2024-02-10T07:44:37.714 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T07:44:37.715 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --description rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} --archive /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/orig.config.yaml 2024-02-10T07:44:37.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:44:37.824 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: null first_in_suite: false job_id: '7555278' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) mon_bind_msgr2: false sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMtn8ILoP4HsAh7Ccwf7LEa4Ga+FAKgUtwKtiedwvBSIwiFxvoItbSI56QhzmKcEEWeDL/2t0sU/IrUVX4ZR+w= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5zvCgoyqaesxlZVllbXdfOTzJx7alf2x1fJNVN5QFM8hnn7LeBY3y4/Bg7B6N3csAJOMjOk9k39nbTE3iCqgo= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_21:34:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T07:44:37.824 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-10T07:44:37.825 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-10T07:44:37.825 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T07:44:37.826 INFO:teuthology.task.internal:Checking packages... 2024-02-10T07:44:37.848 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-10T07:44:37.848 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T07:44:37.849 INFO:teuthology.packaging:ref: None 2024-02-10T07:44:37.849 INFO:teuthology.packaging:tag: None 2024-02-10T07:44:37.849 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T07:44:37.849 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T07:44:37.849 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T07:44:38.018 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-10T07:44:38.019 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T07:44:38.025 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T07:44:38.026 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T07:44:38.074 INFO:teuthology.task.internal:Saving configuration 2024-02-10T07:44:38.086 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T07:44:38.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:44:38.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:39:28.462834', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMtn8ILoP4HsAh7Ccwf7LEa4Ga+FAKgUtwKtiedwvBSIwiFxvoItbSI56QhzmKcEEWeDL/2t0sU/IrUVX4ZR+w='} 2024-02-10T07:44:38.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:39:28.464164', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5zvCgoyqaesxlZVllbXdfOTzJx7alf2x1fJNVN5QFM8hnn7LeBY3y4/Bg7B6N3csAJOMjOk9k39nbTE3iCqgo='} 2024-02-10T07:44:38.141 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T07:44:38.147 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T07:44:38.147 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-10T07:44:38.147 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T07:44:38.241 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f67ed294f70>, signals=[15]) 2024-02-10T07:44:38.242 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T07:44:38.247 INFO:teuthology.task.internal:Opening connections... 2024-02-10T07:44:38.247 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-02-10T07:44:38.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:44:38.326 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-02-10T07:44:38.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:44:38.402 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T07:44:38.409 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-02-10T07:44:38.431 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-02-10T07:44:38.431 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-02-10T07:44:38.491 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T07:44:38.492 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:44:38.493 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-02-10T07:44:38.511 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-02-10T07:44:38.533 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-02-10T07:44:38.534 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T07:44:38.596 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:44:38.597 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:44:38.597 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T07:44:38.597 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:44:38.597 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-02-10T07:44:38.619 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T07:44:38.627 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T07:44:38.649 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T07:44:38.649 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:44:38.652 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:44:38.673 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T07:44:38.678 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T07:44:38.678 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:44:38.713 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:44:38.742 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T07:44:38.797 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T07:44:38.812 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T07:44:38.812 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T07:44:38.812 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:44:38.813 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:44:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.833 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-02-10T07:44:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.834 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-10T07:44:38.894 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-489.el8.x86_64 2024-02-10T07:44:38.894 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-489.el8.x86_64 2024-02-10T07:44:38.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-02-10T07:44:38.896 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-489.el8.x86_64 2024-02-10T07:44:38.896 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-489.el8.x86_64 2024-02-10T07:44:38.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-02-10T07:44:42.857 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 18 MB/s | 27 MB 00:01 2024-02-10T07:44:43.369 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 14 MB/s | 27 MB 00:01 2024-02-10T07:47:54.095 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 56 kB/s | 10 MB 03:06 2024-02-10T07:47:55.871 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 105 kB/s | 18 kB 00:00 2024-02-10T07:47:56.246 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.9 kB 00:00 2024-02-10T07:47:56.697 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 9.6 kB/s | 3.5 kB 00:00 2024-02-10T07:47:56.885 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 56 kB/s | 10 MB 03:08 2024-02-10T07:47:58.794 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 58 kB/s | 18 kB 00:00 2024-02-10T07:47:59.187 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2024-02-10T07:47:59.439 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-02-10T07:47:59.448 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 16 MB 00:02 2024-02-10T07:48:00.164 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-02-10T07:48:03.181 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 453 kB/s | 24 kB 00:00 2024-02-10T07:48:03.906 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 498 kB/s | 24 kB 00:00 2024-02-10T07:48:06.077 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T07:48:06.078 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T07:48:06.190 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T07:48:06.191 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T07:48:06.192 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T07:48:06.192 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2024-02-10T07:48:06.192 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T07:48:06.192 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 90 M 2024-02-10T07:48:06.192 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-02-10T07:48:06.193 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T07:48:06.861 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T07:48:06.861 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T07:48:06.971 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T07:48:06.971 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout:Install 3 Packages 2024-02-10T07:48:06.972 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T07:48:06.973 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 90 M 2024-02-10T07:48:06.973 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 96 M 2024-02-10T07:48:06.973 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T07:54:22.466 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 28 kB/s | 10 MB 06:16 2024-02-10T07:54:36.298 INFO:teuthology.orchestra.run.smithi195.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 27 kB/s | 10 MB 06:29 2024-02-10T08:10:03.573 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 28 kB/s | 36 MB 21:57 2024-02-10T08:10:04.616 INFO:teuthology.orchestra.run.smithi195.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 28 kB/s | 36 MB 21:57 2024-02-10T08:13:58.965 INFO:teuthology.orchestra.run.smithi195.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 29 kB/s | 44 MB 25:51 2024-02-10T08:13:58.966 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:13:58.966 INFO:teuthology.orchestra.run.smithi195.stdout:Total 59 kB/s | 90 MB 25:51 2024-02-10T08:13:59.646 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:14:00.081 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:14:00.081 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:14:01.326 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:14:01.371 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:14:04.226 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:14:04.696 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:08.348 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:08.754 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:12.196 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:12.502 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:12.956 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 28 kB/s | 44 MB 26:06 2024-02-10T08:14:12.956 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:14:12.956 INFO:teuthology.orchestra.run.smithi107.stdout:Total 59 kB/s | 90 MB 26:06 2024-02-10T08:14:13.631 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:14:14.048 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:14:14.048 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:14:15.298 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:14:15.330 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:14:18.111 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:14:18.581 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:22.135 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:22.472 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:25.904 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:26.185 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:34.430 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:34.452 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:34.813 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:34.813 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:34.813 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T08:14:35.408 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:14:35.409 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:14:35.569 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:14:35.838 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:14:35.838 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:14:35.838 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T08:14:35.838 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T08:14:35.838 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T08:14:35.839 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T08:14:35.839 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T08:14:35.839 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:14:36.050 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:14:36.050 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:14:36.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.02 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 129.el8 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Group : System Environment/Base 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 9265586 2024-02-10T08:14:36.084 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Relocations : (not relocatable) 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : CentOS BuildSystem 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T08:14:36.085 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T08:14:36.086 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-02-10T08:14:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:14:36.086 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T08:14:36.087 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T08:14:36.087 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 4.18.0-536.el8.x86_64 2024-02-10T08:14:36.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T08:14:37.974 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-02-10T08:14:39.905 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-02-10T08:14:39.906 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-02-10T08:14:39.931 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.bmTgo9nsX9 2024-02-10T08:14:39.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bmTgo9nsX9 2024-02-10T08:14:40.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.bmTgo9nsX9 2024-02-10T08:14:40.218 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.bmTgo9nsX9 is 6KB 2024-02-10T08:14:40.243 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.bmTgo9nsX9 2024-02-10T08:14:40.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T08:14:40.353 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T08:14:40.353 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T08:14:40.353 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T08:14:40.354 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T08:14:40.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T08:14:40.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-02-10T08:14:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:14:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:14:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:14:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:14:48.844 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:14:48.844 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T08:14:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T08:14:48.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:14:48.845 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:14:48.990 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:14:49.209 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:14:49.210 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:14:49.210 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T08:14:49.210 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T08:14:49.210 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T08:14:49.210 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T08:14:49.210 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T08:14:49.210 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:14:49.421 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:14:49.421 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:14:49.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 129.el8 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T08:14:49.455 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9265586 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:14:49.456 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T08:14:49.457 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T08:14:49.457 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T08:14:49.457 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-02-10T08:14:49.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:14:49.457 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T08:14:49.458 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T08:14:49.458 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-536.el8.x86_64 2024-02-10T08:14:49.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T08:14:51.014 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-02-10T08:14:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-02-10T08:14:52.801 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-02-10T08:14:52.836 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.0q3P4Ne2Mq 2024-02-10T08:14:52.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0q3P4Ne2Mq 2024-02-10T08:14:52.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.0q3P4Ne2Mq 2024-02-10T08:14:53.084 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.0q3P4Ne2Mq is 6KB 2024-02-10T08:14:53.132 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.0q3P4Ne2Mq 2024-02-10T08:14:53.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T08:14:53.223 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T08:14:53.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T08:14:53.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-02-10T08:15:10.602 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T08:15:10.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-10T08:15:10.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:15:23.447 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T08:15:23.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:15:23.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:15:58.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-02-10T08:16:07.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:16:07.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:16:10.608 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T08:16:19.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-10T08:16:19.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:16:20.098 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-10T08:16:20.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-10T08:16:20.400 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-536.el8.x86_64"... 2024-02-10T08:16:20.400 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-10T08:16:20.416 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-536.el8.x86_64 2024-02-10T08:16:20.417 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T08:16:20.417 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T08:16:20.417 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T08:16:21.417 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T08:16:21.418 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T08:16:21.524 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-02-10T08:16:21.537 DEBUG:teuthology.parallel:result is None 2024-02-10T08:16:25.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-02-10T08:16:37.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:16:37.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:16:37.837 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-02-10T08:16:38.144 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:16:38.144 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-536.el8.x86_64"... 2024-02-10T08:16:38.144 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-02-10T08:16:38.161 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-536.el8.x86_64 2024-02-10T08:16:38.161 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T08:16:38.161 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T08:16:38.161 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T08:16:39.162 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T08:16:39.163 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T08:16:39.212 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-02-10T08:16:39.224 DEBUG:teuthology.parallel:result is None 2024-02-10T08:16:39.224 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T08:16:39.232 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T08:16:39.232 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T08:16:39.234 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T08:16:39.257 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T08:16:39.267 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T08:16:39.284 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T08:16:39.285 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T08:16:39.293 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T08:16:39.365 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T08:16:39.372 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T08:16:39.372 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T08:16:39.375 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T08:16:39.413 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:16:39.424 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:16:39.424 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:16:39.434 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:16:39.435 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T08:16:39.442 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T08:16:39.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T08:16:39.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T08:16:39.495 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T08:16:39.506 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T08:16:39.506 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T08:16:39.512 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T08:16:39.555 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:16:39.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:16:39.647 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:16:39.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:16:39.767 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:16:39.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T08:16:39.830 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:16:39.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:16:39.929 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:16:40.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:16:40.058 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:16:40.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T08:16:40.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-02-10T08:16:40.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-02-10T08:16:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:16:40.238 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:16:40.546 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T08:16:40.553 INFO:teuthology.task.internal:Starting timer... 2024-02-10T08:16:40.553 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T08:16:40.605 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T08:16:40.637 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T08:16:40.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-02-10T08:16:40.718 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-02-10T08:16:40.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-02-10T08:16:40.796 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-02-10T08:16:40.799 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T08:16:40.799 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-02-10T08:16:40.823 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-02-10T08:16:40.823 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-02-10T08:16:40.852 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-02-10T08:16:40.853 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-02-10T08:16:40.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:16:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:16:40.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:16:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:16:40.921 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T08:16:40.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-02-10T08:16:40.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-02-10T08:16:41.001 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T08:16:41.011 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T08:16:41.029 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T08:16:41.030 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T08:22:41.092 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-02-10T08:22:41.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:22:41.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:22:41.172 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-02-10T08:22:41.242 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-02-10T08:22:41.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-10T08:22:41.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:22:41.317 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-10T08:22:41.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-10T08:22:41.391 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T08:22:41.403 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T08:22:41.403 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T08:22:41.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:22:41.405 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T08:22:41.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:22:41.435 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T08:22:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T08:22:41.464 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T08:22:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T08:22:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-02-10T08:22:41.527 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-02-10T08:22:41.541 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T08:22:41.542 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-02-10T08:22:41.552 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-02-10T08:22:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T08:22:41.565 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T08:22:41.579 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T08:22:41.625 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.628 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.643 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:22:41.644 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T08:22:41.654 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T08:22:41.654 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2024-02-10T08:22:41.654 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:22:41.655 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2024-02-10T08:22:41.655 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:22:42.451 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 08:21:15 AM UTC. 2024-02-10T08:22:42.466 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:29 ago on Sat 10 Feb 2024 08:21:13 AM UTC. 2024-02-10T08:22:44.429 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:22:44.430 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-10T08:22:44.430 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout:Resetting modules: 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:22:44.432 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:22:44.433 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:44.433 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout:Resetting modules: 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:44.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:22:44.681 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:22:44.698 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:22:45.530 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:32 ago on Sat 10 Feb 2024 08:21:13 AM UTC. 2024-02-10T08:22:45.530 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 08:21:15 AM UTC. 2024-02-10T08:22:47.491 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:22:47.525 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:22:47.525 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:47.525 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:22:47.525 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:47.525 INFO:teuthology.orchestra.run.smithi195.stdout:Installing group/module packages: 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T08:22:47.526 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout:Installing module profiles: 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools/common 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout:Enabling module streams: 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools rhel8 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-02-10T08:22:47.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:22:47.528 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 27 M 2024-02-10T08:22:47.528 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 91 M 2024-02-10T08:22:47.528 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout:Installing group/module packages: 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T08:22:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout:Installing module profiles: 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools/common 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling module streams: 2024-02-10T08:22:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools rhel8 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:22:47.562 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 27 M 2024-02-10T08:22:47.563 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 91 M 2024-02-10T08:22:47.563 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:22:48.228 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.6 MB/s | 1.0 MB 00:00 2024-02-10T08:22:48.394 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 227 kB/s | 38 kB 00:00 2024-02-10T08:22:48.503 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): crun-1.12-1.module_el8+804+f131391c.x86 2.2 MB/s | 248 kB 00:00 2024-02-10T08:22:48.603 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): cockpit-podman-82-1.module_el8+804+f131 760 kB/s | 759 kB 00:00 2024-02-10T08:22:48.665 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.1 MB/s | 1.0 MB 00:00 2024-02-10T08:22:48.912 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 582 kB/s | 179 kB 00:00 2024-02-10T08:22:49.021 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 228 kB/s | 25 kB 00:00 2024-02-10T08:22:49.215 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 69 kB/s | 38 kB 00:00 2024-02-10T08:22:49.229 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 450 kB/s | 94 kB 00:00 2024-02-10T08:22:49.355 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 4.6 MB/s | 4.0 MB 00:00 2024-02-10T08:22:49.356 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): crun-1.12-1.module_el8+804+f131391c.x86 1.7 MB/s | 248 kB 00:00 2024-02-10T08:22:49.582 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): cockpit-podman-82-1.module_el8+804+f131 409 kB/s | 759 kB 00:01 2024-02-10T08:22:49.807 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 798 kB/s | 179 kB 00:00 2024-02-10T08:22:49.855 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): toolbox-0.0.99.4-8.module_el8+804+f1313 5.0 MB/s | 2.5 MB 00:00 2024-02-10T08:22:49.925 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2024-02-10T08:22:49.947 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 575 kB/s | 53 kB 00:00 2024-02-10T08:22:50.025 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 941 kB/s | 94 kB 00:00 2024-02-10T08:22:50.039 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 446 kB/s | 41 kB 00:00 2024-02-10T08:22:51.717 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 1.7 MB/s | 4.0 MB 00:02 2024-02-10T08:22:52.308 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 2.1 MB/s | 9.7 MB 00:04 2024-02-10T08:22:52.509 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 263 kB/s | 53 kB 00:00 2024-02-10T08:22:52.780 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 151 kB/s | 41 kB 00:00 2024-02-10T08:22:52.842 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): toolbox-0.0.99.4-8.module_el8+804+f131 2.2 MB/s | 2.5 MB 00:01 2024-02-10T08:22:53.297 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): skopeo-1.14.0-2.module_el8+804+f131391 2.2 MB/s | 8.8 MB 00:04 2024-02-10T08:22:57.125 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): buildah-1.34.0-1.module_el8+804+f13139 1.0 MB/s | 9.7 MB 00:09 2024-02-10T08:22:57.125 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:22:57.125 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.9 MB/s | 27 MB 00:09 2024-02-10T08:22:57.391 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:22:57.392 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 1.2 MB/s | 8.8 MB 00:07 2024-02-10T08:22:57.392 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:22:57.392 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.8 MB/s | 27 MB 00:09 2024-02-10T08:22:57.411 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:22:57.411 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:22:57.666 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:22:57.679 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:22:57.682 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:22:57.688 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:22:57.688 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:22:57.968 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:22:57.971 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:22:58.386 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:22:58.593 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T08:22:58.683 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:22:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T08:22:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T08:22:58.991 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T08:22:59.138 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T08:22:59.215 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:22:59.321 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T08:22:59.557 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:22:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T08:23:00.595 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T08:23:00.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:23:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T08:23:01.561 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:23:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T08:23:01.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T08:23:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T08:23:02.034 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T08:23:02.277 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T08:23:02.283 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T08:23:02.690 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T08:23:03.213 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:23:03.457 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T08:23:03.637 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:23:03.654 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:23:03.875 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T08:23:04.073 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:23:04.073 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:23:04.074 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T08:23:04.119 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:23:04.557 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:23:04.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T08:23:04.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T08:23:04.557 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:23:04.558 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T08:23:04.617 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:23:04.618 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:23:04.902 DEBUG:teuthology.parallel:result is None 2024-02-10T08:23:05.075 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T08:23:05.076 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:23:05.077 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:23:05.375 DEBUG:teuthology.parallel:result is None 2024-02-10T08:23:05.375 INFO:teuthology.run_tasks:Running task install... 2024-02-10T08:23:05.386 DEBUG:teuthology.task.install:project ceph 2024-02-10T08:23:05.386 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-10T08:23:05.386 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T08:23:05.386 INFO:teuthology.task.install:Using flavor: default 2024-02-10T08:23:05.393 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T08:23:05.393 INFO:teuthology.task.install:extra packages: [] 2024-02-10T08:23:05.393 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-10T08:23:05.394 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:23:05.397 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-10T08:23:05.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:23:05.564 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-10T08:23:05.564 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-10T08:23:05.568 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-10T08:23:05.568 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-10T08:23:05.685 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T08:23:05.685 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:23:05.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T08:23:05.690 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T08:23:05.690 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:23:05.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T08:23:05.715 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T08:23:05.716 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T08:23:05.720 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T08:23:05.720 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T08:23:05.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T08:23:05.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T08:23:05.866 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T08:23:05.867 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T08:23:05.932 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-02-10T08:23:05.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-02-10T08:23:05.938 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-02-10T08:23:05.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-02-10T08:23:06.206 INFO:teuthology.orchestra.run.smithi107.stdout:76 files removed 2024-02-10T08:23:06.216 INFO:teuthology.orchestra.run.smithi195.stdout:76 files removed 2024-02-10T08:23:06.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-02-10T08:23:06.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-02-10T08:23:06.955 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 244 kB/s | 76 kB 00:00 2024-02-10T08:23:06.961 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 246 kB/s | 76 kB 00:00 2024-02-10T08:23:07.370 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2024-02-10T08:23:07.378 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2024-02-10T08:23:07.729 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 4.5 kB/s | 1.4 kB 00:00 2024-02-10T08:23:07.743 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2024-02-10T08:23:17.787 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 2.7 MB/s | 27 MB 00:10 2024-02-10T08:23:28.103 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 1.9 MB/s | 10 MB 00:05 2024-02-10T08:23:29.977 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2024-02-10T08:23:30.411 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-02-10T08:23:32.736 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 1.1 MB/s | 27 MB 00:24 2024-02-10T08:23:33.263 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - PowerTools 1.7 MB/s | 4.8 MB 00:02 2024-02-10T08:23:34.439 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-02-10T08:23:35.548 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-02-10T08:23:39.374 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 352 kB/s | 24 kB 00:00 2024-02-10T08:23:42.885 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-10T08:23:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T08:23:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-10T08:23:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout:Install 21 Packages 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-02-10T08:23:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:23:42.892 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 58 M 2024-02-10T08:23:42.892 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:23:44.233 INFO:teuthology.orchestra.run.smithi107.stdout:(1/23): ceph-base-16.2.14-521.g0e714d9a.el8.x86 5.6 MB/s | 6.1 MB 00:01 2024-02-10T08:23:44.300 INFO:teuthology.orchestra.run.smithi107.stdout:(2/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 385 kB/s | 26 kB 00:00 2024-02-10T08:23:44.369 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 1.5 MB/s | 10 MB 00:06 2024-02-10T08:23:44.441 INFO:teuthology.orchestra.run.smithi107.stdout:(3/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 5.7 MB/s | 825 kB 00:00 2024-02-10T08:23:44.633 INFO:teuthology.orchestra.run.smithi107.stdout:(4/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8. 7.8 MB/s | 12 MB 00:01 2024-02-10T08:23:44.659 INFO:teuthology.orchestra.run.smithi107.stdout:(5/23): libradosstriper1-16.2.14-521.g0e714d9a. 2.2 MB/s | 494 kB 00:00 2024-02-10T08:23:44.734 INFO:teuthology.orchestra.run.smithi107.stdout:(6/23): python3-ceph-argparse-16.2.14-521.g0e71 601 kB/s | 45 kB 00:00 2024-02-10T08:23:44.826 INFO:teuthology.orchestra.run.smithi107.stdout:(7/23): python3-ceph-common-16.2.14-521.g0e714d 1.1 MB/s | 99 kB 00:00 2024-02-10T08:23:44.943 INFO:teuthology.orchestra.run.smithi107.stdout:(8/23): python3-cephfs-16.2.14-521.g0e714d9a.el 1.8 MB/s | 215 kB 00:00 2024-02-10T08:23:45.026 INFO:teuthology.orchestra.run.smithi107.stdout:(9/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_6 9.7 MB/s | 3.8 MB 00:00 2024-02-10T08:23:45.052 INFO:teuthology.orchestra.run.smithi107.stdout:(10/23): python3-rados-16.2.14-521.g0e714d9a.el 3.5 MB/s | 389 kB 00:00 2024-02-10T08:23:45.102 INFO:teuthology.orchestra.run.smithi107.stdout:(11/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 4.8 MB/s | 369 kB 00:00 2024-02-10T08:23:45.136 INFO:teuthology.orchestra.run.smithi107.stdout:(12/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 1.3 MB/s | 114 kB 00:00 2024-02-10T08:23:45.436 INFO:teuthology.orchestra.run.smithi107.stdout:(13/23): ceph-common-16.2.14-521.g0e714d9a.el8. 11 MB/s | 24 MB 00:02 2024-02-10T08:23:46.178 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 84 kB/s | 18 kB 00:00 2024-02-10T08:23:46.253 INFO:teuthology.orchestra.run.smithi107.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 54 kB/s | 44 kB 00:00 2024-02-10T08:23:46.336 INFO:teuthology.orchestra.run.smithi107.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-02-10T08:23:46.362 INFO:teuthology.orchestra.run.smithi107.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 212 kB/s | 259 kB 00:01 2024-02-10T08:23:46.579 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2024-02-10T08:23:46.604 INFO:teuthology.orchestra.run.smithi107.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 749 kB/s | 181 kB 00:00 2024-02-10T08:23:46.629 INFO:teuthology.orchestra.run.smithi107.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2024-02-10T08:23:46.662 INFO:teuthology.orchestra.run.smithi107.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-02-10T08:23:46.688 INFO:teuthology.orchestra.run.smithi107.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2024-02-10T08:23:48.221 INFO:teuthology.orchestra.run.smithi107.stdout:(21/23): librados2-16.2.14-521.g0e714d9a.el8.x8 2.4 MB/s | 3.7 MB 00:01 2024-02-10T08:23:48.246 INFO:teuthology.orchestra.run.smithi107.stdout:(22/23): librdkafka-1.6.1-1.el8.x86_64.rpm 215 kB/s | 677 kB 00:03 2024-02-10T08:23:48.463 INFO:teuthology.orchestra.run.smithi107.stdout:(23/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 2.4 MB/s | 4.2 MB 00:01 2024-02-10T08:23:48.465 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:23:48.465 INFO:teuthology.orchestra.run.smithi107.stdout:Total 10 MB/s | 58 MB 00:05 2024-02-10T08:23:48.510 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:23:48.557 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:23:48.557 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:23:48.928 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:23:48.931 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:23:49.952 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:23:50.202 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-10T08:23:50.305 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - PowerTools 1.3 MB/s | 4.8 MB 00:03 2024-02-10T08:23:50.361 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-10T08:23:50.746 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-10T08:23:50.953 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:23:51.046 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:23:51.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-10T08:23:51.389 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-10T08:23:51.502 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:23:51.895 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:23:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-10T08:23:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-10T08:23:52.165 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2024-02-10T08:23:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-10T08:23:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-10T08:23:52.969 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-10T08:23:53.052 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-10T08:23:53.283 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-10T08:23:53.441 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-10T08:23:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-10T08:23:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-10T08:23:54.186 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-10T08:23:54.335 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-10T08:23:54.396 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-10T08:23:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-10T08:23:54.819 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-10T08:23:55.008 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-10T08:23:55.186 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-10T08:23:55.923 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 367 kB/s | 24 kB 00:00 2024-02-10T08:23:57.705 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:23:57.985 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:23:58.692 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:23:58.985 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-10T08:23:59.125 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-10T08:23:59.259 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:23:59.418 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:23:59.420 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-10T08:23:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-10T08:23:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout:Install 21 Packages 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-02-10T08:23:59.423 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:23:59.424 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 58 M 2024-02-10T08:23:59.424 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:24:00.800 INFO:teuthology.orchestra.run.smithi195.stdout:(1/23): ceph-base-16.2.14-521.g0e714d9a.el8.x86 5.7 MB/s | 6.1 MB 00:01 2024-02-10T08:24:00.866 INFO:teuthology.orchestra.run.smithi195.stdout:(2/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 385 kB/s | 26 kB 00:00 2024-02-10T08:24:01.008 INFO:teuthology.orchestra.run.smithi195.stdout:(3/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 5.7 MB/s | 825 kB 00:00 2024-02-10T08:24:01.275 INFO:teuthology.orchestra.run.smithi195.stdout:(4/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8. 7.4 MB/s | 12 MB 00:01 2024-02-10T08:24:01.301 INFO:teuthology.orchestra.run.smithi195.stdout:(5/23): libradosstriper1-16.2.14-521.g0e714d9a. 1.7 MB/s | 494 kB 00:00 2024-02-10T08:24:01.376 INFO:teuthology.orchestra.run.smithi195.stdout:(6/23): python3-ceph-argparse-16.2.14-521.g0e71 599 kB/s | 45 kB 00:00 2024-02-10T08:24:01.468 INFO:teuthology.orchestra.run.smithi195.stdout:(7/23): python3-ceph-common-16.2.14-521.g0e714d 1.1 MB/s | 99 kB 00:00 2024-02-10T08:24:01.568 INFO:teuthology.orchestra.run.smithi195.stdout:(8/23): python3-cephfs-16.2.14-521.g0e714d9a.el 2.1 MB/s | 215 kB 00:00 2024-02-10T08:24:01.643 INFO:teuthology.orchestra.run.smithi195.stdout:(9/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_6 10 MB/s | 3.8 MB 00:00 2024-02-10T08:24:01.669 INFO:teuthology.orchestra.run.smithi195.stdout:(10/23): python3-rados-16.2.14-521.g0e714d9a.el 3.8 MB/s | 389 kB 00:00 2024-02-10T08:24:01.711 INFO:teuthology.orchestra.run.smithi195.stdout:(11/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 5.4 MB/s | 369 kB 00:00 2024-02-10T08:24:01.728 INFO:teuthology.orchestra.run.smithi195.stdout:(12/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 1.9 MB/s | 114 kB 00:00 2024-02-10T08:24:02.137 INFO:teuthology.orchestra.run.smithi195.stdout:(13/23): ceph-common-16.2.14-521.g0e714d9a.el8. 10 MB/s | 24 MB 00:02 2024-02-10T08:24:02.454 INFO:teuthology.orchestra.run.smithi195.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 139 kB/s | 44 kB 00:00 2024-02-10T08:24:02.570 INFO:teuthology.orchestra.run.smithi195.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 404 kB/s | 47 kB 00:00 2024-02-10T08:24:02.879 INFO:teuthology.orchestra.run.smithi195.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 225 kB/s | 259 kB 00:01 2024-02-10T08:24:02.996 INFO:teuthology.orchestra.run.smithi195.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 720 kB/s | 306 kB 00:00 2024-02-10T08:24:03.067 INFO:teuthology.orchestra.run.smithi195.stdout:(18/23): librdkafka-1.6.1-1.el8.x86_64.rpm 499 kB/s | 677 kB 00:01 2024-02-10T08:24:03.147 INFO:teuthology.orchestra.run.smithi195.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 391 kB/s | 59 kB 00:00 2024-02-10T08:24:03.172 INFO:teuthology.orchestra.run.smithi195.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 618 kB/s | 181 kB 00:00 2024-02-10T08:24:03.289 INFO:teuthology.orchestra.run.smithi195.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 339 kB/s | 75 kB 00:00 2024-02-10T08:24:03.623 INFO:teuthology.orchestra.run.smithi195.stdout:(22/23): librados2-16.2.14-521.g0e714d9a.el8.x8 7.9 MB/s | 3.7 MB 00:00 2024-02-10T08:24:03.681 INFO:teuthology.orchestra.run.smithi195.stdout:(23/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 8.2 MB/s | 4.2 MB 00:00 2024-02-10T08:24:03.683 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:03.683 INFO:teuthology.orchestra.run.smithi195.stdout:Total 14 MB/s | 58 MB 00:04 2024-02-10T08:24:03.728 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:24:03.775 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:24:03.775 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:24:04.137 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:24:04.140 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:24:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:24:05.408 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-10T08:24:05.578 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-10T08:24:05.956 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-10T08:24:06.177 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:24:06.272 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:24:06.512 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-10T08:24:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:24:07.166 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:24:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-10T08:24:07.411 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-10T08:24:07.644 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-10T08:24:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-10T08:24:08.527 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-10T08:24:08.593 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-10T08:24:08.775 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-10T08:24:08.975 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-10T08:24:09.148 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-10T08:24:09.354 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-10T08:24:09.561 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-10T08:24:09.719 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-10T08:24:09.790 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-10T08:24:09.995 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-10T08:24:10.185 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-10T08:24:10.440 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-10T08:24:10.619 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-10T08:24:13.166 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:24:13.393 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:24:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-10T08:24:14.366 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-10T08:24:14.447 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-10T08:24:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:24:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:24:18.117 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-02-10T08:24:18.117 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-02-10T08:24:18.117 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2024-02-10T08:24:18.117 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /run 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2024-02-10T08:24:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:19.315 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-10T08:24:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-10T08:24:19.390 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-10T08:24:19.591 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-10T08:24:19.591 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:20.289 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:20.289 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-10T08:24:20.289 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-10T08:24:20.289 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-10T08:24:20.290 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-10T08:24:20.291 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-10T08:24:20.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:24:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T08:24:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:21.175 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:24:21.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-02-10T08:24:22.219 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:43 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:24:23.705 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:24:23.706 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-02-10T08:24:23.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:23.708 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 53 M 2024-02-10T08:24:23.708 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 197 M 2024-02-10T08:24:23.708 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:24:24.130 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 753 kB/s | 174 kB 00:00 2024-02-10T08:24:24.489 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 525 kB/s | 188 kB 00:00 2024-02-10T08:24:24.664 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-02-10T08:24:24.764 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-02-10T08:24:24.875 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): jq-1.6-8.el8.x86_64.rpm 208 kB/s | 203 kB 00:00 2024-02-10T08:24:26.590 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 19 MB/s | 52 MB 00:02 2024-02-10T08:24:26.592 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:26.592 INFO:teuthology.orchestra.run.smithi107.stdout:Total 18 MB/s | 53 MB 00:02 2024-02-10T08:24:26.634 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:24:26.655 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:24:26.655 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:24:26.941 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:24:26.944 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:24:27.434 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:24:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T08:24:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T08:24:27.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:24:28.019 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:24:28.250 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T08:24:28.403 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-10T08:24:32.989 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-10T08:24:33.259 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T08:24:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T08:24:33.841 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:24:33.841 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-02-10T08:24:33.841 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2024-02-10T08:24:33.842 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-8.el8.x86_64 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-10T08:24:33.974 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-10T08:24:33.975 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T08:24:33.975 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:33.975 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:24:34.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-02-10T08:24:34.947 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:24:35.004 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-10T08:24:35.107 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-10T08:24:35.107 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-10T08:24:35.292 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-10T08:24:35.292 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:35.912 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:35.912 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-10T08:24:35.912 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-10T08:24:35.913 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-10T08:24:35.914 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-10T08:24:36.465 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:24:36.468 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:24:36.468 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:36.468 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T08:24:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T08:24:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout:Install 32 Packages 2024-02-10T08:24:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:36.472 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 32 M 2024-02-10T08:24:36.472 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 104 M 2024-02-10T08:24:36.472 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:24:36.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-10T08:24:36.759 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:36.759 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-02-10T08:24:36.759 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T08:24:36.760 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:36.761 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:36.762 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:24:36.857 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 40 kB/s | 6.4 kB 00:00 2024-02-10T08:24:37.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-02-10T08:24:37.198 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2024-02-10T08:24:37.812 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:42 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:24:37.924 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86_ 2.0 MB/s | 2.4 MB 00:01 2024-02-10T08:24:38.041 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): ceph-mgr-modules-core-16.2.14-521.g0e71 2.0 MB/s | 245 kB 00:00 2024-02-10T08:24:38.249 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86_ 3.7 MB/s | 5.2 MB 00:01 2024-02-10T08:24:38.433 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86_ 15 MB/s | 19 MB 00:01 2024-02-10T08:24:38.558 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 303 kB/s | 157 kB 00:00 2024-02-10T08:24:38.683 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 456 kB/s | 57 kB 00:00 2024-02-10T08:24:38.709 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 84 kB/s | 39 kB 00:00 2024-02-10T08:24:38.884 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 230 kB/s | 103 kB 00:00 2024-02-10T08:24:39.084 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.1 MB/s | 457 kB 00:00 2024-02-10T08:24:39.184 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.1 MB/s | 109 kB 00:00 2024-02-10T08:24:39.321 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:24:39.322 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:39.322 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:24:39.322 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:39.323 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-10T08:24:39.324 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:39.324 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 53 M 2024-02-10T08:24:39.324 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 197 M 2024-02-10T08:24:39.324 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:24:39.435 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-02-10T08:24:39.501 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 138 kB/s | 44 kB 00:00 2024-02-10T08:24:39.582 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 805 kB/s | 174 kB 00:00 2024-02-10T08:24:39.732 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 555 kB/s | 203 kB 00:00 2024-02-10T08:24:39.752 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 690 kB/s | 173 kB 00:00 2024-02-10T08:24:39.835 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 463 kB/s | 185 kB 00:00 2024-02-10T08:24:39.941 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2024-02-10T08:24:39.944 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-02-10T08:24:39.969 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 197 kB/s | 249 kB 00:01 2024-02-10T08:24:39.994 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2024-02-10T08:24:40.041 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 708 kB/s | 71 kB 00:00 2024-02-10T08:24:40.053 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-10T08:24:40.078 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 447 kB/s | 49 kB 00:00 2024-02-10T08:24:40.124 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 347 kB/s | 188 kB 00:00 2024-02-10T08:24:40.187 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2024-02-10T08:24:40.295 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 173 kB/s | 19 kB 00:00 2024-02-10T08:24:40.321 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2024-02-10T08:24:40.346 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): python3-more-itertools-7.2.0-3.el8.noa 167 kB/s | 59 kB 00:00 2024-02-10T08:24:40.405 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 219 kB/s | 24 kB 00:00 2024-02-10T08:24:40.430 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-02-10T08:24:40.455 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-02-10T08:24:40.514 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-02-10T08:24:40.606 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-02-10T08:24:40.631 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-02-10T08:24:40.656 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 426 kB/s | 85 kB 00:00 2024-02-10T08:24:40.657 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:40.658 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.6 MB/s | 32 MB 00:04 2024-02-10T08:24:40.740 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:24:40.779 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:24:40.779 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:24:41.108 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:24:41.111 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:24:42.141 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 19 MB/s | 52 MB 00:02 2024-02-10T08:24:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Total 19 MB/s | 53 MB 00:02 2024-02-10T08:24:42.186 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:24:42.207 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:24:42.207 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:24:42.220 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:24:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T08:24:42.495 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:24:42.498 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:24:42.674 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T08:24:42.967 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:24:43.000 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T08:24:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T08:24:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T08:24:43.366 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T08:24:43.487 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T08:24:43.547 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:24:43.622 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:24:43.694 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T08:24:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T08:24:43.886 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T08:24:43.972 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-10T08:24:44.088 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T08:24:44.306 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T08:24:44.560 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T08:24:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T08:24:45.072 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T08:24:45.467 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T08:24:45.701 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T08:24:45.890 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T08:24:46.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T08:24:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:24:46.764 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T08:24:47.004 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T08:24:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T08:24:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T08:24:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T08:24:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T08:24:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T08:24:48.507 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T08:24:48.584 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-10T08:24:48.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-10T08:24:48.850 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T08:24:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-10T08:24:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-10T08:24:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-10T08:24:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T08:24:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T08:24:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T08:24:49.203 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-10T08:24:49.408 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-10T08:24:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: jq-1.6-8.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:49.583 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:24:49.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-02-10T08:24:50.545 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:24:50.926 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-10T08:24:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-10T08:24:51.591 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-10T08:24:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-10T08:24:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-10T08:24:52.041 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T08:24:52.045 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T08:24:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:24:52.047 INFO:teuthology.orchestra.run.smithi195.stdout:Install 32 Packages 2024-02-10T08:24:52.048 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:24:52.048 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 32 M 2024-02-10T08:24:52.048 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 104 M 2024-02-10T08:24:52.048 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:24:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-10T08:24:52.241 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-10T08:24:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-10T08:24:52.577 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 46 kB/s | 6.4 kB 00:00 2024-02-10T08:24:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T08:24:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T08:24:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T08:24:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T08:24:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T08:24:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T08:24:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T08:24:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T08:24:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T08:24:52.928 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 3.3 MB/s | 1.6 MB 00:00 2024-02-10T08:24:53.020 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86_ 4.2 MB/s | 2.4 MB 00:00 2024-02-10T08:24:53.095 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): ceph-mgr-modules-core-16.2.14-521.g0e71 3.2 MB/s | 245 kB 00:00 2024-02-10T08:24:53.404 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86_ 6.3 MB/s | 5.2 MB 00:00 2024-02-10T08:24:53.620 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 179 kB/s | 39 kB 00:00 2024-02-10T08:24:53.738 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 244 kB/s | 157 kB 00:00 2024-02-10T08:24:53.788 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 619 kB/s | 103 kB 00:00 2024-02-10T08:24:53.944 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T08:24:53.944 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:53.944 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:24:53.944 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:53.944 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T08:24:53.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T08:24:53.946 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T08:24:53.947 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T08:24:53.947 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:53.947 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:24:54.064 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 175 kB/s | 57 kB 00:00 2024-02-10T08:24:54.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-02-10T08:24:54.263 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 963 kB/s | 457 kB 00:00 2024-02-10T08:24:54.506 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86 12 MB/s | 19 MB 00:01 2024-02-10T08:24:54.531 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 533 kB/s | 249 kB 00:00 2024-02-10T08:24:54.573 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2024-02-10T08:24:54.772 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.2 MB/s | 641 kB 00:00 2024-02-10T08:24:54.906 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 116 kB/s | 44 kB 00:00 2024-02-10T08:24:54.948 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:15 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:24:55.057 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 383 kB/s | 185 kB 00:00 2024-02-10T08:24:55.224 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 382 kB/s | 173 kB 00:00 2024-02-10T08:24:55.257 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 55 kB/s | 11 kB 00:00 2024-02-10T08:24:55.283 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-02-10T08:24:55.408 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 96 kB/s | 18 kB 00:00 2024-02-10T08:24:55.500 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 201 kB/s | 49 kB 00:00 2024-02-10T08:24:55.525 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 242 kB/s | 59 kB 00:00 2024-02-10T08:24:55.692 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 995 kB/s | 283 kB 00:00 2024-02-10T08:24:55.717 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 97 kB/s | 19 kB 00:00 2024-02-10T08:24:55.743 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): python3-portend-2.6-1.el8.noarch.rpm 64 kB/s | 16 kB 00:00 2024-02-10T08:24:55.810 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-02-10T08:24:55.885 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 175 kB/s | 29 kB 00:00 2024-02-10T08:24:55.910 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 164 kB/s | 27 kB 00:00 2024-02-10T08:24:55.986 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-02-10T08:24:56.036 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 682 kB/s | 85 kB 00:00 2024-02-10T08:24:56.069 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-02-10T08:24:56.181 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 118 kB/s | 23 kB 00:00 2024-02-10T08:24:56.182 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:56.182 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.7 MB/s | 32 MB 00:04 2024-02-10T08:24:56.264 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:24:56.303 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:24:56.303 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:24:56.390 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:24:56.437 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:24:56.437 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:24:56.437 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:24:56.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-02-10T08:24:56.624 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:24:56.626 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:24:57.282 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:18 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:24:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:24:57.982 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T08:24:58.219 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T08:24:58.536 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T08:24:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T08:24:58.801 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:24:58.802 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:24:58.802 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:58.802 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:24:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 85 k 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 352 k 2024-02-10T08:24:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:24:59.032 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T08:24:59.201 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 214 kB/s | 85 kB 00:00 2024-02-10T08:24:59.202 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:24:59.202 INFO:teuthology.orchestra.run.smithi107.stdout:Total 213 kB/s | 85 kB 00:00 2024-02-10T08:24:59.202 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:24:59.209 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:24:59.210 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:24:59.211 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:24:59.215 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:24:59.237 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T08:24:59.255 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:24:59.422 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T08:24:59.600 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T08:24:59.712 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:24:59.868 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T08:24:59.927 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:00.108 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T08:25:00.154 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:00.326 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T08:25:00.600 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T08:25:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:00.607 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:00.607 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:00.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T08:25:00.994 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T08:25:01.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T08:25:01.493 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T08:25:01.552 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:01.768 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T08:25:02.028 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:25:02.344 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T08:25:02.624 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T08:25:02.804 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T08:25:03.026 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T08:25:03.061 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-10T08:25:03.062 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 161 k 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 470 k 2024-02-10T08:25:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:03.291 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 709 kB/s | 161 kB 00:00 2024-02-10T08:25:03.291 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:03.291 INFO:teuthology.orchestra.run.smithi107.stdout:Total 705 kB/s | 161 kB 00:00 2024-02-10T08:25:03.292 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:03.302 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:03.302 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:03.324 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T08:25:03.349 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:03.352 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:03.540 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:03.543 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T08:25:03.754 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T08:25:04.108 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T08:25:04.421 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T08:25:04.827 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-10T08:25:05.041 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-10T08:25:05.420 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:05.850 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:06.217 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:06.217 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:06.217 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:06.218 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:06.218 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:06.218 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:06.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-02-10T08:25:06.570 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-10T08:25:06.827 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-10T08:25:07.205 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:28 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-10T08:25:07.510 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-10T08:25:07.727 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-10T08:25:07.927 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-10T08:25:07.944 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-10T08:25:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-10T08:25:08.553 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T08:25:08.554 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T08:25:08.555 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T08:25:08.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T08:25:08.556 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T08:25:08.671 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:25:08.718 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:08.719 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:25:08.719 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:08.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T08:25:09.570 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:10.246 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T08:25:10.246 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:10.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T08:25:10.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T08:25:10.248 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:10.249 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:10.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-02-10T08:25:11.089 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T08:25:11.090 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.4 M 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 22 M 2024-02-10T08:25:11.091 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:11.314 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:16 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:11.320 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-grafana-dashboards-16.2.14-521.g0e7 152 kB/s | 24 kB 00:00 2024-02-10T08:25:11.345 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-prometheus-alerts-16.2.14-521.g0e71 80 kB/s | 15 kB 00:00 2024-02-10T08:25:11.429 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 393 kB/s | 43 kB 00:00 2024-02-10T08:25:11.571 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 7.6 MB/s | 3.1 MB 00:00 2024-02-10T08:25:11.596 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2024-02-10T08:25:11.863 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 453 kB/s | 196 kB 00:00 2024-02-10T08:25:11.866 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:11.866 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.4 MB/s | 3.4 MB 00:00 2024-02-10T08:25:11.875 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:11.885 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:11.885 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:12.044 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:12.047 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:12.313 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:12.582 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T08:25:12.784 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:25:12.831 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:12.832 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:25:12.832 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:12.835 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T08:25:12.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-02-10T08:25:13.037 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T08:25:13.183 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-10T08:25:13.686 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:18 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:14.219 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-10T08:25:14.535 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-10T08:25:14.807 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-10T08:25:14.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-10T08:25:14.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-10T08:25:14.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-10T08:25:14.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T08:25:14.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T08:25:15.188 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-10T08:25:15.189 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 85 k 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 352 k 2024-02-10T08:25:15.190 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:25:15.192 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:15.193 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:15.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T08:25:15.403 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 401 kB/s | 85 kB 00:00 2024-02-10T08:25:15.404 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:15.404 INFO:teuthology.orchestra.run.smithi195.stdout:Total 398 kB/s | 85 kB 00:00 2024-02-10T08:25:15.404 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:25:15.411 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:25:15.411 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:25:15.412 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:25:15.416 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:25:15.462 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:25:16.029 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:16.057 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:16.235 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:16.470 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:16.821 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:17.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T08:25:17.557 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T08:25:17.558 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout:Install 8 Packages 2024-02-10T08:25:17.559 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:17.560 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2024-02-10T08:25:17.560 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 210 M 2024-02-10T08:25:17.560 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:17.785 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:18.311 INFO:teuthology.orchestra.run.smithi107.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-52 13 MB/s | 8.6 MB 00:00 2024-02-10T08:25:19.299 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:19.299 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:25:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:19.301 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 161 k 2024-02-10T08:25:19.301 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 470 k 2024-02-10T08:25:19.301 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:25:19.542 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 670 kB/s | 161 kB 00:00 2024-02-10T08:25:19.542 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:19.542 INFO:teuthology.orchestra.run.smithi195.stdout:Total 667 kB/s | 161 kB 00:00 2024-02-10T08:25:19.543 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:25:19.553 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:25:19.554 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:25:19.601 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:25:19.604 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:25:19.717 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:25:19.899 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:21.517 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:22.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-02-10T08:25:22.921 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:23.078 INFO:teuthology.orchestra.run.smithi107.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 890 kB/s | 4.7 MB 00:05 2024-02-10T08:25:23.170 INFO:teuthology.orchestra.run.smithi107.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-10T08:25:24.362 INFO:teuthology.orchestra.run.smithi107.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 632 kB/s | 3.7 MB 00:06 2024-02-10T08:25:24.367 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:25:24.415 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:24.416 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:25:24.416 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:24.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T08:25:24.546 INFO:teuthology.orchestra.run.smithi107.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-02-10T08:25:24.596 INFO:teuthology.orchestra.run.smithi107.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-02-10T08:25:25.277 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:25.679 INFO:teuthology.orchestra.run.smithi107.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 618 kB/s | 4.9 MB 00:08 2024-02-10T08:25:26.363 INFO:teuthology.orchestra.run.smithi107.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 4.4 MB/s | 14 MB 00:03 2024-02-10T08:25:26.364 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:26.364 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.2 MB/s | 37 MB 00:08 2024-02-10T08:25:26.596 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:26.614 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:26.614 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:26.787 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:26.788 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:26.788 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:25:26.788 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:26.788 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:25:26.788 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-10T08:25:26.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:26.790 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.4 M 2024-02-10T08:25:26.790 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 22 M 2024-02-10T08:25:26.790 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:25:26.939 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:26.942 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:27.497 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:27.571 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-prometheus-alerts-16.2.14-521.g0e71 37 kB/s | 15 kB 00:00 2024-02-10T08:25:27.596 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-grafana-dashboards-16.2.14-521.g0e7 57 kB/s | 24 kB 00:00 2024-02-10T08:25:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:25:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:25:27.863 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 126 kB/s | 34 kB 00:00 2024-02-10T08:25:27.963 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 109 kB/s | 43 kB 00:00 2024-02-10T08:25:27.973 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:25:28.131 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 736 kB/s | 196 kB 00:00 2024-02-10T08:25:28.289 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 2.8 MB/s | 3.1 MB 00:01 2024-02-10T08:25:28.291 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:28.291 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.3 MB/s | 3.4 MB 00:01 2024-02-10T08:25:28.303 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:25:28.314 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:25:28.314 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:25:28.481 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:25:28.484 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:25:28.576 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:25:28.739 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:28.754 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:25:29.004 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T08:25:29.249 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T08:25:29.353 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:29.450 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T08:25:29.534 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:25:29.604 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-10T08:25:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:25:30.600 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:25:30.643 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-10T08:25:30.813 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:25:30.961 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-10T08:25:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-10T08:25:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-10T08:25:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-10T08:25:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-10T08:25:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T08:25:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T08:25:31.611 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T08:25:31.611 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:31.611 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:25:31.611 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:31.612 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:31.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T08:25:32.522 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:25:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T08:25:34.021 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-10T08:25:34.042 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T08:25:34.043 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout:Install 8 Packages 2024-02-10T08:25:34.044 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:34.045 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 37 M 2024-02-10T08:25:34.045 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 210 M 2024-02-10T08:25:34.045 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T08:25:34.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T08:25:34.723 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T08:25:34.723 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:34.724 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:34.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-02-10T08:25:35.328 INFO:teuthology.orchestra.run.smithi195.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-52 7.1 MB/s | 8.6 MB 00:01 2024-02-10T08:25:35.588 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:56 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:37.092 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:37.093 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T08:25:37.094 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-02-10T08:25:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:37.096 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.9 M 2024-02-10T08:25:37.096 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2024-02-10T08:25:37.096 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:37.119 INFO:teuthology.orchestra.run.smithi195.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.1 MB/s | 3.7 MB 00:01 2024-02-10T08:25:37.663 INFO:teuthology.orchestra.run.smithi195.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 414 kB/s | 225 kB 00:00 2024-02-10T08:25:37.846 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 158 kB/s | 27 kB 00:00 2024-02-10T08:25:37.954 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 71 kB/s | 20 kB 00:00 2024-02-10T08:25:37.979 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 86 kB/s | 27 kB 00:00 2024-02-10T08:25:38.105 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.2 MB/s | 155 kB 00:00 2024-02-10T08:25:38.147 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2024-02-10T08:25:38.239 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 360 kB/s | 33 kB 00:00 2024-02-10T08:25:38.281 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 389 kB/s | 16 kB 00:00 2024-02-10T08:25:38.306 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 274 kB/s | 126 kB 00:00 2024-02-10T08:25:38.332 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.6 MB/s | 81 kB 00:00 2024-02-10T08:25:38.374 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.7 MB/s | 64 kB 00:00 2024-02-10T08:25:38.399 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 247 kB/s | 110 kB 00:00 2024-02-10T08:25:38.424 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2024-02-10T08:25:38.608 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2024-02-10T08:25:38.610 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:38.610 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-02-10T08:25:38.667 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:38.683 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:38.683 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:38.770 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:38.772 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:39.339 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:39.547 INFO:teuthology.orchestra.run.smithi195.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 917 kB/s | 4.9 MB 00:05 2024-02-10T08:25:39.573 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T08:25:39.762 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T08:25:39.981 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T08:25:40.243 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T08:25:40.380 INFO:teuthology.orchestra.run.smithi195.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 775 kB/s | 645 kB 00:00 2024-02-10T08:25:40.492 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T08:25:40.547 INFO:teuthology.orchestra.run.smithi195.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-02-10T08:25:40.766 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T08:25:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T08:25:41.081 INFO:teuthology.orchestra.run.smithi195.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 4.1 MB/s | 14 MB 00:03 2024-02-10T08:25:41.332 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T08:25:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T08:25:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T08:25:42.884 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T08:25:43.094 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T08:25:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-10T08:25:43.584 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T08:25:43.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T08:25:43.831 INFO:teuthology.orchestra.run.smithi195.stdout:(8/8): openblas-0.3.15-4.el8.x86_64.rpm 499 kB/s | 4.7 MB 00:09 2024-02-10T08:25:43.831 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:43.832 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.8 MB/s | 37 MB 00:09 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T08:25:44.047 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:44.061 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:25:44.079 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:25:44.079 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:25:44.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T08:25:44.419 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:25:44.421 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:25:44.903 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:05 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:25:45.162 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:25:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:25:45.445 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:25:46.045 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:25:46.204 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:46.411 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T08:25:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.8 M 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2024-02-10T08:25:46.413 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:46.867 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 501 kB/s | 116 kB 00:00 2024-02-10T08:25:46.976 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:25:47.184 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 67 kB/s | 21 kB 00:00 2024-02-10T08:25:47.337 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 766 kB/s | 538 kB 00:00 2024-02-10T08:25:47.428 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2024-02-10T08:25:47.545 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 235 kB/s | 49 kB 00:00 2024-02-10T08:25:47.671 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:25:48.057 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:25:48.302 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:25:49.245 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 1.8 MB/s | 4.8 MB 00:02 2024-02-10T08:25:49.248 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:49.248 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.1 MB/s | 5.8 MB 00:02 2024-02-10T08:25:49.301 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:49.312 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:49.312 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:49.417 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:49.420 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:50.000 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T08:25:50.067 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:25:50.329 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T08:25:51.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T08:25:51.367 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T08:25:51.620 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-10T08:25:51.638 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T08:25:51.891 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:25:51.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T08:25:51.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T08:25:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T08:25:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T08:25:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:52.334 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T08:25:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T08:25:52.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-10T08:25:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-10T08:25:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:52.810 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:52.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-02-10T08:25:53.205 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:58 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:25:53.698 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:14 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:54.729 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T08:25:54.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-02-10T08:25:54.732 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:25:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.9 M 2024-02-10T08:25:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2024-02-10T08:25:54.733 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:25:55.224 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:55.225 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-02-10T08:25:55.226 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:55.226 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 994 k 2024-02-10T08:25:55.226 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.8 M 2024-02-10T08:25:55.226 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:25:55.270 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 160 kB/s | 27 kB 00:00 2024-02-10T08:25:55.328 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 116 kB/s | 27 kB 00:00 2024-02-10T08:25:55.354 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 78 kB/s | 20 kB 00:00 2024-02-10T08:25:55.446 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.6 MB/s | 155 kB 00:00 2024-02-10T08:25:55.479 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2024-02-10T08:25:55.504 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 954 kB/s | 83 kB 00:00 2024-02-10T08:25:55.547 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 505 kB/s | 110 kB 00:00 2024-02-10T08:25:55.655 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 328 kB/s | 126 kB 00:00 2024-02-10T08:25:55.746 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 2.7 MB/s | 911 kB 00:00 2024-02-10T08:25:55.747 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:55.747 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.9 MB/s | 994 kB 00:00 2024-02-10T08:25:55.757 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:25:55.777 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:25:55.777 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:25:55.862 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:25:55.865 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:25:55.872 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 375 kB/s | 81 kB 00:00 2024-02-10T08:25:55.897 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 46 kB/s | 16 kB 00:00 2024-02-10T08:25:55.923 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-cachetools-3.1.1-4.el8.noarch. 74 kB/s | 33 kB 00:00 2024-02-10T08:25:56.015 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 667 kB/s | 61 kB 00:00 2024-02-10T08:25:56.040 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 452 kB/s | 64 kB 00:00 2024-02-10T08:25:56.048 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:25:56.324 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T08:25:56.468 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.0 MB/s | 1.2 MB 00:00 2024-02-10T08:25:56.468 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:25:56.468 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-02-10T08:25:56.523 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:25:56.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-10T08:25:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:25:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:25:56.629 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:25:56.632 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:25:56.928 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-10T08:25:56.928 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-10T08:25:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:25:57.479 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T08:25:57.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-02-10T08:25:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T08:25:57.945 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T08:25:58.183 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T08:25:58.295 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:19 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:25:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T08:25:58.674 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T08:25:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T08:25:59.249 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T08:25:59.804 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:25:59.805 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-02-10T08:25:59.805 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:59.805 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:25:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 131 k 2024-02-10T08:25:59.806 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 432 k 2024-02-10T08:25:59.807 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:26:00.010 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 645 kB/s | 131 kB 00:00 2024-02-10T08:26:00.010 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:00.011 INFO:teuthology.orchestra.run.smithi107.stdout:Total 641 kB/s | 131 kB 00:00 2024-02-10T08:26:00.011 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:26:00.016 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:26:00.016 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:26:00.066 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:26:00.069 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:26:00.186 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:26:00.293 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T08:26:00.415 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:00.608 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T08:26:00.725 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:00.808 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T08:26:00.993 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:01.087 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:01.142 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-10T08:26:01.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T08:26:01.482 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T08:26:01.483 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T08:26:01.483 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T08:26:01.930 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T08:26:01.931 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T08:26:01.932 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T08:26:01.932 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T08:26:01.932 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:01.932 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:02.059 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:23 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:02.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T08:26:02.845 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:07 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:03.502 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:03.548 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:03.550 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:03.550 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:04.346 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:04.347 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:04.347 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:04.348 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.8 M 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2024-02-10T08:26:04.349 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:04.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-02-10T08:26:05.171 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:26 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:05.375 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 225 kB/s | 116 kB 00:00 2024-02-10T08:26:06.175 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 408 kB/s | 538 kB 00:01 2024-02-10T08:26:06.567 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 18 kB/s | 21 kB 00:01 2024-02-10T08:26:06.683 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:06.683 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:26:06.684 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:06.686 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 25 k 2024-02-10T08:26:06.686 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 107 k 2024-02-10T08:26:06.686 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:26:06.868 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 163 kB/s | 49 kB 00:00 2024-02-10T08:26:06.909 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 114 kB/s | 25 kB 00:00 2024-02-10T08:26:06.910 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:06.910 INFO:teuthology.orchestra.run.smithi107.stdout:Total 114 kB/s | 25 kB 00:00 2024-02-10T08:26:06.910 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:26:06.914 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:26:06.914 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:26:06.916 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:26:06.919 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:26:06.967 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:26:07.001 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 431 kB/s | 355 kB 00:00 2024-02-10T08:26:07.180 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:07.449 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:07.794 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:07.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-02-10T08:26:08.653 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:29 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:10.105 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:10.151 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:10.152 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:10.152 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:10.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-02-10T08:26:10.851 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 818 kB/s | 4.8 MB 00:05 2024-02-10T08:26:10.852 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:10.852 INFO:teuthology.orchestra.run.smithi195.stdout:Total 920 kB/s | 5.8 MB 00:06 2024-02-10T08:26:10.905 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:10.915 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:10.915 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:10.998 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:31 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:11.021 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:11.024 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:11.347 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T08:26:11.927 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T08:26:12.461 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:12.506 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:12.508 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:12.508 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:12.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-02-10T08:26:12.881 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T08:26:13.237 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T08:26:13.349 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:34 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:13.515 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T08:26:13.708 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-10T08:26:14.002 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-10T08:26:14.002 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-10T08:26:14.003 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T08:26:14.003 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T08:26:14.003 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T08:26:14.003 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T08:26:14.396 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:14.397 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:14.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-02-10T08:26:14.824 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:14.869 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:14.871 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:14.871 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:14.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-02-10T08:26:15.284 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:20 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:15.725 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:36 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:16.816 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:16.816 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:16.816 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-10T08:26:16.816 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:16.816 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:16.817 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 994 k 2024-02-10T08:26:16.818 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.8 M 2024-02-10T08:26:16.818 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:16.960 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 877 kB/s | 83 kB 00:00 2024-02-10T08:26:17.202 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 2.6 MB/s | 911 kB 00:00 2024-02-10T08:26:17.203 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:17.203 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:17.203 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.5 MB/s | 994 kB 00:00 2024-02-10T08:26:17.212 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:17.232 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:17.232 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:17.248 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:17.249 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:17.250 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:17.318 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:17.321 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:17.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-02-10T08:26:17.536 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:17.814 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T08:26:18.007 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-10T08:26:18.093 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:39 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:18.408 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-10T08:26:18.408 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:18.779 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:19.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-02-10T08:26:19.546 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:19.592 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:19.593 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:19.593 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:19.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-02-10T08:26:19.790 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:24 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:20.443 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:41 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:21.311 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:21.311 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:21.311 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:26:21.312 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:21.313 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 131 k 2024-02-10T08:26:21.313 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 432 k 2024-02-10T08:26:21.313 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:21.543 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 569 kB/s | 131 kB 00:00 2024-02-10T08:26:21.544 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:21.544 INFO:teuthology.orchestra.run.smithi195.stdout:Total 566 kB/s | 131 kB 00:00 2024-02-10T08:26:21.544 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:21.549 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:21.549 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:21.602 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:21.605 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:21.714 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:21.910 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:21.955 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:21.956 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-02-10T08:26:21.956 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:21.972 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:22.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-02-10T08:26:22.269 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:22.666 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:22.666 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:22.666 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:22.667 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:22.667 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:22.667 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:22.812 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:43 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:22.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-02-10T08:26:23.622 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:28 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 88 k 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 240 k 2024-02-10T08:26:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:26:24.623 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 286 kB/s | 88 kB 00:00 2024-02-10T08:26:24.624 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:24.624 INFO:teuthology.orchestra.run.smithi107.stdout:Total 285 kB/s | 88 kB 00:00 2024-02-10T08:26:24.624 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:26:24.633 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:26:24.633 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:26:24.683 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:26:24.686 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:26:24.790 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:26:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:25.082 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:25.128 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:25.129 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:25.129 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:25.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-02-10T08:26:25.279 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:25.709 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:25.709 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:25.710 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:26:25.710 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:25.710 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:25.710 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:25.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-02-10T08:26:25.979 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:30 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:26.649 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:47 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:27.484 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:27.484 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:27.484 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-02-10T08:26:27.484 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:27.484 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 25 k 2024-02-10T08:26:27.485 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 107 k 2024-02-10T08:26:27.486 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:27.788 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 84 kB/s | 25 kB 00:00 2024-02-10T08:26:27.788 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:27.789 INFO:teuthology.orchestra.run.smithi195.stdout:Total 84 kB/s | 25 kB 00:00 2024-02-10T08:26:27.789 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:27.795 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:27.796 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:27.796 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:27.798 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:27.845 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:28.157 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:26:28.158 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:28.159 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.7 M 2024-02-10T08:26:28.159 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 15 M 2024-02-10T08:26:28.159 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:26:28.299 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:28.654 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:28.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-02-10T08:26:29.019 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 4.3 MB/s | 3.7 MB 00:00 2024-02-10T08:26:29.020 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:29.020 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.3 MB/s | 3.7 MB 00:00 2024-02-10T08:26:29.020 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:26:29.032 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:26:29.032 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:26:29.094 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:26:29.097 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:26:29.519 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:34 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:29.529 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:26:29.743 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:30.544 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:30.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:30.544 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:26:30.544 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:30.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:30.545 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:30.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-02-10T08:26:30.995 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:31.040 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:31.042 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:31.042 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:31.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-02-10T08:26:31.518 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:52 ago on Sat 10 Feb 2024 08:23:39 AM UTC. 2024-02-10T08:26:31.903 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:36 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-10T08:26:33.017 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 175 k 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 523 k 2024-02-10T08:26:33.018 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-02-10T08:26:33.332 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 559 kB/s | 175 kB 00:00 2024-02-10T08:26:33.333 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:33.333 INFO:teuthology.orchestra.run.smithi107.stdout:Total 556 kB/s | 175 kB 00:00 2024-02-10T08:26:33.333 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-02-10T08:26:33.344 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-02-10T08:26:33.344 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-02-10T08:26:33.373 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:33.393 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-02-10T08:26:33.396 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-02-10T08:26:33.419 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:33.420 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:33.420 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-02-10T08:26:33.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-02-10T08:26:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:34.037 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:34.283 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:39 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:26:34.457 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-02-10T08:26:34.654 DEBUG:teuthology.parallel:result is None 2024-02-10T08:26:35.739 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:35.784 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:35.786 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:35.786 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:35.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-02-10T08:26:36.640 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:41 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:38.101 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:38.146 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:38.147 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:38.147 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:38.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-02-10T08:26:39.015 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:44 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:40.486 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:40.531 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:40.533 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:40.533 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:40.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-02-10T08:26:41.397 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:46 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:42.854 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-10T08:26:42.900 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:42.901 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-10T08:26:42.901 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:43.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-02-10T08:26:43.770 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:48 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:45.285 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:45.286 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 88 k 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 240 k 2024-02-10T08:26:45.287 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:45.786 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 177 kB/s | 88 kB 00:00 2024-02-10T08:26:45.787 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:45.787 INFO:teuthology.orchestra.run.smithi195.stdout:Total 177 kB/s | 88 kB 00:00 2024-02-10T08:26:45.787 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:45.797 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:45.797 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:45.846 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:45.849 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:45.953 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:46.135 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:46.420 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:46.821 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:47.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-02-10T08:26:47.809 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:52 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-10T08:26:49.316 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.7 M 2024-02-10T08:26:49.317 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 15 M 2024-02-10T08:26:49.318 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:50.116 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 4.6 MB/s | 3.7 MB 00:00 2024-02-10T08:26:50.117 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:50.117 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.6 MB/s | 3.7 MB 00:00 2024-02-10T08:26:50.118 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:50.129 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:50.129 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:50.191 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:50.194 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:50.624 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:50.868 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:51.292 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:51.682 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:51.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-02-10T08:26:52.645 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:57 ago on Sat 10 Feb 2024 08:23:55 AM UTC. 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-10T08:26:54.160 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-10T08:26:54.161 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-10T08:26:54.161 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:54.161 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 175 k 2024-02-10T08:26:54.161 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 523 k 2024-02-10T08:26:54.161 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-10T08:26:54.670 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 344 kB/s | 175 kB 00:00 2024-02-10T08:26:54.670 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:26:54.671 INFO:teuthology.orchestra.run.smithi195.stdout:Total 344 kB/s | 175 kB 00:00 2024-02-10T08:26:54.671 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-10T08:26:54.681 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-10T08:26:54.681 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-10T08:26:54.732 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-10T08:26:54.735 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-10T08:26:54.845 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-10T08:26:55.117 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:55.412 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:26:55.799 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-10T08:26:56.043 DEBUG:teuthology.parallel:result is None 2024-02-10T08:26:56.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:26:56.213 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T08:26:56.240 INFO:teuthology.orchestra.run.smithi107.stdout:16.2.14-521.g0e714d9a.el8 2024-02-10T08:26:56.240 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-10T08:26:56.241 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-10T08:26:56.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:26:56.404 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T08:26:56.427 INFO:teuthology.orchestra.run.smithi195.stdout:16.2.14-521.g0e714d9a.el8 2024-02-10T08:26:56.427 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-10T08:26:56.427 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-10T08:26:56.430 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T08:26:56.430 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:26:56.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T08:26:56.464 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:26:56.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T08:26:56.494 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T08:26:56.494 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:26:56.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T08:26:56.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T08:26:56.605 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:26:56.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T08:26:56.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T08:26:56.698 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T08:26:56.699 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:26:56.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T08:26:56.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T08:26:56.798 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:26:56.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T08:26:56.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T08:26:56.891 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T08:26:56.891 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:26:56.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T08:26:56.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T08:26:56.988 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:26:56.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T08:26:57.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T08:26:57.081 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T08:26:57.155 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'cephadm_mode': 'root'} 2024-02-10T08:26:57.155 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:26:57.155 INFO:tasks.cephadm:Cluster fsid is 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:26:57.156 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T08:26:57.156 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.107:6789]', 'mon.c': '[v1:172.21.15.107:6790]', 'mon.b': '[v1:172.21.15.195:6789]'} 2024-02-10T08:26:57.156 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-02-10T08:26:57.156 INFO:tasks.cephadm:First mgr is y 2024-02-10T08:26:57.156 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T08:26:57.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-02-10T08:26:57.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-02-10T08:26:57.213 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 0e714d9a4bd2a821113e6318adb87bd06cf81ec1)... 2024-02-10T08:26:57.213 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.374 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 10 08:26 /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.375 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.506 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 10 08:26 /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.506 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.527 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T08:26:57.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-02-10T08:26:57.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-02-10T08:26:57.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-02-10T08:26:57.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-02-10T08:26:57.691 INFO:tasks.cephadm:Writing seed config... 2024-02-10T08:26:57.692 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T08:26:57.692 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-02-10T08:26:57.693 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T08:26:57.694 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-10T08:26:57.695 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:26:57.695 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T08:26:57.724 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 275c2754-c7ee-11ee-95b8-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-10T08:26:57.724 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T08:26:57.767 DEBUG:teuthology.orchestra.run.smithi107:mgr.y> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T08:26:57.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:26:57.796 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T08:26:57.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -v bootstrap --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.107:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:26:57.822 INFO:journalctl@ceph.mgr.y.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:26:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:-------------------------------------------------------------------------------- 2024-02-10T08:26:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1', '-v', 'bootstrap', '--fsid', '275c2754-c7ee-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.107:6789]', '--skip-admin-label'] 2024-02-10T08:26:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.8.3 2024-02-10T08:26:58.869 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying podman|docker is present... 2024-02-10T08:26:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.8.3 2024-02-10T08:26:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying lvm2 is present... 2024-02-10T08:26:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying time synchronization is in place... 2024-02-10T08:26:58.912 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T08:26:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:26:58.923 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: enabled 2024-02-10T08:26:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: active 2024-02-10T08:26:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:Unit chronyd.service is enabled and running 2024-02-10T08:26:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:Repeating the final host check... 2024-02-10T08:26:58.948 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.8.3 2024-02-10T08:26:58.958 INFO:teuthology.orchestra.run.smithi107.stderr:podman (/bin/podman) version 4.8.3 is present 2024-02-10T08:26:58.958 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl is present 2024-02-10T08:26:58.958 INFO:teuthology.orchestra.run.smithi107.stderr:lvcreate is present 2024-02-10T08:26:58.964 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T08:26:58.969 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:26:58.975 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: enabled 2024-02-10T08:26:58.980 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: active 2024-02-10T08:26:58.981 INFO:teuthology.orchestra.run.smithi107.stderr:Unit chronyd.service is enabled and running 2024-02-10T08:26:58.981 INFO:teuthology.orchestra.run.smithi107.stderr:Host looks OK 2024-02-10T08:26:58.981 INFO:teuthology.orchestra.run.smithi107.stderr:Cluster fsid: 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:26:58.981 INFO:teuthology.orchestra.run.smithi107.stderr:Acquiring lock 139948968872424 on /run/cephadm/275c2754-c7ee-11ee-95b8-87774f69a715.lock 2024-02-10T08:26:58.982 INFO:teuthology.orchestra.run.smithi107.stderr:Lock 139948968872424 acquired on /run/cephadm/275c2754-c7ee-11ee-95b8-87774f69a715.lock 2024-02-10T08:26:58.982 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying IP 172.21.15.107 port 6789 ... 2024-02-10T08:26:58.982 INFO:teuthology.orchestra.run.smithi107.stderr:Base mon IP(s) is [172.21.15.107:6789], mon addrv is [v1:172.21.15.107:6789] 2024-02-10T08:26:58.985 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-02-10T08:26:58.985 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-02-10T08:26:58.988 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T08:26:58.988 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T08:26:58.988 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-02-10T08:26:58.990 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T08:26:58.990 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T08:26:58.990 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T08:26:58.991 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T08:26:58.991 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-02-10T08:26:58.991 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T08:26:58.991 INFO:teuthology.orchestra.run.smithi107.stderr:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-02-10T08:26:58.991 INFO:teuthology.orchestra.run.smithi107.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-02-10T08:26:58.992 INFO:teuthology.orchestra.run.smithi107.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T08:26:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-10T08:26:59.096 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-10T08:26:59.250 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Getting image source signatures 2024-02-10T08:26:59.258 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying blob sha256:955bfa9dba4a4380f93be7209d227ce2b057a09dc4ade787f31ef12d8b977cea 2024-02-10T08:26:59.258 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T08:27:24.644 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying config sha256:1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-10T08:27:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T08:27:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-10T08:27:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:ceph: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-10T08:27:26.219 INFO:teuthology.orchestra.run.smithi107.stderr:Ceph version: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-10T08:27:26.219 INFO:teuthology.orchestra.run.smithi107.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T08:27:26.724 INFO:teuthology.orchestra.run.smithi107.stderr:stat: 167 167 2024-02-10T08:27:27.252 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial keys... 2024-02-10T08:27:27.839 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQBvM8dlOXrLLBAAvmUwrllQNXuGMwHyzCQB7A== 2024-02-10T08:27:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQBwM8dl9aQrMxAAZ8GggnennjTae8vybt0iVA== 2024-02-10T08:27:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQByM8dl2sHmDxAAWn6yFtBt6tY9/l1WGn19kQ== 2024-02-10T08:27:34.984 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial monmap... 2024-02-10T08:27:38.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T08:27:38.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:38.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T08:27:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:monmaptool for a [v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T08:27:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: set fsid to 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T08:27:42.069 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:27:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:Creating mon... 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.523+0000 7f3bd51dd880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.523+0000 7f3bd51dd880 1 imported monmap: 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: last_changed 2024-02-10T08:27:37.431644+0000 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: created 2024-02-10T08:27:37.431644+0000 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 0: v1:172.21.15.107:6789/0 mon.a 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.523+0000 7f3bd51dd880 0 /usr/bin/ceph-mon: set fsid to 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:42.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: DB SUMMARY 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.create_if_missing: 1 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.env: 0x561da334b080 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:27:42.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.info_log: 0x561da4517a80 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.db_log_dir: 2024-02-10T08:27:42.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.write_buffer_manager: 0x561da451d980 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:27:42.666 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.row_cache: None 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.wal_filter: None 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:27:42.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:27:42.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Compression algorithms supported: 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kZSTD supported: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:27:42.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.524+0000 7f3bd51dd880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.merge_operator: 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561da442dd30) 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:27:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache: 0x561da4462d10 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-10T08:27:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-10T08:27:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.num_levels: 7 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:27:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:27:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:27:42.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:27:42.676 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.525+0000 7f3bd51dd880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.526+0000 7f3bd51dd880 4 rocksdb: DB pointer 0x561da452b800 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.526+0000 7f3bbe04f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.526+0000 7f3bbe04f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:27:42.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:42.678 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:27:42.679 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-02-10T08:27:42.680 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.531+0000 7f3bd51dd880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-10T08:27:42.681 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.531+0000 7f3bd51dd880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-10T08:27:42.681 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:27:42.531+0000 7f3bd51dd880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-10T08:27:43.253 INFO:teuthology.orchestra.run.smithi107.stderr:create mon.a on 2024-02-10T08:27:43.267 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-10T08:27:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T08:27:43.452 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target. 2024-02-10T08:27:43.452 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target. 2024-02-10T08:27:43.648 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to reset failed state of unit ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T08:27:43.654 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target.wants/ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715@.service. 2024-02-10T08:27:43.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:43 smithi107 systemd[1]: Starting Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:27:44.356 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-02-10T08:27:44.363 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:27:44.363 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-02-10T08:27:44.363 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T08:27:44.364 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mon to start... 2024-02-10T08:27:44.364 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mon... 2024-02-10T08:27:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.316270+0000 mon.a (mon.0) 0 : [INF] mkfs 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.294211+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T08:27:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:44 smithi107 bash[67709]: 8f9ae8d11780c17cbaf5e153b9330c937d9949af7afd899f27212b0e2c6463d2 2024-02-10T08:27:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:44 smithi107 systemd[1]: Started Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.884+0000 7fe4e6eee700 1 Processor -- start 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.884+0000 7fe4e6eee700 1 -- start start 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.884+0000 7fe4e6eee700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4e01002e0 con 0x7fe4e0070c40 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.884+0000 7fe4e4c8a700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe4e0070c40 0x7fe4e00febc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34800/0 (socket says 172.21.15.107:34800) 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.884+0000 7fe4e4c8a700 1 -- 172.21.15.107:0/201226137 learned_addr learned my addr 172.21.15.107:0/201226137 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515690273 0 0) 0x7fe4e01002e0 con 0x7fe4e0070c40 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d00033f0 con 0x7fe4e0070c40 2024-02-10T08:27:44.887 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142110103 0 0) 0x7fe4d00033f0 con 0x7fe4e0070c40 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0101480 con 0x7fe4e0070c40 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe4d4001590 con 0x7fe4e0070c40 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4d4001880 con 0x7fe4e0070c40 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4df7fe700 1 -- 172.21.15.107:0/201226137 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe4d4001bd0 con 0x7fe4e0070c40 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4e6eee700 1 -- 172.21.15.107:0/201226137 >> v1:172.21.15.107:6789/0 conn(0x7fe4e0070c40 legacy=0x7fe4e00febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4e6eee700 1 -- 172.21.15.107:0/201226137 shutdown_connections 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.885+0000 7fe4e6eee700 1 -- 172.21.15.107:0/201226137 >> 172.21.15.107:0/201226137 conn(0x7fe4e00f6520 msgr2=0x7fe4e00f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 -- 172.21.15.107:0/201226137 shutdown_connections 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 -- 172.21.15.107:0/201226137 wait complete. 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 librados: starting msgr at 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 librados: starting objecter 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 Processor -- start 2024-02-10T08:27:44.888 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 -- start start 2024-02-10T08:27:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 librados: setting wanted keys 2024-02-10T08:27:44.889 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 librados: calling monclient init 2024-02-10T08:27:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e6eee700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4e00fc5d0 con 0x7fe4e0070c40 2024-02-10T08:27:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e4c8a700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe4e0070c40 0x7fe4e00faac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34806/0 (socket says 172.21.15.107:34806) 2024-02-10T08:27:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4e4c8a700 1 -- 172.21.15.107:0/556365156 learned_addr learned my addr 172.21.15.107:0/556365156 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4ddffb700 10 client.?.objecter ms_handle_connect 0x7fe4e0070c40 2024-02-10T08:27:44.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.886+0000 7fe4ddffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298113243 0 0) 0x7fe4e00fc5d0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0004400 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115817838 0 0) 0x7fe4d0004400 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4e00fc5d0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe4d4001610 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1681173318 0 0) 0x7fe4e00fc5d0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00fc760 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00fca10 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4d4004a90 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe4d4004da0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fe4d40051e0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4e6eee700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e00fb4d0 con 0x7fe4e0070c40 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.887+0000 7fe4e6eee700 1 librados: init done 2024-02-10T08:27:44.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7fe4d4005ec0 con 0x7fe4e0070c40 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4ddffb700 10 client.4098.objecter ms_dispatch 0x7fe4e0001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4ddffb700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4ddffb700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4ddffb700 20 client.4098.objecter dump_active .. 0 homeless 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.888+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0047bf0 con 0x7fe4e0070c40 2024-02-10T08:27:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.890+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fe4e00fca10 con 0x7fe4e0070c40 2024-02-10T08:27:44.959 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.957+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe4e005a3b0 con 0x7fe4e0070c40 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.957+0000 7fe4ddffb700 1 -- 172.21.15.107:0/556365156 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 2540902129) 0x7fe4d4005a20 con 0x7fe4e0070c40 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: cluster: 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: id: 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: services: 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.643049s) 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: data: 2024-02-10T08:27:44.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: pgs: 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.957+0000 7fe4e6eee700 10 librados: watch_flush enter 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.957+0000 7fe4e6eee700 10 librados: watch_flush exit 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 20 client.4098.objecter shutdown clearing up homeless session... 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 10 client.4098.objecter successfully canceled tick 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 >> v1:172.21.15.107:6789/0 conn(0x7fe4e0070c40 legacy=0x7fe4e00faac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 shutdown_connections 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 >> 172.21.15.107:0/556365156 conn(0x7fe4e00f6520 msgr2=0x7fe4e00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 shutdown_connections 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 -- 172.21.15.107:0/556365156 wait complete. 2024-02-10T08:27:44.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:44.958+0000 7fe4e6eee700 1 librados: shutdown 2024-02-10T08:27:45.552 INFO:teuthology.orchestra.run.smithi107.stderr:mon is available 2024-02-10T08:27:45.553 INFO:teuthology.orchestra.run.smithi107.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T08:27:45.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.315012+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T08:27:45.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.315189+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.107:6789/0} removed_ranks: {} 2024-02-10T08:27:45.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.318145+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T08:27:45.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.320850+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T08:27:45.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: cluster 2024-02-10T08:27:44.321219+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T08:27:45.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: audit 2024-02-10T08:27:44.958028+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.107:0/556365156' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.075+0000 7fb63b415700 1 Processor -- start 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.075+0000 7fb63b415700 1 -- start start 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb63b415700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb634100400 con 0x7fb6340fd540 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb6391b1700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6340fd540 0x7fb6340fd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34812/0 (socket says 172.21.15.107:34812) 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb6391b1700 1 -- 172.21.15.107:0/1492076875 learned_addr learned my addr 172.21.15.107:0/1492076875 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748974071 0 0) 0x7fb634100400 con 0x7fb6340fd540 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6200033f0 con 0x7fb6340fd540 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160212578 0 0) 0x7fb6200033f0 con 0x7fb6340fd540 2024-02-10T08:27:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.076+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6341015a0 con 0x7fb6340fd540 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb628001590 con 0x7fb6340fd540 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6280018f0 con 0x7fb6340fd540 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb627fff700 1 -- 172.21.15.107:0/1492076875 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb628001c40 con 0x7fb6340fd540 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb63b415700 1 -- 172.21.15.107:0/1492076875 >> v1:172.21.15.107:6789/0 conn(0x7fb6340fd540 legacy=0x7fb6340fd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb63b415700 1 -- 172.21.15.107:0/1492076875 shutdown_connections 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb63b415700 1 -- 172.21.15.107:0/1492076875 >> 172.21.15.107:0/1492076875 conn(0x7fb6340f7690 msgr2=0x7fb6340f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb63b415700 1 -- 172.21.15.107:0/1492076875 shutdown_connections 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.077+0000 7fb63b415700 1 -- 172.21.15.107:0/1492076875 wait complete. 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 librados: starting msgr at 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 librados: starting objecter 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 Processor -- start 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 -- start start 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 librados: setting wanted keys 2024-02-10T08:27:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 librados: calling monclient init 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb63b415700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb63410a880 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb6391b1700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6340fd540 0x7fb63410a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34826/0 (socket says 172.21.15.107:34826) 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb6391b1700 1 -- 172.21.15.107:0/1159756041 learned_addr learned my addr 172.21.15.107:0/1159756041 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb6267fc700 10 client.?.objecter ms_handle_connect 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.078+0000 7fb6267fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070151243 0 0) 0x7fb63410a880 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb620004400 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2458006814 0 0) 0x7fb620004400 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb63410a880 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb628001550 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109900484 0 0) 0x7fb63410a880 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63410aa10 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63410acc0 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb63b415700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63410e9b0 con 0x7fb6340fd540 2024-02-10T08:27:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.079+0000 7fb63b415700 1 librados: init done 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb628004b80 con 0x7fb6340fd540 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb628004e90 con 0x7fb6340fd540 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fb6280052d0 con 0x7fb6340fd540 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7fb6280062c0 con 0x7fb6340fd540 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 10 client.4100.objecter ms_dispatch 0x7fb634001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb6267fc700 20 client.4100.objecter dump_active .. 0 homeless 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.080+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb618004f50 con 0x7fb6340fd540 2024-02-10T08:27:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.083+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fb628006480 con 0x7fb6340fd540 2024-02-10T08:27:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.147+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb618005780 con 0x7fb6340fd540 2024-02-10T08:27:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 10 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fb628005a10 con 0x7fb6340fd540 2024-02-10T08:27:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb6267fc700 1 -- 172.21.15.107:0/1159756041 <== mon.0 v1:172.21.15.107:6789/0 11 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+343 (unknown 1213389831 0 2153410792) 0x7fb628007370 con 0x7fb6340fd540 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [global] 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: fsid = 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.107:6789] 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [mgr] 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [osd] 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T08:27:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 10 librados: watch_flush enter 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 10 librados: watch_flush exit 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 20 client.4100.objecter shutdown clearing up homeless session... 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 10 client.4100.objecter successfully canceled tick 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 >> v1:172.21.15.107:6789/0 conn(0x7fb6340fd540 legacy=0x7fb63410a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 shutdown_connections 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.150+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 >> 172.21.15.107:0/1159756041 conn(0x7fb6340f7690 msgr2=0x7fb6340f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.151+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 shutdown_connections 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.151+0000 7fb63b415700 1 -- 172.21.15.107:0/1159756041 wait complete. 2024-02-10T08:27:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:46.151+0000 7fb63b415700 1 librados: shutdown 2024-02-10T08:27:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: audit 2024-02-10T08:27:46.147954+0000 mon.a (mon.0) 8 : audit [INF] from='client.? v1:172.21.15.107:0/1159756041' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T08:27:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: audit 2024-02-10T08:27:46.150693+0000 mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.107:0/1159756041' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T08:27:46.693 INFO:teuthology.orchestra.run.smithi107.stderr:Generating new minimal ceph.conf... 2024-02-10T08:27:47.219 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.217+0000 7ff8049e9700 1 Processor -- start 2024-02-10T08:27:47.219 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.217+0000 7ff8049e9700 1 -- start start 2024-02-10T08:27:47.219 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.217+0000 7ff8049e9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff800100240 con 0x7ff800070ce0 2024-02-10T08:27:47.219 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fe59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff800070ce0 0x7ff8000feb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34828/0 (socket says 172.21.15.107:34828) 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fe59c700 1 -- 172.21.15.107:0/2379251283 learned_addr learned my addr 172.21.15.107:0/2379251283 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881524679 0 0) 0x7ff800100240 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e80033f0 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471882770 0 0) 0x7ff7e80033f0 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8001013e0 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff7f0001590 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7ff7f0001d10 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.218+0000 7ff7fd59a700 1 -- 172.21.15.107:0/2379251283 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff7f0005750 con 0x7ff800070ce0 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 -- 172.21.15.107:0/2379251283 >> v1:172.21.15.107:6789/0 conn(0x7ff800070ce0 legacy=0x7ff8000feb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:47.220 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 -- 172.21.15.107:0/2379251283 shutdown_connections 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 -- 172.21.15.107:0/2379251283 >> 172.21.15.107:0/2379251283 conn(0x7ff8000f6520 msgr2=0x7ff8000f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 -- 172.21.15.107:0/2379251283 shutdown_connections 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 -- 172.21.15.107:0/2379251283 wait complete. 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 librados: starting msgr at 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.219+0000 7ff8049e9700 1 librados: starting objecter 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff8049e9700 1 Processor -- start 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff8049e9700 1 -- start start 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff8049e9700 1 librados: setting wanted keys 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff8049e9700 1 librados: calling monclient init 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff8049e9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff80010d210 con 0x7ff800070ce0 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7fe59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff800070ce0 0x7ff80010cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34844/0 (socket says 172.21.15.107:34844) 2024-02-10T08:27:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7fe59c700 1 -- 172.21.15.107:0/3641459405 learned_addr learned my addr 172.21.15.107:0/3641459405 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7ef7fe700 10 client.?.objecter ms_handle_connect 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7ef7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843804548 0 0) 0x7ff80010d210 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.220+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e8005580 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2100571353 0 0) 0x7ff7e8005580 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff80010d210 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff7f0000fe0 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 848821717 0 0) 0x7ff80010d210 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80010d3a0 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80010d650 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff8049e9700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff800111340 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff8049e9700 1 librados: init done 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7ff7f00046b0 con 0x7ff800070ce0 2024-02-10T08:27:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff7f0006110 con 0x7ff800070ce0 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7ff7f0006550 con 0x7ff800070ce0 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7ff7f0007470 con 0x7ff800070ce0 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 10 client.4102.objecter ms_dispatch 0x7ff800001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.221+0000 7ff7ef7fe700 20 client.4102.objecter dump_active .. 0 homeless 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.222+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e0004f50 con 0x7ff800070ce0 2024-02-10T08:27:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.224+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7ff7f0005a30 con 0x7ff800070ce0 2024-02-10T08:27:47.289 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.289+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff7e0005200 con 0x7ff800070ce0 2024-02-10T08:27:47.289 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.289+0000 7ff7ef7fe700 1 -- 172.21.15.107:0/3641459405 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+148 (unknown 1452402520 0 3229897886) 0x7ff7f00069e0 con 0x7ff800070ce0 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.289+0000 7ff8049e9700 10 librados: watch_flush enter 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 10 librados: watch_flush exit 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 20 client.4102.objecter shutdown clearing up homeless session... 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 10 client.4102.objecter successfully canceled tick 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 >> v1:172.21.15.107:6789/0 conn(0x7ff800070ce0 legacy=0x7ff80010cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:47.291 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 shutdown_connections 2024-02-10T08:27:47.292 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 >> 172.21.15.107:0/3641459405 conn(0x7ff8000f6520 msgr2=0x7ff8000fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:47.292 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 shutdown_connections 2024-02-10T08:27:47.292 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 -- 172.21.15.107:0/3641459405 wait complete. 2024-02-10T08:27:47.292 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:47.290+0000 7ff8049e9700 1 librados: shutdown 2024-02-10T08:27:47.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: audit 2024-02-10T08:27:47.289915+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.107:0/3641459405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:27:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:Restarting the monitor... 2024-02-10T08:27:48.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:47 smithi107 systemd[1]: Stopping Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:27:48.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:47 smithi107 bash[68051]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T08:27:48.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: debug 2024-02-10T08:27:48.091+0000 7fb669d35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T08:27:48.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[67724]: debug 2024-02-10T08:27:48.091+0000 7fb669d35700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T08:27:48.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 bash[68091]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a 2024-02-10T08:27:49.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service: Succeeded. 2024-02-10T08:27:49.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 systemd[1]: Stopped Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:27:49.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:48 smithi107 systemd[1]: Starting Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:27:49.502 INFO:teuthology.orchestra.run.smithi107.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.448+0000 7fe8bcbb4880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.448+0000 7fe8bcbb4880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.448+0000 7fe8bcbb4880 0 pidfile_write: ignore empty --pid-file 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.453+0000 7fe8bcbb4880 0 load: jerasure load: lrc load: isa 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.453+0000 7fe8bcbb4880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.453+0000 7fe8bcbb4880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.453+0000 7fe8bcbb4880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.453+0000 7fe8bcbb4880 4 rocksdb: DB SUMMARY 2024-02-10T08:27:49.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 91707 ; 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:27:49.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.env: 0x55f499b08080 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.info_log: 0x55f49ba10200 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:27:49.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.db_log_dir: 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:27:49.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.write_buffer_manager: 0x55f49c702390 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:27:49.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.row_cache: None 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.wal_filter: None 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:27:49.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:27:49.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Compression algorithms supported: 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kZSTD supported: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.merge_operator: 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f49b9b5d28) 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cache_index_and_filter_blocks: 1 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: pin_top_level_index_and_filter: 1 2024-02-10T08:27:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: index_type: 0 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: data_block_index_type: 0 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: index_shortening: 1 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: hash_index_allow_collision: 1 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: checksum: 1 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: no_block_cache: 0 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_cache: 0x55f49b9eaf10 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_cache_name: BinnedLRUCache 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_cache_options: 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: capacity : 536870912 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: num_shard_bits : 4 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: strict_capacity_limit : 0 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: high_pri_pool_ratio: 0.000 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_cache_compressed: (nil) 2024-02-10T08:27:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: persistent_cache: (nil) 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_size: 4096 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_size_deviation: 10 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_restart_interval: 16 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: index_block_restart_interval: 1 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: metadata_block_size: 4096 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: partition_filters: 0 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: use_delta_encoding: 1 2024-02-10T08:27:49.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:27:49.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: whole_key_filtering: 1 2024-02-10T08:27:49.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: verify_compression: 0 2024-02-10T08:27:49.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: read_amp_bytes_per_bit: 0 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: format_version: 2 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: enable_index_compression: 1 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: block_align: 0 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.num_levels: 7 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:27:49.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:27:49.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:27:49.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:27:49.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:27:49.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.454+0000 7fe8bcbb4880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.455+0000 7fe8bcbb4880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553669455668, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.455+0000 7fe8bcbb4880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.455+0000 7fe8bcbb4880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.455+0000 7fe8bcbb4880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553669456525, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 87642, "table_properties": {"data_size": 85745, "index_size": 374, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 11481, "raw_average_key_size": 49, "raw_value_size": 79389, "raw_average_value_size": 339, "num_data_blocks": 12, "num_entries": 234, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707553669, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.455+0000 7fe8bcbb4880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.456+0000 7fe8bcbb4880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553669456899, "job": 1, "event": "recovery_finished"} 2024-02-10T08:27:49.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.456+0000 7fe8bcbb4880 4 rocksdb: DB pointer 0x55f49bab3800 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.457+0000 7fe8a39a5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.457+0000 7fe8a39a5700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** DB Stats ** 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** Compaction Stats [default] ** 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:49.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: L0 2/0 87.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Sum 2/0 87.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** Compaction Stats [default] ** 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:27:49.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Cumulative compaction: 0.00 GB write, 31.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Interval compaction: 0.00 GB write, 31.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** Compaction Stats [default] ** 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: L0 2/0 87.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Sum 2/0 87.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** Compaction Stats [default] ** 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Cumulative compaction: 0.00 GB write, 30.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.457+0000 7fe8bcbb4880 0 starting mon.a rank 0 at public addrs v1:172.21.15.107:6789/0 at bind addrs v1:172.21.15.107:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.458+0000 7fe8bcbb4880 1 mon.a@-1(???) e1 preinit fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.458+0000 7fe8bcbb4880 0 mon.a@-1(???).mds e1 new map 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.458+0000 7fe8bcbb4880 0 mon.a@-1(???).mds e1 print_map 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: e1 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: legacy client fscid: -1 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: No filesystems configured 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.459+0000 7fe8bcbb4880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.459+0000 7fe8bcbb4880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:27:49.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.459+0000 7fe8bcbb4880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.459+0000 7fe8bcbb4880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:47.289915+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.107:0/3641459405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:27:49.459+0000 7fe8bcbb4880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 bash[68208]: 220395f13b1c4367cf14527f900f7e1d15c8293247143586aec89dc973b8cfb2 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 systemd[1]: Started Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:49.475652+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:49.475718+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.107:6789/0} removed_ranks: {} 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:49.476067+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:49.476105+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T08:27:49.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:49.476472+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T08:27:50.055 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.049+0000 7f76d1a4d700 1 Processor -- start 2024-02-10T08:27:50.055 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.049+0000 7f76d1a4d700 1 -- start start 2024-02-10T08:27:50.055 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.050+0000 7f76d1a4d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76cc0fc960 con 0x7f76cc070ca0 2024-02-10T08:27:50.055 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.050+0000 7f76caffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76cc070ca0 0x7f76cc100f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48762/0 (socket says 172.21.15.107:48762) 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.050+0000 7f76caffd700 1 -- 172.21.15.107:0/4176677046 learned_addr learned my addr 172.21.15.107:0/4176677046 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629605749 0 0) 0x7f76cc0fc960 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40033f0 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2430058665 0 0) 0x7f76b40033f0 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc1036a0 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f76c0001590 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f76c0001c40 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76c9ffb700 1 -- 172.21.15.107:0/4176677046 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f76c0005610 con 0x7f76cc070ca0 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.051+0000 7f76d1a4d700 1 -- 172.21.15.107:0/4176677046 >> v1:172.21.15.107:6789/0 conn(0x7f76cc070ca0 legacy=0x7f76cc100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- 172.21.15.107:0/4176677046 shutdown_connections 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- 172.21.15.107:0/4176677046 >> 172.21.15.107:0/4176677046 conn(0x7f76cc0f6530 msgr2=0x7f76cc0f89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- 172.21.15.107:0/4176677046 shutdown_connections 2024-02-10T08:27:50.056 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- 172.21.15.107:0/4176677046 wait complete. 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 librados: starting msgr at 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 librados: starting objecter 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 Processor -- start 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- start start 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 librados: setting wanted keys 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 librados: calling monclient init 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.052+0000 7f76d1a4d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76cc10d200 con 0x7f76cc070ca0 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76caffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76cc070ca0 0x7f76cc10caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48776/0 (socket says 172.21.15.107:48776) 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76caffd700 1 -- 172.21.15.107:0/3679390645 learned_addr learned my addr 172.21.15.107:0/3679390645 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 10 client.?.objecter ms_handle_connect 0x7f76cc070ca0 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694573733 0 0) 0x7f76cc10d200 con 0x7f76cc070ca0 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40055d0 con 0x7f76cc070ca0 2024-02-10T08:27:50.057 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531664828 0 0) 0x7f76b40055d0 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc10d200 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f76c0000ff0 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 611050447 0 0) 0x7f76cc10d200 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc10d390 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc10d640 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76d1a4d700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc111330 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76d1a4d700 1 librados: init done 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f76c0004730 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f76c0006160 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7f76c0006600 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f76c0007560 con 0x7f76cc070ca0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 10 client.14098.objecter ms_dispatch 0x7f76cc001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.053+0000 7f76b3fff700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:50.059 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.054+0000 7f76b3fff700 20 client.14098.objecter dump_active .. 0 homeless 2024-02-10T08:27:50.059 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.054+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc05a3b0 con 0x7f76cc070ca0 2024-02-10T08:27:50.059 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.056+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7f76c0005900 con 0x7f76cc070ca0 2024-02-10T08:27:50.120 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.120+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f76cc046d80 con 0x7f76cc070ca0 2024-02-10T08:27:50.123 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.122+0000 7f76b3fff700 1 -- 172.21.15.107:0/3679390645 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (unknown 534127980 0 0) 0x7f76cc046d80 con 0x7f76cc070ca0 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 10 librados: watch_flush enter 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 10 librados: watch_flush exit 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 20 client.14098.objecter shutdown clearing up homeless session... 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 10 client.14098.objecter successfully canceled tick 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 >> v1:172.21.15.107:6789/0 conn(0x7f76cc070ca0 legacy=0x7f76cc10caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 shutdown_connections 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 >> 172.21.15.107:0/3679390645 conn(0x7f76cc0f6530 msgr2=0x7f76cc0f71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:50.124 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 shutdown_connections 2024-02-10T08:27:50.125 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 -- 172.21.15.107:0/3679390645 wait complete. 2024-02-10T08:27:50.125 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:50.123+0000 7f76d1a4d700 1 librados: shutdown 2024-02-10T08:27:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T08:27:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:27:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:Creating mgr... 2024-02-10T08:27:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying port 9283 ... 2024-02-10T08:27:50.828 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to reset failed state of unit ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T08:27:50.834 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715.target.wants/ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-275c2754-c7ee-11ee-95b8-87774f69a715@.service. 2024-02-10T08:27:51.123 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:50 smithi107 systemd[1]: Starting Ceph mgr.y for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:27:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:50.123041+0000 mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.107:0/3679390645' entity='client.admin' 2024-02-10T08:27:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-02-10T08:27:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:27:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-02-10T08:27:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T08:27:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-02-10T08:27:51.518 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:27:51.518 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-02-10T08:27:51.519 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T08:27:51.519 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr to start... 2024-02-10T08:27:51.519 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr... 2024-02-10T08:27:51.751 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:51 smithi107 bash[68488]: 15150c0bc9979abc31f3ea07ca0f55a35b121799bb621ea966a88510bebe5627 2024-02-10T08:27:51.751 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:51 smithi107 systemd[1]: Started Ceph mgr.y for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:27:51.751 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:51.535+0000 7f186cd9c680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:27:51.751 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:51.620+0000 7f186cd9c680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:27:52.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.071+0000 7f3f29387700 1 Processor -- start 2024-02-10T08:27:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.071+0000 7f3f29387700 1 -- start start 2024-02-10T08:27:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.071+0000 7f3f29387700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f24100280 con 0x7f3f24070ca0 2024-02-10T08:27:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.071+0000 7f3f22ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3f24070ca0 0x7f3f240feb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48796/0 (socket says 172.21.15.107:48796) 2024-02-10T08:27:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.071+0000 7f3f22ffd700 1 -- 172.21.15.107:0/1908693552 learned_addr learned my addr 172.21.15.107:0/1908693552 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:52.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121300263 0 0) 0x7f3f24100280 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f0c0033f0 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260885082 0 0) 0x7f3f0c0033f0 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24101420 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f3f14001590 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f3f14001d10 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f21ffb700 1 -- 172.21.15.107:0/1908693552 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f3f14005750 con 0x7f3f24070ca0 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f29387700 1 -- 172.21.15.107:0/1908693552 >> v1:172.21.15.107:6789/0 conn(0x7f3f24070ca0 legacy=0x7f3f240feb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f29387700 1 -- 172.21.15.107:0/1908693552 shutdown_connections 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f29387700 1 -- 172.21.15.107:0/1908693552 >> 172.21.15.107:0/1908693552 conn(0x7f3f240f6520 msgr2=0x7f3f240f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f29387700 1 -- 172.21.15.107:0/1908693552 shutdown_connections 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.072+0000 7f3f29387700 1 -- 172.21.15.107:0/1908693552 wait complete. 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 librados: starting msgr at 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 librados: starting objecter 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 Processor -- start 2024-02-10T08:27:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 -- start start 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 librados: setting wanted keys 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 librados: calling monclient init 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f29387700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f2410b010 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f22ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3f24070ca0 0x7f3f2410a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48812/0 (socket says 172.21.15.107:48812) 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.073+0000 7f3f22ffd700 1 -- 172.21.15.107:0/1711352214 learned_addr learned my addr 172.21.15.107:0/1711352214 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 10 client.?.objecter ms_handle_connect 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963846471 0 0) 0x7f3f2410b010 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f0c005580 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270862155 0 0) 0x7f3f0c005580 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f2410b010 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f3f14000fe0 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3837487439 0 0) 0x7f3f2410b010 con 0x7f3f24070ca0 2024-02-10T08:27:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2410b1a0 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2410b450 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f29387700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2410f140 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f29387700 1 librados: init done 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f3f140046b0 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f3f14006120 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7f3f14006580 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.074+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f3f140074e0 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.075+0000 7f3f0bfff700 10 client.14102.objecter ms_dispatch 0x7f3f24001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.075+0000 7f3f0bfff700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.075+0000 7f3f0bfff700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.075+0000 7f3f0bfff700 20 client.14102.objecter dump_active .. 0 homeless 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.075+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f10004f50 con 0x7f3f24070ca0 2024-02-10T08:27:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.077+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7f3f14005a40 con 0x7f3f24070ca0 2024-02-10T08:27:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.147+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3f10005200 con 0x7f3f24070ca0 2024-02-10T08:27:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.147+0000 7f3f0bfff700 1 -- 172.21.15.107:0/1711352214 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (unknown 4201413639 0 4226792694) 0x7f3f14005be0 con 0x7f3f24070ca0 2024-02-10T08:27:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "a" 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T08:27:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:27:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:27:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": false, 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "nfs", 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:27:44.316616+0000", 2024-02-10T08:27:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 10 librados: watch_flush enter 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 10 librados: watch_flush exit 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 20 client.14102.objecter shutdown clearing up homeless session... 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 10 client.14102.objecter successfully canceled tick 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 >> v1:172.21.15.107:6789/0 conn(0x7f3f24070ca0 legacy=0x7f3f2410a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 shutdown_connections 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.148+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 >> 172.21.15.107:0/1711352214 conn(0x7f3f240f6520 msgr2=0x7f3f240fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.149+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 shutdown_connections 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.149+0000 7f3f29387700 1 -- 172.21.15.107:0/1711352214 wait complete. 2024-02-10T08:27:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:52.149+0000 7f3f29387700 1 librados: shutdown 2024-02-10T08:27:52.172 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:51.996+0000 7f186cd9c680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:27:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:52.148271+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.107:0/1711352214' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:27:52.743 INFO:teuthology.orchestra.run.smithi107.stderr:mgr not available, waiting (1/15)... 2024-02-10T08:27:52.840 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:52.755+0000 7f186cd9c680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:27:53.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:52.839+0000 7f186cd9c680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:27:53.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:53.047+0000 7f186cd9c680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:27:54.266 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:53.974+0000 7f186cd9c680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:27:54.266 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.100+0000 7f186cd9c680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:27:54.266 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.177+0000 7f186cd9c680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:27:54.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.266+0000 7f186cd9c680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:27:54.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.457+0000 7f186cd9c680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:27:55.002 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.731+0000 7f186cd9c680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:27:55.002 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:54.834+0000 7f186cd9c680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.257+0000 7fe272cf9700 1 Processor -- start 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.257+0000 7fe272cf9700 1 -- start start 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.257+0000 7fe272cf9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe26c0fcf70 con 0x7fe26c0fc440 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.257+0000 7fe270a95700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe26c0fc440 0x7fe26c0fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48826/0 (socket says 172.21.15.107:48826) 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.257+0000 7fe270a95700 1 -- 172.21.15.107:0/906949545 learned_addr learned my addr 172.21.15.107:0/906949545 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:55.259 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614210541 0 0) 0x7fe26c0fcf70 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe25c0033f0 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113138861 0 0) 0x7fe25c0033f0 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c1003b0 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe260001590 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fe260001d10 con 0x7fe26c0fc440 2024-02-10T08:27:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.258+0000 7fe26b7fe700 1 -- 172.21.15.107:0/906949545 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe260005750 con 0x7fe26c0fc440 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 -- 172.21.15.107:0/906949545 >> v1:172.21.15.107:6789/0 conn(0x7fe26c0fc440 legacy=0x7fe26c0fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 -- 172.21.15.107:0/906949545 shutdown_connections 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 -- 172.21.15.107:0/906949545 >> 172.21.15.107:0/906949545 conn(0x7fe26c0f5c90 msgr2=0x7fe26c0f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 -- 172.21.15.107:0/906949545 shutdown_connections 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 -- 172.21.15.107:0/906949545 wait complete. 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 librados: starting msgr at 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.259+0000 7fe272cf9700 1 librados: starting objecter 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe272cf9700 1 Processor -- start 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe272cf9700 1 -- start start 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe272cf9700 1 librados: setting wanted keys 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe272cf9700 1 librados: calling monclient init 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe272cf9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe26c0fac90 con 0x7fe26c0fc440 2024-02-10T08:27:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe270a95700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe26c0fc440 0x7fe26c10d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48830/0 (socket says 172.21.15.107:48830) 2024-02-10T08:27:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe270a95700 1 -- 172.21.15.107:0/1402635702 learned_addr learned my addr 172.21.15.107:0/1402635702 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 10 client.?.objecter ms_handle_connect 0x7fe26c0fc440 2024-02-10T08:27:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571890492 0 0) 0x7fe26c0fac90 con 0x7fe26c0fc440 2024-02-10T08:27:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe25c005580 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927335094 0 0) 0x7fe25c005580 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe26c0fac90 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.260+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe260005510 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2756687145 0 0) 0x7fe26c0fac90 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0fae20 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c10ece0 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fe260005730 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe2600062a0 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fe2600067a0 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe272cf9700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c10f110 con 0x7fe26c0fc440 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.261+0000 7fe272cf9700 1 librados: init done 2024-02-10T08:27:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7fe260006c80 con 0x7fe26c0fc440 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe269ffb700 10 client.14104.objecter ms_dispatch 0x7fe26c001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe269ffb700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe269ffb700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe269ffb700 20 client.14104.objecter dump_active .. 0 homeless 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.262+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c05a3b0 con 0x7fe26c0fc440 2024-02-10T08:27:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.264+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fe260006ec0 con 0x7fe26c0fc440 2024-02-10T08:27:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.333+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe26c046d80 con 0x7fe26c0fc440 2024-02-10T08:27:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.333+0000 7fe269ffb700 1 -- 172.21.15.107:0/1402635702 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (unknown 4201413639 0 483829626) 0x7fe26c046d80 con 0x7fe26c0fc440 2024-02-10T08:27:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:27:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "a" 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-10T08:27:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:27:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": false, 2024-02-10T08:27:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "nfs", 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:27:44.316616+0000", 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:27:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 10 librados: watch_flush enter 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 10 librados: watch_flush exit 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 20 client.14104.objecter shutdown clearing up homeless session... 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 10 client.14104.objecter successfully canceled tick 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 >> v1:172.21.15.107:6789/0 conn(0x7fe26c0fc440 legacy=0x7fe26c10d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 shutdown_connections 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 >> 172.21.15.107:0/1402635702 conn(0x7fe26c0f5c90 msgr2=0x7fe26c0f6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 shutdown_connections 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 -- 172.21.15.107:0/1402635702 wait complete. 2024-02-10T08:27:55.345 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:55.334+0000 7fe272cf9700 1 librados: shutdown 2024-02-10T08:27:55.664 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:55.334032+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.107:0/1402635702' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:27:55.664 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:55.580+0000 7f186cd9c680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:27:55.927 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:55.663+0000 7f186cd9c680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:27:55.927 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:55.755+0000 7f186cd9c680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:27:55.955 INFO:teuthology.orchestra.run.smithi107.stderr:mgr not available, waiting (2/15)... 2024-02-10T08:27:56.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:55.926+0000 7f186cd9c680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:27:56.440 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:56.180+0000 7f186cd9c680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:27:56.440 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:56.308+0000 7f186cd9c680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:27:56.695 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:56.439+0000 7f186cd9c680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:27:56.695 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:56.610+0000 7f186cd9c680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:27:57.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:27:56.694+0000 7f186cd9c680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:27:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:56.708479+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-10T08:27:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:56.711180+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00277036s) 2024-02-10T08:27:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.712227+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:27:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.712420+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.712587+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.712747+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.712914+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.713074+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.713244+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.713417+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:56.726165+0000 mon.a (mon.0) 19 : cluster [INF] Manager daemon y is now available 2024-02-10T08:27:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.735265+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' 2024-02-10T08:27:57.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.739834+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:27:57.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.742235+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:27:57.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.745011+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' 2024-02-10T08:27:57.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:56.747192+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 v1:172.21.15.107:0/1305020114' entity='mgr.y' 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.456+0000 7f6ed867f700 1 Processor -- start 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.456+0000 7f6ed867f700 1 -- start start 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.456+0000 7f6ed867f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ed0100400 con 0x7f6ed00fd540 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.456+0000 7f6ed641b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6ed00fd540 0x7f6ed00fd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48902/0 (socket says 172.21.15.107:48902) 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed641b700 1 -- 172.21.15.107:0/1420959041 learned_addr learned my addr 172.21.15.107:0/1420959041 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234628222 0 0) 0x7f6ed0100400 con 0x7f6ed00fd540 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc0033f0 con 0x7f6ed00fd540 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3385904526 0 0) 0x7f6ebc0033f0 con 0x7f6ed00fd540 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed01015a0 con 0x7f6ed00fd540 2024-02-10T08:27:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ec8001590 con 0x7f6ed00fd540 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f6ec8001d10 con 0x7f6ed00fd540 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.457+0000 7f6ed5419700 1 -- 172.21.15.107:0/1420959041 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ec8005750 con 0x7f6ed00fd540 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- 172.21.15.107:0/1420959041 >> v1:172.21.15.107:6789/0 conn(0x7f6ed00fd540 legacy=0x7f6ed00fd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- 172.21.15.107:0/1420959041 shutdown_connections 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- 172.21.15.107:0/1420959041 >> 172.21.15.107:0/1420959041 conn(0x7f6ed00f7690 msgr2=0x7f6ed00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- 172.21.15.107:0/1420959041 shutdown_connections 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- 172.21.15.107:0/1420959041 wait complete. 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 librados: starting msgr at 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 librados: starting objecter 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 Processor -- start 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 -- start start 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 librados: setting wanted keys 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.458+0000 7f6ed867f700 1 librados: calling monclient init 2024-02-10T08:27:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed867f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ed010b080 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed641b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6ed00fd540 0x7f6ed010a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48906/0 (socket says 172.21.15.107:48906) 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed641b700 1 -- 172.21.15.107:0/3053169685 learned_addr learned my addr 172.21.15.107:0/3053169685 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 10 client.?.objecter ms_handle_connect 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060704959 0 0) 0x7f6ed010b080 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc005580 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815411179 0 0) 0x7f6ebc005580 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ed010b080 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ec8000ff0 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 494708515 0 0) 0x7f6ed010b080 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed010b210 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed010b4c0 con 0x7f6ed00fd540 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed867f700 10 client.14110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed010f1b0 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.459+0000 7f6ed867f700 1 librados: init done 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f6ec80046b0 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ec8006240 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 3) v1 ==== 38465+0+0 (unknown 3170577179 0 0) 0x7f6ec800fab0 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f6ec803f7f0 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 10 client.14110.objecter ms_dispatch 0x7f6ed0001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 3 client.14110.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.460+0000 7f6ec77fe700 3 client.14110.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.461+0000 7f6ec77fe700 20 client.14110.objecter dump_active .. 0 homeless 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.461+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed005a3b0 con 0x7f6ed00fd540 2024-02-10T08:27:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.461+0000 7f6ec77fe700 10 client.14110.objecter ms_handle_connect 0x7f6ebc0352a0 2024-02-10T08:27:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.466+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6ec80110a0 con 0x7f6ed00fd540 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.690+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6ed0101110 con 0x7f6ed00fd540 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.690+0000 7f6ec77fe700 1 -- 172.21.15.107:0/3053169685 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (unknown 4201413639 0 1710268723) 0x7f6ec8014830 con 0x7f6ed00fd540 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:27:58.693 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "a" 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:58.694 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:27:58.695 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:27:58.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:27:58.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "nfs", 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:27:58.701 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:27:44.316616+0000", 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 10 librados: watch_flush enter 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 10 librados: watch_flush exit 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 20 client.14110.objecter shutdown clearing up homeless session... 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 10 client.14110.objecter successfully canceled tick 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 >> v1:172.21.15.107:6800/4244374259 conn(0x7f6ebc0352a0 legacy=0x7f6ebc037760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 >> v1:172.21.15.107:6789/0 conn(0x7f6ed00fd540 legacy=0x7f6ed010a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 shutdown_connections 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 >> 172.21.15.107:0/3053169685 conn(0x7f6ed00f7690 msgr2=0x7f6ed00f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 shutdown_connections 2024-02-10T08:27:58.702 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 -- 172.21.15.107:0/3053169685 wait complete. 2024-02-10T08:27:58.703 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:58.692+0000 7f6ed867f700 1 librados: shutdown 2024-02-10T08:27:58.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:57.714216+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e3: y(active, since 1.00581s) 2024-02-10T08:27:58.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:58.691044+0000 mon.a (mon.0) 26 : audit [DBG] from='client.? v1:172.21.15.107:0/3053169685' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:27:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:mgr is available 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.767+0000 7f67d69fb700 1 Processor -- start 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.767+0000 7f67d69fb700 1 -- start start 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.767+0000 7f67d69fb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67d0100240 con 0x7f67d0070ce0 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67cffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f67d0070ce0 0x7f67d00feb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43172/0 (socket says 172.21.15.107:43172) 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67cffff700 1 -- 172.21.15.107:0/3509879582 learned_addr learned my addr 172.21.15.107:0/3509879582 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232914720 0 0) 0x7f67d0100240 con 0x7f67d0070ce0 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b80033f0 con 0x7f67d0070ce0 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2636657827 0 0) 0x7f67b80033f0 con 0x7f67d0070ce0 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d01013e0 con 0x7f67d0070ce0 2024-02-10T08:27:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f67c0001590 con 0x7f67d0070ce0 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f67c0001d10 con 0x7f67d0070ce0 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.768+0000 7f67ceffd700 1 -- 172.21.15.107:0/3509879582 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f67c0005750 con 0x7f67d0070ce0 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- 172.21.15.107:0/3509879582 >> v1:172.21.15.107:6789/0 conn(0x7f67d0070ce0 legacy=0x7f67d00feb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- 172.21.15.107:0/3509879582 shutdown_connections 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- 172.21.15.107:0/3509879582 >> 172.21.15.107:0/3509879582 conn(0x7f67d00f6520 msgr2=0x7f67d00f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- 172.21.15.107:0/3509879582 shutdown_connections 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- 172.21.15.107:0/3509879582 wait complete. 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 librados: starting msgr at 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 librados: starting objecter 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 Processor -- start 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 -- start start 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 librados: setting wanted keys 2024-02-10T08:27:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.769+0000 7f67d69fb700 1 librados: calling monclient init 2024-02-10T08:27:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67d69fb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67d010b010 con 0x7f67d0070ce0 2024-02-10T08:27:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f67d0070ce0 0x7f67d010a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43180/0 (socket says 172.21.15.107:43180) 2024-02-10T08:27:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cffff700 1 -- 172.21.15.107:0/1202613835 learned_addr learned my addr 172.21.15.107:0/1202613835 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:27:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 10 client.?.objecter ms_handle_connect 0x7f67d0070ce0 2024-02-10T08:27:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478954722 0 0) 0x7f67d010b010 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b8005580 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762310710 0 0) 0x7f67b8005580 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d010b010 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f67c0000fe0 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2280886150 0 0) 0x7f67d010b010 con 0x7f67d0070ce0 2024-02-10T08:27:59.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d010b1a0 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.770+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d010b450 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.771+0000 7f67d69fb700 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.771+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d010f140 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.771+0000 7f67d69fb700 1 librados: init done 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.771+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f67c0004b40 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.771+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f67c0006120 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc004f50 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 38571+0+0 (unknown 1487916338 0 0) 0x7f67c0001900 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f67c003ea60 con 0x7f67d0070ce0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 10 client.14112.objecter ms_dispatch 0x7f67d0001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 20 client.14112.objecter dump_active .. 0 homeless 2024-02-10T08:27:59.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.772+0000 7f67cd7fa700 10 client.14112.objecter ms_handle_connect 0x7f67b80352b0 2024-02-10T08:27:59.777 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.776+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f67c0005a40 con 0x7f67d0070ce0 2024-02-10T08:27:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.937+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f67bc005780 con 0x7f67d0070ce0 2024-02-10T08:27:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.937+0000 7f67cd7fa700 1 -- 172.21.15.107:0/1202613835 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+307 (unknown 1187553405 0 4223055987) 0x7f67c0013870 con 0x7f67d0070ce0 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [global] 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: fsid = 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [mgr] 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [osd] 2024-02-10T08:27:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 10 librados: watch_flush enter 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 10 librados: watch_flush exit 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 20 client.14112.objecter shutdown clearing up homeless session... 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 10 client.14112.objecter successfully canceled tick 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 >> v1:172.21.15.107:6800/4244374259 conn(0x7f67b80352b0 legacy=0x7f67b8037770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 >> v1:172.21.15.107:6789/0 conn(0x7f67d0070ce0 legacy=0x7f67d010a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 shutdown_connections 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 >> 172.21.15.107:0/1202613835 conn(0x7f67d00f6520 msgr2=0x7f67d00fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 shutdown_connections 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.939+0000 7f67d69fb700 1 -- 172.21.15.107:0/1202613835 wait complete. 2024-02-10T08:27:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:27:59.940+0000 7f67d69fb700 1 librados: shutdown 2024-02-10T08:28:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:27:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:27:58.715458+0000 mon.a (mon.0) 27 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-02-10T08:28:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling cephadm module... 2024-02-10T08:28:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:27:59.938117+0000 mon.a (mon.0) 28 : audit [INF] from='client.? v1:172.21.15.107:0/1202613835' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T08:28:00.959 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.957+0000 7fce20e1e700 1 Processor -- start 2024-02-10T08:28:00.959 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.957+0000 7fce20e1e700 1 -- start start 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.957+0000 7fce20e1e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce1c070d20 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1a59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce1c0fcae0 0x7fce1c0feed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43184/0 (socket says 172.21.15.107:43184) 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1a59c700 1 -- 172.21.15.107:0/3080507454 learned_addr learned my addr 172.21.15.107:0/3080507454 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188091006 0 0) 0x7fce1c070d20 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce040033f0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371719703 0 0) 0x7fce040033f0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c070ed0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fce10001590 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fce10001d10 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.958+0000 7fce1959a700 1 -- 172.21.15.107:0/3080507454 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fce10005750 con 0x7fce1c0fcae0 2024-02-10T08:28:00.960 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 -- 172.21.15.107:0/3080507454 >> v1:172.21.15.107:6789/0 conn(0x7fce1c0fcae0 legacy=0x7fce1c0feed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:00.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 -- 172.21.15.107:0/3080507454 shutdown_connections 2024-02-10T08:28:00.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 -- 172.21.15.107:0/3080507454 >> 172.21.15.107:0/3080507454 conn(0x7fce1c0f65b0 msgr2=0x7fce1c0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:00.961 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 -- 172.21.15.107:0/3080507454 shutdown_connections 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 -- 172.21.15.107:0/3080507454 wait complete. 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 librados: starting msgr at 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.959+0000 7fce20e1e700 1 librados: starting objecter 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce20e1e700 1 Processor -- start 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce20e1e700 1 -- start start 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce20e1e700 1 librados: setting wanted keys 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce20e1e700 1 librados: calling monclient init 2024-02-10T08:28:00.964 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce20e1e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce1c10b030 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce1a59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce1c0fcae0 0x7fce1c10a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43192/0 (socket says 172.21.15.107:43192) 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce1a59c700 1 -- 172.21.15.107:0/4230095401 learned_addr learned my addr 172.21.15.107:0/4230095401 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce0b7fe700 10 client.?.objecter ms_handle_connect 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce0b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179988329 0 0) 0x7fce1c10b030 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.960+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce04005580 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 78747781 0 0) 0x7fce04005580 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce1c10b030 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fce10000fe0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 883384280 0 0) 0x7fce1c10b030 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c10b1c0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c10b470 con 0x7fce1c0fcae0 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce20e1e700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c10f160 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce20e1e700 1 librados: init done 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fce100046b0 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.961+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fce10006120 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 38571+0+0 (unknown 1487916338 0 0) 0x7fce1000f940 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7fce1003f940 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 10 client.14114.objecter ms_dispatch 0x7fce1c001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 20 client.14114.objecter dump_active .. 0 homeless 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc004f50 con 0x7fce1c0fcae0 2024-02-10T08:28:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.962+0000 7fce0b7fe700 10 client.14114.objecter ms_handle_connect 0x7fce04035300 2024-02-10T08:28:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:00.967+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fce10010f80 con 0x7fce1c0fcae0 2024-02-10T08:28:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.169+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcdfc005200 con 0x7fce1c0fcae0 2024-02-10T08:28:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.718+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mgrmap(e 5) v1 ==== 38582+0+0 (unknown 3253396151 0 0) 0x7fce1003ef30 con 0x7fce1c0fcae0 2024-02-10T08:28:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.719+0000 7fce0b7fe700 1 -- 172.21.15.107:0/4230095401 <== mon.0 v1:172.21.15.107:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7fce10012f10 con 0x7fce1c0fcae0 2024-02-10T08:28:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 10 librados: watch_flush enter 2024-02-10T08:28:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 10 librados: watch_flush exit 2024-02-10T08:28:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 20 client.14114.objecter shutdown clearing up homeless session... 2024-02-10T08:28:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 10 client.14114.objecter successfully canceled tick 2024-02-10T08:28:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 >> v1:172.21.15.107:6800/4244374259 conn(0x7fce04035300 legacy=0x7fce040377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 >> v1:172.21.15.107:6789/0 conn(0x7fce1c0fcae0 legacy=0x7fce1c10a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 shutdown_connections 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.721+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 >> 172.21.15.107:0/4230095401 conn(0x7fce1c0f65b0 msgr2=0x7fce1c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.722+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 shutdown_connections 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.722+0000 7fce20e1e700 1 -- 172.21.15.107:0/4230095401 wait complete. 2024-02-10T08:28:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:01.722+0000 7fce20e1e700 1 librados: shutdown 2024-02-10T08:28:02.087 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:01.170892+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.107:0/4230095401' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T08:28:02.087 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setuser ceph since I am not root 2024-02-10T08:28:02.087 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setgroup ceph since I am not root 2024-02-10T08:28:02.088 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:01.881+0000 7f05f5407680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:28:02.088 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:01.960+0000 7f05f5407680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:28:02.368 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:02.321+0000 7f05f5407680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f07654ae700 1 Processor -- start 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f07654ae700 1 -- start start 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f07654ae700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0760070d20 con 0x7f07600fcad0 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f075effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f07600fcad0 0x7f07600feec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43220/0 (socket says 172.21.15.107:43220) 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f075effd700 1 -- 172.21.15.107:0/3246314621 learned_addr learned my addr 172.21.15.107:0/3246314621 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292573618 0 0) 0x7f0760070d20 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.916+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07480033f0 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276113767 0 0) 0x7f07480033f0 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0760070ed0 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0750001590 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f0750001d10 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f075dffb700 1 -- 172.21.15.107:0/3246314621 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0750005750 con 0x7f07600fcad0 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f07654ae700 1 -- 172.21.15.107:0/3246314621 >> v1:172.21.15.107:6789/0 conn(0x7f07600fcad0 legacy=0x7f07600feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f07654ae700 1 -- 172.21.15.107:0/3246314621 shutdown_connections 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f07654ae700 1 -- 172.21.15.107:0/3246314621 >> 172.21.15.107:0/3246314621 conn(0x7f07600f6520 msgr2=0x7f07600f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f07654ae700 1 -- 172.21.15.107:0/3246314621 shutdown_connections 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.917+0000 7f07654ae700 1 -- 172.21.15.107:0/3246314621 wait complete. 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 librados: starting msgr at 2024-02-10T08:28:02.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 librados: starting objecter 2024-02-10T08:28:02.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 Processor -- start 2024-02-10T08:28:02.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 -- start start 2024-02-10T08:28:02.922 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 librados: setting wanted keys 2024-02-10T08:28:02.922 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 librados: calling monclient init 2024-02-10T08:28:02.922 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.918+0000 7f07654ae700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f076010b020 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f075effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f07600fcad0 0x7f076010a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43224/0 (socket says 172.21.15.107:43224) 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f075effd700 1 -- 172.21.15.107:0/3589588683 learned_addr learned my addr 172.21.15.107:0/3589588683 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 10 client.?.objecter ms_handle_connect 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808449292 0 0) 0x7f076010b020 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748005580 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156107972 0 0) 0x7f0748005580 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f076010b020 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0750000fe0 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2613512233 0 0) 0x7f076010b020 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076010b1b0 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076010b460 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f07654ae700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f076010f150 con 0x7f07600fcad0 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f07654ae700 1 librados: init done 2024-02-10T08:28:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f0750005740 con 0x7f07600fcad0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.919+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0750006230 con 0x7f07600fcad0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.920+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 5) v1 ==== 38582+0+0 (unknown 3253396151 0 0) 0x7f07500063b0 con 0x7f07600fcad0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.920+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c004f50 con 0x7f07600fcad0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f075003ec90 con 0x7f07600fcad0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f0747fff700 10 client.14118.objecter ms_dispatch 0x7f0760001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f0747fff700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f0747fff700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f075e7fc700 1 -- 172.21.15.107:0/3589588683 >> v1:172.21.15.107:6800/4244374259 conn(0x7f07480352b0 legacy=0x7f0748037770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.921+0000 7f0747fff700 20 client.14118.objecter dump_active .. 0 homeless 2024-02-10T08:28:02.925 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:02.925+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0750007ad0 con 0x7f07600fcad0 2024-02-10T08:28:03.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:01.720179+0000 mon.a (mon.0) 30 : audit [INF] from='client.? v1:172.21.15.107:0/4230095401' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T08:28:03.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:01.720228+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e5: y(active, since 5s) 2024-02-10T08:28:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.122+0000 7f075e7fc700 1 -- 172.21.15.107:0/3589588683 >> v1:172.21.15.107:6800/4244374259 conn(0x7f07480352b0 legacy=0x7f0748037770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.174+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f074c005a10 con 0x7f07600fcad0 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.174+0000 7f0747fff700 1 -- 172.21.15.107:0/3589588683 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f0750014970 con 0x7f07600fcad0 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T08:28:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T08:28:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:28:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 10 librados: watch_flush enter 2024-02-10T08:28:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 10 librados: watch_flush exit 2024-02-10T08:28:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 20 client.14118.objecter shutdown clearing up homeless session... 2024-02-10T08:28:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 10 client.14118.objecter successfully canceled tick 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 >> v1:172.21.15.107:6800/4244374259 conn(0x7f07480352b0 legacy=0x7f0748037770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 >> v1:172.21.15.107:6789/0 conn(0x7f07600fcad0 legacy=0x7f076010a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 shutdown_connections 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 >> 172.21.15.107:0/3589588683 conn(0x7f07600f6520 msgr2=0x7f07600fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 shutdown_connections 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.176+0000 7f07654ae700 1 -- 172.21.15.107:0/3589588683 wait complete. 2024-02-10T08:28:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:03.177+0000 7f07654ae700 1 librados: shutdown 2024-02-10T08:28:03.381 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:03.081+0000 7f05f5407680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:28:03.381 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:03.166+0000 7f05f5407680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:28:03.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:03.380+0000 7f05f5407680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:28:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for the mgr to restart... 2024-02-10T08:28:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr epoch 5... 2024-02-10T08:28:04.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:03.175017+0000 mon.a (mon.0) 32 : audit [DBG] from='client.? v1:172.21.15.107:0/3589588683' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T08:28:04.418 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.379+0000 7f085c93a700 1 Processor -- start 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.379+0000 7f085c93a700 1 -- start start 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.379+0000 7f085c93a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0858100280 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.379+0000 7f085659c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0858070ca0 0x7f08580feb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43230/0 (socket says 172.21.15.107:43230) 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.379+0000 7f085659c700 1 -- 172.21.15.107:0/1357013134 learned_addr learned my addr 172.21.15.107:0/1357013134 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809934246 0 0) 0x7f0858100280 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08400033f0 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1011843156 0 0) 0x7f08400033f0 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0858101420 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0848001590 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f0848001d10 con 0x7f0858070ca0 2024-02-10T08:28:04.419 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085559a700 1 -- 172.21.15.107:0/1357013134 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0848005750 con 0x7f0858070ca0 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085c93a700 1 -- 172.21.15.107:0/1357013134 >> v1:172.21.15.107:6789/0 conn(0x7f0858070ca0 legacy=0x7f08580feb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085c93a700 1 -- 172.21.15.107:0/1357013134 shutdown_connections 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.380+0000 7f085c93a700 1 -- 172.21.15.107:0/1357013134 >> 172.21.15.107:0/1357013134 conn(0x7f08580f6520 msgr2=0x7f08580f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 -- 172.21.15.107:0/1357013134 shutdown_connections 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 -- 172.21.15.107:0/1357013134 wait complete. 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 librados: starting msgr at 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 librados: starting objecter 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 Processor -- start 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 -- start start 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 librados: setting wanted keys 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 librados: calling monclient init 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.381+0000 7f085c93a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f08580fc500 con 0x7f0858070ca0 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085659c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0858070ca0 0x7f08580faa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43242/0 (socket says 172.21.15.107:43242) 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085659c700 1 -- 172.21.15.107:0/1633134030 learned_addr learned my addr 172.21.15.107:0/1633134030 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 10 client.?.objecter ms_handle_connect 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986364208 0 0) 0x7f08580fc500 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0840005580 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3671275250 0 0) 0x7f0840005580 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08580fc500 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0848000fe0 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1441380449 0 0) 0x7f08580fc500 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08580fc690 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08580fc940 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085c93a700 10 client.14120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08580fb410 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f085c93a700 1 librados: init done 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f0848001b30 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.382+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0848006160 con 0x7f0858070ca0 2024-02-10T08:28:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 5) v1 ==== 38582+0+0 (unknown 3253396151 0 0) 0x7f084800fa60 con 0x7f0858070ca0 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6800/4244374259 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0838000d40 con 0x7f08400352a0 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 1578396782 0 0) 0x7f084803faf0 con 0x7f0858070ca0 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f0855d9b700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/4244374259 conn(0x7f08400352a0 legacy=0x7f0840037760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 10 client.14120.objecter ms_dispatch 0x7f0858001550 osd_map(1..1 src has 1..1) v4 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 3 client.14120.objecter handle_osd_map got epochs [1,1] > 0 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 3 client.14120.objecter handle_osd_map decoding full epoch 1 2024-02-10T08:28:04.422 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.383+0000 7f08477fe700 20 client.14120.objecter dump_active .. 0 homeless 2024-02-10T08:28:04.585 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.584+0000 7f0855d9b700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/4244374259 conn(0x7f08400352a0 legacy=0x7f0840037760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:04.628 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:04.329+0000 7f05f5407680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:28:04.628 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:04.462+0000 7f05f5407680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:28:04.628 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:04.539+0000 7f05f5407680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:28:04.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:04.628+0000 7f05f5407680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:28:04.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:04.821+0000 7f05f5407680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:28:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:04.985+0000 7f0855d9b700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/4244374259 conn(0x7f08400352a0 legacy=0x7f0840037760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:05.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:05.098+0000 7f05f5407680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:28:05.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:05.193+0000 7f05f5407680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:28:05.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:05.785+0000 7f0855d9b700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/4244374259 conn(0x7f08400352a0 legacy=0x7f0840037760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/4244374259 2024-02-10T08:28:06.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:05.906+0000 7f05f5407680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:28:06.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:05.987+0000 7f05f5407680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:28:06.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.079+0000 7f05f5407680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:28:06.506 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.252+0000 7f05f5407680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:28:06.767 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.505+0000 7f05f5407680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:28:06.767 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.634+0000 7f05f5407680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:28:07.032 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:07.029+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mgrmap(e 6) v1 ==== 38441+0+0 (unknown 2960631281 0 0) 0x7f0848005a40 con 0x7f0858070ca0 2024-02-10T08:28:07.032 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:07.030+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/4244374259 conn(0x7f08400352a0 legacy=0x7f0840037760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:28:07.080 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.765+0000 7f05f5407680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:28:07.081 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:06.933+0000 7f05f5407680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:28:07.081 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:07.015+0000 7f05f5407680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:28:07.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:07.029482+0000 mon.a (mon.0) 33 : cluster [INF] Active manager daemon y restarted 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:07.029696+0000 mon.a (mon.0) 34 : cluster [INF] Activating manager daemon y 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:07.032672+0000 mon.a (mon.0) 35 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:07.032770+0000 mon.a (mon.0) 36 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00315704s) 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.034228+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.034704+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.035287+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.035476+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.035649+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:07.049087+0000 mon.a (mon.0) 42 : cluster [INF] Manager daemon y is now available 2024-02-10T08:28:07.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.060860+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.031+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mgrmap(e 7) v1 ==== 38476+0+0 (unknown 3588192853 0 0) 0x7f084803f0e0 con 0x7f0858070ca0 2024-02-10T08:28:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.031+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6800/2081774553 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0838000d40 con 0x7f0840039cc0 2024-02-10T08:28:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.032+0000 7f08477fe700 10 client.14120.objecter ms_handle_connect 0x7f0840039cc0 2024-02-10T08:28:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.033+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (unknown 0 0 4198994610) 0x7f0838000d40 con 0x7f0840039cc0 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 --> v1:172.21.15.107:6800/2081774553 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0838002610 con 0x7f0840039cc0 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f08477fe700 1 -- 172.21.15.107:0/1633134030 <== mgr.14116 v1:172.21.15.107:6800/2081774553 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 96372106) 0x7f0838002610 con 0x7f0840039cc0 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:28:08.042 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 10 librados: watch_flush enter 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 10 librados: watch_flush exit 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 20 client.14120.objecter shutdown clearing up homeless session... 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 10 client.14120.objecter successfully canceled tick 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6800/2081774553 conn(0x7f0840039cc0 legacy=0x7f084003c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 >> v1:172.21.15.107:6789/0 conn(0x7f0858070ca0 legacy=0x7f08580faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 shutdown_connections 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 >> 172.21.15.107:0/1633134030 conn(0x7f08580f6520 msgr2=0x7f08580fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 shutdown_connections 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 -- 172.21.15.107:0/1633134030 wait complete. 2024-02-10T08:28:08.043 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:08.039+0000 7f085c93a700 1 librados: shutdown 2024-02-10T08:28:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:07.058325+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-10T08:28:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.144146+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.145060+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.146397+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.148516+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.161988+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:28:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:07.164583+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:28:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:08.035633+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e7: y(active, since 1.00602s) 2024-02-10T08:28:09.004 INFO:teuthology.orchestra.run.smithi107.stderr:mgr epoch 5 is available 2024-02-10T08:28:09.004 INFO:teuthology.orchestra.run.smithi107.stderr:Setting orchestrator backend to cephadm... 2024-02-10T08:28:09.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:08.034336+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 v1:172.21.15.107:0/1633134030' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T08:28:09.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:08.040893+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14120 v1:172.21.15.107:0/1633134030' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f9612700 1 Processor -- start 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f9612700 1 -- start start 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f9612700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f40fcf60 con 0x7ff0f40fc430 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff0f40fc430 0x7ff0f40fc850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47450/0 (socket says 172.21.15.107:47450) 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f2ffd700 1 -- 172.21.15.107:0/485253633 learned_addr learned my addr 172.21.15.107:0/485253633 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942434640 0 0) 0x7ff0f40fcf60 con 0x7ff0f40fc430 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.491+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc0033f0 con 0x7ff0f40fc430 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871392400 0 0) 0x7ff0dc0033f0 con 0x7ff0f40fc430 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f41003a0 con 0x7ff0f40fc430 2024-02-10T08:28:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff0e4001590 con 0x7ff0f40fc430 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7ff0e4001d10 con 0x7ff0f40fc430 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f1ffb700 1 -- 172.21.15.107:0/485253633 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff0e4005750 con 0x7ff0f40fc430 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f9612700 1 -- 172.21.15.107:0/485253633 >> v1:172.21.15.107:6789/0 conn(0x7ff0f40fc430 legacy=0x7ff0f40fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f9612700 1 -- 172.21.15.107:0/485253633 shutdown_connections 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f9612700 1 -- 172.21.15.107:0/485253633 >> 172.21.15.107:0/485253633 conn(0x7ff0f40f5cc0 msgr2=0x7ff0f40f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f9612700 1 -- 172.21.15.107:0/485253633 shutdown_connections 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.492+0000 7ff0f9612700 1 -- 172.21.15.107:0/485253633 wait complete. 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 librados: starting msgr at 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 librados: starting objecter 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 Processor -- start 2024-02-10T08:28:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 -- start start 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 librados: setting wanted keys 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 librados: calling monclient init 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f9612700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f4109f40 con 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff0f40fc430 0x7ff0f4109830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47466/0 (socket says 172.21.15.107:47466) 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0f2ffd700 1 -- 172.21.15.107:0/2428376446 learned_addr learned my addr 172.21.15.107:0/2428376446 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0dbfff700 10 client.?.objecter ms_handle_connect 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.493+0000 7ff0dbfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040635565 0 0) 0x7ff0f4109f40 con 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc005580 con 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350198860 0 0) 0x7ff0dc005580 con 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0f4109f40 con 0x7ff0f40fc430 2024-02-10T08:28:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff0e4000ff0 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1196143341 0 0) 0x7ff0f4109f40 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f410a0d0 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f410a380 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0f9612700 10 client.14126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f410e070 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0f9612700 1 librados: init done 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7ff0e40046b0 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.494+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7ff0e40061b0 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7ff0e400fa90 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e0004f50 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7ff0e403ec90 con 0x7ff0f40fc430 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 10 client.14126.objecter ms_dispatch 0x7ff0f4001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 3 client.14126.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 3 client.14126.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 20 client.14126.objecter dump_active .. 0 homeless 2024-02-10T08:28:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.495+0000 7ff0dbfff700 10 client.14126.objecter ms_handle_connect 0x7ff0dc035230 2024-02-10T08:28:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.500+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff0e40111c0 con 0x7ff0f40fc430 2024-02-10T08:28:09.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.674+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0e0000bc0 con 0x7ff0dc035230 2024-02-10T08:28:09.683 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.681+0000 7ff0dbfff700 1 -- 172.21.15.107:0/2428376446 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7ff0e0000bc0 con 0x7ff0dc035230 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 10 librados: watch_flush enter 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 10 librados: watch_flush exit 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 20 client.14126.objecter shutdown clearing up homeless session... 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 10 client.14126.objecter successfully canceled tick 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 >> v1:172.21.15.107:6800/2081774553 conn(0x7ff0dc035230 legacy=0x7ff0dc0376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.683+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 >> v1:172.21.15.107:6789/0 conn(0x7ff0f40fc430 legacy=0x7ff0f4109830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.684+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 shutdown_connections 2024-02-10T08:28:09.687 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.684+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 >> 172.21.15.107:0/2428376446 conn(0x7ff0f40f5cc0 msgr2=0x7ff0f40f6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:09.687 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.684+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 shutdown_connections 2024-02-10T08:28:09.687 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.684+0000 7ff0f9612700 1 -- 172.21.15.107:0/2428376446 wait complete. 2024-02-10T08:28:09.687 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:09.684+0000 7ff0f9612700 1 librados: shutdown 2024-02-10T08:28:10.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:09.146839+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-10T08:28:10.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:09.676553+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 v1:172.21.15.107:0/2428376446' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:10.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:09.683283+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:10.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:09.722500+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.896+0000 7fd6711ce700 1 Processor -- start 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.896+0000 7fd6711ce700 1 -- start start 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd6711ce700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd66c100240 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd66ad9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd66c070ce0 0x7fd66c0feb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47472/0 (socket says 172.21.15.107:47472) 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd66ad9d700 1 -- 172.21.15.107:0/842578257 learned_addr learned my addr 172.21.15.107:0/842578257 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845631620 0 0) 0x7fd66c100240 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6540033f0 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361826367 0 0) 0x7fd6540033f0 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c1013e0 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd660001590 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fd660001c40 con 0x7fd66c070ce0 2024-02-10T08:28:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.897+0000 7fd669d9b700 1 -- 172.21.15.107:0/842578257 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd660005680 con 0x7fd66c070ce0 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.898+0000 7fd6711ce700 1 -- 172.21.15.107:0/842578257 >> v1:172.21.15.107:6789/0 conn(0x7fd66c070ce0 legacy=0x7fd66c0feb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.898+0000 7fd6711ce700 1 -- 172.21.15.107:0/842578257 shutdown_connections 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.898+0000 7fd6711ce700 1 -- 172.21.15.107:0/842578257 >> 172.21.15.107:0/842578257 conn(0x7fd66c0f6520 msgr2=0x7fd66c0f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.898+0000 7fd6711ce700 1 -- 172.21.15.107:0/842578257 shutdown_connections 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.898+0000 7fd6711ce700 1 -- 172.21.15.107:0/842578257 wait complete. 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 librados: starting msgr at 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 librados: starting objecter 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 Processor -- start 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 -- start start 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 librados: setting wanted keys 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 librados: calling monclient init 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.899+0000 7fd6711ce700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd66c10b040 con 0x7fd66c070ce0 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd66ad9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd66c070ce0 0x7fd66c10a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47480/0 (socket says 172.21.15.107:47480) 2024-02-10T08:28:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd66ad9d700 1 -- 172.21.15.107:0/3290257031 learned_addr learned my addr 172.21.15.107:0/3290257031 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 10 client.?.objecter ms_handle_connect 0x7fd66c070ce0 2024-02-10T08:28:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64887056 0 0) 0x7fd66c10b040 con 0x7fd66c070ce0 2024-02-10T08:28:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6540055d0 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659004659 0 0) 0x7fd6540055d0 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd66c10b040 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd660000ff0 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548740058 0 0) 0x7fd66c10b040 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c10b1d0 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c10b480 con 0x7fd66c070ce0 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd6711ce700 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c10f170 con 0x7fd66c070ce0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd6711ce700 1 librados: init done 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fd660004900 con 0x7fd66c070ce0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.900+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd660006170 con 0x7fd66c070ce0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7fd660007440 con 0x7fd66c070ce0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7fd66003fc20 con 0x7fd66c070ce0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 10 client.14128.objecter ms_dispatch 0x7fd66c001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 3 client.14128.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 3 client.14128.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 20 client.14128.objecter dump_active .. 0 homeless 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.901+0000 7fd653fff700 10 client.14128.objecter ms_handle_connect 0x7fd654035160 2024-02-10T08:28:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.902+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd658004f50 con 0x7fd66c070ce0 2024-02-10T08:28:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:10.906+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd6600081e0 con 0x7fd66c070ce0 2024-02-10T08:28:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.071+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd658000bc0 con 0x7fd654035160 2024-02-10T08:28:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.072+0000 7fd653fff700 1 -- 172.21.15.107:0/3290257031 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7fd658000bc0 con 0x7fd654035160 2024-02-10T08:28:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: value unchanged 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 10 librados: watch_flush enter 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 10 librados: watch_flush exit 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 20 client.14128.objecter shutdown clearing up homeless session... 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 10 client.14128.objecter successfully canceled tick 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 >> v1:172.21.15.107:6800/2081774553 conn(0x7fd654035160 legacy=0x7fd654037620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 >> v1:172.21.15.107:6789/0 conn(0x7fd66c070ce0 legacy=0x7fd66c10a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 shutdown_connections 2024-02-10T08:28:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 >> 172.21.15.107:0/3290257031 conn(0x7fd66c0f6520 msgr2=0x7fd66c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 shutdown_connections 2024-02-10T08:28:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 -- 172.21.15.107:0/3290257031 wait complete. 2024-02-10T08:28:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:11.074+0000 7fd6711ce700 1 librados: shutdown 2024-02-10T08:28:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:Generating ssh key... 2024-02-10T08:28:12.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:11.073935+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 v1:172.21.15.107:0/3290257031' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.187+0000 7fb5d938f700 1 Processor -- start 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.187+0000 7fb5d938f700 1 -- start start 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.187+0000 7fb5d938f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb5d4100260 con 0x7fb5d4070cc0 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.187+0000 7fb5d2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb5d4070cc0 0x7fb5d40feb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47496/0 (socket says 172.21.15.107:47496) 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d2ffd700 1 -- 172.21.15.107:0/1673063080 learned_addr learned my addr 172.21.15.107:0/1673063080 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600831189 0 0) 0x7fb5d4100260 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5bc0033f0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776007142 0 0) 0x7fb5bc0033f0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d4101400 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb5c8001590 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fb5c8001d10 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.188+0000 7fb5d1ffb700 1 -- 172.21.15.107:0/1673063080 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb5c8005750 con 0x7fb5d4070cc0 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- 172.21.15.107:0/1673063080 >> v1:172.21.15.107:6789/0 conn(0x7fb5d4070cc0 legacy=0x7fb5d40feb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- 172.21.15.107:0/1673063080 shutdown_connections 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- 172.21.15.107:0/1673063080 >> 172.21.15.107:0/1673063080 conn(0x7fb5d40f6520 msgr2=0x7fb5d40f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- 172.21.15.107:0/1673063080 shutdown_connections 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- 172.21.15.107:0/1673063080 wait complete. 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 librados: starting msgr at 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 librados: starting objecter 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 Processor -- start 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 -- start start 2024-02-10T08:28:12.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 librados: setting wanted keys 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.189+0000 7fb5d938f700 1 librados: calling monclient init 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5d938f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb5d40fc550 con 0x7fb5d4070cc0 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5d2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb5d4070cc0 0x7fb5d40faa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47504/0 (socket says 172.21.15.107:47504) 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5d2ffd700 1 -- 172.21.15.107:0/2897382119 learned_addr learned my addr 172.21.15.107:0/2897382119 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 10 client.?.objecter ms_handle_connect 0x7fb5d4070cc0 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986292872 0 0) 0x7fb5d40fc550 con 0x7fb5d4070cc0 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5bc005580 con 0x7fb5d4070cc0 2024-02-10T08:28:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213553218 0 0) 0x7fb5bc005580 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5d40fc550 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb5c8000fe0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1580175931 0 0) 0x7fb5d40fc550 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d40fc6e0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d40fc990 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.190+0000 7fb5d938f700 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d40fb450 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5d938f700 1 librados: init done 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fb5c8004b40 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fb5c80061d0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7fb5c800f9f0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.191+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c0004f50 con 0x7fb5d4070cc0 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7fb5c803ec00 con 0x7fb5d4070cc0 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 10 client.14130.objecter ms_dispatch 0x7fb5d4001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 20 client.14130.objecter dump_active .. 0 homeless 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.192+0000 7fb5bbfff700 10 client.14130.objecter ms_handle_connect 0x7fb5bc035220 2024-02-10T08:28:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.196+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb5c8010fc0 con 0x7fb5d4070cc0 2024-02-10T08:28:12.360 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:12.358+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5c0000bc0 con 0x7fb5bc035220 2024-02-10T08:28:13.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.005+0000 7fb5bbfff700 1 -- 172.21.15.107:0/2897382119 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7fb5c0000bc0 con 0x7fb5bc035220 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 10 librados: watch_flush enter 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 10 librados: watch_flush exit 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 20 client.14130.objecter shutdown clearing up homeless session... 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 10 client.14130.objecter successfully canceled tick 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 >> v1:172.21.15.107:6800/2081774553 conn(0x7fb5bc035220 legacy=0x7fb5bc0376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 >> v1:172.21.15.107:6789/0 conn(0x7fb5d4070cc0 legacy=0x7fb5d40faa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 shutdown_connections 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.007+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 >> 172.21.15.107:0/2897382119 conn(0x7fb5d40f6520 msgr2=0x7fb5d40fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.008+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 shutdown_connections 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.008+0000 7fb5d938f700 1 -- 172.21.15.107:0/2897382119 wait complete. 2024-02-10T08:28:13.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:13.008+0000 7fb5d938f700 1 librados: shutdown 2024-02-10T08:28:13.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:12.360928+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 v1:172.21.15.107:0/2897382119' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:13.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:12.361130+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: Generating public/private rsa key pair. 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: Your identification has been saved in /tmp/tmpcsjcugq8/key. 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: Your public key has been saved in /tmp/tmpcsjcugq8/key.pub. 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: The key fingerprint is: 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: SHA256:1MtOyeP0NfuMV3ODlTMzEL1NulLb1opx1z8PzzRheA8 ceph-275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: The key's randomart image is: 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: +---[RSA 3072]----+ 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | .o | 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | . . ..| 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | . . .o+| 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | . o o +O.| 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | S O oEBB| 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | = ooo**@| 2024-02-10T08:28:13.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | o .=o=O| 2024-02-10T08:28:13.182 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | . .X+| 2024-02-10T08:28:13.182 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: | ..O| 2024-02-10T08:28:13.182 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: +----[SHA256]-----+ 2024-02-10T08:28:14.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:13.001640+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:14.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:13.006780+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.344+0000 7fe7df314700 1 Processor -- start 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.344+0000 7fe7df314700 1 -- start start 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7df314700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe7d8070d20 con 0x7fe7d80fcad0 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7dd0b0700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe7d80fcad0 0x7fe7d80feec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47512/0 (socket says 172.21.15.107:47512) 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7dd0b0700 1 -- 172.21.15.107:0/2374373917 learned_addr learned my addr 172.21.15.107:0/2374373917 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983861085 0 0) 0x7fe7d8070d20 con 0x7fe7d80fcad0 2024-02-10T08:28:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c40033f0 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 405552536 0 0) 0x7fe7c40033f0 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8070ed0 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe7d4001590 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fe7d4001d10 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.345+0000 7fe7cffff700 1 -- 172.21.15.107:0/2374373917 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe7d4005750 con 0x7fe7d80fcad0 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 -- 172.21.15.107:0/2374373917 >> v1:172.21.15.107:6789/0 conn(0x7fe7d80fcad0 legacy=0x7fe7d80feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 -- 172.21.15.107:0/2374373917 shutdown_connections 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 -- 172.21.15.107:0/2374373917 >> 172.21.15.107:0/2374373917 conn(0x7fe7d80f6520 msgr2=0x7fe7d80f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 -- 172.21.15.107:0/2374373917 shutdown_connections 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 -- 172.21.15.107:0/2374373917 wait complete. 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 librados: starting msgr at 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.346+0000 7fe7df314700 1 librados: starting objecter 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7df314700 1 Processor -- start 2024-02-10T08:28:14.352 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7df314700 1 -- start start 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7df314700 1 librados: setting wanted keys 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7df314700 1 librados: calling monclient init 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7df314700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe7d810b020 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7dd0b0700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe7d80fcad0 0x7fe7d810a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47520/0 (socket says 172.21.15.107:47520) 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7dd0b0700 1 -- 172.21.15.107:0/3748002064 learned_addr learned my addr 172.21.15.107:0/3748002064 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 10 client.?.objecter ms_handle_connect 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873152467 0 0) 0x7fe7d810b020 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4005580 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570324822 0 0) 0x7fe7c4005580 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d810b020 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe7d4000fe0 con 0x7fe7d80fcad0 2024-02-10T08:28:14.353 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3305711787 0 0) 0x7fe7d810b020 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.347+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d810b1b0 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d810b460 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7df314700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d810f150 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7df314700 1 librados: init done 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fe7d4001b30 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fe7d4006300 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7fe7d400fb40 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7fe7d403ec40 con 0x7fe7d80fcad0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 10 client.14132.objecter ms_dispatch 0x7fe7d8001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.348+0000 7fe7ce7fc700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.349+0000 7fe7ce7fc700 20 client.14132.objecter dump_active .. 0 homeless 2024-02-10T08:28:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.349+0000 7fe7ce7fc700 10 client.14132.objecter ms_handle_connect 0x7fe7c4035280 2024-02-10T08:28:14.355 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.349+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc004f50 con 0x7fe7d80fcad0 2024-02-10T08:28:14.355 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.353+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe7d4011180 con 0x7fe7d80fcad0 2024-02-10T08:28:14.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.517+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7bc000bc0 con 0x7fe7c4035280 2024-02-10T08:28:14.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.517+0000 7fe7ce7fc700 1 -- 172.21.15.107:0/3748002064 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 2292581364) 0x7fe7bc000bc0 con 0x7fe7c4035280 2024-02-10T08:28:14.519 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+R2qzAiXC7xfFomIoay0WZufWxnlhb0oRlR+Y3Pv3SZ1Io+SEMR2gGSNUfOOlY3egV+MsjEzisz+HEoa3S5QLRTmatRzvcXXSRUVJb4Lpc5CXv60AMs233ihTPdYeOxQc5rlCJDcw+w2B/fb6OXSetM/XMKHkbEKRbqQImPxNjH6m085T8HXbqyAbMPC4rb0HFjgaCICLFtjzegnsW99IwbQaQ78qcyAY6ZOmtC11F3ITQ10cRWxTz5RQjUK+birIwvUvjgMJxleDbWJiq9qZo/bYs3gL35/hwkoW7ij+iA+gWYbst66thDAhS4HQ+lJzNYvfqxkB9k8eV832O0vN4OgLYVsU6DYqS1Mbvc9YuqlIighhVBYWGfNMlY3rJyUDl6OSln97FGMjA71j00Qk8FCJ7XciPKsOO52pmw6NiqW/YBzN1OEJSlPDifi4+DLiGnhoJxRB7ERE+1R7lrtKuMv7Zm6iAR4cTwMMCDNLQ6URfLASKTxSrIxnqItjl1M= ceph-275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:28:14.521 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 10 librados: watch_flush enter 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 10 librados: watch_flush exit 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 20 client.14132.objecter shutdown clearing up homeless session... 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 10 client.14132.objecter successfully canceled tick 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 >> v1:172.21.15.107:6800/2081774553 conn(0x7fe7c4035280 legacy=0x7fe7c4037740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 >> v1:172.21.15.107:6789/0 conn(0x7fe7d80fcad0 legacy=0x7fe7d810a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 shutdown_connections 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 >> 172.21.15.107:0/3748002064 conn(0x7fe7d80f6520 msgr2=0x7fe7d80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 shutdown_connections 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 -- 172.21.15.107:0/3748002064 wait complete. 2024-02-10T08:28:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:14.519+0000 7fe7df314700 1 librados: shutdown 2024-02-10T08:28:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T08:28:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T08:28:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:Adding host smithi107... 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.660+0000 7fa4d803b700 1 Processor -- start 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.660+0000 7fa4d803b700 1 -- start start 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.660+0000 7fa4d803b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4d00fc940 con 0x7fa4d0070cc0 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.660+0000 7fa4d5dd7700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4d0070cc0 0x7fa4d0100f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47530/0 (socket says 172.21.15.107:47530) 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.660+0000 7fa4d5dd7700 1 -- 172.21.15.107:0/2382727371 learned_addr learned my addr 172.21.15.107:0/2382727371 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466894585 0 0) 0x7fa4d00fc940 con 0x7fa4d0070cc0 2024-02-10T08:28:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc0033f0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1984933842 0 0) 0x7fa4bc0033f0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0103680 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa4cc001590 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fa4cc001d10 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d4dd5700 1 -- 172.21.15.107:0/2382727371 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa4cc0056e0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.661+0000 7fa4d803b700 1 -- 172.21.15.107:0/2382727371 >> v1:172.21.15.107:6789/0 conn(0x7fa4d0070cc0 legacy=0x7fa4d0100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- 172.21.15.107:0/2382727371 shutdown_connections 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- 172.21.15.107:0/2382727371 >> 172.21.15.107:0/2382727371 conn(0x7fa4d00f65b0 msgr2=0x7fa4d00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- 172.21.15.107:0/2382727371 shutdown_connections 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- 172.21.15.107:0/2382727371 wait complete. 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 librados: starting msgr at 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 librados: starting objecter 2024-02-10T08:28:15.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 Processor -- start 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- start start 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 librados: setting wanted keys 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 librados: calling monclient init 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.662+0000 7fa4d803b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4d010d240 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d5dd7700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4d0070cc0 0x7fa4d010cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47546/0 (socket says 172.21.15.107:47546) 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d5dd7700 1 -- 172.21.15.107:0/1145252253 learned_addr learned my addr 172.21.15.107:0/1145252253 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 10 client.?.objecter ms_handle_connect 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146785963 0 0) 0x7fa4d010d240 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc005580 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008630647 0 0) 0x7fa4bc005580 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d010d240 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa4cc000fe0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 593505847 0 0) 0x7fa4d010d240 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d010d3d0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d010d680 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d803b700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0111300 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.663+0000 7fa4d803b700 1 librados: init done 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7fa4cc0046b0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa4cc0061d0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7fa4cc00fad0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b4004f50 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7fa4cc03fbd0 con 0x7fa4d0070cc0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 10 client.14134.objecter ms_dispatch 0x7fa4d0001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.664+0000 7fa4c6ffd700 20 client.14134.objecter dump_active .. 0 homeless 2024-02-10T08:28:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.665+0000 7fa4c6ffd700 10 client.14134.objecter ms_handle_connect 0x7fa4bc035260 2024-02-10T08:28:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.669+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa4cc011110 con 0x7fa4d0070cc0 2024-02-10T08:28:15.835 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:15.834+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4b4000bc0 con 0x7fa4bc035260 2024-02-10T08:28:15.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:14.519348+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 v1:172.21.15.107:0/3748002064' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:16.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [80B blob data] 2024-02-10T08:28:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.446+0000 7fa4c6ffd700 1 -- 172.21.15.107:0/1145252253 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 1030021163) 0x7fa4b4000bc0 con 0x7fa4bc035260 2024-02-10T08:28:16.449 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Added host 'smithi107' with addr '172.21.15.107' 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 10 librados: watch_flush enter 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 10 librados: watch_flush exit 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 20 client.14134.objecter shutdown clearing up homeless session... 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 10 client.14134.objecter successfully canceled tick 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 >> v1:172.21.15.107:6800/2081774553 conn(0x7fa4bc035260 legacy=0x7fa4bc037720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 >> v1:172.21.15.107:6789/0 conn(0x7fa4d0070cc0 legacy=0x7fa4d010cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 shutdown_connections 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 >> 172.21.15.107:0/1145252253 conn(0x7fa4d00f65b0 msgr2=0x7fa4d00f71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 shutdown_connections 2024-02-10T08:28:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.448+0000 7fa4d803b700 1 -- 172.21.15.107:0/1145252253 wait complete. 2024-02-10T08:28:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:16.449+0000 7fa4d803b700 1 librados: shutdown 2024-02-10T08:28:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying unmanaged mon service... 2024-02-10T08:28:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.551+0000 7f9babcba700 1 Processor -- start 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.551+0000 7f9babcba700 1 -- start start 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9babcba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9ba4100250 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba9a56700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9ba4070ce0 0x7f9ba40feb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47552/0 (socket says 172.21.15.107:47552) 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba9a56700 1 -- 172.21.15.107:0/3862043446 learned_addr learned my addr 172.21.15.107:0/3862043446 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546391207 0 0) 0x7f9ba4100250 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b900033f0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187111983 0 0) 0x7f9b900033f0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba41013f0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9ba0001590 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f9ba0001d10 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.552+0000 7f9ba8a54700 1 -- 172.21.15.107:0/3862043446 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9ba0005750 con 0x7f9ba4070ce0 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.553+0000 7f9babcba700 1 -- 172.21.15.107:0/3862043446 >> v1:172.21.15.107:6789/0 conn(0x7f9ba4070ce0 legacy=0x7f9ba40feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.553+0000 7f9babcba700 1 -- 172.21.15.107:0/3862043446 shutdown_connections 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.553+0000 7f9babcba700 1 -- 172.21.15.107:0/3862043446 >> 172.21.15.107:0/3862043446 conn(0x7f9ba40f65b0 msgr2=0x7f9ba40f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.553+0000 7f9babcba700 1 -- 172.21.15.107:0/3862043446 shutdown_connections 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.553+0000 7f9babcba700 1 -- 172.21.15.107:0/3862043446 wait complete. 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 librados: starting msgr at 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 librados: starting objecter 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 Processor -- start 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 -- start start 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 librados: setting wanted keys 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 librados: calling monclient init 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9babcba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9ba410b060 con 0x7f9ba4070ce0 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9ba9a56700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9ba4070ce0 0x7f9ba410a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47560/0 (socket says 172.21.15.107:47560) 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9ba9a56700 1 -- 172.21.15.107:0/694979662 learned_addr learned my addr 172.21.15.107:0/694979662 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9b9affd700 10 client.?.objecter ms_handle_connect 0x7f9ba4070ce0 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9b9affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609497169 0 0) 0x7f9ba410b060 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.554+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b90005580 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603411299 0 0) 0x7f9b90005580 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ba410b060 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9ba0000fe0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1270066075 0 0) 0x7f9ba410b060 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba410b1f0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba410b4a0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9babcba700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba410f190 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9babcba700 1 librados: init done 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f9ba00046b0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.555+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9ba00061d0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7f9ba000f9f0 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b88004f50 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f9ba003ec20 con 0x7f9ba4070ce0 2024-02-10T08:28:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 10 client.14136.objecter ms_dispatch 0x7f9ba4001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:17.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:17.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:17.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 20 client.14136.objecter dump_active .. 0 homeless 2024-02-10T08:28:17.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.556+0000 7f9b9affd700 10 client.14136.objecter ms_handle_connect 0x7f9b90035260 2024-02-10T08:28:17.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.560+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9ba0010d90 con 0x7f9ba4070ce0 2024-02-10T08:28:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:15.836351+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 v1:172.21.15.107:0/1145252253' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:16.447600+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:16.447880+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi107 2024-02-10T08:28:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:16.450860+0000 mon.a (mon.0) 57 : audit [DBG] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.738+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9b88000bc0 con 0x7f9b90035260 2024-02-10T08:28:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.743+0000 7f9b9affd700 1 -- 172.21.15.107:0/694979662 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f9b88000bc0 con 0x7f9b90035260 2024-02-10T08:28:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T08:28:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 10 librados: watch_flush enter 2024-02-10T08:28:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 10 librados: watch_flush exit 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 20 client.14136.objecter shutdown clearing up homeless session... 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 10 client.14136.objecter successfully canceled tick 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b90035260 legacy=0x7f9b90037720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 >> v1:172.21.15.107:6789/0 conn(0x7f9ba4070ce0 legacy=0x7f9ba410a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 shutdown_connections 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 >> 172.21.15.107:0/694979662 conn(0x7f9ba40f65b0 msgr2=0x7f9ba40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 shutdown_connections 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 -- 172.21.15.107:0/694979662 wait complete. 2024-02-10T08:28:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:17.745+0000 7f9babcba700 1 librados: shutdown 2024-02-10T08:28:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:17.740948+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 v1:172.21.15.107:0/694979662' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:17.742133+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-10T08:28:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:17.745149+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:18.956 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying unmanaged mgr service... 2024-02-10T08:28:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.527+0000 7f14de6ba700 1 Processor -- start 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.527+0000 7f14de6ba700 1 -- start start 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.527+0000 7f14de6ba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14d80ff2a0 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.527+0000 7f14d7fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f14d80fc430 0x7f14d80fc850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35002/0 (socket says 172.21.15.107:35002) 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.527+0000 7f14d7fff700 1 -- 172.21.15.107:0/1342857362 learned_addr learned my addr 172.21.15.107:0/1342857362 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463455853 0 0) 0x7f14d80ff2a0 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c00033f0 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083970847 0 0) 0x7f14c00033f0 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8100440 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f14cc001590 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f14cc001d10 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14d6ffd700 1 -- 172.21.15.107:0/1342857362 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f14cc005750 con 0x7f14d80fc430 2024-02-10T08:28:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14de6ba700 1 -- 172.21.15.107:0/1342857362 >> v1:172.21.15.107:6789/0 conn(0x7f14d80fc430 legacy=0x7f14d80fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14de6ba700 1 -- 172.21.15.107:0/1342857362 shutdown_connections 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.528+0000 7f14de6ba700 1 -- 172.21.15.107:0/1342857362 >> 172.21.15.107:0/1342857362 conn(0x7f14d80f65b0 msgr2=0x7f14d80f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 -- 172.21.15.107:0/1342857362 shutdown_connections 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 -- 172.21.15.107:0/1342857362 wait complete. 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 librados: starting msgr at 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 librados: starting objecter 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 Processor -- start 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 -- start start 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 librados: setting wanted keys 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 librados: calling monclient init 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.529+0000 7f14de6ba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14d810d250 con 0x7f14d80fc430 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d7fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f14d80fc430 0x7f14d810cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35008/0 (socket says 172.21.15.107:35008) 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d7fff700 1 -- 172.21.15.107:0/1706669352 learned_addr learned my addr 172.21.15.107:0/1706669352 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 10 client.?.objecter ms_handle_connect 0x7f14d80fc430 2024-02-10T08:28:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25874699 0 0) 0x7f14d810d250 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0005580 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321399859 0 0) 0x7f14c0005580 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d810d250 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f14cc000fe0 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1118822367 0 0) 0x7f14d810d250 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d810d3e0 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d810d690 con 0x7f14d80fc430 2024-02-10T08:28:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14de6ba700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d8111380 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14de6ba700 1 librados: init done 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 886019014 0 0) 0x7f14cc0046b0 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.530+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f14cc006160 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7f14cc00fa40 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f14cc03eca0 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 10 client.14138.objecter ms_dispatch 0x7f14d8001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c4004f50 con 0x7f14d80fc430 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.531+0000 7f14d57fa700 20 client.14138.objecter dump_active .. 0 homeless 2024-02-10T08:28:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.532+0000 7f14d57fa700 10 client.14138.objecter ms_handle_connect 0x7f14c0035230 2024-02-10T08:28:19.538 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.536+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f14cc011170 con 0x7f14d80fc430 2024-02-10T08:28:19.711 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.709+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 --> v1:172.21.15.107:6800/2081774553 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f14c4000bc0 con 0x7f14c0035230 2024-02-10T08:28:19.715 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.713+0000 7f14d57fa700 1 -- 172.21.15.107:0/1706669352 <== mgr.14116 v1:172.21.15.107:6800/2081774553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f14c4000bc0 con 0x7f14c0035230 2024-02-10T08:28:19.715 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 10 librados: watch_flush enter 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 10 librados: watch_flush exit 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 20 client.14138.objecter shutdown clearing up homeless session... 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 10 client.14138.objecter successfully canceled tick 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 >> v1:172.21.15.107:6800/2081774553 conn(0x7f14c0035230 legacy=0x7f14c00376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 >> v1:172.21.15.107:6789/0 conn(0x7f14d80fc430 legacy=0x7f14d810cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 shutdown_connections 2024-02-10T08:28:19.718 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 >> 172.21.15.107:0/1706669352 conn(0x7f14d80f65b0 msgr2=0x7f14d806ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:19.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 shutdown_connections 2024-02-10T08:28:19.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 -- 172.21.15.107:0/1706669352 wait complete. 2024-02-10T08:28:19.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:19.715+0000 7f14de6ba700 1 librados: shutdown 2024-02-10T08:28:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.900+0000 7f850d205700 1 Processor -- start 2024-02-10T08:28:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.900+0000 7f850d205700 1 -- start start 2024-02-10T08:28:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f850d205700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f85080fc950 con 0x7f8508070ca0 2024-02-10T08:28:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8506d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8508070ca0 0x7f8508100f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35024/0 (socket says 172.21.15.107:35024) 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8506d9d700 1 -- 172.21.15.107:0/2179554097 learned_addr learned my addr 172.21.15.107:0/2179554097 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614532975 0 0) 0x7f85080fc950 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f00033f0 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1760865738 0 0) 0x7f84f00033f0 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508103690 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.901+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f84f8001590 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f84f8001d20 con 0x7f8508070ca0 2024-02-10T08:28:20.904 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f8505d9b700 1 -- 172.21.15.107:0/2179554097 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f84f8005710 con 0x7f8508070ca0 2024-02-10T08:28:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f850d205700 1 -- 172.21.15.107:0/2179554097 >> v1:172.21.15.107:6789/0 conn(0x7f8508070ca0 legacy=0x7f8508100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f850d205700 1 -- 172.21.15.107:0/2179554097 shutdown_connections 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f850d205700 1 -- 172.21.15.107:0/2179554097 >> 172.21.15.107:0/2179554097 conn(0x7f85080f6520 msgr2=0x7f85080f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f850d205700 1 -- 172.21.15.107:0/2179554097 shutdown_connections 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.902+0000 7f850d205700 1 -- 172.21.15.107:0/2179554097 wait complete. 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 librados: starting msgr at 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 librados: starting objecter 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 Processor -- start 2024-02-10T08:28:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 -- start start 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 librados: setting wanted keys 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.903+0000 7f850d205700 1 librados: calling monclient init 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f850d205700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f850810d210 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f8506d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8508070ca0 0x7f850810cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35036/0 (socket says 172.21.15.107:35036) 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f8506d9d700 1 -- 172.21.15.107:0/1062113396 learned_addr learned my addr 172.21.15.107:0/1062113396 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 10 client.?.objecter ms_handle_connect 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674449472 0 0) 0x7f850810d210 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f0005580 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3507927671 0 0) 0x7f84f0005580 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f850810d210 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f84f8004800 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.904+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3700345784 0 0) 0x7f850810d210 con 0x7f8508070ca0 2024-02-10T08:28:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850810d3a0 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850810d650 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f850d205700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8508111340 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f850d205700 1 librados: init done 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f84f80011e0 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f84f8006230 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.905+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7f84f800fa50 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f84f803fcc0 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 10 client.14140.objecter ms_dispatch 0x7f8508001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f4004f50 con 0x7f8508070ca0 2024-02-10T08:28:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 20 client.14140.objecter dump_active .. 0 homeless 2024-02-10T08:28:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.906+0000 7f84effff700 10 client.14140.objecter ms_handle_connect 0x7f84f00352a0 2024-02-10T08:28:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:20.910+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f84f8011170 con 0x7f8508070ca0 2024-02-10T08:28:21.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.079+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f84f4005200 con 0x7f8508070ca0 2024-02-10T08:28:21.084 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.082+0000 7f84effff700 1 -- 172.21.15.107:0/1062113396 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7f84f8013250 con 0x7f8508070ca0 2024-02-10T08:28:21.086 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 10 librados: watch_flush enter 2024-02-10T08:28:21.086 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 10 librados: watch_flush exit 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 20 client.14140.objecter shutdown clearing up homeless session... 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 10 client.14140.objecter successfully canceled tick 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 >> v1:172.21.15.107:6800/2081774553 conn(0x7f84f00352a0 legacy=0x7f84f0037760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.084+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 >> v1:172.21.15.107:6789/0 conn(0x7f8508070ca0 legacy=0x7f850810cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.085+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 shutdown_connections 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.085+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 >> 172.21.15.107:0/1062113396 conn(0x7f85080f6520 msgr2=0x7f85080f71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.085+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 shutdown_connections 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.085+0000 7f850d205700 1 -- 172.21.15.107:0/1062113396 wait complete. 2024-02-10T08:28:21.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:21.085+0000 7f850d205700 1 librados: shutdown 2024-02-10T08:28:21.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:19.711636+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 v1:172.21.15.107:0/1706669352' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:21.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:19.712453+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-10T08:28:21.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:19.714708+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:21.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:20.714504+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:21.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:20.954505+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:21.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:21.084309+0000 mon.a (mon.0) 62 : audit [INF] from='client.? v1:172.21.15.107:0/1062113396' entity='client.admin' 2024-02-10T08:28:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.483+0000 7f66f151b700 1 Processor -- start 2024-02-10T08:28:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.483+0000 7f66f151b700 1 -- start start 2024-02-10T08:28:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66f151b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66ec0fcf50 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66eaffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f66ec0fc420 0x7f66ec0fc840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35040/0 (socket says 172.21.15.107:35040) 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66eaffd700 1 -- 172.21.15.107:0/923484501 learned_addr learned my addr 172.21.15.107:0/923484501 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842979860 0 0) 0x7f66ec0fcf50 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d40033f0 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018170361 0 0) 0x7f66d40033f0 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec100340 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f66dc001590 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f66dc001d20 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.484+0000 7f66e9ffb700 1 -- 172.21.15.107:0/923484501 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f66dc005780 con 0x7f66ec0fc420 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.485+0000 7f66f151b700 1 -- 172.21.15.107:0/923484501 >> v1:172.21.15.107:6789/0 conn(0x7f66ec0fc420 legacy=0x7f66ec0fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.485+0000 7f66f151b700 1 -- 172.21.15.107:0/923484501 shutdown_connections 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.485+0000 7f66f151b700 1 -- 172.21.15.107:0/923484501 >> 172.21.15.107:0/923484501 conn(0x7f66ec0f5cc0 msgr2=0x7f66ec0f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.485+0000 7f66f151b700 1 -- 172.21.15.107:0/923484501 shutdown_connections 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.485+0000 7f66f151b700 1 -- 172.21.15.107:0/923484501 wait complete. 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 librados: starting msgr at 2024-02-10T08:28:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 librados: starting objecter 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 Processor -- start 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 -- start start 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 librados: setting wanted keys 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 librados: calling monclient init 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66f151b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66ec109f30 con 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66eaffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f66ec0fc420 0x7f66ec109820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35046/0 (socket says 172.21.15.107:35046) 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66eaffd700 1 -- 172.21.15.107:0/2356552560 learned_addr learned my addr 172.21.15.107:0/2356552560 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66d3fff700 10 client.?.objecter ms_handle_connect 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.486+0000 7f66d3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330055338 0 0) 0x7f66ec109f30 con 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d4005580 con 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728798117 0 0) 0x7f66d4005580 con 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66ec109f30 con 0x7f66ec0fc420 2024-02-10T08:28:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f66dc000fe0 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2380995755 0 0) 0x7f66ec109f30 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec10a0c0 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec10a370 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66f151b700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec10e060 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.487+0000 7f66f151b700 1 librados: init done 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f66dc001430 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f66dc006220 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7f66dc00fb00 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f66dc03fc00 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 10 client.14142.objecter ms_dispatch 0x7f66ec001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec05a3b0 con 0x7f66ec0fc420 2024-02-10T08:28:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.488+0000 7f66d3fff700 20 client.14142.objecter dump_active .. 0 homeless 2024-02-10T08:28:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.489+0000 7f66d3fff700 10 client.14142.objecter ms_handle_connect 0x7f66d40352f0 2024-02-10T08:28:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.493+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f66dc014d80 con 0x7f66ec0fc420 2024-02-10T08:28:22.657 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.655+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f66ec046d80 con 0x7f66ec0fc420 2024-02-10T08:28:22.659 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.658+0000 7f66d3fff700 1 -- 172.21.15.107:0/2356552560 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7f66dc0131f0 con 0x7f66ec0fc420 2024-02-10T08:28:22.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 10 librados: watch_flush enter 2024-02-10T08:28:22.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 10 librados: watch_flush exit 2024-02-10T08:28:22.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 20 client.14142.objecter shutdown clearing up homeless session... 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 10 client.14142.objecter successfully canceled tick 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 >> v1:172.21.15.107:6800/2081774553 conn(0x7f66d40352f0 legacy=0x7f66d40377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 >> v1:172.21.15.107:6789/0 conn(0x7f66ec0fc420 legacy=0x7f66ec109820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 shutdown_connections 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 >> 172.21.15.107:0/2356552560 conn(0x7f66ec0f5cc0 msgr2=0x7f66ec0f6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 shutdown_connections 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 -- 172.21.15.107:0/2356552560 wait complete. 2024-02-10T08:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:22.660+0000 7f66f151b700 1 librados: shutdown 2024-02-10T08:28:23.430 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling the dashboard module... 2024-02-10T08:28:23.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:22.660050+0000 mon.a (mon.0) 63 : audit [INF] from='client.? v1:172.21.15.107:0/2356552560' entity='client.admin' 2024-02-10T08:28:23.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:22.908077+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14116 v1:172.21.15.107:0/795761238' entity='mgr.y' 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.073+0000 7f594d9ca700 1 Processor -- start 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.073+0000 7f594d9ca700 1 -- start start 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.073+0000 7f594d9ca700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5948100250 con 0x7f5948070ce0 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.073+0000 7f5946ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5948070ce0 0x7f59480feb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35054/0 (socket says 172.21.15.107:35054) 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.073+0000 7f5946ffd700 1 -- 172.21.15.107:0/1946329270 learned_addr learned my addr 172.21.15.107:0/1946329270 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008799998 0 0) 0x7f5948100250 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59300033f0 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277290904 0 0) 0x7f59300033f0 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59481013f0 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f5938001590 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5938001d20 con 0x7f5948070ce0 2024-02-10T08:28:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.074+0000 7f5945ffb700 1 -- 172.21.15.107:0/1946329270 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f5938005780 con 0x7f5948070ce0 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.075+0000 7f594d9ca700 1 -- 172.21.15.107:0/1946329270 >> v1:172.21.15.107:6789/0 conn(0x7f5948070ce0 legacy=0x7f59480feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.075+0000 7f594d9ca700 1 -- 172.21.15.107:0/1946329270 shutdown_connections 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.075+0000 7f594d9ca700 1 -- 172.21.15.107:0/1946329270 >> 172.21.15.107:0/1946329270 conn(0x7f59480f65b0 msgr2=0x7f59480f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.075+0000 7f594d9ca700 1 -- 172.21.15.107:0/1946329270 shutdown_connections 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.075+0000 7f594d9ca700 1 -- 172.21.15.107:0/1946329270 wait complete. 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 librados: starting msgr at 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 librados: starting objecter 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 Processor -- start 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 -- start start 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 librados: setting wanted keys 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 librados: calling monclient init 2024-02-10T08:28:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.076+0000 7f594d9ca700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59480fc540 con 0x7f5948070ce0 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f5946ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5948070ce0 0x7f59480faa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35056/0 (socket says 172.21.15.107:35056) 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f5946ffd700 1 -- 172.21.15.107:0/2862820404 learned_addr learned my addr 172.21.15.107:0/2862820404 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 10 client.?.objecter ms_handle_connect 0x7f5948070ce0 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486040972 0 0) 0x7f59480fc540 con 0x7f5948070ce0 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5930005580 con 0x7f5948070ce0 2024-02-10T08:28:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413989733 0 0) 0x7f5930005580 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59480fc540 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.077+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f5938000fe0 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377471237 0 0) 0x7f59480fc540 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59480fc6d0 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59480fc980 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f594d9ca700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59480fb440 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f594d9ca700 1 librados: init done 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f59380049e0 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.078+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f59380061c0 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 1498230328 0 0) 0x7f593800f9e0 con 0x7f5948070ce0 2024-02-10T08:28:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f593803fa40 con 0x7f5948070ce0 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 10 client.14144.objecter ms_dispatch 0x7f5948001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 20 client.14144.objecter dump_active .. 0 homeless 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5934004f50 con 0x7f5948070ce0 2024-02-10T08:28:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.079+0000 7f592ffff700 10 client.14144.objecter ms_handle_connect 0x7f5930035230 2024-02-10T08:28:24.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.084+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5938011140 con 0x7f5948070ce0 2024-02-10T08:28:24.293 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.292+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5934005200 con 0x7f5948070ce0 2024-02-10T08:28:24.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.661+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3866029869 0 0) 0x7f593803efb0 con 0x7f5948070ce0 2024-02-10T08:28:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.663+0000 7f592ffff700 1 -- 172.21.15.107:0/2862820404 <== mon.0 v1:172.21.15.107:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7f5938014df0 con 0x7f5948070ce0 2024-02-10T08:28:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 10 librados: watch_flush enter 2024-02-10T08:28:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 10 librados: watch_flush exit 2024-02-10T08:28:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 20 client.14144.objecter shutdown clearing up homeless session... 2024-02-10T08:28:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 10 client.14144.objecter successfully canceled tick 2024-02-10T08:28:24.667 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 >> v1:172.21.15.107:6800/2081774553 conn(0x7f5930035230 legacy=0x7f59300376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 >> v1:172.21.15.107:6789/0 conn(0x7f5948070ce0 legacy=0x7f59480faa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 shutdown_connections 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 >> 172.21.15.107:0/2862820404 conn(0x7f59480f65b0 msgr2=0x7f59480fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 shutdown_connections 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 -- 172.21.15.107:0/2862820404 wait complete. 2024-02-10T08:28:24.668 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:24.665+0000 7f594d9ca700 1 librados: shutdown 2024-02-10T08:28:24.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:24.293987+0000 mon.a (mon.0) 65 : audit [INF] from='client.? v1:172.21.15.107:0/2862820404' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T08:28:24.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setuser ceph since I am not root 2024-02-10T08:28:24.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setgroup ceph since I am not root 2024-02-10T08:28:24.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:24.824+0000 7fcf79557680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:28:24.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:24.904+0000 7fcf79557680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:28:25.607 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:25.269+0000 7fcf79557680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:28:25.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:24.664860+0000 mon.a (mon.0) 66 : audit [INF] from='client.? v1:172.21.15.107:0/2862820404' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T08:28:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:24.664902+0000 mon.a (mon.0) 67 : cluster [DBG] mgrmap e9: y(active, since 17s) 2024-02-10T08:28:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.971+0000 7fd68f292700 1 Processor -- start 2024-02-10T08:28:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.971+0000 7fd68f292700 1 -- start start 2024-02-10T08:28:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd68f292700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6880fc950 con 0x7fd688070ca0 2024-02-10T08:28:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd68d02e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd688070ca0 0x7fd688100f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35080/0 (socket says 172.21.15.107:35080) 2024-02-10T08:28:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd68d02e700 1 -- 172.21.15.107:0/351064986 learned_addr learned my addr 172.21.15.107:0/351064986 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981617492 0 0) 0x7fd6880fc950 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6740033f0 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2676804264 0 0) 0x7fd6740033f0 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688103690 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.972+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd684001590 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd684001d20 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd67ffff700 1 -- 172.21.15.107:0/351064986 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd684005710 con 0x7fd688070ca0 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd68f292700 1 -- 172.21.15.107:0/351064986 >> v1:172.21.15.107:6789/0 conn(0x7fd688070ca0 legacy=0x7fd688100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd68f292700 1 -- 172.21.15.107:0/351064986 shutdown_connections 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd68f292700 1 -- 172.21.15.107:0/351064986 >> 172.21.15.107:0/351064986 conn(0x7fd6880f6520 msgr2=0x7fd6880f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd68f292700 1 -- 172.21.15.107:0/351064986 shutdown_connections 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.973+0000 7fd68f292700 1 -- 172.21.15.107:0/351064986 wait complete. 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 librados: starting msgr at 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 librados: starting objecter 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 Processor -- start 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 -- start start 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 librados: setting wanted keys 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 librados: calling monclient init 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68f292700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd68810d1f0 con 0x7fd688070ca0 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68d02e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd688070ca0 0x7fd68810cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35094/0 (socket says 172.21.15.107:35094) 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd68d02e700 1 -- 172.21.15.107:0/1124948577 learned_addr learned my addr 172.21.15.107:0/1124948577 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 10 client.?.objecter ms_handle_connect 0x7fd688070ca0 2024-02-10T08:28:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928952433 0 0) 0x7fd68810d1f0 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd674005580 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1878255335 0 0) 0x7fd674005580 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd68810d1f0 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.974+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd684004800 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3247372628 0 0) 0x7fd68810d1f0 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68810d380 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68810d630 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd68f292700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd688111320 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd68f292700 1 librados: init done 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd684001100 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.975+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd684006120 con 0x7fd688070ca0 2024-02-10T08:28:25.980 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3866029869 0 0) 0x7fd68400f950 con 0x7fd688070ca0 2024-02-10T08:28:25.981 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c004f50 con 0x7fd688070ca0 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7fd68403ed90 con 0x7fd688070ca0 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 10 client.14148.objecter ms_dispatch 0x7fd688001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd67e777700 20 client.14148.objecter dump_active .. 0 homeless 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.976+0000 7fd68c82d700 1 -- 172.21.15.107:0/1124948577 >> v1:172.21.15.107:6800/2081774553 conn(0x7fd6740353a0 legacy=0x7fd674037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:25.984+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd6840110c0 con 0x7fd688070ca0 2024-02-10T08:28:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.177+0000 7fd68c82d700 1 -- 172.21.15.107:0/1124948577 >> v1:172.21.15.107:6800/2081774553 conn(0x7fd6740353a0 legacy=0x7fd674037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.227+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd66c005a10 con 0x7fd688070ca0 2024-02-10T08:28:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.227+0000 7fd67e777700 1 -- 172.21.15.107:0/1124948577 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7fd6840131a0 con 0x7fd688070ca0 2024-02-10T08:28:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:28:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-10T08:28:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:28:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-10T08:28:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T08:28:26.229 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:28:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 10 librados: watch_flush enter 2024-02-10T08:28:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 10 librados: watch_flush exit 2024-02-10T08:28:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 20 client.14148.objecter shutdown clearing up homeless session... 2024-02-10T08:28:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 10 client.14148.objecter successfully canceled tick 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 >> v1:172.21.15.107:6800/2081774553 conn(0x7fd6740353a0 legacy=0x7fd674037860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 >> v1:172.21.15.107:6789/0 conn(0x7fd688070ca0 legacy=0x7fd68810cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 shutdown_connections 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 >> 172.21.15.107:0/1124948577 conn(0x7fd6880f6520 msgr2=0x7fd6880f71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 shutdown_connections 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 -- 172.21.15.107:0/1124948577 wait complete. 2024-02-10T08:28:26.231 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:26.229+0000 7fd68f292700 1 librados: shutdown 2024-02-10T08:28:26.299 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:26.036+0000 7fcf79557680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:28:26.299 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:26.121+0000 7fcf79557680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:28:26.580 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:26.340+0000 7fcf79557680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:28:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for the mgr to restart... 2024-02-10T08:28:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr epoch 9... 2024-02-10T08:28:26.885 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:26.228136+0000 mon.a (mon.0) 68 : audit [DBG] from='client.? v1:172.21.15.107:0/1124948577' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T08:28:27.420 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:27.287+0000 7fcf79557680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:28:27.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b57ab1700 1 Processor -- start 2024-02-10T08:28:27.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b57ab1700 1 -- start start 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b57ab1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9b50100240 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b5584d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9b50070ce0 0x7f9b500feb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35096/0 (socket says 172.21.15.107:35096) 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b5584d700 1 -- 172.21.15.107:0/1085043541 learned_addr learned my addr 172.21.15.107:0/1085043541 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.493+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280773184 0 0) 0x7f9b50100240 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c0033f0 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1169141693 0 0) 0x7f9b3c0033f0 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b501013e0 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9b4c001590 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9b4c001d20 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b5484b700 1 -- 172.21.15.107:0/1085043541 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9b4c005780 con 0x7f9b50070ce0 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.494+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1085043541 >> v1:172.21.15.107:6789/0 conn(0x7f9b50070ce0 legacy=0x7f9b500feb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1085043541 shutdown_connections 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1085043541 >> 172.21.15.107:0/1085043541 conn(0x7f9b500f6520 msgr2=0x7f9b500f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:27.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1085043541 shutdown_connections 2024-02-10T08:28:27.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1085043541 wait complete. 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 librados: starting msgr at 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 librados: starting objecter 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 Processor -- start 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- start start 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 librados: setting wanted keys 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 librados: calling monclient init 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.495+0000 7f9b57ab1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9b5010b040 con 0x7f9b50070ce0 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b5584d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9b50070ce0 0x7f9b5010a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35102/0 (socket says 172.21.15.107:35102) 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b5584d700 1 -- 172.21.15.107:0/1893509565 learned_addr learned my addr 172.21.15.107:0/1893509565 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 10 client.?.objecter ms_handle_connect 0x7f9b50070ce0 2024-02-10T08:28:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993452374 0 0) 0x7f9b5010b040 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c005580 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780247681 0 0) 0x7f9b3c005580 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b5010b040 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9b4c000fe0 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1917147515 0 0) 0x7f9b5010b040 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5010b1d0 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5010b480 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b57ab1700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5010f170 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.496+0000 7f9b57ab1700 1 librados: init done 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9b4c004930 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f9b4c006110 con 0x7f9b50070ce0 2024-02-10T08:28:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3866029869 0 0) 0x7f9b4c006270 con 0x7f9b50070ce0 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6800/2081774553 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b3c039070 con 0x7f9b3c0353a0 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b5504c700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b3c0353a0 legacy=0x7f9b3c037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 4240252831 0 0) 0x7f9b4c03fb70 con 0x7f9b50070ce0 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 10 client.14150.objecter ms_dispatch 0x7f9b50001550 osd_map(2..2 src has 1..2) v4 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.497+0000 7f9b46f78700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-02-10T08:28:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.498+0000 7f9b46f78700 20 client.14150.objecter dump_active .. 0 homeless 2024-02-10T08:28:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:27.419+0000 7fcf79557680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:28:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:27.502+0000 7fcf79557680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:28:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:27.592+0000 7fcf79557680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:28:27.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:27.698+0000 7f9b5504c700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b3c0353a0 legacy=0x7f9b3c037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:28.059 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:27.785+0000 7fcf79557680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:28:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:28.099+0000 7f9b5504c700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b3c0353a0 legacy=0x7f9b3c037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:28.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:28.058+0000 7fcf79557680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:28:28.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:28.156+0000 7fcf79557680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:28:28.900 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:28.900+0000 7f9b5504c700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b3c0353a0 legacy=0x7f9b3c037860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/2081774553 2024-02-10T08:28:29.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:28.867+0000 7fcf79557680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:28:29.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:28.947+0000 7fcf79557680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:28:29.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.039+0000 7fcf79557680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:28:29.465 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.211+0000 7fcf79557680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:28:29.728 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.464+0000 7fcf79557680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:28:29.728 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.592+0000 7fcf79557680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:28:29.979 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.727+0000 7fcf79557680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:28:29.979 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.896+0000 7fcf79557680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:28:29.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:29.993+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mgrmap(e 10) v1 ==== 38454+0+0 (unknown 2027814394 0 0) 0x7f9b4c005a60 con 0x7f9b50070ce0 2024-02-10T08:28:29.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:29.994+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/2081774553 conn(0x7f9b3c0353a0 legacy=0x7f9b3c037860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:28:30.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:28:29.979+0000 7fcf79557680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:28:30.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [80B blob data] 2024-02-10T08:28:30.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:29.992650+0000 mon.a (mon.0) 69 : cluster [INF] Active manager daemon y restarted 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:29.992873+0000 mon.a (mon.0) 70 : cluster [INF] Activating manager daemon y 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:29.995699+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:29.995812+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00302821s) 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:29.996720+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:29.997277+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:29.997868+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:29.998027+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:29.998145+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:30.012000+0000 mon.a (mon.0) 78 : cluster [INF] Manager daemon y is now available 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:30.102893+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:30.104093+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:30.129303+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:28:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:30.131917+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:28:30.996 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:30.995+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mgrmap(e 11) v1 ==== 38489+0+0 (unknown 1818118967 0 0) 0x7f9b4c007660 con 0x7f9b50070ce0 2024-02-10T08:28:30.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:30.995+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6800/3360068681 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b3c039070 con 0x7f9b4c03f770 2024-02-10T08:28:30.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:30.995+0000 7f9b46f78700 10 client.14150.objecter ms_handle_connect 0x7f9b4c03f770 2024-02-10T08:28:30.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:30.996+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (unknown 0 0 4198994610) 0x7f9b3c039070 con 0x7f9b4c03f770 2024-02-10T08:28:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.002+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 --> v1:172.21.15.107:6800/3360068681 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9b34002440 con 0x7f9b4c03f770 2024-02-10T08:28:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.002+0000 7f9b46f78700 1 -- 172.21.15.107:0/1893509565 <== mgr.14146 v1:172.21.15.107:6800/3360068681 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7f9b34002440 con 0x7f9b4c03f770 2024-02-10T08:28:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-02-10T08:28:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-10T08:28:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T08:28:31.004 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 10 librados: watch_flush enter 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 10 librados: watch_flush exit 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 20 client.14150.objecter shutdown clearing up homeless session... 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 10 client.14150.objecter successfully canceled tick 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6800/3360068681 conn(0x7f9b4c03f770 legacy=0x7f9b3c03c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 >> v1:172.21.15.107:6789/0 conn(0x7f9b50070ce0 legacy=0x7f9b5010a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 shutdown_connections 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 >> 172.21.15.107:0/1893509565 conn(0x7f9b500f6520 msgr2=0x7f9b500fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 shutdown_connections 2024-02-10T08:28:31.005 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 -- 172.21.15.107:0/1893509565 wait complete. 2024-02-10T08:28:31.006 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:31.003+0000 7f9b57ab1700 1 librados: shutdown 2024-02-10T08:28:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:mgr epoch 9 is available 2024-02-10T08:28:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:Generating a dashboard self-signed certificate... 2024-02-10T08:28:32.252 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:30.996805+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 v1:172.21.15.107:0/1893509565' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T08:28:32.252 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:30.997922+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e11: y(active, since 1.00514s) 2024-02-10T08:28:32.252 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:31.003414+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 v1:172.21.15.107:0/1893509565' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T08:28:32.252 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:31.404507+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.672+0000 7f1cab83f700 1 Processor -- start 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.672+0000 7f1cab83f700 1 -- start start 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.672+0000 7f1cab83f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1ca4100260 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.672+0000 7f1ca95db700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ca4070cc0 0x7f1ca40feb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58032/0 (socket says 172.21.15.107:58032) 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.672+0000 7f1ca95db700 1 -- 172.21.15.107:0/2747103695 learned_addr learned my addr 172.21.15.107:0/2747103695 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034305897 0 0) 0x7f1ca4100260 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c900033f0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679083072 0 0) 0x7f1c900033f0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4101400 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f1ca0001590 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1ca0001d20 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1c9bfff700 1 -- 172.21.15.107:0/2747103695 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f1ca0005780 con 0x7f1ca4070cc0 2024-02-10T08:28:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1cab83f700 1 -- 172.21.15.107:0/2747103695 >> v1:172.21.15.107:6789/0 conn(0x7f1ca4070cc0 legacy=0x7f1ca40feb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1cab83f700 1 -- 172.21.15.107:0/2747103695 shutdown_connections 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1cab83f700 1 -- 172.21.15.107:0/2747103695 >> 172.21.15.107:0/2747103695 conn(0x7f1ca40f6520 msgr2=0x7f1ca40f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1cab83f700 1 -- 172.21.15.107:0/2747103695 shutdown_connections 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.673+0000 7f1cab83f700 1 -- 172.21.15.107:0/2747103695 wait complete. 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 librados: starting msgr at 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 librados: starting objecter 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 Processor -- start 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 -- start start 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 librados: setting wanted keys 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 librados: calling monclient init 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1cab83f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1ca410b050 con 0x7f1ca4070cc0 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1ca95db700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ca4070cc0 0x7f1ca410a940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58042/0 (socket says 172.21.15.107:58042) 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1ca95db700 1 -- 172.21.15.107:0/492979479 learned_addr learned my addr 172.21.15.107:0/492979479 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1c9a7fc700 10 client.?.objecter ms_handle_connect 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1c9a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179278135 0 0) 0x7f1ca410b050 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.674+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90005580 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197345763 0 0) 0x7f1c90005580 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ca410b050 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f1ca0000fe0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1050425838 0 0) 0x7f1ca410b050 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca410b1e0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca410b490 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1cab83f700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca410f180 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1cab83f700 1 librados: init done 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1ca0001430 con 0x7f1ca4070cc0 2024-02-10T08:28:32.699 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.675+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f1ca00062a0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 11) v1 ==== 38489+0+0 (unknown 1818118967 0 0) 0x7f1ca000fa70 con 0x7f1ca4070cc0 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f1ca003ebd0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 10 client.14156.objecter ms_dispatch 0x7f1ca4001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 3 client.14156.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 3 client.14156.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88004f50 con 0x7f1ca4070cc0 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 20 client.14156.objecter dump_active .. 0 homeless 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.676+0000 7f1c9a7fc700 10 client.14156.objecter ms_handle_connect 0x7f1c90035160 2024-02-10T08:28:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.680+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1ca0014db0 con 0x7f1ca4070cc0 2024-02-10T08:28:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.856+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c88000bc0 con 0x7f1c90035160 2024-02-10T08:28:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.969+0000 7f1c9a7fc700 1 -- 172.21.15.107:0/492979479 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7f1c88000bc0 con 0x7f1c90035160 2024-02-10T08:28:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 10 librados: watch_flush enter 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 10 librados: watch_flush exit 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 20 client.14156.objecter shutdown clearing up homeless session... 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 10 client.14156.objecter successfully canceled tick 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 >> v1:172.21.15.107:6800/3360068681 conn(0x7f1c90035160 legacy=0x7f1c90037620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 >> v1:172.21.15.107:6789/0 conn(0x7f1ca4070cc0 legacy=0x7f1ca410a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 shutdown_connections 2024-02-10T08:28:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 >> 172.21.15.107:0/492979479 conn(0x7f1ca40f6520 msgr2=0x7f1ca40fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 shutdown_connections 2024-02-10T08:28:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 -- 172.21.15.107:0/492979479 wait complete. 2024-02-10T08:28:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:32.971+0000 7f1cab83f700 1 librados: shutdown 2024-02-10T08:28:33.605 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial admin user... 2024-02-10T08:28:34.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:32.857630+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 v1:172.21.15.107:0/492979479' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:34.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:32.964279+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:34.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:32.969640+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:34.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:33.001012+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e12: y(active, since 3s) 2024-02-10T08:28:34.456 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.414+0000 7f8208fba700 1 Processor -- start 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.414+0000 7f8208fba700 1 -- start start 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.414+0000 7f8208fba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82040ff290 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820259c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82040fc420 0x7f82040fc840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58054/0 (socket says 172.21.15.107:58054) 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820259c700 1 -- 172.21.15.107:0/1652068667 learned_addr learned my addr 172.21.15.107:0/1652068667 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547769828 0 0) 0x7f82040ff290 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f00033f0 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3416614495 0 0) 0x7f81f00033f0 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204100430 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f81fc001910 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f81fc0020a0 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.415+0000 7f820159a700 1 -- 172.21.15.107:0/1652068667 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f81fc005b00 con 0x7f82040fc420 2024-02-10T08:28:34.457 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 -- 172.21.15.107:0/1652068667 >> v1:172.21.15.107:6789/0 conn(0x7f82040fc420 legacy=0x7f82040fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 -- 172.21.15.107:0/1652068667 shutdown_connections 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 -- 172.21.15.107:0/1652068667 >> 172.21.15.107:0/1652068667 conn(0x7f82040f6520 msgr2=0x7f82040f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 -- 172.21.15.107:0/1652068667 shutdown_connections 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 -- 172.21.15.107:0/1652068667 wait complete. 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 librados: starting msgr at 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 librados: starting objecter 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.416+0000 7f8208fba700 1 Processor -- start 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 -- start start 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 librados: setting wanted keys 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 librados: calling monclient init 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f820410b4a0 con 0x7f82040fc420 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f820259c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f82040fc420 0x7f820410fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58064/0 (socket says 172.21.15.107:58064) 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f820259c700 1 -- 172.21.15.107:0/1848555733 learned_addr learned my addr 172.21.15.107:0/1848555733 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 10 client.?.objecter ms_handle_connect 0x7f82040fc420 2024-02-10T08:28:34.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512796798 0 0) 0x7f820410b4a0 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0005580 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096916739 0 0) 0x7f81f0005580 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f820410b4a0 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f81fc004bf0 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2615810802 0 0) 0x7f820410b4a0 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204070c00 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8204070eb0 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 10 client.14158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8204111410 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f8208fba700 1 librados: init done 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f81fc004f30 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.417+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f81fc006550 con 0x7f82040fc420 2024-02-10T08:28:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 12) v1 ==== 38595+0+0 (unknown 4198815368 0 0) 0x7f81fc00fd80 con 0x7f82040fc420 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f81fc040000 con 0x7f82040fc420 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 10 client.14158.objecter ms_dispatch 0x7f8204001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 3 client.14158.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 3 client.14158.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.418+0000 7f81fb7fe700 20 client.14158.objecter dump_active .. 0 homeless 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.419+0000 7f81fb7fe700 10 client.14158.objecter ms_handle_connect 0x7f81f00352d0 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.419+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e4004f50 con 0x7f82040fc420 2024-02-10T08:28:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.423+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f81fc0114f0 con 0x7f82040fc420 2024-02-10T08:28:34.603 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.603+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f81e4002430 con 0x7f81f00352d0 2024-02-10T08:28:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.822+0000 7f81fb7fe700 1 -- 172.21.15.107:0/1848555733 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 1774777277) 0x7f81e4002430 con 0x7f81f00352d0 2024-02-10T08:28:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$6arbSbU8CsUM56KLFXoyU.pTBU/tb2ec7IlzezTHyAHt0vdjSXjwi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707553714, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T08:28:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.823+0000 7f8208fba700 10 librados: watch_flush enter 2024-02-10T08:28:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.823+0000 7f8208fba700 10 librados: watch_flush exit 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.823+0000 7f8208fba700 20 client.14158.objecter shutdown clearing up homeless session... 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.823+0000 7f8208fba700 10 client.14158.objecter successfully canceled tick 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 >> v1:172.21.15.107:6800/3360068681 conn(0x7f81f00352d0 legacy=0x7f81f0037790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 >> v1:172.21.15.107:6789/0 conn(0x7f82040fc420 legacy=0x7f820410fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 shutdown_connections 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 >> 172.21.15.107:0/1848555733 conn(0x7f82040f6520 msgr2=0x7f82040f71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 shutdown_connections 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 -- 172.21.15.107:0/1848555733 wait complete. 2024-02-10T08:28:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:34.824+0000 7f8208fba700 1 librados: shutdown 2024-02-10T08:28:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:Fetching dashboard port number... 2024-02-10T08:28:35.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:34.604304+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 v1:172.21.15.107:0/1848555733' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:28:35.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:34.822487+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.083+0000 7fa3b03d3700 1 Processor -- start 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.083+0000 7fa3b03d3700 1 -- start start 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.083+0000 7fa3b03d3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3a8100270 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ae16f700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3a8070c70 0x7fa3a80feac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58066/0 (socket says 172.21.15.107:58066) 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ae16f700 1 -- 172.21.15.107:0/1547941643 learned_addr learned my addr 172.21.15.107:0/1547941643 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505021978 0 0) 0x7fa3a8100270 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3940033f0 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2248239743 0 0) 0x7fa3940033f0 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8101410 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa398001590 con 0x7fa3a8070c70 2024-02-10T08:28:36.087 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa398001c50 con 0x7fa3a8070c70 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.084+0000 7fa3ad16d700 1 -- 172.21.15.107:0/1547941643 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa3980056b0 con 0x7fa3a8070c70 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- 172.21.15.107:0/1547941643 >> v1:172.21.15.107:6789/0 conn(0x7fa3a8070c70 legacy=0x7fa3a80feac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- 172.21.15.107:0/1547941643 shutdown_connections 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- 172.21.15.107:0/1547941643 >> 172.21.15.107:0/1547941643 conn(0x7fa3a80f64c0 msgr2=0x7fa3a80f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- 172.21.15.107:0/1547941643 shutdown_connections 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- 172.21.15.107:0/1547941643 wait complete. 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 librados: starting msgr at 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 librados: starting objecter 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 Processor -- start 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 -- start start 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 librados: setting wanted keys 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.085+0000 7fa3b03d3700 1 librados: calling monclient init 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3b03d3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3a80fc4d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3ae16f700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3a8070c70 0x7fa3a80fa9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58082/0 (socket says 172.21.15.107:58082) 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3ae16f700 1 -- 172.21.15.107:0/2930224630 learned_addr learned my addr 172.21.15.107:0/2930224630 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 10 client.?.objecter ms_handle_connect 0x7fa3a8070c70 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262710043 0 0) 0x7fa3a80fc4d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3940055d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610111305 0 0) 0x7fa3940055d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3a80fc4d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa3980047d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 777037647 0 0) 0x7fa3a80fc4d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80fc660 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a80fc910 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3b03d3700 10 client.14160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a80fb3d0 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.086+0000 7fa3b03d3700 1 librados: init done 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.087+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa398001850 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.087+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fa398006210 con 0x7fa3a8070c70 2024-02-10T08:28:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.087+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c004f50 con 0x7fa3a8070c70 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.087+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 12) v1 ==== 38595+0+0 (unknown 4198815368 0 0) 0x7fa39800fa40 con 0x7fa3a8070c70 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7fa39803ed10 con 0x7fa3a8070c70 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 10 client.14160.objecter ms_dispatch 0x7fa3a8001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 3 client.14160.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 3 client.14160.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 20 client.14160.objecter dump_active .. 0 homeless 2024-02-10T08:28:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.088+0000 7fa39f7fe700 10 client.14160.objecter ms_handle_connect 0x7fa3940352e0 2024-02-10T08:28:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.092+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa398011060 con 0x7fa3a8070c70 2024-02-10T08:28:36.255 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.254+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa38c005780 con 0x7fa3a8070c70 2024-02-10T08:28:36.255 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.255+0000 7fa39f7fe700 1 -- 172.21.15.107:0/2930224630 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fa398013140 con 0x7fa3a8070c70 2024-02-10T08:28:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 8443 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 10 librados: watch_flush enter 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 10 librados: watch_flush exit 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 20 client.14160.objecter shutdown clearing up homeless session... 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 10 client.14160.objecter successfully canceled tick 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 >> v1:172.21.15.107:6800/3360068681 conn(0x7fa3940352e0 legacy=0x7fa3940377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 >> v1:172.21.15.107:6789/0 conn(0x7fa3a8070c70 legacy=0x7fa3a80fa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 shutdown_connections 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 >> 172.21.15.107:0/2930224630 conn(0x7fa3a80f64c0 msgr2=0x7fa3a80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 shutdown_connections 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.256+0000 7fa3b03d3700 1 -- 172.21.15.107:0/2930224630 wait complete. 2024-02-10T08:28:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:36.257+0000 7fa3b03d3700 1 librados: shutdown 2024-02-10T08:28:36.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:36.255562+0000 mon.a (mon.0) 89 : audit [DBG] from='client.? v1:172.21.15.107:0/2930224630' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T08:28:36.961 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-02-10T08:28:36.969 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-02-10T08:28:36.969 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-02-10T08:28:36.969 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr:Ceph Dashboard is now available at: 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr: User: admin 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr: Password: qlo7amppch 2024-02-10T08:28:36.972 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling autotune for osd_memory_target 2024-02-10T08:28:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.490+0000 7f028bf4e700 1 Processor -- start 2024-02-10T08:28:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f028bf4e700 1 -- start start 2024-02-10T08:28:37.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f028bf4e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0284070d20 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f0289cea700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02840fcae0 0x7f02840feed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58090/0 (socket says 172.21.15.107:58090) 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f0289cea700 1 -- 172.21.15.107:0/2870116812 learned_addr learned my addr 172.21.15.107:0/2870116812 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244672244 0 0) 0x7f0284070d20 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.491+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02700033f0 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873680071 0 0) 0x7f02700033f0 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0284070ed0 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0274001590 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0274001d20 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f0288ce8700 1 -- 172.21.15.107:0/2870116812 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0274005780 con 0x7f02840fcae0 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f028bf4e700 1 -- 172.21.15.107:0/2870116812 >> v1:172.21.15.107:6789/0 conn(0x7f02840fcae0 legacy=0x7f02840feed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f028bf4e700 1 -- 172.21.15.107:0/2870116812 shutdown_connections 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f028bf4e700 1 -- 172.21.15.107:0/2870116812 >> 172.21.15.107:0/2870116812 conn(0x7f02840f65b0 msgr2=0x7f02840f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f028bf4e700 1 -- 172.21.15.107:0/2870116812 shutdown_connections 2024-02-10T08:28:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.492+0000 7f028bf4e700 1 -- 172.21.15.107:0/2870116812 wait complete. 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 librados: starting msgr at 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 librados: starting objecter 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 Processor -- start 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 -- start start 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 librados: setting wanted keys 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 librados: calling monclient init 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f028bf4e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02840fc500 con 0x7f02840fcae0 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f0289cea700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02840fcae0 0x7f02840fa9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58096/0 (socket says 172.21.15.107:58096) 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f0289cea700 1 -- 172.21.15.107:0/288969218 learned_addr learned my addr 172.21.15.107:0/288969218 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f027affd700 10 client.?.objecter ms_handle_connect 0x7f02840fcae0 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f027affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763032818 0 0) 0x7f02840fc500 con 0x7f02840fcae0 2024-02-10T08:28:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.493+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0270005580 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337519752 0 0) 0x7f0270005580 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02840fc500 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0274000fe0 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3383845504 0 0) 0x7f02840fc500 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840fc690 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02840fc940 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f028bf4e700 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02840fb400 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f028bf4e700 1 librados: init done 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f02740049e0 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.494+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f02740061a0 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f0274007430 con 0x7f02840fcae0 2024-02-10T08:28:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0268004f50 con 0x7f02840fcae0 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f027403ed50 con 0x7f02840fcae0 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 10 client.14162.objecter ms_dispatch 0x7f0284001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 20 client.14162.objecter dump_active .. 0 homeless 2024-02-10T08:28:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.495+0000 7f027affd700 10 client.14162.objecter ms_handle_connect 0x7f0270035320 2024-02-10T08:28:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.500+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0274007bf0 con 0x7f02840fcae0 2024-02-10T08:28:37.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.660+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0268005780 con 0x7f02840fcae0 2024-02-10T08:28:37.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.661+0000 7f027affd700 1 -- 172.21.15.107:0/288969218 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7f0274013240 con 0x7f02840fcae0 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.662+0000 7f028bf4e700 10 librados: watch_flush enter 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.662+0000 7f028bf4e700 10 librados: watch_flush exit 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.662+0000 7f028bf4e700 20 client.14162.objecter shutdown clearing up homeless session... 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.662+0000 7f028bf4e700 10 client.14162.objecter successfully canceled tick 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0270035320 legacy=0x7f02700377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 >> v1:172.21.15.107:6789/0 conn(0x7f02840fcae0 legacy=0x7f02840fa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:37.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 shutdown_connections 2024-02-10T08:28:37.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 >> 172.21.15.107:0/288969218 conn(0x7f02840f65b0 msgr2=0x7f02840fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:37.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 shutdown_connections 2024-02-10T08:28:37.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 -- 172.21.15.107:0/288969218 wait complete. 2024-02-10T08:28:37.665 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:37.663+0000 7f028bf4e700 1 librados: shutdown 2024-02-10T08:28:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:36.474157+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.811+0000 7f07c92a6700 1 Processor -- start 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.811+0000 7f07c92a6700 1 -- start start 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.811+0000 7f07c92a6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f07c40fad90 con 0x7f07c40fa260 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f07c40fa260 0x7f07c40fa680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58102/0 (socket says 172.21.15.107:58102) 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c2ffd700 1 -- 172.21.15.107:0/438769772 learned_addr learned my addr 172.21.15.107:0/438769772 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431194520 0 0) 0x7f07c40fad90 con 0x7f07c40fa260 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac0033f0 con 0x7f07c40fa260 2024-02-10T08:28:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655750381 0 0) 0x7f07ac0033f0 con 0x7f07c40fa260 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c40faf20 con 0x7f07c40fa260 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f07b8001590 con 0x7f07c40fa260 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f07b8001d20 con 0x7f07c40fa260 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.812+0000 7f07c1ffb700 1 -- 172.21.15.107:0/438769772 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f07b8005780 con 0x7f07c40fa260 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- 172.21.15.107:0/438769772 >> v1:172.21.15.107:6789/0 conn(0x7f07c40fa260 legacy=0x7f07c40fa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- 172.21.15.107:0/438769772 shutdown_connections 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- 172.21.15.107:0/438769772 >> 172.21.15.107:0/438769772 conn(0x7f07c40f5cc0 msgr2=0x7f07c40f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- 172.21.15.107:0/438769772 shutdown_connections 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- 172.21.15.107:0/438769772 wait complete. 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 librados: starting msgr at 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 librados: starting objecter 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 Processor -- start 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 -- start start 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 librados: setting wanted keys 2024-02-10T08:28:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.813+0000 7f07c92a6700 1 librados: calling monclient init 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07c92a6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f07c410b040 con 0x7f07c40fa260 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07c2ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f07c40fa260 0x7f07c410a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58114/0 (socket says 172.21.15.107:58114) 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07c2ffd700 1 -- 172.21.15.107:0/351059164 learned_addr learned my addr 172.21.15.107:0/351059164 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 10 client.?.objecter ms_handle_connect 0x7f07c40fa260 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434326277 0 0) 0x7f07c410b040 con 0x7f07c40fa260 2024-02-10T08:28:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac005580 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924130316 0 0) 0x7f07ac005580 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07c410b040 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f07b8000fe0 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2815974316 0 0) 0x7f07c410b040 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c410b1d0 con 0x7f07c40fa260 2024-02-10T08:28:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.814+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c410b480 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07c92a6700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c410f170 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07c92a6700 1 librados: init done 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f07b8001430 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f07b8006220 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f07b800fa80 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f07b803fd50 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 10 client.14164.objecter ms_dispatch 0x7f07c4001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07abfff700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.815+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0004f50 con 0x7f07c40fa260 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.816+0000 7f07abfff700 20 client.14164.objecter dump_active .. 0 homeless 2024-02-10T08:28:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.816+0000 7f07abfff700 10 client.14164.objecter ms_handle_connect 0x7f07ac035390 2024-02-10T08:28:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:38.820+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f07b80113f0 con 0x7f07c40fa260 2024-02-10T08:28:39.070 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.069+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f07b0005780 con 0x7f07c40fa260 2024-02-10T08:28:39.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.072+0000 7f07abfff700 1 -- 172.21.15.107:0/351059164 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v24)=0 set mgr/dashboard/cluster/status v24) v1 ==== 153+0+0 (unknown 2554848587 0 0) 0x7f07b8013320 con 0x7f07c40fa260 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 10 librados: watch_flush enter 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 10 librados: watch_flush exit 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 20 client.14164.objecter shutdown clearing up homeless session... 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 10 client.14164.objecter successfully canceled tick 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 >> v1:172.21.15.107:6800/3360068681 conn(0x7f07ac035390 legacy=0x7f07ac037850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.073+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 >> v1:172.21.15.107:6789/0 conn(0x7f07c40fa260 legacy=0x7f07c410a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.074+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 shutdown_connections 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.074+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 >> 172.21.15.107:0/351059164 conn(0x7f07c40f5cc0 msgr2=0x7f07c40f6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.074+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 shutdown_connections 2024-02-10T08:28:39.075 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.074+0000 7f07c92a6700 1 -- 172.21.15.107:0/351059164 wait complete. 2024-02-10T08:28:39.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-02-10T08:28:39.074+0000 7f07c92a6700 1 librados: shutdown 2024-02-10T08:28:39.076 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr:Or, if you are only running a single cluster on this host: 2024-02-10T08:28:39.727 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: ceph telemetry on 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr:For more information see: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:28:39.728 INFO:teuthology.orchestra.run.smithi107.stderr:Bootstrap complete. 2024-02-10T08:28:39.765 INFO:tasks.cephadm:Fetching config... 2024-02-10T08:28:39.765 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:28:39.766 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T08:28:39.823 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T08:28:39.823 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:28:39.823 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T08:28:39.880 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T08:28:39.880 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:28:39.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T08:28:39.948 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T08:28:39.948 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:28:39.948 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T08:28:40.008 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T08:28:40.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+R2qzAiXC7xfFomIoay0WZufWxnlhb0oRlR+Y3Pv3SZ1Io+SEMR2gGSNUfOOlY3egV+MsjEzisz+HEoa3S5QLRTmatRzvcXXSRUVJb4Lpc5CXv60AMs233ihTPdYeOxQc5rlCJDcw+w2B/fb6OXSetM/XMKHkbEKRbqQImPxNjH6m085T8HXbqyAbMPC4rb0HFjgaCICLFtjzegnsW99IwbQaQ78qcyAY6ZOmtC11F3ITQ10cRWxTz5RQjUK+birIwvUvjgMJxleDbWJiq9qZo/bYs3gL35/hwkoW7ij+iA+gWYbst66thDAhS4HQ+lJzNYvfqxkB9k8eV832O0vN4OgLYVsU6DYqS1Mbvc9YuqlIighhVBYWGfNMlY3rJyUDl6OSln97FGMjA71j00Qk8FCJ7XciPKsOO52pmw6NiqW/YBzN1OEJSlPDifi4+DLiGnhoJxRB7ERE+1R7lrtKuMv7Zm6iAR4cTwMMCDNLQ6URfLASKTxSrIxnqItjl1M= ceph-275c2754-c7ee-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T08:28:40.085 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+R2qzAiXC7xfFomIoay0WZufWxnlhb0oRlR+Y3Pv3SZ1Io+SEMR2gGSNUfOOlY3egV+MsjEzisz+HEoa3S5QLRTmatRzvcXXSRUVJb4Lpc5CXv60AMs233ihTPdYeOxQc5rlCJDcw+w2B/fb6OXSetM/XMKHkbEKRbqQImPxNjH6m085T8HXbqyAbMPC4rb0HFjgaCICLFtjzegnsW99IwbQaQ78qcyAY6ZOmtC11F3ITQ10cRWxTz5RQjUK+birIwvUvjgMJxleDbWJiq9qZo/bYs3gL35/hwkoW7ij+iA+gWYbst66thDAhS4HQ+lJzNYvfqxkB9k8eV832O0vN4OgLYVsU6DYqS1Mbvc9YuqlIighhVBYWGfNMlY3rJyUDl6OSln97FGMjA71j00Qk8FCJ7XciPKsOO52pmw6NiqW/YBzN1OEJSlPDifi4+DLiGnhoJxRB7ERE+1R7lrtKuMv7Zm6iAR4cTwMMCDNLQ6URfLASKTxSrIxnqItjl1M= ceph-275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:28:40.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+R2qzAiXC7xfFomIoay0WZufWxnlhb0oRlR+Y3Pv3SZ1Io+SEMR2gGSNUfOOlY3egV+MsjEzisz+HEoa3S5QLRTmatRzvcXXSRUVJb4Lpc5CXv60AMs233ihTPdYeOxQc5rlCJDcw+w2B/fb6OXSetM/XMKHkbEKRbqQImPxNjH6m085T8HXbqyAbMPC4rb0HFjgaCICLFtjzegnsW99IwbQaQ78qcyAY6ZOmtC11F3ITQ10cRWxTz5RQjUK+birIwvUvjgMJxleDbWJiq9qZo/bYs3gL35/hwkoW7ij+iA+gWYbst66thDAhS4HQ+lJzNYvfqxkB9k8eV832O0vN4OgLYVsU6DYqS1Mbvc9YuqlIighhVBYWGfNMlY3rJyUDl6OSln97FGMjA71j00Qk8FCJ7XciPKsOO52pmw6NiqW/YBzN1OEJSlPDifi4+DLiGnhoJxRB7ERE+1R7lrtKuMv7Zm6iAR4cTwMMCDNLQ6URfLASKTxSrIxnqItjl1M= ceph-275c2754-c7ee-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T08:28:40.141 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+R2qzAiXC7xfFomIoay0WZufWxnlhb0oRlR+Y3Pv3SZ1Io+SEMR2gGSNUfOOlY3egV+MsjEzisz+HEoa3S5QLRTmatRzvcXXSRUVJb4Lpc5CXv60AMs233ihTPdYeOxQc5rlCJDcw+w2B/fb6OXSetM/XMKHkbEKRbqQImPxNjH6m085T8HXbqyAbMPC4rb0HFjgaCICLFtjzegnsW99IwbQaQ78qcyAY6ZOmtC11F3ITQ10cRWxTz5RQjUK+birIwvUvjgMJxleDbWJiq9qZo/bYs3gL35/hwkoW7ij+iA+gWYbst66thDAhS4HQ+lJzNYvfqxkB9k8eV832O0vN4OgLYVsU6DYqS1Mbvc9YuqlIighhVBYWGfNMlY3rJyUDl6OSln97FGMjA71j00Qk8FCJ7XciPKsOO52pmw6NiqW/YBzN1OEJSlPDifi4+DLiGnhoJxRB7ERE+1R7lrtKuMv7Zm6iAR4cTwMMCDNLQ6URfLASKTxSrIxnqItjl1M= ceph-275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:28:40.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T08:28:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:39.072760+0000 mon.a (mon.0) 91 : audit [INF] from='client.? v1:172.21.15.107:0/351059164' entity='client.admin' 2024-02-10T08:28:42.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:41.809989+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:42.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:42.168916+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:28:42.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:42.171984+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:42.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:42.173212+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:42.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:42.178582+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.347+0000 7f0b5e559700 1 -- 172.21.15.107:0/4167102204 >> v1:172.21.15.107:6789/0 conn(0x7f0b58061230 legacy=0x7f0b580fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.347+0000 7f0b5e559700 1 -- 172.21.15.107:0/4167102204 shutdown_connections 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.347+0000 7f0b5e559700 1 -- 172.21.15.107:0/4167102204 >> 172.21.15.107:0/4167102204 conn(0x7f0b580f1bd0 msgr2=0x7f0b580f3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.347+0000 7f0b5e559700 1 -- 172.21.15.107:0/4167102204 shutdown_connections 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.347+0000 7f0b5e559700 1 -- 172.21.15.107:0/4167102204 wait complete. 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 librados: starting msgr at 2024-02-10T08:28:43.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 librados: starting objecter 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 Processor -- start 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 -- start start 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 librados: setting wanted keys 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 librados: calling monclient init 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b5e559700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b580f7b60 con 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b57fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0b58061230 0x7f0b580f6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47092/0 (socket says 172.21.15.107:47092) 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.348+0000 7f0b57fff700 1 -- 172.21.15.107:0/2970760005 learned_addr learned my addr 172.21.15.107:0/2970760005 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 10 client.?.objecter ms_handle_connect 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773211788 0 0) 0x7f0b580f7b60 con 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40005580 con 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321406674 0 0) 0x7f0b40005580 con 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b580f7b60 con 0x7f0b58061230 2024-02-10T08:28:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0b480020c0 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675798692 0 0) 0x7f0b580f7b60 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580f7cf0 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b580f7fa0 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b5e559700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b580f6a60 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b5e559700 1 librados: init done 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0b480028a0 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.349+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0b480038d0 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f0b48003a50 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b44004f50 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f0b4803eb30 con 0x7f0b58061230 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 10 client.14166.objecter ms_dispatch 0x7f0b58001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 20 client.14166.objecter dump_active .. 0 homeless 2024-02-10T08:28:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.350+0000 7f0b557fa700 10 client.14166.objecter ms_handle_connect 0x7f0b40035420 2024-02-10T08:28:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.355+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0b48014ad0 con 0x7f0b58061230 2024-02-10T08:28:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.518+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0b44005780 con 0x7f0b58061230 2024-02-10T08:28:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.521+0000 7f0b557fa700 1 -- 172.21.15.107:0/2970760005 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7f0b48012f40 con 0x7f0b58061230 2024-02-10T08:28:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 10 librados: watch_flush enter 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 10 librados: watch_flush exit 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 20 client.14166.objecter shutdown clearing up homeless session... 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 10 client.14166.objecter successfully canceled tick 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0b40035420 legacy=0x7f0b400378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 >> v1:172.21.15.107:6789/0 conn(0x7f0b58061230 legacy=0x7f0b580f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 shutdown_connections 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 >> 172.21.15.107:0/2970760005 conn(0x7f0b580f1bd0 msgr2=0x7f0b580f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 shutdown_connections 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 -- 172.21.15.107:0/2970760005 wait complete. 2024-02-10T08:28:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:43.523+0000 7f0b5e559700 1 librados: shutdown 2024-02-10T08:28:44.398 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T08:28:44.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T08:28:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:43.521867+0000 mon.a (mon.0) 97 : audit [INF] from='client.? v1:172.21.15.107:0/2970760005' entity='client.admin' 2024-02-10T08:28:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:43.545508+0000 mon.a (mon.0) 98 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:43.549130+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 -- 172.21.15.107:0/2753317369 >> v1:172.21.15.107:6789/0 conn(0x7f458c0f6470 legacy=0x7f458c0f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 -- 172.21.15.107:0/2753317369 shutdown_connections 2024-02-10T08:28:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 -- 172.21.15.107:0/2753317369 >> 172.21.15.107:0/2753317369 conn(0x7f458c0f1bd0 msgr2=0x7f458c0f3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 -- 172.21.15.107:0/2753317369 shutdown_connections 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 -- 172.21.15.107:0/2753317369 wait complete. 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 librados: starting msgr at 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.140+0000 7f45925fc700 1 librados: starting objecter 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45925fc700 1 Processor -- start 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45925fc700 1 -- start start 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45925fc700 1 librados: setting wanted keys 2024-02-10T08:28:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45925fc700 1 librados: calling monclient init 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45925fc700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f458c1072b0 con 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f458bfff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f458c0f6470 0x7f458c106ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47112/0 (socket says 172.21.15.107:47112) 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f458bfff700 1 -- 172.21.15.107:0/3095834391 learned_addr learned my addr 172.21.15.107:0/3095834391 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45897fa700 10 client.?.objecter ms_handle_connect 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925448551 0 0) 0x7f458c1072b0 con 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.141+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4574005580 con 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603311791 0 0) 0x7f4574005580 con 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f458c1072b0 con 0x7f458c0f6470 2024-02-10T08:28:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f457c0020c0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1071843283 0 0) 0x7f458c1072b0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c107440 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c1076f0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45925fc700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c10b3e0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45925fc700 1 librados: init done 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f457c0028a0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f457c003870 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.142+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f457c0039f0 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f457c03fb30 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 10 client.14168.objecter ms_dispatch 0x7f458c001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 20 client.14168.objecter dump_active .. 0 homeless 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578004f50 con 0x7f458c0f6470 2024-02-10T08:28:46.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.143+0000 7f45897fa700 10 client.14168.objecter ms_handle_connect 0x7f4574035440 2024-02-10T08:28:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.147+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f457c003d40 con 0x7f458c0f6470 2024-02-10T08:28:46.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.312+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f4578000bc0 con 0x7f4574035440 2024-02-10T08:28:46.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.316+0000 7f45897fa700 1 -- 172.21.15.107:0/3095834391 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f4578000bc0 con 0x7f4574035440 2024-02-10T08:28:46.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 10 librados: watch_flush enter 2024-02-10T08:28:46.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 10 librados: watch_flush exit 2024-02-10T08:28:46.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 20 client.14168.objecter shutdown clearing up homeless session... 2024-02-10T08:28:46.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 10 client.14168.objecter successfully canceled tick 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 >> v1:172.21.15.107:6800/3360068681 conn(0x7f4574035440 legacy=0x7f4574037900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.317+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 >> v1:172.21.15.107:6789/0 conn(0x7f458c0f6470 legacy=0x7f458c106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.318+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 shutdown_connections 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.318+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 >> 172.21.15.107:0/3095834391 conn(0x7f458c0f1bd0 msgr2=0x7f458c0fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.318+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 shutdown_connections 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.318+0000 7f45925fc700 1 -- 172.21.15.107:0/3095834391 wait complete. 2024-02-10T08:28:46.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:28:46.318+0000 7f45925fc700 1 librados: shutdown 2024-02-10T08:28:47.014 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-02-10T08:28:47.014 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:28:47.014 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-02-10T08:28:47.036 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:28:47.036 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:28:47.097 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-02-10T08:28:47.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch host add smithi195 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.316492+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.318979+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.320272+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.320845+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:46.321818+0000 mgr.y (mgr.14146) 6 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:28:46.337222+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.365499+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:28:46.367746+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:28:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:49.999335+0000 mgr.y (mgr.14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:28:53.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:51.999771+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:28:55.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:54.000101+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:28:57.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:56.000363+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:28:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:28:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:28:58.000592+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:01.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:00.000934+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:02.001281+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:04.001586+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:06.001909+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:08.002268+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:10.002698+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:12.622 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 -- 172.21.15.195:0/4005534067 >> v1:172.21.15.107:6789/0 conn(0x7fd9940f5830 legacy=0x7fd9940f5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 -- 172.21.15.195:0/4005534067 shutdown_connections 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 -- 172.21.15.195:0/4005534067 >> 172.21.15.195:0/4005534067 conn(0x7fd9940f1350 msgr2=0x7fd9940f3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 -- 172.21.15.195:0/4005534067 shutdown_connections 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 -- 172.21.15.195:0/4005534067 wait complete. 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 librados: starting msgr at 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.577+0000 7fd9994af700 1 librados: starting objecter 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd9994af700 1 Processor -- start 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd9994af700 1 -- start start 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd9994af700 1 librados: setting wanted keys 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd9994af700 1 librados: calling monclient init 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd9994af700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9941072e0 con 0x7fd9940f5830 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd992ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9940f5830 0x7fd994106bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:57458/0 (socket says 172.21.15.195:57458) 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd992ffd700 1 -- 172.21.15.195:0/37904882 learned_addr learned my addr 172.21.15.195:0/37904882 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd97bfff700 10 client.?.objecter ms_handle_connect 0x7fd9940f5830 2024-02-10T08:29:12.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.578+0000 7fd97bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957250001 0 0) 0x7fd9941072e0 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c005580 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2275220471 0 0) 0x7fd97c005580 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9941072e0 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd984000ed0 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2193305559 0 0) 0x7fd9941072e0 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994107470 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd994107720 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd9994af700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99410b410 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd9994af700 1 librados: init done 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd984002400 con 0x7fd9940f5830 2024-02-10T08:29:12.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.579+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7fd984003990 con 0x7fd9940f5830 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7fd984003b10 con 0x7fd9940f5830 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7fd98403eb80 con 0x7fd9940f5830 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 10 client.14170.objecter ms_dispatch 0x7fd994001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd97bfff700 20 client.14170.objecter dump_active .. 0 homeless 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.580+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd980004f50 con 0x7fd9940f5830 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.581+0000 7fd97bfff700 10 client.14170.objecter ms_handle_connect 0x7fd97c035470 2024-02-10T08:29:12.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.586+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd984003e80 con 0x7fd9940f5830 2024-02-10T08:29:12.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:12.752+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}) v1 -- 0x7fd980000bc0 con 0x7fd97c035470 2024-02-10T08:29:13.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:29:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [90B blob data] 2024-02-10T08:29:13.374 INFO:teuthology.orchestra.run.smithi195.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-02-10T08:29:13.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.372+0000 7fd97bfff700 1 -- 172.21.15.195:0/37904882 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 187748994) 0x7fd980000bc0 con 0x7fd97c035470 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 10 librados: watch_flush enter 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 10 librados: watch_flush exit 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 20 client.14170.objecter shutdown clearing up homeless session... 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 10 client.14170.objecter successfully canceled tick 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 >> v1:172.21.15.107:6800/3360068681 conn(0x7fd97c035470 legacy=0x7fd97c037930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 >> v1:172.21.15.107:6789/0 conn(0x7fd9940f5830 legacy=0x7fd994106bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:13.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 shutdown_connections 2024-02-10T08:29:13.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 >> 172.21.15.195:0/37904882 conn(0x7fd9940f1350 msgr2=0x7fd9940f3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:13.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 shutdown_connections 2024-02-10T08:29:13.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 -- 172.21.15.195:0/37904882 wait complete. 2024-02-10T08:29:13.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:13.374+0000 7fd9994af700 1 librados: shutdown 2024-02-10T08:29:13.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:29:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [80B blob data] 2024-02-10T08:29:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:12.002983+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:13.372782+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:13.399769+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:14.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T08:29:14.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:12.755317+0000 mgr.y (mgr.14146) 20 : audit [DBG] from='client.14170 v1:172.21.15.195:0/37904882' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:29:14.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:13.373176+0000 mgr.y (mgr.14146) 21 : cephadm [INF] Added host smithi195 2024-02-10T08:29:14.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:14.089716+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:15.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 -- 172.21.15.195:0/2732617339 >> v1:172.21.15.107:6789/0 conn(0x7f0d080f7570 legacy=0x7f0d080f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:15.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 -- 172.21.15.195:0/2732617339 shutdown_connections 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 -- 172.21.15.195:0/2732617339 >> 172.21.15.195:0/2732617339 conn(0x7f0d080f2cd0 msgr2=0x7f0d080f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 -- 172.21.15.195:0/2732617339 shutdown_connections 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 -- 172.21.15.195:0/2732617339 wait complete. 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 librados: starting msgr at 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.900+0000 7f0d0f85e700 1 librados: starting objecter 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0f85e700 1 Processor -- start 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0f85e700 1 -- start start 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0f85e700 1 librados: setting wanted keys 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0f85e700 1 librados: calling monclient init 2024-02-10T08:29:15.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0f85e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d081072c0 con 0x7f0d080f7570 2024-02-10T08:29:15.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0d5fa700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0d080f7570 0x7f0d08106bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:57476/0 (socket says 172.21.15.195:57476) 2024-02-10T08:29:15.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0d0d5fa700 1 -- 172.21.15.195:0/675976563 learned_addr learned my addr 172.21.15.195:0/675976563 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:15.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0cfe7fc700 10 client.?.objecter ms_handle_connect 0x7f0d080f7570 2024-02-10T08:29:15.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0cfe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954170094 0 0) 0x7f0d081072c0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.901+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf4005580 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 215327040 0 0) 0x7f0cf4005580 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d081072c0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0d04000ed0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 287499957 0 0) 0x7f0d081072c0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08107450 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08107700 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0d0f85e700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0810b3f0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0d0f85e700 1 librados: init done 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0d040028a0 con 0x7f0d080f7570 2024-02-10T08:29:15.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.902+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f0d04003990 con 0x7f0d080f7570 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f0d04003b10 con 0x7f0d080f7570 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f0d0403fc00 con 0x7f0d080f7570 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 10 client.14172.objecter ms_dispatch 0x7f0d08001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0cfe7fc700 20 client.14172.objecter dump_active .. 0 homeless 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.903+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec004f50 con 0x7f0d080f7570 2024-02-10T08:29:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.904+0000 7f0cfe7fc700 10 client.14172.objecter ms_handle_connect 0x7f0cf4035370 2024-02-10T08:29:15.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:15.908+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0d04014d00 con 0x7f0d080f7570 2024-02-10T08:29:15.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:14.003417+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:15.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:15.080758+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:16.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.085+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cec000bc0 con 0x7f0cf4035370 2024-02-10T08:29:16.088 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.086+0000 7f0cfe7fc700 1 -- 172.21.15.195:0/675976563 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (unknown 0 0 1244900858) 0x7f0cec000bc0 con 0x7f0cf4035370 2024-02-10T08:29:16.088 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:29:16.088 INFO:teuthology.orchestra.run.smithi195.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 10 librados: watch_flush enter 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 10 librados: watch_flush exit 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 20 client.14172.objecter shutdown clearing up homeless session... 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 10 client.14172.objecter successfully canceled tick 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0cf4035370 legacy=0x7f0cf4037830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 >> v1:172.21.15.107:6789/0 conn(0x7f0d080f7570 legacy=0x7f0d08106bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 shutdown_connections 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 >> 172.21.15.195:0/675976563 conn(0x7f0d080f2cd0 msgr2=0x7f0d080f50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 shutdown_connections 2024-02-10T08:29:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 -- 172.21.15.195:0/675976563 wait complete. 2024-02-10T08:29:16.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:16.088+0000 7f0d0f85e700 1 librados: shutdown 2024-02-10T08:29:17.529 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-02-10T08:29:17.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd crush tunables jewel 2024-02-10T08:29:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:16.003827+0000 mgr.y (mgr.14146) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:16.087673+0000 mgr.y (mgr.14146) 24 : audit [DBG] from='client.14172 v1:172.21.15.195:0/675976563' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:29:19.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 -- 172.21.15.107:0/2198841890 >> v1:172.21.15.107:6789/0 conn(0x7f685c0f85e0 legacy=0x7f685c0f89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 -- 172.21.15.107:0/2198841890 shutdown_connections 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 -- 172.21.15.107:0/2198841890 >> 172.21.15.107:0/2198841890 conn(0x7f685c06c390 msgr2=0x7f685c06c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 -- 172.21.15.107:0/2198841890 shutdown_connections 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 -- 172.21.15.107:0/2198841890 wait complete. 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 librados: starting msgr at 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.377+0000 7f6863479700 1 librados: starting objecter 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 Processor -- start 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 -- start start 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 librados: setting wanted keys 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 librados: calling monclient init 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f685c10d490 con 0x7f685c0f85e0 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6861215700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f685c0f85e0 0x7f685c10cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37552/0 (socket says 172.21.15.107:37552) 2024-02-10T08:29:19.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6861215700 1 -- 172.21.15.107:0/3730814939 learned_addr learned my addr 172.21.15.107:0/3730814939 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 10 client.?.objecter ms_handle_connect 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640494401 0 0) 0x7f685c10d490 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6848005580 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3572062482 0 0) 0x7f6848005580 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f685c10d490 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f684c0020c0 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2912675251 0 0) 0x7f685c10d490 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c10d620 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.378+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c10d8d0 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f6863479700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c10fd50 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f6863479700 1 librados: init done 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f684c002400 con 0x7f685c0f85e0 2024-02-10T08:29:19.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f684c003790 con 0x7f685c0f85e0 2024-02-10T08:29:19.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.379+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f684c0039b0 con 0x7f685c0f85e0 2024-02-10T08:29:19.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6840004f50 con 0x7f685c0f85e0 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 4220010722 0 0) 0x7f684c03ec00 con 0x7f685c0f85e0 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 10 client.14174.objecter ms_dispatch 0x7f685c001550 osd_map(3..3 src has 1..3) v4 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 20 client.14174.objecter dump_active .. 0 homeless 2024-02-10T08:29:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.380+0000 7f68527fc700 10 client.14174.objecter ms_handle_connect 0x7f6848035300 2024-02-10T08:29:19.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.384+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f684c014c60 con 0x7f685c0f85e0 2024-02-10T08:29:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:19.552+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7f6840005200 con 0x7f685c0f85e0 2024-02-10T08:29:19.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:18.004099+0000 mgr.y (mgr.14146) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:19.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.289979+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.508+0000 7f68527fc700 1 -- 172.21.15.107:0/3730814939 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (unknown 2099115640 0 0) 0x7f684c0130d0 con 0x7f685c0f85e0 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 10 librados: watch_flush enter 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 10 librados: watch_flush exit 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 20 client.14174.objecter shutdown clearing up homeless session... 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 10 client.14174.objecter successfully canceled tick 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 >> v1:172.21.15.107:6800/3360068681 conn(0x7f6848035300 legacy=0x7f68480377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 >> v1:172.21.15.107:6789/0 conn(0x7f685c0f85e0 legacy=0x7f685c10cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 shutdown_connections 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 >> 172.21.15.107:0/3730814939 conn(0x7f685c06c390 msgr2=0x7f685c0fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 shutdown_connections 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 -- 172.21.15.107:0/3730814939 wait complete. 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:20.510+0000 7f6863479700 1 librados: shutdown 2024-02-10T08:29:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to jewel 2024-02-10T08:29:20.889 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.554043+0000 mon.a (mon.0) 111 : audit [INF] from='client.? v1:172.21.15.107:0/3730814939' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-02-10T08:29:20.889 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.620149+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.620738+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.621397+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.621887+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:19.622739+0000 mgr.y (mgr.14146) 26 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:19.654965+0000 mgr.y (mgr.14146) 27 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.684289+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:20.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:19.686993+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:21.226 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-02-10T08:29:21.226 INFO:tasks.cephadm:Adding mon.c on smithi107 2024-02-10T08:29:21.227 INFO:tasks.cephadm:Adding mon.b on smithi195 2024-02-10T08:29:21.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi107:[v1:172.21.15.107:6789]=a;smithi107:[v1:172.21.15.107:6790]=c;smithi195:[v1:172.21.15.195:6789]=b' 2024-02-10T08:29:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:20.004429+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:20.509613+0000 mon.a (mon.0) 118 : audit [INF] from='client.? v1:172.21.15.107:0/3730814939' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-02-10T08:29:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:20.509671+0000 mon.a (mon.0) 119 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.120+0000 7f32d4afe700 1 -- 172.21.15.195:0/3485486683 >> v1:172.21.15.107:6789/0 conn(0x7f32d00fc500 legacy=0x7f32d00fc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.120+0000 7f32d4afe700 1 -- 172.21.15.195:0/3485486683 shutdown_connections 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.120+0000 7f32d4afe700 1 -- 172.21.15.195:0/3485486683 >> 172.21.15.195:0/3485486683 conn(0x7f32d006c390 msgr2=0x7f32d006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.120+0000 7f32d4afe700 1 -- 172.21.15.195:0/3485486683 shutdown_connections 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.121+0000 7f32d4afe700 1 -- 172.21.15.195:0/3485486683 wait complete. 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.121+0000 7f32d4afe700 1 librados: starting msgr at 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.121+0000 7f32d4afe700 1 librados: starting objecter 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.121+0000 7f32d4afe700 1 Processor -- start 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32d4afe700 1 -- start start 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32d4afe700 1 librados: setting wanted keys 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32d4afe700 1 librados: calling monclient init 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32d4afe700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32d01083d0 con 0x7f32d00fc500 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32ce59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f32d00fc500 0x7f32d0107cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:53518/0 (socket says 172.21.15.195:53518) 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32ce59c700 1 -- 172.21.15.195:0/1724545550 learned_addr learned my addr 172.21.15.195:0/1724545550 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:23.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32bf7fe700 10 client.?.objecter ms_handle_connect 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081298954 0 0) 0x7f32d01083d0 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.122+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b8005580 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3547206091 0 0) 0x7f32b8005580 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d01083d0 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f32c4000ed0 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1171034780 0 0) 0x7f32d01083d0 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0108560 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0108810 con 0x7f32d00fc500 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32d4afe700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:23.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d010fd30 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32d4afe700 1 librados: init done 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f32c40011d0 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.123+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f32c4003790 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b0004f50 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f32c40039b0 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7f32c403ee70 con 0x7f32d00fc500 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 10 client.14176.objecter ms_dispatch 0x7f32d0001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 3 client.14176.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 3 client.14176.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 20 client.14176.objecter dump_active .. 0 homeless 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.124+0000 7f32bf7fe700 10 client.14176.objecter ms_handle_connect 0x7f32b8035300 2024-02-10T08:29:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.129+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f32c40149e0 con 0x7f32d00fc500 2024-02-10T08:29:23.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.301+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:[v1:172.21.15.107:6789]=a;smithi107:[v1:172.21.15.107:6790]=c;smithi195:[v1:172.21.15.195:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f32b0000ca0 con 0x7f32b8035300 2024-02-10T08:29:23.309 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.308+0000 7f32bf7fe700 1 -- 172.21.15.195:0/1724545550 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f32b0000ca0 con 0x7f32b8035300 2024-02-10T08:29:23.309 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-02-10T08:29:23.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 10 librados: watch_flush enter 2024-02-10T08:29:23.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 10 librados: watch_flush exit 2024-02-10T08:29:23.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 20 client.14176.objecter shutdown clearing up homeless session... 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 10 client.14176.objecter successfully canceled tick 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 >> v1:172.21.15.107:6800/3360068681 conn(0x7f32b8035300 legacy=0x7f32b80377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 >> v1:172.21.15.107:6789/0 conn(0x7f32d00fc500 legacy=0x7f32d0107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 shutdown_connections 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 >> 172.21.15.195:0/1724545550 conn(0x7f32d006c390 msgr2=0x7f32d00fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 shutdown_connections 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 -- 172.21.15.195:0/1724545550 wait complete. 2024-02-10T08:29:23.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:23.309+0000 7f32d4afe700 1 librados: shutdown 2024-02-10T08:29:23.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:22.004746+0000 mgr.y (mgr.14146) 29 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.309010+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.359631+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.361132+0000 mon.a (mon.0) 122 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.361763+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.367865+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.369533+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:23.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.370162+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:23.948 DEBUG:teuthology.orchestra.run.smithi107:mon.c> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T08:29:23.951 DEBUG:teuthology.orchestra.run.smithi195:mon.b> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T08:29:23.953 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T08:29:23.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:29:23.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:29:23.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:29:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:23.304254+0000 mgr.y (mgr.14146) 30 : audit [DBG] from='client.14176 v1:172.21.15.195:0/1724545550' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:[v1:172.21.15.107:6789]=a;smithi107:[v1:172.21.15.107:6790]=c;smithi195:[v1:172.21.15.195:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:29:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:23.306488+0000 mgr.y (mgr.14146) 31 : cephadm [INF] Saving service mon spec with placement smithi107:[v1:172.21.15.107:6789]=a;smithi107:[v1:172.21.15.107:6790]=c;smithi195:[v1:172.21.15.195:6789]=b;count:3 2024-02-10T08:29:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:23.370762+0000 mgr.y (mgr.14146) 32 : cephadm [INF] Deploying daemon mon.b on smithi195 2024-02-10T08:29:25.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:24.005101+0000 mgr.y (mgr.14146) 33 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.075+0000 7f6abc35b700 1 -- 172.21.15.195:0/1259456785 >> v1:172.21.15.107:6789/0 conn(0x7f6ab40fd5c0 legacy=0x7f6ab40fd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.075+0000 7f6abc35b700 1 -- 172.21.15.195:0/1259456785 shutdown_connections 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.075+0000 7f6abc35b700 1 -- 172.21.15.195:0/1259456785 >> 172.21.15.195:0/1259456785 conn(0x7f6ab40f3180 msgr2=0x7f6ab40f55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.075+0000 7f6abc35b700 1 -- 172.21.15.195:0/1259456785 shutdown_connections 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.075+0000 7f6abc35b700 1 -- 172.21.15.195:0/1259456785 wait complete. 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 librados: starting msgr at 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 librados: starting objecter 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 Processor -- start 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 -- start start 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 librados: setting wanted keys 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 librados: calling monclient init 2024-02-10T08:29:27.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6abc35b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ab406cf80 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6aba0f7700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6ab40fd5c0 0x7f6ab406b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:53542/0 (socket says 172.21.15.195:53542) 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6aba0f7700 1 -- 172.21.15.195:0/2142708738 learned_addr learned my addr 172.21.15.195:0/2142708738 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6aab7fe700 10 client.?.objecter ms_handle_connect 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.076+0000 7f6aab7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67848520 0 0) 0x7f6ab406cf80 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa00055d0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3112715802 0 0) 0x7f6aa00055d0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ab406cf80 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ab0001490 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3765130884 0 0) 0x7f6ab406cf80 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab406b880 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab406bb30 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6abc35b700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab4111ff0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6abc35b700 1 librados: init done 2024-02-10T08:29:27.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6ab0001ac0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.077+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 502370982 0 0) 0x7f6ab0003690 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f6ab0003810 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a98004f50 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7f6ab003eba0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 10 client.14178.objecter ms_dispatch 0x7f6ab4001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 3 client.14178.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 3 client.14178.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.078+0000 7f6aab7fe700 20 client.14178.objecter dump_active .. 0 homeless 2024-02-10T08:29:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.079+0000 7f6aab7fe700 10 client.14178.objecter ms_handle_connect 0x7f6aa0035480 2024-02-10T08:29:27.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.083+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6ab0003bb0 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.315+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a98005200 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.315+0000 7f6aab7fe700 1 -- 172.21.15.195:0/2142708738 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+675 (unknown 2237029548 0 2108605748) 0x7f6ab0012f20 con 0x7f6ab40fd5c0 2024-02-10T08:29:27.317 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:29:27.317 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","modified":"2024-02-10T08:27:37.431644Z","created":"2024-02-10T08:27:37.431644Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:29:27.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 10 librados: watch_flush enter 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 10 librados: watch_flush exit 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 20 client.14178.objecter shutdown clearing up homeless session... 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 10 client.14178.objecter successfully canceled tick 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 >> v1:172.21.15.107:6800/3360068681 conn(0x7f6aa0035480 legacy=0x7f6aa0037940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 >> v1:172.21.15.107:6789/0 conn(0x7f6ab40fd5c0 legacy=0x7f6ab406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 shutdown_connections 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 >> 172.21.15.195:0/2142708738 conn(0x7f6ab40f3180 msgr2=0x7f6ab40ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 shutdown_connections 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 -- 172.21.15.195:0/2142708738 wait complete. 2024-02-10T08:29:27.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:27.317+0000 7f6abc35b700 1 librados: shutdown 2024-02-10T08:29:27.320 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-10T08:29:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:26.005489+0000 mgr.y (mgr.14146) 34 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:27.316953+0000 mon.a (mon.0) 127 : audit [DBG] from='client.? v1:172.21.15.195:0/2142708738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:28 smithi195 systemd[1]: Starting Ceph mon.b for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:29:29.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:28.005874+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.988+0000 7fb691f7d880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.988+0000 7fb691f7d880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.988+0000 7fb691f7d880 0 pidfile_write: ignore empty --pid-file 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 0 load: jerasure load: lrc load: isa 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: DB SUMMARY 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T08:29:30.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 476 ; 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.env: 0x5646831c2080 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:29:30.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.info_log: 0x564684ea5980 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:29:30.292 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.db_log_dir: 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:29:30.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.write_buffer_manager: 0x564685ba23f0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:29:30.294 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.row_cache: None 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.wal_filter: None 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:29:30.295 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Compression algorithms supported: 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kZSTD supported: 0 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.993+0000 7fb691f7d880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:29:30.297 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.merge_operator: 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564684e55d28) 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cache_index_and_filter_blocks: 1 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: pin_top_level_index_and_filter: 1 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: index_type: 0 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: data_block_index_type: 0 2024-02-10T08:29:30.298 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: index_shortening: 1 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: hash_index_allow_collision: 1 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: checksum: 1 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: no_block_cache: 0 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_cache: 0x564684e8af10 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_cache_name: BinnedLRUCache 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_cache_options: 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: capacity : 536870912 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: num_shard_bits : 4 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: strict_capacity_limit : 0 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: high_pri_pool_ratio: 0.000 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_cache_compressed: (nil) 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: persistent_cache: (nil) 2024-02-10T08:29:30.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_size: 4096 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_size_deviation: 10 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_restart_interval: 16 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: index_block_restart_interval: 1 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: metadata_block_size: 4096 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: partition_filters: 0 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: use_delta_encoding: 1 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: whole_key_filtering: 1 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: verify_compression: 0 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: read_amp_bytes_per_bit: 0 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: format_version: 2 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: enable_index_compression: 1 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: block_align: 0 2024-02-10T08:29:30.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.num_levels: 7 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:29:30.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:29:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:29:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:29:30.304 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-10T08:29:30.305 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553769995317, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.994+0000 7fb691f7d880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.995+0000 7fb691f7d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553769995777, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707553769, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.995+0000 7fb691f7d880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.995+0000 7fb691f7d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553769996180, "job": 1, "event": "recovery_finished"} 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.996+0000 7fb691f7d880 4 rocksdb: DB pointer 0x564684f53800 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.996+0000 7fb678d6e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.996+0000 7fb678d6e700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** DB Stats ** 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:29:30.306 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** Compaction Stats [default] ** 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** Compaction Stats [default] ** 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:30.307 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** Compaction Stats [default] ** 2024-02-10T08:29:30.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** Compaction Stats [default] ** 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:29:30.309 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.996+0000 7fb691f7d880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.996+0000 7fb691f7d880 0 using public_addrv v1:172.21.15.195:6789/0 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.997+0000 7fb691f7d880 0 starting mon.b rank -1 at public addrs v1:172.21.15.195:6789/0 at bind addrs v1:172.21.15.195:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:29.998+0000 7fb691f7d880 1 mon.b@-1(???) e0 preinit fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb676d6a700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.310 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb680d7e700 4 rocksdb: (Original Log Time 2024/02/10-08:29:30.016048) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb680d7e700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb680d7e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553770016108, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1017, "num_deletes": 10, "total_data_size": 3472516, "memory_usage": 3491672, "flush_reason": "Manual Compaction"} 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb680d7e700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 0 mon.b@-1(synchronizing).mds e1 new map 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: e1 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: legacy client fscid: -1 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: No filesystems configured 2024-02-10T08:29:30.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.015+0000 7fb67b573700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.016+0000 7fb67b573700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.016+0000 7fb67b573700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.016+0000 7fb67b573700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.016+0000 7fb67b573700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:28.005874+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:29:30.016+0000 7fb67b573700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 bash[69548]: c37bb817dd700332c8ddc3d37e7e0ef92d2084846beb48ee6da0fe27d15bf663 2024-02-10T08:29:30.312 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:30 smithi195 systemd[1]: Started Ceph mon.b for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:29:30.490 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T08:29:30.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:29:35.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.069+0000 7fbb42b04700 1 -- 172.21.15.195:0/2551310327 >> v1:172.21.15.107:6789/0 conn(0x7fbb3c0fc500 legacy=0x7fbb3c0fc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:35.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.069+0000 7fbb42b04700 1 -- 172.21.15.195:0/2551310327 shutdown_connections 2024-02-10T08:29:35.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.069+0000 7fbb42b04700 1 -- 172.21.15.195:0/2551310327 >> 172.21.15.195:0/2551310327 conn(0x7fbb3c06c390 msgr2=0x7fbb3c06c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:35.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.069+0000 7fbb42b04700 1 -- 172.21.15.195:0/2551310327 shutdown_connections 2024-02-10T08:29:35.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.069+0000 7fbb42b04700 1 -- 172.21.15.195:0/2551310327 wait complete. 2024-02-10T08:29:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 librados: starting msgr at 2024-02-10T08:29:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 librados: starting objecter 2024-02-10T08:29:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 Processor -- start 2024-02-10T08:29:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 -- start start 2024-02-10T08:29:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 librados: setting wanted keys 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 librados: calling monclient init 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb3c0f9030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.070+0000 7fbb42b04700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb3c11a8e0 con 0x7fbb3c0fc500 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb408a0700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbb3c0fc500 0x7fbb3c0f8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:44390/0 (socket says 172.21.15.195:44390) 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb408a0700 1 -- 172.21.15.195:0/303095711 learned_addr learned my addr 172.21.15.195:0/303095711 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 10 client.?.objecter ms_handle_connect 0x7fbb3c0f9c70 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 10 client.?.objecter ms_handle_connect 0x7fbb3c0fc500 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046048614 0 0) 0x7fbb3c0f9030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.071+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c003030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10971169 0 0) 0x7fbb2c003030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb3c0f9030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 669468621 0 0) 0x7fbb28001800 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1636792134 0 0) 0x7fbb3c0f9030 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 >> v1:172.21.15.195:6789/0 conn(0x7fbb3c0fc500 legacy=0x7fbb3c0f8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:35.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c11ba80 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c11aad0 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.072+0000 7fbb42b04700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.073+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c11af80 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.073+0000 7fbb42b04700 1 librados: init done 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.074+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbb280023a0 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.074+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 669468621 0 0) 0x7fbb28003270 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.074+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c05f100 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.074+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7fbb280033f0 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7fbb2803ce90 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 10 client.14182.objecter ms_dispatch 0x7fbb3c001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 3 client.14182.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 3 client.14182.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:35.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 20 client.14182.objecter dump_active .. 0 homeless 2024-02-10T08:29:35.076 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.075+0000 7fbb39ffb700 10 client.14182.objecter ms_handle_connect 0x7fbb2c035530 2024-02-10T08:29:35.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.079+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbb28011ed0 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.309+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb3c05a3b0 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.310+0000 7fbb39ffb700 1 -- 172.21.15.195:0/303095711 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+893 (unknown 3557084514 0 152608156) 0x7fbb28010340 con 0x7fbb3c0f9c70 2024-02-10T08:29:35.311 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:29:35.311 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","modified":"2024-02-10T08:29:30.020288Z","created":"2024-02-10T08:27:37.431644Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T08:29:35.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.311+0000 7fbb42b04700 10 librados: watch_flush enter 2024-02-10T08:29:35.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.311+0000 7fbb42b04700 10 librados: watch_flush exit 2024-02-10T08:29:35.312 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.311+0000 7fbb42b04700 20 client.14182.objecter shutdown clearing up homeless session... 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.311+0000 7fbb42b04700 10 client.14182.objecter successfully canceled tick 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 >> v1:172.21.15.107:6800/3360068681 conn(0x7fbb2c035530 legacy=0x7fbb2c0379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 >> v1:172.21.15.107:6789/0 conn(0x7fbb3c0f9c70 legacy=0x7fbb3c0f8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 shutdown_connections 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 >> 172.21.15.195:0/303095711 conn(0x7fbb3c06c390 msgr2=0x7fbb3c0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 shutdown_connections 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 -- 172.21.15.195:0/303095711 wait complete. 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:35.312+0000 7fbb42b04700 1 librados: shutdown 2024-02-10T08:29:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:30.006161+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:30.022853+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:30.023034+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:30.035779+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:31.020078+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:32.006523+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:32.020034+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:32.021335+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:33.020327+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:34.006880+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:34.020344+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:35.020496+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.050192+0000 mon.a (mon.0) 137 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.052984+0000 mon.a (mon.0) 138 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0} removed_ranks: {} 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.065354+0000 mon.a (mon.0) 139 : cluster [DBG] fsmap 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.065412+0000 mon.a (mon.0) 140 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.065606+0000 mon.a (mon.0) 141 : cluster [DBG] mgrmap e13: y(active, since 65s) 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:35.068906+0000 mon.a (mon.0) 142 : cluster [INF] overall HEALTH_OK 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:35.071101+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:35.072174+0000 mon.a (mon.0) 144 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:35.074556+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:30.006161+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:30.022853+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:30.023034+0000 mon.a (mon.0) 130 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:30.035779+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:31.020078+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:32.006523+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:32.020034+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:32.021335+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:33.020327+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:34.006880+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:34.020344+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:35.020496+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.050192+0000 mon.a (mon.0) 137 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.052984+0000 mon.a (mon.0) 138 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0} removed_ranks: {} 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.065354+0000 mon.a (mon.0) 139 : cluster [DBG] fsmap 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.065412+0000 mon.a (mon.0) 140 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.065606+0000 mon.a (mon.0) 141 : cluster [DBG] mgrmap e13: y(active, since 65s) 2024-02-10T08:29:35.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:35.068906+0000 mon.a (mon.0) 142 : cluster [INF] overall HEALTH_OK 2024-02-10T08:29:35.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:35.071101+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:35.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:35.072174+0000 mon.a (mon.0) 144 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:35.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:35.074556+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:35.075430+0000 mgr.y (mgr.14146) 39 : cephadm [INF] Deploying daemon mon.c on smithi107 2024-02-10T08:29:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:35.310846+0000 mon.a (mon.0) 146 : audit [DBG] from='client.? v1:172.21.15.195:0/303095711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:36.020418+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:35.075430+0000 mgr.y (mgr.14146) 39 : cephadm [INF] Deploying daemon mon.c on smithi107 2024-02-10T08:29:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:35.310846+0000 mon.a (mon.0) 146 : audit [DBG] from='client.? v1:172.21.15.195:0/303095711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:36.020418+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:36.864 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T08:29:36.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:29:37.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:36.007123+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:36.007123+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:37.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:37 smithi107 systemd[1]: Starting Ceph mon.c for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:29:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.249+0000 7fc84bd3d880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:29:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.249+0000 7fc84bd3d880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-10T08:29:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.249+0000 7fc84bd3d880 0 pidfile_write: ignore empty --pid-file 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.253+0000 7fc84bd3d880 0 load: jerasure load: lrc load: isa 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Compile date Feb 8 2024 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: DB SUMMARY 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: CURRENT file: CURRENT 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-10T08:29:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 566 ; 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.create_if_missing: 0 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.env: 0x5609f9192080 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.info_log: 0x5609f9c1c220 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:29:38.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.db_log_dir: 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:29:38.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.write_buffer_manager: 0x5609fa868450 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:29:38.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.row_cache: None 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.wal_filter: None 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:29:38.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:29:38.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Compression algorithms supported: 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kZSTD supported: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:29:38.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.merge_operator: 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:29:38.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5609f9bc1d28) 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cache_index_and_filter_blocks: 1 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: pin_top_level_index_and_filter: 1 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: index_type: 0 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: data_block_index_type: 0 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: index_shortening: 1 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: hash_index_allow_collision: 1 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: checksum: 1 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: no_block_cache: 0 2024-02-10T08:29:38.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_cache: 0x5609f9bf6f10 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_cache_name: BinnedLRUCache 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_cache_options: 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: capacity : 536870912 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: num_shard_bits : 4 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: strict_capacity_limit : 0 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: high_pri_pool_ratio: 0.000 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_cache_compressed: (nil) 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: persistent_cache: (nil) 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_size: 4096 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_size_deviation: 10 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_restart_interval: 16 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: index_block_restart_interval: 1 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: metadata_block_size: 4096 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: partition_filters: 0 2024-02-10T08:29:38.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: use_delta_encoding: 1 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: whole_key_filtering: 1 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: verify_compression: 0 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: read_amp_bytes_per_bit: 0 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: format_version: 2 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: enable_index_compression: 1 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: block_align: 0 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:29:38.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.num_levels: 7 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:29:38.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:29:38.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:29:38.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:29:38.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:29:38.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:29:38.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.254+0000 7fc84bd3d880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553778255947, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-10T08:29:38.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553778256353, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1492, "table_properties": {"data_size": 578, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707553778, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.255+0000 7fc84bd3d880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.256+0000 7fc84bd3d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553778256722, "job": 1, "event": "recovery_finished"} 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.256+0000 7fc84bd3d880 4 rocksdb: DB pointer 0x5609f9cbf800 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.256+0000 7fc832b2e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.256+0000 7fc832b2e700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** DB Stats ** 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:29:38.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** Compaction Stats [default] ** 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** Compaction Stats [default] ** 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:38.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** Compaction Stats [default] ** 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:38.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** Compaction Stats [default] ** 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:29:38.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.256+0000 7fc84bd3d880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.257+0000 7fc84bd3d880 0 using public_addrv v1:172.21.15.107:6790/0 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.257+0000 7fc84bd3d880 0 starting mon.c rank -1 at public addrs v1:172.21.15.107:6790/0 at bind addrs v1:172.21.15.107:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.258+0000 7fc84bd3d880 1 mon.c@-1(???) e0 preinit fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.271+0000 7fc835333700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.271+0000 7fc835333700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: e1 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T08:29:38.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: legacy client fscid: -1 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: No filesystems configured 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.271+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc830b2a700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc83ab3e700 4 rocksdb: (Original Log Time 2024/02/10-08:29:38.272715) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc83ab3e700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc83ab3e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707553778272762, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1067, "num_deletes": 13, "total_data_size": 3624477, "memory_usage": 3644232, "flush_reason": "Manual Compaction"} 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc83ab3e700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T08:29:38.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:36.007123+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:29:38.272+0000 7fc835333700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 bash[74732]: fd6c6b2d3e6e75c206bedb22ca87dde63f06fa5bd34346e37f5d54a104b47478 2024-02-10T08:29:38.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:38 smithi107 systemd[1]: Started Ceph mon.c for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.301+0000 7f77783fd700 1 -- 172.21.15.195:0/3351359470 >> v1:172.21.15.107:6789/0 conn(0x7f7754003430 legacy=0x7f77540058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.301+0000 7f77783fd700 1 -- 172.21.15.195:0/3351359470 shutdown_connections 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.301+0000 7f77783fd700 1 -- 172.21.15.195:0/3351359470 >> 172.21.15.195:0/3351359470 conn(0x7f777006c390 msgr2=0x7f777006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 -- 172.21.15.195:0/3351359470 shutdown_connections 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 -- 172.21.15.195:0/3351359470 wait complete. 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 librados: starting msgr at 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 librados: starting objecter 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 Processor -- start 2024-02-10T08:29:43.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 -- start start 2024-02-10T08:29:43.304 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 librados: setting wanted keys 2024-02-10T08:29:43.304 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.302+0000 7f77783fd700 1 librados: calling monclient init 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77783fd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7770116f40 con 0x7f7770113250 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77783fd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7770118140 con 0x7f7754003430 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77783fd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7770119340 con 0x7f777010d420 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f7776199700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7754003430 0x7f777010cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50642/0 (socket says 172.21.15.195:50642) 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f7775998700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f777010d420 0x7f7770111b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.195:39498/0 (socket says 172.21.15.195:39498) 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f7776199700 1 -- 172.21.15.195:0/1737458692 learned_addr learned my addr 172.21.15.195:0/1737458692 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter ms_handle_connect 0x7f777010d420 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter ms_handle_connect 0x7f7754003430 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter ms_handle_connect 0x7f7770113250 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:43.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498461239 0 0) 0x7f7770116f40 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.303+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c003a90 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85002638 0 0) 0x7f775c003a90 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7770116f40 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7768001b40 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3498303887 0 0) 0x7f7770116f40 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 >> v1:172.21.15.107:6790/0 conn(0x7f777010d420 legacy=0x7f7770111b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 >> v1:172.21.15.195:6789/0 conn(0x7f7754003430 legacy=0x7f777010cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777011a540 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7770118330 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77783fd700 10 client.14187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701187e0 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77783fd700 1 librados: init done 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7768002690 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.304+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f77680035e0 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f776800ce60 con 0x7f7770113250 2024-02-10T08:29:43.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7f776803d240 con 0x7f7770113250 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 10 client.14187.objecter ms_dispatch 0x7f7770001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 3 client.14187.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 3 client.14187.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 20 client.14187.objecter dump_active .. 0 homeless 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77540069e0 con 0x7f7770113250 2024-02-10T08:29:43.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.305+0000 7f77677fe700 10 client.14187.objecter ms_handle_connect 0x7f775c035390 2024-02-10T08:29:43.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.310+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f776800e6c0 con 0x7f7770113250 2024-02-10T08:29:43.559 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.559+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7754006f40 con 0x7f7770113250 2024-02-10T08:29:43.560 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.559+0000 7f77677fe700 1 -- 172.21.15.195:0/1737458692 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1109 (unknown 409899127 0 1544112014) 0x7f77680107a0 con 0x7f7770113250 2024-02-10T08:29:43.560 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-10T08:29:43.560 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","modified":"2024-02-10T08:29:38.277410Z","created":"2024-02-10T08:27:37.431644Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6790","nonce":0}]},"addr":"172.21.15.107:6790/0","public_addr":"172.21.15.107:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 10 librados: watch_flush enter 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 10 librados: watch_flush exit 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 20 client.14187.objecter shutdown clearing up homeless session... 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 10 client.14187.objecter successfully canceled tick 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 >> v1:172.21.15.107:6800/3360068681 conn(0x7f775c035390 legacy=0x7f775c037850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 >> v1:172.21.15.107:6789/0 conn(0x7f7770113250 legacy=0x7f7770115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 shutdown_connections 2024-02-10T08:29:43.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 >> 172.21.15.195:0/1737458692 conn(0x7f777006c390 msgr2=0x7f77700fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:43.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 shutdown_connections 2024-02-10T08:29:43.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 -- 172.21.15.195:0/1737458692 wait complete. 2024-02-10T08:29:43.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:43.561+0000 7f77783fd700 1 librados: shutdown 2024-02-10T08:29:43.563 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 3 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:38.007485+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:38.281875+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:38.281993+0000 mon.a (mon.0) 150 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:38.282642+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:38.295624+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:38.295736+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:39.276634+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:40.007829+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:40.276828+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:41.277004+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:42.008167+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:42.277062+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:43.277239+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.295776+0000 mon.a (mon.0) 158 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.298837+0000 mon.a (mon.0) 159 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0,c=v1:172.21.15.107:6790/0} removed_ranks: {} 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.299344+0000 mon.a (mon.0) 160 : cluster [DBG] fsmap 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.299391+0000 mon.a (mon.0) 161 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.299595+0000 mon.a (mon.0) 162 : cluster [DBG] mgrmap e13: y(active, since 73s) 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.299760+0000 mon.a (mon.0) 163 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.313177+0000 mon.a (mon.0) 164 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.313195+0000 mon.a (mon.0) 165 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T08:29:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:43.313207+0000 mon.a (mon.0) 166 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.107:6790/0 is down (out of quorum) 2024-02-10T08:29:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:43.315499+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:43.316680+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:38.007485+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:38.281875+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:38.281993+0000 mon.a (mon.0) 150 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:38.282642+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:38.295624+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:29:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:38.295736+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:39.276634+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:40.007829+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:40.276828+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:41.277004+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:42.008167+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:42.277062+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:43.277239+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.295776+0000 mon.a (mon.0) 158 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.298837+0000 mon.a (mon.0) 159 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0,c=v1:172.21.15.107:6790/0} removed_ranks: {} 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.299344+0000 mon.a (mon.0) 160 : cluster [DBG] fsmap 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.299391+0000 mon.a (mon.0) 161 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.299595+0000 mon.a (mon.0) 162 : cluster [DBG] mgrmap e13: y(active, since 73s) 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.299760+0000 mon.a (mon.0) 163 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T08:29:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.313177+0000 mon.a (mon.0) 164 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T08:29:43.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.313195+0000 mon.a (mon.0) 165 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T08:29:43.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:43.313207+0000 mon.a (mon.0) 166 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.107:6790/0 is down (out of quorum) 2024-02-10T08:29:43.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:43.315499+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:43.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:43.316680+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:43.559956+0000 mon.a (mon.0) 169 : audit [DBG] from='client.? v1:172.21.15.195:0/1737458692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:44.277328+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:43.559956+0000 mon.a (mon.0) 169 : audit [DBG] from='client.? v1:172.21.15.195:0/1737458692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:44.277328+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:43.559956+0000 mon.a (mon.0) 169 : audit [DBG] from='client.? v1:172.21.15.195:0/1737458692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:29:45.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:44.277328+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:40.278233+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.295618+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.296020+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.296063+0000 mon.a (mon.0) 174 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.309121+0000 mon.a (mon.0) 175 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316165+0000 mon.a (mon.0) 176 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0,c=v1:172.21.15.107:6790/0} removed_ranks: {} 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316596+0000 mon.a (mon.0) 177 : cluster [DBG] fsmap 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316624+0000 mon.a (mon.0) 178 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316818+0000 mon.a (mon.0) 179 : cluster [DBG] mgrmap e13: y(active, since 75s) 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316962+0000 mon.a (mon.0) 180 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.316980+0000 mon.a (mon.0) 181 : cluster [INF] Cluster is now healthy 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:45.322024+0000 mon.a (mon.0) 182 : cluster [INF] overall HEALTH_OK 2024-02-10T08:29:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:40.278233+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.295618+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.296020+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.296063+0000 mon.a (mon.0) 174 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.309121+0000 mon.a (mon.0) 175 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316165+0000 mon.a (mon.0) 176 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0,c=v1:172.21.15.107:6790/0} removed_ranks: {} 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316596+0000 mon.a (mon.0) 177 : cluster [DBG] fsmap 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316624+0000 mon.a (mon.0) 178 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316818+0000 mon.a (mon.0) 179 : cluster [DBG] mgrmap e13: y(active, since 75s) 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316962+0000 mon.a (mon.0) 180 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.316980+0000 mon.a (mon.0) 181 : cluster [INF] Cluster is now healthy 2024-02-10T08:29:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:45.322024+0000 mon.a (mon.0) 182 : cluster [INF] overall HEALTH_OK 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:40.278233+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.295618+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.296020+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.296063+0000 mon.a (mon.0) 174 : cluster [INF] mon.a calling monitor election 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.309121+0000 mon.a (mon.0) 175 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316165+0000 mon.a (mon.0) 176 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.107:6789/0,b=v1:172.21.15.195:6789/0,c=v1:172.21.15.107:6790/0} removed_ranks: {} 2024-02-10T08:29:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316596+0000 mon.a (mon.0) 177 : cluster [DBG] fsmap 2024-02-10T08:29:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316624+0000 mon.a (mon.0) 178 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:29:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316818+0000 mon.a (mon.0) 179 : cluster [DBG] mgrmap e13: y(active, since 75s) 2024-02-10T08:29:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316962+0000 mon.a (mon.0) 180 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T08:29:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.316980+0000 mon.a (mon.0) 181 : cluster [INF] Cluster is now healthy 2024-02-10T08:29:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:45.322024+0000 mon.a (mon.0) 182 : cluster [INF] overall HEALTH_OK 2024-02-10T08:29:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:46.008904+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:46.277614+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:46.008904+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:46.277614+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:47.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:46.008904+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:47.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:46.277614+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:29:47.760 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T08:29:47.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T08:29:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:48.009355+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.254013+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.254568+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.255068+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:48.255776+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:48.255964+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.274544+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.280510+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.282996+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:48.283934+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.284137+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.284612+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:48.285073+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:48.285453+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi107 2024-02-10T08:29:49.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:48.009355+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.254013+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.254568+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.255068+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:48.255776+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:48.255964+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.274544+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.280510+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.282996+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:48.283934+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.284137+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.284612+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:48.285073+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:48.285453+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi107 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:48.009355+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.254013+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.254568+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.255068+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:48.255776+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:48.255964+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.274544+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.280510+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.282996+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:48.283934+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.284137+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.284612+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:48.285073+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:49.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:48.285453+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi107 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 -- 172.21.15.107:0/2087326682 >> v1:172.21.15.107:6789/0 conn(0x7fe6040fc820 legacy=0x7fe6040fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 -- 172.21.15.107:0/2087326682 shutdown_connections 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 -- 172.21.15.107:0/2087326682 >> 172.21.15.107:0/2087326682 conn(0x7fe60406c800 msgr2=0x7fe60406cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 -- 172.21.15.107:0/2087326682 shutdown_connections 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 -- 172.21.15.107:0/2087326682 wait complete. 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 librados: starting msgr at 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.692+0000 7fe60a3df700 1 librados: starting objecter 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 Processor -- start 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 -- start start 2024-02-10T08:29:49.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 librados: setting wanted keys 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 librados: calling monclient init 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe604117110 con 0x7fe604103160 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6041182f0 con 0x7fe6040fc820 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60a3df700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6041194f0 con 0x7fe6040ff620 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6037fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe6040ff620 0x7fe6040f9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46456/0 (socket says 172.21.15.107:46456) 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe60897c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe604103160 0x7fe6041157f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58850/0 (socket says 172.21.15.107:58850) 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6037fe700 1 -- 172.21.15.107:0/2049431446 learned_addr learned my addr 172.21.15.107:0/2049431446 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter ms_handle_connect 0x7fe6040ff620 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter ms_handle_connect 0x7fe604103160 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter ms_handle_connect 0x7fe6040fc820 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274826407 0 0) 0x7fe604117110 con 0x7fe604103160 2024-02-10T08:29:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec003030 con 0x7fe604103160 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84662709 0 0) 0x7fe6041194f0 con 0x7fe6040ff620 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604117110 con 0x7fe6040ff620 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.693+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136510508 0 0) 0x7fe6041182f0 con 0x7fe6040fc820 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6041194f0 con 0x7fe6040fc820 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2144392251 0 0) 0x7fe5ec003030 con 0x7fe604103160 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6041182f0 con 0x7fe604103160 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe5fc001b40 con 0x7fe604103160 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465302538 0 0) 0x7fe604117110 con 0x7fe6040ff620 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5ec003030 con 0x7fe6040ff620 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe5f4001970 con 0x7fe6040ff620 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428714638 0 0) 0x7fe6041194f0 con 0x7fe6040fc820 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe604117110 con 0x7fe6040fc820 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe5f8002020 con 0x7fe6040fc820 2024-02-10T08:29:49.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2206173535 0 0) 0x7fe6041182f0 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 >> v1:172.21.15.107:6790/0 conn(0x7fe6040ff620 legacy=0x7fe6040f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 >> v1:172.21.15.195:6789/0 conn(0x7fe6040fc820 legacy=0x7fe6040f96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60411a6f0 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe604117300 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe60a3df700 10 client.14193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6041177f0 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe60a3df700 1 librados: init done 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe5fc002620 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.694+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe5fc005d20 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7fe5fc00f660 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7fe5fc03fa80 con 0x7fe604103160 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 10 client.14193.objecter ms_dispatch 0x7fe604001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 3 client.14193.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 3 client.14193.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:49.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 20 client.14193.objecter dump_active .. 0 homeless 2024-02-10T08:29:49.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0004f20 con 0x7fe604103160 2024-02-10T08:29:49.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.695+0000 7fe6017fa700 10 client.14193.objecter ms_handle_connect 0x7fe5ec0353a0 2024-02-10T08:29:49.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.700+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe5fc0110b0 con 0x7fe604103160 2024-02-10T08:29:49.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.870+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe5f00051d0 con 0x7fe604103160 2024-02-10T08:29:49.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.870+0000 7fe6017fa700 1 -- 172.21.15.107:0/2049431446 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+193 (unknown 2189217731 0 898050140) 0x7fe5fc012fe0 con 0x7fe604103160 2024-02-10T08:29:49.871 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:49.871 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-02-10T08:29:49.871 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:29:49.871 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = 172.21.15.107:6789/0 172.21.15.195:6789/0 v1:172.21.15.107:6790/0 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 10 librados: watch_flush enter 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 10 librados: watch_flush exit 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 20 client.14193.objecter shutdown clearing up homeless session... 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 10 client.14193.objecter successfully canceled tick 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 >> v1:172.21.15.107:6800/3360068681 conn(0x7fe5ec0353a0 legacy=0x7fe5ec037820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 >> v1:172.21.15.107:6789/0 conn(0x7fe604103160 legacy=0x7fe6041157f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 shutdown_connections 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 >> 172.21.15.107:0/2049431446 conn(0x7fe60406c800 msgr2=0x7fe604060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 shutdown_connections 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 -- 172.21.15.107:0/2049431446 wait complete. 2024-02-10T08:29:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:29:49.872+0000 7fe60a3df700 1 librados: shutdown 2024-02-10T08:29:50.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:49.871157+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2049431446' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:49.871157+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2049431446' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:49.871157+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2049431446' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:51.151 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T08:29:51.151 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:29:51.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T08:29:51.181 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:29:51.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:29:51.248 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:29:51.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T08:29:51.284 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:29:51.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:29:51.355 INFO:tasks.cephadm:Adding mgr.y on smithi107 2024-02-10T08:29:51.356 INFO:tasks.cephadm:Adding mgr.x on smithi195 2024-02-10T08:29:51.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi107=y;smithi195=x' 2024-02-10T08:29:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:50.009776+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:50.863291+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:50.864048+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:50.864590+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:50.865111+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:50.009776+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:50.863291+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:50.864048+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:50.864590+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:50.865111+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:50.009776+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:50.863291+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:50.864048+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:50.864590+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:50.865111+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:50.863824+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T08:29:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:50.865569+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi107 2024-02-10T08:29:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:52.421718+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:52.431550+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:52.432047+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:52.432484+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:50.863824+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:50.865569+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi107 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:52.421718+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:52.431550+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:52.432047+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:52.432484+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:50.863824+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:50.865569+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi107 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:52.421718+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:52.431550+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:52.432047+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T08:29:52.756 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:52.432484+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:52.010147+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:52.431351+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T08:29:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:52.432891+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi195 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:52.010147+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:52.431351+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:52.432891+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi195 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:52.010147+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:52.431351+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-10T08:29:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:52.432891+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi195 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- 172.21.15.195:0/1561780883 >> v1:172.21.15.195:6789/0 conn(0x7feb400ff9f0 legacy=0x7feb40101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- 172.21.15.195:0/1561780883 shutdown_connections 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- 172.21.15.195:0/1561780883 >> 172.21.15.195:0/1561780883 conn(0x7feb400f3050 msgr2=0x7feb400f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- 172.21.15.195:0/1561780883 shutdown_connections 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- 172.21.15.195:0/1561780883 wait complete. 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 librados: starting msgr at 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 librados: starting objecter 2024-02-10T08:29:54.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 Processor -- start 2024-02-10T08:29:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 -- start start 2024-02-10T08:29:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 librados: setting wanted keys 2024-02-10T08:29:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.412+0000 7feb47a5b700 1 librados: calling monclient init 2024-02-10T08:29:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb40114d90 con 0x7feb400f78f0 2024-02-10T08:29:54.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb40115f70 con 0x7feb400ff9f0 2024-02-10T08:29:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb40117170 con 0x7feb400fbdc0 2024-02-10T08:29:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb45ff8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feb400ff9f0 0x7feb40113450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:57320/0 (socket says 172.21.15.195:57320) 2024-02-10T08:29:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb45ff8700 1 -- 172.21.15.195:0/3835508561 learned_addr learned my addr 172.21.15.195:0/3835508561 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:29:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb457f7700 1 --1- 172.21.15.195:0/3835508561 >> v1:172.21.15.107:6789/0 conn(0x7feb400f78f0 0x7feb4006ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:52650/0 (socket says 172.21.15.195:52650) 2024-02-10T08:29:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb44ff6700 1 --1- 172.21.15.195:0/3835508561 >> v1:172.21.15.107:6790/0 conn(0x7feb400fbdc0 0x7feb4010fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.195:58830/0 (socket says 172.21.15.195:58830) 2024-02-10T08:29:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter ms_handle_connect 0x7feb400ff9f0 2024-02-10T08:29:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter ms_handle_connect 0x7feb400fbdc0 2024-02-10T08:29:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:54.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter ms_handle_connect 0x7feb400f78f0 2024-02-10T08:29:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:29:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872306689 0 0) 0x7feb40115f70 con 0x7feb400ff9f0 2024-02-10T08:29:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c0033f0 con 0x7feb400ff9f0 2024-02-10T08:29:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010513240 0 0) 0x7feb40117170 con 0x7feb400fbdc0 2024-02-10T08:29:54.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40115f70 con 0x7feb400fbdc0 2024-02-10T08:29:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586025669 0 0) 0x7feb40114d90 con 0x7feb400f78f0 2024-02-10T08:29:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40117170 con 0x7feb400f78f0 2024-02-10T08:29:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 40168360 0 0) 0x7feb2c0033f0 con 0x7feb400ff9f0 2024-02-10T08:29:54.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb40114d90 con 0x7feb400ff9f0 2024-02-10T08:29:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7feb38003280 con 0x7feb400ff9f0 2024-02-10T08:29:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902160301 0 0) 0x7feb40115f70 con 0x7feb400fbdc0 2024-02-10T08:29:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb2c0033f0 con 0x7feb400fbdc0 2024-02-10T08:29:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7feb30002890 con 0x7feb400fbdc0 2024-02-10T08:29:54.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615889663 0 0) 0x7feb40117170 con 0x7feb400f78f0 2024-02-10T08:29:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb40115f70 con 0x7feb400f78f0 2024-02-10T08:29:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7feb3c001950 con 0x7feb400f78f0 2024-02-10T08:29:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 429518558 0 0) 0x7feb40114d90 con 0x7feb400ff9f0 2024-02-10T08:29:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 >> v1:172.21.15.107:6790/0 conn(0x7feb400fbdc0 legacy=0x7feb4010fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:54.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 >> v1:172.21.15.107:6789/0 conn(0x7feb400f78f0 legacy=0x7feb4006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40118370 con 0x7feb400ff9f0 2024-02-10T08:29:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb40117360 con 0x7feb400ff9f0 2024-02-10T08:29:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 10 client.24109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:29:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb40117850 con 0x7feb400ff9f0 2024-02-10T08:29:54.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb47a5b700 1 librados: init done 2024-02-10T08:29:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7feb380028e0 con 0x7feb400ff9f0 2024-02-10T08:29:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.413+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7feb38003e40 con 0x7feb400ff9f0 2024-02-10T08:29:54.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7feb38004040 con 0x7feb400ff9f0 2024-02-10T08:29:54.252 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mgr update... 2024-02-10T08:29:54.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7feb3803f120 con 0x7feb400ff9f0 2024-02-10T08:29:54.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 10 client.24109.objecter ms_dispatch 0x7feb40001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:29:54.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 3 client.24109.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:29:54.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 3 client.24109.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:29:54.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24004f20 con 0x7feb400ff9f0 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.414+0000 7feb367fc700 20 client.24109.objecter dump_active .. 0 homeless 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.415+0000 7feb367fc700 10 client.24109.objecter ms_handle_connect 0x7feb2c035470 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.419+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7feb38015160 con 0x7feb400ff9f0 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.596+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi195=x", "target": ["mon-mgr", ""]}) v1 -- 0x7feb24000b90 con 0x7feb2c035470 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.602+0000 7feb367fc700 1 -- 172.21.15.195:0/3835508561 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7feb24000b90 con 0x7feb2c035470 2024-02-10T08:29:54.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 10 librados: watch_flush enter 2024-02-10T08:29:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 10 librados: watch_flush exit 2024-02-10T08:29:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 20 client.24109.objecter shutdown clearing up homeless session... 2024-02-10T08:29:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 10 client.24109.objecter successfully canceled tick 2024-02-10T08:29:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 >> v1:172.21.15.107:6800/3360068681 conn(0x7feb2c035470 legacy=0x7feb2c037930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 >> v1:172.21.15.195:6789/0 conn(0x7feb400ff9f0 legacy=0x7feb40113450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:29:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 shutdown_connections 2024-02-10T08:29:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 >> 172.21.15.195:0/3835508561 conn(0x7feb400f3050 msgr2=0x7feb400f3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:29:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 shutdown_connections 2024-02-10T08:29:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 -- 172.21.15.195:0/3835508561 wait complete. 2024-02-10T08:29:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:29:53.604+0000 7feb47a5b700 1 librados: shutdown 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:53.598575+0000 mgr.y (mgr.14146) 57 : audit [DBG] from='client.24109 v1:172.21.15.195:0/3835508561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:53.599955+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Saving service mgr spec with placement smithi107=y;smithi195=x;count:2 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:53.602889+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.122412+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.123271+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.124931+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.734 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.125681+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:54.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.129709+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.131073+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:54.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.133950+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T08:29:54.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.134609+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:54.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:54.135462+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.736 DEBUG:teuthology.orchestra.run.smithi195:mgr.x> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T08:29:54.739 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T08:29:54.739 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:29:54.739 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T08:29:54.761 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T08:29:54.761 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-02-10T08:29:54.766 INFO:journalctl@ceph.mgr.x.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 192078 Links: 1 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:54.822 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-02-10 08:29:37.723488437 +0000 2024-02-10T08:29:54.823 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-02-10 08:22:04.743893773 +0000 2024-02-10T08:29:54.823 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-02-10 08:22:04.743893773 +0000 2024-02-10T08:29:54.823 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2024-02-10T08:29:54.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T08:29:54.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:53.598575+0000 mgr.y (mgr.14146) 57 : audit [DBG] from='client.24109 v1:172.21.15.195:0/3835508561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:29:54.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:53.599955+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Saving service mgr spec with placement smithi107=y;smithi195=x;count:2 2024-02-10T08:29:54.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:53.602889+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.122412+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.890 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.123271+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.124931+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.125681+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.129709+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.131073+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.133950+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.134609+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:54.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:54.135462+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:53.598575+0000 mgr.y (mgr.14146) 57 : audit [DBG] from='client.24109 v1:172.21.15.195:0/3835508561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:53.599955+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Saving service mgr spec with placement smithi107=y;smithi195=x;count:2 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:53.602889+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.122412+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.123271+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.124931+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.125681+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.129709+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.131073+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.133950+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.134609+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:54.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:54.135462+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:54.893 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-02-10T08:29:54.893 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-02-10T08:29:54.893 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000214816 s, 2.4 MB/s 2024-02-10T08:29:54.894 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T08:29:54.955 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-02-10T08:29:55.014 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T08:29:55.014 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 192134 Links: 1 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-02-10 08:29:37.723488437 +0000 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-02-10 08:22:05.136887379 +0000 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-02-10 08:22:05.136887379 +0000 2024-02-10T08:29:55.015 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2024-02-10T08:29:55.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T08:29:55.082 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-02-10T08:29:55.082 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-02-10T08:29:55.082 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000212403 s, 2.4 MB/s 2024-02-10T08:29:55.083 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T08:29:55.142 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-02-10T08:29:55.200 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T08:29:55.200 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 192195 Links: 1 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-02-10 08:29:37.723488437 +0000 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-02-10 08:22:05.479881799 +0000 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-02-10 08:22:05.479881799 +0000 2024-02-10T08:29:55.201 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2024-02-10T08:29:55.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T08:29:55.267 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-02-10T08:29:55.267 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-02-10T08:29:55.267 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00019169 s, 2.7 MB/s 2024-02-10T08:29:55.268 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T08:29:55.327 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-02-10T08:29:55.385 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T08:29:55.385 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.385 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 199169 Links: 1 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-02-10 08:29:37.723488437 +0000 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-02-10 08:22:05.829876118 +0000 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-02-10 08:22:05.829876118 +0000 2024-02-10T08:29:55.386 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2024-02-10T08:29:55.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T08:29:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-02-10T08:29:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-02-10T08:29:55.451 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210942 s, 2.4 MB/s 2024-02-10T08:29:55.452 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T08:29:55.511 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:29:55.511 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T08:29:55.529 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T08:29:55.529 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 191406 Links: 1 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-10 08:29:28.093657268 +0000 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-10 08:22:03.636192566 +0000 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-10 08:22:03.636192566 +0000 2024-02-10T08:29:55.588 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-02-10T08:29:55.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T08:29:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-10T08:29:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-10T08:29:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000106476 s, 4.8 MB/s 2024-02-10T08:29:55.656 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T08:29:55.715 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-02-10T08:29:55.772 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 192485 Links: 1 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-10 08:29:28.093657268 +0000 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-10 08:22:03.976186175 +0000 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-10 08:22:03.976186175 +0000 2024-02-10T08:29:55.773 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-02-10T08:29:55.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T08:29:55.839 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-10T08:29:55.839 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-10T08:29:55.839 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.8407e-05 s, 5.2 MB/s 2024-02-10T08:29:55.840 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T08:29:55.899 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-02-10T08:29:55.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:54.010430+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:55.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:54.136121+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Deploying daemon mgr.x on smithi195 2024-02-10T08:29:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:54.010430+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:54.136121+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Deploying daemon mgr.x on smithi195 2024-02-10T08:29:55.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:54.010430+0000 mgr.y (mgr.14146) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:55.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:54.136121+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Deploying daemon mgr.x on smithi195 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 197681 Links: 1 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:55.957 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-10 08:29:28.093657268 +0000 2024-02-10T08:29:55.958 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-10 08:22:04.326179596 +0000 2024-02-10T08:29:55.958 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-10 08:22:04.326179596 +0000 2024-02-10T08:29:55.958 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-02-10T08:29:55.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T08:29:56.058 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-10T08:29:56.058 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-10T08:29:56.058 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.00012932 s, 4.0 MB/s 2024-02-10T08:29:56.059 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T08:29:56.077 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 197788 Links: 1 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-02-10 08:29:56.012127828 +0000 2024-02-10T08:29:56.135 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-02-10 08:22:04.675173036 +0000 2024-02-10T08:29:56.136 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-02-10 08:22:04.675173036 +0000 2024-02-10T08:29:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-02-10T08:29:56.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T08:29:56.201 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:56 smithi195 systemd[1]: Starting Ceph mgr.x for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:29:56.203 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-02-10T08:29:56.204 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-02-10T08:29:56.204 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000108586 s, 4.7 MB/s 2024-02-10T08:29:56.205 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T08:29:56.264 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/vg_nvme/lv_4... 2024-02-10T08:29:56.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T08:29:57.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:56 smithi195 bash[71020]: 5ee731709ae6d3e9f1294c7bcd253e7298c0151a47f0812bdffd9f3f741b0720 2024-02-10T08:29:57.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:56 smithi195 systemd[1]: Started Ceph mgr.x for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:29:57.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:56.869+0000 7ff0837d8680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:29:57.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:56.953+0000 7ff0837d8680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:29:57.616 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:57.355+0000 7ff0837d8680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:29:57.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:56.010807+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:57.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:56.860904+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:57.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:56.862296+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:57.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:56.010807+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:57.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:56.860904+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:57.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:56.862296+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:56.010807+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:56.860904+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:56.862296+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:29:58.415 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:58.124+0000 7ff0837d8680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:29:58.415 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:58.207+0000 7ff0837d8680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:29:58.415 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:58.414+0000 7ff0837d8680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148349 s, 707 MB/s 2024-02-10T08:29:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-02-10T08:29:59.146 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.962158+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.962696+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.963217+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.966416+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:57.967671+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.967860+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.968439+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:57.968860+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:29:57.969262+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi107 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:29:58.011072+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.962158+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.962696+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.963217+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.966416+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:57.967671+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.967860+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.968439+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:57.968860+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:29:57.969262+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi107 2024-02-10T08:29:59.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:29:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:29:58.011072+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:59.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-02-10T08:29:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.962158+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.962696+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.963217+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:29:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.966416+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:29:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:57.967671+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-10T08:29:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.967860+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T08:29:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.968439+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T08:29:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:57.968860+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:29:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:29:57.969262+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi107 2024-02-10T08:29:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:29:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:29:58.011072+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:29:59.639 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:59.349+0000 7ff0837d8680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:29:59.639 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:59.473+0000 7ff0837d8680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:29:59.639 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:59.551+0000 7ff0837d8680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:29:59.639 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:59.638+0000 7ff0837d8680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:30:00.037 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:29:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:29:59.831+0000 7ff0837d8680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:30:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:00.106+0000 7ff0837d8680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:30:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:00.204+0000 7ff0837d8680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:30:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:59.871762+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:59.872585+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:59.873670+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:59.874138+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:29:59.877883+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:00.000145+0000 mon.a (mon.0) 226 : cluster [INF] overall HEALTH_OK 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:00.011428+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:00.139622+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:59.871762+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:59.872585+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:59.873670+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:59.874138+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:29:59.877883+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:00.000145+0000 mon.a (mon.0) 226 : cluster [INF] overall HEALTH_OK 2024-02-10T08:30:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:00.011428+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:00.139622+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:00.921+0000 7ff0837d8680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.001+0000 7ff0837d8680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.092+0000 7ff0837d8680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.263+0000 7ff0837d8680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:59.871762+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:59.872585+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:01.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:59.873670+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:01.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:59.874138+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:01.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:29:59.877883+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:01.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:00.000145+0000 mon.a (mon.0) 226 : cluster [INF] overall HEALTH_OK 2024-02-10T08:30:01.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:00.011428+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:01.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:00.139622+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.631+0000 7f53226a8700 1 -- 172.21.15.107:0/3503555501 >> v1:172.21.15.107:6790/0 conn(0x7f531c0fbda0 legacy=0x7f531c0fe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.631+0000 7f53226a8700 1 -- 172.21.15.107:0/3503555501 shutdown_connections 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.631+0000 7f53226a8700 1 -- 172.21.15.107:0/3503555501 >> 172.21.15.107:0/3503555501 conn(0x7f531c0f3050 msgr2=0x7f531c0f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 -- 172.21.15.107:0/3503555501 shutdown_connections 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 -- 172.21.15.107:0/3503555501 wait complete. 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 librados: starting msgr at 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 librados: starting objecter 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 Processor -- start 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 -- start start 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 librados: setting wanted keys 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.632+0000 7f53226a8700 1 librados: calling monclient init 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53226a8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f531c108440 con 0x7f531c0fbda0 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53226a8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f531c113ca0 con 0x7f531c0f78d0 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53226a8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f531c114e80 con 0x7f531c0ff9d0 2024-02-10T08:30:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f5320c45700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f531c0ff9d0 0x7f531c111570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52798/0 (socket says 172.21.15.107:52798) 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f5320c45700 1 -- 172.21.15.107:0/2301245417 learned_addr learned my addr 172.21.15.107:0/2301245417 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f531b7fe700 1 --1- 172.21.15.107:0/2301245417 >> v1:172.21.15.107:6789/0 conn(0x7f531c0fbda0 0x7f531c10de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38370/0 (socket says 172.21.15.107:38370) 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter ms_handle_connect 0x7f531c0ff9d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter ms_handle_connect 0x7f531c0fbda0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter ms_handle_connect 0x7f531c0f78d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493462048 0 0) 0x7f531c114e80 con 0x7f531c0ff9d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53040033f0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448909655 0 0) 0x7f531c108440 con 0x7f531c0fbda0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c114e80 con 0x7f531c0fbda0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014221533 0 0) 0x7f531c113ca0 con 0x7f531c0f78d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c108440 con 0x7f531c0f78d0 2024-02-10T08:30:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952848981 0 0) 0x7f53040033f0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f531c113ca0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5314001cd0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809152375 0 0) 0x7f531c108440 con 0x7f531c0f78d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53040033f0 con 0x7f531c0f78d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206968530 0 0) 0x7f531c114e80 con 0x7f531c0fbda0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f531c108440 con 0x7f531c0fbda0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.633+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f530c001840 con 0x7f531c0f78d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5310000fe0 con 0x7f531c0fbda0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642752979 0 0) 0x7f531c113ca0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 >> v1:172.21.15.195:6789/0 conn(0x7f531c0f78d0 legacy=0x7f531c107870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 >> v1:172.21.15.107:6789/0 conn(0x7f531c0fbda0 legacy=0x7f531c10de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c116060 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c112cf0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53226a8700 10 client.34121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:30:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c1131e0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53226a8700 1 librados: init done 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f53140026c0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5314005d30 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.634+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 2399087135 0 0) 0x7f531400f670 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 3852152495 0 0) 0x7f531403faf0 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 10 client.34121.objecter ms_dispatch 0x7f531c001550 osd_map(4..4 src has 1..4) v4 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 3 client.34121.objecter handle_osd_map got epochs [4,4] > 0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 3 client.34121.objecter handle_osd_map decoding full epoch 4 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 20 client.34121.objecter dump_active .. 0 homeless 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5308004f20 con 0x7f531c0ff9d0 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.635+0000 7f53197fa700 10 client.34121.objecter ms_handle_connect 0x7f5304035420 2024-02-10T08:30:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.639+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5314011070 con 0x7f531c0ff9d0 2024-02-10T08:30:01.784 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.521+0000 7ff0837d8680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:30:01.784 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.652+0000 7ff0837d8680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:30:01.784 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.782+0000 7ff0837d8680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:30:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:01.809+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5308000b90 con 0x7f5304035420 2024-02-10T08:30:02.044 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:01.952+0000 7ff0837d8680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:30:02.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:01.812830+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:02.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:01.814603+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:02.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:01.815052+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:02.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:01.812830+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:02.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:01.814603+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:02.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:01.815052+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:02.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:01.812830+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:02.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:01.814603+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:02.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:01.815052+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:02.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:30:02.034+0000 7ff0837d8680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:30:02.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:02.888+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f531403ec60 con 0x7f531c0ff9d0 2024-02-10T08:30:03.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:01.810789+0000 mgr.y (mgr.14146) 66 : audit [DBG] from='client.34121 v1:172.21.15.107:0/2301245417' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:03.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:02.011711+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:03.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:02.049278+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:30:03.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:02.050178+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:02.050449+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:02.051275+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:02.051485+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:01.810789+0000 mgr.y (mgr.14146) 66 : audit [DBG] from='client.34121 v1:172.21.15.107:0/2301245417' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:02.011711+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:02.049278+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:02.050178+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:02.050449+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:02.051275+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:30:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:02.051485+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:01.810789+0000 mgr.y (mgr.14146) 66 : audit [DBG] from='client.34121 v1:172.21.15.107:0/2301245417' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:02.011711+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:02.049278+0000 mon.a (mon.0) 231 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:02.050178+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:02.050449+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:02.051275+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:30:03.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:02.051485+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.195:0/4176640445' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:30:04.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:02.888003+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 92s), standbys: x 2024-02-10T08:30:04.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:02.888109+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:30:04.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:02.888003+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 92s), standbys: x 2024-02-10T08:30:04.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:02.888109+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:30:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:02.888003+0000 mon.a (mon.0) 232 : cluster [DBG] mgrmap e14: y(active, since 92s), standbys: x 2024-02-10T08:30:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:02.888109+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:30:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:04.011929+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:04.145435+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]: dispatch 2024-02-10T08:30:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:04.153819+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]': finished 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:04.153868+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:04.153956+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:04.712713+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.107:0/2570587947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:04.011929+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:04.145435+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]: dispatch 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:04.153819+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]': finished 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:04.153868+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:04.153956+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:05.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:04.712713+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.107:0/2570587947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:04.011929+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:04.145435+0000 mon.a (mon.0) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]: dispatch 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:04.153819+0000 mon.a (mon.0) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1035035990' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167"}]': finished 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:04.153868+0000 mon.a (mon.0) 236 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:04.153956+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:04.712713+0000 mon.a (mon.0) 238 : audit [DBG] from='client.? v1:172.21.15.107:0/2570587947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:06.635+0000 7f5321ea7700 10 client.34121.objecter tick 2024-02-10T08:30:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:06.012278+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:07.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:06.012278+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:07.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:06.012278+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:09.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:08.012754+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:09.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:08.012754+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:08.012754+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:10.383445+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T08:30:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:10.384072+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:10.383445+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T08:30:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:10.384072+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:10.383445+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T08:30:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:10.384072+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:11.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:11.635+0000 7f5321ea7700 10 client.34121.objecter tick 2024-02-10T08:30:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:10.013144+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:10.384529+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi107 2024-02-10T08:30:11.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:10.013144+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:11.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:10.384529+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi107 2024-02-10T08:30:11.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:10.013144+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:11.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:10.384529+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi107 2024-02-10T08:30:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:12.013464+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:12.013464+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:12.013464+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:15.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:14.013734+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:15.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:15.424893+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:15.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:15.427776+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:15.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:14.013734+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:15.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:15.424893+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:15.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:15.427776+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:14.013734+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:15.424893+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:15.427776+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:16.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:16.636+0000 7f5321ea7700 10 client.34121.objecter tick 2024-02-10T08:30:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:16.013943+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:17.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:16.013943+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:16.013943+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.536+0000 7f53197fa700 1 -- 172.21.15.107:0/2301245417 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1349577908) 0x7f5308000b90 con 0x7f5304035420 2024-02-10T08:30:18.538 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-02-10T08:30:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 10 librados: watch_flush enter 2024-02-10T08:30:18.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 10 librados: watch_flush exit 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 20 client.34121.objecter shutdown clearing up homeless session... 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 10 client.34121.objecter successfully canceled tick 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 >> v1:172.21.15.107:6800/3360068681 conn(0x7f5304035420 legacy=0x7f53040378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 >> v1:172.21.15.107:6790/0 conn(0x7f531c0ff9d0 legacy=0x7f531c111570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 shutdown_connections 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 >> 172.21.15.107:0/2301245417 conn(0x7f531c0f3050 msgr2=0x7f531c0f3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:30:18.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 shutdown_connections 2024-02-10T08:30:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.539+0000 7f53226a8700 1 -- 172.21.15.107:0/2301245417 wait complete. 2024-02-10T08:30:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:18.540+0000 7f53226a8700 1 librados: shutdown 2024-02-10T08:30:19.185 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T08:30:19.187 INFO:tasks.cephadm:Deploying osd.1 on smithi107 with /dev/vg_nvme/lv_3... 2024-02-10T08:30:19.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T08:30:19.216 INFO:journalctl@ceph.osd.0.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:18.014270+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:18.537412+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:18.559345+0000 mon.b (mon.1) 8 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:18.559810+0000 mon.a (mon.0) 244 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:18.586243+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:19.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:18.014270+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:19.777 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:18.537412+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:19.777 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:18.559345+0000 mon.b (mon.1) 8 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:19.777 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:18.559810+0000 mon.a (mon.0) 244 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:19.777 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:18.586243+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:18.014270+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:18.537412+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:18.559345+0000 mon.b (mon.1) 8 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:18.559810+0000 mon.a (mon.0) 244 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T08:30:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:18.586243+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:20.930 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0[78238]: debug 2024-02-10T08:30:20.545+0000 7faac1cab700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T08:30:20.931 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0[78238]: debug 2024-02-10T08:30:20.547+0000 7faabdc26700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:19.539201+0000 mon.a (mon.0) 246 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:19.539266+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:19.539604+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:19.539963+0000 mon.b (mon.1) 9 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:19.544489+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:19.539201+0000 mon.a (mon.0) 246 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:19.539266+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-10T08:30:20.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:19.539604+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:20.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:19.539963+0000 mon.b (mon.1) 9 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:20.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:19.544489+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:19.539201+0000 mon.a (mon.0) 246 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T08:30:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:19.539266+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-10T08:30:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:19.539604+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:19.539963+0000 mon.b (mon.1) 9 : audit [INF] from='osd.0 v1:172.21.15.107:6801/3321596972' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:19.544489+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:20.014632+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:20.543680+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:20.543763+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:20.544080+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:20.547320+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:20.014632+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:21.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:20.543680+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:21.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:20.543763+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-10T08:30:21.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:20.544080+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:21.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:20.547320+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:20.014632+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:20.543680+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:20.543763+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-10T08:30:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:20.544080+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:20.547320+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112037 s, 936 MB/s 2024-02-10T08:30:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-02-10T08:30:22.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:19.564047+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:22.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:19.564104+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:22.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:21.548495+0000 mon.a (mon.0) 254 : cluster [INF] osd.0 v1:172.21.15.107:6801/3321596972 boot 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:21.548533+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:21.548858+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:19.564047+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:19.564104+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:21.548495+0000 mon.a (mon.0) 254 : cluster [INF] osd.0 v1:172.21.15.107:6801/3321596972 boot 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:21.548533+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-10T08:30:22.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:21.548858+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:22.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-02-10T08:30:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:19.564047+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:19.564104+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:21.548495+0000 mon.a (mon.0) 254 : cluster [INF] osd.0 v1:172.21.15.107:6801/3321596972 boot 2024-02-10T08:30:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:21.548533+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-10T08:30:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:21.548858+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:30:23.808 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:22.014907+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:23.808 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:22.562321+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-10T08:30:23.808 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:22.014907+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:23.808 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:22.562321+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-10T08:30:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:22.014907+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:30:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:22.562321+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-10T08:30:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:23.843024+0000 mgr.y (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.850152+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.850943+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:23.851363+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 16098M 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.857352+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.858218+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.858767+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.864622+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:23.866281+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:24.015178+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:24.155173+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:24.155774+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:24.161963+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:23.843024+0000 mgr.y (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.850152+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.850943+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:23.851363+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 16098M 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.857352+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.858218+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.858767+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.864622+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:23.866281+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:24.015178+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:24.155173+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:24.155774+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:24.161963+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.183+0000 7f84a5a85700 1 -- 172.21.15.107:0/4090418255 >> v1:172.21.15.107:6790/0 conn(0x7f84a00fac90 legacy=0x7f84a00fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.183+0000 7f84a5a85700 1 -- 172.21.15.107:0/4090418255 shutdown_connections 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.183+0000 7f84a5a85700 1 -- 172.21.15.107:0/4090418255 >> 172.21.15.107:0/4090418255 conn(0x7f84a00f1f00 msgr2=0x7f84a00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.183+0000 7f84a5a85700 1 -- 172.21.15.107:0/4090418255 shutdown_connections 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.183+0000 7f84a5a85700 1 -- 172.21.15.107:0/4090418255 wait complete. 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 librados: starting msgr at 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 librados: starting objecter 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 Processor -- start 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 -- start start 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 librados: setting wanted keys 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.184+0000 7f84a5a85700 1 librados: calling monclient init 2024-02-10T08:30:25.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84a01074f0 con 0x7f84a00fac90 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84a0112d50 con 0x7f84a00f67c0 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84a0113f30 con 0x7f84a00fe8c0 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f84a00fac90 0x7f84a010cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34870/0 (socket says 172.21.15.107:34870) 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849effd700 1 -- 172.21.15.107:0/4013891351 learned_addr learned my addr 172.21.15.107:0/4013891351 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849ffff700 1 --1- 172.21.15.107:0/4013891351 >> v1:172.21.15.107:6790/0 conn(0x7f84a00fe8c0 0x7f84a0110620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35002/0 (socket says 172.21.15.107:35002) 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter ms_handle_connect 0x7f84a00fac90 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter ms_handle_connect 0x7f84a00fe8c0 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter ms_handle_connect 0x7f84a00f67c0 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066449987 0 0) 0x7f84a01074f0 con 0x7f84a00fac90 2024-02-10T08:30:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488003030 con 0x7f84a00fac90 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862217493 0 0) 0x7f84a0113f30 con 0x7f84a00fe8c0 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a01074f0 con 0x7f84a00fe8c0 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116379629 0 0) 0x7f84a0112d50 con 0x7f84a00f67c0 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a0113f30 con 0x7f84a00f67c0 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027110180 0 0) 0x7f8488003030 con 0x7f84a00fac90 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a0112d50 con 0x7f84a00fac90 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8494002d70 con 0x7f84a00fac90 2024-02-10T08:30:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1472744419 0 0) 0x7f84a01074f0 con 0x7f84a00fe8c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8488003030 con 0x7f84a00fe8c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960047290 0 0) 0x7f84a0113f30 con 0x7f84a00f67c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a01074f0 con 0x7f84a00f67c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8498001b00 con 0x7f84a00fe8c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8490001860 con 0x7f84a00f67c0 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2180024236 0 0) 0x7f84a0112d50 con 0x7f84a00fac90 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 >> v1:172.21.15.107:6790/0 conn(0x7f84a00fe8c0 legacy=0x7f84a0110620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 >> v1:172.21.15.195:6789/0 conn(0x7f84a00f67c0 legacy=0x7f84a0106920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a0115110 con 0x7f84a00fac90 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a0114120 con 0x7f84a00fac90 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 10 client.14238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:30:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a0114610 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.185+0000 7f84a5a85700 1 librados: init done 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.186+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8494002350 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.186+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8494003950 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.186+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c004f20 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f8494003b50 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1484+0+0 (unknown 123526312 0 0) 0x7f84940775e0 con 0x7f84a00fac90 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 10 client.14238.objecter ms_dispatch 0x7f84a0001550 osd_map(9..9 src has 1..9) v4 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 3 client.14238.objecter handle_osd_map got epochs [9,9] > 0 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 3 client.14238.objecter handle_osd_map decoding full epoch 9 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 20 client.14238.objecter dump_active .. 0 homeless 2024-02-10T08:30:25.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.187+0000 7f849cf74700 10 client.14238.objecter ms_handle_connect 0x7f8488063db0 2024-02-10T08:30:25.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.191+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f84940489c0 con 0x7f84a00fac90 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:23.843024+0000 mgr.y (mgr.14146) 79 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.850152+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.850943+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:23.851363+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 16098M 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.857352+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.858218+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.858767+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.864622+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:23.866281+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:24.015178+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:24.155173+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:24.155774+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:24.161963+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:25.364+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f848c000b90 con 0x7f8488063db0 2024-02-10T08:30:26.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:25.366646+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 v1:172.21.15.107:0/4013891351' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:26.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:25.368504+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:26.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:25.370268+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:26.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:25.370769+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:26.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:25.366646+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 v1:172.21.15.107:0/4013891351' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:26.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:25.368504+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:26.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:25.370268+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:26.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:25.370769+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:25.366646+0000 mgr.y (mgr.14146) 82 : audit [DBG] from='client.14238 v1:172.21.15.107:0/4013891351' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:25.368504+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:25.370268+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:25.370769+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:26.015470+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:26.015470+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:27.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:26.015470+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:28.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:27.771834+0000 mon.a (mon.0) 271 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]: dispatch 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:27.776350+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]': finished 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:27.776397+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:27.776493+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:27.771834+0000 mon.a (mon.0) 271 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]: dispatch 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:27.776350+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]': finished 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:27.776397+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-10T08:30:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:27.776493+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:27.771834+0000 mon.a (mon.0) 271 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]: dispatch 2024-02-10T08:30:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:27.776350+0000 mon.a (mon.0) 272 : audit [INF] from='client.? v1:172.21.15.107:0/4242234707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80"}]': finished 2024-02-10T08:30:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:27.776397+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-10T08:30:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:27.776493+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:28.015743+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:28.361565+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.107:0/2481995566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:28.015743+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:28.361565+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.107:0/2481995566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:28.015743+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:29.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:28.361565+0000 mon.c (mon.2) 3 : audit [DBG] from='client.? v1:172.21.15.107:0/2481995566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:30.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:30.187+0000 7f84a5284700 10 client.14238.objecter tick 2024-02-10T08:30:31.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:30.015943+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:31.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:30.015943+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:30.015943+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:32.016160+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:33.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:32.016160+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:33.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:32.016160+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:34.042431+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T08:30:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:34.043066+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:34.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:34.042431+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T08:30:34.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:34.043066+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:34.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:34.042431+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T08:30:34.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:34.043066+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:35.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:35.188+0000 7f84a5284700 10 client.14238.objecter tick 2024-02-10T08:30:35.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:34.016711+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:35.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:34.043526+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi107 2024-02-10T08:30:35.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:34.016711+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:35.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:34.043526+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi107 2024-02-10T08:30:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:34.016711+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:34.043526+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi107 2024-02-10T08:30:37.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:36.016942+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:37.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:36.016942+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:37.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:36.016942+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:38.017196+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:39.135237+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:39.156018+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:38.017196+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:39.135237+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:39.156018+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:38.017196+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:39.135237+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:39.156018+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:40.189+0000 7f84a5284700 10 client.14238.objecter tick 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:40.017437+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:41.309813+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:41.310619+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:41.311220+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:41.315817+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:40.017437+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:41.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:41.309813+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:41.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:41.310619+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:41.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:41.311220+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:41.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:41.315817+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:40.017437+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:41.309813+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:41.310619+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:41.311220+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:41.315817+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:42.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.237197+0000 mon.a (mon.0) 283 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T08:30:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.671+0000 7f849cf74700 1 -- 172.21.15.107:0/4013891351 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 207946240) 0x7f848c000b90 con 0x7f8488063db0 2024-02-10T08:30:42.672 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 1 on host 'smithi107' 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 10 librados: watch_flush enter 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 10 librados: watch_flush exit 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 20 client.14238.objecter shutdown clearing up homeless session... 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 10 client.14238.objecter successfully canceled tick 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 >> v1:172.21.15.107:6800/3360068681 conn(0x7f8488063db0 legacy=0x7f8488066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 >> v1:172.21.15.107:6789/0 conn(0x7f84a00fac90 legacy=0x7f84a010cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 shutdown_connections 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 >> 172.21.15.107:0/4013891351 conn(0x7f84a00f1f00 msgr2=0x7f84a00fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 shutdown_connections 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 -- 172.21.15.107:0/4013891351 wait complete. 2024-02-10T08:30:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:42.673+0000 7f84a5a85700 1 librados: shutdown 2024-02-10T08:30:42.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.237197+0000 mon.a (mon.0) 283 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T08:30:42.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.237197+0000 mon.a (mon.0) 283 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T08:30:43.336 DEBUG:teuthology.orchestra.run.smithi107:osd.1> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T08:30:43.339 INFO:tasks.cephadm:Deploying osd.2 on smithi107 with /dev/vg_nvme/lv_2... 2024-02-10T08:30:43.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T08:30:43.370 INFO:journalctl@ceph.osd.1.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:30:43.623 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:30:43.524+0000 7f40bd7e6700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T08:30:43.623 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:30:43.530+0000 7f40b5f5a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:30:43.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:42.018034+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:43.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.520184+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T08:30:43.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:42.520247+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-10T08:30:43.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.520351+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.520702+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.671312+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:42.684088+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:42.018034+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.520184+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:42.520247+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.520351+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.520702+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.671312+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:43.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:42.684088+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:42.018034+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.520184+0000 mon.a (mon.0) 284 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T08:30:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:42.520247+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-10T08:30:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.520351+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.520702+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:30:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.671312+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:42.684088+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:30:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:43.521869+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:43.521931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-10T08:30:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:43.522275+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:43.528892+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:44.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:43.521869+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:44.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:43.521931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-10T08:30:44.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:43.522275+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:44.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:43.528892+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:44.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:43.521869+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 v1:172.21.15.107:6805/133516126' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:30:44.926 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:43.521931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-10T08:30:44.926 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:43.522275+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:44.926 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:43.528892+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:45.868 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:43.222375+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:45.868 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:43.222420+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:45.868 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:44.018321+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:45.868 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:44.524826+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:44.537866+0000 mon.a (mon.0) 295 : cluster [INF] osd.1 v1:172.21.15.107:6805/133516126 boot 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:44.537901+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:44.538412+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:43.222375+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:43.222420+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:44.018321+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:44.524826+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:44.537866+0000 mon.a (mon.0) 295 : cluster [INF] osd.1 v1:172.21.15.107:6805/133516126 boot 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:44.537901+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-10T08:30:45.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:44.538412+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:43.222375+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:43.222420+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:44.018321+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:44.524826+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:44.537866+0000 mon.a (mon.0) 295 : cluster [INF] osd.1 v1:172.21.15.107:6805/133516126 boot 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:44.537901+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-10T08:30:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:44.538412+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:30:46.910 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:45.543394+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-10T08:30:46.910 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:45.543394+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110275 s, 951 MB/s 2024-02-10T08:30:46.913 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-02-10T08:30:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:45.543394+0000 mon.a (mon.0) 298 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-10T08:30:47.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-02-10T08:30:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:46.018715+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.196868+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:46.018715+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.196868+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:46.018715+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.196868+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:47.192499+0000 mgr.y (mgr.14146) 95 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.803082+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.803677+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.809360+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.810139+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.810667+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:47.814750+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:47.192499+0000 mgr.y (mgr.14146) 95 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.803082+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.803677+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.809360+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.810139+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.810667+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:47.814750+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:47.192499+0000 mgr.y (mgr.14146) 95 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.803082+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.803677+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.809360+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.810139+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.810667+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:30:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:47.814750+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- 172.21.15.107:0/1430913951 >> v1:172.21.15.107:6790/0 conn(0x7faa8c0ff9c0 legacy=0x7faa8c101e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- 172.21.15.107:0/1430913951 shutdown_connections 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- 172.21.15.107:0/1430913951 >> 172.21.15.107:0/1430913951 conn(0x7faa8c0f3020 msgr2=0x7faa8c0f5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- 172.21.15.107:0/1430913951 shutdown_connections 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- 172.21.15.107:0/1430913951 wait complete. 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 librados: starting msgr at 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 librados: starting objecter 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 Processor -- start 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 -- start start 2024-02-10T08:30:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 librados: setting wanted keys 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.417+0000 7faa935d6700 1 librados: calling monclient init 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa8c108430 con 0x7faa8c0fbd90 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa8c113c90 con 0x7faa8c0f78c0 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa8c114e70 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa90b71700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faa8c0fbd90 0x7faa8c10de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34882/0 (socket says 172.21.15.107:34882) 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa90b71700 1 -- 172.21.15.107:0/430992197 learned_addr learned my addr 172.21.15.107:0/430992197 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa91b73700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faa8c0ff9c0 0x7faa8c111560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39698/0 (socket says 172.21.15.107:39698) 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter ms_handle_connect 0x7faa8c0fbd90 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter ms_handle_connect 0x7faa8c0ff9c0 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter ms_handle_connect 0x7faa8c0f78c0 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974150334 0 0) 0x7faa8c108430 con 0x7faa8c0fbd90 2024-02-10T08:30:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa78003030 con 0x7faa8c0fbd90 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052827572 0 0) 0x7faa8c114e70 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c108430 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718146665 0 0) 0x7faa8c113c90 con 0x7faa8c0f78c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c114e70 con 0x7faa8c0f78c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511219163 0 0) 0x7faa78003030 con 0x7faa8c0fbd90 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8c113c90 con 0x7faa8c0fbd90 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa88001940 con 0x7faa8c0fbd90 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258577712 0 0) 0x7faa8c108430 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa78003030 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa840023a0 con 0x7faa8c0ff9c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979449361 0 0) 0x7faa8c114e70 con 0x7faa8c0f78c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8c108430 con 0x7faa8c0f78c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa7c0019f0 con 0x7faa8c0f78c0 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56446144 0 0) 0x7faa8c113c90 con 0x7faa8c0fbd90 2024-02-10T08:30:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 >> v1:172.21.15.107:6790/0 conn(0x7faa8c0ff9c0 legacy=0x7faa8c111560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 >> v1:172.21.15.195:6789/0 conn(0x7faa8c0f78c0 legacy=0x7faa8c107860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c116050 con 0x7faa8c0fbd90 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c115060 con 0x7faa8c0fbd90 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 10 client.14265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.418+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c115550 con 0x7faa8c0fbd90 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.419+0000 7faa935d6700 1 librados: init done 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.419+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7faa88001cd0 con 0x7faa8c0fbd90 2024-02-10T08:30:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.419+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa88005ca0 con 0x7faa8c0fbd90 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa70004f20 con 0x7faa8c0fbd90 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7faa88018b70 con 0x7faa8c0fbd90 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1776+0+0 (unknown 3003082351 0 0) 0x7faa880774a0 con 0x7faa8c0fbd90 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 10 client.14265.objecter ms_dispatch 0x7faa8c001550 osd_map(14..14 src has 1..14) v4 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 3 client.14265.objecter handle_osd_map got epochs [14,14] > 0 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 3 client.14265.objecter handle_osd_map decoding full epoch 14 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 20 client.14265.objecter dump_active .. 0 homeless 2024-02-10T08:30:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.420+0000 7faa827fc700 10 client.14265.objecter ms_handle_connect 0x7faa78063c40 2024-02-10T08:30:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.424+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faa880494f0 con 0x7faa8c0fbd90 2024-02-10T08:30:49.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:49.604+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faa70000b90 con 0x7faa78063c40 2024-02-10T08:30:49.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:47.804065+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 8049M 2024-02-10T08:30:49.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:48.019125+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:49.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:47.804065+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 8049M 2024-02-10T08:30:49.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:48.019125+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:47.804065+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 8049M 2024-02-10T08:30:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:48.019125+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:50.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:49.606113+0000 mgr.y (mgr.14146) 98 : audit [DBG] from='client.14265 v1:172.21.15.107:0/430992197' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:49.608033+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:49.609752+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:49.610187+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:49.606113+0000 mgr.y (mgr.14146) 98 : audit [DBG] from='client.14265 v1:172.21.15.107:0/430992197' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:49.608033+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:49.609752+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:50.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:49.610187+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:49.606113+0000 mgr.y (mgr.14146) 98 : audit [DBG] from='client.14265 v1:172.21.15.107:0/430992197' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:30:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:49.608033+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:30:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:49.609752+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:30:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:49.610187+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:51.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:50.019480+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:50.019480+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:52.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:50.019480+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:51.871367+0000 mon.a (mon.0) 309 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]: dispatch 2024-02-10T08:30:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:51.874880+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]': finished 2024-02-10T08:30:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:51.874932+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-10T08:30:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:51.875025+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:30:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:52.493660+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? v1:172.21.15.107:0/4250476185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:51.871367+0000 mon.a (mon.0) 309 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]: dispatch 2024-02-10T08:30:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:51.874880+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]': finished 2024-02-10T08:30:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:51.874932+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-10T08:30:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:51.875025+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:30:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:52.493660+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? v1:172.21.15.107:0/4250476185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:51.871367+0000 mon.a (mon.0) 309 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]: dispatch 2024-02-10T08:30:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:51.874880+0000 mon.a (mon.0) 310 : audit [INF] from='client.? v1:172.21.15.107:0/27435993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf"}]': finished 2024-02-10T08:30:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:51.874932+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-10T08:30:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:51.875025+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:30:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:52.493660+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? v1:172.21.15.107:0/4250476185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:30:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:52.019816+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:52.019816+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:54.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:52.019816+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:54.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:54.419+0000 7faa92dd5700 10 client.14265.objecter tick 2024-02-10T08:30:55.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:54.020194+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:54.020194+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:54.020194+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:57.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:56.020570+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:56.020570+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:56.020570+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:58.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:58.326435+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T08:30:58.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:30:58.327075+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:58.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:58.326435+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T08:30:58.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:30:58.327075+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:58.326435+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T08:30:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:30:58.327075+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:30:59.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:30:59.420+0000 7faa92dd5700 10 client.14265.objecter tick 2024-02-10T08:30:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:30:58.020969+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:30:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:30:58.327560+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi107 2024-02-10T08:30:59.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:30:58.020969+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:30:59.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:30:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:30:58.327560+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi107 2024-02-10T08:31:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:30:58.020969+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:30:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:30:58.327560+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi107 2024-02-10T08:31:01.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:00.021263+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:01.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:00.021263+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:00.021263+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:02.021649+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:03.552288+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:03.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:03.563302+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:02.021649+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:03.552288+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:03.563302+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:02.021649+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:03.552288+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:03.563302+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:04.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:04.421+0000 7faa92dd5700 10 client.14265.objecter tick 2024-02-10T08:31:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:04.022004+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:04.022004+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:04.022004+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.102+0000 7faa827fc700 1 -- 172.21.15.107:0/430992197 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3898290652) 0x7faa70000b90 con 0x7faa78063c40 2024-02-10T08:31:07.103 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 2 on host 'smithi107' 2024-02-10T08:31:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 10 librados: watch_flush enter 2024-02-10T08:31:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 10 librados: watch_flush exit 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 20 client.14265.objecter shutdown clearing up homeless session... 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 10 client.14265.objecter successfully canceled tick 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 >> v1:172.21.15.107:6800/3360068681 conn(0x7faa78063c40 legacy=0x7faa78066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 >> v1:172.21.15.107:6789/0 conn(0x7faa8c0fbd90 legacy=0x7faa8c10de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 shutdown_connections 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 >> 172.21.15.107:0/430992197 conn(0x7faa8c0f3020 msgr2=0x7faa8c0f3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 shutdown_connections 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 -- 172.21.15.107:0/430992197 wait complete. 2024-02-10T08:31:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:07.104+0000 7faa935d6700 1 librados: shutdown 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:06.022273+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.196578+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.197293+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.197894+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.201517+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.673756+0000 mon.c (mon.2) 5 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:06.674281+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.102266+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.119130+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:06.022273+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.196578+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.197293+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.197894+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.201517+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.673756+0000 mon.c (mon.2) 5 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:06.674281+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.102266+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.119130+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:06.022273+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.196578+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.197293+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.197894+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.201517+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.673756+0000 mon.c (mon.2) 5 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:06.674281+0000 mon.a (mon.0) 321 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T08:31:07.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.102266+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:07.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.119130+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:07.833 DEBUG:teuthology.orchestra.run.smithi107:osd.2> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T08:31:07.835 INFO:tasks.cephadm:Deploying osd.3 on smithi107 with /dev/vg_nvme/lv_1... 2024-02-10T08:31:07.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T08:31:07.865 INFO:journalctl@ceph.osd.2.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:31:08.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.593656+0000 mon.a (mon.0) 324 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T08:31:08.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:07.593747+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.593854+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.594209+0000 mon.c (mon.2) 6 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:07.594489+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.593656+0000 mon.a (mon.0) 324 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:07.593747+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.593854+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.594209+0000 mon.c (mon.2) 6 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:07.594489+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:08.682 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2[87274]: debug 2024-02-10T08:31:08.598+0000 7fbb8f52e700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T08:31:08.682 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:31:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2[87274]: debug 2024-02-10T08:31:08.601+0000 7fbb88ca4700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:31:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.593656+0000 mon.a (mon.0) 324 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T08:31:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:07.593747+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-10T08:31:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.593854+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.594209+0000 mon.c (mon.2) 6 : audit [INF] from='osd.2 v1:172.21.15.107:6809/3575881728' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:07.594489+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:08.022539+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:08.595115+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:08.595194+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:08.595507+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:08.599466+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:08.022539+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:08.595115+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:08.595194+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:08.595507+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:09.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:08.599466+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:10.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:08.022539+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:10.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:08.595115+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:10.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:08.595194+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-10T08:31:10.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:08.595507+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:10.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:08.599466+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:10.512 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T08:31:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T08:31:10.513 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-02-10T08:31:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-02-10T08:31:10.513 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111928 s, 937 MB/s 2024-02-10T08:31:10.513 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:07.694633+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:07.694689+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:09.600716+0000 mon.a (mon.0) 332 : cluster [INF] osd.2 v1:172.21.15.107:6809/3575881728 boot 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:09.600762+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:09.601088+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:09.601465+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:31:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:10.022819+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:10.603252+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:10.603297+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:10.603825+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:07.694633+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:07.694689+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:09.600716+0000 mon.a (mon.0) 332 : cluster [INF] osd.2 v1:172.21.15.107:6809/3575881728 boot 2024-02-10T08:31:11.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:09.600762+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:09.601088+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:09.601465+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:10.022819+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:10.603252+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:10.603297+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-10T08:31:11.040 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:10.603825+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:31:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:07.694633+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:07.694689+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:09.600716+0000 mon.a (mon.0) 332 : cluster [INF] osd.2 v1:172.21.15.107:6809/3575881728 boot 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:09.600762+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:09.601088+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:09.601465+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:10.022819+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:10.603252+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:10.603297+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-10T08:31:11.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:10.603825+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:31:11.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-02-10T08:31:12.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:11.604909+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:31:12.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:11.604996+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-10T08:31:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:11.604909+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:31:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:11.604996+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-10T08:31:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:11.604909+0000 mon.a (mon.0) 339 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:31:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:11.604996+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-10T08:31:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:12.023120+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:12.608954+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-10T08:31:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.491027+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.491863+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.492424+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.492894+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.498929+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.499682+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.500196+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:13.506960+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:12.023120+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:12.608954+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.491027+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.491863+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.492424+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.492894+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:13.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.498929+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.703 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.499682+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:13.703 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.500196+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:13.703 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:13.506960+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.948+0000 7f99ff932700 1 -- 172.21.15.107:0/1906251328 >> v1:172.21.15.107:6789/0 conn(0x7f99f80ff9f0 legacy=0x7f99f8101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.948+0000 7f99ff932700 1 -- 172.21.15.107:0/1906251328 shutdown_connections 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.948+0000 7f99ff932700 1 -- 172.21.15.107:0/1906251328 >> 172.21.15.107:0/1906251328 conn(0x7f99f80f16b0 msgr2=0x7f99f80f3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.948+0000 7f99ff932700 1 -- 172.21.15.107:0/1906251328 shutdown_connections 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.948+0000 7f99ff932700 1 -- 172.21.15.107:0/1906251328 wait complete. 2024-02-10T08:31:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 librados: starting msgr at 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 librados: starting objecter 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 Processor -- start 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 -- start start 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 librados: setting wanted keys 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 librados: calling monclient init 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99f8114c10 con 0x7f99f80ff9f0 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99f8115df0 con 0x7f99f80f5b90 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ff932700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99f8116ff0 con 0x7f99f80fbee0 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99fcecd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f99f80fbee0 0x7f99f810f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40298/0 (socket says 172.21.15.107:40298) 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99fcecd700 1 -- 172.21.15.107:0/2839181136 learned_addr learned my addr 172.21.15.107:0/2839181136 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter ms_handle_connect 0x7f99f80fbee0 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:13.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter ms_handle_connect 0x7f99f80ff9f0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter ms_handle_connect 0x7f99f80f5b90 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.949+0000 7f99ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950249921 0 0) 0x7f99f8116ff0 con 0x7f99f80fbee0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4003030 con 0x7f99f80fbee0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035028114 0 0) 0x7f99f8114c10 con 0x7f99f80ff9f0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8116ff0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785246773 0 0) 0x7f99f8115df0 con 0x7f99f80f5b90 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8114c10 con 0x7f99f80f5b90 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4249487161 0 0) 0x7f99e4003030 con 0x7f99f80fbee0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f8115df0 con 0x7f99f80fbee0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99f4001290 con 0x7f99f80fbee0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1736099397 0 0) 0x7f99f8116ff0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99e4003030 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99f00023f0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908281795 0 0) 0x7f99f8114c10 con 0x7f99f80f5b90 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f8116ff0 con 0x7f99f80f5b90 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99e8001840 con 0x7f99f80f5b90 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 475895969 0 0) 0x7f99e4003030 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 >> v1:172.21.15.107:6790/0 conn(0x7f99f80fbee0 legacy=0x7f99f810f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 >> v1:172.21.15.195:6789/0 conn(0x7f99f80f5b90 legacy=0x7f99f80f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f81181f0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f8114e00 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ff932700 10 client.14292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f81152f0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.950+0000 7f99ff932700 1 librados: init done 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.951+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f99f00015b0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.951+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99f0003a80 con 0x7f99f80ff9f0 2024-02-10T08:31:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.951+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc004f20 con 0x7f99f80ff9f0 2024-02-10T08:31:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.951+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f99f0016970 con 0x7f99f80ff9f0 2024-02-10T08:31:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 2484+0+0 (unknown 3912798270 0 0) 0x7f99f00753e0 con 0x7f99f80ff9f0 2024-02-10T08:31:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 10 client.14292.objecter ms_dispatch 0x7f99f8001550 osd_map(21..21 src has 1..21) v4 2024-02-10T08:31:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 3 client.14292.objecter handle_osd_map got epochs [21,21] > 0 2024-02-10T08:31:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 3 client.14292.objecter handle_osd_map decoding full epoch 21 2024-02-10T08:31:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 20 client.14292.objecter dump_active .. 0 homeless 2024-02-10T08:31:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.952+0000 7f99ee7fc700 10 client.14292.objecter ms_handle_connect 0x7f99e4063c00 2024-02-10T08:31:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:13.956+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f99f0046400 con 0x7f99f80ff9f0 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:12.023120+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:12.608954+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.491027+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.491863+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.492424+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.492894+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.498929+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.499682+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.500196+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:13.506960+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:14.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:14.126+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f99dc000b90 con 0x7f99e4063c00 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:13.486653+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:13.493246+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 5366M 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:14.131263+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:14.132963+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:14.133372+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:13.486653+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:14.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:13.493246+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 5366M 2024-02-10T08:31:14.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:14.131263+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:14.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:14.132963+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:14.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:14.133372+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:13.486653+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:13.493246+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 5366M 2024-02-10T08:31:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:14.131263+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:14.132963+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:14.133372+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:15.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:14.023440+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:15.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:14.129221+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.107:0/2839181136' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:15.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:14.023440+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:15.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:14.129221+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.107:0/2839181136' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:14.023440+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:14.129221+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.107:0/2839181136' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:16.583588+0000 mon.a (mon.0) 353 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]: dispatch 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:16.588805+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]': finished 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:16.588842+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:16.588947+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:16.583588+0000 mon.a (mon.0) 353 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]: dispatch 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:16.588805+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]': finished 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:16.588842+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-10T08:31:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:16.588947+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:16.583588+0000 mon.a (mon.0) 353 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]: dispatch 2024-02-10T08:31:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:16.588805+0000 mon.a (mon.0) 354 : audit [INF] from='client.? v1:172.21.15.107:0/19487504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43"}]': finished 2024-02-10T08:31:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:16.588842+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-10T08:31:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:16.588947+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:16.023761+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:17.176404+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? v1:172.21.15.107:0/1924312715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:16.023761+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:17.176404+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? v1:172.21.15.107:0/1924312715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:16.023761+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:17.176404+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? v1:172.21.15.107:0/1924312715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:18.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:18.951+0000 7f99ff131700 10 client.14292.objecter tick 2024-02-10T08:31:19.814 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:18.024091+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:19.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:18.024091+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:18.024091+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:21.824 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:20.024601+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:21.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:20.024601+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:20.024601+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:22.024883+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:22.866892+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:22.867424+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:22.024883+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:22.866892+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T08:31:23.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:22.867424+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:23.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:23.952+0000 7f99ff131700 10 client.14292.objecter tick 2024-02-10T08:31:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:22.024883+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:22.866892+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T08:31:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:22.867424+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:22.867871+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi107 2024-02-10T08:31:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:22.867871+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi107 2024-02-10T08:31:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:22.867871+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi107 2024-02-10T08:31:25.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:24.025519+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:25.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:24.025519+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:24.025519+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:26.026015+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:26.026015+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:26.026015+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:28.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:28.953+0000 7f99ff131700 10 client.14292.objecter tick 2024-02-10T08:31:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:27.877866+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:27.901265+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:28.026332+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:27.877866+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:27.901265+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:28.026332+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:27.877866+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:27.901265+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:28.026332+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:31.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.286+0000 7f99ee7fc700 1 -- 172.21.15.107:0/2839181136 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3025067880) 0x7f99dc000b90 con 0x7f99e4063c00 2024-02-10T08:31:31.287 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 3 on host 'smithi107' 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 10 librados: watch_flush enter 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 10 librados: watch_flush exit 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 20 client.14292.objecter shutdown clearing up homeless session... 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 10 client.14292.objecter successfully canceled tick 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 >> v1:172.21.15.107:6800/3360068681 conn(0x7f99e4063c00 legacy=0x7f99e40660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 >> v1:172.21.15.107:6789/0 conn(0x7f99f80ff9f0 legacy=0x7f99f81132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 shutdown_connections 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 >> 172.21.15.107:0/2839181136 conn(0x7f99f80f16b0 msgr2=0x7f99f80f3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 shutdown_connections 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 -- 172.21.15.107:0/2839181136 wait complete. 2024-02-10T08:31:31.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:31:31.288+0000 7f99ff932700 1 librados: shutdown 2024-02-10T08:31:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:30.026612+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.637315+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.638076+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.638612+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.643343+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.989093+0000 mon.c (mon.2) 8 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:30.989352+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.286273+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.310159+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:30.026612+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.637315+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.638076+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.638612+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.643343+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.989093+0000 mon.c (mon.2) 8 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:30.989352+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.286273+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.310159+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:31.965 DEBUG:teuthology.orchestra.run.smithi107:osd.3> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T08:31:31.967 INFO:tasks.cephadm:Deploying osd.4 on smithi195 with /dev/vg_nvme/lv_4... 2024-02-10T08:31:31.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T08:31:31.997 INFO:journalctl@ceph.osd.3.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:31:32.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:30.026612+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:32.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.637315+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:32.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.638076+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:32.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.638612+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:32.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.643343+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:32.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.989093+0000 mon.c (mon.2) 8 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:32.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:30.989352+0000 mon.a (mon.0) 365 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T08:31:32.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.286273+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:32.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.310159+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.644910+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T08:31:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:31.644957+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-10T08:31:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.645048+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.645635+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:31.645926+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.644910+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:31.644957+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.645048+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.645635+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:31.645926+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.644910+0000 mon.a (mon.0) 368 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T08:31:32.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:31.644957+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-10T08:31:32.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.645048+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:32.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.645635+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.107:6813/2871825691' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:31.645926+0000 mon.a (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-02-10T08:31:32.928 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3[91952]: debug 2024-02-10T08:31:32.651+0000 7f0dda343700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T08:31:32.928 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:31:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3[91952]: debug 2024-02-10T08:31:32.655+0000 7f0dd6abf700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:31:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:32.026921+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:32.649865+0000 mon.a (mon.0) 372 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:32.649931+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:32.650169+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:32.652620+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:32.026921+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:32.649865+0000 mon.a (mon.0) 372 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:32.649931+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:32.650169+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:32.652620+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:32.026921+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T08:31:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:32.649865+0000 mon.a (mon.0) 372 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-02-10T08:31:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:32.649931+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-10T08:31:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:32.650169+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:32.652620+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00803291 s, 1.3 GB/s 2024-02-10T08:31:34.072 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-10T08:31:34.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:32.023925+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:32.024006+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:33.650129+0000 mon.a (mon.0) 376 : cluster [INF] osd.3 v1:172.21.15.107:6813/2871825691 boot 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:33.650160+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:33.650459+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:32.023925+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:32.024006+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:33.650129+0000 mon.a (mon.0) 376 : cluster [INF] osd.3 v1:172.21.15.107:6813/2871825691 boot 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:33.650160+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-10T08:31:34.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:33.650459+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.777 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:32.023925+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:34.778 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:32.024006+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:34.778 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:33.650129+0000 mon.a (mon.0) 376 : cluster [INF] osd.3 v1:172.21.15.107:6813/2871825691 boot 2024-02-10T08:31:34.778 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:33.650160+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-10T08:31:34.778 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:33.650459+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:31:34.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_4 2024-02-10T08:31:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:34.027223+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:34.658635+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-10T08:31:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.345931+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.346814+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.347322+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.347777+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.348223+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.352265+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.353707+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.354199+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:35.358345+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:34.027223+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:34.658635+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-10T08:31:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.345931+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.346814+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.347322+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.347777+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.348223+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.352265+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.353707+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.354199+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:35.358345+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:34.027223+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:34.658635+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-10T08:31:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.345931+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.346814+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.347322+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.347777+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.348223+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.352265+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.353707+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.354199+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:35.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:35.358345+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- 172.21.15.195:0/1774493967 >> v1:172.21.15.195:6789/0 conn(0x7f54ac0ff9c0 legacy=0x7f54ac101e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- 172.21.15.195:0/1774493967 shutdown_connections 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- 172.21.15.195:0/1774493967 >> 172.21.15.195:0/1774493967 conn(0x7f54ac0f1f20 msgr2=0x7f54ac0f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- 172.21.15.195:0/1774493967 shutdown_connections 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- 172.21.15.195:0/1774493967 wait complete. 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 librados: starting msgr at 2024-02-10T08:31:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 librados: starting objecter 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 Processor -- start 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 -- start start 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 librados: setting wanted keys 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.744+0000 7f54b31f3700 1 librados: calling monclient init 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54b31f3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54ac114ff0 con 0x7f54ac061780 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54b31f3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54ac1161d0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54b31f3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54ac1173d0 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54abfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f54ac0fbcb0 0x7f54ac10fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:42206/0 (socket says 172.21.15.195:42206) 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54abfff700 1 -- 172.21.15.195:0/3009401321 learned_addr learned my addr 172.21.15.195:0/3009401321 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter ms_handle_connect 0x7f54ac0fbcb0 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter ms_handle_connect 0x7f54ac061780 2024-02-10T08:31:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter ms_handle_connect 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745653693 0 0) 0x7f54ac1161d0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549c002220 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219069852 0 0) 0x7f54ac114ff0 con 0x7f54ac061780 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac1161d0 con 0x7f54ac061780 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203420508 0 0) 0x7f54ac1173d0 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac114ff0 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3133387863 0 0) 0x7f549c002220 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ac1173d0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5498001900 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1262803740 0 0) 0x7f54ac114ff0 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f549c002220 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f54a4001c40 con 0x7f54ac0ff9c0 2024-02-10T08:31:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754456450 0 0) 0x7f54ac1161d0 con 0x7f54ac061780 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ac114ff0 con 0x7f54ac061780 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746239949 0 0) 0x7f54ac1173d0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 >> v1:172.21.15.107:6790/0 conn(0x7f54ac0ff9c0 legacy=0x7f54ac1136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 >> v1:172.21.15.107:6789/0 conn(0x7f54ac061780 legacy=0x7f54ac0f7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac1185d0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac1175c0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f54980023e0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.745+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5498005af0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.746+0000 7f54b31f3700 10 client.24229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.746+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac117ab0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.746+0000 7f54b31f3700 1 librados: init done 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.746+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac060300 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f5498001f40 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 2776+0+0 (unknown 3489454960 0 0) 0x7f54980777c0 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 10 client.24229.objecter ms_dispatch 0x7f54ac001550 osd_map(26..26 src has 1..26) v4 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 3 client.24229.objecter handle_osd_map got epochs [26,26] > 0 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 3 client.24229.objecter handle_osd_map decoding full epoch 26 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.747+0000 7f54a9ffb700 20 client.24229.objecter dump_active .. 0 homeless 2024-02-10T08:31:36.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.748+0000 7f54a9ffb700 10 client.24229.objecter ms_handle_connect 0x7f549c063d60 2024-02-10T08:31:36.752 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.752+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f54ac060300 con 0x7f54ac0fbcb0 2024-02-10T08:31:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:35.341106+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:35.348575+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 4024M 2024-02-10T08:31:36.921 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:36.920+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f54ac060300 con 0x7f549c063d60 2024-02-10T08:31:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:35.341106+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:35.348575+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 4024M 2024-02-10T08:31:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:35.341106+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi107 2024-02-10T08:31:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:35.348575+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 4024M 2024-02-10T08:31:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:36.027688+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:36.924797+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:36.926977+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:36.927608+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:36.027688+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:36.924797+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:36.926977+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:36.927608+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:36.027688+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:36.924797+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:31:38.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:36.926977+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:31:38.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:36.927608+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:36.922307+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24229 v1:172.21.15.195:0/3009401321' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:36.922307+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24229 v1:172.21.15.195:0/3009401321' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:36.922307+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24229 v1:172.21.15.195:0/3009401321' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:31:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:38.028058+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:39.391016+0000 mon.b (mon.1) 10 : audit [INF] from='client.? v1:172.21.15.195:0/2822433416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:39.391515+0000 mon.a (mon.0) 392 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:39.398378+0000 mon.a (mon.0) 393 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]': finished 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:39.398471+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:39.398626+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:38.028058+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:39.391016+0000 mon.b (mon.1) 10 : audit [INF] from='client.? v1:172.21.15.195:0/2822433416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:39.391515+0000 mon.a (mon.0) 392 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:39.398378+0000 mon.a (mon.0) 393 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]': finished 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:39.398471+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-10T08:31:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:39.398626+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:38.028058+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:39.391016+0000 mon.b (mon.1) 10 : audit [INF] from='client.? v1:172.21.15.195:0/2822433416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:39.391515+0000 mon.a (mon.0) 392 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]: dispatch 2024-02-10T08:31:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:39.398378+0000 mon.a (mon.0) 393 : audit [INF] from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7529a6-19fc-4435-be7f-28876750838a"}]': finished 2024-02-10T08:31:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:39.398471+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-10T08:31:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:39.398626+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:39.941665+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? v1:172.21.15.195:0/2511592053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:39.941665+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? v1:172.21.15.195:0/2511592053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:39.941665+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? v1:172.21.15.195:0/2511592053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:31:41.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:41.747+0000 7f54b29f2700 10 client.24229.objecter tick 2024-02-10T08:31:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:40.028506+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:40.028506+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:40.028506+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:42.028940+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:42.790259+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:42.790350+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:42.028940+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:42.790259+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:42.790350+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:42.028940+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:42.790259+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:42.790350+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:43.690262+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-10T08:31:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:43.692813+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:43.690262+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-10T08:31:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:43.692813+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:43.690262+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-10T08:31:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:43.692813+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:45.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:44.029334+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:46.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:44.029334+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:44.029334+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:46.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:46.748+0000 7f54b29f2700 10 client.24229.objecter tick 2024-02-10T08:31:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:45.820103+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T08:31:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:45.820951+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:45.820103+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T08:31:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:45.820951+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:45.820103+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T08:31:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:45.820951+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:45.821610+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi195 2024-02-10T08:31:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:46.029679+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:48.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:45.821610+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi195 2024-02-10T08:31:48.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:46.029679+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:48.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:45.821610+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi195 2024-02-10T08:31:48.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:46.029679+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:50.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:48.030056+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:50.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:48.030056+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:48.030056+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:51.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:51.748+0000 7f54b29f2700 10 client.24229.objecter tick 2024-02-10T08:31:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:50.030433+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:51.086107+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:51.111859+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:52.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:50.030433+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:52.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:51.086107+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:52.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:51.111859+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:52.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:50.030433+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:52.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:51.086107+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:52.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:51.111859+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:53.973 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:52.030781+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:54.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:52.030781+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:54.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:52.030781+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:54.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.528+0000 7f54a9ffb700 1 -- 172.21.15.195:0/3009401321 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1101380550) 0x7f54ac060300 con 0x7f549c063d60 2024-02-10T08:31:54.530 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 4 on host 'smithi195' 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 10 librados: watch_flush enter 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 10 librados: watch_flush exit 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 20 client.24229.objecter shutdown clearing up homeless session... 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 10 client.24229.objecter successfully canceled tick 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 >> v1:172.21.15.107:6800/3360068681 conn(0x7f549c063d60 legacy=0x7f549c066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.530+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 >> v1:172.21.15.195:6789/0 conn(0x7f54ac0fbcb0 legacy=0x7f54ac10fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:31:54.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.531+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 shutdown_connections 2024-02-10T08:31:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.531+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 >> 172.21.15.195:0/3009401321 conn(0x7f54ac0f1f20 msgr2=0x7f54ac0f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:31:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.531+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 shutdown_connections 2024-02-10T08:31:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.531+0000 7f54b31f3700 1 -- 172.21.15.195:0/3009401321 wait complete. 2024-02-10T08:31:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:31:54.531+0000 7f54b31f3700 1 librados: shutdown 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.807034+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.807819+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.808602+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.815343+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.976778+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:53.977211+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.176 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:54.031211+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:55.176 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.529456+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.176 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.532958+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:55.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.807034+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.807819+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:55.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.808602+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:55.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.815343+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.976778+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:53.977211+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:54.031211+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.529456+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.532958+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.807034+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.807819+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.808602+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.815343+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.976778+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:53.977211+0000 mon.a (mon.0) 406 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:54.031211+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:55.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.529456+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:31:55.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.532958+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:31:55.206 DEBUG:teuthology.orchestra.run.smithi195:osd.4> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T08:31:55.209 INFO:tasks.cephadm:Deploying osd.5 on smithi195 with /dev/vg_nvme/lv_3... 2024-02-10T08:31:55.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T08:31:55.240 INFO:journalctl@ceph.osd.4.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:31:56.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.823100+0000 mon.a (mon.0) 409 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:31:56.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:54.823200+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-10T08:31:56.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.823346+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:56.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.824137+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:54.824779+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.823100+0000 mon.a (mon.0) 409 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:54.823200+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.823346+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.824137+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:54.824779+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.243 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:31:55.832+0000 7fdb7e967700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T08:31:56.243 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:31:55.837+0000 7fdb798e0700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:31:56.243 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.823100+0000 mon.a (mon.0) 409 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:31:56.243 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:54.823200+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-10T08:31:56.243 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.823346+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:56.243 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.824137+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.195:6800/2499671285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.243 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:54.824779+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:31:56.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:55.828383+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:31:56.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:55.828448+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-10T08:31:56.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:55.828848+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:56.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:55.831521+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:56.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:56.031717+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:56.915 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:56.829257+0000 mon.a (mon.0) 417 : cluster [INF] osd.4 v1:172.21.15.195:6800/2499671285 boot 2024-02-10T08:31:56.915 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:56.829304+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-10T08:31:56.915 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:56.829449+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:55.828383+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:31:57.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:55.828448+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-10T08:31:57.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:55.828848+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:55.831521+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:56.031717+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:56.829257+0000 mon.a (mon.0) 417 : cluster [INF] osd.4 v1:172.21.15.195:6800/2499671285 boot 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:56.829304+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:56.829449+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:55.828383+0000 mon.a (mon.0) 413 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:55.828448+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:55.828848+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:55.831521+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:56.031717+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:56.829257+0000 mon.a (mon.0) 417 : cluster [INF] osd.4 v1:172.21.15.195:6800/2499671285 boot 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:56.829304+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-10T08:31:57.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:56.829449+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:31:58.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:54.965735+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:54.965795+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:56.307792+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:56.307874+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:57.827569+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-10T08:31:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:54.965735+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:54.965795+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:56.307792+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:56.307874+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:57.827569+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-10T08:31:58.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:54.965735+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:54.965795+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:56.307792+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:31:58.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:56.307874+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:31:58.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:57.827569+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00813303 s, 1.3 GB/s 2024-02-10T08:31:58.529 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-10T08:31:59.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:58.032181+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:31:59.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:31:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:31:58.828967+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-10T08:31:59.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:58.032181+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:31:59.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:31:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:31:58.828967+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-10T08:31:59.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:58.032181+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:31:59.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:31:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:31:58.828967+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-10T08:31:59.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_3 2024-02-10T08:32:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:59.137951+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.142457+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.143604+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:31:59.144445+0000 mgr.y (mgr.14146) 144 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 17122M 2024-02-10T08:32:00.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.151127+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.152038+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:00.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.152769+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:00.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:31:59.157226+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:59.137951+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.142457+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.143604+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:31:59.144445+0000 mgr.y (mgr.14146) 144 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 17122M 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.151127+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.152038+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.152769+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:31:59.157226+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:59.137951+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.142457+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.143604+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:31:59.144445+0000 mgr.y (mgr.14146) 144 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 17122M 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.151127+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.152038+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.152769+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:00.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:31:59.157226+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:01.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.402+0000 7fe811df4700 1 -- 172.21.15.195:0/4175902023 >> v1:172.21.15.195:6789/0 conn(0x7fe80c0ff9d0 legacy=0x7fe80c101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.402+0000 7fe811df4700 1 -- 172.21.15.195:0/4175902023 shutdown_connections 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.402+0000 7fe811df4700 1 -- 172.21.15.195:0/4175902023 >> 172.21.15.195:0/4175902023 conn(0x7fe80c0f3050 msgr2=0x7fe80c0f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 -- 172.21.15.195:0/4175902023 shutdown_connections 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 -- 172.21.15.195:0/4175902023 wait complete. 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 librados: starting msgr at 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 librados: starting objecter 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 Processor -- start 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 -- start start 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 librados: setting wanted keys 2024-02-10T08:32:01.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 librados: calling monclient init 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.403+0000 7fe811df4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe80c114dd0 con 0x7fe80c0f78d0 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe80c115fb0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe80c1171b0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe80b7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe80c0f78d0 0x7fe80c06ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:38776/0 (socket says 172.21.15.195:38776) 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe80bfff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe80c0ff9d0 0x7fe80c113490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:59116/0 (socket says 172.21.15.195:59116) 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe80b7fe700 1 -- 172.21.15.195:0/498532983 learned_addr learned my addr 172.21.15.195:0/498532983 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:32:01.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter ms_handle_connect 0x7fe80c0ff9d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter ms_handle_connect 0x7fe80c0f78d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter ms_handle_connect 0x7fe80c0fbda0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560550396 0 0) 0x7fe80c115fb0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f4002220 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242260218 0 0) 0x7fe80c114dd0 con 0x7fe80c0f78d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe80c115fb0 con 0x7fe80c0f78d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846801267 0 0) 0x7fe80c1171b0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe80c114dd0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602927419 0 0) 0x7fe7f4002220 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe80c1171b0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.409 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe8040017b0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2695087 0 0) 0x7fe80c115fb0 con 0x7fe80c0f78d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f4002220 con 0x7fe80c0f78d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe7fc001900 con 0x7fe80c0f78d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210798324 0 0) 0x7fe80c114dd0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe80c115fb0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe8000017c0 con 0x7fe80c0fbda0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1967141721 0 0) 0x7fe80c1171b0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 >> v1:172.21.15.107:6790/0 conn(0x7fe80c0fbda0 legacy=0x7fe80c10fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 >> v1:172.21.15.107:6789/0 conn(0x7fe80c0f78d0 legacy=0x7fe80c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c1183b0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c114fc0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 10 client.24256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:32:01.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c1154b0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.404+0000 7fe811df4700 1 librados: init done 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.405+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe8040028e0 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.405+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe804003a80 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.405+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f8004f20 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7fe804002440 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(33..33 src has 1..33) v4 ==== 3178+0+0 (unknown 451594124 0 0) 0x7fe804077e40 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 10 client.24256.objecter ms_dispatch 0x7fe80c001550 osd_map(33..33 src has 1..33) v4 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 3 client.24256.objecter handle_osd_map got epochs [33,33] > 0 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 3 client.24256.objecter handle_osd_map decoding full epoch 33 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.406+0000 7fe808ff9700 20 client.24256.objecter dump_active .. 0 homeless 2024-02-10T08:32:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.407+0000 7fe808ff9700 10 client.24256.objecter ms_handle_connect 0x7fe7f4063e60 2024-02-10T08:32:01.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.410+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe804048b40 con 0x7fe80c0ff9d0 2024-02-10T08:32:01.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:01.579+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7f8000b90 con 0x7fe7f4063e60 2024-02-10T08:32:01.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:00.032797+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:01.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:00.032797+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:00.032797+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:01.582484+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24256 v1:172.21.15.195:0/498532983' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:01.584759+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:01.587045+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:01.587715+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:01.582484+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24256 v1:172.21.15.195:0/498532983' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:01.584759+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:01.587045+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:01.587715+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:01.582484+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24256 v1:172.21.15.195:0/498532983' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:01.584759+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:01.587045+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:01.587715+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:02.033258+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:02.033258+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:03.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:02.033258+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:04.018519+0000 mon.b (mon.1) 14 : audit [INF] from='client.? v1:172.21.15.195:0/1820868768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:04.019000+0000 mon.a (mon.0) 431 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:04.031068+0000 mon.a (mon.0) 432 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]': finished 2024-02-10T08:32:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:04.031132+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-10T08:32:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:04.031286+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:04.018519+0000 mon.b (mon.1) 14 : audit [INF] from='client.? v1:172.21.15.195:0/1820868768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:04.019000+0000 mon.a (mon.0) 431 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:04.031068+0000 mon.a (mon.0) 432 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]': finished 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:04.031132+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:04.031286+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:04.018519+0000 mon.b (mon.1) 14 : audit [INF] from='client.? v1:172.21.15.195:0/1820868768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:04.019000+0000 mon.a (mon.0) 431 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]: dispatch 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:04.031068+0000 mon.a (mon.0) 432 : audit [INF] from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5"}]': finished 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:04.031132+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-10T08:32:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:04.031286+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:04.033570+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:04.610363+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/324907243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:04.033570+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:04.610363+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/324907243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:04.033570+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:04.610363+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/324907243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:06.407 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:06.406+0000 7fe8115f3700 10 client.24256.objecter tick 2024-02-10T08:32:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:06.033880+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:06.033880+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:06.033880+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:08.034148+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:08.034148+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:09.940 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:08.034148+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:10.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:10.274653+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T08:32:10.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:10.275436+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:10.274653+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T08:32:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:10.275436+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:11.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:10.274653+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T08:32:11.004 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:10.275436+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:11.407 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:11.407+0000 7fe8115f3700 10 client.24256.objecter tick 2024-02-10T08:32:11.652 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:10.034451+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:11.652 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:10.276056+0000 mgr.y (mgr.14146) 152 : cephadm [INF] Deploying daemon osd.5 on smithi195 2024-02-10T08:32:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:10.034451+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:10.276056+0000 mgr.y (mgr.14146) 152 : cephadm [INF] Deploying daemon osd.5 on smithi195 2024-02-10T08:32:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:10.034451+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:10.276056+0000 mgr.y (mgr.14146) 152 : cephadm [INF] Deploying daemon osd.5 on smithi195 2024-02-10T08:32:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:12.034867+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:12.034867+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:12.034867+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:15.782 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:14.035277+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:15.782 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:15.386409+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:15.782 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:15.412525+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:15.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:14.035277+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:15.386409+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:15.412525+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:15.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:14.035277+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:15.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:15.386409+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:15.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:15.412525+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:16.408 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:16.408+0000 7fe8115f3700 10 client.24256.objecter tick 2024-02-10T08:32:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:16.035739+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:16.035739+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:16.035739+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:18.878 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 5 on host 'smithi195' 2024-02-10T08:32:18.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.877+0000 7fe808ff9700 1 -- 172.21.15.195:0/498532983 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 498095474) 0x7fe7f8000b90 con 0x7fe7f4063e60 2024-02-10T08:32:18.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.879+0000 7fe811df4700 10 librados: watch_flush enter 2024-02-10T08:32:18.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.879+0000 7fe811df4700 10 librados: watch_flush exit 2024-02-10T08:32:18.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.879+0000 7fe811df4700 20 client.24256.objecter shutdown clearing up homeless session... 2024-02-10T08:32:18.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.879+0000 7fe811df4700 10 client.24256.objecter successfully canceled tick 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 >> v1:172.21.15.107:6800/3360068681 conn(0x7fe7f4063e60 legacy=0x7fe7f4066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 >> v1:172.21.15.195:6789/0 conn(0x7fe80c0ff9d0 legacy=0x7fe80c113490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 shutdown_connections 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 >> 172.21.15.195:0/498532983 conn(0x7fe80c0f3050 msgr2=0x7fe80c0f3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 shutdown_connections 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 -- 172.21.15.195:0/498532983 wait complete. 2024-02-10T08:32:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:18.880+0000 7fe811df4700 1 librados: shutdown 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:17.963380+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:17.964189+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:17.964984+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:17.972197+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:18.036098+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.518503+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.518887+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.877786+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.161 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.890363+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:17.963380+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:17.964189+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:17.964984+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:17.972197+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:18.036098+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.518503+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.518887+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.877786+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.890363+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:17.963380+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:17.964189+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:17.964984+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:17.972197+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:18.036098+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:19.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.518503+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.518887+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T08:32:19.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.877786+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:19.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.890363+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:19.502 DEBUG:teuthology.orchestra.run.smithi195:osd.5> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T08:32:19.504 INFO:tasks.cephadm:Deploying osd.6 on smithi195 with /dev/vg_nvme/lv_2... 2024-02-10T08:32:19.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T08:32:19.534 INFO:journalctl@ceph.osd.5.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:32:20.091 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:32:19.985+0000 7fa4b2652700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T08:32:20.091 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:32:19.990+0000 7fa4aadc6700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:18.514942+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:18.517978+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.982792+0000 mon.a (mon.0) 446 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:18.982866+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.983017+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.983330+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:20.092 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:18.983702+0000 mon.a (mon.0) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:18.514942+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:32:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:18.517978+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:32:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.982792+0000 mon.a (mon.0) 446 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:18.982866+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.983017+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.983330+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:18.983702+0000 mon.a (mon.0) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:18.514942+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:18.517978+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.982792+0000 mon.a (mon.0) 446 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:18.982866+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.983017+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.983330+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.195:6804/4178321486' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:20.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:18.983702+0000 mon.a (mon.0) 449 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:21.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:19.984028+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:21.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:19.984131+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-10T08:32:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:19.984803+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:19.986902+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:20.036451+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:20.986824+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:19.984028+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:19.984131+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-10T08:32:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:19.984803+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:19.986902+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:20.036451+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:20.986824+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:19.984028+0000 mon.a (mon.0) 450 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:19.984131+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:19.984803+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:19.986902+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:20.036451+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T08:32:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:20.986824+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:19.550093+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:19.550152+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:20.994877+0000 mon.a (mon.0) 455 : cluster [INF] osd.5 v1:172.21.15.195:6804/4178321486 boot 2024-02-10T08:32:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:20.994915+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-10T08:32:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:20.995423+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:19.550093+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:19.550152+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:20.994877+0000 mon.a (mon.0) 455 : cluster [INF] osd.5 v1:172.21.15.195:6804/4178321486 boot 2024-02-10T08:32:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:20.994915+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:20.995423+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:19.550093+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:19.550152+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:20.994877+0000 mon.a (mon.0) 455 : cluster [INF] osd.5 v1:172.21.15.195:6804/4178321486 boot 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:20.994915+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-10T08:32:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:20.995423+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:32:22.572 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T08:32:22.573 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T08:32:22.573 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-10T08:32:22.573 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-10T08:32:22.573 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00830327 s, 1.3 GB/s 2024-02-10T08:32:22.573 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-10T08:32:23.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:21.999364+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-10T08:32:23.266 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:22.036793+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:23.266 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:22.999163+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-10T08:32:23.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:21.999364+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-10T08:32:23.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:22.036793+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:23.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:22.999163+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-10T08:32:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:21.999364+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-10T08:32:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:22.036793+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:22.999163+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-10T08:32:23.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_2 2024-02-10T08:32:25.736 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:24.037202+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:24.460529+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.465924+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.466920+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.467590+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:24.468068+0000 mgr.y (mgr.14146) 161 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 8561M 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.472544+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.473413+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.474039+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:25.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:24.478813+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:24.037202+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:24.460529+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.465924+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.466920+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.467590+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:24.468068+0000 mgr.y (mgr.14146) 161 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 8561M 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.472544+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.473413+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.474039+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:24.478813+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:24.037202+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:24.460529+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.465924+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.466920+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.467590+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:24.468068+0000 mgr.y (mgr.14146) 161 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 8561M 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.472544+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.473413+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.474039+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:24.478813+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:25.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.972+0000 7fc14f9d1700 1 -- 172.21.15.195:0/723966234 >> v1:172.21.15.195:6789/0 conn(0x7fc1480f78b0 legacy=0x7fc1480f7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.972+0000 7fc14f9d1700 1 -- 172.21.15.195:0/723966234 shutdown_connections 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.972+0000 7fc14f9d1700 1 -- 172.21.15.195:0/723966234 >> 172.21.15.195:0/723966234 conn(0x7fc1480f3010 msgr2=0x7fc1480f5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.972+0000 7fc14f9d1700 1 -- 172.21.15.195:0/723966234 shutdown_connections 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 -- 172.21.15.195:0/723966234 wait complete. 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 librados: starting msgr at 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 librados: starting objecter 2024-02-10T08:32:25.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 Processor -- start 2024-02-10T08:32:25.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 -- start start 2024-02-10T08:32:25.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 librados: setting wanted keys 2024-02-10T08:32:25.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.973+0000 7fc14f9d1700 1 librados: calling monclient init 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1481086c0 con 0x7fc1480ff9b0 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc148113f20 con 0x7fc1480f78b0 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14d76d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc1480f78b0 0x7fc148107af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:50230/0 (socket says 172.21.15.195:50230) 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc148115100 con 0x7fc1480fbd80 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14d76d700 1 -- 172.21.15.195:0/500533034 learned_addr learned my addr 172.21.15.195:0/500533034 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter ms_handle_connect 0x7fc1480f78b0 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter ms_handle_connect 0x7fc1480ff9b0 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:25.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter ms_handle_connect 0x7fc1480fbd80 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561276898 0 0) 0x7fc148113f20 con 0x7fc1480f78b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc134002220 con 0x7fc1480f78b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280939680 0 0) 0x7fc1481086c0 con 0x7fc1480ff9b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc148113f20 con 0x7fc1480ff9b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816986719 0 0) 0x7fc148115100 con 0x7fc1480fbd80 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1481086c0 con 0x7fc1480fbd80 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14571425 0 0) 0x7fc134002220 con 0x7fc1480f78b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc148115100 con 0x7fc1480f78b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc138001a60 con 0x7fc1480f78b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750270430 0 0) 0x7fc148113f20 con 0x7fc1480ff9b0 2024-02-10T08:32:25.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc134002220 con 0x7fc1480ff9b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc140001b80 con 0x7fc1480ff9b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2247026377 0 0) 0x7fc1481086c0 con 0x7fc1480fbd80 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc148113f20 con 0x7fc1480fbd80 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc144001730 con 0x7fc1480fbd80 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3101880074 0 0) 0x7fc148115100 con 0x7fc1480f78b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 >> v1:172.21.15.107:6790/0 conn(0x7fc1480fbd80 legacy=0x7fc14810e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 >> v1:172.21.15.107:6789/0 conn(0x7fc1480ff9b0 legacy=0x7fc1481117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1481162e0 con 0x7fc1480f78b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc148112f70 con 0x7fc1480f78b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 10 client.24283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1481133e0 con 0x7fc1480f78b0 2024-02-10T08:32:25.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.974+0000 7fc14f9d1700 1 librados: init done 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.975+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc138001d60 con 0x7fc1480f78b0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.975+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc138003840 con 0x7fc1480f78b0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.975+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c004f20 con 0x7fc1480f78b0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7fc138002fd0 con 0x7fc1480f78b0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3470+0+0 (unknown 2632120439 0 0) 0x7fc138077d40 con 0x7fc1480f78b0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 10 client.24283.objecter ms_dispatch 0x7fc148001550 osd_map(39..39 src has 1..39) v4 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 3 client.24283.objecter handle_osd_map got epochs [39,39] > 0 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 3 client.24283.objecter handle_osd_map decoding full epoch 39 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.976+0000 7fc13e7fc700 20 client.24283.objecter dump_active .. 0 homeless 2024-02-10T08:32:25.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.977+0000 7fc13e7fc700 10 client.24283.objecter ms_handle_connect 0x7fc134063c70 2024-02-10T08:32:25.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:25.980+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc138048920 con 0x7fc1480f78b0 2024-02-10T08:32:26.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:26.150+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12c000b90 con 0x7fc134063c70 2024-02-10T08:32:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:26.154303+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:26.156633+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:26.157354+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:26.154303+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:26.156633+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:26.157354+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:26.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:26.154303+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:26.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:26.156633+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:26.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:26.157354+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:27.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:26.037510+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:27.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:26.152047+0000 mgr.y (mgr.14146) 163 : audit [DBG] from='client.24283 v1:172.21.15.195:0/500533034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:26.037510+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:26.152047+0000 mgr.y (mgr.14146) 163 : audit [DBG] from='client.24283 v1:172.21.15.195:0/500533034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:26.037510+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:26.152047+0000 mgr.y (mgr.14146) 163 : audit [DBG] from='client.24283 v1:172.21.15.195:0/500533034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:28.037886+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:28.666663+0000 mon.b (mon.1) 18 : audit [INF] from='client.? v1:172.21.15.195:0/1432632268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:28.667297+0000 mon.a (mon.0) 470 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:28.671818+0000 mon.a (mon.0) 471 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]': finished 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:28.671870+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:28.672027+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:29.256991+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? v1:172.21.15.195:0/280148175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:29.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:28.037886+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:29.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:28.666663+0000 mon.b (mon.1) 18 : audit [INF] from='client.? v1:172.21.15.195:0/1432632268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:28.667297+0000 mon.a (mon.0) 470 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:28.671818+0000 mon.a (mon.0) 471 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]': finished 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:28.671870+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:28.672027+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:29.256991+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? v1:172.21.15.195:0/280148175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:28.037886+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:28.666663+0000 mon.b (mon.1) 18 : audit [INF] from='client.? v1:172.21.15.195:0/1432632268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:28.667297+0000 mon.a (mon.0) 470 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]: dispatch 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:28.671818+0000 mon.a (mon.0) 471 : audit [INF] from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031"}]': finished 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:28.671870+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-10T08:32:29.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:28.672027+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:29.256991+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? v1:172.21.15.195:0/280148175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:30.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:30.975+0000 7fc14f1d0700 10 client.24283.objecter tick 2024-02-10T08:32:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:30.038252+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:30.038252+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:30.038252+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:32.038781+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:33.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:32.038781+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:32.038781+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:34.039196+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:34.230891+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:34.230948+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:34.540925+0000 mon.a (mon.0) 474 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-10T08:32:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:34.541076+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:34.944512+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T08:32:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:34.945292+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:34.039196+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:34.230891+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:34.230948+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:34.540925+0000 mon.a (mon.0) 474 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:34.541076+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:34.944512+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:34.945292+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:34.039196+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:34.230891+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:34.230948+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:34.540925+0000 mon.a (mon.0) 474 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-10T08:32:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:34.541076+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:34.944512+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T08:32:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:34.945292+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:35.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:35.976+0000 7fc14f1d0700 10 client.24283.objecter tick 2024-02-10T08:32:36.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:34.945942+0000 mgr.y (mgr.14146) 168 : cephadm [INF] Deploying daemon osd.6 on smithi195 2024-02-10T08:32:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:34.945942+0000 mgr.y (mgr.14146) 168 : cephadm [INF] Deploying daemon osd.6 on smithi195 2024-02-10T08:32:36.949 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:34.945942+0000 mgr.y (mgr.14146) 168 : cephadm [INF] Deploying daemon osd.6 on smithi195 2024-02-10T08:32:37.833 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:36.039516+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:36.039516+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:37.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:36.039516+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:39.807 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:38.039918+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:38.039918+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:38.039918+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:40.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:40.977+0000 7fc14f1d0700 10 client.24283.objecter tick 2024-02-10T08:32:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:40.040177+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:40.040274+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:40.082417+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:40.040177+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:40.040274+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:40.082417+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:40.040177+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:40.040274+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:40.082417+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:42.040773+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:43.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:42.934124+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:43.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:42.040773+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:43.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:42.934124+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:42.040773+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:42.934124+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:43.641 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 6 on host 'smithi195' 2024-02-10T08:32:43.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.638+0000 7fc13e7fc700 1 -- 172.21.15.195:0/500533034 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4186859182) 0x7fc12c000b90 con 0x7fc134063c70 2024-02-10T08:32:43.642 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 10 librados: watch_flush enter 2024-02-10T08:32:43.642 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 10 librados: watch_flush exit 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 20 client.24283.objecter shutdown clearing up homeless session... 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 10 client.24283.objecter successfully canceled tick 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 >> v1:172.21.15.107:6800/3360068681 conn(0x7fc134063c70 legacy=0x7fc134066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 >> v1:172.21.15.195:6789/0 conn(0x7fc1480f78b0 legacy=0x7fc148107af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 shutdown_connections 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 >> 172.21.15.195:0/500533034 conn(0x7fc1480f3010 msgr2=0x7fc1480ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 shutdown_connections 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 -- 172.21.15.195:0/500533034 wait complete. 2024-02-10T08:32:43.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:43.641+0000 7fc14f9d1700 1 librados: shutdown 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.157212+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.157811+0000 mon.a (mon.0) 481 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.639560+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.749779+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.750474+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.751072+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.755809+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.756740+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.757491+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.764656+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:43.766951+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.157212+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.157811+0000 mon.a (mon.0) 481 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.639560+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.749779+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.750474+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.751072+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.755809+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.756740+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.757491+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.764656+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:43.766951+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.157212+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.157811+0000 mon.a (mon.0) 481 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.639560+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.749779+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.750474+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.751072+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.755809+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.756740+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.757491+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.764656+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:43.766951+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:32:44.817 DEBUG:teuthology.orchestra.run.smithi195:osd.6> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T08:32:44.819 INFO:tasks.cephadm:Deploying osd.7 on smithi195 with /dev/vg_nvme/lv_1... 2024-02-10T08:32:44.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T08:32:44.849 INFO:journalctl@ceph.osd.6.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:32:45.405 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:43.751538+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 5707M 2024-02-10T08:32:45.405 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:44.041265+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:44.054745+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:44.054828+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:44.055416+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:44.057685+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:44.058296+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:45.053691+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:45.053796+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:45.054089+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:45.056789+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.406 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:32:45.054+0000 7fe9a830f700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T08:32:45.406 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:32:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:32:45.061+0000 7fe9a2a87700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:43.751538+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 5707M 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:44.041265+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:44.054745+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:44.054828+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:44.055416+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:44.057685+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:44.058296+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:45.053691+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:45.053796+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:45.054089+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:45.056789+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:43.751538+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 5707M 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:44.041265+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:44.054745+0000 mon.a (mon.0) 491 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:44.054828+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:44.055416+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.195:6808/1417479017' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:44.057685+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:44.058296+0000 mon.a (mon.0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:45.053691+0000 mon.a (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:32:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:45.053796+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-10T08:32:45.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:45.054089+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:45.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:45.056789+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:47.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:44.174192+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:47.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:44.174239+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:47.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:46.041732+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:47.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:46.056382+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.195:6808/1417479017 boot 2024-02-10T08:32:47.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:46.056442+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-10T08:32:47.328 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:46.056613+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:47.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:44.174192+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:44.174239+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:46.041732+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:46.056382+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.195:6808/1417479017 boot 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:46.056442+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:46.056613+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:44.174192+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:44.174239+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:46.041732+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:46.056382+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v1:172.21.15.195:6808/1417479017 boot 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:46.056442+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-10T08:32:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:46.056613+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00817528 s, 1.3 GB/s 2024-02-10T08:32:47.934 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-02-10T08:32:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:47.057184+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-10T08:32:48.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:47.057184+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-10T08:32:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:47.057184+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-10T08:32:49.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi195:vg_nvme/lv_1 2024-02-10T08:32:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:48.042090+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:48.059805+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-10T08:32:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:48.042090+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:48.059805+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-10T08:32:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:48.042090+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:48.059805+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-10T08:32:50.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:49.343931+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:50.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:49.351406+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:50.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:49.352267+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:50.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:49.353019+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:50.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:49.359938+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:49.343931+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:49.351406+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:49.352267+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:49.353019+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:49.359938+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:49.343931+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:32:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:49.351406+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:50.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:49.352267+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:50.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:49.353019+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:32:50.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:49.359938+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.118+0000 7f5eebcf1700 1 -- 172.21.15.195:0/2698412335 >> v1:172.21.15.195:6789/0 conn(0x7f5ee40f67c0 legacy=0x7f5ee40f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.118+0000 7f5eebcf1700 1 -- 172.21.15.195:0/2698412335 shutdown_connections 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.118+0000 7f5eebcf1700 1 -- 172.21.15.195:0/2698412335 >> 172.21.15.195:0/2698412335 conn(0x7f5ee40f1f20 msgr2=0x7f5ee40f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- 172.21.15.195:0/2698412335 shutdown_connections 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- 172.21.15.195:0/2698412335 wait complete. 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 librados: starting msgr at 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 librados: starting objecter 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 Processor -- start 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- start start 2024-02-10T08:32:51.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 librados: setting wanted keys 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 librados: calling monclient init 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ee4114cb0 con 0x7f5ee40fac90 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ee4115eb0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eebcf1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ee41170b0 con 0x7f5ee40f67c0 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eea28e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5ee40fe8c0 0x7f5ee4113370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:51924/0 (socket says 172.21.15.195:51924) 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.119+0000 7f5eea28e700 1 -- 172.21.15.195:0/3202044992 learned_addr learned my addr 172.21.15.195:0/3202044992 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter ms_handle_connect 0x7f5ee40fe8c0 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter ms_handle_connect 0x7f5ee40fac90 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:51.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter ms_handle_connect 0x7f5ee40f67c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794804910 0 0) 0x7f5ee4115eb0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0002220 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105692914 0 0) 0x7f5ee4114cb0 con 0x7f5ee40fac90 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee4115eb0 con 0x7f5ee40fac90 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713476717 0 0) 0x7f5ee41170b0 con 0x7f5ee40f67c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee4114cb0 con 0x7f5ee40f67c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3249304011 0 0) 0x7f5ed0002220 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee41170b0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5edc002c00 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4208325598 0 0) 0x7f5ee4115eb0 con 0x7f5ee40fac90 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed0002220 con 0x7f5ee40fac90 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5ed40017f0 con 0x7f5ee40fac90 2024-02-10T08:32:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901750088 0 0) 0x7f5ee41170b0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 >> v1:172.21.15.107:6790/0 conn(0x7f5ee40f67c0 legacy=0x7f5ee410c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 >> v1:172.21.15.107:6789/0 conn(0x7f5ee40fac90 legacy=0x7f5ee410fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee41182b0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee4114ea0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5eebcf1700 10 client.24310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee4115310 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.120+0000 7f5eebcf1700 1 librados: init done 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.121+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5edc0016f0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.121+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5edc005d10 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.122+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec8004f20 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.122+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f5edc018c20 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(47..47 src has 1..47) v4 ==== 3762+0+0 (unknown 1008712802 0 0) 0x7f5edc077e70 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 10 client.24310.objecter ms_dispatch 0x7f5ee4001550 osd_map(47..47 src has 1..47) v4 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 3 client.24310.objecter handle_osd_map got epochs [47,47] > 0 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 3 client.24310.objecter handle_osd_map decoding full epoch 47 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 20 client.24310.objecter dump_active .. 0 homeless 2024-02-10T08:32:51.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.123+0000 7f5edaffd700 10 client.24310.objecter ms_handle_connect 0x7f5ed0063990 2024-02-10T08:32:51.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.127+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5edc0495d0 con 0x7f5ee40fe8c0 2024-02-10T08:32:51.296 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:51.295+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ec8000b90 con 0x7f5ed0063990 2024-02-10T08:32:51.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:49.998760+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:51.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:49.998818+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:51.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:50.042536+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:51.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:50.537377+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T08:32:51.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:51.299922+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:51.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:51.302090+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:51.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:51.302833+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:51.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:49.998760+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:51.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:49.998818+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:50.042536+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:50.537377+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:51.299922+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:51.302090+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:51.302833+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:49.998760+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:49.998818+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:50.042536+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:50.537377+0000 mon.a (mon.0) 508 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:51.299922+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T08:32:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:51.302090+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T08:32:51.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:51.302833+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:32:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:51.297571+0000 mgr.y (mgr.14146) 179 : audit [DBG] from='client.24310 v1:172.21.15.195:0/3202044992' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:52.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:51.297571+0000 mgr.y (mgr.14146) 179 : audit [DBG] from='client.24310 v1:172.21.15.195:0/3202044992' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:51.297571+0000 mgr.y (mgr.14146) 179 : audit [DBG] from='client.24310 v1:172.21.15.195:0/3202044992' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:32:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:52.042916+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:52.042916+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:54.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:52.042916+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:53.708730+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.195:0/581334874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:53.709306+0000 mon.a (mon.0) 512 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:53.713696+0000 mon.a (mon.0) 513 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]': finished 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:53.713796+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:53.713940+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:54.300044+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? v1:172.21.15.195:0/2775378900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:53.708730+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.195:0/581334874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:53.709306+0000 mon.a (mon.0) 512 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:53.713696+0000 mon.a (mon.0) 513 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]': finished 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:53.713796+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:53.713940+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:32:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:54.300044+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? v1:172.21.15.195:0/2775378900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:53.708730+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.195:0/581334874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:53.709306+0000 mon.a (mon.0) 512 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]: dispatch 2024-02-10T08:32:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:53.713696+0000 mon.a (mon.0) 513 : audit [INF] from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948"}]': finished 2024-02-10T08:32:55.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:53.713796+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-10T08:32:55.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:53.713940+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:32:55.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:54.300044+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? v1:172.21.15.195:0/2775378900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T08:32:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:54.043417+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:56.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:32:56.121+0000 7f5eeb4f0700 10 client.24310.objecter tick 2024-02-10T08:32:56.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:54.043417+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:56.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:54.043417+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:56.043825+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:58.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:56.043825+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:58.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:56.043825+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:58.866 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:57.928209+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:58.866 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:57.928269+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:59.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:57.928209+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:59.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:57.928269+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:59.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:57.928209+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:32:59.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:57.928269+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:32:59.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:58.044325+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:32:59.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:32:58.724564+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-10T08:32:59.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:32:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:58.724746+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:58.044325+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:32:58.724564+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-10T08:33:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:58.724746+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:58.044325+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:32:58.724564+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-10T08:33:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:32:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:58.724746+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:59.956992+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T08:33:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:32:59.957717+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:01.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:01.122+0000 7f5eeb4f0700 10 client.24310.objecter tick 2024-02-10T08:33:01.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:59.956992+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T08:33:01.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:32:59.957717+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:01.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:59.956992+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T08:33:01.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:32:59.957717+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:01.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:32:59.958330+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi195 2024-02-10T08:33:01.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:00.044803+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:02.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:32:59.958330+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi195 2024-02-10T08:33:02.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:00.044803+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:02.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:32:59.958330+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi195 2024-02-10T08:33:02.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:00.044803+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:02.045297+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:04.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:02.045297+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:04.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:02.045297+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:05.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:04.045632+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:05.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:05.053510+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:05.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:05.064173+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:06.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:06.123+0000 7f5eeb4f0700 10 client.24310.objecter tick 2024-02-10T08:33:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:04.045632+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:05.053510+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:05.064173+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:06.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:04.045632+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:06.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:05.053510+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:05.064173+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:06.046119+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:07.656945+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:07.657772+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:07.658536+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:07.663617+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:06.046119+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:07.656945+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:07.657772+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:07.658536+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:07.663617+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:06.046119+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:07.656945+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:07.657772+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:07.658536+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:07.663617+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:08.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.442+0000 7f5edaffd700 1 -- 172.21.15.195:0/3202044992 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2778433562) 0x7f5ec8000b90 con 0x7f5ed0063990 2024-02-10T08:33:08.443 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 7 on host 'smithi195' 2024-02-10T08:33:08.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.443+0000 7f5eebcf1700 10 librados: watch_flush enter 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.443+0000 7f5eebcf1700 10 librados: watch_flush exit 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.443+0000 7f5eebcf1700 20 client.24310.objecter shutdown clearing up homeless session... 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.443+0000 7f5eebcf1700 10 client.24310.objecter successfully canceled tick 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 >> v1:172.21.15.107:6800/3360068681 conn(0x7f5ed0063990 legacy=0x7f5ed0065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 >> v1:172.21.15.195:6789/0 conn(0x7f5ee40fe8c0 legacy=0x7f5ee4113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:08.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 shutdown_connections 2024-02-10T08:33:08.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 >> 172.21.15.195:0/3202044992 conn(0x7f5ee40f1f20 msgr2=0x7f5ee4060af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:08.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 shutdown_connections 2024-02-10T08:33:08.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 -- 172.21.15.195:0/3202044992 wait complete. 2024-02-10T08:33:08.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:08.444+0000 7f5eebcf1700 1 librados: shutdown 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:07.851505+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:07.851556+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.153574+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.153982+0000 mon.a (mon.0) 526 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.441792+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.481709+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:09.164 DEBUG:teuthology.orchestra.run.smithi195:osd.7> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T08:33:09.166 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T08:33:09.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T08:33:09.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:07.851505+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:09.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:07.851556+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:09.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.153574+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.153982+0000 mon.a (mon.0) 526 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.441792+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.481709+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:07.851505+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:07.851556+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.153574+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.153982+0000 mon.a (mon.0) 526 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.441792+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:09.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.481709+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:09.201 INFO:journalctl@ceph.osd.7.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:08.046530+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.749347+0000 mon.a (mon.0) 529 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:08.749405+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.749848+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.749861+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:08.750251+0000 mon.a (mon.0) 532 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:08.046530+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:09.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.749347+0000 mon.a (mon.0) 529 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T08:33:09.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:08.749405+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-10T08:33:09.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.749848+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:09.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.749861+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:09.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:08.750251+0000 mon.a (mon.0) 532 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:08.046530+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.749347+0000 mon.a (mon.0) 529 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:08.749405+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.749848+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.749861+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.195:6812/1934176928' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:10.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:08.750251+0000 mon.a (mon.0) 532 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:33:10.039 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:33:09.758+0000 7faf3bdd2700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T08:33:10.039 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:33:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:33:09.765+0000 7faf3654a700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:33:10.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:09.752284+0000 mon.a (mon.0) 533 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:33:10.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:09.752368+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-10T08:33:10.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:09.752838+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:09.755291+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:10.046980+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:10.751422+0000 mon.a (mon.0) 537 : cluster [INF] osd.7 v1:172.21.15.195:6812/1934176928 boot 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:10.751462+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:10.751605+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:09.752284+0000 mon.a (mon.0) 533 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:09.752368+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:09.752838+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:09.755291+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:10.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:10.046980+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:10.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:10.751422+0000 mon.a (mon.0) 537 : cluster [INF] osd.7 v1:172.21.15.195:6812/1934176928 boot 2024-02-10T08:33:10.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:10.751462+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T08:33:10.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:10.751605+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:11.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:09.752284+0000 mon.a (mon.0) 533 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-02-10T08:33:11.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:09.752368+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-10T08:33:11.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:09.752838+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:11.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:09.755291+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:11.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:10.046980+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:33:11.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:10.751422+0000 mon.a (mon.0) 537 : cluster [INF] osd.7 v1:172.21.15.195:6812/1934176928 boot 2024-02-10T08:33:11.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:10.751462+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-10T08:33:11.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:10.751605+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.118+0000 7f0faac7d700 1 -- 172.21.15.107:0/3318394045 >> v1:172.21.15.107:6789/0 conn(0x7f0fa40f78d0 legacy=0x7f0fa40f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.118+0000 7f0faac7d700 1 -- 172.21.15.107:0/3318394045 shutdown_connections 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.118+0000 7f0faac7d700 1 -- 172.21.15.107:0/3318394045 >> 172.21.15.107:0/3318394045 conn(0x7f0fa40f3050 msgr2=0x7f0fa40f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.118+0000 7f0faac7d700 1 -- 172.21.15.107:0/3318394045 shutdown_connections 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.118+0000 7f0faac7d700 1 -- 172.21.15.107:0/3318394045 wait complete. 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 librados: starting msgr at 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 librados: starting objecter 2024-02-10T08:33:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 Processor -- start 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 -- start start 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 librados: setting wanted keys 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 librados: calling monclient init 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fa4114c10 con 0x7f0fa40fbda0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fa4115e10 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0faac7d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fa4117010 con 0x7f0fa40f78d0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa3fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0fa40fbda0 0x7f0fa40fe4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38110/0 (socket says 172.21.15.107:38110) 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa3fff700 1 -- 172.21.15.107:0/2739090974 learned_addr learned my addr 172.21.15.107:0/2739090974 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter ms_handle_connect 0x7f0fa40fbda0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter ms_handle_connect 0x7f0fa40ff9d0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter ms_handle_connect 0x7f0fa40f78d0 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.119+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885270071 0 0) 0x7f0fa4114c10 con 0x7f0fa40fbda0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f940033f0 con 0x7f0fa40fbda0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115009255 0 0) 0x7f0fa4115e10 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4114c10 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115845468 0 0) 0x7f0fa4117010 con 0x7f0fa40f78d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4115e10 con 0x7f0fa40f78d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867565827 0 0) 0x7f0f940033f0 con 0x7f0fa40fbda0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa4117010 con 0x7f0fa40fbda0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0f90001980 con 0x7f0fa40fbda0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2831365943 0 0) 0x7f0fa4114c10 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f940033f0 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0f9c001b30 con 0x7f0fa40ff9d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361068317 0 0) 0x7f0fa4115e10 con 0x7f0fa40f78d0 2024-02-10T08:33:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa4114c10 con 0x7f0fa40f78d0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0f98002df0 con 0x7f0fa40f78d0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2914315940 0 0) 0x7f0fa4117010 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 >> v1:172.21.15.107:6790/0 conn(0x7f0fa40f78d0 legacy=0x7f0fa40fdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 >> v1:172.21.15.195:6789/0 conn(0x7f0fa40ff9d0 legacy=0x7f0fa4113460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa4118210 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa4116000 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0f90002100 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0f90005ab0 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0faac7d700 10 client.14424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa41164f0 con 0x7f0fa40fbda0 2024-02-10T08:33:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.120+0000 7f0faac7d700 1 librados: init done 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.121+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa40475e0 con 0x7f0fa40fbda0 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f0f90001c60 con 0x7f0fa40fbda0 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 4087+0+0 (unknown 2636531135 0 0) 0x7f0f90077b60 con 0x7f0fa40fbda0 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 10 client.14424.objecter ms_dispatch 0x7f0fa4001550 osd_map(52..52 src has 1..52) v4 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 3 client.14424.objecter handle_osd_map got epochs [52,52] > 0 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 3 client.14424.objecter handle_osd_map decoding full epoch 52 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 20 client.14424.objecter dump_active .. 0 homeless 2024-02-10T08:33:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.122+0000 7f0fa1ffb700 10 client.14424.objecter ms_handle_connect 0x7f0f94063bf0 2024-02-10T08:33:11.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.126+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0f90048420 con 0x7f0fa40fbda0 2024-02-10T08:33:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.287+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0fa4116890 con 0x7f0fa40fbda0 2024-02-10T08:33:11.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.287+0000 7f0fa1ffb700 1 -- 172.21.15.107:0/2739090974 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v52) v1 ==== 74+0+130 (unknown 217607441 0 1327026452) 0x7f0f9004a350 con 0x7f0fa40fbda0 2024-02-10T08:33:11.289 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 10 librados: watch_flush enter 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 10 librados: watch_flush exit 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 20 client.14424.objecter shutdown clearing up homeless session... 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 10 client.14424.objecter successfully canceled tick 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0f94063bf0 legacy=0x7f0f940660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 >> v1:172.21.15.107:6789/0 conn(0x7f0fa40fbda0 legacy=0x7f0fa40fe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 shutdown_connections 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 >> 172.21.15.107:0/2739090974 conn(0x7f0fa40f3050 msgr2=0x7f0fa406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 shutdown_connections 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 -- 172.21.15.107:0/2739090974 wait complete. 2024-02-10T08:33:11.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:11.289+0000 7f0faac7d700 1 librados: shutdown 2024-02-10T08:33:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:09.106964+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:09.107048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:11.289340+0000 mon.a (mon.0) 540 : audit [DBG] from='client.? v1:172.21.15.107:0/2739090974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T08:33:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:11.755448+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T08:33:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:09.106964+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:09.107048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:11.289340+0000 mon.a (mon.0) 540 : audit [DBG] from='client.? v1:172.21.15.107:0/2739090974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T08:33:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:11.755448+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T08:33:11.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:09.106964+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:11.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:09.107048+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:11.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:11.289340+0000 mon.a (mon.0) 540 : audit [DBG] from='client.? v1:172.21.15.107:0/2739090974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T08:33:11.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:11.755448+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-10T08:33:12.063 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1707553990,"num_in_osds":8,"osd_in_since":1707553973,"num_remapped_pgs":1} 2024-02-10T08:33:12.063 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi107 2024-02-10T08:33:12.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi107=foo.a' 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:12.047324+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:12.184367+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.190844+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.191686+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.192293+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.192843+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.193474+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:12.193925+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4280M 2024-02-10T08:33:13.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.199849+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.420 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.200578+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:13.420 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.201186+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:13.420 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:12.207273+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.420 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:12.759151+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:12.047324+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:12.184367+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.190844+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.191686+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.192293+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.192843+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.193474+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:12.193925+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4280M 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.199849+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.200578+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.201186+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:12.207273+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:12.759151+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:12.047324+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:12.184367+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Detected new or changed devices on smithi195 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.190844+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.191686+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.192293+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.192843+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.193474+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:12.193925+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4280M 2024-02-10T08:33:13.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.199849+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.200578+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:13.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.201186+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:13.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:12.207273+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:13.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:12.759151+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.894+0000 7fa4e6a52700 1 -- 172.21.15.195:0/634636509 >> v1:172.21.15.195:6789/0 conn(0x7fa4e00fbda0 legacy=0x7fa4e00fe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- 172.21.15.195:0/634636509 shutdown_connections 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- 172.21.15.195:0/634636509 >> 172.21.15.195:0/634636509 conn(0x7fa4e00f3010 msgr2=0x7fa4e00f5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- 172.21.15.195:0/634636509 shutdown_connections 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- 172.21.15.195:0/634636509 wait complete. 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 librados: starting msgr at 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 librados: starting objecter 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 Processor -- start 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- start start 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 librados: setting wanted keys 2024-02-10T08:33:13.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 librados: calling monclient init 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4e0114de0 con 0x7fa4e00fbda0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4e0115fe0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.895+0000 7fa4e6a52700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4e01171e0 con 0x7fa4e00f78d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e4fef700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa4e00ff9d0 0x7fa4e01134a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:33648/0 (socket says 172.21.15.195:33648) 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e4fef700 1 -- 172.21.15.195:0/748323887 learned_addr learned my addr 172.21.15.195:0/748323887 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fa4e00ff9d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fa4e00f78d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fa4e00fbda0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836144051 0 0) 0x7fa4e0115fe0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c8002220 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780480353 0 0) 0x7fa4e01171e0 con 0x7fa4e00f78d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e0115fe0 con 0x7fa4e00f78d0 2024-02-10T08:33:13.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806704289 0 0) 0x7fa4e0114de0 con 0x7fa4e00fbda0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e01171e0 con 0x7fa4e00fbda0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452851633 0 0) 0x7fa4c8002220 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4e0114de0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa4d8001cd0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259851450 0 0) 0x7fa4e0115fe0 con 0x7fa4e00f78d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4c8002220 con 0x7fa4e00f78d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa4d00018c0 con 0x7fa4e00f78d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076085422 0 0) 0x7fa4e01171e0 con 0x7fa4e00fbda0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4e0115fe0 con 0x7fa4e00fbda0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa4d4001430 con 0x7fa4e00fbda0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3659667384 0 0) 0x7fa4e0114de0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 >> v1:172.21.15.107:6790/0 conn(0x7fa4e00f78d0 legacy=0x7fa4e00fdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 >> v1:172.21.15.107:6789/0 conn(0x7fa4e00fbda0 legacy=0x7fa4e00fe6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e01183e0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01173d0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e6a52700 10 client.24343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01178c0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4e6a52700 1 librados: init done 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa4d8002060 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.896+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa4d8005f40 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.897+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc004f20 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7fa4d8018e10 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(54..54 src has 1..54) v4 ==== 4054+0+0 (unknown 24812548 0 0) 0x7fa4d8077fe0 con 0x7fa4e00ff9d0 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 10 client.24343.objecter ms_dispatch 0x7fa4e0001550 osd_map(54..54 src has 1..54) v4 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 3 client.24343.objecter handle_osd_map got epochs [54,54] > 0 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 3 client.24343.objecter handle_osd_map decoding full epoch 54 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 20 client.24343.objecter dump_active .. 0 homeless 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.898+0000 7fa4dd7fa700 10 client.24343.objecter ms_handle_connect 0x7fa4c8063dc0 2024-02-10T08:33:13.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:13.902+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa4d8048900 con 0x7fa4e00ff9d0 2024-02-10T08:33:14.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.074+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4cc000b90 con 0x7fa4c8063dc0 2024-02-10T08:33:14.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.081+0000 7fa4dd7fa700 1 -- 172.21.15.195:0/748323887 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7fa4cc000b90 con 0x7fa4c8063dc0 2024-02-10T08:33:14.082 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled rgw.foo.a update... 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 10 librados: watch_flush enter 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 10 librados: watch_flush exit 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 20 client.24343.objecter shutdown clearing up homeless session... 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 10 client.24343.objecter successfully canceled tick 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 >> v1:172.21.15.107:6800/3360068681 conn(0x7fa4c8063dc0 legacy=0x7fa4c8066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 >> v1:172.21.15.195:6789/0 conn(0x7fa4e00ff9d0 legacy=0x7fa4e01134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 shutdown_connections 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 >> 172.21.15.195:0/748323887 conn(0x7fa4e00f3010 msgr2=0x7fa4e006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 shutdown_connections 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 -- 172.21.15.195:0/748323887 wait complete. 2024-02-10T08:33:14.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:14.083+0000 7fa4e6a52700 1 librados: shutdown 2024-02-10T08:33:14.761 DEBUG:teuthology.orchestra.run.smithi107:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@rgw.foo.a.service 2024-02-10T08:33:14.764 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi195 2024-02-10T08:33:14.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool create iscsi 3 3 replicated 2024-02-10T08:33:14.793 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:33:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:14.047821+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.076300+0000 mgr.y (mgr.14146) 195 : audit [DBG] from='client.24343 v1:172.21.15.195:0/748323887' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:14.077931+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.081653+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.127567+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.129097+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.129755+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.137803+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:14.140230+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.146234+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.146822+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.152587+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T08:33:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.159014+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:14.159712+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:14.160379+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Deploying daemon rgw.foo.a on smithi107 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:14.047821+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.076300+0000 mgr.y (mgr.14146) 195 : audit [DBG] from='client.24343 v1:172.21.15.195:0/748323887' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:14.077931+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.081653+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.127567+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.129097+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.129755+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.137803+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:14.140230+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.146234+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.146822+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.152587+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.159014+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:14.159712+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:14.160379+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Deploying daemon rgw.foo.a on smithi107 2024-02-10T08:33:15.458 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:14.047821+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.076300+0000 mgr.y (mgr.14146) 195 : audit [DBG] from='client.24343 v1:172.21.15.195:0/748323887' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:14.077931+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.081653+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.127567+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.129097+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.129755+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.137803+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:14.140230+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.146234+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.146822+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.152587+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.159014+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:15.459 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:14.159712+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:15.460 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:14.160379+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Deploying daemon rgw.foo.a on smithi107 2024-02-10T08:33:15.876 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:33:15 smithi107 systemd[1]: Starting Ceph rgw.foo.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- 172.21.15.195:0/3599339455 >> v1:172.21.15.195:6789/0 conn(0x7f1e4c0ff9d0 legacy=0x7f1e4c101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- 172.21.15.195:0/3599339455 shutdown_connections 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- 172.21.15.195:0/3599339455 >> 172.21.15.195:0/3599339455 conn(0x7f1e4c0f3050 msgr2=0x7f1e4c0f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- 172.21.15.195:0/3599339455 shutdown_connections 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- 172.21.15.195:0/3599339455 wait complete. 2024-02-10T08:33:16.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 librados: starting msgr at 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 librados: starting objecter 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 Processor -- start 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 -- start start 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 librados: setting wanted keys 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.548+0000 7f1e523f0700 1 librados: calling monclient init 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e4c114c70 con 0x7f1e4c0fbda0 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e4c115e50 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e4c117050 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e5098d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1e4c0ff9d0 0x7f1e4c113330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:33680/0 (socket says 172.21.15.195:33680) 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e5098d700 1 -- 172.21.15.195:0/2208723620 learned_addr learned my addr 172.21.15.195:0/2208723620 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e4bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1e4c0f78d0 0x7f1e4c06ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.195:44534/0 (socket says 172.21.15.195:44534) 2024-02-10T08:33:16.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter ms_handle_connect 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter ms_handle_connect 0x7f1e4c0f78d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter ms_handle_connect 0x7f1e4c0fbda0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404288598 0 0) 0x7f1e4c115e50 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e34002220 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464630767 0 0) 0x7f1e4c117050 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e4c115e50 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693987678 0 0) 0x7f1e34002220 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e4c117050 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e44002570 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3455831193 0 0) 0x7f1e4c115e50 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e34002220 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e400018c0 con 0x7f1e4c0f78d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2788165920 0 0) 0x7f1e4c117050 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 >> v1:172.21.15.107:6790/0 conn(0x7f1e4c0f78d0 legacy=0x7f1e4c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 >> v1:172.21.15.107:6789/0 conn(0x7f1e4c0fbda0 legacy=0x7f1e4c10f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c118250 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c117240 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 10 client.24358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c117730 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e523f0700 1 librados: init done 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1e44002bb0 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.549+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e44003db0 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.550+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e38004f20 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f1e440035e0 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(54..54 src has 1..54) v4 ==== 4054+0+0 (unknown 24812548 0 0) 0x7f1e440783d0 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 10 client.24358.objecter ms_dispatch 0x7f1e4c001550 osd_map(54..54 src has 1..54) v4 2024-02-10T08:33:16.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 3 client.24358.objecter handle_osd_map got epochs [54,54] > 0 2024-02-10T08:33:16.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 3 client.24358.objecter handle_osd_map decoding full epoch 54 2024-02-10T08:33:16.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.551+0000 7f1e497fa700 20 client.24358.objecter dump_active .. 0 homeless 2024-02-10T08:33:16.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.552+0000 7f1e497fa700 10 client.24358.objecter ms_handle_connect 0x7f1e34063b10 2024-02-10T08:33:16.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.555+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1e44048d10 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:16.717+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f1e38005750 con 0x7f1e4c0ff9d0 2024-02-10T08:33:16.930 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:33:16 smithi107 bash[94716]: c836eaa20a5dacb96e5958ee852eef468c52e4d9439389751b65dafba626bca0 2024-02-10T08:33:16.930 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:33:16 smithi107 systemd[1]: Started Ceph rgw.foo.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:33:17.090 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.090+0000 7f1e497fa700 1 -- 172.21.15.195:0/2208723620 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'iscsi' created v55) v1 ==== 154+0+0 (unknown 2509653992 0 0) 0x7f1e4404ac40 con 0x7f1e4c0ff9d0 2024-02-10T08:33:17.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.093+0000 7f1e523f0700 10 librados: watch_flush enter 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.093+0000 7f1e523f0700 10 librados: watch_flush exit 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.093+0000 7f1e523f0700 20 client.24358.objecter shutdown clearing up homeless session... 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.093+0000 7f1e523f0700 10 client.24358.objecter successfully canceled tick 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 >> v1:172.21.15.107:6800/3360068681 conn(0x7f1e34063b10 legacy=0x7f1e34065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 >> v1:172.21.15.195:6789/0 conn(0x7f1e4c0ff9d0 legacy=0x7f1e4c113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 shutdown_connections 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 >> 172.21.15.195:0/2208723620 conn(0x7f1e4c0f3050 msgr2=0x7f1e4c0f3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 shutdown_connections 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 -- 172.21.15.195:0/2208723620 wait complete. 2024-02-10T08:33:17.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:17.094+0000 7f1e523f0700 1 librados: shutdown 2024-02-10T08:33:17.097 INFO:teuthology.orchestra.run.smithi195.stderr:pool 'iscsi' created 2024-02-10T08:33:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:16.048128+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:16.718095+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.195:0/2208723620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:16.729307+0000 mon.a (mon.0) 562 : audit [INF] from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:16.963602+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:16.964687+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:16.048128+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:16.718095+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.195:0/2208723620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:16.729307+0000 mon.a (mon.0) 562 : audit [INF] from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:16.963602+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:16.964687+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:17.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:16.048128+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:17.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:16.718095+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.195:0/2208723620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:16.729307+0000 mon.a (mon.0) 562 : audit [INF] from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T08:33:17.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:16.963602+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:17.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:16.964687+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:17.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool application enable iscsi rbd 2024-02-10T08:33:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:17.090633+0000 mon.a (mon.0) 565 : audit [INF] from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T08:33:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:17.090690+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T08:33:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:17.094736+0000 mon.c (mon.2) 10 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:17.098164+0000 mon.a (mon.0) 567 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:17.090633+0000 mon.a (mon.0) 565 : audit [INF] from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:17.090690+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:17.094736+0000 mon.c (mon.2) 10 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:17.098164+0000 mon.a (mon.0) 567 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:17.090633+0000 mon.a (mon.0) 565 : audit [INF] from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:17.090690+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-10T08:33:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:17.094736+0000 mon.c (mon.2) 10 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:17.098164+0000 mon.a (mon.0) 567 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:18.048481+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:18.097227+0000 mon.a (mon.0) 568 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:18.097288+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:18.383532+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:18.384283+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:18.384926+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:19.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:18.388789+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-10T08:33:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:18.048481+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:18.097227+0000 mon.a (mon.0) 568 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T08:33:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:18.097288+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T08:33:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:18.383532+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:18.384283+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:18.384926+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:18.388789+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:18.048481+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v189: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:18.097227+0000 mon.a (mon.0) 568 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:18.097288+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:18.383532+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:18.384283+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:18.384926+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:18.388789+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.528+0000 7fa6bc87d700 1 -- 172.21.15.195:0/1312239134 >> v1:172.21.15.195:6789/0 conn(0x7fa6b80ff9d0 legacy=0x7fa6b8101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.528+0000 7fa6bc87d700 1 -- 172.21.15.195:0/1312239134 shutdown_connections 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.528+0000 7fa6bc87d700 1 -- 172.21.15.195:0/1312239134 >> 172.21.15.195:0/1312239134 conn(0x7fa6b80f3050 msgr2=0x7fa6b80f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.528+0000 7fa6bc87d700 1 -- 172.21.15.195:0/1312239134 shutdown_connections 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.528+0000 7fa6bc87d700 1 -- 172.21.15.195:0/1312239134 wait complete. 2024-02-10T08:33:19.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 librados: starting msgr at 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 librados: starting objecter 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 Processor -- start 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 -- start start 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 librados: setting wanted keys 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 librados: calling monclient init 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6b8108650 con 0x7fa6b80fbda0 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6b8113eb0 con 0x7fa6b80f78d0 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6bc87d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6b8115090 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6b659c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa6b80f78d0 0x7fa6b8107a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:33694/0 (socket says 172.21.15.195:33694) 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6b659c700 1 -- 172.21.15.195:0/711553339 learned_addr learned my addr 172.21.15.195:0/711553339 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter ms_handle_connect 0x7fa6b80f78d0 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:19.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter ms_handle_connect 0x7fa6b80fbda0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter ms_handle_connect 0x7fa6b80ff9d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322816869 0 0) 0x7fa6b8113eb0 con 0x7fa6b80f78d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003030 con 0x7fa6b80f78d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.529+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252327112 0 0) 0x7fa6b8108650 con 0x7fa6b80fbda0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8113eb0 con 0x7fa6b80fbda0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091994233 0 0) 0x7fa6b8115090 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8108650 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097803165 0 0) 0x7fa6a0003030 con 0x7fa6b80f78d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b8115090 con 0x7fa6b80f78d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6ac002970 con 0x7fa6b80f78d0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246138613 0 0) 0x7fa6b8113eb0 con 0x7fa6b80fbda0 2024-02-10T08:33:19.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6a0003030 con 0x7fa6b80fbda0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2553868056 0 0) 0x7fa6b8108650 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b8113eb0 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6a80017f0 con 0x7fa6b80fbda0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6b0003280 con 0x7fa6b80ff9d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 80658570 0 0) 0x7fa6b8115090 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 >> v1:172.21.15.107:6790/0 conn(0x7fa6b80ff9d0 legacy=0x7fa6b8111780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 >> v1:172.21.15.107:6789/0 conn(0x7fa6b80fbda0 legacy=0x7fa6b810e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8116270 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b8112f00 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6bc87d700 10 client.24382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b81133f0 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6bc87d700 1 librados: init done 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa6ac002d20 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.530+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6ac005b20 con 0x7fa6b80f78d0 2024-02-10T08:33:19.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.531+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa698004f20 con 0x7fa6b80f78d0 2024-02-10T08:33:19.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.531+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7fa6ac0015e0 con 0x7fa6b80f78d0 2024-02-10T08:33:19.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(57..57 src has 1..57) v4 ==== 5111+0+0 (unknown 2552772759 0 0) 0x7fa6ac078120 con 0x7fa6b80f78d0 2024-02-10T08:33:19.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 10 client.24382.objecter ms_dispatch 0x7fa6b8001550 osd_map(57..57 src has 1..57) v4 2024-02-10T08:33:19.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 3 client.24382.objecter handle_osd_map got epochs [57,57] > 0 2024-02-10T08:33:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 3 client.24382.objecter handle_osd_map decoding full epoch 57 2024-02-10T08:33:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 20 client.24382.objecter dump_active .. 0 homeless 2024-02-10T08:33:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.532+0000 7fa6a77fe700 10 client.24382.objecter ms_handle_connect 0x7fa6a0063c70 2024-02-10T08:33:19.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.536+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa6ac048580 con 0x7fa6b80f78d0 2024-02-10T08:33:19.700 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:19.700+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"} v 0) v1 -- 0x7fa698005750 con 0x7fa6b80f78d0 2024-02-10T08:33:20.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.103+0000 7fa6a77fe700 1 -- 172.21.15.195:0/711553339 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]=0 enabled application 'rbd' on pool 'iscsi' v58) v1 ==== 147+0+0 (unknown 3603553944 0 0) 0x7fa6ac04a4b0 con 0x7fa6b80f78d0 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 10 librados: watch_flush enter 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 10 librados: watch_flush exit 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 20 client.24382.objecter shutdown clearing up homeless session... 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 10 client.24382.objecter successfully canceled tick 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 >> v1:172.21.15.107:6800/3360068681 conn(0x7fa6a0063c70 legacy=0x7fa6a0066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:20.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.106+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 >> v1:172.21.15.195:6789/0 conn(0x7fa6b80f78d0 legacy=0x7fa6b8107a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:20.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.109+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 shutdown_connections 2024-02-10T08:33:20.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.109+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 >> 172.21.15.195:0/711553339 conn(0x7fa6b80f3050 msgr2=0x7fa6b80f3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:20.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.109+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 shutdown_connections 2024-02-10T08:33:20.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.109+0000 7fa6bc87d700 1 -- 172.21.15.195:0/711553339 wait complete. 2024-02-10T08:33:20.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:20.110+0000 7fa6bc87d700 1 librados: shutdown 2024-02-10T08:33:20.111 INFO:teuthology.orchestra.run.smithi195.stderr:enabled application 'rbd' on pool 'iscsi' 2024-02-10T08:33:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:18.410175+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:18.410235+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:19.104171+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T08:33:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.106593+0000 mon.c (mon.2) 11 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.107084+0000 mon.a (mon.0) 574 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.108271+0000 mon.c (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.108838+0000 mon.a (mon.0) 575 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.701003+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.195:0/711553339' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:19.701689+0000 mon.a (mon.0) 576 : audit [INF] from='client.24382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:18.410175+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:18.410235+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:19.104171+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.106593+0000 mon.c (mon.2) 11 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.107084+0000 mon.a (mon.0) 574 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.108271+0000 mon.c (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.108838+0000 mon.a (mon.0) 575 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.701003+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.195:0/711553339' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:19.701689+0000 mon.a (mon.0) 576 : audit [INF] from='client.24382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:18.410175+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:20.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:18.410235+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:20.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:19.104171+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-10T08:33:20.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.106593+0000 mon.c (mon.2) 11 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.107084+0000 mon.a (mon.0) 574 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.108271+0000 mon.c (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.108838+0000 mon.a (mon.0) 575 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T08:33:20.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.701003+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.195:0/711553339' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:19.701689+0000 mon.a (mon.0) 576 : audit [INF] from='client.24382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-10T08:33:20.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply iscsi iscsi user password --placement '1;smithi195=iscsi.a' 2024-02-10T08:33:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:20.048912+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v192: 68 pgs: 3 creating+peering, 5 active+clean, 60 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:20.103381+0000 mon.a (mon.0) 577 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:20.103499+0000 mon.a (mon.0) 578 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:20.103620+0000 mon.a (mon.0) 579 : audit [INF] from='client.24382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:20.103708+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:20.591388+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:21.106416+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:20.048912+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v192: 68 pgs: 3 creating+peering, 5 active+clean, 60 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:20.103381+0000 mon.a (mon.0) 577 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:20.103499+0000 mon.a (mon.0) 578 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:20.103620+0000 mon.a (mon.0) 579 : audit [INF] from='client.24382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:20.103708+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:20.591388+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:21.106416+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:20.048912+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v192: 68 pgs: 3 creating+peering, 5 active+clean, 60 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:20.103381+0000 mon.a (mon.0) 577 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:20.103499+0000 mon.a (mon.0) 578 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:20.103620+0000 mon.a (mon.0) 579 : audit [INF] from='client.24382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:20.103708+0000 mon.a (mon.0) 580 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:20.591388+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:21.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:21.106416+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:21.111299+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:21.111640+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:21.112359+0000 mon.a (mon.0) 583 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:21.112730+0000 mon.a (mon.0) 584 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:22.108624+0000 mon.a (mon.0) 585 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:22.108724+0000 mon.a (mon.0) 586 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:22.108775+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:21.111299+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:21.111640+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:21.112359+0000 mon.a (mon.0) 583 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:21.112730+0000 mon.a (mon.0) 584 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:22.108624+0000 mon.a (mon.0) 585 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:22.108724+0000 mon.a (mon.0) 586 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:22.108775+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:21.111299+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.107:0/987143937' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:21.111640+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.107:0/1288525501' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:21.112359+0000 mon.a (mon.0) 583 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:21.112730+0000 mon.a (mon.0) 584 : audit [INF] from='client.34379 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:22.108624+0000 mon.a (mon.0) 585 : audit [INF] from='client.34349 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:22.108724+0000 mon.a (mon.0) 586 : audit [INF] from='client.34379 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T08:33:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:22.108775+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- 172.21.15.195:0/3436970721 >> v1:172.21.15.195:6789/0 conn(0x7f99280f67c0 legacy=0x7f99280f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- 172.21.15.195:0/3436970721 shutdown_connections 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- 172.21.15.195:0/3436970721 >> 172.21.15.195:0/3436970721 conn(0x7f99280f1f00 msgr2=0x7f99280f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- 172.21.15.195:0/3436970721 shutdown_connections 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- 172.21.15.195:0/3436970721 wait complete. 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 librados: starting msgr at 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 librados: starting objecter 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 Processor -- start 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 -- start start 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 librados: setting wanted keys 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.551+0000 7f992f2ba700 1 librados: calling monclient init 2024-02-10T08:33:22.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992f2ba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9928107310 con 0x7f99280fac90 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992f2ba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9928112b70 con 0x7f99280f67c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992f2ba700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9928113d50 con 0x7f99280fe8c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992d056700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f99280f67c0 0x7f9928106740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:34336/0 (socket says 172.21.15.195:34336) 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992d056700 1 -- 172.21.15.195:0/1560927267 learned_addr learned my addr 172.21.15.195:0/1560927267 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter ms_handle_connect 0x7f99280f67c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter ms_handle_connect 0x7f99280fe8c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter ms_handle_connect 0x7f99280fac90 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592377472 0 0) 0x7f9928112b70 con 0x7f99280f67c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9914002300 con 0x7f99280f67c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503088103 0 0) 0x7f9928113d50 con 0x7f99280fe8c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928112b70 con 0x7f99280fe8c0 2024-02-10T08:33:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2874927638 0 0) 0x7f9914002300 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9928113d50 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9918001350 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743028470 0 0) 0x7f9928112b70 con 0x7f99280fe8c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9914002300 con 0x7f99280fe8c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99200019a0 con 0x7f99280fe8c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1989879977 0 0) 0x7f9928113d50 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 >> v1:172.21.15.107:6790/0 conn(0x7f99280fe8c0 legacy=0x7f9928110440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 >> v1:172.21.15.107:6789/0 conn(0x7f99280fac90 legacy=0x7f992810cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928114f30 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.552+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9928113f40 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f992f2ba700 10 client.24400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9928114430 con 0x7f99280f67c0 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f992f2ba700 1 librados: init done 2024-02-10T08:33:22.556 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9918001570 con 0x7f99280f67c0 2024-02-10T08:33:22.557 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9918003990 con 0x7f99280f67c0 2024-02-10T08:33:22.557 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.553+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c004f20 con 0x7f99280f67c0 2024-02-10T08:33:22.557 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f9918003140 con 0x7f99280f67c0 2024-02-10T08:33:22.557 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(60..60 src has 1..60) v4 ==== 5502+0+0 (unknown 1361710204 0 0) 0x7f9918078750 con 0x7f99280f67c0 2024-02-10T08:33:22.557 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 10 client.24400.objecter ms_dispatch 0x7f9928001550 osd_map(60..60 src has 1..60) v4 2024-02-10T08:33:22.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 3 client.24400.objecter handle_osd_map got epochs [60,60] > 0 2024-02-10T08:33:22.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 3 client.24400.objecter handle_osd_map decoding full epoch 60 2024-02-10T08:33:22.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.554+0000 7f991e777700 20 client.24400.objecter dump_active .. 0 homeless 2024-02-10T08:33:22.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.555+0000 7f991e777700 10 client.24400.objecter ms_handle_connect 0x7f9914063b20 2024-02-10T08:33:22.559 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.558+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9918048ae0 con 0x7f99280f67c0 2024-02-10T08:33:22.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.730+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi195=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f990c000c40 con 0x7f9914063b20 2024-02-10T08:33:22.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.737+0000 7f991e777700 1 -- 172.21.15.195:0/1560927267 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3282307126) 0x7f990c000c40 con 0x7f9914063b20 2024-02-10T08:33:22.738 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled iscsi.iscsi update... 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 10 librados: watch_flush enter 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 10 librados: watch_flush exit 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 20 client.24400.objecter shutdown clearing up homeless session... 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 10 client.24400.objecter successfully canceled tick 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 >> v1:172.21.15.107:6800/3360068681 conn(0x7f9914063b20 legacy=0x7f9914065fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 >> v1:172.21.15.195:6789/0 conn(0x7f99280f67c0 legacy=0x7f9928106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 shutdown_connections 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 >> 172.21.15.195:0/1560927267 conn(0x7f99280f1f00 msgr2=0x7f99280fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 shutdown_connections 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 -- 172.21.15.195:0/1560927267 wait complete. 2024-02-10T08:33:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:22.739+0000 7f992f2ba700 1 librados: shutdown 2024-02-10T08:33:23.393 DEBUG:teuthology.orchestra.run.smithi195:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@iscsi.iscsi.a.service 2024-02-10T08:33:23.395 INFO:tasks.cephadm:Adding prometheus.a on smithi195 2024-02-10T08:33:23.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi195=a' 2024-02-10T08:33:23.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:33:24.009 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:22.049384+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v195: 100 pgs: 12 creating+peering, 53 active+clean, 35 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-02-10T08:33:24.009 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:22.731712+0000 mgr.y (mgr.14146) 204 : audit [DBG] from='client.24400 v1:172.21.15.195:0/1560927267' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi195=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:22.733443+0000 mgr.y (mgr.14146) 205 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi195=iscsi.a;count:1 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:22.737451+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:23.109957+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:23.111459+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:23.113492+0000 mon.a (mon.0) 590 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.010 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:23.114333+0000 mon.a (mon.0) 591 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:22.049384+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v195: 100 pgs: 12 creating+peering, 53 active+clean, 35 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-02-10T08:33:24.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:22.731712+0000 mgr.y (mgr.14146) 204 : audit [DBG] from='client.24400 v1:172.21.15.195:0/1560927267' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi195=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:24.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:22.733443+0000 mgr.y (mgr.14146) 205 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi195=iscsi.a;count:1 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:22.737451+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:23.109957+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:23.111459+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:23.113492+0000 mon.a (mon.0) 590 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:23.114333+0000 mon.a (mon.0) 591 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:22.049384+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v195: 100 pgs: 12 creating+peering, 53 active+clean, 35 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:22.731712+0000 mgr.y (mgr.14146) 204 : audit [DBG] from='client.24400 v1:172.21.15.195:0/1560927267' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi195=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:22.733443+0000 mgr.y (mgr.14146) 205 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi195=iscsi.a;count:1 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:22.737451+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:23.109957+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-10T08:33:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:23.111459+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:23.113492+0000 mon.a (mon.0) 590 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:24.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:23.114333+0000 mon.a (mon.0) 591 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T08:33:25.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.148+0000 7f15553b5700 1 -- 172.21.15.195:0/747870289 >> v1:172.21.15.195:6789/0 conn(0x7f1550101f90 legacy=0x7f1550104380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:25.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.148+0000 7f15553b5700 1 -- 172.21.15.195:0/747870289 shutdown_connections 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.148+0000 7f15553b5700 1 -- 172.21.15.195:0/747870289 >> 172.21.15.195:0/747870289 conn(0x7f15500f3050 msgr2=0x7f15500f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.148+0000 7f15553b5700 1 -- 172.21.15.195:0/747870289 shutdown_connections 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.148+0000 7f15553b5700 1 -- 172.21.15.195:0/747870289 wait complete. 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 librados: starting msgr at 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 librados: starting objecter 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 Processor -- start 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 -- start start 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 librados: setting wanted keys 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 librados: calling monclient init 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.149+0000 7f15553b5700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15500f7ef0 con 0x7f1550101f90 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f15553b5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15500f80e0 con 0x7f155006cc10 2024-02-10T08:33:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f15553b5700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15500f82d0 con 0x7f155006b820 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f154e7fc700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f155006cc10 0x7f15500f7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:34364/0 (socket says 172.21.15.195:34364) 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f154e7fc700 1 -- 172.21.15.195:0/2986506292 learned_addr learned my addr 172.21.15.195:0/2986506292 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter ms_handle_connect 0x7f155006cc10 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter ms_handle_connect 0x7f155006b820 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter ms_handle_connect 0x7f1550101f90 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168765414 0 0) 0x7f15500f80e0 con 0x7f155006cc10 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15380033f0 con 0x7f155006cc10 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530918794 0 0) 0x7f15500f82d0 con 0x7f155006b820 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15500f80e0 con 0x7f155006b820 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093091389 0 0) 0x7f15500f7ef0 con 0x7f1550101f90 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15500f82d0 con 0x7f1550101f90 2024-02-10T08:33:25.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287430619 0 0) 0x7f15380033f0 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15500f7ef0 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f15440029b0 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406951498 0 0) 0x7f15500f82d0 con 0x7f1550101f90 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15380033f0 con 0x7f1550101f90 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1548002240 con 0x7f1550101f90 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369059477 0 0) 0x7f15500f80e0 con 0x7f155006b820 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15500f82d0 con 0x7f155006b820 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1540001840 con 0x7f155006b820 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 686070805 0 0) 0x7f15500f7ef0 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 >> v1:172.21.15.107:6790/0 conn(0x7f155006b820 legacy=0x7f15500fad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 >> v1:172.21.15.107:6789/0 conn(0x7f1550101f90 legacy=0x7f1550118130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.150+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155011c880 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15501198b0 con 0x7f155006cc10 2024-02-10T08:33:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f15553b5700 10 client.24406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1550119da0 con 0x7f155006cc10 2024-02-10T08:33:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f15553b5700 1 librados: init done 2024-02-10T08:33:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1544002c70 con 0x7f155006cc10 2024-02-10T08:33:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.151+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1544005ab0 con 0x7f155006cc10 2024-02-10T08:33:25.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c004f20 con 0x7f155006cc10 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 1125578376 0 0) 0x7f1544018900 con 0x7f155006cc10 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(63..63 src has 1..63) v4 ==== 5884+0+0 (unknown 3485451785 0 0) 0x7f1544078390 con 0x7f155006cc10 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f1537fff700 10 client.24406.objecter ms_dispatch 0x7f1550001550 osd_map(63..63 src has 1..63) v4 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f1537fff700 3 client.24406.objecter handle_osd_map got epochs [63,63] > 0 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.152+0000 7f1537fff700 3 client.24406.objecter handle_osd_map decoding full epoch 63 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.153+0000 7f1537fff700 20 client.24406.objecter dump_active .. 0 homeless 2024-02-10T08:33:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.153+0000 7f1537fff700 10 client.24406.objecter ms_handle_connect 0x7f1538063d50 2024-02-10T08:33:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.157+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1544048540 con 0x7f155006cc10 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:24.049987+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v198: 132 pgs: 16 creating+peering, 75 active+clean, 41 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 695 B/s wr, 2 op/s 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:24.109627+0000 mon.a (mon.0) 592 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:24.109774+0000 mon.a (mon.0) 593 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:24.109839+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:24.111818+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:24.114342+0000 mon.a (mon.0) 595 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:24.115320+0000 mon.a (mon.0) 596 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:24.127536+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:33:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:24.130483+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:33:25.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.376+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f153c000b90 con 0x7f1538063d50 2024-02-10T08:33:25.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.382+0000 7f1537fff700 1 -- 172.21.15.195:0/2986506292 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7f153c000b90 con 0x7f1538063d50 2024-02-10T08:33:25.383 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled prometheus update... 2024-02-10T08:33:25.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.384+0000 7f15553b5700 10 librados: watch_flush enter 2024-02-10T08:33:25.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.384+0000 7f15553b5700 10 librados: watch_flush exit 2024-02-10T08:33:25.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.384+0000 7f15553b5700 20 client.24406.objecter shutdown clearing up homeless session... 2024-02-10T08:33:25.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.384+0000 7f15553b5700 10 client.24406.objecter successfully canceled tick 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 >> v1:172.21.15.107:6800/3360068681 conn(0x7f1538063d50 legacy=0x7f1538066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 >> v1:172.21.15.195:6789/0 conn(0x7f155006cc10 legacy=0x7f15500f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 shutdown_connections 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 >> 172.21.15.195:0/2986506292 conn(0x7f15500f3050 msgr2=0x7f15500f4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 shutdown_connections 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 -- 172.21.15.195:0/2986506292 wait complete. 2024-02-10T08:33:25.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:25.385+0000 7f15553b5700 1 librados: shutdown 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:24.049987+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v198: 132 pgs: 16 creating+peering, 75 active+clean, 41 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 695 B/s wr, 2 op/s 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:24.109627+0000 mon.a (mon.0) 592 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:24.109774+0000 mon.a (mon.0) 593 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:24.109839+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:24.111818+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:24.114342+0000 mon.a (mon.0) 595 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:24.115320+0000 mon.a (mon.0) 596 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:24.127536+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:24.130483+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:24.049987+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v198: 132 pgs: 16 creating+peering, 75 active+clean, 41 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 695 B/s wr, 2 op/s 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:24.109627+0000 mon.a (mon.0) 592 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:24.109774+0000 mon.a (mon.0) 593 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:24.109839+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:24.111818+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.107:0/4187249279' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:24.114342+0000 mon.a (mon.0) 595 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:24.115320+0000 mon.a (mon.0) 596 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:24.127536+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:33:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:24.130483+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:33:26.017 DEBUG:teuthology.orchestra.run.smithi195:prometheus.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T08:33:26.019 INFO:tasks.cephadm:Adding node-exporter.a on smithi107 2024-02-10T08:33:26.019 INFO:tasks.cephadm:Adding node-exporter.b on smithi195 2024-02-10T08:33:26.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi107=a;smithi195=b' 2024-02-10T08:33:26.052 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:33:26.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.110621+0000 mon.a (mon.0) 597 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.110707+0000 mon.a (mon.0) 598 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:25.110778+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.266766+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.273006+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.280550+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:25.284173+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.292443+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.296435+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.304243+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:25.304918+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi195 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.378170+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24406 v1:172.21.15.195:0/2986506292' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:25.379580+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi195=a;count:1 2024-02-10T08:33:26.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:25.382871+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.110621+0000 mon.a (mon.0) 597 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.110707+0000 mon.a (mon.0) 598 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:25.110778+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.266766+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.273006+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.280550+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:25.284173+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.292443+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.296435+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.304243+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:25.304918+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi195 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.378170+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24406 v1:172.21.15.195:0/2986506292' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:25.379580+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi195=a;count:1 2024-02-10T08:33:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:25.382871+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.110621+0000 mon.a (mon.0) 597 : audit [INF] from='client.24391 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.110707+0000 mon.a (mon.0) 598 : audit [INF] from='client.? v1:172.21.15.107:0/3446877500' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:25.110778+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.266766+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.273006+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.280550+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:25.284173+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.292443+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.296435+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.304243+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:25.304918+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi195 2024-02-10T08:33:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.378170+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24406 v1:172.21.15.195:0/2986506292' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:26.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:25.379580+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi195=a;count:1 2024-02-10T08:33:26.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:25.382871+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:27.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:25.538580+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:27.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:25.538656+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:27.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:26.050694+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v201: 132 pgs: 16 creating+peering, 99 active+clean, 17 unknown; 2.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 3.0 KiB/s wr, 65 op/s 2024-02-10T08:33:27.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:26.392771+0000 mon.a (mon.0) 607 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-10T08:33:27.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:26.392822+0000 mon.a (mon.0) 608 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T08:33:27.527 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:27 smithi195 systemd[1]: Starting Ceph iscsi.iscsi.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:33:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:25.538580+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:25.538656+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:26.050694+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v201: 132 pgs: 16 creating+peering, 99 active+clean, 17 unknown; 2.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 3.0 KiB/s wr, 65 op/s 2024-02-10T08:33:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:26.392771+0000 mon.a (mon.0) 607 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:26.392822+0000 mon.a (mon.0) 608 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:25.538580+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:25.538656+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:26.050694+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v201: 132 pgs: 16 creating+peering, 99 active+clean, 17 unknown; 2.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 3.0 KiB/s wr, 65 op/s 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:26.392771+0000 mon.a (mon.0) 607 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-10T08:33:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:26.392822+0000 mon.a (mon.0) 608 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-10T08:33:28.522 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 -- 172.21.15.195:0/1065291165 >> v1:172.21.15.195:6789/0 conn(0x7fa8180fae00 legacy=0x7fa8180fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 -- 172.21.15.195:0/1065291165 shutdown_connections 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 -- 172.21.15.195:0/1065291165 >> 172.21.15.195:0/1065291165 conn(0x7fa8180f16b0 msgr2=0x7fa8180f3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 -- 172.21.15.195:0/1065291165 shutdown_connections 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 -- 172.21.15.195:0/1065291165 wait complete. 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 librados: starting msgr at 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 librados: starting objecter 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.520+0000 7fa81d0ca700 1 Processor -- start 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 -- start start 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 librados: setting wanted keys 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 librados: calling monclient init 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa818114fb0 con 0x7fa8180f6d80 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa818116190 con 0x7fa8180fae00 2024-02-10T08:33:28.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81d0ca700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa818117390 con 0x7fa8180fe910 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81659c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa8180fae00 0x7fa81810ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:34376/0 (socket says 172.21.15.195:34376) 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa81659c700 1 -- 172.21.15.195:0/2541161844 learned_addr learned my addr 172.21.15.195:0/2541161844 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa816d9d700 1 --1- 172.21.15.195:0/2541161844 >> v1:172.21.15.107:6789/0 conn(0x7fa8180f6d80 0x7fa81810a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:39866/0 (socket says 172.21.15.195:39866) 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter ms_handle_connect 0x7fa8180fe910 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter ms_handle_connect 0x7fa8180fae00 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter ms_handle_connect 0x7fa8180f6d80 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500846171 0 0) 0x7fa818116190 con 0x7fa8180fae00 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800002220 con 0x7fa8180fae00 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538028049 0 0) 0x7fa818117390 con 0x7fa8180fe910 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818116190 con 0x7fa8180fe910 2024-02-10T08:33:28.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634658821 0 0) 0x7fa818114fb0 con 0x7fa8180f6d80 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.521+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818117390 con 0x7fa8180f6d80 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2585326016 0 0) 0x7fa800002220 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa818114fb0 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa80c001070 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336369451 0 0) 0x7fa818116190 con 0x7fa8180fe910 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa800002220 con 0x7fa8180fe910 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa808001b80 con 0x7fa8180fe910 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1441052572 0 0) 0x7fa818114fb0 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 >> v1:172.21.15.107:6790/0 conn(0x7fa8180fe910 legacy=0x7fa818113670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 >> v1:172.21.15.107:6789/0 conn(0x7fa8180f6d80 legacy=0x7fa81810a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa818118590 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa818116380 con 0x7fa8180fae00 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa81d0ca700 10 client.24433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:28.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa818116870 con 0x7fa8180fae00 2024-02-10T08:33:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa81d0ca700 1 librados: init done 2024-02-10T08:33:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa80c0011d0 con 0x7fa8180fae00 2024-02-10T08:33:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.522+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa80c003550 con 0x7fa8180fae00 2024-02-10T08:33:28.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.523+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa804004f20 con 0x7fa8180fae00 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.523+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 15) v1 ==== 77081+0+0 (unknown 2404933324 0 0) 0x7fa80c0036d0 con 0x7fa8180fae00 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.524+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(64..64 src has 1..64) v4 ==== 5884+0+0 (unknown 1482841443 0 0) 0x7fa80c078670 con 0x7fa8180fae00 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.524+0000 7fa7fffff700 10 client.24433.objecter ms_dispatch 0x7fa818001550 osd_map(64..64 src has 1..64) v4 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.524+0000 7fa7fffff700 3 client.24433.objecter handle_osd_map got epochs [64,64] > 0 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.524+0000 7fa7fffff700 3 client.24433.objecter handle_osd_map decoding full epoch 64 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.524+0000 7fa7fffff700 20 client.24433.objecter dump_active .. 0 homeless 2024-02-10T08:33:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.525+0000 7fa7fffff700 10 client.24433.objecter ms_handle_connect 0x7fa800063e10 2024-02-10T08:33:28.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.529+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa80c0486c0 con 0x7fa8180fae00 2024-02-10T08:33:28.707 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.706+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi195=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fa804000b90 con 0x7fa800063e10 2024-02-10T08:33:28.713 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.713+0000 7fa7fffff700 1 -- 172.21.15.195:0/2541161844 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7fa804000b90 con 0x7fa800063e10 2024-02-10T08:33:28.714 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled node-exporter update... 2024-02-10T08:33:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 10 librados: watch_flush enter 2024-02-10T08:33:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 10 librados: watch_flush exit 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 20 client.24433.objecter shutdown clearing up homeless session... 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 10 client.24433.objecter successfully canceled tick 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 >> v1:172.21.15.107:6800/3360068681 conn(0x7fa800063e10 legacy=0x7fa8000662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 >> v1:172.21.15.195:6789/0 conn(0x7fa8180fae00 legacy=0x7fa81810ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 shutdown_connections 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 >> 172.21.15.195:0/2541161844 conn(0x7fa8180f16b0 msgr2=0x7fa8180f5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 shutdown_connections 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 -- 172.21.15.195:0/2541161844 wait complete. 2024-02-10T08:33:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:28.716+0000 7fa81d0ca700 1 librados: shutdown 2024-02-10T08:33:29.407 DEBUG:teuthology.orchestra.run.smithi107:node-exporter.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T08:33:29.411 DEBUG:teuthology.orchestra.run.smithi195:node-exporter.b> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T08:33:29.413 INFO:tasks.cephadm:Adding alertmanager.a on smithi107 2024-02-10T08:33:29.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi107=a' 2024-02-10T08:33:29.444 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:33:29.448 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:33:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:28.051410+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v203: 132 pgs: 7 creating+peering, 125 active+clean; 4.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.1 KiB/s wr, 184 op/s 2024-02-10T08:33:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:28.707980+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24433 v1:172.21.15.195:0/2541161844' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:28.709397+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi107=a;smithi195=b;count:2 2024-02-10T08:33:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:28.713627+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:30.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:29 smithi195 bash[90997]: log file path now is '/var/log/tcmu-runner.log' 2024-02-10T08:33:30.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:29 smithi195 bash[91139]: c62b21cc298646f255bf3e6cf3bf7fa53acf2c2681753a5b9a1153b7be7494eb 2024-02-10T08:33:30.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:29 smithi195 systemd[1]: Started Ceph iscsi.iscsi.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:33:30.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:28.051410+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v203: 132 pgs: 7 creating+peering, 125 active+clean; 4.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.1 KiB/s wr, 184 op/s 2024-02-10T08:33:30.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:28.707980+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24433 v1:172.21.15.195:0/2541161844' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:30.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:28.709397+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi107=a;smithi195=b;count:2 2024-02-10T08:33:30.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:28.713627+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:30.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:28.051410+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v203: 132 pgs: 7 creating+peering, 125 active+clean; 4.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.1 KiB/s wr, 184 op/s 2024-02-10T08:33:30.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:28.707980+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24433 v1:172.21.15.195:0/2541161844' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:30.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:28.709397+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi107=a;smithi195=b;count:2 2024-02-10T08:33:30.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:28.713627+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:30.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Started the configuration object watcher 2024-02-10T08:33:30.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Checking for config object changes every 1s 2024-02-10T08:33:30.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Processing osd blocklist entries for this node 2024-02-10T08:33:30.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Reading the configuration object to update local LIO configuration 2024-02-10T08:33:30.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Configuration does not have an entry for this host(smithi195.front.sepia.ceph.com) - nothing to define to LIO 2024-02-10T08:33:30.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-10T08:33:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:29.968218+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:29.969572+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:30.051984+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-02-10T08:33:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:30.396634+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/3999254052' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T08:33:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:30.600026+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:29.968218+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:29.969572+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:30.051984+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-02-10T08:33:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:30.396634+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/3999254052' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:30.600026+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:29.968218+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:29.969572+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:30.051984+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:30.396634+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.195:0/3999254052' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T08:33:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:30.600026+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:31.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 -- 172.21.15.195:0/1907965666 >> v1:172.21.15.195:6789/0 conn(0x7fb6bc0f78d0 legacy=0x7fb6bc0f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 -- 172.21.15.195:0/1907965666 shutdown_connections 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 -- 172.21.15.195:0/1907965666 >> 172.21.15.195:0/1907965666 conn(0x7fb6bc0f3050 msgr2=0x7fb6bc0f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 -- 172.21.15.195:0/1907965666 shutdown_connections 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 -- 172.21.15.195:0/1907965666 wait complete. 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 librados: starting msgr at 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 librados: starting objecter 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.683+0000 7fb6c3de1700 1 Processor -- start 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 -- start start 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 librados: setting wanted keys 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 librados: calling monclient init 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6bc114e20 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6bc116000 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c3de1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6bc117200 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c137c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb6bc0fbda0 0x7fb6bc0fe720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:42768/0 (socket says 172.21.15.195:42768) 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6c137c700 1 -- 172.21.15.195:0/561264652 learned_addr learned my addr 172.21.15.195:0/561264652 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter ms_handle_connect 0x7fb6bc0fbda0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter ms_handle_connect 0x7fb6bc0ff9d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter ms_handle_connect 0x7fb6bc0f78d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041164395 0 0) 0x7fb6bc116000 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a80033f0 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929838298 0 0) 0x7fb6bc114e20 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc116000 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792628318 0 0) 0x7fb6bc117200 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc114e20 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768001099 0 0) 0x7fb6a80033f0 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6bc117200 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6ac001a10 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127279422 0 0) 0x7fb6bc116000 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a80033f0 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6b4001c40 con 0x7fb6bc0ff9d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880613826 0 0) 0x7fb6bc114e20 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6bc116000 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.684+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6b8002060 con 0x7fb6bc0f78d0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3430627892 0 0) 0x7fb6bc117200 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 >> v1:172.21.15.107:6790/0 conn(0x7fb6bc0f78d0 legacy=0x7fb6bc0fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 >> v1:172.21.15.107:6789/0 conn(0x7fb6bc0ff9d0 legacy=0x7fb6bc1134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc118400 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc115010 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6c3de1700 10 client.24448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc115480 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6c3de1700 1 librados: init done 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb6ac002d00 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6ac005bc0 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.685+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a0004f20 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 15) v1 ==== 77081+0+0 (unknown 2404933324 0 0) 0x7fb6ac002960 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(64..64 src has 1..64) v4 ==== 5884+0+0 (unknown 1482841443 0 0) 0x7fb6ac0786b0 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 10 client.24448.objecter ms_dispatch 0x7fb6bc001550 osd_map(64..64 src has 1..64) v4 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 3 client.24448.objecter handle_osd_map got epochs [64,64] > 0 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 3 client.24448.objecter handle_osd_map decoding full epoch 64 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.686+0000 7fb6b2ffd700 20 client.24448.objecter dump_active .. 0 homeless 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.687+0000 7fb6b2ffd700 10 client.24448.objecter ms_handle_connect 0x7fb6a8063d60 2024-02-10T08:33:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.691+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb6ac048660 con 0x7fb6bc0fbda0 2024-02-10T08:33:31.866 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.866+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6a0000b90 con 0x7fb6a8063d60 2024-02-10T08:33:31.873 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.873+0000 7fb6b2ffd700 1 -- 172.21.15.195:0/561264652 <== mgr.14146 v1:172.21.15.107:6800/3360068681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7fb6a0000b90 con 0x7fb6a8063d60 2024-02-10T08:33:31.874 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled alertmanager update... 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 10 librados: watch_flush enter 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 10 librados: watch_flush exit 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 20 client.24448.objecter shutdown clearing up homeless session... 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 10 client.24448.objecter successfully canceled tick 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 >> v1:172.21.15.107:6800/3360068681 conn(0x7fb6a8063d60 legacy=0x7fb6a8066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 >> v1:172.21.15.195:6789/0 conn(0x7fb6bc0fbda0 legacy=0x7fb6bc0fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 shutdown_connections 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 >> 172.21.15.195:0/561264652 conn(0x7fb6bc0f3050 msgr2=0x7fb6bc06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 shutdown_connections 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 -- 172.21.15.195:0/561264652 wait complete. 2024-02-10T08:33:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:31.875+0000 7fb6c3de1700 1 librados: shutdown 2024-02-10T08:33:32.652 DEBUG:teuthology.orchestra.run.smithi107:alertmanager.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T08:33:32.655 INFO:tasks.cephadm:Adding grafana.a on smithi195 2024-02-10T08:33:32.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi195=a' 2024-02-10T08:33:32.688 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:-- Logs begin at Sat 2024-02-10 08:16:20 UTC. -- 2024-02-10T08:33:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:31.868090+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24448 v1:172.21.15.195:0/561264652' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:31.869395+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi107=a;count:1 2024-02-10T08:33:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:31.873678+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:32.052549+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-02-10T08:33:33.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:31.868090+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24448 v1:172.21.15.195:0/561264652' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:33.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:31.869395+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi107=a;count:1 2024-02-10T08:33:33.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:31.873678+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:33.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:32.052549+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-02-10T08:33:33.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:31.868090+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24448 v1:172.21.15.195:0/561264652' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:33:33.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:31.869395+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi107=a;count:1 2024-02-10T08:33:33.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:31.873678+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:33.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:32.052549+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-02-10T08:33:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.927+0000 7f0e2359e700 1 -- 172.21.15.195:0/2520713042 >> v1:172.21.15.195:6789/0 conn(0x7f0e240ff800 legacy=0x7f0e24101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.927+0000 7f0e2359e700 1 -- 172.21.15.195:0/2520713042 shutdown_connections 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.927+0000 7f0e2359e700 1 -- 172.21.15.195:0/2520713042 >> 172.21.15.195:0/2520713042 conn(0x7f0e240f1e10 msgr2=0x7f0e240f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.927+0000 7f0e2359e700 1 -- 172.21.15.195:0/2520713042 shutdown_connections 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.927+0000 7f0e2359e700 1 -- 172.21.15.195:0/2520713042 wait complete. 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 librados: starting msgr at 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 librados: starting objecter 2024-02-10T08:33:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 Processor -- start 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 -- start start 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 librados: setting wanted keys 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 librados: calling monclient init 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0e240fad70 con 0x7f0e24061180 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0e241160d0 con 0x7f0e240fba90 2024-02-10T08:33:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2359e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0e24117270 con 0x7f0e240ff800 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e21d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0e240fba90 0x7f0e24110270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:42808/0 (socket says 172.21.15.195:42808) 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 learned_addr learned my addr 172.21.15.195:0/3614553889 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.928+0000 7f0e2259c700 1 --1- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6789/0 conn(0x7f0e24061180 0x7f0e240fa250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:60282/0 (socket says 172.21.15.195:60282) 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter ms_handle_connect 0x7f0e240fba90 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter ms_handle_connect 0x7f0e24061180 2024-02-10T08:33:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter ms_handle_connect 0x7f0e240ff800 2024-02-10T08:33:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637335591 0 0) 0x7f0e240fad70 con 0x7f0e24061180 2024-02-10T08:33:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c0033f0 con 0x7f0e24061180 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433247022 0 0) 0x7f0e241160d0 con 0x7f0e240fba90 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e240fad70 con 0x7f0e240fba90 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470142681 0 0) 0x7f0e24117270 con 0x7f0e240ff800 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e241160d0 con 0x7f0e240ff800 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112221759 0 0) 0x7f0e0c0033f0 con 0x7f0e24061180 2024-02-10T08:33:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e24117270 con 0x7f0e24061180 2024-02-10T08:33:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0e14001840 con 0x7f0e24061180 2024-02-10T08:33:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076046444 0 0) 0x7f0e241160d0 con 0x7f0e240ff800 2024-02-10T08:33:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e0c0033f0 con 0x7f0e240ff800 2024-02-10T08:33:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0e1c003280 con 0x7f0e240ff800 2024-02-10T08:33:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67149587 0 0) 0x7f0e240fad70 con 0x7f0e240fba90 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e241160d0 con 0x7f0e240fba90 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0e18001880 con 0x7f0e240fba90 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2578802800 0 0) 0x7f0e24117270 con 0x7f0e24061180 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6790/0 conn(0x7f0e240ff800 legacy=0x7f0e241139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.195:6789/0 conn(0x7f0e240fba90 legacy=0x7f0e24110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24118410 con 0x7f0e24061180 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e24115120 con 0x7f0e24061180 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e2359e700 10 client.14559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e24115610 con 0x7f0e24061180 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e2359e700 1 librados: init done 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0e14001f00 con 0x7f0e24061180 2024-02-10T08:33:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.929+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0e14005990 con 0x7f0e24061180 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.930+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04004f20 con 0x7f0e24061180 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.930+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 16) v1 ==== 77095+0+0 (unknown 1188558149 0 0) 0x7f0e140188a0 con 0x7f0e24061180 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/3360068681 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(64..64 src has 1..64) v4 ==== 5884+0+0 (unknown 1482841443 0 0) 0x7f0e140783b0 con 0x7f0e24061180 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e137fe700 10 client.14559.objecter ms_dispatch 0x7f0e24001550 osd_map(64..64 src has 1..64) v4 2024-02-10T08:33:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e137fe700 3 client.14559.objecter handle_osd_map got epochs [64,64] > 0 2024-02-10T08:33:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e137fe700 3 client.14559.objecter handle_osd_map decoding full epoch 64 2024-02-10T08:33:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.931+0000 7f0e137fe700 20 client.14559.objecter dump_active .. 0 homeless 2024-02-10T08:33:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:34.935+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0e140483a0 con 0x7f0e24061180 2024-02-10T08:33:34.946 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:33.685978+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.946 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:33.686811+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:34.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:33.687641+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:34.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:33.692453+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:33.697590+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T08:33:34.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:33.685978+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:33.686811+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:34.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:33.687641+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:34.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:33.692453+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:33.697590+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T08:33:34.949 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setuser ceph since I am not root 2024-02-10T08:33:34.949 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ignoring --setgroup ceph since I am not root 2024-02-10T08:33:34.949 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:34.864+0000 7f1439438680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:33:34.949 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:34.945+0000 7f1439438680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:33:34.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:33.685978+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:33.686811+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:34.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:33.687641+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:34.950 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:33.692453+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' 2024-02-10T08:33:34.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:33.697590+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T08:33:34.951 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ignoring --setuser ceph since I am not root 2024-02-10T08:33:34.951 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ignoring --setgroup ceph since I am not root 2024-02-10T08:33:34.951 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:34.738+0000 7fddf4c50700 1 -- 172.21.15.195:0/2388845399 <== mon.1 v1:172.21.15.195:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 1793837640 0 0) 0x56540663c9c0 con 0x565406613400 2024-02-10T08:33:34.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:34.861+0000 7fde04e2e680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T08:33:35.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:35.116+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6800/3360068681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e04000b90 con 0x7f0e0c063d00 2024-02-10T08:33:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:35.132+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/3360068681 2024-02-10T08:33:35.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:34.947+0000 7fde04e2e680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T08:33:35.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:35.533+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/3360068681 2024-02-10T08:33:35.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:35.302+0000 7f1439438680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:33:35.700 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:35.312+0000 7fde04e2e680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T08:33:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:34.703761+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T08:33:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:34.703825+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-10T08:33:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:34.703761+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T08:33:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:34.703825+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-10T08:33:36.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:34.703761+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.107:0/4116544856' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T08:33:36.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:34.703825+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-10T08:33:36.045 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:36.038+0000 7f1439438680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:33:36.335 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:36.124+0000 7f1439438680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:33:36.336 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:36.334+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/3360068681 2024-02-10T08:33:36.346 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:36.049+0000 7fde04e2e680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T08:33:36.346 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:36.134+0000 7fde04e2e680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T08:33:36.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:36.334+0000 7f1439438680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:33:36.787 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:36.344+0000 7fde04e2e680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T08:33:37.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:37.270+0000 7fde04e2e680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:33:37.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:37.395+0000 7fde04e2e680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:33:37.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:37.473+0000 7fde04e2e680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:33:37.558 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:37.264+0000 7f1439438680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T08:33:37.559 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:37.390+0000 7f1439438680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:33:37.559 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:37.468+0000 7f1439438680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T08:33:37.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:37.558+0000 7f1439438680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:33:37.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:37.752+0000 7f1439438680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:33:37.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:37.935+0000 7f0e21d9b700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.107:6800/3360068681 2024-02-10T08:33:38.035 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:37.561+0000 7fde04e2e680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T08:33:38.035 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:37.757+0000 7fde04e2e680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T08:33:38.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:38.034+0000 7fde04e2e680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:33:38.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:38.133+0000 7fde04e2e680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:33:38.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:38.027+0000 7f1439438680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T08:33:38.430 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:38.125+0000 7f1439438680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T08:33:39.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:38.841+0000 7f1439438680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:33:39.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:38.923+0000 7f1439438680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:33:39.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.015+0000 7f1439438680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:33:39.191 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:38.848+0000 7fde04e2e680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T08:33:39.191 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:38.928+0000 7fde04e2e680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T08:33:39.191 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.018+0000 7fde04e2e680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T08:33:39.444 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.188+0000 7f1439438680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:33:39.452 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.189+0000 7fde04e2e680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T08:33:39.708 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.443+0000 7f1439438680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:33:39.708 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.576+0000 7f1439438680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:33:39.711 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.450+0000 7fde04e2e680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T08:33:39.711 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.579+0000 7fde04e2e680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T08:33:39.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:39.930+0000 7f0e23fff700 10 client.14559.objecter tick 2024-02-10T08:33:39.961 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.708+0000 7f1439438680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:33:39.961 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.877+0000 7f1439438680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:33:39.975 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.710+0000 7fde04e2e680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T08:33:39.975 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.877+0000 7fde04e2e680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T08:33:39.976 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: debug 2024-02-10T08:33:39.959+0000 7fde04e2e680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:33:39.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:39.979+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mgrmap(e 17) v1 ==== 76859+0+0 (unknown 1343758278 0 0) 0x7f0e140760b0 con 0x7f0e24061180 2024-02-10T08:33:39.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:39.980+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/3360068681 conn(0x7f0e0c063d00 legacy=0x7f0e0c0661c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:33:40.220 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:33:39.960+0000 7f1439438680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T08:33:40.220 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:33:40] ENGINE Bus STARTING 2024-02-10T08:33:40.220 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [80B blob data] 2024-02-10T08:33:40.220 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [80B blob data] 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: [10/Feb/2024:08:33:39] ENGINE Bus STARTING 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: CherryPy Checker: 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: The Application mounted at '' has an empty config. 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: [10/Feb/2024:08:33:40] ENGINE Serving on http://:::9283 2024-02-10T08:33:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: [10/Feb/2024:08:33:40] ENGINE Bus STARTED 2024-02-10T08:33:40.534 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: CherryPy Checker: 2024-02-10T08:33:40.535 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: The Application mounted at '' has an empty config. 2024-02-10T08:33:40.535 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: 2024-02-10T08:33:40.535 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:33:40] ENGINE Serving on http://:::9283 2024-02-10T08:33:40.535 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:33:40] ENGINE Bus STARTED 2024-02-10T08:33:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.974492+0000 mon.a (mon.0) 621 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.974582+0000 mon.a (mon.0) 622 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.974777+0000 mon.a (mon.0) 623 : cluster [INF] Active manager daemon y restarted 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.975023+0000 mon.a (mon.0) 624 : cluster [INF] Activating manager daemon y 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.975615+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.976456+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.982972+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:39.983659+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00871775s), standbys: x 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.986586+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.986784+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.986951+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:33:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.987352+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.987669+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.987673+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.987840+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.988051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.988298+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.988572+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.988850+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.989065+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.989316+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.989569+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.989801+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.990025+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:33:40.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.990169+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:39.990615+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:40.004550+0000 mon.a (mon.0) 627 : cluster [INF] Manager daemon y is now available 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.098818+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.099927+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.135560+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.146533+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.146873+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.169319+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.169643+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.974492+0000 mon.a (mon.0) 621 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T08:33:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.974582+0000 mon.a (mon.0) 622 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.974777+0000 mon.a (mon.0) 623 : cluster [INF] Active manager daemon y restarted 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.975023+0000 mon.a (mon.0) 624 : cluster [INF] Activating manager daemon y 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.975615+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.976456+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.982972+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:39.983659+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00871775s), standbys: x 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.986586+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.986784+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.986951+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.987352+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.987669+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.987673+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.987840+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:33:40.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.988051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.974492+0000 mon.a (mon.0) 621 : cluster [DBG] Standby manager daemon x restarted 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.974582+0000 mon.a (mon.0) 622 : cluster [DBG] Standby manager daemon x started 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.974777+0000 mon.a (mon.0) 623 : cluster [INF] Active manager daemon y restarted 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.975023+0000 mon.a (mon.0) 624 : cluster [INF] Activating manager daemon y 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.975615+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.976456+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.982972+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:39.983659+0000 mon.a (mon.0) 626 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00871775s), standbys: x 2024-02-10T08:33:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.986586+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.986784+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.986951+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.987352+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.987669+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.987673+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.195:0/1360211175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.987840+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.988051+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.988298+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:33:40.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.988572+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.988850+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.989065+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.989316+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.989569+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.989801+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.990025+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.990169+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:39.990615+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:40.004550+0000 mon.a (mon.0) 627 : cluster [INF] Manager daemon y is now available 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.098818+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.099927+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.135560+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.146533+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.146873+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.169319+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.169643+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.988298+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.988572+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.988850+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.989065+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.989316+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.989569+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.989801+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T08:33:40.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.990025+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.990169+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:39.990615+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:40.004550+0000 mon.a (mon.0) 627 : cluster [INF] Manager daemon y is now available 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.098818+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.099927+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.135560+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.146533+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.146873+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.169319+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.169643+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T08:33:40.983 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.982+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mon.0 v1:172.21.15.107:6789/0 11 ==== mgrmap(e 18) v1 ==== 76894+0+0 (unknown 2122934099 0 0) 0x7f0e14076430 con 0x7f0e24061180 2024-02-10T08:33:40.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.982+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e04000b90 con 0x7f0e0c06e120 2024-02-10T08:33:40.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.985+0000 7f0e137fe700 10 client.14559.objecter ms_handle_connect 0x7f0e0c06e120 2024-02-10T08:33:40.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.993+0000 7f0e137fe700 1 -- 172.21.15.195:0/3614553889 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7f0e04000b90 con 0x7f0e0c06e120 2024-02-10T08:33:40.995 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled grafana update... 2024-02-10T08:33:40.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 10 librados: watch_flush enter 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 10 librados: watch_flush exit 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 20 client.14559.objecter shutdown clearing up homeless session... 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 10 client.14559.objecter successfully canceled tick 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0e0c06e120 legacy=0x7f0e0c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 >> v1:172.21.15.107:6789/0 conn(0x7f0e24061180 legacy=0x7f0e240fa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e22d9d700 1 -- 172.21.15.195:0/3614553889 reap_dead start 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 shutdown_connections 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 >> 172.21.15.195:0/3614553889 conn(0x7f0e240f1e10 msgr2=0x7f0e240f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 shutdown_connections 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 -- 172.21.15.195:0/3614553889 wait complete. 2024-02-10T08:33:40.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:40.995+0000 7f0e2359e700 1 librados: shutdown 2024-02-10T08:33:41.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:33:42.053 DEBUG:teuthology.orchestra.run.smithi195:grafana.a> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T08:33:42.055 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T08:33:42.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T08:33:42.084 INFO:journalctl@ceph.grafana.a.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:40.986366+0000 mon.a (mon.0) 631 : cluster [DBG] mgrmap e18: y(active, since 1.01142s), standbys: x 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:40.990712+0000 mgr.y (mgr.14553) 3 : cephadm [INF] Saving service grafana spec with placement smithi195=a;count:1 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:40.991571+0000 mgr.y (mgr.14553) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:40.994324+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.657810+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.939173+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.939578+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.940309+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.940541+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.941149+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.941359+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.941941+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.942148+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:41.946912+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:40.986366+0000 mon.a (mon.0) 631 : cluster [DBG] mgrmap e18: y(active, since 1.01142s), standbys: x 2024-02-10T08:33:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:40.990712+0000 mgr.y (mgr.14553) 3 : cephadm [INF] Saving service grafana spec with placement smithi195=a;count:1 2024-02-10T08:33:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:40.991571+0000 mgr.y (mgr.14553) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:40.994324+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.657810+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.939173+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.939578+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.940309+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.940541+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.941149+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.941359+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.941941+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.942148+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:41.946912+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:40.986366+0000 mon.a (mon.0) 631 : cluster [DBG] mgrmap e18: y(active, since 1.01142s), standbys: x 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:40.990712+0000 mgr.y (mgr.14553) 3 : cephadm [INF] Saving service grafana spec with placement smithi195=a;count:1 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:40.991571+0000 mgr.y (mgr.14553) 4 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:40.994324+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.657810+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.939173+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.939578+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.940309+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.940541+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.941149+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.941359+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.941941+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.942148+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:41.946912+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.881 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:41.942627+0000 mgr.y (mgr.14553) 5 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3768M 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:41.989102+0000 mgr.y (mgr.14553) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.687176+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.967621+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.967885+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.968547+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.968748+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.969342+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.969541+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.970144+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.970332+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.975242+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.976215+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:42.976766+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:43.882 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:42.993794+0000 mon.a (mon.0) 645 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:43.031397+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:43.072314+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:43.076360+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:41.942627+0000 mgr.y (mgr.14553) 5 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3768M 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:41.989102+0000 mgr.y (mgr.14553) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.687176+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.967621+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.967885+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.968547+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.968748+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.969342+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.969541+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.970144+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.970332+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.975242+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.976215+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:42.976766+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:42.993794+0000 mon.a (mon.0) 645 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:43.031397+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:43.072314+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:43.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:43.076360+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:41.942627+0000 mgr.y (mgr.14553) 5 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3768M 2024-02-10T08:33:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:41.989102+0000 mgr.y (mgr.14553) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.687176+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.967621+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.967885+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.968547+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.968748+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.969342+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.969541+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.970144+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.970332+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.975242+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.976215+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:42.976766+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:42.993794+0000 mon.a (mon.0) 645 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:43.031397+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:43.072314+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:43.076360+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 -- 172.21.15.107:0/2453586512 >> v1:172.21.15.107:6790/0 conn(0x7f0dbc0ff5e0 legacy=0x7f0dbc1019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 -- 172.21.15.107:0/2453586512 shutdown_connections 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 -- 172.21.15.107:0/2453586512 >> 172.21.15.107:0/2453586512 conn(0x7f0dbc06c800 msgr2=0x7f0dbc06cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 -- 172.21.15.107:0/2453586512 shutdown_connections 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 -- 172.21.15.107:0/2453586512 wait complete. 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 librados: starting msgr at 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.439+0000 7f0dc1b01700 1 librados: starting objecter 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 Processor -- start 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 -- start start 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 librados: setting wanted keys 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 librados: calling monclient init 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0dbc116130 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0dbc117310 con 0x7f0dbc103150 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dc1b01700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0dbc118510 con 0x7f0dbc0fc840 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dbb7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0dbc0fc840 0x7f0dbc0f8820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43324/0 (socket says 172.21.15.107:43324) 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dbb7fe700 1 -- 172.21.15.107:0/1955770218 learned_addr learned my addr 172.21.15.107:0/1955770218 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0dbaffd700 1 --1- 172.21.15.107:0/1955770218 >> v1:172.21.15.107:6789/0 conn(0x7f0dbc0ff5e0 0x7f0dbc0f8f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46136/0 (socket says 172.21.15.107:46136) 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter ms_handle_connect 0x7f0dbc0ff5e0 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter ms_handle_connect 0x7f0dbc0fc840 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387984249 0 0) 0x7f0dbc116130 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da4003030 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475359910 0 0) 0x7f0dbc118510 con 0x7f0dbc0fc840 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc116130 con 0x7f0dbc0fc840 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter ms_handle_connect 0x7f0dbc103150 2024-02-10T08:33:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.440+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590473773 0 0) 0x7f0da4003030 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc118510 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0db00015d0 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791992555 0 0) 0x7f0dbc117310 con 0x7f0dbc103150 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da4003030 con 0x7f0dbc103150 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926324434 0 0) 0x7f0dbc116130 con 0x7f0dbc0fc840 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc117310 con 0x7f0dbc0fc840 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0dac001840 con 0x7f0dbc0fc840 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4294006216 0 0) 0x7f0dbc118510 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 >> v1:172.21.15.107:6790/0 conn(0x7f0dbc0fc840 legacy=0x7f0dbc0f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 >> v1:172.21.15.195:6789/0 conn(0x7f0dbc103150 legacy=0x7f0dbc1147f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc119710 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc118700 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0dc1b01700 10 client.14574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc118bf0 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0dc1b01700 1 librados: init done 2024-02-10T08:33:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0db0001100 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.441+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0db00036c0 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.442+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8004f20 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.442+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0db0003840 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 6056+0+0 (unknown 4271144760 0 0) 0x7f0db00786f0 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 10 client.14574.objecter ms_dispatch 0x7f0dbc001550 osd_map(65..65 src has 1..65) v4 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 3 client.14574.objecter handle_osd_map got epochs [65,65] > 0 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 3 client.14574.objecter handle_osd_map decoding full epoch 65 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 20 client.14574.objecter dump_active .. 0 homeless 2024-02-10T08:33:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.443+0000 7f0db8ff9700 10 client.14574.objecter ms_handle_connect 0x7f0da4063d70 2024-02-10T08:33:44.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.447+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0db004c3c0 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.681+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0da8005750 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.685+0000 7f0db8ff9700 1 -- 172.21.15.107:0/1955770218 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 3970417695) 0x7f0db004a830 con 0x7f0dbc0ff5e0 2024-02-10T08:33:44.685 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-02-10T08:33:44.686 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDoNMdle8yuKBAA4Z8bm7OrXcHh93BxA6rNMQ== 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 10 librados: watch_flush enter 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 10 librados: watch_flush exit 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 20 client.14574.objecter shutdown clearing up homeless session... 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 10 client.14574.objecter successfully canceled tick 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0da4063d70 legacy=0x7f0da4066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 >> v1:172.21.15.107:6789/0 conn(0x7f0dbc0ff5e0 legacy=0x7f0dbc0f8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 shutdown_connections 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 >> 172.21.15.107:0/1955770218 conn(0x7f0dbc06c800 msgr2=0x7f0dbc06d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 shutdown_connections 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 -- 172.21.15.107:0/1955770218 wait complete. 2024-02-10T08:33:44.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:44.687+0000 7f0dc1b01700 1 librados: shutdown 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:42.970847+0000 mgr.y (mgr.14553) 7 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4024M 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:42.977760+0000 mgr.y (mgr.14553) 8 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:42.977995+0000 mgr.y (mgr.14553) 9 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:43.000487+0000 mgr.y (mgr.14553) 10 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:43.022421+0000 mgr.y (mgr.14553) 11 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:43.086330+0000 mgr.y (mgr.14553) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi107 2024-02-10T08:33:45.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:44.682397+0000 mon.a (mon.0) 649 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:45.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:44.685965+0000 mon.a (mon.0) 650 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:42.970847+0000 mgr.y (mgr.14553) 7 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4024M 2024-02-10T08:33:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:42.977760+0000 mgr.y (mgr.14553) 8 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:33:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:42.977995+0000 mgr.y (mgr.14553) 9 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:33:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:43.000487+0000 mgr.y (mgr.14553) 10 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:43.022421+0000 mgr.y (mgr.14553) 11 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:43.086330+0000 mgr.y (mgr.14553) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi107 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:44.682397+0000 mon.a (mon.0) 649 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:44.685965+0000 mon.a (mon.0) 650 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:42.970847+0000 mgr.y (mgr.14553) 7 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 4024M 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:42.977760+0000 mgr.y (mgr.14553) 8 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.conf 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:42.977995+0000 mgr.y (mgr.14553) 9 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.conf 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:43.000487+0000 mgr.y (mgr.14553) 10 : cephadm [INF] Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:43.022421+0000 mgr.y (mgr.14553) 11 : cephadm [INF] Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:43.086330+0000 mgr.y (mgr.14553) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi107 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:44.682397+0000 mon.a (mon.0) 649 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:44.685965+0000 mon.a (mon.0) 650 : audit [INF] from='client.? v1:172.21.15.107:0/1955770218' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:45.813 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:33:45.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T08:33:45.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T08:33:45.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T08:33:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:43.989575+0000 mgr.y (mgr.14553) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:46.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:43.989575+0000 mgr.y (mgr.14553) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:43.989575+0000 mgr.y (mgr.14553) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:47.500 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.497+0000 7fbc8ffde700 1 -- 172.21.15.195:0/4086344878 >> v1:172.21.15.195:6789/0 conn(0x7fbc880ff620 legacy=0x7fbc88101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.497+0000 7fbc8ffde700 1 -- 172.21.15.195:0/4086344878 shutdown_connections 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.497+0000 7fbc8ffde700 1 -- 172.21.15.195:0/4086344878 >> 172.21.15.195:0/4086344878 conn(0x7fbc8806c800 msgr2=0x7fbc8806cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.497+0000 7fbc8ffde700 1 -- 172.21.15.195:0/4086344878 shutdown_connections 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.497+0000 7fbc8ffde700 1 -- 172.21.15.195:0/4086344878 wait complete. 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 librados: starting msgr at 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 librados: starting objecter 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 Processor -- start 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 -- start start 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 librados: setting wanted keys 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 librados: calling monclient init 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc880fc0e0 con 0x7fbc88103160 2024-02-10T08:33:47.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc8811aa20 con 0x7fbc880fc820 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8ffde700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc8811bbc0 con 0x7fbc880ff620 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8dd7a700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbc880fc820 0x7fbc880fb810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:46630/0 (socket says 172.21.15.195:46630) 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc8dd7a700 1 -- 172.21.15.195:0/2354439159 learned_addr learned my addr 172.21.15.195:0/2354439159 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter ms_handle_connect 0x7fbc880fc820 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268708557 0 0) 0x7fbc8811aa20 con 0x7fbc880fc820 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc740033f0 con 0x7fbc880fc820 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter ms_handle_connect 0x7fbc880ff620 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter ms_handle_connect 0x7fbc88103160 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507029085 0 0) 0x7fbc8811bbc0 con 0x7fbc880ff620 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8811aa20 con 0x7fbc880ff620 2024-02-10T08:33:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753931248 0 0) 0x7fbc880fc0e0 con 0x7fbc88103160 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.498+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8811bbc0 con 0x7fbc88103160 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860243532 0 0) 0x7fbc740033f0 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc880fc0e0 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbc780029b0 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2333047703 0 0) 0x7fbc880fc0e0 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 >> v1:172.21.15.107:6790/0 conn(0x7fbc880ff620 legacy=0x7fbc88114bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 >> v1:172.21.15.107:6789/0 conn(0x7fbc88103160 legacy=0x7fbc881182f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8811cd60 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8811bdb0 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc8ffde700 10 client.24490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8811c220 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc8ffde700 1 librados: init done 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbc78002d40 con 0x7fbc880fc820 2024-02-10T08:33:47.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.499+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbc78005bc0 con 0x7fbc880fc820 2024-02-10T08:33:47.504 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6c004f50 con 0x7fbc880fc820 2024-02-10T08:33:47.504 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbc78001620 con 0x7fbc880fc820 2024-02-10T08:33:47.504 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 6056+0+0 (unknown 4271144760 0 0) 0x7fbc78078510 con 0x7fbc880fc820 2024-02-10T08:33:47.504 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc7effd700 10 client.24490.objecter ms_dispatch 0x7fbc88001550 osd_map(65..65 src has 1..65) v4 2024-02-10T08:33:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc7effd700 3 client.24490.objecter handle_osd_map got epochs [65,65] > 0 2024-02-10T08:33:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.500+0000 7fbc7effd700 3 client.24490.objecter handle_osd_map decoding full epoch 65 2024-02-10T08:33:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.501+0000 7fbc7effd700 20 client.24490.objecter dump_active .. 0 homeless 2024-02-10T08:33:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.501+0000 7fbc7effd700 10 client.24490.objecter ms_handle_connect 0x7fbc740636c0 2024-02-10T08:33:47.506 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.505+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbc7804c120 con 0x7fbc880fc820 2024-02-10T08:33:47.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.731+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fbc6c005780 con 0x7fbc880fc820 2024-02-10T08:33:47.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.735+0000 7fbc7effd700 1 -- 172.21.15.195:0/2354439159 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (unknown 3508017507 0 1664017921) 0x7fbc7804a530 con 0x7fbc880fc820 2024-02-10T08:33:47.736 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2024-02-10T08:33:47.736 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDrNMdlZ6C+KxAAqdWeoEpsK7DnTxYfxKYdPQ== 2024-02-10T08:33:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 10 librados: watch_flush enter 2024-02-10T08:33:47.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 10 librados: watch_flush exit 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 20 client.24490.objecter shutdown clearing up homeless session... 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 10 client.24490.objecter successfully canceled tick 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbc740636c0 legacy=0x7fbc74065b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 >> v1:172.21.15.195:6789/0 conn(0x7fbc880fc820 legacy=0x7fbc880fb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 shutdown_connections 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 >> 172.21.15.195:0/2354439159 conn(0x7fbc8806c800 msgr2=0x7fbc8806d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 shutdown_connections 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 -- 172.21.15.195:0/2354439159 wait complete. 2024-02-10T08:33:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-10T08:33:47.737+0000 7fbc8ffde700 1 librados: shutdown 2024-02-10T08:33:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:45.990078+0000 mgr.y (mgr.14553) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:48.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:45.990078+0000 mgr.y (mgr.14553) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:45.990078+0000 mgr.y (mgr.14553) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:33:48.401 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-10T08:33:48.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T08:33:48.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T08:33:48.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T08:33:49.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:47.733050+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.195:0/2354439159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:47.733786+0000 mon.a (mon.0) 651 : audit [INF] from='client.24490 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:47.736690+0000 mon.a (mon.0) 652 : audit [INF] from='client.24490 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:49.001 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:47.733050+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.195:0/2354439159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.001 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:47.733786+0000 mon.a (mon.0) 651 : audit [INF] from='client.24490 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.001 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:47.736690+0000 mon.a (mon.0) 652 : audit [INF] from='client.24490 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:47.733050+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.195:0/2354439159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:47.733786+0000 mon.a (mon.0) 651 : audit [INF] from='client.24490 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T08:33:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:47.736690+0000 mon.a (mon.0) 652 : audit [INF] from='client.24490 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T08:33:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:47.990625+0000 mgr.y (mgr.14553) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-10T08:33:50.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:47.990625+0000 mgr.y (mgr.14553) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-10T08:33:50.144 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:47.990625+0000 mgr.y (mgr.14553) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-10T08:33:50.421 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:33:51.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:33:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:33:51.741 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:51 smithi107 systemd[1]: Starting Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:33:52.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:49.990990+0000 mgr.y (mgr.14553) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-10T08:33:52.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:49.990990+0000 mgr.y (mgr.14553) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-10T08:33:52.094 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:49.990990+0000 mgr.y (mgr.14553) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.026+0000 7f8932e6c700 1 -- 172.21.15.107:0/3649475823 >> v1:172.21.15.107:6790/0 conn(0x7f892c104330 legacy=0x7f892c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.026+0000 7f8932e6c700 1 -- 172.21.15.107:0/3649475823 shutdown_connections 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.026+0000 7f8932e6c700 1 -- 172.21.15.107:0/3649475823 >> 172.21.15.107:0/3649475823 conn(0x7f892c0f3640 msgr2=0x7f892c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.026+0000 7f8932e6c700 1 -- 172.21.15.107:0/3649475823 shutdown_connections 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.026+0000 7f8932e6c700 1 -- 172.21.15.107:0/3649475823 wait complete. 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 librados: starting msgr at 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 librados: starting objecter 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 Processor -- start 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 -- start start 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 librados: setting wanted keys 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 librados: calling monclient init 2024-02-10T08:33:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f892c117480 con 0x7f892c0fda50 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f892c118680 con 0x7f892c1007f0 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8932e6c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f892c119880 con 0x7f892c104330 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8931409700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f892c104330 0x7f892c115b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48596/0 (socket says 172.21.15.107:48596) 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8930c08700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f892c0fda50 0x7f892c0f9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40100/0 (socket says 172.21.15.107:40100) 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8931409700 1 -- 172.21.15.107:0/336463838 learned_addr learned my addr 172.21.15.107:0/336463838 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter ms_handle_connect 0x7f892c0fda50 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter ms_handle_connect 0x7f892c104330 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter ms_handle_connect 0x7f892c1007f0 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220359376 0 0) 0x7f892c117480 con 0x7f892c0fda50 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c003630 con 0x7f892c0fda50 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.027+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950655674 0 0) 0x7f892c119880 con 0x7f892c104330 2024-02-10T08:33:53.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c117480 con 0x7f892c104330 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413841179 0 0) 0x7f892c118680 con 0x7f892c1007f0 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c119880 con 0x7f892c1007f0 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353366996 0 0) 0x7f891c003630 con 0x7f892c0fda50 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c118680 con 0x7f892c0fda50 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8920001800 con 0x7f892c0fda50 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349981443 0 0) 0x7f892c117480 con 0x7f892c104330 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f891c003630 con 0x7f892c104330 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89240030c0 con 0x7f892c104330 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1276516657 0 0) 0x7f892c119880 con 0x7f892c1007f0 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c117480 con 0x7f892c1007f0 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89180018a0 con 0x7f892c1007f0 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1646647648 0 0) 0x7f891c003630 con 0x7f892c104330 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 >> v1:172.21.15.195:6789/0 conn(0x7f892c1007f0 legacy=0x7f892c0fa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 >> v1:172.21.15.107:6789/0 conn(0x7f892c0fda50 legacy=0x7f892c0f9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:53.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c11aa80 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c119a70 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f89240032e0 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8924003cd0 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8932e6c700 10 client.34505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c119ee0 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.028+0000 7f8932e6c700 1 librados: init done 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.029+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c05f700 con 0x7f892c104330 2024-02-10T08:33:53.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.029+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8924003e30 con 0x7f892c104330 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 6056+0+0 (unknown 4271144760 0 0) 0x7f8924078b20 con 0x7f892c104330 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 10 client.34505.objecter ms_dispatch 0x7f892c001550 osd_map(65..65 src has 1..65) v4 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 3 client.34505.objecter handle_osd_map got epochs [65,65] > 0 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 3 client.34505.objecter handle_osd_map decoding full epoch 65 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 20 client.34505.objecter dump_active .. 0 homeless 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.030+0000 7f8929ffb700 10 client.34505.objecter ms_handle_connect 0x7f891c063ca0 2024-02-10T08:33:53.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.034+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f892404c730 con 0x7f892c104330 2024-02-10T08:33:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:50.995961+0000 mgr.y (mgr.14553) 17 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:33:53.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:50.995961+0000 mgr.y (mgr.14553) 17 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:33:53.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:50.995961+0000 mgr.y (mgr.14553) 17 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:33:53.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.204+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c0475e0 con 0x7f892c104330 2024-02-10T08:33:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.206+0000 7f8929ffb700 1 -- 172.21.15.107:0/336463838 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+19069 (unknown 1960508987 0 1365639330) 0x7f892404aba0 con 0x7f892c104330 2024-02-10T08:33:53.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:33:53.207 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","created":"2024-02-10T08:27:44.316161+0000","modified":"2024-02-10T08:33:39.974814+0000","last_up_change":"2024-02-10T08:33:10.747466+0000","last_in_change":"2024-02-10T08:32:53.709959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T08:31:09.850435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T08:33:16.728206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-10T08:33:16.960794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T08:33:18.379465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T08:33:20.354391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T08:33:22.423222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e30a7307-1dfd-4dfa-b4c4-89b83eb24167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":3321596972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":3321596972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":3321596972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":3321596972}]},"public_addr":"172.21.15.107:6801/3321596972","cluster_addr":"172.21.15.107:6802/3321596972","heartbeat_back_addr":"172.21.15.107:6804/3321596972","heartbeat_front_addr":"172.21.15.107:6803/3321596972","state":["exists","up"]},{"osd":1,"uuid":"c5fe7280-6de7-4c89-a9f2-22eebe4c1c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":133516126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":133516126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":133516126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":133516126}]},"public_addr":"172.21.15.107:6805/133516126","cluster_addr":"172.21.15.107:6806/133516126","heartbeat_back_addr":"172.21.15.107:6808/133516126","heartbeat_front_addr":"172.21.15.107:6807/133516126","state":["exists","up"]},{"osd":2,"uuid":"3ae7131b-69d5-4cbd-b1ca-29556fac7fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":3575881728}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":3575881728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":3575881728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":3575881728}]},"public_addr":"172.21.15.107:6809/3575881728","cluster_addr":"172.21.15.107:6810/3575881728","heartbeat_back_addr":"172.21.15.107:6812/3575881728","heartbeat_front_addr":"172.21.15.107:6811/3575881728","state":["exists","up"]},{"osd":3,"uuid":"1ffb2633-18bb-4048-90eb-7130e0cc7d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6813","nonce":2871825691}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6814","nonce":2871825691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6816","nonce":2871825691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6815","nonce":2871825691}]},"public_addr":"172.21.15.107:6813/2871825691","cluster_addr":"172.21.15.107:6814/2871825691","heartbeat_back_addr":"172.21.15.107:6816/2871825691","heartbeat_front_addr":"172.21.15.107:6815/2871825691","state":["exists","up"]},{"osd":4,"uuid":"1b7529a6-19fc-4435-be7f-28876750838a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6800","nonce":2499671285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6801","nonce":2499671285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6803","nonce":2499671285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6802","nonce":2499671285}]},"public_addr":"172.21.15.195:6800/2499671285","cluster_addr":"172.21.15.195:6801/2499671285","heartbeat_back_addr":"172.21.15.195:6803/2499671285","heartbeat_front_addr":"172.21.15.195:6802/2499671285","state":["exists","up"]},{"osd":5,"uuid":"bdc1b91c-94ce-4287-8019-535b326da3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6804","nonce":4178321486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6805","nonce":4178321486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6807","nonce":4178321486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6806","nonce":4178321486}]},"public_addr":"172.21.15.195:6804/4178321486","cluster_addr":"172.21.15.195:6805/4178321486","heartbeat_back_addr":"172.21.15.195:6807/4178321486","heartbeat_front_addr":"172.21.15.195:6806/4178321486","state":["exists","up"]},{"osd":6,"uuid":"217fdf91-33a7-40e2-a948-51b05ed63031","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6808","nonce":1417479017}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6809","nonce":1417479017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6811","nonce":1417479017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6810","nonce":1417479017}]},"public_addr":"172.21.15.195:6808/1417479017","cluster_addr":"172.21.15.195:6809/1417479017","heartbeat_back_addr":"172.21.15.195:6811/1417479017","heartbeat_front_addr":"172.21.15.195:6810/1417479017","state":["exists","up"]},{"osd":7,"uuid":"f0cd42da-4080-4e6a-843b-d73e10ce0948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6812","nonce":1934176928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6813","nonce":1934176928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6815","nonce":1934176928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6814","nonce":1934176928}]},"public_addr":"172.21.15.195:6812/1934176928","cluster_addr":"172.21.15.195:6813/1934176928","heartbeat_back_addr":"172.21.15.195:6815/1934176928","heartbeat_front_addr":"172.21.15.195:6814/1934176928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:32:49.998821+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:18.410238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:32:34.230950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:32:57.928272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:07.851558+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:25.538658+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:32:44.174241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:09.107053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1675475667":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/682167559":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/3360068681":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/2916003802":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1214732392":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/810931708":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1610921635":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/942960899":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/2081774553":"2024-02-11T08:28:29.992672+0000","172.21.15.107:6800/4244374259":"2024-02-11T08:28:07.029499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T08:33:53.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 10 librados: watch_flush enter 2024-02-10T08:33:53.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 10 librados: watch_flush exit 2024-02-10T08:33:53.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 20 client.34505.objecter shutdown clearing up homeless session... 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 10 client.34505.objecter successfully canceled tick 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 >> v1:172.21.15.107:6800/2634842009 conn(0x7f891c063ca0 legacy=0x7f891c066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.207+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 >> v1:172.21.15.107:6790/0 conn(0x7f892c104330 legacy=0x7f892c115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.208+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 shutdown_connections 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.208+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 >> 172.21.15.107:0/336463838 conn(0x7f892c0f3640 msgr2=0x7f892c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.208+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 shutdown_connections 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.208+0000 7f8932e6c700 1 -- 172.21.15.107:0/336463838 wait complete. 2024-02-10T08:33:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:53.208+0000 7f8932e6c700 1 librados: shutdown 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.143Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.143Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.143Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.145Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.170Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.170Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T08:33:53.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.172Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T08:33:53.431 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:53.172Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T08:33:53.431 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 bash[97150]: 02b1a43e9cac8528469e54f06e87413db0cfcfb104fe3adfaab075243a52b186 2024-02-10T08:33:53.431 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:53 smithi107 systemd[1]: Started Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:33:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:51.991721+0000 mgr.y (mgr.14553) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-10T08:33:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:53.206035+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.107:0/336463838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:33:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:53.336226+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:51.991721+0000 mgr.y (mgr.14553) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-10T08:33:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:53.206035+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.107:0/336463838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:33:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:53.336226+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:53.960 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}] 2024-02-10T08:33:53.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-10T08:33:54.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:51.991721+0000 mgr.y (mgr.14553) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-10T08:33:54.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:53.206035+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.107:0/336463838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:33:54.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:53.336226+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:33:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:53.341208+0000 mgr.y (mgr.14553) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi107 2024-02-10T08:33:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:53.992042+0000 mgr.y (mgr.14553) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:33:54.931 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:54 smithi107 bash[97534]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T08:33:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:53.341208+0000 mgr.y (mgr.14553) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi107 2024-02-10T08:33:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:53.992042+0000 mgr.y (mgr.14553) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:33:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:53.341208+0000 mgr.y (mgr.14553) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi107 2024-02-10T08:33:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:53.992042+0000 mgr.y (mgr.14553) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:33:55.399 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:33:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:33:55.146Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000909354s 2024-02-10T08:33:55.399 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:54 smithi107 bash[97534]: Getting image source signatures 2024-02-10T08:33:55.399 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:54 smithi107 bash[97534]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T08:33:55.399 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:54 smithi107 bash[97534]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T08:33:55.399 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:54 smithi107 bash[97534]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T08:33:56.145 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:55 smithi107 bash[97534]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T08:33:56.145 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:56 smithi107 bash[97534]: Writing manifest to image destination 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.208+0000 7f9787cfb700 1 -- 172.21.15.107:0/2048516607 >> v1:172.21.15.107:6789/0 conn(0x7f97800fc910 legacy=0x7f97800fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.208+0000 7f9787cfb700 1 -- 172.21.15.107:0/2048516607 shutdown_connections 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.208+0000 7f9787cfb700 1 -- 172.21.15.107:0/2048516607 >> 172.21.15.107:0/2048516607 conn(0x7f97800f2510 msgr2=0x7f97800f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 -- 172.21.15.107:0/2048516607 shutdown_connections 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 -- 172.21.15.107:0/2048516607 wait complete. 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 librados: starting msgr at 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 librados: starting objecter 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 Processor -- start 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 -- start start 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 librados: setting wanted keys 2024-02-10T08:33:56.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.209+0000 7f9787cfb700 1 librados: calling monclient init 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97800fbfe0 con 0x7f9780103250 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97800fc1b0 con 0x7f97800fc910 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f978011bb20 con 0x7f97800ff710 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9785296700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f97800ff710 0x7f9780114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48614/0 (socket says 172.21.15.107:48614) 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9785296700 1 -- 172.21.15.107:0/942328447 learned_addr learned my addr 172.21.15.107:0/942328447 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter ms_handle_connect 0x7f9780103250 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter ms_handle_connect 0x7f97800ff710 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter ms_handle_connect 0x7f97800fc910 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482407721 0 0) 0x7f97800fbfe0 con 0x7f9780103250 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976c0038e0 con 0x7f9780103250 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768391822 0 0) 0x7f978011bb20 con 0x7f97800ff710 2024-02-10T08:33:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97800fbfe0 con 0x7f97800ff710 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483969516 0 0) 0x7f97800fc1b0 con 0x7f97800fc910 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978011bb20 con 0x7f97800fc910 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918287174 0 0) 0x7f976c0038e0 con 0x7f9780103250 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97800fc1b0 con 0x7f9780103250 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9778001d70 con 0x7f9780103250 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936631650 0 0) 0x7f97800fbfe0 con 0x7f97800ff710 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f976c0038e0 con 0x7f97800ff710 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9770001840 con 0x7f97800ff710 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915167835 0 0) 0x7f978011bb20 con 0x7f97800fc910 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97800fbfe0 con 0x7f97800fc910 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3044506615 0 0) 0x7f97800fc1b0 con 0x7f9780103250 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 >> v1:172.21.15.107:6790/0 conn(0x7f97800ff710 legacy=0x7f9780114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 >> v1:172.21.15.195:6789/0 conn(0x7f97800fc910 legacy=0x7f97800fb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978011ccc0 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978011bd10 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 10 client.14592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f978011c180 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9787cfb700 1 librados: init done 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.210+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9778002620 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.211+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9778003550 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.211+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764004f20 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9778002180 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 6056+0+0 (unknown 4271144760 0 0) 0x7f9778075f70 con 0x7f9780103250 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 10 client.14592.objecter ms_dispatch 0x7f9780001550 osd_map(65..65 src has 1..65) v4 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 3 client.14592.objecter handle_osd_map got epochs [65,65] > 0 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 3 client.14592.objecter handle_osd_map decoding full epoch 65 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 20 client.14592.objecter dump_active .. 0 homeless 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.212+0000 7f9776ffd700 10 client.14592.objecter ms_handle_connect 0x7f976c063c90 2024-02-10T08:33:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.216+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9778049ba0 con 0x7f9780103250 2024-02-10T08:33:56.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.381+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9764005750 con 0x7f9780103250 2024-02-10T08:33:56.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.381+0000 7f9776ffd700 1 -- 172.21.15.107:0/942328447 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v65) v1 ==== 110+0+10 (unknown 2581670197 0 2170607528) 0x7f9778048010 con 0x7f9780103250 2024-02-10T08:33:56.382 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 10 librados: watch_flush enter 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 10 librados: watch_flush exit 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 20 client.14592.objecter shutdown clearing up homeless session... 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 10 client.14592.objecter successfully canceled tick 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 >> v1:172.21.15.107:6800/2634842009 conn(0x7f976c063c90 legacy=0x7f976c066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 >> v1:172.21.15.107:6789/0 conn(0x7f9780103250 legacy=0x7f97801183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 shutdown_connections 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 >> 172.21.15.107:0/942328447 conn(0x7f97800f2510 msgr2=0x7f97801066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 shutdown_connections 2024-02-10T08:33:56.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 -- 172.21.15.107:0/942328447 wait complete. 2024-02-10T08:33:56.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:56.383+0000 7f9787cfb700 1 librados: shutdown 2024-02-10T08:33:56.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:55.140203+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:56.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:55.140203+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:55.140203+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:57.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:55.992548+0000 mgr.y (mgr.14553) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:33:57.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:56.382152+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? v1:172.21.15.107:0/942328447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T08:33:57.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:55.992548+0000 mgr.y (mgr.14553) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:33:57.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:56.382152+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? v1:172.21.15.107:0/942328447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.147Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.147Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.147Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T08:33:57.417 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T08:33:57.418 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.148Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T08:33:57.419 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T08:33:57.420 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a[97669]: ts=2024-02-10T08:33:57.149Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T08:33:57.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get .rgw.root pg_num 2024-02-10T08:33:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:55.992548+0000 mgr.y (mgr.14553) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:33:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:56.382152+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? v1:172.21.15.107:0/942328447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-10T08:33:57.680 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 bash[97534]: 497f1048db96173fe9b4410c05d4571496f17d589fe4d9e4bac15119fd746be0 2024-02-10T08:33:57.680 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:33:57 smithi107 systemd[1]: Started Ceph node-exporter.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:33:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:33:58.139 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 systemd[1]: Starting Ceph node-exporter.b for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:33:58.626 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 bash[92779]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-10T08:33:58.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:57.623403+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:58.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:33:57.626259+0000 mgr.y (mgr.14553) 22 : cephadm [INF] Deploying daemon node-exporter.b on smithi195 2024-02-10T08:33:58.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:57.623403+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:58.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:33:57.626259+0000 mgr.y (mgr.14553) 22 : cephadm [INF] Deploying daemon node-exporter.b on smithi195 2024-02-10T08:33:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:57.623403+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:33:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:33:57.626259+0000 mgr.y (mgr.14553) 22 : cephadm [INF] Deploying daemon node-exporter.b on smithi195 2024-02-10T08:33:59.037 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 bash[92779]: Getting image source signatures 2024-02-10T08:33:59.037 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 bash[92779]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-10T08:33:59.038 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 bash[92779]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-10T08:33:59.038 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:33:58 smithi195 bash[92779]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-10T08:33:59.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.338+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/3209875200 >> v1:172.21.15.107:6790/0 conn(0x7f9a78103250 legacy=0x7f9a781056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:59.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.338+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/3209875200 shutdown_connections 2024-02-10T08:33:59.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.338+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/3209875200 >> 172.21.15.107:0/3209875200 conn(0x7f9a780f2510 msgr2=0x7f9a780f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.338+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/3209875200 shutdown_connections 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.338+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/3209875200 wait complete. 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 librados: starting msgr at 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 librados: starting objecter 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 Processor -- start 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 -- start start 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 librados: setting wanted keys 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 librados: calling monclient init 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a78116250 con 0x7f9a780ff710 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a78117430 con 0x7f9a78103250 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7f2ca700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a78118630 con 0x7f9a780fc910 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7d066700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9a780fc910 0x7f9a780f8810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42564/0 (socket says 172.21.15.107:42564) 2024-02-10T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a7d066700 1 -- 172.21.15.107:0/2412423462 learned_addr learned my addr 172.21.15.107:0/2412423462 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter ms_handle_connect 0x7f9a780fc910 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter ms_handle_connect 0x7f9a780ff710 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter ms_handle_connect 0x7f9a78103250 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.339+0000 7f9a6e777700 10 client.?.objecter resend_mon_ops 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724797733 0 0) 0x7f9a78116250 con 0x7f9a780ff710 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003800 con 0x7f9a780ff710 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972448312 0 0) 0x7f9a78118630 con 0x7f9a780fc910 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78116250 con 0x7f9a780fc910 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118097009 0 0) 0x7f9a78117430 con 0x7f9a78103250 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78118630 con 0x7f9a78103250 2024-02-10T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086795176 0 0) 0x7f9a64003800 con 0x7f9a780ff710 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a78117430 con 0x7f9a780ff710 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a740018c0 con 0x7f9a780ff710 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2468858757 0 0) 0x7f9a78118630 con 0x7f9a78103250 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a64003800 con 0x7f9a78103250 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1785627769 0 0) 0x7f9a78116250 con 0x7f9a780fc910 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a78118630 con 0x7f9a780fc910 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a70001930 con 0x7f9a78103250 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a680019f0 con 0x7f9a780fc910 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3245155415 0 0) 0x7f9a78117430 con 0x7f9a780ff710 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 >> v1:172.21.15.107:6790/0 conn(0x7f9a780fc910 legacy=0x7f9a780f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 >> v1:172.21.15.195:6789/0 conn(0x7f9a78103250 legacy=0x7f9a78114910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78119830 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a78116440 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a7f2ca700 10 client.14598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a78116930 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a7f2ca700 1 librados: init done 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9a74002040 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.340+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a74003310 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.341+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c004f20 con 0x7f9a780ff710 2024-02-10T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.341+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9a740161a0 con 0x7f9a780ff710 2024-02-10T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 6056+0+0 (unknown 4271144760 0 0) 0x7f9a74075c90 con 0x7f9a780ff710 2024-02-10T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 10 client.14598.objecter ms_dispatch 0x7f9a78001550 osd_map(65..65 src has 1..65) v4 2024-02-10T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 3 client.14598.objecter handle_osd_map got epochs [65,65] > 0 2024-02-10T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 3 client.14598.objecter handle_osd_map decoding full epoch 65 2024-02-10T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 20 client.14598.objecter dump_active .. 0 homeless 2024-02-10T08:33:59.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.342+0000 7f9a6e777700 10 client.14598.objecter ms_handle_connect 0x7f9a64063d10 2024-02-10T08:33:59.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.346+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9a740498c0 con 0x7f9a780ff710 2024-02-10T08:33:59.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.508+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"} v 0) v1 -- 0x7f9a5c005750 con 0x7f9a780ff710 2024-02-10T08:33:59.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.509+0000 7f9a6e777700 1 -- 172.21.15.107:0/2412423462 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]=0 v65) v1 ==== 98+0+11 (unknown 2572135718 0 3519293104) 0x7f9a74047cd0 con 0x7f9a780ff710 2024-02-10T08:33:59.510 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.510+0000 7f9a7f2ca700 10 librados: watch_flush enter 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.510+0000 7f9a7f2ca700 10 librados: watch_flush exit 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.510+0000 7f9a7f2ca700 20 client.14598.objecter shutdown clearing up homeless session... 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.510+0000 7f9a7f2ca700 10 client.14598.objecter successfully canceled tick 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9a64063d10 legacy=0x7f9a640661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 >> v1:172.21.15.107:6789/0 conn(0x7f9a780ff710 legacy=0x7f9a780f8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:33:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 shutdown_connections 2024-02-10T08:33:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 >> 172.21.15.107:0/2412423462 conn(0x7f9a780f2510 msgr2=0x7f9a781066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:33:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 shutdown_connections 2024-02-10T08:33:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 -- 172.21.15.107:0/2412423462 wait complete. 2024-02-10T08:33:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:33:59.511+0000 7f9a7f2ca700 1 librados: shutdown 2024-02-10T08:33:59.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:57.992949+0000 mgr.y (mgr.14553) 23 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:33:59.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:33:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:33:59.509802+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? v1:172.21.15.107:0/2412423462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-10T08:33:59.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:57.992949+0000 mgr.y (mgr.14553) 23 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:33:59.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:33:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:33:59.509802+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? v1:172.21.15.107:0/2412423462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-10T08:34:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:57.992949+0000 mgr.y (mgr.14553) 23 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:33:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:33:59.509802+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? v1:172.21.15.107:0/2412423462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-10T08:34:00.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get iscsi pg_num 2024-02-10T08:34:00.451 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:00.537 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:00 smithi195 bash[92779]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-10T08:34:00.537 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:00 smithi195 bash[92779]: Writing manifest to image destination 2024-02-10T08:34:01.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.068Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.068Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.068Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.068Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=arp 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-10T08:34:01.288 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=edac 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.069Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-10T08:34:01.289 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=os 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-10T08:34:01.290 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=stat 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=time 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=uname 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b[92831]: ts=2024-02-10T08:34:01.070Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 bash[92779]: b3113240a72e741c618a5ebb1fd3900f0cfb195dbc2b24735166a61c7e08015c 2024-02-10T08:34:01.291 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:34:01 smithi195 systemd[1]: Started Ceph node-exporter.b for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:33:59.993345+0000 mgr.y (mgr.14553) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:00.072150+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:00.072230+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:00.633381+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:00.635074+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:00.635389+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:01.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:00.636286+0000 mon.c (mon.2) 48 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:01.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:33:59.993345+0000 mgr.y (mgr.14553) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:00.072150+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:00.072230+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:00.633381+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:00.635074+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:00.635389+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:00.636286+0000 mon.c (mon.2) 48 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:00.636513+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:01.200749+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:00.636513+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:01.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:01.200749+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:33:59.993345+0000 mgr.y (mgr.14553) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:00.072150+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:00.072230+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:00.633381+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-10T08:34:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:00.635074+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:00.635389+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T08:34:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:00.636286+0000 mon.c (mon.2) 48 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:00.636513+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-10T08:34:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:01.200749+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.173+0000 7f946369c700 1 -- 172.21.15.107:0/1444339764 >> v1:172.21.15.107:6789/0 conn(0x7f945c103250 legacy=0x7f945c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.173+0000 7f946369c700 1 -- 172.21.15.107:0/1444339764 shutdown_connections 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.173+0000 7f946369c700 1 -- 172.21.15.107:0/1444339764 >> 172.21.15.107:0/1444339764 conn(0x7f945c0f2510 msgr2=0x7f945c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- 172.21.15.107:0/1444339764 shutdown_connections 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- 172.21.15.107:0/1444339764 wait complete. 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 librados: starting msgr at 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 librados: starting objecter 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 Processor -- start 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- start start 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 librados: setting wanted keys 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 librados: calling monclient init 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f945c119650 con 0x7f945c103250 2024-02-10T08:34:02.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f945c11a830 con 0x7f945c0fc910 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f946369c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f945c11ba30 con 0x7f945c0ff710 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f9460c37700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f945c0ff710 0x7f945c061190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42590/0 (socket says 172.21.15.107:42590) 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f9461c39700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f945c103250 0x7f945c117d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41096/0 (socket says 172.21.15.107:41096) 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.174+0000 7f9460c37700 1 -- 172.21.15.107:0/960099134 learned_addr learned my addr 172.21.15.107:0/960099134 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter ms_handle_connect 0x7f945c103250 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter ms_handle_connect 0x7f945c0ff710 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter ms_handle_connect 0x7f945c0fc910 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495441506 0 0) 0x7f945c119650 con 0x7f945c103250 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003030 con 0x7f945c103250 2024-02-10T08:34:02.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777368392 0 0) 0x7f945c11ba30 con 0x7f945c0ff710 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c119650 con 0x7f945c0ff710 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762281638 0 0) 0x7f945c11a830 con 0x7f945c0fc910 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c11ba30 con 0x7f945c0fc910 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365404530 0 0) 0x7f9448003030 con 0x7f945c103250 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f945c11a830 con 0x7f945c103250 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9454002470 con 0x7f945c103250 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 267845719 0 0) 0x7f945c119650 con 0x7f945c0ff710 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9448003030 con 0x7f945c0ff710 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f944c001840 con 0x7f945c0ff710 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489897871 0 0) 0x7f945c11ba30 con 0x7f945c0fc910 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f945c119650 con 0x7f945c0fc910 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9458001920 con 0x7f945c0fc910 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237746593 0 0) 0x7f945c11a830 con 0x7f945c103250 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 >> v1:172.21.15.107:6790/0 conn(0x7f945c0ff710 legacy=0x7f945c061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 >> v1:172.21.15.195:6789/0 conn(0x7f945c0fc910 legacy=0x7f945c060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c11cc30 con 0x7f945c103250 2024-02-10T08:34:02.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c11bc20 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f946369c700 10 client.14604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c11c090 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f946369c700 1 librados: init done 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9454001560 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.175+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9454003a10 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.176+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9440004f20 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.176+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9454001bc0 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(67..67 src has 1..67) v4 ==== 6056+0+0 (unknown 82449408 0 0) 0x7f9454078ac0 con 0x7f945c103250 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 10 client.14604.objecter ms_dispatch 0x7f945c001550 osd_map(67..67 src has 1..67) v4 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 3 client.14604.objecter handle_osd_map got epochs [67,67] > 0 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 3 client.14604.objecter handle_osd_map decoding full epoch 67 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 20 client.14604.objecter dump_active .. 0 homeless 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.177+0000 7f94527fc700 10 client.14604.objecter ms_handle_connect 0x7f9448063c70 2024-02-10T08:34:02.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.181+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f945404c730 con 0x7f945c103250 2024-02-10T08:34:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.347+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"} v 0) v1 -- 0x7f9440005750 con 0x7f945c103250 2024-02-10T08:34:02.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.348+0000 7f94527fc700 1 -- 172.21.15.107:0/960099134 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]=0 v67) v1 ==== 94+0+10 (unknown 4104206529 0 2787501894) 0x7f945404aba0 con 0x7f945c103250 2024-02-10T08:34:02.348 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 3 2024-02-10T08:34:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.349+0000 7f946369c700 10 librados: watch_flush enter 2024-02-10T08:34:02.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 10 librados: watch_flush exit 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 20 client.14604.objecter shutdown clearing up homeless session... 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 10 client.14604.objecter successfully canceled tick 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9448063c70 legacy=0x7f9448066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 >> v1:172.21.15.107:6789/0 conn(0x7f945c103250 legacy=0x7f945c117d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 shutdown_connections 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 >> 172.21.15.107:0/960099134 conn(0x7f945c0f2510 msgr2=0x7f945c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 shutdown_connections 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 -- 172.21.15.107:0/960099134 wait complete. 2024-02-10T08:34:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:02.350+0000 7f946369c700 1 librados: shutdown 2024-02-10T08:34:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:01.001451+0000 mgr.y (mgr.14553) 25 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:34:01.212847+0000 mgr.y (mgr.14553) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi195 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:01.637600+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:01.637697+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:02.348598+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? v1:172.21.15.107:0/960099134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:01.001451+0000 mgr.y (mgr.14553) 25 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:34:01.212847+0000 mgr.y (mgr.14553) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi195 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:01.637600+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:01.637697+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T08:34:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:02.348598+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? v1:172.21.15.107:0/960099134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-10T08:34:02.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get default.rgw.log pg_num 2024-02-10T08:34:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:01.001451+0000 mgr.y (mgr.14553) 25 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:34:01.212847+0000 mgr.y (mgr.14553) 26 : cephadm [INF] Deploying daemon prometheus.a on smithi195 2024-02-10T08:34:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:01.637600+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-10T08:34:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:01.637697+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-10T08:34:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:02.348598+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? v1:172.21.15.107:0/960099134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-10T08:34:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:03.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:34:03.147Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002348321s 2024-02-10T08:34:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:01.993945+0000 mgr.y (mgr.14553) 27 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:03.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:01.993945+0000 mgr.y (mgr.14553) 27 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:01.993945+0000 mgr.y (mgr.14553) 27 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:05.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.017+0000 7fe2d4374700 1 -- 172.21.15.107:0/822318824 >> v1:172.21.15.107:6790/0 conn(0x7fe2cc104330 legacy=0x7fe2cc1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.017+0000 7fe2d4374700 1 -- 172.21.15.107:0/822318824 shutdown_connections 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.017+0000 7fe2d4374700 1 -- 172.21.15.107:0/822318824 >> 172.21.15.107:0/822318824 conn(0x7fe2cc0f3640 msgr2=0x7fe2cc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.017+0000 7fe2d4374700 1 -- 172.21.15.107:0/822318824 shutdown_connections 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.017+0000 7fe2d4374700 1 -- 172.21.15.107:0/822318824 wait complete. 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 librados: starting msgr at 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 librados: starting objecter 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 Processor -- start 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 -- start start 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 librados: setting wanted keys 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 librados: calling monclient init 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2cc06cc10 con 0x7fe2cc104330 2024-02-10T08:34:05.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2cc06cda0 con 0x7fe2cc0fda50 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d4374700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2cc06cf90 con 0x7fe2cc1007f0 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d2911700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe2cc104330 0x7fe2cc1184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41126/0 (socket says 172.21.15.107:41126) 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d190f700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe2cc1007f0 0x7fe2cc114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42608/0 (socket says 172.21.15.107:42608) 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2d2911700 1 -- 172.21.15.107:0/906085394 learned_addr learned my addr 172.21.15.107:0/906085394 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter ms_handle_connect 0x7fe2cc104330 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter ms_handle_connect 0x7fe2cc1007f0 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter ms_handle_connect 0x7fe2cc0fda50 2024-02-10T08:34:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822272804 0 0) 0x7fe2cc06cf90 con 0x7fe2cc1007f0 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b80033f0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056587691 0 0) 0x7fe2cc06cda0 con 0x7fe2cc0fda50 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc06cf90 con 0x7fe2cc0fda50 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701739360 0 0) 0x7fe2cc06cc10 con 0x7fe2cc104330 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.018+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc06cda0 con 0x7fe2cc104330 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750855231 0 0) 0x7fe2b80033f0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2cc06cc10 con 0x7fe2cc1007f0 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe2c80019d0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839711473 0 0) 0x7fe2cc06cda0 con 0x7fe2cc104330 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b80033f0 con 0x7fe2cc104330 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe2c4002310 con 0x7fe2cc104330 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3316275496 0 0) 0x7fe2cc06cf90 con 0x7fe2cc0fda50 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2cc06cda0 con 0x7fe2cc0fda50 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe2bc0018c0 con 0x7fe2cc0fda50 2024-02-10T08:34:05.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4025145685 0 0) 0x7fe2cc06cc10 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 >> v1:172.21.15.195:6789/0 conn(0x7fe2cc0fda50 legacy=0x7fe2cc0fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 >> v1:172.21.15.107:6789/0 conn(0x7fe2cc104330 legacy=0x7fe2cc1184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc11cc00 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc119c30 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2d4374700 10 client.34529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc11a0a0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2d4374700 1 librados: init done 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe2c8002340 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.019+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe2c8005950 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.020+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b0004f20 con 0x7fe2cc1007f0 2024-02-10T08:34:05.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.020+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe2c80024b0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 6056+0+0 (unknown 1924917797 0 0) 0x7fe2c80783a0 con 0x7fe2cc1007f0 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 10 client.34529.objecter ms_dispatch 0x7fe2cc001550 osd_map(68..68 src has 1..68) v4 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 3 client.34529.objecter handle_osd_map got epochs [68,68] > 0 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 3 client.34529.objecter handle_osd_map decoding full epoch 68 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 20 client.34529.objecter dump_active .. 0 homeless 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.021+0000 7fe2c37fe700 10 client.34529.objecter ms_handle_connect 0x7fe2b8063f40 2024-02-10T08:34:05.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.025+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe2c804bf70 con 0x7fe2cc1007f0 2024-02-10T08:34:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.189+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"} v 0) v1 -- 0x7fe2b0005750 con 0x7fe2cc1007f0 2024-02-10T08:34:05.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.189+0000 7fe2c37fe700 1 -- 172.21.15.107:0/906085394 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (unknown 2415937256 0 3519293104) 0x7fe2c804a380 con 0x7fe2cc1007f0 2024-02-10T08:34:05.190 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 10 librados: watch_flush enter 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 10 librados: watch_flush exit 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 20 client.34529.objecter shutdown clearing up homeless session... 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 10 client.34529.objecter successfully canceled tick 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe2b8063f40 legacy=0x7fe2b8066400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 >> v1:172.21.15.107:6790/0 conn(0x7fe2cc1007f0 legacy=0x7fe2cc114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 shutdown_connections 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 >> 172.21.15.107:0/906085394 conn(0x7fe2cc0f3640 msgr2=0x7fe2cc1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 shutdown_connections 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 -- 172.21.15.107:0/906085394 wait complete. 2024-02-10T08:34:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:05.191+0000 7fe2d4374700 1 librados: shutdown 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:03.994425+0000 mgr.y (mgr.14553) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:04.537706+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:05.150940+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:05.190112+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.107:0/906085394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:03.994425+0000 mgr.y (mgr.14553) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:04.537706+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:05.150940+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:05.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:05.190112+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.107:0/906085394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-10T08:34:05.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get default.rgw.control pg_num 2024-02-10T08:34:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:03.994425+0000 mgr.y (mgr.14553) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:04.537706+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-10T08:34:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:05.150940+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:05.190112+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.107:0/906085394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-10T08:34:06.179 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:07.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:05.995111+0000 mgr.y (mgr.14553) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:34:07.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:05.995111+0000 mgr.y (mgr.14553) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:34:07.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.900+0000 7fb993c15700 1 -- 172.21.15.107:0/1198890455 >> v1:172.21.15.107:6790/0 conn(0x7fb98c100810 legacy=0x7fb98c102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.900+0000 7fb993c15700 1 -- 172.21.15.107:0/1198890455 shutdown_connections 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.900+0000 7fb993c15700 1 -- 172.21.15.107:0/1198890455 >> 172.21.15.107:0/1198890455 conn(0x7fb98c0f3630 msgr2=0x7fb98c0f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.900+0000 7fb993c15700 1 -- 172.21.15.107:0/1198890455 shutdown_connections 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.900+0000 7fb993c15700 1 -- 172.21.15.107:0/1198890455 wait complete. 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 librados: starting msgr at 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 librados: starting objecter 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 Processor -- start 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 -- start start 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 librados: setting wanted keys 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 librados: calling monclient init 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb98c1174b0 con 0x7fb98c100810 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb98c118690 con 0x7fb98c104320 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb993c15700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb98c119890 con 0x7fb98c0fda30 2024-02-10T08:34:07.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb9919b1700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb98c0fda30 0x7fb98c100db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42620/0 (socket says 172.21.15.107:42620) 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb9919b1700 1 -- 172.21.15.107:0/2132865850 learned_addr learned my addr 172.21.15.107:0/2132865850 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter ms_handle_connect 0x7fb98c100810 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter ms_handle_connect 0x7fb98c0fda30 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter ms_handle_connect 0x7fb98c104320 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758896751 0 0) 0x7fb98c1174b0 con 0x7fb98c100810 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9780033f0 con 0x7fb98c100810 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946231107 0 0) 0x7fb98c119890 con 0x7fb98c0fda30 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98c1174b0 con 0x7fb98c0fda30 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504943497 0 0) 0x7fb98c118690 con 0x7fb98c104320 2024-02-10T08:34:07.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.901+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98c119890 con 0x7fb98c104320 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2016911700 0 0) 0x7fb9780033f0 con 0x7fb98c100810 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb98c118690 con 0x7fb98c100810 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3836370415 0 0) 0x7fb98c1174b0 con 0x7fb98c0fda30 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9780033f0 con 0x7fb98c0fda30 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb97c002cb0 con 0x7fb98c100810 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb988001880 con 0x7fb98c0fda30 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625884938 0 0) 0x7fb98c119890 con 0x7fb98c104320 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb98c1174b0 con 0x7fb98c104320 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb984001c40 con 0x7fb98c104320 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1926540586 0 0) 0x7fb98c118690 con 0x7fb98c100810 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 >> v1:172.21.15.107:6790/0 conn(0x7fb98c0fda30 legacy=0x7fb98c100db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 >> v1:172.21.15.195:6789/0 conn(0x7fb98c104320 legacy=0x7fb98c115b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:07.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c11aa90 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98c119a80 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb993c15700 10 client.14616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c119ef0 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb993c15700 1 librados: init done 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb97c002f50 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.902+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb97c003690 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb970004f20 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb97c0037f0 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 6056+0+0 (unknown 1924917797 0 0) 0x7fb97c075eb0 con 0x7fb98c100810 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 10 client.14616.objecter ms_dispatch 0x7fb98c001550 osd_map(68..68 src has 1..68) v4 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 3 client.14616.objecter handle_osd_map got epochs [68,68] > 0 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 3 client.14616.objecter handle_osd_map decoding full epoch 68 2024-02-10T08:34:07.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.903+0000 7fb982ffd700 20 client.14616.objecter dump_active .. 0 homeless 2024-02-10T08:34:07.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.904+0000 7fb982ffd700 10 client.14616.objecter ms_handle_connect 0x7fb978063c50 2024-02-10T08:34:07.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:07.908+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb97c049ae0 con 0x7fb98c100810 2024-02-10T08:34:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:05.995111+0000 mgr.y (mgr.14553) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:34:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.068+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"} v 0) v1 -- 0x7fb970005750 con 0x7fb98c100810 2024-02-10T08:34:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.069+0000 7fb982ffd700 1 -- 172.21.15.107:0/2132865850 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]=0 v68) v1 ==== 108+0+11 (unknown 3370406734 0 3519293104) 0x7fb97c047f50 con 0x7fb98c100810 2024-02-10T08:34:08.069 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 10 librados: watch_flush enter 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 10 librados: watch_flush exit 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 20 client.14616.objecter shutdown clearing up homeless session... 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 10 client.14616.objecter successfully canceled tick 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb978063c50 legacy=0x7fb978066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 >> v1:172.21.15.107:6789/0 conn(0x7fb98c100810 legacy=0x7fb98c1014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 shutdown_connections 2024-02-10T08:34:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 >> 172.21.15.107:0/2132865850 conn(0x7fb98c0f3630 msgr2=0x7fb98c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 shutdown_connections 2024-02-10T08:34:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 -- 172.21.15.107:0/2132865850 wait complete. 2024-02-10T08:34:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:08.070+0000 7fb993c15700 1 librados: shutdown 2024-02-10T08:34:08.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:08.069582+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? v1:172.21.15.107:0/2132865850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-10T08:34:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:08.069582+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? v1:172.21.15.107:0/2132865850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-10T08:34:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:08.069582+0000 mon.a (mon.0) 667 : audit [DBG] from='client.? v1:172.21.15.107:0/2132865850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-10T08:34:08.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool get default.rgw.meta pg_num 2024-02-10T08:34:09.113 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:09.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:07.995447+0000 mgr.y (mgr.14553) 30 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:34:09.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:09.540633+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T08:34:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:07.995447+0000 mgr.y (mgr.14553) 30 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:34:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:09.540633+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T08:34:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:07.995447+0000 mgr.y (mgr.14553) 30 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:34:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:09.540633+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-10T08:34:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:09.016481+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:09.016525+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:09.016481+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:09.016525+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:10.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:09.016481+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:10.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:09.016525+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:10.717 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:10 smithi195 systemd[1]: Starting Ceph prometheus.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.886+0000 7f66939fd700 1 -- 172.21.15.107:0/2380096709 >> v1:172.21.15.107:6789/0 conn(0x7f668c104330 legacy=0x7f668c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.886+0000 7f66939fd700 1 -- 172.21.15.107:0/2380096709 shutdown_connections 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.886+0000 7f66939fd700 1 -- 172.21.15.107:0/2380096709 >> 172.21.15.107:0/2380096709 conn(0x7f668c0f3640 msgr2=0x7f668c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.886+0000 7f66939fd700 1 -- 172.21.15.107:0/2380096709 shutdown_connections 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.886+0000 7f66939fd700 1 -- 172.21.15.107:0/2380096709 wait complete. 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 librados: starting msgr at 2024-02-10T08:34:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 librados: starting objecter 2024-02-10T08:34:10.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 Processor -- start 2024-02-10T08:34:10.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 -- start start 2024-02-10T08:34:10.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 librados: setting wanted keys 2024-02-10T08:34:10.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 librados: calling monclient init 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f668c117330 con 0x7f668c0fda50 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f668c118510 con 0x7f668c1007f0 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66939fd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f668c119710 con 0x7f668c104330 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f6691799700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f668c0fda50 0x7f668c0f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38022/0 (socket says 172.21.15.107:38022) 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f6691f9a700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f668c104330 0x7f668c1159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43074/0 (socket says 172.21.15.107:43074) 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f6691799700 1 -- 172.21.15.107:0/288143422 learned_addr learned my addr 172.21.15.107:0/288143422 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter ms_handle_connect 0x7f668c0fda50 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter ms_handle_connect 0x7f668c104330 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter ms_handle_connect 0x7f668c1007f0 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315905755 0 0) 0x7f668c117330 con 0x7f668c0fda50 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6678003800 con 0x7f668c0fda50 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105706891 0 0) 0x7f668c119710 con 0x7f668c104330 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668c117330 con 0x7f668c104330 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678341773 0 0) 0x7f668c118510 con 0x7f668c1007f0 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.887+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668c119710 con 0x7f668c1007f0 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2166974530 0 0) 0x7f6678003800 con 0x7f668c0fda50 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f668c118510 con 0x7f668c0fda50 2024-02-10T08:34:10.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f667c001980 con 0x7f668c0fda50 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883827450 0 0) 0x7f668c117330 con 0x7f668c104330 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6678003800 con 0x7f668c104330 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66840030d0 con 0x7f668c104330 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85633560 0 0) 0x7f668c119710 con 0x7f668c1007f0 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f668c117330 con 0x7f668c1007f0 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66880018d0 con 0x7f668c1007f0 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 722698484 0 0) 0x7f668c118510 con 0x7f668c0fda50 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 >> v1:172.21.15.107:6790/0 conn(0x7f668c104330 legacy=0x7f668c1159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 >> v1:172.21.15.195:6789/0 conn(0x7f668c1007f0 legacy=0x7f668c0fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c11a910 con 0x7f668c0fda50 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c119900 con 0x7f668c0fda50 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66939fd700 10 client.14622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c119df0 con 0x7f668c0fda50 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66939fd700 1 librados: init done 2024-02-10T08:34:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f667c002100 con 0x7f668c0fda50 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.888+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f667c005a70 con 0x7f668c0fda50 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.889+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6670004f20 con 0x7f668c0fda50 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.889+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f667c001c60 con 0x7f668c0fda50 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 6056+0+0 (unknown 260366265 0 0) 0x7f667c0783b0 con 0x7f668c0fda50 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 10 client.14622.objecter ms_dispatch 0x7f668c001550 osd_map(69..69 src has 1..69) v4 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 3 client.14622.objecter handle_osd_map got epochs [69,69] > 0 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 3 client.14622.objecter handle_osd_map decoding full epoch 69 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 20 client.14622.objecter dump_active .. 0 homeless 2024-02-10T08:34:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.890+0000 7f66827fc700 10 client.14622.objecter ms_handle_connect 0x7f6678063d40 2024-02-10T08:34:10.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:10.894+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f667c04bfc0 con 0x7f668c0fda50 2024-02-10T08:34:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.057+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"} v 0) v1 -- 0x7f6670005750 con 0x7f668c0fda50 2024-02-10T08:34:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.057+0000 7f66827fc700 1 -- 172.21.15.107:0/288143422 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]=0 v69) v1 ==== 105+0+11 (unknown 2571780633 0 3519293104) 0x7f667c046e60 con 0x7f668c0fda50 2024-02-10T08:34:11.058 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 10 librados: watch_flush enter 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 10 librados: watch_flush exit 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 20 client.14622.objecter shutdown clearing up homeless session... 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 10 client.14622.objecter successfully canceled tick 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6678063d40 legacy=0x7f6678066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 >> v1:172.21.15.107:6789/0 conn(0x7f668c0fda50 legacy=0x7f668c0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 shutdown_connections 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 >> 172.21.15.107:0/288143422 conn(0x7f668c0f3640 msgr2=0x7f668c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 shutdown_connections 2024-02-10T08:34:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 -- 172.21.15.107:0/288143422 wait complete. 2024-02-10T08:34:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:11.059+0000 7f66939fd700 1 librados: shutdown 2024-02-10T08:34:11.283 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:11.537 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.283Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-10T08:34:11.537 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.283Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-10T08:34:11.537 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.283Z caller=main.go:518 level=info host_details="(Linux 4.18.0-536.el8.x86_64 #1 SMP Thu Jan 18 15:15:29 UTC 2024 x86_64 smithi195 (none))" 2024-02-10T08:34:11.537 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.283Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.283Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.285Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.285Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.287Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.288Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.288Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.85µs 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.288Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.288Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.288Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.448µs wal_replay_duration=149.272µs total_replay_duration=188.303µs 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.289Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.289Z caller=main.go:947 level=info msg="TSDB started" 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.289Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.304Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.108874ms db_storage=792ns remote_storage=1.22µs web_handler=318ns query_engine=852ns scrape=564µs scrape_sd=22.265µs notify=22.809µs notify_sd=6.355µs rules=14.245477ms 2024-02-10T08:34:11.538 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:34:11.304Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-10T08:34:11.539 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 bash[93292]: e544cd2226d5413462d061b9dc4b87ada3f8b10e9a223d234871d7ef0b358182 2024-02-10T08:34:11.539 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:34:11 smithi195 systemd[1]: Started Ceph prometheus.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:34:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:09.995806+0000 mgr.y (mgr.14553) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.057988+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? v1:172.21.15.107:0/288143422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-10T08:34:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.392726+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.446523+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.454733+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.459702+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.466391+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE Bus STOPPING 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE Bus STOPPED 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE Bus STARTING 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE Serving on http://:::9283 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:11] ENGINE Bus STARTED 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:09.995806+0000 mgr.y (mgr.14553) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.057988+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? v1:172.21.15.107:0/288143422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-10T08:34:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.392726+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.446523+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.454733+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.459702+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.466391+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:11.789 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T08:34:11.789 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T08:34:11.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T08:34:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:09.995806+0000 mgr.y (mgr.14553) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.057988+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? v1:172.21.15.107:0/288143422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-10T08:34:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.392726+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.446523+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.454733+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:12.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.459702+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:12.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.466391+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:12.895 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.008927+0000 mgr.y (mgr.14553) 32 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:12.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:11.460056+0000 mgr.y (mgr.14553) 33 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:12.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:34:11.476402+0000 mgr.y (mgr.14553) 34 : cephadm [INF] Deploying daemon grafana.a on smithi195 2024-02-10T08:34:12.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.008927+0000 mgr.y (mgr.14553) 32 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:12.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:11.460056+0000 mgr.y (mgr.14553) 33 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:12.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:34:11.476402+0000 mgr.y (mgr.14553) 34 : cephadm [INF] Deploying daemon grafana.a on smithi195 2024-02-10T08:34:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.008927+0000 mgr.y (mgr.14553) 32 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:11.460056+0000 mgr.y (mgr.14553) 33 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T08:34:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:34:11.476402+0000 mgr.y (mgr.14553) 34 : cephadm [INF] Deploying daemon grafana.a on smithi195 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- 172.21.15.107:0/669376504 >> v1:172.21.15.107:6790/0 conn(0x7f89e80f9570 legacy=0x7f89e8102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- 172.21.15.107:0/669376504 shutdown_connections 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- 172.21.15.107:0/669376504 >> 172.21.15.107:0/669376504 conn(0x7f89e80f2510 msgr2=0x7f89e80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- 172.21.15.107:0/669376504 shutdown_connections 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- 172.21.15.107:0/669376504 wait complete. 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 librados: starting msgr at 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 librados: starting objecter 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 Processor -- start 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 -- start start 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 librados: setting wanted keys 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.759+0000 7f89f01e9700 1 librados: calling monclient init 2024-02-10T08:34:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89e81173e0 con 0x7f89e80f89c0 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89e81185c0 con 0x7f89e8104350 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89e81197c0 con 0x7f89e80f9570 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89edf85700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f89e80f89c0 0x7f89e80ff510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38042/0 (socket says 172.21.15.107:38042) 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89edf85700 1 -- 172.21.15.107:0/4077473928 learned_addr learned my addr 172.21.15.107:0/4077473928 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter ms_handle_connect 0x7f89e80f89c0 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter ms_handle_connect 0x7f89e80f9570 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter ms_handle_connect 0x7f89e8104350 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062750593 0 0) 0x7f89e81197c0 con 0x7f89e80f9570 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003800 con 0x7f89e80f9570 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053541157 0 0) 0x7f89e81173e0 con 0x7f89e80f89c0 2024-02-10T08:34:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e81197c0 con 0x7f89e80f89c0 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448058664 0 0) 0x7f89e81185c0 con 0x7f89e8104350 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e81173e0 con 0x7f89e8104350 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319770930 0 0) 0x7f89d4003800 con 0x7f89e80f9570 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e81185c0 con 0x7f89e80f9570 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89d8002f60 con 0x7f89e80f9570 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872499262 0 0) 0x7f89e81197c0 con 0x7f89e80f89c0 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89d4003800 con 0x7f89e80f89c0 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89e4001900 con 0x7f89e80f89c0 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484853276 0 0) 0x7f89e81173e0 con 0x7f89e8104350 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e81197c0 con 0x7f89e8104350 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89e0001c50 con 0x7f89e8104350 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650863784 0 0) 0x7f89e81185c0 con 0x7f89e80f9570 2024-02-10T08:34:13.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 >> v1:172.21.15.195:6789/0 conn(0x7f89e8104350 legacy=0x7f89e8115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 >> v1:172.21.15.107:6789/0 conn(0x7f89e80f89c0 legacy=0x7f89e80ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e811a9c0 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e81199b0 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 10 client.34547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e8119e20 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89f01e9700 1 librados: init done 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f89d8001210 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.760+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f89d8003540 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.761+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc004f20 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f89d8002050 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6056+0+0 (unknown 2294645021 0 0) 0x7f89d8075fa0 con 0x7f89e80f9570 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 10 client.34547.objecter ms_dispatch 0x7f89e8001550 osd_map(70..70 src has 1..70) v4 2024-02-10T08:34:13.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 3 client.34547.objecter handle_osd_map got epochs [70,70] > 0 2024-02-10T08:34:13.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 3 client.34547.objecter handle_osd_map decoding full epoch 70 2024-02-10T08:34:13.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 20 client.34547.objecter dump_active .. 0 homeless 2024-02-10T08:34:13.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.762+0000 7f89deffd700 10 client.34547.objecter ms_handle_connect 0x7f89d4063ca0 2024-02-10T08:34:13.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.766+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f89d8049b70 con 0x7f89e80f9570 2024-02-10T08:34:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:11.996362+0000 mgr.y (mgr.14553) 35 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:13.130610+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:13.130672+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:11.996362+0000 mgr.y (mgr.14553) 35 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:13.130610+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:13.130672+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:13.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.974+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f89cc0051d0 con 0x7f89e80f9570 2024-02-10T08:34:13.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.977+0000 7f89deffd700 1 -- 172.21.15.107:0/4077473928 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+151942 (unknown 3383617513 0 240213242) 0x7f89d8047fe0 con 0x7f89e80f9570 2024-02-10T08:34:13.981 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:34:13.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 10 librados: watch_flush enter 2024-02-10T08:34:13.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 10 librados: watch_flush exit 2024-02-10T08:34:13.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 20 client.34547.objecter shutdown clearing up homeless session... 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 10 client.34547.objecter successfully canceled tick 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 >> v1:172.21.15.107:6800/2634842009 conn(0x7f89d4063ca0 legacy=0x7f89d4066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 >> v1:172.21.15.107:6790/0 conn(0x7f89e80f9570 legacy=0x7f89e80ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 shutdown_connections 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.979+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 >> 172.21.15.107:0/4077473928 conn(0x7f89e80f2510 msgr2=0x7f89e8103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.980+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 shutdown_connections 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.980+0000 7f89f01e9700 1 -- 172.21.15.107:0/4077473928 wait complete. 2024-02-10T08:34:13.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:13.980+0000 7f89f01e9700 1 librados: shutdown 2024-02-10T08:34:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:11.996362+0000 mgr.y (mgr.14553) 35 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:13.130610+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:13.130672+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:14.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:13.577782+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T08:34:14.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:13.976881+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.107:0/4077473928' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T08:34:14.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:14.540614+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T08:34:14.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:13.577782+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T08:34:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:13.976881+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.107:0/4077473928' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T08:34:14.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:14.540614+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T08:34:14.861 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"active_gid":14553,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":2634842009}]},"active_addr":"172.21.15.107:6800/2634842009","active_change":"2024-02-10T08:33:39.974929+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":24457,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":65,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2319187403}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":366194623}]}]} 2024-02-10T08:34:14.864 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T08:34:14.864 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T08:34:14.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T08:34:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:13.577782+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-10T08:34:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:13.976881+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.107:0/4077473928' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T08:34:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:14.540614+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-10T08:34:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:15.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:13.996745+0000 mgr.y (mgr.14553) 36 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:15.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:15.169309+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:15.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:13.996745+0000 mgr.y (mgr.14553) 36 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:15.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:15.169309+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:13.996745+0000 mgr.y (mgr.14553) 36 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:15.169309+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:16.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.796+0000 7fd47b632700 1 -- 172.21.15.107:0/823754289 >> v1:172.21.15.107:6789/0 conn(0x7fd4740ff6d0 legacy=0x7fd474101ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.796+0000 7fd47b632700 1 -- 172.21.15.107:0/823754289 shutdown_connections 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.796+0000 7fd47b632700 1 -- 172.21.15.107:0/823754289 >> 172.21.15.107:0/823754289 conn(0x7fd47406cc30 msgr2=0x7fd4740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.796+0000 7fd47b632700 1 -- 172.21.15.107:0/823754289 shutdown_connections 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.796+0000 7fd47b632700 1 -- 172.21.15.107:0/823754289 wait complete. 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 librados: starting msgr at 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 librados: starting objecter 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 Processor -- start 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 -- start start 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 librados: setting wanted keys 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 librados: calling monclient init 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4741172a0 con 0x7fd4740fc8f0 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd474118480 con 0x7fd4740ff6d0 2024-02-10T08:34:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd47b632700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd474119680 con 0x7fd474103210 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd4793ce700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd4740fc8f0 0x7fd4740f6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38060/0 (socket says 172.21.15.107:38060) 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd4793ce700 1 -- 172.21.15.107:0/1369508245 learned_addr learned my addr 172.21.15.107:0/1369508245 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter ms_handle_connect 0x7fd4740fc8f0 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter ms_handle_connect 0x7fd474103210 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter ms_handle_connect 0x7fd4740ff6d0 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.797+0000 7fd46a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807414880 0 0) 0x7fd474119680 con 0x7fd474103210 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd460003800 con 0x7fd474103210 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975406695 0 0) 0x7fd4741172a0 con 0x7fd4740fc8f0 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd474119680 con 0x7fd4740fc8f0 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426445106 0 0) 0x7fd474118480 con 0x7fd4740ff6d0 2024-02-10T08:34:16.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4741172a0 con 0x7fd4740ff6d0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364506571 0 0) 0x7fd460003800 con 0x7fd474103210 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd474118480 con 0x7fd474103210 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd46c001d70 con 0x7fd474103210 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1259637220 0 0) 0x7fd4741172a0 con 0x7fd4740ff6d0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd460003800 con 0x7fd4740ff6d0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd470002c60 con 0x7fd4740ff6d0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241406705 0 0) 0x7fd474119680 con 0x7fd4740fc8f0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4741172a0 con 0x7fd4740fc8f0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd464001840 con 0x7fd4740fc8f0 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2320209463 0 0) 0x7fd474118480 con 0x7fd474103210 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 >> v1:172.21.15.195:6789/0 conn(0x7fd4740ff6d0 legacy=0x7fd4740f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 >> v1:172.21.15.107:6789/0 conn(0x7fd4740fc8f0 legacy=0x7fd4740f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47411a880 con 0x7fd474103210 2024-02-10T08:34:16.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474117490 con 0x7fd474103210 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd47b632700 10 client.34553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd46c002620 con 0x7fd474103210 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd474117980 con 0x7fd474103210 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd47b632700 1 librados: init done 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.798+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd46c003540 con 0x7fd474103210 2024-02-10T08:34:16.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.799+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd458004f20 con 0x7fd474103210 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd46c002180 con 0x7fd474103210 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (unknown 1684438359 0 0) 0x7fd46c075e20 con 0x7fd474103210 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 10 client.34553.objecter ms_dispatch 0x7fd474001550 osd_map(72..72 src has 1..72) v4 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 3 client.34553.objecter handle_osd_map got epochs [72,72] > 0 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 3 client.34553.objecter handle_osd_map decoding full epoch 72 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 20 client.34553.objecter dump_active .. 0 homeless 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.800+0000 7fd46a7fc700 10 client.34553.objecter ms_handle_connect 0x7fd460063dc0 2024-02-10T08:34:16.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.804+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd46c049a50 con 0x7fd474103210 2024-02-10T08:34:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:15.576753+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:15.576802+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:16.173134+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T08:34:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:15.576753+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:15.576802+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:16.173134+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T08:34:16.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.965+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4580051d0 con 0x7fd474103210 2024-02-10T08:34:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.966+0000 7fd46a7fc700 1 -- 172.21.15.107:0/1369508245 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+19069 (unknown 1686360691 0 1396005442) 0x7fd46c047e60 con 0x7fd474103210 2024-02-10T08:34:16.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:34:16.968 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","created":"2024-02-10T08:27:44.316161+0000","modified":"2024-02-10T08:34:16.167775+0000","last_up_change":"2024-02-10T08:33:10.747466+0000","last_in_change":"2024-02-10T08:32:53.709959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T08:31:09.850435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T08:33:16.728206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-10T08:33:16.960794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T08:33:18.379465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T08:33:20.354391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T08:33:22.423222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e30a7307-1dfd-4dfa-b4c4-89b83eb24167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":3321596972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":3321596972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":3321596972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":3321596972}]},"public_addr":"172.21.15.107:6801/3321596972","cluster_addr":"172.21.15.107:6802/3321596972","heartbeat_back_addr":"172.21.15.107:6804/3321596972","heartbeat_front_addr":"172.21.15.107:6803/3321596972","state":["exists","up"]},{"osd":1,"uuid":"c5fe7280-6de7-4c89-a9f2-22eebe4c1c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":133516126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":133516126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":133516126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":133516126}]},"public_addr":"172.21.15.107:6805/133516126","cluster_addr":"172.21.15.107:6806/133516126","heartbeat_back_addr":"172.21.15.107:6808/133516126","heartbeat_front_addr":"172.21.15.107:6807/133516126","state":["exists","up"]},{"osd":2,"uuid":"3ae7131b-69d5-4cbd-b1ca-29556fac7fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":3575881728}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":3575881728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":3575881728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":3575881728}]},"public_addr":"172.21.15.107:6809/3575881728","cluster_addr":"172.21.15.107:6810/3575881728","heartbeat_back_addr":"172.21.15.107:6812/3575881728","heartbeat_front_addr":"172.21.15.107:6811/3575881728","state":["exists","up"]},{"osd":3,"uuid":"1ffb2633-18bb-4048-90eb-7130e0cc7d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6813","nonce":2871825691}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6814","nonce":2871825691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6816","nonce":2871825691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6815","nonce":2871825691}]},"public_addr":"172.21.15.107:6813/2871825691","cluster_addr":"172.21.15.107:6814/2871825691","heartbeat_back_addr":"172.21.15.107:6816/2871825691","heartbeat_front_addr":"172.21.15.107:6815/2871825691","state":["exists","up"]},{"osd":4,"uuid":"1b7529a6-19fc-4435-be7f-28876750838a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6800","nonce":2499671285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6801","nonce":2499671285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6803","nonce":2499671285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6802","nonce":2499671285}]},"public_addr":"172.21.15.195:6800/2499671285","cluster_addr":"172.21.15.195:6801/2499671285","heartbeat_back_addr":"172.21.15.195:6803/2499671285","heartbeat_front_addr":"172.21.15.195:6802/2499671285","state":["exists","up"]},{"osd":5,"uuid":"bdc1b91c-94ce-4287-8019-535b326da3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6804","nonce":4178321486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6805","nonce":4178321486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6807","nonce":4178321486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6806","nonce":4178321486}]},"public_addr":"172.21.15.195:6804/4178321486","cluster_addr":"172.21.15.195:6805/4178321486","heartbeat_back_addr":"172.21.15.195:6807/4178321486","heartbeat_front_addr":"172.21.15.195:6806/4178321486","state":["exists","up"]},{"osd":6,"uuid":"217fdf91-33a7-40e2-a948-51b05ed63031","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6808","nonce":1417479017}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6809","nonce":1417479017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6811","nonce":1417479017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6810","nonce":1417479017}]},"public_addr":"172.21.15.195:6808/1417479017","cluster_addr":"172.21.15.195:6809/1417479017","heartbeat_back_addr":"172.21.15.195:6811/1417479017","heartbeat_front_addr":"172.21.15.195:6810/1417479017","state":["exists","up"]},{"osd":7,"uuid":"f0cd42da-4080-4e6a-843b-d73e10ce0948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6812","nonce":1934176928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6813","nonce":1934176928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6815","nonce":1934176928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6814","nonce":1934176928}]},"public_addr":"172.21.15.195:6812/1934176928","cluster_addr":"172.21.15.195:6813/1934176928","heartbeat_back_addr":"172.21.15.195:6815/1934176928","heartbeat_front_addr":"172.21.15.195:6814/1934176928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:09.016527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:18.410238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:00.072234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:15.576804+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:07.851558+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:25.538658+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:13.130674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:09.107053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1675475667":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/682167559":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/3360068681":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/2916003802":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1214732392":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/810931708":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1610921635":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/942960899":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/2081774553":"2024-02-11T08:28:29.992672+0000","172.21.15.107:6800/4244374259":"2024-02-11T08:28:07.029499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 10 librados: watch_flush enter 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 10 librados: watch_flush exit 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 20 client.34553.objecter shutdown clearing up homeless session... 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 10 client.34553.objecter successfully canceled tick 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd460063dc0 legacy=0x7fd460066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 >> v1:172.21.15.107:6790/0 conn(0x7fd474103210 legacy=0x7fd474115960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 shutdown_connections 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 >> 172.21.15.107:0/1369508245 conn(0x7fd47406cc30 msgr2=0x7fd4741066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 shutdown_connections 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 -- 172.21.15.107:0/1369508245 wait complete. 2024-02-10T08:34:16.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:16.968+0000 7fd47b632700 1 librados: shutdown 2024-02-10T08:34:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:15.576753+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:15.576802+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:16.173134+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-10T08:34:17.611 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T08:34:17.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T08:34:17.874 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:15.997514+0000 mgr.y (mgr.14553) 37 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:34:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:16.967559+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.107:0/1369508245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:15.997514+0000 mgr.y (mgr.14553) 37 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:34:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:16.967559+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.107:0/1369508245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:15.997514+0000 mgr.y (mgr.14553) 37 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:34:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:16.967559+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.107:0/1369508245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:19.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:17.998043+0000 mgr.y (mgr.14553) 38 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:19.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:17.998043+0000 mgr.y (mgr.14553) 38 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:17.998043+0000 mgr.y (mgr.14553) 38 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:34:20.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.136+0000 7f822e802700 1 -- 172.21.15.107:0/3232419148 >> v1:172.21.15.107:6790/0 conn(0x7f8228103250 legacy=0x7f82281056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:20.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.136+0000 7f822e802700 1 -- 172.21.15.107:0/3232419148 shutdown_connections 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.136+0000 7f822e802700 1 -- 172.21.15.107:0/3232419148 >> 172.21.15.107:0/3232419148 conn(0x7f82280f2510 msgr2=0x7f82280f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.136+0000 7f822e802700 1 -- 172.21.15.107:0/3232419148 shutdown_connections 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.136+0000 7f822e802700 1 -- 172.21.15.107:0/3232419148 wait complete. 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 librados: starting msgr at 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 librados: starting objecter 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 Processor -- start 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 -- start start 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 librados: setting wanted keys 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 librados: calling monclient init 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8228117310 con 0x7f8228103250 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82281184f0 con 0x7f82280fc910 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.137+0000 7f822e802700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82281196f0 con 0x7f82280ff710 2024-02-10T08:34:20.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82277fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f82280ff710 0x7f8228061300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54686/0 (socket says 172.21.15.107:54686) 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82277fe700 1 -- 172.21.15.107:0/1131979075 learned_addr learned my addr 172.21.15.107:0/1131979075 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f822cd9f700 1 --1- 172.21.15.107:0/1131979075 >> v1:172.21.15.107:6789/0 conn(0x7f8228103250 0x7f8228115a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46482/0 (socket says 172.21.15.107:46482) 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter ms_handle_connect 0x7f82280ff710 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter ms_handle_connect 0x7f8228103250 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter ms_handle_connect 0x7f82280fc910 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132010852 0 0) 0x7f82281196f0 con 0x7f82280ff710 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210003800 con 0x7f82280ff710 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332771375 0 0) 0x7f8228117310 con 0x7f8228103250 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82281196f0 con 0x7f8228103250 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187971186 0 0) 0x7f82281184f0 con 0x7f82280fc910 2024-02-10T08:34:20.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228117310 con 0x7f82280fc910 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141058651 0 0) 0x7f82281196f0 con 0x7f8228103250 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82281184f0 con 0x7f8228103250 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8220003160 con 0x7f8228103250 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 187775539 0 0) 0x7f8210003800 con 0x7f82280ff710 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82281196f0 con 0x7f82280ff710 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378214476 0 0) 0x7f8228117310 con 0x7f82280fc910 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8210003800 con 0x7f82280fc910 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f821c002890 con 0x7f82280ff710 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f82180018d0 con 0x7f82280fc910 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085820641 0 0) 0x7f82281184f0 con 0x7f8228103250 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 >> v1:172.21.15.107:6790/0 conn(0x7f82280ff710 legacy=0x7f8228061300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 >> v1:172.21.15.195:6789/0 conn(0x7f82280fc910 legacy=0x7f8228060bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:20.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822811a8f0 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8228117500 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f822e802700 10 client.14640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82281179f0 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f822e802700 1 librados: init done 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8220001450 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.138+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8220003b90 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.139+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8214004f20 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8220002260 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (unknown 1684438359 0 0) 0x7f8220078af0 con 0x7f8228103250 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 10 client.14640.objecter ms_dispatch 0x7f8228001550 osd_map(72..72 src has 1..72) v4 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 3 client.14640.objecter handle_osd_map got epochs [72,72] > 0 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 3 client.14640.objecter handle_osd_map decoding full epoch 72 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 20 client.14640.objecter dump_active .. 0 homeless 2024-02-10T08:34:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.140+0000 7f82257fa700 10 client.14640.objecter ms_handle_connect 0x7f8210063cd0 2024-02-10T08:34:20.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.144+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f822004c780 con 0x7f8228103250 2024-02-10T08:34:20.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.308+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82140051d0 con 0x7f8228103250 2024-02-10T08:34:20.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.309+0000 7f82257fa700 1 -- 172.21.15.107:0/1131979075 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+19069 (unknown 1686360691 0 1396005442) 0x7f822004abf0 con 0x7f8228103250 2024-02-10T08:34:20.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:34:20.312 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","created":"2024-02-10T08:27:44.316161+0000","modified":"2024-02-10T08:34:16.167775+0000","last_up_change":"2024-02-10T08:33:10.747466+0000","last_in_change":"2024-02-10T08:32:53.709959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T08:31:09.850435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T08:33:16.728206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-10T08:33:16.960794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T08:33:18.379465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T08:33:20.354391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T08:33:22.423222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e30a7307-1dfd-4dfa-b4c4-89b83eb24167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":3321596972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":3321596972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":3321596972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":3321596972}]},"public_addr":"172.21.15.107:6801/3321596972","cluster_addr":"172.21.15.107:6802/3321596972","heartbeat_back_addr":"172.21.15.107:6804/3321596972","heartbeat_front_addr":"172.21.15.107:6803/3321596972","state":["exists","up"]},{"osd":1,"uuid":"c5fe7280-6de7-4c89-a9f2-22eebe4c1c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":133516126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":133516126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":133516126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":133516126}]},"public_addr":"172.21.15.107:6805/133516126","cluster_addr":"172.21.15.107:6806/133516126","heartbeat_back_addr":"172.21.15.107:6808/133516126","heartbeat_front_addr":"172.21.15.107:6807/133516126","state":["exists","up"]},{"osd":2,"uuid":"3ae7131b-69d5-4cbd-b1ca-29556fac7fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":3575881728}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":3575881728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":3575881728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":3575881728}]},"public_addr":"172.21.15.107:6809/3575881728","cluster_addr":"172.21.15.107:6810/3575881728","heartbeat_back_addr":"172.21.15.107:6812/3575881728","heartbeat_front_addr":"172.21.15.107:6811/3575881728","state":["exists","up"]},{"osd":3,"uuid":"1ffb2633-18bb-4048-90eb-7130e0cc7d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6813","nonce":2871825691}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6814","nonce":2871825691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6816","nonce":2871825691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6815","nonce":2871825691}]},"public_addr":"172.21.15.107:6813/2871825691","cluster_addr":"172.21.15.107:6814/2871825691","heartbeat_back_addr":"172.21.15.107:6816/2871825691","heartbeat_front_addr":"172.21.15.107:6815/2871825691","state":["exists","up"]},{"osd":4,"uuid":"1b7529a6-19fc-4435-be7f-28876750838a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6800","nonce":2499671285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6801","nonce":2499671285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6803","nonce":2499671285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6802","nonce":2499671285}]},"public_addr":"172.21.15.195:6800/2499671285","cluster_addr":"172.21.15.195:6801/2499671285","heartbeat_back_addr":"172.21.15.195:6803/2499671285","heartbeat_front_addr":"172.21.15.195:6802/2499671285","state":["exists","up"]},{"osd":5,"uuid":"bdc1b91c-94ce-4287-8019-535b326da3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6804","nonce":4178321486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6805","nonce":4178321486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6807","nonce":4178321486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6806","nonce":4178321486}]},"public_addr":"172.21.15.195:6804/4178321486","cluster_addr":"172.21.15.195:6805/4178321486","heartbeat_back_addr":"172.21.15.195:6807/4178321486","heartbeat_front_addr":"172.21.15.195:6806/4178321486","state":["exists","up"]},{"osd":6,"uuid":"217fdf91-33a7-40e2-a948-51b05ed63031","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6808","nonce":1417479017}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6809","nonce":1417479017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6811","nonce":1417479017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6810","nonce":1417479017}]},"public_addr":"172.21.15.195:6808/1417479017","cluster_addr":"172.21.15.195:6809/1417479017","heartbeat_back_addr":"172.21.15.195:6811/1417479017","heartbeat_front_addr":"172.21.15.195:6810/1417479017","state":["exists","up"]},{"osd":7,"uuid":"f0cd42da-4080-4e6a-843b-d73e10ce0948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6812","nonce":1934176928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6813","nonce":1934176928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6815","nonce":1934176928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6814","nonce":1934176928}]},"public_addr":"172.21.15.195:6812/1934176928","cluster_addr":"172.21.15.195:6813/1934176928","heartbeat_back_addr":"172.21.15.195:6815/1934176928","heartbeat_front_addr":"172.21.15.195:6814/1934176928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:09.016527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:18.410238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:00.072234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:15.576804+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:07.851558+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:25.538658+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:34:13.130674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:33:09.107053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1675475667":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/682167559":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/3360068681":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/2916003802":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1214732392":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/810931708":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1610921635":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/942960899":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/2081774553":"2024-02-11T08:28:29.992672+0000","172.21.15.107:6800/4244374259":"2024-02-11T08:28:07.029499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T08:34:20.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 10 librados: watch_flush enter 2024-02-10T08:34:20.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 10 librados: watch_flush exit 2024-02-10T08:34:20.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 20 client.14640.objecter shutdown clearing up homeless session... 2024-02-10T08:34:20.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 10 client.14640.objecter successfully canceled tick 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8210063cd0 legacy=0x7f8210066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 >> v1:172.21.15.107:6789/0 conn(0x7f8228103250 legacy=0x7f8228115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 shutdown_connections 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 >> 172.21.15.107:0/1131979075 conn(0x7f82280f2510 msgr2=0x7f82281066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 shutdown_connections 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 -- 172.21.15.107:0/1131979075 wait complete. 2024-02-10T08:34:20.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:20.311+0000 7f822e802700 1 librados: shutdown 2024-02-10T08:34:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:34:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:34:20.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:20.310627+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1131979075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:20.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:20.310627+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1131979075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:21.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T08:34:21.019 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:20.310627+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1131979075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:34:21.282 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:21.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:21.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:19.998616+0000 mgr.y (mgr.14553) 39 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-02-10T08:34:21.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.410237+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.d scrub starts 2024-02-10T08:34:21.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.413268+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2024-02-10T08:34:21.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.648836+0000 osd.5 (osd.5) 5 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:34:21.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.651933+0000 osd.5 (osd.5) 6 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.773147+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.b scrub starts 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:20.776044+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.b scrub ok 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:19.998616+0000 mgr.y (mgr.14553) 39 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.410237+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.d scrub starts 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.413268+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.648836+0000 osd.5 (osd.5) 5 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.651933+0000 osd.5 (osd.5) 6 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.773147+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.b scrub starts 2024-02-10T08:34:21.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:20.776044+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.b scrub ok 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:19.998616+0000 mgr.y (mgr.14553) 39 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.410237+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.d scrub starts 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.413268+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.648836+0000 osd.5 (osd.5) 5 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.651933+0000 osd.5 (osd.5) 6 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.773147+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.b scrub starts 2024-02-10T08:34:22.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:20.776044+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.b scrub ok 2024-02-10T08:34:22.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:21.019729+0000 mgr.y (mgr.14553) 40 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:22.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.122309+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.c scrub starts 2024-02-10T08:34:22.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.123543+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.c scrub ok 2024-02-10T08:34:22.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.658327+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:34:22.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.661210+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:34:22.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.730104+0000 osd.0 (osd.0) 9 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:34:22.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.730932+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:21.019729+0000 mgr.y (mgr.14553) 40 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.122309+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.c scrub starts 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.123543+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.c scrub ok 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.658327+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.661210+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.730104+0000 osd.0 (osd.0) 9 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:34:22.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.730932+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:21.019729+0000 mgr.y (mgr.14553) 40 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.122309+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.c scrub starts 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.123543+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.c scrub ok 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.658327+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.661210+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.730104+0000 osd.0 (osd.0) 9 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:34:22.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.730932+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.104+0000 7f9230a1c700 1 -- 172.21.15.107:0/3452091774 >> v1:172.21.15.107:6789/0 conn(0x7f922c103250 legacy=0x7f922c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 -- 172.21.15.107:0/3452091774 shutdown_connections 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 -- 172.21.15.107:0/3452091774 >> 172.21.15.107:0/3452091774 conn(0x7f922c0f2510 msgr2=0x7f922c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 -- 172.21.15.107:0/3452091774 shutdown_connections 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 -- 172.21.15.107:0/3452091774 wait complete. 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 librados: starting msgr at 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 librados: starting objecter 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 Processor -- start 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 -- start start 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 librados: setting wanted keys 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.105+0000 7f9230a1c700 1 librados: calling monclient init 2024-02-10T08:34:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f922c116170 con 0x7f922c0ff710 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f922c117350 con 0x7f922c0fc910 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f922c118550 con 0x7f922c103250 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9229d9b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f922c0ff710 0x7f922c0f8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46494/0 (socket says 172.21.15.107:46494) 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9229d9b700 1 -- 172.21.15.107:0/1730901875 learned_addr learned my addr 172.21.15.107:0/1730901875 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f922ad9d700 1 --1- 172.21.15.107:0/1730901875 >> v1:172.21.15.107:6790/0 conn(0x7f922c103250 0x7f922c1148d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54698/0 (socket says 172.21.15.107:54698) 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter ms_handle_connect 0x7f922c103250 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter ms_handle_connect 0x7f922c0ff710 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter ms_handle_connect 0x7f922c0fc910 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895279784 0 0) 0x7f922c118550 con 0x7f922c103250 2024-02-10T08:34:23.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92140038e0 con 0x7f922c103250 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094607690 0 0) 0x7f922c117350 con 0x7f922c0fc910 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922c118550 con 0x7f922c0fc910 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456537998 0 0) 0x7f922c116170 con 0x7f922c0ff710 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922c117350 con 0x7f922c0ff710 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930431059 0 0) 0x7f92140038e0 con 0x7f922c103250 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f922c116170 con 0x7f922c103250 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9224001360 con 0x7f922c103250 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126484334 0 0) 0x7f922c117350 con 0x7f922c0ff710 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92140038e0 con 0x7f922c0ff710 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1643512008 0 0) 0x7f922c118550 con 0x7f922c0fc910 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f922c117350 con 0x7f922c0fc910 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9220002850 con 0x7f922c0ff710 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f921c001620 con 0x7f922c0fc910 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3451173408 0 0) 0x7f922c116170 con 0x7f922c103250 2024-02-10T08:34:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 >> v1:172.21.15.195:6789/0 conn(0x7f922c0fc910 legacy=0x7f922c0f8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 >> v1:172.21.15.107:6789/0 conn(0x7f922c0ff710 legacy=0x7f922c0f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c119750 con 0x7f922c103250 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c116360 con 0x7f922c103250 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 10 client.34565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c116850 con 0x7f922c103250 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.106+0000 7f9230a1c700 1 librados: init done 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9224002ab0 con 0x7f922c103250 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9224003ba0 con 0x7f922c103250 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f9230a1c700 10 client.34565.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f9230a1c700 15 client.34565.objecter _session_command_op_assign -1 1 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f9230a1c700 20 client.34565.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f9230a1c700 10 client.34565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:23.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.107+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f920c0028b0 con 0x7f922c103250 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9224002610 con 0x7f922c103250 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (unknown 1684438359 0 0) 0x7f9224078c50 con 0x7f922c103250 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 10 client.34565.objecter ms_dispatch 0x7f922c001550 osd_map(72..72 src has 1..72) v4 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 3 client.34565.objecter handle_osd_map got epochs [72,72] > 0 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 3 client.34565.objecter handle_osd_map decoding full epoch 72 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 10 client.34565.objecter checking command 1 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter _get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 15 client.34565.objecter _session_command_op_remove -1 1 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter _get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 15 client.34565.objecter _session_command_op_assign 0 1 2024-02-10T08:34:23.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 4 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 10 client.34565.objecter _send_command 1 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9214074220 con 0x7f921406e300 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 20 client.34565.objecter dump_active .. 0 homeless 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f9224078fd0 con 0x7f922c103250 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 10 client.34565.objecter ms_handle_connect 0x7f9214063cc0 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.108+0000 7f921b7fe700 10 client.34565.objecter ms_handle_connect 0x7f921406e300 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.109+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9214074220 con 0x7f921406e300 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.109+0000 7f921b7fe700 10 client.34565.objecter ms_dispatch 0x7f922c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.109+0000 7f921b7fe700 10 client.34565.objecter _finish_command 1 = system:0 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.109+0000 7f921b7fe700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 4 2024-02-10T08:34:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.109+0000 7f921b7fe700 15 client.34565.objecter _session_command_op_remove 0 1 2024-02-10T08:34:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 client.34565.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 15 client.34565.objecter _session_command_op_assign -1 2 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter _get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter _get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 15 client.34565.objecter _session_command_op_remove -1 2 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter get_session s=0x7f9224001360 osd=0 3 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 15 client.34565.objecter _session_command_op_assign 0 2 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 4 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 client.34565.objecter _send_command 2 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f920c001140 con 0x7f921406e300 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f921b7fe700 1 -- 172.21.15.107:0/1730901875 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1002017476) 0x7f920c001140 con 0x7f921406e300 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f921b7fe700 10 client.34565.objecter ms_dispatch 0x7f922c001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f921b7fe700 10 client.34565.objecter _finish_command 2 = system:0 2024-02-10T08:34:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f921b7fe700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 4 2024-02-10T08:34:23.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f921b7fe700 15 client.34565.objecter _session_command_op_remove 0 2 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 librados: watch_flush enter 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 librados: watch_flush exit 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 client.34565.objecter close_session for osd.0 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 >> v1:172.21.15.107:6801/3321596972 conn(0x7f921406e300 legacy=0x7f92140707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter put_session s=0x7f9224001360 osd=0 1 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 20 client.34565.objecter shutdown clearing up homeless session... 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 10 client.34565.objecter successfully canceled tick 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9214063cc0 legacy=0x7f9214066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 >> v1:172.21.15.107:6790/0 conn(0x7f922c103250 legacy=0x7f922c1148d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.123+0000 7f922ad9d700 1 -- 172.21.15.107:0/1730901875 reap_dead start 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.124+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 shutdown_connections 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.124+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 >> 172.21.15.107:0/1730901875 conn(0x7f922c0f2510 msgr2=0x7f922c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.124+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 shutdown_connections 2024-02-10T08:34:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.124+0000 7f9230a1c700 1 -- 172.21.15.107:0/1730901875 wait complete. 2024-02-10T08:34:23.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:23.124+0000 7f9230a1c700 1 librados: shutdown 2024-02-10T08:34:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:21.999232+0000 mgr.y (mgr.14553) 41 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:22.583955+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:34:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:22.584851+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:34:23.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:21.999232+0000 mgr.y (mgr.14553) 41 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:23.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:22.583955+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:34:23.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:22.584851+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:34:23.814 INFO:teuthology.orchestra.run.smithi107.stdout:34359738418 2024-02-10T08:34:23.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T08:34:23.893 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:21.999232+0000 mgr.y (mgr.14553) 41 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:23.893 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:22.583955+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:34:23.893 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:22.584851+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:34:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:24.145 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 systemd[1]: Starting Ceph grafana.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:34:24.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.543114+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:34:24.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.543979+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.593307+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.594132+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.673041+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.674399+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.543114+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.543979+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.593307+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.594132+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.673041+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:34:24.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.674399+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:34:24.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.543114+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:34:24.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.543979+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:34:24.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.593307+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:34:24.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.594132+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:34:24.881 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.673041+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:34:24.881 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.674399+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T08:34:24.883 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="App mode production" logger=settings 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-10T08:34:24.884 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-10T08:34:24.885 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-10T08:34:24.887 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-10T08:34:24.888 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-10T08:34:24.889 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-10T08:34:24.890 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-10T08:34:24.891 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-10T08:34:24.892 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-10T08:34:24.893 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-10T08:34:24.894 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-10T08:34:24.895 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-10T08:34:24.896 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-10T08:34:24.897 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 bash[93657]: a2b53e1c16da903acd9a171515700c22e39d6845023af0d9dadee6e3f05d3c70 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T08:34:24.898 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 systemd[1]: Started Ceph grafana.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-10T08:34:24.899 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T08:34:24.900 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T08:34:24.901 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-10T08:34:24.902 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-10T08:34:24.903 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T08:34:24.905 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-10T08:34:24.906 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-10T08:34:24.907 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-10T08:34:24.908 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T08:34:24.909 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T08:34:24.910 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-10T08:34:24.911 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T08:34:24.912 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-10T08:34:24.913 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=164.171433ms 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-10T08:34:24.914 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-10T08:34:24.915 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-10T08:34:25.287 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-10T08:34:25.287 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-10T08:34:25.288 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:34:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:34:24+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T08:34:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 -- 172.21.15.107:0/3949650281 >> v1:172.21.15.107:6790/0 conn(0x7fdae80fc910 legacy=0x7fdae80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 -- 172.21.15.107:0/3949650281 shutdown_connections 2024-02-10T08:34:25.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 -- 172.21.15.107:0/3949650281 >> 172.21.15.107:0/3949650281 conn(0x7fdae80f2510 msgr2=0x7fdae80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 -- 172.21.15.107:0/3949650281 shutdown_connections 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 -- 172.21.15.107:0/3949650281 wait complete. 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 librados: starting msgr at 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 librados: starting objecter 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.717+0000 7fdaf068a700 1 Processor -- start 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 -- start start 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 librados: setting wanted keys 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 librados: calling monclient init 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdae8116270 con 0x7fdae80fc910 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdae8117450 con 0x7fdae80ff710 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaf068a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdae8118650 con 0x7fdae8103250 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaee426700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdae80fc910 0x7fdae80f88f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46514/0 (socket says 172.21.15.107:46514) 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaee426700 1 -- 172.21.15.107:0/1715042060 learned_addr learned my addr 172.21.15.107:0/1715042060 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdaeec27700 1 --1- 172.21.15.107:0/1715042060 >> v1:172.21.15.107:6790/0 conn(0x7fdae8103250 0x7fdae8114930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54708/0 (socket says 172.21.15.107:54708) 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter ms_handle_connect 0x7fdae80fc910 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter ms_handle_connect 0x7fdae8103250 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter ms_handle_connect 0x7fdae80ff710 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501744096 0 0) 0x7fdae8116270 con 0x7fdae80fc910 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad40033f0 con 0x7fdae80fc910 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889565456 0 0) 0x7fdae8117450 con 0x7fdae80ff710 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8116270 con 0x7fdae80ff710 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034301397 0 0) 0x7fdae8118650 con 0x7fdae8103250 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8117450 con 0x7fdae8103250 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823042865 0 0) 0x7fdae8116270 con 0x7fdae80ff710 2024-02-10T08:34:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdae8118650 con 0x7fdae80ff710 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663319515 0 0) 0x7fdae8117450 con 0x7fdae8103250 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdae8116270 con 0x7fdae8103250 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.718+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1029739899 0 0) 0x7fdad40033f0 con 0x7fdae80fc910 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdae8117450 con 0x7fdae80fc910 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdad80018c0 con 0x7fdae80ff710 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdae0001bc0 con 0x7fdae8103250 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdae4000fe0 con 0x7fdae80fc910 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1684560342 0 0) 0x7fdae8118650 con 0x7fdae80ff710 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 >> v1:172.21.15.107:6790/0 conn(0x7fdae8103250 legacy=0x7fdae8114930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 >> v1:172.21.15.107:6789/0 conn(0x7fdae80fc910 legacy=0x7fdae80f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae8119850 con 0x7fdae80ff710 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae8118840 con 0x7fdae80ff710 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdaf068a700 10 client.24556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae8118d30 con 0x7fdae80ff710 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdaf068a700 1 librados: init done 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fdad8002c70 con 0x7fdae80ff710 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.719+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdad8003310 con 0x7fdae80ff710 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdaf068a700 10 client.24556.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdaf068a700 15 client.24556.objecter _session_command_op_assign -1 1 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdaf068a700 20 client.24556.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdaf068a700 10 client.24556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.195:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdacc0028b0 con 0x7fdae80ff710 2024-02-10T08:34:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.720+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fdad8001dd0 con 0x7fdae80ff710 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6056+0+0 (unknown 425281739 0 0) 0x7fdad8075c40 con 0x7fdae80ff710 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 10 client.24556.objecter ms_dispatch 0x7fdae8001550 osd_map(73..73 src has 1..73) v4 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 3 client.24556.objecter handle_osd_map got epochs [73,73] > 0 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 3 client.24556.objecter handle_osd_map decoding full epoch 73 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 10 client.24556.objecter checking command 1 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter _get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 15 client.24556.objecter _session_command_op_remove -1 1 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter _get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 15 client.24556.objecter _session_command_op_assign 1 1 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 4 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 10 client.24556.objecter _send_command 1 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6805/133516126 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdad40720c0 con 0x7fdad406e3b0 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 20 client.24556.objecter dump_active .. 0 homeless 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (unknown 3489808268 0 0) 0x7fdad8076070 con 0x7fdae80ff710 2024-02-10T08:34:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 10 client.24556.objecter ms_handle_connect 0x7fdad4063cf0 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.721+0000 7fdadf7fe700 10 client.24556.objecter ms_handle_connect 0x7fdad406e3b0 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.722+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdad40720c0 con 0x7fdad406e3b0 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.722+0000 7fdadf7fe700 10 client.24556.objecter ms_dispatch 0x7fdae8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.722+0000 7fdadf7fe700 10 client.24556.objecter _finish_command 1 = system:0 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.722+0000 7fdadf7fe700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 4 2024-02-10T08:34:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.722+0000 7fdadf7fe700 15 client.24556.objecter _session_command_op_remove 1 1 2024-02-10T08:34:25.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 10 client.24556.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:25.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 15 client.24556.objecter _session_command_op_assign -1 2 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 20 client.24556.objecter _get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 20 client.24556.objecter _get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 15 client.24556.objecter _session_command_op_remove -1 2 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 20 client.24556.objecter get_session s=0x7fdae4000fe0 osd=1 3 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 15 client.24556.objecter _session_command_op_assign 1 2 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 4 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 10 client.24556.objecter _send_command 2 2024-02-10T08:34:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 --> v1:172.21.15.107:6805/133516126 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdacc001140 con 0x7fdad406e3b0 2024-02-10T08:34:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdadf7fe700 1 -- 172.21.15.107:0/1715042060 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 4200525082) 0x7fdacc001140 con 0x7fdad406e3b0 2024-02-10T08:34:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdadf7fe700 10 client.24556.objecter ms_dispatch 0x7fdae8001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdadf7fe700 10 client.24556.objecter _finish_command 2 = system:0 2024-02-10T08:34:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdadf7fe700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 4 2024-02-10T08:34:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.738+0000 7fdadf7fe700 15 client.24556.objecter _session_command_op_remove 1 2 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 10 librados: watch_flush enter 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 10 librados: watch_flush exit 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 10 client.24556.objecter close_session for osd.1 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 >> v1:172.21.15.107:6805/133516126 conn(0x7fdad406e3b0 legacy=0x7fdad4070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 20 client.24556.objecter put_session s=0x7fdae4000fe0 osd=1 1 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 20 client.24556.objecter shutdown clearing up homeless session... 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 10 client.24556.objecter successfully canceled tick 2024-02-10T08:34:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 >> v1:172.21.15.107:6800/2634842009 conn(0x7fdad4063cf0 legacy=0x7fdad40661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 >> v1:172.21.15.195:6789/0 conn(0x7fdae80ff710 legacy=0x7fdae80f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaeec27700 1 -- 172.21.15.107:0/1715042060 reap_dead start 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 shutdown_connections 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 >> 172.21.15.107:0/1715042060 conn(0x7fdae80f2510 msgr2=0x7fdae81066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 shutdown_connections 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 -- 172.21.15.107:0/1715042060 wait complete. 2024-02-10T08:34:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:25.739+0000 7fdaf068a700 1 librados: shutdown 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:23.999608+0000 mgr.y (mgr.14553) 42 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.114149+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.117902+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.358855+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.358931+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.359436+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.a scrub starts 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.362306+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.a scrub ok 2024-02-10T08:34:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.603558+0000 osd.3 (osd.3) 9 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:34:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.604344+0000 osd.3 (osd.3) 10 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:34:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:24.619451+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T08:34:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:24.813771+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:24.815199+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:25.177022+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:25] "GET /metrics HTTP/1.1" 200 172109 "" "Prometheus/2.33.4" 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:23.999608+0000 mgr.y (mgr.14553) 42 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.114149+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.117902+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.358855+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.358931+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.359436+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.a scrub starts 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.362306+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.a scrub ok 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.603558+0000 osd.3 (osd.3) 9 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.604344+0000 osd.3 (osd.3) 10 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:24.619451+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:24.813771+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:24.815199+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:25.177022+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:23.999608+0000 mgr.y (mgr.14553) 42 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.114149+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.117902+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.358855+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.358931+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.359436+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.a scrub starts 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.362306+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.a scrub ok 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.603558+0000 osd.3 (osd.3) 9 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.604344+0000 osd.3 (osd.3) 10 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:34:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:24.619451+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-10T08:34:25.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:24.813771+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:25.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:24.815199+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:25.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:25.177022+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:26.413 INFO:teuthology.orchestra.run.smithi107.stdout:55834574892 2024-02-10T08:34:26.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T08:34:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:25.081854+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-10T08:34:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:25.085659+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-10T08:34:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:25.704940+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:34:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:25.707844+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:34:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:25.081854+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-10T08:34:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:25.085659+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-10T08:34:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:25.704940+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:34:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:25.707844+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:34:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:25.081854+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-10T08:34:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:25.085659+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-10T08:34:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:25.704940+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:34:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:25.707844+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:34:27.100 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:26.000152+0000 mgr.y (mgr.14553) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:34:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:26.522593+0000 osd.2 (osd.2) 7 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:34:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:26.526198+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:34:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:26.672447+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:34:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:26.673256+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:34:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:26.000152+0000 mgr.y (mgr.14553) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:34:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:26.522593+0000 osd.2 (osd.2) 7 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:34:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:26.526198+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:34:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:26.672447+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:34:27.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:26.673256+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:34:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:26.000152+0000 mgr.y (mgr.14553) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:34:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:26.522593+0000 osd.2 (osd.2) 7 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:34:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:26.526198+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:34:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:26.672447+0000 osd.7 (osd.7) 11 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:34:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:26.673256+0000 osd.7 (osd.7) 12 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:34:28.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:27.274783+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:34:28.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:27.278384+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:34:28.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:27.583584+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:34:28.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:27.584364+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:34:28.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:27.274783+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:34:28.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:27.278384+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:34:28.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:27.583584+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:34:28.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:27.584364+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:34:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:27.274783+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:34:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:27.278384+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:34:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:27.583584+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:34:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:27.584364+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 -- 172.21.15.107:0/2498421739 >> v1:172.21.15.107:6789/0 conn(0x7f74641007b0 legacy=0x7f7464102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 -- 172.21.15.107:0/2498421739 shutdown_connections 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 -- 172.21.15.107:0/2498421739 >> 172.21.15.107:0/2498421739 conn(0x7f74640f35d0 msgr2=0x7f74640f59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 -- 172.21.15.107:0/2498421739 shutdown_connections 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 -- 172.21.15.107:0/2498421739 wait complete. 2024-02-10T08:34:29.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 librados: starting msgr at 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 librados: starting objecter 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.109+0000 7f746c2ea700 1 Processor -- start 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 -- start start 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 librados: setting wanted keys 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 librados: calling monclient init 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7464117380 con 0x7f74641007b0 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7464118580 con 0x7f74641042f0 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746c2ea700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7464119780 con 0x7f74640fd9b0 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746a086700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f74640fd9b0 0x7f74640f9950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54726/0 (socket says 172.21.15.107:54726) 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f746a086700 1 -- 172.21.15.107:0/67275598 learned_addr learned my addr 172.21.15.107:0/67275598 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter ms_handle_connect 0x7f74640fd9b0 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:29.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter ms_handle_connect 0x7f74641007b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter ms_handle_connect 0x7f74641042f0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735976165 0 0) 0x7f7464119780 con 0x7f74640fd9b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74500033f0 con 0x7f74640fd9b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806589271 0 0) 0x7f7464117380 con 0x7f74641007b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7464119780 con 0x7f74641007b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169582449 0 0) 0x7f7464118580 con 0x7f74641042f0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7464117380 con 0x7f74641042f0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319580479 0 0) 0x7f74500033f0 con 0x7f74640fd9b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7464118580 con 0x7f74640fd9b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7460001940 con 0x7f74640fd9b0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094918264 0 0) 0x7f7464117380 con 0x7f74641042f0 2024-02-10T08:34:29.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74500033f0 con 0x7f74641042f0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062405543 0 0) 0x7f7464119780 con 0x7f74641007b0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7464117380 con 0x7f74641007b0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.110+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f745c001c40 con 0x7f74641042f0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7454000fe0 con 0x7f74641007b0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 706082986 0 0) 0x7f7464118580 con 0x7f74640fd9b0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 >> v1:172.21.15.195:6789/0 conn(0x7f74641042f0 legacy=0x7f7464115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 >> v1:172.21.15.107:6789/0 conn(0x7f74641007b0 legacy=0x7f74640fa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746411a980 con 0x7f74640fd9b0 2024-02-10T08:34:29.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7464117570 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 10 client.34577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74641179e0 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 1 librados: init done 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7460002330 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7460005a30 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 10 client.34577.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 15 client.34577.objecter _session_command_op_assign -1 1 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 20 client.34577.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 10 client.34577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.111+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74480028b0 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.112+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7460001e90 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6056+0+0 (unknown 425281739 0 0) 0x7f7460078440 con 0x7f74640fd9b0 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 10 client.34577.objecter ms_dispatch 0x7f7464001550 osd_map(73..73 src has 1..73) v4 2024-02-10T08:34:29.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 3 client.34577.objecter handle_osd_map got epochs [73,73] > 0 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 3 client.34577.objecter handle_osd_map decoding full epoch 73 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 10 client.34577.objecter checking command 1 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter _get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 15 client.34577.objecter _session_command_op_remove -1 1 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter _get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 15 client.34577.objecter _session_command_op_assign 2 1 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 4 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 10 client.34577.objecter _send_command 1 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6809/3575881728 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7450071e40 con 0x7f745006e1d0 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 20 client.34577.objecter dump_active .. 0 homeless 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (unknown 3489808268 0 0) 0x7f74600021c0 con 0x7f74640fd9b0 2024-02-10T08:34:29.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 10 client.34577.objecter ms_handle_connect 0x7f7450063c90 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.113+0000 7f745b779700 10 client.34577.objecter ms_handle_connect 0x7f745006e1d0 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.114+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7450071e40 con 0x7f745006e1d0 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.114+0000 7f745b779700 10 client.34577.objecter ms_dispatch 0x7f7464001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.114+0000 7f745b779700 10 client.34577.objecter _finish_command 1 = system:0 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.114+0000 7f745b779700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 4 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.114+0000 7f745b779700 15 client.34577.objecter _session_command_op_remove 2 1 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 client.34577.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 15 client.34577.objecter _session_command_op_assign -1 2 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter _get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter _get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 15 client.34577.objecter _session_command_op_remove -1 2 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter get_session s=0x7f7454000fe0 osd=2 3 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 15 client.34577.objecter _session_command_op_assign 2 2 2024-02-10T08:34:29.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 4 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 client.34577.objecter _send_command 2 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 --> v1:172.21.15.107:6809/3575881728 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7448001140 con 0x7f745006e1d0 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f745b779700 1 -- 172.21.15.107:0/67275598 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3472381573) 0x7f7448001140 con 0x7f745006e1d0 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f745b779700 10 client.34577.objecter ms_dispatch 0x7f7464001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f745b779700 10 client.34577.objecter _finish_command 2 = system:0 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f745b779700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 4 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f745b779700 15 client.34577.objecter _session_command_op_remove 2 2 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 librados: watch_flush enter 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 librados: watch_flush exit 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 client.34577.objecter close_session for osd.2 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 >> v1:172.21.15.107:6809/3575881728 conn(0x7f745006e1d0 legacy=0x7f7450070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter put_session s=0x7f7454000fe0 osd=2 1 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 20 client.34577.objecter shutdown clearing up homeless session... 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 10 client.34577.objecter successfully canceled tick 2024-02-10T08:34:29.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7450063c90 legacy=0x7f7450066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 >> v1:172.21.15.107:6790/0 conn(0x7f74640fd9b0 legacy=0x7f74640f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746a887700 1 -- 172.21.15.107:0/67275598 reap_dead start 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 shutdown_connections 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 >> 172.21.15.107:0/67275598 conn(0x7f74640f35d0 msgr2=0x7f7464107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 shutdown_connections 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 -- 172.21.15.107:0/67275598 wait complete. 2024-02-10T08:34:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:29.128+0000 7f746c2ea700 1 librados: shutdown 2024-02-10T08:34:29.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.000460+0000 mgr.y (mgr.14553) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.303482+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.306992+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.587473+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.590358+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.597203+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.598974+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.604398+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.605161+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.882356+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:28.882441+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:28.999468+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.545554+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-10T08:34:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.552637+0000 osd.0 (osd.0) 15 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.553906+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.000460+0000 mgr.y (mgr.14553) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.303482+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.306992+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.587473+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.590358+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.597203+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.598974+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.604398+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.605161+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.882356+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:29.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:28.882441+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:29.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:28.999468+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:29.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.545554+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-10T08:34:29.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.552637+0000 osd.0 (osd.0) 15 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:34:29.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.553906+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.000460+0000 mgr.y (mgr.14553) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.303482+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.306992+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.587473+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.590358+0000 osd.5 (osd.5) 10 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.597203+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.598974+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.604398+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.605161+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.882356+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:28.882441+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:28.999468+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:30.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.545554+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-10T08:34:30.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.552637+0000 osd.0 (osd.0) 15 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:34:30.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.553906+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:34:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:34:30.598 INFO:teuthology.orchestra.run.smithi107.stdout:77309411369 2024-02-10T08:34:30.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.399874+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.402870+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.581588+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.584524+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.777676+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:29.780640+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.399874+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.402870+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:34:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.581588+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:34:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.584524+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:34:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.777676+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:34:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:29.780640+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.399874+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.402870+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.581588+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.584524+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.777676+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:34:31.031 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:29.780640+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:34:31.071 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:31.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:30.000965+0000 mgr.y (mgr.14553) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:31.047206+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:31.050735+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:31.051355+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:31.058273+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.543262+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.544044+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:30.000965+0000 mgr.y (mgr.14553) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:31.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:31.047206+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:31.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:31.050735+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:31.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:31.051355+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:31.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:31.058273+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:31.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.543262+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:34:31.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.544044+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:34:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:30.000965+0000 mgr.y (mgr.14553) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:31.047206+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:31.050735+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:31.051355+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:31.058273+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.543262+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:34:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.544044+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:31.030415+0000 mgr.y (mgr.14553) 46 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:34:31.061752+0000 mgr.y (mgr.14553) 47 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:34:31.065528+0000 mgr.y (mgr.14553) 48 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi107 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.358542+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.359542+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.662267+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:34:32.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:31.663878+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:32.204791+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:32.208310+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:31.030415+0000 mgr.y (mgr.14553) 46 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:34:31.061752+0000 mgr.y (mgr.14553) 47 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:34:31.065528+0000 mgr.y (mgr.14553) 48 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi107 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.358542+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.359542+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.662267+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:31.663878+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:32.204791+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:34:32.876 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:32.208310+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:31.030415+0000 mgr.y (mgr.14553) 46 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:34:31.061752+0000 mgr.y (mgr.14553) 47 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:34:31.065528+0000 mgr.y (mgr.14553) 48 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi107 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.358542+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.359542+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.662267+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:31.663878+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:34:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:32.204791+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:34:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:32.208310+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:34:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:32.001625+0000 mgr.y (mgr.14553) 49 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:32.619054+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:34:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:32.620081+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:34:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:33.227732+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:34:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:33.231402+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:32.001625+0000 mgr.y (mgr.14553) 49 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:32.619054+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:32.620081+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:33.227732+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:33.231402+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:34:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:32.001625+0000 mgr.y (mgr.14553) 49 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:32.619054+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:34:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:32.620081+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:34:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:33.227732+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:34:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:33.231402+0000 osd.1 (osd.1) 20 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 -- 172.21.15.107:0/3480667179 >> v1:172.21.15.107:6789/0 conn(0x7f070c0f9570 legacy=0x7f070c102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 -- 172.21.15.107:0/3480667179 shutdown_connections 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 -- 172.21.15.107:0/3480667179 >> 172.21.15.107:0/3480667179 conn(0x7f070c0f2510 msgr2=0x7f070c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 -- 172.21.15.107:0/3480667179 shutdown_connections 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 -- 172.21.15.107:0/3480667179 wait complete. 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 librados: starting msgr at 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 librados: starting objecter 2024-02-10T08:34:34.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.482+0000 7f0711152700 1 Processor -- start 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 -- start start 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 librados: setting wanted keys 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 librados: calling monclient init 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f070c1174c0 con 0x7f070c0f9570 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f070c1186a0 con 0x7f070c0f89c0 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f0711152700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f070c1198a0 con 0x7f070c104350 2024-02-10T08:34:34.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f070a59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f070c0f9570 0x7f070c0fff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36802/0 (socket says 172.21.15.107:36802) 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f070a59c700 1 -- 172.21.15.107:0/423333056 learned_addr learned my addr 172.21.15.107:0/423333056 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f070b59e700 1 --1- 172.21.15.107:0/423333056 >> v1:172.21.15.107:6790/0 conn(0x7f070c104350 0x7f070c115b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35786/0 (socket says 172.21.15.107:35786) 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter ms_handle_connect 0x7f070c0f9570 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter ms_handle_connect 0x7f070c104350 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter ms_handle_connect 0x7f070c0f89c0 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984754643 0 0) 0x7f070c1174c0 con 0x7f070c0f9570 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f4003030 con 0x7f070c0f9570 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140033988 0 0) 0x7f070c1198a0 con 0x7f070c104350 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c1174c0 con 0x7f070c104350 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990100182 0 0) 0x7f070c1186a0 con 0x7f070c0f89c0 2024-02-10T08:34:34.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c1198a0 con 0x7f070c0f89c0 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.483+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2285297175 0 0) 0x7f06f4003030 con 0x7f070c0f9570 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c1186a0 con 0x7f070c0f9570 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f06fc001fa0 con 0x7f070c0f9570 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695909904 0 0) 0x7f070c1174c0 con 0x7f070c104350 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f4003030 con 0x7f070c104350 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125547519 0 0) 0x7f070c1198a0 con 0x7f070c0f89c0 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c1174c0 con 0x7f070c0f89c0 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0704001ce0 con 0x7f070c104350 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f07000017c0 con 0x7f070c0f89c0 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 703491546 0 0) 0x7f070c1186a0 con 0x7f070c0f9570 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 >> v1:172.21.15.107:6790/0 conn(0x7f070c104350 legacy=0x7f070c115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 >> v1:172.21.15.195:6789/0 conn(0x7f070c0f89c0 legacy=0x7f070c0ff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c11aaa0 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c119a90 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f0711152700 10 client.14664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c119f00 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f0711152700 1 librados: init done 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f06fc0025e0 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.484+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f06fc003770 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f0711152700 10 client.14664.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f0711152700 15 client.14664.objecter _session_command_op_assign -1 1 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f0711152700 20 client.14664.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f0711152700 10 client.14664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f06f80028b0 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f06fc003030 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (unknown 2652378735 0 0) 0x7f06fc078860 con 0x7f070c0f9570 2024-02-10T08:34:34.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f06f3fff700 10 client.14664.objecter ms_dispatch 0x7f070c001550 osd_map(74..74 src has 1..74) v4 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f06f3fff700 3 client.14664.objecter handle_osd_map got epochs [74,74] > 0 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.485+0000 7f06f3fff700 3 client.14664.objecter handle_osd_map decoding full epoch 74 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter checking command 1 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter _get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 15 client.14664.objecter _session_command_op_remove -1 1 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter _get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 15 client.14664.objecter _session_command_op_assign 3 1 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 4 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter _send_command 1 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6813/2871825691 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06f4072140 con 0x7f06f406e4d0 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter dump_active .. 0 homeless 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (unknown 3287965567 0 0) 0x7f06fc078c90 con 0x7f070c0f9570 2024-02-10T08:34:34.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter ms_handle_connect 0x7f06f4063d80 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter ms_handle_connect 0x7f06f406e4d0 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f06f4072140 con 0x7f06f406e4d0 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter ms_dispatch 0x7f070c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 10 client.14664.objecter _finish_command 1 = system:0 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 4 2024-02-10T08:34:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.486+0000 7f06f3fff700 15 client.14664.objecter _session_command_op_remove 3 1 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 10 client.14664.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 15 client.14664.objecter _session_command_op_assign -1 2 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 20 client.14664.objecter _get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 20 client.14664.objecter _get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 15 client.14664.objecter _session_command_op_remove -1 2 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 20 client.14664.objecter get_session s=0x7f06fc001fa0 osd=3 3 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 15 client.14664.objecter _session_command_op_assign 3 2 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 4 2024-02-10T08:34:34.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 10 client.14664.objecter _send_command 2 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.500+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 --> v1:172.21.15.107:6813/2871825691 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f06f8001140 con 0x7f06f406e4d0 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f06f3fff700 1 -- 172.21.15.107:0/423333056 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1579616305) 0x7f06f8001140 con 0x7f06f406e4d0 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f06f3fff700 10 client.14664.objecter ms_dispatch 0x7f070c001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f06f3fff700 10 client.14664.objecter _finish_command 2 = system:0 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f06f3fff700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 4 2024-02-10T08:34:34.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f06f3fff700 15 client.14664.objecter _session_command_op_remove 3 2 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 10 librados: watch_flush enter 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 10 librados: watch_flush exit 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 10 client.14664.objecter close_session for osd.3 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 >> v1:172.21.15.107:6813/2871825691 conn(0x7f06f406e4d0 legacy=0x7f06f4070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 20 client.14664.objecter put_session s=0x7f06fc001fa0 osd=3 1 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 20 client.14664.objecter shutdown clearing up homeless session... 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 10 client.14664.objecter successfully canceled tick 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 >> v1:172.21.15.107:6800/2634842009 conn(0x7f06f4063d80 legacy=0x7f06f4066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 >> v1:172.21.15.107:6789/0 conn(0x7f070c0f9570 legacy=0x7f070c0fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:34.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f070b59e700 1 -- 172.21.15.107:0/423333056 reap_dead start 2024-02-10T08:34:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 shutdown_connections 2024-02-10T08:34:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 >> 172.21.15.107:0/423333056 conn(0x7f070c0f2510 msgr2=0x7f070c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 shutdown_connections 2024-02-10T08:34:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 -- 172.21.15.107:0/423333056 wait complete. 2024-02-10T08:34:34.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:34.501+0000 7f0711152700 1 librados: shutdown 2024-02-10T08:34:34.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:33.599416+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:34:34.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:33.601197+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:34:34.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.240009+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:34.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.240053+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:34.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:33.599416+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:34:34.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:33.601197+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:34:34.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.240009+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:34.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.240053+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:33.599416+0000 osd.3 (osd.3) 19 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:34:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:33.601197+0000 osd.3 (osd.3) 20 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:34:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.240009+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.240053+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:35.336 INFO:teuthology.orchestra.run.smithi107.stdout:107374182438 2024-02-10T08:34:35.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T08:34:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.001950+0000 mgr.y (mgr.14553) 50 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.590537+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.b scrub starts 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.592069+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.b scrub ok 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:34.663020+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:35] "GET /metrics HTTP/1.1" 200 172109 "" "Prometheus/2.33.4" 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.001950+0000 mgr.y (mgr.14553) 50 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.590537+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.b scrub starts 2024-02-10T08:34:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.592069+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.b scrub ok 2024-02-10T08:34:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:34.663020+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-10T08:34:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.001950+0000 mgr.y (mgr.14553) 50 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:34:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.590537+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.b scrub starts 2024-02-10T08:34:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.592069+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.b scrub ok 2024-02-10T08:34:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:34.663020+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-10T08:34:36.246 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:36.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:36 smithi107 systemd[1]: Stopping Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:34:36.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:36 smithi107 bash[100983]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T08:34:36.841 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[97185]: level=info ts=2024-02-10T08:34:36.682Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T08:34:36.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.198471+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.e scrub starts 2024-02-10T08:34:36.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.201367+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.e scrub ok 2024-02-10T08:34:36.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.419024+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f deep-scrub starts 2024-02-10T08:34:36.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.421858+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f deep-scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.634111+0000 osd.0 (osd.0) 21 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.635093+0000 osd.0 (osd.0) 22 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.660843+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.662514+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.816643+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:35.819773+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.198471+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.e scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.201367+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.e scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.419024+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f deep-scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.421858+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f deep-scrub ok 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.634111+0000 osd.0 (osd.0) 21 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:34:36.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.635093+0000 osd.0 (osd.0) 22 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.660843+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.662514+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.816643+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:35.819773+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.198471+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.e scrub starts 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.201367+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.e scrub ok 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.419024+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.1f deep-scrub starts 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.421858+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.1f deep-scrub ok 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.634111+0000 osd.0 (osd.0) 21 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:34:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.635093+0000 osd.0 (osd.0) 22 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:34:37.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.660843+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:34:37.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.662514+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:34:37.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.816643+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:34:37.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:35.819773+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:34:37.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:37 smithi107 bash[101076]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T08:34:37.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.002494+0000 mgr.y (mgr.14553) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.153796+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.155800+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.297757+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.300765+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.448060+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.449938+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.647786+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:36.649181+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.002494+0000 mgr.y (mgr.14553) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.153796+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.155800+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:34:37.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.297757+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:34:37.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.300765+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:34:37.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.448060+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:34:37.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.449938+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:34:37.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.647786+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:34:37.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:36.649181+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.002494+0000 mgr.y (mgr.14553) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.153796+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.155800+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.297757+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.300765+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.448060+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:34:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.449938+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:34:38.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.647786+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:34:38.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:36.649181+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:34:38.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 -- 172.21.15.107:0/1070326187 >> v1:172.21.15.107:6790/0 conn(0x7fa5140ff710 legacy=0x7fa514101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 -- 172.21.15.107:0/1070326187 shutdown_connections 2024-02-10T08:34:38.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 -- 172.21.15.107:0/1070326187 >> 172.21.15.107:0/1070326187 conn(0x7fa5140f2510 msgr2=0x7fa5140f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 -- 172.21.15.107:0/1070326187 shutdown_connections 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 -- 172.21.15.107:0/1070326187 wait complete. 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 librados: starting msgr at 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.911+0000 7fa51ad25700 1 librados: starting objecter 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 Processor -- start 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 -- start start 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 librados: setting wanted keys 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 librados: calling monclient init 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa514117380 con 0x7fa5140fc910 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa514118560 con 0x7fa5140ff710 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa51ad25700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa514119760 con 0x7fa514103250 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa518ac1700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa5140fc910 0x7fa5140f9b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36816/0 (socket says 172.21.15.107:36816) 2024-02-10T08:34:38.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa518ac1700 1 -- 172.21.15.107:0/2346510576 learned_addr learned my addr 172.21.15.107:0/2346510576 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter ms_handle_connect 0x7fa514103250 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter ms_handle_connect 0x7fa5140fc910 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087751491 0 0) 0x7fa514119760 con 0x7fa514103250 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504003030 con 0x7fa514103250 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498266968 0 0) 0x7fa514117380 con 0x7fa5140fc910 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa514119760 con 0x7fa5140fc910 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter ms_handle_connect 0x7fa5140ff710 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573003837 0 0) 0x7fa514118560 con 0x7fa5140ff710 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa514117380 con 0x7fa5140ff710 2024-02-10T08:34:38.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762381479 0 0) 0x7fa504003030 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa514118560 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.912+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa50c001d70 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3606281782 0 0) 0x7fa514119760 con 0x7fa5140fc910 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa504003030 con 0x7fa5140fc910 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa5080018c0 con 0x7fa5140fc910 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1659523049 0 0) 0x7fa514118560 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 >> v1:172.21.15.195:6789/0 conn(0x7fa5140ff710 legacy=0x7fa5140fa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 >> v1:172.21.15.107:6789/0 conn(0x7fa5140fc910 legacy=0x7fa5140f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51411a960 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa514117570 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa50c0026a0 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa50c003610 con 0x7fa514103250 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa51ad25700 10 client.34589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:38.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa514117a60 con 0x7fa514103250 2024-02-10T08:34:38.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.913+0000 7fa51ad25700 1 librados: init done 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa51ad25700 10 client.34589.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa51ad25700 15 client.34589.objecter _session_command_op_assign -1 1 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa51ad25700 20 client.34589.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa51ad25700 10 client.34589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa51405f7a0 con 0x7fa514103250 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.914+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa50c016580 con 0x7fa514103250 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 6056+0+0 (unknown 3810993139 0 0) 0x7fa50c076120 con 0x7fa514103250 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 10 client.34589.objecter ms_dispatch 0x7fa514001550 osd_map(75..75 src has 1..75) v4 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 3 client.34589.objecter handle_osd_map got epochs [75,75] > 0 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 3 client.34589.objecter handle_osd_map decoding full epoch 75 2024-02-10T08:34:38.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 10 client.34589.objecter checking command 1 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter _get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 15 client.34589.objecter _session_command_op_remove -1 1 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter _get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 15 client.34589.objecter _session_command_op_assign 4 1 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 4 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 10 client.34589.objecter _send_command 1 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.195:6800/2499671285 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5040764a0 con 0x7fa504072790 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 20 client.34589.objecter dump_active .. 0 homeless 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (unknown 838226561 0 0) 0x7fa50c076550 con 0x7fa514103250 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.915+0000 7fa511ffb700 10 client.34589.objecter ms_handle_connect 0x7fa504068100 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 10 client.34589.objecter ms_handle_connect 0x7fa504072790 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa5040764a0 con 0x7fa504072790 2024-02-10T08:34:38.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 10 client.34589.objecter ms_dispatch 0x7fa514001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:38.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 10 client.34589.objecter _finish_command 1 = system:0 2024-02-10T08:34:38.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 4 2024-02-10T08:34:38.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.916+0000 7fa511ffb700 15 client.34589.objecter _session_command_op_remove 4 1 2024-02-10T08:34:38.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:38 smithi107 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-10T08:34:38.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:38 smithi107 systemd[1]: Stopped Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:34:38.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:38 smithi107 systemd[1]: Starting Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:34:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:37.329044+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:34:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:37.332136+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:34:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:37.615320+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:34:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:37.616425+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:34:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:37.329044+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:34:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:37.332136+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:34:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:37.615320+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:34:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:37.616425+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:34:38.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 10 client.34589.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 15 client.34589.objecter _session_command_op_assign -1 2 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 20 client.34589.objecter _get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 20 client.34589.objecter _get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 15 client.34589.objecter _session_command_op_remove -1 2 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 20 client.34589.objecter get_session s=0x7fa5040720e0 osd=4 3 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 15 client.34589.objecter _session_command_op_assign 4 2 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 4 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 10 client.34589.objecter _send_command 2 2024-02-10T08:34:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.932+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 --> v1:172.21.15.195:6800/2499671285 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5140475e0 con 0x7fa504072790 2024-02-10T08:34:38.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa511ffb700 1 -- 172.21.15.107:0/2346510576 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3860663012) 0x7fa5140475e0 con 0x7fa504072790 2024-02-10T08:34:38.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa511ffb700 10 client.34589.objecter ms_dispatch 0x7fa514001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:38.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa511ffb700 10 client.34589.objecter _finish_command 2 = system:0 2024-02-10T08:34:38.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa511ffb700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 4 2024-02-10T08:34:38.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa511ffb700 15 client.34589.objecter _session_command_op_remove 4 2 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 10 librados: watch_flush enter 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 10 librados: watch_flush exit 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 10 client.34589.objecter close_session for osd.4 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 >> v1:172.21.15.195:6800/2499671285 conn(0x7fa504072790 legacy=0x7fa504074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 20 client.34589.objecter put_session s=0x7fa5040720e0 osd=4 1 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 20 client.34589.objecter shutdown clearing up homeless session... 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 10 client.34589.objecter successfully canceled tick 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa504068100 legacy=0x7fa50406a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 >> v1:172.21.15.107:6790/0 conn(0x7fa514103250 legacy=0x7fa514115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa5192c2700 1 -- 172.21.15.107:0/2346510576 reap_dead start 2024-02-10T08:34:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 shutdown_connections 2024-02-10T08:34:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 >> 172.21.15.107:0/2346510576 conn(0x7fa5140f2510 msgr2=0x7fa5140612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.933+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 shutdown_connections 2024-02-10T08:34:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.934+0000 7fa51ad25700 1 -- 172.21.15.107:0/2346510576 wait complete. 2024-02-10T08:34:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:38.934+0000 7fa51ad25700 1 librados: shutdown 2024-02-10T08:34:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:37.329044+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:34:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:37.332136+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:34:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:37.615320+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:34:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:37.616425+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:34:39.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.856Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-10T08:34:39.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.856Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.857Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.858Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.883Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.884Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.886Z caller=main.go:518 msg=Listening address=:9093 2024-02-10T08:34:39.931 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:39.886Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-10T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:38.002875+0000 mgr.y (mgr.14553) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:38.117351+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:38.118340+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:38.625641+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:38.628556+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.548744+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:38.002875+0000 mgr.y (mgr.14553) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:38.117351+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:38.118340+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:38.625641+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:38.628556+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:34:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.548744+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:38.002875+0000 mgr.y (mgr.14553) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:38.117351+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:38.118340+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:38.625641+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:38.628556+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:34:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.548744+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-10T08:34:40.259 INFO:teuthology.orchestra.run.smithi107.stdout:133143986209 2024-02-10T08:34:40.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T08:34:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:34:40.546 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPING 2024-02-10T08:34:40.546 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:34:40.546 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPED 2024-02-10T08:34:40.547 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTING 2024-02-10T08:34:40.547 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Serving on http://:::9283 2024-02-10T08:34:40.547 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:40 smithi107 bash[101290]: 11b1a7937650ccd42dc15ce3c0818eb8adf369244027a4de911fd1a9eec4dac6 2024-02-10T08:34:40.547 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:40 smithi107 systemd[1]: Started Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:34:40.551 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.354711+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.357763+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.376297+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.378306+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.589729+0000 osd.2 (osd.2) 9 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.590599+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.701406+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:34:40.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:39.703639+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.008271+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.008501+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.008607+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.008710+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.008891+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.009099+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.009424+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.009684+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.406973+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.354711+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.357763+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.376297+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.378306+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.589729+0000 osd.2 (osd.2) 9 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.590599+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.701406+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:39.703639+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.008271+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.008501+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.008607+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.008710+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.008891+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.009099+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:40.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.009424+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.009684+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.406973+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.412628+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.413811+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.418095+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.424763+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.428591+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.433106+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.439933+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.442927+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:40.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.447614+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.454553+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.412628+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.413811+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.418095+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.424763+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.428591+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.433106+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.439933+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.442927+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.447614+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:40.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.454553+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTED 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPING 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPED 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTING 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Serving on http://:::9283 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTED 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPING 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STOPPED 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTING 2024-02-10T08:34:40.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Serving on http://:::9283 2024-02-10T08:34:40.825 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:34:40] ENGINE Bus STARTED 2024-02-10T08:34:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.354711+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:34:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.357763+0000 osd.1 (osd.1) 28 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:34:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.376297+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:34:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.378306+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:34:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.589729+0000 osd.2 (osd.2) 9 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.590599+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.701406+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:39.703639+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.008271+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.008501+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.008607+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.008710+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.008891+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.009099+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.009424+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T08:34:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.009684+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.406973+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.412628+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.413811+0000 mon.c (mon.2) 61 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.418095+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.424763+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.428591+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.433106+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.439933+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.442927+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.447614+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:41.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.454553+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:41.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:41.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:40.003420+0000 mgr.y (mgr.14553) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:41.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.413049+0000 mgr.y (mgr.14553) 54 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:41.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.414025+0000 mgr.y (mgr.14553) 55 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:41.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.425184+0000 mgr.y (mgr.14553) 56 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:41.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.429056+0000 mgr.y (mgr.14553) 57 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.440319+0000 mgr.y (mgr.14553) 58 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.443236+0000 mgr.y (mgr.14553) 59 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.690122+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.690207+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.690276+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:40.690338+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:40.690396+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:40.808174+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:40.808973+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:40.003420+0000 mgr.y (mgr.14553) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:41.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.413049+0000 mgr.y (mgr.14553) 54 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.414025+0000 mgr.y (mgr.14553) 55 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.425184+0000 mgr.y (mgr.14553) 56 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.429056+0000 mgr.y (mgr.14553) 57 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.440319+0000 mgr.y (mgr.14553) 58 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.443236+0000 mgr.y (mgr.14553) 59 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.690122+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.690207+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.690276+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:40.690338+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:40.690396+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:40.808174+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:34:41.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:40.808973+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:40.003420+0000 mgr.y (mgr.14553) 53 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.413049+0000 mgr.y (mgr.14553) 54 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.414025+0000 mgr.y (mgr.14553) 55 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.425184+0000 mgr.y (mgr.14553) 56 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.429056+0000 mgr.y (mgr.14553) 57 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T08:34:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.440319+0000 mgr.y (mgr.14553) 58 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.443236+0000 mgr.y (mgr.14553) 59 : audit [DBG] from='mon.? v1:172.21.15.107:0/269256191' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.690122+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.690207+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.690276+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:40.690338+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:40.690396+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:40.808174+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:34:42.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:40.808973+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:34:42.131 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:41.858Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000103354s 2024-02-10T08:34:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.437+0000 7f624de91700 1 -- 172.21.15.107:0/3033637626 >> v1:172.21.15.107:6789/0 conn(0x7f6248104330 legacy=0x7f62481067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.437+0000 7f624de91700 1 -- 172.21.15.107:0/3033637626 shutdown_connections 2024-02-10T08:34:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.437+0000 7f624de91700 1 -- 172.21.15.107:0/3033637626 >> 172.21.15.107:0/3033637626 conn(0x7f62480f3640 msgr2=0x7f62480f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.437+0000 7f624de91700 1 -- 172.21.15.107:0/3033637626 shutdown_connections 2024-02-10T08:34:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.437+0000 7f624de91700 1 -- 172.21.15.107:0/3033637626 wait complete. 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 librados: starting msgr at 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 librados: starting objecter 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 Processor -- start 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 -- start start 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 librados: setting wanted keys 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 librados: calling monclient init 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f624806cc50 con 0x7f62480fda50 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f624806cde0 con 0x7f6248104330 2024-02-10T08:34:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f624de91700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f624806cfd0 con 0x7f62481007f0 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f62477fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62480fda50 0x7f62480fc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35274/0 (socket says 172.21.15.107:35274) 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.438+0000 7f62477fe700 1 -- 172.21.15.107:0/3756897762 learned_addr learned my addr 172.21.15.107:0/3756897762 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter ms_handle_connect 0x7f62481007f0 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter ms_handle_connect 0x7f62480fda50 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter ms_handle_connect 0x7f6248104330 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980790900 0 0) 0x7f624806cfd0 con 0x7f62481007f0 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6230003800 con 0x7f62481007f0 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305095369 0 0) 0x7f624806cc50 con 0x7f62480fda50 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624806cfd0 con 0x7f62480fda50 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134957317 0 0) 0x7f624806cde0 con 0x7f6248104330 2024-02-10T08:34:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624806cc50 con 0x7f6248104330 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143342190 0 0) 0x7f6230003800 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f624806cde0 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f623c0018c0 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696716601 0 0) 0x7f624806cc50 con 0x7f6248104330 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6230003800 con 0x7f6248104330 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6240002fe0 con 0x7f6248104330 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2722609095 0 0) 0x7f624806cde0 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 >> v1:172.21.15.195:6789/0 conn(0x7f6248104330 legacy=0x7f62481184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 >> v1:172.21.15.107:6789/0 conn(0x7f62480fda50 legacy=0x7f62480fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624811cc00 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6248119c30 con 0x7f62481007f0 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f624de91700 10 client.34595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:42.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f624811a0a0 con 0x7f62481007f0 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f624de91700 1 librados: init done 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f623c001ec0 con 0x7f62481007f0 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.439+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f623c003280 con 0x7f62481007f0 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f624de91700 10 client.34595.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f624de91700 15 client.34595.objecter _session_command_op_assign -1 1 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f624de91700 20 client.34595.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:42.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f624de91700 10 client.34595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62340028b0 con 0x7f62481007f0 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.440+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f623c001a20 con 0x7f62481007f0 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1788720618 0 0) 0x7f623c075d40 con 0x7f62481007f0 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 10 client.34595.objecter ms_dispatch 0x7f6248001550 osd_map(78..78 src has 1..78) v4 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 3 client.34595.objecter handle_osd_map got epochs [78,78] > 0 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 3 client.34595.objecter handle_osd_map decoding full epoch 78 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 10 client.34595.objecter checking command 1 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter _get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 15 client.34595.objecter _session_command_op_remove -1 1 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter _get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 15 client.34595.objecter _session_command_op_assign 5 1 2024-02-10T08:34:42.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 4 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 10 client.34595.objecter _send_command 1 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.195:6804/4178321486 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6230071ff0 con 0x7f623006e2e0 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 20 client.34595.objecter dump_active .. 0 homeless 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7f623c076110 con 0x7f62481007f0 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 10 client.34595.objecter ms_handle_connect 0x7f6230063a30 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.441+0000 7f6244ff9700 10 client.34595.objecter ms_handle_connect 0x7f623006e2e0 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.442+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6230071ff0 con 0x7f623006e2e0 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.442+0000 7f6244ff9700 10 client.34595.objecter ms_dispatch 0x7f6248001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.442+0000 7f6244ff9700 10 client.34595.objecter _finish_command 1 = system:0 2024-02-10T08:34:42.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.442+0000 7f6244ff9700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 4 2024-02-10T08:34:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.442+0000 7f6244ff9700 15 client.34595.objecter _session_command_op_remove 5 1 2024-02-10T08:34:42.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 10 client.34595.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 15 client.34595.objecter _session_command_op_assign -1 2 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 20 client.34595.objecter _get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 20 client.34595.objecter _get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 15 client.34595.objecter _session_command_op_remove -1 2 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 20 client.34595.objecter get_session s=0x7f623006dc30 osd=5 3 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 15 client.34595.objecter _session_command_op_assign 5 2 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 4 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 10 client.34595.objecter _send_command 2 2024-02-10T08:34:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.456+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 --> v1:172.21.15.195:6804/4178321486 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6234001140 con 0x7f623006e2e0 2024-02-10T08:34:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6244ff9700 1 -- 172.21.15.107:0/3756897762 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1699381006) 0x7f6234001140 con 0x7f623006e2e0 2024-02-10T08:34:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6244ff9700 10 client.34595.objecter ms_dispatch 0x7f6248001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6244ff9700 10 client.34595.objecter _finish_command 2 = system:0 2024-02-10T08:34:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6244ff9700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 4 2024-02-10T08:34:42.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6244ff9700 15 client.34595.objecter _session_command_op_remove 5 2 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 10 librados: watch_flush enter 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 10 librados: watch_flush exit 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 10 client.34595.objecter close_session for osd.5 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 >> v1:172.21.15.195:6804/4178321486 conn(0x7f623006e2e0 legacy=0x7f6230070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 20 client.34595.objecter put_session s=0x7f623006dc30 osd=5 1 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 20 client.34595.objecter shutdown clearing up homeless session... 2024-02-10T08:34:42.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 10 client.34595.objecter successfully canceled tick 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6230063a30 legacy=0x7f6230065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 >> v1:172.21.15.107:6790/0 conn(0x7f62481007f0 legacy=0x7f6248114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f6247fff700 1 -- 172.21.15.107:0/3756897762 reap_dead start 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 shutdown_connections 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 >> 172.21.15.107:0/3756897762 conn(0x7f62480f3640 msgr2=0x7f62481077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 shutdown_connections 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 -- 172.21.15.107:0/3756897762 wait complete. 2024-02-10T08:34:42.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:42.457+0000 7f624de91700 1 librados: shutdown 2024-02-10T08:34:42.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:41.040978+0000 mgr.y (mgr.14553) 60 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:42.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:41.645796+0000 osd.0 (osd.0) 25 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:41.648711+0000 osd.0 (osd.0) 26 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:41.689928+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:41.040978+0000 mgr.y (mgr.14553) 60 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:41.645796+0000 osd.0 (osd.0) 25 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:41.648711+0000 osd.0 (osd.0) 26 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:34:42.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:41.689928+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-10T08:34:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:41.040978+0000 mgr.y (mgr.14553) 60 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:41.645796+0000 osd.0 (osd.0) 25 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:34:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:41.648711+0000 osd.0 (osd.0) 26 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:34:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:41.689928+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-10T08:34:43.536 INFO:teuthology.orchestra.run.smithi107.stdout:158913789982 2024-02-10T08:34:43.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T08:34:43.840 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:43.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:42.003825+0000 mgr.y (mgr.14553) 61 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:34:43.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:42.748609+0000 osd.7 (osd.7) 19 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:34:43.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:42.752077+0000 osd.7 (osd.7) 20 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:34:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:42.003825+0000 mgr.y (mgr.14553) 61 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:34:43.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:42.748609+0000 osd.7 (osd.7) 19 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:34:43.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:42.752077+0000 osd.7 (osd.7) 20 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:34:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:42.003825+0000 mgr.y (mgr.14553) 61 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:34:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:42.748609+0000 osd.7 (osd.7) 19 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:34:44.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:42.752077+0000 osd.7 (osd.7) 20 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:43.678710+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:43.680069+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.004421+0000 mgr.y (mgr.14553) 62 : cluster [DBG] pgmap v48: 132 pgs: 1 active+recovering, 129 active+clean, 2 peering; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/576 objects misplaced (0.868%); 41 B/s, 0 objects/s recovering 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.377576+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.a scrub starts 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.381204+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.a scrub ok 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.431573+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.434599+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.809899+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:34:45.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:44.810880+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:45] "GET /metrics HTTP/1.1" 200 172121 "" "Prometheus/2.33.4" 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:43.678710+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:43.680069+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.004421+0000 mgr.y (mgr.14553) 62 : cluster [DBG] pgmap v48: 132 pgs: 1 active+recovering, 129 active+clean, 2 peering; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/576 objects misplaced (0.868%); 41 B/s, 0 objects/s recovering 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.377576+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.a scrub starts 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.381204+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.a scrub ok 2024-02-10T08:34:45.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.431573+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:34:45.875 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.434599+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:34:45.875 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.809899+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:34:45.875 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:44.810880+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:34:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:43.678710+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:34:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:43.680069+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:34:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.004421+0000 mgr.y (mgr.14553) 62 : cluster [DBG] pgmap v48: 132 pgs: 1 active+recovering, 129 active+clean, 2 peering; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/576 objects misplaced (0.868%); 41 B/s, 0 objects/s recovering 2024-02-10T08:34:46.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.377576+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.a scrub starts 2024-02-10T08:34:46.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.381204+0000 osd.6 (osd.6) 20 : cluster [DBG] 4.a scrub ok 2024-02-10T08:34:46.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.431573+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:34:46.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.434599+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:34:46.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.809899+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:34:46.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:44.810880+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.335+0000 7f4c168e1700 1 -- 172.21.15.107:0/1946817272 >> v1:172.21.15.107:6790/0 conn(0x7f4c100ff720 legacy=0x7f4c10101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.335+0000 7f4c168e1700 1 -- 172.21.15.107:0/1946817272 shutdown_connections 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.335+0000 7f4c168e1700 1 -- 172.21.15.107:0/1946817272 >> 172.21.15.107:0/1946817272 conn(0x7f4c100f2540 msgr2=0x7f4c100f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.335+0000 7f4c168e1700 1 -- 172.21.15.107:0/1946817272 shutdown_connections 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.335+0000 7f4c168e1700 1 -- 172.21.15.107:0/1946817272 wait complete. 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 librados: starting msgr at 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 librados: starting objecter 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 Processor -- start 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 -- start start 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 librados: setting wanted keys 2024-02-10T08:34:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 librados: calling monclient init 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4c10116090 con 0x7f4c100fc920 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4c10117270 con 0x7f4c100ff720 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c168e1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4c10118450 con 0x7f4c10103260 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c14e7e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4c10103260 0x7f4c101148e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39442/0 (socket says 172.21.15.107:39442) 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c14e7e700 1 -- 172.21.15.107:0/3812166552 learned_addr learned my addr 172.21.15.107:0/3812166552 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter ms_handle_connect 0x7f4c100fc920 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter ms_handle_connect 0x7f4c10103260 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter ms_handle_connect 0x7f4c100ff720 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124605766 0 0) 0x7f4c10116090 con 0x7f4c100fc920 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.336+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bfc0038e0 con 0x7f4c100fc920 2024-02-10T08:34:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217875102 0 0) 0x7f4c10118450 con 0x7f4c10103260 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10116090 con 0x7f4c10103260 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336805419 0 0) 0x7f4c10117270 con 0x7f4c100ff720 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10118450 con 0x7f4c100ff720 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 182345000 0 0) 0x7f4bfc0038e0 con 0x7f4c100fc920 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c10117270 con 0x7f4c100fc920 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c00001860 con 0x7f4c100fc920 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052176378 0 0) 0x7f4c10116090 con 0x7f4c10103260 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bfc0038e0 con 0x7f4c10103260 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c08001bd0 con 0x7f4c10103260 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120451595 0 0) 0x7f4c10118450 con 0x7f4c100ff720 2024-02-10T08:34:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c10116090 con 0x7f4c100ff720 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4bf80015d0 con 0x7f4c100ff720 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3013213784 0 0) 0x7f4c10117270 con 0x7f4c100fc920 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 >> v1:172.21.15.107:6790/0 conn(0x7f4c10103260 legacy=0x7f4c101148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 >> v1:172.21.15.195:6789/0 conn(0x7f4c100ff720 legacy=0x7f4c100f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10119670 con 0x7f4c100fc920 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c10118640 con 0x7f4c100fc920 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c168e1700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c10118b30 con 0x7f4c100fc920 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.337+0000 7f4c168e1700 1 librados: init done 2024-02-10T08:34:46.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4c00001ff0 con 0x7f4c100fc920 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c168e1700 10 client.14682.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c00003380 con 0x7f4c100fc920 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c168e1700 15 client.14682.objecter _session_command_op_assign -1 1 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c168e1700 20 client.14682.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c168e1700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c100603f0 con 0x7f4c100fc920 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.338+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4c000162f0 con 0x7f4c100fc920 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1788720618 0 0) 0x7f4c00075e90 con 0x7f4c100fc920 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 10 client.14682.objecter ms_dispatch 0x7f4c10001550 osd_map(78..78 src has 1..78) v4 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 3 client.14682.objecter handle_osd_map got epochs [78,78] > 0 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 3 client.14682.objecter handle_osd_map decoding full epoch 78 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 10 client.14682.objecter checking command 1 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter _get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 15 client.14682.objecter _session_command_op_remove -1 1 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter _get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 15 client.14682.objecter _session_command_op_assign 6 1 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 4 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 10 client.14682.objecter _send_command 1 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.195:6808/1417479017 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bfc072370 con 0x7f4bfc06e660 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 20 client.14682.objecter dump_active .. 0 homeless 2024-02-10T08:34:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7f4c000762c0 con 0x7f4c100fc920 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 10 client.14682.objecter ms_handle_connect 0x7f4bfc063d10 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.339+0000 7f4c0d7fa700 10 client.14682.objecter ms_handle_connect 0x7f4bfc06e660 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.340+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bfc072370 con 0x7f4bfc06e660 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.340+0000 7f4c0d7fa700 10 client.14682.objecter ms_dispatch 0x7f4c10001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.340+0000 7f4c0d7fa700 10 client.14682.objecter _finish_command 1 = system:0 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.340+0000 7f4c0d7fa700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 4 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.340+0000 7f4c0d7fa700 15 client.14682.objecter _session_command_op_remove 6 1 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 10 client.14682.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 15 client.14682.objecter _session_command_op_assign -1 2 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 20 client.14682.objecter _get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 20 client.14682.objecter _get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 15 client.14682.objecter _session_command_op_remove -1 2 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 20 client.14682.objecter get_session s=0x7f4bfc06dfb0 osd=6 3 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 15 client.14682.objecter _session_command_op_assign 6 2 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 4 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 10 client.14682.objecter _send_command 2 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.354+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 --> v1:172.21.15.195:6808/1417479017 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c100475e0 con 0x7f4bfc06e660 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c0d7fa700 1 -- 172.21.15.107:0/3812166552 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2307416270) 0x7f4c100475e0 con 0x7f4bfc06e660 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c0d7fa700 10 client.14682.objecter ms_dispatch 0x7f4c10001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c0d7fa700 10 client.14682.objecter _finish_command 2 = system:0 2024-02-10T08:34:46.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c0d7fa700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 4 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c0d7fa700 15 client.14682.objecter _session_command_op_remove 6 2 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 10 librados: watch_flush enter 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 10 librados: watch_flush exit 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 10 client.14682.objecter close_session for osd.6 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 >> v1:172.21.15.195:6808/1417479017 conn(0x7f4bfc06e660 legacy=0x7f4bfc070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 20 client.14682.objecter put_session s=0x7f4bfc06dfb0 osd=6 1 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 20 client.14682.objecter shutdown clearing up homeless session... 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 10 client.14682.objecter successfully canceled tick 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4bfc063d10 legacy=0x7f4bfc0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 >> v1:172.21.15.107:6789/0 conn(0x7f4c100fc920 legacy=0x7f4c100f8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c14e7e700 1 -- 172.21.15.107:0/3812166552 reap_dead start 2024-02-10T08:34:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 shutdown_connections 2024-02-10T08:34:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 >> 172.21.15.107:0/3812166552 conn(0x7f4c100f2540 msgr2=0x7f4c10106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 shutdown_connections 2024-02-10T08:34:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 -- 172.21.15.107:0/3812166552 wait complete. 2024-02-10T08:34:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:46.355+0000 7f4c168e1700 1 librados: shutdown 2024-02-10T08:34:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:45.518203+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.b scrub starts 2024-02-10T08:34:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:45.519109+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.b scrub ok 2024-02-10T08:34:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:45.604357+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:34:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:45.607248+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:34:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:45.518203+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.b scrub starts 2024-02-10T08:34:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:45.519109+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.b scrub ok 2024-02-10T08:34:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:45.604357+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:34:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:45.607248+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:34:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:45.518203+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.b scrub starts 2024-02-10T08:34:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:45.519109+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.b scrub ok 2024-02-10T08:34:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:45.604357+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:34:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:45.607248+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:34:47.563 INFO:teuthology.orchestra.run.smithi107.stdout:188978561050 2024-02-10T08:34:47.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T08:34:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:46.005080+0000 mgr.y (mgr.14553) 63 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 89 B/s, 2 objects/s recovering 2024-02-10T08:34:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:46.449473+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:34:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:46.452501+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:34:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:47.141702+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:46.005080+0000 mgr.y (mgr.14553) 63 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 89 B/s, 2 objects/s recovering 2024-02-10T08:34:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:46.449473+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:34:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:46.452501+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:34:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:47.141702+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:46.005080+0000 mgr.y (mgr.14553) 63 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 89 B/s, 2 objects/s recovering 2024-02-10T08:34:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:46.449473+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:34:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:46.452501+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:34:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:47.141702+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:48.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.414758+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:34:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.420170+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:34:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.545857+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:34:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.549666+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:34:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.414758+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:34:48.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.420170+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:34:48.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.545857+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:34:48.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.549666+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:34:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.414758+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:34:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.420170+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:34:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.545857+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:34:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.549666+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:34:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 -- 172.21.15.107:0/2028026426 >> v1:172.21.15.107:6789/0 conn(0x7faf28104330 legacy=0x7faf281067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 -- 172.21.15.107:0/2028026426 shutdown_connections 2024-02-10T08:34:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 -- 172.21.15.107:0/2028026426 >> 172.21.15.107:0/2028026426 conn(0x7faf280f3640 msgr2=0x7faf280f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 -- 172.21.15.107:0/2028026426 shutdown_connections 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 -- 172.21.15.107:0/2028026426 wait complete. 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 librados: starting msgr at 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.573+0000 7faf2d0ba700 1 librados: starting objecter 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 Processor -- start 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 -- start start 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 librados: setting wanted keys 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 librados: calling monclient init 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf28117390 con 0x7faf280fda50 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf28118570 con 0x7faf281007f0 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2d0ba700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf28119770 con 0x7faf28104330 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf26d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faf280fda50 0x7faf280f9bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54800/0 (socket says 172.21.15.107:54800) 2024-02-10T08:34:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf26d9d700 1 -- 172.21.15.107:0/1655614867 learned_addr learned my addr 172.21.15.107:0/1655614867 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf2759e700 1 --1- 172.21.15.107:0/1655614867 >> v1:172.21.15.107:6790/0 conn(0x7faf28104330 0x7faf28115a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44004/0 (socket says 172.21.15.107:44004) 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf0ffff700 10 client.?.objecter ms_handle_connect 0x7faf280fda50 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf0ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf0ffff700 10 client.?.objecter ms_handle_connect 0x7faf28104330 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.574+0000 7faf0ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 10 client.?.objecter ms_handle_connect 0x7faf281007f0 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568038185 0 0) 0x7faf28117390 con 0x7faf280fda50 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf100038e0 con 0x7faf280fda50 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750942412 0 0) 0x7faf28119770 con 0x7faf28104330 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf28117390 con 0x7faf28104330 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212415127 0 0) 0x7faf28118570 con 0x7faf281007f0 2024-02-10T08:34:49.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf28119770 con 0x7faf281007f0 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586630740 0 0) 0x7faf100038e0 con 0x7faf280fda50 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf28118570 con 0x7faf280fda50 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf1c001940 con 0x7faf280fda50 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858401203 0 0) 0x7faf28117390 con 0x7faf28104330 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf100038e0 con 0x7faf28104330 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf20002180 con 0x7faf28104330 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040167925 0 0) 0x7faf28119770 con 0x7faf281007f0 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf28117390 con 0x7faf281007f0 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf18001620 con 0x7faf281007f0 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270533461 0 0) 0x7faf28118570 con 0x7faf280fda50 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 >> v1:172.21.15.107:6790/0 conn(0x7faf28104330 legacy=0x7faf28115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 >> v1:172.21.15.195:6789/0 conn(0x7faf281007f0 legacy=0x7faf280fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2811a970 con 0x7faf280fda50 2024-02-10T08:34:49.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf28119960 con 0x7faf280fda50 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf2d0ba700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf28119dd0 con 0x7faf280fda50 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf2d0ba700 1 librados: init done 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7faf1c001cd0 con 0x7faf280fda50 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.575+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf1c005ca0 con 0x7faf280fda50 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf2d0ba700 10 client.14688.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf2d0ba700 15 client.14688.objecter _session_command_op_assign -1 1 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf2d0ba700 20 client.14688.objecter _get_session osd=-1 returning homeless 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf2d0ba700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf140028b0 con 0x7faf280fda50 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.576+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7faf1c002c60 con 0x7faf280fda50 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1788720618 0 0) 0x7faf1c078610 con 0x7faf280fda50 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 10 client.14688.objecter ms_dispatch 0x7faf28001550 osd_map(78..78 src has 1..78) v4 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 3 client.14688.objecter handle_osd_map got epochs [78,78] > 0 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 3 client.14688.objecter handle_osd_map decoding full epoch 78 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 10 client.14688.objecter checking command 1 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter _get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 15 client.14688.objecter _session_command_op_remove -1 1 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter _get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 15 client.14688.objecter _session_command_op_assign 7 1 2024-02-10T08:34:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 4 2024-02-10T08:34:49.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 10 client.14688.objecter _send_command 1 2024-02-10T08:34:49.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.195:6812/1934176928 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf10072320 con 0x7faf1006e610 2024-02-10T08:34:49.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 20 client.14688.objecter dump_active .. 0 homeless 2024-02-10T08:34:49.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7faf1c0789b0 con 0x7faf280fda50 2024-02-10T08:34:49.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.577+0000 7faf0ffff700 10 client.14688.objecter ms_handle_connect 0x7faf10063dc0 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 10 client.14688.objecter ms_handle_connect 0x7faf1006e610 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faf10072320 con 0x7faf1006e610 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 10 client.14688.objecter ms_dispatch 0x7faf28001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 10 client.14688.objecter _finish_command 1 = system:0 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 4 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.578+0000 7faf0ffff700 15 client.14688.objecter _session_command_op_remove 7 1 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 10 client.14688.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 15 client.14688.objecter _session_command_op_assign -1 2 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 20 client.14688.objecter _get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 20 client.14688.objecter _get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 15 client.14688.objecter _session_command_op_remove -1 2 2024-02-10T08:34:49.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 20 client.14688.objecter get_session s=0x7faf1006df60 osd=7 3 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 15 client.14688.objecter _session_command_op_assign 7 2 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 4 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 10 client.14688.objecter _send_command 2 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 --> v1:172.21.15.195:6812/1934176928 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf14003be0 con 0x7faf1006e610 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf0ffff700 1 -- 172.21.15.107:0/1655614867 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 460026885) 0x7faf14003be0 con 0x7faf1006e610 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf0ffff700 10 client.14688.objecter ms_dispatch 0x7faf28001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf0ffff700 10 client.14688.objecter _finish_command 2 = system:0 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf0ffff700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 4 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf0ffff700 15 client.14688.objecter _session_command_op_remove 7 2 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 10 librados: watch_flush enter 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 10 librados: watch_flush exit 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.593+0000 7faf2d0ba700 10 client.14688.objecter close_session for osd.7 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 >> v1:172.21.15.195:6812/1934176928 conn(0x7faf1006e610 legacy=0x7faf10070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 20 client.14688.objecter put_session s=0x7faf1006df60 osd=7 1 2024-02-10T08:34:49.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 20 client.14688.objecter shutdown clearing up homeless session... 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 10 client.14688.objecter successfully canceled tick 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 >> v1:172.21.15.107:6800/2634842009 conn(0x7faf10063dc0 legacy=0x7faf10066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 >> v1:172.21.15.107:6789/0 conn(0x7faf280fda50 legacy=0x7faf280f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2759e700 1 -- 172.21.15.107:0/1655614867 reap_dead start 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 shutdown_connections 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 >> 172.21.15.107:0/1655614867 conn(0x7faf280f3640 msgr2=0x7faf281077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 shutdown_connections 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 -- 172.21.15.107:0/1655614867 wait complete. 2024-02-10T08:34:49.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:49.594+0000 7faf2d0ba700 1 librados: shutdown 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.691176+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:47.693021+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:48.005518+0000 mgr.y (mgr.14553) 64 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:48.838132+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:48.841022+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.691176+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:34:49.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:47.693021+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:34:49.619 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:48.005518+0000 mgr.y (mgr.14553) 64 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-02-10T08:34:49.619 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:48.838132+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:34:49.619 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:48.841022+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:34:49.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:34:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:34:49.860Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00191998s 2024-02-10T08:34:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.691176+0000 osd.0 (osd.0) 29 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:34:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:47.693021+0000 osd.0 (osd.0) 30 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:34:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:48.005518+0000 mgr.y (mgr.14553) 64 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-02-10T08:34:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:48.838132+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:34:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:48.841022+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:34:50.320 INFO:teuthology.orchestra.run.smithi107.stdout:223338299412 2024-02-10T08:34:50.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T08:34:50.454 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:34:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:34:50.566 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:48.690913+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:34:50.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:48.691729+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:34:50.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.532318+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:34:50.586 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:48.690913+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:34:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:48.691729+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:34:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.532318+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:34:50.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:48.690913+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:34:50.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:48.691729+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:34:50.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.532318+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:34:51.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.005970+0000 mgr.y (mgr.14553) 65 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.092021+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.093909+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.535807+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:50.894337+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:51.168365+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:51.168972+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:51.173056+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:51.176929+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.005970+0000 mgr.y (mgr.14553) 65 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.092021+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.093909+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.535807+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:50.894337+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:51.168365+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:51.168972+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:51.173056+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:51.176929+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.005970+0000 mgr.y (mgr.14553) 65 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.092021+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.093909+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.535807+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:50.894337+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:51.168365+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:51.168972+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:51.173056+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:51.825 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:51.176929+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:34:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.388+0000 7f008daa7700 1 -- 172.21.15.107:0/2014362794 >> v1:172.21.15.107:6790/0 conn(0x7f00880fda50 legacy=0x7f00880fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.388+0000 7f008daa7700 1 -- 172.21.15.107:0/2014362794 shutdown_connections 2024-02-10T08:34:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.388+0000 7f008daa7700 1 -- 172.21.15.107:0/2014362794 >> 172.21.15.107:0/2014362794 conn(0x7f00880f3640 msgr2=0x7f00880f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.388+0000 7f008daa7700 1 -- 172.21.15.107:0/2014362794 shutdown_connections 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.388+0000 7f008daa7700 1 -- 172.21.15.107:0/2014362794 wait complete. 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 librados: starting msgr at 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 librados: starting objecter 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 Processor -- start 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 -- start start 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 librados: setting wanted keys 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.389+0000 7f008daa7700 1 librados: calling monclient init 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f008daa7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0088117350 con 0x7f00880fda50 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f00877fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f00880fda50 0x7f00880f9900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54828/0 (socket says 172.21.15.107:54828) 2024-02-10T08:34:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f00877fe700 1 -- 172.21.15.107:0/3005581259 learned_addr learned my addr 172.21.15.107:0/3005581259 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0088118530 con 0x7f0088104330 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter ms_handle_connect 0x7f00881007f0 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter ms_handle_connect 0x7f00880fda50 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0088119730 con 0x7f00881007f0 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter ms_handle_connect 0x7f0088104330 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215585435 0 0) 0x7f0088117350 con 0x7f00880fda50 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00740033f0 con 0x7f00880fda50 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332779175 0 0) 0x7f0088119730 con 0x7f00881007f0 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088117350 con 0x7f00881007f0 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582517184 0 0) 0x7f0088118530 con 0x7f0088104330 2024-02-10T08:34:52.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088119730 con 0x7f0088104330 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613296321 0 0) 0x7f00740033f0 con 0x7f00880fda50 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0088118530 con 0x7f00880fda50 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f00780021f0 con 0x7f00880fda50 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852493676 0 0) 0x7f0088117350 con 0x7f00881007f0 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00740033f0 con 0x7f00881007f0 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0070001880 con 0x7f00881007f0 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 552230191 0 0) 0x7f0088119730 con 0x7f0088104330 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0088117350 con 0x7f0088104330 2024-02-10T08:34:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0080001920 con 0x7f0088104330 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1889015197 0 0) 0x7f0088118530 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 >> v1:172.21.15.107:6790/0 conn(0x7f00881007f0 legacy=0x7f00880fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 >> v1:172.21.15.195:6789/0 conn(0x7f0088104330 legacy=0x7f0088115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008811a930 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0088119920 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.390+0000 7f008daa7700 10 client.14694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.391+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0088119d90 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.391+0000 7f008daa7700 1 librados: init done 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.391+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0078002390 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.391+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0078003930 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.391+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0068004f20 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0078001880 con 0x7f00880fda50 2024-02-10T08:34:52.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1788720618 0 0) 0x7f0078078870 con 0x7f00880fda50 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 10 client.14694.objecter ms_dispatch 0x7f0088001550 osd_map(78..78 src has 1..78) v4 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 3 client.14694.objecter handle_osd_map got epochs [78,78] > 0 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 3 client.14694.objecter handle_osd_map decoding full epoch 78 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 20 client.14694.objecter dump_active .. 0 homeless 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.392+0000 7f0084f74700 10 client.14694.objecter ms_handle_connect 0x7f0074063da0 2024-02-10T08:34:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.396+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f007804c480 con 0x7f00880fda50 2024-02-10T08:34:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.567+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f00680051d0 con 0x7f00880fda50 2024-02-10T08:34:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.567+0000 7f0084f74700 1 -- 172.21.15.107:0/3005581259 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 739478307) 0x7f007804a8f0 con 0x7f00880fda50 2024-02-10T08:34:52.568 INFO:teuthology.orchestra.run.smithi107.stdout:34359738423 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 10 librados: watch_flush enter 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 10 librados: watch_flush exit 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 20 client.14694.objecter shutdown clearing up homeless session... 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 10 client.14694.objecter successfully canceled tick 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0074063da0 legacy=0x7f0074066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 >> v1:172.21.15.107:6789/0 conn(0x7f00880fda50 legacy=0x7f00880f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 shutdown_connections 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 >> 172.21.15.107:0/3005581259 conn(0x7f00880f3640 msgr2=0x7f00881077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 shutdown_connections 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 -- 172.21.15.107:0/3005581259 wait complete. 2024-02-10T08:34:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:52.569+0000 7f008daa7700 1 librados: shutdown 2024-02-10T08:34:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.744610+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.a scrub starts 2024-02-10T08:34:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:50.747420+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.a scrub ok 2024-02-10T08:34:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:51.050943+0000 mgr.y (mgr.14553) 66 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.345336+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:34:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.348328+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.495841+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.497236+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.909331+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.910174+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.488527+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.491417+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.744610+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.a scrub starts 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:50.747420+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.a scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:51.050943+0000 mgr.y (mgr.14553) 66 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.345336+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.348328+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:34:52.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.495841+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:34:52.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.497236+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:34:52.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.909331+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:34:52.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.910174+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:34:52.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.488527+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:34:52.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.491417+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.744610+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.a scrub starts 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:50.747420+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.a scrub ok 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:51.050943+0000 mgr.y (mgr.14553) 66 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.345336+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.348328+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.495841+0000 osd.3 (osd.3) 29 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.497236+0000 osd.3 (osd.3) 30 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:34:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.909331+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:34:53.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.910174+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:34:53.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.488527+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:34:53.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.491417+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:34:53.703 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-02-10T08:34:53.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.680121+0000 osd.0 (osd.0) 33 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.680982+0000 osd.0 (osd.0) 34 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.733904+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.d scrub starts 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:51.737386+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.d scrub ok 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.006768+0000 mgr.y (mgr.14553) 67 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 396 B/s wr, 8 op/s; 56 B/s, 1 objects/s recovering 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.282927+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.285818+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.515510+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:34:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.516228+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:34:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:52.568129+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.107:0/3005581259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:34:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.697214+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:34:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:52.698323+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.680121+0000 osd.0 (osd.0) 33 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.680982+0000 osd.0 (osd.0) 34 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.733904+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.d scrub starts 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:51.737386+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.d scrub ok 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.006768+0000 mgr.y (mgr.14553) 67 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 396 B/s wr, 8 op/s; 56 B/s, 1 objects/s recovering 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.282927+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:34:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.285818+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.515510+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.516228+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:52.568129+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.107:0/3005581259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.697214+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:52.698323+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.680121+0000 osd.0 (osd.0) 33 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.680982+0000 osd.0 (osd.0) 34 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.733904+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.d scrub starts 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:51.737386+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.d scrub ok 2024-02-10T08:34:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.006768+0000 mgr.y (mgr.14553) 67 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 396 B/s wr, 8 op/s; 56 B/s, 1 objects/s recovering 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.282927+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.285818+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.515510+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.516228+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:52.568129+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.107:0/3005581259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.697214+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:34:53.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:52.698323+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:34:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:54.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.295333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:34:54.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.298223+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:34:54.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.485373+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:34:54.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.487125+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:34:54.798 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.295333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:34:54.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.298223+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:34:54.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.485373+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:34:54.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.487125+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:34:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.295333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:34:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.298223+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:34:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.485373+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:34:55.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.487125+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.740939+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.f scrub starts 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:53.743803+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.f scrub ok 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:54.007075+0000 mgr.y (mgr.14553) 68 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s; 48 B/s, 1 objects/s recovering 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:54.036018+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 deep-scrub starts 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:54.039580+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 deep-scrub ok 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:55.256964+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.440497+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:34:55.892 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.443387+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:34:55] "GET /metrics HTTP/1.1" 200 172145 "" "Prometheus/2.33.4" 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.740939+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.f scrub starts 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:53.743803+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.f scrub ok 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:54.007075+0000 mgr.y (mgr.14553) 68 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s; 48 B/s, 1 objects/s recovering 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:54.036018+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 deep-scrub starts 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:54.039580+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 deep-scrub ok 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:55.256964+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.440497+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:34:55.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.443387+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.740939+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.f scrub starts 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:53.743803+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.f scrub ok 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:54.007075+0000 mgr.y (mgr.14553) 68 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s; 48 B/s, 1 objects/s recovering 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:54.036018+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 deep-scrub starts 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:54.039580+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 deep-scrub ok 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:55.256964+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.440497+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:34:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.443387+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:34:56.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:54.702805+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.c scrub starts 2024-02-10T08:34:56.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:54.705697+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.c scrub ok 2024-02-10T08:34:56.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.283609+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:34:56.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.286514+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.665376+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.b scrub starts 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.668326+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.b scrub ok 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.704226+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:55.704259+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.472650+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-02-10T08:34:56.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.475525+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-02-10T08:34:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.652+0000 7f480f72d700 1 -- 172.21.15.107:0/2660682962 >> v1:172.21.15.107:6790/0 conn(0x7f4808104330 legacy=0x7f48081067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- 172.21.15.107:0/2660682962 shutdown_connections 2024-02-10T08:34:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- 172.21.15.107:0/2660682962 >> 172.21.15.107:0/2660682962 conn(0x7f48080f3640 msgr2=0x7f48080f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- 172.21.15.107:0/2660682962 shutdown_connections 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- 172.21.15.107:0/2660682962 wait complete. 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 librados: starting msgr at 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 librados: starting objecter 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 Processor -- start 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- start start 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 librados: setting wanted keys 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 librados: calling monclient init 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48081195f0 con 0x7f48081007f0 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f480811a7d0 con 0x7f48080fda50 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480f72d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f480811b9b0 con 0x7f4808104330 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480ccc8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f48081007f0 0x7f4808102c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54852/0 (socket says 172.21.15.107:54852) 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.653+0000 7f480ccc8700 1 -- 172.21.15.107:0/931853297 learned_addr learned my addr 172.21.15.107:0/931853297 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter ms_handle_connect 0x7f48081007f0 2024-02-10T08:34:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter ms_handle_connect 0x7f4808104330 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter ms_handle_connect 0x7f48080fda50 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18757411 0 0) 0x7f48081195f0 con 0x7f48081007f0 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f40038e0 con 0x7f48081007f0 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990253423 0 0) 0x7f480811b9b0 con 0x7f4808104330 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48081195f0 con 0x7f4808104330 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699803606 0 0) 0x7f480811a7d0 con 0x7f48080fda50 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480811b9b0 con 0x7f48080fda50 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687811163 0 0) 0x7f47f40038e0 con 0x7f48081007f0 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f480811a7d0 con 0x7f48081007f0 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47f8001900 con 0x7f48081007f0 2024-02-10T08:34:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1100600497 0 0) 0x7f48081195f0 con 0x7f4808104330 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f40038e0 con 0x7f4808104330 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4800001b40 con 0x7f4808104330 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270906942 0 0) 0x7f480811b9b0 con 0x7f48080fda50 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48081195f0 con 0x7f48080fda50 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f48040019f0 con 0x7f48080fda50 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1694936578 0 0) 0x7f480811a7d0 con 0x7f48081007f0 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 >> v1:172.21.15.107:6790/0 conn(0x7f4808104330 legacy=0x7f4808117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 >> v1:172.21.15.195:6789/0 conn(0x7f48080fda50 legacy=0x7f4808102550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480811cbb0 con 0x7f48081007f0 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480811bba0 con 0x7f48081007f0 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f480f72d700 10 client.14700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f480811c090 con 0x7f48081007f0 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f480f72d700 1 librados: init done 2024-02-10T08:34:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f47f8002420 con 0x7f48081007f0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.654+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47f8005ae0 con 0x7f48081007f0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.655+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec004f20 con 0x7f48081007f0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f47f8001f80 con 0x7f48081007f0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(79..79 src has 1..79) v4 ==== 6172+0+0 (unknown 2108253250 0 0) 0x7f47f8078530 con 0x7f48081007f0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 10 client.14700.objecter ms_dispatch 0x7f4808001550 osd_map(79..79 src has 1..79) v4 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 3 client.14700.objecter handle_osd_map got epochs [79,79] > 0 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 3 client.14700.objecter handle_osd_map decoding full epoch 79 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 20 client.14700.objecter dump_active .. 0 homeless 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.656+0000 7f47fe7fc700 10 client.14700.objecter ms_handle_connect 0x7f47f4063c20 2024-02-10T08:34:56.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.660+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f47f804c080 con 0x7f48081007f0 2024-02-10T08:34:56.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.830+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f47ec0051d0 con 0x7f48081007f0 2024-02-10T08:34:56.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.830+0000 7f47fe7fc700 1 -- 172.21.15.107:0/931853297 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 3109548707) 0x7f47f804a490 con 0x7f48081007f0 2024-02-10T08:34:56.831 INFO:teuthology.orchestra.run.smithi107.stdout:55834574899 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 10 librados: watch_flush enter 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 10 librados: watch_flush exit 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 20 client.14700.objecter shutdown clearing up homeless session... 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 10 client.14700.objecter successfully canceled tick 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 >> v1:172.21.15.107:6800/2634842009 conn(0x7f47f4063c20 legacy=0x7f47f40660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 >> v1:172.21.15.107:6789/0 conn(0x7f48081007f0 legacy=0x7f4808102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 shutdown_connections 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 >> 172.21.15.107:0/931853297 conn(0x7f48080f3640 msgr2=0x7f48081077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 shutdown_connections 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 -- 172.21.15.107:0/931853297 wait complete. 2024-02-10T08:34:56.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:56.832+0000 7f480f72d700 1 librados: shutdown 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:54.702805+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.c scrub starts 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:54.705697+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.c scrub ok 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.283609+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.286514+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.665376+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.b scrub starts 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.668326+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.b scrub ok 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.704226+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:55.704259+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.472650+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-02-10T08:34:56.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.475525+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:54.702805+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.c scrub starts 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:54.705697+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.c scrub ok 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.283609+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.286514+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.665376+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.b scrub starts 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.668326+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.b scrub ok 2024-02-10T08:34:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.704226+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:34:57.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:55.704259+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:34:57.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.472650+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-02-10T08:34:57.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.475525+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-02-10T08:34:57.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574899 for osd.1 2024-02-10T08:34:57.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T08:34:57.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.007850+0000 mgr.y (mgr.14553) 69 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:34:57.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.468369+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.469623+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.614771+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.698443+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.699413+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:56.831342+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.107:0/931853297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:56.949959+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:56.951073+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:56.951656+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.955427+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:34:56.956292+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:56.956474+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.007850+0000 mgr.y (mgr.14553) 69 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.468369+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.469623+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.614771+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.698443+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.699413+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:56.831342+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.107:0/931853297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:56.949959+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:56.951073+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:57.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:56.951656+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:57.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.955427+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:34:57.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:34:56.956292+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:57.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:56.956474+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:34:57.943 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:34:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.007850+0000 mgr.y (mgr.14553) 69 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:34:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.468369+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:34:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.469623+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:34:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.614771+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-10T08:34:58.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.698443+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.699413+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:56.831342+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.107:0/931853297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:56.949959+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:56.951073+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:56.951656+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.955427+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:34:56.956292+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:34:58.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:56.956474+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:34:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.832+0000 7f01cba67700 1 -- 172.21.15.107:0/2921849234 >> v1:172.21.15.107:6790/0 conn(0x7f01c4104330 legacy=0x7f01c41067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.832+0000 7f01cba67700 1 -- 172.21.15.107:0/2921849234 shutdown_connections 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.832+0000 7f01cba67700 1 -- 172.21.15.107:0/2921849234 >> 172.21.15.107:0/2921849234 conn(0x7f01c40f2510 msgr2=0x7f01c40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.832+0000 7f01cba67700 1 -- 172.21.15.107:0/2921849234 shutdown_connections 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.832+0000 7f01cba67700 1 -- 172.21.15.107:0/2921849234 wait complete. 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 librados: starting msgr at 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 librados: starting objecter 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 Processor -- start 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 -- start start 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 librados: setting wanted keys 2024-02-10T08:34:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 librados: calling monclient init 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01c41173b0 con 0x7f01c40f9550 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01c4118550 con 0x7f01c40f8a60 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01cba67700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01c4119750 con 0x7f01c4104330 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01ca004700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f01c4104330 0x7f01c4115a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48608/0 (socket says 172.21.15.107:48608) 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.833+0000 7f01ca004700 1 -- 172.21.15.107:0/3280928105 learned_addr learned my addr 172.21.15.107:0/3280928105 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter ms_handle_connect 0x7f01c4104330 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter ms_handle_connect 0x7f01c40f9550 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter ms_handle_connect 0x7f01c40f8a60 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641275098 0 0) 0x7f01c4119750 con 0x7f01c4104330 2024-02-10T08:34:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b0003800 con 0x7f01c4104330 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274212863 0 0) 0x7f01c4118550 con 0x7f01c40f8a60 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c4119750 con 0x7f01c40f8a60 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449072532 0 0) 0x7f01c41173b0 con 0x7f01c40f9550 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c4118550 con 0x7f01c40f9550 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883866864 0 0) 0x7f01b0003800 con 0x7f01c4104330 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01c41173b0 con 0x7f01c4104330 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f01bc003220 con 0x7f01c4104330 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038600129 0 0) 0x7f01c4119750 con 0x7f01c40f8a60 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01b0003800 con 0x7f01c40f8a60 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f01c00016a0 con 0x7f01c40f8a60 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790624097 0 0) 0x7f01c4118550 con 0x7f01c40f9550 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01c4119750 con 0x7f01c40f9550 2024-02-10T08:34:59.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f01b4001840 con 0x7f01c40f9550 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 571123415 0 0) 0x7f01c41173b0 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 >> v1:172.21.15.195:6789/0 conn(0x7f01c40f8a60 legacy=0x7f01c40ff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 >> v1:172.21.15.107:6789/0 conn(0x7f01c40f9550 legacy=0x7f01c40ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c411a950 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c41175a0 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01cba67700 10 client.34625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c4117a10 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01cba67700 1 librados: init done 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f01bc002350 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.834+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f01bc003ca0 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.835+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a8004f20 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f01bc0034c0 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3609243947 0 0) 0x7f01bc078dc0 con 0x7f01c4104330 2024-02-10T08:34:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 10 client.34625.objecter ms_dispatch 0x7f01c4001550 osd_map(80..80 src has 1..80) v4 2024-02-10T08:34:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 3 client.34625.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:34:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 3 client.34625.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:34:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 20 client.34625.objecter dump_active .. 0 homeless 2024-02-10T08:34:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.836+0000 7f01baf78700 10 client.34625.objecter ms_handle_connect 0x7f01b0063cc0 2024-02-10T08:34:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:34:59.840+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f01bc04c970 con 0x7f01c4104330 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:58.008186+0000 mgr.y (mgr.14553) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:58.058575+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:58.059621+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:58.968695+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:58.971601+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:34:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:59.552643+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:58.008186+0000 mgr.y (mgr.14553) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:58.058575+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:58.059621+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:58.968695+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:58.971601+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:34:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:34:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:59.552643+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-10T08:35:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.004+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f01a80051d0 con 0x7f01c4104330 2024-02-10T08:35:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.004+0000 7f01baf78700 1 -- 172.21.15.107:0/3280928105 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 2868170512) 0x7f01bc04ade0 con 0x7f01c4104330 2024-02-10T08:35:00.005 INFO:teuthology.orchestra.run.smithi107.stdout:77309411375 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 10 librados: watch_flush enter 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 10 librados: watch_flush exit 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 20 client.34625.objecter shutdown clearing up homeless session... 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 10 client.34625.objecter successfully canceled tick 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 >> v1:172.21.15.107:6800/2634842009 conn(0x7f01b0063cc0 legacy=0x7f01b0066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 >> v1:172.21.15.107:6790/0 conn(0x7f01c4104330 legacy=0x7f01c4115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 shutdown_connections 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 >> 172.21.15.107:0/3280928105 conn(0x7f01c40f2510 msgr2=0x7f01c41077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 shutdown_connections 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 -- 172.21.15.107:0/3280928105 wait complete. 2024-02-10T08:35:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:00.006+0000 7f01cba67700 1 librados: shutdown 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:58.008186+0000 mgr.y (mgr.14553) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:58.058575+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:58.059621+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:58.968695+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:58.971601+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:35:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:34:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:59.552643+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-10T08:35:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:00.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411375 for osd.2 2024-02-10T08:35:00.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T08:35:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:59.082475+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:35:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:34:59.084146+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:35:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:00.005553+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.107:0/3280928105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:35:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:59.082475+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:35:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:34:59.084146+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:35:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:00.005553+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.107:0/3280928105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:35:00.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:59.082475+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:35:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:34:59.084146+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:35:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:00.005553+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.107:0/3280928105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:35:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.008536+0000 mgr.y (mgr.14553) 71 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.132923+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:35:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.136385+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:35:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.419001+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.420428+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.638777+0000 osd.0 (osd.0) 35 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.639701+0000 osd.0 (osd.0) 36 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.933103+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:00.936107+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:01.488927+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.1b scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:01.491908+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.1b scrub ok 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.008536+0000 mgr.y (mgr.14553) 71 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.132923+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.136385+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.419001+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:35:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.420428+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.638777+0000 osd.0 (osd.0) 35 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.639701+0000 osd.0 (osd.0) 36 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.933103+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:00.936107+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:01.488927+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.1b scrub starts 2024-02-10T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:01.491908+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.1b scrub ok 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.008536+0000 mgr.y (mgr.14553) 71 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.132923+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.136385+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.419001+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.420428+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.638777+0000 osd.0 (osd.0) 35 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:35:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.639701+0000 osd.0 (osd.0) 36 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:35:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.933103+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-10T08:35:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:00.936107+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-10T08:35:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:01.488927+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.1b scrub starts 2024-02-10T08:35:02.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:01.491908+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.1b scrub ok 2024-02-10T08:35:02.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:01.054245+0000 mgr.y (mgr.14553) 72 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:01.686565+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:01.687382+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.102740+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.105622+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:01.054245+0000 mgr.y (mgr.14553) 72 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:01.686565+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:01.687382+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.102740+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:35:02.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.105622+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:35:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.904+0000 7f7eaf621700 1 -- 172.21.15.107:0/650982841 >> v1:172.21.15.107:6790/0 conn(0x7f7ea80ff720 legacy=0x7f7ea8101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.904+0000 7f7eaf621700 1 -- 172.21.15.107:0/650982841 shutdown_connections 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.904+0000 7f7eaf621700 1 -- 172.21.15.107:0/650982841 >> 172.21.15.107:0/650982841 conn(0x7f7ea80f2540 msgr2=0x7f7ea80f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- 172.21.15.107:0/650982841 shutdown_connections 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- 172.21.15.107:0/650982841 wait complete. 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 librados: starting msgr at 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 librados: starting objecter 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 Processor -- start 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- start start 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 librados: setting wanted keys 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 librados: calling monclient init 2024-02-10T08:35:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ea8117360 con 0x7f7ea8103260 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ea8118500 con 0x7f7ea80fc920 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eaf621700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ea8119700 con 0x7f7ea80ff720 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eacbbc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7ea80ff720 0x7f7ea80fa1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48622/0 (socket says 172.21.15.107:48622) 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eadbbe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7ea8103260 0x7f7ea81159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40922/0 (socket says 172.21.15.107:40922) 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.905+0000 7f7eacbbc700 1 -- 172.21.15.107:0/1206107928 learned_addr learned my addr 172.21.15.107:0/1206107928 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter ms_handle_connect 0x7f7ea8103260 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter ms_handle_connect 0x7f7ea80ff720 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter ms_handle_connect 0x7f7ea80fc920 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837491799 0 0) 0x7f7ea8117360 con 0x7f7ea8103260 2024-02-10T08:35:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e94003030 con 0x7f7ea8103260 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412031898 0 0) 0x7f7ea8119700 con 0x7f7ea80ff720 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8117360 con 0x7f7ea80ff720 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769162965 0 0) 0x7f7ea8118500 con 0x7f7ea80fc920 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8119700 con 0x7f7ea80fc920 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1998901228 0 0) 0x7f7e94003030 con 0x7f7ea8103260 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea8118500 con 0x7f7ea8103260 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7ea0001bc0 con 0x7f7ea8103260 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904538178 0 0) 0x7f7ea8117360 con 0x7f7ea80ff720 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e94003030 con 0x7f7ea80ff720 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7ea4001430 con 0x7f7ea80ff720 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605900504 0 0) 0x7f7ea8119700 con 0x7f7ea80fc920 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea8117360 con 0x7f7ea80fc920 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7e980016a0 con 0x7f7ea80fc920 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 143065181 0 0) 0x7f7ea8118500 con 0x7f7ea8103260 2024-02-10T08:35:02.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 >> v1:172.21.15.107:6790/0 conn(0x7f7ea80ff720 legacy=0x7f7ea80fa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 >> v1:172.21.15.195:6789/0 conn(0x7f7ea80fc920 legacy=0x7f7ea80f9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea811a900 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea81198f0 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7eaf621700 10 client.14712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea8119de0 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7eaf621700 1 librados: init done 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7ea00026a0 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.906+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7ea0005d80 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.907+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c004f20 con 0x7f7ea8103260 2024-02-10T08:35:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.907+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7ea0002200 con 0x7f7ea8103260 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3609243947 0 0) 0x7f7ea0078830 con 0x7f7ea8103260 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 10 client.14712.objecter ms_dispatch 0x7f7ea8001550 osd_map(80..80 src has 1..80) v4 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 3 client.14712.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 3 client.14712.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 20 client.14712.objecter dump_active .. 0 homeless 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.908+0000 7f7e9e7fc700 10 client.14712.objecter ms_handle_connect 0x7f7e94063cc0 2024-02-10T08:35:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:02.912+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7ea004c380 con 0x7f7ea8103260 2024-02-10T08:35:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:01.054245+0000 mgr.y (mgr.14553) 72 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:01.686565+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-10T08:35:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:01.687382+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-10T08:35:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.102740+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:35:03.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.105622+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:35:03.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.074+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7e8c0051d0 con 0x7f7ea8103260 2024-02-10T08:35:03.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.075+0000 7f7e9e7fc700 1 -- 172.21.15.107:0/1206107928 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 240780175) 0x7f7ea004a790 con 0x7f7ea8103260 2024-02-10T08:35:03.075 INFO:teuthology.orchestra.run.smithi107.stdout:107374182444 2024-02-10T08:35:03.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 10 librados: watch_flush enter 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 10 librados: watch_flush exit 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 20 client.14712.objecter shutdown clearing up homeless session... 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 10 client.14712.objecter successfully canceled tick 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7e94063cc0 legacy=0x7f7e94066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 >> v1:172.21.15.107:6789/0 conn(0x7f7ea8103260 legacy=0x7f7ea81159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 shutdown_connections 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.076+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 >> 172.21.15.107:0/1206107928 conn(0x7f7ea80f2540 msgr2=0x7f7ea81066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.077+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 shutdown_connections 2024-02-10T08:35:03.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.077+0000 7f7eaf621700 1 -- 172.21.15.107:0/1206107928 wait complete. 2024-02-10T08:35:03.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:03.077+0000 7f7eaf621700 1 librados: shutdown 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.009442+0000 mgr.y (mgr.14553) 73 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.445405+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.446565+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.704578+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:02.705660+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:03.075784+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? v1:172.21.15.107:0/1206107928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:03.143497+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:35:03.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:03.146986+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.009442+0000 mgr.y (mgr.14553) 73 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.445405+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.446565+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.704578+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:02.705660+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:03.075784+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? v1:172.21.15.107:0/1206107928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:03.143497+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:35:03.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:03.146986+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:35:03.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182438 got 107374182444 for osd.3 2024-02-10T08:35:03.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T08:35:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.009442+0000 mgr.y (mgr.14553) 73 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.445405+0000 osd.3 (osd.3) 39 : cluster [DBG] 5.17 scrub starts 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.446565+0000 osd.3 (osd.3) 40 : cluster [DBG] 5.17 scrub ok 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.704578+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:02.705660+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:35:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:03.075784+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? v1:172.21.15.107:0/1206107928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:35:04.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:03.143497+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:35:04.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:03.146986+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.768+0000 7f3106902700 1 -- 172.21.15.107:0/1302583576 >> v1:172.21.15.107:6789/0 conn(0x7f31000fc930 legacy=0x7f31000fcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.768+0000 7f3106902700 1 -- 172.21.15.107:0/1302583576 shutdown_connections 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.768+0000 7f3106902700 1 -- 172.21.15.107:0/1302583576 >> 172.21.15.107:0/1302583576 conn(0x7f31000f2510 msgr2=0x7f31000f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.768+0000 7f3106902700 1 -- 172.21.15.107:0/1302583576 shutdown_connections 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.768+0000 7f3106902700 1 -- 172.21.15.107:0/1302583576 wait complete. 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 librados: starting msgr at 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 librados: starting objecter 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 Processor -- start 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 -- start start 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 librados: setting wanted keys 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 librados: calling monclient init 2024-02-10T08:35:05.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31000fc080 con 0x7f31000fc930 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31000fc250 con 0x7f31000ff730 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3106902700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f310011bb40 con 0x7f3100103270 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f3104e9f700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3100103270 0x7f3100118400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48644/0 (socket says 172.21.15.107:48644) 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.769+0000 7f30fffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f31000fc930 0x7f31000fb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40940/0 (socket says 172.21.15.107:40940) 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f3104e9f700 1 -- 172.21.15.107:0/205743154 learned_addr learned my addr 172.21.15.107:0/205743154 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter ms_handle_connect 0x7f3100103270 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter ms_handle_connect 0x7f31000fc930 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter ms_handle_connect 0x7f31000ff730 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707293049 0 0) 0x7f310011bb40 con 0x7f3100103270 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e8003030 con 0x7f3100103270 2024-02-10T08:35:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182450286 0 0) 0x7f31000fc080 con 0x7f31000fc930 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310011bb40 con 0x7f31000fc930 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647106527 0 0) 0x7f31000fc250 con 0x7f31000ff730 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31000fc080 con 0x7f31000ff730 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 706100603 0 0) 0x7f30e8003030 con 0x7f3100103270 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31000fc250 con 0x7f3100103270 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465787925 0 0) 0x7f310011bb40 con 0x7f31000fc930 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e8003030 con 0x7f31000fc930 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30f8001ca0 con 0x7f3100103270 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30f0001f50 con 0x7f31000fc930 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 560704360 0 0) 0x7f31000fc080 con 0x7f31000ff730 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f310011bb40 con 0x7f31000ff730 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30f40018c0 con 0x7f31000ff730 2024-02-10T08:35:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3047783932 0 0) 0x7f31000fc250 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 >> v1:172.21.15.195:6789/0 conn(0x7f31000ff730 legacy=0x7f3100114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 >> v1:172.21.15.107:6789/0 conn(0x7f31000fc930 legacy=0x7f31000fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310011cce0 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310011bd30 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f3106902700 10 client.34637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f310011c1a0 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f3106902700 1 librados: init done 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f30f8001e90 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.770+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30f8005e50 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.771+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec004f20 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f30f8002360 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(80..80 src has 30..80) v4 ==== 6172+0+0 (unknown 3594425049 0 0) 0x7f30f8078920 con 0x7f3100103270 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 10 client.34637.objecter ms_dispatch 0x7f3100001550 osd_map(80..80 src has 30..80) v4 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 3 client.34637.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 3 client.34637.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 20 client.34637.objecter dump_active .. 0 homeless 2024-02-10T08:35:05.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.772+0000 7f30fd7fa700 10 client.34637.objecter ms_handle_connect 0x7f30e8063c40 2024-02-10T08:35:05.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.776+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f30f804c4d0 con 0x7f3100103270 2024-02-10T08:35:05.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:05] "GET /metrics HTTP/1.1" 200 172145 "" "Prometheus/2.33.4" 2024-02-10T08:35:05.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:03.717370+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.e scrub starts 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:03.720312+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.e scrub ok 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:04.009773+0000 mgr.y (mgr.14553) 74 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:04.108181+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:04.110046+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:04.455847+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:04.456706+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:05.503586+0000 osd.5 (osd.5) 31 : cluster [DBG] 4.8 deep-scrub starts 2024-02-10T08:35:05.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:05.510623+0000 osd.5 (osd.5) 32 : cluster [DBG] 4.8 deep-scrub ok 2024-02-10T08:35:05.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:03.717370+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.e scrub starts 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:03.720312+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.e scrub ok 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:04.009773+0000 mgr.y (mgr.14553) 74 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:04.108181+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:04.110046+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:04.455847+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:04.456706+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:05.503586+0000 osd.5 (osd.5) 31 : cluster [DBG] 4.8 deep-scrub starts 2024-02-10T08:35:05.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:05.510623+0000 osd.5 (osd.5) 32 : cluster [DBG] 4.8 deep-scrub ok 2024-02-10T08:35:05.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.938+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f30ec0051d0 con 0x7f3100103270 2024-02-10T08:35:05.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.938+0000 7f30fd7fa700 1 -- 172.21.15.107:0/205743154 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 718512730) 0x7f30f804a940 con 0x7f3100103270 2024-02-10T08:35:05.939 INFO:teuthology.orchestra.run.smithi107.stdout:133143986214 2024-02-10T08:35:05.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 10 librados: watch_flush enter 2024-02-10T08:35:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 10 librados: watch_flush exit 2024-02-10T08:35:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 20 client.34637.objecter shutdown clearing up homeless session... 2024-02-10T08:35:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 10 client.34637.objecter successfully canceled tick 2024-02-10T08:35:05.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 >> v1:172.21.15.107:6800/2634842009 conn(0x7f30e8063c40 legacy=0x7f30e8066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 >> v1:172.21.15.107:6790/0 conn(0x7f3100103270 legacy=0x7f3100118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 shutdown_connections 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 >> 172.21.15.107:0/205743154 conn(0x7f31000f2510 msgr2=0x7f31000613e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 shutdown_connections 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 -- 172.21.15.107:0/205743154 wait complete. 2024-02-10T08:35:05.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:05.940+0000 7f3106902700 1 librados: shutdown 2024-02-10T08:35:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:03.717370+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.e scrub starts 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:03.720312+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.e scrub ok 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:04.009773+0000 mgr.y (mgr.14553) 74 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:04.108181+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:04.110046+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:04.455847+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:04.456706+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:05.503586+0000 osd.5 (osd.5) 31 : cluster [DBG] 4.8 deep-scrub starts 2024-02-10T08:35:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:05.510623+0000 osd.5 (osd.5) 32 : cluster [DBG] 4.8 deep-scrub ok 2024-02-10T08:35:06.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986209 got 133143986214 for osd.4 2024-02-10T08:35:06.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:05.939114+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.107:0/205743154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:06.517537+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.f scrub starts 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:06.520517+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.f scrub ok 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:05.939114+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.107:0/205743154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:06.517537+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.f scrub starts 2024-02-10T08:35:06.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:06.520517+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.f scrub ok 2024-02-10T08:35:06.945 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:05.939114+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.107:0/205743154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:35:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:06.517537+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.f scrub starts 2024-02-10T08:35:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:06.520517+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.f scrub ok 2024-02-10T08:35:07.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:06.010368+0000 mgr.y (mgr.14553) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:07.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.223232+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:35:07.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.227582+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:35:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:06.010368+0000 mgr.y (mgr.14553) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:07.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.223232+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:35:07.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.227582+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:35:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:06.010368+0000 mgr.y (mgr.14553) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.223232+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:35:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.227582+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.795+0000 7f86f6fea700 1 -- 172.21.15.107:0/1396611753 >> v1:172.21.15.107:6790/0 conn(0x7f86f0103250 legacy=0x7f86f01056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 -- 172.21.15.107:0/1396611753 shutdown_connections 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 -- 172.21.15.107:0/1396611753 >> 172.21.15.107:0/1396611753 conn(0x7f86f00f2510 msgr2=0x7f86f00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 -- 172.21.15.107:0/1396611753 shutdown_connections 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 -- 172.21.15.107:0/1396611753 wait complete. 2024-02-10T08:35:08.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 librados: starting msgr at 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 librados: starting objecter 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 Processor -- start 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 -- start start 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 librados: setting wanted keys 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.796+0000 7f86f6fea700 1 librados: calling monclient init 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f6fea700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86f0116340 con 0x7f86f00fc910 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f6fea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86f0117540 con 0x7f86f0103250 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f6fea700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86f0118740 con 0x7f86f00ff710 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f4d86700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86f00fc910 0x7f86f00f88f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40956/0 (socket says 172.21.15.107:40956) 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86effff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f86f00ff710 0x7f86f00f90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48668/0 (socket says 172.21.15.107:48668) 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f4d86700 1 -- 172.21.15.107:0/2353050492 learned_addr learned my addr 172.21.15.107:0/2353050492 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter ms_handle_connect 0x7f86f00ff710 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:08.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter ms_handle_connect 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674580620 0 0) 0x7f86f0116340 con 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e00038e0 con 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter ms_handle_connect 0x7f86f0103250 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857978996 0 0) 0x7f86f0118740 con 0x7f86f00ff710 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0116340 con 0x7f86f00ff710 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567427186 0 0) 0x7f86f0117540 con 0x7f86f0103250 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0118740 con 0x7f86f0103250 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245110669 0 0) 0x7f86e00038e0 con 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f0117540 con 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86e4001940 con 0x7f86f00fc910 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654086919 0 0) 0x7f86f0116340 con 0x7f86f00ff710 2024-02-10T08:35:08.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e00038e0 con 0x7f86f00ff710 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86dc0018d0 con 0x7f86f00ff710 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852578517 0 0) 0x7f86f0117540 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 >> v1:172.21.15.107:6790/0 conn(0x7f86f00ff710 legacy=0x7f86f00f90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 >> v1:172.21.15.195:6789/0 conn(0x7f86f0103250 legacy=0x7f86f0114a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0119940 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.797+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0116530 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.798+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f86e4001cd0 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.798+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86e4005ca0 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.798+0000 7f86f6fea700 10 client.14724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.798+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f01169a0 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.798+0000 7f86f6fea700 1 librados: init done 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f005f7a0 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f86e4002270 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(80..80 src has 30..80) v4 ==== 6172+0+0 (unknown 3594425049 0 0) 0x7f86e4078690 con 0x7f86f00fc910 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 10 client.14724.objecter ms_dispatch 0x7f86f0001550 osd_map(80..80 src has 30..80) v4 2024-02-10T08:35:08.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 3 client.14724.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:35:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 3 client.14724.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:35:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.799+0000 7f86edffb700 20 client.14724.objecter dump_active .. 0 homeless 2024-02-10T08:35:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.800+0000 7f86edffb700 10 client.14724.objecter ms_handle_connect 0x7f86e0063b50 2024-02-10T08:35:08.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.803+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f86e404c240 con 0x7f86f00fc910 2024-02-10T08:35:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.037585+0000 osd.4 (osd.4) 31 : cluster [DBG] 5.1c scrub starts 2024-02-10T08:35:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.038719+0000 osd.4 (osd.4) 32 : cluster [DBG] 5.1c scrub ok 2024-02-10T08:35:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.224056+0000 osd.1 (osd.1) 43 : cluster [DBG] 5.c scrub starts 2024-02-10T08:35:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.226943+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.c scrub ok 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.520706+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.523995+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.037585+0000 osd.4 (osd.4) 31 : cluster [DBG] 5.1c scrub starts 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.038719+0000 osd.4 (osd.4) 32 : cluster [DBG] 5.1c scrub ok 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.224056+0000 osd.1 (osd.1) 43 : cluster [DBG] 5.c scrub starts 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.226943+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.c scrub ok 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.520706+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:35:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.523995+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:35:08.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.967+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f86f0108f50 con 0x7f86f00fc910 2024-02-10T08:35:08.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.968+0000 7f86edffb700 1 -- 172.21.15.107:0/2353050492 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 2597790923) 0x7f86e404a650 con 0x7f86f00fc910 2024-02-10T08:35:08.968 INFO:teuthology.orchestra.run.smithi107.stdout:158913789987 2024-02-10T08:35:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 10 librados: watch_flush enter 2024-02-10T08:35:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 10 librados: watch_flush exit 2024-02-10T08:35:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 20 client.14724.objecter shutdown clearing up homeless session... 2024-02-10T08:35:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 10 client.14724.objecter successfully canceled tick 2024-02-10T08:35:08.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 >> v1:172.21.15.107:6800/2634842009 conn(0x7f86e0063b50 legacy=0x7f86e0066010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:08.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 >> v1:172.21.15.107:6789/0 conn(0x7f86f00fc910 legacy=0x7f86f00f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:08.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 shutdown_connections 2024-02-10T08:35:08.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 >> 172.21.15.107:0/2353050492 conn(0x7f86f00f2510 msgr2=0x7f86f01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:08.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 shutdown_connections 2024-02-10T08:35:08.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.969+0000 7f86f6fea700 1 -- 172.21.15.107:0/2353050492 wait complete. 2024-02-10T08:35:08.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:08.970+0000 7f86f6fea700 1 librados: shutdown 2024-02-10T08:35:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.037585+0000 osd.4 (osd.4) 31 : cluster [DBG] 5.1c scrub starts 2024-02-10T08:35:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.038719+0000 osd.4 (osd.4) 32 : cluster [DBG] 5.1c scrub ok 2024-02-10T08:35:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.224056+0000 osd.1 (osd.1) 43 : cluster [DBG] 5.c scrub starts 2024-02-10T08:35:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.226943+0000 osd.1 (osd.1) 44 : cluster [DBG] 5.c scrub ok 2024-02-10T08:35:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.520706+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:35:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.523995+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:35:09.766 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789982 got 158913789987 for osd.5 2024-02-10T08:35:09.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T08:35:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.721744+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:35:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:07.725057+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:35:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.010771+0000 mgr.y (mgr.14553) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:08.968643+0000 mon.a (mon.0) 713 : audit [DBG] from='client.? v1:172.21.15.107:0/2353050492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:35:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:09.043347+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:09.044884+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.721744+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:07.725057+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.010771+0000 mgr.y (mgr.14553) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:08.968643+0000 mon.a (mon.0) 713 : audit [DBG] from='client.? v1:172.21.15.107:0/2353050492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:09.043347+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:35:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:09.044884+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:35:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.721744+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:07.725057+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.010771+0000 mgr.y (mgr.14553) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:08.968643+0000 mon.a (mon.0) 713 : audit [DBG] from='client.? v1:172.21.15.107:0/2353050492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:09.043347+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:35:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:09.044884+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:35:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.717878+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:08.720781+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:09.487362+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:09.492012+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:10.029939+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:35:10.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:10.030933+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.717878+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:08.720781+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:09.487362+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:09.492012+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:10.029939+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:35:10.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:10.030933+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.717878+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:08.720781+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:09.487362+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:09.492012+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:10.029939+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:35:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:10.030933+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:35:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:10.011167+0000 mgr.y (mgr.14553) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.021817+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.024924+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.186564+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.f scrub starts 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.190429+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.f scrub ok 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:10.011167+0000 mgr.y (mgr.14553) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-02-10T08:35:11.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.021817+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:35:11.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.024924+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:35:11.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.186564+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.f scrub starts 2024-02-10T08:35:11.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.190429+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.f scrub ok 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- 172.21.15.107:0/2725819262 >> v1:172.21.15.107:6789/0 conn(0x7faa4c0f9570 legacy=0x7faa4c102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- 172.21.15.107:0/2725819262 shutdown_connections 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- 172.21.15.107:0/2725819262 >> 172.21.15.107:0/2725819262 conn(0x7faa4c0f2510 msgr2=0x7faa4c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- 172.21.15.107:0/2725819262 shutdown_connections 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- 172.21.15.107:0/2725819262 wait complete. 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 librados: starting msgr at 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 librados: starting objecter 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 Processor -- start 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 -- start start 2024-02-10T08:35:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 librados: setting wanted keys 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.982+0000 7faa52257700 1 librados: calling monclient init 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa4c1174a0 con 0x7faa4c104350 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa4c118680 con 0x7faa4c0f9570 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa4c119880 con 0x7faa4c0f89c0 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa4b7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faa4c0f89c0 0x7faa4c0ff790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56960/0 (socket says 172.21.15.107:56960) 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa4b7fe700 1 -- 172.21.15.107:0/3096932824 learned_addr learned my addr 172.21.15.107:0/3096932824 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa4bfff700 1 --1- 172.21.15.107:0/3096932824 >> v1:172.21.15.107:6789/0 conn(0x7faa4c104350 0x7faa4c115b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52576/0 (socket says 172.21.15.107:52576) 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter ms_handle_connect 0x7faa4c104350 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter ms_handle_connect 0x7faa4c0f89c0 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:11.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter ms_handle_connect 0x7faa4c0f9570 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499887140 0 0) 0x7faa4c1174a0 con 0x7faa4c104350 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa380038e0 con 0x7faa4c104350 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233647637 0 0) 0x7faa4c119880 con 0x7faa4c0f89c0 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4c1174a0 con 0x7faa4c0f89c0 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432285953 0 0) 0x7faa4c118680 con 0x7faa4c0f9570 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4c119880 con 0x7faa4c0f9570 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141305791 0 0) 0x7faa380038e0 con 0x7faa4c104350 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa4c118680 con 0x7faa4c104350 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa44001bc0 con 0x7faa4c104350 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3698585741 0 0) 0x7faa4c1174a0 con 0x7faa4c0f89c0 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa380038e0 con 0x7faa4c0f89c0 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa340019f0 con 0x7faa4c0f89c0 2024-02-10T08:35:11.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743371554 0 0) 0x7faa4c119880 con 0x7faa4c0f9570 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa4c1174a0 con 0x7faa4c0f9570 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa3c002070 con 0x7faa4c0f9570 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2115273618 0 0) 0x7faa4c118680 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 >> v1:172.21.15.107:6790/0 conn(0x7faa4c0f89c0 legacy=0x7faa4c0ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 >> v1:172.21.15.195:6789/0 conn(0x7faa4c0f9570 legacy=0x7faa4c0ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c11aa80 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c119a70 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 10 client.14730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c119ee0 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.983+0000 7faa52257700 1 librados: init done 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.984+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7faa440027a0 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.984+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faa44005e30 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.984+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c004f20 con 0x7faa4c104350 2024-02-10T08:35:11.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7faa44002300 con 0x7faa4c104350 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(80..80 src has 30..80) v4 ==== 6172+0+0 (unknown 3594425049 0 0) 0x7faa440789b0 con 0x7faa4c104350 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 10 client.14730.objecter ms_dispatch 0x7faa4c001550 osd_map(80..80 src has 30..80) v4 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 3 client.14730.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 3 client.14730.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 20 client.14730.objecter dump_active .. 0 homeless 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.985+0000 7faa48ff9700 10 client.14730.objecter ms_handle_connect 0x7faa38063d10 2024-02-10T08:35:11.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:11.989+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faa4404c540 con 0x7faa4c104350 2024-02-10T08:35:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:10.011167+0000 mgr.y (mgr.14553) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-02-10T08:35:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.021817+0000 osd.7 (osd.7) 37 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:35:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.024924+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:35:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.186564+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.f scrub starts 2024-02-10T08:35:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.190429+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.f scrub ok 2024-02-10T08:35:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.154+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7faa2c0051d0 con 0x7faa4c104350 2024-02-10T08:35:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.155+0000 7faa48ff9700 1 -- 172.21.15.107:0/3096932824 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 1242847335) 0x7faa4404a9b0 con 0x7faa4c104350 2024-02-10T08:35:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:188978561054 2024-02-10T08:35:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 10 librados: watch_flush enter 2024-02-10T08:35:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 10 librados: watch_flush exit 2024-02-10T08:35:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 20 client.14730.objecter shutdown clearing up homeless session... 2024-02-10T08:35:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 10 client.14730.objecter successfully canceled tick 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 >> v1:172.21.15.107:6800/2634842009 conn(0x7faa38063d10 legacy=0x7faa380661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.156+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 >> v1:172.21.15.107:6789/0 conn(0x7faa4c104350 legacy=0x7faa4c115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.157+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 shutdown_connections 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.157+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 >> 172.21.15.107:0/3096932824 conn(0x7faa4c0f2510 msgr2=0x7faa4c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.157+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 shutdown_connections 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.157+0000 7faa52257700 1 -- 172.21.15.107:0/3096932824 wait complete. 2024-02-10T08:35:12.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:12.157+0000 7faa52257700 1 librados: shutdown 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.023432+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.026904+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:11.065296+0000 mgr.y (mgr.14553) 78 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.540431+0000 osd.3 (osd.3) 45 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:11.542328+0000 osd.3 (osd.3) 46 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:12.156843+0000 mon.a (mon.0) 714 : audit [DBG] from='client.? v1:172.21.15.107:0/3096932824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.023432+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.026904+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:11.065296+0000 mgr.y (mgr.14553) 78 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.540431+0000 osd.3 (osd.3) 45 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:35:12.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:11.542328+0000 osd.3 (osd.3) 46 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:35:12.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:12.156843+0000 mon.a (mon.0) 714 : audit [DBG] from='client.? v1:172.21.15.107:0/3096932824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:35:12.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561050 got 188978561054 for osd.6 2024-02-10T08:35:12.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.023432+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.026904+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:11.065296+0000 mgr.y (mgr.14553) 78 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.540431+0000 osd.3 (osd.3) 45 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:11.542328+0000 osd.3 (osd.3) 46 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:35:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:12.156843+0000 mon.a (mon.0) 714 : audit [DBG] from='client.? v1:172.21.15.107:0/3096932824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:35:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:12.011775+0000 mgr.y (mgr.14553) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:13.028001+0000 osd.7 (osd.7) 39 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:35:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:13.030882+0000 osd.7 (osd.7) 40 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:13.491223+0000 osd.5 (osd.5) 37 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:13.494694+0000 osd.5 (osd.5) 38 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:12.011775+0000 mgr.y (mgr.14553) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:13.028001+0000 osd.7 (osd.7) 39 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:13.030882+0000 osd.7 (osd.7) 40 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:13.491223+0000 osd.5 (osd.5) 37 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:13.494694+0000 osd.5 (osd.5) 38 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:35:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:12.011775+0000 mgr.y (mgr.14553) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:13.028001+0000 osd.7 (osd.7) 39 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:35:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:13.030882+0000 osd.7 (osd.7) 40 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:35:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:13.491223+0000 osd.5 (osd.5) 37 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:35:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:13.494694+0000 osd.5 (osd.5) 38 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:35:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.057+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3906169258 >> v1:172.21.15.107:6790/0 conn(0x7f3b780fda50 legacy=0x7f3b780fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.057+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3906169258 shutdown_connections 2024-02-10T08:35:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.057+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3906169258 >> 172.21.15.107:0/3906169258 conn(0x7f3b780f3640 msgr2=0x7f3b780f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.057+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3906169258 shutdown_connections 2024-02-10T08:35:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.057+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3906169258 wait complete. 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 librados: starting msgr at 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 librados: starting objecter 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 Processor -- start 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 -- start start 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 librados: setting wanted keys 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 librados: calling monclient init 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b781173d0 con 0x7f3b780fda50 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b781185b0 con 0x7f3b78104330 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b7e9ab700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b781197b0 con 0x7f3b781007f0 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b6ffff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3b781007f0 0x7f3b780fa290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56974/0 (socket says 172.21.15.107:56974) 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b6ffff700 1 -- 172.21.15.107:0/3687234985 learned_addr learned my addr 172.21.15.107:0/3687234985 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b75ffb700 10 client.?.objecter ms_handle_connect 0x7f3b781007f0 2024-02-10T08:35:15.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.058+0000 7f3b75ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 10 client.?.objecter ms_handle_connect 0x7f3b780fda50 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 10 client.?.objecter ms_handle_connect 0x7f3b78104330 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826499740 0 0) 0x7f3b781197b0 con 0x7f3b781007f0 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b640038e0 con 0x7f3b781007f0 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150037126 0 0) 0x7f3b781173d0 con 0x7f3b780fda50 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b781197b0 con 0x7f3b780fda50 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131850858 0 0) 0x7f3b781185b0 con 0x7f3b78104330 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b781173d0 con 0x7f3b78104330 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280552092 0 0) 0x7f3b640038e0 con 0x7f3b781007f0 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b781185b0 con 0x7f3b781007f0 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b60001900 con 0x7f3b781007f0 2024-02-10T08:35:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249383338 0 0) 0x7f3b781197b0 con 0x7f3b780fda50 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b640038e0 con 0x7f3b780fda50 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b68001050 con 0x7f3b780fda50 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3154510392 0 0) 0x7f3b781173d0 con 0x7f3b78104330 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b781197b0 con 0x7f3b78104330 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b70001a20 con 0x7f3b78104330 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225704454 0 0) 0x7f3b781185b0 con 0x7f3b781007f0 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 >> v1:172.21.15.195:6789/0 conn(0x7f3b78104330 legacy=0x7f3b78115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 >> v1:172.21.15.107:6789/0 conn(0x7f3b780fda50 legacy=0x7f3b780f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7811a9b0 con 0x7f3b781007f0 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b781175c0 con 0x7f3b781007f0 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b7e9ab700 10 client.34655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b78117a30 con 0x7f3b781007f0 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b7e9ab700 1 librados: init done 2024-02-10T08:35:15.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3b60002420 con 0x7f3b781007f0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.059+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b60005ae0 con 0x7f3b781007f0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.060+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b58004f20 con 0x7f3b781007f0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.060+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3b60001f80 con 0x7f3b781007f0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(80..80 src has 30..80) v4 ==== 6172+0+0 (unknown 3594425049 0 0) 0x7f3b60078600 con 0x7f3b781007f0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 10 client.34655.objecter ms_dispatch 0x7f3b78001550 osd_map(80..80 src has 30..80) v4 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 3 client.34655.objecter handle_osd_map got epochs [80,80] > 0 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 3 client.34655.objecter handle_osd_map decoding full epoch 80 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 20 client.34655.objecter dump_active .. 0 homeless 2024-02-10T08:35:15.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.061+0000 7f3b75ffb700 10 client.34655.objecter ms_handle_connect 0x7f3b64063d10 2024-02-10T08:35:15.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.065+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3b6004c1b0 con 0x7f3b781007f0 2024-02-10T08:35:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.234+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3b580051d0 con 0x7f3b781007f0 2024-02-10T08:35:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.234+0000 7f3b75ffb700 1 -- 172.21.15.107:0/3687234985 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 52473595) 0x7f3b6004a5c0 con 0x7f3b781007f0 2024-02-10T08:35:15.236 INFO:teuthology.orchestra.run.smithi107.stdout:223338299417 2024-02-10T08:35:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 10 librados: watch_flush enter 2024-02-10T08:35:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 10 librados: watch_flush exit 2024-02-10T08:35:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 20 client.34655.objecter shutdown clearing up homeless session... 2024-02-10T08:35:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 10 client.34655.objecter successfully canceled tick 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3b64063d10 legacy=0x7f3b640661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 >> v1:172.21.15.107:6790/0 conn(0x7f3b781007f0 legacy=0x7f3b780fa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 shutdown_connections 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 >> 172.21.15.107:0/3687234985 conn(0x7f3b780f3640 msgr2=0x7f3b781077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 shutdown_connections 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 -- 172.21.15.107:0/3687234985 wait complete. 2024-02-10T08:35:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:15.236+0000 7f3b7e9ab700 1 librados: shutdown 2024-02-10T08:35:15.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:14.012122+0000 mgr.y (mgr.14553) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:35:15.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:15.235984+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/3687234985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:35:15.907 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:14.012122+0000 mgr.y (mgr.14553) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:35:15.908 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:15.235984+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/3687234985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:35:15.908 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:15] "GET /metrics HTTP/1.1" 200 172142 "" "Prometheus/2.33.4" 2024-02-10T08:35:15.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299412 got 223338299417 for osd.7 2024-02-10T08:35:15.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T08:35:15.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:35:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:14.012122+0000 mgr.y (mgr.14553) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:35:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:15.235984+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/3687234985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:35:16.201 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:15.477480+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:35:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:15.478950+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:35:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.506936+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.d scrub starts 2024-02-10T08:35:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.509969+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.d scrub ok 2024-02-10T08:35:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:15.477480+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:35:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:15.478950+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:35:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.506936+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.d scrub starts 2024-02-10T08:35:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.509969+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.d scrub ok 2024-02-10T08:35:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:15.477480+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:35:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:15.478950+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:35:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.506936+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.d scrub starts 2024-02-10T08:35:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.509969+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.d scrub ok 2024-02-10T08:35:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.012764+0000 mgr.y (mgr.14553) 81 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.230233+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.230277+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:16.680070+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-10T08:35:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.012764+0000 mgr.y (mgr.14553) 81 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.230233+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.230277+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:16.680070+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-10T08:35:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.012764+0000 mgr.y (mgr.14553) 81 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.230233+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.230277+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:16.680070+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-10T08:35:18.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.245+0000 7f98c9074700 1 -- 172.21.15.107:0/1127936245 >> v1:172.21.15.107:6790/0 conn(0x7f98c4103250 legacy=0x7f98c41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 -- 172.21.15.107:0/1127936245 shutdown_connections 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 -- 172.21.15.107:0/1127936245 >> 172.21.15.107:0/1127936245 conn(0x7f98c40f2510 msgr2=0x7f98c40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 -- 172.21.15.107:0/1127936245 shutdown_connections 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 -- 172.21.15.107:0/1127936245 wait complete. 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 librados: starting msgr at 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 librados: starting objecter 2024-02-10T08:35:18.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.246+0000 7f98c9074700 1 Processor -- start 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 -- start start 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 librados: setting wanted keys 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 librados: calling monclient init 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f98c41171a0 con 0x7f98c40ff710 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f98c4118380 con 0x7f98c4103250 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c9074700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f98c4119580 con 0x7f98c40fc910 2024-02-10T08:35:18.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c2d9d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f98c40fc910 0x7f98c4060c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56990/0 (socket says 172.21.15.107:56990) 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98c2d9d700 1 -- 172.21.15.107:0/4285973093 learned_addr learned my addr 172.21.15.107:0/4285973093 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter ms_handle_connect 0x7f98c40fc910 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter ms_handle_connect 0x7f98c40ff710 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500350670 0 0) 0x7f98c4119580 con 0x7f98c40fc910 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac003030 con 0x7f98c40fc910 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter ms_handle_connect 0x7f98c4103250 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413109601 0 0) 0x7f98c41171a0 con 0x7f98c40ff710 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4119580 con 0x7f98c40ff710 2024-02-10T08:35:18.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421145213 0 0) 0x7f98c4118380 con 0x7f98c4103250 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c41171a0 con 0x7f98c4103250 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853422688 0 0) 0x7f98ac003030 con 0x7f98c40fc910 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c4118380 con 0x7f98c40fc910 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f98b80019f0 con 0x7f98c40fc910 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896385730 0 0) 0x7f98c4119580 con 0x7f98c40ff710 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98ac003030 con 0x7f98c40ff710 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.247+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f98b4002750 con 0x7f98c40ff710 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 345899667 0 0) 0x7f98c4118380 con 0x7f98c40fc910 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 >> v1:172.21.15.195:6789/0 conn(0x7f98c4103250 legacy=0x7f98c41158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 >> v1:172.21.15.107:6789/0 conn(0x7f98c40ff710 legacy=0x7f98c4061370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c411a780 con 0x7f98c40fc910 2024-02-10T08:35:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c4117390 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98c9074700 10 client.34661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4117880 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98c9074700 1 librados: init done 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f98b8002550 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.248+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f98b8005c20 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b0004f20 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f98b8018ab0 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(81..81 src has 30..81) v4 ==== 6172+0+0 (unknown 3238675313 0 0) 0x7f98b8078670 con 0x7f98c40fc910 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98abfff700 10 client.34661.objecter ms_dispatch 0x7f98c4001550 osd_map(81..81 src has 30..81) v4 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98abfff700 3 client.34661.objecter handle_osd_map got epochs [81,81] > 0 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.249+0000 7f98abfff700 3 client.34661.objecter handle_osd_map decoding full epoch 81 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.250+0000 7f98abfff700 20 client.34661.objecter dump_active .. 0 homeless 2024-02-10T08:35:18.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.250+0000 7f98abfff700 10 client.34661.objecter ms_handle_connect 0x7f98ac063bc0 2024-02-10T08:35:18.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.253+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f98b804c1c0 con 0x7f98c40fc910 2024-02-10T08:35:18.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.415+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98b0000b90 con 0x7f98ac063bc0 2024-02-10T08:35:18.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.425+0000 7f98abfff700 1 -- 172.21.15.107:0/4285973093 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357344 (unknown 2965378022 0 3147899911) 0x7f98b0000b90 con 0x7f98ac063bc0 2024-02-10T08:35:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:35:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 10 librados: watch_flush enter 2024-02-10T08:35:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 10 librados: watch_flush exit 2024-02-10T08:35:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 20 client.34661.objecter shutdown clearing up homeless session... 2024-02-10T08:35:18.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 10 client.34661.objecter successfully canceled tick 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 >> v1:172.21.15.107:6800/2634842009 conn(0x7f98ac063bc0 legacy=0x7f98ac066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 >> v1:172.21.15.107:6790/0 conn(0x7f98c40fc910 legacy=0x7f98c4060c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 shutdown_connections 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.428+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 >> 172.21.15.107:0/4285973093 conn(0x7f98c40f2510 msgr2=0x7f98c40f5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.429+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 shutdown_connections 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.429+0000 7f98c9074700 1 -- 172.21.15.107:0/4285973093 wait complete. 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:18.429+0000 7f98c9074700 1 librados: shutdown 2024-02-10T08:35:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:35:18.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:17.891574+0000 osd.4 (osd.4) 35 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:35:18.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:17.895156+0000 osd.4 (osd.4) 36 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:35:18.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:17.891574+0000 osd.4 (osd.4) 35 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:35:18.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:17.895156+0000 osd.4 (osd.4) 36 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:35:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:17.891574+0000 osd.4 (osd.4) 35 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:35:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:17.895156+0000 osd.4 (osd.4) 36 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:35:19.131 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-02-10T08:35:18.012930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":831,"num_read_kb":583,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":493,"ondisk_log_size":493,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2383264,"kb_used_data":6496,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747577952,"statfs":{"total":767960285184,"available":765519822848,"internally_reserved":0,"allocated":6651904,"data_stored":2720379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002353"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.420536+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:35:00.420536+0000","last_peered":"2024-02-10T08:35:00.420536+0000","last_clean":"2024-02-10T08:35:00.420536+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:35:00.420536+0000","last_undegraded":"2024-02-10T08:35:00.420536+0000","last_fullsized":"2024-02-10T08:35:00.420536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774644+0000","last_change":"2024-02-10T08:33:22.117853+0000","last_active":"2024-02-10T08:34:56.774644+0000","last_peered":"2024-02-10T08:34:56.774644+0000","last_clean":"2024-02-10T08:34:56.774644+0000","last_became_active":"2024-02-10T08:33:22.117534+0000","last_became_peered":"2024-02-10T08:33:22.117534+0000","last_unstale":"2024-02-10T08:34:56.774644+0000","last_undegraded":"2024-02-10T08:34:56.774644+0000","last_fullsized":"2024-02-10T08:34:56.774644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:33:21.099708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.699140+0000","last_change":"2024-02-10T08:34:41.648797+0000","last_active":"2024-02-10T08:34:56.699140+0000","last_peered":"2024-02-10T08:34:56.699140+0000","last_clean":"2024-02-10T08:34:56.699140+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:34:56.699140+0000","last_undegraded":"2024-02-10T08:34:56.699140+0000","last_fullsized":"2024-02-10T08:34:56.699140+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:41.648715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:41.648715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:15.479032+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:35:15.479032+0000","last_peered":"2024-02-10T08:35:15.479032+0000","last_clean":"2024-02-10T08:35:15.479032+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:35:15.479032+0000","last_undegraded":"2024-02-10T08:35:15.479032+0000","last_fullsized":"2024-02-10T08:35:15.479032+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776462+0000","last_change":"2024-02-10T08:34:38.118431+0000","last_active":"2024-02-10T08:34:56.776462+0000","last_peered":"2024-02-10T08:34:56.776462+0000","last_clean":"2024-02-10T08:34:56.776462+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:34:56.776462+0000","last_undegraded":"2024-02-10T08:34:56.776462+0000","last_fullsized":"2024-02-10T08:34:56.776462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.118347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.118347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613119+0000","last_change":"2024-02-10T08:34:37.332223+0000","last_active":"2024-02-10T08:34:56.613119+0000","last_peered":"2024-02-10T08:34:56.613119+0000","last_clean":"2024-02-10T08:34:56.613119+0000","last_became_active":"2024-02-10T08:33:22.121607+0000","last_became_peered":"2024-02-10T08:33:22.121607+0000","last_unstale":"2024-02-10T08:34:56.613119+0000","last_undegraded":"2024-02-10T08:34:56.613119+0000","last_fullsized":"2024-02-10T08:34:56.613119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.332140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.332140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775027+0000","last_change":"2024-02-10T08:34:23.594197+0000","last_active":"2024-02-10T08:34:56.775027+0000","last_peered":"2024-02-10T08:34:56.775027+0000","last_clean":"2024-02-10T08:34:56.775027+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:34:56.775027+0000","last_undegraded":"2024-02-10T08:34:56.775027+0000","last_fullsized":"2024-02-10T08:34:56.775027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.594135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.594135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:04.110147+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:35:04.110147+0000","last_peered":"2024-02-10T08:35:04.110147+0000","last_clean":"2024-02-10T08:35:04.110147+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:35:04.110147+0000","last_undegraded":"2024-02-10T08:35:04.110147+0000","last_fullsized":"2024-02-10T08:35:04.110147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.524105+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:35:08.524105+0000","last_peered":"2024-02-10T08:35:08.524105+0000","last_clean":"2024-02-10T08:35:08.524105+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:35:08.524105+0000","last_undegraded":"2024-02-10T08:35:08.524105+0000","last_fullsized":"2024-02-10T08:35:08.524105+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.936215+0000","last_change":"2024-02-10T08:35:00.936215+0000","last_active":"2024-02-10T08:35:00.936215+0000","last_peered":"2024-02-10T08:35:00.936215+0000","last_clean":"2024-02-10T08:35:00.936215+0000","last_became_active":"2024-02-10T08:33:22.119141+0000","last_became_peered":"2024-02-10T08:33:22.119141+0000","last_unstale":"2024-02-10T08:35:00.936215+0000","last_undegraded":"2024-02-10T08:35:00.936215+0000","last_fullsized":"2024-02-10T08:35:00.936215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775765+0000","last_change":"2024-02-10T08:34:26.673356+0000","last_active":"2024-02-10T08:34:56.775765+0000","last_peered":"2024-02-10T08:34:56.775765+0000","last_clean":"2024-02-10T08:34:56.775765+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:34:56.775765+0000","last_undegraded":"2024-02-10T08:34:56.775765+0000","last_fullsized":"2024-02-10T08:34:56.775765+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:26.673263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.673263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774643+0000","last_change":"2024-02-10T08:34:50.094010+0000","last_active":"2024-02-10T08:34:56.774643+0000","last_peered":"2024-02-10T08:34:56.774643+0000","last_clean":"2024-02-10T08:34:56.774643+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:34:56.774643+0000","last_undegraded":"2024-02-10T08:34:56.774643+0000","last_fullsized":"2024-02-10T08:34:56.774643+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:34:50.093915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.093915+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698222+0000","last_change":"2024-02-10T08:34:51.681056+0000","last_active":"2024-02-10T08:34:56.698222+0000","last_peered":"2024-02-10T08:34:56.698222+0000","last_clean":"2024-02-10T08:34:56.698222+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:34:56.698222+0000","last_undegraded":"2024-02-10T08:34:56.698222+0000","last_fullsized":"2024-02-10T08:34:56.698222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.680986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.680986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:01.492010+0000","last_change":"2024-02-10T08:35:01.492010+0000","last_active":"2024-02-10T08:35:01.492010+0000","last_peered":"2024-02-10T08:35:01.492010+0000","last_clean":"2024-02-10T08:35:01.492010+0000","last_became_active":"2024-02-10T08:33:22.122241+0000","last_became_peered":"2024-02-10T08:33:22.122241+0000","last_unstale":"2024-02-10T08:35:01.492010+0000","last_undegraded":"2024-02-10T08:35:01.492010+0000","last_fullsized":"2024-02-10T08:35:01.492010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.956668+0000","last_change":"2024-02-10T08:34:56.956608+0000","last_active":"2024-02-10T08:34:56.956668+0000","last_peered":"2024-02-10T08:34:56.956668+0000","last_clean":"2024-02-10T08:34:56.956668+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:34:56.956668+0000","last_undegraded":"2024-02-10T08:34:56.956668+0000","last_fullsized":"2024-02-10T08:34:56.956668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.956483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.956483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:59.084255+0000","last_change":"2024-02-10T08:34:59.084224+0000","last_active":"2024-02-10T08:34:59.084255+0000","last_peered":"2024-02-10T08:34:59.084255+0000","last_clean":"2024-02-10T08:34:59.084255+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:34:59.084255+0000","last_undegraded":"2024-02-10T08:34:59.084255+0000","last_fullsized":"2024-02-10T08:34:59.084255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:59.084151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:59.084151+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775240+0000","last_change":"2024-02-10T08:34:52.516328+0000","last_active":"2024-02-10T08:34:56.775240+0000","last_peered":"2024-02-10T08:34:56.775240+0000","last_clean":"2024-02-10T08:34:56.775240+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:34:56.775240+0000","last_undegraded":"2024-02-10T08:34:56.775240+0000","last_fullsized":"2024-02-10T08:34:56.775240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.516236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.516236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:07.038806+0000","last_change":"2024-02-10T08:35:07.038806+0000","last_active":"2024-02-10T08:35:07.038806+0000","last_peered":"2024-02-10T08:35:07.038806+0000","last_clean":"2024-02-10T08:35:07.038806+0000","last_became_active":"2024-02-10T08:33:22.116085+0000","last_became_peered":"2024-02-10T08:33:22.116085+0000","last_unstale":"2024-02-10T08:35:07.038806+0000","last_undegraded":"2024-02-10T08:35:07.038806+0000","last_fullsized":"2024-02-10T08:35:07.038806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774698+0000","last_change":"2024-02-10T08:34:37.616548+0000","last_active":"2024-02-10T08:34:56.774698+0000","last_peered":"2024-02-10T08:34:56.774698+0000","last_clean":"2024-02-10T08:34:56.774698+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:34:56.774698+0000","last_undegraded":"2024-02-10T08:34:56.774698+0000","last_fullsized":"2024-02-10T08:34:56.774698+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.616436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.616436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774864+0000","last_change":"2024-02-10T08:34:33.601286+0000","last_active":"2024-02-10T08:34:56.774864+0000","last_peered":"2024-02-10T08:34:56.774864+0000","last_clean":"2024-02-10T08:34:56.774864+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:34:56.774864+0000","last_undegraded":"2024-02-10T08:34:56.774864+0000","last_fullsized":"2024-02-10T08:34:56.774864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:33.601201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.601201+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776940+0000","last_change":"2024-02-10T08:33:24.126031+0000","last_active":"2024-02-10T08:34:56.776940+0000","last_peered":"2024-02-10T08:34:56.776940+0000","last_clean":"2024-02-10T08:34:56.776940+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:34:56.776940+0000","last_undegraded":"2024-02-10T08:34:56.776940+0000","last_fullsized":"2024-02-10T08:34:56.776940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614387+0000","last_change":"2024-02-10T08:34:39.357828+0000","last_active":"2024-02-10T08:34:56.614387+0000","last_peered":"2024-02-10T08:34:56.614387+0000","last_clean":"2024-02-10T08:34:56.614387+0000","last_became_active":"2024-02-10T08:33:22.120631+0000","last_became_peered":"2024-02-10T08:33:22.120631+0000","last_unstale":"2024-02-10T08:34:56.614387+0000","last_undegraded":"2024-02-10T08:34:56.614387+0000","last_fullsized":"2024-02-10T08:34:56.614387+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.357766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612968+0000","last_change":"2024-02-10T08:34:53.298291+0000","last_active":"2024-02-10T08:34:56.612968+0000","last_peered":"2024-02-10T08:34:56.612968+0000","last_clean":"2024-02-10T08:34:56.612968+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:34:56.612968+0000","last_undegraded":"2024-02-10T08:34:56.612968+0000","last_fullsized":"2024-02-10T08:34:56.612968+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.298226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.298226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":61,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613313+0000","last_change":"2024-02-10T08:34:26.526319+0000","last_active":"2024-02-10T08:34:56.613313+0000","last_peered":"2024-02-10T08:34:56.613313+0000","last_clean":"2024-02-10T08:34:56.613313+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:34:56.613313+0000","last_undegraded":"2024-02-10T08:34:56.613313+0000","last_fullsized":"2024-02-10T08:34:56.613313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:34:26.526206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.526206+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612454+0000","last_change":"2024-02-10T08:34:52.285891+0000","last_active":"2024-02-10T08:34:56.612454+0000","last_peered":"2024-02-10T08:34:56.612454+0000","last_clean":"2024-02-10T08:34:56.612454+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:34:56.612454+0000","last_undegraded":"2024-02-10T08:34:56.612454+0000","last_fullsized":"2024-02-10T08:34:56.612454+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.285824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.285824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:01.687455+0000","last_change":"2024-02-10T08:35:01.687455+0000","last_active":"2024-02-10T08:35:01.687455+0000","last_peered":"2024-02-10T08:35:01.687455+0000","last_clean":"2024-02-10T08:35:01.687455+0000","last_became_active":"2024-02-10T08:33:22.122436+0000","last_became_peered":"2024-02-10T08:33:22.122436+0000","last_unstale":"2024-02-10T08:35:01.687455+0000","last_undegraded":"2024-02-10T08:35:01.687455+0000","last_fullsized":"2024-02-10T08:35:01.687455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.687386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774391+0000","last_change":"2024-02-10T08:34:24.604411+0000","last_active":"2024-02-10T08:34:56.774391+0000","last_peered":"2024-02-10T08:34:56.774391+0000","last_clean":"2024-02-10T08:34:56.774391+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:34:56.774391+0000","last_undegraded":"2024-02-10T08:34:56.774391+0000","last_fullsized":"2024-02-10T08:34:56.774391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.604347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.604347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613427+0000","last_change":"2024-02-10T08:34:36.450017+0000","last_active":"2024-02-10T08:34:56.613427+0000","last_peered":"2024-02-10T08:34:56.613427+0000","last_clean":"2024-02-10T08:34:56.613427+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:34:56.613427+0000","last_undegraded":"2024-02-10T08:34:56.613427+0000","last_fullsized":"2024-02-10T08:34:56.613427+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:36.449943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.449943+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:09.044967+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:35:09.044967+0000","last_peered":"2024-02-10T08:35:09.044967+0000","last_clean":"2024-02-10T08:35:09.044967+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:35:09.044967+0000","last_undegraded":"2024-02-10T08:35:09.044967+0000","last_fullsized":"2024-02-10T08:35:09.044967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'9","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511275+0000","last_change":"2024-02-10T08:34:35.421984+0000","last_active":"2024-02-10T08:35:06.511275+0000","last_peered":"2024-02-10T08:35:06.511275+0000","last_clean":"2024-02-10T08:35:06.511275+0000","last_became_active":"2024-02-10T08:33:22.119251+0000","last_became_peered":"2024-02-10T08:33:22.119251+0000","last_unstale":"2024-02-10T08:35:06.511275+0000","last_undegraded":"2024-02-10T08:35:06.511275+0000","last_fullsized":"2024-02-10T08:35:06.511275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_deep_scrub":"74'9","last_deep_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.421866+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774351+0000","last_change":"2024-02-10T08:34:41.695373+0000","last_active":"2024-02-10T08:34:56.774351+0000","last_peered":"2024-02-10T08:34:56.774351+0000","last_clean":"2024-02-10T08:34:56.774351+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:34:56.774351+0000","last_undegraded":"2024-02-10T08:34:56.774351+0000","last_fullsized":"2024-02-10T08:34:56.774351+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.661214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.661214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698966+0000","last_change":"2024-02-10T08:34:43.680148+0000","last_active":"2024-02-10T08:34:56.698966+0000","last_peered":"2024-02-10T08:34:56.698966+0000","last_clean":"2024-02-10T08:34:56.698966+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:34:56.698966+0000","last_undegraded":"2024-02-10T08:34:56.698966+0000","last_fullsized":"2024-02-10T08:34:56.698966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:43.680073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:43.680073+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774809+0000","last_change":"2024-02-10T08:34:32.620154+0000","last_active":"2024-02-10T08:34:56.774809+0000","last_peered":"2024-02-10T08:34:56.774809+0000","last_clean":"2024-02-10T08:34:56.774809+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:34:56.774809+0000","last_undegraded":"2024-02-10T08:34:56.774809+0000","last_fullsized":"2024-02-10T08:34:56.774809+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:32.620086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.620086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:58.971723+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:34:58.971723+0000","last_peered":"2024-02-10T08:34:58.971723+0000","last_clean":"2024-02-10T08:34:58.971723+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:34:58.971723+0000","last_undegraded":"2024-02-10T08:34:58.971723+0000","last_fullsized":"2024-02-10T08:34:58.971723+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613515+0000","last_change":"2024-02-10T08:34:31.359650+0000","last_active":"2024-02-10T08:34:56.613515+0000","last_peered":"2024-02-10T08:34:56.613515+0000","last_clean":"2024-02-10T08:34:56.613515+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:34:56.613515+0000","last_undegraded":"2024-02-10T08:34:56.613515+0000","last_fullsized":"2024-02-10T08:34:56.613515+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.542445+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:35:11.542445+0000","last_peered":"2024-02-10T08:35:11.542445+0000","last_clean":"2024-02-10T08:35:11.542445+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:35:11.542445+0000","last_undegraded":"2024-02-10T08:35:11.542445+0000","last_fullsized":"2024-02-10T08:35:11.542445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774701+0000","last_change":"2024-02-10T08:34:35.819857+0000","last_active":"2024-02-10T08:34:56.774701+0000","last_peered":"2024-02-10T08:34:56.774701+0000","last_clean":"2024-02-10T08:34:56.774701+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:34:56.774701+0000","last_undegraded":"2024-02-10T08:34:56.774701+0000","last_fullsized":"2024-02-10T08:34:56.774701+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.819777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.819777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612685+0000","last_change":"2024-02-10T08:34:55.286635+0000","last_active":"2024-02-10T08:34:56.612685+0000","last_peered":"2024-02-10T08:34:56.612685+0000","last_clean":"2024-02-10T08:34:56.612685+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:34:56.612685+0000","last_undegraded":"2024-02-10T08:34:56.612685+0000","last_fullsized":"2024-02-10T08:34:56.612685+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.286540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.286540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775137+0000","last_change":"2024-02-10T08:34:28.590439+0000","last_active":"2024-02-10T08:34:56.775137+0000","last_peered":"2024-02-10T08:34:56.775137+0000","last_clean":"2024-02-10T08:34:56.775137+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:34:56.775137+0000","last_undegraded":"2024-02-10T08:34:56.775137+0000","last_fullsized":"2024-02-10T08:34:56.775137+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.590362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.590362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":41,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774395+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:34:56.774395+0000","last_peered":"2024-02-10T08:34:56.774395+0000","last_clean":"2024-02-10T08:34:56.774395+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:34:56.774395+0000","last_undegraded":"2024-02-10T08:34:56.774395+0000","last_fullsized":"2024-02-10T08:34:56.774395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774274+0000","last_change":"2024-02-10T08:34:28.605230+0000","last_active":"2024-02-10T08:34:56.774274+0000","last_peered":"2024-02-10T08:34:56.774274+0000","last_clean":"2024-02-10T08:34:56.774274+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:34:56.774274+0000","last_undegraded":"2024-02-10T08:34:56.774274+0000","last_fullsized":"2024-02-10T08:34:56.774274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.605166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.605166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.105717+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:35:02.105717+0000","last_peered":"2024-02-10T08:35:02.105717+0000","last_clean":"2024-02-10T08:35:02.105717+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:35:02.105717+0000","last_undegraded":"2024-02-10T08:35:02.105717+0000","last_fullsized":"2024-02-10T08:35:02.105717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614392+0000","last_change":"2024-02-10T08:34:36.300831+0000","last_active":"2024-02-10T08:34:56.614392+0000","last_peered":"2024-02-10T08:34:56.614392+0000","last_clean":"2024-02-10T08:34:56.614392+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:34:56.614392+0000","last_undegraded":"2024-02-10T08:34:56.614392+0000","last_fullsized":"2024-02-10T08:34:56.614392+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:36.300770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.300770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774801+0000","last_change":"2024-02-10T08:33:20.118729+0000","last_active":"2024-02-10T08:34:56.774801+0000","last_peered":"2024-02-10T08:34:56.774801+0000","last_clean":"2024-02-10T08:34:56.774801+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:34:56.774801+0000","last_undegraded":"2024-02-10T08:34:56.774801+0000","last_fullsized":"2024-02-10T08:34:56.774801+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:33:19.093397+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697937+0000","last_change":"2024-02-10T08:34:48.691828+0000","last_active":"2024-02-10T08:34:56.697937+0000","last_peered":"2024-02-10T08:34:56.697937+0000","last_clean":"2024-02-10T08:34:56.697937+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:34:56.697937+0000","last_undegraded":"2024-02-10T08:34:56.697937+0000","last_fullsized":"2024-02-10T08:34:56.697937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.691737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.691737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774730+0000","last_change":"2024-02-10T08:34:27.584434+0000","last_active":"2024-02-10T08:34:56.774730+0000","last_peered":"2024-02-10T08:34:56.774730+0000","last_clean":"2024-02-10T08:34:56.774730+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:34:56.774730+0000","last_undegraded":"2024-02-10T08:34:56.774730+0000","last_fullsized":"2024-02-10T08:34:56.774730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:27.584369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.584369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612971+0000","last_change":"2024-02-10T08:34:23.544050+0000","last_active":"2024-02-10T08:34:56.612971+0000","last_peered":"2024-02-10T08:34:56.612971+0000","last_clean":"2024-02-10T08:34:56.612971+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:34:56.612971+0000","last_undegraded":"2024-02-10T08:34:56.612971+0000","last_fullsized":"2024-02-10T08:34:56.612971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.543984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.543984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":56,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613529+0000","last_change":"2024-02-10T08:34:47.420238+0000","last_active":"2024-02-10T08:34:56.613529+0000","last_peered":"2024-02-10T08:34:56.613529+0000","last_clean":"2024-02-10T08:34:56.613529+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:34:56.613529+0000","last_undegraded":"2024-02-10T08:34:56.613529+0000","last_fullsized":"2024-02-10T08:34:56.613529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.420174+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776330+0000","last_change":"2024-02-10T08:33:24.119570+0000","last_active":"2024-02-10T08:34:56.776330+0000","last_peered":"2024-02-10T08:34:56.776330+0000","last_clean":"2024-02-10T08:34:56.776330+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:34:56.776330+0000","last_undegraded":"2024-02-10T08:34:56.776330+0000","last_fullsized":"2024-02-10T08:34:56.776330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511188+0000","last_change":"2024-02-10T08:34:51.497308+0000","last_active":"2024-02-10T08:35:06.511188+0000","last_peered":"2024-02-10T08:35:06.511188+0000","last_clean":"2024-02-10T08:35:06.511188+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:35:06.511188+0000","last_undegraded":"2024-02-10T08:35:06.511188+0000","last_fullsized":"2024-02-10T08:35:06.511188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.497242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.497242+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698419+0000","last_change":"2024-02-10T08:34:35.635203+0000","last_active":"2024-02-10T08:34:56.698419+0000","last_peered":"2024-02-10T08:34:56.698419+0000","last_clean":"2024-02-10T08:34:56.698419+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:34:56.698419+0000","last_undegraded":"2024-02-10T08:34:56.698419+0000","last_fullsized":"2024-02-10T08:34:56.698419+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.635108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.635108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:13.494784+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:35:13.494784+0000","last_peered":"2024-02-10T08:35:13.494784+0000","last_clean":"2024-02-10T08:35:13.494784+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:35:13.494784+0000","last_undegraded":"2024-02-10T08:35:13.494784+0000","last_fullsized":"2024-02-10T08:35:13.494784+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.705789+0000","last_change":"2024-02-10T08:35:02.705789+0000","last_active":"2024-02-10T08:35:02.705789+0000","last_peered":"2024-02-10T08:35:02.705789+0000","last_clean":"2024-02-10T08:35:02.705789+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:35:02.705789+0000","last_undegraded":"2024-02-10T08:35:02.705789+0000","last_fullsized":"2024-02-10T08:35:02.705789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.705679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.705679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511297+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:35:06.511297+0000","last_peered":"2024-02-10T08:35:06.511297+0000","last_clean":"2024-02-10T08:35:06.511297+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:35:06.511297+0000","last_undegraded":"2024-02-10T08:35:06.511297+0000","last_fullsized":"2024-02-10T08:35:06.511297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774530+0000","last_change":"2024-02-10T08:34:29.584600+0000","last_active":"2024-02-10T08:34:56.774530+0000","last_peered":"2024-02-10T08:34:56.774530+0000","last_clean":"2024-02-10T08:34:56.774530+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:34:56.774530+0000","last_undegraded":"2024-02-10T08:34:56.774530+0000","last_fullsized":"2024-02-10T08:34:56.774530+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.584529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.584529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774735+0000","last_change":"2024-02-10T08:34:31.663950+0000","last_active":"2024-02-10T08:34:56.774735+0000","last_peered":"2024-02-10T08:34:56.774735+0000","last_clean":"2024-02-10T08:34:56.774735+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:34:56.774735+0000","last_undegraded":"2024-02-10T08:34:56.774735+0000","last_fullsized":"2024-02-10T08:34:56.774735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:31.663882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.663882+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:10.031019+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:35:10.031019+0000","last_peered":"2024-02-10T08:35:10.031019+0000","last_clean":"2024-02-10T08:35:10.031019+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:35:10.031019+0000","last_undegraded":"2024-02-10T08:35:10.031019+0000","last_fullsized":"2024-02-10T08:35:10.031019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775051+0000","last_change":"2024-02-10T08:34:45.607331+0000","last_active":"2024-02-10T08:34:56.775051+0000","last_peered":"2024-02-10T08:34:56.775051+0000","last_clean":"2024-02-10T08:34:56.775051+0000","last_became_active":"2024-02-10T08:33:22.121591+0000","last_became_peered":"2024-02-10T08:33:22.121591+0000","last_unstale":"2024-02-10T08:34:56.775051+0000","last_undegraded":"2024-02-10T08:34:56.775051+0000","last_fullsized":"2024-02-10T08:34:56.775051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.607254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.607254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612786+0000","last_change":"2024-02-10T08:34:29.402936+0000","last_active":"2024-02-10T08:34:56.612786+0000","last_peered":"2024-02-10T08:34:56.612786+0000","last_clean":"2024-02-10T08:34:56.612786+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:34:56.612786+0000","last_undegraded":"2024-02-10T08:34:56.612786+0000","last_fullsized":"2024-02-10T08:34:56.612786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.402874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.402874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775197+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:34:56.775197+0000","last_peered":"2024-02-10T08:34:56.775197+0000","last_clean":"2024-02-10T08:34:56.775197+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:34:56.775197+0000","last_undegraded":"2024-02-10T08:34:56.775197+0000","last_fullsized":"2024-02-10T08:34:56.775197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:07.725134+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:35:07.725134+0000","last_peered":"2024-02-10T08:35:07.725134+0000","last_clean":"2024-02-10T08:35:07.725134+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:35:07.725134+0000","last_undegraded":"2024-02-10T08:35:07.725134+0000","last_fullsized":"2024-02-10T08:35:07.725134+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.446706+0000","last_change":"2024-02-10T08:35:02.446706+0000","last_active":"2024-02-10T08:35:02.446706+0000","last_peered":"2024-02-10T08:35:02.446706+0000","last_clean":"2024-02-10T08:35:02.446706+0000","last_became_active":"2024-02-10T08:33:22.118323+0000","last_became_peered":"2024-02-10T08:33:22.118323+0000","last_unstale":"2024-02-10T08:35:02.446706+0000","last_undegraded":"2024-02-10T08:35:02.446706+0000","last_fullsized":"2024-02-10T08:35:02.446706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.699547+0000","last_change":"2024-02-10T08:34:56.699511+0000","last_active":"2024-02-10T08:34:56.699547+0000","last_peered":"2024-02-10T08:34:56.699547+0000","last_clean":"2024-02-10T08:34:56.699547+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:34:56.699547+0000","last_undegraded":"2024-02-10T08:34:56.699547+0000","last_fullsized":"2024-02-10T08:34:56.699547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.699420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.699420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'9","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698876+0000","last_change":"2024-02-10T08:34:23.674473+0000","last_active":"2024-02-10T08:34:56.698876+0000","last_peered":"2024-02-10T08:34:56.698876+0000","last_clean":"2024-02-10T08:34:56.698876+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:34:56.698876+0000","last_undegraded":"2024-02-10T08:34:56.698876+0000","last_fullsized":"2024-02-10T08:34:56.698876+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:23.674404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.674404+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774791+0000","last_change":"2024-02-10T08:34:45.519207+0000","last_active":"2024-02-10T08:34:56.774791+0000","last_peered":"2024-02-10T08:34:56.774791+0000","last_clean":"2024-02-10T08:34:56.774791+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:34:56.774791+0000","last_undegraded":"2024-02-10T08:34:56.774791+0000","last_fullsized":"2024-02-10T08:34:56.774791+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.519116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.519116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.136470+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:35:00.136470+0000","last_peered":"2024-02-10T08:35:00.136470+0000","last_clean":"2024-02-10T08:35:00.136470+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:35:00.136470+0000","last_undegraded":"2024-02-10T08:35:00.136470+0000","last_fullsized":"2024-02-10T08:35:00.136470+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776217+0000","last_change":"2024-02-10T08:34:54.039656+0000","last_active":"2024-02-10T08:34:56.776217+0000","last_peered":"2024-02-10T08:34:56.776217+0000","last_clean":"2024-02-10T08:34:56.776217+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:34:56.776217+0000","last_undegraded":"2024-02-10T08:34:56.776217+0000","last_fullsized":"2024-02-10T08:34:56.776217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.039584+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612415+0000","last_change":"2024-02-10T08:34:39.590686+0000","last_active":"2024-02-10T08:34:56.612415+0000","last_peered":"2024-02-10T08:34:56.612415+0000","last_clean":"2024-02-10T08:34:56.612415+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:34:56.612415+0000","last_undegraded":"2024-02-10T08:34:56.612415+0000","last_fullsized":"2024-02-10T08:34:56.612415+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.590604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.590604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697916+0000","last_change":"2024-02-10T08:34:31.544112+0000","last_active":"2024-02-10T08:34:56.697916+0000","last_peered":"2024-02-10T08:34:56.697916+0000","last_clean":"2024-02-10T08:34:56.697916+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:34:56.697916+0000","last_undegraded":"2024-02-10T08:34:56.697916+0000","last_fullsized":"2024-02-10T08:34:56.697916+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.544049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.544049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612915+0000","last_change":"2024-02-10T08:34:46.452573+0000","last_active":"2024-02-10T08:34:56.612915+0000","last_peered":"2024-02-10T08:34:56.612915+0000","last_clean":"2024-02-10T08:34:56.612915+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:34:56.612915+0000","last_undegraded":"2024-02-10T08:34:56.612915+0000","last_fullsized":"2024-02-10T08:34:56.612915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:46.452504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:46.452504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'10","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613672+0000","last_change":"2024-02-10T08:34:33.231522+0000","last_active":"2024-02-10T08:34:56.613672+0000","last_peered":"2024-02-10T08:34:56.613672+0000","last_clean":"2024-02-10T08:34:56.613672+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:34:56.613672+0000","last_undegraded":"2024-02-10T08:34:56.613672+0000","last_fullsized":"2024-02-10T08:34:56.613672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:33.231411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.231411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511185+0000","last_change":"2024-02-10T08:34:47.693139+0000","last_active":"2024-02-10T08:35:06.511185+0000","last_peered":"2024-02-10T08:35:06.511185+0000","last_clean":"2024-02-10T08:35:06.511185+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:35:06.511185+0000","last_undegraded":"2024-02-10T08:35:06.511185+0000","last_fullsized":"2024-02-10T08:35:06.511185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:47.693030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.693030+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:04.456788+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:35:04.456788+0000","last_peered":"2024-02-10T08:35:04.456788+0000","last_clean":"2024-02-10T08:35:04.456788+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:35:04.456788+0000","last_undegraded":"2024-02-10T08:35:04.456788+0000","last_fullsized":"2024-02-10T08:35:04.456788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774621+0000","last_change":"2024-02-10T08:34:20.652012+0000","last_active":"2024-02-10T08:34:56.774621+0000","last_peered":"2024-02-10T08:34:56.774621+0000","last_clean":"2024-02-10T08:34:56.774621+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:34:56.774621+0000","last_undegraded":"2024-02-10T08:34:56.774621+0000","last_fullsized":"2024-02-10T08:34:56.774621+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.651946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.651946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511270+0000","last_change":"2024-02-10T08:34:47.549753+0000","last_active":"2024-02-10T08:35:06.511270+0000","last_peered":"2024-02-10T08:35:06.511270+0000","last_clean":"2024-02-10T08:35:06.511270+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:35:06.511270+0000","last_undegraded":"2024-02-10T08:35:06.511270+0000","last_fullsized":"2024-02-10T08:35:06.511270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:34:47.549671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.549671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.639772+0000","last_change":"2024-02-10T08:35:00.639772+0000","last_active":"2024-02-10T08:35:00.639772+0000","last_peered":"2024-02-10T08:35:00.639772+0000","last_clean":"2024-02-10T08:35:00.639772+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:35:00.639772+0000","last_undegraded":"2024-02-10T08:35:00.639772+0000","last_fullsized":"2024-02-10T08:35:00.639772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.639705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.639705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775933+0000","last_change":"2024-02-10T08:34:51.910276+0000","last_active":"2024-02-10T08:34:56.775933+0000","last_peered":"2024-02-10T08:34:56.775933+0000","last_clean":"2024-02-10T08:34:56.775933+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:34:56.775933+0000","last_undegraded":"2024-02-10T08:34:56.775933+0000","last_fullsized":"2024-02-10T08:34:56.775933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.910184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":153,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:14.260052+0000","last_change":"2024-02-10T08:35:07.227647+0000","last_active":"2024-02-10T08:35:14.260052+0000","last_peered":"2024-02-10T08:35:14.260052+0000","last_clean":"2024-02-10T08:35:14.260052+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:35:14.260052+0000","last_undegraded":"2024-02-10T08:35:14.260052+0000","last_fullsized":"2024-02-10T08:35:14.260052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.227587+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612586+0000","last_change":"2024-02-10T08:34:52.698430+0000","last_active":"2024-02-10T08:34:56.612586+0000","last_peered":"2024-02-10T08:34:56.612586+0000","last_clean":"2024-02-10T08:34:56.612586+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:34:56.612586+0000","last_undegraded":"2024-02-10T08:34:56.612586+0000","last_fullsized":"2024-02-10T08:34:56.612586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.698329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":63,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614951+0000","last_change":"2024-02-10T08:34:27.278457+0000","last_active":"2024-02-10T08:34:56.614951+0000","last_peered":"2024-02-10T08:34:56.614951+0000","last_clean":"2024-02-10T08:34:56.614951+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:34:56.614951+0000","last_undegraded":"2024-02-10T08:34:56.614951+0000","last_fullsized":"2024-02-10T08:34:56.614951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:34:27.278388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.278388+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.720849+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:35:08.720849+0000","last_peered":"2024-02-10T08:35:08.720849+0000","last_clean":"2024-02-10T08:35:08.720849+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:35:08.720849+0000","last_undegraded":"2024-02-10T08:35:08.720849+0000","last_fullsized":"2024-02-10T08:35:08.720849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775867+0000","last_change":"2024-02-10T08:34:55.443465+0000","last_active":"2024-02-10T08:34:56.775867+0000","last_peered":"2024-02-10T08:34:56.775867+0000","last_clean":"2024-02-10T08:34:56.775867+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:34:56.775867+0000","last_undegraded":"2024-02-10T08:34:56.775867+0000","last_fullsized":"2024-02-10T08:34:56.775867+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.443390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.443390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774586+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:34:56.774586+0000","last_peered":"2024-02-10T08:34:56.774586+0000","last_clean":"2024-02-10T08:34:56.774586+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:34:56.774586+0000","last_undegraded":"2024-02-10T08:34:56.774586+0000","last_fullsized":"2024-02-10T08:34:56.774586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775158+0000","last_change":"2024-02-10T08:34:52.491498+0000","last_active":"2024-02-10T08:34:56.775158+0000","last_peered":"2024-02-10T08:34:56.775158+0000","last_clean":"2024-02-10T08:34:56.775158+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:34:56.775158+0000","last_undegraded":"2024-02-10T08:34:56.775158+0000","last_fullsized":"2024-02-10T08:34:56.775158+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.491420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.491420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776453+0000","last_change":"2024-02-10T08:34:48.841104+0000","last_active":"2024-02-10T08:34:56.776453+0000","last_peered":"2024-02-10T08:34:56.776453+0000","last_clean":"2024-02-10T08:34:56.776453+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:34:56.776453+0000","last_undegraded":"2024-02-10T08:34:56.776453+0000","last_fullsized":"2024-02-10T08:34:56.776453+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.841025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.841025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613175+0000","last_change":"2024-02-10T08:34:39.378393+0000","last_active":"2024-02-10T08:34:56.613175+0000","last_peered":"2024-02-10T08:34:56.613175+0000","last_clean":"2024-02-10T08:34:56.613175+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:34:56.613175+0000","last_undegraded":"2024-02-10T08:34:56.613175+0000","last_fullsized":"2024-02-10T08:34:56.613175+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:39.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.378312+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775424+0000","last_change":"2024-02-10T08:34:40.809042+0000","last_active":"2024-02-10T08:34:56.775424+0000","last_peered":"2024-02-10T08:34:56.775424+0000","last_clean":"2024-02-10T08:34:56.775424+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:34:56.775424+0000","last_undegraded":"2024-02-10T08:34:56.775424+0000","last_fullsized":"2024-02-10T08:34:56.775424+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:40.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:40.808978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":33,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612648+0000","last_change":"2024-02-10T08:34:28.307074+0000","last_active":"2024-02-10T08:34:56.612648+0000","last_peered":"2024-02-10T08:34:56.612648+0000","last_clean":"2024-02-10T08:34:56.612648+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:34:56.612648+0000","last_undegraded":"2024-02-10T08:34:56.612648+0000","last_fullsized":"2024-02-10T08:34:56.612648+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:34:28.306997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.306997+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698173+0000","last_change":"2024-02-10T08:34:21.731002+0000","last_active":"2024-02-10T08:34:56.698173+0000","last_peered":"2024-02-10T08:34:56.698173+0000","last_clean":"2024-02-10T08:34:56.698173+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:34:56.698173+0000","last_undegraded":"2024-02-10T08:34:56.698173+0000","last_fullsized":"2024-02-10T08:34:56.698173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.730937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.730937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776370+0000","last_change":"2024-02-10T08:34:25.707975+0000","last_active":"2024-02-10T08:34:56.776370+0000","last_peered":"2024-02-10T08:34:56.776370+0000","last_clean":"2024-02-10T08:34:56.776370+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:34:56.776370+0000","last_undegraded":"2024-02-10T08:34:56.776370+0000","last_fullsized":"2024-02-10T08:34:56.776370+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:25.707852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.707852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698746+0000","last_change":"2024-02-10T08:34:28.599042+0000","last_active":"2024-02-10T08:34:56.698746+0000","last_peered":"2024-02-10T08:34:56.698746+0000","last_clean":"2024-02-10T08:34:56.698746+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:34:56.698746+0000","last_undegraded":"2024-02-10T08:34:56.698746+0000","last_fullsized":"2024-02-10T08:34:56.698746+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:28.598979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.598979+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775788+0000","last_change":"2024-02-10T08:34:38.628666+0000","last_active":"2024-02-10T08:34:56.775788+0000","last_peered":"2024-02-10T08:34:56.775788+0000","last_clean":"2024-02-10T08:34:56.775788+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:34:56.775788+0000","last_undegraded":"2024-02-10T08:34:56.775788+0000","last_fullsized":"2024-02-10T08:34:56.775788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.628559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.628559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613054+0000","last_change":"2024-02-10T08:34:44.434673+0000","last_active":"2024-02-10T08:34:56.613054+0000","last_peered":"2024-02-10T08:34:56.613054+0000","last_clean":"2024-02-10T08:34:56.613054+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:34:56.613054+0000","last_undegraded":"2024-02-10T08:34:56.613054+0000","last_fullsized":"2024-02-10T08:34:56.613054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.434603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.434603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776019+0000","last_change":"2024-02-10T08:34:44.810962+0000","last_active":"2024-02-10T08:34:56.776019+0000","last_peered":"2024-02-10T08:34:56.776019+0000","last_clean":"2024-02-10T08:34:56.776019+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:34:56.776019+0000","last_undegraded":"2024-02-10T08:34:56.776019+0000","last_fullsized":"2024-02-10T08:34:56.776019+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.810886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.810886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":71,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698301+0000","last_change":"2024-02-10T08:34:39.703715+0000","last_active":"2024-02-10T08:34:56.698301+0000","last_peered":"2024-02-10T08:34:56.698301+0000","last_clean":"2024-02-10T08:34:56.698301+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:34:56.698301+0000","last_undegraded":"2024-02-10T08:34:56.698301+0000","last_fullsized":"2024-02-10T08:34:56.698301+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:39.703643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.703643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613055+0000","last_change":"2024-02-10T08:34:51.348414+0000","last_active":"2024-02-10T08:34:56.613055+0000","last_peered":"2024-02-10T08:34:56.613055+0000","last_clean":"2024-02-10T08:34:56.613055+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:34:56.613055+0000","last_undegraded":"2024-02-10T08:34:56.613055+0000","last_fullsized":"2024-02-10T08:34:56.613055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.348334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.348334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776782+0000","last_change":"2024-02-10T08:33:24.119911+0000","last_active":"2024-02-10T08:34:56.776782+0000","last_peered":"2024-02-10T08:34:56.776782+0000","last_clean":"2024-02-10T08:34:56.776782+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:34:56.776782+0000","last_undegraded":"2024-02-10T08:34:56.776782+0000","last_fullsized":"2024-02-10T08:34:56.776782+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:03.147093+0000","last_change":"2024-02-10T08:35:03.147093+0000","last_active":"2024-02-10T08:35:03.147093+0000","last_peered":"2024-02-10T08:35:03.147093+0000","last_clean":"2024-02-10T08:35:03.147093+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:35:03.147093+0000","last_undegraded":"2024-02-10T08:35:03.147093+0000","last_fullsized":"2024-02-10T08:35:03.147093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:35:03.146990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.146990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:09.492086+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:35:09.492086+0000","last_peered":"2024-02-10T08:35:09.492086+0000","last_clean":"2024-02-10T08:35:09.492086+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:35:09.492086+0000","last_undegraded":"2024-02-10T08:35:09.492086+0000","last_fullsized":"2024-02-10T08:35:09.492086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:58.059779+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:34:58.059779+0000","last_peered":"2024-02-10T08:34:58.059779+0000","last_clean":"2024-02-10T08:34:58.059779+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:34:58.059779+0000","last_undegraded":"2024-02-10T08:34:58.059779+0000","last_fullsized":"2024-02-10T08:34:58.059779+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.025052+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:35:11.025052+0000","last_peered":"2024-02-10T08:35:11.025052+0000","last_clean":"2024-02-10T08:35:11.025052+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:35:11.025052+0000","last_undegraded":"2024-02-10T08:35:11.025052+0000","last_fullsized":"2024-02-10T08:35:11.025052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613510+0000","last_change":"2024-02-10T08:34:22.584930+0000","last_active":"2024-02-10T08:34:56.613510+0000","last_peered":"2024-02-10T08:34:56.613510+0000","last_clean":"2024-02-10T08:34:56.613510+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:34:56.613510+0000","last_undegraded":"2024-02-10T08:34:56.613510+0000","last_fullsized":"2024-02-10T08:34:56.613510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:22.584856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:22.584856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776119+0000","last_change":"2024-02-10T08:34:36.155885+0000","last_active":"2024-02-10T08:34:56.776119+0000","last_peered":"2024-02-10T08:34:56.776119+0000","last_clean":"2024-02-10T08:34:56.776119+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:34:56.776119+0000","last_undegraded":"2024-02-10T08:34:56.776119+0000","last_fullsized":"2024-02-10T08:34:56.776119+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:36.155806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.155806+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511177+0000","last_change":"2024-02-10T08:34:36.649251+0000","last_active":"2024-02-10T08:35:06.511177+0000","last_peered":"2024-02-10T08:35:06.511177+0000","last_clean":"2024-02-10T08:35:06.511177+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:35:06.511177+0000","last_undegraded":"2024-02-10T08:35:06.511177+0000","last_fullsized":"2024-02-10T08:35:06.511177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:36.649186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.649186+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774728+0000","last_change":"2024-02-10T08:33:24.126537+0000","last_active":"2024-02-10T08:34:56.774728+0000","last_peered":"2024-02-10T08:34:56.774728+0000","last_clean":"2024-02-10T08:34:56.774728+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:34:56.774728+0000","last_undegraded":"2024-02-10T08:34:56.774728+0000","last_fullsized":"2024-02-10T08:34:56.774728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.520594+0000","last_change":"2024-02-10T08:35:06.520594+0000","last_active":"2024-02-10T08:35:06.520594+0000","last_peered":"2024-02-10T08:35:06.520594+0000","last_clean":"2024-02-10T08:35:06.520594+0000","last_became_active":"2024-02-10T08:33:22.122087+0000","last_became_peered":"2024-02-10T08:33:22.122087+0000","last_unstale":"2024-02-10T08:35:06.520594+0000","last_undegraded":"2024-02-10T08:35:06.520594+0000","last_fullsized":"2024-02-10T08:35:06.520594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776508+0000","last_change":"2024-02-10T08:34:29.780708+0000","last_active":"2024-02-10T08:34:56.776508+0000","last_peered":"2024-02-10T08:34:56.776508+0000","last_clean":"2024-02-10T08:34:56.776508+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:34:56.776508+0000","last_undegraded":"2024-02-10T08:34:56.776508+0000","last_fullsized":"2024-02-10T08:34:56.776508+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.780644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.780644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776948+0000","last_change":"2024-02-10T08:34:25.085734+0000","last_active":"2024-02-10T08:34:56.776948+0000","last_peered":"2024-02-10T08:34:56.776948+0000","last_clean":"2024-02-10T08:34:56.776948+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:34:56.776948+0000","last_undegraded":"2024-02-10T08:34:56.776948+0000","last_fullsized":"2024-02-10T08:34:56.776948+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:25.085663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.085663+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775352+0000","last_change":"2024-02-10T08:33:24.122991+0000","last_active":"2024-02-10T08:34:56.775352+0000","last_peered":"2024-02-10T08:34:56.775352+0000","last_clean":"2024-02-10T08:34:56.775352+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:34:56.775352+0000","last_undegraded":"2024-02-10T08:34:56.775352+0000","last_fullsized":"2024-02-10T08:34:56.775352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.027015+0000","last_change":"2024-02-10T08:35:11.026984+0000","last_active":"2024-02-10T08:35:11.027015+0000","last_peered":"2024-02-10T08:35:11.027015+0000","last_clean":"2024-02-10T08:35:11.027015+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:35:11.027015+0000","last_undegraded":"2024-02-10T08:35:11.027015+0000","last_fullsized":"2024-02-10T08:35:11.027015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613242+0000","last_change":"2024-02-10T08:34:32.208396+0000","last_active":"2024-02-10T08:34:56.613242+0000","last_peered":"2024-02-10T08:34:56.613242+0000","last_clean":"2024-02-10T08:34:56.613242+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:34:56.613242+0000","last_undegraded":"2024-02-10T08:34:56.613242+0000","last_fullsized":"2024-02-10T08:34:56.613242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:34:32.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.208313+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.190561+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:35:11.190561+0000","last_peered":"2024-02-10T08:35:11.190561+0000","last_clean":"2024-02-10T08:35:11.190561+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:35:11.190561+0000","last_undegraded":"2024-02-10T08:35:11.190561+0000","last_fullsized":"2024-02-10T08:35:11.190561+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776817+0000","last_change":"2024-02-10T08:34:35.201454+0000","last_active":"2024-02-10T08:34:56.776817+0000","last_peered":"2024-02-10T08:34:56.776817+0000","last_clean":"2024-02-10T08:34:56.776817+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:34:56.776817+0000","last_undegraded":"2024-02-10T08:34:56.776817+0000","last_fullsized":"2024-02-10T08:34:56.776817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.201371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.201371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511180+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:35:06.511180+0000","last_peered":"2024-02-10T08:35:06.511180+0000","last_clean":"2024-02-10T08:35:06.511180+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:35:06.511180+0000","last_undegraded":"2024-02-10T08:35:06.511180+0000","last_fullsized":"2024-02-10T08:35:06.511180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613457+0000","last_change":"2024-02-10T08:34:24.362389+0000","last_active":"2024-02-10T08:34:56.613457+0000","last_peered":"2024-02-10T08:34:56.613457+0000","last_clean":"2024-02-10T08:34:56.613457+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:34:56.613457+0000","last_undegraded":"2024-02-10T08:34:56.613457+0000","last_fullsized":"2024-02-10T08:34:56.613457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.362309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.362309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775026+0000","last_change":"2024-02-10T08:34:21.123617+0000","last_active":"2024-02-10T08:34:56.775026+0000","last_peered":"2024-02-10T08:34:56.775026+0000","last_clean":"2024-02-10T08:34:56.775026+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:34:56.775026+0000","last_undegraded":"2024-02-10T08:34:56.775026+0000","last_fullsized":"2024-02-10T08:34:56.775026+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:21.123548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.123548+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613294+0000","last_change":"2024-02-10T08:34:53.743882+0000","last_active":"2024-02-10T08:34:56.613294+0000","last_peered":"2024-02-10T08:34:56.613294+0000","last_clean":"2024-02-10T08:34:56.613294+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:34:56.613294+0000","last_undegraded":"2024-02-10T08:34:56.613294+0000","last_fullsized":"2024-02-10T08:34:56.613294+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.743807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.743807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.227030+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:35:08.227030+0000","last_peered":"2024-02-10T08:35:08.227030+0000","last_clean":"2024-02-10T08:35:08.227030+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:35:08.227030+0000","last_undegraded":"2024-02-10T08:35:08.227030+0000","last_fullsized":"2024-02-10T08:35:08.227030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774624+0000","last_change":"2024-02-10T08:34:20.776146+0000","last_active":"2024-02-10T08:34:56.774624+0000","last_peered":"2024-02-10T08:34:56.774624+0000","last_clean":"2024-02-10T08:34:56.774624+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:34:56.774624+0000","last_undegraded":"2024-02-10T08:34:56.774624+0000","last_fullsized":"2024-02-10T08:34:56.774624+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.776052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.776052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775698+0000","last_change":"2024-02-10T08:34:24.118004+0000","last_active":"2024-02-10T08:34:56.775698+0000","last_peered":"2024-02-10T08:34:56.775698+0000","last_clean":"2024-02-10T08:34:56.775698+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:34:56.775698+0000","last_undegraded":"2024-02-10T08:34:56.775698+0000","last_fullsized":"2024-02-10T08:34:56.775698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:34:24.117909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.117909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:13.030970+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:35:13.030970+0000","last_peered":"2024-02-10T08:35:13.030970+0000","last_clean":"2024-02-10T08:35:13.030970+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:35:13.030970+0000","last_undegraded":"2024-02-10T08:35:13.030970+0000","last_fullsized":"2024-02-10T08:35:13.030970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612289+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:34:56.612289+0000","last_peered":"2024-02-10T08:34:56.612289+0000","last_clean":"2024-02-10T08:34:56.612289+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:34:56.612289+0000","last_undegraded":"2024-02-10T08:34:56.612289+0000","last_fullsized":"2024-02-10T08:34:56.612289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613140+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:34:56.613140+0000","last_peered":"2024-02-10T08:34:56.613140+0000","last_clean":"2024-02-10T08:34:56.613140+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:34:56.613140+0000","last_undegraded":"2024-02-10T08:34:56.613140+0000","last_fullsized":"2024-02-10T08:34:56.613140+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":67,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612850+0000","last_change":"2024-02-10T08:34:44.381291+0000","last_active":"2024-02-10T08:34:56.612850+0000","last_peered":"2024-02-10T08:34:56.612850+0000","last_clean":"2024-02-10T08:34:56.612850+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:34:56.612850+0000","last_undegraded":"2024-02-10T08:34:56.612850+0000","last_fullsized":"2024-02-10T08:34:56.612850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:44.381208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.381208+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697976+0000","last_change":"2024-02-10T08:34:29.554010+0000","last_active":"2024-02-10T08:34:56.697976+0000","last_peered":"2024-02-10T08:34:56.697976+0000","last_clean":"2024-02-10T08:34:56.697976+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:34:56.697976+0000","last_undegraded":"2024-02-10T08:34:56.697976+0000","last_fullsized":"2024-02-10T08:34:56.697976+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.553914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.553914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612879+0000","last_change":"2024-02-10T08:34:50.747486+0000","last_active":"2024-02-10T08:34:56.612879+0000","last_peered":"2024-02-10T08:34:56.612879+0000","last_clean":"2024-02-10T08:34:56.612879+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:34:56.612879+0000","last_undegraded":"2024-02-10T08:34:56.612879+0000","last_fullsized":"2024-02-10T08:34:56.612879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:50.747424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.747424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614468+0000","last_change":"2024-02-10T08:34:20.413361+0000","last_active":"2024-02-10T08:34:56.614468+0000","last_peered":"2024-02-10T08:34:56.614468+0000","last_clean":"2024-02-10T08:34:56.614468+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:34:56.614468+0000","last_undegraded":"2024-02-10T08:34:56.614468+0000","last_fullsized":"2024-02-10T08:34:56.614468+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.413281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.413281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698426+0000","last_change":"2024-02-10T08:34:41.701563+0000","last_active":"2024-02-10T08:34:56.698426+0000","last_peered":"2024-02-10T08:34:56.698426+0000","last_clean":"2024-02-10T08:34:56.698426+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:34:56.698426+0000","last_undegraded":"2024-02-10T08:34:56.698426+0000","last_fullsized":"2024-02-10T08:34:56.698426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:34.592074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:34.592074+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774432+0000","last_change":"2024-02-10T08:34:56.475601+0000","last_active":"2024-02-10T08:34:56.774432+0000","last_peered":"2024-02-10T08:34:56.774432+0000","last_clean":"2024-02-10T08:34:56.774432+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:34:56.774432+0000","last_undegraded":"2024-02-10T08:34:56.774432+0000","last_fullsized":"2024-02-10T08:34:56.774432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.475529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.475529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:03.720397+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:35:03.720397+0000","last_peered":"2024-02-10T08:35:03.720397+0000","last_clean":"2024-02-10T08:35:03.720397+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:35:03.720397+0000","last_undegraded":"2024-02-10T08:35:03.720397+0000","last_fullsized":"2024-02-10T08:35:03.720397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:05.510749+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:35:05.510749+0000","last_peered":"2024-02-10T08:35:05.510749+0000","last_clean":"2024-02-10T08:35:05.510749+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:35:05.510749+0000","last_undegraded":"2024-02-10T08:35:05.510749+0000","last_fullsized":"2024-02-10T08:35:05.510749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511282+0000","last_change":"2024-02-10T08:34:42.752203+0000","last_active":"2024-02-10T08:35:06.511282+0000","last_peered":"2024-02-10T08:35:06.511282+0000","last_clean":"2024-02-10T08:35:06.511282+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:35:06.511282+0000","last_undegraded":"2024-02-10T08:35:06.511282+0000","last_fullsized":"2024-02-10T08:35:06.511282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:42.752097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:42.752097+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299418,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297908,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447244,"statfs":{"total":95995035648,"available":95689977856,"internally_reserved":0,"allocated":831488,"data_stored":340065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.034,"15min":1.034},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.054,"15min":1.054},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.496,"15min":0.499},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.503},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.747,"15min":0.747},"last":0.41299999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561056,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297888,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447264,"statfs":{"total":95995035648,"available":95689998336,"internally_reserved":0,"allocated":811008,"data_stored":339973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.69199999999999995}]},{"osd":1,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.498,"15min":0.498},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.623,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.501},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.67000000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.375,"15min":0.372},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56000000000000005,"5min":0.623,"15min":0.623},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.59199999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.39600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":341385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.495,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.309},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.47499999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.377,"15min":0.376},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.23200000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53000000000000003,"5min":0.624,"15min":0.624},"last":0.32000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41899999999999998,"15min":0.42899999999999999},"min":{"1min":0.27100000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.53000000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":1.294,"15min":1.294},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77700000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73299999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297900,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447252,"statfs":{"total":95995035648,"available":95689986048,"internally_reserved":0,"allocated":823296,"data_stored":339089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.30299999999999999},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.308},"min":{"1min":0.27300000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.215}]},{"osd":2,"last update":"Sat Feb 10 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36899999999999999,"15min":0.36499999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.498,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.52600000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.249}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.503,"5min":0.624,"15min":0.624},"last":0.308},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.501,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.29399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75900000000000001,"5min":0.878,"15min":0.878},"last":0.38100000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76900000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.441},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.49399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297896,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447256,"statfs":{"total":95995035648,"available":95689990144,"internally_reserved":0,"allocated":819200,"data_stored":339768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35499999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.38500000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.52800000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81899999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.621,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63400000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":340261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35599999999999998,"15min":0.36899999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39600000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.625}]},{"osd":2,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.437,"15min":0.46000000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57999999999999996,"15min":0.60799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66700000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55800000000000005,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.72499999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.64000000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.88200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.85399999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.87}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297912,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447240,"statfs":{"total":95995035648,"available":95689973760,"internally_reserved":0,"allocated":835584,"data_stored":340431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.998,"5min":1,"15min":1},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.749,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.379,"15min":0.371},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.441,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58799999999999997}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297904,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447248,"statfs":{"total":95995035648,"available":95689981952,"internally_reserved":0,"allocated":827392,"data_stored":339407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.505,"15min":0.505},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76000000000000001,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.253,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":1.081,"15min":1.081},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48499999999999999,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.63},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:35:19.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:35:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:19.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:18.013140+0000 mgr.y (mgr.14553) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:19.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:18.417165+0000 mgr.y (mgr.14553) 83 : audit [DBG] from='client.34661 v1:172.21.15.107:0/4285973093' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:18.013140+0000 mgr.y (mgr.14553) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:18.417165+0000 mgr.y (mgr.14553) 83 : audit [DBG] from='client.34661 v1:172.21.15.107:0/4285973093' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:18.013140+0000 mgr.y (mgr.14553) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:18.417165+0000 mgr.y (mgr.14553) 83 : audit [DBG] from='client.34661 v1:172.21.15.107:0/4285973093' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:20.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:19.421165+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-10T08:35:20.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:19.422082+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-10T08:35:20.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:19.862456+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:35:20.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:19.863352+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:35:20.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:19.421165+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-10T08:35:20.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:19.422082+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-10T08:35:20.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:19.862456+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:35:20.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:19.863352+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:35:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:19.421165+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-10T08:35:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:19.422082+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-10T08:35:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:19.862456+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:35:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:19.863352+0000 osd.4 (osd.4) 38 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:35:21.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 -- 172.21.15.107:0/917251473 >> v1:172.21.15.107:6790/0 conn(0x7fd2cc103250 legacy=0x7fd2cc1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:21.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 -- 172.21.15.107:0/917251473 shutdown_connections 2024-02-10T08:35:21.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 -- 172.21.15.107:0/917251473 >> 172.21.15.107:0/917251473 conn(0x7fd2cc0f2510 msgr2=0x7fd2cc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:21.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 -- 172.21.15.107:0/917251473 shutdown_connections 2024-02-10T08:35:21.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 -- 172.21.15.107:0/917251473 wait complete. 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 librados: starting msgr at 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 librados: starting objecter 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.185+0000 7fd2d2118700 1 Processor -- start 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 -- start start 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 librados: setting wanted keys 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 librados: calling monclient init 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2cc1173d0 con 0x7fd2cc0fc910 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2cc1185b0 con 0x7fd2cc103250 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2d2118700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2cc1197b0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2caffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd2cc0ff710 0x7fd2cc0613f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53064/0 (socket says 172.21.15.107:53064) 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2caffd700 1 -- 172.21.15.107:0/454320160 learned_addr learned my addr 172.21.15.107:0/454320160 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:21.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2cb7fe700 1 --1- 172.21.15.107:0/454320160 >> v1:172.21.15.107:6789/0 conn(0x7fd2cc0fc910 0x7fd2cc060ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37340/0 (socket says 172.21.15.107:37340) 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd2cc0ff710 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd2cc0fc910 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701980759 0 0) 0x7fd2cc1197b0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b4003030 con 0x7fd2cc0ff710 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd2cc103250 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085220239 0 0) 0x7fd2cc1173d0 con 0x7fd2cc0fc910 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.186+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc1197b0 con 0x7fd2cc0fc910 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921966920 0 0) 0x7fd2cc1185b0 con 0x7fd2cc103250 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc1173d0 con 0x7fd2cc103250 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849157587 0 0) 0x7fd2b4003030 con 0x7fd2cc0ff710 2024-02-10T08:35:21.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2cc1185b0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd2bc0019f0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975482000 0 0) 0x7fd2cc1197b0 con 0x7fd2cc0fc910 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b4003030 con 0x7fd2cc0fc910 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd2c0001730 con 0x7fd2cc0fc910 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 787881949 0 0) 0x7fd2cc1173d0 con 0x7fd2cc103250 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2cc1197b0 con 0x7fd2cc103250 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd2c4001950 con 0x7fd2cc103250 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975956816 0 0) 0x7fd2cc1185b0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 >> v1:172.21.15.195:6789/0 conn(0x7fd2cc103250 legacy=0x7fd2cc115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 >> v1:172.21.15.107:6789/0 conn(0x7fd2cc0fc910 legacy=0x7fd2cc060ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc11a9b0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc1175c0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2d2118700 10 client.34667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc117a30 con 0x7fd2cc0ff710 2024-02-10T08:35:21.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2d2118700 1 librados: init done 2024-02-10T08:35:21.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd2bc002510 con 0x7fd2cc0ff710 2024-02-10T08:35:21.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.187+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd2bc005b50 con 0x7fd2cc0ff710 2024-02-10T08:35:21.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.188+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8004f20 con 0x7fd2cc0ff710 2024-02-10T08:35:21.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.188+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd2bc002070 con 0x7fd2cc0ff710 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(81..81 src has 30..81) v4 ==== 6172+0+0 (unknown 3238675313 0 0) 0x7fd2bc0785d0 con 0x7fd2cc0ff710 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 10 client.34667.objecter ms_dispatch 0x7fd2cc001550 osd_map(81..81 src has 30..81) v4 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 3 client.34667.objecter handle_osd_map got epochs [81,81] > 0 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 3 client.34667.objecter handle_osd_map decoding full epoch 81 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 20 client.34667.objecter dump_active .. 0 homeless 2024-02-10T08:35:21.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.189+0000 7fd2c8ff9700 10 client.34667.objecter ms_handle_connect 0x7fd2b4063c20 2024-02-10T08:35:21.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.193+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd2bc04c180 con 0x7fd2cc0ff710 2024-02-10T08:35:21.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.351+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2b8000b90 con 0x7fd2b4063c20 2024-02-10T08:35:21.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.360+0000 7fd2c8ff9700 1 -- 172.21.15.107:0/454320160 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357344 (unknown 2965378022 0 1510453276) 0x7fd2b8000b90 con 0x7fd2b4063c20 2024-02-10T08:35:21.366 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:35:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 10 librados: watch_flush enter 2024-02-10T08:35:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 10 librados: watch_flush exit 2024-02-10T08:35:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 20 client.34667.objecter shutdown clearing up homeless session... 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 10 client.34667.objecter successfully canceled tick 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd2b4063c20 legacy=0x7fd2b40660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 >> v1:172.21.15.107:6790/0 conn(0x7fd2cc0ff710 legacy=0x7fd2cc0613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 shutdown_connections 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.363+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 >> 172.21.15.107:0/454320160 conn(0x7fd2cc0f2510 msgr2=0x7fd2cc1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.364+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 shutdown_connections 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.364+0000 7fd2d2118700 1 -- 172.21.15.107:0/454320160 wait complete. 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:21.364+0000 7fd2d2118700 1 librados: shutdown 2024-02-10T08:35:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:35:21.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:20.013576+0000 mgr.y (mgr.14553) 84 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:22.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:20.013576+0000 mgr.y (mgr.14553) 84 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:22.083 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:20.013576+0000 mgr.y (mgr.14553) 84 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:22.105 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-02-10T08:35:20.013364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":831,"num_read_kb":583,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":493,"ondisk_log_size":493,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2383264,"kb_used_data":6496,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747577952,"statfs":{"total":767960285184,"available":765519822848,"internally_reserved":0,"allocated":6651904,"data_stored":2720379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002441"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.420536+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:35:00.420536+0000","last_peered":"2024-02-10T08:35:00.420536+0000","last_clean":"2024-02-10T08:35:00.420536+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:35:00.420536+0000","last_undegraded":"2024-02-10T08:35:00.420536+0000","last_fullsized":"2024-02-10T08:35:00.420536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774644+0000","last_change":"2024-02-10T08:33:22.117853+0000","last_active":"2024-02-10T08:34:56.774644+0000","last_peered":"2024-02-10T08:34:56.774644+0000","last_clean":"2024-02-10T08:34:56.774644+0000","last_became_active":"2024-02-10T08:33:22.117534+0000","last_became_peered":"2024-02-10T08:33:22.117534+0000","last_unstale":"2024-02-10T08:34:56.774644+0000","last_undegraded":"2024-02-10T08:34:56.774644+0000","last_fullsized":"2024-02-10T08:34:56.774644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:33:21.099708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.699140+0000","last_change":"2024-02-10T08:34:41.648797+0000","last_active":"2024-02-10T08:34:56.699140+0000","last_peered":"2024-02-10T08:34:56.699140+0000","last_clean":"2024-02-10T08:34:56.699140+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:34:56.699140+0000","last_undegraded":"2024-02-10T08:34:56.699140+0000","last_fullsized":"2024-02-10T08:34:56.699140+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:41.648715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:41.648715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:15.479032+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:35:15.479032+0000","last_peered":"2024-02-10T08:35:15.479032+0000","last_clean":"2024-02-10T08:35:15.479032+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:35:15.479032+0000","last_undegraded":"2024-02-10T08:35:15.479032+0000","last_fullsized":"2024-02-10T08:35:15.479032+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776462+0000","last_change":"2024-02-10T08:34:38.118431+0000","last_active":"2024-02-10T08:34:56.776462+0000","last_peered":"2024-02-10T08:34:56.776462+0000","last_clean":"2024-02-10T08:34:56.776462+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:34:56.776462+0000","last_undegraded":"2024-02-10T08:34:56.776462+0000","last_fullsized":"2024-02-10T08:34:56.776462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.118347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.118347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613119+0000","last_change":"2024-02-10T08:34:37.332223+0000","last_active":"2024-02-10T08:34:56.613119+0000","last_peered":"2024-02-10T08:34:56.613119+0000","last_clean":"2024-02-10T08:34:56.613119+0000","last_became_active":"2024-02-10T08:33:22.121607+0000","last_became_peered":"2024-02-10T08:33:22.121607+0000","last_unstale":"2024-02-10T08:34:56.613119+0000","last_undegraded":"2024-02-10T08:34:56.613119+0000","last_fullsized":"2024-02-10T08:34:56.613119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.332140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.332140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775027+0000","last_change":"2024-02-10T08:34:23.594197+0000","last_active":"2024-02-10T08:34:56.775027+0000","last_peered":"2024-02-10T08:34:56.775027+0000","last_clean":"2024-02-10T08:34:56.775027+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:34:56.775027+0000","last_undegraded":"2024-02-10T08:34:56.775027+0000","last_fullsized":"2024-02-10T08:34:56.775027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.594135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.594135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:04.110147+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:35:04.110147+0000","last_peered":"2024-02-10T08:35:04.110147+0000","last_clean":"2024-02-10T08:35:04.110147+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:35:04.110147+0000","last_undegraded":"2024-02-10T08:35:04.110147+0000","last_fullsized":"2024-02-10T08:35:04.110147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.524105+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:35:08.524105+0000","last_peered":"2024-02-10T08:35:08.524105+0000","last_clean":"2024-02-10T08:35:08.524105+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:35:08.524105+0000","last_undegraded":"2024-02-10T08:35:08.524105+0000","last_fullsized":"2024-02-10T08:35:08.524105+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.936215+0000","last_change":"2024-02-10T08:35:00.936215+0000","last_active":"2024-02-10T08:35:00.936215+0000","last_peered":"2024-02-10T08:35:00.936215+0000","last_clean":"2024-02-10T08:35:00.936215+0000","last_became_active":"2024-02-10T08:33:22.119141+0000","last_became_peered":"2024-02-10T08:33:22.119141+0000","last_unstale":"2024-02-10T08:35:00.936215+0000","last_undegraded":"2024-02-10T08:35:00.936215+0000","last_fullsized":"2024-02-10T08:35:00.936215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775765+0000","last_change":"2024-02-10T08:34:26.673356+0000","last_active":"2024-02-10T08:34:56.775765+0000","last_peered":"2024-02-10T08:34:56.775765+0000","last_clean":"2024-02-10T08:34:56.775765+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:34:56.775765+0000","last_undegraded":"2024-02-10T08:34:56.775765+0000","last_fullsized":"2024-02-10T08:34:56.775765+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:26.673263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.673263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774643+0000","last_change":"2024-02-10T08:34:50.094010+0000","last_active":"2024-02-10T08:34:56.774643+0000","last_peered":"2024-02-10T08:34:56.774643+0000","last_clean":"2024-02-10T08:34:56.774643+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:34:56.774643+0000","last_undegraded":"2024-02-10T08:34:56.774643+0000","last_fullsized":"2024-02-10T08:34:56.774643+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:34:50.093915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.093915+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698222+0000","last_change":"2024-02-10T08:34:51.681056+0000","last_active":"2024-02-10T08:34:56.698222+0000","last_peered":"2024-02-10T08:34:56.698222+0000","last_clean":"2024-02-10T08:34:56.698222+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:34:56.698222+0000","last_undegraded":"2024-02-10T08:34:56.698222+0000","last_fullsized":"2024-02-10T08:34:56.698222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.680986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.680986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:01.492010+0000","last_change":"2024-02-10T08:35:01.492010+0000","last_active":"2024-02-10T08:35:01.492010+0000","last_peered":"2024-02-10T08:35:01.492010+0000","last_clean":"2024-02-10T08:35:01.492010+0000","last_became_active":"2024-02-10T08:33:22.122241+0000","last_became_peered":"2024-02-10T08:33:22.122241+0000","last_unstale":"2024-02-10T08:35:01.492010+0000","last_undegraded":"2024-02-10T08:35:01.492010+0000","last_fullsized":"2024-02-10T08:35:01.492010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.956668+0000","last_change":"2024-02-10T08:34:56.956608+0000","last_active":"2024-02-10T08:34:56.956668+0000","last_peered":"2024-02-10T08:34:56.956668+0000","last_clean":"2024-02-10T08:34:56.956668+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:34:56.956668+0000","last_undegraded":"2024-02-10T08:34:56.956668+0000","last_fullsized":"2024-02-10T08:34:56.956668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.956483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.956483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:59.084255+0000","last_change":"2024-02-10T08:34:59.084224+0000","last_active":"2024-02-10T08:34:59.084255+0000","last_peered":"2024-02-10T08:34:59.084255+0000","last_clean":"2024-02-10T08:34:59.084255+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:34:59.084255+0000","last_undegraded":"2024-02-10T08:34:59.084255+0000","last_fullsized":"2024-02-10T08:34:59.084255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:59.084151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:59.084151+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775240+0000","last_change":"2024-02-10T08:34:52.516328+0000","last_active":"2024-02-10T08:34:56.775240+0000","last_peered":"2024-02-10T08:34:56.775240+0000","last_clean":"2024-02-10T08:34:56.775240+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:34:56.775240+0000","last_undegraded":"2024-02-10T08:34:56.775240+0000","last_fullsized":"2024-02-10T08:34:56.775240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.516236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.516236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:07.038806+0000","last_change":"2024-02-10T08:35:07.038806+0000","last_active":"2024-02-10T08:35:07.038806+0000","last_peered":"2024-02-10T08:35:07.038806+0000","last_clean":"2024-02-10T08:35:07.038806+0000","last_became_active":"2024-02-10T08:33:22.116085+0000","last_became_peered":"2024-02-10T08:33:22.116085+0000","last_unstale":"2024-02-10T08:35:07.038806+0000","last_undegraded":"2024-02-10T08:35:07.038806+0000","last_fullsized":"2024-02-10T08:35:07.038806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774698+0000","last_change":"2024-02-10T08:34:37.616548+0000","last_active":"2024-02-10T08:34:56.774698+0000","last_peered":"2024-02-10T08:34:56.774698+0000","last_clean":"2024-02-10T08:34:56.774698+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:34:56.774698+0000","last_undegraded":"2024-02-10T08:34:56.774698+0000","last_fullsized":"2024-02-10T08:34:56.774698+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.616436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.616436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774864+0000","last_change":"2024-02-10T08:34:33.601286+0000","last_active":"2024-02-10T08:34:56.774864+0000","last_peered":"2024-02-10T08:34:56.774864+0000","last_clean":"2024-02-10T08:34:56.774864+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:34:56.774864+0000","last_undegraded":"2024-02-10T08:34:56.774864+0000","last_fullsized":"2024-02-10T08:34:56.774864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:33.601201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.601201+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776940+0000","last_change":"2024-02-10T08:33:24.126031+0000","last_active":"2024-02-10T08:34:56.776940+0000","last_peered":"2024-02-10T08:34:56.776940+0000","last_clean":"2024-02-10T08:34:56.776940+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:34:56.776940+0000","last_undegraded":"2024-02-10T08:34:56.776940+0000","last_fullsized":"2024-02-10T08:34:56.776940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614387+0000","last_change":"2024-02-10T08:34:39.357828+0000","last_active":"2024-02-10T08:34:56.614387+0000","last_peered":"2024-02-10T08:34:56.614387+0000","last_clean":"2024-02-10T08:34:56.614387+0000","last_became_active":"2024-02-10T08:33:22.120631+0000","last_became_peered":"2024-02-10T08:33:22.120631+0000","last_unstale":"2024-02-10T08:34:56.614387+0000","last_undegraded":"2024-02-10T08:34:56.614387+0000","last_fullsized":"2024-02-10T08:34:56.614387+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.357766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612968+0000","last_change":"2024-02-10T08:34:53.298291+0000","last_active":"2024-02-10T08:34:56.612968+0000","last_peered":"2024-02-10T08:34:56.612968+0000","last_clean":"2024-02-10T08:34:56.612968+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:34:56.612968+0000","last_undegraded":"2024-02-10T08:34:56.612968+0000","last_fullsized":"2024-02-10T08:34:56.612968+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.298226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.298226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":61,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613313+0000","last_change":"2024-02-10T08:34:26.526319+0000","last_active":"2024-02-10T08:34:56.613313+0000","last_peered":"2024-02-10T08:34:56.613313+0000","last_clean":"2024-02-10T08:34:56.613313+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:34:56.613313+0000","last_undegraded":"2024-02-10T08:34:56.613313+0000","last_fullsized":"2024-02-10T08:34:56.613313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:34:26.526206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.526206+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612454+0000","last_change":"2024-02-10T08:34:52.285891+0000","last_active":"2024-02-10T08:34:56.612454+0000","last_peered":"2024-02-10T08:34:56.612454+0000","last_clean":"2024-02-10T08:34:56.612454+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:34:56.612454+0000","last_undegraded":"2024-02-10T08:34:56.612454+0000","last_fullsized":"2024-02-10T08:34:56.612454+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.285824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.285824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:01.687455+0000","last_change":"2024-02-10T08:35:01.687455+0000","last_active":"2024-02-10T08:35:01.687455+0000","last_peered":"2024-02-10T08:35:01.687455+0000","last_clean":"2024-02-10T08:35:01.687455+0000","last_became_active":"2024-02-10T08:33:22.122436+0000","last_became_peered":"2024-02-10T08:33:22.122436+0000","last_unstale":"2024-02-10T08:35:01.687455+0000","last_undegraded":"2024-02-10T08:35:01.687455+0000","last_fullsized":"2024-02-10T08:35:01.687455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.687386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774391+0000","last_change":"2024-02-10T08:34:24.604411+0000","last_active":"2024-02-10T08:34:56.774391+0000","last_peered":"2024-02-10T08:34:56.774391+0000","last_clean":"2024-02-10T08:34:56.774391+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:34:56.774391+0000","last_undegraded":"2024-02-10T08:34:56.774391+0000","last_fullsized":"2024-02-10T08:34:56.774391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.604347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.604347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613427+0000","last_change":"2024-02-10T08:34:36.450017+0000","last_active":"2024-02-10T08:34:56.613427+0000","last_peered":"2024-02-10T08:34:56.613427+0000","last_clean":"2024-02-10T08:34:56.613427+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:34:56.613427+0000","last_undegraded":"2024-02-10T08:34:56.613427+0000","last_fullsized":"2024-02-10T08:34:56.613427+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:36.449943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.449943+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:09.044967+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:35:09.044967+0000","last_peered":"2024-02-10T08:35:09.044967+0000","last_clean":"2024-02-10T08:35:09.044967+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:35:09.044967+0000","last_undegraded":"2024-02-10T08:35:09.044967+0000","last_fullsized":"2024-02-10T08:35:09.044967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'9","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511275+0000","last_change":"2024-02-10T08:34:35.421984+0000","last_active":"2024-02-10T08:35:06.511275+0000","last_peered":"2024-02-10T08:35:06.511275+0000","last_clean":"2024-02-10T08:35:06.511275+0000","last_became_active":"2024-02-10T08:33:22.119251+0000","last_became_peered":"2024-02-10T08:33:22.119251+0000","last_unstale":"2024-02-10T08:35:06.511275+0000","last_undegraded":"2024-02-10T08:35:06.511275+0000","last_fullsized":"2024-02-10T08:35:06.511275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_deep_scrub":"74'9","last_deep_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.421866+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774351+0000","last_change":"2024-02-10T08:34:41.695373+0000","last_active":"2024-02-10T08:34:56.774351+0000","last_peered":"2024-02-10T08:34:56.774351+0000","last_clean":"2024-02-10T08:34:56.774351+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:34:56.774351+0000","last_undegraded":"2024-02-10T08:34:56.774351+0000","last_fullsized":"2024-02-10T08:34:56.774351+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.661214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.661214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698966+0000","last_change":"2024-02-10T08:34:43.680148+0000","last_active":"2024-02-10T08:34:56.698966+0000","last_peered":"2024-02-10T08:34:56.698966+0000","last_clean":"2024-02-10T08:34:56.698966+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:34:56.698966+0000","last_undegraded":"2024-02-10T08:34:56.698966+0000","last_fullsized":"2024-02-10T08:34:56.698966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:43.680073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:43.680073+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774809+0000","last_change":"2024-02-10T08:34:32.620154+0000","last_active":"2024-02-10T08:34:56.774809+0000","last_peered":"2024-02-10T08:34:56.774809+0000","last_clean":"2024-02-10T08:34:56.774809+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:34:56.774809+0000","last_undegraded":"2024-02-10T08:34:56.774809+0000","last_fullsized":"2024-02-10T08:34:56.774809+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:32.620086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.620086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:58.971723+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:34:58.971723+0000","last_peered":"2024-02-10T08:34:58.971723+0000","last_clean":"2024-02-10T08:34:58.971723+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:34:58.971723+0000","last_undegraded":"2024-02-10T08:34:58.971723+0000","last_fullsized":"2024-02-10T08:34:58.971723+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613515+0000","last_change":"2024-02-10T08:34:31.359650+0000","last_active":"2024-02-10T08:34:56.613515+0000","last_peered":"2024-02-10T08:34:56.613515+0000","last_clean":"2024-02-10T08:34:56.613515+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:34:56.613515+0000","last_undegraded":"2024-02-10T08:34:56.613515+0000","last_fullsized":"2024-02-10T08:34:56.613515+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.542445+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:35:11.542445+0000","last_peered":"2024-02-10T08:35:11.542445+0000","last_clean":"2024-02-10T08:35:11.542445+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:35:11.542445+0000","last_undegraded":"2024-02-10T08:35:11.542445+0000","last_fullsized":"2024-02-10T08:35:11.542445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774701+0000","last_change":"2024-02-10T08:34:35.819857+0000","last_active":"2024-02-10T08:34:56.774701+0000","last_peered":"2024-02-10T08:34:56.774701+0000","last_clean":"2024-02-10T08:34:56.774701+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:34:56.774701+0000","last_undegraded":"2024-02-10T08:34:56.774701+0000","last_fullsized":"2024-02-10T08:34:56.774701+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.819777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.819777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612685+0000","last_change":"2024-02-10T08:34:55.286635+0000","last_active":"2024-02-10T08:34:56.612685+0000","last_peered":"2024-02-10T08:34:56.612685+0000","last_clean":"2024-02-10T08:34:56.612685+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:34:56.612685+0000","last_undegraded":"2024-02-10T08:34:56.612685+0000","last_fullsized":"2024-02-10T08:34:56.612685+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.286540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.286540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775137+0000","last_change":"2024-02-10T08:34:28.590439+0000","last_active":"2024-02-10T08:34:56.775137+0000","last_peered":"2024-02-10T08:34:56.775137+0000","last_clean":"2024-02-10T08:34:56.775137+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:34:56.775137+0000","last_undegraded":"2024-02-10T08:34:56.775137+0000","last_fullsized":"2024-02-10T08:34:56.775137+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.590362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.590362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":41,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774395+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:34:56.774395+0000","last_peered":"2024-02-10T08:34:56.774395+0000","last_clean":"2024-02-10T08:34:56.774395+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:34:56.774395+0000","last_undegraded":"2024-02-10T08:34:56.774395+0000","last_fullsized":"2024-02-10T08:34:56.774395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774274+0000","last_change":"2024-02-10T08:34:28.605230+0000","last_active":"2024-02-10T08:34:56.774274+0000","last_peered":"2024-02-10T08:34:56.774274+0000","last_clean":"2024-02-10T08:34:56.774274+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:34:56.774274+0000","last_undegraded":"2024-02-10T08:34:56.774274+0000","last_fullsized":"2024-02-10T08:34:56.774274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.605166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.605166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.105717+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:35:02.105717+0000","last_peered":"2024-02-10T08:35:02.105717+0000","last_clean":"2024-02-10T08:35:02.105717+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:35:02.105717+0000","last_undegraded":"2024-02-10T08:35:02.105717+0000","last_fullsized":"2024-02-10T08:35:02.105717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614392+0000","last_change":"2024-02-10T08:34:36.300831+0000","last_active":"2024-02-10T08:34:56.614392+0000","last_peered":"2024-02-10T08:34:56.614392+0000","last_clean":"2024-02-10T08:34:56.614392+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:34:56.614392+0000","last_undegraded":"2024-02-10T08:34:56.614392+0000","last_fullsized":"2024-02-10T08:34:56.614392+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:36.300770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.300770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774801+0000","last_change":"2024-02-10T08:33:20.118729+0000","last_active":"2024-02-10T08:34:56.774801+0000","last_peered":"2024-02-10T08:34:56.774801+0000","last_clean":"2024-02-10T08:34:56.774801+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:34:56.774801+0000","last_undegraded":"2024-02-10T08:34:56.774801+0000","last_fullsized":"2024-02-10T08:34:56.774801+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:33:19.093397+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697937+0000","last_change":"2024-02-10T08:34:48.691828+0000","last_active":"2024-02-10T08:34:56.697937+0000","last_peered":"2024-02-10T08:34:56.697937+0000","last_clean":"2024-02-10T08:34:56.697937+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:34:56.697937+0000","last_undegraded":"2024-02-10T08:34:56.697937+0000","last_fullsized":"2024-02-10T08:34:56.697937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.691737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.691737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774730+0000","last_change":"2024-02-10T08:34:27.584434+0000","last_active":"2024-02-10T08:34:56.774730+0000","last_peered":"2024-02-10T08:34:56.774730+0000","last_clean":"2024-02-10T08:34:56.774730+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:34:56.774730+0000","last_undegraded":"2024-02-10T08:34:56.774730+0000","last_fullsized":"2024-02-10T08:34:56.774730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:27.584369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.584369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612971+0000","last_change":"2024-02-10T08:34:23.544050+0000","last_active":"2024-02-10T08:34:56.612971+0000","last_peered":"2024-02-10T08:34:56.612971+0000","last_clean":"2024-02-10T08:34:56.612971+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:34:56.612971+0000","last_undegraded":"2024-02-10T08:34:56.612971+0000","last_fullsized":"2024-02-10T08:34:56.612971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.543984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.543984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":56,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613529+0000","last_change":"2024-02-10T08:34:47.420238+0000","last_active":"2024-02-10T08:34:56.613529+0000","last_peered":"2024-02-10T08:34:56.613529+0000","last_clean":"2024-02-10T08:34:56.613529+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:34:56.613529+0000","last_undegraded":"2024-02-10T08:34:56.613529+0000","last_fullsized":"2024-02-10T08:34:56.613529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.420174+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776330+0000","last_change":"2024-02-10T08:33:24.119570+0000","last_active":"2024-02-10T08:34:56.776330+0000","last_peered":"2024-02-10T08:34:56.776330+0000","last_clean":"2024-02-10T08:34:56.776330+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:34:56.776330+0000","last_undegraded":"2024-02-10T08:34:56.776330+0000","last_fullsized":"2024-02-10T08:34:56.776330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511188+0000","last_change":"2024-02-10T08:34:51.497308+0000","last_active":"2024-02-10T08:35:06.511188+0000","last_peered":"2024-02-10T08:35:06.511188+0000","last_clean":"2024-02-10T08:35:06.511188+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:35:06.511188+0000","last_undegraded":"2024-02-10T08:35:06.511188+0000","last_fullsized":"2024-02-10T08:35:06.511188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.497242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.497242+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698419+0000","last_change":"2024-02-10T08:34:35.635203+0000","last_active":"2024-02-10T08:34:56.698419+0000","last_peered":"2024-02-10T08:34:56.698419+0000","last_clean":"2024-02-10T08:34:56.698419+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:34:56.698419+0000","last_undegraded":"2024-02-10T08:34:56.698419+0000","last_fullsized":"2024-02-10T08:34:56.698419+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.635108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.635108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:13.494784+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:35:13.494784+0000","last_peered":"2024-02-10T08:35:13.494784+0000","last_clean":"2024-02-10T08:35:13.494784+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:35:13.494784+0000","last_undegraded":"2024-02-10T08:35:13.494784+0000","last_fullsized":"2024-02-10T08:35:13.494784+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.705789+0000","last_change":"2024-02-10T08:35:02.705789+0000","last_active":"2024-02-10T08:35:02.705789+0000","last_peered":"2024-02-10T08:35:02.705789+0000","last_clean":"2024-02-10T08:35:02.705789+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:35:02.705789+0000","last_undegraded":"2024-02-10T08:35:02.705789+0000","last_fullsized":"2024-02-10T08:35:02.705789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.705679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.705679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511297+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:35:06.511297+0000","last_peered":"2024-02-10T08:35:06.511297+0000","last_clean":"2024-02-10T08:35:06.511297+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:35:06.511297+0000","last_undegraded":"2024-02-10T08:35:06.511297+0000","last_fullsized":"2024-02-10T08:35:06.511297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774530+0000","last_change":"2024-02-10T08:34:29.584600+0000","last_active":"2024-02-10T08:34:56.774530+0000","last_peered":"2024-02-10T08:34:56.774530+0000","last_clean":"2024-02-10T08:34:56.774530+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:34:56.774530+0000","last_undegraded":"2024-02-10T08:34:56.774530+0000","last_fullsized":"2024-02-10T08:34:56.774530+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.584529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.584529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774735+0000","last_change":"2024-02-10T08:34:31.663950+0000","last_active":"2024-02-10T08:34:56.774735+0000","last_peered":"2024-02-10T08:34:56.774735+0000","last_clean":"2024-02-10T08:34:56.774735+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:34:56.774735+0000","last_undegraded":"2024-02-10T08:34:56.774735+0000","last_fullsized":"2024-02-10T08:34:56.774735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:31.663882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.663882+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:10.031019+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:35:10.031019+0000","last_peered":"2024-02-10T08:35:10.031019+0000","last_clean":"2024-02-10T08:35:10.031019+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:35:10.031019+0000","last_undegraded":"2024-02-10T08:35:10.031019+0000","last_fullsized":"2024-02-10T08:35:10.031019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775051+0000","last_change":"2024-02-10T08:34:45.607331+0000","last_active":"2024-02-10T08:34:56.775051+0000","last_peered":"2024-02-10T08:34:56.775051+0000","last_clean":"2024-02-10T08:34:56.775051+0000","last_became_active":"2024-02-10T08:33:22.121591+0000","last_became_peered":"2024-02-10T08:33:22.121591+0000","last_unstale":"2024-02-10T08:34:56.775051+0000","last_undegraded":"2024-02-10T08:34:56.775051+0000","last_fullsized":"2024-02-10T08:34:56.775051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.607254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.607254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612786+0000","last_change":"2024-02-10T08:34:29.402936+0000","last_active":"2024-02-10T08:34:56.612786+0000","last_peered":"2024-02-10T08:34:56.612786+0000","last_clean":"2024-02-10T08:34:56.612786+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:34:56.612786+0000","last_undegraded":"2024-02-10T08:34:56.612786+0000","last_fullsized":"2024-02-10T08:34:56.612786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.402874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.402874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775197+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:34:56.775197+0000","last_peered":"2024-02-10T08:34:56.775197+0000","last_clean":"2024-02-10T08:34:56.775197+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:34:56.775197+0000","last_undegraded":"2024-02-10T08:34:56.775197+0000","last_fullsized":"2024-02-10T08:34:56.775197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:07.725134+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:35:07.725134+0000","last_peered":"2024-02-10T08:35:07.725134+0000","last_clean":"2024-02-10T08:35:07.725134+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:35:07.725134+0000","last_undegraded":"2024-02-10T08:35:07.725134+0000","last_fullsized":"2024-02-10T08:35:07.725134+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:02.446706+0000","last_change":"2024-02-10T08:35:02.446706+0000","last_active":"2024-02-10T08:35:02.446706+0000","last_peered":"2024-02-10T08:35:02.446706+0000","last_clean":"2024-02-10T08:35:02.446706+0000","last_became_active":"2024-02-10T08:33:22.118323+0000","last_became_peered":"2024-02-10T08:33:22.118323+0000","last_unstale":"2024-02-10T08:35:02.446706+0000","last_undegraded":"2024-02-10T08:35:02.446706+0000","last_fullsized":"2024-02-10T08:35:02.446706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.699547+0000","last_change":"2024-02-10T08:34:56.699511+0000","last_active":"2024-02-10T08:34:56.699547+0000","last_peered":"2024-02-10T08:34:56.699547+0000","last_clean":"2024-02-10T08:34:56.699547+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:34:56.699547+0000","last_undegraded":"2024-02-10T08:34:56.699547+0000","last_fullsized":"2024-02-10T08:34:56.699547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.699420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.699420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'9","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698876+0000","last_change":"2024-02-10T08:34:23.674473+0000","last_active":"2024-02-10T08:34:56.698876+0000","last_peered":"2024-02-10T08:34:56.698876+0000","last_clean":"2024-02-10T08:34:56.698876+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:34:56.698876+0000","last_undegraded":"2024-02-10T08:34:56.698876+0000","last_fullsized":"2024-02-10T08:34:56.698876+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:23.674404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.674404+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774791+0000","last_change":"2024-02-10T08:34:45.519207+0000","last_active":"2024-02-10T08:34:56.774791+0000","last_peered":"2024-02-10T08:34:56.774791+0000","last_clean":"2024-02-10T08:34:56.774791+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:34:56.774791+0000","last_undegraded":"2024-02-10T08:34:56.774791+0000","last_fullsized":"2024-02-10T08:34:56.774791+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.519116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.519116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.136470+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:35:00.136470+0000","last_peered":"2024-02-10T08:35:00.136470+0000","last_clean":"2024-02-10T08:35:00.136470+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:35:00.136470+0000","last_undegraded":"2024-02-10T08:35:00.136470+0000","last_fullsized":"2024-02-10T08:35:00.136470+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776217+0000","last_change":"2024-02-10T08:34:54.039656+0000","last_active":"2024-02-10T08:34:56.776217+0000","last_peered":"2024-02-10T08:34:56.776217+0000","last_clean":"2024-02-10T08:34:56.776217+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:34:56.776217+0000","last_undegraded":"2024-02-10T08:34:56.776217+0000","last_fullsized":"2024-02-10T08:34:56.776217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.039584+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612415+0000","last_change":"2024-02-10T08:34:39.590686+0000","last_active":"2024-02-10T08:34:56.612415+0000","last_peered":"2024-02-10T08:34:56.612415+0000","last_clean":"2024-02-10T08:34:56.612415+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:34:56.612415+0000","last_undegraded":"2024-02-10T08:34:56.612415+0000","last_fullsized":"2024-02-10T08:34:56.612415+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.590604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.590604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697916+0000","last_change":"2024-02-10T08:34:31.544112+0000","last_active":"2024-02-10T08:34:56.697916+0000","last_peered":"2024-02-10T08:34:56.697916+0000","last_clean":"2024-02-10T08:34:56.697916+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:34:56.697916+0000","last_undegraded":"2024-02-10T08:34:56.697916+0000","last_fullsized":"2024-02-10T08:34:56.697916+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.544049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.544049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612915+0000","last_change":"2024-02-10T08:34:46.452573+0000","last_active":"2024-02-10T08:34:56.612915+0000","last_peered":"2024-02-10T08:34:56.612915+0000","last_clean":"2024-02-10T08:34:56.612915+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:34:56.612915+0000","last_undegraded":"2024-02-10T08:34:56.612915+0000","last_fullsized":"2024-02-10T08:34:56.612915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:46.452504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:46.452504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'10","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613672+0000","last_change":"2024-02-10T08:34:33.231522+0000","last_active":"2024-02-10T08:34:56.613672+0000","last_peered":"2024-02-10T08:34:56.613672+0000","last_clean":"2024-02-10T08:34:56.613672+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:34:56.613672+0000","last_undegraded":"2024-02-10T08:34:56.613672+0000","last_fullsized":"2024-02-10T08:34:56.613672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:33.231411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.231411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511185+0000","last_change":"2024-02-10T08:34:47.693139+0000","last_active":"2024-02-10T08:35:06.511185+0000","last_peered":"2024-02-10T08:35:06.511185+0000","last_clean":"2024-02-10T08:35:06.511185+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:35:06.511185+0000","last_undegraded":"2024-02-10T08:35:06.511185+0000","last_fullsized":"2024-02-10T08:35:06.511185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:47.693030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.693030+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:04.456788+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:35:04.456788+0000","last_peered":"2024-02-10T08:35:04.456788+0000","last_clean":"2024-02-10T08:35:04.456788+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:35:04.456788+0000","last_undegraded":"2024-02-10T08:35:04.456788+0000","last_fullsized":"2024-02-10T08:35:04.456788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774621+0000","last_change":"2024-02-10T08:34:20.652012+0000","last_active":"2024-02-10T08:34:56.774621+0000","last_peered":"2024-02-10T08:34:56.774621+0000","last_clean":"2024-02-10T08:34:56.774621+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:34:56.774621+0000","last_undegraded":"2024-02-10T08:34:56.774621+0000","last_fullsized":"2024-02-10T08:34:56.774621+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.651946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.651946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511270+0000","last_change":"2024-02-10T08:34:47.549753+0000","last_active":"2024-02-10T08:35:06.511270+0000","last_peered":"2024-02-10T08:35:06.511270+0000","last_clean":"2024-02-10T08:35:06.511270+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:35:06.511270+0000","last_undegraded":"2024-02-10T08:35:06.511270+0000","last_fullsized":"2024-02-10T08:35:06.511270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:34:47.549671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.549671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:00.639772+0000","last_change":"2024-02-10T08:35:00.639772+0000","last_active":"2024-02-10T08:35:00.639772+0000","last_peered":"2024-02-10T08:35:00.639772+0000","last_clean":"2024-02-10T08:35:00.639772+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:35:00.639772+0000","last_undegraded":"2024-02-10T08:35:00.639772+0000","last_fullsized":"2024-02-10T08:35:00.639772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.639705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.639705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775933+0000","last_change":"2024-02-10T08:34:51.910276+0000","last_active":"2024-02-10T08:34:56.775933+0000","last_peered":"2024-02-10T08:34:56.775933+0000","last_clean":"2024-02-10T08:34:56.775933+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:34:56.775933+0000","last_undegraded":"2024-02-10T08:34:56.775933+0000","last_fullsized":"2024-02-10T08:34:56.775933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.910184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":153,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:14.260052+0000","last_change":"2024-02-10T08:35:07.227647+0000","last_active":"2024-02-10T08:35:14.260052+0000","last_peered":"2024-02-10T08:35:14.260052+0000","last_clean":"2024-02-10T08:35:14.260052+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:35:14.260052+0000","last_undegraded":"2024-02-10T08:35:14.260052+0000","last_fullsized":"2024-02-10T08:35:14.260052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.227587+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612586+0000","last_change":"2024-02-10T08:34:52.698430+0000","last_active":"2024-02-10T08:34:56.612586+0000","last_peered":"2024-02-10T08:34:56.612586+0000","last_clean":"2024-02-10T08:34:56.612586+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:34:56.612586+0000","last_undegraded":"2024-02-10T08:34:56.612586+0000","last_fullsized":"2024-02-10T08:34:56.612586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.698329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":63,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614951+0000","last_change":"2024-02-10T08:34:27.278457+0000","last_active":"2024-02-10T08:34:56.614951+0000","last_peered":"2024-02-10T08:34:56.614951+0000","last_clean":"2024-02-10T08:34:56.614951+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:34:56.614951+0000","last_undegraded":"2024-02-10T08:34:56.614951+0000","last_fullsized":"2024-02-10T08:34:56.614951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:34:27.278388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.278388+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.720849+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:35:08.720849+0000","last_peered":"2024-02-10T08:35:08.720849+0000","last_clean":"2024-02-10T08:35:08.720849+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:35:08.720849+0000","last_undegraded":"2024-02-10T08:35:08.720849+0000","last_fullsized":"2024-02-10T08:35:08.720849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775867+0000","last_change":"2024-02-10T08:34:55.443465+0000","last_active":"2024-02-10T08:34:56.775867+0000","last_peered":"2024-02-10T08:34:56.775867+0000","last_clean":"2024-02-10T08:34:56.775867+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:34:56.775867+0000","last_undegraded":"2024-02-10T08:34:56.775867+0000","last_fullsized":"2024-02-10T08:34:56.775867+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.443390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.443390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774586+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:34:56.774586+0000","last_peered":"2024-02-10T08:34:56.774586+0000","last_clean":"2024-02-10T08:34:56.774586+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:34:56.774586+0000","last_undegraded":"2024-02-10T08:34:56.774586+0000","last_fullsized":"2024-02-10T08:34:56.774586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775158+0000","last_change":"2024-02-10T08:34:52.491498+0000","last_active":"2024-02-10T08:34:56.775158+0000","last_peered":"2024-02-10T08:34:56.775158+0000","last_clean":"2024-02-10T08:34:56.775158+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:34:56.775158+0000","last_undegraded":"2024-02-10T08:34:56.775158+0000","last_fullsized":"2024-02-10T08:34:56.775158+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.491420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.491420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776453+0000","last_change":"2024-02-10T08:34:48.841104+0000","last_active":"2024-02-10T08:34:56.776453+0000","last_peered":"2024-02-10T08:34:56.776453+0000","last_clean":"2024-02-10T08:34:56.776453+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:34:56.776453+0000","last_undegraded":"2024-02-10T08:34:56.776453+0000","last_fullsized":"2024-02-10T08:34:56.776453+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.841025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.841025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613175+0000","last_change":"2024-02-10T08:34:39.378393+0000","last_active":"2024-02-10T08:34:56.613175+0000","last_peered":"2024-02-10T08:34:56.613175+0000","last_clean":"2024-02-10T08:34:56.613175+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:34:56.613175+0000","last_undegraded":"2024-02-10T08:34:56.613175+0000","last_fullsized":"2024-02-10T08:34:56.613175+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:39.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.378312+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775424+0000","last_change":"2024-02-10T08:34:40.809042+0000","last_active":"2024-02-10T08:34:56.775424+0000","last_peered":"2024-02-10T08:34:56.775424+0000","last_clean":"2024-02-10T08:34:56.775424+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:34:56.775424+0000","last_undegraded":"2024-02-10T08:34:56.775424+0000","last_fullsized":"2024-02-10T08:34:56.775424+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:40.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:40.808978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":33,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612648+0000","last_change":"2024-02-10T08:34:28.307074+0000","last_active":"2024-02-10T08:34:56.612648+0000","last_peered":"2024-02-10T08:34:56.612648+0000","last_clean":"2024-02-10T08:34:56.612648+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:34:56.612648+0000","last_undegraded":"2024-02-10T08:34:56.612648+0000","last_fullsized":"2024-02-10T08:34:56.612648+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:34:28.306997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.306997+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698173+0000","last_change":"2024-02-10T08:34:21.731002+0000","last_active":"2024-02-10T08:34:56.698173+0000","last_peered":"2024-02-10T08:34:56.698173+0000","last_clean":"2024-02-10T08:34:56.698173+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:34:56.698173+0000","last_undegraded":"2024-02-10T08:34:56.698173+0000","last_fullsized":"2024-02-10T08:34:56.698173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.730937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.730937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776370+0000","last_change":"2024-02-10T08:34:25.707975+0000","last_active":"2024-02-10T08:34:56.776370+0000","last_peered":"2024-02-10T08:34:56.776370+0000","last_clean":"2024-02-10T08:34:56.776370+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:34:56.776370+0000","last_undegraded":"2024-02-10T08:34:56.776370+0000","last_fullsized":"2024-02-10T08:34:56.776370+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:25.707852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.707852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698746+0000","last_change":"2024-02-10T08:34:28.599042+0000","last_active":"2024-02-10T08:34:56.698746+0000","last_peered":"2024-02-10T08:34:56.698746+0000","last_clean":"2024-02-10T08:34:56.698746+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:34:56.698746+0000","last_undegraded":"2024-02-10T08:34:56.698746+0000","last_fullsized":"2024-02-10T08:34:56.698746+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:28.598979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.598979+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775788+0000","last_change":"2024-02-10T08:34:38.628666+0000","last_active":"2024-02-10T08:34:56.775788+0000","last_peered":"2024-02-10T08:34:56.775788+0000","last_clean":"2024-02-10T08:34:56.775788+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:34:56.775788+0000","last_undegraded":"2024-02-10T08:34:56.775788+0000","last_fullsized":"2024-02-10T08:34:56.775788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.628559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.628559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613054+0000","last_change":"2024-02-10T08:34:44.434673+0000","last_active":"2024-02-10T08:34:56.613054+0000","last_peered":"2024-02-10T08:34:56.613054+0000","last_clean":"2024-02-10T08:34:56.613054+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:34:56.613054+0000","last_undegraded":"2024-02-10T08:34:56.613054+0000","last_fullsized":"2024-02-10T08:34:56.613054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.434603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.434603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776019+0000","last_change":"2024-02-10T08:34:44.810962+0000","last_active":"2024-02-10T08:34:56.776019+0000","last_peered":"2024-02-10T08:34:56.776019+0000","last_clean":"2024-02-10T08:34:56.776019+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:34:56.776019+0000","last_undegraded":"2024-02-10T08:34:56.776019+0000","last_fullsized":"2024-02-10T08:34:56.776019+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.810886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.810886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":71,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698301+0000","last_change":"2024-02-10T08:34:39.703715+0000","last_active":"2024-02-10T08:34:56.698301+0000","last_peered":"2024-02-10T08:34:56.698301+0000","last_clean":"2024-02-10T08:34:56.698301+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:34:56.698301+0000","last_undegraded":"2024-02-10T08:34:56.698301+0000","last_fullsized":"2024-02-10T08:34:56.698301+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:39.703643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.703643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613055+0000","last_change":"2024-02-10T08:34:51.348414+0000","last_active":"2024-02-10T08:34:56.613055+0000","last_peered":"2024-02-10T08:34:56.613055+0000","last_clean":"2024-02-10T08:34:56.613055+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:34:56.613055+0000","last_undegraded":"2024-02-10T08:34:56.613055+0000","last_fullsized":"2024-02-10T08:34:56.613055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.348334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.348334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776782+0000","last_change":"2024-02-10T08:33:24.119911+0000","last_active":"2024-02-10T08:34:56.776782+0000","last_peered":"2024-02-10T08:34:56.776782+0000","last_clean":"2024-02-10T08:34:56.776782+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:34:56.776782+0000","last_undegraded":"2024-02-10T08:34:56.776782+0000","last_fullsized":"2024-02-10T08:34:56.776782+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:03.147093+0000","last_change":"2024-02-10T08:35:03.147093+0000","last_active":"2024-02-10T08:35:03.147093+0000","last_peered":"2024-02-10T08:35:03.147093+0000","last_clean":"2024-02-10T08:35:03.147093+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:35:03.147093+0000","last_undegraded":"2024-02-10T08:35:03.147093+0000","last_fullsized":"2024-02-10T08:35:03.147093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:35:03.146990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.146990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:09.492086+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:35:09.492086+0000","last_peered":"2024-02-10T08:35:09.492086+0000","last_clean":"2024-02-10T08:35:09.492086+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:35:09.492086+0000","last_undegraded":"2024-02-10T08:35:09.492086+0000","last_fullsized":"2024-02-10T08:35:09.492086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:58.059779+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:34:58.059779+0000","last_peered":"2024-02-10T08:34:58.059779+0000","last_clean":"2024-02-10T08:34:58.059779+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:34:58.059779+0000","last_undegraded":"2024-02-10T08:34:58.059779+0000","last_fullsized":"2024-02-10T08:34:58.059779+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.025052+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:35:11.025052+0000","last_peered":"2024-02-10T08:35:11.025052+0000","last_clean":"2024-02-10T08:35:11.025052+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:35:11.025052+0000","last_undegraded":"2024-02-10T08:35:11.025052+0000","last_fullsized":"2024-02-10T08:35:11.025052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613510+0000","last_change":"2024-02-10T08:34:22.584930+0000","last_active":"2024-02-10T08:34:56.613510+0000","last_peered":"2024-02-10T08:34:56.613510+0000","last_clean":"2024-02-10T08:34:56.613510+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:34:56.613510+0000","last_undegraded":"2024-02-10T08:34:56.613510+0000","last_fullsized":"2024-02-10T08:34:56.613510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:22.584856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:22.584856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776119+0000","last_change":"2024-02-10T08:34:36.155885+0000","last_active":"2024-02-10T08:34:56.776119+0000","last_peered":"2024-02-10T08:34:56.776119+0000","last_clean":"2024-02-10T08:34:56.776119+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:34:56.776119+0000","last_undegraded":"2024-02-10T08:34:56.776119+0000","last_fullsized":"2024-02-10T08:34:56.776119+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:36.155806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.155806+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511177+0000","last_change":"2024-02-10T08:34:36.649251+0000","last_active":"2024-02-10T08:35:06.511177+0000","last_peered":"2024-02-10T08:35:06.511177+0000","last_clean":"2024-02-10T08:35:06.511177+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:35:06.511177+0000","last_undegraded":"2024-02-10T08:35:06.511177+0000","last_fullsized":"2024-02-10T08:35:06.511177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:36.649186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.649186+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774728+0000","last_change":"2024-02-10T08:33:24.126537+0000","last_active":"2024-02-10T08:34:56.774728+0000","last_peered":"2024-02-10T08:34:56.774728+0000","last_clean":"2024-02-10T08:34:56.774728+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:34:56.774728+0000","last_undegraded":"2024-02-10T08:34:56.774728+0000","last_fullsized":"2024-02-10T08:34:56.774728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.520594+0000","last_change":"2024-02-10T08:35:06.520594+0000","last_active":"2024-02-10T08:35:06.520594+0000","last_peered":"2024-02-10T08:35:06.520594+0000","last_clean":"2024-02-10T08:35:06.520594+0000","last_became_active":"2024-02-10T08:33:22.122087+0000","last_became_peered":"2024-02-10T08:33:22.122087+0000","last_unstale":"2024-02-10T08:35:06.520594+0000","last_undegraded":"2024-02-10T08:35:06.520594+0000","last_fullsized":"2024-02-10T08:35:06.520594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776508+0000","last_change":"2024-02-10T08:34:29.780708+0000","last_active":"2024-02-10T08:34:56.776508+0000","last_peered":"2024-02-10T08:34:56.776508+0000","last_clean":"2024-02-10T08:34:56.776508+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:34:56.776508+0000","last_undegraded":"2024-02-10T08:34:56.776508+0000","last_fullsized":"2024-02-10T08:34:56.776508+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.780644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.780644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776948+0000","last_change":"2024-02-10T08:34:25.085734+0000","last_active":"2024-02-10T08:34:56.776948+0000","last_peered":"2024-02-10T08:34:56.776948+0000","last_clean":"2024-02-10T08:34:56.776948+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:34:56.776948+0000","last_undegraded":"2024-02-10T08:34:56.776948+0000","last_fullsized":"2024-02-10T08:34:56.776948+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:25.085663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.085663+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775352+0000","last_change":"2024-02-10T08:33:24.122991+0000","last_active":"2024-02-10T08:34:56.775352+0000","last_peered":"2024-02-10T08:34:56.775352+0000","last_clean":"2024-02-10T08:34:56.775352+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:34:56.775352+0000","last_undegraded":"2024-02-10T08:34:56.775352+0000","last_fullsized":"2024-02-10T08:34:56.775352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.027015+0000","last_change":"2024-02-10T08:35:11.026984+0000","last_active":"2024-02-10T08:35:11.027015+0000","last_peered":"2024-02-10T08:35:11.027015+0000","last_clean":"2024-02-10T08:35:11.027015+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:35:11.027015+0000","last_undegraded":"2024-02-10T08:35:11.027015+0000","last_fullsized":"2024-02-10T08:35:11.027015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613242+0000","last_change":"2024-02-10T08:34:32.208396+0000","last_active":"2024-02-10T08:34:56.613242+0000","last_peered":"2024-02-10T08:34:56.613242+0000","last_clean":"2024-02-10T08:34:56.613242+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:34:56.613242+0000","last_undegraded":"2024-02-10T08:34:56.613242+0000","last_fullsized":"2024-02-10T08:34:56.613242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:34:32.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.208313+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:11.190561+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:35:11.190561+0000","last_peered":"2024-02-10T08:35:11.190561+0000","last_clean":"2024-02-10T08:35:11.190561+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:35:11.190561+0000","last_undegraded":"2024-02-10T08:35:11.190561+0000","last_fullsized":"2024-02-10T08:35:11.190561+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.776817+0000","last_change":"2024-02-10T08:34:35.201454+0000","last_active":"2024-02-10T08:34:56.776817+0000","last_peered":"2024-02-10T08:34:56.776817+0000","last_clean":"2024-02-10T08:34:56.776817+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:34:56.776817+0000","last_undegraded":"2024-02-10T08:34:56.776817+0000","last_fullsized":"2024-02-10T08:34:56.776817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.201371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.201371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511180+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:35:06.511180+0000","last_peered":"2024-02-10T08:35:06.511180+0000","last_clean":"2024-02-10T08:35:06.511180+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:35:06.511180+0000","last_undegraded":"2024-02-10T08:35:06.511180+0000","last_fullsized":"2024-02-10T08:35:06.511180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613457+0000","last_change":"2024-02-10T08:34:24.362389+0000","last_active":"2024-02-10T08:34:56.613457+0000","last_peered":"2024-02-10T08:34:56.613457+0000","last_clean":"2024-02-10T08:34:56.613457+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:34:56.613457+0000","last_undegraded":"2024-02-10T08:34:56.613457+0000","last_fullsized":"2024-02-10T08:34:56.613457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.362309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.362309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775026+0000","last_change":"2024-02-10T08:34:21.123617+0000","last_active":"2024-02-10T08:34:56.775026+0000","last_peered":"2024-02-10T08:34:56.775026+0000","last_clean":"2024-02-10T08:34:56.775026+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:34:56.775026+0000","last_undegraded":"2024-02-10T08:34:56.775026+0000","last_fullsized":"2024-02-10T08:34:56.775026+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:21.123548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.123548+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613294+0000","last_change":"2024-02-10T08:34:53.743882+0000","last_active":"2024-02-10T08:34:56.613294+0000","last_peered":"2024-02-10T08:34:56.613294+0000","last_clean":"2024-02-10T08:34:56.613294+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:34:56.613294+0000","last_undegraded":"2024-02-10T08:34:56.613294+0000","last_fullsized":"2024-02-10T08:34:56.613294+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.743807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.743807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:08.227030+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:35:08.227030+0000","last_peered":"2024-02-10T08:35:08.227030+0000","last_clean":"2024-02-10T08:35:08.227030+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:35:08.227030+0000","last_undegraded":"2024-02-10T08:35:08.227030+0000","last_fullsized":"2024-02-10T08:35:08.227030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774624+0000","last_change":"2024-02-10T08:34:20.776146+0000","last_active":"2024-02-10T08:34:56.774624+0000","last_peered":"2024-02-10T08:34:56.774624+0000","last_clean":"2024-02-10T08:34:56.774624+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:34:56.774624+0000","last_undegraded":"2024-02-10T08:34:56.774624+0000","last_fullsized":"2024-02-10T08:34:56.774624+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.776052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.776052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.775698+0000","last_change":"2024-02-10T08:34:24.118004+0000","last_active":"2024-02-10T08:34:56.775698+0000","last_peered":"2024-02-10T08:34:56.775698+0000","last_clean":"2024-02-10T08:34:56.775698+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:34:56.775698+0000","last_undegraded":"2024-02-10T08:34:56.775698+0000","last_fullsized":"2024-02-10T08:34:56.775698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:34:24.117909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.117909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:13.030970+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:35:13.030970+0000","last_peered":"2024-02-10T08:35:13.030970+0000","last_clean":"2024-02-10T08:35:13.030970+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:35:13.030970+0000","last_undegraded":"2024-02-10T08:35:13.030970+0000","last_fullsized":"2024-02-10T08:35:13.030970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612289+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:34:56.612289+0000","last_peered":"2024-02-10T08:34:56.612289+0000","last_clean":"2024-02-10T08:34:56.612289+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:34:56.612289+0000","last_undegraded":"2024-02-10T08:34:56.612289+0000","last_fullsized":"2024-02-10T08:34:56.612289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.613140+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:34:56.613140+0000","last_peered":"2024-02-10T08:34:56.613140+0000","last_clean":"2024-02-10T08:34:56.613140+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:34:56.613140+0000","last_undegraded":"2024-02-10T08:34:56.613140+0000","last_fullsized":"2024-02-10T08:34:56.613140+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":67,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612850+0000","last_change":"2024-02-10T08:34:44.381291+0000","last_active":"2024-02-10T08:34:56.612850+0000","last_peered":"2024-02-10T08:34:56.612850+0000","last_clean":"2024-02-10T08:34:56.612850+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:34:56.612850+0000","last_undegraded":"2024-02-10T08:34:56.612850+0000","last_fullsized":"2024-02-10T08:34:56.612850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:44.381208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.381208+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.697976+0000","last_change":"2024-02-10T08:34:29.554010+0000","last_active":"2024-02-10T08:34:56.697976+0000","last_peered":"2024-02-10T08:34:56.697976+0000","last_clean":"2024-02-10T08:34:56.697976+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:34:56.697976+0000","last_undegraded":"2024-02-10T08:34:56.697976+0000","last_fullsized":"2024-02-10T08:34:56.697976+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.553914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.553914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.612879+0000","last_change":"2024-02-10T08:34:50.747486+0000","last_active":"2024-02-10T08:34:56.612879+0000","last_peered":"2024-02-10T08:34:56.612879+0000","last_clean":"2024-02-10T08:34:56.612879+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:34:56.612879+0000","last_undegraded":"2024-02-10T08:34:56.612879+0000","last_fullsized":"2024-02-10T08:34:56.612879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:50.747424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.747424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.614468+0000","last_change":"2024-02-10T08:34:20.413361+0000","last_active":"2024-02-10T08:34:56.614468+0000","last_peered":"2024-02-10T08:34:56.614468+0000","last_clean":"2024-02-10T08:34:56.614468+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:34:56.614468+0000","last_undegraded":"2024-02-10T08:34:56.614468+0000","last_fullsized":"2024-02-10T08:34:56.614468+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.413281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.413281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.698426+0000","last_change":"2024-02-10T08:34:41.701563+0000","last_active":"2024-02-10T08:34:56.698426+0000","last_peered":"2024-02-10T08:34:56.698426+0000","last_clean":"2024-02-10T08:34:56.698426+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:34:56.698426+0000","last_undegraded":"2024-02-10T08:34:56.698426+0000","last_fullsized":"2024-02-10T08:34:56.698426+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:34.592074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:34.592074+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-02-10T08:34:56.774432+0000","last_change":"2024-02-10T08:34:56.475601+0000","last_active":"2024-02-10T08:34:56.774432+0000","last_peered":"2024-02-10T08:34:56.774432+0000","last_clean":"2024-02-10T08:34:56.774432+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:34:56.774432+0000","last_undegraded":"2024-02-10T08:34:56.774432+0000","last_fullsized":"2024-02-10T08:34:56.774432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.475529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.475529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:03.720397+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:35:03.720397+0000","last_peered":"2024-02-10T08:35:03.720397+0000","last_clean":"2024-02-10T08:35:03.720397+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:35:03.720397+0000","last_undegraded":"2024-02-10T08:35:03.720397+0000","last_fullsized":"2024-02-10T08:35:03.720397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:05.510749+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:35:05.510749+0000","last_peered":"2024-02-10T08:35:05.510749+0000","last_clean":"2024-02-10T08:35:05.510749+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:35:05.510749+0000","last_undegraded":"2024-02-10T08:35:05.510749+0000","last_fullsized":"2024-02-10T08:35:05.510749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-10T08:35:06.511282+0000","last_change":"2024-02-10T08:34:42.752203+0000","last_active":"2024-02-10T08:35:06.511282+0000","last_peered":"2024-02-10T08:35:06.511282+0000","last_clean":"2024-02-10T08:35:06.511282+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:35:06.511282+0000","last_undegraded":"2024-02-10T08:35:06.511282+0000","last_fullsized":"2024-02-10T08:35:06.511282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:42.752097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:42.752097+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299418,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297908,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447244,"statfs":{"total":95995035648,"available":95689977856,"internally_reserved":0,"allocated":831488,"data_stored":340065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.034,"15min":1.034},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.054,"15min":1.054},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.496,"15min":0.499},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.503},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.747,"15min":0.747},"last":0.41299999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561056,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297888,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447264,"statfs":{"total":95995035648,"available":95689998336,"internally_reserved":0,"allocated":811008,"data_stored":339973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.69199999999999995}]},{"osd":1,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.498,"15min":0.498},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.623,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.501},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.67000000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.375,"15min":0.372},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56000000000000005,"5min":0.623,"15min":0.623},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.59199999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.39600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":341385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.495,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.309},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.47499999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.377,"15min":0.376},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.23200000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53000000000000003,"5min":0.624,"15min":0.624},"last":0.32000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41899999999999998,"15min":0.42899999999999999},"min":{"1min":0.27100000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.53000000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":1.294,"15min":1.294},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77700000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73299999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297900,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447252,"statfs":{"total":95995035648,"available":95689986048,"internally_reserved":0,"allocated":823296,"data_stored":339089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.30299999999999999},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.308},"min":{"1min":0.27300000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.215}]},{"osd":2,"last update":"Sat Feb 10 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36899999999999999,"15min":0.36499999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.498,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.52600000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.249}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.503,"5min":0.624,"15min":0.624},"last":0.308},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.501,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.29399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75900000000000001,"5min":0.878,"15min":0.878},"last":0.38100000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76900000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.441},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.49399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297896,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447256,"statfs":{"total":95995035648,"available":95689990144,"internally_reserved":0,"allocated":819200,"data_stored":339768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35499999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.51100000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.38500000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.52800000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81899999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.621,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.63400000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":340261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35599999999999998,"15min":0.36899999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39600000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.625}]},{"osd":2,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.437,"15min":0.46000000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57999999999999996,"15min":0.60799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66700000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55800000000000005,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.72499999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.64000000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.88200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.85399999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.87}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297912,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447240,"statfs":{"total":95995035648,"available":95689973760,"internally_reserved":0,"allocated":835584,"data_stored":340431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.998,"5min":1,"15min":1},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.749,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.379,"15min":0.371},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.441,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58799999999999997}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297904,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447248,"statfs":{"total":95995035648,"available":95689981952,"internally_reserved":0,"allocated":827392,"data_stored":339407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.373,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.505,"15min":0.505},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76000000000000001,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.253,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":1.081,"15min":1.081},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48499999999999999,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.63},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:35:22.110 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T08:35:22.111 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T08:35:22.111 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T08:35:22.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T08:35:22.380 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:21.076255+0000 mgr.y (mgr.14553) 85 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:21.353034+0000 mgr.y (mgr.14553) 86 : audit [DBG] from='client.34667 v1:172.21.15.107:0/454320160' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:21.076255+0000 mgr.y (mgr.14553) 85 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:21.353034+0000 mgr.y (mgr.14553) 86 : audit [DBG] from='client.34667 v1:172.21.15.107:0/454320160' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:21.076255+0000 mgr.y (mgr.14553) 85 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:21.353034+0000 mgr.y (mgr.14553) 86 : audit [DBG] from='client.34667 v1:172.21.15.107:0/454320160' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:23.912 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:22.014354+0000 mgr.y (mgr.14553) 87 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:23.912 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:22.890881+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:35:23.912 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:22.893257+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:35:23.912 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:22.014354+0000 mgr.y (mgr.14553) 87 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:23.912 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:22.890881+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:35:23.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:22.893257+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:35:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:22.014354+0000 mgr.y (mgr.14553) 87 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:22.890881+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:35:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:22.893257+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.137+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3706787737 >> v1:172.21.15.107:6790/0 conn(0x7f2c34106660 legacy=0x7f2c34108a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.137+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3706787737 shutdown_connections 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.137+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3706787737 >> 172.21.15.107:0/3706787737 conn(0x7f2c340f2510 msgr2=0x7f2c340f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.137+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3706787737 shutdown_connections 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.137+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3706787737 wait complete. 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 librados: starting msgr at 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 librados: starting objecter 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 Processor -- start 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 -- start start 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 librados: setting wanted keys 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 librados: calling monclient init 2024-02-10T08:35:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c34119620 con 0x7f2c34106660 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c3411a800 con 0x7f2c340f7320 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c38c1f700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c3411ba00 con 0x7f2c340f6830 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c3259c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2c340f6830 0x7f2c34102c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53082/0 (socket says 172.21.15.107:53082) 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.138+0000 7f2c3259c700 1 -- 172.21.15.107:0/3845113889 learned_addr learned my addr 172.21.15.107:0/3845113889 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter ms_handle_connect 0x7f2c340f6830 2024-02-10T08:35:24.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter ms_handle_connect 0x7f2c34106660 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765773255 0 0) 0x7f2c3411ba00 con 0x7f2c340f6830 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c1c003030 con 0x7f2c340f6830 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter ms_handle_connect 0x7f2c340f7320 2024-02-10T08:35:24.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184421045 0 0) 0x7f2c34119620 con 0x7f2c34106660 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c3411ba00 con 0x7f2c34106660 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031073335 0 0) 0x7f2c3411a800 con 0x7f2c340f7320 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c34119620 con 0x7f2c340f7320 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490027166 0 0) 0x7f2c1c003030 con 0x7f2c340f6830 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c3411a800 con 0x7f2c340f6830 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2c24001940 con 0x7f2c340f6830 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2249696822 0 0) 0x7f2c3411ba00 con 0x7f2c34106660 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c1c003030 con 0x7f2c34106660 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2c2c0017b0 con 0x7f2c34106660 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534039211 0 0) 0x7f2c34119620 con 0x7f2c340f7320 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c3411ba00 con 0x7f2c340f7320 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2c280019f0 con 0x7f2c340f7320 2024-02-10T08:35:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 754055481 0 0) 0x7f2c3411a800 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 >> v1:172.21.15.195:6789/0 conn(0x7f2c340f7320 legacy=0x7f2c34103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 >> v1:172.21.15.107:6789/0 conn(0x7f2c34106660 legacy=0x7f2c34117ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3411cc00 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c34119810 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c38c1f700 10 client.34673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c34119c80 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c38c1f700 1 librados: init done 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2c24001cd0 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.139+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2c24005bc0 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.140+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c14004f20 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.140+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2c240021b0 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(82..82 src has 30..82) v4 ==== 6172+0+0 (unknown 4166463881 0 0) 0x7f2c240785c0 con 0x7f2c340f6830 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 10 client.34673.objecter ms_dispatch 0x7f2c34001550 osd_map(82..82 src has 30..82) v4 2024-02-10T08:35:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 3 client.34673.objecter handle_osd_map got epochs [82,82] > 0 2024-02-10T08:35:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 3 client.34673.objecter handle_osd_map decoding full epoch 82 2024-02-10T08:35:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 20 client.34673.objecter dump_active .. 0 homeless 2024-02-10T08:35:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.141+0000 7f2c237fe700 10 client.34673.objecter ms_handle_connect 0x7f2c1c063ce0 2024-02-10T08:35:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.145+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2c2404c110 con 0x7f2c340f6830 2024-02-10T08:35:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.345+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2c14005750 con 0x7f2c340f6830 2024-02-10T08:35:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.345+0000 7f2c237fe700 1 -- 172.21.15.107:0/3845113889 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (unknown 3730786198 0 3397129104) 0x7f2c2404a520 con 0x7f2c340f6830 2024-02-10T08:35:24.347 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:35:24.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 10 librados: watch_flush enter 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 10 librados: watch_flush exit 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 20 client.34673.objecter shutdown clearing up homeless session... 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 10 client.34673.objecter successfully canceled tick 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2c1c063ce0 legacy=0x7f2c1c0661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 >> v1:172.21.15.107:6790/0 conn(0x7f2c340f6830 legacy=0x7f2c34102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 shutdown_connections 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 >> 172.21.15.107:0/3845113889 conn(0x7f2c340f2510 msgr2=0x7f2c34109a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 shutdown_connections 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 -- 172.21.15.107:0/3845113889 wait complete. 2024-02-10T08:35:24.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:24.347+0000 7f2c38c1f700 1 librados: shutdown 2024-02-10T08:35:24.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:23.381134+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:23.381185+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:23.698772+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:23.875925+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.18 scrub starts 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:23.876864+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.18 scrub ok 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:24.146445+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:24.146493+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:24.347232+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.107:0/3845113889' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:23.381134+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:24.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:23.381185+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:23.698772+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:23.875925+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.18 scrub starts 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:23.876864+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.18 scrub ok 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:24.146445+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:24.146493+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:24.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:24.347232+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.107:0/3845113889' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:23.381134+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:23.381185+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:23.698772+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:23.875925+0000 osd.4 (osd.4) 41 : cluster [DBG] 5.18 scrub starts 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:23.876864+0000 osd.4 (osd.4) 42 : cluster [DBG] 5.18 scrub ok 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:24.146445+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:24.146493+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:24.347232+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.107:0/3845113889' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T08:35:25.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T08:35:25.110 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T08:35:25.110 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T08:35:25.110 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-10T08:35:25.122 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-10T08:35:25.123 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri10-testing-2024-02-08-0854-pacific'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-tracker-64343-pacific', 'sha1': '57bd6abdec7bb457ae7999d9c96682e9ac678e27'}} 2024-02-10T08:35:25.123 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-10T08:35:25.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-10T08:35:25.395 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:25.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:24.014845+0000 mgr.y (mgr.14553) 88 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:25.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:24.709045+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-10T08:35:25.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:24.014845+0000 mgr.y (mgr.14553) 88 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:25.900 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:24.709045+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-10T08:35:25.900 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:25] "GET /metrics HTTP/1.1" 200 172136 "" "Prometheus/2.33.4" 2024-02-10T08:35:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:24.014845+0000 mgr.y (mgr.14553) 88 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:24.709045+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.223+0000 7f55461da700 1 -- 172.21.15.107:0/3731495055 >> v1:172.21.15.107:6790/0 conn(0x7f55400fc910 legacy=0x7f55400fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.223+0000 7f55461da700 1 -- 172.21.15.107:0/3731495055 shutdown_connections 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.223+0000 7f55461da700 1 -- 172.21.15.107:0/3731495055 >> 172.21.15.107:0/3731495055 conn(0x7f55400f2510 msgr2=0x7f55400f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.223+0000 7f55461da700 1 -- 172.21.15.107:0/3731495055 shutdown_connections 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.223+0000 7f55461da700 1 -- 172.21.15.107:0/3731495055 wait complete. 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 librados: starting msgr at 2024-02-10T08:35:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 librados: starting objecter 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 Processor -- start 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 -- start start 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 librados: setting wanted keys 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 librados: calling monclient init 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5540119520 con 0x7f55400ff710 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f554011a700 con 0x7f5540103250 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.224+0000 7f55461da700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f554011b900 con 0x7f55400fc910 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f55400ff710 0x7f5540061190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37388/0 (socket says 172.21.15.107:37388) 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553effd700 1 -- 172.21.15.107:0/229855065 learned_addr learned my addr 172.21.15.107:0/229855065 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter ms_handle_connect 0x7f55400ff710 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter ms_handle_connect 0x7f55400fc910 2024-02-10T08:35:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter ms_handle_connect 0x7f5540103250 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207571521 0 0) 0x7f5540119520 con 0x7f55400ff710 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5528003630 con 0x7f55400ff710 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997653316 0 0) 0x7f554011b900 con 0x7f55400fc910 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5540119520 con 0x7f55400fc910 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489543011 0 0) 0x7f554011a700 con 0x7f5540103250 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554011b900 con 0x7f5540103250 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825400918 0 0) 0x7f5528003630 con 0x7f55400ff710 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f554011a700 con 0x7f55400ff710 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f55340019d0 con 0x7f55400ff710 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664238977 0 0) 0x7f5540119520 con 0x7f55400fc910 2024-02-10T08:35:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5528003630 con 0x7f55400fc910 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5530002050 con 0x7f55400fc910 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400464096 0 0) 0x7f554011b900 con 0x7f5540103250 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5540119520 con 0x7f5540103250 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5538001b00 con 0x7f5540103250 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1090765085 0 0) 0x7f554011a700 con 0x7f55400ff710 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 >> v1:172.21.15.107:6790/0 conn(0x7f55400fc910 legacy=0x7f5540060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 >> v1:172.21.15.195:6789/0 conn(0x7f5540103250 legacy=0x7f5540117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554011cb00 con 0x7f55400ff710 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554011baf0 con 0x7f55400ff710 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f55461da700 10 client.14760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f554011bfe0 con 0x7f55400ff710 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f55461da700 1 librados: init done 2024-02-10T08:35:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5534002380 con 0x7f55400ff710 2024-02-10T08:35:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.225+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5534005af0 con 0x7f55400ff710 2024-02-10T08:35:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.226+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c004f20 con 0x7f55400ff710 2024-02-10T08:35:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f55340024f0 con 0x7f55400ff710 2024-02-10T08:35:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(83..83 src has 30..83) v4 ==== 6172+0+0 (unknown 4016300065 0 0) 0x7f55340785c0 con 0x7f55400ff710 2024-02-10T08:35:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 10 client.14760.objecter ms_dispatch 0x7f5540001550 osd_map(83..83 src has 30..83) v4 2024-02-10T08:35:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 3 client.14760.objecter handle_osd_map got epochs [83,83] > 0 2024-02-10T08:35:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 3 client.14760.objecter handle_osd_map decoding full epoch 83 2024-02-10T08:35:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 20 client.14760.objecter dump_active .. 0 homeless 2024-02-10T08:35:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.227+0000 7f553cff9700 10 client.14760.objecter ms_handle_connect 0x7f5528063d20 2024-02-10T08:35:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.231+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f553404c110 con 0x7f55400ff710 2024-02-10T08:35:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.395+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 --> v1:172.21.15.107:6789/0 -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f552c005750 con 0x7f55400ff710 2024-02-10T08:35:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.399+0000 7f553cff9700 1 -- 172.21.15.107:0/229855065 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) v1 ==== 136+0+0 (unknown 398629962 0 0) 0x7f553404a520 con 0x7f55400ff710 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 10 librados: watch_flush enter 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 10 librados: watch_flush exit 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 20 client.14760.objecter shutdown clearing up homeless session... 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 10 client.14760.objecter successfully canceled tick 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5528063d20 legacy=0x7f55280661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.401+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 >> v1:172.21.15.107:6789/0 conn(0x7f55400ff710 legacy=0x7f5540061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:27.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.402+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 shutdown_connections 2024-02-10T08:35:27.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.402+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 >> 172.21.15.107:0/229855065 conn(0x7f55400f2510 msgr2=0x7f55401066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:27.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.402+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 shutdown_connections 2024-02-10T08:35:27.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.402+0000 7f55461da700 1 -- 172.21.15.107:0/229855065 wait complete. 2024-02-10T08:35:27.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:27.402+0000 7f55461da700 1 librados: shutdown 2024-02-10T08:35:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE Bus STOPPING 2024-02-10T08:35:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:35:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE Bus STOPPED 2024-02-10T08:35:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE Bus STARTING 2024-02-10T08:35:27.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE Serving on http://:::9283 2024-02-10T08:35:27.681 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:35:27] ENGINE Bus STARTED 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:26.015502+0000 mgr.y (mgr.14553) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:27.048941+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:27.051830+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:27.401334+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.107:0/229855065' entity='client.admin' 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:27.434846+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:27.436219+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:35:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:27.436727+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:35:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:27.441312+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:26.015502+0000 mgr.y (mgr.14553) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:27.048941+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:27.051830+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:27.401334+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.107:0/229855065' entity='client.admin' 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:27.434846+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:27.436219+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:27.436727+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:27.441312+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:28.148 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:26.015502+0000 mgr.y (mgr.14553) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:27.048941+0000 osd.7 (osd.7) 41 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:27.051830+0000 osd.7 (osd.7) 42 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:27.401334+0000 mon.a (mon.0) 718 : audit [INF] from='client.? v1:172.21.15.107:0/229855065' entity='client.admin' 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:27.434846+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:27.436219+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:27.436727+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:35:28.149 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:27.441312+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:28.170 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-10T08:35:28.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T08:35:28.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:35:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:29.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:28.016039+0000 mgr.y (mgr.14553) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:29.913 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:28.016039+0000 mgr.y (mgr.14553) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:28.016039+0000 mgr.y (mgr.14553) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.135+0000 7fe6dba73700 1 -- 172.21.15.107:0/985256205 >> v1:172.21.15.107:6789/0 conn(0x7fe6d40fda50 legacy=0x7fe6d40fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.135+0000 7fe6dba73700 1 -- 172.21.15.107:0/985256205 shutdown_connections 2024-02-10T08:35:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.135+0000 7fe6dba73700 1 -- 172.21.15.107:0/985256205 >> 172.21.15.107:0/985256205 conn(0x7fe6d40f3640 msgr2=0x7fe6d40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.135+0000 7fe6dba73700 1 -- 172.21.15.107:0/985256205 shutdown_connections 2024-02-10T08:35:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.135+0000 7fe6dba73700 1 -- 172.21.15.107:0/985256205 wait complete. 2024-02-10T08:35:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 librados: starting msgr at 2024-02-10T08:35:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 librados: starting objecter 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 Processor -- start 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 -- start start 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 librados: setting wanted keys 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 librados: calling monclient init 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6d40fd000 con 0x7fe6d40fda50 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6d40fd190 con 0x7fe6d41007f0 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6dba73700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6d40fd380 con 0x7fe6d4104330 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6d980f700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe6d40fda50 0x7fe6d40fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42966/0 (socket says 172.21.15.107:42966) 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6d980f700 1 -- 172.21.15.107:0/47902946 learned_addr learned my addr 172.21.15.107:0/47902946 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6da010700 1 --1- 172.21.15.107:0/47902946 >> v1:172.21.15.107:6790/0 conn(0x7fe6d4104330 0x7fe6d41184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39342/0 (socket says 172.21.15.107:39342) 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6caf78700 10 client.?.objecter ms_handle_connect 0x7fe6d40fda50 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6caf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6caf78700 10 client.?.objecter ms_handle_connect 0x7fe6d4104330 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.136+0000 7fe6caf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 10 client.?.objecter ms_handle_connect 0x7fe6d41007f0 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234477891 0 0) 0x7fe6d40fd000 con 0x7fe6d40fda50 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c0003800 con 0x7fe6d40fda50 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778962340 0 0) 0x7fe6d40fd380 con 0x7fe6d4104330 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d40fd000 con 0x7fe6d4104330 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111742760 0 0) 0x7fe6d40fd190 con 0x7fe6d41007f0 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d40fd380 con 0x7fe6d41007f0 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 466904307 0 0) 0x7fe6c0003800 con 0x7fe6d40fda50 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6d40fd190 con 0x7fe6d40fda50 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6d0002f20 con 0x7fe6d40fda50 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927862070 0 0) 0x7fe6d40fd000 con 0x7fe6d4104330 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6c0003800 con 0x7fe6d4104330 2024-02-10T08:35:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6cc001ca0 con 0x7fe6d4104330 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 445877515 0 0) 0x7fe6d40fd190 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 >> v1:172.21.15.107:6790/0 conn(0x7fe6d4104330 legacy=0x7fe6d41184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 >> v1:172.21.15.195:6789/0 conn(0x7fe6d41007f0 legacy=0x7fe6d4114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d411cc00 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d4119c30 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6dba73700 10 client.14766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d411a0a0 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6dba73700 1 librados: init done 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe6d0002790 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.137+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6d00039a0 con 0x7fe6d40fda50 2024-02-10T08:35:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.138+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8004f20 con 0x7fe6d40fda50 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.138+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe6d00031c0 con 0x7fe6d40fda50 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(83..83 src has 30..83) v4 ==== 6172+0+0 (unknown 4016300065 0 0) 0x7fe6d0078990 con 0x7fe6d40fda50 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 10 client.14766.objecter ms_dispatch 0x7fe6d4001550 osd_map(83..83 src has 30..83) v4 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 3 client.14766.objecter handle_osd_map got epochs [83,83] > 0 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 3 client.14766.objecter handle_osd_map decoding full epoch 83 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 20 client.14766.objecter dump_active .. 0 homeless 2024-02-10T08:35:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.139+0000 7fe6caf78700 10 client.14766.objecter ms_handle_connect 0x7fe6c0063c50 2024-02-10T08:35:30.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.143+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe6d004c4e0 con 0x7fe6d40fda50 2024-02-10T08:35:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.302+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6b8000b90 con 0x7fe6c0063c50 2024-02-10T08:35:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.312+0000 7fe6caf78700 1 -- 172.21.15.107:0/47902946 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357358 (unknown 2965378022 0 729608370) 0x7fe6b8000b90 con 0x7fe6c0063c50 2024-02-10T08:35:30.318 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 10 librados: watch_flush enter 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 10 librados: watch_flush exit 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 20 client.14766.objecter shutdown clearing up homeless session... 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 10 client.14766.objecter successfully canceled tick 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe6c0063c50 legacy=0x7fe6c0066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 >> v1:172.21.15.107:6789/0 conn(0x7fe6d40fda50 legacy=0x7fe6d40fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 shutdown_connections 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 >> 172.21.15.107:0/47902946 conn(0x7fe6d40f3640 msgr2=0x7fe6d41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 shutdown_connections 2024-02-10T08:35:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 -- 172.21.15.107:0/47902946 wait complete. 2024-02-10T08:35:30.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:30.315+0000 7fe6dba73700 1 librados: shutdown 2024-02-10T08:35:30.322 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:35:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:29.831199+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:29.834109+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.120486+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.b scrub starts 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.123823+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.b scrub ok 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.238745+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-10T08:35:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.241658+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:29.831199+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:29.834109+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.120486+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.b scrub starts 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.123823+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.b scrub ok 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.238745+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-10T08:35:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.241658+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-10T08:35:31.036 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-02-10T08:35:30.016263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":841,"num_read_kb":593,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":493,"ondisk_log_size":493,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381696,"kb_used_data":4928,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579520,"statfs":{"total":767960285184,"available":765521428480,"internally_reserved":0,"allocated":5046272,"data_stored":2306347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002456"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873473+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:35:23.873473+0000","last_peered":"2024-02-10T08:35:23.873473+0000","last_clean":"2024-02-10T08:35:23.873473+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:35:23.873473+0000","last_undegraded":"2024-02-10T08:35:23.873473+0000","last_fullsized":"2024-02-10T08:35:23.873473+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.876981+0000","last_change":"2024-02-10T08:35:23.876952+0000","last_active":"2024-02-10T08:35:23.876981+0000","last_peered":"2024-02-10T08:35:23.876981+0000","last_clean":"2024-02-10T08:35:23.876981+0000","last_became_active":"2024-02-10T08:33:22.117534+0000","last_became_peered":"2024-02-10T08:33:22.117534+0000","last_unstale":"2024-02-10T08:35:23.876981+0000","last_undegraded":"2024-02-10T08:35:23.876981+0000","last_fullsized":"2024-02-10T08:35:23.876981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:23.876871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:23.876871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873247+0000","last_change":"2024-02-10T08:34:41.648797+0000","last_active":"2024-02-10T08:35:23.873247+0000","last_peered":"2024-02-10T08:35:23.873247+0000","last_clean":"2024-02-10T08:35:23.873247+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:35:23.873247+0000","last_undegraded":"2024-02-10T08:35:23.873247+0000","last_fullsized":"2024-02-10T08:35:23.873247+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:41.648715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:41.648715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":63,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874167+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:35:23.874167+0000","last_peered":"2024-02-10T08:35:23.874167+0000","last_clean":"2024-02-10T08:35:23.874167+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:35:23.874167+0000","last_undegraded":"2024-02-10T08:35:23.874167+0000","last_fullsized":"2024-02-10T08:35:23.874167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874073+0000","last_change":"2024-02-10T08:34:38.118431+0000","last_active":"2024-02-10T08:35:23.874073+0000","last_peered":"2024-02-10T08:35:23.874073+0000","last_clean":"2024-02-10T08:35:23.874073+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:35:23.874073+0000","last_undegraded":"2024-02-10T08:35:23.874073+0000","last_fullsized":"2024-02-10T08:35:23.874073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.118347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.118347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698156+0000","last_change":"2024-02-10T08:34:37.332223+0000","last_active":"2024-02-10T08:35:23.698156+0000","last_peered":"2024-02-10T08:35:23.698156+0000","last_clean":"2024-02-10T08:35:23.698156+0000","last_became_active":"2024-02-10T08:33:22.121607+0000","last_became_peered":"2024-02-10T08:33:22.121607+0000","last_unstale":"2024-02-10T08:35:23.698156+0000","last_undegraded":"2024-02-10T08:35:23.698156+0000","last_fullsized":"2024-02-10T08:35:23.698156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.332140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.332140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874290+0000","last_change":"2024-02-10T08:34:23.594197+0000","last_active":"2024-02-10T08:35:23.874290+0000","last_peered":"2024-02-10T08:35:23.874290+0000","last_clean":"2024-02-10T08:35:23.874290+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:35:23.874290+0000","last_undegraded":"2024-02-10T08:35:23.874290+0000","last_fullsized":"2024-02-10T08:35:23.874290+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.594135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.594135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873372+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:35:23.873372+0000","last_peered":"2024-02-10T08:35:23.873372+0000","last_clean":"2024-02-10T08:35:23.873372+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:35:23.873372+0000","last_undegraded":"2024-02-10T08:35:23.873372+0000","last_fullsized":"2024-02-10T08:35:23.873372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873545+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:35:23.873545+0000","last_peered":"2024-02-10T08:35:23.873545+0000","last_clean":"2024-02-10T08:35:23.873545+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:35:23.873545+0000","last_undegraded":"2024-02-10T08:35:23.873545+0000","last_fullsized":"2024-02-10T08:35:23.873545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698215+0000","last_change":"2024-02-10T08:35:00.936215+0000","last_active":"2024-02-10T08:35:23.698215+0000","last_peered":"2024-02-10T08:35:23.698215+0000","last_clean":"2024-02-10T08:35:23.698215+0000","last_became_active":"2024-02-10T08:33:22.119141+0000","last_became_peered":"2024-02-10T08:33:22.119141+0000","last_unstale":"2024-02-10T08:35:23.698215+0000","last_undegraded":"2024-02-10T08:35:23.698215+0000","last_fullsized":"2024-02-10T08:35:23.698215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698810+0000","last_change":"2024-02-10T08:34:26.673356+0000","last_active":"2024-02-10T08:35:23.698810+0000","last_peered":"2024-02-10T08:35:23.698810+0000","last_clean":"2024-02-10T08:35:23.698810+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:35:23.698810+0000","last_undegraded":"2024-02-10T08:35:23.698810+0000","last_fullsized":"2024-02-10T08:35:23.698810+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:26.673263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.673263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874645+0000","last_change":"2024-02-10T08:34:50.094010+0000","last_active":"2024-02-10T08:35:23.874645+0000","last_peered":"2024-02-10T08:35:23.874645+0000","last_clean":"2024-02-10T08:35:23.874645+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:35:23.874645+0000","last_undegraded":"2024-02-10T08:35:23.874645+0000","last_fullsized":"2024-02-10T08:35:23.874645+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:34:50.093915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.093915+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873384+0000","last_change":"2024-02-10T08:34:51.681056+0000","last_active":"2024-02-10T08:35:23.873384+0000","last_peered":"2024-02-10T08:35:23.873384+0000","last_clean":"2024-02-10T08:35:23.873384+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:35:23.873384+0000","last_undegraded":"2024-02-10T08:35:23.873384+0000","last_fullsized":"2024-02-10T08:35:23.873384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.680986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.680986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874393+0000","last_change":"2024-02-10T08:35:01.492010+0000","last_active":"2024-02-10T08:35:23.874393+0000","last_peered":"2024-02-10T08:35:23.874393+0000","last_clean":"2024-02-10T08:35:23.874393+0000","last_became_active":"2024-02-10T08:33:22.122241+0000","last_became_peered":"2024-02-10T08:33:22.122241+0000","last_unstale":"2024-02-10T08:35:23.874393+0000","last_undegraded":"2024-02-10T08:35:23.874393+0000","last_fullsized":"2024-02-10T08:35:23.874393+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698601+0000","last_change":"2024-02-10T08:34:56.956608+0000","last_active":"2024-02-10T08:35:23.698601+0000","last_peered":"2024-02-10T08:35:23.698601+0000","last_clean":"2024-02-10T08:35:23.698601+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:35:23.698601+0000","last_undegraded":"2024-02-10T08:35:23.698601+0000","last_fullsized":"2024-02-10T08:35:23.698601+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.956483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.956483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874108+0000","last_change":"2024-02-10T08:34:59.084224+0000","last_active":"2024-02-10T08:35:23.874108+0000","last_peered":"2024-02-10T08:35:23.874108+0000","last_clean":"2024-02-10T08:35:23.874108+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:35:23.874108+0000","last_undegraded":"2024-02-10T08:35:23.874108+0000","last_fullsized":"2024-02-10T08:35:23.874108+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:59.084151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:59.084151+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874161+0000","last_change":"2024-02-10T08:34:52.516328+0000","last_active":"2024-02-10T08:35:23.874161+0000","last_peered":"2024-02-10T08:35:23.874161+0000","last_clean":"2024-02-10T08:35:23.874161+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:35:23.874161+0000","last_undegraded":"2024-02-10T08:35:23.874161+0000","last_fullsized":"2024-02-10T08:35:23.874161+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.516236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.516236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874614+0000","last_change":"2024-02-10T08:35:07.038806+0000","last_active":"2024-02-10T08:35:23.874614+0000","last_peered":"2024-02-10T08:35:23.874614+0000","last_clean":"2024-02-10T08:35:23.874614+0000","last_became_active":"2024-02-10T08:33:22.116085+0000","last_became_peered":"2024-02-10T08:33:22.116085+0000","last_unstale":"2024-02-10T08:35:23.874614+0000","last_undegraded":"2024-02-10T08:35:23.874614+0000","last_fullsized":"2024-02-10T08:35:23.874614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873555+0000","last_change":"2024-02-10T08:34:37.616548+0000","last_active":"2024-02-10T08:35:23.873555+0000","last_peered":"2024-02-10T08:35:23.873555+0000","last_clean":"2024-02-10T08:35:23.873555+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:35:23.873555+0000","last_undegraded":"2024-02-10T08:35:23.873555+0000","last_fullsized":"2024-02-10T08:35:23.873555+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.616436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.616436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873925+0000","last_change":"2024-02-10T08:34:33.601286+0000","last_active":"2024-02-10T08:35:23.873925+0000","last_peered":"2024-02-10T08:35:23.873925+0000","last_clean":"2024-02-10T08:35:23.873925+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:35:23.873925+0000","last_undegraded":"2024-02-10T08:35:23.873925+0000","last_fullsized":"2024-02-10T08:35:23.873925+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:33.601201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.601201+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875282+0000","last_change":"2024-02-10T08:35:17.895268+0000","last_active":"2024-02-10T08:35:23.875282+0000","last_peered":"2024-02-10T08:35:23.875282+0000","last_clean":"2024-02-10T08:35:23.875282+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:35:23.875282+0000","last_undegraded":"2024-02-10T08:35:23.875282+0000","last_fullsized":"2024-02-10T08:35:23.875282+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:17.895163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:17.895163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699274+0000","last_change":"2024-02-10T08:34:39.357828+0000","last_active":"2024-02-10T08:35:23.699274+0000","last_peered":"2024-02-10T08:35:23.699274+0000","last_clean":"2024-02-10T08:35:23.699274+0000","last_became_active":"2024-02-10T08:33:22.120631+0000","last_became_peered":"2024-02-10T08:33:22.120631+0000","last_unstale":"2024-02-10T08:35:23.699274+0000","last_undegraded":"2024-02-10T08:35:23.699274+0000","last_fullsized":"2024-02-10T08:35:23.699274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.357766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873911+0000","last_change":"2024-02-10T08:34:53.298291+0000","last_active":"2024-02-10T08:35:23.873911+0000","last_peered":"2024-02-10T08:35:23.873911+0000","last_clean":"2024-02-10T08:35:23.873911+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:35:23.873911+0000","last_undegraded":"2024-02-10T08:35:23.873911+0000","last_fullsized":"2024-02-10T08:35:23.873911+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.298226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.298226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699877+0000","last_change":"2024-02-10T08:34:26.526319+0000","last_active":"2024-02-10T08:35:23.699877+0000","last_peered":"2024-02-10T08:35:23.699877+0000","last_clean":"2024-02-10T08:35:23.699877+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:35:23.699877+0000","last_undegraded":"2024-02-10T08:35:23.699877+0000","last_fullsized":"2024-02-10T08:35:23.699877+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:34:26.526206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.526206+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699094+0000","last_change":"2024-02-10T08:34:52.285891+0000","last_active":"2024-02-10T08:35:23.699094+0000","last_peered":"2024-02-10T08:35:23.699094+0000","last_clean":"2024-02-10T08:35:23.699094+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:35:23.699094+0000","last_undegraded":"2024-02-10T08:35:23.699094+0000","last_fullsized":"2024-02-10T08:35:23.699094+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.285824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.285824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873983+0000","last_change":"2024-02-10T08:35:01.687455+0000","last_active":"2024-02-10T08:35:23.873983+0000","last_peered":"2024-02-10T08:35:23.873983+0000","last_clean":"2024-02-10T08:35:23.873983+0000","last_became_active":"2024-02-10T08:33:22.122436+0000","last_became_peered":"2024-02-10T08:33:22.122436+0000","last_unstale":"2024-02-10T08:35:23.873983+0000","last_undegraded":"2024-02-10T08:35:23.873983+0000","last_fullsized":"2024-02-10T08:35:23.873983+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.687386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873948+0000","last_change":"2024-02-10T08:34:24.604411+0000","last_active":"2024-02-10T08:35:23.873948+0000","last_peered":"2024-02-10T08:35:23.873948+0000","last_clean":"2024-02-10T08:35:23.873948+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:35:23.873948+0000","last_undegraded":"2024-02-10T08:35:23.873948+0000","last_fullsized":"2024-02-10T08:35:23.873948+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.604347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.604347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874593+0000","last_change":"2024-02-10T08:34:36.450017+0000","last_active":"2024-02-10T08:35:23.874593+0000","last_peered":"2024-02-10T08:35:23.874593+0000","last_clean":"2024-02-10T08:35:23.874593+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:35:23.874593+0000","last_undegraded":"2024-02-10T08:35:23.874593+0000","last_fullsized":"2024-02-10T08:35:23.874593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:36.449943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.449943+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698961+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:35:23.698961+0000","last_peered":"2024-02-10T08:35:23.698961+0000","last_clean":"2024-02-10T08:35:23.698961+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:35:23.698961+0000","last_undegraded":"2024-02-10T08:35:23.698961+0000","last_fullsized":"2024-02-10T08:35:23.698961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'9","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874686+0000","last_change":"2024-02-10T08:34:35.421984+0000","last_active":"2024-02-10T08:35:23.874686+0000","last_peered":"2024-02-10T08:35:23.874686+0000","last_clean":"2024-02-10T08:35:23.874686+0000","last_became_active":"2024-02-10T08:33:22.119251+0000","last_became_peered":"2024-02-10T08:33:22.119251+0000","last_unstale":"2024-02-10T08:35:23.874686+0000","last_undegraded":"2024-02-10T08:35:23.874686+0000","last_fullsized":"2024-02-10T08:35:23.874686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_deep_scrub":"74'9","last_deep_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.421866+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875134+0000","last_change":"2024-02-10T08:34:41.695373+0000","last_active":"2024-02-10T08:35:23.875134+0000","last_peered":"2024-02-10T08:35:23.875134+0000","last_clean":"2024-02-10T08:35:23.875134+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:35:23.875134+0000","last_undegraded":"2024-02-10T08:35:23.875134+0000","last_fullsized":"2024-02-10T08:35:23.875134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.661214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.661214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873930+0000","last_change":"2024-02-10T08:34:43.680148+0000","last_active":"2024-02-10T08:35:23.873930+0000","last_peered":"2024-02-10T08:35:23.873930+0000","last_clean":"2024-02-10T08:35:23.873930+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:35:23.873930+0000","last_undegraded":"2024-02-10T08:35:23.873930+0000","last_fullsized":"2024-02-10T08:35:23.873930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:43.680073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:43.680073+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873801+0000","last_change":"2024-02-10T08:34:32.620154+0000","last_active":"2024-02-10T08:35:23.873801+0000","last_peered":"2024-02-10T08:35:23.873801+0000","last_clean":"2024-02-10T08:35:23.873801+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:35:23.873801+0000","last_undegraded":"2024-02-10T08:35:23.873801+0000","last_fullsized":"2024-02-10T08:35:23.873801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:32.620086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.620086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698346+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:35:23.698346+0000","last_peered":"2024-02-10T08:35:23.698346+0000","last_clean":"2024-02-10T08:35:23.698346+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:35:23.698346+0000","last_undegraded":"2024-02-10T08:35:23.698346+0000","last_fullsized":"2024-02-10T08:35:23.698346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874837+0000","last_change":"2024-02-10T08:34:31.359650+0000","last_active":"2024-02-10T08:35:23.874837+0000","last_peered":"2024-02-10T08:35:23.874837+0000","last_clean":"2024-02-10T08:35:23.874837+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:35:23.874837+0000","last_undegraded":"2024-02-10T08:35:23.874837+0000","last_fullsized":"2024-02-10T08:35:23.874837+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":57,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874140+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:35:23.874140+0000","last_peered":"2024-02-10T08:35:23.874140+0000","last_clean":"2024-02-10T08:35:23.874140+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:35:23.874140+0000","last_undegraded":"2024-02-10T08:35:23.874140+0000","last_fullsized":"2024-02-10T08:35:23.874140+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698086+0000","last_change":"2024-02-10T08:34:35.819857+0000","last_active":"2024-02-10T08:35:23.698086+0000","last_peered":"2024-02-10T08:35:23.698086+0000","last_clean":"2024-02-10T08:35:23.698086+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:35:23.698086+0000","last_undegraded":"2024-02-10T08:35:23.698086+0000","last_fullsized":"2024-02-10T08:35:23.698086+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.819777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.819777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873517+0000","last_change":"2024-02-10T08:34:55.286635+0000","last_active":"2024-02-10T08:35:23.873517+0000","last_peered":"2024-02-10T08:35:23.873517+0000","last_clean":"2024-02-10T08:35:23.873517+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:35:23.873517+0000","last_undegraded":"2024-02-10T08:35:23.873517+0000","last_fullsized":"2024-02-10T08:35:23.873517+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.286540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.286540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875714+0000","last_change":"2024-02-10T08:34:28.590439+0000","last_active":"2024-02-10T08:35:23.875714+0000","last_peered":"2024-02-10T08:35:23.875714+0000","last_clean":"2024-02-10T08:35:23.875714+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:35:23.875714+0000","last_undegraded":"2024-02-10T08:35:23.875714+0000","last_fullsized":"2024-02-10T08:35:23.875714+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.590362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.590362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873743+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:35:23.873743+0000","last_peered":"2024-02-10T08:35:23.873743+0000","last_clean":"2024-02-10T08:35:23.873743+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:35:23.873743+0000","last_undegraded":"2024-02-10T08:35:23.873743+0000","last_fullsized":"2024-02-10T08:35:23.873743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873784+0000","last_change":"2024-02-10T08:34:28.605230+0000","last_active":"2024-02-10T08:35:23.873784+0000","last_peered":"2024-02-10T08:35:23.873784+0000","last_clean":"2024-02-10T08:35:23.873784+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:35:23.873784+0000","last_undegraded":"2024-02-10T08:35:23.873784+0000","last_fullsized":"2024-02-10T08:35:23.873784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.605166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.605166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698840+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:35:23.698840+0000","last_peered":"2024-02-10T08:35:23.698840+0000","last_clean":"2024-02-10T08:35:23.698840+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:35:23.698840+0000","last_undegraded":"2024-02-10T08:35:23.698840+0000","last_fullsized":"2024-02-10T08:35:23.698840+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699346+0000","last_change":"2024-02-10T08:34:36.300831+0000","last_active":"2024-02-10T08:35:23.699346+0000","last_peered":"2024-02-10T08:35:23.699346+0000","last_clean":"2024-02-10T08:35:23.699346+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:35:23.699346+0000","last_undegraded":"2024-02-10T08:35:23.699346+0000","last_fullsized":"2024-02-10T08:35:23.699346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:36.300770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.300770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874798+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:35:23.874798+0000","last_peered":"2024-02-10T08:35:23.874798+0000","last_clean":"2024-02-10T08:35:23.874798+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:35:23.874798+0000","last_undegraded":"2024-02-10T08:35:23.874798+0000","last_fullsized":"2024-02-10T08:35:23.874798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872742+0000","last_change":"2024-02-10T08:34:48.691828+0000","last_active":"2024-02-10T08:35:23.872742+0000","last_peered":"2024-02-10T08:35:23.872742+0000","last_clean":"2024-02-10T08:35:23.872742+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:35:23.872742+0000","last_undegraded":"2024-02-10T08:35:23.872742+0000","last_fullsized":"2024-02-10T08:35:23.872742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.691737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.691737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873601+0000","last_change":"2024-02-10T08:34:27.584434+0000","last_active":"2024-02-10T08:35:23.873601+0000","last_peered":"2024-02-10T08:35:23.873601+0000","last_clean":"2024-02-10T08:35:23.873601+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:35:23.873601+0000","last_undegraded":"2024-02-10T08:35:23.873601+0000","last_fullsized":"2024-02-10T08:35:23.873601+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:27.584369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.584369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874118+0000","last_change":"2024-02-10T08:34:23.544050+0000","last_active":"2024-02-10T08:35:23.874118+0000","last_peered":"2024-02-10T08:35:23.874118+0000","last_clean":"2024-02-10T08:35:23.874118+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:35:23.874118+0000","last_undegraded":"2024-02-10T08:35:23.874118+0000","last_fullsized":"2024-02-10T08:35:23.874118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.543984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.543984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":59,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698767+0000","last_change":"2024-02-10T08:34:47.420238+0000","last_active":"2024-02-10T08:35:23.698767+0000","last_peered":"2024-02-10T08:35:23.698767+0000","last_clean":"2024-02-10T08:35:23.698767+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:35:23.698767+0000","last_undegraded":"2024-02-10T08:35:23.698767+0000","last_fullsized":"2024-02-10T08:35:23.698767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.420174+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875563+0000","last_change":"2024-02-10T08:33:24.119570+0000","last_active":"2024-02-10T08:35:23.875563+0000","last_peered":"2024-02-10T08:35:23.875563+0000","last_clean":"2024-02-10T08:35:23.875563+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:35:23.875563+0000","last_undegraded":"2024-02-10T08:35:23.875563+0000","last_fullsized":"2024-02-10T08:35:23.875563+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:33:23.102654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873971+0000","last_change":"2024-02-10T08:34:51.497308+0000","last_active":"2024-02-10T08:35:23.873971+0000","last_peered":"2024-02-10T08:35:23.873971+0000","last_clean":"2024-02-10T08:35:23.873971+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:35:23.873971+0000","last_undegraded":"2024-02-10T08:35:23.873971+0000","last_fullsized":"2024-02-10T08:35:23.873971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.497242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.497242+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873394+0000","last_change":"2024-02-10T08:34:35.635203+0000","last_active":"2024-02-10T08:35:23.873394+0000","last_peered":"2024-02-10T08:35:23.873394+0000","last_clean":"2024-02-10T08:35:23.873394+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:35:23.873394+0000","last_undegraded":"2024-02-10T08:35:23.873394+0000","last_fullsized":"2024-02-10T08:35:23.873394+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.635108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.635108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875035+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:35:23.875035+0000","last_peered":"2024-02-10T08:35:23.875035+0000","last_clean":"2024-02-10T08:35:23.875035+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:35:23.875035+0000","last_undegraded":"2024-02-10T08:35:23.875035+0000","last_fullsized":"2024-02-10T08:35:23.875035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873743+0000","last_change":"2024-02-10T08:35:02.705789+0000","last_active":"2024-02-10T08:35:23.873743+0000","last_peered":"2024-02-10T08:35:23.873743+0000","last_clean":"2024-02-10T08:35:23.873743+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:35:23.873743+0000","last_undegraded":"2024-02-10T08:35:23.873743+0000","last_fullsized":"2024-02-10T08:35:23.873743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.705679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.705679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874877+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:35:23.874877+0000","last_peered":"2024-02-10T08:35:23.874877+0000","last_clean":"2024-02-10T08:35:23.874877+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:35:23.874877+0000","last_undegraded":"2024-02-10T08:35:23.874877+0000","last_fullsized":"2024-02-10T08:35:23.874877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873765+0000","last_change":"2024-02-10T08:34:29.584600+0000","last_active":"2024-02-10T08:35:23.873765+0000","last_peered":"2024-02-10T08:35:23.873765+0000","last_clean":"2024-02-10T08:35:23.873765+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:35:23.873765+0000","last_undegraded":"2024-02-10T08:35:23.873765+0000","last_fullsized":"2024-02-10T08:35:23.873765+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.584529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.584529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873761+0000","last_change":"2024-02-10T08:34:31.663950+0000","last_active":"2024-02-10T08:35:23.873761+0000","last_peered":"2024-02-10T08:35:23.873761+0000","last_clean":"2024-02-10T08:35:23.873761+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:35:23.873761+0000","last_undegraded":"2024-02-10T08:35:23.873761+0000","last_fullsized":"2024-02-10T08:35:23.873761+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:31.663882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.663882+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698704+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:35:23.698704+0000","last_peered":"2024-02-10T08:35:23.698704+0000","last_clean":"2024-02-10T08:35:23.698704+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:35:23.698704+0000","last_undegraded":"2024-02-10T08:35:23.698704+0000","last_fullsized":"2024-02-10T08:35:23.698704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875606+0000","last_change":"2024-02-10T08:34:45.607331+0000","last_active":"2024-02-10T08:35:23.875606+0000","last_peered":"2024-02-10T08:35:23.875606+0000","last_clean":"2024-02-10T08:35:23.875606+0000","last_became_active":"2024-02-10T08:33:22.121591+0000","last_became_peered":"2024-02-10T08:33:22.121591+0000","last_unstale":"2024-02-10T08:35:23.875606+0000","last_undegraded":"2024-02-10T08:35:23.875606+0000","last_fullsized":"2024-02-10T08:35:23.875606+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.607254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.607254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873597+0000","last_change":"2024-02-10T08:34:29.402936+0000","last_active":"2024-02-10T08:35:23.873597+0000","last_peered":"2024-02-10T08:35:23.873597+0000","last_clean":"2024-02-10T08:35:23.873597+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:35:23.873597+0000","last_undegraded":"2024-02-10T08:35:23.873597+0000","last_fullsized":"2024-02-10T08:35:23.873597+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.402874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.402874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874117+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:35:23.874117+0000","last_peered":"2024-02-10T08:35:23.874117+0000","last_clean":"2024-02-10T08:35:23.874117+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:35:23.874117+0000","last_undegraded":"2024-02-10T08:35:23.874117+0000","last_fullsized":"2024-02-10T08:35:23.874117+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699427+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:35:23.699427+0000","last_peered":"2024-02-10T08:35:23.699427+0000","last_clean":"2024-02-10T08:35:23.699427+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:35:23.699427+0000","last_undegraded":"2024-02-10T08:35:23.699427+0000","last_fullsized":"2024-02-10T08:35:23.699427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874222+0000","last_change":"2024-02-10T08:35:02.446706+0000","last_active":"2024-02-10T08:35:23.874222+0000","last_peered":"2024-02-10T08:35:23.874222+0000","last_clean":"2024-02-10T08:35:23.874222+0000","last_became_active":"2024-02-10T08:33:22.118323+0000","last_became_peered":"2024-02-10T08:33:22.118323+0000","last_unstale":"2024-02-10T08:35:23.874222+0000","last_undegraded":"2024-02-10T08:35:23.874222+0000","last_fullsized":"2024-02-10T08:35:23.874222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699445+0000","last_change":"2024-02-10T08:34:56.699511+0000","last_active":"2024-02-10T08:35:23.699445+0000","last_peered":"2024-02-10T08:35:23.699445+0000","last_clean":"2024-02-10T08:35:23.699445+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:35:23.699445+0000","last_undegraded":"2024-02-10T08:35:23.699445+0000","last_fullsized":"2024-02-10T08:35:23.699445+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.699420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.699420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'9","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873837+0000","last_change":"2024-02-10T08:34:23.674473+0000","last_active":"2024-02-10T08:35:23.873837+0000","last_peered":"2024-02-10T08:35:23.873837+0000","last_clean":"2024-02-10T08:35:23.873837+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:35:23.873837+0000","last_undegraded":"2024-02-10T08:35:23.873837+0000","last_fullsized":"2024-02-10T08:35:23.873837+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:23.674404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.674404+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873624+0000","last_change":"2024-02-10T08:34:45.519207+0000","last_active":"2024-02-10T08:35:23.873624+0000","last_peered":"2024-02-10T08:35:23.873624+0000","last_clean":"2024-02-10T08:35:23.873624+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:35:23.873624+0000","last_undegraded":"2024-02-10T08:35:23.873624+0000","last_fullsized":"2024-02-10T08:35:23.873624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.519116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.519116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875360+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:35:23.875360+0000","last_peered":"2024-02-10T08:35:23.875360+0000","last_clean":"2024-02-10T08:35:23.875360+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:35:23.875360+0000","last_undegraded":"2024-02-10T08:35:23.875360+0000","last_fullsized":"2024-02-10T08:35:23.875360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":67,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873873+0000","last_change":"2024-02-10T08:34:54.039656+0000","last_active":"2024-02-10T08:35:23.873873+0000","last_peered":"2024-02-10T08:35:23.873873+0000","last_clean":"2024-02-10T08:35:23.873873+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:35:23.873873+0000","last_undegraded":"2024-02-10T08:35:23.873873+0000","last_fullsized":"2024-02-10T08:35:23.873873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.039584+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699487+0000","last_change":"2024-02-10T08:34:39.590686+0000","last_active":"2024-02-10T08:35:23.699487+0000","last_peered":"2024-02-10T08:35:23.699487+0000","last_clean":"2024-02-10T08:35:23.699487+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:35:23.699487+0000","last_undegraded":"2024-02-10T08:35:23.699487+0000","last_fullsized":"2024-02-10T08:35:23.699487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.590604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.590604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872756+0000","last_change":"2024-02-10T08:34:31.544112+0000","last_active":"2024-02-10T08:35:23.872756+0000","last_peered":"2024-02-10T08:35:23.872756+0000","last_clean":"2024-02-10T08:35:23.872756+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:35:23.872756+0000","last_undegraded":"2024-02-10T08:35:23.872756+0000","last_fullsized":"2024-02-10T08:35:23.872756+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.544049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.544049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698527+0000","last_change":"2024-02-10T08:34:46.452573+0000","last_active":"2024-02-10T08:35:23.698527+0000","last_peered":"2024-02-10T08:35:23.698527+0000","last_clean":"2024-02-10T08:35:23.698527+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:35:23.698527+0000","last_undegraded":"2024-02-10T08:35:23.698527+0000","last_fullsized":"2024-02-10T08:35:23.698527+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:46.452504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:46.452504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'10","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698917+0000","last_change":"2024-02-10T08:34:33.231522+0000","last_active":"2024-02-10T08:35:23.698917+0000","last_peered":"2024-02-10T08:35:23.698917+0000","last_clean":"2024-02-10T08:35:23.698917+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:35:23.698917+0000","last_undegraded":"2024-02-10T08:35:23.698917+0000","last_fullsized":"2024-02-10T08:35:23.698917+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:33.231411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.231411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873483+0000","last_change":"2024-02-10T08:34:47.693139+0000","last_active":"2024-02-10T08:35:23.873483+0000","last_peered":"2024-02-10T08:35:23.873483+0000","last_clean":"2024-02-10T08:35:23.873483+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:35:23.873483+0000","last_undegraded":"2024-02-10T08:35:23.873483+0000","last_fullsized":"2024-02-10T08:35:23.873483+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:47.693030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.693030+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874330+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:35:23.874330+0000","last_peered":"2024-02-10T08:35:23.874330+0000","last_clean":"2024-02-10T08:35:23.874330+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:35:23.874330+0000","last_undegraded":"2024-02-10T08:35:23.874330+0000","last_fullsized":"2024-02-10T08:35:23.874330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873805+0000","last_change":"2024-02-10T08:34:20.652012+0000","last_active":"2024-02-10T08:35:23.873805+0000","last_peered":"2024-02-10T08:35:23.873805+0000","last_clean":"2024-02-10T08:35:23.873805+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:35:23.873805+0000","last_undegraded":"2024-02-10T08:35:23.873805+0000","last_fullsized":"2024-02-10T08:35:23.873805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.651946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.651946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874561+0000","last_change":"2024-02-10T08:34:47.549753+0000","last_active":"2024-02-10T08:35:23.874561+0000","last_peered":"2024-02-10T08:35:23.874561+0000","last_clean":"2024-02-10T08:35:23.874561+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:35:23.874561+0000","last_undegraded":"2024-02-10T08:35:23.874561+0000","last_fullsized":"2024-02-10T08:35:23.874561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:34:47.549671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.549671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873270+0000","last_change":"2024-02-10T08:35:00.639772+0000","last_active":"2024-02-10T08:35:23.873270+0000","last_peered":"2024-02-10T08:35:23.873270+0000","last_clean":"2024-02-10T08:35:23.873270+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:35:23.873270+0000","last_undegraded":"2024-02-10T08:35:23.873270+0000","last_fullsized":"2024-02-10T08:35:23.873270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.639705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.639705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699014+0000","last_change":"2024-02-10T08:34:51.910276+0000","last_active":"2024-02-10T08:35:23.699014+0000","last_peered":"2024-02-10T08:35:23.699014+0000","last_clean":"2024-02-10T08:35:23.699014+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:35:23.699014+0000","last_undegraded":"2024-02-10T08:35:23.699014+0000","last_fullsized":"2024-02-10T08:35:23.699014+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.910184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":166,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708990+0000","last_change":"2024-02-10T08:35:07.227647+0000","last_active":"2024-02-10T08:35:24.708990+0000","last_peered":"2024-02-10T08:35:24.708990+0000","last_clean":"2024-02-10T08:35:24.708990+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:35:24.708990+0000","last_undegraded":"2024-02-10T08:35:24.708990+0000","last_fullsized":"2024-02-10T08:35:24.708990+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.227587+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699035+0000","last_change":"2024-02-10T08:34:52.698430+0000","last_active":"2024-02-10T08:35:23.699035+0000","last_peered":"2024-02-10T08:35:23.699035+0000","last_clean":"2024-02-10T08:35:23.699035+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:35:23.699035+0000","last_undegraded":"2024-02-10T08:35:23.699035+0000","last_fullsized":"2024-02-10T08:35:23.699035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.698329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698809+0000","last_change":"2024-02-10T08:34:27.278457+0000","last_active":"2024-02-10T08:35:23.698809+0000","last_peered":"2024-02-10T08:35:23.698809+0000","last_clean":"2024-02-10T08:35:23.698809+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:35:23.698809+0000","last_undegraded":"2024-02-10T08:35:23.698809+0000","last_fullsized":"2024-02-10T08:35:23.698809+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:34:27.278388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.278388+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699913+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:35:23.699913+0000","last_peered":"2024-02-10T08:35:23.699913+0000","last_clean":"2024-02-10T08:35:23.699913+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:35:23.699913+0000","last_undegraded":"2024-02-10T08:35:23.699913+0000","last_fullsized":"2024-02-10T08:35:23.699913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875650+0000","last_change":"2024-02-10T08:34:55.443465+0000","last_active":"2024-02-10T08:35:23.875650+0000","last_peered":"2024-02-10T08:35:23.875650+0000","last_clean":"2024-02-10T08:35:23.875650+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:35:23.875650+0000","last_undegraded":"2024-02-10T08:35:23.875650+0000","last_fullsized":"2024-02-10T08:35:23.875650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.443390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.443390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873321+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:35:23.873321+0000","last_peered":"2024-02-10T08:35:23.873321+0000","last_clean":"2024-02-10T08:35:23.873321+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:35:23.873321+0000","last_undegraded":"2024-02-10T08:35:23.873321+0000","last_fullsized":"2024-02-10T08:35:23.873321+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874312+0000","last_change":"2024-02-10T08:34:52.491498+0000","last_active":"2024-02-10T08:35:23.874312+0000","last_peered":"2024-02-10T08:35:23.874312+0000","last_clean":"2024-02-10T08:35:23.874312+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:35:23.874312+0000","last_undegraded":"2024-02-10T08:35:23.874312+0000","last_fullsized":"2024-02-10T08:35:23.874312+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.491420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.491420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698072+0000","last_change":"2024-02-10T08:34:48.841104+0000","last_active":"2024-02-10T08:35:23.698072+0000","last_peered":"2024-02-10T08:35:23.698072+0000","last_clean":"2024-02-10T08:35:23.698072+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:35:23.698072+0000","last_undegraded":"2024-02-10T08:35:23.698072+0000","last_fullsized":"2024-02-10T08:35:23.698072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.841025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.841025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875145+0000","last_change":"2024-02-10T08:34:39.378393+0000","last_active":"2024-02-10T08:35:23.875145+0000","last_peered":"2024-02-10T08:35:23.875145+0000","last_clean":"2024-02-10T08:35:23.875145+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:35:23.875145+0000","last_undegraded":"2024-02-10T08:35:23.875145+0000","last_fullsized":"2024-02-10T08:35:23.875145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:39.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.378312+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698718+0000","last_change":"2024-02-10T08:34:40.809042+0000","last_active":"2024-02-10T08:35:23.698718+0000","last_peered":"2024-02-10T08:35:23.698718+0000","last_clean":"2024-02-10T08:35:23.698718+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:35:23.698718+0000","last_undegraded":"2024-02-10T08:35:23.698718+0000","last_fullsized":"2024-02-10T08:35:23.698718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:40.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:40.808978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698234+0000","last_change":"2024-02-10T08:34:28.307074+0000","last_active":"2024-02-10T08:35:23.698234+0000","last_peered":"2024-02-10T08:35:23.698234+0000","last_clean":"2024-02-10T08:35:23.698234+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:35:23.698234+0000","last_undegraded":"2024-02-10T08:35:23.698234+0000","last_fullsized":"2024-02-10T08:35:23.698234+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:34:28.306997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.306997+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873041+0000","last_change":"2024-02-10T08:34:21.731002+0000","last_active":"2024-02-10T08:35:23.873041+0000","last_peered":"2024-02-10T08:35:23.873041+0000","last_clean":"2024-02-10T08:35:23.873041+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:35:23.873041+0000","last_undegraded":"2024-02-10T08:35:23.873041+0000","last_fullsized":"2024-02-10T08:35:23.873041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.730937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.730937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698056+0000","last_change":"2024-02-10T08:34:25.707975+0000","last_active":"2024-02-10T08:35:23.698056+0000","last_peered":"2024-02-10T08:35:23.698056+0000","last_clean":"2024-02-10T08:35:23.698056+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:35:23.698056+0000","last_undegraded":"2024-02-10T08:35:23.698056+0000","last_fullsized":"2024-02-10T08:35:23.698056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:25.707852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.707852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873755+0000","last_change":"2024-02-10T08:34:28.599042+0000","last_active":"2024-02-10T08:35:23.873755+0000","last_peered":"2024-02-10T08:35:23.873755+0000","last_clean":"2024-02-10T08:35:23.873755+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:35:23.873755+0000","last_undegraded":"2024-02-10T08:35:23.873755+0000","last_fullsized":"2024-02-10T08:35:23.873755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:28.598979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.598979+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875613+0000","last_change":"2024-02-10T08:34:38.628666+0000","last_active":"2024-02-10T08:35:23.875613+0000","last_peered":"2024-02-10T08:35:23.875613+0000","last_clean":"2024-02-10T08:35:23.875613+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:35:23.875613+0000","last_undegraded":"2024-02-10T08:35:23.875613+0000","last_fullsized":"2024-02-10T08:35:23.875613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.628559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.628559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698080+0000","last_change":"2024-02-10T08:34:44.434673+0000","last_active":"2024-02-10T08:35:23.698080+0000","last_peered":"2024-02-10T08:35:23.698080+0000","last_clean":"2024-02-10T08:35:23.698080+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:35:23.698080+0000","last_undegraded":"2024-02-10T08:35:23.698080+0000","last_fullsized":"2024-02-10T08:35:23.698080+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.434603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.434603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699102+0000","last_change":"2024-02-10T08:34:44.810962+0000","last_active":"2024-02-10T08:35:23.699102+0000","last_peered":"2024-02-10T08:35:23.699102+0000","last_clean":"2024-02-10T08:35:23.699102+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:35:23.699102+0000","last_undegraded":"2024-02-10T08:35:23.699102+0000","last_fullsized":"2024-02-10T08:35:23.699102+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.810886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.810886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873372+0000","last_change":"2024-02-10T08:34:39.703715+0000","last_active":"2024-02-10T08:35:23.873372+0000","last_peered":"2024-02-10T08:35:23.873372+0000","last_clean":"2024-02-10T08:35:23.873372+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:35:23.873372+0000","last_undegraded":"2024-02-10T08:35:23.873372+0000","last_fullsized":"2024-02-10T08:35:23.873372+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:39.703643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.703643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873988+0000","last_change":"2024-02-10T08:34:51.348414+0000","last_active":"2024-02-10T08:35:23.873988+0000","last_peered":"2024-02-10T08:35:23.873988+0000","last_clean":"2024-02-10T08:35:23.873988+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:35:23.873988+0000","last_undegraded":"2024-02-10T08:35:23.873988+0000","last_fullsized":"2024-02-10T08:35:23.873988+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.348334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.348334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874291+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:35:23.874291+0000","last_peered":"2024-02-10T08:35:23.874291+0000","last_clean":"2024-02-10T08:35:23.874291+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:35:23.874291+0000","last_undegraded":"2024-02-10T08:35:23.874291+0000","last_fullsized":"2024-02-10T08:35:23.874291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698445+0000","last_change":"2024-02-10T08:35:03.147093+0000","last_active":"2024-02-10T08:35:23.698445+0000","last_peered":"2024-02-10T08:35:23.698445+0000","last_clean":"2024-02-10T08:35:23.698445+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:35:23.698445+0000","last_undegraded":"2024-02-10T08:35:23.698445+0000","last_fullsized":"2024-02-10T08:35:23.698445+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:35:03.146990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.146990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872862+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:35:23.872862+0000","last_peered":"2024-02-10T08:35:23.872862+0000","last_clean":"2024-02-10T08:35:23.872862+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:35:23.872862+0000","last_undegraded":"2024-02-10T08:35:23.872862+0000","last_fullsized":"2024-02-10T08:35:23.872862+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873962+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:35:23.873962+0000","last_peered":"2024-02-10T08:35:23.873962+0000","last_clean":"2024-02-10T08:35:23.873962+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:35:23.873962+0000","last_undegraded":"2024-02-10T08:35:23.873962+0000","last_fullsized":"2024-02-10T08:35:23.873962+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698690+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:35:23.698690+0000","last_peered":"2024-02-10T08:35:23.698690+0000","last_clean":"2024-02-10T08:35:23.698690+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:35:23.698690+0000","last_undegraded":"2024-02-10T08:35:23.698690+0000","last_fullsized":"2024-02-10T08:35:23.698690+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874762+0000","last_change":"2024-02-10T08:34:22.584930+0000","last_active":"2024-02-10T08:35:23.874762+0000","last_peered":"2024-02-10T08:35:23.874762+0000","last_clean":"2024-02-10T08:35:23.874762+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:35:23.874762+0000","last_undegraded":"2024-02-10T08:35:23.874762+0000","last_fullsized":"2024-02-10T08:35:23.874762+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:22.584856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:22.584856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":60,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873774+0000","last_change":"2024-02-10T08:34:36.155885+0000","last_active":"2024-02-10T08:35:23.873774+0000","last_peered":"2024-02-10T08:35:23.873774+0000","last_clean":"2024-02-10T08:35:23.873774+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:35:23.873774+0000","last_undegraded":"2024-02-10T08:35:23.873774+0000","last_fullsized":"2024-02-10T08:35:23.873774+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:36.155806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.155806+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":72,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872820+0000","last_change":"2024-02-10T08:34:36.649251+0000","last_active":"2024-02-10T08:35:23.872820+0000","last_peered":"2024-02-10T08:35:23.872820+0000","last_clean":"2024-02-10T08:35:23.872820+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:35:23.872820+0000","last_undegraded":"2024-02-10T08:35:23.872820+0000","last_fullsized":"2024-02-10T08:35:23.872820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:36.649186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.649186+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873181+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:35:23.873181+0000","last_peered":"2024-02-10T08:35:23.873181+0000","last_clean":"2024-02-10T08:35:23.873181+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:35:23.873181+0000","last_undegraded":"2024-02-10T08:35:23.873181+0000","last_fullsized":"2024-02-10T08:35:23.873181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875518+0000","last_change":"2024-02-10T08:35:06.520594+0000","last_active":"2024-02-10T08:35:23.875518+0000","last_peered":"2024-02-10T08:35:23.875518+0000","last_clean":"2024-02-10T08:35:23.875518+0000","last_became_active":"2024-02-10T08:33:22.122087+0000","last_became_peered":"2024-02-10T08:33:22.122087+0000","last_unstale":"2024-02-10T08:35:23.875518+0000","last_undegraded":"2024-02-10T08:35:23.875518+0000","last_fullsized":"2024-02-10T08:35:23.875518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698196+0000","last_change":"2024-02-10T08:34:29.780708+0000","last_active":"2024-02-10T08:35:23.698196+0000","last_peered":"2024-02-10T08:35:23.698196+0000","last_clean":"2024-02-10T08:35:23.698196+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:35:23.698196+0000","last_undegraded":"2024-02-10T08:35:23.698196+0000","last_fullsized":"2024-02-10T08:35:23.698196+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.780644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.780644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875361+0000","last_change":"2024-02-10T08:34:25.085734+0000","last_active":"2024-02-10T08:35:23.875361+0000","last_peered":"2024-02-10T08:35:23.875361+0000","last_clean":"2024-02-10T08:35:23.875361+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:35:23.875361+0000","last_undegraded":"2024-02-10T08:35:23.875361+0000","last_fullsized":"2024-02-10T08:35:23.875361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:25.085663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.085663+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874706+0000","last_change":"2024-02-10T08:35:16.510089+0000","last_active":"2024-02-10T08:35:23.874706+0000","last_peered":"2024-02-10T08:35:23.874706+0000","last_clean":"2024-02-10T08:35:23.874706+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:35:23.874706+0000","last_undegraded":"2024-02-10T08:35:23.874706+0000","last_fullsized":"2024-02-10T08:35:23.874706+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875190+0000","last_change":"2024-02-10T08:35:11.026984+0000","last_active":"2024-02-10T08:35:23.875190+0000","last_peered":"2024-02-10T08:35:23.875190+0000","last_clean":"2024-02-10T08:35:23.875190+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:35:23.875190+0000","last_undegraded":"2024-02-10T08:35:23.875190+0000","last_fullsized":"2024-02-10T08:35:23.875190+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698241+0000","last_change":"2024-02-10T08:34:32.208396+0000","last_active":"2024-02-10T08:35:23.698241+0000","last_peered":"2024-02-10T08:35:23.698241+0000","last_clean":"2024-02-10T08:35:23.698241+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:35:23.698241+0000","last_undegraded":"2024-02-10T08:35:23.698241+0000","last_fullsized":"2024-02-10T08:35:23.698241+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:34:32.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.208313+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698719+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:35:23.698719+0000","last_peered":"2024-02-10T08:35:23.698719+0000","last_clean":"2024-02-10T08:35:23.698719+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:35:23.698719+0000","last_undegraded":"2024-02-10T08:35:23.698719+0000","last_fullsized":"2024-02-10T08:35:23.698719+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875064+0000","last_change":"2024-02-10T08:34:35.201454+0000","last_active":"2024-02-10T08:35:23.875064+0000","last_peered":"2024-02-10T08:35:23.875064+0000","last_clean":"2024-02-10T08:35:23.875064+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:35:23.875064+0000","last_undegraded":"2024-02-10T08:35:23.875064+0000","last_fullsized":"2024-02-10T08:35:23.875064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.201371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.201371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699163+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:35:23.699163+0000","last_peered":"2024-02-10T08:35:23.699163+0000","last_clean":"2024-02-10T08:35:23.699163+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:35:23.699163+0000","last_undegraded":"2024-02-10T08:35:23.699163+0000","last_fullsized":"2024-02-10T08:35:23.699163+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698855+0000","last_change":"2024-02-10T08:34:24.362389+0000","last_active":"2024-02-10T08:35:23.698855+0000","last_peered":"2024-02-10T08:35:23.698855+0000","last_clean":"2024-02-10T08:35:23.698855+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:35:23.698855+0000","last_undegraded":"2024-02-10T08:35:23.698855+0000","last_fullsized":"2024-02-10T08:35:23.698855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.362309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.362309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874452+0000","last_change":"2024-02-10T08:34:21.123617+0000","last_active":"2024-02-10T08:35:23.874452+0000","last_peered":"2024-02-10T08:35:23.874452+0000","last_clean":"2024-02-10T08:35:23.874452+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:35:23.874452+0000","last_undegraded":"2024-02-10T08:35:23.874452+0000","last_fullsized":"2024-02-10T08:35:23.874452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:21.123548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.123548+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699337+0000","last_change":"2024-02-10T08:34:53.743882+0000","last_active":"2024-02-10T08:35:23.699337+0000","last_peered":"2024-02-10T08:35:23.699337+0000","last_clean":"2024-02-10T08:35:23.699337+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:35:23.699337+0000","last_undegraded":"2024-02-10T08:35:23.699337+0000","last_fullsized":"2024-02-10T08:35:23.699337+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.743807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.743807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698321+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:35:23.698321+0000","last_peered":"2024-02-10T08:35:23.698321+0000","last_clean":"2024-02-10T08:35:23.698321+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:35:23.698321+0000","last_undegraded":"2024-02-10T08:35:23.698321+0000","last_fullsized":"2024-02-10T08:35:23.698321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698302+0000","last_change":"2024-02-10T08:34:20.776146+0000","last_active":"2024-02-10T08:35:23.698302+0000","last_peered":"2024-02-10T08:35:23.698302+0000","last_clean":"2024-02-10T08:35:23.698302+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:35:23.698302+0000","last_undegraded":"2024-02-10T08:35:23.698302+0000","last_fullsized":"2024-02-10T08:35:23.698302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.776052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.776052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":67,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873933+0000","last_change":"2024-02-10T08:34:24.118004+0000","last_active":"2024-02-10T08:35:23.873933+0000","last_peered":"2024-02-10T08:35:23.873933+0000","last_clean":"2024-02-10T08:35:23.873933+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:35:23.873933+0000","last_undegraded":"2024-02-10T08:35:23.873933+0000","last_fullsized":"2024-02-10T08:35:23.873933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:34:24.117909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.117909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.697954+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:35:23.697954+0000","last_peered":"2024-02-10T08:35:23.697954+0000","last_clean":"2024-02-10T08:35:23.697954+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:35:23.697954+0000","last_undegraded":"2024-02-10T08:35:23.697954+0000","last_fullsized":"2024-02-10T08:35:23.697954+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699622+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:35:23.699622+0000","last_peered":"2024-02-10T08:35:23.699622+0000","last_clean":"2024-02-10T08:35:23.699622+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:35:23.699622+0000","last_undegraded":"2024-02-10T08:35:23.699622+0000","last_fullsized":"2024-02-10T08:35:23.699622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699621+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:35:23.699621+0000","last_peered":"2024-02-10T08:35:23.699621+0000","last_clean":"2024-02-10T08:35:23.699621+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:35:23.699621+0000","last_undegraded":"2024-02-10T08:35:23.699621+0000","last_fullsized":"2024-02-10T08:35:23.699621+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873419+0000","last_change":"2024-02-10T08:34:44.381291+0000","last_active":"2024-02-10T08:35:23.873419+0000","last_peered":"2024-02-10T08:35:23.873419+0000","last_clean":"2024-02-10T08:35:23.873419+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:35:23.873419+0000","last_undegraded":"2024-02-10T08:35:23.873419+0000","last_fullsized":"2024-02-10T08:35:23.873419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:44.381208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.381208+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872814+0000","last_change":"2024-02-10T08:34:29.554010+0000","last_active":"2024-02-10T08:35:23.872814+0000","last_peered":"2024-02-10T08:35:23.872814+0000","last_clean":"2024-02-10T08:35:23.872814+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:35:23.872814+0000","last_undegraded":"2024-02-10T08:35:23.872814+0000","last_fullsized":"2024-02-10T08:35:23.872814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.553914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.553914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699234+0000","last_change":"2024-02-10T08:34:50.747486+0000","last_active":"2024-02-10T08:35:23.699234+0000","last_peered":"2024-02-10T08:35:23.699234+0000","last_clean":"2024-02-10T08:35:23.699234+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:35:23.699234+0000","last_undegraded":"2024-02-10T08:35:23.699234+0000","last_fullsized":"2024-02-10T08:35:23.699234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:50.747424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.747424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699462+0000","last_change":"2024-02-10T08:34:20.413361+0000","last_active":"2024-02-10T08:35:23.699462+0000","last_peered":"2024-02-10T08:35:23.699462+0000","last_clean":"2024-02-10T08:35:23.699462+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:35:23.699462+0000","last_undegraded":"2024-02-10T08:35:23.699462+0000","last_fullsized":"2024-02-10T08:35:23.699462+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.413281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.413281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873463+0000","last_change":"2024-02-10T08:34:41.701563+0000","last_active":"2024-02-10T08:35:23.873463+0000","last_peered":"2024-02-10T08:35:23.873463+0000","last_clean":"2024-02-10T08:35:23.873463+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:35:23.873463+0000","last_undegraded":"2024-02-10T08:35:23.873463+0000","last_fullsized":"2024-02-10T08:35:23.873463+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:34.592074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:34.592074+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873721+0000","last_change":"2024-02-10T08:34:56.475601+0000","last_active":"2024-02-10T08:35:23.873721+0000","last_peered":"2024-02-10T08:35:23.873721+0000","last_clean":"2024-02-10T08:35:23.873721+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:35:23.873721+0000","last_undegraded":"2024-02-10T08:35:23.873721+0000","last_fullsized":"2024-02-10T08:35:23.873721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.475529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.475529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699986+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:35:23.699986+0000","last_peered":"2024-02-10T08:35:23.699986+0000","last_clean":"2024-02-10T08:35:23.699986+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:35:23.699986+0000","last_undegraded":"2024-02-10T08:35:23.699986+0000","last_fullsized":"2024-02-10T08:35:23.699986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875229+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:35:23.875229+0000","last_peered":"2024-02-10T08:35:23.875229+0000","last_clean":"2024-02-10T08:35:23.875229+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:35:23.875229+0000","last_undegraded":"2024-02-10T08:35:23.875229+0000","last_fullsized":"2024-02-10T08:35:23.875229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698271+0000","last_change":"2024-02-10T08:34:42.752203+0000","last_active":"2024-02-10T08:35:23.698271+0000","last_peered":"2024-02-10T08:35:23.698271+0000","last_clean":"2024-02-10T08:35:23.698271+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:35:23.698271+0000","last_undegraded":"2024-02-10T08:35:23.698271+0000","last_fullsized":"2024-02-10T08:35:23.698271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:42.752097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:42.752097+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299420,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":288311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.034,"15min":1.034},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.054,"15min":1.054},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.496,"15min":0.499},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.73599999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.747,"15min":0.747},"last":0.749}]}]},{"osd":6,"up_from":44,"seq":188978561058,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297692,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447460,"statfs":{"total":95995035648,"available":95690199040,"internally_reserved":0,"allocated":610304,"data_stored":288219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.498,"15min":0.498},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46600000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.623,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.501},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.502}]},{"osd":3,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.375,"15min":0.372},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56000000000000005,"5min":0.623,"15min":0.623},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.434},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.39600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":289631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.495,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.47499999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36799999999999999,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.44800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.309,"5min":0.36199999999999999,"15min":0.36499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":0.624,"15min":0.624},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41899999999999998,"15min":0.42899999999999999},"min":{"1min":0.27100000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.53000000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":1.294,"15min":1.294},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.61899999999999999},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81499999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73299999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.72399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297704,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447448,"statfs":{"total":95995035648,"available":95690186752,"internally_reserved":0,"allocated":622592,"data_stored":287335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.30299999999999999},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.313},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.308},"min":{"1min":0.27300000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48199999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46400000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.23000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.503,"5min":0.624,"15min":0.624},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.501,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.502},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75900000000000001,"5min":0.878,"15min":0.878},"last":0.46899999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76900000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66800000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:35:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84299999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79500000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.58399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":288014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.193,"5min":0.187,"15min":0.187},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.312,"5min":0.35399999999999998,"15min":0.36099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48099999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.313}]},{"osd":1,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35699999999999998,"15min":0.36099999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.41699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.38200000000000001,"15min":0.38300000000000001},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.45700000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.372}]},{"osd":3,"last update":"Sat Feb 10 08:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81899999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.621,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.60799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":288507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35599999999999998,"15min":0.36899999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.23699999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39600000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.437,"15min":0.46000000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.22},{"interface":"front","average":{"1min":0.376,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.246}]},{"osd":4,"last update":"Sat Feb 10 08:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57999999999999996,"15min":0.60799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66700000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55800000000000005,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93400000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.54200000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986219,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":288677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.998,"5min":1,"15min":1},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.62}]},{"osd":2,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.749,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.379,"15min":0.371},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.375}]},{"osd":6,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.441,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53900000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297708,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":626688,"data_stored":287653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.51600000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68000000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68600000000000005,"5min":0.749,"15min":0.749},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.32700000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66100000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70499999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.42699999999999999,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.76400000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":1.081,"15min":1.081},"last":0.72299999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48499999999999999,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.78000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:35:31.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:29.831199+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:29.834109+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.120486+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.b scrub starts 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.123823+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.b scrub ok 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.238745+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-10T08:35:31.044 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.241658+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-10T08:35:31.313 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:31.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.016417+0000 mgr.y (mgr.14553) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:30.304813+0000 mgr.y (mgr.14553) 92 : audit [DBG] from='client.14766 v1:172.21.15.107:0/47902946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.390651+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.390745+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.727560+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.798577+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d scrub starts 2024-02-10T08:35:31.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:30.802302+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d scrub ok 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:31.219455+0000 osd.1 (osd.1) 49 : cluster [DBG] 2.a scrub starts 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:31.222328+0000 osd.1 (osd.1) 50 : cluster [DBG] 2.a scrub ok 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.016417+0000 mgr.y (mgr.14553) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:30.304813+0000 mgr.y (mgr.14553) 92 : audit [DBG] from='client.14766 v1:172.21.15.107:0/47902946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.390651+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.390745+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.727560+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.798577+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d scrub starts 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:30.802302+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d scrub ok 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:31.219455+0000 osd.1 (osd.1) 49 : cluster [DBG] 2.a scrub starts 2024-02-10T08:35:31.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:31.222328+0000 osd.1 (osd.1) 50 : cluster [DBG] 2.a scrub ok 2024-02-10T08:35:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.016417+0000 mgr.y (mgr.14553) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:30.304813+0000 mgr.y (mgr.14553) 92 : audit [DBG] from='client.14766 v1:172.21.15.107:0/47902946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.390651+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.390745+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.727560+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-10T08:35:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.798577+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d scrub starts 2024-02-10T08:35:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:30.802302+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d scrub ok 2024-02-10T08:35:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:31.219455+0000 osd.1 (osd.1) 49 : cluster [DBG] 2.a scrub starts 2024-02-10T08:35:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:31.222328+0000 osd.1 (osd.1) 50 : cluster [DBG] 2.a scrub ok 2024-02-10T08:35:32.894 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:31.087309+0000 mgr.y (mgr.14553) 93 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:32.894 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:31.087309+0000 mgr.y (mgr.14553) 93 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:31.087309+0000 mgr.y (mgr.14553) 93 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- 172.21.15.107:0/592661584 >> v1:172.21.15.107:6790/0 conn(0x7f70bc103250 legacy=0x7f70bc1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- 172.21.15.107:0/592661584 shutdown_connections 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- 172.21.15.107:0/592661584 >> 172.21.15.107:0/592661584 conn(0x7f70bc0f2510 msgr2=0x7f70bc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- 172.21.15.107:0/592661584 shutdown_connections 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- 172.21.15.107:0/592661584 wait complete. 2024-02-10T08:35:33.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 librados: starting msgr at 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 librados: starting objecter 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 Processor -- start 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- start start 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 librados: setting wanted keys 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 librados: calling monclient init 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70bc0fbf40 con 0x7f70bc0fc910 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70bc0fc130 con 0x7f70bc0ff710 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.088+0000 7f70c34e4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70bc0fc320 con 0x7f70bc103250 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c1a81700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f70bc103250 0x7f70bc1183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39374/0 (socket says 172.21.15.107:39374) 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c1a81700 1 -- 172.21.15.107:0/709723698 learned_addr learned my addr 172.21.15.107:0/709723698 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter ms_handle_connect 0x7f70bc103250 2024-02-10T08:35:33.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter ms_handle_connect 0x7f70bc0ff710 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317100764 0 0) 0x7f70bc0fc320 con 0x7f70bc103250 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a80033f0 con 0x7f70bc103250 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter ms_handle_connect 0x7f70bc0fc910 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242894651 0 0) 0x7f70bc0fc130 con 0x7f70bc0ff710 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70bc0fc320 con 0x7f70bc0ff710 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29186923 0 0) 0x7f70bc0fbf40 con 0x7f70bc0fc910 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70bc0fc130 con 0x7f70bc0fc910 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 466052766 0 0) 0x7f70a80033f0 con 0x7f70bc103250 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70bc0fbf40 con 0x7f70bc103250 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70b4002fe0 con 0x7f70bc103250 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2565322154 0 0) 0x7f70bc0fc320 con 0x7f70bc0ff710 2024-02-10T08:35:33.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a80033f0 con 0x7f70bc0ff710 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70ac0019d0 con 0x7f70bc0ff710 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 248046666 0 0) 0x7f70bc0fc130 con 0x7f70bc0fc910 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70bc0fc320 con 0x7f70bc0fc910 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70b80028f0 con 0x7f70bc0fc910 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2433446731 0 0) 0x7f70bc0fbf40 con 0x7f70bc103250 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 >> v1:172.21.15.195:6789/0 conn(0x7f70bc0ff710 legacy=0x7f70bc114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 >> v1:172.21.15.107:6789/0 conn(0x7f70bc0fc910 legacy=0x7f70bc0fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc11cb70 con 0x7f70bc103250 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc119b60 con 0x7f70bc103250 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c34e4700 10 client.34691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc119fd0 con 0x7f70bc103250 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70c34e4700 1 librados: init done 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f70b4001ee0 con 0x7f70bc103250 2024-02-10T08:35:33.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.089+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70b4003b40 con 0x7f70bc103250 2024-02-10T08:35:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.090+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0004f20 con 0x7f70bc103250 2024-02-10T08:35:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f70b4003d40 con 0x7f70bc103250 2024-02-10T08:35:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(84..84 src has 30..84) v4 ==== 6172+0+0 (unknown 2330842233 0 0) 0x7f70b4078bd0 con 0x7f70bc103250 2024-02-10T08:35:33.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 10 client.34691.objecter ms_dispatch 0x7f70bc001550 osd_map(84..84 src has 30..84) v4 2024-02-10T08:35:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 3 client.34691.objecter handle_osd_map got epochs [84,84] > 0 2024-02-10T08:35:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 3 client.34691.objecter handle_osd_map decoding full epoch 84 2024-02-10T08:35:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 20 client.34691.objecter dump_active .. 0 homeless 2024-02-10T08:35:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.091+0000 7f70b27fc700 10 client.34691.objecter ms_handle_connect 0x7f70a8063d00 2024-02-10T08:35:33.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.095+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f70b404c7e0 con 0x7f70bc103250 2024-02-10T08:35:33.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.252+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70a0000b90 con 0x7f70a8063d00 2024-02-10T08:35:33.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.262+0000 7f70b27fc700 1 -- 172.21.15.107:0/709723698 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357360 (unknown 2965378022 0 764905108) 0x7f70a0000b90 con 0x7f70a8063d00 2024-02-10T08:35:33.267 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 10 librados: watch_flush enter 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 10 librados: watch_flush exit 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 20 client.34691.objecter shutdown clearing up homeless session... 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 10 client.34691.objecter successfully canceled tick 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 >> v1:172.21.15.107:6800/2634842009 conn(0x7f70a8063d00 legacy=0x7f70a80661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 >> v1:172.21.15.107:6790/0 conn(0x7f70bc103250 legacy=0x7f70bc1183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 shutdown_connections 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 >> 172.21.15.107:0/709723698 conn(0x7f70bc0f2510 msgr2=0x7f70bc1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 shutdown_connections 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 -- 172.21.15.107:0/709723698 wait complete. 2024-02-10T08:35:33.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:33.265+0000 7f70c34e4700 1 librados: shutdown 2024-02-10T08:35:33.271 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:35:33.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:32.017185+0000 mgr.y (mgr.14553) 94 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:32.017185+0000 mgr.y (mgr.14553) 94 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:32.017185+0000 mgr.y (mgr.14553) 94 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:34.053 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-02-10T08:35:32.016644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":846,"num_read_kb":598,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":493,"ondisk_log_size":493,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381720,"kb_used_data":4952,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579496,"statfs":{"total":767960285184,"available":765521403904,"internally_reserved":0,"allocated":5070848,"data_stored":2319175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002041"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873473+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:35:23.873473+0000","last_peered":"2024-02-10T08:35:23.873473+0000","last_clean":"2024-02-10T08:35:23.873473+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:35:23.873473+0000","last_undegraded":"2024-02-10T08:35:23.873473+0000","last_fullsized":"2024-02-10T08:35:23.873473+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.706907+0000","last_change":"2024-02-10T08:35:23.876952+0000","last_active":"2024-02-10T08:35:24.706907+0000","last_peered":"2024-02-10T08:35:24.706907+0000","last_clean":"2024-02-10T08:35:24.706907+0000","last_became_active":"2024-02-10T08:33:22.117534+0000","last_became_peered":"2024-02-10T08:33:22.117534+0000","last_unstale":"2024-02-10T08:35:24.706907+0000","last_undegraded":"2024-02-10T08:35:24.706907+0000","last_fullsized":"2024-02-10T08:35:24.706907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:23.876871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:23.876871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873247+0000","last_change":"2024-02-10T08:34:41.648797+0000","last_active":"2024-02-10T08:35:23.873247+0000","last_peered":"2024-02-10T08:35:23.873247+0000","last_clean":"2024-02-10T08:35:23.873247+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:35:23.873247+0000","last_undegraded":"2024-02-10T08:35:23.873247+0000","last_fullsized":"2024-02-10T08:35:23.873247+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:41.648715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:41.648715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":63,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874167+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:35:23.874167+0000","last_peered":"2024-02-10T08:35:23.874167+0000","last_clean":"2024-02-10T08:35:23.874167+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:35:23.874167+0000","last_undegraded":"2024-02-10T08:35:23.874167+0000","last_fullsized":"2024-02-10T08:35:23.874167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707096+0000","last_change":"2024-02-10T08:34:38.118431+0000","last_active":"2024-02-10T08:35:24.707096+0000","last_peered":"2024-02-10T08:35:24.707096+0000","last_clean":"2024-02-10T08:35:24.707096+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:35:24.707096+0000","last_undegraded":"2024-02-10T08:35:24.707096+0000","last_fullsized":"2024-02-10T08:35:24.707096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.118347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.118347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707798+0000","last_change":"2024-02-10T08:34:37.332223+0000","last_active":"2024-02-10T08:35:24.707798+0000","last_peered":"2024-02-10T08:35:24.707798+0000","last_clean":"2024-02-10T08:35:24.707798+0000","last_became_active":"2024-02-10T08:33:22.121607+0000","last_became_peered":"2024-02-10T08:33:22.121607+0000","last_unstale":"2024-02-10T08:35:24.707798+0000","last_undegraded":"2024-02-10T08:35:24.707798+0000","last_fullsized":"2024-02-10T08:35:24.707798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.332140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.332140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874290+0000","last_change":"2024-02-10T08:34:23.594197+0000","last_active":"2024-02-10T08:35:23.874290+0000","last_peered":"2024-02-10T08:35:23.874290+0000","last_clean":"2024-02-10T08:35:23.874290+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:35:23.874290+0000","last_undegraded":"2024-02-10T08:35:23.874290+0000","last_fullsized":"2024-02-10T08:35:23.874290+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.594135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.594135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707090+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:35:24.707090+0000","last_peered":"2024-02-10T08:35:24.707090+0000","last_clean":"2024-02-10T08:35:24.707090+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:35:24.707090+0000","last_undegraded":"2024-02-10T08:35:24.707090+0000","last_fullsized":"2024-02-10T08:35:24.707090+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873545+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:35:23.873545+0000","last_peered":"2024-02-10T08:35:23.873545+0000","last_clean":"2024-02-10T08:35:23.873545+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:35:23.873545+0000","last_undegraded":"2024-02-10T08:35:23.873545+0000","last_fullsized":"2024-02-10T08:35:23.873545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698215+0000","last_change":"2024-02-10T08:35:00.936215+0000","last_active":"2024-02-10T08:35:23.698215+0000","last_peered":"2024-02-10T08:35:23.698215+0000","last_clean":"2024-02-10T08:35:23.698215+0000","last_became_active":"2024-02-10T08:33:22.119141+0000","last_became_peered":"2024-02-10T08:33:22.119141+0000","last_unstale":"2024-02-10T08:35:23.698215+0000","last_undegraded":"2024-02-10T08:35:23.698215+0000","last_fullsized":"2024-02-10T08:35:23.698215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698810+0000","last_change":"2024-02-10T08:34:26.673356+0000","last_active":"2024-02-10T08:35:23.698810+0000","last_peered":"2024-02-10T08:35:23.698810+0000","last_clean":"2024-02-10T08:35:23.698810+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:35:23.698810+0000","last_undegraded":"2024-02-10T08:35:23.698810+0000","last_fullsized":"2024-02-10T08:35:23.698810+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:26.673263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.673263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707188+0000","last_change":"2024-02-10T08:34:50.094010+0000","last_active":"2024-02-10T08:35:24.707188+0000","last_peered":"2024-02-10T08:35:24.707188+0000","last_clean":"2024-02-10T08:35:24.707188+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:35:24.707188+0000","last_undegraded":"2024-02-10T08:35:24.707188+0000","last_fullsized":"2024-02-10T08:35:24.707188+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:34:50.093915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.093915+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873384+0000","last_change":"2024-02-10T08:34:51.681056+0000","last_active":"2024-02-10T08:35:23.873384+0000","last_peered":"2024-02-10T08:35:23.873384+0000","last_clean":"2024-02-10T08:35:23.873384+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:35:23.873384+0000","last_undegraded":"2024-02-10T08:35:23.873384+0000","last_fullsized":"2024-02-10T08:35:23.873384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.680986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.680986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874393+0000","last_change":"2024-02-10T08:35:01.492010+0000","last_active":"2024-02-10T08:35:23.874393+0000","last_peered":"2024-02-10T08:35:23.874393+0000","last_clean":"2024-02-10T08:35:23.874393+0000","last_became_active":"2024-02-10T08:33:22.122241+0000","last_became_peered":"2024-02-10T08:33:22.122241+0000","last_unstale":"2024-02-10T08:35:23.874393+0000","last_undegraded":"2024-02-10T08:35:23.874393+0000","last_fullsized":"2024-02-10T08:35:23.874393+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698601+0000","last_change":"2024-02-10T08:34:56.956608+0000","last_active":"2024-02-10T08:35:23.698601+0000","last_peered":"2024-02-10T08:35:23.698601+0000","last_clean":"2024-02-10T08:35:23.698601+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:35:23.698601+0000","last_undegraded":"2024-02-10T08:35:23.698601+0000","last_fullsized":"2024-02-10T08:35:23.698601+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.956483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.956483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":56,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707136+0000","last_change":"2024-02-10T08:34:59.084224+0000","last_active":"2024-02-10T08:35:24.707136+0000","last_peered":"2024-02-10T08:35:24.707136+0000","last_clean":"2024-02-10T08:35:24.707136+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:35:24.707136+0000","last_undegraded":"2024-02-10T08:35:24.707136+0000","last_fullsized":"2024-02-10T08:35:24.707136+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:59.084151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:59.084151+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874161+0000","last_change":"2024-02-10T08:34:52.516328+0000","last_active":"2024-02-10T08:35:23.874161+0000","last_peered":"2024-02-10T08:35:23.874161+0000","last_clean":"2024-02-10T08:35:23.874161+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:35:23.874161+0000","last_undegraded":"2024-02-10T08:35:23.874161+0000","last_fullsized":"2024-02-10T08:35:23.874161+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.516236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.516236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707489+0000","last_change":"2024-02-10T08:35:07.038806+0000","last_active":"2024-02-10T08:35:24.707489+0000","last_peered":"2024-02-10T08:35:24.707489+0000","last_clean":"2024-02-10T08:35:24.707489+0000","last_became_active":"2024-02-10T08:33:22.116085+0000","last_became_peered":"2024-02-10T08:33:22.116085+0000","last_unstale":"2024-02-10T08:35:24.707489+0000","last_undegraded":"2024-02-10T08:35:24.707489+0000","last_fullsized":"2024-02-10T08:35:24.707489+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873555+0000","last_change":"2024-02-10T08:34:37.616548+0000","last_active":"2024-02-10T08:35:23.873555+0000","last_peered":"2024-02-10T08:35:23.873555+0000","last_clean":"2024-02-10T08:35:23.873555+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:35:23.873555+0000","last_undegraded":"2024-02-10T08:35:23.873555+0000","last_fullsized":"2024-02-10T08:35:23.873555+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.616436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.616436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873925+0000","last_change":"2024-02-10T08:34:33.601286+0000","last_active":"2024-02-10T08:35:23.873925+0000","last_peered":"2024-02-10T08:35:23.873925+0000","last_clean":"2024-02-10T08:35:23.873925+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:35:23.873925+0000","last_undegraded":"2024-02-10T08:35:23.873925+0000","last_fullsized":"2024-02-10T08:35:23.873925+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:33.601201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.601201+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707880+0000","last_change":"2024-02-10T08:35:17.895268+0000","last_active":"2024-02-10T08:35:24.707880+0000","last_peered":"2024-02-10T08:35:24.707880+0000","last_clean":"2024-02-10T08:35:24.707880+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:35:24.707880+0000","last_undegraded":"2024-02-10T08:35:24.707880+0000","last_fullsized":"2024-02-10T08:35:24.707880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:17.895163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:17.895163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708351+0000","last_change":"2024-02-10T08:34:39.357828+0000","last_active":"2024-02-10T08:35:24.708351+0000","last_peered":"2024-02-10T08:35:24.708351+0000","last_clean":"2024-02-10T08:35:24.708351+0000","last_became_active":"2024-02-10T08:33:22.120631+0000","last_became_peered":"2024-02-10T08:33:22.120631+0000","last_unstale":"2024-02-10T08:35:24.708351+0000","last_undegraded":"2024-02-10T08:35:24.708351+0000","last_fullsized":"2024-02-10T08:35:24.708351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.357766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873911+0000","last_change":"2024-02-10T08:34:53.298291+0000","last_active":"2024-02-10T08:35:23.873911+0000","last_peered":"2024-02-10T08:35:23.873911+0000","last_clean":"2024-02-10T08:35:23.873911+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:35:23.873911+0000","last_undegraded":"2024-02-10T08:35:23.873911+0000","last_fullsized":"2024-02-10T08:35:23.873911+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.298226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.298226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":65,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.710186+0000","last_change":"2024-02-10T08:34:26.526319+0000","last_active":"2024-02-10T08:35:24.710186+0000","last_peered":"2024-02-10T08:35:24.710186+0000","last_clean":"2024-02-10T08:35:24.710186+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:35:24.710186+0000","last_undegraded":"2024-02-10T08:35:24.710186+0000","last_fullsized":"2024-02-10T08:35:24.710186+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:34:26.526206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.526206+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708232+0000","last_change":"2024-02-10T08:34:52.285891+0000","last_active":"2024-02-10T08:35:24.708232+0000","last_peered":"2024-02-10T08:35:24.708232+0000","last_clean":"2024-02-10T08:35:24.708232+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:35:24.708232+0000","last_undegraded":"2024-02-10T08:35:24.708232+0000","last_fullsized":"2024-02-10T08:35:24.708232+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.285824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.285824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873983+0000","last_change":"2024-02-10T08:35:01.687455+0000","last_active":"2024-02-10T08:35:23.873983+0000","last_peered":"2024-02-10T08:35:23.873983+0000","last_clean":"2024-02-10T08:35:23.873983+0000","last_became_active":"2024-02-10T08:33:22.122436+0000","last_became_peered":"2024-02-10T08:33:22.122436+0000","last_unstale":"2024-02-10T08:35:23.873983+0000","last_undegraded":"2024-02-10T08:35:23.873983+0000","last_fullsized":"2024-02-10T08:35:23.873983+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.687386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873948+0000","last_change":"2024-02-10T08:34:24.604411+0000","last_active":"2024-02-10T08:35:23.873948+0000","last_peered":"2024-02-10T08:35:23.873948+0000","last_clean":"2024-02-10T08:35:23.873948+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:35:23.873948+0000","last_undegraded":"2024-02-10T08:35:23.873948+0000","last_fullsized":"2024-02-10T08:35:23.873948+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.604347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.604347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874593+0000","last_change":"2024-02-10T08:34:36.450017+0000","last_active":"2024-02-10T08:35:23.874593+0000","last_peered":"2024-02-10T08:35:23.874593+0000","last_clean":"2024-02-10T08:35:23.874593+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:35:23.874593+0000","last_undegraded":"2024-02-10T08:35:23.874593+0000","last_fullsized":"2024-02-10T08:35:23.874593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:36.449943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.449943+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698961+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:35:23.698961+0000","last_peered":"2024-02-10T08:35:23.698961+0000","last_clean":"2024-02-10T08:35:23.698961+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:35:23.698961+0000","last_undegraded":"2024-02-10T08:35:23.698961+0000","last_fullsized":"2024-02-10T08:35:23.698961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'9","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.911401+0000","last_change":"2024-02-10T08:34:35.421984+0000","last_active":"2024-02-10T08:35:24.911401+0000","last_peered":"2024-02-10T08:35:24.911401+0000","last_clean":"2024-02-10T08:35:24.911401+0000","last_became_active":"2024-02-10T08:33:22.119251+0000","last_became_peered":"2024-02-10T08:33:22.119251+0000","last_unstale":"2024-02-10T08:35:24.911401+0000","last_undegraded":"2024-02-10T08:35:24.911401+0000","last_fullsized":"2024-02-10T08:35:24.911401+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_deep_scrub":"74'9","last_deep_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.421866+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875134+0000","last_change":"2024-02-10T08:34:41.695373+0000","last_active":"2024-02-10T08:35:23.875134+0000","last_peered":"2024-02-10T08:35:23.875134+0000","last_clean":"2024-02-10T08:35:23.875134+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:35:23.875134+0000","last_undegraded":"2024-02-10T08:35:23.875134+0000","last_fullsized":"2024-02-10T08:35:23.875134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.661214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.661214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873930+0000","last_change":"2024-02-10T08:34:43.680148+0000","last_active":"2024-02-10T08:35:23.873930+0000","last_peered":"2024-02-10T08:35:23.873930+0000","last_clean":"2024-02-10T08:35:23.873930+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:35:23.873930+0000","last_undegraded":"2024-02-10T08:35:23.873930+0000","last_fullsized":"2024-02-10T08:35:23.873930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:43.680073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:43.680073+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873801+0000","last_change":"2024-02-10T08:34:32.620154+0000","last_active":"2024-02-10T08:35:23.873801+0000","last_peered":"2024-02-10T08:35:23.873801+0000","last_clean":"2024-02-10T08:35:23.873801+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:35:23.873801+0000","last_undegraded":"2024-02-10T08:35:23.873801+0000","last_fullsized":"2024-02-10T08:35:23.873801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:32.620086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.620086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698346+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:35:23.698346+0000","last_peered":"2024-02-10T08:35:23.698346+0000","last_clean":"2024-02-10T08:35:23.698346+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:35:23.698346+0000","last_undegraded":"2024-02-10T08:35:23.698346+0000","last_fullsized":"2024-02-10T08:35:23.698346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874837+0000","last_change":"2024-02-10T08:34:31.359650+0000","last_active":"2024-02-10T08:35:23.874837+0000","last_peered":"2024-02-10T08:35:23.874837+0000","last_clean":"2024-02-10T08:35:23.874837+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:35:23.874837+0000","last_undegraded":"2024-02-10T08:35:23.874837+0000","last_fullsized":"2024-02-10T08:35:23.874837+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":57,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874140+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:35:23.874140+0000","last_peered":"2024-02-10T08:35:23.874140+0000","last_clean":"2024-02-10T08:35:23.874140+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:35:23.874140+0000","last_undegraded":"2024-02-10T08:35:23.874140+0000","last_fullsized":"2024-02-10T08:35:23.874140+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698086+0000","last_change":"2024-02-10T08:34:35.819857+0000","last_active":"2024-02-10T08:35:23.698086+0000","last_peered":"2024-02-10T08:35:23.698086+0000","last_clean":"2024-02-10T08:35:23.698086+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:35:23.698086+0000","last_undegraded":"2024-02-10T08:35:23.698086+0000","last_fullsized":"2024-02-10T08:35:23.698086+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.819777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.819777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873517+0000","last_change":"2024-02-10T08:34:55.286635+0000","last_active":"2024-02-10T08:35:23.873517+0000","last_peered":"2024-02-10T08:35:23.873517+0000","last_clean":"2024-02-10T08:35:23.873517+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:35:23.873517+0000","last_undegraded":"2024-02-10T08:35:23.873517+0000","last_fullsized":"2024-02-10T08:35:23.873517+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.286540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.286540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875714+0000","last_change":"2024-02-10T08:34:28.590439+0000","last_active":"2024-02-10T08:35:23.875714+0000","last_peered":"2024-02-10T08:35:23.875714+0000","last_clean":"2024-02-10T08:35:23.875714+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:35:23.875714+0000","last_undegraded":"2024-02-10T08:35:23.875714+0000","last_fullsized":"2024-02-10T08:35:23.875714+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.590362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.590362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873743+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:35:23.873743+0000","last_peered":"2024-02-10T08:35:23.873743+0000","last_clean":"2024-02-10T08:35:23.873743+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:35:23.873743+0000","last_undegraded":"2024-02-10T08:35:23.873743+0000","last_fullsized":"2024-02-10T08:35:23.873743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873784+0000","last_change":"2024-02-10T08:34:28.605230+0000","last_active":"2024-02-10T08:35:23.873784+0000","last_peered":"2024-02-10T08:35:23.873784+0000","last_clean":"2024-02-10T08:35:23.873784+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:35:23.873784+0000","last_undegraded":"2024-02-10T08:35:23.873784+0000","last_fullsized":"2024-02-10T08:35:23.873784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.605166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.605166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707489+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:35:24.707489+0000","last_peered":"2024-02-10T08:35:24.707489+0000","last_clean":"2024-02-10T08:35:24.707489+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:35:24.707489+0000","last_undegraded":"2024-02-10T08:35:24.707489+0000","last_fullsized":"2024-02-10T08:35:24.707489+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708402+0000","last_change":"2024-02-10T08:34:36.300831+0000","last_active":"2024-02-10T08:35:24.708402+0000","last_peered":"2024-02-10T08:35:24.708402+0000","last_clean":"2024-02-10T08:35:24.708402+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:35:24.708402+0000","last_undegraded":"2024-02-10T08:35:24.708402+0000","last_fullsized":"2024-02-10T08:35:24.708402+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:36.300770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.300770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707312+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:35:24.707312+0000","last_peered":"2024-02-10T08:35:24.707312+0000","last_clean":"2024-02-10T08:35:24.707312+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:35:24.707312+0000","last_undegraded":"2024-02-10T08:35:24.707312+0000","last_fullsized":"2024-02-10T08:35:24.707312+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872742+0000","last_change":"2024-02-10T08:34:48.691828+0000","last_active":"2024-02-10T08:35:23.872742+0000","last_peered":"2024-02-10T08:35:23.872742+0000","last_clean":"2024-02-10T08:35:23.872742+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:35:23.872742+0000","last_undegraded":"2024-02-10T08:35:23.872742+0000","last_fullsized":"2024-02-10T08:35:23.872742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.691737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.691737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873601+0000","last_change":"2024-02-10T08:34:27.584434+0000","last_active":"2024-02-10T08:35:23.873601+0000","last_peered":"2024-02-10T08:35:23.873601+0000","last_clean":"2024-02-10T08:35:23.873601+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:35:23.873601+0000","last_undegraded":"2024-02-10T08:35:23.873601+0000","last_fullsized":"2024-02-10T08:35:23.873601+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:27.584369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.584369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874118+0000","last_change":"2024-02-10T08:34:23.544050+0000","last_active":"2024-02-10T08:35:23.874118+0000","last_peered":"2024-02-10T08:35:23.874118+0000","last_clean":"2024-02-10T08:35:23.874118+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:35:23.874118+0000","last_undegraded":"2024-02-10T08:35:23.874118+0000","last_fullsized":"2024-02-10T08:35:23.874118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.543984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.543984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707439+0000","last_change":"2024-02-10T08:34:47.420238+0000","last_active":"2024-02-10T08:35:24.707439+0000","last_peered":"2024-02-10T08:35:24.707439+0000","last_clean":"2024-02-10T08:35:24.707439+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:35:24.707439+0000","last_undegraded":"2024-02-10T08:35:24.707439+0000","last_fullsized":"2024-02-10T08:35:24.707439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.420174+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:29.834237+0000","last_change":"2024-02-10T08:35:29.834206+0000","last_active":"2024-02-10T08:35:29.834237+0000","last_peered":"2024-02-10T08:35:29.834237+0000","last_clean":"2024-02-10T08:35:29.834237+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:35:29.834237+0000","last_undegraded":"2024-02-10T08:35:29.834237+0000","last_fullsized":"2024-02-10T08:35:29.834237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:29.834114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:29.834114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.910289+0000","last_change":"2024-02-10T08:34:51.497308+0000","last_active":"2024-02-10T08:35:24.910289+0000","last_peered":"2024-02-10T08:35:24.910289+0000","last_clean":"2024-02-10T08:35:24.910289+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:35:24.910289+0000","last_undegraded":"2024-02-10T08:35:24.910289+0000","last_fullsized":"2024-02-10T08:35:24.910289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.497242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.497242+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873394+0000","last_change":"2024-02-10T08:34:35.635203+0000","last_active":"2024-02-10T08:35:23.873394+0000","last_peered":"2024-02-10T08:35:23.873394+0000","last_clean":"2024-02-10T08:35:23.873394+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:35:23.873394+0000","last_undegraded":"2024-02-10T08:35:23.873394+0000","last_fullsized":"2024-02-10T08:35:23.873394+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.635108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.635108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875035+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:35:23.875035+0000","last_peered":"2024-02-10T08:35:23.875035+0000","last_clean":"2024-02-10T08:35:23.875035+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:35:23.875035+0000","last_undegraded":"2024-02-10T08:35:23.875035+0000","last_fullsized":"2024-02-10T08:35:23.875035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873743+0000","last_change":"2024-02-10T08:35:02.705789+0000","last_active":"2024-02-10T08:35:23.873743+0000","last_peered":"2024-02-10T08:35:23.873743+0000","last_clean":"2024-02-10T08:35:23.873743+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:35:23.873743+0000","last_undegraded":"2024-02-10T08:35:23.873743+0000","last_fullsized":"2024-02-10T08:35:23.873743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.705679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.705679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":71,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.744642+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:35:24.744642+0000","last_peered":"2024-02-10T08:35:24.744642+0000","last_clean":"2024-02-10T08:35:24.744642+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:35:24.744642+0000","last_undegraded":"2024-02-10T08:35:24.744642+0000","last_fullsized":"2024-02-10T08:35:24.744642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873765+0000","last_change":"2024-02-10T08:34:29.584600+0000","last_active":"2024-02-10T08:35:23.873765+0000","last_peered":"2024-02-10T08:35:23.873765+0000","last_clean":"2024-02-10T08:35:23.873765+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:35:23.873765+0000","last_undegraded":"2024-02-10T08:35:23.873765+0000","last_fullsized":"2024-02-10T08:35:23.873765+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.584529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.584529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873761+0000","last_change":"2024-02-10T08:34:31.663950+0000","last_active":"2024-02-10T08:35:23.873761+0000","last_peered":"2024-02-10T08:35:23.873761+0000","last_clean":"2024-02-10T08:35:23.873761+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:35:23.873761+0000","last_undegraded":"2024-02-10T08:35:23.873761+0000","last_fullsized":"2024-02-10T08:35:23.873761+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:31.663882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.663882+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698704+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:35:23.698704+0000","last_peered":"2024-02-10T08:35:23.698704+0000","last_clean":"2024-02-10T08:35:23.698704+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:35:23.698704+0000","last_undegraded":"2024-02-10T08:35:23.698704+0000","last_fullsized":"2024-02-10T08:35:23.698704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875606+0000","last_change":"2024-02-10T08:34:45.607331+0000","last_active":"2024-02-10T08:35:23.875606+0000","last_peered":"2024-02-10T08:35:23.875606+0000","last_clean":"2024-02-10T08:35:23.875606+0000","last_became_active":"2024-02-10T08:33:22.121591+0000","last_became_peered":"2024-02-10T08:33:22.121591+0000","last_unstale":"2024-02-10T08:35:23.875606+0000","last_undegraded":"2024-02-10T08:35:23.875606+0000","last_fullsized":"2024-02-10T08:35:23.875606+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.607254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.607254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873597+0000","last_change":"2024-02-10T08:34:29.402936+0000","last_active":"2024-02-10T08:35:23.873597+0000","last_peered":"2024-02-10T08:35:23.873597+0000","last_clean":"2024-02-10T08:35:23.873597+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:35:23.873597+0000","last_undegraded":"2024-02-10T08:35:23.873597+0000","last_fullsized":"2024-02-10T08:35:23.873597+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.402874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.402874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874117+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:35:23.874117+0000","last_peered":"2024-02-10T08:35:23.874117+0000","last_clean":"2024-02-10T08:35:23.874117+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:35:23.874117+0000","last_undegraded":"2024-02-10T08:35:23.874117+0000","last_fullsized":"2024-02-10T08:35:23.874117+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709998+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:35:24.709998+0000","last_peered":"2024-02-10T08:35:24.709998+0000","last_clean":"2024-02-10T08:35:24.709998+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:35:24.709998+0000","last_undegraded":"2024-02-10T08:35:24.709998+0000","last_fullsized":"2024-02-10T08:35:24.709998+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874222+0000","last_change":"2024-02-10T08:35:02.446706+0000","last_active":"2024-02-10T08:35:23.874222+0000","last_peered":"2024-02-10T08:35:23.874222+0000","last_clean":"2024-02-10T08:35:23.874222+0000","last_became_active":"2024-02-10T08:33:22.118323+0000","last_became_peered":"2024-02-10T08:33:22.118323+0000","last_unstale":"2024-02-10T08:35:23.874222+0000","last_undegraded":"2024-02-10T08:35:23.874222+0000","last_fullsized":"2024-02-10T08:35:23.874222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708793+0000","last_change":"2024-02-10T08:34:56.699511+0000","last_active":"2024-02-10T08:35:24.708793+0000","last_peered":"2024-02-10T08:35:24.708793+0000","last_clean":"2024-02-10T08:35:24.708793+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:35:24.708793+0000","last_undegraded":"2024-02-10T08:35:24.708793+0000","last_fullsized":"2024-02-10T08:35:24.708793+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.699420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.699420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'9","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873837+0000","last_change":"2024-02-10T08:34:23.674473+0000","last_active":"2024-02-10T08:35:23.873837+0000","last_peered":"2024-02-10T08:35:23.873837+0000","last_clean":"2024-02-10T08:35:23.873837+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:35:23.873837+0000","last_undegraded":"2024-02-10T08:35:23.873837+0000","last_fullsized":"2024-02-10T08:35:23.873837+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:23.674404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.674404+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873624+0000","last_change":"2024-02-10T08:34:45.519207+0000","last_active":"2024-02-10T08:35:23.873624+0000","last_peered":"2024-02-10T08:35:23.873624+0000","last_clean":"2024-02-10T08:35:23.873624+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:35:23.873624+0000","last_undegraded":"2024-02-10T08:35:23.873624+0000","last_fullsized":"2024-02-10T08:35:23.873624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.519116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.519116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707883+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:35:24.707883+0000","last_peered":"2024-02-10T08:35:24.707883+0000","last_clean":"2024-02-10T08:35:24.707883+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:35:24.707883+0000","last_undegraded":"2024-02-10T08:35:24.707883+0000","last_fullsized":"2024-02-10T08:35:24.707883+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":68,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707364+0000","last_change":"2024-02-10T08:34:54.039656+0000","last_active":"2024-02-10T08:35:24.707364+0000","last_peered":"2024-02-10T08:35:24.707364+0000","last_clean":"2024-02-10T08:35:24.707364+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:35:24.707364+0000","last_undegraded":"2024-02-10T08:35:24.707364+0000","last_fullsized":"2024-02-10T08:35:24.707364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.039584+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708845+0000","last_change":"2024-02-10T08:34:39.590686+0000","last_active":"2024-02-10T08:35:24.708845+0000","last_peered":"2024-02-10T08:35:24.708845+0000","last_clean":"2024-02-10T08:35:24.708845+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:35:24.708845+0000","last_undegraded":"2024-02-10T08:35:24.708845+0000","last_fullsized":"2024-02-10T08:35:24.708845+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.590604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.590604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872756+0000","last_change":"2024-02-10T08:34:31.544112+0000","last_active":"2024-02-10T08:35:23.872756+0000","last_peered":"2024-02-10T08:35:23.872756+0000","last_clean":"2024-02-10T08:35:23.872756+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:35:23.872756+0000","last_undegraded":"2024-02-10T08:35:23.872756+0000","last_fullsized":"2024-02-10T08:35:23.872756+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.544049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.544049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708408+0000","last_change":"2024-02-10T08:34:46.452573+0000","last_active":"2024-02-10T08:35:24.708408+0000","last_peered":"2024-02-10T08:35:24.708408+0000","last_clean":"2024-02-10T08:35:24.708408+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:35:24.708408+0000","last_undegraded":"2024-02-10T08:35:24.708408+0000","last_fullsized":"2024-02-10T08:35:24.708408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:46.452504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:46.452504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'10","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707536+0000","last_change":"2024-02-10T08:34:33.231522+0000","last_active":"2024-02-10T08:35:24.707536+0000","last_peered":"2024-02-10T08:35:24.707536+0000","last_clean":"2024-02-10T08:35:24.707536+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:35:24.707536+0000","last_undegraded":"2024-02-10T08:35:24.707536+0000","last_fullsized":"2024-02-10T08:35:24.707536+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:33.231411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.231411+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.910974+0000","last_change":"2024-02-10T08:34:47.693139+0000","last_active":"2024-02-10T08:35:24.910974+0000","last_peered":"2024-02-10T08:35:24.910974+0000","last_clean":"2024-02-10T08:35:24.910974+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:35:24.910974+0000","last_undegraded":"2024-02-10T08:35:24.910974+0000","last_fullsized":"2024-02-10T08:35:24.910974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:47.693030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.693030+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874330+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:35:23.874330+0000","last_peered":"2024-02-10T08:35:23.874330+0000","last_clean":"2024-02-10T08:35:23.874330+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:35:23.874330+0000","last_undegraded":"2024-02-10T08:35:23.874330+0000","last_fullsized":"2024-02-10T08:35:23.874330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873805+0000","last_change":"2024-02-10T08:34:20.652012+0000","last_active":"2024-02-10T08:35:23.873805+0000","last_peered":"2024-02-10T08:35:23.873805+0000","last_clean":"2024-02-10T08:35:23.873805+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:35:23.873805+0000","last_undegraded":"2024-02-10T08:35:23.873805+0000","last_fullsized":"2024-02-10T08:35:23.873805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:20.651946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:20.651946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":52,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.744267+0000","last_change":"2024-02-10T08:34:47.549753+0000","last_active":"2024-02-10T08:35:24.744267+0000","last_peered":"2024-02-10T08:35:24.744267+0000","last_clean":"2024-02-10T08:35:24.744267+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:35:24.744267+0000","last_undegraded":"2024-02-10T08:35:24.744267+0000","last_fullsized":"2024-02-10T08:35:24.744267+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:34:47.549671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.549671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873270+0000","last_change":"2024-02-10T08:35:00.639772+0000","last_active":"2024-02-10T08:35:23.873270+0000","last_peered":"2024-02-10T08:35:23.873270+0000","last_clean":"2024-02-10T08:35:23.873270+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:35:23.873270+0000","last_undegraded":"2024-02-10T08:35:23.873270+0000","last_fullsized":"2024-02-10T08:35:23.873270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.639705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.639705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699014+0000","last_change":"2024-02-10T08:34:51.910276+0000","last_active":"2024-02-10T08:35:23.699014+0000","last_peered":"2024-02-10T08:35:23.699014+0000","last_clean":"2024-02-10T08:35:23.699014+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:35:23.699014+0000","last_undegraded":"2024-02-10T08:35:23.699014+0000","last_fullsized":"2024-02-10T08:35:23.699014+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.910184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":172,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:30.287901+0000","last_change":"2024-02-10T08:35:07.227647+0000","last_active":"2024-02-10T08:35:30.287901+0000","last_peered":"2024-02-10T08:35:30.287901+0000","last_clean":"2024-02-10T08:35:30.287901+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:35:30.287901+0000","last_undegraded":"2024-02-10T08:35:30.287901+0000","last_fullsized":"2024-02-10T08:35:30.287901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.227587+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":121,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709000+0000","last_change":"2024-02-10T08:34:52.698430+0000","last_active":"2024-02-10T08:35:24.709000+0000","last_peered":"2024-02-10T08:35:24.709000+0000","last_clean":"2024-02-10T08:35:24.709000+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:35:24.709000+0000","last_undegraded":"2024-02-10T08:35:24.709000+0000","last_fullsized":"2024-02-10T08:35:24.709000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.698329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708956+0000","last_change":"2024-02-10T08:34:27.278457+0000","last_active":"2024-02-10T08:35:24.708956+0000","last_peered":"2024-02-10T08:35:24.708956+0000","last_clean":"2024-02-10T08:35:24.708956+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:35:24.708956+0000","last_undegraded":"2024-02-10T08:35:24.708956+0000","last_fullsized":"2024-02-10T08:35:24.708956+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:34:27.278388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:27.278388+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709624+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:35:24.709624+0000","last_peered":"2024-02-10T08:35:24.709624+0000","last_clean":"2024-02-10T08:35:24.709624+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:35:24.709624+0000","last_undegraded":"2024-02-10T08:35:24.709624+0000","last_fullsized":"2024-02-10T08:35:24.709624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875650+0000","last_change":"2024-02-10T08:34:55.443465+0000","last_active":"2024-02-10T08:35:23.875650+0000","last_peered":"2024-02-10T08:35:23.875650+0000","last_clean":"2024-02-10T08:35:23.875650+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:35:23.875650+0000","last_undegraded":"2024-02-10T08:35:23.875650+0000","last_fullsized":"2024-02-10T08:35:23.875650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.443390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.443390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873321+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:35:23.873321+0000","last_peered":"2024-02-10T08:35:23.873321+0000","last_clean":"2024-02-10T08:35:23.873321+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:35:23.873321+0000","last_undegraded":"2024-02-10T08:35:23.873321+0000","last_fullsized":"2024-02-10T08:35:23.873321+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874312+0000","last_change":"2024-02-10T08:34:52.491498+0000","last_active":"2024-02-10T08:35:23.874312+0000","last_peered":"2024-02-10T08:35:23.874312+0000","last_clean":"2024-02-10T08:35:23.874312+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:35:23.874312+0000","last_undegraded":"2024-02-10T08:35:23.874312+0000","last_fullsized":"2024-02-10T08:35:23.874312+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.491420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.491420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698072+0000","last_change":"2024-02-10T08:34:48.841104+0000","last_active":"2024-02-10T08:35:23.698072+0000","last_peered":"2024-02-10T08:35:23.698072+0000","last_clean":"2024-02-10T08:35:23.698072+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:35:23.698072+0000","last_undegraded":"2024-02-10T08:35:23.698072+0000","last_fullsized":"2024-02-10T08:35:23.698072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.841025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.841025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875145+0000","last_change":"2024-02-10T08:34:39.378393+0000","last_active":"2024-02-10T08:35:23.875145+0000","last_peered":"2024-02-10T08:35:23.875145+0000","last_clean":"2024-02-10T08:35:23.875145+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:35:23.875145+0000","last_undegraded":"2024-02-10T08:35:23.875145+0000","last_fullsized":"2024-02-10T08:35:23.875145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:39.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.378312+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698718+0000","last_change":"2024-02-10T08:34:40.809042+0000","last_active":"2024-02-10T08:35:23.698718+0000","last_peered":"2024-02-10T08:35:23.698718+0000","last_clean":"2024-02-10T08:35:23.698718+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:35:23.698718+0000","last_undegraded":"2024-02-10T08:35:23.698718+0000","last_fullsized":"2024-02-10T08:35:23.698718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:40.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:40.808978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708143+0000","last_change":"2024-02-10T08:34:28.307074+0000","last_active":"2024-02-10T08:35:24.708143+0000","last_peered":"2024-02-10T08:35:24.708143+0000","last_clean":"2024-02-10T08:35:24.708143+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:35:24.708143+0000","last_undegraded":"2024-02-10T08:35:24.708143+0000","last_fullsized":"2024-02-10T08:35:24.708143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:34:28.306997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.306997+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873041+0000","last_change":"2024-02-10T08:34:21.731002+0000","last_active":"2024-02-10T08:35:23.873041+0000","last_peered":"2024-02-10T08:35:23.873041+0000","last_clean":"2024-02-10T08:35:23.873041+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:35:23.873041+0000","last_undegraded":"2024-02-10T08:35:23.873041+0000","last_fullsized":"2024-02-10T08:35:23.873041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:21.730937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.730937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:27.051918+0000","last_change":"2024-02-10T08:35:27.051918+0000","last_active":"2024-02-10T08:35:27.051918+0000","last_peered":"2024-02-10T08:35:27.051918+0000","last_clean":"2024-02-10T08:35:27.051918+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:35:27.051918+0000","last_undegraded":"2024-02-10T08:35:27.051918+0000","last_fullsized":"2024-02-10T08:35:27.051918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:27.051837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:27.051837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873755+0000","last_change":"2024-02-10T08:34:28.599042+0000","last_active":"2024-02-10T08:35:23.873755+0000","last_peered":"2024-02-10T08:35:23.873755+0000","last_clean":"2024-02-10T08:35:23.873755+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:35:23.873755+0000","last_undegraded":"2024-02-10T08:35:23.873755+0000","last_fullsized":"2024-02-10T08:35:23.873755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:28.598979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.598979+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875613+0000","last_change":"2024-02-10T08:34:38.628666+0000","last_active":"2024-02-10T08:35:23.875613+0000","last_peered":"2024-02-10T08:35:23.875613+0000","last_clean":"2024-02-10T08:35:23.875613+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:35:23.875613+0000","last_undegraded":"2024-02-10T08:35:23.875613+0000","last_fullsized":"2024-02-10T08:35:23.875613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.628559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.628559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707723+0000","last_change":"2024-02-10T08:34:44.434673+0000","last_active":"2024-02-10T08:35:24.707723+0000","last_peered":"2024-02-10T08:35:24.707723+0000","last_clean":"2024-02-10T08:35:24.707723+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:35:24.707723+0000","last_undegraded":"2024-02-10T08:35:24.707723+0000","last_fullsized":"2024-02-10T08:35:24.707723+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.434603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.434603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.699102+0000","last_change":"2024-02-10T08:34:44.810962+0000","last_active":"2024-02-10T08:35:23.699102+0000","last_peered":"2024-02-10T08:35:23.699102+0000","last_clean":"2024-02-10T08:35:23.699102+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:35:23.699102+0000","last_undegraded":"2024-02-10T08:35:23.699102+0000","last_fullsized":"2024-02-10T08:35:23.699102+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.810886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.810886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873372+0000","last_change":"2024-02-10T08:34:39.703715+0000","last_active":"2024-02-10T08:35:23.873372+0000","last_peered":"2024-02-10T08:35:23.873372+0000","last_clean":"2024-02-10T08:35:23.873372+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:35:23.873372+0000","last_undegraded":"2024-02-10T08:35:23.873372+0000","last_fullsized":"2024-02-10T08:35:23.873372+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:39.703643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.703643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873988+0000","last_change":"2024-02-10T08:34:51.348414+0000","last_active":"2024-02-10T08:35:23.873988+0000","last_peered":"2024-02-10T08:35:23.873988+0000","last_clean":"2024-02-10T08:35:23.873988+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:35:23.873988+0000","last_undegraded":"2024-02-10T08:35:23.873988+0000","last_fullsized":"2024-02-10T08:35:23.873988+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.348334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.348334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707396+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:35:24.707396+0000","last_peered":"2024-02-10T08:35:24.707396+0000","last_clean":"2024-02-10T08:35:24.707396+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:35:24.707396+0000","last_undegraded":"2024-02-10T08:35:24.707396+0000","last_fullsized":"2024-02-10T08:35:24.707396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708457+0000","last_change":"2024-02-10T08:35:03.147093+0000","last_active":"2024-02-10T08:35:24.708457+0000","last_peered":"2024-02-10T08:35:24.708457+0000","last_clean":"2024-02-10T08:35:24.708457+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:35:24.708457+0000","last_undegraded":"2024-02-10T08:35:24.708457+0000","last_fullsized":"2024-02-10T08:35:24.708457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:35:03.146990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.146990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":78,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872862+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:35:23.872862+0000","last_peered":"2024-02-10T08:35:23.872862+0000","last_clean":"2024-02-10T08:35:23.872862+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:35:23.872862+0000","last_undegraded":"2024-02-10T08:35:23.872862+0000","last_fullsized":"2024-02-10T08:35:23.872862+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707353+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:35:24.707353+0000","last_peered":"2024-02-10T08:35:24.707353+0000","last_clean":"2024-02-10T08:35:24.707353+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:35:24.707353+0000","last_undegraded":"2024-02-10T08:35:24.707353+0000","last_fullsized":"2024-02-10T08:35:24.707353+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698690+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:35:23.698690+0000","last_peered":"2024-02-10T08:35:23.698690+0000","last_clean":"2024-02-10T08:35:23.698690+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:35:23.698690+0000","last_undegraded":"2024-02-10T08:35:23.698690+0000","last_fullsized":"2024-02-10T08:35:23.698690+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874762+0000","last_change":"2024-02-10T08:34:22.584930+0000","last_active":"2024-02-10T08:35:23.874762+0000","last_peered":"2024-02-10T08:35:23.874762+0000","last_clean":"2024-02-10T08:35:23.874762+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:35:23.874762+0000","last_undegraded":"2024-02-10T08:35:23.874762+0000","last_fullsized":"2024-02-10T08:35:23.874762+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:22.584856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:22.584856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707239+0000","last_change":"2024-02-10T08:34:36.155885+0000","last_active":"2024-02-10T08:35:24.707239+0000","last_peered":"2024-02-10T08:35:24.707239+0000","last_clean":"2024-02-10T08:35:24.707239+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:35:24.707239+0000","last_undegraded":"2024-02-10T08:35:24.707239+0000","last_fullsized":"2024-02-10T08:35:24.707239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:36.155806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.155806+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":73,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.909639+0000","last_change":"2024-02-10T08:34:36.649251+0000","last_active":"2024-02-10T08:35:24.909639+0000","last_peered":"2024-02-10T08:35:24.909639+0000","last_clean":"2024-02-10T08:35:24.909639+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:35:24.909639+0000","last_undegraded":"2024-02-10T08:35:24.909639+0000","last_fullsized":"2024-02-10T08:35:24.909639+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:36.649186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.649186+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873181+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:35:23.873181+0000","last_peered":"2024-02-10T08:35:23.873181+0000","last_clean":"2024-02-10T08:35:23.873181+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:35:23.873181+0000","last_undegraded":"2024-02-10T08:35:23.873181+0000","last_fullsized":"2024-02-10T08:35:23.873181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875518+0000","last_change":"2024-02-10T08:35:06.520594+0000","last_active":"2024-02-10T08:35:23.875518+0000","last_peered":"2024-02-10T08:35:23.875518+0000","last_clean":"2024-02-10T08:35:23.875518+0000","last_became_active":"2024-02-10T08:33:22.122087+0000","last_became_peered":"2024-02-10T08:33:22.122087+0000","last_unstale":"2024-02-10T08:35:23.875518+0000","last_undegraded":"2024-02-10T08:35:23.875518+0000","last_fullsized":"2024-02-10T08:35:23.875518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.698196+0000","last_change":"2024-02-10T08:34:29.780708+0000","last_active":"2024-02-10T08:35:23.698196+0000","last_peered":"2024-02-10T08:35:23.698196+0000","last_clean":"2024-02-10T08:35:23.698196+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:35:23.698196+0000","last_undegraded":"2024-02-10T08:35:23.698196+0000","last_fullsized":"2024-02-10T08:35:23.698196+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.780644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.780644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707904+0000","last_change":"2024-02-10T08:34:25.085734+0000","last_active":"2024-02-10T08:35:24.707904+0000","last_peered":"2024-02-10T08:35:24.707904+0000","last_clean":"2024-02-10T08:35:24.707904+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:35:24.707904+0000","last_undegraded":"2024-02-10T08:35:24.707904+0000","last_fullsized":"2024-02-10T08:35:24.707904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:25.085663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.085663+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.874706+0000","last_change":"2024-02-10T08:35:16.510089+0000","last_active":"2024-02-10T08:35:23.874706+0000","last_peered":"2024-02-10T08:35:23.874706+0000","last_clean":"2024-02-10T08:35:23.874706+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:35:23.874706+0000","last_undegraded":"2024-02-10T08:35:23.874706+0000","last_fullsized":"2024-02-10T08:35:23.874706+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:26.514938+0000","last_change":"2024-02-10T08:35:11.026984+0000","last_active":"2024-02-10T08:35:26.514938+0000","last_peered":"2024-02-10T08:35:26.514938+0000","last_clean":"2024-02-10T08:35:26.514938+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:35:26.514938+0000","last_undegraded":"2024-02-10T08:35:26.514938+0000","last_fullsized":"2024-02-10T08:35:26.514938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707928+0000","last_change":"2024-02-10T08:34:32.208396+0000","last_active":"2024-02-10T08:35:24.707928+0000","last_peered":"2024-02-10T08:35:24.707928+0000","last_clean":"2024-02-10T08:35:24.707928+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:35:24.707928+0000","last_undegraded":"2024-02-10T08:35:24.707928+0000","last_fullsized":"2024-02-10T08:35:24.707928+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:34:32.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.208313+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":65,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708875+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:35:24.708875+0000","last_peered":"2024-02-10T08:35:24.708875+0000","last_clean":"2024-02-10T08:35:24.708875+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:35:24.708875+0000","last_undegraded":"2024-02-10T08:35:24.708875+0000","last_fullsized":"2024-02-10T08:35:24.708875+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707693+0000","last_change":"2024-02-10T08:34:35.201454+0000","last_active":"2024-02-10T08:35:24.707693+0000","last_peered":"2024-02-10T08:35:24.707693+0000","last_clean":"2024-02-10T08:35:24.707693+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:35:24.707693+0000","last_undegraded":"2024-02-10T08:35:24.707693+0000","last_fullsized":"2024-02-10T08:35:24.707693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.201371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.201371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:26.514717+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:35:26.514717+0000","last_peered":"2024-02-10T08:35:26.514717+0000","last_clean":"2024-02-10T08:35:26.514717+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:35:26.514717+0000","last_undegraded":"2024-02-10T08:35:26.514717+0000","last_fullsized":"2024-02-10T08:35:26.514717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707355+0000","last_change":"2024-02-10T08:34:24.362389+0000","last_active":"2024-02-10T08:35:24.707355+0000","last_peered":"2024-02-10T08:35:24.707355+0000","last_clean":"2024-02-10T08:35:24.707355+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:35:24.707355+0000","last_undegraded":"2024-02-10T08:35:24.707355+0000","last_fullsized":"2024-02-10T08:35:24.707355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.362309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.362309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.707453+0000","last_change":"2024-02-10T08:34:21.123617+0000","last_active":"2024-02-10T08:35:24.707453+0000","last_peered":"2024-02-10T08:35:24.707453+0000","last_clean":"2024-02-10T08:35:24.707453+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:35:24.707453+0000","last_undegraded":"2024-02-10T08:35:24.707453+0000","last_fullsized":"2024-02-10T08:35:24.707453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:21.123548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:21.123548+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709826+0000","last_change":"2024-02-10T08:34:53.743882+0000","last_active":"2024-02-10T08:35:24.709826+0000","last_peered":"2024-02-10T08:35:24.709826+0000","last_clean":"2024-02-10T08:35:24.709826+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:35:24.709826+0000","last_undegraded":"2024-02-10T08:35:24.709826+0000","last_fullsized":"2024-02-10T08:35:24.709826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.743807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.743807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.708226+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:35:24.708226+0000","last_peered":"2024-02-10T08:35:24.708226+0000","last_clean":"2024-02-10T08:35:24.708226+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:35:24.708226+0000","last_undegraded":"2024-02-10T08:35:24.708226+0000","last_fullsized":"2024-02-10T08:35:24.708226+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:30.123918+0000","last_change":"2024-02-10T08:35:30.123918+0000","last_active":"2024-02-10T08:35:30.123918+0000","last_peered":"2024-02-10T08:35:30.123918+0000","last_clean":"2024-02-10T08:35:30.123918+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:35:30.123918+0000","last_undegraded":"2024-02-10T08:35:30.123918+0000","last_fullsized":"2024-02-10T08:35:30.123918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.123828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.123828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-10T08:35:30.802385+0000","last_change":"2024-02-10T08:35:30.802385+0000","last_active":"2024-02-10T08:35:30.802385+0000","last_peered":"2024-02-10T08:35:30.802385+0000","last_clean":"2024-02-10T08:35:30.802385+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:35:30.802385+0000","last_undegraded":"2024-02-10T08:35:30.802385+0000","last_fullsized":"2024-02-10T08:35:30.802385+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:35:30.802306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.802306+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.697954+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:35:23.697954+0000","last_peered":"2024-02-10T08:35:23.697954+0000","last_clean":"2024-02-10T08:35:23.697954+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:35:23.697954+0000","last_undegraded":"2024-02-10T08:35:23.697954+0000","last_fullsized":"2024-02-10T08:35:23.697954+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709461+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:35:24.709461+0000","last_peered":"2024-02-10T08:35:24.709461+0000","last_clean":"2024-02-10T08:35:24.709461+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:35:24.709461+0000","last_undegraded":"2024-02-10T08:35:24.709461+0000","last_fullsized":"2024-02-10T08:35:24.709461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.710086+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:35:24.710086+0000","last_peered":"2024-02-10T08:35:24.710086+0000","last_clean":"2024-02-10T08:35:24.710086+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:35:24.710086+0000","last_undegraded":"2024-02-10T08:35:24.710086+0000","last_fullsized":"2024-02-10T08:35:24.710086+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873419+0000","last_change":"2024-02-10T08:34:44.381291+0000","last_active":"2024-02-10T08:35:23.873419+0000","last_peered":"2024-02-10T08:35:23.873419+0000","last_clean":"2024-02-10T08:35:23.873419+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:35:23.873419+0000","last_undegraded":"2024-02-10T08:35:23.873419+0000","last_fullsized":"2024-02-10T08:35:23.873419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:44.381208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.381208+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.872814+0000","last_change":"2024-02-10T08:34:29.554010+0000","last_active":"2024-02-10T08:35:23.872814+0000","last_peered":"2024-02-10T08:35:23.872814+0000","last_clean":"2024-02-10T08:35:23.872814+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:35:23.872814+0000","last_undegraded":"2024-02-10T08:35:23.872814+0000","last_fullsized":"2024-02-10T08:35:23.872814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.553914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.553914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709320+0000","last_change":"2024-02-10T08:34:50.747486+0000","last_active":"2024-02-10T08:35:24.709320+0000","last_peered":"2024-02-10T08:35:24.709320+0000","last_clean":"2024-02-10T08:35:24.709320+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:35:24.709320+0000","last_undegraded":"2024-02-10T08:35:24.709320+0000","last_fullsized":"2024-02-10T08:35:24.709320+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:50.747424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.747424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:30.241761+0000","last_change":"2024-02-10T08:35:30.241737+0000","last_active":"2024-02-10T08:35:30.241761+0000","last_peered":"2024-02-10T08:35:30.241761+0000","last_clean":"2024-02-10T08:35:30.241761+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:35:30.241761+0000","last_undegraded":"2024-02-10T08:35:30.241761+0000","last_fullsized":"2024-02-10T08:35:30.241761+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.241661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.241661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873463+0000","last_change":"2024-02-10T08:34:41.701563+0000","last_active":"2024-02-10T08:35:23.873463+0000","last_peered":"2024-02-10T08:35:23.873463+0000","last_clean":"2024-02-10T08:35:23.873463+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:35:23.873463+0000","last_undegraded":"2024-02-10T08:35:23.873463+0000","last_fullsized":"2024-02-10T08:35:23.873463+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:34.592074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:34.592074+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.873721+0000","last_change":"2024-02-10T08:34:56.475601+0000","last_active":"2024-02-10T08:35:23.873721+0000","last_peered":"2024-02-10T08:35:23.873721+0000","last_clean":"2024-02-10T08:35:23.873721+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:35:23.873721+0000","last_undegraded":"2024-02-10T08:35:23.873721+0000","last_fullsized":"2024-02-10T08:35:23.873721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.475529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.475529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.709707+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:35:24.709707+0000","last_peered":"2024-02-10T08:35:24.709707+0000","last_clean":"2024-02-10T08:35:24.709707+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:35:24.709707+0000","last_undegraded":"2024-02-10T08:35:24.709707+0000","last_fullsized":"2024-02-10T08:35:24.709707+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-10T08:35:23.875229+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:35:23.875229+0000","last_peered":"2024-02-10T08:35:23.875229+0000","last_clean":"2024-02-10T08:35:23.875229+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:35:23.875229+0000","last_undegraded":"2024-02-10T08:35:23.875229+0000","last_fullsized":"2024-02-10T08:35:23.875229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":71,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-10T08:35:24.911177+0000","last_change":"2024-02-10T08:34:42.752203+0000","last_active":"2024-02-10T08:35:24.911177+0000","last_peered":"2024-02-10T08:35:24.911177+0000","last_clean":"2024-02-10T08:35:24.911177+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:35:24.911177+0000","last_undegraded":"2024-02-10T08:35:24.911177+0000","last_fullsized":"2024-02-10T08:35:24.911177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:42.752097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:42.752097+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":121,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299421,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":288311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.034,"15min":1.034},"last":0.64200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.054,"15min":1.054},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.496,"15min":0.499},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.63300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42599999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.503},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52500000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.747,"15min":0.747},"last":0.56299999999999994}]}]},{"osd":6,"up_from":44,"seq":188978561059,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297692,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447460,"statfs":{"total":95995035648,"available":95690199040,"internally_reserved":0,"allocated":610304,"data_stored":288219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.621},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.498,"15min":0.498},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.502},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.623,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.501},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.375,"15min":0.372},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56000000000000005,"5min":0.623,"15min":0.623},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.39600000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.497}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":289631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.495,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.47499999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36799999999999999,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.44800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.309,"5min":0.36199999999999999,"15min":0.36499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":0.624,"15min":0.624},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41899999999999998,"15min":0.42899999999999999},"min":{"1min":0.27100000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.53000000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.371,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.51400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35499999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":1.294,"15min":1.294},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.61899999999999999},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81499999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.67200000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73299999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.79700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297704,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447448,"statfs":{"total":95995035648,"available":95690186752,"internally_reserved":0,"allocated":622592,"data_stored":287335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.30299999999999999},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.308},"min":{"1min":0.27300000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.44}]},{"osd":2,"last update":"Sat Feb 10 08:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48199999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46400000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.503,"5min":0.624,"15min":0.624},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.501,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.621},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75900000000000001,"5min":0.878,"15min":0.878},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78000000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40100000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67900000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76600000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:35:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84299999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79500000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":294428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.193,"5min":0.187,"15min":0.187},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.312,"5min":0.35399999999999998,"15min":0.36099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48099999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35699999999999998,"15min":0.36099999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.41699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.38200000000000001,"15min":0.38300000000000001},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.45700000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.248}]},{"osd":4,"last update":"Sat Feb 10 08:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81899999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.42599999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77800000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.372}]},{"osd":6,"last update":"Sat Feb 10 08:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.505}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.621,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.52900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182450,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":288507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.35599999999999998,"15min":0.36899999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.254}]},{"osd":1,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39600000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41399999999999998,"15min":0.437},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.437,"15min":0.46000000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58199999999999996,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35599999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57999999999999996,"15min":0.60799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66700000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.378},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55800000000000005,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70299999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93400000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.47399999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.41299999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":295091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.998,"5min":1,"15min":1},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.499},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.749,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.437}]},{"osd":3,"last update":"Sat Feb 10 08:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.378},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65800000000000003,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.40300000000000002,"15min":0.38700000000000001},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.441,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53100000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789992,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297708,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":626688,"data_stored":287653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51600000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68000000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.496}]},{"osd":1,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68600000000000005,"5min":0.749,"15min":0.749},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.32700000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66100000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70499999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.42699999999999999,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34499999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":1.081,"15min":1.081},"last":0.44}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48499999999999999,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:35:34.059 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T08:35:34.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump 2024-02-10T08:35:34.330 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:33.253714+0000 mgr.y (mgr.14553) 95 : audit [DBG] from='client.34691 v1:172.21.15.107:0/709723698' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:33.674291+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:33.677372+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:34.244558+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.2 deep-scrub starts 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:34.249420+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.2 deep-scrub ok 2024-02-10T08:35:34.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:33.253714+0000 mgr.y (mgr.14553) 95 : audit [DBG] from='client.34691 v1:172.21.15.107:0/709723698' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:34.900 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:33.674291+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:35:34.900 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:33.677372+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:35:34.900 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:34.244558+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.2 deep-scrub starts 2024-02-10T08:35:34.900 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:34.249420+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.2 deep-scrub ok 2024-02-10T08:35:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:33.253714+0000 mgr.y (mgr.14553) 95 : audit [DBG] from='client.34691 v1:172.21.15.107:0/709723698' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:35:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:33.674291+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:35:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:33.677372+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:35:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:34.244558+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.2 deep-scrub starts 2024-02-10T08:35:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:34.249420+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.2 deep-scrub ok 2024-02-10T08:35:35.635 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:35] "GET /metrics HTTP/1.1" 200 172136 "" "Prometheus/2.33.4" 2024-02-10T08:35:35.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:34.017506+0000 mgr.y (mgr.14553) 96 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:35.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:34.651646+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:35:35.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:34.652544+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:35:35.905 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:34.017506+0000 mgr.y (mgr.14553) 96 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:35.906 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:34.651646+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:35:35.906 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:34.652544+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:35:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:34.017506+0000 mgr.y (mgr.14553) 96 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:35:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:34.651646+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:35:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:34.652544+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:35:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.136+0000 7efebe5d7700 1 -- 172.21.15.107:0/1184703032 >> v1:172.21.15.107:6790/0 conn(0x7efeb80fc910 legacy=0x7efeb80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.136+0000 7efebe5d7700 1 -- 172.21.15.107:0/1184703032 shutdown_connections 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.136+0000 7efebe5d7700 1 -- 172.21.15.107:0/1184703032 >> 172.21.15.107:0/1184703032 conn(0x7efeb80f2510 msgr2=0x7efeb80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.136+0000 7efebe5d7700 1 -- 172.21.15.107:0/1184703032 shutdown_connections 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.136+0000 7efebe5d7700 1 -- 172.21.15.107:0/1184703032 wait complete. 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 librados: starting msgr at 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 librados: starting objecter 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 Processor -- start 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 -- start start 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 librados: setting wanted keys 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 librados: calling monclient init 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb81173b0 con 0x7efeb80ff710 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb8118590 con 0x7efeb80fc910 2024-02-10T08:35:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebe5d7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb8119790 con 0x7efeb8103250 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebcb74700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7efeb8103250 0x7efeb8115a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39382/0 (socket says 172.21.15.107:39382) 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efebcb74700 1 -- 172.21.15.107:0/4006588006 learned_addr learned my addr 172.21.15.107:0/4006588006 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.137+0000 7efeb77fe700 1 --1- 172.21.15.107:0/4006588006 >> v1:172.21.15.107:6789/0 conn(0x7efeb80ff710 0x7efeb80fa250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43014/0 (socket says 172.21.15.107:43014) 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter ms_handle_connect 0x7efeb8103250 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter ms_handle_connect 0x7efeb80ff710 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266856108 0 0) 0x7efeb81173b0 con 0x7efeb80ff710 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea00033f0 con 0x7efeb80ff710 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078940551 0 0) 0x7efeb8119790 con 0x7efeb8103250 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb81173b0 con 0x7efeb8103250 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter ms_handle_connect 0x7efeb80fc910 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108179132 0 0) 0x7efeb8118590 con 0x7efeb80fc910 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8119790 con 0x7efeb80fc910 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694675998 0 0) 0x7efeb81173b0 con 0x7efeb8103250 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb8118590 con 0x7efeb8103250 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efeb0001d50 con 0x7efeb8103250 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583194966 0 0) 0x7efea00033f0 con 0x7efeb80ff710 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb81173b0 con 0x7efeb80ff710 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efeac001840 con 0x7efeb80ff710 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3680078245 0 0) 0x7efeb8119790 con 0x7efeb80fc910 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efea00033f0 con 0x7efeb80fc910 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efea8002050 con 0x7efeb80fc910 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180862652 0 0) 0x7efeb8118590 con 0x7efeb8103250 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 >> v1:172.21.15.195:6789/0 conn(0x7efeb80fc910 legacy=0x7efeb80f9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 >> v1:172.21.15.107:6789/0 conn(0x7efeb80ff710 legacy=0x7efeb80fa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb811a990 con 0x7efeb8103250 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb81175a0 con 0x7efeb8103250 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efebe5d7700 10 client.34697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb8117a10 con 0x7efeb8103250 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efebe5d7700 1 librados: init done 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7efeb00026c0 con 0x7efeb8103250 2024-02-10T08:35:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.138+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efeb0005d40 con 0x7efeb8103250 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea4004f20 con 0x7efeb8103250 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7efeb0002220 con 0x7efeb8103250 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(84..84 src has 30..84) v4 ==== 6172+0+0 (unknown 2330842233 0 0) 0x7efeb0078710 con 0x7efeb8103250 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 10 client.34697.objecter ms_dispatch 0x7efeb8001550 osd_map(84..84 src has 30..84) v4 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 3 client.34697.objecter handle_osd_map got epochs [84,84] > 0 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 3 client.34697.objecter handle_osd_map decoding full epoch 84 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 20 client.34697.objecter dump_active .. 0 homeless 2024-02-10T08:35:36.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.140+0000 7efeb57fa700 10 client.34697.objecter ms_handle_connect 0x7efea0063cf0 2024-02-10T08:35:36.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.144+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7efeb004c2c0 con 0x7efeb8103250 2024-02-10T08:35:36.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.309+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7efea40059e0 con 0x7efeb8103250 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.309+0000 7efeb57fa700 1 -- 172.21.15.107:0/4006588006 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v84) v1 ==== 56+0+4034 (unknown 3828741802 0 442643142) 0x7efeb004a6d0 con 0x7efeb8103250 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 84 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-02-10T08:27:44.316161+0000 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-02-10T08:35:30.716386+0000 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 18 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-02-10T08:35:36.310 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client luminous 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release pacific 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 57 flags hashpspool stripe_width 0 application rgw 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 58 flags hashpspool stripe_width 0 application rbd 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 8 2024-02-10T08:35:36.311 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 8 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6801/3321596972 v1:172.21.15.107:6802/3321596972 exists,up e30a7307-1dfd-4dfa-b4c4-89b83eb24167 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 13 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6805/133516126 v1:172.21.15.107:6806/133516126 exists,up c5fe7280-6de7-4c89-a9f2-22eebe4c1c80 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 18 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6809/3575881728 v1:172.21.15.107:6810/3575881728 exists,up 3ae7131b-69d5-4cbd-b1ca-29556fac7fbf 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 25 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6813/2871825691 v1:172.21.15.107:6814/2871825691 exists,up 1ffb2633-18bb-4048-90eb-7130e0cc7d43 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6800/2499671285 v1:172.21.15.195:6801/2499671285 exists,up 1b7529a6-19fc-4435-be7f-28876750838a 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 37 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6804/4178321486 v1:172.21.15.195:6805/4178321486 exists,up bdc1b91c-94ce-4287-8019-535b326da3e5 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 44 up_thru 59 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6808/1417479017 v1:172.21.15.195:6809/1417479017 exists,up 217fdf91-33a7-40e2-a948-51b05ed63031 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 52 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6812/1934176928 v1:172.21.15.195:6813/1934176928 exists,up f0cd42da-4080-4e6a-843b-d73e10ce0948 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.18 [7,2] 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.4 [1,5] 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.b [1,2] 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.18 [1,2] 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1675475667 expires 2024-02-11T08:33:39.974787+0000 2024-02-10T08:35:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/682167559 expires 2024-02-11T08:33:39.974787+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/3360068681 expires 2024-02-11T08:33:39.974787+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2916003802 expires 2024-02-11T08:28:29.992672+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1214732392 expires 2024-02-11T08:28:07.029499+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/810931708 expires 2024-02-11T08:28:29.992672+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1610921635 expires 2024-02-11T08:28:07.029499+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/942960899 expires 2024-02-11T08:33:39.974787+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/2081774553 expires 2024-02-11T08:28:29.992672+0000 2024-02-10T08:35:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/4244374259 expires 2024-02-11T08:28:07.029499+0000 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 10 librados: watch_flush enter 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 10 librados: watch_flush exit 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 20 client.34697.objecter shutdown clearing up homeless session... 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 10 client.34697.objecter successfully canceled tick 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 >> v1:172.21.15.107:6800/2634842009 conn(0x7efea0063cf0 legacy=0x7efea00661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 >> v1:172.21.15.107:6790/0 conn(0x7efeb8103250 legacy=0x7efeb8115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 shutdown_connections 2024-02-10T08:35:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 >> 172.21.15.107:0/4006588006 conn(0x7efeb80f2510 msgr2=0x7efeb80612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 shutdown_connections 2024-02-10T08:35:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 -- 172.21.15.107:0/4006588006 wait complete. 2024-02-10T08:35:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:36.311+0000 7efebe5d7700 1 librados: shutdown 2024-02-10T08:35:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:36.310091+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.107:0/4006588006' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:35:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:36.510369+0000 osd.5 (osd.5) 41 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:35:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:36.513311+0000 osd.5 (osd.5) 42 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:35:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:36.310091+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.107:0/4006588006' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:35:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:36.510369+0000 osd.5 (osd.5) 41 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:35:36.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:36.513311+0000 osd.5 (osd.5) 42 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:35:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:36.310091+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.107:0/4006588006' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:35:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:36.510369+0000 osd.5 (osd.5) 41 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:35:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:36.513311+0000 osd.5 (osd.5) 42 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:35:37.043 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6801/3321596972 v1:172.21.15.107:6802/3321596972 exists,up e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'osd.1 up in weight 1 up_from 13 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6805/133516126 v1:172.21.15.107:6806/133516126 exists,up c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'osd.2 up in weight 1 up_from 18 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6809/3575881728 v1:172.21.15.107:6810/3575881728 exists,up 3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'osd.3 up in weight 1 up_from 25 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6813/2871825691 v1:172.21.15.107:6814/2871825691 exists,up 1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6800/2499671285 v1:172.21.15.195:6801/2499671285 exists,up 1b7529a6-19fc-4435-be7f-28876750838a', 'osd.5 up in weight 1 up_from 37 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6804/4178321486 v1:172.21.15.195:6805/4178321486 exists,up bdc1b91c-94ce-4287-8019-535b326da3e5', 'osd.6 up in weight 1 up_from 44 up_thru 59 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6808/1417479017 v1:172.21.15.195:6809/1417479017 exists,up 217fdf91-33a7-40e2-a948-51b05ed63031', 'osd.7 up in weight 1 up_from 52 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.195:6812/1934176928 v1:172.21.15.195:6813/1934176928 exists,up f0cd42da-4080-4e6a-843b-d73e10ce0948'] 2024-02-10T08:35:37.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon mon.a config show 2024-02-10T08:35:37.310 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:37.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:36.018044+0000 mgr.y (mgr.14553) 97 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:37.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:36.645861+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:35:37.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:36.647710+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:35:37.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:36.018044+0000 mgr.y (mgr.14553) 97 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:37.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:36.645861+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:35:37.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:36.647710+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:35:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:36.018044+0000 mgr.y (mgr.14553) 97 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:36.645861+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:35:38.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:36.647710+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:35:38.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:37.561561+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:35:38.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:37.562541+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:35:38.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:37.561561+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:35:38.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:37.562541+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:35:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:37.561561+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:35:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:37.562541+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:35:39.105 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:35:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mon.a", 2024-02-10T08:35:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster": "ceph", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket_mode": "", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_client_required": "cephx, none", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_cluster_required": "cephx", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_debug": "true", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_required": "cephx", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_supported": "", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio": "true", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-10T08:35:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_poll_ms": "250", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_reap_max": "16", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_async_discard": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_block_size": "4096", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_inflight_ios": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_enable_discard": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry": "3", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash": "0", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_hipri": "false", 2024-02-10T08:35:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_type": "", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_alloc_size": "1048576", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_allocator": "hybrid", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_buffered_io": "true", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_for_zeros": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_compact_log_sync": "false", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-10T08:35:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_log_runway": "4194304", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_prefetch": "1048576", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_flush_size": "524288", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_log_runway": "1048576", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery": "false", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_sync_write": "false", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocator": "hybrid", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-10T08:35:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_create": "true", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_create": "false", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_path": "", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_size": "0", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_path": "", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_preallocate_file": "false", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_size": "107374182400", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_create": "false", 2024-02-10T08:35:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_path": "", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_size": "100663296", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs": "true", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune": "true", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size": "0", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-10T08:35:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_type": "2q", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_clone_cow": "true", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_mode": "none", 2024-02-10T08:35:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_csum_type": "crc32c", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_freelist": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-10T08:35:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_small_allocations": "0", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_read": "true", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_write": "false", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-10T08:35:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-10T08:35:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-10T08:35:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_ignore_data_csum": "false", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_op_age": "5.000000", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_alloc_size": "0", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size": "0", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-10T08:35:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_deferred_txc": "32", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size": "0", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_nid_prealloc": "1024", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_retry_disk_reads": "3", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cf": "true", 2024-02-10T08:35:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_mem": "512", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-10T08:35:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_tracing": "false", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-10T08:35:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-10T08:35:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_signatures": "false", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_version": "2", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_signatures": "false", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_version": "2", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_signatures": "false", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_version": "2", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_sign_messages": "true", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "chdir": "", 2024-02-10T08:35:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "client_acl_type": "", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_asio_thread_count": "2", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_mid": "0.750000", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_size": "16384", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_caps_release_delay": "5", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_check_pool_perm": "true", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_force_sync_read": "false", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_getattr_caps": "false", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_inject_tick_delay": "0", 2024-02-10T08:35:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_remount": "false", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_dirsize_rbytes": "true", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_force_lazyio": "false", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_fs": "", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_release_failure": "false", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_inline_size": "4096", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mds_namespace": "", 2024-02-10T08:35:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "client_metadata": "", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_gid": "-1", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_timeout": "300", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_uid": "-1", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mountpoint": "/", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_notify_timeout": "10", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc": "true", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty": "104857600", 2024-02-10T08:35:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_objects": "1000", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_size": "209715200", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_target_dirty": "8388608", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_permissions": "true", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota": "true", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota_df": "true", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_bytes": "0", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_periods": "4", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_min": "131072", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_reconnect_stale": "false", 2024-02-10T08:35:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_shutdown_timeout": "30", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_snapdir": ".snap", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_tick_interval": "1", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_trace": "", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_try_dentry_invalidate": "false", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_faked_inos": "false", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_random_mds": "false", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog": "false", 2024-02-10T08:35:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_port": "12201", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_monitors": "default=true", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog": "false", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_level": "info", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "-", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network": "", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network_interface": "", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_isal": "false", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_level": "5", 2024-02-10T08:35:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_winsize": "-15", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zstd_level": "1", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location": "", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook": "", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook_timeout": "10", 2024-02-10T08:35:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "daemonize": "false", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_allow_any_pool_priority": "false", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok": "1/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok_assert_abort": "false", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asserts_on_shutdown": "true", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_auth": "1/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bdev": "1/3", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluefs": "1/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluestore": "1/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_buffer": "0/1", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephfs_mirror": "0/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephsqlite": "0/5", 2024-02-10T08:35:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_civetweb": "1/10", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_client": "0/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_compressor": "1/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_context": "0/1", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crush": "1/1", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crypto": "1/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_deliberately_leak_memory": "false", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_disable_randomized_ping": "false", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_dpdk": "1/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_eventtrace": "1/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filer": "0/1", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filestore": "1/3", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_finisher": "1/1", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_fuse": "1/5", 2024-02-10T08:35:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeat_testing_span": "0", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeatmap": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_javaclient": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journal": "1/3", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journaler": "0/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_kstore": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_leveldb": "4/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_lockdep": "0/1", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_balancer": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_locker": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log": "1/5", 2024-02-10T08:35:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log_expire": "1/5", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_migrator": "1/5", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_memdb": "4/5", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgr": "2/5", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgrc": "20/20", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mon": "20/20", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_monc": "0/10", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ms": "1/1", 2024-02-10T08:35:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_none": "0/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objclass": "0/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objectcacher": "0/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objecter": "0/1", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_optracker": "0/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_osd": "1/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_paxos": "20/20", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_perfcounter": "1/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_prioritycache": "1/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rados": "0/5", 2024-02-10T08:35:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd": "0/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_mirror": "0/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_pwl": "0/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_replay": "0/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_refs": "0/0", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_reserver": "1/1", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw": "1/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_sync": "1/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rocksdb": "4/5", 2024-02-10T08:35:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_striper": "0/1", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_test": "0/5", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_throttle": "1/1", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_timer": "0/1", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_tp": "0/5", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "device_failure_prediction_mode": "none", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_graylog": "false", 2024-02-10T08:35:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_stderr": "true", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_syslog": "false", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event_tracing": "false", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "fake_statfs_for_testing": "0", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "fatal_signal_handlers": "true", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_purge_ops": "10", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_truncate_ops": "128", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_apply_finisher_threads": "1", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_blackhole": "false", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_clone_range": "true", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_snap": "true", 2024-02-10T08:35:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_caller_concurrency": "10", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_collect_device_partition_information": "true", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_commit_timeout": "600.000000", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_inject_read_err": "false", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_omap_check": "false", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_verify_split": "false", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_dump_file": "", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fadvise": "true", 2024-02-10T08:35:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fail_eio": "true", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_shards": "16", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_size": "128", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap": "false", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap_threshold": "4096", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_inject_stall": "0", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_parallel": "false", 2024-02-10T08:35:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_trailing": "false", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_writeahead": "false", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_kill_at": "0", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-10T08:35:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs": "0", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size": "0", 2024-02-10T08:35:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_merge_threshold": "-10", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_odsync_write": "false", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend": "rocksdb", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend_path": "", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-10T08:35:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_timeout": "60", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_threads": "2", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_punch_hole": "false", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-10T08:35:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_ops": "50", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_seek_data_hole": "false", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc": "false", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_splice": "false", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_multiple": "2", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_rand_factor": "20", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_update_to": "1000", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-10T08:35:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_enable": "true", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_xfs_extsize": "false", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_zfs_snap": "false", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "fio_dir": "/tmp/fio", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_allow_other": "true", 2024-02-10T08:35:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_atomic_o_trunc": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_big_writes": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_debug": "false", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_default_permissions": "false", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_disable_pagecache": "false", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_max_write": "0", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_multithreaded": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_require_active_mds": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_set_user_groups": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_move": "true", 2024-02-10T08:35:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_read": "true", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_write": "true", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_use_invalidate_cb": "true", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_target_name": "ceph", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_file": "", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_inject_failure": "0", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_interval": "5", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "host": "", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-10T08:35:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-10T08:35:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "inject_early_sigterm": "false", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_aio": "true", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_align_min_size": "65536", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_align": "true", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_size": "4096", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_dio": "true", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_discard": "false", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_force_aio": "false", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_ignore_corruption": "false", 2024-02-10T08:35:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_bytes": "10485760", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_entries": "100", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_replay_from": "0", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_write_header_frequency": "0", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_zero_on_create": "false", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prefetch_periods": "10", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prezero_periods": "5", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_write_head_interval": "15", 2024-02-10T08:35:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "keyfile": "", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_backend": "rocksdb", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_default_stripe_size": "65536", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount": "false", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_bytes": "67108864", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_ops": "512", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_nid_prealloc": "1024", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_onode_map_size": "1024", 2024-02-10T08:35:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_submit_transaction": "false", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_transaction": "false", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_block_size": "65536", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_bloom_size": "0", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_cache_size": "536870912", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_compact_on_mount": "false", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_compression": "false", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_log": "", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-10T08:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_max_open_files": "0", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_paranoid": "false", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "librados_thread_count": "2", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep": "false", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep_force_backtrace": "false", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_coarse_timestamps": "true", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_flush_on_exit": "false", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_host": "127.0.0.1", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_port": "12201", 2024-02-10T08:35:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_new": "1000", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_recent": "10000", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stderr_prefix": "debug ", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stop_at_utilization": "0.970000", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": "true", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_graylog": "false", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_stderr": "false", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_syslog": "false", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "max_rotating_auth_attempts": "10", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_action_on_write_error": "1", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_alternate_name_max": "8192", 2024-02-10T08:35:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_asio_thread_count": "2", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_export_pin": "true", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_dirs": "true", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_interval": "5", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_interval": "10", 2024-02-10T08:35:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max": "-1", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max_until": "-1", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_merge_size": "50", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_midchunk": "0.300000", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_start": "0.200000", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_minchunk": "0.001000", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_mode": "0", 2024-02-10T08:35:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_max": "1.200000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_min": "0.800000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_bits": "3", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_size": "10000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_target_decay": "10.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_grace": "15.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_interval": "4.000000", 2024-02-10T08:35:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_mid": "0.700000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_release_free_interval": "10", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_reservation": "0.050000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_interval": "1", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_threshold": "262144", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_prealloc_inos": "1000", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-10T08:35:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_connect_bootstrapping": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_damage_table_max_entries": "10000", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_auth_pins": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_frag": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_scatterstat": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_subtrees": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_decay_halflife": "5.000000", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_default_dir_hash": "2", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_defer_session_stale": "true", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_deny_all_reconnect": "false", 2024-02-10T08:35:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_keys_per_op": "16384", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_commit_size": "10", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_entries": "0", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_on_map": "false", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-10T08:35:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_early_reply": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enable_op_tracker": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enforce_unique_name": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-10T08:35:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_summarize_threshold": "10", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_health_dummy": "false", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_migrator_session_race": "false", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_join_fs": "", 2024-02-10T08:35:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_journal_format": "1", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_create_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_export_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_import_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_expire_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_replay_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_link_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_mdstable_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_openc_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_rename_at": "0", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_events_per_segment": "1024", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_events": "-1", 2024-02-10T08:35:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_segments": "128", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_pause": "false", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_segment_size": "0", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_warn_factor": "2.000000", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_caps_per_client": "1048576", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_flushes": "100000", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_requests": "100000", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_export_size": "20971520", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_file_recover": "32", 2024-02-10T08:35:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_files": "64", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops": "8192", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_snaps_per_dir": "100", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_metrics_update_interval": "2", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_per_client": "100", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_working_set": "10000", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_numa_node": "-1", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_complaint_time": "30.000000", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_duration": "600", 2024-02-10T08:35:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_size": "20", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_log_threshold": "5", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_grace": "15", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_interval": "5", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_caps": "30000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_threshold": "262144", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_interval": "1.000000", 2024-02-10T08:35:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_gid": "0", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_uid": "0", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-10T08:35:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_shutdown_check": "0", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_skip_ino": "0", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_max_uid": "4294967294", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_min_uid": "0", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_rstat": "false", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_standby_replay_damaged": "false", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_exports": "0", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_fragments": "0", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_tick_interval": "5.000000", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_valgrind_exit": "false", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_backtrace": "1", 2024-02-10T08:35:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_scatter": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_sessions": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool_debug": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_device_bytes": "1073741824", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_set": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_size": "65536", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_bytes": "134217728", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_messages": "512", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-10T08:35:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_disabled_modules": "", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_max_pg_num_change": "128", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_bytes": "134217728", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_messages": "128", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_bytes": "134217728", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_messages": "128", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_bytes": "536870912", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_messages": "8192", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_standby_modules": "true", 2024-02-10T08:35:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_period": "5", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_threshold": "5", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_tick_period": "2", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_delete": "true", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_size_one": "false", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_auth_validate_all_caps": "true", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-10T08:35:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_bytes": "104857600", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_directed_command_retry": "2", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_parallel": "3", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_log_interval": "1.000000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_interval": "10.000000", 2024-02-10T08:35:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file_level": "debug", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_file": "true", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-10T08:35:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_bootstrap": "false", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_start": "false", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_trim": "true", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cpu_threads": "4", 2024-02-10T08:35:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_crush_min_required_version": "hammer", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_daemon_bytes": "419430400", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_crit": "5", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_warn": "5", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_size_warn": "16106127360", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_json": "false", 2024-02-10T08:35:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_transactions": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_extra_checks": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_octopus": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_pacific": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-10T08:35:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_down_mkfs_grace": "60", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_default_strategy": "1", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_timeout": "5.000000", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_divisor": "2", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_enable_op_tracker": "true", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_fake_pool_delete": "false", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_force_quorum_join": "false", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_globalid_prealloc": "10000", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_detail_to_clog": "true", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_log_update_period": "5", 2024-02-10T08:35:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_max_detail": "50", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog": "true", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_interval": "600", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host": "172.21.15.107:6789/0", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host_override": "", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_initial_members": "", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-10T08:35:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease": "5.000000", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max_summary": "50", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_epochs": "500", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_osd": "10000", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pg_per_osd": "10000", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pool_pg_num": "65536", 2024-02-10T08:35:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_force_trim_to": "0", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_skip_sanity": "false", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_autotune": "true", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_target": "2147483648", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_beacon_grace": "30", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_digest_period": "5", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_inactive_grace": "60", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-10T08:35:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_min_osdmap_epochs": "50", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_complaint_time": "30", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_duration": "600", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_size": "20", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_size": "20", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_log_threshold": "5", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-10T08:35:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_in": "false", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size": "500", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_interval": "600", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-10T08:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_force_trim_to": "0", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_down_reporters": "2", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-10T08:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_report_timeout": "900", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-10T08:35:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_stuck_threshold": "60", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_probe_timeout": "2.000000", 2024-02-10T08:35:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_change": "0.050000", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_osds": "4", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_interval": "86400", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_max_keys": "100", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_timeout": "300", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_session_timeout": "300", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_smart_report_timeout": "5", 2024-02-10T08:35:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stat_smooth_intervals": "6", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_min_size": "2", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_size": "4", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_debug": "false", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_provider_kill_at": "0", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_requester_kill_at": "0", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_timeout": "60.000000", 2024-02-10T08:35:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_target_pg_per_osd": "100", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_tick_interval": "5", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_interval": "300.000000", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_older_version_delay": "604800", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-10T08:35:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_misplaced": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_older_version": "true", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_app": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-10T08:35:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "monmap": "", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_max_op_threads": "5", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_op_threads": "3", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_cm": "false", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_device_name": "", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_dscp": "96", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_local_gid": "", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_port_num": "1", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-10T08:35:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_sl": "3", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_support_srq": "true", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_type": "ib", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_reap_threshold": "5", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_before_connect": "false", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv4": "true", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv6": "false", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr1": "true", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr2": "false", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_max": "7300", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_min": "6800", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-10T08:35:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_count": "3", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_delay": "5", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_client": "false", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mds": "false", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mgr": "false", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mon": "false", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_osd": "false", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_client_mode": "crc secure", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_mode": "crc secure", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_type": "", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_idle_timeout": "900", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_ready_timeout": "10", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_data": "true", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_header": "true", 2024-02-10T08:35:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bad_msg": "false", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bug": "true", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_old_message": "true", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_skipped_message": "false", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_coremask": "0xF", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hugepages": "", 2024-02-10T08:35:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_lro": "true", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_memory_channel": "4", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_pmd": "", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_port_id": "0", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_on_send": "false", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_initial_backoff": "0.200000", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_max": "1.000000", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_type": "", 2024-02-10T08:35:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_socket_failures": "0", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_learn_addr_from_peer": "true", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_accept_failures": "4", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_backoff": "15.000000", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_client_mode": "secure crc", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_service_mode": "secure crc", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_min_cost": "65536", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_public_type": "", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_service_mode": "crc secure", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_listen_backlog": "512", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_nodelay": "true", 2024-02-10T08:35:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_rcvbuf": "0", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_type": "async+posix", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "no_config_file": "false", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_completion_locks_per_session": "32", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_ops": "1024", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_tick_interval": "5.000000", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_timeout": "10.000000", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_blackhole": "false", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "openssl_engine_opts": "", 2024-02-10T08:35:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_delay_time": "5.000000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_hist_halflife": "1000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_low_ops": "2", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_ops": "4", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_slop": "0.020000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_asio_thread_count": "2", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_async_recovery_min_cost": "100", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-10T08:35:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_max": "512", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_min": "64", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_degraded": "false", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_peering": "false", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_unfound": "true", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_beacon_report_interval": "300", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_duration": "30", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_max_block_size": "67108864", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blkin_trace_all": "false", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_for_log_corruption": "false", 2024-02-10T08:35:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_update_on_start": "true", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_cap": "256", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_size_cap": "524288000", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_op_priority": "63", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_watch_timeout": "30", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_max_records": "256", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_thread_timeout": "600", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_compact_on_start": "false", 2024-02-10T08:35:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_on_start": "true", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_weight_set": "true", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_feed_pullee": "-1", 2024-02-10T08:35:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_misdirected_ops": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_acting_change": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_purge_strays": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_op_order": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_shutdown": "false", 2024-02-10T08:35:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_snaps": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_keys": "1024", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_stride": "524288", 2024-02-10T08:35:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_notify_timeout": "30", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep": "0.000000", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_discard_disconnected_ops": "true", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_enable_op_tracker": "true", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-10T08:35:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_info": "true", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown": "true", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_function_tracing": "false", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_grace": "20", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_interval": "6", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_peers": "10", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_size": "2000", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_stale": "600", 2024-02-10T08:35:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_max_size": "100000", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_min_size": "1000", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_size": "5120", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_kill_backfill_at": "0", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_cache_size": "50", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_dedup": "true", 2024-02-10T08:35:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max": "40", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_share_max_epochs": "40", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_name_len": "100", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_size": "0", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_backfills": "1", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_count": "5", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_period": "600", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_name_len": "2048", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_namespace_len": "256", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_size": "134217728", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_blocked_by": "16", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_log_entries": "10000", 2024-02-10T08:35:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pgls": "1024", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_cost": "8388608", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_objects": "10", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_scrubs": "1", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_trimming_pgs": "2", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_op_reply_len": "64", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_size": "90", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-10T08:35:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-10T08:35:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_base": "805306368", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_min": "134217728", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target": "4294967296", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_autotune": "false", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-10T08:35:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_pg_log_entries": "250", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_recovery_priority": "0", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_interval": "5", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_cache_shards": "32", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_op_tracker_shard": "32", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_auto_affinity": "true", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_node": "-1", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_prefer_iface": "true", 2024-02-10T08:35:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objecter_finishers": "1", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore": "bluestore", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_fuse": "false", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_tracing": "false", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_complaint_time": "30.000000", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_duration": "600", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_size": "20", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_size": "20", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_log_threshold": "5", 2024-02-10T08:35:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards": "0", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_hdd": "5", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_ssd": "8", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_min_cost": "65536", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue": "wpq", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue_cut_off": "high", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-10T08:35:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_timeout": "15", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_open_classes_on_start": "true", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_os_flags": "0", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_peering_op_priority": "255", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_cost": "1048576", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_priority": "5", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_max": "10000", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_min": "100", 2024-02-10T08:35:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-10T08:35:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flags": "0", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_min_size": "0", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_num": "32", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pgp_num": "0", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-10T08:35:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_size": "3", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_type": "replicated", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_push_per_object_cost": "1000", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap": "true", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_cost": "20971520", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active": "0", 2024-02-10T08:35:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_single_start": "1", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_priority": "3", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_priority": "5", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep": "0.000000", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-10T08:35:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_repair_during_recovery": "false", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_requested_scrub_priority": "120", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair": "false", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_hour": "0", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_week_day": "0", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_max": "25", 2024-02-10T08:35:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_min": "5", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_cost": "52428800", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_during_recovery": "false", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_hour": "0", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_week_day": "0", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_invalid_stats": "true", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_preemptions": "5", 2024-02-10T08:35:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_priority": "5", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_sleep": "0.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_data_digest": "false", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_smart_report_timeout": "5", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_cost": "1048576", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_priority": "5", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-10T08:35:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_transaction_size": "30", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tracing": "false", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_use_stale_snap": "false", 2024-02-10T08:35:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "osdc_blkin_trace_all": "false", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_kill_at": "0", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_max_join_drift": "10", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min": "500", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min_wait": "0.050000", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_propose_interval": "1.000000", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max": "500", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_min": "10", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_stash_full_interval": "25", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_max": "500", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_min": "250", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "perf": "true", 2024-02-10T08:35:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "pid_file": "", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "-", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrv": "", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "public_bind_addr": "-", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network": "", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network_interface": "", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_enabled": "false", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_mon_op_timeout": "0", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_osd_op_timeout": "0", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_tracing": "false", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_atime_update_interval": "60", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-10T08:35:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_parent_reads": "false", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_snap_reads": "false", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blkin_trace_all": "false", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache": "true", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_policy": "writearound", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_size": "33554432", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-10T08:35:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_clone_copy_on_read": "false", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_compression_hint": "none", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_concurrent_management_ops": "10", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_clone_format": "auto", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_data_pool": "", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_features": "61", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_format": "2", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_map_options": "", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_order": "22", 2024-02-10T08:35:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_pool": "rbd", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_count": "0", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_unit": "0", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_enable_alloc_hint": "true", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler": "simple", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-10T08:35:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_order": "24", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_pool": "", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_splay_width": "4", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_parent_reads": "false", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_snap_reads": "false", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-10T08:35:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-10T08:35:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mtime_update_interval": "60", 2024-02-10T08:35:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_non_blocking_aio": "true", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_thread_timeout": "60", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_threads": "1", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_parent_cache_enabled": "false", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_plugins": "", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst": "0", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_limit": "0", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst": "0", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-10T08:35:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_limit": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-10T08:35:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_read_from_replica_policy": "default", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_skip_partial_discard": "true", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_tracing": "false", 2024-02-10T08:35:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_names": "true", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_pool": "true", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_base_url": "", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_log_level": "", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_acl_grants_max_num": "100", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_admin_entry": "admin", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_barbican_url": "", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-10T08:35:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_enabled": "true", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_expiry_interval": "900", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_lru_size": "10000", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_content_length_compat": "false", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress": "true", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cors_rules_max_num": "100", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cross_domain_policy": "", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_addr": "", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-10T08:35:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_password": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_username": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_require_ssl": "true", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_addr": "", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_auth": "token", 2024-02-10T08:35:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_namespace": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_prefix": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_token_file": "", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_buffersize": "524288", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_time": "300", 2024-02-10T08:35:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_changes_size": "1000", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_num_shards": "128", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_window": "30", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-10T08:35:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-10T08:35:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_name": "", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_s3website_name": "", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding": "true", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_gc_threads": "true", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_lc_threads": "true", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_ops_log": "false", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_quota_threads": "true", 2024-02-10T08:35:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_static_website": "false", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_usage_log": "false", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enforce_swift_acls": "true", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_exit_timeout_secs": "120", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_expose_bucket": "false", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_extended_http_attrs": "", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontends": "beast port=7480", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred": "50", 2024-02-10T08:35:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_objs": "32", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_period": "3600", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_host": "", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-10T08:35:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_init_timeout": "300", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_domain": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password_path": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_project": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_tenant": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token_path": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_user": "", 2024-02-10T08:35:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_api_version": "2", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_domain": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_password": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_project": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_user": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_url": "", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_debug_interval": "-1", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_lock_max_time": "90", 2024-02-10T08:35:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_objs": "32", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_rules": "1000", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_worker": "3", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_thread_delay": "0", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_dnattr": "uid", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchfilter": "", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-10T08:35:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_http_headers": "", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name_utc": "false", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_name_len": "0", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_size": "0", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_chunk_size": "4194304", 2024-02-10T08:35:39.245 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_listing_results": "1000", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_notify_retries": "10", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_param_size": "1048576", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_size": "5368709120", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_slo_entries": "1000", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_log_max_shards": "64", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mp_lock_max_time": "600", 2024-02-10T08:35:39.246 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_max_gc": "300", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-10T08:35:39.247 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_async_rados_threads": "32", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_control_oids": "8", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_rados_handles": "1", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_numa_node": "-1", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_chunk_size": "100", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_gc_interval": "600", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-10T08:35:39.248 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_timeout": "600", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_tracing": "false", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_token": "", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_url": "", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_verify_ssl": "true", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_file_path": "", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_rados": "true", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_socket_path": "", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-10T08:35:39.249 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval": "2.000000", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_port": "", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_continue": "true", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_tracing": "false", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm": "", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region": "", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-10T08:35:39.250 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_request_uri": "", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_batch_size": "64", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_max_aio": "128", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_num_logs": "16", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_thread_interval": "600", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_resolve_cname": "false", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_run_sync_thread": "true", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-10T08:35:39.251 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_scheduler_type": "throttler", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_script_uri": "", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_service_provider_name": "", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_socket_path": "", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_id": "", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_secret": "", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_entry": "sts", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_key": "sts", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-10T08:35:39.252 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_min_session_duration": "900", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_token_introspection_url": "", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_account_in_url": "false", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_entry": "auth", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_url": "", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_custom_header": "", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_need_stats": "true", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_tenant_name": "", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_token_expiration": "86400", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url": "", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url_prefix": "swift", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-10T08:35:39.253 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_lease_period": "120", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_thread_pool_size": "512", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_comment": "", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_createby": "", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_encoding": "", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_flag": "false", 2024-02-10T08:35:39.254 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_origin": "", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_tracker": "", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_trust_forwarded_https": "false", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_shards": "32", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_user_shards": "1", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_use_opa_authz": "false", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_max_buckets": "1000", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-10T08:35:39.255 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_unique_email": "true", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_verify_ssl": "true", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone": "", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup": "", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_block_size": "4096", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-10T08:35:39.256 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_size": "536870912", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_index_type": "binary_search", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_partition_filters": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_perf": "false", 2024-02-10T08:35:39.257 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "run_dir": "/var/run/ceph", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "setgroup": "ceph", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser": "ceph", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser_match_path": "", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "thp": "false", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_default_timeout": "60", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout: "throttler_perf_counter": "true" 2024-02-10T08:35:39.258 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:35:39.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-10T08:35:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.018462+0000 mgr.y (mgr.14553) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.318157+0000 osd.1 (osd.1) 53 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:35:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.321688+0000 osd.1 (osd.1) 54 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:35:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.546189+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.547089+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.777808+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.c scrub starts 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:38.779429+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.c scrub ok 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.018462+0000 mgr.y (mgr.14553) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.318157+0000 osd.1 (osd.1) 53 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.321688+0000 osd.1 (osd.1) 54 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.546189+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.547089+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.777808+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.c scrub starts 2024-02-10T08:35:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:38.779429+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.c scrub ok 2024-02-10T08:35:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.018462+0000 mgr.y (mgr.14553) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.318157+0000 osd.1 (osd.1) 53 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.321688+0000 osd.1 (osd.1) 54 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.546189+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.547089+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.777808+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.c scrub starts 2024-02-10T08:35:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:38.779429+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.c scrub ok 2024-02-10T08:35:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:40.003483+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:40.003570+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:40.003483+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:40.003570+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:40.003483+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:40.003570+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:41.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.742+0000 7f5349e8f700 1 -- 172.21.15.107:0/4020162653 >> v1:172.21.15.107:6790/0 conn(0x7f53440fda50 legacy=0x7f53440fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.742+0000 7f5349e8f700 1 -- 172.21.15.107:0/4020162653 shutdown_connections 2024-02-10T08:35:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.742+0000 7f5349e8f700 1 -- 172.21.15.107:0/4020162653 >> 172.21.15.107:0/4020162653 conn(0x7f53440f3640 msgr2=0x7f53440f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.742+0000 7f5349e8f700 1 -- 172.21.15.107:0/4020162653 shutdown_connections 2024-02-10T08:35:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.742+0000 7f5349e8f700 1 -- 172.21.15.107:0/4020162653 wait complete. 2024-02-10T08:35:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 librados: starting msgr at 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 librados: starting objecter 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 Processor -- start 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 -- start start 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 librados: setting wanted keys 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 librados: calling monclient init 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53440fcf70 con 0x7f5344104330 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53440fd140 con 0x7f53440fda50 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5349e8f700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53440fd310 con 0x7f53441007f0 2024-02-10T08:35:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5342ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f53441007f0 0x7f5344114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55106/0 (socket says 172.21.15.107:55106) 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.743+0000 7f5342ffd700 1 -- 172.21.15.107:0/2973934179 learned_addr learned my addr 172.21.15.107:0/2973934179 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter ms_handle_connect 0x7f53441007f0 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter ms_handle_connect 0x7f5344104330 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter ms_handle_connect 0x7f53440fda50 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026089643 0 0) 0x7f53440fcf70 con 0x7f5344104330 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5330003030 con 0x7f5344104330 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312081192 0 0) 0x7f53440fd310 con 0x7f53441007f0 2024-02-10T08:35:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53440fcf70 con 0x7f53441007f0 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327823931 0 0) 0x7f53440fd140 con 0x7f53440fda50 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53440fd310 con 0x7f53440fda50 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926738249 0 0) 0x7f5330003030 con 0x7f5344104330 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53440fd140 con 0x7f5344104330 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f533c001b00 con 0x7f5344104330 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725900342 0 0) 0x7f53440fcf70 con 0x7f53441007f0 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5330003030 con 0x7f53441007f0 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304356368 0 0) 0x7f53440fd310 con 0x7f53440fda50 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53440fcf70 con 0x7f53440fda50 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53340019f0 con 0x7f53441007f0 2024-02-10T08:35:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f532c0015d0 con 0x7f53440fda50 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3332927650 0 0) 0x7f53440fd140 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6790/0 conn(0x7f53441007f0 legacy=0x7f5344114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.195:6789/0 conn(0x7f53440fda50 legacy=0x7f53440fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534411cc00 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5344119c30 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5349e8f700 10 client.14784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f534411a120 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5349e8f700 1 librados: init done 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f533c0021c0 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.744+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f533c0035f0 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.745+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5324004f40 con 0x7f5344104330 2024-02-10T08:35:41.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f533c001d20 con 0x7f5344104330 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (unknown 2648245713 0 0) 0x7f533c0760c0 con 0x7f5344104330 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 10 client.14784.objecter ms_dispatch 0x7f5344001550 osd_map(85..85 src has 30..85) v4 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 3 client.14784.objecter handle_osd_map got epochs [85,85] > 0 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 3 client.14784.objecter handle_osd_map decoding full epoch 85 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 20 client.14784.objecter dump_active .. 0 homeless 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1111 (unknown 409899127 0 2452703222) 0x7f533c0784f0 con 0x7f5344104330 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f5330063d30 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.746+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f5324004d40 con 0x7f5324002730 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f5324002730 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629577821 0 0) 0x7f5324004d40 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53440fd140 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3675138874 0 0) 0x7f53440fd140 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5324004d40 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53340020a0 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2776110988 0 0) 0x7f5324004d40 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f532400a390 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7f53340022c0 con 0x7f5324002730 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6789/0 conn(0x7f5324002730 legacy=0x7f53240054b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.747+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6789/0 conn(0x7f5324002730 legacy=0x7f53240054b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f532400bcd0 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361723226 0 0) 0x7f532400bcd0 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324004d40 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3009384093 0 0) 0x7f5324004d40 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532400bcd0 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5334005160 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 441525609 0 0) 0x7f532400bcd0 con 0x7f532400a770 2024-02-10T08:35:41.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.754+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f532400bf40 con 0x7f532400a770 2024-02-10T08:35:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.0 v1:172.21.15.107:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7f5334005500 con 0x7f532400a770 2024-02-10T08:35:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6789/0 conn(0x7f532400a770 legacy=0x7f5324008be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6789/0 conn(0x7f532400a770 legacy=0x7f5324008be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.756 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a: {} 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f532400a070 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549786040 0 0) 0x7f532400a070 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532400bcd0 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2931596965 0 0) 0x7f532400bcd0 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532400a070 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.756+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5334005a00 con 0x7f5324009b80 2024-02-10T08:35:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2464419345 0 0) 0x7f532400a070 con 0x7f5324009b80 2024-02-10T08:35:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53240128a0 con 0x7f5324009b80 2024-02-10T08:35:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7f5334007ff0 con 0x7f5324009b80 2024-02-10T08:35:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.195:6789/0 conn(0x7f5324009b80 legacy=0x7f5324001d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.195:6789/0 conn(0x7f5324009b80 legacy=0x7f5324001d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.757+0000 7f5343fff700 1 -- 172.21.15.107:0/2973934179 reap_dead start 2024-02-10T08:35:41.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f5324013040 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972827407 0 0) 0x7f5324013040 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532400a070 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4203968526 0 0) 0x7f532400a070 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5324013040 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5334006410 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 297242008 0 0) 0x7f5324013040 con 0x7f5324012bc0 2024-02-10T08:35:41.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.764+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.195:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f53240142c0 con 0x7f5324012bc0 2024-02-10T08:35:41.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.1 v1:172.21.15.195:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7f5334007020 con 0x7f5324012bc0 2024-02-10T08:35:41.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.195:6789/0 conn(0x7f5324012bc0 legacy=0x7f5324011140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.767 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b: {} 2024-02-10T08:35:41.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.195:6789/0 conn(0x7f5324012bc0 legacy=0x7f5324011140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f5324013ed0 con 0x7f53240132c0 2024-02-10T08:35:41.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f53240132c0 2024-02-10T08:35:41.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.766+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.772+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499813945 0 0) 0x7f5324013ed0 con 0x7f53240132c0 2024-02-10T08:35:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.772+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324013040 con 0x7f53240132c0 2024-02-10T08:35:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.772+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1502682882 0 0) 0x7f5324013040 con 0x7f53240132c0 2024-02-10T08:35:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.772+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5324013ed0 con 0x7f53240132c0 2024-02-10T08:35:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.772+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5334007700 con 0x7f53240132c0 2024-02-10T08:35:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.778+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 151098238 0 0) 0x7f5324013ed0 con 0x7f53240132c0 2024-02-10T08:35:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.778+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f532401ac20 con 0x7f53240132c0 2024-02-10T08:35:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.779+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7f5334007700 con 0x7f53240132c0 2024-02-10T08:35:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.779+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6790/0 conn(0x7f53240132c0 legacy=0x7f53240136d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.779+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6790/0 conn(0x7f53240132c0 legacy=0x7f53240136d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.785+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f532401c380 con 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.785+0000 7f5340ff9700 10 client.14784.objecter ms_handle_connect 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.785+0000 7f5340ff9700 10 client.14784.objecter resend_mon_ops 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645461183 0 0) 0x7f532401c380 con 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324013ed0 con 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2857825373 0 0) 0x7f5324013ed0 con 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f532401c380 con 0x7f532401af40 2024-02-10T08:35:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5334007d50 con 0x7f532401af40 2024-02-10T08:35:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 155936904 0 0) 0x7f532401c380 con 0x7f532401af40 2024-02-10T08:35:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.786+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 --> v1:172.21.15.107:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f532401c580 con 0x7f532401af40 2024-02-10T08:35:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 <== mon.2 v1:172.21.15.107:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7f5334006630 con 0x7f532401af40 2024-02-10T08:35:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6790/0 conn(0x7f532401af40 legacy=0x7f53240194c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5340ff9700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6790/0 conn(0x7f532401af40 legacy=0x7f53240194c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:35:41.788 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c: {} 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 10 librados: watch_flush enter 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 10 librados: watch_flush exit 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 20 client.14784.objecter shutdown clearing up homeless session... 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 10 client.14784.objecter successfully canceled tick 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5330063d30 legacy=0x7f53300661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 >> v1:172.21.15.107:6789/0 conn(0x7f5344104330 legacy=0x7f53441184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5343fff700 1 -- 172.21.15.107:0/2973934179 reap_dead start 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 shutdown_connections 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.787+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 >> 172.21.15.107:0/2973934179 conn(0x7f53440f3640 msgr2=0x7f53441077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.788+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 shutdown_connections 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.788+0000 7f5349e8f700 1 -- 172.21.15.107:0/2973934179 wait complete. 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:41.788+0000 7f5349e8f700 1 librados: shutdown 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-10T08:35:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:40.018929+0000 mgr.y (mgr.14553) 99 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:40.771999+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:41.091247+0000 mgr.y (mgr.14553) 100 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:41.108758+0000 osd.1 (osd.1) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:41.108805+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:42.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:41.746873+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? v1:172.21.15.107:0/2973934179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:35:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:40.018929+0000 mgr.y (mgr.14553) 99 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:40.771999+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-10T08:35:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:41.091247+0000 mgr.y (mgr.14553) 100 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:41.108758+0000 osd.1 (osd.1) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:41.108805+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:41.746873+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? v1:172.21.15.107:0/2973934179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:40.018929+0000 mgr.y (mgr.14553) 99 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:40.771999+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:41.091247+0000 mgr.y (mgr.14553) 100 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:41.108758+0000 osd.1 (osd.1) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:41.108805+0000 osd.1 (osd.1) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:41.746873+0000 mon.a (mon.0) 722 : audit [DBG] from='client.? v1:172.21.15.107:0/2973934179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:35:42.567 INFO:teuthology.run_tasks:Running task workunit... 2024-02-10T08:35:42.590 INFO:tasks.workunit:Pulling workunits from ref 57bd6abdec7bb457ae7999d9c96682e9ac678e27 2024-02-10T08:35:42.590 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-10T08:35:42.591 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-10T08:35:42.591 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-10T08:35:42.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:35:42.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:35:42.592 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-10T08:35:42.592 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:35:42.592 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:35:42.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-10T08:35:42.593 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-10T08:35:42.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:42.594 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-10T08:35:42.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:35:42.594 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-10T08:35:42.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:35:42.595 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-10T08:35:42.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:35:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:35:42.618 INFO:teuthology.orchestra.run.smithi107.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-10T08:35:42.619 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-10T08:35:42.639 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:42.662 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:42.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:42.683 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-10T08:35:42.683 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-10T08:35:42.744 INFO:tasks.workunit:timeout=3h 2024-02-10T08:35:42.744 INFO:tasks.workunit:cleanup=True 2024-02-10T08:35:42.745 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 57bd6abdec7bb457ae7999d9c96682e9ac678e27 2024-02-10T08:35:42.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:42.809 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:41.772845+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:41.794220+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:41.795070+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:42.019392+0000 mgr.y (mgr.14553) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:42.470127+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 deep-scrub starts 2024-02-10T08:35:42.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:42.473017+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 deep-scrub ok 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:41.772845+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:41.794220+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:41.795070+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:42.019392+0000 mgr.y (mgr.14553) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:42.470127+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 deep-scrub starts 2024-02-10T08:35:42.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:42.473017+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 deep-scrub ok 2024-02-10T08:35:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:42.851 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:42.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:42.882 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:35:42.950 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:42.950 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:42.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:42.991 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:41.772845+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:41.794220+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:41.795070+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:42.019392+0000 mgr.y (mgr.14553) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:42.470127+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.2 deep-scrub starts 2024-02-10T08:35:43.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:42.473017+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.2 deep-scrub ok 2024-02-10T08:35:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:43.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:44.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:42.323170+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:35:44.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:42.326622+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:35:44.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:44.124 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:42.323170+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:35:44.124 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:42.326622+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:35:44.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:42.323170+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:35:44.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:42.326622+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:35:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:44.148 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:44.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:44.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:44.293 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:44.442 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:44.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:44.490 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:35:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-10T08:35:44.490 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:35:44.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:44.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:44.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:35:44.725 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:44.847 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:35:44.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:43.332564+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:35:44.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:43.333379+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:35:44.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:44.019744+0000 mgr.y (mgr.14553) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:44.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:43.332564+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:35:44.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:43.333379+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:35:44.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:44.019744+0000 mgr.y (mgr.14553) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:44.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:35:44.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:44.979 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:45.003 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:45.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:45.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:45.108 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:43.332564+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:35:45.108 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:43.333379+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:35:45.108 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:44.019744+0000 mgr.y (mgr.14553) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:35:45.136 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:45.157 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:45.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:45.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:45.285 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:45.304 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:45.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:45.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:45.435 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:45.777 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:45] "GET /metrics HTTP/1.1" 200 172154 "" "Prometheus/2.33.4" 2024-02-10T08:35:46.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:45.303108+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.107:0/1236313980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:45.303396+0000 mon.a (mon.0) 724 : audit [INF] from='client.34715 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:45.303108+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.107:0/1236313980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:45.303396+0000 mon.a (mon.0) 724 : audit [INF] from='client.34715 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.189 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:45.303108+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.107:0/1236313980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:45.303396+0000 mon.a (mon.0) 724 : audit [INF] from='client.34715 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:35:46.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:46.325 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:46.346 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:46.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:35:46.478 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:46.498 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:35:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:35:46.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:46.632 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:46.653 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:46.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.108+0000 7f24e8896700 1 -- 172.21.15.107:0/806859979 >> v1:172.21.15.107:6790/0 conn(0x7f24e40f9570 legacy=0x7f24e4102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:46.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.108+0000 7f24e8896700 1 -- 172.21.15.107:0/806859979 shutdown_connections 2024-02-10T08:35:46.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.108+0000 7f24e8896700 1 -- 172.21.15.107:0/806859979 >> 172.21.15.107:0/806859979 conn(0x7f24e40f2510 msgr2=0x7f24e40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:46.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 -- 172.21.15.107:0/806859979 shutdown_connections 2024-02-10T08:35:46.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 -- 172.21.15.107:0/806859979 wait complete. 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 librados: starting msgr at 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 librados: starting objecter 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 Processor -- start 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 -- start start 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 librados: setting wanted keys 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.109+0000 7f24e8896700 1 librados: calling monclient init 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f24e41199a0 con 0x7f24e4104350 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f24e411ab40 con 0x7f24e40f9570 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f24e411bce0 con 0x7f24e40f89c0 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e259c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f24e40f89c0 0x7f24e410ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55132/0 (socket says 172.21.15.107:55132) 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e2d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24e4104350 0x7f24e4118280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59172/0 (socket says 172.21.15.107:59172) 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e259c700 1 -- 172.21.15.107:0/1236313980 learned_addr learned my addr 172.21.15.107:0/1236313980 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter ms_handle_connect 0x7f24e40f89c0 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:46.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter ms_handle_connect 0x7f24e4104350 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter ms_handle_connect 0x7f24e40f9570 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200988779 0 0) 0x7f24e411bce0 con 0x7f24e40f89c0 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc0033f0 con 0x7f24e40f89c0 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161993582 0 0) 0x7f24e41199a0 con 0x7f24e4104350 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e411bce0 con 0x7f24e4104350 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105124296 0 0) 0x7f24e411ab40 con 0x7f24e40f9570 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e41199a0 con 0x7f24e40f9570 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2100998981 0 0) 0x7f24cc0033f0 con 0x7f24e40f89c0 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24e411ab40 con 0x7f24e40f89c0 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f24d40029d0 con 0x7f24e40f89c0 2024-02-10T08:35:46.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338656542 0 0) 0x7f24e41199a0 con 0x7f24e40f9570 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24cc0033f0 con 0x7f24e40f9570 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f24d8001ec0 con 0x7f24e40f9570 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3198980197 0 0) 0x7f24e411bce0 con 0x7f24e4104350 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24e41199a0 con 0x7f24e4104350 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f24dc002b20 con 0x7f24e4104350 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3221957760 0 0) 0x7f24e411ab40 con 0x7f24e40f89c0 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 >> v1:172.21.15.195:6789/0 conn(0x7f24e40f9570 legacy=0x7f24e4114b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 >> v1:172.21.15.107:6789/0 conn(0x7f24e4104350 legacy=0x7f24e4118280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e411ce80 con 0x7f24e40f89c0 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e411bed0 con 0x7f24e40f89c0 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 10 client.34715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e411c340 con 0x7f24e40f89c0 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.110+0000 7f24e8896700 1 librados: init done 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.111+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f24d4002cd0 con 0x7f24e40f89c0 2024-02-10T08:35:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.111+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f24d4005bc0 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.112+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c4004f20 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.112+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f24d4018a50 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (unknown 2760144681 0 0) 0x7f24d4078600 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 10 client.34715.objecter ms_dispatch 0x7f24e4001550 osd_map(86..86 src has 30..86) v4 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 3 client.34715.objecter handle_osd_map got epochs [86,86] > 0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 3 client.34715.objecter handle_osd_map decoding full epoch 86 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 20 client.34715.objecter dump_active .. 0 homeless 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.113+0000 7f24d37fe700 10 client.34715.objecter ms_handle_connect 0x7f24cc063db0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.117+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f24d404c1b0 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.302+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f24c4005750 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.796+0000 7f24d37fe700 1 -- 172.21.15.107:0/1236313980 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v87) v1 ==== 87+0+0 (unknown 1708980508 0 0) 0x7f24d404a5c0 con 0x7f24e40f89c0 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 10 librados: watch_flush enter 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 10 librados: watch_flush exit 2024-02-10T08:35:46.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 20 client.34715.objecter shutdown clearing up homeless session... 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 10 client.34715.objecter successfully canceled tick 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 >> v1:172.21.15.107:6800/2634842009 conn(0x7f24cc063db0 legacy=0x7f24cc066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 >> v1:172.21.15.107:6790/0 conn(0x7f24e40f89c0 legacy=0x7f24e410ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 shutdown_connections 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 >> 172.21.15.107:0/1236313980 conn(0x7f24e40f2510 msgr2=0x7f24e4103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 shutdown_connections 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 -- 172.21.15.107:0/1236313980 wait complete. 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:45.798+0000 7f24e8896700 1 librados: shutdown 2024-02-10T08:35:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:35:46.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:46.788 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:46.808 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:46.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.541410+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.543107+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.759978+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 deep-scrub starts 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.760866+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 deep-scrub ok 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.788911+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:45.796029+0000 mon.a (mon.0) 726 : audit [INF] from='client.34715 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:45.796120+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-10T08:35:46.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:46.020247+0000 mgr.y (mgr.14553) 103 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:35:46.940 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:46.959 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:46.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:47.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:47.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:47.108 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:47.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.541410+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:35:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.543107+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:35:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.759978+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 deep-scrub starts 2024-02-10T08:35:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.760866+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 deep-scrub ok 2024-02-10T08:35:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.788911+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:45.796029+0000 mon.a (mon.0) 726 : audit [INF] from='client.34715 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:45.796120+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:46.020247+0000 mgr.y (mgr.14553) 103 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.541410+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.543107+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.759978+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 deep-scrub starts 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.760866+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 deep-scrub ok 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.788911+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:45.796029+0000 mon.a (mon.0) 726 : audit [INF] from='client.34715 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:45.796120+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-10T08:35:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:46.020247+0000 mgr.y (mgr.14553) 103 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:35:47.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:47.238 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:47.256 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:47.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:47.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:35:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:48.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:35:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:48.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:35:48.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:48.472 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:48.494 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:48.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:48.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:35:48.628 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:48.645 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:35:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:48.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:35:48.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:48.786 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:48.808 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:48.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:48.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:48.945 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:48.965 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:48.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:49.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:49.097 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:49.116 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:49.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:49.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:49.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:48.020538+0000 mgr.y (mgr.14553) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:35:49.253 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:48.020538+0000 mgr.y (mgr.14553) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:35:49.256 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:49.282 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:49.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:49.288 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:35:49.288 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:35:49.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.148+0000 7ff2fb034700 1 -- 172.21.15.107:0/2776963724 >> v1:172.21.15.107:6790/0 conn(0x7ff2f4104330 legacy=0x7ff2f41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.148+0000 7ff2fb034700 1 -- 172.21.15.107:0/2776963724 shutdown_connections 2024-02-10T08:35:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.148+0000 7ff2fb034700 1 -- 172.21.15.107:0/2776963724 >> 172.21.15.107:0/2776963724 conn(0x7ff2f40f3640 msgr2=0x7ff2f40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.148+0000 7ff2fb034700 1 -- 172.21.15.107:0/2776963724 shutdown_connections 2024-02-10T08:35:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.148+0000 7ff2fb034700 1 -- 172.21.15.107:0/2776963724 wait complete. 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 librados: starting msgr at 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 librados: starting objecter 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 Processor -- start 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 -- start start 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 librados: setting wanted keys 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 librados: calling monclient init 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f40fcfd0 con 0x7ff2f41007f0 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f40fd1a0 con 0x7ff2f40fda50 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2fb034700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f40fd390 con 0x7ff2f4104330 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f95d1700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff2f4104330 0x7ff2f41184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55144/0 (socket says 172.21.15.107:55144) 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f95d1700 1 -- 172.21.15.107:0/1902087863 learned_addr learned my addr 172.21.15.107:0/1902087863 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f3fff700 1 --1- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6789/0 conn(0x7ff2f41007f0 0x7ff2f4114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59186/0 (socket says 172.21.15.107:59186) 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f1ffb700 10 client.?.objecter ms_handle_connect 0x7ff2f4104330 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f1ffb700 10 client.?.objecter ms_handle_connect 0x7ff2f41007f0 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.149+0000 7ff2f1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 10 client.?.objecter ms_handle_connect 0x7ff2f40fda50 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776563555 0 0) 0x7ff2f40fd1a0 con 0x7ff2f40fda50 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4003030 con 0x7ff2f40fda50 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667421482 0 0) 0x7ff2f40fcfd0 con 0x7ff2f41007f0 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f40fd1a0 con 0x7ff2f41007f0 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307106694 0 0) 0x7ff2f40fd390 con 0x7ff2f4104330 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f40fcfd0 con 0x7ff2f4104330 2024-02-10T08:35:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213472600 0 0) 0x7ff2e4003030 con 0x7ff2f40fda50 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2f40fd390 con 0x7ff2f40fda50 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3794668155 0 0) 0x7ff2f40fcfd0 con 0x7ff2f4104330 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e4003030 con 0x7ff2f4104330 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e80019f0 con 0x7ff2f40fda50 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226925026 0 0) 0x7ff2f40fd1a0 con 0x7ff2f41007f0 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2f40fcfd0 con 0x7ff2f41007f0 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2ec002fa0 con 0x7ff2f4104330 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e0001840 con 0x7ff2f41007f0 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4054947151 0 0) 0x7ff2f40fd390 con 0x7ff2f40fda50 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6790/0 conn(0x7ff2f4104330 legacy=0x7ff2f41184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6789/0 conn(0x7ff2f41007f0 legacy=0x7ff2f4114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f411cc00 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4119c30 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff2e8002320 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e80033d0 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2fb034700 10 client.24700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f411a0a0 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.150+0000 7ff2fb034700 1 librados: init done 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.151+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff2f405f700 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff2e8003530 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(87..87 src has 30..87) v4 ==== 6172+0+0 (unknown 3014698625 0 0) 0x7ff2e8075db0 con 0x7ff2f40fda50 2024-02-10T08:35:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 osd_map(87..87 src has 30..87) v4 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 3 client.24700.objecter handle_osd_map got epochs [87,87] > 0 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 3 client.24700.objecter handle_osd_map decoding full epoch 87 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 20 client.24700.objecter dump_active .. 0 homeless 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v87) v1 ==== 54+0+16 (unknown 1393783987 0 2420000192) 0x7ff2e8078150 con 0x7ff2f40fda50 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2e4063ce0 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 10 client.24700.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 1 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 1 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 0 1 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 4 2024-02-10T08:35:49.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 10 client.24700.objecter _send_command 1 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.152+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f411ca70 con 0x7ff2f4059990 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f4059990 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f411ca70 con 0x7ff2f4059990 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 1 = system:0 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 4 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.153+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 0 1 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 10 client.24700.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 2 2024-02-10T08:35:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 2 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f40475e0 osd=0 3 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 0 2 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 4 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 10 client.24700.objecter _send_command 2 2024-02-10T08:35:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.169+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f410a070 con 0x7ff2f4059990 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f410a070 con 0x7ff2f4059990 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 4 2024-02-10T08:35:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 0 2 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 10 client.24700.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 1 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 4 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 10 client.24700.objecter _send_command 3 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.171+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4120a30 con 0x7ff2f4059ee0 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.172+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f4059ee0 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.173+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f4120a30 con 0x7ff2f4059ee0 2024-02-10T08:35:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.173+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:35:49.410 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:35:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.173+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 3 = system:0 2024-02-10T08:35:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.173+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 4 2024-02-10T08:35:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.173+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 1 3 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 10 client.24700.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 4 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 4 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4059d70 osd=1 3 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 1 4 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 4 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 10 client.24700.objecter _send_command 4 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.188+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f4059400 con 0x7ff2f4059ee0 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f4059400 con 0x7ff2f4059ee0 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 4 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 1 4 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 10 client.24700.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 5 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 5 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 2 5 2024-02-10T08:35:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 4 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 10 client.24700.objecter _send_command 5 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.190+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4125450 con 0x7ff2f4105480 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f4105480 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f4125450 con 0x7ff2f4105480 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 5 = system:0 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 4 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.191+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 2 5 2024-02-10T08:35:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 10 client.24700.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 6 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 6 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4059810 osd=2 3 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 2 6 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 4 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 10 client.24700.objecter _send_command 6 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.218+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f41049e0 con 0x7ff2f4105480 2024-02-10T08:35:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f41049e0 con 0x7ff2f4105480 2024-02-10T08:35:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 4 2024-02-10T08:35:49.415 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 2 6 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 10 client.24700.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 7 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 7 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 3 7 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 4 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 10 client.24700.objecter _send_command 7 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.221+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4129e70 con 0x7ff2f4121880 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f4121880 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f4129e70 con 0x7ff2f4121880 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:35:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 7 = system:0 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 4 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.222+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 3 7 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 10 client.24700.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 8 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 8 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4104df0 osd=3 3 2024-02-10T08:35:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 3 8 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 4 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 10 client.24700.objecter _send_command 8 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.238+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f4120de0 con 0x7ff2f4121880 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f4120de0 con 0x7ff2f4121880 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 4 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 3 8 2024-02-10T08:35:49.419 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 10 client.24700.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 9 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 9 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 4 9 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 4 2024-02-10T08:35:49.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 10 client.24700.objecter _send_command 9 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.240+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4126450 con 0x7ff2f4120f80 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.241+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f4120f80 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.242+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f4126450 con 0x7ff2f4120f80 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.242+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.242+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 9 = system:0 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.242+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 4 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.242+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 4 9 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 10 client.24700.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 10 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 10 2024-02-10T08:35:49.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4120de0 osd=4 3 2024-02-10T08:35:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 4 10 2024-02-10T08:35:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 4 2024-02-10T08:35:49.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 10 client.24700.objecter _send_command 10 2024-02-10T08:35:49.422 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.262+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f4126450 con 0x7ff2f4120f80 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f4126450 con 0x7ff2f4120f80 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 4 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 4 10 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 10 client.24700.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 11 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 11 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 5 11 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 4 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 10 client.24700.objecter _send_command 11 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.264+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f41329f0 con 0x7ff2f412eda0 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.265+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f412eda0 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.266+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f41329f0 con 0x7ff2f412eda0 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.266+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.266+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 11 = system:0 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.266+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 4 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.266+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 5 11 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 10 client.24700.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 12 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 12 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4126450 osd=5 3 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 5 12 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 4 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 10 client.24700.objecter _send_command 12 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.282+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f41329f0 con 0x7ff2f412eda0 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f41329f0 con 0x7ff2f412eda0 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 4 2024-02-10T08:35:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 5 12 2024-02-10T08:35:49.426 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:48.020538+0000 mgr.y (mgr.14553) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:35:49.426 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:35:49.426 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 10 client.24700.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 13 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 13 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 6 13 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 4 2024-02-10T08:35:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 10 client.24700.objecter _send_command 13 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4133c10 con 0x7ff2f41346b0 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.285+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f41346b0 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.286+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f4133c10 con 0x7ff2f41346b0 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.286+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.286+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 13 = system:0 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.286+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 4 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.286+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 6 13 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 10 client.24700.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 14 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 14 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f41329f0 osd=6 3 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 6 14 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 4 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 10 client.24700.objecter _send_command 14 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.315+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f4133c10 con 0x7ff2f41346b0 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f4133c10 con 0x7ff2f41346b0 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 4 2024-02-10T08:35:49.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 6 14 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 10 client.24700.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 15 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 15 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 7 15 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 4 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 10 client.24700.objecter _send_command 15 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.317+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f413b3c0 con 0x7ff2f41378b0 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 10 client.24700.objecter ms_handle_connect 0x7ff2f41378b0 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff2f413b3c0 con 0x7ff2f41378b0 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 15 = system:0 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 4 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.318+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 7 15 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 10 client.24700.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign -1 16 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 20 client.24700.objecter _get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_remove -1 16 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 20 client.24700.objecter get_session s=0x7ff2f4133c10 osd=7 3 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 15 client.24700.objecter _session_command_op_assign 7 16 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 4 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 10 client.24700.objecter _send_command 16 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.345+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff2f413b3c0 con 0x7ff2f41378b0 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f1ffb700 1 -- 172.21.15.107:0/1902087863 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff2f413b3c0 con 0x7ff2f41378b0 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f1ffb700 10 client.24700.objecter ms_dispatch 0x7ff2f4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f1ffb700 10 client.24700.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.432 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f1ffb700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 4 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f1ffb700 15 client.24700.objecter _session_command_op_remove 7 16 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 librados: watch_flush enter 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 librados: watch_flush exit 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.0 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6801/3321596972 conn(0x7ff2f4059990 legacy=0x7ff2f411c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f40475e0 osd=0 1 2024-02-10T08:35:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.1 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6805/133516126 conn(0x7ff2f4059ee0 legacy=0x7ff2f411f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059d70 osd=1 1 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.2 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6809/3575881728 conn(0x7ff2f4105480 legacy=0x7ff2f4123d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4059810 osd=2 1 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.3 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6813/2871825691 conn(0x7ff2f4121880 legacy=0x7ff2f4128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4104df0 osd=3 1 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.4 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f95d1700 1 -- 172.21.15.107:0/1902087863 reap_dead start 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.195:6800/2499671285 conn(0x7ff2f4120f80 legacy=0x7ff2f412d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.347+0000 7ff2f95d1700 1 -- 172.21.15.107:0/1902087863 reap_dead start 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4120de0 osd=4 1 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.5 2024-02-10T08:35:49.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.195:6804/4178321486 conn(0x7ff2f412eda0 legacy=0x7ff2f4131190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4126450 osd=5 1 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.6 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.195:6808/1417479017 conn(0x7ff2f41346b0 legacy=0x7ff2f4134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f41329f0 osd=6 1 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 10 client.24700.objecter close_session for osd.7 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.195:6812/1934176928 conn(0x7ff2f41378b0 legacy=0x7ff2f4139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 20 client.24700.objecter put_session s=0x7ff2f4133c10 osd=7 1 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 20 client.24700.objecter shutdown clearing up homeless session... 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 10 client.24700.objecter successfully canceled tick 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff2e4063ce0 legacy=0x7ff2e40661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> v1:172.21.15.195:6789/0 conn(0x7ff2f40fda50 legacy=0x7ff2f40fc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2f95d1700 1 -- 172.21.15.107:0/1902087863 reap_dead start 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2f95d1700 1 -- 172.21.15.107:0/1902087863 reap_dead start 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 shutdown_connections 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 >> 172.21.15.107:0/1902087863 conn(0x7ff2f40f3640 msgr2=0x7ff2f41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 shutdown_connections 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 -- 172.21.15.107:0/1902087863 wait complete. 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:49.348+0000 7ff2fb034700 1 librados: shutdown 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:35:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:49.437 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:49.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:49.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:49.565 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:49.584 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:49.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:49.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:49.716 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:49.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:35:50.128 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:35:50.129 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:35:50.129 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-10T08:35:50.129 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-02-10T08:35:50.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-10T08:35:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:50.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:50.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:50.302 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:50.323 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:50.325 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:35:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:35:50.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:49.152783+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.107:0/1902087863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:35:50.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:49.258869+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:50.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:49.258919+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:50.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:50.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:49.152783+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.107:0/1902087863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:49.258869+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:49.258919+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:49.152783+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.107:0/1902087863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:49.258869+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:35:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:49.258919+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:35:50.455 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:50.474 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:50.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:50.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:50.604 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:50.622 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:50.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:50.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:35:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:35:50.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:35:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:50.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:35:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:51.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.103 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:50.020864+0000 mgr.y (mgr.14553) 105 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-10T08:35:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:50.083456+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-10T08:35:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:50.020864+0000 mgr.y (mgr.14553) 105 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-10T08:35:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:50.083456+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-10T08:35:51.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:51.234 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.253 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:51.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:51.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:50.020864+0000 mgr.y (mgr.14553) 105 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-10T08:35:51.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:50.083456+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-10T08:35:51.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:35:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:35:51.384 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.403 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:51.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:51.534 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:51.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:51.685 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.704 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:51.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:51.834 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:51.853 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:51.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:51.894 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:51.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:35:51.983 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:52.135 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:35:52.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:35:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:52.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:35:52.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:52.267 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:51.102027+0000 mgr.y (mgr.14553) 106 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:52.268 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:51.102027+0000 mgr.y (mgr.14553) 106 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:52.270 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:51.102027+0000 mgr.y (mgr.14553) 106 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:35:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:52.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:52.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:35:52.849 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:52.869 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:35:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:52.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:35:52.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:53.014 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:53.035 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:53.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:53.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:35:53.167 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:53.195 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:35:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:53.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:35:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:52.021397+0000 mgr.y (mgr.14553) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:53.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:53.330 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:53.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:52.021397+0000 mgr.y (mgr.14553) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:53.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:52.021397+0000 mgr.y (mgr.14553) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:53.351 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:53.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:53.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:53.486 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:53.507 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:53.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:53.643 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.870+0000 7f10b3b9d700 1 -- 172.21.15.107:0/2325453246 >> v1:172.21.15.107:6790/0 conn(0x7f10ac0f8a80 legacy=0x7f10ac0f8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.870+0000 7f10b3b9d700 1 -- 172.21.15.107:0/2325453246 shutdown_connections 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.870+0000 7f10b3b9d700 1 -- 172.21.15.107:0/2325453246 >> 172.21.15.107:0/2325453246 conn(0x7f10ac0f2510 msgr2=0x7f10ac0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.870+0000 7f10b3b9d700 1 -- 172.21.15.107:0/2325453246 shutdown_connections 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.870+0000 7f10b3b9d700 1 -- 172.21.15.107:0/2325453246 wait complete. 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 librados: starting msgr at 2024-02-10T08:35:54.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 librados: starting objecter 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 Processor -- start 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 -- start start 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 librados: setting wanted keys 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 librados: calling monclient init 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10ac102600 con 0x7f10ac0f8a80 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10ac11a8a0 con 0x7f10ac104380 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b3b9d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10ac11ba40 con 0x7f10ac0f9570 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b1939700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10ac0f8a80 0x7f10ac101a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45338/0 (socket says 172.21.15.107:45338) 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10b1939700 1 -- 172.21.15.107:0/1320361755 learned_addr learned my addr 172.21.15.107:0/1320361755 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter ms_handle_connect 0x7f10ac0f8a80 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter ms_handle_connect 0x7f10ac0f9570 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:54.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter ms_handle_connect 0x7f10ac104380 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478232364 0 0) 0x7f10ac102600 con 0x7f10ac0f8a80 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1098003030 con 0x7f10ac0f8a80 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854234879 0 0) 0x7f10ac11ba40 con 0x7f10ac0f9570 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ac102600 con 0x7f10ac0f9570 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335022506 0 0) 0x7f10ac11a8a0 con 0x7f10ac104380 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.871+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ac11ba40 con 0x7f10ac104380 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441523688 0 0) 0x7f1098003030 con 0x7f10ac0f8a80 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ac11a8a0 con 0x7f10ac0f8a80 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f10a8002020 con 0x7f10ac0f8a80 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733370023 0 0) 0x7f10ac102600 con 0x7f10ac0f9570 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1098003030 con 0x7f10ac0f9570 2024-02-10T08:35:54.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f109c001620 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2607415994 0 0) 0x7f10ac11ba40 con 0x7f10ac104380 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ac102600 con 0x7f10ac104380 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f10a4001b00 con 0x7f10ac104380 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420233616 0 0) 0x7f1098003030 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 >> v1:172.21.15.195:6789/0 conn(0x7f10ac104380 legacy=0x7f10ac118170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 >> v1:172.21.15.107:6789/0 conn(0x7f10ac0f8a80 legacy=0x7f10ac101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac11cc40 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ac11bc30 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10b3b9d700 10 client.34727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac11c0e0 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10b3b9d700 1 librados: init done 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f109c002280 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.872+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f109c003240 con 0x7f10ac0f9570 2024-02-10T08:35:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.873+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090004f20 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.873+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f109c001de0 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(88..88 src has 30..88) v4 ==== 6172+0+0 (unknown 1872240537 0 0) 0x7f109c075c70 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 10 client.34727.objecter ms_dispatch 0x7f10ac001550 osd_map(88..88 src has 30..88) v4 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 3 client.34727.objecter handle_osd_map got epochs [88,88] > 0 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 3 client.34727.objecter handle_osd_map decoding full epoch 88 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 20 client.34727.objecter dump_active .. 0 homeless 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.874+0000 7f10a2ffd700 10 client.34727.objecter ms_handle_connect 0x7f1098063cc0 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:53.878+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f109c049820 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.074+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f1090005750 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.104+0000 7f10a2ffd700 1 -- 172.21.15.107:0/1320361755 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v89) v1 ==== 133+0+0 (unknown 3792644084 0 0) 0x7f109c047c30 con 0x7f10ac0f9570 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 10 librados: watch_flush enter 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 10 librados: watch_flush exit 2024-02-10T08:35:54.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 20 client.34727.objecter shutdown clearing up homeless session... 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 10 client.34727.objecter successfully canceled tick 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1098063cc0 legacy=0x7f1098066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 >> v1:172.21.15.107:6790/0 conn(0x7f10ac0f9570 legacy=0x7f10ac114a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 shutdown_connections 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 >> 172.21.15.107:0/1320361755 conn(0x7f10ac0f2510 msgr2=0x7f10ac1077f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 shutdown_connections 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 -- 172.21.15.107:0/1320361755 wait complete. 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:54.106+0000 7f10b3b9d700 1 librados: shutdown 2024-02-10T08:35:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-10T08:35:54.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:54.075503+0000 mon.c (mon.2) 81 : audit [INF] from='client.? v1:172.21.15.107:0/1320361755' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:54.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:54.076003+0000 mon.a (mon.0) 729 : audit [INF] from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:54.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:54.075503+0000 mon.c (mon.2) 81 : audit [INF] from='client.? v1:172.21.15.107:0/1320361755' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:54.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:54.076003+0000 mon.a (mon.0) 729 : audit [INF] from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:54.075503+0000 mon.c (mon.2) 81 : audit [INF] from='client.? v1:172.21.15.107:0/1320361755' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:54.076003+0000 mon.a (mon.0) 729 : audit [INF] from='client.34727 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:35:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:54.021715+0000 mgr.y (mgr.14553) 108 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:54.102480+0000 mon.a (mon.0) 730 : audit [INF] from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:35:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:54.102551+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-10T08:35:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:54.021715+0000 mgr.y (mgr.14553) 108 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:54.102480+0000 mon.a (mon.0) 730 : audit [INF] from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:35:55.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:54.102551+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-10T08:35:55.527 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:55.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:54.021715+0000 mgr.y (mgr.14553) 108 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:54.102480+0000 mon.a (mon.0) 730 : audit [INF] from='client.34727 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:35:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:54.102551+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-10T08:35:55.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:55.663 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:55.682 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:55.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:55.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:55.816 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:55.834 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:55.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:55.867 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:35:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:35:55] "GET /metrics HTTP/1.1" 200 172236 "" "Prometheus/2.33.4" 2024-02-10T08:35:55.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:55.965 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:56.117 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:56.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:56.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:56.255 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:56.276 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:56.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:56.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:56.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:55.146690+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:56.395 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:55.146690+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:56.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:55.146690+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:35:56.409 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:56.433 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:56.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:56.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:56.562 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:56.580 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:56.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:35:56.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.612+0000 7fa769bbf700 1 -- 172.21.15.107:0/248637529 >> v1:172.21.15.107:6789/0 conn(0x7fa7640fc910 legacy=0x7fa7640fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.612+0000 7fa769bbf700 1 -- 172.21.15.107:0/248637529 shutdown_connections 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.612+0000 7fa769bbf700 1 -- 172.21.15.107:0/248637529 >> 172.21.15.107:0/248637529 conn(0x7fa7640f2510 msgr2=0x7fa7640f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.612+0000 7fa769bbf700 1 -- 172.21.15.107:0/248637529 shutdown_connections 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.612+0000 7fa769bbf700 1 -- 172.21.15.107:0/248637529 wait complete. 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 librados: starting msgr at 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 librados: starting objecter 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 Processor -- start 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 -- start start 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 librados: setting wanted keys 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 librados: calling monclient init 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa7641196f0 con 0x7fa7640fc910 2024-02-10T08:35:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa76411a8d0 con 0x7fa764103250 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.613+0000 7fa769bbf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa76411bad0 con 0x7fa7640ff710 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa7637fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa7640fc910 0x7fa7640f8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45354/0 (socket says 172.21.15.107:45354) 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa762ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa7640ff710 0x7fa7640f8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59672/0 (socket says 172.21.15.107:59672) 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa7637fe700 1 -- 172.21.15.107:0/2112215073 learned_addr learned my addr 172.21.15.107:0/2112215073 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa763fff700 1 --1- 172.21.15.107:0/2112215073 >> v1:172.21.15.195:6789/0 conn(0x7fa764103250 0x7fa764117db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54686/0 (socket says 172.21.15.107:54686) 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa7640ff710 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa7640fc910 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa764103250 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917589128 0 0) 0x7fa76411bad0 con 0x7fa7640ff710 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c003630 con 0x7fa7640ff710 2024-02-10T08:35:56.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36530611 0 0) 0x7fa7641196f0 con 0x7fa7640fc910 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76411bad0 con 0x7fa7640fc910 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545008316 0 0) 0x7fa76411a8d0 con 0x7fa764103250 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7641196f0 con 0x7fa764103250 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211617189 0 0) 0x7fa76411bad0 con 0x7fa7640fc910 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76411a8d0 con 0x7fa7640fc910 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa754002050 con 0x7fa7640fc910 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573713099 0 0) 0x7fa74c003630 con 0x7fa7640ff710 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76411bad0 con 0x7fa7640ff710 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345873134 0 0) 0x7fa7641196f0 con 0x7fa764103250 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa74c003630 con 0x7fa764103250 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa758001860 con 0x7fa7640ff710 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa75c001b00 con 0x7fa764103250 2024-02-10T08:35:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3933764982 0 0) 0x7fa76411a8d0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 >> v1:172.21.15.107:6790/0 conn(0x7fa7640ff710 legacy=0x7fa7640f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 >> v1:172.21.15.195:6789/0 conn(0x7fa764103250 legacy=0x7fa764117db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76411ccd0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7641198e0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa769bbf700 10 client.14814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa764119dd0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.614+0000 7fa769bbf700 1 librados: init done 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.615+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa7540021b0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.615+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa754003850 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.616+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa750004f20 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.616+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa7540039b0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(89..89 src has 30..89) v4 ==== 6204+0+0 (unknown 2242310687 0 0) 0x7fa7540786a0 con 0x7fa7640fc910 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 10 client.14814.objecter ms_dispatch 0x7fa764001550 osd_map(89..89 src has 30..89) v4 2024-02-10T08:35:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 3 client.14814.objecter handle_osd_map got epochs [89,89] > 0 2024-02-10T08:35:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 3 client.14814.objecter handle_osd_map decoding full epoch 89 2024-02-10T08:35:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 20 client.14814.objecter dump_active .. 0 homeless 2024-02-10T08:35:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.617+0000 7fa760ff9700 10 client.14814.objecter ms_handle_connect 0x7fa74c063d90 2024-02-10T08:35:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.621+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa75404c2f0 con 0x7fa7640fc910 2024-02-10T08:35:56.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:35:56.714 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:56.731 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:35:56.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:56.836+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa7500051d0 con 0x7fa7640fc910 2024-02-10T08:35:56.839 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:35:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:56.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:35:56.850 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:35:56.850 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:35:56.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:35:56.981 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:57.003 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:35:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:57.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:35:57.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:57.141 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.166+0000 7fa760ff9700 1 -- 172.21.15.107:0/2112215073 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v90) v1 ==== 97+0+0 (unknown 2123621796 0 0) 0x7fa75404bee0 con 0x7fa7640fc910 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.169+0000 7fa769bbf700 10 librados: watch_flush enter 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 10 librados: watch_flush exit 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 20 client.14814.objecter shutdown clearing up homeless session... 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 10 client.14814.objecter successfully canceled tick 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa74c063d90 legacy=0x7fa74c066250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 >> v1:172.21.15.107:6789/0 conn(0x7fa7640fc910 legacy=0x7fa7640f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:35:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 shutdown_connections 2024-02-10T08:35:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.170+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 >> 172.21.15.107:0/2112215073 conn(0x7fa7640f2510 msgr2=0x7fa7640612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:35:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.171+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 shutdown_connections 2024-02-10T08:35:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.171+0000 7fa769bbf700 1 -- 172.21.15.107:0/2112215073 wait complete. 2024-02-10T08:35:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:35:57.171+0000 7fa769bbf700 1 librados: shutdown 2024-02-10T08:35:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:35:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:56.022251+0000 mgr.y (mgr.14553) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:56.837369+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:35:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:57.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:57.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:35:57.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:56.022251+0000 mgr.y (mgr.14553) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:57.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:56.837369+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:35:57.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:56.022251+0000 mgr.y (mgr.14553) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:35:57.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:56.837369+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:35:57.428 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:57.449 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:35:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:57.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:35:57.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:35:57.583 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:57.602 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:35:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:57.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:35:57.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:35:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:58.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:35:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:57.154099+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:35:57.166187+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:35:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:57.166236+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-10T08:35:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:57.154099+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:35:57.166187+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:35:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:57.166236+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-10T08:35:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:57.154099+0000 mon.a (mon.0) 734 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:35:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:35:57.166187+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.107:0/2112215073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:35:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:57.166236+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-10T08:35:58.593 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:35:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:58.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:35:58.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:35:58.733 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:59.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:35:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:35:58.022601+0000 mgr.y (mgr.14553) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:35:59.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:35:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:35:58.022601+0000 mgr.y (mgr.14553) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:35:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:35:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:35:58.022601+0000 mgr.y (mgr.14553) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:35:59.792 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:35:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:59.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:35:59.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:35:59.928 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:35:59.950 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:35:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:35:59.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:00.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:00.084 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.100 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:36:00.100 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:36:00.101 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2024-02-10T08:36:00.101 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2024-02-10T08:36:00.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:36:00.105 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:00.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:00.237 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.256 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:00.281 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:00.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.330+0000 7fa28f59e700 1 -- 172.21.15.107:0/1490010521 >> v1:172.21.15.107:6790/0 conn(0x7fa2900fc7c0 legacy=0x7fa2900fcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.330+0000 7fa28f59e700 1 -- 172.21.15.107:0/1490010521 shutdown_connections 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.330+0000 7fa28f59e700 1 -- 172.21.15.107:0/1490010521 >> 172.21.15.107:0/1490010521 conn(0x7fa29006cb60 msgr2=0x7fa2900f3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.330+0000 7fa28f59e700 1 -- 172.21.15.107:0/1490010521 shutdown_connections 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.330+0000 7fa28f59e700 1 -- 172.21.15.107:0/1490010521 wait complete. 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 librados: starting msgr at 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 librados: starting objecter 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 Processor -- start 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 -- start start 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 librados: setting wanted keys 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.331+0000 7fa28f59e700 1 librados: calling monclient init 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa28f59e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2901151f0 con 0x7fa2901030a0 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa28f59e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2901163d0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa28f59e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2901165c0 con 0x7fa2900ff560 2024-02-10T08:36:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa28e59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa2900fc7c0 0x7fa2900f8510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:45380/0 (socket says 172.21.15.107:45380) 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa28e59c700 1 -- 172.21.15.107:0/4281537842 learned_addr learned my addr 172.21.15.107:0/4281537842 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter ms_handle_connect 0x7fa2900fc7c0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611685504 0 0) 0x7fa2901163d0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa278002220 con 0x7fa2900fc7c0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter ms_handle_connect 0x7fa2901030a0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3668384589 0 0) 0x7fa278002220 con 0x7fa2900fc7c0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2901163d0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter ms_handle_connect 0x7fa2900ff560 2024-02-10T08:36:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa280002e90 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710651210 0 0) 0x7fa2901151f0 con 0x7fa2901030a0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.332+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa278002220 con 0x7fa2901030a0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4059614645 0 0) 0x7fa2901163d0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6790/0 conn(0x7fa2900ff560 legacy=0x7fa2900f8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6789/0 conn(0x7fa2901030a0 legacy=0x7fa2901138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2901167b0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa290116a60 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa28f59e700 10 client.24718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa290116ed0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa28f59e700 1 librados: init done 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa2800024f0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.333+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa280003ae0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.334+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa29005f5a0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.334+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa280002050 con 0x7fa2900fc7c0 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(90..90 src has 40..90) v4 ==== 6204+0+0 (unknown 2226038741 0 0) 0x7fa2800789e0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 osd_map(90..90 src has 40..90) v4 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 3 client.24718.objecter handle_osd_map got epochs [90,90] > 0 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 3 client.24718.objecter handle_osd_map decoding full epoch 90 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 20 client.24718.objecter dump_active .. 0 homeless 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v90) v1 ==== 54+0+16 (unknown 623226358 0 2420000192) 0x7fa280078df0 con 0x7fa2900fc7c0 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa28f59e700 10 client.24718.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 1 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.335+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 1 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 0 1 2024-02-10T08:36:00.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 4 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 10 client.24718.objecter _send_command 1 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa290046d80 con 0x7fa290027410 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa278063b00 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.336+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290027410 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.337+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa290046d80 con 0x7fa290027410 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.337+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.337+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 1 = system:0 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.337+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 4 2024-02-10T08:36:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.337+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 0 1 2024-02-10T08:36:00.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 10 client.24718.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 2 2024-02-10T08:36:00.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 2 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa2900f5740 osd=0 3 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 0 2 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 4 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 10 client.24718.objecter _send_command 2 2024-02-10T08:36:00.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.352+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa290107c30 con 0x7fa290027410 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa290107c30 con 0x7fa290027410 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 4 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 0 2 2024-02-10T08:36:00.357 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 10 client.24718.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 1 3 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 4 2024-02-10T08:36:00.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 10 client.24718.objecter _send_command 3 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.355+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29012d320 con 0x7fa290027f00 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.356+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290027f00 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.357+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa29012d320 con 0x7fa290027f00 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.357+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.357+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 3 = system:0 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.357+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 4 2024-02-10T08:36:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.357+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 1 3 2024-02-10T08:36:00.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 10 client.24718.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 4 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 4 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29006e720 osd=1 3 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 1 4 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 4 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 10 client.24718.objecter _send_command 4 2024-02-10T08:36:00.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.372+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa29006e190 con 0x7fa290027f00 2024-02-10T08:36:00.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.374+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa29006e190 con 0x7fa290027f00 2024-02-10T08:36:00.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.374+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.374+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.374+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 4 2024-02-10T08:36:00.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.374+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 1 4 2024-02-10T08:36:00.375 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 10 client.24718.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 5 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 5 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 2 5 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 4 2024-02-10T08:36:00.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 10 client.24718.objecter _send_command 5 2024-02-10T08:36:00.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.376+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa290131d40 con 0x7fa290059ea0 2024-02-10T08:36:00.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.378+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290059ea0 2024-02-10T08:36:00.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.379+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa290131d40 con 0x7fa290059ea0 2024-02-10T08:36:00.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.379+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:00.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.379+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 5 = system:0 2024-02-10T08:36:00.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.379+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 4 2024-02-10T08:36:00.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.379+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 2 5 2024-02-10T08:36:00.386 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 10 client.24718.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 6 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 6 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29006e5a0 osd=2 3 2024-02-10T08:36:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 2 6 2024-02-10T08:36:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 4 2024-02-10T08:36:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 10 client.24718.objecter _send_command 6 2024-02-10T08:36:00.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.395+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa290059460 con 0x7fa290059ea0 2024-02-10T08:36:00.403 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa290059460 con 0x7fa290059ea0 2024-02-10T08:36:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 4 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 2 6 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 10 client.24718.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 7 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 7 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 3 7 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 4 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 10 client.24718.objecter _send_command 7 2024-02-10T08:36:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.398+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29012e380 con 0x7fa290133030 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.403+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290133030 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.404+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa29012e380 con 0x7fa290133030 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.404+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.404+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 7 = system:0 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.404+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 4 2024-02-10T08:36:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.404+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 3 7 2024-02-10T08:36:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 10 client.24718.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 8 2024-02-10T08:36:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 8 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012dc60 osd=3 3 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 3 8 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 4 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 10 client.24718.objecter _send_command 8 2024-02-10T08:36:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.419+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa29012d6d0 con 0x7fa290133030 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.420+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa29012d6d0 con 0x7fa290133030 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.420+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.420+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.420+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 4 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.420+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 3 8 2024-02-10T08:36:00.421 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 10 client.24718.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 9 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 9 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 4 9 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 4 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 10 client.24718.objecter _send_command 9 2024-02-10T08:36:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29012d960 con 0x7fa290132b90 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.421+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290132b90 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.422+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa29012d960 con 0x7fa290132b90 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.422+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.422+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 9 = system:0 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.422+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 4 2024-02-10T08:36:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.422+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 4 9 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 10 client.24718.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 10 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 10 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012d6d0 osd=4 3 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 4 10 2024-02-10T08:36:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 4 2024-02-10T08:36:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 10 client.24718.objecter _send_command 10 2024-02-10T08:36:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.438+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa29012d960 con 0x7fa290132b90 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa29012d960 con 0x7fa290132b90 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 4 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 4 10 2024-02-10T08:36:00.440 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.439+0000 7fa28f59e700 10 client.24718.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 11 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 11 2024-02-10T08:36:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 5 11 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 4 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 10 client.24718.objecter _send_command 11 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa29013f550 con 0x7fa29013b900 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.440+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa29013b900 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.441+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa29013f550 con 0x7fa29013b900 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.441+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.441+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 11 = system:0 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.441+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 4 2024-02-10T08:36:00.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.441+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 5 11 2024-02-10T08:36:00.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 10 client.24718.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 12 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 12 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29012d960 osd=5 3 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 5 12 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 4 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 10 client.24718.objecter _send_command 12 2024-02-10T08:36:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.455+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa29013f550 con 0x7fa29013b900 2024-02-10T08:36:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa29013f550 con 0x7fa29013b900 2024-02-10T08:36:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 4 2024-02-10T08:36:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 5 12 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 10 client.24718.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 13 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 13 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 6 13 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 4 2024-02-10T08:36:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 10 client.24718.objecter _send_command 13 2024-02-10T08:36:00.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.457+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2901407a0 con 0x7fa290141250 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.458+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa290141250 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.459+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa2901407a0 con 0x7fa290141250 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.459+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.459+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 13 = system:0 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.459+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 4 2024-02-10T08:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.459+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 6 13 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 10 client.24718.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 14 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 14 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa29013f550 osd=6 3 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 6 14 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 4 2024-02-10T08:36:00.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 10 client.24718.objecter _send_command 14 2024-02-10T08:36:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.476+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa2901407a0 con 0x7fa290141250 2024-02-10T08:36:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.478+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa2901407a0 con 0x7fa290141250 2024-02-10T08:36:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.478+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.478+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.478+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 4 2024-02-10T08:36:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.478+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 6 14 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 10 client.24718.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 15 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 15 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 7 15 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 4 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 10 client.24718.objecter _send_command 15 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa290147f00 con 0x7fa2901443f0 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.479+0000 7fa27f7fe700 10 client.24718.objecter ms_handle_connect 0x7fa2901443f0 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.480+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa290147f00 con 0x7fa2901443f0 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.480+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.480+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 15 = system:0 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.480+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 4 2024-02-10T08:36:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.480+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 7 15 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 10 client.24718.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign -1 16 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 20 client.24718.objecter _get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_remove -1 16 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 20 client.24718.objecter get_session s=0x7fa2901407a0 osd=7 3 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 15 client.24718.objecter _session_command_op_assign 7 16 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 4 2024-02-10T08:36:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 10 client.24718.objecter _send_command 16 2024-02-10T08:36:00.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.494+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa290147f00 con 0x7fa2901443f0 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.496+0000 7fa27f7fe700 1 -- 172.21.15.107:0/4281537842 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa290147f00 con 0x7fa2901443f0 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.496+0000 7fa27f7fe700 10 client.24718.objecter ms_dispatch 0x7fa290001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.496+0000 7fa27f7fe700 10 client.24718.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.496+0000 7fa27f7fe700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 4 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.496+0000 7fa27f7fe700 15 client.24718.objecter _session_command_op_remove 7 16 2024-02-10T08:36:00.497 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:00.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 librados: watch_flush enter 2024-02-10T08:36:00.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 librados: watch_flush exit 2024-02-10T08:36:00.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.0 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6801/3321596972 conn(0x7fa290027410 legacy=0x7fa2900277f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2900f5740 osd=0 1 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.1 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6805/133516126 conn(0x7fa290027f00 legacy=0x7fa2900282e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e720 osd=1 1 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.2 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6809/3575881728 conn(0x7fa290059ea0 legacy=0x7fa290130620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29006e5a0 osd=2 1 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.3 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6813/2871825691 conn(0x7fa290133030 legacy=0x7fa290135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012dc60 osd=3 1 2024-02-10T08:36:00.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.4 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.195:6800/2499671285 conn(0x7fa290132b90 legacy=0x7fa290139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28ed9d700 1 -- 172.21.15.107:0/4281537842 reap_dead start 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d6d0 osd=4 1 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.5 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.195:6804/4178321486 conn(0x7fa29013b900 legacy=0x7fa29013dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29012d960 osd=5 1 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.6 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.195:6808/1417479017 conn(0x7fa290141250 legacy=0x7fa290141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa29013f550 osd=6 1 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter close_session for osd.7 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.195:6812/1934176928 conn(0x7fa2901443f0 legacy=0x7fa2901467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter put_session s=0x7fa2901407a0 osd=7 1 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 20 client.24718.objecter shutdown clearing up homeless session... 2024-02-10T08:36:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 10 client.24718.objecter successfully canceled tick 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28ed9d700 1 -- 172.21.15.107:0/4281537842 reap_dead start 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28ed9d700 1 -- 172.21.15.107:0/4281537842 reap_dead start 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28ed9d700 1 -- 172.21.15.107:0/4281537842 reap_dead start 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28ed9d700 1 -- 172.21.15.107:0/4281537842 reap_dead start 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa278063b00 legacy=0x7fa278065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> v1:172.21.15.195:6789/0 conn(0x7fa2900fc7c0 legacy=0x7fa2900f8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 shutdown_connections 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 >> 172.21.15.107:0/4281537842 conn(0x7fa29006cb60 msgr2=0x7fa2900f65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 shutdown_connections 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 -- 172.21.15.107:0/4281537842 wait complete. 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:00.497+0000 7fa28f59e700 1 librados: shutdown 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.502 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:36:00.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:00.533 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:00.535 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.553 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:00.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:00.683 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:00.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:00.335727+0000 mon.b (mon.1) 36 : audit [DBG] from='client.? v1:172.21.15.107:0/4281537842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:00.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:00.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:00.335727+0000 mon.b (mon.1) 36 : audit [DBG] from='client.? v1:172.21.15.107:0/4281537842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:00.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:00.335727+0000 mon.b (mon.1) 36 : audit [DBG] from='client.? v1:172.21.15.107:0/4281537842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:00.843 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:00.907 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:00.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:01.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:01.041 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:01.061 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:01.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:01.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:01.247 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:01.265 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:01.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:01.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:01.398 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:01.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:00.023003+0000 mgr.y (mgr.14553) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:36:01.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:36:01.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:00.023003+0000 mgr.y (mgr.14553) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:36:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:00.023003+0000 mgr.y (mgr.14553) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:36:02.253 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:02.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:02.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:02.393 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:02.414 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:02.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:02.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:02.547 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:02.567 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:02.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:02.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:02.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:01.110356+0000 mgr.y (mgr.14553) 112 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:01.110356+0000 mgr.y (mgr.14553) 112 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:02.692 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34349.0:102 5.3 5:c609908c:::notify.5:head [watch watch cookie 94547991869056] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:22.138154+0000", 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 160.556652614, 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013308459999999999, 2024-02-10T08:36:02.699 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34349", 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 102 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138154+0000", 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138154+0000", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138153+0000", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9020000000000004e-06 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138159+0000", 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.909e-06 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138167+0000", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4320000000000004e-06 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138173+0000", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032083600000000002 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138494+0000", 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5847000000000007e-05 2024-02-10T08:36:02.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138580+0000", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000167288 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138747+0000", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000351832 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139099+0000", 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000248791 2024-02-10T08:36:02.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139348+0000", 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8313999999999999e-05 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139386+0000", 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0088999999999999e-05 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139416+0000", 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9105999999999999e-05 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139485+0000", 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013308459999999999 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34379.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94862968492128] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 160.55627128899999, 2024-02-10T08:36:02.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020196910000000001, 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34379", 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 64 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138534+0000", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138541+0000", 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0132999999999999e-05 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138552+0000", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6989999999999997e-06 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138558+0000", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040325199999999999 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138961+0000", 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1836999999999996e-05 2024-02-10T08:36:02.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139043+0000", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047533599999999998 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139519+0000", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5971999999999999e-05 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139575+0000", 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114123 2024-02-10T08:36:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139689+0000", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026445499999999997 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139953+0000", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033451599999999998 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140288+0000", 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000182449 2024-02-10T08:36:02.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140470+0000", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9476999999999999e-05 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140500+0000", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5571e-05 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140555+0000", 2024-02-10T08:36:02.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020196910000000001 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(1.0 REQUEST e62)", 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:24.123419+0000", 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 158.57138759099999, 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001532355, 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123419+0000", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123419+0000", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999957 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123415+0000", 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9350000000000006e-06 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123425+0000", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.061e-05 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123435+0000", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3100000000000001e-06 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.123444+0000", 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014758989999999999 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.124919+0000", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8006e-05 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.124937+0000", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4097000000000001e-05 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:24.124952+0000", 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001532355 2024-02-10T08:36:02.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.119555+0000", 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.57525176499999, 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013179839999999999, 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119555+0000", 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119555+0000", 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119552+0000", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0903999999999999e-05 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119563+0000", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0502999999999999e-05 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119573+0000", 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3450000000000002e-06 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119579+0000", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7434e-05 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119627+0000", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105513 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119732+0000", 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019757299999999999 2024-02-10T08:36:02.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.119930+0000", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050402599999999997 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.120434+0000", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000290103 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.120724+0000", 2024-02-10T08:36:02.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9703000000000005e-05 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.120814+0000", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3884999999999998e-05 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.120848+0000", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5358e-05 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.120873+0000", 2024-02-10T08:36:02.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013179839999999999 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:6 4.3 4:c60da066:gc::gc.4:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.123460+0000", 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.57134728, 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001400114, 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.722 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123460+0000", 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123460+0000", 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123458+0000", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2120000000000003e-06 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123466+0000", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2544999999999999e-05 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123478+0000", 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3939999999999998e-06 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123486+0000", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2018999999999999e-05 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123528+0000", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0328000000000007e-05 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123618+0000", 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019567799999999999 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,7", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.123814+0000", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000338284 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.124152+0000", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041239599999999999 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.124564+0000", 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023828200000000001 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.124803+0000", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0571000000000003e-05 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.124833+0000", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6381999999999999e-05 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.124860+0000", 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001400114 2024-02-10T08:36:02.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:76 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.180775+0000", 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.514031862, 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001815826, 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.728 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 76 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180775+0000", 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180775+0000", 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180773+0000", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1993e-05 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180785+0000", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5429000000000001e-05 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180801+0000", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180808+0000", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4512999999999998e-05 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180863+0000", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111314 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180974+0000", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018145800000000001 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:36:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.181156+0000", 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036055499999999998 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.181516+0000", 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080964600000000002 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182326+0000", 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000179667 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182506+0000", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7615999999999999e-05 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182553+0000", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7602999999999999e-05 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182591+0000", 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001815826 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.733 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:79 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.183407+0000", 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.51140007999999, 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001670265, 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 79 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.734 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183407+0000", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183407+0000", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183405+0000", 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6980000000000003e-06 2024-02-10T08:36:02.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183411+0000", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0611999999999999e-05 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183422+0000", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2899999999999996e-06 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183426+0000", 2024-02-10T08:36:02.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6939e-05 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183463+0000", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2954000000000002e-05 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183506+0000", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011294399999999999 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183619+0000", 2024-02-10T08:36:02.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086169600000000001 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184481+0000", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048265200000000003 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184963+0000", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1109999999999999e-05 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184994+0000", 2024-02-10T08:36:02.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4397999999999997e-05 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185039+0000", 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8300000000000003e-05 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185077+0000", 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001670265 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:80 4.3 4:c31f41d7:gc::gc.10:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.183432+0000", 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.51137471000001, 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001337027, 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 80 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.740 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183432+0000", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183432+0000", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183432+0000", 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6170000000000001e-06 2024-02-10T08:36:02.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183434+0000", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8789999999999999e-06 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183437+0000", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1670000000000002e-06 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183439+0000", 2024-02-10T08:36:02.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1707e-05 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183471+0000", 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6542000000000003e-05 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183518+0000", 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011982200000000001 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,7", 2024-02-10T08:36:02.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.183637+0000", 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00085426999999999996 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184492+0000", 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011898199999999999 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184611+0000", 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8040999999999994e-05 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184689+0000", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1177999999999998e-05 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184730+0000", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9326000000000003e-05 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.184769+0000", 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001337027 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.745 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:84 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.50903962800001, 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021074520000000001, 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 84 2024-02-10T08:36:02.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185766+0000", 2024-02-10T08:36:02.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4710000000000007e-06 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185774+0000", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2590000000000006e-06 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185783+0000", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5350000000000004e-06 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185789+0000", 2024-02-10T08:36:02.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2379999999999996e-05 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185851+0000", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185950+0000", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017519500000000001 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186125+0000", 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037316999999999998 2024-02-10T08:36:02.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186498+0000", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039394499999999998 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186892+0000", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093651699999999999 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187828+0000", 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5956999999999999e-05 2024-02-10T08:36:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187854+0000", 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0312000000000001e-05 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187875+0000", 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021074520000000001 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.751 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:154 4.3 4:c623b173:gc::gc.16:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.212120+0000", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.482686973, 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001332094, 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 154 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212120+0000", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212120+0000", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212118+0000", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.122e-06 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212124+0000", 2024-02-10T08:36:02.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6279999999999994e-06 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212133+0000", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.42e-06 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212137+0000", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9213999999999998e-05 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212167+0000", 2024-02-10T08:36:02.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4949000000000004e-05 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212242+0000", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118218 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,7", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212360+0000", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000244107 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.212604+0000", 2024-02-10T08:36:02.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048410800000000002 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.213088+0000", 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028014099999999997 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.213368+0000", 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3524999999999999e-05 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.213432+0000", 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0299999999999999e-05 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.213452+0000", 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001332094 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:178 4.16 4:6849fd06:gc::gc.17:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.220044+0000", 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.474762801, 2024-02-10T08:36:02.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014918939999999999, 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 178 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220044+0000", 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220044+0000", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220043+0000", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2689999999999996e-06 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220049+0000", 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2591e-05 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220061+0000", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4160000000000001e-06 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220068+0000", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4480999999999997e-05 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220102+0000", 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0840999999999997e-05 2024-02-10T08:36:02.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220143+0000", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6990000000000005e-05 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220240+0000", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035325800000000001 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220593+0000", 2024-02-10T08:36:02.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014066799999999999 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.220734+0000", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071621900000000004 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.221450+0000", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7432000000000003e-05 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.221498+0000", 2024-02-10T08:36:02.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8188000000000001e-05 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.221536+0000", 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014918939999999999 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.24391.0:286 4.1a e63/57 4:5ae4284b:::obj_delete_at_hint.0000000032:head v 63'6, mlcod=63'4)", 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.42490663699999, 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021239140000000002, 2024-02-10T08:36:02.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269899+0000", 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9990000000000008e-06 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269907+0000", 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1367e-05 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269918+0000", 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0509999999999996e-06 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269923+0000", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8180000000000002e-05 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269981+0000", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8329e-05 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.270000+0000", 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001982045 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.271982+0000", 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2318999999999999e-05 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.272024+0000", 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021239140000000002 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:319 4.b 4:d242335b:gc::gc.2:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.285073+0000", 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.40973334899999, 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019274439999999999, 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 319 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285073+0000", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285073+0000", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285071+0000", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0688e-05 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285081+0000", 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3131e-05 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285095+0000", 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1058e-05 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285106+0000", 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5840000000000002e-05 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285151+0000", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4141000000000006e-05 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285236+0000", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019306799999999999 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285429+0000", 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036923000000000001 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285798+0000", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015285299999999999 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.285951+0000", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096042100000000004 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.286911+0000", 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1619000000000002e-05 2024-02-10T08:36:02.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.286963+0000", 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8139000000000002e-05 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.287001+0000", 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019274439999999999 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:327 4.b 4:d242335b:gc::gc.2:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.288302+0000", 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.406505205, 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001690988, 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 327 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288302+0000", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288302+0000", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288300+0000", 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2469999999999999e-06 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288306+0000", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8220000000000007e-06 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288313+0000", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9910000000000004e-06 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288317+0000", 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2623000000000003e-05 2024-02-10T08:36:02.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288350+0000", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3248999999999999e-05 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288393+0000", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104855 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.288498+0000", 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000515534 2024-02-10T08:36:02.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.289014+0000", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087469200000000005 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.289888+0000", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9801000000000001e-05 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.289918+0000", 2024-02-10T08:36:02.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2741999999999998e-05 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.289961+0000", 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1647999999999999e-05 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.289993+0000", 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001690988 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:338 4.3 4:c60da066:gc::gc.4:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.293932+0000", 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.40087501799999, 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001909191, 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 338 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293932+0000", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293932+0000", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293929+0000", 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0132999999999998e-05 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293949+0000", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7757999999999998e-05 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293967+0000", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9170000000000006e-06 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.293975+0000", 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4002000000000003e-05 2024-02-10T08:36:02.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.294019+0000", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0624000000000006e-05 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.294100+0000", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000180803 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,7", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.294280+0000", 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046608199999999999 2024-02-10T08:36:02.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.294746+0000", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8819999999999993e-05 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.294835+0000", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093126999999999999 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.295766+0000", 2024-02-10T08:36:02.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1834e-05 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.295808+0000", 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2666000000000002e-05 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.295841+0000", 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001909191 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.14565.0:2 3.1 3:90ab8ff9:::rbd_task:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected e65)", 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:40.161894+0000", 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 142.532912428, 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001603838, 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.14565", 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2319187403", 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161894+0000", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161894+0000", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161892+0000", 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7319999999999998e-06 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161900+0000", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0747999999999999e-05 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161911+0000", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7110000000000001e-06 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161919+0000", 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2171e-05 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161962+0000", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3550000000000004e-06 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.161967+0000", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014464689999999999 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.163413+0000", 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4740000000000005e-05 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:40.163498+0000", 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001603838 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.0 REQUEST e73)", 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:34:25.703437+0000", 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.991370149000005, 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013441830000000001, 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:02.790 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703437+0000", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703437+0000", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703436+0000", 2024-02-10T08:36:02.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703439+0000", 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4014e-05 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703453+0000", 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8329999999999996e-06 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703460+0000", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9725999999999998e-05 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703510+0000", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5969999999999997e-06 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.703517+0000", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001236107 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.704753+0000", 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6799999999999999e-06 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.704761+0000", 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9440999999999999e-05 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:25.704781+0000", 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013441830000000001 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.10 REQUEST e79)", 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:34:56.696905+0000", 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.997901405999997, 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001377894, 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696905+0000", 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696905+0000", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696905+0000", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.396e-06 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696908+0000", 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0049999999999993e-06 2024-02-10T08:36:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696917+0000", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4739999999999999e-05 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696931+0000", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2870000000000001e-05 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696974+0000", 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.742e-06 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.696981+0000", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00127523 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.698256+0000", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0149999999999999e-05 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.698266+0000", 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7002000000000001e-05 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:34:56.698283+0000", 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001377894 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.f REQUEST e80)", 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:35:00.130283+0000", 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.564523803, 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013458089999999999, 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:02.799 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130283+0000", 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130283+0000", 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967296 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130283+0000", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0700000000000001e-06 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130285+0000", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.61e-06 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130292+0000", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2640000000000002e-06 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130300+0000", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4384000000000001e-05 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130344+0000", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6570000000000002e-06 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.130350+0000", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001247431 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.131597+0000", 2024-02-10T08:36:02.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1980999999999999e-05 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.131609+0000", 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9649000000000001e-05 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:00.131629+0000", 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013458089999999999 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(4.7 REQUEST e86)", 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:35:42.320741+0000", 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.374066149000001, 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00162211, 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320741+0000", 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320741+0000", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320740+0000", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6819999999999999e-06 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320743+0000", 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0798999999999999e-05 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320754+0000", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.601e-06 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320760+0000", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5142000000000002e-05 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320806+0000", 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1690000000000003e-06 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.320813+0000", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015221620000000001 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.322335+0000", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0002e-05 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.322345+0000", 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7866000000000001e-05 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:02.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:35:42.322363+0000", 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00162211 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:02.808 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:01.110356+0000 mgr.y (mgr.14553) 112 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:02.809 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:02.919 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:02.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:03.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:03.059 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:02.023723+0000 mgr.y (mgr.14553) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:02.023723+0000 mgr.y (mgr.14553) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:03.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:36:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:02.023723+0000 mgr.y (mgr.14553) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:04.024110+0000 mgr.y (mgr.14553) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:05.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:05] "GET /metrics HTTP/1.1" 200 172236 "" "Prometheus/2.33.4" 2024-02-10T08:36:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:04.024110+0000 mgr.y (mgr.14553) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:06.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:04.024110+0000 mgr.y (mgr.14553) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:06.024633+0000 mgr.y (mgr.14553) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:06.024633+0000 mgr.y (mgr.14553) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:06.024633+0000 mgr.y (mgr.14553) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:08.391 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:08.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:08.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:08.526 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:08.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:36:09.084 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:09.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:09.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:09.221 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:09.242 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:09.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:09.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:09.374 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:09.394 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:09.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:09.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:08.025004+0000 mgr.y (mgr.14553) 116 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-10T08:36:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:08.025004+0000 mgr.y (mgr.14553) 116 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-10T08:36:09.984 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:09.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:08.025004+0000 mgr.y (mgr.14553) 116 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-02-10T08:36:10.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:10.124 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:10.150 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:10.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:10.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:10.284 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:10.304 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:10.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:10.324 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.369+0000 7f74bfb08700 1 -- 172.21.15.107:0/3200760686 >> v1:172.21.15.195:6789/0 conn(0x7f74b8103250 legacy=0x7f74b81056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.369+0000 7f74bfb08700 1 -- 172.21.15.107:0/3200760686 shutdown_connections 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.369+0000 7f74bfb08700 1 -- 172.21.15.107:0/3200760686 >> 172.21.15.107:0/3200760686 conn(0x7f74b80f2510 msgr2=0x7f74b80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.369+0000 7f74bfb08700 1 -- 172.21.15.107:0/3200760686 shutdown_connections 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.369+0000 7f74bfb08700 1 -- 172.21.15.107:0/3200760686 wait complete. 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 librados: starting msgr at 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 librados: starting objecter 2024-02-10T08:36:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 Processor -- start 2024-02-10T08:36:10.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 -- start start 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 librados: setting wanted keys 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 librados: calling monclient init 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74b8115310 con 0x7f74b80fc910 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74b81164f0 con 0x7f74b8103250 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bfb08700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74b81166e0 con 0x7f74b80ff710 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bd0a3700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f74b80ff710 0x7f74b80f8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:32844/0 (socket says 172.21.15.107:32844) 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74bd0a3700 1 -- 172.21.15.107:0/220261033 learned_addr learned my addr 172.21.15.107:0/220261033 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74aeffd700 10 client.?.objecter ms_handle_connect 0x7f74b80ff710 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.370+0000 7f74aeffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 10 client.?.objecter ms_handle_connect 0x7f74b80fc910 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 10 client.?.objecter ms_handle_connect 0x7f74b8103250 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258366126 0 0) 0x7f74b8115310 con 0x7f74b80fc910 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a4003030 con 0x7f74b80fc910 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843672465 0 0) 0x7f74b81166e0 con 0x7f74b80ff710 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b8115310 con 0x7f74b80ff710 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596778933 0 0) 0x7f74b81164f0 con 0x7f74b8103250 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b81166e0 con 0x7f74b8103250 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1440278922 0 0) 0x7f74b8115310 con 0x7f74b80ff710 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b81164f0 con 0x7f74b80ff710 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815656218 0 0) 0x7f74b81166e0 con 0x7f74b8103250 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b8115310 con 0x7f74b8103250 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f74a80018c0 con 0x7f74b80ff710 2024-02-10T08:36:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f74b0001d10 con 0x7f74b8103250 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106144160 0 0) 0x7f74a4003030 con 0x7f74b80fc910 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b81166e0 con 0x7f74b80fc910 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f74b4001840 con 0x7f74b80fc910 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.371+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 936898847 0 0) 0x7f74b81164f0 con 0x7f74b80ff710 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 >> v1:172.21.15.195:6789/0 conn(0x7f74b8103250 legacy=0x7f74b81139d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 >> v1:172.21.15.107:6789/0 conn(0x7f74b80fc910 legacy=0x7f74b80f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b81168d0 con 0x7f74b80ff710 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b8116b80 con 0x7f74b80ff710 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74bfb08700 10 client.34742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b8116ff0 con 0x7f74b80ff710 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74bfb08700 1 librados: init done 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f74a8002420 con 0x7f74b80ff710 2024-02-10T08:36:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.372+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f74a8005ae0 con 0x7f74b80ff710 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.373+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f74a8005c40 con 0x7f74b80ff710 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.373+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b805f7a0 con 0x7f74b80ff710 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.374+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(90..90 src has 40..90) v4 ==== 6204+0+0 (unknown 2226038741 0 0) 0x7f74a8079450 con 0x7f74b80ff710 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.374+0000 7f74aeffd700 10 client.34742.objecter ms_dispatch 0x7f74b8001550 osd_map(90..90 src has 40..90) v4 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.374+0000 7f74aeffd700 3 client.34742.objecter handle_osd_map got epochs [90,90] > 0 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.374+0000 7f74aeffd700 3 client.34742.objecter handle_osd_map decoding full epoch 90 2024-02-10T08:36:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.374+0000 7f74aeffd700 20 client.34742.objecter dump_active .. 0 homeless 2024-02-10T08:36:10.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.378+0000 7f74aeffd700 10 client.34742.objecter ms_handle_connect 0x7f74a4063c90 2024-02-10T08:36:10.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.380+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f74a804cf80 con 0x7f74b80ff710 2024-02-10T08:36:10.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:10.437 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:10.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:10.686+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f74b80f8140 con 0x7f74b80ff710 2024-02-10T08:36:11.298 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:11.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:11.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.308+0000 7f5c28b10700 1 -- 172.21.15.107:0/1252647049 >> v1:172.21.15.195:6789/0 conn(0x7f5c24107f30 legacy=0x7f5c2410a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.308+0000 7f5c28b10700 1 -- 172.21.15.107:0/1252647049 shutdown_connections 2024-02-10T08:36:11.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.308+0000 7f5c28b10700 1 -- 172.21.15.107:0/1252647049 >> 172.21.15.107:0/1252647049 conn(0x7f5c24063bc0 msgr2=0x7f5c240672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.311+0000 7f5c28b10700 1 -- 172.21.15.107:0/1252647049 shutdown_connections 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.311+0000 7f5c28b10700 1 -- 172.21.15.107:0/1252647049 wait complete. 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.311+0000 7f5c28b10700 1 librados: starting msgr at 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.311+0000 7f5c28b10700 1 librados: starting objecter 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.311+0000 7f5c28b10700 1 Processor -- start 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 -- start start 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 librados: setting wanted keys 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 librados: calling monclient init 2024-02-10T08:36:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c24117290 con 0x7f5c2410ba40 2024-02-10T08:36:11.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c241273f0 con 0x7f5c24103ff0 2024-02-10T08:36:11.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c28b10700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c241285d0 con 0x7f5c24107f30 2024-02-10T08:36:11.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c21d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5c24107f30 0x7f5c24100500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:32870/0 (socket says 172.21.15.107:32870) 2024-02-10T08:36:11.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c21d9b700 1 -- 172.21.15.107:0/40513373 learned_addr learned my addr 172.21.15.107:0/40513373 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c22d9d700 1 --1- 172.21.15.107:0/40513373 >> v1:172.21.15.107:6789/0 conn(0x7f5c2410ba40 0x7f5c24124cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45086/0 (socket says 172.21.15.107:45086) 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter ms_handle_connect 0x7f5c24107f30 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter ms_handle_connect 0x7f5c2410ba40 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter ms_handle_connect 0x7f5c24103ff0 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999382266 0 0) 0x7f5c241285d0 con 0x7f5c24107f30 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c003630 con 0x7f5c24107f30 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.312+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372963170 0 0) 0x7f5c24117290 con 0x7f5c2410ba40 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c241285d0 con 0x7f5c2410ba40 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458306445 0 0) 0x7f5c241273f0 con 0x7f5c24103ff0 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c24117290 con 0x7f5c24103ff0 2024-02-10T08:36:11.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873993108 0 0) 0x7f5c24117290 con 0x7f5c24103ff0 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c241273f0 con 0x7f5c24103ff0 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c14001650 con 0x7f5c24103ff0 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491196472 0 0) 0x7f5c0c003630 con 0x7f5c24107f30 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c24117290 con 0x7f5c24107f30 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 540607589 0 0) 0x7f5c241285d0 con 0x7f5c2410ba40 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c0c003630 con 0x7f5c2410ba40 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c18002d70 con 0x7f5c24107f30 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c1c001be0 con 0x7f5c2410ba40 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2723566545 0 0) 0x7f5c0c003630 con 0x7f5c2410ba40 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 >> v1:172.21.15.107:6790/0 conn(0x7f5c24107f30 legacy=0x7f5c24100500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 >> v1:172.21.15.195:6789/0 conn(0x7f5c24103ff0 legacy=0x7f5c240ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c241297d0 con 0x7f5c2410ba40 2024-02-10T08:36:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c24126440 con 0x7f5c2410ba40 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c28b10700 10 client.14829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c241268b0 con 0x7f5c2410ba40 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.313+0000 7f5c28b10700 1 librados: init done 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.315+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2405ffc0 con 0x7f5c2410ba40 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.315+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5c1c0031a0 con 0x7f5c2410ba40 2024-02-10T08:36:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.315+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c1c005c70 con 0x7f5c2410ba40 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.315+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5c1c018b00 con 0x7f5c2410ba40 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.316+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(90..90 src has 40..90) v4 ==== 6204+0+0 (unknown 2226038741 0 0) 0x7f5c1c079590 con 0x7f5c2410ba40 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.316+0000 7f5c137fe700 10 client.14829.objecter ms_dispatch 0x7f5c24001550 osd_map(90..90 src has 40..90) v4 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.316+0000 7f5c137fe700 3 client.14829.objecter handle_osd_map got epochs [90,90] > 0 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.316+0000 7f5c137fe700 3 client.14829.objecter handle_osd_map decoding full epoch 90 2024-02-10T08:36:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.316+0000 7f5c137fe700 20 client.14829.objecter dump_active .. 0 homeless 2024-02-10T08:36:11.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.321+0000 7f5c137fe700 10 client.14829.objecter ms_handle_connect 0x7f5c0c063ea0 2024-02-10T08:36:11.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.324+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5c1c04ceb0 con 0x7f5c2410ba40 2024-02-10T08:36:11.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:11.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:11.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.615+0000 7f74aeffd700 1 -- 172.21.15.107:0/220261033 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v91) v1 ==== 91+0+0 (unknown 3639485741 0 0) 0x7f74a804b390 con 0x7f74b80ff710 2024-02-10T08:36:11.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 10 librados: watch_flush enter 2024-02-10T08:36:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 10 librados: watch_flush exit 2024-02-10T08:36:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 20 client.34742.objecter shutdown clearing up homeless session... 2024-02-10T08:36:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 10 client.34742.objecter successfully canceled tick 2024-02-10T08:36:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 >> v1:172.21.15.107:6800/2634842009 conn(0x7f74a4063c90 legacy=0x7f74a4066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.625+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 >> v1:172.21.15.107:6790/0 conn(0x7f74b80ff710 legacy=0x7f74b80f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.628+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 shutdown_connections 2024-02-10T08:36:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.628+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 >> 172.21.15.107:0/220261033 conn(0x7f74b80f2510 msgr2=0x7f74b8106a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.628+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 shutdown_connections 2024-02-10T08:36:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.628+0000 7f74bfb08700 1 -- 172.21.15.107:0/220261033 wait complete. 2024-02-10T08:36:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.628+0000 7f74bfb08700 1 librados: shutdown 2024-02-10T08:36:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:36:11.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.642+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c24112930 con 0x7f5c0c063ea0 2024-02-10T08:36:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.658+0000 7f5c137fe700 1 -- 172.21.15.107:0/40513373 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357245 (unknown 2965378022 0 318094606) 0x7f5c24112930 con 0x7f5c0c063ea0 2024-02-10T08:36:11.665 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 10 librados: watch_flush enter 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 10 librados: watch_flush exit 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 20 client.14829.objecter shutdown clearing up homeless session... 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 10 client.14829.objecter successfully canceled tick 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5c0c063ea0 legacy=0x7f5c0c066360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 >> v1:172.21.15.107:6789/0 conn(0x7f5c2410ba40 legacy=0x7f5c24124cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 shutdown_connections 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.663+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 >> 172.21.15.107:0/40513373 conn(0x7f5c24063bc0 msgr2=0x7f5c24067af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.664+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 shutdown_connections 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.664+0000 7f5c28b10700 1 -- 172.21.15.107:0/40513373 wait complete. 2024-02-10T08:36:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:11.664+0000 7f5c28b10700 1 librados: shutdown 2024-02-10T08:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:36:11.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:11.735 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:10.025331+0000 mgr.y (mgr.14553) 117 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:10.687735+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.107:0/220261033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:11.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:10.688062+0000 mon.a (mon.0) 737 : audit [INF] from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:10.025331+0000 mgr.y (mgr.14553) 117 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:10.687735+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.107:0/220261033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:10.688062+0000 mon.a (mon.0) 737 : audit [INF] from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:10.025331+0000 mgr.y (mgr.14553) 117 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:10.687735+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.107:0/220261033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:10.688062+0000 mon.a (mon.0) 737 : audit [INF] from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:36:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:11.118107+0000 mgr.y (mgr.14553) 118 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:11.603649+0000 mon.a (mon.0) 738 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:11.615557+0000 mon.a (mon.0) 739 : audit [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:36:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:11.615619+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-10T08:36:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:11.645802+0000 mgr.y (mgr.14553) 119 : audit [DBG] from='client.14829 v1:172.21.15.107:0/40513373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:11.726895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.e scrub starts 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:11.732457+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.e scrub ok 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:11.118107+0000 mgr.y (mgr.14553) 118 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:11.603649+0000 mon.a (mon.0) 738 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:11.615557+0000 mon.a (mon.0) 739 : audit [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:11.615619+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:11.645802+0000 mgr.y (mgr.14553) 119 : audit [DBG] from='client.14829 v1:172.21.15.107:0/40513373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:11.726895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.e scrub starts 2024-02-10T08:36:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:11.732457+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.e scrub ok 2024-02-10T08:36:12.996 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:13.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:11.118107+0000 mgr.y (mgr.14553) 118 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:11.603649+0000 mon.a (mon.0) 738 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:11.615557+0000 mon.a (mon.0) 739 : audit [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:11.615619+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:11.645802+0000 mgr.y (mgr.14553) 119 : audit [DBG] from='client.14829 v1:172.21.15.107:0/40513373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:11.726895+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.e scrub starts 2024-02-10T08:36:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:11.732457+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.e scrub ok 2024-02-10T08:36:13.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:13.129 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:13.166 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:13.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:13.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:12.025875+0000 mgr.y (mgr.14553) 120 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:12.630473+0000 osd.5 (osd.5) 45 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:36:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:12.633363+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:36:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:12.025875+0000 mgr.y (mgr.14553) 120 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:12.630473+0000 osd.5 (osd.5) 45 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:36:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:12.633363+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:36:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:13.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:12.025875+0000 mgr.y (mgr.14553) 120 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:12.630473+0000 osd.5 (osd.5) 45 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:36:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:12.633363+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:36:14.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:14.092 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:14.115 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:14.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:14.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:14.248 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:14.267 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:14.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:14.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:14.399 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:14.420 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:14.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:14.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:14.552 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:14.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.536881+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:36:14.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.537836+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.647362+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.650976+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.775743+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.778669+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.521590+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:14.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.521655+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.536881+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.537836+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.647362+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.650976+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.775743+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.778669+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-02-10T08:36:14.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.521590+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:14.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.521655+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.536881+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.537836+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.647362+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.650976+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.775743+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.778669+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.521590+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.521655+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:14.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:36:15.407 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:15.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:15.415 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-02-10T08:36:11.612294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":886,"num_read_kb":638,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2382368,"kb_used_data":5600,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747578848,"statfs":{"total":767960285184,"available":765520740352,"internally_reserved":0,"allocated":5734400,"data_stored":2664907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.589520"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106025+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:35:54.106025+0000","last_peered":"2024-02-10T08:35:54.106025+0000","last_clean":"2024-02-10T08:35:54.106025+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:35:54.106025+0000","last_undegraded":"2024-02-10T08:35:54.106025+0000","last_fullsized":"2024-02-10T08:35:54.106025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101840+0000","last_change":"2024-02-10T08:35:23.876952+0000","last_active":"2024-02-10T08:35:54.101840+0000","last_peered":"2024-02-10T08:35:54.101840+0000","last_clean":"2024-02-10T08:35:54.101840+0000","last_became_active":"2024-02-10T08:33:22.117534+0000","last_became_peered":"2024-02-10T08:33:22.117534+0000","last_unstale":"2024-02-10T08:35:54.101840+0000","last_undegraded":"2024-02-10T08:35:54.101840+0000","last_fullsized":"2024-02-10T08:35:54.101840+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:23.876871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:23.876871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780480+0000","last_change":"2024-02-10T08:34:41.648797+0000","last_active":"2024-02-10T08:35:54.780480+0000","last_peered":"2024-02-10T08:35:54.780480+0000","last_clean":"2024-02-10T08:35:54.780480+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:35:54.780480+0000","last_undegraded":"2024-02-10T08:35:54.780480+0000","last_fullsized":"2024-02-10T08:35:54.780480+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:41.648715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:41.648715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105871+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:35:54.105871+0000","last_peered":"2024-02-10T08:35:54.105871+0000","last_clean":"2024-02-10T08:35:54.105871+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:35:54.105871+0000","last_undegraded":"2024-02-10T08:35:54.105871+0000","last_fullsized":"2024-02-10T08:35:54.105871+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102248+0000","last_change":"2024-02-10T08:34:38.118431+0000","last_active":"2024-02-10T08:35:54.102248+0000","last_peered":"2024-02-10T08:35:54.102248+0000","last_clean":"2024-02-10T08:35:54.102248+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:35:54.102248+0000","last_undegraded":"2024-02-10T08:35:54.102248+0000","last_fullsized":"2024-02-10T08:35:54.102248+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.118347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.118347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101013+0000","last_change":"2024-02-10T08:34:37.332223+0000","last_active":"2024-02-10T08:35:54.101013+0000","last_peered":"2024-02-10T08:35:54.101013+0000","last_clean":"2024-02-10T08:35:54.101013+0000","last_became_active":"2024-02-10T08:33:22.121607+0000","last_became_peered":"2024-02-10T08:33:22.121607+0000","last_unstale":"2024-02-10T08:35:54.101013+0000","last_undegraded":"2024-02-10T08:35:54.101013+0000","last_fullsized":"2024-02-10T08:35:54.101013+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.332140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.332140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106639+0000","last_change":"2024-02-10T08:34:23.594197+0000","last_active":"2024-02-10T08:35:54.106639+0000","last_peered":"2024-02-10T08:35:54.106639+0000","last_clean":"2024-02-10T08:35:54.106639+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:35:54.106639+0000","last_undegraded":"2024-02-10T08:35:54.106639+0000","last_fullsized":"2024-02-10T08:35:54.106639+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:23.594135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:23.594135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102000+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:35:54.102000+0000","last_peered":"2024-02-10T08:35:54.102000+0000","last_clean":"2024-02-10T08:35:54.102000+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:35:54.102000+0000","last_undegraded":"2024-02-10T08:35:54.102000+0000","last_fullsized":"2024-02-10T08:35:54.102000+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615458+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:35:54.615458+0000","last_peered":"2024-02-10T08:35:54.615458+0000","last_clean":"2024-02-10T08:35:54.615458+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:35:54.615458+0000","last_undegraded":"2024-02-10T08:35:54.615458+0000","last_fullsized":"2024-02-10T08:35:54.615458+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089243+0000","last_change":"2024-02-10T08:35:00.936215+0000","last_active":"2024-02-10T08:35:55.089243+0000","last_peered":"2024-02-10T08:35:55.089243+0000","last_clean":"2024-02-10T08:35:55.089243+0000","last_became_active":"2024-02-10T08:33:22.119141+0000","last_became_peered":"2024-02-10T08:33:22.119141+0000","last_unstale":"2024-02-10T08:35:55.089243+0000","last_undegraded":"2024-02-10T08:35:55.089243+0000","last_fullsized":"2024-02-10T08:35:55.089243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089898+0000","last_change":"2024-02-10T08:34:26.673356+0000","last_active":"2024-02-10T08:35:55.089898+0000","last_peered":"2024-02-10T08:35:55.089898+0000","last_clean":"2024-02-10T08:35:55.089898+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:35:55.089898+0000","last_undegraded":"2024-02-10T08:35:55.089898+0000","last_fullsized":"2024-02-10T08:35:55.089898+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:26.673263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.673263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102129+0000","last_change":"2024-02-10T08:34:50.094010+0000","last_active":"2024-02-10T08:35:54.102129+0000","last_peered":"2024-02-10T08:35:54.102129+0000","last_clean":"2024-02-10T08:35:54.102129+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:35:54.102129+0000","last_undegraded":"2024-02-10T08:35:54.102129+0000","last_fullsized":"2024-02-10T08:35:54.102129+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:34:50.093915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.093915+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779796+0000","last_change":"2024-02-10T08:34:51.681056+0000","last_active":"2024-02-10T08:35:54.779796+0000","last_peered":"2024-02-10T08:35:54.779796+0000","last_clean":"2024-02-10T08:35:54.779796+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:35:54.779796+0000","last_undegraded":"2024-02-10T08:35:54.779796+0000","last_fullsized":"2024-02-10T08:35:54.779796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.680986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.680986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615896+0000","last_change":"2024-02-10T08:35:01.492010+0000","last_active":"2024-02-10T08:35:54.615896+0000","last_peered":"2024-02-10T08:35:54.615896+0000","last_clean":"2024-02-10T08:35:54.615896+0000","last_became_active":"2024-02-10T08:33:22.122241+0000","last_became_peered":"2024-02-10T08:33:22.122241+0000","last_unstale":"2024-02-10T08:35:54.615896+0000","last_undegraded":"2024-02-10T08:35:54.615896+0000","last_fullsized":"2024-02-10T08:35:54.615896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089858+0000","last_change":"2024-02-10T08:34:56.956608+0000","last_active":"2024-02-10T08:35:55.089858+0000","last_peered":"2024-02-10T08:35:55.089858+0000","last_clean":"2024-02-10T08:35:55.089858+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:35:55.089858+0000","last_undegraded":"2024-02-10T08:35:55.089858+0000","last_fullsized":"2024-02-10T08:35:55.089858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.956483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.956483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102179+0000","last_change":"2024-02-10T08:34:59.084224+0000","last_active":"2024-02-10T08:35:54.102179+0000","last_peered":"2024-02-10T08:35:54.102179+0000","last_clean":"2024-02-10T08:35:54.102179+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:35:54.102179+0000","last_undegraded":"2024-02-10T08:35:54.102179+0000","last_fullsized":"2024-02-10T08:35:54.102179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:59.084151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:59.084151+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106757+0000","last_change":"2024-02-10T08:34:52.516328+0000","last_active":"2024-02-10T08:35:54.106757+0000","last_peered":"2024-02-10T08:35:54.106757+0000","last_clean":"2024-02-10T08:35:54.106757+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:35:54.106757+0000","last_undegraded":"2024-02-10T08:35:54.106757+0000","last_fullsized":"2024-02-10T08:35:54.106757+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.516236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.516236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102415+0000","last_change":"2024-02-10T08:35:07.038806+0000","last_active":"2024-02-10T08:35:54.102415+0000","last_peered":"2024-02-10T08:35:54.102415+0000","last_clean":"2024-02-10T08:35:54.102415+0000","last_became_active":"2024-02-10T08:33:22.116085+0000","last_became_peered":"2024-02-10T08:33:22.116085+0000","last_unstale":"2024-02-10T08:35:54.102415+0000","last_undegraded":"2024-02-10T08:35:54.102415+0000","last_fullsized":"2024-02-10T08:35:54.102415+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106118+0000","last_change":"2024-02-10T08:34:37.616548+0000","last_active":"2024-02-10T08:35:54.106118+0000","last_peered":"2024-02-10T08:35:54.106118+0000","last_clean":"2024-02-10T08:35:54.106118+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:35:54.106118+0000","last_undegraded":"2024-02-10T08:35:54.106118+0000","last_fullsized":"2024-02-10T08:35:54.106118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:37.616436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:37.616436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106582+0000","last_change":"2024-02-10T08:34:33.601286+0000","last_active":"2024-02-10T08:35:54.106582+0000","last_peered":"2024-02-10T08:35:54.106582+0000","last_clean":"2024-02-10T08:35:54.106582+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:35:54.106582+0000","last_undegraded":"2024-02-10T08:35:54.106582+0000","last_fullsized":"2024-02-10T08:35:54.106582+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:34:33.601201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:33.601201+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102775+0000","last_change":"2024-02-10T08:35:17.895268+0000","last_active":"2024-02-10T08:35:54.102775+0000","last_peered":"2024-02-10T08:35:54.102775+0000","last_clean":"2024-02-10T08:35:54.102775+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:35:54.102775+0000","last_undegraded":"2024-02-10T08:35:54.102775+0000","last_fullsized":"2024-02-10T08:35:54.102775+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:17.895163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:17.895163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101930+0000","last_change":"2024-02-10T08:34:39.357828+0000","last_active":"2024-02-10T08:35:54.101930+0000","last_peered":"2024-02-10T08:35:54.101930+0000","last_clean":"2024-02-10T08:35:54.101930+0000","last_became_active":"2024-02-10T08:33:22.120631+0000","last_became_peered":"2024-02-10T08:33:22.120631+0000","last_unstale":"2024-02-10T08:35:54.101930+0000","last_undegraded":"2024-02-10T08:35:54.101930+0000","last_fullsized":"2024-02-10T08:35:54.101930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.357766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517839+0000","last_change":"2024-02-10T08:34:53.298291+0000","last_active":"2024-02-10T08:35:54.517839+0000","last_peered":"2024-02-10T08:35:54.517839+0000","last_clean":"2024-02-10T08:35:54.517839+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:35:54.517839+0000","last_undegraded":"2024-02-10T08:35:54.517839+0000","last_fullsized":"2024-02-10T08:35:54.517839+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.298226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.298226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.703153+0000","last_change":"2024-02-10T08:34:26.526319+0000","last_active":"2024-02-10T08:35:54.703153+0000","last_peered":"2024-02-10T08:35:54.703153+0000","last_clean":"2024-02-10T08:35:54.703153+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:35:54.703153+0000","last_undegraded":"2024-02-10T08:35:54.703153+0000","last_fullsized":"2024-02-10T08:35:54.703153+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:34:26.526206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:26.526206+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101849+0000","last_change":"2024-02-10T08:34:52.285891+0000","last_active":"2024-02-10T08:35:54.101849+0000","last_peered":"2024-02-10T08:35:54.101849+0000","last_clean":"2024-02-10T08:35:54.101849+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:35:54.101849+0000","last_undegraded":"2024-02-10T08:35:54.101849+0000","last_fullsized":"2024-02-10T08:35:54.101849+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.285824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.285824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780413+0000","last_change":"2024-02-10T08:35:01.687455+0000","last_active":"2024-02-10T08:35:54.780413+0000","last_peered":"2024-02-10T08:35:54.780413+0000","last_clean":"2024-02-10T08:35:54.780413+0000","last_became_active":"2024-02-10T08:33:22.122436+0000","last_became_peered":"2024-02-10T08:33:22.122436+0000","last_unstale":"2024-02-10T08:35:54.780413+0000","last_undegraded":"2024-02-10T08:35:54.780413+0000","last_fullsized":"2024-02-10T08:35:54.780413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.687386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105740+0000","last_change":"2024-02-10T08:34:24.604411+0000","last_active":"2024-02-10T08:35:54.105740+0000","last_peered":"2024-02-10T08:35:54.105740+0000","last_clean":"2024-02-10T08:35:54.105740+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:35:54.105740+0000","last_undegraded":"2024-02-10T08:35:54.105740+0000","last_fullsized":"2024-02-10T08:35:54.105740+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:24.604347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:24.604347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.518183+0000","last_change":"2024-02-10T08:35:45.543177+0000","last_active":"2024-02-10T08:35:54.518183+0000","last_peered":"2024-02-10T08:35:54.518183+0000","last_clean":"2024-02-10T08:35:54.518183+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:35:54.518183+0000","last_undegraded":"2024-02-10T08:35:54.518183+0000","last_fullsized":"2024-02-10T08:35:54.518183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:45.543110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.543110+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.090133+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:35:55.090133+0000","last_peered":"2024-02-10T08:35:55.090133+0000","last_clean":"2024-02-10T08:35:55.090133+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:35:55.090133+0000","last_undegraded":"2024-02-10T08:35:55.090133+0000","last_fullsized":"2024-02-10T08:35:55.090133+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'9","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521677+0000","last_change":"2024-02-10T08:34:35.421984+0000","last_active":"2024-02-10T08:36:01.521677+0000","last_peered":"2024-02-10T08:36:01.521677+0000","last_clean":"2024-02-10T08:36:01.521677+0000","last_became_active":"2024-02-10T08:33:22.119251+0000","last_became_peered":"2024-02-10T08:33:22.119251+0000","last_unstale":"2024-02-10T08:36:01.521677+0000","last_undegraded":"2024-02-10T08:36:01.521677+0000","last_fullsized":"2024-02-10T08:36:01.521677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_deep_scrub":"74'9","last_deep_scrub_stamp":"2024-02-10T08:34:35.421866+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.421866+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615516+0000","last_change":"2024-02-10T08:35:36.513411+0000","last_active":"2024-02-10T08:35:54.615516+0000","last_peered":"2024-02-10T08:35:54.615516+0000","last_clean":"2024-02-10T08:35:54.615516+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:35:54.615516+0000","last_undegraded":"2024-02-10T08:35:54.615516+0000","last_fullsized":"2024-02-10T08:35:54.615516+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:36.513316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.513316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780370+0000","last_change":"2024-02-10T08:34:43.680148+0000","last_active":"2024-02-10T08:35:54.780370+0000","last_peered":"2024-02-10T08:35:54.780370+0000","last_clean":"2024-02-10T08:35:54.780370+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:35:54.780370+0000","last_undegraded":"2024-02-10T08:35:54.780370+0000","last_fullsized":"2024-02-10T08:35:54.780370+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:43.680073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:43.680073+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106231+0000","last_change":"2024-02-10T08:34:32.620154+0000","last_active":"2024-02-10T08:35:54.106231+0000","last_peered":"2024-02-10T08:35:54.106231+0000","last_clean":"2024-02-10T08:35:54.106231+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:35:54.106231+0000","last_undegraded":"2024-02-10T08:35:54.106231+0000","last_fullsized":"2024-02-10T08:35:54.106231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:32.620086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.620086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089189+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:35:55.089189+0000","last_peered":"2024-02-10T08:35:55.089189+0000","last_clean":"2024-02-10T08:35:55.089189+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:35:55.089189+0000","last_undegraded":"2024-02-10T08:35:55.089189+0000","last_fullsized":"2024-02-10T08:35:55.089189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.518258+0000","last_change":"2024-02-10T08:34:31.359650+0000","last_active":"2024-02-10T08:35:54.518258+0000","last_peered":"2024-02-10T08:35:54.518258+0000","last_clean":"2024-02-10T08:35:54.518258+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:35:54.518258+0000","last_undegraded":"2024-02-10T08:35:54.518258+0000","last_fullsized":"2024-02-10T08:35:54.518258+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105885+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:35:54.105885+0000","last_peered":"2024-02-10T08:35:54.105885+0000","last_clean":"2024-02-10T08:35:54.105885+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:35:54.105885+0000","last_undegraded":"2024-02-10T08:35:54.105885+0000","last_fullsized":"2024-02-10T08:35:54.105885+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089509+0000","last_change":"2024-02-10T08:34:35.819857+0000","last_active":"2024-02-10T08:35:55.089509+0000","last_peered":"2024-02-10T08:35:55.089509+0000","last_clean":"2024-02-10T08:35:55.089509+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:35:55.089509+0000","last_undegraded":"2024-02-10T08:35:55.089509+0000","last_fullsized":"2024-02-10T08:35:55.089509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.819777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.819777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517354+0000","last_change":"2024-02-10T08:34:55.286635+0000","last_active":"2024-02-10T08:35:54.517354+0000","last_peered":"2024-02-10T08:35:54.517354+0000","last_clean":"2024-02-10T08:35:54.517354+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:35:54.517354+0000","last_undegraded":"2024-02-10T08:35:54.517354+0000","last_fullsized":"2024-02-10T08:35:54.517354+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.286540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.286540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.616183+0000","last_change":"2024-02-10T08:34:28.590439+0000","last_active":"2024-02-10T08:35:54.616183+0000","last_peered":"2024-02-10T08:35:54.616183+0000","last_clean":"2024-02-10T08:35:54.616183+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:35:54.616183+0000","last_undegraded":"2024-02-10T08:35:54.616183+0000","last_fullsized":"2024-02-10T08:35:54.616183+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.590362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.590362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105652+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:35:54.105652+0000","last_peered":"2024-02-10T08:35:54.105652+0000","last_clean":"2024-02-10T08:35:54.105652+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:35:54.105652+0000","last_undegraded":"2024-02-10T08:35:54.105652+0000","last_fullsized":"2024-02-10T08:35:54.105652+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105603+0000","last_change":"2024-02-10T08:34:28.605230+0000","last_active":"2024-02-10T08:35:54.105603+0000","last_peered":"2024-02-10T08:35:54.105603+0000","last_clean":"2024-02-10T08:35:54.105603+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:35:54.105603+0000","last_undegraded":"2024-02-10T08:35:54.105603+0000","last_fullsized":"2024-02-10T08:35:54.105603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:28.605166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:28.605166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101634+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:35:54.101634+0000","last_peered":"2024-02-10T08:35:54.101634+0000","last_clean":"2024-02-10T08:35:54.101634+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:35:54.101634+0000","last_undegraded":"2024-02-10T08:35:54.101634+0000","last_fullsized":"2024-02-10T08:35:54.101634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101953+0000","last_change":"2024-02-10T08:34:36.300831+0000","last_active":"2024-02-10T08:35:54.101953+0000","last_peered":"2024-02-10T08:35:54.101953+0000","last_clean":"2024-02-10T08:35:54.101953+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:35:54.101953+0000","last_undegraded":"2024-02-10T08:35:54.101953+0000","last_fullsized":"2024-02-10T08:35:54.101953+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:36.300770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.300770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102268+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:35:54.102268+0000","last_peered":"2024-02-10T08:35:54.102268+0000","last_clean":"2024-02-10T08:35:54.102268+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:35:54.102268+0000","last_undegraded":"2024-02-10T08:35:54.102268+0000","last_fullsized":"2024-02-10T08:35:54.102268+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779453+0000","last_change":"2024-02-10T08:34:48.691828+0000","last_active":"2024-02-10T08:35:54.779453+0000","last_peered":"2024-02-10T08:35:54.779453+0000","last_clean":"2024-02-10T08:35:54.779453+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:35:54.779453+0000","last_undegraded":"2024-02-10T08:35:54.779453+0000","last_fullsized":"2024-02-10T08:35:54.779453+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.691737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.691737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106154+0000","last_change":"2024-02-10T08:35:43.333489+0000","last_active":"2024-02-10T08:35:54.106154+0000","last_peered":"2024-02-10T08:35:54.106154+0000","last_clean":"2024-02-10T08:35:54.106154+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:35:54.106154+0000","last_undegraded":"2024-02-10T08:35:54.106154+0000","last_fullsized":"2024-02-10T08:35:54.106154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517953+0000","last_change":"2024-02-10T08:35:37.562620+0000","last_active":"2024-02-10T08:35:54.517953+0000","last_peered":"2024-02-10T08:35:54.517953+0000","last_clean":"2024-02-10T08:35:54.517953+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:35:54.517953+0000","last_undegraded":"2024-02-10T08:35:54.517953+0000","last_fullsized":"2024-02-10T08:35:54.517953+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:37.562545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:37.562545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101593+0000","last_change":"2024-02-10T08:34:47.420238+0000","last_active":"2024-02-10T08:35:54.101593+0000","last_peered":"2024-02-10T08:35:54.101593+0000","last_clean":"2024-02-10T08:35:54.101593+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:35:54.101593+0000","last_undegraded":"2024-02-10T08:35:54.101593+0000","last_fullsized":"2024-02-10T08:35:54.101593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:34:47.420174+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.420174+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.103007+0000","last_change":"2024-02-10T08:35:29.834206+0000","last_active":"2024-02-10T08:35:54.103007+0000","last_peered":"2024-02-10T08:35:54.103007+0000","last_clean":"2024-02-10T08:35:54.103007+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:35:54.103007+0000","last_undegraded":"2024-02-10T08:35:54.103007+0000","last_fullsized":"2024-02-10T08:35:54.103007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:29.834114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:29.834114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521364+0000","last_change":"2024-02-10T08:34:51.497308+0000","last_active":"2024-02-10T08:36:01.521364+0000","last_peered":"2024-02-10T08:36:01.521364+0000","last_clean":"2024-02-10T08:36:01.521364+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:36:01.521364+0000","last_undegraded":"2024-02-10T08:36:01.521364+0000","last_fullsized":"2024-02-10T08:36:01.521364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.497242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.497242+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780082+0000","last_change":"2024-02-10T08:35:45.760941+0000","last_active":"2024-02-10T08:35:54.780082+0000","last_peered":"2024-02-10T08:35:54.780082+0000","last_clean":"2024-02-10T08:35:54.780082+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:35:54.780082+0000","last_undegraded":"2024-02-10T08:35:54.780082+0000","last_fullsized":"2024-02-10T08:35:54.780082+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.760869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.616180+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:35:54.616180+0000","last_peered":"2024-02-10T08:35:54.616180+0000","last_clean":"2024-02-10T08:35:54.616180+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:35:54.616180+0000","last_undegraded":"2024-02-10T08:35:54.616180+0000","last_fullsized":"2024-02-10T08:35:54.616180+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780109+0000","last_change":"2024-02-10T08:35:02.705789+0000","last_active":"2024-02-10T08:35:54.780109+0000","last_peered":"2024-02-10T08:35:54.780109+0000","last_clean":"2024-02-10T08:35:54.780109+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:35:54.780109+0000","last_undegraded":"2024-02-10T08:35:54.780109+0000","last_fullsized":"2024-02-10T08:35:54.780109+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.705679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.705679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521493+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:36:01.521493+0000","last_peered":"2024-02-10T08:36:01.521493+0000","last_clean":"2024-02-10T08:36:01.521493+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:36:01.521493+0000","last_undegraded":"2024-02-10T08:36:01.521493+0000","last_fullsized":"2024-02-10T08:36:01.521493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615699+0000","last_change":"2024-02-10T08:34:29.584600+0000","last_active":"2024-02-10T08:35:54.615699+0000","last_peered":"2024-02-10T08:35:54.615699+0000","last_clean":"2024-02-10T08:35:54.615699+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:35:54.615699+0000","last_undegraded":"2024-02-10T08:35:54.615699+0000","last_fullsized":"2024-02-10T08:35:54.615699+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.584529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.584529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106334+0000","last_change":"2024-02-10T08:34:31.663950+0000","last_active":"2024-02-10T08:35:54.106334+0000","last_peered":"2024-02-10T08:35:54.106334+0000","last_clean":"2024-02-10T08:35:54.106334+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:35:54.106334+0000","last_undegraded":"2024-02-10T08:35:54.106334+0000","last_fullsized":"2024-02-10T08:35:54.106334+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:34:31.663882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.663882+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089864+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:35:55.089864+0000","last_peered":"2024-02-10T08:35:55.089864+0000","last_clean":"2024-02-10T08:35:55.089864+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:35:55.089864+0000","last_undegraded":"2024-02-10T08:35:55.089864+0000","last_fullsized":"2024-02-10T08:35:55.089864+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615383+0000","last_change":"2024-02-10T08:34:45.607331+0000","last_active":"2024-02-10T08:35:54.615383+0000","last_peered":"2024-02-10T08:35:54.615383+0000","last_clean":"2024-02-10T08:35:54.615383+0000","last_became_active":"2024-02-10T08:33:22.121591+0000","last_became_peered":"2024-02-10T08:33:22.121591+0000","last_unstale":"2024-02-10T08:35:54.615383+0000","last_undegraded":"2024-02-10T08:35:54.615383+0000","last_fullsized":"2024-02-10T08:35:54.615383+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.607254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.607254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517592+0000","last_change":"2024-02-10T08:34:29.402936+0000","last_active":"2024-02-10T08:35:54.517592+0000","last_peered":"2024-02-10T08:35:54.517592+0000","last_clean":"2024-02-10T08:35:54.517592+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:35:54.517592+0000","last_undegraded":"2024-02-10T08:35:54.517592+0000","last_fullsized":"2024-02-10T08:35:54.517592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.402874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.402874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106681+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:35:54.106681+0000","last_peered":"2024-02-10T08:35:54.106681+0000","last_clean":"2024-02-10T08:35:54.106681+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:35:54.106681+0000","last_undegraded":"2024-02-10T08:35:54.106681+0000","last_fullsized":"2024-02-10T08:35:54.106681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702923+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:35:54.702923+0000","last_peered":"2024-02-10T08:35:54.702923+0000","last_clean":"2024-02-10T08:35:54.702923+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:35:54.702923+0000","last_undegraded":"2024-02-10T08:35:54.702923+0000","last_fullsized":"2024-02-10T08:35:54.702923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106790+0000","last_change":"2024-02-10T08:35:02.446706+0000","last_active":"2024-02-10T08:35:54.106790+0000","last_peered":"2024-02-10T08:35:54.106790+0000","last_clean":"2024-02-10T08:35:54.106790+0000","last_became_active":"2024-02-10T08:33:22.118323+0000","last_became_peered":"2024-02-10T08:33:22.118323+0000","last_unstale":"2024-02-10T08:35:54.106790+0000","last_undegraded":"2024-02-10T08:35:54.106790+0000","last_fullsized":"2024-02-10T08:35:54.106790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702631+0000","last_change":"2024-02-10T08:34:56.699511+0000","last_active":"2024-02-10T08:35:54.702631+0000","last_peered":"2024-02-10T08:35:54.702631+0000","last_clean":"2024-02-10T08:35:54.702631+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:35:54.702631+0000","last_undegraded":"2024-02-10T08:35:54.702631+0000","last_fullsized":"2024-02-10T08:35:54.702631+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.699420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.699420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780284+0000","last_change":"2024-02-10T08:35:33.677439+0000","last_active":"2024-02-10T08:35:54.780284+0000","last_peered":"2024-02-10T08:35:54.780284+0000","last_clean":"2024-02-10T08:35:54.780284+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:35:54.780284+0000","last_undegraded":"2024-02-10T08:35:54.780284+0000","last_fullsized":"2024-02-10T08:35:54.780284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:35:33.677377+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106192+0000","last_change":"2024-02-10T08:34:45.519207+0000","last_active":"2024-02-10T08:35:54.106192+0000","last_peered":"2024-02-10T08:35:54.106192+0000","last_clean":"2024-02-10T08:35:54.106192+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:35:54.106192+0000","last_undegraded":"2024-02-10T08:35:54.106192+0000","last_fullsized":"2024-02-10T08:35:54.106192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:45.519116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:45.519116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102826+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:35:54.102826+0000","last_peered":"2024-02-10T08:35:54.102826+0000","last_clean":"2024-02-10T08:35:54.102826+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:35:54.102826+0000","last_undegraded":"2024-02-10T08:35:54.102826+0000","last_fullsized":"2024-02-10T08:35:54.102826+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102137+0000","last_change":"2024-02-10T08:34:54.039656+0000","last_active":"2024-02-10T08:35:54.102137+0000","last_peered":"2024-02-10T08:35:54.102137+0000","last_clean":"2024-02-10T08:35:54.102137+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:35:54.102137+0000","last_undegraded":"2024-02-10T08:35:54.102137+0000","last_fullsized":"2024-02-10T08:35:54.102137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.039584+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702403+0000","last_change":"2024-02-10T08:34:39.590686+0000","last_active":"2024-02-10T08:35:54.702403+0000","last_peered":"2024-02-10T08:35:54.702403+0000","last_clean":"2024-02-10T08:35:54.702403+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:35:54.702403+0000","last_undegraded":"2024-02-10T08:35:54.702403+0000","last_fullsized":"2024-02-10T08:35:54.702403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:39.590604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.590604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779536+0000","last_change":"2024-02-10T08:34:31.544112+0000","last_active":"2024-02-10T08:35:54.779536+0000","last_peered":"2024-02-10T08:35:54.779536+0000","last_clean":"2024-02-10T08:35:54.779536+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:35:54.779536+0000","last_undegraded":"2024-02-10T08:35:54.779536+0000","last_fullsized":"2024-02-10T08:35:54.779536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:31.544049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:31.544049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101382+0000","last_change":"2024-02-10T08:34:46.452573+0000","last_active":"2024-02-10T08:35:54.101382+0000","last_peered":"2024-02-10T08:35:54.101382+0000","last_clean":"2024-02-10T08:35:54.101382+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:35:54.101382+0000","last_undegraded":"2024-02-10T08:35:54.101382+0000","last_fullsized":"2024-02-10T08:35:54.101382+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:46.452504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:46.452504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101681+0000","last_change":"2024-02-10T08:35:34.249489+0000","last_active":"2024-02-10T08:35:54.101681+0000","last_peered":"2024-02-10T08:35:54.101681+0000","last_clean":"2024-02-10T08:35:54.101681+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:35:54.101681+0000","last_undegraded":"2024-02-10T08:35:54.101681+0000","last_fullsized":"2024-02-10T08:35:54.101681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.249424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521170+0000","last_change":"2024-02-10T08:34:47.693139+0000","last_active":"2024-02-10T08:36:01.521170+0000","last_peered":"2024-02-10T08:36:01.521170+0000","last_clean":"2024-02-10T08:36:01.521170+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:36:01.521170+0000","last_undegraded":"2024-02-10T08:36:01.521170+0000","last_fullsized":"2024-02-10T08:36:01.521170+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:47.693030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.693030+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106696+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:35:54.106696+0000","last_peered":"2024-02-10T08:35:54.106696+0000","last_clean":"2024-02-10T08:35:54.106696+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:35:54.106696+0000","last_undegraded":"2024-02-10T08:35:54.106696+0000","last_fullsized":"2024-02-10T08:35:54.106696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615689+0000","last_change":"2024-02-10T08:35:42.473094+0000","last_active":"2024-02-10T08:35:54.615689+0000","last_peered":"2024-02-10T08:35:54.615689+0000","last_clean":"2024-02-10T08:35:54.615689+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:35:54.615689+0000","last_undegraded":"2024-02-10T08:35:54.615689+0000","last_fullsized":"2024-02-10T08:35:54.615689+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.473021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521484+0000","last_change":"2024-02-10T08:34:47.549753+0000","last_active":"2024-02-10T08:36:01.521484+0000","last_peered":"2024-02-10T08:36:01.521484+0000","last_clean":"2024-02-10T08:36:01.521484+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:36:01.521484+0000","last_undegraded":"2024-02-10T08:36:01.521484+0000","last_fullsized":"2024-02-10T08:36:01.521484+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:34:47.549671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:47.549671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779391+0000","last_change":"2024-02-10T08:35:00.639772+0000","last_active":"2024-02-10T08:35:54.779391+0000","last_peered":"2024-02-10T08:35:54.779391+0000","last_clean":"2024-02-10T08:35:54.779391+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:35:54.779391+0000","last_undegraded":"2024-02-10T08:35:54.779391+0000","last_fullsized":"2024-02-10T08:35:54.779391+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.639705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.639705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.090002+0000","last_change":"2024-02-10T08:34:51.910276+0000","last_active":"2024-02-10T08:35:55.090002+0000","last_peered":"2024-02-10T08:35:55.090002+0000","last_clean":"2024-02-10T08:35:55.090002+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:35:55.090002+0000","last_undegraded":"2024-02-10T08:35:55.090002+0000","last_fullsized":"2024-02-10T08:35:55.090002+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.910184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":218,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:09.355130+0000","last_change":"2024-02-10T08:35:07.227647+0000","last_active":"2024-02-10T08:36:09.355130+0000","last_peered":"2024-02-10T08:36:09.355130+0000","last_clean":"2024-02-10T08:36:09.355130+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:36:09.355130+0000","last_undegraded":"2024-02-10T08:36:09.355130+0000","last_fullsized":"2024-02-10T08:36:09.355130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:35:07.227587+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.227587+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":166,"num_read_kb":161,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702649+0000","last_change":"2024-02-10T08:34:52.698430+0000","last_active":"2024-02-10T08:35:54.702649+0000","last_peered":"2024-02-10T08:35:54.702649+0000","last_clean":"2024-02-10T08:35:54.702649+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:35:54.702649+0000","last_undegraded":"2024-02-10T08:35:54.702649+0000","last_fullsized":"2024-02-10T08:35:54.702649+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.698329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102141+0000","last_change":"2024-02-10T08:35:42.326705+0000","last_active":"2024-02-10T08:35:54.102141+0000","last_peered":"2024-02-10T08:35:54.102141+0000","last_clean":"2024-02-10T08:35:54.102141+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:35:54.102141+0000","last_undegraded":"2024-02-10T08:35:54.102141+0000","last_fullsized":"2024-02-10T08:35:54.102141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:35:42.326627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.326627+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702991+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:35:54.702991+0000","last_peered":"2024-02-10T08:35:54.702991+0000","last_clean":"2024-02-10T08:35:54.702991+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:35:54.702991+0000","last_undegraded":"2024-02-10T08:35:54.702991+0000","last_fullsized":"2024-02-10T08:35:54.702991+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.616314+0000","last_change":"2024-02-10T08:34:55.443465+0000","last_active":"2024-02-10T08:35:54.616314+0000","last_peered":"2024-02-10T08:35:54.616314+0000","last_clean":"2024-02-10T08:35:54.616314+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:35:54.616314+0000","last_undegraded":"2024-02-10T08:35:54.616314+0000","last_fullsized":"2024-02-10T08:35:54.616314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.443390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.443390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105917+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:35:54.105917+0000","last_peered":"2024-02-10T08:35:54.105917+0000","last_clean":"2024-02-10T08:35:54.105917+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:35:54.105917+0000","last_undegraded":"2024-02-10T08:35:54.105917+0000","last_fullsized":"2024-02-10T08:35:54.105917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615888+0000","last_change":"2024-02-10T08:34:52.491498+0000","last_active":"2024-02-10T08:35:54.615888+0000","last_peered":"2024-02-10T08:35:54.615888+0000","last_clean":"2024-02-10T08:35:54.615888+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:35:54.615888+0000","last_undegraded":"2024-02-10T08:35:54.615888+0000","last_fullsized":"2024-02-10T08:35:54.615888+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:52.491420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:52.491420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089548+0000","last_change":"2024-02-10T08:34:48.841104+0000","last_active":"2024-02-10T08:35:55.089548+0000","last_peered":"2024-02-10T08:35:55.089548+0000","last_clean":"2024-02-10T08:35:55.089548+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:35:55.089548+0000","last_undegraded":"2024-02-10T08:35:55.089548+0000","last_fullsized":"2024-02-10T08:35:55.089548+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:48.841025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:34:48.841025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.518135+0000","last_change":"2024-02-10T08:34:39.378393+0000","last_active":"2024-02-10T08:35:54.518135+0000","last_peered":"2024-02-10T08:35:54.518135+0000","last_clean":"2024-02-10T08:35:54.518135+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:35:54.518135+0000","last_undegraded":"2024-02-10T08:35:54.518135+0000","last_fullsized":"2024-02-10T08:35:54.518135+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:39.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.378312+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.090077+0000","last_change":"2024-02-10T08:34:40.809042+0000","last_active":"2024-02-10T08:35:55.090077+0000","last_peered":"2024-02-10T08:35:55.090077+0000","last_clean":"2024-02-10T08:35:55.090077+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:35:55.090077+0000","last_undegraded":"2024-02-10T08:35:55.090077+0000","last_fullsized":"2024-02-10T08:35:55.090077+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:40.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:40.808978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101242+0000","last_change":"2024-02-10T08:35:38.321759+0000","last_active":"2024-02-10T08:35:54.101242+0000","last_peered":"2024-02-10T08:35:54.101242+0000","last_clean":"2024-02-10T08:35:54.101242+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:35:54.101242+0000","last_undegraded":"2024-02-10T08:35:54.101242+0000","last_fullsized":"2024-02-10T08:35:54.101242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:38.321694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.321694+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779762+0000","last_change":"2024-02-10T08:35:41.795140+0000","last_active":"2024-02-10T08:35:54.779762+0000","last_peered":"2024-02-10T08:35:54.779762+0000","last_clean":"2024-02-10T08:35:54.779762+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:35:54.779762+0000","last_undegraded":"2024-02-10T08:35:54.779762+0000","last_fullsized":"2024-02-10T08:35:54.779762+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089414+0000","last_change":"2024-02-10T08:35:27.051918+0000","last_active":"2024-02-10T08:35:55.089414+0000","last_peered":"2024-02-10T08:35:55.089414+0000","last_clean":"2024-02-10T08:35:55.089414+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:35:55.089414+0000","last_undegraded":"2024-02-10T08:35:55.089414+0000","last_fullsized":"2024-02-10T08:35:55.089414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:27.051837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:27.051837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780195+0000","last_change":"2024-02-10T08:35:36.647820+0000","last_active":"2024-02-10T08:35:54.780195+0000","last_peered":"2024-02-10T08:35:54.780195+0000","last_clean":"2024-02-10T08:35:54.780195+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:35:54.780195+0000","last_undegraded":"2024-02-10T08:35:54.780195+0000","last_fullsized":"2024-02-10T08:35:54.780195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:35:36.647719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.647719+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615406+0000","last_change":"2024-02-10T08:34:38.628666+0000","last_active":"2024-02-10T08:35:54.615406+0000","last_peered":"2024-02-10T08:35:54.615406+0000","last_clean":"2024-02-10T08:35:54.615406+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:35:54.615406+0000","last_undegraded":"2024-02-10T08:35:54.615406+0000","last_fullsized":"2024-02-10T08:35:54.615406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:38.628559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:38.628559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.100959+0000","last_change":"2024-02-10T08:34:44.434673+0000","last_active":"2024-02-10T08:35:54.100959+0000","last_peered":"2024-02-10T08:35:54.100959+0000","last_clean":"2024-02-10T08:35:54.100959+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:35:54.100959+0000","last_undegraded":"2024-02-10T08:35:54.100959+0000","last_fullsized":"2024-02-10T08:35:54.100959+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.434603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.434603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.090097+0000","last_change":"2024-02-10T08:34:44.810962+0000","last_active":"2024-02-10T08:35:55.090097+0000","last_peered":"2024-02-10T08:35:55.090097+0000","last_clean":"2024-02-10T08:35:55.090097+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:35:55.090097+0000","last_undegraded":"2024-02-10T08:35:55.090097+0000","last_fullsized":"2024-02-10T08:35:55.090097+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:44.810886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.810886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780045+0000","last_change":"2024-02-10T08:34:39.703715+0000","last_active":"2024-02-10T08:35:54.780045+0000","last_peered":"2024-02-10T08:35:54.780045+0000","last_clean":"2024-02-10T08:35:54.780045+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:35:54.780045+0000","last_undegraded":"2024-02-10T08:35:54.780045+0000","last_fullsized":"2024-02-10T08:35:54.780045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:39.703643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:39.703643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517900+0000","last_change":"2024-02-10T08:34:51.348414+0000","last_active":"2024-02-10T08:35:54.517900+0000","last_peered":"2024-02-10T08:35:54.517900+0000","last_clean":"2024-02-10T08:35:54.517900+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:35:54.517900+0000","last_undegraded":"2024-02-10T08:35:54.517900+0000","last_fullsized":"2024-02-10T08:35:54.517900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:51.348334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.348334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.103044+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:35:54.103044+0000","last_peered":"2024-02-10T08:35:54.103044+0000","last_clean":"2024-02-10T08:35:54.103044+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:35:54.103044+0000","last_undegraded":"2024-02-10T08:35:54.103044+0000","last_fullsized":"2024-02-10T08:35:54.103044+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102270+0000","last_change":"2024-02-10T08:35:03.147093+0000","last_active":"2024-02-10T08:35:54.102270+0000","last_peered":"2024-02-10T08:35:54.102270+0000","last_clean":"2024-02-10T08:35:54.102270+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:35:54.102270+0000","last_undegraded":"2024-02-10T08:35:54.102270+0000","last_fullsized":"2024-02-10T08:35:54.102270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:35:03.146990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.146990+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.105585+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:35:54.105585+0000","last_peered":"2024-02-10T08:35:54.105585+0000","last_clean":"2024-02-10T08:35:54.105585+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:35:54.105585+0000","last_undegraded":"2024-02-10T08:35:54.105585+0000","last_fullsized":"2024-02-10T08:35:54.105585+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102421+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:35:54.102421+0000","last_peered":"2024-02-10T08:35:54.102421+0000","last_clean":"2024-02-10T08:35:54.102421+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:35:54.102421+0000","last_undegraded":"2024-02-10T08:35:54.102421+0000","last_fullsized":"2024-02-10T08:35:54.102421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.090053+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:35:55.090053+0000","last_peered":"2024-02-10T08:35:55.090053+0000","last_clean":"2024-02-10T08:35:55.090053+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:35:55.090053+0000","last_undegraded":"2024-02-10T08:35:55.090053+0000","last_fullsized":"2024-02-10T08:35:55.090053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.518218+0000","last_change":"2024-02-10T08:35:38.547170+0000","last_active":"2024-02-10T08:35:54.518218+0000","last_peered":"2024-02-10T08:35:54.518218+0000","last_clean":"2024-02-10T08:35:54.518218+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:35:54.518218+0000","last_undegraded":"2024-02-10T08:35:54.518218+0000","last_fullsized":"2024-02-10T08:35:54.518218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:38.547094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.547094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102195+0000","last_change":"2024-02-10T08:34:36.155885+0000","last_active":"2024-02-10T08:35:54.102195+0000","last_peered":"2024-02-10T08:35:54.102195+0000","last_clean":"2024-02-10T08:35:54.102195+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:35:54.102195+0000","last_undegraded":"2024-02-10T08:35:54.102195+0000","last_fullsized":"2024-02-10T08:35:54.102195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:34:36.155806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.155806+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521186+0000","last_change":"2024-02-10T08:34:36.649251+0000","last_active":"2024-02-10T08:36:01.521186+0000","last_peered":"2024-02-10T08:36:01.521186+0000","last_clean":"2024-02-10T08:36:01.521186+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:36:01.521186+0000","last_undegraded":"2024-02-10T08:36:01.521186+0000","last_fullsized":"2024-02-10T08:36:01.521186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:36.649186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:36.649186+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.106309+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:35:54.106309+0000","last_peered":"2024-02-10T08:35:54.106309+0000","last_clean":"2024-02-10T08:35:54.106309+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:35:54.106309+0000","last_undegraded":"2024-02-10T08:35:54.106309+0000","last_fullsized":"2024-02-10T08:35:54.106309+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615378+0000","last_change":"2024-02-10T08:35:06.520594+0000","last_active":"2024-02-10T08:35:54.615378+0000","last_peered":"2024-02-10T08:35:54.615378+0000","last_clean":"2024-02-10T08:35:54.615378+0000","last_became_active":"2024-02-10T08:33:22.122087+0000","last_became_peered":"2024-02-10T08:33:22.122087+0000","last_unstale":"2024-02-10T08:35:54.615378+0000","last_undegraded":"2024-02-10T08:35:54.615378+0000","last_fullsized":"2024-02-10T08:35:54.615378+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089533+0000","last_change":"2024-02-10T08:34:29.780708+0000","last_active":"2024-02-10T08:35:55.089533+0000","last_peered":"2024-02-10T08:35:55.089533+0000","last_clean":"2024-02-10T08:35:55.089533+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:35:55.089533+0000","last_undegraded":"2024-02-10T08:35:55.089533+0000","last_fullsized":"2024-02-10T08:35:55.089533+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:29.780644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:29.780644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102805+0000","last_change":"2024-02-10T08:34:25.085734+0000","last_active":"2024-02-10T08:35:54.102805+0000","last_peered":"2024-02-10T08:35:54.102805+0000","last_clean":"2024-02-10T08:35:54.102805+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:35:54.102805+0000","last_undegraded":"2024-02-10T08:35:54.102805+0000","last_fullsized":"2024-02-10T08:35:54.102805+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:34:25.085663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:25.085663+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.616007+0000","last_change":"2024-02-10T08:35:16.510089+0000","last_active":"2024-02-10T08:35:54.616007+0000","last_peered":"2024-02-10T08:35:54.616007+0000","last_clean":"2024-02-10T08:35:54.616007+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:35:54.616007+0000","last_undegraded":"2024-02-10T08:35:54.616007+0000","last_fullsized":"2024-02-10T08:35:54.616007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521490+0000","last_change":"2024-02-10T08:35:11.026984+0000","last_active":"2024-02-10T08:36:01.521490+0000","last_peered":"2024-02-10T08:36:01.521490+0000","last_clean":"2024-02-10T08:36:01.521490+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:36:01.521490+0000","last_undegraded":"2024-02-10T08:36:01.521490+0000","last_fullsized":"2024-02-10T08:36:01.521490+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101111+0000","last_change":"2024-02-10T08:34:32.208396+0000","last_active":"2024-02-10T08:35:54.101111+0000","last_peered":"2024-02-10T08:35:54.101111+0000","last_clean":"2024-02-10T08:35:54.101111+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:35:54.101111+0000","last_undegraded":"2024-02-10T08:35:54.101111+0000","last_fullsized":"2024-02-10T08:35:54.101111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:34:32.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:32.208313+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102098+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:35:54.102098+0000","last_peered":"2024-02-10T08:35:54.102098+0000","last_clean":"2024-02-10T08:35:54.102098+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:35:54.102098+0000","last_undegraded":"2024-02-10T08:35:54.102098+0000","last_fullsized":"2024-02-10T08:35:54.102098+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102571+0000","last_change":"2024-02-10T08:34:35.201454+0000","last_active":"2024-02-10T08:35:54.102571+0000","last_peered":"2024-02-10T08:35:54.102571+0000","last_clean":"2024-02-10T08:35:54.102571+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:35:54.102571+0000","last_undegraded":"2024-02-10T08:35:54.102571+0000","last_fullsized":"2024-02-10T08:35:54.102571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:35.201371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.201371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521616+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:36:01.521616+0000","last_peered":"2024-02-10T08:36:01.521616+0000","last_clean":"2024-02-10T08:36:01.521616+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:36:01.521616+0000","last_undegraded":"2024-02-10T08:36:01.521616+0000","last_fullsized":"2024-02-10T08:36:01.521616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101550+0000","last_change":"2024-02-10T08:35:31.222395+0000","last_active":"2024-02-10T08:35:54.101550+0000","last_peered":"2024-02-10T08:35:54.101550+0000","last_clean":"2024-02-10T08:35:54.101550+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:35:54.101550+0000","last_undegraded":"2024-02-10T08:35:54.101550+0000","last_fullsized":"2024-02-10T08:35:54.101550+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:31.222331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:31.222331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102300+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:35:54.102300+0000","last_peered":"2024-02-10T08:35:54.102300+0000","last_clean":"2024-02-10T08:35:54.102300+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:35:54.102300+0000","last_undegraded":"2024-02-10T08:35:54.102300+0000","last_fullsized":"2024-02-10T08:35:54.102300+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.703064+0000","last_change":"2024-02-10T08:34:53.743882+0000","last_active":"2024-02-10T08:35:54.703064+0000","last_peered":"2024-02-10T08:35:54.703064+0000","last_clean":"2024-02-10T08:35:54.703064+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:35:54.703064+0000","last_undegraded":"2024-02-10T08:35:54.703064+0000","last_fullsized":"2024-02-10T08:35:54.703064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:53.743807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.743807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.101285+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:35:54.101285+0000","last_peered":"2024-02-10T08:35:54.101285+0000","last_clean":"2024-02-10T08:35:54.101285+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:35:54.101285+0000","last_undegraded":"2024-02-10T08:35:54.101285+0000","last_fullsized":"2024-02-10T08:35:54.101285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089673+0000","last_change":"2024-02-10T08:35:30.123918+0000","last_active":"2024-02-10T08:35:55.089673+0000","last_peered":"2024-02-10T08:35:55.089673+0000","last_clean":"2024-02-10T08:35:55.089673+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:35:55.089673+0000","last_undegraded":"2024-02-10T08:35:55.089673+0000","last_fullsized":"2024-02-10T08:35:55.089673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.123828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.123828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102726+0000","last_change":"2024-02-10T08:35:30.802385+0000","last_active":"2024-02-10T08:35:54.102726+0000","last_peered":"2024-02-10T08:35:54.102726+0000","last_clean":"2024-02-10T08:35:54.102726+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:35:54.102726+0000","last_undegraded":"2024-02-10T08:35:54.102726+0000","last_fullsized":"2024-02-10T08:35:54.102726+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:35:30.802306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.802306+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:55.089396+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:35:55.089396+0000","last_peered":"2024-02-10T08:35:55.089396+0000","last_clean":"2024-02-10T08:35:55.089396+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:35:55.089396+0000","last_undegraded":"2024-02-10T08:35:55.089396+0000","last_fullsized":"2024-02-10T08:35:55.089396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702905+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:35:54.702905+0000","last_peered":"2024-02-10T08:35:54.702905+0000","last_clean":"2024-02-10T08:35:54.702905+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:35:54.702905+0000","last_undegraded":"2024-02-10T08:35:54.702905+0000","last_fullsized":"2024-02-10T08:35:54.702905+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.703041+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:35:54.703041+0000","last_peered":"2024-02-10T08:35:54.703041+0000","last_clean":"2024-02-10T08:35:54.703041+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:35:54.703041+0000","last_undegraded":"2024-02-10T08:35:54.703041+0000","last_fullsized":"2024-02-10T08:35:54.703041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.517814+0000","last_change":"2024-02-10T08:34:44.381291+0000","last_active":"2024-02-10T08:35:54.517814+0000","last_peered":"2024-02-10T08:35:54.517814+0000","last_clean":"2024-02-10T08:35:54.517814+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:35:54.517814+0000","last_undegraded":"2024-02-10T08:35:54.517814+0000","last_fullsized":"2024-02-10T08:35:54.517814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:34:44.381208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:44.381208+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.779570+0000","last_change":"2024-02-10T08:35:34.652671+0000","last_active":"2024-02-10T08:35:54.779570+0000","last_peered":"2024-02-10T08:35:54.779570+0000","last_clean":"2024-02-10T08:35:54.779570+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:35:54.779570+0000","last_undegraded":"2024-02-10T08:35:54.779570+0000","last_fullsized":"2024-02-10T08:35:54.779570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:34.652551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.652551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.702411+0000","last_change":"2024-02-10T08:34:50.747486+0000","last_active":"2024-02-10T08:35:54.702411+0000","last_peered":"2024-02-10T08:35:54.702411+0000","last_clean":"2024-02-10T08:35:54.702411+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:35:54.702411+0000","last_undegraded":"2024-02-10T08:35:54.702411+0000","last_fullsized":"2024-02-10T08:35:54.702411+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:50.747424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.747424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.102030+0000","last_change":"2024-02-10T08:35:30.241737+0000","last_active":"2024-02-10T08:35:54.102030+0000","last_peered":"2024-02-10T08:35:54.102030+0000","last_clean":"2024-02-10T08:35:54.102030+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:35:54.102030+0000","last_undegraded":"2024-02-10T08:35:54.102030+0000","last_fullsized":"2024-02-10T08:35:54.102030+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.241661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.241661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.780137+0000","last_change":"2024-02-10T08:34:41.701563+0000","last_active":"2024-02-10T08:35:54.780137+0000","last_peered":"2024-02-10T08:35:54.780137+0000","last_clean":"2024-02-10T08:35:54.780137+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:35:54.780137+0000","last_undegraded":"2024-02-10T08:35:54.780137+0000","last_fullsized":"2024-02-10T08:35:54.780137+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:34:34.592074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:34.592074+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615601+0000","last_change":"2024-02-10T08:34:56.475601+0000","last_active":"2024-02-10T08:35:54.615601+0000","last_peered":"2024-02-10T08:35:54.615601+0000","last_clean":"2024-02-10T08:35:54.615601+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:35:54.615601+0000","last_undegraded":"2024-02-10T08:35:54.615601+0000","last_fullsized":"2024-02-10T08:35:54.615601+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.475529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.475529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.703066+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:35:54.703066+0000","last_peered":"2024-02-10T08:35:54.703066+0000","last_clean":"2024-02-10T08:35:54.703066+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:35:54.703066+0000","last_undegraded":"2024-02-10T08:35:54.703066+0000","last_fullsized":"2024-02-10T08:35:54.703066+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T08:35:54.615608+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:35:54.615608+0000","last_peered":"2024-02-10T08:35:54.615608+0000","last_clean":"2024-02-10T08:35:54.615608+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:35:54.615608+0000","last_undegraded":"2024-02-10T08:35:54.615608+0000","last_fullsized":"2024-02-10T08:35:54.615608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-10T08:36:01.521657+0000","last_change":"2024-02-10T08:34:42.752203+0000","last_active":"2024-02-10T08:36:01.521657+0000","last_peered":"2024-02-10T08:36:01.521657+0000","last_clean":"2024-02-10T08:36:01.521657+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:36:01.521657+0000","last_undegraded":"2024-02-10T08:36:01.521657+0000","last_fullsized":"2024-02-10T08:36:01.521657+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:42.752097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:42.752097+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":166,"num_read_kb":161,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297796,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447356,"statfs":{"total":95995035648,"available":95690092544,"internally_reserved":0,"allocated":716800,"data_stored":333131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.034,"15min":1.034},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.054,"15min":1.054},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.496,"15min":0.499},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.5}]},{"osd":3,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42499999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.443}]},{"osd":5,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.747,"15min":0.747},"last":0.52300000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561067,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297776,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447376,"statfs":{"total":95995035648,"available":95690113024,"internally_reserved":0,"allocated":696320,"data_stored":333039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.36299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47399999999999998,"15min":0.47299999999999998},"min":{"1min":0.35699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.437}]},{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.315,"5min":0.248,"15min":0.248},"max":{"1min":0.60899999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.501,"15min":0.501},"min":{"1min":0.39400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66600000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.40200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.34699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39400000000000002,"15min":0.378},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.63},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.39600000000000002,"15min":0.372},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.436,"15min":0.40400000000000003},"min":{"1min":0.32600000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67100000000000004,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.623}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297816,"kb_used_data":720,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447336,"statfs":{"total":95995035648,"available":95690072064,"internally_reserved":0,"allocated":737280,"data_stored":334451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.20799999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46600000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41099999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36799999999999999,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.44800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.309,"5min":0.36199999999999999,"15min":0.36499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":0.624,"15min":0.624},"last":0.82399999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.441,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.42199999999999999,"15min":0.44600000000000001},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.45300000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.44800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.73899999999999999,"5min":1.294,"15min":1.294},"last":0.90000000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.66700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.61899999999999999},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81499999999999995,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.95399999999999996}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.873},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.47099999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.749}]},{"osd":7,"last update":"Sat Feb 10 08:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.93300000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":332155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34899999999999998,"15min":0.30499999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.35599999999999998,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48299999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.36199999999999999,"15min":0.36299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48199999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.505},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46400000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.48399999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39800000000000002,"15min":0.41599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.624,"15min":0.624},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40999999999999998,"15min":0.42999999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.49299999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79400000000000004,"5min":0.878,"15min":0.878},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78000000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40100000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.875,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.42299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.747,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84299999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79500000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.61499999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297784,"kb_used_data":688,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447368,"statfs":{"total":95995035648,"available":95690104832,"internally_reserved":0,"allocated":704512,"data_stored":332834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.193,"5min":0.187,"15min":0.187},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.629},{"interface":"front","average":{"1min":0.312,"5min":0.35399999999999998,"15min":0.36099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48099999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35699999999999998,"15min":0.36099999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.41699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.317},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.38200000000000001,"15min":0.38300000000000001},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.45700000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.46999999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40400000000000003,"15min":0.42799999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77800000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.62}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82499999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82099999999999995,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.51300000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.621,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.54100000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297816,"kb_used_data":720,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447336,"statfs":{"total":95995035648,"available":95690072064,"internally_reserved":0,"allocated":737280,"data_stored":333327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34200000000000003,"15min":0.36399999999999999},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36099999999999999,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.375,"15min":0.41199999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.45300000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.41099999999999998},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.63500000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79500000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93400000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.42699999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.755}]},{"osd":7,"last update":"Sat Feb 10 08:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.71599999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986228,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297800,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447352,"statfs":{"total":95995035648,"available":95690088448,"internally_reserved":0,"allocated":720896,"data_stored":333497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.35199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56599999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.28100000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":1,"15min":1},"last":0.499},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51700000000000002,"15min":0.54200000000000004},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.55200000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63800000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":2.113}]},{"osd":3,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60899999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72099999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72099999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.378},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65800000000000003,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.40300000000000002,"15min":0.38700000000000001},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67200000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41599999999999998,"15min":0.39900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66500000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.34000000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71099999999999997,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.60099999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.626}]}]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297792,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447360,"statfs":{"total":95995035648,"available":95690096640,"internally_reserved":0,"allocated":712704,"data_stored":332473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.51600000000000001,"15min":0.52400000000000002},"min":{"1min":0.35899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68000000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68600000000000005,"5min":0.749,"15min":0.749},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.32700000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66100000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.41699999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70499999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.42699999999999999,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.432},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.439,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69899999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.374},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72299999999999998,"5min":1.081,"15min":1.081},"last":0.39500000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48499999999999999,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:36:15.422 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-10T08:36:15.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 22 2024-02-10T08:36:15.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:15.544 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:15.565 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:15.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:15.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:15.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.801108+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:36:15.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:13.802260+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:36:15.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.026172+0000 mgr.y (mgr.14553) 121 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:15.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.033293+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.035349+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.613242+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.616135+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.625083+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.746127+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.747513+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.824076+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.b scrub starts 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:14.826313+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.b scrub ok 2024-02-10T08:36:15.682 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:15] "GET /metrics HTTP/1.1" 200 172237 "" "Prometheus/2.33.4" 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.801108+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:13.802260+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.026172+0000 mgr.y (mgr.14553) 121 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.033293+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.035349+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.613242+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.616135+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.625083+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.746127+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.747513+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.824076+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.b scrub starts 2024-02-10T08:36:15.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:14.826313+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.b scrub ok 2024-02-10T08:36:15.696 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:36:15.696 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:36:15.696 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:36:15.696 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:36:15.708 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:15.916 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:15.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:15.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 -- 172.21.15.107:0/1784672025 >> v1:172.21.15.107:6789/0 conn(0x7fd3140ff740 legacy=0x7fd314101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:15.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 -- 172.21.15.107:0/1784672025 shutdown_connections 2024-02-10T08:36:15.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 -- 172.21.15.107:0/1784672025 >> 172.21.15.107:0/1784672025 conn(0x7fd31406fe70 msgr2=0x7fd3140f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:15.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 -- 172.21.15.107:0/1784672025 shutdown_connections 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 -- 172.21.15.107:0/1784672025 wait complete. 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 librados: starting msgr at 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.930+0000 7fd31b407700 1 librados: starting objecter 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 Processor -- start 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 -- start start 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 librados: setting wanted keys 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 librados: calling monclient init 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd314115470 con 0x7fd3140ff740 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd314116650 con 0x7fd3140fc950 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd31b407700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd314116840 con 0x7fd3141032b0 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd3189a2700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd3140ff740 0x7fd3140f8e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45098/0 (socket says 172.21.15.107:45098) 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd3189a2700 1 -- 172.21.15.107:0/4094250969 learned_addr learned my addr 172.21.15.107:0/4094250969 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd30a7fc700 10 client.?.objecter ms_handle_connect 0x7fd3140ff740 2024-02-10T08:36:15.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd30a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683977324 0 0) 0x7fd314115470 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.931+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3000033f0 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 10 client.?.objecter ms_handle_connect 0x7fd3141032b0 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 253532488 0 0) 0x7fd3000033f0 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd314115470 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd304002cb0 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232159524 0 0) 0x7fd314116840 con 0x7fd3141032b0 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3000033f0 con 0x7fd3141032b0 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622857992 0 0) 0x7fd314115470 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6790/0 conn(0x7fd3141032b0 legacy=0x7fd314113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.195:6789/0 conn(0x7fd3140fc950 legacy=0x7fd3140f8720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314116a30 con 0x7fd3140ff740 2024-02-10T08:36:15.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd314116ce0 con 0x7fd3140ff740 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd31b407700 10 client.14835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd314117150 con 0x7fd3140ff740 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.932+0000 7fd31b407700 1 librados: init done 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.933+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd3040024f0 con 0x7fd3140ff740 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.933+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd304003880 con 0x7fd3140ff740 2024-02-10T08:36:15.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd304003aa0 con 0x7fd3140ff740 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd2f8004f30 con 0x7fd3140ff740 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(92..92 src has 40..92) v4 ==== 6204+0+0 (unknown 2616155034 0 0) 0x7fd304078990 con 0x7fd3140ff740 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 osd_map(92..92 src has 40..92) v4 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 3 client.14835.objecter handle_osd_map got epochs [92,92] > 0 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 3 client.14835.objecter handle_osd_map decoding full epoch 92 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.934+0000 7fd30a7fc700 20 client.14835.objecter dump_active .. 0 homeless 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd300063c00 2024-02-10T08:36:15.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v92) v1 ==== 54+0+16 (unknown 457710080 0 2420000192) 0x7fd30404d3c0 con 0x7fd3140ff740 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 10 client.14835.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 1 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 1 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 0 1 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 4 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 10 client.14835.objecter _send_command 1 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.935+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f8002020 con 0x7fd2f8001be0 2024-02-10T08:36:15.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.936+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f8001be0 2024-02-10T08:36:15.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.937+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f8002020 con 0x7fd2f8001be0 2024-02-10T08:36:15.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.937+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:15.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.937+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 1 = system:0 2024-02-10T08:36:15.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.937+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 4 2024-02-10T08:36:15.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.937+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 0 1 2024-02-10T08:36:15.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 10 client.14835.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 2 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 2 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80051d0 osd=0 3 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 0 2 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 4 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 10 client.14835.objecter _send_command 2 2024-02-10T08:36:15.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.966+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f8002400 con 0x7fd2f8001be0 2024-02-10T08:36:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f8002400 con 0x7fd2f8001be0 2024-02-10T08:36:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 4 2024-02-10T08:36:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 0 2 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 10 client.14835.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 3 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 3 2024-02-10T08:36:15.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 1 3 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 4 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 10 client.14835.objecter _send_command 3 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f800bf20 con 0x7fd2f8001790 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.969+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f8001790 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.970+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f800bf20 con 0x7fd2f8001790 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.970+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.970+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 3 = system:0 2024-02-10T08:36:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.970+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 4 2024-02-10T08:36:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.970+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 1 3 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 10 client.14835.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 4 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 4 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8001620 osd=1 3 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 1 4 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 4 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 10 client.14835.objecter _send_command 4 2024-02-10T08:36:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:15.998+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f8004320 con 0x7fd2f8001790 2024-02-10T08:36:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f8004320 con 0x7fd2f8001790 2024-02-10T08:36:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 4 2024-02-10T08:36:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 1 4 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 10 client.14835.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 5 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 5 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 2 5 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 4 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 10 client.14835.objecter _send_command 5 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f800de00 con 0x7fd2f800d240 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.000+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f800d240 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.001+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f800de00 con 0x7fd2f800d240 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.001+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.001+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 5 = system:0 2024-02-10T08:36:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.001+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 4 2024-02-10T08:36:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.001+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 2 5 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 10 client.14835.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 6 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 6 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8003620 osd=2 3 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 2 6 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 4 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 10 client.14835.objecter _send_command 6 2024-02-10T08:36:16.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.016+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f8003150 con 0x7fd2f800d240 2024-02-10T08:36:16.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f8003150 con 0x7fd2f800d240 2024-02-10T08:36:16.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 4 2024-02-10T08:36:16.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 2 6 2024-02-10T08:36:16.020 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:16.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 10 client.14835.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 7 2024-02-10T08:36:16.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 7 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 3 7 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 4 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 10 client.14835.objecter _send_command 7 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f8015aa0 con 0x7fd2f8011f60 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.018+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f8011f60 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.019+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f8015aa0 con 0x7fd2f8011f60 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.019+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.019+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 7 = system:0 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.019+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 4 2024-02-10T08:36:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.019+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 3 7 2024-02-10T08:36:16.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.801108+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:13.802260+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.026172+0000 mgr.y (mgr.14553) 121 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.033293+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.035349+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.613242+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.16 scrub starts 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.616135+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.16 scrub ok 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.625083+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.746127+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:36:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.747513+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:36:16.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.824076+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.b scrub starts 2024-02-10T08:36:16.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:14.826313+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.b scrub ok 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 10 client.14835.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 8 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 8 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80118e0 osd=3 3 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 3 8 2024-02-10T08:36:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 4 2024-02-10T08:36:16.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 10 client.14835.objecter _send_command 8 2024-02-10T08:36:16.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.038+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f800e1b0 con 0x7fd2f8011f60 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.040+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f800e1b0 con 0x7fd2f8011f60 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.040+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.040+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.040+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 4 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.040+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 3 8 2024-02-10T08:36:16.042 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 10 client.14835.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 9 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 9 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 4 9 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 4 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 10 client.14835.objecter _send_command 9 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f800e0e0 con 0x7fd2f800cda0 2024-02-10T08:36:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.041+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f800cda0 2024-02-10T08:36:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.042+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f800e0e0 con 0x7fd2f800cda0 2024-02-10T08:36:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.042+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.042+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 9 = system:0 2024-02-10T08:36:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.042+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 4 2024-02-10T08:36:16.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.042+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 4 9 2024-02-10T08:36:16.052 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 10 client.14835.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 10 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 10 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f800df70 osd=4 3 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 4 10 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 4 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 10 client.14835.objecter _send_command 10 2024-02-10T08:36:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.059+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f800e0e0 con 0x7fd2f800cda0 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.061+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f800e0e0 con 0x7fd2f800cda0 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.061+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.061+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.061+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 4 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.061+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 4 10 2024-02-10T08:36:16.063 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 10 client.14835.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 11 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 11 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 5 11 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 4 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 10 client.14835.objecter _send_command 11 2024-02-10T08:36:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f8016e70 con 0x7fd2f80168a0 2024-02-10T08:36:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.063+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f80168a0 2024-02-10T08:36:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.064+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f8016e70 con 0x7fd2f80168a0 2024-02-10T08:36:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.064+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.064+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 11 = system:0 2024-02-10T08:36:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.064+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 4 2024-02-10T08:36:16.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.064+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 5 11 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 10 client.14835.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 12 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 12 2024-02-10T08:36:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f800e0e0 osd=5 3 2024-02-10T08:36:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 5 12 2024-02-10T08:36:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 4 2024-02-10T08:36:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 10 client.14835.objecter _send_command 12 2024-02-10T08:36:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.085+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f80170b0 con 0x7fd2f80168a0 2024-02-10T08:36:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f80170b0 con 0x7fd2f80168a0 2024-02-10T08:36:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 4 2024-02-10T08:36:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 5 12 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 10 client.14835.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 13 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 13 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 6 13 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 4 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 10 client.14835.objecter _send_command 13 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f8022c00 con 0x7fd2f801f080 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.087+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f801f080 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.088+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f8022c00 con 0x7fd2f801f080 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.088+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.088+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 13 = system:0 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.088+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 4 2024-02-10T08:36:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.088+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 6 13 2024-02-10T08:36:16.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 10 client.14835.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 14 2024-02-10T08:36:16.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 14 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f80170b0 osd=6 3 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 6 14 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 4 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 10 client.14835.objecter _send_command 14 2024-02-10T08:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.103+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f8022c00 con 0x7fd2f801f080 2024-02-10T08:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f8022c00 con 0x7fd2f801f080 2024-02-10T08:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 4 2024-02-10T08:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 6 14 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 10 client.14835.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 15 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 15 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 7 15 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 4 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 10 client.14835.objecter _send_command 15 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.105+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f8023e80 con 0x7fd2f8024900 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 10 client.14835.objecter ms_handle_connect 0x7fd2f8024900 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd2f8023e80 con 0x7fd2f8024900 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 15 = system:0 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 4 2024-02-10T08:36:16.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.106+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 7 15 2024-02-10T08:36:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 10 client.14835.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign -1 16 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 20 client.14835.objecter _get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 15 client.14835.objecter _session_command_op_remove -1 16 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 20 client.14835.objecter get_session s=0x7fd2f8022c00 osd=7 3 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 15 client.14835.objecter _session_command_op_assign 7 16 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 4 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 10 client.14835.objecter _send_command 16 2024-02-10T08:36:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.123+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2f8023e80 con 0x7fd2f8024900 2024-02-10T08:36:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd30a7fc700 1 -- 172.21.15.107:0/4094250969 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd2f8023e80 con 0x7fd2f8024900 2024-02-10T08:36:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd30a7fc700 10 client.14835.objecter ms_dispatch 0x7fd314001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd30a7fc700 10 client.14835.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd30a7fc700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 4 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd30a7fc700 15 client.14835.objecter _session_command_op_remove 7 16 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 librados: watch_flush enter 2024-02-10T08:36:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 librados: watch_flush exit 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.0 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6801/3321596972 conn(0x7fd2f8001be0 legacy=0x7fd2f8002740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80051d0 osd=0 1 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.1 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6805/133516126 conn(0x7fd2f8001790 legacy=0x7fd2f800a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8001620 osd=1 1 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.2 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6809/3575881728 conn(0x7fd2f800d240 legacy=0x7fd2f800d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8003620 osd=2 1 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.3 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6813/2871825691 conn(0x7fd2f8011f60 legacy=0x7fd2f8014350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80118e0 osd=3 1 2024-02-10T08:36:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.4 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.195:6800/2499671285 conn(0x7fd2f800cda0 legacy=0x7fd2f8015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800df70 osd=4 1 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.5 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.195:6804/4178321486 conn(0x7fd2f80168a0 legacy=0x7fd2f801d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f800e0e0 osd=5 1 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.6 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.195:6808/1417479017 conn(0x7fd2f801f080 legacy=0x7fd2f8021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f80170b0 osd=6 1 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter close_session for osd.7 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.195:6812/1934176928 conn(0x7fd2f8024900 legacy=0x7fd2f8024d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter put_session s=0x7fd2f8022c00 osd=7 1 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 20 client.14835.objecter shutdown clearing up homeless session... 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.125+0000 7fd31b407700 10 client.14835.objecter successfully canceled tick 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.126+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd300063c00 legacy=0x7fd3000660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.126+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> v1:172.21.15.107:6789/0 conn(0x7fd3140ff740 legacy=0x7fd3140f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd3199a4700 1 -- 172.21.15.107:0/4094250969 reap_dead start 2024-02-10T08:36:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd3199a4700 1 -- 172.21.15.107:0/4094250969 reap_dead start 2024-02-10T08:36:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd3199a4700 1 -- 172.21.15.107:0/4094250969 reap_dead start 2024-02-10T08:36:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd3199a4700 1 -- 172.21.15.107:0/4094250969 reap_dead start 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd3199a4700 1 -- 172.21.15.107:0/4094250969 reap_dead start 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 shutdown_connections 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.127+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 >> 172.21.15.107:0/4094250969 conn(0x7fd31406fe70 msgr2=0x7fd31406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.129+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 shutdown_connections 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.129+0000 7fd31b407700 1 -- 172.21.15.107:0/4094250969 wait complete. 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:16.129+0000 7fd31b407700 1 librados: shutdown 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.132 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:36:16.158 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:16.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:16.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:16.297 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:16.319 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:16.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:16.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:16.453 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:16.476 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:16.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:16.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:16.614 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.043692+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.045036+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.450123+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.453153+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.642078+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:36:16.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.644994+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:36:16.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.868360+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:36:16.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:15.871056+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:36:16.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:15.936607+0000 mon.a (mon.0) 742 : audit [DBG] from='client.? v1:172.21.15.107:0/4094250969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:16.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.166151+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:36:16.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.169613+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.043692+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.045036+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.450123+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.453153+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.642078+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.644994+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.868360+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:15.871056+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:36:16.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:15.936607+0000 mon.a (mon.0) 742 : audit [DBG] from='client.? v1:172.21.15.107:0/4094250969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:16.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.166151+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:36:16.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.169613+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:36:16.698 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:16.700 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:16.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:16.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:36:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:16.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:16.835 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:16.855 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:16.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:16.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.043692+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:36:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.045036+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:36:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.450123+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:36:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.453153+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.642078+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.644994+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.868360+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:15.871056+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:15.936607+0000 mon.a (mon.0) 742 : audit [DBG] from='client.? v1:172.21.15.107:0/4094250969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.166151+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:36:17.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.169613+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:36:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:17.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:17.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:17.218 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:17.343 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:17.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:17.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:17.485 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:17.506 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:17.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:17.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:17.660 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:17.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:17.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.026941+0000 mgr.y (mgr.14553) 122 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.420250+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:36:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.423782+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:36:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.545621+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:36:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:16.546667+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:36:17.792 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:17.811 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:17.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:17.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.026941+0000 mgr.y (mgr.14553) 122 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.420250+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.423782+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.545621+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:16.546667+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.026941+0000 mgr.y (mgr.14553) 122 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.420250+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:36:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.423782+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:36:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.545621+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:36:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:16.546667+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:36:17.948 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:18.104 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:18.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:18.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:18.245 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.564834+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.a scrub starts 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.568158+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.a scrub ok 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.568848+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.572471+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.727275+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.729244+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.891265+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:36:18.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:17.894331+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.564834+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.a scrub starts 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.568158+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.a scrub ok 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.568848+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.572471+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.727275+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.729244+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:36:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.891265+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:36:18.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:17.894331+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:36:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:19.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.564834+0000 osd.2 (osd.2) 37 : cluster [DBG] 5.a scrub starts 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.568158+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.a scrub ok 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.568848+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.572471+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.727275+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.729244+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.891265+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:36:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:17.894331+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:36:19.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:36:19.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:19.141 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:19.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:19.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:19.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:19.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:19.777 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:19.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:19.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:18.027280+0000 mgr.y (mgr.14553) 123 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:19.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:18.418219+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:36:19.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:18.421255+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:36:19.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:18.027280+0000 mgr.y (mgr.14553) 123 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:19.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:18.418219+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:36:19.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:18.421255+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:36:19.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.020 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:18.027280+0000 mgr.y (mgr.14553) 123 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:18.418219+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.19 scrub starts 2024-02-10T08:36:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:18.421255+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.19 scrub ok 2024-02-10T08:36:20.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:20.158 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.179 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.269+0000 7f3b20720700 1 -- 172.21.15.107:0/1783054938 >> v1:172.21.15.195:6789/0 conn(0x7f3b180feea0 legacy=0x7f3b18101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.270+0000 7f3b20720700 1 -- 172.21.15.107:0/1783054938 shutdown_connections 2024-02-10T08:36:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.270+0000 7f3b20720700 1 -- 172.21.15.107:0/1783054938 >> 172.21.15.107:0/1783054938 conn(0x7f3b1806fe70 msgr2=0x7f3b180f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.270+0000 7f3b20720700 1 -- 172.21.15.107:0/1783054938 shutdown_connections 2024-02-10T08:36:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.270+0000 7f3b20720700 1 -- 172.21.15.107:0/1783054938 wait complete. 2024-02-10T08:36:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 librados: starting msgr at 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 librados: starting objecter 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 Processor -- start 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 -- start start 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 librados: setting wanted keys 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 librados: calling monclient init 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b180fba20 con 0x7f3b180feea0 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b18117610 con 0x7f3b181029b0 2024-02-10T08:36:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b20720700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3b181187b0 con 0x7f3b180fc050 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b1e4bc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3b180fc050 0x7f3b180f7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53124/0 (socket says 172.21.15.107:53124) 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.271+0000 7f3b1e4bc700 1 -- 172.21.15.107:0/731133333 learned_addr learned my addr 172.21.15.107:0/731133333 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter ms_handle_connect 0x7f3b180fc050 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter ms_handle_connect 0x7f3b180feea0 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter ms_handle_connect 0x7f3b181029b0 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165993492 0 0) 0x7f3b181187b0 con 0x7f3b180fc050 2024-02-10T08:36:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04003110 con 0x7f3b180fc050 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292137624 0 0) 0x7f3b180fba20 con 0x7f3b180feea0 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b181187b0 con 0x7f3b180feea0 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377330411 0 0) 0x7f3b04003110 con 0x7f3b180fc050 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b180fba20 con 0x7f3b180fc050 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b08001980 con 0x7f3b180fc050 2024-02-10T08:36:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180294840 0 0) 0x7f3b18117610 con 0x7f3b181029b0 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04003110 con 0x7f3b181029b0 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 142899373 0 0) 0x7f3b180fba20 con 0x7f3b180fc050 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 >> v1:172.21.15.195:6789/0 conn(0x7f3b181029b0 legacy=0x7f3b18114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 >> v1:172.21.15.107:6789/0 conn(0x7f3b180feea0 legacy=0x7f3b180f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18119950 con 0x7f3b180fc050 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b18116660 con 0x7f3b180fc050 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b20720700 10 client.34754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b18116ad0 con 0x7f3b180fc050 2024-02-10T08:36:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.273+0000 7f3b20720700 1 librados: init done 2024-02-10T08:36:20.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.274+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3b08001c60 con 0x7f3b180fc050 2024-02-10T08:36:20.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.274+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3b08005bd0 con 0x7f3b180fc050 2024-02-10T08:36:20.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.275+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3b08018ac0 con 0x7f3b180fc050 2024-02-10T08:36:20.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.276+0000 7f3b0f7fe700 10 client.34754.objecter ms_handle_connect 0x7f3b04068050 2024-02-10T08:36:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.277+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(92..92 src has 40..92) v4 ==== 6204+0+0 (unknown 2616155034 0 0) 0x7f3b0804d310 con 0x7f3b180fc050 2024-02-10T08:36:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.277+0000 7f3b0f7fe700 10 client.34754.objecter ms_dispatch 0x7f3b18001550 osd_map(92..92 src has 40..92) v4 2024-02-10T08:36:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.277+0000 7f3b0f7fe700 3 client.34754.objecter handle_osd_map got epochs [92,92] > 0 2024-02-10T08:36:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.277+0000 7f3b0f7fe700 3 client.34754.objecter handle_osd_map decoding full epoch 92 2024-02-10T08:36:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.277+0000 7f3b0f7fe700 20 client.34754.objecter dump_active .. 0 homeless 2024-02-10T08:36:20.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.282+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc004f50 con 0x7f3b180fc050 2024-02-10T08:36:20.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.291+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3b0804d0e0 con 0x7f3b180fc050 2024-02-10T08:36:20.317 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.347 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:20.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:20.497 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.503 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 -- 172.21.15.107:0/1435810801 >> v1:172.21.15.195:6789/0 conn(0x7f5c64103320 legacy=0x7f5c641057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 -- 172.21.15.107:0/1435810801 shutdown_connections 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 -- 172.21.15.107:0/1435810801 >> 172.21.15.107:0/1435810801 conn(0x7f5c6406c090 msgr2=0x7f5c6406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 -- 172.21.15.107:0/1435810801 shutdown_connections 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 -- 172.21.15.107:0/1435810801 wait complete. 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 librados: starting msgr at 2024-02-10T08:36:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 librados: starting objecter 2024-02-10T08:36:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.510+0000 7f5c69d80700 1 Processor -- start 2024-02-10T08:36:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.511+0000 7f5c69d80700 1 -- start start 2024-02-10T08:36:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.511+0000 7f5c69d80700 1 librados: setting wanted keys 2024-02-10T08:36:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.511+0000 7f5c69d80700 1 librados: calling monclient init 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.515+0000 7f5c69d80700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c64119500 con 0x7f5c640ff720 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.515+0000 7f5c69d80700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c6411a6e0 con 0x7f5c64103320 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.515+0000 7f5c69d80700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c6411b900 con 0x7f5c640fc960 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.516+0000 7f5c62ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c640ff720 0x7f5c640f4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59672/0 (socket says 172.21.15.107:59672) 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.516+0000 7f5c62ffd700 1 -- 172.21.15.107:0/3085037502 learned_addr learned my addr 172.21.15.107:0/3085037502 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.517+0000 7f5c60ff9700 10 client.?.objecter ms_handle_connect 0x7f5c64103320 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.517+0000 7f5c60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.517+0000 7f5c60ff9700 10 client.?.objecter ms_handle_connect 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.517+0000 7f5c60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 10 client.?.objecter ms_handle_connect 0x7f5c640fc960 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952402852 0 0) 0x7f5c6411a6e0 con 0x7f5c64103320 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c50002220 con 0x7f5c64103320 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29629084 0 0) 0x7f5c64119500 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.518+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6411a6e0 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016459044 0 0) 0x7f5c6411b900 con 0x7f5c640fc960 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c64119500 con 0x7f5c640fc960 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 869916602 0 0) 0x7f5c6411a6e0 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c6411b900 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c540016b0 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4075371351 0 0) 0x7f5c6411b900 con 0x7f5c640ff720 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 >> v1:172.21.15.107:6790/0 conn(0x7f5c640fc960 legacy=0x7f5c640f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 >> v1:172.21.15.195:6789/0 conn(0x7f5c64103320 legacy=0x7f5c64117de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6411cb00 con 0x7f5c640ff720 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5c54001c20 con 0x7f5c640ff720 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.519+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5c54005930 con 0x7f5c640ff720 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.520+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c641196f0 con 0x7f5c640ff720 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.520+0000 7f5c69d80700 10 client.14844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.520+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c64119b60 con 0x7f5c640ff720 2024-02-10T08:36:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.520+0000 7f5c69d80700 1 librados: init done 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.524+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5c540187e0 con 0x7f5c640ff720 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.524+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c44004f50 con 0x7f5c640ff720 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.525+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(92..92 src has 40..92) v4 ==== 6204+0+0 (unknown 2616155034 0 0) 0x7f5c54078410 con 0x7f5c640ff720 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.525+0000 7f5c60ff9700 10 client.14844.objecter ms_dispatch 0x7f5c64001550 osd_map(92..92 src has 40..92) v4 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.525+0000 7f5c60ff9700 3 client.14844.objecter handle_osd_map got epochs [92,92] > 0 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.525+0000 7f5c60ff9700 3 client.14844.objecter handle_osd_map decoding full epoch 92 2024-02-10T08:36:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.525+0000 7f5c60ff9700 20 client.14844.objecter dump_active .. 0 homeless 2024-02-10T08:36:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.530+0000 7f5c60ff9700 10 client.14844.objecter ms_handle_connect 0x7f5c50063d40 2024-02-10T08:36:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.532+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5c5404cde0 con 0x7f5c640ff720 2024-02-10T08:36:20.536 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:20.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.592+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"} v 0) v1 -- 0x7f3afc005780 con 0x7f3b180fc050 2024-02-10T08:36:20.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:20.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.653+0000 7f3b0f7fe700 1 -- 172.21.15.107:0/731133333 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]=0 set pool 5 pg_num to 22 v93) v1 ==== 144+0+0 (unknown 2422341326 0 0) 0x7f3b0804b4f0 con 0x7f3b180fc050 2024-02-10T08:36:20.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:19.567158+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:36:20.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:19.570173+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:36:20.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.139457+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.142531+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:20.594810+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.107:0/731133333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:20.595601+0000 mon.a (mon.0) 743 : audit [INF] from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:19.567158+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:19.570173+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.139457+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.142531+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:20.594810+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.107:0/731133333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:20.595601+0000 mon.a (mon.0) 743 : audit [INF] from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.656+0000 7f3b20720700 10 librados: watch_flush enter 2024-02-10T08:36:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.657+0000 7f3b20720700 10 librados: watch_flush exit 2024-02-10T08:36:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.657+0000 7f3b20720700 20 client.34754.objecter shutdown clearing up homeless session... 2024-02-10T08:36:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.657+0000 7f3b20720700 10 client.34754.objecter successfully canceled tick 2024-02-10T08:36:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.657+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3b04068050 legacy=0x7f3b0406a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.657+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 >> v1:172.21.15.107:6790/0 conn(0x7f3b180fc050 legacy=0x7f3b180f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:20.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.658+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 shutdown_connections 2024-02-10T08:36:20.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.658+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 >> 172.21.15.107:0/731133333 conn(0x7f3b1806fe70 msgr2=0x7f3b1806c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.658+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 shutdown_connections 2024-02-10T08:36:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.658+0000 7f3b20720700 1 -- 172.21.15.107:0/731133333 wait complete. 2024-02-10T08:36:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.658+0000 7f3b20720700 1 librados: shutdown 2024-02-10T08:36:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 5 pg_num to 22 2024-02-10T08:36:20.676 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.697 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:20.788+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5c44005200 con 0x7f5c640ff720 2024-02-10T08:36:20.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:20.830 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:20.849 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:20.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:20.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:20.978 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:19.567158+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:36:20.979 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:19.570173+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:36:20.979 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.139457+0000 osd.7 (osd.7) 49 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:36:20.979 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.142531+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:36:20.979 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:20.594810+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.107:0/731133333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.979 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:20.595601+0000 mon.a (mon.0) 743 : audit [INF] from='client.34754 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-10T08:36:20.981 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:21.000 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:21.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:21.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:21.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:21.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.658+0000 7f5c60ff9700 1 -- 172.21.15.107:0/3085037502 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v94) v1 ==== 101+0+0 (unknown 2024750469 0 0) 0x7f5c5404b1f0 con 0x7f5c640ff720 2024-02-10T08:36:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.662+0000 7f5c69d80700 10 librados: watch_flush enter 2024-02-10T08:36:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.662+0000 7f5c69d80700 10 librados: watch_flush exit 2024-02-10T08:36:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.662+0000 7f5c69d80700 20 client.14844.objecter shutdown clearing up homeless session... 2024-02-10T08:36:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.662+0000 7f5c69d80700 10 client.14844.objecter successfully canceled tick 2024-02-10T08:36:21.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.668+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5c50063d40 legacy=0x7f5c50066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:21.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.668+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 >> v1:172.21.15.107:6789/0 conn(0x7f5c640ff720 legacy=0x7f5c640f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:21.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.672+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 shutdown_connections 2024-02-10T08:36:21.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.672+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 >> 172.21.15.107:0/3085037502 conn(0x7f5c6406c090 msgr2=0x7f5c6406d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:21.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.672+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 shutdown_connections 2024-02-10T08:36:21.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.673+0000 7f5c69d80700 1 -- 172.21.15.107:0/3085037502 wait complete. 2024-02-10T08:36:21.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:21.680+0000 7f5c69d80700 1 librados: shutdown 2024-02-10T08:36:21.682 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.027640+0000 mgr.y (mgr.14553) 124 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.045234+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.047268+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.495981+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.500304+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:36:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:20.651298+0000 mon.a (mon.0) 744 : audit [INF] from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.651370+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:20.793871+0000 mon.a (mon.0) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.876306+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:20.877261+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.106752+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.107576+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.027640+0000 mgr.y (mgr.14553) 124 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.045234+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.047268+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:36:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.495981+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.500304+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:20.651298+0000 mon.a (mon.0) 744 : audit [INF] from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.651370+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:20.793871+0000 mon.a (mon.0) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.876306+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:20.877261+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.106752+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:36:21.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.107576+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:36:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:21.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:21.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.027640+0000 mgr.y (mgr.14553) 124 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-10T08:36:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.045234+0000 osd.3 (osd.3) 59 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:36:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.047268+0000 osd.3 (osd.3) 60 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.495981+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.500304+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:20.651298+0000 mon.a (mon.0) 744 : audit [INF] from='client.34754 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.651370+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:20.793871+0000 mon.a (mon.0) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.876306+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:20.877261+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.106752+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:36:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.107576+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:36:22.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:22.136 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:22.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:21.128887+0000 mgr.y (mgr.14553) 125 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.648929+0000 mon.a (mon.0) 747 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:36:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.648951+0000 mon.a (mon.0) 748 : cluster [INF] Cluster is now healthy 2024-02-10T08:36:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:21.659955+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:22.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.660010+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.701881+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.a scrub starts 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:21.704769+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.a scrub ok 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:22.028267+0000 mon.c (mon.2) 84 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:22.028995+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:21.128887+0000 mgr.y (mgr.14553) 125 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.648929+0000 mon.a (mon.0) 747 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.648951+0000 mon.a (mon.0) 748 : cluster [INF] Cluster is now healthy 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:21.659955+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.660010+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.701881+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.a scrub starts 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:21.704769+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.a scrub ok 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:22.028267+0000 mon.c (mon.2) 84 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:22.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:22.028995+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:22.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:23.024 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:21.128887+0000 mgr.y (mgr.14553) 125 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.648929+0000 mon.a (mon.0) 747 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.648951+0000 mon.a (mon.0) 748 : cluster [INF] Cluster is now healthy 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:21.659955+0000 mon.a (mon.0) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3085037502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.660010+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.701881+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.a scrub starts 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:21.704769+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.a scrub ok 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:22.028267+0000 mon.c (mon.2) 84 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:23.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:22.028995+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-10T08:36:23.027 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.048 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:23.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:23.189 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.266 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:23.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:23.402 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.424 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:23.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:23.557 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.577 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.027952+0000 mgr.y (mgr.14553) 126 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.072033+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.072976+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.638276+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.639374+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:36:23.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:22.661764+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-10T08:36:23.673 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.661832+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-10T08:36:23.673 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.711405+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:36:23.673 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:22.714297+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:36:23.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.027952+0000 mgr.y (mgr.14553) 126 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.072033+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.072976+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.638276+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.639374+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:22.661764+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.661832+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.711405+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:36:23.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:22.714297+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.027952+0000 mgr.y (mgr.14553) 126 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.072033+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.072976+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.638276+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.639374+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:22.661764+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.661832+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.711405+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:36:23.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:22.714297+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:36:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:23.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:23.978 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:23.997 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:23.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:24.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.046+0000 7f19da94e700 1 -- 172.21.15.107:0/3941202744 >> v1:172.21.15.107:6789/0 conn(0x7f19d4104330 legacy=0x7f19d41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.046+0000 7f19da94e700 1 -- 172.21.15.107:0/3941202744 shutdown_connections 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.046+0000 7f19da94e700 1 -- 172.21.15.107:0/3941202744 >> 172.21.15.107:0/3941202744 conn(0x7f19d40f3640 msgr2=0x7f19d40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.046+0000 7f19da94e700 1 -- 172.21.15.107:0/3941202744 shutdown_connections 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.046+0000 7f19da94e700 1 -- 172.21.15.107:0/3941202744 wait complete. 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 librados: starting msgr at 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 librados: starting objecter 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 Processor -- start 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 -- start start 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 librados: setting wanted keys 2024-02-10T08:36:24.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.047+0000 7f19da94e700 1 librados: calling monclient init 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19da94e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d40fcfc0 con 0x7f19d40fda50 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19da94e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d40fd150 con 0x7f19d4104330 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19da94e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d40fd340 con 0x7f19d41007f0 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d3fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f19d40fda50 0x7f19d40fc820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59676/0 (socket says 172.21.15.107:59676) 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d3fff700 1 -- 172.21.15.107:0/304300980 learned_addr learned my addr 172.21.15.107:0/304300980 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter ms_handle_connect 0x7f19d41007f0 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter ms_handle_connect 0x7f19d40fda50 2024-02-10T08:36:24.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter ms_handle_connect 0x7f19d4104330 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929097893 0 0) 0x7f19d40fcfc0 con 0x7f19d40fda50 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc0033f0 con 0x7f19d40fda50 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868953515 0 0) 0x7f19d40fd150 con 0x7f19d4104330 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d40fcfc0 con 0x7f19d4104330 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303150990 0 0) 0x7f19d40fd340 con 0x7f19d41007f0 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d40fd150 con 0x7f19d41007f0 2024-02-10T08:36:24.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653963181 0 0) 0x7f19d40fcfc0 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d40fd340 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f19cc002310 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1799181666 0 0) 0x7f19bc0033f0 con 0x7f19d40fda50 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d40fcfc0 con 0x7f19d40fda50 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f19c40028b0 con 0x7f19d40fda50 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1963166213 0 0) 0x7f19d40fd340 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6790/0 conn(0x7f19d41007f0 legacy=0x7f19d4114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6789/0 conn(0x7f19d40fda50 legacy=0x7f19d40fc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.048+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d411cc00 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d4119c30 con 0x7f19d4104330 2024-02-10T08:36:24.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19da94e700 10 client.24751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:24.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d411a120 con 0x7f19d4104330 2024-02-10T08:36:24.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19da94e700 1 librados: init done 2024-02-10T08:36:24.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f19cc003290 con 0x7f19d4104330 2024-02-10T08:36:24.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f19cc003990 con 0x7f19d4104330 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.049+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f19c0004f00 con 0x7f19d4104330 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.050+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f19cc0168a0 con 0x7f19d4104330 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(96..96 src has 40..96) v4 ==== 6204+0+0 (unknown 1600340348 0 0) 0x7f19cc076360 con 0x7f19d4104330 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 osd_map(96..96 src has 40..96) v4 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 3 client.24751.objecter handle_osd_map got epochs [96,96] > 0 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 3 client.24751.objecter handle_osd_map decoding full epoch 96 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 20 client.24751.objecter dump_active .. 0 homeless 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) v1 ==== 54+0+16 (unknown 2433924269 0 2420000192) 0x7f19cc078750 con 0x7f19d4104330 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19bc063c60 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 10 client.24751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 1 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 1 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 0 1 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 4 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 10 client.24751.objecter _send_command 1 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.051+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c0004970 con 0x7f19c0004530 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c0004530 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c0004970 con 0x7f19c0004530 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 10 client.24751.objecter _finish_command 1 = system:0 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 4 2024-02-10T08:36:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.052+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 0 1 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 10 client.24751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 2 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 2 2024-02-10T08:36:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00051a0 osd=0 3 2024-02-10T08:36:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 0 2 2024-02-10T08:36:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 4 2024-02-10T08:36:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 10 client.24751.objecter _send_command 2 2024-02-10T08:36:24.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.073+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c0004d50 con 0x7f19c0004530 2024-02-10T08:36:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c0004d50 con 0x7f19c0004530 2024-02-10T08:36:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 10 client.24751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 4 2024-02-10T08:36:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 0 2 2024-02-10T08:36:24.078 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 10 client.24751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 1 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 4 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 10 client.24751.objecter _send_command 3 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c000bef0 con 0x7f19c0002090 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.076+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c0002090 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.077+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c000bef0 con 0x7f19c0002090 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.077+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:24.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.077+0000 7f19d17fa700 10 client.24751.objecter _finish_command 3 = system:0 2024-02-10T08:36:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.077+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 4 2024-02-10T08:36:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.077+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 1 3 2024-02-10T08:36:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 10 client.24751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 4 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 4 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c0004260 osd=1 3 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 1 4 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 4 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 10 client.24751.objecter _send_command 4 2024-02-10T08:36:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.092+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c00016b0 con 0x7f19c0002090 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c00016b0 con 0x7f19c0002090 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19d17fa700 10 client.24751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 4 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 1 4 2024-02-10T08:36:24.096 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 10 client.24751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 5 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 5 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.094+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.095+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 2 5 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.095+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 4 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.095+0000 7f19da94e700 10 client.24751.objecter _send_command 5 2024-02-10T08:36:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.095+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c000ddd0 con 0x7f19c000d210 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.095+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c000d210 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.096+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c000ddd0 con 0x7f19c000d210 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.096+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.096+0000 7f19d17fa700 10 client.24751.objecter _finish_command 5 = system:0 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.096+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 4 2024-02-10T08:36:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.096+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 2 5 2024-02-10T08:36:24.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:24.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 10 client.24751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 6 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 6 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00035f0 osd=2 3 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 2 6 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 4 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 10 client.24751.objecter _send_command 6 2024-02-10T08:36:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.116+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c0003120 con 0x7f19c000d210 2024-02-10T08:36:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c0003120 con 0x7f19c000d210 2024-02-10T08:36:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19d17fa700 10 client.24751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 4 2024-02-10T08:36:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 2 6 2024-02-10T08:36:24.120 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 10 client.24751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 7 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 7 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 3 7 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 4 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 10 client.24751.objecter _send_command 7 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.118+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c0015a70 con 0x7f19c0011f30 2024-02-10T08:36:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.119+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c0011f30 2024-02-10T08:36:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.120+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c0015a70 con 0x7f19c0011f30 2024-02-10T08:36:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.120+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.120+0000 7f19d17fa700 10 client.24751.objecter _finish_command 7 = system:0 2024-02-10T08:36:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.120+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 4 2024-02-10T08:36:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.120+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 3 7 2024-02-10T08:36:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 10 client.24751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 8 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 8 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00118b0 osd=3 3 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 3 8 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 4 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 10 client.24751.objecter _send_command 8 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.142+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c000e1b0 con 0x7f19c0011f30 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.144+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c000e1b0 con 0x7f19c0011f30 2024-02-10T08:36:24.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.144+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.144+0000 7f19d17fa700 10 client.24751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.144+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 4 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.144+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 3 8 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 10 client.24751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 9 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 9 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 4 9 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 4 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 10 client.24751.objecter _send_command 9 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c000e120 con 0x7f19c000cd70 2024-02-10T08:36:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.145+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c000cd70 2024-02-10T08:36:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.146+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c000e120 con 0x7f19c000cd70 2024-02-10T08:36:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.146+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.146+0000 7f19d17fa700 10 client.24751.objecter _finish_command 9 = system:0 2024-02-10T08:36:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.146+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 4 2024-02-10T08:36:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.146+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 4 9 2024-02-10T08:36:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 10 client.24751.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 10 2024-02-10T08:36:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 10 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c000df70 osd=4 3 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 4 10 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 4 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 10 client.24751.objecter _send_command 10 2024-02-10T08:36:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.165+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c000e120 con 0x7f19c000cd70 2024-02-10T08:36:24.168 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c000e120 con 0x7f19c000cd70 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19d17fa700 10 client.24751.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 4 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 4 10 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 10 client.24751.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 11 2024-02-10T08:36:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 11 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 5 11 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 4 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 10 client.24751.objecter _send_command 11 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.167+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c0016e80 con 0x7f19c00168c0 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c00168c0 2024-02-10T08:36:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c0016e80 con 0x7f19c00168c0 2024-02-10T08:36:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 10 client.24751.objecter _finish_command 11 = system:0 2024-02-10T08:36:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 4 2024-02-10T08:36:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.168+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 5 11 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 10 client.24751.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 12 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 12 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c000e120 osd=5 3 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 5 12 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 4 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 10 client.24751.objecter _send_command 12 2024-02-10T08:36:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.183+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c0016e80 con 0x7f19c00168c0 2024-02-10T08:36:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.184+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c0016e80 con 0x7f19c00168c0 2024-02-10T08:36:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.184+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.184+0000 7f19d17fa700 10 client.24751.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.184+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 4 2024-02-10T08:36:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.184+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 5 12 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 10 client.24751.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 13 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 13 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 6 13 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 4 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 10 client.24751.objecter _send_command 13 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c0022c40 con 0x7f19c001f070 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.185+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c001f070 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.186+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c0022c40 con 0x7f19c001f070 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.186+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.186+0000 7f19d17fa700 10 client.24751.objecter _finish_command 13 = system:0 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.186+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 4 2024-02-10T08:36:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.186+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 6 13 2024-02-10T08:36:24.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 10 client.24751.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 14 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 14 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c0016e80 osd=6 3 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 6 14 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 4 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 10 client.24751.objecter _send_command 14 2024-02-10T08:36:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.208+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c00243f0 con 0x7f19c001f070 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c00243f0 con 0x7f19c001f070 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 10 client.24751.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 4 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 6 14 2024-02-10T08:36:24.211 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 10 client.24751.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 15 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 15 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 7 15 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 4 2024-02-10T08:36:24.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 10 client.24751.objecter _send_command 15 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c0025010 con 0x7f19c0024ad0 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.210+0000 7f19d17fa700 10 client.24751.objecter ms_handle_connect 0x7f19c0024ad0 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.211+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f19c0025010 con 0x7f19c0024ad0 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.211+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.211+0000 7f19d17fa700 10 client.24751.objecter _finish_command 15 = system:0 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.211+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 4 2024-02-10T08:36:24.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.211+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 7 15 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 10 client.24751.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign -1 16 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 20 client.24751.objecter _get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 15 client.24751.objecter _session_command_op_remove -1 16 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 20 client.24751.objecter get_session s=0x7f19c00243f0 osd=7 3 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 15 client.24751.objecter _session_command_op_assign 7 16 2024-02-10T08:36:24.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 4 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 10 client.24751.objecter _send_command 16 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.237+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19c0025010 con 0x7f19c0024ad0 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d17fa700 1 -- 172.21.15.107:0/304300980 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f19c0025010 con 0x7f19c0024ad0 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d17fa700 10 client.24751.objecter ms_dispatch 0x7f19d4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d17fa700 10 client.24751.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d17fa700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 4 2024-02-10T08:36:24.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d17fa700 15 client.24751.objecter _session_command_op_remove 7 16 2024-02-10T08:36:24.241 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:24.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 librados: watch_flush enter 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 librados: watch_flush exit 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.0 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6801/3321596972 conn(0x7f19c0004530 legacy=0x7f19c0002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00051a0 osd=0 1 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.1 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6805/133516126 conn(0x7f19c0002090 legacy=0x7f19c000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0004260 osd=1 1 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.2 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6809/3575881728 conn(0x7f19c000d210 legacy=0x7f19c000d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00035f0 osd=2 1 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.3 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6813/2871825691 conn(0x7f19c0011f30 legacy=0x7f19c0014320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00118b0 osd=3 1 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.4 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.195:6800/2499671285 conn(0x7f19c000cd70 legacy=0x7f19c0015c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000df70 osd=4 1 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.5 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.195:6804/4178321486 conn(0x7f19c00168c0 legacy=0x7f19c001d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c000e120 osd=5 1 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.6 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.195:6808/1417479017 conn(0x7f19c001f070 legacy=0x7f19c0021460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c0016e80 osd=6 1 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter close_session for osd.7 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.195:6812/1934176928 conn(0x7f19c0024ad0 legacy=0x7f19c0022c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter put_session s=0x7f19c00243f0 osd=7 1 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 20 client.24751.objecter shutdown clearing up homeless session... 2024-02-10T08:36:24.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 10 client.24751.objecter successfully canceled tick 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d8eeb700 1 -- 172.21.15.107:0/304300980 reap_dead start 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d8eeb700 1 -- 172.21.15.107:0/304300980 reap_dead start 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d8eeb700 1 -- 172.21.15.107:0/304300980 reap_dead start 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d8eeb700 1 -- 172.21.15.107:0/304300980 reap_dead start 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19d8eeb700 1 -- 172.21.15.107:0/304300980 reap_dead start 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.107:6800/2634842009 conn(0x7f19bc063c60 legacy=0x7f19bc066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.239+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> v1:172.21.15.195:6789/0 conn(0x7f19d4104330 legacy=0x7f19d41184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.240+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 shutdown_connections 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.240+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 >> 172.21.15.107:0/304300980 conn(0x7f19d40f3640 msgr2=0x7f19d41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.240+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 shutdown_connections 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.240+0000 7f19da94e700 1 -- 172.21.15.107:0/304300980 wait complete. 2024-02-10T08:36:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:24.240+0000 7f19da94e700 1 librados: shutdown 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.245 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:36:24.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:23.082956+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.b scrub starts 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:23.084323+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.b scrub ok 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:23.643646+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.a scrub starts 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:23.647161+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub ok 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:23.663425+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:24.052231+0000 mon.b (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.107:0/304300980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.093199+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.096100+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:23.082956+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.b scrub starts 2024-02-10T08:36:24.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:23.084323+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.b scrub ok 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:23.643646+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.a scrub starts 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:23.647161+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub ok 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:23.663425+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:24.052231+0000 mon.b (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.107:0/304300980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.093199+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:36:24.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.096100+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:36:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:23.082956+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.b scrub starts 2024-02-10T08:36:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:23.084323+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.b scrub ok 2024-02-10T08:36:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:23.643646+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.a scrub starts 2024-02-10T08:36:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:23.647161+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub ok 2024-02-10T08:36:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:23.663425+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-10T08:36:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:24.052231+0000 mon.b (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.107:0/304300980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.093199+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:36:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.096100+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:36:25.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:36:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:25] "GET /metrics HTTP/1.1" 200 172237 "" "Prometheus/2.33.4" 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.028346+0000 mgr.y (mgr.14553) 127 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.062476+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.066599+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.762712+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.764016+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.977609+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:24.978880+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.050889+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.053800+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:36:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:25.144997+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.028346+0000 mgr.y (mgr.14553) 127 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.062476+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.066599+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.762712+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.764016+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.977609+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:24.978880+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.050889+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.053800+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:36:25.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:25.144997+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:25.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.028346+0000 mgr.y (mgr.14553) 127 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:36:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.062476+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.066599+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.762712+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.764016+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.977609+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:24.978880+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.050889+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:36:26.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.053800+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:36:26.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:25.144997+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:26.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:26.112 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:36:26.113 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:36:26.113 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:36:26.113 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:36:26.113 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:36:26.128 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:26.152 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:26.153 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:26.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:26.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:26.482 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:26.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:26.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:26.611 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:26.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.500303+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:36:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.503348+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:36:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.636137+0000 osd.3 (osd.3) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:25.636185+0000 osd.3 (osd.3) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.013816+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:36:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.014687+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.500303+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.503348+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.636137+0000 osd.3 (osd.3) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:25.636185+0000 osd.3 (osd.3) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.013816+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.014687+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:26.029568+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:26.030246+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.047182+0000 osd.7 (osd.7) 57 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.048115+0000 osd.7 (osd.7) 58 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:26.147561+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:36:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.147618+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-10T08:36:26.759 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:26.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:26.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:26.897 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:26.918 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:26.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:26.029568+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:26.030246+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.047182+0000 osd.7 (osd.7) 57 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.048115+0000 osd.7 (osd.7) 58 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:26.147561+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:36:26.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.147618+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-10T08:36:27.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.500303+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.503348+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.636137+0000 osd.3 (osd.3) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:25.636185+0000 osd.3 (osd.3) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.013816+0000 osd.3 (osd.3) 69 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.014687+0000 osd.3 (osd.3) 70 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:26.029568+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:26.030246+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:36:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.047182+0000 osd.7 (osd.7) 57 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:36:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.048115+0000 osd.7 (osd.7) 58 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:36:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:26.147561+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:36:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.147618+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-10T08:36:27.054 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:27.075 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:27.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:27.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:27.208 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:27.227 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:27.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:27.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:27.360 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:27.380 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:27.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:27.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to '57bd6abdec7bb457ae7999d9c96682e9ac678e27'. 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.492 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-02-10T08:36:27.493 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at 57bd6abdec7 qa/suites/orch: whitelist warnings that are expected in test environments 2024-02-10T08:36:27.498 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-10T08:36:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:27.567 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T08:36:27.569 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T08:36:27.569 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T08:36:27.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:36:27.711 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:27.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:27.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:27.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:36:27.843 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:27.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.029183+0000 mgr.y (mgr.14553) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:27.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.453892+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:36:27.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.456775+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:36:27.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.575393+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.578281+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.631022+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.633973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.961119+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:26.962330+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:36:27.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.051325+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:36:27.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.052353+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:36:27.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.149947+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-10T08:36:27.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:27.445562+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:36:27.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.029183+0000 mgr.y (mgr.14553) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.453892+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.456775+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.575393+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.578281+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.631022+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:36:27.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.633973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.961119+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:26.962330+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.051325+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.052353+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.149947+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-10T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:27.445562+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:36:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:27.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:27.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.029183+0000 mgr.y (mgr.14553) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.453892+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.456775+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.575393+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.578281+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.631022+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.1d scrub starts 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.633973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.1d scrub ok 2024-02-10T08:36:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.961119+0000 osd.0 (osd.0) 65 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:36:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:26.962330+0000 osd.0 (osd.0) 66 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:36:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.051325+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:36:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.052353+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:36:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.149947+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-10T08:36:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:27.445562+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:36:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:28.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:28.855 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:36:28.855 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:36:28.855 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.log 2024-02-10T08:36:28.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:36:28.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.053303+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.054139+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:27.815087+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:27.815706+0000 mon.c (mon.2) 88 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:27.820354+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.956495+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:27.957922+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:36:28.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.150849+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.053303+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.054139+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:27.815087+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:27.815706+0000 mon.c (mon.2) 88 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:27.820354+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.956495+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:27.957922+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:36:28.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.150849+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-10T08:36:28.977 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:29.013 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.053303+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.054139+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:27.815087+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:27.815706+0000 mon.c (mon.2) 88 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:27.820354+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.956495+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:36:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:27.957922+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:36:29.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.150849+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-10T08:36:29.575 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T08:36:29.736 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T08:36:29.737 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T08:36:29.737 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T08:36:29.777 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T08:36:29.780 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-02-10T08:36:29.780 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-10T08:36:29.826 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:29.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:29.841 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-10T08:36:29.842 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-10T08:36:29.842 DEBUG:teuthology.orchestra.run.smithi107:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57bd6abdec7bb457ae7999d9c96682e9ac678e27 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ parallel=1 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' '' = --serial ']' 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ color= 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -t 1 ']' 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-10T08:36:29.926 INFO:tasks.workunit.client.0.smithi107.stderr:+ declare -A pids 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_aio 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_aio' 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_aio 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_aio 2024-02-10T08:36:29.928 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112398 2024-02-10T08:36:29.928 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_aio on pid 112398' 2024-02-10T08:36:29.928 INFO:tasks.workunit.client.0.smithi107.stdout:test api_aio on pid 112398 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112398 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_aio_pp 2024-02-10T08:36:29.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_aio_pp' 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_aio_pp 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_aio 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_aio: /' 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_aio_pp 2024-02-10T08:36:29.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112406 2024-02-10T08:36:29.931 INFO:tasks.workunit.client.0.smithi107.stdout:test api_aio_pp on pid 112406 2024-02-10T08:36:29.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_aio_pp on pid 112406' 2024-02-10T08:36:29.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112406 2024-02-10T08:36:29.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_io 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_io' 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_io 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_aio_pp 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_io 2024-02-10T08:36:29.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112414 2024-02-10T08:36:29.934 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_io on pid 112414' 2024-02-10T08:36:29.934 INFO:tasks.workunit.client.0.smithi107.stdout:test api_io on pid 112414 2024-02-10T08:36:29.935 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112414 2024-02-10T08:36:29.935 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-10T08:36:29.935 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_io_pp 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_io_pp' 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_io_pp 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_io 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_io.log 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_io: /' 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_io_pp 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112422 2024-02-10T08:36:29.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_io_pp on pid 112422' 2024-02-10T08:36:29.937 INFO:tasks.workunit.client.0.smithi107.stdout:test api_io_pp on pid 112422 2024-02-10T08:36:29.938 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112422 2024-02-10T08:36:29.938 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.938 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.938 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-10T08:36:29.938 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_asio 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_asio' 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_asio 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_io_pp 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_asio 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112430 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_asio on pid 112430' 2024-02-10T08:36:29.939 INFO:tasks.workunit.client.0.smithi107.stdout:test api_asio on pid 112430 2024-02-10T08:36:29.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:29.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112430 2024-02-10T08:36:29.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_list 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_list' 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_list 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_asio 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_asio: /' 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_list 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112438 2024-02-10T08:36:29.942 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_list on pid 112438' 2024-02-10T08:36:29.943 INFO:tasks.workunit.client.0.smithi107.stdout:test api_list on pid 112438 2024-02-10T08:36:29.944 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112438 2024-02-10T08:36:29.944 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_lock 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_lock' 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_lock 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_list 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_list.log 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_list: /' 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_lock 2024-02-10T08:36:29.945 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112446 2024-02-10T08:36:29.946 INFO:tasks.workunit.client.0.smithi107.stdout:test api_lock on pid 112446 2024-02-10T08:36:29.947 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_lock on pid 112446' 2024-02-10T08:36:29.947 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112446 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_lock_pp 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_lock_pp' 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_lock_pp 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_lock 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_lock: /' 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_lock_pp 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112454 2024-02-10T08:36:29.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_lock_pp on pid 112454' 2024-02-10T08:36:29.949 INFO:tasks.workunit.client.0.smithi107.stdout:test api_lock_pp on pid 112454 2024-02-10T08:36:29.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112454 2024-02-10T08:36:29.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-10T08:36:29.950 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_misc 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_misc' 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_misc 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_lock_pp 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-10T08:36:29.951 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-10T08:36:29.952 INFO:tasks.workunit.client.0.smithi107.stdout:test api_misc on pid 112462 2024-02-10T08:36:29.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_misc 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112462 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_misc on pid 112462' 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112462 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_misc_pp 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_misc_pp' 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_misc_pp 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_misc 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-10T08:36:29.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_misc: /' 2024-02-10T08:36:29.954 INFO:tasks.workunit.client.0.smithi107.stdout:test api_misc_pp on pid 112473 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_misc_pp 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112473 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_misc_pp on pid 112473' 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112473 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_tier_pp 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_tier_pp' 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_tier_pp 2024-02-10T08:36:29.955 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_misc_pp 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_tier_pp 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112484 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_tier_pp on pid 112484' 2024-02-10T08:36:29.956 INFO:tasks.workunit.client.0.smithi107.stdout:test api_tier_pp on pid 112484 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112484 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_pool 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_pool' 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_pool 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_tier_pp 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-10T08:36:29.958 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_pool 2024-02-10T08:36:29.959 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112495 2024-02-10T08:36:29.959 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_pool on pid 112495' 2024-02-10T08:36:29.959 INFO:tasks.workunit.client.0.smithi107.stdout:test api_pool on pid 112495 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112495 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_snapshots 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_snapshots' 2024-02-10T08:36:29.960 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_snapshots 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_pool 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_pool: /' 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_snapshots 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112506 2024-02-10T08:36:29.961 INFO:tasks.workunit.client.0.smithi107.stdout:test api_snapshots on pid 112506 2024-02-10T08:36:29.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_snapshots on pid 112506' 2024-02-10T08:36:29.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112506 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_snapshots_pp 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_snapshots_pp' 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_snapshots_pp 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_snapshots 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-10T08:36:29.963 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_snapshots_pp 2024-02-10T08:36:29.964 INFO:tasks.workunit.client.0.smithi107.stdout:test api_snapshots_pp on pid 112518 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112518 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_snapshots_pp on pid 112518' 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112518 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_stat 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_stat' 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_stat 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_snapshots_pp 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-10T08:36:29.965 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-10T08:36:29.966 INFO:tasks.workunit.client.0.smithi107.stdout:test api_stat on pid 112531 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_stat 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112531 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_stat on pid 112531' 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112531 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_stat_pp 2024-02-10T08:36:29.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_stat_pp' 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_stat_pp 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_stat 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_stat: /' 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_stat_pp 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112540 2024-02-10T08:36:29.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_stat_pp on pid 112540' 2024-02-10T08:36:29.969 INFO:tasks.workunit.client.0.smithi107.stdout:test api_stat_pp on pid 112540 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112540 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_watch_notify 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_watch_notify' 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_watch_notify 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_stat_pp 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-10T08:36:29.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-10T08:36:29.971 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_watch_notify 2024-02-10T08:36:29.971 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112548 2024-02-10T08:36:29.971 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_watch_notify on pid 112548' 2024-02-10T08:36:29.971 INFO:tasks.workunit.client.0.smithi107.stdout:test api_watch_notify on pid 112548 2024-02-10T08:36:29.972 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112548 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_watch_notify_pp 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_watch_notify_pp' 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_watch_notify_pp 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_watch_notify 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_watch_notify_pp 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112556 2024-02-10T08:36:29.973 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_watch_notify_pp on pid 112556' 2024-02-10T08:36:29.974 INFO:tasks.workunit.client.0.smithi107.stdout:test api_watch_notify_pp on pid 112556 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112556 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_cmd 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_cmd' 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_cmd 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-10T08:36:29.975 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-10T08:36:29.976 INFO:tasks.workunit.client.0.smithi107.stdout:test api_cmd on pid 112565 2024-02-10T08:36:29.976 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_cmd 2024-02-10T08:36:29.976 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112565 2024-02-10T08:36:29.976 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_cmd on pid 112565' 2024-02-10T08:36:29.976 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112565 2024-02-10T08:36:29.977 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.977 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.977 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-10T08:36:29.977 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_cmd_pp 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.029625+0000 mgr.y (mgr.14553) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.075087+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.076395+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.633171+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.634847+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.635010+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.635606+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:36:29.977 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.742590+0000 osd.4 (osd.4) 61 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:36:29.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:28.744461+0000 osd.4 (osd.4) 62 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:36:29.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:29.151843+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-10T08:36:29.978 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_cmd_pp' 2024-02-10T08:36:29.978 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.029625+0000 mgr.y (mgr.14553) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:29.978 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.075087+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.076395+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.633171+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.634847+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.635010+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.635606+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.742590+0000 osd.4 (osd.4) 61 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:28.744461+0000 osd.4 (osd.4) 62 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:36:29.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:29.151843+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-10T08:36:29.979 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_cmd_pp 2024-02-10T08:36:29.980 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_cmd 2024-02-10T08:36:29.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-10T08:36:29.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_cmd: /' 2024-02-10T08:36:29.980 INFO:tasks.workunit.client.0.smithi107.stdout:test api_cmd_pp on pid 112574 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_cmd_pp 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112574 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_cmd_pp on pid 112574' 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112574 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-10T08:36:29.981 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_service 2024-02-10T08:36:29.981 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:29.982 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_service' 2024-02-10T08:36:29.982 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_service 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_cmd_pp 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_service 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112582 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_service on pid 112582' 2024-02-10T08:36:29.983 INFO:tasks.workunit.client.0.smithi107.stdout:test api_service on pid 112582 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112582 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_service_pp 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_service_pp' 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_service_pp 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_service 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_service.log 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_service: /' 2024-02-10T08:36:29.985 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_service_pp 2024-02-10T08:36:29.986 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112590 2024-02-10T08:36:29.986 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_service_pp on pid 112590' 2024-02-10T08:36:29.986 INFO:tasks.workunit.client.0.smithi107.stdout:test api_service_pp on pid 112590 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112590 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_c_write_operations 2024-02-10T08:36:29.987 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_c_write_operations' 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_c_write_operations 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_service_pp 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_c_write_operations 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112604 2024-02-10T08:36:29.988 INFO:tasks.workunit.client.0.smithi107.stdout:test api_c_write_operations on pid 112604 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_c_write_operations on pid 112604' 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112604 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s api_c_read_operations 2024-02-10T08:36:29.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' api_c_read_operations' 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo api_c_read_operations 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_c_write_operations 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=api_c_read_operations 2024-02-10T08:36:29.993 INFO:tasks.workunit.client.0.smithi107.stdout:test api_c_read_operations on pid 112612 2024-02-10T08:36:29.994 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112612 2024-02-10T08:36:29.994 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test api_c_read_operations on pid 112612' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112612 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s list_parallel 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' list_parallel' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo list_parallel 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_api_c_read_operations 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=list_parallel 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112620 2024-02-10T08:36:29.995 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test list_parallel on pid 112620' 2024-02-10T08:36:29.996 INFO:tasks.workunit.client.0.smithi107.stdout:test list_parallel on pid 112620 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112620 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s open_pools_parallel 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' open_pools_parallel' 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo open_pools_parallel 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:29.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_list_parallel 2024-02-10T08:36:29.998 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-10T08:36:29.998 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ list_parallel: /' 2024-02-10T08:36:29.998 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=open_pools_parallel 2024-02-10T08:36:29.998 INFO:tasks.workunit.client.0.smithi107.stdout:test open_pools_parallel on pid 112628 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112628 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test open_pools_parallel on pid 112628' 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112628 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:++ printf %25s delete_pools_parallel 2024-02-10T08:36:29.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ r=' delete_pools_parallel' 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:++ echo delete_pools_parallel 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '{print $1}' 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_open_pools_parallel 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ ff=delete_pools_parallel 2024-02-10T08:36:30.000 INFO:tasks.workunit.client.0.smithi107.stdout:test delete_pools_parallel on pid 112636 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112636 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo 'test delete_pools_parallel on pid 112636' 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ pids[$f]=112636 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ ret=0 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-10T08:36:30.001 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112506 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112506 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph_test_rados_delete_pools_parallel 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-10T08:36:30.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-10T08:36:30.003 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:30.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.029625+0000 mgr.y (mgr.14553) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.075087+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:36:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.076395+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.633171+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.634847+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.635010+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.635606+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.742590+0000 osd.4 (osd.4) 61 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:28.744461+0000 osd.4 (osd.4) 62 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:36:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:29.151843+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-10T08:36:30.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.133+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/2352946679 >> v1:172.21.15.107:6789/0 conn(0x7fb1a4103250 legacy=0x7fb1a41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.133+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/2352946679 shutdown_connections 2024-02-10T08:36:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.133+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/2352946679 >> 172.21.15.107:0/2352946679 conn(0x7fb1a40f2510 msgr2=0x7fb1a40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.133+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/2352946679 shutdown_connections 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.133+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/2352946679 wait complete. 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 librados: starting msgr at 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 librados: starting objecter 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 Processor -- start 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 -- start start 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 librados: setting wanted keys 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 librados: calling monclient init 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1a4119650 con 0x7fb1a40fc910 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1a411a830 con 0x7fb1a40ff710 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a9ee7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1a411ba30 con 0x7fb1a4103250 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a3fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb1a4103250 0x7fb1a4117d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45798/0 (socket says 172.21.15.107:45798) 2024-02-10T08:36:30.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a3fff700 1 -- 172.21.15.107:0/3200237694 learned_addr learned my addr 172.21.15.107:0/3200237694 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a37fe700 1 --1- 172.21.15.107:0/3200237694 >> v1:172.21.15.107:6789/0 conn(0x7fb1a40fc910 0x7fb1a4060a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57656/0 (socket says 172.21.15.107:57656) 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb1a4103250 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb1a40fc910 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.134+0000 7fb1a0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb1a40ff710 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955655746 0 0) 0x7fb1a411ba30 con 0x7fb1a4103250 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c003030 con 0x7fb1a4103250 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390267455 0 0) 0x7fb1a4119650 con 0x7fb1a40fc910 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a411ba30 con 0x7fb1a40fc910 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325592625 0 0) 0x7fb1a411a830 con 0x7fb1a40ff710 2024-02-10T08:36:30.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a4119650 con 0x7fb1a40ff710 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665783099 0 0) 0x7fb1a411ba30 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a411a830 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb194001b00 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193372546 0 0) 0x7fb18c003030 con 0x7fb1a4103250 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a411ba30 con 0x7fb1a4103250 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb19c002320 con 0x7fb1a4103250 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1732311811 0 0) 0x7fb1a411a830 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 >> v1:172.21.15.107:6790/0 conn(0x7fb1a4103250 legacy=0x7fb1a4117d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 >> v1:172.21.15.195:6789/0 conn(0x7fb1a40ff710 legacy=0x7fb1a4061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a411cc30 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a4119840 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a9ee7700 10 client.14856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a4119cb0 con 0x7fb1a40fc910 2024-02-10T08:36:30.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a9ee7700 1 librados: init done 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb194002470 con 0x7fb1a40fc910 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.135+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb194005a40 con 0x7fb1a40fc910 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.136+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb190004f20 con 0x7fb1a40fc910 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb1940025e0 con 0x7fb1a40fc910 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(100..100 src has 50..100) v4 ==== 6204+0+0 (unknown 1139711067 0 0) 0x7fb194078560 con 0x7fb1a40fc910 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 10 client.14856.objecter ms_dispatch 0x7fb1a4001550 osd_map(100..100 src has 50..100) v4 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 3 client.14856.objecter handle_osd_map got epochs [100,100] > 0 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 3 client.14856.objecter handle_osd_map decoding full epoch 100 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 20 client.14856.objecter dump_active .. 0 homeless 2024-02-10T08:36:30.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.137+0000 7fb1a0ff9700 10 client.14856.objecter ms_handle_connect 0x7fb18c063ca0 2024-02-10T08:36:30.144 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:30.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.142+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb19404c0f0 con 0x7fb1a40fc910 2024-02-10T08:36:30.163 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:30.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:30.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:30.393 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:30.399 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:30.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:30.461+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb190005750 con 0x7fb1a40fc910 2024-02-10T08:36:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:30.601 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:30.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:30.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:30.735 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:30.863 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:30.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:30.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:30.995 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:31.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:32.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:32.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:32.379 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.623+0000 7f297ecf8700 1 -- 172.21.15.107:0/1289854824 >> v1:172.21.15.107:6790/0 conn(0x7f29780f8a00 legacy=0x7f29780f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.623+0000 7f297ecf8700 1 -- 172.21.15.107:0/1289854824 shutdown_connections 2024-02-10T08:36:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.623+0000 7f297ecf8700 1 -- 172.21.15.107:0/1289854824 >> 172.21.15.107:0/1289854824 conn(0x7f29780f2510 msgr2=0x7f29780f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.623+0000 7f297ecf8700 1 -- 172.21.15.107:0/1289854824 shutdown_connections 2024-02-10T08:36:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.624+0000 7f297ecf8700 1 -- 172.21.15.107:0/1289854824 wait complete. 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.624+0000 7f297ecf8700 1 librados: starting msgr at 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.624+0000 7f297ecf8700 1 librados: starting objecter 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 Processor -- start 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 -- start start 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 librados: setting wanted keys 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 librados: calling monclient init 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29780f8680 con 0x7f2978104330 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29780f8850 con 0x7f29780f8a00 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f297ecf8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f297811bc50 con 0x7f29780f95b0 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2977fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f29780f95b0 0x7f2978114da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46512/0 (socket says 172.21.15.107:46512) 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2977fff700 1 -- 172.21.15.107:0/2352163046 learned_addr learned my addr 172.21.15.107:0/2352163046 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:33.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter ms_handle_connect 0x7f29780f95b0 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter ms_handle_connect 0x7f2978104330 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter ms_handle_connect 0x7f29780f8a00 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962643197 0 0) 0x7f297811bc50 con 0x7f29780f95b0 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2968002220 con 0x7f29780f95b0 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.625+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173575426 0 0) 0x7f29780f8850 con 0x7f29780f8a00 2024-02-10T08:36:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f297811bc50 con 0x7f29780f8a00 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908374950 0 0) 0x7f2968002220 con 0x7f29780f95b0 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29780f8850 con 0x7f29780f95b0 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2964001920 con 0x7f29780f95b0 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58453557 0 0) 0x7f297811bc50 con 0x7f29780f8a00 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2968002220 con 0x7f29780f8a00 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f296c001a60 con 0x7f29780f8a00 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1305955610 0 0) 0x7f29780f8850 con 0x7f29780f95b0 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.195:6789/0 conn(0x7f29780f8a00 legacy=0x7f2978102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6789/0 conn(0x7f2978104330 legacy=0x7f29781184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297811ce50 con 0x7f29780f95b0 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297811be40 con 0x7f29780f95b0 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2964001cb0 con 0x7f29780f95b0 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2964005bc0 con 0x7f29780f95b0 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f297ecf8700 10 client.34937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f297811c2b0 con 0x7f29780f95b0 2024-02-10T08:36:33.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.626+0000 7f297ecf8700 1 librados: init done 2024-02-10T08:36:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.627+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f297805f7a0 con 0x7f29780f95b0 2024-02-10T08:36:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.627+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2964002270 con 0x7f29780f95b0 2024-02-10T08:36:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(101..101 src has 50..101) v4 ==== 6961+0+0 (unknown 1186365996 0 0) 0x7f2964078ac0 con 0x7f29780f95b0 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 osd_map(101..101 src has 50..101) v4 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 3 client.34937.objecter handle_osd_map got epochs [101,101] > 0 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 3 client.34937.objecter handle_osd_map decoding full epoch 101 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 20 client.34937.objecter dump_active .. 0 homeless 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v101) v1 ==== 54+0+16 (unknown 2957627552 0 2420000192) 0x7f2964078e70 con 0x7f29780f95b0 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f2968063ad0 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 10 client.34937.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 1 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 1 2024-02-10T08:36:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 0 1 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 4 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 10 client.34937.objecter _send_command 1 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f297810a070 con 0x7f29780f9a10 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.628+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f29780f9a10 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.629+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f297810a070 con 0x7f29780f9a10 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.629+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.629+0000 7f2975ffb700 10 client.34937.objecter _finish_command 1 = system:0 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.629+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 4 2024-02-10T08:36:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.629+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 0 1 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 10 client.34937.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 2 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 2 2024-02-10T08:36:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f29780475e0 osd=0 3 2024-02-10T08:36:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 0 2 2024-02-10T08:36:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 4 2024-02-10T08:36:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 10 client.34937.objecter _send_command 2 2024-02-10T08:36:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.644+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2978028660 con 0x7f29780f9a10 2024-02-10T08:36:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2978028660 con 0x7f29780f9a10 2024-02-10T08:36:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f2975ffb700 10 client.34937.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 4 2024-02-10T08:36:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 0 2 2024-02-10T08:36:33.647 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:33.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 10 client.34937.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 1 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 4 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 10 client.34937.objecter _send_command 3 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.646+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2978120ce0 con 0x7f29780fa740 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f29780fa740 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2978120ce0 con 0x7f29780fa740 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 10 client.34937.objecter _finish_command 3 = system:0 2024-02-10T08:36:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 4 2024-02-10T08:36:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.647+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 1 3 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 10 client.34937.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 4 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 4 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f29780287d0 osd=1 3 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 1 4 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 4 2024-02-10T08:36:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 10 client.34937.objecter _send_command 4 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.662+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f297811c7e0 con 0x7f29780fa740 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f297811c7e0 con 0x7f29780fa740 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 10 client.34937.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 4 2024-02-10T08:36:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 1 4 2024-02-10T08:36:33.665 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 10 client.34937.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 5 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 5 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 2 5 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 4 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 10 client.34937.objecter _send_command 5 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2978125700 con 0x7f2978059ea0 2024-02-10T08:36:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.664+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f2978059ea0 2024-02-10T08:36:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.665+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2978125700 con 0x7f2978059ea0 2024-02-10T08:36:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.665+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.665+0000 7f2975ffb700 10 client.34937.objecter _finish_command 5 = system:0 2024-02-10T08:36:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.665+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 4 2024-02-10T08:36:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.665+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 2 5 2024-02-10T08:36:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 10 client.34937.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 6 2024-02-10T08:36:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 6 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f297811cbf0 osd=2 3 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 2 6 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 4 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 10 client.34937.objecter _send_command 6 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.679+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2978059400 con 0x7f2978059ea0 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2978059400 con 0x7f2978059ea0 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 10 client.34937.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 4 2024-02-10T08:36:33.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 2 6 2024-02-10T08:36:33.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 10 client.34937.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 7 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 7 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 3 7 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 4 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 10 client.34937.objecter _send_command 7 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f297812a150 con 0x7f2978121b30 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.681+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f2978121b30 2024-02-10T08:36:33.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.682+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f297812a150 con 0x7f2978121b30 2024-02-10T08:36:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.682+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.682+0000 7f2975ffb700 10 client.34937.objecter _finish_command 7 = system:0 2024-02-10T08:36:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.682+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 4 2024-02-10T08:36:33.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.682+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 3 7 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 10 client.34937.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 8 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 8 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978059810 osd=3 3 2024-02-10T08:36:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 3 8 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 4 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 10 client.34937.objecter _send_command 8 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.697+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2978121090 con 0x7f2978121b30 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2978121090 con 0x7f2978121b30 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f2975ffb700 10 client.34937.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 4 2024-02-10T08:36:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 3 8 2024-02-10T08:36:33.699 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 10 client.34937.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 9 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 9 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 4 9 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 4 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 10 client.34937.objecter _send_command 9 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.698+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2978126740 con 0x7f2978121230 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f2978121230 2024-02-10T08:36:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2978126740 con 0x7f2978121230 2024-02-10T08:36:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 10 client.34937.objecter _finish_command 9 = system:0 2024-02-10T08:36:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 4 2024-02-10T08:36:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.699+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 4 9 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 10 client.34937.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 10 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 10 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978121090 osd=4 3 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 4 10 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 4 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 10 client.34937.objecter _send_command 10 2024-02-10T08:36:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.714+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2978126740 con 0x7f2978121230 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.716+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2978126740 con 0x7f2978121230 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.716+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.716+0000 7f2975ffb700 10 client.34937.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.716+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 4 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.716+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 4 10 2024-02-10T08:36:33.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 10 client.34937.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 11 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 11 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 5 11 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 4 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 10 client.34937.objecter _send_command 11 2024-02-10T08:36:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f297812c600 con 0x7f297812c090 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.717+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f297812c090 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.718+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f297812c600 con 0x7f297812c090 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.718+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.718+0000 7f2975ffb700 10 client.34937.objecter _finish_command 11 = system:0 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.718+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 4 2024-02-10T08:36:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.718+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 5 11 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 10 client.34937.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 12 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 12 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978126740 osd=5 3 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 5 12 2024-02-10T08:36:33.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 4 2024-02-10T08:36:33.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 10 client.34937.objecter _send_command 12 2024-02-10T08:36:33.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.734+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f297812c600 con 0x7f297812c090 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.735+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f297812c600 con 0x7f297812c090 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.735+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.735+0000 7f2975ffb700 10 client.34937.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.735+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 4 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.735+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 5 12 2024-02-10T08:36:33.736 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 10 client.34937.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 13 2024-02-10T08:36:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 13 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 6 13 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 4 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 10 client.34937.objecter _send_command 13 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2978137430 con 0x7f2978133730 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.736+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f2978133730 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.737+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2978137430 con 0x7f2978133730 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.737+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.737+0000 7f2975ffb700 10 client.34937.objecter _finish_command 13 = system:0 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.737+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 4 2024-02-10T08:36:33.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.737+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 6 13 2024-02-10T08:36:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 10 client.34937.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 14 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 14 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f297812c600 osd=6 3 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 6 14 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 4 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 10 client.34937.objecter _send_command 14 2024-02-10T08:36:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.751+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2978137430 con 0x7f2978133730 2024-02-10T08:36:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2978137430 con 0x7f2978133730 2024-02-10T08:36:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f2975ffb700 10 client.34937.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 4 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 6 14 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 10 client.34937.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 15 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 15 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 7 15 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 4 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 10 client.34937.objecter _send_command 15 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.753+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29780009b0 con 0x7f29781390f0 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 10 client.34937.objecter ms_handle_connect 0x7f29781390f0 2024-02-10T08:36:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f29780009b0 con 0x7f29781390f0 2024-02-10T08:36:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 10 client.34937.objecter _finish_command 15 = system:0 2024-02-10T08:36:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 4 2024-02-10T08:36:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.754+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 7 15 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 10 client.34937.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign -1 16 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 20 client.34937.objecter _get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_remove -1 16 2024-02-10T08:36:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 20 client.34937.objecter get_session s=0x7f2978137430 osd=7 3 2024-02-10T08:36:33.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 15 client.34937.objecter _session_command_op_assign 7 16 2024-02-10T08:36:33.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 4 2024-02-10T08:36:33.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 10 client.34937.objecter _send_command 16 2024-02-10T08:36:33.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.770+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29780009b0 con 0x7f29781390f0 2024-02-10T08:36:33.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f2975ffb700 1 -- 172.21.15.107:0/2352163046 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f29780009b0 con 0x7f29781390f0 2024-02-10T08:36:33.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f2975ffb700 10 client.34937.objecter ms_dispatch 0x7f2978001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:33.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f2975ffb700 10 client.34937.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f2975ffb700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 4 2024-02-10T08:36:33.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f2975ffb700 15 client.34937.objecter _session_command_op_remove 7 16 2024-02-10T08:36:33.773 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 librados: watch_flush enter 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 librados: watch_flush exit 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.0 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6801/3321596972 conn(0x7f29780f9a10 legacy=0x7f29780f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780475e0 osd=0 1 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.1 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6805/133516126 conn(0x7f29780fa740 legacy=0x7f297811f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f29780287d0 osd=1 1 2024-02-10T08:36:33.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.2 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6809/3575881728 conn(0x7f2978059ea0 legacy=0x7f2978123fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297811cbf0 osd=2 1 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.3 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6813/2871825691 conn(0x7f2978121b30 legacy=0x7f2978128a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978059810 osd=3 1 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.4 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.195:6800/2499671285 conn(0x7f2978121230 legacy=0x7f297812a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297d295700 1 -- 172.21.15.107:0/2352163046 reap_dead start 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978121090 osd=4 1 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.5 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.195:6804/4178321486 conn(0x7f297812c090 legacy=0x7f2978131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297d295700 1 -- 172.21.15.107:0/2352163046 reap_dead start 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978126740 osd=5 1 2024-02-10T08:36:33.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.6 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.195:6808/1417479017 conn(0x7f2978133730 legacy=0x7f2978135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f297812c600 osd=6 1 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter close_session for osd.7 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297d295700 1 -- 172.21.15.107:0/2352163046 reap_dead start 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.195:6812/1934176928 conn(0x7f29781390f0 legacy=0x7f2978139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter put_session s=0x7f2978137430 osd=7 1 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 20 client.34937.objecter shutdown clearing up homeless session... 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 10 client.34937.objecter successfully canceled tick 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2968063ad0 legacy=0x7f2968065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.772+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> v1:172.21.15.107:6790/0 conn(0x7f29780f95b0 legacy=0x7f2978114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.773+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 shutdown_connections 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.773+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 >> 172.21.15.107:0/2352163046 conn(0x7f29780f2510 msgr2=0x7f29781077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.773+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 shutdown_connections 2024-02-10T08:36:33.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.773+0000 7f297ecf8700 1 -- 172.21.15.107:0/2352163046 wait complete. 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:33.773+0000 7f297ecf8700 1 librados: shutdown 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:36:33.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:36:34.059 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:34.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:34.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:34.199 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:34.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:34.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:34.378 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:34.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:34.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:34.482 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:36:34.483 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:36:34.514 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:34.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:34.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:35.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:36:35.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.136+0000 7fb1a96e6700 10 client.14856.objecter tick 2024-02-10T08:36:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.888+0000 7fb1a0ff9700 1 -- 172.21.15.107:0/3200237694 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v102) v1 ==== 87+0+0 (unknown 564714410 0 0) 0x7fb19404a500 con 0x7fb1a40fc910 2024-02-10T08:36:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.891+0000 7fb1a9ee7700 10 librados: watch_flush enter 2024-02-10T08:36:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.895+0000 7fb1a9ee7700 10 librados: watch_flush exit 2024-02-10T08:36:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.895+0000 7fb1a9ee7700 20 client.14856.objecter shutdown clearing up homeless session... 2024-02-10T08:36:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.895+0000 7fb1a9ee7700 10 client.14856.objecter successfully canceled tick 2024-02-10T08:36:35.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.897+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb18c063ca0 legacy=0x7fb18c066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:35.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.897+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 >> v1:172.21.15.107:6789/0 conn(0x7fb1a40fc910 legacy=0x7fb1a4060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:35.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.898+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 shutdown_connections 2024-02-10T08:36:35.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.898+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 >> 172.21.15.107:0/3200237694 conn(0x7fb1a40f2510 msgr2=0x7fb1a41066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.902+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 shutdown_connections 2024-02-10T08:36:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.909+0000 7fb1a9ee7700 1 -- 172.21.15.107:0/3200237694 wait complete. 2024-02-10T08:36:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:35.913+0000 7fb1a9ee7700 1 librados: shutdown 2024-02-10T08:36:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:29.047487+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:29.048960+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.030214+0000 mon.c (mon.2) 89 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.030882+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:30.151507+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:36:35.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:30.157468+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:36:35.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.328005+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:35.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.346564+0000 mon.b (mon.1) 39 : audit [DBG] from='client.? v1:172.21.15.107:0/1387719925' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T08:36:35.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:29.047487+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:36:35.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:29.048960+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:36:35.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.030214+0000 mon.c (mon.2) 89 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.030882+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:30.151507+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:30.157468+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.328005+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.346564+0000 mon.b (mon.1) 39 : audit [DBG] from='client.? v1:172.21.15.107:0/1387719925' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T08:36:35.934 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:35] "GET /metrics HTTP/1.1" 200 172237 "" "Prometheus/2.33.4" 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:29.047487+0000 osd.3 (osd.3) 75 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:29.048960+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.030214+0000 mon.c (mon.2) 89 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.030882+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:30.151507+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:36:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:30.157468+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:36:36.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.328005+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:36.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.346564+0000 mon.b (mon.1) 39 : audit [DBG] from='client.? v1:172.21.15.107:0/1387719925' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T08:36:36.782 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:36.816 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:36.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:36.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:29.801866+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:29.804756+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:30.029943+0000 mgr.y (mgr.14553) 130 : cluster [DBG] pgmap v123: 131 pgs: 131 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:30.792612+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:30.795490+0000 osd.5 (osd.5) 62 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.796774+0000 mon.a (mon.0) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.797652+0000 mon.a (mon.0) 765 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.804779+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:30.815400+0000 mon.b (mon.1) 40 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.051 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:31.139797+0000 mgr.y (mgr.14553) 131 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.232693+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.232741+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.430247+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.434025+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.889065+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:31.890382+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.030704+0000 mgr.y (mgr.14553) 132 : cluster [DBG] pgmap v125: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.131890+0000 osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.132756+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.447221+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:36:37.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.450799+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.707665+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:32.709387+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.041727+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.042578+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.448680+0000 osd.1 (osd.1) 75 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.452358+0000 osd.1 (osd.1) 76 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:33.628569+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.107:0/2352163046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.821600+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.824506+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.927886+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:33.928689+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:36:37.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:34.031062+0000 mgr.y (mgr.14553) 133 : cluster [DBG] pgmap v126: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:34.400663+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:34.834555+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:34.837479+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.551809+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/1076227176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:35.688475+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f deep-scrub starts 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:35.691628+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f deep-scrub ok 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:35.870393+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:35.875164+0000 mon.a (mon.0) 768 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.881722+0000 mon.a (mon.0) 769 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.883774+0000 mon.a (mon.0) 770 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.887909+0000 mon.a (mon.0) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.892443+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.107:0/1628114538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.894252+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.107:0/1472191512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.894993+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.107:0/439232726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.895214+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.107:0/3352524603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.895383+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.107:0/3601488252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.901174+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.107:0/1952017327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.907649+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.107:0/1955669538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.907911+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.107:0/3084601356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.908103+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.107:0/2794038004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.908489+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.107:0/110400822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.908709+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.908902+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.107:0/2789504017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.909086+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.107:0/3483335305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.909350+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.107:0/2127225077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.909537+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.107:0/2829047206' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.909736+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.107:0/2188923728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.909948+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.107:0/3458750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:35.916955+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.107:0/1659510119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:36.033630+0000 mgr.y (mgr.14553) 134 : cluster [DBG] pgmap v128: 867 pgs: 647 creating+peering, 2 peering, 89 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:29.801866+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:29.804756+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:30.029943+0000 mgr.y (mgr.14553) 130 : cluster [DBG] pgmap v123: 131 pgs: 131 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:30.792612+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:30.795490+0000 osd.5 (osd.5) 62 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.796774+0000 mon.a (mon.0) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:36:37.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.797652+0000 mon.a (mon.0) 765 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.804779+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:30.815400+0000 mon.b (mon.1) 40 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:31.139797+0000 mgr.y (mgr.14553) 131 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.232693+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.232741+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.430247+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.434025+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.889065+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:31.890382+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.030704+0000 mgr.y (mgr.14553) 132 : cluster [DBG] pgmap v125: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:37.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.131890+0000 osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.132756+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.447221+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.450799+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.707665+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:32.709387+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.041727+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.042578+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.448680+0000 osd.1 (osd.1) 75 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.452358+0000 osd.1 (osd.1) 76 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:33.628569+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.107:0/2352163046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.821600+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:36:37.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.824506+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.927886+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:33.928689+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:34.031062+0000 mgr.y (mgr.14553) 133 : cluster [DBG] pgmap v126: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:34.400663+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:34.834555+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:34.837479+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.551809+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/1076227176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:35.688475+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f deep-scrub starts 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:35.691628+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f deep-scrub ok 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:35.870393+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:35.875164+0000 mon.a (mon.0) 768 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.881722+0000 mon.a (mon.0) 769 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.060 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.883774+0000 mon.a (mon.0) 770 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.887909+0000 mon.a (mon.0) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.892443+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.107:0/1628114538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.894252+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.107:0/1472191512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.894993+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.107:0/439232726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.895214+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.107:0/3352524603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.895383+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.107:0/3601488252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.901174+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.107:0/1952017327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.907649+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.107:0/1955669538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:36.144807+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-10T08:36:37.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.145400+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.146681+0000 mon.a (mon.0) 773 : audit [INF] from='client.34808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.146921+0000 mon.a (mon.0) 774 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.147133+0000 mon.a (mon.0) 775 : audit [INF] from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.147347+0000 mon.a (mon.0) 776 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.147526+0000 mon.a (mon.0) 777 : audit [INF] from='client.34841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.147696+0000 mon.a (mon.0) 778 : audit [INF] from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.147900+0000 mon.a (mon.0) 779 : audit [INF] from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.148593+0000 mon.a (mon.0) 780 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.151704+0000 mon.a (mon.0) 781 : audit [INF] from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.151934+0000 mon.a (mon.0) 782 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.152248+0000 mon.a (mon.0) 783 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.152474+0000 mon.a (mon.0) 784 : audit [INF] from='client.24925 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.152706+0000 mon.a (mon.0) 785 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.153000+0000 mon.a (mon.0) 786 : audit [INF] from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.153211+0000 mon.a (mon.0) 787 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.153444+0000 mon.a (mon.0) 788 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.153702+0000 mon.a (mon.0) 789 : audit [INF] from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.154076+0000 mon.a (mon.0) 790 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.154476+0000 mon.a (mon.0) 791 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.907911+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.107:0/3084601356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.908103+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.107:0/2794038004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.908489+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.107:0/110400822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.908709+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.908902+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.107:0/2789504017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.909086+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.107:0/3483335305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.909350+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.107:0/2127225077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.909537+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.107:0/2829047206' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.909736+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.107:0/2188923728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.909948+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.107:0/3458750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:35.916955+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.107:0/1659510119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:36.033630+0000 mgr.y (mgr.14553) 134 : cluster [DBG] pgmap v128: 867 pgs: 647 creating+peering, 2 peering, 89 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:36.144807+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-10T08:36:37.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.145400+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.146681+0000 mon.a (mon.0) 773 : audit [INF] from='client.34808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.146921+0000 mon.a (mon.0) 774 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.147133+0000 mon.a (mon.0) 775 : audit [INF] from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.147347+0000 mon.a (mon.0) 776 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.147526+0000 mon.a (mon.0) 777 : audit [INF] from='client.34841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.147696+0000 mon.a (mon.0) 778 : audit [INF] from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.147900+0000 mon.a (mon.0) 779 : audit [INF] from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.148593+0000 mon.a (mon.0) 780 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.151704+0000 mon.a (mon.0) 781 : audit [INF] from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.151934+0000 mon.a (mon.0) 782 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.152248+0000 mon.a (mon.0) 783 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.152474+0000 mon.a (mon.0) 784 : audit [INF] from='client.24925 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.152706+0000 mon.a (mon.0) 785 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.153000+0000 mon.a (mon.0) 786 : audit [INF] from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.153211+0000 mon.a (mon.0) 787 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.153444+0000 mon.a (mon.0) 788 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.153702+0000 mon.a (mon.0) 789 : audit [INF] from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.154076+0000 mon.a (mon.0) 790 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.154476+0000 mon.a (mon.0) 791 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:37.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:37.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:29.801866+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:29.804756+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:30.029943+0000 mgr.y (mgr.14553) 130 : cluster [DBG] pgmap v123: 131 pgs: 131 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:30.792612+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:30.795490+0000 osd.5 (osd.5) 62 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.796774+0000 mon.a (mon.0) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:36:37.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.797652+0000 mon.a (mon.0) 765 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.804779+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:30.815400+0000 mon.b (mon.1) 40 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:31.139797+0000 mgr.y (mgr.14553) 131 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.232693+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.232741+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.430247+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.434025+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.889065+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:31.890382+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.030704+0000 mgr.y (mgr.14553) 132 : cluster [DBG] pgmap v125: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.131890+0000 osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.132756+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.447221+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.450799+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:36:37.245 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.707665+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:32.709387+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.041727+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.042578+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.448680+0000 osd.1 (osd.1) 75 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.452358+0000 osd.1 (osd.1) 76 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:33.628569+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.107:0/2352163046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.821600+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.824506+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.927886+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:33.928689+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:34.031062+0000 mgr.y (mgr.14553) 133 : cluster [DBG] pgmap v126: 195 pgs: 57 creating+peering, 2 peering, 7 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:34.400663+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:34.834555+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:34.837479+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:36:37.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.551809+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/1076227176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:35.688475+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f deep-scrub starts 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:35.691628+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f deep-scrub ok 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:35.870393+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:35.875164+0000 mon.a (mon.0) 768 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.881722+0000 mon.a (mon.0) 769 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.883774+0000 mon.a (mon.0) 770 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.887909+0000 mon.a (mon.0) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3200237694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.892443+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.107:0/1628114538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.894252+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.107:0/1472191512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.894993+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.107:0/439232726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.895214+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.107:0/3352524603' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.895383+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.107:0/3601488252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.901174+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.107:0/1952017327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.907649+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.107:0/1955669538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.907911+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.107:0/3084601356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.908103+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.107:0/2794038004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.908489+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.107:0/110400822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.908709+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.908902+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.107:0/2789504017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.909086+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.107:0/3483335305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.909350+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.107:0/2127225077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.909537+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.107:0/2829047206' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.909736+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.107:0/2188923728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.909948+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.107:0/3458750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:35.916955+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.107:0/1659510119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:36.033630+0000 mgr.y (mgr.14553) 134 : cluster [DBG] pgmap v128: 867 pgs: 647 creating+peering, 2 peering, 89 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:36.144807+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.145400+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.146681+0000 mon.a (mon.0) 773 : audit [INF] from='client.34808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.146921+0000 mon.a (mon.0) 774 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.147133+0000 mon.a (mon.0) 775 : audit [INF] from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.147347+0000 mon.a (mon.0) 776 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.147526+0000 mon.a (mon.0) 777 : audit [INF] from='client.34841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.147696+0000 mon.a (mon.0) 778 : audit [INF] from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.147900+0000 mon.a (mon.0) 779 : audit [INF] from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.148593+0000 mon.a (mon.0) 780 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.151704+0000 mon.a (mon.0) 781 : audit [INF] from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.151934+0000 mon.a (mon.0) 782 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.152248+0000 mon.a (mon.0) 783 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.152474+0000 mon.a (mon.0) 784 : audit [INF] from='client.24925 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.152706+0000 mon.a (mon.0) 785 : audit [INF] from='client.24949 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.153000+0000 mon.a (mon.0) 786 : audit [INF] from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.153211+0000 mon.a (mon.0) 787 : audit [INF] from='client.24820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.153444+0000 mon.a (mon.0) 788 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.153702+0000 mon.a (mon.0) 789 : audit [INF] from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.154076+0000 mon.a (mon.0) 790 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:37.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.154476+0000 mon.a (mon.0) 791 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:37.250 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:37.251 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-10T08:36:37.251 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [----------] Global test environment set-up. 2024-02-10T08:36:37.251 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-10T08:36:37.251 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (9 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (16 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (15 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (7 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (7 ms) 2024-02-10T08:36:37.252 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (12 ms) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (10 ms) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [----------] 12 tests from AsioRados (86 ms total) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [----------] Global test environment tear-down 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (7010 ms total) 2024-02-10T08:36:37.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-10T08:36:37.269 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:37.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:37.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:37.403 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:37.424 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:37.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:37.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:37.622 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-02-10T08:36:37.622 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] Global test environment set-up. 2024-02-10T08:36:37.622 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.ListObjects (324 ms) 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (55 ms) 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo1,foo2,foo3 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo1 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo2 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo3 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo1,foo4,foo5 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo4 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo5 2024-02-10T08:36:37.623 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo1 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo6,foo7 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo7 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo6 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo4 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo5 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns2:foo7 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns2:foo6 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo1 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo1 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo2 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo3 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (111 ms) 2024-02-10T08:36:37.624 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 1 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 10 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 13 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 7 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 14 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 0 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 15 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 11 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 5 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 8 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 6 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 3 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 4 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 12 0 2024-02-10T08:36:37.625 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 9 0 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 2 0 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (71 ms) 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: x cursor=MIN 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=1 cursor=25:02547ec2:::1:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=10 cursor=25:52ea6a34:::10:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=13 cursor=25:566253c9:::13:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=7 cursor=25:5c6b0b28:::7:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=14 cursor=25:62a1935d:::14:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=0 cursor=25:6cac518f:::0:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=15 cursor=25:863748b0:::15:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=11 cursor=25:89d3ae78:::11:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=5 cursor=25:b29083e3:::5:head 2024-02-10T08:36:37.626 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=8 cursor=25:bd63b0f1:::8:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=6 cursor=25:c4fdafeb:::6:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=3 cursor=25:cfc208b3:::3:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=4 cursor=25:d83876eb:::4:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=12 cursor=25:de5d7c5f:::12:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=9 cursor=25:e960b815:::9:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > oid=2 cursor=25:f905c69b:::2:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=1 cursor=25:02547ec2:::1:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:02547ec2:::1:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:02547ec2:::1:head -> 1 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=10 cursor=25:52ea6a34:::10:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:52ea6a34:::10:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:52ea6a34:::10:head -> 10 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=13 cursor=25:566253c9:::13:head 2024-02-10T08:36:37.627 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:566253c9:::13:head 2024-02-10T08:36:37.628 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:566253c9:::13:head -> 13 2024-02-10T08:36:37.628 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=7 cursor=25:5c6b0b28:::7:head 2024-02-10T08:36:37.628 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:5c6b0b28:::7:head 2024-02-10T08:36:37.628 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:5c6b0b28:::7:head -> 7 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=14 cursor=25:62a1935d:::14:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:62a1935d:::14:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:62a1935d:::14:head -> 14 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=0 cursor=25:6cac518f:::0:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:6cac518f:::0:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:6cac518f:::0:head -> 0 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=15 cursor=25:863748b0:::15:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:863748b0:::15:head 2024-02-10T08:36:37.679 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:863748b0:::15:head -> 15 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=11 cursor=25:89d3ae78:::11:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:89d3ae78:::11:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:89d3ae78:::11:head -> 11 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=5 cursor=25:b29083e3:::5:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:b29083e3:::5:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:b29083e3:::5:head -> 5 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=8 cursor=25:bd63b0f1:::8:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:bd63b0f1:::8:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:bd63b0f1:::8:head -> 8 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=6 cursor=25:c4fdafeb:::6:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:c4fdafeb:::6:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:c4fdafeb:::6:head -> 6 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=3 cursor=25:cfc208b3:::3:head 2024-02-10T08:36:37.680 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:cfc208b3:::3:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:cfc208b3:::3:head -> 3 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=4 cursor=25:d83876eb:::4:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:d83876eb:::4:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:d83876eb:::4:head -> 4 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=12 cursor=25:de5d7c5f:::12:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:de5d7c5f:::12:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:de5d7c5f:::12:head -> 12 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=9 cursor=25:e960b815:::9:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:e960b815:::9:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:e960b815:::9:head -> 9 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : oid=2 cursor=25:f905c69b:::2:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:f905c69b:::2:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:f905c69b:::2:head -> 2 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:e960b815:::9:head 2024-02-10T08:36:37.681 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:e960b815:::9:head expected=25:e960b815:::9:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:e960b815:::9:head -> 9 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=9 expected=9 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:de5d7c5f:::12:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:de5d7c5f:::12:head expected=25:de5d7c5f:::12:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:de5d7c5f:::12:head -> 12 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=12 expected=12 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:f905c69b:::2:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:f905c69b:::2:head expected=25:f905c69b:::2:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:f905c69b:::2:head -> 2 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=2 expected=2 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:d83876eb:::4:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:d83876eb:::4:head expected=25:d83876eb:::4:head 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:d83876eb:::4:head -> 4 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=4 expected=4 2024-02-10T08:36:37.682 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:cfc208b3:::3:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:cfc208b3:::3:head expected=25:cfc208b3:::3:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:cfc208b3:::3:head -> 3 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=3 expected=3 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:c4fdafeb:::6:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:c4fdafeb:::6:head expected=25:c4fdafeb:::6:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:c4fdafeb:::6:head -> 6 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=6 expected=6 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:bd63b0f1:::8:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:bd63b0f1:::8:head expected=25:bd63b0f1:::8:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:bd63b0f1:::8:head -> 8 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=8 expected=8 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:b29083e3:::5:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:b29083e3:::5:head expected=25:b29083e3:::5:head 2024-02-10T08:36:37.683 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:b29083e3:::5:head -> 5 2024-02-10T08:36:37.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:36:37.852 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=5 expect list_parallel: process_1_[112669]: starting. 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: creating pool ceph_test_rados_list_parallel.smithi107-112625 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: created object 0... 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: created object 25... 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: created object 49... 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: finishing. 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_1_[112669]: shutting down. 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: starting. 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: listing objects. 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: listed object 0... 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: listed object 25... 2024-02-10T08:36:37.853 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: saw 50 objects 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_2_[112670]: shutting down. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: starting. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: creating pool ceph_test_rados_list_parallel.smithi107-112625 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: created object 0... 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: created object 25... 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: created object 49... 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: finishing. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_3_[114433]: shutting down. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: starting. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: listing objects. 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: listed object 0... 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: listed object 25... 2024-02-10T08:36:37.854 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: saw 43 objects 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_4_[114434]: shutting down. 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: starting. 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: removed 25 objects... 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: removed half of the objects 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: removed 50 objects... 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: removed 50 objects 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_5_[114435]: shutting down. 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: starting. 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: creating pool ceph_test_rados_list_parallel.smithi107-112625 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: created object 0... 2024-02-10T08:36:37.855 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: created object 25... 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: created object 49... 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: finishing. 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_6_[114531]: shutting down. 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: starting. 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: listing objects. 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: listed object 0... 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: listed object 25... 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: listed object 50... 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: saw 56 objects 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_7_[114532]: shutting down. 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.856 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: starting. 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: added 25 objects... 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: added half of the objects 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: added 50 objects... 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: added 50 objects 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_8_[114534]: shutting down. 2024-02-10T08:36:37.857 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.142 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.902792+0000 mon.a (mon.0) 792 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.142 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.902857+0000 mon.a (mon.0) 793 : audit [INF] from='client.34808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.142 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.902895+0000 mon.a (mon.0) 794 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.142 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.902926+0000 mon.a (mon.0) 795 : audit [INF] from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.902961+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903000+0000 mon.a (mon.0) 797 : audit [INF] from='client.34841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903047+0000 mon.a (mon.0) 798 : audit [INF] from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903091+0000 mon.a (mon.0) 799 : audit [INF] from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903146+0000 mon.a (mon.0) 800 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903196+0000 mon.a (mon.0) 801 : audit [INF] from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903234+0000 mon.a (mon.0) 802 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903271+0000 mon.a (mon.0) 803 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903304+0000 mon.a (mon.0) 804 : audit [INF] from='client.24925 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903341+0000 mon.a (mon.0) 805 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903388+0000 mon.a (mon.0) 806 : audit [INF] from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903423+0000 mon.a (mon.0) 807 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.143 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903457+0000 mon.a (mon.0) 808 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903502+0000 mon.a (mon.0) 809 : audit [INF] from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903596+0000 mon.a (mon.0) 810 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.903698+0000 mon.a (mon.0) 811 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:36.903769+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.919742+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:36.926011+0000 mon.a (mon.0) 813 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:37.582051+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.107:0/3641509245' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:37.824113+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.902792+0000 mon.a (mon.0) 792 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.902857+0000 mon.a (mon.0) 793 : audit [INF] from='client.34808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.144 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.902895+0000 mon.a (mon.0) 794 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.902926+0000 mon.a (mon.0) 795 : audit [INF] from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.902961+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903000+0000 mon.a (mon.0) 797 : audit [INF] from='client.34841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903047+0000 mon.a (mon.0) 798 : audit [INF] from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903091+0000 mon.a (mon.0) 799 : audit [INF] from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903146+0000 mon.a (mon.0) 800 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903196+0000 mon.a (mon.0) 801 : audit [INF] from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903234+0000 mon.a (mon.0) 802 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903271+0000 mon.a (mon.0) 803 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903304+0000 mon.a (mon.0) 804 : audit [INF] from='client.24925 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903341+0000 mon.a (mon.0) 805 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903388+0000 mon.a (mon.0) 806 : audit [INF] from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903423+0000 mon.a (mon.0) 807 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903457+0000 mon.a (mon.0) 808 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903502+0000 mon.a (mon.0) 809 : audit [INF] from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903596+0000 mon.a (mon.0) 810 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.903698+0000 mon.a (mon.0) 811 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:36.903769+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.919742+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:36.926011+0000 mon.a (mon.0) 813 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:37.582051+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.107:0/3641509245' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:38.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:37.824113+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:38.150 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:38.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:38.223 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ************************** api_c_read_operations: Running main() from gmock_main.cc 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (209 ms) 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (31 ms) 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (42 ms) 2024-02-10T08:36:38.224 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (29 ms) 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (12 ms) 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (9 ms) 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (24 ms) 2024-02-10T08:36:38.225 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (13 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (11 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (19 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (16 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (11 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-02-10T08:36:38.226 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (445 ms total) 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (8055 ms total) 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-10T08:36:38.227 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (156 ms) 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (25 ms) 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (7862 ms) 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (8043 ms total) 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (8043 ms total) 2024-02-10T08:36:38.228 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-10T08:36:38.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: Running main() from gmock_main.cc 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-10T08:36:38.284 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (203 ms) 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (26 ms) 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (7852 ms) 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.902792+0000 mon.a [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.902857+0000 mon.a [INF] from='client.34808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.902895+0000 mon.a [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.902926+0000 mon.a [INF] from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.902961+0000 mon.a [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903000+0000 mon.a [INF] from='client.34841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903047+0000 mon.a [INF] from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.285 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903091+0000 mon.a [INF] from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.286 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903146+0000 mon.a [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.286 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903196+0000 mon.a [INF] from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.286 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903234+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.286 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903271+0000 mon.a [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.902792+0000 mon.a (mon.0) 792 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.902857+0000 mon.a (mon.0) 793 : audit [INF] from='client.34808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi107-112553-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.902895+0000 mon.a (mon.0) 794 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi107-112513-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.902926+0000 mon.a (mon.0) 795 : audit [INF] from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112579-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.902961+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi107-112490-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903000+0000 mon.a (mon.0) 797 : audit [INF] from='client.34841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi107-112411-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903047+0000 mon.a (mon.0) 798 : audit [INF] from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi107-112459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903091+0000 mon.a (mon.0) 799 : audit [INF] from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi107-112427-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903146+0000 mon.a (mon.0) 800 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi107-112545-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903196+0000 mon.a (mon.0) 801 : audit [INF] from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903234+0000 mon.a (mon.0) 802 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi107-112525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903271+0000 mon.a (mon.0) 803 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi107-112443-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903304+0000 mon.a (mon.0) 804 : audit [INF] from='client.24925 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903341+0000 mon.a (mon.0) 805 : audit [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903388+0000 mon.a (mon.0) 806 : audit [INF] from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903423+0000 mon.a (mon.0) 807 : audit [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903457+0000 mon.a (mon.0) 808 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903502+0000 mon.a (mon.0) 809 : audit [INF] from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903596+0000 mon.a (mon.0) 810 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.903698+0000 mon.a (mon.0) 811 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:36.903769+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-10T08:36:38.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.919742+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:36.926011+0000 mon.a (mon.0) 813 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:38.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:37.582051+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.107:0/3641509245' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T08:36:38.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:37.824113+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:38.308 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:38.352 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903304+0000 mon.a [INF] from='client.24925 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable",***** 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: starting. 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: creating pool ceph_test_rados_list_parallel.smithi107-112625 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: created object 0... 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: created object 25... 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: created object 49... 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: finishing. 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_9_[114678]: shutting down. 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: starting. 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: listing objects. 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: listed object 0... 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: listed object 25... 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: listed object 50... 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: listed object 75... 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: saw 98 objects 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_10_[114679]: shutting down. 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: starting. 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: removed 25 objects... 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: removed half of the objects 2024-02-10T08:36:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: removed 50 objects... 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: removed 50 objects 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_13_[114682]: shutting down. 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: starting. 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: added 25 objects... 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: added half of the objects 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: added 50 objects... 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: added 50 objects 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_11_[114680]: shutting down. 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: starting. 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: added 25 objects... 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: added half of the objects 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: added 50 objects... 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: added 50 objects 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_12_[114681]: shutting down. 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: starting. 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: creating pool ceph_test_rados_list_parallel.smithi107-112625 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: created object 0... 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: created object 25... 2024-02-10T08:36:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: created object 49... 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: finishing. 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_14_[114818]: shutting down. 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: starting. 2024-02-10T08:36:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listing objects. 2024-02-10T08:36:38.461 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:38.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:38.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:38.594 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:38.616 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:38.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:38.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:38.749 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:38.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:36:38.772 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:38.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:38.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:38.905 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:38.968 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:36:38.968 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.053298+0000 mon.c (mon.2) 96 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:38.218503+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.220268+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.264758+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.279233+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.281225+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:38.281944+0000 client.admin (client.35171) 0 : cluster [INF] onexx 2024-02-10T08:36:38.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.289007+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.295873+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:39.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.491204+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.737838+0000 mon.a (mon.0) 816 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738071+0000 mon.a (mon.0) 817 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738308+0000 mon.a (mon.0) 818 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.000 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: pr"pool": "LibRadosIo_smithi107-112419-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.000 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903341+0000 mon.a [INF] from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112570-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.000 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903388+0000 mon.a [INF] from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi107-112617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.000 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903423+0000 mon.a [INF] from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi107-112451-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903457+0000 mon.a [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903502+0000 mon.a [INF] from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi107-112403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903596+0000 mon.a [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi107-112537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.903698+0000 mon.a [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.919742+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:36.926011+0000 mon.a [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.053298+0000 mon.c [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.220268+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.264758+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.279233+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.281225+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.001 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.281944+0000 client.admin [INF] onexx 2024-02-10T08:36:39.059 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:36:39.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 -- 172.21.15.107:0/57739235 >> v1:172.21.15.107:6789/0 conn(0x7fbc280fc910 legacy=0x7fbc280fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 -- 172.21.15.107:0/57739235 shutdown_connections 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 -- 172.21.15.107:0/57739235 >> 172.21.15.107:0/57739235 conn(0x7fbc280f2510 msgr2=0x7fbc280f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 -- 172.21.15.107:0/57739235 shutdown_connections 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 -- 172.21.15.107:0/57739235 wait complete. 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 librados: starting msgr at 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.725+0000 7fbc2eb76700 1 librados: starting objecter 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 Processor -- start 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 -- start start 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 librados: setting wanted keys 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 librados: calling monclient init 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc2810e990 con 0x7fbc280ff710 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc280fbf90 con 0x7fbc28103250 2024-02-10T08:36:39.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2eb76700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc280fc120 con 0x7fbc280fc910 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2d113700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbc28103250 0x7fbc2811cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:45504/0 (socket says 172.21.15.107:45504) 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc2d113700 1 -- 172.21.15.107:0/3146110451 learned_addr learned my addr 172.21.15.107:0/3146110451 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter ms_handle_connect 0x7fbc28103250 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter ms_handle_connect 0x7fbc280ff710 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter ms_handle_connect 0x7fbc280fc910 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.726+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447643210 0 0) 0x7fbc280fc120 con 0x7fbc280fc910 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc100033f0 con 0x7fbc280fc910 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267435376 0 0) 0x7fbc2810e990 con 0x7fbc280ff710 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc280fc120 con 0x7fbc280ff710 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617837991 0 0) 0x7fbc100033f0 con 0x7fbc280fc910 2024-02-10T08:36:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc2810e990 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbc1c002d90 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926761372 0 0) 0x7fbc280fc120 con 0x7fbc280ff710 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc100033f0 con 0x7fbc280ff710 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbc180018c0 con 0x7fbc280ff710 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 817882791 0 0) 0x7fbc2810e990 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 >> v1:172.21.15.195:6789/0 conn(0x7fbc28103250 legacy=0x7fbc2811cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 >> v1:172.21.15.107:6789/0 conn(0x7fbc280ff710 legacy=0x7fbc28119490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc280fc2b0 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2811e340 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc2eb76700 10 client.35099.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2811e7b0 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.727+0000 7fbc2eb76700 1 librados: init done 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.729+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbc1c001aa0 con 0x7fbc280fc910 2024-02-10T08:36:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.729+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2805f7a0 con 0x7fbc280fc910 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.729+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbc1c0037b0 con 0x7fbc280fc910 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.729+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbc1c0166c0 con 0x7fbc280fc910 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(103..103 src has 50..103) v4 ==== 15693+0+0 (unknown 2526050364 0 0) 0x7fbc1c079520 con 0x7fbc280fc910 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 10 client.35099.objecter ms_dispatch 0x7fbc28001550 osd_map(103..103 src has 50..103) v4 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 3 client.35099.objecter handle_osd_map got epochs [103,103] > 0 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 3 client.35099.objecter handle_osd_map decoding full epoch 103 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 20 client.35099.objecter dump_active .. 0 homeless 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.730+0000 7fbc25ffb700 10 client.35099.objecter ms_handle_connect 0x7fbc10063bb0 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.735+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbc1c04ab60 con 0x7fbc280fc910 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:37.993+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc2810a0b0 con 0x7fbc10063bb0 2024-02-10T08:36:39.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:38.052+0000 7fbc25ffb700 1 -- 172.21.15.107:0/3146110451 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1289105 (unknown 2965378022 0 3703179480) 0x7fbc2810a0b0 con 0x7fbc10063bb0 2024-02-10T08:36:39.077 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:39.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:39.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.053298+0000 mon.c (mon.2) 96 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:38.218503+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-10T08:36:39.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.220268+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.264758+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.279233+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.281225+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:38.281944+0000 client.admin (client.35171) 0 : cluster [INF] onexx 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.289007+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.295873+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.491204+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.737838+0000 mon.a (mon.0) 816 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738071+0000 mon.a (mon.0) 817 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738308+0000 mon.a (mon.0) 818 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738538+0000 mon.a (mon.0) 819 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738539+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738797+0000 mon.a (mon.0) 820 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.053298+0000 mon.c (mon.2) 96 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:38.218503+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.220268+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.264758+0000 mon.c (mon.2) 98 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.279233+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.281225+0000 mon.c (mon.2) 100 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:38.281944+0000 client.admin (client.35171) 0 : cluster [INF] onexx 2024-02-10T08:36:39.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.289007+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.295873+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.491204+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.737838+0000 mon.a (mon.0) 816 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738071+0000 mon.a (mon.0) 817 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738308+0000 mon.a (mon.0) 818 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738538+0000 mon.a (mon.0) 819 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738539+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738797+0000 mon.a (mon.0) 820 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 10 librados: watch_flush enter 2024-02-10T08:36:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 10 librados: watch_flush exit 2024-02-10T08:36:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 20 client.35099.objecter shutdown clearing up homeless session... 2024-02-10T08:36:39.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 10 client.35099.objecter successfully canceled tick 2024-02-10T08:36:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbc10063bb0 legacy=0x7fbc10066070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 >> v1:172.21.15.107:6790/0 conn(0x7fbc280fc910 legacy=0x7fbc280fb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 shutdown_connections 2024-02-10T08:36:39.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.075+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 >> 172.21.15.107:0/3146110451 conn(0x7fbc280f2510 msgr2=0x7fbc281066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:39.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.076+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 shutdown_connections 2024-02-10T08:36:39.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.076+0000 7fbc2eb76700 1 -- 172.21.15.107:0/3146110451 wait complete. 2024-02-10T08:36:39.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:39.076+0000 7fbc2eb76700 1 librados: shutdown 2024-02-10T08:36:39.103 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:36:39.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:39.213 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:39.233 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:39.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738538+0000 mon.a (mon.0) 819 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738539+0000 mon.c (mon.2) 102 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738797+0000 mon.a (mon.0) 820 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:39.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:39.364 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:39.439 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:39.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:39.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:39.579 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:39.598 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:39.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:39.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:39.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:36:39.728 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:39.745 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:39.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:39.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:40.060 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10Tocess_15_[114819]: listed object 0... 2024-02-10T08:36:40.060 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listed object 25... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listed object 50... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listed object 75... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listed object 100... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: listed object 125... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: saw 150 objects 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_15_[114819]: shutting down. 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: starting. 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: added 25 objects... 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: added half of the objects 2024-02-10T08:36:40.061 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: added 50 objects... 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: added 50 objects 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: process_16_[114820]: shutting down. 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******************************* 2024-02-10T08:36:40.062 INFO:tasks.workunit.client.0.smithi107.stdout: list_parallel: ******* SUCCESS ********** 2024-02-10T08:36:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:40.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:40.188 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:40.215 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:40.233 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:40.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:37.994824+0000 mgr.y (mgr.14553) 135 : audit [DBG] from='client.35099 v1:172.21.15.107:0/3146110451' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:38.051427+0000 mgr.y (mgr.14553) 136 : cluster [DBG] pgmap v131: 546 pgs: 290 creating+peering, 1 peering, 126 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.738988+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.739382+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.825717+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.992233+0000 mon.a (mon.0) 821 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.992534+0000 mon.a (mon.0) 822 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.992848+0000 mon.a (mon.0) 823 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:38.993061+0000 mon.a (mon.0) 824 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.001691+0000 mon.a (mon.0) 825 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.001922+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.002017+0000 mon.a (mon.0) 827 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.015706+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.021363+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.107:0/1946495091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.529153+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.529273+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.701817+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.702519+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.702920+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.703304+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.703670+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.704109+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.704522+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.704899+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.705289+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:40.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.705669+0000 mon.b (mon.1) 73 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:40.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:39.772196+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-10T08:36:40.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.772737+0000 mon.a (mon.0) 829 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T08:36:40.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.772798+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.773283+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.773699+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:40.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:40.364 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:40.382 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:40.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:37.994824+0000 mgr.y (mgr.14553) 135 : audit [DBG] from='client.35099 v1:172.21.15.107:0/3146110451' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:38.051427+0000 mgr.y (mgr.14553) 136 : cluster [DBG] pgmap v131: 546 pgs: 290 creating+peering, 1 peering, 126 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.738988+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.739382+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.825717+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:40.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.992233+0000 mon.a (mon.0) 821 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.992534+0000 mon.a (mon.0) 822 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.992848+0000 mon.a (mon.0) 823 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:38.993061+0000 mon.a (mon.0) 824 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.001691+0000 mon.a (mon.0) 825 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.001922+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.002017+0000 mon.a (mon.0) 827 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:37.994824+0000 mgr.y (mgr.14553) 135 : audit [DBG] from='client.35099 v1:172.21.15.107:0/3146110451' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:40.424 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:38.051427+0000 mgr.y (mgr.14553) 136 : cluster [DBG] pgmap v131: 546 pgs: 290 creating+peering, 1 peering, 126 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.738988+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.739382+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.825717+0000 mon.b (mon.1) 61 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.992233+0000 mon.a (mon.0) 821 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.992534+0000 mon.a (mon.0) 822 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.992848+0000 mon.a (mon.0) 823 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:38.993061+0000 mon.a (mon.0) 824 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.001691+0000 mon.a (mon.0) 825 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi107-112562-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.001922+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.002017+0000 mon.a (mon.0) 827 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.015706+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.021363+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.107:0/1946495091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.529153+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.425 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.529273+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.701817+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.702519+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.702920+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.703304+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.703670+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.704109+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.704522+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.704899+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.705289+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.705669+0000 mon.b (mon.1) 73 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:39.772196+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.015706+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.021363+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.107:0/1946495091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.529153+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.529273+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.701817+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.702519+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.702920+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.703304+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.703670+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.704109+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.704522+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.704899+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:40.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.705289+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.705669+0000 mon.b (mon.1) 73 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:39.772196+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.772737+0000 mon.a (mon.0) 829 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.772798+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.773283+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.773699+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.772737+0000 mon.a (mon.0) 829 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T08:36:40.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.772798+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.773283+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:40.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.773699+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:40.441 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-02-10T08:36:37.917512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":911,"num_read_kb":663,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":1260,"acting":1260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":450,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2382256,"kb_used_data":5412,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747578960,"statfs":{"total":767960285184,"available":765520855040,"internally_reserved":0,"allocated":5541888,"data_stored":2640473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.111972"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918587+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.922516+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917512+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894277+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899007+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805034+0000","last_change":"2024-02-10T08:36:30.805034+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805034+0000","last_undegraded":"2024-02-10T08:36:30.805034+0000","last_fullsized":"2024-02-10T08:36:30.805034+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":108,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526679+0000","last_change":"2024-02-10T08:36:16.169727+0000","last_active":"2024-02-10T08:36:31.526679+0000","last_peered":"2024-02-10T08:36:31.526679+0000","last_clean":"2024-02-10T08:36:31.526679+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:36:31.526679+0000","last_undegraded":"2024-02-10T08:36:31.526679+0000","last_fullsized":"2024-02-10T08:36:31.526679+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:16.169618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.169618+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":90,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806270+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:36:30.806270+0000","last_peered":"2024-02-10T08:36:30.806270+0000","last_clean":"2024-02-10T08:36:30.806270+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:36:30.806270+0000","last_undegraded":"2024-02-10T08:36:30.806270+0000","last_fullsized":"2024-02-10T08:36:30.806270+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.811718+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:36:30.811718+0000","last_peered":"2024-02-10T08:36:30.811718+0000","last_clean":"2024-02-10T08:36:30.811718+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:36:30.811718+0000","last_undegraded":"2024-02-10T08:36:30.811718+0000","last_fullsized":"2024-02-10T08:36:30.811718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812920+0000","last_change":"2024-02-10T08:36:21.704857+0000","last_active":"2024-02-10T08:36:30.812920+0000","last_peered":"2024-02-10T08:36:30.812920+0000","last_clean":"2024-02-10T08:36:30.812920+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:36:30.812920+0000","last_undegraded":"2024-02-10T08:36:30.812920+0000","last_fullsized":"2024-02-10T08:36:30.812920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.704774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.704774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.871362+0000","last_change":"2024-02-10T08:36:35.871362+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.871362+0000","last_undegraded":"2024-02-10T08:36:35.871362+0000","last_fullsized":"2024-02-10T08:36:35.871362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900927+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910534+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.924761+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906408+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894818+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900094+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.871900+0000","last_change":"2024-02-10T08:36:35.871900+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.871900+0000","last_undegraded":"2024-02-10T08:36:35.871900+0000","last_fullsized":"2024-02-10T08:36:35.871900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":97,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810340+0000","last_change":"2024-02-10T08:36:14.826412+0000","last_active":"2024-02-10T08:36:30.810340+0000","last_peered":"2024-02-10T08:36:30.810340+0000","last_clean":"2024-02-10T08:36:30.810340+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:36:30.810340+0000","last_undegraded":"2024-02-10T08:36:30.810340+0000","last_fullsized":"2024-02-10T08:36:30.810340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:14.826321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.826321+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808049+0000","last_change":"2024-02-10T08:35:30.241737+0000","last_active":"2024-02-10T08:36:30.808049+0000","last_peered":"2024-02-10T08:36:30.808049+0000","last_clean":"2024-02-10T08:36:30.808049+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:36:30.808049+0000","last_undegraded":"2024-02-10T08:36:30.808049+0000","last_fullsized":"2024-02-10T08:36:30.808049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.241661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.241661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810170+0000","last_change":"2024-02-10T08:36:17.568277+0000","last_active":"2024-02-10T08:36:30.810170+0000","last_peered":"2024-02-10T08:36:30.810170+0000","last_clean":"2024-02-10T08:36:30.810170+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:36:30.810170+0000","last_undegraded":"2024-02-10T08:36:30.810170+0000","last_fullsized":"2024-02-10T08:36:30.810170+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.568165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.568165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809881+0000","last_change":"2024-02-10T08:35:34.652671+0000","last_active":"2024-02-10T08:36:30.809881+0000","last_peered":"2024-02-10T08:36:30.809881+0000","last_clean":"2024-02-10T08:36:30.809881+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:36:30.809881+0000","last_undegraded":"2024-02-10T08:36:30.809881+0000","last_fullsized":"2024-02-10T08:36:30.809881+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:34.652551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.652551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.809572+0000","last_change":"2024-02-10T08:36:30.809572+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.809572+0000","last_undegraded":"2024-02-10T08:36:30.809572+0000","last_fullsized":"2024-02-10T08:36:30.809572+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902494+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.929373+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.923991+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907948+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894070+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900859+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807003+0000","last_change":"2024-02-10T08:36:30.807003+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807003+0000","last_undegraded":"2024-02-10T08:36:30.807003+0000","last_fullsized":"2024-02-10T08:36:30.807003+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809219+0000","last_change":"2024-02-10T08:36:23.647242+0000","last_active":"2024-02-10T08:36:30.809219+0000","last_peered":"2024-02-10T08:36:30.809219+0000","last_clean":"2024-02-10T08:36:30.809219+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:36:30.809219+0000","last_undegraded":"2024-02-10T08:36:30.809219+0000","last_fullsized":"2024-02-10T08:36:30.809219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:23.647164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.647164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812620+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:36:30.812620+0000","last_peered":"2024-02-10T08:36:30.812620+0000","last_clean":"2024-02-10T08:36:30.812620+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:36:30.812620+0000","last_undegraded":"2024-02-10T08:36:30.812620+0000","last_fullsized":"2024-02-10T08:36:30.812620+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810098+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:36:30.810098+0000","last_peered":"2024-02-10T08:36:30.810098+0000","last_clean":"2024-02-10T08:36:30.810098+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:36:30.810098+0000","last_undegraded":"2024-02-10T08:36:30.810098+0000","last_fullsized":"2024-02-10T08:36:30.810098+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807750+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:36:30.807750+0000","last_peered":"2024-02-10T08:36:30.807750+0000","last_clean":"2024-02-10T08:36:30.807750+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:36:30.807750+0000","last_undegraded":"2024-02-10T08:36:30.807750+0000","last_fullsized":"2024-02-10T08:36:30.807750+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806065+0000","last_change":"2024-02-10T08:36:30.806065+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806065+0000","last_undegraded":"2024-02-10T08:36:30.806065+0000","last_fullsized":"2024-02-10T08:36:30.806065+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913815+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919767+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914131+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.920126+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916867+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904601+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":91,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806064+0000","last_change":"2024-02-10T08:35:30.802385+0000","last_active":"2024-02-10T08:36:30.806064+0000","last_peered":"2024-02-10T08:36:30.806064+0000","last_clean":"2024-02-10T08:36:30.806064+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:36:30.806064+0000","last_undegraded":"2024-02-10T08:36:30.806064+0000","last_fullsized":"2024-02-10T08:36:30.806064+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:35:30.802306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.802306+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809013+0000","last_change":"2024-02-10T08:35:30.123918+0000","last_active":"2024-02-10T08:36:30.809013+0000","last_peered":"2024-02-10T08:36:30.809013+0000","last_clean":"2024-02-10T08:36:30.809013+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:36:30.809013+0000","last_undegraded":"2024-02-10T08:36:30.809013+0000","last_fullsized":"2024-02-10T08:36:30.809013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.123828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.123828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807071+0000","last_change":"2024-02-10T08:35:08.227030+0000","last_active":"2024-02-10T08:36:30.807071+0000","last_peered":"2024-02-10T08:36:30.807071+0000","last_clean":"2024-02-10T08:36:30.807071+0000","last_became_active":"2024-02-10T08:33:22.121165+0000","last_became_peered":"2024-02-10T08:33:22.121165+0000","last_unstale":"2024-02-10T08:36:30.807071+0000","last_undegraded":"2024-02-10T08:36:30.807071+0000","last_fullsized":"2024-02-10T08:36:30.807071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:35.691742+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:36:35.691742+0000","last_peered":"2024-02-10T08:36:35.691742+0000","last_clean":"2024-02-10T08:36:35.691742+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:36:35.691742+0000","last_undegraded":"2024-02-10T08:36:35.691742+0000","last_fullsized":"2024-02-10T08:36:35.691742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.809702+0000","last_change":"2024-02-10T08:36:30.809702+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.809702+0000","last_undegraded":"2024-02-10T08:36:30.809702+0000","last_fullsized":"2024-02-10T08:36:30.809702+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.871363+0000","last_change":"2024-02-10T08:36:35.871363+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.871363+0000","last_undegraded":"2024-02-10T08:36:35.871363+0000","last_fullsized":"2024-02-10T08:36:35.871363+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898086+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917214+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914967+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913646+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.924748+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916439+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904576+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908773+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":78,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.805156+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:36:30.805156+0000","last_peered":"2024-02-10T08:36:30.805156+0000","last_clean":"2024-02-10T08:36:30.805156+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:36:30.805156+0000","last_undegraded":"2024-02-10T08:36:30.805156+0000","last_fullsized":"2024-02-10T08:36:30.805156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806034+0000","last_change":"2024-02-10T08:35:31.222395+0000","last_active":"2024-02-10T08:36:30.806034+0000","last_peered":"2024-02-10T08:36:30.806034+0000","last_clean":"2024-02-10T08:36:30.806034+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:36:30.806034+0000","last_undegraded":"2024-02-10T08:36:30.806034+0000","last_fullsized":"2024-02-10T08:36:30.806034+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:31.222331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:31.222331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"63'8","reported_seq":103,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526470+0000","last_change":"2024-02-10T08:34:51.737455+0000","last_active":"2024-02-10T08:36:31.526470+0000","last_peered":"2024-02-10T08:36:31.526470+0000","last_clean":"2024-02-10T08:36:31.526470+0000","last_became_active":"2024-02-10T08:33:22.123495+0000","last_became_peered":"2024-02-10T08:33:22.123495+0000","last_unstale":"2024-02-10T08:36:31.526470+0000","last_undegraded":"2024-02-10T08:36:31.526470+0000","last_fullsized":"2024-02-10T08:36:31.526470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808219+0000","last_change":"2024-02-10T08:36:13.778764+0000","last_active":"2024-02-10T08:36:30.808219+0000","last_peered":"2024-02-10T08:36:30.808219+0000","last_clean":"2024-02-10T08:36:30.808219+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:36:30.808219+0000","last_undegraded":"2024-02-10T08:36:30.808219+0000","last_fullsized":"2024-02-10T08:36:30.808219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.778673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.778673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807404+0000","last_change":"2024-02-10T08:36:30.807404+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807404+0000","last_undegraded":"2024-02-10T08:36:30.807404+0000","last_fullsized":"2024-02-10T08:36:30.807404+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.811742+0000","last_change":"2024-02-10T08:36:30.811742+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.811742+0000","last_undegraded":"2024-02-10T08:36:30.811742+0000","last_fullsized":"2024-02-10T08:36:30.811742+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916888+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900447+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908167+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917655+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910993+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907981+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.872275+0000","last_change":"2024-02-10T08:36:35.872275+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.872275+0000","last_undegraded":"2024-02-10T08:36:35.872275+0000","last_fullsized":"2024-02-10T08:36:35.872275+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807383+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:36:30.807383+0000","last_peered":"2024-02-10T08:36:30.807383+0000","last_clean":"2024-02-10T08:36:30.807383+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:36:30.807383+0000","last_undegraded":"2024-02-10T08:36:30.807383+0000","last_fullsized":"2024-02-10T08:36:30.807383+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":71,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806153+0000","last_change":"2024-02-10T08:36:16.423854+0000","last_active":"2024-02-10T08:36:30.806153+0000","last_peered":"2024-02-10T08:36:30.806153+0000","last_clean":"2024-02-10T08:36:30.806153+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:36:30.806153+0000","last_undegraded":"2024-02-10T08:36:30.806153+0000","last_fullsized":"2024-02-10T08:36:30.806153+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:16.423786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.423786+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"63'8","reported_seq":103,"reported_epoch":101,"state":"peering","last_fresh":"2024-02-10T08:36:30.808040+0000","last_change":"2024-02-10T08:36:30.808040+0000","last_active":"2024-02-10T08:36:30.807760+0000","last_peered":"2024-02-10T08:36:29.157689+0000","last_clean":"2024-02-10T08:36:29.157689+0000","last_became_active":"2024-02-10T08:33:22.117209+0000","last_became_peered":"2024-02-10T08:33:22.117209+0000","last_unstale":"2024-02-10T08:36:30.808040+0000","last_undegraded":"2024-02-10T08:36:30.808040+0000","last_fullsized":"2024-02-10T08:36:30.808040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813970+0000","last_change":"2024-02-10T08:35:16.510089+0000","last_active":"2024-02-10T08:36:30.813970+0000","last_peered":"2024-02-10T08:36:30.813970+0000","last_clean":"2024-02-10T08:36:30.813970+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:36:30.813970+0000","last_undegraded":"2024-02-10T08:36:30.813970+0000","last_fullsized":"2024-02-10T08:36:30.813970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805424+0000","last_change":"2024-02-10T08:36:30.805424+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805424+0000","last_undegraded":"2024-02-10T08:36:30.805424+0000","last_fullsized":"2024-02-10T08:36:30.805424+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.897399+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911567+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914904+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.926355+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917106+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919781+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898219+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":82,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807148+0000","last_change":"2024-02-10T08:36:11.732545+0000","last_active":"2024-02-10T08:36:30.807148+0000","last_peered":"2024-02-10T08:36:30.807148+0000","last_clean":"2024-02-10T08:36:30.807148+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:36:30.807148+0000","last_undegraded":"2024-02-10T08:36:30.807148+0000","last_fullsized":"2024-02-10T08:36:30.807148+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:36:11.732461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:11.732461+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808000+0000","last_change":"2024-02-10T08:36:20.142711+0000","last_active":"2024-02-10T08:36:30.808000+0000","last_peered":"2024-02-10T08:36:30.808000+0000","last_clean":"2024-02-10T08:36:30.808000+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:36:30.808000+0000","last_undegraded":"2024-02-10T08:36:30.808000+0000","last_fullsized":"2024-02-10T08:36:30.808000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.142541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.142541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526576+0000","last_change":"2024-02-10T08:36:29.158683+0000","last_active":"2024-02-10T08:36:31.526576+0000","last_peered":"2024-02-10T08:36:31.526576+0000","last_clean":"2024-02-10T08:36:31.526576+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:36:31.526576+0000","last_undegraded":"2024-02-10T08:36:31.526576+0000","last_fullsized":"2024-02-10T08:36:31.526576+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817360+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:36:30.817360+0000","last_peered":"2024-02-10T08:36:30.817360+0000","last_clean":"2024-02-10T08:36:30.817360+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:36:30.817360+0000","last_undegraded":"2024-02-10T08:36:30.817360+0000","last_fullsized":"2024-02-10T08:36:30.817360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805235+0000","last_change":"2024-02-10T08:36:30.805235+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805235+0000","last_undegraded":"2024-02-10T08:36:30.805235+0000","last_fullsized":"2024-02-10T08:36:30.805235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.803130+0000","last_change":"2024-02-10T08:36:30.803130+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.803130+0000","last_undegraded":"2024-02-10T08:36:30.803130+0000","last_fullsized":"2024-02-10T08:36:30.803130+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.905875+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912514+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.920038+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916128+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.921479+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911205+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.892974+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.905154+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.804514+0000","last_change":"2024-02-10T08:36:30.804514+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.804514+0000","last_undegraded":"2024-02-10T08:36:30.804514+0000","last_fullsized":"2024-02-10T08:36:30.804514+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":110,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526485+0000","last_change":"2024-02-10T08:36:28.076465+0000","last_active":"2024-02-10T08:36:31.526485+0000","last_peered":"2024-02-10T08:36:31.526485+0000","last_clean":"2024-02-10T08:36:31.526485+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:36:31.526485+0000","last_undegraded":"2024-02-10T08:36:31.526485+0000","last_fullsized":"2024-02-10T08:36:31.526485+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:28.076401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.076401+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":84,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806360+0000","last_change":"2024-02-10T08:36:17.729319+0000","last_active":"2024-02-10T08:36:30.806360+0000","last_peered":"2024-02-10T08:36:30.806360+0000","last_clean":"2024-02-10T08:36:30.806360+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:36:30.806360+0000","last_undegraded":"2024-02-10T08:36:30.806360+0000","last_fullsized":"2024-02-10T08:36:30.806360+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:17.729249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.729249+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808271+0000","last_change":"2024-02-10T08:35:38.547170+0000","last_active":"2024-02-10T08:36:30.808271+0000","last_peered":"2024-02-10T08:36:30.808271+0000","last_clean":"2024-02-10T08:36:30.808271+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:36:30.808271+0000","last_undegraded":"2024-02-10T08:36:30.808271+0000","last_fullsized":"2024-02-10T08:36:30.808271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:38.547094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.547094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809316+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:36:30.809316+0000","last_peered":"2024-02-10T08:36:30.809316+0000","last_clean":"2024-02-10T08:36:30.809316+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:36:30.809316+0000","last_undegraded":"2024-02-10T08:36:30.809316+0000","last_fullsized":"2024-02-10T08:36:30.809316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.812623+0000","last_change":"2024-02-10T08:36:30.812623+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.812623+0000","last_undegraded":"2024-02-10T08:36:30.812623+0000","last_fullsized":"2024-02-10T08:36:30.812623+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910988+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904808+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908330+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.909941+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915446+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.920689+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.905170+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899264+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807037+0000","last_change":"2024-02-10T08:36:30.807037+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807037+0000","last_undegraded":"2024-02-10T08:36:30.807037+0000","last_fullsized":"2024-02-10T08:36:30.807037+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809166+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:36:30.809166+0000","last_peered":"2024-02-10T08:36:30.809166+0000","last_clean":"2024-02-10T08:36:30.809166+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:36:30.809166+0000","last_undegraded":"2024-02-10T08:36:30.809166+0000","last_fullsized":"2024-02-10T08:36:30.809166+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":97,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816374+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:36:30.816374+0000","last_peered":"2024-02-10T08:36:30.816374+0000","last_clean":"2024-02-10T08:36:30.816374+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:36:30.816374+0000","last_undegraded":"2024-02-10T08:36:30.816374+0000","last_fullsized":"2024-02-10T08:36:30.816374+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:32.450940+0000","last_change":"2024-02-10T08:36:32.450905+0000","last_active":"2024-02-10T08:36:32.450940+0000","last_peered":"2024-02-10T08:36:32.450940+0000","last_clean":"2024-02-10T08:36:32.450940+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:36:32.450940+0000","last_undegraded":"2024-02-10T08:36:32.450940+0000","last_fullsized":"2024-02-10T08:36:32.450940+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:32.450807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.450807+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.804883+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:36:30.804883+0000","last_peered":"2024-02-10T08:36:30.804883+0000","last_clean":"2024-02-10T08:36:30.804883+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:36:30.804883+0000","last_undegraded":"2024-02-10T08:36:30.804883+0000","last_fullsized":"2024-02-10T08:36:30.804883+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.813025+0000","last_change":"2024-02-10T08:36:30.813025+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.813025+0000","last_undegraded":"2024-02-10T08:36:30.813025+0000","last_fullsized":"2024-02-10T08:36:30.813025+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.909963+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899469+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918123+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.930521+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919847+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907577+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903579+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902109+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.814353+0000","last_change":"2024-02-10T08:36:30.814353+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.814353+0000","last_undegraded":"2024-02-10T08:36:30.814353+0000","last_fullsized":"2024-02-10T08:36:30.814353+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809397+0000","last_change":"2024-02-10T08:36:26.578369+0000","last_active":"2024-02-10T08:36:30.809397+0000","last_peered":"2024-02-10T08:36:30.809397+0000","last_clean":"2024-02-10T08:36:30.809397+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:36:30.809397+0000","last_undegraded":"2024-02-10T08:36:30.809397+0000","last_fullsized":"2024-02-10T08:36:30.809397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.578284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.578284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":98,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810181+0000","last_change":"2024-02-10T08:36:15.871132+0000","last_active":"2024-02-10T08:36:30.810181+0000","last_peered":"2024-02-10T08:36:30.810181+0000","last_clean":"2024-02-10T08:36:30.810181+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:36:30.810181+0000","last_undegraded":"2024-02-10T08:36:30.810181+0000","last_fullsized":"2024-02-10T08:36:30.810181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:15.871060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.871060+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808350+0000","last_change":"2024-02-10T08:36:26.048208+0000","last_active":"2024-02-10T08:36:30.808350+0000","last_peered":"2024-02-10T08:36:30.808350+0000","last_clean":"2024-02-10T08:36:30.808350+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:36:30.808350+0000","last_undegraded":"2024-02-10T08:36:30.808350+0000","last_fullsized":"2024-02-10T08:36:30.808350+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.048119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.048119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806004+0000","last_change":"2024-02-10T08:36:20.500433+0000","last_active":"2024-02-10T08:36:30.806004+0000","last_peered":"2024-02-10T08:36:30.806004+0000","last_clean":"2024-02-10T08:36:30.806004+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:36:30.806004+0000","last_undegraded":"2024-02-10T08:36:30.806004+0000","last_fullsized":"2024-02-10T08:36:30.806004+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.500313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.804825+0000","last_change":"2024-02-10T08:36:30.804825+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.804825+0000","last_undegraded":"2024-02-10T08:36:30.804825+0000","last_fullsized":"2024-02-10T08:36:30.804825+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900978+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899722+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917967+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911137+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903277+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903793+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.896354+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.872274+0000","last_change":"2024-02-10T08:36:35.872274+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.872274+0000","last_undegraded":"2024-02-10T08:36:35.872274+0000","last_fullsized":"2024-02-10T08:36:35.872274+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.814211+0000","last_change":"2024-02-10T08:36:22.714387+0000","last_active":"2024-02-10T08:36:30.814211+0000","last_peered":"2024-02-10T08:36:30.814211+0000","last_clean":"2024-02-10T08:36:30.814211+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:36:30.814211+0000","last_undegraded":"2024-02-10T08:36:30.814211+0000","last_fullsized":"2024-02-10T08:36:30.814211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.714301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.714301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813706+0000","last_change":"2024-02-10T08:35:36.647820+0000","last_active":"2024-02-10T08:36:30.813706+0000","last_peered":"2024-02-10T08:36:30.813706+0000","last_clean":"2024-02-10T08:36:30.813706+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:36:30.813706+0000","last_undegraded":"2024-02-10T08:36:30.813706+0000","last_fullsized":"2024-02-10T08:36:30.813706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:35:36.647719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.647719+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807839+0000","last_change":"2024-02-10T08:35:27.051918+0000","last_active":"2024-02-10T08:36:30.807839+0000","last_peered":"2024-02-10T08:36:30.807839+0000","last_clean":"2024-02-10T08:36:30.807839+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:36:30.807839+0000","last_undegraded":"2024-02-10T08:36:30.807839+0000","last_fullsized":"2024-02-10T08:36:30.807839+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:27.051837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:27.051837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809678+0000","last_change":"2024-02-10T08:35:41.795140+0000","last_active":"2024-02-10T08:36:30.809678+0000","last_peered":"2024-02-10T08:36:30.809678+0000","last_clean":"2024-02-10T08:36:30.809678+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:36:30.809678+0000","last_undegraded":"2024-02-10T08:36:30.809678+0000","last_fullsized":"2024-02-10T08:36:30.809678+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807145+0000","last_change":"2024-02-10T08:35:38.321759+0000","last_active":"2024-02-10T08:36:30.807145+0000","last_peered":"2024-02-10T08:36:30.807145+0000","last_clean":"2024-02-10T08:36:30.807145+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:36:30.807145+0000","last_undegraded":"2024-02-10T08:36:30.807145+0000","last_fullsized":"2024-02-10T08:36:30.807145+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:38.321694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.321694+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806394+0000","last_change":"2024-02-10T08:36:30.806394+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806394+0000","last_undegraded":"2024-02-10T08:36:30.806394+0000","last_fullsized":"2024-02-10T08:36:30.806394+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914690+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902887+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913940+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.930482+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912408+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.924587+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908337+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903656+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806379+0000","last_change":"2024-02-10T08:36:30.806379+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806379+0000","last_undegraded":"2024-02-10T08:36:30.806379+0000","last_fullsized":"2024-02-10T08:36:30.806379+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809612+0000","last_change":"2024-02-10T08:36:27.052479+0000","last_active":"2024-02-10T08:36:30.809612+0000","last_peered":"2024-02-10T08:36:30.809612+0000","last_clean":"2024-02-10T08:36:30.809612+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:36:30.809612+0000","last_undegraded":"2024-02-10T08:36:30.809612+0000","last_fullsized":"2024-02-10T08:36:30.809612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.052362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.052362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807026+0000","last_change":"2024-02-10T08:36:28.635087+0000","last_active":"2024-02-10T08:36:30.807026+0000","last_peered":"2024-02-10T08:36:30.807026+0000","last_clean":"2024-02-10T08:36:30.807026+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:36:30.807026+0000","last_undegraded":"2024-02-10T08:36:30.807026+0000","last_fullsized":"2024-02-10T08:36:30.807026+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:28.635015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635015+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807922+0000","last_change":"2024-02-10T08:36:25.053910+0000","last_active":"2024-02-10T08:36:30.807922+0000","last_peered":"2024-02-10T08:36:30.807922+0000","last_clean":"2024-02-10T08:36:30.807922+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:36:30.807922+0000","last_undegraded":"2024-02-10T08:36:30.807922+0000","last_fullsized":"2024-02-10T08:36:30.807922+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.053809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.053809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813856+0000","last_change":"2024-02-10T08:36:30.795587+0000","last_active":"2024-02-10T08:36:30.813856+0000","last_peered":"2024-02-10T08:36:30.813856+0000","last_clean":"2024-02-10T08:36:30.813856+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:36:30.813856+0000","last_undegraded":"2024-02-10T08:36:30.813856+0000","last_fullsized":"2024-02-10T08:36:30.813856+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817260+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:36:30.817260+0000","last_peered":"2024-02-10T08:36:30.817260+0000","last_clean":"2024-02-10T08:36:30.817260+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:36:30.817260+0000","last_undegraded":"2024-02-10T08:36:30.817260+0000","last_fullsized":"2024-02-10T08:36:30.817260+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.814332+0000","last_change":"2024-02-10T08:36:29.804845+0000","last_active":"2024-02-10T08:36:30.814332+0000","last_peered":"2024-02-10T08:36:30.814332+0000","last_clean":"2024-02-10T08:36:30.814332+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:36:30.814332+0000","last_undegraded":"2024-02-10T08:36:30.814332+0000","last_fullsized":"2024-02-10T08:36:30.814332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:29.804761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.804761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806674+0000","last_change":"2024-02-10T08:36:30.806674+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806674+0000","last_undegraded":"2024-02-10T08:36:30.806674+0000","last_fullsized":"2024-02-10T08:36:30.806674+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899687+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903294+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906572+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916824+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907923+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899830+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.813388+0000","last_change":"2024-02-10T08:36:30.813388+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.813388+0000","last_undegraded":"2024-02-10T08:36:30.813388+0000","last_fullsized":"2024-02-10T08:36:30.813388+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812415+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:36:30.812415+0000","last_peered":"2024-02-10T08:36:30.812415+0000","last_clean":"2024-02-10T08:36:30.812415+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:36:30.812415+0000","last_undegraded":"2024-02-10T08:36:30.812415+0000","last_fullsized":"2024-02-10T08:36:30.812415+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":90,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807408+0000","last_change":"2024-02-10T08:35:42.326705+0000","last_active":"2024-02-10T08:36:30.807408+0000","last_peered":"2024-02-10T08:36:30.807408+0000","last_clean":"2024-02-10T08:36:30.807408+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:36:30.807408+0000","last_undegraded":"2024-02-10T08:36:30.807408+0000","last_fullsized":"2024-02-10T08:36:30.807408+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:35:42.326627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.326627+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810025+0000","last_change":"2024-02-10T08:36:28.635691+0000","last_active":"2024-02-10T08:36:30.810025+0000","last_peered":"2024-02-10T08:36:30.810025+0000","last_clean":"2024-02-10T08:36:30.810025+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:36:30.810025+0000","last_undegraded":"2024-02-10T08:36:30.810025+0000","last_fullsized":"2024-02-10T08:36:30.810025+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:28.635610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":260,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:35.400261+0000","last_change":"2024-02-10T08:36:33.452480+0000","last_active":"2024-02-10T08:36:35.400261+0000","last_peered":"2024-02-10T08:36:35.400261+0000","last_clean":"2024-02-10T08:36:35.400261+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:36:35.400261+0000","last_undegraded":"2024-02-10T08:36:35.400261+0000","last_fullsized":"2024-02-10T08:36:35.400261+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.452367+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":191,"num_read_kb":186,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:32.132864+0000","last_change":"2024-02-10T08:36:32.132836+0000","last_active":"2024-02-10T08:36:32.132864+0000","last_peered":"2024-02-10T08:36:32.132864+0000","last_clean":"2024-02-10T08:36:32.132864+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:36:32.132864+0000","last_undegraded":"2024-02-10T08:36:32.132864+0000","last_fullsized":"2024-02-10T08:36:32.132864+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:32.132760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.132760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806734+0000","last_change":"2024-02-10T08:36:30.806734+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806734+0000","last_undegraded":"2024-02-10T08:36:30.806734+0000","last_fullsized":"2024-02-10T08:36:30.806734+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917412+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913171+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.929734+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908139+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898401+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894702+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805247+0000","last_change":"2024-02-10T08:36:30.805247+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805247+0000","last_undegraded":"2024-02-10T08:36:30.805247+0000","last_fullsized":"2024-02-10T08:36:30.805247+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812828+0000","last_change":"2024-02-10T08:36:26.962399+0000","last_active":"2024-02-10T08:36:30.812828+0000","last_peered":"2024-02-10T08:36:30.812828+0000","last_clean":"2024-02-10T08:36:30.812828+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:36:30.812828+0000","last_undegraded":"2024-02-10T08:36:30.812828+0000","last_fullsized":"2024-02-10T08:36:30.812828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.962334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.962334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":89,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526662+0000","last_change":"2024-02-10T08:36:17.572570+0000","last_active":"2024-02-10T08:36:31.526662+0000","last_peered":"2024-02-10T08:36:31.526662+0000","last_clean":"2024-02-10T08:36:31.526662+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:36:31.526662+0000","last_undegraded":"2024-02-10T08:36:31.526662+0000","last_fullsized":"2024-02-10T08:36:31.526662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:36:17.572475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.572475+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813411+0000","last_change":"2024-02-10T08:35:42.473094+0000","last_active":"2024-02-10T08:36:30.813411+0000","last_peered":"2024-02-10T08:36:30.813411+0000","last_clean":"2024-02-10T08:36:30.813411+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:36:30.813411+0000","last_undegraded":"2024-02-10T08:36:30.813411+0000","last_fullsized":"2024-02-10T08:36:30.813411+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.473021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817136+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:36:30.817136+0000","last_peered":"2024-02-10T08:36:30.817136+0000","last_clean":"2024-02-10T08:36:30.817136+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:36:30.817136+0000","last_undegraded":"2024-02-10T08:36:30.817136+0000","last_fullsized":"2024-02-10T08:36:30.817136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807664+0000","last_change":"2024-02-10T08:36:30.807664+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807664+0000","last_undegraded":"2024-02-10T08:36:30.807664+0000","last_fullsized":"2024-02-10T08:36:30.807664+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900859+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913242+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906544+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907729+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.924345+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917158+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.896180+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.901700+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.808620+0000","last_change":"2024-02-10T08:36:30.808620+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.808620+0000","last_undegraded":"2024-02-10T08:36:30.808620+0000","last_fullsized":"2024-02-10T08:36:30.808620+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":107,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526479+0000","last_change":"2024-02-10T08:36:27.957994+0000","last_active":"2024-02-10T08:36:31.526479+0000","last_peered":"2024-02-10T08:36:31.526479+0000","last_clean":"2024-02-10T08:36:31.526479+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:36:31.526479+0000","last_undegraded":"2024-02-10T08:36:31.526479+0000","last_fullsized":"2024-02-10T08:36:31.526479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:27.957926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.957926+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806269+0000","last_change":"2024-02-10T08:35:34.249489+0000","last_active":"2024-02-10T08:36:30.806269+0000","last_peered":"2024-02-10T08:36:30.806269+0000","last_clean":"2024-02-10T08:36:30.806269+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:36:30.806269+0000","last_undegraded":"2024-02-10T08:36:30.806269+0000","last_fullsized":"2024-02-10T08:36:30.806269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.249424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806877+0000","last_change":"2024-02-10T08:36:26.456845+0000","last_active":"2024-02-10T08:36:30.806877+0000","last_peered":"2024-02-10T08:36:30.806877+0000","last_clean":"2024-02-10T08:36:30.806877+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:36:30.806877+0000","last_undegraded":"2024-02-10T08:36:30.806877+0000","last_fullsized":"2024-02-10T08:36:30.806877+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.456780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809896+0000","last_change":"2024-02-10T08:36:13.802328+0000","last_active":"2024-02-10T08:36:30.809896+0000","last_peered":"2024-02-10T08:36:30.809896+0000","last_clean":"2024-02-10T08:36:30.809896+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:36:30.809896+0000","last_undegraded":"2024-02-10T08:36:30.809896+0000","last_fullsized":"2024-02-10T08:36:30.809896+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.802264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.802264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.802307+0000","last_change":"2024-02-10T08:36:30.802307+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.802307+0000","last_undegraded":"2024-02-10T08:36:30.802307+0000","last_fullsized":"2024-02-10T08:36:30.802307+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904244+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919401+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919030+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.931595+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910843+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904085+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805924+0000","last_change":"2024-02-10T08:36:30.805924+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805924+0000","last_undegraded":"2024-02-10T08:36:30.805924+0000","last_fullsized":"2024-02-10T08:36:30.805924+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810208+0000","last_change":"2024-02-10T08:36:16.546775+0000","last_active":"2024-02-10T08:36:30.810208+0000","last_peered":"2024-02-10T08:36:30.810208+0000","last_clean":"2024-02-10T08:36:30.810208+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:36:30.810208+0000","last_undegraded":"2024-02-10T08:36:30.810208+0000","last_fullsized":"2024-02-10T08:36:30.810208+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:16.546674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.546674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":91,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806483+0000","last_change":"2024-02-10T08:36:28.744540+0000","last_active":"2024-02-10T08:36:30.806483+0000","last_peered":"2024-02-10T08:36:30.806483+0000","last_clean":"2024-02-10T08:36:30.806483+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:36:30.806483+0000","last_undegraded":"2024-02-10T08:36:30.806483+0000","last_fullsized":"2024-02-10T08:36:30.806483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:36:28.744465+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.744465+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":73,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.805538+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:36:30.805538+0000","last_peered":"2024-02-10T08:36:30.805538+0000","last_clean":"2024-02-10T08:36:30.805538+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:36:30.805538+0000","last_undegraded":"2024-02-10T08:36:30.805538+0000","last_fullsized":"2024-02-10T08:36:30.805538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817015+0000","last_change":"2024-02-10T08:36:23.084433+0000","last_active":"2024-02-10T08:36:30.817015+0000","last_peered":"2024-02-10T08:36:30.817015+0000","last_clean":"2024-02-10T08:36:30.817015+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:36:30.817015+0000","last_undegraded":"2024-02-10T08:36:30.817015+0000","last_fullsized":"2024-02-10T08:36:30.817015+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:23.084331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.084331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.812303+0000","last_change":"2024-02-10T08:36:30.812303+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.812303+0000","last_undegraded":"2024-02-10T08:36:30.812303+0000","last_fullsized":"2024-02-10T08:36:30.812303+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900956+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913957+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914619+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919412+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894542+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903874+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806128+0000","last_change":"2024-02-10T08:36:30.806128+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806128+0000","last_undegraded":"2024-02-10T08:36:30.806128+0000","last_fullsized":"2024-02-10T08:36:30.806128+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":82,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813794+0000","last_change":"2024-02-10T08:35:33.677439+0000","last_active":"2024-02-10T08:36:30.813794+0000","last_peered":"2024-02-10T08:36:30.813794+0000","last_clean":"2024-02-10T08:36:30.813794+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:36:30.813794+0000","last_undegraded":"2024-02-10T08:36:30.813794+0000","last_fullsized":"2024-02-10T08:36:30.813794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:35:33.677377+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810116+0000","last_change":"2024-02-10T08:36:22.639484+0000","last_active":"2024-02-10T08:36:30.810116+0000","last_peered":"2024-02-10T08:36:30.810116+0000","last_clean":"2024-02-10T08:36:30.810116+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:36:30.810116+0000","last_undegraded":"2024-02-10T08:36:30.810116+0000","last_fullsized":"2024-02-10T08:36:30.810116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.639381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.639381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.814299+0000","last_change":"2024-02-10T08:36:23.932743+0000","last_active":"2024-02-10T08:36:30.814299+0000","last_peered":"2024-02-10T08:36:30.814299+0000","last_clean":"2024-02-10T08:36:30.814299+0000","last_became_active":"2024-02-10T08:36:23.932443+0000","last_became_peered":"2024-02-10T08:36:23.932443+0000","last_unstale":"2024-02-10T08:36:30.814299+0000","last_undegraded":"2024-02-10T08:36:30.814299+0000","last_fullsized":"2024-02-10T08:36:30.814299+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810426+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:36:30.810426+0000","last_peered":"2024-02-10T08:36:30.810426+0000","last_clean":"2024-02-10T08:36:30.810426+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:36:30.810426+0000","last_undegraded":"2024-02-10T08:36:30.810426+0000","last_fullsized":"2024-02-10T08:36:30.810426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805946+0000","last_change":"2024-02-10T08:36:30.805946+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805946+0000","last_undegraded":"2024-02-10T08:36:30.805946+0000","last_fullsized":"2024-02-10T08:36:30.805946+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898053+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911433+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906774+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917241+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.926020+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914819+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.909070+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907660+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.803102+0000","last_change":"2024-02-10T08:36:30.803102+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.803102+0000","last_undegraded":"2024-02-10T08:36:30.803102+0000","last_fullsized":"2024-02-10T08:36:30.803102+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816995+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:36:30.816995+0000","last_peered":"2024-02-10T08:36:30.816995+0000","last_clean":"2024-02-10T08:36:30.816995+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:36:30.816995+0000","last_undegraded":"2024-02-10T08:36:30.816995+0000","last_fullsized":"2024-02-10T08:36:30.816995+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809853+0000","last_change":"2024-02-10T08:36:19.570275+0000","last_active":"2024-02-10T08:36:30.809853+0000","last_peered":"2024-02-10T08:36:30.809853+0000","last_clean":"2024-02-10T08:36:30.809853+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:36:30.809853+0000","last_undegraded":"2024-02-10T08:36:30.809853+0000","last_fullsized":"2024-02-10T08:36:30.809853+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:19.570181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:19.570181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812200+0000","last_change":"2024-02-10T08:36:23.671449+0000","last_active":"2024-02-10T08:36:30.812200+0000","last_peered":"2024-02-10T08:36:30.812200+0000","last_clean":"2024-02-10T08:36:30.812200+0000","last_became_active":"2024-02-10T08:36:23.671134+0000","last_became_peered":"2024-02-10T08:36:23.671134+0000","last_unstale":"2024-02-10T08:36:30.812200+0000","last_undegraded":"2024-02-10T08:36:30.812200+0000","last_fullsized":"2024-02-10T08:36:30.812200+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.616139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.616139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807952+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:36:30.807952+0000","last_peered":"2024-02-10T08:36:30.807952+0000","last_clean":"2024-02-10T08:36:30.807952+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:36:30.807952+0000","last_undegraded":"2024-02-10T08:36:30.807952+0000","last_fullsized":"2024-02-10T08:36:30.807952+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.814093+0000","last_change":"2024-02-10T08:36:30.814093+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.814093+0000","last_undegraded":"2024-02-10T08:36:30.814093+0000","last_fullsized":"2024-02-10T08:36:30.814093+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.896981+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911317+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906391+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918836+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.928999+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914937+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898747+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902989+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.802310+0000","last_change":"2024-02-10T08:36:30.802310+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.802310+0000","last_undegraded":"2024-02-10T08:36:30.802310+0000","last_fullsized":"2024-02-10T08:36:30.802310+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":77,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817440+0000","last_change":"2024-02-10T08:36:14.035460+0000","last_active":"2024-02-10T08:36:30.817440+0000","last_peered":"2024-02-10T08:36:30.817440+0000","last_clean":"2024-02-10T08:36:30.817440+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:36:30.817440+0000","last_undegraded":"2024-02-10T08:36:30.817440+0000","last_fullsized":"2024-02-10T08:36:30.817440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:14.035356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.035356+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813024+0000","last_change":"2024-02-10T08:36:12.633463+0000","last_active":"2024-02-10T08:36:30.813024+0000","last_peered":"2024-02-10T08:36:30.813024+0000","last_clean":"2024-02-10T08:36:30.813024+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:36:30.813024+0000","last_undegraded":"2024-02-10T08:36:30.813024+0000","last_fullsized":"2024-02-10T08:36:30.813024+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:12.633368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:12.633368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":103,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526649+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:36:31.526649+0000","last_peered":"2024-02-10T08:36:31.526649+0000","last_clean":"2024-02-10T08:36:31.526649+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:36:31.526649+0000","last_undegraded":"2024-02-10T08:36:31.526649+0000","last_fullsized":"2024-02-10T08:36:31.526649+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:33.928778+0000","last_change":"2024-02-10T08:36:33.928755+0000","last_active":"2024-02-10T08:36:33.928778+0000","last_peered":"2024-02-10T08:36:33.928778+0000","last_clean":"2024-02-10T08:36:33.928778+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:36:33.928778+0000","last_undegraded":"2024-02-10T08:36:33.928778+0000","last_fullsized":"2024-02-10T08:36:33.928778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.928693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805638+0000","last_change":"2024-02-10T08:36:30.805638+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805638+0000","last_undegraded":"2024-02-10T08:36:30.805638+0000","last_fullsized":"2024-02-10T08:36:30.805638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915433+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910644+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912363+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919007+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912812+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915069+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.897903+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908339+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.804495+0000","last_change":"2024-02-10T08:36:30.804495+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.804495+0000","last_undegraded":"2024-02-10T08:36:30.804495+0000","last_fullsized":"2024-02-10T08:36:30.804495+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":74,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813213+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:36:30.813213+0000","last_peered":"2024-02-10T08:36:30.813213+0000","last_clean":"2024-02-10T08:36:30.813213+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:36:30.813213+0000","last_undegraded":"2024-02-10T08:36:30.813213+0000","last_fullsized":"2024-02-10T08:36:30.813213+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810259+0000","last_change":"2024-02-10T08:35:45.760941+0000","last_active":"2024-02-10T08:36:30.810259+0000","last_peered":"2024-02-10T08:36:30.810259+0000","last_clean":"2024-02-10T08:36:30.810259+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:36:30.810259+0000","last_undegraded":"2024-02-10T08:36:30.810259+0000","last_fullsized":"2024-02-10T08:36:30.810259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.760869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":107,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.526476+0000","last_change":"2024-02-10T08:36:29.049034+0000","last_active":"2024-02-10T08:36:31.526476+0000","last_peered":"2024-02-10T08:36:31.526476+0000","last_clean":"2024-02-10T08:36:31.526476+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:36:31.526476+0000","last_undegraded":"2024-02-10T08:36:31.526476+0000","last_fullsized":"2024-02-10T08:36:31.526476+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:29.048965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.048965+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.805878+0000","last_change":"2024-02-10T08:35:29.834206+0000","last_active":"2024-02-10T08:36:30.805878+0000","last_peered":"2024-02-10T08:36:30.805878+0000","last_clean":"2024-02-10T08:36:30.805878+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:36:30.805878+0000","last_undegraded":"2024-02-10T08:36:30.805878+0000","last_fullsized":"2024-02-10T08:36:30.805878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:29.834114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:29.834114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.802765+0000","last_change":"2024-02-10T08:36:30.802765+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.802765+0000","last_undegraded":"2024-02-10T08:36:30.802765+0000","last_fullsized":"2024-02-10T08:36:30.802765+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.898851+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906787+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907279+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908016+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918087+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919799+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.895597+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.814704+0000","last_change":"2024-02-10T08:36:30.814704+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.814704+0000","last_undegraded":"2024-02-10T08:36:30.814704+0000","last_fullsized":"2024-02-10T08:36:30.814704+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.434199+0000","last_change":"2024-02-10T08:36:31.434160+0000","last_active":"2024-02-10T08:36:31.434199+0000","last_peered":"2024-02-10T08:36:31.434199+0000","last_clean":"2024-02-10T08:36:31.434199+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:36:31.434199+0000","last_undegraded":"2024-02-10T08:36:31.434199+0000","last_fullsized":"2024-02-10T08:36:31.434199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.434047+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806619+0000","last_change":"2024-02-10T08:35:37.562620+0000","last_active":"2024-02-10T08:36:30.806619+0000","last_peered":"2024-02-10T08:36:30.806619+0000","last_clean":"2024-02-10T08:36:30.806619+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:36:30.806619+0000","last_undegraded":"2024-02-10T08:36:30.806619+0000","last_fullsized":"2024-02-10T08:36:30.806619+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:37.562545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:37.562545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816934+0000","last_change":"2024-02-10T08:35:43.333489+0000","last_active":"2024-02-10T08:36:30.816934+0000","last_peered":"2024-02-10T08:36:30.816934+0000","last_clean":"2024-02-10T08:36:30.816934+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:36:30.816934+0000","last_undegraded":"2024-02-10T08:36:30.816934+0000","last_fullsized":"2024-02-10T08:36:30.816934+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809815+0000","last_change":"2024-02-10T08:36:24.978997+0000","last_active":"2024-02-10T08:36:30.809815+0000","last_peered":"2024-02-10T08:36:30.809815+0000","last_clean":"2024-02-10T08:36:30.809815+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:36:30.809815+0000","last_undegraded":"2024-02-10T08:36:30.809815+0000","last_fullsized":"2024-02-10T08:36:30.809815+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.978888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.809226+0000","last_change":"2024-02-10T08:36:30.809226+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.809226+0000","last_undegraded":"2024-02-10T08:36:30.809226+0000","last_fullsized":"2024-02-10T08:36:30.809226+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911734+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910974+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918681+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918419+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907422+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915557+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.893927+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.903330+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807707+0000","last_change":"2024-02-10T08:36:30.807707+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807707+0000","last_undegraded":"2024-02-10T08:36:30.807707+0000","last_fullsized":"2024-02-10T08:36:30.807707+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":77,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.804890+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:36:30.804890+0000","last_peered":"2024-02-10T08:36:30.804890+0000","last_clean":"2024-02-10T08:36:30.804890+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:36:30.804890+0000","last_undegraded":"2024-02-10T08:36:30.804890+0000","last_fullsized":"2024-02-10T08:36:30.804890+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807966+0000","last_change":"2024-02-10T08:36:15.453264+0000","last_active":"2024-02-10T08:36:30.807966+0000","last_peered":"2024-02-10T08:36:30.807966+0000","last_clean":"2024-02-10T08:36:30.807966+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:36:30.807966+0000","last_undegraded":"2024-02-10T08:36:30.807966+0000","last_fullsized":"2024-02-10T08:36:30.807966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.453161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.453161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806198+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:36:30.806198+0000","last_peered":"2024-02-10T08:36:30.806198+0000","last_clean":"2024-02-10T08:36:30.806198+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:36:30.806198+0000","last_undegraded":"2024-02-10T08:36:30.806198+0000","last_fullsized":"2024-02-10T08:36:30.806198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816408+0000","last_change":"2024-02-10T08:36:24.066706+0000","last_active":"2024-02-10T08:36:30.816408+0000","last_peered":"2024-02-10T08:36:30.816408+0000","last_clean":"2024-02-10T08:36:30.816408+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:36:30.816408+0000","last_undegraded":"2024-02-10T08:36:30.816408+0000","last_fullsized":"2024-02-10T08:36:30.816408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.066605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.066605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.811270+0000","last_change":"2024-02-10T08:36:30.811270+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.811270+0000","last_undegraded":"2024-02-10T08:36:30.811270+0000","last_fullsized":"2024-02-10T08:36:30.811270+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904883+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912095+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.924322+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.923496+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915068+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.922406+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.901845+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805576+0000","last_change":"2024-02-10T08:36:30.805576+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805576+0000","last_undegraded":"2024-02-10T08:36:30.805576+0000","last_fullsized":"2024-02-10T08:36:30.805576+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816163+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:36:30.816163+0000","last_peered":"2024-02-10T08:36:30.816163+0000","last_clean":"2024-02-10T08:36:30.816163+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:36:30.816163+0000","last_undegraded":"2024-02-10T08:36:30.816163+0000","last_fullsized":"2024-02-10T08:36:30.816163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.813538+0000","last_change":"2024-02-10T08:36:15.645089+0000","last_active":"2024-02-10T08:36:30.813538+0000","last_peered":"2024-02-10T08:36:30.813538+0000","last_clean":"2024-02-10T08:36:30.813538+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:36:30.813538+0000","last_undegraded":"2024-02-10T08:36:30.813538+0000","last_fullsized":"2024-02-10T08:36:30.813538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.644998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.644998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:33.824616+0000","last_change":"2024-02-10T08:36:33.824585+0000","last_active":"2024-02-10T08:36:33.824616+0000","last_peered":"2024-02-10T08:36:33.824616+0000","last_clean":"2024-02-10T08:36:33.824616+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:36:33.824616+0000","last_undegraded":"2024-02-10T08:36:33.824616+0000","last_fullsized":"2024-02-10T08:36:33.824616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.824511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.824511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808340+0000","last_change":"2024-02-10T08:36:24.096165+0000","last_active":"2024-02-10T08:36:30.808340+0000","last_peered":"2024-02-10T08:36:30.808340+0000","last_clean":"2024-02-10T08:36:30.808340+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:36:30.808340+0000","last_undegraded":"2024-02-10T08:36:30.808340+0000","last_fullsized":"2024-02-10T08:36:30.808340+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.096104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.096104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.808049+0000","last_change":"2024-02-10T08:36:30.808049+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.808049+0000","last_undegraded":"2024-02-10T08:36:30.808049+0000","last_fullsized":"2024-02-10T08:36:30.808049+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.904027+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917050+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919028+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915784+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.921739+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.895425+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.897370+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.809093+0000","last_change":"2024-02-10T08:36:30.809093+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.809093+0000","last_undegraded":"2024-02-10T08:36:30.809093+0000","last_fullsized":"2024-02-10T08:36:30.809093+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816703+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:36:30.816703+0000","last_peered":"2024-02-10T08:36:30.816703+0000","last_clean":"2024-02-10T08:36:30.816703+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:36:30.816703+0000","last_undegraded":"2024-02-10T08:36:30.816703+0000","last_fullsized":"2024-02-10T08:36:30.816703+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808191+0000","last_change":"2024-02-10T08:36:13.537919+0000","last_active":"2024-02-10T08:36:30.808191+0000","last_peered":"2024-02-10T08:36:30.808191+0000","last_clean":"2024-02-10T08:36:30.808191+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:36:30.808191+0000","last_undegraded":"2024-02-10T08:36:30.808191+0000","last_fullsized":"2024-02-10T08:36:30.808191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.537840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.537840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808174+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:36:30.808174+0000","last_peered":"2024-02-10T08:36:30.808174+0000","last_clean":"2024-02-10T08:36:30.808174+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:36:30.808174+0000","last_undegraded":"2024-02-10T08:36:30.808174+0000","last_fullsized":"2024-02-10T08:36:30.808174+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:33.042685+0000","last_change":"2024-02-10T08:36:33.042649+0000","last_active":"2024-02-10T08:36:33.042685+0000","last_peered":"2024-02-10T08:36:33.042685+0000","last_clean":"2024-02-10T08:36:33.042685+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:36:33.042685+0000","last_undegraded":"2024-02-10T08:36:33.042685+0000","last_fullsized":"2024-02-10T08:36:33.042685+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.042582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.042582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806837+0000","last_change":"2024-02-10T08:36:30.806837+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806837+0000","last_undegraded":"2024-02-10T08:36:30.806837+0000","last_fullsized":"2024-02-10T08:36:30.806837+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910253+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910630+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918346+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919772+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915399+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917793+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906423+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902172+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.808919+0000","last_change":"2024-02-10T08:36:30.808919+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.808919+0000","last_undegraded":"2024-02-10T08:36:30.808919+0000","last_fullsized":"2024-02-10T08:36:30.808919+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":77,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:31.890497+0000","last_change":"2024-02-10T08:36:31.890457+0000","last_active":"2024-02-10T08:36:31.890497+0000","last_peered":"2024-02-10T08:36:31.890497+0000","last_clean":"2024-02-10T08:36:31.890497+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:36:31.890497+0000","last_undegraded":"2024-02-10T08:36:31.890497+0000","last_fullsized":"2024-02-10T08:36:31.890497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:31.890386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.890386+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":74,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806226+0000","last_change":"2024-02-10T08:35:36.513411+0000","last_active":"2024-02-10T08:36:30.806226+0000","last_peered":"2024-02-10T08:36:30.806226+0000","last_clean":"2024-02-10T08:36:30.806226+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:36:30.806226+0000","last_undegraded":"2024-02-10T08:36:30.806226+0000","last_fullsized":"2024-02-10T08:36:30.806226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:36.513316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.513316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809805+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:36:30.809805+0000","last_peered":"2024-02-10T08:36:30.809805+0000","last_clean":"2024-02-10T08:36:30.809805+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:36:30.809805+0000","last_undegraded":"2024-02-10T08:36:30.809805+0000","last_fullsized":"2024-02-10T08:36:30.809805+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.871412+0000","last_change":"2024-02-10T08:36:35.871412+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.871412+0000","last_undegraded":"2024-02-10T08:36:35.871412+0000","last_fullsized":"2024-02-10T08:36:35.871412+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900380+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.911179+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912940+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917383+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908915+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.929770+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.908709+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899448+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.813702+0000","last_change":"2024-02-10T08:36:30.813702+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.813702+0000","last_undegraded":"2024-02-10T08:36:30.813702+0000","last_fullsized":"2024-02-10T08:36:30.813702+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":82,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808353+0000","last_change":"2024-02-10T08:35:45.543177+0000","last_active":"2024-02-10T08:36:30.808353+0000","last_peered":"2024-02-10T08:36:30.808353+0000","last_clean":"2024-02-10T08:36:30.808353+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:36:30.808353+0000","last_undegraded":"2024-02-10T08:36:30.808353+0000","last_fullsized":"2024-02-10T08:36:30.808353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:45.543110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.543110+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817293+0000","last_change":"2024-02-10T08:36:22.073074+0000","last_active":"2024-02-10T08:36:30.817293+0000","last_peered":"2024-02-10T08:36:30.817293+0000","last_clean":"2024-02-10T08:36:30.817293+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:36:30.817293+0000","last_undegraded":"2024-02-10T08:36:30.817293+0000","last_fullsized":"2024-02-10T08:36:30.817293+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.072982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807843+0000","last_change":"2024-02-10T08:36:25.503414+0000","last_active":"2024-02-10T08:36:30.807843+0000","last_peered":"2024-02-10T08:36:30.807843+0000","last_clean":"2024-02-10T08:36:30.807843+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:36:30.807843+0000","last_undegraded":"2024-02-10T08:36:30.807843+0000","last_fullsized":"2024-02-10T08:36:30.807843+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.503351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.503351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.804559+0000","last_change":"2024-02-10T08:36:30.804559+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.804559+0000","last_undegraded":"2024-02-10T08:36:30.804559+0000","last_fullsized":"2024-02-10T08:36:30.804559+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914931+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912782+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914621+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.899652+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902574+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805556+0000","last_change":"2024-02-10T08:36:30.805556+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805556+0000","last_undegraded":"2024-02-10T08:36:30.805556+0000","last_fullsized":"2024-02-10T08:36:30.805556+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812842+0000","last_change":"2024-02-10T08:36:13.651116+0000","last_active":"2024-02-10T08:36:30.812842+0000","last_peered":"2024-02-10T08:36:30.812842+0000","last_clean":"2024-02-10T08:36:30.812842+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:36:30.812842+0000","last_undegraded":"2024-02-10T08:36:30.812842+0000","last_fullsized":"2024-02-10T08:36:30.812842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:36:13.650984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.650984+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:34.837600+0000","last_change":"2024-02-10T08:36:34.837568+0000","last_active":"2024-02-10T08:36:34.837600+0000","last_peered":"2024-02-10T08:36:34.837600+0000","last_clean":"2024-02-10T08:36:34.837600+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:36:34.837600+0000","last_undegraded":"2024-02-10T08:36:34.837600+0000","last_fullsized":"2024-02-10T08:36:34.837600+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:34.837483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:34.837483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810570+0000","last_change":"2024-02-10T08:36:26.634045+0000","last_active":"2024-02-10T08:36:30.810570+0000","last_peered":"2024-02-10T08:36:30.810570+0000","last_clean":"2024-02-10T08:36:30.810570+0000","last_became_active":"2024-02-10T08:36:23.667607+0000","last_became_peered":"2024-02-10T08:36:23.667607+0000","last_unstale":"2024-02-10T08:36:30.810570+0000","last_undegraded":"2024-02-10T08:36:30.810570+0000","last_fullsized":"2024-02-10T08:36:30.810570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.633977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.633977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807249+0000","last_change":"2024-02-10T08:35:17.895268+0000","last_active":"2024-02-10T08:36:30.807249+0000","last_peered":"2024-02-10T08:36:30.807249+0000","last_clean":"2024-02-10T08:36:30.807249+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:36:30.807249+0000","last_undegraded":"2024-02-10T08:36:30.807249+0000","last_fullsized":"2024-02-10T08:36:30.807249+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:17.895163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:17.895163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.808028+0000","last_change":"2024-02-10T08:36:30.808028+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.808028+0000","last_undegraded":"2024-02-10T08:36:30.808028+0000","last_fullsized":"2024-02-10T08:36:30.808028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916540+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912919+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.906861+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.929407+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918139+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.905382+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.892310+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805016+0000","last_change":"2024-02-10T08:36:30.805016+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805016+0000","last_undegraded":"2024-02-10T08:36:30.805016+0000","last_fullsized":"2024-02-10T08:36:30.805016+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816827+0000","last_change":"2024-02-10T08:36:20.047350+0000","last_active":"2024-02-10T08:36:30.816827+0000","last_peered":"2024-02-10T08:36:30.816827+0000","last_clean":"2024-02-10T08:36:30.816827+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:36:30.816827+0000","last_undegraded":"2024-02-10T08:36:30.816827+0000","last_fullsized":"2024-02-10T08:36:30.816827+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:36:20.047272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.047272+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816844+0000","last_change":"2024-02-10T08:36:15.045147+0000","last_active":"2024-02-10T08:36:30.816844+0000","last_peered":"2024-02-10T08:36:30.816844+0000","last_clean":"2024-02-10T08:36:30.816844+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:36:30.816844+0000","last_undegraded":"2024-02-10T08:36:30.816844+0000","last_fullsized":"2024-02-10T08:36:30.816844+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.045044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.045044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.807243+0000","last_change":"2024-02-10T08:36:23.933053+0000","last_active":"2024-02-10T08:36:30.807243+0000","last_peered":"2024-02-10T08:36:30.807243+0000","last_clean":"2024-02-10T08:36:30.807243+0000","last_became_active":"2024-02-10T08:36:23.932688+0000","last_became_peered":"2024-02-10T08:36:23.932688+0000","last_unstale":"2024-02-10T08:36:30.807243+0000","last_undegraded":"2024-02-10T08:36:30.807243+0000","last_fullsized":"2024-02-10T08:36:30.807243+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.038724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.038724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817120+0000","last_change":"2024-02-10T08:36:27.054202+0000","last_active":"2024-02-10T08:36:30.817120+0000","last_peered":"2024-02-10T08:36:30.817120+0000","last_clean":"2024-02-10T08:36:30.817120+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:36:30.817120+0000","last_undegraded":"2024-02-10T08:36:30.817120+0000","last_fullsized":"2024-02-10T08:36:30.817120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.054145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.054145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.871365+0000","last_change":"2024-02-10T08:36:35.871365+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.871365+0000","last_undegraded":"2024-02-10T08:36:35.871365+0000","last_fullsized":"2024-02-10T08:36:35.871365+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907138+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914735+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.922549+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917743+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.915028+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912418+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902229+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.814312+0000","last_change":"2024-02-10T08:36:30.814312+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.814312+0000","last_undegraded":"2024-02-10T08:36:30.814312+0000","last_fullsized":"2024-02-10T08:36:30.814312+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.804501+0000","last_change":"2024-02-10T08:36:24.764092+0000","last_active":"2024-02-10T08:36:30.804501+0000","last_peered":"2024-02-10T08:36:30.804501+0000","last_clean":"2024-02-10T08:36:30.804501+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:36:30.804501+0000","last_undegraded":"2024-02-10T08:36:30.804501+0000","last_fullsized":"2024-02-10T08:36:30.804501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:24.764020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.764020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809384+0000","last_change":"2024-02-10T08:36:30.157585+0000","last_active":"2024-02-10T08:36:30.809384+0000","last_peered":"2024-02-10T08:36:30.809384+0000","last_clean":"2024-02-10T08:36:30.809384+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:36:30.809384+0000","last_undegraded":"2024-02-10T08:36:30.809384+0000","last_fullsized":"2024-02-10T08:36:30.809384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.157473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.157473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810351+0000","last_change":"2024-02-10T08:36:23.933004+0000","last_active":"2024-02-10T08:36:30.810351+0000","last_peered":"2024-02-10T08:36:30.810351+0000","last_clean":"2024-02-10T08:36:30.810351+0000","last_became_active":"2024-02-10T08:36:23.932727+0000","last_became_peered":"2024-02-10T08:36:23.932727+0000","last_unstale":"2024-02-10T08:36:30.810351+0000","last_undegraded":"2024-02-10T08:36:30.810351+0000","last_fullsized":"2024-02-10T08:36:30.810351+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:01.491913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:01.491913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":58,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810253+0000","last_change":"2024-02-10T08:36:20.877370+0000","last_active":"2024-02-10T08:36:30.810253+0000","last_peered":"2024-02-10T08:36:30.810253+0000","last_clean":"2024-02-10T08:36:30.810253+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:36:30.810253+0000","last_undegraded":"2024-02-10T08:36:30.810253+0000","last_fullsized":"2024-02-10T08:36:30.810253+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.877268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.877268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.805728+0000","last_change":"2024-02-10T08:36:30.805728+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.805728+0000","last_undegraded":"2024-02-10T08:36:30.805728+0000","last_fullsized":"2024-02-10T08:36:30.805728+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900601+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.912437+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.922070+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914970+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.892349+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.807280+0000","last_change":"2024-02-10T08:36:30.807280+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.807280+0000","last_undegraded":"2024-02-10T08:36:30.807280+0000","last_fullsized":"2024-02-10T08:36:30.807280+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":73,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:32.709511+0000","last_change":"2024-02-10T08:36:32.709476+0000","last_active":"2024-02-10T08:36:32.709511+0000","last_peered":"2024-02-10T08:36:32.709511+0000","last_clean":"2024-02-10T08:36:32.709511+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:36:32.709511+0000","last_undegraded":"2024-02-10T08:36:32.709511+0000","last_fullsized":"2024-02-10T08:36:32.709511+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:36:32.709392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.709392+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":65,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.808041+0000","last_change":"2024-02-10T08:36:21.107654+0000","last_active":"2024-02-10T08:36:30.808041+0000","last_peered":"2024-02-10T08:36:30.808041+0000","last_clean":"2024-02-10T08:36:30.808041+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:36:30.808041+0000","last_undegraded":"2024-02-10T08:36:30.808041+0000","last_fullsized":"2024-02-10T08:36:30.808041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.107579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.107579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.810432+0000","last_change":"2024-02-10T08:36:24.066925+0000","last_active":"2024-02-10T08:36:30.810432+0000","last_peered":"2024-02-10T08:36:30.810432+0000","last_clean":"2024-02-10T08:36:30.810432+0000","last_became_active":"2024-02-10T08:36:24.066557+0000","last_became_peered":"2024-02-10T08:36:24.066557+0000","last_unstale":"2024-02-10T08:36:30.810432+0000","last_undegraded":"2024-02-10T08:36:30.810432+0000","last_fullsized":"2024-02-10T08:36:30.810432+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.806181+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:36:30.806181+0000","last_peered":"2024-02-10T08:36:30.806181+0000","last_clean":"2024-02-10T08:36:30.806181+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:36:30.806181+0000","last_undegraded":"2024-02-10T08:36:30.806181+0000","last_fullsized":"2024-02-10T08:36:30.806181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.809202+0000","last_change":"2024-02-10T08:36:30.809202+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.809202+0000","last_undegraded":"2024-02-10T08:36:30.809202+0000","last_fullsized":"2024-02-10T08:36:30.809202+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.909819+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.901354+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.927964+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.925730+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.931262+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.907098+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.894726+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.902148+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.802728+0000","last_change":"2024-02-10T08:36:30.802728+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.802728+0000","last_undegraded":"2024-02-10T08:36:30.802728+0000","last_fullsized":"2024-02-10T08:36:30.802728+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":93,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.804234+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:36:30.804234+0000","last_peered":"2024-02-10T08:36:30.804234+0000","last_clean":"2024-02-10T08:36:30.804234+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:36:30.804234+0000","last_undegraded":"2024-02-10T08:36:30.804234+0000","last_fullsized":"2024-02-10T08:36:30.804234+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.817067+0000","last_change":"2024-02-10T08:36:26.014750+0000","last_active":"2024-02-10T08:36:30.817067+0000","last_peered":"2024-02-10T08:36:30.817067+0000","last_clean":"2024-02-10T08:36:30.817067+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:36:30.817067+0000","last_undegraded":"2024-02-10T08:36:30.817067+0000","last_fullsized":"2024-02-10T08:36:30.817067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.014690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.014690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809531+0000","last_change":"2024-02-10T08:36:23.670233+0000","last_active":"2024-02-10T08:36:30.809531+0000","last_peered":"2024-02-10T08:36:30.809531+0000","last_clean":"2024-02-10T08:36:30.809531+0000","last_became_active":"2024-02-10T08:36:23.669768+0000","last_became_peered":"2024-02-10T08:36:23.669768+0000","last_unstale":"2024-02-10T08:36:30.809531+0000","last_undegraded":"2024-02-10T08:36:30.809531+0000","last_fullsized":"2024-02-10T08:36:30.809531+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:18.421261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:18.421261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.804406+0000","last_change":"2024-02-10T08:36:14.747640+0000","last_active":"2024-02-10T08:36:30.804406+0000","last_peered":"2024-02-10T08:36:30.804406+0000","last_clean":"2024-02-10T08:36:30.804406+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:36:30.804406+0000","last_undegraded":"2024-02-10T08:36:30.804406+0000","last_fullsized":"2024-02-10T08:36:30.804406+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.747521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.747521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.808390+0000","last_change":"2024-02-10T08:36:30.808390+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.808390+0000","last_undegraded":"2024-02-10T08:36:30.808390+0000","last_fullsized":"2024-02-10T08:36:30.808390+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.900038+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.910173+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.917309+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:37.899089+0000","last_change":"2024-02-10T08:36:37.899089+0000","last_active":"2024-02-10T08:36:37.899089+0000","last_peered":"2024-02-10T08:36:37.899089+0000","last_clean":"2024-02-10T08:36:37.899089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:37.899089+0000","last_undegraded":"2024-02-10T08:36:37.899089+0000","last_fullsized":"2024-02-10T08:36:37.899089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:37.899089+0000","last_clean_scrub_stamp":"2024-02-10T08:36:37.899089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:36.884487+0000","last_change":"2024-02-10T08:36:36.884487+0000","last_active":"2024-02-10T08:36:36.884487+0000","last_peered":"2024-02-10T08:36:36.884487+0000","last_clean":"2024-02-10T08:36:36.884487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:36.884487+0000","last_undegraded":"2024-02-10T08:36:36.884487+0000","last_fullsized":"2024-02-10T08:36:36.884487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:36.884487+0000","last_clean_scrub_stamp":"2024-02-10T08:36:36.884487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.913055+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918305+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.919286+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.896435+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.870864+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.806798+0000","last_change":"2024-02-10T08:36:30.806798+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.806798+0000","last_undegraded":"2024-02-10T08:36:30.806798+0000","last_fullsized":"2024-02-10T08:36:30.806798+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":82,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816782+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:36:30.816782+0000","last_peered":"2024-02-10T08:36:30.816782+0000","last_clean":"2024-02-10T08:36:30.816782+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:36:30.816782+0000","last_undegraded":"2024-02-10T08:36:30.816782+0000","last_fullsized":"2024-02-10T08:36:30.816782+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.812962+0000","last_change":"2024-02-10T08:36:17.894440+0000","last_active":"2024-02-10T08:36:30.812962+0000","last_peered":"2024-02-10T08:36:30.812962+0000","last_clean":"2024-02-10T08:36:30.812962+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:36:30.812962+0000","last_undegraded":"2024-02-10T08:36:30.812962+0000","last_fullsized":"2024-02-10T08:36:30.812962+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.894338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.894338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.809955+0000","last_change":"2024-02-10T08:36:23.933121+0000","last_active":"2024-02-10T08:36:30.809955+0000","last_peered":"2024-02-10T08:36:30.809955+0000","last_clean":"2024-02-10T08:36:30.809955+0000","last_became_active":"2024-02-10T08:36:23.932769+0000","last_became_peered":"2024-02-10T08:36:23.932769+0000","last_unstale":"2024-02-10T08:36:30.809955+0000","last_undegraded":"2024-02-10T08:36:30.809955+0000","last_fullsized":"2024-02-10T08:36:30.809955+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:23.876871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:23.876871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-02-10T08:36:30.816629+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:36:30.816629+0000","last_peered":"2024-02-10T08:36:30.816629+0000","last_clean":"2024-02-10T08:36:30.816629+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:36:30.816629+0000","last_undegraded":"2024-02-10T08:36:30.816629+0000","last_fullsized":"2024-02-10T08:36:30.816629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":101,"state":"creating+peering","last_fresh":"2024-02-10T08:36:30.802682+0000","last_change":"2024-02-10T08:36:30.802682+0000","last_active":"2024-02-10T08:36:30.795614+0000","last_peered":"2024-02-10T08:36:30.795614+0000","last_clean":"2024-02-10T08:36:30.795614+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:30.802682+0000","last_undegraded":"2024-02-10T08:36:30.802682+0000","last_fullsized":"2024-02-10T08:36:30.802682+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.916588+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.914480+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:35.870864+0000","last_change":"2024-02-10T08:36:35.918583+0000","last_active":"2024-02-10T08:36:35.870864+0000","last_peered":"2024-02-10T08:36:35.870864+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:35.870864+0000","last_undegraded":"2024-02-10T08:36:35.870864+0000","last_fullsized":"2024-02-10T08:36:35.870864+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":90,"acting":90,"num_store_stats":7},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":75,"acting":75,"num_store_stats":7},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":90,"acting":90,"num_store_stats":7},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":81,"acting":81,"num_store_stats":6},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":30,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57,"ondisk_log_size":57,"up":90,"acting":90,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":191,"num_read_kb":186,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":7},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":87,"acting":87,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299434,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297768,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447384,"statfs":{"total":95995035648,"available":95690121216,"internally_reserved":0,"allocated":688128,"data_stored":327055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.50900000000000001,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.625},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.307,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.44900000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.02,"5min":1.054,"15min":1.054},"last":0.53400000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44,"15min":0.42999999999999999},"min":{"1min":0.312,"5min":0.311,"15min":0.311},"max":{"1min":0.67000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44700000000000001,"15min":0.433},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.44600000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.497,"15min":0.48399999999999999},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.378,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.70399999999999996}]}]},{"osd":6,"up_from":44,"seq":188978561072,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":327278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.36299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47399999999999998,"15min":0.47299999999999998},"min":{"1min":0.35699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.374}]},{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.315,"5min":0.248,"15min":0.248},"max":{"1min":0.60899999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.501,"15min":0.501},"min":{"1min":0.39400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66600000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.40200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.34699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.441}]},{"osd":4,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39400000000000002,"15min":0.378},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.63},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.39600000000000002,"15min":0.372},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.436,"15min":0.40400000000000003},"min":{"1min":0.32600000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67100000000000004,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:36:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47599999999999998,"15min":0.46100000000000002},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.443,"5min":0.317,"15min":0.317},"max":{"1min":0.65300000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":328375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.20799999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46600000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.219},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41099999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36099999999999999,"15min":0.371},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.29099999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.441,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.42199999999999999,"15min":0.44600000000000001},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.45300000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.44800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.73899999999999999,"5min":1.294,"15min":1.294},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:36:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.432},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.44700000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.47099999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.45900000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":349623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34899999999999998,"15min":0.30499999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.35599999999999998,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48299999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.374},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.307}]},{"osd":3,"last update":"Sat Feb 10 08:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39800000000000002,"15min":0.41599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48799999999999999,"5min":0.624,"15min":0.624},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40999999999999998,"15min":0.42999999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.49299999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79400000000000004,"5min":0.878,"15min":0.878},"last":0.58399999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72699999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.501}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.875,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.42299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.747,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83699999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.753,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.50700000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":675840,"data_stored":326758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.371,"15min":0.35899999999999999},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.22}]},{"osd":1,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.254}]},{"osd":3,"last update":"Sat Feb 10 08:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.46999999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40400000000000003,"15min":0.42799999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.22600000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.441,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82499999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82099999999999995,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80100000000000005,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.497,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.54400000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":327251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34200000000000003,"15min":0.36399999999999999},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36099999999999999,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.375,"15min":0.41199999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.45300000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.307},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.41099999999999998},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.63500000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79500000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":8.5670000000000002,"5min":8.5670000000000002,"15min":8.5670000000000002},"last":0.46300000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.42699999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.495}]},{"osd":7,"last update":"Sat Feb 10 08:36:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71599999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56799999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297772,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447380,"statfs":{"total":95995035648,"available":95690117120,"internally_reserved":0,"allocated":692224,"data_stored":327421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.35199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56599999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.28100000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":1,"15min":1},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51700000000000002,"15min":0.54200000000000004},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63800000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.505},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60899999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72099999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.48699999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.42299999999999999,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.435},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67500000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.38600000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41599999999999998,"15min":0.39900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66500000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.34000000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71099999999999997,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.41499999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.36899999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68500000000000005,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.51500000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790005,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297844,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447308,"statfs":{"total":95995035648,"available":95690043392,"internally_reserved":0,"allocated":688128,"data_stored":326712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55000000000000004,"15min":0.55900000000000005},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61399999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52800000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.73699999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55100000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.75}]},{"osd":4,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44400000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67400000000000004,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.44800000000000001,"15min":0.40200000000000002},"min":{"1min":0.34499999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.439,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69899999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72299999999999998,"5min":1.081,"15min":1.081},"last":0.623}]},{"osd":7,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.501,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:36:40.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:40.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:40.543 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:40.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:40.565 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:40.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:40.674 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:40.692 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:40.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:40.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:40.830 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:41.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.702138+0000 mgr.y (mgr.14553) 137 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:41.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.702138+0000 mgr.y (mgr.14553) 137 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.702672+0000 mgr.y (mgr.14553) 138 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.703054+0000 mgr.y (mgr.14553) 139 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.703440+0000 mgr.y (mgr.14553) 140 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.703808+0000 mgr.y (mgr.14553) 141 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.704244+0000 mgr.y (mgr.14553) 142 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.704652+0000 mgr.y (mgr.14553) 143 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.705030+0000 mgr.y (mgr.14553) 144 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.705426+0000 mgr.y (mgr.14553) 145 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.705809+0000 mgr.y (mgr.14553) 146 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:39.719959+0000 osd.4 (osd.4) 65 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:39.721588+0000 osd.4 (osd.4) 66 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:39.774122+0000 client.admin (client.35171) 0 : cluster [INF] twoxx 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.775620+0000 mon.a (mon.0) 830 : audit [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.776112+0000 mon.a (mon.0) 831 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:39.827209+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.036095+0000 mon.a (mon.0) 832 : audit [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.036202+0000 mon.a (mon.0) 833 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:40.036952+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.038675+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.107:0/3074326236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.044075+0000 mon.a (mon.0) 835 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.044564+0000 mon.a (mon.0) 836 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.044939+0000 mon.a (mon.0) 837 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.045252+0000 mon.a (mon.0) 838 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.045720+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.045909+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.107:0/3905136853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.048789+0000 mon.a (mon.0) 839 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.049286+0000 mon.a (mon.0) 840 : audit [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:40.053352+0000 mgr.y (mgr.14553) 147 : cluster [DBG] pgmap v134: 746 pgs: 258 creating+peering, 1 peering, 358 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.059031+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.308782+0000 mon.a (mon.0) 841 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.309142+0000 mon.a (mon.0) 842 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.309387+0000 mon.a (mon.0) 843 : audit [INF] from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.317434+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:41.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:40.828603+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:41.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.702672+0000 mgr.y (mgr.14553) 138 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:41.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.703054+0000 mgr.y (mgr.14553) 139 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:41.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.703440+0000 mgr.y (mgr.14553) 140 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:41.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.703808+0000 mgr.y (mgr.14553) 141 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.704244+0000 mgr.y (mgr.14553) 142 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.704652+0000 mgr.y (mgr.14553) 143 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.705030+0000 mgr.y (mgr.14553) 144 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.705426+0000 mgr.y (mgr.14553) 145 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.705809+0000 mgr.y (mgr.14553) 146 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:39.719959+0000 osd.4 (osd.4) 65 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:39.721588+0000 osd.4 (osd.4) 66 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:39.774122+0000 client.admin (client.35171) 0 : cluster [INF] twoxx 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.775620+0000 mon.a (mon.0) 830 : audit [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.776112+0000 mon.a (mon.0) 831 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:39.827209+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.702138+0000 mgr.y (mgr.14553) 137 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.702672+0000 mgr.y (mgr.14553) 138 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.703054+0000 mgr.y (mgr.14553) 139 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.703440+0000 mgr.y (mgr.14553) 140 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.703808+0000 mgr.y (mgr.14553) 141 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.704244+0000 mgr.y (mgr.14553) 142 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.704652+0000 mgr.y (mgr.14553) 143 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.705030+0000 mgr.y (mgr.14553) 144 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.705426+0000 mgr.y (mgr.14553) 145 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.705809+0000 mgr.y (mgr.14553) 146 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:39.719959+0000 osd.4 (osd.4) 65 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:39.721588+0000 osd.4 (osd.4) 66 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:39.774122+0000 client.admin (client.35171) 0 : cluster [INF] twoxx 2024-02-10T08:36:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.775620+0000 mon.a (mon.0) 830 : audit [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.776112+0000 mon.a (mon.0) 831 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.036095+0000 mon.a (mon.0) 832 : audit [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.036202+0000 mon.a (mon.0) 833 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:40.036952+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.038675+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.107:0/3074326236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.044075+0000 mon.a (mon.0) 835 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.044564+0000 mon.a (mon.0) 836 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.044939+0000 mon.a (mon.0) 837 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.045252+0000 mon.a (mon.0) 838 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.045720+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.045909+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.107:0/3905136853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.048789+0000 mon.a (mon.0) 839 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.049286+0000 mon.a (mon.0) 840 : audit [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:40.053352+0000 mgr.y (mgr.14553) 147 : cluster [DBG] pgmap v134: 746 pgs: 258 creating+peering, 1 peering, 358 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.059031+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.308782+0000 mon.a (mon.0) 841 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.309142+0000 mon.a (mon.0) 842 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.309387+0000 mon.a (mon.0) 843 : audit [INF] from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.317434+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:41.389 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:40.828603+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:39.827209+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.036095+0000 mon.a (mon.0) 832 : audit [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi107-112403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.036202+0000 mon.a (mon.0) 833 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:40.036952+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.038675+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.107:0/3074326236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.044075+0000 mon.a (mon.0) 835 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.044564+0000 mon.a (mon.0) 836 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.044939+0000 mon.a (mon.0) 837 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.045252+0000 mon.a (mon.0) 838 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.045720+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.045909+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.107:0/3905136853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.048789+0000 mon.a (mon.0) 839 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.049286+0000 mon.a (mon.0) 840 : audit [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:40.053352+0000 mgr.y (mgr.14553) 147 : cluster [DBG] pgmap v134: 746 pgs: 258 creating+peering, 1 peering, 358 unknown, 129 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.059031+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.308782+0000 mon.a (mon.0) 841 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.309142+0000 mon.a (mon.0) 842 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.309387+0000 mon.a (mon.0) 843 : audit [INF] from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.317434+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:41.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:40.828603+0000 mon.b (mon.1) 77 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:40.739705+0000 osd.5 (osd.5) 63 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:40.742598+0000 osd.5 (osd.5) 64 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.030855+0000 mon.a (mon.0) 845 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.043467+0000 mon.a (mon.0) 846 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.043552+0000 mon.a (mon.0) 847 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.043628+0000 mon.a (mon.0) 848 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.043889+0000 mon.a (mon.0) 849 : audit [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.043998+0000 mon.a (mon.0) 850 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.044087+0000 mon.a (mon.0) 851 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.044161+0000 mon.a (mon.0) 852 : audit [INF] from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:41.044277+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.044625+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.046146+0000 mon.a (mon.0) 854 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.047211+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.047544+0000 mon.a (mon.0) 855 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.050421+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.050704+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.052113+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.053124+0000 mon.a (mon.0) 856 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.053784+0000 mon.a (mon.0) 857 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.054181+0000 mon.a (mon.0) 858 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.061842+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.062439+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.115444+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.116705+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.117848+0000 mon.a (mon.0) 860 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.118297+0000 mon.a (mon.0) 861 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.120635+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.121190+0000 mon.a (mon.0) 862 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.122124+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.122619+0000 mon.a (mon.0) 863 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.125002+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.125598+0000 mon.a (mon.0) 864 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.126422+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.126983+0000 mon.a (mon.0) 865 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.150198+0000 mgr.y (mgr.14553) 148 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:41.829548+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:40.739705+0000 osd.5 (osd.5) 63 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:36:42.177 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:40.742598+0000 osd.5 (osd.5) 64 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.030855+0000 mon.a (mon.0) 845 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.043467+0000 mon.a (mon.0) 846 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.043552+0000 mon.a (mon.0) 847 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.043628+0000 mon.a (mon.0) 848 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.043889+0000 mon.a (mon.0) 849 : audit [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.043998+0000 mon.a (mon.0) 850 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.044087+0000 mon.a (mon.0) 851 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.044161+0000 mon.a (mon.0) 852 : audit [INF] from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:41.044277+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.044625+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.046146+0000 mon.a (mon.0) 854 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.047211+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.178 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.047544+0000 mon.a (mon.0) 855 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.050421+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.050704+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.052113+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.053124+0000 mon.a (mon.0) 856 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.053784+0000 mon.a (mon.0) 857 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.054181+0000 mon.a (mon.0) 858 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.061842+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.062439+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.115444+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.116705+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.117848+0000 mon.a (mon.0) 860 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.118297+0000 mon.a (mon.0) 861 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.179 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.120635+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.121190+0000 mon.a (mon.0) 862 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.122124+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.122619+0000 mon.a (mon.0) 863 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.125002+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.125598+0000 mon.a (mon.0) 864 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.126422+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.126983+0000 mon.a (mon.0) 865 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.150198+0000 mgr.y (mgr.14553) 148 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:41.829548+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:42.180 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:42.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:42.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:40.739705+0000 osd.5 (osd.5) 63 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:40.742598+0000 osd.5 (osd.5) 64 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.030855+0000 mon.a (mon.0) 845 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.043467+0000 mon.a (mon.0) 846 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.043552+0000 mon.a (mon.0) 847 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.043628+0000 mon.a (mon.0) 848 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.043889+0000 mon.a (mon.0) 849 : audit [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.043998+0000 mon.a (mon.0) 850 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.044087+0000 mon.a (mon.0) 851 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.044161+0000 mon.a (mon.0) 852 : audit [INF] from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:41.044277+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.044625+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.046146+0000 mon.a (mon.0) 854 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.047211+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.047544+0000 mon.a (mon.0) 855 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.050421+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.050704+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.052113+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.053124+0000 mon.a (mon.0) 856 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:42.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.053784+0000 mon.a (mon.0) 857 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.054181+0000 mon.a (mon.0) 858 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.061842+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.062439+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.115444+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.116705+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.117848+0000 mon.a (mon.0) 860 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.118297+0000 mon.a (mon.0) 861 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.120635+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.121190+0000 mon.a (mon.0) 862 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.122124+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.122619+0000 mon.a (mon.0) 863 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.125002+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.125598+0000 mon.a (mon.0) 864 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:42.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.126422+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.126983+0000 mon.a (mon.0) 865 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:42.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.150198+0000 mgr.y (mgr.14553) 148 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:42.291 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:41.829548+0000 mon.b (mon.1) 80 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:42.340 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:42.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:42.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:43.005 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: starting. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: creating pool ceph_test_rados_open_pools_parallel.smithi107-112633 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: created object 0... 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: created object 25... 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: created object 49... 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: finishing. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_1_[112673]: shutting down. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_2_[112674]: starting. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_2_[112674]: rados_pool_create. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_2_[112674]: rados_ioctx_create. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_2_[112674]: shutting down. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: ******************************* 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: starting. 2024-02-10T08:36:43.006 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: creating pool ceph_test_rados_open_pools_parallel.smithi107-112633 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: created object 0... 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: created object 25... 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: created object 49... 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: finishing. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_3_[114876]: shutting down. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: ******************************* 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_4_[114877]: starting. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_4_[114877]: rados_pool_create. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_4_[114877]: rados_ioctx_create. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: process_4_[114877]: shutting down. 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: ******************************* 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: ******************************* 2024-02-10T08:36:43.007 INFO:tasks.workunit.client.0.smithi107.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: starting. 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: creating pool ceph_test_rados_delete_pools_parallel.smithi107-112637 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: created object 0... 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: created object 25... 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: created object 49... 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: finishing. 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_1_[112671]: shutting down. 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_2_[112672]: starting. 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_2_[112672]: deleting pool ceph_test_rados_delete_pools_parallel.smithi107-112637 2024-02-10T08:36:43.008 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_2_[112672]: shutting down. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******************************* 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: starting. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: creating pool ceph_test_rados_delete_pools_parallel.smithi107-112637 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: created object 0... 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: created object 25... 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: created object 49... 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: finishing. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_3_[114868]: shutting down. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******************************* 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: starting. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: listing objects. 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: listed object 0... 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: listed object 25... 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: saw 50 objects 2024-02-10T08:36:43.009 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_5_[114870]: shutting down. 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******************************* 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_4_[114869]: starting. 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_4_[114869]: deleting pool ceph_test_rados_delete_pools_parallel.smithi107-112637 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: process_4_[114869]: shutting down. 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******************************* 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******************************* 2024-02-10T08:36:43.010 INFO:tasks.workunit.client.0.smithi107.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-10T08:36:43.031 INFO:tasks.workunit.client.0.smithi107.stdout:08:36:38.289007+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:43.031 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.295873+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.491204+0000 mon.a [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.737838+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.738071+0000 mon.a [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.738308+0000 mon.a [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.738538+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.738539+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:38.738797+0000 mon.a [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.030855+0000 mon.a [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.043467+0000 mon.a [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi107-112537-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.032 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.043552+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.043628+0000 mon.a [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi107-112419-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.043889+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.043998+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.044087+0000 mon.a [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.044161+0000 mon.a [INF] from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.044625+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.046146+0000 mon.a [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.047211+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.033 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.047544+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.050421+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.050704+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.052113+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.053124+0000 mon.a [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.053784+0000 mon.a [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.054181+0000 mon.a [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.061842+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:43.034 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.062439+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:43.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:43.125 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.115444+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.116705+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.117848+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.118297+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.120635+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.121190+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.122124+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.122619+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.125002+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.125598+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.126422+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:43.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:41.126983+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:43.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.056453+0000 mon.c [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.291327+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:43.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.291978+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.764364+0000 mon.a [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.056453+0000 mon.c (mon.2) 122 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.291327+0000 mon.a (mon.0) 866 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.291978+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.764364+0000 mon.a (mon.0) 867 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.765462+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.830674+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.997786+0000 mon.a (mon.0) 868 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.997883+0000 mon.a (mon.0) 869 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.998041+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.998829+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.999179+0000 mon.c (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:42.999456+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.001162+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.002216+0000 mon.a (mon.0) 872 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.056453+0000 mon.c (mon.2) 122 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.291327+0000 mon.a (mon.0) 866 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.291978+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.764364+0000 mon.a (mon.0) 867 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.765462+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.830674+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:43.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.997786+0000 mon.a (mon.0) 868 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.997883+0000 mon.a (mon.0) 869 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.998041+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.998829+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.999179+0000 mon.c (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:42.999456+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.001162+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.002216+0000 mon.a (mon.0) 872 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.056453+0000 mon.c (mon.2) 122 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.291327+0000 mon.a (mon.0) 866 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.291978+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.764364+0000 mon.a (mon.0) 867 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi107-112411-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.765462+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.830674+0000 mon.b (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.997786+0000 mon.a (mon.0) 868 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.997883+0000 mon.a (mon.0) 869 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.998041+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.998829+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.999179+0000 mon.c (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:42.999456+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.001162+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:43.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.002216+0000 mon.a (mon.0) 872 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:43.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:44.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.765462+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/2801389734' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.997786+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.997883+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi107-112451-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.998829+0000 mon.b [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.999179+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:44.277 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:42.999456+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.001162+0000 mon.a [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.002216+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.031574+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.031945+0000 client.admin [INF] threexx 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.253862+0000 mon.a [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.255906+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.256158+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.259292+0000 mon.a [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.278 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.265723+0000 mon.a [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.265816+0000 mon.a [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd pool cre api_io_pp: Running main() from gmock_main.cc 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: seed 112427 2024-02-10T08:36:44.365 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (279 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (8 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (16 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (17 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (8 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-10T08:36:44.366 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (6 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (7 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (7 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (19 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (4 ms) 2024-02-10T08:36:44.367 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (17 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (7 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (5 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (428 ms total) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1058 ms) 2024-02-10T08:36:44.368 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-10T08:36:44.369 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:41.875684+0000 osd.0 (osd.0) 73 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:41.876859+0000 osd.0 (osd.0) 74 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.054344+0000 mgr.y (mgr.14553) 149 : cluster [DBG] pgmap v137: 481 pgs: 46 creating+peering, 1 clean+premerge+peered, 32 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.205351+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.205402+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.502486+0000 osd.1 (osd.1) 77 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:42.505952+0000 osd.1 (osd.1) 78 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.031574+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:43.031945+0000 client.admin (client.35171) 0 : cluster [INF] threexx 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:43.253862+0000 mon.a (mon.0) 873 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.255906+0000 mon.a (mon.0) 874 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:44.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.256158+0000 mon.a (mon.0) 875 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.259292+0000 mon.a (mon.0) 876 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.265723+0000 mon.a (mon.0) 877 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.265816+0000 mon.a (mon.0) 878 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:41.875684+0000 osd.0 (osd.0) 73 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:41.876859+0000 osd.0 (osd.0) 74 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.054344+0000 mgr.y (mgr.14553) 149 : cluster [DBG] pgmap v137: 481 pgs: 46 creating+peering, 1 clean+premerge+peered, 32 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.205351+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.205402+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.502486+0000 osd.1 (osd.1) 77 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:36:44.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:42.505952+0000 osd.1 (osd.1) 78 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.031574+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:43.031945+0000 client.admin (client.35171) 0 : cluster [INF] threexx 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:43.253862+0000 mon.a (mon.0) 873 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.255906+0000 mon.a (mon.0) 874 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.256158+0000 mon.a (mon.0) 875 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.259292+0000 mon.a (mon.0) 876 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.265723+0000 mon.a (mon.0) 877 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.265816+0000 mon.a (mon.0) 878 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.265873+0000 mon.a (mon.0) 879 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:44.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.265945+0000 mon.a (mon.0) 880 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.266015+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.266069+0000 mon.a (mon.0) 882 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.272024+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.279956+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:43.509549+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.510927+0000 mon.a (mon.0) 884 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.511364+0000 mon.a (mon.0) 885 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.512078+0000 mon.a (mon.0) 886 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.512446+0000 mon.a (mon.0) 887 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:43.832084+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:43.984460+0000 osd.6 (osd.6) 49 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:36:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:43.985587+0000 osd.6 (osd.6) 50 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:36:44.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.265873+0000 mon.a (mon.0) 879 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:44.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.265945+0000 mon.a (mon.0) 880 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:44.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.266015+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-10T08:36:44.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.266069+0000 mon.a (mon.0) 882 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.272024+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.279956+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:43.509549+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.510927+0000 mon.a (mon.0) 884 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.511364+0000 mon.a (mon.0) 885 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.512078+0000 mon.a (mon.0) 886 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.512446+0000 mon.a (mon.0) 887 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:43.832084+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:43.984460+0000 osd.6 (osd.6) 49 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:36:44.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:43.985587+0000 osd.6 (osd.6) 50 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:41.875684+0000 osd.0 (osd.0) 73 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:41.876859+0000 osd.0 (osd.0) 74 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.054344+0000 mgr.y (mgr.14553) 149 : cluster [DBG] pgmap v137: 481 pgs: 46 creating+peering, 1 clean+premerge+peered, 32 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.205351+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.205402+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.502486+0000 osd.1 (osd.1) 77 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:36:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:42.505952+0000 osd.1 (osd.1) 78 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.031574+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:43.031945+0000 client.admin (client.35171) 0 : cluster [INF] threexx 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:43.253862+0000 mon.a (mon.0) 873 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.255906+0000 mon.a (mon.0) 874 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.256158+0000 mon.a (mon.0) 875 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.259292+0000 mon.a (mon.0) 876 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.265723+0000 mon.a (mon.0) 877 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi107-112419-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.265816+0000 mon.a (mon.0) 878 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.265873+0000 mon.a (mon.0) 879 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.265945+0000 mon.a (mon.0) 880 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.266015+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.266069+0000 mon.a (mon.0) 882 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.272024+0000 mon.c (mon.2) 127 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.279956+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:43.509549+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.510927+0000 mon.a (mon.0) 884 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.511364+0000 mon.a (mon.0) 885 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.512078+0000 mon.a (mon.0) 886 : audit [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.512446+0000 mon.a (mon.0) 887 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:43.832084+0000 mon.b (mon.1) 84 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:43.984460+0000 osd.6 (osd.6) 49 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:36:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:43.985587+0000 osd.6 (osd.6) 50 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:36:44.873 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:36:44.873 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:36:45.467 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:36:45.467 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:36:45.467 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2024-02-10T08:36:45.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:36:45.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:44.055075+0000 mgr.y (mgr.14553) 150 : cluster [DBG] pgmap v139: 545 pgs: 46 creating+peering, 1 clean+premerge+peered, 96 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 545 op/s 2024-02-10T08:36:45.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:44.262247+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.768248+0000 mon.a (mon.0) 889 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]': finished 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.768344+0000 mon.a (mon.0) 890 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.768425+0000 mon.a (mon.0) 891 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.768521+0000 mon.a (mon.0) 892 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:44.768714+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.769721+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.107:0/4202373440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.769983+0000 mon.a (mon.0) 894 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.770035+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.107:0/1695708024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.770303+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.107:0/1950840999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.770465+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.770988+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:44.055075+0000 mgr.y (mgr.14553) 150 : cluster [DBG] pgmap v139: 545 pgs: 46 creating+peering, 1 clean+premerge+peered, 96 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 545 op/s 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:44.262247+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.768248+0000 mon.a (mon.0) 889 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]': finished 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.768344+0000 mon.a (mon.0) 890 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.768425+0000 mon.a (mon.0) 891 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.768521+0000 mon.a (mon.0) 892 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:44.768714+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-10T08:36:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.769721+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.107:0/4202373440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.769983+0000 mon.a (mon.0) 894 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.770035+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.107:0/1695708024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.770303+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.107:0/1950840999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.770465+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.770988+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:44.771155+0000 client.admin (client.35171) 0 : cluster [INF] fourxx 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.775204+0000 mon.a (mon.0) 895 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.776364+0000 mon.a (mon.0) 896 : audit [INF] from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.776590+0000 mon.a (mon.0) 897 : audit [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.776802+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.776982+0000 mon.a (mon.0) 899 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.783985+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.784358+0000 mon.a (mon.0) 900 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.787855+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.788385+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.788690+0000 mon.a (mon.0) 901 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:44.833013+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:44.771155+0000 client.admin (client.35171) 0 : cluster [INF] fourxx 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.775204+0000 mon.a (mon.0) 895 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.776364+0000 mon.a (mon.0) 896 : audit [INF] from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.776590+0000 mon.a (mon.0) 897 : audit [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.776802+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.776982+0000 mon.a (mon.0) 899 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.783985+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.784358+0000 mon.a (mon.0) 900 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.787855+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.788385+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.788690+0000 mon.a (mon.0) 901 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.502 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:44.833013+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:45.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.508+0000 7f969d072700 1 -- 172.21.15.107:0/2931546444 >> v1:172.21.15.107:6790/0 conn(0x7f96981043b0 legacy=0x7f9698106870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.508+0000 7f969d072700 1 -- 172.21.15.107:0/2931546444 shutdown_connections 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.508+0000 7f969d072700 1 -- 172.21.15.107:0/2931546444 >> 172.21.15.107:0/2931546444 conn(0x7f969806fe70 msgr2=0x7f96980f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.508+0000 7f969d072700 1 -- 172.21.15.107:0/2931546444 shutdown_connections 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.508+0000 7f969d072700 1 -- 172.21.15.107:0/2931546444 wait complete. 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 librados: starting msgr at 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 librados: starting objecter 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 Processor -- start 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 -- start start 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 librados: setting wanted keys 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.509+0000 7f969d072700 1 librados: calling monclient init 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f969d072700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9698117540 con 0x7f96980fda50 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f969d072700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9698118720 con 0x7f96981043b0 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f969d072700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9698119920 con 0x7f9698100840 2024-02-10T08:36:45.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f969659c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9698100840 0x7f96980fa500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49894/0 (socket says 172.21.15.107:49894) 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f969659c700 1 -- 172.21.15.107:0/3846299225 learned_addr learned my addr 172.21.15.107:0/3846299225 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter ms_handle_connect 0x7f96980fda50 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter ms_handle_connect 0x7f9698100840 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter ms_handle_connect 0x7f96981043b0 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370932812 0 0) 0x7f9698119920 con 0x7f9698100840 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680003630 con 0x7f9698100840 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498904618 0 0) 0x7f9698118720 con 0x7f96981043b0 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698119920 con 0x7f96981043b0 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3956842220 0 0) 0x7f9680003630 con 0x7f9698100840 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9698118720 con 0x7f9698100840 2024-02-10T08:36:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2855860721 0 0) 0x7f9698119920 con 0x7f96981043b0 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9680003630 con 0x7f96981043b0 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f968c001880 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.510+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9690002240 con 0x7f96981043b0 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2453765884 0 0) 0x7f9698118720 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.195:6789/0 conn(0x7f96981043b0 legacy=0x7f9698115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6789/0 conn(0x7f96980fda50 legacy=0x7f96980f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969811ab20 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9698119b10 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f969d072700 10 client.35252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9698119f80 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.511+0000 7f969d072700 1 librados: init done 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.512+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f968c001a70 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.512+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f968c005b30 con 0x7f9698100840 2024-02-10T08:36:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.512+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f968c0189e0 con 0x7f9698100840 2024-02-10T08:36:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.512+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9684004f00 con 0x7f9698100840 2024-02-10T08:36:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(111..111 src has 50..111) v4 ==== 15435+0+0 (unknown 1564101913 0 0) 0x7f968c07a8a0 con 0x7f9698100840 2024-02-10T08:36:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 osd_map(111..111 src has 50..111) v4 2024-02-10T08:36:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 3 client.35252.objecter handle_osd_map got epochs [111,111] > 0 2024-02-10T08:36:45.776 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 3 client.35252.objecter handle_osd_map decoding full epoch 111 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 20 client.35252.objecter dump_active .. 0 homeless 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v111) v1 ==== 54+0+16 (unknown 4076800186 0 2420000192) 0x7f968c07ab80 con 0x7f9698100840 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9680063ae0 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 10 client.35252.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 1 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 1 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 0 1 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 4 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 10 client.35252.objecter _send_command 1 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.513+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684004970 con 0x7f9684004530 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684004530 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684004970 con 0x7f9684004530 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 10 client.35252.objecter _finish_command 1 = system:0 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 4 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.514+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 0 1 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 10 client.35252.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 2 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 2 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f96840051a0 osd=0 3 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 0 2 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 4 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 10 client.35252.objecter _send_command 2 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.529+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9684004d50 con 0x7f9684004530 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.530+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9684004d50 con 0x7f9684004530 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.530+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.530+0000 7f967ffff700 10 client.35252.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.530+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 4 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.530+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 0 2 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 10 client.35252.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 1 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 4 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 10 client.35252.objecter _send_command 3 2024-02-10T08:36:45.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684010370 con 0x7f9684002090 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.531+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684002090 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.532+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684010370 con 0x7f9684002090 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.532+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.532+0000 7f967ffff700 10 client.35252.objecter _finish_command 3 = system:0 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.532+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 4 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.532+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 1 3 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:45.781 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 10 client.35252.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 4 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 4 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f96840042a0 osd=1 3 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 1 4 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 4 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 10 client.35252.objecter _send_command 4 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.546+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96840016b0 con 0x7f9684002090 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f96840016b0 con 0x7f9684002090 2024-02-10T08:36:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f967ffff700 10 client.35252.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 4 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 1 4 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 10 client.35252.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 5 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 5 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 2 5 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 4 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 10 client.35252.objecter _send_command 5 2024-02-10T08:36:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.548+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684011040 con 0x7f9684010510 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684010510 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684011040 con 0x7f9684010510 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 10 client.35252.objecter _finish_command 5 = system:0 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 4 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.549+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 2 5 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 10 client.35252.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 6 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 6 2024-02-10T08:36:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684003630 osd=2 3 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 2 6 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 4 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 10 client.35252.objecter _send_command 6 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.564+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9684011980 con 0x7f9684010510 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9684011980 con 0x7f9684010510 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f967ffff700 10 client.35252.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 4 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 2 6 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f969d072700 10 client.35252.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.566+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 7 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 7 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 3 7 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 4 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 10 client.35252.objecter _send_command 7 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684019e40 con 0x7f9684016220 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684016220 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684019e40 con 0x7f9684016220 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 10 client.35252.objecter _finish_command 7 = system:0 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 4 2024-02-10T08:36:45.786 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.567+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 3 7 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 10 client.35252.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 8 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 8 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684003420 osd=3 3 2024-02-10T08:36:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 3 8 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 4 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 10 client.35252.objecter _send_command 8 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.582+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9684019e40 con 0x7f9684016220 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.583+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9684019e40 con 0x7f9684016220 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.583+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.583+0000 7f967ffff700 10 client.35252.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.583+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 4 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.583+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 3 8 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 10 client.35252.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 9 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 9 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 4 9 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 4 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 10 client.35252.objecter _send_command 9 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684011610 con 0x7f9684012840 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.584+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684012840 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.585+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684011610 con 0x7f9684012840 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.585+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.585+0000 7f967ffff700 10 client.35252.objecter _finish_command 9 = system:0 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.585+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 4 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.585+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 4 9 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 10 client.35252.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 10 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 10 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684019e40 osd=4 3 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 4 10 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 4 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 10 client.35252.objecter _send_command 10 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.600+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9684011610 con 0x7f9684012840 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9684011610 con 0x7f9684012840 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 10 client.35252.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 4 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 4 10 2024-02-10T08:36:45.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 10 client.35252.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 11 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 11 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 5 11 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 4 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 10 client.35252.objecter _send_command 11 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f968401b0e0 con 0x7f968401ac00 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.602+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f968401ac00 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.603+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f968401b0e0 con 0x7f968401ac00 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.603+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.603+0000 7f967ffff700 10 client.35252.objecter _finish_command 11 = system:0 2024-02-10T08:36:45.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.603+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 4 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.603+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 5 11 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 10 client.35252.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 12 2024-02-10T08:36:45.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 12 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684011610 osd=5 3 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 5 12 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 4 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 10 client.35252.objecter _send_command 12 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.617+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f968401b0e0 con 0x7f968401ac00 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f968401b0e0 con 0x7f968401ac00 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f967ffff700 10 client.35252.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 4 2024-02-10T08:36:45.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 5 12 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 10 client.35252.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 13 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 13 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 6 13 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 4 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 10 client.35252.objecter _send_command 13 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.619+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9684026f50 con 0x7f9684023440 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f9684023440 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9684026f50 con 0x7f9684023440 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:45.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 10 client.35252.objecter _finish_command 13 = system:0 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 4 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.620+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 6 13 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 10 client.35252.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 14 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 14 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f968401b0e0 osd=6 3 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 6 14 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 4 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 10 client.35252.objecter _send_command 14 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.635+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9684026f50 con 0x7f9684023440 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9684026f50 con 0x7f9684023440 2024-02-10T08:36:45.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f967ffff700 10 client.35252.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 4 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 6 14 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 10 client.35252.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 15 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 15 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 7 15 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 4 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 10 client.35252.objecter _send_command 15 2024-02-10T08:36:45.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.637+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96840292d0 con 0x7f96840275d0 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 10 client.35252.objecter ms_handle_connect 0x7f96840275d0 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f96840292d0 con 0x7f96840275d0 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 10 client.35252.objecter _finish_command 15 = system:0 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 4 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.641+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 7 15 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 10 client.35252.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign -1 16 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 20 client.35252.objecter _get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 15 client.35252.objecter _session_command_op_remove -1 16 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 20 client.35252.objecter get_session s=0x7f9684026f50 osd=7 3 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 15 client.35252.objecter _session_command_op_assign 7 16 2024-02-10T08:36:45.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 4 2024-02-10T08:36:45.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 10 client.35252.objecter _send_command 16 2024-02-10T08:36:45.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.656+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96840292d0 con 0x7f96840275d0 2024-02-10T08:36:45.798 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:45.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.658+0000 7f967ffff700 1 -- 172.21.15.107:0/3846299225 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f96840292d0 con 0x7f96840275d0 2024-02-10T08:36:45.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.658+0000 7f967ffff700 10 client.35252.objecter ms_dispatch 0x7f9698001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.658+0000 7f967ffff700 10 client.35252.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.658+0000 7f967ffff700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 4 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.658+0000 7f967ffff700 15 client.35252.objecter _session_command_op_remove 7 16 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 librados: watch_flush enter 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 librados: watch_flush exit 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.0 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6801/3321596972 conn(0x7f9684004530 legacy=0x7f9684002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840051a0 osd=0 1 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.1 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6805/133516126 conn(0x7f9684002090 legacy=0x7f968400a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f96840042a0 osd=1 1 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.2 2024-02-10T08:36:45.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6809/3575881728 conn(0x7f9684010510 legacy=0x7f9684010930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003630 osd=2 1 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.3 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6813/2871825691 conn(0x7f9684016220 legacy=0x7f9684018610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684003420 osd=3 1 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.4 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.195:6800/2499671285 conn(0x7f9684012840 legacy=0x7f9684019fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684019e40 osd=4 1 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.5 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.195:6804/4178321486 conn(0x7f968401ac00 legacy=0x7f9684021810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969759e700 1 -- 172.21.15.107:0/3846299225 reap_dead start 2024-02-10T08:36:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684011610 osd=5 1 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.6 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.195:6808/1417479017 conn(0x7f9684023440 legacy=0x7f9684025830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969759e700 1 -- 172.21.15.107:0/3846299225 reap_dead start 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f968401b0e0 osd=6 1 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter close_session for osd.7 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.195:6812/1934176928 conn(0x7f96840275d0 legacy=0x7f9684027a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter put_session s=0x7f9684026f50 osd=7 1 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 20 client.35252.objecter shutdown clearing up homeless session... 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 10 client.35252.objecter successfully canceled tick 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969759e700 1 -- 172.21.15.107:0/3846299225 reap_dead start 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969759e700 1 -- 172.21.15.107:0/3846299225 reap_dead start 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9680063ae0 legacy=0x7f9680065fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> v1:172.21.15.107:6790/0 conn(0x7f9698100840 legacy=0x7f96980fa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:45.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 shutdown_connections 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 >> 172.21.15.107:0/3846299225 conn(0x7f969806fe70 msgr2=0x7f96980f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 shutdown_connections 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 -- 172.21.15.107:0/3846299225 wait complete. 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:45.659+0000 7f969d072700 1 librados: shutdown 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:44.055075+0000 mgr.y (mgr.14553) 150 : cluster [DBG] pgmap v139: 545 pgs: 46 creating+peering, 1 clean+premerge+peered, 96 unknown, 402 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 545 op/s 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:44.262247+0000 mon.a (mon.0) 888 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.768248+0000 mon.a (mon.0) 889 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]': finished 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.768344+0000 mon.a (mon.0) 890 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.768425+0000 mon.a (mon.0) 891 : audit [INF] from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.768521+0000 mon.a (mon.0) 892 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:44.768714+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.769721+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.107:0/4202373440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.769983+0000 mon.a (mon.0) 894 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T08:36:45.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.770035+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.107:0/1695708024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.770303+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.107:0/1950840999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.770465+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.107:0/2601684997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.770988+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.107:0/1261353402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:44.771155+0000 client.admin (client.35171) 0 : cluster [INF] fourxx 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.775204+0000 mon.a (mon.0) 895 : audit [INF] from='client.35171 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.776364+0000 mon.a (mon.0) 896 : audit [INF] from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.776590+0000 mon.a (mon.0) 897 : audit [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.776802+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.776982+0000 mon.a (mon.0) 899 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.783985+0000 mon.c (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.784358+0000 mon.a (mon.0) 900 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.787855+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T08:36:45.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.788385+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.805 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.788690+0000 mon.a (mon.0) 901 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:45.805 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:44.833013+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:45] "GET /metrics HTTP/1.1" 200 221862 "" "Prometheus/2.33.4" 2024-02-10T08:36:46.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:36:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:46.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:46.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:46.532 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:46.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 -- 172.21.15.107:0/3059621446 >> v1:172.21.15.195:6789/0 conn(0x7fbb800f8a40 legacy=0x7fbb800f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:46.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 -- 172.21.15.107:0/3059621446 shutdown_connections 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 -- 172.21.15.107:0/3059621446 >> 172.21.15.107:0/3059621446 conn(0x7fbb800f2510 msgr2=0x7fbb800f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 -- 172.21.15.107:0/3059621446 shutdown_connections 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 -- 172.21.15.107:0/3059621446 wait complete. 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 librados: starting msgr at 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.696+0000 7fbb8840d700 1 librados: starting objecter 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 Processor -- start 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 -- start start 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 librados: setting wanted keys 2024-02-10T08:36:46.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 librados: calling monclient init 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb80102b80 con 0x7fbb800f9530 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb80102d50 con 0x7fbb80104340 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb8840d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb80102f40 con 0x7fbb800f8a40 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb859a8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb800f9530 0x7fbb80114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55060/0 (socket says 172.21.15.107:55060) 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb859a8700 1 -- 172.21.15.107:0/2647519839 learned_addr learned my addr 172.21.15.107:0/2647519839 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 10 client.?.objecter ms_handle_connect 0x7fbb800f9530 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 10 client.?.objecter ms_handle_connect 0x7fbb80104340 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068801383 0 0) 0x7fbb80102d50 con 0x7fbb80104340 2024-02-10T08:36:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.697+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c004010 con 0x7fbb80104340 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687622616 0 0) 0x7fbb6c004010 con 0x7fbb80104340 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb80102d50 con 0x7fbb80104340 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755407989 0 0) 0x7fbb80102b80 con 0x7fbb800f9530 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c004010 con 0x7fbb800f9530 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb78001b90 con 0x7fbb80104340 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1233154398 0 0) 0x7fbb80102d50 con 0x7fbb80104340 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 >> v1:172.21.15.107:6790/0 conn(0x7fbb800f8a40 legacy=0x7fbb801023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 >> v1:172.21.15.107:6789/0 conn(0x7fbb800f9530 legacy=0x7fbb80114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8011cc00 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 10 client.?.objecter ms_handle_connect 0x7fbb800f8a40 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80119c30 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb8840d700 10 client.25312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8011a0a0 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb8840d700 1 librados: init done 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbb780022a0 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.698+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb78003680 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb64004f50 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbb78016530 con 0x7fbb80104340 2024-02-10T08:36:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(112..112 src has 50..112) v4 ==== 13581+0+0 (unknown 1313216833 0 0) 0x7fbb78077d90 con 0x7fbb80104340 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb777fe700 10 client.25312.objecter ms_dispatch 0x7fbb80001550 osd_map(112..112 src has 50..112) v4 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb777fe700 3 client.25312.objecter handle_osd_map got epochs [112,112] > 0 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.699+0000 7fbb777fe700 3 client.25312.objecter handle_osd_map decoding full epoch 112 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.700+0000 7fbb777fe700 20 client.25312.objecter dump_active .. 0 homeless 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.700+0000 7fbb777fe700 10 client.25312.objecter ms_handle_connect 0x7fbb6c063ca0 2024-02-10T08:36:46.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.704+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbb78049bf0 con 0x7fbb80104340 2024-02-10T08:36:46.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:46.882+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fbb64005200 con 0x7fbb80104340 2024-02-10T08:36:46.909 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-10T08:36:46.909 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2024-02-10T08:36:46.909 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-10T08:36:46.909 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (4 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-02-10T08:36:46.910 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (5 ms) 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1138 ms total) 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (16740 ms total) 2024-02-10T08:36:46.911 INFO:tasks.workunit.client.0.smithi107.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-10T08:36:46.913 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: Running main() from gmock_main.cc 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] Global test environment set-up. 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStat.Stat (292 ms) 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStat.StatNS (37 ms) 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (10 ms) 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] 4 tests from LibRadosStat (340 ms total) 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: 2024-02-10T08:36:46.914 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1060 ms) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (15 ms) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1077 ms total) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [----------] Global test environment tear-down 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (16755 ms total) 2024-02-10T08:36:46.915 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat: [ PASSED ] 8 tests. 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: seed 112545 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (299 ms) 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (16 ms) 2024-02-10T08:36:46.919 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (4 ms) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (19 ms) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (338 ms total) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1063 ms) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-10T08:36:46.920 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3 ms) 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7 ms) 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1073 ms total) 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (16763 ms total) 2024-02-10T08:36:46.921 INFO:tasks.workunit.client.0.smithi107.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-10T08:36:46.928 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: Running main() from gmock_main.cc 2024-02-10T08:36:46.928 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [==========] Running 23 tests from 2 test suites. 2024-02-10T08:36:46.928 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] Global test environment set-up. 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] 13 tests from LibRadosIo 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (278 ms) 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: no timeout :/ 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: no timeout :/ 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: no timeout :/ 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: no timeout :/ 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: no timeout :/ 2024-02-10T08:36:46.929 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (65 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.Checksum (17 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (10 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (8 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (9 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-10T08:36:46.930 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (7 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.RmXattr (18 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] 13 tests from LibRadosIo (431 ms total) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1060 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (8 ms) 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-10T08:36:46.931 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (6 ms) 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (6 ms) 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-10T08:36:46.932 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1111 ms total) 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [----------] Global test environment tear-down 2024-02-10T08:36:46.933 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [==========] 23 tests from 2 test suites ran. (16775 ms total) 2024-02-10T08:36:46.934 INFO:tasks.workunit.client.0.smithi107.stdout: api_io: [ PASSED ] 23 tests. 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.278751+0000 mon.a (mon.0) 902 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.278848+0000 mon.a (mon.0) 903 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279109+0000 mon.a (mon.0) 904 : audit [INF] from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279175+0000 mon.a (mon.0) 905 : audit [INF] from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279228+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279280+0000 mon.a (mon.0) 907 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279354+0000 mon.a (mon.0) 908 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.279404+0000 mon.a (mon.0) 909 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.279455+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.288690+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.288975+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.289920+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.290395+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.293162+0000 mon.c (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.296915+0000 mon.a (mon.0) 911 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.308725+0000 mon.a (mon.0) 912 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.309015+0000 mon.a (mon.0) 913 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.309303+0000 mon.a (mon.0) 914 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.309478+0000 mon.a (mon.0) 915 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.309641+0000 mon.a (mon.0) 916 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.315418+0000 mon.c (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.317252+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.318097+0000 mon.a (mon.0) 917 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.513760+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.107:0/3846299225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:46.966 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.833978+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.878741+0000 mon.a (mon.0) 918 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.884644+0000 mon.a (mon.0) 919 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.884732+0000 mon.a (mon.0) 920 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.884833+0000 mon.a (mon.0) 921 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.884893+0000 mon.a (mon.0) 922 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.884952+0000 mon.a (mon.0) 923 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.885053+0000 mon.a (mon.0) 924 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.885176+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.893439+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.897601+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.899772+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.909277+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.910194+0000 mon.a (mon.0) 926 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.910862+0000 mon.a (mon.0) 927 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.918735+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.921541+0000 mon.a (mon.0) 928 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.921929+0000 mon.a (mon.0) 929 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.922361+0000 mon.a (mon.0) 930 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.934160+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:45.948763+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.056825+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:46.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.201623+0000 mon.a (mon.0) 931 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:46.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.278751+0000 mon.a (mon.0) 902 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:46.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.278848+0000 mon.a (mon.0) 903 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:46.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279109+0000 mon.a (mon.0) 904 : audit [INF] from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279175+0000 mon.a (mon.0) 905 : audit [INF] from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279228+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279280+0000 mon.a (mon.0) 907 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279354+0000 mon.a (mon.0) 908 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]': finished 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.279404+0000 mon.a (mon.0) 909 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.279455+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.288690+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.288975+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.289920+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.290395+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.293162+0000 mon.c (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.296915+0000 mon.a (mon.0) 911 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.308725+0000 mon.a (mon.0) 912 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.309015+0000 mon.a (mon.0) 913 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.309303+0000 mon.a (mon.0) 914 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.309478+0000 mon.a (mon.0) 915 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.309641+0000 mon.a (mon.0) 916 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.315418+0000 mon.c (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.317252+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.318097+0000 mon.a (mon.0) 917 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.513760+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.107:0/3846299225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.833978+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.878741+0000 mon.a (mon.0) 918 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:36:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.884644+0000 mon.a (mon.0) 919 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.884732+0000 mon.a (mon.0) 920 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.884833+0000 mon.a (mon.0) 921 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.884893+0000 mon.a (mon.0) 922 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.884952+0000 mon.a (mon.0) 923 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.885053+0000 mon.a (mon.0) 924 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.885176+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.893439+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.897601+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.899772+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.909277+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.910194+0000 mon.a (mon.0) 926 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.910862+0000 mon.a (mon.0) 927 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:46.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.918735+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.921541+0000 mon.a (mon.0) 928 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.921929+0000 mon.a (mon.0) 929 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.922361+0000 mon.a (mon.0) 930 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.934160+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:45.948763+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.056825+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:46.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.201623+0000 mon.a (mon.0) 931 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:46.982 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:46.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.278751+0000 mon.a (mon.0) 902 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi107-112459-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.278848+0000 mon.a (mon.0) 903 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi107-112451-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279109+0000 mon.a (mon.0) 904 : audit [INF] from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279175+0000 mon.a (mon.0) 905 : audit [INF] from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279228+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi107-112403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279280+0000 mon.a (mon.0) 907 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi107-112562-1"}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279354+0000 mon.a (mon.0) 908 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-4"}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.279404+0000 mon.a (mon.0) 909 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.279455+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.288690+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:47.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.288975+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.289920+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.290395+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.293162+0000 mon.c (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.296915+0000 mon.a (mon.0) 911 : audit [INF] from='client.35171 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.308725+0000 mon.a (mon.0) 912 : audit [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.309015+0000 mon.a (mon.0) 913 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.309303+0000 mon.a (mon.0) 914 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.309478+0000 mon.a (mon.0) 915 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.309641+0000 mon.a (mon.0) 916 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.315418+0000 mon.c (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.317252+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.318097+0000 mon.a (mon.0) 917 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.513760+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.107:0/3846299225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:47.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.833978+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.878741+0000 mon.a (mon.0) 918 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.884644+0000 mon.a (mon.0) 919 : audit [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-4", "tierpool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.884732+0000 mon.a (mon.0) 920 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.884833+0000 mon.a (mon.0) 921 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.884893+0000 mon.a (mon.0) 922 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.884952+0000 mon.a (mon.0) 923 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.885053+0000 mon.a (mon.0) 924 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.885176+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.893439+0000 mon.c (mon.2) 140 : audit [INF] from='client.? v1:172.21.15.107:0/3669001896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.897601+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3093492486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.899772+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.107:0/3646985788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.909277+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.910194+0000 mon.a (mon.0) 926 : audit [INF] from='client.25231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.910862+0000 mon.a (mon.0) 927 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.918735+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.107:0/631454208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.921541+0000 mon.a (mon.0) 928 : audit [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.921929+0000 mon.a (mon.0) 929 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.922361+0000 mon.a (mon.0) 930 : audit [INF] from='client.35177 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.934160+0000 mon.c (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:45.948763+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.056825+0000 mon.c (mon.2) 145 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:47.041 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.201623+0000 mon.a (mon.0) 931 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:36:47.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:47.114 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:47.133 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:47.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:47.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:47.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout:ate", "pool": "LibRadosStatEC_smithi107-112537-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.265873+0000 mon.a [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi107-112427-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.265945+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi107-112545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.266015+0000 mon.a [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.266069+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-4", "overlaypool": "test-rados-api-smithi107-112490-4-cache"}]': finished 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.272024+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/4088871154' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.279956+0000 mon.c [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.510927+0000 mon.a [INF] from='client.35048 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.511364+0000 mon.a [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi107-112468-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.315 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.512078+0000 mon.a [INF] from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi107-112411-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: got: 2024-02-10T08:36:43.512446+0000 mon.a [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi107-112562-1"}]: dispatch 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9087 ms) 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (17168 ms total) 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (17168 ms total) 2024-02-10T08:36:47.316 INFO:tasks.workunit.client.0.smithi107.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-10T08:36:47.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:47.415 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:47.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.895+0000 7fbb777fe700 1 -- 172.21.15.107:0/2647519839 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v114) v1 ==== 97+0+0 (unknown 959336279 0 0) 0x7fbb78048000 con 0x7fbb80104340 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.907+0000 7fbb8840d700 10 librados: watch_flush enter 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.909+0000 7fbb8840d700 10 librados: watch_flush exit 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.909+0000 7fbb8840d700 20 client.25312.objecter shutdown clearing up homeless session... 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.909+0000 7fbb8840d700 10 client.25312.objecter successfully canceled tick 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.911+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbb6c063ca0 legacy=0x7fbb6c066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.911+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 >> v1:172.21.15.195:6789/0 conn(0x7fbb80104340 legacy=0x7fbb801184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.913+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 shutdown_connections 2024-02-10T08:36:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.913+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 >> 172.21.15.107:0/2647519839 conn(0x7fbb800f2510 msgr2=0x7fbb801077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.922+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 shutdown_connections 2024-02-10T08:36:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.922+0000 7fbb8840d700 1 -- 172.21.15.107:0/2647519839 wait complete. 2024-02-10T08:36:47.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.777214+0000 osd.5 (osd.5) 65 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:36:47.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.808406+0000 osd.5 (osd.5) 66 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:36:47.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.973344+0000 osd.6 (osd.6) 51 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:36:47.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:45.976245+0000 osd.6 (osd.6) 52 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:46.056537+0000 mgr.y (mgr.14553) 151 : cluster [DBG] pgmap v143: 560 pgs: 32 creating+peering, 2 active+clean+snaptrim, 526 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.698530+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.834976+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.882995+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.107:0/2647519839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886083+0000 mon.a (mon.0) 933 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886231+0000 mon.a (mon.0) 934 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886362+0000 mon.a (mon.0) 935 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886420+0000 mon.a (mon.0) 936 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886504+0000 mon.a (mon.0) 937 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886597+0000 mon.a (mon.0) 938 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:36:47.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.886716+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:46.886790+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.895230+0000 mon.a (mon.0) 941 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.898058+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.903719+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.107:0/2248243388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.909636+0000 mon.a (mon.0) 943 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.913533+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.107:0/3529637480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.913877+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.914583+0000 mon.a (mon.0) 944 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:47.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.915266+0000 mon.a (mon.0) 945 : audit [INF] from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.949371+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.950104+0000 mon.a (mon.0) 946 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.951911+0000 mon.c (mon.2) 148 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.952141+0000 mon.a (mon.0) 947 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.952645+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.952958+0000 mon.a (mon.0) 948 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.970349+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:47.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:46.970953+0000 mon.a (mon.0) 949 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:47.927+0000 7fbb8840d700 1 librados: shutdown 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.777214+0000 osd.5 (osd.5) 65 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.808406+0000 osd.5 (osd.5) 66 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.973344+0000 osd.6 (osd.6) 51 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:45.976245+0000 osd.6 (osd.6) 52 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:46.056537+0000 mgr.y (mgr.14553) 151 : cluster [DBG] pgmap v143: 560 pgs: 32 creating+peering, 2 active+clean+snaptrim, 526 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-02-10T08:36:47.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.698530+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.834976+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.882995+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.107:0/2647519839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886083+0000 mon.a (mon.0) 933 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886231+0000 mon.a (mon.0) 934 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886362+0000 mon.a (mon.0) 935 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886420+0000 mon.a (mon.0) 936 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886504+0000 mon.a (mon.0) 937 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886597+0000 mon.a (mon.0) 938 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:36:47.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.886716+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:46.886790+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.895230+0000 mon.a (mon.0) 941 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.898058+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.903719+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.107:0/2248243388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.909636+0000 mon.a (mon.0) 943 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.913533+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.107:0/3529637480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.913877+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.914583+0000 mon.a (mon.0) 944 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.915266+0000 mon.a (mon.0) 945 : audit [INF] from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:47.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.949371+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.950104+0000 mon.a (mon.0) 946 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.951911+0000 mon.c (mon.2) 148 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.952141+0000 mon.a (mon.0) 947 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.952645+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.952958+0000 mon.a (mon.0) 948 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.970349+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:47.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:46.970953+0000 mon.a (mon.0) 949 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:36:48.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.777214+0000 osd.5 (osd.5) 65 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.808406+0000 osd.5 (osd.5) 66 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.973344+0000 osd.6 (osd.6) 51 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:45.976245+0000 osd.6 (osd.6) 52 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:46.056537+0000 mgr.y (mgr.14553) 151 : cluster [DBG] pgmap v143: 560 pgs: 32 creating+peering, 2 active+clean+snaptrim, 526 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.698530+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.834976+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.882995+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.107:0/2647519839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886083+0000 mon.a (mon.0) 933 : audit [INF] from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi107-112427-22"}]': finished 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886231+0000 mon.a (mon.0) 934 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886362+0000 mon.a (mon.0) 935 : audit [INF] from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi107-112419-15"}]': finished 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886420+0000 mon.a (mon.0) 936 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi107-112545-7"}]': finished 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886504+0000 mon.a (mon.0) 937 : audit [INF] from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi107-112537-6"}]': finished 2024-02-10T08:36:48.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886597+0000 mon.a (mon.0) 938 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.886716+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:46.886790+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.895230+0000 mon.a (mon.0) 941 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.898058+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.903719+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.107:0/2248243388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.909636+0000 mon.a (mon.0) 943 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.913533+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.107:0/3529637480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.913877+0000 mon.c (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.914583+0000 mon.a (mon.0) 944 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.915266+0000 mon.a (mon.0) 945 : audit [INF] from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.949371+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.950104+0000 mon.a (mon.0) 946 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.951911+0000 mon.c (mon.2) 148 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:48.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.952141+0000 mon.a (mon.0) 947 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:48.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.952645+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:48.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.952958+0000 mon.a (mon.0) 948 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:36:48.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.970349+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:48.040 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:46.970953+0000 mon.a (mon.0) 949 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]: dispatch 2024-02-10T08:36:48.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:36:49.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:47.526375+0000 osd.1 (osd.1) 79 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:47.558294+0000 osd.1 (osd.1) 80 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.835972+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:47.883556+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888398+0000 mon.a (mon.0) 951 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888524+0000 mon.a (mon.0) 952 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888599+0000 mon.a (mon.0) 953 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888756+0000 mon.a (mon.0) 954 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888829+0000 mon.a (mon.0) 955 : audit [INF] from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.888945+0000 mon.a (mon.0) 956 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.889021+0000 mon.a (mon.0) 957 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]': finished 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:47.889082+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.895137+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.907401+0000 mon.a (mon.0) 959 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.912218+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.914214+0000 mon.a (mon.0) 960 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.923323+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:47.923780+0000 mon.a (mon.0) 961 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:47.526375+0000 osd.1 (osd.1) 79 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:47.558294+0000 osd.1 (osd.1) 80 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.835972+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:47.883556+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888398+0000 mon.a (mon.0) 951 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888524+0000 mon.a (mon.0) 952 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888599+0000 mon.a (mon.0) 953 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888756+0000 mon.a (mon.0) 954 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888829+0000 mon.a (mon.0) 955 : audit [INF] from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.888945+0000 mon.a (mon.0) 956 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.889021+0000 mon.a (mon.0) 957 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]': finished 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:47.889082+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.895137+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.907401+0000 mon.a (mon.0) 959 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.912218+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.914214+0000 mon.a (mon.0) 960 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.923323+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:47.923780+0000 mon.a (mon.0) 961 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:47.526375+0000 osd.1 (osd.1) 79 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:47.558294+0000 osd.1 (osd.1) 80 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.835972+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:47.883556+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888398+0000 mon.a (mon.0) 951 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888524+0000 mon.a (mon.0) 952 : audit [INF] from='client.? v1:172.21.15.107:0/4285222104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888599+0000 mon.a (mon.0) 953 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi107-112403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888756+0000 mon.a (mon.0) 954 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888829+0000 mon.a (mon.0) 955 : audit [INF] from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.888945+0000 mon.a (mon.0) 956 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.889021+0000 mon.a (mon.0) 957 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pg_num": 4}]': finished 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:47.889082+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.895137+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.907401+0000 mon.a (mon.0) 959 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.912218+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.914214+0000 mon.a (mon.0) 960 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.923323+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3624186468' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:47.923780+0000 mon.a (mon.0) 961 : audit [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:36:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:49.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:49.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:50.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:50.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:50.170 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:50.189 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:50.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:50.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:50.321 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:50.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:50.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:50.476 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:48.057071+0000 mgr.y (mgr.14553) 152 : cluster [DBG] pgmap v146: 532 pgs: 100 unknown, 32 creating+peering, 1 active+clean+snaptrim, 399 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:48.836957+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:48.894573+0000 mon.a (mon.0) 962 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:48.905348+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.146922+0000 mon.a (mon.0) 963 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:49.147182+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.148463+0000 mon.a (mon.0) 965 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.148516+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.148814+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.107:0/2637540386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.148990+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.149487+0000 mon.a (mon.0) 966 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.149954+0000 mon.a (mon.0) 967 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.478 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.150222+0000 mon.a (mon.0) 968 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.478 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:50.496 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:50.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:50.655 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:48.057071+0000 mgr.y (mgr.14553) 152 : cluster [DBG] pgmap v146: 532 pgs: 100 unknown, 32 creating+peering, 1 active+clean+snaptrim, 399 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:48.836957+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:48.894573+0000 mon.a (mon.0) 962 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:48.905348+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.146922+0000 mon.a (mon.0) 963 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:49.147182+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.148463+0000 mon.a (mon.0) 965 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.148516+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.148814+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.107:0/2637540386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.148990+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.149487+0000 mon.a (mon.0) 966 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.149954+0000 mon.a (mon.0) 967 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.150222+0000 mon.a (mon.0) 968 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:48.057071+0000 mgr.y (mgr.14553) 152 : cluster [DBG] pgmap v146: 532 pgs: 100 unknown, 32 creating+peering, 1 active+clean+snaptrim, 399 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:48.836957+0000 mon.b (mon.1) 101 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:48.894573+0000 mon.a (mon.0) 962 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:48.905348+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.146922+0000 mon.a (mon.0) 963 : audit [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi107-112468-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:49.147182+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.148463+0000 mon.a (mon.0) 965 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.148516+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.148814+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.107:0/2637540386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.148990+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.149487+0000 mon.a (mon.0) 966 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:50.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.149954+0000 mon.a (mon.0) 967 : audit [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:50.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.150222+0000 mon.a (mon.0) 968 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:50.662 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:50.952 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: Running main() from gmock_main.cc 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] Global test environment set-up. 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (296 ms) 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.LockShared (18 ms) 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1031 ms) 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1016 ms) 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-10T08:36:50.953 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] 8 tests from LibRadosLock (2374 ms total) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (671 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (3 ms) 2024-02-10T08:36:50.954 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1043 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1030 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2 ms) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (2762 ms total) 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [----------] Global test environment tear-down 2024-02-10T08:36:50.955 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (20794 ms total) 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: seed 112459 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (261 ms) 2024-02-10T08:36:50.956 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (42 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1016 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1013 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (7 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2 ms) 2024-02-10T08:36:50.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2348 ms total) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (642 ms) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (19 ms) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1057 ms) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1025 ms) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3 ms) 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-10T08:36:50.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (6 ms) 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (2759 ms total) 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (20775 ms total) 2024-02-10T08:36:50.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-10T08:36:51.065 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.620+0000 7f2e9b650700 1 -- 172.21.15.107:0/196713580 >> v1:172.21.15.195:6789/0 conn(0x7f2e941132f0 legacy=0x7f2e941156e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.620+0000 7f2e9b650700 1 -- 172.21.15.107:0/196713580 shutdown_connections 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.620+0000 7f2e9b650700 1 -- 172.21.15.107:0/196713580 >> 172.21.15.107:0/196713580 conn(0x7f2e94063bc0 msgr2=0x7f2e940672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.621+0000 7f2e9b650700 1 -- 172.21.15.107:0/196713580 shutdown_connections 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.621+0000 7f2e9b650700 1 -- 172.21.15.107:0/196713580 wait complete. 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 librados: starting msgr at 2024-02-10T08:36:51.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 librados: starting objecter 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 Processor -- start 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 -- start start 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 librados: setting wanted keys 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 librados: calling monclient init 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e9413ff50 con 0x7f2e94069490 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e94141130 con 0x7f2e941132f0 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.622+0000 7f2e9b650700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e94142330 con 0x7f2e9410c3c0 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.623+0000 7f2e99bed700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2e9410c3c0 0x7f2e9413e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44112/0 (socket says 172.21.15.107:44112) 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.623+0000 7f2e99bed700 1 -- 172.21.15.107:0/2567511150 learned_addr learned my addr 172.21.15.107:0/2567511150 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.623+0000 7f2e8a7fc700 10 client.?.objecter ms_handle_connect 0x7f2e9410c3c0 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.623+0000 7f2e8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 10 client.?.objecter ms_handle_connect 0x7f2e94069490 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:51.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942782040 0 0) 0x7f2e94142330 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80002220 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482873710 0 0) 0x7f2e80002220 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e94142330 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.624+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2e8c003150 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.625+0000 7f2e8a7fc700 10 client.?.objecter ms_handle_connect 0x7f2e941132f0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.625+0000 7f2e8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.626+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2380285838 0 0) 0x7f2e94142330 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.626+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 >> v1:172.21.15.195:6789/0 conn(0x7f2e941132f0 legacy=0x7f2e94111980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.626+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 >> v1:172.21.15.107:6789/0 conn(0x7f2e94069490 legacy=0x7f2e9410bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.626+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94143530 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.626+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e94140140 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.627+0000 7f2e9b650700 10 client.35963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.627+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e941405b0 con 0x7f2e9410c3c0 2024-02-10T08:36:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.627+0000 7f2e9b650700 1 librados: init done 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.627+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2e8c0027b0 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.627+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2e8c003c30 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.629+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2e8c003db0 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.629+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(116..116 src has 50..116) v4 ==== 12946+0+0 (unknown 128195346 0 0) 0x7f2e8c07ab50 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e8a7fc700 10 client.35963.objecter ms_dispatch 0x7f2e94001550 osd_map(116..116 src has 50..116) v4 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e8a7fc700 3 client.35963.objecter handle_osd_map got epochs [116,116] > 0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e8a7fc700 3 client.35963.objecter handle_osd_map decoding full epoch 116 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e78004f50 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e8a7fc700 20 client.35963.objecter dump_active .. 0 homeless 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.630+0000 7f2e8a7fc700 10 client.35963.objecter ms_handle_connect 0x7f2e80063a50 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.635+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2e8c04ccf0 con 0x7f2e9410c3c0 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.894+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e78000bc0 con 0x7f2e80063a50 2024-02-10T08:36:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:50.966+0000 7f2e8a7fc700 1 -- 172.21.15.107:0/2567511150 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1321159 (unknown 2965378022 0 655184302) 0x7f2e78000bc0 con 0x7f2e80063a50 2024-02-10T08:36:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.080+0000 7f2e9b650700 10 librados: watch_flush enter 2024-02-10T08:36:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.080+0000 7f2e9b650700 10 librados: watch_flush exit 2024-02-10T08:36:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.080+0000 7f2e9b650700 20 client.35963.objecter shutdown clearing up homeless session... 2024-02-10T08:36:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.080+0000 7f2e9b650700 10 client.35963.objecter successfully canceled tick 2024-02-10T08:36:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.081+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2e80063a50 legacy=0x7f2e80065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.081+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 >> v1:172.21.15.107:6790/0 conn(0x7f2e9410c3c0 legacy=0x7f2e9413e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:51.110 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.086+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 shutdown_connections 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.086+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 >> 172.21.15.107:0/2567511150 conn(0x7f2e94063bc0 msgr2=0x7f2e940647c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.086+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 shutdown_connections 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.086+0000 7f2e9b650700 1 -- 172.21.15.107:0/2567511150 wait complete. 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:51.086+0000 7f2e9b650700 1 librados: shutdown 2024-02-10T08:36:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:36:51.324 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:48.511843+0000 osd.1 (osd.1) 81 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:36:51.324 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:48.514769+0000 osd.1 (osd.1) 82 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:36:51.324 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:49.839908+0000 mon.b (mon.1) 103 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.324 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.057607+0000 mgr.y (mgr.14553) 153 : cluster [DBG] pgmap v148: 419 pgs: 36 unknown, 32 creating+peering, 1 active+clean+snaptrim, 350 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-10T08:36:51.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.059333+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.340506+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.344549+0000 mon.a (mon.0) 970 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:51.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:36:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:48.511843+0000 osd.1 (osd.1) 81 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:36:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:48.514769+0000 osd.1 (osd.1) 82 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:36:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:49.839908+0000 mon.b (mon.1) 103 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.057607+0000 mgr.y (mgr.14553) 153 : cluster [DBG] pgmap v148: 419 pgs: 36 unknown, 32 creating+peering, 1 active+clean+snaptrim, 350 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.059333+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.340506+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.344549+0000 mon.a (mon.0) 970 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.344718+0000 mon.a (mon.0) 971 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.344811+0000 mon.a (mon.0) 972 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.344882+0000 mon.a (mon.0) 973 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.344948+0000 mon.a (mon.0) 974 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.345229+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.346794+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.348748+0000 mon.a (mon.0) 976 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.353576+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.354276+0000 mon.a (mon.0) 977 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.356574+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.107:0/3690313877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.358902+0000 mon.a (mon.0) 978 : audit [INF] from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.378893+0000 mon.c (mon.2) 156 : audit [INF] from='client.? v1:172.21.15.107:0/1000879324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.380575+0000 mon.a (mon.0) 979 : audit [INF] from='client.35294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.387161+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.387601+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.388239+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2171720735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.395676+0000 mon.a (mon.0) 980 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.396017+0000 mon.a (mon.0) 981 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.396281+0000 mon.a (mon.0) 982 : audit [INF] from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.842970+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.880246+0000 mon.a (mon.0) 983 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.880684+0000 mon.a (mon.0) 984 : cluster [WRN] pool 'PoolQuotaPP_smithi107-112411-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.882817+0000 mon.a (mon.0) 985 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:51.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.882851+0000 mon.a (mon.0) 986 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889524+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889605+0000 mon.a (mon.0) 988 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889700+0000 mon.a (mon.0) 989 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889785+0000 mon.a (mon.0) 990 : audit [INF] from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889903+0000 mon.a (mon.0) 991 : audit [INF] from='client.35294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.889986+0000 mon.a (mon.0) 992 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.890045+0000 mon.a (mon.0) 993 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.890106+0000 mon.a (mon.0) 994 : audit [INF] from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:50.890197+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-10T08:36:51.541 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:51.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:51.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.344718+0000 mon.a (mon.0) 971 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:36:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.344811+0000 mon.a (mon.0) 972 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.344882+0000 mon.a (mon.0) 973 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.344948+0000 mon.a (mon.0) 974 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.345229+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.346794+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.348748+0000 mon.a (mon.0) 976 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.353576+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.354276+0000 mon.a (mon.0) 977 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.356574+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.107:0/3690313877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.358902+0000 mon.a (mon.0) 978 : audit [INF] from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.378893+0000 mon.c (mon.2) 156 : audit [INF] from='client.? v1:172.21.15.107:0/1000879324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.380575+0000 mon.a (mon.0) 979 : audit [INF] from='client.35294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.387161+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.387601+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.388239+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2171720735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.395676+0000 mon.a (mon.0) 980 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.396017+0000 mon.a (mon.0) 981 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.396281+0000 mon.a (mon.0) 982 : audit [INF] from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.842970+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.880246+0000 mon.a (mon.0) 983 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.880684+0000 mon.a (mon.0) 984 : cluster [WRN] pool 'PoolQuotaPP_smithi107-112411-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.882817+0000 mon.a (mon.0) 985 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.882851+0000 mon.a (mon.0) 986 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889524+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889605+0000 mon.a (mon.0) 988 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889700+0000 mon.a (mon.0) 989 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889785+0000 mon.a (mon.0) 990 : audit [INF] from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889903+0000 mon.a (mon.0) 991 : audit [INF] from='client.35294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.889986+0000 mon.a (mon.0) 992 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.890045+0000 mon.a (mon.0) 993 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.890106+0000 mon.a (mon.0) 994 : audit [INF] from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:50.890197+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:48.511843+0000 osd.1 (osd.1) 81 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:48.514769+0000 osd.1 (osd.1) 82 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:49.839908+0000 mon.b (mon.1) 103 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.057607+0000 mgr.y (mgr.14553) 153 : cluster [DBG] pgmap v148: 419 pgs: 36 unknown, 32 creating+peering, 1 active+clean+snaptrim, 350 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.059333+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.340506+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.344549+0000 mon.a (mon.0) 970 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi107-112553-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.344718+0000 mon.a (mon.0) 971 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.344811+0000 mon.a (mon.0) 972 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.344882+0000 mon.a (mon.0) 973 : audit [INF] from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi107-112513-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.344948+0000 mon.a (mon.0) 974 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.345229+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.346794+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.348748+0000 mon.a (mon.0) 976 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.353576+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.354276+0000 mon.a (mon.0) 977 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.356574+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.107:0/3690313877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.358902+0000 mon.a (mon.0) 978 : audit [INF] from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.378893+0000 mon.c (mon.2) 156 : audit [INF] from='client.? v1:172.21.15.107:0/1000879324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.380575+0000 mon.a (mon.0) 979 : audit [INF] from='client.35294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.387161+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.107:0/4111016630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.387601+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2138383223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.388239+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2171720735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.395676+0000 mon.a (mon.0) 980 : audit [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.396017+0000 mon.a (mon.0) 981 : audit [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.396281+0000 mon.a (mon.0) 982 : audit [INF] from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.842970+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.880246+0000 mon.a (mon.0) 983 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.880684+0000 mon.a (mon.0) 984 : cluster [WRN] pool 'PoolQuotaPP_smithi107-112411-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.882817+0000 mon.a (mon.0) 985 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.882851+0000 mon.a (mon.0) 986 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889524+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T08:36:51.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889605+0000 mon.a (mon.0) 988 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "mode": "writeback"}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889700+0000 mon.a (mon.0) 989 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889785+0000 mon.a (mon.0) 990 : audit [INF] from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889903+0000 mon.a (mon.0) 991 : audit [INF] from='client.35294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi107-112403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.889986+0000 mon.a (mon.0) 992 : audit [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi107-112459-10"}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.890045+0000 mon.a (mon.0) 993 : audit [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi107-112451-10"}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.890106+0000 mon.a (mon.0) 994 : audit [INF] from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:51.687 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:50.890197+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-10T08:36:51.688 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:36:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:36:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:36:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:36:51.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34379.0:62 5.d 5:b4812045:::notify.4:head [watch watch cookie 94862968482912] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:22.136887+0000", 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.27962255099999, 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022844079999999999, 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34379", 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 62 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136887+0000", 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136887+0000", 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136886+0000", 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8849999999999999e-06 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136894+0000", 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8539999999999997e-06 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136901+0000", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4829999999999996e-06 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.136908+0000", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029457499999999999 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.137202+0000", 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5987000000000003e-05 2024-02-10T08:36:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:36:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.137278+0000", 2024-02-10T08:36:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00069714500000000003 2024-02-10T08:36:51.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.137976+0000", 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0895000000000005e-05 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138046+0000", 2024-02-10T08:36:51.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016886200000000001 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,7", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138215+0000", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000393478 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138609+0000", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042164900000000001 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139030+0000", 2024-02-10T08:36:51.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2827000000000003e-05 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139063+0000", 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8803e-05 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139112+0000", 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9834999999999999e-05 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139172+0000", 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022844079999999999 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:62 4.1c 4:3e426232:gc::gc.9:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.175287+0000", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.24122306300001, 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031576439999999998, 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 62 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175287+0000", 2024-02-10T08:36:51.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175287+0000", 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175284+0000", 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0563e-05 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175295+0000", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2918999999999999e-05 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175308+0000", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7899999999999999e-06 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175313+0000", 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5886999999999997e-05 2024-02-10T08:36:51.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175348+0000", 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8442000000000001e-05 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175397+0000", 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017745099999999999 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.175574+0000", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068077100000000005 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.176255+0000", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021028 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.176465+0000", 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001916839 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.178382+0000", 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9666999999999999e-05 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.178412+0000", 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2675000000000001e-05 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.178445+0000", 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031576439999999998 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:75 4.1c 4:3e426232:gc::gc.9:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.180169+0000", 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.23634074399999, 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027916870000000002, 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.709 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 75 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180169+0000", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180169+0000", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180168+0000", 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5019999999999998e-06 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180174+0000", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1527e-05 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180186+0000", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.199e-06 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180193+0000", 2024-02-10T08:36:51.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8038999999999999e-05 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180241+0000", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3226999999999995e-05 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180314+0000", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017653300000000001 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.180491+0000", 2024-02-10T08:36:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058881700000000005 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.181080+0000", 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020460300000000001 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.181284+0000", 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016177120000000001 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182902+0000", 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7740000000000001e-05 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182940+0000", 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1240999999999999e-05 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.182961+0000", 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027916870000000002 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:186 4.1c 4:3b93e7b4:::obj_delete_at_hint.0000000016:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.223950+0000", 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.19255984599999, 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016635969999999999, 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 186 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223950+0000", 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223950+0000", 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223948+0000", 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6939999999999995e-06 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223956+0000", 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5959999999999993e-06 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223966+0000", 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4100000000000001e-06 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.223970+0000", 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2821999999999998e-05 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.224013+0000", 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5760000000000006e-05 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.224089+0000", 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116754 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.224205+0000", 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058302400000000002 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.224788+0000", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044827199999999999 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.225237+0000", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032470399999999999 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.225561+0000", 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1216000000000001e-05 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.225593+0000", 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1118999999999999e-05 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.225614+0000", 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016635969999999999 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.723 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:189 4.1c 4:3b93e7b4:::obj_delete_at_hint.0000000016:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.226068+0000", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.19044215599999, 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016473449999999999, 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 189 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226068+0000", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226068+0000", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226066+0000", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6850000000000004e-06 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226073+0000", 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1015e-05 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226084+0000", 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4725999999999999e-05 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226099+0000", 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0005000000000002e-05 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226129+0000", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3934999999999999e-05 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226173+0000", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000156291 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.226329+0000", 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000951026 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.227280+0000", 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3757999999999995e-05 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.227344+0000", 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031333199999999999 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.227657+0000", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8778e-05 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.227696+0000", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9355999999999999e-05 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.227715+0000", 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016473449999999999 2024-02-10T08:36:51.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:198 4.1c 4:3fb1de85:::obj_delete_at_hint.0000000018:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.230607+0000", 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.185902951, 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00170284, 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198 2024-02-10T08:36:51.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230607+0000", 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230607+0000", 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230605+0000", 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2250000000000003e-06 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230613+0000", 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3217999999999999e-05 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230626+0000", 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6699999999999997e-06 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230632+0000", 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000301775 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.230934+0000", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011343 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.231048+0000", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018190200000000001 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.231230+0000", 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000537617 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.231767+0000", 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8219999999999994e-05 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.231835+0000", 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000429188 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.232265+0000", 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6414000000000002e-05 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.232291+0000", 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8960000000000001e-05 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.232310+0000", 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00170284 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.736 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:345 4.1c 4:3d9e841d:::obj_delete_at_hint.0000000041:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.297378+0000", 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 206.11913172199999, 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035838269999999999, 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:36:51.737 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 345 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297378+0000", 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297378+0000", 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297375+0000", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1508000000000001e-05 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297387+0000", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9735e-05 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297407+0000", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9760000000000007e-06 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297417+0000", 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6069000000000002e-05 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297463+0000", 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012392699999999999 2024-02-10T08:36:51.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297587+0000", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021037200000000001 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.297797+0000", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035561000000000002 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.298153+0000", 2024-02-10T08:36:51.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020565899999999999 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.298358+0000", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025546990000000001 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.300913+0000", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9765e-05 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.300943+0000", 2024-02-10T08:36:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9466e-05 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.300962+0000", 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035838269999999999 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(4.b REQUEST e92)", 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:14.821771+0000", 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 36.594739427999997, 2024-02-10T08:36:51.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021400899999999999, 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821771+0000", 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821771+0000", 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:51.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821770+0000", 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4829999999999999e-06 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821774+0000", 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5325e-05 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821789+0000", 2024-02-10T08:36:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2239999999999995e-06 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821797+0000", 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9763999999999999e-05 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821857+0000", 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3180000000000007e-06 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.821865+0000", 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002017446 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.823883+0000", 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9539999999999993e-06 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.823892+0000", 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998999999999999e-05 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:14.823911+0000", 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021400899999999999 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(50.0s1 112/111 ECSubWrite(tid=1189, reqid=client.35222.0:3, at_version=112'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:45.968394+0000", 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.4481158040000004, 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015035690000000001, 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:36:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968394+0000", 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968394+0000", 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968393+0000", 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1041000000000001e-05 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968414+0000", 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0005e-05 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968424+0000", 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1889999999999998e-06 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968429+0000", 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6162999999999999e-05 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968465+0000", 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5200000000000003e-06 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:36:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.968472+0000", 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01492856 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.983400+0000", 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9771999999999999e-05 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.983430+0000", 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015035690000000001 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35222.0:6 50.3s0 50:c1c2e34e:test-rados-api-smithi107-112459-13::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.525575108, 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049033968999999997, 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35222", 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4111016630", 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.756 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890932+0000", 2024-02-10T08:36:51.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0301e-05 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890943+0000", 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2490999999999999e-05 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890955+0000", 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.926e-06 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890963+0000", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025738898999999999 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.916702+0000", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6725000000000007e-05 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.916799+0000", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023170130000000001 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.939969+0000", 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049033968999999997 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(50.4s2 epoch 114/111 rep_tid 1204 entries 114'2 (0'0) error 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head by client.35222.0:9 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945942+0000", 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.4705682370000002, 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005355823, 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:36:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945942+0000", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945942+0000", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945941+0000", 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0589999999999992e-06 2024-02-10T08:36:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945950+0000", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1751e-05 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945962+0000", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5779999999999999e-06 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945970+0000", 2024-02-10T08:36:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003596457 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949567+0000", 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001731018 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951298+0000", 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005355823 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25291.0:2 53.6 e114/113 53:602f83fe:::foo:head v 114'1, mlcod=0'0)", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.957743+0000", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.458766936, 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010061706, 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957743+0000", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957743+0000", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957743+0000", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.815e-06 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957752+0000", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.852e-06 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957758+0000", 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3480000000000003e-06 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957764+0000", 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010090240000000001 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.958773+0000", 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5370999999999999e-05 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.958788+0000", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089886510000000003 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967777+0000", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8153e-05 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967805+0000", 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010061706 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.14907.0:2 55.15 e114/113 55:ad551702:::test:head v 114'1, mlcod=0'0)", 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.967758+0000", 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.448751632, 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001842495, 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967758+0000", 2024-02-10T08:36:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967758+0000", 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967758+0000", 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967764+0000", 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7789999999999998e-06 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967771+0000", 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5880000000000003e-06 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967778+0000", 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000165222 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967943+0000", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2183e-05 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967955+0000", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016212399999999999 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969576+0000", 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4419000000000001e-05 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969601+0000", 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001842495 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25153.0:8 41.5 41:a378f8be:::oid_PoolQuotaPP_5:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+full_try e114)", 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:49.788097+0000", 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.628412797, 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020837951, 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25153", 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2801389734", 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788097+0000", 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788097+0000", 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999948 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788092+0000", 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3672999999999998e-05 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788116+0000", 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.703e-05 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788143+0000", 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1275e-05 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788154+0000", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6555000000000002e-05 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788221+0000", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010237499999999999 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788323+0000", 2024-02-10T08:36:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014239600000000001 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1", 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788466+0000", 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016287415999999999 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.804753+0000", 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001431383 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.806184+0000", 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026504710000000002 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.808835+0000", 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6504e-05 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.808891+0000", 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3903999999999998e-05 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.808935+0000", 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020837951 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25153.0:9 41.e e116/107 41:77268589:::oid_PoolQuotaPP_6:head v 116'1, mlcod=0'0)", 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.812614+0000", 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.60389572199999997, 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041087579999999997, 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812614+0000", 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812614+0000", 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999957 2024-02-10T08:36:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812610+0000", 2024-02-10T08:36:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1066999999999999e-05 2024-02-10T08:36:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812641+0000", 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1612000000000002e-05 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812673+0000", 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0927000000000001e-05 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812684+0000", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9789000000000005e-05 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812764+0000", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4579999999999998e-05 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.812788+0000", 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038673010000000001 2024-02-10T08:36:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.816655+0000", 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7599000000000001e-05 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.816723+0000", 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041087579999999997 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.5.0:1233 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'1, mlcod=0'0)", 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.986617+0000", 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.429893162, 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064128400000000004, 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986617+0000", 2024-02-10T08:36:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986617+0000", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986616+0000", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1534e-05 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986628+0000", 2024-02-10T08:36:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3459999999999999e-06 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986634+0000", 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5679999999999999e-06 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986640+0000", 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034309 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986983+0000", 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7921e-05 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.987001+0000", 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059966029999999997 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.992997+0000", 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2431999999999997e-05 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.993030+0000", 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064128400000000004 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:8 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'2, mlcod=0'0)", 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.986649+0000", 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.42986101500000001, 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064016350000000001, 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986649+0000", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986649+0000", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986648+0000", 2024-02-10T08:36:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6639999999999997e-06 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986653+0000", 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.181e-06 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986658+0000", 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986662+0000", 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053575499999999996 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.987198+0000", 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6752000000000002e-05 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.987215+0000", 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005818513 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.993033+0000", 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7609e-05 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.993051+0000", 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064016350000000001 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35294.0:2 61.6 61:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:51.030480+0000", 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.38603005200000001, 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069163430000000001, 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35294", 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1000879324", 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030480+0000", 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030480+0000", 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030479+0000", 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4600000000000002e-06 2024-02-10T08:36:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030484+0000", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030492+0000", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.024e-06 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030499+0000", 2024-02-10T08:36:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2543000000000001e-05 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030561+0000", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013469600000000001 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030696+0000", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015149900000000001 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.030847+0000", 2024-02-10T08:36:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050321899999999995 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.031351+0000", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016277799999999999 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.032978+0000", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043210000000000002 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.037299+0000", 2024-02-10T08:36:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3035000000000003e-05 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.037352+0000", 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3936000000000001e-05 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.037396+0000", 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069163430000000001 2024-02-10T08:36:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.5.0:1257 56.1 e117/114 56:a5beb126:test-rados-api-smithi107-112562-15::foo:head v 117'5, mlcod=117'3)", 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:51.144163+0000", 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.27234705199999998, 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053603260000000003, 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144163+0000", 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144163+0000", 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144161+0000", 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4826e-05 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144176+0000", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9748000000000001e-05 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144195+0000", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5059999999999998e-06 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144203+0000", 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035490579999999999 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.147752+0000", 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1178999999999998e-05 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.147773+0000", 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001709124 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.149482+0000", 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1081999999999997e-05 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.149523+0000", 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053603260000000003 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.805 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:50 56.1 e117/114 56:a5beb126:test-rados-api-smithi107-112562-15::foo:head v 117'6, mlcod=117'3)", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:51.144453+0000", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.27205749000000001, 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050907549999999998, 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144453+0000", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144453+0000", 2024-02-10T08:36:51.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144452+0000", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4250000000000001e-06 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144459+0000", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1510000000000001e-06 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144465+0000", 2024-02-10T08:36:51.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.121e-06 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.144469+0000", 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034642689999999999 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.147934+0000", 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5048999999999999e-05 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:36:51.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.147949+0000", 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015775559999999999 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.149526+0000", 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7002000000000001e-05 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.149543+0000", 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050907549999999998 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:36:51.810 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:36:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:36:51.810 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:51.950 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:51.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:52.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:52.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:36:52.140 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:52.177 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:52.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:52.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:52.521 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:52.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:52.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:36:52.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:50.895782+0000 mgr.y (mgr.14553) 154 : audit [DBG] from='client.35963 v1:172.21.15.107:0/2567511150' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.162104+0000 mgr.y (mgr.14553) 155 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.847089+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:51.924697+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.943176+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.947685+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:52.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.963017+0000 mon.a (mon.0) 997 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:52.651 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:52.676 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:52.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:52.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:52.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:51.970267+0000 mon.a (mon.0) 998 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:52.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:50.895782+0000 mgr.y (mgr.14553) 154 : audit [DBG] from='client.35963 v1:172.21.15.107:0/2567511150' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:52.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.162104+0000 mgr.y (mgr.14553) 155 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:52.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.847089+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:52.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:51.924697+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-10T08:36:52.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.943176+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:52.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.947685+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:52.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.963017+0000 mon.a (mon.0) 997 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:52.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:51.970267+0000 mon.a (mon.0) 998 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:50.895782+0000 mgr.y (mgr.14553) 154 : audit [DBG] from='client.35963 v1:172.21.15.107:0/2567511150' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.162104+0000 mgr.y (mgr.14553) 155 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.847089+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:51.924697+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.943176+0000 mon.c (mon.2) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.947685+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.963017+0000 mon.a (mon.0) 997 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:51.970267+0000 mon.a (mon.0) 998 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.167 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:36:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:53.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:36:53.199 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-02-10T08:36:50.894037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151016182,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":1941,"num_read_kb":49843,"num_write":1590,"num_write_kb":153102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1658,"ondisk_log_size":1658,"up":1149,"acting":1149,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1796,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2885356,"kb_used_data":451244,"kb_used_omap":0,"kb_used_meta":2434048,"kb_avail":747075860,"statfs":{"total":767960285184,"available":765005680640,"internally_reserved":0,"allocated":462073856,"data_stored":457395139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2492465152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-520,"num_objects":-12,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12,"num_whiteouts":0,"num_read":-10,"num_read_kb":-1,"num_write":-14,"num_write_kb":-6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010155"},"pg_stats":[{"pgid":"62.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.891238+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289030+0000","last_change":"2024-02-10T08:36:44.326695+0000","last_active":"2024-02-10T08:36:45.289030+0000","last_peered":"2024-02-10T08:36:45.289030+0000","last_clean":"2024-02-10T08:36:45.289030+0000","last_became_active":"2024-02-10T08:36:44.326375+0000","last_became_peered":"2024-02-10T08:36:44.326375+0000","last_unstale":"2024-02-10T08:36:45.289030+0000","last_undegraded":"2024-02-10T08:36:45.289030+0000","last_fullsized":"2024-02-10T08:36:45.289030+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288611+0000","last_change":"2024-02-10T08:36:42.325498+0000","last_active":"2024-02-10T08:36:45.288611+0000","last_peered":"2024-02-10T08:36:45.288611+0000","last_clean":"2024-02-10T08:36:45.288611+0000","last_became_active":"2024-02-10T08:36:42.325169+0000","last_became_peered":"2024-02-10T08:36:42.325169+0000","last_unstale":"2024-02-10T08:36:45.288611+0000","last_undegraded":"2024-02-10T08:36:45.288611+0000","last_fullsized":"2024-02-10T08:36:45.288611+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306038+0000","last_change":"2024-02-10T08:36:40.086125+0000","last_active":"2024-02-10T08:36:45.306038+0000","last_peered":"2024-02-10T08:36:45.306038+0000","last_clean":"2024-02-10T08:36:45.306038+0000","last_became_active":"2024-02-10T08:36:40.085844+0000","last_became_peered":"2024-02-10T08:36:40.085844+0000","last_unstale":"2024-02-10T08:36:45.306038+0000","last_undegraded":"2024-02-10T08:36:45.306038+0000","last_fullsized":"2024-02-10T08:36:45.306038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"62.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.890324+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.10","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309589+0000","last_change":"2024-02-10T08:36:44.320223+0000","last_active":"2024-02-10T08:36:45.309589+0000","last_peered":"2024-02-10T08:36:45.309589+0000","last_clean":"2024-02-10T08:36:45.309589+0000","last_became_active":"2024-02-10T08:36:44.319950+0000","last_became_peered":"2024-02-10T08:36:44.319950+0000","last_unstale":"2024-02-10T08:36:45.309589+0000","last_undegraded":"2024-02-10T08:36:45.309589+0000","last_fullsized":"2024-02-10T08:36:45.309589+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.284074+0000","last_change":"2024-02-10T08:36:42.329996+0000","last_active":"2024-02-10T08:36:45.284074+0000","last_peered":"2024-02-10T08:36:45.284074+0000","last_clean":"2024-02-10T08:36:45.284074+0000","last_became_active":"2024-02-10T08:36:42.329763+0000","last_became_peered":"2024-02-10T08:36:42.329763+0000","last_unstale":"2024-02-10T08:36:45.284074+0000","last_undegraded":"2024-02-10T08:36:45.284074+0000","last_fullsized":"2024-02-10T08:36:45.284074+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302957+0000","last_change":"2024-02-10T08:36:40.103457+0000","last_active":"2024-02-10T08:36:45.302957+0000","last_peered":"2024-02-10T08:36:45.302957+0000","last_clean":"2024-02-10T08:36:45.302957+0000","last_became_active":"2024-02-10T08:36:40.103180+0000","last_became_peered":"2024-02-10T08:36:40.103180+0000","last_unstale":"2024-02-10T08:36:45.302957+0000","last_undegraded":"2024-02-10T08:36:45.302957+0000","last_fullsized":"2024-02-10T08:36:45.302957+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"62.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.895044+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.13","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290312+0000","last_change":"2024-02-10T08:36:44.299177+0000","last_active":"2024-02-10T08:36:45.290312+0000","last_peered":"2024-02-10T08:36:45.290312+0000","last_clean":"2024-02-10T08:36:45.290312+0000","last_became_active":"2024-02-10T08:36:44.298914+0000","last_became_peered":"2024-02-10T08:36:44.298914+0000","last_unstale":"2024-02-10T08:36:45.290312+0000","last_undegraded":"2024-02-10T08:36:45.290312+0000","last_fullsized":"2024-02-10T08:36:45.290312+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309502+0000","last_change":"2024-02-10T08:36:42.309521+0000","last_active":"2024-02-10T08:36:45.309502+0000","last_peered":"2024-02-10T08:36:45.309502+0000","last_clean":"2024-02-10T08:36:45.309502+0000","last_became_active":"2024-02-10T08:36:42.309263+0000","last_became_peered":"2024-02-10T08:36:42.309263+0000","last_unstale":"2024-02-10T08:36:45.309502+0000","last_undegraded":"2024-02-10T08:36:45.309502+0000","last_fullsized":"2024-02-10T08:36:45.309502+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.317045+0000","last_change":"2024-02-10T08:36:40.088586+0000","last_active":"2024-02-10T08:36:45.317045+0000","last_peered":"2024-02-10T08:36:45.317045+0000","last_clean":"2024-02-10T08:36:45.317045+0000","last_became_active":"2024-02-10T08:36:40.059375+0000","last_became_peered":"2024-02-10T08:36:40.059375+0000","last_unstale":"2024-02-10T08:36:45.317045+0000","last_undegraded":"2024-02-10T08:36:45.317045+0000","last_fullsized":"2024-02-10T08:36:45.317045+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"62.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.887857+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327838+0000","last_change":"2024-02-10T08:36:44.321501+0000","last_active":"2024-02-10T08:36:45.327838+0000","last_peered":"2024-02-10T08:36:45.327838+0000","last_clean":"2024-02-10T08:36:45.327838+0000","last_became_active":"2024-02-10T08:36:44.321238+0000","last_became_peered":"2024-02-10T08:36:44.321238+0000","last_unstale":"2024-02-10T08:36:45.327838+0000","last_undegraded":"2024-02-10T08:36:45.327838+0000","last_fullsized":"2024-02-10T08:36:45.327838+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314579+0000","last_change":"2024-02-10T08:36:42.336637+0000","last_active":"2024-02-10T08:36:45.314579+0000","last_peered":"2024-02-10T08:36:45.314579+0000","last_clean":"2024-02-10T08:36:45.314579+0000","last_became_active":"2024-02-10T08:36:42.336223+0000","last_became_peered":"2024-02-10T08:36:42.336223+0000","last_unstale":"2024-02-10T08:36:45.314579+0000","last_undegraded":"2024-02-10T08:36:45.314579+0000","last_fullsized":"2024-02-10T08:36:45.314579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.304565+0000","last_change":"2024-02-10T08:36:40.096181+0000","last_active":"2024-02-10T08:36:45.304565+0000","last_peered":"2024-02-10T08:36:45.304565+0000","last_clean":"2024-02-10T08:36:45.304565+0000","last_became_active":"2024-02-10T08:36:40.095871+0000","last_became_peered":"2024-02-10T08:36:40.095871+0000","last_unstale":"2024-02-10T08:36:45.304565+0000","last_undegraded":"2024-02-10T08:36:45.304565+0000","last_fullsized":"2024-02-10T08:36:45.304565+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"62.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:47.882934+0000","last_change":"2024-02-10T08:36:47.882934+0000","last_active":"2024-02-10T08:36:47.882934+0000","last_peered":"2024-02-10T08:36:47.882934+0000","last_clean":"2024-02-10T08:36:47.882934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:47.882934+0000","last_undegraded":"2024-02-10T08:36:47.882934+0000","last_fullsized":"2024-02-10T08:36:47.882934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_clean_scrub_stamp":"2024-02-10T08:36:47.882934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.886554+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.15","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.308130+0000","last_change":"2024-02-10T08:36:44.312363+0000","last_active":"2024-02-10T08:36:45.308130+0000","last_peered":"2024-02-10T08:36:45.308130+0000","last_clean":"2024-02-10T08:36:45.308130+0000","last_became_active":"2024-02-10T08:36:44.311804+0000","last_became_peered":"2024-02-10T08:36:44.311804+0000","last_unstale":"2024-02-10T08:36:45.308130+0000","last_undegraded":"2024-02-10T08:36:45.308130+0000","last_fullsized":"2024-02-10T08:36:45.308130+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.328743+0000","last_change":"2024-02-10T08:36:42.308081+0000","last_active":"2024-02-10T08:36:45.328743+0000","last_peered":"2024-02-10T08:36:45.328743+0000","last_clean":"2024-02-10T08:36:45.328743+0000","last_became_active":"2024-02-10T08:36:42.306909+0000","last_became_peered":"2024-02-10T08:36:42.306909+0000","last_unstale":"2024-02-10T08:36:45.328743+0000","last_undegraded":"2024-02-10T08:36:45.328743+0000","last_fullsized":"2024-02-10T08:36:45.328743+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.278888+0000","last_change":"2024-02-10T08:36:40.093933+0000","last_active":"2024-02-10T08:36:45.278888+0000","last_peered":"2024-02-10T08:36:45.278888+0000","last_clean":"2024-02-10T08:36:45.278888+0000","last_became_active":"2024-02-10T08:36:40.093396+0000","last_became_peered":"2024-02-10T08:36:40.093396+0000","last_unstale":"2024-02-10T08:36:45.278888+0000","last_undegraded":"2024-02-10T08:36:45.278888+0000","last_fullsized":"2024-02-10T08:36:45.278888+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"62.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:47.882934+0000","last_change":"2024-02-10T08:36:47.882934+0000","last_active":"2024-02-10T08:36:47.882934+0000","last_peered":"2024-02-10T08:36:47.882934+0000","last_clean":"2024-02-10T08:36:47.882934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:47.882934+0000","last_undegraded":"2024-02-10T08:36:47.882934+0000","last_fullsized":"2024-02-10T08:36:47.882934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_clean_scrub_stamp":"2024-02-10T08:36:47.882934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.892468+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.325786+0000","last_change":"2024-02-10T08:36:44.323045+0000","last_active":"2024-02-10T08:36:45.325786+0000","last_peered":"2024-02-10T08:36:45.325786+0000","last_clean":"2024-02-10T08:36:45.325786+0000","last_became_active":"2024-02-10T08:36:44.322750+0000","last_became_peered":"2024-02-10T08:36:44.322750+0000","last_unstale":"2024-02-10T08:36:45.325786+0000","last_undegraded":"2024-02-10T08:36:45.325786+0000","last_fullsized":"2024-02-10T08:36:45.325786+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.13","version":"110'1","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303136+0000","last_change":"2024-02-10T08:36:42.301634+0000","last_active":"2024-02-10T08:36:45.303136+0000","last_peered":"2024-02-10T08:36:45.303136+0000","last_clean":"2024-02-10T08:36:45.303136+0000","last_became_active":"2024-02-10T08:36:42.301301+0000","last_became_peered":"2024-02-10T08:36:42.301301+0000","last_unstale":"2024-02-10T08:36:45.303136+0000","last_undegraded":"2024-02-10T08:36:45.303136+0000","last_fullsized":"2024-02-10T08:36:45.303136+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292218+0000","last_change":"2024-02-10T08:36:41.152301+0000","last_active":"2024-02-10T08:36:45.292218+0000","last_peered":"2024-02-10T08:36:45.292218+0000","last_clean":"2024-02-10T08:36:45.292218+0000","last_became_active":"2024-02-10T08:36:41.151995+0000","last_became_peered":"2024-02-10T08:36:41.151995+0000","last_unstale":"2024-02-10T08:36:45.292218+0000","last_undegraded":"2024-02-10T08:36:45.292218+0000","last_fullsized":"2024-02-10T08:36:45.292218+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:47.882934+0000","last_change":"2024-02-10T08:36:47.882934+0000","last_active":"2024-02-10T08:36:47.882934+0000","last_peered":"2024-02-10T08:36:47.882934+0000","last_clean":"2024-02-10T08:36:47.882934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:47.882934+0000","last_undegraded":"2024-02-10T08:36:47.882934+0000","last_fullsized":"2024-02-10T08:36:47.882934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_clean_scrub_stamp":"2024-02-10T08:36:47.882934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.895192+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.284058+0000","last_change":"2024-02-10T08:36:44.311229+0000","last_active":"2024-02-10T08:36:45.284058+0000","last_peered":"2024-02-10T08:36:45.284058+0000","last_clean":"2024-02-10T08:36:45.284058+0000","last_became_active":"2024-02-10T08:36:44.310361+0000","last_became_peered":"2024-02-10T08:36:44.310361+0000","last_unstale":"2024-02-10T08:36:45.284058+0000","last_undegraded":"2024-02-10T08:36:45.284058+0000","last_fullsized":"2024-02-10T08:36:45.284058+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.282291+0000","last_change":"2024-02-10T08:36:42.157988+0000","last_active":"2024-02-10T08:36:45.282291+0000","last_peered":"2024-02-10T08:36:45.282291+0000","last_clean":"2024-02-10T08:36:45.282291+0000","last_became_active":"2024-02-10T08:36:42.157745+0000","last_became_peered":"2024-02-10T08:36:42.157745+0000","last_unstale":"2024-02-10T08:36:45.282291+0000","last_undegraded":"2024-02-10T08:36:45.282291+0000","last_fullsized":"2024-02-10T08:36:45.282291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.284703+0000","last_change":"2024-02-10T08:36:41.145416+0000","last_active":"2024-02-10T08:36:45.284703+0000","last_peered":"2024-02-10T08:36:45.284703+0000","last_clean":"2024-02-10T08:36:45.284703+0000","last_became_active":"2024-02-10T08:36:41.145133+0000","last_became_peered":"2024-02-10T08:36:41.145133+0000","last_unstale":"2024-02-10T08:36:45.284703+0000","last_undegraded":"2024-02-10T08:36:45.284703+0000","last_fullsized":"2024-02-10T08:36:45.284703+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"62.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"58.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:47.882934+0000","last_change":"2024-02-10T08:36:47.882934+0000","last_active":"2024-02-10T08:36:47.882934+0000","last_peered":"2024-02-10T08:36:47.882934+0000","last_clean":"2024-02-10T08:36:47.882934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:47.882934+0000","last_undegraded":"2024-02-10T08:36:47.882934+0000","last_fullsized":"2024-02-10T08:36:47.882934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:47.882934+0000","last_clean_scrub_stamp":"2024-02-10T08:36:47.882934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.896287+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.282088+0000","last_change":"2024-02-10T08:36:44.326197+0000","last_active":"2024-02-10T08:36:45.282088+0000","last_peered":"2024-02-10T08:36:45.282088+0000","last_clean":"2024-02-10T08:36:45.282088+0000","last_became_active":"2024-02-10T08:36:44.325866+0000","last_became_peered":"2024-02-10T08:36:44.325866+0000","last_unstale":"2024-02-10T08:36:45.282088+0000","last_undegraded":"2024-02-10T08:36:45.282088+0000","last_fullsized":"2024-02-10T08:36:45.282088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292033+0000","last_change":"2024-02-10T08:36:42.185873+0000","last_active":"2024-02-10T08:36:45.292033+0000","last_peered":"2024-02-10T08:36:45.292033+0000","last_clean":"2024-02-10T08:36:45.292033+0000","last_became_active":"2024-02-10T08:36:42.185610+0000","last_became_peered":"2024-02-10T08:36:42.185610+0000","last_unstale":"2024-02-10T08:36:45.292033+0000","last_undegraded":"2024-02-10T08:36:45.292033+0000","last_fullsized":"2024-02-10T08:36:45.292033+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.324239+0000","last_change":"2024-02-10T08:36:40.100488+0000","last_active":"2024-02-10T08:36:45.324239+0000","last_peered":"2024-02-10T08:36:45.324239+0000","last_clean":"2024-02-10T08:36:45.324239+0000","last_became_active":"2024-02-10T08:36:40.100222+0000","last_became_peered":"2024-02-10T08:36:40.100222+0000","last_unstale":"2024-02-10T08:36:45.324239+0000","last_undegraded":"2024-02-10T08:36:45.324239+0000","last_fullsized":"2024-02-10T08:36:45.324239+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"62.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.890815+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.300962+0000","last_change":"2024-02-10T08:36:44.327876+0000","last_active":"2024-02-10T08:36:45.300962+0000","last_peered":"2024-02-10T08:36:45.300962+0000","last_clean":"2024-02-10T08:36:45.300962+0000","last_became_active":"2024-02-10T08:36:44.327577+0000","last_became_peered":"2024-02-10T08:36:44.327577+0000","last_unstale":"2024-02-10T08:36:45.300962+0000","last_undegraded":"2024-02-10T08:36:45.300962+0000","last_fullsized":"2024-02-10T08:36:45.300962+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327290+0000","last_change":"2024-02-10T08:36:42.325594+0000","last_active":"2024-02-10T08:36:45.327290+0000","last_peered":"2024-02-10T08:36:45.327290+0000","last_clean":"2024-02-10T08:36:45.327290+0000","last_became_active":"2024-02-10T08:36:42.325232+0000","last_became_peered":"2024-02-10T08:36:42.325232+0000","last_unstale":"2024-02-10T08:36:45.327290+0000","last_undegraded":"2024-02-10T08:36:45.327290+0000","last_fullsized":"2024-02-10T08:36:45.327290+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.296279+0000","last_change":"2024-02-10T08:36:40.079651+0000","last_active":"2024-02-10T08:36:45.296279+0000","last_peered":"2024-02-10T08:36:45.296279+0000","last_clean":"2024-02-10T08:36:45.296279+0000","last_became_active":"2024-02-10T08:36:40.079358+0000","last_became_peered":"2024-02-10T08:36:40.079358+0000","last_unstale":"2024-02-10T08:36:45.296279+0000","last_undegraded":"2024-02-10T08:36:45.296279+0000","last_fullsized":"2024-02-10T08:36:45.296279+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"62.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.899350+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.312013+0000","last_change":"2024-02-10T08:36:44.322741+0000","last_active":"2024-02-10T08:36:45.312013+0000","last_peered":"2024-02-10T08:36:45.312013+0000","last_clean":"2024-02-10T08:36:45.312013+0000","last_became_active":"2024-02-10T08:36:44.322069+0000","last_became_peered":"2024-02-10T08:36:44.322069+0000","last_unstale":"2024-02-10T08:36:45.312013+0000","last_undegraded":"2024-02-10T08:36:45.312013+0000","last_fullsized":"2024-02-10T08:36:45.312013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280712+0000","last_change":"2024-02-10T08:36:42.330455+0000","last_active":"2024-02-10T08:36:45.280712+0000","last_peered":"2024-02-10T08:36:45.280712+0000","last_clean":"2024-02-10T08:36:45.280712+0000","last_became_active":"2024-02-10T08:36:42.329951+0000","last_became_peered":"2024-02-10T08:36:42.329951+0000","last_unstale":"2024-02-10T08:36:45.280712+0000","last_undegraded":"2024-02-10T08:36:45.280712+0000","last_fullsized":"2024-02-10T08:36:45.280712+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.284018+0000","last_change":"2024-02-10T08:36:41.093070+0000","last_active":"2024-02-10T08:36:45.284018+0000","last_peered":"2024-02-10T08:36:45.284018+0000","last_clean":"2024-02-10T08:36:45.284018+0000","last_became_active":"2024-02-10T08:36:41.092333+0000","last_became_peered":"2024-02-10T08:36:41.092333+0000","last_unstale":"2024-02-10T08:36:45.284018+0000","last_undegraded":"2024-02-10T08:36:45.284018+0000","last_fullsized":"2024-02-10T08:36:45.284018+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"62.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.902826+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.1b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.322481+0000","last_change":"2024-02-10T08:36:44.324094+0000","last_active":"2024-02-10T08:36:45.322481+0000","last_peered":"2024-02-10T08:36:45.322481+0000","last_clean":"2024-02-10T08:36:45.322481+0000","last_became_active":"2024-02-10T08:36:44.323789+0000","last_became_peered":"2024-02-10T08:36:44.323789+0000","last_unstale":"2024-02-10T08:36:45.322481+0000","last_undegraded":"2024-02-10T08:36:45.322481+0000","last_fullsized":"2024-02-10T08:36:45.322481+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281872+0000","last_change":"2024-02-10T08:36:42.182693+0000","last_active":"2024-02-10T08:36:45.281872+0000","last_peered":"2024-02-10T08:36:45.281872+0000","last_clean":"2024-02-10T08:36:45.281872+0000","last_became_active":"2024-02-10T08:36:42.182464+0000","last_became_peered":"2024-02-10T08:36:42.182464+0000","last_unstale":"2024-02-10T08:36:45.281872+0000","last_undegraded":"2024-02-10T08:36:45.281872+0000","last_fullsized":"2024-02-10T08:36:45.281872+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316437+0000","last_change":"2024-02-10T08:36:41.137643+0000","last_active":"2024-02-10T08:36:45.316437+0000","last_peered":"2024-02-10T08:36:45.316437+0000","last_clean":"2024-02-10T08:36:45.316437+0000","last_became_active":"2024-02-10T08:36:41.137150+0000","last_became_peered":"2024-02-10T08:36:41.137150+0000","last_unstale":"2024-02-10T08:36:45.316437+0000","last_undegraded":"2024-02-10T08:36:45.316437+0000","last_fullsized":"2024-02-10T08:36:45.316437+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.883168+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"46.1a","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.283705+0000","last_change":"2024-02-10T08:36:44.303836+0000","last_active":"2024-02-10T08:36:45.283705+0000","last_peered":"2024-02-10T08:36:45.283705+0000","last_clean":"2024-02-10T08:36:45.283705+0000","last_became_active":"2024-02-10T08:36:44.303557+0000","last_became_peered":"2024-02-10T08:36:44.303557+0000","last_unstale":"2024-02-10T08:36:45.283705+0000","last_undegraded":"2024-02-10T08:36:45.283705+0000","last_fullsized":"2024-02-10T08:36:45.283705+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.308291+0000","last_change":"2024-02-10T08:36:42.173955+0000","last_active":"2024-02-10T08:36:45.308291+0000","last_peered":"2024-02-10T08:36:45.308291+0000","last_clean":"2024-02-10T08:36:45.308291+0000","last_became_active":"2024-02-10T08:36:42.173652+0000","last_became_peered":"2024-02-10T08:36:42.173652+0000","last_unstale":"2024-02-10T08:36:45.308291+0000","last_undegraded":"2024-02-10T08:36:45.308291+0000","last_fullsized":"2024-02-10T08:36:45.308291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.284285+0000","last_change":"2024-02-10T08:36:41.106917+0000","last_active":"2024-02-10T08:36:45.284285+0000","last_peered":"2024-02-10T08:36:45.284285+0000","last_clean":"2024-02-10T08:36:45.284285+0000","last_became_active":"2024-02-10T08:36:41.105326+0000","last_became_peered":"2024-02-10T08:36:41.105326+0000","last_unstale":"2024-02-10T08:36:45.284285+0000","last_undegraded":"2024-02-10T08:36:45.284285+0000","last_fullsized":"2024-02-10T08:36:45.284285+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"62.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.904294+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302782+0000","last_change":"2024-02-10T08:36:44.333337+0000","last_active":"2024-02-10T08:36:45.302782+0000","last_peered":"2024-02-10T08:36:45.302782+0000","last_clean":"2024-02-10T08:36:45.302782+0000","last_became_active":"2024-02-10T08:36:44.333077+0000","last_became_peered":"2024-02-10T08:36:44.333077+0000","last_unstale":"2024-02-10T08:36:45.302782+0000","last_undegraded":"2024-02-10T08:36:45.302782+0000","last_fullsized":"2024-02-10T08:36:45.302782+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279979+0000","last_change":"2024-02-10T08:36:42.161141+0000","last_active":"2024-02-10T08:36:45.279979+0000","last_peered":"2024-02-10T08:36:45.279979+0000","last_clean":"2024-02-10T08:36:45.279979+0000","last_became_active":"2024-02-10T08:36:42.160897+0000","last_became_peered":"2024-02-10T08:36:42.160897+0000","last_unstale":"2024-02-10T08:36:45.279979+0000","last_undegraded":"2024-02-10T08:36:45.279979+0000","last_fullsized":"2024-02-10T08:36:45.279979+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.297939+0000","last_change":"2024-02-10T08:36:40.070956+0000","last_active":"2024-02-10T08:36:45.297939+0000","last_peered":"2024-02-10T08:36:45.297939+0000","last_clean":"2024-02-10T08:36:45.297939+0000","last_became_active":"2024-02-10T08:36:40.070663+0000","last_became_peered":"2024-02-10T08:36:40.070663+0000","last_unstale":"2024-02-10T08:36:45.297939+0000","last_undegraded":"2024-02-10T08:36:45.297939+0000","last_fullsized":"2024-02-10T08:36:45.297939+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"62.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.908122+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.296125+0000","last_change":"2024-02-10T08:36:44.333227+0000","last_active":"2024-02-10T08:36:45.296125+0000","last_peered":"2024-02-10T08:36:45.296125+0000","last_clean":"2024-02-10T08:36:45.296125+0000","last_became_active":"2024-02-10T08:36:44.332834+0000","last_became_peered":"2024-02-10T08:36:44.332834+0000","last_unstale":"2024-02-10T08:36:45.296125+0000","last_undegraded":"2024-02-10T08:36:45.296125+0000","last_fullsized":"2024-02-10T08:36:45.296125+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307508+0000","last_change":"2024-02-10T08:36:42.333171+0000","last_active":"2024-02-10T08:36:45.307508+0000","last_peered":"2024-02-10T08:36:45.307508+0000","last_clean":"2024-02-10T08:36:45.307508+0000","last_became_active":"2024-02-10T08:36:42.332899+0000","last_became_peered":"2024-02-10T08:36:42.332899+0000","last_unstale":"2024-02-10T08:36:45.307508+0000","last_undegraded":"2024-02-10T08:36:45.307508+0000","last_fullsized":"2024-02-10T08:36:45.307508+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.315680+0000","last_change":"2024-02-10T08:36:40.081654+0000","last_active":"2024-02-10T08:36:45.315680+0000","last_peered":"2024-02-10T08:36:45.315680+0000","last_clean":"2024-02-10T08:36:45.315680+0000","last_became_active":"2024-02-10T08:36:40.081383+0000","last_became_peered":"2024-02-10T08:36:40.081383+0000","last_unstale":"2024-02-10T08:36:45.315680+0000","last_undegraded":"2024-02-10T08:36:45.315680+0000","last_fullsized":"2024-02-10T08:36:45.315680+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"62.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.897266+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280394+0000","last_change":"2024-02-10T08:36:44.326355+0000","last_active":"2024-02-10T08:36:45.280394+0000","last_peered":"2024-02-10T08:36:45.280394+0000","last_clean":"2024-02-10T08:36:45.280394+0000","last_became_active":"2024-02-10T08:36:44.326068+0000","last_became_peered":"2024-02-10T08:36:44.326068+0000","last_unstale":"2024-02-10T08:36:45.280394+0000","last_undegraded":"2024-02-10T08:36:45.280394+0000","last_fullsized":"2024-02-10T08:36:45.280394+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287011+0000","last_change":"2024-02-10T08:36:42.179145+0000","last_active":"2024-02-10T08:36:45.287011+0000","last_peered":"2024-02-10T08:36:45.287011+0000","last_clean":"2024-02-10T08:36:45.287011+0000","last_became_active":"2024-02-10T08:36:42.178837+0000","last_became_peered":"2024-02-10T08:36:42.178837+0000","last_unstale":"2024-02-10T08:36:45.287011+0000","last_undegraded":"2024-02-10T08:36:45.287011+0000","last_fullsized":"2024-02-10T08:36:45.287011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.328882+0000","last_change":"2024-02-10T08:36:40.090018+0000","last_active":"2024-02-10T08:36:45.328882+0000","last_peered":"2024-02-10T08:36:45.328882+0000","last_clean":"2024-02-10T08:36:45.328882+0000","last_became_active":"2024-02-10T08:36:40.089827+0000","last_became_peered":"2024-02-10T08:36:40.089827+0000","last_unstale":"2024-02-10T08:36:45.328882+0000","last_undegraded":"2024-02-10T08:36:45.328882+0000","last_fullsized":"2024-02-10T08:36:45.328882+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"62.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.885933+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310385+0000","last_change":"2024-02-10T08:36:44.317228+0000","last_active":"2024-02-10T08:36:45.310385+0000","last_peered":"2024-02-10T08:36:45.310385+0000","last_clean":"2024-02-10T08:36:45.310385+0000","last_became_active":"2024-02-10T08:36:44.316907+0000","last_became_peered":"2024-02-10T08:36:44.316907+0000","last_unstale":"2024-02-10T08:36:45.310385+0000","last_undegraded":"2024-02-10T08:36:45.310385+0000","last_fullsized":"2024-02-10T08:36:45.310385+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306028+0000","last_change":"2024-02-10T08:36:42.328352+0000","last_active":"2024-02-10T08:36:45.306028+0000","last_peered":"2024-02-10T08:36:45.306028+0000","last_clean":"2024-02-10T08:36:45.306028+0000","last_became_active":"2024-02-10T08:36:42.328082+0000","last_became_peered":"2024-02-10T08:36:42.328082+0000","last_unstale":"2024-02-10T08:36:45.306028+0000","last_undegraded":"2024-02-10T08:36:45.306028+0000","last_fullsized":"2024-02-10T08:36:45.306028+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303495+0000","last_change":"2024-02-10T08:36:41.121770+0000","last_active":"2024-02-10T08:36:45.303495+0000","last_peered":"2024-02-10T08:36:45.303495+0000","last_clean":"2024-02-10T08:36:45.303495+0000","last_became_active":"2024-02-10T08:36:41.121558+0000","last_became_peered":"2024-02-10T08:36:41.121558+0000","last_unstale":"2024-02-10T08:36:45.303495+0000","last_undegraded":"2024-02-10T08:36:45.303495+0000","last_fullsized":"2024-02-10T08:36:45.303495+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.902467+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.315088+0000","last_change":"2024-02-10T08:36:44.320107+0000","last_active":"2024-02-10T08:36:45.315088+0000","last_peered":"2024-02-10T08:36:45.315088+0000","last_clean":"2024-02-10T08:36:45.315088+0000","last_became_active":"2024-02-10T08:36:44.319836+0000","last_became_peered":"2024-02-10T08:36:44.319836+0000","last_unstale":"2024-02-10T08:36:45.315088+0000","last_undegraded":"2024-02-10T08:36:45.315088+0000","last_fullsized":"2024-02-10T08:36:45.315088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288613+0000","last_change":"2024-02-10T08:36:42.173814+0000","last_active":"2024-02-10T08:36:45.288613+0000","last_peered":"2024-02-10T08:36:45.288613+0000","last_clean":"2024-02-10T08:36:45.288613+0000","last_became_active":"2024-02-10T08:36:42.173414+0000","last_became_peered":"2024-02-10T08:36:42.173414+0000","last_unstale":"2024-02-10T08:36:45.288613+0000","last_undegraded":"2024-02-10T08:36:45.288613+0000","last_fullsized":"2024-02-10T08:36:45.288613+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310749+0000","last_change":"2024-02-10T08:36:41.107264+0000","last_active":"2024-02-10T08:36:45.310749+0000","last_peered":"2024-02-10T08:36:45.310749+0000","last_clean":"2024-02-10T08:36:45.310749+0000","last_became_active":"2024-02-10T08:36:41.105258+0000","last_became_peered":"2024-02-10T08:36:41.105258+0000","last_unstale":"2024-02-10T08:36:45.310749+0000","last_undegraded":"2024-02-10T08:36:45.310749+0000","last_fullsized":"2024-02-10T08:36:45.310749+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.898116+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.278644+0000","last_change":"2024-02-10T08:36:44.306098+0000","last_active":"2024-02-10T08:36:45.278644+0000","last_peered":"2024-02-10T08:36:45.278644+0000","last_clean":"2024-02-10T08:36:45.278644+0000","last_became_active":"2024-02-10T08:36:44.305820+0000","last_became_peered":"2024-02-10T08:36:44.305820+0000","last_unstale":"2024-02-10T08:36:45.278644+0000","last_undegraded":"2024-02-10T08:36:45.278644+0000","last_fullsized":"2024-02-10T08:36:45.278644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332003+0000","last_change":"2024-02-10T08:36:42.332266+0000","last_active":"2024-02-10T08:36:45.332003+0000","last_peered":"2024-02-10T08:36:45.332003+0000","last_clean":"2024-02-10T08:36:45.332003+0000","last_became_active":"2024-02-10T08:36:42.332025+0000","last_became_peered":"2024-02-10T08:36:42.332025+0000","last_unstale":"2024-02-10T08:36:45.332003+0000","last_undegraded":"2024-02-10T08:36:45.332003+0000","last_fullsized":"2024-02-10T08:36:45.332003+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305806+0000","last_change":"2024-02-10T08:36:40.088010+0000","last_active":"2024-02-10T08:36:45.305806+0000","last_peered":"2024-02-10T08:36:45.305806+0000","last_clean":"2024-02-10T08:36:45.305806+0000","last_became_active":"2024-02-10T08:36:40.087640+0000","last_became_peered":"2024-02-10T08:36:40.087640+0000","last_unstale":"2024-02-10T08:36:45.305806+0000","last_undegraded":"2024-02-10T08:36:45.305806+0000","last_fullsized":"2024-02-10T08:36:45.305806+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"62.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.885316+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279644+0000","last_change":"2024-02-10T08:36:44.309311+0000","last_active":"2024-02-10T08:36:45.279644+0000","last_peered":"2024-02-10T08:36:45.279644+0000","last_clean":"2024-02-10T08:36:45.279644+0000","last_became_active":"2024-02-10T08:36:44.309024+0000","last_became_peered":"2024-02-10T08:36:44.309024+0000","last_unstale":"2024-02-10T08:36:45.279644+0000","last_undegraded":"2024-02-10T08:36:45.279644+0000","last_fullsized":"2024-02-10T08:36:45.279644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.326005+0000","last_change":"2024-02-10T08:36:42.338161+0000","last_active":"2024-02-10T08:36:45.326005+0000","last_peered":"2024-02-10T08:36:45.326005+0000","last_clean":"2024-02-10T08:36:45.326005+0000","last_became_active":"2024-02-10T08:36:42.337880+0000","last_became_peered":"2024-02-10T08:36:42.337880+0000","last_unstale":"2024-02-10T08:36:45.326005+0000","last_undegraded":"2024-02-10T08:36:45.326005+0000","last_fullsized":"2024-02-10T08:36:45.326005+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.282368+0000","last_change":"2024-02-10T08:36:41.124420+0000","last_active":"2024-02-10T08:36:45.282368+0000","last_peered":"2024-02-10T08:36:45.282368+0000","last_clean":"2024-02-10T08:36:45.282368+0000","last_became_active":"2024-02-10T08:36:41.124005+0000","last_became_peered":"2024-02-10T08:36:41.124005+0000","last_unstale":"2024-02-10T08:36:45.282368+0000","last_undegraded":"2024-02-10T08:36:45.282368+0000","last_fullsized":"2024-02-10T08:36:45.282368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"62.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.888306+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302985+0000","last_change":"2024-02-10T08:36:44.319858+0000","last_active":"2024-02-10T08:36:45.302985+0000","last_peered":"2024-02-10T08:36:45.302985+0000","last_clean":"2024-02-10T08:36:45.302985+0000","last_became_active":"2024-02-10T08:36:44.319562+0000","last_became_peered":"2024-02-10T08:36:44.319562+0000","last_unstale":"2024-02-10T08:36:45.302985+0000","last_undegraded":"2024-02-10T08:36:45.302985+0000","last_fullsized":"2024-02-10T08:36:45.302985+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.5","version":"111'1","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.782031+0000","last_change":"2024-02-10T08:36:42.339394+0000","last_active":"2024-02-10T08:36:45.782031+0000","last_peered":"2024-02-10T08:36:45.782031+0000","last_clean":"2024-02-10T08:36:45.782031+0000","last_became_active":"2024-02-10T08:36:42.339039+0000","last_became_peered":"2024-02-10T08:36:42.339039+0000","last_unstale":"2024-02-10T08:36:45.782031+0000","last_undegraded":"2024-02-10T08:36:45.782031+0000","last_fullsized":"2024-02-10T08:36:45.782031+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309886+0000","last_change":"2024-02-10T08:36:40.079878+0000","last_active":"2024-02-10T08:36:45.309886+0000","last_peered":"2024-02-10T08:36:45.309886+0000","last_clean":"2024-02-10T08:36:45.309886+0000","last_became_active":"2024-02-10T08:36:40.079222+0000","last_became_peered":"2024-02-10T08:36:40.079222+0000","last_unstale":"2024-02-10T08:36:45.309886+0000","last_undegraded":"2024-02-10T08:36:45.309886+0000","last_fullsized":"2024-02-10T08:36:45.309886+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"62.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.895345+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302863+0000","last_change":"2024-02-10T08:36:44.323740+0000","last_active":"2024-02-10T08:36:45.302863+0000","last_peered":"2024-02-10T08:36:45.302863+0000","last_clean":"2024-02-10T08:36:45.302863+0000","last_became_active":"2024-02-10T08:36:44.323460+0000","last_became_peered":"2024-02-10T08:36:44.323460+0000","last_unstale":"2024-02-10T08:36:45.302863+0000","last_undegraded":"2024-02-10T08:36:45.302863+0000","last_fullsized":"2024-02-10T08:36:45.302863+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.293810+0000","last_change":"2024-02-10T08:36:42.178297+0000","last_active":"2024-02-10T08:36:45.293810+0000","last_peered":"2024-02-10T08:36:45.293810+0000","last_clean":"2024-02-10T08:36:45.293810+0000","last_became_active":"2024-02-10T08:36:42.175804+0000","last_became_peered":"2024-02-10T08:36:42.175804+0000","last_unstale":"2024-02-10T08:36:45.293810+0000","last_undegraded":"2024-02-10T08:36:45.293810+0000","last_fullsized":"2024-02-10T08:36:45.293810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287325+0000","last_change":"2024-02-10T08:36:41.133800+0000","last_active":"2024-02-10T08:36:45.287325+0000","last_peered":"2024-02-10T08:36:45.287325+0000","last_clean":"2024-02-10T08:36:45.287325+0000","last_became_active":"2024-02-10T08:36:41.133511+0000","last_became_peered":"2024-02-10T08:36:41.133511+0000","last_unstale":"2024-02-10T08:36:45.287325+0000","last_undegraded":"2024-02-10T08:36:45.287325+0000","last_fullsized":"2024-02-10T08:36:45.287325+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.885998+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295635+0000","last_change":"2024-02-10T08:36:44.322512+0000","last_active":"2024-02-10T08:36:45.295635+0000","last_peered":"2024-02-10T08:36:45.295635+0000","last_clean":"2024-02-10T08:36:45.295635+0000","last_became_active":"2024-02-10T08:36:44.322183+0000","last_became_peered":"2024-02-10T08:36:44.322183+0000","last_unstale":"2024-02-10T08:36:45.295635+0000","last_undegraded":"2024-02-10T08:36:45.295635+0000","last_fullsized":"2024-02-10T08:36:45.295635+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307875+0000","last_change":"2024-02-10T08:36:42.162033+0000","last_active":"2024-02-10T08:36:45.307875+0000","last_peered":"2024-02-10T08:36:45.307875+0000","last_clean":"2024-02-10T08:36:45.307875+0000","last_became_active":"2024-02-10T08:36:42.161700+0000","last_became_peered":"2024-02-10T08:36:42.161700+0000","last_unstale":"2024-02-10T08:36:45.307875+0000","last_undegraded":"2024-02-10T08:36:45.307875+0000","last_fullsized":"2024-02-10T08:36:45.307875+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306363+0000","last_change":"2024-02-10T08:36:40.098133+0000","last_active":"2024-02-10T08:36:45.306363+0000","last_peered":"2024-02-10T08:36:45.306363+0000","last_clean":"2024-02-10T08:36:45.306363+0000","last_became_active":"2024-02-10T08:36:40.097500+0000","last_became_peered":"2024-02-10T08:36:40.097500+0000","last_unstale":"2024-02-10T08:36:45.306363+0000","last_undegraded":"2024-02-10T08:36:45.306363+0000","last_fullsized":"2024-02-10T08:36:45.306363+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"62.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.888305+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299419+0000","last_change":"2024-02-10T08:36:44.317273+0000","last_active":"2024-02-10T08:36:45.299419+0000","last_peered":"2024-02-10T08:36:45.299419+0000","last_clean":"2024-02-10T08:36:45.299419+0000","last_became_active":"2024-02-10T08:36:44.317005+0000","last_became_peered":"2024-02-10T08:36:44.317005+0000","last_unstale":"2024-02-10T08:36:45.299419+0000","last_undegraded":"2024-02-10T08:36:45.299419+0000","last_fullsized":"2024-02-10T08:36:45.299419+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.278974+0000","last_change":"2024-02-10T08:36:42.177238+0000","last_active":"2024-02-10T08:36:45.278974+0000","last_peered":"2024-02-10T08:36:45.278974+0000","last_clean":"2024-02-10T08:36:45.278974+0000","last_became_active":"2024-02-10T08:36:42.176913+0000","last_became_peered":"2024-02-10T08:36:42.176913+0000","last_unstale":"2024-02-10T08:36:45.278974+0000","last_undegraded":"2024-02-10T08:36:45.278974+0000","last_fullsized":"2024-02-10T08:36:45.278974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.283871+0000","last_change":"2024-02-10T08:36:40.096203+0000","last_active":"2024-02-10T08:36:45.283871+0000","last_peered":"2024-02-10T08:36:45.283871+0000","last_clean":"2024-02-10T08:36:45.283871+0000","last_became_active":"2024-02-10T08:36:40.095945+0000","last_became_peered":"2024-02-10T08:36:40.095945+0000","last_unstale":"2024-02-10T08:36:45.283871+0000","last_undegraded":"2024-02-10T08:36:45.283871+0000","last_fullsized":"2024-02-10T08:36:45.283871+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"62.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.913703+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.6","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316239+0000","last_change":"2024-02-10T08:36:44.318241+0000","last_active":"2024-02-10T08:36:45.316239+0000","last_peered":"2024-02-10T08:36:45.316239+0000","last_clean":"2024-02-10T08:36:45.316239+0000","last_became_active":"2024-02-10T08:36:44.317945+0000","last_became_peered":"2024-02-10T08:36:44.317945+0000","last_unstale":"2024-02-10T08:36:45.316239+0000","last_undegraded":"2024-02-10T08:36:45.316239+0000","last_fullsized":"2024-02-10T08:36:45.316239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301103+0000","last_change":"2024-02-10T08:36:42.312954+0000","last_active":"2024-02-10T08:36:45.301103+0000","last_peered":"2024-02-10T08:36:45.301103+0000","last_clean":"2024-02-10T08:36:45.301103+0000","last_became_active":"2024-02-10T08:36:42.312553+0000","last_became_peered":"2024-02-10T08:36:42.312553+0000","last_unstale":"2024-02-10T08:36:45.301103+0000","last_undegraded":"2024-02-10T08:36:45.301103+0000","last_fullsized":"2024-02-10T08:36:45.301103+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305225+0000","last_change":"2024-02-10T08:36:41.139503+0000","last_active":"2024-02-10T08:36:45.305225+0000","last_peered":"2024-02-10T08:36:45.305225+0000","last_clean":"2024-02-10T08:36:45.305225+0000","last_became_active":"2024-02-10T08:36:41.139225+0000","last_became_peered":"2024-02-10T08:36:41.139225+0000","last_unstale":"2024-02-10T08:36:45.305225+0000","last_undegraded":"2024-02-10T08:36:45.305225+0000","last_fullsized":"2024-02-10T08:36:45.305225+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.885453+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306239+0000","last_change":"2024-02-10T08:36:44.334344+0000","last_active":"2024-02-10T08:36:45.306239+0000","last_peered":"2024-02-10T08:36:45.306239+0000","last_clean":"2024-02-10T08:36:45.306239+0000","last_became_active":"2024-02-10T08:36:44.334090+0000","last_became_peered":"2024-02-10T08:36:44.334090+0000","last_unstale":"2024-02-10T08:36:45.306239+0000","last_undegraded":"2024-02-10T08:36:45.306239+0000","last_fullsized":"2024-02-10T08:36:45.306239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316136+0000","last_change":"2024-02-10T08:36:42.161942+0000","last_active":"2024-02-10T08:36:45.316136+0000","last_peered":"2024-02-10T08:36:45.316136+0000","last_clean":"2024-02-10T08:36:45.316136+0000","last_became_active":"2024-02-10T08:36:42.161535+0000","last_became_peered":"2024-02-10T08:36:42.161535+0000","last_unstale":"2024-02-10T08:36:45.316136+0000","last_undegraded":"2024-02-10T08:36:45.316136+0000","last_fullsized":"2024-02-10T08:36:45.316136+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.326412+0000","last_change":"2024-02-10T08:36:40.074866+0000","last_active":"2024-02-10T08:36:45.326412+0000","last_peered":"2024-02-10T08:36:45.326412+0000","last_clean":"2024-02-10T08:36:45.326412+0000","last_became_active":"2024-02-10T08:36:40.074369+0000","last_became_peered":"2024-02-10T08:36:40.074369+0000","last_unstale":"2024-02-10T08:36:45.326412+0000","last_undegraded":"2024-02-10T08:36:45.326412+0000","last_fullsized":"2024-02-10T08:36:45.326412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"62.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.899891+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.282215+0000","last_change":"2024-02-10T08:36:44.308893+0000","last_active":"2024-02-10T08:36:45.282215+0000","last_peered":"2024-02-10T08:36:45.282215+0000","last_clean":"2024-02-10T08:36:45.282215+0000","last_became_active":"2024-02-10T08:36:44.308545+0000","last_became_peered":"2024-02-10T08:36:44.308545+0000","last_unstale":"2024-02-10T08:36:45.282215+0000","last_undegraded":"2024-02-10T08:36:45.282215+0000","last_fullsized":"2024-02-10T08:36:45.282215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299596+0000","last_change":"2024-02-10T08:36:42.167047+0000","last_active":"2024-02-10T08:36:45.299596+0000","last_peered":"2024-02-10T08:36:45.299596+0000","last_clean":"2024-02-10T08:36:45.299596+0000","last_became_active":"2024-02-10T08:36:42.165868+0000","last_became_peered":"2024-02-10T08:36:42.165868+0000","last_unstale":"2024-02-10T08:36:45.299596+0000","last_undegraded":"2024-02-10T08:36:45.299596+0000","last_fullsized":"2024-02-10T08:36:45.299596+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305884+0000","last_change":"2024-02-10T08:36:40.087854+0000","last_active":"2024-02-10T08:36:45.305884+0000","last_peered":"2024-02-10T08:36:45.305884+0000","last_clean":"2024-02-10T08:36:45.305884+0000","last_became_active":"2024-02-10T08:36:40.087305+0000","last_became_peered":"2024-02-10T08:36:40.087305+0000","last_unstale":"2024-02-10T08:36:45.305884+0000","last_undegraded":"2024-02-10T08:36:45.305884+0000","last_fullsized":"2024-02-10T08:36:45.305884+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"62.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.903191+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303622+0000","last_change":"2024-02-10T08:36:44.322281+0000","last_active":"2024-02-10T08:36:45.303622+0000","last_peered":"2024-02-10T08:36:45.303622+0000","last_clean":"2024-02-10T08:36:45.303622+0000","last_became_active":"2024-02-10T08:36:44.318843+0000","last_became_peered":"2024-02-10T08:36:44.318843+0000","last_unstale":"2024-02-10T08:36:45.303622+0000","last_undegraded":"2024-02-10T08:36:45.303622+0000","last_fullsized":"2024-02-10T08:36:45.303622+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.282911+0000","last_change":"2024-02-10T08:36:42.339626+0000","last_active":"2024-02-10T08:36:45.282911+0000","last_peered":"2024-02-10T08:36:45.282911+0000","last_clean":"2024-02-10T08:36:45.282911+0000","last_became_active":"2024-02-10T08:36:42.339323+0000","last_became_peered":"2024-02-10T08:36:42.339323+0000","last_unstale":"2024-02-10T08:36:45.282911+0000","last_undegraded":"2024-02-10T08:36:45.282911+0000","last_fullsized":"2024-02-10T08:36:45.282911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327658+0000","last_change":"2024-02-10T08:36:40.093412+0000","last_active":"2024-02-10T08:36:45.327658+0000","last_peered":"2024-02-10T08:36:45.327658+0000","last_clean":"2024-02-10T08:36:45.327658+0000","last_became_active":"2024-02-10T08:36:40.093128+0000","last_became_peered":"2024-02-10T08:36:40.093128+0000","last_unstale":"2024-02-10T08:36:45.327658+0000","last_undegraded":"2024-02-10T08:36:45.327658+0000","last_fullsized":"2024-02-10T08:36:45.327658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"62.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.891476+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.293923+0000","last_change":"2024-02-10T08:36:44.324200+0000","last_active":"2024-02-10T08:36:45.293923+0000","last_peered":"2024-02-10T08:36:45.293923+0000","last_clean":"2024-02-10T08:36:45.293923+0000","last_became_active":"2024-02-10T08:36:44.323896+0000","last_became_peered":"2024-02-10T08:36:44.323896+0000","last_unstale":"2024-02-10T08:36:45.293923+0000","last_undegraded":"2024-02-10T08:36:45.293923+0000","last_fullsized":"2024-02-10T08:36:45.293923+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314683+0000","last_change":"2024-02-10T08:36:42.336482+0000","last_active":"2024-02-10T08:36:45.314683+0000","last_peered":"2024-02-10T08:36:45.314683+0000","last_clean":"2024-02-10T08:36:45.314683+0000","last_became_active":"2024-02-10T08:36:42.335900+0000","last_became_peered":"2024-02-10T08:36:42.335900+0000","last_unstale":"2024-02-10T08:36:45.314683+0000","last_undegraded":"2024-02-10T08:36:45.314683+0000","last_fullsized":"2024-02-10T08:36:45.314683+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.312905+0000","last_change":"2024-02-10T08:36:41.106893+0000","last_active":"2024-02-10T08:36:45.312905+0000","last_peered":"2024-02-10T08:36:45.312905+0000","last_clean":"2024-02-10T08:36:45.312905+0000","last_became_active":"2024-02-10T08:36:41.106576+0000","last_became_peered":"2024-02-10T08:36:41.106576+0000","last_unstale":"2024-02-10T08:36:45.312905+0000","last_undegraded":"2024-02-10T08:36:45.312905+0000","last_fullsized":"2024-02-10T08:36:45.312905+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.892131+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314473+0000","last_change":"2024-02-10T08:36:44.321363+0000","last_active":"2024-02-10T08:36:45.314473+0000","last_peered":"2024-02-10T08:36:45.314473+0000","last_clean":"2024-02-10T08:36:45.314473+0000","last_became_active":"2024-02-10T08:36:44.321087+0000","last_became_peered":"2024-02-10T08:36:44.321087+0000","last_unstale":"2024-02-10T08:36:45.314473+0000","last_undegraded":"2024-02-10T08:36:45.314473+0000","last_fullsized":"2024-02-10T08:36:45.314473+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280151+0000","last_change":"2024-02-10T08:36:42.307672+0000","last_active":"2024-02-10T08:36:45.280151+0000","last_peered":"2024-02-10T08:36:45.280151+0000","last_clean":"2024-02-10T08:36:45.280151+0000","last_became_active":"2024-02-10T08:36:42.307419+0000","last_became_peered":"2024-02-10T08:36:42.307419+0000","last_unstale":"2024-02-10T08:36:45.280151+0000","last_undegraded":"2024-02-10T08:36:45.280151+0000","last_fullsized":"2024-02-10T08:36:45.280151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287406+0000","last_change":"2024-02-10T08:36:41.137048+0000","last_active":"2024-02-10T08:36:45.287406+0000","last_peered":"2024-02-10T08:36:45.287406+0000","last_clean":"2024-02-10T08:36:45.287406+0000","last_became_active":"2024-02-10T08:36:41.136744+0000","last_became_peered":"2024-02-10T08:36:41.136744+0000","last_unstale":"2024-02-10T08:36:45.287406+0000","last_undegraded":"2024-02-10T08:36:45.287406+0000","last_fullsized":"2024-02-10T08:36:45.287406+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"62.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.902902+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.325868+0000","last_change":"2024-02-10T08:36:44.308392+0000","last_active":"2024-02-10T08:36:45.325868+0000","last_peered":"2024-02-10T08:36:45.325868+0000","last_clean":"2024-02-10T08:36:45.325868+0000","last_became_active":"2024-02-10T08:36:44.308092+0000","last_became_peered":"2024-02-10T08:36:44.308092+0000","last_unstale":"2024-02-10T08:36:45.325868+0000","last_undegraded":"2024-02-10T08:36:45.325868+0000","last_fullsized":"2024-02-10T08:36:45.325868+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302945+0000","last_change":"2024-02-10T08:36:42.182946+0000","last_active":"2024-02-10T08:36:45.302945+0000","last_peered":"2024-02-10T08:36:45.302945+0000","last_clean":"2024-02-10T08:36:45.302945+0000","last_became_active":"2024-02-10T08:36:42.182675+0000","last_became_peered":"2024-02-10T08:36:42.182675+0000","last_unstale":"2024-02-10T08:36:45.302945+0000","last_undegraded":"2024-02-10T08:36:45.302945+0000","last_fullsized":"2024-02-10T08:36:45.302945+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279705+0000","last_change":"2024-02-10T08:36:41.111645+0000","last_active":"2024-02-10T08:36:45.279705+0000","last_peered":"2024-02-10T08:36:45.279705+0000","last_clean":"2024-02-10T08:36:45.279705+0000","last_became_active":"2024-02-10T08:36:41.111262+0000","last_became_peered":"2024-02-10T08:36:41.111262+0000","last_unstale":"2024-02-10T08:36:45.279705+0000","last_undegraded":"2024-02-10T08:36:45.279705+0000","last_fullsized":"2024-02-10T08:36:45.279705+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"62.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.894639+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.331509+0000","last_change":"2024-02-10T08:36:44.325286+0000","last_active":"2024-02-10T08:36:45.331509+0000","last_peered":"2024-02-10T08:36:45.331509+0000","last_clean":"2024-02-10T08:36:45.331509+0000","last_became_active":"2024-02-10T08:36:44.324900+0000","last_became_peered":"2024-02-10T08:36:44.324900+0000","last_unstale":"2024-02-10T08:36:45.331509+0000","last_undegraded":"2024-02-10T08:36:45.331509+0000","last_fullsized":"2024-02-10T08:36:45.331509+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287032+0000","last_change":"2024-02-10T08:36:42.309191+0000","last_active":"2024-02-10T08:36:45.287032+0000","last_peered":"2024-02-10T08:36:45.287032+0000","last_clean":"2024-02-10T08:36:45.287032+0000","last_became_active":"2024-02-10T08:36:42.308937+0000","last_became_peered":"2024-02-10T08:36:42.308937+0000","last_unstale":"2024-02-10T08:36:45.287032+0000","last_undegraded":"2024-02-10T08:36:45.287032+0000","last_fullsized":"2024-02-10T08:36:45.287032+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.283961+0000","last_change":"2024-02-10T08:36:40.079682+0000","last_active":"2024-02-10T08:36:45.283961+0000","last_peered":"2024-02-10T08:36:45.283961+0000","last_clean":"2024-02-10T08:36:45.283961+0000","last_became_active":"2024-02-10T08:36:40.078950+0000","last_became_peered":"2024-02-10T08:36:40.078950+0000","last_unstale":"2024-02-10T08:36:45.283961+0000","last_undegraded":"2024-02-10T08:36:45.283961+0000","last_fullsized":"2024-02-10T08:36:45.283961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"62.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"59.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:50.313439+0000","last_change":"2024-02-10T08:36:50.313439+0000","last_active":"2024-02-10T08:36:50.313439+0000","last_peered":"2024-02-10T08:36:50.313439+0000","last_clean":"2024-02-10T08:36:50.313439+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:50.313439+0000","last_undegraded":"2024-02-10T08:36:50.313439+0000","last_fullsized":"2024-02-10T08:36:50.313439+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"57.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:36:48.884899+0000","last_change":"2024-02-10T08:36:48.884899+0000","last_active":"2024-02-10T08:36:48.884899+0000","last_peered":"2024-02-10T08:36:48.884899+0000","last_clean":"2024-02-10T08:36:48.884899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:48.884899+0000","last_undegraded":"2024-02-10T08:36:48.884899+0000","last_fullsized":"2024-02-10T08:36:48.884899+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:48.884899+0000","last_clean_scrub_stamp":"2024-02-10T08:36:48.884899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:36:45.877981+0000","last_change":"2024-02-10T08:36:45.896652+0000","last_active":"2024-02-10T08:36:45.877981+0000","last_peered":"2024-02-10T08:36:45.877981+0000","last_clean":"2024-02-10T08:36:45.877981+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:36:45.877981+0000","last_undegraded":"2024-02-10T08:36:45.877981+0000","last_fullsized":"2024-02-10T08:36:45.877981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.877981+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.877981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310476+0000","last_change":"2024-02-10T08:36:44.310217+0000","last_active":"2024-02-10T08:36:45.310476+0000","last_peered":"2024-02-10T08:36:45.310476+0000","last_clean":"2024-02-10T08:36:45.310476+0000","last_became_active":"2024-02-10T08:36:44.309950+0000","last_became_peered":"2024-02-10T08:36:44.309950+0000","last_unstale":"2024-02-10T08:36:45.310476+0000","last_undegraded":"2024-02-10T08:36:45.310476+0000","last_fullsized":"2024-02-10T08:36:45.310476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.254298+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.254298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306105+0000","last_change":"2024-02-10T08:36:42.338247+0000","last_active":"2024-02-10T08:36:45.306105+0000","last_peered":"2024-02-10T08:36:45.306105+0000","last_clean":"2024-02-10T08:36:45.306105+0000","last_became_active":"2024-02-10T08:36:42.327795+0000","last_became_peered":"2024-02-10T08:36:42.327795+0000","last_unstale":"2024-02-10T08:36:45.306105+0000","last_undegraded":"2024-02-10T08:36:45.306105+0000","last_fullsized":"2024-02-10T08:36:45.306105+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:41.031390+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.031390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306028+0000","last_change":"2024-02-10T08:36:41.123772+0000","last_active":"2024-02-10T08:36:45.306028+0000","last_peered":"2024-02-10T08:36:45.306028+0000","last_clean":"2024-02-10T08:36:45.306028+0000","last_became_active":"2024-02-10T08:36:41.123043+0000","last_became_peered":"2024-02-10T08:36:41.123043+0000","last_unstale":"2024-02-10T08:36:45.306028+0000","last_undegraded":"2024-02-10T08:36:45.306028+0000","last_fullsized":"2024-02-10T08:36:45.306028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:38.986526+0000","last_clean_scrub_stamp":"2024-02-10T08:36:38.986526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316856+0000","last_change":"2024-02-10T08:36:37.149757+0000","last_active":"2024-02-10T08:36:45.316856+0000","last_peered":"2024-02-10T08:36:45.316856+0000","last_clean":"2024-02-10T08:36:45.316856+0000","last_became_active":"2024-02-10T08:36:37.126414+0000","last_became_peered":"2024-02-10T08:36:37.126414+0000","last_unstale":"2024-02-10T08:36:45.316856+0000","last_undegraded":"2024-02-10T08:36:45.316856+0000","last_fullsized":"2024-02-10T08:36:45.316856+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":24,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314561+0000","last_change":"2024-02-10T08:36:40.742684+0000","last_active":"2024-02-10T08:36:45.314561+0000","last_peered":"2024-02-10T08:36:45.314561+0000","last_clean":"2024-02-10T08:36:45.314561+0000","last_became_active":"2024-02-10T08:36:37.030059+0000","last_became_peered":"2024-02-10T08:36:37.030059+0000","last_unstale":"2024-02-10T08:36:45.314561+0000","last_undegraded":"2024-02-10T08:36:45.314561+0000","last_fullsized":"2024-02-10T08:36:45.314561+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:40.742602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:40.742602+0000","last_clean_scrub_stamp":"2024-02-10T08:36:40.742602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"63'8","reported_seq":120,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.296907+0000","last_change":"2024-02-10T08:36:16.169727+0000","last_active":"2024-02-10T08:36:45.296907+0000","last_peered":"2024-02-10T08:36:45.296907+0000","last_clean":"2024-02-10T08:36:45.296907+0000","last_became_active":"2024-02-10T08:33:22.122112+0000","last_became_peered":"2024-02-10T08:33:22.122112+0000","last_unstale":"2024-02-10T08:36:45.296907+0000","last_undegraded":"2024-02-10T08:36:45.296907+0000","last_fullsized":"2024-02-10T08:36:45.296907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:16.169618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.169618+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288264+0000","last_change":"2024-02-10T08:35:05.510749+0000","last_active":"2024-02-10T08:36:45.288264+0000","last_peered":"2024-02-10T08:36:45.288264+0000","last_clean":"2024-02-10T08:36:45.288264+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:36:45.288264+0000","last_undegraded":"2024-02-10T08:36:45.288264+0000","last_fullsized":"2024-02-10T08:36:45.288264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:35:05.510644+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":70,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.311831+0000","last_change":"2024-02-10T08:35:03.720397+0000","last_active":"2024-02-10T08:36:45.311831+0000","last_peered":"2024-02-10T08:36:45.311831+0000","last_clean":"2024-02-10T08:36:45.311831+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:36:45.311831+0000","last_undegraded":"2024-02-10T08:36:45.311831+0000","last_fullsized":"2024-02-10T08:36:45.311831+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:03.720316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:03.720316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301801+0000","last_change":"2024-02-10T08:36:21.704857+0000","last_active":"2024-02-10T08:36:45.301801+0000","last_peered":"2024-02-10T08:36:45.301801+0000","last_clean":"2024-02-10T08:36:45.301801+0000","last_became_active":"2024-02-10T08:33:24.126993+0000","last_became_peered":"2024-02-10T08:33:24.126993+0000","last_unstale":"2024-02-10T08:36:45.301801+0000","last_undegraded":"2024-02-10T08:36:45.301801+0000","last_fullsized":"2024-02-10T08:36:45.301801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.704774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.704774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289207+0000","last_change":"2024-02-10T08:36:37.051266+0000","last_active":"2024-02-10T08:36:45.289207+0000","last_peered":"2024-02-10T08:36:45.289207+0000","last_clean":"2024-02-10T08:36:45.289207+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:36:45.289207+0000","last_undegraded":"2024-02-10T08:36:45.289207+0000","last_fullsized":"2024-02-10T08:36:45.289207+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.888279+0000","last_change":"2024-02-10T08:36:45.281404+0000","last_active":"2024-02-10T08:36:45.888279+0000","last_peered":"2024-02-10T08:36:45.888279+0000","last_clean":"2024-02-10T08:36:45.888279+0000","last_became_active":"2024-02-10T08:36:37.180809+0000","last_became_peered":"2024-02-10T08:36:37.180809+0000","last_unstale":"2024-02-10T08:36:45.888279+0000","last_undegraded":"2024-02-10T08:36:45.888279+0000","last_fullsized":"2024-02-10T08:36:45.888279+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.16","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288779+0000","last_change":"2024-02-10T08:36:37.165629+0000","last_active":"2024-02-10T08:36:45.288779+0000","last_peered":"2024-02-10T08:36:45.288779+0000","last_clean":"2024-02-10T08:36:45.288779+0000","last_became_active":"2024-02-10T08:36:37.164613+0000","last_became_peered":"2024-02-10T08:36:37.164613+0000","last_unstale":"2024-02-10T08:36:45.288779+0000","last_undegraded":"2024-02-10T08:36:45.288779+0000","last_fullsized":"2024-02-10T08:36:45.288779+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.278975+0000","last_change":"2024-02-10T08:36:37.097050+0000","last_active":"2024-02-10T08:36:45.278975+0000","last_peered":"2024-02-10T08:36:45.278975+0000","last_clean":"2024-02-10T08:36:45.278975+0000","last_became_active":"2024-02-10T08:36:37.096176+0000","last_became_peered":"2024-02-10T08:36:37.096176+0000","last_unstale":"2024-02-10T08:36:45.278975+0000","last_undegraded":"2024-02-10T08:36:45.278975+0000","last_fullsized":"2024-02-10T08:36:45.278975+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309090+0000","last_change":"2024-02-10T08:36:14.826412+0000","last_active":"2024-02-10T08:36:45.309090+0000","last_peered":"2024-02-10T08:36:45.309090+0000","last_clean":"2024-02-10T08:36:45.309090+0000","last_became_active":"2024-02-10T08:34:41.694073+0000","last_became_peered":"2024-02-10T08:34:41.694073+0000","last_unstale":"2024-02-10T08:36:45.309090+0000","last_undegraded":"2024-02-10T08:36:45.309090+0000","last_fullsized":"2024-02-10T08:36:45.309090+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:14.826321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.826321+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.286978+0000","last_change":"2024-02-10T08:35:30.241737+0000","last_active":"2024-02-10T08:36:45.286978+0000","last_peered":"2024-02-10T08:36:45.286978+0000","last_clean":"2024-02-10T08:36:45.286978+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:36:45.286978+0000","last_undegraded":"2024-02-10T08:36:45.286978+0000","last_fullsized":"2024-02-10T08:36:45.286978+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.241661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.241661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.297334+0000","last_change":"2024-02-10T08:36:17.568277+0000","last_active":"2024-02-10T08:36:45.297334+0000","last_peered":"2024-02-10T08:36:45.297334+0000","last_clean":"2024-02-10T08:36:45.297334+0000","last_became_active":"2024-02-10T08:33:22.115594+0000","last_became_peered":"2024-02-10T08:33:22.115594+0000","last_unstale":"2024-02-10T08:36:45.297334+0000","last_undegraded":"2024-02-10T08:36:45.297334+0000","last_fullsized":"2024-02-10T08:36:45.297334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.568165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.568165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307556+0000","last_change":"2024-02-10T08:35:34.652671+0000","last_active":"2024-02-10T08:36:45.307556+0000","last_peered":"2024-02-10T08:36:45.307556+0000","last_clean":"2024-02-10T08:36:45.307556+0000","last_became_active":"2024-02-10T08:33:24.121181+0000","last_became_peered":"2024-02-10T08:33:24.121181+0000","last_unstale":"2024-02-10T08:36:45.307556+0000","last_undegraded":"2024-02-10T08:36:45.307556+0000","last_fullsized":"2024-02-10T08:36:45.307556+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:34.652551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.652551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314848+0000","last_change":"2024-02-10T08:36:37.033242+0000","last_active":"2024-02-10T08:36:45.314848+0000","last_peered":"2024-02-10T08:36:45.314848+0000","last_clean":"2024-02-10T08:36:45.314848+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:36:45.314848+0000","last_undegraded":"2024-02-10T08:36:45.314848+0000","last_fullsized":"2024-02-10T08:36:45.314848+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.886638+0000","last_change":"2024-02-10T08:36:45.307874+0000","last_active":"2024-02-10T08:36:45.886638+0000","last_peered":"2024-02-10T08:36:45.886638+0000","last_clean":"2024-02-10T08:36:45.886638+0000","last_became_active":"2024-02-10T08:36:37.129993+0000","last_became_peered":"2024-02-10T08:36:37.129993+0000","last_unstale":"2024-02-10T08:36:45.886638+0000","last_undegraded":"2024-02-10T08:36:45.886638+0000","last_fullsized":"2024-02-10T08:36:45.886638+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.17","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.326754+0000","last_change":"2024-02-10T08:36:37.071055+0000","last_active":"2024-02-10T08:36:45.326754+0000","last_peered":"2024-02-10T08:36:45.326754+0000","last_clean":"2024-02-10T08:36:45.326754+0000","last_became_active":"2024-02-10T08:36:37.070584+0000","last_became_peered":"2024-02-10T08:36:37.070584+0000","last_unstale":"2024-02-10T08:36:45.326754+0000","last_undegraded":"2024-02-10T08:36:45.326754+0000","last_fullsized":"2024-02-10T08:36:45.326754+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"104'2","reported_seq":28,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.319860+0000","last_change":"2024-02-10T08:36:41.876934+0000","last_active":"2024-02-10T08:36:45.319860+0000","last_peered":"2024-02-10T08:36:45.319860+0000","last_clean":"2024-02-10T08:36:45.319860+0000","last_became_active":"2024-02-10T08:36:37.006148+0000","last_became_peered":"2024-02-10T08:36:37.006148+0000","last_unstale":"2024-02-10T08:36:45.319860+0000","last_undegraded":"2024-02-10T08:36:45.319860+0000","last_fullsized":"2024-02-10T08:36:45.319860+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'2","last_scrub_stamp":"2024-02-10T08:36:41.876864+0000","last_deep_scrub":"104'2","last_deep_scrub_stamp":"2024-02-10T08:36:41.876864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:41.876864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":104,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.329243+0000","last_change":"2024-02-10T08:36:23.647242+0000","last_active":"2024-02-10T08:36:45.329243+0000","last_peered":"2024-02-10T08:36:45.329243+0000","last_clean":"2024-02-10T08:36:45.329243+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:36:45.329243+0000","last_undegraded":"2024-02-10T08:36:45.329243+0000","last_fullsized":"2024-02-10T08:36:45.329243+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:23.647164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.647164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":70,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289621+0000","last_change":"2024-02-10T08:34:54.705770+0000","last_active":"2024-02-10T08:36:45.289621+0000","last_peered":"2024-02-10T08:36:45.289621+0000","last_clean":"2024-02-10T08:36:45.289621+0000","last_became_active":"2024-02-10T08:33:18.103536+0000","last_became_peered":"2024-02-10T08:33:18.103536+0000","last_unstale":"2024-02-10T08:36:45.289621+0000","last_undegraded":"2024-02-10T08:36:45.289621+0000","last_fullsized":"2024-02-10T08:36:45.289621+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:54.705701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:54.705701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":66,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291479+0000","last_change":"2024-02-10T08:34:55.668438+0000","last_active":"2024-02-10T08:36:45.291479+0000","last_peered":"2024-02-10T08:36:45.291479+0000","last_clean":"2024-02-10T08:36:45.291479+0000","last_became_active":"2024-02-10T08:33:22.117792+0000","last_became_peered":"2024-02-10T08:33:22.117792+0000","last_unstale":"2024-02-10T08:36:45.291479+0000","last_undegraded":"2024-02-10T08:36:45.291479+0000","last_fullsized":"2024-02-10T08:36:45.291479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:55.668347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:55.668347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299578+0000","last_change":"2024-02-10T08:35:13.030970+0000","last_active":"2024-02-10T08:36:45.299578+0000","last_peered":"2024-02-10T08:36:45.299578+0000","last_clean":"2024-02-10T08:36:45.299578+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:36:45.299578+0000","last_undegraded":"2024-02-10T08:36:45.299578+0000","last_fullsized":"2024-02-10T08:36:45.299578+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:13.030886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.030886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303786+0000","last_change":"2024-02-10T08:36:37.021497+0000","last_active":"2024-02-10T08:36:45.303786+0000","last_peered":"2024-02-10T08:36:45.303786+0000","last_clean":"2024-02-10T08:36:45.303786+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:36:45.303786+0000","last_undegraded":"2024-02-10T08:36:45.303786+0000","last_fullsized":"2024-02-10T08:36:45.303786+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.19","version":"111'1","reported_seq":32,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.901231+0000","last_change":"2024-02-10T08:36:45.283588+0000","last_active":"2024-02-10T08:36:45.901231+0000","last_peered":"2024-02-10T08:36:45.901231+0000","last_clean":"2024-02-10T08:36:45.901231+0000","last_became_active":"2024-02-10T08:36:37.102291+0000","last_became_peered":"2024-02-10T08:36:37.102291+0000","last_unstale":"2024-02-10T08:36:45.901231+0000","last_undegraded":"2024-02-10T08:36:45.901231+0000","last_fullsized":"2024-02-10T08:36:45.901231+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.10","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288354+0000","last_change":"2024-02-10T08:36:37.116021+0000","last_active":"2024-02-10T08:36:45.288354+0000","last_peered":"2024-02-10T08:36:45.288354+0000","last_clean":"2024-02-10T08:36:45.288354+0000","last_became_active":"2024-02-10T08:36:37.115730+0000","last_became_peered":"2024-02-10T08:36:37.115730+0000","last_unstale":"2024-02-10T08:36:45.288354+0000","last_undegraded":"2024-02-10T08:36:45.288354+0000","last_fullsized":"2024-02-10T08:36:45.288354+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285893+0000","last_change":"2024-02-10T08:35:30.802385+0000","last_active":"2024-02-10T08:36:45.285893+0000","last_peered":"2024-02-10T08:36:45.285893+0000","last_clean":"2024-02-10T08:36:45.285893+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:36:45.285893+0000","last_undegraded":"2024-02-10T08:36:45.285893+0000","last_fullsized":"2024-02-10T08:36:45.285893+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:35:30.802306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.802306+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.294916+0000","last_change":"2024-02-10T08:35:30.123918+0000","last_active":"2024-02-10T08:36:45.294916+0000","last_peered":"2024-02-10T08:36:45.294916+0000","last_clean":"2024-02-10T08:36:45.294916+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:36:45.294916+0000","last_undegraded":"2024-02-10T08:36:45.294916+0000","last_fullsized":"2024-02-10T08:36:45.294916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.123828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.123828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":84,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.950978+0000","last_change":"2024-02-10T08:36:45.950978+0000","last_active":"2024-02-10T08:36:45.950978+0000","last_peered":"2024-02-10T08:36:45.950978+0000","last_clean":"2024-02-10T08:36:45.950978+0000","last_became_active":"2024-02-10T08:36:45.950656+0000","last_became_peered":"2024-02-10T08:36:45.950656+0000","last_unstale":"2024-02-10T08:36:45.950978+0000","last_undegraded":"2024-02-10T08:36:45.950978+0000","last_fullsized":"2024-02-10T08:36:45.950978+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.226947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303423+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:36:45.303423+0000","last_peered":"2024-02-10T08:36:45.303423+0000","last_clean":"2024-02-10T08:36:45.303423+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:36:45.303423+0000","last_undegraded":"2024-02-10T08:36:45.303423+0000","last_fullsized":"2024-02-10T08:36:45.303423+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"103'1","reported_seq":25,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291266+0000","last_change":"2024-02-10T08:36:42.506026+0000","last_active":"2024-02-10T08:36:45.291266+0000","last_peered":"2024-02-10T08:36:45.291266+0000","last_clean":"2024-02-10T08:36:45.291266+0000","last_became_active":"2024-02-10T08:36:37.168715+0000","last_became_peered":"2024-02-10T08:36:37.168715+0000","last_unstale":"2024-02-10T08:36:45.291266+0000","last_undegraded":"2024-02-10T08:36:45.291266+0000","last_fullsized":"2024-02-10T08:36:45.291266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'1","last_scrub_stamp":"2024-02-10T08:36:42.505956+0000","last_deep_scrub":"103'1","last_deep_scrub_stamp":"2024-02-10T08:36:42.505956+0000","last_clean_scrub_stamp":"2024-02-10T08:36:42.505956+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279924+0000","last_change":"2024-02-10T08:36:37.075652+0000","last_active":"2024-02-10T08:36:45.279924+0000","last_peered":"2024-02-10T08:36:45.279924+0000","last_clean":"2024-02-10T08:36:45.279924+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:36:45.279924+0000","last_undegraded":"2024-02-10T08:36:45.279924+0000","last_fullsized":"2024-02-10T08:36:45.279924+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.904059+0000","last_change":"2024-02-10T08:36:45.334924+0000","last_active":"2024-02-10T08:36:45.904059+0000","last_peered":"2024-02-10T08:36:45.904059+0000","last_clean":"2024-02-10T08:36:45.904059+0000","last_became_active":"2024-02-10T08:36:37.170256+0000","last_became_peered":"2024-02-10T08:36:37.170256+0000","last_unstale":"2024-02-10T08:36:45.904059+0000","last_undegraded":"2024-02-10T08:36:45.904059+0000","last_fullsized":"2024-02-10T08:36:45.904059+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.11","version":"103'7","reported_seq":25,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306438+0000","last_change":"2024-02-10T08:36:37.024851+0000","last_active":"2024-02-10T08:36:45.306438+0000","last_peered":"2024-02-10T08:36:45.306438+0000","last_clean":"2024-02-10T08:36:45.306438+0000","last_became_active":"2024-02-10T08:36:36.991924+0000","last_became_peered":"2024-02-10T08:36:36.991924+0000","last_unstale":"2024-02-10T08:36:45.306438+0000","last_undegraded":"2024-02-10T08:36:45.306438+0000","last_fullsized":"2024-02-10T08:36:45.306438+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.286183+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:36:45.286183+0000","last_peered":"2024-02-10T08:36:45.286183+0000","last_clean":"2024-02-10T08:36:45.286183+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:36:45.286183+0000","last_undegraded":"2024-02-10T08:36:45.286183+0000","last_fullsized":"2024-02-10T08:36:45.286183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290791+0000","last_change":"2024-02-10T08:35:31.222395+0000","last_active":"2024-02-10T08:36:45.290791+0000","last_peered":"2024-02-10T08:36:45.290791+0000","last_clean":"2024-02-10T08:36:45.290791+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:36:45.290791+0000","last_undegraded":"2024-02-10T08:36:45.290791+0000","last_fullsized":"2024-02-10T08:36:45.290791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:31.222331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:31.222331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":135,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292550+0000","last_change":"2024-02-10T08:36:42.181308+0000","last_active":"2024-02-10T08:36:45.292550+0000","last_peered":"2024-02-10T08:36:45.292550+0000","last_clean":"2024-02-10T08:36:45.292550+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:36:45.292550+0000","last_undegraded":"2024-02-10T08:36:45.292550+0000","last_fullsized":"2024-02-10T08:36:45.292550+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:51.737390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:51.737390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289472+0000","last_change":"2024-02-10T08:36:13.778764+0000","last_active":"2024-02-10T08:36:45.289472+0000","last_peered":"2024-02-10T08:36:45.289472+0000","last_clean":"2024-02-10T08:36:45.289472+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:36:45.289472+0000","last_undegraded":"2024-02-10T08:36:45.289472+0000","last_fullsized":"2024-02-10T08:36:45.289472+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.778673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.778673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":21,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.294945+0000","last_change":"2024-02-10T08:36:42.156065+0000","last_active":"2024-02-10T08:36:45.294945+0000","last_peered":"2024-02-10T08:36:45.294945+0000","last_clean":"2024-02-10T08:36:45.294945+0000","last_became_active":"2024-02-10T08:36:37.150853+0000","last_became_peered":"2024-02-10T08:36:37.150853+0000","last_unstale":"2024-02-10T08:36:45.294945+0000","last_undegraded":"2024-02-10T08:36:45.294945+0000","last_fullsized":"2024-02-10T08:36:45.294945+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316294+0000","last_change":"2024-02-10T08:36:37.118827+0000","last_active":"2024-02-10T08:36:45.316294+0000","last_peered":"2024-02-10T08:36:45.316294+0000","last_clean":"2024-02-10T08:36:45.316294+0000","last_became_active":"2024-02-10T08:36:37.118328+0000","last_became_peered":"2024-02-10T08:36:37.118328+0000","last_unstale":"2024-02-10T08:36:45.316294+0000","last_undegraded":"2024-02-10T08:36:45.316294+0000","last_fullsized":"2024-02-10T08:36:45.316294+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.906187+0000","last_change":"2024-02-10T08:36:45.288324+0000","last_active":"2024-02-10T08:36:45.906187+0000","last_peered":"2024-02-10T08:36:45.906187+0000","last_clean":"2024-02-10T08:36:45.906187+0000","last_became_active":"2024-02-10T08:36:37.070097+0000","last_became_peered":"2024-02-10T08:36:37.070097+0000","last_unstale":"2024-02-10T08:36:45.906187+0000","last_undegraded":"2024-02-10T08:36:45.906187+0000","last_fullsized":"2024-02-10T08:36:45.906187+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.12","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291118+0000","last_change":"2024-02-10T08:36:37.179572+0000","last_active":"2024-02-10T08:36:45.291118+0000","last_peered":"2024-02-10T08:36:45.291118+0000","last_clean":"2024-02-10T08:36:45.291118+0000","last_became_active":"2024-02-10T08:36:37.178108+0000","last_became_peered":"2024-02-10T08:36:37.178108+0000","last_unstale":"2024-02-10T08:36:45.291118+0000","last_undegraded":"2024-02-10T08:36:45.291118+0000","last_fullsized":"2024-02-10T08:36:45.291118+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"105'1","reported_seq":24,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280083+0000","last_change":"2024-02-10T08:36:37.045219+0000","last_active":"2024-02-10T08:36:45.280083+0000","last_peered":"2024-02-10T08:36:45.280083+0000","last_clean":"2024-02-10T08:36:45.280083+0000","last_became_active":"2024-02-10T08:36:37.044969+0000","last_became_peered":"2024-02-10T08:36:37.044969+0000","last_unstale":"2024-02-10T08:36:45.280083+0000","last_undegraded":"2024-02-10T08:36:45.280083+0000","last_fullsized":"2024-02-10T08:36:45.280083+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":93,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279572+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:36:45.279572+0000","last_peered":"2024-02-10T08:36:45.279572+0000","last_clean":"2024-02-10T08:36:45.279572+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:36:45.279572+0000","last_undegraded":"2024-02-10T08:36:45.279572+0000","last_fullsized":"2024-02-10T08:36:45.279572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":81,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291415+0000","last_change":"2024-02-10T08:36:16.423854+0000","last_active":"2024-02-10T08:36:45.291415+0000","last_peered":"2024-02-10T08:36:45.291415+0000","last_clean":"2024-02-10T08:36:45.291415+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:36:45.291415+0000","last_undegraded":"2024-02-10T08:36:45.291415+0000","last_fullsized":"2024-02-10T08:36:45.291415+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:16.423786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.423786+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"105'10","reported_seq":134,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289445+0000","last_change":"2024-02-10T08:36:39.056431+0000","last_active":"2024-02-10T08:36:45.289445+0000","last_peered":"2024-02-10T08:36:45.289445+0000","last_clean":"2024-02-10T08:36:45.289445+0000","last_became_active":"2024-02-10T08:36:39.055080+0000","last_became_peered":"2024-02-10T08:36:39.055080+0000","last_unstale":"2024-02-10T08:36:45.289445+0000","last_undegraded":"2024-02-10T08:36:45.289445+0000","last_fullsized":"2024-02-10T08:36:45.289445+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:35:11.026909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.026909+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310931+0000","last_change":"2024-02-10T08:35:16.510089+0000","last_active":"2024-02-10T08:36:45.310931+0000","last_peered":"2024-02-10T08:36:45.310931+0000","last_clean":"2024-02-10T08:36:45.310931+0000","last_became_active":"2024-02-10T08:33:24.122585+0000","last_became_peered":"2024-02-10T08:33:24.122585+0000","last_unstale":"2024-02-10T08:36:45.310931+0000","last_undegraded":"2024-02-10T08:36:45.310931+0000","last_fullsized":"2024-02-10T08:36:45.310931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303343+0000","last_change":"2024-02-10T08:36:37.148745+0000","last_active":"2024-02-10T08:36:45.303343+0000","last_peered":"2024-02-10T08:36:45.303343+0000","last_clean":"2024-02-10T08:36:45.303343+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:36:45.303343+0000","last_undegraded":"2024-02-10T08:36:45.303343+0000","last_fullsized":"2024-02-10T08:36:45.303343+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.910991+0000","last_change":"2024-02-10T08:36:45.292249+0000","last_active":"2024-02-10T08:36:45.910991+0000","last_peered":"2024-02-10T08:36:45.910991+0000","last_clean":"2024-02-10T08:36:45.910991+0000","last_became_active":"2024-02-10T08:36:37.018966+0000","last_became_peered":"2024-02-10T08:36:37.018966+0000","last_unstale":"2024-02-10T08:36:45.910991+0000","last_undegraded":"2024-02-10T08:36:45.910991+0000","last_fullsized":"2024-02-10T08:36:45.910991+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.13","version":"103'9","reported_seq":27,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287523+0000","last_change":"2024-02-10T08:36:37.095477+0000","last_active":"2024-02-10T08:36:45.287523+0000","last_peered":"2024-02-10T08:36:45.287523+0000","last_clean":"2024-02-10T08:36:45.287523+0000","last_became_active":"2024-02-10T08:36:37.095163+0000","last_became_peered":"2024-02-10T08:36:37.095163+0000","last_unstale":"2024-02-10T08:36:45.287523+0000","last_undegraded":"2024-02-10T08:36:45.287523+0000","last_fullsized":"2024-02-10T08:36:45.287523+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289198+0000","last_change":"2024-02-10T08:36:11.732545+0000","last_active":"2024-02-10T08:36:45.289198+0000","last_peered":"2024-02-10T08:36:45.289198+0000","last_clean":"2024-02-10T08:36:45.289198+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:36:45.289198+0000","last_undegraded":"2024-02-10T08:36:45.289198+0000","last_fullsized":"2024-02-10T08:36:45.289198+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:36:11.732461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:11.732461+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299864+0000","last_change":"2024-02-10T08:36:20.142711+0000","last_active":"2024-02-10T08:36:45.299864+0000","last_peered":"2024-02-10T08:36:45.299864+0000","last_clean":"2024-02-10T08:36:45.299864+0000","last_became_active":"2024-02-10T08:33:18.108226+0000","last_became_peered":"2024-02-10T08:33:18.108226+0000","last_unstale":"2024-02-10T08:36:45.299864+0000","last_undegraded":"2024-02-10T08:36:45.299864+0000","last_fullsized":"2024-02-10T08:36:45.299864+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.142541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.142541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314475+0000","last_change":"2024-02-10T08:36:29.158683+0000","last_active":"2024-02-10T08:36:45.314475+0000","last_peered":"2024-02-10T08:36:45.314475+0000","last_clean":"2024-02-10T08:36:45.314475+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:36:45.314475+0000","last_undegraded":"2024-02-10T08:36:45.314475+0000","last_fullsized":"2024-02-10T08:36:45.314475+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":63,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307248+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:36:45.307248+0000","last_peered":"2024-02-10T08:36:45.307248+0000","last_clean":"2024-02-10T08:36:45.307248+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:36:45.307248+0000","last_undegraded":"2024-02-10T08:36:45.307248+0000","last_fullsized":"2024-02-10T08:36:45.307248+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"103'6","reported_seq":30,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280957+0000","last_change":"2024-02-10T08:36:39.721668+0000","last_active":"2024-02-10T08:36:45.280957+0000","last_peered":"2024-02-10T08:36:45.280957+0000","last_clean":"2024-02-10T08:36:45.280957+0000","last_became_active":"2024-02-10T08:36:37.094562+0000","last_became_peered":"2024-02-10T08:36:37.094562+0000","last_unstale":"2024-02-10T08:36:45.280957+0000","last_undegraded":"2024-02-10T08:36:45.280957+0000","last_fullsized":"2024-02-10T08:36:45.280957+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'6","last_scrub_stamp":"2024-02-10T08:36:39.721593+0000","last_deep_scrub":"103'6","last_deep_scrub_stamp":"2024-02-10T08:36:39.721593+0000","last_clean_scrub_stamp":"2024-02-10T08:36:39.721593+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.298710+0000","last_change":"2024-02-10T08:36:37.134622+0000","last_active":"2024-02-10T08:36:45.298710+0000","last_peered":"2024-02-10T08:36:45.298710+0000","last_clean":"2024-02-10T08:36:45.298710+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:36:45.298710+0000","last_undegraded":"2024-02-10T08:36:45.298710+0000","last_fullsized":"2024-02-10T08:36:45.298710+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.897832+0000","last_change":"2024-02-10T08:36:45.314432+0000","last_active":"2024-02-10T08:36:45.897832+0000","last_peered":"2024-02-10T08:36:45.897832+0000","last_clean":"2024-02-10T08:36:45.897832+0000","last_became_active":"2024-02-10T08:36:37.078479+0000","last_became_peered":"2024-02-10T08:36:37.078479+0000","last_unstale":"2024-02-10T08:36:45.897832+0000","last_undegraded":"2024-02-10T08:36:45.897832+0000","last_fullsized":"2024-02-10T08:36:45.897832+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1c","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.286311+0000","last_change":"2024-02-10T08:36:37.065813+0000","last_active":"2024-02-10T08:36:45.286311+0000","last_peered":"2024-02-10T08:36:45.286311+0000","last_clean":"2024-02-10T08:36:45.286311+0000","last_became_active":"2024-02-10T08:36:37.065010+0000","last_became_peered":"2024-02-10T08:36:37.065010+0000","last_unstale":"2024-02-10T08:36:45.286311+0000","last_undegraded":"2024-02-10T08:36:45.286311+0000","last_fullsized":"2024-02-10T08:36:45.286311+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"105'2","reported_seq":21,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309461+0000","last_change":"2024-02-10T08:36:37.033787+0000","last_active":"2024-02-10T08:36:45.309461+0000","last_peered":"2024-02-10T08:36:45.309461+0000","last_clean":"2024-02-10T08:36:45.309461+0000","last_became_active":"2024-02-10T08:36:37.033569+0000","last_became_peered":"2024-02-10T08:36:37.033569+0000","last_unstale":"2024-02-10T08:36:45.309461+0000","last_undegraded":"2024-02-10T08:36:45.309461+0000","last_fullsized":"2024-02-10T08:36:45.309461+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":122,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295193+0000","last_change":"2024-02-10T08:36:28.076465+0000","last_active":"2024-02-10T08:36:45.295193+0000","last_peered":"2024-02-10T08:36:45.295193+0000","last_clean":"2024-02-10T08:36:45.295193+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:36:45.295193+0000","last_undegraded":"2024-02-10T08:36:45.295193+0000","last_fullsized":"2024-02-10T08:36:45.295193+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:28.076401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.076401+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":94,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280475+0000","last_change":"2024-02-10T08:36:17.729319+0000","last_active":"2024-02-10T08:36:45.280475+0000","last_peered":"2024-02-10T08:36:45.280475+0000","last_clean":"2024-02-10T08:36:45.280475+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:36:45.280475+0000","last_undegraded":"2024-02-10T08:36:45.280475+0000","last_fullsized":"2024-02-10T08:36:45.280475+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:17.729249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.729249+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332856+0000","last_change":"2024-02-10T08:35:38.547170+0000","last_active":"2024-02-10T08:36:45.332856+0000","last_peered":"2024-02-10T08:36:45.332856+0000","last_clean":"2024-02-10T08:36:45.332856+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:36:45.332856+0000","last_undegraded":"2024-02-10T08:36:45.332856+0000","last_fullsized":"2024-02-10T08:36:45.332856+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:38.547094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.547094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295499+0000","last_change":"2024-02-10T08:35:11.025052+0000","last_active":"2024-02-10T08:36:45.295499+0000","last_peered":"2024-02-10T08:36:45.295499+0000","last_clean":"2024-02-10T08:36:45.295499+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:36:45.295499+0000","last_undegraded":"2024-02-10T08:36:45.295499+0000","last_fullsized":"2024-02-10T08:36:45.295499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:11.024934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.024934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327353+0000","last_change":"2024-02-10T08:36:37.149649+0000","last_active":"2024-02-10T08:36:45.327353+0000","last_peered":"2024-02-10T08:36:45.327353+0000","last_clean":"2024-02-10T08:36:45.327353+0000","last_became_active":"2024-02-10T08:36:37.149192+0000","last_became_peered":"2024-02-10T08:36:37.149192+0000","last_unstale":"2024-02-10T08:36:45.327353+0000","last_undegraded":"2024-02-10T08:36:45.327353+0000","last_fullsized":"2024-02-10T08:36:45.327353+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.902084+0000","last_change":"2024-02-10T08:36:45.305000+0000","last_active":"2024-02-10T08:36:45.902084+0000","last_peered":"2024-02-10T08:36:45.902084+0000","last_clean":"2024-02-10T08:36:45.902084+0000","last_became_active":"2024-02-10T08:36:37.123824+0000","last_became_peered":"2024-02-10T08:36:37.123824+0000","last_unstale":"2024-02-10T08:36:45.902084+0000","last_undegraded":"2024-02-10T08:36:45.902084+0000","last_fullsized":"2024-02-10T08:36:45.902084+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1d","version":"103'7","reported_seq":25,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309293+0000","last_change":"2024-02-10T08:36:37.149279+0000","last_active":"2024-02-10T08:36:45.309293+0000","last_peered":"2024-02-10T08:36:45.309293+0000","last_clean":"2024-02-10T08:36:45.309293+0000","last_became_active":"2024-02-10T08:36:37.148954+0000","last_became_peered":"2024-02-10T08:36:37.148954+0000","last_unstale":"2024-02-10T08:36:45.309293+0000","last_undegraded":"2024-02-10T08:36:45.309293+0000","last_fullsized":"2024-02-10T08:36:45.309293+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.317036+0000","last_change":"2024-02-10T08:36:37.121482+0000","last_active":"2024-02-10T08:36:45.317036+0000","last_peered":"2024-02-10T08:36:45.317036+0000","last_clean":"2024-02-10T08:36:45.317036+0000","last_became_active":"2024-02-10T08:36:37.113210+0000","last_became_peered":"2024-02-10T08:36:37.113210+0000","last_unstale":"2024-02-10T08:36:45.317036+0000","last_undegraded":"2024-02-10T08:36:45.317036+0000","last_fullsized":"2024-02-10T08:36:45.317036+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":66,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281303+0000","last_change":"2024-02-10T08:34:58.059739+0000","last_active":"2024-02-10T08:36:45.281303+0000","last_peered":"2024-02-10T08:36:45.281303+0000","last_clean":"2024-02-10T08:36:45.281303+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:36:45.281303+0000","last_undegraded":"2024-02-10T08:36:45.281303+0000","last_fullsized":"2024-02-10T08:36:45.281303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.059637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.059637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295094+0000","last_change":"2024-02-10T08:35:09.492086+0000","last_active":"2024-02-10T08:36:45.295094+0000","last_peered":"2024-02-10T08:36:45.295094+0000","last_clean":"2024-02-10T08:36:45.295094+0000","last_became_active":"2024-02-10T08:33:20.256436+0000","last_became_peered":"2024-02-10T08:33:20.256436+0000","last_unstale":"2024-02-10T08:36:45.295094+0000","last_undegraded":"2024-02-10T08:36:45.295094+0000","last_fullsized":"2024-02-10T08:36:45.295094+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":82,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289340+0000","last_change":"2024-02-10T08:36:32.450905+0000","last_active":"2024-02-10T08:36:45.289340+0000","last_peered":"2024-02-10T08:36:45.289340+0000","last_clean":"2024-02-10T08:36:45.289340+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:36:45.289340+0000","last_undegraded":"2024-02-10T08:36:45.289340+0000","last_fullsized":"2024-02-10T08:36:45.289340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:32.450807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.450807+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281565+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:36:45.281565+0000","last_peered":"2024-02-10T08:36:45.281565+0000","last_clean":"2024-02-10T08:36:45.281565+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:36:45.281565+0000","last_undegraded":"2024-02-10T08:36:45.281565+0000","last_fullsized":"2024-02-10T08:36:45.281565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316970+0000","last_change":"2024-02-10T08:36:37.121328+0000","last_active":"2024-02-10T08:36:45.316970+0000","last_peered":"2024-02-10T08:36:45.316970+0000","last_clean":"2024-02-10T08:36:45.316970+0000","last_became_active":"2024-02-10T08:36:37.112855+0000","last_became_peered":"2024-02-10T08:36:37.112855+0000","last_unstale":"2024-02-10T08:36:45.316970+0000","last_undegraded":"2024-02-10T08:36:45.316970+0000","last_fullsized":"2024-02-10T08:36:45.316970+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.913860+0000","last_change":"2024-02-10T08:36:45.316375+0000","last_active":"2024-02-10T08:36:45.913860+0000","last_peered":"2024-02-10T08:36:45.913860+0000","last_clean":"2024-02-10T08:36:45.913860+0000","last_became_active":"2024-02-10T08:36:37.133630+0000","last_became_peered":"2024-02-10T08:36:37.133630+0000","last_unstale":"2024-02-10T08:36:45.913860+0000","last_undegraded":"2024-02-10T08:36:45.913860+0000","last_fullsized":"2024-02-10T08:36:45.913860+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1e","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.313287+0000","last_change":"2024-02-10T08:36:37.085343+0000","last_active":"2024-02-10T08:36:45.313287+0000","last_peered":"2024-02-10T08:36:45.313287+0000","last_clean":"2024-02-10T08:36:45.313287+0000","last_became_active":"2024-02-10T08:36:37.078274+0000","last_became_peered":"2024-02-10T08:36:37.078274+0000","last_unstale":"2024-02-10T08:36:45.313287+0000","last_undegraded":"2024-02-10T08:36:45.313287+0000","last_fullsized":"2024-02-10T08:36:45.313287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.325814+0000","last_change":"2024-02-10T08:36:37.157729+0000","last_active":"2024-02-10T08:36:45.325814+0000","last_peered":"2024-02-10T08:36:45.325814+0000","last_clean":"2024-02-10T08:36:45.325814+0000","last_became_active":"2024-02-10T08:36:37.156958+0000","last_became_peered":"2024-02-10T08:36:37.156958+0000","last_unstale":"2024-02-10T08:36:45.325814+0000","last_undegraded":"2024-02-10T08:36:45.325814+0000","last_fullsized":"2024-02-10T08:36:45.325814+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.329328+0000","last_change":"2024-02-10T08:36:26.578369+0000","last_active":"2024-02-10T08:36:45.329328+0000","last_peered":"2024-02-10T08:36:45.329328+0000","last_clean":"2024-02-10T08:36:45.329328+0000","last_became_active":"2024-02-10T08:33:22.119355+0000","last_became_peered":"2024-02-10T08:33:22.119355+0000","last_unstale":"2024-02-10T08:36:45.329328+0000","last_undegraded":"2024-02-10T08:36:45.329328+0000","last_fullsized":"2024-02-10T08:36:45.329328+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.578284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.578284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":108,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.308938+0000","last_change":"2024-02-10T08:36:15.871132+0000","last_active":"2024-02-10T08:36:45.308938+0000","last_peered":"2024-02-10T08:36:45.308938+0000","last_clean":"2024-02-10T08:36:45.308938+0000","last_became_active":"2024-02-10T08:33:20.256279+0000","last_became_peered":"2024-02-10T08:33:20.256279+0000","last_unstale":"2024-02-10T08:36:45.308938+0000","last_undegraded":"2024-02-10T08:36:45.308938+0000","last_fullsized":"2024-02-10T08:36:45.308938+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:15.871060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.871060+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305797+0000","last_change":"2024-02-10T08:36:26.048208+0000","last_active":"2024-02-10T08:36:45.305797+0000","last_peered":"2024-02-10T08:36:45.305797+0000","last_clean":"2024-02-10T08:36:45.305797+0000","last_became_active":"2024-02-10T08:33:18.106040+0000","last_became_peered":"2024-02-10T08:33:18.106040+0000","last_unstale":"2024-02-10T08:36:45.305797+0000","last_undegraded":"2024-02-10T08:36:45.305797+0000","last_fullsized":"2024-02-10T08:36:45.305797+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.048119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.048119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291557+0000","last_change":"2024-02-10T08:36:20.500433+0000","last_active":"2024-02-10T08:36:45.291557+0000","last_peered":"2024-02-10T08:36:45.291557+0000","last_clean":"2024-02-10T08:36:45.291557+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:36:45.291557+0000","last_undegraded":"2024-02-10T08:36:45.291557+0000","last_fullsized":"2024-02-10T08:36:45.291557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.500313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302641+0000","last_change":"2024-02-10T08:36:37.101187+0000","last_active":"2024-02-10T08:36:45.302641+0000","last_peered":"2024-02-10T08:36:45.302641+0000","last_clean":"2024-02-10T08:36:45.302641+0000","last_became_active":"2024-02-10T08:36:37.100263+0000","last_became_peered":"2024-02-10T08:36:37.100263+0000","last_unstale":"2024-02-10T08:36:45.302641+0000","last_undegraded":"2024-02-10T08:36:45.302641+0000","last_fullsized":"2024-02-10T08:36:45.302641+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"105'1","reported_seq":32,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.886987+0000","last_change":"2024-02-10T08:36:45.310068+0000","last_active":"2024-02-10T08:36:45.886987+0000","last_peered":"2024-02-10T08:36:45.886987+0000","last_clean":"2024-02-10T08:36:45.886987+0000","last_became_active":"2024-02-10T08:36:37.048925+0000","last_became_peered":"2024-02-10T08:36:37.048925+0000","last_unstale":"2024-02-10T08:36:45.886987+0000","last_undegraded":"2024-02-10T08:36:45.886987+0000","last_fullsized":"2024-02-10T08:36:45.886987+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1b","version":"103'10","reported_seq":28,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.308416+0000","last_change":"2024-02-10T08:36:37.147200+0000","last_active":"2024-02-10T08:36:45.308416+0000","last_peered":"2024-02-10T08:36:45.308416+0000","last_clean":"2024-02-10T08:36:45.308416+0000","last_became_active":"2024-02-10T08:36:37.146933+0000","last_became_peered":"2024-02-10T08:36:37.146933+0000","last_unstale":"2024-02-10T08:36:45.308416+0000","last_undegraded":"2024-02-10T08:36:45.308416+0000","last_fullsized":"2024-02-10T08:36:45.308416+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"105'3","reported_seq":22,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287106+0000","last_change":"2024-02-10T08:36:37.035006+0000","last_active":"2024-02-10T08:36:45.287106+0000","last_peered":"2024-02-10T08:36:45.287106+0000","last_clean":"2024-02-10T08:36:45.287106+0000","last_became_active":"2024-02-10T08:36:37.034740+0000","last_became_peered":"2024-02-10T08:36:37.034740+0000","last_unstale":"2024-02-10T08:36:45.287106+0000","last_undegraded":"2024-02-10T08:36:45.287106+0000","last_fullsized":"2024-02-10T08:36:45.287106+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165839,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314570+0000","last_change":"2024-02-10T08:36:22.714387+0000","last_active":"2024-02-10T08:36:45.314570+0000","last_peered":"2024-02-10T08:36:45.314570+0000","last_clean":"2024-02-10T08:36:45.314570+0000","last_became_active":"2024-02-10T08:33:22.124212+0000","last_became_peered":"2024-02-10T08:33:22.124212+0000","last_unstale":"2024-02-10T08:36:45.314570+0000","last_undegraded":"2024-02-10T08:36:45.314570+0000","last_fullsized":"2024-02-10T08:36:45.314570+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.714301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.714301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":90,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.320488+0000","last_change":"2024-02-10T08:35:36.647820+0000","last_active":"2024-02-10T08:36:45.320488+0000","last_peered":"2024-02-10T08:36:45.320488+0000","last_clean":"2024-02-10T08:36:45.320488+0000","last_became_active":"2024-02-10T08:33:20.117217+0000","last_became_peered":"2024-02-10T08:33:20.117217+0000","last_unstale":"2024-02-10T08:36:45.320488+0000","last_undegraded":"2024-02-10T08:36:45.320488+0000","last_fullsized":"2024-02-10T08:36:45.320488+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:35:36.647719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.647719+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301163+0000","last_change":"2024-02-10T08:35:27.051918+0000","last_active":"2024-02-10T08:36:45.301163+0000","last_peered":"2024-02-10T08:36:45.301163+0000","last_clean":"2024-02-10T08:36:45.301163+0000","last_became_active":"2024-02-10T08:33:18.108310+0000","last_became_peered":"2024-02-10T08:33:18.108310+0000","last_unstale":"2024-02-10T08:36:45.301163+0000","last_undegraded":"2024-02-10T08:36:45.301163+0000","last_fullsized":"2024-02-10T08:36:45.301163+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:27.051837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:27.051837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306640+0000","last_change":"2024-02-10T08:35:41.795140+0000","last_active":"2024-02-10T08:36:45.306640+0000","last_peered":"2024-02-10T08:36:45.306640+0000","last_clean":"2024-02-10T08:36:45.306640+0000","last_became_active":"2024-02-10T08:33:18.105349+0000","last_became_peered":"2024-02-10T08:33:18.105349+0000","last_unstale":"2024-02-10T08:36:45.306640+0000","last_undegraded":"2024-02-10T08:36:45.306640+0000","last_fullsized":"2024-02-10T08:36:45.306640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":70,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290255+0000","last_change":"2024-02-10T08:35:38.321759+0000","last_active":"2024-02-10T08:36:45.290255+0000","last_peered":"2024-02-10T08:36:45.290255+0000","last_clean":"2024-02-10T08:36:45.290255+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:36:45.290255+0000","last_undegraded":"2024-02-10T08:36:45.290255+0000","last_fullsized":"2024-02-10T08:36:45.290255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:38.321694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.321694+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.328958+0000","last_change":"2024-02-10T08:36:37.034653+0000","last_active":"2024-02-10T08:36:45.328958+0000","last_peered":"2024-02-10T08:36:45.328958+0000","last_clean":"2024-02-10T08:36:45.328958+0000","last_became_active":"2024-02-10T08:36:37.034311+0000","last_became_peered":"2024-02-10T08:36:37.034311+0000","last_unstale":"2024-02-10T08:36:45.328958+0000","last_undegraded":"2024-02-10T08:36:45.328958+0000","last_fullsized":"2024-02-10T08:36:45.328958+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.902324+0000","last_change":"2024-02-10T08:36:45.289049+0000","last_active":"2024-02-10T08:36:45.902324+0000","last_peered":"2024-02-10T08:36:45.902324+0000","last_clean":"2024-02-10T08:36:45.902324+0000","last_became_active":"2024-02-10T08:36:37.046478+0000","last_became_peered":"2024-02-10T08:36:37.046478+0000","last_unstale":"2024-02-10T08:36:45.902324+0000","last_undegraded":"2024-02-10T08:36:45.902324+0000","last_fullsized":"2024-02-10T08:36:45.902324+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.18","version":"103'2","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.313434+0000","last_change":"2024-02-10T08:36:37.050041+0000","last_active":"2024-02-10T08:36:45.313434+0000","last_peered":"2024-02-10T08:36:45.313434+0000","last_clean":"2024-02-10T08:36:45.313434+0000","last_became_active":"2024-02-10T08:36:36.992643+0000","last_became_peered":"2024-02-10T08:36:36.992643+0000","last_unstale":"2024-02-10T08:36:45.313434+0000","last_undegraded":"2024-02-10T08:36:45.313434+0000","last_fullsized":"2024-02-10T08:36:45.313434+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":24,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.976340+0000","last_change":"2024-02-10T08:36:45.976340+0000","last_active":"2024-02-10T08:36:45.976340+0000","last_peered":"2024-02-10T08:36:45.976340+0000","last_clean":"2024-02-10T08:36:45.976340+0000","last_became_active":"2024-02-10T08:36:37.041246+0000","last_became_peered":"2024-02-10T08:36:37.041246+0000","last_unstale":"2024-02-10T08:36:45.976340+0000","last_undegraded":"2024-02-10T08:36:45.976340+0000","last_fullsized":"2024-02-10T08:36:45.976340+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:45.976254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:45.976254+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.976254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299798+0000","last_change":"2024-02-10T08:36:27.052479+0000","last_active":"2024-02-10T08:36:45.299798+0000","last_peered":"2024-02-10T08:36:45.299798+0000","last_clean":"2024-02-10T08:36:45.299798+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:36:45.299798+0000","last_undegraded":"2024-02-10T08:36:45.299798+0000","last_fullsized":"2024-02-10T08:36:45.299798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.052362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.052362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":95,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332176+0000","last_change":"2024-02-10T08:36:28.635087+0000","last_active":"2024-02-10T08:36:45.332176+0000","last_peered":"2024-02-10T08:36:45.332176+0000","last_clean":"2024-02-10T08:36:45.332176+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:36:45.332176+0000","last_undegraded":"2024-02-10T08:36:45.332176+0000","last_fullsized":"2024-02-10T08:36:45.332176+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:28.635015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635015+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299740+0000","last_change":"2024-02-10T08:36:25.053910+0000","last_active":"2024-02-10T08:36:45.299740+0000","last_peered":"2024-02-10T08:36:45.299740+0000","last_clean":"2024-02-10T08:36:45.299740+0000","last_became_active":"2024-02-10T08:33:13.521573+0000","last_became_peered":"2024-02-10T08:33:13.521573+0000","last_unstale":"2024-02-10T08:36:45.299740+0000","last_undegraded":"2024-02-10T08:36:45.299740+0000","last_fullsized":"2024-02-10T08:36:45.299740+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.053809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.053809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303901+0000","last_change":"2024-02-10T08:36:30.795587+0000","last_active":"2024-02-10T08:36:45.303901+0000","last_peered":"2024-02-10T08:36:45.303901+0000","last_clean":"2024-02-10T08:36:45.303901+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:36:45.303901+0000","last_undegraded":"2024-02-10T08:36:45.303901+0000","last_fullsized":"2024-02-10T08:36:45.303901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302151+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:36:45.302151+0000","last_peered":"2024-02-10T08:36:45.302151+0000","last_clean":"2024-02-10T08:36:45.302151+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:36:45.302151+0000","last_undegraded":"2024-02-10T08:36:45.302151+0000","last_fullsized":"2024-02-10T08:36:45.302151+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314770+0000","last_change":"2024-02-10T08:36:29.804845+0000","last_active":"2024-02-10T08:36:45.314770+0000","last_peered":"2024-02-10T08:36:45.314770+0000","last_clean":"2024-02-10T08:36:45.314770+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:36:45.314770+0000","last_undegraded":"2024-02-10T08:36:45.314770+0000","last_fullsized":"2024-02-10T08:36:45.314770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:29.804761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.804761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292500+0000","last_change":"2024-02-10T08:36:37.114544+0000","last_active":"2024-02-10T08:36:45.292500+0000","last_peered":"2024-02-10T08:36:45.292500+0000","last_clean":"2024-02-10T08:36:45.292500+0000","last_became_active":"2024-02-10T08:36:37.114240+0000","last_became_peered":"2024-02-10T08:36:37.114240+0000","last_unstale":"2024-02-10T08:36:45.292500+0000","last_undegraded":"2024-02-10T08:36:45.292500+0000","last_fullsized":"2024-02-10T08:36:45.292500+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.950844+0000","last_change":"2024-02-10T08:36:45.325691+0000","last_active":"2024-02-10T08:36:45.950844+0000","last_peered":"2024-02-10T08:36:45.950844+0000","last_clean":"2024-02-10T08:36:45.950844+0000","last_became_active":"2024-02-10T08:36:37.078599+0000","last_became_peered":"2024-02-10T08:36:37.078599+0000","last_unstale":"2024-02-10T08:36:45.950844+0000","last_undegraded":"2024-02-10T08:36:45.950844+0000","last_fullsized":"2024-02-10T08:36:45.950844+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1a","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305392+0000","last_change":"2024-02-10T08:36:37.131190+0000","last_active":"2024-02-10T08:36:45.305392+0000","last_peered":"2024-02-10T08:36:45.305392+0000","last_clean":"2024-02-10T08:36:45.305392+0000","last_became_active":"2024-02-10T08:36:37.128609+0000","last_became_peered":"2024-02-10T08:36:37.128609+0000","last_unstale":"2024-02-10T08:36:45.305392+0000","last_undegraded":"2024-02-10T08:36:45.305392+0000","last_fullsized":"2024-02-10T08:36:45.305392+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"103'6","reported_seq":25,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316372+0000","last_change":"2024-02-10T08:36:37.156906+0000","last_active":"2024-02-10T08:36:45.316372+0000","last_peered":"2024-02-10T08:36:45.316372+0000","last_clean":"2024-02-10T08:36:45.316372+0000","last_became_active":"2024-02-10T08:36:37.156334+0000","last_became_peered":"2024-02-10T08:36:37.156334+0000","last_unstale":"2024-02-10T08:36:45.316372+0000","last_undegraded":"2024-02-10T08:36:45.316372+0000","last_fullsized":"2024-02-10T08:36:45.316372+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":66,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.312106+0000","last_change":"2024-02-10T08:35:08.720849+0000","last_active":"2024-02-10T08:36:45.312106+0000","last_peered":"2024-02-10T08:36:45.312106+0000","last_clean":"2024-02-10T08:36:45.312106+0000","last_became_active":"2024-02-10T08:33:22.116503+0000","last_became_peered":"2024-02-10T08:33:22.116503+0000","last_unstale":"2024-02-10T08:36:45.312106+0000","last_undegraded":"2024-02-10T08:36:45.312106+0000","last_fullsized":"2024-02-10T08:36:45.312106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.720784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.720784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279109+0000","last_change":"2024-02-10T08:35:42.326705+0000","last_active":"2024-02-10T08:36:45.279109+0000","last_peered":"2024-02-10T08:36:45.279109+0000","last_clean":"2024-02-10T08:36:45.279109+0000","last_became_active":"2024-02-10T08:33:20.116505+0000","last_became_peered":"2024-02-10T08:33:20.116505+0000","last_unstale":"2024-02-10T08:36:45.279109+0000","last_undegraded":"2024-02-10T08:36:45.279109+0000","last_fullsized":"2024-02-10T08:36:45.279109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:35:42.326627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.326627+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281650+0000","last_change":"2024-02-10T08:36:28.635691+0000","last_active":"2024-02-10T08:36:45.281650+0000","last_peered":"2024-02-10T08:36:45.281650+0000","last_clean":"2024-02-10T08:36:45.281650+0000","last_became_active":"2024-02-10T08:33:18.104092+0000","last_became_peered":"2024-02-10T08:33:18.104092+0000","last_unstale":"2024-02-10T08:36:45.281650+0000","last_undegraded":"2024-02-10T08:36:45.281650+0000","last_fullsized":"2024-02-10T08:36:45.281650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:28.635610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":280,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.415337+0000","last_change":"2024-02-10T08:36:33.452480+0000","last_active":"2024-02-10T08:36:45.415337+0000","last_peered":"2024-02-10T08:36:45.415337+0000","last_clean":"2024-02-10T08:36:45.415337+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:36:45.415337+0000","last_undegraded":"2024-02-10T08:36:45.415337+0000","last_fullsized":"2024-02-10T08:36:45.415337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.452367+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305664+0000","last_change":"2024-02-10T08:36:32.132836+0000","last_active":"2024-02-10T08:36:45.305664+0000","last_peered":"2024-02-10T08:36:45.305664+0000","last_clean":"2024-02-10T08:36:45.305664+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:36:45.305664+0000","last_undegraded":"2024-02-10T08:36:45.305664+0000","last_fullsized":"2024-02-10T08:36:45.305664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:32.132760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.132760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280888+0000","last_change":"2024-02-10T08:36:37.181546+0000","last_active":"2024-02-10T08:36:45.280888+0000","last_peered":"2024-02-10T08:36:45.280888+0000","last_clean":"2024-02-10T08:36:45.280888+0000","last_became_active":"2024-02-10T08:36:37.171740+0000","last_became_peered":"2024-02-10T08:36:37.171740+0000","last_unstale":"2024-02-10T08:36:45.280888+0000","last_undegraded":"2024-02-10T08:36:45.280888+0000","last_fullsized":"2024-02-10T08:36:45.280888+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.955414+0000","last_change":"2024-02-10T08:36:45.326053+0000","last_active":"2024-02-10T08:36:45.955414+0000","last_peered":"2024-02-10T08:36:45.955414+0000","last_clean":"2024-02-10T08:36:45.955414+0000","last_became_active":"2024-02-10T08:36:37.164464+0000","last_became_peered":"2024-02-10T08:36:37.164464+0000","last_unstale":"2024-02-10T08:36:45.955414+0000","last_undegraded":"2024-02-10T08:36:45.955414+0000","last_fullsized":"2024-02-10T08:36:45.955414+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.19","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281385+0000","last_change":"2024-02-10T08:36:37.077971+0000","last_active":"2024-02-10T08:36:45.281385+0000","last_peered":"2024-02-10T08:36:45.281385+0000","last_clean":"2024-02-10T08:36:45.281385+0000","last_became_active":"2024-02-10T08:36:37.074340+0000","last_became_peered":"2024-02-10T08:36:37.074340+0000","last_unstale":"2024-02-10T08:36:45.281385+0000","last_undegraded":"2024-02-10T08:36:45.281385+0000","last_fullsized":"2024-02-10T08:36:45.281385+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"105'2","reported_seq":34,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.808565+0000","last_change":"2024-02-10T08:36:45.808527+0000","last_active":"2024-02-10T08:36:45.808565+0000","last_peered":"2024-02-10T08:36:45.808565+0000","last_clean":"2024-02-10T08:36:45.808565+0000","last_became_active":"2024-02-10T08:36:37.079083+0000","last_became_peered":"2024-02-10T08:36:37.079083+0000","last_unstale":"2024-02-10T08:36:45.808565+0000","last_undegraded":"2024-02-10T08:36:45.808565+0000","last_fullsized":"2024-02-10T08:36:45.808565+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-02-10T08:36:45.808419+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-02-10T08:36:45.808419+0000","last_clean_scrub_stamp":"2024-02-10T08:36:45.808419+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":70,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327047+0000","last_change":"2024-02-10T08:36:26.962399+0000","last_active":"2024-02-10T08:36:45.327047+0000","last_peered":"2024-02-10T08:36:45.327047+0000","last_clean":"2024-02-10T08:36:45.327047+0000","last_became_active":"2024-02-10T08:33:22.120856+0000","last_became_peered":"2024-02-10T08:33:22.120856+0000","last_unstale":"2024-02-10T08:36:45.327047+0000","last_undegraded":"2024-02-10T08:36:45.327047+0000","last_fullsized":"2024-02-10T08:36:45.327047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.962334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.962334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310560+0000","last_change":"2024-02-10T08:36:17.572570+0000","last_active":"2024-02-10T08:36:45.310560+0000","last_peered":"2024-02-10T08:36:45.310560+0000","last_clean":"2024-02-10T08:36:45.310560+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:36:45.310560+0000","last_undegraded":"2024-02-10T08:36:45.310560+0000","last_fullsized":"2024-02-10T08:36:45.310560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:36:17.572475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.572475+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302015+0000","last_change":"2024-02-10T08:35:42.473094+0000","last_active":"2024-02-10T08:36:45.302015+0000","last_peered":"2024-02-10T08:36:45.302015+0000","last_clean":"2024-02-10T08:36:45.302015+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:36:45.302015+0000","last_undegraded":"2024-02-10T08:36:45.302015+0000","last_fullsized":"2024-02-10T08:36:45.302015+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.473021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":63,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.313813+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:36:45.313813+0000","last_peered":"2024-02-10T08:36:45.313813+0000","last_clean":"2024-02-10T08:36:45.313813+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:36:45.313813+0000","last_undegraded":"2024-02-10T08:36:45.313813+0000","last_fullsized":"2024-02-10T08:36:45.313813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279159+0000","last_change":"2024-02-10T08:36:36.996181+0000","last_active":"2024-02-10T08:36:45.279159+0000","last_peered":"2024-02-10T08:36:45.279159+0000","last_clean":"2024-02-10T08:36:45.279159+0000","last_became_active":"2024-02-10T08:36:36.995408+0000","last_became_peered":"2024-02-10T08:36:36.995408+0000","last_unstale":"2024-02-10T08:36:45.279159+0000","last_undegraded":"2024-02-10T08:36:45.279159+0000","last_fullsized":"2024-02-10T08:36:45.279159+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.953533+0000","last_change":"2024-02-10T08:36:45.330636+0000","last_active":"2024-02-10T08:36:45.953533+0000","last_peered":"2024-02-10T08:36:45.953533+0000","last_clean":"2024-02-10T08:36:45.953533+0000","last_became_active":"2024-02-10T08:36:37.157268+0000","last_became_peered":"2024-02-10T08:36:37.157268+0000","last_unstale":"2024-02-10T08:36:45.953533+0000","last_undegraded":"2024-02-10T08:36:45.953533+0000","last_fullsized":"2024-02-10T08:36:45.953533+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1f","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327783+0000","last_change":"2024-02-10T08:36:37.005646+0000","last_active":"2024-02-10T08:36:45.327783+0000","last_peered":"2024-02-10T08:36:45.327783+0000","last_clean":"2024-02-10T08:36:45.327783+0000","last_became_active":"2024-02-10T08:36:37.004719+0000","last_became_peered":"2024-02-10T08:36:37.004719+0000","last_unstale":"2024-02-10T08:36:45.327783+0000","last_undegraded":"2024-02-10T08:36:45.327783+0000","last_fullsized":"2024-02-10T08:36:45.327783+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"103'1","reported_seq":21,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.319785+0000","last_change":"2024-02-10T08:36:37.024570+0000","last_active":"2024-02-10T08:36:45.319785+0000","last_peered":"2024-02-10T08:36:45.319785+0000","last_clean":"2024-02-10T08:36:45.319785+0000","last_became_active":"2024-02-10T08:36:37.023071+0000","last_became_peered":"2024-02-10T08:36:37.023071+0000","last_unstale":"2024-02-10T08:36:45.319785+0000","last_undegraded":"2024-02-10T08:36:45.319785+0000","last_fullsized":"2024-02-10T08:36:45.319785+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"63'8","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309184+0000","last_change":"2024-02-10T08:36:27.957994+0000","last_active":"2024-02-10T08:36:45.309184+0000","last_peered":"2024-02-10T08:36:45.309184+0000","last_clean":"2024-02-10T08:36:45.309184+0000","last_became_active":"2024-02-10T08:33:22.123203+0000","last_became_peered":"2024-02-10T08:33:22.123203+0000","last_unstale":"2024-02-10T08:36:45.309184+0000","last_undegraded":"2024-02-10T08:36:45.309184+0000","last_fullsized":"2024-02-10T08:36:45.309184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:27.957926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.957926+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":93,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291005+0000","last_change":"2024-02-10T08:35:34.249489+0000","last_active":"2024-02-10T08:36:45.291005+0000","last_peered":"2024-02-10T08:36:45.291005+0000","last_clean":"2024-02-10T08:36:45.291005+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:36:45.291005+0000","last_undegraded":"2024-02-10T08:36:45.291005+0000","last_fullsized":"2024-02-10T08:36:45.291005+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.249424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290038+0000","last_change":"2024-02-10T08:36:26.456845+0000","last_active":"2024-02-10T08:36:45.290038+0000","last_peered":"2024-02-10T08:36:45.290038+0000","last_clean":"2024-02-10T08:36:45.290038+0000","last_became_active":"2024-02-10T08:33:18.108085+0000","last_became_peered":"2024-02-10T08:33:18.108085+0000","last_unstale":"2024-02-10T08:36:45.290038+0000","last_undegraded":"2024-02-10T08:36:45.290038+0000","last_fullsized":"2024-02-10T08:36:45.290038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.456780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.313238+0000","last_change":"2024-02-10T08:36:13.802328+0000","last_active":"2024-02-10T08:36:45.313238+0000","last_peered":"2024-02-10T08:36:45.313238+0000","last_clean":"2024-02-10T08:36:45.313238+0000","last_became_active":"2024-02-10T08:33:24.122197+0000","last_became_peered":"2024-02-10T08:33:24.122197+0000","last_unstale":"2024-02-10T08:36:45.313238+0000","last_undegraded":"2024-02-10T08:36:45.313238+0000","last_fullsized":"2024-02-10T08:36:45.313238+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.802264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.802264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310472+0000","last_change":"2024-02-10T08:36:37.155744+0000","last_active":"2024-02-10T08:36:45.310472+0000","last_peered":"2024-02-10T08:36:45.310472+0000","last_clean":"2024-02-10T08:36:45.310472+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:36:45.310472+0000","last_undegraded":"2024-02-10T08:36:45.310472+0000","last_fullsized":"2024-02-10T08:36:45.310472+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.921752+0000","last_change":"2024-02-10T08:36:45.297269+0000","last_active":"2024-02-10T08:36:45.921752+0000","last_peered":"2024-02-10T08:36:45.921752+0000","last_clean":"2024-02-10T08:36:45.921752+0000","last_became_active":"2024-02-10T08:36:37.117066+0000","last_became_peered":"2024-02-10T08:36:37.117066+0000","last_unstale":"2024-02-10T08:36:45.921752+0000","last_undegraded":"2024-02-10T08:36:45.921752+0000","last_fullsized":"2024-02-10T08:36:45.921752+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.14","version":"103'2","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285344+0000","last_change":"2024-02-10T08:36:37.065315+0000","last_active":"2024-02-10T08:36:45.285344+0000","last_peered":"2024-02-10T08:36:45.285344+0000","last_clean":"2024-02-10T08:36:45.285344+0000","last_became_active":"2024-02-10T08:36:37.063912+0000","last_became_peered":"2024-02-10T08:36:37.063912+0000","last_unstale":"2024-02-10T08:36:45.285344+0000","last_undegraded":"2024-02-10T08:36:45.285344+0000","last_fullsized":"2024-02-10T08:36:45.285344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":24,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332458+0000","last_change":"2024-02-10T08:36:43.985689+0000","last_active":"2024-02-10T08:36:45.332458+0000","last_peered":"2024-02-10T08:36:45.332458+0000","last_clean":"2024-02-10T08:36:45.332458+0000","last_became_active":"2024-02-10T08:36:37.010946+0000","last_became_peered":"2024-02-10T08:36:37.010946+0000","last_unstale":"2024-02-10T08:36:45.332458+0000","last_undegraded":"2024-02-10T08:36:45.332458+0000","last_fullsized":"2024-02-10T08:36:45.332458+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:43.985592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:43.985592+0000","last_clean_scrub_stamp":"2024-02-10T08:36:43.985592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279790+0000","last_change":"2024-02-10T08:36:16.546775+0000","last_active":"2024-02-10T08:36:45.279790+0000","last_peered":"2024-02-10T08:36:45.279790+0000","last_clean":"2024-02-10T08:36:45.279790+0000","last_became_active":"2024-02-10T08:33:22.117969+0000","last_became_peered":"2024-02-10T08:33:22.117969+0000","last_unstale":"2024-02-10T08:36:45.279790+0000","last_undegraded":"2024-02-10T08:36:45.279790+0000","last_fullsized":"2024-02-10T08:36:45.279790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:16.546674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.546674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280581+0000","last_change":"2024-02-10T08:36:28.744540+0000","last_active":"2024-02-10T08:36:45.280581+0000","last_peered":"2024-02-10T08:36:45.280581+0000","last_clean":"2024-02-10T08:36:45.280581+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:36:45.280581+0000","last_undegraded":"2024-02-10T08:36:45.280581+0000","last_fullsized":"2024-02-10T08:36:45.280581+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:36:28.744465+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.744465+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285932+0000","last_change":"2024-02-10T08:35:00.136470+0000","last_active":"2024-02-10T08:36:45.285932+0000","last_peered":"2024-02-10T08:36:45.285932+0000","last_clean":"2024-02-10T08:36:45.285932+0000","last_became_active":"2024-02-10T08:33:18.102929+0000","last_became_peered":"2024-02-10T08:33:18.102929+0000","last_unstale":"2024-02-10T08:36:45.285932+0000","last_undegraded":"2024-02-10T08:36:45.285932+0000","last_fullsized":"2024-02-10T08:36:45.285932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:35:00.136390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.136390+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316924+0000","last_change":"2024-02-10T08:36:23.084433+0000","last_active":"2024-02-10T08:36:45.316924+0000","last_peered":"2024-02-10T08:36:45.316924+0000","last_clean":"2024-02-10T08:36:45.316924+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:36:45.316924+0000","last_undegraded":"2024-02-10T08:36:45.316924+0000","last_fullsized":"2024-02-10T08:36:45.316924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:23.084331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.084331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287486+0000","last_change":"2024-02-10T08:36:37.053619+0000","last_active":"2024-02-10T08:36:45.287486+0000","last_peered":"2024-02-10T08:36:45.287486+0000","last_clean":"2024-02-10T08:36:45.287486+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:36:45.287486+0000","last_undegraded":"2024-02-10T08:36:45.287486+0000","last_fullsized":"2024-02-10T08:36:45.287486+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.894020+0000","last_change":"2024-02-10T08:36:45.307018+0000","last_active":"2024-02-10T08:36:45.894020+0000","last_peered":"2024-02-10T08:36:45.894020+0000","last_clean":"2024-02-10T08:36:45.894020+0000","last_became_active":"2024-02-10T08:36:37.122545+0000","last_became_peered":"2024-02-10T08:36:37.122545+0000","last_unstale":"2024-02-10T08:36:45.894020+0000","last_undegraded":"2024-02-10T08:36:45.894020+0000","last_fullsized":"2024-02-10T08:36:45.894020+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.b","version":"103'2","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.283431+0000","last_change":"2024-02-10T08:36:37.095396+0000","last_active":"2024-02-10T08:36:45.283431+0000","last_peered":"2024-02-10T08:36:45.283431+0000","last_clean":"2024-02-10T08:36:45.283431+0000","last_became_active":"2024-02-10T08:36:37.094868+0000","last_became_peered":"2024-02-10T08:36:37.094868+0000","last_unstale":"2024-02-10T08:36:45.283431+0000","last_undegraded":"2024-02-10T08:36:45.283431+0000","last_fullsized":"2024-02-10T08:36:45.283431+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306249+0000","last_change":"2024-02-10T08:36:37.037394+0000","last_active":"2024-02-10T08:36:45.306249+0000","last_peered":"2024-02-10T08:36:45.306249+0000","last_clean":"2024-02-10T08:36:45.306249+0000","last_became_active":"2024-02-10T08:36:37.016196+0000","last_became_peered":"2024-02-10T08:36:37.016196+0000","last_unstale":"2024-02-10T08:36:45.306249+0000","last_undegraded":"2024-02-10T08:36:45.306249+0000","last_fullsized":"2024-02-10T08:36:45.306249+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.320576+0000","last_change":"2024-02-10T08:35:33.677439+0000","last_active":"2024-02-10T08:36:45.320576+0000","last_peered":"2024-02-10T08:36:45.320576+0000","last_clean":"2024-02-10T08:36:45.320576+0000","last_became_active":"2024-02-10T08:33:20.256099+0000","last_became_peered":"2024-02-10T08:33:20.256099+0000","last_unstale":"2024-02-10T08:36:45.320576+0000","last_undegraded":"2024-02-10T08:36:45.320576+0000","last_fullsized":"2024-02-10T08:36:45.320576+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:35:33.677377+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279704+0000","last_change":"2024-02-10T08:36:22.639484+0000","last_active":"2024-02-10T08:36:45.279704+0000","last_peered":"2024-02-10T08:36:45.279704+0000","last_clean":"2024-02-10T08:36:45.279704+0000","last_became_active":"2024-02-10T08:33:18.104911+0000","last_became_peered":"2024-02-10T08:33:18.104911+0000","last_unstale":"2024-02-10T08:36:45.279704+0000","last_undegraded":"2024-02-10T08:36:45.279704+0000","last_fullsized":"2024-02-10T08:36:45.279704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.639381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.639381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":26,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314677+0000","last_change":"2024-02-10T08:36:23.932743+0000","last_active":"2024-02-10T08:36:45.314677+0000","last_peered":"2024-02-10T08:36:45.314677+0000","last_clean":"2024-02-10T08:36:45.314677+0000","last_became_active":"2024-02-10T08:36:23.932443+0000","last_became_peered":"2024-02-10T08:36:23.932443+0000","last_unstale":"2024-02-10T08:36:45.314677+0000","last_undegraded":"2024-02-10T08:36:45.314677+0000","last_fullsized":"2024-02-10T08:36:45.314677+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.446574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.446574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289351+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:36:45.289351+0000","last_peered":"2024-02-10T08:36:45.289351+0000","last_clean":"2024-02-10T08:36:45.289351+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:36:45.289351+0000","last_undegraded":"2024-02-10T08:36:45.289351+0000","last_fullsized":"2024-02-10T08:36:45.289351+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281657+0000","last_change":"2024-02-10T08:36:37.041210+0000","last_active":"2024-02-10T08:36:45.281657+0000","last_peered":"2024-02-10T08:36:45.281657+0000","last_clean":"2024-02-10T08:36:45.281657+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:36:45.281657+0000","last_undegraded":"2024-02-10T08:36:45.281657+0000","last_fullsized":"2024-02-10T08:36:45.281657+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.930302+0000","last_change":"2024-02-10T08:36:45.324750+0000","last_active":"2024-02-10T08:36:45.930302+0000","last_peered":"2024-02-10T08:36:45.930302+0000","last_clean":"2024-02-10T08:36:45.930302+0000","last_became_active":"2024-02-10T08:36:37.101632+0000","last_became_peered":"2024-02-10T08:36:37.101632+0000","last_unstale":"2024-02-10T08:36:45.930302+0000","last_undegraded":"2024-02-10T08:36:45.930302+0000","last_fullsized":"2024-02-10T08:36:45.930302+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.a","version":"103'10","reported_seq":28,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303866+0000","last_change":"2024-02-10T08:36:37.038036+0000","last_active":"2024-02-10T08:36:45.303866+0000","last_peered":"2024-02-10T08:36:45.303866+0000","last_clean":"2024-02-10T08:36:45.303866+0000","last_became_active":"2024-02-10T08:36:37.033591+0000","last_became_peered":"2024-02-10T08:36:37.033591+0000","last_unstale":"2024-02-10T08:36:45.303866+0000","last_undegraded":"2024-02-10T08:36:45.303866+0000","last_fullsized":"2024-02-10T08:36:45.303866+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285474+0000","last_change":"2024-02-10T08:36:36.997751+0000","last_active":"2024-02-10T08:36:45.285474+0000","last_peered":"2024-02-10T08:36:45.285474+0000","last_clean":"2024-02-10T08:36:45.285474+0000","last_became_active":"2024-02-10T08:36:36.996981+0000","last_became_peered":"2024-02-10T08:36:36.996981+0000","last_unstale":"2024-02-10T08:36:45.285474+0000","last_undegraded":"2024-02-10T08:36:45.285474+0000","last_fullsized":"2024-02-10T08:36:45.285474+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.325967+0000","last_change":"2024-02-10T08:34:35.662595+0000","last_active":"2024-02-10T08:36:45.325967+0000","last_peered":"2024-02-10T08:36:45.325967+0000","last_clean":"2024-02-10T08:36:45.325967+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:36:45.325967+0000","last_undegraded":"2024-02-10T08:36:45.325967+0000","last_fullsized":"2024-02-10T08:36:45.325967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:34:35.662519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:35.662519+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316711+0000","last_change":"2024-02-10T08:36:19.570275+0000","last_active":"2024-02-10T08:36:45.316711+0000","last_peered":"2024-02-10T08:36:45.316711+0000","last_clean":"2024-02-10T08:36:45.316711+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:36:45.316711+0000","last_undegraded":"2024-02-10T08:36:45.316711+0000","last_fullsized":"2024-02-10T08:36:45.316711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:19.570181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:19.570181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":26,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.311917+0000","last_change":"2024-02-10T08:36:23.671449+0000","last_active":"2024-02-10T08:36:45.311917+0000","last_peered":"2024-02-10T08:36:45.311917+0000","last_clean":"2024-02-10T08:36:45.311917+0000","last_became_active":"2024-02-10T08:36:23.671134+0000","last_became_peered":"2024-02-10T08:36:23.671134+0000","last_unstale":"2024-02-10T08:36:45.311917+0000","last_undegraded":"2024-02-10T08:36:45.311917+0000","last_fullsized":"2024-02-10T08:36:45.311917+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.616139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.616139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305422+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:36:45.305422+0000","last_peered":"2024-02-10T08:36:45.305422+0000","last_clean":"2024-02-10T08:36:45.305422+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:36:45.305422+0000","last_undegraded":"2024-02-10T08:36:45.305422+0000","last_fullsized":"2024-02-10T08:36:45.305422+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.303573+0000","last_change":"2024-02-10T08:36:37.051573+0000","last_active":"2024-02-10T08:36:45.303573+0000","last_peered":"2024-02-10T08:36:45.303573+0000","last_clean":"2024-02-10T08:36:45.303573+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:36:45.303573+0000","last_undegraded":"2024-02-10T08:36:45.303573+0000","last_fullsized":"2024-02-10T08:36:45.303573+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.911159+0000","last_change":"2024-02-10T08:36:45.292574+0000","last_active":"2024-02-10T08:36:45.911159+0000","last_peered":"2024-02-10T08:36:45.911159+0000","last_clean":"2024-02-10T08:36:45.911159+0000","last_became_active":"2024-02-10T08:36:37.046024+0000","last_became_peered":"2024-02-10T08:36:37.046024+0000","last_unstale":"2024-02-10T08:36:45.911159+0000","last_undegraded":"2024-02-10T08:36:45.911159+0000","last_fullsized":"2024-02-10T08:36:45.911159+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.9","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281120+0000","last_change":"2024-02-10T08:36:37.103137+0000","last_active":"2024-02-10T08:36:45.281120+0000","last_peered":"2024-02-10T08:36:45.281120+0000","last_clean":"2024-02-10T08:36:45.281120+0000","last_became_active":"2024-02-10T08:36:37.102650+0000","last_became_peered":"2024-02-10T08:36:37.102650+0000","last_unstale":"2024-02-10T08:36:45.281120+0000","last_undegraded":"2024-02-10T08:36:45.281120+0000","last_fullsized":"2024-02-10T08:36:45.281120+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279871+0000","last_change":"2024-02-10T08:36:37.075256+0000","last_active":"2024-02-10T08:36:45.279871+0000","last_peered":"2024-02-10T08:36:45.279871+0000","last_clean":"2024-02-10T08:36:45.279871+0000","last_became_active":"2024-02-10T08:36:37.074967+0000","last_became_peered":"2024-02-10T08:36:37.074967+0000","last_unstale":"2024-02-10T08:36:45.279871+0000","last_undegraded":"2024-02-10T08:36:45.279871+0000","last_fullsized":"2024-02-10T08:36:45.279871+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307336+0000","last_change":"2024-02-10T08:36:14.035460+0000","last_active":"2024-02-10T08:36:45.307336+0000","last_peered":"2024-02-10T08:36:45.307336+0000","last_clean":"2024-02-10T08:36:45.307336+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:36:45.307336+0000","last_undegraded":"2024-02-10T08:36:45.307336+0000","last_fullsized":"2024-02-10T08:36:45.307336+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:14.035356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.035356+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301934+0000","last_change":"2024-02-10T08:36:12.633463+0000","last_active":"2024-02-10T08:36:45.301934+0000","last_peered":"2024-02-10T08:36:45.301934+0000","last_clean":"2024-02-10T08:36:45.301934+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:36:45.301934+0000","last_undegraded":"2024-02-10T08:36:45.301934+0000","last_fullsized":"2024-02-10T08:36:45.301934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:12.633368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:12.633368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"63'8","reported_seq":115,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310572+0000","last_change":"2024-02-10T08:34:50.535880+0000","last_active":"2024-02-10T08:36:45.310572+0000","last_peered":"2024-02-10T08:36:45.310572+0000","last_clean":"2024-02-10T08:36:45.310572+0000","last_became_active":"2024-02-10T08:33:22.122282+0000","last_became_peered":"2024-02-10T08:33:22.122282+0000","last_unstale":"2024-02-10T08:36:45.310572+0000","last_undegraded":"2024-02-10T08:36:45.310572+0000","last_fullsized":"2024-02-10T08:36:45.310572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:34:50.535813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:50.535813+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.319942+0000","last_change":"2024-02-10T08:36:33.928755+0000","last_active":"2024-02-10T08:36:45.319942+0000","last_peered":"2024-02-10T08:36:45.319942+0000","last_clean":"2024-02-10T08:36:45.319942+0000","last_became_active":"2024-02-10T08:33:24.122375+0000","last_became_peered":"2024-02-10T08:33:24.122375+0000","last_unstale":"2024-02-10T08:36:45.319942+0000","last_undegraded":"2024-02-10T08:36:45.319942+0000","last_fullsized":"2024-02-10T08:36:45.319942+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.928693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314434+0000","last_change":"2024-02-10T08:36:37.050344+0000","last_active":"2024-02-10T08:36:45.314434+0000","last_peered":"2024-02-10T08:36:45.314434+0000","last_clean":"2024-02-10T08:36:45.314434+0000","last_became_active":"2024-02-10T08:36:37.044433+0000","last_became_peered":"2024-02-10T08:36:37.044433+0000","last_unstale":"2024-02-10T08:36:45.314434+0000","last_undegraded":"2024-02-10T08:36:45.314434+0000","last_fullsized":"2024-02-10T08:36:45.314434+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.901697+0000","last_change":"2024-02-10T08:36:45.341946+0000","last_active":"2024-02-10T08:36:45.901697+0000","last_peered":"2024-02-10T08:36:45.901697+0000","last_clean":"2024-02-10T08:36:45.901697+0000","last_became_active":"2024-02-10T08:36:37.117016+0000","last_became_peered":"2024-02-10T08:36:37.117016+0000","last_unstale":"2024-02-10T08:36:45.901697+0000","last_undegraded":"2024-02-10T08:36:45.901697+0000","last_fullsized":"2024-02-10T08:36:45.901697+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.8","version":"103'2","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327731+0000","last_change":"2024-02-10T08:36:37.035179+0000","last_active":"2024-02-10T08:36:45.327731+0000","last_peered":"2024-02-10T08:36:45.327731+0000","last_clean":"2024-02-10T08:36:45.327731+0000","last_became_active":"2024-02-10T08:36:37.023527+0000","last_became_peered":"2024-02-10T08:36:37.023527+0000","last_unstale":"2024-02-10T08:36:45.327731+0000","last_undegraded":"2024-02-10T08:36:45.327731+0000","last_fullsized":"2024-02-10T08:36:45.327731+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"104'1","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292870+0000","last_change":"2024-02-10T08:36:37.076039+0000","last_active":"2024-02-10T08:36:45.292870+0000","last_peered":"2024-02-10T08:36:45.292870+0000","last_clean":"2024-02-10T08:36:45.292870+0000","last_became_active":"2024-02-10T08:36:37.075826+0000","last_became_peered":"2024-02-10T08:36:37.075826+0000","last_unstale":"2024-02-10T08:36:45.292870+0000","last_undegraded":"2024-02-10T08:36:45.292870+0000","last_fullsized":"2024-02-10T08:36:45.292870+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309615+0000","last_change":"2024-02-10T08:35:13.494784+0000","last_active":"2024-02-10T08:36:45.309615+0000","last_peered":"2024-02-10T08:36:45.309615+0000","last_clean":"2024-02-10T08:36:45.309615+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:36:45.309615+0000","last_undegraded":"2024-02-10T08:36:45.309615+0000","last_fullsized":"2024-02-10T08:36:45.309615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:13.494700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:13.494700+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309010+0000","last_change":"2024-02-10T08:35:45.760941+0000","last_active":"2024-02-10T08:36:45.309010+0000","last_peered":"2024-02-10T08:36:45.309010+0000","last_clean":"2024-02-10T08:36:45.309010+0000","last_became_active":"2024-02-10T08:33:18.106025+0000","last_became_peered":"2024-02-10T08:33:18.106025+0000","last_unstale":"2024-02-10T08:36:45.309010+0000","last_undegraded":"2024-02-10T08:36:45.309010+0000","last_fullsized":"2024-02-10T08:36:45.309010+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.760869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.308535+0000","last_change":"2024-02-10T08:36:29.049034+0000","last_active":"2024-02-10T08:36:45.308535+0000","last_peered":"2024-02-10T08:36:45.308535+0000","last_clean":"2024-02-10T08:36:45.308535+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:36:45.308535+0000","last_undegraded":"2024-02-10T08:36:45.308535+0000","last_fullsized":"2024-02-10T08:36:45.308535+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:29.048965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.048965+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.287961+0000","last_change":"2024-02-10T08:35:29.834206+0000","last_active":"2024-02-10T08:36:45.287961+0000","last_peered":"2024-02-10T08:36:45.287961+0000","last_clean":"2024-02-10T08:36:45.287961+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:36:45.287961+0000","last_undegraded":"2024-02-10T08:36:45.287961+0000","last_fullsized":"2024-02-10T08:36:45.287961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:29.834114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:29.834114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":31,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281341+0000","last_change":"2024-02-10T08:36:37.041395+0000","last_active":"2024-02-10T08:36:45.281341+0000","last_peered":"2024-02-10T08:36:45.281341+0000","last_clean":"2024-02-10T08:36:45.281341+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:36:45.281341+0000","last_undegraded":"2024-02-10T08:36:45.281341+0000","last_fullsized":"2024-02-10T08:36:45.281341+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.951019+0000","last_change":"2024-02-10T08:36:45.325835+0000","last_active":"2024-02-10T08:36:45.951019+0000","last_peered":"2024-02-10T08:36:45.951019+0000","last_clean":"2024-02-10T08:36:45.951019+0000","last_became_active":"2024-02-10T08:36:37.081776+0000","last_became_peered":"2024-02-10T08:36:37.081776+0000","last_unstale":"2024-02-10T08:36:45.951019+0000","last_undegraded":"2024-02-10T08:36:45.951019+0000","last_fullsized":"2024-02-10T08:36:45.951019+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.f","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.315043+0000","last_change":"2024-02-10T08:36:37.114769+0000","last_active":"2024-02-10T08:36:45.315043+0000","last_peered":"2024-02-10T08:36:45.315043+0000","last_clean":"2024-02-10T08:36:45.315043+0000","last_became_active":"2024-02-10T08:36:37.114495+0000","last_became_peered":"2024-02-10T08:36:37.114495+0000","last_unstale":"2024-02-10T08:36:45.315043+0000","last_undegraded":"2024-02-10T08:36:45.315043+0000","last_fullsized":"2024-02-10T08:36:45.315043+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1e","version":"105'1","reported_seq":24,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310582+0000","last_change":"2024-02-10T08:36:37.131330+0000","last_active":"2024-02-10T08:36:45.310582+0000","last_peered":"2024-02-10T08:36:45.310582+0000","last_clean":"2024-02-10T08:36:45.310582+0000","last_became_active":"2024-02-10T08:36:37.126890+0000","last_became_peered":"2024-02-10T08:36:37.126890+0000","last_unstale":"2024-02-10T08:36:45.310582+0000","last_undegraded":"2024-02-10T08:36:45.310582+0000","last_fullsized":"2024-02-10T08:36:45.310582+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":93,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290883+0000","last_change":"2024-02-10T08:36:31.434160+0000","last_active":"2024-02-10T08:36:45.290883+0000","last_peered":"2024-02-10T08:36:45.290883+0000","last_clean":"2024-02-10T08:36:45.290883+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:36:45.290883+0000","last_undegraded":"2024-02-10T08:36:45.290883+0000","last_fullsized":"2024-02-10T08:36:45.290883+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.434047+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.326562+0000","last_change":"2024-02-10T08:35:37.562620+0000","last_active":"2024-02-10T08:36:45.326562+0000","last_peered":"2024-02-10T08:36:45.326562+0000","last_clean":"2024-02-10T08:36:45.326562+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:36:45.326562+0000","last_undegraded":"2024-02-10T08:36:45.326562+0000","last_fullsized":"2024-02-10T08:36:45.326562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:37.562545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:37.562545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":70,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316843+0000","last_change":"2024-02-10T08:35:43.333489+0000","last_active":"2024-02-10T08:36:45.316843+0000","last_peered":"2024-02-10T08:36:45.316843+0000","last_clean":"2024-02-10T08:36:45.316843+0000","last_became_active":"2024-02-10T08:33:22.120350+0000","last_became_peered":"2024-02-10T08:33:22.120350+0000","last_unstale":"2024-02-10T08:36:45.316843+0000","last_undegraded":"2024-02-10T08:36:45.316843+0000","last_fullsized":"2024-02-10T08:36:45.316843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310308+0000","last_change":"2024-02-10T08:36:24.978997+0000","last_active":"2024-02-10T08:36:45.310308+0000","last_peered":"2024-02-10T08:36:45.310308+0000","last_clean":"2024-02-10T08:36:45.310308+0000","last_became_active":"2024-02-10T08:33:24.124367+0000","last_became_peered":"2024-02-10T08:33:24.124367+0000","last_unstale":"2024-02-10T08:36:45.310308+0000","last_undegraded":"2024-02-10T08:36:45.310308+0000","last_fullsized":"2024-02-10T08:36:45.310308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.978888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.331910+0000","last_change":"2024-02-10T08:36:37.051660+0000","last_active":"2024-02-10T08:36:45.331910+0000","last_peered":"2024-02-10T08:36:45.331910+0000","last_clean":"2024-02-10T08:36:45.331910+0000","last_became_active":"2024-02-10T08:36:37.049952+0000","last_became_peered":"2024-02-10T08:36:37.049952+0000","last_unstale":"2024-02-10T08:36:45.331910+0000","last_undegraded":"2024-02-10T08:36:45.331910+0000","last_fullsized":"2024-02-10T08:36:45.331910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"111'6","reported_seq":35,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:36:45.919648+0000","last_change":"2024-02-10T08:36:45.294981+0000","last_active":"2024-02-10T08:36:45.919648+0000","last_peered":"2024-02-10T08:36:45.919648+0000","last_clean":"2024-02-10T08:36:45.919648+0000","last_became_active":"2024-02-10T08:36:37.091828+0000","last_became_peered":"2024-02-10T08:36:37.091828+0000","last_unstale":"2024-02-10T08:36:45.919648+0000","last_undegraded":"2024-02-10T08:36:45.919648+0000","last_fullsized":"2024-02-10T08:36:45.919648+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"25.e","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.319706+0000","last_change":"2024-02-10T08:36:37.124863+0000","last_active":"2024-02-10T08:36:45.319706+0000","last_peered":"2024-02-10T08:36:45.319706+0000","last_clean":"2024-02-10T08:36:45.319706+0000","last_became_active":"2024-02-10T08:36:37.124466+0000","last_became_peered":"2024-02-10T08:36:37.124466+0000","last_unstale":"2024-02-10T08:36:45.319706+0000","last_undegraded":"2024-02-10T08:36:45.319706+0000","last_fullsized":"2024-02-10T08:36:45.319706+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.331918+0000","last_change":"2024-02-10T08:36:37.064761+0000","last_active":"2024-02-10T08:36:45.331918+0000","last_peered":"2024-02-10T08:36:45.331918+0000","last_clean":"2024-02-10T08:36:45.331918+0000","last_became_active":"2024-02-10T08:36:37.064204+0000","last_became_peered":"2024-02-10T08:36:37.064204+0000","last_unstale":"2024-02-10T08:36:45.331918+0000","last_undegraded":"2024-02-10T08:36:45.331918+0000","last_fullsized":"2024-02-10T08:36:45.331918+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.283679+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:36:45.283679+0000","last_peered":"2024-02-10T08:36:45.283679+0000","last_clean":"2024-02-10T08:36:45.283679+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:36:45.283679+0000","last_undegraded":"2024-02-10T08:36:45.283679+0000","last_fullsized":"2024-02-10T08:36:45.283679+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.286089+0000","last_change":"2024-02-10T08:36:15.453264+0000","last_active":"2024-02-10T08:36:45.286089+0000","last_peered":"2024-02-10T08:36:45.286089+0000","last_clean":"2024-02-10T08:36:45.286089+0000","last_became_active":"2024-02-10T08:33:18.107986+0000","last_became_peered":"2024-02-10T08:33:18.107986+0000","last_unstale":"2024-02-10T08:36:45.286089+0000","last_undegraded":"2024-02-10T08:36:45.286089+0000","last_fullsized":"2024-02-10T08:36:45.286089+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.453161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.453161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":66,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.290974+0000","last_change":"2024-02-10T08:35:02.105717+0000","last_active":"2024-02-10T08:36:45.290974+0000","last_peered":"2024-02-10T08:36:45.290974+0000","last_clean":"2024-02-10T08:36:45.290974+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:36:45.290974+0000","last_undegraded":"2024-02-10T08:36:45.290974+0000","last_fullsized":"2024-02-10T08:36:45.290974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:02.105625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:02.105625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301440+0000","last_change":"2024-02-10T08:36:24.066706+0000","last_active":"2024-02-10T08:36:45.301440+0000","last_peered":"2024-02-10T08:36:45.301440+0000","last_clean":"2024-02-10T08:36:45.301440+0000","last_became_active":"2024-02-10T08:33:24.117837+0000","last_became_peered":"2024-02-10T08:33:24.117837+0000","last_unstale":"2024-02-10T08:36:45.301440+0000","last_undegraded":"2024-02-10T08:36:45.301440+0000","last_fullsized":"2024-02-10T08:36:45.301440+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.066605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.066605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.322135+0000","last_change":"2024-02-10T08:36:37.146078+0000","last_active":"2024-02-10T08:36:45.322135+0000","last_peered":"2024-02-10T08:36:45.322135+0000","last_clean":"2024-02-10T08:36:45.322135+0000","last_became_active":"2024-02-10T08:36:37.145330+0000","last_became_peered":"2024-02-10T08:36:37.145330+0000","last_unstale":"2024-02-10T08:36:45.322135+0000","last_undegraded":"2024-02-10T08:36:45.322135+0000","last_fullsized":"2024-02-10T08:36:45.322135+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.903300+0000","last_change":"2024-02-10T08:36:45.321759+0000","last_active":"2024-02-10T08:36:45.903300+0000","last_peered":"2024-02-10T08:36:45.903300+0000","last_clean":"2024-02-10T08:36:45.903300+0000","last_became_active":"2024-02-10T08:36:37.089069+0000","last_became_peered":"2024-02-10T08:36:37.089069+0000","last_unstale":"2024-02-10T08:36:45.903300+0000","last_undegraded":"2024-02-10T08:36:45.903300+0000","last_fullsized":"2024-02-10T08:36:45.903300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.d","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332348+0000","last_change":"2024-02-10T08:36:37.082707+0000","last_active":"2024-02-10T08:36:45.332348+0000","last_peered":"2024-02-10T08:36:45.332348+0000","last_clean":"2024-02-10T08:36:45.332348+0000","last_became_active":"2024-02-10T08:36:37.081295+0000","last_became_peered":"2024-02-10T08:36:37.081295+0000","last_unstale":"2024-02-10T08:36:45.332348+0000","last_undegraded":"2024-02-10T08:36:45.332348+0000","last_fullsized":"2024-02-10T08:36:45.332348+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"104'1001","reported_seq":2020,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310566+0000","last_change":"2024-02-10T08:36:37.024274+0000","last_active":"2024-02-10T08:36:45.310566+0000","last_peered":"2024-02-10T08:36:45.310566+0000","last_clean":"2024-02-10T08:36:45.310566+0000","last_became_active":"2024-02-10T08:36:37.023900+0000","last_became_peered":"2024-02-10T08:36:37.023900+0000","last_unstale":"2024-02-10T08:36:45.310566+0000","last_undegraded":"2024-02-10T08:36:45.310566+0000","last_fullsized":"2024-02-10T08:36:45.310566+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1001,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":73,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295008+0000","last_change":"2024-02-10T08:34:53.487239+0000","last_active":"2024-02-10T08:36:45.295008+0000","last_peered":"2024-02-10T08:36:45.295008+0000","last_clean":"2024-02-10T08:36:45.295008+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:36:45.295008+0000","last_undegraded":"2024-02-10T08:36:45.295008+0000","last_fullsized":"2024-02-10T08:36:45.295008+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:34:53.487135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:34:53.487135+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.317516+0000","last_change":"2024-02-10T08:36:15.645089+0000","last_active":"2024-02-10T08:36:45.317516+0000","last_peered":"2024-02-10T08:36:45.317516+0000","last_clean":"2024-02-10T08:36:45.317516+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:36:45.317516+0000","last_undegraded":"2024-02-10T08:36:45.317516+0000","last_fullsized":"2024-02-10T08:36:45.317516+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.644998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.644998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":71,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316624+0000","last_change":"2024-02-10T08:36:33.824585+0000","last_active":"2024-02-10T08:36:45.316624+0000","last_peered":"2024-02-10T08:36:45.316624+0000","last_clean":"2024-02-10T08:36:45.316624+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:36:45.316624+0000","last_undegraded":"2024-02-10T08:36:45.316624+0000","last_fullsized":"2024-02-10T08:36:45.316624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.824511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.824511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292498+0000","last_change":"2024-02-10T08:36:24.096165+0000","last_active":"2024-02-10T08:36:45.292498+0000","last_peered":"2024-02-10T08:36:45.292498+0000","last_clean":"2024-02-10T08:36:45.292498+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:36:45.292498+0000","last_undegraded":"2024-02-10T08:36:45.292498+0000","last_fullsized":"2024-02-10T08:36:45.292498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.096104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.096104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306581+0000","last_change":"2024-02-10T08:36:37.165190+0000","last_active":"2024-02-10T08:36:45.306581+0000","last_peered":"2024-02-10T08:36:45.306581+0000","last_clean":"2024-02-10T08:36:45.306581+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:36:45.306581+0000","last_undegraded":"2024-02-10T08:36:45.306581+0000","last_fullsized":"2024-02-10T08:36:45.306581+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.888195+0000","last_change":"2024-02-10T08:36:45.289305+0000","last_active":"2024-02-10T08:36:45.888195+0000","last_peered":"2024-02-10T08:36:45.888195+0000","last_clean":"2024-02-10T08:36:45.888195+0000","last_became_active":"2024-02-10T08:36:37.095234+0000","last_became_peered":"2024-02-10T08:36:37.095234+0000","last_unstale":"2024-02-10T08:36:45.888195+0000","last_undegraded":"2024-02-10T08:36:45.888195+0000","last_fullsized":"2024-02-10T08:36:45.888195+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.c","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.311257+0000","last_change":"2024-02-10T08:36:37.127556+0000","last_active":"2024-02-10T08:36:45.311257+0000","last_peered":"2024-02-10T08:36:45.311257+0000","last_clean":"2024-02-10T08:36:45.311257+0000","last_became_active":"2024-02-10T08:36:37.127287+0000","last_became_peered":"2024-02-10T08:36:37.127287+0000","last_unstale":"2024-02-10T08:36:45.311257+0000","last_undegraded":"2024-02-10T08:36:45.311257+0000","last_fullsized":"2024-02-10T08:36:45.311257+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"105'4","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.317010+0000","last_change":"2024-02-10T08:36:37.127419+0000","last_active":"2024-02-10T08:36:45.317010+0000","last_peered":"2024-02-10T08:36:45.317010+0000","last_clean":"2024-02-10T08:36:45.317010+0000","last_became_active":"2024-02-10T08:36:37.120154+0000","last_became_peered":"2024-02-10T08:36:37.120154+0000","last_unstale":"2024-02-10T08:36:45.317010+0000","last_undegraded":"2024-02-10T08:36:45.317010+0000","last_fullsized":"2024-02-10T08:36:45.317010+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":27325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":86,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301960+0000","last_change":"2024-02-10T08:35:11.542445+0000","last_active":"2024-02-10T08:36:45.301960+0000","last_peered":"2024-02-10T08:36:45.301960+0000","last_clean":"2024-02-10T08:36:45.301960+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:36:45.301960+0000","last_undegraded":"2024-02-10T08:36:45.301960+0000","last_fullsized":"2024-02-10T08:36:45.301960+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:11.542338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.542338+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.332712+0000","last_change":"2024-02-10T08:36:13.537919+0000","last_active":"2024-02-10T08:36:45.332712+0000","last_peered":"2024-02-10T08:36:45.332712+0000","last_clean":"2024-02-10T08:36:45.332712+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:36:45.332712+0000","last_undegraded":"2024-02-10T08:36:45.332712+0000","last_fullsized":"2024-02-10T08:36:45.332712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.537840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.537840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":66,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.300009+0000","last_change":"2024-02-10T08:34:58.971694+0000","last_active":"2024-02-10T08:36:45.300009+0000","last_peered":"2024-02-10T08:36:45.300009+0000","last_clean":"2024-02-10T08:36:45.300009+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:36:45.300009+0000","last_undegraded":"2024-02-10T08:36:45.300009+0000","last_fullsized":"2024-02-10T08:36:45.300009+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:58.971607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:34:58.971607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.317079+0000","last_change":"2024-02-10T08:36:33.042649+0000","last_active":"2024-02-10T08:36:45.317079+0000","last_peered":"2024-02-10T08:36:45.317079+0000","last_clean":"2024-02-10T08:36:45.317079+0000","last_became_active":"2024-02-10T08:33:24.125966+0000","last_became_peered":"2024-02-10T08:33:24.125966+0000","last_unstale":"2024-02-10T08:36:45.317079+0000","last_undegraded":"2024-02-10T08:36:45.317079+0000","last_fullsized":"2024-02-10T08:36:45.317079+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.042582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.042582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316929+0000","last_change":"2024-02-10T08:36:37.060344+0000","last_active":"2024-02-10T08:36:45.316929+0000","last_peered":"2024-02-10T08:36:45.316929+0000","last_clean":"2024-02-10T08:36:45.316929+0000","last_became_active":"2024-02-10T08:36:37.059898+0000","last_became_peered":"2024-02-10T08:36:37.059898+0000","last_unstale":"2024-02-10T08:36:45.316929+0000","last_undegraded":"2024-02-10T08:36:45.316929+0000","last_fullsized":"2024-02-10T08:36:45.316929+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.924711+0000","last_change":"2024-02-10T08:36:45.305165+0000","last_active":"2024-02-10T08:36:45.924711+0000","last_peered":"2024-02-10T08:36:45.924711+0000","last_clean":"2024-02-10T08:36:45.924711+0000","last_became_active":"2024-02-10T08:36:37.116157+0000","last_became_peered":"2024-02-10T08:36:37.116157+0000","last_unstale":"2024-02-10T08:36:45.924711+0000","last_undegraded":"2024-02-10T08:36:45.924711+0000","last_fullsized":"2024-02-10T08:36:45.924711+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.3","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.294771+0000","last_change":"2024-02-10T08:36:37.052377+0000","last_active":"2024-02-10T08:36:45.294771+0000","last_peered":"2024-02-10T08:36:45.294771+0000","last_clean":"2024-02-10T08:36:45.294771+0000","last_became_active":"2024-02-10T08:36:37.050808+0000","last_became_peered":"2024-02-10T08:36:37.050808+0000","last_unstale":"2024-02-10T08:36:45.294771+0000","last_undegraded":"2024-02-10T08:36:45.294771+0000","last_fullsized":"2024-02-10T08:36:45.294771+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.329039+0000","last_change":"2024-02-10T08:36:37.028862+0000","last_active":"2024-02-10T08:36:45.329039+0000","last_peered":"2024-02-10T08:36:45.329039+0000","last_clean":"2024-02-10T08:36:45.329039+0000","last_became_active":"2024-02-10T08:36:37.012947+0000","last_became_peered":"2024-02-10T08:36:37.012947+0000","last_unstale":"2024-02-10T08:36:45.329039+0000","last_undegraded":"2024-02-10T08:36:45.329039+0000","last_fullsized":"2024-02-10T08:36:45.329039+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.321378+0000","last_change":"2024-02-10T08:36:31.890457+0000","last_active":"2024-02-10T08:36:45.321378+0000","last_peered":"2024-02-10T08:36:45.321378+0000","last_clean":"2024-02-10T08:36:45.321378+0000","last_became_active":"2024-02-10T08:33:20.256491+0000","last_became_peered":"2024-02-10T08:33:20.256491+0000","last_unstale":"2024-02-10T08:36:45.321378+0000","last_undegraded":"2024-02-10T08:36:45.321378+0000","last_fullsized":"2024-02-10T08:36:45.321378+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:31.890386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.890386+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288431+0000","last_change":"2024-02-10T08:35:36.513411+0000","last_active":"2024-02-10T08:36:45.288431+0000","last_peered":"2024-02-10T08:36:45.288431+0000","last_clean":"2024-02-10T08:36:45.288431+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:36:45.288431+0000","last_undegraded":"2024-02-10T08:36:45.288431+0000","last_fullsized":"2024-02-10T08:36:45.288431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:36.513316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.513316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":65,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299951+0000","last_change":"2024-02-10T08:35:09.044967+0000","last_active":"2024-02-10T08:36:45.299951+0000","last_peered":"2024-02-10T08:36:45.299951+0000","last_clean":"2024-02-10T08:36:45.299951+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:36:45.299951+0000","last_undegraded":"2024-02-10T08:36:45.299951+0000","last_fullsized":"2024-02-10T08:36:45.299951+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:09.044890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.044890+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.296578+0000","last_change":"2024-02-10T08:36:37.049548+0000","last_active":"2024-02-10T08:36:45.296578+0000","last_peered":"2024-02-10T08:36:45.296578+0000","last_clean":"2024-02-10T08:36:45.296578+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:36:45.296578+0000","last_undegraded":"2024-02-10T08:36:45.296578+0000","last_fullsized":"2024-02-10T08:36:45.296578+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.902029+0000","last_change":"2024-02-10T08:36:45.341586+0000","last_active":"2024-02-10T08:36:45.902029+0000","last_peered":"2024-02-10T08:36:45.902029+0000","last_clean":"2024-02-10T08:36:45.902029+0000","last_became_active":"2024-02-10T08:36:37.127153+0000","last_became_peered":"2024-02-10T08:36:37.127153+0000","last_unstale":"2024-02-10T08:36:45.902029+0000","last_undegraded":"2024-02-10T08:36:45.902029+0000","last_fullsized":"2024-02-10T08:36:45.902029+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.2","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292181+0000","last_change":"2024-02-10T08:36:37.074211+0000","last_active":"2024-02-10T08:36:45.292181+0000","last_peered":"2024-02-10T08:36:45.292181+0000","last_clean":"2024-02-10T08:36:45.292181+0000","last_became_active":"2024-02-10T08:36:37.073890+0000","last_became_peered":"2024-02-10T08:36:37.073890+0000","last_unstale":"2024-02-10T08:36:45.292181+0000","last_undegraded":"2024-02-10T08:36:45.292181+0000","last_fullsized":"2024-02-10T08:36:45.292181+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.315679+0000","last_change":"2024-02-10T08:36:37.157054+0000","last_active":"2024-02-10T08:36:45.315679+0000","last_peered":"2024-02-10T08:36:45.315679+0000","last_clean":"2024-02-10T08:36:45.315679+0000","last_became_active":"2024-02-10T08:36:37.156550+0000","last_became_peered":"2024-02-10T08:36:37.156550+0000","last_unstale":"2024-02-10T08:36:45.315679+0000","last_undegraded":"2024-02-10T08:36:45.315679+0000","last_fullsized":"2024-02-10T08:36:45.315679+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.333024+0000","last_change":"2024-02-10T08:35:45.543177+0000","last_active":"2024-02-10T08:36:45.333024+0000","last_peered":"2024-02-10T08:36:45.333024+0000","last_clean":"2024-02-10T08:36:45.333024+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:36:45.333024+0000","last_undegraded":"2024-02-10T08:36:45.333024+0000","last_fullsized":"2024-02-10T08:36:45.333024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:45.543110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:45.543110+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.301718+0000","last_change":"2024-02-10T08:36:22.073074+0000","last_active":"2024-02-10T08:36:45.301718+0000","last_peered":"2024-02-10T08:36:45.301718+0000","last_clean":"2024-02-10T08:36:45.301718+0000","last_became_active":"2024-02-10T08:33:18.107580+0000","last_became_peered":"2024-02-10T08:33:18.107580+0000","last_unstale":"2024-02-10T08:36:45.301718+0000","last_undegraded":"2024-02-10T08:36:45.301718+0000","last_fullsized":"2024-02-10T08:36:45.301718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.072982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285505+0000","last_change":"2024-02-10T08:36:25.503414+0000","last_active":"2024-02-10T08:36:45.285505+0000","last_peered":"2024-02-10T08:36:45.285505+0000","last_clean":"2024-02-10T08:36:45.285505+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:36:45.285505+0000","last_undegraded":"2024-02-10T08:36:45.285505+0000","last_fullsized":"2024-02-10T08:36:45.285505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.503351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.503351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.280761+0000","last_change":"2024-02-10T08:36:37.124706+0000","last_active":"2024-02-10T08:36:45.280761+0000","last_peered":"2024-02-10T08:36:45.280761+0000","last_clean":"2024-02-10T08:36:45.280761+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:36:45.280761+0000","last_undegraded":"2024-02-10T08:36:45.280761+0000","last_fullsized":"2024-02-10T08:36:45.280761+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.c","version":"103'1","reported_seq":32,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.907558+0000","last_change":"2024-02-10T08:36:45.308137+0000","last_active":"2024-02-10T08:36:45.907558+0000","last_peered":"2024-02-10T08:36:45.907558+0000","last_clean":"2024-02-10T08:36:45.907558+0000","last_became_active":"2024-02-10T08:36:37.170094+0000","last_became_peered":"2024-02-10T08:36:37.170094+0000","last_unstale":"2024-02-10T08:36:45.907558+0000","last_undegraded":"2024-02-10T08:36:45.907558+0000","last_fullsized":"2024-02-10T08:36:45.907558+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.1","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281735+0000","last_change":"2024-02-10T08:36:37.063791+0000","last_active":"2024-02-10T08:36:45.281735+0000","last_peered":"2024-02-10T08:36:45.281735+0000","last_clean":"2024-02-10T08:36:45.281735+0000","last_became_active":"2024-02-10T08:36:37.063485+0000","last_became_peered":"2024-02-10T08:36:37.063485+0000","last_unstale":"2024-02-10T08:36:45.281735+0000","last_undegraded":"2024-02-10T08:36:45.281735+0000","last_fullsized":"2024-02-10T08:36:45.281735+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.10","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299480+0000","last_change":"2024-02-10T08:36:37.021161+0000","last_active":"2024-02-10T08:36:45.299480+0000","last_peered":"2024-02-10T08:36:45.299480+0000","last_clean":"2024-02-10T08:36:45.299480+0000","last_became_active":"2024-02-10T08:36:37.017583+0000","last_became_peered":"2024-02-10T08:36:37.017583+0000","last_unstale":"2024-02-10T08:36:45.299480+0000","last_undegraded":"2024-02-10T08:36:45.299480+0000","last_fullsized":"2024-02-10T08:36:45.299480+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":98,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289834+0000","last_change":"2024-02-10T08:36:13.651116+0000","last_active":"2024-02-10T08:36:45.289834+0000","last_peered":"2024-02-10T08:36:45.289834+0000","last_clean":"2024-02-10T08:36:45.289834+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:36:45.289834+0000","last_undegraded":"2024-02-10T08:36:45.289834+0000","last_fullsized":"2024-02-10T08:36:45.289834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:36:13.650984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.650984+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.329405+0000","last_change":"2024-02-10T08:36:34.837568+0000","last_active":"2024-02-10T08:36:45.329405+0000","last_peered":"2024-02-10T08:36:45.329405+0000","last_clean":"2024-02-10T08:36:45.329405+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:36:45.329405+0000","last_undegraded":"2024-02-10T08:36:45.329405+0000","last_fullsized":"2024-02-10T08:36:45.329405+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:34.837483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:34.837483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.289278+0000","last_change":"2024-02-10T08:35:17.895268+0000","last_active":"2024-02-10T08:36:45.289278+0000","last_peered":"2024-02-10T08:36:45.289278+0000","last_clean":"2024-02-10T08:36:45.289278+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:36:45.289278+0000","last_undegraded":"2024-02-10T08:36:45.289278+0000","last_fullsized":"2024-02-10T08:36:45.289278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:17.895163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:17.895163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314510+0000","last_change":"2024-02-10T08:36:37.166424+0000","last_active":"2024-02-10T08:36:45.314510+0000","last_peered":"2024-02-10T08:36:45.314510+0000","last_clean":"2024-02-10T08:36:45.314510+0000","last_became_active":"2024-02-10T08:36:37.165494+0000","last_became_peered":"2024-02-10T08:36:37.165494+0000","last_unstale":"2024-02-10T08:36:45.314510+0000","last_undegraded":"2024-02-10T08:36:45.314510+0000","last_fullsized":"2024-02-10T08:36:45.314510+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.953702+0000","last_change":"2024-02-10T08:36:45.332064+0000","last_active":"2024-02-10T08:36:45.953702+0000","last_peered":"2024-02-10T08:36:45.953702+0000","last_clean":"2024-02-10T08:36:45.953702+0000","last_became_active":"2024-02-10T08:36:37.117150+0000","last_became_peered":"2024-02-10T08:36:37.117150+0000","last_unstale":"2024-02-10T08:36:45.953702+0000","last_undegraded":"2024-02-10T08:36:45.953702+0000","last_fullsized":"2024-02-10T08:36:45.953702+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"8.11","version":"105'10","reported_seq":30,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292997+0000","last_change":"2024-02-10T08:36:37.056718+0000","last_active":"2024-02-10T08:36:45.292997+0000","last_peered":"2024-02-10T08:36:45.292997+0000","last_clean":"2024-02-10T08:36:45.292997+0000","last_became_active":"2024-02-10T08:36:37.056435+0000","last_became_peered":"2024-02-10T08:36:37.056435+0000","last_unstale":"2024-02-10T08:36:45.292997+0000","last_undegraded":"2024-02-10T08:36:45.292997+0000","last_fullsized":"2024-02-10T08:36:45.292997+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165892,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":95,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.307203+0000","last_change":"2024-02-10T08:36:20.047350+0000","last_active":"2024-02-10T08:36:45.307203+0000","last_peered":"2024-02-10T08:36:45.307203+0000","last_clean":"2024-02-10T08:36:45.307203+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:36:45.307203+0000","last_undegraded":"2024-02-10T08:36:45.307203+0000","last_fullsized":"2024-02-10T08:36:45.307203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:36:20.047272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.047272+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316753+0000","last_change":"2024-02-10T08:36:15.045147+0000","last_active":"2024-02-10T08:36:45.316753+0000","last_peered":"2024-02-10T08:36:45.316753+0000","last_clean":"2024-02-10T08:36:45.316753+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:36:45.316753+0000","last_undegraded":"2024-02-10T08:36:45.316753+0000","last_fullsized":"2024-02-10T08:36:45.316753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.045044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.045044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.326113+0000","last_change":"2024-02-10T08:36:27.054202+0000","last_active":"2024-02-10T08:36:45.326113+0000","last_peered":"2024-02-10T08:36:45.326113+0000","last_clean":"2024-02-10T08:36:45.326113+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:36:45.326113+0000","last_undegraded":"2024-02-10T08:36:45.326113+0000","last_fullsized":"2024-02-10T08:36:45.326113+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.054145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.054145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.291695+0000","last_change":"2024-02-10T08:36:37.038364+0000","last_active":"2024-02-10T08:36:45.291695+0000","last_peered":"2024-02-10T08:36:45.291695+0000","last_clean":"2024-02-10T08:36:45.291695+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:36:45.291695+0000","last_undegraded":"2024-02-10T08:36:45.291695+0000","last_fullsized":"2024-02-10T08:36:45.291695+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.909840+0000","last_change":"2024-02-10T08:36:45.311963+0000","last_active":"2024-02-10T08:36:45.909840+0000","last_peered":"2024-02-10T08:36:45.909840+0000","last_clean":"2024-02-10T08:36:45.909840+0000","last_became_active":"2024-02-10T08:36:37.165380+0000","last_became_peered":"2024-02-10T08:36:37.165380+0000","last_unstale":"2024-02-10T08:36:45.909840+0000","last_undegraded":"2024-02-10T08:36:45.909840+0000","last_fullsized":"2024-02-10T08:36:45.909840+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.0","version":"103'5","reported_seq":23,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327633+0000","last_change":"2024-02-10T08:36:37.017992+0000","last_active":"2024-02-10T08:36:45.327633+0000","last_peered":"2024-02-10T08:36:45.327633+0000","last_clean":"2024-02-10T08:36:45.327633+0000","last_became_active":"2024-02-10T08:36:37.016870+0000","last_became_peered":"2024-02-10T08:36:37.016870+0000","last_unstale":"2024-02-10T08:36:45.327633+0000","last_undegraded":"2024-02-10T08:36:45.327633+0000","last_fullsized":"2024-02-10T08:36:45.327633+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327981+0000","last_change":"2024-02-10T08:36:37.005181+0000","last_active":"2024-02-10T08:36:45.327981+0000","last_peered":"2024-02-10T08:36:45.327981+0000","last_clean":"2024-02-10T08:36:45.327981+0000","last_became_active":"2024-02-10T08:36:37.001706+0000","last_became_peered":"2024-02-10T08:36:37.001706+0000","last_unstale":"2024-02-10T08:36:45.327981+0000","last_undegraded":"2024-02-10T08:36:45.327981+0000","last_fullsized":"2024-02-10T08:36:45.327981+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310676+0000","last_change":"2024-02-10T08:36:37.085781+0000","last_active":"2024-02-10T08:36:45.310676+0000","last_peered":"2024-02-10T08:36:45.310676+0000","last_clean":"2024-02-10T08:36:45.310676+0000","last_became_active":"2024-02-10T08:36:37.082091+0000","last_became_peered":"2024-02-10T08:36:37.082091+0000","last_unstale":"2024-02-10T08:36:45.310676+0000","last_undegraded":"2024-02-10T08:36:45.310676+0000","last_fullsized":"2024-02-10T08:36:45.310676+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":89,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281221+0000","last_change":"2024-02-10T08:36:24.764092+0000","last_active":"2024-02-10T08:36:45.281221+0000","last_peered":"2024-02-10T08:36:45.281221+0000","last_clean":"2024-02-10T08:36:45.281221+0000","last_became_active":"2024-02-10T08:33:20.115281+0000","last_became_peered":"2024-02-10T08:33:20.115281+0000","last_unstale":"2024-02-10T08:36:45.281221+0000","last_undegraded":"2024-02-10T08:36:45.281221+0000","last_fullsized":"2024-02-10T08:36:45.281221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:24.764020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.764020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.299578+0000","last_change":"2024-02-10T08:36:30.157585+0000","last_active":"2024-02-10T08:36:45.299578+0000","last_peered":"2024-02-10T08:36:45.299578+0000","last_clean":"2024-02-10T08:36:45.299578+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:36:45.299578+0000","last_undegraded":"2024-02-10T08:36:45.299578+0000","last_fullsized":"2024-02-10T08:36:45.299578+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.157473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.157473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":68,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.314287+0000","last_change":"2024-02-10T08:36:20.877370+0000","last_active":"2024-02-10T08:36:45.314287+0000","last_peered":"2024-02-10T08:36:45.314287+0000","last_clean":"2024-02-10T08:36:45.314287+0000","last_became_active":"2024-02-10T08:33:24.123937+0000","last_became_peered":"2024-02-10T08:33:24.123937+0000","last_unstale":"2024-02-10T08:36:45.314287+0000","last_undegraded":"2024-02-10T08:36:45.314287+0000","last_fullsized":"2024-02-10T08:36:45.314287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.877268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.877268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.311617+0000","last_change":"2024-02-10T08:36:36.992978+0000","last_active":"2024-02-10T08:36:45.311617+0000","last_peered":"2024-02-10T08:36:45.311617+0000","last_clean":"2024-02-10T08:36:45.311617+0000","last_became_active":"2024-02-10T08:36:36.991506+0000","last_became_peered":"2024-02-10T08:36:36.991506+0000","last_unstale":"2024-02-10T08:36:45.311617+0000","last_undegraded":"2024-02-10T08:36:45.311617+0000","last_fullsized":"2024-02-10T08:36:45.311617+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.909371+0000","last_change":"2024-02-10T08:36:45.283803+0000","last_active":"2024-02-10T08:36:45.909371+0000","last_peered":"2024-02-10T08:36:45.909371+0000","last_clean":"2024-02-10T08:36:45.909371+0000","last_became_active":"2024-02-10T08:36:37.040895+0000","last_became_peered":"2024-02-10T08:36:37.040895+0000","last_unstale":"2024-02-10T08:36:45.909371+0000","last_undegraded":"2024-02-10T08:36:45.909371+0000","last_fullsized":"2024-02-10T08:36:45.909371+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.6","version":"103'9","reported_seq":27,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.311436+0000","last_change":"2024-02-10T08:36:37.066456+0000","last_active":"2024-02-10T08:36:45.311436+0000","last_peered":"2024-02-10T08:36:45.311436+0000","last_clean":"2024-02-10T08:36:45.311436+0000","last_became_active":"2024-02-10T08:36:37.065893+0000","last_became_peered":"2024-02-10T08:36:37.065893+0000","last_unstale":"2024-02-10T08:36:45.311436+0000","last_undegraded":"2024-02-10T08:36:45.311436+0000","last_fullsized":"2024-02-10T08:36:45.311436+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.17","version":"103'1","reported_seq":21,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305529+0000","last_change":"2024-02-10T08:36:37.091857+0000","last_active":"2024-02-10T08:36:45.305529+0000","last_peered":"2024-02-10T08:36:45.305529+0000","last_clean":"2024-02-10T08:36:45.305529+0000","last_became_active":"2024-02-10T08:36:37.090503+0000","last_became_peered":"2024-02-10T08:36:37.090503+0000","last_unstale":"2024-02-10T08:36:45.305529+0000","last_undegraded":"2024-02-10T08:36:45.305529+0000","last_fullsized":"2024-02-10T08:36:45.305529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.285243+0000","last_change":"2024-02-10T08:36:32.709476+0000","last_active":"2024-02-10T08:36:45.285243+0000","last_peered":"2024-02-10T08:36:45.285243+0000","last_clean":"2024-02-10T08:36:45.285243+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:36:45.285243+0000","last_undegraded":"2024-02-10T08:36:45.285243+0000","last_fullsized":"2024-02-10T08:36:45.285243+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:36:32.709392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.709392+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305508+0000","last_change":"2024-02-10T08:36:21.107654+0000","last_active":"2024-02-10T08:36:45.305508+0000","last_peered":"2024-02-10T08:36:45.305508+0000","last_clean":"2024-02-10T08:36:45.305508+0000","last_became_active":"2024-02-10T08:33:18.110388+0000","last_became_peered":"2024-02-10T08:33:18.110388+0000","last_unstale":"2024-02-10T08:36:45.305508+0000","last_undegraded":"2024-02-10T08:36:45.305508+0000","last_fullsized":"2024-02-10T08:36:45.305508+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.107579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.107579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":26,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.297525+0000","last_change":"2024-02-10T08:36:24.066925+0000","last_active":"2024-02-10T08:36:45.297525+0000","last_peered":"2024-02-10T08:36:45.297525+0000","last_clean":"2024-02-10T08:36:45.297525+0000","last_became_active":"2024-02-10T08:36:24.066557+0000","last_became_peered":"2024-02-10T08:36:24.066557+0000","last_unstale":"2024-02-10T08:36:45.297525+0000","last_undegraded":"2024-02-10T08:36:45.297525+0000","last_fullsized":"2024-02-10T08:36:45.297525+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.936114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.936114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":64,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.292815+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:36:45.292815+0000","last_peered":"2024-02-10T08:36:45.292815+0000","last_clean":"2024-02-10T08:36:45.292815+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:36:45.292815+0000","last_undegraded":"2024-02-10T08:36:45.292815+0000","last_fullsized":"2024-02-10T08:36:45.292815+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.327634+0000","last_change":"2024-02-10T08:36:37.064446+0000","last_active":"2024-02-10T08:36:45.327634+0000","last_peered":"2024-02-10T08:36:45.327634+0000","last_clean":"2024-02-10T08:36:45.327634+0000","last_became_active":"2024-02-10T08:36:37.063451+0000","last_became_peered":"2024-02-10T08:36:37.063451+0000","last_unstale":"2024-02-10T08:36:45.327634+0000","last_undegraded":"2024-02-10T08:36:45.327634+0000","last_fullsized":"2024-02-10T08:36:45.327634+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.934573+0000","last_change":"2024-02-10T08:36:45.300823+0000","last_active":"2024-02-10T08:36:45.934573+0000","last_peered":"2024-02-10T08:36:45.934573+0000","last_clean":"2024-02-10T08:36:45.934573+0000","last_became_active":"2024-02-10T08:36:36.996164+0000","last_became_peered":"2024-02-10T08:36:36.996164+0000","last_unstale":"2024-02-10T08:36:45.934573+0000","last_undegraded":"2024-02-10T08:36:45.934573+0000","last_fullsized":"2024-02-10T08:36:45.934573+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.5","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.306803+0000","last_change":"2024-02-10T08:36:37.078717+0000","last_active":"2024-02-10T08:36:45.306803+0000","last_peered":"2024-02-10T08:36:45.306803+0000","last_clean":"2024-02-10T08:36:45.306803+0000","last_became_active":"2024-02-10T08:36:37.077729+0000","last_became_peered":"2024-02-10T08:36:37.077729+0000","last_unstale":"2024-02-10T08:36:45.306803+0000","last_undegraded":"2024-02-10T08:36:45.306803+0000","last_fullsized":"2024-02-10T08:36:45.306803+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"103'1","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.288074+0000","last_change":"2024-02-10T08:36:37.090336+0000","last_active":"2024-02-10T08:36:45.288074+0000","last_peered":"2024-02-10T08:36:45.288074+0000","last_clean":"2024-02-10T08:36:45.288074+0000","last_became_active":"2024-02-10T08:36:37.089875+0000","last_became_peered":"2024-02-10T08:36:37.089875+0000","last_unstale":"2024-02-10T08:36:45.288074+0000","last_undegraded":"2024-02-10T08:36:45.288074+0000","last_fullsized":"2024-02-10T08:36:45.288074+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279568+0000","last_change":"2024-02-10T08:35:04.110147+0000","last_active":"2024-02-10T08:36:45.279568+0000","last_peered":"2024-02-10T08:36:45.279568+0000","last_clean":"2024-02-10T08:36:45.279568+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:36:45.279568+0000","last_undegraded":"2024-02-10T08:36:45.279568+0000","last_fullsized":"2024-02-10T08:36:45.279568+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:35:04.110053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.110053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.310922+0000","last_change":"2024-02-10T08:36:26.014750+0000","last_active":"2024-02-10T08:36:45.310922+0000","last_peered":"2024-02-10T08:36:45.310922+0000","last_clean":"2024-02-10T08:36:45.310922+0000","last_became_active":"2024-02-10T08:33:18.114558+0000","last_became_peered":"2024-02-10T08:33:18.114558+0000","last_unstale":"2024-02-10T08:36:45.310922+0000","last_undegraded":"2024-02-10T08:36:45.310922+0000","last_fullsized":"2024-02-10T08:36:45.310922+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.014690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.014690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":28,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.295514+0000","last_change":"2024-02-10T08:36:23.670233+0000","last_active":"2024-02-10T08:36:45.295514+0000","last_peered":"2024-02-10T08:36:45.295514+0000","last_clean":"2024-02-10T08:36:45.295514+0000","last_became_active":"2024-02-10T08:36:23.669768+0000","last_became_peered":"2024-02-10T08:36:23.669768+0000","last_unstale":"2024-02-10T08:36:45.295514+0000","last_undegraded":"2024-02-10T08:36:45.295514+0000","last_fullsized":"2024-02-10T08:36:45.295514+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:18.421261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:18.421261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":69,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.281128+0000","last_change":"2024-02-10T08:36:14.747640+0000","last_active":"2024-02-10T08:36:45.281128+0000","last_peered":"2024-02-10T08:36:45.281128+0000","last_clean":"2024-02-10T08:36:45.281128+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:36:45.281128+0000","last_undegraded":"2024-02-10T08:36:45.281128+0000","last_fullsized":"2024-02-10T08:36:45.281128+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.747521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.747521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.309725+0000","last_change":"2024-02-10T08:36:37.115544+0000","last_active":"2024-02-10T08:36:45.309725+0000","last_peered":"2024-02-10T08:36:45.309725+0000","last_clean":"2024-02-10T08:36:45.309725+0000","last_became_active":"2024-02-10T08:36:37.114661+0000","last_became_peered":"2024-02-10T08:36:37.114661+0000","last_unstale":"2024-02-10T08:36:45.309725+0000","last_undegraded":"2024-02-10T08:36:45.309725+0000","last_fullsized":"2024-02-10T08:36:45.309725+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.913578+0000","last_change":"2024-02-10T08:36:45.314245+0000","last_active":"2024-02-10T08:36:45.913578+0000","last_peered":"2024-02-10T08:36:45.913578+0000","last_clean":"2024-02-10T08:36:45.913578+0000","last_became_active":"2024-02-10T08:36:37.103545+0000","last_became_peered":"2024-02-10T08:36:37.103545+0000","last_unstale":"2024-02-10T08:36:45.913578+0000","last_undegraded":"2024-02-10T08:36:45.913578+0000","last_fullsized":"2024-02-10T08:36:45.913578+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"25.4","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.313491+0000","last_change":"2024-02-10T08:36:37.164878+0000","last_active":"2024-02-10T08:36:45.313491+0000","last_peered":"2024-02-10T08:36:45.313491+0000","last_clean":"2024-02-10T08:36:45.313491+0000","last_became_active":"2024-02-10T08:36:37.163315+0000","last_became_peered":"2024-02-10T08:36:37.163315+0000","last_unstale":"2024-02-10T08:36:45.313491+0000","last_undegraded":"2024-02-10T08:36:45.313491+0000","last_fullsized":"2024-02-10T08:36:45.313491+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"104'5","reported_seq":28,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.305329+0000","last_change":"2024-02-10T08:36:37.044078+0000","last_active":"2024-02-10T08:36:45.305329+0000","last_peered":"2024-02-10T08:36:45.305329+0000","last_clean":"2024-02-10T08:36:45.305329+0000","last_became_active":"2024-02-10T08:36:37.043640+0000","last_became_peered":"2024-02-10T08:36:37.043640+0000","last_unstale":"2024-02-10T08:36:45.305329+0000","last_undegraded":"2024-02-10T08:36:45.305329+0000","last_fullsized":"2024-02-10T08:36:45.305329+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.302044+0000","last_change":"2024-02-10T08:35:15.479032+0000","last_active":"2024-02-10T08:36:45.302044+0000","last_peered":"2024-02-10T08:36:45.302044+0000","last_clean":"2024-02-10T08:36:45.302044+0000","last_became_active":"2024-02-10T08:33:20.114272+0000","last_became_peered":"2024-02-10T08:33:20.114272+0000","last_unstale":"2024-02-10T08:36:45.302044+0000","last_undegraded":"2024-02-10T08:36:45.302044+0000","last_fullsized":"2024-02-10T08:36:45.302044+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.330228+0000","last_change":"2024-02-10T08:36:17.894440+0000","last_active":"2024-02-10T08:36:45.330228+0000","last_peered":"2024-02-10T08:36:45.330228+0000","last_clean":"2024-02-10T08:36:45.330228+0000","last_became_active":"2024-02-10T08:33:18.105185+0000","last_became_peered":"2024-02-10T08:33:18.105185+0000","last_unstale":"2024-02-10T08:36:45.330228+0000","last_undegraded":"2024-02-10T08:36:45.330228+0000","last_fullsized":"2024-02-10T08:36:45.330228+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.894338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.894338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":26,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.279485+0000","last_change":"2024-02-10T08:36:23.933121+0000","last_active":"2024-02-10T08:36:45.279485+0000","last_peered":"2024-02-10T08:36:45.279485+0000","last_clean":"2024-02-10T08:36:45.279485+0000","last_became_active":"2024-02-10T08:36:23.932769+0000","last_became_peered":"2024-02-10T08:36:23.932769+0000","last_unstale":"2024-02-10T08:36:45.279485+0000","last_undegraded":"2024-02-10T08:36:45.279485+0000","last_fullsized":"2024-02-10T08:36:45.279485+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:23.876871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:23.876871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":63,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.316460+0000","last_change":"2024-02-10T08:35:00.420536+0000","last_active":"2024-02-10T08:36:45.316460+0000","last_peered":"2024-02-10T08:36:45.316460+0000","last_clean":"2024-02-10T08:36:45.316460+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:36:45.316460+0000","last_undegraded":"2024-02-10T08:36:45.316460+0000","last_fullsized":"2024-02-10T08:36:45.316460+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:00.420443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:00.420443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.328575+0000","last_change":"2024-02-10T08:36:37.083207+0000","last_active":"2024-02-10T08:36:45.328575+0000","last_peered":"2024-02-10T08:36:45.328575+0000","last_clean":"2024-02-10T08:36:45.328575+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:36:45.328575+0000","last_undegraded":"2024-02-10T08:36:45.328575+0000","last_fullsized":"2024-02-10T08:36:45.328575+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-10T08:36:45.915352+0000","last_change":"2024-02-10T08:36:45.303183+0000","last_active":"2024-02-10T08:36:45.915352+0000","last_peered":"2024-02-10T08:36:45.915352+0000","last_clean":"2024-02-10T08:36:45.915352+0000","last_became_active":"2024-02-10T08:36:37.082214+0000","last_became_peered":"2024-02-10T08:36:37.082214+0000","last_unstale":"2024-02-10T08:36:45.915352+0000","last_undegraded":"2024-02-10T08:36:45.915352+0000","last_fullsized":"2024-02-10T08:36:45.915352+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]}],"pool_stats":[{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":59,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":58,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":57,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":56,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":52,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":62,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":27,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":81,"acting":81,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":8192,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":150998978,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1043,"num_write_kb":152978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453267456,"data_stored":452996934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1053,"ondisk_log_size":1053,"up":96,"acting":96,"num_store_stats":8},{"poolid":46,"num_pg":32,"stat_sum":{"num_bytes":651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299436,"num_pgs":207,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":380136,"kb_used_data":74848,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93365016,"statfs":{"total":95995035648,"available":95605776384,"internally_reserved":0,"allocated":76644352,"data_stored":76095611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.50900000000000001,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":22.143000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.307,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":8.7850000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":7.593},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.02,"5min":1.054,"15min":1.054},"last":7.7759999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.7130000000000001,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":5.9850000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":19.369}]},{"osd":3,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":6.9829999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":22.102}]},{"osd":4,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44,"15min":0.42999999999999999},"min":{"1min":0.312,"5min":0.311,"15min":0.311},"max":{"1min":0.67000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":8.827},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44700000000000001,"15min":0.433},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":13.646000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.44600000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":19.326000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.497,"15min":0.48399999999999999},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":1.6220000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.378,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":22.687000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":1.7450000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561074,"num_pgs":196,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355540,"kb_used_data":50252,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389612,"statfs":{"total":95995035648,"available":95630962688,"internally_reserved":0,"allocated":51458048,"data_stored":50887027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.36299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47399999999999998,"15min":0.47299999999999998},"min":{"1min":0.35699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.315,"5min":0.248,"15min":0.248},"max":{"1min":0.60899999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.501,"15min":0.501},"min":{"1min":0.39400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66600000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.372}]},{"osd":3,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.40200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.34699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39400000000000002,"15min":0.378},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60799999999999998,"5min":0.623,"15min":0.623},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.38,"15min":0.35799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.63},"last":0.38600000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.39600000000000002,"15min":0.372},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.436,"15min":0.40400000000000003},"min":{"1min":0.32600000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67100000000000004,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.47199999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:36:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47599999999999998,"15min":0.46100000000000002},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.497},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.443,"5min":0.317,"15min":0.317},"max":{"1min":0.65300000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.44600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":250,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376044,"kb_used_data":74852,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93369108,"statfs":{"total":95995035648,"available":95609966592,"internally_reserved":0,"allocated":76648448,"data_stored":76036133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.20799999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46600000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":1.8220000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41099999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36099999999999999,"15min":0.371},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.28299999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:36:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38600000000000001,"15min":0.41799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.71399999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.41999999999999998,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.44800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.73899999999999999,"5min":1.294,"15min":1.294},"last":0.35899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:36:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.81799999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.73199999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.47099999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.69699999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.45900000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":254,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326888,"kb_used_data":25696,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93418264,"statfs":{"total":95995035648,"available":95660302336,"internally_reserved":0,"allocated":26312704,"data_stored":25712606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34899999999999998,"15min":0.30499999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.35599999999999998,"15min":0.31},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.48299999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.374},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39200000000000002,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56299999999999994,"5min":0.624,"15min":0.624},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40300000000000002,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.27900000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.313},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79400000000000004,"5min":0.878,"15min":0.878},"last":0.40100000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72699999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.35099999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.875,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.373},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.42299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.747,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42299999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83699999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.432},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.753,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48499999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326860,"kb_used_data":25668,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93418292,"statfs":{"total":95995035648,"available":95660331008,"internally_reserved":0,"allocated":26284032,"data_stored":25705802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.371,"15min":0.35899999999999999},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.245},{"interface":"front","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.253}]},{"osd":1,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.214},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.46999999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.315},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40400000000000003,"15min":0.42799999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.27300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.623}]},{"osd":5,"last update":"Sat Feb 10 08:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.441,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.53400000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82499999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82099999999999995,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80100000000000005,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.629},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.497,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.54900000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":246,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376020,"kb_used_data":74828,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93369132,"statfs":{"total":95995035648,"available":95609991168,"internally_reserved":0,"allocated":76623872,"data_stored":76043410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.47899999999999998,"15min":0.40999999999999998},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.744,"5min":14.744,"15min":14.744},"last":0.38},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.2859999999999996,"5min":5.2859999999999996,"15min":5.2859999999999996},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":0.52100000000000002,"15min":0.46100000000000002},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.634,"5min":16.634,"15min":16.634},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.3740000000000001,"5min":0.57599999999999996,"15min":0.49099999999999999},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":12.438000000000001,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":0.63600000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":16.536000000000001,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.508,"5min":0.60599999999999998,"15min":0.48299999999999998},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":23.294,"5min":23.294,"15min":23.294},"last":0.40899999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79500000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.44700000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":8.5670000000000002,"5min":8.5670000000000002,"15min":8.5670000000000002},"last":0.36399999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:35:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.42699999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.39000000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:36:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71599999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.73299999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":223,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335040,"kb_used_data":25656,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93410112,"statfs":{"total":95995035648,"available":95651954688,"internally_reserved":0,"allocated":26271744,"data_stored":25706736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.35199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56599999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":19.940999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.28100000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.221}]},{"osd":1,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.29599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":1,"15min":1},"last":4.0830000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51700000000000002,"15min":0.54200000000000004},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":1.1439999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63800000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":5.4960000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":4.7670000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60899999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":8.2309999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72099999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":25.27}]},{"osd":5,"last update":"Sat Feb 10 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.42299999999999999,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":1.6120000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67500000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41599999999999998,"15min":0.39900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66500000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":13.342000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.34000000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71099999999999997,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":2.3580000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.36899999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68500000000000005,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":1.883}]}]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":238,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408828,"kb_used_data":99444,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93336324,"statfs":{"total":95995035648,"available":95576395776,"internally_reserved":0,"allocated":101830656,"data_stored":101207814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.501},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55000000000000004,"15min":0.55900000000000005},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61399999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.73699999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55100000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44400000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67400000000000004,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.44800000000000001,"15min":0.40200000000000002},"min":{"1min":0.34499999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.439,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69899999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72299999999999998,"5min":1.081,"15min":1.081},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.501,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25198592,"data_stored":25166896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75530240,"data_stored":75497525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25198592,"data_stored":25168667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75522048,"data_stored":75497521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25194496,"data_stored":25165879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100666239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50364416,"data_stored":50332782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":75542528,"data_stored":75501425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:36:53.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:53.496 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:53.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:51.667083+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:51.667138+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:52.061031+0000 mgr.y (mgr.14553) 156 : cluster [DBG] pgmap v152: 451 pgs: 14 creating+activating, 2 peering, 32 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:52.857148+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.008625+0000 mon.a (mon.0) 999 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.008712+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.013505+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.107:0/110610435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.021279+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.107:0/1465655171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:53.023854+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:53.023928+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.031208+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.084752+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:53.346317+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.348323+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.348576+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.348785+0000 mon.a (mon.0) 1004 : audit [INF] from='client.26956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.349221+0000 mon.a (mon.0) 1005 : audit [INF] from='client.26977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.355173+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:51.667083+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:51.667138+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:52.061031+0000 mgr.y (mgr.14553) 156 : cluster [DBG] pgmap v152: 451 pgs: 14 creating+activating, 2 peering, 32 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:52.857148+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.008625+0000 mon.a (mon.0) 999 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.008712+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.013505+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.107:0/110610435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.021279+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.107:0/1465655171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:53.023854+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:53.023928+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.031208+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.084752+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:53.346317+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.348323+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.348576+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.348785+0000 mon.a (mon.0) 1004 : audit [INF] from='client.26956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.349221+0000 mon.a (mon.0) 1005 : audit [INF] from='client.26977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.355173+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:53.929 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:51.667083+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:51.667138+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:52.061031+0000 mgr.y (mgr.14553) 156 : cluster [DBG] pgmap v152: 451 pgs: 14 creating+activating, 2 peering, 32 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:52.857148+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.008625+0000 mon.a (mon.0) 999 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.008712+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.013505+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.107:0/110610435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.021279+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.107:0/1465655171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:53.023854+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:53.023928+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.031208+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2054498539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.084752+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:53.346317+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.348323+0000 mon.a (mon.0) 1002 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]: dispatch 2024-02-10T08:36:53.930 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.348576+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.931 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.348785+0000 mon.a (mon.0) 1004 : audit [INF] from='client.26956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.931 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.349221+0000 mon.a (mon.0) 1005 : audit [INF] from='client.26977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:53.931 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.355173+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:36:53.931 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:53.949 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:53.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:54.085 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify_test_cb 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (317 ms) 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94609607020448 err -107 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (27 ms) 2024-02-10T08:36:54.086 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94609607039280 err -107 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (35 ms) 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631489 cookie 94609607058304 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (11 ms) 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631490 cookie 94609607088992 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (12 ms) 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631491 cookie 94609607088992 2024-02-10T08:36:54.087 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (9 ms) 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631488 cookie 94609607084432 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631488 cookie 94609607088992 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (12 ms) 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 442381631489 cookie 94609607084432 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 446676598786 cookie 94609607084432 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94609607084288 err -107 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_cb from 34808 notify_id 476741369859 cookie 94609607084288 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-02-10T08:36:54.088 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94609607084288 err -107 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9440 ms total) 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: watch_notify_test_cb 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (617 ms) 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (617 ms total) 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-10T08:36:54.089 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (23938 ms total) 2024-02-10T08:36:54.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.090 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-10T08:36:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:54.199 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2146 ms) 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (8 ms) 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2154 ms total) 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-10T08:36:54.200 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (60 ms) 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4070 ms) 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (9 ms) 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (20 ms) 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846601600 notify_id 502511173637 notifier_gid 25276 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (17 ms) 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846601600 notify_id 502511173632 notifier_gid 25276 2024-02-10T08:36:54.201 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (14 ms) 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846601600 notify_id 502511173638 notifier_gid 25276 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (9 ms) 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 502511173639 notifier_gid 25276 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (21 ms) 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 502511173640 notifier_gid 25276 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (12 ms) 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 502511173641 notifier_gid 25276 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (29 ms) 2024-02-10T08:36:54.202 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-10T08:36:54.203 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: trying... 2024-02-10T08:36:54.203 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 502511173633 notifier_gid 25276 2024-02-10T08:36:54.203 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: timed out 2024-02-10T08:36:54.203 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: flushing 2024-02-10T08:36:54.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:36:54.236 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.255 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:36:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:36:54.322 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:54.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:54.386 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.406 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:54.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:36:54.536 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.556 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:36:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:36:54.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:54.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:53.859017+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:54.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.033836+0000 mon.a (mon.0) 1007 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:54.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.033909+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.033959+0000 mon.a (mon.0) 1009 : audit [INF] from='client.26956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.034028+0000 mon.a (mon.0) 1010 : audit [INF] from='client.26977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.034106+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.034168+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.039567+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.048150+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.058229+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.072121+0000 mon.c (mon.2) 162 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:54.687 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.705 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:53.859017+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.033836+0000 mon.a (mon.0) 1007 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.033909+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.033959+0000 mon.a (mon.0) 1009 : audit [INF] from='client.26956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.034028+0000 mon.a (mon.0) 1010 : audit [INF] from='client.26977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.034106+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.034168+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-10T08:36:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.039567+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.048150+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.058229+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.072121+0000 mon.c (mon.2) 162 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.086398+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:54.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:36:54.835 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:54.854 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:36:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:54.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:36:54.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:36:54.983 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.086398+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:53.859017+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.033836+0000 mon.a (mon.0) 1007 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi107-112553-12"}]': finished 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.033909+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.107:0/1414958239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.033959+0000 mon.a (mon.0) 1009 : audit [INF] from='client.26956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.034028+0000 mon.a (mon.0) 1010 : audit [INF] from='client.26977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi107-112525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.034106+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.034168+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.039567+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.048150+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.058229+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.072121+0000 mon.c (mon.2) 162 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:54.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.086398+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-10T08:36:55.019 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:55.080 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:36:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:55.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:36:55.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:36:55.372 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:36:55.636 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:36:55] "GET /metrics HTTP/1.1" 200 207295 "" "Prometheus/2.33.4" 2024-02-10T08:36:55.799 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:36:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:55.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:36:55.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.806+0000 7f6e1514d700 1 -- 172.21.15.107:0/4210286646 >> v1:172.21.15.195:6789/0 conn(0x7f6e100fd9e0 legacy=0x7f6e100fddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:55.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.807+0000 7f6e1514d700 1 -- 172.21.15.107:0/4210286646 shutdown_connections 2024-02-10T08:36:55.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.807+0000 7f6e1514d700 1 -- 172.21.15.107:0/4210286646 >> 172.21.15.107:0/4210286646 conn(0x7f6e1006fe70 msgr2=0x7f6e100f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:55.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.811+0000 7f6e1514d700 1 -- 172.21.15.107:0/4210286646 shutdown_connections 2024-02-10T08:36:55.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.811+0000 7f6e1514d700 1 -- 172.21.15.107:0/4210286646 wait complete. 2024-02-10T08:36:55.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.811+0000 7f6e1514d700 1 librados: starting msgr at 2024-02-10T08:36:55.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.811+0000 7f6e1514d700 1 librados: starting objecter 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 Processor -- start 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 -- start start 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 librados: setting wanted keys 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 librados: calling monclient init 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e10115200 con 0x7f6e100fd9e0 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e101163e0 con 0x7f6e10104340 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e1514d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e101175e0 con 0x7f6e10100830 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e0e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6e10100830 0x7f6e10102c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57322/0 (socket says 172.21.15.107:57322) 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e0f59e700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6e10104340 0x7f6e101139b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54642/0 (socket says 172.21.15.107:54642) 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.812+0000 7f6e0e59c700 1 -- 172.21.15.107:0/848338837 learned_addr learned my addr 172.21.15.107:0/848338837 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 10 client.?.objecter ms_handle_connect 0x7f6e10104340 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127753555 0 0) 0x7f6e101163e0 con 0x7f6e10104340 2024-02-10T08:36:55.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df8003f60 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 10 client.?.objecter ms_handle_connect 0x7f6e10100830 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928669922 0 0) 0x7f6df8003f60 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e101163e0 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6e04001840 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003124871 0 0) 0x7f6e101175e0 con 0x7f6e10100830 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df8003f60 con 0x7f6e10100830 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1542660225 0 0) 0x7f6e101163e0 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6790/0 conn(0x7f6e10100830 legacy=0x7f6e10102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6789/0 conn(0x7f6e100fd9e0 legacy=0x7f6e10102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.813+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e101177d0 con 0x7f6e10104340 2024-02-10T08:36:55.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6e04002460 con 0x7f6e10104340 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6e04005b20 con 0x7f6e10104340 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e10117a80 con 0x7f6e10104340 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6e1514d700 10 client.29932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e10117ef0 con 0x7f6e10104340 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.814+0000 7f6e1514d700 1 librados: init done 2024-02-10T08:36:55.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6e04001f80 con 0x7f6e10104340 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(121..121 src has 50..121) v4 ==== 11533+0+0 (unknown 3065708684 0 0) 0x7f6e04079b30 con 0x7f6e10104340 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 osd_map(121..121 src has 50..121) v4 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 3 client.29932.objecter handle_osd_map got epochs [121,121] > 0 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 3 client.29932.objecter handle_osd_map decoding full epoch 121 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 10 client.29932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=122}) v3 -- 0x7f6e101163e0 con 0x7f6e10104340 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.817+0000 7f6df7fff700 20 client.29932.objecter dump_active .. 0 homeless 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.819+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6df8063df0 2024-02-10T08:36:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.819+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6e10103cf0 con 0x7f6e10104340 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.819+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v121) v1 ==== 54+0+16 (unknown 1649006044 0 2420000192) 0x7f6e0404c190 con 0x7f6e10104340 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 10 client.29932.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 1 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 1 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 0 1 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 4 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 10 client.29932.objecter _send_command 1 2024-02-10T08:36:55.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.820+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1005f780 con 0x7f6e10116a80 2024-02-10T08:36:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e10116a80 2024-02-10T08:36:55.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e1005f780 con 0x7f6e10116a80 2024-02-10T08:36:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:36:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 10 client.29932.objecter _finish_command 1 = system:0 2024-02-10T08:36:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 4 2024-02-10T08:36:55.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.825+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 0 1 2024-02-10T08:36:55.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 10 client.29932.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:55.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 2 2024-02-10T08:36:55.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 2 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e101183e0 osd=0 3 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 0 2 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 4 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 10 client.29932.objecter _send_command 2 2024-02-10T08:36:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.853+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e100f93f0 con 0x7f6e10116a80 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.857+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e100f93f0 con 0x7f6e10116a80 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.857+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.857+0000 7f6df7fff700 10 client.29932.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.857+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 4 2024-02-10T08:36:55.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.857+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 0 2 2024-02-10T08:36:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 10 client.29932.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 3 2024-02-10T08:36:55.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 1 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 4 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 10 client.29932.objecter _send_command 3 2024-02-10T08:36:55.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.863+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e100f4e10 con 0x7f6e10059ea0 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.865+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e10059ea0 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.866+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e100f4e10 con 0x7f6e10059ea0 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.866+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.866+0000 7f6df7fff700 10 client.29932.objecter _finish_command 3 = system:0 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.866+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 4 2024-02-10T08:36:55.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.866+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 1 3 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 10 client.29932.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 4 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 4 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10108e00 osd=1 3 2024-02-10T08:36:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 1 4 2024-02-10T08:36:55.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 4 2024-02-10T08:36:55.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 10 client.29932.objecter _send_command 4 2024-02-10T08:36:55.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.894+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e100f4fb0 con 0x7f6e10059ea0 2024-02-10T08:36:55.897 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e100f4fb0 con 0x7f6e10059ea0 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6df7fff700 10 client.29932.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 4 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 1 4 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 10 client.29932.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 5 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 5 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 2 5 2024-02-10T08:36:55.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 4 2024-02-10T08:36:55.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 10 client.29932.objecter _send_command 5 2024-02-10T08:36:55.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.897+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e10028800 con 0x7f6e1012e5c0 2024-02-10T08:36:55.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.898+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e1012e5c0 2024-02-10T08:36:55.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:36:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.907+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e10028800 con 0x7f6e1012e5c0 2024-02-10T08:36:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.907+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:36:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.907+0000 7f6df7fff700 10 client.29932.objecter _finish_command 5 = system:0 2024-02-10T08:36:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.907+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 4 2024-02-10T08:36:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.907+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 2 5 2024-02-10T08:36:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.062142+0000 mgr.y (mgr.14553) 157 : cluster [DBG] pgmap v155: 578 pgs: 14 creating+activating, 1 peering, 160 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-02-10T08:36:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.076560+0000 osd.3 (osd.3) 79 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.092368+0000 osd.3 (osd.3) 80 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.340664+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:54.340724+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:54.865371+0000 mon.b (mon.1) 116 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.025589+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.025731+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.025833+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:55.025901+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.238679+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.456820+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.464854+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:55.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.820331+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? v1:172.21.15.107:0/848338837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.062142+0000 mgr.y (mgr.14553) 157 : cluster [DBG] pgmap v155: 578 pgs: 14 creating+activating, 1 peering, 160 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.076560+0000 osd.3 (osd.3) 79 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.092368+0000 osd.3 (osd.3) 80 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.340664+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:54.340724+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:54.865371+0000 mon.b (mon.1) 116 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.025589+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.025731+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.025833+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-10T08:36:55.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:55.025901+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-10T08:36:55.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.238679+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:55.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.456820+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:55.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.464854+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:55.934 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.820331+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? v1:172.21.15.107:0/848338837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 10 client.29932.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 6 2024-02-10T08:36:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 6 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10059640 osd=2 3 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 2 6 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 4 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 10 client.29932.objecter _send_command 6 2024-02-10T08:36:55.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.930+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e10027d60 con 0x7f6e1012e5c0 2024-02-10T08:36:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.933+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e10027d60 con 0x7f6e1012e5c0 2024-02-10T08:36:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.933+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.933+0000 7f6df7fff700 10 client.29932.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.933+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 4 2024-02-10T08:36:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.933+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 2 6 2024-02-10T08:36:55.945 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:36:55.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 10 client.29932.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 7 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 7 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 3 7 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 4 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 10 client.29932.objecter _send_command 7 2024-02-10T08:36:55.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.934+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e100f7e20 con 0x7f6e10133250 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e10133250 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e100f7e20 con 0x7f6e10133250 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 10 client.29932.objecter _finish_command 7 = system:0 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 4 2024-02-10T08:36:55.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.935+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 3 7 2024-02-10T08:36:55.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 10 client.29932.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 8 2024-02-10T08:36:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 8 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e100f77a0 osd=3 3 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 3 8 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 4 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 10 client.29932.objecter _send_command 8 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.975+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e10134550 con 0x7f6e10133250 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.978+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e10134550 con 0x7f6e10133250 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.978+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.978+0000 7f6df7fff700 10 client.29932.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.978+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 4 2024-02-10T08:36:55.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.978+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 3 8 2024-02-10T08:36:55.992 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:55.993 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:36:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 10 client.29932.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 9 2024-02-10T08:36:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:55.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 9 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 4 9 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 4 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 10 client.29932.objecter _send_command 9 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.985+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1013b3b0 con 0x7f6e101378a0 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.987+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e101378a0 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.988+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e1013b3b0 con 0x7f6e101378a0 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.988+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.988+0000 7f6df7fff700 10 client.29932.objecter _finish_command 9 = system:0 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.988+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 4 2024-02-10T08:36:55.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:55.988+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 4 9 2024-02-10T08:36:56.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 10 client.29932.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:56.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 10 2024-02-10T08:36:56.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:56.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:56.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 10 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e10134550 osd=4 3 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 4 10 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 4 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 10 client.29932.objecter _send_command 10 2024-02-10T08:36:56.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.012+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e1013b3b0 con 0x7f6e101378a0 2024-02-10T08:36:56.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.014+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e1013b3b0 con 0x7f6e101378a0 2024-02-10T08:36:56.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.014+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:56.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.014+0000 7f6df7fff700 10 client.29932.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.014+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 4 2024-02-10T08:36:56.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.014+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 4 10 2024-02-10T08:36:56.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 10 client.29932.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 11 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 11 2024-02-10T08:36:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 5 11 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 4 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 10 client.29932.objecter _send_command 11 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1013f720 con 0x7f6e1013ba60 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.015+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e1013ba60 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.016+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e1013f720 con 0x7f6e1013ba60 2024-02-10T08:36:56.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.016+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:36:56.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.016+0000 7f6df7fff700 10 client.29932.objecter _finish_command 11 = system:0 2024-02-10T08:36:56.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.016+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 4 2024-02-10T08:36:56.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.016+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 5 11 2024-02-10T08:36:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== mon.1 v1:172.21.15.195:6789/0 10 ==== osd_map(122..122 src has 50..122) v4 ==== 2408+0+0 (unknown 3293872758 0 0) 0x7f6e0404c3c0 con 0x7f6e10104340 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 osd_map(122..122 src has 50..122) v4 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 3 client.29932.objecter handle_osd_map got epochs [122,122] > 121 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 3 client.29932.objecter handle_osd_map decoding incremental epoch 122 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 10 client.29932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=123}) v3 -- 0x7f6e10115200 con 0x7f6e10104340 2024-02-10T08:36:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.025+0000 7f6df7fff700 20 client.29932.objecter dump_active .. 0 homeless 2024-02-10T08:36:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 10 client.29932.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 12 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 12 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e1013b3b0 osd=5 3 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 5 12 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 4 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 10 client.29932.objecter _send_command 12 2024-02-10T08:36:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.039+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e1013f720 con 0x7f6e1013ba60 2024-02-10T08:36:56.056 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:36:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.052+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e1013f720 con 0x7f6e1013ba60 2024-02-10T08:36:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.052+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.052+0000 7f6df7fff700 10 client.29932.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.052+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 4 2024-02-10T08:36:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.052+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 5 12 2024-02-10T08:36:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 10 client.29932.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 13 2024-02-10T08:36:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 13 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 6 13 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 4 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 10 client.29932.objecter _send_command 13 2024-02-10T08:36:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.055+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e101408d0 con 0x7f6e101413e0 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.065+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e101413e0 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.065+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e101408d0 con 0x7f6e101413e0 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.065+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.065+0000 7f6df7fff700 10 client.29932.objecter _finish_command 13 = system:0 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.066+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 4 2024-02-10T08:36:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.066+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 6 13 2024-02-10T08:36:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 10 client.29932.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 14 2024-02-10T08:36:56.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 14 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e1013f720 osd=6 3 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 6 14 2024-02-10T08:36:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 4 2024-02-10T08:36:56.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 10 client.29932.objecter _send_command 14 2024-02-10T08:36:56.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.085+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e101408d0 con 0x7f6e101413e0 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e101408d0 con 0x7f6e101413e0 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6df7fff700 10 client.29932.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 4 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 6 14 2024-02-10T08:36:56.101 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6e1514d700 10 client.29932.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.100+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 15 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 15 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 7 15 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 4 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 10 client.29932.objecter _send_command 15 2024-02-10T08:36:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.101+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e101480f0 con 0x7f6e101445e0 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 10 client.29932.objecter ms_handle_connect 0x7f6e101445e0 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e101480f0 con 0x7f6e101445e0 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 10 client.29932.objecter _finish_command 15 = system:0 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 4 2024-02-10T08:36:56.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.121+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 7 15 2024-02-10T08:36:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 10 client.29932.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:36:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign -1 16 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 20 client.29932.objecter _get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_remove -1 16 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 20 client.29932.objecter get_session s=0x7f6e101408d0 osd=7 3 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 15 client.29932.objecter _session_command_op_assign 7 16 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 4 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 10 client.29932.objecter _send_command 16 2024-02-10T08:36:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.153+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e101480f0 con 0x7f6e101445e0 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6df7fff700 1 -- 172.21.15.107:0/848338837 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6e101480f0 con 0x7f6e101445e0 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6df7fff700 10 client.29932.objecter ms_dispatch 0x7f6e10001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6df7fff700 10 client.29932.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6df7fff700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 4 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6df7fff700 15 client.29932.objecter _session_command_op_remove 7 16 2024-02-10T08:36:56.159 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.158+0000 7f6e1514d700 10 librados: watch_flush enter 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 librados: watch_flush exit 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.0 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6801/3321596972 conn(0x7f6e10116a80 legacy=0x7f6e10116e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101183e0 osd=0 1 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.1 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6805/133516126 conn(0x7f6e10059ea0 legacy=0x7f6e10027410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10108e00 osd=1 1 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.2 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6809/3575881728 conn(0x7f6e1012e5c0 legacy=0x7f6e101309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10059640 osd=2 1 2024-02-10T08:36:56.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.3 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6813/2871825691 conn(0x7f6e10133250 legacy=0x7f6e10133670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e100f77a0 osd=3 1 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.4 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.195:6800/2499671285 conn(0x7f6e101378a0 legacy=0x7f6e10139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e10134550 osd=4 1 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.5 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.195:6804/4178321486 conn(0x7f6e1013ba60 legacy=0x7f6e1013de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013b3b0 osd=5 1 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.6 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.195:6808/1417479017 conn(0x7f6e101413e0 legacy=0x7f6e10141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e1013f720 osd=6 1 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter close_session for osd.7 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.195:6812/1934176928 conn(0x7f6e101445e0 legacy=0x7f6e101469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter put_session s=0x7f6e101408d0 osd=7 1 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 20 client.29932.objecter shutdown clearing up homeless session... 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 10 client.29932.objecter successfully canceled tick 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6df8063df0 legacy=0x7f6df80662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.159+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> v1:172.21.15.195:6789/0 conn(0x7f6e10104340 legacy=0x7f6e101139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e0f59e700 1 -- 172.21.15.107:0/848338837 reap_dead start 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 shutdown_connections 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 >> 172.21.15.107:0/848338837 conn(0x7f6e1006fe70 msgr2=0x7f6e10107270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 shutdown_connections 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.160+0000 7f6e1514d700 1 -- 172.21.15.107:0/848338837 wait complete. 2024-02-10T08:36:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:36:56.161+0000 7f6e1514d700 1 librados: shutdown 2024-02-10T08:36:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.062142+0000 mgr.y (mgr.14553) 157 : cluster [DBG] pgmap v155: 578 pgs: 14 creating+activating, 1 peering, 160 unknown, 403 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.076560+0000 osd.3 (osd.3) 79 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.092368+0000 osd.3 (osd.3) 80 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.340664+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:54.340724+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:54.865371+0000 mon.b (mon.1) 116 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.025589+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? v1:172.21.15.107:0/2323349663' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi107-112403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.025731+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.025833+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:55.025901+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.238679+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.456820+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.464854+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:36:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.820331+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? v1:172.21.15.107:0/848338837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:36:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:55.874297+0000 mon.b (mon.1) 119 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:57.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:55.886867+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.029546+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:56.029630+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.049120+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.050710+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.052913+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.107:0/3496182721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.055348+0000 mon.a (mon.0) 1026 : audit [INF] from='client.29710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:56.066520+0000 mgr.y (mgr.14553) 158 : cluster [DBG] pgmap v158: 546 pgs: 1 active+clean+snaptrim, 2 peering, 64 unknown, 479 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:55.874297+0000 mon.b (mon.1) 119 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:55.886867+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.029546+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:56.029630+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.049120+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.050710+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.052913+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.107:0/3496182721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.055348+0000 mon.a (mon.0) 1026 : audit [INF] from='client.29710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:56.066520+0000 mgr.y (mgr.14553) 158 : cluster [DBG] pgmap v158: 546 pgs: 1 active+clean+snaptrim, 2 peering, 64 unknown, 479 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-02-10T08:36:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:55.874297+0000 mon.b (mon.1) 119 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:55.886867+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:36:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.029546+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:56.029630+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.049120+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.050710+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]: dispatch 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.052913+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.107:0/3496182721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.055348+0000 mon.a (mon.0) 1026 : audit [INF] from='client.29710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:56.066520+0000 mgr.y (mgr.14553) 158 : cluster [DBG] pgmap v158: 546 pgs: 1 active+clean+snaptrim, 2 peering, 64 unknown, 479 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-02-10T08:36:57.412 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:36:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:57.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:36:57.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:36:57.560 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:57.579 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:36:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:36:57.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api api_service: [==========] Running 4 tests from 1 test suite. 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [----------] Global test environment set-up. 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5201 ms) 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ OK ] LibRadosService.RegisterLate (25 ms) 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-10T08:36:57.666 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: cluster: 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: id: 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: health: HEALTH_OK 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: services: 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: data: 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: pools: 6 pools, 132 pgs 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: objects: 192 objects, 5.7 KiB 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:57.667 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: pgs: 132 active+clean 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: io: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: client: 953 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: recovery: 0 B/s, 0 objects/s 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: cluster: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: id: 275c2754-c7ee-11ee-95b8-87774f69a715 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: health: HEALTH_WARN 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: noscrub flag(s) set 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: services: 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-02-10T08:36:57.668 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: flags noscrub 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: data: 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: pools: 29 pools, 867 pgs 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: objects: 192 objects, 5.7 KiB 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: pgs: 10.265% pgs unknown 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 74.856% pgs not active 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 647 creating+peering 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 129 active+clean 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 89 unknown 2024-02-10T08:36:57.669 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2 peering 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: io: 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: client: 1.5 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: recovery: 0 B/s, 0 objects/s 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2218 ms) 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ OK ] LibRadosService.Status (20049 ms) 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [----------] 4 tests from LibRadosService (27493 ms total) 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: 2024-02-10T08:36:57.670 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [----------] Global test environment tear-down 2024-02-10T08:36:57.671 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27493 ms total) 2024-02-10T08:36:57.671 INFO:tasks.workunit.client.0.smithi107.stdout: api_service: [ PASSED ] 4 tests. 2024-02-10T08:36:57.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:36:57.760 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:36:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:56.877878+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:57.029795+0000 mon.a (mon.0) 1027 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T08:36:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:57.064080+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:57.064158+0000 mon.a (mon.0) 1029 : audit [INF] from='client.29710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:57.064303+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:57.068333+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:57.880461+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:56.877878+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:57.029795+0000 mon.a (mon.0) 1027 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:57.064080+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:57.064158+0000 mon.a (mon.0) 1029 : audit [INF] from='client.29710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:57.064303+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:57.068333+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:58.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:57.880461+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:58.135 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-10T08:36:58.136 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (8082 ms) 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (4771 ms) 2024-02-10T08:36:58.137 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2901 ms) 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3245 ms) 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2806 ms) 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-10T08:36:58.138 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3092 ms) 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3080 ms) 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (27977 ms total) 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-10T08:36:58.139 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (27977 ms total) 2024-02-10T08:36:58.140 INFO:tasks.workunit.client.0.smithi107.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-10T08:36:58.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:36:58.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:36:58.221 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:36:58.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd out 0 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:56.877878+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:57.029795+0000 mon.a (mon.0) 1027 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:57.064080+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-7"}]': finished 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:57.064158+0000 mon.a (mon.0) 1029 : audit [INF] from='client.29710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:57.064303+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-10T08:36:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:57.068333+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:58.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:57.880461+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:58.067085+0000 mgr.y (mgr.14553) 159 : cluster [DBG] pgmap v160: 513 pgs: 1 active+clean+snaptrim, 1 peering, 64 unknown, 447 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-10T08:36:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:58.076191+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:58.087758+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:58.087983+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-10T08:36:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:58.120095+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:58.120726+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:58.881940+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:58.067085+0000 mgr.y (mgr.14553) 159 : cluster [DBG] pgmap v160: 513 pgs: 1 active+clean+snaptrim, 1 peering, 64 unknown, 447 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:58.076191+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:58.087758+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:58.087983+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:58.120095+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:58.120726+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:36:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:58.881940+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:58.067085+0000 mgr.y (mgr.14553) 159 : cluster [DBG] pgmap v160: 513 pgs: 1 active+clean+snaptrim, 1 peering, 64 unknown, 447 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:58.076191+0000 mon.c (mon.2) 163 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:58.087758+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.107:0/2648724832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi107-112403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:58.087983+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:58.120095+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:58.120726+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:36:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:36:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:58.881940+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:00.272 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:00.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:00.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:00.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.109707+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-10T08:37:00.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.109795+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:36:59.109951+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-10T08:37:00.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.115953+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.119928+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.107:0/3378273510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.120379+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.121762+0000 mon.a (mon.0) 1040 : audit [INF] from='client.40535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:36:59.890935+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.118168+0000 mon.a (mon.0) 1041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.118263+0000 mon.a (mon.0) 1042 : audit [INF] from='client.40535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:00.118441+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.109707+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.109795+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:36:59.109951+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.115953+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.119928+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.107:0/3378273510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.120379+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.121762+0000 mon.a (mon.0) 1040 : audit [INF] from='client.40535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:36:59.890935+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.118168+0000 mon.a (mon.0) 1041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.118263+0000 mon.a (mon.0) 1042 : audit [INF] from='client.40535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:00.118441+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.109707+0000 mon.a (mon.0) 1036 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.109795+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.107:0/3446435660' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:36:59.109951+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.115953+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.119928+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.107:0/3378273510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.120379+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.121762+0000 mon.a (mon.0) 1040 : audit [INF] from='client.40535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:36:59.890935+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.118168+0000 mon.a (mon.0) 1041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.118263+0000 mon.a (mon.0) 1042 : audit [INF] from='client.40535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:00.118441+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-10T08:37:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5126 ms) 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (28 ms) 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (21032 ms) 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: attempt 0 of 20 2024-02-10T08:37:01.509 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5105 ms) 2024-02-10T08:37:01.510 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31292 ms total) 2024-02-10T08:37:01.510 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: 2024-02-10T08:37:01.510 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-10T08:37:01.510 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31292 ms total) 2024-02-10T08:37:01.510 INFO:tasks.workunit.client.0.smithi107.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-10T08:37:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:00.072096+0000 mgr.y (mgr.14553) 160 : cluster [DBG] pgmap v163: 481 pgs: 1 active+clean+snaptrim, 1 peering, 96 unknown, 383 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:37:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.154331+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.107:0/2930993599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.166331+0000 mon.a (mon.0) 1044 : audit [INF] from='client.32608 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:00.179362+0000 osd.1 (osd.1) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:00.179414+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.218029+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.219920+0000 mon.a (mon.0) 1045 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.224639+0000 mon.c (mon.2) 165 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.231196+0000 mon.a (mon.0) 1046 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.236093+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.236464+0000 mon.a (mon.0) 1047 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.236971+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.237214+0000 mon.a (mon.0) 1048 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.903349+0000 mon.a (mon.0) 1049 : audit [INF] from='client.32608 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.903514+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.903660+0000 mon.a (mon.0) 1051 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:00.903753+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.904103+0000 mon.b (mon.1) 129 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.906352+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.914410+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.925201+0000 mon.c (mon.2) 168 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:00.933288+0000 mon.a (mon.0) 1054 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:00.072096+0000 mgr.y (mgr.14553) 160 : cluster [DBG] pgmap v163: 481 pgs: 1 active+clean+snaptrim, 1 peering, 96 unknown, 383 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.154331+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.107:0/2930993599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.166331+0000 mon.a (mon.0) 1044 : audit [INF] from='client.32608 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:00.179362+0000 osd.1 (osd.1) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:00.179414+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.218029+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.219920+0000 mon.a (mon.0) 1045 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.224639+0000 mon.c (mon.2) 165 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.231196+0000 mon.a (mon.0) 1046 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.236093+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.236464+0000 mon.a (mon.0) 1047 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.236971+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.237214+0000 mon.a (mon.0) 1048 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.903349+0000 mon.a (mon.0) 1049 : audit [INF] from='client.32608 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.903514+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.903660+0000 mon.a (mon.0) 1051 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:00.903753+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.904103+0000 mon.b (mon.1) 129 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.906352+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.914410+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.925201+0000 mon.c (mon.2) 168 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:00.933288+0000 mon.a (mon.0) 1054 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:00.072096+0000 mgr.y (mgr.14553) 160 : cluster [DBG] pgmap v163: 481 pgs: 1 active+clean+snaptrim, 1 peering, 96 unknown, 383 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.154331+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.107:0/2930993599' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.166331+0000 mon.a (mon.0) 1044 : audit [INF] from='client.32608 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:00.179362+0000 osd.1 (osd.1) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:00.179414+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.218029+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.219920+0000 mon.a (mon.0) 1045 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.224639+0000 mon.c (mon.2) 165 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.231196+0000 mon.a (mon.0) 1046 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.236093+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.236464+0000 mon.a (mon.0) 1047 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.236971+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.237214+0000 mon.a (mon.0) 1048 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.903349+0000 mon.a (mon.0) 1049 : audit [INF] from='client.32608 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi107-112403-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.903514+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.903660+0000 mon.a (mon.0) 1051 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:00.903753+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.904103+0000 mon.b (mon.1) 129 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.906352+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.914410+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:01.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.925201+0000 mon.c (mon.2) 168 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:01.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:00.933288+0000 mon.a (mon.0) 1054 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.172920+0000 mgr.y (mgr.14553) 161 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.474006+0000 mon.a (mon.0) 1055 : audit [DBG] from='client.? v1:172.21.15.107:0/3884870031' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.475185+0000 mgr.y (mgr.14553) 162 : audit [DBG] from='mon.0 v1:172.21.15.107:0/3103220898' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.905452+0000 mon.b (mon.1) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.919718+0000 mon.a (mon.0) 1056 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:01.937951+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.073464+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:02.262203+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.267937+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.268227+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.172920+0000 mgr.y (mgr.14553) 161 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.474006+0000 mon.a (mon.0) 1055 : audit [DBG] from='client.? v1:172.21.15.107:0/3884870031' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.475185+0000 mgr.y (mgr.14553) 162 : audit [DBG] from='mon.0 v1:172.21.15.107:0/3103220898' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.905452+0000 mon.b (mon.1) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.919718+0000 mon.a (mon.0) 1056 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:01.937951+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.073464+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:02.262203+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.267937+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.268227+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.172920+0000 mgr.y (mgr.14553) 161 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.474006+0000 mon.a (mon.0) 1055 : audit [DBG] from='client.? v1:172.21.15.107:0/3884870031' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.475185+0000 mgr.y (mgr.14553) 162 : audit [DBG] from='mon.0 v1:172.21.15.107:0/3103220898' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.905452+0000 mon.b (mon.1) 131 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.919718+0000 mon.a (mon.0) 1056 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:01.937951+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.073464+0000 mon.c (mon.2) 169 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:02.262203+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-10T08:37:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.267937+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-10T08:37:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.268227+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]: dispatch 2024-02-10T08:37:03.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:02.073093+0000 mgr.y (mgr.14553) 163 : cluster [DBG] pgmap v167: 416 pgs: 1 active+clean+snaptrim, 1 peering, 32 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.274977+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.107:0/3029299480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.286488+0000 mon.a (mon.0) 1060 : audit [INF] from='client.34111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:02.908899+0000 mon.b (mon.1) 134 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:03.068010+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check update: 2 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.099304+0000 mon.a (mon.0) 1062 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.099459+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.099528+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]': finished 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.099630+0000 mon.a (mon.0) 1065 : audit [INF] from='client.34111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:03.099997+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.110285+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.107:0/3289858881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.113461+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.107:0/613092260' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.114415+0000 mon.a (mon.0) 1067 : audit [INF] from='client.43223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.114942+0000 mon.a (mon.0) 1068 : audit [INF] from='client.35314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.252585+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.253286+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.257392+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:03.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.257783+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:02.073093+0000 mgr.y (mgr.14553) 163 : cluster [DBG] pgmap v167: 416 pgs: 1 active+clean+snaptrim, 1 peering, 32 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T08:37:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.274977+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.107:0/3029299480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.286488+0000 mon.a (mon.0) 1060 : audit [INF] from='client.34111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:02.908899+0000 mon.b (mon.1) 134 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:03.068010+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check update: 2 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.099304+0000 mon.a (mon.0) 1062 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.099459+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.099528+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]': finished 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.099630+0000 mon.a (mon.0) 1065 : audit [INF] from='client.34111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:03.099997+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.110285+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.107:0/3289858881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.113461+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.107:0/613092260' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.114415+0000 mon.a (mon.0) 1067 : audit [INF] from='client.43223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.114942+0000 mon.a (mon.0) 1068 : audit [INF] from='client.35314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.252585+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.253286+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.257392+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:03.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.257783+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:02.073093+0000 mgr.y (mgr.14553) 163 : cluster [DBG] pgmap v167: 416 pgs: 1 active+clean+snaptrim, 1 peering, 32 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T08:37:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.274977+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.107:0/3029299480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.286488+0000 mon.a (mon.0) 1060 : audit [INF] from='client.34111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:02.908899+0000 mon.b (mon.1) 134 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:03.068010+0000 mon.a (mon.0) 1061 : cluster [WRN] Health check update: 2 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.099304+0000 mon.a (mon.0) 1062 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi107-112513-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.099459+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.099528+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-9", "mode": "writeback"}]': finished 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.099630+0000 mon.a (mon.0) 1065 : audit [INF] from='client.34111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi107-112411-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:03.099997+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.110285+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.107:0/3289858881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.113461+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.107:0/613092260' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.114415+0000 mon.a (mon.0) 1067 : audit [INF] from='client.43223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.114942+0000 mon.a (mon.0) 1068 : audit [INF] from='client.35314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.252585+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.253286+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:03.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.257392+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:03.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.257783+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]: dispatch 2024-02-10T08:37:04.373 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:04.373 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.358+0000 7f162e5b7700 1 -- 172.21.15.107:0/1985985854 >> v1:172.21.15.195:6789/0 conn(0x7f162806cfc0 legacy=0x7f162806f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.359+0000 7f162e5b7700 1 -- 172.21.15.107:0/1985985854 shutdown_connections 2024-02-10T08:37:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.359+0000 7f162e5b7700 1 -- 172.21.15.107:0/1985985854 >> 172.21.15.107:0/1985985854 conn(0x7f1628063bc0 msgr2=0x7f16280672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.359+0000 7f162e5b7700 1 -- 172.21.15.107:0/1985985854 shutdown_connections 2024-02-10T08:37:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.359+0000 7f162e5b7700 1 -- 172.21.15.107:0/1985985854 wait complete. 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 librados: starting msgr at 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 librados: starting objecter 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 Processor -- start 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 -- start start 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 librados: setting wanted keys 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 librados: calling monclient init 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f162812ea00 con 0x7f1628070ba0 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f162812fbe0 con 0x7f162806cfc0 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f162e5b7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1628130de0 con 0x7f1628069540 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f1627fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1628069540 0x7f16280721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44100/0 (socket says 172.21.15.107:44100) 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f1627fff700 1 -- 172.21.15.107:0/3238348888 learned_addr learned my addr 172.21.15.107:0/3238348888 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f16257fa700 10 client.?.objecter ms_handle_connect 0x7f162806cfc0 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.360+0000 7f16257fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 10 client.?.objecter ms_handle_connect 0x7f1628070ba0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350441749 0 0) 0x7f162812fbe0 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1610002220 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540514325 0 0) 0x7f1610002220 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162812fbe0 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f161c001e30 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 690470990 0 0) 0x7f162812fbe0 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 >> v1:172.21.15.107:6790/0 conn(0x7f1628069540 legacy=0x7f16280721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 >> v1:172.21.15.107:6789/0 conn(0x7f1628070ba0 legacy=0x7f162812d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1628131fe0 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162812ebf0 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f162e5b7700 10 client.36850.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f162812f060 con 0x7f162806cfc0 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f162e5b7700 1 librados: init done 2024-02-10T08:37:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f161c002280 con 0x7f162806cfc0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.361+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f161c0037b0 con 0x7f162806cfc0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f161c001900 con 0x7f162806cfc0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1614004f50 con 0x7f162806cfc0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(130..130 src has 50..130) v4 ==== 11628+0+0 (unknown 3703673471 0 0) 0x7f161c077790 con 0x7f162806cfc0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 10 client.36850.objecter ms_dispatch 0x7f1628001550 osd_map(130..130 src has 50..130) v4 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 3 client.36850.objecter handle_osd_map got epochs [130,130] > 0 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 3 client.36850.objecter handle_osd_map decoding full epoch 130 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.369+0000 7f16257fa700 20 client.36850.objecter dump_active .. 0 homeless 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.370+0000 7f16257fa700 10 client.36850.objecter ms_handle_connect 0x7f1610063a40 2024-02-10T08:37:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.375+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f161c049e50 con 0x7f162806cfc0 2024-02-10T08:37:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:04.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:04.704+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f1614005200 con 0x7f162806cfc0 2024-02-10T08:37:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:04.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:04.895 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:05.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:37:05.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.145+0000 7f16257fa700 1 -- 172.21.15.107:0/3238348888 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v131) v1 ==== 91+0+0 (unknown 3473174974 0 0) 0x7f161c0482c0 con 0x7f162806cfc0 2024-02-10T08:37:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.152+0000 7f162e5b7700 10 librados: watch_flush enter 2024-02-10T08:37:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.158+0000 7f162e5b7700 10 librados: watch_flush exit 2024-02-10T08:37:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.158+0000 7f162e5b7700 20 client.36850.objecter shutdown clearing up homeless session... 2024-02-10T08:37:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.158+0000 7f162e5b7700 10 client.36850.objecter successfully canceled tick 2024-02-10T08:37:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.160+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1610063a40 legacy=0x7f1610065f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.160+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 >> v1:172.21.15.195:6789/0 conn(0x7f162806cfc0 legacy=0x7f16280728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.161+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 shutdown_connections 2024-02-10T08:37:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.161+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 >> 172.21.15.107:0/3238348888 conn(0x7f1628063bc0 msgr2=0x7f1628063fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.167+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 shutdown_connections 2024-02-10T08:37:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.168+0000 7f162e5b7700 1 -- 172.21.15.107:0/3238348888 wait complete. 2024-02-10T08:37:05.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:05.174+0000 7f162e5b7700 1 librados: shutdown 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:03.913108+0000 mon.b (mon.1) 138 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.106015+0000 mon.a (mon.0) 1071 : audit [INF] from='client.43223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.106078+0000 mon.a (mon.0) 1072 : audit [INF] from='client.35314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.106173+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.106209+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]': finished 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:04.106240+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-10T08:37:05.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.107143+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.115506+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.132412+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.132730+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:03.913108+0000 mon.b (mon.1) 138 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:05.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.106015+0000 mon.a (mon.0) 1071 : audit [INF] from='client.43223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.106078+0000 mon.a (mon.0) 1072 : audit [INF] from='client.35314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.106173+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.106209+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]': finished 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:04.106240+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.107143+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.115506+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.132412+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.132730+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.191 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:37:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:05.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:05.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:03.913108+0000 mon.b (mon.1) 138 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.106015+0000 mon.a (mon.0) 1071 : audit [INF] from='client.43223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.106078+0000 mon.a (mon.0) 1072 : audit [INF] from='client.35314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi107-112403-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.106173+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.106209+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4"}]': finished 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:04.106240+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.107143+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.115506+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.132412+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]: dispatch 2024-02-10T08:37:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.132730+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]: dispatch 2024-02-10T08:37:05.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:05.331 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:05.350 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:05.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:05.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:05.661 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:05.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:04.073745+0000 mgr.y (mgr.14553) 164 : cluster [DBG] pgmap v169: 488 pgs: 1 active+clean+snaptrim, 1 peering, 104 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:37:05.857 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:05] "GET /metrics HTTP/1.1" 200 207295 "" "Prometheus/2.33.4" 2024-02-10T08:37:06.061 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:06.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:06.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.705084+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3238348888' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.862312+0000 mon.a (mon.0) 1078 : audit [INF] from='client.36850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:04.920650+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.105739+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.105775+0000 mon.a (mon.0) 1080 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 2 cache pools are missing hit_sets) 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.142854+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.142938+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.143081+0000 mon.a (mon.0) 1083 : audit [INF] from='client.36850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.143131+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.168490+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2013717673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.176183+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.176987+0000 mon.a (mon.0) 1085 : audit [INF] from='client.44318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.177321+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:04.073745+0000 mgr.y (mgr.14553) 164 : cluster [DBG] pgmap v169: 488 pgs: 1 active+clean+snaptrim, 1 peering, 104 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.705084+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3238348888' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.862312+0000 mon.a (mon.0) 1078 : audit [INF] from='client.36850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:04.920650+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.105739+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.105775+0000 mon.a (mon.0) 1080 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 2 cache pools are missing hit_sets) 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.142854+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.142938+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.143081+0000 mon.a (mon.0) 1083 : audit [INF] from='client.36850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:06.185 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.143131+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-10T08:37:06.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.168490+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2013717673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.176183+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.176987+0000 mon.a (mon.0) 1085 : audit [INF] from='client.44318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.177321+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:04.073745+0000 mgr.y (mgr.14553) 164 : cluster [DBG] pgmap v169: 488 pgs: 1 active+clean+snaptrim, 1 peering, 104 unknown, 382 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:37:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.705084+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.107:0/3238348888' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.862312+0000 mon.a (mon.0) 1078 : audit [INF] from='client.36850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:04.920650+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.105739+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.105775+0000 mon.a (mon.0) 1080 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 2 cache pools are missing hit_sets) 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.142854+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6"}]': finished 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.142938+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-9"}]': finished 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.143081+0000 mon.a (mon.0) 1083 : audit [INF] from='client.36850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.143131+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.168490+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2013717673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.176183+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.107:0/2938278666' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.176987+0000 mon.a (mon.0) 1085 : audit [INF] from='client.44318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.177321+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T08:37:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:06.392 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:06.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:06.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:06.595 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:06.616 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:06.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:06.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:06.818 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout:_watch_notify_pp: flushed 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3028 ms) 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: trying... 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 515396075522 notifier_gid 25276 2024-02-10T08:37:06.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: timed out 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: flushing 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: flushed 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3017 ms) 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: List watches 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify2 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846601600 notify_id 528280977418 notifier_gid 25276 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify2 done 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: watch_check 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: unwatch2 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: flushing 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: done 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3027 ms) 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-10T08:37:06.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: List watches 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify2 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: handle_notify cookie 94754846605040 notify_id 541165879307 notifier_gid 25276 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: notify2 done 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: watch_check 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: unwatch2 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: flushing 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: done 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3014 ms) 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16347 ms total) 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (36797 ms total) 2024-02-10T08:37:06.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-10T08:37:07.031 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:07.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:07.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.789904+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.796017+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.909401+0000 mon.a (mon.0) 1087 : audit [INF] from='client.44318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.909484+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:05.909674+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.927244+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.930835+0000 mon.c (mon.2) 172 : audit [INF] from='client.? v1:172.21.15.107:0/4199901723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:05.931251+0000 mon.a (mon.0) 1090 : audit [INF] from='client.45152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.078450+0000 mgr.y (mgr.14553) 165 : cluster [DBG] pgmap v173: 451 pgs: 32 creating+peering, 1 active+clean+snaptrim, 418 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.082037+0000 mon.c (mon.2) 173 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.287482+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.789904+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.796017+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.909401+0000 mon.a (mon.0) 1087 : audit [INF] from='client.44318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.909484+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:05.909674+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.927244+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.930835+0000 mon.c (mon.2) 172 : audit [INF] from='client.? v1:172.21.15.107:0/4199901723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:05.931251+0000 mon.a (mon.0) 1090 : audit [INF] from='client.45152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.078450+0000 mgr.y (mgr.14553) 165 : cluster [DBG] pgmap v173: 451 pgs: 32 creating+peering, 1 active+clean+snaptrim, 418 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-10T08:37:07.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.082037+0000 mon.c (mon.2) 173 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.287482+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:07.259 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.789904+0000 osd.5 (osd.5) 67 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.796017+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.909401+0000 mon.a (mon.0) 1087 : audit [INF] from='client.44318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi107-112411-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.909484+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi107-112562-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:05.909674+0000 mon.a (mon.0) 1089 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.927244+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.930835+0000 mon.c (mon.2) 172 : audit [INF] from='client.? v1:172.21.15.107:0/4199901723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:05.931251+0000 mon.a (mon.0) 1090 : audit [INF] from='client.45152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.078450+0000 mgr.y (mgr.14553) 165 : cluster [DBG] pgmap v173: 451 pgs: 32 creating+peering, 1 active+clean+snaptrim, 418 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-10T08:37:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.082037+0000 mon.c (mon.2) 173 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.287482+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-10T08:37:07.571 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:07.582 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:07.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:07.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:07.717 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:07.736 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:07.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:07.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.670966+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.c scrub starts 2024-02-10T08:37:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.682467+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.c scrub ok 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.763526+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.b scrub starts 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.766808+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.b scrub ok 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.927402+0000 mon.a (mon.0) 1092 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.933508+0000 mon.b (mon.1) 145 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.939692+0000 mon.a (mon.0) 1093 : audit [INF] from='client.45152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.939942+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:06.940000+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.956755+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:06.966732+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:37:08.172 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.670966+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.c scrub starts 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.682467+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.c scrub ok 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.763526+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.b scrub starts 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.766808+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.b scrub ok 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.927402+0000 mon.a (mon.0) 1092 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.933508+0000 mon.b (mon.1) 145 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.939692+0000 mon.a (mon.0) 1093 : audit [INF] from='client.45152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.939942+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:06.940000+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.956755+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:06.966732+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.670966+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.c scrub starts 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.682467+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.c scrub ok 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.763526+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.b scrub starts 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.766808+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.b scrub ok 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.927402+0000 mon.a (mon.0) 1092 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.933508+0000 mon.b (mon.1) 145 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.939692+0000 mon.a (mon.0) 1093 : audit [INF] from='client.45152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi107-112403-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.939942+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:06.940000+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.956755+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.174 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:06.966732+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:08.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.601+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/925576392 >> v1:172.21.15.195:6789/0 conn(0x7f6fe006cfc0 legacy=0x7f6fe006f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/925576392 shutdown_connections 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/925576392 >> 172.21.15.107:0/925576392 conn(0x7f6fe0063bc0 msgr2=0x7f6fe00672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/925576392 shutdown_connections 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/925576392 wait complete. 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 librados: starting msgr at 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 librados: starting objecter 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 Processor -- start 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 -- start start 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 librados: setting wanted keys 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.602+0000 7f6fe7d5b700 1 librados: calling monclient init 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fe7d5b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6fe0132e00 con 0x7f6fe006cfc0 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fe7d5b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6fe0134000 con 0x7f6fe0069540 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fe7d5b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6fe0135200 con 0x7f6fe0070ba0 2024-02-10T08:37:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fe5af7700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6fe0069540 0x7f6fe00730e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:42922/0 (socket says 172.21.15.107:42922) 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fe5af7700 1 -- 172.21.15.107:0/3396498434 learned_addr learned my addr 172.21.15.107:0/3396498434 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fd6ffd700 10 client.?.objecter ms_handle_connect 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fd6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906788410 0 0) 0x7f6fe0134000 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.603+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc003630 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1942602000 0 0) 0x7f6fcc003630 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe0134000 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6fdc001800 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3815030607 0 0) 0x7f6fe0134000 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 >> v1:172.21.15.107:6790/0 conn(0x7f6fe0070ba0 legacy=0x7f6fe01314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 >> v1:172.21.15.107:6789/0 conn(0x7f6fe006cfc0 legacy=0x7f6fe00737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0136400 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe01353f0 con 0x7f6fe0069540 2024-02-10T08:37:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fe7d5b700 10 client.40087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe0135860 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fe7d5b700 1 librados: init done 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6fdc004c70 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.604+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6fdc005b10 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.606+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6fdc0047d0 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.606+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(134..134 src has 50..134) v4 ==== 10846+0+0 (unknown 2091227561 0 0) 0x7f6fdc079810 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.606+0000 7f6fd6ffd700 10 client.40087.objecter ms_dispatch 0x7f6fe0001550 osd_map(134..134 src has 50..134) v4 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.606+0000 7f6fd6ffd700 3 client.40087.objecter handle_osd_map got epochs [134,134] > 0 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.606+0000 7f6fd6ffd700 3 client.40087.objecter handle_osd_map decoding full epoch 134 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.607+0000 7f6fd6ffd700 20 client.40087.objecter dump_active .. 0 homeless 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.607+0000 7f6fd6ffd700 10 client.40087.objecter ms_handle_connect 0x7f6fcc063ba0 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.608+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc4004f50 con 0x7f6fe0069540 2024-02-10T08:37:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.614+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6fdc04c120 con 0x7f6fe0069540 2024-02-10T08:37:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:08.952+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd out", "ids": ["0"]} v 0) v1 -- 0x7f6fc4005200 con 0x7f6fe0069540 2024-02-10T08:37:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.020+0000 7f6fd6ffd700 1 -- 172.21.15.107:0/3396498434 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["0"]}]=0 marked out osd.0. v135) v1 ==== 87+0+0 (unknown 3404683187 0 0) 0x7f6fdc04a530 con 0x7f6fe0069540 2024-02-10T08:37:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.036+0000 7f6fe7d5b700 10 librados: watch_flush enter 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.038+0000 7f6fe7d5b700 10 librados: watch_flush exit 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.038+0000 7f6fe7d5b700 20 client.40087.objecter shutdown clearing up homeless session... 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.038+0000 7f6fe7d5b700 10 client.40087.objecter successfully canceled tick 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.041+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6fcc063ba0 legacy=0x7f6fcc066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.041+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 >> v1:172.21.15.195:6789/0 conn(0x7f6fe0069540 legacy=0x7f6fe00730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.041+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 shutdown_connections 2024-02-10T08:37:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.041+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 >> 172.21.15.107:0/3396498434 conn(0x7f6fe0063bc0 msgr2=0x7f6fe00652d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:09.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.056+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 shutdown_connections 2024-02-10T08:37:09.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.056+0000 7f6fe7d5b700 1 -- 172.21.15.107:0/3396498434 wait complete. 2024-02-10T08:37:09.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:09.056+0000 7f6fe7d5b700 1 librados: shutdown 2024-02-10T08:37:09.077 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.0. 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:37:08.956+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e134 definitely_dead 0 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.239754+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.241242+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.574213+0000 osd.4 (osd.4) 69 : cluster [DBG] 2.f scrub starts 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.577826+0000 osd.4 (osd.4) 70 : cluster [DBG] 2.f scrub ok 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.692976+0000 osd.1 (osd.1) 87 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.696595+0000 osd.1 (osd.1) 88 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.837229+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:37:09.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.840733+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:07.941079+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:07.975091+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:07.975327+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:07.998398+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.107:0/4059147222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.012083+0000 mon.a (mon.0) 1099 : audit [INF] from='client.38938 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.019971+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.021450+0000 mon.a (mon.0) 1100 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.086116+0000 mgr.y (mgr.14553) 166 : cluster [DBG] pgmap v176: 419 pgs: 96 unknown, 1 active+clean+snaptrim, 322 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 514 B/s wr, 0 op/s 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.943965+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.956532+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3396498434' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:08.956921+0000 mon.a (mon.0) 1101 : audit [INF] from='client.40087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.956990+0000 mon.a (mon.0) 1102 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-10T08:37:09.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.239754+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.241242+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.574213+0000 osd.4 (osd.4) 69 : cluster [DBG] 2.f scrub starts 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.577826+0000 osd.4 (osd.4) 70 : cluster [DBG] 2.f scrub ok 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.692976+0000 osd.1 (osd.1) 87 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.696595+0000 osd.1 (osd.1) 88 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.837229+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.840733+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:07.941079+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:07.975091+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:07.975327+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:07.998398+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.107:0/4059147222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.012083+0000 mon.a (mon.0) 1099 : audit [INF] from='client.38938 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.019971+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.021450+0000 mon.a (mon.0) 1100 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.086116+0000 mgr.y (mgr.14553) 166 : cluster [DBG] pgmap v176: 419 pgs: 96 unknown, 1 active+clean+snaptrim, 322 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 514 B/s wr, 0 op/s 2024-02-10T08:37:09.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.943965+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.956532+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3396498434' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:08.956921+0000 mon.a (mon.0) 1101 : audit [INF] from='client.40087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.956990+0000 mon.a (mon.0) 1102 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-10T08:37:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.239754+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:37:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.241242+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:37:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.574213+0000 osd.4 (osd.4) 69 : cluster [DBG] 2.f scrub starts 2024-02-10T08:37:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.577826+0000 osd.4 (osd.4) 70 : cluster [DBG] 2.f scrub ok 2024-02-10T08:37:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.692976+0000 osd.1 (osd.1) 87 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.696595+0000 osd.1 (osd.1) 88 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.837229+0000 osd.5 (osd.5) 69 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.840733+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:07.941079+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:07.975091+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:07.975327+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:07.998398+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.107:0/4059147222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.012083+0000 mon.a (mon.0) 1099 : audit [INF] from='client.38938 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.019971+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.021450+0000 mon.a (mon.0) 1100 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.086116+0000 mgr.y (mgr.14553) 166 : cluster [DBG] pgmap v176: 419 pgs: 96 unknown, 1 active+clean+snaptrim, 322 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 514 B/s wr, 0 op/s 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.943965+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.956532+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.107:0/3396498434' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:08.956921+0000 mon.a (mon.0) 1101 : audit [INF] from='client.40087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-10T08:37:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.956990+0000 mon.a (mon.0) 1102 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-10T08:37:09.927 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:09.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:10.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:10.057 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:10.084 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:10.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:10.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:10.238 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:10.259 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:10.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:10.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:10.388 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.271473+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.277347+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.743034+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.744621+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.817101+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.c scrub starts 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:08.820580+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.c scrub ok 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.005909+0000 mon.a (mon.0) 1103 : audit [INF] from='client.38938 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.006000+0000 mon.a (mon.0) 1104 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.006144+0000 mon.a (mon.0) 1105 : audit [INF] from='client.40087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-10T08:37:10.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:09.006187+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e135: 8 total, 8 up, 7 in 2024-02-10T08:37:10.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.041803+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.107:0/2992111292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.060060+0000 mon.a (mon.0) 1107 : audit [INF] from='client.39757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:09.945700+0000 mon.b (mon.1) 153 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:10.390 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.271473+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.277347+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.743034+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.744621+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.817101+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.c scrub starts 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:08.820580+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.c scrub ok 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.005909+0000 mon.a (mon.0) 1103 : audit [INF] from='client.38938 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.006000+0000 mon.a (mon.0) 1104 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.006144+0000 mon.a (mon.0) 1105 : audit [INF] from='client.40087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:09.006187+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e135: 8 total, 8 up, 7 in 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.041803+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.107:0/2992111292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.060060+0000 mon.a (mon.0) 1107 : audit [INF] from='client.39757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:09.945700+0000 mon.b (mon.1) 153 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.271473+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.277347+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.743034+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.744621+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.817101+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.c scrub starts 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:08.820580+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.c scrub ok 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.005909+0000 mon.a (mon.0) 1103 : audit [INF] from='client.38938 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi107-112411-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.006000+0000 mon.a (mon.0) 1104 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112501-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.006144+0000 mon.a (mon.0) 1105 : audit [INF] from='client.40087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:09.006187+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e135: 8 total, 8 up, 7 in 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.041803+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.107:0/2992111292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.060060+0000 mon.a (mon.0) 1107 : audit [INF] from='client.39757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:10.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:09.945700+0000 mon.b (mon.1) 153 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:10.410 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:10.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:10.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:10.680 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:10.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:11.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:11.159 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:11.227 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:11.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:11.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:11.426 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:11.460 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:11.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:09.211905+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:09.215253+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.076953+0000 mon.a (mon.0) 1108 : audit [INF] from='client.39757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:10.077090+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e136: 8 total, 8 up, 7 in 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.085080+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:10.093155+0000 mgr.y (mgr.14553) 167 : cluster [DBG] pgmap v178: 450 pgs: 128 unknown, 1 active+clean+snaptrim, 321 active+clean; 144 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.099205+0000 mon.a (mon.0) 1110 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.413046+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.543662+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.560246+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.735355+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.735927+0000 mgr.y (mgr.14553) 168 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.738100+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.739760+0000 mgr.y (mgr.14553) 169 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.750798+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.755301+0000 mgr.y (mgr.14553) 170 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.757007+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.758549+0000 mgr.y (mgr.14553) 171 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.759329+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.760927+0000 mgr.y (mgr.14553) 172 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.761771+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.764755+0000 mgr.y (mgr.14553) 173 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.768338+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.768604+0000 mgr.y (mgr.14553) 174 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.769762+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.770090+0000 mgr.y (mgr.14553) 175 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.771226+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.771714+0000 mgr.y (mgr.14553) 176 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.772774+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.773392+0000 mgr.y (mgr.14553) 177 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.919895+0000 mon.a (mon.0) 1113 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]': finished 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.920063+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:10.920144+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.930453+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.930670+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.938474+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:10.939863+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 105'10 2024-02-10T08:37:11.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.960151+0000 mon.a (mon.0) 1117 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.960504+0000 mon.a (mon.0) 1118 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:10.963897+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:11.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:11.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:09.211905+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:37:11.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:09.215253+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.076953+0000 mon.a (mon.0) 1108 : audit [INF] from='client.39757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:10.077090+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e136: 8 total, 8 up, 7 in 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.085080+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:10.093155+0000 mgr.y (mgr.14553) 167 : cluster [DBG] pgmap v178: 450 pgs: 128 unknown, 1 active+clean+snaptrim, 321 active+clean; 144 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.099205+0000 mon.a (mon.0) 1110 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.413046+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.543662+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.560246+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.735355+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.735927+0000 mgr.y (mgr.14553) 168 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.738100+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.739760+0000 mgr.y (mgr.14553) 169 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.750798+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.755301+0000 mgr.y (mgr.14553) 170 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.757007+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.758549+0000 mgr.y (mgr.14553) 171 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.759329+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.760927+0000 mgr.y (mgr.14553) 172 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.761771+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.764755+0000 mgr.y (mgr.14553) 173 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.768338+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.768604+0000 mgr.y (mgr.14553) 174 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.769762+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.770090+0000 mgr.y (mgr.14553) 175 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.771226+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.771714+0000 mgr.y (mgr.14553) 176 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.691 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.772774+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.773392+0000 mgr.y (mgr.14553) 177 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.919895+0000 mon.a (mon.0) 1113 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]': finished 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.920063+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:10.920144+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.930453+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.930670+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.938474+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:10.939863+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 105'10 2024-02-10T08:37:11.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.960151+0000 mon.a (mon.0) 1117 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.960504+0000 mon.a (mon.0) 1118 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:10.963897+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:11.694 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:09.211905+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:37:11.694 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:09.215253+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:37:11.694 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.076953+0000 mon.a (mon.0) 1108 : audit [INF] from='client.39757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi107-112403-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:11.694 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:10.077090+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e136: 8 total, 8 up, 7 in 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.085080+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:10.093155+0000 mgr.y (mgr.14553) 167 : cluster [DBG] pgmap v178: 450 pgs: 128 unknown, 1 active+clean+snaptrim, 321 active+clean; 144 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.099205+0000 mon.a (mon.0) 1110 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.413046+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.543662+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.560246+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.735355+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.735927+0000 mgr.y (mgr.14553) 168 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.0"}]: dispatch 2024-02-10T08:37:11.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.738100+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.739760+0000 mgr.y (mgr.14553) 169 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.1"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.750798+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.755301+0000 mgr.y (mgr.14553) 170 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.2"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.757007+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.758549+0000 mgr.y (mgr.14553) 171 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.3"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.759329+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.760927+0000 mgr.y (mgr.14553) 172 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.4"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.761771+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.764755+0000 mgr.y (mgr.14553) 173 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.5"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.768338+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.768604+0000 mgr.y (mgr.14553) 174 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.6"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.769762+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.770090+0000 mgr.y (mgr.14553) 175 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.7"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.771226+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.771714+0000 mgr.y (mgr.14553) 176 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.8"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.772774+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.773392+0000 mgr.y (mgr.14553) 177 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "8.9"}]: dispatch 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.919895+0000 mon.a (mon.0) 1113 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache", "force_nonempty":""}]': finished 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.920063+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:10.920144+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e137: 8 total, 8 up, 7 in 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.930453+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.930670+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.938474+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:10.939863+0000 osd.4 (osd.4) 71 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 105'10 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.960151+0000 mon.a (mon.0) 1117 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:11.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.960504+0000 mon.a (mon.0) 1118 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:11.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:10.963897+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:11.834 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:11.834 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:12.110 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:12.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:12.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:12.340 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.189856+0000 mgr.y (mgr.14553) 178 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.908206+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? v1:172.21.15.107:0/2695461290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.926088+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.926183+0000 mon.a (mon.0) 1120 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.926269+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:11.926406+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.936193+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.107:0/787475483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.938297+0000 mon.a (mon.0) 1123 : audit [INF] from='client.41800 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.938833+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.939210+0000 mon.a (mon.0) 1124 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.951074+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.953572+0000 mon.a (mon.0) 1125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:11.975373+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.189856+0000 mgr.y (mgr.14553) 178 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.908206+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? v1:172.21.15.107:0/2695461290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.926088+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.926183+0000 mon.a (mon.0) 1120 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.926269+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:11.926406+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.936193+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.107:0/787475483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.938297+0000 mon.a (mon.0) 1123 : audit [INF] from='client.41800 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.938833+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.939210+0000 mon.a (mon.0) 1124 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.951074+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.953572+0000 mon.a (mon.0) 1125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:11.975373+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:12.731 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:12.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.189856+0000 mgr.y (mgr.14553) 178 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.908206+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? v1:172.21.15.107:0/2695461290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.926088+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? v1:172.21.15.107:0/952359198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi107-112411-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.926183+0000 mon.a (mon.0) 1120 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112501-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.926269+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:11.926406+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e138: 8 total, 8 up, 7 in 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.936193+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.107:0/787475483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.938297+0000 mon.a (mon.0) 1123 : audit [INF] from='client.41800 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.938833+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.107:0/2075174279' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.939210+0000 mon.a (mon.0) 1124 : audit [INF] from='client.38944 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]: dispatch 2024-02-10T08:37:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.951074+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.953572+0000 mon.a (mon.0) 1125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]: dispatch 2024-02-10T08:37:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:11.975373+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:12.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:12.877 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:12.895 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:12.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:12.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:13.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:37:13.257 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:13.303 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.901+0000 7f4619ba3700 1 -- 172.21.15.107:0/2038103590 >> v1:172.21.15.107:6789/0 conn(0x7f46140fc910 legacy=0x7f46140fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.901+0000 7f4619ba3700 1 -- 172.21.15.107:0/2038103590 shutdown_connections 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.901+0000 7f4619ba3700 1 -- 172.21.15.107:0/2038103590 >> 172.21.15.107:0/2038103590 conn(0x7f46140f2510 msgr2=0x7f46140f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.902+0000 7f4619ba3700 1 -- 172.21.15.107:0/2038103590 shutdown_connections 2024-02-10T08:37:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.902+0000 7f4619ba3700 1 -- 172.21.15.107:0/2038103590 wait complete. 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.902+0000 7f4619ba3700 1 librados: starting msgr at 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.902+0000 7f4619ba3700 1 librados: starting objecter 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.902+0000 7f4619ba3700 1 Processor -- start 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 -- start start 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 librados: setting wanted keys 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 librados: calling monclient init 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f461411dbd0 con 0x7f46140ff710 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f461411ed70 con 0x7f4614103250 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4619ba3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f461411ff10 con 0x7f46140fc910 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4612ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f46140ff710 0x7f4614061260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45026/0 (socket says 172.21.15.107:45026) 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4612ffd700 1 -- 172.21.15.107:0/2695461290 learned_addr learned my addr 172.21.15.107:0/2695461290 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4610ff9700 10 client.?.objecter ms_handle_connect 0x7f46140ff710 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.903+0000 7f4610ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 10 client.?.objecter ms_handle_connect 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781528677 0 0) 0x7f461411ed70 con 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc002220 con 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906167425 0 0) 0x7f461411dbd0 con 0x7f46140ff710 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461411ed70 con 0x7f46140ff710 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963914052 0 0) 0x7f45fc002220 con 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f461411dbd0 con 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f460c001b80 con 0x7f4614103250 2024-02-10T08:37:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926276684 0 0) 0x7f461411ed70 con 0x7f46140ff710 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45fc002220 con 0x7f46140ff710 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4608001820 con 0x7f46140ff710 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 10 client.?.objecter ms_handle_connect 0x7f46140fc910 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100391137 0 0) 0x7f461411dbd0 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6790/0 conn(0x7f46140fc910 legacy=0x7f4614060b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6789/0 conn(0x7f46140ff710 legacy=0x7f4614061260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46141210b0 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.904+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461411ef60 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4619ba3700 10 client.42424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f461411f3d0 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4619ba3700 1 librados: init done 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f460c002240 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f460c003620 con 0x7f4614103250 2024-02-10T08:37:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.905+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4600004f00 con 0x7f4614103250 2024-02-10T08:37:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.906+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f460c004870 con 0x7f4614103250 2024-02-10T08:37:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(137..137 src has 50..137) v4 ==== 11370+0+0 (unknown 1831120440 0 0) 0x7f460c0774e0 con 0x7f4614103250 2024-02-10T08:37:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 osd_map(137..137 src has 50..137) v4 2024-02-10T08:37:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 3 client.42424.objecter handle_osd_map got epochs [137,137] > 0 2024-02-10T08:37:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 3 client.42424.objecter handle_osd_map decoding full epoch 137 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 20 client.42424.objecter dump_active .. 0 homeless 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.907+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v137) v1 ==== 54+0+16 (unknown 1090232970 0 2420000192) 0x7f460c0798c0 con 0x7f4614103250 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 10 client.42424.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 1 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 1 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 0 1 2024-02-10T08:37:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 4 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 10 client.42424.objecter _send_command 1 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.908+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4600004950 con 0x7f4600004530 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.910+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f4600004530 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4600004950 con 0x7f4600004530 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 10 client.42424.objecter _finish_command 1 = system:0 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 4 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 0 1 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.912+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f45fc063cc0 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 10 client.42424.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 2 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 2 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f46000051a0 osd=0 3 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 0 2 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 4 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 10 client.42424.objecter _send_command 2 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.949+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4600004d30 con 0x7f4600004530 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.955+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4600004d30 con 0x7f4600004530 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.955+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.955+0000 7f4610ff9700 10 client.42424.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.955+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 4 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.955+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 0 2 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 10 client.42424.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 3 2024-02-10T08:37:13.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 1 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 4 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 10 client.42424.objecter _send_command 3 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.958+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460000c000 con 0x7f4600004150 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.959+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f4600004150 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.960+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f460000c000 con 0x7f4600004150 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.960+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.960+0000 7f4610ff9700 10 client.42424.objecter _finish_command 3 = system:0 2024-02-10T08:37:13.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.960+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.960+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 1 3 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 10 client.42424.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600003fe0 osd=1 3 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 1 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 10 client.42424.objecter _send_command 4 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.992+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f46000018b0 con 0x7f4600004150 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.996+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f46000018b0 con 0x7f4600004150 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.996+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.996+0000 7f4610ff9700 10 client.42424.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.996+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 4 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:11.996+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 1 4 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 10 client.42424.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 5 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.314 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 5 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 2 5 2024-02-10T08:37:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 4 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 10 client.42424.objecter _send_command 5 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.005+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460000de90 con 0x7f460000d2f0 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.007+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f460000d2f0 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.014+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f460000de90 con 0x7f460000d2f0 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.014+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.014+0000 7f4610ff9700 10 client.42424.objecter _finish_command 5 = system:0 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.014+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 4 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.014+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 2 5 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 10 client.42424.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 6 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 6 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600003630 osd=2 3 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 2 6 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 4 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 10 client.42424.objecter _send_command 6 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.046+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4600003160 con 0x7f460000d2f0 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.048+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4600003160 con 0x7f460000d2f0 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.048+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.048+0000 7f4610ff9700 10 client.42424.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.048+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 4 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.048+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 2 6 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 10 client.42424.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 7 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 7 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 3 7 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 4 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 10 client.42424.objecter _send_command 7 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.058+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4600015bb0 con 0x7f4600011ff0 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f4600011ff0 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4600015bb0 con 0x7f4600011ff0 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 10 client.42424.objecter _finish_command 7 = system:0 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 4 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.062+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 3 7 2024-02-10T08:37:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 10 client.42424.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 8 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 8 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600011970 osd=3 3 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 3 8 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 4 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 10 client.42424.objecter _send_command 8 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.086+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460000e270 con 0x7f4600011ff0 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.092+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f460000e270 con 0x7f4600011ff0 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.092+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.092+0000 7f4610ff9700 10 client.42424.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.092+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 4 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.092+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 3 8 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 10 client.42424.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 9 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 9 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 4 9 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 4 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 10 client.42424.objecter _send_command 9 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.101+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460000e1a0 con 0x7f460000ce50 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f460000ce50 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f460000e1a0 con 0x7f460000ce50 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 10 client.42424.objecter _finish_command 9 = system:0 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 4 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.103+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 4 9 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 10 client.42424.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 10 2024-02-10T08:37:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 10 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f460000e030 osd=4 3 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 4 10 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 4 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 10 client.42424.objecter _send_command 10 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.128+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460000e1a0 con 0x7f460000ce50 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f460000e1a0 con 0x7f460000ce50 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4610ff9700 10 client.42424.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 4 2024-02-10T08:37:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 4 10 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 10 client.42424.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 11 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 11 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 5 11 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 4 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 10 client.42424.objecter _send_command 11 2024-02-10T08:37:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.129+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4600016f80 con 0x7f4600016970 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f4600016970 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4600016f80 con 0x7f4600016970 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 10 client.42424.objecter _finish_command 11 = system:0 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 4 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.136+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 5 11 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 10 client.42424.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 12 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 12 2024-02-10T08:37:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f460000e1a0 osd=5 3 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 5 12 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 4 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 10 client.42424.objecter _send_command 12 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.157+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4600016f80 con 0x7f4600016970 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.171+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4600016f80 con 0x7f4600016970 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.171+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.171+0000 7f4610ff9700 10 client.42424.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.171+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 4 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.171+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 5 12 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.184+0000 7f4619ba3700 10 client.42424.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.184+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 13 2024-02-10T08:37:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 13 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 6 13 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 4 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 10 client.42424.objecter _send_command 13 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.185+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46000171c0 con 0x7f460001f1f0 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f460001f1f0 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f46000171c0 con 0x7f460001f1f0 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 10 client.42424.objecter _finish_command 13 = system:0 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 4 2024-02-10T08:37:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.188+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 6 13 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 10 client.42424.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 14 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 14 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f4600016f80 osd=6 3 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 6 14 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 4 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 10 client.42424.objecter _send_command 14 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.211+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f46000171c0 con 0x7f460001f1f0 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.213+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f46000171c0 con 0x7f460001f1f0 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.213+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.213+0000 7f4610ff9700 10 client.42424.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.213+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 4 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.213+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 6 14 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 10 client.42424.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 15 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 15 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 7 15 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 4 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 10 client.42424.objecter _send_command 15 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4600023f40 con 0x7f4600024a10 2024-02-10T08:37:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.214+0000 7f4610ff9700 10 client.42424.objecter ms_handle_connect 0x7f4600024a10 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.215+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4600023f40 con 0x7f4600024a10 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.215+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.215+0000 7f4610ff9700 10 client.42424.objecter _finish_command 15 = system:0 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.215+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 4 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.215+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 7 15 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 10 client.42424.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign -1 16 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 20 client.42424.objecter _get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_remove -1 16 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 20 client.42424.objecter get_session s=0x7f46000171c0 osd=7 3 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 15 client.42424.objecter _session_command_op_assign 7 16 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 4 2024-02-10T08:37:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 10 client.42424.objecter _send_command 16 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.250+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4600023f40 con 0x7f4600024a10 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.267+0000 7f4610ff9700 1 -- 172.21.15.107:0/2695461290 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4600023f40 con 0x7f4600024a10 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.267+0000 7f4610ff9700 10 client.42424.objecter ms_dispatch 0x7f4614001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.267+0000 7f4610ff9700 10 client.42424.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.267+0000 7f4610ff9700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 4 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.267+0000 7f4610ff9700 15 client.42424.objecter _session_command_op_remove 7 16 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.285+0000 7f4619ba3700 10 librados: watch_flush enter 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 librados: watch_flush exit 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.0 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6801/3321596972 conn(0x7f4600004530 legacy=0x7f4600002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000051a0 osd=0 1 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.1 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6805/133516126 conn(0x7f4600004150 legacy=0x7f460000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003fe0 osd=1 1 2024-02-10T08:37:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.2 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6809/3575881728 conn(0x7f460000d2f0 legacy=0x7f460000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600003630 osd=2 1 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.3 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6813/2871825691 conn(0x7f4600011ff0 legacy=0x7f46000143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600011970 osd=3 1 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.4 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.195:6800/2499671285 conn(0x7f460000ce50 legacy=0x7f4600015d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e030 osd=4 1 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.5 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.195:6804/4178321486 conn(0x7f4600016970 legacy=0x7f460001d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f460000e1a0 osd=5 1 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.6 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.195:6808/1417479017 conn(0x7f460001f1f0 legacy=0x7f4600021640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f4600016f80 osd=6 1 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter close_session for osd.7 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.195:6812/1934176928 conn(0x7f4600024a10 legacy=0x7f4600024e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter put_session s=0x7f46000171c0 osd=7 1 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 20 client.42424.objecter shutdown clearing up homeless session... 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.286+0000 7f4619ba3700 10 client.42424.objecter successfully canceled tick 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.107:6800/2634842009 conn(0x7f45fc063cc0 legacy=0x7f45fc066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> v1:172.21.15.195:6789/0 conn(0x7f4614103250 legacy=0x7f461411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4613fff700 1 -- 172.21.15.107:0/2695461290 reap_dead start 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 shutdown_connections 2024-02-10T08:37:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.287+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 >> 172.21.15.107:0/2695461290 conn(0x7f46140f2510 msgr2=0x7f46141066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.293+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 shutdown_connections 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.294+0000 7f4619ba3700 1 -- 172.21.15.107:0/2695461290 wait complete. 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:12.294+0000 7f4619ba3700 1 librados: shutdown 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:37:13.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:11.814586+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.d scrub starts 2024-02-10T08:37:13.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:11.819436+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.d scrub ok 2024-02-10T08:37:13.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.093801+0000 mgr.y (mgr.14553) 179 : cluster [DBG] pgmap v182: 482 pgs: 397 active+clean, 1 active+recovery_wait, 1 active+recovering, 2 active+clean+snaptrim, 32 unknown, 32 creating+peering, 1 active+remapped+backfilling, 14 creating+activating, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 766 B/s rd, 18 MiB/s wr, 5 op/s; 13/789 objects misplaced (1.648%); 6.0 MiB/s, 249 keys/s, 17 objects/s recovering 2024-02-10T08:37:13.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.221412+0000 osd.6 (osd.6) 55 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.222217+0000 osd.6 (osd.6) 56 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.396045+0000 osd.3 (osd.3) 83 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.400098+0000 osd.3 (osd.3) 84 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.613106+0000 osd.4 (osd.4) 72 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.616191+0000 osd.4 (osd.4) 73 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.709604+0000 osd.1 (osd.1) 89 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.735478+0000 osd.1 (osd.1) 90 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.926766+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.932369+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.953918+0000 mon.a (mon.0) 1128 : audit [INF] from='client.41800 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.953994+0000 mon.a (mon.0) 1129 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]': finished 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.954100+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]': finished 2024-02-10T08:37:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.954181+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-10T08:37:13.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.972940+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.973786+0000 mon.a (mon.0) 1132 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:12.978710+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:11.814586+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.d scrub starts 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:11.819436+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.d scrub ok 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.093801+0000 mgr.y (mgr.14553) 179 : cluster [DBG] pgmap v182: 482 pgs: 397 active+clean, 1 active+recovery_wait, 1 active+recovering, 2 active+clean+snaptrim, 32 unknown, 32 creating+peering, 1 active+remapped+backfilling, 14 creating+activating, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 766 B/s rd, 18 MiB/s wr, 5 op/s; 13/789 objects misplaced (1.648%); 6.0 MiB/s, 249 keys/s, 17 objects/s recovering 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.221412+0000 osd.6 (osd.6) 55 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.222217+0000 osd.6 (osd.6) 56 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:37:13.713 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.396045+0000 osd.3 (osd.3) 83 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.400098+0000 osd.3 (osd.3) 84 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.613106+0000 osd.4 (osd.4) 72 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.616191+0000 osd.4 (osd.4) 73 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.709604+0000 osd.1 (osd.1) 89 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.735478+0000 osd.1 (osd.1) 90 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.926766+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.932369+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.953918+0000 mon.a (mon.0) 1128 : audit [INF] from='client.41800 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.953994+0000 mon.a (mon.0) 1129 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]': finished 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.954100+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]': finished 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.954181+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.972940+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.973786+0000 mon.a (mon.0) 1132 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:12.978710+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:11.814586+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.d scrub starts 2024-02-10T08:37:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:11.819436+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.d scrub ok 2024-02-10T08:37:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.093801+0000 mgr.y (mgr.14553) 179 : cluster [DBG] pgmap v182: 482 pgs: 397 active+clean, 1 active+recovery_wait, 1 active+recovering, 2 active+clean+snaptrim, 32 unknown, 32 creating+peering, 1 active+remapped+backfilling, 14 creating+activating, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 766 B/s rd, 18 MiB/s wr, 5 op/s; 13/789 objects misplaced (1.648%); 6.0 MiB/s, 249 keys/s, 17 objects/s recovering 2024-02-10T08:37:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.221412+0000 osd.6 (osd.6) 55 : cluster [DBG] 8.5 deep-scrub starts 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.222217+0000 osd.6 (osd.6) 56 : cluster [DBG] 8.5 deep-scrub ok 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.396045+0000 osd.3 (osd.3) 83 : cluster [DBG] 8.0 deep-scrub starts 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.400098+0000 osd.3 (osd.3) 84 : cluster [DBG] 8.0 deep-scrub ok 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.613106+0000 osd.4 (osd.4) 72 : cluster [DBG] 8.2 deep-scrub starts 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.616191+0000 osd.4 (osd.4) 73 : cluster [DBG] 8.2 deep-scrub ok 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.709604+0000 osd.1 (osd.1) 89 : cluster [DBG] 8.1 deep-scrub starts 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.735478+0000 osd.1 (osd.1) 90 : cluster [DBG] 8.1 deep-scrub ok 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.926766+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.932369+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.953918+0000 mon.a (mon.0) 1128 : audit [INF] from='client.41800 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi107-112403-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.953994+0000 mon.a (mon.0) 1129 : audit [INF] from='client.38944 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112501-10", "tierpool":"test-rados-api-smithi107-112501-10-cache"}]': finished 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.954100+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-11", "mode": "writeback"}]': finished 2024-02-10T08:37:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.954181+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e139: 8 total, 8 up, 7 in 2024-02-10T08:37:13.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.972940+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.973786+0000 mon.a (mon.0) 1132 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:13.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:12.978710+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:14.041 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:14.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:14.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:14.172 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:14.191 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:14.217 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:14.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:14.322 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:14.341 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:14.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:14.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:14.773 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.771684+0000 osd.5 (osd.5) 71 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:12.777496+0000 osd.5 (osd.5) 72 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:13.424863+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:13.426871+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:13.577532+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:13.579184+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:13.983742+0000 mon.b (mon.1) 175 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:13.999261+0000 mon.a (mon.0) 1133 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:13.999384+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.012765+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.014004+0000 mon.a (mon.0) 1135 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.015014+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.107:0/4276406080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.017464+0000 mon.a (mon.0) 1136 : audit [INF] from='client.43309 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.082567+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.092840+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:14.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.771684+0000 osd.5 (osd.5) 71 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:12.777496+0000 osd.5 (osd.5) 72 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:13.424863+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:13.426871+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:13.577532+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:13.579184+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:13.983742+0000 mon.b (mon.1) 175 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:13.999261+0000 mon.a (mon.0) 1133 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:13.999384+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.012765+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.779 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.014004+0000 mon.a (mon.0) 1135 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.015014+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.107:0/4276406080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.017464+0000 mon.a (mon.0) 1136 : audit [INF] from='client.43309 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.082567+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.092840+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.771684+0000 osd.5 (osd.5) 71 : cluster [DBG] 8.4 deep-scrub starts 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:12.777496+0000 osd.5 (osd.5) 72 : cluster [DBG] 8.4 deep-scrub ok 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:13.424863+0000 osd.3 (osd.3) 85 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:13.426871+0000 osd.3 (osd.3) 86 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:13.577532+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:13.579184+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:13.983742+0000 mon.b (mon.1) 175 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:13.999261+0000 mon.a (mon.0) 1133 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:13.999384+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e140: 8 total, 8 up, 7 in 2024-02-10T08:37:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.012765+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.107:0/1633394816' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.014004+0000 mon.a (mon.0) 1135 : audit [INF] from='client.41744 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]: dispatch 2024-02-10T08:37:14.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.015014+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.107:0/4276406080' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.017464+0000 mon.a (mon.0) 1136 : audit [INF] from='client.43309 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:14.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.082567+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.092840+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:37:14.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:14.905 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:14.925 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:14.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:15.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: Running main() from gmock_main.cc 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [----------] Global test environment set-up. 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8066 ms) 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4073 ms) 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3629 ms) 2024-02-10T08:37:15.063 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3230 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2779 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5145 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5231 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4672 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8081 ms) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [----------] 9 tests from LibRadosPools (44906 ms total) 2024-02-10T08:37:15.064 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: 2024-02-10T08:37:15.065 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [----------] Global test environment tear-down 2024-02-10T08:37:15.065 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (44906 ms total) 2024-02-10T08:37:15.065 INFO:tasks.workunit.client.0.smithi107.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-10T08:37:15.138 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:15.244 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:15.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:15.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:14.094400+0000 mgr.y (mgr.14553) 180 : cluster [DBG] pgmap v185: 410 pgs: 372 active+clean, 1 active+recovery_wait, 1 active+recovering, 1 active+clean+snaptrim, 32 unknown, 1 active+remapped+backfilling, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 512 B/s rd, 18 MiB/s wr, 3 op/s; 13/777 objects misplaced (1.673%); 6.0 MiB/s, 250 keys/s, 18 objects/s recovering 2024-02-10T08:37:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:14.259145+0000 osd.6 (osd.6) 57 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:37:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:14.267507+0000 osd.6 (osd.6) 58 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:37:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:14.568051+0000 osd.4 (osd.4) 76 : cluster [DBG] 5.e scrub starts 2024-02-10T08:37:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:14.574792+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.e scrub ok 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:14.986057+0000 mon.b (mon.1) 178 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.001443+0000 mon.a (mon.0) 1138 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.024173+0000 mon.a (mon.0) 1139 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.024264+0000 mon.a (mon.0) 1140 : audit [INF] from='client.43309 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.024416+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.024485+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.028318+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.050092+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.064359+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.107:0/1027110602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.064982+0000 mon.a (mon.0) 1144 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.083622+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.087073+0000 mon.a (mon.0) 1145 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.098119+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.100390+0000 mon.a (mon.0) 1146 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.102040+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.103953+0000 mon.a (mon.0) 1147 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:15.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:15.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:14.094400+0000 mgr.y (mgr.14553) 180 : cluster [DBG] pgmap v185: 410 pgs: 372 active+clean, 1 active+recovery_wait, 1 active+recovering, 1 active+clean+snaptrim, 32 unknown, 1 active+remapped+backfilling, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 512 B/s rd, 18 MiB/s wr, 3 op/s; 13/777 objects misplaced (1.673%); 6.0 MiB/s, 250 keys/s, 18 objects/s recovering 2024-02-10T08:37:15.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:14.259145+0000 osd.6 (osd.6) 57 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:37:15.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:14.267507+0000 osd.6 (osd.6) 58 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:37:15.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:14.568051+0000 osd.4 (osd.4) 76 : cluster [DBG] 5.e scrub starts 2024-02-10T08:37:15.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:14.574792+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.e scrub ok 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:14.986057+0000 mon.b (mon.1) 178 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.001443+0000 mon.a (mon.0) 1138 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.024173+0000 mon.a (mon.0) 1139 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.024264+0000 mon.a (mon.0) 1140 : audit [INF] from='client.43309 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.024416+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.024485+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.028318+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.050092+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.064359+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.107:0/1027110602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.064982+0000 mon.a (mon.0) 1144 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.083622+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.087073+0000 mon.a (mon.0) 1145 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.098119+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.885 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.100390+0000 mon.a (mon.0) 1146 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.885 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.102040+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.885 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.103953+0000 mon.a (mon.0) 1147 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.885 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:15] "GET /metrics HTTP/1.1" 200 203097 "" "Prometheus/2.33.4" 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:14.094400+0000 mgr.y (mgr.14553) 180 : cluster [DBG] pgmap v185: 410 pgs: 372 active+clean, 1 active+recovery_wait, 1 active+recovering, 1 active+clean+snaptrim, 32 unknown, 1 active+remapped+backfilling, 2 active+recovering+degraded; 217 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 512 B/s rd, 18 MiB/s wr, 3 op/s; 13/777 objects misplaced (1.673%); 6.0 MiB/s, 250 keys/s, 18 objects/s recovering 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:14.259145+0000 osd.6 (osd.6) 57 : cluster [DBG] 8.6 deep-scrub starts 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:14.267507+0000 osd.6 (osd.6) 58 : cluster [DBG] 8.6 deep-scrub ok 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:14.568051+0000 osd.4 (osd.4) 76 : cluster [DBG] 5.e scrub starts 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:14.574792+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.e scrub ok 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:14.986057+0000 mon.b (mon.1) 178 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.001443+0000 mon.a (mon.0) 1138 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.024173+0000 mon.a (mon.0) 1139 : audit [INF] from='client.41744 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi107-112513-9"}]': finished 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.024264+0000 mon.a (mon.0) 1140 : audit [INF] from='client.43309 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.024416+0000 mon.a (mon.0) 1141 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.024485+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e141: 8 total, 8 up, 7 in 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.028318+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.050092+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.064359+0000 mon.c (mon.2) 176 : audit [INF] from='client.? v1:172.21.15.107:0/1027110602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.064982+0000 mon.a (mon.0) 1144 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.083622+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.087073+0000 mon.a (mon.0) 1145 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.098119+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.100390+0000 mon.a (mon.0) 1146 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.102040+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.887 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.103953+0000 mon.a (mon.0) 1147 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:15.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:16.011 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:16.030 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:16.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:16.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:16.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:37:16.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:16.238 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:16.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:16.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:16.558 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:16.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.250013+0000 osd.7 (osd.7) 71 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:37:16.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.252893+0000 osd.7 (osd.7) 72 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:37:16.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.364857+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:37:16.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.367491+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.908669+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.919506+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.919618+0000 mon.a (mon.0) 1150 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.919758+0000 mon.a (mon.0) 1151 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.919835+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.921403+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.929620+0000 mon.a (mon.0) 1153 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:15.987716+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.250013+0000 osd.7 (osd.7) 71 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.252893+0000 osd.7 (osd.7) 72 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:37:16.688 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.364857+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.367491+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.908669+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.919506+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.919618+0000 mon.a (mon.0) 1150 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.919758+0000 mon.a (mon.0) 1151 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.919835+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.921403+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.929620+0000 mon.a (mon.0) 1153 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.689 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:15.987716+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:16.724 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:16.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.250013+0000 osd.7 (osd.7) 71 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.252893+0000 osd.7 (osd.7) 72 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.364857+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.367491+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.908669+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:37:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.919506+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-11"}]': finished 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.919618+0000 mon.a (mon.0) 1150 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi107-112403-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.919758+0000 mon.a (mon.0) 1151 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.919835+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e142: 8 total, 8 up, 7 in 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.921403+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.929620+0000 mon.a (mon.0) 1153 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:16.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:15.987716+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:16.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:17.094 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:17.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:17.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:17.280 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:17.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34379.0:56 5.0 5:05bf5b68:::notify.1:head [watch watch cookie 94862968474064] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:22.129583+0000", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.39345787400001, 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052884619999999999, 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34379", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 56 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129583+0000", 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129583+0000", 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129582+0000", 2024-02-10T08:37:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4490000000000002e-06 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129589+0000", 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0358e-05 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129599+0000", 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4474999999999999e-05 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.129614+0000", 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00069249700000000003 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.130306+0000", 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8820999999999996e-05 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.130385+0000", 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028973649999999998 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.133283+0000", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0346999999999995e-05 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.133363+0000", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188271 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.133551+0000", 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046244500000000002 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.134014+0000", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027748999999999997 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.134291+0000", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046225500000000001 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.134753+0000", 2024-02-10T08:37:17.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3808999999999999e-05 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.134807+0000", 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4727999999999994e-05 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.134872+0000", 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052884619999999999 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(50.3s2 epoch 113/111 rep_tid 146 entries 113'2 (0'0) error 50:c1c2e34e:test-rados-api-smithi107-112459-13::foo:head by client.35222.0:6 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.933496+0000", 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.589545438000002, 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054173119999999996, 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933496+0000", 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933496+0000", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933496+0000", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6709999999999999e-06 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933503+0000", 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1187e-05 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933514+0000", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2059999999999996e-06 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.933524+0000", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003798577 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937322+0000", 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015910060000000001 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.938913+0000", 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054173119999999996 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25276.0:2 52.14 52:28e9c4a8:test-rados-api-smithi107-112562-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.588012034999998, 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999, 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.637 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25276", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:37:17.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935027+0000", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935035+0000", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0684e-05 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935046+0000", 2024-02-10T08:37:17.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7470000000000005e-06 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935054+0000", 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020156359999999999 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937070+0000", 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4700000000000002e-06 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:37:17.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937078+0000", 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020263032 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957341+0000", 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4965999999999999e-05 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957426+0000", 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699000000000001e-05 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957524+0000", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062096199999999997 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958145+0000", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0729999999999994e-06 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958154+0000", 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048278610000000001 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962982+0000", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0270999999999999e-05 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963042+0000", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6792000000000002e-05 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963089+0000", 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.643 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35219.0:8 51.1s0 51:967778f1:test-rados-api-smithi107-112451-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.035097+0000", 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.487944314, 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049056799999999999, 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35219", 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8 2024-02-10T08:37:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035097+0000", 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035097+0000", 2024-02-10T08:37:17.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:37:17.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035094+0000", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3686000000000001e-05 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035108+0000", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8838000000000002e-05 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035127+0000", 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3300000000000001e-06 2024-02-10T08:37:17.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035134+0000", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3806999999999994e-05 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035198+0000", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000157035 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035355+0000", 2024-02-10T08:37:17.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049071800000000001 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.035846+0000", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035313199999999998 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.036199+0000", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003763041 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.039962+0000", 2024-02-10T08:37:17.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0754000000000003e-05 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.040003+0000", 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049056799999999999 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25291.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.596305352000002, 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003, 2024-02-10T08:37:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25291", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2248243388", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926734+0000", 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8688000000000001e-05 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926753+0000", 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8120000000000002e-06 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926762+0000", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6359999999999994e-06 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926770+0000", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018594099999999999 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926956+0000", 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4220000000000003e-06 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926963+0000", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025476736 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952440+0000", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103766 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952544+0000", 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013851099999999999 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952682+0000", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047535930000000004 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957436+0000", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065080549999999996 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963944+0000", 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004709319 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968653+0000", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4684999999999995e-05 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968718+0000", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6301999999999999e-05 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968754+0000", 2024-02-10T08:37:17.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35219.0:9 51.1s0 51:967778f1:test-rados-api-smithi107-112451-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.577900838000001, 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237, 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35219", 2024-02-10T08:37:17.657 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945138+0000", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7849999999999992e-06 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945146+0000", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5050000000000007e-06 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945155+0000", 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0849999999999999e-06 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945162+0000", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001228563 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946390+0000", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8227999999999996e-05 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946479+0000", 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382076999999998 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.962861+0000", 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.14907.0:2 55.15 55:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.577285624000002, 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724, 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.661 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.14907", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4285222104", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945754+0000", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9549999999999997e-06 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945760+0000", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3450000000000006e-06 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945769+0000", 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2589999999999997e-06 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945775+0000", 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01446833 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960243+0000", 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118359 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960362+0000", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016521099999999999 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960527+0000", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010993870000000001 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.961626+0000", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060821130000000001 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967708+0000", 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002001966 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969710+0000", 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3856999999999998e-05 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969744+0000", 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1362e-05 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969765+0000", 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.24889.0:19 23.d e114/102 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head v 114'3, mlcod=0'0)", 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.577195048, 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503, 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5839999999999995e-06 2024-02-10T08:37:17.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945854+0000", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3709999999999999e-06 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945860+0000", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7637999999999999e-05 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945918+0000", 2024-02-10T08:37:17.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014029747 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959947+0000", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2229999999999999e-05 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959969+0000", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076815219999999997 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967651+0000", 2024-02-10T08:37:17.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5574000000000001e-05 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967697+0000", 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(50.4s1 epoch 114/111 rep_tid 1204 entries 114'2 (0'0) error 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head by client.35222.0:9 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 29.577169207000001, 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002, 2024-02-10T08:37:17.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9130000000000003e-06 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945880+0000", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056525589999999997 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951532+0000", 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6671e-05 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951549+0000", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056510120000000004 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957200+0000", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067084960000000004 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963908+0000", 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.5.0:1233 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'1, mlcod=0'0)", 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.538962497, 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001, 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984078+0000", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7410000000000004e-06 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984087+0000", 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.772e-06 2024-02-10T08:37:17.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984096+0000", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8090000000000004e-06 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984103+0000", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000327542 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984430+0000", 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8281999999999999e-05 2024-02-10T08:37:17.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984449+0000", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017470151999999999 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001919+0000", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4270999999999999e-05 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001943+0000", 2024-02-10T08:37:17.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:8 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'2, mlcod=0'0)", 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.538929172, 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001, 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8529999999999997e-06 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984117+0000", 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9869999999999999e-06 2024-02-10T08:37:17.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984122+0000", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.625e-06 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984126+0000", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046452199999999998 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984591+0000", 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6535999999999999e-05 2024-02-10T08:37:17.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984607+0000", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023819407000000001 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008427+0000", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7294e-05 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008454+0000", 2024-02-10T08:37:17.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:20 56.2 e117/114 56:573ae330:test-rados-api-smithi107-112562-10::foo:head v 117'2, mlcod=0'0)", 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:51.072689+0000", 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.450351950000002, 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008579171, 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072689+0000", 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072689+0000", 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072689+0000", 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6869999999999999e-06 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072694+0000", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9259999999999999e-06 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072699+0000", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7189999999999999e-06 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072703+0000", 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027345700000000001 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072976+0000", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4878999999999999e-05 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.072991+0000", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082250239999999992 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.081216+0000", 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2258e-05 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.081269+0000", 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008579171 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:4 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.456109346000002, 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002, 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.688 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:37:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066929+0000", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8779999999999993e-06 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066939+0000", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.431e-05 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066953+0000", 2024-02-10T08:37:17.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3720000000000005e-06 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066962+0000", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040994600000000001 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071061+0000", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0001153 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071177+0000", 2024-02-10T08:37:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014616299999999999 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071323+0000", 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020771520000000001 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.073400+0000", 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00123847 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.074638+0000", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082236259999999995 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082862+0000", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7891e-05 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082900+0000", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6568e-05 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082926+0000", 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(8.0 GRANT e119)", 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:54.069289+0000", 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 23.453752435999998, 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072487000000000003, 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.069289+0000", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.069289+0000", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.069289+0000", 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0430000000000002e-06 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.069291+0000", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035779129999999998 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.072868+0000", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.072876+0000", 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0036404440000000001 2024-02-10T08:37:17.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.076516+0000", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1958e-05 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.076528+0000", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5600000000000001e-06 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.076530+0000", 2024-02-10T08:37:17.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6399999999999997e-06 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.076538+0000", 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072487000000000003 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:5 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e121)", 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 22.483306753000001, 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002, 2024-02-10T08:37:17.698 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039732+0000", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1054e-05 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039743+0000", 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2641e-05 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039755+0000", 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9560000000000003e-06 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039764+0000", 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9907999999999997e-05 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039814+0000", 2024-02-10T08:37:17.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6459999999999993e-06 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039822+0000", 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030549936 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070372+0000", 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4599999999999996e-05 2024-02-10T08:37:17.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070456+0000", 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019660001 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.090116+0000", 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.705 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.41800.0:2 89.6 e139/138 89:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:13.016040+0000", 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5070018169999999, 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011567604, 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016040+0000", 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016040+0000", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016039+0000", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3409999999999992e-06 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016048+0000", 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0270000000000001e-06 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016054+0000", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9140000000000002e-06 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.016061+0000", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035231310000000001 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.019585+0000", 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0282000000000001e-05 2024-02-10T08:37:17.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.019595+0000", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079582490000000006 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.027553+0000", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4048000000000002e-05 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.027607+0000", 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011567604 2024-02-10T08:37:17.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.2.0:177 83.16 e139/135 83:6a183d1d:test-rados-api-smithi107-112490-12::bar:head v 139'1, mlcod=0'0)", 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:13.100997+0000", 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.4220439000000002, 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044994739999999998, 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.100997+0000", 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.100997+0000", 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.100995+0000", 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2638e-05 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.101008+0000", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6232999999999999e-05 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.101024+0000", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4270000000000002e-06 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.101032+0000", 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003763022 2024-02-10T08:37:17.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.104795+0000", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8539e-05 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.104813+0000", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00065723500000000005 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.105470+0000", 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6461e-05 2024-02-10T08:37:17.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.105497+0000", 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044994739999999998 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26977.0:63 66.1b 66:d9f0651a:test-rados-api-smithi107-112525-13::foo:head [write 0~128] snapc 16=[16] ondisk+write+known_if_redirected e141)", 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.4863147130000001, 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635, 2024-02-10T08:37:17.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26977", 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 63 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036723+0000", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4849e-05 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036738+0000", 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1770000000000001e-05 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036749+0000", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1699999999999997e-06 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036758+0000", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000209371 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036967+0000", 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0199999999999997e-06 2024-02-10T08:37:17.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036974+0000", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020105096999999999 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057079+0000", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120828 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057200+0000", 2024-02-10T08:37:17.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015114300000000001 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057351+0000", 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021251266000000001 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.078602+0000", 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032546469999999998 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081857+0000", 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6645e-05 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081884+0000", 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8924000000000001e-05 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081933+0000", 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7667999999999998e-05 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081980+0000", 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.50096.0:2 91.6 91:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.54732194, 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012101086, 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.721 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.50096", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1027110602", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:37:17.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975717+0000", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5640000000000002e-06 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975726+0000", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6456e-05 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975743+0000", 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6969999999999999e-06 2024-02-10T08:37:17.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975751+0000", 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3288999999999998e-05 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975805+0000", 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5888000000000002e-05 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975891+0000", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5280999999999998e-05 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975986+0000", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091295500000000002 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.976899+0000", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029384300000000001 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.977193+0000", 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010554218000000001 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987747+0000", 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4509999999999999e-05 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987791+0000", 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9099e-05 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987820+0000", 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012101086 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.50096.0:3 91.6 91:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e142)", 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:16.013926+0000", 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.5091149989999999, 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046731189999999999, 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:37:17.727 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.50096", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1027110602", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013926+0000", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013926+0000", 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:37:17.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013923+0000", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1471000000000001e-05 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013934+0000", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2997e-05 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013957+0000", 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0420000000000008e-06 2024-02-10T08:37:17.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.013967+0000", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0327000000000001e-05 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.014017+0000", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1570999999999995e-05 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.014098+0000", 2024-02-10T08:37:17.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045010099999999997 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:16.018599+0000", 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046731189999999999 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:37:17.731 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:17.792 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:37:17.792 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:37:17.793 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2024-02-10T08:37:17.793 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2024-02-10T08:37:17.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:37:17.798 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:17.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:17.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.748027+0000 osd.5 (osd.5) 73 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:37:17.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:15.839106+0000 osd.5 (osd.5) 74 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:37:17.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.097444+0000 mgr.y (mgr.14553) 181 : cluster [DBG] pgmap v188: 378 pgs: 2 active+clean+snaptrim_wait, 371 active+clean, 5 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 374 B/s, 3 objects/s recovering 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.227966+0000 osd.6 (osd.6) 59 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.232324+0000 osd.6 (osd.6) 60 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.270098+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.273965+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.487829+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:16.490722+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:16.989273+0000 mon.b (mon.1) 185 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.269923+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.272058+0000 mon.a (mon.0) 1155 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.275364+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.107:0/4152427963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.280108+0000 mon.a (mon.0) 1156 : audit [INF] from='client.51281 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.748027+0000 osd.5 (osd.5) 73 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:37:17.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:15.839106+0000 osd.5 (osd.5) 74 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:37:17.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.097444+0000 mgr.y (mgr.14553) 181 : cluster [DBG] pgmap v188: 378 pgs: 2 active+clean+snaptrim_wait, 371 active+clean, 5 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 374 B/s, 3 objects/s recovering 2024-02-10T08:37:17.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.227966+0000 osd.6 (osd.6) 59 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.232324+0000 osd.6 (osd.6) 60 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.270098+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.273965+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.487829+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:16.490722+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:16.989273+0000 mon.b (mon.1) 185 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.269923+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.272058+0000 mon.a (mon.0) 1155 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-02-10T08:37:17.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.275364+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.107:0/4152427963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.280108+0000 mon.a (mon.0) 1156 : audit [INF] from='client.51281 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.748027+0000 osd.5 (osd.5) 73 : cluster [DBG] 8.8 deep-scrub starts 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:15.839106+0000 osd.5 (osd.5) 74 : cluster [DBG] 8.8 deep-scrub ok 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.097444+0000 mgr.y (mgr.14553) 181 : cluster [DBG] pgmap v188: 378 pgs: 2 active+clean+snaptrim_wait, 371 active+clean, 5 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 374 B/s, 3 objects/s recovering 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.227966+0000 osd.6 (osd.6) 59 : cluster [DBG] 8.9 deep-scrub starts 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.232324+0000 osd.6 (osd.6) 60 : cluster [DBG] 8.9 deep-scrub ok 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.270098+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.273965+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.487829+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:16.490722+0000 osd.4 (osd.4) 79 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:16.989273+0000 mon.b (mon.1) 185 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:17.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.269923+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e143: 8 total, 8 up, 7 in 2024-02-10T08:37:17.939 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.272058+0000 mon.a (mon.0) 1155 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-02-10T08:37:17.939 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.275364+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.107:0/4152427963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.939 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.280108+0000 mon.a (mon.0) 1156 : audit [INF] from='client.51281 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:17.939 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:17.970 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:17.974 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-10T08:37:17.974 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] Global test environment set-up. 2024-02-10T08:37:17.974 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-10T08:37:17.974 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:36:30.355+0000 7f3c2ce4b700 -1 failed for service _ceph-mon._tcp 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:36:30.355+0000 7f3c2ce4b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (134 ms) 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2010 ms) 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2144 ms total) 2024-02-10T08:37:17.975 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: started 0x560fc43af110 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: started 0x560fc43b85f0 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: started 2 aios 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: waiting 0x560fc43af110 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: waiting 0x560fc43b85f0 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: done. 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (9927 ms) 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (9927 ms total) 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-10T08:37:17.976 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.Exec (13 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (3 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4362 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (28748 ms) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] 8 tests from LibRadosMisc (33128 ms total) 2024-02-10T08:37:17.977 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: 2024-02-10T08:37:17.978 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [----------] Global test environment tear-down 2024-02-10T08:37:17.978 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (47736 ms total) 2024-02-10T08:37:17.978 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-10T08:37:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:17.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:18.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:18.106 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:18.126 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:18.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:18.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:18.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:37:18.257 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:18.275 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:18.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:18.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:18.406 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:18.464 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:18.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:18.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 -- 172.21.15.107:0/3843775383 >> v1:172.21.15.195:6789/0 conn(0x7f4150103250 legacy=0x7f41501056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 -- 172.21.15.107:0/3843775383 shutdown_connections 2024-02-10T08:37:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 -- 172.21.15.107:0/3843775383 >> 172.21.15.107:0/3843775383 conn(0x7f41500f2510 msgr2=0x7f41500f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 -- 172.21.15.107:0/3843775383 shutdown_connections 2024-02-10T08:37:18.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 -- 172.21.15.107:0/3843775383 wait complete. 2024-02-10T08:37:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 librados: starting msgr at 2024-02-10T08:37:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.466+0000 7f415606c700 1 librados: starting objecter 2024-02-10T08:37:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 Processor -- start 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 -- start start 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 librados: setting wanted keys 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 librados: calling monclient init 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4150119840 con 0x7f41500fc910 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f415011aa20 con 0x7f4150103250 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f415606c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f415011bc20 con 0x7f41500ff710 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414effd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f41500ff710 0x7f415010af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33096/0 (socket says 172.21.15.107:33096) 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414effd700 1 -- 172.21.15.107:0/3804642615 learned_addr learned my addr 172.21.15.107:0/3804642615 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414cff9700 10 client.?.objecter ms_handle_connect 0x7f41500ff710 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414cff9700 10 client.?.objecter ms_handle_connect 0x7f4150103250 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.467+0000 7f414cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 10 client.?.objecter ms_handle_connect 0x7f41500fc910 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290043208 0 0) 0x7f415011aa20 con 0x7f4150103250 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4138002220 con 0x7f4150103250 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721249174 0 0) 0x7f415011bc20 con 0x7f41500ff710 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415011aa20 con 0x7f41500ff710 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790799027 0 0) 0x7f4150119840 con 0x7f41500fc910 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415011bc20 con 0x7f41500fc910 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 251857432 0 0) 0x7f4138002220 con 0x7f4150103250 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4150119840 con 0x7f4150103250 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f41480012d0 con 0x7f4150103250 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595512541 0 0) 0x7f415011aa20 con 0x7f41500ff710 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4138002220 con 0x7f41500ff710 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4144001810 con 0x7f41500ff710 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742469288 0 0) 0x7f415011bc20 con 0x7f41500fc910 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415011aa20 con 0x7f41500fc910 2024-02-10T08:37:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f41400017f0 con 0x7f41500fc910 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102934316 0 0) 0x7f4150119840 con 0x7f4150103250 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 >> v1:172.21.15.107:6790/0 conn(0x7f41500ff710 legacy=0x7f415010af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 >> v1:172.21.15.107:6789/0 conn(0x7f41500fc910 legacy=0x7f415010a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415011ce20 con 0x7f4150103250 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415011ac10 con 0x7f4150103250 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f415606c700 10 client.47359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415011b080 con 0x7f4150103250 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f415606c700 1 librados: init done 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4148003310 con 0x7f4150103250 2024-02-10T08:37:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.468+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4148003b60 con 0x7f4150103250 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.469+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c004f20 con 0x7f4150103250 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4148002650 con 0x7f4150103250 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(144..144 src has 50..144) v4 ==== 10136+0+0 (unknown 1098713928 0 0) 0x7f41480771a0 con 0x7f4150103250 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 10 client.47359.objecter ms_dispatch 0x7f4150001550 osd_map(144..144 src has 50..144) v4 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 3 client.47359.objecter handle_osd_map got epochs [144,144] > 0 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 3 client.47359.objecter handle_osd_map decoding full epoch 144 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.470+0000 7f414cff9700 20 client.47359.objecter dump_active .. 0 homeless 2024-02-10T08:37:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.471+0000 7f414cff9700 10 client.47359.objecter ms_handle_connect 0x7f4138063d00 2024-02-10T08:37:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.475+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4148049de0 con 0x7f4150103250 2024-02-10T08:37:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:18.521 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:18.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:18.594 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.659+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f413c0051d0 con 0x7f4150103250 2024-02-10T08:37:18.686 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:18.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:18.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:18.819 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:18.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.937+0000 7f414cff9700 1 -- 172.21.15.107:0/3804642615 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v145) v1 ==== 101+0+0 (unknown 3016494619 0 0) 0x7f4148048250 con 0x7f4150103250 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.941+0000 7f415606c700 10 librados: watch_flush enter 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.941+0000 7f415606c700 10 librados: watch_flush exit 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.941+0000 7f415606c700 20 client.47359.objecter shutdown clearing up homeless session... 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.941+0000 7f415606c700 10 client.47359.objecter successfully canceled tick 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.942+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4138063d00 legacy=0x7f41380661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:18.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.942+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 >> v1:172.21.15.195:6789/0 conn(0x7f4150103250 legacy=0x7f4150117f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.942+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 shutdown_connections 2024-02-10T08:37:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.942+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 >> 172.21.15.107:0/3804642615 conn(0x7f41500f2510 msgr2=0x7f4150060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.947+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 shutdown_connections 2024-02-10T08:37:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.947+0000 7f415606c700 1 -- 172.21.15.107:0/3804642615 wait complete. 2024-02-10T08:37:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:18.947+0000 7f415606c700 1 librados: shutdown 2024-02-10T08:37:18.953 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:37:18.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.613829+0000 osd.1 (osd.1) 91 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.673996+0000 osd.1 (osd.1) 92 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.777700+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.794054+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.930612+0000 mon.a (mon.0) 1157 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.930800+0000 mon.a (mon.0) 1158 : audit [INF] from='client.51281 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:17.930874+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.943830+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.107:0/1796167416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:17.992512+0000 mon.b (mon.1) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.097971+0000 mgr.y (mgr.14553) 182 : cluster [DBG] pgmap v191: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 767 B/s rd, 1 op/s; 246 B/s, 3 objects/s recovering 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.195349+0000 mon.a (mon.0) 1160 : audit [INF] from='client.46837 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.661103+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.107:0/3804642615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:18.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.661593+0000 mon.a (mon.0) 1161 : audit [INF] from='client.47359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.613829+0000 osd.1 (osd.1) 91 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.673996+0000 osd.1 (osd.1) 92 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.777700+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.794054+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.930612+0000 mon.a (mon.0) 1157 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.930800+0000 mon.a (mon.0) 1158 : audit [INF] from='client.51281 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:17.930874+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.943830+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.107:0/1796167416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:17.992512+0000 mon.b (mon.1) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:18.977 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.097971+0000 mgr.y (mgr.14553) 182 : cluster [DBG] pgmap v191: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 767 B/s rd, 1 op/s; 246 B/s, 3 objects/s recovering 2024-02-10T08:37:18.978 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.195349+0000 mon.a (mon.0) 1160 : audit [INF] from='client.46837 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:18.978 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.661103+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.107:0/3804642615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:18.978 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.661593+0000 mon.a (mon.0) 1161 : audit [INF] from='client.47359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.613829+0000 osd.1 (osd.1) 91 : cluster [DBG] 8.3 deep-scrub starts 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.673996+0000 osd.1 (osd.1) 92 : cluster [DBG] 8.3 deep-scrub ok 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.777700+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.794054+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.930612+0000 mon.a (mon.0) 1157 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi107-112513-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.930800+0000 mon.a (mon.0) 1158 : audit [INF] from='client.51281 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi107-112411-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:17.930874+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e144: 8 total, 8 up, 7 in 2024-02-10T08:37:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.943830+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.107:0/1796167416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:17.992512+0000 mon.b (mon.1) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.097971+0000 mgr.y (mgr.14553) 182 : cluster [DBG] pgmap v191: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 767 B/s rd, 1 op/s; 246 B/s, 3 objects/s recovering 2024-02-10T08:37:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.195349+0000 mon.a (mon.0) 1160 : audit [INF] from='client.46837 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.661103+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.107:0/3804642615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.661593+0000 mon.a (mon.0) 1161 : audit [INF] from='client.47359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:19.978 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.259188+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.260163+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.564549+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.565295+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.719342+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.722811+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.927832+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.937215+0000 mon.a (mon.0) 1163 : audit [INF] from='client.46837 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.937296+0000 mon.a (mon.0) 1164 : audit [INF] from='client.47359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:18.937340+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.951677+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.955975+0000 mon.a (mon.0) 1166 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:18.993364+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.259188+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.260163+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.564549+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.565295+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:37:19.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.719342+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.722811+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.927832+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.937215+0000 mon.a (mon.0) 1163 : audit [INF] from='client.46837 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.937296+0000 mon.a (mon.0) 1164 : audit [INF] from='client.47359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:18.937340+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.951677+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.955975+0000 mon.a (mon.0) 1166 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:19.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:18.993364+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:19.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:20.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:20.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.259188+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:37:20.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.260163+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:37:20.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.564549+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:37:20.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.565295+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:37:20.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.719342+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.722811+0000 osd.5 (osd.5) 76 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.927832+0000 mon.a (mon.0) 1162 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.937215+0000 mon.a (mon.0) 1163 : audit [INF] from='client.46837 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi107-112403-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.937296+0000 mon.a (mon.0) 1164 : audit [INF] from='client.47359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:18.937340+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e145: 8 total, 8 up, 7 in 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.951677+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.955975+0000 mon.a (mon.0) 1166 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:20.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:18.993364+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:20.901 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:20.901 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:37:20.901 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:37:20.901 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:20.961 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:20.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:21.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:21.096 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:21.148 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:21.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.309320+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.310248+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.380974+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.382699+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.599798+0000 osd.1 (osd.1) 93 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.602888+0000 osd.1 (osd.1) 94 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.830286+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.e deep-scrub starts 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.833283+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.e deep-scrub ok 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:19.945499+0000 mon.a (mon.0) 1167 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:19.945689+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:19.947586+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.107:0/875340312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:19.948531+0000 mon.a (mon.0) 1169 : audit [INF] from='client.52592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:19.994112+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:20.098493+0000 mgr.y (mgr.14553) 183 : cluster [DBG] pgmap v194: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:20.917283+0000 mon.a (mon.0) 1170 : audit [INF] from='client.52592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:20.917516+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:20.920902+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.107:0/3974906254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.203 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:20.923115+0000 mon.a (mon.0) 1172 : audit [INF] from='client.47371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:21.280 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:21.333 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.309320+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.310248+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.380974+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.382699+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.599798+0000 osd.1 (osd.1) 93 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.602888+0000 osd.1 (osd.1) 94 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.830286+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.e deep-scrub starts 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.833283+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.e deep-scrub ok 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:19.945499+0000 mon.a (mon.0) 1167 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:19.945689+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:19.947586+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.107:0/875340312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:19.948531+0000 mon.a (mon.0) 1169 : audit [INF] from='client.52592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:19.994112+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:20.098493+0000 mgr.y (mgr.14553) 183 : cluster [DBG] pgmap v194: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:20.917283+0000 mon.a (mon.0) 1170 : audit [INF] from='client.52592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:20.917516+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:20.920902+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.107:0/3974906254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:20.923115+0000 mon.a (mon.0) 1172 : audit [INF] from='client.47371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.309320+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.310248+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.380974+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.382699+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.599798+0000 osd.1 (osd.1) 93 : cluster [DBG] 8.7 deep-scrub starts 2024-02-10T08:37:21.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.602888+0000 osd.1 (osd.1) 94 : cluster [DBG] 8.7 deep-scrub ok 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.830286+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.e deep-scrub starts 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.833283+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.e deep-scrub ok 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:19.945499+0000 mon.a (mon.0) 1167 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:19.945689+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e146: 8 total, 8 up, 7 in 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:19.947586+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.107:0/875340312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:19.948531+0000 mon.a (mon.0) 1169 : audit [INF] from='client.52592 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:19.994112+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:20.098493+0000 mgr.y (mgr.14553) 183 : cluster [DBG] pgmap v194: 354 pgs: 72 unknown, 279 active+clean, 3 active+clean+snaptrim; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:20.917283+0000 mon.a (mon.0) 1170 : audit [INF] from='client.52592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi107-112411-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:20.917516+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e147: 8 total, 8 up, 7 in 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:20.920902+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.107:0/3974906254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:20.923115+0000 mon.a (mon.0) 1172 : audit [INF] from='client.47371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:21.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:21.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T08:37:21.728 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:21.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:21.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:21.859 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:21.877 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:21.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:21.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:21.983 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:22.016 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:22.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.089+0000 7f746b761700 1 -- 172.21.15.107:0/1120714711 >> v1:172.21.15.107:6790/0 conn(0x7f74640fc920 legacy=0x7f74640fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.089+0000 7f746b761700 1 -- 172.21.15.107:0/1120714711 shutdown_connections 2024-02-10T08:37:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.089+0000 7f746b761700 1 -- 172.21.15.107:0/1120714711 >> 172.21.15.107:0/1120714711 conn(0x7f74640f2540 msgr2=0x7f74640f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.089+0000 7f746b761700 1 -- 172.21.15.107:0/1120714711 shutdown_connections 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.089+0000 7f746b761700 1 -- 172.21.15.107:0/1120714711 wait complete. 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 librados: starting msgr at 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 librados: starting objecter 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 Processor -- start 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 -- start start 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 librados: setting wanted keys 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 librados: calling monclient init 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74641055b0 con 0x7f74640fc920 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f746411ab20 con 0x7f74640ff720 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f746b761700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f746411bcc0 con 0x7f7464103260 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f74694fd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f74640fc920 0x7f74640fb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43256/0 (socket says 172.21.15.107:43256) 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.090+0000 7f74694fd700 1 -- 172.21.15.107:0/947283264 learned_addr learned my addr 172.21.15.107:0/947283264 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter ms_handle_connect 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter ms_handle_connect 0x7f74640fc920 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter ms_handle_connect 0x7f7464103260 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845830825 0 0) 0x7f746411ab20 con 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7450003630 con 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380912901 0 0) 0x7f746411bcc0 con 0x7f7464103260 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746411ab20 con 0x7f7464103260 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022478467 0 0) 0x7f7450003630 con 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f746411bcc0 con 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f74540028f0 con 0x7f74640ff720 2024-02-10T08:37:22.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898474048 0 0) 0x7f746411ab20 con 0x7f7464103260 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7450003630 con 0x7f7464103260 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f745c001770 con 0x7f7464103260 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3319773424 0 0) 0x7f746411bcc0 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 >> v1:172.21.15.107:6790/0 conn(0x7f7464103260 legacy=0x7f74641183f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 >> v1:172.21.15.107:6789/0 conn(0x7f74640fc920 legacy=0x7f74640fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746411ce60 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7464119b70 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f746b761700 10 client.47377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f74540017d0 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f746411a020 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f746b761700 1 librados: init done 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.091+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7454003310 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.092+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7448004f20 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7454003490 con 0x7f74640ff720 2024-02-10T08:37:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(148..148 src has 50..148) v4 ==== 10160+0+0 (unknown 745367412 0 0) 0x7f7454076cd0 con 0x7f74640ff720 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 10 client.47377.objecter ms_dispatch 0x7f7464001550 osd_map(148..148 src has 50..148) v4 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 3 client.47377.objecter handle_osd_map got epochs [148,148] > 0 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 3 client.47377.objecter handle_osd_map decoding full epoch 148 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.093+0000 7f745a7fc700 20 client.47377.objecter dump_active .. 0 homeless 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.094+0000 7f745a7fc700 10 client.47377.objecter ms_handle_connect 0x7f7450063bd0 2024-02-10T08:37:22.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.098+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7454049890 con 0x7f74640ff720 2024-02-10T08:37:22.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:22.234 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:22.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.276+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7448000b90 con 0x7f7450063bd0 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 -- 172.21.15.107:0/3244727365 >> v1:172.21.15.107:6790/0 conn(0x7fdaa4104350 legacy=0x7fdaa41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 -- 172.21.15.107:0/3244727365 shutdown_connections 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 -- 172.21.15.107:0/3244727365 >> 172.21.15.107:0/3244727365 conn(0x7fdaa40f2510 msgr2=0x7fdaa40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 -- 172.21.15.107:0/3244727365 shutdown_connections 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 -- 172.21.15.107:0/3244727365 wait complete. 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 librados: starting msgr at 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.197+0000 7fdaa9876700 1 librados: starting objecter 2024-02-10T08:37:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 Processor -- start 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 -- start start 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 librados: setting wanted keys 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 librados: calling monclient init 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdaa4117470 con 0x7fdaa40f95d0 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdaa4118650 con 0x7fdaa4104350 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa9876700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdaa4119850 con 0x7fdaa40f8a20 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa27fc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdaa40f95d0 0x7fdaa40ffed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43268/0 (socket says 172.21.15.107:43268) 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fdaa27fc700 1 -- 172.21.15.107:0/1619648577 learned_addr learned my addr 172.21.15.107:0/1619648577 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter ms_handle_connect 0x7fdaa40f95d0 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter ms_handle_connect 0x7fdaa40f8a20 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter ms_handle_connect 0x7fdaa4104350 2024-02-10T08:37:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692055653 0 0) 0x7fdaa4119850 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda8c002220 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174911837 0 0) 0x7fdaa4118650 con 0x7fdaa4104350 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.198+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4119850 con 0x7fdaa4104350 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122629230 0 0) 0x7fda8c002220 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa4118650 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fda94001800 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852349509 0 0) 0x7fdaa4119850 con 0x7fdaa4104350 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda8c002220 con 0x7fdaa4104350 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fda9c002ea0 con 0x7fdaa4104350 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2236361523 0 0) 0x7fdaa4118650 con 0x7fdaa40f8a20 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.195:6789/0 conn(0x7fdaa4104350 legacy=0x7fdaa4115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6789/0 conn(0x7fdaa40f95d0 legacy=0x7fdaa40ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa411aa50 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa4119a40 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fdaa9876700 10 client.52616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa4119eb0 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fdaa9876700 1 librados: init done 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fda94001ec0 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.199+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fda94003360 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.200+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fda90004f00 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.200+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fda94001a20 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(148..148 src has 50..148) v4 ==== 10160+0+0 (unknown 745367412 0 0) 0x7fda94076d00 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 osd_map(148..148 src has 50..148) v4 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 3 client.52616.objecter handle_osd_map got epochs [148,148] > 0 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 3 client.52616.objecter handle_osd_map decoding full epoch 148 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 20 client.52616.objecter dump_active .. 0 homeless 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v148) v1 ==== 54+0+16 (unknown 2551583071 0 2420000192) 0x7fda94079110 con 0x7fdaa40f8a20 2024-02-10T08:37:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda8c063b40 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 10 client.52616.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 1 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 1 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 0 1 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 4 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 10 client.52616.objecter _send_command 1 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda90004950 con 0x7fda90004530 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.201+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda90004530 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.202+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda90004950 con 0x7fda90004530 2024-02-10T08:37:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.202+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.202+0000 7fda8bfff700 10 client.52616.objecter _finish_command 1 = system:0 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.202+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 4 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.202+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 0 1 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 10 client.52616.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 2 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 2 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda900051a0 osd=0 3 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 0 2 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 4 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 10 client.52616.objecter _send_command 2 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.221+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda90004d30 con 0x7fda90004530 2024-02-10T08:37:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda90004d30 con 0x7fda90004530 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fda8bfff700 10 client.52616.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 4 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 0 2 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fdaa9876700 10 client.52616.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.222+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 1 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 4 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 10 client.52616.objecter _send_command 3 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda9000c000 con 0x7fda90004150 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda90004150 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda9000c000 con 0x7fda90004150 2024-02-10T08:37:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 10 client.52616.objecter _finish_command 3 = system:0 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.223+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 1 3 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 10 client.52616.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90003fe0 osd=1 3 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 1 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 10 client.52616.objecter _send_command 4 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.240+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda90001850 con 0x7fda90004150 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.241+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda90001850 con 0x7fda90004150 2024-02-10T08:37:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.241+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.241+0000 7fda8bfff700 10 client.52616.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.241+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 4 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.241+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 1 4 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 10 client.52616.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 5 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 5 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 2 5 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 4 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 10 client.52616.objecter _send_command 5 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda9000cce0 con 0x7fda9000c170 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.242+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda9000c170 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.243+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda9000cce0 con 0x7fda9000c170 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.243+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.243+0000 7fda8bfff700 10 client.52616.objecter _finish_command 5 = system:0 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.243+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 4 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.243+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 2 5 2024-02-10T08:37:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 10 client.52616.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 6 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 6 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90001b60 osd=2 3 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 2 6 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 4 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 10 client.52616.objecter _send_command 6 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.262+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda900032e0 con 0x7fda9000c170 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda900032e0 con 0x7fda9000c170 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 10 client.52616.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 4 2024-02-10T08:37:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 2 6 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 10 client.52616.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 7 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 7 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 3 7 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 4 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 10 client.52616.objecter _send_command 7 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda90019f30 con 0x7fda90016290 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.264+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda90016290 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.265+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda90019f30 con 0x7fda90016290 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.265+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.265+0000 7fda8bfff700 10 client.52616.objecter _finish_command 7 = system:0 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.265+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 4 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.265+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 3 7 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 10 client.52616.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 8 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 8 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda900036f0 osd=3 3 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 3 8 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 4 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 10 client.52616.objecter _send_command 8 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.282+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda9000d0c0 con 0x7fda90016290 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda9000d0c0 con 0x7fda90016290 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fda8bfff700 10 client.52616.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 4 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 3 8 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 10 client.52616.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 9 2024-02-10T08:37:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 9 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 4 9 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 4 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 10 client.52616.objecter _send_command 9 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.283+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda9000d390 con 0x7fda9000e460 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.284+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda9000e460 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.285+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda9000d390 con 0x7fda9000e460 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.285+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.285+0000 7fda8bfff700 10 client.52616.objecter _finish_command 9 = system:0 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.285+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 4 2024-02-10T08:37:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.285+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 4 9 2024-02-10T08:37:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:22.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 10 client.52616.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 10 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 10 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9000d0e0 osd=4 3 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 4 10 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 4 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 10 client.52616.objecter _send_command 10 2024-02-10T08:37:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.313+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda9000d390 con 0x7fda9000e460 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.314+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda9000d390 con 0x7fda9000e460 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.314+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.314+0000 7fda8bfff700 10 client.52616.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.314+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 4 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.314+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 4 10 2024-02-10T08:37:22.316 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 10 client.52616.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 11 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 11 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 5 11 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 4 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 10 client.52616.objecter _send_command 11 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda9001c410 con 0x7fda9001be70 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.315+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda9001be70 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.316+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda9001c410 con 0x7fda9001be70 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.316+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.316+0000 7fda8bfff700 10 client.52616.objecter _finish_command 11 = system:0 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.316+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 4 2024-02-10T08:37:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.316+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 5 11 2024-02-10T08:37:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.318+0000 7f745a7fc700 1 -- 172.21.15.107:0/947283264 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+871692 (unknown 2965378022 0 45729077) 0x7f7448000b90 con 0x7f7450063bd0 2024-02-10T08:37:22.331 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 10 librados: watch_flush enter 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 10 librados: watch_flush exit 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 20 client.47377.objecter shutdown clearing up homeless session... 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 10 client.47377.objecter successfully canceled tick 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7450063bd0 legacy=0x7f7450066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 >> v1:172.21.15.195:6789/0 conn(0x7f74640ff720 legacy=0x7f7464114cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 shutdown_connections 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 >> 172.21.15.107:0/947283264 conn(0x7f74640f2540 msgr2=0x7f74641066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 shutdown_connections 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 -- 172.21.15.107:0/947283264 wait complete. 2024-02-10T08:37:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.325+0000 7f746b761700 1 librados: shutdown 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 10 client.52616.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 12 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 12 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9000d390 osd=5 3 2024-02-10T08:37:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 5 12 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 4 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 10 client.52616.objecter _send_command 12 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.334+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda9001c410 con 0x7fda9001be70 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.336+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda9001c410 con 0x7fda9001be70 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.336+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.336+0000 7fda8bfff700 10 client.52616.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.336+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 4 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.336+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 5 12 2024-02-10T08:37:22.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 10 client.52616.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 13 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 13 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 6 13 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 4 2024-02-10T08:37:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 10 client.52616.objecter _send_command 13 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda900270c0 con 0x7fda90023530 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.337+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda90023530 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.338+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda900270c0 con 0x7fda90023530 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.338+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.338+0000 7fda8bfff700 10 client.52616.objecter _finish_command 13 = system:0 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.338+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 4 2024-02-10T08:37:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.338+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 6 13 2024-02-10T08:37:22.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:20.331659+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:37:22.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:20.333341+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:37:22.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:20.994965+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:22.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:21.201305+0000 mgr.y (mgr.14553) 184 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:22.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:21.919757+0000 mon.a (mon.0) 1173 : audit [INF] from='client.47371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.919958+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:20.331659+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:20.333341+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:20.994965+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:21.201305+0000 mgr.y (mgr.14553) 184 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:21.919757+0000 mon.a (mon.0) 1173 : audit [INF] from='client.47371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:22.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.919958+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-10T08:37:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 10 client.52616.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 14 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 14 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda9001c410 osd=6 3 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 6 14 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 4 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 10 client.52616.objecter _send_command 14 2024-02-10T08:37:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.353+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda90027080 con 0x7fda90023530 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda90027080 con 0x7fda90023530 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 10 client.52616.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 4 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 6 14 2024-02-10T08:37:22.356 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 10 client.52616.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 15 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 15 2024-02-10T08:37:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 7 15 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 4 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 10 client.52616.objecter _send_command 15 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda90028480 con 0x7fda90027740 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.355+0000 7fda8bfff700 10 client.52616.objecter ms_handle_connect 0x7fda90027740 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.356+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fda90028480 con 0x7fda90027740 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.356+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.356+0000 7fda8bfff700 10 client.52616.objecter _finish_command 15 = system:0 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.356+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 4 2024-02-10T08:37:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.356+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 7 15 2024-02-10T08:37:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 10 client.52616.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign -1 16 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 20 client.52616.objecter _get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_remove -1 16 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 20 client.52616.objecter get_session s=0x7fda90027080 osd=7 3 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 15 client.52616.objecter _session_command_op_assign 7 16 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 4 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 10 client.52616.objecter _send_command 16 2024-02-10T08:37:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.370+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda90028650 con 0x7fda90027740 2024-02-10T08:37:22.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fda8bfff700 1 -- 172.21.15.107:0/1619648577 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fda90028650 con 0x7fda90027740 2024-02-10T08:37:22.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fda8bfff700 10 client.52616.objecter ms_dispatch 0x7fdaa4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:22.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fda8bfff700 10 client.52616.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fda8bfff700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 4 2024-02-10T08:37:22.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fda8bfff700 15 client.52616.objecter _session_command_op_remove 7 16 2024-02-10T08:37:22.374 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 librados: watch_flush enter 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 librados: watch_flush exit 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.0 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6801/3321596972 conn(0x7fda90004530 legacy=0x7fda90002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900051a0 osd=0 1 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.1 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6805/133516126 conn(0x7fda90004150 legacy=0x7fda9000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90003fe0 osd=1 1 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.2 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6809/3575881728 conn(0x7fda9000c170 legacy=0x7fda9000c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90001b60 osd=2 1 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.3 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6813/2871825691 conn(0x7fda90016290 legacy=0x7fda90018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda900036f0 osd=3 1 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.4 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa37fe700 1 -- 172.21.15.107:0/1619648577 reap_dead start 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.195:6800/2499671285 conn(0x7fda9000e460 legacy=0x7fda9001a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa37fe700 1 -- 172.21.15.107:0/1619648577 reap_dead start 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d0e0 osd=4 1 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.5 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.195:6804/4178321486 conn(0x7fda9001be70 legacy=0x7fda90021900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9000d390 osd=5 1 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.6 2024-02-10T08:37:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.195:6808/1417479017 conn(0x7fda90023530 legacy=0x7fda90025920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda9001c410 osd=6 1 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter close_session for osd.7 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.195:6812/1934176928 conn(0x7fda90027740 legacy=0x7fda90027ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter put_session s=0x7fda90027080 osd=7 1 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 20 client.52616.objecter shutdown clearing up homeless session... 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 10 client.52616.objecter successfully canceled tick 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6800/2634842009 conn(0x7fda8c063b40 legacy=0x7fda8c066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> v1:172.21.15.107:6790/0 conn(0x7fdaa40f8a20 legacy=0x7fdaa40ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa37fe700 1 -- 172.21.15.107:0/1619648577 reap_dead start 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 shutdown_connections 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.372+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 >> 172.21.15.107:0/1619648577 conn(0x7fdaa40f2510 msgr2=0x7fdaa41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.373+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 shutdown_connections 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.373+0000 7fdaa9876700 1 -- 172.21.15.107:0/1619648577 wait complete. 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:22.373+0000 7fdaa9876700 1 librados: shutdown 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:37:22.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:22.442 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:22.461 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:20.331659+0000 osd.6 (osd.6) 65 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:20.333341+0000 osd.6 (osd.6) 66 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:20.994965+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:21.201305+0000 mgr.y (mgr.14553) 184 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:21.919757+0000 mon.a (mon.0) 1173 : audit [INF] from='client.47371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi107-112403-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:22.463 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.919958+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e148: 8 total, 8 up, 7 in 2024-02-10T08:37:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:22.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:22.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:22.592 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:22.740 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:22.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:22.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:22.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:37:22.996 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:23.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:23.023 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-02-10T08:37:22.098649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226526964,"num_objects":407,"num_object_clones":55,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":1988,"num_read_kb":49885,"num_write":1776,"num_write_kb":228046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":25171856,"num_keys_recovered":1000,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":1062,"acting":1062,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":999,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2811588,"kb_used_data":678284,"kb_used_omap":0,"kb_used_meta":2133248,"kb_avail":653404476,"statfs":{"total":671965249536,"available":669086183424,"internally_reserved":0,"allocated":694562816,"data_stored":686899889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184445952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17778,"num_objects":155,"num_object_clones":54,"num_object_copies":465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":6,"num_read_kb":13,"num_write":156,"num_write_kb":170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001036"},"pg_stats":[{"pgid":"97.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922435+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.921780+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922796+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"97.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.926752+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"97.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.928264+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"97.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.923253+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"97.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.919004+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"97.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.939620+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.932210+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"97.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.921437+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.928205+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"97.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.923641+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.924263+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.933543+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922037+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.926977+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"97.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922131+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922946+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"97.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.925464+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"97.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.927255+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.924943+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"97.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.933215+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922836+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"97.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.939289+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.927768+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"97.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.917968+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"97.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.921524+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.925065+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"97.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922937+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"97.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.921360+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"97.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.922405+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"97.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:20.910602+0000","last_change":"2024-02-10T08:37:20.934881+0000","last_active":"2024-02-10T08:37:20.910602+0000","last_peered":"2024-02-10T08:37:20.910602+0000","last_clean":"2024-02-10T08:37:20.910602+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:20.910602+0000","last_undegraded":"2024-02-10T08:37:20.910602+0000","last_fullsized":"2024-02-10T08:37:20.910602+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:20.910602+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.910602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.0","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979686+0000","last_change":"2024-02-10T08:37:19.978982+0000","last_active":"2024-02-10T08:37:19.979686+0000","last_peered":"2024-02-10T08:37:19.979686+0000","last_clean":"2024-02-10T08:37:19.979686+0000","last_became_active":"2024-02-10T08:37:19.978613+0000","last_became_peered":"2024-02-10T08:37:19.978613+0000","last_unstale":"2024-02-10T08:37:19.979686+0000","last_undegraded":"2024-02-10T08:37:19.979686+0000","last_fullsized":"2024-02-10T08:37:19.979686+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"93.2","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.933175+0000","last_change":"2024-02-10T08:37:20.933608+0000","last_active":"2024-02-10T08:37:20.933175+0000","last_peered":"2024-02-10T08:37:20.933175+0000","last_clean":"2024-02-10T08:37:20.933175+0000","last_became_active":"2024-02-10T08:37:18.963829+0000","last_became_peered":"2024-02-10T08:37:18.963829+0000","last_unstale":"2024-02-10T08:37:20.933175+0000","last_undegraded":"2024-02-10T08:37:20.933175+0000","last_fullsized":"2024-02-10T08:37:20.933175+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1d","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.917856+0000","last_change":"2024-02-10T08:37:16.993956+0000","last_active":"2024-02-10T08:37:20.917856+0000","last_peered":"2024-02-10T08:37:20.917856+0000","last_clean":"2024-02-10T08:37:20.917856+0000","last_became_active":"2024-02-10T08:37:10.274801+0000","last_became_peered":"2024-02-10T08:37:10.274801+0000","last_unstale":"2024-02-10T08:37:20.917856+0000","last_undegraded":"2024-02-10T08:37:20.917856+0000","last_fullsized":"2024-02-10T08:37:20.917856+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.983339+0000","last_change":"2024-02-10T08:37:19.977809+0000","last_active":"2024-02-10T08:37:19.983339+0000","last_peered":"2024-02-10T08:37:19.983339+0000","last_clean":"2024-02-10T08:37:19.983339+0000","last_became_active":"2024-02-10T08:37:19.977530+0000","last_became_peered":"2024-02-10T08:37:19.977530+0000","last_unstale":"2024-02-10T08:37:19.983339+0000","last_undegraded":"2024-02-10T08:37:19.983339+0000","last_fullsized":"2024-02-10T08:37:19.983339+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"93.3","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.929393+0000","last_change":"2024-02-10T08:37:20.929662+0000","last_active":"2024-02-10T08:37:20.929393+0000","last_peered":"2024-02-10T08:37:20.929393+0000","last_clean":"2024-02-10T08:37:20.929393+0000","last_became_active":"2024-02-10T08:37:18.972606+0000","last_became_peered":"2024-02-10T08:37:18.972606+0000","last_unstale":"2024-02-10T08:37:20.929393+0000","last_undegraded":"2024-02-10T08:37:20.929393+0000","last_fullsized":"2024-02-10T08:37:20.929393+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1c","version":"137'5","reported_seq":84,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.928171+0000","last_change":"2024-02-10T08:37:16.971725+0000","last_active":"2024-02-10T08:37:20.928171+0000","last_peered":"2024-02-10T08:37:20.928171+0000","last_clean":"2024-02-10T08:37:20.928171+0000","last_became_active":"2024-02-10T08:36:54.078016+0000","last_became_peered":"2024-02-10T08:36:54.078016+0000","last_unstale":"2024-02-10T08:37:20.928171+0000","last_undegraded":"2024-02-10T08:37:20.928171+0000","last_fullsized":"2024-02-10T08:37:20.928171+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.2","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.981878+0000","last_change":"2024-02-10T08:37:19.980405+0000","last_active":"2024-02-10T08:37:19.981878+0000","last_peered":"2024-02-10T08:37:19.981878+0000","last_clean":"2024-02-10T08:37:19.981878+0000","last_became_active":"2024-02-10T08:37:19.980183+0000","last_became_peered":"2024-02-10T08:37:19.980183+0000","last_unstale":"2024-02-10T08:37:19.981878+0000","last_undegraded":"2024-02-10T08:37:19.981878+0000","last_fullsized":"2024-02-10T08:37:19.981878+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"93.0","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.937539+0000","last_change":"2024-02-10T08:37:20.939241+0000","last_active":"2024-02-10T08:37:20.937539+0000","last_peered":"2024-02-10T08:37:20.937539+0000","last_clean":"2024-02-10T08:37:20.937539+0000","last_became_active":"2024-02-10T08:37:18.975736+0000","last_became_peered":"2024-02-10T08:37:18.975736+0000","last_unstale":"2024-02-10T08:37:20.937539+0000","last_undegraded":"2024-02-10T08:37:20.937539+0000","last_fullsized":"2024-02-10T08:37:20.937539+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1f","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.931624+0000","last_change":"2024-02-10T08:37:16.946744+0000","last_active":"2024-02-10T08:37:20.931624+0000","last_peered":"2024-02-10T08:37:20.931624+0000","last_clean":"2024-02-10T08:37:20.931624+0000","last_became_active":"2024-02-10T08:36:54.050958+0000","last_became_peered":"2024-02-10T08:36:54.050958+0000","last_unstale":"2024-02-10T08:37:20.931624+0000","last_undegraded":"2024-02-10T08:37:20.931624+0000","last_fullsized":"2024-02-10T08:37:20.931624+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.3","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979955+0000","last_change":"2024-02-10T08:37:19.979157+0000","last_active":"2024-02-10T08:37:19.979955+0000","last_peered":"2024-02-10T08:37:19.979955+0000","last_clean":"2024-02-10T08:37:19.979955+0000","last_became_active":"2024-02-10T08:37:19.978876+0000","last_became_peered":"2024-02-10T08:37:19.978876+0000","last_unstale":"2024-02-10T08:37:19.979955+0000","last_undegraded":"2024-02-10T08:37:19.979955+0000","last_fullsized":"2024-02-10T08:37:19.979955+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"93.1","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.935330+0000","last_change":"2024-02-10T08:37:20.936366+0000","last_active":"2024-02-10T08:37:20.935330+0000","last_peered":"2024-02-10T08:37:20.935330+0000","last_clean":"2024-02-10T08:37:20.935330+0000","last_became_active":"2024-02-10T08:37:18.965029+0000","last_became_peered":"2024-02-10T08:37:18.965029+0000","last_unstale":"2024-02-10T08:37:20.935330+0000","last_undegraded":"2024-02-10T08:37:20.935330+0000","last_fullsized":"2024-02-10T08:37:20.935330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1e","version":"134'20","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.922064+0000","last_change":"2024-02-10T08:37:16.996169+0000","last_active":"2024-02-10T08:37:20.922064+0000","last_peered":"2024-02-10T08:37:20.922064+0000","last_clean":"2024-02-10T08:37:20.922064+0000","last_became_active":"2024-02-10T08:37:10.217958+0000","last_became_peered":"2024-02-10T08:37:10.217958+0000","last_unstale":"2024-02-10T08:37:20.922064+0000","last_undegraded":"2024-02-10T08:37:20.922064+0000","last_fullsized":"2024-02-10T08:37:20.922064+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.4","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.983294+0000","last_change":"2024-02-10T08:37:19.977197+0000","last_active":"2024-02-10T08:37:19.983294+0000","last_peered":"2024-02-10T08:37:19.983294+0000","last_clean":"2024-02-10T08:37:19.983294+0000","last_became_active":"2024-02-10T08:37:19.976935+0000","last_became_peered":"2024-02-10T08:37:19.976935+0000","last_unstale":"2024-02-10T08:37:19.983294+0000","last_undegraded":"2024-02-10T08:37:19.983294+0000","last_fullsized":"2024-02-10T08:37:19.983294+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"93.6","version":"147'5","reported_seq":18,"reported_epoch":147,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:37:20.939251+0000","last_change":"2024-02-10T08:37:20.939251+0000","last_active":"2024-02-10T08:37:20.939251+0000","last_peered":"2024-02-10T08:37:20.939251+0000","last_clean":"2024-02-10T08:37:20.939251+0000","last_became_active":"2024-02-10T08:37:18.964811+0000","last_became_peered":"2024-02-10T08:37:18.964811+0000","last_unstale":"2024-02-10T08:37:20.939251+0000","last_undegraded":"2024-02-10T08:37:20.939251+0000","last_fullsized":"2024-02-10T08:37:20.939251+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":16384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.19","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.940680+0000","last_change":"2024-02-10T08:37:16.945777+0000","last_active":"2024-02-10T08:37:20.940680+0000","last_peered":"2024-02-10T08:37:20.940680+0000","last_clean":"2024-02-10T08:37:20.940680+0000","last_became_active":"2024-02-10T08:37:10.135782+0000","last_became_peered":"2024-02-10T08:37:10.135782+0000","last_unstale":"2024-02-10T08:37:20.940680+0000","last_undegraded":"2024-02-10T08:37:20.940680+0000","last_fullsized":"2024-02-10T08:37:20.940680+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.5","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.974292+0000","last_change":"2024-02-10T08:37:19.973098+0000","last_active":"2024-02-10T08:37:19.974292+0000","last_peered":"2024-02-10T08:37:19.974292+0000","last_clean":"2024-02-10T08:37:19.974292+0000","last_became_active":"2024-02-10T08:37:19.972802+0000","last_became_peered":"2024-02-10T08:37:19.972802+0000","last_unstale":"2024-02-10T08:37:19.974292+0000","last_undegraded":"2024-02-10T08:37:19.974292+0000","last_fullsized":"2024-02-10T08:37:19.974292+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"93.7","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.932146+0000","last_change":"2024-02-10T08:37:20.933134+0000","last_active":"2024-02-10T08:37:20.932146+0000","last_peered":"2024-02-10T08:37:20.932146+0000","last_clean":"2024-02-10T08:37:20.932146+0000","last_became_active":"2024-02-10T08:37:18.976594+0000","last_became_peered":"2024-02-10T08:37:18.976594+0000","last_unstale":"2024-02-10T08:37:20.932146+0000","last_undegraded":"2024-02-10T08:37:20.932146+0000","last_fullsized":"2024-02-10T08:37:20.932146+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.18","version":"127'9","reported_seq":82,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925052+0000","last_change":"2024-02-10T08:37:16.941679+0000","last_active":"2024-02-10T08:37:20.925052+0000","last_peered":"2024-02-10T08:37:20.925052+0000","last_clean":"2024-02-10T08:37:20.925052+0000","last_became_active":"2024-02-10T08:37:10.124176+0000","last_became_peered":"2024-02-10T08:37:10.124176+0000","last_unstale":"2024-02-10T08:37:20.925052+0000","last_undegraded":"2024-02-10T08:37:20.925052+0000","last_fullsized":"2024-02-10T08:37:20.925052+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.6","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979907+0000","last_change":"2024-02-10T08:37:19.978945+0000","last_active":"2024-02-10T08:37:19.979907+0000","last_peered":"2024-02-10T08:37:19.979907+0000","last_clean":"2024-02-10T08:37:19.979907+0000","last_became_active":"2024-02-10T08:37:19.978647+0000","last_became_peered":"2024-02-10T08:37:19.978647+0000","last_unstale":"2024-02-10T08:37:19.979907+0000","last_undegraded":"2024-02-10T08:37:19.979907+0000","last_fullsized":"2024-02-10T08:37:19.979907+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"93.4","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.922467+0000","last_change":"2024-02-10T08:37:20.923118+0000","last_active":"2024-02-10T08:37:20.922467+0000","last_peered":"2024-02-10T08:37:20.922467+0000","last_clean":"2024-02-10T08:37:20.922467+0000","last_became_active":"2024-02-10T08:37:18.969387+0000","last_became_peered":"2024-02-10T08:37:18.969387+0000","last_unstale":"2024-02-10T08:37:20.922467+0000","last_undegraded":"2024-02-10T08:37:20.922467+0000","last_fullsized":"2024-02-10T08:37:20.922467+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1b","version":"141'1","reported_seq":39,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.928502+0000","last_change":"2024-02-10T08:37:16.970638+0000","last_active":"2024-02-10T08:37:20.928502+0000","last_peered":"2024-02-10T08:37:20.928502+0000","last_clean":"2024-02-10T08:37:20.928502+0000","last_became_active":"2024-02-10T08:37:10.262302+0000","last_became_peered":"2024-02-10T08:37:10.262302+0000","last_unstale":"2024-02-10T08:37:20.928502+0000","last_undegraded":"2024-02-10T08:37:20.928502+0000","last_fullsized":"2024-02-10T08:37:20.928502+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.7","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.977770+0000","last_change":"2024-02-10T08:37:19.976304+0000","last_active":"2024-02-10T08:37:19.977770+0000","last_peered":"2024-02-10T08:37:19.977770+0000","last_clean":"2024-02-10T08:37:19.977770+0000","last_became_active":"2024-02-10T08:37:19.975860+0000","last_became_peered":"2024-02-10T08:37:19.975860+0000","last_unstale":"2024-02-10T08:37:19.977770+0000","last_undegraded":"2024-02-10T08:37:19.977770+0000","last_fullsized":"2024-02-10T08:37:19.977770+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"93.5","version":"0'0","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.922405+0000","last_change":"2024-02-10T08:37:20.923311+0000","last_active":"2024-02-10T08:37:20.922405+0000","last_peered":"2024-02-10T08:37:20.922405+0000","last_clean":"2024-02-10T08:37:20.922405+0000","last_became_active":"2024-02-10T08:37:18.976145+0000","last_became_peered":"2024-02-10T08:37:18.976145+0000","last_unstale":"2024-02-10T08:37:20.922405+0000","last_undegraded":"2024-02-10T08:37:20.922405+0000","last_fullsized":"2024-02-10T08:37:20.922405+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:17.925274+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.925274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"66.1a","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.924849+0000","last_change":"2024-02-10T08:37:16.942815+0000","last_active":"2024-02-10T08:37:20.924849+0000","last_peered":"2024-02-10T08:37:20.924849+0000","last_clean":"2024-02-10T08:37:20.924849+0000","last_became_active":"2024-02-10T08:36:54.061830+0000","last_became_peered":"2024-02-10T08:36:54.061830+0000","last_unstale":"2024-02-10T08:37:20.924849+0000","last_undegraded":"2024-02-10T08:37:20.924849+0000","last_fullsized":"2024-02-10T08:37:20.924849+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.8","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.971613+0000","last_change":"2024-02-10T08:37:19.970670+0000","last_active":"2024-02-10T08:37:19.971613+0000","last_peered":"2024-02-10T08:37:19.971613+0000","last_clean":"2024-02-10T08:37:19.971613+0000","last_became_active":"2024-02-10T08:37:19.970400+0000","last_became_peered":"2024-02-10T08:37:19.970400+0000","last_unstale":"2024-02-10T08:37:19.971613+0000","last_undegraded":"2024-02-10T08:37:19.971613+0000","last_fullsized":"2024-02-10T08:37:19.971613+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.15","version":"140'4","reported_seq":83,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.935749+0000","last_change":"2024-02-10T08:37:16.954360+0000","last_active":"2024-02-10T08:37:20.935749+0000","last_peered":"2024-02-10T08:37:20.935749+0000","last_clean":"2024-02-10T08:37:20.935749+0000","last_became_active":"2024-02-10T08:36:54.075571+0000","last_became_peered":"2024-02-10T08:36:54.075571+0000","last_unstale":"2024-02-10T08:37:20.935749+0000","last_undegraded":"2024-02-10T08:37:20.935749+0000","last_fullsized":"2024-02-10T08:37:20.935749+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.9","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.983358+0000","last_change":"2024-02-10T08:37:19.982980+0000","last_active":"2024-02-10T08:37:19.983358+0000","last_peered":"2024-02-10T08:37:19.983358+0000","last_clean":"2024-02-10T08:37:19.983358+0000","last_became_active":"2024-02-10T08:37:19.982669+0000","last_became_peered":"2024-02-10T08:37:19.982669+0000","last_unstale":"2024-02-10T08:37:19.983358+0000","last_undegraded":"2024-02-10T08:37:19.983358+0000","last_fullsized":"2024-02-10T08:37:19.983358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.928651+0000","last_change":"2024-02-10T08:37:16.963858+0000","last_active":"2024-02-10T08:37:20.928651+0000","last_peered":"2024-02-10T08:37:20.928651+0000","last_clean":"2024-02-10T08:37:20.928651+0000","last_became_active":"2024-02-10T08:36:54.062360+0000","last_became_peered":"2024-02-10T08:36:54.062360+0000","last_unstale":"2024-02-10T08:37:20.928651+0000","last_undegraded":"2024-02-10T08:37:20.928651+0000","last_fullsized":"2024-02-10T08:37:20.928651+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979521+0000","last_change":"2024-02-10T08:37:19.978822+0000","last_active":"2024-02-10T08:37:19.979521+0000","last_peered":"2024-02-10T08:37:19.979521+0000","last_clean":"2024-02-10T08:37:19.979521+0000","last_became_active":"2024-02-10T08:37:19.978486+0000","last_became_peered":"2024-02-10T08:37:19.978486+0000","last_unstale":"2024-02-10T08:37:19.979521+0000","last_undegraded":"2024-02-10T08:37:19.979521+0000","last_fullsized":"2024-02-10T08:37:19.979521+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.17","version":"124'6","reported_seq":63,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.937805+0000","last_change":"2024-02-10T08:37:16.957932+0000","last_active":"2024-02-10T08:37:20.937805+0000","last_peered":"2024-02-10T08:37:20.937805+0000","last_clean":"2024-02-10T08:37:20.937805+0000","last_became_active":"2024-02-10T08:37:10.140473+0000","last_became_peered":"2024-02-10T08:37:10.140473+0000","last_unstale":"2024-02-10T08:37:20.937805+0000","last_undegraded":"2024-02-10T08:37:20.937805+0000","last_fullsized":"2024-02-10T08:37:20.937805+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.977946+0000","last_change":"2024-02-10T08:37:19.973471+0000","last_active":"2024-02-10T08:37:19.977946+0000","last_peered":"2024-02-10T08:37:19.977946+0000","last_clean":"2024-02-10T08:37:19.977946+0000","last_became_active":"2024-02-10T08:37:19.973202+0000","last_became_peered":"2024-02-10T08:37:19.973202+0000","last_unstale":"2024-02-10T08:37:19.977946+0000","last_undegraded":"2024-02-10T08:37:19.977946+0000","last_fullsized":"2024-02-10T08:37:19.977946+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.922231+0000","last_change":"2024-02-10T08:37:16.997510+0000","last_active":"2024-02-10T08:37:20.922231+0000","last_peered":"2024-02-10T08:37:20.922231+0000","last_clean":"2024-02-10T08:37:20.922231+0000","last_became_active":"2024-02-10T08:36:54.062972+0000","last_became_peered":"2024-02-10T08:36:54.062972+0000","last_unstale":"2024-02-10T08:37:20.922231+0000","last_undegraded":"2024-02-10T08:37:20.922231+0000","last_fullsized":"2024-02-10T08:37:20.922231+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.978333+0000","last_change":"2024-02-10T08:37:19.974182+0000","last_active":"2024-02-10T08:37:19.978333+0000","last_peered":"2024-02-10T08:37:19.978333+0000","last_clean":"2024-02-10T08:37:19.978333+0000","last_became_active":"2024-02-10T08:37:19.973885+0000","last_became_peered":"2024-02-10T08:37:19.973885+0000","last_unstale":"2024-02-10T08:37:19.978333+0000","last_undegraded":"2024-02-10T08:37:19.978333+0000","last_fullsized":"2024-02-10T08:37:19.978333+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.927929+0000","last_change":"2024-02-10T08:37:16.952389+0000","last_active":"2024-02-10T08:37:20.927929+0000","last_peered":"2024-02-10T08:37:20.927929+0000","last_clean":"2024-02-10T08:37:20.927929+0000","last_became_active":"2024-02-10T08:36:54.068919+0000","last_became_peered":"2024-02-10T08:36:54.068919+0000","last_unstale":"2024-02-10T08:37:20.927929+0000","last_undegraded":"2024-02-10T08:37:20.927929+0000","last_fullsized":"2024-02-10T08:37:20.927929+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.975365+0000","last_change":"2024-02-10T08:37:19.973940+0000","last_active":"2024-02-10T08:37:19.975365+0000","last_peered":"2024-02-10T08:37:19.975365+0000","last_clean":"2024-02-10T08:37:19.975365+0000","last_became_active":"2024-02-10T08:37:19.973610+0000","last_became_peered":"2024-02-10T08:37:19.973610+0000","last_unstale":"2024-02-10T08:37:19.975365+0000","last_undegraded":"2024-02-10T08:37:19.975365+0000","last_fullsized":"2024-02-10T08:37:19.975365+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.10","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925295+0000","last_change":"2024-02-10T08:37:16.941845+0000","last_active":"2024-02-10T08:37:20.925295+0000","last_peered":"2024-02-10T08:37:20.925295+0000","last_clean":"2024-02-10T08:37:20.925295+0000","last_became_active":"2024-02-10T08:37:10.217346+0000","last_became_peered":"2024-02-10T08:37:10.217346+0000","last_unstale":"2024-02-10T08:37:20.925295+0000","last_undegraded":"2024-02-10T08:37:20.925295+0000","last_fullsized":"2024-02-10T08:37:20.925295+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.977836+0000","last_change":"2024-02-10T08:37:19.977529+0000","last_active":"2024-02-10T08:37:19.977836+0000","last_peered":"2024-02-10T08:37:19.977836+0000","last_clean":"2024-02-10T08:37:19.977836+0000","last_became_active":"2024-02-10T08:37:19.977277+0000","last_became_peered":"2024-02-10T08:37:19.977277+0000","last_unstale":"2024-02-10T08:37:19.977836+0000","last_undegraded":"2024-02-10T08:37:19.977836+0000","last_fullsized":"2024-02-10T08:37:19.977836+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.13","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.927546+0000","last_change":"2024-02-10T08:37:16.964273+0000","last_active":"2024-02-10T08:37:20.927546+0000","last_peered":"2024-02-10T08:37:20.927546+0000","last_clean":"2024-02-10T08:37:20.927546+0000","last_became_active":"2024-02-10T08:37:10.200768+0000","last_became_peered":"2024-02-10T08:37:10.200768+0000","last_unstale":"2024-02-10T08:37:20.927546+0000","last_undegraded":"2024-02-10T08:37:20.927546+0000","last_fullsized":"2024-02-10T08:37:20.927546+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.975362+0000","last_change":"2024-02-10T08:37:19.974722+0000","last_active":"2024-02-10T08:37:19.975362+0000","last_peered":"2024-02-10T08:37:19.975362+0000","last_clean":"2024-02-10T08:37:19.975362+0000","last_became_active":"2024-02-10T08:37:19.974437+0000","last_became_peered":"2024-02-10T08:37:19.974437+0000","last_unstale":"2024-02-10T08:37:19.975362+0000","last_undegraded":"2024-02-10T08:37:19.975362+0000","last_fullsized":"2024-02-10T08:37:19.975362+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"66.12","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.929330+0000","last_change":"2024-02-10T08:37:16.952861+0000","last_active":"2024-02-10T08:37:20.929330+0000","last_peered":"2024-02-10T08:37:20.929330+0000","last_clean":"2024-02-10T08:37:20.929330+0000","last_became_active":"2024-02-10T08:36:54.080124+0000","last_became_peered":"2024-02-10T08:36:54.080124+0000","last_unstale":"2024-02-10T08:37:20.929330+0000","last_undegraded":"2024-02-10T08:37:20.929330+0000","last_fullsized":"2024-02-10T08:37:20.929330+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.10","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979667+0000","last_change":"2024-02-10T08:37:19.978870+0000","last_active":"2024-02-10T08:37:19.979667+0000","last_peered":"2024-02-10T08:37:19.979667+0000","last_clean":"2024-02-10T08:37:19.979667+0000","last_became_active":"2024-02-10T08:37:19.978206+0000","last_became_peered":"2024-02-10T08:37:19.978206+0000","last_unstale":"2024-02-10T08:37:19.979667+0000","last_undegraded":"2024-02-10T08:37:19.979667+0000","last_fullsized":"2024-02-10T08:37:19.979667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.d","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.917039+0000","last_change":"2024-02-10T08:37:16.991220+0000","last_active":"2024-02-10T08:37:20.917039+0000","last_peered":"2024-02-10T08:37:20.917039+0000","last_clean":"2024-02-10T08:37:20.917039+0000","last_became_active":"2024-02-10T08:37:10.163822+0000","last_became_peered":"2024-02-10T08:37:10.163822+0000","last_unstale":"2024-02-10T08:37:20.917039+0000","last_undegraded":"2024-02-10T08:37:20.917039+0000","last_fullsized":"2024-02-10T08:37:20.917039+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.11","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.979723+0000","last_change":"2024-02-10T08:37:19.978349+0000","last_active":"2024-02-10T08:37:19.979723+0000","last_peered":"2024-02-10T08:37:19.979723+0000","last_clean":"2024-02-10T08:37:19.979723+0000","last_became_active":"2024-02-10T08:37:19.978062+0000","last_became_peered":"2024-02-10T08:37:19.978062+0000","last_unstale":"2024-02-10T08:37:19.979723+0000","last_undegraded":"2024-02-10T08:37:19.979723+0000","last_fullsized":"2024-02-10T08:37:19.979723+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.c","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925816+0000","last_change":"2024-02-10T08:37:16.954051+0000","last_active":"2024-02-10T08:37:20.925816+0000","last_peered":"2024-02-10T08:37:20.925816+0000","last_clean":"2024-02-10T08:37:20.925816+0000","last_became_active":"2024-02-10T08:36:54.064037+0000","last_became_peered":"2024-02-10T08:36:54.064037+0000","last_unstale":"2024-02-10T08:37:20.925816+0000","last_undegraded":"2024-02-10T08:37:20.925816+0000","last_fullsized":"2024-02-10T08:37:20.925816+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.12","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.976501+0000","last_change":"2024-02-10T08:37:19.976162+0000","last_active":"2024-02-10T08:37:19.976501+0000","last_peered":"2024-02-10T08:37:19.976501+0000","last_clean":"2024-02-10T08:37:19.976501+0000","last_became_active":"2024-02-10T08:37:19.975883+0000","last_became_peered":"2024-02-10T08:37:19.975883+0000","last_unstale":"2024-02-10T08:37:19.976501+0000","last_undegraded":"2024-02-10T08:37:19.976501+0000","last_fullsized":"2024-02-10T08:37:19.976501+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.918869+0000","last_change":"2024-02-10T08:37:16.993543+0000","last_active":"2024-02-10T08:37:20.918869+0000","last_peered":"2024-02-10T08:37:20.918869+0000","last_clean":"2024-02-10T08:37:20.918869+0000","last_became_active":"2024-02-10T08:37:10.173213+0000","last_became_peered":"2024-02-10T08:37:10.173213+0000","last_unstale":"2024-02-10T08:37:20.918869+0000","last_undegraded":"2024-02-10T08:37:20.918869+0000","last_fullsized":"2024-02-10T08:37:20.918869+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.13","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969098+0000","last_change":"2024-02-10T08:37:19.968238+0000","last_active":"2024-02-10T08:37:19.969098+0000","last_peered":"2024-02-10T08:37:19.969098+0000","last_clean":"2024-02-10T08:37:19.969098+0000","last_became_active":"2024-02-10T08:37:19.967850+0000","last_became_peered":"2024-02-10T08:37:19.967850+0000","last_unstale":"2024-02-10T08:37:19.969098+0000","last_undegraded":"2024-02-10T08:37:19.969098+0000","last_fullsized":"2024-02-10T08:37:19.969098+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.e","version":"0'0","reported_seq":38,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925459+0000","last_change":"2024-02-10T08:37:16.963866+0000","last_active":"2024-02-10T08:37:20.925459+0000","last_peered":"2024-02-10T08:37:20.925459+0000","last_clean":"2024-02-10T08:37:20.925459+0000","last_became_active":"2024-02-10T08:37:10.167455+0000","last_became_peered":"2024-02-10T08:37:10.167455+0000","last_unstale":"2024-02-10T08:37:20.925459+0000","last_undegraded":"2024-02-10T08:37:20.925459+0000","last_fullsized":"2024-02-10T08:37:20.925459+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.14","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.981643+0000","last_change":"2024-02-10T08:37:19.981261+0000","last_active":"2024-02-10T08:37:19.981643+0000","last_peered":"2024-02-10T08:37:19.981643+0000","last_clean":"2024-02-10T08:37:19.981643+0000","last_became_active":"2024-02-10T08:37:19.980963+0000","last_became_peered":"2024-02-10T08:37:19.980963+0000","last_unstale":"2024-02-10T08:37:19.981643+0000","last_undegraded":"2024-02-10T08:37:19.981643+0000","last_fullsized":"2024-02-10T08:37:19.981643+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.9","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.920360+0000","last_change":"2024-02-10T08:37:16.987795+0000","last_active":"2024-02-10T08:37:20.920360+0000","last_peered":"2024-02-10T08:37:20.920360+0000","last_clean":"2024-02-10T08:37:20.920360+0000","last_became_active":"2024-02-10T08:36:54.066141+0000","last_became_peered":"2024-02-10T08:36:54.066141+0000","last_unstale":"2024-02-10T08:37:20.920360+0000","last_undegraded":"2024-02-10T08:37:20.920360+0000","last_fullsized":"2024-02-10T08:37:20.920360+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.15","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.975195+0000","last_change":"2024-02-10T08:37:19.974402+0000","last_active":"2024-02-10T08:37:19.975195+0000","last_peered":"2024-02-10T08:37:19.975195+0000","last_clean":"2024-02-10T08:37:19.975195+0000","last_became_active":"2024-02-10T08:37:19.974111+0000","last_became_peered":"2024-02-10T08:37:19.974111+0000","last_unstale":"2024-02-10T08:37:19.975195+0000","last_undegraded":"2024-02-10T08:37:19.975195+0000","last_fullsized":"2024-02-10T08:37:19.975195+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":38,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.922748+0000","last_change":"2024-02-10T08:37:16.963619+0000","last_active":"2024-02-10T08:37:20.922748+0000","last_peered":"2024-02-10T08:37:20.922748+0000","last_clean":"2024-02-10T08:37:20.922748+0000","last_became_active":"2024-02-10T08:37:10.127701+0000","last_became_peered":"2024-02-10T08:37:10.127701+0000","last_unstale":"2024-02-10T08:37:20.922748+0000","last_undegraded":"2024-02-10T08:37:20.922748+0000","last_fullsized":"2024-02-10T08:37:20.922748+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.16","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965676+0000","last_change":"2024-02-10T08:37:19.965323+0000","last_active":"2024-02-10T08:37:19.965676+0000","last_peered":"2024-02-10T08:37:19.965676+0000","last_clean":"2024-02-10T08:37:19.965676+0000","last_became_active":"2024-02-10T08:37:19.965052+0000","last_became_peered":"2024-02-10T08:37:19.965052+0000","last_unstale":"2024-02-10T08:37:19.965676+0000","last_undegraded":"2024-02-10T08:37:19.965676+0000","last_fullsized":"2024-02-10T08:37:19.965676+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"66.b","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.930063+0000","last_change":"2024-02-10T08:37:16.952282+0000","last_active":"2024-02-10T08:37:20.930063+0000","last_peered":"2024-02-10T08:37:20.930063+0000","last_clean":"2024-02-10T08:37:20.930063+0000","last_became_active":"2024-02-10T08:36:54.069260+0000","last_became_peered":"2024-02-10T08:36:54.069260+0000","last_unstale":"2024-02-10T08:37:20.930063+0000","last_undegraded":"2024-02-10T08:37:20.930063+0000","last_fullsized":"2024-02-10T08:37:20.930063+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.17","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.974945+0000","last_change":"2024-02-10T08:37:19.972928+0000","last_active":"2024-02-10T08:37:19.974945+0000","last_peered":"2024-02-10T08:37:19.974945+0000","last_clean":"2024-02-10T08:37:19.974945+0000","last_became_active":"2024-02-10T08:37:19.972613+0000","last_became_peered":"2024-02-10T08:37:19.972613+0000","last_unstale":"2024-02-10T08:37:19.974945+0000","last_undegraded":"2024-02-10T08:37:19.974945+0000","last_fullsized":"2024-02-10T08:37:19.974945+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.a","version":"0'0","reported_seq":38,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.928603+0000","last_change":"2024-02-10T08:37:16.951747+0000","last_active":"2024-02-10T08:37:20.928603+0000","last_peered":"2024-02-10T08:37:20.928603+0000","last_clean":"2024-02-10T08:37:20.928603+0000","last_became_active":"2024-02-10T08:37:10.203941+0000","last_became_peered":"2024-02-10T08:37:10.203941+0000","last_unstale":"2024-02-10T08:37:20.928603+0000","last_undegraded":"2024-02-10T08:37:20.928603+0000","last_fullsized":"2024-02-10T08:37:20.928603+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.18","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.973808+0000","last_change":"2024-02-10T08:37:19.970569+0000","last_active":"2024-02-10T08:37:19.973808+0000","last_peered":"2024-02-10T08:37:19.973808+0000","last_clean":"2024-02-10T08:37:19.973808+0000","last_became_active":"2024-02-10T08:37:19.970250+0000","last_became_peered":"2024-02-10T08:37:19.970250+0000","last_unstale":"2024-02-10T08:37:19.973808+0000","last_undegraded":"2024-02-10T08:37:19.973808+0000","last_fullsized":"2024-02-10T08:37:19.973808+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.5","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.917204+0000","last_change":"2024-02-10T08:37:16.993007+0000","last_active":"2024-02-10T08:37:20.917204+0000","last_peered":"2024-02-10T08:37:20.917204+0000","last_clean":"2024-02-10T08:37:20.917204+0000","last_became_active":"2024-02-10T08:37:10.247104+0000","last_became_peered":"2024-02-10T08:37:10.247104+0000","last_unstale":"2024-02-10T08:37:20.917204+0000","last_undegraded":"2024-02-10T08:37:20.917204+0000","last_fullsized":"2024-02-10T08:37:20.917204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.19","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.981445+0000","last_change":"2024-02-10T08:37:19.979642+0000","last_active":"2024-02-10T08:37:19.981445+0000","last_peered":"2024-02-10T08:37:19.981445+0000","last_clean":"2024-02-10T08:37:19.981445+0000","last_became_active":"2024-02-10T08:37:19.979331+0000","last_became_peered":"2024-02-10T08:37:19.979331+0000","last_unstale":"2024-02-10T08:37:19.981445+0000","last_undegraded":"2024-02-10T08:37:19.981445+0000","last_fullsized":"2024-02-10T08:37:19.981445+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925971+0000","last_change":"2024-02-10T08:37:16.954058+0000","last_active":"2024-02-10T08:37:20.925971+0000","last_peered":"2024-02-10T08:37:20.925971+0000","last_clean":"2024-02-10T08:37:20.925971+0000","last_became_active":"2024-02-10T08:36:54.065706+0000","last_became_peered":"2024-02-10T08:36:54.065706+0000","last_unstale":"2024-02-10T08:37:20.925971+0000","last_undegraded":"2024-02-10T08:37:20.925971+0000","last_fullsized":"2024-02-10T08:37:20.925971+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.983270+0000","last_change":"2024-02-10T08:37:19.974654+0000","last_active":"2024-02-10T08:37:19.983270+0000","last_peered":"2024-02-10T08:37:19.983270+0000","last_clean":"2024-02-10T08:37:19.983270+0000","last_became_active":"2024-02-10T08:37:19.974362+0000","last_became_peered":"2024-02-10T08:37:19.974362+0000","last_unstale":"2024-02-10T08:37:19.983270+0000","last_undegraded":"2024-02-10T08:37:19.983270+0000","last_fullsized":"2024-02-10T08:37:19.983270+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.932242+0000","last_change":"2024-02-10T08:37:16.945701+0000","last_active":"2024-02-10T08:37:20.932242+0000","last_peered":"2024-02-10T08:37:20.932242+0000","last_clean":"2024-02-10T08:37:20.932242+0000","last_became_active":"2024-02-10T08:36:54.089365+0000","last_became_peered":"2024-02-10T08:36:54.089365+0000","last_unstale":"2024-02-10T08:37:20.932242+0000","last_undegraded":"2024-02-10T08:37:20.932242+0000","last_fullsized":"2024-02-10T08:37:20.932242+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.982932+0000","last_change":"2024-02-10T08:37:19.980003+0000","last_active":"2024-02-10T08:37:19.982932+0000","last_peered":"2024-02-10T08:37:19.982932+0000","last_clean":"2024-02-10T08:37:19.982932+0000","last_became_active":"2024-02-10T08:37:19.979527+0000","last_became_peered":"2024-02-10T08:37:19.979527+0000","last_unstale":"2024-02-10T08:37:19.982932+0000","last_undegraded":"2024-02-10T08:37:19.982932+0000","last_fullsized":"2024-02-10T08:37:19.982932+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.6","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.925603+0000","last_change":"2024-02-10T08:37:16.964228+0000","last_active":"2024-02-10T08:37:20.925603+0000","last_peered":"2024-02-10T08:37:20.925603+0000","last_clean":"2024-02-10T08:37:20.925603+0000","last_became_active":"2024-02-10T08:36:54.043606+0000","last_became_peered":"2024-02-10T08:36:54.043606+0000","last_unstale":"2024-02-10T08:37:20.925603+0000","last_undegraded":"2024-02-10T08:37:20.925603+0000","last_fullsized":"2024-02-10T08:37:20.925603+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.974048+0000","last_change":"2024-02-10T08:37:19.970573+0000","last_active":"2024-02-10T08:37:19.974048+0000","last_peered":"2024-02-10T08:37:19.974048+0000","last_clean":"2024-02-10T08:37:19.974048+0000","last_became_active":"2024-02-10T08:37:19.970263+0000","last_became_peered":"2024-02-10T08:37:19.970263+0000","last_unstale":"2024-02-10T08:37:19.974048+0000","last_undegraded":"2024-02-10T08:37:19.974048+0000","last_fullsized":"2024-02-10T08:37:19.974048+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.935355+0000","last_change":"2024-02-10T08:37:16.945959+0000","last_active":"2024-02-10T08:37:20.935355+0000","last_peered":"2024-02-10T08:37:20.935355+0000","last_clean":"2024-02-10T08:37:20.935355+0000","last_became_active":"2024-02-10T08:36:54.069174+0000","last_became_peered":"2024-02-10T08:36:54.069174+0000","last_unstale":"2024-02-10T08:37:20.935355+0000","last_undegraded":"2024-02-10T08:37:20.935355+0000","last_fullsized":"2024-02-10T08:37:20.935355+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969612+0000","last_change":"2024-02-10T08:37:19.968330+0000","last_active":"2024-02-10T08:37:19.969612+0000","last_peered":"2024-02-10T08:37:19.969612+0000","last_clean":"2024-02-10T08:37:19.969612+0000","last_became_active":"2024-02-10T08:37:19.968018+0000","last_became_peered":"2024-02-10T08:37:19.968018+0000","last_unstale":"2024-02-10T08:37:19.969612+0000","last_undegraded":"2024-02-10T08:37:19.969612+0000","last_fullsized":"2024-02-10T08:37:19.969612+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.937726+0000","last_change":"2024-02-10T08:37:16.954306+0000","last_active":"2024-02-10T08:37:20.937726+0000","last_peered":"2024-02-10T08:37:20.937726+0000","last_clean":"2024-02-10T08:37:20.937726+0000","last_became_active":"2024-02-10T08:37:10.108757+0000","last_became_peered":"2024-02-10T08:37:10.108757+0000","last_unstale":"2024-02-10T08:37:20.937726+0000","last_undegraded":"2024-02-10T08:37:20.937726+0000","last_fullsized":"2024-02-10T08:37:20.937726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.975270+0000","last_change":"2024-02-10T08:37:19.973173+0000","last_active":"2024-02-10T08:37:19.975270+0000","last_peered":"2024-02-10T08:37:19.975270+0000","last_clean":"2024-02-10T08:37:19.975270+0000","last_became_active":"2024-02-10T08:37:19.972861+0000","last_became_peered":"2024-02-10T08:37:19.972861+0000","last_unstale":"2024-02-10T08:37:19.975270+0000","last_undegraded":"2024-02-10T08:37:19.975270+0000","last_fullsized":"2024-02-10T08:37:19.975270+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":89,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.935576+0000","last_change":"2024-02-10T08:37:16.953564+0000","last_active":"2024-02-10T08:37:20.935576+0000","last_peered":"2024-02-10T08:37:20.935576+0000","last_clean":"2024-02-10T08:37:20.935576+0000","last_became_active":"2024-02-10T08:37:10.149515+0000","last_became_peered":"2024-02-10T08:37:10.149515+0000","last_unstale":"2024-02-10T08:37:20.935576+0000","last_undegraded":"2024-02-10T08:37:20.935576+0000","last_fullsized":"2024-02-10T08:37:20.935576+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"95.1f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.978079+0000","last_change":"2024-02-10T08:37:19.973710+0000","last_active":"2024-02-10T08:37:19.978079+0000","last_peered":"2024-02-10T08:37:19.978079+0000","last_clean":"2024-02-10T08:37:19.978079+0000","last_became_active":"2024-02-10T08:37:19.972842+0000","last_became_peered":"2024-02-10T08:37:19.972842+0000","last_unstale":"2024-02-10T08:37:19.978079+0000","last_undegraded":"2024-02-10T08:37:19.978079+0000","last_fullsized":"2024-02-10T08:37:19.978079+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":80,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.916996+0000","last_change":"2024-02-10T08:37:16.987981+0000","last_active":"2024-02-10T08:37:20.916996+0000","last_peered":"2024-02-10T08:37:20.916996+0000","last_clean":"2024-02-10T08:37:20.916996+0000","last_became_active":"2024-02-10T08:36:54.067906+0000","last_became_peered":"2024-02-10T08:36:54.067906+0000","last_unstale":"2024-02-10T08:37:20.916996+0000","last_undegraded":"2024-02-10T08:37:20.916996+0000","last_fullsized":"2024-02-10T08:37:20.916996+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:52.992803+0000","last_clean_scrub_stamp":"2024-02-10T08:36:52.992803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.3","version":"146'2","reported_seq":45,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.053903+0000","last_change":"2024-02-10T08:37:14.030514+0000","last_active":"2024-02-10T08:37:20.053903+0000","last_peered":"2024-02-10T08:37:20.053903+0000","last_clean":"2024-02-10T08:37:20.053903+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:37:20.053903+0000","last_undegraded":"2024-02-10T08:37:20.053903+0000","last_fullsized":"2024-02-10T08:37:20.053903+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.2","version":"146'1","reported_seq":44,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.063457+0000","last_change":"2024-02-10T08:37:14.007544+0000","last_active":"2024-02-10T08:37:20.063457+0000","last_peered":"2024-02-10T08:37:20.063457+0000","last_clean":"2024-02-10T08:37:20.063457+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:37:20.063457+0000","last_undegraded":"2024-02-10T08:37:20.063457+0000","last_fullsized":"2024-02-10T08:37:20.063457+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1","version":"147'4","reported_seq":46,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.940294+0000","last_change":"2024-02-10T08:37:14.005032+0000","last_active":"2024-02-10T08:37:20.940294+0000","last_peered":"2024-02-10T08:37:20.940294+0000","last_clean":"2024-02-10T08:37:20.940294+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:37:20.940294+0000","last_undegraded":"2024-02-10T08:37:20.940294+0000","last_fullsized":"2024-02-10T08:37:20.940294+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.0","version":"147'16","reported_seq":63,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.998480+0000","last_change":"2024-02-10T08:37:14.044939+0000","last_active":"2024-02-10T08:37:20.998480+0000","last_peered":"2024-02-10T08:37:20.998480+0000","last_clean":"2024-02-10T08:37:20.998480+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:37:20.998480+0000","last_undegraded":"2024-02-10T08:37:20.998480+0000","last_fullsized":"2024-02-10T08:37:20.998480+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.7","version":"147'4","reported_seq":30,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.931811+0000","last_change":"2024-02-10T08:37:14.030045+0000","last_active":"2024-02-10T08:37:20.931811+0000","last_peered":"2024-02-10T08:37:20.931811+0000","last_clean":"2024-02-10T08:37:20.931811+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:37:20.931811+0000","last_undegraded":"2024-02-10T08:37:20.931811+0000","last_fullsized":"2024-02-10T08:37:20.931811+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.6","version":"147'10","reported_seq":50,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.994000+0000","last_change":"2024-02-10T08:37:14.056078+0000","last_active":"2024-02-10T08:37:20.994000+0000","last_peered":"2024-02-10T08:37:20.994000+0000","last_clean":"2024-02-10T08:37:20.994000+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:37:20.994000+0000","last_undegraded":"2024-02-10T08:37:20.994000+0000","last_fullsized":"2024-02-10T08:37:20.994000+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.5","version":"147'4","reported_seq":55,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.930498+0000","last_change":"2024-02-10T08:37:14.029759+0000","last_active":"2024-02-10T08:37:20.930498+0000","last_peered":"2024-02-10T08:37:20.930498+0000","last_clean":"2024-02-10T08:37:20.930498+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:37:20.930498+0000","last_undegraded":"2024-02-10T08:37:20.930498+0000","last_fullsized":"2024-02-10T08:37:20.930498+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.4","version":"147'3","reported_seq":45,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.930349+0000","last_change":"2024-02-10T08:37:14.030358+0000","last_active":"2024-02-10T08:37:20.930349+0000","last_peered":"2024-02-10T08:37:20.930349+0000","last_clean":"2024-02-10T08:37:20.930349+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:37:20.930349+0000","last_undegraded":"2024-02-10T08:37:20.930349+0000","last_fullsized":"2024-02-10T08:37:20.930349+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.b","version":"147'22","reported_seq":66,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.969263+0000","last_change":"2024-02-10T08:37:16.060525+0000","last_active":"2024-02-10T08:37:20.969263+0000","last_peered":"2024-02-10T08:37:20.969263+0000","last_clean":"2024-02-10T08:37:20.969263+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:37:20.969263+0000","last_undegraded":"2024-02-10T08:37:20.969263+0000","last_fullsized":"2024-02-10T08:37:20.969263+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":91,"num_objects":11,"num_object_clones":3,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.a","version":"146'6","reported_seq":50,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.042684+0000","last_change":"2024-02-10T08:37:16.049354+0000","last_active":"2024-02-10T08:37:20.042684+0000","last_peered":"2024-02-10T08:37:20.042684+0000","last_clean":"2024-02-10T08:37:20.042684+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:37:20.042684+0000","last_undegraded":"2024-02-10T08:37:20.042684+0000","last_fullsized":"2024-02-10T08:37:20.042684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.9","version":"147'14","reported_seq":37,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.986784+0000","last_change":"2024-02-10T08:37:14.030841+0000","last_active":"2024-02-10T08:37:20.986784+0000","last_peered":"2024-02-10T08:37:20.986784+0000","last_clean":"2024-02-10T08:37:20.986784+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:37:20.986784+0000","last_undegraded":"2024-02-10T08:37:20.986784+0000","last_fullsized":"2024-02-10T08:37:20.986784+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.8","version":"147'11","reported_seq":51,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:21.000541+0000","last_change":"2024-02-10T08:37:14.028055+0000","last_active":"2024-02-10T08:37:21.000541+0000","last_peered":"2024-02-10T08:37:21.000541+0000","last_clean":"2024-02-10T08:37:21.000541+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:37:21.000541+0000","last_undegraded":"2024-02-10T08:37:21.000541+0000","last_fullsized":"2024-02-10T08:37:21.000541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":73,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.f","version":"147'8","reported_seq":51,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.937060+0000","last_change":"2024-02-10T08:37:14.036091+0000","last_active":"2024-02-10T08:37:20.937060+0000","last_peered":"2024-02-10T08:37:20.937060+0000","last_clean":"2024-02-10T08:37:20.937060+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:37:20.937060+0000","last_undegraded":"2024-02-10T08:37:20.937060+0000","last_fullsized":"2024-02-10T08:37:20.937060+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.e","version":"147'7","reported_seq":32,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.975317+0000","last_change":"2024-02-10T08:37:14.035113+0000","last_active":"2024-02-10T08:37:20.975317+0000","last_peered":"2024-02-10T08:37:20.975317+0000","last_clean":"2024-02-10T08:37:20.975317+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:37:20.975317+0000","last_undegraded":"2024-02-10T08:37:20.975317+0000","last_fullsized":"2024-02-10T08:37:20.975317+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.d","version":"147'5","reported_seq":47,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.927674+0000","last_change":"2024-02-10T08:37:14.035480+0000","last_active":"2024-02-10T08:37:20.927674+0000","last_peered":"2024-02-10T08:37:20.927674+0000","last_clean":"2024-02-10T08:37:20.927674+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:37:20.927674+0000","last_undegraded":"2024-02-10T08:37:20.927674+0000","last_fullsized":"2024-02-10T08:37:20.927674+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.c","version":"147'10","reported_seq":59,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:21.002757+0000","last_change":"2024-02-10T08:37:14.011968+0000","last_active":"2024-02-10T08:37:21.002757+0000","last_peered":"2024-02-10T08:37:21.002757+0000","last_clean":"2024-02-10T08:37:21.002757+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:37:21.002757+0000","last_undegraded":"2024-02-10T08:37:21.002757+0000","last_fullsized":"2024-02-10T08:37:21.002757+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.13","version":"147'7","reported_seq":48,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.984624+0000","last_change":"2024-02-10T08:37:14.013065+0000","last_active":"2024-02-10T08:37:20.984624+0000","last_peered":"2024-02-10T08:37:20.984624+0000","last_clean":"2024-02-10T08:37:20.984624+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:37:20.984624+0000","last_undegraded":"2024-02-10T08:37:20.984624+0000","last_fullsized":"2024-02-10T08:37:20.984624+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.12","version":"147'10","reported_seq":60,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.972932+0000","last_change":"2024-02-10T08:37:14.015667+0000","last_active":"2024-02-10T08:37:20.972932+0000","last_peered":"2024-02-10T08:37:20.972932+0000","last_clean":"2024-02-10T08:37:20.972932+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:37:20.972932+0000","last_undegraded":"2024-02-10T08:37:20.972932+0000","last_fullsized":"2024-02-10T08:37:20.972932+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.11","version":"147'12","reported_seq":52,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.966809+0000","last_change":"2024-02-10T08:37:14.025959+0000","last_active":"2024-02-10T08:37:20.966809+0000","last_peered":"2024-02-10T08:37:20.966809+0000","last_clean":"2024-02-10T08:37:20.966809+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:37:20.966809+0000","last_undegraded":"2024-02-10T08:37:20.966809+0000","last_fullsized":"2024-02-10T08:37:20.966809+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.10","version":"147'6","reported_seq":57,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.937993+0000","last_change":"2024-02-10T08:37:14.016722+0000","last_active":"2024-02-10T08:37:20.937993+0000","last_peered":"2024-02-10T08:37:20.937993+0000","last_clean":"2024-02-10T08:37:20.937993+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:37:20.937993+0000","last_undegraded":"2024-02-10T08:37:20.937993+0000","last_fullsized":"2024-02-10T08:37:20.937993+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.17","version":"147'7","reported_seq":58,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.932401+0000","last_change":"2024-02-10T08:37:14.017461+0000","last_active":"2024-02-10T08:37:20.932401+0000","last_peered":"2024-02-10T08:37:20.932401+0000","last_clean":"2024-02-10T08:37:20.932401+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:37:20.932401+0000","last_undegraded":"2024-02-10T08:37:20.932401+0000","last_fullsized":"2024-02-10T08:37:20.932401+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.16","version":"147'14","reported_seq":68,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:21.009171+0000","last_change":"2024-02-10T08:37:16.048167+0000","last_active":"2024-02-10T08:37:21.009171+0000","last_peered":"2024-02-10T08:37:21.009171+0000","last_clean":"2024-02-10T08:37:21.009171+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:37:21.009171+0000","last_undegraded":"2024-02-10T08:37:21.009171+0000","last_fullsized":"2024-02-10T08:37:21.009171+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.15","version":"146'1","reported_seq":44,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.043594+0000","last_change":"2024-02-10T08:37:14.029348+0000","last_active":"2024-02-10T08:37:20.043594+0000","last_peered":"2024-02-10T08:37:20.043594+0000","last_clean":"2024-02-10T08:37:20.043594+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:37:20.043594+0000","last_undegraded":"2024-02-10T08:37:20.043594+0000","last_fullsized":"2024-02-10T08:37:20.043594+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.14","version":"146'2","reported_seq":45,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.083039+0000","last_change":"2024-02-10T08:37:14.048854+0000","last_active":"2024-02-10T08:37:20.083039+0000","last_peered":"2024-02-10T08:37:20.083039+0000","last_clean":"2024-02-10T08:37:20.083039+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:37:20.083039+0000","last_undegraded":"2024-02-10T08:37:20.083039+0000","last_fullsized":"2024-02-10T08:37:20.083039+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1b","version":"147'10","reported_seq":34,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.970391+0000","last_change":"2024-02-10T08:37:14.028568+0000","last_active":"2024-02-10T08:37:20.970391+0000","last_peered":"2024-02-10T08:37:20.970391+0000","last_clean":"2024-02-10T08:37:20.970391+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:37:20.970391+0000","last_undegraded":"2024-02-10T08:37:20.970391+0000","last_fullsized":"2024-02-10T08:37:20.970391+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1a","version":"147'8","reported_seq":49,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.990937+0000","last_change":"2024-02-10T08:37:14.045308+0000","last_active":"2024-02-10T08:37:20.990937+0000","last_peered":"2024-02-10T08:37:20.990937+0000","last_clean":"2024-02-10T08:37:20.990937+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:37:20.990937+0000","last_undegraded":"2024-02-10T08:37:20.990937+0000","last_fullsized":"2024-02-10T08:37:20.990937+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.19","version":"147'4","reported_seq":55,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.920715+0000","last_change":"2024-02-10T08:37:14.031039+0000","last_active":"2024-02-10T08:37:20.920715+0000","last_peered":"2024-02-10T08:37:20.920715+0000","last_clean":"2024-02-10T08:37:20.920715+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:37:20.920715+0000","last_undegraded":"2024-02-10T08:37:20.920715+0000","last_fullsized":"2024-02-10T08:37:20.920715+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.18","version":"147'6","reported_seq":31,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.953824+0000","last_change":"2024-02-10T08:37:14.027697+0000","last_active":"2024-02-10T08:37:20.953824+0000","last_peered":"2024-02-10T08:37:20.953824+0000","last_clean":"2024-02-10T08:37:20.953824+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:37:20.953824+0000","last_undegraded":"2024-02-10T08:37:20.953824+0000","last_fullsized":"2024-02-10T08:37:20.953824+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1f","version":"147'6","reported_seq":56,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.989835+0000","last_change":"2024-02-10T08:37:14.012763+0000","last_active":"2024-02-10T08:37:20.989835+0000","last_peered":"2024-02-10T08:37:20.989835+0000","last_clean":"2024-02-10T08:37:20.989835+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:37:20.989835+0000","last_undegraded":"2024-02-10T08:37:20.989835+0000","last_fullsized":"2024-02-10T08:37:20.989835+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1e","version":"146'1","reported_seq":28,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.054883+0000","last_change":"2024-02-10T08:37:14.044672+0000","last_active":"2024-02-10T08:37:20.054883+0000","last_peered":"2024-02-10T08:37:20.054883+0000","last_clean":"2024-02-10T08:37:20.054883+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:37:20.054883+0000","last_undegraded":"2024-02-10T08:37:20.054883+0000","last_fullsized":"2024-02-10T08:37:20.054883+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1d","version":"147'3","reported_seq":29,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.930204+0000","last_change":"2024-02-10T08:37:14.038234+0000","last_active":"2024-02-10T08:37:20.930204+0000","last_peered":"2024-02-10T08:37:20.930204+0000","last_clean":"2024-02-10T08:37:20.930204+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:37:20.930204+0000","last_undegraded":"2024-02-10T08:37:20.930204+0000","last_fullsized":"2024-02-10T08:37:20.930204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"60.1c","version":"147'6","reported_seq":47,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.991939+0000","last_change":"2024-02-10T08:37:14.011754+0000","last_active":"2024-02-10T08:37:20.991939+0000","last_peered":"2024-02-10T08:37:20.991939+0000","last_clean":"2024-02-10T08:37:20.991939+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:37:20.991939+0000","last_undegraded":"2024-02-10T08:37:20.991939+0000","last_fullsized":"2024-02-10T08:37:20.991939+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"25.15","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958107+0000","last_change":"2024-02-10T08:37:10.250275+0000","last_active":"2024-02-10T08:37:19.958107+0000","last_peered":"2024-02-10T08:37:19.958107+0000","last_clean":"2024-02-10T08:37:19.958107+0000","last_became_active":"2024-02-10T08:37:10.249567+0000","last_became_peered":"2024-02-10T08:37:10.249567+0000","last_unstale":"2024-02-10T08:37:19.958107+0000","last_undegraded":"2024-02-10T08:37:19.958107+0000","last_fullsized":"2024-02-10T08:37:19.958107+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":64,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966915+0000","last_change":"2024-02-10T08:37:12.777559+0000","last_active":"2024-02-10T08:37:19.966915+0000","last_peered":"2024-02-10T08:37:19.966915+0000","last_clean":"2024-02-10T08:37:19.966915+0000","last_became_active":"2024-02-10T08:36:37.030059+0000","last_became_peered":"2024-02-10T08:36:37.030059+0000","last_unstale":"2024-02-10T08:37:19.966915+0000","last_undegraded":"2024-02-10T08:37:19.966915+0000","last_fullsized":"2024-02-10T08:37:19.966915+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:12.777499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:12.777499+0000","last_clean_scrub_stamp":"2024-02-10T08:37:12.777499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951568+0000","last_change":"2024-02-10T08:36:58.130213+0000","last_active":"2024-02-10T08:37:19.951568+0000","last_peered":"2024-02-10T08:37:19.951568+0000","last_clean":"2024-02-10T08:37:19.951568+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:37:19.951568+0000","last_undegraded":"2024-02-10T08:37:19.951568+0000","last_fullsized":"2024-02-10T08:37:19.951568+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:16.169618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.169618+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952556+0000","last_change":"2024-02-10T08:37:07.840809+0000","last_active":"2024-02-10T08:37:19.952556+0000","last_peered":"2024-02-10T08:37:19.952556+0000","last_clean":"2024-02-10T08:37:19.952556+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:37:19.952556+0000","last_undegraded":"2024-02-10T08:37:19.952556+0000","last_fullsized":"2024-02-10T08:37:19.952556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:37:07.840738+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.840738+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959576+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:37:19.959576+0000","last_peered":"2024-02-10T08:37:19.959576+0000","last_clean":"2024-02-10T08:37:19.959576+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:37:19.959576+0000","last_undegraded":"2024-02-10T08:37:19.959576+0000","last_fullsized":"2024-02-10T08:37:19.959576+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":113,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954173+0000","last_change":"2024-02-10T08:37:10.168797+0000","last_active":"2024-02-10T08:37:19.954173+0000","last_peered":"2024-02-10T08:37:19.954173+0000","last_clean":"2024-02-10T08:37:19.954173+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:37:19.954173+0000","last_undegraded":"2024-02-10T08:37:19.954173+0000","last_fullsized":"2024-02-10T08:37:19.954173+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.704774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.704774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952769+0000","last_change":"2024-02-10T08:36:37.051266+0000","last_active":"2024-02-10T08:37:19.952769+0000","last_peered":"2024-02-10T08:37:19.952769+0000","last_clean":"2024-02-10T08:37:19.952769+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:37:19.952769+0000","last_undegraded":"2024-02-10T08:37:19.952769+0000","last_fullsized":"2024-02-10T08:37:19.952769+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"103'5","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961909+0000","last_change":"2024-02-10T08:37:11.012191+0000","last_active":"2024-02-10T08:37:19.961909+0000","last_peered":"2024-02-10T08:37:19.961909+0000","last_clean":"2024-02-10T08:37:19.961909+0000","last_became_active":"2024-02-10T08:37:10.219488+0000","last_became_peered":"2024-02-10T08:37:10.219488+0000","last_unstale":"2024-02-10T08:37:19.961909+0000","last_undegraded":"2024-02-10T08:37:19.961909+0000","last_fullsized":"2024-02-10T08:37:19.961909+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":64,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955271+0000","last_change":"2024-02-10T08:37:19.602954+0000","last_active":"2024-02-10T08:37:19.955271+0000","last_peered":"2024-02-10T08:37:19.955271+0000","last_clean":"2024-02-10T08:37:19.955271+0000","last_became_active":"2024-02-10T08:36:37.096176+0000","last_became_peered":"2024-02-10T08:36:37.096176+0000","last_unstale":"2024-02-10T08:37:19.955271+0000","last_undegraded":"2024-02-10T08:37:19.955271+0000","last_fullsized":"2024-02-10T08:37:19.955271+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.602894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.602894+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.602894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":82,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952050+0000","last_change":"2024-02-10T08:37:10.933335+0000","last_active":"2024-02-10T08:37:19.952050+0000","last_peered":"2024-02-10T08:37:19.952050+0000","last_clean":"2024-02-10T08:37:19.952050+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:37:19.952050+0000","last_undegraded":"2024-02-10T08:37:19.952050+0000","last_fullsized":"2024-02-10T08:37:19.952050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:14.826321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.826321+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959642+0000","last_change":"2024-02-10T08:35:30.241737+0000","last_active":"2024-02-10T08:37:19.959642+0000","last_peered":"2024-02-10T08:37:19.959642+0000","last_clean":"2024-02-10T08:37:19.959642+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:37:19.959642+0000","last_undegraded":"2024-02-10T08:37:19.959642+0000","last_fullsized":"2024-02-10T08:37:19.959642+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.241661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.241661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":124,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952818+0000","last_change":"2024-02-10T08:36:55.097291+0000","last_active":"2024-02-10T08:37:19.952818+0000","last_peered":"2024-02-10T08:37:19.952818+0000","last_clean":"2024-02-10T08:37:19.952818+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:37:19.952818+0000","last_undegraded":"2024-02-10T08:37:19.952818+0000","last_fullsized":"2024-02-10T08:37:19.952818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.568165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.568165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951291+0000","last_change":"2024-02-10T08:37:10.214995+0000","last_active":"2024-02-10T08:37:19.951291+0000","last_peered":"2024-02-10T08:37:19.951291+0000","last_clean":"2024-02-10T08:37:19.951291+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:37:19.951291+0000","last_undegraded":"2024-02-10T08:37:19.951291+0000","last_fullsized":"2024-02-10T08:37:19.951291+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:34.652551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.652551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965871+0000","last_change":"2024-02-10T08:36:37.033242+0000","last_active":"2024-02-10T08:37:19.965871+0000","last_peered":"2024-02-10T08:37:19.965871+0000","last_clean":"2024-02-10T08:37:19.965871+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:37:19.965871+0000","last_undegraded":"2024-02-10T08:37:19.965871+0000","last_fullsized":"2024-02-10T08:37:19.965871+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.17","version":"103'5","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.967554+0000","last_change":"2024-02-10T08:37:10.688144+0000","last_active":"2024-02-10T08:37:19.967554+0000","last_peered":"2024-02-10T08:37:19.967554+0000","last_clean":"2024-02-10T08:37:19.967554+0000","last_became_active":"2024-02-10T08:37:10.259720+0000","last_became_peered":"2024-02-10T08:37:10.259720+0000","last_unstale":"2024-02-10T08:37:19.967554+0000","last_undegraded":"2024-02-10T08:37:19.967554+0000","last_fullsized":"2024-02-10T08:37:19.967554+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"140'3","reported_seq":44,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962310+0000","last_change":"2024-02-10T08:37:14.267579+0000","last_active":"2024-02-10T08:37:19.962310+0000","last_peered":"2024-02-10T08:37:19.962310+0000","last_clean":"2024-02-10T08:37:19.962310+0000","last_became_active":"2024-02-10T08:37:10.202865+0000","last_became_peered":"2024-02-10T08:37:10.202865+0000","last_unstale":"2024-02-10T08:37:19.962310+0000","last_undegraded":"2024-02-10T08:37:19.962310+0000","last_fullsized":"2024-02-10T08:37:19.962310+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'3","last_scrub_stamp":"2024-02-10T08:37:14.267513+0000","last_deep_scrub":"140'3","last_deep_scrub_stamp":"2024-02-10T08:37:14.267513+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.267513+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":139,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962108+0000","last_change":"2024-02-10T08:36:23.647242+0000","last_active":"2024-02-10T08:37:19.962108+0000","last_peered":"2024-02-10T08:37:19.962108+0000","last_clean":"2024-02-10T08:37:19.962108+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:37:19.962108+0000","last_undegraded":"2024-02-10T08:37:19.962108+0000","last_fullsized":"2024-02-10T08:37:19.962108+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:23.647164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.647164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952957+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:37:19.952957+0000","last_peered":"2024-02-10T08:37:19.952957+0000","last_clean":"2024-02-10T08:37:19.952957+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:37:19.952957+0000","last_undegraded":"2024-02-10T08:37:19.952957+0000","last_fullsized":"2024-02-10T08:37:19.952957+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951976+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:37:19.951976+0000","last_peered":"2024-02-10T08:37:19.951976+0000","last_clean":"2024-02-10T08:37:19.951976+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:37:19.951976+0000","last_undegraded":"2024-02-10T08:37:19.951976+0000","last_fullsized":"2024-02-10T08:37:19.951976+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953448+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:37:19.953448+0000","last_peered":"2024-02-10T08:37:19.953448+0000","last_clean":"2024-02-10T08:37:19.953448+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:37:19.953448+0000","last_undegraded":"2024-02-10T08:37:19.953448+0000","last_fullsized":"2024-02-10T08:37:19.953448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952876+0000","last_change":"2024-02-10T08:36:37.021497+0000","last_active":"2024-02-10T08:37:19.952876+0000","last_peered":"2024-02-10T08:37:19.952876+0000","last_clean":"2024-02-10T08:37:19.952876+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:37:19.952876+0000","last_undegraded":"2024-02-10T08:37:19.952876+0000","last_fullsized":"2024-02-10T08:37:19.952876+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953386+0000","last_change":"2024-02-10T08:37:10.200689+0000","last_active":"2024-02-10T08:37:19.953386+0000","last_peered":"2024-02-10T08:37:19.953386+0000","last_clean":"2024-02-10T08:37:19.953386+0000","last_became_active":"2024-02-10T08:37:10.200347+0000","last_became_peered":"2024-02-10T08:37:10.200347+0000","last_unstale":"2024-02-10T08:37:19.953386+0000","last_undegraded":"2024-02-10T08:37:19.953386+0000","last_fullsized":"2024-02-10T08:37:19.953386+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955246+0000","last_change":"2024-02-10T08:35:30.802385+0000","last_active":"2024-02-10T08:37:19.955246+0000","last_peered":"2024-02-10T08:37:19.955246+0000","last_clean":"2024-02-10T08:37:19.955246+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:37:19.955246+0000","last_undegraded":"2024-02-10T08:37:19.955246+0000","last_fullsized":"2024-02-10T08:37:19.955246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:35:30.802306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.802306+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949651+0000","last_change":"2024-02-10T08:35:30.123918+0000","last_active":"2024-02-10T08:37:19.949651+0000","last_peered":"2024-02-10T08:37:19.949651+0000","last_clean":"2024-02-10T08:37:19.949651+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:37:19.949651+0000","last_undegraded":"2024-02-10T08:37:19.949651+0000","last_fullsized":"2024-02-10T08:37:19.949651+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:30.123828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:30.123828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953254+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:37:19.953254+0000","last_peered":"2024-02-10T08:37:19.953254+0000","last_clean":"2024-02-10T08:37:19.953254+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:37:19.953254+0000","last_undegraded":"2024-02-10T08:37:19.953254+0000","last_fullsized":"2024-02-10T08:37:19.953254+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.956180+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:37:19.956180+0000","last_peered":"2024-02-10T08:37:19.956180+0000","last_clean":"2024-02-10T08:37:19.956180+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:37:19.956180+0000","last_undegraded":"2024-02-10T08:37:19.956180+0000","last_fullsized":"2024-02-10T08:37:19.956180+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"138'2","reported_seq":79,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953461+0000","last_change":"2024-02-10T08:37:12.735610+0000","last_active":"2024-02-10T08:37:19.953461+0000","last_peered":"2024-02-10T08:37:19.953461+0000","last_clean":"2024-02-10T08:37:19.953461+0000","last_became_active":"2024-02-10T08:37:10.181504+0000","last_became_peered":"2024-02-10T08:37:10.181504+0000","last_unstale":"2024-02-10T08:37:19.953461+0000","last_undegraded":"2024-02-10T08:37:19.953461+0000","last_fullsized":"2024-02-10T08:37:19.953461+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'2","last_scrub_stamp":"2024-02-10T08:37:12.735487+0000","last_deep_scrub":"138'2","last_deep_scrub_stamp":"2024-02-10T08:37:12.735487+0000","last_clean_scrub_stamp":"2024-02-10T08:37:12.735487+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":7,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968419+0000","last_change":"2024-02-10T08:36:37.075652+0000","last_active":"2024-02-10T08:37:19.968419+0000","last_peered":"2024-02-10T08:37:19.968419+0000","last_clean":"2024-02-10T08:37:19.968419+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:37:19.968419+0000","last_undegraded":"2024-02-10T08:37:19.968419+0000","last_fullsized":"2024-02-10T08:37:19.968419+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"103'7","reported_seq":43,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968337+0000","last_change":"2024-02-10T08:37:10.141080+0000","last_active":"2024-02-10T08:37:19.968337+0000","last_peered":"2024-02-10T08:37:19.968337+0000","last_clean":"2024-02-10T08:37:19.968337+0000","last_became_active":"2024-02-10T08:37:10.115979+0000","last_became_peered":"2024-02-10T08:37:10.115979+0000","last_unstale":"2024-02-10T08:37:19.968337+0000","last_undegraded":"2024-02-10T08:37:19.968337+0000","last_fullsized":"2024-02-10T08:37:19.968337+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":123,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961675+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:37:19.961675+0000","last_peered":"2024-02-10T08:37:19.961675+0000","last_clean":"2024-02-10T08:37:19.961675+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:37:19.961675+0000","last_undegraded":"2024-02-10T08:37:19.961675+0000","last_fullsized":"2024-02-10T08:37:19.961675+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954426+0000","last_change":"2024-02-10T08:35:31.222395+0000","last_active":"2024-02-10T08:37:19.954426+0000","last_peered":"2024-02-10T08:37:19.954426+0000","last_clean":"2024-02-10T08:37:19.954426+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:37:19.954426+0000","last_undegraded":"2024-02-10T08:37:19.954426+0000","last_fullsized":"2024-02-10T08:37:19.954426+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:31.222331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:31.222331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955563+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:37:19.955563+0000","last_peered":"2024-02-10T08:37:19.955563+0000","last_clean":"2024-02-10T08:37:19.955563+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:37:19.955563+0000","last_undegraded":"2024-02-10T08:37:19.955563+0000","last_fullsized":"2024-02-10T08:37:19.955563+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962765+0000","last_change":"2024-02-10T08:36:13.778764+0000","last_active":"2024-02-10T08:37:19.962765+0000","last_peered":"2024-02-10T08:37:19.962765+0000","last_clean":"2024-02-10T08:37:19.962765+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:37:19.962765+0000","last_undegraded":"2024-02-10T08:37:19.962765+0000","last_fullsized":"2024-02-10T08:37:19.962765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.778673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.778673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":66,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951100+0000","last_change":"2024-02-10T08:37:12.400226+0000","last_active":"2024-02-10T08:37:19.951100+0000","last_peered":"2024-02-10T08:37:19.951100+0000","last_clean":"2024-02-10T08:37:19.951100+0000","last_became_active":"2024-02-10T08:36:37.150853+0000","last_became_peered":"2024-02-10T08:36:37.150853+0000","last_unstale":"2024-02-10T08:37:19.951100+0000","last_undegraded":"2024-02-10T08:37:19.951100+0000","last_fullsized":"2024-02-10T08:37:19.951100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:12.400106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:12.400106+0000","last_clean_scrub_stamp":"2024-02-10T08:37:12.400106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951018+0000","last_change":"2024-02-10T08:37:10.263630+0000","last_active":"2024-02-10T08:37:19.951018+0000","last_peered":"2024-02-10T08:37:19.951018+0000","last_clean":"2024-02-10T08:37:19.951018+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:37:19.951018+0000","last_undegraded":"2024-02-10T08:37:19.951018+0000","last_fullsized":"2024-02-10T08:37:19.951018+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954838+0000","last_change":"2024-02-10T08:37:10.183873+0000","last_active":"2024-02-10T08:37:19.954838+0000","last_peered":"2024-02-10T08:37:19.954838+0000","last_clean":"2024-02-10T08:37:19.954838+0000","last_became_active":"2024-02-10T08:37:10.183562+0000","last_became_peered":"2024-02-10T08:37:10.183562+0000","last_unstale":"2024-02-10T08:37:19.954838+0000","last_undegraded":"2024-02-10T08:37:19.954838+0000","last_fullsized":"2024-02-10T08:37:19.954838+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"143'4","reported_seq":72,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.947223+0000","last_change":"2024-02-10T08:37:17.674068+0000","last_active":"2024-02-10T08:37:19.947223+0000","last_peered":"2024-02-10T08:37:19.947223+0000","last_clean":"2024-02-10T08:37:19.947223+0000","last_became_active":"2024-02-10T08:36:37.044969+0000","last_became_peered":"2024-02-10T08:36:37.044969+0000","last_unstale":"2024-02-10T08:37:19.947223+0000","last_undegraded":"2024-02-10T08:37:19.947223+0000","last_fullsized":"2024-02-10T08:37:19.947223+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4","last_scrub_stamp":"2024-02-10T08:37:17.674001+0000","last_deep_scrub":"143'4","last_deep_scrub_stamp":"2024-02-10T08:37:17.674001+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.674001+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25168636,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":128,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955431+0000","last_change":"2024-02-10T08:35:11.190561+0000","last_active":"2024-02-10T08:37:19.955431+0000","last_peered":"2024-02-10T08:37:19.955431+0000","last_clean":"2024-02-10T08:37:19.955431+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:37:19.955431+0000","last_undegraded":"2024-02-10T08:37:19.955431+0000","last_fullsized":"2024-02-10T08:37:19.955431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:11.190437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:11.190437+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":116,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953575+0000","last_change":"2024-02-10T08:36:16.423854+0000","last_active":"2024-02-10T08:37:19.953575+0000","last_peered":"2024-02-10T08:37:19.953575+0000","last_clean":"2024-02-10T08:37:19.953575+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:37:19.953575+0000","last_undegraded":"2024-02-10T08:37:19.953575+0000","last_fullsized":"2024-02-10T08:37:19.953575+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:16.423786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.423786+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962799+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:37:19.962799+0000","last_peered":"2024-02-10T08:37:19.962799+0000","last_clean":"2024-02-10T08:37:19.962799+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:37:19.962799+0000","last_undegraded":"2024-02-10T08:37:19.962799+0000","last_fullsized":"2024-02-10T08:37:19.962799+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":108,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954914+0000","last_change":"2024-02-10T08:37:10.149327+0000","last_active":"2024-02-10T08:37:19.954914+0000","last_peered":"2024-02-10T08:37:19.954914+0000","last_clean":"2024-02-10T08:37:19.954914+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:37:19.954914+0000","last_undegraded":"2024-02-10T08:37:19.954914+0000","last_fullsized":"2024-02-10T08:37:19.954914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:16.509977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:16.509977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964236+0000","last_change":"2024-02-10T08:36:37.148745+0000","last_active":"2024-02-10T08:37:19.964236+0000","last_peered":"2024-02-10T08:37:19.964236+0000","last_clean":"2024-02-10T08:37:19.964236+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:37:19.964236+0000","last_undegraded":"2024-02-10T08:37:19.964236+0000","last_fullsized":"2024-02-10T08:37:19.964236+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.13","version":"103'9","reported_seq":79,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957533+0000","last_change":"2024-02-10T08:37:10.221042+0000","last_active":"2024-02-10T08:37:19.957533+0000","last_peered":"2024-02-10T08:37:19.957533+0000","last_clean":"2024-02-10T08:37:19.957533+0000","last_became_active":"2024-02-10T08:37:10.136194+0000","last_became_peered":"2024-02-10T08:37:10.136194+0000","last_unstale":"2024-02-10T08:37:19.957533+0000","last_undegraded":"2024-02-10T08:37:19.957533+0000","last_fullsized":"2024-02-10T08:37:19.957533+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":127,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950594+0000","last_change":"2024-02-10T08:36:11.732545+0000","last_active":"2024-02-10T08:37:19.950594+0000","last_peered":"2024-02-10T08:37:19.950594+0000","last_clean":"2024-02-10T08:37:19.950594+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:37:19.950594+0000","last_undegraded":"2024-02-10T08:37:19.950594+0000","last_fullsized":"2024-02-10T08:37:19.950594+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:36:11.732461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:11.732461+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953757+0000","last_change":"2024-02-10T08:37:10.150021+0000","last_active":"2024-02-10T08:37:19.953757+0000","last_peered":"2024-02-10T08:37:19.953757+0000","last_clean":"2024-02-10T08:37:19.953757+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:37:19.953757+0000","last_undegraded":"2024-02-10T08:37:19.953757+0000","last_fullsized":"2024-02-10T08:37:19.953757+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.142541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.142541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":130,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965322+0000","last_change":"2024-02-10T08:36:29.158683+0000","last_active":"2024-02-10T08:37:19.965322+0000","last_peered":"2024-02-10T08:37:19.965322+0000","last_clean":"2024-02-10T08:37:19.965322+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:37:19.965322+0000","last_undegraded":"2024-02-10T08:37:19.965322+0000","last_fullsized":"2024-02-10T08:37:19.965322+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:06.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:06.520521+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":98,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955388+0000","last_change":"2024-02-10T08:35:19.422149+0000","last_active":"2024-02-10T08:37:19.955388+0000","last_peered":"2024-02-10T08:37:19.955388+0000","last_clean":"2024-02-10T08:37:19.955388+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:37:19.955388+0000","last_undegraded":"2024-02-10T08:37:19.955388+0000","last_fullsized":"2024-02-10T08:37:19.955388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.422086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.422086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"138'8","reported_seq":87,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961802+0000","last_change":"2024-02-10T08:37:12.616256+0000","last_active":"2024-02-10T08:37:19.961802+0000","last_peered":"2024-02-10T08:37:19.961802+0000","last_clean":"2024-02-10T08:37:19.961802+0000","last_became_active":"2024-02-10T08:37:10.213411+0000","last_became_peered":"2024-02-10T08:37:10.213411+0000","last_unstale":"2024-02-10T08:37:19.961802+0000","last_undegraded":"2024-02-10T08:37:19.961802+0000","last_fullsized":"2024-02-10T08:37:19.961802+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'8","last_scrub_stamp":"2024-02-10T08:37:12.616196+0000","last_deep_scrub":"138'8","last_deep_scrub_stamp":"2024-02-10T08:37:12.616196+0000","last_clean_scrub_stamp":"2024-02-10T08:37:12.616196+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":7,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950291+0000","last_change":"2024-02-10T08:36:37.134622+0000","last_active":"2024-02-10T08:37:19.950291+0000","last_peered":"2024-02-10T08:37:19.950291+0000","last_clean":"2024-02-10T08:37:19.950291+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:37:19.950291+0000","last_undegraded":"2024-02-10T08:37:19.950291+0000","last_fullsized":"2024-02-10T08:37:19.950291+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961403+0000","last_change":"2024-02-10T08:36:37.065813+0000","last_active":"2024-02-10T08:37:19.961403+0000","last_peered":"2024-02-10T08:37:19.961403+0000","last_clean":"2024-02-10T08:37:19.961403+0000","last_became_active":"2024-02-10T08:36:37.065010+0000","last_became_peered":"2024-02-10T08:36:37.065010+0000","last_unstale":"2024-02-10T08:37:19.961403+0000","last_undegraded":"2024-02-10T08:37:19.961403+0000","last_fullsized":"2024-02-10T08:37:19.961403+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"105'2","reported_seq":57,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954362+0000","last_change":"2024-02-10T08:36:37.033787+0000","last_active":"2024-02-10T08:37:19.954362+0000","last_peered":"2024-02-10T08:37:19.954362+0000","last_clean":"2024-02-10T08:37:19.954362+0000","last_became_active":"2024-02-10T08:36:37.033569+0000","last_became_peered":"2024-02-10T08:36:37.033569+0000","last_unstale":"2024-02-10T08:37:19.954362+0000","last_undegraded":"2024-02-10T08:37:19.954362+0000","last_fullsized":"2024-02-10T08:37:19.954362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":164,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951436+0000","last_change":"2024-02-10T08:36:28.076465+0000","last_active":"2024-02-10T08:37:19.951436+0000","last_peered":"2024-02-10T08:37:19.951436+0000","last_clean":"2024-02-10T08:37:19.951436+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:37:19.951436+0000","last_undegraded":"2024-02-10T08:37:19.951436+0000","last_fullsized":"2024-02-10T08:37:19.951436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:28.076401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.076401+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":129,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968838+0000","last_change":"2024-02-10T08:36:17.729319+0000","last_active":"2024-02-10T08:37:19.968838+0000","last_peered":"2024-02-10T08:37:19.968838+0000","last_clean":"2024-02-10T08:37:19.968838+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:37:19.968838+0000","last_undegraded":"2024-02-10T08:37:19.968838+0000","last_fullsized":"2024-02-10T08:37:19.968838+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:17.729249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.729249+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969479+0000","last_change":"2024-02-10T08:37:19.310327+0000","last_active":"2024-02-10T08:37:19.969479+0000","last_peered":"2024-02-10T08:37:19.969479+0000","last_clean":"2024-02-10T08:37:19.969479+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:37:19.969479+0000","last_undegraded":"2024-02-10T08:37:19.969479+0000","last_fullsized":"2024-02-10T08:37:19.969479+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.310252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.310252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949984+0000","last_change":"2024-02-10T08:37:09.215324+0000","last_active":"2024-02-10T08:37:19.949984+0000","last_peered":"2024-02-10T08:37:19.949984+0000","last_clean":"2024-02-10T08:37:19.949984+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:37:19.949984+0000","last_undegraded":"2024-02-10T08:37:19.949984+0000","last_fullsized":"2024-02-10T08:37:19.949984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:09.215257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:09.215257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961648+0000","last_change":"2024-02-10T08:37:10.160633+0000","last_active":"2024-02-10T08:37:19.961648+0000","last_peered":"2024-02-10T08:37:19.961648+0000","last_clean":"2024-02-10T08:37:19.961648+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:37:19.961648+0000","last_undegraded":"2024-02-10T08:37:19.961648+0000","last_fullsized":"2024-02-10T08:37:19.961648+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"103'7","reported_seq":60,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954269+0000","last_change":"2024-02-10T08:36:37.149279+0000","last_active":"2024-02-10T08:37:19.954269+0000","last_peered":"2024-02-10T08:37:19.954269+0000","last_clean":"2024-02-10T08:37:19.954269+0000","last_became_active":"2024-02-10T08:36:37.148954+0000","last_became_peered":"2024-02-10T08:36:37.148954+0000","last_unstale":"2024-02-10T08:37:19.954269+0000","last_undegraded":"2024-02-10T08:37:19.954269+0000","last_fullsized":"2024-02-10T08:37:19.954269+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"136'1","reported_seq":28,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955046+0000","last_change":"2024-02-10T08:37:10.244649+0000","last_active":"2024-02-10T08:37:19.955046+0000","last_peered":"2024-02-10T08:37:19.955046+0000","last_clean":"2024-02-10T08:37:19.955046+0000","last_became_active":"2024-02-10T08:37:10.243439+0000","last_became_peered":"2024-02-10T08:37:10.243439+0000","last_unstale":"2024-02-10T08:37:19.955046+0000","last_undegraded":"2024-02-10T08:37:19.955046+0000","last_fullsized":"2024-02-10T08:37:19.955046+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969093+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:37:19.969093+0000","last_peered":"2024-02-10T08:37:19.969093+0000","last_clean":"2024-02-10T08:37:19.969093+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:37:19.969093+0000","last_undegraded":"2024-02-10T08:37:19.969093+0000","last_fullsized":"2024-02-10T08:37:19.969093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":169,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951351+0000","last_change":"2024-02-10T08:37:11.087703+0000","last_active":"2024-02-10T08:37:19.951351+0000","last_peered":"2024-02-10T08:37:19.951351+0000","last_clean":"2024-02-10T08:37:19.951351+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:37:19.951351+0000","last_undegraded":"2024-02-10T08:37:19.951351+0000","last_fullsized":"2024-02-10T08:37:19.951351+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:35:09.492017+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":117,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961979+0000","last_change":"2024-02-10T08:36:32.450905+0000","last_active":"2024-02-10T08:37:19.961979+0000","last_peered":"2024-02-10T08:37:19.961979+0000","last_clean":"2024-02-10T08:37:19.961979+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:37:19.961979+0000","last_undegraded":"2024-02-10T08:37:19.961979+0000","last_fullsized":"2024-02-10T08:37:19.961979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:32.450807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.450807+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":99,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962259+0000","last_change":"2024-02-10T08:35:19.863450+0000","last_active":"2024-02-10T08:37:19.962259+0000","last_peered":"2024-02-10T08:37:19.962259+0000","last_clean":"2024-02-10T08:37:19.962259+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:37:19.962259+0000","last_undegraded":"2024-02-10T08:37:19.962259+0000","last_fullsized":"2024-02-10T08:37:19.962259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:19.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:19.863356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955129+0000","last_change":"2024-02-10T08:37:10.260060+0000","last_active":"2024-02-10T08:37:19.955129+0000","last_peered":"2024-02-10T08:37:19.955129+0000","last_clean":"2024-02-10T08:37:19.955129+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:37:19.955129+0000","last_undegraded":"2024-02-10T08:37:19.955129+0000","last_fullsized":"2024-02-10T08:37:19.955129+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963318+0000","last_change":"2024-02-10T08:37:10.221508+0000","last_active":"2024-02-10T08:37:19.963318+0000","last_peered":"2024-02-10T08:37:19.963318+0000","last_clean":"2024-02-10T08:37:19.963318+0000","last_became_active":"2024-02-10T08:37:10.221122+0000","last_became_peered":"2024-02-10T08:37:10.221122+0000","last_unstale":"2024-02-10T08:37:19.963318+0000","last_undegraded":"2024-02-10T08:37:19.963318+0000","last_fullsized":"2024-02-10T08:37:19.963318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":54,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959780+0000","last_change":"2024-02-10T08:36:37.157729+0000","last_active":"2024-02-10T08:37:19.959780+0000","last_peered":"2024-02-10T08:37:19.959780+0000","last_clean":"2024-02-10T08:37:19.959780+0000","last_became_active":"2024-02-10T08:36:37.156958+0000","last_became_peered":"2024-02-10T08:36:37.156958+0000","last_unstale":"2024-02-10T08:37:19.959780+0000","last_undegraded":"2024-02-10T08:37:19.959780+0000","last_fullsized":"2024-02-10T08:37:19.959780+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":115,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962186+0000","last_change":"2024-02-10T08:37:10.231389+0000","last_active":"2024-02-10T08:37:19.962186+0000","last_peered":"2024-02-10T08:37:19.962186+0000","last_clean":"2024-02-10T08:37:19.962186+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:37:19.962186+0000","last_undegraded":"2024-02-10T08:37:19.962186+0000","last_fullsized":"2024-02-10T08:37:19.962186+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.578284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.578284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":89,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964662+0000","last_change":"2024-02-10T08:37:10.868785+0000","last_active":"2024-02-10T08:37:19.964662+0000","last_peered":"2024-02-10T08:37:19.964662+0000","last_clean":"2024-02-10T08:37:19.964662+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:37:19.964662+0000","last_undegraded":"2024-02-10T08:37:19.964662+0000","last_fullsized":"2024-02-10T08:37:19.964662+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:15.871060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.871060+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949937+0000","last_change":"2024-02-10T08:37:10.151669+0000","last_active":"2024-02-10T08:37:19.949937+0000","last_peered":"2024-02-10T08:37:19.949937+0000","last_clean":"2024-02-10T08:37:19.949937+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:37:19.949937+0000","last_undegraded":"2024-02-10T08:37:19.949937+0000","last_fullsized":"2024-02-10T08:37:19.949937+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.048119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.048119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953623+0000","last_change":"2024-02-10T08:36:20.500433+0000","last_active":"2024-02-10T08:37:19.953623+0000","last_peered":"2024-02-10T08:37:19.953623+0000","last_clean":"2024-02-10T08:37:19.953623+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:37:19.953623+0000","last_undegraded":"2024-02-10T08:37:19.953623+0000","last_fullsized":"2024-02-10T08:37:19.953623+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.500313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955924+0000","last_change":"2024-02-10T08:37:10.233957+0000","last_active":"2024-02-10T08:37:19.955924+0000","last_peered":"2024-02-10T08:37:19.955924+0000","last_clean":"2024-02-10T08:37:19.955924+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:37:19.955924+0000","last_undegraded":"2024-02-10T08:37:19.955924+0000","last_fullsized":"2024-02-10T08:37:19.955924+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"103'10","reported_seq":78,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953996+0000","last_change":"2024-02-10T08:37:10.445260+0000","last_active":"2024-02-10T08:37:19.953996+0000","last_peered":"2024-02-10T08:37:19.953996+0000","last_clean":"2024-02-10T08:37:19.953996+0000","last_became_active":"2024-02-10T08:37:10.261034+0000","last_became_peered":"2024-02-10T08:37:10.261034+0000","last_unstale":"2024-02-10T08:37:19.953996+0000","last_undegraded":"2024-02-10T08:37:19.953996+0000","last_fullsized":"2024-02-10T08:37:19.953996+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"105'3","reported_seq":61,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954200+0000","last_change":"2024-02-10T08:36:37.035006+0000","last_active":"2024-02-10T08:37:19.954200+0000","last_peered":"2024-02-10T08:37:19.954200+0000","last_clean":"2024-02-10T08:37:19.954200+0000","last_became_active":"2024-02-10T08:36:37.034740+0000","last_became_peered":"2024-02-10T08:36:37.034740+0000","last_unstale":"2024-02-10T08:37:19.954200+0000","last_undegraded":"2024-02-10T08:37:19.954200+0000","last_fullsized":"2024-02-10T08:37:19.954200+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165839,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965824+0000","last_change":"2024-02-10T08:37:10.149313+0000","last_active":"2024-02-10T08:37:19.965824+0000","last_peered":"2024-02-10T08:37:19.965824+0000","last_clean":"2024-02-10T08:37:19.965824+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:37:19.965824+0000","last_undegraded":"2024-02-10T08:37:19.965824+0000","last_fullsized":"2024-02-10T08:37:19.965824+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.714301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.714301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":67,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962099+0000","last_change":"2024-02-10T08:37:11.422394+0000","last_active":"2024-02-10T08:37:19.962099+0000","last_peered":"2024-02-10T08:37:19.962099+0000","last_clean":"2024-02-10T08:37:19.962099+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:37:19.962099+0000","last_undegraded":"2024-02-10T08:37:19.962099+0000","last_fullsized":"2024-02-10T08:37:19.962099+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:35:36.647719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.647719+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953526+0000","last_change":"2024-02-10T08:37:10.213655+0000","last_active":"2024-02-10T08:37:19.953526+0000","last_peered":"2024-02-10T08:37:19.953526+0000","last_clean":"2024-02-10T08:37:19.953526+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:37:19.953526+0000","last_undegraded":"2024-02-10T08:37:19.953526+0000","last_fullsized":"2024-02-10T08:37:19.953526+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:27.051837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:27.051837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968705+0000","last_change":"2024-02-10T08:37:10.215017+0000","last_active":"2024-02-10T08:37:19.968705+0000","last_peered":"2024-02-10T08:37:19.968705+0000","last_clean":"2024-02-10T08:37:19.968705+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:37:19.968705+0000","last_undegraded":"2024-02-10T08:37:19.968705+0000","last_fullsized":"2024-02-10T08:37:19.968705+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":105,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953343+0000","last_change":"2024-02-10T08:35:38.321759+0000","last_active":"2024-02-10T08:37:19.953343+0000","last_peered":"2024-02-10T08:37:19.953343+0000","last_clean":"2024-02-10T08:37:19.953343+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:37:19.953343+0000","last_undegraded":"2024-02-10T08:37:19.953343+0000","last_fullsized":"2024-02-10T08:37:19.953343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:35:38.321694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.321694+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961772+0000","last_change":"2024-02-10T08:37:10.153879+0000","last_active":"2024-02-10T08:37:19.961772+0000","last_peered":"2024-02-10T08:37:19.961772+0000","last_clean":"2024-02-10T08:37:19.961772+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:37:19.961772+0000","last_undegraded":"2024-02-10T08:37:19.961772+0000","last_fullsized":"2024-02-10T08:37:19.961772+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"103'2","reported_seq":36,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949509+0000","last_change":"2024-02-10T08:37:10.979010+0000","last_active":"2024-02-10T08:37:19.949509+0000","last_peered":"2024-02-10T08:37:19.949509+0000","last_clean":"2024-02-10T08:37:19.949509+0000","last_became_active":"2024-02-10T08:37:10.155420+0000","last_became_peered":"2024-02-10T08:37:10.155420+0000","last_unstale":"2024-02-10T08:37:19.949509+0000","last_undegraded":"2024-02-10T08:37:19.949509+0000","last_fullsized":"2024-02-10T08:37:19.949509+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":73,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.971487+0000","last_change":"2024-02-10T08:37:16.232414+0000","last_active":"2024-02-10T08:37:19.971487+0000","last_peered":"2024-02-10T08:37:19.971487+0000","last_clean":"2024-02-10T08:37:19.971487+0000","last_became_active":"2024-02-10T08:37:10.199231+0000","last_became_peered":"2024-02-10T08:37:10.199231+0000","last_unstale":"2024-02-10T08:37:19.971487+0000","last_undegraded":"2024-02-10T08:37:19.971487+0000","last_fullsized":"2024-02-10T08:37:19.971487+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.232331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:16.232331+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.232331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":106,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953072+0000","last_change":"2024-02-10T08:36:27.052479+0000","last_active":"2024-02-10T08:37:19.953072+0000","last_peered":"2024-02-10T08:37:19.953072+0000","last_clean":"2024-02-10T08:37:19.953072+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:37:19.953072+0000","last_undegraded":"2024-02-10T08:37:19.953072+0000","last_fullsized":"2024-02-10T08:37:19.953072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.052362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.052362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":130,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966544+0000","last_change":"2024-02-10T08:36:28.635087+0000","last_active":"2024-02-10T08:37:19.966544+0000","last_peered":"2024-02-10T08:37:19.966544+0000","last_clean":"2024-02-10T08:37:19.966544+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:37:19.966544+0000","last_undegraded":"2024-02-10T08:37:19.966544+0000","last_fullsized":"2024-02-10T08:37:19.966544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:28.635015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635015+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":127,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953679+0000","last_change":"2024-02-10T08:37:10.149914+0000","last_active":"2024-02-10T08:37:19.953679+0000","last_peered":"2024-02-10T08:37:19.953679+0000","last_clean":"2024-02-10T08:37:19.953679+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:37:19.953679+0000","last_undegraded":"2024-02-10T08:37:19.953679+0000","last_fullsized":"2024-02-10T08:37:19.953679+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.053809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.053809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964564+0000","last_change":"2024-02-10T08:36:30.795587+0000","last_active":"2024-02-10T08:37:19.964564+0000","last_peered":"2024-02-10T08:37:19.964564+0000","last_clean":"2024-02-10T08:37:19.964564+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:37:19.964564+0000","last_undegraded":"2024-02-10T08:37:19.964564+0000","last_fullsized":"2024-02-10T08:37:19.964564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950561+0000","last_change":"2024-02-10T08:34:56.469736+0000","last_active":"2024-02-10T08:37:19.950561+0000","last_peered":"2024-02-10T08:37:19.950561+0000","last_clean":"2024-02-10T08:37:19.950561+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:37:19.950561+0000","last_undegraded":"2024-02-10T08:37:19.950561+0000","last_fullsized":"2024-02-10T08:37:19.950561+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:34:56.469631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:34:56.469631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965856+0000","last_change":"2024-02-10T08:36:29.804845+0000","last_active":"2024-02-10T08:37:19.965856+0000","last_peered":"2024-02-10T08:37:19.965856+0000","last_clean":"2024-02-10T08:37:19.965856+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:37:19.965856+0000","last_undegraded":"2024-02-10T08:37:19.965856+0000","last_fullsized":"2024-02-10T08:37:19.965856+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:29.804761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.804761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957248+0000","last_change":"2024-02-10T08:37:10.151100+0000","last_active":"2024-02-10T08:37:19.957248+0000","last_peered":"2024-02-10T08:37:19.957248+0000","last_clean":"2024-02-10T08:37:19.957248+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:37:19.957248+0000","last_undegraded":"2024-02-10T08:37:19.957248+0000","last_fullsized":"2024-02-10T08:37:19.957248+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1a","version":"103'5","reported_seq":39,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950745+0000","last_change":"2024-02-10T08:37:10.316958+0000","last_active":"2024-02-10T08:37:19.950745+0000","last_peered":"2024-02-10T08:37:19.950745+0000","last_clean":"2024-02-10T08:37:19.950745+0000","last_became_active":"2024-02-10T08:37:10.208613+0000","last_became_peered":"2024-02-10T08:37:10.208613+0000","last_unstale":"2024-02-10T08:37:19.950745+0000","last_undegraded":"2024-02-10T08:37:19.950745+0000","last_fullsized":"2024-02-10T08:37:19.950745+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"103'6","reported_seq":60,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954105+0000","last_change":"2024-02-10T08:36:37.156906+0000","last_active":"2024-02-10T08:37:19.954105+0000","last_peered":"2024-02-10T08:37:19.954105+0000","last_clean":"2024-02-10T08:37:19.954105+0000","last_became_active":"2024-02-10T08:36:37.156334+0000","last_became_peered":"2024-02-10T08:36:37.156334+0000","last_unstale":"2024-02-10T08:37:19.954105+0000","last_undegraded":"2024-02-10T08:37:19.954105+0000","last_fullsized":"2024-02-10T08:37:19.954105+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":124,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958288+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:37:19.958288+0000","last_peered":"2024-02-10T08:37:19.958288+0000","last_clean":"2024-02-10T08:37:19.958288+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:37:19.958288+0000","last_undegraded":"2024-02-10T08:37:19.958288+0000","last_fullsized":"2024-02-10T08:37:19.958288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":156,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955351+0000","last_change":"2024-02-10T08:37:10.647803+0000","last_active":"2024-02-10T08:37:19.955351+0000","last_peered":"2024-02-10T08:37:19.955351+0000","last_clean":"2024-02-10T08:37:19.955351+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:37:19.955351+0000","last_undegraded":"2024-02-10T08:37:19.955351+0000","last_fullsized":"2024-02-10T08:37:19.955351+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:35:42.326627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.326627+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951304+0000","last_change":"2024-02-10T08:37:10.234765+0000","last_active":"2024-02-10T08:37:19.951304+0000","last_peered":"2024-02-10T08:37:19.951304+0000","last_clean":"2024-02-10T08:37:19.951304+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:37:19.951304+0000","last_undegraded":"2024-02-10T08:37:19.951304+0000","last_fullsized":"2024-02-10T08:37:19.951304+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:28.635610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":350,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.546080+0000","last_change":"2024-02-10T08:36:33.452480+0000","last_active":"2024-02-10T08:37:20.546080+0000","last_peered":"2024-02-10T08:37:20.546080+0000","last_clean":"2024-02-10T08:37:20.546080+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:37:20.546080+0000","last_undegraded":"2024-02-10T08:37:20.546080+0000","last_fullsized":"2024-02-10T08:37:20.546080+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.452367+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":236,"num_read_kb":231,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949855+0000","last_change":"2024-02-10T08:36:32.132836+0000","last_active":"2024-02-10T08:37:19.949855+0000","last_peered":"2024-02-10T08:37:19.949855+0000","last_clean":"2024-02-10T08:37:19.949855+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:37:19.949855+0000","last_undegraded":"2024-02-10T08:37:19.949855+0000","last_fullsized":"2024-02-10T08:37:19.949855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:32.132760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.132760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.947564+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:37:19.947564+0000","last_peered":"2024-02-10T08:37:19.947564+0000","last_clean":"2024-02-10T08:37:19.947564+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:37:19.947564+0000","last_undegraded":"2024-02-10T08:37:19.947564+0000","last_fullsized":"2024-02-10T08:37:19.947564+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969174+0000","last_change":"2024-02-10T08:36:37.077971+0000","last_active":"2024-02-10T08:37:19.969174+0000","last_peered":"2024-02-10T08:37:19.969174+0000","last_clean":"2024-02-10T08:37:19.969174+0000","last_became_active":"2024-02-10T08:36:37.074340+0000","last_became_peered":"2024-02-10T08:36:37.074340+0000","last_unstale":"2024-02-10T08:37:19.969174+0000","last_undegraded":"2024-02-10T08:37:19.969174+0000","last_fullsized":"2024-02-10T08:37:19.969174+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"141'3","reported_seq":75,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953722+0000","last_change":"2024-02-10T08:37:15.839182+0000","last_active":"2024-02-10T08:37:19.953722+0000","last_peered":"2024-02-10T08:37:19.953722+0000","last_clean":"2024-02-10T08:37:19.953722+0000","last_became_active":"2024-02-10T08:36:37.079083+0000","last_became_peered":"2024-02-10T08:36:37.079083+0000","last_unstale":"2024-02-10T08:37:19.953722+0000","last_undegraded":"2024-02-10T08:37:19.953722+0000","last_fullsized":"2024-02-10T08:37:19.953722+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'3","last_scrub_stamp":"2024-02-10T08:37:15.839118+0000","last_deep_scrub":"141'3","last_deep_scrub_stamp":"2024-02-10T08:37:15.839118+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.839118+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959367+0000","last_change":"2024-02-10T08:37:10.160466+0000","last_active":"2024-02-10T08:37:19.959367+0000","last_peered":"2024-02-10T08:37:19.959367+0000","last_clean":"2024-02-10T08:37:19.959367+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:37:19.959367+0000","last_undegraded":"2024-02-10T08:37:19.959367+0000","last_fullsized":"2024-02-10T08:37:19.959367+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.962334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.962334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":143,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966291+0000","last_change":"2024-02-10T08:36:17.572570+0000","last_active":"2024-02-10T08:37:19.966291+0000","last_peered":"2024-02-10T08:37:19.966291+0000","last_clean":"2024-02-10T08:37:19.966291+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:37:19.966291+0000","last_undegraded":"2024-02-10T08:37:19.966291+0000","last_fullsized":"2024-02-10T08:37:19.966291+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:36:17.572475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.572475+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954334+0000","last_change":"2024-02-10T08:35:42.473094+0000","last_active":"2024-02-10T08:37:19.954334+0000","last_peered":"2024-02-10T08:37:19.954334+0000","last_clean":"2024-02-10T08:37:19.954334+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:37:19.954334+0000","last_undegraded":"2024-02-10T08:37:19.954334+0000","last_fullsized":"2024-02-10T08:37:19.954334+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:35:42.473021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963259+0000","last_change":"2024-02-10T08:35:04.456788+0000","last_active":"2024-02-10T08:37:19.963259+0000","last_peered":"2024-02-10T08:37:19.963259+0000","last_clean":"2024-02-10T08:37:19.963259+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:37:19.963259+0000","last_undegraded":"2024-02-10T08:37:19.963259+0000","last_fullsized":"2024-02-10T08:37:19.963259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:04.456710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:04.456710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949686+0000","last_change":"2024-02-10T08:37:10.175199+0000","last_active":"2024-02-10T08:37:19.949686+0000","last_peered":"2024-02-10T08:37:19.949686+0000","last_clean":"2024-02-10T08:37:19.949686+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:37:19.949686+0000","last_undegraded":"2024-02-10T08:37:19.949686+0000","last_fullsized":"2024-02-10T08:37:19.949686+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"103'5","reported_seq":39,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955682+0000","last_change":"2024-02-10T08:37:10.332044+0000","last_active":"2024-02-10T08:37:19.955682+0000","last_peered":"2024-02-10T08:37:19.955682+0000","last_clean":"2024-02-10T08:37:19.955682+0000","last_became_active":"2024-02-10T08:37:10.204992+0000","last_became_peered":"2024-02-10T08:37:10.204992+0000","last_unstale":"2024-02-10T08:37:19.955682+0000","last_undegraded":"2024-02-10T08:37:19.955682+0000","last_fullsized":"2024-02-10T08:37:19.955682+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"103'1","reported_seq":36,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957352+0000","last_change":"2024-02-10T08:37:10.670782+0000","last_active":"2024-02-10T08:37:19.957352+0000","last_peered":"2024-02-10T08:37:19.957352+0000","last_clean":"2024-02-10T08:37:19.957352+0000","last_became_active":"2024-02-10T08:37:10.187643+0000","last_became_peered":"2024-02-10T08:37:10.187643+0000","last_unstale":"2024-02-10T08:37:19.957352+0000","last_undegraded":"2024-02-10T08:37:19.957352+0000","last_fullsized":"2024-02-10T08:37:19.957352+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":48,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.971566+0000","last_change":"2024-02-10T08:37:10.406267+0000","last_active":"2024-02-10T08:37:19.971566+0000","last_peered":"2024-02-10T08:37:19.971566+0000","last_clean":"2024-02-10T08:37:19.971566+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:37:19.971566+0000","last_undegraded":"2024-02-10T08:37:19.971566+0000","last_fullsized":"2024-02-10T08:37:19.971566+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:27.957926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.957926+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":128,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954692+0000","last_change":"2024-02-10T08:35:34.249489+0000","last_active":"2024-02-10T08:37:19.954692+0000","last_peered":"2024-02-10T08:37:19.954692+0000","last_clean":"2024-02-10T08:37:19.954692+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:37:19.954692+0000","last_undegraded":"2024-02-10T08:37:19.954692+0000","last_fullsized":"2024-02-10T08:37:19.954692+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:35:34.249424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953035+0000","last_change":"2024-02-10T08:37:10.183206+0000","last_active":"2024-02-10T08:37:19.953035+0000","last_peered":"2024-02-10T08:37:19.953035+0000","last_clean":"2024-02-10T08:37:19.953035+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:37:19.953035+0000","last_undegraded":"2024-02-10T08:37:19.953035+0000","last_fullsized":"2024-02-10T08:37:19.953035+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.456780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951263+0000","last_change":"2024-02-10T08:37:10.263452+0000","last_active":"2024-02-10T08:37:19.951263+0000","last_peered":"2024-02-10T08:37:19.951263+0000","last_clean":"2024-02-10T08:37:19.951263+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:37:19.951263+0000","last_undegraded":"2024-02-10T08:37:19.951263+0000","last_fullsized":"2024-02-10T08:37:19.951263+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.802264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.802264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963052+0000","last_change":"2024-02-10T08:36:37.155744+0000","last_active":"2024-02-10T08:37:19.963052+0000","last_peered":"2024-02-10T08:37:19.963052+0000","last_clean":"2024-02-10T08:37:19.963052+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:37:19.963052+0000","last_undegraded":"2024-02-10T08:37:19.963052+0000","last_fullsized":"2024-02-10T08:37:19.963052+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.14","version":"103'2","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962659+0000","last_change":"2024-02-10T08:36:37.065315+0000","last_active":"2024-02-10T08:37:19.962659+0000","last_peered":"2024-02-10T08:37:19.962659+0000","last_clean":"2024-02-10T08:37:19.962659+0000","last_became_active":"2024-02-10T08:36:37.063912+0000","last_became_peered":"2024-02-10T08:36:37.063912+0000","last_unstale":"2024-02-10T08:37:19.962659+0000","last_undegraded":"2024-02-10T08:37:19.962659+0000","last_fullsized":"2024-02-10T08:37:19.962659+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":64,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966888+0000","last_change":"2024-02-10T08:37:12.222315+0000","last_active":"2024-02-10T08:37:19.966888+0000","last_peered":"2024-02-10T08:37:19.966888+0000","last_clean":"2024-02-10T08:37:19.966888+0000","last_became_active":"2024-02-10T08:36:37.010946+0000","last_became_peered":"2024-02-10T08:36:37.010946+0000","last_unstale":"2024-02-10T08:37:19.966888+0000","last_undegraded":"2024-02-10T08:37:19.966888+0000","last_fullsized":"2024-02-10T08:37:19.966888+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:12.222224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:12.222224+0000","last_clean_scrub_stamp":"2024-02-10T08:37:12.222224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950004+0000","last_change":"2024-02-10T08:37:10.216642+0000","last_active":"2024-02-10T08:37:19.950004+0000","last_peered":"2024-02-10T08:37:19.950004+0000","last_clean":"2024-02-10T08:37:19.950004+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:37:19.950004+0000","last_undegraded":"2024-02-10T08:37:19.950004+0000","last_fullsized":"2024-02-10T08:37:19.950004+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:16.546674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.546674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968929+0000","last_change":"2024-02-10T08:36:28.744540+0000","last_active":"2024-02-10T08:37:19.968929+0000","last_peered":"2024-02-10T08:37:19.968929+0000","last_clean":"2024-02-10T08:37:19.968929+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:37:19.968929+0000","last_undegraded":"2024-02-10T08:37:19.968929+0000","last_fullsized":"2024-02-10T08:37:19.968929+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:36:28.744465+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.744465+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":138,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949541+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:37:19.949541+0000","last_peered":"2024-02-10T08:37:19.949541+0000","last_clean":"2024-02-10T08:37:19.949541+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:37:19.949541+0000","last_undegraded":"2024-02-10T08:37:19.949541+0000","last_fullsized":"2024-02-10T08:37:19.949541+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":103,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954388+0000","last_change":"2024-02-10T08:36:23.084433+0000","last_active":"2024-02-10T08:37:19.954388+0000","last_peered":"2024-02-10T08:37:19.954388+0000","last_clean":"2024-02-10T08:37:19.954388+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:37:19.954388+0000","last_undegraded":"2024-02-10T08:37:19.954388+0000","last_fullsized":"2024-02-10T08:37:19.954388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:23.084331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.084331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955483+0000","last_change":"2024-02-10T08:36:37.053619+0000","last_active":"2024-02-10T08:37:19.955483+0000","last_peered":"2024-02-10T08:37:19.955483+0000","last_clean":"2024-02-10T08:37:19.955483+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:37:19.955483+0000","last_undegraded":"2024-02-10T08:37:19.955483+0000","last_fullsized":"2024-02-10T08:37:19.955483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.b","version":"103'2","reported_seq":68,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957616+0000","last_change":"2024-02-10T08:37:10.677108+0000","last_active":"2024-02-10T08:37:19.957616+0000","last_peered":"2024-02-10T08:37:19.957616+0000","last_clean":"2024-02-10T08:37:19.957616+0000","last_became_active":"2024-02-10T08:37:10.216339+0000","last_became_peered":"2024-02-10T08:37:10.216339+0000","last_unstale":"2024-02-10T08:37:19.957616+0000","last_undegraded":"2024-02-10T08:37:19.957616+0000","last_fullsized":"2024-02-10T08:37:19.957616+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"103'1","reported_seq":36,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.967090+0000","last_change":"2024-02-10T08:37:10.407567+0000","last_active":"2024-02-10T08:37:19.967090+0000","last_peered":"2024-02-10T08:37:19.967090+0000","last_clean":"2024-02-10T08:37:19.967090+0000","last_became_active":"2024-02-10T08:37:10.152928+0000","last_became_peered":"2024-02-10T08:37:10.152928+0000","last_unstale":"2024-02-10T08:37:19.967090+0000","last_undegraded":"2024-02-10T08:37:19.967090+0000","last_fullsized":"2024-02-10T08:37:19.967090+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":7,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":89,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963308+0000","last_change":"2024-02-10T08:37:10.892256+0000","last_active":"2024-02-10T08:37:19.963308+0000","last_peered":"2024-02-10T08:37:19.963308+0000","last_clean":"2024-02-10T08:37:19.963308+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:37:19.963308+0000","last_undegraded":"2024-02-10T08:37:19.963308+0000","last_fullsized":"2024-02-10T08:37:19.963308+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:35:33.677377+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949928+0000","last_change":"2024-02-10T08:37:10.214852+0000","last_active":"2024-02-10T08:37:19.949928+0000","last_peered":"2024-02-10T08:37:19.949928+0000","last_clean":"2024-02-10T08:37:19.949928+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:37:19.949928+0000","last_undegraded":"2024-02-10T08:37:19.949928+0000","last_fullsized":"2024-02-10T08:37:19.949928+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.639381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.639381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":99,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952879+0000","last_change":"2024-02-10T08:35:07.725134+0000","last_active":"2024-02-10T08:37:19.952879+0000","last_peered":"2024-02-10T08:37:19.952879+0000","last_clean":"2024-02-10T08:37:19.952879+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:37:19.952879+0000","last_undegraded":"2024-02-10T08:37:19.952879+0000","last_fullsized":"2024-02-10T08:37:19.952879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:07.725060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:07.725060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961704+0000","last_change":"2024-02-10T08:36:37.041210+0000","last_active":"2024-02-10T08:37:19.961704+0000","last_peered":"2024-02-10T08:37:19.961704+0000","last_clean":"2024-02-10T08:37:19.961704+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:37:19.961704+0000","last_undegraded":"2024-02-10T08:37:19.961704+0000","last_fullsized":"2024-02-10T08:37:19.961704+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"103'10","reported_seq":46,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952606+0000","last_change":"2024-02-10T08:37:10.582273+0000","last_active":"2024-02-10T08:37:19.952606+0000","last_peered":"2024-02-10T08:37:19.952606+0000","last_clean":"2024-02-10T08:37:19.952606+0000","last_became_active":"2024-02-10T08:37:10.186893+0000","last_became_peered":"2024-02-10T08:37:10.186893+0000","last_unstale":"2024-02-10T08:37:19.952606+0000","last_undegraded":"2024-02-10T08:37:19.952606+0000","last_fullsized":"2024-02-10T08:37:19.952606+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":54,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958116+0000","last_change":"2024-02-10T08:36:36.997751+0000","last_active":"2024-02-10T08:37:19.958116+0000","last_peered":"2024-02-10T08:37:19.958116+0000","last_clean":"2024-02-10T08:37:19.958116+0000","last_became_active":"2024-02-10T08:36:36.996981+0000","last_became_peered":"2024-02-10T08:36:36.996981+0000","last_unstale":"2024-02-10T08:37:19.958116+0000","last_undegraded":"2024-02-10T08:37:19.958116+0000","last_fullsized":"2024-02-10T08:37:19.958116+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":116,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.967338+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:37:19.967338+0000","last_peered":"2024-02-10T08:37:19.967338+0000","last_clean":"2024-02-10T08:37:19.967338+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:37:19.967338+0000","last_undegraded":"2024-02-10T08:37:19.967338+0000","last_fullsized":"2024-02-10T08:37:19.967338+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.971755+0000","last_change":"2024-02-10T08:36:19.570275+0000","last_active":"2024-02-10T08:37:19.971755+0000","last_peered":"2024-02-10T08:37:19.971755+0000","last_clean":"2024-02-10T08:37:19.971755+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:37:19.971755+0000","last_undegraded":"2024-02-10T08:37:19.971755+0000","last_fullsized":"2024-02-10T08:37:19.971755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:19.570181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:19.570181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":99,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949622+0000","last_change":"2024-02-10T08:35:10.031019+0000","last_active":"2024-02-10T08:37:19.949622+0000","last_peered":"2024-02-10T08:37:19.949622+0000","last_clean":"2024-02-10T08:37:19.949622+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:37:19.949622+0000","last_undegraded":"2024-02-10T08:37:19.949622+0000","last_fullsized":"2024-02-10T08:37:19.949622+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:10.030939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:10.030939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964421+0000","last_change":"2024-02-10T08:36:37.051573+0000","last_active":"2024-02-10T08:37:19.964421+0000","last_peered":"2024-02-10T08:37:19.964421+0000","last_clean":"2024-02-10T08:37:19.964421+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:37:19.964421+0000","last_undegraded":"2024-02-10T08:37:19.964421+0000","last_fullsized":"2024-02-10T08:37:19.964421+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951017+0000","last_change":"2024-02-10T08:37:10.215944+0000","last_active":"2024-02-10T08:37:19.951017+0000","last_peered":"2024-02-10T08:37:19.951017+0000","last_clean":"2024-02-10T08:37:19.951017+0000","last_became_active":"2024-02-10T08:37:10.205941+0000","last_became_peered":"2024-02-10T08:37:10.205941+0000","last_unstale":"2024-02-10T08:37:19.951017+0000","last_undegraded":"2024-02-10T08:37:19.951017+0000","last_fullsized":"2024-02-10T08:37:19.951017+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"103'1","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955771+0000","last_change":"2024-02-10T08:36:37.075256+0000","last_active":"2024-02-10T08:37:19.955771+0000","last_peered":"2024-02-10T08:37:19.955771+0000","last_clean":"2024-02-10T08:37:19.955771+0000","last_became_active":"2024-02-10T08:36:37.074967+0000","last_became_peered":"2024-02-10T08:36:37.074967+0000","last_unstale":"2024-02-10T08:37:19.955771+0000","last_undegraded":"2024-02-10T08:37:19.955771+0000","last_fullsized":"2024-02-10T08:37:19.955771+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":122,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955466+0000","last_change":"2024-02-10T08:36:14.035460+0000","last_active":"2024-02-10T08:37:19.955466+0000","last_peered":"2024-02-10T08:37:19.955466+0000","last_clean":"2024-02-10T08:37:19.955466+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:37:19.955466+0000","last_undegraded":"2024-02-10T08:37:19.955466+0000","last_fullsized":"2024-02-10T08:37:19.955466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:14.035356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.035356+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954255+0000","last_change":"2024-02-10T08:36:12.633463+0000","last_active":"2024-02-10T08:37:19.954255+0000","last_peered":"2024-02-10T08:37:19.954255+0000","last_clean":"2024-02-10T08:37:19.954255+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:37:19.954255+0000","last_undegraded":"2024-02-10T08:37:19.954255+0000","last_fullsized":"2024-02-10T08:37:19.954255+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:12.633368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:12.633368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954699+0000","last_change":"2024-02-10T08:37:10.531826+0000","last_active":"2024-02-10T08:37:19.954699+0000","last_peered":"2024-02-10T08:37:19.954699+0000","last_clean":"2024-02-10T08:37:19.954699+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:37:19.954699+0000","last_undegraded":"2024-02-10T08:37:19.954699+0000","last_fullsized":"2024-02-10T08:37:19.954699+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:37:05.796026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:05.796026+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952922+0000","last_change":"2024-02-10T08:37:10.242150+0000","last_active":"2024-02-10T08:37:19.952922+0000","last_peered":"2024-02-10T08:37:19.952922+0000","last_clean":"2024-02-10T08:37:19.952922+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:37:19.952922+0000","last_undegraded":"2024-02-10T08:37:19.952922+0000","last_fullsized":"2024-02-10T08:37:19.952922+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.928693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949192+0000","last_change":"2024-02-10T08:37:10.232375+0000","last_active":"2024-02-10T08:37:19.949192+0000","last_peered":"2024-02-10T08:37:19.949192+0000","last_clean":"2024-02-10T08:37:19.949192+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:37:19.949192+0000","last_undegraded":"2024-02-10T08:37:19.949192+0000","last_fullsized":"2024-02-10T08:37:19.949192+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.8","version":"103'2","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949360+0000","last_change":"2024-02-10T08:36:37.035179+0000","last_active":"2024-02-10T08:37:19.949360+0000","last_peered":"2024-02-10T08:37:19.949360+0000","last_clean":"2024-02-10T08:37:19.949360+0000","last_became_active":"2024-02-10T08:36:37.023527+0000","last_became_peered":"2024-02-10T08:36:37.023527+0000","last_unstale":"2024-02-10T08:37:19.949360+0000","last_undegraded":"2024-02-10T08:37:19.949360+0000","last_fullsized":"2024-02-10T08:37:19.949360+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"104'1","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957552+0000","last_change":"2024-02-10T08:37:10.224100+0000","last_active":"2024-02-10T08:37:19.957552+0000","last_peered":"2024-02-10T08:37:19.957552+0000","last_clean":"2024-02-10T08:37:19.957552+0000","last_became_active":"2024-02-10T08:37:10.127797+0000","last_became_peered":"2024-02-10T08:37:10.127797+0000","last_unstale":"2024-02-10T08:37:19.957552+0000","last_undegraded":"2024-02-10T08:37:19.957552+0000","last_fullsized":"2024-02-10T08:37:19.957552+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":124,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954544+0000","last_change":"2024-02-10T08:37:18.722889+0000","last_active":"2024-02-10T08:37:19.954544+0000","last_peered":"2024-02-10T08:37:19.954544+0000","last_clean":"2024-02-10T08:37:19.954544+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:37:19.954544+0000","last_undegraded":"2024-02-10T08:37:19.954544+0000","last_fullsized":"2024-02-10T08:37:19.954544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:18.722818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.722818+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957842+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:37:19.957842+0000","last_peered":"2024-02-10T08:37:19.957842+0000","last_clean":"2024-02-10T08:37:19.957842+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:37:19.957842+0000","last_undegraded":"2024-02-10T08:37:19.957842+0000","last_fullsized":"2024-02-10T08:37:19.957842+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":161,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962984+0000","last_change":"2024-02-10T08:36:29.049034+0000","last_active":"2024-02-10T08:37:19.962984+0000","last_peered":"2024-02-10T08:37:19.962984+0000","last_clean":"2024-02-10T08:37:19.962984+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:37:19.962984+0000","last_undegraded":"2024-02-10T08:37:19.962984+0000","last_fullsized":"2024-02-10T08:37:19.962984+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:29.048965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.048965+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":99,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949818+0000","last_change":"2024-02-10T08:35:29.834206+0000","last_active":"2024-02-10T08:37:19.949818+0000","last_peered":"2024-02-10T08:37:19.949818+0000","last_clean":"2024-02-10T08:37:19.949818+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:37:19.949818+0000","last_undegraded":"2024-02-10T08:37:19.949818+0000","last_fullsized":"2024-02-10T08:37:19.949818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:29.834114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:29.834114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":66,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951101+0000","last_change":"2024-02-10T08:36:37.041395+0000","last_active":"2024-02-10T08:37:19.951101+0000","last_peered":"2024-02-10T08:37:19.951101+0000","last_clean":"2024-02-10T08:37:19.951101+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:37:19.951101+0000","last_undegraded":"2024-02-10T08:37:19.951101+0000","last_fullsized":"2024-02-10T08:37:19.951101+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.965943+0000","last_change":"2024-02-10T08:37:10.129525+0000","last_active":"2024-02-10T08:37:19.965943+0000","last_peered":"2024-02-10T08:37:19.965943+0000","last_clean":"2024-02-10T08:37:19.965943+0000","last_became_active":"2024-02-10T08:37:10.129023+0000","last_became_peered":"2024-02-10T08:37:10.129023+0000","last_unstale":"2024-02-10T08:37:19.965943+0000","last_undegraded":"2024-02-10T08:37:19.965943+0000","last_fullsized":"2024-02-10T08:37:19.965943+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1e","version":"105'1","reported_seq":59,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963052+0000","last_change":"2024-02-10T08:36:37.131330+0000","last_active":"2024-02-10T08:37:19.963052+0000","last_peered":"2024-02-10T08:37:19.963052+0000","last_clean":"2024-02-10T08:37:19.963052+0000","last_became_active":"2024-02-10T08:36:37.126890+0000","last_became_peered":"2024-02-10T08:36:37.126890+0000","last_unstale":"2024-02-10T08:37:19.963052+0000","last_undegraded":"2024-02-10T08:37:19.963052+0000","last_fullsized":"2024-02-10T08:37:19.963052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":128,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954505+0000","last_change":"2024-02-10T08:36:31.434160+0000","last_active":"2024-02-10T08:37:19.954505+0000","last_peered":"2024-02-10T08:37:19.954505+0000","last_clean":"2024-02-10T08:37:19.954505+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:37:19.954505+0000","last_undegraded":"2024-02-10T08:37:19.954505+0000","last_fullsized":"2024-02-10T08:37:19.954505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.434047+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":115,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961883+0000","last_change":"2024-02-10T08:37:18.260244+0000","last_active":"2024-02-10T08:37:19.961883+0000","last_peered":"2024-02-10T08:37:19.961883+0000","last_clean":"2024-02-10T08:37:19.961883+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:37:19.961883+0000","last_undegraded":"2024-02-10T08:37:19.961883+0000","last_fullsized":"2024-02-10T08:37:19.961883+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.260171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.260171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":114,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954349+0000","last_change":"2024-02-10T08:37:10.315695+0000","last_active":"2024-02-10T08:37:19.954349+0000","last_peered":"2024-02-10T08:37:19.954349+0000","last_clean":"2024-02-10T08:37:19.954349+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:37:19.954349+0000","last_undegraded":"2024-02-10T08:37:19.954349+0000","last_fullsized":"2024-02-10T08:37:19.954349+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953567+0000","last_change":"2024-02-10T08:37:10.150087+0000","last_active":"2024-02-10T08:37:19.953567+0000","last_peered":"2024-02-10T08:37:19.953567+0000","last_clean":"2024-02-10T08:37:19.953567+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:37:19.953567+0000","last_undegraded":"2024-02-10T08:37:19.953567+0000","last_fullsized":"2024-02-10T08:37:19.953567+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.978888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962717+0000","last_change":"2024-02-10T08:37:10.202917+0000","last_active":"2024-02-10T08:37:19.962717+0000","last_peered":"2024-02-10T08:37:19.962717+0000","last_clean":"2024-02-10T08:37:19.962717+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:37:19.962717+0000","last_undegraded":"2024-02-10T08:37:19.962717+0000","last_fullsized":"2024-02-10T08:37:19.962717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950342+0000","last_change":"2024-02-10T08:37:10.242206+0000","last_active":"2024-02-10T08:37:19.950342+0000","last_peered":"2024-02-10T08:37:19.950342+0000","last_clean":"2024-02-10T08:37:19.950342+0000","last_became_active":"2024-02-10T08:37:10.242025+0000","last_became_peered":"2024-02-10T08:37:10.242025+0000","last_unstale":"2024-02-10T08:37:19.950342+0000","last_undegraded":"2024-02-10T08:37:19.950342+0000","last_fullsized":"2024-02-10T08:37:19.950342+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":24,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964434+0000","last_change":"2024-02-10T08:37:10.208515+0000","last_active":"2024-02-10T08:37:19.964434+0000","last_peered":"2024-02-10T08:37:19.964434+0000","last_clean":"2024-02-10T08:37:19.964434+0000","last_became_active":"2024-02-10T08:37:10.208173+0000","last_became_peered":"2024-02-10T08:37:10.208173+0000","last_unstale":"2024-02-10T08:37:19.964434+0000","last_undegraded":"2024-02-10T08:37:19.964434+0000","last_fullsized":"2024-02-10T08:37:19.964434+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":122,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957765+0000","last_change":"2024-02-10T08:35:22.893394+0000","last_active":"2024-02-10T08:37:19.957765+0000","last_peered":"2024-02-10T08:37:19.957765+0000","last_clean":"2024-02-10T08:37:19.957765+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:37:19.957765+0000","last_undegraded":"2024-02-10T08:37:19.957765+0000","last_fullsized":"2024-02-10T08:37:19.957765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:35:22.893268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:22.893268+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959536+0000","last_change":"2024-02-10T08:37:10.151238+0000","last_active":"2024-02-10T08:37:19.959536+0000","last_peered":"2024-02-10T08:37:19.959536+0000","last_clean":"2024-02-10T08:37:19.959536+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:37:19.959536+0000","last_undegraded":"2024-02-10T08:37:19.959536+0000","last_fullsized":"2024-02-10T08:37:19.959536+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.453161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.453161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":106,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954595+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:37:19.954595+0000","last_peered":"2024-02-10T08:37:19.954595+0000","last_clean":"2024-02-10T08:37:19.954595+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:37:19.954595+0000","last_undegraded":"2024-02-10T08:37:19.954595+0000","last_fullsized":"2024-02-10T08:37:19.954595+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":112,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950085+0000","last_change":"2024-02-10T08:37:10.284262+0000","last_active":"2024-02-10T08:37:19.950085+0000","last_peered":"2024-02-10T08:37:19.950085+0000","last_clean":"2024-02-10T08:37:19.950085+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:37:19.950085+0000","last_undegraded":"2024-02-10T08:37:19.950085+0000","last_fullsized":"2024-02-10T08:37:19.950085+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.066605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.066605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959679+0000","last_change":"2024-02-10T08:37:10.260574+0000","last_active":"2024-02-10T08:37:19.959679+0000","last_peered":"2024-02-10T08:37:19.959679+0000","last_clean":"2024-02-10T08:37:19.959679+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:37:19.959679+0000","last_undegraded":"2024-02-10T08:37:19.959679+0000","last_fullsized":"2024-02-10T08:37:19.959679+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"103'5","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966692+0000","last_change":"2024-02-10T08:37:10.322611+0000","last_active":"2024-02-10T08:37:19.966692+0000","last_peered":"2024-02-10T08:37:19.966692+0000","last_clean":"2024-02-10T08:37:19.966692+0000","last_became_active":"2024-02-10T08:37:10.149021+0000","last_became_peered":"2024-02-10T08:37:10.149021+0000","last_unstale":"2024-02-10T08:37:19.966692+0000","last_undegraded":"2024-02-10T08:37:19.966692+0000","last_fullsized":"2024-02-10T08:37:19.966692+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"104'1001","reported_seq":2055,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966193+0000","last_change":"2024-02-10T08:36:37.024274+0000","last_active":"2024-02-10T08:37:19.966193+0000","last_peered":"2024-02-10T08:37:19.966193+0000","last_clean":"2024-02-10T08:37:19.966193+0000","last_became_active":"2024-02-10T08:36:37.023900+0000","last_became_peered":"2024-02-10T08:36:37.023900+0000","last_unstale":"2024-02-10T08:37:19.966193+0000","last_undegraded":"2024-02-10T08:37:19.966193+0000","last_fullsized":"2024-02-10T08:37:19.966193+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1001,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":113,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951176+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:37:19.951176+0000","last_peered":"2024-02-10T08:37:19.951176+0000","last_clean":"2024-02-10T08:37:19.951176+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:37:19.951176+0000","last_undegraded":"2024-02-10T08:37:19.951176+0000","last_fullsized":"2024-02-10T08:37:19.951176+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963095+0000","last_change":"2024-02-10T08:36:15.645089+0000","last_active":"2024-02-10T08:37:19.963095+0000","last_peered":"2024-02-10T08:37:19.963095+0000","last_clean":"2024-02-10T08:37:19.963095+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:37:19.963095+0000","last_undegraded":"2024-02-10T08:37:19.963095+0000","last_fullsized":"2024-02-10T08:37:19.963095+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.644998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.644998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":106,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.971675+0000","last_change":"2024-02-10T08:36:33.824585+0000","last_active":"2024-02-10T08:37:19.971675+0000","last_peered":"2024-02-10T08:37:19.971675+0000","last_clean":"2024-02-10T08:37:19.971675+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:37:19.971675+0000","last_undegraded":"2024-02-10T08:37:19.971675+0000","last_fullsized":"2024-02-10T08:37:19.971675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.824511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.824511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959599+0000","last_change":"2024-02-10T08:36:24.096165+0000","last_active":"2024-02-10T08:37:19.959599+0000","last_peered":"2024-02-10T08:37:19.959599+0000","last_clean":"2024-02-10T08:37:19.959599+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:37:19.959599+0000","last_undegraded":"2024-02-10T08:37:19.959599+0000","last_fullsized":"2024-02-10T08:37:19.959599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.096104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.096104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955250+0000","last_change":"2024-02-10T08:36:37.165190+0000","last_active":"2024-02-10T08:37:19.955250+0000","last_peered":"2024-02-10T08:37:19.955250+0000","last_clean":"2024-02-10T08:37:19.955250+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:37:19.955250+0000","last_undegraded":"2024-02-10T08:37:19.955250+0000","last_fullsized":"2024-02-10T08:37:19.955250+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966780+0000","last_change":"2024-02-10T08:36:37.127556+0000","last_active":"2024-02-10T08:37:19.966780+0000","last_peered":"2024-02-10T08:37:19.966780+0000","last_clean":"2024-02-10T08:37:19.966780+0000","last_became_active":"2024-02-10T08:36:37.127287+0000","last_became_peered":"2024-02-10T08:36:37.127287+0000","last_unstale":"2024-02-10T08:37:19.966780+0000","last_undegraded":"2024-02-10T08:37:19.966780+0000","last_fullsized":"2024-02-10T08:37:19.966780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"136'5","reported_seq":49,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958739+0000","last_change":"2024-02-10T08:37:10.905903+0000","last_active":"2024-02-10T08:37:19.958739+0000","last_peered":"2024-02-10T08:37:19.958739+0000","last_clean":"2024-02-10T08:37:19.958739+0000","last_became_active":"2024-02-10T08:37:10.274074+0000","last_became_peered":"2024-02-10T08:37:10.274074+0000","last_unstale":"2024-02-10T08:37:19.958739+0000","last_undegraded":"2024-02-10T08:37:19.958739+0000","last_fullsized":"2024-02-10T08:37:19.958739+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50331656,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":51902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":25165828,"num_keys_recovered":1000,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":126,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950566+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:37:19.950566+0000","last_peered":"2024-02-10T08:37:19.950566+0000","last_clean":"2024-02-10T08:37:19.950566+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:37:19.950566+0000","last_undegraded":"2024-02-10T08:37:19.950566+0000","last_fullsized":"2024-02-10T08:37:19.950566+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.969400+0000","last_change":"2024-02-10T08:36:13.537919+0000","last_active":"2024-02-10T08:37:19.969400+0000","last_peered":"2024-02-10T08:37:19.969400+0000","last_clean":"2024-02-10T08:37:19.969400+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:37:19.969400+0000","last_undegraded":"2024-02-10T08:37:19.969400+0000","last_fullsized":"2024-02-10T08:37:19.969400+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.537840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.537840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":106,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953897+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:37:19.953897+0000","last_peered":"2024-02-10T08:37:19.953897+0000","last_clean":"2024-02-10T08:37:19.953897+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:37:19.953897+0000","last_undegraded":"2024-02-10T08:37:19.953897+0000","last_fullsized":"2024-02-10T08:37:19.953897+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":112,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954434+0000","last_change":"2024-02-10T08:37:10.262623+0000","last_active":"2024-02-10T08:37:19.954434+0000","last_peered":"2024-02-10T08:37:19.954434+0000","last_clean":"2024-02-10T08:37:19.954434+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:37:19.954434+0000","last_undegraded":"2024-02-10T08:37:19.954434+0000","last_fullsized":"2024-02-10T08:37:19.954434+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.042582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.042582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.967035+0000","last_change":"2024-02-10T08:37:10.148262+0000","last_active":"2024-02-10T08:37:19.967035+0000","last_peered":"2024-02-10T08:37:19.967035+0000","last_clean":"2024-02-10T08:37:19.967035+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:37:19.967035+0000","last_undegraded":"2024-02-10T08:37:19.967035+0000","last_fullsized":"2024-02-10T08:37:19.967035+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"103'5","reported_seq":58,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949556+0000","last_change":"2024-02-10T08:36:37.052377+0000","last_active":"2024-02-10T08:37:19.949556+0000","last_peered":"2024-02-10T08:37:19.949556+0000","last_clean":"2024-02-10T08:37:19.949556+0000","last_became_active":"2024-02-10T08:36:37.050808+0000","last_became_peered":"2024-02-10T08:36:37.050808+0000","last_unstale":"2024-02-10T08:37:19.949556+0000","last_undegraded":"2024-02-10T08:37:19.949556+0000","last_fullsized":"2024-02-10T08:37:19.949556+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"103'1","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961858+0000","last_change":"2024-02-10T08:36:37.028862+0000","last_active":"2024-02-10T08:37:19.961858+0000","last_peered":"2024-02-10T08:37:19.961858+0000","last_clean":"2024-02-10T08:37:19.961858+0000","last_became_active":"2024-02-10T08:36:37.012947+0000","last_became_peered":"2024-02-10T08:36:37.012947+0000","last_unstale":"2024-02-10T08:37:19.961858+0000","last_undegraded":"2024-02-10T08:37:19.961858+0000","last_fullsized":"2024-02-10T08:37:19.961858+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953080+0000","last_change":"2024-02-10T08:37:10.466717+0000","last_active":"2024-02-10T08:37:19.953080+0000","last_peered":"2024-02-10T08:37:19.953080+0000","last_clean":"2024-02-10T08:37:19.953080+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:37:19.953080+0000","last_undegraded":"2024-02-10T08:37:19.953080+0000","last_fullsized":"2024-02-10T08:37:19.953080+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:31.890386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.890386+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953475+0000","last_change":"2024-02-10T08:35:36.513411+0000","last_active":"2024-02-10T08:37:19.953475+0000","last_peered":"2024-02-10T08:37:19.953475+0000","last_clean":"2024-02-10T08:37:19.953475+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:37:19.953475+0000","last_undegraded":"2024-02-10T08:37:19.953475+0000","last_fullsized":"2024-02-10T08:37:19.953475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:36.513316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:36.513316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":105,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953222+0000","last_change":"2024-02-10T08:37:07.241312+0000","last_active":"2024-02-10T08:37:19.953222+0000","last_peered":"2024-02-10T08:37:19.953222+0000","last_clean":"2024-02-10T08:37:19.953222+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:37:19.953222+0000","last_undegraded":"2024-02-10T08:37:19.953222+0000","last_fullsized":"2024-02-10T08:37:19.953222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:07.241247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.241247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952747+0000","last_change":"2024-02-10T08:36:37.049548+0000","last_active":"2024-02-10T08:37:19.952747+0000","last_peered":"2024-02-10T08:37:19.952747+0000","last_clean":"2024-02-10T08:37:19.952747+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:37:19.952747+0000","last_undegraded":"2024-02-10T08:37:19.952747+0000","last_fullsized":"2024-02-10T08:37:19.952747+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959194+0000","last_change":"2024-02-10T08:36:37.074211+0000","last_active":"2024-02-10T08:37:19.959194+0000","last_peered":"2024-02-10T08:37:19.959194+0000","last_clean":"2024-02-10T08:37:19.959194+0000","last_became_active":"2024-02-10T08:36:37.073890+0000","last_became_peered":"2024-02-10T08:36:37.073890+0000","last_unstale":"2024-02-10T08:37:19.959194+0000","last_undegraded":"2024-02-10T08:37:19.959194+0000","last_fullsized":"2024-02-10T08:37:19.959194+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":54,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954071+0000","last_change":"2024-02-10T08:36:37.157054+0000","last_active":"2024-02-10T08:37:19.954071+0000","last_peered":"2024-02-10T08:37:19.954071+0000","last_clean":"2024-02-10T08:37:19.954071+0000","last_became_active":"2024-02-10T08:36:37.156550+0000","last_became_peered":"2024-02-10T08:36:37.156550+0000","last_unstale":"2024-02-10T08:37:19.954071+0000","last_undegraded":"2024-02-10T08:37:19.954071+0000","last_fullsized":"2024-02-10T08:37:19.954071+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":132,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:20.333435+0000","last_change":"2024-02-10T08:37:20.333410+0000","last_active":"2024-02-10T08:37:20.333435+0000","last_peered":"2024-02-10T08:37:20.333435+0000","last_clean":"2024-02-10T08:37:20.333435+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:37:20.333435+0000","last_undegraded":"2024-02-10T08:37:20.333435+0000","last_fullsized":"2024-02-10T08:37:20.333435+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:20.333345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.333345+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":118,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950267+0000","last_change":"2024-02-10T08:37:10.284428+0000","last_active":"2024-02-10T08:37:19.950267+0000","last_peered":"2024-02-10T08:37:19.950267+0000","last_clean":"2024-02-10T08:37:19.950267+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:37:19.950267+0000","last_undegraded":"2024-02-10T08:37:19.950267+0000","last_fullsized":"2024-02-10T08:37:19.950267+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.072982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959307+0000","last_change":"2024-02-10T08:36:25.503414+0000","last_active":"2024-02-10T08:37:19.959307+0000","last_peered":"2024-02-10T08:37:19.959307+0000","last_clean":"2024-02-10T08:37:19.959307+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:37:19.959307+0000","last_undegraded":"2024-02-10T08:37:19.959307+0000","last_fullsized":"2024-02-10T08:37:19.959307+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.503351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.503351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.947815+0000","last_change":"2024-02-10T08:36:37.124706+0000","last_active":"2024-02-10T08:37:19.947815+0000","last_peered":"2024-02-10T08:37:19.947815+0000","last_clean":"2024-02-10T08:37:19.947815+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:37:19.947815+0000","last_undegraded":"2024-02-10T08:37:19.947815+0000","last_fullsized":"2024-02-10T08:37:19.947815+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"103'5","reported_seq":58,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.951386+0000","last_change":"2024-02-10T08:36:37.063791+0000","last_active":"2024-02-10T08:37:19.951386+0000","last_peered":"2024-02-10T08:37:19.951386+0000","last_clean":"2024-02-10T08:37:19.951386+0000","last_became_active":"2024-02-10T08:36:37.063485+0000","last_became_peered":"2024-02-10T08:36:37.063485+0000","last_unstale":"2024-02-10T08:37:19.951386+0000","last_undegraded":"2024-02-10T08:37:19.951386+0000","last_fullsized":"2024-02-10T08:37:19.951386+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.10","version":"103'1","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953303+0000","last_change":"2024-02-10T08:36:37.021161+0000","last_active":"2024-02-10T08:37:19.953303+0000","last_peered":"2024-02-10T08:37:19.953303+0000","last_clean":"2024-02-10T08:37:19.953303+0000","last_became_active":"2024-02-10T08:36:37.017583+0000","last_became_peered":"2024-02-10T08:36:37.017583+0000","last_unstale":"2024-02-10T08:37:19.953303+0000","last_undegraded":"2024-02-10T08:37:19.953303+0000","last_fullsized":"2024-02-10T08:37:19.953303+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"63'15","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953161+0000","last_change":"2024-02-10T08:36:13.651116+0000","last_active":"2024-02-10T08:37:19.953161+0000","last_peered":"2024-02-10T08:37:19.953161+0000","last_clean":"2024-02-10T08:37:19.953161+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:37:19.953161+0000","last_undegraded":"2024-02-10T08:37:19.953161+0000","last_fullsized":"2024-02-10T08:37:19.953161+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:36:13.650984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.650984+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.968829+0000","last_change":"2024-02-10T08:36:34.837568+0000","last_active":"2024-02-10T08:37:19.968829+0000","last_peered":"2024-02-10T08:37:19.968829+0000","last_clean":"2024-02-10T08:37:19.968829+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:37:19.968829+0000","last_undegraded":"2024-02-10T08:37:19.968829+0000","last_fullsized":"2024-02-10T08:37:19.968829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:34.837483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:34.837483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950684+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:37:19.950684+0000","last_peered":"2024-02-10T08:37:19.950684+0000","last_clean":"2024-02-10T08:37:19.950684+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:37:19.950684+0000","last_undegraded":"2024-02-10T08:37:19.950684+0000","last_fullsized":"2024-02-10T08:37:19.950684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949102+0000","last_change":"2024-02-10T08:37:10.232241+0000","last_active":"2024-02-10T08:37:19.949102+0000","last_peered":"2024-02-10T08:37:19.949102+0000","last_clean":"2024-02-10T08:37:19.949102+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:37:19.949102+0000","last_undegraded":"2024-02-10T08:37:19.949102+0000","last_fullsized":"2024-02-10T08:37:19.949102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"105'10","reported_seq":69,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957725+0000","last_change":"2024-02-10T08:36:37.056718+0000","last_active":"2024-02-10T08:37:19.957725+0000","last_peered":"2024-02-10T08:37:19.957725+0000","last_clean":"2024-02-10T08:37:19.957725+0000","last_became_active":"2024-02-10T08:36:37.056435+0000","last_became_peered":"2024-02-10T08:36:37.056435+0000","last_unstale":"2024-02-10T08:37:19.957725+0000","last_undegraded":"2024-02-10T08:37:19.957725+0000","last_fullsized":"2024-02-10T08:37:19.957725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165892,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":130,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958312+0000","last_change":"2024-02-10T08:36:20.047350+0000","last_active":"2024-02-10T08:37:19.958312+0000","last_peered":"2024-02-10T08:37:19.958312+0000","last_clean":"2024-02-10T08:37:19.958312+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:37:19.958312+0000","last_undegraded":"2024-02-10T08:37:19.958312+0000","last_fullsized":"2024-02-10T08:37:19.958312+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:36:20.047272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.047272+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":109,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954306+0000","last_change":"2024-02-10T08:36:15.045147+0000","last_active":"2024-02-10T08:37:19.954306+0000","last_peered":"2024-02-10T08:37:19.954306+0000","last_clean":"2024-02-10T08:37:19.954306+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:37:19.954306+0000","last_undegraded":"2024-02-10T08:37:19.954306+0000","last_fullsized":"2024-02-10T08:37:19.954306+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.045044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.045044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":103,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.967456+0000","last_change":"2024-02-10T08:36:27.054202+0000","last_active":"2024-02-10T08:37:19.967456+0000","last_peered":"2024-02-10T08:37:19.967456+0000","last_clean":"2024-02-10T08:37:19.967456+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:37:19.967456+0000","last_undegraded":"2024-02-10T08:37:19.967456+0000","last_fullsized":"2024-02-10T08:37:19.967456+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.054145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.054145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.953994+0000","last_change":"2024-02-10T08:36:37.038364+0000","last_active":"2024-02-10T08:37:19.953994+0000","last_peered":"2024-02-10T08:37:19.953994+0000","last_clean":"2024-02-10T08:37:19.953994+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:37:19.953994+0000","last_undegraded":"2024-02-10T08:37:19.953994+0000","last_fullsized":"2024-02-10T08:37:19.953994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"103'5","reported_seq":58,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949277+0000","last_change":"2024-02-10T08:36:37.017992+0000","last_active":"2024-02-10T08:37:19.949277+0000","last_peered":"2024-02-10T08:37:19.949277+0000","last_clean":"2024-02-10T08:37:19.949277+0000","last_became_active":"2024-02-10T08:36:37.016870+0000","last_became_peered":"2024-02-10T08:36:37.016870+0000","last_unstale":"2024-02-10T08:37:19.949277+0000","last_undegraded":"2024-02-10T08:37:19.949277+0000","last_fullsized":"2024-02-10T08:37:19.949277+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958744+0000","last_change":"2024-02-10T08:37:10.112153+0000","last_active":"2024-02-10T08:37:19.958744+0000","last_peered":"2024-02-10T08:37:19.958744+0000","last_clean":"2024-02-10T08:37:19.958744+0000","last_became_active":"2024-02-10T08:37:10.111814+0000","last_became_peered":"2024-02-10T08:37:10.111814+0000","last_unstale":"2024-02-10T08:37:19.958744+0000","last_undegraded":"2024-02-10T08:37:19.958744+0000","last_fullsized":"2024-02-10T08:37:19.958744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"103'1","reported_seq":55,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963128+0000","last_change":"2024-02-10T08:36:37.085781+0000","last_active":"2024-02-10T08:37:19.963128+0000","last_peered":"2024-02-10T08:37:19.963128+0000","last_clean":"2024-02-10T08:37:19.963128+0000","last_became_active":"2024-02-10T08:36:37.082091+0000","last_became_peered":"2024-02-10T08:36:37.082091+0000","last_unstale":"2024-02-10T08:37:19.963128+0000","last_undegraded":"2024-02-10T08:37:19.963128+0000","last_fullsized":"2024-02-10T08:37:19.963128+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":143,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.962013+0000","last_change":"2024-02-10T08:37:10.483725+0000","last_active":"2024-02-10T08:37:19.962013+0000","last_peered":"2024-02-10T08:37:19.962013+0000","last_clean":"2024-02-10T08:37:19.962013+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:37:19.962013+0000","last_undegraded":"2024-02-10T08:37:19.962013+0000","last_fullsized":"2024-02-10T08:37:19.962013+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:24.764020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.764020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.952838+0000","last_change":"2024-02-10T08:36:30.157585+0000","last_active":"2024-02-10T08:37:19.952838+0000","last_peered":"2024-02-10T08:37:19.952838+0000","last_clean":"2024-02-10T08:37:19.952838+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:37:19.952838+0000","last_undegraded":"2024-02-10T08:37:19.952838+0000","last_fullsized":"2024-02-10T08:37:19.952838+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.157473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.157473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.947137+0000","last_change":"2024-02-10T08:37:10.187798+0000","last_active":"2024-02-10T08:37:19.947137+0000","last_peered":"2024-02-10T08:37:19.947137+0000","last_clean":"2024-02-10T08:37:19.947137+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:37:19.947137+0000","last_undegraded":"2024-02-10T08:37:19.947137+0000","last_fullsized":"2024-02-10T08:37:19.947137+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.877268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.877268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957575+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:37:19.957575+0000","last_peered":"2024-02-10T08:37:19.957575+0000","last_clean":"2024-02-10T08:37:19.957575+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:37:19.957575+0000","last_undegraded":"2024-02-10T08:37:19.957575+0000","last_fullsized":"2024-02-10T08:37:19.957575+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.6","version":"103'9","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957438+0000","last_change":"2024-02-10T08:36:37.066456+0000","last_active":"2024-02-10T08:37:19.957438+0000","last_peered":"2024-02-10T08:37:19.957438+0000","last_clean":"2024-02-10T08:37:19.957438+0000","last_became_active":"2024-02-10T08:36:37.065893+0000","last_became_peered":"2024-02-10T08:36:37.065893+0000","last_unstale":"2024-02-10T08:37:19.957438+0000","last_undegraded":"2024-02-10T08:37:19.957438+0000","last_fullsized":"2024-02-10T08:37:19.957438+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.17","version":"103'1","reported_seq":69,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.959026+0000","last_change":"2024-02-10T08:37:10.222838+0000","last_active":"2024-02-10T08:37:19.959026+0000","last_peered":"2024-02-10T08:37:19.959026+0000","last_clean":"2024-02-10T08:37:19.959026+0000","last_became_active":"2024-02-10T08:37:10.182209+0000","last_became_peered":"2024-02-10T08:37:10.182209+0000","last_unstale":"2024-02-10T08:37:19.959026+0000","last_undegraded":"2024-02-10T08:37:19.959026+0000","last_fullsized":"2024-02-10T08:37:19.959026+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":118,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957924+0000","last_change":"2024-02-10T08:36:32.709476+0000","last_active":"2024-02-10T08:37:19.957924+0000","last_peered":"2024-02-10T08:37:19.957924+0000","last_clean":"2024-02-10T08:37:19.957924+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:37:19.957924+0000","last_undegraded":"2024-02-10T08:37:19.957924+0000","last_fullsized":"2024-02-10T08:37:19.957924+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:36:32.709392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.709392+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949703+0000","last_change":"2024-02-10T08:37:10.243920+0000","last_active":"2024-02-10T08:37:19.949703+0000","last_peered":"2024-02-10T08:37:19.949703+0000","last_clean":"2024-02-10T08:37:19.949703+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:37:19.949703+0000","last_undegraded":"2024-02-10T08:37:19.949703+0000","last_fullsized":"2024-02-10T08:37:19.949703+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.107579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.107579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":99,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.957462+0000","last_change":"2024-02-10T08:35:08.524105+0000","last_active":"2024-02-10T08:37:19.957462+0000","last_peered":"2024-02-10T08:37:19.957462+0000","last_clean":"2024-02-10T08:37:19.957462+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:37:19.957462+0000","last_undegraded":"2024-02-10T08:37:19.957462+0000","last_fullsized":"2024-02-10T08:37:19.957462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:08.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:35:08.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.964513+0000","last_change":"2024-02-10T08:37:10.149854+0000","last_active":"2024-02-10T08:37:19.964513+0000","last_peered":"2024-02-10T08:37:19.964513+0000","last_clean":"2024-02-10T08:37:19.964513+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:37:19.964513+0000","last_undegraded":"2024-02-10T08:37:19.964513+0000","last_fullsized":"2024-02-10T08:37:19.964513+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958182+0000","last_change":"2024-02-10T08:36:37.078717+0000","last_active":"2024-02-10T08:37:19.958182+0000","last_peered":"2024-02-10T08:37:19.958182+0000","last_clean":"2024-02-10T08:37:19.958182+0000","last_became_active":"2024-02-10T08:36:37.077729+0000","last_became_peered":"2024-02-10T08:36:37.077729+0000","last_unstale":"2024-02-10T08:37:19.958182+0000","last_undegraded":"2024-02-10T08:37:19.958182+0000","last_fullsized":"2024-02-10T08:37:19.958182+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"136'2","reported_seq":72,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961874+0000","last_change":"2024-02-10T08:37:10.192268+0000","last_active":"2024-02-10T08:37:19.961874+0000","last_peered":"2024-02-10T08:37:19.961874+0000","last_clean":"2024-02-10T08:37:19.961874+0000","last_became_active":"2024-02-10T08:37:10.150428+0000","last_became_peered":"2024-02-10T08:37:10.150428+0000","last_unstale":"2024-02-10T08:37:19.961874+0000","last_undegraded":"2024-02-10T08:37:19.961874+0000","last_fullsized":"2024-02-10T08:37:19.961874+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165835,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":7,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":143,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.955450+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:37:19.955450+0000","last_peered":"2024-02-10T08:37:19.955450+0000","last_clean":"2024-02-10T08:37:19.955450+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:37:19.955450+0000","last_undegraded":"2024-02-10T08:37:19.955450+0000","last_fullsized":"2024-02-10T08:37:19.955450+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":118,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963258+0000","last_change":"2024-02-10T08:37:10.306481+0000","last_active":"2024-02-10T08:37:19.963258+0000","last_peered":"2024-02-10T08:37:19.963258+0000","last_clean":"2024-02-10T08:37:19.963258+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:37:19.963258+0000","last_undegraded":"2024-02-10T08:37:19.963258+0000","last_fullsized":"2024-02-10T08:37:19.963258+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.014690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.014690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":104,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.961928+0000","last_change":"2024-02-10T08:36:14.747640+0000","last_active":"2024-02-10T08:37:19.961928+0000","last_peered":"2024-02-10T08:37:19.961928+0000","last_clean":"2024-02-10T08:37:19.961928+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:37:19.961928+0000","last_undegraded":"2024-02-10T08:37:19.961928+0000","last_fullsized":"2024-02-10T08:37:19.961928+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.747521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.747521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":62,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966100+0000","last_change":"2024-02-10T08:37:10.145097+0000","last_active":"2024-02-10T08:37:19.966100+0000","last_peered":"2024-02-10T08:37:19.966100+0000","last_clean":"2024-02-10T08:37:19.966100+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:37:19.966100+0000","last_undegraded":"2024-02-10T08:37:19.966100+0000","last_fullsized":"2024-02-10T08:37:19.966100+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.963170+0000","last_change":"2024-02-10T08:36:37.164878+0000","last_active":"2024-02-10T08:37:19.963170+0000","last_peered":"2024-02-10T08:37:19.963170+0000","last_clean":"2024-02-10T08:37:19.963170+0000","last_became_active":"2024-02-10T08:36:37.163315+0000","last_became_peered":"2024-02-10T08:36:37.163315+0000","last_unstale":"2024-02-10T08:37:19.963170+0000","last_undegraded":"2024-02-10T08:37:19.963170+0000","last_fullsized":"2024-02-10T08:37:19.963170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"104'5","reported_seq":63,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.949552+0000","last_change":"2024-02-10T08:36:37.044078+0000","last_active":"2024-02-10T08:37:19.949552+0000","last_peered":"2024-02-10T08:37:19.949552+0000","last_clean":"2024-02-10T08:37:19.949552+0000","last_became_active":"2024-02-10T08:36:37.043640+0000","last_became_peered":"2024-02-10T08:36:37.043640+0000","last_unstale":"2024-02-10T08:37:19.949552+0000","last_undegraded":"2024-02-10T08:37:19.949552+0000","last_fullsized":"2024-02-10T08:37:19.949552+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:30.795614+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795614+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.950648+0000","last_change":"2024-02-10T08:37:10.430238+0000","last_active":"2024-02-10T08:37:19.950648+0000","last_peered":"2024-02-10T08:37:19.950648+0000","last_clean":"2024-02-10T08:37:19.950648+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:37:19.950648+0000","last_undegraded":"2024-02-10T08:37:19.950648+0000","last_fullsized":"2024-02-10T08:37:19.950648+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:35:15.478955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:15.478955+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.958964+0000","last_change":"2024-02-10T08:37:10.112164+0000","last_active":"2024-02-10T08:37:19.958964+0000","last_peered":"2024-02-10T08:37:19.958964+0000","last_clean":"2024-02-10T08:37:19.958964+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:37:19.958964+0000","last_undegraded":"2024-02-10T08:37:19.958964+0000","last_fullsized":"2024-02-10T08:37:19.958964+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:17.894338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.894338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":103,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.954154+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:37:19.954154+0000","last_peered":"2024-02-10T08:37:19.954154+0000","last_clean":"2024-02-10T08:37:19.954154+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:37:19.954154+0000","last_undegraded":"2024-02-10T08:37:19.954154+0000","last_fullsized":"2024-02-10T08:37:19.954154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":53,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-10T08:37:19.966400+0000","last_change":"2024-02-10T08:36:37.083207+0000","last_active":"2024-02-10T08:37:19.966400+0000","last_peered":"2024-02-10T08:37:19.966400+0000","last_clean":"2024-02-10T08:37:19.966400+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:37:19.966400+0000","last_undegraded":"2024-02-10T08:37:19.966400+0000","last_fullsized":"2024-02-10T08:37:19.966400+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":95,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":93,"num_pg":8,"stat_sum":{"num_bytes":16384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":24,"acting":24,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":28,"num_write_kb":1049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":7424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":96,"acting":96,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":6272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":97,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":1421,"num_objects":159,"num_object_clones":54,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":164,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":4403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":236,"num_read_kb":231,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25166896,"num_keys_recovered":1000,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679809024,"data_stored":679497786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299443,"num_pgs":141,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422252,"kb_used_data":125028,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93322900,"statfs":{"total":95995035648,"available":95562649600,"internally_reserved":0,"allocated":128028672,"data_stored":126885935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":0.76200000000000001,"15min":0.60199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":22.143000000000001,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.361,"5min":0.68700000000000006,"15min":0.57699999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.7850000000000001,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.60899999999999999,"15min":0.54400000000000004},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.593,"5min":7.593,"15min":7.593},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.68400000000000005,"15min":0.59699999999999998},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.7759999999999998,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.65100000000000002,"15min":0.55100000000000005},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.9099999999999999,"5min":0.84399999999999997,"15min":0.64900000000000002},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":19.369,"5min":19.369,"15min":19.369},"last":0.433}]},{"osd":3,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.69699999999999995,"15min":0.56499999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.9710000000000001,"5min":0.83199999999999996,"15min":0.63200000000000001},"min":{"1min":0.34899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":22.102,"5min":22.102,"15min":22.102},"last":0.46200000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.50700000000000001,"15min":0.45200000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.827,"5min":8.827,"15min":8.827},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.03,"5min":0.56799999999999995,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.646000000000001,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.32300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.64300000000000002,"15min":0.505},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":19.326000000000001,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.378}]},{"osd":6,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.71199999999999997,"15min":0.56299999999999994},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":22.687000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.36299999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561081,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409740,"kb_used_data":100356,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93335412,"statfs":{"total":95995035648,"available":95575461888,"internally_reserved":0,"allocated":102764544,"data_stored":101730884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.29599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.505}]},{"osd":2,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.501},"min":{"1min":0.31,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.53000000000000003,"15min":0.49399999999999999},"min":{"1min":0.39500000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.49299999999999999},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.435}]},{"osd":4,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.39800000000000002,"15min":0.36399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55300000000000005,"5min":0.63,"15min":0.63},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.41499999999999998,"15min":0.379},"min":{"1min":0.26000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.46999999999999997,"15min":0.45900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64900000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":153,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409856,"kb_used_data":100472,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93335296,"statfs":{"total":95995035648,"available":95575343104,"internally_reserved":0,"allocated":102883328,"data_stored":101732898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.41299999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":1.103,"15min":0.61799999999999999},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":34.985999999999997,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":2.9609999999999999,"5min":0.879,"15min":0.53700000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.31}]},{"osd":3,"last update":"Sat Feb 10 08:36:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38600000000000001,"15min":0.41799999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.71399999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.41999999999999998,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:37:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.502}]},{"osd":5,"last update":"Sat Feb 10 08:36:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.40799999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95399999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.312,"15min":0.312},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.64200000000000002,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":0.71399999999999997,"15min":0.59399999999999997},"min":{"1min":0.35199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":9.4640000000000004,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.841,"5min":0.84099999999999997,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.9199999999999999,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.58099999999999996}]}]},{"osd":0,"up_from":0,"seq":34359738454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360628,"kb_used_data":51244,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93384524,"statfs":{"total":95995035648,"available":95625752576,"internally_reserved":0,"allocated":52473856,"data_stored":51387383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.371,"15min":0.35899999999999999},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":1.5740000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.622}]},{"osd":3,"last update":"Sat Feb 10 08:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.40999999999999998,"15min":0.41999999999999998},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":1.6319999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.44800000000000001,"15min":0.443},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.48,"5min":3.48,"15min":3.48},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Feb 10 08:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":0.74399999999999999,"15min":0.64200000000000002},"min":{"1min":0.52700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":9.2370000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":1.7270000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.441,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":1.8160000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":0.72699999999999998,"15min":0.63500000000000001},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.266,"5min":9.266,"15min":9.266},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.45300000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.76}]},{"osd":7,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.79700000000000004,"15min":0.69199999999999995},"min":{"1min":0.56699999999999995,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.5350000000000001,"5min":4.5350000000000001,"15min":4.5350000000000001},"last":1.7889999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":434360,"kb_used_data":124976,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93310792,"statfs":{"total":95995035648,"available":95550251008,"internally_reserved":0,"allocated":127975424,"data_stored":126880407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.47899999999999998,"15min":0.40999999999999998},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.744,"5min":14.744,"15min":14.744},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.2859999999999996,"5min":5.2859999999999996,"15min":5.2859999999999996},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":0.52100000000000002,"15min":0.46100000000000002},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":16.634,"5min":16.634,"15min":16.634},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.3740000000000001,"5min":0.57599999999999996,"15min":0.49099999999999999},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":12.438000000000001,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":0.63600000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":16.536000000000001,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.508,"5min":0.60599999999999998,"15min":0.48299999999999998},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":23.294,"5min":23.294,"15min":23.294},"last":0.25}]},{"osd":4,"last update":"Sat Feb 10 08:37:06 2024","interfaces":[{"interface":"back","average":{"1min":2.129,"5min":0.86899999999999999,"15min":0.70499999999999996},"min":{"1min":0.38900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":12.371,"5min":12.371,"15min":12.371},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.7190000000000001,"5min":0.77700000000000002,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":0.34200000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:36:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":8.5670000000000002,"5min":8.5670000000000002,"15min":8.5670000000000002},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":0.81100000000000005,"15min":0.627},"min":{"1min":0.502,"5min":0.374,"15min":0.374},"max":{"1min":16.091000000000001,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.7969999999999999,"5min":0.81000000000000005,"15min":0.63700000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":16.475000000000001,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.40200000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:36:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55600000000000005,"15min":0.54400000000000004},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.498},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71599999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.442}]}]},{"osd":4,"up_from":31,"seq":133143986242,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":352568,"kb_used_data":51248,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93392584,"statfs":{"total":95995035648,"available":95634006016,"internally_reserved":0,"allocated":52477952,"data_stored":51383783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":0.80200000000000005,"15min":0.63400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":19.940999999999999,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.157,"5min":0.623,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":9.1760000000000002,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":0.67900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":0.73199999999999998,"15min":0.61399999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.438}]},{"osd":2,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.74399999999999999,"15min":0.60999999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.38},{"interface":"front","average":{"1min":1.373,"5min":0.68700000000000006,"15min":0.59099999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.202,"5min":9.202,"15min":9.202},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1880000000000002,"5min":0.84099999999999997,"15min":0.64500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.5909999999999993,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.29499999999999998},{"interface":"front","average":{"1min":3.1360000000000001,"5min":1.044,"15min":0.70399999999999996},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":25.27,"5min":25.27,"15min":25.27},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.42299999999999999,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67500000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":0.57699999999999996,"15min":0.45300000000000001},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":13.342000000000001,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.48699999999999999,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.36599999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.497},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.41999999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":141,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422184,"kb_used_data":124960,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93322968,"statfs":{"total":95995035648,"available":95562719232,"internally_reserved":0,"allocated":127959040,"data_stored":126898599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.311},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.503,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.33100000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55000000000000004,"15min":0.55900000000000005},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61399999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.73699999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55100000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44400000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67400000000000004,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.44800000000000001,"15min":0.40200000000000002},"min":{"1min":0.34499999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34100000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:36:56 2024","interfaces":[{"interface":"back","average":{"1min":4.1600000000000001,"5min":1.1890000000000001,"15min":0.67100000000000004},"min":{"1min":0.28799999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":72.241,"5min":72.241,"15min":72.241},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.712,"5min":0.72599999999999998,"15min":0.54600000000000004},"min":{"1min":0.33700000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":22.469999999999999,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.36599999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:37:18 2024","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":1.1830000000000001,"15min":0.71099999999999997},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":66.844999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":4.1180000000000003,"5min":1.248,"15min":0.73999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":72.257000000000005,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100708352,"data_stored":100666160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50376704,"data_stored":50335527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":125865984,"data_stored":125829184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":50364416,"data_stored":50331707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":125894656,"data_stored":125834874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100712448,"data_stored":100667246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":125886464,"data_stored":125833088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":93,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":97,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:37:23.046 INFO:tasks.cephadm.ceph_manager.ceph:too many creating 2024-02-10T08:37:23.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:23.130 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.274726+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.277777+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.365331+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.366280+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.580121+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.f scrub starts 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.583840+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.f scrub ok 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.743240+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.746706+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.766529+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:21.769421+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:21.995792+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.099017+0000 mgr.y (mgr.14553) 185 : cluster [DBG] pgmap v197: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:23.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:22.202566+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.107:0/1619648577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:22.278446+0000 mgr.y (mgr.14553) 186 : audit [DBG] from='client.47377 v1:172.21.15.107:0/947283264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.914425+0000 mon.a (mon.0) 1175 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.920601+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:22.926362+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.107:0/2698823352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:22.928329+0000 mon.a (mon.0) 1177 : audit [INF] from='client.52604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.274726+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.277777+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.365331+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.366280+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.580121+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.f scrub starts 2024-02-10T08:37:23.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.583840+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.f scrub ok 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.743240+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.746706+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.766529+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:21.769421+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:21.995792+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.099017+0000 mgr.y (mgr.14553) 185 : cluster [DBG] pgmap v197: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:22.202566+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.107:0/1619648577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:22.278446+0000 mgr.y (mgr.14553) 186 : audit [DBG] from='client.47377 v1:172.21.15.107:0/947283264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.914425+0000 mon.a (mon.0) 1175 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.920601+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:22.926362+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.107:0/2698823352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:22.928329+0000 mon.a (mon.0) 1177 : audit [INF] from='client.52604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:23.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:23.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:23.466 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.274726+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:37:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.277777+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:37:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.365331+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.366280+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.580121+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.f scrub starts 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.583840+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.f scrub ok 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.743240+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.746706+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.766529+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:21.769421+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:21.995792+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.099017+0000 mgr.y (mgr.14553) 185 : cluster [DBG] pgmap v197: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:22.202566+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.107:0/1619648577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:22.278446+0000 mgr.y (mgr.14553) 186 : audit [DBG] from='client.47377 v1:172.21.15.107:0/947283264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.914425+0000 mon.a (mon.0) 1175 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.920601+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e149: 8 total, 8 up, 7 in 2024-02-10T08:37:23.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:22.926362+0000 mon.c (mon.2) 180 : audit [INF] from='client.? v1:172.21.15.107:0/2698823352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:22.928329+0000 mon.a (mon.0) 1177 : audit [INF] from='client.52604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:23.539 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:23.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:23.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:23.671 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:23.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:23.786 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:23.786 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:23.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:23.871 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:23.875 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:23.899 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:23.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:24.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:24.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.264345+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:37:24.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.265157+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:37:24.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.717406+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:37:24.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:22.720291+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:37:24.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:22.999998+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:23.367541+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:23.368536+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:23.925014+0000 mon.a (mon.0) 1178 : audit [INF] from='client.52604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:23.925210+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:23.928395+0000 mon.c (mon.2) 181 : audit [INF] from='client.? v1:172.21.15.107:0/1231717860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:23.928957+0000 mon.a (mon.0) 1180 : audit [INF] from='client.52622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.000808+0000 mon.b (mon.1) 196 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.068160+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.068557+0000 mon.a (mon.0) 1181 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.264345+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.265157+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.717406+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:22.720291+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:22.999998+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:23.367541+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:23.368536+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:23.925014+0000 mon.a (mon.0) 1178 : audit [INF] from='client.52604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:23.925210+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:23.928395+0000 mon.c (mon.2) 181 : audit [INF] from='client.? v1:172.21.15.107:0/1231717860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:23.928957+0000 mon.a (mon.0) 1180 : audit [INF] from='client.52622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.000808+0000 mon.b (mon.1) 196 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.068160+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.068557+0000 mon.a (mon.0) 1181 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.512 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.264345+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.265157+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.717406+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:22.720291+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:22.999998+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:23.367541+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:23.368536+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:23.925014+0000 mon.a (mon.0) 1178 : audit [INF] from='client.52604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi107-112411-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:23.925210+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e150: 8 total, 8 up, 7 in 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:23.928395+0000 mon.c (mon.2) 181 : audit [INF] from='client.? v1:172.21.15.107:0/1231717860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:23.928957+0000 mon.a (mon.0) 1180 : audit [INF] from='client.52622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.000808+0000 mon.b (mon.1) 196 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.068160+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.513 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.068557+0000 mon.a (mon.0) 1181 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:37:24.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T08:37:24.587 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:24.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:24.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:24.737 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:24.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:24.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:24.868 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:25.116 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:25.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:25.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:25.247 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:25.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:24.099516+0000 mgr.y (mgr.14553) 187 : cluster [DBG] pgmap v200: 386 pgs: 1 active+clean+snaptrim, 64 unknown, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:25.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.929810+0000 mon.a (mon.0) 1182 : audit [INF] from='client.52622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:25.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.929989+0000 mon.a (mon.0) 1183 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:25.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:24.930055+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.946775+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:24.947436+0000 mon.a (mon.0) 1185 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:24.099516+0000 mgr.y (mgr.14553) 187 : cluster [DBG] pgmap v200: 386 pgs: 1 active+clean+snaptrim, 64 unknown, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.929810+0000 mon.a (mon.0) 1182 : audit [INF] from='client.52622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.929989+0000 mon.a (mon.0) 1183 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:24.930055+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.946775+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:24.947436+0000 mon.a (mon.0) 1185 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:24.099516+0000 mgr.y (mgr.14553) 187 : cluster [DBG] pgmap v200: 386 pgs: 1 active+clean+snaptrim, 64 unknown, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 42 KiB/s wr, 40 op/s 2024-02-10T08:37:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.929810+0000 mon.a (mon.0) 1182 : audit [INF] from='client.52622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi107-112403-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.929989+0000 mon.a (mon.0) 1183 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:37:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:24.930055+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e151: 8 total, 8 up, 7 in 2024-02-10T08:37:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.946775+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:24.947436+0000 mon.a (mon.0) 1185 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:37:25.595 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:25.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:25.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.698+0000 7f5cfe008700 1 -- 172.21.15.107:0/3792368568 >> v1:172.21.15.107:6789/0 conn(0x7f5cf80f89c0 legacy=0x7f5cf80f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.698+0000 7f5cfe008700 1 -- 172.21.15.107:0/3792368568 shutdown_connections 2024-02-10T08:37:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.698+0000 7f5cfe008700 1 -- 172.21.15.107:0/3792368568 >> 172.21.15.107:0/3792368568 conn(0x7f5cf80f2510 msgr2=0x7f5cf80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.698+0000 7f5cfe008700 1 -- 172.21.15.107:0/3792368568 shutdown_connections 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.698+0000 7f5cfe008700 1 -- 172.21.15.107:0/3792368568 wait complete. 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 librados: starting msgr at 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 librados: starting objecter 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 Processor -- start 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 -- start start 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 librados: setting wanted keys 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 librados: calling monclient init 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cf81173e0 con 0x7f5cf80f9570 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cf81185c0 con 0x7f5cf8104350 2024-02-10T08:37:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cfe008700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cf81197c0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf77fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5cf80f89c0 0x7f5cf80ff510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40842/0 (socket says 172.21.15.107:40842) 2024-02-10T08:37:25.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf77fe700 1 -- 172.21.15.107:0/1467881293 learned_addr learned my addr 172.21.15.107:0/1467881293 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:25.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf6ffd700 1 --1- 172.21.15.107:0/1467881293 >> v1:172.21.15.107:6789/0 conn(0x7f5cf80f9570 0x7f5cf80ffc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43344/0 (socket says 172.21.15.107:43344) 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 10 client.?.objecter ms_handle_connect 0x7f5cf80f89c0 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 10 client.?.objecter ms_handle_connect 0x7f5cf80f9570 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223507893 0 0) 0x7f5cf81197c0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.699+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce0003030 con 0x7f5cf80f89c0 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223570155 0 0) 0x7f5cf81173e0 con 0x7f5cf80f9570 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf81197c0 con 0x7f5cf80f9570 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 10 client.?.objecter ms_handle_connect 0x7f5cf8104350 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039724545 0 0) 0x7f5cf81185c0 con 0x7f5cf8104350 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf81173e0 con 0x7f5cf8104350 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206706514 0 0) 0x7f5ce0003030 con 0x7f5cf80f89c0 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cf81185c0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5cec002070 con 0x7f5cf80f89c0 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3738047183 0 0) 0x7f5cf81197c0 con 0x7f5cf80f9570 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce0003030 con 0x7f5cf80f9570 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5ce8001920 con 0x7f5cf80f9570 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1881569151 0 0) 0x7f5cf81185c0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 >> v1:172.21.15.195:6789/0 conn(0x7f5cf8104350 legacy=0x7f5cf8115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 >> v1:172.21.15.107:6789/0 conn(0x7f5cf80f9570 legacy=0x7f5cf80ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf811a9c0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf81175d0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cfe008700 10 client.52634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf8117a40 con 0x7f5cf80f89c0 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cfe008700 1 librados: init done 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5cec002d60 con 0x7f5cf80f89c0 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.700+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5cec003990 con 0x7f5cf80f89c0 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.701+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4004f20 con 0x7f5cf80f89c0 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.701+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5cec002210 con 0x7f5cf80f89c0 2024-02-10T08:37:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(151..151 src has 50..151) v4 ==== 10189+0+0 (unknown 2573580298 0 0) 0x7f5cec079860 con 0x7f5cf80f89c0 2024-02-10T08:37:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 10 client.52634.objecter ms_dispatch 0x7f5cf8001550 osd_map(151..151 src has 50..151) v4 2024-02-10T08:37:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 3 client.52634.objecter handle_osd_map got epochs [151,151] > 0 2024-02-10T08:37:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 3 client.52634.objecter handle_osd_map decoding full epoch 151 2024-02-10T08:37:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 20 client.52634.objecter dump_active .. 0 homeless 2024-02-10T08:37:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.702+0000 7f5cf4ff9700 10 client.52634.objecter ms_handle_connect 0x7f5ce0063990 2024-02-10T08:37:25.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.706+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5cec04c4e0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.727 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:25] "GET /metrics HTTP/1.1" 200 196442 "" "Prometheus/2.33.4" 2024-02-10T08:37:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:25.749 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:25.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:25.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:37:25.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:25.880 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.889+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f5ce40051d0 con 0x7f5cf80f89c0 2024-02-10T08:37:25.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.917+0000 7f5cf4ff9700 1 -- 172.21.15.107:0/1467881293 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v152) v1 ==== 87+0+0 (unknown 124104692 0 0) 0x7f5cec04a950 con 0x7f5cf80f89c0 2024-02-10T08:37:25.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.922+0000 7f5cfe008700 10 librados: watch_flush enter 2024-02-10T08:37:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.927+0000 7f5cfe008700 10 librados: watch_flush exit 2024-02-10T08:37:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.927+0000 7f5cfe008700 20 client.52634.objecter shutdown clearing up homeless session... 2024-02-10T08:37:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.927+0000 7f5cfe008700 10 client.52634.objecter successfully canceled tick 2024-02-10T08:37:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.933+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5ce0063990 legacy=0x7f5ce0065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.933+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 >> v1:172.21.15.107:6790/0 conn(0x7f5cf80f89c0 legacy=0x7f5cf80ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.935+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 shutdown_connections 2024-02-10T08:37:25.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.935+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 >> 172.21.15.107:0/1467881293 conn(0x7f5cf80f2510 msgr2=0x7f5cf81077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:25.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.937+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 shutdown_connections 2024-02-10T08:37:25.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.939+0000 7f5cfe008700 1 -- 172.21.15.107:0/1467881293 wait complete. 2024-02-10T08:37:25.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:25.939+0000 7f5cfe008700 1 librados: shutdown 2024-02-10T08:37:25.952 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:37:26.078 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:26.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:26.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:26.208 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:26.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:26.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:26.367 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:26.386 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:26.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:26.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:26.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:24.677500+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.d scrub starts 2024-02-10T08:37:26.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:24.680404+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2024-02-10T08:37:26.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.001585+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.229000+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.328722+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:25.416990+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:25.417770+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.891525+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.107:0/1467881293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.891849+0000 mon.a (mon.0) 1188 : audit [INF] from='client.52634 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:25.913223+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.918777+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.918892+0000 mon.a (mon.0) 1191 : audit [INF] from='client.52634 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:25.918948+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.921447+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.928534+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:25.945432+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.002422+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:24.677500+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.d scrub starts 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:24.680404+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.001585+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.229000+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.328722+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:25.416990+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:25.417770+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.891525+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.107:0/1467881293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.891849+0000 mon.a (mon.0) 1188 : audit [INF] from='client.52634 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:25.913223+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:26.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.918777+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.918892+0000 mon.a (mon.0) 1191 : audit [INF] from='client.52634 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:25.918948+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.921447+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.928534+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:25.945432+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.002422+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.517 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:24.677500+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.d scrub starts 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:24.680404+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.001585+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.229000+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.328722+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:25.416990+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:25.417770+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.891525+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.107:0/1467881293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.891849+0000 mon.a (mon.0) 1188 : audit [INF] from='client.52634 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:25.913223+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.918777+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.918892+0000 mon.a (mon.0) 1191 : audit [INF] from='client.52634 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:25.918948+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e152: 8 total, 8 up, 7 in 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.921447+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.928534+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:25.945432+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]: dispatch 2024-02-10T08:37:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.002422+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:26.631 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:26.631 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:26.664 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:26.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:26.675 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:26.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:26.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:26.999 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:27.029 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:26.100639+0000 mgr.y (mgr.14553) 188 : cluster [DBG] pgmap v203: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 90 KiB/s wr, 88 op/s 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:26.915473+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.921142+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.921282+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]': finished 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:26.921390+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-10T08:37:27.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.934978+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.107:0/3275010171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.935104+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.946420+0000 mon.a (mon.0) 1199 : audit [INF] from='client.52640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:26.952878+0000 mon.a (mon.0) 1200 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.003239+0000 mon.b (mon.1) 203 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:27.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:37:27.380 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:27.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:27.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:27.510 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:27.529 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:27.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:27.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:27.651 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:26.100639+0000 mgr.y (mgr.14553) 188 : cluster [DBG] pgmap v203: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 90 KiB/s wr, 88 op/s 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:26.915473+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.921142+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.921282+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]': finished 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:26.921390+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.934978+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.107:0/3275010171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.935104+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.946420+0000 mon.a (mon.0) 1199 : audit [INF] from='client.52640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:26.952878+0000 mon.a (mon.0) 1200 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.003239+0000 mon.b (mon.1) 203 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:26.100639+0000 mgr.y (mgr.14553) 188 : cluster [DBG] pgmap v203: 354 pgs: 1 active+clean+snaptrim, 32 creating+peering, 321 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 3.2 KiB/s rd, 90 KiB/s wr, 88 op/s 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:26.915473+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.921142+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? v1:172.21.15.107:0/1426906385' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi107-112411-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.921282+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-13", "mode": "writeback"}]': finished 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:26.921390+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e153: 8 total, 8 up, 7 in 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.934978+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.107:0/3275010171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.935104+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.946420+0000 mon.a (mon.0) 1199 : audit [INF] from='client.52640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:27.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:26.952878+0000 mon.a (mon.0) 1200 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:27.661 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.003239+0000 mon.b (mon.1) 203 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:27.661 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:27.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:27.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:28.047 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:37:28.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:37:28.047 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-10T08:37:28.047 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:37:28.047 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:37:28.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-10T08:37:28.240 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:28.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.409623+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.409846+0000 mgr.y (mgr.14553) 189 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.410171+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.410365+0000 mgr.y (mgr.14553) 190 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.410806+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.410926+0000 mgr.y (mgr.14553) 191 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.411182+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.411307+0000 mgr.y (mgr.14553) 192 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.411547+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.411677+0000 mgr.y (mgr.14553) 193 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.411932+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.412050+0000 mgr.y (mgr.14553) 194 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.412278+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.412394+0000 mgr.y (mgr.14553) 195 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.412644+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.412774+0000 mgr.y (mgr.14553) 196 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.413019+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.413170+0000 mgr.y (mgr.14553) 197 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.413418+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.413543+0000 mgr.y (mgr.14553) 198 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:27.574617+0000 osd.4 (osd.4) 82 : cluster [DBG] 95.1 scrub starts 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:27.576270+0000 osd.4 (osd.4) 83 : cluster [DBG] 95.1 scrub ok 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.824847+0000 mon.c (mon.2) 184 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.924036+0000 mon.a (mon.0) 1201 : audit [INF] from='client.52640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.924118+0000 mon.a (mon.0) 1202 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:28.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:27.924165+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.936025+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:27.936693+0000 mon.a (mon.0) 1204 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.004151+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.409623+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.409846+0000 mgr.y (mgr.14553) 189 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.410171+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.410365+0000 mgr.y (mgr.14553) 190 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.410806+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.410926+0000 mgr.y (mgr.14553) 191 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.411182+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.411307+0000 mgr.y (mgr.14553) 192 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.411547+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.411677+0000 mgr.y (mgr.14553) 193 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.411932+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.412050+0000 mgr.y (mgr.14553) 194 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.412278+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.412394+0000 mgr.y (mgr.14553) 195 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.412644+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.412774+0000 mgr.y (mgr.14553) 196 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.413019+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.413170+0000 mgr.y (mgr.14553) 197 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.413418+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.413543+0000 mgr.y (mgr.14553) 198 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:27.574617+0000 osd.4 (osd.4) 82 : cluster [DBG] 95.1 scrub starts 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:27.576270+0000 osd.4 (osd.4) 83 : cluster [DBG] 95.1 scrub ok 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.824847+0000 mon.c (mon.2) 184 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.924036+0000 mon.a (mon.0) 1201 : audit [INF] from='client.52640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.924118+0000 mon.a (mon.0) 1202 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:27.924165+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.936025+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:27.936693+0000 mon.a (mon.0) 1204 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.004151+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:28.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:28.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:28.376 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.409623+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.409846+0000 mgr.y (mgr.14553) 189 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-10T08:37:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.410171+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.410365+0000 mgr.y (mgr.14553) 190 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-10T08:37:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.410806+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.410926+0000 mgr.y (mgr.14553) 191 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.411182+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.411307+0000 mgr.y (mgr.14553) 192 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.411547+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.411677+0000 mgr.y (mgr.14553) 193 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.411932+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.412050+0000 mgr.y (mgr.14553) 194 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.412278+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.412394+0000 mgr.y (mgr.14553) 195 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.412644+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.412774+0000 mgr.y (mgr.14553) 196 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.413019+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.413170+0000 mgr.y (mgr.14553) 197 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-10T08:37:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.413418+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.413543+0000 mgr.y (mgr.14553) 198 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:27.574617+0000 osd.4 (osd.4) 82 : cluster [DBG] 95.1 scrub starts 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:27.576270+0000 osd.4 (osd.4) 83 : cluster [DBG] 95.1 scrub ok 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.824847+0000 mon.c (mon.2) 184 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.924036+0000 mon.a (mon.0) 1201 : audit [INF] from='client.52640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi107-112403-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.924118+0000 mon.a (mon.0) 1202 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:27.924165+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e154: 8 total, 8 up, 7 in 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.936025+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.107:0/21513973' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:27.936693+0000 mon.a (mon.0) 1204 : audit [INF] from='client.44944 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]: dispatch 2024-02-10T08:37:28.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.004151+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:28.574 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:28.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:28.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:37:28.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:28.739 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.786+0000 7f50d68cd700 1 -- 172.21.15.107:0/2776290969 >> v1:172.21.15.107:6789/0 conn(0x7f50d0100830 legacy=0x7f50d0102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.786+0000 7f50d68cd700 1 -- 172.21.15.107:0/2776290969 shutdown_connections 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.786+0000 7f50d68cd700 1 -- 172.21.15.107:0/2776290969 >> 172.21.15.107:0/2776290969 conn(0x7f50d006fe70 msgr2=0x7f50d00f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 -- 172.21.15.107:0/2776290969 shutdown_connections 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 -- 172.21.15.107:0/2776290969 wait complete. 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 librados: starting msgr at 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 librados: starting objecter 2024-02-10T08:37:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 Processor -- start 2024-02-10T08:37:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 -- start start 2024-02-10T08:37:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 librados: setting wanted keys 2024-02-10T08:37:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.787+0000 7f50d68cd700 1 librados: calling monclient init 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50d68cd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50d0117370 con 0x7f50d00fd9e0 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50d68cd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50d0118550 con 0x7f50d0100830 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50d68cd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50d0119750 con 0x7f50d0104340 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50d4e6a700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f50d0104340 0x7f50d0115a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40908/0 (socket says 172.21.15.107:40908) 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 learned_addr learned my addr 172.21.15.107:0/3922910312 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cffff700 1 --1- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6789/0 conn(0x7f50d00fd9e0 0x7f50d00f9ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43400/0 (socket says 172.21.15.107:43400) 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter ms_handle_connect 0x7f50d0104340 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter ms_handle_connect 0x7f50d00fd9e0 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter ms_handle_connect 0x7f50d0100830 2024-02-10T08:37:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716373573 0 0) 0x7f50d0119750 con 0x7f50d0104340 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b80033f0 con 0x7f50d0104340 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272612850 0 0) 0x7f50d0117370 con 0x7f50d00fd9e0 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0119750 con 0x7f50d00fd9e0 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383041922 0 0) 0x7f50d0118550 con 0x7f50d0100830 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.788+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0117370 con 0x7f50d0100830 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806453697 0 0) 0x7f50b80033f0 con 0x7f50d0104340 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50d0118550 con 0x7f50d0104340 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f50c8002d40 con 0x7f50d0104340 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301168100 0 0) 0x7f50d0119750 con 0x7f50d00fd9e0 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b80033f0 con 0x7f50d00fd9e0 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049662409 0 0) 0x7f50d0117370 con 0x7f50d0100830 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50d0119750 con 0x7f50d0100830 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f50c0001910 con 0x7f50d00fd9e0 2024-02-10T08:37:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f50c4001f10 con 0x7f50d0100830 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3838410030 0 0) 0x7f50d0118550 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.195:6789/0 conn(0x7f50d0100830 legacy=0x7f50d00fa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6789/0 conn(0x7f50d00fd9e0 legacy=0x7f50d00f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d011a950 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d0117560 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50d68cd700 10 client.52652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d0117910 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50d68cd700 1 librados: init done 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f50c8003000 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.789+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f50c8005ed0 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.790+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f50bc004f00 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.790+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f50c8018d60 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(154..154 src has 50..154) v4 ==== 9355+0+0 (unknown 2418919699 0 0) 0x7f50c80796a0 con 0x7f50d0104340 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 osd_map(154..154 src has 50..154) v4 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 3 client.52652.objecter handle_osd_map got epochs [154,154] > 0 2024-02-10T08:37:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 3 client.52652.objecter handle_osd_map decoding full epoch 154 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 20 client.52652.objecter dump_active .. 0 homeless 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50b8063e00 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v154) v1 ==== 54+0+16 (unknown 4044755937 0 2420000192) 0x7f50c807ba90 con 0x7f50d0104340 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 10 client.52652.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 1 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 1 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 0 1 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 4 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 10 client.52652.objecter _send_command 1 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.791+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc004950 con 0x7f50bc004530 2024-02-10T08:37:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc004530 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc004950 con 0x7f50bc004530 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 1 = system:0 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 4 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.792+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 0 1 2024-02-10T08:37:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:28.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 10 client.52652.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 2 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 2 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0051a0 osd=0 3 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 0 2 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 4 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 10 client.52652.objecter _send_command 2 2024-02-10T08:37:28.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.807+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc004d30 con 0x7f50bc004530 2024-02-10T08:37:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc004d30 con 0x7f50bc004530 2024-02-10T08:37:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 4 2024-02-10T08:37:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 0 2 2024-02-10T08:37:28.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:28.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 10 client.52652.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 1 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 4 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 10 client.52652.objecter _send_command 3 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc00c000 con 0x7f50bc004150 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.809+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc004150 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.810+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc00c000 con 0x7f50bc004150 2024-02-10T08:37:28.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.810+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:28.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.810+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 3 = system:0 2024-02-10T08:37:28.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.810+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 4 2024-02-10T08:37:28.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.810+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 1 3 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 10 client.52652.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 4 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 4 2024-02-10T08:37:28.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc003fe0 osd=1 3 2024-02-10T08:37:28.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 1 4 2024-02-10T08:37:28.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 4 2024-02-10T08:37:28.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 10 client.52652.objecter _send_command 4 2024-02-10T08:37:28.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.825+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc0018b0 con 0x7f50bc004150 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.827+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc0018b0 con 0x7f50bc004150 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.827+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.827+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.827+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 4 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.827+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 1 4 2024-02-10T08:37:28.829 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 10 client.52652.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 5 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 5 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 2 5 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 4 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 10 client.52652.objecter _send_command 5 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc003e40 con 0x7f50bc00d2f0 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.828+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc00d2f0 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.829+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc003e40 con 0x7f50bc00d2f0 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.829+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.829+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 5 = system:0 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.829+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 4 2024-02-10T08:37:28.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.829+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 2 5 2024-02-10T08:37:28.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 10 client.52652.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 6 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 6 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc003630 osd=2 3 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 2 6 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 4 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 10 client.52652.objecter _send_command 6 2024-02-10T08:37:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.843+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc00e720 con 0x7f50bc00d2f0 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.844+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc00e720 con 0x7f50bc00d2f0 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.844+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.844+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.844+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 4 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.844+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 2 6 2024-02-10T08:37:28.846 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 10 client.52652.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 7 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 7 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 3 7 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 4 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 10 client.52652.objecter _send_command 7 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc015a10 con 0x7f50bc011e50 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.845+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc011e50 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.846+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc015a10 con 0x7f50bc011e50 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.846+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.846+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 7 = system:0 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.846+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 4 2024-02-10T08:37:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.846+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 3 7 2024-02-10T08:37:28.858 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 10 client.52652.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 8 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 8 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0033c0 osd=3 3 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 3 8 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 4 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 10 client.52652.objecter _send_command 8 2024-02-10T08:37:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.860+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc00e190 con 0x7f50bc011e50 2024-02-10T08:37:28.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc00e190 con 0x7f50bc011e50 2024-02-10T08:37:28.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 4 2024-02-10T08:37:28.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 3 8 2024-02-10T08:37:28.864 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:28.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 10 client.52652.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 9 2024-02-10T08:37:28.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 9 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 4 9 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 4 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 10 client.52652.objecter _send_command 9 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.862+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc0162c0 con 0x7f50bc00ce50 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.863+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc00ce50 2024-02-10T08:37:28.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.864+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc0162c0 con 0x7f50bc00ce50 2024-02-10T08:37:28.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.864+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:28.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.864+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 9 = system:0 2024-02-10T08:37:28.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.864+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 4 2024-02-10T08:37:28.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.864+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 4 9 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 10 client.52652.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 10 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 10 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc00e190 osd=4 3 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 4 10 2024-02-10T08:37:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 4 2024-02-10T08:37:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 10 client.52652.objecter _send_command 10 2024-02-10T08:37:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.879+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc0162c0 con 0x7f50bc00ce50 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.880+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc0162c0 con 0x7f50bc00ce50 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.880+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.880+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.880+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 4 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.880+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 4 10 2024-02-10T08:37:28.882 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 10 client.52652.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 11 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 11 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 5 11 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 4 2024-02-10T08:37:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 10 client.52652.objecter _send_command 11 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc016ff0 con 0x7f50bc016a50 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.881+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc016a50 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.882+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc016ff0 con 0x7f50bc016a50 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.882+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.882+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 11 = system:0 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.882+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 4 2024-02-10T08:37:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.882+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 5 11 2024-02-10T08:37:28.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 10 client.52652.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 12 2024-02-10T08:37:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 12 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc0162c0 osd=5 3 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 5 12 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 4 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 10 client.52652.objecter _send_command 12 2024-02-10T08:37:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.899+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc016ff0 con 0x7f50bc016a50 2024-02-10T08:37:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc016ff0 con 0x7f50bc016a50 2024-02-10T08:37:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 4 2024-02-10T08:37:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 5 12 2024-02-10T08:37:28.904 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 10 client.52652.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 13 2024-02-10T08:37:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 13 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 6 13 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 4 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 10 client.52652.objecter _send_command 13 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.901+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc01fb80 con 0x7f50bc020620 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.902+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc020620 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.903+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc01fb80 con 0x7f50bc020620 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.903+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.903+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 13 = system:0 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.903+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 4 2024-02-10T08:37:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.903+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 6 13 2024-02-10T08:37:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 10 client.52652.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 14 2024-02-10T08:37:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 14 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc016ff0 osd=6 3 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 6 14 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 4 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 10 client.52652.objecter _send_command 14 2024-02-10T08:37:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.920+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc01fb80 con 0x7f50bc020620 2024-02-10T08:37:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc01fb80 con 0x7f50bc020620 2024-02-10T08:37:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 4 2024-02-10T08:37:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 6 14 2024-02-10T08:37:28.924 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 10 client.52652.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 15 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 15 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 7 15 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 4 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 10 client.52652.objecter _send_command 15 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.922+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50bc027330 con 0x7f50bc023820 2024-02-10T08:37:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.923+0000 7f50cd7fa700 10 client.52652.objecter ms_handle_connect 0x7f50bc023820 2024-02-10T08:37:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.925+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f50bc027330 con 0x7f50bc023820 2024-02-10T08:37:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.925+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.925+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 15 = system:0 2024-02-10T08:37:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.925+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 4 2024-02-10T08:37:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.925+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 7 15 2024-02-10T08:37:28.937 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-10T08:37:28.937 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2080 ms) 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3290 ms) 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2999 ms) 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1608 ms) 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (9977 ms total) 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-10T08:37:28.938 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4678 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4073 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8751 ms total) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2808 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2079 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2099 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1835 ms) 2024-02-10T08:37:28.939 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8821 ms total) 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4009 ms) 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3993 ms) 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8002 ms total) 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (58787 ms total) 2024-02-10T08:37:28.940 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112604 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112604 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112430 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112430 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112556 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112556 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112518 2024-02-10T08:37:28.941 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112518 2024-02-10T08:37:28.944 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 10 client.52652.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign -1 16 2024-02-10T08:37:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 20 client.52652.objecter _get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_remove -1 16 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 20 client.52652.objecter get_session s=0x7f50bc01fb80 osd=7 3 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 15 client.52652.objecter _session_command_op_assign 7 16 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 4 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 10 client.52652.objecter _send_command 16 2024-02-10T08:37:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.955+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f50bc027330 con 0x7f50bc023820 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.957+0000 7f50cd7fa700 1 -- 172.21.15.107:0/3922910312 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f50bc027330 con 0x7f50bc023820 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.957+0000 7f50cd7fa700 10 client.52652.objecter ms_dispatch 0x7f50d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.957+0000 7f50cd7fa700 10 client.52652.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.957+0000 7f50cd7fa700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 4 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.957+0000 7f50cd7fa700 15 client.52652.objecter _session_command_op_remove 7 16 2024-02-10T08:37:28.959 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:28.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 librados: watch_flush enter 2024-02-10T08:37:28.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 librados: watch_flush exit 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.0 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6801/3321596972 conn(0x7f50bc004530 legacy=0x7f50bc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0051a0 osd=0 1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6805/133516126 conn(0x7f50bc004150 legacy=0x7f50bc00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003fe0 osd=1 1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.2 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6809/3575881728 conn(0x7f50bc00d2f0 legacy=0x7f50bc00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc003630 osd=2 1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.3 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6813/2871825691 conn(0x7f50bc011e50 legacy=0x7f50bc014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0033c0 osd=3 1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.4 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.195:6800/2499671285 conn(0x7f50bc00ce50 legacy=0x7f50bc015bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc00e190 osd=4 1 2024-02-10T08:37:28.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.5 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.195:6804/4178321486 conn(0x7f50bc016a50 legacy=0x7f50bc01d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc0162c0 osd=5 1 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.6 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.195:6808/1417479017 conn(0x7f50bc020620 legacy=0x7f50bc020aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc016ff0 osd=6 1 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter close_session for osd.7 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.195:6812/1934176928 conn(0x7f50bc023820 legacy=0x7f50bc025c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter put_session s=0x7f50bc01fb80 osd=7 1 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 20 client.52652.objecter shutdown clearing up homeless session... 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d68cd700 10 client.52652.objecter successfully canceled tick 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 reap_dead start 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 reap_dead start 2024-02-10T08:37:28.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 reap_dead start 2024-02-10T08:37:28.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 reap_dead start 2024-02-10T08:37:28.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.958+0000 7f50d4e6a700 1 -- 172.21.15.107:0/3922910312 reap_dead start 2024-02-10T08:37:28.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6800/2634842009 conn(0x7f50b8063e00 legacy=0x7f50b80662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> v1:172.21.15.107:6790/0 conn(0x7f50d0104340 legacy=0x7f50d0115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 shutdown_connections 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 >> 172.21.15.107:0/3922910312 conn(0x7f50d006fe70 msgr2=0x7f50d00f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 shutdown_connections 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 -- 172.21.15.107:0/3922910312 wait complete. 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:28.960+0000 7f50d68cd700 1 librados: shutdown 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.966 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:37:28.973 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:29.087 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:29.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:29.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:29.222 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:27.662640+0000 osd.1 (osd.1) 97 : cluster [DBG] 95.8 scrub starts 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:27.666858+0000 osd.1 (osd.1) 98 : cluster [DBG] 95.8 scrub ok 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.101069+0000 mgr.y (mgr.14553) 199 : cluster [DBG] pgmap v206: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 82 KiB/s wr, 86 op/s 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.463718+0000 mon.c (mon.2) 185 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.464344+0000 mon.c (mon.2) 186 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.465133+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.465845+0000 mgr.y (mgr.14553) 200 : cluster [DBG] pgmap v207: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 93 KiB/s wr, 98 op/s 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.467106+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-10T08:37:29.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.469325+0000 mon.a (mon.0) 1205 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.574812+0000 osd.4 (osd.4) 84 : cluster [DBG] 95.4 scrub starts 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.576388+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.4 scrub ok 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.792492+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/3922910312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.918493+0000 mon.a (mon.0) 1206 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.926299+0000 mon.a (mon.0) 1207 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:28.926402+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.934415+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.107:0/4246505494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:28.946475+0000 mon.a (mon.0) 1209 : audit [INF] from='client.52646 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:29.005098+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:27.662640+0000 osd.1 (osd.1) 97 : cluster [DBG] 95.8 scrub starts 2024-02-10T08:37:29.242 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:27.666858+0000 osd.1 (osd.1) 98 : cluster [DBG] 95.8 scrub ok 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.101069+0000 mgr.y (mgr.14553) 199 : cluster [DBG] pgmap v206: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 82 KiB/s wr, 86 op/s 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.463718+0000 mon.c (mon.2) 185 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.464344+0000 mon.c (mon.2) 186 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.465133+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.465845+0000 mgr.y (mgr.14553) 200 : cluster [DBG] pgmap v207: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 93 KiB/s wr, 98 op/s 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.467106+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.469325+0000 mon.a (mon.0) 1205 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.574812+0000 osd.4 (osd.4) 84 : cluster [DBG] 95.4 scrub starts 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.576388+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.4 scrub ok 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.792492+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/3922910312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.918493+0000 mon.a (mon.0) 1206 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.926299+0000 mon.a (mon.0) 1207 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:28.926402+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-10T08:37:29.243 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.934415+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.107:0/4246505494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:28.946475+0000 mon.a (mon.0) 1209 : audit [INF] from='client.52646 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:29.005098+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:27.662640+0000 osd.1 (osd.1) 97 : cluster [DBG] 95.8 scrub starts 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:27.666858+0000 osd.1 (osd.1) 98 : cluster [DBG] 95.8 scrub ok 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.101069+0000 mgr.y (mgr.14553) 199 : cluster [DBG] pgmap v206: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 82 KiB/s wr, 86 op/s 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.463718+0000 mon.c (mon.2) 185 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.464344+0000 mon.c (mon.2) 186 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.465133+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.465845+0000 mgr.y (mgr.14553) 200 : cluster [DBG] pgmap v207: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s rd, 93 KiB/s wr, 98 op/s 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.467106+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.469325+0000 mon.a (mon.0) 1205 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.574812+0000 osd.4 (osd.4) 84 : cluster [DBG] 95.4 scrub starts 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.576388+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.4 scrub ok 2024-02-10T08:37:29.339 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.792492+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.107:0/3922910312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.918493+0000 mon.a (mon.0) 1206 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.926299+0000 mon.a (mon.0) 1207 : audit [INF] from='client.44944 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi107-112513-14"}]': finished 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:28.926402+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e155: 8 total, 8 up, 7 in 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.934415+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.107:0/4246505494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:28.946475+0000 mon.a (mon.0) 1209 : audit [INF] from='client.52646 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:29.340 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:29.005098+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:29.873 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:29.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:29.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:30.246 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:30.340 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:30.341 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:37:30.341 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:37:30.341 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:30.353 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:30.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:30.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:30.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:30.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:30.653 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:30.851 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:30.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:30.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:30.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:37:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:31.145 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:31.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:31.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:29.771507+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:29.772073+0000 osd.5 (osd.5) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:29.928550+0000 mon.a (mon.0) 1210 : audit [INF] from='client.52646 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:29.928690+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:29.930458+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:30.005887+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:30.466272+0000 mgr.y (mgr.14553) 201 : cluster [DBG] pgmap v210: 378 pgs: 64 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:30.914644+0000 mon.a (mon.0) 1213 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:30.920384+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:30.920538+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-10T08:37:31.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:29.771507+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:29.772073+0000 osd.5 (osd.5) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:29.928550+0000 mon.a (mon.0) 1210 : audit [INF] from='client.52646 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:29.928690+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:29.930458+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:30.005887+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:31.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:30.466272+0000 mgr.y (mgr.14553) 201 : cluster [DBG] pgmap v210: 378 pgs: 64 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:30.914644+0000 mon.a (mon.0) 1213 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:30.920384+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:30.920538+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:29.771507+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:29.772073+0000 osd.5 (osd.5) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:29.928550+0000 mon.a (mon.0) 1210 : audit [INF] from='client.52646 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi107-112411-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:29.928690+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e156: 8 total, 8 up, 7 in 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:29.930458+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:30.005887+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:30.466272+0000 mgr.y (mgr.14553) 201 : cluster [DBG] pgmap v210: 378 pgs: 64 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:30.914644+0000 mon.a (mon.0) 1213 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:30.920384+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? v1:172.21.15.107:0/709535540' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi107-112403-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:30.920538+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 7 in 2024-02-10T08:37:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:31.379 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:31.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 -- 172.21.15.107:0/3440105254 >> v1:172.21.15.107:6790/0 conn(0x7f59ec103250 legacy=0x7f59ec1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 -- 172.21.15.107:0/3440105254 shutdown_connections 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 -- 172.21.15.107:0/3440105254 >> 172.21.15.107:0/3440105254 conn(0x7f59ec0f2510 msgr2=0x7f59ec0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 -- 172.21.15.107:0/3440105254 shutdown_connections 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 -- 172.21.15.107:0/3440105254 wait complete. 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 librados: starting msgr at 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.387+0000 7f59f453c700 1 librados: starting objecter 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 Processor -- start 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 -- start start 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 librados: setting wanted keys 2024-02-10T08:37:31.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 librados: calling monclient init 2024-02-10T08:37:31.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59ec0fbfd0 con 0x7f59ec0fc910 2024-02-10T08:37:31.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59ec0fc1a0 con 0x7f59ec0ff710 2024-02-10T08:37:31.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f453c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f59ec11bb20 con 0x7f59ec103250 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f2ad9700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f59ec103250 0x7f59ec1183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52044/0 (socket says 172.21.15.107:52044) 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59f2ad9700 1 -- 172.21.15.107:0/1120638739 learned_addr learned my addr 172.21.15.107:0/1120638739 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter ms_handle_connect 0x7f59ec103250 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter ms_handle_connect 0x7f59ec0ff710 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512430824 0 0) 0x7f59ec0fc1a0 con 0x7f59ec0ff710 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d80033f0 con 0x7f59ec0ff710 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter ms_handle_connect 0x7f59ec0fc910 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026784547 0 0) 0x7f59ec11bb20 con 0x7f59ec103250 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ec0fc1a0 con 0x7f59ec103250 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403316823 0 0) 0x7f59ec0fbfd0 con 0x7f59ec0fc910 2024-02-10T08:37:31.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ec11bb20 con 0x7f59ec0fc910 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778327307 0 0) 0x7f59d80033f0 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59ec0fbfd0 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f59dc0018c0 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982745957 0 0) 0x7f59ec0fc1a0 con 0x7f59ec103250 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d80033f0 con 0x7f59ec103250 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.388+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f59e40023d0 con 0x7f59ec103250 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3496502857 0 0) 0x7f59ec0fbfd0 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 >> v1:172.21.15.107:6790/0 conn(0x7f59ec103250 legacy=0x7f59ec1183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 >> v1:172.21.15.107:6789/0 conn(0x7f59ec0fc910 legacy=0x7f59ec0fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec11ccc0 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec11bd10 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59f453c700 10 client.47425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec11c180 con 0x7f59ec0ff710 2024-02-10T08:37:31.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59f453c700 1 librados: init done 2024-02-10T08:37:31.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f59dc001f80 con 0x7f59ec0ff710 2024-02-10T08:37:31.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.389+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f59dc0032a0 con 0x7f59ec0ff710 2024-02-10T08:37:31.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.390+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0004f20 con 0x7f59ec0ff710 2024-02-10T08:37:31.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.390+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f59dc0026c0 con 0x7f59ec0ff710 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(157..157 src has 50..157) v4 ==== 9229+0+0 (unknown 4053708927 0 0) 0x7f59dc0769d0 con 0x7f59ec0ff710 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 10 client.47425.objecter ms_dispatch 0x7f59ec001550 osd_map(157..157 src has 50..157) v4 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 3 client.47425.objecter handle_osd_map got epochs [157,157] > 0 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 3 client.47425.objecter handle_osd_map decoding full epoch 157 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 20 client.47425.objecter dump_active .. 0 homeless 2024-02-10T08:37:31.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.391+0000 7f59e37fe700 10 client.47425.objecter ms_handle_connect 0x7f59d8063bc0 2024-02-10T08:37:31.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.395+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f59dc049930 con 0x7f59ec0ff710 2024-02-10T08:37:31.421 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:31.421 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-10T08:37:31.421 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:31.451 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:31.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:31.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:31.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.578+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f59d00051d0 con 0x7f59ec0ff710 2024-02-10T08:37:31.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:31.660 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:31.678 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:31.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:31.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:31.812 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:31.831 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:31.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:31.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.946+0000 7f59e37fe700 1 -- 172.21.15.107:0/1120638739 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v158) v1 ==== 97+0+0 (unknown 2960769672 0 0) 0x7f59dc047d40 con 0x7f59ec0ff710 2024-02-10T08:37:31.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.956+0000 7f59f453c700 10 librados: watch_flush enter 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 10 librados: watch_flush exit 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 20 client.47425.objecter shutdown clearing up homeless session... 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 10 client.47425.objecter successfully canceled tick 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 >> v1:172.21.15.107:6800/2634842009 conn(0x7f59d8063bc0 legacy=0x7f59d8066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 >> v1:172.21.15.195:6789/0 conn(0x7f59ec0ff710 legacy=0x7f59ec114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 shutdown_connections 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 >> 172.21.15.107:0/1120638739 conn(0x7f59ec0f2510 msgr2=0x7f59ec1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 shutdown_connections 2024-02-10T08:37:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 -- 172.21.15.107:0/1120638739 wait complete. 2024-02-10T08:37:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:31.961+0000 7f59f453c700 1 librados: shutdown 2024-02-10T08:37:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:37:31.981 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.006820+0000 mon.b (mon.1) 218 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.209959+0000 mgr.y (mgr.14553) 202 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.579422+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.107:0/1120638739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.579898+0000 mon.a (mon.0) 1216 : audit [INF] from='client.47425 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:31.937107+0000 mon.a (mon.0) 1217 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.942496+0000 mon.a (mon.0) 1218 : audit [INF] from='client.47425 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:32.070 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:31.942569+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-10T08:37:32.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.006820+0000 mon.b (mon.1) 218 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.209959+0000 mgr.y (mgr.14553) 202 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.579422+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.107:0/1120638739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.579898+0000 mon.a (mon.0) 1216 : audit [INF] from='client.47425 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:31.937107+0000 mon.a (mon.0) 1217 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.942496+0000 mon.a (mon.0) 1218 : audit [INF] from='client.47425 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:32.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:31.942569+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-10T08:37:32.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.006820+0000 mon.b (mon.1) 218 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:32.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.209959+0000 mgr.y (mgr.14553) 202 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:32.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.579422+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.107:0/1120638739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.579898+0000 mon.a (mon.0) 1216 : audit [INF] from='client.47425 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:32.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:31.937107+0000 mon.a (mon.0) 1217 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:32.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.942496+0000 mon.a (mon.0) 1218 : audit [INF] from='client.47425 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:32.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:31.942569+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e158: 8 total, 8 up, 7 in 2024-02-10T08:37:32.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:37:32.631 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:32.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:32.696 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:32.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:32.764 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:33.112 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:33.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:33.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:31.454647+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.0 scrub starts 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:31.458786+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.0 scrub ok 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.948424+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.107:0/2248689821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:31.958054+0000 mon.a (mon.0) 1220 : audit [INF] from='client.52661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:32.007906+0000 mon.b (mon.1) 220 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.466736+0000 mgr.y (mgr.14553) 203 : cluster [DBG] pgmap v213: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:32.943097+0000 mon.a (mon.0) 1221 : audit [INF] from='client.52661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.943270+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:32.956932+0000 mon.c (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.107:0/419848273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:32.959096+0000 mon.a (mon.0) 1223 : audit [INF] from='client.52673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.244 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:33.316 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:33.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:31.454647+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.0 scrub starts 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:31.458786+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.0 scrub ok 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.948424+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.107:0/2248689821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:31.958054+0000 mon.a (mon.0) 1220 : audit [INF] from='client.52661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:32.007906+0000 mon.b (mon.1) 220 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:33.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.466736+0000 mgr.y (mgr.14553) 203 : cluster [DBG] pgmap v213: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:32.943097+0000 mon.a (mon.0) 1221 : audit [INF] from='client.52661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.943270+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:32.956932+0000 mon.c (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.107:0/419848273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:32.959096+0000 mon.a (mon.0) 1223 : audit [INF] from='client.52673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:31.454647+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.0 scrub starts 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:31.458786+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.0 scrub ok 2024-02-10T08:37:33.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.948424+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.107:0/2248689821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:31.958054+0000 mon.a (mon.0) 1220 : audit [INF] from='client.52661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:32.007906+0000 mon.b (mon.1) 220 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.466736+0000 mgr.y (mgr.14553) 203 : cluster [DBG] pgmap v213: 346 pgs: 32 unknown, 314 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:32.943097+0000 mon.a (mon.0) 1221 : audit [INF] from='client.52661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi107-112411-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.943270+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e159: 8 total, 8 up, 7 in 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:32.956932+0000 mon.c (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.107:0/419848273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:32.959096+0000 mon.a (mon.0) 1223 : audit [INF] from='client.52673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:33.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:33.448 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:33.556 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:33.556 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:37:33.556 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:37:33.556 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:37:33.556 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:33.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:33.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:33.779 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:33.799 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:33.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:33.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:33.929 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:34.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:37:34.299 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:34.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:34.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.495265+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.497340+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.612122+0000 osd.5 (osd.5) 85 : cluster [DBG] 95.3 scrub starts 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:32.615835+0000 osd.5 (osd.5) 86 : cluster [DBG] 95.3 scrub ok 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:33.008829+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:33.945943+0000 mon.a (mon.0) 1224 : audit [INF] from='client.52673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:34.416 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:33.946092+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:33.949769+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.107:0/989472285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:33.950405+0000 mon.a (mon.0) 1226 : audit [INF] from='client.52679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.495265+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.497340+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.612122+0000 osd.5 (osd.5) 85 : cluster [DBG] 95.3 scrub starts 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:32.615835+0000 osd.5 (osd.5) 86 : cluster [DBG] 95.3 scrub ok 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:33.008829+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:33.945943+0000 mon.a (mon.0) 1224 : audit [INF] from='client.52673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:33.946092+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:33.949769+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.107:0/989472285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:33.950405+0000 mon.a (mon.0) 1226 : audit [INF] from='client.52679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.428 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:34.446 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:34.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.495265+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.497340+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.612122+0000 osd.5 (osd.5) 85 : cluster [DBG] 95.3 scrub starts 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:32.615835+0000 osd.5 (osd.5) 86 : cluster [DBG] 95.3 scrub ok 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:33.008829+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:33.945943+0000 mon.a (mon.0) 1224 : audit [INF] from='client.52673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi107-112403-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:33.946092+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e160: 8 total, 8 up, 7 in 2024-02-10T08:37:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:33.949769+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.107:0/989472285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:33.950405+0000 mon.a (mon.0) 1226 : audit [INF] from='client.52679 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:34.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:34.568 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:34.579 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:34.599 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:34.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:34.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:37:34.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.654+0000 7fd9b19f7700 1 -- 172.21.15.107:0/2643719324 >> v1:172.21.15.107:6790/0 conn(0x7fd9ac0fda50 legacy=0x7fd9ac0fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.654+0000 7fd9b19f7700 1 -- 172.21.15.107:0/2643719324 shutdown_connections 2024-02-10T08:37:34.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.654+0000 7fd9b19f7700 1 -- 172.21.15.107:0/2643719324 >> 172.21.15.107:0/2643719324 conn(0x7fd9ac0f3640 msgr2=0x7fd9ac0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:34.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.654+0000 7fd9b19f7700 1 -- 172.21.15.107:0/2643719324 shutdown_connections 2024-02-10T08:37:34.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.654+0000 7fd9b19f7700 1 -- 172.21.15.107:0/2643719324 wait complete. 2024-02-10T08:37:34.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 librados: starting msgr at 2024-02-10T08:37:34.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 librados: starting objecter 2024-02-10T08:37:34.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 Processor -- start 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 -- start start 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 librados: setting wanted keys 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 librados: calling monclient init 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9ac117330 con 0x7fd9ac104330 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9ac118510 con 0x7fd9ac1007f0 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9b19f7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9ac119710 con 0x7fd9ac0fda50 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.655+0000 7fd9aaffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd9ac0fda50 0x7fd9ac0f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52114/0 (socket says 172.21.15.107:52114) 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd9aaffd700 1 -- 172.21.15.107:0/676496404 learned_addr learned my addr 172.21.15.107:0/676496404 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter ms_handle_connect 0x7fd9ac0fda50 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter ms_handle_connect 0x7fd9ac104330 2024-02-10T08:37:34.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter ms_handle_connect 0x7fd9ac1007f0 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569235594 0 0) 0x7fd9ac117330 con 0x7fd9ac104330 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9940033f0 con 0x7fd9ac104330 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781198450 0 0) 0x7fd9ac119710 con 0x7fd9ac0fda50 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ac117330 con 0x7fd9ac0fda50 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351145100 0 0) 0x7fd9ac118510 con 0x7fd9ac1007f0 2024-02-10T08:37:34.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ac119710 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252346337 0 0) 0x7fd9ac119710 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9ac118510 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd9a0001840 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55975067 0 0) 0x7fd9940033f0 con 0x7fd9ac104330 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9ac119710 con 0x7fd9ac104330 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd9a4001bb0 con 0x7fd9ac104330 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3503056657 0 0) 0x7fd9ac117330 con 0x7fd9ac0fda50 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9940033f0 con 0x7fd9ac0fda50 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3327881663 0 0) 0x7fd9ac118510 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6790/0 conn(0x7fd9ac0fda50 legacy=0x7fd9ac0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6789/0 conn(0x7fd9ac104330 legacy=0x7fd9ac1159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac11a910 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.656+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac117520 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd9b19f7700 10 client.47446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac117a10 con 0x7fd9ac1007f0 2024-02-10T08:37:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd9b19f7700 1 librados: init done 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd9a0001e40 con 0x7fd9ac1007f0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd9a00031c0 con 0x7fd9ac1007f0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.657+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd998004f00 con 0x7fd9ac1007f0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.658+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd9a00019a0 con 0x7fd9ac1007f0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.658+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(160..160 src has 50..160) v4 ==== 9193+0+0 (unknown 2008323697 0 0) 0x7fd9a00766f0 con 0x7fd9ac1007f0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.658+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 osd_map(160..160 src has 50..160) v4 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.658+0000 7fd993fff700 3 client.47446.objecter handle_osd_map got epochs [160,160] > 0 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.658+0000 7fd993fff700 3 client.47446.objecter handle_osd_map decoding full epoch 160 2024-02-10T08:37:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd993fff700 20 client.47446.objecter dump_active .. 0 homeless 2024-02-10T08:37:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v160) v1 ==== 54+0+16 (unknown 1159791290 0 2420000192) 0x7fd9a0078ae0 con 0x7fd9ac1007f0 2024-02-10T08:37:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd994063e10 2024-02-10T08:37:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 1 2024-02-10T08:37:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 1 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 0 1 2024-02-10T08:37:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 4 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 10 client.47446.objecter _send_command 1 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd998004950 con 0x7fd998004530 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.659+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd998004530 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.660+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd998004950 con 0x7fd998004530 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.660+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.660+0000 7fd993fff700 10 client.47446.objecter _finish_command 1 = system:0 2024-02-10T08:37:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.660+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 4 2024-02-10T08:37:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.660+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 0 1 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 2 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 2 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd9980051a0 osd=0 3 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 0 2 2024-02-10T08:37:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 4 2024-02-10T08:37:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 10 client.47446.objecter _send_command 2 2024-02-10T08:37:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.677+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd998004d30 con 0x7fd998004530 2024-02-10T08:37:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd998004d30 con 0x7fd998004530 2024-02-10T08:37:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd993fff700 10 client.47446.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 4 2024-02-10T08:37:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 0 2 2024-02-10T08:37:34.679 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 3 2024-02-10T08:37:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 3 2024-02-10T08:37:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 1 3 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 4 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 10 client.47446.objecter _send_command 3 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.678+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd99800c000 con 0x7fd998004110 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd998004110 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd99800c000 con 0x7fd998004110 2024-02-10T08:37:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:34.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 10 client.47446.objecter _finish_command 3 = system:0 2024-02-10T08:37:34.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 4 2024-02-10T08:37:34.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.679+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 1 3 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 4 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 4 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998003fa0 osd=1 3 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 1 4 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 4 2024-02-10T08:37:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 10 client.47446.objecter _send_command 4 2024-02-10T08:37:34.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.694+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd9980018b0 con 0x7fd998004110 2024-02-10T08:37:34.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd9980018b0 con 0x7fd998004110 2024-02-10T08:37:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 10 client.47446.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 4 2024-02-10T08:37:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 1 4 2024-02-10T08:37:34.701 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:34.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 5 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 5 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 2 5 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 4 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 10 client.47446.objecter _send_command 5 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd998003c70 con 0x7fd99800c170 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.696+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd99800c170 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.697+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd998003c70 con 0x7fd99800c170 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.697+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.697+0000 7fd993fff700 10 client.47446.objecter _finish_command 5 = system:0 2024-02-10T08:37:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.697+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 4 2024-02-10T08:37:34.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.697+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 2 5 2024-02-10T08:37:34.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 6 2024-02-10T08:37:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 6 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd9980035f0 osd=2 3 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 2 6 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 4 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 10 client.47446.objecter _send_command 6 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.711+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd998003de0 con 0x7fd99800c170 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd998003de0 con 0x7fd99800c170 2024-02-10T08:37:34.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd993fff700 10 client.47446.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 4 2024-02-10T08:37:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 2 6 2024-02-10T08:37:34.714 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 7 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 7 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 3 7 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 4 2024-02-10T08:37:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 10 client.47446.objecter _send_command 7 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.713+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd998019e80 con 0x7fd99800f600 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd99800f600 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd998019e80 con 0x7fd99800f600 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 10 client.47446.objecter _finish_command 7 = system:0 2024-02-10T08:37:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 4 2024-02-10T08:37:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.714+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 3 7 2024-02-10T08:37:34.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 8 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 8 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998003350 osd=3 3 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 3 8 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 4 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 10 client.47446.objecter _send_command 8 2024-02-10T08:37:34.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.729+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd99800eb60 con 0x7fd99800f600 2024-02-10T08:37:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd99800eb60 con 0x7fd99800f600 2024-02-10T08:37:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 10 client.47446.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 4 2024-02-10T08:37:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 3 8 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 9 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 9 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 4 9 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 4 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 10 client.47446.objecter _send_command 9 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd998016350 con 0x7fd99800ecd0 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.731+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd99800ecd0 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:34.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.732+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd998016350 con 0x7fd99800ecd0 2024-02-10T08:37:34.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.732+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:34.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.732+0000 7fd993fff700 10 client.47446.objecter _finish_command 9 = system:0 2024-02-10T08:37:34.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.732+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 4 2024-02-10T08:37:34.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.732+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 4 9 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 10 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 10 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd99800eb60 osd=4 3 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 4 10 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 4 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 10 client.47446.objecter _send_command 10 2024-02-10T08:37:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.747+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd998016350 con 0x7fd99800ecd0 2024-02-10T08:37:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd998016350 con 0x7fd99800ecd0 2024-02-10T08:37:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd993fff700 10 client.47446.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 4 2024-02-10T08:37:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 4 10 2024-02-10T08:37:34.751 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:34.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 11 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 11 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 5 11 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 4 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 10 client.47446.objecter _send_command 11 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.750+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd99801af00 con 0x7fd99801aaa0 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd99801aaa0 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd99801af00 con 0x7fd99801aaa0 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:34.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 10 client.47446.objecter _finish_command 11 = system:0 2024-02-10T08:37:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 4 2024-02-10T08:37:34.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.751+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 5 11 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 12 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 12 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998016350 osd=5 3 2024-02-10T08:37:34.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 5 12 2024-02-10T08:37:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 4 2024-02-10T08:37:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 10 client.47446.objecter _send_command 12 2024-02-10T08:37:34.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.766+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd99801af00 con 0x7fd99801aaa0 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.767+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd99801af00 con 0x7fd99801aaa0 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.767+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.767+0000 7fd993fff700 10 client.47446.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.767+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 4 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.767+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 5 12 2024-02-10T08:37:34.768 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 13 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 13 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 6 13 2024-02-10T08:37:34.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 4 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 10 client.47446.objecter _send_command 13 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd998026f60 con 0x7fd998023450 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.768+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd998023450 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.769+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd998026f60 con 0x7fd998023450 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.769+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.769+0000 7fd993fff700 10 client.47446.objecter _finish_command 13 = system:0 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.769+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 4 2024-02-10T08:37:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.769+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 6 13 2024-02-10T08:37:34.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 14 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 14 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd99801af00 osd=6 3 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 6 14 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 4 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 10 client.47446.objecter _send_command 14 2024-02-10T08:37:34.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.783+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd998026f60 con 0x7fd998023450 2024-02-10T08:37:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd998026f60 con 0x7fd998023450 2024-02-10T08:37:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd993fff700 10 client.47446.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 4 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 6 14 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 15 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 15 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 7 15 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 4 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 10 client.47446.objecter _send_command 15 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.785+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd99802b240 con 0x7fd9980275e0 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 10 client.47446.objecter ms_handle_connect 0x7fd9980275e0 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd99802b240 con 0x7fd9980275e0 2024-02-10T08:37:34.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 10 client.47446.objecter _finish_command 15 = system:0 2024-02-10T08:37:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 4 2024-02-10T08:37:34.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.786+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 7 15 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 10 client.47446.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign -1 16 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 20 client.47446.objecter _get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_remove -1 16 2024-02-10T08:37:34.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 20 client.47446.objecter get_session s=0x7fd998026f60 osd=7 3 2024-02-10T08:37:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 15 client.47446.objecter _session_command_op_assign 7 16 2024-02-10T08:37:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 4 2024-02-10T08:37:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 10 client.47446.objecter _send_command 16 2024-02-10T08:37:34.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.801+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd99802c9f0 con 0x7fd9980275e0 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.802+0000 7fd993fff700 1 -- 172.21.15.107:0/676496404 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd99802c9f0 con 0x7fd9980275e0 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.802+0000 7fd993fff700 10 client.47446.objecter ms_dispatch 0x7fd9ac001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.802+0000 7fd993fff700 10 client.47446.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.802+0000 7fd993fff700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 4 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.802+0000 7fd993fff700 15 client.47446.objecter _session_command_op_remove 7 16 2024-02-10T08:37:34.803 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:34.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 librados: watch_flush enter 2024-02-10T08:37:34.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 librados: watch_flush exit 2024-02-10T08:37:34.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.0 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6801/3321596972 conn(0x7fd998004530 legacy=0x7fd998002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980051a0 osd=0 1 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.1 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6805/133516126 conn(0x7fd998004110 legacy=0x7fd99800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003fa0 osd=1 1 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.2 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6809/3575881728 conn(0x7fd99800c170 legacy=0x7fd998013eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd9980035f0 osd=2 1 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.3 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6813/2871825691 conn(0x7fd99800f600 legacy=0x7fd998018760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998003350 osd=3 1 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.4 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.195:6800/2499671285 conn(0x7fd99800ecd0 legacy=0x7fd998019e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9ab7fe700 1 -- 172.21.15.107:0/676496404 reap_dead start 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99800eb60 osd=4 1 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.5 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.195:6804/4178321486 conn(0x7fd99801aaa0 legacy=0x7fd998021820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9ab7fe700 1 -- 172.21.15.107:0/676496404 reap_dead start 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998016350 osd=5 1 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.6 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.195:6808/1417479017 conn(0x7fd998023450 legacy=0x7fd998025840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd99801af00 osd=6 1 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter close_session for osd.7 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.195:6812/1934176928 conn(0x7fd9980275e0 legacy=0x7fd9980299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter put_session s=0x7fd998026f60 osd=7 1 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 20 client.47446.objecter shutdown clearing up homeless session... 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 10 client.47446.objecter successfully canceled tick 2024-02-10T08:37:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9ab7fe700 1 -- 172.21.15.107:0/676496404 reap_dead start 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd994063e10 legacy=0x7fd9940662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> v1:172.21.15.195:6789/0 conn(0x7fd9ac1007f0 legacy=0x7fd9ac0fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 shutdown_connections 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 >> 172.21.15.107:0/676496404 conn(0x7fd9ac0f3640 msgr2=0x7fd9ac0f64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 shutdown_connections 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 -- 172.21.15.107:0/676496404 wait complete. 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:34.803+0000 7fd9b19f7700 1 librados: shutdown 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.808 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:37:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:34.809 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:34.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:34.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:35.007 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:35.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:35.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:35.286 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:35.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:35.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:34.009693+0000 mon.b (mon.1) 222 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:34.467173+0000 mgr.y (mgr.14553) 204 : cluster [DBG] pgmap v216: 346 pgs: 64 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 56 KiB/s rd, 73 op/s 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:34.659228+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? v1:172.21.15.107:0/676496404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:34.947022+0000 mon.a (mon.0) 1227 : audit [INF] from='client.52679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:34.947215+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:34.952879+0000 mon.c (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.107:0/218760049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:34.953380+0000 mon.a (mon.0) 1229 : audit [INF] from='client.52685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.418 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:34.009693+0000 mon.b (mon.1) 222 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:34.467173+0000 mgr.y (mgr.14553) 204 : cluster [DBG] pgmap v216: 346 pgs: 64 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 56 KiB/s rd, 73 op/s 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:34.659228+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? v1:172.21.15.107:0/676496404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:34.947022+0000 mon.a (mon.0) 1227 : audit [INF] from='client.52679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:34.947215+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:34.952879+0000 mon.c (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.107:0/218760049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:34.953380+0000 mon.a (mon.0) 1229 : audit [INF] from='client.52685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:34.009693+0000 mon.b (mon.1) 222 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:34.467173+0000 mgr.y (mgr.14553) 204 : cluster [DBG] pgmap v216: 346 pgs: 64 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 56 KiB/s rd, 73 op/s 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:34.659228+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? v1:172.21.15.107:0/676496404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:34.947022+0000 mon.a (mon.0) 1227 : audit [INF] from='client.52679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112525-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:34.947215+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e161: 8 total, 8 up, 7 in 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:34.952879+0000 mon.c (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.107:0/218760049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:34.953380+0000 mon.a (mon.0) 1229 : audit [INF] from='client.52685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:35.521 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:35.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:35.560 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:35.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:35.649 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:35.781 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:35.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:35.813 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:35] "GET /metrics HTTP/1.1" 200 196442 "" "Prometheus/2.33.4" 2024-02-10T08:37:35.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:36.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:36.197 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:36.269 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:36.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:36.278 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:36.278 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:35.010482+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:35.445666+0000 osd.7 (osd.7) 79 : cluster [DBG] 95.2 scrub starts 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:35.449188+0000 osd.7 (osd.7) 80 : cluster [DBG] 95.2 scrub ok 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:35.948865+0000 mon.a (mon.0) 1230 : audit [INF] from='client.52685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:35.949031+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:35.950611+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.107:0/1942633859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:35.951195+0000 mon.a (mon.0) 1232 : audit [INF] from='client.52700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:36.011238+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:35.010482+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:35.445666+0000 osd.7 (osd.7) 79 : cluster [DBG] 95.2 scrub starts 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:35.449188+0000 osd.7 (osd.7) 80 : cluster [DBG] 95.2 scrub ok 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:35.948865+0000 mon.a (mon.0) 1230 : audit [INF] from='client.52685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:35.949031+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-10T08:37:36.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:35.950611+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.107:0/1942633859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:35.951195+0000 mon.a (mon.0) 1232 : audit [INF] from='client.52700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:36.011238+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:35.010482+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:35.445666+0000 osd.7 (osd.7) 79 : cluster [DBG] 95.2 scrub starts 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:35.449188+0000 osd.7 (osd.7) 80 : cluster [DBG] 95.2 scrub ok 2024-02-10T08:37:36.401 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:35.948865+0000 mon.a (mon.0) 1230 : audit [INF] from='client.52685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi107-112411-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:36.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:35.949031+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e162: 8 total, 8 up, 7 in 2024-02-10T08:37:36.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:35.950611+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.107:0/1942633859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:35.951195+0000 mon.a (mon.0) 1232 : audit [INF] from='client.52700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:36.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:36.011238+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:36.500 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:36.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:36.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:36.641 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:37:37.090 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:37.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:37.220 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.244 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:36.468359+0000 mgr.y (mgr.14553) 205 : cluster [DBG] pgmap v219: 378 pgs: 32 creating+peering, 346 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:36.495947+0000 osd.7 (osd.7) 81 : cluster [DBG] 95.7 scrub starts 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:36.497621+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.7 scrub ok 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:36.952893+0000 mon.a (mon.0) 1233 : audit [INF] from='client.52700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:36.953084+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-10T08:37:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.012054+0000 mon.b (mon.1) 226 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:37.327 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:37.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.321+0000 7f66cead3700 1 -- 172.21.15.107:0/2519071460 >> v1:172.21.15.107:6789/0 conn(0x7f66c8100840 legacy=0x7f66c8102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.321+0000 7f66cead3700 1 -- 172.21.15.107:0/2519071460 shutdown_connections 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.321+0000 7f66cead3700 1 -- 172.21.15.107:0/2519071460 >> 172.21.15.107:0/2519071460 conn(0x7f66c806c090 msgr2=0x7f66c806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.321+0000 7f66cead3700 1 -- 172.21.15.107:0/2519071460 shutdown_connections 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.321+0000 7f66cead3700 1 -- 172.21.15.107:0/2519071460 wait complete. 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 librados: starting msgr at 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 librados: starting objecter 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 Processor -- start 2024-02-10T08:37:37.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 -- start start 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 librados: setting wanted keys 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 librados: calling monclient init 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66c80faaa0 con 0x7f66c8104350 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66c8118850 con 0x7f66c8100840 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cead3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66c81199f0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cc86f700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f66c80fd9f0 0x7f66c80f9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52238/0 (socket says 172.21.15.107:52238) 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66cc86f700 1 -- 172.21.15.107:0/3269555164 learned_addr learned my addr 172.21.15.107:0/3269555164 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter ms_handle_connect 0x7f66c80fd9f0 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter ms_handle_connect 0x7f66c8104350 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter ms_handle_connect 0x7f66c8100840 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055327486 0 0) 0x7f66c81199f0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b80033f0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037619000 0 0) 0x7f66c80faaa0 con 0x7f66c8104350 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c81199f0 con 0x7f66c8104350 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807950711 0 0) 0x7f66c8118850 con 0x7f66c8100840 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.322+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c80faaa0 con 0x7f66c8100840 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839412423 0 0) 0x7f66b80033f0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66c8118850 con 0x7f66c80fd9f0 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66bc002990 con 0x7f66c80fd9f0 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 503154836 0 0) 0x7f66c81199f0 con 0x7f66c8104350 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b80033f0 con 0x7f66c8104350 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66c0001b70 con 0x7f66c8104350 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224666962 0 0) 0x7f66c80faaa0 con 0x7f66c8100840 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66c81199f0 con 0x7f66c8100840 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66b4002ca0 con 0x7f66c8100840 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2611699494 0 0) 0x7f66c8118850 con 0x7f66c80fd9f0 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 >> v1:172.21.15.195:6789/0 conn(0x7f66c8100840 legacy=0x7f66c81129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 >> v1:172.21.15.107:6789/0 conn(0x7f66c8104350 legacy=0x7f66c8116120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:37.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c811abd0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c81178a0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f66bc002d20 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66bc005c50 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66cead3700 10 client.52724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c8117d10 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.323+0000 7f66cead3700 1 librados: init done 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.324+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c805f580 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.324+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f66bc0016a0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(163..163 src has 50..163) v4 ==== 9264+0+0 (unknown 231134301 0 0) 0x7f66bc079360 con 0x7f66c80fd9f0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 10 client.52724.objecter ms_dispatch 0x7f66c8001550 osd_map(163..163 src has 50..163) v4 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 3 client.52724.objecter handle_osd_map got epochs [163,163] > 0 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 3 client.52724.objecter handle_osd_map decoding full epoch 163 2024-02-10T08:37:37.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 20 client.52724.objecter dump_active .. 0 homeless 2024-02-10T08:37:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.325+0000 7f66c5f76700 10 client.52724.objecter ms_handle_connect 0x7f66b8063c30 2024-02-10T08:37:37.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.329+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f66bc04c2a0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:36.468359+0000 mgr.y (mgr.14553) 205 : cluster [DBG] pgmap v219: 378 pgs: 32 creating+peering, 346 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:36.495947+0000 osd.7 (osd.7) 81 : cluster [DBG] 95.7 scrub starts 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:36.497621+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.7 scrub ok 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:36.952893+0000 mon.a (mon.0) 1233 : audit [INF] from='client.52700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:36.953084+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.012054+0000 mon.b (mon.1) 226 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:37.374 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:36.468359+0000 mgr.y (mgr.14553) 205 : cluster [DBG] pgmap v219: 378 pgs: 32 creating+peering, 346 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:37.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:36.495947+0000 osd.7 (osd.7) 81 : cluster [DBG] 95.7 scrub starts 2024-02-10T08:37:37.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:36.497621+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.7 scrub ok 2024-02-10T08:37:37.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:36.952893+0000 mon.a (mon.0) 1233 : audit [INF] from='client.52700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112403-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:37.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:36.953084+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e163: 8 total, 8 up, 7 in 2024-02-10T08:37:37.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.012054+0000 mon.b (mon.1) 226 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:37.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:37.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.507+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f66c8109fa0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:37:37.527 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.545 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:37.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:37.651 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:37:37.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:37:37.651 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-10T08:37:37.651 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-02-10T08:37:37.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-10T08:37:37.678 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:37.802 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:37.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:37.836 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:37.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.956+0000 7f66c5f76700 1 -- 172.21.15.107:0/3269555164 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v164) v1 ==== 91+0+0 (unknown 718092229 0 0) 0x7f66bc04a6b0 con 0x7f66c80fd9f0 2024-02-10T08:37:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.960+0000 7f66cead3700 10 librados: watch_flush enter 2024-02-10T08:37:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.960+0000 7f66cead3700 10 librados: watch_flush exit 2024-02-10T08:37:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.960+0000 7f66cead3700 20 client.52724.objecter shutdown clearing up homeless session... 2024-02-10T08:37:37.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.960+0000 7f66cead3700 10 client.52724.objecter successfully canceled tick 2024-02-10T08:37:37.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.966+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 >> v1:172.21.15.107:6800/2634842009 conn(0x7f66b8063c30 legacy=0x7f66b80660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:37.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.966+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 >> v1:172.21.15.107:6790/0 conn(0x7f66c80fd9f0 legacy=0x7f66c80f9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.970+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 shutdown_connections 2024-02-10T08:37:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.970+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 >> 172.21.15.107:0/3269555164 conn(0x7f66c806c090 msgr2=0x7f66c806c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.980+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 shutdown_connections 2024-02-10T08:37:37.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.980+0000 7f66cead3700 1 -- 172.21.15.107:0/3269555164 wait complete. 2024-02-10T08:37:37.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:37.982+0000 7f66cead3700 1 librados: shutdown 2024-02-10T08:37:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:37:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:37.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:38.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:37:38.121 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:38.140 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:37:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:38.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.508777+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.107:0/3269555164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.509142+0000 mon.a (mon.0) 1235 : audit [INF] from='client.52724 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:37.947738+0000 mon.a (mon.0) 1236 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.950942+0000 mon.a (mon.0) 1237 : audit [INF] from='client.52724 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:37.951023+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.969342+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.107:0/2182655262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:38.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:37.969817+0000 mon.a (mon.0) 1239 : audit [INF] from='client.52718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:38.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:38.285 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:38.521 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.508777+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.107:0/3269555164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.509142+0000 mon.a (mon.0) 1235 : audit [INF] from='client.52724 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:37.947738+0000 mon.a (mon.0) 1236 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.950942+0000 mon.a (mon.0) 1237 : audit [INF] from='client.52724 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:37.951023+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.969342+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.107:0/2182655262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:38.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:37.969817+0000 mon.a (mon.0) 1239 : audit [INF] from='client.52718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.508777+0000 mon.c (mon.2) 194 : audit [INF] from='client.? v1:172.21.15.107:0/3269555164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.509142+0000 mon.a (mon.0) 1235 : audit [INF] from='client.52724 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:37.947738+0000 mon.a (mon.0) 1236 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.950942+0000 mon.a (mon.0) 1237 : audit [INF] from='client.52724 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:37.951023+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e164: 8 total, 8 up, 7 in 2024-02-10T08:37:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.969342+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.107:0/2182655262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:38.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:37.969817+0000 mon.a (mon.0) 1239 : audit [INF] from='client.52718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:39.039 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:39.039 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:39.134 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:39.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:39.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:39.269 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:39.288 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:39.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:39.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.012838+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.013507+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.246237+0000 mon.a (mon.0) 1240 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.246786+0000 mon.c (mon.2) 197 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.250867+0000 mon.a (mon.0) 1241 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.251292+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.251512+0000 mon.a (mon.0) 1242 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.468777+0000 mgr.y (mgr.14553) 206 : cluster [DBG] pgmap v222: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.526744+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.527649+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.542878+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.b scrub starts 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.545774+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.b scrub ok 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.953221+0000 mon.a (mon.0) 1243 : audit [INF] from='client.52718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.953335+0000 mon.a (mon.0) 1244 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.953388+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-10T08:37:39.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.958354+0000 mon.c (mon.2) 199 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.958963+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:39.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:38.959923+0000 mon.a (mon.0) 1247 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:39.013612+0000 mon.b (mon.1) 228 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:39.419 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.012838+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.013507+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.246237+0000 mon.a (mon.0) 1240 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.246786+0000 mon.c (mon.2) 197 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.250867+0000 mon.a (mon.0) 1241 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.251292+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.251512+0000 mon.a (mon.0) 1242 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.468777+0000 mgr.y (mgr.14553) 206 : cluster [DBG] pgmap v222: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.526744+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.527649+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.542878+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.b scrub starts 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.545774+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.b scrub ok 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.953221+0000 mon.a (mon.0) 1243 : audit [INF] from='client.52718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.953335+0000 mon.a (mon.0) 1244 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.953388+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.958354+0000 mon.c (mon.2) 199 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.958963+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:38.959923+0000 mon.a (mon.0) 1247 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:39.013612+0000 mon.b (mon.1) 228 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.012838+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.013507+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.246237+0000 mon.a (mon.0) 1240 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.246786+0000 mon.c (mon.2) 197 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.250867+0000 mon.a (mon.0) 1241 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.251292+0000 mon.c (mon.2) 198 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.251512+0000 mon.a (mon.0) 1242 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.468777+0000 mgr.y (mgr.14553) 206 : cluster [DBG] pgmap v222: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:39.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.526744+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.527649+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.542878+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.b scrub starts 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.545774+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.b scrub ok 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.953221+0000 mon.a (mon.0) 1243 : audit [INF] from='client.52718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.953335+0000 mon.a (mon.0) 1244 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.953388+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e165: 8 total, 8 up, 7 in 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.958354+0000 mon.c (mon.2) 199 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.958963+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:38.959923+0000 mon.a (mon.0) 1247 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:39.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:39.013612+0000 mon.b (mon.1) 228 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:39.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:37:39.768 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:39.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:39.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:39.901 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:40.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:40.015 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:40.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:40.132 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:40.257 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:40.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:40.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:40.388 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:40.474 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:40.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:40.572 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:40.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.671+0000 7f559a42d700 1 -- 172.21.15.107:0/152830385 >> v1:172.21.15.107:6790/0 conn(0x7f5594103250 legacy=0x7f55941056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- 172.21.15.107:0/152830385 shutdown_connections 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- 172.21.15.107:0/152830385 >> 172.21.15.107:0/152830385 conn(0x7f55940f2550 msgr2=0x7f55940f4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- 172.21.15.107:0/152830385 shutdown_connections 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- 172.21.15.107:0/152830385 wait complete. 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 librados: starting msgr at 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 librados: starting objecter 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 Processor -- start 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- start start 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 librados: setting wanted keys 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 librados: calling monclient init 2024-02-10T08:37:40.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55940fbf10 con 0x7f55940ff710 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.672+0000 7f559a42d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55940fc0e0 con 0x7f5594103250 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f559a42d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55940fc2d0 con 0x7f55940fc910 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f5593fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f55940fc910 0x7f55940fb700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41392/0 (socket says 172.21.15.107:41392) 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f5593fff700 1 -- 172.21.15.107:0/3266514546 learned_addr learned my addr 172.21.15.107:0/3266514546 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter ms_handle_connect 0x7f55940fc910 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter ms_handle_connect 0x7f55940ff710 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter ms_handle_connect 0x7f5594103250 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342292708 0 0) 0x7f55940fc2d0 con 0x7f55940fc910 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c0038e0 con 0x7f55940fc910 2024-02-10T08:37:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170978157 0 0) 0x7f55940fbf10 con 0x7f55940ff710 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55940fc2d0 con 0x7f55940ff710 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962533956 0 0) 0x7f55940fc0e0 con 0x7f5594103250 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55940fbf10 con 0x7f5594103250 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84917094 0 0) 0x7f557c0038e0 con 0x7f55940fc910 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55940fc0e0 con 0x7f55940fc910 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5584001940 con 0x7f55940fc910 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459503739 0 0) 0x7f55940fc2d0 con 0x7f55940ff710 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f557c0038e0 con 0x7f55940ff710 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5588001620 con 0x7f55940ff710 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395911692 0 0) 0x7f55940fbf10 con 0x7f5594103250 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55940fc2d0 con 0x7f5594103250 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f558c002180 con 0x7f5594103250 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3776088336 0 0) 0x7f55940fc0e0 con 0x7f55940fc910 2024-02-10T08:37:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.195:6789/0 conn(0x7f5594103250 legacy=0x7f55941183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6789/0 conn(0x7f55940ff710 legacy=0x7f5594114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559411cb30 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5594119b60 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f559a42d700 10 client.52751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f559411a050 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f559a42d700 1 librados: init done 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5584002460 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.673+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5584005ae0 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.674+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5580004f00 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5584002c30 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(166..166 src has 50..166) v4 ==== 9265+0+0 (unknown 2048166564 0 0) 0x7f55840790f0 con 0x7f55940fc910 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 osd_map(166..166 src has 50..166) v4 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 3 client.52751.objecter handle_osd_map got epochs [166,166] > 0 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 3 client.52751.objecter handle_osd_map decoding full epoch 166 2024-02-10T08:37:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 20 client.52751.objecter dump_active .. 0 homeless 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v166) v1 ==== 54+0+16 (unknown 2068624716 0 2420000192) 0x7f558407b500 con 0x7f55940fc910 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f557c063cb0 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 10 client.52751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 1 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 1 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 0 1 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 4 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 10 client.52751.objecter _send_command 1 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.675+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580004950 con 0x7f5580004530 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f5580004530 2024-02-10T08:37:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580004950 con 0x7f5580004530 2024-02-10T08:37:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 10 client.52751.objecter _finish_command 1 = system:0 2024-02-10T08:37:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 4 2024-02-10T08:37:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.676+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 0 1 2024-02-10T08:37:40.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.677698+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.a scrub starts 2024-02-10T08:37:40.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:38.680605+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.a scrub ok 2024-02-10T08:37:40.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.540475+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:37:40.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.542433+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:37:40.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.544260+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.a scrub starts 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.547732+0000 osd.6 (osd.6) 74 : cluster [DBG] 4.a scrub ok 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:39.956015+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.014557+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.201282+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.217376+0000 mon.a (mon.0) 1250 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.677698+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.a scrub starts 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:38.680605+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.a scrub ok 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.540475+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.542433+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.544260+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.a scrub starts 2024-02-10T08:37:40.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.547732+0000 osd.6 (osd.6) 74 : cluster [DBG] 4.a scrub ok 2024-02-10T08:37:40.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:39.956015+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:40.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.014557+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:40.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.201282+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-10T08:37:40.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.217376+0000 mon.a (mon.0) 1250 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 10 client.52751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 2 2024-02-10T08:37:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 2 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f55800051a0 osd=0 3 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 0 2 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 4 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 10 client.52751.objecter _send_command 2 2024-02-10T08:37:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.691+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5580004d30 con 0x7f5580004530 2024-02-10T08:37:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5580004d30 con 0x7f5580004530 2024-02-10T08:37:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 10 client.52751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 4 2024-02-10T08:37:40.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 0 2 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 10 client.52751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 3 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 3 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 1 3 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 4 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 10 client.52751.objecter _send_command 3 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f558000c000 con 0x7f5580004150 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.693+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f5580004150 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.694+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f558000c000 con 0x7f5580004150 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.694+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.694+0000 7f55917fa700 10 client.52751.objecter _finish_command 3 = system:0 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.694+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 4 2024-02-10T08:37:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.694+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 1 3 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 10 client.52751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 4 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 4 2024-02-10T08:37:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003fe0 osd=1 3 2024-02-10T08:37:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 1 4 2024-02-10T08:37:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 4 2024-02-10T08:37:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 10 client.52751.objecter _send_command 4 2024-02-10T08:37:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.709+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f55800018b0 con 0x7f5580004150 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f55800018b0 con 0x7f5580004150 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f55917fa700 10 client.52751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 4 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 1 4 2024-02-10T08:37:40.711 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.710+0000 7f559a42d700 10 client.52751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 5 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 5 2024-02-10T08:37:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 2 5 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 4 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 10 client.52751.objecter _send_command 5 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580003e70 con 0x7f558000d2f0 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f558000d2f0 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580003e70 con 0x7f558000d2f0 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 10 client.52751.objecter _finish_command 5 = system:0 2024-02-10T08:37:40.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 4 2024-02-10T08:37:40.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.711+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 2 5 2024-02-10T08:37:40.719 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:40.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 10 client.52751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 6 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 6 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003630 osd=2 3 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 2 6 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 4 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 10 client.52751.objecter _send_command 6 2024-02-10T08:37:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.726+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f558000e730 con 0x7f558000d2f0 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f558000e730 con 0x7f558000d2f0 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 10 client.52751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 4 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 2 6 2024-02-10T08:37:40.728 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:40.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 10 client.52751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 7 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 7 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 3 7 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 4 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 10 client.52751.objecter _send_command 7 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580015ac0 con 0x7f5580011e80 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.728+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f5580011e80 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.729+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580015ac0 con 0x7f5580011e80 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.729+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:40.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.729+0000 7f55917fa700 10 client.52751.objecter _finish_command 7 = system:0 2024-02-10T08:37:40.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.729+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 4 2024-02-10T08:37:40.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.729+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 3 7 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 10 client.52751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 8 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 8 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580003410 osd=3 3 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 3 8 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 4 2024-02-10T08:37:40.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 10 client.52751.objecter _send_command 8 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.743+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f558000e1a0 con 0x7f5580011e80 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f558000e1a0 con 0x7f5580011e80 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 10 client.52751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 4 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 3 8 2024-02-10T08:37:40.745 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:40.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 10 client.52751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 9 2024-02-10T08:37:40.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 9 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 4 9 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 4 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 10 client.52751.objecter _send_command 9 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f558000e4a0 con 0x7f558000ce50 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.745+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f558000ce50 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.746+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f558000e4a0 con 0x7f558000ce50 2024-02-10T08:37:40.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.746+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:40.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.746+0000 7f55917fa700 10 client.52751.objecter _finish_command 9 = system:0 2024-02-10T08:37:40.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.746+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 4 2024-02-10T08:37:40.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.746+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 4 9 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 10 client.52751.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 10 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 10 2024-02-10T08:37:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f558000e1a0 osd=4 3 2024-02-10T08:37:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 4 10 2024-02-10T08:37:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 4 2024-02-10T08:37:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 10 client.52751.objecter _send_command 10 2024-02-10T08:37:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.761+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f558000e4a0 con 0x7f558000ce50 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.762+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f558000e4a0 con 0x7f558000ce50 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.762+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.762+0000 7f55917fa700 10 client.52751.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.762+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 4 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.762+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 4 10 2024-02-10T08:37:40.763 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 10 client.52751.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 11 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 11 2024-02-10T08:37:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 5 11 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 4 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 10 client.52751.objecter _send_command 11 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580017ff0 con 0x7f5580017a00 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.763+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f5580017a00 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.764+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580017ff0 con 0x7f5580017a00 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.764+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.764+0000 7f55917fa700 10 client.52751.objecter _finish_command 11 = system:0 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.764+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 4 2024-02-10T08:37:40.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.764+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 5 11 2024-02-10T08:37:40.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 10 client.52751.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 12 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 12 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f558000e4a0 osd=5 3 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 5 12 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 4 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 10 client.52751.objecter _send_command 12 2024-02-10T08:37:40.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.779+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5580017ff0 con 0x7f5580017a00 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5580017ff0 con 0x7f5580017a00 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 10 client.52751.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 4 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 5 12 2024-02-10T08:37:40.781 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 10 client.52751.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 13 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 13 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 6 13 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 4 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 10 client.52751.objecter _send_command 13 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580022c00 con 0x7f558001f0c0 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.781+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f558001f0c0 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.782+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580022c00 con 0x7f558001f0c0 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.782+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.782+0000 7f55917fa700 10 client.52751.objecter _finish_command 13 = system:0 2024-02-10T08:37:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.782+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 4 2024-02-10T08:37:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.782+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 6 13 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.677698+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.a scrub starts 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:38.680605+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.a scrub ok 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.540475+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.542433+0000 osd.3 (osd.3) 100 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.544260+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.a scrub starts 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.547732+0000 osd.6 (osd.6) 74 : cluster [DBG] 4.a scrub ok 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:39.956015+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.107:0/2447688014' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi107-112403-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.014557+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.201282+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e166: 8 total, 8 up, 7 in 2024-02-10T08:37:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.217376+0000 mon.a (mon.0) 1250 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:37:40.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 10 client.52751.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 14 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 14 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f5580017ff0 osd=6 3 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 6 14 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 4 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 10 client.52751.objecter _send_command 14 2024-02-10T08:37:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.796+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f55800243b0 con 0x7f558001f0c0 2024-02-10T08:37:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f55800243b0 con 0x7f558001f0c0 2024-02-10T08:37:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f55917fa700 10 client.52751.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 4 2024-02-10T08:37:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 6 14 2024-02-10T08:37:40.799 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 10 client.52751.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 15 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 15 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 7 15 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 4 2024-02-10T08:37:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 10 client.52751.objecter _send_command 15 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.798+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5580024fe0 con 0x7f5580024a30 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.799+0000 7f55917fa700 10 client.52751.objecter ms_handle_connect 0x7f5580024a30 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.800+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5580024fe0 con 0x7f5580024a30 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.800+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.800+0000 7f55917fa700 10 client.52751.objecter _finish_command 15 = system:0 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.800+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 4 2024-02-10T08:37:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.800+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 7 15 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 10 client.52751.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign -1 16 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 20 client.52751.objecter _get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 15 client.52751.objecter _session_command_op_remove -1 16 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 20 client.52751.objecter get_session s=0x7f55800243b0 osd=7 3 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 15 client.52751.objecter _session_command_op_assign 7 16 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 4 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 10 client.52751.objecter _send_command 16 2024-02-10T08:37:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.814+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5580024fe0 con 0x7f5580024a30 2024-02-10T08:37:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55917fa700 1 -- 172.21.15.107:0/3266514546 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5580024fe0 con 0x7f5580024a30 2024-02-10T08:37:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55917fa700 10 client.52751.objecter ms_dispatch 0x7f5594001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:37:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55917fa700 10 client.52751.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55917fa700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 4 2024-02-10T08:37:40.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55917fa700 15 client.52751.objecter _session_command_op_remove 7 16 2024-02-10T08:37:40.817 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 librados: watch_flush enter 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 librados: watch_flush exit 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.0 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6801/3321596972 conn(0x7f5580004530 legacy=0x7f5580002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800051a0 osd=0 1 2024-02-10T08:37:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.1 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6805/133516126 conn(0x7f5580004150 legacy=0x7f558000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003fe0 osd=1 1 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.2 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6809/3575881728 conn(0x7f558000d2f0 legacy=0x7f558000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003630 osd=2 1 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.3 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6813/2871825691 conn(0x7f5580011e80 legacy=0x7f5580014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f55989ca700 1 -- 172.21.15.107:0/3266514546 reap_dead start 2024-02-10T08:37:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580003410 osd=3 1 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.4 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.195:6800/2499671285 conn(0x7f558000ce50 legacy=0x7f5580015c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e1a0 osd=4 1 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.5 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.195:6804/4178321486 conn(0x7f5580017a00 legacy=0x7f558001d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f558000e4a0 osd=5 1 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.6 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.195:6808/1417479017 conn(0x7f558001f0c0 legacy=0x7f55800214b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f5580017ff0 osd=6 1 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter close_session for osd.7 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.195:6812/1934176928 conn(0x7f5580024a30 legacy=0x7f5580022c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter put_session s=0x7f55800243b0 osd=7 1 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 20 client.52751.objecter shutdown clearing up homeless session... 2024-02-10T08:37:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.816+0000 7f559a42d700 10 client.52751.objecter successfully canceled tick 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f55989ca700 1 -- 172.21.15.107:0/3266514546 reap_dead start 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6800/2634842009 conn(0x7f557c063cb0 legacy=0x7f557c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> v1:172.21.15.107:6790/0 conn(0x7f55940fc910 legacy=0x7f55940fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 shutdown_connections 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 >> 172.21.15.107:0/3266514546 conn(0x7f55940f2550 msgr2=0x7f55940f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 shutdown_connections 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 -- 172.21.15.107:0/3266514546 wait complete. 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:40.817+0000 7f559a42d700 1 librados: shutdown 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:37:40.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:40.851 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:41.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:37:41.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:41.602 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:41.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:41.669 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:41.669 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:37:41.669 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:37:41.669 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:37:41.669 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.570318+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:39.573221+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.469169+0000 mgr.y (mgr.14553) 207 : cluster [DBG] pgmap v225: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.518736+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.9 scrub starts 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.521733+0000 osd.4 (osd.4) 87 : cluster [DBG] 95.9 scrub ok 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.591234+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.592925+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.675908+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.107:0/3266514546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:41.690 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.781807+0000 mon.b (mon.1) 230 : audit [DBG] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.782325+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.782638+0000 mon.a (mon.0) 1251 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.924318+0000 mon.a (mon.0) 1252 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.924441+0000 mon.a (mon.0) 1253 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.924498+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.925857+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.947521+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2"}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.950141+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:40.950467+0000 mon.a (mon.0) 1256 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.691 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.015337+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:41.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:41.738 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:41.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.570318+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:37:41.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:39.573221+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.469169+0000 mgr.y (mgr.14553) 207 : cluster [DBG] pgmap v225: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.518736+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.9 scrub starts 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.521733+0000 osd.4 (osd.4) 87 : cluster [DBG] 95.9 scrub ok 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.591234+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.592925+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.675908+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.107:0/3266514546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.781807+0000 mon.b (mon.1) 230 : audit [DBG] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.782325+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.782638+0000 mon.a (mon.0) 1251 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.924318+0000 mon.a (mon.0) 1252 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.924441+0000 mon.a (mon.0) 1253 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.924498+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.925857+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.947521+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2"}]: dispatch 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.950141+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:40.950467+0000 mon.a (mon.0) 1256 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.015337+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.570318+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:39.573221+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.469169+0000 mgr.y (mgr.14553) 207 : cluster [DBG] pgmap v225: 314 pgs: 32 unknown, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.518736+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.9 scrub starts 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.521733+0000 osd.4 (osd.4) 87 : cluster [DBG] 95.9 scrub ok 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.591234+0000 osd.6 (osd.6) 75 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:37:41.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.592925+0000 osd.6 (osd.6) 76 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.675908+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.107:0/3266514546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.781807+0000 mon.b (mon.1) 230 : audit [DBG] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.782325+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.782638+0000 mon.a (mon.0) 1251 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.924318+0000 mon.a (mon.0) 1252 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi107-112525-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.924441+0000 mon.a (mon.0) 1253 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.924498+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.925857+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.947521+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2"}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.950141+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:40.950467+0000 mon.a (mon.0) 1256 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:41.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.015337+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:41.937 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:41.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.962+0000 7f664c50c700 1 -- 172.21.15.107:0/2172462882 >> v1:172.21.15.107:6789/0 conn(0x7f664410ba40 legacy=0x7f664410dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 -- 172.21.15.107:0/2172462882 shutdown_connections 2024-02-10T08:37:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 -- 172.21.15.107:0/2172462882 >> 172.21.15.107:0/2172462882 conn(0x7f6644063bc0 msgr2=0x7f66440672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 -- 172.21.15.107:0/2172462882 shutdown_connections 2024-02-10T08:37:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 -- 172.21.15.107:0/2172462882 wait complete. 2024-02-10T08:37:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 librados: starting msgr at 2024-02-10T08:37:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.963+0000 7f664c50c700 1 librados: starting objecter 2024-02-10T08:37:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 Processor -- start 2024-02-10T08:37:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 -- start start 2024-02-10T08:37:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 librados: setting wanted keys 2024-02-10T08:37:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 librados: calling monclient init 2024-02-10T08:37:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6644117290 con 0x7f6644103ff0 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66441273f0 con 0x7f664410ba40 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664c50c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66441285d0 con 0x7f6644107f30 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664a2a8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6644103ff0 0x7f66440ffdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58938/0 (socket says 172.21.15.107:58938) 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f664a2a8700 1 -- 172.21.15.107:0/913362717 learned_addr learned my addr 172.21.15.107:0/913362717 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 10 client.?.objecter ms_handle_connect 0x7f6644103ff0 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 10 client.?.objecter ms_handle_connect 0x7f664410ba40 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564511737 0 0) 0x7f66441273f0 con 0x7f664410ba40 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66300033f0 con 0x7f664410ba40 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223544858 0 0) 0x7f6644117290 con 0x7f6644103ff0 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.964+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66441273f0 con 0x7f6644103ff0 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 10 client.?.objecter ms_handle_connect 0x7f6644107f30 2024-02-10T08:37:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421500374 0 0) 0x7f66300033f0 con 0x7f664410ba40 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6644117290 con 0x7f664410ba40 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f663c002320 con 0x7f664410ba40 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152437590 0 0) 0x7f66441285d0 con 0x7f6644107f30 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66300033f0 con 0x7f6644107f30 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040773802 0 0) 0x7f66441273f0 con 0x7f6644103ff0 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66441285d0 con 0x7f6644103ff0 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6638001900 con 0x7f6644103ff0 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4225691796 0 0) 0x7f6644117290 con 0x7f664410ba40 2024-02-10T08:37:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 >> v1:172.21.15.107:6790/0 conn(0x7f6644107f30 legacy=0x7f6644100500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 >> v1:172.21.15.107:6789/0 conn(0x7f6644103ff0 legacy=0x7f66440ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66441297d0 con 0x7f664410ba40 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66441275e0 con 0x7f664410ba40 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f664c50c700 10 client.47509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6644127a50 con 0x7f664410ba40 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f664c50c700 1 librados: init done 2024-02-10T08:37:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f663c0024c0 con 0x7f664410ba40 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.965+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f663c003a30 con 0x7f664410ba40 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.966+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6628004f20 con 0x7f664410ba40 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.967+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f663c002e50 con 0x7f664410ba40 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.967+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(168..168 src has 50..168) v4 ==== 10099+0+0 (unknown 2989007567 0 0) 0x7f663c0772a0 con 0x7f664410ba40 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.967+0000 7f66377fe700 10 client.47509.objecter ms_dispatch 0x7f6644001550 osd_map(168..168 src has 50..168) v4 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.967+0000 7f66377fe700 3 client.47509.objecter handle_osd_map got epochs [168,168] > 0 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.967+0000 7f66377fe700 3 client.47509.objecter handle_osd_map decoding full epoch 168 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.968+0000 7f66377fe700 20 client.47509.objecter dump_active .. 0 homeless 2024-02-10T08:37:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.968+0000 7f66377fe700 10 client.47509.objecter ms_handle_connect 0x7f6630063cf0 2024-02-10T08:37:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:41.972+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f663c049ea0 con 0x7f664410ba40 2024-02-10T08:37:42.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.156+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f66280059e0 con 0x7f664410ba40 2024-02-10T08:37:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:42.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:42.317 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:42.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:42.401 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:42.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:37:42.649 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:42.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.893113+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.b scrub starts 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:40.896689+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.b scrub ok 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.219897+0000 mgr.y (mgr.14553) 208 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.600752+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.604240+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.749430+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.752898+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.926984+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.927161+0000 mon.a (mon.0) 1258 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.927251+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.932127+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.941978+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.943986+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:41.946254+0000 mon.a (mon.0) 1261 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.016180+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.157423+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.107:0/913362717' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.157815+0000 mon.a (mon.0) 1262 : audit [INF] from='client.47509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.893113+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.b scrub starts 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:40.896689+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.b scrub ok 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.219897+0000 mgr.y (mgr.14553) 208 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.600752+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:37:42.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.604240+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.749430+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.752898+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.926984+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.927161+0000 mon.a (mon.0) 1258 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.927251+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.932127+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.941978+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.943986+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:41.946254+0000 mon.a (mon.0) 1261 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.016180+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.157423+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.107:0/913362717' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.157815+0000 mon.a (mon.0) 1262 : audit [INF] from='client.47509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:42.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:42.712 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:42.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.893113+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.b scrub starts 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:40.896689+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.b scrub ok 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.219897+0000 mgr.y (mgr.14553) 208 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.600752+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.604240+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.749430+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.752898+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:37:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.926984+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.107:0/1908017200' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi107-112411-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.927161+0000 mon.a (mon.0) 1258 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi107-112480-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.927251+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.932127+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.941978+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.943986+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:41.946254+0000 mon.a (mon.0) 1261 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.016180+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.157423+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.107:0/913362717' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.157815+0000 mon.a (mon.0) 1262 : audit [INF] from='client.47509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:37:42.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.937+0000 7f66377fe700 1 -- 172.21.15.107:0/913362717 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v169) v1 ==== 133+0+0 (unknown 4119418035 0 0) 0x7f663c048310 con 0x7f664410ba40 2024-02-10T08:37:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.940+0000 7f664c50c700 10 librados: watch_flush enter 2024-02-10T08:37:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.943+0000 7f664c50c700 10 librados: watch_flush exit 2024-02-10T08:37:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.943+0000 7f664c50c700 20 client.47509.objecter shutdown clearing up homeless session... 2024-02-10T08:37:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.943+0000 7f664c50c700 10 client.47509.objecter successfully canceled tick 2024-02-10T08:37:42.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.944+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6630063cf0 legacy=0x7f66300661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:42.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.944+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 >> v1:172.21.15.195:6789/0 conn(0x7f664410ba40 legacy=0x7f6644124cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.945+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 shutdown_connections 2024-02-10T08:37:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.945+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 >> 172.21.15.107:0/913362717 conn(0x7f6644063bc0 msgr2=0x7f664406a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:42.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.951+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 shutdown_connections 2024-02-10T08:37:42.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.955+0000 7f664c50c700 1 -- 172.21.15.107:0/913362717 wait complete. 2024-02-10T08:37:42.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:42.959+0000 7f664c50c700 1 librados: shutdown 2024-02-10T08:37:42.961 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-10T08:37:43.365 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:43.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:43.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.595660+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.598538+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.942395+0000 osd.2 (osd.2) 63 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:41.943884+0000 osd.2 (osd.2) 64 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.431606+0000 osd.2 (osd.2) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.432374+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.469787+0000 mgr.y (mgr.14553) 209 : cluster [DBG] pgmap v228: 354 pgs: 32 unknown, 40 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.524149+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.525042+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.563833+0000 osd.6 (osd.6) 79 : cluster [DBG] 8.1a scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.567440+0000 osd.6 (osd.6) 80 : cluster [DBG] 8.1a scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.775379+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.d scrub starts 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.778282+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.d scrub ok 2024-02-10T08:37:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.934349+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.934480+0000 mon.a (mon.0) 1264 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.934579+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.934629+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.959984+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:42.965188+0000 mon.a (mon.0) 1267 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.017141+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.595660+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.598538+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.942395+0000 osd.2 (osd.2) 63 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:37:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:41.943884+0000 osd.2 (osd.2) 64 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.431606+0000 osd.2 (osd.2) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.432374+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.469787+0000 mgr.y (mgr.14553) 209 : cluster [DBG] pgmap v228: 354 pgs: 32 unknown, 40 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.524149+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.525042+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.563833+0000 osd.6 (osd.6) 79 : cluster [DBG] 8.1a scrub starts 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.567440+0000 osd.6 (osd.6) 80 : cluster [DBG] 8.1a scrub ok 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.775379+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.d scrub starts 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.778282+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.d scrub ok 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.934349+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.934480+0000 mon.a (mon.0) 1264 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.934579+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.934629+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.959984+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:42.965188+0000 mon.a (mon.0) 1267 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.017141+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.595660+0000 osd.5 (osd.5) 89 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.598538+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.942395+0000 osd.2 (osd.2) 63 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:41.943884+0000 osd.2 (osd.2) 64 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.431606+0000 osd.2 (osd.2) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.432374+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.469787+0000 mgr.y (mgr.14553) 209 : cluster [DBG] pgmap v228: 354 pgs: 32 unknown, 40 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.524149+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.525042+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.563833+0000 osd.6 (osd.6) 79 : cluster [DBG] 8.1a scrub starts 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.567440+0000 osd.6 (osd.6) 80 : cluster [DBG] 8.1a scrub ok 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.775379+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.d scrub starts 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.778282+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.d scrub ok 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.934349+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.107:0/4005123178' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi107-112403-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.934480+0000 mon.a (mon.0) 1264 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.934579+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.934629+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.959984+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:42.965188+0000 mon.a (mon.0) 1267 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T08:37:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.017141+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:43.846 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:43.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:43.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:43.989 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:44.237 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:44.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:44.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:37:44.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:44.370 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:44.467 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:44.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:33:22.111789+0000", 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 262.36856108500001, 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013340619, 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111789+0000", 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111789+0000", 2024-02-10T08:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111786+0000", 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8430000000000005e-06 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111796+0000", 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7703999999999999e-05 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111813+0000", 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.234e-06 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.111823+0000", 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022998440000000001 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114122+0000", 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3599999999999996e-06 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114131+0000", 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006010479 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.120141+0000", 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3954000000000005e-05 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.120205+0000", 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012155900000000001 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.120327+0000", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012793679999999999 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.121606+0000", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019692210000000002 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.123575+0000", 2024-02-10T08:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014832339999999999 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.125059+0000", 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7920999999999998e-05 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.125096+0000", 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2777000000000001e-05 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.125129+0000", 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013340619 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34379.0:53 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:33:22.114650+0000", 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 262.36569965500001, 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371145, 2024-02-10T08:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34379", 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114650+0000", 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114650+0000", 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114648+0000", 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6499999999999996e-06 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114655+0000", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0679999999999999e-05 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114666+0000", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.618e-06 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114673+0000", 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2568999999999998e-05 2024-02-10T08:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114715+0000", 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3380000000000004e-06 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.114721+0000", 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004953542 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.119674+0000", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012466300000000001 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.119799+0000", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014907500000000001 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.119948+0000", 2024-02-10T08:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001651781 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.121600+0000", 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001940946 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.123541+0000", 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013865469999999999 2024-02-10T08:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.124927+0000", 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2889000000000001e-05 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.124980+0000", 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1066999999999998e-05 2024-02-10T08:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.125021+0000", 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371145 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34379.0:54 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94862968472960] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:33:22.127284+0000", 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 262.35306597300001, 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018104009999999999, 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34379", 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127284+0000", 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127284+0000", 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127282+0000", 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9299999999999997e-06 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127289+0000", 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5599999999999999e-06 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127298+0000", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9549999999999997e-06 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127304+0000", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5604e-05 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127340+0000", 2024-02-10T08:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1737999999999997e-05 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127401+0000", 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109483 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127511+0000", 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028287100000000002 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.127794+0000", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027676099999999999 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.128070+0000", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00090530099999999996 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.128976+0000", 2024-02-10T08:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1365e-05 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.129027+0000", 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6958000000000005e-05 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:33:22.129094+0000", 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018104009999999999 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(4.e REQUEST e91)", 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:11.724870+0000", 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 92.755479957999995, 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018240750000000001, 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724870+0000", 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724870+0000", 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724870+0000", 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0989999999999999e-06 2024-02-10T08:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724873+0000", 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1936000000000001e-05 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724885+0000", 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1820000000000006e-06 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724893+0000", 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8389e-05 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724941+0000", 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9280000000000005e-06 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.724951+0000", 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001694252 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.726645+0000", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2318e-05 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.726658+0000", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6201000000000001e-05 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:11.726694+0000", 2024-02-10T08:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018240750000000001 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35273.0:1 58.1s0 58:8fcb97c6:test-rados-api-smithi107-112553-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e116)", 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 54.088684800000003, 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.58733886400000002, 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35273", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2054498539", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391663+0000", 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4849999999999997e-06 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391670+0000", 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4370000000000007e-06 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391678+0000", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9359999999999998e-06 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391683+0000", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6999999999999997e-05 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391730+0000", 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7719999999999999e-06 2024-02-10T08:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391733+0000", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.56671075800000004 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.958444+0000", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014385299999999999 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.958587+0000", 2024-02-10T08:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043469100000000002 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.959022+0000", 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043462299999999998 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.959457+0000", 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019472974000000001 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.978930+0000", 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4040000000000003e-05 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979004+0000", 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.58733886400000002 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35273.0:2 58.1s0 58:8fcb97c6:test-rados-api-smithi107-112553-13::foo:head [watch watch cookie 94609606614144] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 53.496625049999999, 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014181251000000001, 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35273", 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2054498539", 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983723+0000", 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2670000000000003e-06 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983729+0000", 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7109999999999995e-06 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983737+0000", 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0190000000000003e-06 2024-02-10T08:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983743+0000", 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039794899999999998 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984141+0000", 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4467000000000003e-05 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984225+0000", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039063500000000002 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984616+0000", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028579099999999997 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984902+0000", 2024-02-10T08:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012934038 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.997836+0000", 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0287e-05 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.997906+0000", 2024-02-10T08:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014181251000000001 2024-02-10T08:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35273.0:5 58.1s0 58:8fcb97c6:test-rados-api-smithi107-112553-13::foo:head [watch unwatch cookie 94609606614144] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.001275+0000", 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 53.479074759, 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039198669999999996, 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35273", 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2054498539", 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 5 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001275+0000", 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001275+0000", 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001274+0000", 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2430000000000001e-06 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001277+0000", 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8260000000000004e-06 2024-02-10T08:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001284+0000", 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9899999999999999e-06 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001288+0000", 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6812999999999999e-05 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001315+0000", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9712000000000001e-05 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001365+0000", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025568900000000002 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001620+0000", 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019743899999999999 2024-02-10T08:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.001818+0000", 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033084770000000002 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.005126+0000", 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8460999999999994e-05 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.005195+0000", 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039198669999999996 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:5 57.1a e118/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head v 118'3, mlcod=0'0)", 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.959304+0000", 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.521045852, 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073123809999999997, 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959304+0000", 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959304+0000", 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959303+0000", 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9857999999999999e-05 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959323+0000", 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0489999999999993e-06 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959332+0000", 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3629999999999996e-06 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959340+0000", 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001656902 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.960996+0000", 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3317000000000001e-05 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.961020+0000", 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055585820000000003 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.966578+0000", 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7826e-05 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.966616+0000", 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073123809999999997 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3 60.f 60:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.453928718, 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026465336999999999, 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026419+0000", 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6020000000000006e-06 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026428+0000", 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6636000000000001e-05 2024-02-10T08:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026445+0000", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6056e-05 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026461+0000", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026510+0000", 2024-02-10T08:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013219600000000001 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026642+0000", 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014158500000000001 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026784+0000", 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000910372 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.027694+0000", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034724220000000002 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.031166+0000", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021647165999999999 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052814+0000", 2024-02-10T08:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1364e-05 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052845+0000", 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1374000000000001e-05 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052886+0000", 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026465336999999999 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.7.0:168 57.1a e120/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:3 v 120'5, mlcod=117'1)", 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:54.038618+0000", 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.441731502000003, 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036767729999999999, 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038618+0000", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038618+0000", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038618+0000", 2024-02-10T08:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4409999999999998e-06 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038624+0000", 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2079000000000001e-05 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038636+0000", 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3950000000000003e-06 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038644+0000", 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000334624 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.038978+0000", 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2345000000000003e-05 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.039011+0000", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032562670000000002 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.042267+0000", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8007000000000001e-05 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:54.042295+0000", 2024-02-10T08:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036767729999999999 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:10 57.1a e121/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head v 121'6, mlcod=118'3)", 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.437047243999999, 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012434328, 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3654000000000001e-05 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043326+0000", 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4459999999999998e-06 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043332+0000", 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4690000000000002e-06 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043339+0000", 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032279800000000001 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043661+0000", 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455999999999999e-05 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043682+0000", 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012017435 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055699+0000", 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7508000000000001e-05 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055737+0000", 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012434328 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:12 57.10 e122/115 57:0d24be56:test-rados-api-smithi107-112513-8::foo:head v 122'1, mlcod=0'0)", 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:56.047722+0000", 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 48.432627717999999, 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024058560000000001, 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047722+0000", 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047722+0000", 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047721+0000", 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0142e-05 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047731+0000", 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0151e-05 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047741+0000", 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3640000000000001e-06 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.047748+0000", 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015209290000000001 2024-02-10T08:37:44.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.049269+0000", 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9939e-05 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.049289+0000", 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00081112299999999997 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.050100+0000", 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7540000000000001e-05 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.050128+0000", 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024058560000000001 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(93.6s1 145/144 ECSubWrite(tid=1290, reqid=client.44944.0:2, at_version=145'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:18.972245+0000", 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 25.508104967000001, 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026896730000000001, 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972245+0000", 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972245+0000", 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972244+0000", 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.747e-05 2024-02-10T08:37:44.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972272+0000", 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5440000000000001e-06 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972279+0000", 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0659999999999998e-06 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972286+0000", 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025932 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972546+0000", 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.371e-06 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.972554+0000", 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002362121 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.974916+0000", 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8128999999999999e-05 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.974934+0000", 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026896730000000001 2024-02-10T08:37:44.565 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(93.6s1 146/144 ECSubWrite(tid=1292, reqid=client.44944.0:4, at_version=146'3, trim_to=0'0, roll_forward_to=145'1))", 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:19.962638+0000", 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 24.517711894000001, 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065358120000000002, 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962638+0000", 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962638+0000", 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962637+0000", 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0340000000000001e-05 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962668+0000", 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0090000000000007e-06 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962676+0000", 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9500000000000004e-06 2024-02-10T08:37:44.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.962683+0000", 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002460828 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.965143+0000", 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4267999999999999e-05 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.965168+0000", 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039778280000000001 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.969145+0000", 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8175999999999999e-05 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.969174+0000", 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065358120000000002 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.570 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:404 60.0 e146/135 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head v 146'1, mlcod=0'0)", 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:19.973985+0000", 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 24.506364384000001, 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002329476, 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.973985+0000", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.973985+0000", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.973984+0000", 2024-02-10T08:37:44.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3377000000000001e-05 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.973998+0000", 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5229999999999995e-06 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.974006+0000", 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5819999999999998e-06 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.974013+0000", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000149443 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.974162+0000", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9093000000000002e-05 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.974181+0000", 2024-02-10T08:37:44.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020967799999999999 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.976278+0000", 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6681000000000003e-05 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.976315+0000", 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002329476 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.575 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:505 60.0 e147/135 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head v 147'8, mlcod=146'5)", 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:20.928548+0000", 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.551801672, 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055081770000000004, 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.576 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928548+0000", 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928548+0000", 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928548+0000", 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7061000000000001e-05 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928565+0000", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2450000000000001e-06 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928572+0000", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8170000000000001e-06 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928579+0000", 2024-02-10T08:37:44.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109288 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928688+0000", 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2779e-05 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.928711+0000", 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053157209999999998 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.934027+0000", 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9640000000000001e-05 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.934056+0000", 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055081770000000004 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(93.6s1 147/144 ECSubWrite(tid=1306, reqid=osd.5.0:1306, at_version=147'5, trim_to=0'0, roll_forward_to=146'3))", 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:20.943013+0000", 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.537336704000001, 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019153729999999999, 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943013+0000", 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943013+0000", 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943013+0000", 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0055999999999999e-05 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943033+0000", 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2350000000000002e-06 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943040+0000", 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8369999999999997e-06 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943048+0000", 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025059000000000002 2024-02-10T08:37:44.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943298+0000", 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0859000000000001e-05 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943309+0000", 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001604746 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.944914+0000", 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.435e-05 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.944928+0000", 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019153729999999999 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:606 60.0 e148/135 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head v 148'20, mlcod=147'16)", 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.545740243000001, 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013072240000000001, 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9150000000000002e-06 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934618+0000", 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1688999999999999e-05 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934640+0000", 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1930000000000004e-06 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934647+0000", 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082309480000000001 2024-02-10T08:37:44.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942878+0000", 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0771000000000002e-05 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942899+0000", 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047591389999999999 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.947658+0000", 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3985000000000002e-05 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.947682+0000", 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013072240000000001 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.590 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(93.6s1 148/144 ECSubWrite(tid=1320, reqid=client.44944.0:9, at_version=148'6, trim_to=0'0, roll_forward_to=147'5))", 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.938334+0000", 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.542015596999999, 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006404585, 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.938334+0000", 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.938334+0000", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.938334+0000", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1841999999999999e-05 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.938355+0000", 2024-02-10T08:37:44.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038455400000000002 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942201+0000", 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7780000000000002e-06 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942211+0000", 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015594799999999999 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942367+0000", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5279999999999992e-06 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942376+0000", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023490630000000002 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.944725+0000", 2024-02-10T08:37:44.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3327e-05 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.944739+0000", 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006404585 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(93.6s1 148/144 ECSubWrite(tid=1321, reqid=client.44944.0:9, at_version=0'0, trim_to=0'0, roll_forward_to=148'6))", 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.953051+0000", 2024-02-10T08:37:44.595 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.527298831, 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043613189999999998, 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953051+0000", 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:37:44.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953051+0000", 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953050+0000", 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0941000000000001e-05 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953061+0000", 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5379999999999997e-06 2024-02-10T08:37:44.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953069+0000", 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2880000000000004e-06 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953076+0000", 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3489999999999999e-05 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:37:44.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953120+0000", 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1864e-05 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.953131+0000", 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042582230000000002 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.957390+0000", 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2586999999999999e-05 2024-02-10T08:37:44.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.957412+0000", 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043613189999999998 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:37:44.600 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:37:44.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.926278+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:42.929164+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:43.517308+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:43.520303+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.935778+0000 mon.a (mon.0) 1268 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:43.935916+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.944176+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.107:0/1173755769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.945453+0000 mon.a (mon.0) 1270 : audit [INF] from='client.52769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.955831+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:43.956401+0000 mon.a (mon.0) 1271 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:44.018059+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.926278+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:42.929164+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:43.517308+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:43.520303+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.935778+0000 mon.a (mon.0) 1268 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:43.935916+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-10T08:37:44.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.944176+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.107:0/1173755769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.945453+0000 mon.a (mon.0) 1270 : audit [INF] from='client.52769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.955831+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:43.956401+0000 mon.a (mon.0) 1271 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:44.018059+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:44.628 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:44.725 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:44.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.926278+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-02-10T08:37:44.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:42.929164+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-02-10T08:37:44.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:43.517308+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:37:44.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:43.520303+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:37:44.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.935778+0000 mon.a (mon.0) 1268 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:43.935916+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.944176+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.107:0/1173755769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.945453+0000 mon.a (mon.0) 1270 : audit [INF] from='client.52769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.955831+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.107:0/2651721103' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:43.956401+0000 mon.a (mon.0) 1271 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T08:37:44.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:44.018059+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:44.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:44.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.792+0000 7fa553820700 1 -- 172.21.15.107:0/2040888414 >> v1:172.21.15.107:6790/0 conn(0x7fa54c0f89c0 legacy=0x7fa54c0f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:44.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.792+0000 7fa553820700 1 -- 172.21.15.107:0/2040888414 shutdown_connections 2024-02-10T08:37:44.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.792+0000 7fa553820700 1 -- 172.21.15.107:0/2040888414 >> 172.21.15.107:0/2040888414 conn(0x7fa54c0f2510 msgr2=0x7fa54c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:44.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.792+0000 7fa553820700 1 -- 172.21.15.107:0/2040888414 shutdown_connections 2024-02-10T08:37:44.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 -- 172.21.15.107:0/2040888414 wait complete. 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 librados: starting msgr at 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 librados: starting objecter 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 Processor -- start 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 -- start start 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 librados: setting wanted keys 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.793+0000 7fa553820700 1 librados: calling monclient init 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa553820700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa54c117440 con 0x7fa54c0f89c0 2024-02-10T08:37:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa553820700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa54c118620 con 0x7fa54c104350 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa553820700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa54c119820 con 0x7fa54c0f9570 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5515bc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa54c0f89c0 0x7fa54c0ff730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58980/0 (socket says 172.21.15.107:58980) 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa550dbb700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa54c0f9570 0x7fa54c0ffe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41476/0 (socket says 172.21.15.107:41476) 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5515bc700 1 -- 172.21.15.107:0/2294644570 learned_addr learned my addr 172.21.15.107:0/2294644570 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter ms_handle_connect 0x7fa54c0f9570 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter ms_handle_connect 0x7fa54c0f89c0 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082712078 0 0) 0x7fa54c119820 con 0x7fa54c0f9570 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538003030 con 0x7fa54c0f9570 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter ms_handle_connect 0x7fa54c104350 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150630518 0 0) 0x7fa54c117440 con 0x7fa54c0f89c0 2024-02-10T08:37:44.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa54c119820 con 0x7fa54c0f89c0 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619034552 0 0) 0x7fa54c118620 con 0x7fa54c104350 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa54c117440 con 0x7fa54c104350 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 812801762 0 0) 0x7fa538003030 con 0x7fa54c0f9570 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54c118620 con 0x7fa54c0f9570 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa53c0018c0 con 0x7fa54c0f9570 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361198845 0 0) 0x7fa54c119820 con 0x7fa54c0f89c0 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa538003030 con 0x7fa54c0f89c0 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.794+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa5480015d0 con 0x7fa54c0f89c0 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695499005 0 0) 0x7fa54c117440 con 0x7fa54c104350 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54c119820 con 0x7fa54c104350 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa544001ca0 con 0x7fa54c104350 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2728218220 0 0) 0x7fa54c118620 con 0x7fa54c0f9570 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 >> v1:172.21.15.195:6789/0 conn(0x7fa54c104350 legacy=0x7fa54c115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 >> v1:172.21.15.107:6789/0 conn(0x7fa54c0f89c0 legacy=0x7fa54c0ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:44.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c11aa20 con 0x7fa54c0f9570 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c117630 con 0x7fa54c0f9570 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa553820700 10 client.52781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c117aa0 con 0x7fa54c0f9570 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa553820700 1 librados: init done 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa53c002290 con 0x7fa54c0f9570 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.795+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa53c005920 con 0x7fa54c0f9570 2024-02-10T08:37:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.796+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa530004f20 con 0x7fa54c0f9570 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.796+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa53c001df0 con 0x7fa54c0f9570 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(170..170 src has 50..170) v4 ==== 9810+0+0 (unknown 2157317687 0 0) 0x7fa53c0790e0 con 0x7fa54c0f9570 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 10 client.52781.objecter ms_dispatch 0x7fa54c001550 osd_map(170..170 src has 50..170) v4 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 3 client.52781.objecter handle_osd_map got epochs [170,170] > 0 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 3 client.52781.objecter handle_osd_map decoding full epoch 170 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 20 client.52781.objecter dump_active .. 0 homeless 2024-02-10T08:37:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.797+0000 7fa5427fc700 10 client.52781.objecter ms_handle_connect 0x7fa538063cc0 2024-02-10T08:37:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:44.804+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa53c04be20 con 0x7fa54c0f9570 2024-02-10T08:37:44.816 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:44.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:44.860 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.010+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa5300051d0 con 0x7fa54c0f9570 2024-02-10T08:37:45.109 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:45.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:45.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:45.241 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:45.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:37:45.590 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:45.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:45.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:45.723 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:43.914363+0000 osd.2 (osd.2) 69 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:43.917276+0000 osd.2 (osd.2) 70 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.470171+0000 mgr.y (mgr.14553) 210 : cluster [DBG] pgmap v231: 322 pgs: 32 unknown, 8 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.515199+0000 osd.7 (osd.7) 85 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.518081+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.531406+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.533185+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:44.939119+0000 mon.a (mon.0) 1272 : audit [INF] from='client.52769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:44.939277+0000 mon.a (mon.0) 1273 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.939378+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:44.943023+0000 mon.c (mon.2) 202 : audit [INF] from='client.? v1:172.21.15.107:0/1963374674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:44.946417+0000 mon.a (mon.0) 1275 : audit [INF] from='client.52775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.011650+0000 mon.c (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.107:0/2294644570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.011922+0000 mon.a (mon.0) 1276 : audit [INF] from='client.52781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.019107+0000 mon.b (mon.1) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:45] "GET /metrics HTTP/1.1" 200 191974 "" "Prometheus/2.33.4" 2024-02-10T08:37:45.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:43.914363+0000 osd.2 (osd.2) 69 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:43.917276+0000 osd.2 (osd.2) 70 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.470171+0000 mgr.y (mgr.14553) 210 : cluster [DBG] pgmap v231: 322 pgs: 32 unknown, 8 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.515199+0000 osd.7 (osd.7) 85 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.518081+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.531406+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.533185+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:44.939119+0000 mon.a (mon.0) 1272 : audit [INF] from='client.52769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:44.939277+0000 mon.a (mon.0) 1273 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.939378+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:44.943023+0000 mon.c (mon.2) 202 : audit [INF] from='client.? v1:172.21.15.107:0/1963374674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:44.946417+0000 mon.a (mon.0) 1275 : audit [INF] from='client.52775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.011650+0000 mon.c (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.107:0/2294644570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.011922+0000 mon.a (mon.0) 1276 : audit [INF] from='client.52781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.019107+0000 mon.b (mon.1) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:43.914363+0000 osd.2 (osd.2) 69 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:43.917276+0000 osd.2 (osd.2) 70 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.470171+0000 mgr.y (mgr.14553) 210 : cluster [DBG] pgmap v231: 322 pgs: 32 unknown, 8 creating+peering, 282 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.515199+0000 osd.7 (osd.7) 85 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.518081+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.531406+0000 osd.3 (osd.3) 101 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.533185+0000 osd.3 (osd.3) 102 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:44.939119+0000 mon.a (mon.0) 1272 : audit [INF] from='client.52769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi107-112411-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:44.939277+0000 mon.a (mon.0) 1273 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi107-112480-1","app":"app1","key":"key1"}]': finished 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.939378+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-10T08:37:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:44.943023+0000 mon.c (mon.2) 202 : audit [INF] from='client.? v1:172.21.15.107:0/1963374674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:44.946417+0000 mon.a (mon.0) 1275 : audit [INF] from='client.52775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:45.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.011650+0000 mon.c (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.107:0/2294644570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.011922+0000 mon.a (mon.0) 1276 : audit [INF] from='client.52781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:37:45.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.019107+0000 mon.b (mon.1) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.943+0000 7fa5427fc700 1 -- 172.21.15.107:0/2294644570 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v172) v1 ==== 101+0+0 (unknown 4093632292 0 0) 0x7fa53c04a290 con 0x7fa54c0f9570 2024-02-10T08:37:45.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.948+0000 7fa553820700 10 librados: watch_flush enter 2024-02-10T08:37:45.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.949+0000 7fa553820700 10 librados: watch_flush exit 2024-02-10T08:37:45.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.949+0000 7fa553820700 20 client.52781.objecter shutdown clearing up homeless session... 2024-02-10T08:37:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.949+0000 7fa553820700 10 client.52781.objecter successfully canceled tick 2024-02-10T08:37:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.949+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa538063cc0 legacy=0x7fa538066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.949+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 >> v1:172.21.15.107:6790/0 conn(0x7fa54c0f9570 legacy=0x7fa54c0ffe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:45.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.961+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 shutdown_connections 2024-02-10T08:37:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.961+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 >> 172.21.15.107:0/2294644570 conn(0x7fa54c0f2510 msgr2=0x7fa54c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.962+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 shutdown_connections 2024-02-10T08:37:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.962+0000 7fa553820700 1 -- 172.21.15.107:0/2294644570 wait complete. 2024-02-10T08:37:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:45.962+0000 7fa553820700 1 librados: shutdown 2024-02-10T08:37:45.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-10T08:37:45.967 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: seed 112480 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (10 ms) 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (252 ms) 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (26 ms) 2024-02-10T08:37:45.968 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (35 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (26 ms) 2024-02-10T08:37:45.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (7 ms) 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1618 ms) 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (474 ms) 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-10T08:37:45.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: done waiting, doing copies 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: done waiting 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61369 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (3 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4173 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-10T08:37:45.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-10T08:37:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:37:46.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.594425+0000 osd.5 (osd.5) 91 : cluster [DBG] 6.a scrub starts 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.597330+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.a scrub ok 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.869577+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:44.870567+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.546918+0000 osd.6 (osd.6) 83 : cluster [DBG] 25.d scrub starts 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.548292+0000 osd.6 (osd.6) 84 : cluster [DBG] 25.d scrub ok 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.821565+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.825148+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.936549+0000 mon.a (mon.0) 1277 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.942119+0000 mon.a (mon.0) 1278 : audit [INF] from='client.52775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.942335+0000 mon.a (mon.0) 1279 : audit [INF] from='client.52781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.942387+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.980585+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.980917+0000 mon.a (mon.0) 1281 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.982092+0000 mon.c (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.982383+0000 mon.a (mon.0) 1282 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.982786+0000 mon.c (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.594425+0000 osd.5 (osd.5) 91 : cluster [DBG] 6.a scrub starts 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.597330+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.a scrub ok 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.869577+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:44.870567+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.546918+0000 osd.6 (osd.6) 83 : cluster [DBG] 25.d scrub starts 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.548292+0000 osd.6 (osd.6) 84 : cluster [DBG] 25.d scrub ok 2024-02-10T08:37:46.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.821565+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.825148+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.936549+0000 mon.a (mon.0) 1277 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.942119+0000 mon.a (mon.0) 1278 : audit [INF] from='client.52775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.942335+0000 mon.a (mon.0) 1279 : audit [INF] from='client.52781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.942387+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.980585+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.980917+0000 mon.a (mon.0) 1281 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.982092+0000 mon.c (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.982383+0000 mon.a (mon.0) 1282 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.982786+0000 mon.c (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:45.983049+0000 mon.a (mon.0) 1283 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:46.022179+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:46.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:45.983049+0000 mon.a (mon.0) 1283 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:46.022179+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:46.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:46.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.594425+0000 osd.5 (osd.5) 91 : cluster [DBG] 6.a scrub starts 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.597330+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.a scrub ok 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.869577+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:44.870567+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.546918+0000 osd.6 (osd.6) 83 : cluster [DBG] 25.d scrub starts 2024-02-10T08:37:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.548292+0000 osd.6 (osd.6) 84 : cluster [DBG] 25.d scrub ok 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.821565+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.825148+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.936549+0000 mon.a (mon.0) 1277 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.942119+0000 mon.a (mon.0) 1278 : audit [INF] from='client.52775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi107-112403-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.942335+0000 mon.a (mon.0) 1279 : audit [INF] from='client.52781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.942387+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.980585+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.980917+0000 mon.a (mon.0) 1281 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.982092+0000 mon.c (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.982383+0000 mon.a (mon.0) 1282 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.982786+0000 mon.c (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:45.983049+0000 mon.a (mon.0) 1283 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:46.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:46.022179+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:46.807 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:47.057 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:47.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:47.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:47.190 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:47.211 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:47.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.277+0000 7f7135176700 1 -- 172.21.15.107:0/2382414455 >> v1:172.21.15.107:6790/0 conn(0x7f7130103250 legacy=0x7f71301056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.277+0000 7f7135176700 1 -- 172.21.15.107:0/2382414455 shutdown_connections 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.277+0000 7f7135176700 1 -- 172.21.15.107:0/2382414455 >> 172.21.15.107:0/2382414455 conn(0x7f71300f2510 msgr2=0x7f71300f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.277+0000 7f7135176700 1 -- 172.21.15.107:0/2382414455 shutdown_connections 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.277+0000 7f7135176700 1 -- 172.21.15.107:0/2382414455 wait complete. 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 librados: starting msgr at 2024-02-10T08:37:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 librados: starting objecter 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 Processor -- start 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 -- start start 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 librados: setting wanted keys 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 librados: calling monclient init 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301162e0 con 0x7f71300ff710 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301174e0 con 0x7f71300fc910 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.278+0000 7f7135176700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301186e0 con 0x7f7130103250 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f712e59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71300ff710 0x7f71300f9340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59050/0 (socket says 172.21.15.107:59050) 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f712e59c700 1 -- 172.21.15.107:0/3399772062 learned_addr learned my addr 172.21.15.107:0/3399772062 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f71300ff710 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f7130103250 2024-02-10T08:37:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f71300fc910 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325408853 0 0) 0x7f71301186e0 con 0x7f7130103250 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7118003030 con 0x7f7130103250 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265137637 0 0) 0x7f71301162e0 con 0x7f71300ff710 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71301186e0 con 0x7f71300ff710 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611458028 0 0) 0x7f71301174e0 con 0x7f71300fc910 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71301162e0 con 0x7f71300fc910 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2605224792 0 0) 0x7f7118003030 con 0x7f7130103250 2024-02-10T08:37:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71301174e0 con 0x7f7130103250 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7128002fa0 con 0x7f7130103250 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1728818323 0 0) 0x7f71301186e0 con 0x7f71300ff710 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7118003030 con 0x7f71300ff710 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71200019f0 con 0x7f71300ff710 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091607786 0 0) 0x7f71301162e0 con 0x7f71300fc910 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71301186e0 con 0x7f71300fc910 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71240018c0 con 0x7f71300fc910 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1412191690 0 0) 0x7f71301174e0 con 0x7f7130103250 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.195:6789/0 conn(0x7f71300fc910 legacy=0x7f71300f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6789/0 conn(0x7f71300ff710 legacy=0x7f71300f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71301198e0 con 0x7f7130103250 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71301188d0 con 0x7f7130103250 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7135176700 10 client.52802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7130118d40 con 0x7f7130103250 2024-02-10T08:37:47.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.279+0000 7f7135176700 1 librados: init done 2024-02-10T08:37:47.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.280+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f71280032c0 con 0x7f7130103250 2024-02-10T08:37:47.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.280+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7128003a80 con 0x7f7130103250 2024-02-10T08:37:47.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.280+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f711c004f00 con 0x7f7130103250 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f71280023d0 con 0x7f7130103250 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(173..173 src has 50..173) v4 ==== 9676+0+0 (unknown 518480987 0 0) 0x7f7128077120 con 0x7f7130103250 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 osd_map(173..173 src has 50..173) v4 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 3 client.52802.objecter handle_osd_map got epochs [173,173] > 0 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 3 client.52802.objecter handle_osd_map decoding full epoch 173 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 20 client.52802.objecter dump_active .. 0 homeless 2024-02-10T08:37:47.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v173) v1 ==== 54+0+16 (unknown 857201663 0 2420000192) 0x7f7128079530 con 0x7f7130103250 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.281+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f7118063cc0 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 10 client.52802.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 1 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 1 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 0 1 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 10 client.52802.objecter _send_command 1 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c004950 con 0x7f711c004530 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c004530 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c004950 con 0x7f711c004530 2024-02-10T08:37:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:37:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 10 client.52802.objecter _finish_command 1 = system:0 2024-02-10T08:37:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:37:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.282+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 0 1 2024-02-10T08:37:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 10 client.52802.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 2 2024-02-10T08:37:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 2 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 0 2 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 10 client.52802.objecter _send_command 2 2024-02-10T08:37:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.297+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c004d30 con 0x7f711c004530 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.298+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c004d30 con 0x7f711c004530 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.298+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.298+0000 7f7117fff700 10 client.52802.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.298+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.298+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 0 2 2024-02-10T08:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 10 client.52802.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 1 3 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 10 client.52802.objecter _send_command 3 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c00e210 con 0x7f711c004150 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.299+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c004150 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.300+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c00e210 con 0x7f711c004150 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.300+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.300+0000 7f7117fff700 10 client.52802.objecter _finish_command 3 = system:0 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.300+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:37:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.300+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 1 3 2024-02-10T08:37:47.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 10 client.52802.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 4 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 4 2024-02-10T08:37:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:37:47.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 1 4 2024-02-10T08:37:47.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:37:47.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 10 client.52802.objecter _send_command 4 2024-02-10T08:37:47.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.314+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c0018b0 con 0x7f711c004150 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.315+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c0018b0 con 0x7f711c004150 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.315+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.315+0000 7f7117fff700 10 client.52802.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.315+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.315+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 1 4 2024-02-10T08:37:47.317 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 10 client.52802.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 5 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 5 2024-02-10T08:37:47.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 2 5 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 10 client.52802.objecter _send_command 5 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c0100a0 con 0x7f711c00f500 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c00f500 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c0100a0 con 0x7f711c00f500 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 10 client.52802.objecter _finish_command 5 = system:0 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:37:47.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.316+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 2 5 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 10 client.52802.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 6 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 6 2024-02-10T08:37:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:37:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 2 6 2024-02-10T08:37:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:37:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 10 client.52802.objecter _send_command 6 2024-02-10T08:37:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.331+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c0030a0 con 0x7f711c00f500 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c0030a0 con 0x7f711c00f500 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7117fff700 10 client.52802.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 2 6 2024-02-10T08:37:47.333 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 10 client.52802.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 7 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 7 2024-02-10T08:37:47.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 3 7 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 4 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 10 client.52802.objecter _send_command 7 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.332+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c017ca0 con 0x7f711c014090 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c014090 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c017ca0 con 0x7f711c014090 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 10 client.52802.objecter _finish_command 7 = system:0 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 4 2024-02-10T08:37:47.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.333+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 3 7 2024-02-10T08:37:47.343 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:47.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 10 client.52802.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 8 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 8 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c003480 osd=3 3 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 3 8 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 4 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 10 client.52802.objecter _send_command 8 2024-02-10T08:37:47.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.347+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c010480 con 0x7f711c014090 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c010480 con 0x7f711c014090 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 10 client.52802.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 4 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 3 8 2024-02-10T08:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 10 client.52802.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 9 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 9 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 4 9 2024-02-10T08:37:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 4 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 10 client.52802.objecter _send_command 9 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c010680 con 0x7f711c00f060 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.349+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c00f060 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.350+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c010680 con 0x7f711c00f060 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.350+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.350+0000 7f7117fff700 10 client.52802.objecter _finish_command 9 = system:0 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.350+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 4 2024-02-10T08:37:47.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.350+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 4 9 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 10 client.52802.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 10 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 10 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c010480 osd=4 3 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 4 10 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 4 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 10 client.52802.objecter _send_command 10 2024-02-10T08:37:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.365+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c010680 con 0x7f711c00f060 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.366+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c010680 con 0x7f711c00f060 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.366+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.366+0000 7f7117fff700 10 client.52802.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.366+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 4 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.366+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 4 10 2024-02-10T08:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 10 client.52802.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 11 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 11 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 5 11 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 4 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 10 client.52802.objecter _send_command 11 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c018e70 con 0x7f711c018a30 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.367+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c018a30 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.368+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c018e70 con 0x7f711c018a30 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.368+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.368+0000 7f7117fff700 10 client.52802.objecter _finish_command 11 = system:0 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.368+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 4 2024-02-10T08:37:47.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.368+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 5 11 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 10 client.52802.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 12 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 12 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c010680 osd=5 3 2024-02-10T08:37:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 5 12 2024-02-10T08:37:47.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 4 2024-02-10T08:37:47.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 10 client.52802.objecter _send_command 12 2024-02-10T08:37:47.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.382+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c018e70 con 0x7f711c018a30 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.383+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c018e70 con 0x7f711c018a30 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.383+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.383+0000 7f7117fff700 10 client.52802.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.383+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 4 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.383+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 5 12 2024-02-10T08:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 10 client.52802.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 13 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 13 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 6 13 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 4 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 10 client.52802.objecter _send_command 13 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c019270 con 0x7f711c0236a0 2024-02-10T08:37:47.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.384+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c0236a0 2024-02-10T08:37:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.385+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c019270 con 0x7f711c0236a0 2024-02-10T08:37:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.385+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:37:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.385+0000 7f7117fff700 10 client.52802.objecter _finish_command 13 = system:0 2024-02-10T08:37:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.385+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 4 2024-02-10T08:37:47.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.385+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 6 13 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 10 client.52802.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 14 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 14 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c018e70 osd=6 3 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 6 14 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 4 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 10 client.52802.objecter _send_command 14 2024-02-10T08:37:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.398+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c019270 con 0x7f711c0236a0 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c019270 con 0x7f711c0236a0 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7117fff700 10 client.52802.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 4 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 6 14 2024-02-10T08:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7135176700 10 client.52802.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 15 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.400+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 15 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 7 15 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 4 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 10 client.52802.objecter _send_command 15 2024-02-10T08:37:47.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c028230 con 0x7f711c028cd0 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.401+0000 7f7117fff700 10 client.52802.objecter ms_handle_connect 0x7f711c028cd0 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.402+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c028230 con 0x7f711c028cd0 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.402+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.402+0000 7f7117fff700 10 client.52802.objecter _finish_command 15 = system:0 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.402+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 4 2024-02-10T08:37:47.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.402+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 7 15 2024-02-10T08:37:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 10 client.52802.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:37:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign -1 16 2024-02-10T08:37:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 20 client.52802.objecter _get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 15 client.52802.objecter _session_command_op_remove -1 16 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 20 client.52802.objecter get_session s=0x7f711c019270 osd=7 3 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 15 client.52802.objecter _session_command_op_assign 7 16 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 4 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 10 client.52802.objecter _send_command 16 2024-02-10T08:37:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.416+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c028230 con 0x7f711c028cd0 2024-02-10T08:37:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7117fff700 1 -- 172.21.15.107:0/3399772062 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c028230 con 0x7f711c028cd0 2024-02-10T08:37:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7117fff700 10 client.52802.objecter ms_dispatch 0x7f7130001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:37:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7117fff700 10 client.52802.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7117fff700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 4 2024-02-10T08:37:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7117fff700 15 client.52802.objecter _session_command_op_remove 7 16 2024-02-10T08:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 librados: watch_flush enter 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 librados: watch_flush exit 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.0 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6801/3321596972 conn(0x7f711c004530 legacy=0x7f711c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c0051a0 osd=0 1 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.1 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6805/133516126 conn(0x7f711c004150 legacy=0x7f711c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003fe0 osd=1 1 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.2 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6809/3575881728 conn(0x7f711c00f500 legacy=0x7f711c00f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003630 osd=2 1 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.3 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6813/2871825691 conn(0x7f711c014090 legacy=0x7f711c016480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c003480 osd=3 1 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.4 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f712f59e700 1 -- 172.21.15.107:0/3399772062 reap_dead start 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.195:6800/2499671285 conn(0x7f711c00f060 legacy=0x7f711c017e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f712f59e700 1 -- 172.21.15.107:0/3399772062 reap_dead start 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010480 osd=4 1 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.5 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.195:6804/4178321486 conn(0x7f711c018a30 legacy=0x7f711c021a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c010680 osd=5 1 2024-02-10T08:37:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.6 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.195:6808/1417479017 conn(0x7f711c0236a0 legacy=0x7f711c025a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c018e70 osd=6 1 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter close_session for osd.7 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.195:6812/1934176928 conn(0x7f711c028cd0 legacy=0x7f711c029150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter put_session s=0x7f711c019270 osd=7 1 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 20 client.52802.objecter shutdown clearing up homeless session... 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 10 client.52802.objecter successfully canceled tick 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7118063cc0 legacy=0x7f7118066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> v1:172.21.15.107:6790/0 conn(0x7f7130103250 legacy=0x7f71301149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f712f59e700 1 -- 172.21.15.107:0/3399772062 reap_dead start 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 shutdown_connections 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 >> 172.21.15.107:0/3399772062 conn(0x7f71300f2510 msgr2=0x7f71301066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 shutdown_connections 2024-02-10T08:37:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 -- 172.21.15.107:0/3399772062 wait complete. 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:47.418+0000 7f7135176700 1 librados: shutdown 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:37:47.441 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:47.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:47.448 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:47.448 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:47.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:47.573 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:47.696 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:47.697 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.870444+0000 osd.2 (osd.2) 73 : cluster [DBG] 25.1 scrub starts 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:45.871876+0000 osd.2 (osd.2) 74 : cluster [DBG] 25.1 scrub ok 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.471198+0000 mgr.y (mgr.14553) 211 : cluster [DBG] pgmap v234: 290 pgs: 14 creating+activating, 276 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.580473+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.583381+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.937635+0000 mon.a (mon.0) 1284 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:46.945371+0000 mon.a (mon.0) 1285 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.945642+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:46.956880+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:46.957658+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:46.957949+0000 mon.a (mon.0) 1288 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:47.023202+0000 mon.b (mon.1) 245 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:47.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:47.281959+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.107:0/3399772062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.870444+0000 osd.2 (osd.2) 73 : cluster [DBG] 25.1 scrub starts 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:45.871876+0000 osd.2 (osd.2) 74 : cluster [DBG] 25.1 scrub ok 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.471198+0000 mgr.y (mgr.14553) 211 : cluster [DBG] pgmap v234: 290 pgs: 14 creating+activating, 276 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.580473+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.583381+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.937635+0000 mon.a (mon.0) 1284 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:46.945371+0000 mon.a (mon.0) 1285 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.945642+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:46.956880+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:46.957658+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:47.699 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:46.957949+0000 mon.a (mon.0) 1288 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:47.023202+0000 mon.b (mon.1) 245 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:47.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:47.281959+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.107:0/3399772062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:47.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.870444+0000 osd.2 (osd.2) 73 : cluster [DBG] 25.1 scrub starts 2024-02-10T08:37:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:45.871876+0000 osd.2 (osd.2) 74 : cluster [DBG] 25.1 scrub ok 2024-02-10T08:37:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.471198+0000 mgr.y (mgr.14553) 211 : cluster [DBG] pgmap v234: 290 pgs: 14 creating+activating, 276 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.580473+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:37:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.583381+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.937635+0000 mon.a (mon.0) 1284 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:46.945371+0000 mon.a (mon.0) 1285 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112480-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.945642+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:46.956880+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:46.957658+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:46.957949+0000 mon.a (mon.0) 1288 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:47.023202+0000 mon.b (mon.1) 245 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:47.281959+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.107:0/3399772062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:37:47.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:47.830 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:37:48.178 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:48.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:48.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:48.310 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.329 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:48.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:48.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:48.445 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:48.460 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.479 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:48.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:48.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:48.610 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.573696+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:37:48.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:46.576657+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:47.585269+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:47.587043+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:47.811130+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:47.814583+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:47.950551+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:48.024047+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.190537+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:48.192823+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.573696+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:46.576657+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:37:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:47.585269+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:47.587043+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:47.811130+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:47.814583+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:47.950551+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:48.024047+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.190537+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-10T08:37:48.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:48.192823+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:48.682 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:48.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:48.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:37:48.743 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.573696+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:37:48.743 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:46.576657+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:47.585269+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:47.587043+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:47.811130+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:47.814583+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:47.950551+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.107:0/2239155362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi107-112411-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:48.024047+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.190537+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-10T08:37:48.744 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:48.192823+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:48.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:48.811 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:48.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:48.851 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:37:48.851 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:37:48.851 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.meta 2024-02-10T08:37:48.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:37:48.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:48.968 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:48.990 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.468106+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.469777+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.471611+0000 mgr.y (mgr.14553) 212 : cluster [DBG] pgmap v237: 322 pgs: 64 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:48.955587+0000 mon.a (mon.0) 1292 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.468106+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.469777+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:37:49.488 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.471611+0000 mgr.y (mgr.14553) 212 : cluster [DBG] pgmap v237: 322 pgs: 64 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:49.489 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:48.955587+0000 mon.a (mon.0) 1292 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:49.489 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:48.955768+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:49.489 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.955854+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-10T08:37:49.489 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:49.025051+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.468106+0000 osd.3 (osd.3) 103 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.469777+0000 osd.3 (osd.3) 104 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.471611+0000 mgr.y (mgr.14553) 212 : cluster [DBG] pgmap v237: 322 pgs: 64 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:48.955587+0000 mon.a (mon.0) 1292 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112480-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:48.955768+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:49.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.955854+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-10T08:37:49.667 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:49.025051+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:49.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:48.955768+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? v1:172.21.15.107:0/23173584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi107-112403-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:49.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.955854+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-10T08:37:49.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:49.025051+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:50.037 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:50.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:50.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:37:50.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:50.170 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:50.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:37:50.459 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:50.459 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.895144+0000 osd.2 (osd.2) 75 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:48.901976+0000 osd.2 (osd.2) 76 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:49.654065+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:49.657206+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:49.885550+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:49.889141+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:49.959177+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:49.961249+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:49.964490+0000 mon.a (mon.0) 1296 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:49.966899+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.107:0/2010278204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:49.973919+0000 mon.a (mon.0) 1297 : audit [INF] from='client.52808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.709+0000 7f84fbd78700 1 -- 172.21.15.107:0/1629867869 >> v1:172.21.15.107:6790/0 conn(0x7f84f40fc910 legacy=0x7f84f40fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.709+0000 7f84fbd78700 1 -- 172.21.15.107:0/1629867869 shutdown_connections 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.709+0000 7f84fbd78700 1 -- 172.21.15.107:0/1629867869 >> 172.21.15.107:0/1629867869 conn(0x7f84f40f2510 msgr2=0x7f84f40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.709+0000 7f84fbd78700 1 -- 172.21.15.107:0/1629867869 shutdown_connections 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.709+0000 7f84fbd78700 1 -- 172.21.15.107:0/1629867869 wait complete. 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 librados: starting msgr at 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 librados: starting objecter 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 Processor -- start 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 -- start start 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 librados: setting wanted keys 2024-02-10T08:37:50.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 librados: calling monclient init 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84f4117350 con 0x7f84f40ff710 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84f41184f0 con 0x7f84f40fc910 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fbd78700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84f41196f0 con 0x7f84f4103250 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84f9313700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f84f40ff710 0x7f84f40fa1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58248/0 (socket says 172.21.15.107:58248) 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fa315700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f84f4103250 0x7f84f41159e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41688/0 (socket says 172.21.15.107:41688) 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84f9b14700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f84f40fc910 0x7f84f40f9aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:52146/0 (socket says 172.21.15.107:52146) 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84fa315700 1 -- 172.21.15.107:0/2199584269 learned_addr learned my addr 172.21.15.107:0/2199584269 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter ms_handle_connect 0x7f84f4103250 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter ms_handle_connect 0x7f84f40fc910 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter ms_handle_connect 0x7f84f40ff710 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.710+0000 7f84eaffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374403388 0 0) 0x7f84f41196f0 con 0x7f84f4103250 2024-02-10T08:37:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e0003630 con 0x7f84f4103250 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177368789 0 0) 0x7f84f41184f0 con 0x7f84f40fc910 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f41196f0 con 0x7f84f40fc910 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210550637 0 0) 0x7f84f4117350 con 0x7f84f40ff710 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f41184f0 con 0x7f84f40ff710 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529451489 0 0) 0x7f84e0003630 con 0x7f84f4103250 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84f4117350 con 0x7f84f4103250 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f84ec001c80 con 0x7f84f4103250 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060713725 0 0) 0x7f84f41184f0 con 0x7f84f40ff710 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e0003630 con 0x7f84f40ff710 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389366227 0 0) 0x7f84f41196f0 con 0x7f84f40fc910 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84f41184f0 con 0x7f84f40fc910 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f84f00018b0 con 0x7f84f40ff710 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f84e4002d80 con 0x7f84f40fc910 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865230733 0 0) 0x7f84f4117350 con 0x7f84f4103250 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 >> v1:172.21.15.195:6789/0 conn(0x7f84f40fc910 legacy=0x7f84f40f9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 >> v1:172.21.15.107:6789/0 conn(0x7f84f40ff710 legacy=0x7f84f40fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f411a8f0 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f41186e0 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84fbd78700 10 client.52820.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f4118bd0 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84fbd78700 1 librados: init done 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f84ec0027a0 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.711+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f84ec005e30 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.712+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d8004f20 con 0x7f84f4103250 2024-02-10T08:37:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.712+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f84ec002300 con 0x7f84f4103250 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(176..176 src has 50..176) v4 ==== 10597+0+0 (unknown 1295810125 0 0) 0x7f84ec079ba0 con 0x7f84f4103250 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 10 client.52820.objecter ms_dispatch 0x7f84f4001550 osd_map(176..176 src has 50..176) v4 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 3 client.52820.objecter handle_osd_map got epochs [176,176] > 0 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 3 client.52820.objecter handle_osd_map decoding full epoch 176 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 20 client.52820.objecter dump_active .. 0 homeless 2024-02-10T08:37:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.713+0000 7f84eaffd700 10 client.52820.objecter ms_handle_connect 0x7f84e0063ce0 2024-02-10T08:37:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.717+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f84ec04c5d0 con 0x7f84f4103250 2024-02-10T08:37:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:50.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.895144+0000 osd.2 (osd.2) 75 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:48.901976+0000 osd.2 (osd.2) 76 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:49.654065+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:49.657206+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:49.885550+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:49.889141+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:49.959177+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-10T08:37:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:49.961249+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:49.964490+0000 mon.a (mon.0) 1296 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:49.966899+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.107:0/2010278204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:49.973919+0000 mon.a (mon.0) 1297 : audit [INF] from='client.52808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.026067+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:50.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:50.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.026067+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:50.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.895144+0000 osd.2 (osd.2) 75 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:48.901976+0000 osd.2 (osd.2) 76 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:49.654065+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:49.657206+0000 osd.7 (osd.7) 88 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:49.885550+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:49.889141+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:49.959177+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:49.961249+0000 mon.c (mon.2) 209 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:49.964490+0000 mon.a (mon.0) 1296 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:49.966899+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.107:0/2010278204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:49.973919+0000 mon.a (mon.0) 1297 : audit [INF] from='client.52808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:50.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.026067+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:50.851 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:50.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.903+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f84d80051d0 con 0x7f84f4103250 2024-02-10T08:37:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.935+0000 7f84eaffd700 1 -- 172.21.15.107:0/2199584269 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v177) v1 ==== 87+0+0 (unknown 3858247306 0 0) 0x7f84ec04a9e0 con 0x7f84f4103250 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.937+0000 7f84fbd78700 10 librados: watch_flush enter 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.937+0000 7f84fbd78700 10 librados: watch_flush exit 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.937+0000 7f84fbd78700 20 client.52820.objecter shutdown clearing up homeless session... 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.937+0000 7f84fbd78700 10 client.52820.objecter successfully canceled tick 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.938+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 >> v1:172.21.15.107:6800/2634842009 conn(0x7f84e0063ce0 legacy=0x7f84e00661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:50.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.938+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 >> v1:172.21.15.107:6790/0 conn(0x7f84f4103250 legacy=0x7f84f41159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.938+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 shutdown_connections 2024-02-10T08:37:50.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.938+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 >> 172.21.15.107:0/2199584269 conn(0x7f84f40f2510 msgr2=0x7f84f41066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.947+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 shutdown_connections 2024-02-10T08:37:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.948+0000 7f84fbd78700 1 -- 172.21.15.107:0/2199584269 wait complete. 2024-02-10T08:37:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:50.952+0000 7f84fbd78700 1 librados: shutdown 2024-02-10T08:37:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:37:51.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:37:51.424 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:37:51.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:37:51.631 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:51.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.472105+0000 mgr.y (mgr.14553) 213 : cluster [DBG] pgmap v240: 330 pgs: 72 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:51.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.583307+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.d scrub starts 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.586877+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub ok 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.655399+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.656131+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.904496+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.107:0/2199584269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.904798+0000 mon.a (mon.0) 1298 : audit [INF] from='client.52820 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.922303+0000 mon.a (mon.0) 1299 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.928560+0000 mon.a (mon.0) 1300 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.928753+0000 mon.a (mon.0) 1301 : audit [INF] from='client.52808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.928885+0000 mon.a (mon.0) 1302 : audit [INF] from='client.52820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:50.928939+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.939264+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.940112+0000 mon.c (mon.2) 213 : audit [INF] from='client.? v1:172.21.15.107:0/1615631491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.950277+0000 mon.a (mon.0) 1304 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.950497+0000 mon.a (mon.0) 1305 : audit [INF] from='client.52814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.953176+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:50.954147+0000 mon.a (mon.0) 1306 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.026999+0000 mon.b (mon.1) 249 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.472105+0000 mgr.y (mgr.14553) 213 : cluster [DBG] pgmap v240: 330 pgs: 72 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.583307+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.d scrub starts 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.586877+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub ok 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.655399+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.656131+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:37:51.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.904496+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.107:0/2199584269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.904798+0000 mon.a (mon.0) 1298 : audit [INF] from='client.52820 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.922303+0000 mon.a (mon.0) 1299 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.928560+0000 mon.a (mon.0) 1300 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.928753+0000 mon.a (mon.0) 1301 : audit [INF] from='client.52808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.928885+0000 mon.a (mon.0) 1302 : audit [INF] from='client.52820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:50.928939+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.939264+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.940112+0000 mon.c (mon.2) 213 : audit [INF] from='client.? v1:172.21.15.107:0/1615631491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.950277+0000 mon.a (mon.0) 1304 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.950497+0000 mon.a (mon.0) 1305 : audit [INF] from='client.52814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.953176+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:50.954147+0000 mon.a (mon.0) 1306 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.026999+0000 mon.b (mon.1) 249 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:51.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:51.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.472105+0000 mgr.y (mgr.14553) 213 : cluster [DBG] pgmap v240: 330 pgs: 72 unknown, 258 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.583307+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.d scrub starts 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.586877+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub ok 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.655399+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.656131+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.904496+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.107:0/2199584269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.904798+0000 mon.a (mon.0) 1298 : audit [INF] from='client.52820 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.922303+0000 mon.a (mon.0) 1299 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.928560+0000 mon.a (mon.0) 1300 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.928753+0000 mon.a (mon.0) 1301 : audit [INF] from='client.52808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi107-112411-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:51.763 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.928885+0000 mon.a (mon.0) 1302 : audit [INF] from='client.52820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:50.928939+0000 mon.a (mon.0) 1303 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.939264+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.940112+0000 mon.c (mon.2) 213 : audit [INF] from='client.? v1:172.21.15.107:0/1615631491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.950277+0000 mon.a (mon.0) 1304 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.950497+0000 mon.a (mon.0) 1305 : audit [INF] from='client.52814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.953176+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:50.954147+0000 mon.a (mon.0) 1306 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:51.764 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.026999+0000 mon.b (mon.1) 249 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:51.765 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:51.787 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:51.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:51.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:51.918 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.220979+0000 mgr.y (mgr.14553) 214 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.928515+0000 mon.a (mon.0) 1307 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.928600+0000 mon.a (mon.0) 1308 : audit [INF] from='client.52814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.928674+0000 mon.a (mon.0) 1309 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:51.928773+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.934864+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:51.935535+0000 mon.a (mon.0) 1311 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.027822+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.220979+0000 mgr.y (mgr.14553) 214 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.928515+0000 mon.a (mon.0) 1307 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.928600+0000 mon.a (mon.0) 1308 : audit [INF] from='client.52814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.928674+0000 mon.a (mon.0) 1309 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:51.928773+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.934864+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:51.935535+0000 mon.a (mon.0) 1311 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.027822+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.220979+0000 mgr.y (mgr.14553) 214 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.928515+0000 mon.a (mon.0) 1307 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.928600+0000 mon.a (mon.0) 1308 : audit [INF] from='client.52814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi107-112403-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.928674+0000 mon.a (mon.0) 1309 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112480-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:51.928773+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-10T08:37:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.934864+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.107:0/2956897450' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:51.935535+0000 mon.a (mon.0) 1311 : audit [INF] from='client.52739 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]: dispatch 2024-02-10T08:37:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.027822+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:52.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.659+0000 7f30eb3e6700 1 -- 172.21.15.107:0/565250724 >> v1:172.21.15.107:6790/0 conn(0x7f30e4103250 legacy=0x7f30e41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:52.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.659+0000 7f30eb3e6700 1 -- 172.21.15.107:0/565250724 shutdown_connections 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.659+0000 7f30eb3e6700 1 -- 172.21.15.107:0/565250724 >> 172.21.15.107:0/565250724 conn(0x7f30e40f2510 msgr2=0x7f30e40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.659+0000 7f30eb3e6700 1 -- 172.21.15.107:0/565250724 shutdown_connections 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.659+0000 7f30eb3e6700 1 -- 172.21.15.107:0/565250724 wait complete. 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 librados: starting msgr at 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 librados: starting objecter 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 Processor -- start 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 -- start start 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 librados: setting wanted keys 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 librados: calling monclient init 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30e40fbf00 con 0x7f30e40ff710 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30e40fc0d0 con 0x7f30e40fc910 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30eb3e6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30e40fc2c0 con 0x7f30e4103250 2024-02-10T08:37:52.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30e8981700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f30e40ff710 0x7f30e4114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58282/0 (socket says 172.21.15.107:58282) 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30e9983700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f30e4103250 0x7f30e41183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41724/0 (socket says 172.21.15.107:41724) 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30e8981700 1 -- 172.21.15.107:0/2884148211 learned_addr learned my addr 172.21.15.107:0/2884148211 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter ms_handle_connect 0x7f30e40ff710 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter ms_handle_connect 0x7f30e4103250 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter ms_handle_connect 0x7f30e40fc910 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400902304 0 0) 0x7f30e40fbf00 con 0x7f30e40ff710 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d0003800 con 0x7f30e40ff710 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186780957 0 0) 0x7f30e40fc2c0 con 0x7f30e4103250 2024-02-10T08:37:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e40fbf00 con 0x7f30e4103250 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972701464 0 0) 0x7f30e40fc0d0 con 0x7f30e40fc910 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.660+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e40fc2c0 con 0x7f30e40fc910 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736047491 0 0) 0x7f30d0003800 con 0x7f30e40ff710 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e40fc0d0 con 0x7f30e40ff710 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30d4002910 con 0x7f30e40ff710 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277016314 0 0) 0x7f30e40fbf00 con 0x7f30e4103250 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d0003800 con 0x7f30e4103250 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 451726252 0 0) 0x7f30e40fc2c0 con 0x7f30e40fc910 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e40fbf00 con 0x7f30e40fc910 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30dc003150 con 0x7f30e4103250 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30e00018d0 con 0x7f30e40fc910 2024-02-10T08:37:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3127301040 0 0) 0x7f30e40fc0d0 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 >> v1:172.21.15.107:6790/0 conn(0x7f30e4103250 legacy=0x7f30e41183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 >> v1:172.21.15.195:6789/0 conn(0x7f30e40fc910 legacy=0x7f30e40fb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e411cb30 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4119b60 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30eb3e6700 10 client.27333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e411a050 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30eb3e6700 1 librados: init done 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f30d4002b80 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.661+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30d4005b80 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.662+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c8004f20 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.662+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f30d40015e0 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(178..178 src has 50..178) v4 ==== 9842+0+0 (unknown 3587376675 0 0) 0x7f30d4079310 con 0x7f30e40ff710 2024-02-10T08:37:52.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 10 client.27333.objecter ms_dispatch 0x7f30e4001550 osd_map(178..178 src has 50..178) v4 2024-02-10T08:37:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 3 client.27333.objecter handle_osd_map got epochs [178,178] > 0 2024-02-10T08:37:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 3 client.27333.objecter handle_osd_map decoding full epoch 178 2024-02-10T08:37:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 20 client.27333.objecter dump_active .. 0 homeless 2024-02-10T08:37:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.663+0000 7f30da7fc700 10 client.27333.objecter ms_handle_connect 0x7f30d0063ce0 2024-02-10T08:37:52.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.667+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f30d404c010 con 0x7f30e40ff710 2024-02-10T08:37:52.714 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.832+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30c8000b90 con 0x7f30d0063ce0 2024-02-10T08:37:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.854+0000 7f30da7fc700 1 -- 172.21.15.107:0/2884148211 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+795203 (unknown 2965378022 0 2186981881) 0x7f30c8000b90 con 0x7f30d0063ce0 2024-02-10T08:37:52.864 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:37:52.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 10 librados: watch_flush enter 2024-02-10T08:37:52.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 10 librados: watch_flush exit 2024-02-10T08:37:52.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 20 client.27333.objecter shutdown clearing up homeless session... 2024-02-10T08:37:52.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 10 client.27333.objecter successfully canceled tick 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 >> v1:172.21.15.107:6800/2634842009 conn(0x7f30d0063ce0 legacy=0x7f30d00661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 >> v1:172.21.15.107:6789/0 conn(0x7f30e40ff710 legacy=0x7f30e4114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.858+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 shutdown_connections 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.859+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 >> 172.21.15.107:0/2884148211 conn(0x7f30e40f2510 msgr2=0x7f30e40612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.859+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 shutdown_connections 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.859+0000 7f30eb3e6700 1 -- 172.21.15.107:0/2884148211 wait complete. 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:37:52.859+0000 7f30eb3e6700 1 librados: shutdown 2024-02-10T08:37:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:37:52.885 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:52.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:52.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-02-10T08:37:52.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-02-10T08:37:52.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-10T08:37:52.957 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: seed 112525 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2101 ms) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3751 ms) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2523 ms) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1602 ms) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4014 ms) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (13991 ms total) 2024-02-10T08:37:52.958 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4157 ms) 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4072 ms) 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5748 ms) 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3913 ms) 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2125 ms) 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi107-112525-7 2024-02-10T08:37:52.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18217 ms) 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (38232 ms total) 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5038 ms) 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5045 ms total) 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-10T08:37:52.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3003 ms) 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2003 ms) 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2242 ms) 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1770 ms) 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9018 ms total) 2024-02-10T08:37:52.961 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: 2024-02-10T08:37:52.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:37:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:53.148 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:37:53.148 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:37:53.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:52.472643+0000 mgr.y (mgr.14553) 215 : cluster [DBG] pgmap v243: 322 pgs: 32 creating+peering, 290 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:53.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.833892+0000 mgr.y (mgr.14553) 216 : audit [DBG] from='client.27333 v1:172.21.15.107:0/2884148211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:53.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:52.925661+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.935883+0000 mon.a (mon.0) 1313 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:52.936011+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.941182+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.107:0/3076020731' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.943451+0000 mon.a (mon.0) 1315 : audit [INF] from='client.52826 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.973123+0000 mon.c (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.973938+0000 mon.a (mon.0) 1316 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.978755+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.979270+0000 mon.a (mon.0) 1317 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.981034+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.981762+0000 mon.c (mon.2) 220 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:52.472643+0000 mgr.y (mgr.14553) 215 : cluster [DBG] pgmap v243: 322 pgs: 32 creating+peering, 290 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.833892+0000 mgr.y (mgr.14553) 216 : audit [DBG] from='client.27333 v1:172.21.15.107:0/2884148211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:52.925661+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.935883+0000 mon.a (mon.0) 1313 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:52.936011+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.941182+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.107:0/3076020731' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.943451+0000 mon.a (mon.0) 1315 : audit [INF] from='client.52826 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.973123+0000 mon.c (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.973938+0000 mon.a (mon.0) 1316 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.978755+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.979270+0000 mon.a (mon.0) 1317 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.981034+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.981762+0000 mon.c (mon.2) 220 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.982334+0000 mon.a (mon.0) 1318 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.982762+0000 mon.a (mon.0) 1319 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.983722+0000 mon.c (mon.2) 221 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.984058+0000 mon.a (mon.0) 1320 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.984652+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:52.984932+0000 mon.a (mon.0) 1321 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.028746+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.982334+0000 mon.a (mon.0) 1318 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.982762+0000 mon.a (mon.0) 1319 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.983722+0000 mon.c (mon.2) 221 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.984058+0000 mon.a (mon.0) 1320 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.984652+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:52.984932+0000 mon.a (mon.0) 1321 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.028746+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:53.625 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-02-10T08:37:52.472328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15357,"num_objects":935,"num_object_clones":541,"num_object_copies":2805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":1297,"num_read_kb":964,"num_write":1045,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":3296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1738,"ondisk_log_size":1738,"up":966,"acting":966,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":914,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2130964,"kb_used_data":25808,"kb_used_omap":0,"kb_used_meta":2105088,"kb_avail":654085100,"statfs":{"total":671965249536,"available":669783142400,"internally_reserved":0,"allocated":26427392,"data_stored":9817936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2155610112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-384,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-3,"num_write_kb":-3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000974"},"pg_stats":[{"pgid":"121.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.935750+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.8","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.959916+0000","last_change":"2024-02-10T08:37:50.959916+0000","last_active":"2024-02-10T08:37:50.959916+0000","last_peered":"2024-02-10T08:37:50.959916+0000","last_clean":"2024-02-10T08:37:50.959916+0000","last_became_active":"2024-02-10T08:37:50.959669+0000","last_became_peered":"2024-02-10T08:37:50.959669+0000","last_unstale":"2024-02-10T08:37:50.959916+0000","last_undegraded":"2024-02-10T08:37:50.959916+0000","last_fullsized":"2024-02-10T08:37:50.959916+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"121.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.932360+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.9","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.950543+0000","last_change":"2024-02-10T08:37:50.950543+0000","last_active":"2024-02-10T08:37:50.950543+0000","last_peered":"2024-02-10T08:37:50.950543+0000","last_clean":"2024-02-10T08:37:50.950543+0000","last_became_active":"2024-02-10T08:37:50.950265+0000","last_became_peered":"2024-02-10T08:37:50.950265+0000","last_unstale":"2024-02-10T08:37:50.950543+0000","last_undegraded":"2024-02-10T08:37:50.950543+0000","last_fullsized":"2024-02-10T08:37:50.950543+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.926865+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"119.a","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.946983+0000","last_change":"2024-02-10T08:37:50.946983+0000","last_active":"2024-02-10T08:37:50.946983+0000","last_peered":"2024-02-10T08:37:50.946983+0000","last_clean":"2024-02-10T08:37:50.946983+0000","last_became_active":"2024-02-10T08:37:50.946698+0000","last_became_peered":"2024-02-10T08:37:50.946698+0000","last_unstale":"2024-02-10T08:37:50.946983+0000","last_undegraded":"2024-02-10T08:37:50.946983+0000","last_fullsized":"2024-02-10T08:37:50.946983+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"121.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.937567+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.b","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.964720+0000","last_change":"2024-02-10T08:37:50.964720+0000","last_active":"2024-02-10T08:37:50.964720+0000","last_peered":"2024-02-10T08:37:50.964720+0000","last_clean":"2024-02-10T08:37:50.964720+0000","last_became_active":"2024-02-10T08:37:50.964074+0000","last_became_peered":"2024-02-10T08:37:50.964074+0000","last_unstale":"2024-02-10T08:37:50.964720+0000","last_undegraded":"2024-02-10T08:37:50.964720+0000","last_fullsized":"2024-02-10T08:37:50.964720+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"121.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.938011+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.c","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.956449+0000","last_change":"2024-02-10T08:37:50.956449+0000","last_active":"2024-02-10T08:37:50.956449+0000","last_peered":"2024-02-10T08:37:50.956449+0000","last_clean":"2024-02-10T08:37:50.956449+0000","last_became_active":"2024-02-10T08:37:50.956016+0000","last_became_peered":"2024-02-10T08:37:50.956016+0000","last_unstale":"2024-02-10T08:37:50.956449+0000","last_undegraded":"2024-02-10T08:37:50.956449+0000","last_fullsized":"2024-02-10T08:37:50.956449+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.931181+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"119.d","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.950427+0000","last_change":"2024-02-10T08:37:50.950427+0000","last_active":"2024-02-10T08:37:50.950427+0000","last_peered":"2024-02-10T08:37:50.950427+0000","last_clean":"2024-02-10T08:37:50.950427+0000","last_became_active":"2024-02-10T08:37:50.950193+0000","last_became_peered":"2024-02-10T08:37:50.950193+0000","last_unstale":"2024-02-10T08:37:50.950427+0000","last_undegraded":"2024-02-10T08:37:50.950427+0000","last_fullsized":"2024-02-10T08:37:50.950427+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"121.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.938271+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.e","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.965038+0000","last_change":"2024-02-10T08:37:50.965038+0000","last_active":"2024-02-10T08:37:50.965038+0000","last_peered":"2024-02-10T08:37:50.965038+0000","last_clean":"2024-02-10T08:37:50.965038+0000","last_became_active":"2024-02-10T08:37:50.964709+0000","last_became_peered":"2024-02-10T08:37:50.964709+0000","last_unstale":"2024-02-10T08:37:50.965038+0000","last_undegraded":"2024-02-10T08:37:50.965038+0000","last_fullsized":"2024-02-10T08:37:50.965038+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"121.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.927802+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.f","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.954296+0000","last_change":"2024-02-10T08:37:50.954296+0000","last_active":"2024-02-10T08:37:50.954296+0000","last_peered":"2024-02-10T08:37:50.954296+0000","last_clean":"2024-02-10T08:37:50.954296+0000","last_became_active":"2024-02-10T08:37:50.954002+0000","last_became_peered":"2024-02-10T08:37:50.954002+0000","last_unstale":"2024-02-10T08:37:50.954296+0000","last_undegraded":"2024-02-10T08:37:50.954296+0000","last_fullsized":"2024-02-10T08:37:50.954296+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.936130+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.0","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.959428+0000","last_change":"2024-02-10T08:37:50.959428+0000","last_active":"2024-02-10T08:37:50.959428+0000","last_peered":"2024-02-10T08:37:50.959428+0000","last_clean":"2024-02-10T08:37:50.959428+0000","last_became_active":"2024-02-10T08:37:50.959168+0000","last_became_peered":"2024-02-10T08:37:50.959168+0000","last_unstale":"2024-02-10T08:37:50.959428+0000","last_undegraded":"2024-02-10T08:37:50.959428+0000","last_fullsized":"2024-02-10T08:37:50.959428+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.1","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.993764+0000","last_change":"2024-02-10T08:37:49.992927+0000","last_active":"2024-02-10T08:37:49.993764+0000","last_peered":"2024-02-10T08:37:49.993764+0000","last_clean":"2024-02-10T08:37:49.993764+0000","last_became_active":"2024-02-10T08:37:49.992134+0000","last_became_peered":"2024-02-10T08:37:49.992134+0000","last_unstale":"2024-02-10T08:37:49.993764+0000","last_undegraded":"2024-02-10T08:37:49.993764+0000","last_fullsized":"2024-02-10T08:37:49.993764+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"121.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.940006+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.1","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.951677+0000","last_change":"2024-02-10T08:37:50.951677+0000","last_active":"2024-02-10T08:37:50.951677+0000","last_peered":"2024-02-10T08:37:50.951677+0000","last_clean":"2024-02-10T08:37:50.951677+0000","last_became_active":"2024-02-10T08:37:50.951352+0000","last_became_peered":"2024-02-10T08:37:50.951352+0000","last_unstale":"2024-02-10T08:37:50.951677+0000","last_undegraded":"2024-02-10T08:37:50.951677+0000","last_fullsized":"2024-02-10T08:37:50.951677+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"118.0","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.992989+0000","last_change":"2024-02-10T08:37:49.992552+0000","last_active":"2024-02-10T08:37:49.992989+0000","last_peered":"2024-02-10T08:37:49.992989+0000","last_clean":"2024-02-10T08:37:49.992989+0000","last_became_active":"2024-02-10T08:37:49.992222+0000","last_became_peered":"2024-02-10T08:37:49.992222+0000","last_unstale":"2024-02-10T08:37:49.992989+0000","last_undegraded":"2024-02-10T08:37:49.992989+0000","last_fullsized":"2024-02-10T08:37:49.992989+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"121.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.933400+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.2","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.960105+0000","last_change":"2024-02-10T08:37:50.960105+0000","last_active":"2024-02-10T08:37:50.960105+0000","last_peered":"2024-02-10T08:37:50.960105+0000","last_clean":"2024-02-10T08:37:50.960105+0000","last_became_active":"2024-02-10T08:37:50.959819+0000","last_became_peered":"2024-02-10T08:37:50.959819+0000","last_unstale":"2024-02-10T08:37:50.960105+0000","last_undegraded":"2024-02-10T08:37:50.960105+0000","last_fullsized":"2024-02-10T08:37:50.960105+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"118.3","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.992483+0000","last_change":"2024-02-10T08:37:49.992134+0000","last_active":"2024-02-10T08:37:49.992483+0000","last_peered":"2024-02-10T08:37:49.992483+0000","last_clean":"2024-02-10T08:37:49.992483+0000","last_became_active":"2024-02-10T08:37:49.991780+0000","last_became_peered":"2024-02-10T08:37:49.991780+0000","last_unstale":"2024-02-10T08:37:49.992483+0000","last_undegraded":"2024-02-10T08:37:49.992483+0000","last_fullsized":"2024-02-10T08:37:49.992483+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"121.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.931478+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.3","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.946640+0000","last_change":"2024-02-10T08:37:50.946640+0000","last_active":"2024-02-10T08:37:50.946640+0000","last_peered":"2024-02-10T08:37:50.946640+0000","last_clean":"2024-02-10T08:37:50.946640+0000","last_became_active":"2024-02-10T08:37:50.946352+0000","last_became_peered":"2024-02-10T08:37:50.946352+0000","last_unstale":"2024-02-10T08:37:50.946640+0000","last_undegraded":"2024-02-10T08:37:50.946640+0000","last_fullsized":"2024-02-10T08:37:50.946640+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"118.2","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.989765+0000","last_change":"2024-02-10T08:37:49.985763+0000","last_active":"2024-02-10T08:37:49.989765+0000","last_peered":"2024-02-10T08:37:49.989765+0000","last_clean":"2024-02-10T08:37:49.989765+0000","last_became_active":"2024-02-10T08:37:49.985426+0000","last_became_peered":"2024-02-10T08:37:49.985426+0000","last_unstale":"2024-02-10T08:37:49.989765+0000","last_undegraded":"2024-02-10T08:37:49.989765+0000","last_fullsized":"2024-02-10T08:37:49.989765+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"121.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.941446+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.4","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.957352+0000","last_change":"2024-02-10T08:37:50.957352+0000","last_active":"2024-02-10T08:37:50.957352+0000","last_peered":"2024-02-10T08:37:50.957352+0000","last_clean":"2024-02-10T08:37:50.957352+0000","last_became_active":"2024-02-10T08:37:50.957016+0000","last_became_peered":"2024-02-10T08:37:50.957016+0000","last_unstale":"2024-02-10T08:37:50.957352+0000","last_undegraded":"2024-02-10T08:37:50.957352+0000","last_fullsized":"2024-02-10T08:37:50.957352+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"118.5","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.985761+0000","last_change":"2024-02-10T08:37:49.985535+0000","last_active":"2024-02-10T08:37:49.985761+0000","last_peered":"2024-02-10T08:37:49.985761+0000","last_clean":"2024-02-10T08:37:49.985761+0000","last_became_active":"2024-02-10T08:37:49.985182+0000","last_became_peered":"2024-02-10T08:37:49.985182+0000","last_unstale":"2024-02-10T08:37:49.985761+0000","last_undegraded":"2024-02-10T08:37:49.985761+0000","last_fullsized":"2024-02-10T08:37:49.985761+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.931503+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"119.5","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.953951+0000","last_change":"2024-02-10T08:37:50.953951+0000","last_active":"2024-02-10T08:37:50.953951+0000","last_peered":"2024-02-10T08:37:50.953951+0000","last_clean":"2024-02-10T08:37:50.953951+0000","last_became_active":"2024-02-10T08:37:50.953658+0000","last_became_peered":"2024-02-10T08:37:50.953658+0000","last_unstale":"2024-02-10T08:37:50.953951+0000","last_undegraded":"2024-02-10T08:37:50.953951+0000","last_fullsized":"2024-02-10T08:37:50.953951+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"118.4","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.986608+0000","last_change":"2024-02-10T08:37:49.983975+0000","last_active":"2024-02-10T08:37:49.986608+0000","last_peered":"2024-02-10T08:37:49.986608+0000","last_clean":"2024-02-10T08:37:49.986608+0000","last_became_active":"2024-02-10T08:37:49.983662+0000","last_became_peered":"2024-02-10T08:37:49.983662+0000","last_unstale":"2024-02-10T08:37:49.986608+0000","last_undegraded":"2024-02-10T08:37:49.986608+0000","last_fullsized":"2024-02-10T08:37:49.986608+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"121.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.927080+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"119.6","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.965642+0000","last_change":"2024-02-10T08:37:50.965642+0000","last_active":"2024-02-10T08:37:50.965642+0000","last_peered":"2024-02-10T08:37:50.965642+0000","last_clean":"2024-02-10T08:37:50.965642+0000","last_became_active":"2024-02-10T08:37:50.965347+0000","last_became_peered":"2024-02-10T08:37:50.965347+0000","last_unstale":"2024-02-10T08:37:50.965642+0000","last_undegraded":"2024-02-10T08:37:50.965642+0000","last_fullsized":"2024-02-10T08:37:50.965642+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.7","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.987402+0000","last_change":"2024-02-10T08:37:49.987014+0000","last_active":"2024-02-10T08:37:49.987402+0000","last_peered":"2024-02-10T08:37:49.987402+0000","last_clean":"2024-02-10T08:37:49.987402+0000","last_became_active":"2024-02-10T08:37:49.986723+0000","last_became_peered":"2024-02-10T08:37:49.986723+0000","last_unstale":"2024-02-10T08:37:49.987402+0000","last_undegraded":"2024-02-10T08:37:49.987402+0000","last_fullsized":"2024-02-10T08:37:49.987402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"121.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.942064+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.7","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.967601+0000","last_change":"2024-02-10T08:37:50.967601+0000","last_active":"2024-02-10T08:37:50.967601+0000","last_peered":"2024-02-10T08:37:50.967601+0000","last_clean":"2024-02-10T08:37:50.967601+0000","last_became_active":"2024-02-10T08:37:50.967293+0000","last_became_peered":"2024-02-10T08:37:50.967293+0000","last_unstale":"2024-02-10T08:37:50.967601+0000","last_undegraded":"2024-02-10T08:37:50.967601+0000","last_fullsized":"2024-02-10T08:37:50.967601+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"118.6","version":"0'0","reported_seq":10,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.990439+0000","last_change":"2024-02-10T08:37:49.990180+0000","last_active":"2024-02-10T08:37:49.990439+0000","last_peered":"2024-02-10T08:37:49.990439+0000","last_clean":"2024-02-10T08:37:49.990439+0000","last_became_active":"2024-02-10T08:37:49.989890+0000","last_became_peered":"2024-02-10T08:37:49.989890+0000","last_unstale":"2024-02-10T08:37:49.990439+0000","last_undegraded":"2024-02-10T08:37:49.990439+0000","last_fullsized":"2024-02-10T08:37:49.990439+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:48.945028+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.945028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"121.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.933968+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"119.18","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.958780+0000","last_change":"2024-02-10T08:37:50.958780+0000","last_active":"2024-02-10T08:37:50.958780+0000","last_peered":"2024-02-10T08:37:50.958780+0000","last_clean":"2024-02-10T08:37:50.958780+0000","last_became_active":"2024-02-10T08:37:50.958498+0000","last_became_peered":"2024-02-10T08:37:50.958498+0000","last_unstale":"2024-02-10T08:37:50.958780+0000","last_undegraded":"2024-02-10T08:37:50.958780+0000","last_fullsized":"2024-02-10T08:37:50.958780+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"121.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.940451+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.19","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.952370+0000","last_change":"2024-02-10T08:37:50.952370+0000","last_active":"2024-02-10T08:37:50.952370+0000","last_peered":"2024-02-10T08:37:50.952370+0000","last_clean":"2024-02-10T08:37:50.952370+0000","last_became_active":"2024-02-10T08:37:50.952076+0000","last_became_peered":"2024-02-10T08:37:50.952076+0000","last_unstale":"2024-02-10T08:37:50.952370+0000","last_undegraded":"2024-02-10T08:37:50.952370+0000","last_fullsized":"2024-02-10T08:37:50.952370+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"121.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.929425+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.1a","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.956317+0000","last_change":"2024-02-10T08:37:50.956317+0000","last_active":"2024-02-10T08:37:50.956317+0000","last_peered":"2024-02-10T08:37:50.956317+0000","last_clean":"2024-02-10T08:37:50.956317+0000","last_became_active":"2024-02-10T08:37:50.955858+0000","last_became_peered":"2024-02-10T08:37:50.955858+0000","last_unstale":"2024-02-10T08:37:50.956317+0000","last_undegraded":"2024-02-10T08:37:50.956317+0000","last_fullsized":"2024-02-10T08:37:50.956317+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"121.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.929603+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.1b","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.960017+0000","last_change":"2024-02-10T08:37:50.960017+0000","last_active":"2024-02-10T08:37:50.960017+0000","last_peered":"2024-02-10T08:37:50.960017+0000","last_clean":"2024-02-10T08:37:50.960017+0000","last_became_active":"2024-02-10T08:37:50.959748+0000","last_became_peered":"2024-02-10T08:37:50.959748+0000","last_unstale":"2024-02-10T08:37:50.960017+0000","last_undegraded":"2024-02-10T08:37:50.960017+0000","last_fullsized":"2024-02-10T08:37:50.960017+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"121.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.929260+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.1c","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.952955+0000","last_change":"2024-02-10T08:37:50.952955+0000","last_active":"2024-02-10T08:37:50.952955+0000","last_peered":"2024-02-10T08:37:50.952955+0000","last_clean":"2024-02-10T08:37:50.952955+0000","last_became_active":"2024-02-10T08:37:50.952654+0000","last_became_peered":"2024-02-10T08:37:50.952654+0000","last_unstale":"2024-02-10T08:37:50.952955+0000","last_undegraded":"2024-02-10T08:37:50.952955+0000","last_fullsized":"2024-02-10T08:37:50.952955+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"121.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.938292+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"119.1d","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.954470+0000","last_change":"2024-02-10T08:37:50.954470+0000","last_active":"2024-02-10T08:37:50.954470+0000","last_peered":"2024-02-10T08:37:50.954470+0000","last_clean":"2024-02-10T08:37:50.954470+0000","last_became_active":"2024-02-10T08:37:50.954194+0000","last_became_peered":"2024-02-10T08:37:50.954194+0000","last_unstale":"2024-02-10T08:37:50.954470+0000","last_undegraded":"2024-02-10T08:37:50.954470+0000","last_fullsized":"2024-02-10T08:37:50.954470+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"121.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.928183+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.1e","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.954201+0000","last_change":"2024-02-10T08:37:50.954201+0000","last_active":"2024-02-10T08:37:50.954201+0000","last_peered":"2024-02-10T08:37:50.954201+0000","last_clean":"2024-02-10T08:37:50.954201+0000","last_became_active":"2024-02-10T08:37:50.953922+0000","last_became_peered":"2024-02-10T08:37:50.953922+0000","last_unstale":"2024-02-10T08:37:50.954201+0000","last_undegraded":"2024-02-10T08:37:50.954201+0000","last_fullsized":"2024-02-10T08:37:50.954201+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.942375+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.1f","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.950826+0000","last_change":"2024-02-10T08:37:50.950826+0000","last_active":"2024-02-10T08:37:50.950826+0000","last_peered":"2024-02-10T08:37:50.950826+0000","last_clean":"2024-02-10T08:37:50.950826+0000","last_became_active":"2024-02-10T08:37:50.950466+0000","last_became_peered":"2024-02-10T08:37:50.950466+0000","last_unstale":"2024-02-10T08:37:50.950826+0000","last_undegraded":"2024-02-10T08:37:50.950826+0000","last_fullsized":"2024-02-10T08:37:50.950826+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"121.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.936471+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.10","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.960039+0000","last_change":"2024-02-10T08:37:50.960039+0000","last_active":"2024-02-10T08:37:50.960039+0000","last_peered":"2024-02-10T08:37:50.960039+0000","last_clean":"2024-02-10T08:37:50.960039+0000","last_became_active":"2024-02-10T08:37:50.959755+0000","last_became_peered":"2024-02-10T08:37:50.959755+0000","last_unstale":"2024-02-10T08:37:50.960039+0000","last_undegraded":"2024-02-10T08:37:50.960039+0000","last_fullsized":"2024-02-10T08:37:50.960039+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"121.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.931366+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"119.11","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.952938+0000","last_change":"2024-02-10T08:37:50.952938+0000","last_active":"2024-02-10T08:37:50.952938+0000","last_peered":"2024-02-10T08:37:50.952938+0000","last_clean":"2024-02-10T08:37:50.952938+0000","last_became_active":"2024-02-10T08:37:50.952510+0000","last_became_peered":"2024-02-10T08:37:50.952510+0000","last_unstale":"2024-02-10T08:37:50.952938+0000","last_undegraded":"2024-02-10T08:37:50.952938+0000","last_fullsized":"2024-02-10T08:37:50.952938+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"121.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.933717+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"119.12","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.967976+0000","last_change":"2024-02-10T08:37:50.967976+0000","last_active":"2024-02-10T08:37:50.967976+0000","last_peered":"2024-02-10T08:37:50.967976+0000","last_clean":"2024-02-10T08:37:50.967976+0000","last_became_active":"2024-02-10T08:37:50.967694+0000","last_became_peered":"2024-02-10T08:37:50.967694+0000","last_unstale":"2024-02-10T08:37:50.967976+0000","last_undegraded":"2024-02-10T08:37:50.967976+0000","last_fullsized":"2024-02-10T08:37:50.967976+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"121.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.940480+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.13","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.951031+0000","last_change":"2024-02-10T08:37:50.951031+0000","last_active":"2024-02-10T08:37:50.951031+0000","last_peered":"2024-02-10T08:37:50.951031+0000","last_clean":"2024-02-10T08:37:50.951031+0000","last_became_active":"2024-02-10T08:37:50.950863+0000","last_became_peered":"2024-02-10T08:37:50.950863+0000","last_unstale":"2024-02-10T08:37:50.951031+0000","last_undegraded":"2024-02-10T08:37:50.951031+0000","last_fullsized":"2024-02-10T08:37:50.951031+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"121.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.929755+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"119.14","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.943220+0000","last_change":"2024-02-10T08:37:50.943220+0000","last_active":"2024-02-10T08:37:50.943220+0000","last_peered":"2024-02-10T08:37:50.943220+0000","last_clean":"2024-02-10T08:37:50.943220+0000","last_became_active":"2024-02-10T08:37:50.942533+0000","last_became_peered":"2024-02-10T08:37:50.942533+0000","last_unstale":"2024-02-10T08:37:50.943220+0000","last_undegraded":"2024-02-10T08:37:50.943220+0000","last_fullsized":"2024-02-10T08:37:50.943220+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"121.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.940682+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"119.15","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.952373+0000","last_change":"2024-02-10T08:37:50.952373+0000","last_active":"2024-02-10T08:37:50.952373+0000","last_peered":"2024-02-10T08:37:50.952373+0000","last_clean":"2024-02-10T08:37:50.952373+0000","last_became_active":"2024-02-10T08:37:50.952110+0000","last_became_peered":"2024-02-10T08:37:50.952110+0000","last_unstale":"2024-02-10T08:37:50.952373+0000","last_undegraded":"2024-02-10T08:37:50.952373+0000","last_fullsized":"2024-02-10T08:37:50.952373+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"121.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.943043+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.16","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.953482+0000","last_change":"2024-02-10T08:37:50.953482+0000","last_active":"2024-02-10T08:37:50.953482+0000","last_peered":"2024-02-10T08:37:50.953482+0000","last_clean":"2024-02-10T08:37:50.953482+0000","last_became_active":"2024-02-10T08:37:50.952792+0000","last_became_peered":"2024-02-10T08:37:50.952792+0000","last_unstale":"2024-02-10T08:37:50.953482+0000","last_undegraded":"2024-02-10T08:37:50.953482+0000","last_fullsized":"2024-02-10T08:37:50.953482+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"121.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:37:50.921460+0000","last_change":"2024-02-10T08:37:50.942742+0000","last_active":"2024-02-10T08:37:50.921460+0000","last_peered":"2024-02-10T08:37:50.921460+0000","last_clean":"2024-02-10T08:37:50.921460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:37:50.921460+0000","last_undegraded":"2024-02-10T08:37:50.921460+0000","last_fullsized":"2024-02-10T08:37:50.921460+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:50.921460+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.921460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"119.17","version":"0'0","reported_seq":9,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.955395+0000","last_change":"2024-02-10T08:37:50.955395+0000","last_active":"2024-02-10T08:37:50.955395+0000","last_peered":"2024-02-10T08:37:50.955395+0000","last_clean":"2024-02-10T08:37:50.955395+0000","last_became_active":"2024-02-10T08:37:50.955183+0000","last_became_peered":"2024-02-10T08:37:50.955183+0000","last_unstale":"2024-02-10T08:37:50.955395+0000","last_undegraded":"2024-02-10T08:37:50.955395+0000","last_fullsized":"2024-02-10T08:37:50.955395+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:49.952233+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.952233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.0","version":"153'19","reported_seq":130,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965853+0000","last_change":"2024-02-10T08:37:31.458914+0000","last_active":"2024-02-10T08:37:49.965853+0000","last_peered":"2024-02-10T08:37:49.965853+0000","last_clean":"2024-02-10T08:37:49.965853+0000","last_became_active":"2024-02-10T08:37:19.978613+0000","last_became_peered":"2024-02-10T08:37:19.978613+0000","last_unstale":"2024-02-10T08:37:49.965853+0000","last_undegraded":"2024-02-10T08:37:49.965853+0000","last_fullsized":"2024-02-10T08:37:49.965853+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'19","last_scrub_stamp":"2024-02-10T08:37:31.458795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:31.458795+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":194,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":19,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1","version":"153'3","reported_seq":50,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961860+0000","last_change":"2024-02-10T08:37:27.576383+0000","last_active":"2024-02-10T08:37:49.961860+0000","last_peered":"2024-02-10T08:37:49.961860+0000","last_clean":"2024-02-10T08:37:49.961860+0000","last_became_active":"2024-02-10T08:37:19.977530+0000","last_became_peered":"2024-02-10T08:37:19.977530+0000","last_unstale":"2024-02-10T08:37:49.961860+0000","last_undegraded":"2024-02-10T08:37:49.961860+0000","last_fullsized":"2024-02-10T08:37:49.961860+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'3","last_scrub_stamp":"2024-02-10T08:37:27.576278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:27.576278+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.2","version":"153'3","reported_seq":51,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970918+0000","last_change":"2024-02-10T08:37:35.449266+0000","last_active":"2024-02-10T08:37:49.970918+0000","last_peered":"2024-02-10T08:37:49.970918+0000","last_clean":"2024-02-10T08:37:49.970918+0000","last_became_active":"2024-02-10T08:37:19.980183+0000","last_became_peered":"2024-02-10T08:37:19.980183+0000","last_unstale":"2024-02-10T08:37:49.970918+0000","last_undegraded":"2024-02-10T08:37:49.970918+0000","last_fullsized":"2024-02-10T08:37:49.970918+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'3","last_scrub_stamp":"2024-02-10T08:37:35.449192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:35.449192+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.3","version":"153'4","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970076+0000","last_change":"2024-02-10T08:37:32.615906+0000","last_active":"2024-02-10T08:37:49.970076+0000","last_peered":"2024-02-10T08:37:49.970076+0000","last_clean":"2024-02-10T08:37:49.970076+0000","last_became_active":"2024-02-10T08:37:19.978876+0000","last_became_peered":"2024-02-10T08:37:19.978876+0000","last_unstale":"2024-02-10T08:37:49.970076+0000","last_undegraded":"2024-02-10T08:37:49.970076+0000","last_fullsized":"2024-02-10T08:37:49.970076+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4","last_scrub_stamp":"2024-02-10T08:37:32.615839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:32.615839+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.4","version":"153'3","reported_seq":51,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967064+0000","last_change":"2024-02-10T08:37:28.576463+0000","last_active":"2024-02-10T08:37:49.967064+0000","last_peered":"2024-02-10T08:37:49.967064+0000","last_clean":"2024-02-10T08:37:49.967064+0000","last_became_active":"2024-02-10T08:37:19.976935+0000","last_became_peered":"2024-02-10T08:37:19.976935+0000","last_unstale":"2024-02-10T08:37:49.967064+0000","last_undegraded":"2024-02-10T08:37:49.967064+0000","last_fullsized":"2024-02-10T08:37:49.967064+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'3","last_scrub_stamp":"2024-02-10T08:37:28.576392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:28.576392+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.5","version":"153'5","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969317+0000","last_change":"2024-02-10T08:37:28.467183+0000","last_active":"2024-02-10T08:37:49.969317+0000","last_peered":"2024-02-10T08:37:49.969317+0000","last_clean":"2024-02-10T08:37:49.969317+0000","last_became_active":"2024-02-10T08:37:19.972802+0000","last_became_peered":"2024-02-10T08:37:19.972802+0000","last_unstale":"2024-02-10T08:37:49.969317+0000","last_undegraded":"2024-02-10T08:37:49.969317+0000","last_fullsized":"2024-02-10T08:37:49.969317+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'5","last_scrub_stamp":"2024-02-10T08:37:28.467111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:28.467111+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.6","version":"153'10","reported_seq":63,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968158+0000","last_change":"2024-02-10T08:37:32.497415+0000","last_active":"2024-02-10T08:37:49.968158+0000","last_peered":"2024-02-10T08:37:49.968158+0000","last_clean":"2024-02-10T08:37:49.968158+0000","last_became_active":"2024-02-10T08:37:19.978647+0000","last_became_peered":"2024-02-10T08:37:19.978647+0000","last_unstale":"2024-02-10T08:37:49.968158+0000","last_undegraded":"2024-02-10T08:37:49.968158+0000","last_fullsized":"2024-02-10T08:37:49.968158+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'10","last_scrub_stamp":"2024-02-10T08:37:32.497344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:32.497344+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.7","version":"153'1","reported_seq":47,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977205+0000","last_change":"2024-02-10T08:37:36.497696+0000","last_active":"2024-02-10T08:37:49.977205+0000","last_peered":"2024-02-10T08:37:49.977205+0000","last_clean":"2024-02-10T08:37:49.977205+0000","last_became_active":"2024-02-10T08:37:19.975860+0000","last_became_peered":"2024-02-10T08:37:19.975860+0000","last_unstale":"2024-02-10T08:37:49.977205+0000","last_undegraded":"2024-02-10T08:37:49.977205+0000","last_fullsized":"2024-02-10T08:37:49.977205+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'1","last_scrub_stamp":"2024-02-10T08:37:36.497625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:36.497625+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.8","version":"153'16","reported_seq":73,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958905+0000","last_change":"2024-02-10T08:37:27.666949+0000","last_active":"2024-02-10T08:37:49.958905+0000","last_peered":"2024-02-10T08:37:49.958905+0000","last_clean":"2024-02-10T08:37:49.958905+0000","last_became_active":"2024-02-10T08:37:19.970400+0000","last_became_peered":"2024-02-10T08:37:19.970400+0000","last_unstale":"2024-02-10T08:37:49.958905+0000","last_undegraded":"2024-02-10T08:37:49.958905+0000","last_fullsized":"2024-02-10T08:37:49.958905+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-02-10T08:37:27.666863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:27.666863+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":16,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.9","version":"153'17","reported_seq":74,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961774+0000","last_change":"2024-02-10T08:37:40.521865+0000","last_active":"2024-02-10T08:37:49.961774+0000","last_peered":"2024-02-10T08:37:49.961774+0000","last_clean":"2024-02-10T08:37:49.961774+0000","last_became_active":"2024-02-10T08:37:19.982669+0000","last_became_peered":"2024-02-10T08:37:19.982669+0000","last_unstale":"2024-02-10T08:37:49.961774+0000","last_undegraded":"2024-02-10T08:37:49.961774+0000","last_fullsized":"2024-02-10T08:37:49.961774+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'17","last_scrub_stamp":"2024-02-10T08:37:40.521742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.521742+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":172,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":17,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.a","version":"153'3","reported_seq":45,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.983066+0000","last_change":"2024-02-10T08:37:19.978822+0000","last_active":"2024-02-10T08:37:49.983066+0000","last_peered":"2024-02-10T08:37:49.983066+0000","last_clean":"2024-02-10T08:37:49.983066+0000","last_became_active":"2024-02-10T08:37:19.978486+0000","last_became_peered":"2024-02-10T08:37:19.978486+0000","last_unstale":"2024-02-10T08:37:49.983066+0000","last_undegraded":"2024-02-10T08:37:49.983066+0000","last_fullsized":"2024-02-10T08:37:49.983066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.b","version":"153'10","reported_seq":56,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969994+0000","last_change":"2024-02-10T08:37:19.973471+0000","last_active":"2024-02-10T08:37:49.969994+0000","last_peered":"2024-02-10T08:37:49.969994+0000","last_clean":"2024-02-10T08:37:49.969994+0000","last_became_active":"2024-02-10T08:37:19.973202+0000","last_became_peered":"2024-02-10T08:37:19.973202+0000","last_unstale":"2024-02-10T08:37:49.969994+0000","last_undegraded":"2024-02-10T08:37:49.969994+0000","last_fullsized":"2024-02-10T08:37:49.969994+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.c","version":"153'6","reported_seq":51,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980554+0000","last_change":"2024-02-10T08:37:19.974182+0000","last_active":"2024-02-10T08:37:49.980554+0000","last_peered":"2024-02-10T08:37:49.980554+0000","last_clean":"2024-02-10T08:37:49.980554+0000","last_became_active":"2024-02-10T08:37:19.973885+0000","last_became_peered":"2024-02-10T08:37:19.973885+0000","last_unstale":"2024-02-10T08:37:49.980554+0000","last_undegraded":"2024-02-10T08:37:49.980554+0000","last_fullsized":"2024-02-10T08:37:49.980554+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.d","version":"153'3","reported_seq":46,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970535+0000","last_change":"2024-02-10T08:37:19.973940+0000","last_active":"2024-02-10T08:37:49.970535+0000","last_peered":"2024-02-10T08:37:49.970535+0000","last_clean":"2024-02-10T08:37:49.970535+0000","last_became_active":"2024-02-10T08:37:19.973610+0000","last_became_peered":"2024-02-10T08:37:19.973610+0000","last_unstale":"2024-02-10T08:37:49.970535+0000","last_undegraded":"2024-02-10T08:37:49.970535+0000","last_fullsized":"2024-02-10T08:37:49.970535+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.e","version":"153'5","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.978174+0000","last_change":"2024-02-10T08:37:19.977529+0000","last_active":"2024-02-10T08:37:49.978174+0000","last_peered":"2024-02-10T08:37:49.978174+0000","last_clean":"2024-02-10T08:37:49.978174+0000","last_became_active":"2024-02-10T08:37:19.977277+0000","last_became_peered":"2024-02-10T08:37:19.977277+0000","last_unstale":"2024-02-10T08:37:49.978174+0000","last_undegraded":"2024-02-10T08:37:49.978174+0000","last_fullsized":"2024-02-10T08:37:49.978174+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.f","version":"153'6","reported_seq":50,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969691+0000","last_change":"2024-02-10T08:37:19.974722+0000","last_active":"2024-02-10T08:37:49.969691+0000","last_peered":"2024-02-10T08:37:49.969691+0000","last_clean":"2024-02-10T08:37:49.969691+0000","last_became_active":"2024-02-10T08:37:19.974437+0000","last_became_peered":"2024-02-10T08:37:19.974437+0000","last_unstale":"2024-02-10T08:37:49.969691+0000","last_undegraded":"2024-02-10T08:37:49.969691+0000","last_fullsized":"2024-02-10T08:37:49.969691+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.10","version":"153'11","reported_seq":59,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965497+0000","last_change":"2024-02-10T08:37:19.978870+0000","last_active":"2024-02-10T08:37:49.965497+0000","last_peered":"2024-02-10T08:37:49.965497+0000","last_clean":"2024-02-10T08:37:49.965497+0000","last_became_active":"2024-02-10T08:37:19.978206+0000","last_became_peered":"2024-02-10T08:37:19.978206+0000","last_unstale":"2024-02-10T08:37:49.965497+0000","last_undegraded":"2024-02-10T08:37:49.965497+0000","last_fullsized":"2024-02-10T08:37:49.965497+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.11","version":"153'14","reported_seq":65,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958686+0000","last_change":"2024-02-10T08:37:19.978349+0000","last_active":"2024-02-10T08:37:49.958686+0000","last_peered":"2024-02-10T08:37:49.958686+0000","last_clean":"2024-02-10T08:37:49.958686+0000","last_became_active":"2024-02-10T08:37:19.978062+0000","last_became_peered":"2024-02-10T08:37:19.978062+0000","last_unstale":"2024-02-10T08:37:49.958686+0000","last_undegraded":"2024-02-10T08:37:49.958686+0000","last_fullsized":"2024-02-10T08:37:49.958686+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":145,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.12","version":"153'16","reported_seq":70,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965885+0000","last_change":"2024-02-10T08:37:19.976162+0000","last_active":"2024-02-10T08:37:49.965885+0000","last_peered":"2024-02-10T08:37:49.965885+0000","last_clean":"2024-02-10T08:37:49.965885+0000","last_became_active":"2024-02-10T08:37:19.975883+0000","last_became_peered":"2024-02-10T08:37:19.975883+0000","last_unstale":"2024-02-10T08:37:49.965885+0000","last_undegraded":"2024-02-10T08:37:49.965885+0000","last_fullsized":"2024-02-10T08:37:49.965885+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":16,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.13","version":"153'7","reported_seq":52,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972837+0000","last_change":"2024-02-10T08:37:19.968238+0000","last_active":"2024-02-10T08:37:49.972837+0000","last_peered":"2024-02-10T08:37:49.972837+0000","last_clean":"2024-02-10T08:37:49.972837+0000","last_became_active":"2024-02-10T08:37:19.967850+0000","last_became_peered":"2024-02-10T08:37:19.967850+0000","last_unstale":"2024-02-10T08:37:49.972837+0000","last_undegraded":"2024-02-10T08:37:49.972837+0000","last_fullsized":"2024-02-10T08:37:49.972837+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.14","version":"153'2","reported_seq":44,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980484+0000","last_change":"2024-02-10T08:37:19.981261+0000","last_active":"2024-02-10T08:37:49.980484+0000","last_peered":"2024-02-10T08:37:49.980484+0000","last_clean":"2024-02-10T08:37:49.980484+0000","last_became_active":"2024-02-10T08:37:19.980963+0000","last_became_peered":"2024-02-10T08:37:19.980963+0000","last_unstale":"2024-02-10T08:37:49.980484+0000","last_undegraded":"2024-02-10T08:37:49.980484+0000","last_fullsized":"2024-02-10T08:37:49.980484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.15","version":"153'2","reported_seq":43,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960411+0000","last_change":"2024-02-10T08:37:19.974402+0000","last_active":"2024-02-10T08:37:49.960411+0000","last_peered":"2024-02-10T08:37:49.960411+0000","last_clean":"2024-02-10T08:37:49.960411+0000","last_became_active":"2024-02-10T08:37:19.974111+0000","last_became_peered":"2024-02-10T08:37:19.974111+0000","last_unstale":"2024-02-10T08:37:49.960411+0000","last_undegraded":"2024-02-10T08:37:49.960411+0000","last_fullsized":"2024-02-10T08:37:49.960411+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.16","version":"153'10","reported_seq":57,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968452+0000","last_change":"2024-02-10T08:37:19.965323+0000","last_active":"2024-02-10T08:37:49.968452+0000","last_peered":"2024-02-10T08:37:49.968452+0000","last_clean":"2024-02-10T08:37:49.968452+0000","last_became_active":"2024-02-10T08:37:19.965052+0000","last_became_peered":"2024-02-10T08:37:19.965052+0000","last_unstale":"2024-02-10T08:37:49.968452+0000","last_undegraded":"2024-02-10T08:37:49.968452+0000","last_fullsized":"2024-02-10T08:37:49.968452+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.17","version":"153'5","reported_seq":49,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.979590+0000","last_change":"2024-02-10T08:37:19.972928+0000","last_active":"2024-02-10T08:37:49.979590+0000","last_peered":"2024-02-10T08:37:49.979590+0000","last_clean":"2024-02-10T08:37:49.979590+0000","last_became_active":"2024-02-10T08:37:19.972613+0000","last_became_peered":"2024-02-10T08:37:19.972613+0000","last_unstale":"2024-02-10T08:37:49.979590+0000","last_undegraded":"2024-02-10T08:37:49.979590+0000","last_fullsized":"2024-02-10T08:37:49.979590+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.18","version":"153'4","reported_seq":47,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965414+0000","last_change":"2024-02-10T08:37:19.970569+0000","last_active":"2024-02-10T08:37:49.965414+0000","last_peered":"2024-02-10T08:37:49.965414+0000","last_clean":"2024-02-10T08:37:49.965414+0000","last_became_active":"2024-02-10T08:37:19.970250+0000","last_became_peered":"2024-02-10T08:37:19.970250+0000","last_unstale":"2024-02-10T08:37:49.965414+0000","last_undegraded":"2024-02-10T08:37:49.965414+0000","last_fullsized":"2024-02-10T08:37:49.965414+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.19","version":"153'6","reported_seq":51,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958586+0000","last_change":"2024-02-10T08:37:19.979642+0000","last_active":"2024-02-10T08:37:49.958586+0000","last_peered":"2024-02-10T08:37:49.958586+0000","last_clean":"2024-02-10T08:37:49.958586+0000","last_became_active":"2024-02-10T08:37:19.979331+0000","last_became_peered":"2024-02-10T08:37:19.979331+0000","last_unstale":"2024-02-10T08:37:49.958586+0000","last_undegraded":"2024-02-10T08:37:49.958586+0000","last_fullsized":"2024-02-10T08:37:49.958586+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.1a","version":"153'4","reported_seq":46,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962107+0000","last_change":"2024-02-10T08:37:19.974654+0000","last_active":"2024-02-10T08:37:49.962107+0000","last_peered":"2024-02-10T08:37:49.962107+0000","last_clean":"2024-02-10T08:37:49.962107+0000","last_became_active":"2024-02-10T08:37:19.974362+0000","last_became_peered":"2024-02-10T08:37:19.974362+0000","last_unstale":"2024-02-10T08:37:49.962107+0000","last_undegraded":"2024-02-10T08:37:49.962107+0000","last_fullsized":"2024-02-10T08:37:49.962107+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.1b","version":"153'10","reported_seq":58,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.973248+0000","last_change":"2024-02-10T08:37:19.980003+0000","last_active":"2024-02-10T08:37:49.973248+0000","last_peered":"2024-02-10T08:37:49.973248+0000","last_clean":"2024-02-10T08:37:49.973248+0000","last_became_active":"2024-02-10T08:37:19.979527+0000","last_became_peered":"2024-02-10T08:37:19.979527+0000","last_unstale":"2024-02-10T08:37:49.973248+0000","last_undegraded":"2024-02-10T08:37:49.973248+0000","last_fullsized":"2024-02-10T08:37:49.973248+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1c","version":"153'7","reported_seq":54,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976667+0000","last_change":"2024-02-10T08:37:19.970573+0000","last_active":"2024-02-10T08:37:49.976667+0000","last_peered":"2024-02-10T08:37:49.976667+0000","last_clean":"2024-02-10T08:37:49.976667+0000","last_became_active":"2024-02-10T08:37:19.970263+0000","last_became_peered":"2024-02-10T08:37:19.970263+0000","last_unstale":"2024-02-10T08:37:49.976667+0000","last_undegraded":"2024-02-10T08:37:49.976667+0000","last_fullsized":"2024-02-10T08:37:49.976667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1d","version":"153'2","reported_seq":44,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964940+0000","last_change":"2024-02-10T08:37:19.968330+0000","last_active":"2024-02-10T08:37:49.964940+0000","last_peered":"2024-02-10T08:37:49.964940+0000","last_clean":"2024-02-10T08:37:49.964940+0000","last_became_active":"2024-02-10T08:37:19.968018+0000","last_became_peered":"2024-02-10T08:37:19.968018+0000","last_unstale":"2024-02-10T08:37:49.964940+0000","last_undegraded":"2024-02-10T08:37:49.964940+0000","last_fullsized":"2024-02-10T08:37:49.964940+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1e","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972181+0000","last_change":"2024-02-10T08:37:19.973173+0000","last_active":"2024-02-10T08:37:49.972181+0000","last_peered":"2024-02-10T08:37:49.972181+0000","last_clean":"2024-02-10T08:37:49.972181+0000","last_became_active":"2024-02-10T08:37:19.972861+0000","last_became_peered":"2024-02-10T08:37:19.972861+0000","last_unstale":"2024-02-10T08:37:49.972181+0000","last_undegraded":"2024-02-10T08:37:49.972181+0000","last_fullsized":"2024-02-10T08:37:49.972181+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1f","version":"153'7","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.979509+0000","last_change":"2024-02-10T08:37:19.973710+0000","last_active":"2024-02-10T08:37:49.979509+0000","last_peered":"2024-02-10T08:37:49.979509+0000","last_clean":"2024-02-10T08:37:49.979509+0000","last_became_active":"2024-02-10T08:37:19.972842+0000","last_became_peered":"2024-02-10T08:37:19.972842+0000","last_unstale":"2024-02-10T08:37:49.979509+0000","last_undegraded":"2024-02-10T08:37:49.979509+0000","last_fullsized":"2024-02-10T08:37:49.979509+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:18.927243+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.927243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.3","version":"150'18","reported_seq":89,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.959200+0000","last_change":"2024-02-10T08:37:14.030514+0000","last_active":"2024-02-10T08:37:49.959200+0000","last_peered":"2024-02-10T08:37:49.959200+0000","last_clean":"2024-02-10T08:37:49.959200+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:37:49.959200+0000","last_undegraded":"2024-02-10T08:37:49.959200+0000","last_fullsized":"2024-02-10T08:37:49.959200+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"150'9","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962370+0000","last_change":"2024-02-10T08:37:14.007544+0000","last_active":"2024-02-10T08:37:49.962370+0000","last_peered":"2024-02-10T08:37:49.962370+0000","last_clean":"2024-02-10T08:37:49.962370+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:37:49.962370+0000","last_undegraded":"2024-02-10T08:37:49.962370+0000","last_fullsized":"2024-02-10T08:37:49.962370+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"150'18","reported_seq":88,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962121+0000","last_change":"2024-02-10T08:37:14.005032+0000","last_active":"2024-02-10T08:37:49.962121+0000","last_peered":"2024-02-10T08:37:49.962121+0000","last_clean":"2024-02-10T08:37:49.962121+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:37:49.962121+0000","last_undegraded":"2024-02-10T08:37:49.962121+0000","last_fullsized":"2024-02-10T08:37:49.962121+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"150'54","reported_seq":138,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964830+0000","last_change":"2024-02-10T08:37:14.044939+0000","last_active":"2024-02-10T08:37:49.964830+0000","last_peered":"2024-02-10T08:37:49.964830+0000","last_clean":"2024-02-10T08:37:49.964830+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:37:49.964830+0000","last_undegraded":"2024-02-10T08:37:49.964830+0000","last_fullsized":"2024-02-10T08:37:49.964830+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"150'18","reported_seq":70,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963451+0000","last_change":"2024-02-10T08:37:14.030045+0000","last_active":"2024-02-10T08:37:49.963451+0000","last_peered":"2024-02-10T08:37:49.963451+0000","last_clean":"2024-02-10T08:37:49.963451+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:37:49.963451+0000","last_undegraded":"2024-02-10T08:37:49.963451+0000","last_fullsized":"2024-02-10T08:37:49.963451+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"150'36","reported_seq":107,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969566+0000","last_change":"2024-02-10T08:37:14.056078+0000","last_active":"2024-02-10T08:37:49.969566+0000","last_peered":"2024-02-10T08:37:49.969566+0000","last_clean":"2024-02-10T08:37:49.969566+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:37:49.969566+0000","last_undegraded":"2024-02-10T08:37:49.969566+0000","last_fullsized":"2024-02-10T08:37:49.969566+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"150'18","reported_seq":100,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968610+0000","last_change":"2024-02-10T08:37:14.029759+0000","last_active":"2024-02-10T08:37:49.968610+0000","last_peered":"2024-02-10T08:37:49.968610+0000","last_clean":"2024-02-10T08:37:49.968610+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:37:49.968610+0000","last_undegraded":"2024-02-10T08:37:49.968610+0000","last_fullsized":"2024-02-10T08:37:49.968610+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"150'9","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966936+0000","last_change":"2024-02-10T08:37:14.030358+0000","last_active":"2024-02-10T08:37:49.966936+0000","last_peered":"2024-02-10T08:37:49.966936+0000","last_clean":"2024-02-10T08:37:49.966936+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:37:49.966936+0000","last_undegraded":"2024-02-10T08:37:49.966936+0000","last_fullsized":"2024-02-10T08:37:49.966936+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"150'56","reported_seq":128,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.974298+0000","last_change":"2024-02-10T08:37:16.060525+0000","last_active":"2024-02-10T08:37:49.974298+0000","last_peered":"2024-02-10T08:37:49.974298+0000","last_clean":"2024-02-10T08:37:49.974298+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:37:49.974298+0000","last_undegraded":"2024-02-10T08:37:49.974298+0000","last_fullsized":"2024-02-10T08:37:49.974298+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"150'14","reported_seq":89,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962278+0000","last_change":"2024-02-10T08:37:16.049354+0000","last_active":"2024-02-10T08:37:49.962278+0000","last_peered":"2024-02-10T08:37:49.962278+0000","last_clean":"2024-02-10T08:37:49.962278+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:37:49.962278+0000","last_undegraded":"2024-02-10T08:37:49.962278+0000","last_fullsized":"2024-02-10T08:37:49.962278+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"150'54","reported_seq":110,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958823+0000","last_change":"2024-02-10T08:37:14.030841+0000","last_active":"2024-02-10T08:37:49.958823+0000","last_peered":"2024-02-10T08:37:49.958823+0000","last_clean":"2024-02-10T08:37:49.958823+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:37:49.958823+0000","last_undegraded":"2024-02-10T08:37:49.958823+0000","last_fullsized":"2024-02-10T08:37:49.958823+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"150'45","reported_seq":120,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960862+0000","last_change":"2024-02-10T08:37:14.028055+0000","last_active":"2024-02-10T08:37:49.960862+0000","last_peered":"2024-02-10T08:37:49.960862+0000","last_clean":"2024-02-10T08:37:49.960862+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:37:49.960862+0000","last_undegraded":"2024-02-10T08:37:49.960862+0000","last_fullsized":"2024-02-10T08:37:49.960862+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"150'46","reported_seq":109,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976769+0000","last_change":"2024-02-10T08:37:14.036091+0000","last_active":"2024-02-10T08:37:49.976769+0000","last_peered":"2024-02-10T08:37:49.976769+0000","last_clean":"2024-02-10T08:37:49.976769+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:37:49.976769+0000","last_undegraded":"2024-02-10T08:37:49.976769+0000","last_fullsized":"2024-02-10T08:37:49.976769+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":26,"num_object_clones":20,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"150'27","reported_seq":80,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972399+0000","last_change":"2024-02-10T08:37:14.035113+0000","last_active":"2024-02-10T08:37:49.972399+0000","last_peered":"2024-02-10T08:37:49.972399+0000","last_clean":"2024-02-10T08:37:49.972399+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:37:49.972399+0000","last_undegraded":"2024-02-10T08:37:49.972399+0000","last_fullsized":"2024-02-10T08:37:49.972399+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"150'27","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.973317+0000","last_change":"2024-02-10T08:37:14.035480+0000","last_active":"2024-02-10T08:37:49.973317+0000","last_peered":"2024-02-10T08:37:49.973317+0000","last_clean":"2024-02-10T08:37:49.973317+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:37:49.973317+0000","last_undegraded":"2024-02-10T08:37:49.973317+0000","last_fullsized":"2024-02-10T08:37:49.973317+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"150'36","reported_seq":111,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969235+0000","last_change":"2024-02-10T08:37:14.011968+0000","last_active":"2024-02-10T08:37:49.969235+0000","last_peered":"2024-02-10T08:37:49.969235+0000","last_clean":"2024-02-10T08:37:49.969235+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:37:49.969235+0000","last_undegraded":"2024-02-10T08:37:49.969235+0000","last_fullsized":"2024-02-10T08:37:49.969235+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"150'27","reported_seq":99,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970646+0000","last_change":"2024-02-10T08:37:14.013065+0000","last_active":"2024-02-10T08:37:49.970646+0000","last_peered":"2024-02-10T08:37:49.970646+0000","last_clean":"2024-02-10T08:37:49.970646+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:37:49.970646+0000","last_undegraded":"2024-02-10T08:37:49.970646+0000","last_fullsized":"2024-02-10T08:37:49.970646+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"150'54","reported_seq":135,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970651+0000","last_change":"2024-02-10T08:37:14.015667+0000","last_active":"2024-02-10T08:37:49.970651+0000","last_peered":"2024-02-10T08:37:49.970651+0000","last_clean":"2024-02-10T08:37:49.970651+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:37:49.970651+0000","last_undegraded":"2024-02-10T08:37:49.970651+0000","last_fullsized":"2024-02-10T08:37:49.970651+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"150'54","reported_seq":123,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968547+0000","last_change":"2024-02-10T08:37:14.025959+0000","last_active":"2024-02-10T08:37:49.968547+0000","last_peered":"2024-02-10T08:37:49.968547+0000","last_clean":"2024-02-10T08:37:49.968547+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:37:49.968547+0000","last_undegraded":"2024-02-10T08:37:49.968547+0000","last_fullsized":"2024-02-10T08:37:49.968547+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"150'36","reported_seq":118,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962780+0000","last_change":"2024-02-10T08:37:14.016722+0000","last_active":"2024-02-10T08:37:49.962780+0000","last_peered":"2024-02-10T08:37:49.962780+0000","last_clean":"2024-02-10T08:37:49.962780+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:37:49.962780+0000","last_undegraded":"2024-02-10T08:37:49.962780+0000","last_fullsized":"2024-02-10T08:37:49.962780+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"150'45","reported_seq":116,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977414+0000","last_change":"2024-02-10T08:37:14.017461+0000","last_active":"2024-02-10T08:37:49.977414+0000","last_peered":"2024-02-10T08:37:49.977414+0000","last_clean":"2024-02-10T08:37:49.977414+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:37:49.977414+0000","last_undegraded":"2024-02-10T08:37:49.977414+0000","last_fullsized":"2024-02-10T08:37:49.977414+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"150'32","reported_seq":121,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972303+0000","last_change":"2024-02-10T08:37:16.048167+0000","last_active":"2024-02-10T08:37:49.972303+0000","last_peered":"2024-02-10T08:37:49.972303+0000","last_clean":"2024-02-10T08:37:49.972303+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:37:49.972303+0000","last_undegraded":"2024-02-10T08:37:49.972303+0000","last_fullsized":"2024-02-10T08:37:49.972303+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":161,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"150'9","reported_seq":82,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968520+0000","last_change":"2024-02-10T08:37:14.029348+0000","last_active":"2024-02-10T08:37:49.968520+0000","last_peered":"2024-02-10T08:37:49.968520+0000","last_clean":"2024-02-10T08:37:49.968520+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:37:49.968520+0000","last_undegraded":"2024-02-10T08:37:49.968520+0000","last_fullsized":"2024-02-10T08:37:49.968520+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"150'18","reported_seq":88,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970594+0000","last_change":"2024-02-10T08:37:14.048854+0000","last_active":"2024-02-10T08:37:49.970594+0000","last_peered":"2024-02-10T08:37:49.970594+0000","last_clean":"2024-02-10T08:37:49.970594+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:37:49.970594+0000","last_undegraded":"2024-02-10T08:37:49.970594+0000","last_fullsized":"2024-02-10T08:37:49.970594+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"150'36","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962804+0000","last_change":"2024-02-10T08:37:14.028568+0000","last_active":"2024-02-10T08:37:49.962804+0000","last_peered":"2024-02-10T08:37:49.962804+0000","last_clean":"2024-02-10T08:37:49.962804+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:37:49.962804+0000","last_undegraded":"2024-02-10T08:37:49.962804+0000","last_fullsized":"2024-02-10T08:37:49.962804+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"150'36","reported_seq":100,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958912+0000","last_change":"2024-02-10T08:37:14.045308+0000","last_active":"2024-02-10T08:37:49.958912+0000","last_peered":"2024-02-10T08:37:49.958912+0000","last_clean":"2024-02-10T08:37:49.958912+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:37:49.958912+0000","last_undegraded":"2024-02-10T08:37:49.958912+0000","last_fullsized":"2024-02-10T08:37:49.958912+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"150'18","reported_seq":101,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.958910+0000","last_change":"2024-02-10T08:37:14.031039+0000","last_active":"2024-02-10T08:37:49.958910+0000","last_peered":"2024-02-10T08:37:49.958910+0000","last_clean":"2024-02-10T08:37:49.958910+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:37:49.958910+0000","last_undegraded":"2024-02-10T08:37:49.958910+0000","last_fullsized":"2024-02-10T08:37:49.958910+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"150'18","reported_seq":73,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960779+0000","last_change":"2024-02-10T08:37:14.027697+0000","last_active":"2024-02-10T08:37:49.960779+0000","last_peered":"2024-02-10T08:37:49.960779+0000","last_clean":"2024-02-10T08:37:49.960779+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:37:49.960779+0000","last_undegraded":"2024-02-10T08:37:49.960779+0000","last_fullsized":"2024-02-10T08:37:49.960779+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"150'18","reported_seq":102,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980336+0000","last_change":"2024-02-10T08:37:14.012763+0000","last_active":"2024-02-10T08:37:49.980336+0000","last_peered":"2024-02-10T08:37:49.980336+0000","last_clean":"2024-02-10T08:37:49.980336+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:37:49.980336+0000","last_undegraded":"2024-02-10T08:37:49.980336+0000","last_fullsized":"2024-02-10T08:37:49.980336+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"150'9","reported_seq":63,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960525+0000","last_change":"2024-02-10T08:37:14.044672+0000","last_active":"2024-02-10T08:37:49.960525+0000","last_peered":"2024-02-10T08:37:49.960525+0000","last_clean":"2024-02-10T08:37:49.960525+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:37:49.960525+0000","last_undegraded":"2024-02-10T08:37:49.960525+0000","last_fullsized":"2024-02-10T08:37:49.960525+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"150'9","reported_seq":66,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969999+0000","last_change":"2024-02-10T08:37:14.038234+0000","last_active":"2024-02-10T08:37:49.969999+0000","last_peered":"2024-02-10T08:37:49.969999+0000","last_clean":"2024-02-10T08:37:49.969999+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:37:49.969999+0000","last_undegraded":"2024-02-10T08:37:49.969999+0000","last_fullsized":"2024-02-10T08:37:49.969999+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"150'18","reported_seq":93,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969052+0000","last_change":"2024-02-10T08:37:14.011754+0000","last_active":"2024-02-10T08:37:49.969052+0000","last_peered":"2024-02-10T08:37:49.969052+0000","last_clean":"2024-02-10T08:37:49.969052+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:37:49.969052+0000","last_undegraded":"2024-02-10T08:37:49.969052+0000","last_fullsized":"2024-02-10T08:37:49.969052+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970186+0000","last_change":"2024-02-10T08:37:10.250275+0000","last_active":"2024-02-10T08:37:49.970186+0000","last_peered":"2024-02-10T08:37:49.970186+0000","last_clean":"2024-02-10T08:37:49.970186+0000","last_became_active":"2024-02-10T08:37:10.249567+0000","last_became_peered":"2024-02-10T08:37:10.249567+0000","last_unstale":"2024-02-10T08:37:49.970186+0000","last_undegraded":"2024-02-10T08:37:49.970186+0000","last_fullsized":"2024-02-10T08:37:49.970186+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":218,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966914+0000","last_change":"2024-02-10T08:36:58.130213+0000","last_active":"2024-02-10T08:37:49.966914+0000","last_peered":"2024-02-10T08:37:49.966914+0000","last_clean":"2024-02-10T08:37:49.966914+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:37:49.966914+0000","last_undegraded":"2024-02-10T08:37:49.966914+0000","last_fullsized":"2024-02-10T08:37:49.966914+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:16.169618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.169618+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":170,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975781+0000","last_change":"2024-02-10T08:37:07.840809+0000","last_active":"2024-02-10T08:37:49.975781+0000","last_peered":"2024-02-10T08:37:49.975781+0000","last_clean":"2024-02-10T08:37:49.975781+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:37:49.975781+0000","last_undegraded":"2024-02-10T08:37:49.975781+0000","last_fullsized":"2024-02-10T08:37:49.975781+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:37:07.840738+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.840738+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969357+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:37:49.969357+0000","last_peered":"2024-02-10T08:37:49.969357+0000","last_clean":"2024-02-10T08:37:49.969357+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:37:49.969357+0000","last_undegraded":"2024-02-10T08:37:49.969357+0000","last_fullsized":"2024-02-10T08:37:49.969357+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":148,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.982740+0000","last_change":"2024-02-10T08:37:44.597397+0000","last_active":"2024-02-10T08:37:49.982740+0000","last_peered":"2024-02-10T08:37:49.982740+0000","last_clean":"2024-02-10T08:37:49.982740+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:37:49.982740+0000","last_undegraded":"2024-02-10T08:37:49.982740+0000","last_fullsized":"2024-02-10T08:37:49.982740+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.597335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.597335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969370+0000","last_change":"2024-02-10T08:36:37.051266+0000","last_active":"2024-02-10T08:37:49.969370+0000","last_peered":"2024-02-10T08:37:49.969370+0000","last_clean":"2024-02-10T08:37:49.969370+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:37:49.969370+0000","last_undegraded":"2024-02-10T08:37:49.969370+0000","last_fullsized":"2024-02-10T08:37:49.969370+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"103'5","reported_seq":101,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960766+0000","last_change":"2024-02-10T08:37:11.012191+0000","last_active":"2024-02-10T08:37:49.960766+0000","last_peered":"2024-02-10T08:37:49.960766+0000","last_clean":"2024-02-10T08:37:49.960766+0000","last_became_active":"2024-02-10T08:37:10.219488+0000","last_became_peered":"2024-02-10T08:37:10.219488+0000","last_unstale":"2024-02-10T08:37:49.960766+0000","last_undegraded":"2024-02-10T08:37:49.960766+0000","last_fullsized":"2024-02-10T08:37:49.960766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":117,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.974567+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:37:49.974567+0000","last_peered":"2024-02-10T08:37:49.974567+0000","last_clean":"2024-02-10T08:37:49.974567+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:37:49.974567+0000","last_undegraded":"2024-02-10T08:37:49.974567+0000","last_fullsized":"2024-02-10T08:37:49.974567+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961299+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:37:49.961299+0000","last_peered":"2024-02-10T08:37:49.961299+0000","last_clean":"2024-02-10T08:37:49.961299+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:37:49.961299+0000","last_undegraded":"2024-02-10T08:37:49.961299+0000","last_fullsized":"2024-02-10T08:37:49.961299+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":159,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967852+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:37:49.967852+0000","last_peered":"2024-02-10T08:37:49.967852+0000","last_clean":"2024-02-10T08:37:49.967852+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:37:49.967852+0000","last_undegraded":"2024-02-10T08:37:49.967852+0000","last_fullsized":"2024-02-10T08:37:49.967852+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":58,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966656+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:37:49.966656+0000","last_peered":"2024-02-10T08:37:49.966656+0000","last_clean":"2024-02-10T08:37:49.966656+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:37:49.966656+0000","last_undegraded":"2024-02-10T08:37:49.966656+0000","last_fullsized":"2024-02-10T08:37:49.966656+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980890+0000","last_change":"2024-02-10T08:36:37.033242+0000","last_active":"2024-02-10T08:37:49.980890+0000","last_peered":"2024-02-10T08:37:49.980890+0000","last_clean":"2024-02-10T08:37:49.980890+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:37:49.980890+0000","last_undegraded":"2024-02-10T08:37:49.980890+0000","last_fullsized":"2024-02-10T08:37:49.980890+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.17","version":"103'5","reported_seq":101,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.973048+0000","last_change":"2024-02-10T08:37:10.688144+0000","last_active":"2024-02-10T08:37:49.973048+0000","last_peered":"2024-02-10T08:37:49.973048+0000","last_clean":"2024-02-10T08:37:49.973048+0000","last_became_active":"2024-02-10T08:37:10.259720+0000","last_became_peered":"2024-02-10T08:37:10.259720+0000","last_unstale":"2024-02-10T08:37:49.973048+0000","last_undegraded":"2024-02-10T08:37:49.973048+0000","last_fullsized":"2024-02-10T08:37:49.973048+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":174,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971239+0000","last_change":"2024-02-10T08:37:39.547805+0000","last_active":"2024-02-10T08:37:49.971239+0000","last_peered":"2024-02-10T08:37:49.971239+0000","last_clean":"2024-02-10T08:37:49.971239+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:37:49.971239+0000","last_undegraded":"2024-02-10T08:37:49.971239+0000","last_fullsized":"2024-02-10T08:37:49.971239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:37:39.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.547736+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970052+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:37:49.970052+0000","last_peered":"2024-02-10T08:37:49.970052+0000","last_clean":"2024-02-10T08:37:49.970052+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:37:49.970052+0000","last_undegraded":"2024-02-10T08:37:49.970052+0000","last_fullsized":"2024-02-10T08:37:49.970052+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":163,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.974432+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:37:49.974432+0000","last_peered":"2024-02-10T08:37:49.974432+0000","last_clean":"2024-02-10T08:37:49.974432+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:37:49.974432+0000","last_undegraded":"2024-02-10T08:37:49.974432+0000","last_fullsized":"2024-02-10T08:37:49.974432+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963476+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:37:49.963476+0000","last_peered":"2024-02-10T08:37:49.963476+0000","last_clean":"2024-02-10T08:37:49.963476+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:37:49.963476+0000","last_undegraded":"2024-02-10T08:37:49.963476+0000","last_fullsized":"2024-02-10T08:37:49.963476+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.978325+0000","last_change":"2024-02-10T08:36:37.021497+0000","last_active":"2024-02-10T08:37:49.978325+0000","last_peered":"2024-02-10T08:37:49.978325+0000","last_clean":"2024-02-10T08:37:49.978325+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:37:49.978325+0000","last_undegraded":"2024-02-10T08:37:49.978325+0000","last_fullsized":"2024-02-10T08:37:49.978325+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975858+0000","last_change":"2024-02-10T08:37:10.200689+0000","last_active":"2024-02-10T08:37:49.975858+0000","last_peered":"2024-02-10T08:37:49.975858+0000","last_clean":"2024-02-10T08:37:49.975858+0000","last_became_active":"2024-02-10T08:37:10.200347+0000","last_became_peered":"2024-02-10T08:37:10.200347+0000","last_unstale":"2024-02-10T08:37:49.975858+0000","last_undegraded":"2024-02-10T08:37:49.975858+0000","last_fullsized":"2024-02-10T08:37:49.975858+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":171,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.586980+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:37:50.586980+0000","last_peered":"2024-02-10T08:37:50.586980+0000","last_clean":"2024-02-10T08:37:50.586980+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:37:50.586980+0000","last_undegraded":"2024-02-10T08:37:50.586980+0000","last_fullsized":"2024-02-10T08:37:50.586980+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970012+0000","last_change":"2024-02-10T08:37:38.545880+0000","last_active":"2024-02-10T08:37:49.970012+0000","last_peered":"2024-02-10T08:37:49.970012+0000","last_clean":"2024-02-10T08:37:49.970012+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:37:49.970012+0000","last_undegraded":"2024-02-10T08:37:49.970012+0000","last_fullsized":"2024-02-10T08:37:49.970012+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":163,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963762+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:37:49.963762+0000","last_peered":"2024-02-10T08:37:49.963762+0000","last_clean":"2024-02-10T08:37:49.963762+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:37:49.963762+0000","last_undegraded":"2024-02-10T08:37:49.963762+0000","last_fullsized":"2024-02-10T08:37:49.963762+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971358+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:37:49.971358+0000","last_peered":"2024-02-10T08:37:49.971358+0000","last_clean":"2024-02-10T08:37:49.971358+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:37:49.971358+0000","last_undegraded":"2024-02-10T08:37:49.971358+0000","last_fullsized":"2024-02-10T08:37:49.971358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962285+0000","last_change":"2024-02-10T08:36:37.075652+0000","last_active":"2024-02-10T08:37:49.962285+0000","last_peered":"2024-02-10T08:37:49.962285+0000","last_clean":"2024-02-10T08:37:49.962285+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:37:49.962285+0000","last_undegraded":"2024-02-10T08:37:49.962285+0000","last_fullsized":"2024-02-10T08:37:49.962285+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"103'7","reported_seq":73,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962208+0000","last_change":"2024-02-10T08:37:10.141080+0000","last_active":"2024-02-10T08:37:49.962208+0000","last_peered":"2024-02-10T08:37:49.962208+0000","last_clean":"2024-02-10T08:37:49.962208+0000","last_became_active":"2024-02-10T08:37:10.115979+0000","last_became_peered":"2024-02-10T08:37:10.115979+0000","last_unstale":"2024-02-10T08:37:49.962208+0000","last_undegraded":"2024-02-10T08:37:49.962208+0000","last_fullsized":"2024-02-10T08:37:49.962208+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":153,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967569+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:37:49.967569+0000","last_peered":"2024-02-10T08:37:49.967569+0000","last_clean":"2024-02-10T08:37:49.967569+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:37:49.967569+0000","last_undegraded":"2024-02-10T08:37:49.967569+0000","last_fullsized":"2024-02-10T08:37:49.967569+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963189+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:37:49.963189+0000","last_peered":"2024-02-10T08:37:49.963189+0000","last_clean":"2024-02-10T08:37:49.963189+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:37:49.963189+0000","last_undegraded":"2024-02-10T08:37:49.963189+0000","last_fullsized":"2024-02-10T08:37:49.963189+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":218,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968464+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:37:49.968464+0000","last_peered":"2024-02-10T08:37:49.968464+0000","last_clean":"2024-02-10T08:37:49.968464+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:37:49.968464+0000","last_undegraded":"2024-02-10T08:37:49.968464+0000","last_fullsized":"2024-02-10T08:37:49.968464+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963718+0000","last_change":"2024-02-10T08:36:13.778764+0000","last_active":"2024-02-10T08:37:49.963718+0000","last_peered":"2024-02-10T08:37:49.963718+0000","last_clean":"2024-02-10T08:37:49.963718+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:37:49.963718+0000","last_undegraded":"2024-02-10T08:37:49.963718+0000","last_fullsized":"2024-02-10T08:37:49.963718+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.778673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.778673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965932+0000","last_change":"2024-02-10T08:37:10.263630+0000","last_active":"2024-02-10T08:37:49.965932+0000","last_peered":"2024-02-10T08:37:49.965932+0000","last_clean":"2024-02-10T08:37:49.965932+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:37:49.965932+0000","last_undegraded":"2024-02-10T08:37:49.965932+0000","last_fullsized":"2024-02-10T08:37:49.965932+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963744+0000","last_change":"2024-02-10T08:37:10.183873+0000","last_active":"2024-02-10T08:37:49.963744+0000","last_peered":"2024-02-10T08:37:49.963744+0000","last_clean":"2024-02-10T08:37:49.963744+0000","last_became_active":"2024-02-10T08:37:10.183562+0000","last_became_peered":"2024-02-10T08:37:10.183562+0000","last_unstale":"2024-02-10T08:37:49.963744+0000","last_undegraded":"2024-02-10T08:37:49.963744+0000","last_fullsized":"2024-02-10T08:37:49.963744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":163,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964910+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:37:49.964910+0000","last_peered":"2024-02-10T08:37:49.964910+0000","last_clean":"2024-02-10T08:37:49.964910+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:37:49.964910+0000","last_undegraded":"2024-02-10T08:37:49.964910+0000","last_fullsized":"2024-02-10T08:37:49.964910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":146,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.959609+0000","last_change":"2024-02-10T08:36:16.423854+0000","last_active":"2024-02-10T08:37:49.959609+0000","last_peered":"2024-02-10T08:37:49.959609+0000","last_clean":"2024-02-10T08:37:49.959609+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:37:49.959609+0000","last_undegraded":"2024-02-10T08:37:49.959609+0000","last_fullsized":"2024-02-10T08:37:49.959609+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:16.423786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.423786+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":243,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963631+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:37:49.963631+0000","last_peered":"2024-02-10T08:37:49.963631+0000","last_clean":"2024-02-10T08:37:49.963631+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:37:49.963631+0000","last_undegraded":"2024-02-10T08:37:49.963631+0000","last_fullsized":"2024-02-10T08:37:49.963631+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":143,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964693+0000","last_change":"2024-02-10T08:37:24.680471+0000","last_active":"2024-02-10T08:37:49.964693+0000","last_peered":"2024-02-10T08:37:49.964693+0000","last_clean":"2024-02-10T08:37:49.964693+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:37:49.964693+0000","last_undegraded":"2024-02-10T08:37:49.964693+0000","last_fullsized":"2024-02-10T08:37:49.964693+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:24.680407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:24.680407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970803+0000","last_change":"2024-02-10T08:36:37.148745+0000","last_active":"2024-02-10T08:37:49.970803+0000","last_peered":"2024-02-10T08:37:49.970803+0000","last_clean":"2024-02-10T08:37:49.970803+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:37:49.970803+0000","last_undegraded":"2024-02-10T08:37:49.970803+0000","last_fullsized":"2024-02-10T08:37:49.970803+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.13","version":"103'9","reported_seq":109,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963017+0000","last_change":"2024-02-10T08:37:10.221042+0000","last_active":"2024-02-10T08:37:49.963017+0000","last_peered":"2024-02-10T08:37:49.963017+0000","last_clean":"2024-02-10T08:37:49.963017+0000","last_became_active":"2024-02-10T08:37:10.136194+0000","last_became_peered":"2024-02-10T08:37:10.136194+0000","last_unstale":"2024-02-10T08:37:49.963017+0000","last_undegraded":"2024-02-10T08:37:49.963017+0000","last_fullsized":"2024-02-10T08:37:49.963017+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":157,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963395+0000","last_change":"2024-02-10T08:36:11.732545+0000","last_active":"2024-02-10T08:37:49.963395+0000","last_peered":"2024-02-10T08:37:49.963395+0000","last_clean":"2024-02-10T08:37:49.963395+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:37:49.963395+0000","last_undegraded":"2024-02-10T08:37:49.963395+0000","last_fullsized":"2024-02-10T08:37:49.963395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:36:11.732461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:11.732461+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963779+0000","last_change":"2024-02-10T08:37:10.150021+0000","last_active":"2024-02-10T08:37:49.963779+0000","last_peered":"2024-02-10T08:37:49.963779+0000","last_clean":"2024-02-10T08:37:49.963779+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:37:49.963779+0000","last_undegraded":"2024-02-10T08:37:49.963779+0000","last_fullsized":"2024-02-10T08:37:49.963779+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.142541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.142541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":171,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980968+0000","last_change":"2024-02-10T08:37:21.746790+0000","last_active":"2024-02-10T08:37:49.980968+0000","last_peered":"2024-02-10T08:37:49.980968+0000","last_clean":"2024-02-10T08:37:49.980968+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:37:49.980968+0000","last_undegraded":"2024-02-10T08:37:49.980968+0000","last_fullsized":"2024-02-10T08:37:49.980968+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:37:21.746710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.746710+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971055+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:37:49.971055+0000","last_peered":"2024-02-10T08:37:49.971055+0000","last_clean":"2024-02-10T08:37:49.971055+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:37:49.971055+0000","last_undegraded":"2024-02-10T08:37:49.971055+0000","last_fullsized":"2024-02-10T08:37:49.971055+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966089+0000","last_change":"2024-02-10T08:36:37.134622+0000","last_active":"2024-02-10T08:37:49.966089+0000","last_peered":"2024-02-10T08:37:49.966089+0000","last_clean":"2024-02-10T08:37:49.966089+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:37:49.966089+0000","last_undegraded":"2024-02-10T08:37:49.966089+0000","last_fullsized":"2024-02-10T08:37:49.966089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967110+0000","last_change":"2024-02-10T08:36:37.065813+0000","last_active":"2024-02-10T08:37:49.967110+0000","last_peered":"2024-02-10T08:37:49.967110+0000","last_clean":"2024-02-10T08:37:49.967110+0000","last_became_active":"2024-02-10T08:36:37.065010+0000","last_became_peered":"2024-02-10T08:36:37.065010+0000","last_unstale":"2024-02-10T08:37:49.967110+0000","last_undegraded":"2024-02-10T08:37:49.967110+0000","last_fullsized":"2024-02-10T08:37:49.967110+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":200,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966237+0000","last_change":"2024-02-10T08:36:28.076465+0000","last_active":"2024-02-10T08:37:49.966237+0000","last_peered":"2024-02-10T08:37:49.966237+0000","last_clean":"2024-02-10T08:37:49.966237+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:37:49.966237+0000","last_undegraded":"2024-02-10T08:37:49.966237+0000","last_fullsized":"2024-02-10T08:37:49.966237+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:28.076401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.076401+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":159,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962605+0000","last_change":"2024-02-10T08:36:17.729319+0000","last_active":"2024-02-10T08:37:49.962605+0000","last_peered":"2024-02-10T08:37:49.962605+0000","last_clean":"2024-02-10T08:37:49.962605+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:37:49.962605+0000","last_undegraded":"2024-02-10T08:37:49.962605+0000","last_fullsized":"2024-02-10T08:37:49.962605+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:17.729249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.729249+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977153+0000","last_change":"2024-02-10T08:37:19.310327+0000","last_active":"2024-02-10T08:37:49.977153+0000","last_peered":"2024-02-10T08:37:49.977153+0000","last_clean":"2024-02-10T08:37:49.977153+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:37:49.977153+0000","last_undegraded":"2024-02-10T08:37:49.977153+0000","last_fullsized":"2024-02-10T08:37:49.977153+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.310252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.310252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970174+0000","last_change":"2024-02-10T08:37:09.215324+0000","last_active":"2024-02-10T08:37:49.970174+0000","last_peered":"2024-02-10T08:37:49.970174+0000","last_clean":"2024-02-10T08:37:49.970174+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:37:49.970174+0000","last_undegraded":"2024-02-10T08:37:49.970174+0000","last_fullsized":"2024-02-10T08:37:49.970174+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:09.215257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:09.215257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961442+0000","last_change":"2024-02-10T08:37:10.160633+0000","last_active":"2024-02-10T08:37:49.961442+0000","last_peered":"2024-02-10T08:37:49.961442+0000","last_clean":"2024-02-10T08:37:49.961442+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:37:49.961442+0000","last_undegraded":"2024-02-10T08:37:49.961442+0000","last_fullsized":"2024-02-10T08:37:49.961442+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"103'7","reported_seq":90,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964168+0000","last_change":"2024-02-10T08:36:37.149279+0000","last_active":"2024-02-10T08:37:49.964168+0000","last_peered":"2024-02-10T08:37:49.964168+0000","last_clean":"2024-02-10T08:37:49.964168+0000","last_became_active":"2024-02-10T08:36:37.148954+0000","last_became_peered":"2024-02-10T08:36:37.148954+0000","last_unstale":"2024-02-10T08:37:49.964168+0000","last_undegraded":"2024-02-10T08:37:49.964168+0000","last_fullsized":"2024-02-10T08:37:49.964168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":136,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962780+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:37:49.962780+0000","last_peered":"2024-02-10T08:37:49.962780+0000","last_clean":"2024-02-10T08:37:49.962780+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:37:49.962780+0000","last_undegraded":"2024-02-10T08:37:49.962780+0000","last_fullsized":"2024-02-10T08:37:49.962780+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":204,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966168+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:37:49.966168+0000","last_peered":"2024-02-10T08:37:49.966168+0000","last_clean":"2024-02-10T08:37:49.966168+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:37:49.966168+0000","last_undegraded":"2024-02-10T08:37:49.966168+0000","last_fullsized":"2024-02-10T08:37:49.966168+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":147,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960850+0000","last_change":"2024-02-10T08:36:32.450905+0000","last_active":"2024-02-10T08:37:49.960850+0000","last_peered":"2024-02-10T08:37:49.960850+0000","last_clean":"2024-02-10T08:37:49.960850+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:37:49.960850+0000","last_undegraded":"2024-02-10T08:37:49.960850+0000","last_fullsized":"2024-02-10T08:37:49.960850+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:32.450807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.450807+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963317+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:37:49.963317+0000","last_peered":"2024-02-10T08:37:49.963317+0000","last_clean":"2024-02-10T08:37:49.963317+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:37:49.963317+0000","last_undegraded":"2024-02-10T08:37:49.963317+0000","last_fullsized":"2024-02-10T08:37:49.963317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970791+0000","last_change":"2024-02-10T08:37:10.260060+0000","last_active":"2024-02-10T08:37:49.970791+0000","last_peered":"2024-02-10T08:37:49.970791+0000","last_clean":"2024-02-10T08:37:49.970791+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:37:49.970791+0000","last_undegraded":"2024-02-10T08:37:49.970791+0000","last_fullsized":"2024-02-10T08:37:49.970791+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972742+0000","last_change":"2024-02-10T08:37:10.221508+0000","last_active":"2024-02-10T08:37:49.972742+0000","last_peered":"2024-02-10T08:37:49.972742+0000","last_clean":"2024-02-10T08:37:49.972742+0000","last_became_active":"2024-02-10T08:37:10.221122+0000","last_became_peered":"2024-02-10T08:37:10.221122+0000","last_unstale":"2024-02-10T08:37:49.972742+0000","last_undegraded":"2024-02-10T08:37:49.972742+0000","last_fullsized":"2024-02-10T08:37:49.972742+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":150,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971320+0000","last_change":"2024-02-10T08:37:43.520384+0000","last_active":"2024-02-10T08:37:49.971320+0000","last_peered":"2024-02-10T08:37:49.971320+0000","last_clean":"2024-02-10T08:37:49.971320+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:37:49.971320+0000","last_undegraded":"2024-02-10T08:37:49.971320+0000","last_fullsized":"2024-02-10T08:37:49.971320+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.520307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.520307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":119,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971092+0000","last_change":"2024-02-10T08:37:10.868785+0000","last_active":"2024-02-10T08:37:49.971092+0000","last_peered":"2024-02-10T08:37:49.971092+0000","last_clean":"2024-02-10T08:37:49.971092+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:37:49.971092+0000","last_undegraded":"2024-02-10T08:37:49.971092+0000","last_fullsized":"2024-02-10T08:37:49.971092+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:15.871060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.871060+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969780+0000","last_change":"2024-02-10T08:37:10.151669+0000","last_active":"2024-02-10T08:37:49.969780+0000","last_peered":"2024-02-10T08:37:49.969780+0000","last_clean":"2024-02-10T08:37:49.969780+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:37:49.969780+0000","last_undegraded":"2024-02-10T08:37:49.969780+0000","last_fullsized":"2024-02-10T08:37:49.969780+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.048119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.048119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961580+0000","last_change":"2024-02-10T08:36:20.500433+0000","last_active":"2024-02-10T08:37:49.961580+0000","last_peered":"2024-02-10T08:37:49.961580+0000","last_clean":"2024-02-10T08:37:49.961580+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:37:49.961580+0000","last_undegraded":"2024-02-10T08:37:49.961580+0000","last_fullsized":"2024-02-10T08:37:49.961580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.500313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970061+0000","last_change":"2024-02-10T08:37:10.233957+0000","last_active":"2024-02-10T08:37:49.970061+0000","last_peered":"2024-02-10T08:37:49.970061+0000","last_clean":"2024-02-10T08:37:49.970061+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:37:49.970061+0000","last_undegraded":"2024-02-10T08:37:49.970061+0000","last_fullsized":"2024-02-10T08:37:49.970061+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"103'10","reported_seq":108,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976144+0000","last_change":"2024-02-10T08:37:10.445260+0000","last_active":"2024-02-10T08:37:49.976144+0000","last_peered":"2024-02-10T08:37:49.976144+0000","last_clean":"2024-02-10T08:37:49.976144+0000","last_became_active":"2024-02-10T08:37:10.261034+0000","last_became_peered":"2024-02-10T08:37:10.261034+0000","last_unstale":"2024-02-10T08:37:49.976144+0000","last_undegraded":"2024-02-10T08:37:49.976144+0000","last_fullsized":"2024-02-10T08:37:49.976144+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":163,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980718+0000","last_change":"2024-02-10T08:37:10.149313+0000","last_active":"2024-02-10T08:37:49.980718+0000","last_peered":"2024-02-10T08:37:49.980718+0000","last_clean":"2024-02-10T08:37:49.980718+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:37:49.980718+0000","last_undegraded":"2024-02-10T08:37:49.980718+0000","last_fullsized":"2024-02-10T08:37:49.980718+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.714301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.714301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":102,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961075+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:37:49.961075+0000","last_peered":"2024-02-10T08:37:49.961075+0000","last_clean":"2024-02-10T08:37:49.961075+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:37:49.961075+0000","last_undegraded":"2024-02-10T08:37:49.961075+0000","last_fullsized":"2024-02-10T08:37:49.961075+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":154,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963553+0000","last_change":"2024-02-10T08:37:21.277884+0000","last_active":"2024-02-10T08:37:49.963553+0000","last_peered":"2024-02-10T08:37:49.963553+0000","last_clean":"2024-02-10T08:37:49.963553+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:37:49.963553+0000","last_undegraded":"2024-02-10T08:37:49.963553+0000","last_fullsized":"2024-02-10T08:37:49.963553+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.277786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.277786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962468+0000","last_change":"2024-02-10T08:37:10.215017+0000","last_active":"2024-02-10T08:37:49.962468+0000","last_peered":"2024-02-10T08:37:49.962468+0000","last_clean":"2024-02-10T08:37:49.962468+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:37:49.962468+0000","last_undegraded":"2024-02-10T08:37:49.962468+0000","last_fullsized":"2024-02-10T08:37:49.962468+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963843+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:37:49.963843+0000","last_peered":"2024-02-10T08:37:49.963843+0000","last_clean":"2024-02-10T08:37:49.963843+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:37:49.963843+0000","last_undegraded":"2024-02-10T08:37:49.963843+0000","last_fullsized":"2024-02-10T08:37:49.963843+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970960+0000","last_change":"2024-02-10T08:37:10.153879+0000","last_active":"2024-02-10T08:37:49.970960+0000","last_peered":"2024-02-10T08:37:49.970960+0000","last_clean":"2024-02-10T08:37:49.970960+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:37:49.970960+0000","last_undegraded":"2024-02-10T08:37:49.970960+0000","last_fullsized":"2024-02-10T08:37:49.970960+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"103'2","reported_seq":66,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966233+0000","last_change":"2024-02-10T08:37:10.979010+0000","last_active":"2024-02-10T08:37:49.966233+0000","last_peered":"2024-02-10T08:37:49.966233+0000","last_clean":"2024-02-10T08:37:49.966233+0000","last_became_active":"2024-02-10T08:37:10.155420+0000","last_became_peered":"2024-02-10T08:37:10.155420+0000","last_unstale":"2024-02-10T08:37:49.966233+0000","last_undegraded":"2024-02-10T08:37:49.966233+0000","last_fullsized":"2024-02-10T08:37:49.966233+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":136,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969758+0000","last_change":"2024-02-10T08:36:27.052479+0000","last_active":"2024-02-10T08:37:49.969758+0000","last_peered":"2024-02-10T08:37:49.969758+0000","last_clean":"2024-02-10T08:37:49.969758+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:37:49.969758+0000","last_undegraded":"2024-02-10T08:37:49.969758+0000","last_fullsized":"2024-02-10T08:37:49.969758+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.052362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.052362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":165,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975828+0000","last_change":"2024-02-10T08:37:40.592997+0000","last_active":"2024-02-10T08:37:49.975828+0000","last_peered":"2024-02-10T08:37:49.975828+0000","last_clean":"2024-02-10T08:37:49.975828+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:37:49.975828+0000","last_undegraded":"2024-02-10T08:37:49.975828+0000","last_fullsized":"2024-02-10T08:37:49.975828+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:37:40.592930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.592930+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":157,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963703+0000","last_change":"2024-02-10T08:37:10.149914+0000","last_active":"2024-02-10T08:37:49.963703+0000","last_peered":"2024-02-10T08:37:49.963703+0000","last_clean":"2024-02-10T08:37:49.963703+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:37:49.963703+0000","last_undegraded":"2024-02-10T08:37:49.963703+0000","last_fullsized":"2024-02-10T08:37:49.963703+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.053809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.053809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970998+0000","last_change":"2024-02-10T08:36:30.795587+0000","last_active":"2024-02-10T08:37:49.970998+0000","last_peered":"2024-02-10T08:37:49.970998+0000","last_clean":"2024-02-10T08:37:49.970998+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:37:49.970998+0000","last_undegraded":"2024-02-10T08:37:49.970998+0000","last_fullsized":"2024-02-10T08:37:49.970998+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":139,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966391+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:37:49.966391+0000","last_peered":"2024-02-10T08:37:49.966391+0000","last_clean":"2024-02-10T08:37:49.966391+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:37:49.966391+0000","last_undegraded":"2024-02-10T08:37:49.966391+0000","last_fullsized":"2024-02-10T08:37:49.966391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980799+0000","last_change":"2024-02-10T08:36:29.804845+0000","last_active":"2024-02-10T08:37:49.980799+0000","last_peered":"2024-02-10T08:37:49.980799+0000","last_clean":"2024-02-10T08:37:49.980799+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:37:49.980799+0000","last_undegraded":"2024-02-10T08:37:49.980799+0000","last_fullsized":"2024-02-10T08:37:49.980799+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:29.804761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.804761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968634+0000","last_change":"2024-02-10T08:37:10.151100+0000","last_active":"2024-02-10T08:37:49.968634+0000","last_peered":"2024-02-10T08:37:49.968634+0000","last_clean":"2024-02-10T08:37:49.968634+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:37:49.968634+0000","last_undegraded":"2024-02-10T08:37:49.968634+0000","last_fullsized":"2024-02-10T08:37:49.968634+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1a","version":"103'5","reported_seq":69,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966530+0000","last_change":"2024-02-10T08:37:10.316958+0000","last_active":"2024-02-10T08:37:49.966530+0000","last_peered":"2024-02-10T08:37:49.966530+0000","last_clean":"2024-02-10T08:37:49.966530+0000","last_became_active":"2024-02-10T08:37:10.208613+0000","last_became_peered":"2024-02-10T08:37:10.208613+0000","last_unstale":"2024-02-10T08:37:49.966530+0000","last_undegraded":"2024-02-10T08:37:49.966530+0000","last_fullsized":"2024-02-10T08:37:49.966530+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":154,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969775+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:37:49.969775+0000","last_peered":"2024-02-10T08:37:49.969775+0000","last_clean":"2024-02-10T08:37:49.969775+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:37:49.969775+0000","last_undegraded":"2024-02-10T08:37:49.969775+0000","last_fullsized":"2024-02-10T08:37:49.969775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":191,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964830+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:37:49.964830+0000","last_peered":"2024-02-10T08:37:49.964830+0000","last_clean":"2024-02-10T08:37:49.964830+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:37:49.964830+0000","last_undegraded":"2024-02-10T08:37:49.964830+0000","last_fullsized":"2024-02-10T08:37:49.964830+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967590+0000","last_change":"2024-02-10T08:37:10.234765+0000","last_active":"2024-02-10T08:37:49.967590+0000","last_peered":"2024-02-10T08:37:49.967590+0000","last_clean":"2024-02-10T08:37:49.967590+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:37:49.967590+0000","last_undegraded":"2024-02-10T08:37:49.967590+0000","last_fullsized":"2024-02-10T08:37:49.967590+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:28.635610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":410,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.585987+0000","last_change":"2024-02-10T08:36:33.452480+0000","last_active":"2024-02-10T08:37:50.585987+0000","last_peered":"2024-02-10T08:37:50.585987+0000","last_clean":"2024-02-10T08:37:50.585987+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:37:50.585987+0000","last_undegraded":"2024-02-10T08:37:50.585987+0000","last_fullsized":"2024-02-10T08:37:50.585987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.452367+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969660+0000","last_change":"2024-02-10T08:36:32.132836+0000","last_active":"2024-02-10T08:37:49.969660+0000","last_peered":"2024-02-10T08:37:49.969660+0000","last_clean":"2024-02-10T08:37:49.969660+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:37:49.969660+0000","last_undegraded":"2024-02-10T08:37:49.969660+0000","last_fullsized":"2024-02-10T08:37:49.969660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:32.132760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.132760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961089+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:37:49.961089+0000","last_peered":"2024-02-10T08:37:49.961089+0000","last_clean":"2024-02-10T08:37:49.961089+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:37:49.961089+0000","last_undegraded":"2024-02-10T08:37:49.961089+0000","last_fullsized":"2024-02-10T08:37:49.961089+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962859+0000","last_change":"2024-02-10T08:36:37.077971+0000","last_active":"2024-02-10T08:37:49.962859+0000","last_peered":"2024-02-10T08:37:49.962859+0000","last_clean":"2024-02-10T08:37:49.962859+0000","last_became_active":"2024-02-10T08:36:37.074340+0000","last_became_peered":"2024-02-10T08:36:37.074340+0000","last_unstale":"2024-02-10T08:37:49.962859+0000","last_undegraded":"2024-02-10T08:37:49.962859+0000","last_fullsized":"2024-02-10T08:37:49.962859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960994+0000","last_change":"2024-02-10T08:37:10.160466+0000","last_active":"2024-02-10T08:37:49.960994+0000","last_peered":"2024-02-10T08:37:49.960994+0000","last_clean":"2024-02-10T08:37:49.960994+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:37:49.960994+0000","last_undegraded":"2024-02-10T08:37:49.960994+0000","last_fullsized":"2024-02-10T08:37:49.960994+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.962334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.962334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":179,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980665+0000","last_change":"2024-02-10T08:36:17.572570+0000","last_active":"2024-02-10T08:37:49.980665+0000","last_peered":"2024-02-10T08:37:49.980665+0000","last_clean":"2024-02-10T08:37:49.980665+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:37:49.980665+0000","last_undegraded":"2024-02-10T08:37:49.980665+0000","last_fullsized":"2024-02-10T08:37:49.980665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:36:17.572475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.572475+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.982884+0000","last_change":"2024-02-10T08:37:41.598616+0000","last_active":"2024-02-10T08:37:49.982884+0000","last_peered":"2024-02-10T08:37:49.982884+0000","last_clean":"2024-02-10T08:37:49.982884+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:37:49.982884+0000","last_undegraded":"2024-02-10T08:37:49.982884+0000","last_fullsized":"2024-02-10T08:37:49.982884+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.598541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.598541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972648+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:37:49.972648+0000","last_peered":"2024-02-10T08:37:49.972648+0000","last_clean":"2024-02-10T08:37:49.972648+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:37:49.972648+0000","last_undegraded":"2024-02-10T08:37:49.972648+0000","last_fullsized":"2024-02-10T08:37:49.972648+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965039+0000","last_change":"2024-02-10T08:37:10.175199+0000","last_active":"2024-02-10T08:37:49.965039+0000","last_peered":"2024-02-10T08:37:49.965039+0000","last_clean":"2024-02-10T08:37:49.965039+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:37:49.965039+0000","last_undegraded":"2024-02-10T08:37:49.965039+0000","last_fullsized":"2024-02-10T08:37:49.965039+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"103'5","reported_seq":69,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969864+0000","last_change":"2024-02-10T08:37:10.332044+0000","last_active":"2024-02-10T08:37:49.969864+0000","last_peered":"2024-02-10T08:37:49.969864+0000","last_clean":"2024-02-10T08:37:49.969864+0000","last_became_active":"2024-02-10T08:37:10.204992+0000","last_became_peered":"2024-02-10T08:37:10.204992+0000","last_unstale":"2024-02-10T08:37:49.969864+0000","last_undegraded":"2024-02-10T08:37:49.969864+0000","last_fullsized":"2024-02-10T08:37:49.969864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":89,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.982422+0000","last_change":"2024-02-10T08:37:41.604353+0000","last_active":"2024-02-10T08:37:49.982422+0000","last_peered":"2024-02-10T08:37:49.982422+0000","last_clean":"2024-02-10T08:37:49.982422+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:37:49.982422+0000","last_undegraded":"2024-02-10T08:37:49.982422+0000","last_fullsized":"2024-02-10T08:37:49.982422+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:37:41.604248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.604248+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":163,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963519+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:37:49.963519+0000","last_peered":"2024-02-10T08:37:49.963519+0000","last_clean":"2024-02-10T08:37:49.963519+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:37:49.963519+0000","last_undegraded":"2024-02-10T08:37:49.963519+0000","last_fullsized":"2024-02-10T08:37:49.963519+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963618+0000","last_change":"2024-02-10T08:37:10.183206+0000","last_active":"2024-02-10T08:37:49.963618+0000","last_peered":"2024-02-10T08:37:49.963618+0000","last_clean":"2024-02-10T08:37:49.963618+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:37:49.963618+0000","last_undegraded":"2024-02-10T08:37:49.963618+0000","last_fullsized":"2024-02-10T08:37:49.963618+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.456780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966097+0000","last_change":"2024-02-10T08:37:10.263452+0000","last_active":"2024-02-10T08:37:49.966097+0000","last_peered":"2024-02-10T08:37:49.966097+0000","last_clean":"2024-02-10T08:37:49.966097+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:37:49.966097+0000","last_undegraded":"2024-02-10T08:37:49.966097+0000","last_fullsized":"2024-02-10T08:37:49.966097+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.802264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.802264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972483+0000","last_change":"2024-02-10T08:36:37.155744+0000","last_active":"2024-02-10T08:37:49.972483+0000","last_peered":"2024-02-10T08:37:49.972483+0000","last_clean":"2024-02-10T08:37:49.972483+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:37:49.972483+0000","last_undegraded":"2024-02-10T08:37:49.972483+0000","last_fullsized":"2024-02-10T08:37:49.972483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.14","version":"103'2","reported_seq":85,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967175+0000","last_change":"2024-02-10T08:36:37.065315+0000","last_active":"2024-02-10T08:37:49.967175+0000","last_peered":"2024-02-10T08:37:49.967175+0000","last_clean":"2024-02-10T08:37:49.967175+0000","last_became_active":"2024-02-10T08:36:37.063912+0000","last_became_peered":"2024-02-10T08:36:37.063912+0000","last_unstale":"2024-02-10T08:37:49.967175+0000","last_undegraded":"2024-02-10T08:37:49.967175+0000","last_fullsized":"2024-02-10T08:37:49.967175+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":168,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965356+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:37:49.965356+0000","last_peered":"2024-02-10T08:37:49.965356+0000","last_clean":"2024-02-10T08:37:49.965356+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:37:49.965356+0000","last_undegraded":"2024-02-10T08:37:49.965356+0000","last_fullsized":"2024-02-10T08:37:49.965356+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":166,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962697+0000","last_change":"2024-02-10T08:36:28.744540+0000","last_active":"2024-02-10T08:37:49.962697+0000","last_peered":"2024-02-10T08:37:49.962697+0000","last_clean":"2024-02-10T08:37:49.962697+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:37:49.962697+0000","last_undegraded":"2024-02-10T08:37:49.962697+0000","last_fullsized":"2024-02-10T08:37:49.962697+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:36:28.744465+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.744465+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":168,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963608+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:37:49.963608+0000","last_peered":"2024-02-10T08:37:49.963608+0000","last_clean":"2024-02-10T08:37:49.963608+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:37:49.963608+0000","last_undegraded":"2024-02-10T08:37:49.963608+0000","last_fullsized":"2024-02-10T08:37:49.963608+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976477+0000","last_change":"2024-02-10T08:36:23.084433+0000","last_active":"2024-02-10T08:37:49.976477+0000","last_peered":"2024-02-10T08:37:49.976477+0000","last_clean":"2024-02-10T08:37:49.976477+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:37:49.976477+0000","last_undegraded":"2024-02-10T08:37:49.976477+0000","last_fullsized":"2024-02-10T08:37:49.976477+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:23.084331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.084331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":88,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968388+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:37:49.968388+0000","last_peered":"2024-02-10T08:37:49.968388+0000","last_clean":"2024-02-10T08:37:49.968388+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:37:49.968388+0000","last_undegraded":"2024-02-10T08:37:49.968388+0000","last_fullsized":"2024-02-10T08:37:49.968388+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.b","version":"103'2","reported_seq":98,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963106+0000","last_change":"2024-02-10T08:37:10.677108+0000","last_active":"2024-02-10T08:37:49.963106+0000","last_peered":"2024-02-10T08:37:49.963106+0000","last_clean":"2024-02-10T08:37:49.963106+0000","last_became_active":"2024-02-10T08:37:10.216339+0000","last_became_peered":"2024-02-10T08:37:10.216339+0000","last_unstale":"2024-02-10T08:37:49.963106+0000","last_undegraded":"2024-02-10T08:37:49.963106+0000","last_fullsized":"2024-02-10T08:37:49.963106+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":124,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972712+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:37:49.972712+0000","last_peered":"2024-02-10T08:37:49.972712+0000","last_clean":"2024-02-10T08:37:49.972712+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:37:49.972712+0000","last_undegraded":"2024-02-10T08:37:49.972712+0000","last_fullsized":"2024-02-10T08:37:49.972712+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":154,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965282+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:37:49.965282+0000","last_peered":"2024-02-10T08:37:49.965282+0000","last_clean":"2024-02-10T08:37:49.965282+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:37:49.965282+0000","last_undegraded":"2024-02-10T08:37:49.965282+0000","last_fullsized":"2024-02-10T08:37:49.965282+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969978+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:37:49.969978+0000","last_peered":"2024-02-10T08:37:49.969978+0000","last_clean":"2024-02-10T08:37:49.969978+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:37:49.969978+0000","last_undegraded":"2024-02-10T08:37:49.969978+0000","last_fullsized":"2024-02-10T08:37:49.969978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962735+0000","last_change":"2024-02-10T08:36:37.041210+0000","last_active":"2024-02-10T08:37:49.962735+0000","last_peered":"2024-02-10T08:37:49.962735+0000","last_clean":"2024-02-10T08:37:49.962735+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:37:49.962735+0000","last_undegraded":"2024-02-10T08:37:49.962735+0000","last_fullsized":"2024-02-10T08:37:49.962735+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"103'10","reported_seq":76,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967770+0000","last_change":"2024-02-10T08:37:10.582273+0000","last_active":"2024-02-10T08:37:49.967770+0000","last_peered":"2024-02-10T08:37:49.967770+0000","last_clean":"2024-02-10T08:37:49.967770+0000","last_became_active":"2024-02-10T08:37:10.186893+0000","last_became_peered":"2024-02-10T08:37:10.186893+0000","last_unstale":"2024-02-10T08:37:49.967770+0000","last_undegraded":"2024-02-10T08:37:49.967770+0000","last_fullsized":"2024-02-10T08:37:49.967770+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":146,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972868+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:37:49.972868+0000","last_peered":"2024-02-10T08:37:49.972868+0000","last_clean":"2024-02-10T08:37:49.972868+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:37:49.972868+0000","last_undegraded":"2024-02-10T08:37:49.972868+0000","last_fullsized":"2024-02-10T08:37:49.972868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967065+0000","last_change":"2024-02-10T08:36:19.570275+0000","last_active":"2024-02-10T08:37:49.967065+0000","last_peered":"2024-02-10T08:37:49.967065+0000","last_clean":"2024-02-10T08:37:49.967065+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:37:49.967065+0000","last_undegraded":"2024-02-10T08:37:49.967065+0000","last_fullsized":"2024-02-10T08:37:49.967065+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:19.570181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:19.570181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968906+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:37:49.968906+0000","last_peered":"2024-02-10T08:37:49.968906+0000","last_clean":"2024-02-10T08:37:49.968906+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:37:49.968906+0000","last_undegraded":"2024-02-10T08:37:49.968906+0000","last_fullsized":"2024-02-10T08:37:49.968906+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970875+0000","last_change":"2024-02-10T08:36:37.051573+0000","last_active":"2024-02-10T08:37:49.970875+0000","last_peered":"2024-02-10T08:37:49.970875+0000","last_clean":"2024-02-10T08:37:49.970875+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:37:49.970875+0000","last_undegraded":"2024-02-10T08:37:49.970875+0000","last_fullsized":"2024-02-10T08:37:49.970875+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967068+0000","last_change":"2024-02-10T08:37:10.215944+0000","last_active":"2024-02-10T08:37:49.967068+0000","last_peered":"2024-02-10T08:37:49.967068+0000","last_clean":"2024-02-10T08:37:49.967068+0000","last_became_active":"2024-02-10T08:37:10.205941+0000","last_became_peered":"2024-02-10T08:37:10.205941+0000","last_unstale":"2024-02-10T08:37:49.967068+0000","last_undegraded":"2024-02-10T08:37:49.967068+0000","last_fullsized":"2024-02-10T08:37:49.967068+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":152,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971152+0000","last_change":"2024-02-10T08:36:14.035460+0000","last_active":"2024-02-10T08:37:49.971152+0000","last_peered":"2024-02-10T08:37:49.971152+0000","last_clean":"2024-02-10T08:37:49.971152+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:37:49.971152+0000","last_undegraded":"2024-02-10T08:37:49.971152+0000","last_fullsized":"2024-02-10T08:37:49.971152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:14.035356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.035356+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.982816+0000","last_change":"2024-02-10T08:37:46.576724+0000","last_active":"2024-02-10T08:37:49.982816+0000","last_peered":"2024-02-10T08:37:49.982816+0000","last_clean":"2024-02-10T08:37:49.982816+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:37:49.982816+0000","last_undegraded":"2024-02-10T08:37:49.982816+0000","last_fullsized":"2024-02-10T08:37:49.982816+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.576661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.576661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":211,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964475+0000","last_change":"2024-02-10T08:37:10.531826+0000","last_active":"2024-02-10T08:37:49.964475+0000","last_peered":"2024-02-10T08:37:49.964475+0000","last_clean":"2024-02-10T08:37:49.964475+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:37:49.964475+0000","last_undegraded":"2024-02-10T08:37:49.964475+0000","last_fullsized":"2024-02-10T08:37:49.964475+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:37:05.796026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:05.796026+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.978371+0000","last_change":"2024-02-10T08:37:10.242150+0000","last_active":"2024-02-10T08:37:49.978371+0000","last_peered":"2024-02-10T08:37:49.978371+0000","last_clean":"2024-02-10T08:37:49.978371+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:37:49.978371+0000","last_undegraded":"2024-02-10T08:37:49.978371+0000","last_fullsized":"2024-02-10T08:37:49.978371+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.928693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965932+0000","last_change":"2024-02-10T08:37:10.232375+0000","last_active":"2024-02-10T08:37:49.965932+0000","last_peered":"2024-02-10T08:37:49.965932+0000","last_clean":"2024-02-10T08:37:49.965932+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:37:49.965932+0000","last_undegraded":"2024-02-10T08:37:49.965932+0000","last_fullsized":"2024-02-10T08:37:49.965932+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.8","version":"103'2","reported_seq":85,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966097+0000","last_change":"2024-02-10T08:36:37.035179+0000","last_active":"2024-02-10T08:37:49.966097+0000","last_peered":"2024-02-10T08:37:49.966097+0000","last_clean":"2024-02-10T08:37:49.966097+0000","last_became_active":"2024-02-10T08:36:37.023527+0000","last_became_peered":"2024-02-10T08:36:37.023527+0000","last_unstale":"2024-02-10T08:37:49.966097+0000","last_undegraded":"2024-02-10T08:37:49.966097+0000","last_fullsized":"2024-02-10T08:37:49.966097+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":154,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.964330+0000","last_change":"2024-02-10T08:37:18.722889+0000","last_active":"2024-02-10T08:37:49.964330+0000","last_peered":"2024-02-10T08:37:49.964330+0000","last_clean":"2024-02-10T08:37:49.964330+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:37:49.964330+0000","last_undegraded":"2024-02-10T08:37:49.964330+0000","last_fullsized":"2024-02-10T08:37:49.964330+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:18.722818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.722818+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963289+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:37:49.963289+0000","last_peered":"2024-02-10T08:37:49.963289+0000","last_clean":"2024-02-10T08:37:49.963289+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:37:49.963289+0000","last_undegraded":"2024-02-10T08:37:49.963289+0000","last_fullsized":"2024-02-10T08:37:49.963289+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":197,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971474+0000","last_change":"2024-02-10T08:36:29.049034+0000","last_active":"2024-02-10T08:37:49.971474+0000","last_peered":"2024-02-10T08:37:49.971474+0000","last_clean":"2024-02-10T08:37:49.971474+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:37:49.971474+0000","last_undegraded":"2024-02-10T08:37:49.971474+0000","last_fullsized":"2024-02-10T08:37:49.971474+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:29.048965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.048965+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971938+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:37:49.971938+0000","last_peered":"2024-02-10T08:37:49.971938+0000","last_clean":"2024-02-10T08:37:49.971938+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:37:49.971938+0000","last_undegraded":"2024-02-10T08:37:49.971938+0000","last_fullsized":"2024-02-10T08:37:49.971938+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":96,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967163+0000","last_change":"2024-02-10T08:36:37.041395+0000","last_active":"2024-02-10T08:37:49.967163+0000","last_peered":"2024-02-10T08:37:49.967163+0000","last_clean":"2024-02-10T08:37:49.967163+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:37:49.967163+0000","last_undegraded":"2024-02-10T08:37:49.967163+0000","last_fullsized":"2024-02-10T08:37:49.967163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980965+0000","last_change":"2024-02-10T08:37:10.129525+0000","last_active":"2024-02-10T08:37:49.980965+0000","last_peered":"2024-02-10T08:37:49.980965+0000","last_clean":"2024-02-10T08:37:49.980965+0000","last_became_active":"2024-02-10T08:37:10.129023+0000","last_became_peered":"2024-02-10T08:37:10.129023+0000","last_unstale":"2024-02-10T08:37:49.980965+0000","last_undegraded":"2024-02-10T08:37:49.980965+0000","last_fullsized":"2024-02-10T08:37:49.980965+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":158,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963267+0000","last_change":"2024-02-10T08:36:31.434160+0000","last_active":"2024-02-10T08:37:49.963267+0000","last_peered":"2024-02-10T08:37:49.963267+0000","last_clean":"2024-02-10T08:37:49.963267+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:37:49.963267+0000","last_undegraded":"2024-02-10T08:37:49.963267+0000","last_fullsized":"2024-02-10T08:37:49.963267+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.434047+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976921+0000","last_change":"2024-02-10T08:37:18.260244+0000","last_active":"2024-02-10T08:37:49.976921+0000","last_peered":"2024-02-10T08:37:49.976921+0000","last_clean":"2024-02-10T08:37:49.976921+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:37:49.976921+0000","last_undegraded":"2024-02-10T08:37:49.976921+0000","last_fullsized":"2024-02-10T08:37:49.976921+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.260171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.260171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":144,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976424+0000","last_change":"2024-02-10T08:37:10.315695+0000","last_active":"2024-02-10T08:37:49.976424+0000","last_peered":"2024-02-10T08:37:49.976424+0000","last_clean":"2024-02-10T08:37:49.976424+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:37:49.976424+0000","last_undegraded":"2024-02-10T08:37:49.976424+0000","last_fullsized":"2024-02-10T08:37:49.976424+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976503+0000","last_change":"2024-02-10T08:37:10.150087+0000","last_active":"2024-02-10T08:37:49.976503+0000","last_peered":"2024-02-10T08:37:49.976503+0000","last_clean":"2024-02-10T08:37:49.976503+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:37:49.976503+0000","last_undegraded":"2024-02-10T08:37:49.976503+0000","last_fullsized":"2024-02-10T08:37:49.976503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.978888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969226+0000","last_change":"2024-02-10T08:37:10.202917+0000","last_active":"2024-02-10T08:37:49.969226+0000","last_peered":"2024-02-10T08:37:49.969226+0000","last_clean":"2024-02-10T08:37:49.969226+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:37:49.969226+0000","last_undegraded":"2024-02-10T08:37:49.969226+0000","last_fullsized":"2024-02-10T08:37:49.969226+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963206+0000","last_change":"2024-02-10T08:37:10.242206+0000","last_active":"2024-02-10T08:37:49.963206+0000","last_peered":"2024-02-10T08:37:49.963206+0000","last_clean":"2024-02-10T08:37:49.963206+0000","last_became_active":"2024-02-10T08:37:10.242025+0000","last_became_peered":"2024-02-10T08:37:10.242025+0000","last_unstale":"2024-02-10T08:37:49.963206+0000","last_undegraded":"2024-02-10T08:37:49.963206+0000","last_fullsized":"2024-02-10T08:37:49.963206+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":157,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963792+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:37:49.963792+0000","last_peered":"2024-02-10T08:37:49.963792+0000","last_clean":"2024-02-10T08:37:49.963792+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:37:49.963792+0000","last_undegraded":"2024-02-10T08:37:49.963792+0000","last_fullsized":"2024-02-10T08:37:49.963792+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961153+0000","last_change":"2024-02-10T08:37:10.151238+0000","last_active":"2024-02-10T08:37:49.961153+0000","last_peered":"2024-02-10T08:37:49.961153+0000","last_clean":"2024-02-10T08:37:49.961153+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:37:49.961153+0000","last_undegraded":"2024-02-10T08:37:49.961153+0000","last_fullsized":"2024-02-10T08:37:49.961153+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.453161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.453161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":136,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963271+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:37:49.963271+0000","last_peered":"2024-02-10T08:37:49.963271+0000","last_clean":"2024-02-10T08:37:49.963271+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:37:49.963271+0000","last_undegraded":"2024-02-10T08:37:49.963271+0000","last_fullsized":"2024-02-10T08:37:49.963271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":142,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966514+0000","last_change":"2024-02-10T08:37:10.284262+0000","last_active":"2024-02-10T08:37:49.966514+0000","last_peered":"2024-02-10T08:37:49.966514+0000","last_clean":"2024-02-10T08:37:49.966514+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:37:49.966514+0000","last_undegraded":"2024-02-10T08:37:49.966514+0000","last_fullsized":"2024-02-10T08:37:49.966514+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.066605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.066605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972657+0000","last_change":"2024-02-10T08:37:10.260574+0000","last_active":"2024-02-10T08:37:49.972657+0000","last_peered":"2024-02-10T08:37:49.972657+0000","last_clean":"2024-02-10T08:37:49.972657+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:37:49.972657+0000","last_undegraded":"2024-02-10T08:37:49.972657+0000","last_fullsized":"2024-02-10T08:37:49.972657+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"103'5","reported_seq":106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975982+0000","last_change":"2024-02-10T08:37:45.548365+0000","last_active":"2024-02-10T08:37:49.975982+0000","last_peered":"2024-02-10T08:37:49.975982+0000","last_clean":"2024-02-10T08:37:49.975982+0000","last_became_active":"2024-02-10T08:37:10.149021+0000","last_became_peered":"2024-02-10T08:37:10.149021+0000","last_unstale":"2024-02-10T08:37:49.975982+0000","last_undegraded":"2024-02-10T08:37:49.975982+0000","last_fullsized":"2024-02-10T08:37:49.975982+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'5","last_scrub_stamp":"2024-02-10T08:37:45.548297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.548297+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":143,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966016+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:37:49.966016+0000","last_peered":"2024-02-10T08:37:49.966016+0000","last_clean":"2024-02-10T08:37:49.966016+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:37:49.966016+0000","last_undegraded":"2024-02-10T08:37:49.966016+0000","last_fullsized":"2024-02-10T08:37:49.966016+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.983247+0000","last_change":"2024-02-10T08:36:15.645089+0000","last_active":"2024-02-10T08:37:49.983247+0000","last_peered":"2024-02-10T08:37:49.983247+0000","last_clean":"2024-02-10T08:37:49.983247+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:37:49.983247+0000","last_undegraded":"2024-02-10T08:37:49.983247+0000","last_fullsized":"2024-02-10T08:37:49.983247+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.644998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.644998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":136,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966991+0000","last_change":"2024-02-10T08:36:33.824585+0000","last_active":"2024-02-10T08:37:49.966991+0000","last_peered":"2024-02-10T08:37:49.966991+0000","last_clean":"2024-02-10T08:37:49.966991+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:37:49.966991+0000","last_undegraded":"2024-02-10T08:37:49.966991+0000","last_fullsized":"2024-02-10T08:37:49.966991+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.824511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.824511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":139,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971005+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:37:49.971005+0000","last_peered":"2024-02-10T08:37:49.971005+0000","last_clean":"2024-02-10T08:37:49.971005+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:37:49.971005+0000","last_undegraded":"2024-02-10T08:37:49.971005+0000","last_fullsized":"2024-02-10T08:37:49.971005+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970914+0000","last_change":"2024-02-10T08:36:37.165190+0000","last_active":"2024-02-10T08:37:49.970914+0000","last_peered":"2024-02-10T08:37:49.970914+0000","last_clean":"2024-02-10T08:37:49.970914+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:37:49.970914+0000","last_undegraded":"2024-02-10T08:37:49.970914+0000","last_fullsized":"2024-02-10T08:37:49.970914+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.981720+0000","last_change":"2024-02-10T08:36:37.127556+0000","last_active":"2024-02-10T08:37:49.981720+0000","last_peered":"2024-02-10T08:37:49.981720+0000","last_clean":"2024-02-10T08:37:49.981720+0000","last_became_active":"2024-02-10T08:36:37.127287+0000","last_became_peered":"2024-02-10T08:36:37.127287+0000","last_unstale":"2024-02-10T08:37:49.981720+0000","last_undegraded":"2024-02-10T08:37:49.981720+0000","last_fullsized":"2024-02-10T08:37:49.981720+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":156,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966948+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:37:49.966948+0000","last_peered":"2024-02-10T08:37:49.966948+0000","last_clean":"2024-02-10T08:37:49.966948+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:37:49.966948+0000","last_undegraded":"2024-02-10T08:37:49.966948+0000","last_fullsized":"2024-02-10T08:37:49.966948+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":145,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977082+0000","last_change":"2024-02-10T08:37:38.527763+0000","last_active":"2024-02-10T08:37:49.977082+0000","last_peered":"2024-02-10T08:37:49.977082+0000","last_clean":"2024-02-10T08:37:49.977082+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:37:49.977082+0000","last_undegraded":"2024-02-10T08:37:49.977082+0000","last_fullsized":"2024-02-10T08:37:49.977082+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.527658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.527658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":136,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963921+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:37:49.963921+0000","last_peered":"2024-02-10T08:37:49.963921+0000","last_clean":"2024-02-10T08:37:49.963921+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:37:49.963921+0000","last_undegraded":"2024-02-10T08:37:49.963921+0000","last_fullsized":"2024-02-10T08:37:49.963921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":142,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976472+0000","last_change":"2024-02-10T08:37:10.262623+0000","last_active":"2024-02-10T08:37:49.976472+0000","last_peered":"2024-02-10T08:37:49.976472+0000","last_clean":"2024-02-10T08:37:49.976472+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:37:49.976472+0000","last_undegraded":"2024-02-10T08:37:49.976472+0000","last_fullsized":"2024-02-10T08:37:49.976472+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.042582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.042582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976230+0000","last_change":"2024-02-10T08:37:10.148262+0000","last_active":"2024-02-10T08:37:49.976230+0000","last_peered":"2024-02-10T08:37:49.976230+0000","last_clean":"2024-02-10T08:37:49.976230+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:37:49.976230+0000","last_undegraded":"2024-02-10T08:37:49.976230+0000","last_fullsized":"2024-02-10T08:37:49.976230+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"103'5","reported_seq":88,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969968+0000","last_change":"2024-02-10T08:36:37.052377+0000","last_active":"2024-02-10T08:37:49.969968+0000","last_peered":"2024-02-10T08:37:49.969968+0000","last_clean":"2024-02-10T08:37:49.969968+0000","last_became_active":"2024-02-10T08:36:37.050808+0000","last_became_peered":"2024-02-10T08:36:37.050808+0000","last_unstale":"2024-02-10T08:37:49.969968+0000","last_undegraded":"2024-02-10T08:37:49.969968+0000","last_fullsized":"2024-02-10T08:37:49.969968+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.978476+0000","last_change":"2024-02-10T08:37:10.466717+0000","last_active":"2024-02-10T08:37:49.978476+0000","last_peered":"2024-02-10T08:37:49.978476+0000","last_clean":"2024-02-10T08:37:49.978476+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:37:49.978476+0000","last_undegraded":"2024-02-10T08:37:49.978476+0000","last_fullsized":"2024-02-10T08:37:49.978476+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:31.890386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.890386+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":154,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975938+0000","last_change":"2024-02-10T08:37:39.573283+0000","last_active":"2024-02-10T08:37:49.975938+0000","last_peered":"2024-02-10T08:37:49.975938+0000","last_clean":"2024-02-10T08:37:49.975938+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:37:49.975938+0000","last_undegraded":"2024-02-10T08:37:49.975938+0000","last_fullsized":"2024-02-10T08:37:49.975938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:39.573225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.573225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":135,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969881+0000","last_change":"2024-02-10T08:37:07.241312+0000","last_active":"2024-02-10T08:37:49.969881+0000","last_peered":"2024-02-10T08:37:49.969881+0000","last_clean":"2024-02-10T08:37:49.969881+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:37:49.969881+0000","last_undegraded":"2024-02-10T08:37:49.969881+0000","last_fullsized":"2024-02-10T08:37:49.969881+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:07.241247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.241247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967810+0000","last_change":"2024-02-10T08:36:37.049548+0000","last_active":"2024-02-10T08:37:49.967810+0000","last_peered":"2024-02-10T08:37:49.967810+0000","last_clean":"2024-02-10T08:37:49.967810+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:37:49.967810+0000","last_undegraded":"2024-02-10T08:37:49.967810+0000","last_fullsized":"2024-02-10T08:37:49.967810+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970807+0000","last_change":"2024-02-10T08:36:37.074211+0000","last_active":"2024-02-10T08:37:49.970807+0000","last_peered":"2024-02-10T08:37:49.970807+0000","last_clean":"2024-02-10T08:37:49.970807+0000","last_became_active":"2024-02-10T08:36:37.073890+0000","last_became_peered":"2024-02-10T08:36:37.073890+0000","last_unstale":"2024-02-10T08:37:49.970807+0000","last_undegraded":"2024-02-10T08:37:49.970807+0000","last_fullsized":"2024-02-10T08:37:49.970807+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":162,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977561+0000","last_change":"2024-02-10T08:37:20.333410+0000","last_active":"2024-02-10T08:37:49.977561+0000","last_peered":"2024-02-10T08:37:49.977561+0000","last_clean":"2024-02-10T08:37:49.977561+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:37:49.977561+0000","last_undegraded":"2024-02-10T08:37:49.977561+0000","last_fullsized":"2024-02-10T08:37:49.977561+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:20.333345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.333345+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":148,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966719+0000","last_change":"2024-02-10T08:37:10.284428+0000","last_active":"2024-02-10T08:37:49.966719+0000","last_peered":"2024-02-10T08:37:49.966719+0000","last_clean":"2024-02-10T08:37:49.966719+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:37:49.966719+0000","last_undegraded":"2024-02-10T08:37:49.966719+0000","last_fullsized":"2024-02-10T08:37:49.966719+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.072982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960914+0000","last_change":"2024-02-10T08:36:25.503414+0000","last_active":"2024-02-10T08:37:49.960914+0000","last_peered":"2024-02-10T08:37:49.960914+0000","last_clean":"2024-02-10T08:37:49.960914+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:37:49.960914+0000","last_undegraded":"2024-02-10T08:37:49.960914+0000","last_fullsized":"2024-02-10T08:37:49.960914+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.503351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.503351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.961009+0000","last_change":"2024-02-10T08:36:37.124706+0000","last_active":"2024-02-10T08:37:49.961009+0000","last_peered":"2024-02-10T08:37:49.961009+0000","last_clean":"2024-02-10T08:37:49.961009+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:37:49.961009+0000","last_undegraded":"2024-02-10T08:37:49.961009+0000","last_fullsized":"2024-02-10T08:37:49.961009+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"103'5","reported_seq":93,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967689+0000","last_change":"2024-02-10T08:37:45.871952+0000","last_active":"2024-02-10T08:37:49.967689+0000","last_peered":"2024-02-10T08:37:49.967689+0000","last_clean":"2024-02-10T08:37:49.967689+0000","last_became_active":"2024-02-10T08:36:37.063485+0000","last_became_peered":"2024-02-10T08:36:37.063485+0000","last_unstale":"2024-02-10T08:37:49.967689+0000","last_undegraded":"2024-02-10T08:37:49.967689+0000","last_fullsized":"2024-02-10T08:37:49.967689+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'5","last_scrub_stamp":"2024-02-10T08:37:45.871881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.871881+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":175,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970357+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:37:49.970357+0000","last_peered":"2024-02-10T08:37:49.970357+0000","last_clean":"2024-02-10T08:37:49.970357+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:37:49.970357+0000","last_undegraded":"2024-02-10T08:37:49.970357+0000","last_fullsized":"2024-02-10T08:37:49.970357+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972205+0000","last_change":"2024-02-10T08:36:34.837568+0000","last_active":"2024-02-10T08:37:49.972205+0000","last_peered":"2024-02-10T08:37:49.972205+0000","last_clean":"2024-02-10T08:37:49.972205+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:37:49.972205+0000","last_undegraded":"2024-02-10T08:37:49.972205+0000","last_fullsized":"2024-02-10T08:37:49.972205+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:34.837483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:34.837483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963479+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:37:49.963479+0000","last_peered":"2024-02-10T08:37:49.963479+0000","last_clean":"2024-02-10T08:37:49.963479+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:37:49.963479+0000","last_undegraded":"2024-02-10T08:37:49.963479+0000","last_fullsized":"2024-02-10T08:37:49.963479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965772+0000","last_change":"2024-02-10T08:37:10.232241+0000","last_active":"2024-02-10T08:37:49.965772+0000","last_peered":"2024-02-10T08:37:49.965772+0000","last_clean":"2024-02-10T08:37:49.965772+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:37:49.965772+0000","last_undegraded":"2024-02-10T08:37:49.965772+0000","last_fullsized":"2024-02-10T08:37:49.965772+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":160,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970395+0000","last_change":"2024-02-10T08:36:20.047350+0000","last_active":"2024-02-10T08:37:49.970395+0000","last_peered":"2024-02-10T08:37:49.970395+0000","last_clean":"2024-02-10T08:37:49.970395+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:37:49.970395+0000","last_undegraded":"2024-02-10T08:37:49.970395+0000","last_fullsized":"2024-02-10T08:37:49.970395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:36:20.047272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.047272+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":139,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976386+0000","last_change":"2024-02-10T08:36:15.045147+0000","last_active":"2024-02-10T08:37:49.976386+0000","last_peered":"2024-02-10T08:37:49.976386+0000","last_clean":"2024-02-10T08:37:49.976386+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:37:49.976386+0000","last_undegraded":"2024-02-10T08:37:49.976386+0000","last_fullsized":"2024-02-10T08:37:49.976386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.045044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.045044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972956+0000","last_change":"2024-02-10T08:36:27.054202+0000","last_active":"2024-02-10T08:37:49.972956+0000","last_peered":"2024-02-10T08:37:49.972956+0000","last_clean":"2024-02-10T08:37:49.972956+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:37:49.972956+0000","last_undegraded":"2024-02-10T08:37:49.972956+0000","last_fullsized":"2024-02-10T08:37:49.972956+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.054145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.054145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963334+0000","last_change":"2024-02-10T08:36:37.038364+0000","last_active":"2024-02-10T08:37:49.963334+0000","last_peered":"2024-02-10T08:37:49.963334+0000","last_clean":"2024-02-10T08:37:49.963334+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:37:49.963334+0000","last_undegraded":"2024-02-10T08:37:49.963334+0000","last_fullsized":"2024-02-10T08:37:49.963334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"103'5","reported_seq":88,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.966015+0000","last_change":"2024-02-10T08:36:37.017992+0000","last_active":"2024-02-10T08:37:49.966015+0000","last_peered":"2024-02-10T08:37:49.966015+0000","last_clean":"2024-02-10T08:37:49.966015+0000","last_became_active":"2024-02-10T08:36:37.016870+0000","last_became_peered":"2024-02-10T08:36:37.016870+0000","last_unstale":"2024-02-10T08:37:49.966015+0000","last_undegraded":"2024-02-10T08:37:49.966015+0000","last_fullsized":"2024-02-10T08:37:49.966015+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.977695+0000","last_change":"2024-02-10T08:37:10.112153+0000","last_active":"2024-02-10T08:37:49.977695+0000","last_peered":"2024-02-10T08:37:49.977695+0000","last_clean":"2024-02-10T08:37:49.977695+0000","last_became_active":"2024-02-10T08:37:10.111814+0000","last_became_peered":"2024-02-10T08:37:10.111814+0000","last_unstale":"2024-02-10T08:37:49.977695+0000","last_undegraded":"2024-02-10T08:37:49.977695+0000","last_fullsized":"2024-02-10T08:37:49.977695+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":173,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962980+0000","last_change":"2024-02-10T08:37:10.483725+0000","last_active":"2024-02-10T08:37:49.962980+0000","last_peered":"2024-02-10T08:37:49.962980+0000","last_clean":"2024-02-10T08:37:49.962980+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:37:49.962980+0000","last_undegraded":"2024-02-10T08:37:49.962980+0000","last_fullsized":"2024-02-10T08:37:49.962980+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:24.764020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.764020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":140,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.965188+0000","last_change":"2024-02-10T08:36:30.157585+0000","last_active":"2024-02-10T08:37:49.965188+0000","last_peered":"2024-02-10T08:37:49.965188+0000","last_clean":"2024-02-10T08:37:49.965188+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:37:49.965188+0000","last_undegraded":"2024-02-10T08:37:49.965188+0000","last_fullsized":"2024-02-10T08:37:49.965188+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.157473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.157473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.959064+0000","last_change":"2024-02-10T08:37:10.187798+0000","last_active":"2024-02-10T08:37:49.959064+0000","last_peered":"2024-02-10T08:37:49.959064+0000","last_clean":"2024-02-10T08:37:49.959064+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:37:49.959064+0000","last_undegraded":"2024-02-10T08:37:49.959064+0000","last_fullsized":"2024-02-10T08:37:49.959064+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.877268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.877268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969279+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:37:49.969279+0000","last_peered":"2024-02-10T08:37:49.969279+0000","last_clean":"2024-02-10T08:37:49.969279+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:37:49.969279+0000","last_undegraded":"2024-02-10T08:37:49.969279+0000","last_fullsized":"2024-02-10T08:37:49.969279+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.6","version":"103'9","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.969195+0000","last_change":"2024-02-10T08:36:37.066456+0000","last_active":"2024-02-10T08:37:49.969195+0000","last_peered":"2024-02-10T08:37:49.969195+0000","last_clean":"2024-02-10T08:37:49.969195+0000","last_became_active":"2024-02-10T08:36:37.065893+0000","last_became_peered":"2024-02-10T08:36:37.065893+0000","last_unstale":"2024-02-10T08:37:49.969195+0000","last_undegraded":"2024-02-10T08:37:49.969195+0000","last_fullsized":"2024-02-10T08:37:49.969195+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":148,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.963836+0000","last_change":"2024-02-10T08:36:32.709476+0000","last_active":"2024-02-10T08:37:49.963836+0000","last_peered":"2024-02-10T08:37:49.963836+0000","last_clean":"2024-02-10T08:37:49.963836+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:37:49.963836+0000","last_undegraded":"2024-02-10T08:37:49.963836+0000","last_fullsized":"2024-02-10T08:37:49.963836+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:36:32.709392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.709392+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":149,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968990+0000","last_change":"2024-02-10T08:37:10.243920+0000","last_active":"2024-02-10T08:37:49.968990+0000","last_peered":"2024-02-10T08:37:49.968990+0000","last_clean":"2024-02-10T08:37:49.968990+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:37:49.968990+0000","last_undegraded":"2024-02-10T08:37:49.968990+0000","last_fullsized":"2024-02-10T08:37:49.968990+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.107579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.107579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.968780+0000","last_change":"2024-02-10T08:37:22.720373+0000","last_active":"2024-02-10T08:37:49.968780+0000","last_peered":"2024-02-10T08:37:49.968780+0000","last_clean":"2024-02-10T08:37:49.968780+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:37:49.968780+0000","last_undegraded":"2024-02-10T08:37:49.968780+0000","last_fullsized":"2024-02-10T08:37:49.968780+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.720295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.720295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":53,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980417+0000","last_change":"2024-02-10T08:37:10.149854+0000","last_active":"2024-02-10T08:37:49.980417+0000","last_peered":"2024-02-10T08:37:49.980417+0000","last_clean":"2024-02-10T08:37:49.980417+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:37:49.980417+0000","last_undegraded":"2024-02-10T08:37:49.980417+0000","last_fullsized":"2024-02-10T08:37:49.980417+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.970259+0000","last_change":"2024-02-10T08:36:37.078717+0000","last_active":"2024-02-10T08:37:49.970259+0000","last_peered":"2024-02-10T08:37:49.970259+0000","last_clean":"2024-02-10T08:37:49.970259+0000","last_became_active":"2024-02-10T08:36:37.077729+0000","last_became_peered":"2024-02-10T08:36:37.077729+0000","last_unstale":"2024-02-10T08:37:49.970259+0000","last_undegraded":"2024-02-10T08:37:49.970259+0000","last_fullsized":"2024-02-10T08:37:49.970259+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":173,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.960957+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:37:49.960957+0000","last_peered":"2024-02-10T08:37:49.960957+0000","last_clean":"2024-02-10T08:37:49.960957+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:37:49.960957+0000","last_undegraded":"2024-02-10T08:37:49.960957+0000","last_fullsized":"2024-02-10T08:37:49.960957+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":148,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.972616+0000","last_change":"2024-02-10T08:37:10.306481+0000","last_active":"2024-02-10T08:37:49.972616+0000","last_peered":"2024-02-10T08:37:49.972616+0000","last_clean":"2024-02-10T08:37:49.972616+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:37:49.972616+0000","last_undegraded":"2024-02-10T08:37:49.972616+0000","last_fullsized":"2024-02-10T08:37:49.972616+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.014690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.014690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.962889+0000","last_change":"2024-02-10T08:36:14.747640+0000","last_active":"2024-02-10T08:37:49.962889+0000","last_peered":"2024-02-10T08:37:49.962889+0000","last_clean":"2024-02-10T08:37:49.962889+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:37:49.962889+0000","last_undegraded":"2024-02-10T08:37:49.962889+0000","last_fullsized":"2024-02-10T08:37:49.962889+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.747521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.747521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":92,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.980599+0000","last_change":"2024-02-10T08:37:10.145097+0000","last_active":"2024-02-10T08:37:49.980599+0000","last_peered":"2024-02-10T08:37:49.980599+0000","last_clean":"2024-02-10T08:37:49.980599+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:37:49.980599+0000","last_undegraded":"2024-02-10T08:37:49.980599+0000","last_fullsized":"2024-02-10T08:37:49.980599+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.971713+0000","last_change":"2024-02-10T08:36:37.164878+0000","last_active":"2024-02-10T08:37:49.971713+0000","last_peered":"2024-02-10T08:37:49.971713+0000","last_clean":"2024-02-10T08:37:49.971713+0000","last_became_active":"2024-02-10T08:36:37.163315+0000","last_became_peered":"2024-02-10T08:36:37.163315+0000","last_unstale":"2024-02-10T08:37:49.971713+0000","last_undegraded":"2024-02-10T08:37:49.971713+0000","last_fullsized":"2024-02-10T08:37:49.971713+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":187,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.967033+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:37:49.967033+0000","last_peered":"2024-02-10T08:37:49.967033+0000","last_clean":"2024-02-10T08:37:49.967033+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:37:49.967033+0000","last_undegraded":"2024-02-10T08:37:49.967033+0000","last_fullsized":"2024-02-10T08:37:49.967033+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":58,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:50.656236+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:37:50.656236+0000","last_peered":"2024-02-10T08:37:50.656236+0000","last_clean":"2024-02-10T08:37:50.656236+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:37:50.656236+0000","last_undegraded":"2024-02-10T08:37:50.656236+0000","last_fullsized":"2024-02-10T08:37:50.656236+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.976282+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:37:49.976282+0000","last_peered":"2024-02-10T08:37:49.976282+0000","last_clean":"2024-02-10T08:37:49.976282+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:37:49.976282+0000","last_undegraded":"2024-02-10T08:37:49.976282+0000","last_fullsized":"2024-02-10T08:37:49.976282+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":83,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-02-10T08:37:49.975733+0000","last_change":"2024-02-10T08:36:37.083207+0000","last_active":"2024-02-10T08:37:49.975733+0000","last_peered":"2024-02-10T08:37:49.975733+0000","last_clean":"2024-02-10T08:37:49.975733+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:37:49.975733+0000","last_undegraded":"2024-02-10T08:37:49.975733+0000","last_fullsized":"2024-02-10T08:37:49.975733+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":121,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":119,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":118,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":7},{"poolid":95,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2715648,"data_stored":6843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":7},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":6272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":5227,"num_objects":505,"num_object_clones":400,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6209536,"data_stored":15689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309300,"kb_used_data":3880,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435852,"statfs":{"total":95995035648,"available":95678312448,"internally_reserved":0,"allocated":3973120,"data_stored":1399954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":0.76200000000000001,"15min":0.60199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":22.143000000000001,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.361,"5min":0.68700000000000006,"15min":0.57699999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.7850000000000001,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.60899999999999999,"15min":0.54400000000000004},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.593,"5min":7.593,"15min":7.593},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.68400000000000005,"15min":0.59699999999999998},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.7759999999999998,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.65100000000000002,"15min":0.55100000000000005},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.9099999999999999,"5min":0.84399999999999997,"15min":0.64900000000000002},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":19.369,"5min":19.369,"15min":19.369},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.69699999999999995,"15min":0.56499999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.9710000000000001,"5min":0.83199999999999996,"15min":0.63200000000000001},"min":{"1min":0.34899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":22.102,"5min":22.102,"15min":22.102},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.50700000000000001,"15min":0.45200000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.827,"5min":8.827,"15min":8.827},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.03,"5min":0.56799999999999995,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.646000000000001,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.29599999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.64300000000000002,"15min":0.505},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":19.326000000000001,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.316},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.41899999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.71199999999999997,"15min":0.56299999999999994},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":22.687000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.35699999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561087,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300888,"kb_used_data":3660,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444264,"statfs":{"total":95995035648,"available":95686926336,"internally_reserved":0,"allocated":3747840,"data_stored":1399348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.29599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.503},"min":{"1min":0.34499999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.441},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.501},"min":{"1min":0.31,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.53000000000000003,"15min":0.49399999999999999},"min":{"1min":0.39500000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.49299999999999999},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.39800000000000002,"15min":0.36399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55300000000000005,"5min":0.63,"15min":0.63},"last":0.28699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.41499999999999998,"15min":0.379},"min":{"1min":0.26000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.313}]},{"osd":7,"last update":"Sat Feb 10 08:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.46999999999999997,"15min":0.45900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64900000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.39000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301120,"kb_used_data":3896,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444032,"statfs":{"total":95995035648,"available":95686688768,"internally_reserved":0,"allocated":3989504,"data_stored":1409873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.41299999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.249,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.255}]},{"osd":2,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":1.103,"15min":0.61799999999999999},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":34.985999999999997,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.9609999999999999,"5min":0.879,"15min":0.53700000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.31}]},{"osd":3,"last update":"Sat Feb 10 08:37:43 2024","interfaces":[{"interface":"back","average":{"1min":2.323,"5min":0.76400000000000001,"15min":0.54400000000000004},"min":{"1min":0.246,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":24.773,"5min":24.773,"15min":24.773},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.9750000000000001,"5min":0.72399999999999998,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":16.120999999999999,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:37:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.44}]},{"osd":5,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.67700000000000005,"15min":0.61899999999999999},"min":{"1min":0.42099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.5600000000000005,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.609,"5min":0.79100000000000004,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.8479999999999999,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.45600000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.312,"15min":0.312},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.64200000000000002,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.0220000000000002,"5min":6.0220000000000002,"15min":6.0220000000000002},"last":0.53200000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":0.71399999999999997,"15min":0.59399999999999997},"min":{"1min":0.35199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":9.4640000000000004,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.841,"5min":0.84099999999999997,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.9199999999999999,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.41599999999999998}]}]},{"osd":0,"up_from":0,"seq":34359738460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300848,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444304,"statfs":{"total":95995035648,"available":95686967296,"internally_reserved":0,"allocated":3710976,"data_stored":1398806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.44800000000000001,"15min":0.38500000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.187},"max":{"1min":2.8660000000000001,"5min":2.8660000000000001,"15min":2.8660000000000001},"last":3.3330000000000002},{"interface":"front","average":{"1min":1.4490000000000001,"5min":0.58499999999999996,"15min":0.438},"min":{"1min":0.22,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":1.012}]},{"osd":1,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":0.54400000000000004,"15min":0.42399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.0239999999999991,"5min":8.0239999999999991,"15min":8.0239999999999991},"last":12.512},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.501,"15min":0.42299999999999999},"min":{"1min":0.246,"5min":0.22600000000000001,"15min":0.223},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":9.6769999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.48999999999999999,"15min":0.44600000000000001},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.22},"max":{"1min":4.617,"5min":4.617,"15min":4.617},"last":2.1949999999999998},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.54200000000000004,"15min":0.47399999999999998},"min":{"1min":0.307,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1109999999999998,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":2.6139999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":0.74399999999999999,"15min":0.64200000000000002},"min":{"1min":0.52700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":9.2370000000000001},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81100000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:37:32 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":0.77500000000000002,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":7.4859999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.65500000000000003,"15min":0.60099999999999998},"min":{"1min":0.35999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":1.8540000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":0.72699999999999998,"15min":0.63500000000000001},"min":{"1min":0.48799999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.266,"5min":9.266,"15min":9.266},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.45300000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.155}]},{"osd":7,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.79700000000000004,"15min":0.69199999999999995},"min":{"1min":0.56699999999999995,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":2.7090000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.5350000000000001,"5min":4.5350000000000001,"15min":4.5350000000000001},"last":4.944}]}]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300884,"kb_used_data":3660,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444268,"statfs":{"total":95995035648,"available":95686930432,"internally_reserved":0,"allocated":3747840,"data_stored":1399331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":0.82499999999999996,"15min":0.52500000000000002},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":11.550000000000001,"5min":14.744,"15min":14.744},"last":0.41299999999999998},{"interface":"front","average":{"1min":2.7389999999999999,"5min":0.88,"15min":0.55300000000000005},"min":{"1min":0.27900000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.808,"5min":11.808,"15min":11.808},"last":0.442}]},{"osd":1,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.9350000000000001,"5min":0.82199999999999995,"15min":0.56100000000000005},"min":{"1min":0.27300000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":10.444000000000001,"5min":16.634,"15min":16.634},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.7330000000000001,"5min":0.83299999999999996,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":11.494999999999999,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.43099999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":0.873,"15min":0.60499999999999998},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":11.516999999999999,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":2.8180000000000001,"5min":1.0860000000000001,"15min":0.64300000000000002},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":17.722000000000001,"5min":23.294,"15min":23.294},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:37:06 2024","interfaces":[{"interface":"back","average":{"1min":2.129,"5min":0.86899999999999999,"15min":0.70499999999999996},"min":{"1min":0.38900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":12.371,"5min":12.371,"15min":12.371},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.7190000000000001,"5min":0.77700000000000002,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":0.97499999999999998,"15min":0.70399999999999996},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":12.579000000000001,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.26,"5min":0.78500000000000003,"15min":0.65300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":0.62}]},{"osd":6,"last update":"Sat Feb 10 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":0.81100000000000005,"15min":0.627},"min":{"1min":0.502,"5min":0.374,"15min":0.374},"max":{"1min":16.091000000000001,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.7969999999999999,"5min":0.81000000000000005,"15min":0.63700000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":16.475000000000001,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1829999999999998,"5min":0.88500000000000001,"15min":0.65300000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":14.773999999999999,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":2.5659999999999998,"5min":0.96799999999999997,"15min":0.68600000000000005},"min":{"1min":0.442,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.606999999999999,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.64600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986248,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309056,"kb_used_data":3640,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436096,"statfs":{"total":95995035648,"available":95678562304,"internally_reserved":0,"allocated":3727360,"data_stored":1407930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":0.80200000000000005,"15min":0.63400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":19.940999999999999,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.157,"5min":0.623,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":9.1760000000000002,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.441}]},{"osd":1,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":0.67900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.6259999999999999,"5min":0.73199999999999998,"15min":0.61399999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.74399999999999999,"15min":0.60999999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.311},{"interface":"front","average":{"1min":1.373,"5min":0.68700000000000006,"15min":0.59099999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.202,"5min":9.202,"15min":9.202},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1880000000000002,"5min":0.84099999999999997,"15min":0.64500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.5909999999999993,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.46000000000000002},{"interface":"front","average":{"1min":3.1360000000000001,"5min":1.044,"15min":0.70399999999999996},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":25.27,"5min":25.27,"15min":25.27},"last":0.40300000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:37:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.501,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.45600000000000002,"15min":0.40500000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":3.8290000000000002,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.38400000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":0.57699999999999996,"15min":0.45300000000000001},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":13.342000000000001,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.48699999999999999,"15min":0.45100000000000001},"min":{"1min":0.33400000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.432}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.45200000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790020,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308868,"kb_used_data":3448,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436284,"statfs":{"total":95995035648,"available":95678754816,"internally_reserved":0,"allocated":3530752,"data_stored":1402694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9880000000000004,"5min":2.0009999999999999,"15min":1.0069999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":107.551,"5min":107.551,"15min":107.551},"last":0.41799999999999998},{"interface":"front","average":{"1min":7.3719999999999999,"5min":1.8720000000000001,"15min":0.97599999999999998},"min":{"1min":0.27700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":88.792000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":6.444,"5min":1.7170000000000001,"15min":0.94999999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":107.205,"5min":107.205,"15min":107.205},"last":0.375},{"interface":"front","average":{"1min":4.7750000000000004,"5min":1.3919999999999999,"15min":0.83999999999999997},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":72.272000000000006,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.246}]},{"osd":2,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":7.274,"5min":1.8640000000000001,"15min":0.95799999999999996},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":54.174999999999997,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":7.6349999999999998,"5min":1.9510000000000001,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":86.183000000000007,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":3.5299999999999998,"5min":1.119,"15min":0.70899999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":44.395000000000003,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":10.414,"5min":2.528,"15min":1.2090000000000001},"min":{"1min":0.35799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":114.526,"5min":114.526,"15min":114.526},"last":0.29099999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":4.0199999999999996,"5min":1.169,"15min":0.65100000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":72.216999999999999,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":2.617,"5min":0.89500000000000002,"15min":0.55100000000000005},"min":{"1min":0.27000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":44.345999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.40899999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:36:56 2024","interfaces":[{"interface":"back","average":{"1min":4.1600000000000001,"5min":1.1890000000000001,"15min":0.67100000000000004},"min":{"1min":0.28799999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":72.241,"5min":72.241,"15min":72.241},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.712,"5min":0.72599999999999998,"15min":0.54600000000000004},"min":{"1min":0.33700000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":22.469999999999999,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.316}]},{"osd":7,"last update":"Sat Feb 10 08:37:18 2024","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":1.1830000000000001,"15min":0.71099999999999997},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":66.844999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":4.1180000000000003,"5min":1.248,"15min":0.73999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":72.257000000000005,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":2297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":872448,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":897024,"data_stored":2259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1003520,"data_stored":2548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":2032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1060864,"data_stored":2675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":1357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":119,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":121,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:37:53.646 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:37:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:53.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:37:53.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:52.472643+0000 mgr.y (mgr.14553) 215 : cluster [DBG] pgmap v243: 322 pgs: 32 creating+peering, 290 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:53.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.833892+0000 mgr.y (mgr.14553) 216 : audit [DBG] from='client.27333 v1:172.21.15.107:0/2884148211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:37:53.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:52.925661+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:37:53.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.935883+0000 mon.a (mon.0) 1313 : audit [INF] from='client.52739 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi107-112525-15"}]': finished 2024-02-10T08:37:53.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:52.936011+0000 mon.a (mon.0) 1314 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.941182+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.107:0/3076020731' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.943451+0000 mon.a (mon.0) 1315 : audit [INF] from='client.52826 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.973123+0000 mon.c (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.973938+0000 mon.a (mon.0) 1316 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.978755+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.979270+0000 mon.a (mon.0) 1317 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.981034+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.981762+0000 mon.c (mon.2) 220 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.982334+0000 mon.a (mon.0) 1318 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.982762+0000 mon.a (mon.0) 1319 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.983722+0000 mon.c (mon.2) 221 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.984058+0000 mon.a (mon.0) 1320 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:53.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.984652+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.742 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:52.984932+0000 mon.a (mon.0) 1321 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:37:53.742 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.028746+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:53.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:53.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:37:53.782 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:53.802 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:53.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:53.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:53.934 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:53.966 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:53.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:54.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:54.096 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:54.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:54.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:54.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:54.244 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:54.262 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:54.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:54.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:37:54.395 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:52.685143+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:52.685206+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.935304+0000 mon.a (mon.0) 1322 : audit [INF] from='client.52826 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.935469+0000 mon.a (mon.0) 1323 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.935541+0000 mon.a (mon.0) 1324 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:53.935599+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.942809+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.943087+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.943216+0000 mon.a (mon.0) 1326 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.943315+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.943733+0000 mon.a (mon.0) 1327 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:53.944036+0000 mon.a (mon.0) 1328 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:54.029662+0000 mon.b (mon.1) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:52.685143+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:52.685206+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.935304+0000 mon.a (mon.0) 1322 : audit [INF] from='client.52826 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.935469+0000 mon.a (mon.0) 1323 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.935541+0000 mon.a (mon.0) 1324 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:53.935599+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.942809+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.943087+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.943216+0000 mon.a (mon.0) 1326 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.943315+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.943733+0000 mon.a (mon.0) 1327 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:53.944036+0000 mon.a (mon.0) 1328 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:54.029662+0000 mon.b (mon.1) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:52.685143+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:52.685206+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.935304+0000 mon.a (mon.0) 1322 : audit [INF] from='client.52826 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi107-112411-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.935469+0000 mon.a (mon.0) 1323 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi107-112403-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.935541+0000 mon.a (mon.0) 1324 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:53.935599+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-10T08:37:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.942809+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.943087+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.943216+0000 mon.a (mon.0) 1326 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.943315+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.943733+0000 mon.a (mon.0) 1327 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:53.944036+0000 mon.a (mon.0) 1328 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:37:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:54.029662+0000 mon.b (mon.1) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:54.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:37:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:37:55] "GET /metrics HTTP/1.1" 200 187220 "" "Prometheus/2.33.4" 2024-02-10T08:37:55.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:54.473056+0000 mgr.y (mgr.14553) 217 : cluster [DBG] pgmap v246: 282 pgs: 32 unknown, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.030762+0000 mon.b (mon.1) 253 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.185614+0000 mon.a (mon.0) 1329 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.186237+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:55.420685+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.421421+0000 mon.a (mon.0) 1331 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:54.473056+0000 mgr.y (mgr.14553) 217 : cluster [DBG] pgmap v246: 282 pgs: 32 unknown, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.030762+0000 mon.b (mon.1) 253 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.185614+0000 mon.a (mon.0) 1329 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.186237+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:55.420685+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-10T08:37:55.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.421421+0000 mon.a (mon.0) 1331 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:54.473056+0000 mgr.y (mgr.14553) 217 : cluster [DBG] pgmap v246: 282 pgs: 32 unknown, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.030762+0000 mon.b (mon.1) 253 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.185614+0000 mon.a (mon.0) 1329 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.186237+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.107:0/1916230586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:55.420685+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-10T08:37:56.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.421421+0000 mon.a (mon.0) 1331 : audit [INF] from='client.52784 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]: dispatch 2024-02-10T08:37:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.945859+0000 mon.a (mon.0) 1332 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.945969+0000 mon.a (mon.0) 1333 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:37:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.946036+0000 mon.a (mon.0) 1334 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:55.946120+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.953548+0000 mon.c (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.107:0/1051324779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:55.954410+0000 mon.a (mon.0) 1336 : audit [INF] from='client.52850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:56.008621+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:56.009270+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:56.031860+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:56.474215+0000 mgr.y (mgr.14553) 218 : cluster [DBG] pgmap v249: 298 pgs: 48 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.945859+0000 mon.a (mon.0) 1332 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.945969+0000 mon.a (mon.0) 1333 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.946036+0000 mon.a (mon.0) 1334 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:55.946120+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.953548+0000 mon.c (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.107:0/1051324779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:55.954410+0000 mon.a (mon.0) 1336 : audit [INF] from='client.52850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:56.008621+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:56.009270+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:56.031860+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:56.474215+0000 mgr.y (mgr.14553) 218 : cluster [DBG] pgmap v249: 298 pgs: 48 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.945859+0000 mon.a (mon.0) 1332 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi107-112403-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.945969+0000 mon.a (mon.0) 1333 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:37:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.946036+0000 mon.a (mon.0) 1334 : audit [INF] from='client.52784 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112480-24"}]': finished 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:55.946120+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.953548+0000 mon.c (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.107:0/1051324779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:55.954410+0000 mon.a (mon.0) 1336 : audit [INF] from='client.52850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:56.008621+0000 osd.3 (osd.3) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:56.009270+0000 osd.3 (osd.3) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:56.031860+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:57.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:56.474215+0000 mgr.y (mgr.14553) 218 : cluster [DBG] pgmap v249: 298 pgs: 48 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:37:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:58.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:37:58.102 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:58.120 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:37:58.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:37:58.185 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:56.952225+0000 mon.a (mon.0) 1337 : audit [INF] from='client.52850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:56.952408+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-10T08:37:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:56.962293+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.107:0/1851412551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:56.962818+0000 mon.a (mon.0) 1339 : audit [INF] from='client.52856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:57.032985+0000 mon.b (mon.1) 255 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:56.952225+0000 mon.a (mon.0) 1337 : audit [INF] from='client.52850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:56.952408+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:56.962293+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.107:0/1851412551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:56.962818+0000 mon.a (mon.0) 1339 : audit [INF] from='client.52856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:57.032985+0000 mon.b (mon.1) 255 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:56.952225+0000 mon.a (mon.0) 1337 : audit [INF] from='client.52850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi107-112411-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:56.952408+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:56.962293+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.107:0/1851412551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:56.962818+0000 mon.a (mon.0) 1339 : audit [INF] from='client.52856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:58.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:57.032985+0000 mon.b (mon.1) 255 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:37:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:58.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:37:58.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:37:58.520 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:58.539 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:37:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:58.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:37:58.639 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:37:58.639 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:37:58.639 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:37:58.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:37:58.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:58.672 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:58.692 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:58.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:58.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:58.823 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:58.841 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:58.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:58.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:58.993 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:59.242 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:59.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:57.955276+0000 mon.a (mon.0) 1340 : audit [INF] from='client.52856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:59.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:57.955497+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:57.965551+0000 mon.c (mon.2) 229 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:57.966273+0000 mon.a (mon.0) 1342 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:58.033909+0000 mon.b (mon.1) 256 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:58.474687+0000 mgr.y (mgr.14553) 219 : cluster [DBG] pgmap v252: 290 pgs: 32 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:58.956504+0000 mon.a (mon.0) 1343 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:58.956671+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:58.960077+0000 mon.c (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:58.962308+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:58.962915+0000 mon.a (mon.0) 1346 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.244 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:57.955276+0000 mon.a (mon.0) 1340 : audit [INF] from='client.52856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:57.955497+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:57.965551+0000 mon.c (mon.2) 229 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:57.966273+0000 mon.a (mon.0) 1342 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:58.033909+0000 mon.b (mon.1) 256 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:58.474687+0000 mgr.y (mgr.14553) 219 : cluster [DBG] pgmap v252: 290 pgs: 32 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:58.956504+0000 mon.a (mon.0) 1343 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:58.956671+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:58.960077+0000 mon.c (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:58.962308+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:59.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:58.962915+0000 mon.a (mon.0) 1346 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:59.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:57.955276+0000 mon.a (mon.0) 1340 : audit [INF] from='client.52856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi107-112480-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:57.955497+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:57.965551+0000 mon.c (mon.2) 229 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:57.966273+0000 mon.a (mon.0) 1342 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:58.033909+0000 mon.b (mon.1) 256 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:58.474687+0000 mgr.y (mgr.14553) 219 : cluster [DBG] pgmap v252: 290 pgs: 32 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:58.956504+0000 mon.a (mon.0) 1343 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:58.956671+0000 mon.a (mon.0) 1344 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-10T08:37:59.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:58.960077+0000 mon.c (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.107:0/2924108889' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:58.962308+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:37:59.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:58.962915+0000 mon.a (mon.0) 1346 : audit [INF] from='client.52838 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]: dispatch 2024-02-10T08:37:59.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:37:59.375 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:59.394 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:37:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:59.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:37:59.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:37:59.524 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:59.542 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:37:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:59.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:37:59.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:37:59.671 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_ api_aio: Running main() from gmock_main.cc 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] Global test environment set-up. 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.TooBig (7750 ms) 2024-02-10T08:37:59.969 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (5067 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2909 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3243 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2799 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4091 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3093 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3136 ms) 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-10T08:37:59.970 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2876 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2873 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2916 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3089 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3250 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2691 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.Flush (2970 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3005 ms) 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-10T08:37:59.971 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3002 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3007 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3010 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3007 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2957 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3026 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3010 ms) 2024-02-10T08:37:59.972 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3009 ms) 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2978 ms) 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] 25 tests from LibRadosAio (82766 ms total) 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-02-10T08:37:59.973 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7019 ms) 2024-02-10T08:37:59.973 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:37:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:37:59.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:00.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:00.110 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:00.131 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:00.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:00.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.034948+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.958068+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.958220+0000 mon.a (mon.0) 1348 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:37:59.958375+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.976015+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2429415340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.976504+0000 mon.a (mon.0) 1350 : audit [INF] from='client.52868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.034948+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.958068+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.958220+0000 mon.a (mon.0) 1348 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:37:59.958375+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.976015+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2429415340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:37:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.976504+0000 mon.a (mon.0) 1350 : audit [INF] from='client.52868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.034948+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.958068+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.107:0/2589364342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi107-112411-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.958220+0000 mon.a (mon.0) 1348 : audit [INF] from='client.52838 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi107-112403-26"}]': finished 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:37:59.958375+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.976015+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.107:0/2429415340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:37:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.976504+0000 mon.a (mon.0) 1350 : audit [INF] from='client.52868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:00.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:00.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:00.443 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:00.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:00.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:00.614 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:01.162 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:01.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:01.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:01.293 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:01.312 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:01.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:01.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.995309+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.995943+0000 mon.a (mon.0) 1351 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.997315+0000 mon.c (mon.2) 233 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.997626+0000 mon.a (mon.0) 1352 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.998080+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:37:59.998336+0000 mon.a (mon.0) 1353 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.035956+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.350027+0000 mon.a (mon.0) 1354 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:00.475214+0000 mgr.y (mgr.14553) 220 : cluster [DBG] pgmap v255: 322 pgs: 64 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.929921+0000 mon.a (mon.0) 1355 : audit [INF] from='client.52868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.930029+0000 mon.a (mon.0) 1356 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:00.930086+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.940822+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:00.947811+0000 mon.a (mon.0) 1358 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.995309+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.995943+0000 mon.a (mon.0) 1351 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.997315+0000 mon.c (mon.2) 233 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.997626+0000 mon.a (mon.0) 1352 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.998080+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:37:59.998336+0000 mon.a (mon.0) 1353 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.035956+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.350027+0000 mon.a (mon.0) 1354 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:00.475214+0000 mgr.y (mgr.14553) 220 : cluster [DBG] pgmap v255: 322 pgs: 64 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.929921+0000 mon.a (mon.0) 1355 : audit [INF] from='client.52868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.930029+0000 mon.a (mon.0) 1356 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:00.930086+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.940822+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:00.947811+0000 mon.a (mon.0) 1358 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.995309+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.995943+0000 mon.a (mon.0) 1351 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.997315+0000 mon.c (mon.2) 233 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.997626+0000 mon.a (mon.0) 1352 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.998080+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:37:59.998336+0000 mon.a (mon.0) 1353 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.035956+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.350027+0000 mon.a (mon.0) 1354 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:00.475214+0000 mgr.y (mgr.14553) 220 : cluster [DBG] pgmap v255: 322 pgs: 64 unknown, 8 creating+peering, 250 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.929921+0000 mon.a (mon.0) 1355 : audit [INF] from='client.52868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi107-112480-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.930029+0000 mon.a (mon.0) 1356 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi107-112403-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:00.930086+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.940822+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:00.947811+0000 mon.a (mon.0) 1358 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.555+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/1131217407 >> v1:172.21.15.107:6790/0 conn(0x7fd3cc1007f0 legacy=0x7fd3cc102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/1131217407 shutdown_connections 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/1131217407 >> 172.21.15.107:0/1131217407 conn(0x7fd3cc0f3640 msgr2=0x7fd3cc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/1131217407 shutdown_connections 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/1131217407 wait complete. 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 librados: starting msgr at 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.556+0000 7fd3d1d7d700 1 librados: starting objecter 2024-02-10T08:38:01.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 Processor -- start 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 -- start start 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 librados: setting wanted keys 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 librados: calling monclient init 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd3cc117330 con 0x7fd3cc1007f0 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd3cc118510 con 0x7fd3cc0fda50 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3d1d7d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd3cc119710 con 0x7fd3cc104330 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3caffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd3cc1007f0 0x7fd3cc0fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42928/0 (socket says 172.21.15.107:42928) 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3caffd700 1 -- 172.21.15.107:0/2050385615 learned_addr learned my addr 172.21.15.107:0/2050385615 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd3cc104330 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd3cc1007f0 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.557+0000 7fd3c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 10 client.?.objecter ms_handle_connect 0x7fd3cc0fda50 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728855995 0 0) 0x7fd3cc119710 con 0x7fd3cc104330 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b80033f0 con 0x7fd3cc104330 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693056116 0 0) 0x7fd3cc117330 con 0x7fd3cc1007f0 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc119710 con 0x7fd3cc1007f0 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332850247 0 0) 0x7fd3cc118510 con 0x7fd3cc0fda50 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc117330 con 0x7fd3cc0fda50 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444907661 0 0) 0x7fd3b80033f0 con 0x7fd3cc104330 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3cc118510 con 0x7fd3cc104330 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd3c4001c30 con 0x7fd3cc104330 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824459891 0 0) 0x7fd3cc119710 con 0x7fd3cc1007f0 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3b80033f0 con 0x7fd3cc1007f0 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd3b4000ed0 con 0x7fd3cc1007f0 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173711664 0 0) 0x7fd3cc117330 con 0x7fd3cc0fda50 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3cc119710 con 0x7fd3cc0fda50 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd3bc001840 con 0x7fd3cc0fda50 2024-02-10T08:38:01.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3884958461 0 0) 0x7fd3cc118510 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.195:6789/0 conn(0x7fd3cc0fda50 legacy=0x7fd3cc0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6789/0 conn(0x7fd3cc1007f0 legacy=0x7fd3cc0fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc11a910 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc117520 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3d1d7d700 10 client.52883.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc117990 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3d1d7d700 1 librados: init done 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd3c40026b0 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.558+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd3c40036b0 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.559+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd3ac004f00 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.559+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd3c40038b0 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(187..187 src has 50..187) v4 ==== 9839+0+0 (unknown 1492845850 0 0) 0x7fd3c4076f50 con 0x7fd3cc104330 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 osd_map(187..187 src has 50..187) v4 2024-02-10T08:38:01.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 3 client.52883.objecter handle_osd_map got epochs [187,187] > 0 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 3 client.52883.objecter handle_osd_map decoding full epoch 187 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 20 client.52883.objecter dump_active .. 0 homeless 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v187) v1 ==== 54+0+16 (unknown 2745239193 0 2420000192) 0x7fd3c4079360 con 0x7fd3cc104330 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3b80680e0 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:01.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 1 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 1 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 0 1 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 4 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 1 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac004950 con 0x7fd3ac004530 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.560+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac004530 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.561+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac004950 con 0x7fd3ac004530 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.561+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.561+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 1 = system:0 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.561+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 4 2024-02-10T08:38:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.561+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 0 1 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 2 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 2 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac0051a0 osd=0 3 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 0 2 2024-02-10T08:38:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 4 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 2 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.587+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac004d30 con 0x7fd3ac004530 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac004d30 con 0x7fd3ac004530 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 4 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 0 2 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 3 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 3 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 1 3 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 4 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 3 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.589+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac00c000 con 0x7fd3ac004150 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac004150 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac00c000 con 0x7fd3ac004150 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 3 = system:0 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 4 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.590+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 1 3 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 4 2024-02-10T08:38:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 4 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003fe0 osd=1 3 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 1 4 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 4 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 4 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.605+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac0018b0 con 0x7fd3ac004150 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac0018b0 con 0x7fd3ac004150 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 4 2024-02-10T08:38:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 1 4 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 5 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 5 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 2 5 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 4 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 5 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac00de90 con 0x7fd3ac00d2f0 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.607+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac00d2f0 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.608+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac00de90 con 0x7fd3ac00d2f0 2024-02-10T08:38:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.608+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.608+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 5 = system:0 2024-02-10T08:38:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.608+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 4 2024-02-10T08:38:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.608+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 2 5 2024-02-10T08:38:01.621 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 6 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 6 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003630 osd=2 3 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 2 6 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 4 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 6 2024-02-10T08:38:01.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.622+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac0030a0 con 0x7fd3ac00d2f0 2024-02-10T08:38:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:01.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac0030a0 con 0x7fd3ac00d2f0 2024-02-10T08:38:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 4 2024-02-10T08:38:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 2 6 2024-02-10T08:38:01.626 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 7 2024-02-10T08:38:01.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 7 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 3 7 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 4 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 7 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.624+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac015a60 con 0x7fd3ac011e80 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac011e80 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac015a60 con 0x7fd3ac011e80 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 7 = system:0 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 4 2024-02-10T08:38:01.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.625+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 3 7 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 8 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 8 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac003480 osd=3 3 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 3 8 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 4 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 8 2024-02-10T08:38:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.642+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac00e270 con 0x7fd3ac011e80 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac00e270 con 0x7fd3ac011e80 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 4 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 3 8 2024-02-10T08:38:01.645 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 9 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 9 2024-02-10T08:38:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 4 9 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 4 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 9 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac00e190 con 0x7fd3ac00ce50 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.645+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac00ce50 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.646+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac00e190 con 0x7fd3ac00ce50 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.646+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.646+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 9 = system:0 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.646+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 4 2024-02-10T08:38:01.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.646+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 4 9 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 10 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 10 2024-02-10T08:38:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac00e690 osd=4 3 2024-02-10T08:38:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 4 10 2024-02-10T08:38:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 4 2024-02-10T08:38:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 10 2024-02-10T08:38:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.661+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac00e190 con 0x7fd3ac00ce50 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac00e190 con 0x7fd3ac00ce50 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 4 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 4 10 2024-02-10T08:38:01.663 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 11 2024-02-10T08:38:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 11 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 5 11 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 4 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 11 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac016db0 con 0x7fd3ac016830 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.663+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac016830 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.664+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac016db0 con 0x7fd3ac016830 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.664+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.664+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 11 = system:0 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.664+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 4 2024-02-10T08:38:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.664+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 5 11 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 12 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 12 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac00e190 osd=5 3 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 5 12 2024-02-10T08:38:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 4 2024-02-10T08:38:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 12 2024-02-10T08:38:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.679+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac016db0 con 0x7fd3ac016830 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac016db0 con 0x7fd3ac016830 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 4 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 5 12 2024-02-10T08:38:01.681 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.680+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 13 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 13 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 6 13 2024-02-10T08:38:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 4 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 13 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac022b70 con 0x7fd3ac01f060 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.681+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac01f060 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.682+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac022b70 con 0x7fd3ac01f060 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.682+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.682+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 13 = system:0 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.682+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 4 2024-02-10T08:38:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.682+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 6 13 2024-02-10T08:38:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 14 2024-02-10T08:38:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 14 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac016db0 osd=6 3 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 6 14 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 4 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 14 2024-02-10T08:38:01.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.696+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac0242f0 con 0x7fd3ac01f060 2024-02-10T08:38:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac0242f0 con 0x7fd3ac01f060 2024-02-10T08:38:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 4 2024-02-10T08:38:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 6 14 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 15 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 15 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 7 15 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 4 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 15 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3ac024f70 con 0x7fd3ac0249b0 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.698+0000 7fd3c8ff9700 10 client.52883.objecter ms_handle_connect 0x7fd3ac0249b0 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.699+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd3ac024f70 con 0x7fd3ac0249b0 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.699+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:01.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.699+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 15 = system:0 2024-02-10T08:38:01.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.699+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 4 2024-02-10T08:38:01.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.699+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 7 15 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 10 client.52883.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign -1 16 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 20 client.52883.objecter _get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_remove -1 16 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 20 client.52883.objecter get_session s=0x7fd3ac0242f0 osd=7 3 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 15 client.52883.objecter _session_command_op_assign 7 16 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 4 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 10 client.52883.objecter _send_command 16 2024-02-10T08:38:01.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.713+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd3ac024f30 con 0x7fd3ac0249b0 2024-02-10T08:38:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3c8ff9700 1 -- 172.21.15.107:0/2050385615 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd3ac024f30 con 0x7fd3ac0249b0 2024-02-10T08:38:01.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3c8ff9700 10 client.52883.objecter ms_dispatch 0x7fd3cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:01.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3c8ff9700 10 client.52883.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3c8ff9700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 4 2024-02-10T08:38:01.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3c8ff9700 15 client.52883.objecter _session_command_op_remove 7 16 2024-02-10T08:38:01.716 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:01.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 librados: watch_flush enter 2024-02-10T08:38:01.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 librados: watch_flush exit 2024-02-10T08:38:01.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.0 2024-02-10T08:38:01.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6801/3321596972 conn(0x7fd3ac004530 legacy=0x7fd3ac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0051a0 osd=0 1 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.1 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6805/133516126 conn(0x7fd3ac004150 legacy=0x7fd3ac00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003fe0 osd=1 1 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.2 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6809/3575881728 conn(0x7fd3ac00d2f0 legacy=0x7fd3ac00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003630 osd=2 1 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.3 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6813/2871825691 conn(0x7fd3ac011e80 legacy=0x7fd3ac014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac003480 osd=3 1 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.4 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.195:6800/2499671285 conn(0x7fd3ac00ce50 legacy=0x7fd3ac015bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e690 osd=4 1 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.5 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.195:6804/4178321486 conn(0x7fd3ac016830 legacy=0x7fd3ac01d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac00e190 osd=5 1 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.6 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.195:6808/1417479017 conn(0x7fd3ac01f060 legacy=0x7fd3ac021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac016db0 osd=6 1 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter close_session for osd.7 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.195:6812/1934176928 conn(0x7fd3ac0249b0 legacy=0x7fd3ac022b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter put_session s=0x7fd3ac0242f0 osd=7 1 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 20 client.52883.objecter shutdown clearing up homeless session... 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 10 client.52883.objecter successfully canceled tick 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3cbfff700 1 -- 172.21.15.107:0/2050385615 reap_dead start 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3cbfff700 1 -- 172.21.15.107:0/2050385615 reap_dead start 2024-02-10T08:38:01.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd3b80680e0 legacy=0x7fd3b806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> v1:172.21.15.107:6790/0 conn(0x7fd3cc104330 legacy=0x7fd3cc1159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.715+0000 7fd3cbfff700 1 -- 172.21.15.107:0/2050385615 reap_dead start 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3cbfff700 1 -- 172.21.15.107:0/2050385615 reap_dead start 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3cbfff700 1 -- 172.21.15.107:0/2050385615 reap_dead start 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 shutdown_connections 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 >> 172.21.15.107:0/2050385615 conn(0x7fd3cc0f3640 msgr2=0x7fd3cc1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 shutdown_connections 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3d1d7d700 1 -- 172.21.15.107:0/2050385615 wait complete. 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.716+0000 7fd3d1d7d700 1 librados: shutdown 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:38:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.629+0000 7fee26844700 1 -- 172.21.15.107:0/2677066862 >> v1:172.21.15.107:6790/0 conn(0x7fee20104330 legacy=0x7fee201067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 -- 172.21.15.107:0/2677066862 shutdown_connections 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 -- 172.21.15.107:0/2677066862 >> 172.21.15.107:0/2677066862 conn(0x7fee200f3640 msgr2=0x7fee200f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 -- 172.21.15.107:0/2677066862 shutdown_connections 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 -- 172.21.15.107:0/2677066862 wait complete. 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 librados: starting msgr at 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 librados: starting objecter 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 Processor -- start 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 -- start start 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 librados: setting wanted keys 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.630+0000 7fee26844700 1 librados: calling monclient init 2024-02-10T08:38:01.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee26844700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee20117400 con 0x7fee201007f0 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee26844700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee201185a0 con 0x7fee200fda50 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee26844700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee201197a0 con 0x7fee20104330 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1f7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fee201007f0 0x7fee200fa2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42946/0 (socket says 172.21.15.107:42946) 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1f7fe700 1 -- 172.21.15.107:0/2621394740 learned_addr learned my addr 172.21.15.107:0/2621394740 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter ms_handle_connect 0x7fee20104330 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter ms_handle_connect 0x7fee201007f0 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter ms_handle_connect 0x7fee200fda50 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657089420 0 0) 0x7fee201197a0 con 0x7fee20104330 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee08003800 con 0x7fee20104330 2024-02-10T08:38:01.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644621322 0 0) 0x7fee20117400 con 0x7fee201007f0 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee201197a0 con 0x7fee201007f0 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512645753 0 0) 0x7fee201185a0 con 0x7fee200fda50 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee20117400 con 0x7fee200fda50 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429957206 0 0) 0x7fee08003800 con 0x7fee20104330 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee201185a0 con 0x7fee20104330 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fee18003260 con 0x7fee20104330 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138358134 0 0) 0x7fee201197a0 con 0x7fee201007f0 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee08003800 con 0x7fee201007f0 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fee14001920 con 0x7fee201007f0 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643008252 0 0) 0x7fee20117400 con 0x7fee200fda50 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee201197a0 con 0x7fee200fda50 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fee100018c0 con 0x7fee200fda50 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 902437839 0 0) 0x7fee201185a0 con 0x7fee20104330 2024-02-10T08:38:01.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 >> v1:172.21.15.195:6789/0 conn(0x7fee200fda50 legacy=0x7fee200f9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 >> v1:172.21.15.107:6789/0 conn(0x7fee201007f0 legacy=0x7fee200fa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2011a9a0 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.631+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee201175f0 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.632+0000 7fee26844700 10 client.52889.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.632+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20117a60 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.632+0000 7fee26844700 1 librados: init done 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.632+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fee18001920 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.632+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fee180039c0 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c004f20 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fee18001b00 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(187..187 src has 50..187) v4 ==== 9839+0+0 (unknown 1492845850 0 0) 0x7fee18077170 con 0x7fee20104330 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 10 client.52889.objecter ms_dispatch 0x7fee20001550 osd_map(187..187 src has 50..187) v4 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 3 client.52889.objecter handle_osd_map got epochs [187,187] > 0 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 3 client.52889.objecter handle_osd_map decoding full epoch 187 2024-02-10T08:38:01.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.633+0000 7fee1d7fa700 20 client.52889.objecter dump_active .. 0 homeless 2024-02-10T08:38:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.634+0000 7fee1d7fa700 10 client.52889.objecter ms_handle_connect 0x7fee08063d10 2024-02-10T08:38:01.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.637+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fee18049ef0 con 0x7fee20104330 2024-02-10T08:38:01.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:01.752 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:01.770 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:01.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:01.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:01.820+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fee0c0051d0 con 0x7fee20104330 2024-02-10T08:38:01.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:01.900 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:01.918 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:01.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:02.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:02.050 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.193+0000 7fee1d7fa700 1 -- 172.21.15.107:0/2621394740 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v188) v1 ==== 97+0+0 (unknown 631645187 0 0) 0x7fee18048360 con 0x7fee20104330 2024-02-10T08:38:02.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 10 librados: watch_flush enter 2024-02-10T08:38:02.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 10 librados: watch_flush exit 2024-02-10T08:38:02.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 20 client.52889.objecter shutdown clearing up homeless session... 2024-02-10T08:38:02.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 10 client.52889.objecter successfully canceled tick 2024-02-10T08:38:02.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 >> v1:172.21.15.107:6800/2634842009 conn(0x7fee08063d10 legacy=0x7fee080661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:02.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.196+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 >> v1:172.21.15.107:6790/0 conn(0x7fee20104330 legacy=0x7fee20115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:02.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.200+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 shutdown_connections 2024-02-10T08:38:02.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.200+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 >> 172.21.15.107:0/2621394740 conn(0x7fee200f3640 msgr2=0x7fee201077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:02.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.200+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 shutdown_connections 2024-02-10T08:38:02.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.200+0000 7fee26844700 1 -- 172.21.15.107:0/2621394740 wait complete. 2024-02-10T08:38:02.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:02.200+0000 7fee26844700 1 librados: shutdown 2024-02-10T08:38:02.201 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:38:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:02.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:02.445 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:02.445 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:38:02.445 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:38:02.445 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:02.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.036707+0000 mon.b (mon.1) 259 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.052899+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.227572+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.230958+0000 mgr.y (mgr.14553) 221 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.560626+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.107:0/2050385615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.821884+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.107:0/2621394740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:01.822194+0000 mon.a (mon.0) 1360 : audit [INF] from='client.52889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:01.944386+0000 mon.a (mon.0) 1361 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.037547+0000 mon.b (mon.1) 261 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.193779+0000 mon.a (mon.0) 1362 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.193862+0000 mon.a (mon.0) 1363 : audit [INF] from='client.52889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:02.193932+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.195033+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.195431+0000 mon.a (mon.0) 1365 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.195488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.107:0/2466810138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.195948+0000 mon.a (mon.0) 1366 : audit [INF] from='client.52877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.036707+0000 mon.b (mon.1) 259 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.052899+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.227572+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.230958+0000 mgr.y (mgr.14553) 221 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.560626+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.107:0/2050385615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.821884+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.107:0/2621394740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:01.822194+0000 mon.a (mon.0) 1360 : audit [INF] from='client.52889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:01.944386+0000 mon.a (mon.0) 1361 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.037547+0000 mon.b (mon.1) 261 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.193779+0000 mon.a (mon.0) 1362 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.193862+0000 mon.a (mon.0) 1363 : audit [INF] from='client.52889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:02.193932+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-10T08:38:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.195033+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.195431+0000 mon.a (mon.0) 1365 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.195488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.107:0/2466810138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:02.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.195948+0000 mon.a (mon.0) 1366 : audit [INF] from='client.52877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.036707+0000 mon.b (mon.1) 259 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.052899+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.227572+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.230958+0000 mgr.y (mgr.14553) 221 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.560626+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.107:0/2050385615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.821884+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.107:0/2621394740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:01.822194+0000 mon.a (mon.0) 1360 : audit [INF] from='client.52889 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:01.944386+0000 mon.a (mon.0) 1361 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.037547+0000 mon.b (mon.1) 261 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.193779+0000 mon.a (mon.0) 1362 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.193862+0000 mon.a (mon.0) 1363 : audit [INF] from='client.52889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:02.193932+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.195033+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.195431+0000 mon.a (mon.0) 1365 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.195488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.107:0/2466810138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.195948+0000 mon.a (mon.0) 1366 : audit [INF] from='client.52877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:03.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:03.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:38:03.693 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:03.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:02.475903+0000 mgr.y (mgr.14553) 222 : cluster [DBG] pgmap v258: 290 pgs: 58 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 32 unknown, 186 active+clean; 46 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:02.946434+0000 mon.a (mon.0) 1367 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.951655+0000 mon.a (mon.0) 1368 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.951829+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.951889+0000 mon.a (mon.0) 1370 : audit [INF] from='client.52877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:02.952025+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.959817+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.107:0/3225455625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:02.962161+0000 mon.a (mon.0) 1372 : audit [INF] from='client.52895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:03.038324+0000 mon.b (mon.1) 263 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:02.475903+0000 mgr.y (mgr.14553) 222 : cluster [DBG] pgmap v258: 290 pgs: 58 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 32 unknown, 186 active+clean; 46 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:02.946434+0000 mon.a (mon.0) 1367 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.951655+0000 mon.a (mon.0) 1368 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.951829+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.951889+0000 mon.a (mon.0) 1370 : audit [INF] from='client.52877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:02.952025+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.959817+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.107:0/3225455625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:02.962161+0000 mon.a (mon.0) 1372 : audit [INF] from='client.52895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:03.038324+0000 mon.b (mon.1) 263 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:02.475903+0000 mgr.y (mgr.14553) 222 : cluster [DBG] pgmap v258: 290 pgs: 58 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 32 unknown, 186 active+clean; 46 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:02.946434+0000 mon.a (mon.0) 1367 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.951655+0000 mon.a (mon.0) 1368 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi107-112403-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.951829+0000 mon.a (mon.0) 1369 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-13"}]': finished 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.951889+0000 mon.a (mon.0) 1370 : audit [INF] from='client.52877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi107-112411-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:02.952025+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-10T08:38:03.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.959817+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.107:0/3225455625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:02.962161+0000 mon.a (mon.0) 1372 : audit [INF] from='client.52895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:03.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:03.038324+0000 mon.b (mon.1) 263 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:03.834 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:03.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:03.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:03.994 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:04.126 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:04.153 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.172 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:04.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:04.301 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.320 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:04.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.469 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:04.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:04.599 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.617 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:04.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:03.459158+0000 mon.a (mon.0) 1373 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:04.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:03.951800+0000 mon.a (mon.0) 1374 : audit [INF] from='client.52895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:04.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:03.951991+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-10T08:38:04.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.039251+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:04.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:03.459158+0000 mon.a (mon.0) 1373 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:04.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:03.951800+0000 mon.a (mon.0) 1374 : audit [INF] from='client.52895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:04.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:03.951991+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-10T08:38:04.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.039251+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:04.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:04.745 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:03.459158+0000 mon.a (mon.0) 1373 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:04.745 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:03.951800+0000 mon.a (mon.0) 1374 : audit [INF] from='client.52895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi107-112480-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:04.745 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:03.951991+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-10T08:38:04.745 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.039251+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:04.747 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.765 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:04.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:04.898 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:04.918 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:04.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:04.922 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:05.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:05.049 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:05.068 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:05.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:05.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:05.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:38:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:05.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:05.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:05.656 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:05.675 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:05.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:05.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:04.476272+0000 mgr.y (mgr.14553) 223 : cluster [DBG] pgmap v261: 266 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 40 unknown, 186 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 10 KiB/s wr, 1 op/s 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:04.956625+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.962354+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.962629+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.107:0/4263800241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.962764+0000 mon.a (mon.0) 1377 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.963419+0000 mon.a (mon.0) 1378 : audit [INF] from='client.52901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.972666+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.973471+0000 mon.a (mon.0) 1379 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.993113+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.993436+0000 mon.a (mon.0) 1380 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.994396+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.994644+0000 mon.a (mon.0) 1381 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.994982+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:04.995194+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:05.040191+0000 mon.b (mon.1) 269 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:05.784 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:04.476272+0000 mgr.y (mgr.14553) 223 : cluster [DBG] pgmap v261: 266 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 40 unknown, 186 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 10 KiB/s wr, 1 op/s 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:04.956625+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.962354+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.962629+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.107:0/4263800241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.962764+0000 mon.a (mon.0) 1377 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.963419+0000 mon.a (mon.0) 1378 : audit [INF] from='client.52901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.972666+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.973471+0000 mon.a (mon.0) 1379 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.993113+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.993436+0000 mon.a (mon.0) 1380 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.994396+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.994644+0000 mon.a (mon.0) 1381 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.994982+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:04.995194+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:05.040191+0000 mon.b (mon.1) 269 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:05.786 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:05] "GET /metrics HTTP/1.1" 200 187220 "" "Prometheus/2.33.4" 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:04.476272+0000 mgr.y (mgr.14553) 223 : cluster [DBG] pgmap v261: 266 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 40 unknown, 186 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 10 KiB/s wr, 1 op/s 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:04.956625+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.962354+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.962629+0000 mon.c (mon.2) 241 : audit [INF] from='client.? v1:172.21.15.107:0/4263800241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.962764+0000 mon.a (mon.0) 1377 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.963419+0000 mon.a (mon.0) 1378 : audit [INF] from='client.52901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.972666+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.973471+0000 mon.a (mon.0) 1379 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.993113+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.993436+0000 mon.a (mon.0) 1380 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.994396+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.994644+0000 mon.a (mon.0) 1381 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.994982+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:04.995194+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:05.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:05.040191+0000 mon.b (mon.1) 269 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:05.808 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:06.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:38:06.136 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:06.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:06.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 -- 172.21.15.107:0/2475622844 >> v1:172.21.15.195:6789/0 conn(0x7ff9a80ff710 legacy=0x7ff9a8101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:06.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 -- 172.21.15.107:0/2475622844 shutdown_connections 2024-02-10T08:38:06.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 -- 172.21.15.107:0/2475622844 >> 172.21.15.107:0/2475622844 conn(0x7ff9a80f2510 msgr2=0x7ff9a80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:06.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 -- 172.21.15.107:0/2475622844 shutdown_connections 2024-02-10T08:38:06.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 -- 172.21.15.107:0/2475622844 wait complete. 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 librados: starting msgr at 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.147+0000 7ff9aefe6700 1 librados: starting objecter 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 Processor -- start 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 -- start start 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 librados: setting wanted keys 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 librados: calling monclient init 2024-02-10T08:38:06.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9a81194f0 con 0x7ff9a80fc910 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9a811a6f0 con 0x7ff9a8103250 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9aefe6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9a811b8f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff9a80ff710 0x7ff9a8061190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48364/0 (socket says 172.21.15.107:48364) 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a7fff700 1 -- 172.21.15.107:0/729982628 learned_addr learned my addr 172.21.15.107:0/729982628 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9a80ff710 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9a8103250 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9a80fc910 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:06.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.148+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176408390 0 0) 0x7ff9a81194f0 con 0x7ff9a80fc910 2024-02-10T08:38:06.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9980033f0 con 0x7ff9a80fc910 2024-02-10T08:38:06.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593280067 0 0) 0x7ff9a811b8f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a81194f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397942538 0 0) 0x7ff9a811a6f0 con 0x7ff9a8103250 2024-02-10T08:38:06.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a811b8f0 con 0x7ff9a8103250 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4203982005 0 0) 0x7ff9a81194f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9a811a6f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff994002070 con 0x7ff9a80ff710 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093473213 0 0) 0x7ff9980033f0 con 0x7ff9a80fc910 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9a81194f0 con 0x7ff9a80fc910 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff99c001830 con 0x7ff9a80fc910 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52309144 0 0) 0x7ff9a811b8f0 con 0x7ff9a8103250 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9980033f0 con 0x7ff9a8103250 2024-02-10T08:38:06.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9a0001a20 con 0x7ff9a8103250 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4206631975 0 0) 0x7ff9a811a6f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 >> v1:172.21.15.195:6789/0 conn(0x7ff9a8103250 legacy=0x7ff9a8117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 >> v1:172.21.15.107:6789/0 conn(0x7ff9a80fc910 legacy=0x7ff9a8060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a811caf0 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a811a8e0 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff994002210 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff994003900 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.149+0000 7ff9aefe6700 10 client.52913.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.150+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a811ad50 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.150+0000 7ff9aefe6700 1 librados: init done 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff9940017f0 con 0x7ff9a80ff710 2024-02-10T08:38:06.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(192..192 src has 50..192) v4 ==== 9891+0+0 (unknown 3121978416 0 0) 0x7ff994079760 con 0x7ff9a80ff710 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9a5ffb700 10 client.52913.objecter ms_dispatch 0x7ff9a8001550 osd_map(192..192 src has 50..192) v4 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9a5ffb700 3 client.52913.objecter handle_osd_map got epochs [192,192] > 0 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9a5ffb700 3 client.52913.objecter handle_osd_map decoding full epoch 192 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.151+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a805f7a0 con 0x7ff9a80ff710 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.152+0000 7ff9a5ffb700 20 client.52913.objecter dump_active .. 0 homeless 2024-02-10T08:38:06.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.152+0000 7ff9a5ffb700 10 client.52913.objecter ms_handle_connect 0x7ff998063e70 2024-02-10T08:38:06.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.156+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff99404c510 con 0x7ff9a80ff710 2024-02-10T08:38:06.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.339+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff9a80f8140 con 0x7ff9a80ff710 2024-02-10T08:38:06.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.341+0000 7ff9a5ffb700 1 -- 172.21.15.107:0/729982628 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v192) v1 ==== 81+0+55584 (unknown 4185383213 0 3209505329) 0x7ff99404a980 con 0x7ff9a80ff710 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 192, 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:38:05.950640+0000", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:33:10.747466+0000", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:37:08.966515+0000", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:38:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 37, 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:38:06.344 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 133, 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:38:06.346 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "67", 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:38:06.350 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.351 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.353 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.354 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.355 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:38:06.357 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.358 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.359 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.360 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.361 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.362 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.363 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.364 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.365 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.366 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:38:06.367 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.368 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.370 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.371 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.373 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.374 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.376 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.377 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.378 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.379 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.380 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 25, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "LibRadosList_smithi107-112443-1", 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:34.873249+0000", 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.381 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.382 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.383 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.384 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 60, 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-6", 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:49.742959+0000", 2024-02-10T08:38:06.385 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.386 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "189", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "188", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "188", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "188", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 11, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 187, 2024-02-10T08:38:06.387 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.388 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:37:55.420600+0000", 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:38:06.390 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 3, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.391 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "192", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 11, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 192, 2024-02-10T08:38:06.392 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.393 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20", 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.394 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-15", 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:38:04.220716+0000", 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.395 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.396 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "192", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.397 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.398 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 133, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "OmapPP_smithi107-112411-26", 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:38:04.463625+0000", 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:38:06.400 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.401 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "192", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:38:06.402 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:38:06.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.404 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.405 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:38:06.407 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.408 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.410 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.411 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.412 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:38:06.413 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:38:06.415 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:38:06.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:38:06.418 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.419 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.421 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.422 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.423 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:38:06.424 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.427 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.429 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.430 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 191, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:38:06.431 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.433 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:37:52.685209+0000", 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.435 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:37:00.179417+0000", 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:37:42.432376+0000", 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:37:56.009272+0000", 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.437 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:36:54.340726+0000", 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.438 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:37:29.772075+0000", 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:36:53.023932+0000", 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:38:06.439 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:36:51.667140+0000", 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [], 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.18", 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.440 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 7, 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:38:06.441 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.442 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:38:06.443 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-LibRadosMiscECPP_smithi107-112480-36": { 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:38:06.444 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20": { 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.446 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:38:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.448 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 60, 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:38:06.449 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.450 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.451 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:38:06.453 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.454 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.455 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:38:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:38:06.457 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 10 librados: watch_flush enter 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 10 librados: watch_flush exit 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 20 client.52913.objecter shutdown clearing up homeless session... 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 10 client.52913.objecter successfully canceled tick 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff998063e70 legacy=0x7ff998066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 >> v1:172.21.15.107:6790/0 conn(0x7ff9a80ff710 legacy=0x7ff9a8061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 shutdown_connections 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 >> 172.21.15.107:0/729982628 conn(0x7ff9a80f2510 msgr2=0x7ff9a81066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 shutdown_connections 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 -- 172.21.15.107:0/729982628 wait complete. 2024-02-10T08:38:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:06.343+0000 7ff9aefe6700 1 librados: shutdown 2024-02-10T08:38:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:06.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:06.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:06.754 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:06.773 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:06.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:06.811 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:06.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:06.904 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:06.922 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:06.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:07.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:05.998260+0000 mon.a (mon.0) 1383 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:05.998335+0000 mon.a (mon.0) 1384 : audit [INF] from='client.52901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:05.998433+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:05.998541+0000 mon.a (mon.0) 1386 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:05.998599+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.002615+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.007409+0000 mon.a (mon.0) 1388 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.009795+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.010429+0000 mon.a (mon.0) 1389 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.041052+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:06.340812+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/729982628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:38:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:06.477438+0000 mgr.y (mgr.14553) 224 : cluster [DBG] pgmap v264: 290 pgs: 45 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 205 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:05.998260+0000 mon.a (mon.0) 1383 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:05.998335+0000 mon.a (mon.0) 1384 : audit [INF] from='client.52901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:05.998433+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:05.998541+0000 mon.a (mon.0) 1386 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:05.998599+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.002615+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.007409+0000 mon.a (mon.0) 1388 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.009795+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.010429+0000 mon.a (mon.0) 1389 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.041052+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:06.340812+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/729982628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:06.477438+0000 mgr.y (mgr.14553) 224 : cluster [DBG] pgmap v264: 290 pgs: 45 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 205 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:05.998260+0000 mon.a (mon.0) 1383 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:05.998335+0000 mon.a (mon.0) 1384 : audit [INF] from='client.52901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi107-112411-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:05.998433+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:05.998541+0000 mon.a (mon.0) 1386 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:05.998599+0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.002615+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.007409+0000 mon.a (mon.0) 1388 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.009795+0000 mon.c (mon.2) 242 : audit [INF] from='client.? v1:172.21.15.107:0/1154190771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.010429+0000 mon.a (mon.0) 1389 : audit [INF] from='client.52871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.041052+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:06.340812+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.107:0/729982628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:38:07.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:06.477438+0000 mgr.y (mgr.14553) 224 : cluster [DBG] pgmap v264: 290 pgs: 45 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 205 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:07.302 INFO:tasks.thrashosds.thrasher:j is {'epoch': 192, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:38:05.950640+0000', 'last_up_change': '2024-02-10T08:33:10.747466+0000', 'last_in_change': '2024-02-10T08:37:08.966515+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 37, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 133, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 25, 'pool_name': 'LibRadosList_smithi107-112443-1', 'create_time': '2024-02-10T08:36:34.873249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 60, 'pool_name': 'test-rados-api-smithi107-112490-6', 'create_time': '2024-02-10T08:36:49.742959+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '188', 'last_force_op_resend_prenautilus': '188', 'last_force_op_resend_preluminous': '188', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 11, 'snap_epoch': 187, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 124, 'pool_name': 'LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20', 'create_time': '2024-02-10T08:37:55.420600+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 3, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 11, 'snap_epoch': 192, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 132, 'pool_name': 'test-rados-api-smithi107-112490-15', 'create_time': '2024-02-10T08:38:04.220716+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 133, 'pool_name': 'OmapPP_smithi107-112411-26', 'create_time': '2024-02-10T08:38:04.463625+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:37:52.685209+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:37:00.179417+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:37:42.432376+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:37:56.009272+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:36:54.340726+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:37:29.772075+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:36:53.023932+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:36:51.667140+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosMiscECPP_smithi107-112480-36': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 60, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:38:07.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:38:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:07.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:07.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:07.437 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:07.455 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:07.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:07.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:07.586 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:07.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:07.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:07.734 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:07.752 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:07.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:07.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:07.883 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.042027+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.245148+0000 mon.a (mon.0) 1390 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:07.245271+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-10T08:38:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.247677+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.248033+0000 mon.a (mon.0) 1392 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.261686+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.262198+0000 mon.a (mon.0) 1393 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.262886+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.263150+0000 mon.a (mon.0) 1394 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.263525+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:07.517439+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.042027+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.245148+0000 mon.a (mon.0) 1390 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:07.245271+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.247677+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.248033+0000 mon.a (mon.0) 1392 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.261686+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.262198+0000 mon.a (mon.0) 1393 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.262886+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.263150+0000 mon.a (mon.0) 1394 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.263525+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:07.517439+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.042027+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.245148+0000 mon.a (mon.0) 1390 : audit [INF] from='client.52871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi107-112403-27"}]': finished 2024-02-10T08:38:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:07.245271+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-10T08:38:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.247677+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.248033+0000 mon.a (mon.0) 1392 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.261686+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.262198+0000 mon.a (mon.0) 1393 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.262886+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.263150+0000 mon.a (mon.0) 1394 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.263525+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:07.517439+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:08.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:08.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:08.664 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:08.683 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:08.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:08.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:08.817 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:09.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:09.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:09.176 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.008722+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.008849+0000 mon.a (mon.0) 1397 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.008959+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:08.009002+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.014129+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.015058+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.017540+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.107:0/1937376491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.017793+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.018568+0000 mon.a (mon.0) 1401 : audit [INF] from='client.47650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.018811+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:08.043023+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:08.478088+0000 mgr.y (mgr.14553) 225 : cluster [DBG] pgmap v267: 298 pgs: 40 unknown, 26 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.008722+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.008849+0000 mon.a (mon.0) 1397 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.008959+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:08.009002+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.014129+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.015058+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.017540+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.107:0/1937376491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.017793+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.018568+0000 mon.a (mon.0) 1401 : audit [INF] from='client.47650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.018811+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:08.043023+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:09.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:08.478088+0000 mgr.y (mgr.14553) 225 : cluster [DBG] pgmap v267: 298 pgs: 40 unknown, 26 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-02-10T08:38:09.340 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:38:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:38:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:38:09.341 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.008722+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi107-112480-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.008849+0000 mon.a (mon.0) 1397 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.008959+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi107-112403-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:08.009002+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.014129+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.015058+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.017540+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.107:0/1937376491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.017793+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.018568+0000 mon.a (mon.0) 1401 : audit [INF] from='client.47650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.018811+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:08.043023+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:08.478088+0000 mgr.y (mgr.14553) 225 : cluster [DBG] pgmap v267: 298 pgs: 40 unknown, 26 creating+activating, 26 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 52 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-02-10T08:38:09.924 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:09.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:10.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:10.068 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:10.088 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:10.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:10.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:10.218 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:10.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:38:10.416 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:10.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.015574+0000 mon.a (mon.0) 1403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.015707+0000 mon.a (mon.0) 1404 : audit [INF] from='client.47650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.022846+0000 mon.c (mon.2) 244 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.024992+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.043802+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:09.265074+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.266090+0000 mon.a (mon.0) 1406 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:09.266321+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:10.009813+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.014134+0000 mon.a (mon.0) 1409 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.014229+0000 mon.a (mon.0) 1410 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.014285+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]': finished 2024-02-10T08:38:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:10.014368+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-10T08:38:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.016677+0000 mon.c (mon.2) 245 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.023057+0000 mon.a (mon.0) 1413 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.027626+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.029040+0000 mon.a (mon.0) 1414 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.044657+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.015574+0000 mon.a (mon.0) 1403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.015707+0000 mon.a (mon.0) 1404 : audit [INF] from='client.47650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.022846+0000 mon.c (mon.2) 244 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.024992+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.043802+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:09.265074+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.266090+0000 mon.a (mon.0) 1406 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:09.266321+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:10.009813+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.014134+0000 mon.a (mon.0) 1409 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.014229+0000 mon.a (mon.0) 1410 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.014285+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]': finished 2024-02-10T08:38:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:10.014368+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.016677+0000 mon.c (mon.2) 245 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.023057+0000 mon.a (mon.0) 1413 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.027626+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.029040+0000 mon.a (mon.0) 1414 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.044657+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.015574+0000 mon.a (mon.0) 1403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.015707+0000 mon.a (mon.0) 1404 : audit [INF] from='client.47650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi107-112411-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.022846+0000 mon.c (mon.2) 244 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.024992+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.043802+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:09.265074+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.266090+0000 mon.a (mon.0) 1406 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:09.266321+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:10.009813+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.014134+0000 mon.a (mon.0) 1409 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi107-112403-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.014229+0000 mon.a (mon.0) 1410 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.014285+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-15", "mode": "writeback"}]': finished 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:10.014368+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.016677+0000 mon.c (mon.2) 245 : audit [INF] from='client.? v1:172.21.15.107:0/2993270293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.023057+0000 mon.a (mon.0) 1413 : audit [INF] from='client.52844 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.027626+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.029040+0000 mon.a (mon.0) 1414 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:10.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.044657+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:10.547 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:10.789 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:10.803 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:10.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:10.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplepp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4032 ms) 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3971 ms) 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4059 ms) 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12062 ms total) 2024-02-10T08:38:10.959 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: 2024-02-10T08:38:10.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-10T08:38:10.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (100789 ms total) 2024-02-10T08:38:10.960 INFO:tasks.workunit.client.0.smithi107.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-10T08:38:10.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:38:10.960 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112462 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112462 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112454 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112454 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112438 2024-02-10T08:38:10.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112438 2024-02-10T08:38:11.192 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:11.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:11.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:11.326 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:11.346 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:11.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:11.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:11.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:10.478511+0000 mgr.y (mgr.14553) 226 : cluster [DBG] pgmap v270: 258 pgs: 8 unknown, 26 creating+activating, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 12 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:11.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.936228+0000 mon.a (mon.0) 1415 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.936316+0000 mon.a (mon.0) 1416 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:10.936427+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.940920+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.942296+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.107:0/4202333394' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.946042+0000 mon.a (mon.0) 1418 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.953378+0000 mon.a (mon.0) 1419 : audit [INF] from='client.52931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.967079+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:10.967464+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.045616+0000 mon.b (mon.1) 287 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:11.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:10.478511+0000 mgr.y (mgr.14553) 226 : cluster [DBG] pgmap v270: 258 pgs: 8 unknown, 26 creating+activating, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 12 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.936228+0000 mon.a (mon.0) 1415 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.936316+0000 mon.a (mon.0) 1416 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:10.936427+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.940920+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.942296+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.107:0/4202333394' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.946042+0000 mon.a (mon.0) 1418 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.953378+0000 mon.a (mon.0) 1419 : audit [INF] from='client.52931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.967079+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:10.967464+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.045616+0000 mon.b (mon.1) 287 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:11.480 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:10.478511+0000 mgr.y (mgr.14553) 226 : cluster [DBG] pgmap v270: 258 pgs: 8 unknown, 26 creating+activating, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 192 active+clean; 12 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.936228+0000 mon.a (mon.0) 1415 : audit [INF] from='client.52844 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi107-112525-20"}]': finished 2024-02-10T08:38:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.936316+0000 mon.a (mon.0) 1416 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:10.936427+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.940920+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.107:0/4023540388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.942296+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.107:0/4202333394' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.946042+0000 mon.a (mon.0) 1418 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.953378+0000 mon.a (mon.0) 1419 : audit [INF] from='client.52931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.967079+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:10.967464+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.045616+0000 mon.b (mon.1) 287 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:11.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:11.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:11.710 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:11.807 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:11.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:11.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 -- 172.21.15.107:0/3040732652 >> v1:172.21.15.107:6789/0 conn(0x7f88400ff710 legacy=0x7f8840101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:11.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 -- 172.21.15.107:0/3040732652 shutdown_connections 2024-02-10T08:38:11.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 -- 172.21.15.107:0/3040732652 >> 172.21.15.107:0/3040732652 conn(0x7f88400f2510 msgr2=0x7f88400f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:11.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 -- 172.21.15.107:0/3040732652 shutdown_connections 2024-02-10T08:38:11.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 -- 172.21.15.107:0/3040732652 wait complete. 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 librados: starting msgr at 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.817+0000 7f8845168700 1 librados: starting objecter 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 Processor -- start 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 -- start start 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 librados: setting wanted keys 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 librados: calling monclient init 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8840117200 con 0x7f88400fc910 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f88401183e0 con 0x7f88400ff710 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8845168700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f88401195e0 con 0x7f8840103250 2024-02-10T08:38:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f883f59e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8840103250 0x7f88401158e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46762/0 (socket says 172.21.15.107:46762) 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f883ed9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f88400fc910 0x7f88400f97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51446/0 (socket says 172.21.15.107:51446) 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f883f59e700 1 -- 172.21.15.107:0/4057322491 learned_addr learned my addr 172.21.15.107:0/4057322491 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter ms_handle_connect 0x7f88400fc910 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter ms_handle_connect 0x7f8840103250 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter ms_handle_connect 0x7f88400ff710 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543316940 0 0) 0x7f88401195e0 con 0x7f8840103250 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88280038e0 con 0x7f8840103250 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414525581 0 0) 0x7f8840117200 con 0x7f88400fc910 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88401195e0 con 0x7f88400fc910 2024-02-10T08:38:11.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096801655 0 0) 0x7f88401183e0 con 0x7f88400ff710 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.818+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8840117200 con 0x7f88400ff710 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3701731146 0 0) 0x7f88280038e0 con 0x7f8840103250 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88401183e0 con 0x7f8840103250 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8838001c00 con 0x7f8840103250 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319316732 0 0) 0x7f88401195e0 con 0x7f88400fc910 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88280038e0 con 0x7f88400fc910 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8834001a80 con 0x7f88400fc910 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 115214240 0 0) 0x7f8840117200 con 0x7f88400ff710 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88401195e0 con 0x7f88400ff710 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8830001ed0 con 0x7f88400ff710 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754383271 0 0) 0x7f88401183e0 con 0x7f8840103250 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 >> v1:172.21.15.195:6789/0 conn(0x7f88400ff710 legacy=0x7f88400f9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 >> v1:172.21.15.107:6789/0 conn(0x7f88400fc910 legacy=0x7f88400f97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884011a7e0 con 0x7f8840103250 2024-02-10T08:38:11.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88401173f0 con 0x7f8840103250 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8845168700 10 client.52937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88401178e0 con 0x7f8840103250 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8845168700 1 librados: init done 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f88380026d0 con 0x7f8840103250 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.819+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8838005da0 con 0x7f8840103250 2024-02-10T08:38:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.820+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c004f20 con 0x7f8840103250 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.820+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8838002230 con 0x7f8840103250 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(197..197 src has 50..197) v4 ==== 9623+0+0 (unknown 1489413434 0 0) 0x7f8838079660 con 0x7f8840103250 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 10 client.52937.objecter ms_dispatch 0x7f8840001550 osd_map(197..197 src has 50..197) v4 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 3 client.52937.objecter handle_osd_map got epochs [197,197] > 0 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 3 client.52937.objecter handle_osd_map decoding full epoch 197 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 20 client.52937.objecter dump_active .. 0 homeless 2024-02-10T08:38:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.821+0000 7f8827fff700 10 client.52937.objecter ms_handle_connect 0x7f8828063d10 2024-02-10T08:38:11.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:11.825+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f883804c460 con 0x7f8840103250 2024-02-10T08:38:11.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:11.948 INFO:tasks.workunit.client.0.smithi107.stdout:P (68032 ms total) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (113 ms) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (113 ms total) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (33 ms) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (38 ms total) 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.949 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (29 ms) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (32 ms total) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (43 ms) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (11 ms) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (54 ms total) 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.950 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1027 ms) 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1027 ms total) 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (101777 ms total) 2024-02-10T08:38:11.951 INFO:tasks.workunit.client.0.smithi107.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-10T08:38:11.951 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:11.979 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:11.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:12.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.021+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f882c0051d0 con 0x7f8840103250 2024-02-10T08:38:12.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:12.131 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:12.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:12.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.241786+0000 mgr.y (mgr.14553) 227 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:11.291884+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:11.499855+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:11.500242+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.936976+0000 mon.a (mon.0) 1422 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.937111+0000 mon.a (mon.0) 1423 : audit [INF] from='client.52931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:12.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.937178+0000 mon.a (mon.0) 1424 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:11.937212+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.940483+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.945911+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.950294+0000 mon.a (mon.0) 1426 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:11.952800+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.023176+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.107:0/4057322491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.023486+0000 mon.a (mon.0) 1428 : audit [INF] from='client.52937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.046418+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.241786+0000 mgr.y (mgr.14553) 227 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:12.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:11.291884+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:11.499855+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:11.500242+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.936976+0000 mon.a (mon.0) 1422 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.937111+0000 mon.a (mon.0) 1423 : audit [INF] from='client.52931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.937178+0000 mon.a (mon.0) 1424 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:11.937212+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.940483+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.945911+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.950294+0000 mon.a (mon.0) 1426 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:11.952800+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.023176+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.107:0/4057322491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.023486+0000 mon.a (mon.0) 1428 : audit [INF] from='client.52937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.046418+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.241786+0000 mgr.y (mgr.14553) 227 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:11.291884+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:38:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:11.499855+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:11.500242+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.936976+0000 mon.a (mon.0) 1422 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi107-112480-36"}]': finished 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.937111+0000 mon.a (mon.0) 1423 : audit [INF] from='client.52931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi107-112411-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.937178+0000 mon.a (mon.0) 1424 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:11.937212+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.940483+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.945911+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.950294+0000 mon.a (mon.0) 1426 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:11.952800+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.023176+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.107:0/4057322491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.023486+0000 mon.a (mon.0) 1428 : audit [INF] from='client.52937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:12.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.046418+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.943+0000 7f8827fff700 1 -- 172.21.15.107:0/4057322491 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v199) v1 ==== 91+0+0 (unknown 1107273682 0 0) 0x7f883804a870 con 0x7f8840103250 2024-02-10T08:38:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 10 librados: watch_flush enter 2024-02-10T08:38:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 10 librados: watch_flush exit 2024-02-10T08:38:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 20 client.52937.objecter shutdown clearing up homeless session... 2024-02-10T08:38:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 10 client.52937.objecter successfully canceled tick 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8828063d10 legacy=0x7f88280661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.946+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 >> v1:172.21.15.107:6790/0 conn(0x7f8840103250 legacy=0x7f88401158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.947+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 shutdown_connections 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.947+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 >> 172.21.15.107:0/4057322491 conn(0x7f88400f2510 msgr2=0x7f88401066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.947+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 shutdown_connections 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.947+0000 7f8845168700 1 -- 172.21.15.107:0/4057322491 wait complete. 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:12.947+0000 7f8845168700 1 librados: shutdown 2024-02-10T08:38:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:38:13.259 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:12.478936+0000 mgr.y (mgr.14553) 228 : cluster [DBG] pgmap v273: 282 pgs: 32 creating+peering, 50 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:12.932833+0000 mon.a (mon.0) 1429 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:12.932855+0000 mon.a (mon.0) 1430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.939325+0000 mon.a (mon.0) 1431 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.939396+0000 mon.a (mon.0) 1432 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.939472+0000 mon.a (mon.0) 1433 : audit [INF] from='client.52937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:12.939516+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.948010+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:12.948499+0000 mon.a (mon.0) 1435 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:13.047255+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:12.478936+0000 mgr.y (mgr.14553) 228 : cluster [DBG] pgmap v273: 282 pgs: 32 creating+peering, 50 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:12.932833+0000 mon.a (mon.0) 1429 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:12.932855+0000 mon.a (mon.0) 1430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.939325+0000 mon.a (mon.0) 1431 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.939396+0000 mon.a (mon.0) 1432 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.939472+0000 mon.a (mon.0) 1433 : audit [INF] from='client.52937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:12.939516+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.948010+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:12.948499+0000 mon.a (mon.0) 1435 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:13.047255+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:12.478936+0000 mgr.y (mgr.14553) 228 : cluster [DBG] pgmap v273: 282 pgs: 32 creating+peering, 50 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:12.932833+0000 mon.a (mon.0) 1429 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:12.932855+0000 mon.a (mon.0) 1430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.939325+0000 mon.a (mon.0) 1431 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.939396+0000 mon.a (mon.0) 1432 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-15"}]': finished 2024-02-10T08:38:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.939472+0000 mon.a (mon.0) 1433 : audit [INF] from='client.52937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:12.939516+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-10T08:38:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.948010+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.107:0/993932140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:12.948499+0000 mon.a (mon.0) 1435 : audit [INF] from='client.47647 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]: dispatch 2024-02-10T08:38:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:13.047255+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:13.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.910+0000 7fc64442c700 1 -- 172.21.15.107:0/1145865538 >> v1:172.21.15.107:6790/0 conn(0x7fc63c100840 legacy=0x7fc63c102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.910+0000 7fc64442c700 1 -- 172.21.15.107:0/1145865538 shutdown_connections 2024-02-10T08:38:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.910+0000 7fc64442c700 1 -- 172.21.15.107:0/1145865538 >> 172.21.15.107:0/1145865538 conn(0x7fc63c06c090 msgr2=0x7fc63c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.910+0000 7fc64442c700 1 -- 172.21.15.107:0/1145865538 shutdown_connections 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 -- 172.21.15.107:0/1145865538 wait complete. 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 librados: starting msgr at 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 librados: starting objecter 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 Processor -- start 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 -- start start 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 librados: setting wanted keys 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 librados: calling monclient init 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c0fd2a0 con 0x7fc63c104350 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c0fd430 con 0x7fc63c100840 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc64442c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c11bc10 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc6421c8700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc63c0fd9f0 0x7fc63c0fcb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46806/0 (socket says 172.21.15.107:46806) 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.911+0000 7fc6421c8700 1 -- 172.21.15.107:0/2598038392 learned_addr learned my addr 172.21.15.107:0/2598038392 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter ms_handle_connect 0x7fc63c0fd9f0 2024-02-10T08:38:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter ms_handle_connect 0x7fc63c104350 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942152785 0 0) 0x7fc63c11bc10 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc624003630 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466853064 0 0) 0x7fc63c0fd2a0 con 0x7fc63c104350 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c11bc10 con 0x7fc63c104350 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter ms_handle_connect 0x7fc63c100840 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192821866 0 0) 0x7fc63c0fd430 con 0x7fc63c100840 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c0fd2a0 con 0x7fc63c100840 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749882656 0 0) 0x7fc63c11bc10 con 0x7fc63c104350 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc63c0fd430 con 0x7fc63c104350 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3693112098 0 0) 0x7fc624003630 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc63c11bc10 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc62c001be0 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc6340018d0 con 0x7fc63c0fd9f0 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 16227006 0 0) 0x7fc63c0fd430 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 >> v1:172.21.15.107:6790/0 conn(0x7fc63c0fd9f0 legacy=0x7fc63c0fcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 >> v1:172.21.15.195:6789/0 conn(0x7fc63c100840 legacy=0x7fc63c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c11cdb0 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c11be00 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc64442c700 10 client.27438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c11c270 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc64442c700 1 librados: init done 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc62c002220 con 0x7fc63c104350 2024-02-10T08:38:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.912+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc62c003540 con 0x7fc63c104350 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.913+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc628004f20 con 0x7fc63c104350 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc62c002960 con 0x7fc63c104350 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(199..199 src has 50..199) v4 ==== 8382+0+0 (unknown 81201704 0 0) 0x7fc62c0767e0 con 0x7fc63c104350 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 10 client.27438.objecter ms_dispatch 0x7fc63c001550 osd_map(199..199 src has 50..199) v4 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 3 client.27438.objecter handle_osd_map got epochs [199,199] > 0 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 3 client.27438.objecter handle_osd_map decoding full epoch 199 2024-02-10T08:38:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 20 client.27438.objecter dump_active .. 0 homeless 2024-02-10T08:38:13.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.914+0000 7fc63b7fe700 10 client.27438.objecter ms_handle_connect 0x7fc6240639e0 2024-02-10T08:38:13.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:13.918+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc62c049a90 con 0x7fc63c104350 2024-02-10T08:38:13.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:13.966 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:13.987 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:13.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:14.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.106+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc628000b90 con 0x7fc6240639e0 2024-02-10T08:38:14.119 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:14.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.123+0000 7fc63b7fe700 1 -- 172.21.15.107:0/2598038392 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+625126 (unknown 2965378022 0 583603967) 0x7fc628000b90 con 0x7fc6240639e0 2024-02-10T08:38:14.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:38:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.127+0000 7fc64442c700 10 librados: watch_flush enter 2024-02-10T08:38:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.127+0000 7fc64442c700 10 librados: watch_flush exit 2024-02-10T08:38:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.127+0000 7fc64442c700 20 client.27438.objecter shutdown clearing up homeless session... 2024-02-10T08:38:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.127+0000 7fc64442c700 10 client.27438.objecter successfully canceled tick 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc6240639e0 legacy=0x7fc624065ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 >> v1:172.21.15.107:6789/0 conn(0x7fc63c104350 legacy=0x7fc63c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 shutdown_connections 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 >> 172.21.15.107:0/2598038392 conn(0x7fc63c06c090 msgr2=0x7fc63c1067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 shutdown_connections 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 -- 172.21.15.107:0/2598038392 wait complete. 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:14.128+0000 7fc64442c700 1 librados: shutdown 2024-02-10T08:38:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:38:14.139 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:14.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:14.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:14.269 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:14.287 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:14.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:14.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.462743+0000 osd.3 (osd.3) 107 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.463641+0000 osd.3 (osd.3) 108 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.598408+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.601380+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.654512+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.655500+0000 osd.7 (osd.7) 92 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.915214+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.915596+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:14.945 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:13.938869+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 -- 172.21.15.107:0/93950340 >> v1:172.21.15.107:6789/0 conn(0x7f01381007f0 legacy=0x7f0138102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 -- 172.21.15.107:0/93950340 shutdown_connections 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 -- 172.21.15.107:0/93950340 >> 172.21.15.107:0/93950340 conn(0x7f01380f3640 msgr2=0x7f01380f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 -- 172.21.15.107:0/93950340 shutdown_connections 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 -- 172.21.15.107:0/93950340 wait complete. 2024-02-10T08:38:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 librados: starting msgr at 2024-02-10T08:38:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.159+0000 7f013e0b4700 1 librados: starting objecter 2024-02-10T08:38:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 Processor -- start 2024-02-10T08:38:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 -- start start 2024-02-10T08:38:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 librados: setting wanted keys 2024-02-10T08:38:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 librados: calling monclient init 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0138117520 con 0x7f0138104330 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0138118700 con 0x7f01380fda50 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f013e0b4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0138119900 con 0x7f01381007f0 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0136ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f01381007f0 0x7f01380fa480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46832/0 (socket says 172.21.15.107:46832) 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0136ffd700 1 -- 172.21.15.107:0/2512040328 learned_addr learned my addr 172.21.15.107:0/2512040328 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter ms_handle_connect 0x7f0138104330 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter ms_handle_connect 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823744360 0 0) 0x7f0138119900 con 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0120003030 con 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter ms_handle_connect 0x7f01380fda50 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789044348 0 0) 0x7f0138117520 con 0x7f0138104330 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0138119900 con 0x7f0138104330 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.160+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248754339 0 0) 0x7f0138118700 con 0x7f01380fda50 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0138117520 con 0x7f01380fda50 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1511095234 0 0) 0x7f0120003030 con 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0138118700 con 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f012c002df0 con 0x7f01381007f0 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156916006 0 0) 0x7f0138119900 con 0x7f0138104330 2024-02-10T08:38:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0120003030 con 0x7f0138104330 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0130001cb0 con 0x7f0138104330 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1533758925 0 0) 0x7f0138117520 con 0x7f01380fda50 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0138119900 con 0x7f01380fda50 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2138659313 0 0) 0x7f0138118700 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.195:6789/0 conn(0x7f01380fda50 legacy=0x7f01380f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6789/0 conn(0x7f0138104330 legacy=0x7f0138115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013811ab00 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0138119af0 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f013e0b4700 10 client.52961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0138119f60 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f013e0b4700 1 librados: init done 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f012c001aa0 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.161+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f012c0035c0 con 0x7f01381007f0 2024-02-10T08:38:15.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.162+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0124004f00 con 0x7f01381007f0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.162+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f012c003800 con 0x7f01381007f0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(201..201 src has 50..201) v4 ==== 8940+0+0 (unknown 3672846268 0 0) 0x7f012c079290 con 0x7f01381007f0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 osd_map(201..201 src has 50..201) v4 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 3 client.52961.objecter handle_osd_map got epochs [201,201] > 0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 3 client.52961.objecter handle_osd_map decoding full epoch 201 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 20 client.52961.objecter dump_active .. 0 homeless 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v201) v1 ==== 54+0+16 (unknown 4031702312 0 2420000192) 0x7f012c07b6a0 con 0x7f01381007f0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0120063ca0 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 10 client.52961.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 1 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 1 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 0 1 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 4 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 10 client.52961.objecter _send_command 1 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0124004950 con 0x7f0124004530 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.163+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0124004530 2024-02-10T08:38:15.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.164+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0124004950 con 0x7f0124004530 2024-02-10T08:38:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.164+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.164+0000 7f0134ff9700 10 client.52961.objecter _finish_command 1 = system:0 2024-02-10T08:38:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.164+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 4 2024-02-10T08:38:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.164+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 0 1 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.462743+0000 osd.3 (osd.3) 107 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.463641+0000 osd.3 (osd.3) 108 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.598408+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.601380+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.654512+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:38:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.655500+0000 osd.7 (osd.7) 92 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.915214+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.915596+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:13.938869+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:13.938971+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:13.955797+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:13.972464+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.048173+0000 mon.b (mon.1) 293 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.107982+0000 mgr.y (mgr.14553) 229 : audit [DBG] from='client.27438 v1:172.21.15.107:0/2598038392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.212335+0000 mon.a (mon.0) 1439 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.212869+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.213111+0000 mon.a (mon.0) 1440 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.213472+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.213708+0000 mon.a (mon.0) 1441 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:14.479416+0000 mgr.y (mgr.14553) 230 : cluster [DBG] pgmap v276: 250 pgs: 32 unknown, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.462743+0000 osd.3 (osd.3) 107 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.463641+0000 osd.3 (osd.3) 108 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.598408+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.601380+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.654512+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:38:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.655500+0000 osd.7 (osd.7) 92 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.915214+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.915596+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:13.938869+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47647 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi107-112403-28"}]': finished 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:13.938971+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:13.955797+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:13.972464+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.048173+0000 mon.b (mon.1) 293 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.107982+0000 mgr.y (mgr.14553) 229 : audit [DBG] from='client.27438 v1:172.21.15.107:0/2598038392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.212335+0000 mon.a (mon.0) 1439 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.212869+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.213111+0000 mon.a (mon.0) 1440 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.213472+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.213708+0000 mon.a (mon.0) 1441 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:14.479416+0000 mgr.y (mgr.14553) 230 : cluster [DBG] pgmap v276: 250 pgs: 32 unknown, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 10 client.52961.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 2 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 2 2024-02-10T08:38:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f01240051a0 osd=0 3 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 0 2 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 4 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 10 client.52961.objecter _send_command 2 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.180+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0124004d30 con 0x7f0124004530 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0124004d30 con 0x7f0124004530 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f0134ff9700 10 client.52961.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 4 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 0 2 2024-02-10T08:38:15.185 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 10 client.52961.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 3 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 3 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 1 3 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 4 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 10 client.52961.objecter _send_command 3 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.181+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f012400c000 con 0x7f0124004150 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0124004150 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f012400c000 con 0x7f0124004150 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 10 client.52961.objecter _finish_command 3 = system:0 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 4 2024-02-10T08:38:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.182+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 1 3 2024-02-10T08:38:15.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 10 client.52961.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 4 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 4 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124003fe0 osd=1 3 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 1 4 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 4 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 10 client.52961.objecter _send_command 4 2024-02-10T08:38:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.198+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f01240018b0 con 0x7f0124004150 2024-02-10T08:38:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f01240018b0 con 0x7f0124004150 2024-02-10T08:38:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 10 client.52961.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 4 2024-02-10T08:38:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 1 4 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 10 client.52961.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 5 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 5 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 2 5 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 4 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 10 client.52961.objecter _send_command 5 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f012400de90 con 0x7f012400d2f0 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.200+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f012400d2f0 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.201+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f012400de90 con 0x7f012400d2f0 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.201+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.201+0000 7f0134ff9700 10 client.52961.objecter _finish_command 5 = system:0 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.201+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 4 2024-02-10T08:38:15.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.201+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 2 5 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 10 client.52961.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 6 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 6 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124003630 osd=2 3 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 2 6 2024-02-10T08:38:15.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 4 2024-02-10T08:38:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 10 client.52961.objecter _send_command 6 2024-02-10T08:38:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.216+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f01240030a0 con 0x7f012400d2f0 2024-02-10T08:38:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f01240030a0 con 0x7f012400d2f0 2024-02-10T08:38:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f0134ff9700 10 client.52961.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 4 2024-02-10T08:38:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 2 6 2024-02-10T08:38:15.219 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 10 client.52961.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 7 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 7 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 3 7 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 4 2024-02-10T08:38:15.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 10 client.52961.objecter _send_command 7 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.217+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0124015b00 con 0x7f0124011ff0 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0124011ff0 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0124015b00 con 0x7f0124011ff0 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 10 client.52961.objecter _finish_command 7 = system:0 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 4 2024-02-10T08:38:15.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.218+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 3 7 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 10 client.52961.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 8 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 8 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124011970 osd=3 3 2024-02-10T08:38:15.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 3 8 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 4 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 10 client.52961.objecter _send_command 8 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.233+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f012400e270 con 0x7f0124011ff0 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f012400e270 con 0x7f0124011ff0 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f0134ff9700 10 client.52961.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 4 2024-02-10T08:38:15.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 3 8 2024-02-10T08:38:15.236 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:15.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 10 client.52961.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 9 2024-02-10T08:38:15.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 9 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 4 9 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 4 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 10 client.52961.objecter _send_command 9 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.234+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f012400e1e0 con 0x7f012400ce50 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.235+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f012400ce50 2024-02-10T08:38:15.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.235+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f012400e1e0 con 0x7f012400ce50 2024-02-10T08:38:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.235+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.236+0000 7f0134ff9700 10 client.52961.objecter _finish_command 9 = system:0 2024-02-10T08:38:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.236+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 4 2024-02-10T08:38:15.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.236+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 4 9 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 10 client.52961.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 10 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 10 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f012400e030 osd=4 3 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 4 10 2024-02-10T08:38:15.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 4 2024-02-10T08:38:15.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 10 client.52961.objecter _send_command 10 2024-02-10T08:38:15.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.251+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f012400e1e0 con 0x7f012400ce50 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f012400e1e0 con 0x7f012400ce50 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 10 client.52961.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 4 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 4 10 2024-02-10T08:38:15.256 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 10 client.52961.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 11 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 11 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 5 11 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 4 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 10 client.52961.objecter _send_command 11 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0124018050 con 0x7f0124017a60 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.253+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0124017a60 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.254+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0124018050 con 0x7f0124017a60 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.254+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.254+0000 7f0134ff9700 10 client.52961.objecter _finish_command 11 = system:0 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.254+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 4 2024-02-10T08:38:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.254+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 5 11 2024-02-10T08:38:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 10 client.52961.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 12 2024-02-10T08:38:15.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 12 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f012400e1e0 osd=5 3 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 5 12 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 4 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 10 client.52961.objecter _send_command 12 2024-02-10T08:38:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.269+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0124018050 con 0x7f0124017a60 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.271+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0124018050 con 0x7f0124017a60 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.271+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.271+0000 7f0134ff9700 10 client.52961.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.271+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 4 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.271+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 5 12 2024-02-10T08:38:15.273 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 10 client.52961.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 13 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 13 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 6 13 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 4 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 10 client.52961.objecter _send_command 13 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0124022c30 con 0x7f012401f080 2024-02-10T08:38:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.272+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f012401f080 2024-02-10T08:38:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.273+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0124022c30 con 0x7f012401f080 2024-02-10T08:38:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.273+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.273+0000 7f0134ff9700 10 client.52961.objecter _finish_command 13 = system:0 2024-02-10T08:38:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.273+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 4 2024-02-10T08:38:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.273+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 6 13 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:13.938971+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:13.955797+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:13.972464+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.048173+0000 mon.b (mon.1) 293 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.107982+0000 mgr.y (mgr.14553) 229 : audit [DBG] from='client.27438 v1:172.21.15.107:0/2598038392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.212335+0000 mon.a (mon.0) 1439 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.212869+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.213111+0000 mon.a (mon.0) 1440 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.213472+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.213708+0000 mon.a (mon.0) 1441 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:14.479416+0000 mgr.y (mgr.14553) 230 : cluster [DBG] pgmap v276: 250 pgs: 32 unknown, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 186 active+clean; 11 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 10 client.52961.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 14 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 14 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f0124018050 osd=6 3 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 6 14 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 4 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 10 client.52961.objecter _send_command 14 2024-02-10T08:38:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.287+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f01240243e0 con 0x7f012401f080 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.289+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f01240243e0 con 0x7f012401f080 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.289+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.289+0000 7f0134ff9700 10 client.52961.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.289+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 4 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.289+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 6 14 2024-02-10T08:38:15.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 10 client.52961.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 15 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 15 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 7 15 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 4 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 10 client.52961.objecter _send_command 15 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0124025050 con 0x7f0124024aa0 2024-02-10T08:38:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.290+0000 7f0134ff9700 10 client.52961.objecter ms_handle_connect 0x7f0124024aa0 2024-02-10T08:38:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.291+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0124025050 con 0x7f0124024aa0 2024-02-10T08:38:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.291+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.291+0000 7f0134ff9700 10 client.52961.objecter _finish_command 15 = system:0 2024-02-10T08:38:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.291+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 4 2024-02-10T08:38:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.291+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 7 15 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 10 client.52961.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign -1 16 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 20 client.52961.objecter _get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_remove -1 16 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 20 client.52961.objecter get_session s=0x7f01240243e0 osd=7 3 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 15 client.52961.objecter _session_command_op_assign 7 16 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 4 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 10 client.52961.objecter _send_command 16 2024-02-10T08:38:15.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.305+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0124025050 con 0x7f0124024aa0 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.307+0000 7f0134ff9700 1 -- 172.21.15.107:0/2512040328 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0124025050 con 0x7f0124024aa0 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.307+0000 7f0134ff9700 10 client.52961.objecter ms_dispatch 0x7f0138001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.307+0000 7f0134ff9700 10 client.52961.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.307+0000 7f0134ff9700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 4 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.307+0000 7f0134ff9700 15 client.52961.objecter _session_command_op_remove 7 16 2024-02-10T08:38:15.309 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 librados: watch_flush enter 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 librados: watch_flush exit 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.0 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6801/3321596972 conn(0x7f0124004530 legacy=0x7f0124002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240051a0 osd=0 1 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.1 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6805/133516126 conn(0x7f0124004150 legacy=0x7f012400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003fe0 osd=1 1 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.2 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6809/3575881728 conn(0x7f012400d2f0 legacy=0x7f012400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124003630 osd=2 1 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.3 2024-02-10T08:38:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6813/2871825691 conn(0x7f0124011ff0 legacy=0x7f01240143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f0137fff700 1 -- 172.21.15.107:0/2512040328 reap_dead start 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124011970 osd=3 1 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.4 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.195:6800/2499671285 conn(0x7f012400ce50 legacy=0x7f0124016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f0137fff700 1 -- 172.21.15.107:0/2512040328 reap_dead start 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e030 osd=4 1 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.5 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.195:6804/4178321486 conn(0x7f0124017a60 legacy=0x7f012401d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f012400e1e0 osd=5 1 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.6 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.195:6808/1417479017 conn(0x7f012401f080 legacy=0x7f0124021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f0124018050 osd=6 1 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter close_session for osd.7 2024-02-10T08:38:15.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.195:6812/1934176928 conn(0x7f0124024aa0 legacy=0x7f0124022c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter put_session s=0x7f01240243e0 osd=7 1 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 20 client.52961.objecter shutdown clearing up homeless session... 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 10 client.52961.objecter successfully canceled tick 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0120063ca0 legacy=0x7f0120066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> v1:172.21.15.107:6790/0 conn(0x7f01381007f0 legacy=0x7f01380fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f0137fff700 1 -- 172.21.15.107:0/2512040328 reap_dead start 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 shutdown_connections 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 >> 172.21.15.107:0/2512040328 conn(0x7f01380f3640 msgr2=0x7f01381077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 shutdown_connections 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 -- 172.21.15.107:0/2512040328 wait complete. 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:15.308+0000 7f013e0b4700 1 librados: shutdown 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.313 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:38:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:15.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:15.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:15.834 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:15.854 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:15.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:15] "GET /metrics HTTP/1.1" 200 189891 "" "Prometheus/2.33.4" 2024-02-10T08:38:15.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:16.192 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-10T08:38:16.192 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [==========] Running 56 tests from 4 test suites. 2024-02-10T08:38:16.192 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-10T08:38:16.192 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (7802 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (20171 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6008 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2855 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3140 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2874 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2942 ms) 2024-02-10T08:38:16.193 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3024 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2965 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3017 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2999 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2991 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3020 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3014 ms) 2024-02-10T08:38:16.194 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3235 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2748 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3025 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2984 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2981 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3479 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2551 ms) 2024-02-10T08:38:16.195 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2976 ms) 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3020 ms) 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3277 ms) 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2783 ms) 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2919 ms) 2024-02-10T08:38:16.196 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-10T08:38:16.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:14.833933+0000 osd.2 (osd.2) 77 : cluster [DBG] 16.a scrub starts 2024-02-10T08:38:16.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:14.837148+0000 osd.2 (osd.2) 78 : cluster [DBG] 16.a scrub ok 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.946297+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.946416+0000 mon.a (mon.0) 1443 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:14.946493+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.949631+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.955034+0000 mon.a (mon.0) 1445 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.960103+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:14.960912+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:15.049146+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:15.164614+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2512040328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:15.536856+0000 osd.5 (osd.5) 97 : cluster [DBG] 16.f scrub starts 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:15.539841+0000 osd.5 (osd.5) 98 : cluster [DBG] 16.f scrub ok 2024-02-10T08:38:16.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:14.833933+0000 osd.2 (osd.2) 77 : cluster [DBG] 16.a scrub starts 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:14.837148+0000 osd.2 (osd.2) 78 : cluster [DBG] 16.a scrub ok 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.946297+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.946416+0000 mon.a (mon.0) 1443 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:14.946493+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.949631+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.955034+0000 mon.a (mon.0) 1445 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.960103+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:14.960912+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:15.049146+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:15.164614+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2512040328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:15.536856+0000 osd.5 (osd.5) 97 : cluster [DBG] 16.f scrub starts 2024-02-10T08:38:16.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:15.539841+0000 osd.5 (osd.5) 98 : cluster [DBG] 16.f scrub ok 2024-02-10T08:38:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:16.345 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-02-10T08:38:13.936765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11546,"num_objects":552,"num_object_clones":237,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":1096,"num_read_kb":762,"num_write":1047,"num_write_kb":617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":3296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1848,"ondisk_log_size":1848,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":818,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2145532,"kb_used_data":23964,"kb_used_omap":0,"kb_used_meta":2121472,"kb_avail":654070532,"statfs":{"total":671965249536,"available":669768224768,"internally_reserved":0,"allocated":24539136,"data_stored":11476964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2172387328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-515,"num_objects":-58,"num_object_clones":-59,"num_object_copies":-174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-58,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.924389"},"pg_stats":[{"pgid":"138.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:38:13.934038+0000","last_change":"2024-02-10T08:38:13.934038+0000","last_active":"2024-02-10T08:38:13.934038+0000","last_peered":"2024-02-10T08:38:13.934038+0000","last_clean":"2024-02-10T08:38:13.934038+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:38:13.934038+0000","last_undegraded":"2024-02-10T08:38:13.934038+0000","last_fullsized":"2024-02-10T08:38:13.934038+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:13.934038+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.934038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.3","version":"194'26","reported_seq":124,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.030987+0000","last_change":"2024-02-10T08:38:10.030987+0000","last_active":"2024-02-10T08:38:10.030987+0000","last_peered":"2024-02-10T08:38:10.030987+0000","last_clean":"2024-02-10T08:38:10.030987+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:38:10.030987+0000","last_undegraded":"2024-02-10T08:38:10.030987+0000","last_fullsized":"2024-02-10T08:38:10.030987+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.2","version":"194'14","reported_seq":115,"reported_epoch":197,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.949424+0000","last_change":"2024-02-10T08:38:10.034861+0000","last_active":"2024-02-10T08:38:10.949424+0000","last_peered":"2024-02-10T08:38:10.949424+0000","last_clean":"2024-02-10T08:38:10.949424+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:38:10.949424+0000","last_undegraded":"2024-02-10T08:38:10.949424+0000","last_fullsized":"2024-02-10T08:38:10.949424+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.1","version":"190'22","reported_seq":116,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015322+0000","last_change":"2024-02-10T08:38:06.032521+0000","last_active":"2024-02-10T08:38:10.015322+0000","last_peered":"2024-02-10T08:38:10.015322+0000","last_clean":"2024-02-10T08:38:10.015322+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:38:10.015322+0000","last_undegraded":"2024-02-10T08:38:10.015322+0000","last_fullsized":"2024-02-10T08:38:10.015322+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.0","version":"196'74","reported_seq":179,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.045345+0000","last_change":"2024-02-10T08:38:08.025136+0000","last_active":"2024-02-10T08:38:10.045345+0000","last_peered":"2024-02-10T08:38:10.045345+0000","last_clean":"2024-02-10T08:38:10.045345+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:38:10.045345+0000","last_undegraded":"2024-02-10T08:38:10.045345+0000","last_fullsized":"2024-02-10T08:38:10.045345+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":214,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.7","version":"196'26","reported_seq":102,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.049136+0000","last_change":"2024-02-10T08:38:10.037399+0000","last_active":"2024-02-10T08:38:10.049136+0000","last_peered":"2024-02-10T08:38:10.049136+0000","last_clean":"2024-02-10T08:38:10.049136+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:38:10.049136+0000","last_undegraded":"2024-02-10T08:38:10.049136+0000","last_fullsized":"2024-02-10T08:38:10.049136+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.6","version":"194'44","reported_seq":138,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.032274+0000","last_change":"2024-02-10T08:38:10.032274+0000","last_active":"2024-02-10T08:38:10.032274+0000","last_peered":"2024-02-10T08:38:10.032274+0000","last_clean":"2024-02-10T08:38:10.032274+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:38:10.032274+0000","last_undegraded":"2024-02-10T08:38:10.032274+0000","last_fullsized":"2024-02-10T08:38:10.032274+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.5","version":"196'22","reported_seq":125,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.049002+0000","last_change":"2024-02-10T08:38:10.034930+0000","last_active":"2024-02-10T08:38:10.049002+0000","last_peered":"2024-02-10T08:38:10.049002+0000","last_clean":"2024-02-10T08:38:10.049002+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:38:10.049002+0000","last_undegraded":"2024-02-10T08:38:10.049002+0000","last_fullsized":"2024-02-10T08:38:10.049002+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"188'11","reported_seq":110,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.016733+0000","last_change":"2024-02-10T08:38:04.008036+0000","last_active":"2024-02-10T08:38:10.016733+0000","last_peered":"2024-02-10T08:38:10.016733+0000","last_clean":"2024-02-10T08:38:10.016733+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:38:10.016733+0000","last_undegraded":"2024-02-10T08:38:10.016733+0000","last_fullsized":"2024-02-10T08:38:10.016733+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.b","version":"196'70","reported_seq":165,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.043462+0000","last_change":"2024-02-10T08:38:10.032314+0000","last_active":"2024-02-10T08:38:10.043462+0000","last_peered":"2024-02-10T08:38:10.043462+0000","last_clean":"2024-02-10T08:38:10.043462+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:38:10.043462+0000","last_undegraded":"2024-02-10T08:38:10.043462+0000","last_fullsized":"2024-02-10T08:38:10.043462+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":207,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.a","version":"192'16","reported_seq":116,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015141+0000","last_change":"2024-02-10T08:38:08.035122+0000","last_active":"2024-02-10T08:38:10.015141+0000","last_peered":"2024-02-10T08:38:10.015141+0000","last_clean":"2024-02-10T08:38:10.015141+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:38:10.015141+0000","last_undegraded":"2024-02-10T08:38:10.015141+0000","last_fullsized":"2024-02-10T08:38:10.015141+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.9","version":"192'66","reported_seq":144,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.020472+0000","last_change":"2024-02-10T08:38:08.034397+0000","last_active":"2024-02-10T08:38:10.020472+0000","last_peered":"2024-02-10T08:38:10.020472+0000","last_clean":"2024-02-10T08:38:10.020472+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:38:10.020472+0000","last_undegraded":"2024-02-10T08:38:10.020472+0000","last_fullsized":"2024-02-10T08:38:10.020472+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":264,"num_objects":24,"num_object_clones":18,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.8","version":"192'55","reported_seq":153,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015418+0000","last_change":"2024-02-10T08:38:08.033068+0000","last_active":"2024-02-10T08:38:10.015418+0000","last_peered":"2024-02-10T08:38:10.015418+0000","last_clean":"2024-02-10T08:38:10.015418+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:38:10.015418+0000","last_undegraded":"2024-02-10T08:38:10.015418+0000","last_fullsized":"2024-02-10T08:38:10.015418+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":220,"num_objects":20,"num_object_clones":15,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.f","version":"196'70","reported_seq":157,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.049671+0000","last_change":"2024-02-10T08:38:10.037855+0000","last_active":"2024-02-10T08:38:10.049671+0000","last_peered":"2024-02-10T08:38:10.049671+0000","last_clean":"2024-02-10T08:38:10.049671+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:38:10.049671+0000","last_undegraded":"2024-02-10T08:38:10.049671+0000","last_fullsized":"2024-02-10T08:38:10.049671+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":147,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.e","version":"194'39","reported_seq":119,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.032421+0000","last_change":"2024-02-10T08:38:10.032421+0000","last_active":"2024-02-10T08:38:10.032421+0000","last_peered":"2024-02-10T08:38:10.032421+0000","last_clean":"2024-02-10T08:38:10.032421+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:38:10.032421+0000","last_undegraded":"2024-02-10T08:38:10.032421+0000","last_fullsized":"2024-02-10T08:38:10.032421+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"60.d","version":"196'45","reported_seq":138,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.047124+0000","last_change":"2024-02-10T08:38:08.032955+0000","last_active":"2024-02-10T08:38:10.047124+0000","last_peered":"2024-02-10T08:38:10.047124+0000","last_clean":"2024-02-10T08:38:10.047124+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:38:10.047124+0000","last_undegraded":"2024-02-10T08:38:10.047124+0000","last_fullsized":"2024-02-10T08:38:10.047124+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.c","version":"191'44","reported_seq":142,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015418+0000","last_change":"2024-02-10T08:38:08.024484+0000","last_active":"2024-02-10T08:38:10.015418+0000","last_peered":"2024-02-10T08:38:10.015418+0000","last_clean":"2024-02-10T08:38:10.015418+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:38:10.015418+0000","last_undegraded":"2024-02-10T08:38:10.015418+0000","last_fullsized":"2024-02-10T08:38:10.015418+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.13","version":"196'43","reported_seq":141,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.050405+0000","last_change":"2024-02-10T08:38:10.046852+0000","last_active":"2024-02-10T08:38:10.050405+0000","last_peered":"2024-02-10T08:38:10.050405+0000","last_clean":"2024-02-10T08:38:10.050405+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:38:10.050405+0000","last_undegraded":"2024-02-10T08:38:10.050405+0000","last_fullsized":"2024-02-10T08:38:10.050405+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.12","version":"196'74","reported_seq":176,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.044113+0000","last_change":"2024-02-10T08:38:08.022411+0000","last_active":"2024-02-10T08:38:10.044113+0000","last_peered":"2024-02-10T08:38:10.044113+0000","last_clean":"2024-02-10T08:38:10.044113+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:38:10.044113+0000","last_undegraded":"2024-02-10T08:38:10.044113+0000","last_fullsized":"2024-02-10T08:38:10.044113+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":214,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.11","version":"196'78","reported_seq":166,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.049010+0000","last_change":"2024-02-10T08:38:06.012290+0000","last_active":"2024-02-10T08:38:10.049010+0000","last_peered":"2024-02-10T08:38:10.049010+0000","last_clean":"2024-02-10T08:38:10.049010+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:38:10.049010+0000","last_undegraded":"2024-02-10T08:38:10.049010+0000","last_fullsized":"2024-02-10T08:38:10.049010+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.10","version":"192'52","reported_seq":159,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.029905+0000","last_change":"2024-02-10T08:38:08.025726+0000","last_active":"2024-02-10T08:38:10.029905+0000","last_peered":"2024-02-10T08:38:10.029905+0000","last_clean":"2024-02-10T08:38:10.029905+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:38:10.029905+0000","last_undegraded":"2024-02-10T08:38:10.029905+0000","last_fullsized":"2024-02-10T08:38:10.029905+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.17","version":"194'59","reported_seq":153,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.026431+0000","last_change":"2024-02-10T08:38:09.270297+0000","last_active":"2024-02-10T08:38:10.026431+0000","last_peered":"2024-02-10T08:38:10.026431+0000","last_clean":"2024-02-10T08:38:10.026431+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:38:10.026431+0000","last_undegraded":"2024-02-10T08:38:10.026431+0000","last_fullsized":"2024-02-10T08:38:10.026431+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.16","version":"194'44","reported_seq":160,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.030409+0000","last_change":"2024-02-10T08:38:10.030409+0000","last_active":"2024-02-10T08:38:10.030409+0000","last_peered":"2024-02-10T08:38:10.030409+0000","last_clean":"2024-02-10T08:38:10.030409+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:38:10.030409+0000","last_undegraded":"2024-02-10T08:38:10.030409+0000","last_fullsized":"2024-02-10T08:38:10.030409+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"60.15","version":"194'11","reported_seq":109,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.037321+0000","last_change":"2024-02-10T08:38:10.037321+0000","last_active":"2024-02-10T08:38:10.037321+0000","last_peered":"2024-02-10T08:38:10.037321+0000","last_clean":"2024-02-10T08:38:10.037321+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:38:10.037321+0000","last_undegraded":"2024-02-10T08:38:10.037321+0000","last_fullsized":"2024-02-10T08:38:10.037321+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.14","version":"196'30","reported_seq":127,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.038507+0000","last_change":"2024-02-10T08:38:10.032492+0000","last_active":"2024-02-10T08:38:10.038507+0000","last_peered":"2024-02-10T08:38:10.038507+0000","last_clean":"2024-02-10T08:38:10.038507+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:38:10.038507+0000","last_undegraded":"2024-02-10T08:38:10.038507+0000","last_fullsized":"2024-02-10T08:38:10.038507+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.1b","version":"186'44","reported_seq":123,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.017717+0000","last_change":"2024-02-10T08:38:01.996768+0000","last_active":"2024-02-10T08:38:10.017717+0000","last_peered":"2024-02-10T08:38:10.017717+0000","last_clean":"2024-02-10T08:38:10.017717+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:38:10.017717+0000","last_undegraded":"2024-02-10T08:38:10.017717+0000","last_fullsized":"2024-02-10T08:38:10.017717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.1a","version":"196'52","reported_seq":138,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.043895+0000","last_change":"2024-02-10T08:38:08.034442+0000","last_active":"2024-02-10T08:38:10.043895+0000","last_peered":"2024-02-10T08:38:10.043895+0000","last_clean":"2024-02-10T08:38:10.043895+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:38:10.043895+0000","last_undegraded":"2024-02-10T08:38:10.043895+0000","last_fullsized":"2024-02-10T08:38:10.043895+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"60.19","version":"196'26","reported_seq":133,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.043520+0000","last_change":"2024-02-10T08:38:10.031012+0000","last_active":"2024-02-10T08:38:10.043520+0000","last_peered":"2024-02-10T08:38:10.043520+0000","last_clean":"2024-02-10T08:38:10.043520+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:38:10.043520+0000","last_undegraded":"2024-02-10T08:38:10.043520+0000","last_fullsized":"2024-02-10T08:38:10.043520+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.18","version":"186'22","reported_seq":101,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015286+0000","last_change":"2024-02-10T08:38:01.998557+0000","last_active":"2024-02-10T08:38:10.015286+0000","last_peered":"2024-02-10T08:38:10.015286+0000","last_clean":"2024-02-10T08:38:10.015286+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:38:10.015286+0000","last_undegraded":"2024-02-10T08:38:10.015286+0000","last_fullsized":"2024-02-10T08:38:10.015286+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"60.1f","version":"194'26","reported_seq":136,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.046815+0000","last_change":"2024-02-10T08:38:10.046815+0000","last_active":"2024-02-10T08:38:10.046815+0000","last_peered":"2024-02-10T08:38:10.046815+0000","last_clean":"2024-02-10T08:38:10.046815+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:38:10.046815+0000","last_undegraded":"2024-02-10T08:38:10.046815+0000","last_fullsized":"2024-02-10T08:38:10.046815+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"60.1e","version":"191'11","reported_seq":90,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.023156+0000","last_change":"2024-02-10T08:38:08.021556+0000","last_active":"2024-02-10T08:38:10.023156+0000","last_peered":"2024-02-10T08:38:10.023156+0000","last_clean":"2024-02-10T08:38:10.023156+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:38:10.023156+0000","last_undegraded":"2024-02-10T08:38:10.023156+0000","last_fullsized":"2024-02-10T08:38:10.023156+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"60.1d","version":"196'15","reported_seq":102,"reported_epoch":196,"state":"active+clean+snaptrim","last_fresh":"2024-02-10T08:38:10.041497+0000","last_change":"2024-02-10T08:38:10.031727+0000","last_active":"2024-02-10T08:38:10.041497+0000","last_peered":"2024-02-10T08:38:10.041497+0000","last_clean":"2024-02-10T08:38:10.041497+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:38:10.041497+0000","last_undegraded":"2024-02-10T08:38:10.041497+0000","last_fullsized":"2024-02-10T08:38:10.041497+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"60.1c","version":"190'22","reported_seq":121,"reported_epoch":196,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-10T08:38:10.015254+0000","last_change":"2024-02-10T08:38:05.999649+0000","last_active":"2024-02-10T08:38:10.015254+0000","last_peered":"2024-02-10T08:38:10.015254+0000","last_clean":"2024-02-10T08:38:10.015254+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:38:10.015254+0000","last_undegraded":"2024-02-10T08:38:10.015254+0000","last_fullsized":"2024-02-10T08:38:10.015254+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"25.15","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022766+0000","last_change":"2024-02-10T08:37:10.250275+0000","last_active":"2024-02-10T08:38:10.022766+0000","last_peered":"2024-02-10T08:38:10.022766+0000","last_clean":"2024-02-10T08:38:10.022766+0000","last_became_active":"2024-02-10T08:37:10.249567+0000","last_became_peered":"2024-02-10T08:37:10.249567+0000","last_unstale":"2024-02-10T08:38:10.022766+0000","last_undegraded":"2024-02-10T08:38:10.022766+0000","last_fullsized":"2024-02-10T08:38:10.022766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":242,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028273+0000","last_change":"2024-02-10T08:36:58.130213+0000","last_active":"2024-02-10T08:38:10.028273+0000","last_peered":"2024-02-10T08:38:10.028273+0000","last_clean":"2024-02-10T08:38:10.028273+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:38:10.028273+0000","last_undegraded":"2024-02-10T08:38:10.028273+0000","last_fullsized":"2024-02-10T08:38:10.028273+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:16.169618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.169618+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":189,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027769+0000","last_change":"2024-02-10T08:37:07.840809+0000","last_active":"2024-02-10T08:38:10.027769+0000","last_peered":"2024-02-10T08:38:10.027769+0000","last_clean":"2024-02-10T08:38:10.027769+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:38:10.027769+0000","last_undegraded":"2024-02-10T08:38:10.027769+0000","last_fullsized":"2024-02-10T08:38:10.027769+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:37:07.840738+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.840738+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":160,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.014821+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:38:10.014821+0000","last_peered":"2024-02-10T08:38:10.014821+0000","last_clean":"2024-02-10T08:38:10.014821+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:38:10.014821+0000","last_undegraded":"2024-02-10T08:38:10.014821+0000","last_fullsized":"2024-02-10T08:38:10.014821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":167,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026410+0000","last_change":"2024-02-10T08:37:44.597397+0000","last_active":"2024-02-10T08:38:10.026410+0000","last_peered":"2024-02-10T08:38:10.026410+0000","last_clean":"2024-02-10T08:38:10.026410+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:38:10.026410+0000","last_undegraded":"2024-02-10T08:38:10.026410+0000","last_fullsized":"2024-02-10T08:38:10.026410+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.597335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.597335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015544+0000","last_change":"2024-02-10T08:36:37.051266+0000","last_active":"2024-02-10T08:38:10.015544+0000","last_peered":"2024-02-10T08:38:10.015544+0000","last_clean":"2024-02-10T08:38:10.015544+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:38:10.015544+0000","last_undegraded":"2024-02-10T08:38:10.015544+0000","last_fullsized":"2024-02-10T08:38:10.015544+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"103'5","reported_seq":121,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023368+0000","last_change":"2024-02-10T08:37:11.012191+0000","last_active":"2024-02-10T08:38:10.023368+0000","last_peered":"2024-02-10T08:38:10.023368+0000","last_clean":"2024-02-10T08:38:10.023368+0000","last_became_active":"2024-02-10T08:37:10.219488+0000","last_became_peered":"2024-02-10T08:37:10.219488+0000","last_unstale":"2024-02-10T08:38:10.023368+0000","last_undegraded":"2024-02-10T08:38:10.023368+0000","last_fullsized":"2024-02-10T08:38:10.023368+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":137,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023832+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:38:10.023832+0000","last_peered":"2024-02-10T08:38:10.023832+0000","last_clean":"2024-02-10T08:38:10.023832+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:38:10.023832+0000","last_undegraded":"2024-02-10T08:38:10.023832+0000","last_fullsized":"2024-02-10T08:38:10.023832+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":165,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019329+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:38:10.019329+0000","last_peered":"2024-02-10T08:38:10.019329+0000","last_clean":"2024-02-10T08:38:10.019329+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:38:10.019329+0000","last_undegraded":"2024-02-10T08:38:10.019329+0000","last_fullsized":"2024-02-10T08:38:10.019329+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024339+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:38:10.024339+0000","last_peered":"2024-02-10T08:38:10.024339+0000","last_clean":"2024-02-10T08:38:10.024339+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:38:10.024339+0000","last_undegraded":"2024-02-10T08:38:10.024339+0000","last_fullsized":"2024-02-10T08:38:10.024339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":78,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027972+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:38:10.027972+0000","last_peered":"2024-02-10T08:38:10.027972+0000","last_clean":"2024-02-10T08:38:10.027972+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:38:10.027972+0000","last_undegraded":"2024-02-10T08:38:10.027972+0000","last_fullsized":"2024-02-10T08:38:10.027972+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":102,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.044476+0000","last_change":"2024-02-10T08:36:37.033242+0000","last_active":"2024-02-10T08:38:10.044476+0000","last_peered":"2024-02-10T08:38:10.044476+0000","last_clean":"2024-02-10T08:38:10.044476+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:38:10.044476+0000","last_undegraded":"2024-02-10T08:38:10.044476+0000","last_fullsized":"2024-02-10T08:38:10.044476+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.17","version":"103'5","reported_seq":121,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.029828+0000","last_change":"2024-02-10T08:37:10.688144+0000","last_active":"2024-02-10T08:38:10.029828+0000","last_peered":"2024-02-10T08:38:10.029828+0000","last_clean":"2024-02-10T08:38:10.029828+0000","last_became_active":"2024-02-10T08:37:10.259720+0000","last_became_peered":"2024-02-10T08:37:10.259720+0000","last_unstale":"2024-02-10T08:38:10.029828+0000","last_undegraded":"2024-02-10T08:38:10.029828+0000","last_fullsized":"2024-02-10T08:38:10.029828+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019113+0000","last_change":"2024-02-10T08:37:39.547805+0000","last_active":"2024-02-10T08:38:10.019113+0000","last_peered":"2024-02-10T08:38:10.019113+0000","last_clean":"2024-02-10T08:38:10.019113+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:38:10.019113+0000","last_undegraded":"2024-02-10T08:38:10.019113+0000","last_fullsized":"2024-02-10T08:38:10.019113+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:37:39.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.547736+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015879+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:38:10.015879+0000","last_peered":"2024-02-10T08:38:10.015879+0000","last_clean":"2024-02-10T08:38:10.015879+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:38:10.015879+0000","last_undegraded":"2024-02-10T08:38:10.015879+0000","last_fullsized":"2024-02-10T08:38:10.015879+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023616+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:38:10.023616+0000","last_peered":"2024-02-10T08:38:10.023616+0000","last_clean":"2024-02-10T08:38:10.023616+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:38:10.023616+0000","last_undegraded":"2024-02-10T08:38:10.023616+0000","last_fullsized":"2024-02-10T08:38:10.023616+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030120+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:38:10.030120+0000","last_peered":"2024-02-10T08:38:10.030120+0000","last_clean":"2024-02-10T08:38:10.030120+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:38:10.030120+0000","last_undegraded":"2024-02-10T08:38:10.030120+0000","last_fullsized":"2024-02-10T08:38:10.030120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032612+0000","last_change":"2024-02-10T08:36:37.021497+0000","last_active":"2024-02-10T08:38:10.032612+0000","last_peered":"2024-02-10T08:38:10.032612+0000","last_clean":"2024-02-10T08:38:10.032612+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:38:10.032612+0000","last_undegraded":"2024-02-10T08:38:10.032612+0000","last_fullsized":"2024-02-10T08:38:10.032612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027861+0000","last_change":"2024-02-10T08:37:10.200689+0000","last_active":"2024-02-10T08:38:10.027861+0000","last_peered":"2024-02-10T08:38:10.027861+0000","last_clean":"2024-02-10T08:38:10.027861+0000","last_became_active":"2024-02-10T08:37:10.200347+0000","last_became_peered":"2024-02-10T08:37:10.200347+0000","last_unstale":"2024-02-10T08:38:10.027861+0000","last_undegraded":"2024-02-10T08:38:10.027861+0000","last_fullsized":"2024-02-10T08:38:10.027861+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":191,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021805+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:38:10.021805+0000","last_peered":"2024-02-10T08:38:10.021805+0000","last_clean":"2024-02-10T08:38:10.021805+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:38:10.021805+0000","last_undegraded":"2024-02-10T08:38:10.021805+0000","last_fullsized":"2024-02-10T08:38:10.021805+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":165,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.025036+0000","last_change":"2024-02-10T08:37:38.545880+0000","last_active":"2024-02-10T08:38:10.025036+0000","last_peered":"2024-02-10T08:38:10.025036+0000","last_clean":"2024-02-10T08:38:10.025036+0000","last_became_active":"2024-02-10T08:33:18.105679+0000","last_became_peered":"2024-02-10T08:33:18.105679+0000","last_unstale":"2024-02-10T08:38:10.025036+0000","last_undegraded":"2024-02-10T08:38:10.025036+0000","last_fullsized":"2024-02-10T08:38:10.025036+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023929+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:38:10.023929+0000","last_peered":"2024-02-10T08:38:10.023929+0000","last_clean":"2024-02-10T08:38:10.023929+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:38:10.023929+0000","last_undegraded":"2024-02-10T08:38:10.023929+0000","last_fullsized":"2024-02-10T08:38:10.023929+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019085+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:38:10.019085+0000","last_peered":"2024-02-10T08:38:10.019085+0000","last_clean":"2024-02-10T08:38:10.019085+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:38:10.019085+0000","last_undegraded":"2024-02-10T08:38:10.019085+0000","last_fullsized":"2024-02-10T08:38:10.019085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015424+0000","last_change":"2024-02-10T08:36:37.075652+0000","last_active":"2024-02-10T08:38:10.015424+0000","last_peered":"2024-02-10T08:38:10.015424+0000","last_clean":"2024-02-10T08:38:10.015424+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:38:10.015424+0000","last_undegraded":"2024-02-10T08:38:10.015424+0000","last_fullsized":"2024-02-10T08:38:10.015424+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"103'7","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015559+0000","last_change":"2024-02-10T08:37:10.141080+0000","last_active":"2024-02-10T08:38:10.015559+0000","last_peered":"2024-02-10T08:38:10.015559+0000","last_clean":"2024-02-10T08:38:10.015559+0000","last_became_active":"2024-02-10T08:37:10.115979+0000","last_became_peered":"2024-02-10T08:37:10.115979+0000","last_unstale":"2024-02-10T08:38:10.015559+0000","last_undegraded":"2024-02-10T08:38:10.015559+0000","last_fullsized":"2024-02-10T08:38:10.015559+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.017163+0000","last_change":"2024-02-10T08:35:38.779527+0000","last_active":"2024-02-10T08:38:10.017163+0000","last_peered":"2024-02-10T08:38:10.017163+0000","last_clean":"2024-02-10T08:38:10.017163+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:38:10.017163+0000","last_undegraded":"2024-02-10T08:38:10.017163+0000","last_fullsized":"2024-02-10T08:38:10.017163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:35:38.779433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:35:38.779433+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":165,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019880+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:38:10.019880+0000","last_peered":"2024-02-10T08:38:10.019880+0000","last_clean":"2024-02-10T08:38:10.019880+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:38:10.019880+0000","last_undegraded":"2024-02-10T08:38:10.019880+0000","last_fullsized":"2024-02-10T08:38:10.019880+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":242,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022827+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:38:10.022827+0000","last_peered":"2024-02-10T08:38:10.022827+0000","last_clean":"2024-02-10T08:38:10.022827+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:38:10.022827+0000","last_undegraded":"2024-02-10T08:38:10.022827+0000","last_fullsized":"2024-02-10T08:38:10.022827+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023079+0000","last_change":"2024-02-10T08:36:13.778764+0000","last_active":"2024-02-10T08:38:10.023079+0000","last_peered":"2024-02-10T08:38:10.023079+0000","last_clean":"2024-02-10T08:38:10.023079+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:38:10.023079+0000","last_undegraded":"2024-02-10T08:38:10.023079+0000","last_fullsized":"2024-02-10T08:38:10.023079+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.778673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.778673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020127+0000","last_change":"2024-02-10T08:37:10.263630+0000","last_active":"2024-02-10T08:38:10.020127+0000","last_peered":"2024-02-10T08:38:10.020127+0000","last_clean":"2024-02-10T08:38:10.020127+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:38:10.020127+0000","last_undegraded":"2024-02-10T08:38:10.020127+0000","last_fullsized":"2024-02-10T08:38:10.020127+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020077+0000","last_change":"2024-02-10T08:37:10.183873+0000","last_active":"2024-02-10T08:38:10.020077+0000","last_peered":"2024-02-10T08:38:10.020077+0000","last_clean":"2024-02-10T08:38:10.020077+0000","last_became_active":"2024-02-10T08:37:10.183562+0000","last_became_peered":"2024-02-10T08:37:10.183562+0000","last_unstale":"2024-02-10T08:38:10.020077+0000","last_undegraded":"2024-02-10T08:38:10.020077+0000","last_fullsized":"2024-02-10T08:38:10.020077+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020329+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:38:10.020329+0000","last_peered":"2024-02-10T08:38:10.020329+0000","last_clean":"2024-02-10T08:38:10.020329+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:38:10.020329+0000","last_undegraded":"2024-02-10T08:38:10.020329+0000","last_fullsized":"2024-02-10T08:38:10.020329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":166,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020964+0000","last_change":"2024-02-10T08:36:16.423854+0000","last_active":"2024-02-10T08:38:10.020964+0000","last_peered":"2024-02-10T08:38:10.020964+0000","last_clean":"2024-02-10T08:38:10.020964+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:38:10.020964+0000","last_undegraded":"2024-02-10T08:38:10.020964+0000","last_fullsized":"2024-02-10T08:38:10.020964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:16.423786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:16.423786+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":267,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022979+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:38:10.022979+0000","last_peered":"2024-02-10T08:38:10.022979+0000","last_clean":"2024-02-10T08:38:10.022979+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:38:10.022979+0000","last_undegraded":"2024-02-10T08:38:10.022979+0000","last_fullsized":"2024-02-10T08:38:10.022979+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":162,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031585+0000","last_change":"2024-02-10T08:37:24.680471+0000","last_active":"2024-02-10T08:38:10.031585+0000","last_peered":"2024-02-10T08:38:10.031585+0000","last_clean":"2024-02-10T08:38:10.031585+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:38:10.031585+0000","last_undegraded":"2024-02-10T08:38:10.031585+0000","last_fullsized":"2024-02-10T08:38:10.031585+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:24.680407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:24.680407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":102,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032194+0000","last_change":"2024-02-10T08:36:37.148745+0000","last_active":"2024-02-10T08:38:10.032194+0000","last_peered":"2024-02-10T08:38:10.032194+0000","last_clean":"2024-02-10T08:38:10.032194+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:38:10.032194+0000","last_undegraded":"2024-02-10T08:38:10.032194+0000","last_fullsized":"2024-02-10T08:38:10.032194+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.13","version":"103'9","reported_seq":129,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018035+0000","last_change":"2024-02-10T08:37:10.221042+0000","last_active":"2024-02-10T08:38:10.018035+0000","last_peered":"2024-02-10T08:38:10.018035+0000","last_clean":"2024-02-10T08:38:10.018035+0000","last_became_active":"2024-02-10T08:37:10.136194+0000","last_became_peered":"2024-02-10T08:37:10.136194+0000","last_unstale":"2024-02-10T08:38:10.018035+0000","last_undegraded":"2024-02-10T08:38:10.018035+0000","last_fullsized":"2024-02-10T08:38:10.018035+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019535+0000","last_change":"2024-02-10T08:36:11.732545+0000","last_active":"2024-02-10T08:38:10.019535+0000","last_peered":"2024-02-10T08:38:10.019535+0000","last_clean":"2024-02-10T08:38:10.019535+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:38:10.019535+0000","last_undegraded":"2024-02-10T08:38:10.019535+0000","last_fullsized":"2024-02-10T08:38:10.019535+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:36:11.732461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:11.732461+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030415+0000","last_change":"2024-02-10T08:37:10.150021+0000","last_active":"2024-02-10T08:38:10.030415+0000","last_peered":"2024-02-10T08:38:10.030415+0000","last_clean":"2024-02-10T08:38:10.030415+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:38:10.030415+0000","last_undegraded":"2024-02-10T08:38:10.030415+0000","last_fullsized":"2024-02-10T08:38:10.030415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.142541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.142541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":194,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.044259+0000","last_change":"2024-02-10T08:37:21.746790+0000","last_active":"2024-02-10T08:38:10.044259+0000","last_peered":"2024-02-10T08:38:10.044259+0000","last_clean":"2024-02-10T08:38:10.044259+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:38:10.044259+0000","last_undegraded":"2024-02-10T08:38:10.044259+0000","last_fullsized":"2024-02-10T08:38:10.044259+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:37:21.746710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.746710+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028147+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:38:10.028147+0000","last_peered":"2024-02-10T08:38:10.028147+0000","last_clean":"2024-02-10T08:38:10.028147+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:38:10.028147+0000","last_undegraded":"2024-02-10T08:38:10.028147+0000","last_fullsized":"2024-02-10T08:38:10.028147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.017927+0000","last_change":"2024-02-10T08:36:37.134622+0000","last_active":"2024-02-10T08:38:10.017927+0000","last_peered":"2024-02-10T08:38:10.017927+0000","last_clean":"2024-02-10T08:38:10.017927+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:38:10.017927+0000","last_undegraded":"2024-02-10T08:38:10.017927+0000","last_fullsized":"2024-02-10T08:38:10.017927+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016883+0000","last_change":"2024-02-10T08:36:37.065813+0000","last_active":"2024-02-10T08:38:10.016883+0000","last_peered":"2024-02-10T08:38:10.016883+0000","last_clean":"2024-02-10T08:38:10.016883+0000","last_became_active":"2024-02-10T08:36:37.065010+0000","last_became_peered":"2024-02-10T08:36:37.065010+0000","last_unstale":"2024-02-10T08:38:10.016883+0000","last_undegraded":"2024-02-10T08:38:10.016883+0000","last_fullsized":"2024-02-10T08:38:10.016883+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020307+0000","last_change":"2024-02-10T08:36:28.076465+0000","last_active":"2024-02-10T08:38:10.020307+0000","last_peered":"2024-02-10T08:38:10.020307+0000","last_clean":"2024-02-10T08:38:10.020307+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:38:10.020307+0000","last_undegraded":"2024-02-10T08:38:10.020307+0000","last_fullsized":"2024-02-10T08:38:10.020307+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:28.076401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.076401+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015897+0000","last_change":"2024-02-10T08:36:17.729319+0000","last_active":"2024-02-10T08:38:10.015897+0000","last_peered":"2024-02-10T08:38:10.015897+0000","last_clean":"2024-02-10T08:38:10.015897+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:38:10.015897+0000","last_undegraded":"2024-02-10T08:38:10.015897+0000","last_fullsized":"2024-02-10T08:38:10.015897+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:36:17.729249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.729249+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023432+0000","last_change":"2024-02-10T08:37:19.310327+0000","last_active":"2024-02-10T08:38:10.023432+0000","last_peered":"2024-02-10T08:38:10.023432+0000","last_clean":"2024-02-10T08:38:10.023432+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:38:10.023432+0000","last_undegraded":"2024-02-10T08:38:10.023432+0000","last_fullsized":"2024-02-10T08:38:10.023432+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.310252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.310252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.025559+0000","last_change":"2024-02-10T08:37:09.215324+0000","last_active":"2024-02-10T08:38:10.025559+0000","last_peered":"2024-02-10T08:38:10.025559+0000","last_clean":"2024-02-10T08:38:10.025559+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:38:10.025559+0000","last_undegraded":"2024-02-10T08:38:10.025559+0000","last_fullsized":"2024-02-10T08:38:10.025559+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:09.215257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:09.215257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019467+0000","last_change":"2024-02-10T08:37:10.160633+0000","last_active":"2024-02-10T08:38:10.019467+0000","last_peered":"2024-02-10T08:38:10.019467+0000","last_clean":"2024-02-10T08:38:10.019467+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:38:10.019467+0000","last_undegraded":"2024-02-10T08:38:10.019467+0000","last_fullsized":"2024-02-10T08:38:10.019467+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"103'7","reported_seq":109,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031887+0000","last_change":"2024-02-10T08:36:37.149279+0000","last_active":"2024-02-10T08:38:10.031887+0000","last_peered":"2024-02-10T08:38:10.031887+0000","last_clean":"2024-02-10T08:38:10.031887+0000","last_became_active":"2024-02-10T08:36:37.148954+0000","last_became_peered":"2024-02-10T08:36:37.148954+0000","last_unstale":"2024-02-10T08:38:10.031887+0000","last_undegraded":"2024-02-10T08:38:10.031887+0000","last_fullsized":"2024-02-10T08:38:10.031887+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":156,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016115+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:38:10.016115+0000","last_peered":"2024-02-10T08:38:10.016115+0000","last_clean":"2024-02-10T08:38:10.016115+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:38:10.016115+0000","last_undegraded":"2024-02-10T08:38:10.016115+0000","last_fullsized":"2024-02-10T08:38:10.016115+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020265+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:38:10.020265+0000","last_peered":"2024-02-10T08:38:10.020265+0000","last_clean":"2024-02-10T08:38:10.020265+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:38:10.020265+0000","last_undegraded":"2024-02-10T08:38:10.020265+0000","last_fullsized":"2024-02-10T08:38:10.020265+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":167,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023460+0000","last_change":"2024-02-10T08:36:32.450905+0000","last_active":"2024-02-10T08:38:10.023460+0000","last_peered":"2024-02-10T08:38:10.023460+0000","last_clean":"2024-02-10T08:38:10.023460+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:38:10.023460+0000","last_undegraded":"2024-02-10T08:38:10.023460+0000","last_fullsized":"2024-02-10T08:38:10.023460+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:36:32.450807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.450807+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018287+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:38:10.018287+0000","last_peered":"2024-02-10T08:38:10.018287+0000","last_clean":"2024-02-10T08:38:10.018287+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:38:10.018287+0000","last_undegraded":"2024-02-10T08:38:10.018287+0000","last_fullsized":"2024-02-10T08:38:10.018287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027889+0000","last_change":"2024-02-10T08:37:10.260060+0000","last_active":"2024-02-10T08:38:10.027889+0000","last_peered":"2024-02-10T08:38:10.027889+0000","last_clean":"2024-02-10T08:38:10.027889+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:38:10.027889+0000","last_undegraded":"2024-02-10T08:38:10.027889+0000","last_fullsized":"2024-02-10T08:38:10.027889+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024865+0000","last_change":"2024-02-10T08:37:10.221508+0000","last_active":"2024-02-10T08:38:10.024865+0000","last_peered":"2024-02-10T08:38:10.024865+0000","last_clean":"2024-02-10T08:38:10.024865+0000","last_became_active":"2024-02-10T08:37:10.221122+0000","last_became_peered":"2024-02-10T08:37:10.221122+0000","last_unstale":"2024-02-10T08:38:10.024865+0000","last_undegraded":"2024-02-10T08:38:10.024865+0000","last_fullsized":"2024-02-10T08:38:10.024865+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019203+0000","last_change":"2024-02-10T08:37:43.520384+0000","last_active":"2024-02-10T08:38:10.019203+0000","last_peered":"2024-02-10T08:38:10.019203+0000","last_clean":"2024-02-10T08:38:10.019203+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:38:10.019203+0000","last_undegraded":"2024-02-10T08:38:10.019203+0000","last_fullsized":"2024-02-10T08:38:10.019203+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.520307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.520307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":138,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032527+0000","last_change":"2024-02-10T08:37:10.868785+0000","last_active":"2024-02-10T08:38:10.032527+0000","last_peered":"2024-02-10T08:38:10.032527+0000","last_clean":"2024-02-10T08:38:10.032527+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:38:10.032527+0000","last_undegraded":"2024-02-10T08:38:10.032527+0000","last_fullsized":"2024-02-10T08:38:10.032527+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:36:15.871060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.871060+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031984+0000","last_change":"2024-02-10T08:37:10.151669+0000","last_active":"2024-02-10T08:38:10.031984+0000","last_peered":"2024-02-10T08:38:10.031984+0000","last_clean":"2024-02-10T08:38:10.031984+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:38:10.031984+0000","last_undegraded":"2024-02-10T08:38:10.031984+0000","last_fullsized":"2024-02-10T08:38:10.031984+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.048119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.048119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021108+0000","last_change":"2024-02-10T08:36:20.500433+0000","last_active":"2024-02-10T08:38:10.021108+0000","last_peered":"2024-02-10T08:38:10.021108+0000","last_clean":"2024-02-10T08:38:10.021108+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:38:10.021108+0000","last_undegraded":"2024-02-10T08:38:10.021108+0000","last_fullsized":"2024-02-10T08:38:10.021108+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.500313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.500313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018840+0000","last_change":"2024-02-10T08:37:10.233957+0000","last_active":"2024-02-10T08:38:10.018840+0000","last_peered":"2024-02-10T08:38:10.018840+0000","last_clean":"2024-02-10T08:38:10.018840+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:38:10.018840+0000","last_undegraded":"2024-02-10T08:38:10.018840+0000","last_fullsized":"2024-02-10T08:38:10.018840+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"103'10","reported_seq":128,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021736+0000","last_change":"2024-02-10T08:37:10.445260+0000","last_active":"2024-02-10T08:38:10.021736+0000","last_peered":"2024-02-10T08:38:10.021736+0000","last_clean":"2024-02-10T08:38:10.021736+0000","last_became_active":"2024-02-10T08:37:10.261034+0000","last_became_peered":"2024-02-10T08:37:10.261034+0000","last_unstale":"2024-02-10T08:38:10.021736+0000","last_undegraded":"2024-02-10T08:38:10.021736+0000","last_fullsized":"2024-02-10T08:38:10.021736+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.044325+0000","last_change":"2024-02-10T08:37:10.149313+0000","last_active":"2024-02-10T08:38:10.044325+0000","last_peered":"2024-02-10T08:38:10.044325+0000","last_clean":"2024-02-10T08:38:10.044325+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:38:10.044325+0000","last_undegraded":"2024-02-10T08:38:10.044325+0000","last_fullsized":"2024-02-10T08:38:10.044325+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.714301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.714301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":122,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023612+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:38:10.023612+0000","last_peered":"2024-02-10T08:38:10.023612+0000","last_clean":"2024-02-10T08:38:10.023612+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:38:10.023612+0000","last_undegraded":"2024-02-10T08:38:10.023612+0000","last_fullsized":"2024-02-10T08:38:10.023612+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030174+0000","last_change":"2024-02-10T08:37:21.277884+0000","last_active":"2024-02-10T08:38:10.030174+0000","last_peered":"2024-02-10T08:38:10.030174+0000","last_clean":"2024-02-10T08:38:10.030174+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:38:10.030174+0000","last_undegraded":"2024-02-10T08:38:10.030174+0000","last_fullsized":"2024-02-10T08:38:10.030174+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.277786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.277786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015744+0000","last_change":"2024-02-10T08:37:10.215017+0000","last_active":"2024-02-10T08:38:10.015744+0000","last_peered":"2024-02-10T08:38:10.015744+0000","last_clean":"2024-02-10T08:38:10.015744+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:38:10.015744+0000","last_undegraded":"2024-02-10T08:38:10.015744+0000","last_fullsized":"2024-02-10T08:38:10.015744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:41.795076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:35:41.795076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":160,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024021+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:38:10.024021+0000","last_peered":"2024-02-10T08:38:10.024021+0000","last_clean":"2024-02-10T08:38:10.024021+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:38:10.024021+0000","last_undegraded":"2024-02-10T08:38:10.024021+0000","last_fullsized":"2024-02-10T08:38:10.024021+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018900+0000","last_change":"2024-02-10T08:37:10.153879+0000","last_active":"2024-02-10T08:38:10.018900+0000","last_peered":"2024-02-10T08:38:10.018900+0000","last_clean":"2024-02-10T08:38:10.018900+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:38:10.018900+0000","last_undegraded":"2024-02-10T08:38:10.018900+0000","last_fullsized":"2024-02-10T08:38:10.018900+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"103'2","reported_seq":85,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026586+0000","last_change":"2024-02-10T08:37:10.979010+0000","last_active":"2024-02-10T08:38:10.026586+0000","last_peered":"2024-02-10T08:38:10.026586+0000","last_clean":"2024-02-10T08:38:10.026586+0000","last_became_active":"2024-02-10T08:37:10.155420+0000","last_became_peered":"2024-02-10T08:37:10.155420+0000","last_unstale":"2024-02-10T08:38:10.026586+0000","last_undegraded":"2024-02-10T08:38:10.026586+0000","last_fullsized":"2024-02-10T08:38:10.026586+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":156,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.029019+0000","last_change":"2024-02-10T08:36:27.052479+0000","last_active":"2024-02-10T08:38:10.029019+0000","last_peered":"2024-02-10T08:38:10.029019+0000","last_clean":"2024-02-10T08:38:10.029019+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:38:10.029019+0000","last_undegraded":"2024-02-10T08:38:10.029019+0000","last_fullsized":"2024-02-10T08:38:10.029019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.052362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.052362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":184,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022560+0000","last_change":"2024-02-10T08:37:40.592997+0000","last_active":"2024-02-10T08:38:10.022560+0000","last_peered":"2024-02-10T08:38:10.022560+0000","last_clean":"2024-02-10T08:38:10.022560+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:38:10.022560+0000","last_undegraded":"2024-02-10T08:38:10.022560+0000","last_fullsized":"2024-02-10T08:38:10.022560+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:37:40.592930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.592930+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030331+0000","last_change":"2024-02-10T08:37:10.149914+0000","last_active":"2024-02-10T08:38:10.030331+0000","last_peered":"2024-02-10T08:38:10.030331+0000","last_clean":"2024-02-10T08:38:10.030331+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:38:10.030331+0000","last_undegraded":"2024-02-10T08:38:10.030331+0000","last_fullsized":"2024-02-10T08:38:10.030331+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.053809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.053809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032434+0000","last_change":"2024-02-10T08:36:30.795587+0000","last_active":"2024-02-10T08:38:10.032434+0000","last_peered":"2024-02-10T08:38:10.032434+0000","last_clean":"2024-02-10T08:38:10.032434+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:38:10.032434+0000","last_undegraded":"2024-02-10T08:38:10.032434+0000","last_fullsized":"2024-02-10T08:38:10.032434+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.795494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.795494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018180+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:38:10.018180+0000","last_peered":"2024-02-10T08:38:10.018180+0000","last_clean":"2024-02-10T08:38:10.018180+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:38:10.018180+0000","last_undegraded":"2024-02-10T08:38:10.018180+0000","last_fullsized":"2024-02-10T08:38:10.018180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.044414+0000","last_change":"2024-02-10T08:36:29.804845+0000","last_active":"2024-02-10T08:38:10.044414+0000","last_peered":"2024-02-10T08:38:10.044414+0000","last_clean":"2024-02-10T08:38:10.044414+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:38:10.044414+0000","last_undegraded":"2024-02-10T08:38:10.044414+0000","last_fullsized":"2024-02-10T08:38:10.044414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:29.804761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.804761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021848+0000","last_change":"2024-02-10T08:37:10.151100+0000","last_active":"2024-02-10T08:38:10.021848+0000","last_peered":"2024-02-10T08:38:10.021848+0000","last_clean":"2024-02-10T08:38:10.021848+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:38:10.021848+0000","last_undegraded":"2024-02-10T08:38:10.021848+0000","last_fullsized":"2024-02-10T08:38:10.021848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1a","version":"103'5","reported_seq":89,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018279+0000","last_change":"2024-02-10T08:37:10.316958+0000","last_active":"2024-02-10T08:38:10.018279+0000","last_peered":"2024-02-10T08:38:10.018279+0000","last_clean":"2024-02-10T08:38:10.018279+0000","last_became_active":"2024-02-10T08:37:10.208613+0000","last_became_peered":"2024-02-10T08:37:10.208613+0000","last_unstale":"2024-02-10T08:38:10.018279+0000","last_undegraded":"2024-02-10T08:38:10.018279+0000","last_fullsized":"2024-02-10T08:38:10.018279+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015008+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:38:10.015008+0000","last_peered":"2024-02-10T08:38:10.015008+0000","last_clean":"2024-02-10T08:38:10.015008+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:38:10.015008+0000","last_undegraded":"2024-02-10T08:38:10.015008+0000","last_fullsized":"2024-02-10T08:38:10.015008+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":211,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020325+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:38:10.020325+0000","last_peered":"2024-02-10T08:38:10.020325+0000","last_clean":"2024-02-10T08:38:10.020325+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:38:10.020325+0000","last_undegraded":"2024-02-10T08:38:10.020325+0000","last_fullsized":"2024-02-10T08:38:10.020325+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019292+0000","last_change":"2024-02-10T08:37:10.234765+0000","last_active":"2024-02-10T08:38:10.019292+0000","last_peered":"2024-02-10T08:38:10.019292+0000","last_clean":"2024-02-10T08:38:10.019292+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:38:10.019292+0000","last_undegraded":"2024-02-10T08:38:10.019292+0000","last_fullsized":"2024-02-10T08:38:10.019292+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:28.635610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.635610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":450,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.620565+0000","last_change":"2024-02-10T08:36:33.452480+0000","last_active":"2024-02-10T08:38:10.620565+0000","last_peered":"2024-02-10T08:38:10.620565+0000","last_clean":"2024-02-10T08:38:10.620565+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:38:10.620565+0000","last_undegraded":"2024-02-10T08:38:10.620565+0000","last_fullsized":"2024-02-10T08:38:10.620565+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.452367+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":286,"num_read_kb":281,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031904+0000","last_change":"2024-02-10T08:36:32.132836+0000","last_active":"2024-02-10T08:38:10.031904+0000","last_peered":"2024-02-10T08:38:10.031904+0000","last_clean":"2024-02-10T08:38:10.031904+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:38:10.031904+0000","last_undegraded":"2024-02-10T08:38:10.031904+0000","last_fullsized":"2024-02-10T08:38:10.031904+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:32.132760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.132760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018382+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:38:10.018382+0000","last_peered":"2024-02-10T08:38:10.018382+0000","last_clean":"2024-02-10T08:38:10.018382+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:38:10.018382+0000","last_undegraded":"2024-02-10T08:38:10.018382+0000","last_fullsized":"2024-02-10T08:38:10.018382+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016270+0000","last_change":"2024-02-10T08:36:37.077971+0000","last_active":"2024-02-10T08:38:10.016270+0000","last_peered":"2024-02-10T08:38:10.016270+0000","last_clean":"2024-02-10T08:38:10.016270+0000","last_became_active":"2024-02-10T08:36:37.074340+0000","last_became_peered":"2024-02-10T08:36:37.074340+0000","last_unstale":"2024-02-10T08:38:10.016270+0000","last_undegraded":"2024-02-10T08:38:10.016270+0000","last_fullsized":"2024-02-10T08:38:10.016270+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019056+0000","last_change":"2024-02-10T08:37:10.160466+0000","last_active":"2024-02-10T08:38:10.019056+0000","last_peered":"2024-02-10T08:38:10.019056+0000","last_clean":"2024-02-10T08:38:10.019056+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:38:10.019056+0000","last_undegraded":"2024-02-10T08:38:10.019056+0000","last_fullsized":"2024-02-10T08:38:10.019056+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.962334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.962334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":202,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.037157+0000","last_change":"2024-02-10T08:36:17.572570+0000","last_active":"2024-02-10T08:38:10.037157+0000","last_peered":"2024-02-10T08:38:10.037157+0000","last_clean":"2024-02-10T08:38:10.037157+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:38:10.037157+0000","last_undegraded":"2024-02-10T08:38:10.037157+0000","last_fullsized":"2024-02-10T08:38:10.037157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:36:17.572475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:17.572475+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026652+0000","last_change":"2024-02-10T08:37:41.598616+0000","last_active":"2024-02-10T08:38:10.026652+0000","last_peered":"2024-02-10T08:38:10.026652+0000","last_clean":"2024-02-10T08:38:10.026652+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:38:10.026652+0000","last_undegraded":"2024-02-10T08:38:10.026652+0000","last_fullsized":"2024-02-10T08:38:10.026652+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.598541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.598541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022487+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:38:10.022487+0000","last_peered":"2024-02-10T08:38:10.022487+0000","last_clean":"2024-02-10T08:38:10.022487+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:38:10.022487+0000","last_undegraded":"2024-02-10T08:38:10.022487+0000","last_fullsized":"2024-02-10T08:38:10.022487+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019177+0000","last_change":"2024-02-10T08:37:10.175199+0000","last_active":"2024-02-10T08:38:10.019177+0000","last_peered":"2024-02-10T08:38:10.019177+0000","last_clean":"2024-02-10T08:38:10.019177+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:38:10.019177+0000","last_undegraded":"2024-02-10T08:38:10.019177+0000","last_fullsized":"2024-02-10T08:38:10.019177+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"103'5","reported_seq":89,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018639+0000","last_change":"2024-02-10T08:37:10.332044+0000","last_active":"2024-02-10T08:38:10.018639+0000","last_peered":"2024-02-10T08:38:10.018639+0000","last_clean":"2024-02-10T08:38:10.018639+0000","last_became_active":"2024-02-10T08:37:10.204992+0000","last_became_peered":"2024-02-10T08:37:10.204992+0000","last_unstale":"2024-02-10T08:38:10.018639+0000","last_undegraded":"2024-02-10T08:38:10.018639+0000","last_fullsized":"2024-02-10T08:38:10.018639+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018156+0000","last_change":"2024-02-10T08:37:41.604353+0000","last_active":"2024-02-10T08:38:10.018156+0000","last_peered":"2024-02-10T08:38:10.018156+0000","last_clean":"2024-02-10T08:38:10.018156+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:38:10.018156+0000","last_undegraded":"2024-02-10T08:38:10.018156+0000","last_fullsized":"2024-02-10T08:38:10.018156+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:37:41.604248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.604248+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019993+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:38:10.019993+0000","last_peered":"2024-02-10T08:38:10.019993+0000","last_clean":"2024-02-10T08:38:10.019993+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:38:10.019993+0000","last_undegraded":"2024-02-10T08:38:10.019993+0000","last_fullsized":"2024-02-10T08:38:10.019993+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023831+0000","last_change":"2024-02-10T08:37:10.183206+0000","last_active":"2024-02-10T08:38:10.023831+0000","last_peered":"2024-02-10T08:38:10.023831+0000","last_clean":"2024-02-10T08:38:10.023831+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:38:10.023831+0000","last_undegraded":"2024-02-10T08:38:10.023831+0000","last_fullsized":"2024-02-10T08:38:10.023831+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.456780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020219+0000","last_change":"2024-02-10T08:37:10.263452+0000","last_active":"2024-02-10T08:38:10.020219+0000","last_peered":"2024-02-10T08:38:10.020219+0000","last_clean":"2024-02-10T08:38:10.020219+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:38:10.020219+0000","last_undegraded":"2024-02-10T08:38:10.020219+0000","last_fullsized":"2024-02-10T08:38:10.020219+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:13.802264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:13.802264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022338+0000","last_change":"2024-02-10T08:36:37.155744+0000","last_active":"2024-02-10T08:38:10.022338+0000","last_peered":"2024-02-10T08:38:10.022338+0000","last_clean":"2024-02-10T08:38:10.022338+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:38:10.022338+0000","last_undegraded":"2024-02-10T08:38:10.022338+0000","last_fullsized":"2024-02-10T08:38:10.022338+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.14","version":"103'2","reported_seq":105,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016983+0000","last_change":"2024-02-10T08:36:37.065315+0000","last_active":"2024-02-10T08:38:10.016983+0000","last_peered":"2024-02-10T08:38:10.016983+0000","last_clean":"2024-02-10T08:38:10.016983+0000","last_became_active":"2024-02-10T08:36:37.063912+0000","last_became_peered":"2024-02-10T08:36:37.063912+0000","last_unstale":"2024-02-10T08:38:10.016983+0000","last_undegraded":"2024-02-10T08:38:10.016983+0000","last_fullsized":"2024-02-10T08:38:10.016983+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":188,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019510+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:38:10.019510+0000","last_peered":"2024-02-10T08:38:10.019510+0000","last_clean":"2024-02-10T08:38:10.019510+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:38:10.019510+0000","last_undegraded":"2024-02-10T08:38:10.019510+0000","last_fullsized":"2024-02-10T08:38:10.019510+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":186,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016009+0000","last_change":"2024-02-10T08:36:28.744540+0000","last_active":"2024-02-10T08:38:10.016009+0000","last_peered":"2024-02-10T08:38:10.016009+0000","last_clean":"2024-02-10T08:38:10.016009+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:38:10.016009+0000","last_undegraded":"2024-02-10T08:38:10.016009+0000","last_fullsized":"2024-02-10T08:38:10.016009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:36:28.744465+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:36:28.744465+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":188,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022471+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:38:10.022471+0000","last_peered":"2024-02-10T08:38:10.022471+0000","last_clean":"2024-02-10T08:38:10.022471+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:38:10.022471+0000","last_undegraded":"2024-02-10T08:38:10.022471+0000","last_fullsized":"2024-02-10T08:38:10.022471+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022146+0000","last_change":"2024-02-10T08:36:23.084433+0000","last_active":"2024-02-10T08:38:10.022146+0000","last_peered":"2024-02-10T08:38:10.022146+0000","last_clean":"2024-02-10T08:38:10.022146+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:38:10.022146+0000","last_undegraded":"2024-02-10T08:38:10.022146+0000","last_fullsized":"2024-02-10T08:38:10.022146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:23.084331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:23.084331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":108,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022751+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:38:10.022751+0000","last_peered":"2024-02-10T08:38:10.022751+0000","last_clean":"2024-02-10T08:38:10.022751+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:38:10.022751+0000","last_undegraded":"2024-02-10T08:38:10.022751+0000","last_fullsized":"2024-02-10T08:38:10.022751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.b","version":"103'2","reported_seq":118,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018151+0000","last_change":"2024-02-10T08:37:10.677108+0000","last_active":"2024-02-10T08:38:10.018151+0000","last_peered":"2024-02-10T08:38:10.018151+0000","last_clean":"2024-02-10T08:38:10.018151+0000","last_became_active":"2024-02-10T08:37:10.216339+0000","last_became_peered":"2024-02-10T08:37:10.216339+0000","last_unstale":"2024-02-10T08:38:10.018151+0000","last_undegraded":"2024-02-10T08:38:10.018151+0000","last_fullsized":"2024-02-10T08:38:10.018151+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":144,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022539+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:38:10.022539+0000","last_peered":"2024-02-10T08:38:10.022539+0000","last_clean":"2024-02-10T08:38:10.022539+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:38:10.022539+0000","last_undegraded":"2024-02-10T08:38:10.022539+0000","last_fullsized":"2024-02-10T08:38:10.022539+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019430+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:38:10.019430+0000","last_peered":"2024-02-10T08:38:10.019430+0000","last_clean":"2024-02-10T08:38:10.019430+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:38:10.019430+0000","last_undegraded":"2024-02-10T08:38:10.019430+0000","last_fullsized":"2024-02-10T08:38:10.019430+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015704+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:38:10.015704+0000","last_peered":"2024-02-10T08:38:10.015704+0000","last_clean":"2024-02-10T08:38:10.015704+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:38:10.015704+0000","last_undegraded":"2024-02-10T08:38:10.015704+0000","last_fullsized":"2024-02-10T08:38:10.015704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015982+0000","last_change":"2024-02-10T08:36:37.041210+0000","last_active":"2024-02-10T08:38:10.015982+0000","last_peered":"2024-02-10T08:38:10.015982+0000","last_clean":"2024-02-10T08:38:10.015982+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:38:10.015982+0000","last_undegraded":"2024-02-10T08:38:10.015982+0000","last_fullsized":"2024-02-10T08:38:10.015982+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"103'10","reported_seq":96,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024208+0000","last_change":"2024-02-10T08:37:10.582273+0000","last_active":"2024-02-10T08:38:10.024208+0000","last_peered":"2024-02-10T08:38:10.024208+0000","last_clean":"2024-02-10T08:38:10.024208+0000","last_became_active":"2024-02-10T08:37:10.186893+0000","last_became_peered":"2024-02-10T08:37:10.186893+0000","last_unstale":"2024-02-10T08:38:10.024208+0000","last_undegraded":"2024-02-10T08:38:10.024208+0000","last_fullsized":"2024-02-10T08:38:10.024208+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":166,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028833+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:38:10.028833+0000","last_peered":"2024-02-10T08:38:10.028833+0000","last_clean":"2024-02-10T08:38:10.028833+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:38:10.028833+0000","last_undegraded":"2024-02-10T08:38:10.028833+0000","last_fullsized":"2024-02-10T08:38:10.028833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018394+0000","last_change":"2024-02-10T08:36:19.570275+0000","last_active":"2024-02-10T08:38:10.018394+0000","last_peered":"2024-02-10T08:38:10.018394+0000","last_clean":"2024-02-10T08:38:10.018394+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:38:10.018394+0000","last_undegraded":"2024-02-10T08:38:10.018394+0000","last_fullsized":"2024-02-10T08:38:10.018394+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:19.570181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:19.570181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031665+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:38:10.031665+0000","last_peered":"2024-02-10T08:38:10.031665+0000","last_clean":"2024-02-10T08:38:10.031665+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:38:10.031665+0000","last_undegraded":"2024-02-10T08:38:10.031665+0000","last_fullsized":"2024-02-10T08:38:10.031665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":102,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032298+0000","last_change":"2024-02-10T08:36:37.051573+0000","last_active":"2024-02-10T08:38:10.032298+0000","last_peered":"2024-02-10T08:38:10.032298+0000","last_clean":"2024-02-10T08:38:10.032298+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:38:10.032298+0000","last_undegraded":"2024-02-10T08:38:10.032298+0000","last_fullsized":"2024-02-10T08:38:10.032298+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019004+0000","last_change":"2024-02-10T08:37:10.215944+0000","last_active":"2024-02-10T08:38:10.019004+0000","last_peered":"2024-02-10T08:38:10.019004+0000","last_clean":"2024-02-10T08:38:10.019004+0000","last_became_active":"2024-02-10T08:37:10.205941+0000","last_became_peered":"2024-02-10T08:37:10.205941+0000","last_unstale":"2024-02-10T08:38:10.019004+0000","last_undegraded":"2024-02-10T08:38:10.019004+0000","last_fullsized":"2024-02-10T08:38:10.019004+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028229+0000","last_change":"2024-02-10T08:36:14.035460+0000","last_active":"2024-02-10T08:38:10.028229+0000","last_peered":"2024-02-10T08:38:10.028229+0000","last_clean":"2024-02-10T08:38:10.028229+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:38:10.028229+0000","last_undegraded":"2024-02-10T08:38:10.028229+0000","last_fullsized":"2024-02-10T08:38:10.028229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:14.035356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.035356+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026488+0000","last_change":"2024-02-10T08:37:46.576724+0000","last_active":"2024-02-10T08:38:10.026488+0000","last_peered":"2024-02-10T08:38:10.026488+0000","last_clean":"2024-02-10T08:38:10.026488+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:38:10.026488+0000","last_undegraded":"2024-02-10T08:38:10.026488+0000","last_fullsized":"2024-02-10T08:38:10.026488+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.576661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.576661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":234,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031421+0000","last_change":"2024-02-10T08:37:10.531826+0000","last_active":"2024-02-10T08:38:10.031421+0000","last_peered":"2024-02-10T08:38:10.031421+0000","last_clean":"2024-02-10T08:38:10.031421+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:38:10.031421+0000","last_undegraded":"2024-02-10T08:38:10.031421+0000","last_fullsized":"2024-02-10T08:38:10.031421+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:37:05.796026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:05.796026+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032709+0000","last_change":"2024-02-10T08:37:10.242150+0000","last_active":"2024-02-10T08:38:10.032709+0000","last_peered":"2024-02-10T08:38:10.032709+0000","last_clean":"2024-02-10T08:38:10.032709+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:38:10.032709+0000","last_undegraded":"2024-02-10T08:38:10.032709+0000","last_fullsized":"2024-02-10T08:38:10.032709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.928693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":72,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024278+0000","last_change":"2024-02-10T08:37:10.232375+0000","last_active":"2024-02-10T08:38:10.024278+0000","last_peered":"2024-02-10T08:38:10.024278+0000","last_clean":"2024-02-10T08:38:10.024278+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:38:10.024278+0000","last_undegraded":"2024-02-10T08:38:10.024278+0000","last_fullsized":"2024-02-10T08:38:10.024278+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.8","version":"103'2","reported_seq":104,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026426+0000","last_change":"2024-02-10T08:36:37.035179+0000","last_active":"2024-02-10T08:38:10.026426+0000","last_peered":"2024-02-10T08:38:10.026426+0000","last_clean":"2024-02-10T08:38:10.026426+0000","last_became_active":"2024-02-10T08:36:37.023527+0000","last_became_peered":"2024-02-10T08:36:37.023527+0000","last_unstale":"2024-02-10T08:38:10.026426+0000","last_undegraded":"2024-02-10T08:38:10.026426+0000","last_fullsized":"2024-02-10T08:38:10.026426+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031233+0000","last_change":"2024-02-10T08:37:18.722889+0000","last_active":"2024-02-10T08:38:10.031233+0000","last_peered":"2024-02-10T08:38:10.031233+0000","last_clean":"2024-02-10T08:38:10.031233+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:38:10.031233+0000","last_undegraded":"2024-02-10T08:38:10.031233+0000","last_fullsized":"2024-02-10T08:38:10.031233+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:18.722818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.722818+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018808+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:38:10.018808+0000","last_peered":"2024-02-10T08:38:10.018808+0000","last_clean":"2024-02-10T08:38:10.018808+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:38:10.018808+0000","last_undegraded":"2024-02-10T08:38:10.018808+0000","last_fullsized":"2024-02-10T08:38:10.018808+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":221,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028446+0000","last_change":"2024-02-10T08:36:29.049034+0000","last_active":"2024-02-10T08:38:10.028446+0000","last_peered":"2024-02-10T08:38:10.028446+0000","last_clean":"2024-02-10T08:38:10.028446+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:38:10.028446+0000","last_undegraded":"2024-02-10T08:38:10.028446+0000","last_fullsized":"2024-02-10T08:38:10.028446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:36:29.048965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:29.048965+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022722+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:38:10.022722+0000","last_peered":"2024-02-10T08:38:10.022722+0000","last_clean":"2024-02-10T08:38:10.022722+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:38:10.022722+0000","last_undegraded":"2024-02-10T08:38:10.022722+0000","last_fullsized":"2024-02-10T08:38:10.022722+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":116,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019090+0000","last_change":"2024-02-10T08:36:37.041395+0000","last_active":"2024-02-10T08:38:10.019090+0000","last_peered":"2024-02-10T08:38:10.019090+0000","last_clean":"2024-02-10T08:38:10.019090+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:38:10.019090+0000","last_undegraded":"2024-02-10T08:38:10.019090+0000","last_fullsized":"2024-02-10T08:38:10.019090+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.044547+0000","last_change":"2024-02-10T08:37:10.129525+0000","last_active":"2024-02-10T08:38:10.044547+0000","last_peered":"2024-02-10T08:38:10.044547+0000","last_clean":"2024-02-10T08:38:10.044547+0000","last_became_active":"2024-02-10T08:37:10.129023+0000","last_became_peered":"2024-02-10T08:37:10.129023+0000","last_unstale":"2024-02-10T08:38:10.044547+0000","last_undegraded":"2024-02-10T08:38:10.044547+0000","last_fullsized":"2024-02-10T08:38:10.044547+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":178,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019906+0000","last_change":"2024-02-10T08:36:31.434160+0000","last_active":"2024-02-10T08:38:10.019906+0000","last_peered":"2024-02-10T08:38:10.019906+0000","last_clean":"2024-02-10T08:38:10.019906+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:38:10.019906+0000","last_undegraded":"2024-02-10T08:38:10.019906+0000","last_fullsized":"2024-02-10T08:38:10.019906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.434047+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020155+0000","last_change":"2024-02-10T08:37:18.260244+0000","last_active":"2024-02-10T08:38:10.020155+0000","last_peered":"2024-02-10T08:38:10.020155+0000","last_clean":"2024-02-10T08:38:10.020155+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:38:10.020155+0000","last_undegraded":"2024-02-10T08:38:10.020155+0000","last_fullsized":"2024-02-10T08:38:10.020155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.260171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.260171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022072+0000","last_change":"2024-02-10T08:37:10.315695+0000","last_active":"2024-02-10T08:38:10.022072+0000","last_peered":"2024-02-10T08:38:10.022072+0000","last_clean":"2024-02-10T08:38:10.022072+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:38:10.022072+0000","last_undegraded":"2024-02-10T08:38:10.022072+0000","last_fullsized":"2024-02-10T08:38:10.022072+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:35:43.333386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:35:43.333386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":72,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028027+0000","last_change":"2024-02-10T08:37:10.150087+0000","last_active":"2024-02-10T08:38:10.028027+0000","last_peered":"2024-02-10T08:38:10.028027+0000","last_clean":"2024-02-10T08:38:10.028027+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:38:10.028027+0000","last_undegraded":"2024-02-10T08:38:10.028027+0000","last_fullsized":"2024-02-10T08:38:10.028027+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.978888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022495+0000","last_change":"2024-02-10T08:37:10.202917+0000","last_active":"2024-02-10T08:38:10.022495+0000","last_peered":"2024-02-10T08:38:10.022495+0000","last_clean":"2024-02-10T08:38:10.022495+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:38:10.022495+0000","last_undegraded":"2024-02-10T08:38:10.022495+0000","last_fullsized":"2024-02-10T08:38:10.022495+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019284+0000","last_change":"2024-02-10T08:37:10.242206+0000","last_active":"2024-02-10T08:38:10.019284+0000","last_peered":"2024-02-10T08:38:10.019284+0000","last_clean":"2024-02-10T08:38:10.019284+0000","last_became_active":"2024-02-10T08:37:10.242025+0000","last_became_peered":"2024-02-10T08:37:10.242025+0000","last_unstale":"2024-02-10T08:38:10.019284+0000","last_undegraded":"2024-02-10T08:38:10.019284+0000","last_fullsized":"2024-02-10T08:38:10.019284+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018710+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:38:10.018710+0000","last_peered":"2024-02-10T08:38:10.018710+0000","last_clean":"2024-02-10T08:38:10.018710+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:38:10.018710+0000","last_undegraded":"2024-02-10T08:38:10.018710+0000","last_fullsized":"2024-02-10T08:38:10.018710+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019190+0000","last_change":"2024-02-10T08:37:10.151238+0000","last_active":"2024-02-10T08:38:10.019190+0000","last_peered":"2024-02-10T08:38:10.019190+0000","last_clean":"2024-02-10T08:38:10.019190+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:38:10.019190+0000","last_undegraded":"2024-02-10T08:38:10.019190+0000","last_fullsized":"2024-02-10T08:38:10.019190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.453161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.453161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":156,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019959+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:38:10.019959+0000","last_peered":"2024-02-10T08:38:10.019959+0000","last_clean":"2024-02-10T08:38:10.019959+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:38:10.019959+0000","last_undegraded":"2024-02-10T08:38:10.019959+0000","last_fullsized":"2024-02-10T08:38:10.019959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018840+0000","last_change":"2024-02-10T08:37:10.284262+0000","last_active":"2024-02-10T08:38:10.018840+0000","last_peered":"2024-02-10T08:38:10.018840+0000","last_clean":"2024-02-10T08:38:10.018840+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:38:10.018840+0000","last_undegraded":"2024-02-10T08:38:10.018840+0000","last_fullsized":"2024-02-10T08:38:10.018840+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:24.066605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.066605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.025626+0000","last_change":"2024-02-10T08:37:10.260574+0000","last_active":"2024-02-10T08:38:10.025626+0000","last_peered":"2024-02-10T08:38:10.025626+0000","last_clean":"2024-02-10T08:38:10.025626+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:38:10.025626+0000","last_undegraded":"2024-02-10T08:38:10.025626+0000","last_fullsized":"2024-02-10T08:38:10.025626+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"103'5","reported_seq":125,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022715+0000","last_change":"2024-02-10T08:37:45.548365+0000","last_active":"2024-02-10T08:38:10.022715+0000","last_peered":"2024-02-10T08:38:10.022715+0000","last_clean":"2024-02-10T08:38:10.022715+0000","last_became_active":"2024-02-10T08:37:10.149021+0000","last_became_peered":"2024-02-10T08:37:10.149021+0000","last_unstale":"2024-02-10T08:38:10.022715+0000","last_undegraded":"2024-02-10T08:38:10.022715+0000","last_fullsized":"2024-02-10T08:38:10.022715+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'5","last_scrub_stamp":"2024-02-10T08:37:45.548297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.548297+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":163,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.020173+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:38:10.020173+0000","last_peered":"2024-02-10T08:38:10.020173+0000","last_clean":"2024-02-10T08:38:10.020173+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:38:10.020173+0000","last_undegraded":"2024-02-10T08:38:10.020173+0000","last_fullsized":"2024-02-10T08:38:10.020173+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.036551+0000","last_change":"2024-02-10T08:36:15.645089+0000","last_active":"2024-02-10T08:38:10.036551+0000","last_peered":"2024-02-10T08:38:10.036551+0000","last_clean":"2024-02-10T08:38:10.036551+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:38:10.036551+0000","last_undegraded":"2024-02-10T08:38:10.036551+0000","last_fullsized":"2024-02-10T08:38:10.036551+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.644998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.644998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018281+0000","last_change":"2024-02-10T08:36:33.824585+0000","last_active":"2024-02-10T08:38:10.018281+0000","last_peered":"2024-02-10T08:38:10.018281+0000","last_clean":"2024-02-10T08:38:10.018281+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:38:10.018281+0000","last_undegraded":"2024-02-10T08:38:10.018281+0000","last_fullsized":"2024-02-10T08:38:10.018281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.824511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.824511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027386+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:38:10.027386+0000","last_peered":"2024-02-10T08:38:10.027386+0000","last_clean":"2024-02-10T08:38:10.027386+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:38:10.027386+0000","last_undegraded":"2024-02-10T08:38:10.027386+0000","last_fullsized":"2024-02-10T08:38:10.027386+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028012+0000","last_change":"2024-02-10T08:36:37.165190+0000","last_active":"2024-02-10T08:38:10.028012+0000","last_peered":"2024-02-10T08:38:10.028012+0000","last_clean":"2024-02-10T08:38:10.028012+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:38:10.028012+0000","last_undegraded":"2024-02-10T08:38:10.028012+0000","last_fullsized":"2024-02-10T08:38:10.028012+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":102,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.041956+0000","last_change":"2024-02-10T08:36:37.127556+0000","last_active":"2024-02-10T08:38:10.041956+0000","last_peered":"2024-02-10T08:38:10.041956+0000","last_clean":"2024-02-10T08:38:10.041956+0000","last_became_active":"2024-02-10T08:36:37.127287+0000","last_became_peered":"2024-02-10T08:36:37.127287+0000","last_unstale":"2024-02-10T08:38:10.041956+0000","last_undegraded":"2024-02-10T08:38:10.041956+0000","last_fullsized":"2024-02-10T08:38:10.041956+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019250+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:38:10.019250+0000","last_peered":"2024-02-10T08:38:10.019250+0000","last_clean":"2024-02-10T08:38:10.019250+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:38:10.019250+0000","last_undegraded":"2024-02-10T08:38:10.019250+0000","last_fullsized":"2024-02-10T08:38:10.019250+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023269+0000","last_change":"2024-02-10T08:37:38.527763+0000","last_active":"2024-02-10T08:38:10.023269+0000","last_peered":"2024-02-10T08:38:10.023269+0000","last_clean":"2024-02-10T08:38:10.023269+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:38:10.023269+0000","last_undegraded":"2024-02-10T08:38:10.023269+0000","last_fullsized":"2024-02-10T08:38:10.023269+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.527658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.527658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":156,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030572+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:38:10.030572+0000","last_peered":"2024-02-10T08:38:10.030572+0000","last_clean":"2024-02-10T08:38:10.030572+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:38:10.030572+0000","last_undegraded":"2024-02-10T08:38:10.030572+0000","last_fullsized":"2024-02-10T08:38:10.030572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":162,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022226+0000","last_change":"2024-02-10T08:37:10.262623+0000","last_active":"2024-02-10T08:38:10.022226+0000","last_peered":"2024-02-10T08:38:10.022226+0000","last_clean":"2024-02-10T08:38:10.022226+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:38:10.022226+0000","last_undegraded":"2024-02-10T08:38:10.022226+0000","last_fullsized":"2024-02-10T08:38:10.022226+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:33.042582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:33.042582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":72,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022934+0000","last_change":"2024-02-10T08:37:10.148262+0000","last_active":"2024-02-10T08:38:10.022934+0000","last_peered":"2024-02-10T08:38:10.022934+0000","last_clean":"2024-02-10T08:38:10.022934+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:38:10.022934+0000","last_undegraded":"2024-02-10T08:38:10.022934+0000","last_fullsized":"2024-02-10T08:38:10.022934+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"103'5","reported_seq":108,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024859+0000","last_change":"2024-02-10T08:36:37.052377+0000","last_active":"2024-02-10T08:38:10.024859+0000","last_peered":"2024-02-10T08:38:10.024859+0000","last_clean":"2024-02-10T08:38:10.024859+0000","last_became_active":"2024-02-10T08:36:37.050808+0000","last_became_peered":"2024-02-10T08:36:37.050808+0000","last_unstale":"2024-02-10T08:38:10.024859+0000","last_undegraded":"2024-02-10T08:38:10.024859+0000","last_fullsized":"2024-02-10T08:38:10.024859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.032921+0000","last_change":"2024-02-10T08:37:10.466717+0000","last_active":"2024-02-10T08:38:10.032921+0000","last_peered":"2024-02-10T08:38:10.032921+0000","last_clean":"2024-02-10T08:38:10.032921+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:38:10.032921+0000","last_undegraded":"2024-02-10T08:38:10.032921+0000","last_fullsized":"2024-02-10T08:38:10.032921+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:36:31.890386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:31.890386+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.027943+0000","last_change":"2024-02-10T08:37:39.573283+0000","last_active":"2024-02-10T08:38:10.027943+0000","last_peered":"2024-02-10T08:38:10.027943+0000","last_clean":"2024-02-10T08:38:10.027943+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:38:10.027943+0000","last_undegraded":"2024-02-10T08:38:10.027943+0000","last_fullsized":"2024-02-10T08:38:10.027943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:39.573225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.573225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":155,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.029165+0000","last_change":"2024-02-10T08:37:07.241312+0000","last_active":"2024-02-10T08:38:10.029165+0000","last_peered":"2024-02-10T08:38:10.029165+0000","last_clean":"2024-02-10T08:38:10.029165+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:38:10.029165+0000","last_undegraded":"2024-02-10T08:38:10.029165+0000","last_fullsized":"2024-02-10T08:38:10.029165+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:07.241247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.241247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024281+0000","last_change":"2024-02-10T08:36:37.049548+0000","last_active":"2024-02-10T08:38:10.024281+0000","last_peered":"2024-02-10T08:38:10.024281+0000","last_clean":"2024-02-10T08:38:10.024281+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:38:10.024281+0000","last_undegraded":"2024-02-10T08:38:10.024281+0000","last_fullsized":"2024-02-10T08:38:10.024281+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.026141+0000","last_change":"2024-02-10T08:36:37.074211+0000","last_active":"2024-02-10T08:38:10.026141+0000","last_peered":"2024-02-10T08:38:10.026141+0000","last_clean":"2024-02-10T08:38:10.026141+0000","last_became_active":"2024-02-10T08:36:37.073890+0000","last_became_peered":"2024-02-10T08:36:37.073890+0000","last_unstale":"2024-02-10T08:38:10.026141+0000","last_undegraded":"2024-02-10T08:38:10.026141+0000","last_fullsized":"2024-02-10T08:38:10.026141+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":181,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.023519+0000","last_change":"2024-02-10T08:37:20.333410+0000","last_active":"2024-02-10T08:38:10.023519+0000","last_peered":"2024-02-10T08:38:10.023519+0000","last_clean":"2024-02-10T08:38:10.023519+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:38:10.023519+0000","last_undegraded":"2024-02-10T08:38:10.023519+0000","last_fullsized":"2024-02-10T08:38:10.023519+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:20.333345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:20.333345+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":168,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019024+0000","last_change":"2024-02-10T08:37:10.284428+0000","last_active":"2024-02-10T08:38:10.019024+0000","last_peered":"2024-02-10T08:38:10.019024+0000","last_clean":"2024-02-10T08:38:10.019024+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:38:10.019024+0000","last_undegraded":"2024-02-10T08:38:10.019024+0000","last_fullsized":"2024-02-10T08:38:10.019024+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:36:22.072982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018975+0000","last_change":"2024-02-10T08:36:25.503414+0000","last_active":"2024-02-10T08:38:10.018975+0000","last_peered":"2024-02-10T08:38:10.018975+0000","last_clean":"2024-02-10T08:38:10.018975+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:38:10.018975+0000","last_undegraded":"2024-02-10T08:38:10.018975+0000","last_fullsized":"2024-02-10T08:38:10.018975+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:25.503351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:25.503351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022656+0000","last_change":"2024-02-10T08:36:37.124706+0000","last_active":"2024-02-10T08:38:10.022656+0000","last_peered":"2024-02-10T08:38:10.022656+0000","last_clean":"2024-02-10T08:38:10.022656+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:38:10.022656+0000","last_undegraded":"2024-02-10T08:38:10.022656+0000","last_fullsized":"2024-02-10T08:38:10.022656+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"103'5","reported_seq":113,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019374+0000","last_change":"2024-02-10T08:37:45.871952+0000","last_active":"2024-02-10T08:38:10.019374+0000","last_peered":"2024-02-10T08:38:10.019374+0000","last_clean":"2024-02-10T08:38:10.019374+0000","last_became_active":"2024-02-10T08:36:37.063485+0000","last_became_peered":"2024-02-10T08:36:37.063485+0000","last_unstale":"2024-02-10T08:38:10.019374+0000","last_undegraded":"2024-02-10T08:38:10.019374+0000","last_fullsized":"2024-02-10T08:38:10.019374+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'5","last_scrub_stamp":"2024-02-10T08:37:45.871881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.871881+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":195,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022298+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:38:10.022298+0000","last_peered":"2024-02-10T08:38:10.022298+0000","last_clean":"2024-02-10T08:38:10.022298+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:38:10.022298+0000","last_undegraded":"2024-02-10T08:38:10.022298+0000","last_fullsized":"2024-02-10T08:38:10.022298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019565+0000","last_change":"2024-02-10T08:36:34.837568+0000","last_active":"2024-02-10T08:38:10.019565+0000","last_peered":"2024-02-10T08:38:10.019565+0000","last_clean":"2024-02-10T08:38:10.019565+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:38:10.019565+0000","last_undegraded":"2024-02-10T08:38:10.019565+0000","last_fullsized":"2024-02-10T08:38:10.019565+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:34.837483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:34.837483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022823+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:38:10.022823+0000","last_peered":"2024-02-10T08:38:10.022823+0000","last_clean":"2024-02-10T08:38:10.022823+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:38:10.022823+0000","last_undegraded":"2024-02-10T08:38:10.022823+0000","last_fullsized":"2024-02-10T08:38:10.022823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":72,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024194+0000","last_change":"2024-02-10T08:37:10.232241+0000","last_active":"2024-02-10T08:38:10.024194+0000","last_peered":"2024-02-10T08:38:10.024194+0000","last_clean":"2024-02-10T08:38:10.024194+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:38:10.024194+0000","last_undegraded":"2024-02-10T08:38:10.024194+0000","last_fullsized":"2024-02-10T08:38:10.024194+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022963+0000","last_change":"2024-02-10T08:36:20.047350+0000","last_active":"2024-02-10T08:38:10.022963+0000","last_peered":"2024-02-10T08:38:10.022963+0000","last_clean":"2024-02-10T08:38:10.022963+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:38:10.022963+0000","last_undegraded":"2024-02-10T08:38:10.022963+0000","last_fullsized":"2024-02-10T08:38:10.022963+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:36:20.047272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.047272+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":159,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021998+0000","last_change":"2024-02-10T08:36:15.045147+0000","last_active":"2024-02-10T08:38:10.021998+0000","last_peered":"2024-02-10T08:38:10.021998+0000","last_clean":"2024-02-10T08:38:10.021998+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:38:10.021998+0000","last_undegraded":"2024-02-10T08:38:10.021998+0000","last_fullsized":"2024-02-10T08:38:10.021998+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:15.045044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:15.045044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028978+0000","last_change":"2024-02-10T08:36:27.054202+0000","last_active":"2024-02-10T08:38:10.028978+0000","last_peered":"2024-02-10T08:38:10.028978+0000","last_clean":"2024-02-10T08:38:10.028978+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:38:10.028978+0000","last_undegraded":"2024-02-10T08:38:10.028978+0000","last_fullsized":"2024-02-10T08:38:10.028978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:27.054145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:27.054145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021238+0000","last_change":"2024-02-10T08:36:37.038364+0000","last_active":"2024-02-10T08:38:10.021238+0000","last_peered":"2024-02-10T08:38:10.021238+0000","last_clean":"2024-02-10T08:38:10.021238+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:38:10.021238+0000","last_undegraded":"2024-02-10T08:38:10.021238+0000","last_fullsized":"2024-02-10T08:38:10.021238+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"103'5","reported_seq":107,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.024364+0000","last_change":"2024-02-10T08:36:37.017992+0000","last_active":"2024-02-10T08:38:10.024364+0000","last_peered":"2024-02-10T08:38:10.024364+0000","last_clean":"2024-02-10T08:38:10.024364+0000","last_became_active":"2024-02-10T08:36:37.016870+0000","last_became_peered":"2024-02-10T08:36:37.016870+0000","last_unstale":"2024-02-10T08:38:10.024364+0000","last_undegraded":"2024-02-10T08:38:10.024364+0000","last_fullsized":"2024-02-10T08:38:10.024364+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.030804+0000","last_change":"2024-02-10T08:37:10.112153+0000","last_active":"2024-02-10T08:38:10.030804+0000","last_peered":"2024-02-10T08:38:10.030804+0000","last_clean":"2024-02-10T08:38:10.030804+0000","last_became_active":"2024-02-10T08:37:10.111814+0000","last_became_peered":"2024-02-10T08:37:10.111814+0000","last_unstale":"2024-02-10T08:38:10.030804+0000","last_undegraded":"2024-02-10T08:38:10.030804+0000","last_fullsized":"2024-02-10T08:38:10.030804+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.017957+0000","last_change":"2024-02-10T08:37:10.483725+0000","last_active":"2024-02-10T08:38:10.017957+0000","last_peered":"2024-02-10T08:38:10.017957+0000","last_clean":"2024-02-10T08:38:10.017957+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:38:10.017957+0000","last_undegraded":"2024-02-10T08:38:10.017957+0000","last_fullsized":"2024-02-10T08:38:10.017957+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:36:24.764020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:24.764020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":160,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028798+0000","last_change":"2024-02-10T08:36:30.157585+0000","last_active":"2024-02-10T08:38:10.028798+0000","last_peered":"2024-02-10T08:38:10.028798+0000","last_clean":"2024-02-10T08:38:10.028798+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:38:10.028798+0000","last_undegraded":"2024-02-10T08:38:10.028798+0000","last_fullsized":"2024-02-10T08:38:10.028798+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:30.157473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:30.157473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":73,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.014246+0000","last_change":"2024-02-10T08:37:10.187798+0000","last_active":"2024-02-10T08:38:10.014246+0000","last_peered":"2024-02-10T08:38:10.014246+0000","last_clean":"2024-02-10T08:38:10.014246+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:38:10.014246+0000","last_undegraded":"2024-02-10T08:38:10.014246+0000","last_fullsized":"2024-02-10T08:38:10.014246+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:20.877268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:20.877268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.014748+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:38:10.014748+0000","last_peered":"2024-02-10T08:38:10.014748+0000","last_clean":"2024-02-10T08:38:10.014748+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:38:10.014748+0000","last_undegraded":"2024-02-10T08:38:10.014748+0000","last_fullsized":"2024-02-10T08:38:10.014748+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.6","version":"103'9","reported_seq":112,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.014656+0000","last_change":"2024-02-10T08:36:37.066456+0000","last_active":"2024-02-10T08:38:10.014656+0000","last_peered":"2024-02-10T08:38:10.014656+0000","last_clean":"2024-02-10T08:38:10.014656+0000","last_became_active":"2024-02-10T08:36:37.065893+0000","last_became_peered":"2024-02-10T08:36:37.065893+0000","last_unstale":"2024-02-10T08:38:10.014656+0000","last_undegraded":"2024-02-10T08:38:10.014656+0000","last_fullsized":"2024-02-10T08:38:10.014656+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"63'5","reported_seq":168,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.018907+0000","last_change":"2024-02-10T08:36:32.709476+0000","last_active":"2024-02-10T08:38:10.018907+0000","last_peered":"2024-02-10T08:38:10.018907+0000","last_clean":"2024-02-10T08:38:10.018907+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:38:10.018907+0000","last_undegraded":"2024-02-10T08:38:10.018907+0000","last_fullsized":"2024-02-10T08:38:10.018907+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'5","last_scrub_stamp":"2024-02-10T08:36:32.709392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:36:32.709392+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":169,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031753+0000","last_change":"2024-02-10T08:37:10.243920+0000","last_active":"2024-02-10T08:38:10.031753+0000","last_peered":"2024-02-10T08:38:10.031753+0000","last_clean":"2024-02-10T08:38:10.031753+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:38:10.031753+0000","last_undegraded":"2024-02-10T08:38:10.031753+0000","last_fullsized":"2024-02-10T08:38:10.031753+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:21.107579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:21.107579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022015+0000","last_change":"2024-02-10T08:37:22.720373+0000","last_active":"2024-02-10T08:38:10.022015+0000","last_peered":"2024-02-10T08:38:10.022015+0000","last_clean":"2024-02-10T08:38:10.022015+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:38:10.022015+0000","last_undegraded":"2024-02-10T08:38:10.022015+0000","last_fullsized":"2024-02-10T08:38:10.022015+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.720295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.720295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":72,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.043897+0000","last_change":"2024-02-10T08:37:10.149854+0000","last_active":"2024-02-10T08:38:10.043897+0000","last_peered":"2024-02-10T08:38:10.043897+0000","last_clean":"2024-02-10T08:38:10.043897+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:38:10.043897+0000","last_undegraded":"2024-02-10T08:38:10.043897+0000","last_fullsized":"2024-02-10T08:38:10.043897+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022839+0000","last_change":"2024-02-10T08:36:37.078717+0000","last_active":"2024-02-10T08:38:10.022839+0000","last_peered":"2024-02-10T08:38:10.022839+0000","last_clean":"2024-02-10T08:38:10.022839+0000","last_became_active":"2024-02-10T08:36:37.077729+0000","last_became_peered":"2024-02-10T08:36:37.077729+0000","last_unstale":"2024-02-10T08:38:10.022839+0000","last_undegraded":"2024-02-10T08:38:10.022839+0000","last_fullsized":"2024-02-10T08:38:10.022839+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.015678+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:38:10.015678+0000","last_peered":"2024-02-10T08:38:10.015678+0000","last_clean":"2024-02-10T08:38:10.015678+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:38:10.015678+0000","last_undegraded":"2024-02-10T08:38:10.015678+0000","last_fullsized":"2024-02-10T08:38:10.015678+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":168,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022455+0000","last_change":"2024-02-10T08:37:10.306481+0000","last_active":"2024-02-10T08:38:10.022455+0000","last_peered":"2024-02-10T08:38:10.022455+0000","last_clean":"2024-02-10T08:38:10.022455+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:38:10.022455+0000","last_undegraded":"2024-02-10T08:38:10.022455+0000","last_fullsized":"2024-02-10T08:38:10.022455+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:26.014690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:36:26.014690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.016154+0000","last_change":"2024-02-10T08:36:14.747640+0000","last_active":"2024-02-10T08:38:10.016154+0000","last_peered":"2024-02-10T08:38:10.016154+0000","last_clean":"2024-02-10T08:38:10.016154+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:38:10.016154+0000","last_undegraded":"2024-02-10T08:38:10.016154+0000","last_fullsized":"2024-02-10T08:38:10.016154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:14.747521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:36:14.747521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":111,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.037055+0000","last_change":"2024-02-10T08:37:10.145097+0000","last_active":"2024-02-10T08:38:10.037055+0000","last_peered":"2024-02-10T08:38:10.037055+0000","last_clean":"2024-02-10T08:38:10.037055+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:38:10.037055+0000","last_undegraded":"2024-02-10T08:38:10.037055+0000","last_fullsized":"2024-02-10T08:38:10.037055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":103,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.028574+0000","last_change":"2024-02-10T08:36:37.164878+0000","last_active":"2024-02-10T08:38:10.028574+0000","last_peered":"2024-02-10T08:38:10.028574+0000","last_clean":"2024-02-10T08:38:10.028574+0000","last_became_active":"2024-02-10T08:36:37.163315+0000","last_became_peered":"2024-02-10T08:36:37.163315+0000","last_unstale":"2024-02-10T08:38:10.028574+0000","last_undegraded":"2024-02-10T08:38:10.028574+0000","last_fullsized":"2024-02-10T08:38:10.028574+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":207,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.019334+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:38:10.019334+0000","last_peered":"2024-02-10T08:38:10.019334+0000","last_clean":"2024-02-10T08:38:10.019334+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:38:10.019334+0000","last_undegraded":"2024-02-10T08:38:10.019334+0000","last_fullsized":"2024-02-10T08:38:10.019334+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":78,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.031041+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:38:10.031041+0000","last_peered":"2024-02-10T08:38:10.031041+0000","last_clean":"2024-02-10T08:38:10.031041+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:38:10.031041+0000","last_undegraded":"2024-02-10T08:38:10.031041+0000","last_fullsized":"2024-02-10T08:38:10.031041+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":153,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.021853+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:38:10.021853+0000","last_peered":"2024-02-10T08:38:10.021853+0000","last_clean":"2024-02-10T08:38:10.021853+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:38:10.021853+0000","last_undegraded":"2024-02-10T08:38:10.021853+0000","last_fullsized":"2024-02-10T08:38:10.021853+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":102,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-02-10T08:38:10.022470+0000","last_change":"2024-02-10T08:36:37.083207+0000","last_active":"2024-02-10T08:38:10.022470+0000","last_peered":"2024-02-10T08:38:10.022470+0000","last_clean":"2024-02-10T08:38:10.022470+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:38:10.022470+0000","last_undegraded":"2024-02-10T08:38:10.022470+0000","last_fullsized":"2024-02-10T08:38:10.022470+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":138,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":6272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":3697,"num_objects":343,"num_object_clones":237,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":512,"num_write_kb":511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":11099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1253,"ondisk_log_size":1253,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":286,"num_read_kb":281,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313104,"kb_used_data":3580,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432048,"statfs":{"total":95995035648,"available":95674417152,"internally_reserved":0,"allocated":3665920,"data_stored":1639783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":0.76200000000000001,"15min":0.60199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":22.143000000000001,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.361,"5min":0.68700000000000006,"15min":0.57699999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.7850000000000001,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.60899999999999999,"15min":0.54400000000000004},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.593,"5min":7.593,"15min":7.593},"last":0.623},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.68400000000000005,"15min":0.59699999999999998},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.7759999999999998,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.65100000000000002,"15min":0.55100000000000005},"min":{"1min":0.31900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.9099999999999999,"5min":0.84399999999999997,"15min":0.64900000000000002},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":19.369,"5min":19.369,"15min":19.369},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.69699999999999995,"15min":0.56499999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":0.83199999999999996,"15min":0.63200000000000001},"min":{"1min":0.34899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":22.102,"5min":22.102,"15min":22.102},"last":0.66200000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.50700000000000001,"15min":0.45200000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.827,"5min":8.827,"15min":8.827},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.03,"5min":0.56799999999999995,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.646000000000001,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.64300000000000002,"15min":0.505},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":19.326000000000001,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.71199999999999997,"15min":0.56299999999999994},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":22.687000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.55500000000000005}]}]},{"osd":6,"up_from":44,"seq":188978561091,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304616,"kb_used_data":3288,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440536,"statfs":{"total":95995035648,"available":95683108864,"internally_reserved":0,"allocated":3366912,"data_stored":1638994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":0.63500000000000001,"15min":0.51800000000000002},"min":{"1min":0.26400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":0.65500000000000003,"15min":0.53300000000000003},"min":{"1min":0.28199999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.4629999999999992,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.496}]},{"osd":1,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.59999999999999998,"15min":0.53200000000000003},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":8.3680000000000003,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.57399999999999995,"15min":0.497},"min":{"1min":0.29399999999999998,"5min":0.248,"15min":0.248},"max":{"1min":8.3770000000000007,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.47499999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.57599999999999996,"15min":0.52700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.798,"5min":4.798,"15min":4.798},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.57799999999999996,"15min":0.52700000000000002},"min":{"1min":0.29399999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.46500000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":0.82299999999999995,"15min":0.59199999999999997},"min":{"1min":0.33800000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":16.823,"5min":16.823,"15min":16.823},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.58299999999999996,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.42599999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39700000000000002,"15min":0.379},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.49199999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.40300000000000002,"15min":0.36499999999999999},"min":{"1min":0.25,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.622,"5min":0.63,"15min":0.63},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41399999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.45100000000000001,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58899999999999997,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.38}]},{"osd":7,"last update":"Sat Feb 10 08:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.46999999999999997,"15min":0.45900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.441},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64900000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304776,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440376,"statfs":{"total":95995035648,"available":95682945024,"internally_reserved":0,"allocated":3538944,"data_stored":1641177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.57199999999999995,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.8780000000000001,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.8009999999999999,"5min":0.65500000000000003,"15min":0.46000000000000002},"min":{"1min":0.255,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":16.166,"5min":16.166,"15min":16.166},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":1.103,"15min":0.61799999999999999},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":34.985999999999997,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.624},{"interface":"front","average":{"1min":2.9609999999999999,"5min":0.879,"15min":0.53700000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:37:43 2024","interfaces":[{"interface":"back","average":{"1min":2.323,"5min":0.76400000000000001,"15min":0.54400000000000004},"min":{"1min":0.246,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":24.773,"5min":24.773,"15min":24.773},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":0.72399999999999998,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":16.120999999999999,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":0.749,"15min":0.623},"min":{"1min":0.35199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":16.074000000000002,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.67700000000000005,"15min":0.61199999999999999},"min":{"1min":0.38700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.98999999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.67700000000000005,"15min":0.61899999999999999},"min":{"1min":0.42099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.5600000000000005,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.609,"5min":0.79100000000000004,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.8479999999999999,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.876}]},{"osd":6,"last update":"Sat Feb 10 08:37:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.68700000000000006,"15min":0.61299999999999999},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.312},"max":{"1min":9.532,"5min":9.532,"15min":9.532},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":0.79400000000000004,"15min":0.65800000000000003},"min":{"1min":0.432,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.9000000000000004,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":1.0009999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":0.71399999999999997,"15min":0.59399999999999997},"min":{"1min":0.35199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":9.4640000000000004,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.841,"5min":0.84099999999999997,"15min":0.66700000000000004},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.9199999999999999,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":1.02}]}]},{"osd":0,"up_from":0,"seq":34359738464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304752,"kb_used_data":3428,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440400,"statfs":{"total":95995035648,"available":95682969600,"internally_reserved":0,"allocated":3510272,"data_stored":1638897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.44800000000000001,"15min":0.38500000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.187},"max":{"1min":2.8660000000000001,"5min":2.8660000000000001,"15min":2.8660000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.4490000000000001,"5min":0.58499999999999996,"15min":0.438},"min":{"1min":0.22,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":0.54400000000000004,"15min":0.42399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.0239999999999991,"5min":8.0239999999999991,"15min":8.0239999999999991},"last":0.372},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.501,"15min":0.42299999999999999},"min":{"1min":0.246,"5min":0.22600000000000001,"15min":0.223},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.308}]},{"osd":3,"last update":"Sat Feb 10 08:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.48999999999999999,"15min":0.44600000000000001},"min":{"1min":0.27900000000000003,"5min":0.22,"15min":0.22},"max":{"1min":4.617,"5min":4.617,"15min":4.617},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.54200000000000004,"15min":0.47399999999999998},"min":{"1min":0.307,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1109999999999998,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.52300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:38:05 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":0.86499999999999999,"15min":0.68300000000000005},"min":{"1min":0.34100000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":10.920999999999999,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.514,"5min":0.78000000000000003,"15min":0.66600000000000004},"min":{"1min":0.36799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":12.292999999999999,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:37:32 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":0.77500000000000002,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.65500000000000003,"15min":0.60099999999999998},"min":{"1min":0.35999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:37:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.65000000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.431,"5min":9.266,"15min":9.266},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.68300000000000005,"15min":0.63300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.5910000000000002,"5min":4.5910000000000002,"15min":4.5910000000000002},"last":0.70099999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.79700000000000004,"15min":0.69199999999999995},"min":{"1min":0.56699999999999995,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.5350000000000001,"5min":4.5350000000000001,"15min":4.5350000000000001},"last":0.68400000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300668,"kb_used_data":3440,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444484,"statfs":{"total":95995035648,"available":95687151616,"internally_reserved":0,"allocated":3522560,"data_stored":1639356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":0.82499999999999996,"15min":0.52500000000000002},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":11.550000000000001,"5min":14.744,"15min":14.744},"last":0.245},{"interface":"front","average":{"1min":2.7389999999999999,"5min":0.88,"15min":0.55300000000000005},"min":{"1min":0.27900000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.808,"5min":11.808,"15min":11.808},"last":0.313}]},{"osd":1,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.9350000000000001,"5min":0.82199999999999995,"15min":0.56100000000000005},"min":{"1min":0.27300000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":10.444000000000001,"5min":16.634,"15min":16.634},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.7330000000000001,"5min":0.83299999999999996,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":11.494999999999999,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":0.873,"15min":0.60499999999999998},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":11.516999999999999,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.8180000000000001,"5min":1.0860000000000001,"15min":0.64300000000000002},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":17.722000000000001,"5min":23.294,"15min":23.294},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:38:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.83999999999999997,"15min":0.69499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66300000000000003,"5min":12.371,"15min":12.371},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83099999999999996,"15min":0.66600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.6139999999999999,"5min":14.711,"15min":14.711},"last":0.38300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:37:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":0.97499999999999998,"15min":0.70399999999999996},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":12.579000000000001,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.379},{"interface":"front","average":{"1min":1.26,"5min":0.78500000000000003,"15min":0.65300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":0.42599999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:37:58 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":0.98899999999999999,"15min":0.68600000000000005},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.374},"max":{"1min":10.381,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.179,"5min":0.93500000000000005,"15min":0.67900000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":11.472,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:37:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1829999999999998,"5min":0.88500000000000001,"15min":0.65300000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":14.773999999999999,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":2.5659999999999998,"5min":0.96799999999999997,"15min":0.68600000000000005},"min":{"1min":0.442,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.606999999999999,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.48299999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986252,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308952,"kb_used_data":3528,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436200,"statfs":{"total":95995035648,"available":95678668800,"internally_reserved":0,"allocated":3612672,"data_stored":1640079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":0.80200000000000005,"15min":0.63400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":19.940999999999999,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.157,"5min":0.623,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":9.1760000000000002,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":0.67900000000000005,"15min":0.57899999999999996},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":0.73199999999999998,"15min":0.61399999999999999},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.74399999999999999,"15min":0.60999999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.441},{"interface":"front","average":{"1min":1.373,"5min":0.68700000000000006,"15min":0.59099999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.202,"5min":9.202,"15min":9.202},"last":0.94699999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:37:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1880000000000002,"5min":0.84099999999999997,"15min":0.64500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.5909999999999993,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.46800000000000003},{"interface":"front","average":{"1min":3.1360000000000001,"5min":1.044,"15min":0.70399999999999996},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":25.27,"5min":25.27,"15min":25.27},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:37:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.501,"15min":0.41199999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.8730000000000002,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.45600000000000002,"15min":0.40500000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":3.8290000000000002,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.64800000000000002,"15min":0.47599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.3119999999999994,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.496},{"interface":"front","average":{"1min":1.101,"5min":0.621,"15min":0.495},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":13.753,"5min":13.753,"15min":13.753},"last":0.5}]},{"osd":7,"last update":"Sat Feb 10 08:37:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.51500000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308664,"kb_used_data":3244,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436488,"statfs":{"total":95995035648,"available":95678963712,"internally_reserved":0,"allocated":3321856,"data_stored":1638678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9880000000000004,"5min":2.0009999999999999,"15min":1.0069999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":107.551,"5min":107.551,"15min":107.551},"last":0.38800000000000001},{"interface":"front","average":{"1min":7.3719999999999999,"5min":1.8720000000000001,"15min":0.97599999999999998},"min":{"1min":0.27700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":88.792000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.442}]},{"osd":1,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":6.444,"5min":1.7170000000000001,"15min":0.94999999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":107.205,"5min":107.205,"15min":107.205},"last":0.45700000000000002},{"interface":"front","average":{"1min":4.7750000000000004,"5min":1.3919999999999999,"15min":0.83999999999999997},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":72.272000000000006,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":7.274,"5min":1.8640000000000001,"15min":0.95799999999999996},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":54.174999999999997,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":7.6349999999999998,"5min":1.9510000000000001,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":86.183000000000007,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":3.5299999999999998,"5min":1.119,"15min":0.70899999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":44.395000000000003,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":10.414,"5min":2.528,"15min":1.2090000000000001},"min":{"1min":0.35799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":114.526,"5min":114.526,"15min":114.526},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:37:27 2024","interfaces":[{"interface":"back","average":{"1min":4.0199999999999996,"5min":1.169,"15min":0.65100000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":72.216999999999999,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.371},{"interface":"front","average":{"1min":2.617,"5min":0.89500000000000002,"15min":0.55100000000000005},"min":{"1min":0.27000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":44.345999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.53400000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:37:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":1.1830000000000001,"15min":0.66900000000000004},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67800000000000005,"5min":72.241,"15min":72.241},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.71099999999999997,"15min":0.54100000000000004},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.46700000000000003,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Feb 10 08:37:18 2024","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":1.1830000000000001,"15min":0.71099999999999997},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":66.844999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":4.1180000000000003,"5min":1.248,"15min":0.73999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":72.257000000000005,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":1587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":1638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":610304,"data_stored":1591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":708608,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":1104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":1307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":765952,"data_stored":1994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:38:16.355 INFO:tasks.thrashosds.thrasher:Setting 2.b to [3, 1, 6] 2024-02-10T08:38:16.355 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.b', '3', '1', '6'] 2024-02-10T08:38:16.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 2.b 3 1 6 2024-02-10T08:38:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:16.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:16.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:16.484 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:14.833933+0000 osd.2 (osd.2) 77 : cluster [DBG] 16.a scrub starts 2024-02-10T08:38:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:14.837148+0000 osd.2 (osd.2) 78 : cluster [DBG] 16.a scrub ok 2024-02-10T08:38:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.946297+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.107:0/4236394162' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi107-112411-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.946416+0000 mon.a (mon.0) 1443 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi107-112403-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:14.946493+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.949631+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.955034+0000 mon.a (mon.0) 1445 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.960103+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:14.960912+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:15.049146+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:15.164614+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.107:0/2512040328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:15.536856+0000 osd.5 (osd.5) 97 : cluster [DBG] 16.f scrub starts 2024-02-10T08:38:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:15.539841+0000 osd.5 (osd.5) 98 : cluster [DBG] 16.f scrub ok 2024-02-10T08:38:16.832 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:16.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:16.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:17.265 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.051798+0000 mon.b (mon.1) 296 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.190325+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:16.190406+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-10T08:38:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.195147+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.197246+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:16.425005+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:16.425939+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:16.480291+0000 mgr.y (mgr.14553) 231 : cluster [DBG] pgmap v279: 250 pgs: 12 creating+activating, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.952440+0000 mon.a (mon.0) 1450 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.952536+0000 mon.a (mon.0) 1451 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:16.952634+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.959561+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.961183+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:16.961870+0000 mon.a (mon.0) 1454 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:17.055047+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.051798+0000 mon.b (mon.1) 296 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.190325+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:16.190406+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.195147+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.197246+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:16.425005+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:16.425939+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:16.480291+0000 mgr.y (mgr.14553) 231 : cluster [DBG] pgmap v279: 250 pgs: 12 creating+activating, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.952440+0000 mon.a (mon.0) 1450 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.952536+0000 mon.a (mon.0) 1451 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:16.952634+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.959561+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:17.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.961183+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:16.961870+0000 mon.a (mon.0) 1454 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:17.055047+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.051798+0000 mon.b (mon.1) 296 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.190325+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:16.190406+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.195147+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.197246+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:16.425005+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:38:17.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:16.425939+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:16.480291+0000 mgr.y (mgr.14553) 231 : cluster [DBG] pgmap v279: 250 pgs: 12 creating+activating, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.952440+0000 mon.a (mon.0) 1450 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi107-112403-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.952536+0000 mon.a (mon.0) 1451 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:16.952634+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.959561+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.961183+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:16.961870+0000 mon.a (mon.0) 1454 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:17.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:17.055047+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:17.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:38:17.823 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:17.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:17.831 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:38:17.831 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:38:17.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:17.959 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.214 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:18.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:18.349 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.370 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:18.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:18.502 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.523 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:18.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:18.653 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.672 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:18.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:18.803 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:18.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:18.953 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:18.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:17.726015+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub starts 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:17.727382+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.c scrub ok 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:17.950129+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:17.950265+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:17.950341+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:17.952616+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:17.955557+0000 mon.a (mon.0) 1458 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.005234+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.008840+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:38:19.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:18.056117+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.480769+0000 mgr.y (mgr.14553) 232 : cluster [DBG] pgmap v282: 290 pgs: 12 creating+activating, 40 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:17.726015+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub starts 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:17.727382+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.c scrub ok 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:17.950129+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:17.950265+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:17.950341+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:17.952616+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:17.955557+0000 mon.a (mon.0) 1458 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.005234+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:38:19.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.008840+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:38:19.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:18.056117+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:19.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.480769+0000 mgr.y (mgr.14553) 232 : cluster [DBG] pgmap v282: 290 pgs: 12 creating+activating, 40 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:19.048 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:19.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:38:19.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:19.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:38:19.261 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:19.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:17.726015+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub starts 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:17.727382+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.c scrub ok 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:17.950129+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.107:0/372041220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi107-112411-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:17.950265+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:17.950341+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:17.952616+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:17.955557+0000 mon.a (mon.0) 1458 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]: dispatch 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.005234+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.008840+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:38:19.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:18.056117+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:19.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.480769+0000 mgr.y (mgr.14553) 232 : cluster [DBG] pgmap v282: 290 pgs: 12 creating+activating, 40 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:19.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:19.392 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:19.430 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:19.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:19.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:19.569 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:19.589 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:19.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:19.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:19.722 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:19.741 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:19.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:19.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:19.872 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:19.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.474042+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:38:19.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.475502+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:38:19.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.538804+0000 osd.6 (osd.6) 87 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:38:19.956 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.541688+0000 osd.6 (osd.6) 88 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:38:19.956 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.584280+0000 osd.5 (osd.5) 99 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:38:19.956 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.587945+0000 osd.5 (osd.5) 100 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.474042+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.475502+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.538804+0000 osd.6 (osd.6) 87 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.541688+0000 osd.6 (osd.6) 88 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:38:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.584280+0000 osd.5 (osd.5) 99 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.587945+0000 osd.5 (osd.5) 100 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.758027+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.758839+0000 osd.4 (osd.4) 99 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.947759+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:18.956415+0000 mon.a (mon.0) 1460 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]': finished 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:18.956518+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:18.964909+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:18.965209+0000 mon.a (mon.0) 1462 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.030067+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.057068+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:20.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.237588+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.474042+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.475502+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.538804+0000 osd.6 (osd.6) 87 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.541688+0000 osd.6 (osd.6) 88 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.584280+0000 osd.5 (osd.5) 99 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:38:20.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.587945+0000 osd.5 (osd.5) 100 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.758027+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.758839+0000 osd.4 (osd.4) 99 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.947759+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:18.956415+0000 mon.a (mon.0) 1460 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]': finished 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:18.956518+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:18.964909+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:18.965209+0000 mon.a (mon.0) 1462 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.030067+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.057068+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.237588+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.758027+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:38:20.371 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.758839+0000 osd.4 (osd.4) 99 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.947759+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:18.956415+0000 mon.a (mon.0) 1460 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-17", "mode": "writeback"}]': finished 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:18.956518+0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:18.964909+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:18.965209+0000 mon.a (mon.0) 1462 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.030067+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.057068+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:20.372 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.237588+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2382404789 >> v1:172.21.15.107:6789/0 conn(0x7f0ff0104330 legacy=0x7f0ff01067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2382404789 shutdown_connections 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2382404789 >> 172.21.15.107:0/2382404789 conn(0x7f0ff00f2510 msgr2=0x7f0ff00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2382404789 shutdown_connections 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2382404789 wait complete. 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 librados: starting msgr at 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 librados: starting objecter 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.594+0000 7f0ff53b4700 1 Processor -- start 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 -- start start 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 librados: setting wanted keys 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 librados: calling monclient init 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ff0117480 con 0x7f0ff00f8a60 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ff0118660 con 0x7f0ff00f9550 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0ff53b4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ff0119860 con 0x7f0ff0104330 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0feeffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0ff00f8a60 0x7f0ff00ff820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35948/0 (socket says 172.21.15.107:35948) 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0feeffd700 1 -- 172.21.15.107:0/2510223421 learned_addr learned my addr 172.21.15.107:0/2510223421 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fef7fe700 1 --1- 172.21.15.107:0/2510223421 >> v1:172.21.15.107:6790/0 conn(0x7f0ff0104330 0x7f0ff0115b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36506/0 (socket says 172.21.15.107:36506) 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter ms_handle_connect 0x7f0ff00f8a60 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter ms_handle_connect 0x7f0ff0104330 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter ms_handle_connect 0x7f0ff00f9550 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487743124 0 0) 0x7f0ff0119860 con 0x7f0ff0104330 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd80038e0 con 0x7f0ff0104330 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990840719 0 0) 0x7f0ff0117480 con 0x7f0ff00f8a60 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0119860 con 0x7f0ff00f8a60 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685451987 0 0) 0x7f0ff0118660 con 0x7f0ff00f9550 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0117480 con 0x7f0ff00f9550 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1397846449 0 0) 0x7f0fd80038e0 con 0x7f0ff0104330 2024-02-10T08:38:20.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff0118660 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.595+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0fe0001490 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096459977 0 0) 0x7f0ff0119860 con 0x7f0ff00f8a60 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fd80038e0 con 0x7f0ff00f8a60 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0fe8001ca0 con 0x7f0ff00f8a60 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077717668 0 0) 0x7f0ff0118660 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 >> v1:172.21.15.195:6789/0 conn(0x7f0ff00f9550 legacy=0x7f0ff00fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 >> v1:172.21.15.107:6789/0 conn(0x7f0ff00f8a60 legacy=0x7f0ff00ff820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff011aa60 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff0119a50 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0ff53b4700 10 client.52979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff0119ec0 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0ff53b4700 1 librados: init done 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0fe0002200 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.596+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0fe00037a0 con 0x7f0ff0104330 2024-02-10T08:38:20.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.597+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc004f20 con 0x7f0ff0104330 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.597+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0fe0003090 con 0x7f0ff0104330 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(206..206 src has 50..206) v4 ==== 8767+0+0 (unknown 601247802 0 0) 0x7f0fe0079230 con 0x7f0ff0104330 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 10 client.52979.objecter ms_dispatch 0x7f0ff0001550 osd_map(206..206 src has 50..206) v4 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 3 client.52979.objecter handle_osd_map got epochs [206,206] > 0 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 3 client.52979.objecter handle_osd_map decoding full epoch 206 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 20 client.52979.objecter dump_active .. 0 homeless 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.598+0000 7f0fd7fff700 10 client.52979.objecter ms_handle_connect 0x7f0fd8063a00 2024-02-10T08:38:20.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.602+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0fe004c3c0 con 0x7f0ff0104330 2024-02-10T08:38:20.621 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:20.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:20.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:20.757 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:20.778 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:20.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:20.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.787+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]} v 0) v1 -- 0x7f0fdc0059e0 con 0x7f0ff0104330 2024-02-10T08:38:20.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.942+0000 7f0fd7fff700 1 -- 172.21.15.107:0/2510223421 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]=0 set 2.b pg_upmap mapping to [3,1,6] v207) v1 ==== 127+0+0 (unknown 3374977013 0 0) 0x7f0fe004a830 con 0x7f0ff0104330 2024-02-10T08:38:20.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 10 librados: watch_flush enter 2024-02-10T08:38:20.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 10 librados: watch_flush exit 2024-02-10T08:38:20.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 20 client.52979.objecter shutdown clearing up homeless session... 2024-02-10T08:38:20.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 10 client.52979.objecter successfully canceled tick 2024-02-10T08:38:20.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0fd8063a00 legacy=0x7f0fd8065ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:20.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.946+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 >> v1:172.21.15.107:6790/0 conn(0x7f0ff0104330 legacy=0x7f0ff0115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.950+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 shutdown_connections 2024-02-10T08:38:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.950+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 >> 172.21.15.107:0/2510223421 conn(0x7f0ff00f2510 msgr2=0x7f0ff01077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.951+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 shutdown_connections 2024-02-10T08:38:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.951+0000 7f0ff53b4700 1 -- 172.21.15.107:0/2510223421 wait complete. 2024-02-10T08:38:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:20.951+0000 7f0ff53b4700 1 librados: shutdown 2024-02-10T08:38:20.959 INFO:teuthology.orchestra.run.smithi107.stderr:set 2.b pg_upmap mapping to [3,1,6] 2024-02-10T08:38:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.451325+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.453055+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.609737+0000 osd.5 (osd.5) 101 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.613417+0000 osd.5 (osd.5) 102 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.647393+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.647767+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.846244+0000 osd.2 (osd.2) 79 : cluster [DBG] 25.6 scrub starts 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.849706+0000 osd.2 (osd.2) 80 : cluster [DBG] 25.6 scrub ok 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.960843+0000 mon.a (mon.0) 1464 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.960974+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:19.961034+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.962611+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.962802+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.963263+0000 mon.a (mon.0) 1467 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.963471+0000 mon.a (mon.0) 1468 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:19.964304+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.058116+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.481366+0000 mgr.y (mgr.14553) 233 : cluster [DBG] pgmap v285: 282 pgs: 12 creating+activating, 32 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.789485+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.107:0/2510223421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.789753+0000 mon.a (mon.0) 1470 : audit [INF] from='client.52979 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.932800+0000 mon.a (mon.0) 1471 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.940482+0000 mon.a (mon.0) 1472 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.940556+0000 mon.a (mon.0) 1473 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.940632+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.940757+0000 mon.a (mon.0) 1475 : audit [INF] from='client.52979 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]': finished 2024-02-10T08:38:21.039 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.940821+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-10T08:38:21.080 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:21.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.451325+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:38:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.453055+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:38:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.609737+0000 osd.5 (osd.5) 101 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.613417+0000 osd.5 (osd.5) 102 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.647393+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.647767+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.846244+0000 osd.2 (osd.2) 79 : cluster [DBG] 25.6 scrub starts 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.849706+0000 osd.2 (osd.2) 80 : cluster [DBG] 25.6 scrub ok 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.960843+0000 mon.a (mon.0) 1464 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.960974+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:19.961034+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.962611+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.962802+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.963263+0000 mon.a (mon.0) 1467 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.963471+0000 mon.a (mon.0) 1468 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:19.964304+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.451325+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.453055+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.609737+0000 osd.5 (osd.5) 101 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.613417+0000 osd.5 (osd.5) 102 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.647393+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.647767+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.846244+0000 osd.2 (osd.2) 79 : cluster [DBG] 25.6 scrub starts 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.849706+0000 osd.2 (osd.2) 80 : cluster [DBG] 25.6 scrub ok 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.960843+0000 mon.a (mon.0) 1464 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.960974+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:21.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:19.961034+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.962611+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.107:0/3051098383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.962802+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.963263+0000 mon.a (mon.0) 1467 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.963471+0000 mon.a (mon.0) 1468 : audit [INF] from='client.52955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:19.964304+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.058116+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.481366+0000 mgr.y (mgr.14553) 233 : cluster [DBG] pgmap v285: 282 pgs: 12 creating+activating, 32 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.789485+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.107:0/2510223421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.789753+0000 mon.a (mon.0) 1470 : audit [INF] from='client.52979 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.932800+0000 mon.a (mon.0) 1471 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.940482+0000 mon.a (mon.0) 1472 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.940556+0000 mon.a (mon.0) 1473 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.940632+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.940757+0000 mon.a (mon.0) 1475 : audit [INF] from='client.52979 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]': finished 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.940821+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.058116+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.481366+0000 mgr.y (mgr.14553) 233 : cluster [DBG] pgmap v285: 282 pgs: 12 creating+activating, 32 unknown, 5 creating+peering, 18 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 201 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.789485+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.107:0/2510223421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.789753+0000 mon.a (mon.0) 1470 : audit [INF] from='client.52979 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]: dispatch 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.932800+0000 mon.a (mon.0) 1471 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:21.184 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.940482+0000 mon.a (mon.0) 1472 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-17"}]': finished 2024-02-10T08:38:21.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.940556+0000 mon.a (mon.0) 1473 : audit [INF] from='client.52955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi107-112403-29"}]': finished 2024-02-10T08:38:21.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.940632+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? v1:172.21.15.107:0/3550655139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi107-112411-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:21.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.940757+0000 mon.a (mon.0) 1475 : audit [INF] from='client.52979 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.b", "id": [3, 1, 6]}]': finished 2024-02-10T08:38:21.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.940821+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-10T08:38:21.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:21.213 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:21.336 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:21.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:21.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:21.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:21.761 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:22.515 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:22.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:22.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:22.650 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:22.671 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:22.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.444426+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.445966+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.589142+0000 osd.5 (osd.5) 103 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:20.592060+0000 osd.5 (osd.5) 104 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.978503+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.980270+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:20.980592+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:21.059286+0000 mon.b (mon.1) 306 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:21.251921+0000 mgr.y (mgr.14553) 234 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:21.958332+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:21.958429+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:21.963270+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.444426+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.445966+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.589142+0000 osd.5 (osd.5) 103 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:20.592060+0000 osd.5 (osd.5) 104 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.978503+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.980270+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:20.980592+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:21.059286+0000 mon.b (mon.1) 306 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:21.251921+0000 mgr.y (mgr.14553) 234 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:21.958332+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:21.958429+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-10T08:38:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:21.963270+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.444426+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.445966+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.589142+0000 osd.5 (osd.5) 103 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:20.592060+0000 osd.5 (osd.5) 104 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.978503+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.980270+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:20.980592+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:21.059286+0000 mon.b (mon.1) 306 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:21.251921+0000 mgr.y (mgr.14553) 234 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:21.958332+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi107-112403-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:21.958429+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-10T08:38:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:21.963270+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:22.806 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:22.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:22.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:22.991 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:22.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:23.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:23.125 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:23.146 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:23.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:23.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.617+0000 7f5aae1a3700 1 -- 172.21.15.107:0/2636411158 >> v1:172.21.15.107:6790/0 conn(0x7f5aa8104330 legacy=0x7f5aa81067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:23.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 -- 172.21.15.107:0/2636411158 shutdown_connections 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 -- 172.21.15.107:0/2636411158 >> 172.21.15.107:0/2636411158 conn(0x7f5aa80f3640 msgr2=0x7f5aa80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 -- 172.21.15.107:0/2636411158 shutdown_connections 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 -- 172.21.15.107:0/2636411158 wait complete. 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 librados: starting msgr at 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 librados: starting objecter 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 Processor -- start 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 -- start start 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 librados: setting wanted keys 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.618+0000 7f5aae1a3700 1 librados: calling monclient init 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aa8117200 con 0x7f5aa81007f0 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aa81183a0 con 0x7f5aa8104330 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aa81195a0 con 0x7f5aa80fda50 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa77fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5aa80fda50 0x7f5aa80f9800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36564/0 (socket says 172.21.15.107:36564) 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa77fe700 1 -- 172.21.15.107:0/4159828581 learned_addr learned my addr 172.21.15.107:0/4159828581 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:23.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa6ffd700 1 --1- 172.21.15.107:0/4159828581 >> v1:172.21.15.107:6789/0 conn(0x7f5aa81007f0 0x7f5aa80f9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35990/0 (socket says 172.21.15.107:35990) 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter ms_handle_connect 0x7f5aa80fda50 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter ms_handle_connect 0x7f5aa81007f0 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter ms_handle_connect 0x7f5aa8104330 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193829899 0 0) 0x7f5aa8117200 con 0x7f5aa81007f0 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90003030 con 0x7f5aa81007f0 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095774132 0 0) 0x7f5aa81195a0 con 0x7f5aa80fda50 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8117200 con 0x7f5aa80fda50 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753596173 0 0) 0x7f5aa81183a0 con 0x7f5aa8104330 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa81195a0 con 0x7f5aa8104330 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206521458 0 0) 0x7f5a90003030 con 0x7f5aa81007f0 2024-02-10T08:38:23.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa81183a0 con 0x7f5aa81007f0 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5a9c0019d0 con 0x7f5aa81007f0 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335205905 0 0) 0x7f5aa8117200 con 0x7f5aa80fda50 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a90003030 con 0x7f5aa80fda50 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5a980018d0 con 0x7f5aa80fda50 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717609514 0 0) 0x7f5aa81195a0 con 0x7f5aa8104330 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa8117200 con 0x7f5aa8104330 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5aa00023d0 con 0x7f5aa8104330 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1277093735 0 0) 0x7f5aa81183a0 con 0x7f5aa81007f0 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 >> v1:172.21.15.107:6790/0 conn(0x7f5aa80fda50 legacy=0x7f5aa80f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 >> v1:172.21.15.195:6789/0 conn(0x7f5aa8104330 legacy=0x7f5aa81159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa811a7a0 con 0x7f5aa81007f0 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa81173f0 con 0x7f5aa81007f0 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 10 client.27474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:23.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa81178e0 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.619+0000 7f5aae1a3700 1 librados: init done 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.620+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5a9c002340 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.620+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5a9c005910 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.620+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a94004f20 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.621+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5a9c0024b0 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(208..208 src has 50..208) v4 ==== 8216+0+0 (unknown 784552850 0 0) 0x7f5a9c078be0 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 10 client.27474.objecter ms_dispatch 0x7f5aa8001550 osd_map(208..208 src has 50..208) v4 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 3 client.27474.objecter handle_osd_map got epochs [208,208] > 0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 3 client.27474.objecter handle_osd_map decoding full epoch 208 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 20 client.27474.objecter dump_active .. 0 homeless 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.622+0000 7f5aa4ff9700 10 client.27474.objecter ms_handle_connect 0x7f5a90063d50 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.625+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5a9c04bfa0 con 0x7f5aa81007f0 2024-02-10T08:38:23.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:22.811+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5a940051d0 con 0x7f5aa81007f0 2024-02-10T08:38:23.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.203+0000 7f5aa4ff9700 1 -- 172.21.15.107:0/4159828581 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v209) v1 ==== 101+0+0 (unknown 3237803460 0 0) 0x7f5a9c04a410 con 0x7f5aa81007f0 2024-02-10T08:38:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 10 librados: watch_flush enter 2024-02-10T08:38:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 10 librados: watch_flush exit 2024-02-10T08:38:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 20 client.27474.objecter shutdown clearing up homeless session... 2024-02-10T08:38:23.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 10 client.27474.objecter successfully canceled tick 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5a90063d50 legacy=0x7f5a90066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.206+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 >> v1:172.21.15.107:6789/0 conn(0x7f5aa81007f0 legacy=0x7f5aa80f9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.207+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 shutdown_connections 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.207+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 >> 172.21.15.107:0/4159828581 conn(0x7f5aa80f3640 msgr2=0x7f5aa80f6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.207+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 shutdown_connections 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.207+0000 7f5aae1a3700 1 -- 172.21.15.107:0/4159828581 wait complete. 2024-02-10T08:38:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:23.207+0000 7f5aae1a3700 1 librados: shutdown 2024-02-10T08:38:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:38:23.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:23.278 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:23.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:23.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:23.429 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:21.485051+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:21.488558+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:22.060372+0000 mon.b (mon.1) 307 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.139732+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.140011+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.406341+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.b scrub starts 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.407469+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.b scrub ok 2024-02-10T08:38:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.481754+0000 mgr.y (mgr.14553) 235 : cluster [DBG] pgmap v288: 218 pgs: 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:22.813260+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:22.954307+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.061366+0000 mon.b (mon.1) 308 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.205166+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:23.205232+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.206099+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.107:0/3713373242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.206508+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.206957+0000 mon.a (mon.0) 1487 : audit [INF] from='client.52985 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.207329+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:21.485051+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:21.488558+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:22.060372+0000 mon.b (mon.1) 307 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.139732+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.140011+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.406341+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.b scrub starts 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.407469+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.b scrub ok 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.481754+0000 mgr.y (mgr.14553) 235 : cluster [DBG] pgmap v288: 218 pgs: 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:22.813260+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:22.954307+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.061366+0000 mon.b (mon.1) 308 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.205166+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:23.205232+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.206099+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.107:0/3713373242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.206508+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.206957+0000 mon.a (mon.0) 1487 : audit [INF] from='client.52985 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.207329+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:21.485051+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:21.488558+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:22.060372+0000 mon.b (mon.1) 307 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.139732+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.140011+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.406341+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.b scrub starts 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.407469+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.b scrub ok 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.481754+0000 mgr.y (mgr.14553) 235 : cluster [DBG] pgmap v288: 218 pgs: 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:22.813260+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:23.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:22.954307+0000 mon.a (mon.0) 1484 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.061366+0000 mon.b (mon.1) 308 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.205166+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? v1:172.21.15.107:0/4159828581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:23.205232+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.206099+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.107:0/3713373242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.206508+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.206957+0000 mon.a (mon.0) 1487 : audit [INF] from='client.52985 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:23.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.207329+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:24.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:24.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:25.072 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:23.399446+0000 osd.3 (osd.3) 119 : cluster [DBG] 25.1b scrub starts 2024-02-10T08:38:25.072 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:23.400980+0000 osd.3 (osd.3) 120 : cluster [DBG] 25.1b scrub ok 2024-02-10T08:38:25.072 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.961512+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:25.072 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.961692+0000 mon.a (mon.0) 1490 : audit [INF] from='client.52985 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.072 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.961765+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:23.961855+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-10T08:38:25.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.970205+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:23.971595+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:24.062523+0000 mon.b (mon.1) 311 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:25.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:24.482293+0000 mgr.y (mgr.14553) 236 : cluster [DBG] pgmap v291: 290 pgs: 72 unknown, 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:23.399446+0000 osd.3 (osd.3) 119 : cluster [DBG] 25.1b scrub starts 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:23.400980+0000 osd.3 (osd.3) 120 : cluster [DBG] 25.1b scrub ok 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.961512+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.961692+0000 mon.a (mon.0) 1490 : audit [INF] from='client.52985 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.961765+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:23.961855+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.970205+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:23.971595+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:24.062523+0000 mon.b (mon.1) 311 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:25.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:24.482293+0000 mgr.y (mgr.14553) 236 : cluster [DBG] pgmap v291: 290 pgs: 72 unknown, 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:23.399446+0000 osd.3 (osd.3) 119 : cluster [DBG] 25.1b scrub starts 2024-02-10T08:38:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:23.400980+0000 osd.3 (osd.3) 120 : cluster [DBG] 25.1b scrub ok 2024-02-10T08:38:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.961512+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi107-112403-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.961692+0000 mon.a (mon.0) 1490 : audit [INF] from='client.52985 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi107-112411-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.961765+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:23.961855+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-10T08:38:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.970205+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:23.971595+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:24.062523+0000 mon.b (mon.1) 311 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:24.482293+0000 mgr.y (mgr.14553) 236 : cluster [DBG] pgmap v291: 290 pgs: 72 unknown, 1 peering, 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 188 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.573+0000 7ffad1aa4700 1 -- 172.21.15.107:0/1904770588 >> v1:172.21.15.107:6790/0 conn(0x7ffacc1007f0 legacy=0x7ffacc102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 -- 172.21.15.107:0/1904770588 shutdown_connections 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 -- 172.21.15.107:0/1904770588 >> 172.21.15.107:0/1904770588 conn(0x7ffacc0f3640 msgr2=0x7ffacc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 -- 172.21.15.107:0/1904770588 shutdown_connections 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 -- 172.21.15.107:0/1904770588 wait complete. 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 librados: starting msgr at 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 librados: starting objecter 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 Processor -- start 2024-02-10T08:38:25.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 -- start start 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 librados: setting wanted keys 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.574+0000 7ffad1aa4700 1 librados: calling monclient init 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffacc0fa920 con 0x7ffacc104330 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffacc1187f0 con 0x7ffacc0fda50 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffacc119990 con 0x7ffacc1007f0 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffacaffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ffacc1007f0 0x7ffacc112990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36614/0 (socket says 172.21.15.107:36614) 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffacaffd700 1 -- 172.21.15.107:0/2603337646 learned_addr learned my addr 172.21.15.107:0/2603337646 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter ms_handle_connect 0x7ffacc1007f0 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter ms_handle_connect 0x7ffacc104330 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter ms_handle_connect 0x7ffacc0fda50 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304315544 0 0) 0x7ffacc0fa920 con 0x7ffacc104330 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4003030 con 0x7ffacc104330 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55376438 0 0) 0x7ffacc119990 con 0x7ffacc1007f0 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffacc0fa920 con 0x7ffacc1007f0 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972789929 0 0) 0x7ffacc1187f0 con 0x7ffacc0fda50 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffacc119990 con 0x7ffacc0fda50 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3714376958 0 0) 0x7ffacc0fa920 con 0x7ffacc1007f0 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffacc1187f0 con 0x7ffacc1007f0 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ffac0002050 con 0x7ffacc1007f0 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872934841 0 0) 0x7ffab4003030 con 0x7ffacc104330 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffacc0fa920 con 0x7ffacc104330 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ffac4001b00 con 0x7ffacc104330 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43298144 0 0) 0x7ffacc119990 con 0x7ffacc0fda50 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffab4003030 con 0x7ffacc0fda50 2024-02-10T08:38:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ffabc0018a0 con 0x7ffacc0fda50 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2546206569 0 0) 0x7ffacc1187f0 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.195:6789/0 conn(0x7ffacc0fda50 legacy=0x7ffacc0f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6789/0 conn(0x7ffacc104330 legacy=0x7ffacc1160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc11ab70 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc117840 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 10 client.53003.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc117cb0 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffad1aa4700 1 librados: init done 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ffac0002390 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.575+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ffac0003800 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.576+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffab8004f00 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ffac0001840 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(211..211 src has 50..211) v4 ==== 9142+0+0 (unknown 2610459335 0 0) 0x7ffac00793c0 con 0x7ffacc1007f0 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 osd_map(211..211 src has 50..211) v4 2024-02-10T08:38:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 3 client.53003.objecter handle_osd_map got epochs [211,211] > 0 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 3 client.53003.objecter handle_osd_map decoding full epoch 211 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 20 client.53003.objecter dump_active .. 0 homeless 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v211) v1 ==== 54+0+16 (unknown 472078242 0 2420000192) 0x7ffac007b7d0 con 0x7ffacc1007f0 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.577+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab4063da0 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 1 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 1 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 0 1 2024-02-10T08:38:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 4 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 10 client.53003.objecter _send_command 1 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab8004950 con 0x7ffab8004530 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab8004530 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab8004950 con 0x7ffab8004530 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 10 client.53003.objecter _finish_command 1 = system:0 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 4 2024-02-10T08:38:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.578+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 0 1 2024-02-10T08:38:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 2 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 2 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab80051a0 osd=0 3 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 0 2 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 4 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 10 client.53003.objecter _send_command 2 2024-02-10T08:38:25.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.593+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab8004d30 con 0x7ffab8004530 2024-02-10T08:38:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab8004d30 con 0x7ffab8004530 2024-02-10T08:38:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 10 client.53003.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 4 2024-02-10T08:38:25.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 0 2 2024-02-10T08:38:25.597 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:25.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 3 2024-02-10T08:38:25.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 1 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 4 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 10 client.53003.objecter _send_command 3 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab800c000 con 0x7ffab8004150 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.595+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab8004150 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.596+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab800c000 con 0x7ffab8004150 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.596+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.596+0000 7ffac8f74700 10 client.53003.objecter _finish_command 3 = system:0 2024-02-10T08:38:25.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.596+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 4 2024-02-10T08:38:25.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.596+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 1 3 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 4 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 4 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8003fe0 osd=1 3 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 1 4 2024-02-10T08:38:25.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 4 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 10 client.53003.objecter _send_command 4 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.610+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab80018b0 con 0x7ffab8004150 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab80018b0 con 0x7ffab8004150 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 10 client.53003.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 4 2024-02-10T08:38:25.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 1 4 2024-02-10T08:38:25.614 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 5 2024-02-10T08:38:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 5 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 2 5 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 4 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 10 client.53003.objecter _send_command 5 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab800de90 con 0x7ffab800d2f0 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.612+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab800d2f0 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.613+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab800de90 con 0x7ffab800d2f0 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.613+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.613+0000 7ffac8f74700 10 client.53003.objecter _finish_command 5 = system:0 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.613+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 4 2024-02-10T08:38:25.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.613+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 2 5 2024-02-10T08:38:25.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 6 2024-02-10T08:38:25.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 6 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8003630 osd=2 3 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 2 6 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 4 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 10 client.53003.objecter _send_command 6 2024-02-10T08:38:25.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.627+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab80030a0 con 0x7ffab800d2f0 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab80030a0 con 0x7ffab800d2f0 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffac8f74700 10 client.53003.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 4 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 2 6 2024-02-10T08:38:25.630 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:25.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 7 2024-02-10T08:38:25.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 7 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 3 7 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 4 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 10 client.53003.objecter _send_command 7 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.628+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab8019f10 con 0x7ffab8016400 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab8016400 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab8019f10 con 0x7ffab8016400 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 10 client.53003.objecter _finish_command 7 = system:0 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 4 2024-02-10T08:38:25.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.629+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 3 7 2024-02-10T08:38:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:25.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:25.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 8 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 8 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8015d80 osd=3 3 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 3 8 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 4 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 10 client.53003.objecter _send_command 8 2024-02-10T08:38:25.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.644+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab800e240 con 0x7ffab8016400 2024-02-10T08:38:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab800e240 con 0x7ffab8016400 2024-02-10T08:38:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 10 client.53003.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 4 2024-02-10T08:38:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 3 8 2024-02-10T08:38:25.648 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 9 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 9 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 4 9 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 4 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 10 client.53003.objecter _send_command 9 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab800e1b0 con 0x7ffab800ce50 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.646+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab800ce50 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.647+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab800e1b0 con 0x7ffab800ce50 2024-02-10T08:38:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.647+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.647+0000 7ffac8f74700 10 client.53003.objecter _finish_command 9 = system:0 2024-02-10T08:38:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.647+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 4 2024-02-10T08:38:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.647+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 4 9 2024-02-10T08:38:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 10 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 10 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab800e000 osd=4 3 2024-02-10T08:38:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 4 10 2024-02-10T08:38:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 4 2024-02-10T08:38:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 10 client.53003.objecter _send_command 10 2024-02-10T08:38:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.662+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab800e1b0 con 0x7ffab800ce50 2024-02-10T08:38:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab800e1b0 con 0x7ffab800ce50 2024-02-10T08:38:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 10 client.53003.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 4 2024-02-10T08:38:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 4 10 2024-02-10T08:38:25.667 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 11 2024-02-10T08:38:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 11 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 5 11 2024-02-10T08:38:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 4 2024-02-10T08:38:25.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 10 client.53003.objecter _send_command 11 2024-02-10T08:38:25.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab801c480 con 0x7ffab801be70 2024-02-10T08:38:25.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.664+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab801be70 2024-02-10T08:38:25.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.665+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab801c480 con 0x7ffab801be70 2024-02-10T08:38:25.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.665+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:25.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.665+0000 7ffac8f74700 10 client.53003.objecter _finish_command 11 = system:0 2024-02-10T08:38:25.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.665+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 4 2024-02-10T08:38:25.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.665+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 5 11 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 12 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 12 2024-02-10T08:38:25.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab800e1b0 osd=5 3 2024-02-10T08:38:25.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 5 12 2024-02-10T08:38:25.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 4 2024-02-10T08:38:25.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 10 client.53003.objecter _send_command 12 2024-02-10T08:38:25.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.679+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab801c480 con 0x7ffab801be70 2024-02-10T08:38:25.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab801c480 con 0x7ffab801be70 2024-02-10T08:38:25.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 10 client.53003.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 4 2024-02-10T08:38:25.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 5 12 2024-02-10T08:38:25.684 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 13 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 13 2024-02-10T08:38:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 6 13 2024-02-10T08:38:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 4 2024-02-10T08:38:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 10 client.53003.objecter _send_command 13 2024-02-10T08:38:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab8027060 con 0x7ffab8023490 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.682+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab8023490 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.683+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab8027060 con 0x7ffab8023490 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.683+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.683+0000 7ffac8f74700 10 client.53003.objecter _finish_command 13 = system:0 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.683+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 4 2024-02-10T08:38:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.683+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 6 13 2024-02-10T08:38:25.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 14 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 14 2024-02-10T08:38:25.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab801c480 osd=6 3 2024-02-10T08:38:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 6 14 2024-02-10T08:38:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 4 2024-02-10T08:38:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 10 client.53003.objecter _send_command 14 2024-02-10T08:38:25.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.697+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab8027060 con 0x7ffab8023490 2024-02-10T08:38:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab8027060 con 0x7ffab8023490 2024-02-10T08:38:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffac8f74700 10 client.53003.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 4 2024-02-10T08:38:25.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 6 14 2024-02-10T08:38:25.702 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 15 2024-02-10T08:38:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 15 2024-02-10T08:38:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 7 15 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 4 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 10 client.53003.objecter _send_command 15 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.699+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffab80282b0 con 0x7ffab8028d60 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 10 client.53003.objecter ms_handle_connect 0x7ffab8028d60 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffab80282b0 con 0x7ffab8028d60 2024-02-10T08:38:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 10 client.53003.objecter _finish_command 15 = system:0 2024-02-10T08:38:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 4 2024-02-10T08:38:25.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.700+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 7 15 2024-02-10T08:38:25.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 10 client.53003.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign -1 16 2024-02-10T08:38:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 20 client.53003.objecter _get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_remove -1 16 2024-02-10T08:38:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 20 client.53003.objecter get_session s=0x7ffab8027060 osd=7 3 2024-02-10T08:38:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 15 client.53003.objecter _session_command_op_assign 7 16 2024-02-10T08:38:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 4 2024-02-10T08:38:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 10 client.53003.objecter _send_command 16 2024-02-10T08:38:25.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.714+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffab80282b0 con 0x7ffab8028d60 2024-02-10T08:38:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffac8f74700 1 -- 172.21.15.107:0/2603337646 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffab80282b0 con 0x7ffab8028d60 2024-02-10T08:38:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffac8f74700 10 client.53003.objecter ms_dispatch 0x7ffacc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:25.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffac8f74700 10 client.53003.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffac8f74700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 4 2024-02-10T08:38:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffac8f74700 15 client.53003.objecter _session_command_op_remove 7 16 2024-02-10T08:38:25.719 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffad1aa4700 10 librados: watch_flush enter 2024-02-10T08:38:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.716+0000 7ffad1aa4700 10 librados: watch_flush exit 2024-02-10T08:38:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.0 2024-02-10T08:38:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6801/3321596972 conn(0x7ffab8004530 legacy=0x7ffab8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab80051a0 osd=0 1 2024-02-10T08:38:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.1 2024-02-10T08:38:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6805/133516126 conn(0x7ffab8004150 legacy=0x7ffab800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003fe0 osd=1 1 2024-02-10T08:38:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.2 2024-02-10T08:38:25.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6809/3575881728 conn(0x7ffab800d2f0 legacy=0x7ffab800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8003630 osd=2 1 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.3 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6813/2871825691 conn(0x7ffab8016400 legacy=0x7ffab80187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8015d80 osd=3 1 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.4 2024-02-10T08:38:25.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.195:6800/2499671285 conn(0x7ffab800ce50 legacy=0x7ffab801b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffacbfff700 1 -- 172.21.15.107:0/2603337646 reap_dead start 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffacbfff700 1 -- 172.21.15.107:0/2603337646 reap_dead start 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e000 osd=4 1 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.5 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.195:6804/4178321486 conn(0x7ffab801be70 legacy=0x7ffab8021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab800e1b0 osd=5 1 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.6 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.195:6808/1417479017 conn(0x7ffab8023490 legacy=0x7ffab8025880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab801c480 osd=6 1 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter close_session for osd.7 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.195:6812/1934176928 conn(0x7ffab8028d60 legacy=0x7ffab80291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter put_session s=0x7ffab8027060 osd=7 1 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 20 client.53003.objecter shutdown clearing up homeless session... 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 10 client.53003.objecter successfully canceled tick 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6800/2634842009 conn(0x7ffab4063da0 legacy=0x7ffab4066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> v1:172.21.15.107:6790/0 conn(0x7ffacc1007f0 legacy=0x7ffacc112990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffacbfff700 1 -- 172.21.15.107:0/2603337646 reap_dead start 2024-02-10T08:38:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 shutdown_connections 2024-02-10T08:38:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 >> 172.21.15.107:0/2603337646 conn(0x7ffacc0f3640 msgr2=0x7ffacc1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 shutdown_connections 2024-02-10T08:38:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 -- 172.21.15.107:0/2603337646 wait complete. 2024-02-10T08:38:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:25.717+0000 7ffad1aa4700 1 librados: shutdown 2024-02-10T08:38:25.728 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.730 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:38:25.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:25.774 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:25.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:25] "GET /metrics HTTP/1.1" 200 190122 "" "Prometheus/2.33.4" 2024-02-10T08:38:26.224 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:26.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:24.611482+0000 osd.5 (osd.5) 105 : cluster [DBG] 25.c deep-scrub starts 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:24.614421+0000 osd.5 (osd.5) 106 : cluster [DBG] 25.c deep-scrub ok 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:24.971310+0000 mon.a (mon.0) 1494 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:24.971438+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:24.976581+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:24.977143+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.063674+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.249795+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.579014+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2603337646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:26.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.939451+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.939556+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.942447+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.948915+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.949532+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.951996+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.107:0/1556895985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:25.956848+0000 mon.a (mon.0) 1502 : audit [INF] from='client.52997 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:24.611482+0000 osd.5 (osd.5) 105 : cluster [DBG] 25.c deep-scrub starts 2024-02-10T08:38:26.228 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:24.614421+0000 osd.5 (osd.5) 106 : cluster [DBG] 25.c deep-scrub ok 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:24.971310+0000 mon.a (mon.0) 1494 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:24.971438+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:24.976581+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:24.977143+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.063674+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.249795+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.579014+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2603337646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.939451+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.939556+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.942447+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.229 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.948915+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:26.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.949532+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.951996+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.107:0/1556895985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:25.956848+0000 mon.a (mon.0) 1502 : audit [INF] from='client.52997 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:26.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:24.611482+0000 osd.5 (osd.5) 105 : cluster [DBG] 25.c deep-scrub starts 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:24.614421+0000 osd.5 (osd.5) 106 : cluster [DBG] 25.c deep-scrub ok 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:24.971310+0000 mon.a (mon.0) 1494 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:24.971438+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:24.976581+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:24.977143+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.063674+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.249795+0000 mon.a (mon.0) 1497 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.579014+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.107:0/2603337646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.939451+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:26.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.939556+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-10T08:38:26.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.942447+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.948915+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:26.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.949532+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]: dispatch 2024-02-10T08:38:26.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.951996+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.107:0/1556895985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:25.956848+0000 mon.a (mon.0) 1502 : audit [INF] from='client.52997 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:26.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:26.363 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:26.385 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:26.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:26.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:26.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:38:26.665 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:26.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:26.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:26.784 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:38:26.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:38:26.784 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-10T08:38:26.784 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:38:26.785 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:38:26.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-10T08:38:26.806 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:26.827 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:26.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:26.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:26.975 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:26.996 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:26.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:27.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:27.126 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.630745+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:38:27.126 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.633603+0000 osd.5 (osd.5) 108 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:38:27.126 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.848705+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.e scrub starts 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.851572+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.e scrub ok 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.900862+0000 osd.2 (osd.2) 81 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:25.903750+0000 osd.2 (osd.2) 82 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.064722+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.483271+0000 mgr.y (mgr.14553) 237 : cluster [DBG] pgmap v294: 282 pgs: 32 creating+peering, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.936504+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.943451+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.943531+0000 mon.a (mon.0) 1505 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]': finished 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.943643+0000 mon.a (mon.0) 1506 : audit [INF] from='client.52997 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.943706+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-10T08:38:27.127 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.945091+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:27.128 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.148 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.630745+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:38:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.633603+0000 osd.5 (osd.5) 108 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:38:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.848705+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.e scrub starts 2024-02-10T08:38:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.851572+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.e scrub ok 2024-02-10T08:38:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.900862+0000 osd.2 (osd.2) 81 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:25.903750+0000 osd.2 (osd.2) 82 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.064722+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.483271+0000 mgr.y (mgr.14553) 237 : cluster [DBG] pgmap v294: 282 pgs: 32 creating+peering, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.936504+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.943451+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.943531+0000 mon.a (mon.0) 1505 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]': finished 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.943643+0000 mon.a (mon.0) 1506 : audit [INF] from='client.52997 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.943706+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.945091+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.630745+0000 osd.5 (osd.5) 107 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.633603+0000 osd.5 (osd.5) 108 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.848705+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.e scrub starts 2024-02-10T08:38:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.851572+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.e scrub ok 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.900862+0000 osd.2 (osd.2) 81 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:25.903750+0000 osd.2 (osd.2) 82 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.064722+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.483271+0000 mgr.y (mgr.14553) 237 : cluster [DBG] pgmap v294: 282 pgs: 32 creating+peering, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.936504+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.943451+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.943531+0000 mon.a (mon.0) 1505 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-19", "mode": "writeback"}]': finished 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.943643+0000 mon.a (mon.0) 1506 : audit [INF] from='client.52997 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi107-112411-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.943706+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-10T08:38:27.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.945091+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]: dispatch 2024-02-10T08:38:27.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:27.279 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.298 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:27.301 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:27.301 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:38:27.302 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:38:27.302 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:27.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:27.429 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.482 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:27.518 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:27.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.632 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:27.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.782 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:27.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:27.914 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:27.934 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:27.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:27.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:38:28.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:28.093 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:28.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:28.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:28.223 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:28.242 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:28.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:28.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:28.373 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:28.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.674720+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:38:28.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.677691+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:38:28.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.869372+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:38:28.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.871096+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.941032+0000 osd.1 (osd.1) 115 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:26.946525+0000 osd.1 (osd.1) 116 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.997491+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:26.997872+0000 mon.a (mon.0) 1509 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.065851+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.942828+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.942927+0000 mon.a (mon.0) 1511 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:27.942978+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.949948+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.950359+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.968673+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.969001+0000 mon.a (mon.0) 1514 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.970595+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.971027+0000 mon.a (mon.0) 1515 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.971439+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:27.971683+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.674720+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.677691+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.869372+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.871096+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.941032+0000 osd.1 (osd.1) 115 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:26.946525+0000 osd.1 (osd.1) 116 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.997491+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:26.997872+0000 mon.a (mon.0) 1509 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.065851+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:28.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.942828+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.942927+0000 mon.a (mon.0) 1511 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:27.942978+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.949948+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.950359+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.968673+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.969001+0000 mon.a (mon.0) 1514 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.970595+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.971027+0000 mon.a (mon.0) 1515 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.971439+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:28.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:27.971683+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.674720+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:38:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.677691+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:38:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.869372+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:38:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.871096+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:38:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.941032+0000 osd.1 (osd.1) 115 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:26.946525+0000 osd.1 (osd.1) 116 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.997491+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:26.997872+0000 mon.a (mon.0) 1509 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.065851+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.942828+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? v1:172.21.15.107:0/1645625687' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi107-112403-30"}]': finished 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.942927+0000 mon.a (mon.0) 1511 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:27.942978+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.949948+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.950359+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.968673+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.969001+0000 mon.a (mon.0) 1514 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.970595+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.971027+0000 mon.a (mon.0) 1515 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.971439+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:27.971683+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:29.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:38:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:29.265 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:27.938763+0000 osd.1 (osd.1) 117 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:27.952511+0000 osd.1 (osd.1) 118 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.068019+0000 mon.b (mon.1) 322 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.473711+0000 mon.c (mon.2) 259 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:28.483836+0000 mgr.y (mgr.14553) 238 : cluster [DBG] pgmap v297: 250 pgs: 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.845939+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:28.939693+0000 mon.a (mon.0) 1518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.946265+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.946353+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:29.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:28.946436+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.955638+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.107:0/29749952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.956210+0000 mon.a (mon.0) 1522 : audit [INF] from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.956962+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:28.957818+0000 mon.a (mon.0) 1523 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:29.068909+0000 mon.b (mon.1) 325 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:27.938763+0000 osd.1 (osd.1) 117 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:27.952511+0000 osd.1 (osd.1) 118 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.068019+0000 mon.b (mon.1) 322 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.473711+0000 mon.c (mon.2) 259 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:38:29.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:28.483836+0000 mgr.y (mgr.14553) 238 : cluster [DBG] pgmap v297: 250 pgs: 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.845939+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:28.939693+0000 mon.a (mon.0) 1518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.946265+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.946353+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:28.946436+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.955638+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.107:0/29749952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.956210+0000 mon.a (mon.0) 1522 : audit [INF] from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.956962+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:28.957818+0000 mon.a (mon.0) 1523 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:29.068909+0000 mon.b (mon.1) 325 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:29.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:27.938763+0000 osd.1 (osd.1) 117 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:27.952511+0000 osd.1 (osd.1) 118 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.068019+0000 mon.b (mon.1) 322 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.473711+0000 mon.c (mon.2) 259 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:28.483836+0000 mgr.y (mgr.14553) 238 : cluster [DBG] pgmap v297: 250 pgs: 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.845939+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:28.939693+0000 mon.a (mon.0) 1518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.946265+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-19"}]': finished 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.946353+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi107-112403-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:28.946436+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-10T08:38:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.955638+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.107:0/29749952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.956210+0000 mon.a (mon.0) 1522 : audit [INF] from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.956962+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:28.957818+0000 mon.a (mon.0) 1523 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:29.068909+0000 mon.b (mon.1) 325 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:29.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:29.640 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:29.660 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:29.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:29.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:29.792 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:29.810 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:29.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:29.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:28.903338+0000 osd.1 (osd.1) 119 : cluster [DBG] 16.b deep-scrub starts 2024-02-10T08:38:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:28.906464+0000 osd.1 (osd.1) 120 : cluster [DBG] 16.b deep-scrub ok 2024-02-10T08:38:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:29.380726+0000 mon.c (mon.2) 260 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:29.381358+0000 mon.c (mon.2) 261 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:29.382499+0000 mgr.y (mgr.14553) 239 : cluster [DBG] pgmap v299: 282 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:29.386102+0000 mon.a (mon.0) 1524 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:29.945758+0000 mon.a (mon.0) 1525 : audit [INF] from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:30.069845+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.188472+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:28.903338+0000 osd.1 (osd.1) 119 : cluster [DBG] 16.b deep-scrub starts 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:28.906464+0000 osd.1 (osd.1) 120 : cluster [DBG] 16.b deep-scrub ok 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:29.380726+0000 mon.c (mon.2) 260 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:29.381358+0000 mon.c (mon.2) 261 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:29.382499+0000 mgr.y (mgr.14553) 239 : cluster [DBG] pgmap v299: 282 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:29.386102+0000 mon.a (mon.0) 1524 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:29.945758+0000 mon.a (mon.0) 1525 : audit [INF] from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:30.069845+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:30.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.188472+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:28.903338+0000 osd.1 (osd.1) 119 : cluster [DBG] 16.b deep-scrub starts 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:28.906464+0000 osd.1 (osd.1) 120 : cluster [DBG] 16.b deep-scrub ok 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:29.380726+0000 mon.c (mon.2) 260 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:29.381358+0000 mon.c (mon.2) 261 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:29.382499+0000 mgr.y (mgr.14553) 239 : cluster [DBG] pgmap v299: 282 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 226 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:29.386102+0000 mon.a (mon.0) 1524 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:29.945758+0000 mon.a (mon.0) 1525 : audit [INF] from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi107-112411-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:30.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:30.069845+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:30.893 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.188472+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-10T08:38:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:30.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:31.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:31.032 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:31.050 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:31.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:31.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:31.179 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:31.197 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:31.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:31.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:31.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:31.330 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:31.367 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:31.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:31.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-10T08:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:29.720993+0000 osd.5 (osd.5) 111 : cluster [DBG] 25.1d scrub starts 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:29.724505+0000 osd.5 (osd.5) 112 : cluster [DBG] 25.1d scrub ok 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.288652+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.291534+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.388341+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.1d deep-scrub starts 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.389072+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.1d deep-scrub ok 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.429358+0000 mon.a (mon.0) 1527 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:30.942265+0000 mon.a (mon.0) 1528 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.942397+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:30.955406+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:30.958894+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.070773+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:29.720993+0000 osd.5 (osd.5) 111 : cluster [DBG] 25.1d scrub starts 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:29.724505+0000 osd.5 (osd.5) 112 : cluster [DBG] 25.1d scrub ok 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.288652+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.291534+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.388341+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.1d deep-scrub starts 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.389072+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.1d deep-scrub ok 2024-02-10T08:38:31.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.429358+0000 mon.a (mon.0) 1527 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:30.942265+0000 mon.a (mon.0) 1528 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.942397+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:30.955406+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:30.958894+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.070773+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:29.720993+0000 osd.5 (osd.5) 111 : cluster [DBG] 25.1d scrub starts 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:29.724505+0000 osd.5 (osd.5) 112 : cluster [DBG] 25.1d scrub ok 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.288652+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.291534+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.388341+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.1d deep-scrub starts 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.389072+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.1d deep-scrub ok 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.429358+0000 mon.a (mon.0) 1527 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-02-10T08:38:31.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:30.942265+0000 mon.a (mon.0) 1528 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi107-112403-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:31.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.942397+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-10T08:38:31.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:30.955406+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:30.958894+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:31.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.070773+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.692660+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.c scrub starts 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:30.695679+0000 osd.5 (osd.5) 114 : cluster [DBG] 16.c scrub ok 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.262569+0000 mgr.y (mgr.14553) 240 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.289699+0000 osd.6 (osd.6) 91 : cluster [DBG] 25.8 scrub starts 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.292840+0000 osd.6 (osd.6) 92 : cluster [DBG] 25.8 scrub ok 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.383582+0000 mgr.y (mgr.14553) 241 : cluster [DBG] pgmap v302: 258 pgs: 40 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 297 B/s wr, 1 op/s 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.435039+0000 osd.3 (osd.3) 121 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.435988+0000 osd.3 (osd.3) 122 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.838276+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:38:32.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.840214+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.945360+0000 mon.a (mon.0) 1531 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.945503+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.951860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.107:0/2081909442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.967407+0000 mon.a (mon.0) 1533 : audit [INF] from='client.53021 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.973740+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:31.974004+0000 mon.a (mon.0) 1534 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.071720+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.692660+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.c scrub starts 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:30.695679+0000 osd.5 (osd.5) 114 : cluster [DBG] 16.c scrub ok 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.262569+0000 mgr.y (mgr.14553) 240 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.289699+0000 osd.6 (osd.6) 91 : cluster [DBG] 25.8 scrub starts 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.292840+0000 osd.6 (osd.6) 92 : cluster [DBG] 25.8 scrub ok 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.383582+0000 mgr.y (mgr.14553) 241 : cluster [DBG] pgmap v302: 258 pgs: 40 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 297 B/s wr, 1 op/s 2024-02-10T08:38:32.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.435039+0000 osd.3 (osd.3) 121 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.435988+0000 osd.3 (osd.3) 122 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.838276+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.840214+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.945360+0000 mon.a (mon.0) 1531 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.945503+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.951860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.107:0/2081909442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.967407+0000 mon.a (mon.0) 1533 : audit [INF] from='client.53021 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.973740+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:31.974004+0000 mon.a (mon.0) 1534 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.678 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.071720+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.692660+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.c scrub starts 2024-02-10T08:38:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:30.695679+0000 osd.5 (osd.5) 114 : cluster [DBG] 16.c scrub ok 2024-02-10T08:38:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.262569+0000 mgr.y (mgr.14553) 240 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.289699+0000 osd.6 (osd.6) 91 : cluster [DBG] 25.8 scrub starts 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.292840+0000 osd.6 (osd.6) 92 : cluster [DBG] 25.8 scrub ok 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.383582+0000 mgr.y (mgr.14553) 241 : cluster [DBG] pgmap v302: 258 pgs: 40 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 297 B/s wr, 1 op/s 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.435039+0000 osd.3 (osd.3) 121 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.435988+0000 osd.3 (osd.3) 122 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.838276+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.840214+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.945360+0000 mon.a (mon.0) 1531 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.945503+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.951860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? v1:172.21.15.107:0/2081909442' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.967407+0000 mon.a (mon.0) 1533 : audit [INF] from='client.53021 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.973740+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:31.974004+0000 mon.a (mon.0) 1534 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.071720+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:33.552 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:33.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:33.558 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:33.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:33.690 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:33.712 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:33.713 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.742360+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:31.745358+0000 osd.5 (osd.5) 116 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.339236+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.342192+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.442122+0000 osd.3 (osd.3) 123 : cluster [DBG] 25.1a deep-scrub starts 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.444407+0000 osd.3 (osd.3) 124 : cluster [DBG] 25.1a deep-scrub ok 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.884132+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.886086+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.948091+0000 mon.a (mon.0) 1535 : audit [INF] from='client.53021 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.948229+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.948285+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.955524+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.956163+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.958352+0000 mon.a (mon.0) 1538 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:32.958605+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.072750+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:33.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.742360+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:31.745358+0000 osd.5 (osd.5) 116 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.339236+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.342192+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.442122+0000 osd.3 (osd.3) 123 : cluster [DBG] 25.1a deep-scrub starts 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.444407+0000 osd.3 (osd.3) 124 : cluster [DBG] 25.1a deep-scrub ok 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.884132+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.886086+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:38:33.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.948091+0000 mon.a (mon.0) 1535 : audit [INF] from='client.53021 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.948229+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.948285+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.955524+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.956163+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.958352+0000 mon.a (mon.0) 1538 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:32.958605+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.072750+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.742360+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:31.745358+0000 osd.5 (osd.5) 116 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.339236+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:38:33.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.342192+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.442122+0000 osd.3 (osd.3) 123 : cluster [DBG] 25.1a deep-scrub starts 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.444407+0000 osd.3 (osd.3) 124 : cluster [DBG] 25.1a deep-scrub ok 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.884132+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.886086+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.948091+0000 mon.a (mon.0) 1535 : audit [INF] from='client.53021 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.948229+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.948285+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.955524+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.956163+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.958352+0000 mon.a (mon.0) 1538 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:32.958605+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:33.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.072750+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:33.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:33.845 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:34.494 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:34.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:34.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:34.630 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:34.652 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:34.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.693+0000 7fdc333bd700 1 -- 172.21.15.107:0/3983580008 >> v1:172.21.15.107:6789/0 conn(0x7fdc2c0fc910 legacy=0x7fdc2c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.693+0000 7fdc333bd700 1 -- 172.21.15.107:0/3983580008 shutdown_connections 2024-02-10T08:38:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.693+0000 7fdc333bd700 1 -- 172.21.15.107:0/3983580008 >> 172.21.15.107:0/3983580008 conn(0x7fdc2c0f2510 msgr2=0x7fdc2c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.693+0000 7fdc333bd700 1 -- 172.21.15.107:0/3983580008 shutdown_connections 2024-02-10T08:38:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.693+0000 7fdc333bd700 1 -- 172.21.15.107:0/3983580008 wait complete. 2024-02-10T08:38:34.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 librados: starting msgr at 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 librados: starting objecter 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 Processor -- start 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 -- start start 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 librados: setting wanted keys 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 librados: calling monclient init 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc2c1173b0 con 0x7fdc2c103250 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc2c118590 con 0x7fdc2c0fc910 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc333bd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc2c119790 con 0x7fdc2c0ff710 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc3195a700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdc2c103250 0x7fdc2c115a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37724/0 (socket says 172.21.15.107:37724) 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc3195a700 1 -- 172.21.15.107:0/3032486899 learned_addr learned my addr 172.21.15.107:0/3032486899 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter ms_handle_connect 0x7fdc2c0ff710 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter ms_handle_connect 0x7fdc2c103250 2024-02-10T08:38:34.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter ms_handle_connect 0x7fdc2c0fc910 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840588452 0 0) 0x7fdc2c119790 con 0x7fdc2c0ff710 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc18003030 con 0x7fdc2c0ff710 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155694240 0 0) 0x7fdc2c1173b0 con 0x7fdc2c103250 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c119790 con 0x7fdc2c103250 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.694+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552760651 0 0) 0x7fdc2c118590 con 0x7fdc2c0fc910 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c1173b0 con 0x7fdc2c0fc910 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831242366 0 0) 0x7fdc18003030 con 0x7fdc2c0ff710 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc2c118590 con 0x7fdc2c0ff710 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc280019f0 con 0x7fdc2c0ff710 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3818260962 0 0) 0x7fdc2c119790 con 0x7fdc2c103250 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc18003030 con 0x7fdc2c103250 2024-02-10T08:38:34.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc24001bc0 con 0x7fdc2c103250 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961578138 0 0) 0x7fdc2c1173b0 con 0x7fdc2c0fc910 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc2c119790 con 0x7fdc2c0fc910 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc1c000fe0 con 0x7fdc2c0fc910 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 348359539 0 0) 0x7fdc2c118590 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 >> v1:172.21.15.195:6789/0 conn(0x7fdc2c0fc910 legacy=0x7fdc2c060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 >> v1:172.21.15.107:6789/0 conn(0x7fdc2c103250 legacy=0x7fdc2c115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c11a990 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c1175a0 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc333bd700 10 client.53033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c117a10 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc333bd700 1 librados: init done 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fdc280023a0 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.695+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc28003280 con 0x7fdc2c0ff710 2024-02-10T08:38:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.696+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc10004f20 con 0x7fdc2c0ff710 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.696+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fdc28001f00 con 0x7fdc2c0ff710 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(220..220 src has 50..220) v4 ==== 9189+0+0 (unknown 2335968722 0 0) 0x7fdc280769b0 con 0x7fdc2c0ff710 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 10 client.53033.objecter ms_dispatch 0x7fdc2c001550 osd_map(220..220 src has 50..220) v4 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 3 client.53033.objecter handle_osd_map got epochs [220,220] > 0 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 3 client.53033.objecter handle_osd_map decoding full epoch 220 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 20 client.53033.objecter dump_active .. 0 homeless 2024-02-10T08:38:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.697+0000 7fdc227fc700 10 client.53033.objecter ms_handle_connect 0x7fdc18063e20 2024-02-10T08:38:34.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.701+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdc28049960 con 0x7fdc2c0ff710 2024-02-10T08:38:34.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.742003+0000 osd.5 (osd.5) 117 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:32.744967+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.360207+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.363187+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.383982+0000 mgr.y (mgr.14553) 242 : cluster [DBG] pgmap v305: 282 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.471488+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.472451+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:38:34.783 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.952423+0000 mon.a (mon.0) 1540 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.952503+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.952583+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.959192+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.959391+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.107:0/3052214244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.959958+0000 mon.a (mon.0) 1543 : audit [INF] from='client.53027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.960262+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.960290+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:33.963217+0000 mon.a (mon.0) 1545 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.784 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.073743+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:34.785 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:34.804 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:34.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.742003+0000 osd.5 (osd.5) 117 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:32.744967+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.360207+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.363187+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.383982+0000 mgr.y (mgr.14553) 242 : cluster [DBG] pgmap v305: 282 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.471488+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.472451+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.952423+0000 mon.a (mon.0) 1540 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.952503+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.952583+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-10T08:38:34.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.959192+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.959391+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.107:0/3052214244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.959958+0000 mon.a (mon.0) 1543 : audit [INF] from='client.53027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.960262+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.960290+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:33.963217+0000 mon.a (mon.0) 1545 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.073743+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.742003+0000 osd.5 (osd.5) 117 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:32.744967+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.360207+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.363187+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.383982+0000 mgr.y (mgr.14553) 242 : cluster [DBG] pgmap v305: 282 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:34.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.471488+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.472451+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.952423+0000 mon.a (mon.0) 1540 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.952503+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.952583+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.959192+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.959391+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.107:0/3052214244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.959958+0000 mon.a (mon.0) 1543 : audit [INF] from='client.53027 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.960262+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.107:0/241311821' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.960290+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:33.963217+0000 mon.a (mon.0) 1545 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]: dispatch 2024-02-10T08:38:34.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.073743+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:34.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.883+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fdc100051d0 con 0x7fdc2c0ff710 2024-02-10T08:38:34.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.957+0000 7fdc227fc700 1 -- 172.21.15.107:0/3032486899 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v221) v1 ==== 87+0+0 (unknown 653434284 0 0) 0x7fdc28047d70 con 0x7fdc2c0ff710 2024-02-10T08:38:34.964 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:34.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 10 librados: watch_flush enter 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 10 librados: watch_flush exit 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 20 client.53033.objecter shutdown clearing up homeless session... 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 10 client.53033.objecter successfully canceled tick 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 >> v1:172.21.15.107:6800/2634842009 conn(0x7fdc18063e20 legacy=0x7fdc180662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 >> v1:172.21.15.107:6790/0 conn(0x7fdc2c0ff710 legacy=0x7fdc2c061390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 shutdown_connections 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.967+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 >> 172.21.15.107:0/3032486899 conn(0x7fdc2c0f2510 msgr2=0x7fdc2c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.968+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 shutdown_connections 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.968+0000 7fdc333bd700 1 -- 172.21.15.107:0/3032486899 wait complete. 2024-02-10T08:38:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:34.968+0000 7fdc333bd700 1 librados: shutdown 2024-02-10T08:38:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:38:35.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:35.094 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:35.113 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:35.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:35.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:35.322 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:35.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:35.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:35.454 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:35.552 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:35.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.917727+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:33.920626+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:34.375736+0000 osd.6 (osd.6) 95 : cluster [DBG] 16.d scrub starts 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:34.376756+0000 osd.6 (osd.6) 96 : cluster [DBG] 16.d scrub ok 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.884512+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.107:0/3032486899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.884808+0000 mon.a (mon.0) 1546 : audit [INF] from='client.53033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:34.950394+0000 mon.a (mon.0) 1547 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:34.950430+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.954314+0000 mon.a (mon.0) 1549 : audit [INF] from='client.53027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.954389+0000 mon.a (mon.0) 1550 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]': finished 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.954466+0000 mon.a (mon.0) 1551 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.954519+0000 mon.a (mon.0) 1552 : audit [INF] from='client.53033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:34.954569+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.987205+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.987554+0000 mon.a (mon.0) 1554 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.988747+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.917727+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:33.920626+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:34.375736+0000 osd.6 (osd.6) 95 : cluster [DBG] 16.d scrub starts 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:34.376756+0000 osd.6 (osd.6) 96 : cluster [DBG] 16.d scrub ok 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.884512+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.107:0/3032486899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.884808+0000 mon.a (mon.0) 1546 : audit [INF] from='client.53033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:34.950394+0000 mon.a (mon.0) 1547 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:34.950430+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:35.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.954314+0000 mon.a (mon.0) 1549 : audit [INF] from='client.53027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.954389+0000 mon.a (mon.0) 1550 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]': finished 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.954466+0000 mon.a (mon.0) 1551 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.954519+0000 mon.a (mon.0) 1552 : audit [INF] from='client.53033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:34.954569+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.987205+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.987554+0000 mon.a (mon.0) 1554 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.988747+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.988962+0000 mon.a (mon.0) 1555 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.989408+0000 mon.b (mon.1) 339 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:34.989698+0000 mon.a (mon.0) 1556 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.014217+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.074583+0000 mon.b (mon.1) 341 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:35.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.249446+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.988962+0000 mon.a (mon.0) 1555 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.989408+0000 mon.b (mon.1) 339 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:34.989698+0000 mon.a (mon.0) 1556 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.014217+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.074583+0000 mon.b (mon.1) 341 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:35.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.249446+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:35.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:35.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:35.686 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.917727+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:38:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:33.920626+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:38:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:34.375736+0000 osd.6 (osd.6) 95 : cluster [DBG] 16.d scrub starts 2024-02-10T08:38:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:34.376756+0000 osd.6 (osd.6) 96 : cluster [DBG] 16.d scrub ok 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.884512+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.107:0/3032486899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.884808+0000 mon.a (mon.0) 1546 : audit [INF] from='client.53033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:34.950394+0000 mon.a (mon.0) 1547 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:34.950430+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.954314+0000 mon.a (mon.0) 1549 : audit [INF] from='client.53027 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.954389+0000 mon.a (mon.0) 1550 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-21", "mode": "writeback"}]': finished 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.954466+0000 mon.a (mon.0) 1551 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi107-112403-31"}]': finished 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.954519+0000 mon.a (mon.0) 1552 : audit [INF] from='client.53033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:34.954569+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.987205+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.987554+0000 mon.a (mon.0) 1554 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.988747+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.988962+0000 mon.a (mon.0) 1555 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:35.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.989408+0000 mon.b (mon.1) 339 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:34.989698+0000 mon.a (mon.0) 1556 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:35.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.014217+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.074583+0000 mon.b (mon.1) 341 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:35.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.249446+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:35] "GET /metrics HTTP/1.1" 200 190122 "" "Prometheus/2.33.4" 2024-02-10T08:38:36.534 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:35.384535+0000 mgr.y (mgr.14553) 243 : cluster [DBG] pgmap v308: 314 pgs: 64 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.944551+0000 mon.a (mon.0) 1558 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.944647+0000 mon.a (mon.0) 1559 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:35.944687+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.956485+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.956987+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.961187+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.107:0/2054489822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.961933+0000 mon.a (mon.0) 1561 : audit [INF] from='client.53042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.964011+0000 mon.a (mon.0) 1562 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:35.964423+0000 mon.a (mon.0) 1563 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:36.075750+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:35.384535+0000 mgr.y (mgr.14553) 243 : cluster [DBG] pgmap v308: 314 pgs: 64 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.944551+0000 mon.a (mon.0) 1558 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.944647+0000 mon.a (mon.0) 1559 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:35.944687+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.956485+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.956987+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.961187+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.107:0/2054489822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.961933+0000 mon.a (mon.0) 1561 : audit [INF] from='client.53042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.964011+0000 mon.a (mon.0) 1562 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:35.964423+0000 mon.a (mon.0) 1563 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:36.075750+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:36.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:36.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:36.768 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:36.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:35.384535+0000 mgr.y (mgr.14553) 243 : cluster [DBG] pgmap v308: 314 pgs: 64 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 201 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.944551+0000 mon.a (mon.0) 1558 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi107-112403-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.944647+0000 mon.a (mon.0) 1559 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:35.944687+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.956485+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.956987+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.961187+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.107:0/2054489822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.961933+0000 mon.a (mon.0) 1561 : audit [INF] from='client.53042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:36.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.964011+0000 mon.a (mon.0) 1562 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:36.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:35.964423+0000 mon.a (mon.0) 1563 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]: dispatch 2024-02-10T08:38:36.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:36.075750+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:36.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:36.900 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:36.958 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:36.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:37.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:37.086 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:37.086 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:37.086 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:37.086 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 -- 172.21.15.107:0/66893228 >> v1:172.21.15.107:6790/0 conn(0x7f23500f9570 legacy=0x7f2350102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 -- 172.21.15.107:0/66893228 shutdown_connections 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 -- 172.21.15.107:0/66893228 >> 172.21.15.107:0/66893228 conn(0x7f23500f2510 msgr2=0x7f23500f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 -- 172.21.15.107:0/66893228 shutdown_connections 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 -- 172.21.15.107:0/66893228 wait complete. 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 librados: starting msgr at 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 librados: starting objecter 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.984+0000 7f235502a700 1 Processor -- start 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 -- start start 2024-02-10T08:38:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 librados: setting wanted keys 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 librados: calling monclient init 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2350102b70 con 0x7f2350104350 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2350102d40 con 0x7f23500f9570 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f235502a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2350102f30 con 0x7f23500f89c0 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f234ed9d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f23500f89c0 0x7f23501023d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45180/0 (socket says 172.21.15.107:45180) 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f234ed9d700 1 -- 172.21.15.107:0/1814751143 learned_addr learned my addr 172.21.15.107:0/1814751143 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter ms_handle_connect 0x7f23500f89c0 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter ms_handle_connect 0x7f2350104350 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter ms_handle_connect 0x7f23500f9570 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114234709 0 0) 0x7f2350102f30 con 0x7f23500f89c0 2024-02-10T08:38:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2338002220 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453473173 0 0) 0x7f2350102d40 con 0x7f23500f9570 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2350102f30 con 0x7f23500f9570 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091719379 0 0) 0x7f2338002220 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2350102d40 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.985+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2340001940 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1079914847 0 0) 0x7f2350102f30 con 0x7f23500f9570 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2338002220 con 0x7f23500f9570 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2344001e50 con 0x7f23500f9570 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4125663303 0 0) 0x7f2350102d40 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.195:6789/0 conn(0x7f23500f9570 legacy=0x7f2350114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6789/0 conn(0x7f2350104350 legacy=0x7f23501184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235011cc20 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2350119c50 con 0x7f23500f89c0 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f235502a700 10 client.53048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f235011a0c0 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f235502a700 1 librados: init done 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2340001cd0 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.986+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2340005ca0 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.987+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f233c004f00 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.987+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2340002270 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(223..223 src has 50..223) v4 ==== 9862+0+0 (unknown 519197619 0 0) 0x7f2340079570 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 osd_map(223..223 src has 50..223) v4 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 3 client.53048.objecter handle_osd_map got epochs [223,223] > 0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 3 client.53048.objecter handle_osd_map decoding full epoch 223 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 20 client.53048.objecter dump_active .. 0 homeless 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v223) v1 ==== 54+0+16 (unknown 1626880078 0 2420000192) 0x7f234007b980 con 0x7f23500f89c0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f2338063ad0 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 10 client.53048.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 1 2024-02-10T08:38:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 1 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 0 1 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 4 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 10 client.53048.objecter _send_command 1 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c004950 con 0x7f233c004530 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.988+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c004530 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.989+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c004950 con 0x7f233c004530 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.989+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.989+0000 7f2337fff700 10 client.53048.objecter _finish_command 1 = system:0 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.989+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 4 2024-02-10T08:38:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:36.989+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 0 1 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 10 client.53048.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 2 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 2 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c0051a0 osd=0 3 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 0 2 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 4 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 10 client.53048.objecter _send_command 2 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.004+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c004d30 con 0x7f233c004530 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c004d30 con 0x7f233c004530 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f2337fff700 10 client.53048.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 4 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 0 2 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.005+0000 7f235502a700 10 client.53048.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 1 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 4 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 10 client.53048.objecter _send_command 3 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c00c000 con 0x7f233c004150 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c004150 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c00c000 con 0x7f233c004150 2024-02-10T08:38:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 10 client.53048.objecter _finish_command 3 = system:0 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.006+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 1 3 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 10 client.53048.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003fe0 osd=1 3 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 1 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 10 client.53048.objecter _send_command 4 2024-02-10T08:38:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.021+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c0018b0 con 0x7f233c004150 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.022+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c0018b0 con 0x7f233c004150 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.022+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.022+0000 7f2337fff700 10 client.53048.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.022+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 4 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.022+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 1 4 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 10 client.53048.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 5 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 5 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 2 5 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 4 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 10 client.53048.objecter _send_command 5 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c003e70 con 0x7f233c00d2f0 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.023+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c00d2f0 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.024+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c003e70 con 0x7f233c00d2f0 2024-02-10T08:38:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.024+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.024+0000 7f2337fff700 10 client.53048.objecter _finish_command 5 = system:0 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.024+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 4 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.024+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 2 5 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.038+0000 7f235502a700 10 client.53048.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.038+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 6 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.038+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 6 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003630 osd=2 3 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 2 6 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 4 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 10 client.53048.objecter _send_command 6 2024-02-10T08:38:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.039+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c00e730 con 0x7f233c00d2f0 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c00e730 con 0x7f233c00d2f0 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f2337fff700 10 client.53048.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 4 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 2 6 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 10 client.53048.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 7 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 7 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 3 7 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 4 2024-02-10T08:38:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 10 client.53048.objecter _send_command 7 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.041+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c015a60 con 0x7f233c011e80 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c011e80 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c015a60 con 0x7f233c011e80 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 10 client.53048.objecter _finish_command 7 = system:0 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 4 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.042+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 3 7 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 10 client.53048.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 8 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 8 2024-02-10T08:38:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c003410 osd=3 3 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 3 8 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 4 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 10 client.53048.objecter _send_command 8 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.057+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c015a60 con 0x7f233c011e80 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c015a60 con 0x7f233c011e80 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f2337fff700 10 client.53048.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 4 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 3 8 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 10 client.53048.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 9 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 9 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 4 9 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 4 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 10 client.53048.objecter _send_command 9 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.059+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c00cf20 con 0x7f233c00c980 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c00c980 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c00cf20 con 0x7f233c00c980 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 10 client.53048.objecter _finish_command 9 = system:0 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 4 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.060+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 4 9 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 10 client.53048.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 10 2024-02-10T08:38:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 10 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c015a60 osd=4 3 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 4 10 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 4 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 10 client.53048.objecter _send_command 10 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.076+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c00d160 con 0x7f233c00c980 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.077+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c00d160 con 0x7f233c00c980 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.077+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.077+0000 7f2337fff700 10 client.53048.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.077+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 4 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.077+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 4 10 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 10 client.53048.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 11 2024-02-10T08:38:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 11 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 5 11 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 4 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 10 client.53048.objecter _send_command 11 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c017df0 con 0x7f233c0179a0 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.078+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c0179a0 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.079+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c017df0 con 0x7f233c0179a0 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.079+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.079+0000 7f2337fff700 10 client.53048.objecter _finish_command 11 = system:0 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.079+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 4 2024-02-10T08:38:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.079+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 5 11 2024-02-10T08:38:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 10 client.53048.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 12 2024-02-10T08:38:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 12 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c00d160 osd=5 3 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 5 12 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 4 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 10 client.53048.objecter _send_command 12 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.096+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c017df0 con 0x7f233c0179a0 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c017df0 con 0x7f233c0179a0 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f2337fff700 10 client.53048.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 4 2024-02-10T08:38:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 5 12 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 10 client.53048.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 13 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 13 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 6 13 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 4 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 10 client.53048.objecter _send_command 13 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.098+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c022b70 con 0x7f233c01f060 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c01f060 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c022b70 con 0x7f233c01f060 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 10 client.53048.objecter _finish_command 13 = system:0 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 4 2024-02-10T08:38:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.099+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 6 13 2024-02-10T08:38:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 10 client.53048.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 14 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 14 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c017df0 osd=6 3 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 6 14 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 4 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 10 client.53048.objecter _send_command 14 2024-02-10T08:38:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.117+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c022b70 con 0x7f233c01f060 2024-02-10T08:38:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c022b70 con 0x7f233c01f060 2024-02-10T08:38:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f2337fff700 10 client.53048.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 4 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 6 14 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 10 client.53048.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 15 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 15 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 7 15 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 4 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 10 client.53048.objecter _send_command 15 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.119+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c026e60 con 0x7f233c0231f0 2024-02-10T08:38:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.120+0000 7f2337fff700 10 client.53048.objecter ms_handle_connect 0x7f233c0231f0 2024-02-10T08:38:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.121+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c026e60 con 0x7f233c0231f0 2024-02-10T08:38:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.121+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.121+0000 7f2337fff700 10 client.53048.objecter _finish_command 15 = system:0 2024-02-10T08:38:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.121+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 4 2024-02-10T08:38:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.121+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 7 15 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 10 client.53048.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign -1 16 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 20 client.53048.objecter _get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 15 client.53048.objecter _session_command_op_remove -1 16 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 20 client.53048.objecter get_session s=0x7f233c022b70 osd=7 3 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 15 client.53048.objecter _session_command_op_assign 7 16 2024-02-10T08:38:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 4 2024-02-10T08:38:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 10 client.53048.objecter _send_command 16 2024-02-10T08:38:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.135+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f233c028610 con 0x7f233c0231f0 2024-02-10T08:38:37.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f2337fff700 1 -- 172.21.15.107:0/1814751143 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f233c028610 con 0x7f233c0231f0 2024-02-10T08:38:37.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f2337fff700 10 client.53048.objecter ms_dispatch 0x7f2350001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:37.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f2337fff700 10 client.53048.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f2337fff700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 4 2024-02-10T08:38:37.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f2337fff700 15 client.53048.objecter _session_command_op_remove 7 16 2024-02-10T08:38:37.141 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 librados: watch_flush enter 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 librados: watch_flush exit 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.0 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6801/3321596972 conn(0x7f233c004530 legacy=0x7f233c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c0051a0 osd=0 1 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.1 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6805/133516126 conn(0x7f233c004150 legacy=0x7f233c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003fe0 osd=1 1 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.2 2024-02-10T08:38:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6809/3575881728 conn(0x7f233c00d2f0 legacy=0x7f233c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003630 osd=2 1 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.3 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6813/2871825691 conn(0x7f233c011e80 legacy=0x7f233c014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f234f59e700 1 -- 172.21.15.107:0/1814751143 reap_dead start 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c003410 osd=3 1 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.4 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.195:6800/2499671285 conn(0x7f233c00c980 legacy=0x7f233c015c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f234f59e700 1 -- 172.21.15.107:0/1814751143 reap_dead start 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c015a60 osd=4 1 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.5 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.195:6804/4178321486 conn(0x7f233c0179a0 legacy=0x7f233c01d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c00d160 osd=5 1 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 10 client.53048.objecter close_session for osd.6 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.195:6808/1417479017 conn(0x7f233c01f060 legacy=0x7f233c021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.137+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c017df0 osd=6 1 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 10 client.53048.objecter close_session for osd.7 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.195:6812/1934176928 conn(0x7f233c0231f0 legacy=0x7f233c0255e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 20 client.53048.objecter put_session s=0x7f233c022b70 osd=7 1 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 20 client.53048.objecter shutdown clearing up homeless session... 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 10 client.53048.objecter successfully canceled tick 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2338063ad0 legacy=0x7f2338065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> v1:172.21.15.107:6790/0 conn(0x7f23500f89c0 legacy=0x7f23501023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f234f59e700 1 -- 172.21.15.107:0/1814751143 reap_dead start 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 shutdown_connections 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 >> 172.21.15.107:0/1814751143 conn(0x7f23500f2510 msgr2=0x7f23501077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 shutdown_connections 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 -- 172.21.15.107:0/1814751143 wait complete. 2024-02-10T08:38:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:37.138+0000 7f235502a700 1 librados: shutdown 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:38:37.253 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:38:37.254 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:38:37.696 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:38:37.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:38:37.697 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-10T08:38:37.697 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:38:37.697 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:38:37.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:36.942888+0000 mon.a (mon.0) 1564 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:36.948619+0000 mon.a (mon.0) 1565 : audit [INF] from='client.53042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:36.988412+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.107:0/1814751143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:37.076913+0000 mon.b (mon.1) 345 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:37.198815+0000 mon.a (mon.0) 1566 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:37.727 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:37.198878+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-10T08:38:37.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:36.942888+0000 mon.a (mon.0) 1564 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:37.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:36.948619+0000 mon.a (mon.0) 1565 : audit [INF] from='client.53042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:37.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:36.988412+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.107:0/1814751143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:37.076913+0000 mon.b (mon.1) 345 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:37.198815+0000 mon.a (mon.0) 1566 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:37.198878+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:36.942888+0000 mon.a (mon.0) 1564 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:36.948619+0000 mon.a (mon.0) 1565 : audit [INF] from='client.53042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi107-112411-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:36.988412+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.107:0/1814751143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:37.076913+0000 mon.b (mon.1) 345 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:37.198815+0000 mon.a (mon.0) 1566 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-21"}]': finished 2024-02-10T08:38:37.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:37.198878+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-10T08:38:37.989 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:38:38.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:38:38.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:38:38.145 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:38.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:38.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:38.285 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:38.304 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:38.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:38.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:38.434 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:37.385119+0000 mgr.y (mgr.14553) 244 : cluster [DBG] pgmap v311: 346 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 299 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:38:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:37.953129+0000 mon.a (mon.0) 1568 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:37.953256+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-10T08:38:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:38.077902+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:38.638 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:38.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:38.738 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:38.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:38.761 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:38.771 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:38.870 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:38.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:37.385119+0000 mgr.y (mgr.14553) 244 : cluster [DBG] pgmap v311: 346 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 299 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:37.953129+0000 mon.a (mon.0) 1568 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:37.953256+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:38.077902+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:37.385119+0000 mgr.y (mgr.14553) 244 : cluster [DBG] pgmap v311: 346 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 299 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:37.953129+0000 mon.a (mon.0) 1568 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi107-112403-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:37.953256+0000 mon.a (mon.0) 1569 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-10T08:38:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:38.077902+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:38.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:39.004 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:39.452 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:39.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:39.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:39.582 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:39.655 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:39.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:39.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:39.786 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:39.806 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:39.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:39.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:39.920 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:38:39.920 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-02-10T08:38:39.920 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:38:39.938 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:39.957 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:39.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:40.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:40.092 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:38.957232+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:38.965091+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:38.966854+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:39.078997+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:39.385541+0000 mgr.y (mgr.14553) 245 : cluster [DBG] pgmap v314: 290 pgs: 40 unknown, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:38.957232+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-10T08:38:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:38.965091+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:38.966854+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:39.078997+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:39.385541+0000 mgr.y (mgr.14553) 245 : cluster [DBG] pgmap v314: 290 pgs: 40 unknown, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:38:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:38.957232+0000 mon.a (mon.0) 1570 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-10T08:38:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:38.965091+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:38.966854+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:39.078997+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:39.385541+0000 mgr.y (mgr.14553) 245 : cluster [DBG] pgmap v314: 290 pgs: 40 unknown, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:38:40.291 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:40.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:40.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:40.422 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:40.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:40.527 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:40.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:40.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:38:40.653 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:40.672 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:40.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:40.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:40.803 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:40.820 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:40.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:40.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:40.956 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:41.157 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:41.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:41.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:41.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:39.964740+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:41.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:39.964843+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:39.967939+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:39.985472+0000 mon.a (mon.0) 1574 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.079987+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.248411+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:40.560147+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:40.560371+0000 osd.5 (osd.5) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.944751+0000 mon.a (mon.0) 1576 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:40.944893+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.946616+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.949747+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.959525+0000 mon.a (mon.0) 1579 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.964890+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:40.965270+0000 mon.a (mon.0) 1580 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:39.964740+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:39.964843+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:39.967939+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:39.985472+0000 mon.a (mon.0) 1574 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.079987+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.248411+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:40.560147+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:40.560371+0000 osd.5 (osd.5) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.944751+0000 mon.a (mon.0) 1576 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:40.944893+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.946616+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.949747+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.959525+0000 mon.a (mon.0) 1579 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.964890+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:40.965270+0000 mon.a (mon.0) 1580 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:39.964740+0000 mon.a (mon.0) 1572 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:39.964843+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:39.967939+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:39.985472+0000 mon.a (mon.0) 1574 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.079987+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.248411+0000 mon.a (mon.0) 1575 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:40.560147+0000 osd.5 (osd.5) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:40.560371+0000 osd.5 (osd.5) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.944751+0000 mon.a (mon.0) 1576 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:41.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:40.944893+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-10T08:38:41.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.946616+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:41.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.949747+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.107:0/1988965652' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.959525+0000 mon.a (mon.0) 1579 : audit [INF] from='client.47758 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]: dispatch 2024-02-10T08:38:41.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.964890+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:40.965270+0000 mon.a (mon.0) 1580 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:41.391 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:41.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:41.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:41.521 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:41.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:41.558 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:38:41.558 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:38:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:41.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:41.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:41.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.080849+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:41.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.264269+0000 mgr.y (mgr.14553) 246 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:41.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:41.386522+0000 mgr.y (mgr.14553) 247 : cluster [DBG] pgmap v317: 282 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.948557+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.948654+0000 mon.a (mon.0) 1582 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.948735+0000 mon.a (mon.0) 1583 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:41.948786+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.949704+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.950129+0000 mon.a (mon.0) 1585 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.080849+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.264269+0000 mgr.y (mgr.14553) 246 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:41.386522+0000 mgr.y (mgr.14553) 247 : cluster [DBG] pgmap v317: 282 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-02-10T08:38:41.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.948557+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:41.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.948654+0000 mon.a (mon.0) 1582 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:41.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.948735+0000 mon.a (mon.0) 1583 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:41.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:41.948786+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-10T08:38:41.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.949704+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:41.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.950129+0000 mon.a (mon.0) 1585 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:42.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.080849+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.264269+0000 mgr.y (mgr.14553) 246 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:41.386522+0000 mgr.y (mgr.14553) 247 : cluster [DBG] pgmap v317: 282 pgs: 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.948557+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.107:0/688128758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.948654+0000 mon.a (mon.0) 1582 : audit [INF] from='client.47758 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi107-112403-32"}]': finished 2024-02-10T08:38:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.948735+0000 mon.a (mon.0) 1583 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:41.948786+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-10T08:38:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.949704+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.950129+0000 mon.a (mon.0) 1585 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:42.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:42.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:42.518 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:42.539 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:42.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:42.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:42.672 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:42.745 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:42.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:42.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:42.880 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:42.900 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:42.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:43.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.129 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:43.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:41.988465+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.081820+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.232735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.233217+0000 mon.c (mon.2) 268 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.233472+0000 mon.a (mon.0) 1587 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.233828+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.234068+0000 mon.a (mon.0) 1588 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.952342+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.952470+0000 mon.a (mon.0) 1590 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:42.952545+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.953214+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.953818+0000 mon.a (mon.0) 1592 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.959821+0000 mon.c (mon.2) 271 : audit [INF] from='client.? v1:172.21.15.107:0/2385614971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.960271+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.960490+0000 mon.a (mon.0) 1593 : audit [INF] from='client.53060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:42.963455+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.261 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.281 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:41.988465+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.081820+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.232735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.233217+0000 mon.c (mon.2) 268 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.233472+0000 mon.a (mon.0) 1587 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.233828+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.234068+0000 mon.a (mon.0) 1588 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.952342+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.952470+0000 mon.a (mon.0) 1590 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:42.952545+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-10T08:38:43.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.953214+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.953818+0000 mon.a (mon.0) 1592 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:41.988465+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.081820+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.232735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.233217+0000 mon.c (mon.2) 268 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.233472+0000 mon.a (mon.0) 1587 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.233828+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.234068+0000 mon.a (mon.0) 1588 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.952342+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.952470+0000 mon.a (mon.0) 1590 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi107-112403-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:42.952545+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-10T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.953214+0000 mon.c (mon.2) 270 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.953818+0000 mon.a (mon.0) 1592 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.959821+0000 mon.c (mon.2) 271 : audit [INF] from='client.? v1:172.21.15.107:0/2385614971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.960271+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.960490+0000 mon.a (mon.0) 1593 : audit [INF] from='client.53060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:42.963455+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.959821+0000 mon.c (mon.2) 271 : audit [INF] from='client.? v1:172.21.15.107:0/2385614971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.960271+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.960490+0000 mon.a (mon.0) 1593 : audit [INF] from='client.53060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:43.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:42.963455+0000 mon.a (mon.0) 1594 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]: dispatch 2024-02-10T08:38:43.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:43.413 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.431 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:43.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:43.563 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.581 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:43.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:43.713 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.732 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 -- 172.21.15.107:0/810360335 >> v1:172.21.15.107:6790/0 conn(0x7f8c8c103250 legacy=0x7f8c8c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 -- 172.21.15.107:0/810360335 shutdown_connections 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 -- 172.21.15.107:0/810360335 >> 172.21.15.107:0/810360335 conn(0x7f8c8c0f2510 msgr2=0x7f8c8c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 -- 172.21.15.107:0/810360335 shutdown_connections 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 -- 172.21.15.107:0/810360335 wait complete. 2024-02-10T08:38:43.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 librados: starting msgr at 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.852+0000 7f8c9291d700 1 librados: starting objecter 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 Processor -- start 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 -- start start 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 librados: setting wanted keys 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 librados: calling monclient init 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c116470 con 0x7f8c8c0ff710 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c117670 con 0x7f8c8c103250 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c9291d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c118870 con 0x7f8c8c0fc910 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c8bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8c8c0fc910 0x7f8c8c0f8c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57850/0 (socket says 172.21.15.107:57850) 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c8bfff700 1 -- 172.21.15.107:0/1000076357 learned_addr learned my addr 172.21.15.107:0/1000076357 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c0fc910 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c0ff710 2024-02-10T08:38:43.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119080533 0 0) 0x7f8c8c118870 con 0x7f8c8c0fc910 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c740033f0 con 0x7f8c8c0fc910 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.853+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c103250 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758681102 0 0) 0x7f8c8c116470 con 0x7f8c8c0ff710 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c118870 con 0x7f8c8c0ff710 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134938420 0 0) 0x7f8c8c117670 con 0x7f8c8c103250 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c116470 con 0x7f8c8c103250 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1764828650 0 0) 0x7f8c740033f0 con 0x7f8c8c0fc910 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c8c117670 con 0x7f8c8c0fc910 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8c7c001980 con 0x7f8c8c0fc910 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 149886066 0 0) 0x7f8c8c118870 con 0x7f8c8c0ff710 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c740033f0 con 0x7f8c8c0ff710 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8c800018c0 con 0x7f8c8c0ff710 2024-02-10T08:38:43.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019015993 0 0) 0x7f8c8c116470 con 0x7f8c8c103250 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c8c118870 con 0x7f8c8c103250 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8c84002240 con 0x7f8c8c103250 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 895409908 0 0) 0x7f8c8c117670 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 >> v1:172.21.15.195:6789/0 conn(0x7f8c8c103250 legacy=0x7f8c8c114b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 >> v1:172.21.15.107:6789/0 conn(0x7f8c8c0ff710 legacy=0x7f8c8c0f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c119a70 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c116660 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c9291d700 10 client.53072.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c116b50 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c9291d700 1 librados: init done 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8c7c002100 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.854+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8c7c005ae0 con 0x7f8c8c0fc910 2024-02-10T08:38:43.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.855+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78004f20 con 0x7f8c8c0fc910 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.855+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8c7c0028a0 con 0x7f8c8c0fc910 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(228..228 src has 50..228) v4 ==== 8699+0+0 (unknown 2599730802 0 0) 0x7f8c7c079090 con 0x7f8c8c0fc910 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 10 client.53072.objecter ms_dispatch 0x7f8c8c001550 osd_map(228..228 src has 50..228) v4 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 3 client.53072.objecter handle_osd_map got epochs [228,228] > 0 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 3 client.53072.objecter handle_osd_map decoding full epoch 228 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 20 client.53072.objecter dump_active .. 0 homeless 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.856+0000 7f8c897fa700 10 client.53072.objecter ms_handle_connect 0x7f8c74063d00 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:42.860+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8c7c04c220 con 0x7f8c8c0fc910 2024-02-10T08:38:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:43.055+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8c780051d0 con 0x7f8c8c0fc910 2024-02-10T08:38:43.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:43.862 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:43.881 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:43.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:43.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:44.016 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:44.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:44.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:44.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.209+0000 7f8c897fa700 1 -- 172.21.15.107:0/1000076357 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v230) v1 ==== 97+0+0 (unknown 2025799069 0 0) 0x7f8c7c04a630 con 0x7f8c8c0fc910 2024-02-10T08:38:44.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.212+0000 7f8c9291d700 10 librados: watch_flush enter 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.212+0000 7f8c9291d700 10 librados: watch_flush exit 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.212+0000 7f8c9291d700 20 client.53072.objecter shutdown clearing up homeless session... 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.212+0000 7f8c9291d700 10 client.53072.objecter successfully canceled tick 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.212+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8c74063d00 legacy=0x7f8c740661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.213+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 >> v1:172.21.15.107:6790/0 conn(0x7f8c8c0fc910 legacy=0x7f8c8c0f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.213+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 shutdown_connections 2024-02-10T08:38:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.213+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 >> 172.21.15.107:0/1000076357 conn(0x7f8c8c0f2510 msgr2=0x7f8c8c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.214+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 shutdown_connections 2024-02-10T08:38:44.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.214+0000 7f8c9291d700 1 -- 172.21.15.107:0/1000076357 wait complete. 2024-02-10T08:38:44.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.218+0000 7f8c9291d700 1 librados: shutdown 2024-02-10T08:38:44.223 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:43.055935+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.107:0/1000076357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:43.056252+0000 mon.a (mon.0) 1595 : audit [INF] from='client.53072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:43.082825+0000 mon.b (mon.1) 357 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:43.387048+0000 mgr.y (mgr.14553) 248 : cluster [DBG] pgmap v320: 314 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:43.946583+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:44.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:43.946614+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:43.055935+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.107:0/1000076357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:43.056252+0000 mon.a (mon.0) 1595 : audit [INF] from='client.53072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:43.082825+0000 mon.b (mon.1) 357 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:43.387048+0000 mgr.y (mgr.14553) 248 : cluster [DBG] pgmap v320: 314 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:43.946583+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:44.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:43.946614+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.277+0000 7f994aaaf700 1 -- 172.21.15.107:0/967196528 >> v1:172.21.15.107:6790/0 conn(0x7f99440fda50 legacy=0x7f99440fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.277+0000 7f994aaaf700 1 -- 172.21.15.107:0/967196528 shutdown_connections 2024-02-10T08:38:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.277+0000 7f994aaaf700 1 -- 172.21.15.107:0/967196528 >> 172.21.15.107:0/967196528 conn(0x7f99440f3640 msgr2=0x7f99440f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.277+0000 7f994aaaf700 1 -- 172.21.15.107:0/967196528 shutdown_connections 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.277+0000 7f994aaaf700 1 -- 172.21.15.107:0/967196528 wait complete. 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 librados: starting msgr at 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 librados: starting objecter 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 Processor -- start 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 -- start start 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 librados: setting wanted keys 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 librados: calling monclient init 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99441174a0 con 0x7f99440fda50 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9944118680 con 0x7f9944104330 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994aaaf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9944119880 con 0x7f99441007f0 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.278+0000 7f994884b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f99440fda50 0x7f99440f9c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59834/0 (socket says 172.21.15.107:59834) 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f994884b700 1 -- 172.21.15.107:0/2979270354 learned_addr learned my addr 172.21.15.107:0/2979270354 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter ms_handle_connect 0x7f99440fda50 2024-02-10T08:38:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter ms_handle_connect 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter ms_handle_connect 0x7f9944104330 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522914616 0 0) 0x7f9944119880 con 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9934003630 con 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716965630 0 0) 0x7f9944118680 con 0x7f9944104330 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9944119880 con 0x7f9944104330 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057875277 0 0) 0x7f9934003630 con 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9944118680 con 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9930001730 con 0x7f99441007f0 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273150455 0 0) 0x7f9944119880 con 0x7f9944104330 2024-02-10T08:38:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9934003630 con 0x7f9944104330 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f993c001bc0 con 0x7f9944104330 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1758939141 0 0) 0x7f9944118680 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.195:6789/0 conn(0x7f9944104330 legacy=0x7f9944115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6789/0 conn(0x7f99440fda50 legacy=0x7f99440f9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994411aa80 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9944119a70 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9930002240 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9930003180 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f994aaaf700 10 client.53084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.279+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9944119f60 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.280+0000 7f994aaaf700 1 librados: init done 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.280+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f994405f740 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9930001da0 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(230..230 src has 50..230) v4 ==== 9475+0+0 (unknown 1107888448 0 0) 0x7f9930076910 con 0x7f99441007f0 2024-02-10T08:38:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 osd_map(230..230 src has 50..230) v4 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 3 client.53084.objecter handle_osd_map got epochs [230,230] > 0 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 3 client.53084.objecter handle_osd_map decoding full epoch 230 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 20 client.53084.objecter dump_active .. 0 homeless 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v230) v1 ==== 54+0+16 (unknown 3422155502 0 2420000192) 0x7f9930076ce0 con 0x7f99441007f0 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.281+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9934063d40 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 10 client.53084.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 1 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 1 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 0 1 2024-02-10T08:38:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 4 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 10 client.53084.objecter _send_command 1 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99440475e0 con 0x7f9944118810 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9944118810 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f99440475e0 con 0x7f9944118810 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 10 client.53084.objecter _finish_command 1 = system:0 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 4 2024-02-10T08:38:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.282+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 0 1 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 10 client.53084.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 2 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 2 2024-02-10T08:38:44.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99440fd450 osd=0 3 2024-02-10T08:38:44.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 0 2 2024-02-10T08:38:44.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 4 2024-02-10T08:38:44.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 10 client.53084.objecter _send_command 2 2024-02-10T08:38:44.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.307+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f994410a050 con 0x7f9944118810 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.308+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f994410a050 con 0x7f9944118810 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.308+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.308+0000 7f9941f76700 10 client.53084.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.308+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 4 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.308+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 0 2 2024-02-10T08:38:44.309 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 10 client.53084.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 3 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 3 2024-02-10T08:38:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 1 3 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 4 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 10 client.53084.objecter _send_command 3 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99440f44e0 con 0x7f99440f40a0 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.309+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f99440f40a0 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.310+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f99440f44e0 con 0x7f99440f40a0 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.310+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.310+0000 7f9941f76700 10 client.53084.objecter _finish_command 3 = system:0 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.310+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 4 2024-02-10T08:38:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.310+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 1 3 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 10 client.53084.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 4 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 4 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99440f3a20 osd=1 3 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 1 4 2024-02-10T08:38:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 4 2024-02-10T08:38:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 10 client.53084.objecter _send_command 4 2024-02-10T08:38:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.334+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f99440f48c0 con 0x7f99440f40a0 2024-02-10T08:38:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f99440f48c0 con 0x7f99440f40a0 2024-02-10T08:38:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f9941f76700 10 client.53084.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 4 2024-02-10T08:38:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 1 4 2024-02-10T08:38:44.336 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 10 client.53084.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 5 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 5 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 2 5 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 4 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 10 client.53084.objecter _send_command 5 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.335+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9944028800 con 0x7f994411a5c0 2024-02-10T08:38:44.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f994411a5c0 2024-02-10T08:38:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9944028800 con 0x7f994411a5c0 2024-02-10T08:38:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 10 client.53084.objecter _finish_command 5 = system:0 2024-02-10T08:38:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 4 2024-02-10T08:38:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.336+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 2 5 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 10 client.53084.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 6 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 6 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f994411a450 osd=2 3 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 2 6 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 4 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 10 client.53084.objecter _send_command 6 2024-02-10T08:38:44.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.351+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f994406cfc0 con 0x7f994411a5c0 2024-02-10T08:38:44.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f994406cfc0 con 0x7f994411a5c0 2024-02-10T08:38:44.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 10 client.53084.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 4 2024-02-10T08:38:44.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 2 6 2024-02-10T08:38:44.353 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 10 client.53084.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 7 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 7 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 3 7 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 4 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 10 client.53084.objecter _send_command 7 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9944122100 con 0x7f9944126870 2024-02-10T08:38:44.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.352+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9944126870 2024-02-10T08:38:44.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.353+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9944122100 con 0x7f9944126870 2024-02-10T08:38:44.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.353+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:44.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.353+0000 7f9941f76700 10 client.53084.objecter _finish_command 7 = system:0 2024-02-10T08:38:44.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.353+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 4 2024-02-10T08:38:44.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.353+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 3 7 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 10 client.53084.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 8 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 8 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f9944027fa0 osd=3 3 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 3 8 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 4 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 10 client.53084.objecter _send_command 8 2024-02-10T08:38:44.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.367+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f99441222f0 con 0x7f9944126870 2024-02-10T08:38:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f99441222f0 con 0x7f9944126870 2024-02-10T08:38:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f9941f76700 10 client.53084.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 4 2024-02-10T08:38:44.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 3 8 2024-02-10T08:38:44.370 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:44.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 10 client.53084.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 9 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 9 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 4 9 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 4 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 10 client.53084.objecter _send_command 9 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.369+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9944059f50 con 0x7f9944059990 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9944059990 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9944059f50 con 0x7f9944059990 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 10 client.53084.objecter _finish_command 9 = system:0 2024-02-10T08:38:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 4 2024-02-10T08:38:44.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.370+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 4 9 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 10 client.53084.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 10 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 10 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441222f0 osd=4 3 2024-02-10T08:38:44.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 4 10 2024-02-10T08:38:44.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 4 2024-02-10T08:38:44.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 10 client.53084.objecter _send_command 10 2024-02-10T08:38:44.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.385+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9944059f50 con 0x7f9944059990 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.386+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9944059f50 con 0x7f9944059990 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.386+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.386+0000 7f9941f76700 10 client.53084.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.386+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 4 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.386+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 4 10 2024-02-10T08:38:44.387 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 10 client.53084.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 11 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 11 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 5 11 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 4 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 10 client.53084.objecter _send_command 11 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9944132cf0 con 0x7f994412f140 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.387+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f994412f140 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.388+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9944132cf0 con 0x7f994412f140 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.388+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.388+0000 7f9941f76700 10 client.53084.objecter _finish_command 11 = system:0 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.388+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 4 2024-02-10T08:38:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.388+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 5 11 2024-02-10T08:38:44.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 10 client.53084.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 12 2024-02-10T08:38:44.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 12 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f9944059f50 osd=5 3 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 5 12 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 4 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 10 client.53084.objecter _send_command 12 2024-02-10T08:38:44.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.402+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f99441344a0 con 0x7f994412f140 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f99441344a0 con 0x7f994412f140 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 10 client.53084.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 4 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 5 12 2024-02-10T08:38:44.404 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 10 client.53084.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 13 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 13 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 6 13 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 4 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 10 client.53084.objecter _send_command 13 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99441350a0 con 0x7f9944134b60 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.404+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9944134b60 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.405+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f99441350a0 con 0x7f9944134b60 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.405+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.405+0000 7f9941f76700 10 client.53084.objecter _finish_command 13 = system:0 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.405+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 4 2024-02-10T08:38:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.405+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 6 13 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 10 client.53084.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 14 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 14 2024-02-10T08:38:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441344a0 osd=6 3 2024-02-10T08:38:44.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 6 14 2024-02-10T08:38:44.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 4 2024-02-10T08:38:44.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 10 client.53084.objecter _send_command 14 2024-02-10T08:38:44.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.419+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f99441350a0 con 0x7f9944134b60 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.420+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f99441350a0 con 0x7f9944134b60 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.420+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.420+0000 7f9941f76700 10 client.53084.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.420+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 4 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.420+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 6 14 2024-02-10T08:38:44.421 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 10 client.53084.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 15 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 15 2024-02-10T08:38:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 7 15 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 4 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 10 client.53084.objecter _send_command 15 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f994413b6a0 con 0x7f9944137b90 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.421+0000 7f9941f76700 10 client.53084.objecter ms_handle_connect 0x7f9944137b90 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.422+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f994413b6a0 con 0x7f9944137b90 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.422+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.422+0000 7f9941f76700 10 client.53084.objecter _finish_command 15 = system:0 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.422+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 4 2024-02-10T08:38:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.422+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 7 15 2024-02-10T08:38:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 10 client.53084.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:38:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign -1 16 2024-02-10T08:38:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 20 client.53084.objecter _get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_remove -1 16 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 20 client.53084.objecter get_session s=0x7f99441350a0 osd=7 3 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 15 client.53084.objecter _session_command_op_assign 7 16 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 4 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 10 client.53084.objecter _send_command 16 2024-02-10T08:38:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.436+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f994413b6a0 con 0x7f9944137b90 2024-02-10T08:38:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f9941f76700 1 -- 172.21.15.107:0/2979270354 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f994413b6a0 con 0x7f9944137b90 2024-02-10T08:38:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f9941f76700 10 client.53084.objecter ms_dispatch 0x7f9944001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:38:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f9941f76700 10 client.53084.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f9941f76700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 4 2024-02-10T08:38:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f9941f76700 15 client.53084.objecter _session_command_op_remove 7 16 2024-02-10T08:38:44.439 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 librados: watch_flush enter 2024-02-10T08:38:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 librados: watch_flush exit 2024-02-10T08:38:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.0 2024-02-10T08:38:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6801/3321596972 conn(0x7f9944118810 legacy=0x7f9944118bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440fd450 osd=0 1 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.1 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6805/133516126 conn(0x7f99440f40a0 legacy=0x7f9944027410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99440f3a20 osd=1 1 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.2 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6809/3575881728 conn(0x7f994411a5c0 legacy=0x7f994411fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f994411a450 osd=2 1 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.3 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994904c700 1 -- 172.21.15.107:0/2979270354 reap_dead start 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6813/2871825691 conn(0x7f9944126870 legacy=0x7f9944128c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944027fa0 osd=3 1 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.4 2024-02-10T08:38:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.195:6800/2499671285 conn(0x7f9944059990 legacy=0x7f994412d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441222f0 osd=4 1 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.5 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.195:6804/4178321486 conn(0x7f994412f140 legacy=0x7f9944131530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f9944059f50 osd=5 1 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.6 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.195:6808/1417479017 conn(0x7f9944134b60 legacy=0x7f9944132cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441344a0 osd=6 1 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter close_session for osd.7 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.195:6812/1934176928 conn(0x7f9944137b90 legacy=0x7f9944139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter put_session s=0x7f99441350a0 osd=7 1 2024-02-10T08:38:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 20 client.53084.objecter shutdown clearing up homeless session... 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 10 client.53084.objecter successfully canceled tick 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994904c700 1 -- 172.21.15.107:0/2979270354 reap_dead start 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9934063d40 legacy=0x7f9934066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> v1:172.21.15.107:6790/0 conn(0x7f99441007f0 legacy=0x7f99440fa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 shutdown_connections 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 >> 172.21.15.107:0/2979270354 conn(0x7f99440f3640 msgr2=0x7f99441077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 shutdown_connections 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 -- 172.21.15.107:0/2979270354 wait complete. 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:44.438+0000 7f994aaaf700 1 librados: shutdown 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:38:44.478 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:43.055935+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.107:0/1000076357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:43.056252+0000 mon.a (mon.0) 1595 : audit [INF] from='client.53072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:43.082825+0000 mon.b (mon.1) 357 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:43.387048+0000 mgr.y (mgr.14553) 248 : cluster [DBG] pgmap v320: 314 pgs: 32 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:43.946583+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:43.946614+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.083821+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.209551+0000 mon.a (mon.0) 1598 : audit [INF] from='client.53060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.209667+0000 mon.a (mon.0) 1599 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]': finished 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.209727+0000 mon.a (mon.0) 1600 : audit [INF] from='client.53072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:44.209774+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.281901+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.107:0/2979270354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.953924+0000 mon.a (mon.0) 1602 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:44.954068+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-10T08:38:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.960737+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.984136+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:44.984566+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:45.084769+0000 mon.b (mon.1) 360 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.547 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:38:45.547 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:38:45.547 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-10T08:38:45.547 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:38:45.547 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:38:45.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-10T08:38:45.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.083821+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.209551+0000 mon.a (mon.0) 1598 : audit [INF] from='client.53060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.209667+0000 mon.a (mon.0) 1599 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]': finished 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.209727+0000 mon.a (mon.0) 1600 : audit [INF] from='client.53072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:44.209774+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.281901+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.107:0/2979270354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.953924+0000 mon.a (mon.0) 1602 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:44.954068+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.960737+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.984136+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:44.984566+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:45.084769+0000 mon.b (mon.1) 360 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.083821+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.209551+0000 mon.a (mon.0) 1598 : audit [INF] from='client.53060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.209667+0000 mon.a (mon.0) 1599 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-23", "mode": "writeback"}]': finished 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.209727+0000 mon.a (mon.0) 1600 : audit [INF] from='client.53072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:44.209774+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.281901+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.107:0/2979270354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.953924+0000 mon.a (mon.0) 1602 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi107-112403-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:44.954068+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.960737+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.984136+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:44.984566+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:45.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:45.084769+0000 mon.b (mon.1) 360 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:45.822 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:38:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:45] "GET /metrics HTTP/1.1" 200 187838 "" "Prometheus/2.33.4" 2024-02-10T08:38:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:45.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:46.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:46.101 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:45.387552+0000 mgr.y (mgr.14553) 249 : cluster [DBG] pgmap v323: 354 pgs: 72 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:45.958335+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:45.958479+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:45.958530+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:45.961176+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:45.961638+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.085620+0000 mon.b (mon.1) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:45.387552+0000 mgr.y (mgr.14553) 249 : cluster [DBG] pgmap v323: 354 pgs: 72 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:45.958335+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:45.958479+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:45.958530+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:45.961176+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:45.961638+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.085620+0000 mon.b (mon.1) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:45.387552+0000 mgr.y (mgr.14553) 249 : cluster [DBG] pgmap v323: 354 pgs: 72 unknown, 32 creating+peering, 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 235 active+clean; 9.0 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:45.958335+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? v1:172.21.15.107:0/3026999653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:45.958479+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:45.958530+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:45.961176+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:45.961638+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]: dispatch 2024-02-10T08:38:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.085620+0000 mon.b (mon.1) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:46.752 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:46.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:46.884 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:46.982 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:46.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:47.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:47.113 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:47.262 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:47.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:47.270 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:38:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:38:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:38:47.271 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:46.955354+0000 mon.a (mon.0) 1610 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.961462+0000 mon.a (mon.0) 1611 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:46.961559+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.967717+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.107:0/1614701628' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.967908+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.968140+0000 mon.a (mon.0) 1613 : audit [INF] from='client.53090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:46.968393+0000 mon.a (mon.0) 1614 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:47.086564+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:46.955354+0000 mon.a (mon.0) 1610 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.961462+0000 mon.a (mon.0) 1611 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:46.961559+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.967717+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.107:0/1614701628' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.967908+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.968140+0000 mon.a (mon.0) 1613 : audit [INF] from='client.53090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:46.968393+0000 mon.a (mon.0) 1614 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:47.086564+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:46.955354+0000 mon.a (mon.0) 1610 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.961462+0000 mon.a (mon.0) 1611 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-23"}]': finished 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:46.961559+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.967717+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.107:0/1614701628' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.967908+0000 mon.c (mon.2) 275 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.968140+0000 mon.a (mon.0) 1613 : audit [INF] from='client.53090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:46.968393+0000 mon.a (mon.0) 1614 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:47.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:47.086564+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:47.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:47.397 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:47.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:38:47.497 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:47.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:47.518 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:38:47.518 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-02-10T08:38:47.518 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:38:47.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:47.629 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:47.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:38:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:47.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:48.040 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:48.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:48.117 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:48.316 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:48.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:48.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:48.447 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:48.465 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:48.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:47.388021+0000 mgr.y (mgr.14553) 250 : cluster [DBG] pgmap v326: 378 pgs: 3 creating+activating, 32 unknown, 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 325 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:47.961198+0000 mon.a (mon.0) 1615 : audit [INF] from='client.53090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:47.961271+0000 mon.a (mon.0) 1616 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:47.961362+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:47.964177+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:47.968616+0000 mon.a (mon.0) 1618 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.087742+0000 mon.b (mon.1) 364 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:48.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:47.388021+0000 mgr.y (mgr.14553) 250 : cluster [DBG] pgmap v326: 378 pgs: 3 creating+activating, 32 unknown, 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 325 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:47.961198+0000 mon.a (mon.0) 1615 : audit [INF] from='client.53090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:47.961271+0000 mon.a (mon.0) 1616 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:47.961362+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:47.964177+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:47.968616+0000 mon.a (mon.0) 1618 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.087742+0000 mon.b (mon.1) 364 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:47.388021+0000 mgr.y (mgr.14553) 250 : cluster [DBG] pgmap v326: 378 pgs: 3 creating+activating, 32 unknown, 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 325 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-10T08:38:48.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:47.961198+0000 mon.a (mon.0) 1615 : audit [INF] from='client.53090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:48.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:47.961271+0000 mon.a (mon.0) 1616 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:48.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:47.961362+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-10T08:38:48.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:47.964177+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.107:0/2713829649' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:47.968616+0000 mon.a (mon.0) 1618 : audit [INF] from='client.53066 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]: dispatch 2024-02-10T08:38:48.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.087742+0000 mon.b (mon.1) 364 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:48.596 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:48.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:38:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:48.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:48.875 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:48.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:48.993 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:49.013 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:49.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:49.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:49.491 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:49.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:49.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:49.629 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:49.651 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:49.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:49.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:49.784 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- 172.21.15.107:0/343948043 >> v1:172.21.15.107:6790/0 conn(0x7f02d00ff710 legacy=0x7f02d0101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- 172.21.15.107:0/343948043 shutdown_connections 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- 172.21.15.107:0/343948043 >> 172.21.15.107:0/343948043 conn(0x7f02d00f2510 msgr2=0x7f02d00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- 172.21.15.107:0/343948043 shutdown_connections 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- 172.21.15.107:0/343948043 wait complete. 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 librados: starting msgr at 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 librados: starting objecter 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 Processor -- start 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- start start 2024-02-10T08:38:49.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 librados: setting wanted keys 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 librados: calling monclient init 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d01172c0 con 0x7f02d00ff710 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d01184a0 con 0x7f02d00fc910 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.936+0000 7f02d7ea3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d01196a0 con 0x7f02d0103250 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d543e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02d00ff710 0x7f02d00fa010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57304/0 (socket says 172.21.15.107:57304) 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d543e700 1 -- 172.21.15.107:0/201349834 learned_addr learned my addr 172.21.15.107:0/201349834 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d6440700 1 --1- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6790/0 conn(0x7f02d0103250 0x7f02d0115980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34110/0 (socket says 172.21.15.107:34110) 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter ms_handle_connect 0x7f02d00ff710 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter ms_handle_connect 0x7f02d0103250 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter ms_handle_connect 0x7f02d00fc910 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:49.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732427941 0 0) 0x7f02d01172c0 con 0x7f02d00ff710 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc003630 con 0x7f02d00ff710 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213310567 0 0) 0x7f02d01196a0 con 0x7f02d0103250 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d01172c0 con 0x7f02d0103250 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624175325 0 0) 0x7f02d01184a0 con 0x7f02d00fc910 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d01196a0 con 0x7f02d00fc910 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053382798 0 0) 0x7f02bc003630 con 0x7f02d00ff710 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d01184a0 con 0x7f02d00ff710 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02c00020a0 con 0x7f02d00ff710 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2071485767 0 0) 0x7f02d01172c0 con 0x7f02d0103250 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02bc003630 con 0x7f02d0103250 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02c8001b00 con 0x7f02d0103250 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 943196827 0 0) 0x7f02d01196a0 con 0x7f02d00fc910 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d01172c0 con 0x7f02d00fc910 2024-02-10T08:38:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02cc001620 con 0x7f02d00fc910 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3704421763 0 0) 0x7f02d01184a0 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6790/0 conn(0x7f02d0103250 legacy=0x7f02d0115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.195:6789/0 conn(0x7f02d00fc910 legacy=0x7f02d00f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d011a8a0 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d0119890 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d7ea3700 10 client.27570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d0119d00 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02d7ea3700 1 librados: init done 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f02c0002730 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.937+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02c00038e0 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.938+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f02b4004f00 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f02c0003180 con 0x7f02d00ff710 2024-02-10T08:38:49.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(235..235 src has 50..235) v4 ==== 10233+0+0 (unknown 3978658318 0 0) 0x7f02c0079750 con 0x7f02d00ff710 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 osd_map(235..235 src has 50..235) v4 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 3 client.27570.objecter handle_osd_map got epochs [235,235] > 0 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 3 client.27570.objecter handle_osd_map decoding full epoch 235 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 20 client.27570.objecter dump_active .. 0 homeless 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v235) v1 ==== 54+0+16 (unknown 3184470955 0 2420000192) 0x7f02c0079b10 con 0x7f02d00ff710 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02bc063dd0 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 1 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 1 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 0 1 2024-02-10T08:38:49.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 4 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 10 client.27570.objecter _send_command 1 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.939+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b4004950 con 0x7f02b4004530 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b4004530 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b4004950 con 0x7f02b4004530 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 1 = system:0 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 4 2024-02-10T08:38:49.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.940+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 0 1 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 2 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 2 2024-02-10T08:38:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b40051a0 osd=0 3 2024-02-10T08:38:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 0 2 2024-02-10T08:38:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 4 2024-02-10T08:38:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 10 client.27570.objecter _send_command 2 2024-02-10T08:38:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.955+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b4004d30 con 0x7f02b4004530 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.956+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b4004d30 con 0x7f02b4004530 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.956+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.956+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.956+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 4 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.956+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 0 2 2024-02-10T08:38:49.957 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:38:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 3 2024-02-10T08:38:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 3 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 1 3 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 4 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 10 client.27570.objecter _send_command 3 2024-02-10T08:38:49.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b400bf60 con 0x7f02b4003b40 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b4003b40 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b400bf60 con 0x7f02b4003b40 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 3 = system:0 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 4 2024-02-10T08:38:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.957+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 1 3 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 4 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 4 2024-02-10T08:38:49.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4004270 osd=1 3 2024-02-10T08:38:49.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 1 4 2024-02-10T08:38:49.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 4 2024-02-10T08:38:49.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 10 client.27570.objecter _send_command 4 2024-02-10T08:38:49.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.979+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b400d280 con 0x7f02b4003b40 2024-02-10T08:38:49.982 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:49.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:49.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.990+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b400d280 con 0x7f02b4003b40 2024-02-10T08:38:49.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.990+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:49.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.990+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:49.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.990+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 4 2024-02-10T08:38:49.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.990+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 1 4 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 5 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:49.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 5 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 2 5 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 4 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 10 client.27570.objecter _send_command 5 2024-02-10T08:38:49.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.992+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b40034a0 con 0x7f02b4002070 2024-02-10T08:38:49.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:49.999+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b4002070 2024-02-10T08:38:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.001+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b40034a0 con 0x7f02b4002070 2024-02-10T08:38:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.001+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:38:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.001+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 5 = system:0 2024-02-10T08:38:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.001+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 4 2024-02-10T08:38:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.001+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 2 5 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 6 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 6 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4003160 osd=2 3 2024-02-10T08:38:50.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 2 6 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 4 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 10 client.27570.objecter _send_command 6 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.020+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b4001630 con 0x7f02b4002070 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b4001630 con 0x7f02b4002070 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 4 2024-02-10T08:38:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 2 6 2024-02-10T08:38:50.023 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:38:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 7 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 7 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 3 7 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 4 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 10 client.27570.objecter _send_command 7 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b400d040 con 0x7f02b40152e0 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.022+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b40152e0 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.023+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b400d040 con 0x7f02b40152e0 2024-02-10T08:38:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.023+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:38:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.023+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 7 = system:0 2024-02-10T08:38:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.023+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 4 2024-02-10T08:38:50.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.023+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 3 7 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 8 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 8 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b400c8d0 osd=3 3 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 3 8 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 4 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 10 client.27570.objecter _send_command 8 2024-02-10T08:38:50.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.038+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b4016600 con 0x7f02b40152e0 2024-02-10T08:38:50.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b4016600 con 0x7f02b40152e0 2024-02-10T08:38:50.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 4 2024-02-10T08:38:50.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 3 8 2024-02-10T08:38:50.041 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:38:50.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:50.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 9 2024-02-10T08:38:50.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 9 2024-02-10T08:38:50.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 4 9 2024-02-10T08:38:50.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 4 2024-02-10T08:38:50.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 10 client.27570.objecter _send_command 9 2024-02-10T08:38:50.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.040+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b401e690 con 0x7f02b401ab10 2024-02-10T08:38:50.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b401ab10 2024-02-10T08:38:50.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b401e690 con 0x7f02b401ab10 2024-02-10T08:38:50.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:38:50.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 9 = system:0 2024-02-10T08:38:50.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 4 2024-02-10T08:38:50.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.041+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 4 9 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 10 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 10 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4016600 osd=4 3 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 4 10 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 4 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 10 client.27570.objecter _send_command 10 2024-02-10T08:38:50.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.057+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b401e690 con 0x7f02b401ab10 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b401e690 con 0x7f02b401ab10 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 4 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 4 10 2024-02-10T08:38:50.060 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 11 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 11 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 5 11 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 4 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 10 client.27570.objecter _send_command 11 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.059+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b401f8b0 con 0x7f02b4020350 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.060+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b4020350 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.061+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b401f8b0 con 0x7f02b4020350 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.061+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.061+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 11 = system:0 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.061+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 4 2024-02-10T08:38:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.061+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 5 11 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 12 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 12 2024-02-10T08:38:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b401e690 osd=5 3 2024-02-10T08:38:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 5 12 2024-02-10T08:38:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 4 2024-02-10T08:38:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 10 client.27570.objecter _send_command 12 2024-02-10T08:38:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.075+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b401f8b0 con 0x7f02b4020350 2024-02-10T08:38:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.076+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b401f8b0 con 0x7f02b4020350 2024-02-10T08:38:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.076+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.076+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.076+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 4 2024-02-10T08:38:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.076+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 5 12 2024-02-10T08:38:50.078 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 13 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 13 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 6 13 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 4 2024-02-10T08:38:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 10 client.27570.objecter _send_command 13 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b4027060 con 0x7f02b4023550 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.077+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b4023550 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.078+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b4027060 con 0x7f02b4023550 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.078+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.078+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 13 = system:0 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.078+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 4 2024-02-10T08:38:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.078+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 6 13 2024-02-10T08:38:50.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 14 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 14 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b401f8b0 osd=6 3 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 6 14 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 4 2024-02-10T08:38:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 10 client.27570.objecter _send_command 14 2024-02-10T08:38:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.092+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b4027060 con 0x7f02b4023550 2024-02-10T08:38:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b4027060 con 0x7f02b4023550 2024-02-10T08:38:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 4 2024-02-10T08:38:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 6 14 2024-02-10T08:38:50.095 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 15 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 15 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 7 15 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 4 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 10 client.27570.objecter _send_command 15 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.094+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b402b370 con 0x7f02b40276e0 2024-02-10T08:38:50.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 10 client.27570.objecter ms_handle_connect 0x7f02b40276e0 2024-02-10T08:38:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02b402b370 con 0x7f02b40276e0 2024-02-10T08:38:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:38:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 15 = system:0 2024-02-10T08:38:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 4 2024-02-10T08:38:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.095+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 7 15 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 10 client.27570.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign -1 16 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 20 client.27570.objecter _get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_remove -1 16 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 20 client.27570.objecter get_session s=0x7f02b4027060 osd=7 3 2024-02-10T08:38:50.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 15 client.27570.objecter _session_command_op_assign 7 16 2024-02-10T08:38:50.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 4 2024-02-10T08:38:50.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 10 client.27570.objecter _send_command 16 2024-02-10T08:38:50.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.110+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02b402b370 con 0x7f02b40276e0 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.112+0000 7f02c6ffd700 1 -- 172.21.15.107:0/201349834 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02b402b370 con 0x7f02b40276e0 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.112+0000 7f02c6ffd700 10 client.27570.objecter ms_dispatch 0x7f02d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.112+0000 7f02c6ffd700 10 client.27570.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.112+0000 7f02c6ffd700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 4 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.112+0000 7f02c6ffd700 15 client.27570.objecter _session_command_op_remove 7 16 2024-02-10T08:38:50.113 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:38:50.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 librados: watch_flush enter 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 librados: watch_flush exit 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.0 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6801/3321596972 conn(0x7f02b4004530 legacy=0x7f02b4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b40051a0 osd=0 1 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.1 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6805/133516126 conn(0x7f02b4003b40 legacy=0x7f02b400a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4004270 osd=1 1 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.2 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6809/3575881728 conn(0x7f02b4002070 legacy=0x7f02b400f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4003160 osd=2 1 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.3 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6813/2871825691 conn(0x7f02b40152e0 legacy=0x7f02b4015720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b400c8d0 osd=3 1 2024-02-10T08:38:50.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.4 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.195:6800/2499671285 conn(0x7f02b401ab10 legacy=0x7f02b401cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4016600 osd=4 1 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.5 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.195:6804/4178321486 conn(0x7f02b4020350 legacy=0x7f02b40207d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401e690 osd=5 1 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.6 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.195:6808/1417479017 conn(0x7f02b4023550 legacy=0x7f02b4025940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b401f8b0 osd=6 1 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d6440700 1 -- 172.21.15.107:0/201349834 reap_dead start 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter close_session for osd.7 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.195:6812/1934176928 conn(0x7f02b40276e0 legacy=0x7f02b4029af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d6440700 1 -- 172.21.15.107:0/201349834 reap_dead start 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d6440700 1 -- 172.21.15.107:0/201349834 reap_dead start 2024-02-10T08:38:50.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d6440700 1 -- 172.21.15.107:0/201349834 reap_dead start 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter put_session s=0x7f02b4027060 osd=7 1 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 20 client.27570.objecter shutdown clearing up homeless session... 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 10 client.27570.objecter successfully canceled tick 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6800/2634842009 conn(0x7f02bc063dd0 legacy=0x7f02bc066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> v1:172.21.15.107:6789/0 conn(0x7f02d00ff710 legacy=0x7f02d00fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 shutdown_connections 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 >> 172.21.15.107:0/201349834 conn(0x7f02d00f2510 msgr2=0x7f02d00612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 shutdown_connections 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 -- 172.21.15.107:0/201349834 wait complete. 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:50.113+0000 7f02d7ea3700 1 librados: shutdown 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.117 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.118 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.118 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.118 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:38:50.119 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:50.139 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.967466+0000 mon.a (mon.0) 1619 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:50.139 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:48.967560+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-10T08:38:50.139 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.977389+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.107:0/2622016933' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.139 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.978430+0000 mon.a (mon.0) 1621 : audit [INF] from='client.47812 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.985921+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:48.986897+0000 mon.a (mon.0) 1622 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.006877+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.007271+0000 mon.a (mon.0) 1623 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.008147+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.008441+0000 mon.a (mon.0) 1624 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.008816+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.009100+0000 mon.a (mon.0) 1625 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.088701+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:50.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:49.388584+0000 mgr.y (mgr.14553) 251 : cluster [DBG] pgmap v329: 410 pgs: 3 creating+activating, 96 unknown, 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 298 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.939327+0000 mon.a (mon.0) 1626 : audit [DBG] from='client.? v1:172.21.15.107:0/201349834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.967466+0000 mon.a (mon.0) 1619 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:48.967560+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.977389+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.107:0/2622016933' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.978430+0000 mon.a (mon.0) 1621 : audit [INF] from='client.47812 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.985921+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:48.986897+0000 mon.a (mon.0) 1622 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.006877+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.007271+0000 mon.a (mon.0) 1623 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.008147+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.008441+0000 mon.a (mon.0) 1624 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.008816+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.009100+0000 mon.a (mon.0) 1625 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.088701+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:50.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:49.388584+0000 mgr.y (mgr.14553) 251 : cluster [DBG] pgmap v329: 410 pgs: 3 creating+activating, 96 unknown, 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 298 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:38:50.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.939327+0000 mon.a (mon.0) 1626 : audit [DBG] from='client.? v1:172.21.15.107:0/201349834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:50.142 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:50.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:50.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:38:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.967466+0000 mon.a (mon.0) 1619 : audit [INF] from='client.53066 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi107-112403-33"}]': finished 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:48.967560+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.977389+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.107:0/2622016933' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.978430+0000 mon.a (mon.0) 1621 : audit [INF] from='client.47812 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.985921+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:48.986897+0000 mon.a (mon.0) 1622 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.006877+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.007271+0000 mon.a (mon.0) 1623 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.008147+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.008441+0000 mon.a (mon.0) 1624 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.008816+0000 mon.c (mon.2) 279 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.009100+0000 mon.a (mon.0) 1625 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.088701+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:49.388584+0000 mgr.y (mgr.14553) 251 : cluster [DBG] pgmap v329: 410 pgs: 3 creating+activating, 96 unknown, 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 298 active+clean; 9.1 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:38:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.939327+0000 mon.a (mon.0) 1626 : audit [DBG] from='client.? v1:172.21.15.107:0/201349834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:38:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:50.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:50.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:50.521 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:50.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:38:50.663 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:51.212 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:38:51.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.976702+0000 mon.a (mon.0) 1627 : audit [INF] from='client.47812 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.976828+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.976893+0000 mon.a (mon.0) 1629 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:49.976951+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.980121+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:49.983428+0000 mon.a (mon.0) 1631 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:50.089719+0000 mon.b (mon.1) 368 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.976702+0000 mon.a (mon.0) 1627 : audit [INF] from='client.47812 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.976828+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.976893+0000 mon.a (mon.0) 1629 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:49.976951+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.980121+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:49.983428+0000 mon.a (mon.0) 1631 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.216 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:50.089719+0000 mon.b (mon.1) 368 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:51.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:38:51.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:51.344 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:51.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:38:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.976702+0000 mon.a (mon.0) 1627 : audit [INF] from='client.47812 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi107-112411-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.976828+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.976893+0000 mon.a (mon.0) 1629 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi107-112403-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:51.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:49.976951+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-10T08:38:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.980121+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:49.983428+0000 mon.a (mon.0) 1631 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:51.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:50.089719+0000 mon.b (mon.1) 368 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:51.728 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:52.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.090687+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:51.194021+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.199112+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.201612+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.275236+0000 mgr.y (mgr.14553) 252 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:51.389785+0000 mgr.y (mgr.14553) 253 : cluster [DBG] pgmap v332: 378 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 374 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.950992+0000 mon.a (mon.0) 1634 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.951088+0000 mon.a (mon.0) 1635 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:51.951172+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.955999+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:51.957067+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:52.091539+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.090687+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:51.194021+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.199112+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.201612+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.275236+0000 mgr.y (mgr.14553) 252 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:51.389785+0000 mgr.y (mgr.14553) 253 : cluster [DBG] pgmap v332: 378 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 374 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.950992+0000 mon.a (mon.0) 1634 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.951088+0000 mon.a (mon.0) 1635 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:51.951172+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-10T08:38:52.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.955999+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.525 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:51.957067+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.525 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:52.091539+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:52.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.090687+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:51.194021+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.199112+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.201612+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.275236+0000 mgr.y (mgr.14553) 252 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:51.389785+0000 mgr.y (mgr.14553) 253 : cluster [DBG] pgmap v332: 378 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 374 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.950992+0000 mon.a (mon.0) 1634 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi107-112403-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.951088+0000 mon.a (mon.0) 1635 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:51.951172+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.955999+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:51.957067+0000 mon.a (mon.0) 1637 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:52.091539+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:52.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:52.661 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:52.681 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:52.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:52.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:52.814 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:52.834 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:52.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:52.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:52.980 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:52.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:38:52.998 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:53.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:53.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:53.147 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:53.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:53.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:53.303 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:38:53.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T08:38:53.530 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:53.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:53.541 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:38:53.541 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:38:53.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:53.715 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:53.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:53.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:53.851 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:54.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:52.954699+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:52.954845+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:52.962427+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:52.964322+0000 mon.a (mon.0) 1640 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:53.092424+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:53.390170+0000 mgr.y (mgr.14553) 254 : cluster [DBG] pgmap v335: 322 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 310 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:52.954699+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:52.954845+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-10T08:38:54.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:52.962427+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:52.964322+0000 mon.a (mon.0) 1640 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:53.092424+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:54.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:53.390170+0000 mgr.y (mgr.14553) 254 : cluster [DBG] pgmap v335: 322 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 310 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:52.954699+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:52.954845+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:52.962427+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:52.964322+0000 mon.a (mon.0) 1640 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]: dispatch 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:53.092424+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:54.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:53.390170+0000 mgr.y (mgr.14553) 254 : cluster [DBG] pgmap v335: 322 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 310 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:38:54.400 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:54.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:54.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:54.533 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:54.553 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:54.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:54.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:54.684 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:54.956 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:53.949924+0000 mon.a (mon.0) 1641 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:55.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 -- 172.21.15.107:0/102501141 >> v1:172.21.15.107:6789/0 conn(0x7fe3cc104340 legacy=0x7fe3cc1067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 -- 172.21.15.107:0/102501141 shutdown_connections 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 -- 172.21.15.107:0/102501141 >> 172.21.15.107:0/102501141 conn(0x7fe3cc06fe70 msgr2=0x7fe3cc0f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 -- 172.21.15.107:0/102501141 shutdown_connections 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 -- 172.21.15.107:0/102501141 wait complete. 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 librados: starting msgr at 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.281+0000 7fe3d3e10700 1 librados: starting objecter 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 Processor -- start 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 -- start start 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 librados: setting wanted keys 2024-02-10T08:38:55.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 librados: calling monclient init 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3cc117220 con 0x7fe3cc104340 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3cc118400 con 0x7fe3cc100830 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d3e10700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3cc119600 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d1bac700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe3cc0fd9e0 0x7fe3cc0f9ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34142/0 (socket says 172.21.15.107:34142) 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3d1bac700 1 -- 172.21.15.107:0/342422474 learned_addr learned my addr 172.21.15.107:0/342422474 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter ms_handle_connect 0x7fe3cc0fd9e0 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter ms_handle_connect 0x7fe3cc104340 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter ms_handle_connect 0x7fe3cc100830 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048365394 0 0) 0x7fe3cc119600 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b8003030 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299457935 0 0) 0x7fe3cc117220 con 0x7fe3cc104340 2024-02-10T08:38:55.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3cc119600 con 0x7fe3cc104340 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666392720 0 0) 0x7fe3cc118400 con 0x7fe3cc100830 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.282+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3cc117220 con 0x7fe3cc100830 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1492258220 0 0) 0x7fe3b8003030 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3cc118400 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3c8001960 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717918042 0 0) 0x7fe3cc117220 con 0x7fe3cc100830 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3b8003030 con 0x7fe3cc100830 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2716043192 0 0) 0x7fe3cc119600 con 0x7fe3cc104340 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3cc117220 con 0x7fe3cc104340 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3bc001840 con 0x7fe3cc100830 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3c4001890 con 0x7fe3cc104340 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 920867473 0 0) 0x7fe3cc118400 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 >> v1:172.21.15.195:6789/0 conn(0x7fe3cc100830 legacy=0x7fe3cc0fa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 >> v1:172.21.15.107:6789/0 conn(0x7fe3cc104340 legacy=0x7fe3cc1159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:55.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc11a800 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc117410 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3d3e10700 10 client.53117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:38:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc117880 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.283+0000 7fe3d3e10700 1 librados: init done 2024-02-10T08:38:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:53.949924+0000 mon.a (mon.0) 1641 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:53.956729+0000 mon.a (mon.0) 1642 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]': finished 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:53.956813+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:53.966097+0000 mon.c (mon.2) 281 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:53.966758+0000 mon.a (mon.0) 1644 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.004078+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.004387+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.093330+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:55.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.284+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe3c8002370 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.284+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc05f780 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.284+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3c8005a50 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe3c80189c0 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(241..241 src has 50..241) v4 ==== 8463+0+0 (unknown 1177837919 0 0) 0x7fe3c8079d00 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 10 client.53117.objecter ms_dispatch 0x7fe3cc001550 osd_map(241..241 src has 50..241) v4 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 3 client.53117.objecter handle_osd_map got epochs [241,241] > 0 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 3 client.53117.objecter handle_osd_map decoding full epoch 241 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 20 client.53117.objecter dump_active .. 0 homeless 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.285+0000 7fe3c2ffd700 10 client.53117.objecter ms_handle_connect 0x7fe3b8068270 2024-02-10T08:38:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.289+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe3c804cf80 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.332 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:53.949924+0000 mon.a (mon.0) 1641 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:53.956729+0000 mon.a (mon.0) 1642 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]': finished 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:53.956813+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:53.966097+0000 mon.c (mon.2) 281 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:53.966758+0000 mon.a (mon.0) 1644 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.004078+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.004387+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.093330+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:53.956729+0000 mon.a (mon.0) 1642 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-25", "mode": "writeback"}]': finished 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:53.956813+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:53.966097+0000 mon.c (mon.2) 281 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:53.966758+0000 mon.a (mon.0) 1644 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.004078+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.004387+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:38:55.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.093330+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:55.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:55.366 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:55.376 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:55.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:55.468 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:55.489 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:55.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:55.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.503+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fe3cc109fd0 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:55.648 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:55.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:55.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:55.784 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:55.805 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:55.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:55.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:38:55] "GET /metrics HTTP/1.1" 200 187957 "" "Prometheus/2.33.4" 2024-02-10T08:38:55.939 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:55.960 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:55.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:55.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.974+0000 7fe3c2ffd700 1 -- 172.21.15.107:0/342422474 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v242) v1 ==== 91+0+0 (unknown 448766630 0 0) 0x7fe3c804b390 con 0x7fe3cc0fd9e0 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 10 librados: watch_flush enter 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 10 librados: watch_flush exit 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 20 client.53117.objecter shutdown clearing up homeless session... 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 10 client.53117.objecter successfully canceled tick 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe3b8068270 legacy=0x7fe3b806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.986+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 >> v1:172.21.15.107:6790/0 conn(0x7fe3cc0fd9e0 legacy=0x7fe3cc0f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:38:55.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.987+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 shutdown_connections 2024-02-10T08:38:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.987+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 >> 172.21.15.107:0/342422474 conn(0x7fe3cc06fe70 msgr2=0x7fe3cc0f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:38:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.988+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 shutdown_connections 2024-02-10T08:38:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.988+0000 7fe3d3e10700 1 -- 172.21.15.107:0/342422474 wait complete. 2024-02-10T08:38:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:38:55.988+0000 7fe3d3e10700 1 librados: shutdown 2024-02-10T08:38:55.991 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.967884+0000 mon.a (mon.0) 1646 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.967970+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:54.968016+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.971767+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.972886+0000 mon.a (mon.0) 1649 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.974412+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:54.974917+0000 mon.a (mon.0) 1650 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.094279+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:56.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.249115+0000 mon.a (mon.0) 1651 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:56.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:55.391040+0000 mgr.y (mgr.14553) 255 : cluster [DBG] pgmap v338: 250 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 246 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:56.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.504002+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.107:0/342422474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:56.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.504356+0000 mon.a (mon.0) 1652 : audit [INF] from='client.53117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:56.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.967884+0000 mon.a (mon.0) 1646 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.967970+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:54.968016+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.971767+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.972886+0000 mon.a (mon.0) 1649 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.974412+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:54.974917+0000 mon.a (mon.0) 1650 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.094279+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.249115+0000 mon.a (mon.0) 1651 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:55.391040+0000 mgr.y (mgr.14553) 255 : cluster [DBG] pgmap v338: 250 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 246 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:56.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.504002+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.107:0/342422474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.504356+0000 mon.a (mon.0) 1652 : audit [INF] from='client.53117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.967884+0000 mon.a (mon.0) 1646 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.967970+0000 mon.a (mon.0) 1647 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:54.968016+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.971767+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.972886+0000 mon.a (mon.0) 1649 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.974412+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.107:0/3207694689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:54.974917+0000 mon.a (mon.0) 1650 : audit [INF] from='client.53099 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.094279+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.249115+0000 mon.a (mon.0) 1651 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:55.391040+0000 mgr.y (mgr.14553) 255 : cluster [DBG] pgmap v338: 250 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 246 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:38:56.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.504002+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.107:0/342422474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:56.348 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.504356+0000 mon.a (mon.0) 1652 : audit [INF] from='client.53117 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:38:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:55.962920+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:55.962954+0000 mon.a (mon.0) 1654 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.974362+0000 mon.a (mon.0) 1655 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.974495+0000 mon.a (mon.0) 1656 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.974599+0000 mon.a (mon.0) 1657 : audit [INF] from='client.53117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:55.974634+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.976790+0000 mon.c (mon.2) 284 : audit [INF] from='client.? v1:172.21.15.107:0/1877759425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:55.988489+0000 mon.a (mon.0) 1659 : audit [INF] from='client.53111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.008188+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.009576+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.009931+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.095293+0000 mon.b (mon.1) 379 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:56.212448+0000 osd.2 (osd.2) 85 : cluster [DBG] 25.9 scrub starts 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:56.213369+0000 osd.2 (osd.2) 86 : cluster [DBG] 25.9 scrub ok 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:56.339959+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:56.340141+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.974354+0000 mon.a (mon.0) 1663 : audit [INF] from='client.53111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.974464+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:56.974542+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-10T08:38:57.401 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:38:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:55.962920+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:55.962954+0000 mon.a (mon.0) 1654 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.974362+0000 mon.a (mon.0) 1655 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.974495+0000 mon.a (mon.0) 1656 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.974599+0000 mon.a (mon.0) 1657 : audit [INF] from='client.53117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:55.974634+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.976790+0000 mon.c (mon.2) 284 : audit [INF] from='client.? v1:172.21.15.107:0/1877759425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:55.988489+0000 mon.a (mon.0) 1659 : audit [INF] from='client.53111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.008188+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.009576+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.009931+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.095293+0000 mon.b (mon.1) 379 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:56.212448+0000 osd.2 (osd.2) 85 : cluster [DBG] 25.9 scrub starts 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:56.213369+0000 osd.2 (osd.2) 86 : cluster [DBG] 25.9 scrub ok 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:56.339959+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:56.340141+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.974354+0000 mon.a (mon.0) 1663 : audit [INF] from='client.53111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.974464+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:56.974542+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:55.962920+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:55.962954+0000 mon.a (mon.0) 1654 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.974362+0000 mon.a (mon.0) 1655 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-25"}]': finished 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.974495+0000 mon.a (mon.0) 1656 : audit [INF] from='client.53099 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi107-112403-34"}]': finished 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.974599+0000 mon.a (mon.0) 1657 : audit [INF] from='client.53117 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:55.974634+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.976790+0000 mon.c (mon.2) 284 : audit [INF] from='client.? v1:172.21.15.107:0/1877759425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:55.988489+0000 mon.a (mon.0) 1659 : audit [INF] from='client.53111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.008188+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.009576+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.009931+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.095293+0000 mon.b (mon.1) 379 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:56.212448+0000 osd.2 (osd.2) 85 : cluster [DBG] 25.9 scrub starts 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:56.213369+0000 osd.2 (osd.2) 86 : cluster [DBG] 25.9 scrub ok 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:56.339959+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:56.340141+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.974354+0000 mon.a (mon.0) 1663 : audit [INF] from='client.53111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi107-112411-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.974464+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi107-112403-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:38:57.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:56.974542+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-10T08:38:57.642 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:57.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:57.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:57.781 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:57.799 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:57.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:57.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:38:57.932 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:57.952 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:38:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:57.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:38:58.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:38:58.083 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:58.101 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:38:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:58.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:38:58.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:38:58.240 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:56.977557+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:57.096457+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:57.167506+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:38:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:57.168516+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:38:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:57.391544+0000 mgr.y (mgr.14553) 256 : cluster [DBG] pgmap v341: 250 pgs: 32 creating+peering, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:56.977557+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:57.096457+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:57.167506+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:38:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:57.168516+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:38:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:57.391544+0000 mgr.y (mgr.14553) 256 : cluster [DBG] pgmap v341: 250 pgs: 32 creating+peering, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:58.438 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:38:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:58.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:38:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:56.977557+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:38:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:57.096457+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:57.167506+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:38:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:57.168516+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:38:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:57.391544+0000 mgr.y (mgr.14553) 256 : cluster [DBG] pgmap v341: 250 pgs: 32 creating+peering, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:38:58.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:38:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:59.017 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:38:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:59.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:38:59.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:59.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:38:59.170 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:59.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:59.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:38:59.303 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:59.322 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:38:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:59.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:38:59.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:59.453 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:57.916651+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:57.920136+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.097411+0000 mon.b (mon.1) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.216558+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.217864+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.224389+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.726831+0000 osd.4 (osd.4) 110 : cluster [DBG] 25.b scrub starts 2024-02-10T08:38:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.727762+0000 osd.4 (osd.4) 111 : cluster [DBG] 25.b scrub ok 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.976149+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.976341+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.976433+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.983725+0000 mon.c (mon.2) 285 : audit [INF] from='client.? v1:172.21.15.107:0/1187186007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:58.988231+0000 mon.a (mon.0) 1672 : audit [INF] from='client.53126 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:38:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:59.098436+0000 mon.b (mon.1) 383 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:57.916651+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:38:59.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:57.920136+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.097411+0000 mon.b (mon.1) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.216558+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.217864+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.224389+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.726831+0000 osd.4 (osd.4) 110 : cluster [DBG] 25.b scrub starts 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.727762+0000 osd.4 (osd.4) 111 : cluster [DBG] 25.b scrub ok 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.976149+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.976341+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.976433+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.983725+0000 mon.c (mon.2) 285 : audit [INF] from='client.? v1:172.21.15.107:0/1187186007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:58.988231+0000 mon.a (mon.0) 1672 : audit [INF] from='client.53126 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:59.098436+0000 mon.b (mon.1) 383 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:57.916651+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:57.920136+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.097411+0000 mon.b (mon.1) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.216558+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.217864+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.224389+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.726831+0000 osd.4 (osd.4) 110 : cluster [DBG] 25.b scrub starts 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.727762+0000 osd.4 (osd.4) 111 : cluster [DBG] 25.b scrub ok 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.976149+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi107-112403-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.976341+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:38:59.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.976433+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-10T08:38:59.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.983725+0000 mon.c (mon.2) 285 : audit [INF] from='client.? v1:172.21.15.107:0/1187186007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:58.988231+0000 mon.a (mon.0) 1672 : audit [INF] from='client.53126 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:38:59.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:38:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:59.098436+0000 mon.b (mon.1) 383 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:38:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:38:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:59.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:38:59.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:38:59.689 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:38:59.710 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:38:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:38:59.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:38:59.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:38:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:00.039 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:00.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:00.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:00.172 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:00.193 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:00.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:00.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:00.324 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:00.343 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:00.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:00.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:00.472 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:00.472 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.913815+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:39:00.472 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:58.916993+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.392168+0000 mgr.y (mgr.14553) 257 : cluster [DBG] pgmap v344: 290 pgs: 72 unknown, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.741542+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub starts 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.745110+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.e scrub ok 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:38:59.978299+0000 mon.a (mon.0) 1673 : audit [INF] from='client.53126 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.978408+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.007091+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.007400+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.099510+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:00.474 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:00.492 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:00.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:00.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.913815+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:58.916993+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.392168+0000 mgr.y (mgr.14553) 257 : cluster [DBG] pgmap v344: 290 pgs: 72 unknown, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.741542+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub starts 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.745110+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.e scrub ok 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:38:59.978299+0000 mon.a (mon.0) 1673 : audit [INF] from='client.53126 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.978408+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.007091+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.007400+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.099510+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.913815+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:39:00.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:58.916993+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.392168+0000 mgr.y (mgr.14553) 257 : cluster [DBG] pgmap v344: 290 pgs: 72 unknown, 218 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.741542+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub starts 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.745110+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.e scrub ok 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:38:59.978299+0000 mon.a (mon.0) 1673 : audit [INF] from='client.53126 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi107-112411-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.978408+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.007091+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.007400+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:00.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.099510+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:00.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:00.624 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:00.723 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:00.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:00.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:00.854 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.973+0000 7faffc3e9700 1 -- 172.21.15.107:0/2861210914 >> v1:172.21.15.107:6789/0 conn(0x7faff4069f80 legacy=0x7faff4111bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.973+0000 7faffc3e9700 1 -- 172.21.15.107:0/2861210914 shutdown_connections 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.973+0000 7faffc3e9700 1 -- 172.21.15.107:0/2861210914 >> 172.21.15.107:0/2861210914 conn(0x7faff4063bc0 msgr2=0x7faff40672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- 172.21.15.107:0/2861210914 shutdown_connections 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- 172.21.15.107:0/2861210914 wait complete. 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 librados: starting msgr at 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 librados: starting objecter 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 Processor -- start 2024-02-10T08:39:01.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- start start 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 librados: setting wanted keys 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 librados: calling monclient init 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faff41267b0 con 0x7faff41132f0 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faff4127990 con 0x7faff41229b0 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.974+0000 7faffc3e9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faff4128b90 con 0x7faff4069490 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffa185700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faff4069490 0x7faff41014e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44490/0 (socket says 172.21.15.107:44490) 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faff9984700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faff41132f0 0x7faff4101bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60778/0 (socket says 172.21.15.107:60778) 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffa185700 1 -- 172.21.15.107:0/3102873523 learned_addr learned my addr 172.21.15.107:0/3102873523 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter ms_handle_connect 0x7faff4069490 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter ms_handle_connect 0x7faff41132f0 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter ms_handle_connect 0x7faff41229b0 2024-02-10T08:39:01.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597596038 0 0) 0x7faff4128b90 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0002220 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265545765 0 0) 0x7faff41267b0 con 0x7faff41132f0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff4128b90 con 0x7faff41132f0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938759404 0 0) 0x7faff4127990 con 0x7faff41229b0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff41267b0 con 0x7faff41229b0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441811667 0 0) 0x7fafe0002220 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff4127990 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafe4001950 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765632356 0 0) 0x7faff41267b0 con 0x7faff41229b0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe0002220 con 0x7faff41229b0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafec001a30 con 0x7faff41229b0 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3630731238 0 0) 0x7faff4127990 con 0x7faff4069490 2024-02-10T08:39:01.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 >> v1:172.21.15.195:6789/0 conn(0x7faff41229b0 legacy=0x7faff4124e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 >> v1:172.21.15.107:6789/0 conn(0x7faff41132f0 legacy=0x7faff4101bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff4129d90 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff41269a0 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffc3e9700 10 client.53129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faff4126e10 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.975+0000 7faffc3e9700 1 librados: init done 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.976+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fafe4002460 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.976+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafe4005b50 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.977+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd8004f20 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.977+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fafe40189e0 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(247..247 src has 50..247) v4 ==== 8755+0+0 (unknown 3780530888 0 0) 0x7fafe40790c0 con 0x7faff4069490 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 10 client.53129.objecter ms_dispatch 0x7faff4001550 osd_map(247..247 src has 50..247) v4 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 3 client.53129.objecter handle_osd_map got epochs [247,247] > 0 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 3 client.53129.objecter handle_osd_map decoding full epoch 247 2024-02-10T08:39:01.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 20 client.53129.objecter dump_active .. 0 homeless 2024-02-10T08:39:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.978+0000 7fafeb7fe700 10 client.53129.objecter ms_handle_connect 0x7fafe0063ab0 2024-02-10T08:39:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:00.982+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fafe404c220 con 0x7faff4069490 2024-02-10T08:39:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.169+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fafd80051d0 con 0x7faff4069490 2024-02-10T08:39:01.271 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:01.271 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.746937+0000 osd.6 (osd.6) 97 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.747759+0000 osd.6 (osd.6) 98 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.952272+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:38:59.955269+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:00.643985+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:00.645354+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.955291+0000 mon.a (mon.0) 1676 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:00.955358+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.960509+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.964727+0000 mon.a (mon.0) 1678 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.968557+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.985943+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.986312+0000 mon.a (mon.0) 1680 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.987504+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.987770+0000 mon.a (mon.0) 1681 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.988262+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:00.988522+0000 mon.a (mon.0) 1682 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.100549+0000 mon.b (mon.1) 390 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.170322+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.107:0/3102873523' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.170655+0000 mon.a (mon.0) 1683 : audit [INF] from='client.53129 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.746937+0000 osd.6 (osd.6) 97 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.747759+0000 osd.6 (osd.6) 98 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.952272+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:38:59.955269+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:00.643985+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:00.645354+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.955291+0000 mon.a (mon.0) 1676 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:00.955358+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.960509+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.964727+0000 mon.a (mon.0) 1678 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.968557+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.985943+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.986312+0000 mon.a (mon.0) 1680 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.987504+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.987770+0000 mon.a (mon.0) 1681 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.988262+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:00.988522+0000 mon.a (mon.0) 1682 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.100549+0000 mon.b (mon.1) 390 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.170322+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.107:0/3102873523' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.170655+0000 mon.a (mon.0) 1683 : audit [INF] from='client.53129 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.746937+0000 osd.6 (osd.6) 97 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:39:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.747759+0000 osd.6 (osd.6) 98 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:39:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.952272+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:38:59.955269+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:00.643985+0000 osd.3 (osd.3) 127 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:00.645354+0000 osd.3 (osd.3) 128 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.955291+0000 mon.a (mon.0) 1676 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:00.955358+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.960509+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.964727+0000 mon.a (mon.0) 1678 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.968557+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.985943+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.986312+0000 mon.a (mon.0) 1680 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.987504+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.987770+0000 mon.a (mon.0) 1681 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.988262+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:00.988522+0000 mon.a (mon.0) 1682 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.100549+0000 mon.b (mon.1) 390 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.170322+0000 mon.c (mon.2) 286 : audit [INF] from='client.? v1:172.21.15.107:0/3102873523' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.170655+0000 mon.a (mon.0) 1683 : audit [INF] from='client.53129 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:39:01.703 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:01.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:01.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:01.836 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:01.856 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:01.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 -- 172.21.15.107:0/1447078364 >> v1:172.21.15.107:6789/0 conn(0x7fc1640ff710 legacy=0x7fc164101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 -- 172.21.15.107:0/1447078364 shutdown_connections 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 -- 172.21.15.107:0/1447078364 >> 172.21.15.107:0/1447078364 conn(0x7fc1640f2510 msgr2=0x7fc1640f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 -- 172.21.15.107:0/1447078364 shutdown_connections 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 -- 172.21.15.107:0/1447078364 wait complete. 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 librados: starting msgr at 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 librados: starting objecter 2024-02-10T08:39:01.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.903+0000 7fc169d2e700 1 Processor -- start 2024-02-10T08:39:01.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 -- start start 2024-02-10T08:39:01.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 librados: setting wanted keys 2024-02-10T08:39:01.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 librados: calling monclient init 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1641173e0 con 0x7fc1640fc910 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1641185c0 con 0x7fc1640ff710 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc169d2e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1641197c0 con 0x7fc164103250 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc1637fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1640fc910 0x7fc1640f9b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60804/0 (socket says 172.21.15.107:60804) 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc1637fe700 1 -- 172.21.15.107:0/439945549 learned_addr learned my addr 172.21.15.107:0/439945549 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter ms_handle_connect 0x7fc1640fc910 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter ms_handle_connect 0x7fc164103250 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter ms_handle_connect 0x7fc1640ff710 2024-02-10T08:39:01.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449667651 0 0) 0x7fc1641173e0 con 0x7fc1640fc910 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc14c003030 con 0x7fc1640fc910 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872984236 0 0) 0x7fc1641197c0 con 0x7fc164103250 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1641173e0 con 0x7fc164103250 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825036388 0 0) 0x7fc1641185c0 con 0x7fc1640ff710 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1641197c0 con 0x7fc1640ff710 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.904+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549071713 0 0) 0x7fc14c003030 con 0x7fc1640fc910 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1641185c0 con 0x7fc1640fc910 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc158001940 con 0x7fc1640fc910 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532528378 0 0) 0x7fc1641173e0 con 0x7fc164103250 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc14c003030 con 0x7fc164103250 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc15c0018d0 con 0x7fc164103250 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047655990 0 0) 0x7fc1641197c0 con 0x7fc1640ff710 2024-02-10T08:39:01.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1641173e0 con 0x7fc1640ff710 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc154001430 con 0x7fc1640ff710 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015363194 0 0) 0x7fc1641185c0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6790/0 conn(0x7fc164103250 legacy=0x7fc164115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.195:6789/0 conn(0x7fc1640ff710 legacy=0x7fc1640fa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16411a9c0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1641199b0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc169d2e700 10 client.27612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc164119ea0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc169d2e700 1 librados: init done 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc158001cd0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.905+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc158005bc0 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.906+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc150004f00 con 0x7fc1640fc910 2024-02-10T08:39:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.906+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc158002ba0 con 0x7fc1640fc910 2024-02-10T08:39:01.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(247..247 src has 50..247) v4 ==== 8755+0+0 (unknown 3780530888 0 0) 0x7fc158078e90 con 0x7fc1640fc910 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 osd_map(247..247 src has 50..247) v4 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 3 client.27612.objecter handle_osd_map got epochs [247,247] > 0 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 3 client.27612.objecter handle_osd_map decoding full epoch 247 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 20 client.27612.objecter dump_active .. 0 homeless 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v247) v1 ==== 54+0+16 (unknown 1875848919 0 2420000192) 0x7fc158079220 con 0x7fc1640fc910 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc14c063c20 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 10 client.27612.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 1 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 1 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 0 1 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 4 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 10 client.27612.objecter _send_command 1 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc150004950 con 0x7fc150004530 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.907+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150004530 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.908+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc150004950 con 0x7fc150004530 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.908+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.908+0000 7fc160ff9700 10 client.27612.objecter _finish_command 1 = system:0 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.908+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 4 2024-02-10T08:39:01.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.908+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 0 1 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 10 client.27612.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 2 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 2 2024-02-10T08:39:01.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc1500051a0 osd=0 3 2024-02-10T08:39:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 0 2 2024-02-10T08:39:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 4 2024-02-10T08:39:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 10 client.27612.objecter _send_command 2 2024-02-10T08:39:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.923+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150004d30 con 0x7fc150004530 2024-02-10T08:39:01.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150004d30 con 0x7fc150004530 2024-02-10T08:39:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc160ff9700 10 client.27612.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 4 2024-02-10T08:39:01.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 0 2 2024-02-10T08:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 10 client.27612.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 1 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 4 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 10 client.27612.objecter _send_command 3 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.924+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc15000bf60 con 0x7fc150003b40 2024-02-10T08:39:01.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150003b40 2024-02-10T08:39:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc15000bf60 con 0x7fc150003b40 2024-02-10T08:39:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 10 client.27612.objecter _finish_command 3 = system:0 2024-02-10T08:39:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 4 2024-02-10T08:39:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.925+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 1 3 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 10 client.27612.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 4 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 4 2024-02-10T08:39:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150004270 osd=1 3 2024-02-10T08:39:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 1 4 2024-02-10T08:39:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 4 2024-02-10T08:39:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 10 client.27612.objecter _send_command 4 2024-02-10T08:39:01.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.940+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc15000d280 con 0x7fc150003b40 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.941+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc15000d280 con 0x7fc150003b40 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.941+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.941+0000 7fc160ff9700 10 client.27612.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.941+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 4 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.941+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 1 4 2024-02-10T08:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 10 client.27612.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 5 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 5 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 2 5 2024-02-10T08:39:01.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 4 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 10 client.27612.objecter _send_command 5 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc150010d20 con 0x7fc150002070 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150002070 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc150010d20 con 0x7fc150002070 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 10 client.27612.objecter _finish_command 5 = system:0 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 4 2024-02-10T08:39:01.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.942+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 2 5 2024-02-10T08:39:01.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.955+0000 7fafeb7fe700 1 -- 172.21.15.107:0/3102873523 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v248) v1 ==== 113+0+0 (unknown 1746484628 0 0) 0x7fafe404a630 con 0x7faff4069490 2024-02-10T08:39:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 10 librados: watch_flush enter 2024-02-10T08:39:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 10 librados: watch_flush exit 2024-02-10T08:39:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 20 client.53129.objecter shutdown clearing up homeless session... 2024-02-10T08:39:01.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 10 client.53129.objecter successfully canceled tick 2024-02-10T08:39:01.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 >> v1:172.21.15.107:6800/2634842009 conn(0x7fafe0063ab0 legacy=0x7fafe0065f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.959+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 >> v1:172.21.15.107:6790/0 conn(0x7faff4069490 legacy=0x7faff41014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 shutdown_connections 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 >> 172.21.15.107:0/3102873523 conn(0x7faff4063bc0 msgr2=0x7faff40647c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 shutdown_connections 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7faffc3e9700 1 -- 172.21.15.107:0/3102873523 wait complete. 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7faffc3e9700 1 librados: shutdown 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T08:39:01.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.960+0000 7fc169d2e700 10 client.27612.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 6 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 6 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150003400 osd=2 3 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 2 6 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 4 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 10 client.27612.objecter _send_command 6 2024-02-10T08:39:01.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.961+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150001630 con 0x7fc150002070 2024-02-10T08:39:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150001630 con 0x7fc150002070 2024-02-10T08:39:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc160ff9700 10 client.27612.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 4 2024-02-10T08:39:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 2 6 2024-02-10T08:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 10 client.27612.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 7 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 7 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 3 7 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 4 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 10 client.27612.objecter _send_command 7 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.962+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc15000d050 con 0x7fc150010e90 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.963+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150010e90 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.964+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc15000d050 con 0x7fc150010e90 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.964+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.964+0000 7fc160ff9700 10 client.27612.objecter _finish_command 7 = system:0 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.964+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 4 2024-02-10T08:39:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.964+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 3 7 2024-02-10T08:39:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 10 client.27612.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 8 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 8 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc15000c8d0 osd=3 3 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 3 8 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 4 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 10 client.27612.objecter _send_command 8 2024-02-10T08:39:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.981+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150012200 con 0x7fc150010e90 2024-02-10T08:39:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150012200 con 0x7fc150010e90 2024-02-10T08:39:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc160ff9700 10 client.27612.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 4 2024-02-10T08:39:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 3 8 2024-02-10T08:39:01.984 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 10 client.27612.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 9 2024-02-10T08:39:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 9 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 4 9 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 4 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 10 client.27612.objecter _send_command 9 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.983+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc150018c20 con 0x7fc1500166e0 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc1500166e0 2024-02-10T08:39:01.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc150018c20 con 0x7fc1500166e0 2024-02-10T08:39:01.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:01.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 10 client.27612.objecter _finish_command 9 = system:0 2024-02-10T08:39:01.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 4 2024-02-10T08:39:01.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:01.984+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 4 9 2024-02-10T08:39:01.989 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 10 client.27612.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 10 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 10 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150012200 osd=4 3 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 4 10 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 4 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 10 client.27612.objecter _send_command 10 2024-02-10T08:39:02.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.000+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150018c20 con 0x7fc1500166e0 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150018c20 con 0x7fc1500166e0 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc160ff9700 10 client.27612.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 4 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 4 10 2024-02-10T08:39:02.003 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 10 client.27612.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 11 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 11 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 5 11 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 4 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 10 client.27612.objecter _send_command 11 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.002+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc150022b50 con 0x7fc15001ee80 2024-02-10T08:39:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.003+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc15001ee80 2024-02-10T08:39:02.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.004+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc150022b50 con 0x7fc15001ee80 2024-02-10T08:39:02.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.004+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:02.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.004+0000 7fc160ff9700 10 client.27612.objecter _finish_command 11 = system:0 2024-02-10T08:39:02.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.004+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 4 2024-02-10T08:39:02.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.004+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 5 11 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 10 client.27612.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 12 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 12 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150018c20 osd=5 3 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 5 12 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 4 2024-02-10T08:39:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 10 client.27612.objecter _send_command 12 2024-02-10T08:39:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.019+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150024300 con 0x7fc15001ee80 2024-02-10T08:39:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150024300 con 0x7fc15001ee80 2024-02-10T08:39:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 10 client.27612.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 4 2024-02-10T08:39:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 5 12 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 10 client.27612.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 13 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 13 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 6 13 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 4 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 10 client.27612.objecter _send_command 13 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc150024f30 con 0x7fc150024980 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.021+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150024980 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.022+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc150024f30 con 0x7fc150024980 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.022+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.022+0000 7fc160ff9700 10 client.27612.objecter _finish_command 13 = system:0 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.022+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 4 2024-02-10T08:39:02.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.022+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 6 13 2024-02-10T08:39:02.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 10 client.27612.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 14 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 14 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150024300 osd=6 3 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 6 14 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 4 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 10 client.27612.objecter _send_command 14 2024-02-10T08:39:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.036+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc150024f30 con 0x7fc150024980 2024-02-10T08:39:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc150024f30 con 0x7fc150024980 2024-02-10T08:39:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc160ff9700 10 client.27612.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 4 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 6 14 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 10 client.27612.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 15 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 15 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 7 15 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 4 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 10 client.27612.objecter _send_command 15 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.038+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc15002b520 con 0x7fc150027a10 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 10 client.27612.objecter ms_handle_connect 0x7fc150027a10 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc15002b520 con 0x7fc150027a10 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 10 client.27612.objecter _finish_command 15 = system:0 2024-02-10T08:39:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 4 2024-02-10T08:39:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.039+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 7 15 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 10 client.27612.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign -1 16 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 20 client.27612.objecter _get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_remove -1 16 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 20 client.27612.objecter get_session s=0x7fc150024f30 osd=7 3 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 15 client.27612.objecter _session_command_op_assign 7 16 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 4 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 10 client.27612.objecter _send_command 16 2024-02-10T08:39:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.054+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc15002b520 con 0x7fc150027a10 2024-02-10T08:39:02.056 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.055+0000 7fc160ff9700 1 -- 172.21.15.107:0/439945549 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc15002b520 con 0x7fc150027a10 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.055+0000 7fc160ff9700 10 client.27612.objecter ms_dispatch 0x7fc164001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.055+0000 7fc160ff9700 10 client.27612.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.055+0000 7fc160ff9700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 4 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.055+0000 7fc160ff9700 15 client.27612.objecter _session_command_op_remove 7 16 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 librados: watch_flush enter 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 librados: watch_flush exit 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.0 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6801/3321596972 conn(0x7fc150004530 legacy=0x7fc150002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc1500051a0 osd=0 1 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.1 2024-02-10T08:39:02.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6805/133516126 conn(0x7fc150003b40 legacy=0x7fc15000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150004270 osd=1 1 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.2 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6809/3575881728 conn(0x7fc150002070 legacy=0x7fc15000f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150003400 osd=2 1 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.3 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6813/2871825691 conn(0x7fc150010e90 legacy=0x7fc1500112d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc15000c8d0 osd=3 1 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.4 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.195:6800/2499671285 conn(0x7fc1500166e0 legacy=0x7fc15001d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150012200 osd=4 1 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc163fff700 1 -- 172.21.15.107:0/439945549 reap_dead start 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.5 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.195:6804/4178321486 conn(0x7fc15001ee80 legacy=0x7fc1500212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc163fff700 1 -- 172.21.15.107:0/439945549 reap_dead start 2024-02-10T08:39:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150018c20 osd=5 1 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.6 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.195:6808/1417479017 conn(0x7fc150024980 legacy=0x7fc150022b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024300 osd=6 1 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter close_session for osd.7 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.195:6812/1934176928 conn(0x7fc150027a10 legacy=0x7fc150029e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter put_session s=0x7fc150024f30 osd=7 1 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 20 client.27612.objecter shutdown clearing up homeless session... 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 10 client.27612.objecter successfully canceled tick 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc163fff700 1 -- 172.21.15.107:0/439945549 reap_dead start 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc14c063c20 legacy=0x7fc14c0660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> v1:172.21.15.107:6789/0 conn(0x7fc1640fc910 legacy=0x7fc1640f9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 shutdown_connections 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 >> 172.21.15.107:0/439945549 conn(0x7fc1640f2510 msgr2=0x7fc1640612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 shutdown_connections 2024-02-10T08:39:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 -- 172.21.15.107:0/439945549 wait complete. 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:02.056+0000 7fc169d2e700 1 librados: shutdown 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:39:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:00.748664+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:39:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:00.751580+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:39:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.283123+0000 mgr.y (mgr.14553) 258 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.393224+0000 mgr.y (mgr.14553) 259 : cluster [DBG] pgmap v347: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.635705+0000 osd.3 (osd.3) 129 : cluster [DBG] 25.5 scrub starts 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.637105+0000 osd.3 (osd.3) 130 : cluster [DBG] 25.5 scrub ok 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.907070+0000 mon.a (mon.0) 1684 : audit [DBG] from='client.? v1:172.21.15.107:0/439945549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.955679+0000 mon.a (mon.0) 1685 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.955764+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.955820+0000 mon.a (mon.0) 1687 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.955878+0000 mon.a (mon.0) 1688 : audit [INF] from='client.53129 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.955930+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.956286+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.956401+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.956515+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.956733+0000 mon.a (mon.0) 1691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:01.957082+0000 mon.a (mon.0) 1692 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:02.101546+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:00.748664+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:00.751580+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.283123+0000 mgr.y (mgr.14553) 258 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.393224+0000 mgr.y (mgr.14553) 259 : cluster [DBG] pgmap v347: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.635705+0000 osd.3 (osd.3) 129 : cluster [DBG] 25.5 scrub starts 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.637105+0000 osd.3 (osd.3) 130 : cluster [DBG] 25.5 scrub ok 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.907070+0000 mon.a (mon.0) 1684 : audit [DBG] from='client.? v1:172.21.15.107:0/439945549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.955679+0000 mon.a (mon.0) 1685 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.955764+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.955820+0000 mon.a (mon.0) 1687 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.955878+0000 mon.a (mon.0) 1688 : audit [INF] from='client.53129 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:39:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.955930+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.956286+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.956401+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.956515+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.956733+0000 mon.a (mon.0) 1691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:01.957082+0000 mon.a (mon.0) 1692 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:02.101546+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:00.748664+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:00.751580+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.283123+0000 mgr.y (mgr.14553) 258 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.393224+0000 mgr.y (mgr.14553) 259 : cluster [DBG] pgmap v347: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.635705+0000 osd.3 (osd.3) 129 : cluster [DBG] 25.5 scrub starts 2024-02-10T08:39:02.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.637105+0000 osd.3 (osd.3) 130 : cluster [DBG] 25.5 scrub ok 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.907070+0000 mon.a (mon.0) 1684 : audit [DBG] from='client.? v1:172.21.15.107:0/439945549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.955679+0000 mon.a (mon.0) 1685 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.955764+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.955820+0000 mon.a (mon.0) 1687 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.955878+0000 mon.a (mon.0) 1688 : audit [INF] from='client.53129 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.955930+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.956286+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.956401+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.956515+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.956733+0000 mon.a (mon.0) 1691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:01.957082+0000 mon.a (mon.0) 1692 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:02.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:02.101546+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.737800+0000 osd.6 (osd.6) 99 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:39:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.740682+0000 osd.6 (osd.6) 100 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.761999+0000 osd.5 (osd.5) 123 : cluster [DBG] 25.10 scrub starts 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:01.765005+0000 osd.5 (osd.5) 124 : cluster [DBG] 25.10 scrub ok 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:02.249530+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:02.250712+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:02.952225+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:02.957491+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:02.957576+0000 mon.a (mon.0) 1695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]': finished 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.102443+0000 mon.b (mon.1) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.103005+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.190183+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.191064+0000 mon.a (mon.0) 1697 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.206567+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.206878+0000 mon.a (mon.0) 1698 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.207613+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.207849+0000 mon.a (mon.0) 1699 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.208192+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.208424+0000 mon.a (mon.0) 1700 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.737800+0000 osd.6 (osd.6) 99 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.740682+0000 osd.6 (osd.6) 100 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.761999+0000 osd.5 (osd.5) 123 : cluster [DBG] 25.10 scrub starts 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:01.765005+0000 osd.5 (osd.5) 124 : cluster [DBG] 25.10 scrub ok 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:02.249530+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:02.250712+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:02.952225+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:02.957491+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:02.957576+0000 mon.a (mon.0) 1695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]': finished 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.102443+0000 mon.b (mon.1) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:03.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.103005+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.190183+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.191064+0000 mon.a (mon.0) 1697 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.206567+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.206878+0000 mon.a (mon.0) 1698 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.207613+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.207849+0000 mon.a (mon.0) 1699 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.208192+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.208424+0000 mon.a (mon.0) 1700 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.737800+0000 osd.6 (osd.6) 99 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:39:03.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.740682+0000 osd.6 (osd.6) 100 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.761999+0000 osd.5 (osd.5) 123 : cluster [DBG] 25.10 scrub starts 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:01.765005+0000 osd.5 (osd.5) 124 : cluster [DBG] 25.10 scrub ok 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:02.249530+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:02.250712+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:02.952225+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:02.957491+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? v1:172.21.15.107:0/3883710145' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi107-112403-35"}]': finished 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:02.957576+0000 mon.a (mon.0) 1695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-27", "mode": "writeback"}]': finished 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.102443+0000 mon.b (mon.1) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.103005+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.190183+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.191064+0000 mon.a (mon.0) 1697 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.206567+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.206878+0000 mon.a (mon.0) 1698 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.207613+0000 mon.c (mon.2) 288 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.207849+0000 mon.a (mon.0) 1699 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:03.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.208192+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.208424+0000 mon.a (mon.0) 1700 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:03.643 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:03.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:03.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:03.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:03.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:03.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:04.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:04.094 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:04.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:04.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:04.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:04.247 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:04.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:04.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:04.397 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:04.420 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:04.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:04.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.250582+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:39:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.251581+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:39:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.393599+0000 mgr.y (mgr.14553) 260 : cluster [DBG] pgmap v350: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.544853+0000 osd.3 (osd.3) 131 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.545851+0000 osd.3 (osd.3) 132 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.746925+0000 osd.4 (osd.4) 114 : cluster [DBG] 25.1c scrub starts 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.747846+0000 osd.4 (osd.4) 115 : cluster [DBG] 25.1c scrub ok 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.959275+0000 mon.a (mon.0) 1701 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.959388+0000 mon.a (mon.0) 1702 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.959457+0000 mon.a (mon.0) 1703 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:03.959520+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.964982+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.965648+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.966309+0000 mon.a (mon.0) 1705 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.966470+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.967226+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.967517+0000 mon.a (mon.0) 1706 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.984116+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:03.984456+0000 mon.a (mon.0) 1707 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.250582+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.251581+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.393599+0000 mgr.y (mgr.14553) 260 : cluster [DBG] pgmap v350: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.544853+0000 osd.3 (osd.3) 131 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.545851+0000 osd.3 (osd.3) 132 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:39:04.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.746925+0000 osd.4 (osd.4) 114 : cluster [DBG] 25.1c scrub starts 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.747846+0000 osd.4 (osd.4) 115 : cluster [DBG] 25.1c scrub ok 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.959275+0000 mon.a (mon.0) 1701 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.959388+0000 mon.a (mon.0) 1702 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.959457+0000 mon.a (mon.0) 1703 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:03.959520+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.964982+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.965648+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.966309+0000 mon.a (mon.0) 1705 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.966470+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.967226+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.967517+0000 mon.a (mon.0) 1706 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.984116+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:03.984456+0000 mon.a (mon.0) 1707 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.250582+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.251581+0000 osd.7 (osd.7) 104 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.393599+0000 mgr.y (mgr.14553) 260 : cluster [DBG] pgmap v350: 250 pgs: 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.544853+0000 osd.3 (osd.3) 131 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.545851+0000 osd.3 (osd.3) 132 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.746925+0000 osd.4 (osd.4) 114 : cluster [DBG] 25.1c scrub starts 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.747846+0000 osd.4 (osd.4) 115 : cluster [DBG] 25.1c scrub ok 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.959275+0000 mon.a (mon.0) 1701 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.959388+0000 mon.a (mon.0) 1702 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:04.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.959457+0000 mon.a (mon.0) 1703 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:03.959520+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.964982+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.965648+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.966309+0000 mon.a (mon.0) 1705 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.966470+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.967226+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.967517+0000 mon.a (mon.0) 1706 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.984116+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:03.984456+0000 mon.a (mon.0) 1707 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:04.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:04.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:04.906 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:04.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:05.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:05.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 -- 172.21.15.107:0/4246625848 >> v1:172.21.15.107:6790/0 conn(0x7f608006b1b0 legacy=0x7f608006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:05.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 -- 172.21.15.107:0/4246625848 shutdown_connections 2024-02-10T08:39:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 -- 172.21.15.107:0/4246625848 >> 172.21.15.107:0/4246625848 conn(0x7f60800f3640 msgr2=0x7f60800f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 -- 172.21.15.107:0/4246625848 shutdown_connections 2024-02-10T08:39:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 -- 172.21.15.107:0/4246625848 wait complete. 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 librados: starting msgr at 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.011+0000 7f6087cf5700 1 librados: starting objecter 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 Processor -- start 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 -- start start 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 librados: setting wanted keys 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 librados: calling monclient init 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60801062f0 con 0x7f608006b1b0 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f608011aaa0 con 0x7f608006bca0 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6087cf5700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f608011bc80 con 0x7f6080106590 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6086292700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6080106590 0x7f6080118370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44552/0 (socket says 172.21.15.107:44552) 2024-02-10T08:39:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6086292700 1 -- 172.21.15.107:0/1783611761 learned_addr learned my addr 172.21.15.107:0/1783611761 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6085a91700 1 --1- 172.21.15.107:0/1783611761 >> v1:172.21.15.107:6789/0 conn(0x7f608006b1b0 0x7f6080105740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60834/0 (socket says 172.21.15.107:60834) 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter ms_handle_connect 0x7f608006b1b0 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter ms_handle_connect 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter ms_handle_connect 0x7f608006bca0 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799162405 0 0) 0x7f608011bc80 con 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c002220 con 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638236582 0 0) 0x7f608011aaa0 con 0x7f608006bca0 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f608011bc80 con 0x7f608006bca0 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596831954 0 0) 0x7f606c002220 con 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f608011aaa0 con 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.012+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6078001c80 con 0x7f6080106590 2024-02-10T08:39:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816183532 0 0) 0x7f608011bc80 con 0x7f608006bca0 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f606c002220 con 0x7f608006bca0 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f60700017f0 con 0x7f608006bca0 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321982851 0 0) 0x7f608011aaa0 con 0x7f6080106590 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 >> v1:172.21.15.195:6789/0 conn(0x7f608006bca0 legacy=0x7f6080114c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 >> v1:172.21.15.107:6789/0 conn(0x7f608006b1b0 legacy=0x7f6080105740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608011ce80 con 0x7f6080106590 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608011be70 con 0x7f6080106590 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6087cf5700 10 client.53150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f608011c2e0 con 0x7f6080106590 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6087cf5700 1 librados: init done 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6078002010 con 0x7f6080106590 2024-02-10T08:39:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.013+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6078005f40 con 0x7f6080106590 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.014+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f60780025f0 con 0x7f6080106590 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.014+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6064004f20 con 0x7f6080106590 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(251..251 src has 50..251) v4 ==== 9583+0+0 (unknown 4020184481 0 0) 0x7f6078079810 con 0x7f6080106590 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 10 client.53150.objecter ms_dispatch 0x7f6080001550 osd_map(251..251 src has 50..251) v4 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 3 client.53150.objecter handle_osd_map got epochs [251,251] > 0 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 3 client.53150.objecter handle_osd_map decoding full epoch 251 2024-02-10T08:39:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 20 client.53150.objecter dump_active .. 0 homeless 2024-02-10T08:39:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.015+0000 7f6076ffd700 10 client.53150.objecter ms_handle_connect 0x7f606c063bb0 2024-02-10T08:39:05.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.020+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f607804c630 con 0x7f6080106590 2024-02-10T08:39:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:05.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.203+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f60640051d0 con 0x7f6080106590 2024-02-10T08:39:05.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:39:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:05.493 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:05.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.365718+0000 osd.2 (osd.2) 91 : cluster [DBG] 25.1f scrub starts 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.369175+0000 osd.2 (osd.2) 92 : cluster [DBG] 25.1f scrub ok 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.204778+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.107:0/1783611761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.209719+0000 mon.a (mon.0) 1708 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.209863+0000 mon.a (mon.0) 1709 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.209927+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-10T08:39:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.210718+0000 mon.a (mon.0) 1711 : audit [INF] from='client.53150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.210961+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:05.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.211356+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.211605+0000 mon.a (mon.0) 1712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.365718+0000 osd.2 (osd.2) 91 : cluster [DBG] 25.1f scrub starts 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.369175+0000 osd.2 (osd.2) 92 : cluster [DBG] 25.1f scrub ok 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.204778+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.107:0/1783611761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.209719+0000 mon.a (mon.0) 1708 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.209863+0000 mon.a (mon.0) 1709 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.209927+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.210718+0000 mon.a (mon.0) 1711 : audit [INF] from='client.53150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.210961+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.211356+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.211605+0000 mon.a (mon.0) 1712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.365718+0000 osd.2 (osd.2) 91 : cluster [DBG] 25.1f scrub starts 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.369175+0000 osd.2 (osd.2) 92 : cluster [DBG] 25.1f scrub ok 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.204778+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.107:0/1783611761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.209719+0000 mon.a (mon.0) 1708 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.209863+0000 mon.a (mon.0) 1709 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.209927+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.210718+0000 mon.a (mon.0) 1711 : audit [INF] from='client.53150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.210961+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.211356+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.211605+0000 mon.a (mon.0) 1712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]: dispatch 2024-02-10T08:39:05.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:05.622 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:05.701 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:05.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:05.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:05.853 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:05] "GET /metrics HTTP/1.1" 200 187957 "" "Prometheus/2.33.4" 2024-02-10T08:39:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:05.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.974+0000 7f6076ffd700 1 -- 172.21.15.107:0/1783611761 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v252) v1 ==== 101+0+0 (unknown 1603157192 0 0) 0x7f607804aa40 con 0x7f6080106590 2024-02-10T08:39:05.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.984+0000 7f6087cf5700 10 librados: watch_flush enter 2024-02-10T08:39:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.986+0000 7f6087cf5700 10 librados: watch_flush exit 2024-02-10T08:39:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.986+0000 7f6087cf5700 20 client.53150.objecter shutdown clearing up homeless session... 2024-02-10T08:39:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.986+0000 7f6087cf5700 10 client.53150.objecter successfully canceled tick 2024-02-10T08:39:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.986+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 >> v1:172.21.15.107:6800/2634842009 conn(0x7f606c063bb0 legacy=0x7f606c066070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:05.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.986+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 >> v1:172.21.15.107:6790/0 conn(0x7f6080106590 legacy=0x7f6080118370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:05.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.989+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 shutdown_connections 2024-02-10T08:39:05.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.989+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 >> 172.21.15.107:0/1783611761 conn(0x7f60800f3640 msgr2=0x7f60800f5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:05.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.990+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 shutdown_connections 2024-02-10T08:39:05.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.990+0000 7f6087cf5700 1 -- 172.21.15.107:0/1783611761 wait complete. 2024-02-10T08:39:05.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:05.990+0000 7f6087cf5700 1 librados: shutdown 2024-02-10T08:39:05.993 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:39:06.105 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:06.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:06.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T08:39:06.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:06.238 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:06.257 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:06.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:06.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.781788+0000 osd.5 (osd.5) 125 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.785234+0000 osd.5 (osd.5) 126 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.933199+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:04.936170+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.277804+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.278315+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.278551+0000 mon.a (mon.0) 1713 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.393925+0000 mgr.y (mgr.14553) 261 : cluster [DBG] pgmap v353: 258 pgs: 8 unknown, 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.394260+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.395221+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.789125+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.790001+0000 osd.4 (osd.4) 117 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.960604+0000 mon.a (mon.0) 1715 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.960632+0000 mon.a (mon.0) 1716 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.968080+0000 mon.a (mon.0) 1717 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.968187+0000 mon.a (mon.0) 1718 : audit [INF] from='client.53150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.781788+0000 osd.5 (osd.5) 125 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.785234+0000 osd.5 (osd.5) 126 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.933199+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:04.936170+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.277804+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.278315+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.278551+0000 mon.a (mon.0) 1713 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.393925+0000 mgr.y (mgr.14553) 261 : cluster [DBG] pgmap v353: 258 pgs: 8 unknown, 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.394260+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.395221+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.789125+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.790001+0000 osd.4 (osd.4) 117 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.960604+0000 mon.a (mon.0) 1715 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.960632+0000 mon.a (mon.0) 1716 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.968080+0000 mon.a (mon.0) 1717 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.968187+0000 mon.a (mon.0) 1718 : audit [INF] from='client.53150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:06.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.968257+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.968352+0000 mon.a (mon.0) 1720 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.968416+0000 mon.a (mon.0) 1721 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.968481+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.982077+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.984174+0000 mon.a (mon.0) 1723 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.987179+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.989494+0000 mon.b (mon.1) 406 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.992230+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:05.993327+0000 mon.a (mon.0) 1724 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.968257+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.968352+0000 mon.a (mon.0) 1720 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.968416+0000 mon.a (mon.0) 1721 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.968481+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.982077+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.984174+0000 mon.a (mon.0) 1723 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.987179+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.989494+0000 mon.b (mon.1) 406 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.992230+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:05.993327+0000 mon.a (mon.0) 1724 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.389 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:06.534 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.781788+0000 osd.5 (osd.5) 125 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.785234+0000 osd.5 (osd.5) 126 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.933199+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:04.936170+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.277804+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.278315+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.278551+0000 mon.a (mon.0) 1713 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.393925+0000 mgr.y (mgr.14553) 261 : cluster [DBG] pgmap v353: 258 pgs: 8 unknown, 250 active+clean; 8.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.394260+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.395221+0000 mon.a (mon.0) 1714 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.789125+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.790001+0000 osd.4 (osd.4) 117 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.960604+0000 mon.a (mon.0) 1715 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.960632+0000 mon.a (mon.0) 1716 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.968080+0000 mon.a (mon.0) 1717 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi107-112403-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.968187+0000 mon.a (mon.0) 1718 : audit [INF] from='client.53150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.968257+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-27"}]': finished 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.968352+0000 mon.a (mon.0) 1720 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pg_num","val":"11"}]': finished 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.968416+0000 mon.a (mon.0) 1721 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.968481+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.982077+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.984174+0000 mon.a (mon.0) 1723 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.987179+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.989494+0000 mon.b (mon.1) 406 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:06.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.992230+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:05.993327+0000 mon.a (mon.0) 1724 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T08:39:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:06.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:06.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:06.671 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:06.692 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:06.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:06.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:06.825 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:06.845 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:06.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:06.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:07.020 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:07.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:07.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:07.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.785158+0000 osd.5 (osd.5) 127 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.788114+0000 osd.5 (osd.5) 128 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.944942+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:05.948575+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:06.767935+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:06.770846+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:39:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:06.968571+0000 mon.a (mon.0) 1725 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:06.968705+0000 mon.a (mon.0) 1726 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:06.968795+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:06.984611+0000 mon.b (mon.1) 408 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:06.984809+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:06.991385+0000 mon.a (mon.0) 1728 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.785158+0000 osd.5 (osd.5) 127 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.788114+0000 osd.5 (osd.5) 128 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.944942+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:05.948575+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:06.767935+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:06.770846+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:06.968571+0000 mon.a (mon.0) 1725 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:06.968705+0000 mon.a (mon.0) 1726 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:06.968795+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-10T08:39:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:06.984611+0000 mon.b (mon.1) 408 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:06.984809+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:06.991385+0000 mon.a (mon.0) 1728 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:07.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.785158+0000 osd.5 (osd.5) 127 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.788114+0000 osd.5 (osd.5) 128 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.944942+0000 osd.1 (osd.1) 129 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:05.948575+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:06.767935+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:06.770846+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:06.968571+0000 mon.a (mon.0) 1725 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:06.968705+0000 mon.a (mon.0) 1726 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi107-112443-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T08:39:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:06.968795+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-10T08:39:07.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:06.984611+0000 mon.b (mon.1) 408 : audit [INF] from='client.? v1:172.21.15.107:0/3596812208' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:07.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:06.984809+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:07.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:06.991385+0000 mon.a (mon.0) 1728 : audit [INF] from='client.47857 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]: dispatch 2024-02-10T08:39:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:08.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:08.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:08.467 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:08.485 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:08.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:06.962247+0000 osd.1 (osd.1) 131 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:39:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:06.965148+0000 osd.1 (osd.1) 132 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:07.394379+0000 mgr.y (mgr.14553) 262 : cluster [DBG] pgmap v356: 226 pgs: 21 peering, 8 creating+peering, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.970743+0000 mon.a (mon.0) 1729 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:07.970841+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.972749+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.973073+0000 mon.a (mon.0) 1731 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.978881+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.979471+0000 mon.a (mon.0) 1732 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:07.991518+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.000004+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.000282+0000 mon.a (mon.0) 1733 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.001575+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.001872+0000 mon.a (mon.0) 1734 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.002315+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.002610+0000 mon.a (mon.0) 1735 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:08.615 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:06.962247+0000 osd.1 (osd.1) 131 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:06.965148+0000 osd.1 (osd.1) 132 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:07.394379+0000 mgr.y (mgr.14553) 262 : cluster [DBG] pgmap v356: 226 pgs: 21 peering, 8 creating+peering, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.970743+0000 mon.a (mon.0) 1729 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:07.970841+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.972749+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.973073+0000 mon.a (mon.0) 1731 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.978881+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.979471+0000 mon.a (mon.0) 1732 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:07.991518+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.000004+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.000282+0000 mon.a (mon.0) 1733 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.001575+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.001872+0000 mon.a (mon.0) 1734 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.002315+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.002610+0000 mon.a (mon.0) 1735 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:06.962247+0000 osd.1 (osd.1) 131 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:06.965148+0000 osd.1 (osd.1) 132 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:07.394379+0000 mgr.y (mgr.14553) 262 : cluster [DBG] pgmap v356: 226 pgs: 21 peering, 8 creating+peering, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.970743+0000 mon.a (mon.0) 1729 : audit [INF] from='client.47857 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-45"}]': finished 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:07.970841+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.972749+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.973073+0000 mon.a (mon.0) 1731 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.978881+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.979471+0000 mon.a (mon.0) 1732 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:07.991518+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.000004+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.000282+0000 mon.a (mon.0) 1733 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.001575+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.001872+0000 mon.a (mon.0) 1734 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.002315+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.619 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.002610+0000 mon.a (mon.0) 1735 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:08.670 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:08.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:08.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:08.801 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:08.820 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:08.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:08.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:39:08.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:08.950 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:08.968 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:08.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:09.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:09.099 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:09.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:09.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:39:09.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.256832+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T08:39:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.627974+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.629115+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.671398+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.671571+0000 osd.3 (osd.3) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.763712+0000 osd.6 (osd.6) 103 : cluster [DBG] 25.18 deep-scrub starts 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.766624+0000 osd.6 (osd.6) 104 : cluster [DBG] 25.18 deep-scrub ok 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.974598+0000 mon.a (mon.0) 1737 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.974751+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.974846+0000 mon.a (mon.0) 1739 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:08.974911+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.979988+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.980517+0000 mon.a (mon.0) 1741 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.982338+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.982974+0000 mon.a (mon.0) 1742 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.988841+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.989428+0000 mon.a (mon.0) 1743 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:08.992924+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.256832+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.627974+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.629115+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.671398+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.671571+0000 osd.3 (osd.3) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.763712+0000 osd.6 (osd.6) 103 : cluster [DBG] 25.18 deep-scrub starts 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.766624+0000 osd.6 (osd.6) 104 : cluster [DBG] 25.18 deep-scrub ok 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.974598+0000 mon.a (mon.0) 1737 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.974751+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.974846+0000 mon.a (mon.0) 1739 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:08.974911+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.979988+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.980517+0000 mon.a (mon.0) 1741 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.982338+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.982974+0000 mon.a (mon.0) 1742 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.988841+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.989428+0000 mon.a (mon.0) 1743 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:08.992924+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.256832+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-02-10T08:39:09.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.627974+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.629115+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.671398+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.671571+0000 osd.3 (osd.3) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.763712+0000 osd.6 (osd.6) 103 : cluster [DBG] 25.18 deep-scrub starts 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.766624+0000 osd.6 (osd.6) 104 : cluster [DBG] 25.18 deep-scrub ok 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.974598+0000 mon.a (mon.0) 1737 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.974751+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.974846+0000 mon.a (mon.0) 1739 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi107-112411-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:08.974911+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.979988+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.107:0/2407674176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.980517+0000 mon.a (mon.0) 1741 : audit [INF] from='client.53144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]: dispatch 2024-02-10T08:39:09.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.982338+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.982974+0000 mon.a (mon.0) 1742 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:09.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.988841+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.989428+0000 mon.a (mon.0) 1743 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:09.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:08.992924+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:09.765 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:09.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:09.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:09.897 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:09.917 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:09.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:10.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:10.053 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:09.394829+0000 mgr.y (mgr.14553) 263 : cluster [DBG] pgmap v359: 250 pgs: 21 peering, 32 unknown, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:09.397176+0000 osd.7 (osd.7) 105 : cluster [DBG] 25.13 scrub starts 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:09.400648+0000 osd.7 (osd.7) 106 : cluster [DBG] 25.13 scrub ok 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:09.973811+0000 mon.a (mon.0) 1744 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:09.993761+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:09.996809+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.221508+0000 mon.a (mon.0) 1745 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:10.221582+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.222217+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.222303+0000 mon.a (mon.0) 1747 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.222798+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.223357+0000 mon.a (mon.0) 1748 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.223653+0000 mon.a (mon.0) 1749 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.226044+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.227274+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:10.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.227647+0000 mon.a (mon.0) 1751 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.561+0000 7fd743d88700 1 -- 172.21.15.107:0/2564919144 >> v1:172.21.15.107:6790/0 conn(0x7fd73c103250 legacy=0x7fd73c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:10.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.561+0000 7fd743d88700 1 -- 172.21.15.107:0/2564919144 shutdown_connections 2024-02-10T08:39:10.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.561+0000 7fd743d88700 1 -- 172.21.15.107:0/2564919144 >> 172.21.15.107:0/2564919144 conn(0x7fd73c0f2510 msgr2=0x7fd73c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:10.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.561+0000 7fd743d88700 1 -- 172.21.15.107:0/2564919144 shutdown_connections 2024-02-10T08:39:10.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.561+0000 7fd743d88700 1 -- 172.21.15.107:0/2564919144 wait complete. 2024-02-10T08:39:10.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 librados: starting msgr at 2024-02-10T08:39:10.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 librados: starting objecter 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 Processor -- start 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 -- start start 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 librados: setting wanted keys 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 librados: calling monclient init 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c116440 con 0x7fd73c103250 2024-02-10T08:39:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c117620 con 0x7fd73c0ff710 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd743d88700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c118820 con 0x7fd73c0fc910 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd741b24700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd73c0fc910 0x7fd73c0f8c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48156/0 (socket says 172.21.15.107:48156) 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd741b24700 1 -- 172.21.15.107:0/2446123377 learned_addr learned my addr 172.21.15.107:0/2446123377 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd742325700 1 --1- 172.21.15.107:0/2446123377 >> v1:172.21.15.107:6789/0 conn(0x7fd73c103250 0x7fd73c114b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52072/0 (socket says 172.21.15.107:52072) 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd732ffd700 10 client.?.objecter ms_handle_connect 0x7fd73c0fc910 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.562+0000 7fd732ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 10 client.?.objecter ms_handle_connect 0x7fd73c103250 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 10 client.?.objecter ms_handle_connect 0x7fd73c0ff710 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420039825 0 0) 0x7fd73c118820 con 0x7fd73c0fc910 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd728003630 con 0x7fd73c0fc910 2024-02-10T08:39:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529142498 0 0) 0x7fd73c117620 con 0x7fd73c0ff710 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c118820 con 0x7fd73c0ff710 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2405447584 0 0) 0x7fd728003630 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c117620 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd72c001980 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198167390 0 0) 0x7fd73c118820 con 0x7fd73c0ff710 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd728003630 con 0x7fd73c0ff710 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd738001860 con 0x7fd73c0ff710 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240012215 0 0) 0x7fd73c117620 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 >> v1:172.21.15.195:6789/0 conn(0x7fd73c0ff710 legacy=0x7fd73c0f9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 >> v1:172.21.15.107:6789/0 conn(0x7fd73c103250 legacy=0x7fd73c114b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c119a20 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c116630 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd743d88700 10 client.53168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c116b20 con 0x7fd73c0fc910 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.563+0000 7fd743d88700 1 librados: init done 2024-02-10T08:39:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.564+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd72c001d10 con 0x7fd73c0fc910 2024-02-10T08:39:10.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.564+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd72c005c30 con 0x7fd73c0fc910 2024-02-10T08:39:10.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.565+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd720004f20 con 0x7fd73c0fc910 2024-02-10T08:39:10.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.565+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd72c018b80 con 0x7fd73c0fc910 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(256..256 src has 50..256) v4 ==== 8757+0+0 (unknown 416284978 0 0) 0x7fd72c0791d0 con 0x7fd73c0fc910 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 10 client.53168.objecter ms_dispatch 0x7fd73c001550 osd_map(256..256 src has 50..256) v4 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 3 client.53168.objecter handle_osd_map got epochs [256,256] > 0 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 3 client.53168.objecter handle_osd_map decoding full epoch 256 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 20 client.53168.objecter dump_active .. 0 homeless 2024-02-10T08:39:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.566+0000 7fd732ffd700 10 client.53168.objecter ms_handle_connect 0x7fd728065d50 2024-02-10T08:39:10.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.570+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd72c049500 con 0x7fd73c0fc910 2024-02-10T08:39:10.648 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:39:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:39:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:39:10.649 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:39:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:09.394829+0000 mgr.y (mgr.14553) 263 : cluster [DBG] pgmap v359: 250 pgs: 21 peering, 32 unknown, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:09.397176+0000 osd.7 (osd.7) 105 : cluster [DBG] 25.13 scrub starts 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:09.400648+0000 osd.7 (osd.7) 106 : cluster [DBG] 25.13 scrub ok 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:09.973811+0000 mon.a (mon.0) 1744 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:09.993761+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:09.996809+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.221508+0000 mon.a (mon.0) 1745 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:10.221582+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.222217+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.222303+0000 mon.a (mon.0) 1747 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.222798+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.223357+0000 mon.a (mon.0) 1748 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.223653+0000 mon.a (mon.0) 1749 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.226044+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.227274+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.227647+0000 mon.a (mon.0) 1751 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:09.394829+0000 mgr.y (mgr.14553) 263 : cluster [DBG] pgmap v359: 250 pgs: 21 peering, 32 unknown, 197 active+clean; 7.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:09.397176+0000 osd.7 (osd.7) 105 : cluster [DBG] 25.13 scrub starts 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:09.400648+0000 osd.7 (osd.7) 106 : cluster [DBG] 25.13 scrub ok 2024-02-10T08:39:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:09.973811+0000 mon.a (mon.0) 1744 : audit [INF] from='client.53144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi107-112403-36"}]': finished 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:09.993761+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:09.996809+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.221508+0000 mon.a (mon.0) 1745 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:10.221582+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.222217+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.222303+0000 mon.a (mon.0) 1747 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.222798+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.223357+0000 mon.a (mon.0) 1748 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.223653+0000 mon.a (mon.0) 1749 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.226044+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.227274+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:10.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.227647+0000 mon.a (mon.0) 1751 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:10.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.748+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fd7200051d0 con 0x7fd73c0fc910 2024-02-10T08:39:10.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.957+0000 7fd732ffd700 1 -- 172.21.15.107:0/2446123377 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v257) v1 ==== 113+0+0 (unknown 1138526874 0 0) 0x7fd72c04bec0 con 0x7fd73c0fc910 2024-02-10T08:39:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.966+0000 7fd743d88700 10 librados: watch_flush enter 2024-02-10T08:39:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.966+0000 7fd743d88700 10 librados: watch_flush exit 2024-02-10T08:39:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.966+0000 7fd743d88700 20 client.53168.objecter shutdown clearing up homeless session... 2024-02-10T08:39:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.966+0000 7fd743d88700 10 client.53168.objecter successfully canceled tick 2024-02-10T08:39:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.966+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd728065d50 legacy=0x7fd728068210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.967+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 >> v1:172.21.15.107:6790/0 conn(0x7fd73c0fc910 legacy=0x7fd73c0f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.967+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 shutdown_connections 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.967+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 >> 172.21.15.107:0/2446123377 conn(0x7fd73c0f2510 msgr2=0x7fd73c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.968+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 shutdown_connections 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.971+0000 7fd743d88700 1 -- 172.21.15.107:0/2446123377 wait complete. 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:10.971+0000 7fd743d88700 1 librados: shutdown 2024-02-10T08:39:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T08:39:11.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:39:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:10.419907+0000 osd.7 (osd.7) 107 : cluster [DBG] 25.17 scrub starts 2024-02-10T08:39:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:10.422663+0000 osd.7 (osd.7) 108 : cluster [DBG] 25.17 scrub ok 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.749487+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.107:0/2446123377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.749753+0000 mon.a (mon.0) 1752 : audit [INF] from='client.53168 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.956676+0000 mon.a (mon.0) 1753 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.956764+0000 mon.a (mon.0) 1754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.956840+0000 mon.a (mon.0) 1755 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.956914+0000 mon.a (mon.0) 1756 : audit [INF] from='client.53168 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:10.956973+0000 mon.a (mon.0) 1757 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.959256+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.962331+0000 mon.a (mon.0) 1758 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.963227+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.972613+0000 mon.a (mon.0) 1759 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:10.995246+0000 mon.b (mon.1) 425 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:10.419907+0000 osd.7 (osd.7) 107 : cluster [DBG] 25.17 scrub starts 2024-02-10T08:39:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:10.422663+0000 osd.7 (osd.7) 108 : cluster [DBG] 25.17 scrub ok 2024-02-10T08:39:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.749487+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.107:0/2446123377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.749753+0000 mon.a (mon.0) 1752 : audit [INF] from='client.53168 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.956676+0000 mon.a (mon.0) 1753 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.956764+0000 mon.a (mon.0) 1754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.956840+0000 mon.a (mon.0) 1755 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.956914+0000 mon.a (mon.0) 1756 : audit [INF] from='client.53168 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:10.956973+0000 mon.a (mon.0) 1757 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.959256+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.962331+0000 mon.a (mon.0) 1758 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.963227+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.972613+0000 mon.a (mon.0) 1759 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:10.995246+0000 mon.b (mon.1) 425 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:10.419907+0000 osd.7 (osd.7) 107 : cluster [DBG] 25.17 scrub starts 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:10.422663+0000 osd.7 (osd.7) 108 : cluster [DBG] 25.17 scrub ok 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.749487+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.107:0/2446123377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.749753+0000 mon.a (mon.0) 1752 : audit [INF] from='client.53168 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.956676+0000 mon.a (mon.0) 1753 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi107-112411-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.956764+0000 mon.a (mon.0) 1754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.956840+0000 mon.a (mon.0) 1755 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi107-112403-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.956914+0000 mon.a (mon.0) 1756 : audit [INF] from='client.53168 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:10.956973+0000 mon.a (mon.0) 1757 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.959256+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.962331+0000 mon.a (mon.0) 1758 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.963227+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.972613+0000 mon.a (mon.0) 1759 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]: dispatch 2024-02-10T08:39:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:10.995246+0000 mon.b (mon.1) 425 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:11.802 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:11.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:11.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:12.032 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:12.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:12.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:12.164 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:12.265 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:12.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:12.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:11.293139+0000 mgr.y (mgr.14553) 264 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.395724+0000 mgr.y (mgr.14553) 265 : cluster [DBG] pgmap v362: 258 pgs: 8 creating+peering, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:11.395997+0000 mon.c (mon.2) 296 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:11.396931+0000 mon.a (mon.0) 1760 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.787960+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.790860+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.954080+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:11.996533+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.202459+0000 mon.a (mon.0) 1762 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]': finished 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.202532+0000 mon.a (mon.0) 1763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.202601+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:11.293139+0000 mgr.y (mgr.14553) 264 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.395724+0000 mgr.y (mgr.14553) 265 : cluster [DBG] pgmap v362: 258 pgs: 8 creating+peering, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:11.395997+0000 mon.c (mon.2) 296 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:11.396931+0000 mon.a (mon.0) 1760 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.787960+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.790860+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.954080+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:11.996533+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:12.396 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.202459+0000 mon.a (mon.0) 1762 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]': finished 2024-02-10T08:39:12.397 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.202532+0000 mon.a (mon.0) 1763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:39:12.397 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.202601+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-10T08:39:12.397 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:11.293139+0000 mgr.y (mgr.14553) 264 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.395724+0000 mgr.y (mgr.14553) 265 : cluster [DBG] pgmap v362: 258 pgs: 8 creating+peering, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:11.395997+0000 mon.c (mon.2) 296 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:11.396931+0000 mon.a (mon.0) 1760 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T08:39:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.787960+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.790860+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.954080+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:11.996533+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.202459+0000 mon.a (mon.0) 1762 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-29", "mode": "writeback"}]': finished 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.202532+0000 mon.a (mon.0) 1763 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T08:39:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.202601+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-10T08:39:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.656448+0000 osd.5 (osd.5) 129 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:11.660044+0000 osd.5 (osd.5) 130 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.082132+0000 osd.1 (osd.1) 133 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.085255+0000 osd.1 (osd.1) 134 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.269438+0000 mon.a (mon.0) 1765 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.273101+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.274648+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.343038+0000 osd.2 (osd.2) 93 : cluster [DBG] 25.19 scrub starts 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.343956+0000 osd.2 (osd.2) 94 : cluster [DBG] 25.19 scrub ok 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.537322+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.537431+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.686925+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:39:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.687914+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.961083+0000 mon.a (mon.0) 1767 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.961183+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.961232+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.969137+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.971762+0000 mon.a (mon.0) 1770 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.972400+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.972725+0000 mon.a (mon.0) 1771 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:12.997550+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.656448+0000 osd.5 (osd.5) 129 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:11.660044+0000 osd.5 (osd.5) 130 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:39:13.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.082132+0000 osd.1 (osd.1) 133 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.085255+0000 osd.1 (osd.1) 134 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.269438+0000 mon.a (mon.0) 1765 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.273101+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.274648+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.343038+0000 osd.2 (osd.2) 93 : cluster [DBG] 25.19 scrub starts 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.343956+0000 osd.2 (osd.2) 94 : cluster [DBG] 25.19 scrub ok 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.537322+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.537431+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.686925+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.687914+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.961083+0000 mon.a (mon.0) 1767 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.961183+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.961232+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.969137+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.971762+0000 mon.a (mon.0) 1770 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.972400+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.972725+0000 mon.a (mon.0) 1771 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:12.997550+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:13.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T08:39:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.656448+0000 osd.5 (osd.5) 129 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:39:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:11.660044+0000 osd.5 (osd.5) 130 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.082132+0000 osd.1 (osd.1) 133 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.085255+0000 osd.1 (osd.1) 134 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.269438+0000 mon.a (mon.0) 1765 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.273101+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.274648+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.343038+0000 osd.2 (osd.2) 93 : cluster [DBG] 25.19 scrub starts 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.343956+0000 osd.2 (osd.2) 94 : cluster [DBG] 25.19 scrub ok 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.537322+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.537431+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.686925+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.687914+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.961083+0000 mon.a (mon.0) 1767 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi107-112403-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.961183+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.961232+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.969137+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.971762+0000 mon.a (mon.0) 1770 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]: dispatch 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.972400+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.972725+0000 mon.a (mon.0) 1771 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:13.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:12.997550+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:14.147 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:14.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:14.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:14.283 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:14.305 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:14.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:14.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.670399+0000 osd.5 (osd.5) 131 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:39:14.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:12.673900+0000 osd.5 (osd.5) 132 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:39:14.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.328785+0000 osd.2 (osd.2) 95 : cluster [DBG] 25.1e scrub starts 2024-02-10T08:39:14.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.331944+0000 osd.2 (osd.2) 96 : cluster [DBG] 25.1e scrub ok 2024-02-10T08:39:14.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.396250+0000 mgr.y (mgr.14553) 266 : cluster [DBG] pgmap v365: 258 pgs: 8 unknown, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.494462+0000 osd.3 (osd.3) 137 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.495336+0000 osd.3 (osd.3) 138 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.959055+0000 mon.a (mon.0) 1772 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:13.963756+0000 mon.a (mon.0) 1773 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:13.963820+0000 mon.a (mon.0) 1774 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:13.963883+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:13.968728+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:13.969142+0000 mon.a (mon.0) 1776 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:13.998385+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.670399+0000 osd.5 (osd.5) 131 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:39:14.384 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:12.673900+0000 osd.5 (osd.5) 132 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.328785+0000 osd.2 (osd.2) 95 : cluster [DBG] 25.1e scrub starts 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.331944+0000 osd.2 (osd.2) 96 : cluster [DBG] 25.1e scrub ok 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.396250+0000 mgr.y (mgr.14553) 266 : cluster [DBG] pgmap v365: 258 pgs: 8 unknown, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.494462+0000 osd.3 (osd.3) 137 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.495336+0000 osd.3 (osd.3) 138 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.959055+0000 mon.a (mon.0) 1772 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:13.963756+0000 mon.a (mon.0) 1773 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:13.963820+0000 mon.a (mon.0) 1774 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:13.963883+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:13.968728+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:13.969142+0000 mon.a (mon.0) 1776 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:13.998385+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:14.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:14.438 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:14.458 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:14.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 -- 172.21.15.107:0/3087135059 >> v1:172.21.15.107:6790/0 conn(0x7ff9e8103250 legacy=0x7ff9e81056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 -- 172.21.15.107:0/3087135059 shutdown_connections 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 -- 172.21.15.107:0/3087135059 >> 172.21.15.107:0/3087135059 conn(0x7ff9e80f2510 msgr2=0x7ff9e80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 -- 172.21.15.107:0/3087135059 shutdown_connections 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 -- 172.21.15.107:0/3087135059 wait complete. 2024-02-10T08:39:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 librados: starting msgr at 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.370+0000 7ff9eec05700 1 librados: starting objecter 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 Processor -- start 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 -- start start 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 librados: setting wanted keys 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 librados: calling monclient init 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9e8117350 con 0x7ff9e80ff710 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9e8118530 con 0x7ff9e80fc910 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9eec05700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9e8119730 con 0x7ff9e8103250 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9ed1a2700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff9e8103250 0x7ff9e8115a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48182/0 (socket says 172.21.15.107:48182) 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9ed1a2700 1 -- 172.21.15.107:0/3443746379 learned_addr learned my addr 172.21.15.107:0/3443746379 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9e8103250 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter ms_handle_connect 0x7ff9e80fc910 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075107306 0 0) 0x7ff9e8119730 con 0x7ff9e8103250 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d8003030 con 0x7ff9e8103250 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456116134 0 0) 0x7ff9e8117350 con 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e8119730 con 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335526511 0 0) 0x7ff9e8118530 con 0x7ff9e80fc910 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e8117350 con 0x7ff9e80fc910 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1442516685 0 0) 0x7ff9e8119730 con 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e8118530 con 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 856046622 0 0) 0x7ff9d8003030 con 0x7ff9e8103250 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e8119730 con 0x7ff9e8103250 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9d4001970 con 0x7ff9e80ff710 2024-02-10T08:39:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9e0002420 con 0x7ff9e8103250 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.371+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745968835 0 0) 0x7ff9e8117350 con 0x7ff9e80fc910 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9d8003030 con 0x7ff9e80fc910 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9dc0018c0 con 0x7ff9e80fc910 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 213464349 0 0) 0x7ff9e8118530 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 >> v1:172.21.15.107:6790/0 conn(0x7ff9e8103250 legacy=0x7ff9e8115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 >> v1:172.21.15.195:6789/0 conn(0x7ff9e80fc910 legacy=0x7ff9e80f9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e811a930 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8117540 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff9d40022a0 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9d4003270 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9eec05700 10 client.27633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81179b0 con 0x7ff9e80ff710 2024-02-10T08:39:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.372+0000 7ff9eec05700 1 librados: init done 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.373+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e805f7a0 con 0x7ff9e80ff710 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.373+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff9d4001e00 con 0x7ff9e80ff710 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(260..260 src has 50..260) v4 ==== 9235+0+0 (unknown 1695583300 0 0) 0x7ff9d40768e0 con 0x7ff9e80ff710 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 10 client.27633.objecter ms_dispatch 0x7ff9e8001550 osd_map(260..260 src has 50..260) v4 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 3 client.27633.objecter handle_osd_map got epochs [260,260] > 0 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 3 client.27633.objecter handle_osd_map decoding full epoch 260 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 20 client.27633.objecter dump_active .. 0 homeless 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.374+0000 7ff9e5ffb700 10 client.27633.objecter ms_handle_connect 0x7ff9d8063d40 2024-02-10T08:39:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.378+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff9d4049860 con 0x7ff9e80ff710 2024-02-10T08:39:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.670399+0000 osd.5 (osd.5) 131 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:39:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:12.673900+0000 osd.5 (osd.5) 132 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:39:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.328785+0000 osd.2 (osd.2) 95 : cluster [DBG] 25.1e scrub starts 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.331944+0000 osd.2 (osd.2) 96 : cluster [DBG] 25.1e scrub ok 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.396250+0000 mgr.y (mgr.14553) 266 : cluster [DBG] pgmap v365: 258 pgs: 8 unknown, 250 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 9.0 KiB/s wr, 17 op/s; 543 B/s, 6 objects/s recovering 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.494462+0000 osd.3 (osd.3) 137 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.495336+0000 osd.3 (osd.3) 138 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.959055+0000 mon.a (mon.0) 1772 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:13.963756+0000 mon.a (mon.0) 1773 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-29"}]': finished 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:13.963820+0000 mon.a (mon.0) 1774 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:13.963883+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:13.968728+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.107:0/1475210826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:13.969142+0000 mon.a (mon.0) 1776 : audit [INF] from='client.47878 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]: dispatch 2024-02-10T08:39:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:13.998385+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:14.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.557+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff9e80f8140 con 0x7ff9e80ff710 2024-02-10T08:39:14.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:14.937 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:14.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:14.975 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.973+0000 7ff9e5ffb700 1 -- 172.21.15.107:0/3443746379 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v261) v1 ==== 87+0+0 (unknown 3603576413 0 0) 0x7ff9d4047c70 con 0x7ff9e80ff710 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 10 librados: watch_flush enter 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 10 librados: watch_flush exit 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 20 client.27633.objecter shutdown clearing up homeless session... 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 10 client.27633.objecter successfully canceled tick 2024-02-10T08:39:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff9d8063d40 legacy=0x7ff9d8066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.976+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 >> v1:172.21.15.107:6789/0 conn(0x7ff9e80ff710 legacy=0x7ff9e80fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.977+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 shutdown_connections 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.977+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 >> 172.21.15.107:0/3443746379 conn(0x7ff9e80f2510 msgr2=0x7ff9e80612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.978+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 shutdown_connections 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.978+0000 7ff9eec05700 1 -- 172.21.15.107:0/3443746379 wait complete. 2024-02-10T08:39:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:14.979+0000 7ff9eec05700 1 librados: shutdown 2024-02-10T08:39:14.985 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:39:15.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:15.072 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:15.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:15.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:15.303 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:15.323 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:15.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:15.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.305218+0000 osd.2 (osd.2) 97 : cluster [DBG] 25.16 scrub starts 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.308721+0000 osd.2 (osd.2) 98 : cluster [DBG] 25.16 scrub ok 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.544602+0000 osd.7 (osd.7) 109 : cluster [DBG] 25.3 scrub starts 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.548087+0000 osd.7 (osd.7) 110 : cluster [DBG] 25.3 scrub ok 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.558969+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.659904+0000 osd.6 (osd.6) 107 : cluster [DBG] 25.0 scrub starts 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.663497+0000 osd.6 (osd.6) 108 : cluster [DBG] 25.0 scrub ok 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.682428+0000 osd.4 (osd.4) 120 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.683239+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.961900+0000 mon.a (mon.0) 1778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.972104+0000 mon.a (mon.0) 1779 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.972236+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:14.972278+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.976942+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.979977+0000 mon.a (mon.0) 1782 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:14.999903+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.010127+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.239899+0000 mon.a (mon.0) 1783 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.240707+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.240948+0000 mon.a (mon.0) 1784 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.241348+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.241583+0000 mon.a (mon.0) 1785 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.305218+0000 osd.2 (osd.2) 97 : cluster [DBG] 25.16 scrub starts 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.308721+0000 osd.2 (osd.2) 98 : cluster [DBG] 25.16 scrub ok 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.544602+0000 osd.7 (osd.7) 109 : cluster [DBG] 25.3 scrub starts 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.548087+0000 osd.7 (osd.7) 110 : cluster [DBG] 25.3 scrub ok 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.558969+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.659904+0000 osd.6 (osd.6) 107 : cluster [DBG] 25.0 scrub starts 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.663497+0000 osd.6 (osd.6) 108 : cluster [DBG] 25.0 scrub ok 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.682428+0000 osd.4 (osd.4) 120 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.683239+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.961900+0000 mon.a (mon.0) 1778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.972104+0000 mon.a (mon.0) 1779 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.972236+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:14.972278+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.976942+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.979977+0000 mon.a (mon.0) 1782 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:14.999903+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.010127+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.239899+0000 mon.a (mon.0) 1783 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.240707+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.240948+0000 mon.a (mon.0) 1784 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.241348+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.241583+0000 mon.a (mon.0) 1785 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.455 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:15.474 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:15.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.305218+0000 osd.2 (osd.2) 97 : cluster [DBG] 25.16 scrub starts 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.308721+0000 osd.2 (osd.2) 98 : cluster [DBG] 25.16 scrub ok 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.544602+0000 osd.7 (osd.7) 109 : cluster [DBG] 25.3 scrub starts 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.548087+0000 osd.7 (osd.7) 110 : cluster [DBG] 25.3 scrub ok 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.558969+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.659904+0000 osd.6 (osd.6) 107 : cluster [DBG] 25.0 scrub starts 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.663497+0000 osd.6 (osd.6) 108 : cluster [DBG] 25.0 scrub ok 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.682428+0000 osd.4 (osd.4) 120 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.683239+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.961900+0000 mon.a (mon.0) 1778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.972104+0000 mon.a (mon.0) 1779 : audit [INF] from='client.47878 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi107-112411-46"}]': finished 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.972236+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? v1:172.21.15.107:0/3443746379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:14.972278+0000 mon.a (mon.0) 1781 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.976942+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.979977+0000 mon.a (mon.0) 1782 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:14.999903+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.010127+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.239899+0000 mon.a (mon.0) 1783 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.240707+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.240948+0000 mon.a (mon.0) 1784 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.241348+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.241583+0000 mon.a (mon.0) 1785 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:15.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.612+0000 7ff51aa12700 1 -- 172.21.15.107:0/1217041103 >> v1:172.21.15.107:6790/0 conn(0x7ff514104330 legacy=0x7ff5141067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.612+0000 7ff51aa12700 1 -- 172.21.15.107:0/1217041103 shutdown_connections 2024-02-10T08:39:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.612+0000 7ff51aa12700 1 -- 172.21.15.107:0/1217041103 >> 172.21.15.107:0/1217041103 conn(0x7ff5140f3640 msgr2=0x7ff5140f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 -- 172.21.15.107:0/1217041103 shutdown_connections 2024-02-10T08:39:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 -- 172.21.15.107:0/1217041103 wait complete. 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 librados: starting msgr at 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 librados: starting objecter 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 Processor -- start 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 -- start start 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 librados: setting wanted keys 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.613+0000 7ff51aa12700 1 librados: calling monclient init 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff51aa12700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff514117520 con 0x7ff5140fda50 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff51aa12700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff514118700 con 0x7ff5141007f0 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff51aa12700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff514119900 con 0x7ff514104330 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff513fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff5140fda50 0x7ff5140f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52130/0 (socket says 172.21.15.107:52130) 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff518faf700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff514104330 0x7ff514115be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48232/0 (socket says 172.21.15.107:48232) 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff513fff700 1 -- 172.21.15.107:0/431344422 learned_addr learned my addr 172.21.15.107:0/431344422 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter ms_handle_connect 0x7ff5140fda50 2024-02-10T08:39:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter ms_handle_connect 0x7ff514104330 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter ms_handle_connect 0x7ff5141007f0 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928246581 0 0) 0x7ff514117520 con 0x7ff5140fda50 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4fc0033f0 con 0x7ff5140fda50 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427899380 0 0) 0x7ff514119900 con 0x7ff514104330 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff514117520 con 0x7ff514104330 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579231768 0 0) 0x7ff514118700 con 0x7ff5141007f0 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff514119900 con 0x7ff5141007f0 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981247468 0 0) 0x7ff4fc0033f0 con 0x7ff5140fda50 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff514118700 con 0x7ff5140fda50 2024-02-10T08:39:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613862416 0 0) 0x7ff514117520 con 0x7ff514104330 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4fc0033f0 con 0x7ff514104330 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff504002a10 con 0x7ff5140fda50 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff50c0013f0 con 0x7ff514104330 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645041944 0 0) 0x7ff514119900 con 0x7ff5141007f0 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff514117520 con 0x7ff5141007f0 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.614+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff508001840 con 0x7ff5141007f0 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108751368 0 0) 0x7ff514118700 con 0x7ff5140fda50 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6790/0 conn(0x7ff514104330 legacy=0x7ff514115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.195:6789/0 conn(0x7ff5141007f0 legacy=0x7ff5140fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51411ab00 con 0x7ff5140fda50 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff514117710 con 0x7ff5140fda50 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff51aa12700 10 client.27639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff514117b80 con 0x7ff5140fda50 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff51aa12700 1 librados: init done 2024-02-10T08:39:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff504002da0 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.615+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff504005bd0 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.616+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff500004f00 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.616+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff5040016e0 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(261..261 src has 50..261) v4 ==== 8329+0+0 (unknown 2560986684 0 0) 0x7ff504078e40 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 osd_map(261..261 src has 50..261) v4 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 3 client.27639.objecter handle_osd_map got epochs [261,261] > 0 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 3 client.27639.objecter handle_osd_map decoding full epoch 261 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 20 client.27639.objecter dump_active .. 0 homeless 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v261) v1 ==== 54+0+16 (unknown 2443954105 0 2420000192) 0x7ff504079220 con 0x7ff5140fda50 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff4fc063cd0 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 10 client.27639.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 1 2024-02-10T08:39:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 1 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 0 1 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 4 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 10 client.27639.objecter _send_command 1 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff500004950 con 0x7ff500004530 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.617+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500004530 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.618+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff500004950 con 0x7ff500004530 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.618+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.618+0000 7ff5117fa700 10 client.27639.objecter _finish_command 1 = system:0 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.618+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 4 2024-02-10T08:39:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.618+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 0 1 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 10 client.27639.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 2 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 2 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff5000051a0 osd=0 3 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 0 2 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 4 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 10 client.27639.objecter _send_command 2 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.634+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff500004d30 con 0x7ff500004530 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff500004d30 con 0x7ff500004530 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff5117fa700 10 client.27639.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 4 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 0 2 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 10 client.27639.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 3 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 3 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 1 3 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 4 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 10 client.27639.objecter _send_command 3 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.636+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff500010380 con 0x7ff500003b40 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500003b40 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff500010380 con 0x7ff500003b40 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 10 client.27639.objecter _finish_command 3 = system:0 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 4 2024-02-10T08:39:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.637+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 1 3 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 10 client.27639.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 4 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500004270 osd=1 3 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 1 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 10 client.27639.objecter _send_command 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.652+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff5000116a0 con 0x7ff500003b40 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff5000116a0 con 0x7ff500003b40 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff5117fa700 10 client.27639.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 1 4 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 10 client.27639.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 5 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 5 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 2 5 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 4 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 10 client.27639.objecter _send_command 5 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.654+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff500015260 con 0x7ff500002070 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500002070 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff500015260 con 0x7ff500002070 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 10 client.27639.objecter _finish_command 5 = system:0 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 4 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.655+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 2 5 2024-02-10T08:39:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 10 client.27639.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 6 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 6 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500011970 osd=2 3 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 2 6 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 4 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 10 client.27639.objecter _send_command 6 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.669+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff500001630 con 0x7ff500002070 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff500001630 con 0x7ff500002070 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 10 client.27639.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 4 2024-02-10T08:39:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 2 6 2024-02-10T08:39:15.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 10 client.27639.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 7 2024-02-10T08:39:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 7 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 3 7 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 4 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 10 client.27639.objecter _send_command 7 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5000114c0 con 0x7ff500016550 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.671+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500016550 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.672+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff5000114c0 con 0x7ff500016550 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.672+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.672+0000 7ff5117fa700 10 client.27639.objecter _finish_command 7 = system:0 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.672+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 4 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.672+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 3 7 2024-02-10T08:39:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 10 client.27639.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 8 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 8 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010cf0 osd=3 3 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 3 8 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 4 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 10 client.27639.objecter _send_command 8 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.686+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff500010760 con 0x7ff500016550 2024-02-10T08:39:15.865 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff500010760 con 0x7ff500016550 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff5117fa700 10 client.27639.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 4 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 3 8 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 10 client.27639.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 9 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 9 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 4 9 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 4 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 10 client.27639.objecter _send_command 9 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.688+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff500010690 con 0x7ff5000160b0 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff5000160b0 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff500010690 con 0x7ff5000160b0 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 10 client.27639.objecter _finish_command 9 = system:0 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 4 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.689+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 4 9 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 10 client.27639.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 10 2024-02-10T08:39:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 10 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010520 osd=4 3 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 4 10 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 4 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 10 client.27639.objecter _send_command 10 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.713+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff500010690 con 0x7ff5000160b0 2024-02-10T08:39:15.868 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.714+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff500010690 con 0x7ff5000160b0 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.714+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.714+0000 7ff5117fa700 10 client.27639.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.714+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 4 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.714+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 4 10 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 10 client.27639.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 11 2024-02-10T08:39:15.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 11 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 5 11 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 4 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 10 client.27639.objecter _send_command 11 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff50001c400 con 0x7ff50001be40 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.715+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff50001be40 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.716+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff50001c400 con 0x7ff50001be40 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.716+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.716+0000 7ff5117fa700 10 client.27639.objecter _finish_command 11 = system:0 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.716+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 4 2024-02-10T08:39:15.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.716+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 5 11 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 10 client.27639.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 12 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 12 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500010690 osd=5 3 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 5 12 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 4 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 10 client.27639.objecter _send_command 12 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.730+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff50001c400 con 0x7ff50001be40 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff50001c400 con 0x7ff50001be40 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff5117fa700 10 client.27639.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 4 2024-02-10T08:39:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 5 12 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 10 client.27639.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 13 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 13 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 6 13 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 4 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 10 client.27639.objecter _send_command 13 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.732+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff500026fe0 con 0x7ff500023490 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500023490 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff500026fe0 con 0x7ff500023490 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 10 client.27639.objecter _finish_command 13 = system:0 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 4 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.733+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 6 13 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 10 client.27639.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 14 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 14 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff50001c400 osd=6 3 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 6 14 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 4 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 10 client.27639.objecter _send_command 14 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.747+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff500028790 con 0x7ff500023490 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff500028790 con 0x7ff500023490 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff5117fa700 10 client.27639.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 4 2024-02-10T08:39:15.874 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 6 14 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 10 client.27639.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 15 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 15 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 7 15 2024-02-10T08:39:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 4 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 10 client.27639.objecter _send_command 15 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.749+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5000293c0 con 0x7ff500028e10 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 10 client.27639.objecter ms_handle_connect 0x7ff500028e10 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff5000293c0 con 0x7ff500028e10 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 10 client.27639.objecter _finish_command 15 = system:0 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 4 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.750+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 7 15 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 10 client.27639.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign -1 16 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 20 client.27639.objecter _get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_remove -1 16 2024-02-10T08:39:15.877 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 20 client.27639.objecter get_session s=0x7ff500028790 osd=7 3 2024-02-10T08:39:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 15 client.27639.objecter _session_command_op_assign 7 16 2024-02-10T08:39:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 4 2024-02-10T08:39:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 10 client.27639.objecter _send_command 16 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.765+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff5000293c0 con 0x7ff500028e10 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.766+0000 7ff5117fa700 1 -- 172.21.15.107:0/431344422 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff5000293c0 con 0x7ff500028e10 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.766+0000 7ff5117fa700 10 client.27639.objecter ms_dispatch 0x7ff514001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.766+0000 7ff5117fa700 10 client.27639.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.766+0000 7ff5117fa700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 4 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.766+0000 7ff5117fa700 15 client.27639.objecter _session_command_op_remove 7 16 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 librados: watch_flush enter 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 librados: watch_flush exit 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.0 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6801/3321596972 conn(0x7ff500004530 legacy=0x7ff500002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff5000051a0 osd=0 1 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.1 2024-02-10T08:39:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6805/133516126 conn(0x7ff500003b40 legacy=0x7ff50000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500004270 osd=1 1 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.2 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6809/3575881728 conn(0x7ff500002070 legacy=0x7ff500013b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500011970 osd=2 1 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.3 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6813/2871825691 conn(0x7ff500016550 legacy=0x7ff500018940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010cf0 osd=3 1 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.4 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff518faf700 1 -- 172.21.15.107:0/431344422 reap_dead start 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.195:6800/2499671285 conn(0x7ff5000160b0 legacy=0x7ff50001b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010520 osd=4 1 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.5 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff518faf700 1 -- 172.21.15.107:0/431344422 reap_dead start 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.195:6804/4178321486 conn(0x7ff50001be40 legacy=0x7ff500021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500010690 osd=5 1 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.6 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.195:6808/1417479017 conn(0x7ff500023490 legacy=0x7ff500025880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff50001c400 osd=6 1 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter close_session for osd.7 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.195:6812/1934176928 conn(0x7ff500028e10 legacy=0x7ff500026fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter put_session s=0x7ff500028790 osd=7 1 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 20 client.27639.objecter shutdown clearing up homeless session... 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 10 client.27639.objecter successfully canceled tick 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff4fc063cd0 legacy=0x7ff4fc066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> v1:172.21.15.107:6789/0 conn(0x7ff5140fda50 legacy=0x7ff5140f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff518faf700 1 -- 172.21.15.107:0/431344422 reap_dead start 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 shutdown_connections 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 >> 172.21.15.107:0/431344422 conn(0x7ff5140f3640 msgr2=0x7ff5141077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 shutdown_connections 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 -- 172.21.15.107:0/431344422 wait complete. 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:15.767+0000 7ff51aa12700 1 librados: shutdown 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:39:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:15] "GET /metrics HTTP/1.1" 200 188139 "" "Prometheus/2.33.4" 2024-02-10T08:39:16.467 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:16.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:15.396614+0000 mgr.y (mgr.14553) 267 : cluster [DBG] pgmap v368: 217 pgs: 217 active+clean; 8.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.396917+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.397875+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.618084+0000 mon.a (mon.0) 1787 : audit [DBG] from='client.? v1:172.21.15.107:0/431344422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.972025+0000 mon.a (mon.0) 1788 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.972137+0000 mon.a (mon.0) 1789 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.972209+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:15.972269+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.979877+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.980217+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.980602+0000 mon.a (mon.0) 1792 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.983332+0000 mon.a (mon.0) 1793 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.984967+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:15.986082+0000 mon.a (mon.0) 1794 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:16.002359+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:16.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:16.604 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:16.625 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:16.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:15.396614+0000 mgr.y (mgr.14553) 267 : cluster [DBG] pgmap v368: 217 pgs: 217 active+clean; 8.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.396917+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.397875+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.618084+0000 mon.a (mon.0) 1787 : audit [DBG] from='client.? v1:172.21.15.107:0/431344422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.972025+0000 mon.a (mon.0) 1788 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.972137+0000 mon.a (mon.0) 1789 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.972209+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:15.972269+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.979877+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.980217+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.980602+0000 mon.a (mon.0) 1792 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.983332+0000 mon.a (mon.0) 1793 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.984967+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:15.986082+0000 mon.a (mon.0) 1794 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:16.002359+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:15.396614+0000 mgr.y (mgr.14553) 267 : cluster [DBG] pgmap v368: 217 pgs: 217 active+clean; 8.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.396917+0000 mon.c (mon.2) 297 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.397875+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.618084+0000 mon.a (mon.0) 1787 : audit [DBG] from='client.? v1:172.21.15.107:0/431344422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.972025+0000 mon.a (mon.0) 1788 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.972137+0000 mon.a (mon.0) 1789 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi107-112411-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.972209+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:15.972269+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.979877+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.107:0/4001272799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.980217+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.980602+0000 mon.a (mon.0) 1792 : audit [INF] from='client.47884 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]: dispatch 2024-02-10T08:39:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.983332+0000 mon.a (mon.0) 1793 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:16.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.984967+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:15.986082+0000 mon.a (mon.0) 1794 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:16.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:16.002359+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:16.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:17.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:17.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:17.938 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:17.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:18.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:18.079 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:18.100 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:18.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:18.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:16.972908+0000 mon.a (mon.0) 1795 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:18.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.001349+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.003124+0000 mon.b (mon.1) 442 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.216762+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:17.216849+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e263: 8 total, 8 up, 7 in 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.218149+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.218460+0000 mon.a (mon.0) 1798 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.218925+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.219326+0000 mon.a (mon.0) 1799 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.219734+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.219823+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.220081+0000 mon.a (mon.0) 1801 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:17.397086+0000 mgr.y (mgr.14553) 268 : cluster [DBG] pgmap v371: 249 pgs: 2 peering, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:16.972908+0000 mon.a (mon.0) 1795 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.001349+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.003124+0000 mon.b (mon.1) 442 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.216762+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:17.216849+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e263: 8 total, 8 up, 7 in 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.218149+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.218460+0000 mon.a (mon.0) 1798 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.218925+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.219326+0000 mon.a (mon.0) 1799 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.219734+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.219823+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.220081+0000 mon.a (mon.0) 1801 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:17.397086+0000 mgr.y (mgr.14553) 268 : cluster [DBG] pgmap v371: 249 pgs: 2 peering, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:18.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:16.972908+0000 mon.a (mon.0) 1795 : audit [INF] from='client.47884 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi107-112403-37"}]': finished 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.001349+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.003124+0000 mon.b (mon.1) 442 : audit [DBG] from='client.? v1:172.21.15.107:0/2868250403' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.216762+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:17.216849+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e263: 8 total, 8 up, 7 in 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.218149+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.218460+0000 mon.a (mon.0) 1798 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.218925+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.219326+0000 mon.a (mon.0) 1799 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:18.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.219734+0000 mon.c (mon.2) 300 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.219823+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:18.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.220081+0000 mon.a (mon.0) 1801 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:18.232 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:17.397086+0000 mgr.y (mgr.14553) 268 : cluster [DBG] pgmap v371: 249 pgs: 2 peering, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:39:18.233 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:18.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:18.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:18.385 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:18.404 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:18.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:18.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:18.788 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:18.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:18.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:18.929 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:18.948 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:18.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:19.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:19.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:19.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:19.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.980511+0000 mon.a (mon.0) 1802 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:19.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.980595+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:19.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.980649+0000 mon.a (mon.0) 1804 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:19.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:17.980697+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e264: 8 total, 8 up, 7 in 2024-02-10T08:39:19.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.981251+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.984093+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.984393+0000 mon.a (mon.0) 1806 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:17.984840+0000 mon.a (mon.0) 1807 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.980511+0000 mon.a (mon.0) 1802 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.980595+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.980649+0000 mon.a (mon.0) 1804 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:17.980697+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e264: 8 total, 8 up, 7 in 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.981251+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.984093+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.984393+0000 mon.a (mon.0) 1806 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:17.984840+0000 mon.a (mon.0) 1807 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.980511+0000 mon.a (mon.0) 1802 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi107-112411-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.980595+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.980649+0000 mon.a (mon.0) 1804 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi107-112403-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:17.980697+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e264: 8 total, 8 up, 7 in 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.981251+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.984093+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.984393+0000 mon.a (mon.0) 1806 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:17.984840+0000 mon.a (mon.0) 1807 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:19.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.781+0000 7f2720914700 1 -- 172.21.15.107:0/400782761 >> v1:172.21.15.107:6790/0 conn(0x7f271c0fda30 legacy=0x7f271c0fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.781+0000 7f2720914700 1 -- 172.21.15.107:0/400782761 shutdown_connections 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.781+0000 7f2720914700 1 -- 172.21.15.107:0/400782761 >> 172.21.15.107:0/400782761 conn(0x7f271c0f3630 msgr2=0x7f271c0f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.781+0000 7f2720914700 1 -- 172.21.15.107:0/400782761 shutdown_connections 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.781+0000 7f2720914700 1 -- 172.21.15.107:0/400782761 wait complete. 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 librados: starting msgr at 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 librados: starting objecter 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 Processor -- start 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 -- start start 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 librados: setting wanted keys 2024-02-10T08:39:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 librados: calling monclient init 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f271c1194d0 con 0x7f271c104320 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f271c11a670 con 0x7f271c0fda30 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2720914700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f271c11b810 con 0x7f271c100810 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f271ad9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f271c104320 0x7f271c117db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42956/0 (socket says 172.21.15.107:42956) 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f2719d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f271c100810 0x7f271c06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35784/0 (socket says 172.21.15.107:35784) 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f271a59c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f271c0fda30 0x7f271c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:50154/0 (socket says 172.21.15.107:50154) 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f271ad9d700 1 -- 172.21.15.107:0/1454158873 learned_addr learned my addr 172.21.15.107:0/1454158873 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter ms_handle_connect 0x7f271c100810 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter ms_handle_connect 0x7f271c0fda30 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter ms_handle_connect 0x7f271c104320 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.782+0000 7f270b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263445587 0 0) 0x7f271c11a670 con 0x7f271c0fda30 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2704003030 con 0x7f271c0fda30 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585198814 0 0) 0x7f271c1194d0 con 0x7f271c104320 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c11a670 con 0x7f271c104320 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692463989 0 0) 0x7f271c11b810 con 0x7f271c100810 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c1194d0 con 0x7f271c100810 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206972282 0 0) 0x7f2704003030 con 0x7f271c0fda30 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f271c11b810 con 0x7f271c0fda30 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3914603528 0 0) 0x7f271c1194d0 con 0x7f271c100810 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2704003030 con 0x7f271c100810 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 68308081 0 0) 0x7f271c11a670 con 0x7f271c104320 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f271c1194d0 con 0x7f271c104320 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f270c002c20 con 0x7f271c0fda30 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2710001a80 con 0x7f271c100810 2024-02-10T08:39:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2714001b80 con 0x7f271c104320 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2580305056 0 0) 0x7f271c11b810 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 >> v1:172.21.15.107:6790/0 conn(0x7f271c100810 legacy=0x7f271c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 >> v1:172.21.15.107:6789/0 conn(0x7f271c104320 legacy=0x7f271c117db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c11ca10 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c11ba00 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f2720914700 10 client.47920.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c11be70 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f2720914700 1 librados: init done 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f270c001b20 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.783+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f270c003710 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.784+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc004f20 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f270c003870 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(265..265 src has 50..265) v4 ==== 9543+0+0 (unknown 3289849274 0 0) 0x7f270c079390 con 0x7f271c0fda30 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 10 client.47920.objecter ms_dispatch 0x7f271c001550 osd_map(265..265 src has 50..265) v4 2024-02-10T08:39:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 3 client.47920.objecter handle_osd_map got epochs [265,265] > 0 2024-02-10T08:39:19.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 3 client.47920.objecter handle_osd_map decoding full epoch 265 2024-02-10T08:39:19.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 20 client.47920.objecter dump_active .. 0 homeless 2024-02-10T08:39:19.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.785+0000 7f270b7fe700 10 client.47920.objecter ms_handle_connect 0x7f2704063e30 2024-02-10T08:39:19.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.789+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f270c04c280 con 0x7f271c0fda30 2024-02-10T08:39:19.958 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:19.959 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:19.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.975+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f26fc0051d0 con 0x7f271c0fda30 2024-02-10T08:39:19.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.988+0000 7f270b7fe700 1 -- 172.21.15.107:0/1454158873 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v266) v1 ==== 113+0+0 (unknown 4145129021 0 0) 0x7f270c04a6f0 con 0x7f271c0fda30 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.990+0000 7f2720914700 10 librados: watch_flush enter 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.993+0000 7f2720914700 10 librados: watch_flush exit 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.993+0000 7f2720914700 20 client.47920.objecter shutdown clearing up homeless session... 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.993+0000 7f2720914700 10 client.47920.objecter successfully canceled tick 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.993+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2704063e30 legacy=0x7f27040662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:19.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.993+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 >> v1:172.21.15.195:6789/0 conn(0x7f271c0fda30 legacy=0x7f271c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:19.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.995+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 shutdown_connections 2024-02-10T08:39:19.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.995+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 >> 172.21.15.107:0/1454158873 conn(0x7f271c0f3630 msgr2=0x7f271c0f6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:19.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.996+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 shutdown_connections 2024-02-10T08:39:19.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.997+0000 7f2720914700 1 -- 172.21.15.107:0/1454158873 wait complete. 2024-02-10T08:39:20.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:19.999+0000 7f2720914700 1 librados: shutdown 2024-02-10T08:39:20.001 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T08:39:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:20.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:20.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:20.343 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:20.362 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:20.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:20.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:39:20.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:20.489 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:18.980777+0000 mon.a (mon.0) 1808 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:18.981706+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:19.226969+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e265: 8 total, 8 up, 7 in 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.227846+0000 mon.a (mon.0) 1810 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:19.397546+0000 mgr.y (mgr.14553) 269 : cluster [DBG] pgmap v374: 256 pgs: 1 peering, 8 unknown, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-10T08:39:20.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.397788+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.398860+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.977133+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.107:0/1454158873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.977555+0000 mon.a (mon.0) 1812 : audit [INF] from='client.47920 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:19.978188+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.982226+0000 mon.a (mon.0) 1814 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.982312+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]': finished 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.982376+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.982478+0000 mon.a (mon.0) 1817 : audit [INF] from='client.47920 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:19.982530+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e266: 8 total, 8 up, 7 in 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.995784+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:19.997434+0000 mon.a (mon.0) 1819 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.021194+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.022420+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.022723+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:20.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.092922+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.093294+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.492 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:20.509 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:20.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:18.980777+0000 mon.a (mon.0) 1808 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:18.981706+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:19.226969+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e265: 8 total, 8 up, 7 in 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.227846+0000 mon.a (mon.0) 1810 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:19.397546+0000 mgr.y (mgr.14553) 269 : cluster [DBG] pgmap v374: 256 pgs: 1 peering, 8 unknown, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-10T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.397788+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.398860+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.977133+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.107:0/1454158873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.977555+0000 mon.a (mon.0) 1812 : audit [INF] from='client.47920 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:19.978188+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.982226+0000 mon.a (mon.0) 1814 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.982312+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]': finished 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.982376+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:18.980777+0000 mon.a (mon.0) 1808 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:18.981706+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:19.226969+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e265: 8 total, 8 up, 7 in 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.227846+0000 mon.a (mon.0) 1810 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]: dispatch 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:19.397546+0000 mgr.y (mgr.14553) 269 : cluster [DBG] pgmap v374: 256 pgs: 1 peering, 8 unknown, 32 creating+peering, 215 active+clean; 8.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.397788+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.398860+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.977133+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.107:0/1454158873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.977555+0000 mon.a (mon.0) 1812 : audit [INF] from='client.47920 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:19.978188+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.982226+0000 mon.a (mon.0) 1814 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi107-112403-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.982312+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-31", "mode": "writeback"}]': finished 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.982376+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi107-112443-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-10T08:39:20.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.982478+0000 mon.a (mon.0) 1817 : audit [INF] from='client.47920 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:19.982530+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e266: 8 total, 8 up, 7 in 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.995784+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:19.997434+0000 mon.a (mon.0) 1819 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.021194+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.022420+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.022723+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.092922+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.093294+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.982478+0000 mon.a (mon.0) 1817 : audit [INF] from='client.47920 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:19.982530+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e266: 8 total, 8 up, 7 in 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.995784+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:19.997434+0000 mon.a (mon.0) 1819 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.021194+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.022420+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.022723+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.092922+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.093294+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:20.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:20.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T08:39:20.988 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:20.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:21.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:21.118 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:21.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:39:21.574 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:21.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:21.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:21.707 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:21.709 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:21.730 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:21.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:21.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:21.863 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:21.882 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:21.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:21.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.960313+0000 mon.a (mon.0) 1824 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.960426+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.960490+0000 mon.a (mon.0) 1826 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:20.960556+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e267: 8 total, 8 up, 7 in 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.961235+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.965122+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.966053+0000 mon.a (mon.0) 1828 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.966596+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:20.966813+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:21.304044+0000 mgr.y (mgr.14553) 270 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:21.398471+0000 mgr.y (mgr.14553) 271 : cluster [DBG] pgmap v377: 226 pgs: 4 creating+activating, 222 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.960313+0000 mon.a (mon.0) 1824 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.960426+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.960490+0000 mon.a (mon.0) 1826 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:20.960556+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e267: 8 total, 8 up, 7 in 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.961235+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.965122+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.966053+0000 mon.a (mon.0) 1828 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.966596+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:20.966813+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:22.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:21.304044+0000 mgr.y (mgr.14553) 270 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:22.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:21.398471+0000 mgr.y (mgr.14553) 271 : cluster [DBG] pgmap v377: 226 pgs: 4 creating+activating, 222 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.960313+0000 mon.a (mon.0) 1824 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.960426+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi107-112443-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.960490+0000 mon.a (mon.0) 1826 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:20.960556+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e267: 8 total, 8 up, 7 in 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.961235+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.107:0/2005230489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.965122+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.966053+0000 mon.a (mon.0) 1828 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]: dispatch 2024-02-10T08:39:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.966596+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]: dispatch 2024-02-10T08:39:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:20.966813+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:21.304044+0000 mgr.y (mgr.14553) 270 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:21.398471+0000 mgr.y (mgr.14553) 271 : cluster [DBG] pgmap v377: 226 pgs: 4 creating+activating, 222 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-10T08:39:22.364 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:22.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:22.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:22.487 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:22.497 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:22.518 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:22.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:22.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:22.858 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:21.957923+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:21.971172+0000 mon.a (mon.0) 1832 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:21.971240+0000 mon.a (mon.0) 1833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:21.997421+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:22.215631+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e268: 8 total, 8 up, 7 in 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.216378+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.216603+0000 mon.a (mon.0) 1835 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.217255+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.219950+0000 mon.a (mon.0) 1836 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.220445+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.220674+0000 mon.a (mon.0) 1837 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.221751+0000 mon.a (mon.0) 1838 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:21.957923+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:21.971172+0000 mon.a (mon.0) 1832 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:21.971240+0000 mon.a (mon.0) 1833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:21.997421+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:22.215631+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e268: 8 total, 8 up, 7 in 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.216378+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.216603+0000 mon.a (mon.0) 1835 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.217255+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.219950+0000 mon.a (mon.0) 1836 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.220445+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.220674+0000 mon.a (mon.0) 1837 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.221751+0000 mon.a (mon.0) 1838 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:21.957923+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:21.971172+0000 mon.a (mon.0) 1832 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi107-112411-47"}]': finished 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:21.971240+0000 mon.a (mon.0) 1833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-31"}]': finished 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:21.997421+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:22.215631+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e268: 8 total, 8 up, 7 in 2024-02-10T08:39:23.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.216378+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.216603+0000 mon.a (mon.0) 1835 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.217255+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.219950+0000 mon.a (mon.0) 1836 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.220445+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.220674+0000 mon.a (mon.0) 1837 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:23.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.221751+0000 mon.a (mon.0) 1838 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioed=5 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:89d3ae78:::11:head 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:89d3ae78:::11:head expected=25:89d3ae78:::11:head 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:89d3ae78:::11:head -> 11 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=11 expected=11 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:863748b0:::15:head 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:863748b0:::15:head expected=25:863748b0:::15:head 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:863748b0:::15:head -> 15 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=15 expected=15 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:02547ec2:::1:head 2024-02-10T08:39:24.038 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:02547ec2:::1:head expected=25:02547ec2:::1:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:02547ec2:::1:head -> 1 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=1 expected=1 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:5c6b0b28:::7:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:5c6b0b28:::7:head expected=25:5c6b0b28:::7:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:5c6b0b28:::7:head -> 7 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=7 expected=7 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:6cac518f:::0:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:6cac518f:::0:head expected=25:6cac518f:::0:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:6cac518f:::0:head -> 0 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=0 expected=0 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:62a1935d:::14:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:62a1935d:::14:head expected=25:62a1935d:::14:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:62a1935d:::14:head -> 14 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=14 expected=14 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:566253c9:::13:head 2024-02-10T08:39:24.039 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:566253c9:::13:head expected=25:566253c9:::13:head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:566253c9:::13:head -> 13 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=13 expected=13 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : seek to 25:52ea6a34:::10:head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : cursor()=25:52ea6a34:::10:head expected=25:52ea6a34:::10:head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: > 25:52ea6a34:::10:head -> 10 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: : entry=10 expected=10 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (298 ms) 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (147462 ms) 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: split 0/5 -> MIN 25:33333333::::head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: split 1/5 -> 25:33333333::::head 25:66666666::::head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: split 2/5 -> 25:66666666::::head 25:99999999::::head 2024-02-10T08:39:24.040 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: split 3/5 -> 25:99999999::::head 25:cccccccc::::head 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: split 4/5 -> 25:cccccccc::::head MAX 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (14396 ms) 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 7 tests from LibRadosList (162718 ms total) 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1023 ms) 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo1,foo2,foo3 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo1 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo2 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo3 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo1,foo4,foo5 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo4 2024-02-10T08:39:24.041 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo5 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo1 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset foo6,foo7 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo7 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: foo6 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo4 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo5 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns2:foo7 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns2:foo6 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: ns1:foo1 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo1 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo2 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: :foo3 2024-02-10T08:39:24.042 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (33 ms) 2024-02-10T08:39:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.970421+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.970527+0000 mon.a (mon.0) 1840 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.970613+0000 mon.a (mon.0) 1841 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:22.970688+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e269: 8 total, 8 up, 7 in 2024-02-10T08:39:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.981084+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.981569+0000 mon.a (mon.0) 1843 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:22.986113+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:23.233381+0000 mon.a (mon.0) 1844 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:23.398899+0000 mgr.y (mgr.14553) 272 : cluster [DBG] pgmap v380: 194 pgs: 8 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:24.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.970421+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:24.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.970527+0000 mon.a (mon.0) 1840 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:24.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.970613+0000 mon.a (mon.0) 1841 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:24.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:22.970688+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e269: 8 total, 8 up, 7 in 2024-02-10T08:39:24.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.981084+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.981569+0000 mon.a (mon.0) 1843 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:22.986113+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:23.233381+0000 mon.a (mon.0) 1844 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:23.398899+0000 mgr.y (mgr.14553) 272 : cluster [DBG] pgmap v380: 194 pgs: 8 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.970421+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi107-112443-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.970527+0000 mon.a (mon.0) 1840 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi107-112411-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.970613+0000 mon.a (mon.0) 1841 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:22.970688+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e269: 8 total, 8 up, 7 in 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.981084+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.981569+0000 mon.a (mon.0) 1843 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:24.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:22.986113+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.107:0/131797111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:23.233381+0000 mon.a (mon.0) 1844 : audit [INF] from='client.53189 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]: dispatch 2024-02-10T08:39:24.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:23.398899+0000 mgr.y (mgr.14553) 272 : cluster [DBG] pgmap v380: 194 pgs: 8 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:24.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:24.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:24.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:24.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:25.036 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:25.090 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:25.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:25.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.100+0000 7f4b31f03700 1 -- 172.21.15.107:0/2717580406 >> v1:172.21.15.107:6790/0 conn(0x7f4b2c0f7300 legacy=0x7f4b2c104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:25.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 -- 172.21.15.107:0/2717580406 shutdown_connections 2024-02-10T08:39:25.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 -- 172.21.15.107:0/2717580406 >> 172.21.15.107:0/2717580406 conn(0x7f4b2c0f2510 msgr2=0x7f4b2c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:25.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 -- 172.21.15.107:0/2717580406 shutdown_connections 2024-02-10T08:39:25.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 -- 172.21.15.107:0/2717580406 wait complete. 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 librados: starting msgr at 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.101+0000 7f4b31f03700 1 librados: starting objecter 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 Processor -- start 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 -- start start 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 librados: setting wanted keys 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 librados: calling monclient init 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b2c119630 con 0x7f4b2c0f7300 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b2c11a810 con 0x7f4b2c106640 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b31f03700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b2c11ba10 con 0x7f4b2c0f6810 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b2b7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4b2c0f6810 0x7f4b2c102c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35858/0 (socket says 172.21.15.107:35858) 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b2b7fe700 1 -- 172.21.15.107:0/1573906400 learned_addr learned my addr 172.21.15.107:0/1573906400 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:25.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter ms_handle_connect 0x7f4b2c0f6810 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter ms_handle_connect 0x7f4b2c0f7300 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter ms_handle_connect 0x7f4b2c106640 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124319305 0 0) 0x7f4b2c11ba10 con 0x7f4b2c0f6810 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b14003630 con 0x7f4b2c0f6810 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592610829 0 0) 0x7f4b2c119630 con 0x7f4b2c0f7300 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c11ba10 con 0x7f4b2c0f7300 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.102+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496882322 0 0) 0x7f4b2c11a810 con 0x7f4b2c106640 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c119630 con 0x7f4b2c106640 2024-02-10T08:39:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228007270 0 0) 0x7f4b14003630 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b2c11a810 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b1c002990 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4211320570 0 0) 0x7f4b2c11ba10 con 0x7f4b2c0f7300 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b14003630 con 0x7f4b2c0f7300 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897596584 0 0) 0x7f4b2c119630 con 0x7f4b2c106640 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b2c11ba10 con 0x7f4b2c106640 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b20002f60 con 0x7f4b2c0f7300 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b24001a30 con 0x7f4b2c106640 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637158268 0 0) 0x7f4b2c11a810 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 >> v1:172.21.15.195:6789/0 conn(0x7f4b2c106640 legacy=0x7f4b2c117cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 >> v1:172.21.15.107:6789/0 conn(0x7f4b2c0f7300 legacy=0x7f4b2c103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c11cc10 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c119820 con 0x7f4b2c0f6810 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b31f03700 10 client.53216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:25.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c119d10 con 0x7f4b2c0f6810 2024-02-10T08:39:25.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.103+0000 7f4b31f03700 1 librados: init done 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.104+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4b1c002d20 con 0x7f4b2c0f6810 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.104+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b1c005c30 con 0x7f4b2c0f6810 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.104+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4b1c018b40 con 0x7f4b2c0f6810 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.104+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b18004f20 con 0x7f4b2c0f6810 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(271..271 src has 50..271) v4 ==== 9433+0+0 (unknown 3027153650 0 0) 0x7f4b1c07a280 con 0x7f4b2c0f6810 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 10 client.53216.objecter ms_dispatch 0x7f4b2c001550 osd_map(271..271 src has 50..271) v4 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 3 client.53216.objecter handle_osd_map got epochs [271,271] > 0 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 3 client.53216.objecter handle_osd_map decoding full epoch 271 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 20 client.53216.objecter dump_active .. 0 homeless 2024-02-10T08:39:25.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.105+0000 7f4b28ff9700 10 client.53216.objecter ms_handle_connect 0x7f4b14063e20 2024-02-10T08:39:25.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.109+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4b1c04d110 con 0x7f4b2c0f6810 2024-02-10T08:39:25.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.231356+0000 mon.a (mon.0) 1845 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:25.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:24.231442+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e270: 8 total, 8 up, 7 in 2024-02-10T08:39:25.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.232407+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.233741+0000 mon.a (mon.0) 1847 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.248431+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.248719+0000 mon.a (mon.0) 1848 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.249477+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.249708+0000 mon.a (mon.0) 1849 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.250047+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.250256+0000 mon.a (mon.0) 1850 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.231356+0000 mon.a (mon.0) 1845 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:24.231442+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e270: 8 total, 8 up, 7 in 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.232407+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.233741+0000 mon.a (mon.0) 1847 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.248431+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.248719+0000 mon.a (mon.0) 1848 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.249477+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.249708+0000 mon.a (mon.0) 1849 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.250047+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.250256+0000 mon.a (mon.0) 1850 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:25.226 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:25.247 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:25.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.231356+0000 mon.a (mon.0) 1845 : audit [INF] from='client.53189 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi107-112403-38"}]': finished 2024-02-10T08:39:25.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:24.231442+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e270: 8 total, 8 up, 7 in 2024-02-10T08:39:25.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.232407+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.233741+0000 mon.a (mon.0) 1847 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.248431+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.248719+0000 mon.a (mon.0) 1848 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.249477+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.249708+0000 mon.a (mon.0) 1849 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.250047+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.250256+0000 mon.a (mon.0) 1850 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:25.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:25.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:25.301+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f4b180051d0 con 0x7f4b2c0f6810 2024-02-10T08:39:25.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:25.382 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:25.403 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:25.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:25.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:25.535 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:25.555 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:25.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:25.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:25] "GET /metrics HTTP/1.1" 200 188389 "" "Prometheus/2.33.4" 2024-02-10T08:39:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.291+0000 7f4b28ff9700 1 -- 172.21.15.107:0/1573906400 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v272) v1 ==== 97+0+0 (unknown 2596541350 0 0) 0x7f4b1c04b520 con 0x7f4b2c0f6810 2024-02-10T08:39:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.293+0000 7f4b31f03700 10 librados: watch_flush enter 2024-02-10T08:39:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.293+0000 7f4b31f03700 10 librados: watch_flush exit 2024-02-10T08:39:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.293+0000 7f4b31f03700 20 client.53216.objecter shutdown clearing up homeless session... 2024-02-10T08:39:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.293+0000 7f4b31f03700 10 client.53216.objecter successfully canceled tick 2024-02-10T08:39:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.293+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4b14063e20 legacy=0x7f4b140662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 >> v1:172.21.15.107:6790/0 conn(0x7f4b2c0f6810 legacy=0x7f4b2c102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 shutdown_connections 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 >> 172.21.15.107:0/1573906400 conn(0x7f4b2c0f2510 msgr2=0x7f4b2c109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 shutdown_connections 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 -- 172.21.15.107:0/1573906400 wait complete. 2024-02-10T08:39:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:26.294+0000 7f4b31f03700 1 librados: shutdown 2024-02-10T08:39:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:39:26.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.994633+0000 mon.a (mon.0) 1851 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:26.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.994858+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.994939+0000 mon.a (mon.0) 1853 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:24.995004+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e271: 8 total, 8 up, 7 in 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.996036+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.996130+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:24.996497+0000 mon.a (mon.0) 1856 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:25.017361+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:25.018904+0000 mon.a (mon.0) 1857 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:25.153709+0000 mon.a (mon.0) 1858 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:25.303651+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.107:0/1573906400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:25.303946+0000 mon.a (mon.0) 1859 : audit [INF] from='client.53216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:25.399288+0000 mgr.y (mgr.14553) 273 : cluster [DBG] pgmap v383: 226 pgs: 40 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.994633+0000 mon.a (mon.0) 1851 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.994858+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.994939+0000 mon.a (mon.0) 1853 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:24.995004+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e271: 8 total, 8 up, 7 in 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.996036+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.996130+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:24.996497+0000 mon.a (mon.0) 1856 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:25.017361+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:25.018904+0000 mon.a (mon.0) 1857 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:25.153709+0000 mon.a (mon.0) 1858 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:25.303651+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.107:0/1573906400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:25.303946+0000 mon.a (mon.0) 1859 : audit [INF] from='client.53216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.322 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:25.399288+0000 mgr.y (mgr.14553) 273 : cluster [DBG] pgmap v383: 226 pgs: 40 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.994633+0000 mon.a (mon.0) 1851 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi107-112411-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.994858+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.994939+0000 mon.a (mon.0) 1853 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi107-112403-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:24.995004+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e271: 8 total, 8 up, 7 in 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.996036+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.996130+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:24.996497+0000 mon.a (mon.0) 1856 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:25.017361+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:25.018904+0000 mon.a (mon.0) 1857 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:25.153709+0000 mon.a (mon.0) 1858 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:25.303651+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.107:0/1573906400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:25.303946+0000 mon.a (mon.0) 1859 : audit [INF] from='client.53216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:25.399288+0000 mgr.y (mgr.14553) 273 : cluster [DBG] pgmap v383: 226 pgs: 40 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:25.989502+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.047217+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.292872+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.292942+0000 mon.a (mon.0) 1863 : audit [INF] from='client.53216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:26.293005+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e272: 8 total, 8 up, 7 in 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.293751+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.294003+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:26.294382+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.048737+0000 mon.a (mon.0) 1867 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.048827+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.048877+0000 mon.a (mon.0) 1869 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:27.048951+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e273: 8 total, 8 up, 7 in 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.050983+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.052017+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.054125+0000 mon.a (mon.0) 1871 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.054563+0000 mon.a (mon.0) 1872 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:25.989502+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.047217+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.292872+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.292942+0000 mon.a (mon.0) 1863 : audit [INF] from='client.53216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:27.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:26.293005+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e272: 8 total, 8 up, 7 in 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.293751+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.294003+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:26.294382+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.048737+0000 mon.a (mon.0) 1867 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.048827+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.048877+0000 mon.a (mon.0) 1869 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:27.048951+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e273: 8 total, 8 up, 7 in 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.050983+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.052017+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.054125+0000 mon.a (mon.0) 1871 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.054563+0000 mon.a (mon.0) 1872 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:25.989502+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.047217+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.292872+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.292942+0000 mon.a (mon.0) 1863 : audit [INF] from='client.53216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:26.293005+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e272: 8 total, 8 up, 7 in 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.293751+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.294003+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]: dispatch 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:26.294382+0000 mon.a (mon.0) 1866 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.048737+0000 mon.a (mon.0) 1867 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi107-112403-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.048827+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? v1:172.21.15.107:0/1714952529' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi107-112443-2"}]': finished 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.048877+0000 mon.a (mon.0) 1869 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:27.048951+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e273: 8 total, 8 up, 7 in 2024-02-10T08:39:27.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.050983+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.052017+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.054125+0000 mon.a (mon.0) 1871 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]: dispatch 2024-02-10T08:39:27.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.054563+0000 mon.a (mon.0) 1872 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:27.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.873+0000 7f1216118700 1 -- 172.21.15.107:0/3550541978 >> v1:172.21.15.107:6790/0 conn(0x7f12100ff710 legacy=0x7f1210101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.873+0000 7f1216118700 1 -- 172.21.15.107:0/3550541978 shutdown_connections 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.873+0000 7f1216118700 1 -- 172.21.15.107:0/3550541978 >> 172.21.15.107:0/3550541978 conn(0x7f12100f2510 msgr2=0x7f12100f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.873+0000 7f1216118700 1 -- 172.21.15.107:0/3550541978 shutdown_connections 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.873+0000 7f1216118700 1 -- 172.21.15.107:0/3550541978 wait complete. 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 librados: starting msgr at 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 librados: starting objecter 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 Processor -- start 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 -- start start 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 librados: setting wanted keys 2024-02-10T08:39:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 librados: calling monclient init 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1210117350 con 0x7f12100ff710 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1210118530 con 0x7f12100fc910 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f1216118700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1210119730 con 0x7f1210103250 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f120effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f12100ff710 0x7f12100fa210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43114/0 (socket says 172.21.15.107:43114) 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f120ffff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1210103250 0x7f1210115a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35922/0 (socket says 172.21.15.107:35922) 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.874+0000 7f120effd700 1 -- 172.21.15.107:0/3597477642 learned_addr learned my addr 172.21.15.107:0/3597477642 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter ms_handle_connect 0x7f1210103250 2024-02-10T08:39:27.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter ms_handle_connect 0x7f12100ff710 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter ms_handle_connect 0x7f12100fc910 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285924900 0 0) 0x7f1210119730 con 0x7f1210103250 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8003030 con 0x7f1210103250 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446526044 0 0) 0x7f1210117350 con 0x7f12100ff710 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1210119730 con 0x7f12100ff710 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144967045 0 0) 0x7f1210118530 con 0x7f12100fc910 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1210117350 con 0x7f12100fc910 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3627449637 0 0) 0x7f11f8003030 con 0x7f1210103250 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1210118530 con 0x7f1210103250 2024-02-10T08:39:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1208001bc0 con 0x7f1210103250 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4260903845 0 0) 0x7f1210119730 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f8003030 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1502107299 0 0) 0x7f1210117350 con 0x7f12100fc910 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1210119730 con 0x7f12100fc910 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1204001440 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1200001620 con 0x7f12100fc910 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512814350 0 0) 0x7f11f8003030 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 >> v1:172.21.15.107:6790/0 conn(0x7f1210103250 legacy=0x7f1210115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 >> v1:172.21.15.195:6789/0 conn(0x7f12100fc910 legacy=0x7f12100f9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121011a930 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1210117540 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f1216118700 10 client.27672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12101179b0 con 0x7f12100ff710 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.875+0000 7f1216118700 1 librados: init done 2024-02-10T08:39:27.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.876+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f12040021b0 con 0x7f12100ff710 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.876+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f12040036e0 con 0x7f12100ff710 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.876+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc004f20 con 0x7f12100ff710 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1204001720 con 0x7f12100ff710 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(273..273 src has 50..273) v4 ==== 9146+0+0 (unknown 3639460671 0 0) 0x7f1204079390 con 0x7f12100ff710 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 10 client.27672.objecter ms_dispatch 0x7f1210001550 osd_map(273..273 src has 50..273) v4 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 3 client.27672.objecter handle_osd_map got epochs [273,273] > 0 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 3 client.27672.objecter handle_osd_map decoding full epoch 273 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 20 client.27672.objecter dump_active .. 0 homeless 2024-02-10T08:39:27.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.877+0000 7f120cff9700 10 client.27672.objecter ms_handle_connect 0x7f11f8063c70 2024-02-10T08:39:27.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.881+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f120404c3c0 con 0x7f12100ff710 2024-02-10T08:39:27.938 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:27.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 -- 172.21.15.107:0/1488191124 >> v1:172.21.15.107:6790/0 conn(0x7f03780f9570 legacy=0x7f0378102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 -- 172.21.15.107:0/1488191124 shutdown_connections 2024-02-10T08:39:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 -- 172.21.15.107:0/1488191124 >> 172.21.15.107:0/1488191124 conn(0x7f03780f2510 msgr2=0x7f03780f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 -- 172.21.15.107:0/1488191124 shutdown_connections 2024-02-10T08:39:28.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 -- 172.21.15.107:0/1488191124 wait complete. 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 librados: starting msgr at 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.109+0000 7f038032c700 1 librados: starting objecter 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 Processor -- start 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 -- start start 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 librados: setting wanted keys 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 librados: calling monclient init 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0378102cf0 con 0x7f03780f89c0 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0378102ec0 con 0x7f03780f9570 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f038032c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f037811bc10 con 0x7f0378104350 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f037e0c8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03780f89c0 0x7f0378102550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43086/0 (socket says 172.21.15.107:43086) 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f037e8c9700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0378104350 0x7f03781184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35902/0 (socket says 172.21.15.107:35902) 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f037e0c8700 1 -- 172.21.15.107:0/3062634032 learned_addr learned my addr 172.21.15.107:0/3062634032 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:28.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter ms_handle_connect 0x7f03780f89c0 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter ms_handle_connect 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter ms_handle_connect 0x7f03780f9570 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330850660 0 0) 0x7f037811bc10 con 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0364003630 con 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195879720 0 0) 0x7f0378102ec0 con 0x7f03780f9570 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037811bc10 con 0x7f03780f9570 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107523140 0 0) 0x7f0364003630 con 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0378102ec0 con 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.110+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0370001cd0 con 0x7f0378104350 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487092732 0 0) 0x7f037811bc10 con 0x7f03780f9570 2024-02-10T08:39:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0364003630 con 0x7f03780f9570 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0368002f00 con 0x7f03780f9570 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3496430402 0 0) 0x7f0378102ec0 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.195:6789/0 conn(0x7f03780f9570 legacy=0x7f0378114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6789/0 conn(0x7f03780f89c0 legacy=0x7f0378102550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037811cdb0 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037811be00 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f038032c700 10 client.53228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f037811c270 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f038032c700 1 librados: init done 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0370002460 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0370005e10 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.111+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f035c004f00 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.112+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0370001fc0 con 0x7f0378104350 2024-02-10T08:39:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(273..273 src has 50..273) v4 ==== 9146+0+0 (unknown 3639460671 0 0) 0x7f03700794a0 con 0x7f0378104350 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 osd_map(273..273 src has 50..273) v4 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 3 client.53228.objecter handle_osd_map got epochs [273,273] > 0 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 3 client.53228.objecter handle_osd_map decoding full epoch 273 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 20 client.53228.objecter dump_active .. 0 homeless 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v273) v1 ==== 54+0+16 (unknown 348551053 0 2420000192) 0x7f0370079880 con 0x7f0378104350 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f0364063ad0 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 10 client.53228.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 1 2024-02-10T08:39:28.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 1 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 0 1 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 4 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 10 client.53228.objecter _send_command 1 2024-02-10T08:39:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c004950 con 0x7f035c004530 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.113+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c004530 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.114+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c004950 con 0x7f035c004530 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.114+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.114+0000 7f036f7fe700 10 client.53228.objecter _finish_command 1 = system:0 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.114+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 4 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.114+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 0 1 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 10 client.53228.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 2 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 2 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0051a0 osd=0 3 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 0 2 2024-02-10T08:39:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 4 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 10 client.53228.objecter _send_command 2 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.129+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c004d30 con 0x7f035c004530 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.131+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c004d30 con 0x7f035c004530 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.131+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.131+0000 7f036f7fe700 10 client.53228.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.131+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 4 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.131+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 0 2 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 10 client.53228.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 3 2024-02-10T08:39:28.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 1 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 4 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 10 client.53228.objecter _send_command 3 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c00bf60 con 0x7f035c003b40 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c003b40 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c00bf60 con 0x7f035c003b40 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 10 client.53228.objecter _finish_command 3 = system:0 2024-02-10T08:39:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.132+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 1 3 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 10 client.53228.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c004270 osd=1 3 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 1 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 10 client.53228.objecter _send_command 4 2024-02-10T08:39:28.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.147+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c00d280 con 0x7f035c003b40 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.148+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c00d280 con 0x7f035c003b40 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.148+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.148+0000 7f036f7fe700 10 client.53228.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.148+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 4 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.148+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 1 4 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 10 client.53228.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 5 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 5 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 2 5 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 4 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 10 client.53228.objecter _send_command 5 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c010d30 con 0x7f035c002070 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c002070 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c010d30 con 0x7f035c002070 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 10 client.53228.objecter _finish_command 5 = system:0 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 4 2024-02-10T08:39:28.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.149+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 2 5 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 10 client.53228.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 6 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 6 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0033a0 osd=2 3 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 2 6 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 4 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 10 client.53228.objecter _send_command 6 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.165+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c001630 con 0x7f035c002070 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c001630 con 0x7f035c002070 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f036f7fe700 10 client.53228.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 4 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 2 6 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 10 client.53228.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 7 2024-02-10T08:39:28.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 7 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 3 7 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 4 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 10 client.53228.objecter _send_command 7 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.166+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c00d000 con 0x7f035c010ea0 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c010ea0 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c00d000 con 0x7f035c010ea0 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 10 client.53228.objecter _finish_command 7 = system:0 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 4 2024-02-10T08:39:28.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.167+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 3 7 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 10 client.53228.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 8 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 8 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c00c8d0 osd=3 3 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 3 8 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 4 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 10 client.53228.objecter _send_command 8 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.182+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c0121c0 con 0x7f035c010ea0 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c0121c0 con 0x7f035c010ea0 2024-02-10T08:39:28.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 10 client.53228.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 4 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 3 8 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 10 client.53228.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 9 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 9 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 4 9 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 4 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 10 client.53228.objecter _send_command 9 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c01a270 con 0x7f035c0166f0 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.184+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c0166f0 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.185+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c01a270 con 0x7f035c0166f0 2024-02-10T08:39:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.185+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.185+0000 7f036f7fe700 10 client.53228.objecter _finish_command 9 = system:0 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.185+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 4 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.185+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 4 9 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 10 client.53228.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 10 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 10 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c0119f0 osd=4 3 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 4 10 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 4 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 10 client.53228.objecter _send_command 10 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.200+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c01a270 con 0x7f035c0166f0 2024-02-10T08:39:28.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c01a270 con 0x7f035c0166f0 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f036f7fe700 10 client.53228.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 4 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 4 10 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 10 client.53228.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 11 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 11 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 5 11 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 4 2024-02-10T08:39:28.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 10 client.53228.objecter _send_command 11 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.202+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c01b4c0 con 0x7f035c01bf30 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c01bf30 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c01b4c0 con 0x7f035c01bf30 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 10 client.53228.objecter _finish_command 11 = system:0 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 4 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.203+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 5 11 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 10 client.53228.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 12 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 12 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c01a270 osd=5 3 2024-02-10T08:39:28.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 5 12 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 4 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 10 client.53228.objecter _send_command 12 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.218+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c01b4c0 con 0x7f035c01bf30 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.219+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c01b4c0 con 0x7f035c01bf30 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.219+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.219+0000 7f036f7fe700 10 client.53228.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.219+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 4 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.219+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 5 12 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 10 client.53228.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 13 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 13 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 6 13 2024-02-10T08:39:28.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 4 2024-02-10T08:39:28.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 10 client.53228.objecter _send_command 13 2024-02-10T08:39:28.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c022c20 con 0x7f035c01f110 2024-02-10T08:39:28.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.220+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c01f110 2024-02-10T08:39:28.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.221+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c022c20 con 0x7f035c01f110 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.221+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.221+0000 7f036f7fe700 10 client.53228.objecter _finish_command 13 = system:0 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.221+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 4 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.221+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 6 13 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 10 client.53228.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 14 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 14 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c01b4c0 osd=6 3 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 6 14 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 4 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 10 client.53228.objecter _send_command 14 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.235+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c022c20 con 0x7f035c01f110 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c022c20 con 0x7f035c01f110 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f036f7fe700 10 client.53228.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 4 2024-02-10T08:39:28.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 6 14 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 10 client.53228.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 15 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 15 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 7 15 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 4 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 10 client.53228.objecter _send_command 15 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.237+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035c027040 con 0x7f035c0232e0 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.238+0000 7f036f7fe700 10 client.53228.objecter ms_handle_connect 0x7f035c0232e0 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.239+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f035c027040 con 0x7f035c0232e0 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.239+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:28.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.239+0000 7f036f7fe700 10 client.53228.objecter _finish_command 15 = system:0 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.239+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 4 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.239+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 7 15 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 10 client.53228.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign -1 16 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 20 client.53228.objecter _get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 15 client.53228.objecter _session_command_op_remove -1 16 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 20 client.53228.objecter get_session s=0x7f035c022c20 osd=7 3 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 15 client.53228.objecter _session_command_op_assign 7 16 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 4 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 10 client.53228.objecter _send_command 16 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.253+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f035c027000 con 0x7f035c0232e0 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f036f7fe700 1 -- 172.21.15.107:0/3062634032 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f035c027000 con 0x7f035c0232e0 2024-02-10T08:39:28.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f036f7fe700 10 client.53228.objecter ms_dispatch 0x7f0378001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f036f7fe700 10 client.53228.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f036f7fe700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 4 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f036f7fe700 15 client.53228.objecter _session_command_op_remove 7 16 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 librados: watch_flush enter 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 librados: watch_flush exit 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.0 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6801/3321596972 conn(0x7f035c004530 legacy=0x7f035c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0051a0 osd=0 1 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.1 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6805/133516126 conn(0x7f035c003b40 legacy=0x7f035c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c004270 osd=1 1 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.2 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6809/3575881728 conn(0x7f035c002070 legacy=0x7f035c00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0033a0 osd=2 1 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.3 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6813/2871825691 conn(0x7f035c010ea0 legacy=0x7f035c0112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c00c8d0 osd=3 1 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f037e8c9700 1 -- 172.21.15.107:0/3062634032 reap_dead start 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.4 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.195:6800/2499671285 conn(0x7f035c0166f0 legacy=0x7f035c018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f037e8c9700 1 -- 172.21.15.107:0/3062634032 reap_dead start 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c0119f0 osd=4 1 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.5 2024-02-10T08:39:28.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.195:6804/4178321486 conn(0x7f035c01bf30 legacy=0x7f035c01c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01a270 osd=5 1 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.6 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.195:6808/1417479017 conn(0x7f035c01f110 legacy=0x7f035c021500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c01b4c0 osd=6 1 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter close_session for osd.7 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.195:6812/1934176928 conn(0x7f035c0232e0 legacy=0x7f035c025750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter put_session s=0x7f035c022c20 osd=7 1 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 20 client.53228.objecter shutdown clearing up homeless session... 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 10 client.53228.objecter successfully canceled tick 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0364063ad0 legacy=0x7f0364065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> v1:172.21.15.107:6790/0 conn(0x7f0378104350 legacy=0x7f03781184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f037e8c9700 1 -- 172.21.15.107:0/3062634032 reap_dead start 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 shutdown_connections 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 >> 172.21.15.107:0/3062634032 conn(0x7f03780f2510 msgr2=0x7f03781077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:28.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 shutdown_connections 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 -- 172.21.15.107:0/3062634032 wait complete. 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:27.255+0000 7f038032c700 1 librados: shutdown 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:39:28.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:28.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:28.070+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f11fc0051d0 con 0x7f12100ff710 2024-02-10T08:39:28.092 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:28.111 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:28.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:28.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:27.113781+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.107:0/3062634032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:27.399785+0000 mgr.y (mgr.14553) 274 : cluster [DBG] pgmap v386: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:28.046652+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.049852+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]': finished 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.049918+0000 mon.a (mon.0) 1875 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:28.050022+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e274: 8 total, 8 up, 7 in 2024-02-10T08:39:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.053306+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.054651+0000 mon.a (mon.0) 1877 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.058597+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.059504+0000 mon.a (mon.0) 1878 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:28.072583+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:27.113781+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.107:0/3062634032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:27.399785+0000 mgr.y (mgr.14553) 274 : cluster [DBG] pgmap v386: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:28.046652+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.049852+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]': finished 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.049918+0000 mon.a (mon.0) 1875 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:28.050022+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e274: 8 total, 8 up, 7 in 2024-02-10T08:39:28.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.053306+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.054651+0000 mon.a (mon.0) 1877 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.058597+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.059504+0000 mon.a (mon.0) 1878 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:28.072583+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:27.113781+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.107:0/3062634032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:27.399785+0000 mgr.y (mgr.14553) 274 : cluster [DBG] pgmap v386: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:28.046652+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.049852+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-33", "mode": "writeback"}]': finished 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.049918+0000 mon.a (mon.0) 1875 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:28.050022+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e274: 8 total, 8 up, 7 in 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.053306+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.107:0/176366095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.054651+0000 mon.a (mon.0) 1877 : audit [INF] from='client.47932 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]: dispatch 2024-02-10T08:39:28.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.058597+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.059504+0000 mon.a (mon.0) 1878 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:28.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:28.072583+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:39:28.593 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:28.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:28.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:28.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:28.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:28.930 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:28.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:29.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:29.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.056+0000 7f120cff9700 1 -- 172.21.15.107:0/3597477642 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v275) v1 ==== 113+0+0 (unknown 1908222532 0 0) 0x7f120404a830 con 0x7f12100ff710 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.060+0000 7f1216118700 10 librados: watch_flush enter 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.065+0000 7f1216118700 10 librados: watch_flush exit 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.065+0000 7f1216118700 20 client.27672.objecter shutdown clearing up homeless session... 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.065+0000 7f1216118700 10 client.27672.objecter successfully canceled tick 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.065+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 >> v1:172.21.15.107:6800/2634842009 conn(0x7f11f8063c70 legacy=0x7f11f8066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.065+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 >> v1:172.21.15.107:6789/0 conn(0x7f12100ff710 legacy=0x7f12100fa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.067+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 shutdown_connections 2024-02-10T08:39:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.067+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 >> 172.21.15.107:0/3597477642 conn(0x7f12100f2510 msgr2=0x7f12100612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.068+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 shutdown_connections 2024-02-10T08:39:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.069+0000 7f1216118700 1 -- 172.21.15.107:0/3597477642 wait complete. 2024-02-10T08:39:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:29.069+0000 7f1216118700 1 librados: shutdown 2024-02-10T08:39:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T08:39:29.099 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:29.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:29.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:29.234 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:29.254 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:29.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:29.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:29.385 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:29.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:39:29.834 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:29.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:29.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:29.967 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:29.988 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:29.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 1 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 10 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 13 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 7 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 14 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 0 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 15 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 11 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 5 0 2024-02-10T08:39:30.073 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 8 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 6 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 3 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 4 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 12 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 9 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 2 0 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (45 ms) 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 3 tests from LibRadosListEC (1101 ms total) 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3014 ms) 2024-02-10T08:39:30.074 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] 1 test from LibRadosListNP (3014 ms total) 2024-02-10T08:39:30.075 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: 2024-02-10T08:39:30.075 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [----------] Global test environment tear-down 2024-02-10T08:39:30.075 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [==========] 11 tests from 3 test suites ran. (179836 ms total) 2024-02-10T08:39:30.075 INFO:tasks.workunit.client.0.smithi107.stdout: api_list: [ PASSED ] 11 tests. 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112548 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112548 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112540 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112540 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112612 2024-02-10T08:39:30.076 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112612 2024-02-10T08:39:30.077 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:39:30.077 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112406 2024-02-10T08:39:30.077 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112406 2024-02-10T08:39:30.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25276.0:2 52.14 52:28e9c4a8:test-rados-api-smithi107-112562-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:39:30.117 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 163.17918212999999, 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999, 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25276", 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935027+0000", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935035+0000", 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0684e-05 2024-02-10T08:39:30.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935046+0000", 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7470000000000005e-06 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935054+0000", 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020156359999999999 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937070+0000", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4700000000000002e-06 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937078+0000", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020263032 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957341+0000", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4965999999999999e-05 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957426+0000", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699000000000001e-05 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957524+0000", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062096199999999997 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958145+0000", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0729999999999994e-06 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958154+0000", 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048278610000000001 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962982+0000", 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0270999999999999e-05 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963042+0000", 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6792000000000002e-05 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963089+0000", 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.124 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25291.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.187475447, 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003, 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25291", 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2248243388", 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926734+0000", 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8688000000000001e-05 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926753+0000", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8120000000000002e-06 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926762+0000", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6359999999999994e-06 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926770+0000", 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018594099999999999 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926956+0000", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4220000000000003e-06 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926963+0000", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025476736 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952440+0000", 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103766 2024-02-10T08:39:30.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952544+0000", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013851099999999999 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952682+0000", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047535930000000004 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957436+0000", 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065080549999999996 2024-02-10T08:39:30.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963944+0000", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004709319 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968653+0000", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4684999999999995e-05 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968718+0000", 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6301999999999999e-05 2024-02-10T08:39:30.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968754+0000", 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35219.0:9 51.1s0 51:967778f1:test-rados-api-smithi107-112451-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.169070933, 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237, 2024-02-10T08:39:30.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35219", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945138+0000", 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7849999999999992e-06 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945146+0000", 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5050000000000007e-06 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945155+0000", 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0849999999999999e-06 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945162+0000", 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001228563 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946390+0000", 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8227999999999996e-05 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946479+0000", 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382076999999998 2024-02-10T08:39:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.962861+0000", 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.14907.0:2 55.15 55:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.16845571900001, 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724, 2024-02-10T08:39:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.14907", 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4285222104", 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:39:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945754+0000", 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9549999999999997e-06 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945760+0000", 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3450000000000006e-06 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945769+0000", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2589999999999997e-06 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945775+0000", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01446833 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960243+0000", 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118359 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960362+0000", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016521099999999999 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960527+0000", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010993870000000001 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.961626+0000", 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060821130000000001 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967708+0000", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002001966 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969710+0000", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3856999999999998e-05 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969744+0000", 2024-02-10T08:39:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1362e-05 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969765+0000", 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.24889.0:19 23.d e114/102 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head v 114'3, mlcod=0'0)", 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.16836514299999, 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503, 2024-02-10T08:39:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5839999999999995e-06 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945854+0000", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3709999999999999e-06 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945860+0000", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7637999999999999e-05 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945918+0000", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014029747 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959947+0000", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2229999999999999e-05 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959969+0000", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076815219999999997 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967651+0000", 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5574000000000001e-05 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967697+0000", 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(50.4s1 epoch 114/111 rep_tid 1204 entries 114'2 (0'0) error 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head by client.35222.0:9 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:39:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.16833930199999, 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002, 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9130000000000003e-06 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945880+0000", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056525589999999997 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951532+0000", 2024-02-10T08:39:30.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6671e-05 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951549+0000", 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056510120000000004 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957200+0000", 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067084960000000004 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963908+0000", 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.5.0:1233 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'1, mlcod=0'0)", 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:39:30.156 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 159.130132592, 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001, 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984078+0000", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7410000000000004e-06 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984087+0000", 2024-02-10T08:39:30.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.772e-06 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984096+0000", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8090000000000004e-06 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984103+0000", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000327542 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984430+0000", 2024-02-10T08:39:30.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8281999999999999e-05 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984449+0000", 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017470151999999999 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001919+0000", 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4270999999999999e-05 2024-02-10T08:39:30.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001943+0000", 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:8 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'2, mlcod=0'0)", 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 159.13009926699999, 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001, 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.161 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8529999999999997e-06 2024-02-10T08:39:30.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984117+0000", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9869999999999999e-06 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984122+0000", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.625e-06 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984126+0000", 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046452199999999998 2024-02-10T08:39:30.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984591+0000", 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6535999999999999e-05 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984607+0000", 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023819407000000001 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008427+0000", 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7294e-05 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008454+0000", 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.165 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:4 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 157.047279441, 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002, 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066929+0000", 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8779999999999993e-06 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066939+0000", 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.431e-05 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066953+0000", 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3720000000000005e-06 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066962+0000", 2024-02-10T08:39:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040994600000000001 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071061+0000", 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0001153 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071177+0000", 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014616299999999999 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071323+0000", 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020771520000000001 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.073400+0000", 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00123847 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.074638+0000", 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082236259999999995 2024-02-10T08:39:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082862+0000", 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7891e-05 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082900+0000", 2024-02-10T08:39:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6568e-05 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082926+0000", 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:5 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e121)", 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 155.07447684799999, 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002, 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:39:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:39:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039732+0000", 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1054e-05 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039743+0000", 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2641e-05 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039755+0000", 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9560000000000003e-06 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039764+0000", 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9907999999999997e-05 2024-02-10T08:39:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039814+0000", 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6459999999999993e-06 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039822+0000", 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030549936 2024-02-10T08:39:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070372+0000", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4599999999999996e-05 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070456+0000", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019660001 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.090116+0000", 2024-02-10T08:39:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26977.0:63 66.1b 66:d9f0651a:test-rados-api-smithi107-112525-13::foo:head [write 0~128] snapc 16=[16] ondisk+write+known_if_redirected e141)", 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 135.07748480800001, 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635, 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26977", 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 63 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036723+0000", 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4849e-05 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036738+0000", 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1770000000000001e-05 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036749+0000", 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1699999999999997e-06 2024-02-10T08:39:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036758+0000", 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000209371 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036967+0000", 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0199999999999997e-06 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036974+0000", 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020105096999999999 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057079+0000", 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120828 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057200+0000", 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015114300000000001 2024-02-10T08:39:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057351+0000", 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021251266000000001 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.078602+0000", 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032546469999999998 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081857+0000", 2024-02-10T08:39:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6645e-05 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081884+0000", 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8924000000000001e-05 2024-02-10T08:39:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081933+0000", 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7667999999999998e-05 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081980+0000", 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.50096.0:2 91.6 91:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 134.13849203500001, 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012101086, 2024-02-10T08:39:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.50096", 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1027110602", 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975719+0000", 2024-02-10T08:39:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975717+0000", 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5640000000000002e-06 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975726+0000", 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6456e-05 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975743+0000", 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6969999999999999e-06 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975751+0000", 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3288999999999998e-05 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975805+0000", 2024-02-10T08:39:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5888000000000002e-05 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975891+0000", 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5280999999999998e-05 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-02-10T08:39:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.975986+0000", 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091295500000000002 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.976899+0000", 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029384300000000001 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.977193+0000", 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010554218000000001 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987747+0000", 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4509999999999999e-05 2024-02-10T08:39:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987791+0000", 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9099e-05 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.987820+0000", 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012101086 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47371.0:2 97.6 97:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-02-10T08:39:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:39:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 128.180451011, 2024-02-10T08:39:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020953844999999999, 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47371", 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3974906254", 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933758+0000", 2024-02-10T08:39:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9530000000000002e-06 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933765+0000", 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7169999999999997e-06 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933774+0000", 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0809999999999996e-06 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933781+0000", 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8953999999999999e-05 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933830+0000", 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4800000000000004e-06 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933838+0000", 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014440671 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948278+0000", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011152 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948390+0000", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139682 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948530+0000", 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001290404 2024-02-10T08:39:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.949820+0000", 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4445e-05 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.949854+0000", 2024-02-10T08:39:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047572279999999996 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954612+0000", 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9101999999999998e-05 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954671+0000", 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3550000000000001e-05 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954714+0000", 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020953844999999999 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52871.0:1 130.6s0 130:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 87.154987603999999, 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0506345260000001, 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52871", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1154190771", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959222+0000", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959230+0000", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0638e-05 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959241+0000", 2024-02-10T08:39:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3639999999999999e-06 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959249+0000", 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6088999999999999e-05 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959295+0000", 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.23e-06 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959298+0000", 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.042640617 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.001939+0000", 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8879e-05 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.002038+0000", 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024591899999999999 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.002284+0000", 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001147835 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.003432+0000", 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063665730000000004 2024-02-10T08:39:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.009798+0000", 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0049000000000001e-05 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.009858+0000", 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0506345260000001 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.1.0:395 60.1e e191/135 60:7a4cebd2:test-rados-api-smithi107-112490-14::foo69:4 v 191'11, mlcod=149'7)", 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 84.111080035000001, 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017016453000000001, 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2029999999999992e-06 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003139+0000", 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0289999999999994e-06 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003148+0000", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3269999999999997e-06 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003155+0000", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01388257 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017037+0000", 2024-02-10T08:39:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8848e-05 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017056+0000", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030742999999999999 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.020130+0000", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7609e-05 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.020148+0000", 2024-02-10T08:39:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017016453000000001 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.2.0:345 60.c e191/135 60:37d01833:test-rados-api-smithi107-112490-14::foo42:4 v 191'44, mlcod=190'38)", 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 84.106224538000006, 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013120199000000001, 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0859999999999996e-06 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007994+0000", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007999+0000", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.019e-06 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.008003+0000", 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099050619999999992 2024-02-10T08:39:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017908+0000", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8536e-05 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017927+0000", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031526549999999999 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.021080+0000", 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7427999999999999e-05 2024-02-10T08:39:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.021107+0000", 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013120199000000001 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47632.0:1 134.5s0 134:a12dce25:test-rados-api-smithi107-112480-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected e194)", 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 82.094953740999998, 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.020938202, 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47632", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4023540388", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:39:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019256+0000", 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8900000000000001e-06 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019263+0000", 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019270+0000", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4020000000000001e-06 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019276+0000", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9276000000000001e-05 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019315+0000", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2820000000000001e-06 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019318+0000", 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.010518837 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.029837+0000", 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012234799999999999 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.029959+0000", 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032286800000000003 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.030282+0000", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077056499999999997 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.031053+0000", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009073434 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.040126+0000", 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9586999999999996e-05 2024-02-10T08:39:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.040196+0000", 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.020938202 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52931.0:2 137.a 137:53c7e431:::oid_AioUnlockPP_:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e198)", 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 78.167286113000003, 2024-02-10T08:39:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022407323, 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52931", 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4202333394", 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946923+0000", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.064e-06 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946932+0000", 2024-02-10T08:39:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0265000000000001e-05 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946942+0000", 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.328e-06 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946950+0000", 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000488782 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.947438+0000", 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1780000000000003e-06 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.947447+0000", 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013418744 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.960865+0000", 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000128928 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.960994+0000", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000137465 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.961132+0000", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029011100000000001 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.961422+0000", 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074843169999999999 2024-02-10T08:39:30.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.968906+0000", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000371484 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969278+0000", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1267999999999999e-05 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969309+0000", 2024-02-10T08:39:30.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3768999999999999e-05 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969333+0000", 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022407323 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52997.0:2 146.6 146:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-02-10T08:39:30.245 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 63.158145787000002, 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015476029000000001, 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52997", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1556895985", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:39:30.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956064+0000", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0999999999999998e-06 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956071+0000", 2024-02-10T08:39:30.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4700000000000008e-06 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956081+0000", 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5719999999999999e-06 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956087+0000", 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102502 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956190+0000", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6790000000000002e-06 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956196+0000", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099999990000000007 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966196+0000", 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114827 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966310+0000", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014374799999999999 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966454+0000", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00066427700000000001 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.967118+0000", 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025337910000000001 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.969652+0000", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017909740000000001 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971443+0000", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0241000000000002e-05 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971493+0000", 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8279999999999999e-05 2024-02-10T08:39:30.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971542+0000", 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015476029000000001 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47884.0:1 171.6s0 171:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e259)", 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 17.140909666999999, 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0029141699999999, 2024-02-10T08:39:30.252 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47884", 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4001272799", 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973300+0000", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973307+0000", 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.827e-06 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973314+0000", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1540000000000001e-06 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973319+0000", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4014999999999999e-05 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973353+0000", 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5909999999999998e-06 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973355+0000", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.999621396 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.972977+0000", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9870999999999994e-05 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973067+0000", 2024-02-10T08:39:30.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027885300000000002 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973345+0000", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035942999999999999 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973705+0000", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024675819999999998 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.976172+0000", 2024-02-10T08:39:30.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3427999999999998e-05 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.976216+0000", 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0029141699999999 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:39:30.258 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:39:30.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.055655+0000 mon.a (mon.0) 1880 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:30.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.055786+0000 mon.a (mon.0) 1881 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:30.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.055889+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:39:30.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:29.055951+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e275: 8 total, 8 up, 7 in 2024-02-10T08:39:30.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.059206+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.060581+0000 mon.a (mon.0) 1884 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.085511+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.085939+0000 mon.a (mon.0) 1885 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.093196+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.095309+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.095616+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.392868+0000 mon.c (mon.2) 314 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.055655+0000 mon.a (mon.0) 1880 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:30.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.055786+0000 mon.a (mon.0) 1881 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.055889+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:29.055951+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e275: 8 total, 8 up, 7 in 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.059206+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.060581+0000 mon.a (mon.0) 1884 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.085511+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.085939+0000 mon.a (mon.0) 1885 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.093196+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.095309+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.095616+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:30.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.392868+0000 mon.c (mon.2) 314 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:39:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:30.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:29.400159+0000 mgr.y (mgr.14553) 275 : cluster [DBG] pgmap v389: 250 pgs: 32 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:30.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.631469+0000 mon.a (mon.0) 1889 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.910689+0000 mon.c (mon.2) 315 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:39:30.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.911301+0000 mon.c (mon.2) 316 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:39:30.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:29.917071+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:29.400159+0000 mgr.y (mgr.14553) 275 : cluster [DBG] pgmap v389: 250 pgs: 32 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:30.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.631469+0000 mon.a (mon.0) 1889 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.910689+0000 mon.c (mon.2) 315 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:39:30.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.911301+0000 mon.c (mon.2) 316 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:39:30.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:29.917071+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.263 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:30.263 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:30.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:30.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:30.390 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:30.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.055655+0000 mon.a (mon.0) 1880 : audit [INF] from='client.47932 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi107-112411-48"}]': finished 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.055786+0000 mon.a (mon.0) 1881 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112443-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.055889+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? v1:172.21.15.107:0/3597477642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:29.055951+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e275: 8 total, 8 up, 7 in 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.059206+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.060581+0000 mon.a (mon.0) 1884 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.085511+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.107:0/2739571000' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.085939+0000 mon.a (mon.0) 1885 : audit [INF] from='client.53222 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.093196+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.095309+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.095616+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.392868+0000 mon.c (mon.2) 314 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:39:30.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:29.400159+0000 mgr.y (mgr.14553) 275 : cluster [DBG] pgmap v389: 250 pgs: 32 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.631469+0000 mon.a (mon.0) 1889 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.910689+0000 mon.c (mon.2) 315 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:39:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.911301+0000 mon.c (mon.2) 316 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:39:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:29.917071+0000 mon.a (mon.0) 1890 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:30.392 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:30.411 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:30.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:30.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:30.540 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:30.559 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:30.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:30.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:39:30.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T08:39:30.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:30.688 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:30.706 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:30.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:30.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:30.836 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:30.853 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:30.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:30.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:30.983 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.002 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:31.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:31.131 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.150 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:31.157 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:31.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.058893+0000 mon.a (mon.0) 1891 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.058978+0000 mon.a (mon.0) 1892 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.059045+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:30.059094+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.060872+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.067905+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.068247+0000 mon.a (mon.0) 1896 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.116968+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.278 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:30.117312+0000 mon.a (mon.0) 1897 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.279 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.298 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:31.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:31.428 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.058893+0000 mon.a (mon.0) 1891 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.058978+0000 mon.a (mon.0) 1892 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.059045+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:30.059094+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.060872+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.067905+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.068247+0000 mon.a (mon.0) 1896 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.116968+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:30.117312+0000 mon.a (mon.0) 1897 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.058893+0000 mon.a (mon.0) 1891 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.058978+0000 mon.a (mon.0) 1892 : audit [INF] from='client.53222 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi107-112443-3","pool2":"test-rados-api-smithi107-112443-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.059045+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi107-112411-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:30.059094+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.060872+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.067905+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.107:0/583093704' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.068247+0000 mon.a (mon.0) 1896 : audit [INF] from='client.53210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]: dispatch 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.116968+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:30.117312+0000 mon.a (mon.0) 1897 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:31.446 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:31.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:31.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:31.578 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.604 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:31.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:31.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:39:31.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:31.732 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.752 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:31.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:31.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:31.892 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:32.005 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:32.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:32.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:32.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:32.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:32.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.209012+0000 mon.a (mon.0) 1898 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:32.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.209283+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:32.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:31.209370+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-10T08:39:32.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.221880+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.222220+0000 mon.a (mon.0) 1901 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.233214+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.233512+0000 mon.a (mon.0) 1902 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.234809+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.235108+0000 mon.a (mon.0) 1903 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.235539+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.235786+0000 mon.a (mon.0) 1904 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.314023+0000 mgr.y (mgr.14553) 276 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:31.401012+0000 mgr.y (mgr.14553) 277 : cluster [DBG] pgmap v392: 218 pgs: 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.965202+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.965283+0000 mon.a (mon.0) 1906 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.965370+0000 mon.a (mon.0) 1907 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:31.965432+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-10T08:39:32.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.966861+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:31.967344+0000 mon.a (mon.0) 1909 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.390 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.209012+0000 mon.a (mon.0) 1898 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.209283+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:31.209370+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.221880+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.222220+0000 mon.a (mon.0) 1901 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.233214+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.233512+0000 mon.a (mon.0) 1902 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.234809+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.235108+0000 mon.a (mon.0) 1903 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.235539+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.235786+0000 mon.a (mon.0) 1904 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.314023+0000 mgr.y (mgr.14553) 276 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:31.401012+0000 mgr.y (mgr.14553) 277 : cluster [DBG] pgmap v392: 218 pgs: 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.209012+0000 mon.a (mon.0) 1898 : audit [INF] from='client.53210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi107-112403-39"}]': finished 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.209283+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:31.209370+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.221880+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.222220+0000 mon.a (mon.0) 1901 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.233214+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.233512+0000 mon.a (mon.0) 1902 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.234809+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.235108+0000 mon.a (mon.0) 1903 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.235539+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.235786+0000 mon.a (mon.0) 1904 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.314023+0000 mgr.y (mgr.14553) 276 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:31.401012+0000 mgr.y (mgr.14553) 277 : cluster [DBG] pgmap v392: 218 pgs: 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.965202+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.965283+0000 mon.a (mon.0) 1906 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.965370+0000 mon.a (mon.0) 1907 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:31.965432+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.966861+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:31.967344+0000 mon.a (mon.0) 1909 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.965202+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi107-112411-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.965283+0000 mon.a (mon.0) 1906 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.965370+0000 mon.a (mon.0) 1907 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi107-112403-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:31.965432+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.966861+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:31.967344+0000 mon.a (mon.0) 1909 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:32.740 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:32.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:32.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:32.872 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:33.739 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:39:33.740 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:39:33.740 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:39:33.740 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:39:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:33.211610+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-10T08:39:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.223248+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.223627+0000 mon.a (mon.0) 1911 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:33.401353+0000 mgr.y (mgr.14553) 278 : cluster [DBG] pgmap v395: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:34.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:33.211610+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-10T08:39:34.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.223248+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.223627+0000 mon.a (mon.0) 1911 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:33.401353+0000 mgr.y (mgr.14553) 278 : cluster [DBG] pgmap v395: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:34.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:33.211610+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-10T08:39:34.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.223248+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.223627+0000 mon.a (mon.0) 1911 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:34.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:33.401353+0000 mgr.y (mgr.14553) 278 : cluster [DBG] pgmap v395: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-10T08:39:34.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T08:39:34.681 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:33.813120+0000 osd.6 (osd.6) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:33.813259+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.973518+0000 mon.a (mon.0) 1912 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.973613+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:33.973674+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-10T08:39:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.977424+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.977570+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:33.985999+0000 mon.a (mon.0) 1916 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:33.813120+0000 osd.6 (osd.6) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:33.813259+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.973518+0000 mon.a (mon.0) 1912 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.973613+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:33.973674+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.977424+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.977570+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:33.985999+0000 mon.a (mon.0) 1916 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:33.813120+0000 osd.6 (osd.6) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:33.813259+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.973518+0000 mon.a (mon.0) 1912 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi107-112403-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.973613+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:33.973674+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-10T08:39:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.977424+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.977570+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:33.985999+0000 mon.a (mon.0) 1916 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]: dispatch 2024-02-10T08:39:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.012+0000 7fcea7bb3700 1 -- 172.21.15.107:0/563419097 >> v1:172.21.15.107:6789/0 conn(0x7fcea00fc910 legacy=0x7fcea00fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.012+0000 7fcea7bb3700 1 -- 172.21.15.107:0/563419097 shutdown_connections 2024-02-10T08:39:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.012+0000 7fcea7bb3700 1 -- 172.21.15.107:0/563419097 >> 172.21.15.107:0/563419097 conn(0x7fcea00f2510 msgr2=0x7fcea00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- 172.21.15.107:0/563419097 shutdown_connections 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- 172.21.15.107:0/563419097 wait complete. 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 librados: starting msgr at 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 librados: starting objecter 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 Processor -- start 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- start start 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 librados: setting wanted keys 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 librados: calling monclient init 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcea0117350 con 0x7fcea00ff710 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcea01184f0 con 0x7fcea00fc910 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea7bb3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcea01196f0 con 0x7fcea0103250 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea6150700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcea0103250 0x7fcea01159e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36108/0 (socket says 172.21.15.107:36108) 2024-02-10T08:39:35.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea514e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcea00ff710 0x7fcea00fa1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46732/0 (socket says 172.21.15.107:46732) 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fcea6150700 1 -- 172.21.15.107:0/1210996869 learned_addr learned my addr 172.21.15.107:0/1210996869 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fce96ffd700 10 client.?.objecter ms_handle_connect 0x7fcea00ff710 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fce96ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fce96ffd700 10 client.?.objecter ms_handle_connect 0x7fcea0103250 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.013+0000 7fce96ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 10 client.?.objecter ms_handle_connect 0x7fcea00fc910 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904220083 0 0) 0x7fcea01196f0 con 0x7fcea0103250 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c003800 con 0x7fcea0103250 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109518274 0 0) 0x7fcea0117350 con 0x7fcea00ff710 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea01196f0 con 0x7fcea00ff710 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657756246 0 0) 0x7fcea01184f0 con 0x7fcea00fc910 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea0117350 con 0x7fcea00fc910 2024-02-10T08:39:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907286693 0 0) 0x7fce8c003800 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea01184f0 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce98001d70 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809741237 0 0) 0x7fcea01196f0 con 0x7fcea00ff710 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8c003800 con 0x7fcea00ff710 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1970673202 0 0) 0x7fcea0117350 con 0x7fcea00fc910 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea01196f0 con 0x7fcea00fc910 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce900018c0 con 0x7fcea00ff710 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce9c002080 con 0x7fcea00fc910 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1634775064 0 0) 0x7fcea01184f0 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 >> v1:172.21.15.195:6789/0 conn(0x7fcea00fc910 legacy=0x7fcea00f9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 >> v1:172.21.15.107:6789/0 conn(0x7fcea00ff710 legacy=0x7fcea00fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea011a8f0 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01198e0 con 0x7fcea0103250 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fcea7bb3700 10 client.53252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea0119dd0 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fcea7bb3700 1 librados: init done 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fce980026e0 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.014+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce98005d10 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.015+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce84004f20 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.015+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fce98002850 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(281..281 src has 50..281) v4 ==== 8915+0+0 (unknown 3272959435 0 0) 0x7fce98079160 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 10 client.53252.objecter ms_dispatch 0x7fcea0001550 osd_map(281..281 src has 50..281) v4 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 3 client.53252.objecter handle_osd_map got epochs [281,281] > 0 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 3 client.53252.objecter handle_osd_map decoding full epoch 281 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 20 client.53252.objecter dump_active .. 0 homeless 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.016+0000 7fce96ffd700 10 client.53252.objecter ms_handle_connect 0x7fce8c063e90 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.020+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fce9804c200 con 0x7fcea0103250 2024-02-10T08:39:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.199+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fce840051d0 con 0x7fcea0103250 2024-02-10T08:39:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:35.924 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:35] "GET /metrics HTTP/1.1" 200 188389 "" "Prometheus/2.33.4" 2024-02-10T08:39:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:35.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:35.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.990+0000 7fce96ffd700 1 -- 172.21.15.107:0/1210996869 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v282) v1 ==== 91+0+0 (unknown 1868652518 0 0) 0x7fce9804a670 con 0x7fcea0103250 2024-02-10T08:39:35.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 10 librados: watch_flush enter 2024-02-10T08:39:35.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 10 librados: watch_flush exit 2024-02-10T08:39:35.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 20 client.53252.objecter shutdown clearing up homeless session... 2024-02-10T08:39:35.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 10 client.53252.objecter successfully canceled tick 2024-02-10T08:39:35.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 >> v1:172.21.15.107:6800/2634842009 conn(0x7fce8c063e90 legacy=0x7fce8c066350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:35.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.993+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 >> v1:172.21.15.107:6790/0 conn(0x7fcea0103250 legacy=0x7fcea01159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.996+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 shutdown_connections 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.996+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 >> 172.21.15.107:0/1210996869 conn(0x7fcea00f2510 msgr2=0x7fcea01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.996+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 shutdown_connections 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.996+0000 7fcea7bb3700 1 -- 172.21.15.107:0/1210996869 wait complete. 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:35.997+0000 7fcea7bb3700 1 librados: shutdown 2024-02-10T08:39:35.998 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:39:36.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:36.084 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:36.136 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:36.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:34.970787+0000 mon.a (mon.0) 1917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:36.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:34.975390+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:36.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:34.975469+0000 mon.a (mon.0) 1919 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:36.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:34.975552+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-10T08:39:36.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:34.982111+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:36.140 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.200596+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.107:0/1210996869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.140 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.200833+0000 mon.a (mon.0) 1922 : audit [INF] from='client.53252 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.140 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:35.401916+0000 mgr.y (mgr.14553) 279 : cluster [DBG] pgmap v398: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:36.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:36.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:34.970787+0000 mon.a (mon.0) 1917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:36.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:34.975390+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:36.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:34.975469+0000 mon.a (mon.0) 1919 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:36.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:34.975552+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-10T08:39:36.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:34.982111+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.200596+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.107:0/1210996869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.200833+0000 mon.a (mon.0) 1922 : audit [INF] from='client.53252 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:35.401916+0000 mgr.y (mgr.14553) 279 : cluster [DBG] pgmap v398: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:34.970787+0000 mon.a (mon.0) 1917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:34.975390+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:34.975469+0000 mon.a (mon.0) 1919 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-33"}]': finished 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:34.975552+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:34.982111+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.200596+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.107:0/1210996869' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.200833+0000 mon.a (mon.0) 1922 : audit [INF] from='client.53252 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:39:36.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:35.401916+0000 mgr.y (mgr.14553) 279 : cluster [DBG] pgmap v398: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:36.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:36.269 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:36.368 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:36.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:36.387 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:39:36.387 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:39:36.387 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:39:36.387 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:39:36.387 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:39:36.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:36.512 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:36.631 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:36.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:36.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:36.762 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:37.011 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:37.012 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:35.978426+0000 mon.a (mon.0) 1923 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:37.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.987885+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:37.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.987992+0000 mon.a (mon.0) 1925 : audit [INF] from='client.53252 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:39:37.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:35.988045+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-10T08:39:37.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.994252+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:35.994738+0000 mon.a (mon.0) 1927 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:37.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:37.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T08:39:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.052+0000 7fd74147e700 1 -- 172.21.15.107:0/1571504917 >> v1:172.21.15.107:6790/0 conn(0x7fd73c104350 legacy=0x7fd73c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.052+0000 7fd74147e700 1 -- 172.21.15.107:0/1571504917 shutdown_connections 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.052+0000 7fd74147e700 1 -- 172.21.15.107:0/1571504917 >> 172.21.15.107:0/1571504917 conn(0x7fd73c0f2510 msgr2=0x7fd73c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 -- 172.21.15.107:0/1571504917 shutdown_connections 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 -- 172.21.15.107:0/1571504917 wait complete. 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 librados: starting msgr at 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 librados: starting objecter 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 Processor -- start 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 -- start start 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 librados: setting wanted keys 2024-02-10T08:39:37.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.053+0000 7fd74147e700 1 librados: calling monclient init 2024-02-10T08:39:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c102b70 con 0x7fd73c0f89c0 2024-02-10T08:39:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c102d40 con 0x7fd73c104350 2024-02-10T08:39:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd73c102f30 con 0x7fd73c0f9570 2024-02-10T08:39:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd73affd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd73c0f89c0 0x7fd73c1023d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46782/0 (socket says 172.21.15.107:46782) 2024-02-10T08:39:37.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd73affd700 1 -- 172.21.15.107:0/682888550 learned_addr learned my addr 172.21.15.107:0/682888550 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter ms_handle_connect 0x7fd73c0f89c0 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter ms_handle_connect 0x7fd73c0f9570 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter ms_handle_connect 0x7fd73c104350 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682025672 0 0) 0x7fd73c102b70 con 0x7fd73c0f89c0 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7240033f0 con 0x7fd73c0f89c0 2024-02-10T08:39:37.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891484554 0 0) 0x7fd73c102d40 con 0x7fd73c104350 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c102b70 con 0x7fd73c104350 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097548181 0 0) 0x7fd73c102f30 con 0x7fd73c0f9570 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c102d40 con 0x7fd73c0f9570 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191605674 0 0) 0x7fd73c102b70 con 0x7fd73c104350 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c102f30 con 0x7fd73c104350 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3440726368 0 0) 0x7fd7240033f0 con 0x7fd73c0f89c0 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c102b70 con 0x7fd73c0f89c0 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd734002570 con 0x7fd73c104350 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd72c001900 con 0x7fd73c0f89c0 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913224678 0 0) 0x7fd73c102d40 con 0x7fd73c0f9570 2024-02-10T08:39:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7240033f0 con 0x7fd73c0f9570 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd730001830 con 0x7fd73c0f9570 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2642663493 0 0) 0x7fd73c102f30 con 0x7fd73c104350 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6790/0 conn(0x7fd73c0f9570 legacy=0x7fd73c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6789/0 conn(0x7fd73c0f89c0 legacy=0x7fd73c1023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c11cc20 con 0x7fd73c104350 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c119c50 con 0x7fd73c104350 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 10 client.47995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c11a0c0 con 0x7fd73c104350 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.054+0000 7fd74147e700 1 librados: init done 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.055+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd7340013f0 con 0x7fd73c104350 2024-02-10T08:39:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.055+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd734003940 con 0x7fd73c104350 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.055+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd728004f00 con 0x7fd73c104350 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd734001c00 con 0x7fd73c104350 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(283..283 src has 50..283) v4 ==== 8461+0+0 (unknown 3434291695 0 0) 0x7fd734076c90 con 0x7fd73c104350 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 osd_map(283..283 src has 50..283) v4 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 3 client.47995.objecter handle_osd_map got epochs [283,283] > 0 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 3 client.47995.objecter handle_osd_map decoding full epoch 283 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 20 client.47995.objecter dump_active .. 0 homeless 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v283) v1 ==== 54+0+16 (unknown 1450225559 0 2420000192) 0x7fd734079080 con 0x7fd73c104350 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.056+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd724063d50 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 10 client.47995.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 1 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 1 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 0 1 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 4 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 10 client.47995.objecter _send_command 1 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728004950 con 0x7fd728004530 2024-02-10T08:39:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd728004530 2024-02-10T08:39:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728004950 con 0x7fd728004530 2024-02-10T08:39:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 10 client.47995.objecter _finish_command 1 = system:0 2024-02-10T08:39:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 4 2024-02-10T08:39:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.057+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 0 1 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 10 client.47995.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 2 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 2 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd7280051a0 osd=0 3 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 0 2 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 4 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 10 client.47995.objecter _send_command 2 2024-02-10T08:39:37.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.072+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728004d30 con 0x7fd728004530 2024-02-10T08:39:37.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728004d30 con 0x7fd728004530 2024-02-10T08:39:37.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd723fff700 10 client.47995.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 4 2024-02-10T08:39:37.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 0 2 2024-02-10T08:39:37.075 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 10 client.47995.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 1 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 4 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 10 client.47995.objecter _send_command 3 2024-02-10T08:39:37.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.074+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd72800c000 con 0x7fd728004150 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd728004150 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd72800c000 con 0x7fd728004150 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 10 client.47995.objecter _finish_command 3 = system:0 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 4 2024-02-10T08:39:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.075+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 1 3 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 10 client.47995.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 4 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 4 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003fe0 osd=1 3 2024-02-10T08:39:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 1 4 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 4 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 10 client.47995.objecter _send_command 4 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.090+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728001850 con 0x7fd728004150 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728001850 con 0x7fd728004150 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 10 client.47995.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 4 2024-02-10T08:39:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 1 4 2024-02-10T08:39:37.093 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 10 client.47995.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 5 2024-02-10T08:39:37.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 5 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 2 5 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 4 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 10 client.47995.objecter _send_command 5 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728003e30 con 0x7fd72800d320 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.092+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd72800d320 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.093+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728003e30 con 0x7fd72800d320 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.093+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.093+0000 7fd723fff700 10 client.47995.objecter _finish_command 5 = system:0 2024-02-10T08:39:37.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.093+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 4 2024-02-10T08:39:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.093+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 2 5 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 10 client.47995.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 6 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 6 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003630 osd=2 3 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 2 6 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 4 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 10 client.47995.objecter _send_command 6 2024-02-10T08:39:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.107+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd72800e710 con 0x7fd72800d320 2024-02-10T08:39:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd72800e710 con 0x7fd72800d320 2024-02-10T08:39:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 10 client.47995.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 4 2024-02-10T08:39:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 2 6 2024-02-10T08:39:37.110 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 10 client.47995.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 7 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 7 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 3 7 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 4 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 10 client.47995.objecter _send_command 7 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728015a00 con 0x7fd728011eb0 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.109+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd728011eb0 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.110+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728015a00 con 0x7fd728011eb0 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.110+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.110+0000 7fd723fff700 10 client.47995.objecter _finish_command 7 = system:0 2024-02-10T08:39:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.110+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 4 2024-02-10T08:39:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.110+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 3 7 2024-02-10T08:39:37.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 10 client.47995.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 8 2024-02-10T08:39:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 8 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728003340 osd=3 3 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 3 8 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 4 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 10 client.47995.objecter _send_command 8 2024-02-10T08:39:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.125+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728015a00 con 0x7fd728011eb0 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728015a00 con 0x7fd728011eb0 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 10 client.47995.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 4 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 3 8 2024-02-10T08:39:37.128 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:37.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 10 client.47995.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 9 2024-02-10T08:39:37.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 9 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 4 9 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 4 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 10 client.47995.objecter _send_command 9 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd72800cf40 con 0x7fd72800c9b0 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.128+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd72800c9b0 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.129+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd72800cf40 con 0x7fd72800c9b0 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.129+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.129+0000 7fd723fff700 10 client.47995.objecter _finish_command 9 = system:0 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.129+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 4 2024-02-10T08:39:37.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.129+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 4 9 2024-02-10T08:39:37.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:35.978426+0000 mon.a (mon.0) 1923 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:37.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.987885+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.987992+0000 mon.a (mon.0) 1925 : audit [INF] from='client.53252 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:35.988045+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.994252+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:35.994738+0000 mon.a (mon.0) 1927 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:35.978426+0000 mon.a (mon.0) 1923 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.987885+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.107:0/1617103425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi107-112411-49"}]': finished 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.987992+0000 mon.a (mon.0) 1925 : audit [INF] from='client.53252 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:35.988045+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-10T08:39:37.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.994252+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:35.994738+0000 mon.a (mon.0) 1927 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 10 client.47995.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 10 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 10 2024-02-10T08:39:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728015a00 osd=4 3 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 4 10 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 4 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 10 client.47995.objecter _send_command 10 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.144+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd72800cf40 con 0x7fd72800c9b0 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd72800cf40 con 0x7fd72800c9b0 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd723fff700 10 client.47995.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 4 2024-02-10T08:39:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 4 10 2024-02-10T08:39:37.148 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 10 client.47995.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 11 2024-02-10T08:39:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 11 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 5 11 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 4 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 10 client.47995.objecter _send_command 11 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.146+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728016c30 con 0x7fd7280167c0 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd7280167c0 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728016c30 con 0x7fd7280167c0 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 10 client.47995.objecter _finish_command 11 = system:0 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 4 2024-02-10T08:39:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.147+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 5 11 2024-02-10T08:39:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 10 client.47995.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 12 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 12 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd72800cf40 osd=5 3 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 5 12 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 4 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 10 client.47995.objecter _send_command 12 2024-02-10T08:39:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.162+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728016c30 con 0x7fd7280167c0 2024-02-10T08:39:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.008+0000 7f18e6209700 1 -- 172.21.15.107:0/4053125414 >> v1:172.21.15.107:6790/0 conn(0x7f18e00fc820 legacy=0x7f18e00fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.008+0000 7f18e6209700 1 -- 172.21.15.107:0/4053125414 shutdown_connections 2024-02-10T08:39:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.008+0000 7f18e6209700 1 -- 172.21.15.107:0/4053125414 >> 172.21.15.107:0/4053125414 conn(0x7f18e006fe70 msgr2=0x7f18e00f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.009+0000 7f18e6209700 1 -- 172.21.15.107:0/4053125414 shutdown_connections 2024-02-10T08:39:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.009+0000 7f18e6209700 1 -- 172.21.15.107:0/4053125414 wait complete. 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 librados: starting msgr at 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 librados: starting objecter 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 Processor -- start 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 -- start start 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 librados: setting wanted keys 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.010+0000 7f18e6209700 1 librados: calling monclient init 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18e6209700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18e0126160 con 0x7f18e00fc820 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18e6209700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18e0127340 con 0x7f18e00fb090 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18e6209700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18e0128520 con 0x7f18e00fa4d0 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f18e00fc820 0x7f18e006a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46766/0 (socket says 172.21.15.107:46766) 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dffff700 1 -- 172.21.15.107:0/2690071782 learned_addr learned my addr 172.21.15.107:0/2690071782 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter ms_handle_connect 0x7f18e00fc820 2024-02-10T08:39:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter ms_handle_connect 0x7f18e00fb090 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter ms_handle_connect 0x7f18e00fa4d0 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235983834 0 0) 0x7f18e0126160 con 0x7f18e00fc820 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c8003030 con 0x7f18e00fc820 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844777901 0 0) 0x7f18e0127340 con 0x7f18e00fb090 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0126160 con 0x7f18e00fb090 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.011+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644728542 0 0) 0x7f18e0128520 con 0x7f18e00fa4d0 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0127340 con 0x7f18e00fa4d0 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1276164307 0 0) 0x7f18c8003030 con 0x7f18e00fc820 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e0128520 con 0x7f18e00fc820 2024-02-10T08:39:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541649337 0 0) 0x7f18e0127340 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18c8003030 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18d0002950 con 0x7f18e00fc820 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18d8003190 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346994488 0 0) 0x7f18e0126160 con 0x7f18e00fb090 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e0127340 con 0x7f18e00fb090 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18d4001900 con 0x7f18e00fb090 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1411854328 0 0) 0x7f18c8003030 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 >> v1:172.21.15.195:6789/0 conn(0x7f18e00fb090 legacy=0x7f18e0109fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 >> v1:172.21.15.107:6789/0 conn(0x7f18e00fc820 legacy=0x7f18e006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.012+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0129720 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e0127530 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18e6209700 10 client.53264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e01279a0 con 0x7f18e00fa4d0 2024-02-10T08:39:37.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18e6209700 1 librados: init done 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f18d8001cb0 con 0x7f18e00fa4d0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.013+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18d8003850 con 0x7f18e00fa4d0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.014+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc004f20 con 0x7f18e00fa4d0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.014+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f18d8001250 con 0x7f18e00fa4d0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(283..283 src has 50..283) v4 ==== 8461+0+0 (unknown 3434291695 0 0) 0x7f18d8076960 con 0x7f18e00fa4d0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 10 client.53264.objecter ms_dispatch 0x7f18e0001550 osd_map(283..283 src has 50..283) v4 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 3 client.53264.objecter handle_osd_map got epochs [283,283] > 0 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 3 client.53264.objecter handle_osd_map decoding full epoch 283 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 20 client.53264.objecter dump_active .. 0 homeless 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.015+0000 7f18dcff9700 10 client.53264.objecter ms_handle_connect 0x7f18c8063e60 2024-02-10T08:39:37.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.019+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f18d8049c20 con 0x7f18e00fa4d0 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728016c30 con 0x7fd7280167c0 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd723fff700 10 client.47995.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 4 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 5 12 2024-02-10T08:39:37.169 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 10 client.47995.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 13 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 13 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 6 13 2024-02-10T08:39:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 4 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 10 client.47995.objecter _send_command 13 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.164+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728022b10 con 0x7fd72801f000 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd72801f000 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728022b10 con 0x7fd72801f000 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 10 client.47995.objecter _finish_command 13 = system:0 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 4 2024-02-10T08:39:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.165+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 6 13 2024-02-10T08:39:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:37.172 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:37.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 10 client.47995.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 14 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 14 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728016c30 osd=6 3 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 6 14 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 4 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 10 client.47995.objecter _send_command 14 2024-02-10T08:39:37.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.180+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728022b10 con 0x7fd72801f000 2024-02-10T08:39:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728022b10 con 0x7fd72801f000 2024-02-10T08:39:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 10 client.47995.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 4 2024-02-10T08:39:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 6 14 2024-02-10T08:39:37.183 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 10 client.47995.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 15 2024-02-10T08:39:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 15 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 7 15 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 4 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 10 client.47995.objecter _send_command 15 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd728026e80 con 0x7fd728023190 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.182+0000 7fd723fff700 10 client.47995.objecter ms_handle_connect 0x7fd728023190 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.183+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd728026e80 con 0x7fd728023190 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.183+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.183+0000 7fd723fff700 10 client.47995.objecter _finish_command 15 = system:0 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.183+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 4 2024-02-10T08:39:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.183+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 7 15 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 10 client.47995.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign -1 16 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 20 client.47995.objecter _get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 15 client.47995.objecter _session_command_op_remove -1 16 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 20 client.47995.objecter get_session s=0x7fd728022b10 osd=7 3 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 15 client.47995.objecter _session_command_op_assign 7 16 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 4 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 10 client.47995.objecter _send_command 16 2024-02-10T08:39:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.197+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd728028630 con 0x7fd728023190 2024-02-10T08:39:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd723fff700 1 -- 172.21.15.107:0/682888550 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd728028630 con 0x7fd728023190 2024-02-10T08:39:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd723fff700 10 client.47995.objecter ms_dispatch 0x7fd73c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd723fff700 10 client.47995.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd723fff700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 4 2024-02-10T08:39:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd723fff700 15 client.47995.objecter _session_command_op_remove 7 16 2024-02-10T08:39:37.200 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 librados: watch_flush enter 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 librados: watch_flush exit 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.0 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6801/3321596972 conn(0x7fd728004530 legacy=0x7fd728002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd7280051a0 osd=0 1 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.1 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6805/133516126 conn(0x7fd728004150 legacy=0x7fd72800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003fe0 osd=1 1 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.2 2024-02-10T08:39:37.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6809/3575881728 conn(0x7fd72800d320 legacy=0x7fd72800d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003630 osd=2 1 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.3 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6813/2871825691 conn(0x7fd728011eb0 legacy=0x7fd7280142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728003340 osd=3 1 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.4 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.195:6800/2499671285 conn(0x7fd72800c9b0 legacy=0x7fd728015ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728015a00 osd=4 1 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.5 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.195:6804/4178321486 conn(0x7fd7280167c0 legacy=0x7fd72801d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd72800cf40 osd=5 1 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.6 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.195:6808/1417479017 conn(0x7fd72801f000 legacy=0x7fd7280213f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728016c30 osd=6 1 2024-02-10T08:39:37.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter close_session for osd.7 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.195:6812/1934176928 conn(0x7fd728023190 legacy=0x7fd728025600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter put_session s=0x7fd728022b10 osd=7 1 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 20 client.47995.objecter shutdown clearing up homeless session... 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 10 client.47995.objecter successfully canceled tick 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd73b7fe700 1 -- 172.21.15.107:0/682888550 reap_dead start 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd73b7fe700 1 -- 172.21.15.107:0/682888550 reap_dead start 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd73b7fe700 1 -- 172.21.15.107:0/682888550 reap_dead start 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd73b7fe700 1 -- 172.21.15.107:0/682888550 reap_dead start 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd73b7fe700 1 -- 172.21.15.107:0/682888550 reap_dead start 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd724063d50 legacy=0x7fd724066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.199+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> v1:172.21.15.195:6789/0 conn(0x7fd73c104350 legacy=0x7fd73c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.200+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 shutdown_connections 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.200+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 >> 172.21.15.107:0/682888550 conn(0x7fd73c0f2510 msgr2=0x7fd73c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.200+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 shutdown_connections 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.200+0000 7fd74147e700 1 -- 172.21.15.107:0/682888550 wait complete. 2024-02-10T08:39:37.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.200+0000 7fd74147e700 1 librados: shutdown 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.204 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:39:37.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:37.211+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f18cc0051d0 con 0x7f18e00fa4d0 2024-02-10T08:39:37.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:37.337 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:36.636564+0000 osd.7 (osd.7) 111 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:36.639456+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:36.870672+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:36.871519+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:36.996511+0000 mon.a (mon.0) 1928 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:36.996623+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:36.998460+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.107:0/4232189232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.001672+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.002696+0000 mon.a (mon.0) 1930 : audit [INF] from='client.53258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.003378+0000 mon.a (mon.0) 1931 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.005439+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.005913+0000 mon.a (mon.0) 1932 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.007 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.057086+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? v1:172.21.15.107:0/682888550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:38.007 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.212339+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.107:0/2690071782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.007 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:37.212611+0000 mon.a (mon.0) 1933 : audit [INF] from='client.53264 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.007 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:37.402300+0000 mgr.y (mgr.14553) 280 : cluster [DBG] pgmap v401: 250 pgs: 64 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:38.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.009+0000 7f18dcff9700 1 -- 172.21.15.107:0/2690071782 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v284) v1 ==== 113+0+0 (unknown 106316887 0 0) 0x7f18d8048090 con 0x7f18e00fa4d0 2024-02-10T08:39:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 10 librados: watch_flush enter 2024-02-10T08:39:38.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 10 librados: watch_flush exit 2024-02-10T08:39:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 20 client.53264.objecter shutdown clearing up homeless session... 2024-02-10T08:39:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 10 client.53264.objecter successfully canceled tick 2024-02-10T08:39:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 >> v1:172.21.15.107:6800/2634842009 conn(0x7f18c8063e60 legacy=0x7f18c8066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.012+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 >> v1:172.21.15.107:6790/0 conn(0x7f18e00fa4d0 legacy=0x7f18e01098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 shutdown_connections 2024-02-10T08:39:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 >> 172.21.15.107:0/2690071782 conn(0x7f18e006fe70 msgr2=0x7f18e010d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 shutdown_connections 2024-02-10T08:39:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.013+0000 7f18e6209700 1 -- 172.21.15.107:0/2690071782 wait complete. 2024-02-10T08:39:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:38.013+0000 7f18e6209700 1 librados: shutdown 2024-02-10T08:39:38.027 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T08:39:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:36.636564+0000 osd.7 (osd.7) 111 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:39:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:36.639456+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:39:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:36.870672+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:39:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:36.871519+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:36.996511+0000 mon.a (mon.0) 1928 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:36.996623+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:36.998460+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.107:0/4232189232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.001672+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.002696+0000 mon.a (mon.0) 1930 : audit [INF] from='client.53258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.003378+0000 mon.a (mon.0) 1931 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.005439+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.005913+0000 mon.a (mon.0) 1932 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.057086+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? v1:172.21.15.107:0/682888550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.212339+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.107:0/2690071782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:37.212611+0000 mon.a (mon.0) 1933 : audit [INF] from='client.53264 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:37.402300+0000 mgr.y (mgr.14553) 280 : cluster [DBG] pgmap v401: 250 pgs: 64 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:38.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:36.636564+0000 osd.7 (osd.7) 111 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:39:38.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:36.639456+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:36.870672+0000 osd.3 (osd.3) 139 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:36.871519+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:36.996511+0000 mon.a (mon.0) 1928 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:36.996623+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:36.998460+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.107:0/4232189232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.001672+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.107:0/3424795470' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.002696+0000 mon.a (mon.0) 1930 : audit [INF] from='client.53258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.003378+0000 mon.a (mon.0) 1931 : audit [INF] from='client.47971 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.005439+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.005913+0000 mon.a (mon.0) 1932 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.057086+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? v1:172.21.15.107:0/682888550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.212339+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.107:0/2690071782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:37.212611+0000 mon.a (mon.0) 1933 : audit [INF] from='client.53264 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:39:38.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:37.402300+0000 mgr.y (mgr.14553) 280 : cluster [DBG] pgmap v401: 250 pgs: 64 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:38.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:38.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:38.481 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:38.937 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:38.937 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:37.850243+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:37.851366+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:37.860216+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:37.860371+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.004074+0000 mon.a (mon.0) 1934 : audit [INF] from='client.53258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.004222+0000 mon.a (mon.0) 1935 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.004303+0000 mon.a (mon.0) 1936 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.004369+0000 mon.a (mon.0) 1937 : audit [INF] from='client.53264 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.004431+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.007435+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.013104+0000 mon.a (mon.0) 1939 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.037701+0000 osd.1 (osd.1) 135 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.041177+0000 osd.1 (osd.1) 136 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.041900+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.043335+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:38.043600+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.100173+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.100374+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:37.850243+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:37.851366+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:37.860216+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:37.860371+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.004074+0000 mon.a (mon.0) 1934 : audit [INF] from='client.53258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.004222+0000 mon.a (mon.0) 1935 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.004303+0000 mon.a (mon.0) 1936 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.004369+0000 mon.a (mon.0) 1937 : audit [INF] from='client.53264 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.004431+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.007435+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.013104+0000 mon.a (mon.0) 1939 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.037701+0000 osd.1 (osd.1) 135 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.041177+0000 osd.1 (osd.1) 136 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.041900+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.043335+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:38.043600+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.100173+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.100374+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:37.850243+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:37.851366+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:37.860216+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:37.860371+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.004074+0000 mon.a (mon.0) 1934 : audit [INF] from='client.53258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi107-112411-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.004222+0000 mon.a (mon.0) 1935 : audit [INF] from='client.47971 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi107-112403-40"}]': finished 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.004303+0000 mon.a (mon.0) 1936 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.004369+0000 mon.a (mon.0) 1937 : audit [INF] from='client.53264 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.004431+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.007435+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.013104+0000 mon.a (mon.0) 1939 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:39.282 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.037701+0000 osd.1 (osd.1) 135 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.041177+0000 osd.1 (osd.1) 136 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.041900+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.043335+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:38.043600+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.100173+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:39.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.100374+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:39.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T08:39:39.330 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:39.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:39.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:39.464 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:39.582 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:39.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:39:39.589 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:39.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:39.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:39.719 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:39.792 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:39.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:39.851 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:39.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:40.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:39:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.572085+0000 osd.4 (osd.4) 122 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:39:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.573668+0000 osd.4 (osd.4) 123 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.644754+0000 osd.7 (osd.7) 115 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.645523+0000 osd.7 (osd.7) 116 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.851555+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:38.852327+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.007390+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.007499+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.007553+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.012338+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.013168+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.016062+0000 mon.a (mon.0) 1947 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.027253+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.027569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.028339+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.028582+0000 mon.a (mon.0) 1949 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.028984+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:39.029215+0000 mon.a (mon.0) 1950 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.402698+0000 mgr.y (mgr.14553) 281 : cluster [DBG] pgmap v404: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.572085+0000 osd.4 (osd.4) 122 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:39:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.573668+0000 osd.4 (osd.4) 123 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.644754+0000 osd.7 (osd.7) 115 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.645523+0000 osd.7 (osd.7) 116 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.851555+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:38.852327+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.007390+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.007499+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.007553+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.012338+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.013168+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.016062+0000 mon.a (mon.0) 1947 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.027253+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.027569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.572085+0000 osd.4 (osd.4) 122 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.573668+0000 osd.4 (osd.4) 123 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.644754+0000 osd.7 (osd.7) 115 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.645523+0000 osd.7 (osd.7) 116 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.851555+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:38.852327+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.007390+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.007499+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi107-112403-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.007553+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.012338+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.013168+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.016062+0000 mon.a (mon.0) 1947 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:40.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.027253+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.027569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.028339+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.028582+0000 mon.a (mon.0) 1949 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.028984+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:39.029215+0000 mon.a (mon.0) 1950 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.402698+0000 mgr.y (mgr.14553) 281 : cluster [DBG] pgmap v404: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.028339+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.028582+0000 mon.a (mon.0) 1949 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.028984+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:39.029215+0000 mon.a (mon.0) 1950 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:40.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.402698+0000 mgr.y (mgr.14553) 281 : cluster [DBG] pgmap v404: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:40.477 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:40.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:40.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:40.618 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:40.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:39:41.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.820706+0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.820859+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.821494+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:39.822298+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.260621+0000 mon.a (mon.0) 1951 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.260723+0000 mon.a (mon.0) 1952 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.260779+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.261374+0000 mon.b (mon.1) 475 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.261380+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.262682+0000 mon.a (mon.0) 1954 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.263002+0000 mon.a (mon.0) 1955 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.963029+0000 mon.a (mon.0) 1956 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:41.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.968792+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:41.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:41.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.820706+0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.820859+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.821494+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:39.822298+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.260621+0000 mon.a (mon.0) 1951 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.260723+0000 mon.a (mon.0) 1952 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.260779+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.261374+0000 mon.b (mon.1) 475 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.261380+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.262682+0000 mon.a (mon.0) 1954 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.263002+0000 mon.a (mon.0) 1955 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.386 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.963029+0000 mon.a (mon.0) 1956 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.968792+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.820706+0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.820859+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.821494+0000 osd.3 (osd.3) 145 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:39.822298+0000 osd.3 (osd.3) 146 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.260621+0000 mon.a (mon.0) 1951 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.260723+0000 mon.a (mon.0) 1952 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.260779+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.261374+0000 mon.b (mon.1) 475 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.261380+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.262682+0000 mon.a (mon.0) 1954 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]: dispatch 2024-02-10T08:39:41.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.263002+0000 mon.a (mon.0) 1955 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:41.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.963029+0000 mon.a (mon.0) 1956 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:41.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.968792+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi107-112403-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:41.500 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:41.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:41.504 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:41.504 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:41.529 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:41.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.559+0000 7f5150758700 1 -- 172.21.15.107:0/3471757559 >> v1:172.21.15.107:6790/0 conn(0x7f51480ff710 legacy=0x7f5148101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 -- 172.21.15.107:0/3471757559 shutdown_connections 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 -- 172.21.15.107:0/3471757559 >> 172.21.15.107:0/3471757559 conn(0x7f51480f2510 msgr2=0x7f51480f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 -- 172.21.15.107:0/3471757559 shutdown_connections 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 -- 172.21.15.107:0/3471757559 wait complete. 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 librados: starting msgr at 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 librados: starting objecter 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 Processor -- start 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 -- start start 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 librados: setting wanted keys 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.560+0000 7f5150758700 1 librados: calling monclient init 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f5150758700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5148116310 con 0x7f5148103250 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f5150758700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f51481174d0 con 0x7f51480fc910 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f5150758700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f51481186d0 con 0x7f51480ff710 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f514dcf3700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f51480ff710 0x7f51480f9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56396/0 (socket says 172.21.15.107:56396) 2024-02-10T08:39:41.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f514dcf3700 1 -- 172.21.15.107:0/4142100374 learned_addr learned my addr 172.21.15.107:0/4142100374 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f514ecf5700 1 --1- 172.21.15.107:0/4142100374 >> v1:172.21.15.107:6789/0 conn(0x7f5148103250 0x7f51481149d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48646/0 (socket says 172.21.15.107:48646) 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter ms_handle_connect 0x7f51480ff710 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter ms_handle_connect 0x7f5148103250 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter ms_handle_connect 0x7f51480fc910 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774886772 0 0) 0x7f51481186d0 con 0x7f51480ff710 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51340038e0 con 0x7f51480ff710 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578754429 0 0) 0x7f5148116310 con 0x7f5148103250 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51481186d0 con 0x7f5148103250 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497141708 0 0) 0x7f51481174d0 con 0x7f51480fc910 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5148116310 con 0x7f51480fc910 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1868722736 0 0) 0x7f51340038e0 con 0x7f51480ff710 2024-02-10T08:39:41.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51481174d0 con 0x7f51480ff710 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5144002de0 con 0x7f51480ff710 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454530986 0 0) 0x7f5148116310 con 0x7f51480fc910 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51340038e0 con 0x7f51480fc910 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 31305752 0 0) 0x7f51481186d0 con 0x7f5148103250 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5148116310 con 0x7f5148103250 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5138001920 con 0x7f51480fc910 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5140001b40 con 0x7f5148103250 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.561+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3184940286 0 0) 0x7f51481174d0 con 0x7f51480ff710 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 >> v1:172.21.15.195:6789/0 conn(0x7f51480fc910 legacy=0x7f51480f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 >> v1:172.21.15.107:6789/0 conn(0x7f5148103250 legacy=0x7f51481149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51481198d0 con 0x7f51480ff710 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5148116500 con 0x7f51480ff710 2024-02-10T08:39:41.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f5150758700 10 client.53285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5148116970 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f5150758700 1 librados: init done 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f51440023a0 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.562+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5144003750 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c004f20 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5144001f00 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(287..287 src has 50..287) v4 ==== 9222+0+0 (unknown 1174479844 0 0) 0x7f5144079580 con 0x7f51480ff710 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 10 client.53285.objecter ms_dispatch 0x7f5148001550 osd_map(287..287 src has 50..287) v4 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 3 client.53285.objecter handle_osd_map got epochs [287,287] > 0 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 3 client.53285.objecter handle_osd_map decoding full epoch 287 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.563+0000 7f513f7fe700 20 client.53285.objecter dump_active .. 0 homeless 2024-02-10T08:39:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.564+0000 7f513f7fe700 10 client.53285.objecter ms_handle_connect 0x7f5134063d40 2024-02-10T08:39:41.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.568+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f514404c570 con 0x7f51480ff710 2024-02-10T08:39:41.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:41.632 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:41.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.758+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f512c0051d0 con 0x7f51480ff710 2024-02-10T08:39:41.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:41.792 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.976+0000 7f513f7fe700 1 -- 172.21.15.107:0/4142100374 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v288) v1 ==== 113+0+0 (unknown 3951031396 0 0) 0x7f514404a9e0 con 0x7f51480ff710 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 10 librados: watch_flush enter 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 10 librados: watch_flush exit 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 20 client.53285.objecter shutdown clearing up homeless session... 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 10 client.53285.objecter successfully canceled tick 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5134063d40 legacy=0x7f5134066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 >> v1:172.21.15.107:6790/0 conn(0x7f51480ff710 legacy=0x7f51480f9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 shutdown_connections 2024-02-10T08:39:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.979+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 >> 172.21.15.107:0/4142100374 conn(0x7f51480f2510 msgr2=0x7f51481066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.980+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 shutdown_connections 2024-02-10T08:39:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.980+0000 7f5150758700 1 -- 172.21.15.107:0/4142100374 wait complete. 2024-02-10T08:39:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:41.980+0000 7f5150758700 1 librados: shutdown 2024-02-10T08:39:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T08:39:42.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:39:42.219 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:42.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.480180+0000 osd.4 (osd.4) 126 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.483150+0000 osd.4 (osd.4) 127 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.830783+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:40.831887+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:40.968870+0000 mon.a (mon.0) 1958 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]': finished 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.002745+0000 mon.b (mon.1) 476 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.214874+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.219750+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.324926+0000 mgr.y (mgr.14553) 282 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.403808+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:39:42.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.403834+0000 mgr.y (mgr.14553) 283 : cluster [DBG] pgmap v407: 226 pgs: 6 creating+peering, 2 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.404720+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.481813+0000 osd.4 (osd.4) 128 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.482669+0000 osd.4 (osd.4) 129 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.689305+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.691081+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.759126+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.107:0/4142100374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.759413+0000 mon.a (mon.0) 1961 : audit [INF] from='client.53285 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.480180+0000 osd.4 (osd.4) 126 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.483150+0000 osd.4 (osd.4) 127 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:39:42.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.830783+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:40.831887+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:40.968870+0000 mon.a (mon.0) 1958 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]': finished 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.002745+0000 mon.b (mon.1) 476 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.214874+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.219750+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.324926+0000 mgr.y (mgr.14553) 282 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.403808+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.403834+0000 mgr.y (mgr.14553) 283 : cluster [DBG] pgmap v407: 226 pgs: 6 creating+peering, 2 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.404720+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.481813+0000 osd.4 (osd.4) 128 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.482669+0000 osd.4 (osd.4) 129 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.689305+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.691081+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:39:42.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.759126+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.107:0/4142100374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.759413+0000 mon.a (mon.0) 1961 : audit [INF] from='client.53285 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.794373+0000 osd.3 (osd.3) 149 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.795220+0000 osd.3 (osd.3) 150 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.971769+0000 mon.a (mon.0) 1962 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.971964+0000 mon.a (mon.0) 1963 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.972029+0000 mon.a (mon.0) 1964 : audit [INF] from='client.53285 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.972085+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.973977+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:41.979960+0000 mon.a (mon.0) 1966 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.794373+0000 osd.3 (osd.3) 149 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.795220+0000 osd.3 (osd.3) 150 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.971769+0000 mon.a (mon.0) 1962 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.971964+0000 mon.a (mon.0) 1963 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.972029+0000 mon.a (mon.0) 1964 : audit [INF] from='client.53285 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.972085+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.973977+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:41.979960+0000 mon.a (mon.0) 1966 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:42.475 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.480180+0000 osd.4 (osd.4) 126 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:39:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.483150+0000 osd.4 (osd.4) 127 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.830783+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:40.831887+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:40.968870+0000 mon.a (mon.0) 1958 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-35", "mode": "writeback"}]': finished 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.002745+0000 mon.b (mon.1) 476 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.214874+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.219750+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.324926+0000 mgr.y (mgr.14553) 282 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.403808+0000 osd.2 (osd.2) 99 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.403834+0000 mgr.y (mgr.14553) 283 : cluster [DBG] pgmap v407: 226 pgs: 6 creating+peering, 2 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.404720+0000 osd.2 (osd.2) 100 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.481813+0000 osd.4 (osd.4) 128 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.482669+0000 osd.4 (osd.4) 129 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.689305+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:39:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.691081+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.759126+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.107:0/4142100374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.759413+0000 mon.a (mon.0) 1961 : audit [INF] from='client.53285 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.794373+0000 osd.3 (osd.3) 149 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.795220+0000 osd.3 (osd.3) 150 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.971769+0000 mon.a (mon.0) 1962 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi107-112411-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.971964+0000 mon.a (mon.0) 1963 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.972029+0000 mon.a (mon.0) 1964 : audit [INF] from='client.53285 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.972085+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.973977+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:41.979960+0000 mon.a (mon.0) 1966 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]: dispatch 2024-02-10T08:39:42.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T08:39:42.802 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:42.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:42.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:42.934 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:42.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:43.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:43.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.552449+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:39:43.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.555335+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.575005+0000 osd.5 (osd.5) 133 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:41.578524+0000 osd.5 (osd.5) 134 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.509120+0000 osd.4 (osd.4) 130 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.510600+0000 osd.4 (osd.4) 131 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.967084+0000 mon.a (mon.0) 1967 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:42.971591+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.971710+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:42.975576+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.552449+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.555335+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:39:43.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.575005+0000 osd.5 (osd.5) 133 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:41.578524+0000 osd.5 (osd.5) 134 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.509120+0000 osd.4 (osd.4) 130 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.510600+0000 osd.4 (osd.4) 131 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.967084+0000 mon.a (mon.0) 1967 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:42.971591+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.971710+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-10T08:39:43.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:42.975576+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.552449+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.555335+0000 osd.6 (osd.6) 112 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.575005+0000 osd.5 (osd.5) 133 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:41.578524+0000 osd.5 (osd.5) 134 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.509120+0000 osd.4 (osd.4) 130 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.510600+0000 osd.4 (osd.4) 131 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.967084+0000 mon.a (mon.0) 1967 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:42.971591+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-35"}]': finished 2024-02-10T08:39:43.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.971710+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-10T08:39:43.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:42.975576+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:43.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:43.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:43.664 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:43.812 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:43.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:43.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:43.945 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:43.998 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:44.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:44.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:44.130 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:44.238 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.590375+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:39:44.238 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:42.593936+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:39:44.238 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.404302+0000 mgr.y (mgr.14553) 284 : cluster [DBG] pgmap v410: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:44.238 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.631247+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:39:44.238 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.632043+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.709945+0000 osd.3 (osd.3) 151 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.711715+0000 osd.3 (osd.3) 152 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:43.972717+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.972808+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:43.974973+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:44.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:43.975753+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:43.977321+0000 mon.a (mon.0) 1974 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.156163+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? v1:172.21.15.107:0/1462339423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:44.259 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:44.259 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.590375+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:42.593936+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.404302+0000 mgr.y (mgr.14553) 284 : cluster [DBG] pgmap v410: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.631247+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.632043+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.709945+0000 osd.3 (osd.3) 151 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:39:44.280 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.711715+0000 osd.3 (osd.3) 152 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:43.972717+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.972808+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:43.974973+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:43.975753+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:43.977321+0000 mon.a (mon.0) 1974 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.281 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.156163+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? v1:172.21.15.107:0/1462339423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:44.381 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:44.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:44.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 -- 172.21.15.107:0/3849634462 >> v1:172.21.15.107:6790/0 conn(0x7fd5440f8cd0 legacy=0x7fd5440f66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 -- 172.21.15.107:0/3849634462 shutdown_connections 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 -- 172.21.15.107:0/3849634462 >> 172.21.15.107:0/3849634462 conn(0x7fd54406fe70 msgr2=0x7fd5440f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 -- 172.21.15.107:0/3849634462 shutdown_connections 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 -- 172.21.15.107:0/3849634462 wait complete. 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 librados: starting msgr at 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 librados: starting objecter 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.152+0000 7fd54bebd700 1 Processor -- start 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 -- start start 2024-02-10T08:39:44.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 librados: setting wanted keys 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 librados: calling monclient init 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd544119590 con 0x7fd544106740 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd54411a770 con 0x7fd5440f8cd0 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd54bebd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd54411b970 con 0x7fd5440f8830 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd549c59700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd5440f8830 0x7fd544102aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56416/0 (socket says 172.21.15.107:56416) 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd549c59700 1 -- 172.21.15.107:0/1462339423 learned_addr learned my addr 172.21.15.107:0/1462339423 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter ms_handle_connect 0x7fd5440f8830 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter ms_handle_connect 0x7fd544106740 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter ms_handle_connect 0x7fd5440f8cd0 2024-02-10T08:39:44.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10862440 0 0) 0x7fd54411b970 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd530002220 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025649664 0 0) 0x7fd54411a770 con 0x7fd5440f8cd0 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54411b970 con 0x7fd5440f8cd0 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843409418 0 0) 0x7fd530002220 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd54411a770 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd540001940 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2706701333 0 0) 0x7fd54411b970 con 0x7fd5440f8cd0 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd530002220 con 0x7fd5440f8cd0 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd534000fe0 con 0x7fd5440f8cd0 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.153+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273999829 0 0) 0x7fd54411a770 con 0x7fd5440f8830 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.195:6789/0 conn(0x7fd5440f8cd0 legacy=0x7fd5441031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6789/0 conn(0x7fd544106740 legacy=0x7fd544117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54411cb70 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd544119780 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd54bebd700 10 client.53291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd544119bf0 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd54bebd700 1 librados: init done 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd540002460 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd540005ae0 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.154+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd528004f00 con 0x7fd5440f8830 2024-02-10T08:39:44.513 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.155+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd540018990 con 0x7fd5440f8830 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.155+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(290..290 src has 50..290) v4 ==== 8102+0+0 (unknown 4042765641 0 0) 0x7fd540078e10 con 0x7fd5440f8830 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.155+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 osd_map(290..290 src has 50..290) v4 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.155+0000 7fd53affd700 3 client.53291.objecter handle_osd_map got epochs [290,290] > 0 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.155+0000 7fd53affd700 3 client.53291.objecter handle_osd_map decoding full epoch 290 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd53affd700 20 client.53291.objecter dump_active .. 0 homeless 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v290) v1 ==== 54+0+16 (unknown 4252067127 0 2420000192) 0x7fd54007b200 con 0x7fd5440f8830 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd530063b70 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 10 client.53291.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 1 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 1 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 0 1 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 4 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 10 client.53291.objecter _send_command 1 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd528004950 con 0x7fd528004530 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.156+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd528004530 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.157+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd528004950 con 0x7fd528004530 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.157+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.157+0000 7fd53affd700 10 client.53291.objecter _finish_command 1 = system:0 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.157+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 4 2024-02-10T08:39:44.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.157+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 0 1 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 10 client.53291.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 2 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 2 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd5280051a0 osd=0 3 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 0 2 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 4 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 10 client.53291.objecter _send_command 2 2024-02-10T08:39:44.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.172+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd528004d30 con 0x7fd528004530 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd528004d30 con 0x7fd528004530 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 10 client.53291.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 4 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 0 2 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 10 client.53291.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 3 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 3 2024-02-10T08:39:44.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 1 3 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 4 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 10 client.53291.objecter _send_command 3 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd52800c000 con 0x7fd528004150 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.174+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd528004150 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.175+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd52800c000 con 0x7fd528004150 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.175+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:44.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.175+0000 7fd53affd700 10 client.53291.objecter _finish_command 3 = system:0 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.175+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.175+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 1 3 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 10 client.53291.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528003fe0 osd=1 3 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 1 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 10 client.53291.objecter _send_command 4 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.190+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd5280018b0 con 0x7fd528004150 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd5280018b0 con 0x7fd528004150 2024-02-10T08:39:44.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 10 client.53291.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 4 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 1 4 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 10 client.53291.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 5 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 5 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 2 5 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 4 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 10 client.53291.objecter _send_command 5 2024-02-10T08:39:44.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd52800de60 con 0x7fd52800d2f0 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.191+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd52800d2f0 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.192+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd52800de60 con 0x7fd52800d2f0 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.192+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.192+0000 7fd53affd700 10 client.53291.objecter _finish_command 5 = system:0 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.192+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 4 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.192+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 2 5 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 10 client.53291.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 6 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 6 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528003630 osd=2 3 2024-02-10T08:39:44.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 2 6 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 4 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 10 client.53291.objecter _send_command 6 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.206+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd5280030a0 con 0x7fd52800d2f0 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd5280030a0 con 0x7fd52800d2f0 2024-02-10T08:39:44.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 10 client.53291.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 4 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 2 6 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 10 client.53291.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 7 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 7 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 3 7 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 4 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 10 client.53291.objecter _send_command 7 2024-02-10T08:39:44.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd528015b20 con 0x7fd528011e80 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.208+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd528011e80 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.209+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd528015b20 con 0x7fd528011e80 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.209+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.209+0000 7fd53affd700 10 client.53291.objecter _finish_command 7 = system:0 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.209+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 4 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.209+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 3 7 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 10 client.53291.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 8 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 8 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd5280034b0 osd=3 3 2024-02-10T08:39:44.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 3 8 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 4 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 10 client.53291.objecter _send_command 8 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.223+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd528015b20 con 0x7fd528011e80 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd528015b20 con 0x7fd528011e80 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 10 client.53291.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 4 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 3 8 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 10 client.53291.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 9 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 9 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 4 9 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 4 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 10 client.53291.objecter _send_command 9 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd52800e4e0 con 0x7fd52800ce90 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.225+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd52800ce90 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.226+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd52800e4e0 con 0x7fd52800ce90 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.226+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.226+0000 7fd53affd700 10 client.53291.objecter _finish_command 9 = system:0 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.226+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 4 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.226+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 4 9 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 10 client.53291.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 10 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 10 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528015b20 osd=4 3 2024-02-10T08:39:44.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 4 10 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 4 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 10 client.53291.objecter _send_command 10 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.242+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd52800e4e0 con 0x7fd52800ce90 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.244+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd52800e4e0 con 0x7fd52800ce90 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.244+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.244+0000 7fd53affd700 10 client.53291.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.244+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 4 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.244+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 4 10 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 10 client.53291.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 11 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 11 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 5 11 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 4 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 10 client.53291.objecter _send_command 11 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd528017f80 con 0x7fd528017a60 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.245+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd528017a60 2024-02-10T08:39:44.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.246+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd528017f80 con 0x7fd528017a60 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.246+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.246+0000 7fd53affd700 10 client.53291.objecter _finish_command 11 = system:0 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.246+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 4 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.246+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 5 11 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 10 client.53291.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 12 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 12 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd52800e4e0 osd=5 3 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 5 12 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 4 2024-02-10T08:39:44.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 10 client.53291.objecter _send_command 12 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.260+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd528017f80 con 0x7fd528017a60 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd528017f80 con 0x7fd528017a60 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 10 client.53291.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 4 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 5 12 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 10 client.53291.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 13 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 13 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 6 13 2024-02-10T08:39:44.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 4 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 10 client.53291.objecter _send_command 13 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd528022c30 con 0x7fd52801f120 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.262+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd52801f120 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.263+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd528022c30 con 0x7fd52801f120 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.263+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:44.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.263+0000 7fd53affd700 10 client.53291.objecter _finish_command 13 = system:0 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.263+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 4 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.263+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 6 13 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 10 client.53291.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 14 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 14 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528017f80 osd=6 3 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 6 14 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 4 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 10 client.53291.objecter _send_command 14 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.277+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd528022c30 con 0x7fd52801f120 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd528022c30 con 0x7fd52801f120 2024-02-10T08:39:44.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd53affd700 10 client.53291.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 4 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 6 14 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 10 client.53291.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 15 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 15 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 7 15 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 4 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 10 client.53291.objecter _send_command 15 2024-02-10T08:39:44.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.279+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd528027010 con 0x7fd528023320 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 10 client.53291.objecter ms_handle_connect 0x7fd528023320 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd528027010 con 0x7fd528023320 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 10 client.53291.objecter _finish_command 15 = system:0 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 4 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.280+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 7 15 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 10 client.53291.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign -1 16 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 20 client.53291.objecter _get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_remove -1 16 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 20 client.53291.objecter get_session s=0x7fd528022c30 osd=7 3 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 15 client.53291.objecter _session_command_op_assign 7 16 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 4 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 10 client.53291.objecter _send_command 16 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.294+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd528026fd0 con 0x7fd528023320 2024-02-10T08:39:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd53affd700 1 -- 172.21.15.107:0/1462339423 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd528026fd0 con 0x7fd528023320 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd53affd700 10 client.53291.objecter ms_dispatch 0x7fd544001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd53affd700 10 client.53291.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd53affd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 4 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd53affd700 15 client.53291.objecter _session_command_op_remove 7 16 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 librados: watch_flush enter 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 librados: watch_flush exit 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.0 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6801/3321596972 conn(0x7fd528004530 legacy=0x7fd528002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280051a0 osd=0 1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6805/133516126 conn(0x7fd528004150 legacy=0x7fd52800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003fe0 osd=1 1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.2 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6809/3575881728 conn(0x7fd52800d2f0 legacy=0x7fd52800d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528003630 osd=2 1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.3 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6813/2871825691 conn(0x7fd528011e80 legacy=0x7fd528014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd5280034b0 osd=3 1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.4 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.195:6800/2499671285 conn(0x7fd52800ce90 legacy=0x7fd528015cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528015b20 osd=4 1 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.5 2024-02-10T08:39:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.195:6804/4178321486 conn(0x7fd528017a60 legacy=0x7fd52801d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd52800e4e0 osd=5 1 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.6 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.195:6808/1417479017 conn(0x7fd52801f120 legacy=0x7fd528021510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528017f80 osd=6 1 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter close_session for osd.7 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.195:6812/1934176928 conn(0x7fd528023320 legacy=0x7fd528025750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54a45a700 1 -- 172.21.15.107:0/1462339423 reap_dead start 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter put_session s=0x7fd528022c30 osd=7 1 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 20 client.53291.objecter shutdown clearing up homeless session... 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 10 client.53291.objecter successfully canceled tick 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54a45a700 1 -- 172.21.15.107:0/1462339423 reap_dead start 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54a45a700 1 -- 172.21.15.107:0/1462339423 reap_dead start 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54a45a700 1 -- 172.21.15.107:0/1462339423 reap_dead start 2024-02-10T08:39:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54a45a700 1 -- 172.21.15.107:0/1462339423 reap_dead start 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd530063b70 legacy=0x7fd530066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> v1:172.21.15.107:6790/0 conn(0x7fd5440f8830 legacy=0x7fd544102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 shutdown_connections 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 >> 172.21.15.107:0/1462339423 conn(0x7fd54406fe70 msgr2=0x7fd5440f4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 shutdown_connections 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 -- 172.21.15.107:0/1462339423 wait complete. 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:44.296+0000 7fd54bebd700 1 librados: shutdown 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.590375+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:42.593936+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.404302+0000 mgr.y (mgr.14553) 284 : cluster [DBG] pgmap v410: 226 pgs: 8 unknown, 218 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.631247+0000 osd.7 (osd.7) 119 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.632043+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.709945+0000 osd.3 (osd.3) 151 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.711715+0000 osd.3 (osd.3) 152 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:43.972717+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.972808+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-10T08:39:44.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:43.974973+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]: dispatch 2024-02-10T08:39:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:43.975753+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:43.977321+0000 mon.a (mon.0) 1974 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.156163+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? v1:172.21.15.107:0/1462339423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:44.542 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:44.692 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:44.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:39:44.984 INFO:tasks.workunit.client.0.smithi107.stdout:te 2024-02-10T08:39:44.984 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7279 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6709 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6991 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7006 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7012 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7004 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7018 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7001 ms) 2024-02-10T08:39:44.985 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7203 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6786 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7003 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7253 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6979 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6795 ms) 2024-02-10T08:39:44.986 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6975 ms) 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112034 ms total) 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [----------] Global test environment tear-down 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (194800 ms total) 2024-02-10T08:39:44.987 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio: [ PASSED ] 41 tests. 2024-02-10T08:39:45.143 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:45.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:45.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:45.292 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:45.389 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:45.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:45.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.508646+0000 osd.6 (osd.6) 113 : cluster [DBG] 60.f scrub starts 2024-02-10T08:39:45.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.510779+0000 osd.6 (osd.6) 114 : cluster [DBG] 60.f scrub ok 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.633052+0000 osd.5 (osd.5) 137 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.636530+0000 osd.5 (osd.5) 138 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.951068+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:43.954534+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.584744+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.585667+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.975648+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.975725+0000 mon.a (mon.0) 1976 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.975794+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.977958+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.980469+0000 mon.a (mon.0) 1978 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.983757+0000 mon.b (mon.1) 478 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:44.984097+0000 mon.a (mon.0) 1979 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.508646+0000 osd.6 (osd.6) 113 : cluster [DBG] 60.f scrub starts 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.510779+0000 osd.6 (osd.6) 114 : cluster [DBG] 60.f scrub ok 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.633052+0000 osd.5 (osd.5) 137 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.636530+0000 osd.5 (osd.5) 138 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.951068+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:43.954534+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.584744+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.585667+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.975648+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.975725+0000 mon.a (mon.0) 1976 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:45.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.975794+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-10T08:39:45.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.977958+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.980469+0000 mon.a (mon.0) 1978 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.983757+0000 mon.b (mon.1) 478 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:44.984097+0000 mon.a (mon.0) 1979 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.508646+0000 osd.6 (osd.6) 113 : cluster [DBG] 60.f scrub starts 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.510779+0000 osd.6 (osd.6) 114 : cluster [DBG] 60.f scrub ok 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.633052+0000 osd.5 (osd.5) 137 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.636530+0000 osd.5 (osd.5) 138 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.951068+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:43.954534+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.584744+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.585667+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.975648+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? v1:172.21.15.107:0/454441613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi107-112403-41"}]': finished 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.975725+0000 mon.a (mon.0) 1976 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:45.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.975794+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-10T08:39:45.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.977958+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.107:0/1220740578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.980469+0000 mon.a (mon.0) 1978 : audit [INF] from='client.53279 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]: dispatch 2024-02-10T08:39:45.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.983757+0000 mon.b (mon.1) 478 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:44.984097+0000 mon.a (mon.0) 1979 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:45.524 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:45.577 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:45.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:45.708 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:45.894 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:45] "GET /metrics HTTP/1.1" 200 186104 "" "Prometheus/2.33.4" 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.468255+0000 osd.6 (osd.6) 115 : cluster [DBG] 60.d scrub starts 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.469918+0000 osd.6 (osd.6) 116 : cluster [DBG] 60.d scrub ok 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.682431+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.f scrub starts 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:44.685896+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.f scrub ok 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:45.404931+0000 mgr.y (mgr.14553) 285 : cluster [DBG] pgmap v413: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:45.984932+0000 mon.a (mon.0) 1980 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:45.985060+0000 mon.a (mon.0) 1981 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:45.985128+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-10T08:39:46.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:45.990642+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:45.991683+0000 mon.a (mon.0) 1983 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.019089+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.019553+0000 mon.a (mon.0) 1984 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.019939+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.020283+0000 mon.a (mon.0) 1985 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.020678+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.020923+0000 mon.a (mon.0) 1986 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.468255+0000 osd.6 (osd.6) 115 : cluster [DBG] 60.d scrub starts 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.469918+0000 osd.6 (osd.6) 116 : cluster [DBG] 60.d scrub ok 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.682431+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.f scrub starts 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:44.685896+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.f scrub ok 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:45.404931+0000 mgr.y (mgr.14553) 285 : cluster [DBG] pgmap v413: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:45.984932+0000 mon.a (mon.0) 1980 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:45.985060+0000 mon.a (mon.0) 1981 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:45.985128+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:45.990642+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:45.991683+0000 mon.a (mon.0) 1983 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.019089+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.019553+0000 mon.a (mon.0) 1984 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.019939+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.020283+0000 mon.a (mon.0) 1985 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.020678+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.020923+0000 mon.a (mon.0) 1986 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.468255+0000 osd.6 (osd.6) 115 : cluster [DBG] 60.d scrub starts 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.469918+0000 osd.6 (osd.6) 116 : cluster [DBG] 60.d scrub ok 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.682431+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.f scrub starts 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:44.685896+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.f scrub ok 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:45.404931+0000 mgr.y (mgr.14553) 285 : cluster [DBG] pgmap v413: 218 pgs: 32 unknown, 186 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:46.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:45.984932+0000 mon.a (mon.0) 1980 : audit [INF] from='client.53279 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi107-112411-51"}]': finished 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:45.985060+0000 mon.a (mon.0) 1981 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:45.985128+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:45.990642+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:45.991683+0000 mon.a (mon.0) 1983 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.019089+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.019553+0000 mon.a (mon.0) 1984 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.019939+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.020283+0000 mon.a (mon.0) 1985 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.020678+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.020923+0000 mon.a (mon.0) 1986 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:46.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:46.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:46.694 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:46.794 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:46.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:46.799 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:39:46.799 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:39:46.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:39:46.799 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:545 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94547991872512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e77)", 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 305.28836067600002, 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19536709899999999, 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 545 2024-02-10T08:39:46.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506554+0000", 2024-02-10T08:39:46.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8359999999999992e-06 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506562+0000", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.31e-06 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506567+0000", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5569999999999997e-06 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506572+0000", 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4697999999999998e-05 2024-02-10T08:39:46.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506617+0000", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2550000000000003e-06 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506623+0000", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19260776499999999 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.699231+0000", 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011743699999999999 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.699348+0000", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011991020000000001 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700547+0000", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116805 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700664+0000", 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029738300000000002 2024-02-10T08:39:46.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700961+0000", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0784000000000004e-05 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.701042+0000", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00087991499999999999 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.701922+0000", 2024-02-10T08:39:46.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19536709899999999 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:735 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 200.26807026700001, 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999, 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:39:46.806 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 735 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526842+0000", 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2619999999999999e-05 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526855+0000", 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5109e-05 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526880+0000", 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1497e-05 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526891+0000", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017202499999999999 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527063+0000", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3579999999999997e-06 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527071+0000", 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62625176299999996 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153322+0000", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0579e-05 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153353+0000", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:736 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:39:46.810 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 200.268005268, 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998, 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 736 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526909+0000", 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.87e-06 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526914+0000", 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7660000000000004e-06 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526923+0000", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7569999999999998e-06 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526930+0000", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016634899999999999 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527096+0000", 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3100000000000001e-06 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527104+0000", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62646392500000003 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153568+0000", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1849999999999998e-06 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153575+0000", 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.814 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:737 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 3] snapc 0=[] ondisk+write+known_if_redirected e99)", 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 198.643186034, 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172, 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 737 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.815 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151728+0000", 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.536e-06 2024-02-10T08:39:46.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151734+0000", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.661e-06 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151741+0000", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0990000000000004e-06 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151747+0000", 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069385099999999999 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152441+0000", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7069999999999998e-06 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152450+0000", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0056702749999999 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158120+0000", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000171347 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158292+0000", 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000874083 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.159166+0000", 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35222.0:3 50.0s0 50:1147e795:test-rados-api-smithi107-112459-12::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:45.967699+0000", 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 180.827216703, 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015907063999999999, 2024-02-10T08:39:46.819 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35222", 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4111016630", 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967699+0000", 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967699+0000", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967697+0000", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0739999999999993e-06 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967705+0000", 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7109999999999995e-06 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967713+0000", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9729999999999997e-06 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967718+0000", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5847000000000002e-05 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967774+0000", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134077 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.967908+0000", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040994600000000001 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.968318+0000", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020356500000000001 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.968521+0000", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015050022999999999 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.983571+0000", 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4536000000000003e-05 2024-02-10T08:39:46.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:45.983606+0000", 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015907063999999999 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35219.0:6 51.6s0 51:6ab6d0da:test-rados-api-smithi107-112451-13::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 179.899221646, 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344, 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.824 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35219", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 6 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:46.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895691+0000", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0546e-05 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895702+0000", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0954999999999999e-05 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895713+0000", 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0730000000000005e-06 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895721+0000", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012145484999999999 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907866+0000", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101523 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907968+0000", 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015504442 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.923472+0000", 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35222.0:9 50.4s0 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 178.849965148, 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020488565, 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35222", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4111016630", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:39:46.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944948+0000", 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0909999999999993e-06 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944956+0000", 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4697e-05 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944971+0000", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2599999999999999e-06 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944978+0000", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4249000000000001e-05 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.945042+0000", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011398 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.945156+0000", 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020282664999999998 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.965439+0000", 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020488565 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25276.0:8 56.1 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 175.82816387599999, 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999, 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25276", 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 8 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966750+0000", 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4189999999999999e-06 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966756+0000", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4120000000000004e-06 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966764+0000", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5940000000000004e-06 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966768+0000", 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9002999999999999e-05 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966807+0000", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016647400000000001 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966974+0000", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012392504 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979366+0000", 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6769999999999999e-05 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979443+0000", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110282 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979553+0000", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036667200000000002 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979920+0000", 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013358821 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.993279+0000", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025924506 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019203+0000", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9867999999999998e-05 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019253+0000", 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0630999999999998e-05 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019294+0000", 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25276.0:20 56.2 56:573ae330:test-rados-api-smithi107-112562-10::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.069840+0000", 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 175.72507542100001, 2024-02-10T08:39:46.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011724361000000001, 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25276", 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 20 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069840+0000", 2024-02-10T08:39:46.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069840+0000", 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069839+0000", 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2809999999999997e-06 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069845+0000", 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0308e-05 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069855+0000", 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.639e-06 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069860+0000", 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8822000000000002e-05 2024-02-10T08:39:46.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.069899+0000", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013141100000000001 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.070030+0000", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072923300000000003 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.070759+0000", 2024-02-10T08:39:46.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8869000000000002e-05 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.070818+0000", 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9017999999999994e-05 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.070907+0000", 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000321837 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.071229+0000", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000160426 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.071390+0000", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010123738 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.081513+0000", 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9646e-05 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.081543+0000", 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1585000000000001e-05 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.081564+0000", 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011724361000000001 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.846 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.43309.0:2 90.13 90:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 151.73948940099999, 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514, 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.43309", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4276406080", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055424+0000", 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2330000000000001e-05 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055437+0000", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5120000000000002e-06 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055444+0000", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3789999999999997e-06 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055450+0000", 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2941000000000002e-05 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055482+0000", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5740000000000003e-06 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055488+0000", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016882637999999998 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072371+0000", 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010682 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072477+0000", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115103 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072593+0000", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031995 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072913+0000", 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013010588 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.085923+0000", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012255667 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098179+0000", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1485e-05 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098220+0000", 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6390999999999998e-05 2024-02-10T08:39:46.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098257+0000", 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:2 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e145)", 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:18.964357+0000", 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 147.83055863499999, 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011720466000000001, 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.853 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964357+0000", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964357+0000", 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:46.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964354+0000", 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7503000000000002e-05 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964371+0000", 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1785e-05 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964383+0000", 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5820000000000008e-06 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.964393+0000", 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011575439999999999 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.965550+0000", 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128615 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.965679+0000", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039922800000000001 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.966078+0000", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033193250000000001 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.969397+0000", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066035720000000003 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.976001+0000", 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6334000000000005e-05 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:18.976077+0000", 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011720466000000001 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:4 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e146)", 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 146.84835720000001, 2024-02-10T08:39:46.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001, 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 4 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946556+0000", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7836e-05 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946574+0000", 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7283999999999999e-05 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946591+0000", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.516e-06 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946601+0000", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8421000000000001e-05 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946669+0000", 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5779999999999999e-06 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946678+0000", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098724250000000006 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956550+0000", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012298100000000001 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956673+0000", 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00064271 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.957316+0000", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011186949999999999 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.968503+0000", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01058679 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979089+0000", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3553000000000005e-05 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979173+0000", 2024-02-10T08:39:46.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52592.0:2 96.e 96:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e147)", 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:20.940829+0000", 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.85408665400001, 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010889612, 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52592", 2024-02-10T08:39:46.864 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/875340312", 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940829+0000", 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940829+0000", 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940827+0000", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2299999999999996e-06 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940833+0000", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7570000000000004e-06 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940841+0000", 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3210000000000004e-06 2024-02-10T08:39:46.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940846+0000", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000162168 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.941009+0000", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6309999999999997e-06 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.941015+0000", 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00584725 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.946863+0000", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9649999999999996e-05 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.946962+0000", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013539199999999999 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.947098+0000", 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040582769999999999 2024-02-10T08:39:46.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.951156+0000", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035668700000000001 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.951513+0000", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7559000000000006e-05 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.951610+0000", 2024-02-10T08:39:46.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7272000000000002e-05 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.951677+0000", 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1128999999999999e-05 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.951719+0000", 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010889612 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:9 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 144.871080283, 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001, 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923833+0000", 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7350000000000002e-06 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923842+0000", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6348999999999999e-05 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923858+0000", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8599999999999999e-06 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923867+0000", 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7614000000000001e-05 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923925+0000", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0160000000000008e-06 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923933+0000", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088728850000000005 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932805+0000", 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9782000000000001e-05 2024-02-10T08:39:46.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932905+0000", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040048699999999999 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.933306+0000", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075809029999999999 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.940887+0000", 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011264712 2024-02-10T08:39:46.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952151+0000", 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8347000000000001e-05 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952220+0000", 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27168.0:2 103.6 103:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 135.85135849900001, 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017574042000000002, 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27168", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/709535540", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943554+0000", 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1824000000000001e-05 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943566+0000", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2244e-05 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943578+0000", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2460000000000003e-06 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943587+0000", 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7071999999999998e-05 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943654+0000", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0369e-05 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943664+0000", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084003250000000002 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952064+0000", 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115461 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952180+0000", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014554699999999999 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952325+0000", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003713872 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.956039+0000", 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046460440000000002 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.960685+0000", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035895300000000002 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961044+0000", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7293000000000003e-05 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961092+0000", 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9542000000000002e-05 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961131+0000", 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017574042000000002 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(osd.5.0:1426 60.1f e190/135 ondisk, result = 0)", 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:03.986620+0000", 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 102.808295148, 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012421293, 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986620+0000", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986620+0000", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986620+0000", 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7049999999999998e-06 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986624+0000", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9012999999999999e-05 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986643+0000", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0970000000000002e-06 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986649+0000", 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012370335999999999 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999020+0000", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0246e-05 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999030+0000", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1765e-05 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999042+0000", 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012421293 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(osd.5.0:1427 60.1f e190/135 ondisk, result = 0)", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:03.988246+0000", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 102.806669927, 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011030854, 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988246+0000", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988246+0000", 2024-02-10T08:39:46.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988244+0000", 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8669999999999999e-06 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988250+0000", 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3849999999999996e-06 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988258+0000", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6219999999999997e-06 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.988263+0000", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010996578 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999260+0000", 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5669999999999997e-06 2024-02-10T08:39:46.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999267+0000", 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0669999999999998e-06 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.999276+0000", 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011030854 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.890 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47650.0:2 135.1f 135:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 97.762256398000005, 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394, 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47650", 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1937376491", 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.891 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032657+0000", 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2010000000000002e-06 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032664+0000", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1249999999999999e-06 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032673+0000", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4060000000000004e-06 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032679+0000", 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0767999999999999e-05 2024-02-10T08:39:46.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032730+0000", 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4550000000000001e-06 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032739+0000", 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23991668399999999 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272656+0000", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124196 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272780+0000", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152177 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272932+0000", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001504109 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.274436+0000", 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001049901 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.275486+0000", 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027108150000000001 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278197+0000", 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3831e-05 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278251+0000", 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0742999999999999e-05 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278292+0000", 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47734.0:2 147.1a 147:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-10T08:39:46.897 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 76.838153759999997, 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332, 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47734", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/29749952", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:39:46.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956760+0000", 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956768+0000", 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956776+0000", 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0839999999999999e-06 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956781+0000", 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7848000000000001e-05 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956819+0000", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.19e-06 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956825+0000", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090413761999999995 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047239+0000", 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112608 2024-02-10T08:39:46.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047352+0000", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014200000000000001 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047494+0000", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012910860000000001 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.048785+0000", 2024-02-10T08:39:46.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063636400000000003 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.055149+0000", 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023764580000000001 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057525+0000", 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8439000000000001e-05 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057583+0000", 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1635999999999998e-05 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057635+0000", 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53021.0:2 150.1e 150:7cdf4dc8:::oid_XattrsRoundTripPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-10T08:39:46.904 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:32.962819+0000", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 73.832096582000005, 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011295113000000001, 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53021", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2081909442", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962819+0000", 2024-02-10T08:39:46.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962819+0000", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962816+0000", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2784e-05 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962829+0000", 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1686e-05 2024-02-10T08:39:46.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962840+0000", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4003000000000004e-05 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.962904+0000", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110543 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.963015+0000", 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4200000000000007e-06 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.963023+0000", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.009213812 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.972237+0000", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122447 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.972360+0000", 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011072499999999999 2024-02-10T08:39:46.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,7", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.972470+0000", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000810946 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.973281+0000", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041046799999999999 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.973692+0000", 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034071299999999998 2024-02-10T08:39:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.974032+0000", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7893999999999999e-05 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.974080+0000", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3711999999999998e-05 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:32.974114+0000", 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011295113000000001 2024-02-10T08:39:46.910 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.911 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:39:46.911 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:39:46.911 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:39:46.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.796+0000 7fe712d80700 1 -- 172.21.15.107:0/1299371703 >> v1:172.21.15.107:6789/0 conn(0x7fe70c104330 legacy=0x7fe70c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- 172.21.15.107:0/1299371703 shutdown_connections 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- 172.21.15.107:0/1299371703 >> 172.21.15.107:0/1299371703 conn(0x7fe70c0f3640 msgr2=0x7fe70c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- 172.21.15.107:0/1299371703 shutdown_connections 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- 172.21.15.107:0/1299371703 wait complete. 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 librados: starting msgr at 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 librados: starting objecter 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 Processor -- start 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- start start 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 librados: setting wanted keys 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 librados: calling monclient init 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe70c117570 con 0x7fe70c0fda50 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe70c118750 con 0x7fe70c104330 2024-02-10T08:39:46.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe712d80700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe70c119950 con 0x7fe70c1007f0 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe710b1c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe70c0fda50 0x7fe70c0f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48702/0 (socket says 172.21.15.107:48702) 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe710b1c700 1 -- 172.21.15.107:0/2405427173 learned_addr learned my addr 172.21.15.107:0/2405427173 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.797+0000 7fe70bfff700 1 --1- 172.21.15.107:0/2405427173 >> v1:172.21.15.107:6790/0 conn(0x7fe70c1007f0 0x7fe70c0fa4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56454/0 (socket says 172.21.15.107:56454) 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter ms_handle_connect 0x7fe70c1007f0 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter ms_handle_connect 0x7fe70c0fda50 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter ms_handle_connect 0x7fe70c104330 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097788060 0 0) 0x7fe70c117570 con 0x7fe70c0fda50 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6fc0033f0 con 0x7fe70c0fda50 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255512547 0 0) 0x7fe70c119950 con 0x7fe70c1007f0 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c117570 con 0x7fe70c1007f0 2024-02-10T08:39:46.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237079716 0 0) 0x7fe70c118750 con 0x7fe70c104330 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c119950 con 0x7fe70c104330 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868376047 0 0) 0x7fe6fc0033f0 con 0x7fe70c0fda50 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c118750 con 0x7fe70c0fda50 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347497358 0 0) 0x7fe70c117570 con 0x7fe70c1007f0 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6fc0033f0 con 0x7fe70c1007f0 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe700002970 con 0x7fe70c0fda50 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6f8002910 con 0x7fe70c1007f0 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485171605 0 0) 0x7fe70c119950 con 0x7fe70c104330 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c117570 con 0x7fe70c104330 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe704003020 con 0x7fe70c104330 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1573894614 0 0) 0x7fe70c118750 con 0x7fe70c0fda50 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 >> v1:172.21.15.107:6790/0 conn(0x7fe70c1007f0 legacy=0x7fe70c0fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 >> v1:172.21.15.195:6789/0 conn(0x7fe70c104330 legacy=0x7fe70c115c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:46.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c11ab50 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c117760 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe700002ba0 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe700005bd0 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe712d80700 10 client.27732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c117bd0 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.798+0000 7fe712d80700 1 librados: init done 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.799+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c05f740 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe700001660 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(292..292 src has 50..292) v4 ==== 7967+0+0 (unknown 2574545839 0 0) 0x7fe700078d50 con 0x7fe70c0fda50 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 10 client.27732.objecter ms_dispatch 0x7fe70c001550 osd_map(292..292 src has 50..292) v4 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 3 client.27732.objecter handle_osd_map got epochs [292,292] > 0 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 3 client.27732.objecter handle_osd_map decoding full epoch 292 2024-02-10T08:39:46.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 20 client.27732.objecter dump_active .. 0 homeless 2024-02-10T08:39:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.800+0000 7fe709ffb700 10 client.27732.objecter ms_handle_connect 0x7fe6fc063cb0 2024-02-10T08:39:46.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.804+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe70004c1e0 con 0x7fe70c0fda50 2024-02-10T08:39:46.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:46.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:46.993+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe70c0f9240 con 0x7fe70c0fda50 2024-02-10T08:39:47.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:39:47.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:39:47.593 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:47.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:47.672 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:47.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:47.725 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:47.771 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:47.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.826+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3869016628 >> v1:172.21.15.107:6790/0 conn(0x7f1a740fc910 legacy=0x7f1a740fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.826+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3869016628 shutdown_connections 2024-02-10T08:39:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3869016628 >> 172.21.15.107:0/3869016628 conn(0x7f1a740f2510 msgr2=0x7f1a740f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3869016628 shutdown_connections 2024-02-10T08:39:47.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3869016628 wait complete. 2024-02-10T08:39:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 librados: starting msgr at 2024-02-10T08:39:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 librados: starting objecter 2024-02-10T08:39:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 Processor -- start 2024-02-10T08:39:47.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 -- start start 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 librados: setting wanted keys 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.827+0000 7f1a79ebe700 1 librados: calling monclient init 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a79ebe700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a741171d0 con 0x7f1a74103250 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a79ebe700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a741183b0 con 0x7f1a740ff710 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a79ebe700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a741195b0 con 0x7f1a740fc910 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a737fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1a740fc910 0x7f1a740f9790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56468/0 (socket says 172.21.15.107:56468) 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a737fe700 1 -- 172.21.15.107:0/3196838146 learned_addr learned my addr 172.21.15.107:0/3196838146 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter ms_handle_connect 0x7f1a74103250 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter ms_handle_connect 0x7f1a740fc910 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818988396 0 0) 0x7f1a741171d0 con 0x7f1a74103250 2024-02-10T08:39:47.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c0038e0 con 0x7f1a74103250 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter ms_handle_connect 0x7f1a740ff710 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182250363 0 0) 0x7f1a741195b0 con 0x7f1a740fc910 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a741171d0 con 0x7f1a740fc910 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755304924 0 0) 0x7f1a741183b0 con 0x7f1a740ff710 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a741195b0 con 0x7f1a740ff710 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4169447454 0 0) 0x7f1a5c0038e0 con 0x7f1a74103250 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a741183b0 con 0x7f1a74103250 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1a6c001950 con 0x7f1a74103250 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.828+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 835001053 0 0) 0x7f1a741171d0 con 0x7f1a740fc910 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a5c0038e0 con 0x7f1a740fc910 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1a64002e80 con 0x7f1a740fc910 2024-02-10T08:39:47.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133263423 0 0) 0x7f1a741195b0 con 0x7f1a740ff710 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a741171d0 con 0x7f1a740ff710 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1a68001840 con 0x7f1a740ff710 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1743217828 0 0) 0x7f1a5c0038e0 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 >> v1:172.21.15.195:6789/0 conn(0x7f1a740ff710 legacy=0x7f1a740f9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 >> v1:172.21.15.107:6789/0 conn(0x7f1a74103250 legacy=0x7f1a741158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7411a7b0 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a741197a0 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a79ebe700 10 client.53309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a74119c90 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a79ebe700 1 librados: init done 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1a64001e30 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.829+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1a64003740 con 0x7f1a740fc910 2024-02-10T08:39:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.830+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60004f20 con 0x7f1a740fc910 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.830+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1a64003940 con 0x7f1a740fc910 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(293..293 src has 50..293) v4 ==== 8269+0+0 (unknown 4209058285 0 0) 0x7f1a64079030 con 0x7f1a740fc910 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 10 client.53309.objecter ms_dispatch 0x7f1a74001550 osd_map(293..293 src has 50..293) v4 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 3 client.53309.objecter handle_osd_map got epochs [293,293] > 0 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 3 client.53309.objecter handle_osd_map decoding full epoch 293 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 20 client.53309.objecter dump_active .. 0 homeless 2024-02-10T08:39:47.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.831+0000 7f1a70ff9700 10 client.53309.objecter ms_handle_connect 0x7f1a5c063c30 2024-02-10T08:39:47.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:47.835+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1a6404c580 con 0x7f1a740fc910 2024-02-10T08:39:47.856 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:47.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:47.900 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:48.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.038+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f1a600051d0 con 0x7f1a740fc910 2024-02-10T08:39:48.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.230+0000 7fe709ffb700 1 -- 172.21.15.107:0/2405427173 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v294) v1 ==== 101+0+0 (unknown 1271510546 0 0) 0x7fe70004a5f0 con 0x7fe70c0fda50 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 10 librados: watch_flush enter 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 10 librados: watch_flush exit 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 20 client.27732.objecter shutdown clearing up homeless session... 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 10 client.27732.objecter successfully canceled tick 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe6fc063cb0 legacy=0x7fe6fc066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 >> v1:172.21.15.107:6789/0 conn(0x7fe70c0fda50 legacy=0x7fe70c0f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 shutdown_connections 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 >> 172.21.15.107:0/2405427173 conn(0x7fe70c0f3640 msgr2=0x7fe70c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 shutdown_connections 2024-02-10T08:39:48.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 -- 172.21.15.107:0/2405427173 wait complete. 2024-02-10T08:39:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:48.232+0000 7fe712d80700 1 librados: shutdown 2024-02-10T08:39:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.474858+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.475668+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.638191+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.641679+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.642912+0000 osd.7 (osd.7) 123 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.646450+0000 osd.7 (osd.7) 124 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:39:48.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.925067+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.927949+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.984427+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.984517+0000 mon.a (mon.0) 1988 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:46.984584+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.987725+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.988044+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.988154+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.988444+0000 mon.a (mon.0) 1991 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:46.994282+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:48.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.405371+0000 mgr.y (mgr.14553) 286 : cluster [DBG] pgmap v416: 218 pgs: 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.474858+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.475668+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.638191+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.641679+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.642912+0000 osd.7 (osd.7) 123 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.646450+0000 osd.7 (osd.7) 124 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.925067+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.927949+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.984427+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.984517+0000 mon.a (mon.0) 1988 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:46.984584+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-10T08:39:48.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.987725+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.988044+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.988154+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.988444+0000 mon.a (mon.0) 1991 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:46.994282+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.405371+0000 mgr.y (mgr.14553) 286 : cluster [DBG] pgmap v416: 218 pgs: 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.474858+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.475668+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.638191+0000 osd.5 (osd.5) 141 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.641679+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.642912+0000 osd.7 (osd.7) 123 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.646450+0000 osd.7 (osd.7) 124 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.925067+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.927949+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.984427+0000 mon.a (mon.0) 1987 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:48.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.984517+0000 mon.a (mon.0) 1988 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:46.984584+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.987725+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.988044+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.988154+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.988444+0000 mon.a (mon.0) 1991 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:46.994282+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:39:48.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.405371+0000 mgr.y (mgr.14553) 286 : cluster [DBG] pgmap v416: 218 pgs: 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:48.750 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:48.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:48.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:48.886 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.002+0000 7f1a70ff9700 1 -- 172.21.15.107:0/3196838146 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v295) v1 ==== 113+0+0 (unknown 523528686 0 0) 0x7f1a6404a9f0 con 0x7f1a740fc910 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 10 librados: watch_flush enter 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 10 librados: watch_flush exit 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 20 client.53309.objecter shutdown clearing up homeless session... 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 10 client.53309.objecter successfully canceled tick 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1a5c063c30 legacy=0x7f1a5c0660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 >> v1:172.21.15.107:6790/0 conn(0x7f1a740fc910 legacy=0x7f1a740f9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 shutdown_connections 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 >> 172.21.15.107:0/3196838146 conn(0x7f1a740f2510 msgr2=0x7f1a741066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 shutdown_connections 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 -- 172.21.15.107:0/3196838146 wait complete. 2024-02-10T08:39:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:49.005+0000 7f1a79ebe700 1 librados: shutdown 2024-02-10T08:39:49.007 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.471257+0000 osd.6 (osd.6) 119 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.473002+0000 osd.6 (osd.6) 120 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.634005+0000 osd.4 (osd.4) 132 : cluster [DBG] 60.a scrub starts 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.635593+0000 osd.4 (osd.4) 133 : cluster [DBG] 60.a scrub ok 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.647386+0000 osd.7 (osd.7) 125 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.648973+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:47.982200+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:47.989602+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:47.990217+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.039832+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.107:0/3196838146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.022 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.231060+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:49.023 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:48.231120+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-10T08:39:49.023 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.231776+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.023 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.231958+0000 mon.a (mon.0) 1998 : audit [INF] from='client.53309 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.471257+0000 osd.6 (osd.6) 119 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:39:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.473002+0000 osd.6 (osd.6) 120 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:39:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.634005+0000 osd.4 (osd.4) 132 : cluster [DBG] 60.a scrub starts 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.635593+0000 osd.4 (osd.4) 133 : cluster [DBG] 60.a scrub ok 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.647386+0000 osd.7 (osd.7) 125 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.648973+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:47.982200+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:47.989602+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:47.990217+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.039832+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.107:0/3196838146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.231060+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:48.231120+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.231776+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.231958+0000 mon.a (mon.0) 1998 : audit [INF] from='client.53309 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.471257+0000 osd.6 (osd.6) 119 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.473002+0000 osd.6 (osd.6) 120 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.634005+0000 osd.4 (osd.4) 132 : cluster [DBG] 60.a scrub starts 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.635593+0000 osd.4 (osd.4) 133 : cluster [DBG] 60.a scrub ok 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.647386+0000 osd.7 (osd.7) 125 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.648973+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:47.982200+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:47.989602+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:47.990217+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.039832+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.107:0/3196838146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.231060+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? v1:172.21.15.107:0/2405427173' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:48.231120+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.231776+0000 mon.a (mon.0) 1997 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]: dispatch 2024-02-10T08:39:49.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.231958+0000 mon.a (mon.0) 1998 : audit [INF] from='client.53309 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:39:49.942 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:49.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:50.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:50.076 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:48.519733+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:48.520932+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:48.986411+0000 mon.a (mon.0) 1999 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.999066+0000 mon.a (mon.0) 2000 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.999159+0000 mon.a (mon.0) 2001 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]': finished 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:48.999226+0000 mon.a (mon.0) 2002 : audit [INF] from='client.53309 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:39:50.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:48.999281+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:49.071950+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:49.072259+0000 mon.a (mon.0) 2004 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.405826+0000 mgr.y (mgr.14553) 287 : cluster [DBG] pgmap v419: 226 pgs: 8 unknown, 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.447114+0000 osd.2 (osd.2) 101 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.450640+0000 osd.2 (osd.2) 102 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:48.519733+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:48.520932+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:48.986411+0000 mon.a (mon.0) 1999 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.999066+0000 mon.a (mon.0) 2000 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.999159+0000 mon.a (mon.0) 2001 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]': finished 2024-02-10T08:39:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:48.999226+0000 mon.a (mon.0) 2002 : audit [INF] from='client.53309 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:48.999281+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:49.071950+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:49.072259+0000 mon.a (mon.0) 2004 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.405826+0000 mgr.y (mgr.14553) 287 : cluster [DBG] pgmap v419: 226 pgs: 8 unknown, 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.447114+0000 osd.2 (osd.2) 101 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:39:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.450640+0000 osd.2 (osd.2) 102 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:39:50.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:48.519733+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:39:50.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:48.520932+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:39:50.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:48.986411+0000 mon.a (mon.0) 1999 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:39:50.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.999066+0000 mon.a (mon.0) 2000 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi107-112411-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:50.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.999159+0000 mon.a (mon.0) 2001 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-37", "mode": "writeback"}]': finished 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:48.999226+0000 mon.a (mon.0) 2002 : audit [INF] from='client.53309 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:48.999281+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:49.071950+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:49.072259+0000 mon.a (mon.0) 2004 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.405826+0000 mgr.y (mgr.14553) 287 : cluster [DBG] pgmap v419: 226 pgs: 8 unknown, 2 creating+activating, 216 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.447114+0000 osd.2 (osd.2) 101 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:39:50.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.450640+0000 osd.2 (osd.2) 102 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:39:50.198 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:39:50.198 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:39:50.198 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:39:50.198 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:39:50.200 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:50.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:50.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:50.332 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:50.428 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T08:39:50.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:39:50.431 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:50.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:50.532 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:39:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:39:50.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:50.560 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:50.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:50.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:50.815 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:50.835 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:50.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:50.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:39:50.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.915+0000 7f32cd182700 1 -- 172.21.15.107:0/2247649568 >> v1:172.21.15.107:6789/0 conn(0x7f32c80fc910 legacy=0x7f32c80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.915+0000 7f32cd182700 1 -- 172.21.15.107:0/2247649568 shutdown_connections 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.915+0000 7f32cd182700 1 -- 172.21.15.107:0/2247649568 >> 172.21.15.107:0/2247649568 conn(0x7f32c80f2510 msgr2=0x7f32c80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.915+0000 7f32cd182700 1 -- 172.21.15.107:0/2247649568 shutdown_connections 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.915+0000 7f32cd182700 1 -- 172.21.15.107:0/2247649568 wait complete. 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 librados: starting msgr at 2024-02-10T08:39:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 librados: starting objecter 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 Processor -- start 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 -- start start 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 librados: setting wanted keys 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 librados: calling monclient init 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c80f9880 con 0x7f32c80ff710 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c8117710 con 0x7f32c80fc910 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32cd182700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c81188f0 con 0x7f32c8103250 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32c659c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f32c80ff710 0x7f32c81118b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57302/0 (socket says 172.21.15.107:57302) 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32c659c700 1 -- 172.21.15.107:0/2613758849 learned_addr learned my addr 172.21.15.107:0/2613758849 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32c759e700 1 --1- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6790/0 conn(0x7f32c8103250 0x7f32c8114fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55542/0 (socket says 172.21.15.107:55542) 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32affff700 10 client.?.objecter ms_handle_connect 0x7f32c80ff710 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32affff700 10 client.?.objecter ms_handle_connect 0x7f32c8103250 2024-02-10T08:39:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.916+0000 7f32affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 10 client.?.objecter ms_handle_connect 0x7f32c80fc910 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729638006 0 0) 0x7f32c80f9880 con 0x7f32c80ff710 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b00033f0 con 0x7f32c80ff710 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939599683 0 0) 0x7f32c81188f0 con 0x7f32c8103250 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c80f9880 con 0x7f32c8103250 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332207200 0 0) 0x7f32c8117710 con 0x7f32c80fc910 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c81188f0 con 0x7f32c80fc910 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437611560 0 0) 0x7f32b00033f0 con 0x7f32c80ff710 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32c8117710 con 0x7f32c80ff710 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f32b8002930 con 0x7f32c80ff710 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083307096 0 0) 0x7f32c80f9880 con 0x7f32c8103250 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b00033f0 con 0x7f32c8103250 2024-02-10T08:39:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908063691 0 0) 0x7f32c81188f0 con 0x7f32c80fc910 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32c80f9880 con 0x7f32c80fc910 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f32c0001bc0 con 0x7f32c8103250 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f32bc002ca0 con 0x7f32c80fc910 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 495452976 0 0) 0x7f32c8117710 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6790/0 conn(0x7f32c8103250 legacy=0x7f32c8114fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.195:6789/0 conn(0x7f32c80fc910 legacy=0x7f32c80f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8119ad0 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c8118ae0 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32cd182700 10 client.27744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c8118f50 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32cd182700 1 librados: init done 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f32b8001810 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.917+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f32b8005bc0 con 0x7f32c80ff710 2024-02-10T08:39:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.918+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f32b4004f00 con 0x7f32c80ff710 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f32b8001370 con 0x7f32c80ff710 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(296..296 src has 50..296) v4 ==== 8807+0+0 (unknown 3518428092 0 0) 0x7f32b8079020 con 0x7f32c80ff710 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 osd_map(296..296 src has 50..296) v4 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 3 client.27744.objecter handle_osd_map got epochs [296,296] > 0 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 3 client.27744.objecter handle_osd_map decoding full epoch 296 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 20 client.27744.objecter dump_active .. 0 homeless 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v296) v1 ==== 54+0+16 (unknown 3217538349 0 2420000192) 0x7f32b807b430 con 0x7f32c80ff710 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b0063c90 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 10 client.27744.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 1 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 1 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 0 1 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 4 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 10 client.27744.objecter _send_command 1 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.919+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b4004950 con 0x7f32b4004530 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4004530 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b4004950 con 0x7f32b4004530 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 10 client.27744.objecter _finish_command 1 = system:0 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 4 2024-02-10T08:39:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.920+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 0 1 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 10 client.27744.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 2 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 2 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b40051a0 osd=0 3 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 0 2 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 4 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 10 client.27744.objecter _send_command 2 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.935+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b4004d30 con 0x7f32b4004530 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.936+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b4004d30 con 0x7f32b4004530 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.936+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.936+0000 7f32affff700 10 client.27744.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.936+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 4 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.936+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 0 2 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 10 client.27744.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 3 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 3 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 1 3 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 4 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 10 client.27744.objecter _send_command 3 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b400a8e0 con 0x7f32b4004150 2024-02-10T08:39:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.937+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4004150 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.938+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b400a8e0 con 0x7f32b4004150 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.938+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.938+0000 7f32affff700 10 client.27744.objecter _finish_command 3 = system:0 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.938+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 4 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.938+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 1 3 2024-02-10T08:39:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 10 client.27744.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 4 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 4 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4003fe0 osd=1 3 2024-02-10T08:39:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 1 4 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 4 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 10 client.27744.objecter _send_command 4 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.955+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b40017f0 con 0x7f32b4004150 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b40017f0 con 0x7f32b4004150 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32affff700 10 client.27744.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 4 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 1 4 2024-02-10T08:39:50.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 10 client.27744.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 5 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 5 2024-02-10T08:39:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 2 5 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 4 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 10 client.27744.objecter _send_command 5 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.957+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b40152a0 con 0x7f32b4011790 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4011790 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b40152a0 con 0x7f32b4011790 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 10 client.27744.objecter _finish_command 5 = system:0 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 4 2024-02-10T08:39:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.958+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 2 5 2024-02-10T08:39:50.970 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:50.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 10 client.27744.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:50.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 6 2024-02-10T08:39:50.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 6 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4003630 osd=2 3 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 2 6 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 4 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 10 client.27744.objecter _send_command 6 2024-02-10T08:39:50.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.981+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b40030a0 con 0x7f32b4011790 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.984+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b40030a0 con 0x7f32b4011790 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.984+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.984+0000 7f32affff700 10 client.27744.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.984+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 4 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.984+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 2 6 2024-02-10T08:39:50.985 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 10 client.27744.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 7 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 7 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 3 7 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 4 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 10 client.27744.objecter _send_command 7 2024-02-10T08:39:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.985+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b4011520 con 0x7f32b4016590 2024-02-10T08:39:50.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.987+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4016590 2024-02-10T08:39:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.990+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b4011520 con 0x7f32b4016590 2024-02-10T08:39:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.990+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:39:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.990+0000 7f32affff700 10 client.27744.objecter _finish_command 7 = system:0 2024-02-10T08:39:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.990+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 4 2024-02-10T08:39:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:50.990+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 3 7 2024-02-10T08:39:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:39:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:51.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:39:51.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 10 client.27744.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:51.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 8 2024-02-10T08:39:51.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 8 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4010de0 osd=3 3 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 3 8 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 4 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 10 client.27744.objecter _send_command 8 2024-02-10T08:39:51.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.012+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b4010850 con 0x7f32b4016590 2024-02-10T08:39:51.014 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:51.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b4010850 con 0x7f32b4016590 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 10 client.27744.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 4 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 3 8 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 10 client.27744.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 9 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 9 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 4 9 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 4 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 10 client.27744.objecter _send_command 9 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b40107b0 con 0x7f32b40160f0 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.014+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b40160f0 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.015+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b40107b0 con 0x7f32b40160f0 2024-02-10T08:39:51.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.015+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:39:51.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.015+0000 7f32affff700 10 client.27744.objecter _finish_command 9 = system:0 2024-02-10T08:39:51.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.015+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 4 2024-02-10T08:39:51.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.015+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 4 9 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 10 client.27744.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 10 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 10 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4010610 osd=4 3 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 4 10 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 4 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 10 client.27744.objecter _send_command 10 2024-02-10T08:39:51.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.032+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b40107b0 con 0x7f32b40160f0 2024-02-10T08:39:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b40107b0 con 0x7f32b40160f0 2024-02-10T08:39:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32affff700 10 client.27744.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 4 2024-02-10T08:39:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 4 10 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 10 client.27744.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 11 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 11 2024-02-10T08:39:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 5 11 2024-02-10T08:39:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 4 2024-02-10T08:39:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 10 client.27744.objecter _send_command 11 2024-02-10T08:39:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.034+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b4020b40 con 0x7f32b401ee60 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b401ee60 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b4020b40 con 0x7f32b401ee60 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 10 client.27744.objecter _finish_command 11 = system:0 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 4 2024-02-10T08:39:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.035+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 5 11 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 10 client.27744.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 12 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 12 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b40107b0 osd=5 3 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 5 12 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 4 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 10 client.27744.objecter _send_command 12 2024-02-10T08:39:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.050+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b4020b40 con 0x7f32b401ee60 2024-02-10T08:39:51.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.052+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b4020b40 con 0x7f32b401ee60 2024-02-10T08:39:51.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.052+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:51.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.052+0000 7f32affff700 10 client.27744.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.052+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 4 2024-02-10T08:39:51.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.052+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 5 12 2024-02-10T08:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 10 client.27744.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 13 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 13 2024-02-10T08:39:51.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 6 13 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 4 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 10 client.27744.objecter _send_command 13 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b4020f30 con 0x7f32b4023620 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.053+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4023620 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.054+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32c000cad0 con 0x7f32b4023620 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.054+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.054+0000 7f32affff700 10 client.27744.objecter _finish_command 13 = system:0 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.054+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 4 2024-02-10T08:39:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.054+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 6 13 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 10 client.27744.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 14 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 14 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4020b40 osd=6 3 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 6 14 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 4 2024-02-10T08:39:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 10 client.27744.objecter _send_command 14 2024-02-10T08:39:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.068+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b4028770 con 0x7f32b4023620 2024-02-10T08:39:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b4028770 con 0x7f32b4023620 2024-02-10T08:39:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32affff700 10 client.27744.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 4 2024-02-10T08:39:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 6 14 2024-02-10T08:39:51.071 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:39:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 10 client.27744.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 15 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 15 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 7 15 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 4 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 10 client.27744.objecter _send_command 15 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.070+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b4029420 con 0x7f32b4028df0 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 10 client.27744.objecter ms_handle_connect 0x7f32b4028df0 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f32b4029420 con 0x7f32b4028df0 2024-02-10T08:39:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:39:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 10 client.27744.objecter _finish_command 15 = system:0 2024-02-10T08:39:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 4 2024-02-10T08:39:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.071+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 7 15 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 10 client.27744.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign -1 16 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 20 client.27744.objecter _get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 15 client.27744.objecter _session_command_op_remove -1 16 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 20 client.27744.objecter get_session s=0x7f32b4028770 osd=7 3 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 15 client.27744.objecter _session_command_op_assign 7 16 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 4 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 10 client.27744.objecter _send_command 16 2024-02-10T08:39:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.085+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f32b40293e0 con 0x7f32b4028df0 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.087+0000 7f32affff700 1 -- 172.21.15.107:0/2613758849 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f32b40293e0 con 0x7f32b4028df0 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.087+0000 7f32affff700 10 client.27744.objecter ms_dispatch 0x7f32c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.087+0000 7f32affff700 10 client.27744.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.087+0000 7f32affff700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 4 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.087+0000 7f32affff700 15 client.27744.objecter _session_command_op_remove 7 16 2024-02-10T08:39:51.088 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:39:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 librados: watch_flush enter 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 librados: watch_flush exit 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.0 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6801/3321596972 conn(0x7f32b4004530 legacy=0x7f32b4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40051a0 osd=0 1 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.1 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6805/133516126 conn(0x7f32b4004150 legacy=0x7f32b400eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003fe0 osd=1 1 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.2 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6809/3575881728 conn(0x7f32b4011790 legacy=0x7f32b4013b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4003630 osd=2 1 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.3 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6813/2871825691 conn(0x7f32b4016590 legacy=0x7f32b4018980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010de0 osd=3 1 2024-02-10T08:39:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.4 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.195:6800/2499671285 conn(0x7f32b40160f0 legacy=0x7f32b401d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32c759e700 1 -- 172.21.15.107:0/2613758849 reap_dead start 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4010610 osd=4 1 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.5 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.195:6804/4178321486 conn(0x7f32b401ee60 legacy=0x7f32b401f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32c759e700 1 -- 172.21.15.107:0/2613758849 reap_dead start 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b40107b0 osd=5 1 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.6 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.195:6808/1417479017 conn(0x7f32b4023620 legacy=0x7f32b4025a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4020b40 osd=6 1 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter close_session for osd.7 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.195:6812/1934176928 conn(0x7f32b4028df0 legacy=0x7f32b4027130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter put_session s=0x7f32b4028770 osd=7 1 2024-02-10T08:39:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 20 client.27744.objecter shutdown clearing up homeless session... 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 10 client.27744.objecter successfully canceled tick 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32c759e700 1 -- 172.21.15.107:0/2613758849 reap_dead start 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6800/2634842009 conn(0x7f32b0063c90 legacy=0x7f32b0066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> v1:172.21.15.107:6789/0 conn(0x7f32c80ff710 legacy=0x7f32c81118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 shutdown_connections 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 >> 172.21.15.107:0/2613758849 conn(0x7f32c80f2510 msgr2=0x7f32c81066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 shutdown_connections 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 -- 172.21.15.107:0/2613758849 wait complete. 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:51.088+0000 7f32cd182700 1 librados: shutdown 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.093 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:39:51.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:51.164 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.528863+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:39:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.529720+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:39:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.712171+0000 osd.3 (osd.3) 153 : cluster [DBG] 60.14 deep-scrub starts 2024-02-10T08:39:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:49.714830+0000 osd.3 (osd.3) 154 : cluster [DBG] 60.14 deep-scrub ok 2024-02-10T08:39:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.002717+0000 mon.a (mon.0) 2005 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:50.002785+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.007212+0000 mon.b (mon.1) 483 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.007830+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.919853+0000 mon.a (mon.0) 2008 : audit [DBG] from='client.? v1:172.21.15.107:0/2613758849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:50.967600+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.973193+0000 mon.a (mon.0) 2010 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:50.973297+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.980870+0000 mon.c (mon.2) 334 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:50.988723+0000 mon.a (mon.0) 2012 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.528863+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.529720+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.712171+0000 osd.3 (osd.3) 153 : cluster [DBG] 60.14 deep-scrub starts 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:49.714830+0000 osd.3 (osd.3) 154 : cluster [DBG] 60.14 deep-scrub ok 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.002717+0000 mon.a (mon.0) 2005 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:50.002785+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.007212+0000 mon.b (mon.1) 483 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.007830+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.919853+0000 mon.a (mon.0) 2008 : audit [DBG] from='client.? v1:172.21.15.107:0/2613758849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:50.967600+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.973193+0000 mon.a (mon.0) 2010 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:50.973297+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-10T08:39:51.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.980870+0000 mon.c (mon.2) 334 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:50.988723+0000 mon.a (mon.0) 2012 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.528863+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.529720+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.712171+0000 osd.3 (osd.3) 153 : cluster [DBG] 60.14 deep-scrub starts 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:49.714830+0000 osd.3 (osd.3) 154 : cluster [DBG] 60.14 deep-scrub ok 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.002717+0000 mon.a (mon.0) 2005 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:50.002785+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.007212+0000 mon.b (mon.1) 483 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.007830+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]: dispatch 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.919853+0000 mon.a (mon.0) 2008 : audit [DBG] from='client.? v1:172.21.15.107:0/2613758849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:50.967600+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:39:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.973193+0000 mon.a (mon.0) 2010 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-37"}]': finished 2024-02-10T08:39:51.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:50.973297+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-10T08:39:51.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.980870+0000 mon.c (mon.2) 334 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:50.988723+0000 mon.a (mon.0) 2012 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:51.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:51.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:51.520 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:51.540 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:51.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:51.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:39:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:39:51.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:51.670 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:51.818 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:51.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:51.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:39:51.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:51.948 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [==========] Running 72 tests from 4 test suites. 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: seed 112490 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (298 ms) 2024-02-10T08:39:51.993 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10711 ms) 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (14 ms) 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11023 ms total) 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7166 ms) 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7820 ms) 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi107-112490-12 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 19 more times... 2024-02-10T08:39:51.994 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (12006 ms) 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: my_snaps [4] 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: my_snaps [5,4] 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: my_snaps [6,5,4] 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: promoting some heads 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: promoting from clones for snap 7 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: waiting for scrubs... 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: done waiting 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46025 ms) 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9994 ms) 2024-02-10T08:39:51.995 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8012 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8222 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7779 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10007 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9011 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9999 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7997 ms) 2024-02-10T08:39:51.996 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7997 ms) 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13013 ms) 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7986 ms) 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-10T08:39:51.997 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8014 ms) 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:50.628321+0000 osd.4 (osd.4) 134 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:50.630034+0000 osd.4 (osd.4) 135 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:51.335703+0000 mgr.y (mgr.14553) 288 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:51.406591+0000 mgr.y (mgr.14553) 289 : cluster [DBG] pgmap v422: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 66 op/s 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:51.448580+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:51.451476+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:51.988646+0000 mon.a (mon.0) 2013 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:51.988726+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:51.993138+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:51.993468+0000 mon.a (mon.0) 2015 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:50.628321+0000 osd.4 (osd.4) 134 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:50.630034+0000 osd.4 (osd.4) 135 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:51.335703+0000 mgr.y (mgr.14553) 288 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:51.406591+0000 mgr.y (mgr.14553) 289 : cluster [DBG] pgmap v422: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 66 op/s 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:51.448580+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-10T08:39:52.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:51.451476+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:51.988646+0000 mon.a (mon.0) 2013 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:51.988726+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:51.993138+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:51.993468+0000 mon.a (mon.0) 2015 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:50.628321+0000 osd.4 (osd.4) 134 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:50.630034+0000 osd.4 (osd.4) 135 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:51.335703+0000 mgr.y (mgr.14553) 288 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:51.406591+0000 mgr.y (mgr.14553) 289 : cluster [DBG] pgmap v422: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 66 op/s 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:51.448580+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:51.451476+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:51.988646+0000 mon.a (mon.0) 2013 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:52.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:51.988726+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-10T08:39:52.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:51.993138+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.107:0/3193245591' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:51.993468+0000 mon.a (mon.0) 2015 : audit [INF] from='client.53297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]: dispatch 2024-02-10T08:39:52.820 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:52.820 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:53.132 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:53.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:53.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:53.264 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:53.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:53.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:53.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:39:53.519 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:53.538 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:53.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:53.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- 172.21.15.107:0/3950276916 >> v1:172.21.15.107:6789/0 conn(0x7f0b240f7300 legacy=0x7f0b24104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- 172.21.15.107:0/3950276916 shutdown_connections 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- 172.21.15.107:0/3950276916 >> 172.21.15.107:0/3950276916 conn(0x7f0b240f2510 msgr2=0x7f0b240f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- 172.21.15.107:0/3950276916 shutdown_connections 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- 172.21.15.107:0/3950276916 wait complete. 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 librados: starting msgr at 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 librados: starting objecter 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 Processor -- start 2024-02-10T08:39:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 -- start start 2024-02-10T08:39:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 librados: setting wanted keys 2024-02-10T08:39:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.647+0000 7f0b29cbc700 1 librados: calling monclient init 2024-02-10T08:39:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b29cbc700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b24119560 con 0x7f0b240f7300 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b29cbc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b2411a740 con 0x7f0b240f6810 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b29cbc700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b2411b940 con 0x7f0b24106640 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b237fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0b240f6810 0x7f0b24102920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:56566/0 (socket says 172.21.15.107:56566) 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b22ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0b240f7300 0x7f0b24103030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57322/0 (socket says 172.21.15.107:57322) 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b237fe700 1 -- 172.21.15.107:0/2766486693 learned_addr learned my addr 172.21.15.107:0/2766486693 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b23fff700 1 --1- 172.21.15.107:0/2766486693 >> v1:172.21.15.107:6790/0 conn(0x7f0b24106640 0x7f0b24117c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55584/0 (socket says 172.21.15.107:55584) 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter ms_handle_connect 0x7f0b240f7300 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter ms_handle_connect 0x7f0b240f6810 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter ms_handle_connect 0x7f0b24106640 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540530564 0 0) 0x7f0b2411a740 con 0x7f0b240f6810 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c0033f0 con 0x7f0b240f6810 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045270353 0 0) 0x7f0b2411b940 con 0x7f0b24106640 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2411a740 con 0x7f0b24106640 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425671989 0 0) 0x7f0b24119560 con 0x7f0b240f7300 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2411b940 con 0x7f0b240f7300 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254124517 0 0) 0x7f0b0c0033f0 con 0x7f0b240f6810 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b24119560 con 0x7f0b240f6810 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.648+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b14001a50 con 0x7f0b240f6810 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 752067673 0 0) 0x7f0b2411a740 con 0x7f0b24106640 2024-02-10T08:39:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b0c0033f0 con 0x7f0b24106640 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218004550 0 0) 0x7f0b2411b940 con 0x7f0b240f7300 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2411a740 con 0x7f0b240f7300 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b1c001c40 con 0x7f0b24106640 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b18000ed0 con 0x7f0b240f7300 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1482775179 0 0) 0x7f0b24119560 con 0x7f0b240f6810 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 >> v1:172.21.15.107:6790/0 conn(0x7f0b24106640 legacy=0x7f0b24117c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 >> v1:172.21.15.107:6789/0 conn(0x7f0b240f7300 legacy=0x7f0b24103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2411cb40 con 0x7f0b240f6810 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2411a930 con 0x7f0b240f6810 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b29cbc700 10 client.48049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2411ada0 con 0x7f0b240f6810 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b29cbc700 1 librados: init done 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0b140024d0 con 0x7f0b240f6810 2024-02-10T08:39:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.649+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b14005b20 con 0x7f0b240f6810 2024-02-10T08:39:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.650+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b10004f20 con 0x7f0b240f6810 2024-02-10T08:39:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.650+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0b14002030 con 0x7f0b240f6810 2024-02-10T08:39:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(299..299 src has 50..299) v4 ==== 7954+0+0 (unknown 1847985360 0 0) 0x7f0b14078c40 con 0x7f0b240f6810 2024-02-10T08:39:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 10 client.48049.objecter ms_dispatch 0x7f0b24001550 osd_map(299..299 src has 50..299) v4 2024-02-10T08:39:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 3 client.48049.objecter handle_osd_map got epochs [299,299] > 0 2024-02-10T08:39:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 3 client.48049.objecter handle_osd_map decoding full epoch 299 2024-02-10T08:39:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 20 client.48049.objecter dump_active .. 0 homeless 2024-02-10T08:39:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.651+0000 7f0b20ff9700 10 client.48049.objecter ms_handle_connect 0x7f0b0c06c580 2024-02-10T08:39:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.659+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0b1404c0a0 con 0x7f0b240f6810 2024-02-10T08:39:53.670 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:53.695 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:53.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:53.783 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:53.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:39:53.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.825+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b10000b90 con 0x7f0b0c06c580 2024-02-10T08:39:53.828 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.840+0000 7f0b20ff9700 1 -- 172.21.15.107:0/2766486693 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550018 (unknown 2965378022 0 4228011132) 0x7f0b10000b90 con 0x7f0b0c06c580 2024-02-10T08:39:53.848 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:39:53.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 10 librados: watch_flush enter 2024-02-10T08:39:53.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 10 librados: watch_flush exit 2024-02-10T08:39:53.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 20 client.48049.objecter shutdown clearing up homeless session... 2024-02-10T08:39:53.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 10 client.48049.objecter successfully canceled tick 2024-02-10T08:39:53.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0b0c06c580 legacy=0x7f0b0c06ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 >> v1:172.21.15.195:6789/0 conn(0x7f0b240f6810 legacy=0x7f0b24102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 shutdown_connections 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.844+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 >> 172.21.15.107:0/2766486693 conn(0x7f0b240f2510 msgr2=0x7f0b24109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.845+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 shutdown_connections 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.845+0000 7f0b29cbc700 1 -- 172.21.15.107:0/2766486693 wait complete. 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:53.845+0000 7f0b29cbc700 1 librados: shutdown 2024-02-10T08:39:53.855 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:39:54.026 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:39:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:54.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.545386+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.548834+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.579512+0000 osd.4 (osd.4) 136 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.582969+0000 osd.4 (osd.4) 137 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.705016+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.708407+0000 osd.5 (osd.5) 144 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:52.992107+0000 mon.a (mon.0) 2016 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:52.992193+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-10T08:39:54.087 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:52.996919+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:52.997328+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.016223+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.016539+0000 mon.a (mon.0) 2019 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.017580+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.017830+0000 mon.a (mon.0) 2020 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.018324+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.018556+0000 mon.a (mon.0) 2021 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.041655+0000 osd.1 (osd.1) 143 : cluster [DBG] 60.9 deep-scrub starts 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.048145+0000 osd.1 (osd.1) 144 : cluster [DBG] 60.9 deep-scrub ok 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.406982+0000 mgr.y (mgr.14553) 290 : cluster [DBG] pgmap v425: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T08:39:54.088 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.826024+0000 mgr.y (mgr.14553) 291 : audit [DBG] from='client.48049 v1:172.21.15.107:0/2766486693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:39:54.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:54.159 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:54.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.545386+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:39:54.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.548834+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.579512+0000 osd.4 (osd.4) 136 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.582969+0000 osd.4 (osd.4) 137 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.705016+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.708407+0000 osd.5 (osd.5) 144 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:52.992107+0000 mon.a (mon.0) 2016 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:52.992193+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:52.996919+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:52.997328+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.016223+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.016539+0000 mon.a (mon.0) 2019 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.017580+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.017830+0000 mon.a (mon.0) 2020 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.018324+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.018556+0000 mon.a (mon.0) 2021 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.041655+0000 osd.1 (osd.1) 143 : cluster [DBG] 60.9 deep-scrub starts 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.048145+0000 osd.1 (osd.1) 144 : cluster [DBG] 60.9 deep-scrub ok 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.406982+0000 mgr.y (mgr.14553) 290 : cluster [DBG] pgmap v425: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.826024+0000 mgr.y (mgr.14553) 291 : audit [DBG] from='client.48049 v1:172.21.15.107:0/2766486693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.545386+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.548834+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.579512+0000 osd.4 (osd.4) 136 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.582969+0000 osd.4 (osd.4) 137 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.705016+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:39:54.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.708407+0000 osd.5 (osd.5) 144 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:52.992107+0000 mon.a (mon.0) 2016 : audit [INF] from='client.53297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi107-112411-52"}]': finished 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:52.992193+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:52.996919+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:52.997328+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.016223+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.016539+0000 mon.a (mon.0) 2019 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.017580+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.017830+0000 mon.a (mon.0) 2020 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.018324+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.018556+0000 mon.a (mon.0) 2021 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.041655+0000 osd.1 (osd.1) 143 : cluster [DBG] 60.9 deep-scrub starts 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.048145+0000 osd.1 (osd.1) 144 : cluster [DBG] 60.9 deep-scrub ok 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.406982+0000 mgr.y (mgr.14553) 290 : cluster [DBG] pgmap v425: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T08:39:54.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.826024+0000 mgr.y (mgr.14553) 291 : audit [DBG] from='client.48049 v1:172.21.15.107:0/2766486693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:39:54.257 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:54.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:54.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:39:54.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:54.388 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:54.632 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-02-10T08:39:53.406728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007021,"num_objects":316,"num_object_clones":2,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1207,"num_read_kb":865,"num_write":983,"num_write_kb":4497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2292,"ondisk_log_size":2292,"up":558,"acting":558,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":649,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2183360,"kb_used_data":45384,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":654032704,"statfs":{"total":671965249536,"available":669729488896,"internally_reserved":0,"allocated":46473216,"data_stored":31062611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4000008,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":3907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001142"},"pg_stats":[{"pgid":"188.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"188.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:39:52.986532+0000","last_change":"2024-02-10T08:39:52.986532+0000","last_active":"2024-02-10T08:39:52.986532+0000","last_peered":"2024-02-10T08:39:52.986532+0000","last_clean":"2024-02-10T08:39:52.986532+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:39:52.986532+0000","last_undegraded":"2024-02-10T08:39:52.986532+0000","last_fullsized":"2024-02-10T08:39:52.986532+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003304+0000","last_change":"2024-02-10T08:39:32.971969+0000","last_active":"2024-02-10T08:39:50.003304+0000","last_peered":"2024-02-10T08:39:50.003304+0000","last_clean":"2024-02-10T08:39:50.003304+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:39:50.003304+0000","last_undegraded":"2024-02-10T08:39:50.003304+0000","last_fullsized":"2024-02-10T08:39:50.003304+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"236'27","reported_seq":268,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000585+0000","last_change":"2024-02-10T08:39:38.573741+0000","last_active":"2024-02-10T08:39:50.000585+0000","last_peered":"2024-02-10T08:39:50.000585+0000","last_clean":"2024-02-10T08:39:50.000585+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:39:50.000585+0000","last_undegraded":"2024-02-10T08:39:50.000585+0000","last_fullsized":"2024-02-10T08:39:50.000585+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"250'39","reported_seq":267,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000148+0000","last_change":"2024-02-10T08:39:40.483224+0000","last_active":"2024-02-10T08:39:50.000148+0000","last_peered":"2024-02-10T08:39:50.000148+0000","last_clean":"2024-02-10T08:39:50.000148+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:39:50.000148+0000","last_undegraded":"2024-02-10T08:39:50.000148+0000","last_fullsized":"2024-02-10T08:39:50.000148+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"231'107","reported_seq":336,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006432+0000","last_change":"2024-02-10T08:39:32.974249+0000","last_active":"2024-02-10T08:39:50.006432+0000","last_peered":"2024-02-10T08:39:50.006432+0000","last_clean":"2024-02-10T08:39:50.006432+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:39:50.006432+0000","last_undegraded":"2024-02-10T08:39:50.006432+0000","last_fullsized":"2024-02-10T08:39:50.006432+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"222'34","reported_seq":240,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001190+0000","last_change":"2024-02-10T08:39:32.970237+0000","last_active":"2024-02-10T08:39:50.001190+0000","last_peered":"2024-02-10T08:39:50.001190+0000","last_clean":"2024-02-10T08:39:50.001190+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:39:50.001190+0000","last_undegraded":"2024-02-10T08:39:50.001190+0000","last_fullsized":"2024-02-10T08:39:50.001190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"226'71","reported_seq":299,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008484+0000","last_change":"2024-02-10T08:39:49.450775+0000","last_active":"2024-02-10T08:39:50.008484+0000","last_peered":"2024-02-10T08:39:50.008484+0000","last_clean":"2024-02-10T08:39:50.008484+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:39:50.008484+0000","last_undegraded":"2024-02-10T08:39:50.008484+0000","last_fullsized":"2024-02-10T08:39:50.008484+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":279,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001137+0000","last_change":"2024-02-10T08:39:34.971975+0000","last_active":"2024-02-10T08:39:50.001137+0000","last_peered":"2024-02-10T08:39:50.001137+0000","last_clean":"2024-02-10T08:39:50.001137+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:39:50.001137+0000","last_undegraded":"2024-02-10T08:39:50.001137+0000","last_fullsized":"2024-02-10T08:39:50.001137+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"250'22","reported_seq":261,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000728+0000","last_change":"2024-02-10T08:39:42.510684+0000","last_active":"2024-02-10T08:39:50.000728+0000","last_peered":"2024-02-10T08:39:50.000728+0000","last_clean":"2024-02-10T08:39:50.000728+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:39:50.000728+0000","last_undegraded":"2024-02-10T08:39:50.000728+0000","last_fullsized":"2024-02-10T08:39:50.000728+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"250'101","reported_seq":322,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008068+0000","last_change":"2024-02-10T08:39:32.974027+0000","last_active":"2024-02-10T08:39:50.008068+0000","last_peered":"2024-02-10T08:39:50.008068+0000","last_clean":"2024-02-10T08:39:50.008068+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:39:50.008068+0000","last_undegraded":"2024-02-10T08:39:50.008068+0000","last_fullsized":"2024-02-10T08:39:50.008068+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"228'22","reported_seq":260,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000321+0000","last_change":"2024-02-10T08:39:47.635717+0000","last_active":"2024-02-10T08:39:50.000321+0000","last_peered":"2024-02-10T08:39:50.000321+0000","last_clean":"2024-02-10T08:39:50.000321+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:39:50.000321+0000","last_undegraded":"2024-02-10T08:39:50.000321+0000","last_fullsized":"2024-02-10T08:39:50.000321+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"266'104","reported_seq":308,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002509+0000","last_change":"2024-02-10T08:39:32.970954+0000","last_active":"2024-02-10T08:39:50.002509+0000","last_peered":"2024-02-10T08:39:50.002509+0000","last_clean":"2024-02-10T08:39:50.002509+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:39:50.002509+0000","last_undegraded":"2024-02-10T08:39:50.002509+0000","last_fullsized":"2024-02-10T08:39:50.002509+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"236'85","reported_seq":315,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.630154+0000","last_change":"2024-02-10T08:39:50.630114+0000","last_active":"2024-02-10T08:39:50.630154+0000","last_peered":"2024-02-10T08:39:50.630154+0000","last_clean":"2024-02-10T08:39:50.630154+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:39:50.630154+0000","last_undegraded":"2024-02-10T08:39:50.630154+0000","last_fullsized":"2024-02-10T08:39:50.630154+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"231'93","reported_seq":313,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120563+0000","last_change":"2024-02-10T08:39:43.510896+0000","last_active":"2024-02-10T08:39:50.120563+0000","last_peered":"2024-02-10T08:39:50.120563+0000","last_clean":"2024-02-10T08:39:50.120563+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:39:50.120563+0000","last_undegraded":"2024-02-10T08:39:50.120563+0000","last_fullsized":"2024-02-10T08:39:50.120563+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'93","last_scrub_stamp":"2024-02-10T08:39:43.510789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.510789+0000","log_size":93,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":72,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":262,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414582+0000","last_change":"2024-02-10T08:39:33.017060+0000","last_active":"2024-02-10T08:39:50.414582+0000","last_peered":"2024-02-10T08:39:50.414582+0000","last_clean":"2024-02-10T08:39:50.414582+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:39:50.414582+0000","last_undegraded":"2024-02-10T08:39:50.414582+0000","last_fullsized":"2024-02-10T08:39:50.414582+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"287'56","reported_seq":281,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120413+0000","last_change":"2024-02-10T08:39:44.469992+0000","last_active":"2024-02-10T08:39:50.120413+0000","last_peered":"2024-02-10T08:39:50.120413+0000","last_clean":"2024-02-10T08:39:50.120413+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:39:50.120413+0000","last_undegraded":"2024-02-10T08:39:50.120413+0000","last_fullsized":"2024-02-10T08:39:50.120413+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-02-10T08:39:44.469923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.469923+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"226'68","reported_seq":293,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009410+0000","last_change":"2024-02-10T08:39:32.974869+0000","last_active":"2024-02-10T08:39:50.009410+0000","last_peered":"2024-02-10T08:39:50.009410+0000","last_clean":"2024-02-10T08:39:50.009410+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:39:50.009410+0000","last_undegraded":"2024-02-10T08:39:50.009410+0000","last_fullsized":"2024-02-10T08:39:50.009410+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"295'55","reported_seq":286,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006130+0000","last_change":"2024-02-10T08:39:33.014018+0000","last_active":"2024-02-10T08:39:50.006130+0000","last_peered":"2024-02-10T08:39:50.006130+0000","last_clean":"2024-02-10T08:39:50.006130+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:39:50.006130+0000","last_undegraded":"2024-02-10T08:39:50.006130+0000","last_fullsized":"2024-02-10T08:39:50.006130+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'54","last_scrub_stamp":"2024-02-10T08:39:12.673905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.673905+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000054,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":3924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":334,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005883+0000","last_change":"2024-02-10T08:39:46.646522+0000","last_active":"2024-02-10T08:39:50.005883+0000","last_peered":"2024-02-10T08:39:50.005883+0000","last_clean":"2024-02-10T08:39:50.005883+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:39:50.005883+0000","last_undegraded":"2024-02-10T08:39:50.005883+0000","last_fullsized":"2024-02-10T08:39:50.005883+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:39:46.646454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.646454+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"258'103","reported_seq":322,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006194+0000","last_change":"2024-02-10T08:39:41.578640+0000","last_active":"2024-02-10T08:39:50.006194+0000","last_peered":"2024-02-10T08:39:50.006194+0000","last_clean":"2024-02-10T08:39:50.006194+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:39:50.006194+0000","last_undegraded":"2024-02-10T08:39:50.006194+0000","last_fullsized":"2024-02-10T08:39:50.006194+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"209'68","reported_seq":300,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005511+0000","last_change":"2024-02-10T08:39:32.974271+0000","last_active":"2024-02-10T08:39:50.005511+0000","last_peered":"2024-02-10T08:39:50.005511+0000","last_clean":"2024-02-10T08:39:50.005511+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:39:50.005511+0000","last_undegraded":"2024-02-10T08:39:50.005511+0000","last_fullsized":"2024-02-10T08:39:50.005511+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"215'85","reported_seq":306,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006689+0000","last_change":"2024-02-10T08:39:32.975858+0000","last_active":"2024-02-10T08:39:50.006689+0000","last_peered":"2024-02-10T08:39:50.006689+0000","last_clean":"2024-02-10T08:39:50.006689+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:39:50.006689+0000","last_undegraded":"2024-02-10T08:39:50.006689+0000","last_fullsized":"2024-02-10T08:39:50.006689+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":312,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414705+0000","last_change":"2024-02-10T08:39:43.711787+0000","last_active":"2024-02-10T08:39:50.414705+0000","last_peered":"2024-02-10T08:39:50.414705+0000","last_clean":"2024-02-10T08:39:50.414705+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:39:50.414705+0000","last_undegraded":"2024-02-10T08:39:50.414705+0000","last_fullsized":"2024-02-10T08:39:50.414705+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"230'17","reported_seq":248,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000724+0000","last_change":"2024-02-10T08:39:32.970028+0000","last_active":"2024-02-10T08:39:50.000724+0000","last_peered":"2024-02-10T08:39:50.000724+0000","last_clean":"2024-02-10T08:39:50.000724+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:39:50.000724+0000","last_undegraded":"2024-02-10T08:39:50.000724+0000","last_fullsized":"2024-02-10T08:39:50.000724+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416060+0000","last_change":"2024-02-10T08:39:49.714905+0000","last_active":"2024-02-10T08:39:50.416060+0000","last_peered":"2024-02-10T08:39:50.416060+0000","last_clean":"2024-02-10T08:39:50.416060+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:39:50.416060+0000","last_undegraded":"2024-02-10T08:39:50.416060+0000","last_fullsized":"2024-02-10T08:39:50.416060+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"228'68","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000339+0000","last_change":"2024-02-10T08:39:32.968736+0000","last_active":"2024-02-10T08:39:50.000339+0000","last_peered":"2024-02-10T08:39:50.000339+0000","last_clean":"2024-02-10T08:39:50.000339+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:39:50.000339+0000","last_undegraded":"2024-02-10T08:39:50.000339+0000","last_fullsized":"2024-02-10T08:39:50.000339+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"214'68","reported_seq":282,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002699+0000","last_change":"2024-02-10T08:39:32.970019+0000","last_active":"2024-02-10T08:39:50.002699+0000","last_peered":"2024-02-10T08:39:50.002699+0000","last_clean":"2024-02-10T08:39:50.002699+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:39:50.002699+0000","last_undegraded":"2024-02-10T08:39:50.002699+0000","last_fullsized":"2024-02-10T08:39:50.002699+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"208'34","reported_seq":271,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002526+0000","last_change":"2024-02-10T08:39:32.971984+0000","last_active":"2024-02-10T08:39:50.002526+0000","last_peered":"2024-02-10T08:39:50.002526+0000","last_clean":"2024-02-10T08:39:50.002526+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:39:50.002526+0000","last_undegraded":"2024-02-10T08:39:50.002526+0000","last_fullsized":"2024-02-10T08:39:50.002526+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"224'34","reported_seq":243,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000103+0000","last_change":"2024-02-10T08:39:32.967110+0000","last_active":"2024-02-10T08:39:50.000103+0000","last_peered":"2024-02-10T08:39:50.000103+0000","last_clean":"2024-02-10T08:39:50.000103+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:39:50.000103+0000","last_undegraded":"2024-02-10T08:39:50.000103+0000","last_fullsized":"2024-02-10T08:39:50.000103+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"200'34","reported_seq":276,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006472+0000","last_change":"2024-02-10T08:39:33.014676+0000","last_active":"2024-02-10T08:39:50.006472+0000","last_peered":"2024-02-10T08:39:50.006472+0000","last_clean":"2024-02-10T08:39:50.006472+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:39:50.006472+0000","last_undegraded":"2024-02-10T08:39:50.006472+0000","last_fullsized":"2024-02-10T08:39:50.006472+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'34","last_scrub_stamp":"2024-02-10T08:39:04.785239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.785239+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":234,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004607+0000","last_change":"2024-02-10T08:39:38.041298+0000","last_active":"2024-02-10T08:39:50.004607+0000","last_peered":"2024-02-10T08:39:50.004607+0000","last_clean":"2024-02-10T08:39:50.004607+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:39:50.004607+0000","last_undegraded":"2024-02-10T08:39:50.004607+0000","last_fullsized":"2024-02-10T08:39:50.004607+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:39:38.041185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.041185+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"202'17","reported_seq":232,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414075+0000","last_change":"2024-02-10T08:39:33.016451+0000","last_active":"2024-02-10T08:39:50.414075+0000","last_peered":"2024-02-10T08:39:50.414075+0000","last_clean":"2024-02-10T08:39:50.414075+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:39:50.414075+0000","last_undegraded":"2024-02-10T08:39:50.414075+0000","last_fullsized":"2024-02-10T08:39:50.414075+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"222'34","reported_seq":263,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009194+0000","last_change":"2024-02-10T08:39:32.974056+0000","last_active":"2024-02-10T08:39:50.009194+0000","last_peered":"2024-02-10T08:39:50.009194+0000","last_clean":"2024-02-10T08:39:50.009194+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:39:50.009194+0000","last_undegraded":"2024-02-10T08:39:50.009194+0000","last_fullsized":"2024-02-10T08:39:50.009194+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":367,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005833+0000","last_change":"2024-02-10T08:38:21.488646+0000","last_active":"2024-02-10T08:39:50.005833+0000","last_peered":"2024-02-10T08:39:50.005833+0000","last_clean":"2024-02-10T08:39:50.005833+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:39:50.005833+0000","last_undegraded":"2024-02-10T08:39:50.005833+0000","last_fullsized":"2024-02-10T08:39:50.005833+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:38:21.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:21.488563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":294,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005481+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:39:50.005481+0000","last_peered":"2024-02-10T08:39:50.005481+0000","last_clean":"2024-02-10T08:39:50.005481+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:39:50.005481+0000","last_undegraded":"2024-02-10T08:39:50.005481+0000","last_fullsized":"2024-02-10T08:39:50.005481+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":260,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009146+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:39:50.009146+0000","last_peered":"2024-02-10T08:39:50.009146+0000","last_clean":"2024-02-10T08:39:50.009146+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:39:50.009146+0000","last_undegraded":"2024-02-10T08:39:50.009146+0000","last_fullsized":"2024-02-10T08:39:50.009146+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":267,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007964+0000","last_change":"2024-02-10T08:37:44.597397+0000","last_active":"2024-02-10T08:39:50.007964+0000","last_peered":"2024-02-10T08:39:50.007964+0000","last_clean":"2024-02-10T08:39:50.007964+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:39:50.007964+0000","last_undegraded":"2024-02-10T08:39:50.007964+0000","last_fullsized":"2024-02-10T08:39:50.007964+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.597335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.597335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009511+0000","last_change":"2024-02-10T08:38:33.920702+0000","last_active":"2024-02-10T08:39:50.009511+0000","last_peered":"2024-02-10T08:39:50.009511+0000","last_clean":"2024-02-10T08:39:50.009511+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:39:50.009511+0000","last_undegraded":"2024-02-10T08:39:50.009511+0000","last_fullsized":"2024-02-10T08:39:50.009511+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.920630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.920630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":237,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008970+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:39:50.008970+0000","last_peered":"2024-02-10T08:39:50.008970+0000","last_clean":"2024-02-10T08:39:50.008970+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:39:50.008970+0000","last_undegraded":"2024-02-10T08:39:50.008970+0000","last_fullsized":"2024-02-10T08:39:50.008970+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004936+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:39:50.004936+0000","last_peered":"2024-02-10T08:39:50.004936+0000","last_clean":"2024-02-10T08:39:50.004936+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:39:50.004936+0000","last_undegraded":"2024-02-10T08:39:50.004936+0000","last_fullsized":"2024-02-10T08:39:50.004936+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":279,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007588+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:39:50.007588+0000","last_peered":"2024-02-10T08:39:50.007588+0000","last_clean":"2024-02-10T08:39:50.007588+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:39:50.007588+0000","last_undegraded":"2024-02-10T08:39:50.007588+0000","last_fullsized":"2024-02-10T08:39:50.007588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005527+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:39:50.005527+0000","last_peered":"2024-02-10T08:39:50.005527+0000","last_clean":"2024-02-10T08:39:50.005527+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:39:50.005527+0000","last_undegraded":"2024-02-10T08:39:50.005527+0000","last_fullsized":"2024-02-10T08:39:50.005527+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":207,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009074+0000","last_change":"2024-02-10T08:38:26.677780+0000","last_active":"2024-02-10T08:39:50.009074+0000","last_peered":"2024-02-10T08:39:50.009074+0000","last_clean":"2024-02-10T08:39:50.009074+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:39:50.009074+0000","last_undegraded":"2024-02-10T08:39:50.009074+0000","last_fullsized":"2024-02-10T08:39:50.009074+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.677701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.677701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":293,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.119045+0000","last_change":"2024-02-10T08:37:39.547805+0000","last_active":"2024-02-10T08:39:50.119045+0000","last_peered":"2024-02-10T08:39:50.119045+0000","last_clean":"2024-02-10T08:39:50.119045+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:39:50.119045+0000","last_undegraded":"2024-02-10T08:39:50.119045+0000","last_fullsized":"2024-02-10T08:39:50.119045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:37:39.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.547736+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009768+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:39:50.009768+0000","last_peered":"2024-02-10T08:39:50.009768+0000","last_clean":"2024-02-10T08:39:50.009768+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:39:50.009768+0000","last_undegraded":"2024-02-10T08:39:50.009768+0000","last_fullsized":"2024-02-10T08:39:50.009768+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":283,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008375+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:39:50.008375+0000","last_peered":"2024-02-10T08:39:50.008375+0000","last_clean":"2024-02-10T08:39:50.008375+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:39:50.008375+0000","last_undegraded":"2024-02-10T08:39:50.008375+0000","last_fullsized":"2024-02-10T08:39:50.008375+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005696+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:39:50.005696+0000","last_peered":"2024-02-10T08:39:50.005696+0000","last_clean":"2024-02-10T08:39:50.005696+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:39:50.005696+0000","last_undegraded":"2024-02-10T08:39:50.005696+0000","last_fullsized":"2024-02-10T08:39:50.005696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006483+0000","last_change":"2024-02-10T08:39:44.585775+0000","last_active":"2024-02-10T08:39:50.006483+0000","last_peered":"2024-02-10T08:39:50.006483+0000","last_clean":"2024-02-10T08:39:50.006483+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:39:50.006483+0000","last_undegraded":"2024-02-10T08:39:50.006483+0000","last_fullsized":"2024-02-10T08:39:50.006483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:44.585674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.585674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":291,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001323+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:39:50.001323+0000","last_peered":"2024-02-10T08:39:50.001323+0000","last_clean":"2024-02-10T08:39:50.001323+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:39:50.001323+0000","last_undegraded":"2024-02-10T08:39:50.001323+0000","last_fullsized":"2024-02-10T08:39:50.001323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":99,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414565+0000","last_change":"2024-02-10T08:38:22.013062+0000","last_active":"2024-02-10T08:39:50.414565+0000","last_peered":"2024-02-10T08:39:50.414565+0000","last_clean":"2024-02-10T08:39:50.414565+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:39:50.414565+0000","last_undegraded":"2024-02-10T08:39:50.414565+0000","last_fullsized":"2024-02-10T08:39:50.414565+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":283,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004026+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:39:50.004026+0000","last_peered":"2024-02-10T08:39:50.004026+0000","last_clean":"2024-02-10T08:39:50.004026+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:39:50.004026+0000","last_undegraded":"2024-02-10T08:39:50.004026+0000","last_fullsized":"2024-02-10T08:39:50.004026+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009622+0000","last_change":"2024-02-10T08:36:35.691710+0000","last_active":"2024-02-10T08:39:50.009622+0000","last_peered":"2024-02-10T08:39:50.009622+0000","last_clean":"2024-02-10T08:39:50.009622+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:39:50.009622+0000","last_undegraded":"2024-02-10T08:39:50.009622+0000","last_fullsized":"2024-02-10T08:39:50.009622+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.691633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000200+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:39:50.000200+0000","last_peered":"2024-02-10T08:39:50.000200+0000","last_clean":"2024-02-10T08:39:50.000200+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:39:50.000200+0000","last_undegraded":"2024-02-10T08:39:50.000200+0000","last_fullsized":"2024-02-10T08:39:50.000200+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":278,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000980+0000","last_change":"2024-02-10T08:38:17.727465+0000","last_active":"2024-02-10T08:39:50.000980+0000","last_peered":"2024-02-10T08:39:50.000980+0000","last_clean":"2024-02-10T08:39:50.000980+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:39:50.000980+0000","last_undegraded":"2024-02-10T08:39:50.000980+0000","last_fullsized":"2024-02-10T08:39:50.000980+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:17.727387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:17.727387+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002992+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:39:50.002992+0000","last_peered":"2024-02-10T08:39:50.002992+0000","last_clean":"2024-02-10T08:39:50.002992+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:39:50.002992+0000","last_undegraded":"2024-02-10T08:39:50.002992+0000","last_fullsized":"2024-02-10T08:39:50.002992+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":362,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008369+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:39:50.008369+0000","last_peered":"2024-02-10T08:39:50.008369+0000","last_clean":"2024-02-10T08:39:50.008369+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:39:50.008369+0000","last_undegraded":"2024-02-10T08:39:50.008369+0000","last_fullsized":"2024-02-10T08:39:50.008369+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001596+0000","last_change":"2024-02-10T08:38:25.851680+0000","last_active":"2024-02-10T08:39:50.001596+0000","last_peered":"2024-02-10T08:39:50.001596+0000","last_clean":"2024-02-10T08:39:50.001596+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:39:50.001596+0000","last_undegraded":"2024-02-10T08:39:50.001596+0000","last_fullsized":"2024-02-10T08:39:50.001596+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.851576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.851576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":173,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414179+0000","last_change":"2024-02-10T08:37:10.263630+0000","last_active":"2024-02-10T08:39:50.414179+0000","last_peered":"2024-02-10T08:39:50.414179+0000","last_clean":"2024-02-10T08:39:50.414179+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:39:50.414179+0000","last_undegraded":"2024-02-10T08:39:50.414179+0000","last_fullsized":"2024-02-10T08:39:50.414179+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":283,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005046+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:39:50.005046+0000","last_peered":"2024-02-10T08:39:50.005046+0000","last_clean":"2024-02-10T08:39:50.005046+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:39:50.005046+0000","last_undegraded":"2024-02-10T08:39:50.005046+0000","last_fullsized":"2024-02-10T08:39:50.005046+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002945+0000","last_change":"2024-02-10T08:38:18.008913+0000","last_active":"2024-02-10T08:39:50.002945+0000","last_peered":"2024-02-10T08:39:50.002945+0000","last_clean":"2024-02-10T08:39:50.002945+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:39:50.002945+0000","last_undegraded":"2024-02-10T08:39:50.002945+0000","last_fullsized":"2024-02-10T08:39:50.002945+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:18.008846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.008846+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":387,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001358+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:39:50.001358+0000","last_peered":"2024-02-10T08:39:50.001358+0000","last_clean":"2024-02-10T08:39:50.001358+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:39:50.001358+0000","last_undegraded":"2024-02-10T08:39:50.001358+0000","last_fullsized":"2024-02-10T08:39:50.001358+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":262,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008454+0000","last_change":"2024-02-10T08:37:24.680471+0000","last_active":"2024-02-10T08:39:50.008454+0000","last_peered":"2024-02-10T08:39:50.008454+0000","last_clean":"2024-02-10T08:39:50.008454+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:39:50.008454+0000","last_undegraded":"2024-02-10T08:39:50.008454+0000","last_fullsized":"2024-02-10T08:39:50.008454+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:24.680407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:24.680407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":207,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006418+0000","last_change":"2024-02-10T08:39:05.788196+0000","last_active":"2024-02-10T08:39:50.006418+0000","last_peered":"2024-02-10T08:39:50.006418+0000","last_clean":"2024-02-10T08:39:50.006418+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:39:50.006418+0000","last_undegraded":"2024-02-10T08:39:50.006418+0000","last_fullsized":"2024-02-10T08:39:50.006418+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.788118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.788118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":282,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001205+0000","last_change":"2024-02-10T08:38:59.745198+0000","last_active":"2024-02-10T08:39:50.001205+0000","last_peered":"2024-02-10T08:39:50.001205+0000","last_clean":"2024-02-10T08:39:50.001205+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:39:50.001205+0000","last_undegraded":"2024-02-10T08:39:50.001205+0000","last_fullsized":"2024-02-10T08:39:50.001205+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:38:59.745114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.745114+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007040+0000","last_change":"2024-02-10T08:38:33.363322+0000","last_active":"2024-02-10T08:39:50.007040+0000","last_peered":"2024-02-10T08:39:50.007040+0000","last_clean":"2024-02-10T08:39:50.007040+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:39:50.007040+0000","last_undegraded":"2024-02-10T08:39:50.007040+0000","last_fullsized":"2024-02-10T08:39:50.007040+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.363195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":319,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008746+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:39:50.008746+0000","last_peered":"2024-02-10T08:39:50.008746+0000","last_clean":"2024-02-10T08:39:50.008746+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:39:50.008746+0000","last_undegraded":"2024-02-10T08:39:50.008746+0000","last_fullsized":"2024-02-10T08:39:50.008746+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":253,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.417071+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:39:50.417071+0000","last_peered":"2024-02-10T08:39:50.417071+0000","last_clean":"2024-02-10T08:39:50.417071+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:39:50.417071+0000","last_undegraded":"2024-02-10T08:39:50.417071+0000","last_fullsized":"2024-02-10T08:39:50.417071+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.415955+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:39:50.415955+0000","last_peered":"2024-02-10T08:39:50.415955+0000","last_clean":"2024-02-10T08:39:50.415955+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:39:50.415955+0000","last_undegraded":"2024-02-10T08:39:50.415955+0000","last_fullsized":"2024-02-10T08:39:50.415955+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":349,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.415371+0000","last_change":"2024-02-10T08:38:18.475575+0000","last_active":"2024-02-10T08:39:50.415371+0000","last_peered":"2024-02-10T08:39:50.415371+0000","last_clean":"2024-02-10T08:39:50.415371+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:39:50.415371+0000","last_undegraded":"2024-02-10T08:39:50.415371+0000","last_fullsized":"2024-02-10T08:39:50.415371+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:38:18.475509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.475509+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":284,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000477+0000","last_change":"2024-02-10T08:38:31.840285+0000","last_active":"2024-02-10T08:39:50.000477+0000","last_peered":"2024-02-10T08:39:50.000477+0000","last_clean":"2024-02-10T08:39:50.000477+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:39:50.000477+0000","last_undegraded":"2024-02-10T08:39:50.000477+0000","last_fullsized":"2024-02-10T08:39:50.000477+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:38:31.840218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.840218+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120861+0000","last_change":"2024-02-10T08:39:48.521060+0000","last_active":"2024-02-10T08:39:50.120861+0000","last_peered":"2024-02-10T08:39:50.120861+0000","last_clean":"2024-02-10T08:39:50.120861+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:39:50.120861+0000","last_undegraded":"2024-02-10T08:39:50.120861+0000","last_fullsized":"2024-02-10T08:39:50.120861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:48.520941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:48.520941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005782+0000","last_change":"2024-02-10T08:37:09.215324+0000","last_active":"2024-02-10T08:39:50.005782+0000","last_peered":"2024-02-10T08:39:50.005782+0000","last_clean":"2024-02-10T08:39:50.005782+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:39:50.005782+0000","last_undegraded":"2024-02-10T08:39:50.005782+0000","last_fullsized":"2024-02-10T08:39:50.005782+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:09.215257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:09.215257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005089+0000","last_change":"2024-02-10T08:39:12.085361+0000","last_active":"2024-02-10T08:39:50.005089+0000","last_peered":"2024-02-10T08:39:50.005089+0000","last_clean":"2024-02-10T08:39:50.005089+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:39:50.005089+0000","last_undegraded":"2024-02-10T08:39:50.005089+0000","last_fullsized":"2024-02-10T08:39:50.005089+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.085261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.085261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":256,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000668+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:39:50.000668+0000","last_peered":"2024-02-10T08:39:50.000668+0000","last_clean":"2024-02-10T08:39:50.000668+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:39:50.000668+0000","last_undegraded":"2024-02-10T08:39:50.000668+0000","last_fullsized":"2024-02-10T08:39:50.000668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":324,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413307+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:39:50.413307+0000","last_peered":"2024-02-10T08:39:50.413307+0000","last_clean":"2024-02-10T08:39:50.413307+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:39:50.413307+0000","last_undegraded":"2024-02-10T08:39:50.413307+0000","last_fullsized":"2024-02-10T08:39:50.413307+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":272,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004825+0000","last_change":"2024-02-10T08:38:57.920239+0000","last_active":"2024-02-10T08:39:50.004825+0000","last_peered":"2024-02-10T08:39:50.004825+0000","last_clean":"2024-02-10T08:39:50.004825+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:39:50.004825+0000","last_undegraded":"2024-02-10T08:39:50.004825+0000","last_fullsized":"2024-02-10T08:39:50.004825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:57.920144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.920144+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001093+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:39:50.001093+0000","last_peered":"2024-02-10T08:39:50.001093+0000","last_clean":"2024-02-10T08:39:50.001093+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:39:50.001093+0000","last_undegraded":"2024-02-10T08:39:50.001093+0000","last_fullsized":"2024-02-10T08:39:50.001093+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413542+0000","last_change":"2024-02-10T08:39:08.629231+0000","last_active":"2024-02-10T08:39:50.413542+0000","last_peered":"2024-02-10T08:39:50.413542+0000","last_clean":"2024-02-10T08:39:50.413542+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:39:50.413542+0000","last_undegraded":"2024-02-10T08:39:50.413542+0000","last_fullsized":"2024-02-10T08:39:50.413542+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.119043+0000","last_change":"2024-02-10T08:37:43.520384+0000","last_active":"2024-02-10T08:39:50.119043+0000","last_peered":"2024-02-10T08:39:50.119043+0000","last_clean":"2024-02-10T08:39:50.119043+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:39:50.119043+0000","last_undegraded":"2024-02-10T08:39:50.119043+0000","last_fullsized":"2024-02-10T08:39:50.119043+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.520307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.520307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":243,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006575+0000","last_change":"2024-02-10T08:38:18.588026+0000","last_active":"2024-02-10T08:39:50.006575+0000","last_peered":"2024-02-10T08:39:50.006575+0000","last_clean":"2024-02-10T08:39:50.006575+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:39:50.006575+0000","last_undegraded":"2024-02-10T08:39:50.006575+0000","last_fullsized":"2024-02-10T08:39:50.006575+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:38:18.587949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.587949+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006518+0000","last_change":"2024-02-10T08:38:13.655575+0000","last_active":"2024-02-10T08:39:50.006518+0000","last_peered":"2024-02-10T08:39:50.006518+0000","last_clean":"2024-02-10T08:39:50.006518+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:39:50.006518+0000","last_undegraded":"2024-02-10T08:39:50.006518+0000","last_fullsized":"2024-02-10T08:39:50.006518+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.655504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003243+0000","last_change":"2024-02-10T08:38:26.946622+0000","last_active":"2024-02-10T08:39:50.003243+0000","last_peered":"2024-02-10T08:39:50.003243+0000","last_clean":"2024-02-10T08:39:50.003243+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:39:50.003243+0000","last_undegraded":"2024-02-10T08:39:50.003243+0000","last_fullsized":"2024-02-10T08:39:50.003243+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.946531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.946531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":217,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009381+0000","last_change":"2024-02-10T08:38:25.903816+0000","last_active":"2024-02-10T08:39:50.009381+0000","last_peered":"2024-02-10T08:39:50.009381+0000","last_clean":"2024-02-10T08:39:50.009381+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:39:50.009381+0000","last_undegraded":"2024-02-10T08:39:50.009381+0000","last_fullsized":"2024-02-10T08:39:50.009381+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.903753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.903753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":287,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008746+0000","last_change":"2024-02-10T08:38:31.745438+0000","last_active":"2024-02-10T08:39:50.008746+0000","last_peered":"2024-02-10T08:39:50.008746+0000","last_clean":"2024-02-10T08:39:50.008746+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:39:50.008746+0000","last_undegraded":"2024-02-10T08:39:50.008746+0000","last_fullsized":"2024-02-10T08:39:50.008746+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.745362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.745362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":222,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005000+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:39:50.005000+0000","last_peered":"2024-02-10T08:39:50.005000+0000","last_clean":"2024-02-10T08:39:50.005000+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:39:50.005000+0000","last_undegraded":"2024-02-10T08:39:50.005000+0000","last_fullsized":"2024-02-10T08:39:50.005000+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005807+0000","last_change":"2024-02-10T08:37:21.277884+0000","last_active":"2024-02-10T08:39:50.005807+0000","last_peered":"2024-02-10T08:39:50.005807+0000","last_clean":"2024-02-10T08:39:50.005807+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:39:50.005807+0000","last_undegraded":"2024-02-10T08:39:50.005807+0000","last_fullsized":"2024-02-10T08:39:50.005807+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.277786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.277786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000301+0000","last_change":"2024-02-10T08:38:18.758911+0000","last_active":"2024-02-10T08:39:50.000301+0000","last_peered":"2024-02-10T08:39:50.000301+0000","last_clean":"2024-02-10T08:39:50.000301+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:39:50.000301+0000","last_undegraded":"2024-02-10T08:39:50.000301+0000","last_fullsized":"2024-02-10T08:39:50.000301+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.758843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.758843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":260,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004110+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:39:50.004110+0000","last_peered":"2024-02-10T08:39:50.004110+0000","last_clean":"2024-02-10T08:39:50.004110+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:39:50.004110+0000","last_undegraded":"2024-02-10T08:39:50.004110+0000","last_fullsized":"2024-02-10T08:39:50.004110+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":216,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.118918+0000","last_change":"2024-02-10T08:39:06.770922+0000","last_active":"2024-02-10T08:39:50.118918+0000","last_peered":"2024-02-10T08:39:50.118918+0000","last_clean":"2024-02-10T08:39:50.118918+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:39:50.118918+0000","last_undegraded":"2024-02-10T08:39:50.118918+0000","last_fullsized":"2024-02-10T08:39:50.118918+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.770850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.770850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":261,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006121+0000","last_change":"2024-02-10T08:39:38.645588+0000","last_active":"2024-02-10T08:39:50.006121+0000","last_peered":"2024-02-10T08:39:50.006121+0000","last_clean":"2024-02-10T08:39:50.006121+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:39:50.006121+0000","last_undegraded":"2024-02-10T08:39:50.006121+0000","last_fullsized":"2024-02-10T08:39:50.006121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.645527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.645527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":284,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120504+0000","last_change":"2024-02-10T08:37:40.592997+0000","last_active":"2024-02-10T08:39:50.120504+0000","last_peered":"2024-02-10T08:39:50.120504+0000","last_clean":"2024-02-10T08:39:50.120504+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:39:50.120504+0000","last_undegraded":"2024-02-10T08:39:50.120504+0000","last_fullsized":"2024-02-10T08:39:50.120504+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:37:40.592930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.592930+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":282,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006940+0000","last_change":"2024-02-10T08:39:36.639528+0000","last_active":"2024-02-10T08:39:50.006940+0000","last_peered":"2024-02-10T08:39:50.006940+0000","last_clean":"2024-02-10T08:39:50.006940+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:39:50.006940+0000","last_undegraded":"2024-02-10T08:39:50.006940+0000","last_fullsized":"2024-02-10T08:39:50.006940+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.639461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.639461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006564+0000","last_change":"2024-02-10T08:38:25.633733+0000","last_active":"2024-02-10T08:39:50.006564+0000","last_peered":"2024-02-10T08:39:50.006564+0000","last_clean":"2024-02-10T08:39:50.006564+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:39:50.006564+0000","last_undegraded":"2024-02-10T08:39:50.006564+0000","last_fullsized":"2024-02-10T08:39:50.006564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.633608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.633608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413378+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:39:50.413378+0000","last_peered":"2024-02-10T08:39:50.413378+0000","last_clean":"2024-02-10T08:39:50.413378+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:39:50.413378+0000","last_undegraded":"2024-02-10T08:39:50.413378+0000","last_fullsized":"2024-02-10T08:39:50.413378+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":258,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009010+0000","last_change":"2024-02-10T08:39:00.751654+0000","last_active":"2024-02-10T08:39:50.009010+0000","last_peered":"2024-02-10T08:39:50.009010+0000","last_clean":"2024-02-10T08:39:50.009010+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:39:50.009010+0000","last_undegraded":"2024-02-10T08:39:50.009010+0000","last_fullsized":"2024-02-10T08:39:50.009010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:00.751584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.751584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":218,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006272+0000","last_change":"2024-02-10T08:39:11.660136+0000","last_active":"2024-02-10T08:39:50.006272+0000","last_peered":"2024-02-10T08:39:50.006272+0000","last_clean":"2024-02-10T08:39:50.006272+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:39:50.006272+0000","last_undegraded":"2024-02-10T08:39:50.006272+0000","last_fullsized":"2024-02-10T08:39:50.006272+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:11.660049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.660049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009340+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:39:50.009340+0000","last_peered":"2024-02-10T08:39:50.009340+0000","last_clean":"2024-02-10T08:39:50.009340+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:39:50.009340+0000","last_undegraded":"2024-02-10T08:39:50.009340+0000","last_fullsized":"2024-02-10T08:39:50.009340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":311,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004953+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:39:50.004953+0000","last_peered":"2024-02-10T08:39:50.004953+0000","last_clean":"2024-02-10T08:39:50.004953+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:39:50.004953+0000","last_undegraded":"2024-02-10T08:39:50.004953+0000","last_fullsized":"2024-02-10T08:39:50.004953+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":276,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007492+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:39:50.007492+0000","last_peered":"2024-02-10T08:39:50.007492+0000","last_clean":"2024-02-10T08:39:50.007492+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:39:50.007492+0000","last_undegraded":"2024-02-10T08:39:50.007492+0000","last_fullsized":"2024-02-10T08:39:50.007492+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":655,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.785694+0000","last_change":"2024-02-10T08:39:05.948676+0000","last_active":"2024-02-10T08:39:50.785694+0000","last_peered":"2024-02-10T08:39:50.785694+0000","last_clean":"2024-02-10T08:39:50.785694+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:39:50.785694+0000","last_undegraded":"2024-02-10T08:39:50.785694+0000","last_fullsized":"2024-02-10T08:39:50.785694+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:39:05.948579+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.948579+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":386,"num_read_kb":381,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006438+0000","last_change":"2024-02-10T08:39:03.251683+0000","last_active":"2024-02-10T08:39:50.006438+0000","last_peered":"2024-02-10T08:39:50.006438+0000","last_clean":"2024-02-10T08:39:50.006438+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:39:50.006438+0000","last_undegraded":"2024-02-10T08:39:50.006438+0000","last_fullsized":"2024-02-10T08:39:50.006438+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.251585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.251585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":212,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004072+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:39:50.004072+0000","last_peered":"2024-02-10T08:39:50.004072+0000","last_clean":"2024-02-10T08:39:50.004072+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:39:50.004072+0000","last_undegraded":"2024-02-10T08:39:50.004072+0000","last_fullsized":"2024-02-10T08:39:50.004072+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004641+0000","last_change":"2024-02-10T08:39:06.965218+0000","last_active":"2024-02-10T08:39:50.004641+0000","last_peered":"2024-02-10T08:39:50.004641+0000","last_clean":"2024-02-10T08:39:50.004641+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:39:50.004641+0000","last_undegraded":"2024-02-10T08:39:50.004641+0000","last_fullsized":"2024-02-10T08:39:50.004641+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.965152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.965152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":327,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006817+0000","last_change":"2024-02-10T08:38:19.613486+0000","last_active":"2024-02-10T08:39:50.006817+0000","last_peered":"2024-02-10T08:39:50.006817+0000","last_clean":"2024-02-10T08:39:50.006817+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:39:50.006817+0000","last_undegraded":"2024-02-10T08:39:50.006817+0000","last_fullsized":"2024-02-10T08:39:50.006817+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:38:19.613422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.613422+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008056+0000","last_change":"2024-02-10T08:37:41.598616+0000","last_active":"2024-02-10T08:39:50.008056+0000","last_peered":"2024-02-10T08:39:50.008056+0000","last_clean":"2024-02-10T08:39:50.008056+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:39:50.008056+0000","last_undegraded":"2024-02-10T08:39:50.008056+0000","last_fullsized":"2024-02-10T08:39:50.008056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.598541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.598541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":253,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416646+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:39:50.416646+0000","last_peered":"2024-02-10T08:39:50.416646+0000","last_clean":"2024-02-10T08:39:50.416646+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:39:50.416646+0000","last_undegraded":"2024-02-10T08:39:50.416646+0000","last_fullsized":"2024-02-10T08:39:50.416646+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":217,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006537+0000","last_change":"2024-02-10T08:38:57.168613+0000","last_active":"2024-02-10T08:39:50.006537+0000","last_peered":"2024-02-10T08:39:50.006537+0000","last_clean":"2024-02-10T08:39:50.006537+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:39:50.006537+0000","last_undegraded":"2024-02-10T08:39:50.006537+0000","last_fullsized":"2024-02-10T08:39:50.006537+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:57.168523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.168523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":232,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.119721+0000","last_change":"2024-02-10T08:37:41.604353+0000","last_active":"2024-02-10T08:39:50.119721+0000","last_peered":"2024-02-10T08:39:50.119721+0000","last_clean":"2024-02-10T08:39:50.119721+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:39:50.119721+0000","last_undegraded":"2024-02-10T08:39:50.119721+0000","last_fullsized":"2024-02-10T08:39:50.119721+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:37:41.604248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.604248+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":283,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003887+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:39:50.003887+0000","last_peered":"2024-02-10T08:39:50.003887+0000","last_clean":"2024-02-10T08:39:50.003887+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:39:50.003887+0000","last_undegraded":"2024-02-10T08:39:50.003887+0000","last_fullsized":"2024-02-10T08:39:50.003887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003829+0000","last_change":"2024-02-10T08:38:59.955336+0000","last_active":"2024-02-10T08:39:50.003829+0000","last_peered":"2024-02-10T08:39:50.003829+0000","last_clean":"2024-02-10T08:39:50.003829+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:39:50.003829+0000","last_undegraded":"2024-02-10T08:39:50.003829+0000","last_fullsized":"2024-02-10T08:39:50.003829+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.955272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.955272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414046+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:39:50.414046+0000","last_peered":"2024-02-10T08:39:50.414046+0000","last_clean":"2024-02-10T08:39:50.414046+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:39:50.414046+0000","last_undegraded":"2024-02-10T08:39:50.414046+0000","last_fullsized":"2024-02-10T08:39:50.414046+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416505+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:39:50.416505+0000","last_peered":"2024-02-10T08:39:50.416505+0000","last_clean":"2024-02-10T08:39:50.416505+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:39:50.416505+0000","last_undegraded":"2024-02-10T08:39:50.416505+0000","last_fullsized":"2024-02-10T08:39:50.416505+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":288,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006818+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:39:50.006818+0000","last_peered":"2024-02-10T08:39:50.006818+0000","last_clean":"2024-02-10T08:39:50.006818+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:39:50.006818+0000","last_undegraded":"2024-02-10T08:39:50.006818+0000","last_fullsized":"2024-02-10T08:39:50.006818+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":291,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000569+0000","last_change":"2024-02-10T08:38:32.886182+0000","last_active":"2024-02-10T08:39:50.000569+0000","last_peered":"2024-02-10T08:39:50.000569+0000","last_clean":"2024-02-10T08:39:50.000569+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:39:50.000569+0000","last_undegraded":"2024-02-10T08:39:50.000569+0000","last_fullsized":"2024-02-10T08:39:50.000569+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:38:32.886090+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.886090+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":288,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001307+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:39:50.001307+0000","last_peered":"2024-02-10T08:39:50.001307+0000","last_clean":"2024-02-10T08:39:50.001307+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:39:50.001307+0000","last_undegraded":"2024-02-10T08:39:50.001307+0000","last_fullsized":"2024-02-10T08:39:50.001307+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":258,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416465+0000","last_change":"2024-02-10T08:38:22.407565+0000","last_active":"2024-02-10T08:39:50.416465+0000","last_peered":"2024-02-10T08:39:50.416465+0000","last_clean":"2024-02-10T08:39:50.416465+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:39:50.416465+0000","last_undegraded":"2024-02-10T08:39:50.416465+0000","last_fullsized":"2024-02-10T08:39:50.416465+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:22.407476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:22.407476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008331+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:39:50.008331+0000","last_peered":"2024-02-10T08:39:50.008331+0000","last_clean":"2024-02-10T08:39:50.008331+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:39:50.008331+0000","last_undegraded":"2024-02-10T08:39:50.008331+0000","last_fullsized":"2024-02-10T08:39:50.008331+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":244,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413945+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:39:50.413945+0000","last_peered":"2024-02-10T08:39:50.413945+0000","last_clean":"2024-02-10T08:39:50.413945+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:39:50.413945+0000","last_undegraded":"2024-02-10T08:39:50.413945+0000","last_fullsized":"2024-02-10T08:39:50.413945+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006785+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:39:50.006785+0000","last_peered":"2024-02-10T08:39:50.006785+0000","last_clean":"2024-02-10T08:39:50.006785+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:39:50.006785+0000","last_undegraded":"2024-02-10T08:39:50.006785+0000","last_fullsized":"2024-02-10T08:39:50.006785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009653+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:39:50.009653+0000","last_peered":"2024-02-10T08:39:50.009653+0000","last_clean":"2024-02-10T08:39:50.009653+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:39:50.009653+0000","last_undegraded":"2024-02-10T08:39:50.009653+0000","last_fullsized":"2024-02-10T08:39:50.009653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000754+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:39:50.000754+0000","last_peered":"2024-02-10T08:39:50.000754+0000","last_clean":"2024-02-10T08:39:50.000754+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:39:50.000754+0000","last_undegraded":"2024-02-10T08:39:50.000754+0000","last_fullsized":"2024-02-10T08:39:50.000754+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":266,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416492+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:39:50.416492+0000","last_peered":"2024-02-10T08:39:50.416492+0000","last_clean":"2024-02-10T08:39:50.416492+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:39:50.416492+0000","last_undegraded":"2024-02-10T08:39:50.416492+0000","last_fullsized":"2024-02-10T08:39:50.416492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.118839+0000","last_change":"2024-02-10T08:38:30.291597+0000","last_active":"2024-02-10T08:39:50.118839+0000","last_peered":"2024-02-10T08:39:50.118839+0000","last_clean":"2024-02-10T08:39:50.118839+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:39:50.118839+0000","last_undegraded":"2024-02-10T08:39:50.118839+0000","last_fullsized":"2024-02-10T08:39:50.118839+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.291538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.291538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006146+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:39:50.006146+0000","last_peered":"2024-02-10T08:39:50.006146+0000","last_clean":"2024-02-10T08:39:50.006146+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:39:50.006146+0000","last_undegraded":"2024-02-10T08:39:50.006146+0000","last_fullsized":"2024-02-10T08:39:50.006146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":207,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006478+0000","last_change":"2024-02-10T08:38:32.745050+0000","last_active":"2024-02-10T08:39:50.006478+0000","last_peered":"2024-02-10T08:39:50.006478+0000","last_clean":"2024-02-10T08:39:50.006478+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:39:50.006478+0000","last_undegraded":"2024-02-10T08:39:50.006478+0000","last_fullsized":"2024-02-10T08:39:50.006478+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.744972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.744972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":277,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.417209+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:39:50.417209+0000","last_peered":"2024-02-10T08:39:50.417209+0000","last_clean":"2024-02-10T08:39:50.417209+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:39:50.417209+0000","last_undegraded":"2024-02-10T08:39:50.417209+0000","last_fullsized":"2024-02-10T08:39:50.417209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008023+0000","last_change":"2024-02-10T08:37:46.576724+0000","last_active":"2024-02-10T08:39:50.008023+0000","last_peered":"2024-02-10T08:39:50.008023+0000","last_clean":"2024-02-10T08:39:50.008023+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:39:50.008023+0000","last_undegraded":"2024-02-10T08:39:50.008023+0000","last_fullsized":"2024-02-10T08:39:50.008023+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.576661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.576661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":359,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008358+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:39:50.008358+0000","last_peered":"2024-02-10T08:39:50.008358+0000","last_clean":"2024-02-10T08:39:50.008358+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:39:50.008358+0000","last_undegraded":"2024-02-10T08:39:50.008358+0000","last_fullsized":"2024-02-10T08:39:50.008358+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006570+0000","last_change":"2024-02-10T08:39:43.632113+0000","last_active":"2024-02-10T08:39:50.006570+0000","last_peered":"2024-02-10T08:39:50.006570+0000","last_clean":"2024-02-10T08:39:50.006570+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:39:50.006570+0000","last_undegraded":"2024-02-10T08:39:50.006570+0000","last_fullsized":"2024-02-10T08:39:50.006570+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:43.632048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.632048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":177,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.118441+0000","last_change":"2024-02-10T08:39:01.740757+0000","last_active":"2024-02-10T08:39:50.118441+0000","last_peered":"2024-02-10T08:39:50.118441+0000","last_clean":"2024-02-10T08:39:50.118441+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:39:50.118441+0000","last_undegraded":"2024-02-10T08:39:50.118441+0000","last_fullsized":"2024-02-10T08:39:50.118441+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:01.740686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:01.740686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":278,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.008355+0000","last_change":"2024-02-10T08:39:46.641751+0000","last_active":"2024-02-10T08:39:50.008355+0000","last_peered":"2024-02-10T08:39:50.008355+0000","last_clean":"2024-02-10T08:39:50.008355+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:39:50.008355+0000","last_undegraded":"2024-02-10T08:39:50.008355+0000","last_fullsized":"2024-02-10T08:39:50.008355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:39:46.641683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.641683+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":173,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000605+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:39:50.000605+0000","last_peered":"2024-02-10T08:39:50.000605+0000","last_clean":"2024-02-10T08:39:50.000605+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:39:50.000605+0000","last_undegraded":"2024-02-10T08:39:50.000605+0000","last_fullsized":"2024-02-10T08:39:50.000605+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":346,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.417318+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:39:50.417318+0000","last_peered":"2024-02-10T08:39:50.417318+0000","last_clean":"2024-02-10T08:39:50.417318+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:39:50.417318+0000","last_undegraded":"2024-02-10T08:39:50.417318+0000","last_fullsized":"2024-02-10T08:39:50.417318+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001654+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:39:50.001654+0000","last_peered":"2024-02-10T08:39:50.001654+0000","last_clean":"2024-02-10T08:39:50.001654+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:39:50.001654+0000","last_undegraded":"2024-02-10T08:39:50.001654+0000","last_fullsized":"2024-02-10T08:39:50.001654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":216,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007360+0000","last_change":"2024-02-10T08:36:37.041395+0000","last_active":"2024-02-10T08:39:50.007360+0000","last_peered":"2024-02-10T08:39:50.007360+0000","last_clean":"2024-02-10T08:39:50.007360+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:39:50.007360+0000","last_undegraded":"2024-02-10T08:39:50.007360+0000","last_fullsized":"2024-02-10T08:39:50.007360+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":285,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003223+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:39:50.003223+0000","last_peered":"2024-02-10T08:39:50.003223+0000","last_clean":"2024-02-10T08:39:50.003223+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:39:50.003223+0000","last_undegraded":"2024-02-10T08:39:50.003223+0000","last_fullsized":"2024-02-10T08:39:50.003223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120092+0000","last_change":"2024-02-10T08:39:46.475770+0000","last_active":"2024-02-10T08:39:50.120092+0000","last_peered":"2024-02-10T08:39:50.120092+0000","last_clean":"2024-02-10T08:39:50.120092+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:39:50.120092+0000","last_undegraded":"2024-02-10T08:39:50.120092+0000","last_fullsized":"2024-02-10T08:39:50.120092+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.475678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.475678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413540+0000","last_change":"2024-02-10T08:38:13.463715+0000","last_active":"2024-02-10T08:39:50.413540+0000","last_peered":"2024-02-10T08:39:50.413540+0000","last_clean":"2024-02-10T08:39:50.413540+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:39:50.413540+0000","last_undegraded":"2024-02-10T08:39:50.413540+0000","last_fullsized":"2024-02-10T08:39:50.413540+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.463644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.463644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":177,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007699+0000","last_change":"2024-02-10T08:38:20.592140+0000","last_active":"2024-02-10T08:39:50.007699+0000","last_peered":"2024-02-10T08:39:50.007699+0000","last_clean":"2024-02-10T08:39:50.007699+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:39:50.007699+0000","last_undegraded":"2024-02-10T08:39:50.007699+0000","last_fullsized":"2024-02-10T08:39:50.007699+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:20.592065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.592065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":216,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120429+0000","last_change":"2024-02-10T08:39:12.687989+0000","last_active":"2024-02-10T08:39:50.120429+0000","last_peered":"2024-02-10T08:39:50.120429+0000","last_clean":"2024-02-10T08:39:50.120429+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:39:50.120429+0000","last_undegraded":"2024-02-10T08:39:50.120429+0000","last_fullsized":"2024-02-10T08:39:50.120429+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.687918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.687918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":277,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000633+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:39:50.000633+0000","last_peered":"2024-02-10T08:39:50.000633+0000","last_clean":"2024-02-10T08:39:50.000633+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:39:50.000633+0000","last_undegraded":"2024-02-10T08:39:50.000633+0000","last_fullsized":"2024-02-10T08:39:50.000633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":276,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004949+0000","last_change":"2024-02-10T08:39:04.936255+0000","last_active":"2024-02-10T08:39:50.004949+0000","last_peered":"2024-02-10T08:39:50.004949+0000","last_clean":"2024-02-10T08:39:50.004949+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:39:50.004949+0000","last_undegraded":"2024-02-10T08:39:50.004949+0000","last_fullsized":"2024-02-10T08:39:50.004949+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:04.936174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.936174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":256,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004256+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:39:50.004256+0000","last_peered":"2024-02-10T08:39:50.004256+0000","last_clean":"2024-02-10T08:39:50.004256+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:39:50.004256+0000","last_undegraded":"2024-02-10T08:39:50.004256+0000","last_fullsized":"2024-02-10T08:39:50.004256+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":267,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413755+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:39:50.413755+0000","last_peered":"2024-02-10T08:39:50.413755+0000","last_clean":"2024-02-10T08:39:50.413755+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:39:50.413755+0000","last_undegraded":"2024-02-10T08:39:50.413755+0000","last_fullsized":"2024-02-10T08:39:50.413755+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":217,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414128+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:39:50.414128+0000","last_peered":"2024-02-10T08:39:50.414128+0000","last_clean":"2024-02-10T08:39:50.414128+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:39:50.414128+0000","last_undegraded":"2024-02-10T08:39:50.414128+0000","last_fullsized":"2024-02-10T08:39:50.414128+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":263,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.415134+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:39:50.415134+0000","last_peered":"2024-02-10T08:39:50.415134+0000","last_clean":"2024-02-10T08:39:50.415134+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:39:50.415134+0000","last_undegraded":"2024-02-10T08:39:50.415134+0000","last_fullsized":"2024-02-10T08:39:50.415134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007941+0000","last_change":"2024-02-10T08:38:13.601496+0000","last_active":"2024-02-10T08:39:50.007941+0000","last_peered":"2024-02-10T08:39:50.007941+0000","last_clean":"2024-02-10T08:39:50.007941+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:39:50.007941+0000","last_undegraded":"2024-02-10T08:39:50.007941+0000","last_fullsized":"2024-02-10T08:39:50.007941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.601390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.601390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":262,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.118472+0000","last_change":"2024-02-10T08:39:41.555411+0000","last_active":"2024-02-10T08:39:50.118472+0000","last_peered":"2024-02-10T08:39:50.118472+0000","last_clean":"2024-02-10T08:39:50.118472+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:39:50.118472+0000","last_undegraded":"2024-02-10T08:39:50.118472+0000","last_fullsized":"2024-02-10T08:39:50.118472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.555339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.555339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006278+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:39:50.006278+0000","last_peered":"2024-02-10T08:39:50.006278+0000","last_clean":"2024-02-10T08:39:50.006278+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:39:50.006278+0000","last_undegraded":"2024-02-10T08:39:50.006278+0000","last_fullsized":"2024-02-10T08:39:50.006278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.417749+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:39:50.417749+0000","last_peered":"2024-02-10T08:39:50.417749+0000","last_clean":"2024-02-10T08:39:50.417749+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:39:50.417749+0000","last_undegraded":"2024-02-10T08:39:50.417749+0000","last_fullsized":"2024-02-10T08:39:50.417749+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":276,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.415634+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:39:50.415634+0000","last_peered":"2024-02-10T08:39:50.415634+0000","last_clean":"2024-02-10T08:39:50.415634+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:39:50.415634+0000","last_undegraded":"2024-02-10T08:39:50.415634+0000","last_fullsized":"2024-02-10T08:39:50.415634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":269,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120783+0000","last_change":"2024-02-10T08:39:49.529787+0000","last_active":"2024-02-10T08:39:50.120783+0000","last_peered":"2024-02-10T08:39:50.120783+0000","last_clean":"2024-02-10T08:39:50.120783+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:39:50.120783+0000","last_undegraded":"2024-02-10T08:39:50.120783+0000","last_fullsized":"2024-02-10T08:39:50.120783+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:49.529724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.529724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":256,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007246+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:39:50.007246+0000","last_peered":"2024-02-10T08:39:50.007246+0000","last_clean":"2024-02-10T08:39:50.007246+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:39:50.007246+0000","last_undegraded":"2024-02-10T08:39:50.007246+0000","last_fullsized":"2024-02-10T08:39:50.007246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413422+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:39:50.413422+0000","last_peered":"2024-02-10T08:39:50.413422+0000","last_clean":"2024-02-10T08:39:50.413422+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:39:50.413422+0000","last_undegraded":"2024-02-10T08:39:50.413422+0000","last_fullsized":"2024-02-10T08:39:50.413422+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":177,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120642+0000","last_change":"2024-02-10T08:38:59.747834+0000","last_active":"2024-02-10T08:39:50.120642+0000","last_peered":"2024-02-10T08:39:50.120642+0000","last_clean":"2024-02-10T08:39:50.120642+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:39:50.120642+0000","last_undegraded":"2024-02-10T08:39:50.120642+0000","last_fullsized":"2024-02-10T08:39:50.120642+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.747762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.747762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":217,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007096+0000","last_change":"2024-02-10T08:39:41.691153+0000","last_active":"2024-02-10T08:39:50.007096+0000","last_peered":"2024-02-10T08:39:50.007096+0000","last_clean":"2024-02-10T08:39:50.007096+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:39:50.007096+0000","last_undegraded":"2024-02-10T08:39:50.007096+0000","last_fullsized":"2024-02-10T08:39:50.007096+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:39:41.691086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.691086+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007593+0000","last_change":"2024-02-10T08:37:39.573283+0000","last_active":"2024-02-10T08:39:50.007593+0000","last_peered":"2024-02-10T08:39:50.007593+0000","last_clean":"2024-02-10T08:39:50.007593+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:39:50.007593+0000","last_undegraded":"2024-02-10T08:39:50.007593+0000","last_fullsized":"2024-02-10T08:39:50.007593+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:39.573225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.573225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":260,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006231+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:39:50.006231+0000","last_peered":"2024-02-10T08:39:50.006231+0000","last_clean":"2024-02-10T08:39:50.006231+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:39:50.006231+0000","last_undegraded":"2024-02-10T08:39:50.006231+0000","last_fullsized":"2024-02-10T08:39:50.006231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007508+0000","last_change":"2024-02-10T08:38:14.837278+0000","last_active":"2024-02-10T08:39:50.007508+0000","last_peered":"2024-02-10T08:39:50.007508+0000","last_clean":"2024-02-10T08:39:50.007508+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:39:50.007508+0000","last_undegraded":"2024-02-10T08:39:50.007508+0000","last_fullsized":"2024-02-10T08:39:50.007508+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:14.837154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:14.837154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":286,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120940+0000","last_change":"2024-02-10T08:39:47.473081+0000","last_active":"2024-02-10T08:39:50.120940+0000","last_peered":"2024-02-10T08:39:50.120940+0000","last_clean":"2024-02-10T08:39:50.120940+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:39:50.120940+0000","last_undegraded":"2024-02-10T08:39:50.120940+0000","last_fullsized":"2024-02-10T08:39:50.120940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:39:47.473007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.473007+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413569+0000","last_change":"2024-02-10T08:38:31.436092+0000","last_active":"2024-02-10T08:39:50.413569+0000","last_peered":"2024-02-10T08:39:50.413569+0000","last_clean":"2024-02-10T08:39:50.413569+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:39:50.413569+0000","last_undegraded":"2024-02-10T08:39:50.413569+0000","last_fullsized":"2024-02-10T08:39:50.413569+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.435994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.435994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004557+0000","last_change":"2024-02-10T08:38:58.917062+0000","last_active":"2024-02-10T08:39:50.004557+0000","last_peered":"2024-02-10T08:39:50.004557+0000","last_clean":"2024-02-10T08:39:50.004557+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:39:50.004557+0000","last_undegraded":"2024-02-10T08:39:50.004557+0000","last_fullsized":"2024-02-10T08:39:50.004557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:58.916996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:58.916996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.004149+0000","last_change":"2024-02-10T08:38:28.906569+0000","last_active":"2024-02-10T08:39:50.004149+0000","last_peered":"2024-02-10T08:39:50.004149+0000","last_clean":"2024-02-10T08:39:50.004149+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:39:50.004149+0000","last_undegraded":"2024-02-10T08:39:50.004149+0000","last_fullsized":"2024-02-10T08:39:50.004149+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:38:28.906474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":295,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009887+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:39:50.009887+0000","last_peered":"2024-02-10T08:39:50.009887+0000","last_clean":"2024-02-10T08:39:50.009887+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:39:50.009887+0000","last_undegraded":"2024-02-10T08:39:50.009887+0000","last_fullsized":"2024-02-10T08:39:50.009887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120171+0000","last_change":"2024-02-10T08:38:32.342305+0000","last_active":"2024-02-10T08:39:50.120171+0000","last_peered":"2024-02-10T08:39:50.120171+0000","last_clean":"2024-02-10T08:39:50.120171+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:39:50.120171+0000","last_undegraded":"2024-02-10T08:39:50.120171+0000","last_fullsized":"2024-02-10T08:39:50.120171+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.342200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.342200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":254,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001281+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:39:50.001281+0000","last_peered":"2024-02-10T08:39:50.001281+0000","last_clean":"2024-02-10T08:39:50.001281+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:39:50.001281+0000","last_undegraded":"2024-02-10T08:39:50.001281+0000","last_fullsized":"2024-02-10T08:39:50.001281+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":177,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.118439+0000","last_change":"2024-02-10T08:38:18.541756+0000","last_active":"2024-02-10T08:39:50.118439+0000","last_peered":"2024-02-10T08:39:50.118439+0000","last_clean":"2024-02-10T08:39:50.118439+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:39:50.118439+0000","last_undegraded":"2024-02-10T08:39:50.118439+0000","last_fullsized":"2024-02-10T08:39:50.118439+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.541693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.541693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":285,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416332+0000","last_change":"2024-02-10T08:38:19.453131+0000","last_active":"2024-02-10T08:39:50.416332+0000","last_peered":"2024-02-10T08:39:50.416332+0000","last_clean":"2024-02-10T08:39:50.416332+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:39:50.416332+0000","last_undegraded":"2024-02-10T08:39:50.416332+0000","last_fullsized":"2024-02-10T08:39:50.416332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:38:19.453060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.453060+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":264,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413717+0000","last_change":"2024-02-10T08:38:16.426049+0000","last_active":"2024-02-10T08:39:50.413717+0000","last_peered":"2024-02-10T08:39:50.413717+0000","last_clean":"2024-02-10T08:39:50.413717+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:39:50.413717+0000","last_undegraded":"2024-02-10T08:39:50.413717+0000","last_fullsized":"2024-02-10T08:39:50.413717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:16.425945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:16.425945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":258,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416632+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:39:50.416632+0000","last_peered":"2024-02-10T08:39:50.416632+0000","last_clean":"2024-02-10T08:39:50.416632+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:39:50.416632+0000","last_undegraded":"2024-02-10T08:39:50.416632+0000","last_fullsized":"2024-02-10T08:39:50.416632+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":208,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.003398+0000","last_change":"2024-02-10T08:39:46.928020+0000","last_active":"2024-02-10T08:39:50.003398+0000","last_peered":"2024-02-10T08:39:50.003398+0000","last_clean":"2024-02-10T08:39:50.003398+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:39:50.003398+0000","last_undegraded":"2024-02-10T08:39:50.003398+0000","last_fullsized":"2024-02-10T08:39:50.003398+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.927953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":298,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.001135+0000","last_change":"2024-02-10T08:38:26.871174+0000","last_active":"2024-02-10T08:39:50.001135+0000","last_peered":"2024-02-10T08:39:50.001135+0000","last_clean":"2024-02-10T08:39:50.001135+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:39:50.001135+0000","last_undegraded":"2024-02-10T08:39:50.001135+0000","last_fullsized":"2024-02-10T08:39:50.001135+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:38:26.871100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.871100+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":265,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.005973+0000","last_change":"2024-02-10T08:39:02.250833+0000","last_active":"2024-02-10T08:39:50.005973+0000","last_peered":"2024-02-10T08:39:50.005973+0000","last_clean":"2024-02-10T08:39:50.005973+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:39:50.005973+0000","last_undegraded":"2024-02-10T08:39:50.005973+0000","last_fullsized":"2024-02-10T08:39:50.005973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:02.250719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:02.250719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.002596+0000","last_change":"2024-02-10T08:38:27.952601+0000","last_active":"2024-02-10T08:39:50.002596+0000","last_peered":"2024-02-10T08:39:50.002596+0000","last_clean":"2024-02-10T08:39:50.002596+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:39:50.002596+0000","last_undegraded":"2024-02-10T08:39:50.002596+0000","last_fullsized":"2024-02-10T08:39:50.002596+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:27.952518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:27.952518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":212,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.009068+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:39:50.009068+0000","last_peered":"2024-02-10T08:39:50.009068+0000","last_clean":"2024-02-10T08:39:50.009068+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:39:50.009068+0000","last_undegraded":"2024-02-10T08:39:50.009068+0000","last_fullsized":"2024-02-10T08:39:50.009068+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":275,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000709+0000","last_change":"2024-02-10T08:39:11.790943+0000","last_active":"2024-02-10T08:39:50.000709+0000","last_peered":"2024-02-10T08:39:50.000709+0000","last_clean":"2024-02-10T08:39:50.000709+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:39:50.000709+0000","last_undegraded":"2024-02-10T08:39:50.000709+0000","last_fullsized":"2024-02-10T08:39:50.000709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.790866+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":274,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006234+0000","last_change":"2024-02-10T08:38:30.389134+0000","last_active":"2024-02-10T08:39:50.006234+0000","last_peered":"2024-02-10T08:39:50.006234+0000","last_clean":"2024-02-10T08:39:50.006234+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:39:50.006234+0000","last_undegraded":"2024-02-10T08:39:50.006234+0000","last_fullsized":"2024-02-10T08:39:50.006234+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.389076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":253,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006419+0000","last_change":"2024-02-10T08:37:22.720373+0000","last_active":"2024-02-10T08:39:50.006419+0000","last_peered":"2024-02-10T08:39:50.006419+0000","last_clean":"2024-02-10T08:39:50.006419+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:39:50.006419+0000","last_undegraded":"2024-02-10T08:39:50.006419+0000","last_fullsized":"2024-02-10T08:39:50.006419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.720295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.720295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":177,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006618+0000","last_change":"2024-02-10T08:38:15.539954+0000","last_active":"2024-02-10T08:39:50.006618+0000","last_peered":"2024-02-10T08:39:50.006618+0000","last_clean":"2024-02-10T08:39:50.006618+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:39:50.006618+0000","last_undegraded":"2024-02-10T08:39:50.006618+0000","last_fullsized":"2024-02-10T08:39:50.006618+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:15.539850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:15.539850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":293,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000286+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:39:50.000286+0000","last_peered":"2024-02-10T08:39:50.000286+0000","last_clean":"2024-02-10T08:39:50.000286+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:39:50.000286+0000","last_undegraded":"2024-02-10T08:39:50.000286+0000","last_fullsized":"2024-02-10T08:39:50.000286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":273,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.414241+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:39:50.414241+0000","last_peered":"2024-02-10T08:39:50.414241+0000","last_clean":"2024-02-10T08:39:50.414241+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:39:50.414241+0000","last_undegraded":"2024-02-10T08:39:50.414241+0000","last_fullsized":"2024-02-10T08:39:50.414241+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":259,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.000909+0000","last_change":"2024-02-10T08:39:05.790083+0000","last_active":"2024-02-10T08:39:50.000909+0000","last_peered":"2024-02-10T08:39:50.000909+0000","last_clean":"2024-02-10T08:39:50.000909+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:39:50.000909+0000","last_undegraded":"2024-02-10T08:39:50.000909+0000","last_fullsized":"2024-02-10T08:39:50.000909+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.790005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.790005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":216,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.006732+0000","last_change":"2024-02-10T08:38:30.695748+0000","last_active":"2024-02-10T08:39:50.006732+0000","last_peered":"2024-02-10T08:39:50.006732+0000","last_clean":"2024-02-10T08:39:50.006732+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:39:50.006732+0000","last_undegraded":"2024-02-10T08:39:50.006732+0000","last_fullsized":"2024-02-10T08:39:50.006732+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.695684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.695684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":307,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.413333+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:39:50.413333+0000","last_peered":"2024-02-10T08:39:50.413333+0000","last_clean":"2024-02-10T08:39:50.413333+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:39:50.413333+0000","last_undegraded":"2024-02-10T08:39:50.413333+0000","last_fullsized":"2024-02-10T08:39:50.413333+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":178,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.007027+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:39:50.007027+0000","last_peered":"2024-02-10T08:39:50.007027+0000","last_clean":"2024-02-10T08:39:50.007027+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:39:50.007027+0000","last_undegraded":"2024-02-10T08:39:50.007027+0000","last_fullsized":"2024-02-10T08:39:50.007027+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":253,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.416279+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:39:50.416279+0000","last_peered":"2024-02-10T08:39:50.416279+0000","last_clean":"2024-02-10T08:39:50.416279+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:39:50.416279+0000","last_undegraded":"2024-02-10T08:39:50.416279+0000","last_fullsized":"2024-02-10T08:39:50.416279+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":207,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-02-10T08:39:50.120460+0000","last_change":"2024-02-10T08:38:34.376839+0000","last_active":"2024-02-10T08:39:50.120460+0000","last_peered":"2024-02-10T08:39:50.120460+0000","last_clean":"2024-02-10T08:39:50.120460+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:39:50.120460+0000","last_undegraded":"2024-02-10T08:39:50.120460+0000","last_fullsized":"2024-02-10T08:39:50.120460+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:34.376762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:34.376762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001220,"num_objects":123,"num_object_clones":2,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":548,"num_write_kb":4449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13496320,"data_stored":12003668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1795,"ondisk_log_size":1795,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":188,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":386,"num_read_kb":381,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299473,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306180,"kb_used_data":4848,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438972,"statfs":{"total":95995035648,"available":95681507328,"internally_reserved":0,"allocated":4964352,"data_stored":2719481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.76400000000000001,"15min":0.60299999999999998},"min":{"1min":0.30399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.70399999999999996,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67100000000000004,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59399999999999997,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.62,"15min":0.54800000000000004},"min":{"1min":0.314,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.623,"5min":7.593,"15min":7.593},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65100000000000002,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65600000000000003,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.63800000000000001,"15min":0.54600000000000004},"min":{"1min":0.36299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.85399999999999998,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":19.369,"15min":19.369},"last":0.53800000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.68700000000000006,"15min":0.56200000000000006},"min":{"1min":0.378,"5min":0.307,"15min":0.307},"max":{"1min":0.63500000000000001,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84999999999999998,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72799999999999998,"5min":22.102,"15min":22.102},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.498,"15min":0.44900000000000001},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":8.827,"15min":8.827},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.56999999999999995,"15min":0.47399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67000000000000004,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.497}]},{"osd":5,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.67400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70799999999999996,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68899999999999995,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.70799999999999996,"15min":0.56200000000000006},"min":{"1min":0.313,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89100000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.443},{"interface":"front","average":{"1min":0.499,"5min":0.503,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69699999999999995,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.52300000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561111,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318188,"kb_used_data":8664,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426964,"statfs":{"total":95995035648,"available":95669211136,"internally_reserved":0,"allocated":8871936,"data_stored":6719248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.63100000000000001,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26400000000000001,"15min":0.19800000000000001},"max":{"1min":0.65800000000000003,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.65400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.66600000000000004,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.59499999999999997,"15min":0.53000000000000003},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.311},{"interface":"front","average":{"1min":0.442,"5min":0.56999999999999995,"15min":0.496},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":0.59799999999999998,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3400000000000001,"5min":4.798,"15min":4.798},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58499999999999996,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.321,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.503}]},{"osd":3,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.82999999999999996,"15min":0.59399999999999997},"min":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":1.329,"5min":16.823,"15min":16.823},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.57999999999999996,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.60399999999999998,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.443,"15min":0.39500000000000002},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.433},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.44500000000000001,"15min":0.379},"min":{"1min":0.32900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.46899999999999997,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.7429999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.41999999999999998},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1339999999999999,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.378}]},{"osd":7,"last update":"Sat Feb 10 08:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.50800000000000001,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314372,"kb_used_data":4852,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430780,"statfs":{"total":95995035648,"available":95673118720,"internally_reserved":0,"allocated":4968448,"data_stored":2730850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.56599999999999995,"15min":0.40999999999999998},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.64400000000000002,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.63600000000000001,"15min":0.45800000000000002},"min":{"1min":0.23999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":16.166,"15min":16.166},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":1.0980000000000001,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.55300000000000005,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.86099999999999999,"15min":0.53300000000000003},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54700000000000004,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.74399999999999999,"15min":0.53200000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":24.773,"15min":24.773},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71199999999999997,"15min":0.53700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.60399999999999998,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.442}]},{"osd":4,"last update":"Sat Feb 10 08:39:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.752,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89000000000000001,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.67800000000000005,"15min":0.60999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.98999999999999999,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67200000000000004,"15min":0.61199999999999999},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.78600000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.38700000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.67600000000000005,"15min":0.60999999999999999},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.95599999999999996,"5min":9.532,"15min":9.532},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.78800000000000003,"15min":0.65600000000000003},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0009999999999999,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.72499999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.83999999999999997,"15min":0.66700000000000004},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.57299999999999995}]}]},{"osd":0,"up_from":0,"seq":34359738484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314316,"kb_used_data":4796,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430836,"statfs":{"total":95995035648,"available":95673176064,"internally_reserved":0,"allocated":4911104,"data_stored":2718983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.5,"15min":0.40300000000000002},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.58599999999999997,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.318,"5min":0.65200000000000002,"15min":0.46000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56799999999999995,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.70099999999999996,"15min":0.47699999999999998},"min":{"1min":0.24399999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":12.512,"15min":12.512},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.60999999999999999,"15min":0.46200000000000002},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.223},"max":{"1min":0.50800000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49399999999999999,"15min":0.44800000000000001},"min":{"1min":0.22500000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.1949999999999998,"5min":4.617,"15min":4.617},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.59799999999999998,"15min":0.49099999999999999},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.6779999999999999,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:39:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.85599999999999998,"15min":0.67800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.875,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.77500000000000002,"15min":0.66100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.84099999999999997,"15min":0.65400000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68799999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.71599999999999997,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:38:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80600000000000005,"15min":0.66200000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.3090000000000002,"5min":9.266,"15min":9.266},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.73099999999999998,"15min":0.64900000000000002},"min":{"1min":0.45100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.4630000000000001,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.80100000000000005,"15min":0.69299999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84099999999999997,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.77200000000000002,"15min":0.66800000000000004},"min":{"1min":0.46600000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.60799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182502,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318252,"kb_used_data":8736,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426900,"statfs":{"total":95995035648,"available":95669145600,"internally_reserved":0,"allocated":8945664,"data_stored":6727570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.81699999999999995,"15min":0.51600000000000001},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.42299999999999999,"5min":14.744,"15min":14.744},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.872,"15min":0.54500000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.498,"5min":11.808,"15min":11.808},"last":0.498}]},{"osd":1,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.82599999999999996,"15min":0.55100000000000005},"min":{"1min":0.23200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.262,"5min":16.634,"15min":16.634},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.82099999999999995,"15min":0.56100000000000005},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":0.55100000000000005,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.86399999999999999,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":1.079,"15min":0.63500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.51500000000000001,"5min":23.294,"15min":23.294},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:39:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.84799999999999998,"15min":0.68999999999999995},"min":{"1min":0.32400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.589,"5min":12.371,"15min":12.371},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.84699999999999998,"15min":0.66700000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0409999999999999,"5min":14.711,"15min":14.711},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.97199999999999998,"15min":0.69999999999999996},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73399999999999999,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.76600000000000001,"15min":0.64100000000000001},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70199999999999996,"5min":11.448,"15min":11.448},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.98599999999999999,"15min":0.68500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.92300000000000004,"15min":0.67500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.879,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72199999999999998,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":1.0469999999999999,"15min":0.71199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68100000000000005,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.65000000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306148,"kb_used_data":4816,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439004,"statfs":{"total":95995035648,"available":95681540096,"internally_reserved":0,"allocated":4931584,"data_stored":2719480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.84199999999999997,"15min":0.64400000000000002},"min":{"1min":0.29499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71999999999999997,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.65600000000000003,"15min":0.56200000000000006},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.626,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.59399999999999997},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70599999999999996,"5min":5.774,"15min":5.774},"last":0.621},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.74099999999999999,"15min":0.61099999999999999},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67100000000000004,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.76100000000000001,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69399999999999995,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.77400000000000002,"15min":0.61399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63700000000000001,"5min":10.01,"15min":10.01},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.91900000000000004,"15min":0.66500000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.62,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.0960000000000001,"15min":0.71999999999999997},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.82999999999999996,"5min":25.27,"15min":25.27},"last":0.39500000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.51000000000000001,"15min":0.41899999999999998},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67300000000000004,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47699999999999998,"15min":0.41299999999999998},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71499999999999997,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:38:55 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.65700000000000003,"15min":0.47899999999999998},"min":{"1min":0.29399999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.54000000000000004,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.63700000000000001,"15min":0.501},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.085,"5min":13.753,"15min":13.753},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.61699999999999999,"15min":0.50700000000000001},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69499999999999995,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.67900000000000005,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0509999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.68999999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790044,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305904,"kb_used_data":8672,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439248,"statfs":{"total":95995035648,"available":95681789952,"internally_reserved":0,"allocated":8880128,"data_stored":6726999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.9710000000000001,"15min":0.999},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66400000000000003,"5min":107.551,"15min":107.551},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.845,"15min":0.96399999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":0.61899999999999999,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.6910000000000001,"15min":0.93999999999999995},"min":{"1min":0.34899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":107.205,"15min":107.205},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":1.363,"15min":0.82899999999999996},"min":{"1min":0.32100000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.65700000000000003,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.498}]},{"osd":2,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":1.8480000000000001,"15min":0.95399999999999996},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67800000000000005,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.927,"15min":0.995},"min":{"1min":0.311,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":0.65800000000000003,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.1479999999999999,"15min":0.71899999999999997},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2669999999999999,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":2.5219999999999998,"15min":1.2070000000000001},"min":{"1min":0.41699999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61299999999999999,"5min":114.526,"15min":114.526},"last":0.38500000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":1.1699999999999999,"15min":0.65300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.627,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.90200000000000002,"15min":0.55900000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.58699999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.36299999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:38:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.1919999999999999,"15min":0.67200000000000004},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63300000000000001,"5min":72.241,"15min":72.241},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.71099999999999997,"15min":0.54100000000000004},"min":{"1min":0.307,"5min":0.248,"15min":0.248},"max":{"1min":0.621,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.40799999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":1.1819999999999999,"15min":0.71099999999999997},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.60899999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.31},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.2450000000000001,"15min":0.73899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4000411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4190208,"data_stored":4000483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:39:54.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:39:55.359 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:55.360 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:55.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.628900+0000 osd.4 (osd.4) 138 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:39:55.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.632407+0000 osd.4 (osd.4) 139 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:39:55.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.849907+0000 osd.3 (osd.3) 155 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:39:55.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.850760+0000 osd.3 (osd.3) 156 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.998338+0000 mon.a (mon.0) 2022 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:53.998455+0000 mon.a (mon.0) 2023 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:53.998517+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:54.002887+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:54.003889+0000 mon.a (mon.0) 2025 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:54.010383+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:54.013367+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.435426+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:39:55.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.437264+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.628900+0000 osd.4 (osd.4) 138 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.632407+0000 osd.4 (osd.4) 139 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.849907+0000 osd.3 (osd.3) 155 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.850760+0000 osd.3 (osd.3) 156 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.998338+0000 mon.a (mon.0) 2022 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:53.998455+0000 mon.a (mon.0) 2023 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:53.998517+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:54.002887+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:54.003889+0000 mon.a (mon.0) 2025 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:54.010383+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:54.013367+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.435426+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.437264+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.628900+0000 osd.4 (osd.4) 138 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.632407+0000 osd.4 (osd.4) 139 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.849907+0000 osd.3 (osd.3) 155 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.850760+0000 osd.3 (osd.3) 156 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.998338+0000 mon.a (mon.0) 2022 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:53.998455+0000 mon.a (mon.0) 2023 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi107-112411-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:53.998517+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:54.002887+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:54.003889+0000 mon.a (mon.0) 2025 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:54.010383+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:54.013367+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:39:55.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.435426+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:39:55.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.437264+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:39:55.545 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:55.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:55.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:55.679 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:55.716 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:55.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:55.804 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:39:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:39:55] "GET /metrics HTTP/1.1" 200 186240 "" "Prometheus/2.33.4" 2024-02-10T08:39:55.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:55.853 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.982+0000 7fe6c0877700 1 -- 172.21.15.107:0/739417355 >> v1:172.21.15.107:6789/0 conn(0x7fe6bc103250 legacy=0x7fe6bc1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- 172.21.15.107:0/739417355 shutdown_connections 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- 172.21.15.107:0/739417355 >> 172.21.15.107:0/739417355 conn(0x7fe6bc0f2510 msgr2=0x7fe6bc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- 172.21.15.107:0/739417355 shutdown_connections 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- 172.21.15.107:0/739417355 wait complete. 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 librados: starting msgr at 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 librados: starting objecter 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 Processor -- start 2024-02-10T08:39:56.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- start start 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 librados: setting wanted keys 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 librados: calling monclient init 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.983+0000 7fe6c0877700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6bc1173e0 con 0x7fe6bc103250 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6bc1185c0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6bc1197c0 con 0x7fe6bc0fc910 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ba59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe6bc0fc910 0x7fe6bc0f9b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55598/0 (socket says 172.21.15.107:55598) 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ba59c700 1 -- 172.21.15.107:0/392774232 learned_addr learned my addr 172.21.15.107:0/392774232 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter ms_handle_connect 0x7fe6bc0fc910 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter ms_handle_connect 0x7fe6bc103250 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter ms_handle_connect 0x7fe6bc0ff710 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226540438 0 0) 0x7fe6bc1197c0 con 0x7fe6bc0fc910 2024-02-10T08:39:56.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a4003800 con 0x7fe6bc0fc910 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887832948 0 0) 0x7fe6bc1173e0 con 0x7fe6bc103250 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc1197c0 con 0x7fe6bc103250 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939014003 0 0) 0x7fe6bc1185c0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc1173e0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093674203 0 0) 0x7fe6a4003800 con 0x7fe6bc0fc910 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6bc1185c0 con 0x7fe6bc0fc910 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383165580 0 0) 0x7fe6bc1173e0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6a4003800 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 835524789 0 0) 0x7fe6bc1197c0 con 0x7fe6bc103250 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6bc1173e0 con 0x7fe6bc103250 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6ac002a10 con 0x7fe6bc0fc910 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6b00019f0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6b4003220 con 0x7fe6bc103250 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3506622236 0 0) 0x7fe6a4003800 con 0x7fe6bc0ff710 2024-02-10T08:39:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 >> v1:172.21.15.107:6790/0 conn(0x7fe6bc0fc910 legacy=0x7fe6bc0f9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 >> v1:172.21.15.107:6789/0 conn(0x7fe6bc103250 legacy=0x7fe6bc115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc11a9c0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc1175d0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 10 client.48055.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc117a40 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6c0877700 1 librados: init done 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe6b0001640 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.984+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6b0003130 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.985+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c004f20 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe6b0001f00 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(301..301 src has 50..301) v4 ==== 8372+0+0 (unknown 2339425143 0 0) 0x7fe6b00765f0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 10 client.48055.objecter ms_dispatch 0x7fe6bc001550 osd_map(301..301 src has 50..301) v4 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 3 client.48055.objecter handle_osd_map got epochs [301,301] > 0 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 3 client.48055.objecter handle_osd_map decoding full epoch 301 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 20 client.48055.objecter dump_active .. 0 homeless 2024-02-10T08:39:56.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.986+0000 7fe6ab7fe700 10 client.48055.objecter ms_handle_connect 0x7fe6a4063c70 2024-02-10T08:39:56.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:55.990+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe6b00498b0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:39:56.051 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:56.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:56.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:56.172 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:56.196+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fe69c0051d0 con 0x7fe6bc0ff710 2024-02-10T08:39:56.211 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:39:56.214 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:56.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:56.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.514919+0000 osd.7 (osd.7) 127 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:39:56.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.516882+0000 osd.7 (osd.7) 128 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:39:56.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.601098+0000 osd.4 (osd.4) 140 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:39:56.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.602584+0000 osd.4 (osd.4) 141 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.514919+0000 osd.7 (osd.7) 127 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.516882+0000 osd.7 (osd.7) 128 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.601098+0000 osd.4 (osd.4) 140 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.602584+0000 osd.4 (osd.4) 141 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.669324+0000 osd.5 (osd.5) 145 : cluster [DBG] 6.d scrub starts 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.672283+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.d scrub ok 2024-02-10T08:39:56.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.892010+0000 osd.3 (osd.3) 157 : cluster [DBG] 60.e scrub starts 2024-02-10T08:39:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.514919+0000 osd.7 (osd.7) 127 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:39:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.516882+0000 osd.7 (osd.7) 128 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:39:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.601098+0000 osd.4 (osd.4) 140 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.602584+0000 osd.4 (osd.4) 141 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.669324+0000 osd.5 (osd.5) 145 : cluster [DBG] 6.d scrub starts 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.672283+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.d scrub ok 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.892010+0000 osd.3 (osd.3) 157 : cluster [DBG] 60.e scrub starts 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:54.894087+0000 osd.3 (osd.3) 158 : cluster [DBG] 60.e scrub ok 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:55.240769+0000 mon.a (mon.0) 2027 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:55.240868+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:55.241704+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:55.249531+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:55.253910+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:55.407533+0000 mgr.y (mgr.14553) 292 : cluster [DBG] pgmap v428: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:55.520859+0000 osd.7 (osd.7) 129 : cluster [DBG] 60.10 deep-scrub starts 2024-02-10T08:39:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:55.527848+0000 osd.7 (osd.7) 130 : cluster [DBG] 60.10 deep-scrub ok 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.002371+0000 mon.a (mon.0) 2031 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.002475+0000 mon.a (mon.0) 2032 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.002542+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.008476+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.008835+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.197743+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.107:0/392774232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:56.198178+0000 mon.a (mon.0) 2035 : audit [INF] from='client.48055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.317 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:56.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:56.345 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:56.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.669324+0000 osd.5 (osd.5) 145 : cluster [DBG] 6.d scrub starts 2024-02-10T08:39:56.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.672283+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.d scrub ok 2024-02-10T08:39:56.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.892010+0000 osd.3 (osd.3) 157 : cluster [DBG] 60.e scrub starts 2024-02-10T08:39:56.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:54.894087+0000 osd.3 (osd.3) 158 : cluster [DBG] 60.e scrub ok 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:55.240769+0000 mon.a (mon.0) 2027 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:55.240868+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:55.241704+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:55.249531+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:55.253910+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:55.407533+0000 mgr.y (mgr.14553) 292 : cluster [DBG] pgmap v428: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:55.520859+0000 osd.7 (osd.7) 129 : cluster [DBG] 60.10 deep-scrub starts 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:55.527848+0000 osd.7 (osd.7) 130 : cluster [DBG] 60.10 deep-scrub ok 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.002371+0000 mon.a (mon.0) 2031 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.002475+0000 mon.a (mon.0) 2032 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.002542+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.008476+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.008835+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.197743+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.107:0/392774232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:56.198178+0000 mon.a (mon.0) 2035 : audit [INF] from='client.48055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:54.894087+0000 osd.3 (osd.3) 158 : cluster [DBG] 60.e scrub ok 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:55.240769+0000 mon.a (mon.0) 2027 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:55.240868+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:55.241704+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:55.249531+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:39:56.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:55.253910+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:55.407533+0000 mgr.y (mgr.14553) 292 : cluster [DBG] pgmap v428: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:55.520859+0000 osd.7 (osd.7) 129 : cluster [DBG] 60.10 deep-scrub starts 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:55.527848+0000 osd.7 (osd.7) 130 : cluster [DBG] 60.10 deep-scrub ok 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.002371+0000 mon.a (mon.0) 2031 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi107-112411-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.002475+0000 mon.a (mon.0) 2032 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.002542+0000 mon.a (mon.0) 2033 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.008476+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.008835+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.197743+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.107:0/392774232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:56.198178+0000 mon.a (mon.0) 2035 : audit [INF] from='client.48055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:39:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:56.569 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:56.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:56.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:56.697 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:57.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.004+0000 7fe6ab7fe700 1 -- 172.21.15.107:0/392774232 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v303) v1 ==== 87+0+0 (unknown 545948506 0 0) 0x7fe6b0047cc0 con 0x7fe6bc0ff710 2024-02-10T08:39:57.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.007+0000 7fe6c0877700 10 librados: watch_flush enter 2024-02-10T08:39:57.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.009+0000 7fe6c0877700 10 librados: watch_flush exit 2024-02-10T08:39:57.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.009+0000 7fe6c0877700 20 client.48055.objecter shutdown clearing up homeless session... 2024-02-10T08:39:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.009+0000 7fe6c0877700 10 client.48055.objecter successfully canceled tick 2024-02-10T08:39:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.010+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe6a4063c70 legacy=0x7fe6a4066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.010+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 >> v1:172.21.15.195:6789/0 conn(0x7fe6bc0ff710 legacy=0x7fe6bc0fa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.010+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 shutdown_connections 2024-02-10T08:39:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.010+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 >> 172.21.15.107:0/392774232 conn(0x7fe6bc0f2510 msgr2=0x7fe6bc1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:57.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.010+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 shutdown_connections 2024-02-10T08:39:57.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.011+0000 7fe6c0877700 1 -- 172.21.15.107:0/392774232 wait complete. 2024-02-10T08:39:57.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:57.011+0000 7fe6c0877700 1 librados: shutdown 2024-02-10T08:39:57.012 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:39:57.045 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:57.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:57.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:57.176 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:57.204 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:57.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:57.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:39:57.336 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.541479+0000 osd.4 (osd.4) 142 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:39:57.336 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.545037+0000 osd.4 (osd.4) 143 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:39:57.336 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.565296+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.568186+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:56.997250+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:57.004566+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:57.004733+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:57.004793+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e303: 8 total, 8 up, 7 in 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:57.005375+0000 mon.b (mon.1) 493 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:57.009400+0000 mon.a (mon.0) 2040 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.338 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:57.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.541479+0000 osd.4 (osd.4) 142 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:39:57.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.545037+0000 osd.4 (osd.4) 143 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:39:57.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.565296+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:39:57.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.568186+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:39:57.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:56.997250+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:57.004566+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:57.004733+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:57.004793+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e303: 8 total, 8 up, 7 in 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:57.005375+0000 mon.b (mon.1) 493 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:57.009400+0000 mon.a (mon.0) 2040 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.541479+0000 osd.4 (osd.4) 142 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.545037+0000 osd.4 (osd.4) 143 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.565296+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.568186+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:56.997250+0000 mon.a (mon.0) 2036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:57.004566+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:57.004733+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:39:57.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:57.004793+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e303: 8 total, 8 up, 7 in 2024-02-10T08:39:57.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:57.005375+0000 mon.b (mon.1) 493 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:57.009400+0000 mon.a (mon.0) 2040 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:39:57.541 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:39:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:57.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:39:57.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:39:57.676 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:57.987 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:39:57.988 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:39:58.307 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:57.408189+0000 mgr.y (mgr.14553) 293 : cluster [DBG] pgmap v431: 226 pgs: 5 creating+peering, 3 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:58.003685+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:58.003788+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e304: 8 total, 8 up, 7 in 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:58.007393+0000 mon.b (mon.1) 494 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:58.007701+0000 mon.a (mon.0) 2043 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.310 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:57.408189+0000 mgr.y (mgr.14553) 293 : cluster [DBG] pgmap v431: 226 pgs: 5 creating+peering, 3 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:58.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:58.003685+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:39:58.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:58.003788+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e304: 8 total, 8 up, 7 in 2024-02-10T08:39:58.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:58.007393+0000 mon.b (mon.1) 494 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:39:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:58.007701+0000 mon.a (mon.0) 2043 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:58.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:39:58.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:58.442 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:58.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:57.408189+0000 mgr.y (mgr.14553) 293 : cluster [DBG] pgmap v431: 226 pgs: 5 creating+peering, 3 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:39:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:58.003685+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:39:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:58.003788+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e304: 8 total, 8 up, 7 in 2024-02-10T08:39:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:58.007393+0000 mon.b (mon.1) 494 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:39:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:58.007701+0000 mon.a (mon.0) 2043 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:39:58.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:58.593 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:58.645 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:39:58.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:58.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:39:58.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:39:58.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:58.782 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:58.924 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:39:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:58.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:59.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:39:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:59.182 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:39:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:59.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:39:59.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:39:59.315 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:59.352 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:39:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:59.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- 172.21.15.107:0/3650160259 >> v1:172.21.15.107:6790/0 conn(0x7fd7d81007f0 legacy=0x7fd7d8102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- 172.21.15.107:0/3650160259 shutdown_connections 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- 172.21.15.107:0/3650160259 >> 172.21.15.107:0/3650160259 conn(0x7fd7d80f3640 msgr2=0x7fd7d80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- 172.21.15.107:0/3650160259 shutdown_connections 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- 172.21.15.107:0/3650160259 wait complete. 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 librados: starting msgr at 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 librados: starting objecter 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 Processor -- start 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 -- start start 2024-02-10T08:39:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 librados: setting wanted keys 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.424+0000 7fd7de81d700 1 librados: calling monclient init 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7de81d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7d80fd000 con 0x7fd7d8104330 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7de81d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7d80fd190 con 0x7fd7d80fda50 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7de81d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7d80fd380 con 0x7fd7d81007f0 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7dcdba700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd7d8104330 0x7fd7d81184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50824/0 (socket says 172.21.15.107:50824) 2024-02-10T08:39:59.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d77fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd7d81007f0 0x7fd7d8114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48072/0 (socket says 172.21.15.107:48072) 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7dcdba700 1 -- 172.21.15.107:0/3264944720 learned_addr learned my addr 172.21.15.107:0/3264944720 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter ms_handle_connect 0x7fd7d81007f0 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter ms_handle_connect 0x7fd7d8104330 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:59.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter ms_handle_connect 0x7fd7d80fda50 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601077708 0 0) 0x7fd7d80fd000 con 0x7fd7d8104330 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c0003800 con 0x7fd7d8104330 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397043144 0 0) 0x7fd7d80fd380 con 0x7fd7d81007f0 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d80fd000 con 0x7fd7d81007f0 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579586595 0 0) 0x7fd7d80fd190 con 0x7fd7d80fda50 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d80fd380 con 0x7fd7d80fda50 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931953851 0 0) 0x7fd7c0003800 con 0x7fd7d8104330 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7d80fd190 con 0x7fd7d8104330 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd7d0002c90 con 0x7fd7d8104330 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961263217 0 0) 0x7fd7d80fd000 con 0x7fd7d81007f0 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c0003800 con 0x7fd7d81007f0 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd7cc002f60 con 0x7fd7d81007f0 2024-02-10T08:39:59.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 704187001 0 0) 0x7fd7d80fd380 con 0x7fd7d80fda50 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7d80fd000 con 0x7fd7d80fda50 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.425+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd7c8001550 con 0x7fd7d80fda50 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2793098495 0 0) 0x7fd7d80fd190 con 0x7fd7d8104330 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 >> v1:172.21.15.107:6790/0 conn(0x7fd7d81007f0 legacy=0x7fd7d8114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 >> v1:172.21.15.195:6789/0 conn(0x7fd7d80fda50 legacy=0x7fd7d80fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d811cc00 con 0x7fd7d8104330 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d8119c30 con 0x7fd7d8104330 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7de81d700 10 client.27768.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d811a0a0 con 0x7fd7d8104330 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7de81d700 1 librados: init done 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd7d0001c10 con 0x7fd7d8104330 2024-02-10T08:39:59.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.426+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd7d0005e10 con 0x7fd7d8104330 2024-02-10T08:39:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.427+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c4004f20 con 0x7fd7d8104330 2024-02-10T08:39:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.427+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd7d0001770 con 0x7fd7d8104330 2024-02-10T08:39:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(305..305 src has 50..305) v4 ==== 8083+0+0 (unknown 360826008 0 0) 0x7fd7d0079100 con 0x7fd7d8104330 2024-02-10T08:39:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 10 client.27768.objecter ms_dispatch 0x7fd7d8001550 osd_map(305..305 src has 50..305) v4 2024-02-10T08:39:59.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 3 client.27768.objecter handle_osd_map got epochs [305,305] > 0 2024-02-10T08:39:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 3 client.27768.objecter handle_osd_map decoding full epoch 305 2024-02-10T08:39:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 20 client.27768.objecter dump_active .. 0 homeless 2024-02-10T08:39:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.428+0000 7fd7d57fa700 10 client.27768.objecter ms_handle_connect 0x7fd7c0063d40 2024-02-10T08:39:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.432+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd7d004c520 con 0x7fd7d8104330 2024-02-10T08:39:59.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:39:59.481 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:39:59.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.594+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7c4000b90 con 0x7fd7c0063d40 2024-02-10T08:39:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.609+0000 7fd7d57fa700 1 -- 172.21.15.107:0/3264944720 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553708 (unknown 2965378022 0 1521887428) 0x7fd7c4000b90 con 0x7fd7c0063d40 2024-02-10T08:39:59.617 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 10 librados: watch_flush enter 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 10 librados: watch_flush exit 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 20 client.27768.objecter shutdown clearing up homeless session... 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 10 client.27768.objecter successfully canceled tick 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd7c0063d40 legacy=0x7fd7c0066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.612+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 >> v1:172.21.15.107:6789/0 conn(0x7fd7d8104330 legacy=0x7fd7d81184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.613+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 shutdown_connections 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.613+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 >> 172.21.15.107:0/3264944720 conn(0x7fd7d80f3640 msgr2=0x7fd7d81077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.613+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 shutdown_connections 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.613+0000 7fd7de81d700 1 -- 172.21.15.107:0/3264944720 wait complete. 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:39:59.613+0000 7fd7de81d700 1 librados: shutdown 2024-02-10T08:39:59.623 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:39:59.629 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:39:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:39:59.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:39:59.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:39:59.767 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:00.027 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.005054+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:59.005124+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e305: 8 total, 8 up, 7 in 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.008742+0000 mon.b (mon.1) 495 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.009026+0000 mon.a (mon.0) 2046 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.027035+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.027348+0000 mon.a (mon.0) 2047 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.027839+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.028061+0000 mon.a (mon.0) 2048 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:39:59.408789+0000 mgr.y (mgr.14553) 294 : cluster [DBG] pgmap v434: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:39:59.595392+0000 mgr.y (mgr.14553) 295 : audit [DBG] from='client.27768 v1:172.21.15.107:0/3264944720' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.000160+0000 mon.a (mon.0) 2049 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.000204+0000 mon.a (mon.0) 2050 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T08:40:00.028 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.000221+0000 mon.a (mon.0) 2051 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.000236+0000 mon.a (mon.0) 2052 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:40:00.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.000251+0000 mon.a (mon.0) 2053 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:40:00.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.005054+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:00.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:59.005124+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e305: 8 total, 8 up, 7 in 2024-02-10T08:40:00.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.008742+0000 mon.b (mon.1) 495 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.009026+0000 mon.a (mon.0) 2046 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.027035+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.027348+0000 mon.a (mon.0) 2047 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.027839+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.028061+0000 mon.a (mon.0) 2048 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:39:59.408789+0000 mgr.y (mgr.14553) 294 : cluster [DBG] pgmap v434: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:39:59.595392+0000 mgr.y (mgr.14553) 295 : audit [DBG] from='client.27768 v1:172.21.15.107:0/3264944720' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.000160+0000 mon.a (mon.0) 2049 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.000204+0000 mon.a (mon.0) 2050 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.000221+0000 mon.a (mon.0) 2051 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.000236+0000 mon.a (mon.0) 2052 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.000251+0000 mon.a (mon.0) 2053 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:40:00.166 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.005054+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:59.005124+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e305: 8 total, 8 up, 7 in 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.008742+0000 mon.b (mon.1) 495 : audit [INF] from='client.? v1:172.21.15.107:0/143310587' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.009026+0000 mon.a (mon.0) 2046 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.027035+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.027348+0000 mon.a (mon.0) 2047 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.027839+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.028061+0000 mon.a (mon.0) 2048 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:39:59.408789+0000 mgr.y (mgr.14553) 294 : cluster [DBG] pgmap v434: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:39:59.595392+0000 mgr.y (mgr.14553) 295 : audit [DBG] from='client.27768 v1:172.21.15.107:0/3264944720' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.000160+0000 mon.a (mon.0) 2049 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.000204+0000 mon.a (mon.0) 2050 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.000221+0000 mon.a (mon.0) 2051 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.000236+0000 mon.a (mon.0) 2052 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:40:00.167 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.000251+0000 mon.a (mon.0) 2053 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:40:00.279 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.191+0000 7f3731b29700 1 -- 172.21.15.107:0/2834335674 >> v1:172.21.15.107:6789/0 conn(0x7f372c100840 legacy=0x7f372c102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.191+0000 7f3731b29700 1 -- 172.21.15.107:0/2834335674 shutdown_connections 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.191+0000 7f3731b29700 1 -- 172.21.15.107:0/2834335674 >> 172.21.15.107:0/2834335674 conn(0x7f372c06c090 msgr2=0x7f372c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.191+0000 7f3731b29700 1 -- 172.21.15.107:0/2834335674 shutdown_connections 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.191+0000 7f3731b29700 1 -- 172.21.15.107:0/2834335674 wait complete. 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 librados: starting msgr at 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 librados: starting objecter 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 Processor -- start 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 -- start start 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 librados: setting wanted keys 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 librados: calling monclient init 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f372c0fd0c0 con 0x7f372c100840 2024-02-10T08:40:00.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f372c0fd290 con 0x7f372c0fd9f0 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f3731b29700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f372c0fd4a0 con 0x7f372c104350 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f372bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f372c104350 0x7f372c1184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48128/0 (socket says 172.21.15.107:48128) 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.192+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 learned_addr learned my addr 172.21.15.107:0/1165078207 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter ms_handle_connect 0x7f372c104350 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter ms_handle_connect 0x7f372c100840 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter ms_handle_connect 0x7f372c0fd9f0 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643717948 0 0) 0x7f372c0fd4a0 con 0x7f372c104350 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714003800 con 0x7f372c104350 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659561383 0 0) 0x7f372c0fd0c0 con 0x7f372c100840 2024-02-10T08:40:00.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f372c0fd4a0 con 0x7f372c100840 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907490585 0 0) 0x7f372c0fd290 con 0x7f372c0fd9f0 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f372c0fd0c0 con 0x7f372c0fd9f0 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3174748942 0 0) 0x7f3714003800 con 0x7f372c104350 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f372c0fd290 con 0x7f372c104350 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3724001d20 con 0x7f372c104350 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1199861683 0 0) 0x7f372c0fd4a0 con 0x7f372c100840 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3714003800 con 0x7f372c100840 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138341272 0 0) 0x7f372c0fd0c0 con 0x7f372c0fd9f0 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f372c0fd4a0 con 0x7f372c0fd9f0 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3720002ea0 con 0x7f372c100840 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f371c0017f0 con 0x7f372c0fd9f0 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2188656000 0 0) 0x7f372c0fd290 con 0x7f372c104350 2024-02-10T08:40:00.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.195:6789/0 conn(0x7f372c0fd9f0 legacy=0x7f372c0fc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6789/0 conn(0x7f372c100840 legacy=0x7f372c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c11cc20 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c119c50 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3731b29700 10 client.53351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c11a0c0 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3731b29700 1 librados: init done 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3724002780 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.193+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3724005e00 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.194+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3718004f00 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.194+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f37240022e0 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(306..306 src has 50..306) v4 ==== 7967+0+0 (unknown 2925107919 0 0) 0x7f3724078f10 con 0x7f372c104350 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 osd_map(306..306 src has 50..306) v4 2024-02-10T08:40:00.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 3 client.53351.objecter handle_osd_map got epochs [306,306] > 0 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 3 client.53351.objecter handle_osd_map decoding full epoch 306 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 20 client.53351.objecter dump_active .. 0 homeless 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f3714063c80 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v306) v1 ==== 54+0+16 (unknown 1403654055 0 2420000192) 0x7f372407b2b0 con 0x7f372c104350 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 10 client.53351.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 1 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 1 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 0 1 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 4 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 10 client.53351.objecter _send_command 1 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.195+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3718004950 con 0x7f3718004530 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f3718004530 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3718004950 con 0x7f3718004530 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 10 client.53351.objecter _finish_command 1 = system:0 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 4 2024-02-10T08:40:00.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.196+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 0 1 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 10 client.53351.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 2 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 2 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f37180051a0 osd=0 3 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 0 2 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 4 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 10 client.53351.objecter _send_command 2 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.211+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3718004d30 con 0x7f3718004530 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.212+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3718004d30 con 0x7f3718004530 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.212+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.212+0000 7f3728ff9700 10 client.53351.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.212+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 4 2024-02-10T08:40:00.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.212+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 0 2 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 10 client.53351.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 1 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 4 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 10 client.53351.objecter _send_command 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f371800c000 con 0x7f3718004150 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.213+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f3718004150 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.214+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f371800c000 con 0x7f3718004150 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.214+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.214+0000 7f3728ff9700 10 client.53351.objecter _finish_command 3 = system:0 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.214+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 4 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.214+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 1 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 10 client.53351.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 4 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 4 2024-02-10T08:40:00.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718003fe0 osd=1 3 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 1 4 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 4 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 10 client.53351.objecter _send_command 4 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.228+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f37180018b0 con 0x7f3718004150 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.229+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f37180018b0 con 0x7f3718004150 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.229+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.229+0000 7f3728ff9700 10 client.53351.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.229+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 4 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.229+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 1 4 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 10 client.53351.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 5 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 5 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 2 5 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 4 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 10 client.53351.objecter _send_command 5 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f371800dec0 con 0x7f371800d320 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f371800d320 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f371800dec0 con 0x7f371800d320 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:00.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 10 client.53351.objecter _finish_command 5 = system:0 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 4 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.230+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 2 5 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 10 client.53351.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 6 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 6 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f37180035f0 osd=2 3 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 2 6 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 4 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 10 client.53351.objecter _send_command 6 2024-02-10T08:40:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.245+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3718003060 con 0x7f371800d320 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3718003060 con 0x7f371800d320 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3728ff9700 10 client.53351.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 4 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 2 6 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 10 client.53351.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 7 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 7 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 3 7 2024-02-10T08:40:00.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 4 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 10 client.53351.objecter _send_command 7 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.246+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3718015a30 con 0x7f3718011eb0 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f3718011eb0 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3718015a30 con 0x7f3718011eb0 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 10 client.53351.objecter _finish_command 7 = system:0 2024-02-10T08:40:00.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 4 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.247+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 3 7 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 10 client.53351.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 8 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 8 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718003470 osd=3 3 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 3 8 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 4 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 10 client.53351.objecter _send_command 8 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.261+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f371800e270 con 0x7f3718011eb0 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f371800e270 con 0x7f3718011eb0 2024-02-10T08:40:00.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 10 client.53351.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 4 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 3 8 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 10 client.53351.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 9 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 9 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 4 9 2024-02-10T08:40:00.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 4 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 10 client.53351.objecter _send_command 9 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f371800e570 con 0x7f371800ce80 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.263+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f371800ce80 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.264+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f371800e570 con 0x7f371800ce80 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.264+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.264+0000 7f3728ff9700 10 client.53351.objecter _finish_command 9 = system:0 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.264+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 4 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.264+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 4 9 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 10 client.53351.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 10 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 10 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f371800e270 osd=4 3 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 4 10 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 4 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 10 client.53351.objecter _send_command 10 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.279+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f371800e570 con 0x7f371800ce80 2024-02-10T08:40:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f371800e570 con 0x7f371800ce80 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 10 client.53351.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 4 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 4 10 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 10 client.53351.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 11 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 11 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 5 11 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 4 2024-02-10T08:40:00.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 10 client.53351.objecter _send_command 11 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3718016d60 con 0x7f37180167c0 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.281+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f37180167c0 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.282+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3718016d60 con 0x7f37180167c0 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.282+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.282+0000 7f3728ff9700 10 client.53351.objecter _finish_command 11 = system:0 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.282+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 4 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.282+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 5 11 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 10 client.53351.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 12 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 12 2024-02-10T08:40:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f371800e570 osd=5 3 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 5 12 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 4 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 10 client.53351.objecter _send_command 12 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.296+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3718016d60 con 0x7f37180167c0 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3718016d60 con 0x7f37180167c0 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 10 client.53351.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 4 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 5 12 2024-02-10T08:40:00.317 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:00.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 10 client.53351.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 13 2024-02-10T08:40:00.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 13 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 6 13 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 4 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 10 client.53351.objecter _send_command 13 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3718022bb0 con 0x7f371801f030 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.298+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f371801f030 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.299+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3718022bb0 con 0x7f371801f030 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.299+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.299+0000 7f3728ff9700 10 client.53351.objecter _finish_command 13 = system:0 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.299+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 4 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.299+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 6 13 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 10 client.53351.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 14 2024-02-10T08:40:00.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 14 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718016fa0 osd=6 3 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 6 14 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 4 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 10 client.53351.objecter _send_command 14 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.313+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3718024360 con 0x7f371801f030 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3718024360 con 0x7f371801f030 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 10 client.53351.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 4 2024-02-10T08:40:00.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 6 14 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 10 client.53351.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 15 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 15 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 7 15 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 4 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 10 client.53351.objecter _send_command 15 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3718024f80 con 0x7f3718024a20 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.315+0000 7f3728ff9700 10 client.53351.objecter ms_handle_connect 0x7f3718024a20 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.316+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3718024f80 con 0x7f3718024a20 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.316+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.316+0000 7f3728ff9700 10 client.53351.objecter _finish_command 15 = system:0 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.316+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 4 2024-02-10T08:40:00.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.316+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 7 15 2024-02-10T08:40:00.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 10 client.53351.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:00.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign -1 16 2024-02-10T08:40:00.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 20 client.53351.objecter _get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 15 client.53351.objecter _session_command_op_remove -1 16 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 20 client.53351.objecter get_session s=0x7f3718024360 osd=7 3 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 15 client.53351.objecter _session_command_op_assign 7 16 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 4 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 10 client.53351.objecter _send_command 16 2024-02-10T08:40:00.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.330+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3718024f80 con 0x7f3718024a20 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.331+0000 7f3728ff9700 1 -- 172.21.15.107:0/1165078207 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3718024f80 con 0x7f3718024a20 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.331+0000 7f3728ff9700 10 client.53351.objecter ms_dispatch 0x7f372c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.331+0000 7f3728ff9700 10 client.53351.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.331+0000 7f3728ff9700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 4 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.331+0000 7f3728ff9700 15 client.53351.objecter _session_command_op_remove 7 16 2024-02-10T08:40:00.332 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 librados: watch_flush enter 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 librados: watch_flush exit 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.0 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6801/3321596972 conn(0x7f3718004530 legacy=0x7f3718002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180051a0 osd=0 1 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.1 2024-02-10T08:40:00.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6805/133516126 conn(0x7f3718004150 legacy=0x7f371800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003fe0 osd=1 1 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.2 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6809/3575881728 conn(0x7f371800d320 legacy=0x7f371800d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f37180035f0 osd=2 1 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.3 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6813/2871825691 conn(0x7f3718011eb0 legacy=0x7f37180142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718003470 osd=3 1 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.4 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.195:6800/2499671285 conn(0x7f371800ce80 legacy=0x7f3718015ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e270 osd=4 1 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.5 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.195:6804/4178321486 conn(0x7f37180167c0 legacy=0x7f371801d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f371800e570 osd=5 1 2024-02-10T08:40:00.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.6 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.195:6808/1417479017 conn(0x7f371801f030 legacy=0x7f3718021420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718016fa0 osd=6 1 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter close_session for osd.7 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.195:6812/1934176928 conn(0x7f3718024a20 legacy=0x7f3718022bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter put_session s=0x7f3718024360 osd=7 1 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 20 client.53351.objecter shutdown clearing up homeless session... 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 10 client.53351.objecter successfully canceled tick 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 reap_dead start 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 reap_dead start 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 reap_dead start 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 reap_dead start 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f372bfff700 1 -- 172.21.15.107:0/1165078207 reap_dead start 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3714063c80 legacy=0x7f3714066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> v1:172.21.15.107:6790/0 conn(0x7f372c104350 legacy=0x7f372c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 shutdown_connections 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 >> 172.21.15.107:0/1165078207 conn(0x7f372c06c090 msgr2=0x7f372c06d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 shutdown_connections 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 -- 172.21.15.107:0/1165078207 wait complete. 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:00.332+0000 7f3731b29700 1 librados: shutdown 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:40:00.554 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:40:00.554 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:40:00.554 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:40:00.555 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:40:00.555 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:40:00.716 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-02-10T08:39:59.408571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007021,"num_objects":316,"num_object_clones":2,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1212,"num_read_kb":870,"num_write":983,"num_write_kb":4497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2302,"ondisk_log_size":2302,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":636,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2183796,"kb_used_data":45860,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":654032268,"statfs":{"total":671965249536,"available":669729042432,"internally_reserved":0,"allocated":46960640,"data_stored":31335824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001366"},"pg_stats":[{"pgid":"188.1f","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522683+0000","last_change":"2024-02-10T08:39:54.011817+0000","last_active":"2024-02-10T08:39:55.522683+0000","last_peered":"2024-02-10T08:39:55.522683+0000","last_clean":"2024-02-10T08:39:55.522683+0000","last_became_active":"2024-02-10T08:39:54.011588+0000","last_became_peered":"2024-02-10T08:39:54.011588+0000","last_unstale":"2024-02-10T08:39:55.522683+0000","last_undegraded":"2024-02-10T08:39:55.522683+0000","last_fullsized":"2024-02-10T08:39:55.522683+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"188.1e","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999438+0000","last_change":"2024-02-10T08:39:54.016382+0000","last_active":"2024-02-10T08:39:54.999438+0000","last_peered":"2024-02-10T08:39:54.999438+0000","last_clean":"2024-02-10T08:39:54.999438+0000","last_became_active":"2024-02-10T08:39:54.016138+0000","last_became_peered":"2024-02-10T08:39:54.016138+0000","last_unstale":"2024-02-10T08:39:54.999438+0000","last_undegraded":"2024-02-10T08:39:54.999438+0000","last_fullsized":"2024-02-10T08:39:54.999438+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.1d","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999243+0000","last_change":"2024-02-10T08:39:54.020842+0000","last_active":"2024-02-10T08:39:54.999243+0000","last_peered":"2024-02-10T08:39:54.999243+0000","last_clean":"2024-02-10T08:39:54.999243+0000","last_became_active":"2024-02-10T08:39:54.020548+0000","last_became_peered":"2024-02-10T08:39:54.020548+0000","last_unstale":"2024-02-10T08:39:54.999243+0000","last_undegraded":"2024-02-10T08:39:54.999243+0000","last_fullsized":"2024-02-10T08:39:54.999243+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"188.1c","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.017683+0000","last_change":"2024-02-10T08:39:54.017487+0000","last_active":"2024-02-10T08:39:54.017683+0000","last_peered":"2024-02-10T08:39:54.017683+0000","last_clean":"2024-02-10T08:39:54.017683+0000","last_became_active":"2024-02-10T08:39:54.017183+0000","last_became_peered":"2024-02-10T08:39:54.017183+0000","last_unstale":"2024-02-10T08:39:54.017683+0000","last_undegraded":"2024-02-10T08:39:54.017683+0000","last_fullsized":"2024-02-10T08:39:54.017683+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.1b","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997658+0000","last_change":"2024-02-10T08:39:54.017813+0000","last_active":"2024-02-10T08:39:54.997658+0000","last_peered":"2024-02-10T08:39:54.997658+0000","last_clean":"2024-02-10T08:39:54.997658+0000","last_became_active":"2024-02-10T08:39:54.017523+0000","last_became_peered":"2024-02-10T08:39:54.017523+0000","last_unstale":"2024-02-10T08:39:54.997658+0000","last_undegraded":"2024-02-10T08:39:54.997658+0000","last_fullsized":"2024-02-10T08:39:54.997658+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"188.1a","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521948+0000","last_change":"2024-02-10T08:39:54.012543+0000","last_active":"2024-02-10T08:39:55.521948+0000","last_peered":"2024-02-10T08:39:55.521948+0000","last_clean":"2024-02-10T08:39:55.521948+0000","last_became_active":"2024-02-10T08:39:54.012231+0000","last_became_peered":"2024-02-10T08:39:54.012231+0000","last_unstale":"2024-02-10T08:39:55.521948+0000","last_undegraded":"2024-02-10T08:39:55.521948+0000","last_fullsized":"2024-02-10T08:39:55.521948+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"188.19","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999903+0000","last_change":"2024-02-10T08:39:54.017248+0000","last_active":"2024-02-10T08:39:54.999903+0000","last_peered":"2024-02-10T08:39:54.999903+0000","last_clean":"2024-02-10T08:39:54.999903+0000","last_became_active":"2024-02-10T08:39:54.016946+0000","last_became_peered":"2024-02-10T08:39:54.016946+0000","last_unstale":"2024-02-10T08:39:54.999903+0000","last_undegraded":"2024-02-10T08:39:54.999903+0000","last_fullsized":"2024-02-10T08:39:54.999903+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.18","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997847+0000","last_change":"2024-02-10T08:39:54.020760+0000","last_active":"2024-02-10T08:39:54.997847+0000","last_peered":"2024-02-10T08:39:54.997847+0000","last_clean":"2024-02-10T08:39:54.997847+0000","last_became_active":"2024-02-10T08:39:54.020459+0000","last_became_peered":"2024-02-10T08:39:54.020459+0000","last_unstale":"2024-02-10T08:39:54.997847+0000","last_undegraded":"2024-02-10T08:39:54.997847+0000","last_fullsized":"2024-02-10T08:39:54.997847+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"188.17","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.013302+0000","last_change":"2024-02-10T08:39:54.013038+0000","last_active":"2024-02-10T08:39:54.013302+0000","last_peered":"2024-02-10T08:39:54.013302+0000","last_clean":"2024-02-10T08:39:54.013302+0000","last_became_active":"2024-02-10T08:39:54.012766+0000","last_became_peered":"2024-02-10T08:39:54.012766+0000","last_unstale":"2024-02-10T08:39:54.013302+0000","last_undegraded":"2024-02-10T08:39:54.013302+0000","last_fullsized":"2024-02-10T08:39:54.013302+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.16","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998941+0000","last_change":"2024-02-10T08:39:54.018239+0000","last_active":"2024-02-10T08:39:54.998941+0000","last_peered":"2024-02-10T08:39:54.998941+0000","last_clean":"2024-02-10T08:39:54.998941+0000","last_became_active":"2024-02-10T08:39:54.017948+0000","last_became_peered":"2024-02-10T08:39:54.017948+0000","last_unstale":"2024-02-10T08:39:54.998941+0000","last_undegraded":"2024-02-10T08:39:54.998941+0000","last_fullsized":"2024-02-10T08:39:54.998941+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.15","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581476+0000","last_change":"2024-02-10T08:39:54.014145+0000","last_active":"2024-02-10T08:39:55.581476+0000","last_peered":"2024-02-10T08:39:55.581476+0000","last_clean":"2024-02-10T08:39:55.581476+0000","last_became_active":"2024-02-10T08:39:54.013913+0000","last_became_peered":"2024-02-10T08:39:54.013913+0000","last_unstale":"2024-02-10T08:39:55.581476+0000","last_undegraded":"2024-02-10T08:39:55.581476+0000","last_fullsized":"2024-02-10T08:39:55.581476+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"188.14","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521925+0000","last_change":"2024-02-10T08:39:54.012301+0000","last_active":"2024-02-10T08:39:55.521925+0000","last_peered":"2024-02-10T08:39:55.521925+0000","last_clean":"2024-02-10T08:39:55.521925+0000","last_became_active":"2024-02-10T08:39:54.012092+0000","last_became_peered":"2024-02-10T08:39:54.012092+0000","last_unstale":"2024-02-10T08:39:55.521925+0000","last_undegraded":"2024-02-10T08:39:55.521925+0000","last_fullsized":"2024-02-10T08:39:55.521925+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"188.13","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581590+0000","last_change":"2024-02-10T08:39:54.009503+0000","last_active":"2024-02-10T08:39:55.581590+0000","last_peered":"2024-02-10T08:39:55.581590+0000","last_clean":"2024-02-10T08:39:55.581590+0000","last_became_active":"2024-02-10T08:39:54.009326+0000","last_became_peered":"2024-02-10T08:39:54.009326+0000","last_unstale":"2024-02-10T08:39:55.581590+0000","last_undegraded":"2024-02-10T08:39:55.581590+0000","last_fullsized":"2024-02-10T08:39:55.581590+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"188.12","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997785+0000","last_change":"2024-02-10T08:39:54.016085+0000","last_active":"2024-02-10T08:39:54.997785+0000","last_peered":"2024-02-10T08:39:54.997785+0000","last_clean":"2024-02-10T08:39:54.997785+0000","last_became_active":"2024-02-10T08:39:54.015780+0000","last_became_peered":"2024-02-10T08:39:54.015780+0000","last_unstale":"2024-02-10T08:39:54.997785+0000","last_undegraded":"2024-02-10T08:39:54.997785+0000","last_fullsized":"2024-02-10T08:39:54.997785+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"188.11","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997558+0000","last_change":"2024-02-10T08:39:54.015168+0000","last_active":"2024-02-10T08:39:54.997558+0000","last_peered":"2024-02-10T08:39:54.997558+0000","last_clean":"2024-02-10T08:39:54.997558+0000","last_became_active":"2024-02-10T08:39:54.014723+0000","last_became_peered":"2024-02-10T08:39:54.014723+0000","last_unstale":"2024-02-10T08:39:54.997558+0000","last_undegraded":"2024-02-10T08:39:54.997558+0000","last_fullsized":"2024-02-10T08:39:54.997558+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"188.10","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997496+0000","last_change":"2024-02-10T08:39:54.016074+0000","last_active":"2024-02-10T08:39:54.997496+0000","last_peered":"2024-02-10T08:39:54.997496+0000","last_clean":"2024-02-10T08:39:54.997496+0000","last_became_active":"2024-02-10T08:39:54.015734+0000","last_became_peered":"2024-02-10T08:39:54.015734+0000","last_unstale":"2024-02-10T08:39:54.997496+0000","last_undegraded":"2024-02-10T08:39:54.997496+0000","last_fullsized":"2024-02-10T08:39:54.997496+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"188.f","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.009182+0000","last_change":"2024-02-10T08:39:54.008504+0000","last_active":"2024-02-10T08:39:54.009182+0000","last_peered":"2024-02-10T08:39:54.009182+0000","last_clean":"2024-02-10T08:39:54.009182+0000","last_became_active":"2024-02-10T08:39:54.008190+0000","last_became_peered":"2024-02-10T08:39:54.008190+0000","last_unstale":"2024-02-10T08:39:54.009182+0000","last_undegraded":"2024-02-10T08:39:54.009182+0000","last_fullsized":"2024-02-10T08:39:54.009182+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.e","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581839+0000","last_change":"2024-02-10T08:39:54.015455+0000","last_active":"2024-02-10T08:39:55.581839+0000","last_peered":"2024-02-10T08:39:55.581839+0000","last_clean":"2024-02-10T08:39:55.581839+0000","last_became_active":"2024-02-10T08:39:54.015173+0000","last_became_peered":"2024-02-10T08:39:54.015173+0000","last_unstale":"2024-02-10T08:39:55.581839+0000","last_undegraded":"2024-02-10T08:39:55.581839+0000","last_fullsized":"2024-02-10T08:39:55.581839+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"188.d","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999643+0000","last_change":"2024-02-10T08:39:54.016334+0000","last_active":"2024-02-10T08:39:54.999643+0000","last_peered":"2024-02-10T08:39:54.999643+0000","last_clean":"2024-02-10T08:39:54.999643+0000","last_became_active":"2024-02-10T08:39:54.015698+0000","last_became_peered":"2024-02-10T08:39:54.015698+0000","last_unstale":"2024-02-10T08:39:54.999643+0000","last_undegraded":"2024-02-10T08:39:54.999643+0000","last_fullsized":"2024-02-10T08:39:54.999643+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"188.c","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998543+0000","last_change":"2024-02-10T08:39:54.019913+0000","last_active":"2024-02-10T08:39:54.998543+0000","last_peered":"2024-02-10T08:39:54.998543+0000","last_clean":"2024-02-10T08:39:54.998543+0000","last_became_active":"2024-02-10T08:39:54.019568+0000","last_became_peered":"2024-02-10T08:39:54.019568+0000","last_unstale":"2024-02-10T08:39:54.998543+0000","last_undegraded":"2024-02-10T08:39:54.998543+0000","last_fullsized":"2024-02-10T08:39:54.998543+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.b","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999879+0000","last_change":"2024-02-10T08:39:54.008164+0000","last_active":"2024-02-10T08:39:54.999879+0000","last_peered":"2024-02-10T08:39:54.999879+0000","last_clean":"2024-02-10T08:39:54.999879+0000","last_became_active":"2024-02-10T08:39:54.007867+0000","last_became_peered":"2024-02-10T08:39:54.007867+0000","last_unstale":"2024-02-10T08:39:54.999879+0000","last_undegraded":"2024-02-10T08:39:54.999879+0000","last_fullsized":"2024-02-10T08:39:54.999879+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.a","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.013504+0000","last_change":"2024-02-10T08:39:54.013366+0000","last_active":"2024-02-10T08:39:54.013504+0000","last_peered":"2024-02-10T08:39:54.013504+0000","last_clean":"2024-02-10T08:39:54.013504+0000","last_became_active":"2024-02-10T08:39:54.013107+0000","last_became_peered":"2024-02-10T08:39:54.013107+0000","last_unstale":"2024-02-10T08:39:54.013504+0000","last_undegraded":"2024-02-10T08:39:54.013504+0000","last_fullsized":"2024-02-10T08:39:54.013504+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.9","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997563+0000","last_change":"2024-02-10T08:39:54.014995+0000","last_active":"2024-02-10T08:39:54.997563+0000","last_peered":"2024-02-10T08:39:54.997563+0000","last_clean":"2024-02-10T08:39:54.997563+0000","last_became_active":"2024-02-10T08:39:54.014311+0000","last_became_peered":"2024-02-10T08:39:54.014311+0000","last_unstale":"2024-02-10T08:39:54.997563+0000","last_undegraded":"2024-02-10T08:39:54.997563+0000","last_fullsized":"2024-02-10T08:39:54.997563+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"188.8","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.017303+0000","last_change":"2024-02-10T08:39:54.016919+0000","last_active":"2024-02-10T08:39:54.017303+0000","last_peered":"2024-02-10T08:39:54.017303+0000","last_clean":"2024-02-10T08:39:54.017303+0000","last_became_active":"2024-02-10T08:39:54.016649+0000","last_became_peered":"2024-02-10T08:39:54.016649+0000","last_unstale":"2024-02-10T08:39:54.017303+0000","last_undegraded":"2024-02-10T08:39:54.017303+0000","last_fullsized":"2024-02-10T08:39:54.017303+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.7","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777528+0000","last_change":"2024-02-10T08:39:54.005503+0000","last_active":"2024-02-10T08:39:55.777528+0000","last_peered":"2024-02-10T08:39:55.777528+0000","last_clean":"2024-02-10T08:39:55.777528+0000","last_became_active":"2024-02-10T08:39:54.004806+0000","last_became_peered":"2024-02-10T08:39:54.004806+0000","last_unstale":"2024-02-10T08:39:55.777528+0000","last_undegraded":"2024-02-10T08:39:55.777528+0000","last_fullsized":"2024-02-10T08:39:55.777528+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"188.6","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999031+0000","last_change":"2024-02-10T08:39:54.016283+0000","last_active":"2024-02-10T08:39:54.999031+0000","last_peered":"2024-02-10T08:39:54.999031+0000","last_clean":"2024-02-10T08:39:54.999031+0000","last_became_active":"2024-02-10T08:39:54.016117+0000","last_became_peered":"2024-02-10T08:39:54.016117+0000","last_unstale":"2024-02-10T08:39:54.999031+0000","last_undegraded":"2024-02-10T08:39:54.999031+0000","last_fullsized":"2024-02-10T08:39:54.999031+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.5","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999932+0000","last_change":"2024-02-10T08:39:54.018504+0000","last_active":"2024-02-10T08:39:54.999932+0000","last_peered":"2024-02-10T08:39:54.999932+0000","last_clean":"2024-02-10T08:39:54.999932+0000","last_became_active":"2024-02-10T08:39:54.018011+0000","last_became_peered":"2024-02-10T08:39:54.018011+0000","last_unstale":"2024-02-10T08:39:54.999932+0000","last_undegraded":"2024-02-10T08:39:54.999932+0000","last_fullsized":"2024-02-10T08:39:54.999932+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"188.4","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.013281+0000","last_change":"2024-02-10T08:39:54.012307+0000","last_active":"2024-02-10T08:39:54.013281+0000","last_peered":"2024-02-10T08:39:54.013281+0000","last_clean":"2024-02-10T08:39:54.013281+0000","last_became_active":"2024-02-10T08:39:54.012009+0000","last_became_peered":"2024-02-10T08:39:54.012009+0000","last_unstale":"2024-02-10T08:39:54.013281+0000","last_undegraded":"2024-02-10T08:39:54.013281+0000","last_fullsized":"2024-02-10T08:39:54.013281+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"188.3","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776671+0000","last_change":"2024-02-10T08:39:54.007657+0000","last_active":"2024-02-10T08:39:55.776671+0000","last_peered":"2024-02-10T08:39:55.776671+0000","last_clean":"2024-02-10T08:39:55.776671+0000","last_became_active":"2024-02-10T08:39:54.007473+0000","last_became_peered":"2024-02-10T08:39:54.007473+0000","last_unstale":"2024-02-10T08:39:55.776671+0000","last_undegraded":"2024-02-10T08:39:55.776671+0000","last_fullsized":"2024-02-10T08:39:55.776671+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"188.2","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581109+0000","last_change":"2024-02-10T08:39:54.014974+0000","last_active":"2024-02-10T08:39:55.581109+0000","last_peered":"2024-02-10T08:39:55.581109+0000","last_clean":"2024-02-10T08:39:55.581109+0000","last_became_active":"2024-02-10T08:39:54.014653+0000","last_became_peered":"2024-02-10T08:39:54.014653+0000","last_unstale":"2024-02-10T08:39:55.581109+0000","last_undegraded":"2024-02-10T08:39:55.581109+0000","last_fullsized":"2024-02-10T08:39:55.581109+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"188.1","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776254+0000","last_change":"2024-02-10T08:39:54.007160+0000","last_active":"2024-02-10T08:39:55.776254+0000","last_peered":"2024-02-10T08:39:55.776254+0000","last_clean":"2024-02-10T08:39:55.776254+0000","last_became_active":"2024-02-10T08:39:54.006880+0000","last_became_peered":"2024-02-10T08:39:54.006880+0000","last_unstale":"2024-02-10T08:39:55.776254+0000","last_undegraded":"2024-02-10T08:39:55.776254+0000","last_fullsized":"2024-02-10T08:39:55.776254+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"188.0","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997912+0000","last_change":"2024-02-10T08:39:54.018160+0000","last_active":"2024-02-10T08:39:54.997912+0000","last_peered":"2024-02-10T08:39:54.997912+0000","last_clean":"2024-02-10T08:39:54.997912+0000","last_became_active":"2024-02-10T08:39:54.017871+0000","last_became_peered":"2024-02-10T08:39:54.017871+0000","last_unstale":"2024-02-10T08:39:54.997912+0000","last_undegraded":"2024-02-10T08:39:54.997912+0000","last_fullsized":"2024-02-10T08:39:54.997912+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":299,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:52.986532+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.986532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000096+0000","last_change":"2024-02-10T08:39:32.971969+0000","last_active":"2024-02-10T08:39:55.000096+0000","last_peered":"2024-02-10T08:39:55.000096+0000","last_clean":"2024-02-10T08:39:55.000096+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:39:55.000096+0000","last_undegraded":"2024-02-10T08:39:55.000096+0000","last_fullsized":"2024-02-10T08:39:55.000096+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"236'27","reported_seq":273,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581272+0000","last_change":"2024-02-10T08:39:38.573741+0000","last_active":"2024-02-10T08:39:55.581272+0000","last_peered":"2024-02-10T08:39:55.581272+0000","last_clean":"2024-02-10T08:39:55.581272+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:39:55.581272+0000","last_undegraded":"2024-02-10T08:39:55.581272+0000","last_fullsized":"2024-02-10T08:39:55.581272+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"250'39","reported_seq":272,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581085+0000","last_change":"2024-02-10T08:39:40.483224+0000","last_active":"2024-02-10T08:39:55.581085+0000","last_peered":"2024-02-10T08:39:55.581085+0000","last_clean":"2024-02-10T08:39:55.581085+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:39:55.581085+0000","last_undegraded":"2024-02-10T08:39:55.581085+0000","last_fullsized":"2024-02-10T08:39:55.581085+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"231'107","reported_seq":341,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997362+0000","last_change":"2024-02-10T08:39:32.974249+0000","last_active":"2024-02-10T08:39:54.997362+0000","last_peered":"2024-02-10T08:39:54.997362+0000","last_clean":"2024-02-10T08:39:54.997362+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:39:54.997362+0000","last_undegraded":"2024-02-10T08:39:54.997362+0000","last_fullsized":"2024-02-10T08:39:54.997362+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"222'34","reported_seq":250,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582003+0000","last_change":"2024-02-10T08:39:53.632526+0000","last_active":"2024-02-10T08:39:55.582003+0000","last_peered":"2024-02-10T08:39:55.582003+0000","last_clean":"2024-02-10T08:39:55.582003+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:39:55.582003+0000","last_undegraded":"2024-02-10T08:39:55.582003+0000","last_fullsized":"2024-02-10T08:39:55.582003+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"226'71","reported_seq":304,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998896+0000","last_change":"2024-02-10T08:39:49.450775+0000","last_active":"2024-02-10T08:39:54.998896+0000","last_peered":"2024-02-10T08:39:54.998896+0000","last_clean":"2024-02-10T08:39:54.998896+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:39:54.998896+0000","last_undegraded":"2024-02-10T08:39:54.998896+0000","last_fullsized":"2024-02-10T08:39:54.998896+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":289,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581660+0000","last_change":"2024-02-10T08:39:52.583051+0000","last_active":"2024-02-10T08:39:55.581660+0000","last_peered":"2024-02-10T08:39:55.581660+0000","last_clean":"2024-02-10T08:39:55.581660+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:39:55.581660+0000","last_undegraded":"2024-02-10T08:39:55.581660+0000","last_fullsized":"2024-02-10T08:39:55.581660+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"250'22","reported_seq":266,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581684+0000","last_change":"2024-02-10T08:39:42.510684+0000","last_active":"2024-02-10T08:39:55.581684+0000","last_peered":"2024-02-10T08:39:55.581684+0000","last_clean":"2024-02-10T08:39:55.581684+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:39:55.581684+0000","last_undegraded":"2024-02-10T08:39:55.581684+0000","last_fullsized":"2024-02-10T08:39:55.581684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"250'101","reported_seq":327,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997884+0000","last_change":"2024-02-10T08:39:32.974027+0000","last_active":"2024-02-10T08:39:54.997884+0000","last_peered":"2024-02-10T08:39:54.997884+0000","last_clean":"2024-02-10T08:39:54.997884+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:39:54.997884+0000","last_undegraded":"2024-02-10T08:39:54.997884+0000","last_fullsized":"2024-02-10T08:39:54.997884+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"228'22","reported_seq":265,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581246+0000","last_change":"2024-02-10T08:39:47.635717+0000","last_active":"2024-02-10T08:39:55.581246+0000","last_peered":"2024-02-10T08:39:55.581246+0000","last_clean":"2024-02-10T08:39:55.581246+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:39:55.581246+0000","last_undegraded":"2024-02-10T08:39:55.581246+0000","last_fullsized":"2024-02-10T08:39:55.581246+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"299'110","reported_seq":326,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999287+0000","last_change":"2024-02-10T08:39:53.048253+0000","last_active":"2024-02-10T08:39:54.999287+0000","last_peered":"2024-02-10T08:39:54.999287+0000","last_clean":"2024-02-10T08:39:54.999287+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:39:54.999287+0000","last_undegraded":"2024-02-10T08:39:54.999287+0000","last_fullsized":"2024-02-10T08:39:54.999287+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"236'85","reported_seq":320,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581108+0000","last_change":"2024-02-10T08:39:50.630114+0000","last_active":"2024-02-10T08:39:55.581108+0000","last_peered":"2024-02-10T08:39:55.581108+0000","last_clean":"2024-02-10T08:39:55.581108+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:39:55.581108+0000","last_undegraded":"2024-02-10T08:39:55.581108+0000","last_fullsized":"2024-02-10T08:39:55.581108+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"231'93","reported_seq":318,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999937+0000","last_change":"2024-02-10T08:39:43.510896+0000","last_active":"2024-02-10T08:39:54.999937+0000","last_peered":"2024-02-10T08:39:54.999937+0000","last_clean":"2024-02-10T08:39:54.999937+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:39:54.999937+0000","last_undegraded":"2024-02-10T08:39:54.999937+0000","last_fullsized":"2024-02-10T08:39:54.999937+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'93","last_scrub_stamp":"2024-02-10T08:39:43.510789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.510789+0000","log_size":93,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":72,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":272,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777461+0000","last_change":"2024-02-10T08:39:54.894186+0000","last_active":"2024-02-10T08:39:55.777461+0000","last_peered":"2024-02-10T08:39:55.777461+0000","last_clean":"2024-02-10T08:39:55.777461+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:39:55.777461+0000","last_undegraded":"2024-02-10T08:39:55.777461+0000","last_fullsized":"2024-02-10T08:39:55.777461+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"287'56","reported_seq":286,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999235+0000","last_change":"2024-02-10T08:39:44.469992+0000","last_active":"2024-02-10T08:39:54.999235+0000","last_peered":"2024-02-10T08:39:54.999235+0000","last_clean":"2024-02-10T08:39:54.999235+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:39:54.999235+0000","last_undegraded":"2024-02-10T08:39:54.999235+0000","last_fullsized":"2024-02-10T08:39:54.999235+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-02-10T08:39:44.469923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.469923+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"226'68","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000730+0000","last_change":"2024-02-10T08:39:32.974869+0000","last_active":"2024-02-10T08:39:55.000730+0000","last_peered":"2024-02-10T08:39:55.000730+0000","last_clean":"2024-02-10T08:39:55.000730+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:39:55.000730+0000","last_undegraded":"2024-02-10T08:39:55.000730+0000","last_fullsized":"2024-02-10T08:39:55.000730+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"295'55","reported_seq":290,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.008006+0000","last_change":"2024-02-10T08:39:33.014018+0000","last_active":"2024-02-10T08:39:54.008006+0000","last_peered":"2024-02-10T08:39:54.008006+0000","last_clean":"2024-02-10T08:39:54.008006+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:39:54.008006+0000","last_undegraded":"2024-02-10T08:39:54.008006+0000","last_fullsized":"2024-02-10T08:39:54.008006+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'54","last_scrub_stamp":"2024-02-10T08:39:12.673905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.673905+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000054,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":3924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":339,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521594+0000","last_change":"2024-02-10T08:39:46.646522+0000","last_active":"2024-02-10T08:39:55.521594+0000","last_peered":"2024-02-10T08:39:55.521594+0000","last_clean":"2024-02-10T08:39:55.521594+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:39:55.521594+0000","last_undegraded":"2024-02-10T08:39:55.521594+0000","last_fullsized":"2024-02-10T08:39:55.521594+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:39:46.646454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.646454+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"258'103","reported_seq":326,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002321+0000","last_change":"2024-02-10T08:39:41.578640+0000","last_active":"2024-02-10T08:39:54.002321+0000","last_peered":"2024-02-10T08:39:54.002321+0000","last_clean":"2024-02-10T08:39:54.002321+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:39:54.002321+0000","last_undegraded":"2024-02-10T08:39:54.002321+0000","last_fullsized":"2024-02-10T08:39:54.002321+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":314,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.527944+0000","last_change":"2024-02-10T08:39:55.527916+0000","last_active":"2024-02-10T08:39:55.527944+0000","last_peered":"2024-02-10T08:39:55.527944+0000","last_clean":"2024-02-10T08:39:55.527944+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:39:55.527944+0000","last_undegraded":"2024-02-10T08:39:55.527944+0000","last_fullsized":"2024-02-10T08:39:55.527944+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:39:55.527854+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"215'85","reported_seq":316,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522498+0000","last_change":"2024-02-10T08:39:54.516959+0000","last_active":"2024-02-10T08:39:55.522498+0000","last_peered":"2024-02-10T08:39:55.522498+0000","last_clean":"2024-02-10T08:39:55.522498+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:39:55.522498+0000","last_undegraded":"2024-02-10T08:39:55.522498+0000","last_fullsized":"2024-02-10T08:39:55.522498+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'85","last_scrub_stamp":"2024-02-10T08:39:54.516887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.516887+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":317,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777435+0000","last_change":"2024-02-10T08:39:43.711787+0000","last_active":"2024-02-10T08:39:55.777435+0000","last_peered":"2024-02-10T08:39:55.777435+0000","last_clean":"2024-02-10T08:39:55.777435+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:39:55.777435+0000","last_undegraded":"2024-02-10T08:39:55.777435+0000","last_fullsized":"2024-02-10T08:39:55.777435+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"230'17","reported_seq":253,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581588+0000","last_change":"2024-02-10T08:39:32.970028+0000","last_active":"2024-02-10T08:39:55.581588+0000","last_peered":"2024-02-10T08:39:55.581588+0000","last_clean":"2024-02-10T08:39:55.581588+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:39:55.581588+0000","last_undegraded":"2024-02-10T08:39:55.581588+0000","last_fullsized":"2024-02-10T08:39:55.581588+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":270,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.775997+0000","last_change":"2024-02-10T08:39:49.714905+0000","last_active":"2024-02-10T08:39:55.775997+0000","last_peered":"2024-02-10T08:39:55.775997+0000","last_clean":"2024-02-10T08:39:55.775997+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:39:55.775997+0000","last_undegraded":"2024-02-10T08:39:55.775997+0000","last_fullsized":"2024-02-10T08:39:55.775997+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"228'68","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581412+0000","last_change":"2024-02-10T08:39:32.968736+0000","last_active":"2024-02-10T08:39:55.581412+0000","last_peered":"2024-02-10T08:39:55.581412+0000","last_clean":"2024-02-10T08:39:55.581412+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:39:55.581412+0000","last_undegraded":"2024-02-10T08:39:55.581412+0000","last_fullsized":"2024-02-10T08:39:55.581412+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"214'68","reported_seq":287,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998630+0000","last_change":"2024-02-10T08:39:32.970019+0000","last_active":"2024-02-10T08:39:54.998630+0000","last_peered":"2024-02-10T08:39:54.998630+0000","last_clean":"2024-02-10T08:39:54.998630+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:39:54.998630+0000","last_undegraded":"2024-02-10T08:39:54.998630+0000","last_fullsized":"2024-02-10T08:39:54.998630+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"208'34","reported_seq":276,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999389+0000","last_change":"2024-02-10T08:39:32.971984+0000","last_active":"2024-02-10T08:39:54.999389+0000","last_peered":"2024-02-10T08:39:54.999389+0000","last_clean":"2024-02-10T08:39:54.999389+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:39:54.999389+0000","last_undegraded":"2024-02-10T08:39:54.999389+0000","last_fullsized":"2024-02-10T08:39:54.999389+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"224'34","reported_seq":253,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581055+0000","last_change":"2024-02-10T08:39:54.602656+0000","last_active":"2024-02-10T08:39:55.581055+0000","last_peered":"2024-02-10T08:39:55.581055+0000","last_clean":"2024-02-10T08:39:55.581055+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:39:55.581055+0000","last_undegraded":"2024-02-10T08:39:55.581055+0000","last_fullsized":"2024-02-10T08:39:55.581055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"200'34","reported_seq":280,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.006405+0000","last_change":"2024-02-10T08:39:33.014676+0000","last_active":"2024-02-10T08:39:54.006405+0000","last_peered":"2024-02-10T08:39:54.006405+0000","last_clean":"2024-02-10T08:39:54.006405+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:39:54.006405+0000","last_undegraded":"2024-02-10T08:39:54.006405+0000","last_fullsized":"2024-02-10T08:39:54.006405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'34","last_scrub_stamp":"2024-02-10T08:39:04.785239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.785239+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":239,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999330+0000","last_change":"2024-02-10T08:39:38.041298+0000","last_active":"2024-02-10T08:39:54.999330+0000","last_peered":"2024-02-10T08:39:54.999330+0000","last_clean":"2024-02-10T08:39:54.999330+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:39:54.999330+0000","last_undegraded":"2024-02-10T08:39:54.999330+0000","last_fullsized":"2024-02-10T08:39:54.999330+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:39:38.041185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.041185+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"202'17","reported_seq":237,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777288+0000","last_change":"2024-02-10T08:39:33.016451+0000","last_active":"2024-02-10T08:39:55.777288+0000","last_peered":"2024-02-10T08:39:55.777288+0000","last_clean":"2024-02-10T08:39:55.777288+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:39:55.777288+0000","last_undegraded":"2024-02-10T08:39:55.777288+0000","last_fullsized":"2024-02-10T08:39:55.777288+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"222'34","reported_seq":268,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000567+0000","last_change":"2024-02-10T08:39:32.974056+0000","last_active":"2024-02-10T08:39:55.000567+0000","last_peered":"2024-02-10T08:39:55.000567+0000","last_clean":"2024-02-10T08:39:55.000567+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:39:55.000567+0000","last_undegraded":"2024-02-10T08:39:55.000567+0000","last_fullsized":"2024-02-10T08:39:55.000567+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":373,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521894+0000","last_change":"2024-02-10T08:38:21.488646+0000","last_active":"2024-02-10T08:39:55.521894+0000","last_peered":"2024-02-10T08:39:55.521894+0000","last_clean":"2024-02-10T08:39:55.521894+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:39:55.521894+0000","last_undegraded":"2024-02-10T08:39:55.521894+0000","last_fullsized":"2024-02-10T08:39:55.521894+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:38:21.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:21.488563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":298,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.001715+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:39:54.001715+0000","last_peered":"2024-02-10T08:39:54.001715+0000","last_clean":"2024-02-10T08:39:54.001715+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:39:54.001715+0000","last_undegraded":"2024-02-10T08:39:54.001715+0000","last_fullsized":"2024-02-10T08:39:54.001715+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":265,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999116+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:39:54.999116+0000","last_peered":"2024-02-10T08:39:54.999116+0000","last_clean":"2024-02-10T08:39:54.999116+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:39:54.999116+0000","last_undegraded":"2024-02-10T08:39:54.999116+0000","last_fullsized":"2024-02-10T08:39:54.999116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":271,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002207+0000","last_change":"2024-02-10T08:37:44.597397+0000","last_active":"2024-02-10T08:39:54.002207+0000","last_peered":"2024-02-10T08:39:54.002207+0000","last_clean":"2024-02-10T08:39:54.002207+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:39:54.002207+0000","last_undegraded":"2024-02-10T08:39:54.002207+0000","last_fullsized":"2024-02-10T08:39:54.002207+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.597335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.597335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000910+0000","last_change":"2024-02-10T08:38:33.920702+0000","last_active":"2024-02-10T08:39:55.000910+0000","last_peered":"2024-02-10T08:39:55.000910+0000","last_clean":"2024-02-10T08:39:55.000910+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:39:55.000910+0000","last_undegraded":"2024-02-10T08:39:55.000910+0000","last_fullsized":"2024-02-10T08:39:55.000910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.920630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.920630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":242,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998144+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:39:54.998144+0000","last_peered":"2024-02-10T08:39:54.998144+0000","last_clean":"2024-02-10T08:39:54.998144+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:39:54.998144+0000","last_undegraded":"2024-02-10T08:39:54.998144+0000","last_fullsized":"2024-02-10T08:39:54.998144+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":270,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000270+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:39:55.000270+0000","last_peered":"2024-02-10T08:39:55.000270+0000","last_clean":"2024-02-10T08:39:55.000270+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:39:55.000270+0000","last_undegraded":"2024-02-10T08:39:55.000270+0000","last_fullsized":"2024-02-10T08:39:55.000270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":284,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998847+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:39:54.998847+0000","last_peered":"2024-02-10T08:39:54.998847+0000","last_clean":"2024-02-10T08:39:54.998847+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:39:54.998847+0000","last_undegraded":"2024-02-10T08:39:54.998847+0000","last_fullsized":"2024-02-10T08:39:54.998847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521664+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:39:55.521664+0000","last_peered":"2024-02-10T08:39:55.521664+0000","last_clean":"2024-02-10T08:39:55.521664+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:39:55.521664+0000","last_undegraded":"2024-02-10T08:39:55.521664+0000","last_fullsized":"2024-02-10T08:39:55.521664+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":211,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.007113+0000","last_change":"2024-02-10T08:38:26.677780+0000","last_active":"2024-02-10T08:39:54.007113+0000","last_peered":"2024-02-10T08:39:54.007113+0000","last_clean":"2024-02-10T08:39:54.007113+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:39:54.007113+0000","last_undegraded":"2024-02-10T08:39:54.007113+0000","last_fullsized":"2024-02-10T08:39:54.007113+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.677701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.677701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997978+0000","last_change":"2024-02-10T08:37:39.547805+0000","last_active":"2024-02-10T08:39:54.997978+0000","last_peered":"2024-02-10T08:39:54.997978+0000","last_clean":"2024-02-10T08:39:54.997978+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:39:54.997978+0000","last_undegraded":"2024-02-10T08:39:54.997978+0000","last_fullsized":"2024-02-10T08:39:54.997978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:37:39.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.547736+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.001209+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:39:55.001209+0000","last_peered":"2024-02-10T08:39:55.001209+0000","last_clean":"2024-02-10T08:39:55.001209+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:39:55.001209+0000","last_undegraded":"2024-02-10T08:39:55.001209+0000","last_fullsized":"2024-02-10T08:39:55.001209+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998014+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:39:54.998014+0000","last_peered":"2024-02-10T08:39:54.998014+0000","last_clean":"2024-02-10T08:39:54.998014+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:39:54.998014+0000","last_undegraded":"2024-02-10T08:39:54.998014+0000","last_fullsized":"2024-02-10T08:39:54.998014+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521670+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:39:55.521670+0000","last_peered":"2024-02-10T08:39:55.521670+0000","last_clean":"2024-02-10T08:39:55.521670+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:39:55.521670+0000","last_undegraded":"2024-02-10T08:39:55.521670+0000","last_fullsized":"2024-02-10T08:39:55.521670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522559+0000","last_change":"2024-02-10T08:39:44.585775+0000","last_active":"2024-02-10T08:39:55.522559+0000","last_peered":"2024-02-10T08:39:55.522559+0000","last_clean":"2024-02-10T08:39:55.522559+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:39:55.522559+0000","last_undegraded":"2024-02-10T08:39:55.522559+0000","last_fullsized":"2024-02-10T08:39:55.522559+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:44.585674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.585674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582153+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:39:55.582153+0000","last_peered":"2024-02-10T08:39:55.582153+0000","last_clean":"2024-02-10T08:39:55.582153+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:39:55.582153+0000","last_undegraded":"2024-02-10T08:39:55.582153+0000","last_fullsized":"2024-02-10T08:39:55.582153+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":104,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776827+0000","last_change":"2024-02-10T08:38:22.013062+0000","last_active":"2024-02-10T08:39:55.776827+0000","last_peered":"2024-02-10T08:39:55.776827+0000","last_clean":"2024-02-10T08:39:55.776827+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:39:55.776827+0000","last_undegraded":"2024-02-10T08:39:55.776827+0000","last_fullsized":"2024-02-10T08:39:55.776827+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998944+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:39:54.998944+0000","last_peered":"2024-02-10T08:39:54.998944+0000","last_clean":"2024-02-10T08:39:54.998944+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:39:54.998944+0000","last_undegraded":"2024-02-10T08:39:54.998944+0000","last_fullsized":"2024-02-10T08:39:54.998944+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.001067+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:39:55.001067+0000","last_peered":"2024-02-10T08:39:55.001067+0000","last_clean":"2024-02-10T08:39:55.001067+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:39:55.001067+0000","last_undegraded":"2024-02-10T08:39:55.001067+0000","last_fullsized":"2024-02-10T08:39:55.001067+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581080+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:39:55.581080+0000","last_peered":"2024-02-10T08:39:55.581080+0000","last_clean":"2024-02-10T08:39:55.581080+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:39:55.581080+0000","last_undegraded":"2024-02-10T08:39:55.581080+0000","last_fullsized":"2024-02-10T08:39:55.581080+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":283,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581880+0000","last_change":"2024-02-10T08:38:17.727465+0000","last_active":"2024-02-10T08:39:55.581880+0000","last_peered":"2024-02-10T08:39:55.581880+0000","last_clean":"2024-02-10T08:39:55.581880+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:39:55.581880+0000","last_undegraded":"2024-02-10T08:39:55.581880+0000","last_fullsized":"2024-02-10T08:39:55.581880+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:17.727387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:17.727387+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":270,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998996+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:39:54.998996+0000","last_peered":"2024-02-10T08:39:54.998996+0000","last_clean":"2024-02-10T08:39:54.998996+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:39:54.998996+0000","last_undegraded":"2024-02-10T08:39:54.998996+0000","last_fullsized":"2024-02-10T08:39:54.998996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":368,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999660+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:39:54.999660+0000","last_peered":"2024-02-10T08:39:54.999660+0000","last_clean":"2024-02-10T08:39:54.999660+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:39:54.999660+0000","last_undegraded":"2024-02-10T08:39:54.999660+0000","last_fullsized":"2024-02-10T08:39:54.999660+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582178+0000","last_change":"2024-02-10T08:38:25.851680+0000","last_active":"2024-02-10T08:39:55.582178+0000","last_peered":"2024-02-10T08:39:55.582178+0000","last_clean":"2024-02-10T08:39:55.582178+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:39:55.582178+0000","last_undegraded":"2024-02-10T08:39:55.582178+0000","last_fullsized":"2024-02-10T08:39:55.582178+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.851576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.851576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776169+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:39:55.776169+0000","last_peered":"2024-02-10T08:39:55.776169+0000","last_clean":"2024-02-10T08:39:55.776169+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:39:55.776169+0000","last_undegraded":"2024-02-10T08:39:55.776169+0000","last_fullsized":"2024-02-10T08:39:55.776169+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998093+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:39:54.998093+0000","last_peered":"2024-02-10T08:39:54.998093+0000","last_clean":"2024-02-10T08:39:54.998093+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:39:54.998093+0000","last_undegraded":"2024-02-10T08:39:54.998093+0000","last_fullsized":"2024-02-10T08:39:54.998093+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":278,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999750+0000","last_change":"2024-02-10T08:38:18.008913+0000","last_active":"2024-02-10T08:39:54.999750+0000","last_peered":"2024-02-10T08:39:54.999750+0000","last_clean":"2024-02-10T08:39:54.999750+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:39:54.999750+0000","last_undegraded":"2024-02-10T08:39:54.999750+0000","last_fullsized":"2024-02-10T08:39:54.999750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:18.008846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.008846+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":393,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582149+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:39:55.582149+0000","last_peered":"2024-02-10T08:39:55.582149+0000","last_clean":"2024-02-10T08:39:55.582149+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:39:55.582149+0000","last_undegraded":"2024-02-10T08:39:55.582149+0000","last_fullsized":"2024-02-10T08:39:55.582149+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":271,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.672423+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:39:54.672423+0000","last_peered":"2024-02-10T08:39:54.672423+0000","last_clean":"2024-02-10T08:39:54.672423+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:39:54.672423+0000","last_undegraded":"2024-02-10T08:39:54.672423+0000","last_fullsized":"2024-02-10T08:39:54.672423+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":211,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.008118+0000","last_change":"2024-02-10T08:39:05.788196+0000","last_active":"2024-02-10T08:39:54.008118+0000","last_peered":"2024-02-10T08:39:54.008118+0000","last_clean":"2024-02-10T08:39:54.008118+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:39:54.008118+0000","last_undegraded":"2024-02-10T08:39:54.008118+0000","last_fullsized":"2024-02-10T08:39:54.008118+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.788118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.788118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":287,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582009+0000","last_change":"2024-02-10T08:38:59.745198+0000","last_active":"2024-02-10T08:39:55.582009+0000","last_peered":"2024-02-10T08:39:55.582009+0000","last_clean":"2024-02-10T08:39:55.582009+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:39:55.582009+0000","last_undegraded":"2024-02-10T08:39:55.582009+0000","last_fullsized":"2024-02-10T08:39:55.582009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:38:59.745114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.745114+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521795+0000","last_change":"2024-02-10T08:38:33.363322+0000","last_active":"2024-02-10T08:39:55.521795+0000","last_peered":"2024-02-10T08:39:55.521795+0000","last_clean":"2024-02-10T08:39:55.521795+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:39:55.521795+0000","last_undegraded":"2024-02-10T08:39:55.521795+0000","last_fullsized":"2024-02-10T08:39:55.521795+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.363195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":324,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.006859+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:39:54.006859+0000","last_peered":"2024-02-10T08:39:54.006859+0000","last_clean":"2024-02-10T08:39:54.006859+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:39:54.006859+0000","last_undegraded":"2024-02-10T08:39:54.006859+0000","last_fullsized":"2024-02-10T08:39:54.006859+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":258,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776988+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:39:55.776988+0000","last_peered":"2024-02-10T08:39:55.776988+0000","last_clean":"2024-02-10T08:39:55.776988+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:39:55.776988+0000","last_undegraded":"2024-02-10T08:39:55.776988+0000","last_fullsized":"2024-02-10T08:39:55.776988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776567+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:39:55.776567+0000","last_peered":"2024-02-10T08:39:55.776567+0000","last_clean":"2024-02-10T08:39:55.776567+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:39:55.776567+0000","last_undegraded":"2024-02-10T08:39:55.776567+0000","last_fullsized":"2024-02-10T08:39:55.776567+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":355,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776334+0000","last_change":"2024-02-10T08:38:18.475575+0000","last_active":"2024-02-10T08:39:55.776334+0000","last_peered":"2024-02-10T08:39:55.776334+0000","last_clean":"2024-02-10T08:39:55.776334+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:39:55.776334+0000","last_undegraded":"2024-02-10T08:39:55.776334+0000","last_fullsized":"2024-02-10T08:39:55.776334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:38:18.475509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.475509+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":289,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581345+0000","last_change":"2024-02-10T08:38:31.840285+0000","last_active":"2024-02-10T08:39:55.581345+0000","last_peered":"2024-02-10T08:39:55.581345+0000","last_clean":"2024-02-10T08:39:55.581345+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:39:55.581345+0000","last_undegraded":"2024-02-10T08:39:55.581345+0000","last_fullsized":"2024-02-10T08:39:55.581345+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:38:31.840218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.840218+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000008+0000","last_change":"2024-02-10T08:39:48.521060+0000","last_active":"2024-02-10T08:39:55.000008+0000","last_peered":"2024-02-10T08:39:55.000008+0000","last_clean":"2024-02-10T08:39:55.000008+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:39:55.000008+0000","last_undegraded":"2024-02-10T08:39:55.000008+0000","last_fullsized":"2024-02-10T08:39:55.000008+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:48.520941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:48.520941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522046+0000","last_change":"2024-02-10T08:37:09.215324+0000","last_active":"2024-02-10T08:39:55.522046+0000","last_peered":"2024-02-10T08:39:55.522046+0000","last_clean":"2024-02-10T08:39:55.522046+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:39:55.522046+0000","last_undegraded":"2024-02-10T08:39:55.522046+0000","last_fullsized":"2024-02-10T08:39:55.522046+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:09.215257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:09.215257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000349+0000","last_change":"2024-02-10T08:39:12.085361+0000","last_active":"2024-02-10T08:39:55.000349+0000","last_peered":"2024-02-10T08:39:55.000349+0000","last_clean":"2024-02-10T08:39:55.000349+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:39:55.000349+0000","last_undegraded":"2024-02-10T08:39:55.000349+0000","last_fullsized":"2024-02-10T08:39:55.000349+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.085261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.085261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":261,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581441+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:39:55.581441+0000","last_peered":"2024-02-10T08:39:55.581441+0000","last_clean":"2024-02-10T08:39:55.581441+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:39:55.581441+0000","last_undegraded":"2024-02-10T08:39:55.581441+0000","last_fullsized":"2024-02-10T08:39:55.581441+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":329,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776297+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:39:55.776297+0000","last_peered":"2024-02-10T08:39:55.776297+0000","last_clean":"2024-02-10T08:39:55.776297+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:39:55.776297+0000","last_undegraded":"2024-02-10T08:39:55.776297+0000","last_fullsized":"2024-02-10T08:39:55.776297+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":277,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999429+0000","last_change":"2024-02-10T08:38:57.920239+0000","last_active":"2024-02-10T08:39:54.999429+0000","last_peered":"2024-02-10T08:39:54.999429+0000","last_clean":"2024-02-10T08:39:54.999429+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:39:54.999429+0000","last_undegraded":"2024-02-10T08:39:54.999429+0000","last_fullsized":"2024-02-10T08:39:54.999429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:57.920144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.920144+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581600+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:39:55.581600+0000","last_peered":"2024-02-10T08:39:55.581600+0000","last_clean":"2024-02-10T08:39:55.581600+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:39:55.581600+0000","last_undegraded":"2024-02-10T08:39:55.581600+0000","last_fullsized":"2024-02-10T08:39:55.581600+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777438+0000","last_change":"2024-02-10T08:39:08.629231+0000","last_active":"2024-02-10T08:39:55.777438+0000","last_peered":"2024-02-10T08:39:55.777438+0000","last_clean":"2024-02-10T08:39:55.777438+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:39:55.777438+0000","last_undegraded":"2024-02-10T08:39:55.777438+0000","last_fullsized":"2024-02-10T08:39:55.777438+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998057+0000","last_change":"2024-02-10T08:37:43.520384+0000","last_active":"2024-02-10T08:39:54.998057+0000","last_peered":"2024-02-10T08:39:54.998057+0000","last_clean":"2024-02-10T08:39:54.998057+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:39:54.998057+0000","last_undegraded":"2024-02-10T08:39:54.998057+0000","last_fullsized":"2024-02-10T08:39:54.998057+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.520307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.520307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":247,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.008339+0000","last_change":"2024-02-10T08:38:18.588026+0000","last_active":"2024-02-10T08:39:54.008339+0000","last_peered":"2024-02-10T08:39:54.008339+0000","last_clean":"2024-02-10T08:39:54.008339+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:39:54.008339+0000","last_undegraded":"2024-02-10T08:39:54.008339+0000","last_fullsized":"2024-02-10T08:39:54.008339+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:38:18.587949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.587949+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522565+0000","last_change":"2024-02-10T08:38:13.655575+0000","last_active":"2024-02-10T08:39:55.522565+0000","last_peered":"2024-02-10T08:39:55.522565+0000","last_clean":"2024-02-10T08:39:55.522565+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:39:55.522565+0000","last_undegraded":"2024-02-10T08:39:55.522565+0000","last_fullsized":"2024-02-10T08:39:55.522565+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.655504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999982+0000","last_change":"2024-02-10T08:38:26.946622+0000","last_active":"2024-02-10T08:39:54.999982+0000","last_peered":"2024-02-10T08:39:54.999982+0000","last_clean":"2024-02-10T08:39:54.999982+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:39:54.999982+0000","last_undegraded":"2024-02-10T08:39:54.999982+0000","last_fullsized":"2024-02-10T08:39:54.999982+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.946531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.946531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":222,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000804+0000","last_change":"2024-02-10T08:38:25.903816+0000","last_active":"2024-02-10T08:39:55.000804+0000","last_peered":"2024-02-10T08:39:55.000804+0000","last_clean":"2024-02-10T08:39:55.000804+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:39:55.000804+0000","last_undegraded":"2024-02-10T08:39:55.000804+0000","last_fullsized":"2024-02-10T08:39:55.000804+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.903753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.903753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":291,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.006945+0000","last_change":"2024-02-10T08:38:31.745438+0000","last_active":"2024-02-10T08:39:54.006945+0000","last_peered":"2024-02-10T08:39:54.006945+0000","last_clean":"2024-02-10T08:39:54.006945+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:39:54.006945+0000","last_undegraded":"2024-02-10T08:39:54.006945+0000","last_fullsized":"2024-02-10T08:39:54.006945+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.745362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.745362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":227,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999541+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:39:54.999541+0000","last_peered":"2024-02-10T08:39:54.999541+0000","last_clean":"2024-02-10T08:39:54.999541+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:39:54.999541+0000","last_undegraded":"2024-02-10T08:39:54.999541+0000","last_fullsized":"2024-02-10T08:39:54.999541+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521695+0000","last_change":"2024-02-10T08:37:21.277884+0000","last_active":"2024-02-10T08:39:55.521695+0000","last_peered":"2024-02-10T08:39:55.521695+0000","last_clean":"2024-02-10T08:39:55.521695+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:39:55.521695+0000","last_undegraded":"2024-02-10T08:39:55.521695+0000","last_fullsized":"2024-02-10T08:39:55.521695+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.277786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.277786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581239+0000","last_change":"2024-02-10T08:38:18.758911+0000","last_active":"2024-02-10T08:39:55.581239+0000","last_peered":"2024-02-10T08:39:55.581239+0000","last_clean":"2024-02-10T08:39:55.581239+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:39:55.581239+0000","last_undegraded":"2024-02-10T08:39:55.581239+0000","last_fullsized":"2024-02-10T08:39:55.581239+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.758843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.758843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":265,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999115+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:39:54.999115+0000","last_peered":"2024-02-10T08:39:54.999115+0000","last_clean":"2024-02-10T08:39:54.999115+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:39:54.999115+0000","last_undegraded":"2024-02-10T08:39:54.999115+0000","last_fullsized":"2024-02-10T08:39:54.999115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":221,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998208+0000","last_change":"2024-02-10T08:39:06.770922+0000","last_active":"2024-02-10T08:39:54.998208+0000","last_peered":"2024-02-10T08:39:54.998208+0000","last_clean":"2024-02-10T08:39:54.998208+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:39:54.998208+0000","last_undegraded":"2024-02-10T08:39:54.998208+0000","last_fullsized":"2024-02-10T08:39:54.998208+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.770850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.770850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":266,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522238+0000","last_change":"2024-02-10T08:39:38.645588+0000","last_active":"2024-02-10T08:39:55.522238+0000","last_peered":"2024-02-10T08:39:55.522238+0000","last_clean":"2024-02-10T08:39:55.522238+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:39:55.522238+0000","last_undegraded":"2024-02-10T08:39:55.522238+0000","last_fullsized":"2024-02-10T08:39:55.522238+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.645527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.645527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":289,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999389+0000","last_change":"2024-02-10T08:37:40.592997+0000","last_active":"2024-02-10T08:39:54.999389+0000","last_peered":"2024-02-10T08:39:54.999389+0000","last_clean":"2024-02-10T08:39:54.999389+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:39:54.999389+0000","last_undegraded":"2024-02-10T08:39:54.999389+0000","last_fullsized":"2024-02-10T08:39:54.999389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:37:40.592930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.592930+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":287,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.521759+0000","last_change":"2024-02-10T08:39:36.639528+0000","last_active":"2024-02-10T08:39:55.521759+0000","last_peered":"2024-02-10T08:39:55.521759+0000","last_clean":"2024-02-10T08:39:55.521759+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:39:55.521759+0000","last_undegraded":"2024-02-10T08:39:55.521759+0000","last_fullsized":"2024-02-10T08:39:55.521759+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.639461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.639461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":268,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.008270+0000","last_change":"2024-02-10T08:38:25.633733+0000","last_active":"2024-02-10T08:39:54.008270+0000","last_peered":"2024-02-10T08:39:54.008270+0000","last_clean":"2024-02-10T08:39:54.008270+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:39:54.008270+0000","last_undegraded":"2024-02-10T08:39:54.008270+0000","last_fullsized":"2024-02-10T08:39:54.008270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.633608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.633608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776716+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:39:55.776716+0000","last_peered":"2024-02-10T08:39:55.776716+0000","last_clean":"2024-02-10T08:39:55.776716+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:39:55.776716+0000","last_undegraded":"2024-02-10T08:39:55.776716+0000","last_fullsized":"2024-02-10T08:39:55.776716+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":262,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.007032+0000","last_change":"2024-02-10T08:39:00.751654+0000","last_active":"2024-02-10T08:39:54.007032+0000","last_peered":"2024-02-10T08:39:54.007032+0000","last_clean":"2024-02-10T08:39:54.007032+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:39:54.007032+0000","last_undegraded":"2024-02-10T08:39:54.007032+0000","last_fullsized":"2024-02-10T08:39:54.007032+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:00.751584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.751584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":222,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002397+0000","last_change":"2024-02-10T08:39:11.660136+0000","last_active":"2024-02-10T08:39:54.002397+0000","last_peered":"2024-02-10T08:39:54.002397+0000","last_clean":"2024-02-10T08:39:54.002397+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:39:54.002397+0000","last_undegraded":"2024-02-10T08:39:54.002397+0000","last_fullsized":"2024-02-10T08:39:54.002397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:11.660049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.660049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999365+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:39:54.999365+0000","last_peered":"2024-02-10T08:39:54.999365+0000","last_clean":"2024-02-10T08:39:54.999365+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:39:54.999365+0000","last_undegraded":"2024-02-10T08:39:54.999365+0000","last_fullsized":"2024-02-10T08:39:54.999365+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":316,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998004+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:39:54.998004+0000","last_peered":"2024-02-10T08:39:54.998004+0000","last_clean":"2024-02-10T08:39:54.998004+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:39:54.998004+0000","last_undegraded":"2024-02-10T08:39:54.998004+0000","last_fullsized":"2024-02-10T08:39:54.998004+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":281,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000429+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:39:55.000429+0000","last_peered":"2024-02-10T08:39:55.000429+0000","last_clean":"2024-02-10T08:39:55.000429+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:39:55.000429+0000","last_undegraded":"2024-02-10T08:39:55.000429+0000","last_fullsized":"2024-02-10T08:39:55.000429+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":665,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.793534+0000","last_change":"2024-02-10T08:39:05.948676+0000","last_active":"2024-02-10T08:39:55.793534+0000","last_peered":"2024-02-10T08:39:55.793534+0000","last_clean":"2024-02-10T08:39:55.793534+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:39:55.793534+0000","last_undegraded":"2024-02-10T08:39:55.793534+0000","last_fullsized":"2024-02-10T08:39:55.793534+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:39:05.948579+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.948579+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":391,"num_read_kb":386,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522537+0000","last_change":"2024-02-10T08:39:03.251683+0000","last_active":"2024-02-10T08:39:55.522537+0000","last_peered":"2024-02-10T08:39:55.522537+0000","last_clean":"2024-02-10T08:39:55.522537+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:39:55.522537+0000","last_undegraded":"2024-02-10T08:39:55.522537+0000","last_fullsized":"2024-02-10T08:39:55.522537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.251585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.251585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":217,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000458+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:39:55.000458+0000","last_peered":"2024-02-10T08:39:55.000458+0000","last_clean":"2024-02-10T08:39:55.000458+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:39:55.000458+0000","last_undegraded":"2024-02-10T08:39:55.000458+0000","last_fullsized":"2024-02-10T08:39:55.000458+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000045+0000","last_change":"2024-02-10T08:39:06.965218+0000","last_active":"2024-02-10T08:39:55.000045+0000","last_peered":"2024-02-10T08:39:55.000045+0000","last_clean":"2024-02-10T08:39:55.000045+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:39:55.000045+0000","last_undegraded":"2024-02-10T08:39:55.000045+0000","last_fullsized":"2024-02-10T08:39:55.000045+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.965152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.965152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":332,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.005136+0000","last_change":"2024-02-10T08:38:19.613486+0000","last_active":"2024-02-10T08:39:54.005136+0000","last_peered":"2024-02-10T08:39:54.005136+0000","last_clean":"2024-02-10T08:39:54.005136+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:39:54.005136+0000","last_undegraded":"2024-02-10T08:39:54.005136+0000","last_fullsized":"2024-02-10T08:39:54.005136+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:38:19.613422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.613422+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":268,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.004957+0000","last_change":"2024-02-10T08:37:41.598616+0000","last_active":"2024-02-10T08:39:54.004957+0000","last_peered":"2024-02-10T08:39:54.004957+0000","last_clean":"2024-02-10T08:39:54.004957+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:39:54.004957+0000","last_undegraded":"2024-02-10T08:39:54.004957+0000","last_fullsized":"2024-02-10T08:39:54.004957+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.598541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.598541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":258,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777702+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:39:55.777702+0000","last_peered":"2024-02-10T08:39:55.777702+0000","last_clean":"2024-02-10T08:39:55.777702+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:39:55.777702+0000","last_undegraded":"2024-02-10T08:39:55.777702+0000","last_fullsized":"2024-02-10T08:39:55.777702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":222,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997482+0000","last_change":"2024-02-10T08:38:57.168613+0000","last_active":"2024-02-10T08:39:54.997482+0000","last_peered":"2024-02-10T08:39:54.997482+0000","last_clean":"2024-02-10T08:39:54.997482+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:39:54.997482+0000","last_undegraded":"2024-02-10T08:39:54.997482+0000","last_fullsized":"2024-02-10T08:39:54.997482+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:57.168523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.168523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":243,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999421+0000","last_change":"2024-02-10T08:39:52.548918+0000","last_active":"2024-02-10T08:39:54.999421+0000","last_peered":"2024-02-10T08:39:54.999421+0000","last_clean":"2024-02-10T08:39:54.999421+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:39:54.999421+0000","last_undegraded":"2024-02-10T08:39:54.999421+0000","last_fullsized":"2024-02-10T08:39:54.999421+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:52.548839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.548839+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999689+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:39:54.999689+0000","last_peered":"2024-02-10T08:39:54.999689+0000","last_clean":"2024-02-10T08:39:54.999689+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:39:54.999689+0000","last_undegraded":"2024-02-10T08:39:54.999689+0000","last_fullsized":"2024-02-10T08:39:54.999689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998836+0000","last_change":"2024-02-10T08:38:59.955336+0000","last_active":"2024-02-10T08:39:54.998836+0000","last_peered":"2024-02-10T08:39:54.998836+0000","last_clean":"2024-02-10T08:39:54.998836+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:39:54.998836+0000","last_undegraded":"2024-02-10T08:39:54.998836+0000","last_fullsized":"2024-02-10T08:39:54.998836+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.955272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.955272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776252+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:39:55.776252+0000","last_peered":"2024-02-10T08:39:55.776252+0000","last_clean":"2024-02-10T08:39:55.776252+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:39:55.776252+0000","last_undegraded":"2024-02-10T08:39:55.776252+0000","last_fullsized":"2024-02-10T08:39:55.776252+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777515+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:39:55.777515+0000","last_peered":"2024-02-10T08:39:55.777515+0000","last_clean":"2024-02-10T08:39:55.777515+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:39:55.777515+0000","last_undegraded":"2024-02-10T08:39:55.777515+0000","last_fullsized":"2024-02-10T08:39:55.777515+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":293,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997769+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:39:54.997769+0000","last_peered":"2024-02-10T08:39:54.997769+0000","last_clean":"2024-02-10T08:39:54.997769+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:39:54.997769+0000","last_undegraded":"2024-02-10T08:39:54.997769+0000","last_fullsized":"2024-02-10T08:39:54.997769+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581353+0000","last_change":"2024-02-10T08:38:32.886182+0000","last_active":"2024-02-10T08:39:55.581353+0000","last_peered":"2024-02-10T08:39:55.581353+0000","last_clean":"2024-02-10T08:39:55.581353+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:39:55.581353+0000","last_undegraded":"2024-02-10T08:39:55.581353+0000","last_fullsized":"2024-02-10T08:39:55.581353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:38:32.886090+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.886090+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":293,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582158+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:39:55.582158+0000","last_peered":"2024-02-10T08:39:55.582158+0000","last_clean":"2024-02-10T08:39:55.582158+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:39:55.582158+0000","last_undegraded":"2024-02-10T08:39:55.582158+0000","last_fullsized":"2024-02-10T08:39:55.582158+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":263,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777328+0000","last_change":"2024-02-10T08:38:22.407565+0000","last_active":"2024-02-10T08:39:55.777328+0000","last_peered":"2024-02-10T08:39:55.777328+0000","last_clean":"2024-02-10T08:39:55.777328+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:39:55.777328+0000","last_undegraded":"2024-02-10T08:39:55.777328+0000","last_fullsized":"2024-02-10T08:39:55.777328+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:22.407476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:22.407476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999580+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:39:54.999580+0000","last_peered":"2024-02-10T08:39:54.999580+0000","last_clean":"2024-02-10T08:39:54.999580+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:39:54.999580+0000","last_undegraded":"2024-02-10T08:39:54.999580+0000","last_fullsized":"2024-02-10T08:39:54.999580+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":249,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777755+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:39:55.777755+0000","last_peered":"2024-02-10T08:39:55.777755+0000","last_clean":"2024-02-10T08:39:55.777755+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:39:55.777755+0000","last_undegraded":"2024-02-10T08:39:55.777755+0000","last_fullsized":"2024-02-10T08:39:55.777755+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997700+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:39:54.997700+0000","last_peered":"2024-02-10T08:39:54.997700+0000","last_clean":"2024-02-10T08:39:54.997700+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:39:54.997700+0000","last_undegraded":"2024-02-10T08:39:54.997700+0000","last_fullsized":"2024-02-10T08:39:54.997700+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.001124+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:39:55.001124+0000","last_peered":"2024-02-10T08:39:55.001124+0000","last_clean":"2024-02-10T08:39:55.001124+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:39:55.001124+0000","last_undegraded":"2024-02-10T08:39:55.001124+0000","last_fullsized":"2024-02-10T08:39:55.001124+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581332+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:39:55.581332+0000","last_peered":"2024-02-10T08:39:55.581332+0000","last_clean":"2024-02-10T08:39:55.581332+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:39:55.581332+0000","last_undegraded":"2024-02-10T08:39:55.581332+0000","last_fullsized":"2024-02-10T08:39:55.581332+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":271,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777812+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:39:55.777812+0000","last_peered":"2024-02-10T08:39:55.777812+0000","last_clean":"2024-02-10T08:39:55.777812+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:39:55.777812+0000","last_undegraded":"2024-02-10T08:39:55.777812+0000","last_fullsized":"2024-02-10T08:39:55.777812+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":269,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997781+0000","last_change":"2024-02-10T08:38:30.291597+0000","last_active":"2024-02-10T08:39:54.997781+0000","last_peered":"2024-02-10T08:39:54.997781+0000","last_clean":"2024-02-10T08:39:54.997781+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:39:54.997781+0000","last_undegraded":"2024-02-10T08:39:54.997781+0000","last_fullsized":"2024-02-10T08:39:54.997781+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.291538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.291538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522350+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:39:55.522350+0000","last_peered":"2024-02-10T08:39:55.522350+0000","last_clean":"2024-02-10T08:39:55.522350+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:39:55.522350+0000","last_undegraded":"2024-02-10T08:39:55.522350+0000","last_fullsized":"2024-02-10T08:39:55.522350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":211,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.008183+0000","last_change":"2024-02-10T08:38:32.745050+0000","last_active":"2024-02-10T08:39:54.008183+0000","last_peered":"2024-02-10T08:39:54.008183+0000","last_clean":"2024-02-10T08:39:54.008183+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:39:54.008183+0000","last_undegraded":"2024-02-10T08:39:54.008183+0000","last_fullsized":"2024-02-10T08:39:54.008183+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.744972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.744972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":282,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777035+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:39:55.777035+0000","last_peered":"2024-02-10T08:39:55.777035+0000","last_clean":"2024-02-10T08:39:55.777035+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:39:55.777035+0000","last_undegraded":"2024-02-10T08:39:55.777035+0000","last_fullsized":"2024-02-10T08:39:55.777035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":268,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002279+0000","last_change":"2024-02-10T08:37:46.576724+0000","last_active":"2024-02-10T08:39:54.002279+0000","last_peered":"2024-02-10T08:39:54.002279+0000","last_clean":"2024-02-10T08:39:54.002279+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:39:54.002279+0000","last_undegraded":"2024-02-10T08:39:54.002279+0000","last_fullsized":"2024-02-10T08:39:54.002279+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.576661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.576661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":364,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002433+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:39:54.002433+0000","last_peered":"2024-02-10T08:39:54.002433+0000","last_clean":"2024-02-10T08:39:54.002433+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:39:54.002433+0000","last_undegraded":"2024-02-10T08:39:54.002433+0000","last_fullsized":"2024-02-10T08:39:54.002433+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522647+0000","last_change":"2024-02-10T08:39:43.632113+0000","last_active":"2024-02-10T08:39:55.522647+0000","last_peered":"2024-02-10T08:39:55.522647+0000","last_clean":"2024-02-10T08:39:55.522647+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:39:55.522647+0000","last_undegraded":"2024-02-10T08:39:55.522647+0000","last_fullsized":"2024-02-10T08:39:55.522647+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:43.632048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.632048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":182,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997718+0000","last_change":"2024-02-10T08:39:01.740757+0000","last_active":"2024-02-10T08:39:54.997718+0000","last_peered":"2024-02-10T08:39:54.997718+0000","last_clean":"2024-02-10T08:39:54.997718+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:39:54.997718+0000","last_undegraded":"2024-02-10T08:39:54.997718+0000","last_fullsized":"2024-02-10T08:39:54.997718+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:01.740686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:01.740686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":282,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002294+0000","last_change":"2024-02-10T08:39:46.641751+0000","last_active":"2024-02-10T08:39:54.002294+0000","last_peered":"2024-02-10T08:39:54.002294+0000","last_clean":"2024-02-10T08:39:54.002294+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:39:54.002294+0000","last_undegraded":"2024-02-10T08:39:54.002294+0000","last_fullsized":"2024-02-10T08:39:54.002294+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:39:46.641683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.641683+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":178,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581674+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:39:55.581674+0000","last_peered":"2024-02-10T08:39:55.581674+0000","last_clean":"2024-02-10T08:39:55.581674+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:39:55.581674+0000","last_undegraded":"2024-02-10T08:39:55.581674+0000","last_fullsized":"2024-02-10T08:39:55.581674+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":352,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777215+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:39:55.777215+0000","last_peered":"2024-02-10T08:39:55.777215+0000","last_clean":"2024-02-10T08:39:55.777215+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:39:55.777215+0000","last_undegraded":"2024-02-10T08:39:55.777215+0000","last_fullsized":"2024-02-10T08:39:55.777215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582219+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:39:55.582219+0000","last_peered":"2024-02-10T08:39:55.582219+0000","last_clean":"2024-02-10T08:39:55.582219+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:39:55.582219+0000","last_undegraded":"2024-02-10T08:39:55.582219+0000","last_fullsized":"2024-02-10T08:39:55.582219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":226,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000243+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:39:55.000243+0000","last_peered":"2024-02-10T08:39:55.000243+0000","last_clean":"2024-02-10T08:39:55.000243+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:39:55.000243+0000","last_undegraded":"2024-02-10T08:39:55.000243+0000","last_fullsized":"2024-02-10T08:39:55.000243+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":290,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000221+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:39:55.000221+0000","last_peered":"2024-02-10T08:39:55.000221+0000","last_clean":"2024-02-10T08:39:55.000221+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:39:55.000221+0000","last_undegraded":"2024-02-10T08:39:55.000221+0000","last_fullsized":"2024-02-10T08:39:55.000221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999313+0000","last_change":"2024-02-10T08:39:46.475770+0000","last_active":"2024-02-10T08:39:54.999313+0000","last_peered":"2024-02-10T08:39:54.999313+0000","last_clean":"2024-02-10T08:39:54.999313+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:39:54.999313+0000","last_undegraded":"2024-02-10T08:39:54.999313+0000","last_fullsized":"2024-02-10T08:39:54.999313+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.475678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.475678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777153+0000","last_change":"2024-02-10T08:38:13.463715+0000","last_active":"2024-02-10T08:39:55.777153+0000","last_peered":"2024-02-10T08:39:55.777153+0000","last_clean":"2024-02-10T08:39:55.777153+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:39:55.777153+0000","last_undegraded":"2024-02-10T08:39:55.777153+0000","last_fullsized":"2024-02-10T08:39:55.777153+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.463644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.463644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":181,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.003207+0000","last_change":"2024-02-10T08:38:20.592140+0000","last_active":"2024-02-10T08:39:54.003207+0000","last_peered":"2024-02-10T08:39:54.003207+0000","last_clean":"2024-02-10T08:39:54.003207+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:39:54.003207+0000","last_undegraded":"2024-02-10T08:39:54.003207+0000","last_fullsized":"2024-02-10T08:39:54.003207+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:20.592065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.592065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":221,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000055+0000","last_change":"2024-02-10T08:39:12.687989+0000","last_active":"2024-02-10T08:39:55.000055+0000","last_peered":"2024-02-10T08:39:55.000055+0000","last_clean":"2024-02-10T08:39:55.000055+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:39:55.000055+0000","last_undegraded":"2024-02-10T08:39:55.000055+0000","last_fullsized":"2024-02-10T08:39:55.000055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.687918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.687918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":282,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581550+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:39:55.581550+0000","last_peered":"2024-02-10T08:39:55.581550+0000","last_clean":"2024-02-10T08:39:55.581550+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:39:55.581550+0000","last_undegraded":"2024-02-10T08:39:55.581550+0000","last_fullsized":"2024-02-10T08:39:55.581550+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":281,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000165+0000","last_change":"2024-02-10T08:39:04.936255+0000","last_active":"2024-02-10T08:39:55.000165+0000","last_peered":"2024-02-10T08:39:55.000165+0000","last_clean":"2024-02-10T08:39:55.000165+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:39:55.000165+0000","last_undegraded":"2024-02-10T08:39:55.000165+0000","last_fullsized":"2024-02-10T08:39:55.000165+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:04.936174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.936174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":261,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999594+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:39:54.999594+0000","last_peered":"2024-02-10T08:39:54.999594+0000","last_clean":"2024-02-10T08:39:54.999594+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:39:54.999594+0000","last_undegraded":"2024-02-10T08:39:54.999594+0000","last_fullsized":"2024-02-10T08:39:54.999594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":272,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776307+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:39:55.776307+0000","last_peered":"2024-02-10T08:39:55.776307+0000","last_clean":"2024-02-10T08:39:55.776307+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:39:55.776307+0000","last_undegraded":"2024-02-10T08:39:55.776307+0000","last_fullsized":"2024-02-10T08:39:55.776307+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":222,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777739+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:39:55.777739+0000","last_peered":"2024-02-10T08:39:55.777739+0000","last_clean":"2024-02-10T08:39:55.777739+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:39:55.777739+0000","last_undegraded":"2024-02-10T08:39:55.777739+0000","last_fullsized":"2024-02-10T08:39:55.777739+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":268,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776208+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:39:55.776208+0000","last_peered":"2024-02-10T08:39:55.776208+0000","last_clean":"2024-02-10T08:39:55.776208+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:39:55.776208+0000","last_undegraded":"2024-02-10T08:39:55.776208+0000","last_fullsized":"2024-02-10T08:39:55.776208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":268,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.003001+0000","last_change":"2024-02-10T08:38:13.601496+0000","last_active":"2024-02-10T08:39:54.003001+0000","last_peered":"2024-02-10T08:39:54.003001+0000","last_clean":"2024-02-10T08:39:54.003001+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:39:54.003001+0000","last_undegraded":"2024-02-10T08:39:54.003001+0000","last_fullsized":"2024-02-10T08:39:54.003001+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.601390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.601390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998055+0000","last_change":"2024-02-10T08:39:41.555411+0000","last_active":"2024-02-10T08:39:54.998055+0000","last_peered":"2024-02-10T08:39:54.998055+0000","last_clean":"2024-02-10T08:39:54.998055+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:39:54.998055+0000","last_undegraded":"2024-02-10T08:39:54.998055+0000","last_fullsized":"2024-02-10T08:39:54.998055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.555339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.555339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522233+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:39:55.522233+0000","last_peered":"2024-02-10T08:39:55.522233+0000","last_clean":"2024-02-10T08:39:55.522233+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:39:55.522233+0000","last_undegraded":"2024-02-10T08:39:55.522233+0000","last_fullsized":"2024-02-10T08:39:55.522233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776857+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:39:55.776857+0000","last_peered":"2024-02-10T08:39:55.776857+0000","last_clean":"2024-02-10T08:39:55.776857+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:39:55.776857+0000","last_undegraded":"2024-02-10T08:39:55.776857+0000","last_fullsized":"2024-02-10T08:39:55.776857+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":281,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776597+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:39:55.776597+0000","last_peered":"2024-02-10T08:39:55.776597+0000","last_clean":"2024-02-10T08:39:55.776597+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:39:55.776597+0000","last_undegraded":"2024-02-10T08:39:55.776597+0000","last_fullsized":"2024-02-10T08:39:55.776597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":274,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999984+0000","last_change":"2024-02-10T08:39:49.529787+0000","last_active":"2024-02-10T08:39:54.999984+0000","last_peered":"2024-02-10T08:39:54.999984+0000","last_clean":"2024-02-10T08:39:54.999984+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:39:54.999984+0000","last_undegraded":"2024-02-10T08:39:54.999984+0000","last_fullsized":"2024-02-10T08:39:54.999984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:49.529724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.529724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":261,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522083+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:39:55.522083+0000","last_peered":"2024-02-10T08:39:55.522083+0000","last_clean":"2024-02-10T08:39:55.522083+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:39:55.522083+0000","last_undegraded":"2024-02-10T08:39:55.522083+0000","last_fullsized":"2024-02-10T08:39:55.522083+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":272,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777251+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:39:55.777251+0000","last_peered":"2024-02-10T08:39:55.777251+0000","last_clean":"2024-02-10T08:39:55.777251+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:39:55.777251+0000","last_undegraded":"2024-02-10T08:39:55.777251+0000","last_fullsized":"2024-02-10T08:39:55.777251+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":182,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999691+0000","last_change":"2024-02-10T08:38:59.747834+0000","last_active":"2024-02-10T08:39:54.999691+0000","last_peered":"2024-02-10T08:39:54.999691+0000","last_clean":"2024-02-10T08:39:54.999691+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:39:54.999691+0000","last_undegraded":"2024-02-10T08:39:54.999691+0000","last_fullsized":"2024-02-10T08:39:54.999691+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.747762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.747762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":222,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522853+0000","last_change":"2024-02-10T08:39:41.691153+0000","last_active":"2024-02-10T08:39:55.522853+0000","last_peered":"2024-02-10T08:39:55.522853+0000","last_clean":"2024-02-10T08:39:55.522853+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:39:55.522853+0000","last_undegraded":"2024-02-10T08:39:55.522853+0000","last_fullsized":"2024-02-10T08:39:55.522853+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:39:41.691086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.691086+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":277,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.001783+0000","last_change":"2024-02-10T08:37:39.573283+0000","last_active":"2024-02-10T08:39:54.001783+0000","last_peered":"2024-02-10T08:39:54.001783+0000","last_clean":"2024-02-10T08:39:54.001783+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:39:54.001783+0000","last_undegraded":"2024-02-10T08:39:54.001783+0000","last_fullsized":"2024-02-10T08:39:54.001783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:39.573225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.573225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":265,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522333+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:39:55.522333+0000","last_peered":"2024-02-10T08:39:55.522333+0000","last_clean":"2024-02-10T08:39:55.522333+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:39:55.522333+0000","last_undegraded":"2024-02-10T08:39:55.522333+0000","last_fullsized":"2024-02-10T08:39:55.522333+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998772+0000","last_change":"2024-02-10T08:38:14.837278+0000","last_active":"2024-02-10T08:39:54.998772+0000","last_peered":"2024-02-10T08:39:54.998772+0000","last_clean":"2024-02-10T08:39:54.998772+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:39:54.998772+0000","last_undegraded":"2024-02-10T08:39:54.998772+0000","last_fullsized":"2024-02-10T08:39:54.998772+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:14.837154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:14.837154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":291,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000070+0000","last_change":"2024-02-10T08:39:47.473081+0000","last_active":"2024-02-10T08:39:55.000070+0000","last_peered":"2024-02-10T08:39:55.000070+0000","last_clean":"2024-02-10T08:39:55.000070+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:39:55.000070+0000","last_undegraded":"2024-02-10T08:39:55.000070+0000","last_fullsized":"2024-02-10T08:39:55.000070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:39:47.473007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.473007+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":278,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776463+0000","last_change":"2024-02-10T08:38:31.436092+0000","last_active":"2024-02-10T08:39:55.776463+0000","last_peered":"2024-02-10T08:39:55.776463+0000","last_clean":"2024-02-10T08:39:55.776463+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:39:55.776463+0000","last_undegraded":"2024-02-10T08:39:55.776463+0000","last_fullsized":"2024-02-10T08:39:55.776463+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.435994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.435994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999980+0000","last_change":"2024-02-10T08:38:58.917062+0000","last_active":"2024-02-10T08:39:54.999980+0000","last_peered":"2024-02-10T08:39:54.999980+0000","last_clean":"2024-02-10T08:39:54.999980+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:39:54.999980+0000","last_undegraded":"2024-02-10T08:39:54.999980+0000","last_fullsized":"2024-02-10T08:39:54.999980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:58.916996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:58.916996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000393+0000","last_change":"2024-02-10T08:38:28.906569+0000","last_active":"2024-02-10T08:39:55.000393+0000","last_peered":"2024-02-10T08:39:55.000393+0000","last_clean":"2024-02-10T08:39:55.000393+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:39:55.000393+0000","last_undegraded":"2024-02-10T08:39:55.000393+0000","last_fullsized":"2024-02-10T08:39:55.000393+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:38:28.906474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":300,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.001290+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:39:55.001290+0000","last_peered":"2024-02-10T08:39:55.001290+0000","last_clean":"2024-02-10T08:39:55.001290+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:39:55.001290+0000","last_undegraded":"2024-02-10T08:39:55.001290+0000","last_fullsized":"2024-02-10T08:39:55.001290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":269,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998107+0000","last_change":"2024-02-10T08:38:32.342305+0000","last_active":"2024-02-10T08:39:54.998107+0000","last_peered":"2024-02-10T08:39:54.998107+0000","last_clean":"2024-02-10T08:39:54.998107+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:39:54.998107+0000","last_undegraded":"2024-02-10T08:39:54.998107+0000","last_fullsized":"2024-02-10T08:39:54.998107+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.342200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.342200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":259,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.582028+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:39:55.582028+0000","last_peered":"2024-02-10T08:39:55.582028+0000","last_clean":"2024-02-10T08:39:55.582028+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:39:55.582028+0000","last_undegraded":"2024-02-10T08:39:55.582028+0000","last_fullsized":"2024-02-10T08:39:55.582028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":182,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997720+0000","last_change":"2024-02-10T08:38:18.541756+0000","last_active":"2024-02-10T08:39:54.997720+0000","last_peered":"2024-02-10T08:39:54.997720+0000","last_clean":"2024-02-10T08:39:54.997720+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:39:54.997720+0000","last_undegraded":"2024-02-10T08:39:54.997720+0000","last_fullsized":"2024-02-10T08:39:54.997720+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.541693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.541693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":290,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777377+0000","last_change":"2024-02-10T08:38:19.453131+0000","last_active":"2024-02-10T08:39:55.777377+0000","last_peered":"2024-02-10T08:39:55.777377+0000","last_clean":"2024-02-10T08:39:55.777377+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:39:55.777377+0000","last_undegraded":"2024-02-10T08:39:55.777377+0000","last_fullsized":"2024-02-10T08:39:55.777377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:38:19.453060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.453060+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":269,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777143+0000","last_change":"2024-02-10T08:38:16.426049+0000","last_active":"2024-02-10T08:39:55.777143+0000","last_peered":"2024-02-10T08:39:55.777143+0000","last_clean":"2024-02-10T08:39:55.777143+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:39:55.777143+0000","last_undegraded":"2024-02-10T08:39:55.777143+0000","last_fullsized":"2024-02-10T08:39:55.777143+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:16.425945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:16.425945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":263,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777861+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:39:55.777861+0000","last_peered":"2024-02-10T08:39:55.777861+0000","last_clean":"2024-02-10T08:39:55.777861+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:39:55.777861+0000","last_undegraded":"2024-02-10T08:39:55.777861+0000","last_fullsized":"2024-02-10T08:39:55.777861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":213,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.000195+0000","last_change":"2024-02-10T08:39:46.928020+0000","last_active":"2024-02-10T08:39:55.000195+0000","last_peered":"2024-02-10T08:39:55.000195+0000","last_clean":"2024-02-10T08:39:55.000195+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:39:55.000195+0000","last_undegraded":"2024-02-10T08:39:55.000195+0000","last_fullsized":"2024-02-10T08:39:55.000195+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.927953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":303,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581448+0000","last_change":"2024-02-10T08:38:26.871174+0000","last_active":"2024-02-10T08:39:55.581448+0000","last_peered":"2024-02-10T08:39:55.581448+0000","last_clean":"2024-02-10T08:39:55.581448+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:39:55.581448+0000","last_undegraded":"2024-02-10T08:39:55.581448+0000","last_fullsized":"2024-02-10T08:39:55.581448+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:38:26.871100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.871100+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":270,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522076+0000","last_change":"2024-02-10T08:39:02.250833+0000","last_active":"2024-02-10T08:39:55.522076+0000","last_peered":"2024-02-10T08:39:55.522076+0000","last_clean":"2024-02-10T08:39:55.522076+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:39:55.522076+0000","last_undegraded":"2024-02-10T08:39:55.522076+0000","last_fullsized":"2024-02-10T08:39:55.522076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:02.250719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:02.250719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.997994+0000","last_change":"2024-02-10T08:38:27.952601+0000","last_active":"2024-02-10T08:39:54.997994+0000","last_peered":"2024-02-10T08:39:54.997994+0000","last_clean":"2024-02-10T08:39:54.997994+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:39:54.997994+0000","last_undegraded":"2024-02-10T08:39:54.997994+0000","last_fullsized":"2024-02-10T08:39:54.997994+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:27.952518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:27.952518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":217,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.998978+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:39:54.998978+0000","last_peered":"2024-02-10T08:39:54.998978+0000","last_clean":"2024-02-10T08:39:54.998978+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:39:54.998978+0000","last_undegraded":"2024-02-10T08:39:54.998978+0000","last_fullsized":"2024-02-10T08:39:54.998978+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":280,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581700+0000","last_change":"2024-02-10T08:39:11.790943+0000","last_active":"2024-02-10T08:39:55.581700+0000","last_peered":"2024-02-10T08:39:55.581700+0000","last_clean":"2024-02-10T08:39:55.581700+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:39:55.581700+0000","last_undegraded":"2024-02-10T08:39:55.581700+0000","last_fullsized":"2024-02-10T08:39:55.581700+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.790866+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":279,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522427+0000","last_change":"2024-02-10T08:38:30.389134+0000","last_active":"2024-02-10T08:39:55.522427+0000","last_peered":"2024-02-10T08:39:55.522427+0000","last_clean":"2024-02-10T08:39:55.522427+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:39:55.522427+0000","last_undegraded":"2024-02-10T08:39:55.522427+0000","last_fullsized":"2024-02-10T08:39:55.522427+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.389076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":262,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.002541+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:39:54.002541+0000","last_peered":"2024-02-10T08:39:54.002541+0000","last_clean":"2024-02-10T08:39:54.002541+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:39:54.002541+0000","last_undegraded":"2024-02-10T08:39:54.002541+0000","last_fullsized":"2024-02-10T08:39:54.002541+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":181,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.006486+0000","last_change":"2024-02-10T08:38:15.539954+0000","last_active":"2024-02-10T08:39:54.006486+0000","last_peered":"2024-02-10T08:39:54.006486+0000","last_clean":"2024-02-10T08:39:54.006486+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:39:54.006486+0000","last_undegraded":"2024-02-10T08:39:54.006486+0000","last_fullsized":"2024-02-10T08:39:54.006486+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:15.539850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:15.539850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581161+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:39:55.581161+0000","last_peered":"2024-02-10T08:39:55.581161+0000","last_clean":"2024-02-10T08:39:55.581161+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:39:55.581161+0000","last_undegraded":"2024-02-10T08:39:55.581161+0000","last_fullsized":"2024-02-10T08:39:55.581161+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":278,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.777668+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:39:55.777668+0000","last_peered":"2024-02-10T08:39:55.777668+0000","last_clean":"2024-02-10T08:39:55.777668+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:39:55.777668+0000","last_undegraded":"2024-02-10T08:39:55.777668+0000","last_fullsized":"2024-02-10T08:39:55.777668+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.581436+0000","last_change":"2024-02-10T08:39:05.790083+0000","last_active":"2024-02-10T08:39:55.581436+0000","last_peered":"2024-02-10T08:39:55.581436+0000","last_clean":"2024-02-10T08:39:55.581436+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:39:55.581436+0000","last_undegraded":"2024-02-10T08:39:55.581436+0000","last_fullsized":"2024-02-10T08:39:55.581436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.790005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.790005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":220,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.005047+0000","last_change":"2024-02-10T08:38:30.695748+0000","last_active":"2024-02-10T08:39:54.005047+0000","last_peered":"2024-02-10T08:39:54.005047+0000","last_clean":"2024-02-10T08:39:54.005047+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:39:54.005047+0000","last_undegraded":"2024-02-10T08:39:54.005047+0000","last_fullsized":"2024-02-10T08:39:54.005047+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.695684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.695684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":312,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776624+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:39:55.776624+0000","last_peered":"2024-02-10T08:39:55.776624+0000","last_clean":"2024-02-10T08:39:55.776624+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:39:55.776624+0000","last_undegraded":"2024-02-10T08:39:55.776624+0000","last_fullsized":"2024-02-10T08:39:55.776624+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":183,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.522765+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:39:55.522765+0000","last_peered":"2024-02-10T08:39:55.522765+0000","last_clean":"2024-02-10T08:39:55.522765+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:39:55.522765+0000","last_undegraded":"2024-02-10T08:39:55.522765+0000","last_fullsized":"2024-02-10T08:39:55.522765+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":258,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:55.776992+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:39:55.776992+0000","last_peered":"2024-02-10T08:39:55.776992+0000","last_clean":"2024-02-10T08:39:55.776992+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:39:55.776992+0000","last_undegraded":"2024-02-10T08:39:55.776992+0000","last_fullsized":"2024-02-10T08:39:55.776992+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":212,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-02-10T08:39:54.999311+0000","last_change":"2024-02-10T08:38:34.376839+0000","last_active":"2024-02-10T08:39:54.999311+0000","last_peered":"2024-02-10T08:39:54.999311+0000","last_clean":"2024-02-10T08:39:54.999311+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:39:54.999311+0000","last_undegraded":"2024-02-10T08:39:54.999311+0000","last_fullsized":"2024-02-10T08:39:54.999311+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:34.376762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:34.376762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001220,"num_objects":123,"num_object_clones":2,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":548,"num_write_kb":4449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13496320,"data_stored":12003668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1805,"ondisk_log_size":1805,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":188,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":391,"num_read_kb":386,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299474,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306232,"kb_used_data":4912,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438920,"statfs":{"total":95995035648,"available":95681454080,"internally_reserved":0,"allocated":5029888,"data_stored":2756368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.76400000000000001,"15min":0.60299999999999998},"min":{"1min":0.30399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.70399999999999996,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67100000000000004,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59399999999999997,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.62,"15min":0.54800000000000004},"min":{"1min":0.314,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.623,"5min":7.593,"15min":7.593},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65100000000000002,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65600000000000003,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.63800000000000001,"15min":0.54600000000000004},"min":{"1min":0.36299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.37},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.85399999999999998,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":19.369,"15min":19.369},"last":0.308}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.68700000000000006,"15min":0.56200000000000006},"min":{"1min":0.378,"5min":0.307,"15min":0.307},"max":{"1min":0.63500000000000001,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84999999999999998,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72799999999999998,"5min":22.102,"15min":22.102},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.498,"15min":0.44900000000000001},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":8.827,"15min":8.827},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.56999999999999995,"15min":0.47399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67000000000000004,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.34300000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.67400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70799999999999996,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68899999999999995,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.70799999999999996,"15min":0.56200000000000006},"min":{"1min":0.313,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89100000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.376},{"interface":"front","average":{"1min":0.499,"5min":0.503,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69699999999999995,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.38800000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561112,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318260,"kb_used_data":8744,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426892,"statfs":{"total":95995035648,"available":95669137408,"internally_reserved":0,"allocated":8953856,"data_stored":6766461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.63100000000000001,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26400000000000001,"15min":0.19800000000000001},"max":{"1min":0.65800000000000003,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.65400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.66600000000000004,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.59499999999999997,"15min":0.53000000000000003},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.56999999999999995,"15min":0.496},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":0.59799999999999998,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3400000000000001,"5min":4.798,"15min":4.798},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58499999999999996,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.321,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.82999999999999996,"15min":0.59399999999999997},"min":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":1.329,"5min":16.823,"15min":16.823},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.57999999999999996,"15min":0.51500000000000001},"min":{"1min":0.32400000000000001,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.60399999999999998,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.443,"15min":0.39500000000000002},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.44500000000000001,"15min":0.379},"min":{"1min":0.32900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.38}]},{"osd":5,"last update":"Sat Feb 10 08:38:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.46899999999999997,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.7429999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.41999999999999998},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1339999999999999,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.434}]},{"osd":7,"last update":"Sat Feb 10 08:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.50800000000000001,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314436,"kb_used_data":4912,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430716,"statfs":{"total":95995035648,"available":95673053184,"internally_reserved":0,"allocated":5029888,"data_stored":2768147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.56599999999999995,"15min":0.40999999999999998},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.64400000000000002,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.63600000000000001,"15min":0.45800000000000002},"min":{"1min":0.23999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":16.166,"15min":16.166},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":1.0980000000000001,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.55300000000000005,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.86099999999999999,"15min":0.53300000000000003},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54700000000000004,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.74399999999999999,"15min":0.53200000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":24.773,"15min":24.773},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71199999999999997,"15min":0.53700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.60399999999999998,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.442}]},{"osd":4,"last update":"Sat Feb 10 08:39:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.752,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89000000000000001,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.67800000000000005,"15min":0.60999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.98999999999999999,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67200000000000004,"15min":0.61199999999999999},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.78600000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.38700000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.67600000000000005,"15min":0.60999999999999999},"min":{"1min":0.32500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.95599999999999996,"5min":9.532,"15min":9.532},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.78800000000000003,"15min":0.65600000000000003},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0009999999999999,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.72499999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.83999999999999997,"15min":0.66700000000000004},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.57299999999999995}]}]},{"osd":0,"up_from":0,"seq":34359738485,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314392,"kb_used_data":4876,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430760,"statfs":{"total":95995035648,"available":95673098240,"internally_reserved":0,"allocated":4993024,"data_stored":2766196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.5,"15min":0.40300000000000002},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.58599999999999997,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.318,"5min":0.65200000000000002,"15min":0.46000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56799999999999995,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.70099999999999996,"15min":0.47699999999999998},"min":{"1min":0.24399999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":12.512,"15min":12.512},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.60999999999999999,"15min":0.46200000000000002},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.223},"max":{"1min":0.50800000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.6520000000000001,"5min":4.617,"15min":4.617},"last":0.436},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.59399999999999997,"15min":0.48599999999999999},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:39:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.85599999999999998,"15min":0.67800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.875,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.77500000000000002,"15min":0.66100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.623}]},{"osd":5,"last update":"Sat Feb 10 08:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.84099999999999997,"15min":0.65400000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68799999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.71599999999999997,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:38:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80600000000000005,"15min":0.66200000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.3090000000000002,"5min":9.266,"15min":9.266},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.73099999999999998,"15min":0.64900000000000002},"min":{"1min":0.45100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.4630000000000001,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.80100000000000005,"15min":0.69299999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84099999999999997,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.77200000000000002,"15min":0.66800000000000004},"min":{"1min":0.46600000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.55800000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182503,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318324,"kb_used_data":8808,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426828,"statfs":{"total":95995035648,"available":95669071872,"internally_reserved":0,"allocated":9019392,"data_stored":6766591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.81699999999999995,"15min":0.51600000000000001},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.42299999999999999,"5min":14.744,"15min":14.744},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.872,"15min":0.54500000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.498,"5min":11.808,"15min":11.808},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.82599999999999996,"15min":0.55100000000000005},"min":{"1min":0.23200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.262,"5min":16.634,"15min":16.634},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.82099999999999995,"15min":0.56100000000000005},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":0.55100000000000005,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.316}]},{"osd":2,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.86399999999999999,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":1.079,"15min":0.63500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.51500000000000001,"5min":23.294,"15min":23.294},"last":0.40300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:39:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.84799999999999998,"15min":0.68999999999999995},"min":{"1min":0.32400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.589,"5min":12.371,"15min":12.371},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.84699999999999998,"15min":0.66700000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0409999999999999,"5min":14.711,"15min":14.711},"last":0.59199999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.97199999999999998,"15min":0.69999999999999996},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73399999999999999,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.76600000000000001,"15min":0.64100000000000001},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70199999999999996,"5min":11.448,"15min":11.448},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.98599999999999999,"15min":0.68500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.92300000000000004,"15min":0.67500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.879,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72199999999999998,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":1.0469999999999999,"15min":0.71199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68100000000000005,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.63300000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306200,"kb_used_data":4880,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438952,"statfs":{"total":95995035648,"available":95681486848,"internally_reserved":0,"allocated":4997120,"data_stored":2756367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.84199999999999997,"15min":0.64400000000000002},"min":{"1min":0.29499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71999999999999997,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.65600000000000003,"15min":0.56200000000000006},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.626,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.59399999999999997},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70599999999999996,"5min":5.774,"15min":5.774},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.74099999999999999,"15min":0.61099999999999999},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67100000000000004,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.76100000000000001,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69399999999999995,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.77400000000000002,"15min":0.61399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63700000000000001,"5min":10.01,"15min":10.01},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.91900000000000004,"15min":0.66500000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.62,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.0960000000000001,"15min":0.71999999999999997},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.82999999999999996,"5min":25.27,"15min":25.27},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.51000000000000001,"15min":0.41899999999999998},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67300000000000004,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47699999999999998,"15min":0.41299999999999998},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71499999999999997,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.438}]},{"osd":6,"last update":"Sat Feb 10 08:38:55 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.65700000000000003,"15min":0.47899999999999998},"min":{"1min":0.29399999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.54000000000000004,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.63700000000000001,"15min":0.501},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.085,"5min":13.753,"15min":13.753},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.61699999999999999,"15min":0.50700000000000001},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69499999999999995,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.67900000000000005,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0509999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.50800000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790045,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305952,"kb_used_data":8728,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439200,"statfs":{"total":95995035648,"available":95681740800,"internally_reserved":0,"allocated":8937472,"data_stored":6755694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.9710000000000001,"15min":0.999},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66400000000000003,"5min":107.551,"15min":107.551},"last":0.377},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.845,"15min":0.96399999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":0.61899999999999999,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.6910000000000001,"15min":0.93999999999999995},"min":{"1min":0.34899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":107.205,"15min":107.205},"last":0.436},{"interface":"front","average":{"1min":0.48699999999999999,"5min":1.363,"15min":0.82899999999999996},"min":{"1min":0.32100000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.65700000000000003,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":1.8480000000000001,"15min":0.95399999999999996},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67800000000000005,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.927,"15min":0.995},"min":{"1min":0.311,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":0.65800000000000003,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.39700000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.1479999999999999,"15min":0.71899999999999997},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2669999999999999,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":2.5219999999999998,"15min":1.2070000000000001},"min":{"1min":0.41699999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61299999999999999,"5min":114.526,"15min":114.526},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":1.1699999999999999,"15min":0.65300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.627,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.90200000000000002,"15min":0.55900000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.58699999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.317}]},{"osd":6,"last update":"Sat Feb 10 08:38:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.1919999999999999,"15min":0.67200000000000004},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63300000000000001,"5min":72.241,"15min":72.241},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.71099999999999997,"15min":0.54100000000000004},"min":{"1min":0.307,"5min":0.248,"15min":0.248},"max":{"1min":0.621,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.41999999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":1.1819999999999999,"15min":0.71099999999999997},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.60899999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.2450000000000001,"15min":0.73899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4000411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4190208,"data_stored":4000483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":188,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:40:00.725 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T08:40:00.725 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:40:00.725 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-10T08:40:00.725 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T08:40:00.725 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T08:40:00.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-10T08:40:00.917 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:00.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:01.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:01.049 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:01.247 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:01.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:01.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:40:01.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:40:01.381 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:01.403 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:01.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.010389+0000 mon.a (mon.0) 2054 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.010461+0000 mon.a (mon.0) 2055 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]': finished 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.010514+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e306: 8 total, 8 up, 7 in 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.030750+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.031251+0000 mon.a (mon.0) 2057 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.031978+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.032219+0000 mon.a (mon.0) 2058 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.032657+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.032926+0000 mon.a (mon.0) 2059 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.195759+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.107:0/1165078207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.975008+0000 mon.a (mon.0) 2060 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:01.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:00.975106+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e307: 8 total, 8 up, 7 in 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.010389+0000 mon.a (mon.0) 2054 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.010461+0000 mon.a (mon.0) 2055 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]': finished 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.010514+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e306: 8 total, 8 up, 7 in 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.030750+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.031251+0000 mon.a (mon.0) 2057 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.031978+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.032219+0000 mon.a (mon.0) 2058 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.032657+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.032926+0000 mon.a (mon.0) 2059 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.195759+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.107:0/1165078207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.975008+0000 mon.a (mon.0) 2060 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:00.975106+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e307: 8 total, 8 up, 7 in 2024-02-10T08:40:01.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:01.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.010389+0000 mon.a (mon.0) 2054 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi107-112411-53"}]': finished 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.010461+0000 mon.a (mon.0) 2055 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-39"}]': finished 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.010514+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e306: 8 total, 8 up, 7 in 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.030750+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.031251+0000 mon.a (mon.0) 2057 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.031978+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.032219+0000 mon.a (mon.0) 2058 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.032657+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.032926+0000 mon.a (mon.0) 2059 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.195759+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.107:0/1165078207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.975008+0000 mon.a (mon.0) 2060 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi107-112411-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:00.975106+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e307: 8 total, 8 up, 7 in 2024-02-10T08:40:01.546 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:01.565 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:01.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:01.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:01.698 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:01.795 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:01.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:01.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:01.926 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:02.231 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:02.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:02.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:02.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:00.980762+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:01.227198+0000 mon.a (mon.0) 2062 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:01.345979+0000 mgr.y (mgr.14553) 296 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:02.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:01.409793+0000 mgr.y (mgr.14553) 297 : cluster [DBG] pgmap v437: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:01.995549+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:02.227027+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e308: 8 total, 8 up, 7 in 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:00.980762+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:01.227198+0000 mon.a (mon.0) 2062 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:01.345979+0000 mgr.y (mgr.14553) 296 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:01.409793+0000 mgr.y (mgr.14553) 297 : cluster [DBG] pgmap v437: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:01.995549+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:02.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:02.227027+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e308: 8 total, 8 up, 7 in 2024-02-10T08:40:02.364 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:02.384 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:02.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:02.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:00.980762+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:01.227198+0000 mon.a (mon.0) 2062 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:01.345979+0000 mgr.y (mgr.14553) 296 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:01.409793+0000 mgr.y (mgr.14553) 297 : cluster [DBG] pgmap v437: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:01.995549+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:02.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:02.227027+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e308: 8 total, 8 up, 7 in 2024-02-10T08:40:02.495 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:40:02.516 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:02.717 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:02.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:02.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:02.872 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:02.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:02.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:03.035 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:03.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:03.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:03.169 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:03.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:03.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:03.448 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:03.466 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:03.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:02.227810+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:02.991724+0000 mon.a (mon.0) 2065 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:02.991888+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:02.991946+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e309: 8 total, 8 up, 7 in 2024-02-10T08:40:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.001876+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.004157+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.004869+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:03.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:02.227810+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:02.991724+0000 mon.a (mon.0) 2065 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:02.991888+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:02.991946+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e309: 8 total, 8 up, 7 in 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.001876+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.004157+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.004869+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:02.227810+0000 mon.a (mon.0) 2064 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:02.991724+0000 mon.a (mon.0) 2065 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi107-112411-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:02.991888+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:02.991946+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e309: 8 total, 8 up, 7 in 2024-02-10T08:40:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.001876+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.004157+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.004869+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:03.722 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:03.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:03.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:03.852 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:03.951 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:03.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:04.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:04.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:40:04.082 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:04.101 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:04.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:04.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:04.224 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:40:04.224 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-02-10T08:40:04.224 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:40:04.231 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:03.410178+0000 mgr.y (mgr.14553) 298 : cluster [DBG] pgmap v440: 226 pgs: 40 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.992643+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:03.992726+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e310: 8 total, 8 up, 7 in 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.994010+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:03.999963+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:03.410178+0000 mgr.y (mgr.14553) 298 : cluster [DBG] pgmap v440: 226 pgs: 40 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.992643+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:03.992726+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e310: 8 total, 8 up, 7 in 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.994010+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:03.999963+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:03.410178+0000 mgr.y (mgr.14553) 298 : cluster [DBG] pgmap v440: 226 pgs: 40 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:04.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.992643+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:04.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:03.992726+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e310: 8 total, 8 up, 7 in 2024-02-10T08:40:04.462 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.994010+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.462 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:03.999963+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T08:40:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:04.882 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:04.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:04.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:05.017 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:05.083 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:40:05.083 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:40:05.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:40:05.083 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:3 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e117)", 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 193.52918040700001, 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003, 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.084 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907503+0000", 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1326e-05 2024-02-10T08:40:05.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907514+0000", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0261999999999999e-05 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907524+0000", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5769999999999999e-06 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907531+0000", 2024-02-10T08:40:05.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046026000000000001 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912134+0000", 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131586 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912265+0000", 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156537 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:40:05.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912422+0000", 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046267600000000003 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912884+0000", 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097266900000000001 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.913857+0000", 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035933647999999999 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949791+0000", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0361e-05 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949851+0000", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0658999999999998e-05 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949892+0000", 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003 2024-02-10T08:40:05.089 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:5 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected e118)", 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 192.491062935, 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006, 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 5 2024-02-10T08:40:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945621+0000", 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3675e-05 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945634+0000", 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1625e-05 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945656+0000", 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2959999999999999e-06 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945664+0000", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027177170000000001 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948382+0000", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9475000000000006e-05 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948472+0000", 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202852 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948674+0000", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010601493 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959276+0000", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075265439999999996 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.966802+0000", 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054018193999999999 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020821+0000", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5893999999999999e-05 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020857+0000", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9193999999999999e-05 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020886+0000", 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006 2024-02-10T08:40:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:787 5.9 5.84ada7c9 (undecoded) ondisk+write+known_if_redirected e121)", 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 189.41377216999999, 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001, 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:40:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 787 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022911+0000", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4569999999999994e-06 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022920+0000", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3113e-05 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022933+0000", 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9019999999999999e-06 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022941+0000", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011509515999999999 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034451+0000", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3540000000000006e-06 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034460+0000", 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.018581161 2024-02-10T08:40:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053041+0000", 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2311e-05 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053064+0000", 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:10 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e121)", 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 189.403934491, 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002, 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 10 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032749+0000", 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1640000000000004e-06 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032756+0000", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.67e-06 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032765+0000", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6840000000000002e-06 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032774+0000", 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073123999999999997 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033505+0000", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4190000000000005e-05 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033599+0000", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016687399999999999 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033766+0000", 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01101619 2024-02-10T08:40:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.044782+0000", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011076585 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055859+0000", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01553619 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071395+0000", 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8683999999999994e-05 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071474+0000", 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7649999999999999e-05 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071521+0000", 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:12 57.10 e122/115 57:0d24be56:test-rados-api-smithi107-112513-8::foo:head v 122'1, mlcod=0'0)", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 188.39629019200001, 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013632729, 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040393+0000", 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2047e-05 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040405+0000", 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6800000000000005e-06 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040415+0000", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2280000000000001e-06 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040422+0000", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012024386 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.052447+0000", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3753999999999998e-05 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.052530+0000", 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001465528 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053996+0000", 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2224999999999997e-05 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.054028+0000", 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013632729 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:256 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 171.451699848, 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001, 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 256 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984983+0000", 2024-02-10T08:40:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0989999999999999e-06 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984991+0000", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6600000000000007e-06 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985001+0000", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5549999999999997e-06 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985008+0000", 2024-02-10T08:40:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2148999999999998e-05 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985060+0000", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023027400000000001 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985291+0000", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033679382000000001 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.018970+0000", 2024-02-10T08:40:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9383999999999997e-05 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.019059+0000", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061927570000000001 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025252+0000", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6524000000000003e-05 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025309+0000", 2024-02-10T08:40:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104486 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025413+0000", 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:57 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:40:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 171.447392565, 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001, 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 57 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989292+0000", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989298+0000", 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5689999999999994e-06 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989307+0000", 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8519999999999998e-06 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989314+0000", 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2061000000000003e-05 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989356+0000", 2024-02-10T08:40:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8673999999999999e-05 2024-02-10T08:40:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989435+0000", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010247 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989537+0000", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000341224 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989878+0000", 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017592000000000001 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.990054+0000", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026858147999999998 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016912+0000", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4705000000000001e-05 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016947+0000", 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3204999999999997e-05 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016990+0000", 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:257 83.b 83:d4dba154:test-rados-api-smithi107-112490-12::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 171.408231511, 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017712418000000001, 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 257 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028452+0000", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0000000000000002e-06 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028458+0000", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1627000000000001e-05 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028470+0000", 2024-02-10T08:40:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.899e-06 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028475+0000", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1108e-05 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028516+0000", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014721600000000001 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028663+0000", 2024-02-10T08:40:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010976935 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.039640+0000", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9827000000000003e-05 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.039690+0000", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063247499999999996 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046014+0000", 2024-02-10T08:40:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2419000000000001e-05 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046067+0000", 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9597000000000002e-05 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046167+0000", 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017712418000000001 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:258 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 171.38862631699999, 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999, 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 258 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048057+0000", 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4479999999999999e-06 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048064+0000", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0129000000000001e-05 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048074+0000", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3009999999999997e-06 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048079+0000", 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9587e-05 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048109+0000", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012509199999999999 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048234+0000", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010028115000000001 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058262+0000", 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9953e-05 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058322+0000", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019508307999999999 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077830+0000", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2973999999999995e-05 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077913+0000", 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102942 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.078016+0000", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:59 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 15=[15,14] ondisk+write+ordersnap+known_if_redirected e140)", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 170.43363004899999, 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998, 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:40:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 59 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003053+0000", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1612e-05 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003064+0000", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4388e-05 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003079+0000", 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0430000000000008e-06 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003087+0000", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011723245 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014810+0000", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4276000000000003e-05 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014904+0000", 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002265094 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.017169+0000", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057871249999999997 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.022957+0000", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060097370000000002 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.028966+0000", 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013099348 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042066+0000", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7976000000000002e-05 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042124+0000", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4863000000000001e-05 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042159+0000", 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.51281.0:2 92.18 92:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 166.49452200900001, 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998, 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.51281", 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4152427963", 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:40:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942161+0000", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3089999999999996e-06 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942170+0000", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3520000000000007e-06 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942178+0000", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5319999999999997e-06 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942184+0000", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001344175 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943528+0000", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5959999999999998e-06 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943535+0000", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038514623999999997 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982050+0000", 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112311 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982162+0000", 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014368000000000001 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982306+0000", 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010079780000000001 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983314+0000", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036102999999999998 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983675+0000", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001564005 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985239+0000", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8597999999999998e-05 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985288+0000", 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7169999999999998e-05 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985325+0000", 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:505 60.0 e147/135 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head v 147'8, mlcod=146'5)", 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:20.933308+0000", 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 163.50337734499999, 2024-02-10T08:40:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010058958999999999, 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933308+0000", 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933308+0000", 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933308+0000", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3095000000000001e-05 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933321+0000", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933330+0000", 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3980000000000002e-06 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.933337+0000", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073648150000000003 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940702+0000", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1299000000000001e-05 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.940723+0000", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026139539999999999 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943337+0000", 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0032e-05 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:20.943367+0000", 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010058958999999999 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52646.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 154.50054629900001, 2024-02-10T08:40:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018858733999999999, 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52646", 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4246505494", 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936137+0000", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1719999999999996e-06 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936146+0000", 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3692999999999999e-05 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936160+0000", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2989999999999997e-06 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936168+0000", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010850230000000001 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.937253+0000", 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4284999999999999e-05 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.937278+0000", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099907429999999998 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.947268+0000", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2969999999999998e-06 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.947276+0000", 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061726589999999996 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953448+0000", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8418000000000006e-05 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953537+0000", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4968000000000003e-05 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953632+0000", 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00059628800000000005 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954228+0000", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8599999999999993e-06 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954236+0000", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00071013100000000002 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954946+0000", 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6857999999999999e-05 2024-02-10T08:40:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954973+0000", 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5301000000000001e-05 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954998+0000", 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018858733999999999 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52769.0:2 114.13 114:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-02-10T08:40:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 139.48436925799999, 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01324033, 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52769", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1173755769", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952314+0000", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7787000000000001e-05 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952332+0000", 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0054000000000001e-05 2024-02-10T08:40:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952342+0000", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2800000000000003e-06 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952350+0000", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00064678100000000003 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952997+0000", 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9650000000000008e-06 2024-02-10T08:40:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.953005+0000", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084007990000000005 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961406+0000", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012829000000000001 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961534+0000", 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017568099999999999 2024-02-10T08:40:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961710+0000", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016440770000000001 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.963354+0000", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6365999999999999e-05 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.963370+0000", 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002110373 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965481+0000", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9601000000000003e-05 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965520+0000", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6538999999999997e-05 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965557+0000", 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01324033 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.2.0:352 60.0 e196/135 60:01eb2b79:test-rados-api-smithi107-112490-14::foo33:4 v 196'72, mlcod=194'68)", 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.402999326, 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010767107, 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:40:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033685+0000", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1747e-05 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033697+0000", 2024-02-10T08:40:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1090000000000003e-06 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033703+0000", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.561e-06 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033708+0000", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021149799999999999 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033919+0000", 2024-02-10T08:40:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6986000000000001e-05 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033956+0000", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010464941 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.044421+0000", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2191999999999999e-05 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.044453+0000", 2024-02-10T08:40:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010767107 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.2.0:353 60.0 e196/135 60:053d865c:test-rados-api-smithi107-112490-14::foo38:4 v 196'74, mlcod=194'68)", 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.40297338800001, 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011387497999999999, 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.023e-06 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033717+0000", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7820000000000001e-06 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033722+0000", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4669999999999998e-06 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033726+0000", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073303079999999998 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.041056+0000", 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3583000000000001e-05 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.041080+0000", 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039962649999999997 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.045076+0000", 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3532000000000001e-05 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.045100+0000", 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011387497999999999 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47647.0:1 136.6s0 136:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e196)", 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.400768664, 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997, 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47647", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/993932140", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035914+0000", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6919999999999997e-06 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035924+0000", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3081000000000001e-05 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035937+0000", 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0079999999999996e-06 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035944+0000", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1489000000000001e-05 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035995+0000", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2339999999999999e-06 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035999+0000", 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91614844299999998 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952147+0000", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116906 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952264+0000", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048505899999999998 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952749+0000", 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069718000000000004 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.953446+0000", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013142239999999999 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954760+0000", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9851999999999995e-05 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954830+0000", 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.1.0:446 143.16 e213/209 143:6b2337e7:test-rados-api-smithi107-112490-20::foo:head v 213'1, mlcod=0'0)", 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:26.957557+0000", 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 97.479128826999997, 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097892699999999992, 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957557+0000", 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957557+0000", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957555+0000", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.364e-05 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957569+0000", 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2275e-05 2024-02-10T08:40:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957581+0000", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0039999999999999e-06 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957589+0000", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4145000000000003e-05 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957653+0000", 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.194e-05 2024-02-10T08:40:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.957675+0000", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096282529999999998 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.967304+0000", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2496000000000001e-05 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:26.967346+0000", 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097892699999999992 2024-02-10T08:40:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:1847 149.f 149:f30fcc83:test-rados-api-smithi107-112490-22::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e221)", 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 89.477229195000007, 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021778952000000001, 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1847 2024-02-10T08:40:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959454+0000", 2024-02-10T08:40:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3935e-05 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959477+0000", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3018e-05 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959491+0000", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4710000000000007e-06 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959499+0000", 2024-02-10T08:40:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9802999999999996e-05 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959569+0000", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8009999999999998e-06 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959576+0000", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015099468 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.974675+0000", 2024-02-10T08:40:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001121443 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.975796+0000", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043233430000000003 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.980120+0000", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0450000000000003e-05 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.980210+0000", 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00085321500000000005 2024-02-10T08:40:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981063+0000", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9449e-05 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981113+0000", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122416 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981235+0000", 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021778952000000001 2024-02-10T08:40:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.3.0:478 60.16 e231/135 60:6ce498fc:test-rados-api-smithi107-112490-24::bar:e v 231'62, mlcod=226'58)", 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 79.477282889999998, 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013769422999999999, 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:40:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959402+0000", 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6172999999999999e-05 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:40:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959418+0000", 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4239999999999998e-06 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959425+0000", 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.622e-06 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959432+0000", 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104139 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959536+0000", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6367999999999999e-05 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959563+0000", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013555905 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.973119+0000", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3390000000000002e-05 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:40:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.973172+0000", 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013769422999999999 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:40:05.192 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:40:05.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:40:05.669 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:05.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:05.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:05.900 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:05.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:05] "GET /metrics HTTP/1.1" 200 186240 "" "Prometheus/2.33.4" 2024-02-10T08:40:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:05.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:06.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:06.035 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.054 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:06.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:04.998532+0000 mon.a (mon.0) 2072 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]': finished 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:04.998624+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e311: 8 total, 8 up, 7 in 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:04.999719+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:05.000068+0000 mon.a (mon.0) 2074 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:05.000781+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:05.001136+0000 mon.a (mon.0) 2075 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:05.410510+0000 mgr.y (mgr.14553) 299 : cluster [DBG] pgmap v443: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:04.998532+0000 mon.a (mon.0) 2072 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]': finished 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:04.998624+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e311: 8 total, 8 up, 7 in 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:04.999719+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:05.000068+0000 mon.a (mon.0) 2074 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:05.000781+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:05.001136+0000 mon.a (mon.0) 2075 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:05.410510+0000 mgr.y (mgr.14553) 299 : cluster [DBG] pgmap v443: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:04.998532+0000 mon.a (mon.0) 2072 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_count","val": "8"}]': finished 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:04.998624+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e311: 8 total, 8 up, 7 in 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:04.999719+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:05.000068+0000 mon.a (mon.0) 2074 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:05.000781+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:05.001136+0000 mon.a (mon.0) 2075 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:06.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:05.410510+0000 mgr.y (mgr.14553) 299 : cluster [DBG] pgmap v443: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:06.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:06.336 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.358 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:06.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:06.490 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.506+0000 7f4d95284700 1 -- 172.21.15.107:0/817667506 >> v1:172.21.15.107:6789/0 conn(0x7f4d900ff710 legacy=0x7f4d90101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.506+0000 7f4d95284700 1 -- 172.21.15.107:0/817667506 shutdown_connections 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.506+0000 7f4d95284700 1 -- 172.21.15.107:0/817667506 >> 172.21.15.107:0/817667506 conn(0x7f4d900f2510 msgr2=0x7f4d900f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.506+0000 7f4d95284700 1 -- 172.21.15.107:0/817667506 shutdown_connections 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.506+0000 7f4d95284700 1 -- 172.21.15.107:0/817667506 wait complete. 2024-02-10T08:40:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.507+0000 7f4d95284700 1 librados: starting msgr at 2024-02-10T08:40:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.507+0000 7f4d95284700 1 librados: starting objecter 2024-02-10T08:40:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.507+0000 7f4d95284700 1 Processor -- start 2024-02-10T08:40:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 -- start start 2024-02-10T08:40:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 librados: setting wanted keys 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 librados: calling monclient init 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d90119480 con 0x7f4d90103250 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d9011a620 con 0x7f4d900ff710 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d95284700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d9011b800 con 0x7f4d900fc910 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d8effd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4d900fc910 0x7f4d900fb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48158/0 (socket says 172.21.15.107:48158) 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d8effd700 1 -- 172.21.15.107:0/1211038925 learned_addr learned my addr 172.21.15.107:0/1211038925 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter ms_handle_connect 0x7f4d900fc910 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter ms_handle_connect 0x7f4d90103250 2024-02-10T08:40:06.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter ms_handle_connect 0x7f4d900ff710 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171611431 0 0) 0x7f4d9011b800 con 0x7f4d900fc910 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d780038e0 con 0x7f4d900fc910 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571979523 0 0) 0x7f4d90119480 con 0x7f4d90103250 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.508+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9011b800 con 0x7f4d90103250 2024-02-10T08:40:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30567295 0 0) 0x7f4d9011a620 con 0x7f4d900ff710 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d90119480 con 0x7f4d900ff710 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3837085240 0 0) 0x7f4d780038e0 con 0x7f4d900fc910 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d9011a620 con 0x7f4d900fc910 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4d88001990 con 0x7f4d900fc910 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078230367 0 0) 0x7f4d9011b800 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d780038e0 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4d84001360 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068407335 0 0) 0x7f4d90119480 con 0x7f4d900ff710 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d9011b800 con 0x7f4d900ff710 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4d80002050 con 0x7f4d900ff710 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3915575044 0 0) 0x7f4d780038e0 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6790/0 conn(0x7f4d900fc910 legacy=0x7f4d900fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.195:6789/0 conn(0x7f4d900ff710 legacy=0x7f4d900fbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9011ca00 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d90119670 con 0x7f4d90103250 2024-02-10T08:40:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d95284700 10 client.27786.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d90119ae0 con 0x7f4d90103250 2024-02-10T08:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d95284700 1 librados: init done 2024-02-10T08:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4d840024c0 con 0x7f4d90103250 2024-02-10T08:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.509+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4d840033c0 con 0x7f4d90103250 2024-02-10T08:40:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.510+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4d7c004f00 con 0x7f4d90103250 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.510+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4d84016310 con 0x7f4d90103250 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(312..312 src has 50..312) v4 ==== 8079+0+0 (unknown 603070060 0 0) 0x7f4d84076500 con 0x7f4d90103250 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 osd_map(312..312 src has 50..312) v4 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 3 client.27786.objecter handle_osd_map got epochs [312,312] > 0 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 3 client.27786.objecter handle_osd_map decoding full epoch 312 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 20 client.27786.objecter dump_active .. 0 homeless 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v312) v1 ==== 54+0+16 (unknown 287102909 0 2420000192) 0x7f4d84078950 con 0x7f4d90103250 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 10 client.27786.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 1 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 1 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 0 1 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 4 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 10 client.27786.objecter _send_command 1 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.511+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c004950 con 0x7f4d7c004530 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d78063df0 2024-02-10T08:40:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c004530 2024-02-10T08:40:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c004950 con 0x7f4d7c004530 2024-02-10T08:40:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 10 client.27786.objecter _finish_command 1 = system:0 2024-02-10T08:40:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 4 2024-02-10T08:40:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.512+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 0 1 2024-02-10T08:40:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 10 client.27786.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 2 2024-02-10T08:40:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 2 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c0051a0 osd=0 3 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 0 2 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 4 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 10 client.27786.objecter _send_command 2 2024-02-10T08:40:06.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.527+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c004d30 con 0x7f4d7c004530 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.528+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c004d30 con 0x7f4d7c004530 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.528+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.528+0000 7f4d77fff700 10 client.27786.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.528+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 4 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.528+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 0 2 2024-02-10T08:40:06.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 10 client.27786.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 1 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 4 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 10 client.27786.objecter _send_command 3 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c00c000 con 0x7f4d7c004150 2024-02-10T08:40:06.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.529+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c004150 2024-02-10T08:40:06.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.530+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c00c000 con 0x7f4d7c004150 2024-02-10T08:40:06.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.530+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:06.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.530+0000 7f4d77fff700 10 client.27786.objecter _finish_command 3 = system:0 2024-02-10T08:40:06.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.530+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 4 2024-02-10T08:40:06.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.530+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 1 3 2024-02-10T08:40:06.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 10 client.27786.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 4 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 4 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c003fe0 osd=1 3 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 1 4 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 4 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 10 client.27786.objecter _send_command 4 2024-02-10T08:40:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.544+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c0018b0 con 0x7f4d7c004150 2024-02-10T08:40:06.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c0018b0 con 0x7f4d7c004150 2024-02-10T08:40:06.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 10 client.27786.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 4 2024-02-10T08:40:06.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 1 4 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 10 client.27786.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 5 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 5 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 2 5 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 4 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 10 client.27786.objecter _send_command 5 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c00de80 con 0x7f4d7c00d2f0 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.546+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c00d2f0 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.547+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c00de80 con 0x7f4d7c00d2f0 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.547+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.547+0000 7f4d77fff700 10 client.27786.objecter _finish_command 5 = system:0 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.547+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 4 2024-02-10T08:40:06.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.547+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 2 5 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 10 client.27786.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 6 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 6 2024-02-10T08:40:06.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c003630 osd=2 3 2024-02-10T08:40:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 2 6 2024-02-10T08:40:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 4 2024-02-10T08:40:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 10 client.27786.objecter _send_command 6 2024-02-10T08:40:06.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.561+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c0030a0 con 0x7f4d7c00d2f0 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c0030a0 con 0x7f4d7c00d2f0 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d77fff700 10 client.27786.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 4 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 2 6 2024-02-10T08:40:06.563 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d95284700 10 client.27786.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.562+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 7 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 7 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 3 7 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 4 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 10 client.27786.objecter _send_command 7 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c015aa0 con 0x7f4d7c011e50 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c011e50 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c015aa0 con 0x7f4d7c011e50 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 10 client.27786.objecter _finish_command 7 = system:0 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 4 2024-02-10T08:40:06.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.563+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 3 7 2024-02-10T08:40:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 10 client.27786.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 8 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 8 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c0034b0 osd=3 3 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 3 8 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 4 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 10 client.27786.objecter _send_command 8 2024-02-10T08:40:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.578+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c015aa0 con 0x7f4d7c011e50 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c015aa0 con 0x7f4d7c011e50 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 10 client.27786.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 4 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 3 8 2024-02-10T08:40:06.580 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 10 client.27786.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 9 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 9 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 4 9 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 4 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 10 client.27786.objecter _send_command 9 2024-02-10T08:40:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c016350 con 0x7f4d7c00ce50 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.580+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c00ce50 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.581+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c016350 con 0x7f4d7c00ce50 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.581+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.581+0000 7f4d77fff700 10 client.27786.objecter _finish_command 9 = system:0 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.581+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 4 2024-02-10T08:40:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.581+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 4 9 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 10 client.27786.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 10 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 10 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c015aa0 osd=4 3 2024-02-10T08:40:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 4 10 2024-02-10T08:40:06.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 4 2024-02-10T08:40:06.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 10 client.27786.objecter _send_command 10 2024-02-10T08:40:06.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.596+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c016350 con 0x7f4d7c00ce50 2024-02-10T08:40:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c016350 con 0x7f4d7c00ce50 2024-02-10T08:40:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d77fff700 10 client.27786.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 4 2024-02-10T08:40:06.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 4 10 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 10 client.27786.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 11 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 11 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 5 11 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 4 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 10 client.27786.objecter _send_command 11 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.598+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c016fb0 con 0x7f4d7c016a00 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c016a00 2024-02-10T08:40:06.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c016fb0 con 0x7f4d7c016a00 2024-02-10T08:40:06.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:06.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 10 client.27786.objecter _finish_command 11 = system:0 2024-02-10T08:40:06.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 4 2024-02-10T08:40:06.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.599+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 5 11 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 10 client.27786.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 12 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 12 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c016350 osd=5 3 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 5 12 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 4 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 10 client.27786.objecter _send_command 12 2024-02-10T08:40:06.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.613+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c016fb0 con 0x7f4d7c016a00 2024-02-10T08:40:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c016fb0 con 0x7f4d7c016a00 2024-02-10T08:40:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d77fff700 10 client.27786.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 4 2024-02-10T08:40:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 5 12 2024-02-10T08:40:06.616 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 10 client.27786.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 13 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 13 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 6 13 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 4 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 10 client.27786.objecter _send_command 13 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.615+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c022d60 con 0x7f4d7c01f0a0 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c01f0a0 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c022d60 con 0x7f4d7c01f0a0 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 10 client.27786.objecter _finish_command 13 = system:0 2024-02-10T08:40:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 4 2024-02-10T08:40:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.616+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 6 13 2024-02-10T08:40:06.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 10 client.27786.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 14 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 14 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c016fb0 osd=6 3 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 6 14 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 4 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 10 client.27786.objecter _send_command 14 2024-02-10T08:40:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.630+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c024510 con 0x7f4d7c01f0a0 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.632+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c024510 con 0x7f4d7c01f0a0 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.632+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.632+0000 7f4d77fff700 10 client.27786.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.632+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 4 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.632+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 6 14 2024-02-10T08:40:06.633 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 10 client.27786.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 15 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 15 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 7 15 2024-02-10T08:40:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 4 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 10 client.27786.objecter _send_command 15 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c025130 con 0x7f4d7c024bd0 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.633+0000 7f4d77fff700 10 client.27786.objecter ms_handle_connect 0x7f4d7c024bd0 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.634+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4d7c025130 con 0x7f4d7c024bd0 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.634+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.634+0000 7f4d77fff700 10 client.27786.objecter _finish_command 15 = system:0 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.634+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 4 2024-02-10T08:40:06.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.634+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 7 15 2024-02-10T08:40:06.645 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 10 client.27786.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign -1 16 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 20 client.27786.objecter _get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 15 client.27786.objecter _session_command_op_remove -1 16 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 20 client.27786.objecter get_session s=0x7f4d7c024510 osd=7 3 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 15 client.27786.objecter _session_command_op_assign 7 16 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 4 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 10 client.27786.objecter _send_command 16 2024-02-10T08:40:06.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.647+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7c025130 con 0x7f4d7c024bd0 2024-02-10T08:40:06.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d77fff700 1 -- 172.21.15.107:0/1211038925 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4d7c025130 con 0x7f4d7c024bd0 2024-02-10T08:40:06.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d77fff700 10 client.27786.objecter ms_dispatch 0x7f4d90001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:06.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d77fff700 10 client.27786.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d77fff700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 4 2024-02-10T08:40:06.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d77fff700 15 client.27786.objecter _session_command_op_remove 7 16 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 librados: watch_flush enter 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 librados: watch_flush exit 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.0 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6801/3321596972 conn(0x7f4d7c004530 legacy=0x7f4d7c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0051a0 osd=0 1 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.1 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6805/133516126 conn(0x7f4d7c004150 legacy=0x7f4d7c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003fe0 osd=1 1 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.2 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6809/3575881728 conn(0x7f4d7c00d2f0 legacy=0x7f4d7c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c003630 osd=2 1 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.3 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6813/2871825691 conn(0x7f4d7c011e50 legacy=0x7f4d7c014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c0034b0 osd=3 1 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.4 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.195:6800/2499671285 conn(0x7f4d7c00ce50 legacy=0x7f4d7c015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c015aa0 osd=4 1 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.5 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.195:6804/4178321486 conn(0x7f4d7c016a00 legacy=0x7f4d7c01d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d8f7fe700 1 -- 172.21.15.107:0/1211038925 reap_dead start 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d8f7fe700 1 -- 172.21.15.107:0/1211038925 reap_dead start 2024-02-10T08:40:06.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016350 osd=5 1 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.6 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.195:6808/1417479017 conn(0x7f4d7c01f0a0 legacy=0x7f4d7c021510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c016fb0 osd=6 1 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter close_session for osd.7 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.195:6812/1934176928 conn(0x7f4d7c024bd0 legacy=0x7f4d7c022d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter put_session s=0x7f4d7c024510 osd=7 1 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 20 client.27786.objecter shutdown clearing up homeless session... 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d95284700 10 client.27786.objecter successfully canceled tick 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.649+0000 7f4d8f7fe700 1 -- 172.21.15.107:0/1211038925 reap_dead start 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4d78063df0 legacy=0x7f4d780662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> v1:172.21.15.107:6789/0 conn(0x7f4d90103250 legacy=0x7f4d90117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 shutdown_connections 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 >> 172.21.15.107:0/1211038925 conn(0x7f4d900f2510 msgr2=0x7f4d900f5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 shutdown_connections 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 -- 172.21.15.107:0/1211038925 wait complete. 2024-02-10T08:40:06.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:06.650+0000 7f4d95284700 1 librados: shutdown 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.654 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:40:06.663 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:06.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:06.793 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:06.812 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:06.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:06.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:06.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.001859+0000 mon.a (mon.0) 2076 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.001953+0000 mon.a (mon.0) 2077 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:06.002024+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e312: 8 total, 8 up, 7 in 2024-02-10T08:40:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.002865+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.002945+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.003132+0000 mon.a (mon.0) 2079 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.003495+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:06.511770+0000 mon.a (mon.0) 2081 : audit [DBG] from='client.? v1:172.21.15.107:0/1211038925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.001859+0000 mon.a (mon.0) 2076 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.001953+0000 mon.a (mon.0) 2077 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:06.002024+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e312: 8 total, 8 up, 7 in 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.002865+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.002945+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.003132+0000 mon.a (mon.0) 2079 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.003495+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:06.511770+0000 mon.a (mon.0) 2081 : audit [DBG] from='client.? v1:172.21.15.107:0/1211038925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:07.253 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: ok, hit_set contains 188:602f83fe:::foo:head 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8990 ms) 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg_num = 32 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 0 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 1 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 2 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 3 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 4 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 5 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 6 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 7 ls 1707554408,0 2024-02-10T08:40:07.254 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 8 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 9 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 10 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 11 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 12 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 13 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 14 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 15 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 16 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 17 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 18 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 19 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 20 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 21 ls 1707554408,0 2024-02-10T08:40:07.255 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 22 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 23 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 24 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 25 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 26 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 27 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 28 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 29 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 30 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg 31 ls 1707554408,0 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: pg_num = 32 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6cac518f:::0:head 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:02547ec2:::1:head 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f905c69b:::2:head 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:cfc208b3:::3:head 2024-02-10T08:40:07.256 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d83876eb:::4:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b29083e3:::5:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c4fdafeb:::6:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:5c6b0b28:::7:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:bd63b0f1:::8:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e960b815:::9:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:52ea6a34:::10:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:89d3ae78:::11:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:de5d7c5f:::12:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:566253c9:::13:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:62a1935d:::14:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:863748b0:::15:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3958e169:::16:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4d4dabf9:::17:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:8391935d:::18:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:28883081:::19:head 2024-02-10T08:40:07.257 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:69259c59:::20:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4bdb80b7:::21:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a11c5d71:::22:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:271af37b:::23:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:95b121be:::24:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:58d1031b:::25:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:0a050783:::26:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c709704c:::27:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:cbe56eaf:::28:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:86b4b162:::29:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:70d89383:::30:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:dd450c7c:::31:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6d5729b1:::32:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c388f3fb:::33:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:56cfea31:::34:head 2024-02-10T08:40:07.258 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9dbc1bf7:::35:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:40b74ccd:::36:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4d5aaf42:::37:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:920f362c:::38:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6cc53222:::39:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9cad833f:::40:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:1ea84d41:::41:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c4480ef6:::42:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a694361e:::43:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d1bd33e9:::44:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ddc2cd5d:::45:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:2b782207:::46:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7b187fca:::47:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:90ecdf6f:::48:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a5ed95fe:::49:head 2024-02-10T08:40:07.259 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ea0eaa55:::50:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f33ef17b:::51:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a0d1b2f6:::52:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:60c5229e:::53:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:edcbc575:::54:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:102cf253:::55:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:efb7fb0b:::56:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:50d0a326:::57:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d4dc5daf:::58:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3a130462:::59:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ec87ed71:::60:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d5bc9454:::61:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3ddfe313:::62:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7c2816b9:::63:head 2024-02-10T08:40:07.260 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:47e00e4d:::64:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c6410c18:::65:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b48ed237:::66:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:cd63ad31:::67:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b179e92b:::68:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:0d9f741a:::69:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6d3352ae:::70:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c6d5c19e:::71:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:bc4729c3:::72:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:77e930b9:::73:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:0abeecfd:::74:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b7c37e15:::75:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b6378398:::76:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:02bd68de:::77:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:cc795d2d:::78:head 2024-02-10T08:40:07.261 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:630d4fea:::79:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e0d29ef5:::80:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:fd6f13d2:::81:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:606461d5:::82:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:eadbdc43:::83:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:8761d0bb:::84:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9ef0186f:::85:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e0d41294:::86:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:961de695:::87:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:1423148f:::88:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:633a8fa2:::89:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a8653809:::90:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3dac8b33:::91:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:35aad435:::92:head 2024-02-10T08:40:07.262 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f6dcc343:::93:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:dbbdad87:::94:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:1cb48ce0:::95:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:03cd461c:::96:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:17a4ea99:::97:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9993c9a7:::98:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6394211c:::99:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:94c7ae57:::100:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6fdee5bb:::101:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9a477fd1:::102:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:eb850916:::103:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:affc56b9:::104:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b42dc814:::105:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f319f8f0:::106:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9a40b9de:::107:head 2024-02-10T08:40:07.263 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:8b524f28:::108:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e3de589f:::109:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:90f90a5b:::110:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a7b4f1d7:::111:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:af51766e:::112:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b6f90bd1:::113:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e0261208:::114:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c9569ef7:::115:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:61bebe50:::116:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:fe93412b:::117:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d3d38bee:::118:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3100ba0c:::119:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d0560ada:::120:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f0ea8b35:::121:head 2024-02-10T08:40:07.264 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:766f231a:::122:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a07a2582:::123:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:bd7c6b3a:::124:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:fb2ddaff:::125:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4408e1fe:::126:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ee1df7a7:::127:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c3002909:::128:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4f48ffa9:::129:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:edf38733:::130:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c08425c0:::131:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:5f902d98:::132:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:41ea2c93:::133:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:813cee13:::134:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:0131818d:::135:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:26ba5a85:::136:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:381b8a5a:::137:head 2024-02-10T08:40:07.265 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:28797e47:::138:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:bfca7f22:::139:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:36807075:::140:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:80b03975:::141:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:5c15709b:::142:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f39ea15e:::143:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ea992956:::144:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:48887b1c:::145:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:9f24a9dd:::146:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:987f100b:::147:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d2dd3581:::148:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7fed1808:::149:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c80b70e9:::150:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:85ed90f9:::151:head 2024-02-10T08:40:07.266 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:36428b24:::152:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d044c34a:::153:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7c18bf58:::154:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d1c21232:::155:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a7a3c575:::156:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:87da0633:::157:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d5ac3822:::158:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3f20522d:::159:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6ca26563:::160:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:532ce135:::161:head 2024-02-10T08:40:07.267 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:c78863e6:::162:head 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.001859+0000 mon.a (mon.0) 2076 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.001953+0000 mon.a (mon.0) 2077 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:06.002024+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e312: 8 total, 8 up, 7 in 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.002865+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.107:0/446603345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.002945+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.003132+0000 mon.a (mon.0) 2079 : audit [INF] from='client.53345 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]: dispatch 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.003495+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T08:40:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:06.511770+0000 mon.a (mon.0) 2081 : audit [DBG] from='client.? v1:172.21.15.107:0/1211038925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.004184+0000 mon.a (mon.0) 2082 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.004276+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T08:40:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:07.004340+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e313: 8 total, 8 up, 7 in 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.030212+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.031514+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.031831+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.217176+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-41","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.314041+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.314404+0000 mon.a (mon.0) 2088 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.314838+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:07.315006+0000 mon.a (mon.0) 2089 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:07.356848+0000 osd.5 (osd.5) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:08.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:07.356959+0000 osd.5 (osd.5) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:08.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:07.410945+0000 mgr.y (mgr.14553) 300 : cluster [DBG] pgmap v446: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.004184+0000 mon.a (mon.0) 2082 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.004276+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:07.004340+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e313: 8 total, 8 up, 7 in 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.030212+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.031514+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.031831+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.217176+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-41","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.314041+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.314404+0000 mon.a (mon.0) 2088 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.314838+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:07.315006+0000 mon.a (mon.0) 2089 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:07.356848+0000 osd.5 (osd.5) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:07.356959+0000 osd.5 (osd.5) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:07.410945+0000 mgr.y (mgr.14553) 300 : cluster [DBG] pgmap v446: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.004184+0000 mon.a (mon.0) 2082 : audit [INF] from='client.53345 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi107-112411-54"}]': finished 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.004276+0000 mon.a (mon.0) 2083 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:07.004340+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e313: 8 total, 8 up, 7 in 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.030212+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.031514+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.031831+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.217176+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi107-112490-41","var": "pg_num","format": "json"}]: dispatch 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.314041+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.314404+0000 mon.a (mon.0) 2088 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:08.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.314838+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:07.315006+0000 mon.a (mon.0) 2089 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]: dispatch 2024-02-10T08:40:08.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:07.356848+0000 osd.5 (osd.5) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:08.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:07.356959+0000 osd.5 (osd.5) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:08.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:07.410945+0000 mgr.y (mgr.14553) 300 : cluster [DBG] pgmap v446: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:08.494 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:08.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:08.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:08.626 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:08.645 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:08.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:08.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:08.777 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:08.797 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:08.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:08.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:08.930 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.932+0000 7f11e57ed700 1 -- 172.21.15.107:0/1182592490 >> v1:172.21.15.107:6789/0 conn(0x7f11e00f8a40 legacy=0x7f11e00f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 -- 172.21.15.107:0/1182592490 shutdown_connections 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 -- 172.21.15.107:0/1182592490 >> 172.21.15.107:0/1182592490 conn(0x7f11e00f2510 msgr2=0x7f11e00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 -- 172.21.15.107:0/1182592490 shutdown_connections 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 -- 172.21.15.107:0/1182592490 wait complete. 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 librados: starting msgr at 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 librados: starting objecter 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 Processor -- start 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 -- start start 2024-02-10T08:40:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 librados: setting wanted keys 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.933+0000 7f11e57ed700 1 librados: calling monclient init 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11e57ed700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11e0117350 con 0x7f11e00f9530 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11e57ed700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11e0118530 con 0x7f11e0104340 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11e57ed700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11e0119730 con 0x7f11e00f8a40 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11de7fc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f11e00f9530 0x7f11e00ffc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50908/0 (socket says 172.21.15.107:50908) 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11de7fc700 1 -- 172.21.15.107:0/2357500636 learned_addr learned my addr 172.21.15.107:0/2357500636 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter ms_handle_connect 0x7f11e00f8a40 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter ms_handle_connect 0x7f11e00f9530 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258283069 0 0) 0x7f11e0119730 con 0x7f11e00f8a40 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003800 con 0x7f11e00f8a40 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185154250 0 0) 0x7f11e0117350 con 0x7f11e00f9530 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0119730 con 0x7f11e00f9530 2024-02-10T08:40:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter ms_handle_connect 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627243257 0 0) 0x7f11e0118530 con 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0117350 con 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382994029 0 0) 0x7f11c8003800 con 0x7f11e00f8a40 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e0118530 con 0x7f11e00f8a40 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11d0002000 con 0x7f11e00f8a40 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180429577 0 0) 0x7f11e0119730 con 0x7f11e00f9530 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c8003800 con 0x7f11e00f9530 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11d40018d0 con 0x7f11e00f9530 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.934+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010742300 0 0) 0x7f11e0117350 con 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e0119730 con 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11d8001c40 con 0x7f11e0104340 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3309397912 0 0) 0x7f11e0118530 con 0x7f11e00f8a40 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 >> v1:172.21.15.195:6789/0 conn(0x7f11e0104340 legacy=0x7f11e0115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 >> v1:172.21.15.107:6789/0 conn(0x7f11e00f9530 legacy=0x7f11e00ffc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e011a930 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e0117540 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11e57ed700 10 client.53369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e01179b0 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11e57ed700 1 librados: init done 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f11d0001a80 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.935+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11d0003750 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.936+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc004f20 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.936+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f11d0002b90 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(314..314 src has 50..314) v4 ==== 8257+0+0 (unknown 3344346975 0 0) 0x7f11d0076900 con 0x7f11e00f8a40 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 10 client.53369.objecter ms_dispatch 0x7f11e0001550 osd_map(314..314 src has 50..314) v4 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 3 client.53369.objecter handle_osd_map got epochs [314,314] > 0 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 3 client.53369.objecter handle_osd_map decoding full epoch 314 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 20 client.53369.objecter dump_active .. 0 homeless 2024-02-10T08:40:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.937+0000 7f11c7fff700 10 client.53369.objecter ms_handle_connect 0x7f11c8063d00 2024-02-10T08:40:08.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:08.941+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f11d0049c50 con 0x7f11e00f8a40 2024-02-10T08:40:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:08.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:09.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:09.088 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:09.162+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f11cc0051d0 con 0x7f11e00f8a40 2024-02-10T08:40:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:09.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:09.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:09.343 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:08.019794+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:08.019881+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]': finished 2024-02-10T08:40:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:08.019951+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-02-10T08:40:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:08.020597+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:09.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:08.019794+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:09.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:08.019881+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]': finished 2024-02-10T08:40:09.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:08.019951+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-02-10T08:40:09.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:08.020597+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:08.019794+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi107-112411-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:08.019881+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-41"}]': finished 2024-02-10T08:40:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:08.019951+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-02-10T08:40:09.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:08.020597+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:09.770 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:09.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:09.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:09.889 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:40:09.889 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:40:09.890 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2024-02-10T08:40:09.890 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2024-02-10T08:40:09.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:40:09.907 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:10.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.023+0000 7f11c7fff700 1 -- 172.21.15.107:0/2357500636 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v316) v1 ==== 97+0+0 (unknown 25229231 0 0) 0x7f11d00480c0 con 0x7f11e00f8a40 2024-02-10T08:40:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.026+0000 7f11e57ed700 10 librados: watch_flush enter 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 10 librados: watch_flush exit 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 20 client.53369.objecter shutdown clearing up homeless session... 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 10 client.53369.objecter successfully canceled tick 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 >> v1:172.21.15.107:6800/2634842009 conn(0x7f11c8063d00 legacy=0x7f11c80661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 >> v1:172.21.15.107:6790/0 conn(0x7f11e00f8a40 legacy=0x7f11e00ff540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 shutdown_connections 2024-02-10T08:40:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.029+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 >> 172.21.15.107:0/2357500636 conn(0x7f11e00f2510 msgr2=0x7f11e01077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.030+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 shutdown_connections 2024-02-10T08:40:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.031+0000 7f11e57ed700 1 -- 172.21.15.107:0/2357500636 wait complete. 2024-02-10T08:40:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:10.032+0000 7f11e57ed700 1 librados: shutdown 2024-02-10T08:40:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:09.163777+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.107:0/2357500636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:09.274583+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:09.275254+0000 mon.a (mon.0) 2095 : audit [INF] from='client.53369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:09.411310+0000 mgr.y (mgr.14553) 301 : cluster [DBG] pgmap v449: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:10.016619+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.022900+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.022982+0000 mon.a (mon.0) 2098 : audit [INF] from='client.53369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:10.023058+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.026004+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.028109+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.155239+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:09.163777+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.107:0/2357500636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:09.274583+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:09.275254+0000 mon.a (mon.0) 2095 : audit [INF] from='client.53369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:09.411310+0000 mgr.y (mgr.14553) 301 : cluster [DBG] pgmap v449: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:10.016619+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.022900+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.022982+0000 mon.a (mon.0) 2098 : audit [INF] from='client.53369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:10.023058+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.026004+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.028109+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.155239+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:10.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:40:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:09.163777+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.107:0/2357500636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:09.274583+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-02-10T08:40:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:09.275254+0000 mon.a (mon.0) 2095 : audit [INF] from='client.53369 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:09.411310+0000 mgr.y (mgr.14553) 301 : cluster [DBG] pgmap v449: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:10.016619+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.022900+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi107-112411-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.022982+0000 mon.a (mon.0) 2098 : audit [INF] from='client.53369 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:10.023058+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.026004+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.028109+0000 mon.a (mon.0) 2100 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.155239+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:11.436 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:40:11.436 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:40:11.458 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:11.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:11.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:11.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:11.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:11.776 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:11.778 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:11.800 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:11.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:11.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:11.936 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:11.957 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:11.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:10.979082+0000 mon.a (mon.0) 2102 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:10.979196+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-02-10T08:40:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.002872+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.003585+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.356947+0000 mgr.y (mgr.14553) 302 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:12.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:11.412176+0000 mgr.y (mgr.14553) 303 : cluster [DBG] pgmap v452: 226 pgs: 226 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:12.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:12.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:12.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:12.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:10.979082+0000 mon.a (mon.0) 2102 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:12.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:10.979196+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.002872+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.003585+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.356947+0000 mgr.y (mgr.14553) 302 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:11.412176+0000 mgr.y (mgr.14553) 303 : cluster [DBG] pgmap v452: 226 pgs: 226 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:10.979082+0000 mon.a (mon.0) 2102 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:10.979196+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.002872+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.003585+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.356947+0000 mgr.y (mgr.14553) 302 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:12.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:11.412176+0000 mgr.y (mgr.14553) 303 : cluster [DBG] pgmap v452: 226 pgs: 226 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:12.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:12.240 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:12.439 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:12.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:12.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:40:12.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:12.609 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:12.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:12.740 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:12.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:13.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:13.070 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:13.168 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:13.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:13.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.980814+0000 mon.a (mon.0) 2105 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:13.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:11.980884+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-02-10T08:40:13.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.984372+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.986173+0000 mon.a (mon.0) 2107 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:11.986365+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:12.982382+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:12.982466+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:12.982503+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.980814+0000 mon.a (mon.0) 2105 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:11.980884+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.984372+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.986173+0000 mon.a (mon.0) 2107 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:11.986365+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:13.206 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:12.982382+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:40:13.207 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:12.982466+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:13.207 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:12.982503+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-02-10T08:40:13.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:40:13.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.980814+0000 mon.a (mon.0) 2105 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:11.980884+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.984372+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.986173+0000 mon.a (mon.0) 2107 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:11.986365+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:12.982382+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:12.982466+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:12.982503+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-02-10T08:40:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:13.325 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:13.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:13.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:13.459 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:13.478 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:13.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:13.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:13.610 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:13.628 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:13.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:13.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:13.758 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:13.777 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:13.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:13.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:13.908 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:12.984810+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:12.987097+0000 mon.a (mon.0) 2112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:12.987259+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:13.412524+0000 mgr.y (mgr.14553) 304 : cluster [DBG] pgmap v455: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:13.984298+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:13.984382+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:13.984423+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:12.984810+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:12.987097+0000 mon.a (mon.0) 2112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:12.987259+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:13.412524+0000 mgr.y (mgr.14553) 304 : cluster [DBG] pgmap v455: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:13.984298+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:13.984382+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:14.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:13.984423+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:12.984810+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:12.987097+0000 mon.a (mon.0) 2112 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:12.987259+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]: dispatch 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:13.412524+0000 mgr.y (mgr.14553) 304 : cluster [DBG] pgmap v455: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:13.984298+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:13.984382+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.107:0/133081161' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi107-112411-55"}]': finished 2024-02-10T08:40:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:13.984423+0000 mon.a (mon.0) 2116 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-02-10T08:40:15.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:13.988791+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:13.989040+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.004644+0000 mon.c (mon.2) 344 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.004918+0000 mon.a (mon.0) 2118 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.005693+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.005887+0000 mon.a (mon.0) 2119 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.006216+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.006417+0000 mon.a (mon.0) 2120 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.986493+0000 mon.a (mon.0) 2121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.986573+0000 mon.a (mon.0) 2122 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:14.986641+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.987263+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.080 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:13.988791+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:13.989040+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.004644+0000 mon.c (mon.2) 344 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.004918+0000 mon.a (mon.0) 2118 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.005693+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.005887+0000 mon.a (mon.0) 2119 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.006216+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.006417+0000 mon.a (mon.0) 2120 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.986493+0000 mon.a (mon.0) 2121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.986573+0000 mon.a (mon.0) 2122 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:14.986641+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-02-10T08:40:15.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.987263+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.987318+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.987604+0000 mon.a (mon.0) 2124 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:14.988010+0000 mon.a (mon.0) 2125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.987318+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.987604+0000 mon.a (mon.0) 2124 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.082 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:14.988010+0000 mon.a (mon.0) 2125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:13.988791+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:13.989040+0000 mon.a (mon.0) 2117 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.004644+0000 mon.c (mon.2) 344 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.004918+0000 mon.a (mon.0) 2118 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.005693+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.005887+0000 mon.a (mon.0) 2119 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.006216+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.006417+0000 mon.a (mon.0) 2120 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.986493+0000 mon.a (mon.0) 2121 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.986573+0000 mon.a (mon.0) 2122 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi107-112411-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:14.986641+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.987263+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.987318+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.987604+0000 mon.a (mon.0) 2124 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:15.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:14.988010+0000 mon.a (mon.0) 2125 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 -- 172.21.15.107:0/40557673 >> v1:172.21.15.107:6789/0 conn(0x7f90e4104350 legacy=0x7f90e4106810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 -- 172.21.15.107:0/40557673 shutdown_connections 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 -- 172.21.15.107:0/40557673 >> 172.21.15.107:0/40557673 conn(0x7f90e406c090 msgr2=0x7f90e406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 -- 172.21.15.107:0/40557673 shutdown_connections 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 -- 172.21.15.107:0/40557673 wait complete. 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 librados: starting msgr at 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 librados: starting objecter 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.544+0000 7f90e9fd1700 1 Processor -- start 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 -- start start 2024-02-10T08:40:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 librados: setting wanted keys 2024-02-10T08:40:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 librados: calling monclient init 2024-02-10T08:40:15.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90e40f8eb0 con 0x7f90e40fd9f0 2024-02-10T08:40:15.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90e40f9040 con 0x7f90e4100840 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e9fd1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90e40f9230 con 0x7f90e4104350 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e37fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f90e40fd9f0 0x7f90e40fcae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54326/0 (socket says 172.21.15.107:54326) 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e37fe700 1 -- 172.21.15.107:0/4286745215 learned_addr learned my addr 172.21.15.107:0/4286745215 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter ms_handle_connect 0x7f90e4104350 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter ms_handle_connect 0x7f90e40fd9f0 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963275698 0 0) 0x7f90e40f9230 con 0x7f90e4104350 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc003030 con 0x7f90e4104350 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993317028 0 0) 0x7f90e40f8eb0 con 0x7f90e40fd9f0 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e40f9230 con 0x7f90e40fd9f0 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter ms_handle_connect 0x7f90e4100840 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:15.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 681083748 0 0) 0x7f90cc003030 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e40f8eb0 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809800487 0 0) 0x7f90e40f9040 con 0x7f90e4100840 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc003030 con 0x7f90e4100840 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.545+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f90dc002320 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054688118 0 0) 0x7f90e40f9230 con 0x7f90e40fd9f0 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e40f9040 con 0x7f90e40fd9f0 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f90d4001800 con 0x7f90e40fd9f0 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3185759129 0 0) 0x7f90e40f8eb0 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 >> v1:172.21.15.195:6789/0 conn(0x7f90e4100840 legacy=0x7f90e4114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 >> v1:172.21.15.107:6789/0 conn(0x7f90e40fd9f0 legacy=0x7f90e40fcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e411cc20 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e4119c50 con 0x7f90e4104350 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e9fd1700 10 client.53381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:15.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e411a0c0 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e9fd1700 1 librados: init done 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f90dc001da0 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.546+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f90dc003a90 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d0004f20 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f90dc0169e0 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(321..321 src has 50..321) v4 ==== 8285+0+0 (unknown 3568786271 0 0) 0x7f90dc076c20 con 0x7f90e4104350 2024-02-10T08:40:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e0ff9700 10 client.53381.objecter ms_dispatch 0x7f90e4001550 osd_map(321..321 src has 50..321) v4 2024-02-10T08:40:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e0ff9700 3 client.53381.objecter handle_osd_map got epochs [321,321] > 0 2024-02-10T08:40:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.547+0000 7f90e0ff9700 3 client.53381.objecter handle_osd_map decoding full epoch 321 2024-02-10T08:40:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.548+0000 7f90e0ff9700 20 client.53381.objecter dump_active .. 0 homeless 2024-02-10T08:40:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.548+0000 7f90e0ff9700 10 client.53381.objecter ms_handle_connect 0x7f90cc063cf0 2024-02-10T08:40:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.552+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f90dc04a100 con 0x7f90e4104350 2024-02-10T08:40:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:15.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:15.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:15.693 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:15.714 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.715+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90d0000b90 con 0x7f90cc063cf0 2024-02-10T08:40:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:15.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.735+0000 7f90e0ff9700 1 -- 172.21.15.107:0/4286745215 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553562 (unknown 2965378022 0 272628513) 0x7f90d0000b90 con 0x7f90cc063cf0 2024-02-10T08:40:15.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:40:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 10 librados: watch_flush enter 2024-02-10T08:40:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 10 librados: watch_flush exit 2024-02-10T08:40:15.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 20 client.53381.objecter shutdown clearing up homeless session... 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 10 client.53381.objecter successfully canceled tick 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 >> v1:172.21.15.107:6800/2634842009 conn(0x7f90cc063cf0 legacy=0x7f90cc0661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 >> v1:172.21.15.107:6790/0 conn(0x7f90e4104350 legacy=0x7f90e41184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 shutdown_connections 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 >> 172.21.15.107:0/4286745215 conn(0x7f90e406c090 msgr2=0x7f90e41067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 shutdown_connections 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 -- 172.21.15.107:0/4286745215 wait complete. 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:15.739+0000 7f90e9fd1700 1 librados: shutdown 2024-02-10T08:40:15.752 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:40:15.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:15] "GET /metrics HTTP/1.1" 200 188897 "" "Prometheus/2.33.4" 2024-02-10T08:40:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:16.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:16.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:16.333 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:16.333 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:16.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:15.412945+0000 mgr.y (mgr.14553) 305 : cluster [DBG] pgmap v458: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:16.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:15.717714+0000 mgr.y (mgr.14553) 306 : audit [DBG] from='client.53381 v1:172.21.15.107:0/4286745215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:16.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:15.412945+0000 mgr.y (mgr.14553) 305 : cluster [DBG] pgmap v458: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:16.335 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:15.717714+0000 mgr.y (mgr.14553) 306 : audit [DBG] from='client.53381 v1:172.21.15.107:0/4286745215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:16.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:16.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:16.465 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:15.412945+0000 mgr.y (mgr.14553) 305 : cluster [DBG] pgmap v458: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:15.717714+0000 mgr.y (mgr.14553) 306 : audit [DBG] from='client.53381 v1:172.21.15.107:0/4286745215' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:16.888 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-02-10T08:40:15.412728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007021,"num_objects":316,"num_object_clones":2,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1227,"num_read_kb":885,"num_write":983,"num_write_kb":4497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2302,"ondisk_log_size":2302,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":631,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2193688,"kb_used_data":47532,"kb_used_omap":0,"kb_used_meta":2146048,"kb_avail":654022376,"statfs":{"total":671965249536,"available":669718913024,"internally_reserved":0,"allocated":48672768,"data_stored":32357911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2197553152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001241"},"pg_stats":[{"pgid":"193.1f","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:11.001150+0000","last_change":"2024-02-10T08:40:11.001150+0000","last_active":"2024-02-10T08:40:11.001150+0000","last_peered":"2024-02-10T08:40:11.001150+0000","last_clean":"2024-02-10T08:40:11.001150+0000","last_became_active":"2024-02-10T08:40:11.000866+0000","last_became_peered":"2024-02-10T08:40:11.000866+0000","last_unstale":"2024-02-10T08:40:11.001150+0000","last_undegraded":"2024-02-10T08:40:11.001150+0000","last_fullsized":"2024-02-10T08:40:11.001150+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"193.1e","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.992060+0000","last_change":"2024-02-10T08:40:10.992060+0000","last_active":"2024-02-10T08:40:10.992060+0000","last_peered":"2024-02-10T08:40:10.992060+0000","last_clean":"2024-02-10T08:40:10.992060+0000","last_became_active":"2024-02-10T08:40:10.991418+0000","last_became_peered":"2024-02-10T08:40:10.991418+0000","last_unstale":"2024-02-10T08:40:10.992060+0000","last_undegraded":"2024-02-10T08:40:10.992060+0000","last_fullsized":"2024-02-10T08:40:10.992060+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.1d","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.997978+0000","last_change":"2024-02-10T08:40:10.997978+0000","last_active":"2024-02-10T08:40:10.997978+0000","last_peered":"2024-02-10T08:40:10.997978+0000","last_clean":"2024-02-10T08:40:10.997978+0000","last_became_active":"2024-02-10T08:40:10.997663+0000","last_became_peered":"2024-02-10T08:40:10.997663+0000","last_unstale":"2024-02-10T08:40:10.997978+0000","last_undegraded":"2024-02-10T08:40:10.997978+0000","last_fullsized":"2024-02-10T08:40:10.997978+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.1c","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.999320+0000","last_change":"2024-02-10T08:40:10.999320+0000","last_active":"2024-02-10T08:40:10.999320+0000","last_peered":"2024-02-10T08:40:10.999320+0000","last_clean":"2024-02-10T08:40:10.999320+0000","last_became_active":"2024-02-10T08:40:10.999012+0000","last_became_peered":"2024-02-10T08:40:10.999012+0000","last_unstale":"2024-02-10T08:40:10.999320+0000","last_undegraded":"2024-02-10T08:40:10.999320+0000","last_fullsized":"2024-02-10T08:40:10.999320+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.1b","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.996492+0000","last_change":"2024-02-10T08:40:10.996492+0000","last_active":"2024-02-10T08:40:10.996492+0000","last_peered":"2024-02-10T08:40:10.996492+0000","last_clean":"2024-02-10T08:40:10.996492+0000","last_became_active":"2024-02-10T08:40:10.996185+0000","last_became_peered":"2024-02-10T08:40:10.996185+0000","last_unstale":"2024-02-10T08:40:10.996492+0000","last_undegraded":"2024-02-10T08:40:10.996492+0000","last_fullsized":"2024-02-10T08:40:10.996492+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.1a","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.999833+0000","last_change":"2024-02-10T08:40:10.999833+0000","last_active":"2024-02-10T08:40:10.999833+0000","last_peered":"2024-02-10T08:40:10.999833+0000","last_clean":"2024-02-10T08:40:10.999833+0000","last_became_active":"2024-02-10T08:40:10.999545+0000","last_became_peered":"2024-02-10T08:40:10.999545+0000","last_unstale":"2024-02-10T08:40:10.999833+0000","last_undegraded":"2024-02-10T08:40:10.999833+0000","last_fullsized":"2024-02-10T08:40:10.999833+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.19","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:11.001145+0000","last_change":"2024-02-10T08:40:11.001145+0000","last_active":"2024-02-10T08:40:11.001145+0000","last_peered":"2024-02-10T08:40:11.001145+0000","last_clean":"2024-02-10T08:40:11.001145+0000","last_became_active":"2024-02-10T08:40:11.000863+0000","last_became_peered":"2024-02-10T08:40:11.000863+0000","last_unstale":"2024-02-10T08:40:11.001145+0000","last_undegraded":"2024-02-10T08:40:11.001145+0000","last_fullsized":"2024-02-10T08:40:11.001145+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.18","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.999780+0000","last_change":"2024-02-10T08:40:10.999780+0000","last_active":"2024-02-10T08:40:10.999780+0000","last_peered":"2024-02-10T08:40:10.999780+0000","last_clean":"2024-02-10T08:40:10.999780+0000","last_became_active":"2024-02-10T08:40:10.999470+0000","last_became_peered":"2024-02-10T08:40:10.999470+0000","last_unstale":"2024-02-10T08:40:10.999780+0000","last_undegraded":"2024-02-10T08:40:10.999780+0000","last_fullsized":"2024-02-10T08:40:10.999780+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"193.17","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.995557+0000","last_change":"2024-02-10T08:40:10.995557+0000","last_active":"2024-02-10T08:40:10.995557+0000","last_peered":"2024-02-10T08:40:10.995557+0000","last_clean":"2024-02-10T08:40:10.995557+0000","last_became_active":"2024-02-10T08:40:10.995284+0000","last_became_peered":"2024-02-10T08:40:10.995284+0000","last_unstale":"2024-02-10T08:40:10.995557+0000","last_undegraded":"2024-02-10T08:40:10.995557+0000","last_fullsized":"2024-02-10T08:40:10.995557+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.16","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.992850+0000","last_change":"2024-02-10T08:40:10.992850+0000","last_active":"2024-02-10T08:40:10.992850+0000","last_peered":"2024-02-10T08:40:10.992850+0000","last_clean":"2024-02-10T08:40:10.992850+0000","last_became_active":"2024-02-10T08:40:10.992503+0000","last_became_peered":"2024-02-10T08:40:10.992503+0000","last_unstale":"2024-02-10T08:40:10.992850+0000","last_undegraded":"2024-02-10T08:40:10.992850+0000","last_fullsized":"2024-02-10T08:40:10.992850+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.15","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.999873+0000","last_change":"2024-02-10T08:40:10.999873+0000","last_active":"2024-02-10T08:40:10.999873+0000","last_peered":"2024-02-10T08:40:10.999873+0000","last_clean":"2024-02-10T08:40:10.999873+0000","last_became_active":"2024-02-10T08:40:10.999588+0000","last_became_peered":"2024-02-10T08:40:10.999588+0000","last_unstale":"2024-02-10T08:40:10.999873+0000","last_undegraded":"2024-02-10T08:40:10.999873+0000","last_fullsized":"2024-02-10T08:40:10.999873+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.14","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.994674+0000","last_change":"2024-02-10T08:40:10.994674+0000","last_active":"2024-02-10T08:40:10.994674+0000","last_peered":"2024-02-10T08:40:10.994674+0000","last_clean":"2024-02-10T08:40:10.994674+0000","last_became_active":"2024-02-10T08:40:10.994408+0000","last_became_peered":"2024-02-10T08:40:10.994408+0000","last_unstale":"2024-02-10T08:40:10.994674+0000","last_undegraded":"2024-02-10T08:40:10.994674+0000","last_fullsized":"2024-02-10T08:40:10.994674+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.13","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.988435+0000","last_change":"2024-02-10T08:40:10.988435+0000","last_active":"2024-02-10T08:40:10.988435+0000","last_peered":"2024-02-10T08:40:10.988435+0000","last_clean":"2024-02-10T08:40:10.988435+0000","last_became_active":"2024-02-10T08:40:10.988148+0000","last_became_peered":"2024-02-10T08:40:10.988148+0000","last_unstale":"2024-02-10T08:40:10.988435+0000","last_undegraded":"2024-02-10T08:40:10.988435+0000","last_fullsized":"2024-02-10T08:40:10.988435+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"193.12","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.989837+0000","last_change":"2024-02-10T08:40:10.989837+0000","last_active":"2024-02-10T08:40:10.989837+0000","last_peered":"2024-02-10T08:40:10.989837+0000","last_clean":"2024-02-10T08:40:10.989837+0000","last_became_active":"2024-02-10T08:40:10.989231+0000","last_became_peered":"2024-02-10T08:40:10.989231+0000","last_unstale":"2024-02-10T08:40:10.989837+0000","last_undegraded":"2024-02-10T08:40:10.989837+0000","last_fullsized":"2024-02-10T08:40:10.989837+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"193.11","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:11.001016+0000","last_change":"2024-02-10T08:40:11.001016+0000","last_active":"2024-02-10T08:40:11.001016+0000","last_peered":"2024-02-10T08:40:11.001016+0000","last_clean":"2024-02-10T08:40:11.001016+0000","last_became_active":"2024-02-10T08:40:11.000699+0000","last_became_peered":"2024-02-10T08:40:11.000699+0000","last_unstale":"2024-02-10T08:40:11.001016+0000","last_undegraded":"2024-02-10T08:40:11.001016+0000","last_fullsized":"2024-02-10T08:40:11.001016+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"193.10","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.990331+0000","last_change":"2024-02-10T08:40:10.990331+0000","last_active":"2024-02-10T08:40:10.990331+0000","last_peered":"2024-02-10T08:40:10.990331+0000","last_clean":"2024-02-10T08:40:10.990331+0000","last_became_active":"2024-02-10T08:40:10.989985+0000","last_became_peered":"2024-02-10T08:40:10.989985+0000","last_unstale":"2024-02-10T08:40:10.990331+0000","last_undegraded":"2024-02-10T08:40:10.990331+0000","last_fullsized":"2024-02-10T08:40:10.990331+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.f","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:11.001494+0000","last_change":"2024-02-10T08:40:11.001494+0000","last_active":"2024-02-10T08:40:11.001494+0000","last_peered":"2024-02-10T08:40:11.001494+0000","last_clean":"2024-02-10T08:40:11.001494+0000","last_became_active":"2024-02-10T08:40:11.001139+0000","last_became_peered":"2024-02-10T08:40:11.001139+0000","last_unstale":"2024-02-10T08:40:11.001494+0000","last_undegraded":"2024-02-10T08:40:11.001494+0000","last_fullsized":"2024-02-10T08:40:11.001494+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"193.e","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.999940+0000","last_change":"2024-02-10T08:40:10.999940+0000","last_active":"2024-02-10T08:40:10.999940+0000","last_peered":"2024-02-10T08:40:10.999940+0000","last_clean":"2024-02-10T08:40:10.999940+0000","last_became_active":"2024-02-10T08:40:10.999525+0000","last_became_peered":"2024-02-10T08:40:10.999525+0000","last_unstale":"2024-02-10T08:40:10.999940+0000","last_undegraded":"2024-02-10T08:40:10.999940+0000","last_fullsized":"2024-02-10T08:40:10.999940+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.d","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.992819+0000","last_change":"2024-02-10T08:40:10.992819+0000","last_active":"2024-02-10T08:40:10.992819+0000","last_peered":"2024-02-10T08:40:10.992819+0000","last_clean":"2024-02-10T08:40:10.992819+0000","last_became_active":"2024-02-10T08:40:10.992506+0000","last_became_peered":"2024-02-10T08:40:10.992506+0000","last_unstale":"2024-02-10T08:40:10.992819+0000","last_undegraded":"2024-02-10T08:40:10.992819+0000","last_fullsized":"2024-02-10T08:40:10.992819+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.c","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.990543+0000","last_change":"2024-02-10T08:40:10.990543+0000","last_active":"2024-02-10T08:40:10.990543+0000","last_peered":"2024-02-10T08:40:10.990543+0000","last_clean":"2024-02-10T08:40:10.990543+0000","last_became_active":"2024-02-10T08:40:10.990234+0000","last_became_peered":"2024-02-10T08:40:10.990234+0000","last_unstale":"2024-02-10T08:40:10.990543+0000","last_undegraded":"2024-02-10T08:40:10.990543+0000","last_fullsized":"2024-02-10T08:40:10.990543+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"193.b","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.995527+0000","last_change":"2024-02-10T08:40:10.995527+0000","last_active":"2024-02-10T08:40:10.995527+0000","last_peered":"2024-02-10T08:40:10.995527+0000","last_clean":"2024-02-10T08:40:10.995527+0000","last_became_active":"2024-02-10T08:40:10.995247+0000","last_became_peered":"2024-02-10T08:40:10.995247+0000","last_unstale":"2024-02-10T08:40:10.995527+0000","last_undegraded":"2024-02-10T08:40:10.995527+0000","last_fullsized":"2024-02-10T08:40:10.995527+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"193.a","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.994448+0000","last_change":"2024-02-10T08:40:10.994448+0000","last_active":"2024-02-10T08:40:10.994448+0000","last_peered":"2024-02-10T08:40:10.994448+0000","last_clean":"2024-02-10T08:40:10.994448+0000","last_became_active":"2024-02-10T08:40:10.994128+0000","last_became_peered":"2024-02-10T08:40:10.994128+0000","last_unstale":"2024-02-10T08:40:10.994448+0000","last_undegraded":"2024-02-10T08:40:10.994448+0000","last_fullsized":"2024-02-10T08:40:10.994448+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"193.9","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.994252+0000","last_change":"2024-02-10T08:40:10.994252+0000","last_active":"2024-02-10T08:40:10.994252+0000","last_peered":"2024-02-10T08:40:10.994252+0000","last_clean":"2024-02-10T08:40:10.994252+0000","last_became_active":"2024-02-10T08:40:10.994011+0000","last_became_peered":"2024-02-10T08:40:10.994011+0000","last_unstale":"2024-02-10T08:40:10.994252+0000","last_undegraded":"2024-02-10T08:40:10.994252+0000","last_fullsized":"2024-02-10T08:40:10.994252+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"193.8","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:11.000209+0000","last_change":"2024-02-10T08:40:11.000209+0000","last_active":"2024-02-10T08:40:11.000209+0000","last_peered":"2024-02-10T08:40:11.000209+0000","last_clean":"2024-02-10T08:40:11.000209+0000","last_became_active":"2024-02-10T08:40:10.999922+0000","last_became_peered":"2024-02-10T08:40:10.999922+0000","last_unstale":"2024-02-10T08:40:11.000209+0000","last_undegraded":"2024-02-10T08:40:11.000209+0000","last_fullsized":"2024-02-10T08:40:11.000209+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.6","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.995898+0000","last_change":"2024-02-10T08:40:10.995898+0000","last_active":"2024-02-10T08:40:10.995898+0000","last_peered":"2024-02-10T08:40:10.995898+0000","last_clean":"2024-02-10T08:40:10.995898+0000","last_became_active":"2024-02-10T08:40:10.995565+0000","last_became_peered":"2024-02-10T08:40:10.995565+0000","last_unstale":"2024-02-10T08:40:10.995898+0000","last_undegraded":"2024-02-10T08:40:10.995898+0000","last_fullsized":"2024-02-10T08:40:10.995898+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.7","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.995591+0000","last_change":"2024-02-10T08:40:10.995591+0000","last_active":"2024-02-10T08:40:10.995591+0000","last_peered":"2024-02-10T08:40:10.995591+0000","last_clean":"2024-02-10T08:40:10.995591+0000","last_became_active":"2024-02-10T08:40:10.995271+0000","last_became_peered":"2024-02-10T08:40:10.995271+0000","last_unstale":"2024-02-10T08:40:10.995591+0000","last_undegraded":"2024-02-10T08:40:10.995591+0000","last_fullsized":"2024-02-10T08:40:10.995591+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"193.4","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.990623+0000","last_change":"2024-02-10T08:40:10.990623+0000","last_active":"2024-02-10T08:40:10.990623+0000","last_peered":"2024-02-10T08:40:10.990623+0000","last_clean":"2024-02-10T08:40:10.990623+0000","last_became_active":"2024-02-10T08:40:10.990243+0000","last_became_peered":"2024-02-10T08:40:10.990243+0000","last_unstale":"2024-02-10T08:40:10.990623+0000","last_undegraded":"2024-02-10T08:40:10.990623+0000","last_fullsized":"2024-02-10T08:40:10.990623+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"193.5","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.990425+0000","last_change":"2024-02-10T08:40:10.990425+0000","last_active":"2024-02-10T08:40:10.990425+0000","last_peered":"2024-02-10T08:40:10.990425+0000","last_clean":"2024-02-10T08:40:10.990425+0000","last_became_active":"2024-02-10T08:40:10.990152+0000","last_became_peered":"2024-02-10T08:40:10.990152+0000","last_unstale":"2024-02-10T08:40:10.990425+0000","last_undegraded":"2024-02-10T08:40:10.990425+0000","last_fullsized":"2024-02-10T08:40:10.990425+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"193.2","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.990809+0000","last_change":"2024-02-10T08:40:10.990809+0000","last_active":"2024-02-10T08:40:10.990809+0000","last_peered":"2024-02-10T08:40:10.990809+0000","last_clean":"2024-02-10T08:40:10.990809+0000","last_became_active":"2024-02-10T08:40:10.990499+0000","last_became_peered":"2024-02-10T08:40:10.990499+0000","last_unstale":"2024-02-10T08:40:10.990809+0000","last_undegraded":"2024-02-10T08:40:10.990809+0000","last_fullsized":"2024-02-10T08:40:10.990809+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"193.3","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.991145+0000","last_change":"2024-02-10T08:40:10.991145+0000","last_active":"2024-02-10T08:40:10.991145+0000","last_peered":"2024-02-10T08:40:10.991145+0000","last_clean":"2024-02-10T08:40:10.991145+0000","last_became_active":"2024-02-10T08:40:10.990868+0000","last_became_peered":"2024-02-10T08:40:10.990868+0000","last_unstale":"2024-02-10T08:40:10.991145+0000","last_undegraded":"2024-02-10T08:40:10.991145+0000","last_fullsized":"2024-02-10T08:40:10.991145+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"193.0","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.996672+0000","last_change":"2024-02-10T08:40:10.996672+0000","last_active":"2024-02-10T08:40:10.996672+0000","last_peered":"2024-02-10T08:40:10.996672+0000","last_clean":"2024-02-10T08:40:10.996672+0000","last_became_active":"2024-02-10T08:40:10.996367+0000","last_became_peered":"2024-02-10T08:40:10.996367+0000","last_unstale":"2024-02-10T08:40:10.996672+0000","last_undegraded":"2024-02-10T08:40:10.996672+0000","last_fullsized":"2024-02-10T08:40:10.996672+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"193.1","version":"0'0","reported_seq":9,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.991245+0000","last_change":"2024-02-10T08:40:10.991245+0000","last_active":"2024-02-10T08:40:10.991245+0000","last_peered":"2024-02-10T08:40:10.991245+0000","last_clean":"2024-02-10T08:40:10.991245+0000","last_became_active":"2024-02-10T08:40:10.990354+0000","last_became_peered":"2024-02-10T08:40:10.990354+0000","last_unstale":"2024-02-10T08:40:10.991245+0000","last_undegraded":"2024-02-10T08:40:10.991245+0000","last_fullsized":"2024-02-10T08:40:10.991245+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":316,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:10.015876+0000","last_clean_scrub_stamp":"2024-02-10T08:40:10.015876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021747+0000","last_change":"2024-02-10T08:39:32.971969+0000","last_active":"2024-02-10T08:40:10.021747+0000","last_peered":"2024-02-10T08:40:10.021747+0000","last_clean":"2024-02-10T08:40:10.021747+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:40:10.021747+0000","last_undegraded":"2024-02-10T08:40:10.021747+0000","last_fullsized":"2024-02-10T08:40:10.021747+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"236'27","reported_seq":288,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021894+0000","last_change":"2024-02-10T08:39:38.573741+0000","last_active":"2024-02-10T08:40:10.021894+0000","last_peered":"2024-02-10T08:40:10.021894+0000","last_clean":"2024-02-10T08:40:10.021894+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:40:10.021894+0000","last_undegraded":"2024-02-10T08:40:10.021894+0000","last_fullsized":"2024-02-10T08:40:10.021894+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"250'39","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021128+0000","last_change":"2024-02-10T08:39:40.483224+0000","last_active":"2024-02-10T08:40:10.021128+0000","last_peered":"2024-02-10T08:40:10.021128+0000","last_clean":"2024-02-10T08:40:10.021128+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:40:10.021128+0000","last_undegraded":"2024-02-10T08:40:10.021128+0000","last_fullsized":"2024-02-10T08:40:10.021128+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"231'107","reported_seq":356,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026285+0000","last_change":"2024-02-10T08:39:32.974249+0000","last_active":"2024-02-10T08:40:10.026285+0000","last_peered":"2024-02-10T08:40:10.026285+0000","last_clean":"2024-02-10T08:40:10.026285+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:40:10.026285+0000","last_undegraded":"2024-02-10T08:40:10.026285+0000","last_fullsized":"2024-02-10T08:40:10.026285+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"222'34","reported_seq":265,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022705+0000","last_change":"2024-02-10T08:39:53.632526+0000","last_active":"2024-02-10T08:40:10.022705+0000","last_peered":"2024-02-10T08:40:10.022705+0000","last_clean":"2024-02-10T08:40:10.022705+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:40:10.022705+0000","last_undegraded":"2024-02-10T08:40:10.022705+0000","last_fullsized":"2024-02-10T08:40:10.022705+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"226'71","reported_seq":319,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029507+0000","last_change":"2024-02-10T08:39:49.450775+0000","last_active":"2024-02-10T08:40:10.029507+0000","last_peered":"2024-02-10T08:40:10.029507+0000","last_clean":"2024-02-10T08:40:10.029507+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:40:10.029507+0000","last_undegraded":"2024-02-10T08:40:10.029507+0000","last_fullsized":"2024-02-10T08:40:10.029507+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022328+0000","last_change":"2024-02-10T08:39:52.583051+0000","last_active":"2024-02-10T08:40:10.022328+0000","last_peered":"2024-02-10T08:40:10.022328+0000","last_clean":"2024-02-10T08:40:10.022328+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:40:10.022328+0000","last_undegraded":"2024-02-10T08:40:10.022328+0000","last_fullsized":"2024-02-10T08:40:10.022328+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"250'22","reported_seq":281,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022175+0000","last_change":"2024-02-10T08:39:42.510684+0000","last_active":"2024-02-10T08:40:10.022175+0000","last_peered":"2024-02-10T08:40:10.022175+0000","last_clean":"2024-02-10T08:40:10.022175+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:40:10.022175+0000","last_undegraded":"2024-02-10T08:40:10.022175+0000","last_fullsized":"2024-02-10T08:40:10.022175+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"250'101","reported_seq":342,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027264+0000","last_change":"2024-02-10T08:39:32.974027+0000","last_active":"2024-02-10T08:40:10.027264+0000","last_peered":"2024-02-10T08:40:10.027264+0000","last_clean":"2024-02-10T08:40:10.027264+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:40:10.027264+0000","last_undegraded":"2024-02-10T08:40:10.027264+0000","last_fullsized":"2024-02-10T08:40:10.027264+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"228'22","reported_seq":280,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021881+0000","last_change":"2024-02-10T08:39:47.635717+0000","last_active":"2024-02-10T08:40:10.021881+0000","last_peered":"2024-02-10T08:40:10.021881+0000","last_clean":"2024-02-10T08:40:10.021881+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:40:10.021881+0000","last_undegraded":"2024-02-10T08:40:10.021881+0000","last_fullsized":"2024-02-10T08:40:10.021881+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"299'110","reported_seq":341,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021378+0000","last_change":"2024-02-10T08:39:53.048253+0000","last_active":"2024-02-10T08:40:10.021378+0000","last_peered":"2024-02-10T08:40:10.021378+0000","last_clean":"2024-02-10T08:40:10.021378+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:40:10.021378+0000","last_undegraded":"2024-02-10T08:40:10.021378+0000","last_fullsized":"2024-02-10T08:40:10.021378+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"236'85","reported_seq":335,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021357+0000","last_change":"2024-02-10T08:39:50.630114+0000","last_active":"2024-02-10T08:40:10.021357+0000","last_peered":"2024-02-10T08:40:10.021357+0000","last_clean":"2024-02-10T08:40:10.021357+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:40:10.021357+0000","last_undegraded":"2024-02-10T08:40:10.021357+0000","last_fullsized":"2024-02-10T08:40:10.021357+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"231'93","reported_seq":333,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034232+0000","last_change":"2024-02-10T08:39:43.510896+0000","last_active":"2024-02-10T08:40:10.034232+0000","last_peered":"2024-02-10T08:40:10.034232+0000","last_clean":"2024-02-10T08:40:10.034232+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:40:10.034232+0000","last_undegraded":"2024-02-10T08:40:10.034232+0000","last_fullsized":"2024-02-10T08:40:10.034232+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'93","last_scrub_stamp":"2024-02-10T08:39:43.510789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.510789+0000","log_size":93,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":72,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030762+0000","last_change":"2024-02-10T08:39:54.894186+0000","last_active":"2024-02-10T08:40:10.030762+0000","last_peered":"2024-02-10T08:40:10.030762+0000","last_clean":"2024-02-10T08:40:10.030762+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:40:10.030762+0000","last_undegraded":"2024-02-10T08:40:10.030762+0000","last_fullsized":"2024-02-10T08:40:10.030762+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"287'56","reported_seq":301,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033181+0000","last_change":"2024-02-10T08:39:44.469992+0000","last_active":"2024-02-10T08:40:10.033181+0000","last_peered":"2024-02-10T08:40:10.033181+0000","last_clean":"2024-02-10T08:40:10.033181+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:40:10.033181+0000","last_undegraded":"2024-02-10T08:40:10.033181+0000","last_fullsized":"2024-02-10T08:40:10.033181+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-02-10T08:39:44.469923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.469923+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"226'68","reported_seq":313,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028004+0000","last_change":"2024-02-10T08:39:32.974869+0000","last_active":"2024-02-10T08:40:10.028004+0000","last_peered":"2024-02-10T08:40:10.028004+0000","last_clean":"2024-02-10T08:40:10.028004+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:40:10.028004+0000","last_undegraded":"2024-02-10T08:40:10.028004+0000","last_fullsized":"2024-02-10T08:40:10.028004+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"295'55","reported_seq":306,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029568+0000","last_change":"2024-02-10T08:39:33.014018+0000","last_active":"2024-02-10T08:40:10.029568+0000","last_peered":"2024-02-10T08:40:10.029568+0000","last_clean":"2024-02-10T08:40:10.029568+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:40:10.029568+0000","last_undegraded":"2024-02-10T08:40:10.029568+0000","last_fullsized":"2024-02-10T08:40:10.029568+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'54","last_scrub_stamp":"2024-02-10T08:39:12.673905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.673905+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000054,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":3924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":353,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025580+0000","last_change":"2024-02-10T08:39:46.646522+0000","last_active":"2024-02-10T08:40:10.025580+0000","last_peered":"2024-02-10T08:40:10.025580+0000","last_clean":"2024-02-10T08:40:10.025580+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:40:10.025580+0000","last_undegraded":"2024-02-10T08:40:10.025580+0000","last_fullsized":"2024-02-10T08:40:10.025580+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:39:46.646454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.646454+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"258'103","reported_seq":342,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027434+0000","last_change":"2024-02-10T08:39:41.578640+0000","last_active":"2024-02-10T08:40:10.027434+0000","last_peered":"2024-02-10T08:40:10.027434+0000","last_clean":"2024-02-10T08:40:10.027434+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:40:10.027434+0000","last_undegraded":"2024-02-10T08:40:10.027434+0000","last_fullsized":"2024-02-10T08:40:10.027434+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":328,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025029+0000","last_change":"2024-02-10T08:39:55.527916+0000","last_active":"2024-02-10T08:40:10.025029+0000","last_peered":"2024-02-10T08:40:10.025029+0000","last_clean":"2024-02-10T08:40:10.025029+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:40:10.025029+0000","last_undegraded":"2024-02-10T08:40:10.025029+0000","last_fullsized":"2024-02-10T08:40:10.025029+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:39:55.527854+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"215'85","reported_seq":330,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030615+0000","last_change":"2024-02-10T08:39:54.516959+0000","last_active":"2024-02-10T08:40:10.030615+0000","last_peered":"2024-02-10T08:40:10.030615+0000","last_clean":"2024-02-10T08:40:10.030615+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:40:10.030615+0000","last_undegraded":"2024-02-10T08:40:10.030615+0000","last_fullsized":"2024-02-10T08:40:10.030615+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'85","last_scrub_stamp":"2024-02-10T08:39:54.516887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.516887+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":332,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030671+0000","last_change":"2024-02-10T08:39:43.711787+0000","last_active":"2024-02-10T08:40:10.030671+0000","last_peered":"2024-02-10T08:40:10.030671+0000","last_clean":"2024-02-10T08:40:10.030671+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:40:10.030671+0000","last_undegraded":"2024-02-10T08:40:10.030671+0000","last_fullsized":"2024-02-10T08:40:10.030671+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"230'17","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022517+0000","last_change":"2024-02-10T08:39:56.545155+0000","last_active":"2024-02-10T08:40:10.022517+0000","last_peered":"2024-02-10T08:40:10.022517+0000","last_clean":"2024-02-10T08:40:10.022517+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:40:10.022517+0000","last_undegraded":"2024-02-10T08:40:10.022517+0000","last_fullsized":"2024-02-10T08:40:10.022517+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":285,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029579+0000","last_change":"2024-02-10T08:39:49.714905+0000","last_active":"2024-02-10T08:40:10.029579+0000","last_peered":"2024-02-10T08:40:10.029579+0000","last_clean":"2024-02-10T08:40:10.029579+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:40:10.029579+0000","last_undegraded":"2024-02-10T08:40:10.029579+0000","last_fullsized":"2024-02-10T08:40:10.029579+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"228'68","reported_seq":294,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021988+0000","last_change":"2024-02-10T08:39:32.968736+0000","last_active":"2024-02-10T08:40:10.021988+0000","last_peered":"2024-02-10T08:40:10.021988+0000","last_clean":"2024-02-10T08:40:10.021988+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:40:10.021988+0000","last_undegraded":"2024-02-10T08:40:10.021988+0000","last_fullsized":"2024-02-10T08:40:10.021988+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"214'68","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021627+0000","last_change":"2024-02-10T08:39:32.970019+0000","last_active":"2024-02-10T08:40:10.021627+0000","last_peered":"2024-02-10T08:40:10.021627+0000","last_clean":"2024-02-10T08:40:10.021627+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:40:10.021627+0000","last_undegraded":"2024-02-10T08:40:10.021627+0000","last_fullsized":"2024-02-10T08:40:10.021627+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"208'34","reported_seq":291,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021490+0000","last_change":"2024-02-10T08:39:32.971984+0000","last_active":"2024-02-10T08:40:10.021490+0000","last_peered":"2024-02-10T08:40:10.021490+0000","last_clean":"2024-02-10T08:40:10.021490+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:40:10.021490+0000","last_undegraded":"2024-02-10T08:40:10.021490+0000","last_fullsized":"2024-02-10T08:40:10.021490+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"224'34","reported_seq":268,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021260+0000","last_change":"2024-02-10T08:39:54.602656+0000","last_active":"2024-02-10T08:40:10.021260+0000","last_peered":"2024-02-10T08:40:10.021260+0000","last_clean":"2024-02-10T08:40:10.021260+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:40:10.021260+0000","last_undegraded":"2024-02-10T08:40:10.021260+0000","last_fullsized":"2024-02-10T08:40:10.021260+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"200'34","reported_seq":296,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032799+0000","last_change":"2024-02-10T08:39:33.014676+0000","last_active":"2024-02-10T08:40:10.032799+0000","last_peered":"2024-02-10T08:40:10.032799+0000","last_clean":"2024-02-10T08:40:10.032799+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:40:10.032799+0000","last_undegraded":"2024-02-10T08:40:10.032799+0000","last_fullsized":"2024-02-10T08:40:10.032799+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'34","last_scrub_stamp":"2024-02-10T08:39:04.785239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.785239+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":254,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022563+0000","last_change":"2024-02-10T08:39:38.041298+0000","last_active":"2024-02-10T08:40:10.022563+0000","last_peered":"2024-02-10T08:40:10.022563+0000","last_clean":"2024-02-10T08:40:10.022563+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:40:10.022563+0000","last_undegraded":"2024-02-10T08:40:10.022563+0000","last_fullsized":"2024-02-10T08:40:10.022563+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:39:38.041185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.041185+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"202'17","reported_seq":252,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026906+0000","last_change":"2024-02-10T08:39:33.016451+0000","last_active":"2024-02-10T08:40:10.026906+0000","last_peered":"2024-02-10T08:40:10.026906+0000","last_clean":"2024-02-10T08:40:10.026906+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:40:10.026906+0000","last_undegraded":"2024-02-10T08:40:10.026906+0000","last_fullsized":"2024-02-10T08:40:10.026906+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"222'34","reported_seq":283,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027730+0000","last_change":"2024-02-10T08:39:32.974056+0000","last_active":"2024-02-10T08:40:10.027730+0000","last_peered":"2024-02-10T08:40:10.027730+0000","last_clean":"2024-02-10T08:40:10.027730+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:40:10.027730+0000","last_undegraded":"2024-02-10T08:40:10.027730+0000","last_fullsized":"2024-02-10T08:40:10.027730+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":390,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026923+0000","last_change":"2024-02-10T08:38:21.488646+0000","last_active":"2024-02-10T08:40:10.026923+0000","last_peered":"2024-02-10T08:40:10.026923+0000","last_clean":"2024-02-10T08:40:10.026923+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:40:10.026923+0000","last_undegraded":"2024-02-10T08:40:10.026923+0000","last_fullsized":"2024-02-10T08:40:10.026923+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:38:21.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:21.488563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":314,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028043+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:40:10.028043+0000","last_peered":"2024-02-10T08:40:10.028043+0000","last_clean":"2024-02-10T08:40:10.028043+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:40:10.028043+0000","last_undegraded":"2024-02-10T08:40:10.028043+0000","last_fullsized":"2024-02-10T08:40:10.028043+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":280,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029677+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:40:10.029677+0000","last_peered":"2024-02-10T08:40:10.029677+0000","last_clean":"2024-02-10T08:40:10.029677+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:40:10.029677+0000","last_undegraded":"2024-02-10T08:40:10.029677+0000","last_fullsized":"2024-02-10T08:40:10.029677+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027756+0000","last_change":"2024-02-10T08:37:44.597397+0000","last_active":"2024-02-10T08:40:10.027756+0000","last_peered":"2024-02-10T08:40:10.027756+0000","last_clean":"2024-02-10T08:40:10.027756+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:40:10.027756+0000","last_undegraded":"2024-02-10T08:40:10.027756+0000","last_fullsized":"2024-02-10T08:40:10.027756+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.597335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.597335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028122+0000","last_change":"2024-02-10T08:38:33.920702+0000","last_active":"2024-02-10T08:40:10.028122+0000","last_peered":"2024-02-10T08:40:10.028122+0000","last_clean":"2024-02-10T08:40:10.028122+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:40:10.028122+0000","last_undegraded":"2024-02-10T08:40:10.028122+0000","last_fullsized":"2024-02-10T08:40:10.028122+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.920630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.920630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":257,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027533+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:40:10.027533+0000","last_peered":"2024-02-10T08:40:10.027533+0000","last_clean":"2024-02-10T08:40:10.027533+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:40:10.027533+0000","last_undegraded":"2024-02-10T08:40:10.027533+0000","last_fullsized":"2024-02-10T08:40:10.027533+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":285,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022520+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:40:10.022520+0000","last_peered":"2024-02-10T08:40:10.022520+0000","last_clean":"2024-02-10T08:40:10.022520+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:40:10.022520+0000","last_undegraded":"2024-02-10T08:40:10.022520+0000","last_fullsized":"2024-02-10T08:40:10.022520+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":299,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026958+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:40:10.026958+0000","last_peered":"2024-02-10T08:40:10.026958+0000","last_clean":"2024-02-10T08:40:10.026958+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:40:10.026958+0000","last_undegraded":"2024-02-10T08:40:10.026958+0000","last_fullsized":"2024-02-10T08:40:10.026958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026360+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:40:10.026360+0000","last_peered":"2024-02-10T08:40:10.026360+0000","last_clean":"2024-02-10T08:40:10.026360+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:40:10.026360+0000","last_undegraded":"2024-02-10T08:40:10.026360+0000","last_fullsized":"2024-02-10T08:40:10.026360+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":227,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033604+0000","last_change":"2024-02-10T08:38:26.677780+0000","last_active":"2024-02-10T08:40:10.033604+0000","last_peered":"2024-02-10T08:40:10.033604+0000","last_clean":"2024-02-10T08:40:10.033604+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:40:10.033604+0000","last_undegraded":"2024-02-10T08:40:10.033604+0000","last_fullsized":"2024-02-10T08:40:10.033604+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.677701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.677701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":313,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032738+0000","last_change":"2024-02-10T08:37:39.547805+0000","last_active":"2024-02-10T08:40:10.032738+0000","last_peered":"2024-02-10T08:40:10.032738+0000","last_clean":"2024-02-10T08:40:10.032738+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:40:10.032738+0000","last_undegraded":"2024-02-10T08:40:10.032738+0000","last_fullsized":"2024-02-10T08:40:10.032738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:37:39.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.547736+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028413+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:40:10.028413+0000","last_peered":"2024-02-10T08:40:10.028413+0000","last_clean":"2024-02-10T08:40:10.028413+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:40:10.028413+0000","last_undegraded":"2024-02-10T08:40:10.028413+0000","last_fullsized":"2024-02-10T08:40:10.028413+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":303,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027395+0000","last_change":"2024-02-10T08:37:10.186145+0000","last_active":"2024-02-10T08:40:10.027395+0000","last_peered":"2024-02-10T08:40:10.027395+0000","last_clean":"2024-02-10T08:40:10.027395+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:40:10.027395+0000","last_undegraded":"2024-02-10T08:40:10.027395+0000","last_fullsized":"2024-02-10T08:40:10.027395+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.766815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025278+0000","last_change":"2024-02-10T08:37:16.274040+0000","last_active":"2024-02-10T08:40:10.025278+0000","last_peered":"2024-02-10T08:40:10.025278+0000","last_clean":"2024-02-10T08:40:10.025278+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:40:10.025278+0000","last_undegraded":"2024-02-10T08:40:10.025278+0000","last_fullsized":"2024-02-10T08:40:10.025278+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.273968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.273968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":227,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029802+0000","last_change":"2024-02-10T08:39:44.585775+0000","last_active":"2024-02-10T08:40:10.029802+0000","last_peered":"2024-02-10T08:40:10.029802+0000","last_clean":"2024-02-10T08:40:10.029802+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:40:10.029802+0000","last_undegraded":"2024-02-10T08:40:10.029802+0000","last_fullsized":"2024-02-10T08:40:10.029802+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:44.585674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.585674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":311,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022991+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:40:10.022991+0000","last_peered":"2024-02-10T08:40:10.022991+0000","last_clean":"2024-02-10T08:40:10.022991+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:40:10.022991+0000","last_undegraded":"2024-02-10T08:40:10.022991+0000","last_fullsized":"2024-02-10T08:40:10.022991+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":119,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027487+0000","last_change":"2024-02-10T08:38:22.013062+0000","last_active":"2024-02-10T08:40:10.027487+0000","last_peered":"2024-02-10T08:40:10.027487+0000","last_clean":"2024-02-10T08:40:10.027487+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:40:10.027487+0000","last_undegraded":"2024-02-10T08:40:10.027487+0000","last_fullsized":"2024-02-10T08:40:10.027487+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":303,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022525+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:40:10.022525+0000","last_peered":"2024-02-10T08:40:10.022525+0000","last_clean":"2024-02-10T08:40:10.022525+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:40:10.022525+0000","last_undegraded":"2024-02-10T08:40:10.022525+0000","last_fullsized":"2024-02-10T08:40:10.022525+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029081+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:40:10.029081+0000","last_peered":"2024-02-10T08:40:10.029081+0000","last_clean":"2024-02-10T08:40:10.029081+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:40:10.029081+0000","last_undegraded":"2024-02-10T08:40:10.029081+0000","last_fullsized":"2024-02-10T08:40:10.029081+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021190+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:40:10.021190+0000","last_peered":"2024-02-10T08:40:10.021190+0000","last_clean":"2024-02-10T08:40:10.021190+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:40:10.021190+0000","last_undegraded":"2024-02-10T08:40:10.021190+0000","last_fullsized":"2024-02-10T08:40:10.021190+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":298,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022320+0000","last_change":"2024-02-10T08:38:17.727465+0000","last_active":"2024-02-10T08:40:10.022320+0000","last_peered":"2024-02-10T08:40:10.022320+0000","last_clean":"2024-02-10T08:40:10.022320+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:40:10.022320+0000","last_undegraded":"2024-02-10T08:40:10.022320+0000","last_fullsized":"2024-02-10T08:40:10.022320+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:17.727387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:17.727387+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":285,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022105+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:40:10.022105+0000","last_peered":"2024-02-10T08:40:10.022105+0000","last_clean":"2024-02-10T08:40:10.022105+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:40:10.022105+0000","last_undegraded":"2024-02-10T08:40:10.022105+0000","last_fullsized":"2024-02-10T08:40:10.022105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":386,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028563+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:40:10.028563+0000","last_peered":"2024-02-10T08:40:10.028563+0000","last_clean":"2024-02-10T08:40:10.028563+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:40:10.028563+0000","last_undegraded":"2024-02-10T08:40:10.028563+0000","last_fullsized":"2024-02-10T08:40:10.028563+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022832+0000","last_change":"2024-02-10T08:38:25.851680+0000","last_active":"2024-02-10T08:40:10.022832+0000","last_peered":"2024-02-10T08:40:10.022832+0000","last_clean":"2024-02-10T08:40:10.022832+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:40:10.022832+0000","last_undegraded":"2024-02-10T08:40:10.022832+0000","last_fullsized":"2024-02-10T08:40:10.022832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.851576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.851576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025617+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:40:10.025617+0000","last_peered":"2024-02-10T08:40:10.025617+0000","last_clean":"2024-02-10T08:40:10.025617+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:40:10.025617+0000","last_undegraded":"2024-02-10T08:40:10.025617+0000","last_fullsized":"2024-02-10T08:40:10.025617+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":303,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023298+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:40:10.023298+0000","last_peered":"2024-02-10T08:40:10.023298+0000","last_clean":"2024-02-10T08:40:10.023298+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:40:10.023298+0000","last_undegraded":"2024-02-10T08:40:10.023298+0000","last_fullsized":"2024-02-10T08:40:10.023298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021785+0000","last_change":"2024-02-10T08:38:18.008913+0000","last_active":"2024-02-10T08:40:10.021785+0000","last_peered":"2024-02-10T08:40:10.021785+0000","last_clean":"2024-02-10T08:40:10.021785+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:40:10.021785+0000","last_undegraded":"2024-02-10T08:40:10.021785+0000","last_fullsized":"2024-02-10T08:40:10.021785+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:18.008846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.008846+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":411,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022786+0000","last_change":"2024-02-10T08:37:14.574872+0000","last_active":"2024-02-10T08:40:10.022786+0000","last_peered":"2024-02-10T08:40:10.022786+0000","last_clean":"2024-02-10T08:40:10.022786+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:40:10.022786+0000","last_undegraded":"2024-02-10T08:40:10.022786+0000","last_fullsized":"2024-02-10T08:40:10.022786+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'12","last_scrub_stamp":"2024-02-10T08:37:14.574796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:14.574796+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028783+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:40:10.028783+0000","last_peered":"2024-02-10T08:40:10.028783+0000","last_clean":"2024-02-10T08:40:10.028783+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:40:10.028783+0000","last_undegraded":"2024-02-10T08:40:10.028783+0000","last_fullsized":"2024-02-10T08:40:10.028783+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":227,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029730+0000","last_change":"2024-02-10T08:39:05.788196+0000","last_active":"2024-02-10T08:40:10.029730+0000","last_peered":"2024-02-10T08:40:10.029730+0000","last_clean":"2024-02-10T08:40:10.029730+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:40:10.029730+0000","last_undegraded":"2024-02-10T08:40:10.029730+0000","last_fullsized":"2024-02-10T08:40:10.029730+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.788118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.788118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022664+0000","last_change":"2024-02-10T08:38:59.745198+0000","last_active":"2024-02-10T08:40:10.022664+0000","last_peered":"2024-02-10T08:40:10.022664+0000","last_clean":"2024-02-10T08:40:10.022664+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:40:10.022664+0000","last_undegraded":"2024-02-10T08:40:10.022664+0000","last_fullsized":"2024-02-10T08:40:10.022664+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:38:59.745114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.745114+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025687+0000","last_change":"2024-02-10T08:38:33.363322+0000","last_active":"2024-02-10T08:40:10.025687+0000","last_peered":"2024-02-10T08:40:10.025687+0000","last_clean":"2024-02-10T08:40:10.025687+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:40:10.025687+0000","last_undegraded":"2024-02-10T08:40:10.025687+0000","last_fullsized":"2024-02-10T08:40:10.025687+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.363195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":343,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033328+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:40:10.033328+0000","last_peered":"2024-02-10T08:40:10.033328+0000","last_clean":"2024-02-10T08:40:10.033328+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:40:10.033328+0000","last_undegraded":"2024-02-10T08:40:10.033328+0000","last_fullsized":"2024-02-10T08:40:10.033328+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029890+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:40:10.029890+0000","last_peered":"2024-02-10T08:40:10.029890+0000","last_clean":"2024-02-10T08:40:10.029890+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:40:10.029890+0000","last_undegraded":"2024-02-10T08:40:10.029890+0000","last_fullsized":"2024-02-10T08:40:10.029890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026280+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:40:10.026280+0000","last_peered":"2024-02-10T08:40:10.026280+0000","last_clean":"2024-02-10T08:40:10.026280+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:40:10.026280+0000","last_undegraded":"2024-02-10T08:40:10.026280+0000","last_fullsized":"2024-02-10T08:40:10.026280+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":373,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025767+0000","last_change":"2024-02-10T08:38:18.475575+0000","last_active":"2024-02-10T08:40:10.025767+0000","last_peered":"2024-02-10T08:40:10.025767+0000","last_clean":"2024-02-10T08:40:10.025767+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:40:10.025767+0000","last_undegraded":"2024-02-10T08:40:10.025767+0000","last_fullsized":"2024-02-10T08:40:10.025767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:38:18.475509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.475509+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021416+0000","last_change":"2024-02-10T08:38:31.840285+0000","last_active":"2024-02-10T08:40:10.021416+0000","last_peered":"2024-02-10T08:40:10.021416+0000","last_clean":"2024-02-10T08:40:10.021416+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:40:10.021416+0000","last_undegraded":"2024-02-10T08:40:10.021416+0000","last_fullsized":"2024-02-10T08:40:10.021416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:38:31.840218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.840218+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034352+0000","last_change":"2024-02-10T08:39:48.521060+0000","last_active":"2024-02-10T08:40:10.034352+0000","last_peered":"2024-02-10T08:40:10.034352+0000","last_clean":"2024-02-10T08:40:10.034352+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:40:10.034352+0000","last_undegraded":"2024-02-10T08:40:10.034352+0000","last_fullsized":"2024-02-10T08:40:10.034352+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:48.520941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:48.520941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030438+0000","last_change":"2024-02-10T08:39:56.568250+0000","last_active":"2024-02-10T08:40:10.030438+0000","last_peered":"2024-02-10T08:40:10.030438+0000","last_clean":"2024-02-10T08:40:10.030438+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:40:10.030438+0000","last_undegraded":"2024-02-10T08:40:10.030438+0000","last_fullsized":"2024-02-10T08:40:10.030438+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:56.568189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.568189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022659+0000","last_change":"2024-02-10T08:39:12.085361+0000","last_active":"2024-02-10T08:40:10.022659+0000","last_peered":"2024-02-10T08:40:10.022659+0000","last_clean":"2024-02-10T08:40:10.022659+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:40:10.022659+0000","last_undegraded":"2024-02-10T08:40:10.022659+0000","last_fullsized":"2024-02-10T08:40:10.022659+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.085261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.085261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":276,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021509+0000","last_change":"2024-02-10T08:37:18.565362+0000","last_active":"2024-02-10T08:40:10.021509+0000","last_peered":"2024-02-10T08:40:10.021509+0000","last_clean":"2024-02-10T08:40:10.021509+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:40:10.021509+0000","last_undegraded":"2024-02-10T08:40:10.021509+0000","last_fullsized":"2024-02-10T08:40:10.021509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:18.565300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:18.565300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":344,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025759+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:40:10.025759+0000","last_peered":"2024-02-10T08:40:10.025759+0000","last_clean":"2024-02-10T08:40:10.025759+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:40:10.025759+0000","last_undegraded":"2024-02-10T08:40:10.025759+0000","last_fullsized":"2024-02-10T08:40:10.025759+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":292,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022696+0000","last_change":"2024-02-10T08:38:57.920239+0000","last_active":"2024-02-10T08:40:10.022696+0000","last_peered":"2024-02-10T08:40:10.022696+0000","last_clean":"2024-02-10T08:40:10.022696+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:40:10.022696+0000","last_undegraded":"2024-02-10T08:40:10.022696+0000","last_fullsized":"2024-02-10T08:40:10.022696+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:57.920144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.920144+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022737+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:40:10.022737+0000","last_peered":"2024-02-10T08:40:10.022737+0000","last_clean":"2024-02-10T08:40:10.022737+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:40:10.022737+0000","last_undegraded":"2024-02-10T08:40:10.022737+0000","last_fullsized":"2024-02-10T08:40:10.022737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029729+0000","last_change":"2024-02-10T08:39:08.629231+0000","last_active":"2024-02-10T08:40:10.029729+0000","last_peered":"2024-02-10T08:40:10.029729+0000","last_clean":"2024-02-10T08:40:10.029729+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:40:10.029729+0000","last_undegraded":"2024-02-10T08:40:10.029729+0000","last_fullsized":"2024-02-10T08:40:10.029729+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032801+0000","last_change":"2024-02-10T08:37:43.520384+0000","last_active":"2024-02-10T08:40:10.032801+0000","last_peered":"2024-02-10T08:40:10.032801+0000","last_clean":"2024-02-10T08:40:10.032801+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:40:10.032801+0000","last_undegraded":"2024-02-10T08:40:10.032801+0000","last_fullsized":"2024-02-10T08:40:10.032801+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.520307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.520307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":263,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029986+0000","last_change":"2024-02-10T08:38:18.588026+0000","last_active":"2024-02-10T08:40:10.029986+0000","last_peered":"2024-02-10T08:40:10.029986+0000","last_clean":"2024-02-10T08:40:10.029986+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:40:10.029986+0000","last_undegraded":"2024-02-10T08:40:10.029986+0000","last_fullsized":"2024-02-10T08:40:10.029986+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:38:18.587949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.587949+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.037227+0000","last_change":"2024-02-10T08:38:13.655575+0000","last_active":"2024-02-10T08:40:10.037227+0000","last_peered":"2024-02-10T08:40:10.037227+0000","last_clean":"2024-02-10T08:40:10.037227+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:40:10.037227+0000","last_undegraded":"2024-02-10T08:40:10.037227+0000","last_fullsized":"2024-02-10T08:40:10.037227+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.655504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021930+0000","last_change":"2024-02-10T08:38:26.946622+0000","last_active":"2024-02-10T08:40:10.021930+0000","last_peered":"2024-02-10T08:40:10.021930+0000","last_clean":"2024-02-10T08:40:10.021930+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:40:10.021930+0000","last_undegraded":"2024-02-10T08:40:10.021930+0000","last_fullsized":"2024-02-10T08:40:10.021930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.946531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.946531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":237,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028824+0000","last_change":"2024-02-10T08:38:25.903816+0000","last_active":"2024-02-10T08:40:10.028824+0000","last_peered":"2024-02-10T08:40:10.028824+0000","last_clean":"2024-02-10T08:40:10.028824+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:40:10.028824+0000","last_undegraded":"2024-02-10T08:40:10.028824+0000","last_fullsized":"2024-02-10T08:40:10.028824+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.903753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.903753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":307,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033402+0000","last_change":"2024-02-10T08:38:31.745438+0000","last_active":"2024-02-10T08:40:10.033402+0000","last_peered":"2024-02-10T08:40:10.033402+0000","last_clean":"2024-02-10T08:40:10.033402+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:40:10.033402+0000","last_undegraded":"2024-02-10T08:40:10.033402+0000","last_fullsized":"2024-02-10T08:40:10.033402+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.745362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.745362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":242,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022752+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:40:10.022752+0000","last_peered":"2024-02-10T08:40:10.022752+0000","last_clean":"2024-02-10T08:40:10.022752+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:40:10.022752+0000","last_undegraded":"2024-02-10T08:40:10.022752+0000","last_fullsized":"2024-02-10T08:40:10.022752+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025398+0000","last_change":"2024-02-10T08:37:21.277884+0000","last_active":"2024-02-10T08:40:10.025398+0000","last_peered":"2024-02-10T08:40:10.025398+0000","last_clean":"2024-02-10T08:40:10.025398+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:40:10.025398+0000","last_undegraded":"2024-02-10T08:40:10.025398+0000","last_fullsized":"2024-02-10T08:40:10.025398+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.277786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.277786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021316+0000","last_change":"2024-02-10T08:38:18.758911+0000","last_active":"2024-02-10T08:40:10.021316+0000","last_peered":"2024-02-10T08:40:10.021316+0000","last_clean":"2024-02-10T08:40:10.021316+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:40:10.021316+0000","last_undegraded":"2024-02-10T08:40:10.021316+0000","last_fullsized":"2024-02-10T08:40:10.021316+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.758843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.758843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":280,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022602+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:40:10.022602+0000","last_peered":"2024-02-10T08:40:10.022602+0000","last_clean":"2024-02-10T08:40:10.022602+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:40:10.022602+0000","last_undegraded":"2024-02-10T08:40:10.022602+0000","last_fullsized":"2024-02-10T08:40:10.022602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":236,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032477+0000","last_change":"2024-02-10T08:39:06.770922+0000","last_active":"2024-02-10T08:40:10.032477+0000","last_peered":"2024-02-10T08:40:10.032477+0000","last_clean":"2024-02-10T08:40:10.032477+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:40:10.032477+0000","last_undegraded":"2024-02-10T08:40:10.032477+0000","last_fullsized":"2024-02-10T08:40:10.032477+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.770850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.770850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":280,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033971+0000","last_change":"2024-02-10T08:39:38.645588+0000","last_active":"2024-02-10T08:40:10.033971+0000","last_peered":"2024-02-10T08:40:10.033971+0000","last_clean":"2024-02-10T08:40:10.033971+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:40:10.033971+0000","last_undegraded":"2024-02-10T08:40:10.033971+0000","last_fullsized":"2024-02-10T08:40:10.033971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.645527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.645527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033366+0000","last_change":"2024-02-10T08:37:40.592997+0000","last_active":"2024-02-10T08:40:10.033366+0000","last_peered":"2024-02-10T08:40:10.033366+0000","last_clean":"2024-02-10T08:40:10.033366+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:40:10.033366+0000","last_undegraded":"2024-02-10T08:40:10.033366+0000","last_fullsized":"2024-02-10T08:40:10.033366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:37:40.592930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.592930+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":301,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025518+0000","last_change":"2024-02-10T08:39:36.639528+0000","last_active":"2024-02-10T08:40:10.025518+0000","last_peered":"2024-02-10T08:40:10.025518+0000","last_clean":"2024-02-10T08:40:10.025518+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:40:10.025518+0000","last_undegraded":"2024-02-10T08:40:10.025518+0000","last_fullsized":"2024-02-10T08:40:10.025518+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.639461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.639461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029895+0000","last_change":"2024-02-10T08:38:25.633733+0000","last_active":"2024-02-10T08:40:10.029895+0000","last_peered":"2024-02-10T08:40:10.029895+0000","last_clean":"2024-02-10T08:40:10.029895+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:40:10.029895+0000","last_undegraded":"2024-02-10T08:40:10.029895+0000","last_fullsized":"2024-02-10T08:40:10.029895+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.633608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.633608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026562+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:40:10.026562+0000","last_peered":"2024-02-10T08:40:10.026562+0000","last_clean":"2024-02-10T08:40:10.026562+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:40:10.026562+0000","last_undegraded":"2024-02-10T08:40:10.026562+0000","last_fullsized":"2024-02-10T08:40:10.026562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033509+0000","last_change":"2024-02-10T08:39:00.751654+0000","last_active":"2024-02-10T08:40:10.033509+0000","last_peered":"2024-02-10T08:40:10.033509+0000","last_clean":"2024-02-10T08:40:10.033509+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:40:10.033509+0000","last_undegraded":"2024-02-10T08:40:10.033509+0000","last_fullsized":"2024-02-10T08:40:10.033509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:00.751584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.751584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":238,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027534+0000","last_change":"2024-02-10T08:39:11.660136+0000","last_active":"2024-02-10T08:40:10.027534+0000","last_peered":"2024-02-10T08:40:10.027534+0000","last_clean":"2024-02-10T08:40:10.027534+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:40:10.027534+0000","last_undegraded":"2024-02-10T08:40:10.027534+0000","last_fullsized":"2024-02-10T08:40:10.027534+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:11.660049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.660049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":294,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029839+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:40:10.029839+0000","last_peered":"2024-02-10T08:40:10.029839+0000","last_clean":"2024-02-10T08:40:10.029839+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:40:10.029839+0000","last_undegraded":"2024-02-10T08:40:10.029839+0000","last_fullsized":"2024-02-10T08:40:10.029839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":331,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023214+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:40:10.023214+0000","last_peered":"2024-02-10T08:40:10.023214+0000","last_clean":"2024-02-10T08:40:10.023214+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:40:10.023214+0000","last_undegraded":"2024-02-10T08:40:10.023214+0000","last_fullsized":"2024-02-10T08:40:10.023214+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026474+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:40:10.026474+0000","last_peered":"2024-02-10T08:40:10.026474+0000","last_clean":"2024-02-10T08:40:10.026474+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:40:10.026474+0000","last_undegraded":"2024-02-10T08:40:10.026474+0000","last_fullsized":"2024-02-10T08:40:10.026474+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":695,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.816076+0000","last_change":"2024-02-10T08:39:05.948676+0000","last_active":"2024-02-10T08:40:10.816076+0000","last_peered":"2024-02-10T08:40:10.816076+0000","last_clean":"2024-02-10T08:40:10.816076+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:40:10.816076+0000","last_undegraded":"2024-02-10T08:40:10.816076+0000","last_fullsized":"2024-02-10T08:40:10.816076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:39:05.948579+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.948579+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":406,"num_read_kb":401,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.037111+0000","last_change":"2024-02-10T08:39:03.251683+0000","last_active":"2024-02-10T08:40:10.037111+0000","last_peered":"2024-02-10T08:40:10.037111+0000","last_clean":"2024-02-10T08:40:10.037111+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:40:10.037111+0000","last_undegraded":"2024-02-10T08:40:10.037111+0000","last_fullsized":"2024-02-10T08:40:10.037111+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.251585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.251585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":232,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023151+0000","last_change":"2024-02-10T08:37:10.197017+0000","last_active":"2024-02-10T08:40:10.023151+0000","last_peered":"2024-02-10T08:40:10.023151+0000","last_clean":"2024-02-10T08:40:10.023151+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:40:10.023151+0000","last_undegraded":"2024-02-10T08:40:10.023151+0000","last_fullsized":"2024-02-10T08:40:10.023151+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022264+0000","last_change":"2024-02-10T08:39:06.965218+0000","last_active":"2024-02-10T08:40:10.022264+0000","last_peered":"2024-02-10T08:40:10.022264+0000","last_clean":"2024-02-10T08:40:10.022264+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:40:10.022264+0000","last_undegraded":"2024-02-10T08:40:10.022264+0000","last_fullsized":"2024-02-10T08:40:10.022264+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.965152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.965152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":351,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027308+0000","last_change":"2024-02-10T08:38:19.613486+0000","last_active":"2024-02-10T08:40:10.027308+0000","last_peered":"2024-02-10T08:40:10.027308+0000","last_clean":"2024-02-10T08:40:10.027308+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:40:10.027308+0000","last_undegraded":"2024-02-10T08:40:10.027308+0000","last_fullsized":"2024-02-10T08:40:10.027308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:38:19.613422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.613422+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028475+0000","last_change":"2024-02-10T08:37:41.598616+0000","last_active":"2024-02-10T08:40:10.028475+0000","last_peered":"2024-02-10T08:40:10.028475+0000","last_clean":"2024-02-10T08:40:10.028475+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:40:10.028475+0000","last_undegraded":"2024-02-10T08:40:10.028475+0000","last_fullsized":"2024-02-10T08:40:10.028475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.598541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.598541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031304+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:40:10.031304+0000","last_peered":"2024-02-10T08:40:10.031304+0000","last_clean":"2024-02-10T08:40:10.031304+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:40:10.031304+0000","last_undegraded":"2024-02-10T08:40:10.031304+0000","last_fullsized":"2024-02-10T08:40:10.031304+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":237,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026416+0000","last_change":"2024-02-10T08:38:57.168613+0000","last_active":"2024-02-10T08:40:10.026416+0000","last_peered":"2024-02-10T08:40:10.026416+0000","last_clean":"2024-02-10T08:40:10.026416+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:40:10.026416+0000","last_undegraded":"2024-02-10T08:40:10.026416+0000","last_fullsized":"2024-02-10T08:40:10.026416+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:57.168523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.168523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":261,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032962+0000","last_change":"2024-02-10T08:39:52.548918+0000","last_active":"2024-02-10T08:40:10.032962+0000","last_peered":"2024-02-10T08:40:10.032962+0000","last_clean":"2024-02-10T08:40:10.032962+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:40:10.032962+0000","last_undegraded":"2024-02-10T08:40:10.032962+0000","last_fullsized":"2024-02-10T08:40:10.032962+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:52.548839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.548839+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":303,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022890+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:40:10.022890+0000","last_peered":"2024-02-10T08:40:10.022890+0000","last_clean":"2024-02-10T08:40:10.022890+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:40:10.022890+0000","last_undegraded":"2024-02-10T08:40:10.022890+0000","last_fullsized":"2024-02-10T08:40:10.022890+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022347+0000","last_change":"2024-02-10T08:38:59.955336+0000","last_active":"2024-02-10T08:40:10.022347+0000","last_peered":"2024-02-10T08:40:10.022347+0000","last_clean":"2024-02-10T08:40:10.022347+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:40:10.022347+0000","last_undegraded":"2024-02-10T08:40:10.022347+0000","last_fullsized":"2024-02-10T08:40:10.022347+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.955272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.955272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025709+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:40:10.025709+0000","last_peered":"2024-02-10T08:40:10.025709+0000","last_clean":"2024-02-10T08:40:10.025709+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:40:10.025709+0000","last_undegraded":"2024-02-10T08:40:10.025709+0000","last_fullsized":"2024-02-10T08:40:10.025709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030922+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:40:10.030922+0000","last_peered":"2024-02-10T08:40:10.030922+0000","last_clean":"2024-02-10T08:40:10.030922+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:40:10.030922+0000","last_undegraded":"2024-02-10T08:40:10.030922+0000","last_fullsized":"2024-02-10T08:40:10.030922+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":308,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026737+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:40:10.026737+0000","last_peered":"2024-02-10T08:40:10.026737+0000","last_clean":"2024-02-10T08:40:10.026737+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:40:10.026737+0000","last_undegraded":"2024-02-10T08:40:10.026737+0000","last_fullsized":"2024-02-10T08:40:10.026737+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":311,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021466+0000","last_change":"2024-02-10T08:38:32.886182+0000","last_active":"2024-02-10T08:40:10.021466+0000","last_peered":"2024-02-10T08:40:10.021466+0000","last_clean":"2024-02-10T08:40:10.021466+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:40:10.021466+0000","last_undegraded":"2024-02-10T08:40:10.021466+0000","last_fullsized":"2024-02-10T08:40:10.021466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:38:32.886090+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.886090+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":308,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023084+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:40:10.023084+0000","last_peered":"2024-02-10T08:40:10.023084+0000","last_clean":"2024-02-10T08:40:10.023084+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:40:10.023084+0000","last_undegraded":"2024-02-10T08:40:10.023084+0000","last_fullsized":"2024-02-10T08:40:10.023084+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028248+0000","last_change":"2024-02-10T08:38:22.407565+0000","last_active":"2024-02-10T08:40:10.028248+0000","last_peered":"2024-02-10T08:40:10.028248+0000","last_clean":"2024-02-10T08:40:10.028248+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:40:10.028248+0000","last_undegraded":"2024-02-10T08:40:10.028248+0000","last_fullsized":"2024-02-10T08:40:10.028248+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:22.407476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:22.407476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028458+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:40:10.028458+0000","last_peered":"2024-02-10T08:40:10.028458+0000","last_clean":"2024-02-10T08:40:10.028458+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:40:10.028458+0000","last_undegraded":"2024-02-10T08:40:10.028458+0000","last_fullsized":"2024-02-10T08:40:10.028458+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":264,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031385+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:40:10.031385+0000","last_peered":"2024-02-10T08:40:10.031385+0000","last_clean":"2024-02-10T08:40:10.031385+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:40:10.031385+0000","last_undegraded":"2024-02-10T08:40:10.031385+0000","last_fullsized":"2024-02-10T08:40:10.031385+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":294,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026646+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:40:10.026646+0000","last_peered":"2024-02-10T08:40:10.026646+0000","last_clean":"2024-02-10T08:40:10.026646+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:40:10.026646+0000","last_undegraded":"2024-02-10T08:40:10.026646+0000","last_fullsized":"2024-02-10T08:40:10.026646+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028251+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:40:10.028251+0000","last_peered":"2024-02-10T08:40:10.028251+0000","last_clean":"2024-02-10T08:40:10.028251+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:40:10.028251+0000","last_undegraded":"2024-02-10T08:40:10.028251+0000","last_fullsized":"2024-02-10T08:40:10.028251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022005+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:40:10.022005+0000","last_peered":"2024-02-10T08:40:10.022005+0000","last_clean":"2024-02-10T08:40:10.022005+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:40:10.022005+0000","last_undegraded":"2024-02-10T08:40:10.022005+0000","last_fullsized":"2024-02-10T08:40:10.022005+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":286,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029266+0000","last_change":"2024-02-10T08:37:08.277488+0000","last_active":"2024-02-10T08:40:10.029266+0000","last_peered":"2024-02-10T08:40:10.029266+0000","last_clean":"2024-02-10T08:40:10.029266+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:40:10.029266+0000","last_undegraded":"2024-02-10T08:40:10.029266+0000","last_fullsized":"2024-02-10T08:40:10.029266+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:37:08.277355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.277355+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033243+0000","last_change":"2024-02-10T08:38:30.291597+0000","last_active":"2024-02-10T08:40:10.033243+0000","last_peered":"2024-02-10T08:40:10.033243+0000","last_clean":"2024-02-10T08:40:10.033243+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:40:10.033243+0000","last_undegraded":"2024-02-10T08:40:10.033243+0000","last_fullsized":"2024-02-10T08:40:10.033243+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.291538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.291538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.036287+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:40:10.036287+0000","last_peered":"2024-02-10T08:40:10.036287+0000","last_clean":"2024-02-10T08:40:10.036287+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:40:10.036287+0000","last_undegraded":"2024-02-10T08:40:10.036287+0000","last_fullsized":"2024-02-10T08:40:10.036287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":227,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029810+0000","last_change":"2024-02-10T08:38:32.745050+0000","last_active":"2024-02-10T08:40:10.029810+0000","last_peered":"2024-02-10T08:40:10.029810+0000","last_clean":"2024-02-10T08:40:10.029810+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:40:10.029810+0000","last_undegraded":"2024-02-10T08:40:10.029810+0000","last_fullsized":"2024-02-10T08:40:10.029810+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.744972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.744972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":297,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029936+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:40:10.029936+0000","last_peered":"2024-02-10T08:40:10.029936+0000","last_clean":"2024-02-10T08:40:10.029936+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:40:10.029936+0000","last_undegraded":"2024-02-10T08:40:10.029936+0000","last_fullsized":"2024-02-10T08:40:10.029936+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028381+0000","last_change":"2024-02-10T08:37:46.576724+0000","last_active":"2024-02-10T08:40:10.028381+0000","last_peered":"2024-02-10T08:40:10.028381+0000","last_clean":"2024-02-10T08:40:10.028381+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:40:10.028381+0000","last_undegraded":"2024-02-10T08:40:10.028381+0000","last_fullsized":"2024-02-10T08:40:10.028381+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.576661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.576661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":383,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028679+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:40:10.028679+0000","last_peered":"2024-02-10T08:40:10.028679+0000","last_clean":"2024-02-10T08:40:10.028679+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:40:10.028679+0000","last_undegraded":"2024-02-10T08:40:10.028679+0000","last_fullsized":"2024-02-10T08:40:10.028679+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029941+0000","last_change":"2024-02-10T08:39:43.632113+0000","last_active":"2024-02-10T08:40:10.029941+0000","last_peered":"2024-02-10T08:40:10.029941+0000","last_clean":"2024-02-10T08:40:10.029941+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:40:10.029941+0000","last_undegraded":"2024-02-10T08:40:10.029941+0000","last_fullsized":"2024-02-10T08:40:10.029941+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:43.632048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.632048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032119+0000","last_change":"2024-02-10T08:39:01.740757+0000","last_active":"2024-02-10T08:40:10.032119+0000","last_peered":"2024-02-10T08:40:10.032119+0000","last_clean":"2024-02-10T08:40:10.032119+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:40:10.032119+0000","last_undegraded":"2024-02-10T08:40:10.032119+0000","last_fullsized":"2024-02-10T08:40:10.032119+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:01.740686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:01.740686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":298,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028553+0000","last_change":"2024-02-10T08:39:46.641751+0000","last_active":"2024-02-10T08:40:10.028553+0000","last_peered":"2024-02-10T08:40:10.028553+0000","last_clean":"2024-02-10T08:40:10.028553+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:40:10.028553+0000","last_undegraded":"2024-02-10T08:40:10.028553+0000","last_fullsized":"2024-02-10T08:40:10.028553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:39:46.641683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.641683+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":193,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022090+0000","last_change":"2024-02-10T08:37:10.217060+0000","last_active":"2024-02-10T08:40:10.022090+0000","last_peered":"2024-02-10T08:40:10.022090+0000","last_clean":"2024-02-10T08:40:10.022090+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:40:10.022090+0000","last_undegraded":"2024-02-10T08:40:10.022090+0000","last_fullsized":"2024-02-10T08:40:10.022090+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.744627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.744627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":370,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030035+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:40:10.030035+0000","last_peered":"2024-02-10T08:40:10.030035+0000","last_clean":"2024-02-10T08:40:10.030035+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:40:10.030035+0000","last_undegraded":"2024-02-10T08:40:10.030035+0000","last_fullsized":"2024-02-10T08:40:10.030035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023370+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:40:10.023370+0000","last_peered":"2024-02-10T08:40:10.023370+0000","last_clean":"2024-02-10T08:40:10.023370+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:40:10.023370+0000","last_undegraded":"2024-02-10T08:40:10.023370+0000","last_fullsized":"2024-02-10T08:40:10.023370+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":241,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026285+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:40:10.026285+0000","last_peered":"2024-02-10T08:40:10.026285+0000","last_clean":"2024-02-10T08:40:10.026285+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:40:10.026285+0000","last_undegraded":"2024-02-10T08:40:10.026285+0000","last_fullsized":"2024-02-10T08:40:10.026285+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":305,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022190+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:40:10.022190+0000","last_peered":"2024-02-10T08:40:10.022190+0000","last_clean":"2024-02-10T08:40:10.022190+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:40:10.022190+0000","last_undegraded":"2024-02-10T08:40:10.022190+0000","last_fullsized":"2024-02-10T08:40:10.022190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033382+0000","last_change":"2024-02-10T08:39:46.475770+0000","last_active":"2024-02-10T08:40:10.033382+0000","last_peered":"2024-02-10T08:40:10.033382+0000","last_clean":"2024-02-10T08:40:10.033382+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:40:10.033382+0000","last_undegraded":"2024-02-10T08:40:10.033382+0000","last_fullsized":"2024-02-10T08:40:10.033382+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.475678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.475678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028167+0000","last_change":"2024-02-10T08:38:13.463715+0000","last_active":"2024-02-10T08:40:10.028167+0000","last_peered":"2024-02-10T08:40:10.028167+0000","last_clean":"2024-02-10T08:40:10.028167+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:40:10.028167+0000","last_undegraded":"2024-02-10T08:40:10.028167+0000","last_fullsized":"2024-02-10T08:40:10.028167+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.463644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.463644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028242+0000","last_change":"2024-02-10T08:38:20.592140+0000","last_active":"2024-02-10T08:40:10.028242+0000","last_peered":"2024-02-10T08:40:10.028242+0000","last_clean":"2024-02-10T08:40:10.028242+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:40:10.028242+0000","last_undegraded":"2024-02-10T08:40:10.028242+0000","last_fullsized":"2024-02-10T08:40:10.028242+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:20.592065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.592065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":236,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034033+0000","last_change":"2024-02-10T08:39:12.687989+0000","last_active":"2024-02-10T08:40:10.034033+0000","last_peered":"2024-02-10T08:40:10.034033+0000","last_clean":"2024-02-10T08:40:10.034033+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:40:10.034033+0000","last_undegraded":"2024-02-10T08:40:10.034033+0000","last_fullsized":"2024-02-10T08:40:10.034033+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.687918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.687918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":297,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022066+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:40:10.022066+0000","last_peered":"2024-02-10T08:40:10.022066+0000","last_clean":"2024-02-10T08:40:10.022066+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:40:10.022066+0000","last_undegraded":"2024-02-10T08:40:10.022066+0000","last_fullsized":"2024-02-10T08:40:10.022066+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":296,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022375+0000","last_change":"2024-02-10T08:39:04.936255+0000","last_active":"2024-02-10T08:40:10.022375+0000","last_peered":"2024-02-10T08:40:10.022375+0000","last_clean":"2024-02-10T08:40:10.022375+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:40:10.022375+0000","last_undegraded":"2024-02-10T08:40:10.022375+0000","last_fullsized":"2024-02-10T08:40:10.022375+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:04.936174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.936174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":276,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022818+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:40:10.022818+0000","last_peered":"2024-02-10T08:40:10.022818+0000","last_clean":"2024-02-10T08:40:10.022818+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:40:10.022818+0000","last_undegraded":"2024-02-10T08:40:10.022818+0000","last_fullsized":"2024-02-10T08:40:10.022818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031540+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:40:10.031540+0000","last_peered":"2024-02-10T08:40:10.031540+0000","last_clean":"2024-02-10T08:40:10.031540+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:40:10.031540+0000","last_undegraded":"2024-02-10T08:40:10.031540+0000","last_fullsized":"2024-02-10T08:40:10.031540+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":237,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029064+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:40:10.029064+0000","last_peered":"2024-02-10T08:40:10.029064+0000","last_clean":"2024-02-10T08:40:10.029064+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:40:10.029064+0000","last_undegraded":"2024-02-10T08:40:10.029064+0000","last_fullsized":"2024-02-10T08:40:10.029064+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":283,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.025669+0000","last_change":"2024-02-10T08:37:13.426993+0000","last_active":"2024-02-10T08:40:10.025669+0000","last_peered":"2024-02-10T08:40:10.025669+0000","last_clean":"2024-02-10T08:40:10.025669+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:40:10.025669+0000","last_undegraded":"2024-02-10T08:40:10.025669+0000","last_fullsized":"2024-02-10T08:40:10.025669+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:37:13.426878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.426878+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028285+0000","last_change":"2024-02-10T08:38:13.601496+0000","last_active":"2024-02-10T08:40:10.028285+0000","last_peered":"2024-02-10T08:40:10.028285+0000","last_clean":"2024-02-10T08:40:10.028285+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:40:10.028285+0000","last_undegraded":"2024-02-10T08:40:10.028285+0000","last_fullsized":"2024-02-10T08:40:10.028285+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.601390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.601390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":282,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033159+0000","last_change":"2024-02-10T08:39:41.555411+0000","last_active":"2024-02-10T08:40:10.033159+0000","last_peered":"2024-02-10T08:40:10.033159+0000","last_clean":"2024-02-10T08:40:10.033159+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:40:10.033159+0000","last_undegraded":"2024-02-10T08:40:10.033159+0000","last_fullsized":"2024-02-10T08:40:10.033159+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.555339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.555339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.035170+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:40:10.035170+0000","last_peered":"2024-02-10T08:40:10.035170+0000","last_clean":"2024-02-10T08:40:10.035170+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:40:10.035170+0000","last_undegraded":"2024-02-10T08:40:10.035170+0000","last_fullsized":"2024-02-10T08:40:10.035170+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029806+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:40:10.029806+0000","last_peered":"2024-02-10T08:40:10.029806+0000","last_clean":"2024-02-10T08:40:10.029806+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:40:10.029806+0000","last_undegraded":"2024-02-10T08:40:10.029806+0000","last_fullsized":"2024-02-10T08:40:10.029806+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":296,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031811+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:40:10.031811+0000","last_peered":"2024-02-10T08:40:10.031811+0000","last_clean":"2024-02-10T08:40:10.031811+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:40:10.031811+0000","last_undegraded":"2024-02-10T08:40:10.031811+0000","last_fullsized":"2024-02-10T08:40:10.031811+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":289,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034323+0000","last_change":"2024-02-10T08:39:49.529787+0000","last_active":"2024-02-10T08:40:10.034323+0000","last_peered":"2024-02-10T08:40:10.034323+0000","last_clean":"2024-02-10T08:40:10.034323+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:40:10.034323+0000","last_undegraded":"2024-02-10T08:40:10.034323+0000","last_fullsized":"2024-02-10T08:40:10.034323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:49.529724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.529724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":275,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026166+0000","last_change":"2024-02-10T08:37:15.252972+0000","last_active":"2024-02-10T08:40:10.026166+0000","last_peered":"2024-02-10T08:40:10.026166+0000","last_clean":"2024-02-10T08:40:10.026166+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:40:10.026166+0000","last_undegraded":"2024-02-10T08:40:10.026166+0000","last_fullsized":"2024-02-10T08:40:10.026166+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.252897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.252897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":287,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028332+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:40:10.028332+0000","last_peered":"2024-02-10T08:40:10.028332+0000","last_clean":"2024-02-10T08:40:10.028332+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:40:10.028332+0000","last_undegraded":"2024-02-10T08:40:10.028332+0000","last_fullsized":"2024-02-10T08:40:10.028332+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033623+0000","last_change":"2024-02-10T08:38:59.747834+0000","last_active":"2024-02-10T08:40:10.033623+0000","last_peered":"2024-02-10T08:40:10.033623+0000","last_clean":"2024-02-10T08:40:10.033623+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:40:10.033623+0000","last_undegraded":"2024-02-10T08:40:10.033623+0000","last_fullsized":"2024-02-10T08:40:10.033623+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.747762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.747762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":236,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034562+0000","last_change":"2024-02-10T08:39:41.691153+0000","last_active":"2024-02-10T08:40:10.034562+0000","last_peered":"2024-02-10T08:40:10.034562+0000","last_clean":"2024-02-10T08:40:10.034562+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:40:10.034562+0000","last_undegraded":"2024-02-10T08:40:10.034562+0000","last_fullsized":"2024-02-10T08:40:10.034562+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:39:41.691086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.691086+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028138+0000","last_change":"2024-02-10T08:37:39.573283+0000","last_active":"2024-02-10T08:40:10.028138+0000","last_peered":"2024-02-10T08:40:10.028138+0000","last_clean":"2024-02-10T08:40:10.028138+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:40:10.028138+0000","last_undegraded":"2024-02-10T08:40:10.028138+0000","last_fullsized":"2024-02-10T08:40:10.028138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:39.573225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.573225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.030368+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:40:10.030368+0000","last_peered":"2024-02-10T08:40:10.030368+0000","last_clean":"2024-02-10T08:40:10.030368+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:40:10.030368+0000","last_undegraded":"2024-02-10T08:40:10.030368+0000","last_fullsized":"2024-02-10T08:40:10.030368+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.026924+0000","last_change":"2024-02-10T08:38:14.837278+0000","last_active":"2024-02-10T08:40:10.026924+0000","last_peered":"2024-02-10T08:40:10.026924+0000","last_clean":"2024-02-10T08:40:10.026924+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:40:10.026924+0000","last_undegraded":"2024-02-10T08:40:10.026924+0000","last_fullsized":"2024-02-10T08:40:10.026924+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:14.837154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:14.837154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":306,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034410+0000","last_change":"2024-02-10T08:39:47.473081+0000","last_active":"2024-02-10T08:40:10.034410+0000","last_peered":"2024-02-10T08:40:10.034410+0000","last_clean":"2024-02-10T08:40:10.034410+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:40:10.034410+0000","last_undegraded":"2024-02-10T08:40:10.034410+0000","last_fullsized":"2024-02-10T08:40:10.034410+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:39:47.473007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.473007+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031634+0000","last_change":"2024-02-10T08:38:31.436092+0000","last_active":"2024-02-10T08:40:10.031634+0000","last_peered":"2024-02-10T08:40:10.031634+0000","last_clean":"2024-02-10T08:40:10.031634+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:40:10.031634+0000","last_undegraded":"2024-02-10T08:40:10.031634+0000","last_fullsized":"2024-02-10T08:40:10.031634+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.435994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.435994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022230+0000","last_change":"2024-02-10T08:38:58.917062+0000","last_active":"2024-02-10T08:40:10.022230+0000","last_peered":"2024-02-10T08:40:10.022230+0000","last_clean":"2024-02-10T08:40:10.022230+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:40:10.022230+0000","last_undegraded":"2024-02-10T08:40:10.022230+0000","last_fullsized":"2024-02-10T08:40:10.022230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:58.916996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:58.916996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.023068+0000","last_change":"2024-02-10T08:38:28.906569+0000","last_active":"2024-02-10T08:40:10.023068+0000","last_peered":"2024-02-10T08:40:10.023068+0000","last_clean":"2024-02-10T08:40:10.023068+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:40:10.023068+0000","last_undegraded":"2024-02-10T08:40:10.023068+0000","last_fullsized":"2024-02-10T08:40:10.023068+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:38:28.906474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":315,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028596+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:40:10.028596+0000","last_peered":"2024-02-10T08:40:10.028596+0000","last_clean":"2024-02-10T08:40:10.028596+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:40:10.028596+0000","last_undegraded":"2024-02-10T08:40:10.028596+0000","last_fullsized":"2024-02-10T08:40:10.028596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032861+0000","last_change":"2024-02-10T08:38:32.342305+0000","last_active":"2024-02-10T08:40:10.032861+0000","last_peered":"2024-02-10T08:40:10.032861+0000","last_clean":"2024-02-10T08:40:10.032861+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:40:10.032861+0000","last_undegraded":"2024-02-10T08:40:10.032861+0000","last_fullsized":"2024-02-10T08:40:10.032861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.342200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.342200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022665+0000","last_change":"2024-02-10T08:37:16.490795+0000","last_active":"2024-02-10T08:40:10.022665+0000","last_peered":"2024-02-10T08:40:10.022665+0000","last_clean":"2024-02-10T08:40:10.022665+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:40:10.022665+0000","last_undegraded":"2024-02-10T08:40:10.022665+0000","last_fullsized":"2024-02-10T08:40:10.022665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:16.490727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:16.490727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031884+0000","last_change":"2024-02-10T08:38:18.541756+0000","last_active":"2024-02-10T08:40:10.031884+0000","last_peered":"2024-02-10T08:40:10.031884+0000","last_clean":"2024-02-10T08:40:10.031884+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:40:10.031884+0000","last_undegraded":"2024-02-10T08:40:10.031884+0000","last_fullsized":"2024-02-10T08:40:10.031884+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.541693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.541693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":305,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027086+0000","last_change":"2024-02-10T08:38:19.453131+0000","last_active":"2024-02-10T08:40:10.027086+0000","last_peered":"2024-02-10T08:40:10.027086+0000","last_clean":"2024-02-10T08:40:10.027086+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:40:10.027086+0000","last_undegraded":"2024-02-10T08:40:10.027086+0000","last_fullsized":"2024-02-10T08:40:10.027086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:38:19.453060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.453060+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.028094+0000","last_change":"2024-02-10T08:38:16.426049+0000","last_active":"2024-02-10T08:40:10.028094+0000","last_peered":"2024-02-10T08:40:10.028094+0000","last_clean":"2024-02-10T08:40:10.028094+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:40:10.028094+0000","last_undegraded":"2024-02-10T08:40:10.028094+0000","last_fullsized":"2024-02-10T08:40:10.028094+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:16.425945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:16.425945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029401+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:40:10.029401+0000","last_peered":"2024-02-10T08:40:10.029401+0000","last_clean":"2024-02-10T08:40:10.029401+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:40:10.029401+0000","last_undegraded":"2024-02-10T08:40:10.029401+0000","last_fullsized":"2024-02-10T08:40:10.029401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":228,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022071+0000","last_change":"2024-02-10T08:39:46.928020+0000","last_active":"2024-02-10T08:40:10.022071+0000","last_peered":"2024-02-10T08:40:10.022071+0000","last_clean":"2024-02-10T08:40:10.022071+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:40:10.022071+0000","last_undegraded":"2024-02-10T08:40:10.022071+0000","last_fullsized":"2024-02-10T08:40:10.022071+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.927953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":318,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022146+0000","last_change":"2024-02-10T08:38:26.871174+0000","last_active":"2024-02-10T08:40:10.022146+0000","last_peered":"2024-02-10T08:40:10.022146+0000","last_clean":"2024-02-10T08:40:10.022146+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:40:10.022146+0000","last_undegraded":"2024-02-10T08:40:10.022146+0000","last_fullsized":"2024-02-10T08:40:10.022146+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:38:26.871100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.871100+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029966+0000","last_change":"2024-02-10T08:39:02.250833+0000","last_active":"2024-02-10T08:40:10.029966+0000","last_peered":"2024-02-10T08:40:10.029966+0000","last_clean":"2024-02-10T08:40:10.029966+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:40:10.029966+0000","last_undegraded":"2024-02-10T08:40:10.029966+0000","last_fullsized":"2024-02-10T08:40:10.029966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:02.250719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:02.250719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":198,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021487+0000","last_change":"2024-02-10T08:38:27.952601+0000","last_active":"2024-02-10T08:40:10.021487+0000","last_peered":"2024-02-10T08:40:10.021487+0000","last_clean":"2024-02-10T08:40:10.021487+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:40:10.021487+0000","last_undegraded":"2024-02-10T08:40:10.021487+0000","last_fullsized":"2024-02-10T08:40:10.021487+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:27.952518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:27.952518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":232,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.029584+0000","last_change":"2024-02-10T08:37:10.218417+0000","last_active":"2024-02-10T08:40:10.029584+0000","last_peered":"2024-02-10T08:40:10.029584+0000","last_clean":"2024-02-10T08:40:10.029584+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:40:10.029584+0000","last_undegraded":"2024-02-10T08:40:10.029584+0000","last_fullsized":"2024-02-10T08:40:10.029584+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:36:35.870864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022190+0000","last_change":"2024-02-10T08:39:11.790943+0000","last_active":"2024-02-10T08:40:10.022190+0000","last_peered":"2024-02-10T08:40:10.022190+0000","last_clean":"2024-02-10T08:40:10.022190+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:40:10.022190+0000","last_undegraded":"2024-02-10T08:40:10.022190+0000","last_fullsized":"2024-02-10T08:40:10.022190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.790866+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.036405+0000","last_change":"2024-02-10T08:38:30.389134+0000","last_active":"2024-02-10T08:40:10.036405+0000","last_peered":"2024-02-10T08:40:10.036405+0000","last_clean":"2024-02-10T08:40:10.036405+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:40:10.036405+0000","last_undegraded":"2024-02-10T08:40:10.036405+0000","last_fullsized":"2024-02-10T08:40:10.036405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.389076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":278,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027713+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:40:10.027713+0000","last_peered":"2024-02-10T08:40:10.027713+0000","last_clean":"2024-02-10T08:40:10.027713+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:40:10.027713+0000","last_undegraded":"2024-02-10T08:40:10.027713+0000","last_fullsized":"2024-02-10T08:40:10.027713+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.032892+0000","last_change":"2024-02-10T08:38:15.539954+0000","last_active":"2024-02-10T08:40:10.032892+0000","last_peered":"2024-02-10T08:40:10.032892+0000","last_clean":"2024-02-10T08:40:10.032892+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:40:10.032892+0000","last_undegraded":"2024-02-10T08:40:10.032892+0000","last_fullsized":"2024-02-10T08:40:10.032892+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:15.539850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:15.539850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":313,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.021451+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:40:10.021451+0000","last_peered":"2024-02-10T08:40:10.021451+0000","last_clean":"2024-02-10T08:40:10.021451+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:40:10.021451+0000","last_undegraded":"2024-02-10T08:40:10.021451+0000","last_fullsized":"2024-02-10T08:40:10.021451+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031214+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:40:10.031214+0000","last_peered":"2024-02-10T08:40:10.031214+0000","last_clean":"2024-02-10T08:40:10.031214+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:40:10.031214+0000","last_undegraded":"2024-02-10T08:40:10.031214+0000","last_fullsized":"2024-02-10T08:40:10.031214+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":279,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.022101+0000","last_change":"2024-02-10T08:39:05.790083+0000","last_active":"2024-02-10T08:40:10.022101+0000","last_peered":"2024-02-10T08:40:10.022101+0000","last_clean":"2024-02-10T08:40:10.022101+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:40:10.022101+0000","last_undegraded":"2024-02-10T08:40:10.022101+0000","last_fullsized":"2024-02-10T08:40:10.022101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.790005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.790005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":236,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027198+0000","last_change":"2024-02-10T08:38:30.695748+0000","last_active":"2024-02-10T08:40:10.027198+0000","last_peered":"2024-02-10T08:40:10.027198+0000","last_clean":"2024-02-10T08:40:10.027198+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:40:10.027198+0000","last_undegraded":"2024-02-10T08:40:10.027198+0000","last_fullsized":"2024-02-10T08:40:10.027198+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.695684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.695684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":327,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.031901+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:40:10.031901+0000","last_peered":"2024-02-10T08:40:10.031901+0000","last_clean":"2024-02-10T08:40:10.031901+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:40:10.031901+0000","last_undegraded":"2024-02-10T08:40:10.031901+0000","last_fullsized":"2024-02-10T08:40:10.031901+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":197,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.034825+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:40:10.034825+0000","last_peered":"2024-02-10T08:40:10.034825+0000","last_clean":"2024-02-10T08:40:10.034825+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:40:10.034825+0000","last_undegraded":"2024-02-10T08:40:10.034825+0000","last_fullsized":"2024-02-10T08:40:10.034825+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":273,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.027900+0000","last_change":"2024-02-10T08:37:15.367612+0000","last_active":"2024-02-10T08:40:10.027900+0000","last_peered":"2024-02-10T08:40:10.027900+0000","last_clean":"2024-02-10T08:40:10.027900+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:40:10.027900+0000","last_undegraded":"2024-02-10T08:40:10.027900+0000","last_fullsized":"2024-02-10T08:40:10.027900+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:15.367499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:15.367499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":227,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-02-10T08:40:10.033264+0000","last_change":"2024-02-10T08:38:34.376839+0000","last_active":"2024-02-10T08:40:10.033264+0000","last_peered":"2024-02-10T08:40:10.033264+0000","last_clean":"2024-02-10T08:40:10.033264+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:40:10.033264+0000","last_undegraded":"2024-02-10T08:40:10.033264+0000","last_fullsized":"2024-02-10T08:40:10.033264+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:34.376762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:34.376762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":193,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001220,"num_objects":123,"num_object_clones":2,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":548,"num_write_kb":4449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13496320,"data_stored":12003668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1805,"ondisk_log_size":1805,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":406,"num_read_kb":401,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299477,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306484,"kb_used_data":5160,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438668,"statfs":{"total":95995035648,"available":95681196032,"internally_reserved":0,"allocated":5283840,"data_stored":2908281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.76400000000000001,"15min":0.60299999999999998},"min":{"1min":0.30399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.70399999999999996,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67100000000000004,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59399999999999997,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.62,"15min":0.54800000000000004},"min":{"1min":0.314,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.623,"5min":7.593,"15min":7.593},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65100000000000002,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65600000000000003,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.63800000000000001,"15min":0.54600000000000004},"min":{"1min":0.36299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.85399999999999998,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":19.369,"15min":19.369},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.68700000000000006,"15min":0.56200000000000006},"min":{"1min":0.378,"5min":0.307,"15min":0.307},"max":{"1min":0.63500000000000001,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.84999999999999998,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72799999999999998,"5min":22.102,"15min":22.102},"last":0.55900000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.498,"15min":0.44900000000000001},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.624,"5min":8.827,"15min":8.827},"last":0.247},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.56999999999999995,"15min":0.47399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67000000000000004,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.31}]},{"osd":5,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.67400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70799999999999996,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68899999999999995,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.32700000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.70799999999999996,"15min":0.56200000000000006},"min":{"1min":0.313,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89100000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.503,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69699999999999995,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.35099999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561115,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318496,"kb_used_data":8976,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426656,"statfs":{"total":95995035648,"available":95668895744,"internally_reserved":0,"allocated":9191424,"data_stored":6908048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.628,"15min":0.51600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.19800000000000001},"max":{"1min":0.625,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.378},{"interface":"front","average":{"1min":0.441,"5min":0.65000000000000002,"15min":0.53100000000000003},"min":{"1min":0.28799999999999998,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.58799999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.58199999999999996,"15min":0.52600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72499999999999998,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.502},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.56899999999999995,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.66000000000000003,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.46800000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.57599999999999996,"15min":0.52700000000000002},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64900000000000002,"5min":4.798,"15min":4.798},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58299999999999996,"15min":0.52800000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.69599999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.44600000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.82099999999999995,"15min":0.59299999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":16.823,"15min":16.823},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.57299999999999995,"15min":0.51300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.629,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.39700000000000002},"min":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56899999999999995,"5min":3.657,"15min":3.657},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.38500000000000001},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60099999999999998,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.40200000000000002},"min":{"1min":0.317,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.62,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47299999999999998,"15min":0.42399999999999999},"min":{"1min":0.34100000000000003,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.50800000000000001,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318764,"kb_used_data":5144,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426388,"statfs":{"total":95995035648,"available":95668621312,"internally_reserved":0,"allocated":5267456,"data_stored":2909734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.56999999999999995,"15min":0.41099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":0.58099999999999996,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.64300000000000002,"15min":0.45900000000000002},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.19500000000000001},"max":{"1min":0.56599999999999995,"5min":16.166,"15min":16.166},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":1.0980000000000001,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.55300000000000005,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.86099999999999999,"15min":0.53300000000000003},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54700000000000004,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.315}]},{"osd":3,"last update":"Sat Feb 10 08:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.74399999999999999,"15min":0.53200000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":24.773,"15min":24.773},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71199999999999997,"15min":0.53700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.60399999999999998,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.377}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.75700000000000001,"15min":0.61899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.67600000000000005,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.67600000000000005,"15min":0.60799999999999998},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.45200000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67200000000000004,"15min":0.61199999999999999},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.78600000000000003,"15min":0.67500000000000004},"min":{"1min":0.40400000000000003,"5min":0.38700000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.68300000000000005,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.76900000000000002,"5min":9.532,"15min":9.532},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.79500000000000004,"15min":0.65700000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.72499999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.83999999999999997,"15min":0.66700000000000004},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.57499999999999996}]}]},{"osd":0,"up_from":0,"seq":34359738488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314624,"kb_used_data":5108,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430528,"statfs":{"total":95995035648,"available":95672860672,"internally_reserved":0,"allocated":5230592,"data_stored":2907783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.5,"15min":0.40300000000000002},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.58599999999999997,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.65200000000000002,"15min":0.46000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56799999999999995,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.31}]},{"osd":1,"last update":"Sat Feb 10 08:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.70099999999999996,"15min":0.47699999999999998},"min":{"1min":0.24399999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":12.512,"15min":12.512},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.60999999999999999,"15min":0.46200000000000002},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.223},"max":{"1min":0.50800000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.25900000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.6520000000000001,"5min":4.617,"15min":4.617},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.59399999999999997,"15min":0.48599999999999999},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.878,"15min":0.68000000000000005},"min":{"1min":0.32800000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.7370000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.77700000000000002,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":0.79000000000000004,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.84099999999999997,"15min":0.65400000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68799999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.71599999999999997,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.50900000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.81299999999999994,"15min":0.66300000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.6759999999999999,"5min":9.266,"15min":9.266},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.73099999999999998,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.80100000000000005,"15min":0.69299999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84099999999999997,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.63},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.77200000000000002,"15min":0.66800000000000004},"min":{"1min":0.46600000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.57999999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182506,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318564,"kb_used_data":9040,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426588,"statfs":{"total":95995035648,"available":95668826112,"internally_reserved":0,"allocated":9256960,"data_stored":6908178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.81699999999999995,"15min":0.51600000000000001},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.42299999999999999,"5min":14.744,"15min":14.744},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.872,"15min":0.54500000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.498,"5min":11.808,"15min":11.808},"last":3.5579999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.82599999999999996,"15min":0.55100000000000005},"min":{"1min":0.23200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.262,"5min":16.634,"15min":16.634},"last":9.3100000000000005},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.82099999999999995,"15min":0.56100000000000005},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":0.55100000000000005,"5min":12.438000000000001,"15min":12.438000000000001},"last":5.4370000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.86399999999999999,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":16.536000000000001,"15min":16.536000000000001},"last":3.1549999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":1.079,"15min":0.63500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.51500000000000001,"5min":23.294,"15min":23.294},"last":0.307}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.84799999999999998,"15min":0.68400000000000005},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":0.70799999999999996,"5min":12.371,"15min":12.371},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.0089999999999999,"15min":0.71699999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":14.821999999999999,"5min":14.821999999999999,"15min":14.821999999999999},"last":14.821999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.97199999999999998,"15min":0.69999999999999996},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73399999999999999,"5min":12.579000000000001,"15min":12.579000000000001},"last":3.6040000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.76600000000000001,"15min":0.64100000000000001},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70199999999999996,"5min":11.448,"15min":11.448},"last":4.9909999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:40:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.97399999999999998,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":16.091000000000001,"15min":16.091000000000001},"last":3.48},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.91800000000000004,"15min":0.67300000000000004},"min":{"1min":0.378,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72899999999999998,"5min":16.475000000000001,"15min":16.475000000000001},"last":3.2469999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.879,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72199999999999998,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":1.0469999999999999,"15min":0.71199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68100000000000005,"5min":12.606999999999999,"15min":12.606999999999999},"last":3.52}]}]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306452,"kb_used_data":5128,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438700,"statfs":{"total":95995035648,"available":95681228800,"internally_reserved":0,"allocated":5251072,"data_stored":2908280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.84199999999999997,"15min":0.64400000000000002},"min":{"1min":0.29499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71999999999999997,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.65600000000000003,"15min":0.56200000000000006},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.626,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.497}]},{"osd":1,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.59399999999999997},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70599999999999996,"5min":5.774,"15min":5.774},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.74099999999999999,"15min":0.61099999999999999},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67100000000000004,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.76100000000000001,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69399999999999995,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.77400000000000002,"15min":0.61399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63700000000000001,"5min":10.01,"15min":10.01},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.91900000000000004,"15min":0.66500000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.62,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.0960000000000001,"15min":0.71999999999999997},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.82999999999999996,"5min":25.27,"15min":25.27},"last":0.48299999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.51000000000000001,"15min":0.41899999999999998},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67300000000000004,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47699999999999998,"15min":0.41299999999999998},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71499999999999997,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.44}]},{"osd":6,"last update":"Sat Feb 10 08:39:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.65600000000000003,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.70499999999999996,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.63800000000000001,"15min":0.504},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":13.753,"15min":13.753},"last":0.52300000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.61699999999999999,"15min":0.50700000000000001},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69499999999999995,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.67900000000000005,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0509999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.56100000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790048,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310304,"kb_used_data":8976,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434848,"statfs":{"total":95995035648,"available":95677284352,"internally_reserved":0,"allocated":9191424,"data_stored":6907607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.9710000000000001,"15min":0.999},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66400000000000003,"5min":107.551,"15min":107.551},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.845,"15min":0.96399999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":0.61899999999999999,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.6910000000000001,"15min":0.93999999999999995},"min":{"1min":0.34899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":107.205,"15min":107.205},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":1.363,"15min":0.82899999999999996},"min":{"1min":0.32100000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.65700000000000003,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":1.8480000000000001,"15min":0.95399999999999996},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67800000000000005,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.623},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.927,"15min":0.995},"min":{"1min":0.311,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":0.65800000000000003,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.60399999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.1479999999999999,"15min":0.71899999999999997},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2669999999999999,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":2.5219999999999998,"15min":1.2070000000000001},"min":{"1min":0.41699999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61299999999999999,"5min":114.526,"15min":114.526},"last":0.70599999999999996}]},{"osd":4,"last update":"Sat Feb 10 08:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":1.1699999999999999,"15min":0.65300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.627,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.505},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.90200000000000002,"15min":0.55900000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.58699999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.52500000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.1910000000000001,"15min":0.67300000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64800000000000002,"5min":72.241,"15min":72.241},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.434,"5min":0.70299999999999996,"15min":0.53900000000000003},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.56899999999999995,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":1.1819999999999999,"15min":0.71099999999999997},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.60899999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.2450000000000001,"15min":0.73899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4000411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4190208,"data_stored":4000483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":193,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:40:16.896 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-10T08:40:16.914 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:16.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:17.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:17.144 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:17.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:17.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:17.279 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:17.299 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:17.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:17.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:16.230964+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/1701852073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:16.234706+0000 mon.a (mon.0) 2126 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:40:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:16.234787+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-02-10T08:40:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:16.317328+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:16.317480+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:16.991207+0000 mon.a (mon.0) 2128 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:16.991291+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:16.230964+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/1701852073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:16.234706+0000 mon.a (mon.0) 2126 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:16.234787+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:16.317328+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:16.317480+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:16.991207+0000 mon.a (mon.0) 2128 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:17.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:16.991291+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-10T08:40:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:17.451 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:17.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:16.230964+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.107:0/1701852073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:16.234706+0000 mon.a (mon.0) 2126 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:16.234787+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:16.317328+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:16.317480+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:16.991207+0000 mon.a (mon.0) 2128 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi107-112411-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:16.991291+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-10T08:40:17.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:17.654 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:17.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.225+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1815426162 >> v1:172.21.15.107:6790/0 conn(0x7fb7d40fc910 legacy=0x7fb7d40fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.225+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1815426162 shutdown_connections 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.225+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1815426162 >> 172.21.15.107:0/1815426162 conn(0x7fb7d40f2510 msgr2=0x7fb7d40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1815426162 shutdown_connections 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1815426162 wait complete. 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 librados: starting msgr at 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 librados: starting objecter 2024-02-10T08:40:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 Processor -- start 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 -- start start 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 librados: setting wanted keys 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.226+0000 7fb7da5b5700 1 librados: calling monclient init 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d40f4970 con 0x7fb7d4103250 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d411ab10 con 0x7fb7d40fc910 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7d411bcf0 con 0x7fb7d40ff710 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d37fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb7d40ff710 0x7fb7d4114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60342/0 (socket says 172.21.15.107:60342) 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d37fe700 1 -- 172.21.15.107:0/1701852073 learned_addr learned my addr 172.21.15.107:0/1701852073 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter ms_handle_connect 0x7fb7d40ff710 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter ms_handle_connect 0x7fb7d4103250 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter ms_handle_connect 0x7fb7d40fc910 2024-02-10T08:40:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445774732 0 0) 0x7fb7d411bcf0 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc002220 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169081933 0 0) 0x7fb7d411ab10 con 0x7fb7d40fc910 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d411bcf0 con 0x7fb7d40fc910 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754809489 0 0) 0x7fb7bc002220 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7d411ab10 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb7c8001920 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214999653 0 0) 0x7fb7d411bcf0 con 0x7fb7d40fc910 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7bc002220 con 0x7fb7d40fc910 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb7c4001a60 con 0x7fb7d40fc910 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1169390150 0 0) 0x7fb7d411ab10 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.195:6789/0 conn(0x7fb7d40fc910 legacy=0x7fb7d40fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6789/0 conn(0x7fb7d4103250 legacy=0x7fb7d41183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d411cf10 con 0x7fb7d40ff710 2024-02-10T08:40:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d4119b60 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 10 client.53387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4119fd0 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.227+0000 7fb7da5b5700 1 librados: init done 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.228+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb7c8001cb0 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.228+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb7c8005bc0 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.228+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb7c0004f00 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb7c8002970 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(322..322 src has 50..322) v4 ==== 8388+0+0 (unknown 2685286674 0 0) 0x7fb7c8078ff0 con 0x7fb7d40ff710 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 osd_map(322..322 src has 50..322) v4 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 3 client.53387.objecter handle_osd_map got epochs [322,322] > 0 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 3 client.53387.objecter handle_osd_map decoding full epoch 322 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 20 client.53387.objecter dump_active .. 0 homeless 2024-02-10T08:40:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v322) v1 ==== 54+0+16 (unknown 362709316 0 2420000192) 0x7fb7c807b390 con 0x7fb7d40ff710 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7bc063ad0 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 1 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 1 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 0 1 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 4 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 10 client.53387.objecter _send_command 1 2024-02-10T08:40:17.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.229+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c0004950 con 0x7fb7c0004530 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c0004530 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c0004950 con 0x7fb7c0004530 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 1 = system:0 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 4 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.230+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 0 1 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 2 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 2 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c00051a0 osd=0 3 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 0 2 2024-02-10T08:40:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 4 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 10 client.53387.objecter _send_command 2 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.246+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c0004d30 con 0x7fb7c0004530 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c0004d30 con 0x7fb7c0004530 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 4 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 0 2 2024-02-10T08:40:17.743 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 1 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 4 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 10 client.53387.objecter _send_command 3 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c000c000 con 0x7fb7c0004150 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.248+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c0004150 2024-02-10T08:40:17.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.249+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c000c000 con 0x7fb7c0004150 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.249+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.249+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 3 = system:0 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.249+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.249+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 1 3 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0003fe0 osd=1 3 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 1 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 10 client.53387.objecter _send_command 4 2024-02-10T08:40:17.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.264+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c00018b0 con 0x7fb7c0004150 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c00018b0 con 0x7fb7c0004150 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 4 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 1 4 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 5 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 5 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 2 5 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 4 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 10 client.53387.objecter _send_command 5 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.265+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c000de90 con 0x7fb7c000d2f0 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c000d2f0 2024-02-10T08:40:17.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c000de90 con 0x7fb7c000d2f0 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 5 = system:0 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 4 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.266+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 2 5 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 6 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 6 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0003630 osd=2 3 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 2 6 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 4 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 10 client.53387.objecter _send_command 6 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.281+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c00030a0 con 0x7fb7c000d2f0 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c00030a0 con 0x7fb7c000d2f0 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 4 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 2 6 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 7 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 7 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 3 7 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 4 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 10 client.53387.objecter _send_command 7 2024-02-10T08:40:17.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c0015b00 con 0x7fb7c0011ff0 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.282+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c0011ff0 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.283+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c0015b00 con 0x7fb7c0011ff0 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.283+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.283+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 7 = system:0 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.283+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 4 2024-02-10T08:40:17.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.283+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 3 7 2024-02-10T08:40:17.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:17.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 8 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 8 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0011970 osd=3 3 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 3 8 2024-02-10T08:40:17.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 4 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 10 client.53387.objecter _send_command 8 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.297+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c000e240 con 0x7fb7c0011ff0 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c000e240 con 0x7fb7c0011ff0 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 4 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 3 8 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 9 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 9 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 4 9 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 4 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 10 client.53387.objecter _send_command 9 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.299+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c000e1b0 con 0x7fb7c000ce50 2024-02-10T08:40:17.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c000ce50 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c000e1b0 con 0x7fb7c000ce50 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 9 = system:0 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 4 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.300+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 4 9 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 10 2024-02-10T08:40:17.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.758 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 10 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c000e000 osd=4 3 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 4 10 2024-02-10T08:40:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 4 2024-02-10T08:40:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 10 client.53387.objecter _send_command 10 2024-02-10T08:40:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.315+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c000e1b0 con 0x7fb7c000ce50 2024-02-10T08:40:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.317+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c000e1b0 con 0x7fb7c000ce50 2024-02-10T08:40:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.317+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.317+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.317+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 4 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.317+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 4 10 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 11 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 11 2024-02-10T08:40:17.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 5 11 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 4 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 10 client.53387.objecter _send_command 11 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c0018010 con 0x7fb7c0017a60 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.318+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c0017a60 2024-02-10T08:40:17.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.319+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c0018010 con 0x7fb7c0017a60 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.319+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.319+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 11 = system:0 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.319+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 4 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.319+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 5 11 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 12 2024-02-10T08:40:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 12 2024-02-10T08:40:17.765 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:17.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c000e1b0 osd=5 3 2024-02-10T08:40:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 5 12 2024-02-10T08:40:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 4 2024-02-10T08:40:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 10 client.53387.objecter _send_command 12 2024-02-10T08:40:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.335+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c0018250 con 0x7fb7c0017a60 2024-02-10T08:40:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.337+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c8002270 con 0x7fb7c0017a60 2024-02-10T08:40:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.337+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.337+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.337+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 4 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.337+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 5 12 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 13 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 13 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 6 13 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 4 2024-02-10T08:40:17.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 10 client.53387.objecter _send_command 13 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c0022d80 con 0x7fb7c001f080 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.338+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c001f080 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.339+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c0022d80 con 0x7fb7c001f080 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.339+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.339+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 13 = system:0 2024-02-10T08:40:17.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.339+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 4 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.339+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 6 13 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 14 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 14 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0018010 osd=6 3 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 6 14 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 4 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 10 client.53387.objecter _send_command 14 2024-02-10T08:40:17.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.356+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c0022d80 con 0x7fb7c001f080 2024-02-10T08:40:17.773 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c0022d80 con 0x7fb7c001f080 2024-02-10T08:40:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 4 2024-02-10T08:40:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 6 14 2024-02-10T08:40:17.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 15 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 15 2024-02-10T08:40:17.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 7 15 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 4 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 10 client.53387.objecter _send_command 15 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.358+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7c0023fa0 con 0x7fb7c0024a80 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 10 client.53387.objecter ms_handle_connect 0x7fb7c0024a80 2024-02-10T08:40:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb7c0023fa0 con 0x7fb7c0024a80 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 15 = system:0 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 4 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.359+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 7 15 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 10 client.53387.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:17.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign -1 16 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 20 client.53387.objecter _get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_remove -1 16 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 20 client.53387.objecter get_session s=0x7fb7c0022d80 osd=7 3 2024-02-10T08:40:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 15 client.53387.objecter _session_command_op_assign 7 16 2024-02-10T08:40:17.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 4 2024-02-10T08:40:17.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 10 client.53387.objecter _send_command 16 2024-02-10T08:40:17.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.374+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb7c0023fa0 con 0x7fb7c0024a80 2024-02-10T08:40:17.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d17fa700 1 -- 172.21.15.107:0/1701852073 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb7c0023fa0 con 0x7fb7c0024a80 2024-02-10T08:40:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d17fa700 10 client.53387.objecter ms_dispatch 0x7fb7d4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d17fa700 10 client.53387.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d17fa700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 4 2024-02-10T08:40:17.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d17fa700 15 client.53387.objecter _session_command_op_remove 7 16 2024-02-10T08:40:17.781 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:17.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 librados: watch_flush enter 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 librados: watch_flush exit 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.0 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6801/3321596972 conn(0x7fb7c0004530 legacy=0x7fb7c0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c00051a0 osd=0 1 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.1 2024-02-10T08:40:17.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6805/133516126 conn(0x7fb7c0004150 legacy=0x7fb7c000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003fe0 osd=1 1 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.2 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6809/3575881728 conn(0x7fb7c000d2f0 legacy=0x7fb7c000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0003630 osd=2 1 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.3 2024-02-10T08:40:17.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6813/2871825691 conn(0x7fb7c0011ff0 legacy=0x7fb7c00143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0011970 osd=3 1 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.4 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.195:6800/2499671285 conn(0x7fb7c000ce50 legacy=0x7fb7c0016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d8b52700 1 -- 172.21.15.107:0/1701852073 reap_dead start 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d8b52700 1 -- 172.21.15.107:0/1701852073 reap_dead start 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e000 osd=4 1 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.5 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.195:6804/4178321486 conn(0x7fb7c0017a60 legacy=0x7fb7c001d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c000e1b0 osd=5 1 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.6 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.195:6808/1417479017 conn(0x7fb7c001f080 legacy=0x7fb7c00214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0018010 osd=6 1 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter close_session for osd.7 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.195:6812/1934176928 conn(0x7fb7c0024a80 legacy=0x7fb7c0024ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter put_session s=0x7fb7c0022d80 osd=7 1 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 20 client.53387.objecter shutdown clearing up homeless session... 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 10 client.53387.objecter successfully canceled tick 2024-02-10T08:40:17.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb7bc063ad0 legacy=0x7fb7bc065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> v1:172.21.15.107:6790/0 conn(0x7fb7d40ff710 legacy=0x7fb7d4114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d8b52700 1 -- 172.21.15.107:0/1701852073 reap_dead start 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7d8b52700 1 -- 172.21.15.107:0/1701852073 reap_dead start 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 shutdown_connections 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 >> 172.21.15.107:0/1701852073 conn(0x7fb7d40f2510 msgr2=0x7fb7d41066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 shutdown_connections 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 -- 172.21.15.107:0/1701852073 wait complete. 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:16.376+0000 7fb7da5b5700 1 librados: shutdown 2024-02-10T08:40:17.786 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:40:17.813 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:17.413321+0000 mgr.y (mgr.14553) 307 : cluster [DBG] pgmap v461: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:17.998734+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-10T08:40:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:17.413321+0000 mgr.y (mgr.14553) 307 : cluster [DBG] pgmap v461: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:17.998734+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-10T08:40:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:17.413321+0000 mgr.y (mgr.14553) 307 : cluster [DBG] pgmap v461: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:17.998734+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-10T08:40:19.395 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:40:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:40:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:40:19.395 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:40:19.395 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:40:20.164 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:20.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:20.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:18.994677+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:19.000299+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:19.000672+0000 mon.a (mon.0) 2132 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:19.413705+0000 mgr.y (mgr.14553) 308 : cluster [DBG] pgmap v464: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:18.994677+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:19.000299+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:19.000672+0000 mon.a (mon.0) 2132 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:19.413705+0000 mgr.y (mgr.14553) 308 : cluster [DBG] pgmap v464: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:20.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:20.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:18.994677+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-10T08:40:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:19.000299+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:19.000672+0000 mon.a (mon.0) 2132 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:20.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:19.413705+0000 mgr.y (mgr.14553) 308 : cluster [DBG] pgmap v464: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:20.298 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:20.319 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:20.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:20.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:20.451 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:20.470 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:20.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:20.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:40:20.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:20.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:40:20.857 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:20.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:20.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:20.996 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- 172.21.15.107:0/282658966 >> v1:172.21.15.107:6789/0 conn(0x7fa4340f89d0 legacy=0x7fa4340f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- 172.21.15.107:0/282658966 shutdown_connections 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- 172.21.15.107:0/282658966 >> 172.21.15.107:0/282658966 conn(0x7fa4340f2540 msgr2=0x7fa4340f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- 172.21.15.107:0/282658966 shutdown_connections 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- 172.21.15.107:0/282658966 wait complete. 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 librados: starting msgr at 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 librados: starting objecter 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 Processor -- start 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- start start 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 librados: setting wanted keys 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 librados: calling monclient init 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa434117380 con 0x7fa4340f89d0 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa434118560 con 0x7fa434104360 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.081+0000 7fa43b21a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa434119760 con 0x7fa4340f9580 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa433fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa4340f9580 0x7fa4340ffbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57936/0 (socket says 172.21.15.107:57936) 2024-02-10T08:40:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa438fb6700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4340f89d0 0x7fa4340ff4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33298/0 (socket says 172.21.15.107:33298) 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa433fff700 1 -- 172.21.15.107:0/2786828911 learned_addr learned my addr 172.21.15.107:0/2786828911 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter ms_handle_connect 0x7fa4340f89d0 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter ms_handle_connect 0x7fa4340f9580 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter ms_handle_connect 0x7fa434104360 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474743186 0 0) 0x7fa434119760 con 0x7fa4340f9580 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4240033f0 con 0x7fa4340f9580 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384168004 0 0) 0x7fa434118560 con 0x7fa434104360 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa434119760 con 0x7fa434104360 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923350645 0 0) 0x7fa434117380 con 0x7fa4340f89d0 2024-02-10T08:40:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa434118560 con 0x7fa4340f89d0 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2634517785 0 0) 0x7fa4240033f0 con 0x7fa4340f9580 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa434117380 con 0x7fa4340f9580 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa420002910 con 0x7fa4340f9580 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342323680 0 0) 0x7fa434119760 con 0x7fa434104360 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4240033f0 con 0x7fa434104360 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa42c001c30 con 0x7fa434104360 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042381593 0 0) 0x7fa434118560 con 0x7fa4340f89d0 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa434119760 con 0x7fa4340f89d0 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa428002050 con 0x7fa4340f89d0 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2023215554 0 0) 0x7fa434117380 con 0x7fa4340f9580 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 >> v1:172.21.15.195:6789/0 conn(0x7fa434104360 legacy=0x7fa434115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 >> v1:172.21.15.107:6789/0 conn(0x7fa4340f89d0 legacy=0x7fa4340ff4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43411a960 con 0x7fa4340f9580 2024-02-10T08:40:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa434118750 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa4200017d0 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.082+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa420005ab0 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.083+0000 7fa43b21a700 10 client.53399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.083+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa434118bc0 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.083+0000 7fa43b21a700 1 librados: init done 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4340603f0 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa420018a00 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(327..327 src has 50..327) v4 ==== 8001+0+0 (unknown 3865831895 0 0) 0x7fa420078d60 con 0x7fa4340f9580 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa431ffb700 10 client.53399.objecter ms_dispatch 0x7fa434001550 osd_map(327..327 src has 50..327) v4 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa431ffb700 3 client.53399.objecter handle_osd_map got epochs [327,327] > 0 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.084+0000 7fa431ffb700 3 client.53399.objecter handle_osd_map decoding full epoch 327 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.085+0000 7fa431ffb700 20 client.53399.objecter dump_active .. 0 homeless 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.085+0000 7fa431ffb700 10 client.53399.objecter ms_handle_connect 0x7fa42406c560 2024-02-10T08:40:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.089+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa42004c1f0 con 0x7fa4340f9580 2024-02-10T08:40:21.125 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:19.999919+0000 mon.a (mon.0) 2133 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:19.999991+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:20.002903+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:20.003171+0000 mon.a (mon.0) 2135 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:20.985314+0000 mon.a (mon.0) 2136 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:20.985360+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-10T08:40:21.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:19.999919+0000 mon.a (mon.0) 2133 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:19.999991+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-10T08:40:21.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:20.002903+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:20.003171+0000 mon.a (mon.0) 2135 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:20.985314+0000 mon.a (mon.0) 2136 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:20.985360+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-10T08:40:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:21.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:21.183 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:21.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:21.256 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:21.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:19.999919+0000 mon.a (mon.0) 2133 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:19.999991+0000 mon.a (mon.0) 2134 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-10T08:40:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:20.002903+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.107:0/4067602482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:20.003171+0000 mon.a (mon.0) 2135 : audit [INF] from='client.53375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]: dispatch 2024-02-10T08:40:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:20.985314+0000 mon.a (mon.0) 2136 : audit [INF] from='client.53375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi107-112411-56"}]': finished 2024-02-10T08:40:21.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:20.985360+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-10T08:40:21.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:21.300+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fa4340475e0 con 0x7fa4340f9580 2024-02-10T08:40:21.381 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:21.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:21.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:21.515 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:21.616 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:21.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:21.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:21.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:21.897 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:40:21.897 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:40:21.897 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:40:21.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:40:22.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.008+0000 7fa431ffb700 1 -- 172.21.15.107:0/2786828911 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v328) v1 ==== 91+0+0 (unknown 1978542059 0 0) 0x7fa42004a600 con 0x7fa4340f9580 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 10 librados: watch_flush enter 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 10 librados: watch_flush exit 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 20 client.53399.objecter shutdown clearing up homeless session... 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 10 client.53399.objecter successfully canceled tick 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa42406c560 legacy=0x7fa42406ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 >> v1:172.21.15.107:6790/0 conn(0x7fa4340f9580 legacy=0x7fa4340ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 shutdown_connections 2024-02-10T08:40:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.011+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 >> 172.21.15.107:0/2786828911 conn(0x7fa4340f2540 msgr2=0x7fa4341077f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.012+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 shutdown_connections 2024-02-10T08:40:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.012+0000 7fa43b21a700 1 -- 172.21.15.107:0/2786828911 wait complete. 2024-02-10T08:40:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:22.012+0000 7fa43b21a700 1 librados: shutdown 2024-02-10T08:40:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.011769+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.012307+0000 mon.a (mon.0) 2138 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.013265+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.013474+0000 mon.a (mon.0) 2139 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.013945+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.014156+0000 mon.a (mon.0) 2140 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.302178+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.107:0/2786828911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.302449+0000 mon.a (mon.0) 2141 : audit [INF] from='client.53399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:21.359009+0000 mgr.y (mgr.14553) 309 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:21.414506+0000 mgr.y (mgr.14553) 310 : cluster [DBG] pgmap v467: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:22.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.011769+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.012307+0000 mon.a (mon.0) 2138 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.013265+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.013474+0000 mon.a (mon.0) 2139 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.013945+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.014156+0000 mon.a (mon.0) 2140 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.302178+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.107:0/2786828911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.302449+0000 mon.a (mon.0) 2141 : audit [INF] from='client.53399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:21.359009+0000 mgr.y (mgr.14553) 309 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:21.414506+0000 mgr.y (mgr.14553) 310 : cluster [DBG] pgmap v467: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.011769+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.012307+0000 mon.a (mon.0) 2138 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.013265+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.013474+0000 mon.a (mon.0) 2139 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.013945+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.014156+0000 mon.a (mon.0) 2140 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.302178+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.107:0/2786828911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.302449+0000 mon.a (mon.0) 2141 : audit [INF] from='client.53399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:21.359009+0000 mgr.y (mgr.14553) 309 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:22.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:21.414506+0000 mgr.y (mgr.14553) 310 : cluster [DBG] pgmap v467: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:21.998310+0000 mon.a (mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:22.008991+0000 mon.a (mon.0) 2143 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:22.009077+0000 mon.a (mon.0) 2144 : audit [INF] from='client.53399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:40:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:22.009126+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-10T08:40:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:22.010639+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:22.011242+0000 mon.a (mon.0) 2146 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:22.230904+0000 osd.1 (osd.1) 145 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:40:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:22.234266+0000 osd.1 (osd.1) 146 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:40:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:22.356797+0000 osd.2 (osd.2) 109 : cluster [DBG] 16.e scrub starts 2024-02-10T08:40:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:22.359698+0000 osd.2 (osd.2) 110 : cluster [DBG] 16.e scrub ok 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:21.998310+0000 mon.a (mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:22.008991+0000 mon.a (mon.0) 2143 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:22.009077+0000 mon.a (mon.0) 2144 : audit [INF] from='client.53399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:22.009126+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:22.010639+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:22.011242+0000 mon.a (mon.0) 2146 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:22.230904+0000 osd.1 (osd.1) 145 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:22.234266+0000 osd.1 (osd.1) 146 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:22.356797+0000 osd.2 (osd.2) 109 : cluster [DBG] 16.e scrub starts 2024-02-10T08:40:23.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:22.359698+0000 osd.2 (osd.2) 110 : cluster [DBG] 16.e scrub ok 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:21.998310+0000 mon.a (mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:22.008991+0000 mon.a (mon.0) 2143 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi107-112411-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:22.009077+0000 mon.a (mon.0) 2144 : audit [INF] from='client.53399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:22.009126+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:22.010639+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:22.011242+0000 mon.a (mon.0) 2146 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:22.230904+0000 osd.1 (osd.1) 145 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:22.234266+0000 osd.1 (osd.1) 146 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:22.356797+0000 osd.2 (osd.2) 109 : cluster [DBG] 16.e scrub starts 2024-02-10T08:40:23.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:22.359698+0000 osd.2 (osd.2) 110 : cluster [DBG] 16.e scrub ok 2024-02-10T08:40:24.000 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:24.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:24.135 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:24.156 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:24.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.251942+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-10T08:40:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.342072+0000 osd.2 (osd.2) 111 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:40:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.345534+0000 osd.2 (osd.2) 112 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:40:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.414923+0000 mgr.y (mgr.14553) 311 : cluster [DBG] pgmap v470: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:24.289 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:24.309 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:24.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.251942+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.342072+0000 osd.2 (osd.2) 111 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.345534+0000 osd.2 (osd.2) 112 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.414923+0000 mgr.y (mgr.14553) 311 : cluster [DBG] pgmap v470: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.251942+0000 mon.a (mon.0) 2147 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.342072+0000 osd.2 (osd.2) 111 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:40:24.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.345534+0000 osd.2 (osd.2) 112 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:40:24.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.414923+0000 mgr.y (mgr.14553) 311 : cluster [DBG] pgmap v470: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:40:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:24.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:24.614 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:24.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:24.850 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:24.872 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:24.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34379.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94862968492128] snapc 0=[] ondisk+write+known_if_redirected e60)", 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 422.82887921000003, 2024-02-10T08:40:24.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020196910000000001, 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34379", 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1288525501", 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 64 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138536+0000", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138534+0000", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138541+0000", 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0132999999999999e-05 2024-02-10T08:40:24.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138552+0000", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6989999999999997e-06 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138558+0000", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040325199999999999 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.138961+0000", 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1836999999999996e-05 2024-02-10T08:40:24.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139043+0000", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047533599999999998 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139519+0000", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5971999999999999e-05 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139575+0000", 2024-02-10T08:40:24.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114123 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139689+0000", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026445499999999997 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.139953+0000", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033451599999999998 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140288+0000", 2024-02-10T08:40:24.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000182449 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140470+0000", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9476999999999999e-05 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140500+0000", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5571e-05 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:22.140555+0000", 2024-02-10T08:40:24.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020196910000000001 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24391.0:84 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e63)", 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 419.78164754900001, 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021074520000000001, 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24391", 2024-02-10T08:40:24.977 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4187249279", 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 84 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185767+0000", 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185766+0000", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4710000000000007e-06 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185774+0000", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2590000000000006e-06 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185783+0000", 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5350000000000004e-06 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185789+0000", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2379999999999996e-05 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185851+0000", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.185950+0000", 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017519500000000001 2024-02-10T08:40:24.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186125+0000", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037316999999999998 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186498+0000", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039394499999999998 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.186892+0000", 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093651699999999999 2024-02-10T08:40:24.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187828+0000", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5956999999999999e-05 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187854+0000", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0312000000000001e-05 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.187875+0000", 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021074520000000001 2024-02-10T08:40:24.982 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.24391.0:286 4.1a e63/57 4:5ae4284b:::obj_delete_at_hint.0000000032:head v 63'6, mlcod=63'4)", 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 419.69751455800002, 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021239140000000002, 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:40:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269900+0000", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269899+0000", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9990000000000008e-06 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269907+0000", 2024-02-10T08:40:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1367e-05 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269918+0000", 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0509999999999996e-06 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269923+0000", 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8180000000000002e-05 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.269981+0000", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8329e-05 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.270000+0000", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001982045 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.271982+0000", 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2318999999999999e-05 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:33:25.272024+0000", 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021239140000000002 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:17 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [write 0~3] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.07346584999999, 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057249742999999999, 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:24.987 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:24.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893947+0000", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9770000000000006e-06 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893956+0000", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2445e-05 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893968+0000", 2024-02-10T08:40:24.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8250000000000005e-06 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893976+0000", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0333000000000001e-05 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.894026+0000", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6769999999999998e-06 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.894033+0000", 2024-02-10T08:40:24.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047954573 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.941987+0000", 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014102099999999999 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.942129+0000", 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016330500000000001 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:40:24.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.942292+0000", 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005302813 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.947595+0000", 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033408940000000001 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.950936+0000", 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016446899999999999 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951100+0000", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3541000000000001e-05 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951154+0000", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.511e-05 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951199+0000", 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057249742999999999 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:2 52.14 e113/112 52:28e9c4a8:test-rados-api-smithi107-112562-5::foo:head v 113'1, mlcod=0'0)", 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 218.00866490600001, 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040456570000000002, 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958749+0000", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022999999999999e-05 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958760+0000", 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4339999999999997e-06 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958767+0000", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1820000000000003e-06 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958773+0000", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011023809999999999 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.959875+0000", 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1713e-05 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.959897+0000", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028442839999999999 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962741+0000", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4208e-05 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962795+0000", 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040456570000000002 2024-02-10T08:40:24.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(51.6s1 113/111 ECSubWrite(tid=1200, reqid=client.35219.0:7, at_version=113'3, trim_to=0'0, roll_forward_to=112'1))", 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.938914743, 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054912240000000003, 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:40:24.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028497+0000", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3481e-05 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028530+0000", 2024-02-10T08:40:24.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0550000000000001e-05 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028551+0000", 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7580000000000004e-06 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028561+0000", 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3418999999999998e-05 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028614+0000", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028623+0000", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053399140000000003 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.033963+0000", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8592e-05 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.033991+0000", 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054912240000000003 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:19 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e114)", 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.029846781, 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030457504999999999, 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:25.002 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 19 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:25.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937566+0000", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0020000000000003e-06 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937572+0000", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5529999999999993e-06 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937581+0000", 2024-02-10T08:40:25.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2579999999999998e-06 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937588+0000", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062451119999999997 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.943833+0000", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4054000000000004e-05 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.943927+0000", 2024-02-10T08:40:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021535700000000001 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.944143+0000", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028311209999999998 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946974+0000", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067872269999999998 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953761+0000", 2024-02-10T08:40:25.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01420008 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967961+0000", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3223999999999999e-05 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967994+0000", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1270999999999999e-05 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968025+0000", 2024-02-10T08:40:25.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030457504999999999 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.24889.0:19 23.d e114/102 ondisk, result = 0)", 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.018088205, 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044369580000000004, 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949326+0000", 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9810000000000003e-06 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949331+0000", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1840000000000005e-06 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949340+0000", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0509999999999999e-06 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949347+0000", 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043975309999999997 2024-02-10T08:40:25.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953744+0000", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8240000000000006e-06 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953753+0000", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0251999999999999e-05 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953763+0000", 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044369580000000004 2024-02-10T08:40:25.011 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(50.7s2 114/111 ECSubWrite(tid=148, reqid=client.35222.0:14, at_version=114'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 216.895341476, 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019889280000000001, 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072072+0000", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7529000000000001e-05 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072089+0000", 2024-02-10T08:40:25.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2250000000000006e-06 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072099+0000", 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6750000000000001e-06 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072103+0000", 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4253999999999999e-05 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072158+0000", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1949999999999997e-06 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072165+0000", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001880217 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.074045+0000", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7237000000000001e-05 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.074062+0000", 2024-02-10T08:40:25.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019889280000000001 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:21 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.818849086, 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022309029999999998, 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:40:25.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 21 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148562+0000", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3216e-05 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148575+0000", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0112999999999999e-05 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148595+0000", 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0395e-05 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148606+0000", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0525999999999995e-05 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148676+0000", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010333299999999999 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148780+0000", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000200175 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:40:25.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148980+0000", 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091076999999999998 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.149891+0000", 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00075057699999999999 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150641+0000", 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6660999999999998e-05 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150698+0000", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.291e-05 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150751+0000", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5816999999999999e-05 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150797+0000", 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022309029999999998 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.021 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25153.0:8 41.5 e115/107 41:a378f8be:::oid_PoolQuotaPP_5:head v 115'2, mlcod=0'0)", 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.17885628799999, 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013390228000000001, 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788556+0000", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788573+0000", 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0271e-05 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788593+0000", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1259999999999998e-06 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788602+0000", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6830999999999995e-05 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788698+0000", 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3570999999999998e-05 2024-02-10T08:40:25.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788722+0000", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013167371000000001 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.801889+0000", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9398000000000003e-05 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.801949+0000", 2024-02-10T08:40:25.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013390228000000001 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(osd.0.0:283 23.d e116/102 ondisk, result = 0)", 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.57777047100001, 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021669789999999999, 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389643+0000", 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2249999999999999e-06 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389649+0000", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8700000000000003e-06 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389656+0000", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9599999999999997e-06 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389661+0000", 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021301509999999998 2024-02-10T08:40:25.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391792+0000", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3749999999999992e-06 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391801+0000", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0264e-05 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391811+0000", 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021669789999999999 2024-02-10T08:40:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=251, reqid=client.35273.0:1, at_version=117'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 214.006681308, 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012908147999999999, 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.402e-05 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960757+0000", 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6120000000000004e-06 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960764+0000", 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.968e-06 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960771+0000", 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00065651399999999997 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.961427+0000", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6250000000000002e-06 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.961437+0000", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012188005999999999 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.973625+0000", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6701000000000002e-05 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.973642+0000", 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012908147999999999 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:4 52.1d 52:bb1f5efa:test-rados-api-smithi107-112562-7::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e117)", 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.995564621, 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035486110000000001, 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971847+0000", 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4769999999999999e-05 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971862+0000", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4911e-05 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971877+0000", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.893e-06 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971887+0000", 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034160739999999998 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975303+0000", 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5656999999999996e-05 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975399+0000", 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035486110000000001 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:5 52.1d 52:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e117)", 2024-02-10T08:40:25.037 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.99551301599999, 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035821059999999998, 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971901+0000", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4009999999999996e-06 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971906+0000", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2639999999999997e-06 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971913+0000", 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0470000000000004e-06 2024-02-10T08:40:25.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971917+0000", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035043299999999999 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975421+0000", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2872000000000004e-05 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975484+0000", 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035821059999999998 2024-02-10T08:40:25.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=252, reqid=client.35273.0:1, at_version=0'0, trim_to=0'0, roll_forward_to=117'1))", 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.986586149, 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016757220999999999, 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980827+0000", 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2146e-05 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980839+0000", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1671000000000001e-05 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980851+0000", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1720000000000001e-06 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980858+0000", 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4196999999999995e-05 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980922+0000", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1598e-05 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980934+0000", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016631714999999998 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997566+0000", 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0282e-05 2024-02-10T08:40:25.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997586+0000", 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016757220999999999 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=253, reqid=client.35273.0:2, at_version=117'2, trim_to=0'0, roll_forward_to=117'1))", 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 213.98167913899999, 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011863251, 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:40:25.045 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985734+0000", 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3638e-05 2024-02-10T08:40:25.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985758+0000", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5441e-05 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985773+0000", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7770000000000002e-06 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985781+0000", 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051802100000000002 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986299+0000", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0242e-05 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986309+0000", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011282307 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997592+0000", 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1210000000000001e-06 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997599+0000", 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011863251 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26977.0:3 66.17 66:ef3fc2a1:test-rados-api-smithi107-112525-8::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e121)", 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.893602571, 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013570198, 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:25.049 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26977", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073810+0000", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9479999999999997e-06 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073819+0000", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0499999999999999e-05 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073830+0000", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2830000000000004e-06 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073837+0000", 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001023461 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.074860+0000", 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012824000000000001 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.074989+0000", 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000145761 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.075134+0000", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012714499999999999 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.076406+0000", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010855439999999999 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087261+0000", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0412999999999998e-05 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:40:25.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087292+0000", 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0078999999999999e-05 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087342+0000", 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0708999999999999e-05 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087382+0000", 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013570198 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.26977.0:3 66.17 e121/119 ondisk, result = 0)", 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.88655559200001, 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065275589999999996, 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080858+0000", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.896e-06 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080864+0000", 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3210000000000005e-06 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080872+0000", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2320000000000004e-06 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080879+0000", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063623159999999998 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087242+0000", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0676000000000001e-05 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087252+0000", 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013427 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087387+0000", 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065275589999999996 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.15969.0:2 67.6 e121/120 67:602f83fe:::foo:head v 121'1, mlcod=0'0)", 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 209.87981252599999, 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042885639999999999, 2024-02-10T08:40:25.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:40:25.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7630000000000003e-06 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087608+0000", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0780000000000001e-06 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087615+0000", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3210000000000001e-06 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:40:25.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087621+0000", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055538500000000004 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.088176+0000", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6487e-05 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.088193+0000", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003665031 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.091858+0000", 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3052999999999997e-05 2024-02-10T08:40:25.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.091891+0000", 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042885639999999999 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:25.062 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:40:25.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:25.092 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:25.241 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:25.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.403222+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.406223+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.815174+0000 osd.5 (osd.5) 149 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.818427+0000 osd.5 (osd.5) 150 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.859136+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:23.860523+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:24.022843+0000 mon.a (mon.0) 2148 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.022921+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.162+0000 7f66dcac0700 1 -- 172.21.15.107:0/2242690729 >> v1:172.21.15.107:6789/0 conn(0x7f66d80f9570 legacy=0x7f66d8102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.162+0000 7f66dcac0700 1 -- 172.21.15.107:0/2242690729 shutdown_connections 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.162+0000 7f66dcac0700 1 -- 172.21.15.107:0/2242690729 >> 172.21.15.107:0/2242690729 conn(0x7f66d80f2510 msgr2=0x7f66d80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.162+0000 7f66dcac0700 1 -- 172.21.15.107:0/2242690729 shutdown_connections 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.162+0000 7f66dcac0700 1 -- 172.21.15.107:0/2242690729 wait complete. 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 librados: starting msgr at 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 librados: starting objecter 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 Processor -- start 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 -- start start 2024-02-10T08:40:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 librados: setting wanted keys 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 librados: calling monclient init 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66d81194e0 con 0x7f66d80f9570 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66d811a6c0 con 0x7f66d8104350 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66dcac0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66d811b8c0 con 0x7f66d80f89c0 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66d5d9b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f66d80f9570 0x7f66d81029b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33326/0 (socket says 172.21.15.107:33326) 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.163+0000 7f66d5d9b700 1 -- 172.21.15.107:0/3394359245 learned_addr learned my addr 172.21.15.107:0/3394359245 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter ms_handle_connect 0x7f66d80f9570 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter ms_handle_connect 0x7f66d8104350 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter ms_handle_connect 0x7f66d80f89c0 2024-02-10T08:40:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781598286 0 0) 0x7f66d81194e0 con 0x7f66d80f9570 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c0003800 con 0x7f66d80f9570 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010281976 0 0) 0x7f66d811a6c0 con 0x7f66d8104350 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d81194e0 con 0x7f66d8104350 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399507295 0 0) 0x7f66d811b8c0 con 0x7f66d80f89c0 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d811a6c0 con 0x7f66d80f89c0 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171906886 0 0) 0x7f66d81194e0 con 0x7f66d8104350 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d811b8c0 con 0x7f66d8104350 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66d0001d10 con 0x7f66d8104350 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 499135271 0 0) 0x7f66d811a6c0 con 0x7f66d80f89c0 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d81194e0 con 0x7f66d80f89c0 2024-02-10T08:40:25.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589545624 0 0) 0x7f66c0003800 con 0x7f66d80f9570 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d811a6c0 con 0x7f66d80f9570 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66c8000fe0 con 0x7f66d80f89c0 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66cc002f60 con 0x7f66d80f9570 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575798148 0 0) 0x7f66d811b8c0 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6790/0 conn(0x7f66d80f89c0 legacy=0x7f66d81022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6789/0 conn(0x7f66d80f9570 legacy=0x7f66d81029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d811cac0 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d811a8b0 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66dcac0700 10 client.48124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d811ad20 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66dcac0700 1 librados: init done 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f66d0002680 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.164+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66d0003700 con 0x7f66d8104350 2024-02-10T08:40:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.165+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f66b8004f00 con 0x7f66d8104350 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f66d00021e0 con 0x7f66d8104350 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(331..331 src has 50..331) v4 ==== 8813+0+0 (unknown 121240438 0 0) 0x7f66d00769a0 con 0x7f66d8104350 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 osd_map(331..331 src has 50..331) v4 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 3 client.48124.objecter handle_osd_map got epochs [331,331] > 0 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 3 client.48124.objecter handle_osd_map decoding full epoch 331 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 20 client.48124.objecter dump_active .. 0 homeless 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v331) v1 ==== 54+0+16 (unknown 1209959077 0 2420000192) 0x7f66d004d020 con 0x7f66d8104350 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.166+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66c0063dc0 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 10 client.48124.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 1 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 1 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 0 1 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 4 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 10 client.48124.objecter _send_command 1 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b8004950 con 0x7f66b8004530 2024-02-10T08:40:25.336 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b8004530 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b8004950 con 0x7f66b8004530 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 10 client.48124.objecter _finish_command 1 = system:0 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 4 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.167+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 0 1 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 10 client.48124.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 2 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 2 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b80051a0 osd=0 3 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 0 2 2024-02-10T08:40:25.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 4 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 10 client.48124.objecter _send_command 2 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.191+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b8004d30 con 0x7f66b8004530 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b8004d30 con 0x7f66b8004530 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 10 client.48124.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 4 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 0 2 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 10 client.48124.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 3 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 3 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 1 3 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 4 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 10 client.48124.objecter _send_command 3 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b800c000 con 0x7f66b8004150 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.193+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b8004150 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.194+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b800c000 con 0x7f66b8004150 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.194+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.194+0000 7f66c77fe700 10 client.48124.objecter _finish_command 3 = system:0 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.194+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 4 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.194+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 1 3 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 10 client.48124.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 4 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 4 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8003fe0 osd=1 3 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 1 4 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 4 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 10 client.48124.objecter _send_command 4 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.209+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b80018b0 con 0x7f66b8004150 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b80018b0 con 0x7f66b8004150 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 10 client.48124.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 4 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 1 4 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 10 client.48124.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 5 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 5 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 2 5 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 4 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 10 client.48124.objecter _send_command 5 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b800de90 con 0x7f66b800d2f0 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.211+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b800d2f0 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.212+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b800de90 con 0x7f66b800d2f0 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.212+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.212+0000 7f66c77fe700 10 client.48124.objecter _finish_command 5 = system:0 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.212+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 4 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.212+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 2 5 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 10 client.48124.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 6 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 6 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8003630 osd=2 3 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 2 6 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 4 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 10 client.48124.objecter _send_command 6 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.226+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b80030a0 con 0x7f66b800d2f0 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b80030a0 con 0x7f66b800d2f0 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 10 client.48124.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 4 2024-02-10T08:40:25.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 2 6 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 10 client.48124.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 7 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 7 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 3 7 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 4 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 10 client.48124.objecter _send_command 7 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b8015b00 con 0x7f66b8011ff0 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.228+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b8011ff0 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.229+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b8015b00 con 0x7f66b8011ff0 2024-02-10T08:40:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.229+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.229+0000 7f66c77fe700 10 client.48124.objecter _finish_command 7 = system:0 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.229+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 4 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.229+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 3 7 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 10 client.48124.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 8 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 8 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8011970 osd=3 3 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 3 8 2024-02-10T08:40:25.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 4 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 10 client.48124.objecter _send_command 8 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.245+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b800e240 con 0x7f66b8011ff0 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b800e240 con 0x7f66b8011ff0 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66c77fe700 10 client.48124.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 4 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 3 8 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 10 client.48124.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 9 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 9 2024-02-10T08:40:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 4 9 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 4 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 10 client.48124.objecter _send_command 9 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.247+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b800e5b0 con 0x7f66b800ce50 2024-02-10T08:40:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b800ce50 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b800e5b0 con 0x7f66b800ce50 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 10 client.48124.objecter _finish_command 9 = system:0 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 4 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.248+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 4 9 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 10 client.48124.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 10 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 10 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b800e410 osd=4 3 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 4 10 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 4 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 10 client.48124.objecter _send_command 10 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.265+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b800e5b0 con 0x7f66b800ce50 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.266+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b800e5b0 con 0x7f66b800ce50 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.266+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.266+0000 7f66c77fe700 10 client.48124.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 4 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 4 10 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 10 client.48124.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 11 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 11 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 5 11 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 4 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 10 client.48124.objecter _send_command 11 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b801e5e0 con 0x7f66b801a900 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.267+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b801a900 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.268+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b801e5e0 con 0x7f66b801a900 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.268+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.268+0000 7f66c77fe700 10 client.48124.objecter _finish_command 11 = system:0 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.268+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 4 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.268+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 5 11 2024-02-10T08:40:25.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 10 client.48124.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 12 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 12 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b800e5b0 osd=5 3 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 5 12 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 4 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 10 client.48124.objecter _send_command 12 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.282+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b801fd90 con 0x7f66b801a900 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b801fd90 con 0x7f66b801a900 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 10 client.48124.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 4 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 5 12 2024-02-10T08:40:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 10 client.48124.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 13 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 13 2024-02-10T08:40:25.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 6 13 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 4 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 10 client.48124.objecter _send_command 13 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b8020a00 con 0x7f66b8020450 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.285+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b8020450 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.286+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b8020a00 con 0x7f66b8020450 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.286+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.286+0000 7f66c77fe700 10 client.48124.objecter _finish_command 13 = system:0 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.286+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 4 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.286+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 6 13 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 10 client.48124.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 14 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 14 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b801fd90 osd=6 3 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 6 14 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 4 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 10 client.48124.objecter _send_command 14 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.300+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b8020a00 con 0x7f66b8020450 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b8020a00 con 0x7f66b8020450 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66c77fe700 10 client.48124.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 4 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 6 14 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 10 client.48124.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:25.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 15 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 15 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 7 15 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 4 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 10 client.48124.objecter _send_command 15 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.302+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b8026fb0 con 0x7f66b80234a0 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.303+0000 7f66c77fe700 10 client.48124.objecter ms_handle_connect 0x7f66b80234a0 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.304+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66b8026fb0 con 0x7f66b80234a0 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.304+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.304+0000 7f66c77fe700 10 client.48124.objecter _finish_command 15 = system:0 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.304+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 4 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.304+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 7 15 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 10 client.48124.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign -1 16 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 20 client.48124.objecter _get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_remove -1 16 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 20 client.48124.objecter get_session s=0x7f66b8020a00 osd=7 3 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 15 client.48124.objecter _session_command_op_assign 7 16 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 4 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 10 client.48124.objecter _send_command 16 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.318+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66b8026fb0 con 0x7f66b80234a0 2024-02-10T08:40:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66c77fe700 1 -- 172.21.15.107:0/3394359245 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66b8026fb0 con 0x7f66b80234a0 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66c77fe700 10 client.48124.objecter ms_dispatch 0x7f66d8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66c77fe700 10 client.48124.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66c77fe700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 4 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66c77fe700 15 client.48124.objecter _session_command_op_remove 7 16 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 librados: watch_flush enter 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 librados: watch_flush exit 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.0 2024-02-10T08:40:25.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6801/3321596972 conn(0x7f66b8004530 legacy=0x7f66b8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b80051a0 osd=0 1 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.1 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6805/133516126 conn(0x7f66b8004150 legacy=0x7f66b800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003fe0 osd=1 1 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.2 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6809/3575881728 conn(0x7f66b800d2f0 legacy=0x7f66b800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8003630 osd=2 1 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.3 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6813/2871825691 conn(0x7f66b8011ff0 legacy=0x7f66b80143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8011970 osd=3 1 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.4 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66d6d9d700 1 -- 172.21.15.107:0/3394359245 reap_dead start 2024-02-10T08:40:25.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.195:6800/2499671285 conn(0x7f66b800ce50 legacy=0x7f66b8018c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66d6d9d700 1 -- 172.21.15.107:0/3394359245 reap_dead start 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e410 osd=4 1 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.5 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.195:6804/4178321486 conn(0x7f66b801a900 legacy=0x7f66b801cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b800e5b0 osd=5 1 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.6 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.195:6808/1417479017 conn(0x7f66b8020450 legacy=0x7f66b801e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b801fd90 osd=6 1 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter close_session for osd.7 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.195:6812/1934176928 conn(0x7f66b80234a0 legacy=0x7f66b8025890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter put_session s=0x7f66b8020a00 osd=7 1 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 20 client.48124.objecter shutdown clearing up homeless session... 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 10 client.48124.objecter successfully canceled tick 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.107:6800/2634842009 conn(0x7f66c0063dc0 legacy=0x7f66c0066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> v1:172.21.15.195:6789/0 conn(0x7f66d8104350 legacy=0x7f66d8117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66d6d9d700 1 -- 172.21.15.107:0/3394359245 reap_dead start 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 shutdown_connections 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 >> 172.21.15.107:0/3394359245 conn(0x7f66d80f2510 msgr2=0x7f66d80f4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 shutdown_connections 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 -- 172.21.15.107:0/3394359245 wait complete. 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:25.320+0000 7f66dcac0700 1 librados: shutdown 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.403222+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.406223+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.815174+0000 osd.5 (osd.5) 149 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.818427+0000 osd.5 (osd.5) 150 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.859136+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:23.860523+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:24.022843+0000 mon.a (mon.0) 2148 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:25.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.022921+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.403222+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.406223+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.815174+0000 osd.5 (osd.5) 149 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.818427+0000 osd.5 (osd.5) 150 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.859136+0000 osd.3 (osd.3) 159 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:23.860523+0000 osd.3 (osd.3) 160 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:24.022843+0000 mon.a (mon.0) 2148 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi107-112411-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:25.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.022921+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-10T08:40:25.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:25.390 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:25.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:25.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:25.873 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:25] "GET /metrics HTTP/1.1" 200 189026 "" "Prometheus/2.33.4" 2024-02-10T08:40:25.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:40:26.222 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:26.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:26.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.391921+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.395537+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.581324+0000 osd.4 (osd.4) 144 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.583218+0000 osd.4 (osd.4) 145 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.825035+0000 osd.3 (osd.3) 161 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:24.826217+0000 osd.3 (osd.3) 162 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.025916+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:25.168194+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? v1:172.21.15.107:0/3394359245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.415332+0000 mgr.y (mgr.14553) 312 : cluster [DBG] pgmap v473: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:26.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.493983+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.494082+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.391921+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.395537+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.581324+0000 osd.4 (osd.4) 144 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.583218+0000 osd.4 (osd.4) 145 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.825035+0000 osd.3 (osd.3) 161 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:24.826217+0000 osd.3 (osd.3) 162 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.025916+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:25.168194+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? v1:172.21.15.107:0/3394359245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.415332+0000 mgr.y (mgr.14553) 312 : cluster [DBG] pgmap v473: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.493983+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:26.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.494082+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:26.353 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.391921+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.a scrub starts 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.395537+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.a scrub ok 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.581324+0000 osd.4 (osd.4) 144 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.583218+0000 osd.4 (osd.4) 145 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.825035+0000 osd.3 (osd.3) 161 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:24.826217+0000 osd.3 (osd.3) 162 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.025916+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:25.168194+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? v1:172.21.15.107:0/3394359245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.415332+0000 mgr.y (mgr.14553) 312 : cluster [DBG] pgmap v473: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.493983+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.494082+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:26.542 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:26.551 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:26.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:26.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:40:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:26.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:26.807 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:26.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:26.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:26.938 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:27.087 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:27.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:27.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.628205+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.e scrub starts 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:25.631844+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.e scrub ok 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:26.075832+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:26.085053+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:26.085437+0000 mon.a (mon.0) 2152 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:26.420199+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.b scrub starts 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:26.423187+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.b scrub ok 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.628205+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.e scrub starts 2024-02-10T08:40:27.214 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:25.631844+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.e scrub ok 2024-02-10T08:40:27.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:26.075832+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-10T08:40:27.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:26.085053+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:26.085437+0000 mon.a (mon.0) 2152 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:26.420199+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.b scrub starts 2024-02-10T08:40:27.215 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:26.423187+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.b scrub ok 2024-02-10T08:40:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.628205+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.e scrub starts 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:25.631844+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.e scrub ok 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:26.075832+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:26.085053+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:26.085437+0000 mon.a (mon.0) 2152 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:26.420199+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.b scrub starts 2024-02-10T08:40:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:26.423187+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.b scrub ok 2024-02-10T08:40:27.669 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:27.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:27.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:28.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:26.856689+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub starts 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:26.859694+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.a scrub ok 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:27.080112+0000 mon.a (mon.0) 2153 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.080179+0000 mon.a (mon.0) 2154 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:27.081757+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:27.082184+0000 mon.a (mon.0) 2155 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.415881+0000 mgr.y (mgr.14553) 313 : cluster [DBG] pgmap v476: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.637448+0000 osd.4 (osd.4) 148 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.638300+0000 osd.4 (osd.4) 149 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:26.856689+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub starts 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:26.859694+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.a scrub ok 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:27.080112+0000 mon.a (mon.0) 2153 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.080179+0000 mon.a (mon.0) 2154 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:27.081757+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:27.082184+0000 mon.a (mon.0) 2155 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.415881+0000 mgr.y (mgr.14553) 313 : cluster [DBG] pgmap v476: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.637448+0000 osd.4 (osd.4) 148 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:40:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.638300+0000 osd.4 (osd.4) 149 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:26.856689+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub starts 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:26.859694+0000 osd.5 (osd.5) 152 : cluster [DBG] 6.a scrub ok 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:27.080112+0000 mon.a (mon.0) 2153 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.080179+0000 mon.a (mon.0) 2154 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:27.081757+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.107:0/4089636740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:27.082184+0000 mon.a (mon.0) 2155 : audit [INF] from='client.53393 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]: dispatch 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.415881+0000 mgr.y (mgr.14553) 313 : cluster [DBG] pgmap v476: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.637448+0000 osd.4 (osd.4) 148 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:40:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.638300+0000 osd.4 (osd.4) 149 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:28.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:29.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:29.082 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:29.158 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:29.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:29.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:29.291 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:29.311 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:29.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:29.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.388564+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.388805+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.810129+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:27.813392+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.085021+0000 mon.a (mon.0) 2156 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.085094+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-10T08:40:29.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.101529+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.102604+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.102878+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.183039+0000 osd.1 (osd.1) 147 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.186480+0000 osd.1 (osd.1) 148 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.280423+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.280855+0000 mon.a (mon.0) 2161 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.281485+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:28.281727+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.459767+0000 osd.2 (osd.2) 115 : cluster [DBG] 60.b scrub starts 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.463421+0000 osd.2 (osd.2) 116 : cluster [DBG] 60.b scrub ok 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.388564+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.388805+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.810129+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:27.813392+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.085021+0000 mon.a (mon.0) 2156 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.085094+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.101529+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.102604+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.102878+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.183039+0000 osd.1 (osd.1) 147 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.186480+0000 osd.1 (osd.1) 148 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.280423+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.280855+0000 mon.a (mon.0) 2161 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.281485+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:28.281727+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.459767+0000 osd.2 (osd.2) 115 : cluster [DBG] 60.b scrub starts 2024-02-10T08:40:29.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.463421+0000 osd.2 (osd.2) 116 : cluster [DBG] 60.b scrub ok 2024-02-10T08:40:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.388564+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.388805+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.810129+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:27.813392+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.085021+0000 mon.a (mon.0) 2156 : audit [INF] from='client.53393 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi107-112411-57"}]': finished 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.085094+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.101529+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.102604+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.102878+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.183039+0000 osd.1 (osd.1) 147 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.186480+0000 osd.1 (osd.1) 148 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.280423+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.280855+0000 mon.a (mon.0) 2161 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.281485+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:28.281727+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]: dispatch 2024-02-10T08:40:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.459767+0000 osd.2 (osd.2) 115 : cluster [DBG] 60.b scrub starts 2024-02-10T08:40:29.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.463421+0000 osd.2 (osd.2) 116 : cluster [DBG] 60.b scrub ok 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.422851+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:28.425986+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:29.092018+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:29.092091+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]': finished 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:29.092144+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-10T08:40:30.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:29.094991+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:29.416460+0000 mgr.y (mgr.14553) 314 : cluster [DBG] pgmap v479: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:29.921247+0000 mon.c (mon.2) 358 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.422851+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:28.425986+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:29.092018+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:29.092091+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]': finished 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:29.092144+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:29.094991+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:29.416460+0000 mgr.y (mgr.14553) 314 : cluster [DBG] pgmap v479: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:30.355 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:29.921247+0000 mon.c (mon.2) 358 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:40:30.397 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:30.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:30.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:30.529 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.422851+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:28.425986+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:29.092018+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:29.092091+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-43"}]': finished 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:29.092144+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:29.094991+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:29.416460+0000 mgr.y (mgr.14553) 314 : cluster [DBG] pgmap v479: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T08:40:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:29.921247+0000 mon.c (mon.2) 358 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:40:30.780 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:30.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:30.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:30.912 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- 172.21.15.107:0/2783745323 >> v1:172.21.15.107:6789/0 conn(0x7fe08c0f9570 legacy=0x7fe08c102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- 172.21.15.107:0/2783745323 shutdown_connections 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- 172.21.15.107:0/2783745323 >> 172.21.15.107:0/2783745323 conn(0x7fe08c0f2510 msgr2=0x7fe08c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- 172.21.15.107:0/2783745323 shutdown_connections 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- 172.21.15.107:0/2783745323 wait complete. 2024-02-10T08:40:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 librados: starting msgr at 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 librados: starting objecter 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 Processor -- start 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 -- start start 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 librados: setting wanted keys 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.005+0000 7fe0917a5700 1 librados: calling monclient init 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe08c1173f0 con 0x7fe08c104350 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe08c1185d0 con 0x7fe08c0f9570 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe08c1197d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe08b7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe08c104350 0x7fe08c115ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45082/0 (socket says 172.21.15.107:45082) 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe08b7fe700 1 -- 172.21.15.107:0/2921118955 learned_addr learned my addr 172.21.15.107:0/2921118955 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe08affd700 1 --1- 172.21.15.107:0/2921118955 >> v1:172.21.15.107:6790/0 conn(0x7fe08c0f89c0 0x7fe08c0ff710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54322/0 (socket says 172.21.15.107:54322) 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter ms_handle_connect 0x7fe08c0f89c0 2024-02-10T08:40:31.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter ms_handle_connect 0x7fe08c104350 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter ms_handle_connect 0x7fe08c0f9570 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773664294 0 0) 0x7fe08c1173f0 con 0x7fe08c104350 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0780033f0 con 0x7fe08c104350 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463228364 0 0) 0x7fe08c1197d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c1173f0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087674893 0 0) 0x7fe08c1185d0 con 0x7fe08c0f9570 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c1197d0 con 0x7fe08c0f9570 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026792445 0 0) 0x7fe08c1173f0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe08c1185d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837103919 0 0) 0x7fe0780033f0 con 0x7fe08c104350 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe08c1173f0 con 0x7fe08c104350 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe074001a60 con 0x7fe08c0f89c0 2024-02-10T08:40:31.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133170556 0 0) 0x7fe08c1197d0 con 0x7fe08c0f9570 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0780033f0 con 0x7fe08c0f9570 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe084002c50 con 0x7fe08c104350 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe07c000fe0 con 0x7fe08c0f9570 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191295764 0 0) 0x7fe08c1185d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 >> v1:172.21.15.195:6789/0 conn(0x7fe08c0f9570 legacy=0x7fe08c0ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 >> v1:172.21.15.107:6789/0 conn(0x7fe08c104350 legacy=0x7fe08c115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c11a9d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c1199c0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 10 client.53417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c119e30 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.006+0000 7fe0917a5700 1 librados: init done 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.007+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe074002480 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.007+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe074005ab0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.007+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c004f20 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe074001fe0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(337..337 src has 50..337) v4 ==== 8798+0+0 (unknown 1327337566 0 0) 0x7fe074079000 con 0x7fe08c0f89c0 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 10 client.53417.objecter ms_dispatch 0x7fe08c001550 osd_map(337..337 src has 50..337) v4 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 3 client.53417.objecter handle_osd_map got epochs [337,337] > 0 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 3 client.53417.objecter handle_osd_map decoding full epoch 337 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.008+0000 7fe083fff700 20 client.53417.objecter dump_active .. 0 homeless 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.009+0000 7fe083fff700 10 client.53417.objecter ms_handle_connect 0x7fe078063c30 2024-02-10T08:40:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.012+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe07404c130 con 0x7fe08c0f89c0 2024-02-10T08:40:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:31.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.108+0000 7f90b6fe1700 1 -- 172.21.15.107:0/52498075 >> v1:172.21.15.107:6789/0 conn(0x7f90b00f9570 legacy=0x7f90b0102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 -- 172.21.15.107:0/52498075 shutdown_connections 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 -- 172.21.15.107:0/52498075 >> 172.21.15.107:0/52498075 conn(0x7f90b00f2510 msgr2=0x7f90b00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 -- 172.21.15.107:0/52498075 shutdown_connections 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 -- 172.21.15.107:0/52498075 wait complete. 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 librados: starting msgr at 2024-02-10T08:40:31.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 librados: starting objecter 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 Processor -- start 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 -- start start 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 librados: setting wanted keys 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.109+0000 7f90b6fe1700 1 librados: calling monclient init 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90b6fe1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90b01194e0 con 0x7f90b00f9570 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90b6fe1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90b011a6c0 con 0x7f90b0104350 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90b6fe1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90b011b8c0 con 0x7f90b00f89c0 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90affff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f90b00f9570 0x7f90b01029b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45092/0 (socket says 172.21.15.107:45092) 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90affff700 1 -- 172.21.15.107:0/2928702511 learned_addr learned my addr 172.21.15.107:0/2928702511 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter ms_handle_connect 0x7f90b00f9570 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter ms_handle_connect 0x7f90b0104350 2024-02-10T08:40:31.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824823201 0 0) 0x7f90b011a6c0 con 0x7f90b0104350 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003030 con 0x7f90b0104350 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657146275 0 0) 0x7f90b01194e0 con 0x7f90b00f9570 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b011a6c0 con 0x7f90b00f9570 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter ms_handle_connect 0x7f90b00f89c0 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 473385018 0 0) 0x7f90a0003030 con 0x7f90b0104350 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b01194e0 con 0x7f90b0104350 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f90a8001d70 con 0x7f90b0104350 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282401720 0 0) 0x7f90b011b8c0 con 0x7f90b00f89c0 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.110+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003030 con 0x7f90b00f89c0 2024-02-10T08:40:31.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 377647691 0 0) 0x7f90b01194e0 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 >> v1:172.21.15.107:6790/0 conn(0x7f90b00f89c0 legacy=0x7f90b01022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 >> v1:172.21.15.107:6789/0 conn(0x7f90b00f9570 legacy=0x7f90b01029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b011cac0 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b011a850 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f90a8002620 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f90a8003600 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90b6fe1700 10 client.48142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b011acc0 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.111+0000 7f90b6fe1700 1 librados: init done 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.112+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b005f7a0 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.112+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f90a8016550 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(337..337 src has 50..337) v4 ==== 8798+0+0 (unknown 1327337566 0 0) 0x7f90a8076b00 con 0x7f90b0104350 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 10 client.48142.objecter ms_dispatch 0x7f90b0001550 osd_map(337..337 src has 50..337) v4 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 3 client.48142.objecter handle_osd_map got epochs [337,337] > 0 2024-02-10T08:40:31.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 3 client.48142.objecter handle_osd_map decoding full epoch 337 2024-02-10T08:40:31.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 20 client.48142.objecter dump_active .. 0 homeless 2024-02-10T08:40:31.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.113+0000 7f90adffb700 10 client.48142.objecter ms_handle_connect 0x7f90a0063940 2024-02-10T08:40:31.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.117+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f90a8049c10 con 0x7f90b0104350 2024-02-10T08:40:31.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.189+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fe06c0051d0 con 0x7fe08c0f89c0 2024-02-10T08:40:31.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.191+0000 7fe083fff700 1 -- 172.21.15.107:0/2921118955 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v337) v1 ==== 81+0+50643 (unknown 2552587577 0 2344365148) 0x7fe07404a540 con 0x7fe08c0f89c0 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 337, 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:40:31.193 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:40:30.982012+0000", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:33:10.747466+0000", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:37:08.966515+0000", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5804032, 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 89, 2024-02-10T08:40:31.194 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 197, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:40:31.195 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.196 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.197 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "67", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.198 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.199 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.200 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:40:31.201 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.202 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:40:31.203 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.204 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.205 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.206 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.207 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.208 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.209 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.211 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.212 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.213 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.214 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.215 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.216 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:40:31.218 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.219 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.220 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.221 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:40:31.222 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.223 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.224 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.225 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:40:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 60, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-6", 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:49.742959+0000", 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "335", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "296", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "296", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "296", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 21, 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 318, 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 196, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "RoundTripWriteFullPP_smithi107-112411-58", 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:40:30.336855+0000", 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "337", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-RoundTripWriteFullPP_smithi107-112411-58", 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:40:31.242 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 197, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-45", 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:40:30.588310+0000", 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:40:31.243 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:40:31.244 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "337", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:40:31.245 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:40:31.246 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:40:31.247 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:40:31.248 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:40:31.249 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.250 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:40:31.251 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.253 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.254 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.255 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.256 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.257 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:40:31.258 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.259 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:40:31.260 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.261 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 316, 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.264 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.265 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.266 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:40:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.269 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.270 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:40:31.271 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.272 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.273 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.274 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:40:31.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.276 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.277 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:40:31.278 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 330, 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.279 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:40:31.280 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.281 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.282 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:25.494086+0000", 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:39:38.100377+0000", 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:16.317482+0000", 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:27.388809+0000", 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.285 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:39:39.820862+0000", 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:07.356961+0000", 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:39:33.813260+0000", 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:40:31.287 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:39:37.860373+0000", 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:40:31.288 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.18", 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 7, 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:40:31.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:40:31.292 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-RoundTripWriteFullPP_smithi107-112411-58": { 2024-02-10T08:40:31.293 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.294 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.297 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:40:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.305 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.306 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:40:31.307 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:40:31.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:31.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.192+0000 7fe0917a5700 10 librados: watch_flush enter 2024-02-10T08:40:31.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.192+0000 7fe0917a5700 10 librados: watch_flush exit 2024-02-10T08:40:31.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 20 client.53417.objecter shutdown clearing up homeless session... 2024-02-10T08:40:31.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 10 client.53417.objecter successfully canceled tick 2024-02-10T08:40:31.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe078063c30 legacy=0x7fe0780660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 >> v1:172.21.15.107:6790/0 conn(0x7fe08c0f89c0 legacy=0x7fe08c0ff710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 shutdown_connections 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 >> 172.21.15.107:0/2921118955 conn(0x7fe08c0f2510 msgr2=0x7fe08c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 shutdown_connections 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 -- 172.21.15.107:0/2921118955 wait complete. 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.193+0000 7fe0917a5700 1 librados: shutdown 2024-02-10T08:40:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.300+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f90b0109fe0 con 0x7f90b0104350 2024-02-10T08:40:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:31.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.336942+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-10T08:40:31.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:30.988109+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:31.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.988210+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-10T08:40:31.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:30.997165+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.002910+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.336942+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-10T08:40:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:30.988109+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.988210+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-10T08:40:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:30.997165+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.002910+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.336942+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-10T08:40:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:30.988109+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi107-112411-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.988210+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-10T08:40:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:30.997165+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.002910+0000 mon.a (mon.0) 2170 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:31.586 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:31.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:31.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:31.721 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:31.724 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:31.744 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:31.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:31.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:31.878 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:31.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:31.997+0000 7f90adffb700 1 -- 172.21.15.107:0/2928702511 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v338) v1 ==== 101+0+0 (unknown 2827711393 0 0) 0x7f90a8048020 con 0x7f90b0104350 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 10 librados: watch_flush enter 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 10 librados: watch_flush exit 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 20 client.48142.objecter shutdown clearing up homeless session... 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 10 client.48142.objecter successfully canceled tick 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 >> v1:172.21.15.107:6800/2634842009 conn(0x7f90a0063940 legacy=0x7f90a0065e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.000+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 >> v1:172.21.15.195:6789/0 conn(0x7f90b0104350 legacy=0x7f90b0117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.006+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 shutdown_connections 2024-02-10T08:40:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.006+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 >> 172.21.15.107:0/2928702511 conn(0x7f90b00f2510 msgr2=0x7f90b01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.007+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 shutdown_connections 2024-02-10T08:40:32.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.010+0000 7f90b6fe1700 1 -- 172.21.15.107:0/2928702511 wait complete. 2024-02-10T08:40:32.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:32.010+0000 7f90b6fe1700 1 librados: shutdown 2024-02-10T08:40:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:40:32.122 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:40:32.122 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:40:32.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.442674+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:40:32.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.446341+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:40:32.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.881360+0000 osd.3 (osd.3) 165 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:30.882869+0000 osd.3 (osd.3) 166 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.191575+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.107:0/2921118955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.302515+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.107:0/2928702511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.302848+0000 mon.a (mon.0) 2171 : audit [INF] from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.370031+0000 mgr.y (mgr.14553) 315 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:31.417313+0000 mgr.y (mgr.14553) 316 : cluster [DBG] pgmap v482: 226 pgs: 40 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.666144+0000 mon.c (mon.2) 360 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.666750+0000 mon.c (mon.2) 361 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.671304+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:31.984859+0000 mon.a (mon.0) 2173 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.996081+0000 mon.a (mon.0) 2174 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:31.996162+0000 mon.a (mon.0) 2175 : audit [INF] from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:31.996212+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-10T08:40:32.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.048159+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.048501+0000 mon.a (mon.0) 2177 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.442674+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.446341+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.881360+0000 osd.3 (osd.3) 165 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:30.882869+0000 osd.3 (osd.3) 166 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.191575+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.107:0/2921118955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.302515+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.107:0/2928702511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.302848+0000 mon.a (mon.0) 2171 : audit [INF] from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.370031+0000 mgr.y (mgr.14553) 315 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:31.417313+0000 mgr.y (mgr.14553) 316 : cluster [DBG] pgmap v482: 226 pgs: 40 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.666144+0000 mon.c (mon.2) 360 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.666750+0000 mon.c (mon.2) 361 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.671304+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:32.146 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:31.984859+0000 mon.a (mon.0) 2173 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:40:32.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.996081+0000 mon.a (mon.0) 2174 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:32.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:31.996162+0000 mon.a (mon.0) 2175 : audit [INF] from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:32.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:31.996212+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-10T08:40:32.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.048159+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.147 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.048501+0000 mon.a (mon.0) 2177 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.527 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:32.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.442674+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:40:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.446341+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:40:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.881360+0000 osd.3 (osd.3) 165 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:40:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:30.882869+0000 osd.3 (osd.3) 166 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:40:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.191575+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.107:0/2921118955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.302515+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.107:0/2928702511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.302848+0000 mon.a (mon.0) 2171 : audit [INF] from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.370031+0000 mgr.y (mgr.14553) 315 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:31.417313+0000 mgr.y (mgr.14553) 316 : cluster [DBG] pgmap v482: 226 pgs: 40 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.666144+0000 mon.c (mon.2) 360 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.666750+0000 mon.c (mon.2) 361 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.671304+0000 mon.a (mon.0) 2172 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:31.984859+0000 mon.a (mon.0) 2173 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.996081+0000 mon.a (mon.0) 2174 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:31.996162+0000 mon.a (mon.0) 2175 : audit [INF] from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:31.996212+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.048159+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.048501+0000 mon.a (mon.0) 2177 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:32.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:32.660 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:32.972 INFO:tasks.thrashosds.thrasher:j is {'epoch': 337, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:40:30.982012+0000', 'last_up_change': '2024-02-10T08:33:10.747466+0000', 'last_in_change': '2024-02-10T08:37:08.966515+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 89, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 197, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 60, 'pool_name': 'test-rados-api-smithi107-112490-6', 'create_time': '2024-02-10T08:36:49.742959+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '335', 'last_force_op_resend': '296', 'last_force_op_resend_prenautilus': '296', 'last_force_op_resend_preluminous': '296', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 318, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 196, 'pool_name': 'RoundTripWriteFullPP_smithi107-112411-58', 'create_time': '2024-02-10T08:40:30.336855+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '337', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-RoundTripWriteFullPP_smithi107-112411-58', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 197, 'pool_name': 'test-rados-api-smithi107-112490-45', 'create_time': '2024-02-10T08:40:30.588310+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '337', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 316, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:40:25.494086+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:39:38.100377+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:16.317482+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:27.388809+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:39:39.820862+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:07.356961+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:39:33.813260+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:39:37.860373+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-RoundTripWriteFullPP_smithi107-112411-58': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:40:32.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:40:32.978 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:33.109 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:33.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:33.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:33.271 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:33.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:33.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:33.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:31.847438+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:31.849087+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:32.181051+0000 osd.1 (osd.1) 149 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:32.184825+0000 osd.1 (osd.1) 150 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.993823+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:32.993891+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.994818+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.995698+0000 mon.a (mon.0) 2180 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:32.996424+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:33.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:31.847438+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:31.849087+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:32.181051+0000 osd.1 (osd.1) 149 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:32.184825+0000 osd.1 (osd.1) 150 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.993823+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:32.993891+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.994818+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.995698+0000 mon.a (mon.0) 2180 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:32.996424+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:33.404 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:31.847438+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:40:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:31.849087+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:40:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:32.181051+0000 osd.1 (osd.1) 149 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:40:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:32.184825+0000 osd.1 (osd.1) 150 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:40:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.993823+0000 mon.a (mon.0) 2178 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:32.993891+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-10T08:40:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.994818+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.995698+0000 mon.a (mon.0) 2180 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:32.996424+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:33.600 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:33.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:33.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:32.409883+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:32.412758+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:33.417657+0000 mgr.y (mgr.14553) 317 : cluster [DBG] pgmap v485: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:33.994158+0000 mon.a (mon.0) 2182 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:33.994243+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:33.994282+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:33.996283+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:33.996584+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:33.996832+0000 mon.a (mon.0) 2186 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:32.409883+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:32.412758+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:33.417657+0000 mgr.y (mgr.14553) 317 : cluster [DBG] pgmap v485: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:33.994158+0000 mon.a (mon.0) 2182 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:33.994243+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:33.994282+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:33.996283+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:33.996584+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:33.996832+0000 mon.a (mon.0) 2186 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:32.409883+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:32.412758+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:33.417657+0000 mgr.y (mgr.14553) 317 : cluster [DBG] pgmap v485: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:33.994158+0000 mon.a (mon.0) 2182 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:33.994243+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:33.994282+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-10T08:40:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:33.996283+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]: dispatch 2024-02-10T08:40:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:33.996584+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:33.996832+0000 mon.a (mon.0) 2186 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]: dispatch 2024-02-10T08:40:34.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:40:35.017 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:34.991536+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:34.995414+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:34.995472+0000 mon.a (mon.0) 2189 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]': finished 2024-02-10T08:40:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:34.995526+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:34.997305+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:34.998269+0000 mon.a (mon.0) 2191 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:34.991536+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:34.995414+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:34.995472+0000 mon.a (mon.0) 2189 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]': finished 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:34.995526+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:34.997305+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:34.998269+0000 mon.a (mon.0) 2191 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:34.991536+0000 mon.a (mon.0) 2187 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:34.995414+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.107:0/2611749197' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi107-112411-58"}]': finished 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:34.995472+0000 mon.a (mon.0) 2189 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-45", "mode": "writeback"}]': finished 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:34.995526+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:34.997305+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:34.998269+0000 mon.a (mon.0) 2191 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:40:35.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:40:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:35.815 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:35] "GET /metrics HTTP/1.1" 200 189026 "" "Prometheus/2.33.4" 2024-02-10T08:40:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:35.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:35.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:35.950 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:35.970 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:35.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:36.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:36.103 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:36.122 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:36.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:36.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.233+0000 7fc0cea9e700 1 -- 172.21.15.107:0/743572146 >> v1:172.21.15.107:6790/0 conn(0x7fc0c80fda50 legacy=0x7fc0c80fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.233+0000 7fc0cea9e700 1 -- 172.21.15.107:0/743572146 shutdown_connections 2024-02-10T08:40:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.233+0000 7fc0cea9e700 1 -- 172.21.15.107:0/743572146 >> 172.21.15.107:0/743572146 conn(0x7fc0c80f3640 msgr2=0x7fc0c80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- 172.21.15.107:0/743572146 shutdown_connections 2024-02-10T08:40:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- 172.21.15.107:0/743572146 wait complete. 2024-02-10T08:40:36.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 librados: starting msgr at 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 librados: starting objecter 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 Processor -- start 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- start start 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 librados: setting wanted keys 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 librados: calling monclient init 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0c80fa920 con 0x7fc0c80fda50 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0c8118870 con 0x7fc0c8104330 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0cea9e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0c8119a10 con 0x7fc0c81007f0 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0c7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc0c81007f0 0x7fc0c8112a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54396/0 (socket says 172.21.15.107:54396) 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.234+0000 7fc0c7fff700 1 -- 172.21.15.107:0/3001170826 learned_addr learned my addr 172.21.15.107:0/3001170826 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter ms_handle_connect 0x7fc0c81007f0 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:36.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter ms_handle_connect 0x7fc0c80fda50 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter ms_handle_connect 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735918428 0 0) 0x7fc0c8119a10 con 0x7fc0c81007f0 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b80033f0 con 0x7fc0c81007f0 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798649583 0 0) 0x7fc0c8118870 con 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c8119a10 con 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096639356 0 0) 0x7fc0b80033f0 con 0x7fc0c81007f0 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c8118870 con 0x7fc0c81007f0 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc0b4002990 con 0x7fc0c81007f0 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2112787547 0 0) 0x7fc0c8119a10 con 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b80033f0 con 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc0c0001b40 con 0x7fc0c8104330 2024-02-10T08:40:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339434600 0 0) 0x7fc0c8118870 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.195:6789/0 conn(0x7fc0c8104330 legacy=0x7fc0c8116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6789/0 conn(0x7fc0c80fda50 legacy=0x7fc0c80f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c811abb0 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c81178c0 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc0b4001890 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc0b4005b30 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.235+0000 7fc0cea9e700 10 client.53435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.236+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c8117d70 con 0x7fc0c81007f0 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.236+0000 7fc0cea9e700 1 librados: init done 2024-02-10T08:40:36.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.236+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc0c805f700 con 0x7fc0c81007f0 2024-02-10T08:40:36.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc0b4018aa0 con 0x7fc0c81007f0 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(342..342 src has 50..342) v4 ==== 8355+0+0 (unknown 1129045321 0 0) 0x7fc0b4078ff0 con 0x7fc0c81007f0 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 osd_map(342..342 src has 50..342) v4 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 3 client.53435.objecter handle_osd_map got epochs [342,342] > 0 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 3 client.53435.objecter handle_osd_map decoding full epoch 342 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 20 client.53435.objecter dump_active .. 0 homeless 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.237+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v342) v1 ==== 54+0+16 (unknown 2431857008 0 2420000192) 0x7fc0b407b400 con 0x7fc0c81007f0 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0b8063b10 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 1 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 1 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 0 1 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 4 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 10 client.53435.objecter _send_command 1 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c80475e0 con 0x7fc0c8118a00 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.238+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c8118a00 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.239+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c80475e0 con 0x7fc0c8118a00 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.239+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.239+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 1 = system:0 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.239+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 4 2024-02-10T08:40:36.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.239+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 0 1 2024-02-10T08:40:36.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 2 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 2 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80f9270 osd=0 3 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 0 2 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 4 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 10 client.53435.objecter _send_command 2 2024-02-10T08:40:36.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.255+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c8109f80 con 0x7fc0c8118a00 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c8109f80 con 0x7fc0c8118a00 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 4 2024-02-10T08:40:36.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 0 2 2024-02-10T08:40:36.258 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 1 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 4 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 10 client.53435.objecter _send_command 3 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c806cfc0 con 0x7fc0c80f40a0 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.257+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c80f40a0 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.258+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c806cfc0 con 0x7fc0c80f40a0 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.258+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.258+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 3 = system:0 2024-02-10T08:40:36.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.258+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 4 2024-02-10T08:40:36.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.258+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 1 3 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 4 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 4 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80f3a20 osd=1 3 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 1 4 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 4 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 10 client.53435.objecter _send_command 4 2024-02-10T08:40:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.274+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c8028660 con 0x7fc0c80f40a0 2024-02-10T08:40:36.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.276+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c8028660 con 0x7fc0c80f40a0 2024-02-10T08:40:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.276+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.276+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.276+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 4 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.276+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 1 4 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 5 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 5 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 2 5 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 4 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 10 client.53435.objecter _send_command 5 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c8059df0 con 0x7fc0c8059990 2024-02-10T08:40:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.277+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c8059990 2024-02-10T08:40:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.278+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c8059df0 con 0x7fc0c8059990 2024-02-10T08:40:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.278+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.278+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 5 = system:0 2024-02-10T08:40:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.278+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 4 2024-02-10T08:40:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.278+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 2 5 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.292+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 6 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 6 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c80287d0 osd=2 3 2024-02-10T08:40:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 2 6 2024-02-10T08:40:36.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 4 2024-02-10T08:40:36.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 10 client.53435.objecter _send_command 6 2024-02-10T08:40:36.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.293+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c805a1a0 con 0x7fc0c8059990 2024-02-10T08:40:36.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c805a1a0 con 0x7fc0c8059990 2024-02-10T08:40:36.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.295 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 4 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 2 6 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 7 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 7 2024-02-10T08:40:36.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 3 7 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 4 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 10 client.53435.objecter _send_command 7 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.294+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c8122090 con 0x7fc0c8059570 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c8059570 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c8122090 con 0x7fc0c8059570 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 7 = system:0 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 4 2024-02-10T08:40:36.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.295+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 3 7 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 8 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 8 2024-02-10T08:40:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8059400 osd=3 3 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 3 8 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 4 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 10 client.53435.objecter _send_command 8 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.309+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c8121460 con 0x7fc0c8059570 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c8121460 con 0x7fc0c8059570 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 4 2024-02-10T08:40:36.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 3 8 2024-02-10T08:40:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 9 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 9 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 4 9 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 4 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 10 client.53435.objecter _send_command 9 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.311+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c81268e0 con 0x7fc0c81215d0 2024-02-10T08:40:36.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c81215d0 2024-02-10T08:40:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c81268e0 con 0x7fc0c81215d0 2024-02-10T08:40:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 9 = system:0 2024-02-10T08:40:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 4 2024-02-10T08:40:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.312+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 4 9 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 10 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 10 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8121460 osd=4 3 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 4 10 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 4 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 10 client.53435.objecter _send_command 10 2024-02-10T08:40:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.327+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c81268e0 con 0x7fc0c81215d0 2024-02-10T08:40:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c81268e0 con 0x7fc0c81215d0 2024-02-10T08:40:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 4 2024-02-10T08:40:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 4 10 2024-02-10T08:40:36.330 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 11 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 11 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 5 11 2024-02-10T08:40:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 4 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 10 client.53435.objecter _send_command 11 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.329+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c8132dc0 con 0x7fc0c812f140 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.330+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c812f140 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.331+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c8132dc0 con 0x7fc0c812f140 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.331+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.331+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 11 = system:0 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.331+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 4 2024-02-10T08:40:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.331+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 5 11 2024-02-10T08:40:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 12 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 12 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c81268e0 osd=5 3 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 5 12 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 4 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 10 client.53435.objecter _send_command 12 2024-02-10T08:40:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.345+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c8134570 con 0x7fc0c812f140 2024-02-10T08:40:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c8134570 con 0x7fc0c812f140 2024-02-10T08:40:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 4 2024-02-10T08:40:36.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 5 12 2024-02-10T08:40:36.348 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 13 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 13 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 6 13 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 4 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 10 client.53435.objecter _send_command 13 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.347+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c8135180 con 0x7fc0c8134bf0 2024-02-10T08:40:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c8134bf0 2024-02-10T08:40:36.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c8135180 con 0x7fc0c8134bf0 2024-02-10T08:40:36.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:36.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 13 = system:0 2024-02-10T08:40:36.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 4 2024-02-10T08:40:36.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.348+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 6 13 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 14 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 14 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8134570 osd=6 3 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 6 14 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 4 2024-02-10T08:40:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 10 client.53435.objecter _send_command 14 2024-02-10T08:40:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.364+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c8135180 con 0x7fc0c8134bf0 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.366+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c8135180 con 0x7fc0c8134bf0 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.366+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.366+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.366+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 4 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.366+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 6 14 2024-02-10T08:40:36.367 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 15 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 15 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 7 15 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 4 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 10 client.53435.objecter _send_command 15 2024-02-10T08:40:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c813b770 con 0x7fc0c8137c60 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.367+0000 7fc0c5f76700 10 client.53435.objecter ms_handle_connect 0x7fc0c8137c60 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.368+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc0c813b770 con 0x7fc0c8137c60 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.368+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.368+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 15 = system:0 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.368+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 4 2024-02-10T08:40:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.368+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 7 15 2024-02-10T08:40:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 10 client.53435.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign -1 16 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 20 client.53435.objecter _get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_remove -1 16 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 20 client.53435.objecter get_session s=0x7fc0c8135180 osd=7 3 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 15 client.53435.objecter _session_command_op_assign 7 16 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 4 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 10 client.53435.objecter _send_command 16 2024-02-10T08:40:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.382+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc0c813b770 con 0x7fc0c8137c60 2024-02-10T08:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0c5f76700 1 -- 172.21.15.107:0/3001170826 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fc0c813b770 con 0x7fc0c8137c60 2024-02-10T08:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0c5f76700 10 client.53435.objecter ms_dispatch 0x7fc0c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0c5f76700 10 client.53435.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0c5f76700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 4 2024-02-10T08:40:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0c5f76700 15 client.53435.objecter _session_command_op_remove 7 16 2024-02-10T08:40:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 librados: watch_flush enter 2024-02-10T08:40:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 librados: watch_flush exit 2024-02-10T08:40:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.0 2024-02-10T08:40:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6801/3321596972 conn(0x7fc0c8118a00 legacy=0x7fc0c8118de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f9270 osd=0 1 2024-02-10T08:40:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.1 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6805/133516126 conn(0x7fc0c80f40a0 legacy=0x7fc0c80f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80f3a20 osd=1 1 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.2 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6809/3575881728 conn(0x7fc0c8059990 legacy=0x7fc0c81243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c80287d0 osd=2 1 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.3 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6813/2871825691 conn(0x7fc0c8059570 legacy=0x7fc0c8128c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8059400 osd=3 1 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.4 2024-02-10T08:40:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.195:6800/2499671285 conn(0x7fc0c81215d0 legacy=0x7fc0c812d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8121460 osd=4 1 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.5 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.195:6804/4178321486 conn(0x7fc0c812f140 legacy=0x7fc0c8131530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cd03b700 1 -- 172.21.15.107:0/3001170826 reap_dead start 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c81268e0 osd=5 1 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.6 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.195:6808/1417479017 conn(0x7fc0c8134bf0 legacy=0x7fc0c8132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cd03b700 1 -- 172.21.15.107:0/3001170826 reap_dead start 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8134570 osd=6 1 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter close_session for osd.7 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.195:6812/1934176928 conn(0x7fc0c8137c60 legacy=0x7fc0c813a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter put_session s=0x7fc0c8135180 osd=7 1 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 20 client.53435.objecter shutdown clearing up homeless session... 2024-02-10T08:40:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 10 client.53435.objecter successfully canceled tick 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cd03b700 1 -- 172.21.15.107:0/3001170826 reap_dead start 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cd03b700 1 -- 172.21.15.107:0/3001170826 reap_dead start 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc0b8063b10 legacy=0x7fc0b8065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> v1:172.21.15.107:6790/0 conn(0x7fc0c81007f0 legacy=0x7fc0c8112a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 shutdown_connections 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 >> 172.21.15.107:0/3001170826 conn(0x7fc0c80f3640 msgr2=0x7fc0c81077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 shutdown_connections 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.384+0000 7fc0cea9e700 1 -- 172.21.15.107:0/3001170826 wait complete. 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:36.385+0000 7fc0cea9e700 1 librados: shutdown 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.391 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:40:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:34.621892+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 deep-scrub starts 2024-02-10T08:40:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:34.622863+0000 osd.4 (osd.4) 151 : cluster [DBG] 5.1 deep-scrub ok 2024-02-10T08:40:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.017462+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:40:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.021162+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:40:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.304011+0000 osd.1 (osd.1) 151 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:40:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.307604+0000 osd.1 (osd.1) 152 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:40:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.372658+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:40:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.376853+0000 osd.6 (osd.6) 132 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.400387+0000 osd.2 (osd.2) 117 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.402554+0000 osd.2 (osd.2) 118 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.418112+0000 mgr.y (mgr.14553) 318 : cluster [DBG] pgmap v488: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.617399+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:35.620354+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:36.001344+0000 mon.a (mon.0) 2192 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.001446+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-10T08:40:36.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:36.005950+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:36.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:36.006324+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:36.006715+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:36.238017+0000 mon.c (mon.2) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/3001170826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:34.621892+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 deep-scrub starts 2024-02-10T08:40:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:34.622863+0000 osd.4 (osd.4) 151 : cluster [DBG] 5.1 deep-scrub ok 2024-02-10T08:40:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.017462+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:40:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.021162+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.304011+0000 osd.1 (osd.1) 151 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.307604+0000 osd.1 (osd.1) 152 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.372658+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.376853+0000 osd.6 (osd.6) 132 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.400387+0000 osd.2 (osd.2) 117 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.402554+0000 osd.2 (osd.2) 118 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.418112+0000 mgr.y (mgr.14553) 318 : cluster [DBG] pgmap v488: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.617399+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:35.620354+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:36.001344+0000 mon.a (mon.0) 2192 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.001446+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:36.005950+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:36.006324+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:36.006715+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:36.238017+0000 mon.c (mon.2) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/3001170826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:34.621892+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 deep-scrub starts 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:34.622863+0000 osd.4 (osd.4) 151 : cluster [DBG] 5.1 deep-scrub ok 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.017462+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.021162+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.304011+0000 osd.1 (osd.1) 151 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.307604+0000 osd.1 (osd.1) 152 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.372658+0000 osd.6 (osd.6) 131 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.376853+0000 osd.6 (osd.6) 132 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.400387+0000 osd.2 (osd.2) 117 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:40:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.402554+0000 osd.2 (osd.2) 118 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.418112+0000 mgr.y (mgr.14553) 318 : cluster [DBG] pgmap v488: 218 pgs: 32 creating+peering, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.617399+0000 osd.4 (osd.4) 152 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:35.620354+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:36.001344+0000 mon.a (mon.0) 2192 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.001446+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:36.005950+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:36.006324+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:36.006715+0000 mon.a (mon.0) 2195 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:40:36.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:36.238017+0000 mon.c (mon.2) 362 : audit [DBG] from='client.? v1:172.21.15.107:0/3001170826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:37.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:37.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:37.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.350897+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.12 deep-scrub starts 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.354064+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.12 deep-scrub ok 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.366847+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d scrub starts 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.368357+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d scrub ok 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.383061+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.383954+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.426591+0000 osd.2 (osd.2) 119 : cluster [DBG] 60.c scrub starts 2024-02-10T08:40:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:36.430065+0000 osd.2 (osd.2) 120 : cluster [DBG] 60.c scrub ok 2024-02-10T08:40:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:37.000964+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:37.001069+0000 mon.a (mon.0) 2197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.001132+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-10T08:40:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:37.006528+0000 mon.b (mon.1) 523 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:37.006913+0000 mon.a (mon.0) 2199 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.350897+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.12 deep-scrub starts 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.354064+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.12 deep-scrub ok 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.366847+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d scrub starts 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.368357+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d scrub ok 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.383061+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.383954+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.426591+0000 osd.2 (osd.2) 119 : cluster [DBG] 60.c scrub starts 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:36.430065+0000 osd.2 (osd.2) 120 : cluster [DBG] 60.c scrub ok 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:37.000964+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:37.001069+0000 mon.a (mon.0) 2197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.001132+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:37.006528+0000 mon.b (mon.1) 523 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:37.006913+0000 mon.a (mon.0) 2199 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.350897+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.12 deep-scrub starts 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.354064+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.12 deep-scrub ok 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.366847+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d scrub starts 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.368357+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d scrub ok 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.383061+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.383954+0000 osd.7 (osd.7) 140 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.426591+0000 osd.2 (osd.2) 119 : cluster [DBG] 60.c scrub starts 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:36.430065+0000 osd.2 (osd.2) 120 : cluster [DBG] 60.c scrub ok 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:37.000964+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? v1:172.21.15.107:0/248829415' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi107-112411-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:37.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:37.001069+0000 mon.a (mon.0) 2197 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:40:37.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.001132+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-10T08:40:37.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:37.006528+0000 mon.b (mon.1) 523 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:37.006913+0000 mon.a (mon.0) 2199 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:40:37.543 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:37.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:37.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:37.943 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:37.963 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:37.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:38.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.993+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/2845790338 >> v1:172.21.15.107:6789/0 conn(0x7f3ac00ff710 legacy=0x7f3ac0101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.993+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/2845790338 shutdown_connections 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.993+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/2845790338 >> 172.21.15.107:0/2845790338 conn(0x7f3ac00f2510 msgr2=0x7f3ac00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.993+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/2845790338 shutdown_connections 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.993+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/2845790338 wait complete. 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 librados: starting msgr at 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 librados: starting objecter 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 Processor -- start 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 -- start start 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 librados: setting wanted keys 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 librados: calling monclient init 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ac0119480 con 0x7f3ac00fc910 2024-02-10T08:40:38.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ac011a620 con 0x7f3ac0103250 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac7ed4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ac011b800 con 0x7f3ac00ff710 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac546f700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3ac00ff710 0x7f3ac00fbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54410/0 (socket says 172.21.15.107:54410) 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac5c70700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3ac00fc910 0x7f3ac00fb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45132/0 (socket says 172.21.15.107:45132) 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ac546f700 1 -- 172.21.15.107:0/1477359855 learned_addr learned my addr 172.21.15.107:0/1477359855 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.994+0000 7f3ab6ffd700 10 client.?.objecter ms_handle_connect 0x7f3ac00ff710 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 10 client.?.objecter ms_handle_connect 0x7f3ac00fc910 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 10 client.?.objecter ms_handle_connect 0x7f3ac0103250 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367451272 0 0) 0x7f3ac011b800 con 0x7f3ac00ff710 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac003030 con 0x7f3ac00ff710 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377377155 0 0) 0x7f3ac0119480 con 0x7f3ac00fc910 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac011b800 con 0x7f3ac00fc910 2024-02-10T08:40:38.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512190890 0 0) 0x7f3ac011a620 con 0x7f3ac0103250 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac0119480 con 0x7f3ac0103250 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1430688248 0 0) 0x7f3aac003030 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ac011a620 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3ab0001510 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3872290160 0 0) 0x7f3ac011b800 con 0x7f3ac00fc910 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aac003030 con 0x7f3ac00fc910 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3abc001a80 con 0x7f3ac00fc910 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 628974589 0 0) 0x7f3ac0119480 con 0x7f3ac0103250 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ac011b800 con 0x7f3ac0103250 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3ab8001bc0 con 0x7f3ac0103250 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4070278276 0 0) 0x7f3ac011a620 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 >> v1:172.21.15.195:6789/0 conn(0x7f3ac0103250 legacy=0x7f3ac0117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 >> v1:172.21.15.107:6789/0 conn(0x7f3ac00fc910 legacy=0x7f3ac00fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac011ca00 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac0119670 con 0x7f3ac00ff710 2024-02-10T08:40:38.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ac7ed4700 10 client.53441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac0119ae0 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ac7ed4700 1 librados: init done 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3ab0002280 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.995+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3ab0003820 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.996+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa4004f20 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.996+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3ab0003110 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(343..343 src has 50..343) v4 ==== 8368+0+0 (unknown 4244736166 0 0) 0x7f3ab0079110 con 0x7f3ac00ff710 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 10 client.53441.objecter ms_dispatch 0x7f3ac0001550 osd_map(343..343 src has 50..343) v4 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 3 client.53441.objecter handle_osd_map got epochs [343,343] > 0 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 3 client.53441.objecter handle_osd_map decoding full epoch 343 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 20 client.53441.objecter dump_active .. 0 homeless 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:37.997+0000 7f3ab6ffd700 10 client.53441.objecter ms_handle_connect 0x7f3aac063dd0 2024-02-10T08:40:38.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.001+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3ab004c3f0 con 0x7f3ac00ff710 2024-02-10T08:40:38.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:38.094 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.183+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3aa4000b90 con 0x7f3aac063dd0 2024-02-10T08:40:38.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.199+0000 7f3ab6ffd700 1 -- 172.21.15.107:0/1477359855 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553680 (unknown 2965378022 0 1018835412) 0x7f3aa4000b90 con 0x7f3aac063dd0 2024-02-10T08:40:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.202+0000 7f3ac7ed4700 10 librados: watch_flush enter 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.202+0000 7f3ac7ed4700 10 librados: watch_flush exit 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.202+0000 7f3ac7ed4700 20 client.53441.objecter shutdown clearing up homeless session... 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.202+0000 7f3ac7ed4700 10 client.53441.objecter successfully canceled tick 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3aac063dd0 legacy=0x7f3aac066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 >> v1:172.21.15.107:6790/0 conn(0x7f3ac00ff710 legacy=0x7f3ac00fbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 shutdown_connections 2024-02-10T08:40:38.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 >> 172.21.15.107:0/1477359855 conn(0x7f3ac00f2510 msgr2=0x7f3ac01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:38.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 shutdown_connections 2024-02-10T08:40:38.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 -- 172.21.15.107:0/1477359855 wait complete. 2024-02-10T08:40:38.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:38.203+0000 7f3ac7ed4700 1 librados: shutdown 2024-02-10T08:40:38.213 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:40:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.414221+0000 osd.2 (osd.2) 121 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:40:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.417297+0000 osd.2 (osd.2) 122 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.418473+0000 mgr.y (mgr.14553) 319 : cluster [DBG] pgmap v491: 250 pgs: 32 creating+peering, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.662802+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.664580+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:37.997858+0000 mon.a (mon.0) 2200 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.004836+0000 mon.a (mon.0) 2201 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:38.004927+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.007265+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.009689+0000 mon.a (mon.0) 2203 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.028127+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.029535+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.029818+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.414221+0000 osd.2 (osd.2) 121 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.417297+0000 osd.2 (osd.2) 122 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.418473+0000 mgr.y (mgr.14553) 319 : cluster [DBG] pgmap v491: 250 pgs: 32 creating+peering, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.662802+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.664580+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:37.997858+0000 mon.a (mon.0) 2200 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.004836+0000 mon.a (mon.0) 2201 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:38.004927+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.007265+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.009689+0000 mon.a (mon.0) 2203 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.028127+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.029535+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.029818+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:38.445 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:38.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.414221+0000 osd.2 (osd.2) 121 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:40:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.417297+0000 osd.2 (osd.2) 122 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:40:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.418473+0000 mgr.y (mgr.14553) 319 : cluster [DBG] pgmap v491: 250 pgs: 32 creating+peering, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.662802+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.664580+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:37.997858+0000 mon.a (mon.0) 2200 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.004836+0000 mon.a (mon.0) 2201 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:38.004927+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.007265+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.009689+0000 mon.a (mon.0) 2203 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.028127+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.029535+0000 mon.a (mon.0) 2205 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.029818+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:38.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:38.603 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:38.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:38.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:38.650 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:38.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:38.736 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:38.756 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:38.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:38.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:39.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:39.052 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-02-10T08:40:38.005802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007181,"num_objects":336,"num_object_clones":2,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":1252,"num_read_kb":910,"num_write":1003,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2322,"ondisk_log_size":2322,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":655,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2184276,"kb_used_data":50424,"kb_used_omap":0,"kb_used_meta":2133760,"kb_avail":654031788,"statfs":{"total":671965249536,"available":669728550912,"internally_reserved":0,"allocated":51634176,"data_stored":33956338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184970240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":160,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014105"},"pg_stats":[{"pgid":"197.1f","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997214+0000","last_change":"2024-02-10T08:40:32.006122+0000","last_active":"2024-02-10T08:40:34.997214+0000","last_peered":"2024-02-10T08:40:34.997214+0000","last_clean":"2024-02-10T08:40:34.997214+0000","last_became_active":"2024-02-10T08:40:32.005792+0000","last_became_peered":"2024-02-10T08:40:32.005792+0000","last_unstale":"2024-02-10T08:40:34.997214+0000","last_undegraded":"2024-02-10T08:40:34.997214+0000","last_fullsized":"2024-02-10T08:40:34.997214+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.1e","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997670+0000","last_change":"2024-02-10T08:40:32.014038+0000","last_active":"2024-02-10T08:40:34.997670+0000","last_peered":"2024-02-10T08:40:34.997670+0000","last_clean":"2024-02-10T08:40:34.997670+0000","last_became_active":"2024-02-10T08:40:32.013172+0000","last_became_peered":"2024-02-10T08:40:32.013172+0000","last_unstale":"2024-02-10T08:40:34.997670+0000","last_undegraded":"2024-02-10T08:40:34.997670+0000","last_fullsized":"2024-02-10T08:40:34.997670+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.1d","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374919+0000","last_change":"2024-02-10T08:40:32.020290+0000","last_active":"2024-02-10T08:40:35.374919+0000","last_peered":"2024-02-10T08:40:35.374919+0000","last_clean":"2024-02-10T08:40:35.374919+0000","last_became_active":"2024-02-10T08:40:32.019972+0000","last_became_peered":"2024-02-10T08:40:32.019972+0000","last_unstale":"2024-02-10T08:40:35.374919+0000","last_undegraded":"2024-02-10T08:40:35.374919+0000","last_fullsized":"2024-02-10T08:40:35.374919+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"197.1c","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996116+0000","last_change":"2024-02-10T08:40:32.010804+0000","last_active":"2024-02-10T08:40:34.996116+0000","last_peered":"2024-02-10T08:40:34.996116+0000","last_clean":"2024-02-10T08:40:34.996116+0000","last_became_active":"2024-02-10T08:40:32.010461+0000","last_became_peered":"2024-02-10T08:40:32.010461+0000","last_unstale":"2024-02-10T08:40:34.996116+0000","last_undegraded":"2024-02-10T08:40:34.996116+0000","last_fullsized":"2024-02-10T08:40:34.996116+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.1b","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995467+0000","last_change":"2024-02-10T08:40:32.009818+0000","last_active":"2024-02-10T08:40:34.995467+0000","last_peered":"2024-02-10T08:40:34.995467+0000","last_clean":"2024-02-10T08:40:34.995467+0000","last_became_active":"2024-02-10T08:40:32.009353+0000","last_became_peered":"2024-02-10T08:40:32.009353+0000","last_unstale":"2024-02-10T08:40:34.995467+0000","last_undegraded":"2024-02-10T08:40:34.995467+0000","last_fullsized":"2024-02-10T08:40:34.995467+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.1a","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018586+0000","last_change":"2024-02-10T08:40:32.021287+0000","last_active":"2024-02-10T08:40:35.018586+0000","last_peered":"2024-02-10T08:40:35.018586+0000","last_clean":"2024-02-10T08:40:35.018586+0000","last_became_active":"2024-02-10T08:40:32.020998+0000","last_became_peered":"2024-02-10T08:40:32.020998+0000","last_unstale":"2024-02-10T08:40:35.018586+0000","last_undegraded":"2024-02-10T08:40:35.018586+0000","last_fullsized":"2024-02-10T08:40:35.018586+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"197.19","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995644+0000","last_change":"2024-02-10T08:40:32.011647+0000","last_active":"2024-02-10T08:40:34.995644+0000","last_peered":"2024-02-10T08:40:34.995644+0000","last_clean":"2024-02-10T08:40:34.995644+0000","last_became_active":"2024-02-10T08:40:32.011377+0000","last_became_peered":"2024-02-10T08:40:32.011377+0000","last_unstale":"2024-02-10T08:40:34.995644+0000","last_undegraded":"2024-02-10T08:40:34.995644+0000","last_fullsized":"2024-02-10T08:40:34.995644+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.18","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994851+0000","last_change":"2024-02-10T08:40:32.007697+0000","last_active":"2024-02-10T08:40:34.994851+0000","last_peered":"2024-02-10T08:40:34.994851+0000","last_clean":"2024-02-10T08:40:34.994851+0000","last_became_active":"2024-02-10T08:40:32.007339+0000","last_became_peered":"2024-02-10T08:40:32.007339+0000","last_unstale":"2024-02-10T08:40:34.994851+0000","last_undegraded":"2024-02-10T08:40:34.994851+0000","last_fullsized":"2024-02-10T08:40:34.994851+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.17","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997557+0000","last_change":"2024-02-10T08:40:32.016810+0000","last_active":"2024-02-10T08:40:34.997557+0000","last_peered":"2024-02-10T08:40:34.997557+0000","last_clean":"2024-02-10T08:40:34.997557+0000","last_became_active":"2024-02-10T08:40:32.016530+0000","last_became_peered":"2024-02-10T08:40:32.016530+0000","last_unstale":"2024-02-10T08:40:34.997557+0000","last_undegraded":"2024-02-10T08:40:34.997557+0000","last_fullsized":"2024-02-10T08:40:34.997557+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.16","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997772+0000","last_change":"2024-02-10T08:40:32.015823+0000","last_active":"2024-02-10T08:40:34.997772+0000","last_peered":"2024-02-10T08:40:34.997772+0000","last_clean":"2024-02-10T08:40:34.997772+0000","last_became_active":"2024-02-10T08:40:32.015528+0000","last_became_peered":"2024-02-10T08:40:32.015528+0000","last_unstale":"2024-02-10T08:40:34.997772+0000","last_undegraded":"2024-02-10T08:40:34.997772+0000","last_fullsized":"2024-02-10T08:40:34.997772+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.15","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019810+0000","last_change":"2024-02-10T08:40:32.022212+0000","last_active":"2024-02-10T08:40:35.019810+0000","last_peered":"2024-02-10T08:40:35.019810+0000","last_clean":"2024-02-10T08:40:35.019810+0000","last_became_active":"2024-02-10T08:40:32.021858+0000","last_became_peered":"2024-02-10T08:40:32.021858+0000","last_unstale":"2024-02-10T08:40:35.019810+0000","last_undegraded":"2024-02-10T08:40:35.019810+0000","last_fullsized":"2024-02-10T08:40:35.019810+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"197.14","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021589+0000","last_change":"2024-02-10T08:40:32.013315+0000","last_active":"2024-02-10T08:40:35.021589+0000","last_peered":"2024-02-10T08:40:35.021589+0000","last_clean":"2024-02-10T08:40:35.021589+0000","last_became_active":"2024-02-10T08:40:32.013012+0000","last_became_peered":"2024-02-10T08:40:32.013012+0000","last_unstale":"2024-02-10T08:40:35.021589+0000","last_undegraded":"2024-02-10T08:40:35.021589+0000","last_fullsized":"2024-02-10T08:40:35.021589+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.13","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995541+0000","last_change":"2024-02-10T08:40:32.010101+0000","last_active":"2024-02-10T08:40:34.995541+0000","last_peered":"2024-02-10T08:40:34.995541+0000","last_clean":"2024-02-10T08:40:34.995541+0000","last_became_active":"2024-02-10T08:40:32.009067+0000","last_became_peered":"2024-02-10T08:40:32.009067+0000","last_unstale":"2024-02-10T08:40:34.995541+0000","last_undegraded":"2024-02-10T08:40:34.995541+0000","last_fullsized":"2024-02-10T08:40:34.995541+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.12","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994519+0000","last_change":"2024-02-10T08:40:32.013565+0000","last_active":"2024-02-10T08:40:34.994519+0000","last_peered":"2024-02-10T08:40:34.994519+0000","last_clean":"2024-02-10T08:40:34.994519+0000","last_became_active":"2024-02-10T08:40:32.013284+0000","last_became_peered":"2024-02-10T08:40:32.013284+0000","last_unstale":"2024-02-10T08:40:34.994519+0000","last_undegraded":"2024-02-10T08:40:34.994519+0000","last_fullsized":"2024-02-10T08:40:34.994519+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.11","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995754+0000","last_change":"2024-02-10T08:40:32.010819+0000","last_active":"2024-02-10T08:40:34.995754+0000","last_peered":"2024-02-10T08:40:34.995754+0000","last_clean":"2024-02-10T08:40:34.995754+0000","last_became_active":"2024-02-10T08:40:32.010422+0000","last_became_peered":"2024-02-10T08:40:32.010422+0000","last_unstale":"2024-02-10T08:40:34.995754+0000","last_undegraded":"2024-02-10T08:40:34.995754+0000","last_fullsized":"2024-02-10T08:40:34.995754+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.10","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994106+0000","last_change":"2024-02-10T08:40:32.012215+0000","last_active":"2024-02-10T08:40:34.994106+0000","last_peered":"2024-02-10T08:40:34.994106+0000","last_clean":"2024-02-10T08:40:34.994106+0000","last_became_active":"2024-02-10T08:40:32.011943+0000","last_became_peered":"2024-02-10T08:40:32.011943+0000","last_unstale":"2024-02-10T08:40:34.994106+0000","last_undegraded":"2024-02-10T08:40:34.994106+0000","last_fullsized":"2024-02-10T08:40:34.994106+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.f","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023166+0000","last_change":"2024-02-10T08:40:32.013676+0000","last_active":"2024-02-10T08:40:35.023166+0000","last_peered":"2024-02-10T08:40:35.023166+0000","last_clean":"2024-02-10T08:40:35.023166+0000","last_became_active":"2024-02-10T08:40:32.013214+0000","last_became_peered":"2024-02-10T08:40:32.013214+0000","last_unstale":"2024-02-10T08:40:35.023166+0000","last_undegraded":"2024-02-10T08:40:35.023166+0000","last_fullsized":"2024-02-10T08:40:35.023166+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.e","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019417+0000","last_change":"2024-02-10T08:40:32.021923+0000","last_active":"2024-02-10T08:40:35.019417+0000","last_peered":"2024-02-10T08:40:35.019417+0000","last_clean":"2024-02-10T08:40:35.019417+0000","last_became_active":"2024-02-10T08:40:32.021619+0000","last_became_peered":"2024-02-10T08:40:32.021619+0000","last_unstale":"2024-02-10T08:40:35.019417+0000","last_undegraded":"2024-02-10T08:40:35.019417+0000","last_fullsized":"2024-02-10T08:40:35.019417+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"197.d","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995325+0000","last_change":"2024-02-10T08:40:32.016710+0000","last_active":"2024-02-10T08:40:34.995325+0000","last_peered":"2024-02-10T08:40:34.995325+0000","last_clean":"2024-02-10T08:40:34.995325+0000","last_became_active":"2024-02-10T08:40:32.016388+0000","last_became_peered":"2024-02-10T08:40:32.016388+0000","last_unstale":"2024-02-10T08:40:34.995325+0000","last_undegraded":"2024-02-10T08:40:34.995325+0000","last_fullsized":"2024-02-10T08:40:34.995325+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.c","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997278+0000","last_change":"2024-02-10T08:40:32.013864+0000","last_active":"2024-02-10T08:40:34.997278+0000","last_peered":"2024-02-10T08:40:34.997278+0000","last_clean":"2024-02-10T08:40:34.997278+0000","last_became_active":"2024-02-10T08:40:32.013573+0000","last_became_peered":"2024-02-10T08:40:32.013573+0000","last_unstale":"2024-02-10T08:40:34.997278+0000","last_undegraded":"2024-02-10T08:40:34.997278+0000","last_fullsized":"2024-02-10T08:40:34.997278+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.b","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996737+0000","last_change":"2024-02-10T08:40:32.010606+0000","last_active":"2024-02-10T08:40:34.996737+0000","last_peered":"2024-02-10T08:40:34.996737+0000","last_clean":"2024-02-10T08:40:34.996737+0000","last_became_active":"2024-02-10T08:40:32.010338+0000","last_became_peered":"2024-02-10T08:40:32.010338+0000","last_unstale":"2024-02-10T08:40:34.996737+0000","last_undegraded":"2024-02-10T08:40:34.996737+0000","last_fullsized":"2024-02-10T08:40:34.996737+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.a","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401155+0000","last_change":"2024-02-10T08:40:32.020344+0000","last_active":"2024-02-10T08:40:35.401155+0000","last_peered":"2024-02-10T08:40:35.401155+0000","last_clean":"2024-02-10T08:40:35.401155+0000","last_became_active":"2024-02-10T08:40:32.020078+0000","last_became_peered":"2024-02-10T08:40:32.020078+0000","last_unstale":"2024-02-10T08:40:35.401155+0000","last_undegraded":"2024-02-10T08:40:35.401155+0000","last_fullsized":"2024-02-10T08:40:35.401155+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"197.9","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994911+0000","last_change":"2024-02-10T08:40:32.008529+0000","last_active":"2024-02-10T08:40:34.994911+0000","last_peered":"2024-02-10T08:40:34.994911+0000","last_clean":"2024-02-10T08:40:34.994911+0000","last_became_active":"2024-02-10T08:40:32.008226+0000","last_became_peered":"2024-02-10T08:40:32.008226+0000","last_unstale":"2024-02-10T08:40:34.994911+0000","last_undegraded":"2024-02-10T08:40:34.994911+0000","last_fullsized":"2024-02-10T08:40:34.994911+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.8","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.373671+0000","last_change":"2024-02-10T08:40:32.020996+0000","last_active":"2024-02-10T08:40:35.373671+0000","last_peered":"2024-02-10T08:40:35.373671+0000","last_clean":"2024-02-10T08:40:35.373671+0000","last_became_active":"2024-02-10T08:40:32.020438+0000","last_became_peered":"2024-02-10T08:40:32.020438+0000","last_unstale":"2024-02-10T08:40:35.373671+0000","last_undegraded":"2024-02-10T08:40:35.373671+0000","last_fullsized":"2024-02-10T08:40:35.373671+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"197.6","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402397+0000","last_change":"2024-02-10T08:40:32.016597+0000","last_active":"2024-02-10T08:40:35.402397+0000","last_peered":"2024-02-10T08:40:35.402397+0000","last_clean":"2024-02-10T08:40:35.402397+0000","last_became_active":"2024-02-10T08:40:32.016282+0000","last_became_peered":"2024-02-10T08:40:32.016282+0000","last_unstale":"2024-02-10T08:40:35.402397+0000","last_undegraded":"2024-02-10T08:40:35.402397+0000","last_fullsized":"2024-02-10T08:40:35.402397+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"197.7","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997548+0000","last_change":"2024-02-10T08:40:32.011717+0000","last_active":"2024-02-10T08:40:34.997548+0000","last_peered":"2024-02-10T08:40:34.997548+0000","last_clean":"2024-02-10T08:40:34.997548+0000","last_became_active":"2024-02-10T08:40:32.011309+0000","last_became_peered":"2024-02-10T08:40:32.011309+0000","last_unstale":"2024-02-10T08:40:34.997548+0000","last_undegraded":"2024-02-10T08:40:34.997548+0000","last_fullsized":"2024-02-10T08:40:34.997548+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.4","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374816+0000","last_change":"2024-02-10T08:40:32.010400+0000","last_active":"2024-02-10T08:40:35.374816+0000","last_peered":"2024-02-10T08:40:35.374816+0000","last_clean":"2024-02-10T08:40:35.374816+0000","last_became_active":"2024-02-10T08:40:32.010086+0000","last_became_peered":"2024-02-10T08:40:32.010086+0000","last_unstale":"2024-02-10T08:40:35.374816+0000","last_undegraded":"2024-02-10T08:40:35.374816+0000","last_fullsized":"2024-02-10T08:40:35.374816+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"197.5","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022783+0000","last_change":"2024-02-10T08:40:32.018367+0000","last_active":"2024-02-10T08:40:35.022783+0000","last_peered":"2024-02-10T08:40:35.022783+0000","last_clean":"2024-02-10T08:40:35.022783+0000","last_became_active":"2024-02-10T08:40:32.017942+0000","last_became_peered":"2024-02-10T08:40:32.017942+0000","last_unstale":"2024-02-10T08:40:35.022783+0000","last_undegraded":"2024-02-10T08:40:35.022783+0000","last_fullsized":"2024-02-10T08:40:35.022783+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.2","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995293+0000","last_change":"2024-02-10T08:40:32.016016+0000","last_active":"2024-02-10T08:40:34.995293+0000","last_peered":"2024-02-10T08:40:34.995293+0000","last_clean":"2024-02-10T08:40:34.995293+0000","last_became_active":"2024-02-10T08:40:32.015681+0000","last_became_peered":"2024-02-10T08:40:32.015681+0000","last_unstale":"2024-02-10T08:40:34.995293+0000","last_undegraded":"2024-02-10T08:40:34.995293+0000","last_fullsized":"2024-02-10T08:40:34.995293+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.3","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995767+0000","last_change":"2024-02-10T08:40:32.016647+0000","last_active":"2024-02-10T08:40:34.995767+0000","last_peered":"2024-02-10T08:40:34.995767+0000","last_clean":"2024-02-10T08:40:34.995767+0000","last_became_active":"2024-02-10T08:40:32.016242+0000","last_became_peered":"2024-02-10T08:40:32.016242+0000","last_unstale":"2024-02-10T08:40:34.995767+0000","last_undegraded":"2024-02-10T08:40:34.995767+0000","last_fullsized":"2024-02-10T08:40:34.995767+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.0","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994107+0000","last_change":"2024-02-10T08:40:32.013362+0000","last_active":"2024-02-10T08:40:34.994107+0000","last_peered":"2024-02-10T08:40:34.994107+0000","last_clean":"2024-02-10T08:40:34.994107+0000","last_became_active":"2024-02-10T08:40:32.013070+0000","last_became_peered":"2024-02-10T08:40:32.013070+0000","last_unstale":"2024-02-10T08:40:34.994107+0000","last_undegraded":"2024-02-10T08:40:34.994107+0000","last_fullsized":"2024-02-10T08:40:34.994107+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.1","version":"0'0","reported_seq":13,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018559+0000","last_change":"2024-02-10T08:40:32.014578+0000","last_active":"2024-02-10T08:40:35.018559+0000","last_peered":"2024-02-10T08:40:35.018559+0000","last_clean":"2024-02-10T08:40:35.018559+0000","last_became_active":"2024-02-10T08:40:32.014257+0000","last_became_peered":"2024-02-10T08:40:32.014257+0000","last_unstale":"2024-02-10T08:40:35.018559+0000","last_undegraded":"2024-02-10T08:40:35.018559+0000","last_fullsized":"2024-02-10T08:40:35.018559+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":337,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:30.982012+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.982012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":309,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996992+0000","last_change":"2024-02-10T08:40:32.184907+0000","last_active":"2024-02-10T08:40:34.996992+0000","last_peered":"2024-02-10T08:40:34.996992+0000","last_clean":"2024-02-10T08:40:34.996992+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:40:34.996992+0000","last_undegraded":"2024-02-10T08:40:34.996992+0000","last_fullsized":"2024-02-10T08:40:34.996992+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:40:32.184829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.184829+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"338'29","reported_seq":315,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995429+0000","last_change":"2024-02-10T08:39:38.573741+0000","last_active":"2024-02-10T08:40:34.995429+0000","last_peered":"2024-02-10T08:40:34.995429+0000","last_clean":"2024-02-10T08:40:34.995429+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:40:34.995429+0000","last_undegraded":"2024-02-10T08:40:34.995429+0000","last_fullsized":"2024-02-10T08:40:34.995429+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"250'39","reported_seq":312,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995134+0000","last_change":"2024-02-10T08:39:40.483224+0000","last_active":"2024-02-10T08:40:34.995134+0000","last_peered":"2024-02-10T08:40:34.995134+0000","last_clean":"2024-02-10T08:40:34.995134+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:40:34.995134+0000","last_undegraded":"2024-02-10T08:40:34.995134+0000","last_fullsized":"2024-02-10T08:40:34.995134+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"231'107","reported_seq":386,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402740+0000","last_change":"2024-02-10T08:40:35.402698+0000","last_active":"2024-02-10T08:40:35.402740+0000","last_peered":"2024-02-10T08:40:35.402740+0000","last_clean":"2024-02-10T08:40:35.402740+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:40:35.402740+0000","last_undegraded":"2024-02-10T08:40:35.402740+0000","last_fullsized":"2024-02-10T08:40:35.402740+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"222'34","reported_seq":290,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997717+0000","last_change":"2024-02-10T08:39:53.632526+0000","last_active":"2024-02-10T08:40:34.997717+0000","last_peered":"2024-02-10T08:40:34.997717+0000","last_clean":"2024-02-10T08:40:34.997717+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:40:34.997717+0000","last_undegraded":"2024-02-10T08:40:34.997717+0000","last_fullsized":"2024-02-10T08:40:34.997717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"338'73","reported_seq":346,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402581+0000","last_change":"2024-02-10T08:39:49.450775+0000","last_active":"2024-02-10T08:40:35.402581+0000","last_peered":"2024-02-10T08:40:35.402581+0000","last_clean":"2024-02-10T08:40:35.402581+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:40:35.402581+0000","last_undegraded":"2024-02-10T08:40:35.402581+0000","last_fullsized":"2024-02-10T08:40:35.402581+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":329,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996924+0000","last_change":"2024-02-10T08:39:52.583051+0000","last_active":"2024-02-10T08:40:34.996924+0000","last_peered":"2024-02-10T08:40:34.996924+0000","last_clean":"2024-02-10T08:40:34.996924+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:40:34.996924+0000","last_undegraded":"2024-02-10T08:40:34.996924+0000","last_fullsized":"2024-02-10T08:40:34.996924+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"338'23","reported_seq":307,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995950+0000","last_change":"2024-02-10T08:39:42.510684+0000","last_active":"2024-02-10T08:40:34.995950+0000","last_peered":"2024-02-10T08:40:34.995950+0000","last_clean":"2024-02-10T08:40:34.995950+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:40:34.995950+0000","last_undegraded":"2024-02-10T08:40:34.995950+0000","last_fullsized":"2024-02-10T08:40:34.995950+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"250'101","reported_seq":372,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.400854+0000","last_change":"2024-02-10T08:40:28.463491+0000","last_active":"2024-02-10T08:40:35.400854+0000","last_peered":"2024-02-10T08:40:35.400854+0000","last_clean":"2024-02-10T08:40:35.400854+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:40:35.400854+0000","last_undegraded":"2024-02-10T08:40:35.400854+0000","last_fullsized":"2024-02-10T08:40:35.400854+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'101","last_scrub_stamp":"2024-02-10T08:40:28.463426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.463426+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"228'22","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995398+0000","last_change":"2024-02-10T08:39:47.635717+0000","last_active":"2024-02-10T08:40:34.995398+0000","last_peered":"2024-02-10T08:40:34.995398+0000","last_clean":"2024-02-10T08:40:34.995398+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:40:34.995398+0000","last_undegraded":"2024-02-10T08:40:34.995398+0000","last_fullsized":"2024-02-10T08:40:34.995398+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"338'111","reported_seq":367,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995895+0000","last_change":"2024-02-10T08:39:53.048253+0000","last_active":"2024-02-10T08:40:34.995895+0000","last_peered":"2024-02-10T08:40:34.995895+0000","last_clean":"2024-02-10T08:40:34.995895+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:40:34.995895+0000","last_undegraded":"2024-02-10T08:40:34.995895+0000","last_fullsized":"2024-02-10T08:40:34.995895+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"338'86","reported_seq":361,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995063+0000","last_change":"2024-02-10T08:39:50.630114+0000","last_active":"2024-02-10T08:40:34.995063+0000","last_peered":"2024-02-10T08:40:34.995063+0000","last_clean":"2024-02-10T08:40:34.995063+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:40:34.995063+0000","last_undegraded":"2024-02-10T08:40:34.995063+0000","last_fullsized":"2024-02-10T08:40:34.995063+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"231'93","reported_seq":357,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375501+0000","last_change":"2024-02-10T08:39:43.510896+0000","last_active":"2024-02-10T08:40:35.375501+0000","last_peered":"2024-02-10T08:40:35.375501+0000","last_clean":"2024-02-10T08:40:35.375501+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:40:35.375501+0000","last_undegraded":"2024-02-10T08:40:35.375501+0000","last_fullsized":"2024-02-10T08:40:35.375501+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'93","last_scrub_stamp":"2024-02-10T08:39:43.510789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.510789+0000","log_size":93,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":72,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":312,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997864+0000","last_change":"2024-02-10T08:39:54.894186+0000","last_active":"2024-02-10T08:40:34.997864+0000","last_peered":"2024-02-10T08:40:34.997864+0000","last_clean":"2024-02-10T08:40:34.997864+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:40:34.997864+0000","last_undegraded":"2024-02-10T08:40:34.997864+0000","last_fullsized":"2024-02-10T08:40:34.997864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"338'57","reported_seq":326,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375126+0000","last_change":"2024-02-10T08:39:44.469992+0000","last_active":"2024-02-10T08:40:35.375126+0000","last_peered":"2024-02-10T08:40:35.375126+0000","last_clean":"2024-02-10T08:40:35.375126+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:40:35.375126+0000","last_undegraded":"2024-02-10T08:40:35.375126+0000","last_fullsized":"2024-02-10T08:40:35.375126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-02-10T08:39:44.469923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.469923+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"338'70","reported_seq":340,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401598+0000","last_change":"2024-02-10T08:39:32.974869+0000","last_active":"2024-02-10T08:40:35.401598+0000","last_peered":"2024-02-10T08:40:35.401598+0000","last_clean":"2024-02-10T08:40:35.401598+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:40:35.401598+0000","last_undegraded":"2024-02-10T08:40:35.401598+0000","last_fullsized":"2024-02-10T08:40:35.401598+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:36:50.313439+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"338'56","reported_seq":331,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018998+0000","last_change":"2024-02-10T08:39:33.014018+0000","last_active":"2024-02-10T08:40:35.018998+0000","last_peered":"2024-02-10T08:40:35.018998+0000","last_clean":"2024-02-10T08:40:35.018998+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:40:35.018998+0000","last_undegraded":"2024-02-10T08:40:35.018998+0000","last_fullsized":"2024-02-10T08:40:35.018998+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'54","last_scrub_stamp":"2024-02-10T08:39:12.673905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.673905+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":378,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021521+0000","last_change":"2024-02-10T08:39:46.646522+0000","last_active":"2024-02-10T08:40:35.021521+0000","last_peered":"2024-02-10T08:40:35.021521+0000","last_clean":"2024-02-10T08:40:35.021521+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:40:35.021521+0000","last_undegraded":"2024-02-10T08:40:35.021521+0000","last_fullsized":"2024-02-10T08:40:35.021521+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:39:46.646454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.646454+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"258'103","reported_seq":366,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018219+0000","last_change":"2024-02-10T08:39:41.578640+0000","last_active":"2024-02-10T08:40:35.018219+0000","last_peered":"2024-02-10T08:40:35.018219+0000","last_clean":"2024-02-10T08:40:35.018219+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:40:35.018219+0000","last_undegraded":"2024-02-10T08:40:35.018219+0000","last_fullsized":"2024-02-10T08:40:35.018219+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":353,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021353+0000","last_change":"2024-02-10T08:39:55.527916+0000","last_active":"2024-02-10T08:40:35.021353+0000","last_peered":"2024-02-10T08:40:35.021353+0000","last_clean":"2024-02-10T08:40:35.021353+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:40:35.021353+0000","last_undegraded":"2024-02-10T08:40:35.021353+0000","last_fullsized":"2024-02-10T08:40:35.021353+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:39:55.527854+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"215'85","reported_seq":355,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023324+0000","last_change":"2024-02-10T08:39:54.516959+0000","last_active":"2024-02-10T08:40:35.023324+0000","last_peered":"2024-02-10T08:40:35.023324+0000","last_clean":"2024-02-10T08:40:35.023324+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:40:35.023324+0000","last_undegraded":"2024-02-10T08:40:35.023324+0000","last_fullsized":"2024-02-10T08:40:35.023324+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'85","last_scrub_stamp":"2024-02-10T08:39:54.516887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.516887+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":357,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997860+0000","last_change":"2024-02-10T08:39:43.711787+0000","last_active":"2024-02-10T08:40:34.997860+0000","last_peered":"2024-02-10T08:40:34.997860+0000","last_clean":"2024-02-10T08:40:34.997860+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:40:34.997860+0000","last_undegraded":"2024-02-10T08:40:34.997860+0000","last_fullsized":"2024-02-10T08:40:34.997860+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"338'18","reported_seq":299,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996885+0000","last_change":"2024-02-10T08:39:56.545155+0000","last_active":"2024-02-10T08:40:34.996885+0000","last_peered":"2024-02-10T08:40:34.996885+0000","last_clean":"2024-02-10T08:40:34.996885+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:40:34.996885+0000","last_undegraded":"2024-02-10T08:40:34.996885+0000","last_fullsized":"2024-02-10T08:40:34.996885+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":310,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997435+0000","last_change":"2024-02-10T08:39:49.714905+0000","last_active":"2024-02-10T08:40:34.997435+0000","last_peered":"2024-02-10T08:40:34.997435+0000","last_clean":"2024-02-10T08:40:34.997435+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:40:34.997435+0000","last_undegraded":"2024-02-10T08:40:34.997435+0000","last_fullsized":"2024-02-10T08:40:34.997435+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"338'69","reported_seq":325,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995670+0000","last_change":"2024-02-10T08:40:24.583295+0000","last_active":"2024-02-10T08:40:34.995670+0000","last_peered":"2024-02-10T08:40:34.995670+0000","last_clean":"2024-02-10T08:40:34.995670+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:40:34.995670+0000","last_undegraded":"2024-02-10T08:40:34.995670+0000","last_fullsized":"2024-02-10T08:40:34.995670+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"338'69","reported_seq":333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.307708+0000","last_change":"2024-02-10T08:40:35.307685+0000","last_active":"2024-02-10T08:40:35.307708+0000","last_peered":"2024-02-10T08:40:35.307708+0000","last_clean":"2024-02-10T08:40:35.307708+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:40:35.307708+0000","last_undegraded":"2024-02-10T08:40:35.307708+0000","last_fullsized":"2024-02-10T08:40:35.307708+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:40:35.307608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.307608+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995981+0000","last_change":"2024-02-10T08:40:28.186561+0000","last_active":"2024-02-10T08:40:34.995981+0000","last_peered":"2024-02-10T08:40:34.995981+0000","last_clean":"2024-02-10T08:40:34.995981+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:40:34.995981+0000","last_undegraded":"2024-02-10T08:40:34.995981+0000","last_fullsized":"2024-02-10T08:40:34.995981+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'34","last_scrub_stamp":"2024-02-10T08:40:28.186484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.186484+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"338'35","reported_seq":294,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995000+0000","last_change":"2024-02-10T08:39:54.602656+0000","last_active":"2024-02-10T08:40:34.995000+0000","last_peered":"2024-02-10T08:40:34.995000+0000","last_clean":"2024-02-10T08:40:34.995000+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:40:34.995000+0000","last_undegraded":"2024-02-10T08:40:34.995000+0000","last_fullsized":"2024-02-10T08:40:34.995000+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"338'36","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021355+0000","last_change":"2024-02-10T08:39:33.014676+0000","last_active":"2024-02-10T08:40:35.021355+0000","last_peered":"2024-02-10T08:40:35.021355+0000","last_clean":"2024-02-10T08:40:35.021355+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:40:35.021355+0000","last_undegraded":"2024-02-10T08:40:35.021355+0000","last_fullsized":"2024-02-10T08:40:35.021355+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'34","last_scrub_stamp":"2024-02-10T08:39:04.785239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.785239+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":279,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994707+0000","last_change":"2024-02-10T08:39:38.041298+0000","last_active":"2024-02-10T08:40:34.994707+0000","last_peered":"2024-02-10T08:40:34.994707+0000","last_clean":"2024-02-10T08:40:34.994707+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:40:34.994707+0000","last_undegraded":"2024-02-10T08:40:34.994707+0000","last_fullsized":"2024-02-10T08:40:34.994707+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:39:38.041185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.041185+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"338'19","reported_seq":284,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997445+0000","last_change":"2024-02-10T08:40:30.882955+0000","last_active":"2024-02-10T08:40:34.997445+0000","last_peered":"2024-02-10T08:40:34.997445+0000","last_clean":"2024-02-10T08:40:34.997445+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:40:34.997445+0000","last_undegraded":"2024-02-10T08:40:34.997445+0000","last_fullsized":"2024-02-10T08:40:34.997445+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"222'34","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.400885+0000","last_change":"2024-02-10T08:40:23.345616+0000","last_active":"2024-02-10T08:40:35.400885+0000","last_peered":"2024-02-10T08:40:35.400885+0000","last_clean":"2024-02-10T08:40:35.400885+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:40:35.400885+0000","last_undegraded":"2024-02-10T08:40:35.400885+0000","last_fullsized":"2024-02-10T08:40:35.400885+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:40:23.345538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.345538+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":420,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022791+0000","last_change":"2024-02-10T08:38:21.488646+0000","last_active":"2024-02-10T08:40:35.022791+0000","last_peered":"2024-02-10T08:40:35.022791+0000","last_clean":"2024-02-10T08:40:35.022791+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:40:35.022791+0000","last_undegraded":"2024-02-10T08:40:35.022791+0000","last_fullsized":"2024-02-10T08:40:35.022791+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:38:21.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:21.488563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":338,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018515+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:40:35.018515+0000","last_peered":"2024-02-10T08:40:35.018515+0000","last_clean":"2024-02-10T08:40:35.018515+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:40:35.018515+0000","last_undegraded":"2024-02-10T08:40:35.018515+0000","last_fullsized":"2024-02-10T08:40:35.018515+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402666+0000","last_change":"2024-02-10T08:37:19.833350+0000","last_active":"2024-02-10T08:40:35.402666+0000","last_peered":"2024-02-10T08:40:35.402666+0000","last_clean":"2024-02-10T08:40:35.402666+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:40:35.402666+0000","last_undegraded":"2024-02-10T08:40:35.402666+0000","last_fullsized":"2024-02-10T08:40:35.402666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.833287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":316,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018897+0000","last_change":"2024-02-10T08:40:26.859788+0000","last_active":"2024-02-10T08:40:35.018897+0000","last_peered":"2024-02-10T08:40:35.018897+0000","last_clean":"2024-02-10T08:40:35.018897+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:40:35.018897+0000","last_undegraded":"2024-02-10T08:40:35.018897+0000","last_fullsized":"2024-02-10T08:40:35.018897+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.859697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.859697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401731+0000","last_change":"2024-02-10T08:38:33.920702+0000","last_active":"2024-02-10T08:40:35.401731+0000","last_peered":"2024-02-10T08:40:35.401731+0000","last_clean":"2024-02-10T08:40:35.401731+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:40:35.401731+0000","last_undegraded":"2024-02-10T08:40:35.401731+0000","last_fullsized":"2024-02-10T08:40:35.401731+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.920630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.920630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":282,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402048+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:40:35.402048+0000","last_peered":"2024-02-10T08:40:35.402048+0000","last_clean":"2024-02-10T08:40:35.402048+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:40:35.402048+0000","last_undegraded":"2024-02-10T08:40:35.402048+0000","last_fullsized":"2024-02-10T08:40:35.402048+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":310,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995973+0000","last_change":"2024-02-10T08:37:42.778359+0000","last_active":"2024-02-10T08:40:34.995973+0000","last_peered":"2024-02-10T08:40:34.995973+0000","last_clean":"2024-02-10T08:40:34.995973+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:40:34.995973+0000","last_undegraded":"2024-02-10T08:40:34.995973+0000","last_fullsized":"2024-02-10T08:40:34.995973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.778288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.778288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":324,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401605+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:40:35.401605+0000","last_peered":"2024-02-10T08:40:35.401605+0000","last_clean":"2024-02-10T08:40:35.401605+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:40:35.401605+0000","last_undegraded":"2024-02-10T08:40:35.401605+0000","last_fullsized":"2024-02-10T08:40:35.401605+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":222,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022313+0000","last_change":"2024-02-10T08:37:44.518168+0000","last_active":"2024-02-10T08:40:35.022313+0000","last_peered":"2024-02-10T08:40:35.022313+0000","last_clean":"2024-02-10T08:40:35.022313+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:40:35.022313+0000","last_undegraded":"2024-02-10T08:40:35.022313+0000","last_fullsized":"2024-02-10T08:40:35.022313+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.518084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.518084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":251,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021759+0000","last_change":"2024-02-10T08:38:26.677780+0000","last_active":"2024-02-10T08:40:35.021759+0000","last_peered":"2024-02-10T08:40:35.021759+0000","last_clean":"2024-02-10T08:40:35.021759+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:40:35.021759+0000","last_undegraded":"2024-02-10T08:40:35.021759+0000","last_fullsized":"2024-02-10T08:40:35.021759+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.677701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.677701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":342,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374351+0000","last_change":"2024-02-10T08:40:24.395617+0000","last_active":"2024-02-10T08:40:35.374351+0000","last_peered":"2024-02-10T08:40:35.374351+0000","last_clean":"2024-02-10T08:40:35.374351+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:40:35.374351+0000","last_undegraded":"2024-02-10T08:40:35.374351+0000","last_fullsized":"2024-02-10T08:40:35.374351+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:40:24.395541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.395541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":314,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402042+0000","last_change":"2024-02-10T08:37:10.205470+0000","last_active":"2024-02-10T08:40:35.402042+0000","last_peered":"2024-02-10T08:40:35.402042+0000","last_clean":"2024-02-10T08:40:35.402042+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:40:35.402042+0000","last_undegraded":"2024-02-10T08:40:35.402042+0000","last_fullsized":"2024-02-10T08:40:35.402042+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:08.820587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:08.820587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401912+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:40:35.401912+0000","last_peered":"2024-02-10T08:40:35.401912+0000","last_clean":"2024-02-10T08:40:35.401912+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:40:35.401912+0000","last_undegraded":"2024-02-10T08:40:35.401912+0000","last_fullsized":"2024-02-10T08:40:35.401912+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021445+0000","last_change":"2024-02-10T08:40:28.426111+0000","last_active":"2024-02-10T08:40:35.021445+0000","last_peered":"2024-02-10T08:40:35.021445+0000","last_clean":"2024-02-10T08:40:35.021445+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:40:35.021445+0000","last_undegraded":"2024-02-10T08:40:35.021445+0000","last_fullsized":"2024-02-10T08:40:35.021445+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:28.425994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.425994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":252,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022898+0000","last_change":"2024-02-10T08:39:44.585775+0000","last_active":"2024-02-10T08:40:35.022898+0000","last_peered":"2024-02-10T08:40:35.022898+0000","last_clean":"2024-02-10T08:40:35.022898+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:40:35.022898+0000","last_undegraded":"2024-02-10T08:40:35.022898+0000","last_fullsized":"2024-02-10T08:40:35.022898+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:44.585674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.585674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":336,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997291+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:40:34.997291+0000","last_peered":"2024-02-10T08:40:34.997291+0000","last_clean":"2024-02-10T08:40:34.997291+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:40:34.997291+0000","last_undegraded":"2024-02-10T08:40:34.997291+0000","last_fullsized":"2024-02-10T08:40:34.997291+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":144,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997088+0000","last_change":"2024-02-10T08:38:22.013062+0000","last_active":"2024-02-10T08:40:34.997088+0000","last_peered":"2024-02-10T08:40:34.997088+0000","last_clean":"2024-02-10T08:40:34.997088+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:40:34.997088+0000","last_undegraded":"2024-02-10T08:40:34.997088+0000","last_fullsized":"2024-02-10T08:40:34.997088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996508+0000","last_change":"2024-02-10T08:37:10.149585+0000","last_active":"2024-02-10T08:40:34.996508+0000","last_peered":"2024-02-10T08:40:34.996508+0000","last_clean":"2024-02-10T08:40:34.996508+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:40:34.996508+0000","last_undegraded":"2024-02-10T08:40:34.996508+0000","last_fullsized":"2024-02-10T08:40:34.996508+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:06.682474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:06.682474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402795+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:40:35.402795+0000","last_peered":"2024-02-10T08:40:35.402795+0000","last_clean":"2024-02-10T08:40:35.402795+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:40:35.402795+0000","last_undegraded":"2024-02-10T08:40:35.402795+0000","last_fullsized":"2024-02-10T08:40:35.402795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995449+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:40:34.995449+0000","last_peered":"2024-02-10T08:40:34.995449+0000","last_clean":"2024-02-10T08:40:34.995449+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:40:34.995449+0000","last_undegraded":"2024-02-10T08:40:34.995449+0000","last_fullsized":"2024-02-10T08:40:34.995449+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":323,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996133+0000","last_change":"2024-02-10T08:38:17.727465+0000","last_active":"2024-02-10T08:40:34.996133+0000","last_peered":"2024-02-10T08:40:34.996133+0000","last_clean":"2024-02-10T08:40:34.996133+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:40:34.996133+0000","last_undegraded":"2024-02-10T08:40:34.996133+0000","last_fullsized":"2024-02-10T08:40:34.996133+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:17.727387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:17.727387+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":310,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995149+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:40:34.995149+0000","last_peered":"2024-02-10T08:40:34.995149+0000","last_clean":"2024-02-10T08:40:34.995149+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:40:34.995149+0000","last_undegraded":"2024-02-10T08:40:34.995149+0000","last_fullsized":"2024-02-10T08:40:34.995149+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":416,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402512+0000","last_change":"2024-02-10T08:37:11.819571+0000","last_active":"2024-02-10T08:40:35.402512+0000","last_peered":"2024-02-10T08:40:35.402512+0000","last_clean":"2024-02-10T08:40:35.402512+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:40:35.402512+0000","last_undegraded":"2024-02-10T08:40:35.402512+0000","last_fullsized":"2024-02-10T08:40:35.402512+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:37:11.819443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:11.819443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997498+0000","last_change":"2024-02-10T08:38:25.851680+0000","last_active":"2024-02-10T08:40:34.997498+0000","last_peered":"2024-02-10T08:40:34.997498+0000","last_clean":"2024-02-10T08:40:34.997498+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:40:34.997498+0000","last_undegraded":"2024-02-10T08:40:34.997498+0000","last_fullsized":"2024-02-10T08:40:34.997498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.851576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.851576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995129+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:40:34.995129+0000","last_peered":"2024-02-10T08:40:34.995129+0000","last_clean":"2024-02-10T08:40:34.995129+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:40:34.995129+0000","last_undegraded":"2024-02-10T08:40:34.995129+0000","last_fullsized":"2024-02-10T08:40:34.995129+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997850+0000","last_change":"2024-02-10T08:37:21.583921+0000","last_active":"2024-02-10T08:40:34.997850+0000","last_peered":"2024-02-10T08:40:34.997850+0000","last_clean":"2024-02-10T08:40:34.997850+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:40:34.997850+0000","last_undegraded":"2024-02-10T08:40:34.997850+0000","last_fullsized":"2024-02-10T08:40:34.997850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:21.583846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.583846+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996310+0000","last_change":"2024-02-10T08:38:18.008913+0000","last_active":"2024-02-10T08:40:34.996310+0000","last_peered":"2024-02-10T08:40:34.996310+0000","last_clean":"2024-02-10T08:40:34.996310+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:40:34.996310+0000","last_undegraded":"2024-02-10T08:40:34.996310+0000","last_fullsized":"2024-02-10T08:40:34.996310+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:18.008846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.008846+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":446,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997410+0000","last_change":"2024-02-10T08:40:25.631962+0000","last_active":"2024-02-10T08:40:34.997410+0000","last_peered":"2024-02-10T08:40:34.997410+0000","last_clean":"2024-02-10T08:40:34.997410+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:40:34.997410+0000","last_undegraded":"2024-02-10T08:40:34.997410+0000","last_fullsized":"2024-02-10T08:40:34.997410+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:40:25.631854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:25.631854+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":311,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.020354+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:40:35.020354+0000","last_peered":"2024-02-10T08:40:35.020354+0000","last_clean":"2024-02-10T08:40:35.020354+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:40:35.020354+0000","last_undegraded":"2024-02-10T08:40:35.020354+0000","last_fullsized":"2024-02-10T08:40:35.020354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":251,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019130+0000","last_change":"2024-02-10T08:39:05.788196+0000","last_active":"2024-02-10T08:40:35.019130+0000","last_peered":"2024-02-10T08:40:35.019130+0000","last_clean":"2024-02-10T08:40:35.019130+0000","last_became_active":"2024-02-10T08:36:37.148530+0000","last_became_peered":"2024-02-10T08:36:37.148530+0000","last_unstale":"2024-02-10T08:40:35.019130+0000","last_undegraded":"2024-02-10T08:40:35.019130+0000","last_fullsized":"2024-02-10T08:40:35.019130+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.788118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.788118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":327,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997307+0000","last_change":"2024-02-10T08:38:59.745198+0000","last_active":"2024-02-10T08:40:34.997307+0000","last_peered":"2024-02-10T08:40:34.997307+0000","last_clean":"2024-02-10T08:40:34.997307+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:40:34.997307+0000","last_undegraded":"2024-02-10T08:40:34.997307+0000","last_fullsized":"2024-02-10T08:40:34.997307+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:38:59.745114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.745114+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021584+0000","last_change":"2024-02-10T08:38:33.363322+0000","last_active":"2024-02-10T08:40:35.021584+0000","last_peered":"2024-02-10T08:40:35.021584+0000","last_clean":"2024-02-10T08:40:35.021584+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:40:35.021584+0000","last_undegraded":"2024-02-10T08:40:35.021584+0000","last_fullsized":"2024-02-10T08:40:35.021584+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.363195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":372,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021653+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:40:35.021653+0000","last_peered":"2024-02-10T08:40:35.021653+0000","last_clean":"2024-02-10T08:40:35.021653+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:40:35.021653+0000","last_undegraded":"2024-02-10T08:40:35.021653+0000","last_fullsized":"2024-02-10T08:40:35.021653+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":298,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997813+0000","last_change":"2024-02-10T08:37:23.368638+0000","last_active":"2024-02-10T08:40:34.997813+0000","last_peered":"2024-02-10T08:40:34.997813+0000","last_clean":"2024-02-10T08:40:34.997813+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:40:34.997813+0000","last_undegraded":"2024-02-10T08:40:34.997813+0000","last_fullsized":"2024-02-10T08:40:34.997813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:23.368545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:23.368545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994648+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:40:34.994648+0000","last_peered":"2024-02-10T08:40:34.994648+0000","last_clean":"2024-02-10T08:40:34.994648+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:40:34.994648+0000","last_undegraded":"2024-02-10T08:40:34.994648+0000","last_fullsized":"2024-02-10T08:40:34.994648+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":403,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995436+0000","last_change":"2024-02-10T08:38:18.475575+0000","last_active":"2024-02-10T08:40:34.995436+0000","last_peered":"2024-02-10T08:40:34.995436+0000","last_clean":"2024-02-10T08:40:34.995436+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:40:34.995436+0000","last_undegraded":"2024-02-10T08:40:34.995436+0000","last_fullsized":"2024-02-10T08:40:34.995436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:38:18.475509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.475509+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":329,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995681+0000","last_change":"2024-02-10T08:38:31.840285+0000","last_active":"2024-02-10T08:40:34.995681+0000","last_peered":"2024-02-10T08:40:34.995681+0000","last_clean":"2024-02-10T08:40:34.995681+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:40:34.995681+0000","last_undegraded":"2024-02-10T08:40:34.995681+0000","last_fullsized":"2024-02-10T08:40:34.995681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:38:31.840218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.840218+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375596+0000","last_change":"2024-02-10T08:39:48.521060+0000","last_active":"2024-02-10T08:40:35.375596+0000","last_peered":"2024-02-10T08:40:35.375596+0000","last_clean":"2024-02-10T08:40:35.375596+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:40:35.375596+0000","last_undegraded":"2024-02-10T08:40:35.375596+0000","last_fullsized":"2024-02-10T08:40:35.375596+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:48.520941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:48.520941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023032+0000","last_change":"2024-02-10T08:39:56.568250+0000","last_active":"2024-02-10T08:40:35.023032+0000","last_peered":"2024-02-10T08:40:35.023032+0000","last_clean":"2024-02-10T08:40:35.023032+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:40:35.023032+0000","last_undegraded":"2024-02-10T08:40:35.023032+0000","last_fullsized":"2024-02-10T08:40:35.023032+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:56.568189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.568189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996050+0000","last_change":"2024-02-10T08:39:12.085361+0000","last_active":"2024-02-10T08:40:34.996050+0000","last_peered":"2024-02-10T08:40:34.996050+0000","last_clean":"2024-02-10T08:40:34.996050+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:40:34.996050+0000","last_undegraded":"2024-02-10T08:40:34.996050+0000","last_fullsized":"2024-02-10T08:40:34.996050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.085261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.085261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995838+0000","last_change":"2024-02-10T08:40:34.622947+0000","last_active":"2024-02-10T08:40:34.995838+0000","last_peered":"2024-02-10T08:40:34.995838+0000","last_clean":"2024-02-10T08:40:34.995838+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:40:34.995838+0000","last_undegraded":"2024-02-10T08:40:34.995838+0000","last_fullsized":"2024-02-10T08:40:34.995838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:34.622869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":369,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995349+0000","last_change":"2024-02-10T08:37:39.542513+0000","last_active":"2024-02-10T08:40:34.995349+0000","last_peered":"2024-02-10T08:40:34.995349+0000","last_clean":"2024-02-10T08:40:34.995349+0000","last_became_active":"2024-02-10T08:37:10.303971+0000","last_became_peered":"2024-02-10T08:37:10.303971+0000","last_unstale":"2024-02-10T08:40:34.995349+0000","last_undegraded":"2024-02-10T08:40:34.995349+0000","last_fullsized":"2024-02-10T08:40:34.995349+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":317,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994881+0000","last_change":"2024-02-10T08:38:57.920239+0000","last_active":"2024-02-10T08:40:34.994881+0000","last_peered":"2024-02-10T08:40:34.994881+0000","last_clean":"2024-02-10T08:40:34.994881+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:40:34.994881+0000","last_undegraded":"2024-02-10T08:40:34.994881+0000","last_fullsized":"2024-02-10T08:40:34.994881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:57.920144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.920144+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":299,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996025+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:40:34.996025+0000","last_peered":"2024-02-10T08:40:34.996025+0000","last_clean":"2024-02-10T08:40:34.996025+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:40:34.996025+0000","last_undegraded":"2024-02-10T08:40:34.996025+0000","last_fullsized":"2024-02-10T08:40:34.996025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997575+0000","last_change":"2024-02-10T08:39:08.629231+0000","last_active":"2024-02-10T08:40:34.997575+0000","last_peered":"2024-02-10T08:40:34.997575+0000","last_clean":"2024-02-10T08:40:34.997575+0000","last_became_active":"2024-02-10T08:37:10.244361+0000","last_became_peered":"2024-02-10T08:37:10.244361+0000","last_unstale":"2024-02-10T08:40:34.997575+0000","last_undegraded":"2024-02-10T08:40:34.997575+0000","last_fullsized":"2024-02-10T08:40:34.997575+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374402+0000","last_change":"2024-02-10T08:40:32.412832+0000","last_active":"2024-02-10T08:40:35.374402+0000","last_peered":"2024-02-10T08:40:35.374402+0000","last_clean":"2024-02-10T08:40:35.374402+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:40:35.374402+0000","last_undegraded":"2024-02-10T08:40:35.374402+0000","last_fullsized":"2024-02-10T08:40:35.374402+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:32.412763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.412763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":287,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019293+0000","last_change":"2024-02-10T08:38:18.588026+0000","last_active":"2024-02-10T08:40:35.019293+0000","last_peered":"2024-02-10T08:40:35.019293+0000","last_clean":"2024-02-10T08:40:35.019293+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:40:35.019293+0000","last_undegraded":"2024-02-10T08:40:35.019293+0000","last_fullsized":"2024-02-10T08:40:35.019293+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:38:18.587949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.587949+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023133+0000","last_change":"2024-02-10T08:38:13.655575+0000","last_active":"2024-02-10T08:40:35.023133+0000","last_peered":"2024-02-10T08:40:35.023133+0000","last_clean":"2024-02-10T08:40:35.023133+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:40:35.023133+0000","last_undegraded":"2024-02-10T08:40:35.023133+0000","last_fullsized":"2024-02-10T08:40:35.023133+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.655504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.655504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996466+0000","last_change":"2024-02-10T08:38:26.946622+0000","last_active":"2024-02-10T08:40:34.996466+0000","last_peered":"2024-02-10T08:40:34.996466+0000","last_clean":"2024-02-10T08:40:34.996466+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:40:34.996466+0000","last_undegraded":"2024-02-10T08:40:34.996466+0000","last_fullsized":"2024-02-10T08:40:34.996466+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.946531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.946531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402698+0000","last_change":"2024-02-10T08:38:25.903816+0000","last_active":"2024-02-10T08:40:35.402698+0000","last_peered":"2024-02-10T08:40:35.402698+0000","last_clean":"2024-02-10T08:40:35.402698+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:40:35.402698+0000","last_undegraded":"2024-02-10T08:40:35.402698+0000","last_fullsized":"2024-02-10T08:40:35.402698+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.903753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.903753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":331,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021671+0000","last_change":"2024-02-10T08:38:31.745438+0000","last_active":"2024-02-10T08:40:35.021671+0000","last_peered":"2024-02-10T08:40:35.021671+0000","last_clean":"2024-02-10T08:40:35.021671+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:40:35.021671+0000","last_undegraded":"2024-02-10T08:40:35.021671+0000","last_fullsized":"2024-02-10T08:40:35.021671+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.745362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.745362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":267,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995020+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:40:34.995020+0000","last_peered":"2024-02-10T08:40:34.995020+0000","last_clean":"2024-02-10T08:40:34.995020+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:40:34.995020+0000","last_undegraded":"2024-02-10T08:40:34.995020+0000","last_fullsized":"2024-02-10T08:40:34.995020+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":323,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021463+0000","last_change":"2024-02-10T08:40:23.406297+0000","last_active":"2024-02-10T08:40:35.021463+0000","last_peered":"2024-02-10T08:40:35.021463+0000","last_clean":"2024-02-10T08:40:35.021463+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:40:35.021463+0000","last_undegraded":"2024-02-10T08:40:35.021463+0000","last_fullsized":"2024-02-10T08:40:35.021463+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.406227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.406227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995359+0000","last_change":"2024-02-10T08:38:18.758911+0000","last_active":"2024-02-10T08:40:34.995359+0000","last_peered":"2024-02-10T08:40:34.995359+0000","last_clean":"2024-02-10T08:40:34.995359+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:40:34.995359+0000","last_undegraded":"2024-02-10T08:40:34.995359+0000","last_fullsized":"2024-02-10T08:40:34.995359+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.758843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.758843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996578+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:40:34.996578+0000","last_peered":"2024-02-10T08:40:34.996578+0000","last_clean":"2024-02-10T08:40:34.996578+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:40:34.996578+0000","last_undegraded":"2024-02-10T08:40:34.996578+0000","last_fullsized":"2024-02-10T08:40:34.996578+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":260,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374205+0000","last_change":"2024-02-10T08:39:06.770922+0000","last_active":"2024-02-10T08:40:35.374205+0000","last_peered":"2024-02-10T08:40:35.374205+0000","last_clean":"2024-02-10T08:40:35.374205+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:40:35.374205+0000","last_undegraded":"2024-02-10T08:40:35.374205+0000","last_fullsized":"2024-02-10T08:40:35.374205+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.770850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.770850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022659+0000","last_change":"2024-02-10T08:39:38.645588+0000","last_active":"2024-02-10T08:40:35.022659+0000","last_peered":"2024-02-10T08:40:35.022659+0000","last_clean":"2024-02-10T08:40:35.022659+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:40:35.022659+0000","last_undegraded":"2024-02-10T08:40:35.022659+0000","last_fullsized":"2024-02-10T08:40:35.022659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.645527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.645527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.377020+0000","last_change":"2024-02-10T08:40:35.376945+0000","last_active":"2024-02-10T08:40:35.377020+0000","last_peered":"2024-02-10T08:40:35.377020+0000","last_clean":"2024-02-10T08:40:35.377020+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:40:35.377020+0000","last_undegraded":"2024-02-10T08:40:35.377020+0000","last_fullsized":"2024-02-10T08:40:35.377020+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:40:35.376859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.376859+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":326,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021542+0000","last_change":"2024-02-10T08:39:36.639528+0000","last_active":"2024-02-10T08:40:35.021542+0000","last_peered":"2024-02-10T08:40:35.021542+0000","last_clean":"2024-02-10T08:40:35.021542+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:40:35.021542+0000","last_undegraded":"2024-02-10T08:40:35.021542+0000","last_fullsized":"2024-02-10T08:40:35.021542+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.639461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.639461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019355+0000","last_change":"2024-02-10T08:38:25.633733+0000","last_active":"2024-02-10T08:40:35.019355+0000","last_peered":"2024-02-10T08:40:35.019355+0000","last_clean":"2024-02-10T08:40:35.019355+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:40:35.019355+0000","last_undegraded":"2024-02-10T08:40:35.019355+0000","last_fullsized":"2024-02-10T08:40:35.019355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.633608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.633608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994920+0000","last_change":"2024-02-10T08:37:25.417836+0000","last_active":"2024-02-10T08:40:34.994920+0000","last_peered":"2024-02-10T08:40:34.994920+0000","last_clean":"2024-02-10T08:40:34.994920+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:40:34.994920+0000","last_undegraded":"2024-02-10T08:40:34.994920+0000","last_fullsized":"2024-02-10T08:40:34.994920+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:25.417773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:25.417773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":302,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021712+0000","last_change":"2024-02-10T08:39:00.751654+0000","last_active":"2024-02-10T08:40:35.021712+0000","last_peered":"2024-02-10T08:40:35.021712+0000","last_clean":"2024-02-10T08:40:35.021712+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:40:35.021712+0000","last_undegraded":"2024-02-10T08:40:35.021712+0000","last_fullsized":"2024-02-10T08:40:35.021712+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:00.751584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.751584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018795+0000","last_change":"2024-02-10T08:39:11.660136+0000","last_active":"2024-02-10T08:40:35.018795+0000","last_peered":"2024-02-10T08:40:35.018795+0000","last_clean":"2024-02-10T08:40:35.018795+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:40:35.018795+0000","last_undegraded":"2024-02-10T08:40:35.018795+0000","last_fullsized":"2024-02-10T08:40:35.018795+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:11.660049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.660049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":319,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402759+0000","last_change":"2024-02-10T08:37:17.794187+0000","last_active":"2024-02-10T08:40:35.402759+0000","last_peered":"2024-02-10T08:40:35.402759+0000","last_clean":"2024-02-10T08:40:35.402759+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:40:35.402759+0000","last_undegraded":"2024-02-10T08:40:35.402759+0000","last_fullsized":"2024-02-10T08:40:35.402759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:17.794061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:17.794061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":356,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997770+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:40:34.997770+0000","last_peered":"2024-02-10T08:40:34.997770+0000","last_clean":"2024-02-10T08:40:34.997770+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:40:34.997770+0000","last_undegraded":"2024-02-10T08:40:34.997770+0000","last_fullsized":"2024-02-10T08:40:34.997770+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":321,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401366+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:40:35.401366+0000","last_peered":"2024-02-10T08:40:35.401366+0000","last_clean":"2024-02-10T08:40:35.401366+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:40:35.401366+0000","last_undegraded":"2024-02-10T08:40:35.401366+0000","last_fullsized":"2024-02-10T08:40:35.401366+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":745,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.859627+0000","last_change":"2024-02-10T08:39:05.948676+0000","last_active":"2024-02-10T08:40:35.859627+0000","last_peered":"2024-02-10T08:40:35.859627+0000","last_clean":"2024-02-10T08:40:35.859627+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:40:35.859627+0000","last_undegraded":"2024-02-10T08:40:35.859627+0000","last_fullsized":"2024-02-10T08:40:35.859627+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:39:05.948579+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.948579+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023048+0000","last_change":"2024-02-10T08:39:03.251683+0000","last_active":"2024-02-10T08:40:35.023048+0000","last_peered":"2024-02-10T08:40:35.023048+0000","last_clean":"2024-02-10T08:40:35.023048+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:40:35.023048+0000","last_undegraded":"2024-02-10T08:40:35.023048+0000","last_fullsized":"2024-02-10T08:40:35.023048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.251585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.251585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997423+0000","last_change":"2024-02-10T08:40:22.234338+0000","last_active":"2024-02-10T08:40:34.997423+0000","last_peered":"2024-02-10T08:40:34.997423+0000","last_clean":"2024-02-10T08:40:34.997423+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:40:34.997423+0000","last_undegraded":"2024-02-10T08:40:34.997423+0000","last_fullsized":"2024-02-10T08:40:34.997423+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.234270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.234270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995824+0000","last_change":"2024-02-10T08:39:06.965218+0000","last_active":"2024-02-10T08:40:34.995824+0000","last_peered":"2024-02-10T08:40:34.995824+0000","last_clean":"2024-02-10T08:40:34.995824+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:40:34.995824+0000","last_undegraded":"2024-02-10T08:40:34.995824+0000","last_fullsized":"2024-02-10T08:40:34.995824+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.965152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.965152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":380,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019213+0000","last_change":"2024-02-10T08:38:19.613486+0000","last_active":"2024-02-10T08:40:35.019213+0000","last_peered":"2024-02-10T08:40:35.019213+0000","last_clean":"2024-02-10T08:40:35.019213+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:40:35.019213+0000","last_undegraded":"2024-02-10T08:40:35.019213+0000","last_fullsized":"2024-02-10T08:40:35.019213+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:38:19.613422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.613422+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019008+0000","last_change":"2024-02-10T08:40:23.818554+0000","last_active":"2024-02-10T08:40:35.019008+0000","last_peered":"2024-02-10T08:40:35.019008+0000","last_clean":"2024-02-10T08:40:35.019008+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:40:35.019008+0000","last_undegraded":"2024-02-10T08:40:35.019008+0000","last_fullsized":"2024-02-10T08:40:35.019008+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.818434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.818434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":298,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998163+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:40:34.998163+0000","last_peered":"2024-02-10T08:40:34.998163+0000","last_clean":"2024-02-10T08:40:34.998163+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:40:34.998163+0000","last_undegraded":"2024-02-10T08:40:34.998163+0000","last_fullsized":"2024-02-10T08:40:34.998163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401201+0000","last_change":"2024-02-10T08:38:57.168613+0000","last_active":"2024-02-10T08:40:35.401201+0000","last_peered":"2024-02-10T08:40:35.401201+0000","last_clean":"2024-02-10T08:40:35.401201+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:40:35.401201+0000","last_undegraded":"2024-02-10T08:40:35.401201+0000","last_fullsized":"2024-02-10T08:40:35.401201+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:57.168523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.168523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":290,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374889+0000","last_change":"2024-02-10T08:39:52.548918+0000","last_active":"2024-02-10T08:40:35.374889+0000","last_peered":"2024-02-10T08:40:35.374889+0000","last_clean":"2024-02-10T08:40:35.374889+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:40:35.374889+0000","last_undegraded":"2024-02-10T08:40:35.374889+0000","last_fullsized":"2024-02-10T08:40:35.374889+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:52.548839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.548839+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995390+0000","last_change":"2024-02-10T08:37:45.825263+0000","last_active":"2024-02-10T08:40:34.995390+0000","last_peered":"2024-02-10T08:40:34.995390+0000","last_clean":"2024-02-10T08:40:34.995390+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:40:34.995390+0000","last_undegraded":"2024-02-10T08:40:34.995390+0000","last_fullsized":"2024-02-10T08:40:34.995390+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:37:45.825155+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:37:45.825155+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":319,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996383+0000","last_change":"2024-02-10T08:38:59.955336+0000","last_active":"2024-02-10T08:40:34.996383+0000","last_peered":"2024-02-10T08:40:34.996383+0000","last_clean":"2024-02-10T08:40:34.996383+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:40:34.996383+0000","last_undegraded":"2024-02-10T08:40:34.996383+0000","last_fullsized":"2024-02-10T08:40:34.996383+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.955272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.955272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995263+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:40:34.995263+0000","last_peered":"2024-02-10T08:40:34.995263+0000","last_clean":"2024-02-10T08:40:34.995263+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:40:34.995263+0000","last_undegraded":"2024-02-10T08:40:34.995263+0000","last_fullsized":"2024-02-10T08:40:34.995263+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997934+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:40:34.997934+0000","last_peered":"2024-02-10T08:40:34.997934+0000","last_clean":"2024-02-10T08:40:34.997934+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:40:34.997934+0000","last_undegraded":"2024-02-10T08:40:34.997934+0000","last_fullsized":"2024-02-10T08:40:34.997934+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401350+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:40:35.401350+0000","last_peered":"2024-02-10T08:40:35.401350+0000","last_clean":"2024-02-10T08:40:35.401350+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:40:35.401350+0000","last_undegraded":"2024-02-10T08:40:35.401350+0000","last_fullsized":"2024-02-10T08:40:35.401350+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":336,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995820+0000","last_change":"2024-02-10T08:38:32.886182+0000","last_active":"2024-02-10T08:40:34.995820+0000","last_peered":"2024-02-10T08:40:34.995820+0000","last_clean":"2024-02-10T08:40:34.995820+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:40:34.995820+0000","last_undegraded":"2024-02-10T08:40:34.995820+0000","last_fullsized":"2024-02-10T08:40:34.995820+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:38:32.886090+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.886090+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997830+0000","last_change":"2024-02-10T08:37:10.321065+0000","last_active":"2024-02-10T08:40:34.997830+0000","last_peered":"2024-02-10T08:40:34.997830+0000","last_clean":"2024-02-10T08:40:34.997830+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:40:34.997830+0000","last_undegraded":"2024-02-10T08:40:34.997830+0000","last_fullsized":"2024-02-10T08:40:34.997830+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:37:07.577830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.577830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997504+0000","last_change":"2024-02-10T08:38:22.407565+0000","last_active":"2024-02-10T08:40:34.997504+0000","last_peered":"2024-02-10T08:40:34.997504+0000","last_clean":"2024-02-10T08:40:34.997504+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:40:34.997504+0000","last_undegraded":"2024-02-10T08:40:34.997504+0000","last_fullsized":"2024-02-10T08:40:34.997504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:22.407476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:22.407476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402472+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:40:35.402472+0000","last_peered":"2024-02-10T08:40:35.402472+0000","last_clean":"2024-02-10T08:40:35.402472+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:40:35.402472+0000","last_undegraded":"2024-02-10T08:40:35.402472+0000","last_fullsized":"2024-02-10T08:40:35.402472+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":289,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998204+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:40:34.998204+0000","last_peered":"2024-02-10T08:40:34.998204+0000","last_clean":"2024-02-10T08:40:34.998204+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:40:34.998204+0000","last_undegraded":"2024-02-10T08:40:34.998204+0000","last_fullsized":"2024-02-10T08:40:34.998204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":319,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401337+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:40:35.401337+0000","last_peered":"2024-02-10T08:40:35.401337+0000","last_clean":"2024-02-10T08:40:35.401337+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:40:35.401337+0000","last_undegraded":"2024-02-10T08:40:35.401337+0000","last_fullsized":"2024-02-10T08:40:35.401337+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":299,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401863+0000","last_change":"2024-02-10T08:37:21.769494+0000","last_active":"2024-02-10T08:40:35.401863+0000","last_peered":"2024-02-10T08:40:35.401863+0000","last_clean":"2024-02-10T08:40:35.401863+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:40:35.401863+0000","last_undegraded":"2024-02-10T08:40:35.401863+0000","last_fullsized":"2024-02-10T08:40:35.401863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.769424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.769424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995512+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:40:34.995512+0000","last_peered":"2024-02-10T08:40:34.995512+0000","last_clean":"2024-02-10T08:40:34.995512+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:40:34.995512+0000","last_undegraded":"2024-02-10T08:40:34.995512+0000","last_fullsized":"2024-02-10T08:40:34.995512+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":316,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998656+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:40:34.998656+0000","last_peered":"2024-02-10T08:40:34.998656+0000","last_clean":"2024-02-10T08:40:34.998656+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:40:34.998656+0000","last_undegraded":"2024-02-10T08:40:34.998656+0000","last_fullsized":"2024-02-10T08:40:34.998656+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374232+0000","last_change":"2024-02-10T08:38:30.291597+0000","last_active":"2024-02-10T08:40:35.374232+0000","last_peered":"2024-02-10T08:40:35.374232+0000","last_clean":"2024-02-10T08:40:35.374232+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:40:35.374232+0000","last_undegraded":"2024-02-10T08:40:35.374232+0000","last_fullsized":"2024-02-10T08:40:35.374232+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.291538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.291538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":298,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022940+0000","last_change":"2024-02-10T08:37:22.265229+0000","last_active":"2024-02-10T08:40:35.022940+0000","last_peered":"2024-02-10T08:40:35.022940+0000","last_clean":"2024-02-10T08:40:35.022940+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:40:35.022940+0000","last_undegraded":"2024-02-10T08:40:35.022940+0000","last_fullsized":"2024-02-10T08:40:35.022940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:22.265162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:22.265162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":251,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019145+0000","last_change":"2024-02-10T08:38:32.745050+0000","last_active":"2024-02-10T08:40:35.019145+0000","last_peered":"2024-02-10T08:40:35.019145+0000","last_clean":"2024-02-10T08:40:35.019145+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:40:35.019145+0000","last_undegraded":"2024-02-10T08:40:35.019145+0000","last_fullsized":"2024-02-10T08:40:35.019145+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.744972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.744972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997896+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:40:34.997896+0000","last_peered":"2024-02-10T08:40:34.997896+0000","last_clean":"2024-02-10T08:40:34.997896+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:40:34.997896+0000","last_undegraded":"2024-02-10T08:40:34.997896+0000","last_fullsized":"2024-02-10T08:40:34.997896+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021314+0000","last_change":"2024-02-10T08:40:35.021264+0000","last_active":"2024-02-10T08:40:35.021314+0000","last_peered":"2024-02-10T08:40:35.021314+0000","last_clean":"2024-02-10T08:40:35.021314+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:40:35.021314+0000","last_undegraded":"2024-02-10T08:40:35.021314+0000","last_fullsized":"2024-02-10T08:40:35.021314+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.021169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.021169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":412,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.020192+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:40:35.020192+0000","last_peered":"2024-02-10T08:40:35.020192+0000","last_clean":"2024-02-10T08:40:35.020192+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:40:35.020192+0000","last_undegraded":"2024-02-10T08:40:35.020192+0000","last_fullsized":"2024-02-10T08:40:35.020192+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":222,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023038+0000","last_change":"2024-02-10T08:39:43.632113+0000","last_active":"2024-02-10T08:40:35.023038+0000","last_peered":"2024-02-10T08:40:35.023038+0000","last_clean":"2024-02-10T08:40:35.023038+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:40:35.023038+0000","last_undegraded":"2024-02-10T08:40:35.023038+0000","last_fullsized":"2024-02-10T08:40:35.023038+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:43.632048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.632048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.373924+0000","last_change":"2024-02-10T08:39:01.740757+0000","last_active":"2024-02-10T08:40:35.373924+0000","last_peered":"2024-02-10T08:40:35.373924+0000","last_clean":"2024-02-10T08:40:35.373924+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:40:35.373924+0000","last_undegraded":"2024-02-10T08:40:35.373924+0000","last_fullsized":"2024-02-10T08:40:35.373924+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:01.740686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:01.740686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.020042+0000","last_change":"2024-02-10T08:39:46.641751+0000","last_active":"2024-02-10T08:40:35.020042+0000","last_peered":"2024-02-10T08:40:35.020042+0000","last_clean":"2024-02-10T08:40:35.020042+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:40:35.020042+0000","last_undegraded":"2024-02-10T08:40:35.020042+0000","last_fullsized":"2024-02-10T08:40:35.020042+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:39:46.641683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.641683+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995912+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:40:34.995912+0000","last_peered":"2024-02-10T08:40:34.995912+0000","last_clean":"2024-02-10T08:40:34.995912+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:40:34.995912+0000","last_undegraded":"2024-02-10T08:40:34.995912+0000","last_fullsized":"2024-02-10T08:40:34.995912+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":400,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998088+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:40:34.998088+0000","last_peered":"2024-02-10T08:40:34.998088+0000","last_clean":"2024-02-10T08:40:34.998088+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:40:34.998088+0000","last_undegraded":"2024-02-10T08:40:34.998088+0000","last_fullsized":"2024-02-10T08:40:34.998088+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":299,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997971+0000","last_change":"2024-02-10T08:37:46.583489+0000","last_active":"2024-02-10T08:40:34.997971+0000","last_peered":"2024-02-10T08:40:34.997971+0000","last_clean":"2024-02-10T08:40:34.997971+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:40:34.997971+0000","last_undegraded":"2024-02-10T08:40:34.997971+0000","last_fullsized":"2024-02-10T08:40:34.997971+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:46.583388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:46.583388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":266,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401225+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:40:35.401225+0000","last_peered":"2024-02-10T08:40:35.401225+0000","last_clean":"2024-02-10T08:40:35.401225+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:40:35.401225+0000","last_undegraded":"2024-02-10T08:40:35.401225+0000","last_fullsized":"2024-02-10T08:40:35.401225+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":330,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995233+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:40:34.995233+0000","last_peered":"2024-02-10T08:40:34.995233+0000","last_clean":"2024-02-10T08:40:34.995233+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:40:34.995233+0000","last_undegraded":"2024-02-10T08:40:34.995233+0000","last_fullsized":"2024-02-10T08:40:34.995233+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375056+0000","last_change":"2024-02-10T08:39:46.475770+0000","last_active":"2024-02-10T08:40:35.375056+0000","last_peered":"2024-02-10T08:40:35.375056+0000","last_clean":"2024-02-10T08:40:35.375056+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:40:35.375056+0000","last_undegraded":"2024-02-10T08:40:35.375056+0000","last_fullsized":"2024-02-10T08:40:35.375056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.475678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.475678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":314,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997474+0000","last_change":"2024-02-10T08:38:13.463715+0000","last_active":"2024-02-10T08:40:34.997474+0000","last_peered":"2024-02-10T08:40:34.997474+0000","last_clean":"2024-02-10T08:40:34.997474+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:40:34.997474+0000","last_undegraded":"2024-02-10T08:40:34.997474+0000","last_fullsized":"2024-02-10T08:40:34.997474+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.463644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.463644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018664+0000","last_change":"2024-02-10T08:38:20.592140+0000","last_active":"2024-02-10T08:40:35.018664+0000","last_peered":"2024-02-10T08:40:35.018664+0000","last_clean":"2024-02-10T08:40:35.018664+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:40:35.018664+0000","last_undegraded":"2024-02-10T08:40:35.018664+0000","last_fullsized":"2024-02-10T08:40:35.018664+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:20.592065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.592065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":260,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375334+0000","last_change":"2024-02-10T08:39:12.687989+0000","last_active":"2024-02-10T08:40:35.375334+0000","last_peered":"2024-02-10T08:40:35.375334+0000","last_clean":"2024-02-10T08:40:35.375334+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:40:35.375334+0000","last_undegraded":"2024-02-10T08:40:35.375334+0000","last_fullsized":"2024-02-10T08:40:35.375334+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.687918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.687918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995880+0000","last_change":"2024-02-10T08:37:47.587113+0000","last_active":"2024-02-10T08:40:34.995880+0000","last_peered":"2024-02-10T08:40:34.995880+0000","last_clean":"2024-02-10T08:40:34.995880+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:40:34.995880+0000","last_undegraded":"2024-02-10T08:40:34.995880+0000","last_fullsized":"2024-02-10T08:40:34.995880+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:47.587047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.587047+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":321,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995899+0000","last_change":"2024-02-10T08:39:04.936255+0000","last_active":"2024-02-10T08:40:34.995899+0000","last_peered":"2024-02-10T08:40:34.995899+0000","last_clean":"2024-02-10T08:40:34.995899+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:40:34.995899+0000","last_undegraded":"2024-02-10T08:40:34.995899+0000","last_fullsized":"2024-02-10T08:40:34.995899+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:04.936174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.936174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":301,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995312+0000","last_change":"2024-02-10T08:37:07.696715+0000","last_active":"2024-02-10T08:40:34.995312+0000","last_peered":"2024-02-10T08:40:34.995312+0000","last_clean":"2024-02-10T08:40:34.995312+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:40:34.995312+0000","last_undegraded":"2024-02-10T08:40:34.995312+0000","last_fullsized":"2024-02-10T08:40:34.995312+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:07.696599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:07.696599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":312,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995178+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:40:34.995178+0000","last_peered":"2024-02-10T08:40:34.995178+0000","last_clean":"2024-02-10T08:40:34.995178+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:40:34.995178+0000","last_undegraded":"2024-02-10T08:40:34.995178+0000","last_fullsized":"2024-02-10T08:40:34.995178+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998470+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:40:34.998470+0000","last_peered":"2024-02-10T08:40:34.998470+0000","last_clean":"2024-02-10T08:40:34.998470+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:40:34.998470+0000","last_undegraded":"2024-02-10T08:40:34.998470+0000","last_fullsized":"2024-02-10T08:40:34.998470+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995178+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:40:34.995178+0000","last_peered":"2024-02-10T08:40:34.995178+0000","last_clean":"2024-02-10T08:40:34.995178+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:40:34.995178+0000","last_undegraded":"2024-02-10T08:40:34.995178+0000","last_fullsized":"2024-02-10T08:40:34.995178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.020801+0000","last_change":"2024-02-10T08:38:13.601496+0000","last_active":"2024-02-10T08:40:35.020801+0000","last_peered":"2024-02-10T08:40:35.020801+0000","last_clean":"2024-02-10T08:40:35.020801+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:40:35.020801+0000","last_undegraded":"2024-02-10T08:40:35.020801+0000","last_fullsized":"2024-02-10T08:40:35.020801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.601390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.601390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":306,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374156+0000","last_change":"2024-02-10T08:39:41.555411+0000","last_active":"2024-02-10T08:40:35.374156+0000","last_peered":"2024-02-10T08:40:35.374156+0000","last_clean":"2024-02-10T08:40:35.374156+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:40:35.374156+0000","last_undegraded":"2024-02-10T08:40:35.374156+0000","last_fullsized":"2024-02-10T08:40:35.374156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.555339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.555339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022017+0000","last_change":"2024-02-10T08:37:49.657292+0000","last_active":"2024-02-10T08:40:35.022017+0000","last_peered":"2024-02-10T08:40:35.022017+0000","last_clean":"2024-02-10T08:40:35.022017+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:40:35.022017+0000","last_undegraded":"2024-02-10T08:40:35.022017+0000","last_fullsized":"2024-02-10T08:40:35.022017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:49.657209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.657209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997649+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:40:34.997649+0000","last_peered":"2024-02-10T08:40:34.997649+0000","last_clean":"2024-02-10T08:40:34.997649+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:40:34.997649+0000","last_undegraded":"2024-02-10T08:40:34.997649+0000","last_fullsized":"2024-02-10T08:40:34.997649+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":321,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995655+0000","last_change":"2024-02-10T08:37:19.382772+0000","last_active":"2024-02-10T08:40:34.995655+0000","last_peered":"2024-02-10T08:40:34.995655+0000","last_clean":"2024-02-10T08:40:34.995655+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:40:34.995655+0000","last_undegraded":"2024-02-10T08:40:34.995655+0000","last_fullsized":"2024-02-10T08:40:34.995655+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:37:19.382703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:19.382703+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":313,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375556+0000","last_change":"2024-02-10T08:39:49.529787+0000","last_active":"2024-02-10T08:40:35.375556+0000","last_peered":"2024-02-10T08:40:35.375556+0000","last_clean":"2024-02-10T08:40:35.375556+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:40:35.375556+0000","last_undegraded":"2024-02-10T08:40:35.375556+0000","last_fullsized":"2024-02-10T08:40:35.375556+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:49.529724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.529724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021760+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:40:35.021760+0000","last_peered":"2024-02-10T08:40:35.021760+0000","last_clean":"2024-02-10T08:40:35.021760+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:40:35.021760+0000","last_undegraded":"2024-02-10T08:40:35.021760+0000","last_fullsized":"2024-02-10T08:40:35.021760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":312,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997556+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:40:34.997556+0000","last_peered":"2024-02-10T08:40:34.997556+0000","last_clean":"2024-02-10T08:40:34.997556+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:40:34.997556+0000","last_undegraded":"2024-02-10T08:40:34.997556+0000","last_fullsized":"2024-02-10T08:40:34.997556+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375487+0000","last_change":"2024-02-10T08:38:59.747834+0000","last_active":"2024-02-10T08:40:35.375487+0000","last_peered":"2024-02-10T08:40:35.375487+0000","last_clean":"2024-02-10T08:40:35.375487+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:40:35.375487+0000","last_undegraded":"2024-02-10T08:40:35.375487+0000","last_fullsized":"2024-02-10T08:40:35.375487+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.747762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.747762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":261,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023135+0000","last_change":"2024-02-10T08:39:41.691153+0000","last_active":"2024-02-10T08:40:35.023135+0000","last_peered":"2024-02-10T08:40:35.023135+0000","last_clean":"2024-02-10T08:40:35.023135+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:40:35.023135+0000","last_undegraded":"2024-02-10T08:40:35.023135+0000","last_fullsized":"2024-02-10T08:40:35.023135+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:39:41.691086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.691086+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":322,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018569+0000","last_change":"2024-02-10T08:40:27.813479+0000","last_active":"2024-02-10T08:40:35.018569+0000","last_peered":"2024-02-10T08:40:35.018569+0000","last_clean":"2024-02-10T08:40:35.018569+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:40:35.018569+0000","last_undegraded":"2024-02-10T08:40:35.018569+0000","last_fullsized":"2024-02-10T08:40:35.018569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.813396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.813396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022823+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:40:35.022823+0000","last_peered":"2024-02-10T08:40:35.022823+0000","last_clean":"2024-02-10T08:40:35.022823+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:40:35.022823+0000","last_undegraded":"2024-02-10T08:40:35.022823+0000","last_fullsized":"2024-02-10T08:40:35.022823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.401534+0000","last_change":"2024-02-10T08:38:14.837278+0000","last_active":"2024-02-10T08:40:35.401534+0000","last_peered":"2024-02-10T08:40:35.401534+0000","last_clean":"2024-02-10T08:40:35.401534+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:40:35.401534+0000","last_undegraded":"2024-02-10T08:40:35.401534+0000","last_fullsized":"2024-02-10T08:40:35.401534+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:14.837154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:14.837154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":330,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375644+0000","last_change":"2024-02-10T08:39:47.473081+0000","last_active":"2024-02-10T08:40:35.375644+0000","last_peered":"2024-02-10T08:40:35.375644+0000","last_clean":"2024-02-10T08:40:35.375644+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:40:35.375644+0000","last_undegraded":"2024-02-10T08:40:35.375644+0000","last_fullsized":"2024-02-10T08:40:35.375644+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:39:47.473007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.473007+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995422+0000","last_change":"2024-02-10T08:38:31.436092+0000","last_active":"2024-02-10T08:40:34.995422+0000","last_peered":"2024-02-10T08:40:34.995422+0000","last_clean":"2024-02-10T08:40:34.995422+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:40:34.995422+0000","last_undegraded":"2024-02-10T08:40:34.995422+0000","last_fullsized":"2024-02-10T08:40:34.995422+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.435994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.435994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995771+0000","last_change":"2024-02-10T08:38:58.917062+0000","last_active":"2024-02-10T08:40:34.995771+0000","last_peered":"2024-02-10T08:40:34.995771+0000","last_clean":"2024-02-10T08:40:34.995771+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:40:34.995771+0000","last_undegraded":"2024-02-10T08:40:34.995771+0000","last_fullsized":"2024-02-10T08:40:34.995771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:58.916996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:58.916996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997350+0000","last_change":"2024-02-10T08:38:28.906569+0000","last_active":"2024-02-10T08:40:34.997350+0000","last_peered":"2024-02-10T08:40:34.997350+0000","last_clean":"2024-02-10T08:40:34.997350+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:40:34.997350+0000","last_undegraded":"2024-02-10T08:40:34.997350+0000","last_fullsized":"2024-02-10T08:40:34.997350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:38:28.906474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":340,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402258+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:40:35.402258+0000","last_peered":"2024-02-10T08:40:35.402258+0000","last_clean":"2024-02-10T08:40:35.402258+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:40:35.402258+0000","last_undegraded":"2024-02-10T08:40:35.402258+0000","last_fullsized":"2024-02-10T08:40:35.402258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.374452+0000","last_change":"2024-02-10T08:38:32.342305+0000","last_active":"2024-02-10T08:40:35.374452+0000","last_peered":"2024-02-10T08:40:35.374452+0000","last_clean":"2024-02-10T08:40:35.374452+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:40:35.374452+0000","last_undegraded":"2024-02-10T08:40:35.374452+0000","last_fullsized":"2024-02-10T08:40:35.374452+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.342200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.342200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.620467+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:40:35.620467+0000","last_peered":"2024-02-10T08:40:35.620467+0000","last_clean":"2024-02-10T08:40:35.620467+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:40:35.620467+0000","last_undegraded":"2024-02-10T08:40:35.620467+0000","last_fullsized":"2024-02-10T08:40:35.620467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.373834+0000","last_change":"2024-02-10T08:38:18.541756+0000","last_active":"2024-02-10T08:40:35.373834+0000","last_peered":"2024-02-10T08:40:35.373834+0000","last_clean":"2024-02-10T08:40:35.373834+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:40:35.373834+0000","last_undegraded":"2024-02-10T08:40:35.373834+0000","last_fullsized":"2024-02-10T08:40:35.373834+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.541693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.541693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":330,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997609+0000","last_change":"2024-02-10T08:38:19.453131+0000","last_active":"2024-02-10T08:40:34.997609+0000","last_peered":"2024-02-10T08:40:34.997609+0000","last_clean":"2024-02-10T08:40:34.997609+0000","last_became_active":"2024-02-10T08:33:20.113706+0000","last_became_peered":"2024-02-10T08:33:20.113706+0000","last_unstale":"2024-02-10T08:40:34.997609+0000","last_undegraded":"2024-02-10T08:40:34.997609+0000","last_fullsized":"2024-02-10T08:40:34.997609+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:38:19.453060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.453060+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":309,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997420+0000","last_change":"2024-02-10T08:38:16.426049+0000","last_active":"2024-02-10T08:40:34.997420+0000","last_peered":"2024-02-10T08:40:34.997420+0000","last_clean":"2024-02-10T08:40:34.997420+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:40:34.997420+0000","last_undegraded":"2024-02-10T08:40:34.997420+0000","last_fullsized":"2024-02-10T08:40:34.997420+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:16.425945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:16.425945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998793+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:40:34.998793+0000","last_peered":"2024-02-10T08:40:34.998793+0000","last_clean":"2024-02-10T08:40:34.998793+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:40:34.998793+0000","last_undegraded":"2024-02-10T08:40:34.998793+0000","last_fullsized":"2024-02-10T08:40:34.998793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":253,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.996553+0000","last_change":"2024-02-10T08:39:46.928020+0000","last_active":"2024-02-10T08:40:34.996553+0000","last_peered":"2024-02-10T08:40:34.996553+0000","last_clean":"2024-02-10T08:40:34.996553+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:40:34.996553+0000","last_undegraded":"2024-02-10T08:40:34.996553+0000","last_fullsized":"2024-02-10T08:40:34.996553+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.927953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":343,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995892+0000","last_change":"2024-02-10T08:38:26.871174+0000","last_active":"2024-02-10T08:40:34.995892+0000","last_peered":"2024-02-10T08:40:34.995892+0000","last_clean":"2024-02-10T08:40:34.995892+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:40:34.995892+0000","last_undegraded":"2024-02-10T08:40:34.995892+0000","last_fullsized":"2024-02-10T08:40:34.995892+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:38:26.871100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.871100+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":309,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022457+0000","last_change":"2024-02-10T08:39:02.250833+0000","last_active":"2024-02-10T08:40:35.022457+0000","last_peered":"2024-02-10T08:40:35.022457+0000","last_clean":"2024-02-10T08:40:35.022457+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:40:35.022457+0000","last_undegraded":"2024-02-10T08:40:35.022457+0000","last_fullsized":"2024-02-10T08:40:35.022457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:02.250719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:02.250719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":223,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.994466+0000","last_change":"2024-02-10T08:38:27.952601+0000","last_active":"2024-02-10T08:40:34.994466+0000","last_peered":"2024-02-10T08:40:34.994466+0000","last_clean":"2024-02-10T08:40:34.994466+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:40:34.994466+0000","last_undegraded":"2024-02-10T08:40:34.994466+0000","last_fullsized":"2024-02-10T08:40:34.994466+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:27.952518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:27.952518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":262,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.402611+0000","last_change":"2024-02-10T08:40:22.359771+0000","last_active":"2024-02-10T08:40:35.402611+0000","last_peered":"2024-02-10T08:40:35.402611+0000","last_clean":"2024-02-10T08:40:35.402611+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:40:35.402611+0000","last_undegraded":"2024-02-10T08:40:35.402611+0000","last_fullsized":"2024-02-10T08:40:35.402611+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.359703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.359703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":320,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995977+0000","last_change":"2024-02-10T08:39:11.790943+0000","last_active":"2024-02-10T08:40:34.995977+0000","last_peered":"2024-02-10T08:40:34.995977+0000","last_clean":"2024-02-10T08:40:34.995977+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:40:34.995977+0000","last_undegraded":"2024-02-10T08:40:34.995977+0000","last_fullsized":"2024-02-10T08:40:34.995977+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.790866+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.022914+0000","last_change":"2024-02-10T08:38:30.389134+0000","last_active":"2024-02-10T08:40:35.022914+0000","last_peered":"2024-02-10T08:40:35.022914+0000","last_clean":"2024-02-10T08:40:35.022914+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:40:35.022914+0000","last_undegraded":"2024-02-10T08:40:35.022914+0000","last_fullsized":"2024-02-10T08:40:35.022914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.389076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":302,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.018884+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:40:35.018884+0000","last_peered":"2024-02-10T08:40:35.018884+0000","last_clean":"2024-02-10T08:40:35.018884+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:40:35.018884+0000","last_undegraded":"2024-02-10T08:40:35.018884+0000","last_fullsized":"2024-02-10T08:40:35.018884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.021394+0000","last_change":"2024-02-10T08:38:15.539954+0000","last_active":"2024-02-10T08:40:35.021394+0000","last_peered":"2024-02-10T08:40:35.021394+0000","last_clean":"2024-02-10T08:40:35.021394+0000","last_became_active":"2024-02-10T08:37:10.149012+0000","last_became_peered":"2024-02-10T08:37:10.149012+0000","last_unstale":"2024-02-10T08:40:35.021394+0000","last_undegraded":"2024-02-10T08:40:35.021394+0000","last_fullsized":"2024-02-10T08:40:35.021394+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:15.539850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:15.539850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":338,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995108+0000","last_change":"2024-02-10T08:37:13.579260+0000","last_active":"2024-02-10T08:40:34.995108+0000","last_peered":"2024-02-10T08:40:34.995108+0000","last_clean":"2024-02-10T08:40:34.995108+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:40:34.995108+0000","last_undegraded":"2024-02-10T08:40:34.995108+0000","last_fullsized":"2024-02-10T08:40:34.995108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:13.579189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:13.579189+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":318,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.998143+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:40:34.998143+0000","last_peered":"2024-02-10T08:40:34.998143+0000","last_clean":"2024-02-10T08:40:34.998143+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:40:34.998143+0000","last_undegraded":"2024-02-10T08:40:34.998143+0000","last_fullsized":"2024-02-10T08:40:34.998143+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995678+0000","last_change":"2024-02-10T08:39:05.790083+0000","last_active":"2024-02-10T08:40:34.995678+0000","last_peered":"2024-02-10T08:40:34.995678+0000","last_clean":"2024-02-10T08:40:34.995678+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:40:34.995678+0000","last_undegraded":"2024-02-10T08:40:34.995678+0000","last_fullsized":"2024-02-10T08:40:34.995678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.790005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.790005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":260,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.019101+0000","last_change":"2024-02-10T08:38:30.695748+0000","last_active":"2024-02-10T08:40:35.019101+0000","last_peered":"2024-02-10T08:40:35.019101+0000","last_clean":"2024-02-10T08:40:35.019101+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:40:35.019101+0000","last_undegraded":"2024-02-10T08:40:35.019101+0000","last_fullsized":"2024-02-10T08:40:35.019101+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.695684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.695684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":352,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.995726+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:40:34.995726+0000","last_peered":"2024-02-10T08:40:34.995726+0000","last_clean":"2024-02-10T08:40:34.995726+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:40:34.995726+0000","last_undegraded":"2024-02-10T08:40:34.995726+0000","last_fullsized":"2024-02-10T08:40:34.995726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":222,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.023698+0000","last_change":"2024-02-10T08:37:50.656200+0000","last_active":"2024-02-10T08:40:35.023698+0000","last_peered":"2024-02-10T08:40:35.023698+0000","last_clean":"2024-02-10T08:40:35.023698+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:40:35.023698+0000","last_undegraded":"2024-02-10T08:40:35.023698+0000","last_fullsized":"2024-02-10T08:40:35.023698+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:50.656135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.656135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":303,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:34.997316+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:40:34.997316+0000","last_peered":"2024-02-10T08:40:34.997316+0000","last_clean":"2024-02-10T08:40:34.997316+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:40:34.997316+0000","last_undegraded":"2024-02-10T08:40:34.997316+0000","last_fullsized":"2024-02-10T08:40:34.997316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":251,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-10T08:40:35.375253+0000","last_change":"2024-02-10T08:38:34.376839+0000","last_active":"2024-02-10T08:40:35.375253+0000","last_peered":"2024-02-10T08:40:35.375253+0000","last_clean":"2024-02-10T08:40:35.375253+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:40:35.375253+0000","last_undegraded":"2024-02-10T08:40:35.375253+0000","last_fullsized":"2024-02-10T08:40:35.375253+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:34.376762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:34.376762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":197,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001380,"num_objects":143,"num_object_clones":2,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":568,"num_write_kb":4469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13742080,"data_stored":12004148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1825,"ondisk_log_size":1825,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299482,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306908,"kb_used_data":5580,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438244,"statfs":{"total":95995035648,"available":95680761856,"internally_reserved":0,"allocated":5713920,"data_stored":3140691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.76000000000000001,"15min":0.60199999999999998},"min":{"1min":0.36799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.66500000000000004,"15min":0.56799999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66200000000000003,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.61399999999999999,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68799999999999994,"5min":7.593,"15min":7.593},"last":0.63},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.64000000000000001,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.58499999999999996,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.63500000000000001,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.84899999999999998,"15min":0.64900000000000002},"min":{"1min":0.308,"5min":0.251,"15min":0.251},"max":{"1min":0.75,"5min":19.369,"15min":19.369},"last":0.47599999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.68500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.83999999999999997,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33100000000000002,"15min":0.27900000000000003},"max":{"1min":0.72599999999999998,"5min":22.102,"15min":22.102},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.48299999999999998,"15min":0.44700000000000001},"min":{"1min":0.247,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.495,"5min":8.827,"15min":8.827},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.55600000000000005,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67700000000000005,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.38}]},{"osd":5,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.65700000000000003,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64000000000000001,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65200000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.45900000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.68799999999999994,"15min":0.55900000000000005},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60399999999999998,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.35099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71499999999999997,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.44800000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561120,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322988,"kb_used_data":9376,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93422164,"statfs":{"total":95995035648,"available":95664295936,"internally_reserved":0,"allocated":9601024,"data_stored":7130969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.628,"15min":0.51600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.19800000000000001},"max":{"1min":0.625,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.65000000000000002,"15min":0.53100000000000003},"min":{"1min":0.28799999999999998,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.58799999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.56999999999999995}]},{"osd":1,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.58199999999999996,"15min":0.52600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72499999999999998,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.627},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.56899999999999995,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.66000000000000003,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.68000000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.57599999999999996,"15min":0.52700000000000002},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64900000000000002,"5min":4.798,"15min":4.798},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58299999999999996,"15min":0.52800000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.69599999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.82099999999999995,"15min":0.59299999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":16.823,"15min":16.823},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.57299999999999995,"15min":0.51300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.629,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.70299999999999996}]},{"osd":4,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.39700000000000002},"min":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56899999999999995,"5min":3.657,"15min":3.657},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.38500000000000001},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60099999999999998,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.42099999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47499999999999998,"15min":0.40200000000000002},"min":{"1min":0.317,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.62,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47299999999999998,"15min":0.42399999999999999},"min":{"1min":0.34100000000000003,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.47699999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.501,"15min":0.47199999999999998},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.63500000000000001,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.49199999999999999},"min":{"1min":0.40200000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.51400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302792,"kb_used_data":5564,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442360,"statfs":{"total":95995035648,"available":95684976640,"internally_reserved":0,"allocated":5697536,"data_stored":3142144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.56999999999999995,"15min":0.41099999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":0.58099999999999996,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.315},{"interface":"front","average":{"1min":0.37,"5min":0.64300000000000002,"15min":0.45900000000000002},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.19500000000000001},"max":{"1min":0.56599999999999995,"5min":16.166,"15min":16.166},"last":0.377}]},{"osd":2,"last update":"Sat Feb 10 08:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":1.101,"15min":0.61499999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54200000000000004,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.442},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.86399999999999999,"15min":0.53100000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42999999999999999,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.74399999999999999,"15min":0.53200000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.627,"5min":24.773,"15min":24.773},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71199999999999997,"15min":0.53700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.60399999999999998,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.75700000000000001,"15min":0.61899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.67600000000000005,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.67600000000000005,"15min":0.60799999999999998},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67300000000000004,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.33200000000000002,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.77500000000000002,"15min":0.66700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.69099999999999995,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.68300000000000005,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.76900000000000002,"5min":9.532,"15min":9.532},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.79500000000000004,"15min":0.65700000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.72399999999999998,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.83799999999999997,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.754,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.61799999999999999}]}]},{"osd":0,"up_from":0,"seq":34359738493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315040,"kb_used_data":5524,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430112,"statfs":{"total":95995035648,"available":95672434688,"internally_reserved":0,"allocated":5656576,"data_stored":3130736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.41099999999999998},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.187},"max":{"1min":3.0640000000000001,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.379},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.67300000000000004,"15min":0.46400000000000002},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":1.4950000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.70999999999999996,"15min":0.47699999999999998},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.63400000000000001,"5min":12.512,"15min":12.512},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64900000000000002,"15min":0.47099999999999997},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":3.0960000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.441}]},{"osd":3,"last update":"Sat Feb 10 08:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.6520000000000001,"5min":4.617,"15min":4.617},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.59399999999999997,"15min":0.48599999999999999},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.41399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.878,"15min":0.68000000000000005},"min":{"1min":0.32800000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.7370000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.77700000000000002,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":0.79000000000000004,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.86099999999999999,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.764,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.72699999999999998,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.78900000000000003,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.81299999999999994,"15min":0.66300000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.6759999999999999,"5min":9.266,"15min":9.266},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.73099999999999998,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.81999999999999995,"15min":0.69399999999999995},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.708,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.77200000000000002,"15min":0.66500000000000004},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.56399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318972,"kb_used_data":9456,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426180,"statfs":{"total":95995035648,"available":95668408320,"internally_reserved":0,"allocated":9682944,"data_stored":7140580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.81699999999999995,"15min":0.51600000000000001},"min":{"1min":0.253,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.42299999999999999,"5min":14.744,"15min":14.744},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.872,"15min":0.54500000000000004},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.498,"5min":11.808,"15min":11.808},"last":0.31900000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.82599999999999996,"15min":0.55100000000000005},"min":{"1min":0.23200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.262,"5min":16.634,"15min":16.634},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.82099999999999995,"15min":0.56100000000000005},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":0.55100000000000005,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.86399999999999999,"15min":0.59099999999999997},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":1.079,"15min":0.63500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.51500000000000001,"5min":23.294,"15min":23.294},"last":0.312}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.84799999999999998,"15min":0.68400000000000005},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":0.70799999999999996,"5min":12.371,"15min":12.371},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.0089999999999999,"15min":0.71699999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":14.821999999999999,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.6040000000000001,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.80500000000000005,"15min":0.65400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.9909999999999997,"5min":11.448,"15min":11.448},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:40:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.97399999999999998,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.91800000000000004,"15min":0.67300000000000004},"min":{"1min":0.378,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72899999999999998,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.876,"15min":0.65300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.079,"15min":0.72599999999999998},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.52,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.75800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986281,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306876,"kb_used_data":5548,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438276,"statfs":{"total":95995035648,"available":95680794624,"internally_reserved":0,"allocated":5681152,"data_stored":3140690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.84799999999999998,"15min":0.64100000000000001},"min":{"1min":0.35899999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84999999999999998,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.65800000000000003,"15min":0.56100000000000005},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.91800000000000004,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.73099999999999998,"15min":0.59199999999999997},"min":{"1min":0.35599999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93799999999999994,"5min":5.774,"15min":5.774},"last":0.438},{"interface":"front","average":{"1min":0.505,"5min":0.73799999999999999,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.76100000000000001,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.251,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.498},{"interface":"front","average":{"1min":0.497,"5min":0.77600000000000002,"15min":0.61099999999999999},"min":{"1min":0.34599999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":10.01,"15min":10.01},"last":0.45000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.91800000000000004,"15min":0.66100000000000003},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.92700000000000005,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.0980000000000001,"15min":0.71999999999999997},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.82099999999999995,"5min":25.27,"15min":25.27},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.51000000000000001,"15min":0.41899999999999998},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67300000000000004,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47699999999999998,"15min":0.41299999999999998},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71499999999999997,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:39:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.65600000000000003,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.70499999999999996,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.63800000000000001,"15min":0.504},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":13.753,"15min":13.753},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.623,"15min":0.51100000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.68100000000000005,"15min":0.57899999999999996},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78300000000000003,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.55100000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790053,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310700,"kb_used_data":9376,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434452,"statfs":{"total":95995035648,"available":95676878848,"internally_reserved":0,"allocated":9601024,"data_stored":7130528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.968,"15min":0.996},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67900000000000005,"5min":107.551,"15min":107.551},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.498,"5min":1.843,"15min":0.96199999999999997},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":0.76900000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":1.6930000000000001,"15min":0.93799999999999994},"min":{"1min":0.39300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67100000000000004,"5min":107.205,"15min":107.205},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.359,"15min":0.82799999999999996},"min":{"1min":0.38500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.63100000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":1.839,"15min":0.95299999999999996},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.623,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.9219999999999999,"15min":0.995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1379999999999999,"15min":0.71899999999999997},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":2.5169999999999999,"15min":1.206},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77400000000000002,"5min":114.526,"15min":114.526},"last":0.77400000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":1.151,"15min":0.65700000000000003},"min":{"1min":0.30499999999999999,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.88700000000000001,"15min":0.56299999999999994},"min":{"1min":0.317,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.58999999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.1910000000000001,"15min":0.67300000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64800000000000002,"5min":72.241,"15min":72.241},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.434,"5min":0.70299999999999996,"15min":0.53900000000000003},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.56899999999999995,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.1699999999999999,"15min":0.70999999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.2370000000000001,"15min":0.74099999999999999},"min":{"1min":0.39200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72899999999999998,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":4000599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4000467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:40:39.061 INFO:tasks.thrashosds.thrasher:Setting 4.1 to [4, 0, 7] 2024-02-10T08:40:39.061 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.1', '4', '0', '7'] 2024-02-10T08:40:39.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 4.1 4 0 7 2024-02-10T08:40:39.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:39.165 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:39.354 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:39.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:39.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:38.184641+0000 mgr.y (mgr.14553) 320 : audit [DBG] from='client.53441 v1:172.21.15.107:0/1477359855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:38.664452+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.f scrub starts 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:38.667919+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub ok 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:39.007204+0000 mon.a (mon.0) 2207 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:39.007298+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.007375+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-10T08:40:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:39.008401+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:39.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:39.010284+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:39.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:39.010553+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:38.184641+0000 mgr.y (mgr.14553) 320 : audit [DBG] from='client.53441 v1:172.21.15.107:0/1477359855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:38.664452+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.f scrub starts 2024-02-10T08:40:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:38.667919+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub ok 2024-02-10T08:40:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:39.007204+0000 mon.a (mon.0) 2207 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:40:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:39.007298+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.007375+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:39.008401+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:39.010284+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:39.010553+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:38.184641+0000 mgr.y (mgr.14553) 320 : audit [DBG] from='client.53441 v1:172.21.15.107:0/1477359855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:38.664452+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.f scrub starts 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:38.667919+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub ok 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:39.007204+0000 mon.a (mon.0) 2207 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:39.007298+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi107-112411-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.007375+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-10T08:40:39.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:39.008401+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:39.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:39.010284+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:39.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:39.010553+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:40:40.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:40.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:40.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:40.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:40:40.467 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:40.506 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:40.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.347067+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:40:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.350658+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.418919+0000 mgr.y (mgr.14553) 321 : cluster [DBG] pgmap v494: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.436261+0000 osd.6 (osd.6) 135 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.439235+0000 osd.6 (osd.6) 136 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.890758+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:39.891555+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:40:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.245747+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:40:40.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.245839+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-10T08:40:40.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.246707+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.247146+0000 mon.a (mon.0) 2214 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.250082+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:40.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.347067+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.350658+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.418919+0000 mgr.y (mgr.14553) 321 : cluster [DBG] pgmap v494: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.436261+0000 osd.6 (osd.6) 135 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.439235+0000 osd.6 (osd.6) 136 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.890758+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:39.891555+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.245747+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:40:40.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.245839+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.246707+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.247146+0000 mon.a (mon.0) 2214 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.250082+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.347067+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.350658+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.418919+0000 mgr.y (mgr.14553) 321 : cluster [DBG] pgmap v494: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.436261+0000 osd.6 (osd.6) 135 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.439235+0000 osd.6 (osd.6) 136 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.890758+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:39.891555+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.245747+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.245839+0000 mon.a (mon.0) 2213 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-10T08:40:40.641 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.246707+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.642 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.247146+0000 mon.a (mon.0) 2214 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:40:40.642 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.250082+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:40.642 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:40.969 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:40.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:41.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:41.105 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:41.123 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:41.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:41.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:41.273 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:41.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:41.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.094604+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.098511+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.310872+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.c deep-scrub starts 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.313932+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.c deep-scrub ok 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.704525+0000 mon.c (mon.2) 363 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.704809+0000 mon.c (mon.2) 364 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.704924+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705041+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705273+0000 mon.c (mon.2) 366 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705330+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705508+0000 mon.c (mon.2) 367 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705657+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705753+0000 mon.c (mon.2) 368 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.705965+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.706196+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.706368+0000 mon.a (mon.0) 2221 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.892317+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.c deep-scrub starts 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.893253+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.c deep-scrub ok 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990366+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990441+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990509+0000 mon.a (mon.0) 2224 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990578+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990644+0000 mon.a (mon.0) 2226 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990723+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]': finished 2024-02-10T08:40:41.382 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990786+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]': finished 2024-02-10T08:40:41.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:40.990852+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]': finished 2024-02-10T08:40:41.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.990911+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-10T08:40:41.403 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:41.421 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:41.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:41.523 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:41.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:41.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.094604+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:40:41.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.098511+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:40:41.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.310872+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.c deep-scrub starts 2024-02-10T08:40:41.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.313932+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.c deep-scrub ok 2024-02-10T08:40:41.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.704525+0000 mon.c (mon.2) 363 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.704809+0000 mon.c (mon.2) 364 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.704924+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705041+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705273+0000 mon.c (mon.2) 366 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705330+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705508+0000 mon.c (mon.2) 367 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705657+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705753+0000 mon.c (mon.2) 368 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.705965+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.706196+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.094604+0000 osd.5 (osd.5) 157 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.098511+0000 osd.5 (osd.5) 158 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.310872+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.c deep-scrub starts 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.313932+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.c deep-scrub ok 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.704525+0000 mon.c (mon.2) 363 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.704809+0000 mon.c (mon.2) 364 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.704924+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705041+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705273+0000 mon.c (mon.2) 366 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705330+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705508+0000 mon.c (mon.2) 367 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705657+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]: dispatch 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705753+0000 mon.c (mon.2) 368 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.705965+0000 mon.a (mon.0) 2219 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.706196+0000 mon.a (mon.0) 2220 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]: dispatch 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.706368+0000 mon.a (mon.0) 2221 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.892317+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.c deep-scrub starts 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.893253+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.c deep-scrub ok 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990366+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990441+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990509+0000 mon.a (mon.0) 2224 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-02-10T08:40:41.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990578+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990644+0000 mon.a (mon.0) 2226 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]': finished 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990723+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]': finished 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990786+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]': finished 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:40.990852+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]': finished 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.990911+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.706368+0000 mon.a (mon.0) 2221 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]: dispatch 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.892317+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.c deep-scrub starts 2024-02-10T08:40:41.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.893253+0000 osd.3 (osd.3) 172 : cluster [DBG] 6.c deep-scrub ok 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990366+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi107-112411-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990441+0000 mon.a (mon.0) 2223 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990509+0000 mon.a (mon.0) 2224 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990578+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990644+0000 mon.a (mon.0) 2226 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 3]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990723+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.14", "id": [1, 7]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990786+0000 mon.a (mon.0) 2228 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1b", "id": [1, 7]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:40.990852+0000 mon.a (mon.0) 2229 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "197.c", "id": [4, 7]}]': finished 2024-02-10T08:40:41.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.990911+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-10T08:40:41.556 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:41.652 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:41.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:41.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:41.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:41.799 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:42.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:42.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:42.176 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.498888+0000 osd.1 (osd.1) 155 : cluster [DBG] 5.c scrub starts 2024-02-10T08:40:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:40.501864+0000 osd.1 (osd.1) 156 : cluster [DBG] 5.c scrub ok 2024-02-10T08:40:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:41.273438+0000 osd.2 (osd.2) 125 : cluster [DBG] 5.d scrub starts 2024-02-10T08:40:42.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:41.276896+0000 osd.2 (osd.2) 126 : cluster [DBG] 5.d scrub ok 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:41.380839+0000 mgr.y (mgr.14553) 322 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:41.419991+0000 mgr.y (mgr.14553) 323 : cluster [DBG] pgmap v497: 226 pgs: 6 peering, 8 creating+peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:41.535173+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:41.538135+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.004594+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:42.043367+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:42.043740+0000 mon.a (mon.0) 2232 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.498888+0000 osd.1 (osd.1) 155 : cluster [DBG] 5.c scrub starts 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:40.501864+0000 osd.1 (osd.1) 156 : cluster [DBG] 5.c scrub ok 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:41.273438+0000 osd.2 (osd.2) 125 : cluster [DBG] 5.d scrub starts 2024-02-10T08:40:42.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:41.276896+0000 osd.2 (osd.2) 126 : cluster [DBG] 5.d scrub ok 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:41.380839+0000 mgr.y (mgr.14553) 322 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:41.419991+0000 mgr.y (mgr.14553) 323 : cluster [DBG] pgmap v497: 226 pgs: 6 peering, 8 creating+peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:41.535173+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:41.538135+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.004594+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:42.043367+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.327 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:42.043740+0000 mon.a (mon.0) 2232 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:42.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:42.335 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:40:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:40:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:40:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:40:42.335 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:40:42.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:42.461 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:42.483 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:42.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:42.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.498888+0000 osd.1 (osd.1) 155 : cluster [DBG] 5.c scrub starts 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:40.501864+0000 osd.1 (osd.1) 156 : cluster [DBG] 5.c scrub ok 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:41.273438+0000 osd.2 (osd.2) 125 : cluster [DBG] 5.d scrub starts 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:41.276896+0000 osd.2 (osd.2) 126 : cluster [DBG] 5.d scrub ok 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:41.380839+0000 mgr.y (mgr.14553) 322 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:41.419991+0000 mgr.y (mgr.14553) 323 : cluster [DBG] pgmap v497: 226 pgs: 6 peering, 8 creating+peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:41.535173+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:41.538135+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.004594+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-10T08:40:42.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:42.043367+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.616 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:42.043740+0000 mon.a (mon.0) 2232 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:42.619 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:42.637 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:42.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:42.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:43.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:40:43.020 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:43.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:43.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:43.155 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:43.175 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:43.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:43.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:43.307 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.229937+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.e scrub starts 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.232902+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.e scrub ok 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.334980+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.335954+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.008333+0000 mon.a (mon.0) 2233 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:43.008397+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.010832+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.015712+0000 mon.a (mon.0) 2235 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.016769+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.229937+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.e scrub starts 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.232902+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.e scrub ok 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.334980+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.335954+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.008333+0000 mon.a (mon.0) 2233 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:43.008397+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-10T08:40:43.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.010832+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.015712+0000 mon.a (mon.0) 2235 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.016769+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.229937+0000 osd.2 (osd.2) 127 : cluster [DBG] 2.e scrub starts 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.232902+0000 osd.2 (osd.2) 128 : cluster [DBG] 2.e scrub ok 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.334980+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.335954+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.008333+0000 mon.a (mon.0) 2233 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:43.008397+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.010832+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.015712+0000 mon.a (mon.0) 2235 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:43.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.016769+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 -- 172.21.15.107:0/3675021570 >> v1:172.21.15.107:6790/0 conn(0x7ff5f80fda50 legacy=0x7ff5f80fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 -- 172.21.15.107:0/3675021570 shutdown_connections 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 -- 172.21.15.107:0/3675021570 >> 172.21.15.107:0/3675021570 conn(0x7ff5f80f3640 msgr2=0x7ff5f80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 -- 172.21.15.107:0/3675021570 shutdown_connections 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 -- 172.21.15.107:0/3675021570 wait complete. 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 librados: starting msgr at 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 librados: starting objecter 2024-02-10T08:40:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.718+0000 7ff5fda24700 1 Processor -- start 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 -- start start 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 librados: setting wanted keys 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 librados: calling monclient init 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5f806cdb0 con 0x7ff5f81007f0 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5f806cf40 con 0x7ff5f80fda50 2024-02-10T08:40:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5fda24700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5f811fcc0 con 0x7ff5f8104330 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5f77fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff5f8104330 0x7ff5f811c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37648/0 (socket says 172.21.15.107:37648) 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5f77fe700 1 -- 172.21.15.107:0/872486079 learned_addr learned my addr 172.21.15.107:0/872486079 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter ms_handle_connect 0x7ff5f8104330 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter ms_handle_connect 0x7ff5f81007f0 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter ms_handle_connect 0x7ff5f80fda50 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304398891 0 0) 0x7ff5f811fcc0 con 0x7ff5f8104330 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e00038e0 con 0x7ff5f8104330 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565060949 0 0) 0x7ff5f806cdb0 con 0x7ff5f81007f0 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f811fcc0 con 0x7ff5f81007f0 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637640086 0 0) 0x7ff5f806cf40 con 0x7ff5f80fda50 2024-02-10T08:40:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.719+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f806cdb0 con 0x7ff5f80fda50 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1447219637 0 0) 0x7ff5e00038e0 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f806cf40 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff5f0001cd0 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228408514 0 0) 0x7ff5f806cdb0 con 0x7ff5f80fda50 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e00038e0 con 0x7ff5f80fda50 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff5e8000fe0 con 0x7ff5f80fda50 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 529412522 0 0) 0x7ff5f806cf40 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 >> v1:172.21.15.195:6789/0 conn(0x7ff5f80fda50 legacy=0x7ff5f80fc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 >> v1:172.21.15.107:6789/0 conn(0x7ff5f81007f0 legacy=0x7ff5f80fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8120e60 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f811feb0 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5fda24700 10 client.53453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f81203a0 con 0x7ff5f8104330 2024-02-10T08:40:43.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5fda24700 1 librados: init done 2024-02-10T08:40:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff5f0002720 con 0x7ff5f8104330 2024-02-10T08:40:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.720+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff5f0005df0 con 0x7ff5f8104330 2024-02-10T08:40:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.721+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4004f20 con 0x7ff5f8104330 2024-02-10T08:40:43.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.721+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff5f0018ca0 con 0x7ff5f8104330 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(349..349 src has 50..349) v4 ==== 8572+0+0 (unknown 2985874899 0 0) 0x7ff5f0079240 con 0x7ff5f8104330 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 10 client.53453.objecter ms_dispatch 0x7ff5f8001550 osd_map(349..349 src has 50..349) v4 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 3 client.53453.objecter handle_osd_map got epochs [349,349] > 0 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 3 client.53453.objecter handle_osd_map decoding full epoch 349 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 20 client.53453.objecter dump_active .. 0 homeless 2024-02-10T08:40:43.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.722+0000 7ff5dffff700 10 client.53453.objecter ms_handle_connect 0x7ff5e0063bc0 2024-02-10T08:40:43.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.726+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff5f004c580 con 0x7ff5f8104330 2024-02-10T08:40:43.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:43.906+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff5e40051d0 con 0x7ff5f8104330 2024-02-10T08:40:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.010+0000 7ff5dffff700 1 -- 172.21.15.107:0/872486079 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v350) v1 ==== 87+0+0 (unknown 2014236479 0 0) 0x7ff5f004a990 con 0x7ff5f8104330 2024-02-10T08:40:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 10 librados: watch_flush enter 2024-02-10T08:40:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 10 librados: watch_flush exit 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 20 client.53453.objecter shutdown clearing up homeless session... 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 10 client.53453.objecter successfully canceled tick 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff5e0063bc0 legacy=0x7ff5e0066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 >> v1:172.21.15.107:6790/0 conn(0x7ff5f8104330 legacy=0x7ff5f811c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 shutdown_connections 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 >> 172.21.15.107:0/872486079 conn(0x7ff5f80f3640 msgr2=0x7ff5f81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 shutdown_connections 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 -- 172.21.15.107:0/872486079 wait complete. 2024-02-10T08:40:44.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:44.013+0000 7ff5fda24700 1 librados: shutdown 2024-02-10T08:40:44.015 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:40:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:44.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:44.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:44.290 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:44.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:44.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.482383+0000 osd.1 (osd.1) 157 : cluster [DBG] 4.f scrub starts 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:42.486017+0000 osd.1 (osd.1) 158 : cluster [DBG] 4.f scrub ok 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:43.420365+0000 mgr.y (mgr.14553) 324 : cluster [DBG] pgmap v500: 218 pgs: 6 peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:43.491230+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:43.494167+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:40:44.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.907824+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.107:0/872486079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:43.908087+0000 mon.a (mon.0) 2237 : audit [INF] from='client.53453 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:44.003489+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.009242+0000 mon.a (mon.0) 2239 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.009322+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.009366+0000 mon.a (mon.0) 2241 : audit [INF] from='client.53453 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:44.009417+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.011996+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.034391+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.034728+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.035213+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:44.035404+0000 mon.a (mon.0) 2245 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.482383+0000 osd.1 (osd.1) 157 : cluster [DBG] 4.f scrub starts 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:42.486017+0000 osd.1 (osd.1) 158 : cluster [DBG] 4.f scrub ok 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:43.420365+0000 mgr.y (mgr.14553) 324 : cluster [DBG] pgmap v500: 218 pgs: 6 peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:43.491230+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:43.494167+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.907824+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.107:0/872486079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:43.908087+0000 mon.a (mon.0) 2237 : audit [INF] from='client.53453 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:44.003489+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.009242+0000 mon.a (mon.0) 2239 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:44.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.009322+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.009366+0000 mon.a (mon.0) 2241 : audit [INF] from='client.53453 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:44.009417+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.011996+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.034391+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.034728+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.035213+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:44.035404+0000 mon.a (mon.0) 2245 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.444 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:44.464 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:44.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:44.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.482383+0000 osd.1 (osd.1) 157 : cluster [DBG] 4.f scrub starts 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:42.486017+0000 osd.1 (osd.1) 158 : cluster [DBG] 4.f scrub ok 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:43.420365+0000 mgr.y (mgr.14553) 324 : cluster [DBG] pgmap v500: 218 pgs: 6 peering, 212 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:43.491230+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:43.494167+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.907824+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.107:0/872486079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:43.908087+0000 mon.a (mon.0) 2237 : audit [INF] from='client.53453 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:44.003489+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.009242+0000 mon.a (mon.0) 2239 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]': finished 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.009322+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.009366+0000 mon.a (mon.0) 2241 : audit [INF] from='client.53453 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:44.009417+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.011996+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]: dispatch 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.034391+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.034728+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.035213+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:44.035404+0000 mon.a (mon.0) 2245 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-45"}]: dispatch 2024-02-10T08:40:44.940 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp.RoundTripAppendPP (3243 ms) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2773 ms) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3011 ms) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3003 ms) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2974 ms) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio (117806 ms total) 2024-02-10T08:40:45.019 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3006 ms) 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9011 ms) 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15007 ms) 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3241 ms) 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30265 ms total) 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: 2024-02-10T08:40:45.020 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2749 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2749 ms total) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14022 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6990 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7090 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6926 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3015 ms) 2024-02-10T08:40:45.021 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6981 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7006 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7016 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7000 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6976 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7001 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7095 ms) 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-10T08:40:45.022 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6912 ms) 2024-02-10T08:40:45.023 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-02-10T08:40:45.023 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3008 ms) 2024-02-10T08:40:45.023 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-10T08:40:45.023 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7012 ms) 2024-02-10T08:40:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:45.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:45.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:45.578 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:45.701 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:45] "GET /metrics HTTP/1.1" 200 186718 "" "Prometheus/2.33.4" 2024-02-10T08:40:45.702 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:45.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:45.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:45.839 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:45.859 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:45.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:45.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:45.992 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:46.098 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:46.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:46.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.009731+0000 mon.a (mon.0) 2246 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:45.009806+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.034126+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.263955+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.265479+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.265761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.266474+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:45.266788+0000 mon.a (mon.0) 2250 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:45.420851+0000 mgr.y (mgr.14553) 325 : cluster [DBG] pgmap v503: 186 pgs: 5 peering, 181 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.009731+0000 mon.a (mon.0) 2246 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:45.009806+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-10T08:40:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.034126+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.263955+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.265479+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.265761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.266474+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:45.266788+0000 mon.a (mon.0) 2250 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:45.420851+0000 mgr.y (mgr.14553) 325 : cluster [DBG] pgmap v503: 186 pgs: 5 peering, 181 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.009731+0000 mon.a (mon.0) 2246 : audit [INF] from='client.? v1:172.21.15.107:0/10211293' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi107-112411-60"}]': finished 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:45.009806+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.034126+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.263955+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.265479+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.265761+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.266474+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:45.266788+0000 mon.a (mon.0) 2250 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:46.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:45.420851+0000 mgr.y (mgr.14553) 325 : cluster [DBG] pgmap v503: 186 pgs: 5 peering, 181 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:46.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.130+0000 7f6d75c48700 1 -- 172.21.15.107:0/4217608436 >> v1:172.21.15.107:6790/0 conn(0x7f6d701007f0 legacy=0x7f6d70102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:46.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.130+0000 7f6d75c48700 1 -- 172.21.15.107:0/4217608436 shutdown_connections 2024-02-10T08:40:46.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.130+0000 7f6d75c48700 1 -- 172.21.15.107:0/4217608436 >> 172.21.15.107:0/4217608436 conn(0x7f6d700f3640 msgr2=0x7f6d700f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:46.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.130+0000 7f6d75c48700 1 -- 172.21.15.107:0/4217608436 shutdown_connections 2024-02-10T08:40:46.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.130+0000 7f6d75c48700 1 -- 172.21.15.107:0/4217608436 wait complete. 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 librados: starting msgr at 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 librados: starting objecter 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 Processor -- start 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 -- start start 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 librados: setting wanted keys 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 librados: calling monclient init 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d701194c0 con 0x7f6d701007f0 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d7011a660 con 0x7f6d70104330 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d75c48700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d7011b840 con 0x7f6d700fda50 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6d700fda50 0x7f6d700fc6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37670/0 (socket says 172.21.15.107:37670) 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6f7fe700 1 -- 172.21.15.107:0/2381572922 learned_addr learned my addr 172.21.15.107:0/2381572922 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6effd700 1 --1- 172.21.15.107:0/2381572922 >> v1:172.21.15.107:6789/0 conn(0x7f6d701007f0 0x7f6d700fcdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46926/0 (socket says 172.21.15.107:46926) 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6cff9700 10 client.?.objecter ms_handle_connect 0x7f6d700fda50 2024-02-10T08:40:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6cff9700 10 client.?.objecter ms_handle_connect 0x7f6d701007f0 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.131+0000 7f6d6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099026191 0 0) 0x7f6d7011b840 con 0x7f6d700fda50 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d580038e0 con 0x7f6d700fda50 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 10 client.?.objecter ms_handle_connect 0x7f6d70104330 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843108081 0 0) 0x7f6d701194c0 con 0x7f6d701007f0 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7011b840 con 0x7f6d701007f0 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618783403 0 0) 0x7f6d7011a660 con 0x7f6d70104330 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d701194c0 con 0x7f6d70104330 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480699775 0 0) 0x7f6d580038e0 con 0x7f6d700fda50 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7011a660 con 0x7f6d700fda50 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d60001a30 con 0x7f6d700fda50 2024-02-10T08:40:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241246945 0 0) 0x7f6d7011b840 con 0x7f6d701007f0 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d580038e0 con 0x7f6d701007f0 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d64002c60 con 0x7f6d701007f0 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725361212 0 0) 0x7f6d701194c0 con 0x7f6d70104330 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7011b840 con 0x7f6d70104330 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d68001aa0 con 0x7f6d70104330 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3740511070 0 0) 0x7f6d7011a660 con 0x7f6d700fda50 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 >> v1:172.21.15.195:6789/0 conn(0x7f6d70104330 legacy=0x7f6d70117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 >> v1:172.21.15.107:6789/0 conn(0x7f6d701007f0 legacy=0x7f6d700fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7011ca20 con 0x7f6d700fda50 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d701196b0 con 0x7f6d700fda50 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d75c48700 10 client.53465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d70119b20 con 0x7f6d700fda50 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d75c48700 1 librados: init done 2024-02-10T08:40:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6d60002510 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.132+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d60005ba0 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.133+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c004f20 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.133+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6d60018b10 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(352..352 src has 275..352) v4 ==== 8390+0+0 (unknown 707396139 0 0) 0x7f6d60078f60 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 10 client.53465.objecter ms_dispatch 0x7f6d70001550 osd_map(352..352 src has 275..352) v4 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 3 client.53465.objecter handle_osd_map got epochs [352,352] > 0 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 3 client.53465.objecter handle_osd_map decoding full epoch 352 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 20 client.53465.objecter dump_active .. 0 homeless 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.134+0000 7f6d6cff9700 10 client.53465.objecter ms_handle_connect 0x7f6d58063c20 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.138+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6d6004c380 con 0x7f6d700fda50 2024-02-10T08:40:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:46.328+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]} v 0) v1 -- 0x7f6d5c0059e0 con 0x7f6d700fda50 2024-02-10T08:40:46.477 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:46.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:46.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:46.609 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:46.627 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:46.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:46.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:46.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:40:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:47.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:47.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:47.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.272+0000 7f6d6cff9700 1 -- 172.21.15.107:0/2381572922 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]=0 set 4.1 pg_upmap mapping to [4,0,7] v353) v1 ==== 127+0+0 (unknown 1121074285 0 0) 0x7f6d6004a790 con 0x7f6d700fda50 2024-02-10T08:40:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.274+0000 7f6d75c48700 10 librados: watch_flush enter 2024-02-10T08:40:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 10 librados: watch_flush exit 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 20 client.53465.objecter shutdown clearing up homeless session... 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 10 client.53465.objecter successfully canceled tick 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6d58063c20 legacy=0x7f6d580660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 >> v1:172.21.15.107:6790/0 conn(0x7f6d700fda50 legacy=0x7f6d700fc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 shutdown_connections 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.275+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 >> 172.21.15.107:0/2381572922 conn(0x7f6d700f3640 msgr2=0x7f6d701077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.276+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 shutdown_connections 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.276+0000 7f6d75c48700 1 -- 172.21.15.107:0/2381572922 wait complete. 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.276+0000 7f6d75c48700 1 librados: shutdown 2024-02-10T08:40:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:set 4.1 pg_upmap mapping to [4,0,7] 2024-02-10T08:40:47.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:47.374 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.025786+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:46.025864+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.028009+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.032069+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.035179+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.035498+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.329694+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.107:0/2381572922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:46.330049+0000 mon.a (mon.0) 2255 : audit [INF] from='client.53465 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.025786+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:46.025864+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.028009+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.032069+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.035179+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.035498+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.329694+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.107:0/2381572922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:46.330049+0000 mon.a (mon.0) 2255 : audit [INF] from='client.53465 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.025786+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:46.025864+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.028009+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.032069+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.035179+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.035498+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.329694+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.107:0/2381572922' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:46.330049+0000 mon.a (mon.0) 2255 : audit [INF] from='client.53465 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]: dispatch 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.530+0000 7f637c550700 1 -- 172.21.15.107:0/476110154 >> v1:172.21.15.107:6789/0 conn(0x7f6374104330 legacy=0x7f63741067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.530+0000 7f637c550700 1 -- 172.21.15.107:0/476110154 shutdown_connections 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.530+0000 7f637c550700 1 -- 172.21.15.107:0/476110154 >> 172.21.15.107:0/476110154 conn(0x7f63740f2510 msgr2=0x7f63740f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.530+0000 7f637c550700 1 -- 172.21.15.107:0/476110154 shutdown_connections 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.530+0000 7f637c550700 1 -- 172.21.15.107:0/476110154 wait complete. 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 librados: starting msgr at 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 librados: starting objecter 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 Processor -- start 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 -- start start 2024-02-10T08:40:47.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 librados: setting wanted keys 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.531+0000 7f637c550700 1 librados: calling monclient init 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63741173d0 con 0x7f6374104330 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6374118570 con 0x7f63740f8a00 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6374119770 con 0x7f63740f95b0 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f6379aeb700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f63740f95b0 0x7f63740ffc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37692/0 (socket says 172.21.15.107:37692) 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f6379aeb700 1 -- 172.21.15.107:0/2653273624 learned_addr learned my addr 172.21.15.107:0/2653273624 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637aaed700 1 --1- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6789/0 conn(0x7f6374104330 0x7f6374115aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46940/0 (socket says 172.21.15.107:46940) 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter ms_handle_connect 0x7f63740f95b0 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter ms_handle_connect 0x7f6374104330 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter ms_handle_connect 0x7f63740f8a00 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717021389 0 0) 0x7f6374119770 con 0x7f63740f95b0 2024-02-10T08:40:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63600033f0 con 0x7f63740f95b0 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653292502 0 0) 0x7f63741173d0 con 0x7f6374104330 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374119770 con 0x7f6374104330 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757451041 0 0) 0x7f6374118570 con 0x7f63740f8a00 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63741173d0 con 0x7f63740f8a00 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953622787 0 0) 0x7f63600033f0 con 0x7f63740f95b0 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6374118570 con 0x7f63740f95b0 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f63640019d0 con 0x7f63740f95b0 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656433967 0 0) 0x7f6374119770 con 0x7f6374104330 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63600033f0 con 0x7f6374104330 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f636c0021c0 con 0x7f6374104330 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796370002 0 0) 0x7f63741173d0 con 0x7f63740f8a00 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6374119770 con 0x7f63740f8a00 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f63700018c0 con 0x7f63740f8a00 2024-02-10T08:40:47.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2935287843 0 0) 0x7f6374118570 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.195:6789/0 conn(0x7f63740f8a00 legacy=0x7f63740ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6789/0 conn(0x7f6374104330 legacy=0x7f6374115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637411a970 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63741175c0 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 10 client.53471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6374117a30 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.532+0000 7f637c550700 1 librados: init done 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.533+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6364001ce0 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.533+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6364005be0 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.533+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6358004f00 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6364002270 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(353..353 src has 275..353) v4 ==== 8546+0+0 (unknown 82189628 0 0) 0x7f63640790e0 con 0x7f63740f95b0 2024-02-10T08:40:47.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 osd_map(353..353 src has 275..353) v4 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 3 client.53471.objecter handle_osd_map got epochs [353,353] > 0 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 3 client.53471.objecter handle_osd_map decoding full epoch 353 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 20 client.53471.objecter dump_active .. 0 homeless 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v353) v1 ==== 54+0+16 (unknown 1391655278 0 2420000192) 0x7f63640794c0 con 0x7f63740f95b0 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6360063c60 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 10 client.53471.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 1 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 1 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 0 1 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 4 2024-02-10T08:40:47.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 10 client.53471.objecter _send_command 1 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.534+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6358004950 con 0x7f6358004530 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358004530 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6358004950 con 0x7f6358004530 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 10 client.53471.objecter _finish_command 1 = system:0 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 4 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.535+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 0 1 2024-02-10T08:40:47.734 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 10 client.53471.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 2 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 2 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f63580051a0 osd=0 3 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 0 2 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 4 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 10 client.53471.objecter _send_command 2 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.550+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358004d30 con 0x7f6358004530 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358004d30 con 0x7f6358004530 2024-02-10T08:40:47.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f636b7fe700 10 client.53471.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 4 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 0 2 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 10 client.53471.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 1 3 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 4 2024-02-10T08:40:47.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 10 client.53471.objecter _send_command 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.551+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f635800bf60 con 0x7f6358003b40 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358003b40 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f635800bf60 con 0x7f6358003b40 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 10 client.53471.objecter _finish_command 3 = system:0 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 4 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.552+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 1 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 10 client.53471.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 4 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 4 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358004270 osd=1 3 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 1 4 2024-02-10T08:40:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 4 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 10 client.53471.objecter _send_command 4 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.566+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f635800d280 con 0x7f6358003b40 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f635800d280 con 0x7f6358003b40 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 10 client.53471.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 4 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 1 4 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 10 client.53471.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 5 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 5 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 2 5 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 4 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 10 client.53471.objecter _send_command 5 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6358010d30 con 0x7f6358002070 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.568+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358002070 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.569+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6358010d30 con 0x7f6358002070 2024-02-10T08:40:47.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.569+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.569+0000 7f636b7fe700 10 client.53471.objecter _finish_command 5 = system:0 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.569+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 4 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.569+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 2 5 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 10 client.53471.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 6 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 6 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f63580033a0 osd=2 3 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 2 6 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 4 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 10 client.53471.objecter _send_command 6 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.583+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358001630 con 0x7f6358002070 2024-02-10T08:40:47.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358001630 con 0x7f6358002070 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 10 client.53471.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 4 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 2 6 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:47.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 10 client.53471.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 7 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 7 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 3 7 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 4 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 10 client.53471.objecter _send_command 7 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f635800d050 con 0x7f6358010ed0 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.585+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358010ed0 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.586+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f635800d050 con 0x7f6358010ed0 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.586+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.586+0000 7f636b7fe700 10 client.53471.objecter _finish_command 7 = system:0 2024-02-10T08:40:47.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.586+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 4 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.586+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 3 7 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 10 client.53471.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 8 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 8 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f635800c8d0 osd=3 3 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 3 8 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 4 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 10 client.53471.objecter _send_command 8 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.600+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358012240 con 0x7f6358010ed0 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358012240 con 0x7f6358010ed0 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f636b7fe700 10 client.53471.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 4 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 3 8 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 10 client.53471.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 9 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 9 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 4 9 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 4 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 10 client.53471.objecter _send_command 9 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.601+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f635801e6d0 con 0x7f635801ab00 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f635801ab00 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f635801e6d0 con 0x7f635801ab00 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 10 client.53471.objecter _finish_command 9 = system:0 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 4 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.602+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 4 9 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 10 client.53471.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 10 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 10 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358011a70 osd=4 3 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 4 10 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 4 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 10 client.53471.objecter _send_command 10 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.617+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f635801e6d0 con 0x7f635801ab00 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f635801e6d0 con 0x7f635801ab00 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f636b7fe700 10 client.53471.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 4 2024-02-10T08:40:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 4 10 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 10 client.53471.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 11 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 11 2024-02-10T08:40:47.747 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 5 11 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 4 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 10 client.53471.objecter _send_command 11 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.619+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6358013210 con 0x7f6358012cb0 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358012cb0 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6358013210 con 0x7f6358012cb0 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 10 client.53471.objecter _finish_command 11 = system:0 2024-02-10T08:40:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 4 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.620+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 5 11 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 10 client.53471.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 12 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 12 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f635801e6d0 osd=5 3 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 5 12 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 4 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 10 client.53471.objecter _send_command 12 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.634+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358013210 con 0x7f6358012cb0 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358013210 con 0x7f6358012cb0 2024-02-10T08:40:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f636b7fe700 10 client.53471.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 4 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 5 12 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f637c550700 10 client.53471.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.636+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 13 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 13 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 6 13 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 4 2024-02-10T08:40:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 10 client.53471.objecter _send_command 13 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6358020a50 con 0x7f6358020610 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.637+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358020610 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.638+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6358020a50 con 0x7f6358020610 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.638+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.638+0000 7f636b7fe700 10 client.53471.objecter _finish_command 13 = system:0 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.638+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 4 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.638+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 6 13 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 10 client.53471.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 14 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 14 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358013210 osd=6 3 2024-02-10T08:40:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 6 14 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 4 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 10 client.53471.objecter _send_command 14 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.652+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358020a50 con 0x7f6358020610 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358020a50 con 0x7f6358020610 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 10 client.53471.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 4 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 6 14 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 10 client.53471.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 15 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 15 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 7 15 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 4 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 10 client.53471.objecter _send_command 15 2024-02-10T08:40:47.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6358020e50 con 0x7f6358027cd0 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.654+0000 7f636b7fe700 10 client.53471.objecter ms_handle_connect 0x7f6358027cd0 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.655+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6358020e50 con 0x7f6358027cd0 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.655+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.655+0000 7f636b7fe700 10 client.53471.objecter _finish_command 15 = system:0 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.655+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 4 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.655+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 7 15 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 10 client.53471.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign -1 16 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 20 client.53471.objecter _get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 15 client.53471.objecter _session_command_op_remove -1 16 2024-02-10T08:40:47.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 20 client.53471.objecter get_session s=0x7f6358020a50 osd=7 3 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 15 client.53471.objecter _session_command_op_assign 7 16 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 4 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 10 client.53471.objecter _send_command 16 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.670+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6358020e50 con 0x7f6358027cd0 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f636b7fe700 1 -- 172.21.15.107:0/2653273624 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6358020e50 con 0x7f6358027cd0 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f636b7fe700 10 client.53471.objecter ms_dispatch 0x7f6374001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f636b7fe700 10 client.53471.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f636b7fe700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 4 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f636b7fe700 15 client.53471.objecter _session_command_op_remove 7 16 2024-02-10T08:40:47.755 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 librados: watch_flush enter 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 librados: watch_flush exit 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.0 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6801/3321596972 conn(0x7f6358004530 legacy=0x7f6358002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580051a0 osd=0 1 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.1 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6805/133516126 conn(0x7f6358003b40 legacy=0x7f635800a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358004270 osd=1 1 2024-02-10T08:40:47.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.2 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6809/3575881728 conn(0x7f6358002070 legacy=0x7f635800f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f63580033a0 osd=2 1 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.3 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6813/2871825691 conn(0x7f6358010ed0 legacy=0x7f6358011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635800c8d0 osd=3 1 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.4 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.195:6800/2499671285 conn(0x7f635801ab00 legacy=0x7f635801cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358011a70 osd=4 1 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.5 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.195:6804/4178321486 conn(0x7f6358012cb0 legacy=0x7f635801e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637aaed700 1 -- 172.21.15.107:0/2653273624 reap_dead start 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f635801e6d0 osd=5 1 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.6 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.195:6808/1417479017 conn(0x7f6358020610 legacy=0x7f63580260a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637aaed700 1 -- 172.21.15.107:0/2653273624 reap_dead start 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358013210 osd=6 1 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter close_session for osd.7 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.195:6812/1934176928 conn(0x7f6358027cd0 legacy=0x7f635802a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter put_session s=0x7f6358020a50 osd=7 1 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637aaed700 1 -- 172.21.15.107:0/2653273624 reap_dead start 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 20 client.53471.objecter shutdown clearing up homeless session... 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 10 client.53471.objecter successfully canceled tick 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637aaed700 1 -- 172.21.15.107:0/2653273624 reap_dead start 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6360063c60 legacy=0x7f6360066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> v1:172.21.15.107:6790/0 conn(0x7f63740f95b0 legacy=0x7f63740ffc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 shutdown_connections 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 >> 172.21.15.107:0/2653273624 conn(0x7f63740f2510 msgr2=0x7f6374103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 shutdown_connections 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 -- 172.21.15.107:0/2653273624 wait complete. 2024-02-10T08:40:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:47.672+0000 7f637c550700 1 librados: shutdown 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:40:47.759 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:48.007 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:48.224 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:48.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:48.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:48.355 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:48.453 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:48.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:48.479 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:47.272617+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:47.272779+0000 mon.a (mon.0) 2257 : audit [INF] from='client.53465 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]': finished 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:47.272858+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:47.276836+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:47.282659+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:47.421242+0000 mgr.y (mgr.14553) 326 : cluster [DBG] pgmap v506: 218 pgs: 13 creating+peering, 19 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:47.534461+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.107:0/2653273624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:48.029659+0000 mon.a (mon.0) 2260 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:48.029766+0000 mon.a (mon.0) 2261 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:48.029801+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:48.031004+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:48.031415+0000 mon.a (mon.0) 2263 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:47.272617+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:47.272779+0000 mon.a (mon.0) 2257 : audit [INF] from='client.53465 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]': finished 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:47.272858+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:47.276836+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:47.282659+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:47.421242+0000 mgr.y (mgr.14553) 326 : cluster [DBG] pgmap v506: 218 pgs: 13 creating+peering, 19 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:47.534461+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.107:0/2653273624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:48.029659+0000 mon.a (mon.0) 2260 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:48.029766+0000 mon.a (mon.0) 2261 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:48.029801+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:48.031004+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:48.031415+0000 mon.a (mon.0) 2263 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:47.272617+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:47.272779+0000 mon.a (mon.0) 2257 : audit [INF] from='client.53465 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1", "id": [4, 0, 7]}]': finished 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:47.272858+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:47.276836+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:47.282659+0000 mon.a (mon.0) 2259 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:47.421242+0000 mgr.y (mgr.14553) 326 : cluster [DBG] pgmap v506: 218 pgs: 13 creating+peering, 19 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:47.534461+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.107:0/2653273624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:48.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:48.029659+0000 mon.a (mon.0) 2260 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi107-112411-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:48.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:48.029766+0000 mon.a (mon.0) 2261 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:40:48.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:48.029801+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-10T08:40:48.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:48.031004+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:48.031415+0000 mon.a (mon.0) 2263 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:40:48.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:40:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:50.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:49.029318+0000 mon.a (mon.0) 2264 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:40:50.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:49.029428+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-10T08:40:50.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:49.031981+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:49.032634+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:49.421753+0000 mgr.y (mgr.14553) 327 : cluster [DBG] pgmap v509: 226 pgs: 13 creating+peering, 27 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:49.029318+0000 mon.a (mon.0) 2264 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:49.029428+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:49.031981+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:49.032634+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:49.421753+0000 mgr.y (mgr.14553) 327 : cluster [DBG] pgmap v509: 226 pgs: 13 creating+peering, 27 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:50.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:40:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:49.029318+0000 mon.a (mon.0) 2264 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:40:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:49.029428+0000 mon.a (mon.0) 2265 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-10T08:40:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:49.031981+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:49.032634+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]: dispatch 2024-02-10T08:40:50.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:49.421753+0000 mgr.y (mgr.14553) 327 : cluster [DBG] pgmap v509: 226 pgs: 13 creating+peering, 27 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 109 B/s, 3 objects/s recovering 2024-02-10T08:40:50.536 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:40:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:50.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:40:50.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:50.694 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:50.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:50.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:50.829 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:50.848 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:50.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 -- 172.21.15.107:0/2306128875 >> v1:172.21.15.107:6789/0 conn(0x7f669c0f89d0 legacy=0x7f669c0f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 -- 172.21.15.107:0/2306128875 shutdown_connections 2024-02-10T08:40:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 -- 172.21.15.107:0/2306128875 >> 172.21.15.107:0/2306128875 conn(0x7f669c0f2540 msgr2=0x7f669c0f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 -- 172.21.15.107:0/2306128875 shutdown_connections 2024-02-10T08:40:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 -- 172.21.15.107:0/2306128875 wait complete. 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 librados: starting msgr at 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.853+0000 7f66a3113700 1 librados: starting objecter 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 Processor -- start 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 -- start start 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 librados: setting wanted keys 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 librados: calling monclient init 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f669c1004d0 con 0x7f669c104360 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f669c118840 con 0x7f669c0f9580 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a3113700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f669c1199e0 con 0x7f669c0f89d0 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a0eaf700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f669c0f89d0 0x7f669c0ff8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50236/0 (socket says 172.21.15.107:50236) 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a0eaf700 1 -- 172.21.15.107:0/228564985 learned_addr learned my addr 172.21.15.107:0/228564985 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f66a16b0700 1 --1- 172.21.15.107:0/228564985 >> v1:172.21.15.107:6789/0 conn(0x7f669c104360 0x7f669c116110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33730/0 (socket says 172.21.15.107:33730) 2024-02-10T08:40:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter ms_handle_connect 0x7f669c104360 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter ms_handle_connect 0x7f669c0f89d0 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter ms_handle_connect 0x7f669c0f9580 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025193916 0 0) 0x7f669c118840 con 0x7f669c0f9580 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668c003030 con 0x7f669c0f9580 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580799934 0 0) 0x7f669c1004d0 con 0x7f669c104360 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c118840 con 0x7f669c104360 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712539721 0 0) 0x7f669c1199e0 con 0x7f669c0f89d0 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.854+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c1004d0 con 0x7f669c0f89d0 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1302738927 0 0) 0x7f669c1004d0 con 0x7f669c0f89d0 2024-02-10T08:40:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f669c1199e0 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6690002d60 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1411827581 0 0) 0x7f668c003030 con 0x7f669c0f9580 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f669c1004d0 con 0x7f669c0f9580 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3770823123 0 0) 0x7f669c118840 con 0x7f669c104360 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f668c003030 con 0x7f669c104360 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6688001910 con 0x7f669c0f9580 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6694001b00 con 0x7f669c104360 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1543548981 0 0) 0x7f669c1199e0 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 >> v1:172.21.15.195:6789/0 conn(0x7f669c0f9580 legacy=0x7f669c1129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 >> v1:172.21.15.107:6789/0 conn(0x7f669c104360 legacy=0x7f669c116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c11abc0 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c118a30 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6690002560 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6690003ac0 con 0x7f669c0f89d0 2024-02-10T08:40:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f66a3113700 10 client.53477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c118ea0 con 0x7f669c0f89d0 2024-02-10T08:40:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.855+0000 7f66a3113700 1 librados: init done 2024-02-10T08:40:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.856+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c0603f0 con 0x7f669c0f89d0 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.856+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f66900020c0 con 0x7f669c0f89d0 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(356..356 src has 275..356) v4 ==== 8554+0+0 (unknown 3323375022 0 0) 0x7f6690079250 con 0x7f669c0f89d0 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 10 client.53477.objecter ms_dispatch 0x7f669c001550 osd_map(356..356 src has 275..356) v4 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 3 client.53477.objecter handle_osd_map got epochs [356,356] > 0 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 3 client.53477.objecter handle_osd_map decoding full epoch 356 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 20 client.53477.objecter dump_active .. 0 homeless 2024-02-10T08:40:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.857+0000 7f6699ffb700 10 client.53477.objecter ms_handle_connect 0x7f668c063e20 2024-02-10T08:40:50.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:50.861+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f669004c510 con 0x7f669c0f89d0 2024-02-10T08:40:50.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:51.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:51.054+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f669c061550 con 0x7f669c0f89d0 2024-02-10T08:40:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:50.026968+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:50.035398+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]': finished 2024-02-10T08:40:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:50.035502+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-10T08:40:51.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:50.037851+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:50.040081+0000 mon.a (mon.0) 2270 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:50.998218+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:50.998289+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:50.999388+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:51.005088+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:50.026968+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:50.035398+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]': finished 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:50.035502+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:50.037851+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:50.040081+0000 mon.a (mon.0) 2270 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:50.998218+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:51.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:50.998289+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-10T08:40:51.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:50.999388+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:51.005088+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:50.026968+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:50.035398+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-47", "mode": "readproxy"}]': finished 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:50.035502+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:50.037851+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:50.040081+0000 mon.a (mon.0) 2270 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:50.998218+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:50.998289+0000 mon.a (mon.0) 2272 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-10T08:40:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:50.999388+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.107:0/4227792630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:51.005088+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48178 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]: dispatch 2024-02-10T08:40:51.736 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:51.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:51.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:40:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:40:51.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:51.871 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:51.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:51.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:52.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.006+0000 7f6699ffb700 1 -- 172.21.15.107:0/228564985 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v358) v1 ==== 97+0+0 (unknown 1553591857 0 0) 0x7f669004a980 con 0x7f669c0f89d0 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.010+0000 7f66a3113700 10 librados: watch_flush enter 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.010+0000 7f66a3113700 10 librados: watch_flush exit 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.010+0000 7f66a3113700 20 client.53477.objecter shutdown clearing up homeless session... 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.010+0000 7f66a3113700 10 client.53477.objecter successfully canceled tick 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.010+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 >> v1:172.21.15.107:6800/2634842009 conn(0x7f668c063e20 legacy=0x7f668c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.011+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 >> v1:172.21.15.107:6790/0 conn(0x7f669c0f89d0 legacy=0x7f669c0ff8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.012+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 shutdown_connections 2024-02-10T08:40:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.012+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 >> 172.21.15.107:0/228564985 conn(0x7f669c0f2540 msgr2=0x7f669c1037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.012+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 shutdown_connections 2024-02-10T08:40:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.012+0000 7f66a3113700 1 -- 172.21.15.107:0/228564985 wait complete. 2024-02-10T08:40:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:52.012+0000 7f66a3113700 1 librados: shutdown 2024-02-10T08:40:52.013 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:40:52.027 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:52.050 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:52.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:52.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:52.185 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:52.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:51.055475+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.107:0/228564985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:51.055753+0000 mon.a (mon.0) 2274 : audit [INF] from='client.53477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:51.391070+0000 mgr.y (mgr.14553) 328 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:51.422572+0000 mgr.y (mgr.14553) 329 : cluster [DBG] pgmap v512: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:52.002479+0000 mon.a (mon.0) 2275 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:52.006557+0000 mon.a (mon.0) 2276 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:52.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:52.006617+0000 mon.a (mon.0) 2277 : audit [INF] from='client.53477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:52.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:52.006661+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-10T08:40:52.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:52.025650+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:52.026736+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:52.027007+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:51.055475+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.107:0/228564985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:51.055753+0000 mon.a (mon.0) 2274 : audit [INF] from='client.53477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:51.391070+0000 mgr.y (mgr.14553) 328 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:51.422572+0000 mgr.y (mgr.14553) 329 : cluster [DBG] pgmap v512: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:52.002479+0000 mon.a (mon.0) 2275 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:52.263 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:52.006557+0000 mon.a (mon.0) 2276 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:52.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:52.006617+0000 mon.a (mon.0) 2277 : audit [INF] from='client.53477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:52.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:52.006661+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-10T08:40:52.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:52.025650+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:52.026736+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:52.027007+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:52.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:51.055475+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.107:0/228564985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:51.055753+0000 mon.a (mon.0) 2274 : audit [INF] from='client.53477 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:51.391070+0000 mgr.y (mgr.14553) 328 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:51.422572+0000 mgr.y (mgr.14553) 329 : cluster [DBG] pgmap v512: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:52.002479+0000 mon.a (mon.0) 2275 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:52.006557+0000 mon.a (mon.0) 2276 : audit [INF] from='client.48178 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi107-112411-61"}]': finished 2024-02-10T08:40:52.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:52.006617+0000 mon.a (mon.0) 2277 : audit [INF] from='client.53477 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:40:52.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:52.006661+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-10T08:40:52.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:52.025650+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:52.026736+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:52.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:52.027007+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:40:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:52.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:52.930 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:40:52.930 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:40:52.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:52.974 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:52.995 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:52.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:53.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:53.130 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:53.150 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:53.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:53.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:53.283 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:52.063425+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:52.063746+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:53.014229+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:53.014301+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-10T08:40:53.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:53.015894+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:53.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:52.063425+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:53.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:52.063746+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:53.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:53.014229+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:53.014301+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:53.015894+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:52.063425+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:52.063746+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:53.014229+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi107-112411-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:53.014301+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-10T08:40:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:53.015894+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:53.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:53.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:40:53.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:40:53.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:53.766 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:53.787 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:53.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:53.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:53.920 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:53.940 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:53.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:53.959 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:54.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:54.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:54.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:54.350 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:54.368 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:54.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:53.422927+0000 mgr.y (mgr.14553) 330 : cluster [DBG] pgmap v515: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:53.422927+0000 mgr.y (mgr.14553) 330 : cluster [DBG] pgmap v515: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:54.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:40:54.499 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:54.518 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:40:54.519 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:53.422927+0000 mgr.y (mgr.14553) 330 : cluster [DBG] pgmap v515: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:40:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:54.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:40:54.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:54.654 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:54.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:40:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:55.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:55.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:55.136 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:55.156 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:55.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:55.288 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:55.321 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:40:55.321 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:40:55.322 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:40:55.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:40:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:54.255375+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-10T08:40:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:55.011773+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:55.011845+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-10T08:40:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:55.155279+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:54.255375+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-10T08:40:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:55.011773+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:55.011845+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-10T08:40:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:55.155279+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:55.486 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:55.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:54.255375+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-10T08:40:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:55.011773+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi107-112411-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:55.011845+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-10T08:40:55.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:55.155279+0000 mon.a (mon.0) 2288 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:40:55.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:55.916 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:40:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:40:55] "GET /metrics HTTP/1.1" 200 189370 "" "Prometheus/2.33.4" 2024-02-10T08:40:56.465 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:56.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:40:56.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:55.423300+0000 mgr.y (mgr.14553) 331 : cluster [DBG] pgmap v518: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:56.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:56.015630+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-10T08:40:56.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:55.423300+0000 mgr.y (mgr.14553) 331 : cluster [DBG] pgmap v518: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:56.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:56.015630+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-10T08:40:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:55.423300+0000 mgr.y (mgr.14553) 331 : cluster [DBG] pgmap v518: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:40:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:56.015630+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-10T08:40:56.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:56.599 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:56.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:56.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:40:56.833 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:56.855 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:40:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:56.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:40:56.931 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:40:56.931 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:40:56.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:40:56.988 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:57.009 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:40:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:57.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:40:57.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:57.141 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:57.161 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:57.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:57.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:40:57.277 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:56.082710+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:57.277 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:56.083021+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:57.277 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:57.016866+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-10T08:40:57.278 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:57.021510+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.376+0000 7f0213d25700 1 -- 172.21.15.107:0/1323632610 >> v1:172.21.15.107:6790/0 conn(0x7f020c103250 legacy=0x7f020c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.376+0000 7f0213d25700 1 -- 172.21.15.107:0/1323632610 shutdown_connections 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.376+0000 7f0213d25700 1 -- 172.21.15.107:0/1323632610 >> 172.21.15.107:0/1323632610 conn(0x7f020c0f2510 msgr2=0x7f020c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.376+0000 7f0213d25700 1 -- 172.21.15.107:0/1323632610 shutdown_connections 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.376+0000 7f0213d25700 1 -- 172.21.15.107:0/1323632610 wait complete. 2024-02-10T08:40:57.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 librados: starting msgr at 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 librados: starting objecter 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 Processor -- start 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 -- start start 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 librados: setting wanted keys 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 librados: calling monclient init 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f020c117420 con 0x7f020c0fc910 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f020c1185e0 con 0x7f020c0ff710 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0213d25700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f020c1197e0 con 0x7f020c103250 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0211ac1700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f020c0fc910 0x7f020c060d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33748/0 (socket says 172.21.15.107:33748) 2024-02-10T08:40:57.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0211ac1700 1 -- 172.21.15.107:0/2080645700 learned_addr learned my addr 172.21.15.107:0/2080645700 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter ms_handle_connect 0x7f020c103250 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter ms_handle_connect 0x7f020c0fc910 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227190215 0 0) 0x7f020c117420 con 0x7f020c0fc910 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f80038e0 con 0x7f020c0fc910 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter ms_handle_connect 0x7f020c0ff710 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644647820 0 0) 0x7f020c1197e0 con 0x7f020c103250 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c117420 con 0x7f020c103250 2024-02-10T08:40:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943015343 0 0) 0x7f020c1185e0 con 0x7f020c0ff710 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.377+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c1197e0 con 0x7f020c0ff710 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1554162024 0 0) 0x7f020c117420 con 0x7f020c103250 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f020c1185e0 con 0x7f020c103250 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0204001ce0 con 0x7f020c103250 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816138307 0 0) 0x7f01f80038e0 con 0x7f020c0fc910 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f020c117420 con 0x7f020c0fc910 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f01fc001840 con 0x7f020c0fc910 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296214070 0 0) 0x7f020c1197e0 con 0x7f020c0ff710 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f80038e0 con 0x7f020c0ff710 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02080019f0 con 0x7f020c0ff710 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4233086938 0 0) 0x7f020c1185e0 con 0x7f020c103250 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.195:6789/0 conn(0x7f020c0ff710 legacy=0x7f020c061460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6789/0 conn(0x7f020c0fc910 legacy=0x7f020c060d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c11a9e0 con 0x7f020c103250 2024-02-10T08:40:57.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c117610 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0213d25700 10 client.53489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c117a80 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0213d25700 1 librados: init done 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0204001360 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.378+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02040038a0 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.379+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f01f0004f00 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.379+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0204002250 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(363..363 src has 275..363) v4 ==== 8550+0+0 (unknown 3471237124 0 0) 0x7f0204076be0 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 osd_map(363..363 src has 275..363) v4 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 3 client.53489.objecter handle_osd_map got epochs [363,363] > 0 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 3 client.53489.objecter handle_osd_map decoding full epoch 363 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 20 client.53489.objecter dump_active .. 0 homeless 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v363) v1 ==== 54+0+16 (unknown 272900468 0 2420000192) 0x7f0204076f90 con 0x7f020c103250 2024-02-10T08:40:57.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f8063d00 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 10 client.53489.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 1 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 1 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 0 1 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 4 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 10 client.53489.objecter _send_command 1 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f0004950 con 0x7f01f0004530 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.380+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0004530 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.381+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f0004950 con 0x7f01f0004530 2024-02-10T08:40:57.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.381+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.381+0000 7f0202ffd700 10 client.53489.objecter _finish_command 1 = system:0 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.381+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 4 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.381+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 0 1 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 10 client.53489.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 2 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 2 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f00051a0 osd=0 3 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 0 2 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 4 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 10 client.53489.objecter _send_command 2 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.396+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f0004d30 con 0x7f01f0004530 2024-02-10T08:40:57.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f0004d30 con 0x7f01f0004530 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0202ffd700 10 client.53489.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 4 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 0 2 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:40:57.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 10 client.53489.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 1 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 4 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 10 client.53489.objecter _send_command 3 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.397+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f000c000 con 0x7f01f0004150 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0004150 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f000c000 con 0x7f01f0004150 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 10 client.53489.objecter _finish_command 3 = system:0 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 4 2024-02-10T08:40:57.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.398+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 1 3 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 10 client.53489.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 4 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 4 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003fe0 osd=1 3 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 1 4 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 4 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 10 client.53489.objecter _send_command 4 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.413+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f00018b0 con 0x7f01f0004150 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.414+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f00018b0 con 0x7f01f0004150 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.414+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.414+0000 7f0202ffd700 10 client.53489.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.414+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 4 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.414+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 1 4 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 10 client.53489.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 5 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:40:57.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 5 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 2 5 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 4 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 10 client.53489.objecter _send_command 5 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f000de90 con 0x7f01f000d2f0 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f000d2f0 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f000de90 con 0x7f01f000d2f0 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 10 client.53489.objecter _finish_command 5 = system:0 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 4 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.415+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 2 5 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 10 client.53489.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 6 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 6 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003630 osd=2 3 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 2 6 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 4 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 10 client.53489.objecter _send_command 6 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.430+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f00030a0 con 0x7f01f000d2f0 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.431+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f00030a0 con 0x7f01f000d2f0 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.431+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.431+0000 7f0202ffd700 10 client.53489.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.431+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 4 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.431+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 2 6 2024-02-10T08:40:57.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 10 client.53489.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 7 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 7 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 3 7 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 4 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 10 client.53489.objecter _send_command 7 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f0015a60 con 0x7f01f0011e80 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0011e80 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f0015a60 con 0x7f01f0011e80 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:40:57.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 10 client.53489.objecter _finish_command 7 = system:0 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 4 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.432+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 3 7 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 10 client.53489.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 8 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 8 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0003480 osd=3 3 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 3 8 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 4 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 10 client.53489.objecter _send_command 8 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.447+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f000e270 con 0x7f01f0011e80 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f000e270 con 0x7f01f0011e80 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0202ffd700 10 client.53489.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 4 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 3 8 2024-02-10T08:40:57.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 10 client.53489.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 9 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 9 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 4 9 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 4 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 10 client.53489.objecter _send_command 9 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.448+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f000e190 con 0x7f01f000ce50 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.449+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f000ce50 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.450+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f000e190 con 0x7f01f000ce50 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.450+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:40:57.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.450+0000 7f0202ffd700 10 client.53489.objecter _finish_command 9 = system:0 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.450+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 4 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.450+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 4 9 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 10 client.53489.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 10 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 10 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f000e690 osd=4 3 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 4 10 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 4 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 10 client.53489.objecter _send_command 10 2024-02-10T08:40:57.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.464+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f000e190 con 0x7f01f000ce50 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.466+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f000e190 con 0x7f01f000ce50 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.466+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.466+0000 7f0202ffd700 10 client.53489.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.466+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 4 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.466+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 4 10 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 10 client.53489.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 11 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:40:57.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 11 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 5 11 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 4 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 10 client.53489.objecter _send_command 11 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f0016de0 con 0x7f01f0016830 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.467+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0016830 2024-02-10T08:40:57.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.468+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f0016de0 con 0x7f01f0016830 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.468+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.468+0000 7f0202ffd700 10 client.53489.objecter _finish_command 11 = system:0 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.468+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 4 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.468+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 5 11 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 10 client.53489.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 12 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 12 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f000e190 osd=5 3 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 5 12 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 4 2024-02-10T08:40:57.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 10 client.53489.objecter _send_command 12 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.482+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f0016de0 con 0x7f01f0016830 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f0016de0 con 0x7f01f0016830 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0202ffd700 10 client.53489.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 4 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 5 12 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 10 client.53489.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 13 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 13 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 6 13 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 4 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 10 client.53489.objecter _send_command 13 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.484+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f0026f80 con 0x7f01f0023470 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0023470 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f0026f80 con 0x7f01f0023470 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 10 client.53489.objecter _finish_command 13 = system:0 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 4 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.485+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 6 13 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 10 client.53489.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 14 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 14 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0016de0 osd=6 3 2024-02-10T08:40:57.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 6 14 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 4 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 10 client.53489.objecter _send_command 14 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.500+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f0028730 con 0x7f01f0023470 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.501+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f0028730 con 0x7f01f0023470 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.501+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.501+0000 7f0202ffd700 10 client.53489.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.501+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 4 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.501+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 6 14 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 10 client.53489.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 15 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 15 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 7 15 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 4 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 10 client.53489.objecter _send_command 15 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01f00293b0 con 0x7f01f0028df0 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.502+0000 7f0202ffd700 10 client.53489.objecter ms_handle_connect 0x7f01f0028df0 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.503+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f01f00293b0 con 0x7f01f0028df0 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.503+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.503+0000 7f0202ffd700 10 client.53489.objecter _finish_command 15 = system:0 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.503+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 4 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.503+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 7 15 2024-02-10T08:40:57.535 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 10 client.53489.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign -1 16 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 20 client.53489.objecter _get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 15 client.53489.objecter _session_command_op_remove -1 16 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 20 client.53489.objecter get_session s=0x7f01f0028730 osd=7 3 2024-02-10T08:40:57.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 15 client.53489.objecter _session_command_op_assign 7 16 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 4 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 10 client.53489.objecter _send_command 16 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.521+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f01f0029370 con 0x7f01f0028df0 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0202ffd700 1 -- 172.21.15.107:0/2080645700 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f01f0029370 con 0x7f01f0028df0 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0202ffd700 10 client.53489.objecter ms_dispatch 0x7f020c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0202ffd700 10 client.53489.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0202ffd700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 4 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0202ffd700 15 client.53489.objecter _session_command_op_remove 7 16 2024-02-10T08:40:57.537 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:40:57.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:56.082710+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:57.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:56.083021+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:57.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:57.016866+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-10T08:40:57.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:57.021510+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 librados: watch_flush enter 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 librados: watch_flush exit 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.0 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6801/3321596972 conn(0x7f01f0004530 legacy=0x7f01f0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f00051a0 osd=0 1 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.1 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6805/133516126 conn(0x7f01f0004150 legacy=0x7f01f000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003fe0 osd=1 1 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.2 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6809/3575881728 conn(0x7f01f000d2f0 legacy=0x7f01f000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003630 osd=2 1 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.3 2024-02-10T08:40:57.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6813/2871825691 conn(0x7f01f0011e80 legacy=0x7f01f0014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0003480 osd=3 1 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.4 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.195:6800/2499671285 conn(0x7f01f000ce50 legacy=0x7f01f0015bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e690 osd=4 1 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.5 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.195:6804/4178321486 conn(0x7f01f0016830 legacy=0x7f01f001d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f000e190 osd=5 1 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.6 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.195:6808/1417479017 conn(0x7f01f0023470 legacy=0x7f01f0025860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0016de0 osd=6 1 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter close_session for osd.7 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.195:6812/1934176928 conn(0x7f01f0028df0 legacy=0x7f01f0026f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter put_session s=0x7f01f0028730 osd=7 1 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 20 client.53489.objecter shutdown clearing up homeless session... 2024-02-10T08:40:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 10 client.53489.objecter successfully canceled tick 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f02122c2700 1 -- 172.21.15.107:0/2080645700 reap_dead start 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f02122c2700 1 -- 172.21.15.107:0/2080645700 reap_dead start 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f02122c2700 1 -- 172.21.15.107:0/2080645700 reap_dead start 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f02122c2700 1 -- 172.21.15.107:0/2080645700 reap_dead start 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f02122c2700 1 -- 172.21.15.107:0/2080645700 reap_dead start 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.524+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6800/2634842009 conn(0x7f01f8063d00 legacy=0x7f01f80661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> v1:172.21.15.107:6790/0 conn(0x7f020c103250 legacy=0x7f020c115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 shutdown_connections 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 >> 172.21.15.107:0/2080645700 conn(0x7f020c0f2510 msgr2=0x7f020c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 shutdown_connections 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 -- 172.21.15.107:0/2080645700 wait complete. 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:40:57.525+0000 7f0213d25700 1 librados: shutdown 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.542 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.542 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.542 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:40:57.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:56.082710+0000 osd.1 (osd.1) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:40:57.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:56.083021+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:40:57.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:57.016866+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-10T08:40:57.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:57.021510+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:57.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:40:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:40:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:58.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:40:58.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:58.177 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:58.201 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:58.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:58.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:40:58.334 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:58.355 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:40:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:58.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:40:58.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:40:58.487 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:57.380474+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.107:0/2080645700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:57.423684+0000 mgr.y (mgr.14553) 332 : cluster [DBG] pgmap v521: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:58.017901+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:58.017953+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-10T08:40:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:40:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:58.018629+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:57.380474+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.107:0/2080645700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:57.423684+0000 mgr.y (mgr.14553) 332 : cluster [DBG] pgmap v521: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:58.017901+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:58.017953+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:58.018629+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:57.380474+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.107:0/2080645700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:57.423684+0000 mgr.y (mgr.14553) 332 : cluster [DBG] pgmap v521: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:58.017901+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:58.017953+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-10T08:40:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:40:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:58.018629+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]: dispatch 2024-02-10T08:40:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:40:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:59.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:40:59.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:40:59.668 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:40:59.917 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:40:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:40:59.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:00.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:59.025273+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:41:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:59.025319+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-10T08:41:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:59.043159+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:59.044264+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:40:59.044515+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:40:59.424057+0000 mgr.y (mgr.14553) 333 : cluster [DBG] pgmap v524: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:59.025273+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:59.025319+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:59.043159+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:59.044264+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:40:59.044515+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:00.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:40:59.424057+0000 mgr.y (mgr.14553) 333 : cluster [DBG] pgmap v524: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:59.025273+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.107:0/1557677608' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi107-112411-62"}]': finished 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:59.025319+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:59.043159+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:59.044264+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:00.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:40:59.044515+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:00.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:40:59.424057+0000 mgr.y (mgr.14553) 333 : cluster [DBG] pgmap v524: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.897+0000 7f3d4d752700 1 -- 172.21.15.107:0/2925444876 >> v1:172.21.15.107:6790/0 conn(0x7f3d48100810 legacy=0x7f3d48102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.897+0000 7f3d4d752700 1 -- 172.21.15.107:0/2925444876 shutdown_connections 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.897+0000 7f3d4d752700 1 -- 172.21.15.107:0/2925444876 >> 172.21.15.107:0/2925444876 conn(0x7f3d480f3630 msgr2=0x7f3d480f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.897+0000 7f3d4d752700 1 -- 172.21.15.107:0/2925444876 shutdown_connections 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.897+0000 7f3d4d752700 1 -- 172.21.15.107:0/2925444876 wait complete. 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 librados: starting msgr at 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 librados: starting objecter 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 Processor -- start 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 -- start start 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 librados: setting wanted keys 2024-02-10T08:41:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.898+0000 7f3d4d752700 1 librados: calling monclient init 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d4d752700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d48117240 con 0x7f3d48104320 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d4d752700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d48118420 con 0x7f3d48100810 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d4d752700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d48119620 con 0x7f3d480fda30 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d477fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3d48104320 0x7f3d481159c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42966/0 (socket says 172.21.15.107:42966) 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d477fe700 1 -- 172.21.15.107:0/1106629512 learned_addr learned my addr 172.21.15.107:0/1106629512 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter ms_handle_connect 0x7f3d48104320 2024-02-10T08:41:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter ms_handle_connect 0x7f3d480fda30 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter ms_handle_connect 0x7f3d48100810 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230910854 0 0) 0x7f3d48117240 con 0x7f3d48104320 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003030 con 0x7f3d48104320 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095193276 0 0) 0x7f3d48119620 con 0x7f3d480fda30 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.899+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48117240 con 0x7f3d480fda30 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454490958 0 0) 0x7f3d48118420 con 0x7f3d48100810 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48119620 con 0x7f3d48100810 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683942657 0 0) 0x7f3d34003030 con 0x7f3d48104320 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d48118420 con 0x7f3d48104320 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183492402 0 0) 0x7f3d48117240 con 0x7f3d480fda30 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d34003030 con 0x7f3d480fda30 2024-02-10T08:41:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3d40001c10 con 0x7f3d48104320 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3d300018d0 con 0x7f3d480fda30 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060912006 0 0) 0x7f3d48119620 con 0x7f3d48100810 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d48117240 con 0x7f3d48100810 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3d38001430 con 0x7f3d48100810 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2138768820 0 0) 0x7f3d34003030 con 0x7f3d480fda30 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 >> v1:172.21.15.195:6789/0 conn(0x7f3d48100810 legacy=0x7f3d48101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 >> v1:172.21.15.107:6789/0 conn(0x7f3d48104320 legacy=0x7f3d481159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4811a820 con 0x7f3d480fda30 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48119810 con 0x7f3d480fda30 2024-02-10T08:41:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:00.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d4d752700 10 client.53501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:00.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d48119c80 con 0x7f3d480fda30 2024-02-10T08:41:00.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d4d752700 1 librados: init done 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3d30002180 con 0x7f3d480fda30 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.900+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3d30003170 con 0x7f3d480fda30 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.901+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d28004f20 con 0x7f3d480fda30 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3d300032d0 con 0x7f3d480fda30 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(366..366 src has 275..366) v4 ==== 8444+0+0 (unknown 1125238811 0 0) 0x7f3d30076340 con 0x7f3d480fda30 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 10 client.53501.objecter ms_dispatch 0x7f3d48001550 osd_map(366..366 src has 275..366) v4 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 3 client.53501.objecter handle_osd_map got epochs [366,366] > 0 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 3 client.53501.objecter handle_osd_map decoding full epoch 366 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 20 client.53501.objecter dump_active .. 0 homeless 2024-02-10T08:41:00.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.902+0000 7f3d3ffff700 10 client.53501.objecter ms_handle_connect 0x7f3d34063e30 2024-02-10T08:41:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:00.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:00.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:00.907+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3d30001e70 con 0x7f3d480fda30 2024-02-10T08:41:01.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:01.038 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:01.061 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:01.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.101+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3d280051d0 con 0x7f3d480fda30 2024-02-10T08:41:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.102+0000 7f3d3ffff700 1 -- 172.21.15.107:0/1106629512 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v367) v1 ==== 81+0+49213 (unknown 1383236968 0 799825868) 0x7f3d30049160 con 0x7f3d480fda30 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 367, 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:41:00.996523+0000", 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:33:10.747466+0000", 2024-02-10T08:41:01.104 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:37:08.966515+0000", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 97, 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:41:01.105 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 202, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.106 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:41:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "67", 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.108 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.109 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.110 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.111 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.113 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.114 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.115 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.116 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.117 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.118 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.119 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.121 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.122 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.123 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.124 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.125 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.130 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.131 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:41:01.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.133 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.134 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.135 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 60, 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-6", 2024-02-10T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:49.742959+0000", 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "367", 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "367", 2024-02-10T08:41:01.144 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "367", 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "367", 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 21, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 354, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [ 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: 200 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:41:01.145 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.146 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 200, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-47", 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:40:45.263643+0000", 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8201, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:41:01.148 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.149 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "367", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "367", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "367", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "367", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 21, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 354, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": 60, 2024-02-10T08:41:01.150 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "readproxy", 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:41:01.151 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.155 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.157 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.165 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:41:01.166 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.168 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.169 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:41:01.170 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.173 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.174 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.175 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.176 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 354, 2024-02-10T08:41:01.177 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.179 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:41:01.180 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:41:01.181 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 361, 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:41:01.183 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:41:01.184 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.185 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:25.494086+0000", 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:56.083023+0000", 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:16.317482+0000", 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.187 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:27.388809+0000", 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.188 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:41:00.371652+0000", 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:07.356961+0000", 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:41:01.189 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:40:52.063753+0000", 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:41:01.190 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:39:37.860373+0000", 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.191 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.18", 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 7, 2024-02-10T08:41:01.192 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.0", 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.193 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.194 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1d", 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.f", 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 3 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.14", 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 7 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.1b", 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:41:01.197 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 7 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:41:01.198 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:41:01.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-ExecuteClassPP_smithi107-112411-63": { 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.201 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:41:01.202 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:41:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.207 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:41:01.208 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.210 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.211 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.212 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:01.213 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:41:01.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 10 librados: watch_flush enter 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 10 librados: watch_flush exit 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 20 client.53501.objecter shutdown clearing up homeless session... 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 10 client.53501.objecter successfully canceled tick 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3d34063e30 legacy=0x7f3d340662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.104+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 >> v1:172.21.15.107:6790/0 conn(0x7f3d480fda30 legacy=0x7f3d48100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.105+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 shutdown_connections 2024-02-10T08:41:01.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.105+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 >> 172.21.15.107:0/1106629512 conn(0x7f3d480f3630 msgr2=0x7f3d480f6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:01.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.105+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 shutdown_connections 2024-02-10T08:41:01.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.105+0000 7f3d4d752700 1 -- 172.21.15.107:0/1106629512 wait complete. 2024-02-10T08:41:01.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.105+0000 7f3d4d752700 1 librados: shutdown 2024-02-10T08:41:01.246 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:01.268 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:01.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:01.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:00.028957+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:01.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:00.029021+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-10T08:41:01.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:00.030334+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:01.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:00.235874+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:00.236345+0000 mon.a (mon.0) 2303 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:01.401 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:01.421 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:01.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:00.028957+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:00.029021+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-10T08:41:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:00.030334+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:00.235874+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:00.236345+0000 mon.a (mon.0) 2303 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:00.028957+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi107-112411-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:00.029021+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-10T08:41:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:00.030334+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:00.235874+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:00.236345+0000 mon.a (mon.0) 2303 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:01.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:01.554 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:01.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.566+0000 7f653fac2700 1 -- 172.21.15.107:0/2329973904 >> v1:172.21.15.107:6789/0 conn(0x7f6538100840 legacy=0x7f6538102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:01.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.566+0000 7f653fac2700 1 -- 172.21.15.107:0/2329973904 shutdown_connections 2024-02-10T08:41:01.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.566+0000 7f653fac2700 1 -- 172.21.15.107:0/2329973904 >> 172.21.15.107:0/2329973904 conn(0x7f653806c090 msgr2=0x7f653806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.566+0000 7f653fac2700 1 -- 172.21.15.107:0/2329973904 shutdown_connections 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 -- 172.21.15.107:0/2329973904 wait complete. 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 librados: starting msgr at 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 librados: starting objecter 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 Processor -- start 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 -- start start 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 librados: setting wanted keys 2024-02-10T08:41:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 librados: calling monclient init 2024-02-10T08:41:01.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65380fd0c0 con 0x7f6538104350 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65380fd290 con 0x7f65380fd9f0 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653fac2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65380fd480 con 0x7f6538100840 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653d05d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6538100840 0x7f6538114da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50340/0 (socket says 172.21.15.107:50340) 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653d05d700 1 -- 172.21.15.107:0/1845686839 learned_addr learned my addr 172.21.15.107:0/1845686839 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f653e05f700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6538104350 0x7f65381184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42984/0 (socket says 172.21.15.107:42984) 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f652ef78700 10 client.?.objecter ms_handle_connect 0x7f6538100840 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.567+0000 7f652ef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 10 client.?.objecter ms_handle_connect 0x7f6538104350 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 10 client.?.objecter ms_handle_connect 0x7f65380fd9f0 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183286430 0 0) 0x7f65380fd0c0 con 0x7f6538104350 2024-02-10T08:41:01.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65240033f0 con 0x7f6538104350 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325353522 0 0) 0x7f65380fd290 con 0x7f65380fd9f0 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65380fd0c0 con 0x7f65380fd9f0 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984820671 0 0) 0x7f65380fd480 con 0x7f6538100840 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65380fd290 con 0x7f6538100840 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1461493772 0 0) 0x7f65240033f0 con 0x7f6538104350 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65380fd480 con 0x7f6538104350 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6530001c40 con 0x7f6538104350 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286695373 0 0) 0x7f65380fd0c0 con 0x7f65380fd9f0 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65240033f0 con 0x7f65380fd9f0 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f65340018f0 con 0x7f65380fd9f0 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830191028 0 0) 0x7f65380fd290 con 0x7f6538100840 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65380fd0c0 con 0x7f6538100840 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6528001f10 con 0x7f6538100840 2024-02-10T08:41:01.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4137197226 0 0) 0x7f65380fd480 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 >> v1:172.21.15.107:6790/0 conn(0x7f6538100840 legacy=0x7f6538114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 >> v1:172.21.15.195:6789/0 conn(0x7f65380fd9f0 legacy=0x7f65380fc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653811cc20 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6538119c50 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f653fac2700 10 client.27921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f653811a0c0 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f653fac2700 1 librados: init done 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6530002380 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.568+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6530005e60 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.569+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c004f20 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6530001ee0 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(367..367 src has 275..367) v4 ==== 8554+0+0 (unknown 3850579649 0 0) 0x7f6530079100 con 0x7f6538104350 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 10 client.27921.objecter ms_dispatch 0x7f6538001550 osd_map(367..367 src has 275..367) v4 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 3 client.27921.objecter handle_osd_map got epochs [367,367] > 0 2024-02-10T08:41:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 3 client.27921.objecter handle_osd_map decoding full epoch 367 2024-02-10T08:41:01.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 20 client.27921.objecter dump_active .. 0 homeless 2024-02-10T08:41:01.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.570+0000 7f652ef78700 10 client.27921.objecter ms_handle_connect 0x7f6524063c60 2024-02-10T08:41:01.574 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:01.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.574+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f653004c300 con 0x7f6538104350 2024-02-10T08:41:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:01.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:01.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:01.708 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:01.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:01.761+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f651c0051d0 con 0x7f6538104350 2024-02-10T08:41:01.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.004+0000 7f652ef78700 1 -- 172.21.15.107:0/1845686839 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v368) v1 ==== 91+0+0 (unknown 3655423246 0 0) 0x7f653004a710 con 0x7f6538104350 2024-02-10T08:41:02.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 10 librados: watch_flush enter 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 10 librados: watch_flush exit 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 20 client.27921.objecter shutdown clearing up homeless session... 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 10 client.27921.objecter successfully canceled tick 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6524063c60 legacy=0x7f6524066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 >> v1:172.21.15.107:6789/0 conn(0x7f6538104350 legacy=0x7f65381184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 shutdown_connections 2024-02-10T08:41:02.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.008+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 >> 172.21.15.107:0/1845686839 conn(0x7f653806c090 msgr2=0x7f6538106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:02.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.009+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 shutdown_connections 2024-02-10T08:41:02.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.009+0000 7f653fac2700 1 -- 172.21.15.107:0/1845686839 wait complete. 2024-02-10T08:41:02.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:02.011+0000 7f653fac2700 1 librados: shutdown 2024-02-10T08:41:02.012 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:41:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:02.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:02.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:02.194 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:00.371444+0000 osd.4 (osd.4) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:00.371648+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.102549+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/1106629512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.252505+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:01.252575+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.253593+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.256638+0000 mon.a (mon.0) 2306 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.400956+0000 mgr.y (mgr.14553) 334 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:01.424940+0000 mgr.y (mgr.14553) 335 : cluster [DBG] pgmap v527: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:01.762202+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:01.998878+0000 mon.a (mon.0) 2308 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:01.998905+0000 mon.a (mon.0) 2309 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.004480+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.004570+0000 mon.a (mon.0) 2311 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.004634+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:02.004687+0000 mon.a (mon.0) 2313 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-10T08:41:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.025774+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.026084+0000 mon.a (mon.0) 2314 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.026603+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:02.026796+0000 mon.a (mon.0) 2315 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:00.371444+0000 osd.4 (osd.4) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:00.371648+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.102549+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/1106629512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.252505+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:01.252575+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.253593+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.256638+0000 mon.a (mon.0) 2306 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.400956+0000 mgr.y (mgr.14553) 334 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:01.424940+0000 mgr.y (mgr.14553) 335 : cluster [DBG] pgmap v527: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:01.762202+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:01.998878+0000 mon.a (mon.0) 2308 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:01.998905+0000 mon.a (mon.0) 2309 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.004480+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.004570+0000 mon.a (mon.0) 2311 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.004634+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:02.004687+0000 mon.a (mon.0) 2313 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.025774+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.026084+0000 mon.a (mon.0) 2314 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.026603+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.297 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:02.026796+0000 mon.a (mon.0) 2315 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:02.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:02.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:00.371444+0000 osd.4 (osd.4) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:00.371648+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.102549+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.107:0/1106629512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.252505+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:01.252575+0000 mon.a (mon.0) 2305 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-10T08:41:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.253593+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.256638+0000 mon.a (mon.0) 2306 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.400956+0000 mgr.y (mgr.14553) 334 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:01.424940+0000 mgr.y (mgr.14553) 335 : cluster [DBG] pgmap v527: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:01.762202+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:01.998878+0000 mon.a (mon.0) 2308 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:01.998905+0000 mon.a (mon.0) 2309 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.004480+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi107-112411-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.004570+0000 mon.a (mon.0) 2311 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]': finished 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.004634+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? v1:172.21.15.107:0/1845686839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:02.004687+0000 mon.a (mon.0) 2313 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.025774+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.026084+0000 mon.a (mon.0) 2314 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:02.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.026603+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:02.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:02.026796+0000 mon.a (mon.0) 2315 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-47"}]: dispatch 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: : checking for 190:7462ddf6:::163:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:5d165639:::164:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f43765fc:::165:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b4c720e9:::166:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e694b040:::167:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:afa38db2:::168:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:77ba9f53:::169:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:87495034:::170:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7c96bf0e:::171:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:dbe346cc:::172:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:e943ec24:::173:head 2024-02-10T08:41:03.009 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f97a9c0c:::174:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:6f26e74d:::175:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4f95e106:::176:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:0e6f2f8f:::177:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:05db05f1:::178:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:38a78d66:::179:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:d095610b:::180:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:a1a9d709:::181:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:1e5d39db:::182:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:f7df4fb9:::183:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:03a7f161:::184:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ba70721e:::185:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:28e5662d:::186:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:973d52de:::187:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:4303eb1c:::188:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:b990b48e:::189:head 2024-02-10T08:41:03.010 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:29b8165b:::190:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:3547f197:::191:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:7e260936:::192:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:1abec7b1:::193:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:10fdda93:::194:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:15817eea:::195:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:770bab57:::196:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:ed9e13e7:::197:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:71471a8f:::198:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: checking for 190:10fb1d02:::199:head 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8292 ms) 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,0 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: first is 1707554417 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,0 2024-02-10T08:41:03.011 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,1707554425,1707554426,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,1707554425,1707554426,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554417,1707554419,1707554420,1707554422,1707554423,1707554425,1707554426,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554420,1707554422,1707554423,1707554425,1707554426,1707554428,1707554429,0 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: first now 1707554420, trimmed 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21062 ms) 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-10T08:41:03.012 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: foo0 2024-02-10T08:41:03.013 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-10T08:41:03.013 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14680 ms) 2024-02-10T08:41:03.013 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-10T08:41:03.257 INFO:tasks.thrashosds.thrasher:j is {'epoch': 367, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:41:00.996523+0000', 'last_up_change': '2024-02-10T08:33:10.747466+0000', 'last_in_change': '2024-02-10T08:37:08.966515+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 97, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 202, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 60, 'pool_name': 'test-rados-api-smithi107-112490-6', 'create_time': '2024-02-10T08:36:49.742959+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '367', 'last_force_op_resend': '367', 'last_force_op_resend_prenautilus': '367', 'last_force_op_resend_preluminous': '367', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 354, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [200], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 200, 'pool_name': 'test-rados-api-smithi107-112490-47', 'create_time': '2024-02-10T08:40:45.263643+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '367', 'last_force_op_resend': '367', 'last_force_op_resend_prenautilus': '367', 'last_force_op_resend_preluminous': '367', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 354, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 60, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'readproxy', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:40:25.494086+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:56.083023+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:16.317482+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:27.388809+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:41:00.371652+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:07.356961+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:40:52.063753+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:39:37.860373+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '4.1', 'osds': [4, 0, 7]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.0', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '16.f', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '16.14', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '16.1b', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-ExecuteClassPP_smithi107-112411-63': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:41:03.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:41:03.269 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:03.275 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:03.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:03.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:03.410 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:03.558 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:03.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:03.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:03.696 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:03.715 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:03.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:03.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:03.848 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:03.867 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:03.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:03.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:03.999 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:04.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:04.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:41:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:04.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:02.857417+0000 osd.5 (osd.5) 159 : cluster [DBG] 16.f scrub starts 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:02.860768+0000 osd.5 (osd.5) 160 : cluster [DBG] 16.f scrub ok 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.005019+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.235135+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.238031+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:41:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.425277+0000 mgr.y (mgr.14553) 336 : cluster [DBG] pgmap v530: 194 pgs: 8 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:04.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:02.857417+0000 osd.5 (osd.5) 159 : cluster [DBG] 16.f scrub starts 2024-02-10T08:41:04.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:02.860768+0000 osd.5 (osd.5) 160 : cluster [DBG] 16.f scrub ok 2024-02-10T08:41:04.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.005019+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-10T08:41:04.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.235135+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:41:04.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.238031+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.425277+0000 mgr.y (mgr.14553) 336 : cluster [DBG] pgmap v530: 194 pgs: 8 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:02.857417+0000 osd.5 (osd.5) 159 : cluster [DBG] 16.f scrub starts 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:02.860768+0000 osd.5 (osd.5) 160 : cluster [DBG] 16.f scrub ok 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.005019+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.235135+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.238031+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:41:04.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.425277+0000 mgr.y (mgr.14553) 336 : cluster [DBG] pgmap v530: 194 pgs: 8 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:04.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:04.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:04.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:04.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:41:05.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:05.075 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:05.095 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:05.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:05.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:05.227 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:05.247 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:05.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.328669+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.331543+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.580766+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.584568+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.768836+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:03.770705+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:04.015822+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:04.023537+0000 mon.b (mon.1) 545 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:04.023660+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:05.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:04.024056+0000 mon.a (mon.0) 2319 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:05.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.328669+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:41:05.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.331543+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.580766+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.584568+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.768836+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:03.770705+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:04.015822+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:04.023537+0000 mon.b (mon.1) 545 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:04.023660+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:04.024056+0000 mon.a (mon.0) 2319 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.328669+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.331543+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.580766+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:41:05.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.584568+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.768836+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:03.770705+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:04.015822+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:04.023537+0000 mon.b (mon.1) 545 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:04.023660+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:05.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:04.024056+0000 mon.a (mon.0) 2319 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:05.379 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:05.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:05.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:05.860 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:05.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:05] "GET /metrics HTTP/1.1" 200 189370 "" "Prometheus/2.33.4" 2024-02-10T08:41:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:06.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:06.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:04.368680+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:04.369529+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:04.462359+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:04.462565+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:05.014408+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:05.014488+0000 mon.a (mon.0) 2321 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.014565+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:05.020491+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:06.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:05.041411+0000 mon.b (mon.1) 546 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:05.041757+0000 mon.a (mon.0) 2324 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.425803+0000 mgr.y (mgr.14553) 337 : cluster [DBG] pgmap v533: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.016258+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.016325+0000 mon.a (mon.0) 2326 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:06.016384+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.017948+0000 mon.b (mon.1) 547 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.020975+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:06.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:06.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:04.368680+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:04.369529+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:04.462359+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:04.462565+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:05.014408+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:05.014488+0000 mon.a (mon.0) 2321 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.014565+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:05.020491+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:05.041411+0000 mon.b (mon.1) 546 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:05.041757+0000 mon.a (mon.0) 2324 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.425803+0000 mgr.y (mgr.14553) 337 : cluster [DBG] pgmap v533: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.016258+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.016325+0000 mon.a (mon.0) 2326 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:41:06.342 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:06.016384+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-10T08:41:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.017948+0000 mon.b (mon.1) 547 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.020975+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:04.368680+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:04.369529+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:04.462359+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:04.462565+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:05.014408+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:05.014488+0000 mon.a (mon.0) 2321 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.014565+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:05.020491+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]: dispatch 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:05.041411+0000 mon.b (mon.1) 546 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:05.041757+0000 mon.a (mon.0) 2324 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.425803+0000 mgr.y (mgr.14553) 337 : cluster [DBG] pgmap v533: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:06.473 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.016258+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? v1:172.21.15.107:0/901028778' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi107-112411-63"}]': finished 2024-02-10T08:41:06.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.016325+0000 mon.a (mon.0) 2326 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:41:06.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:06.016384+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-10T08:41:06.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.017948+0000 mon.b (mon.1) 547 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.020975+0000 mon.a (mon.0) 2328 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:06.474 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:06.496 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:06.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:06.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:06.629 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:06.776 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:06.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:06.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:06.915 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:06.933 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:06.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:07.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:07.067 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.086 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 -- 172.21.15.107:0/737800435 >> v1:172.21.15.107:6790/0 conn(0x7fc1a0103250 legacy=0x7fc1a01056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 -- 172.21.15.107:0/737800435 shutdown_connections 2024-02-10T08:41:07.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 -- 172.21.15.107:0/737800435 >> 172.21.15.107:0/737800435 conn(0x7fc1a00f2510 msgr2=0x7fc1a00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 -- 172.21.15.107:0/737800435 shutdown_connections 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 -- 172.21.15.107:0/737800435 wait complete. 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 librados: starting msgr at 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.994+0000 7fc1a4f85700 1 librados: starting objecter 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 Processor -- start 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 -- start start 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 librados: setting wanted keys 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 librados: calling monclient init 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1a0116370 con 0x7fc1a0103250 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1a0117570 con 0x7fc1a00ff710 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc1a4f85700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1a0118770 con 0x7fc1a00fc910 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc19ed9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1a0103250 0x7fc1a0114a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43020/0 (socket says 172.21.15.107:43020) 2024-02-10T08:41:07.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc19e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc1a00fc910 0x7fc1a00f8ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50384/0 (socket says 172.21.15.107:50384) 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc19ed9d700 1 -- 172.21.15.107:0/1247562334 learned_addr learned my addr 172.21.15.107:0/1247562334 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter ms_handle_connect 0x7fc1a0103250 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter ms_handle_connect 0x7fc1a00fc910 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter ms_handle_connect 0x7fc1a00ff710 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138266020 0 0) 0x7fc1a0116370 con 0x7fc1a0103250 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc188003030 con 0x7fc1a0103250 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018932157 0 0) 0x7fc1a0118770 con 0x7fc1a00fc910 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0116370 con 0x7fc1a00fc910 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164152471 0 0) 0x7fc1a0117570 con 0x7fc1a00ff710 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.995+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0118770 con 0x7fc1a00ff710 2024-02-10T08:41:07.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075912024 0 0) 0x7fc188003030 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a0117570 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc198001950 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2342018989 0 0) 0x7fc1a0118770 con 0x7fc1a00ff710 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc188003030 con 0x7fc1a00ff710 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc1900018c0 con 0x7fc1a00ff710 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2139692068 0 0) 0x7fc1a0116370 con 0x7fc1a00fc910 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a0118770 con 0x7fc1a00fc910 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc194001b00 con 0x7fc1a00fc910 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 393107313 0 0) 0x7fc1a0117570 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 >> v1:172.21.15.107:6790/0 conn(0x7fc1a00fc910 legacy=0x7fc1a00f8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 >> v1:172.21.15.195:6789/0 conn(0x7fc1a00ff710 legacy=0x7fc1a00f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a0119970 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a0118960 con 0x7fc1a0103250 2024-02-10T08:41:07.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc1a4f85700 10 client.27933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a0118dd0 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc1a4f85700 1 librados: init done 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc198001e10 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.996+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc198003b60 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.997+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc180004f20 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.997+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc198003d60 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(372..372 src has 275..372) v4 ==== 8153+0+0 (unknown 3134385342 0 0) 0x7fc198079300 con 0x7fc1a0103250 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 10 client.27933.objecter ms_dispatch 0x7fc1a0001550 osd_map(372..372 src has 275..372) v4 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 3 client.27933.objecter handle_osd_map got epochs [372,372] > 0 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 3 client.27933.objecter handle_osd_map decoding full epoch 372 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 20 client.27933.objecter dump_active .. 0 homeless 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:06.998+0000 7fc18f7fe700 10 client.27933.objecter ms_handle_connect 0x7fc188063d20 2024-02-10T08:41:07.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.002+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc19804c7b0 con 0x7fc1a0103250 2024-02-10T08:41:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.171+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc180000b90 con 0x7fc188063d20 2024-02-10T08:41:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.186+0000 7fc18f7fe700 1 -- 172.21.15.107:0/1247562334 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553840 (unknown 2965378022 0 2129996067) 0x7fc180000b90 con 0x7fc188063d20 2024-02-10T08:41:07.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:07.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:41:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 10 librados: watch_flush enter 2024-02-10T08:41:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 10 librados: watch_flush exit 2024-02-10T08:41:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 20 client.27933.objecter shutdown clearing up homeless session... 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 10 client.27933.objecter successfully canceled tick 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc188063d20 legacy=0x7fc1880661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 >> v1:172.21.15.107:6789/0 conn(0x7fc1a0103250 legacy=0x7fc1a0114a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 shutdown_connections 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 >> 172.21.15.107:0/1247562334 conn(0x7fc1a00f2510 msgr2=0x7fc1a01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 shutdown_connections 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 -- 172.21.15.107:0/1247562334 wait complete. 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:07.190+0000 7fc1a4f85700 1 librados: shutdown 2024-02-10T08:41:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.556476+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.d scrub starts 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.559389+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.d scrub ok 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.680639+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.682572+0000 osd.4 (osd.4) 163 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.906459+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:05.909355+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.043501+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.044612+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:06.044888+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.017541+0000 mon.a (mon.0) 2332 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.017629+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.017680+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.022529+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.022620+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.022957+0000 mon.a (mon.0) 2336 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.231 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.251 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:07.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:07.381 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.400 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.556476+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.d scrub starts 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.559389+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.d scrub ok 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.680639+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.682572+0000 osd.4 (osd.4) 163 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.906459+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:41:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:05.909355+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.043501+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.044612+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:06.044888+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.017541+0000 mon.a (mon.0) 2332 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.017629+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.017680+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.022529+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.022620+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.022957+0000 mon.a (mon.0) 2336 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.556476+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.d scrub starts 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.559389+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.d scrub ok 2024-02-10T08:41:07.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.680639+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.682572+0000 osd.4 (osd.4) 163 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.906459+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:05.909355+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.043501+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.044612+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:06.044888+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.017541+0000 mon.a (mon.0) 2332 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.017629+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi107-112411-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.017680+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.022529+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.022620+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.022957+0000 mon.a (mon.0) 2336 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]: dispatch 2024-02-10T08:41:07.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:07.531 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.550 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:07.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:07.681 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.699 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:07.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:07.829 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.847 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:07.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:07.977 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:07.994 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:07.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:08.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:08.125 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:08.143 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:08.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:08.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:08.250 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-02-10T08:41:07.016266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007221,"num_objects":341,"num_object_clones":2,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":1286,"num_read_kb":943,"num_write":1008,"num_write_kb":4522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2327,"ondisk_log_size":2327,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":637,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2163504,"kb_used_data":29688,"kb_used_omap":0,"kb_used_meta":2133760,"kb_avail":654052560,"statfs":{"total":671965249536,"available":669749821440,"internally_reserved":0,"allocated":30400512,"data_stored":19096563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184970240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013768"},"pg_stats":[{"pgid":"204.1f","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.030687+0000","last_change":"2024-02-10T08:41:05.028253+0000","last_active":"2024-02-10T08:41:05.030687+0000","last_peered":"2024-02-10T08:41:05.030687+0000","last_clean":"2024-02-10T08:41:05.030687+0000","last_became_active":"2024-02-10T08:41:05.027950+0000","last_became_peered":"2024-02-10T08:41:05.027950+0000","last_unstale":"2024-02-10T08:41:05.030687+0000","last_undegraded":"2024-02-10T08:41:05.030687+0000","last_fullsized":"2024-02-10T08:41:05.030687+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.1e","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029554+0000","last_change":"2024-02-10T08:41:05.029554+0000","last_active":"2024-02-10T08:41:05.029554+0000","last_peered":"2024-02-10T08:41:05.029554+0000","last_clean":"2024-02-10T08:41:05.029554+0000","last_became_active":"2024-02-10T08:41:05.029302+0000","last_became_peered":"2024-02-10T08:41:05.029302+0000","last_unstale":"2024-02-10T08:41:05.029554+0000","last_undegraded":"2024-02-10T08:41:05.029554+0000","last_fullsized":"2024-02-10T08:41:05.029554+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"204.1d","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029141+0000","last_change":"2024-02-10T08:41:05.029141+0000","last_active":"2024-02-10T08:41:05.029141+0000","last_peered":"2024-02-10T08:41:05.029141+0000","last_clean":"2024-02-10T08:41:05.029141+0000","last_became_active":"2024-02-10T08:41:05.028362+0000","last_became_peered":"2024-02-10T08:41:05.028362+0000","last_unstale":"2024-02-10T08:41:05.029141+0000","last_undegraded":"2024-02-10T08:41:05.029141+0000","last_fullsized":"2024-02-10T08:41:05.029141+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.1c","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.033566+0000","last_change":"2024-02-10T08:41:05.033566+0000","last_active":"2024-02-10T08:41:05.033566+0000","last_peered":"2024-02-10T08:41:05.033566+0000","last_clean":"2024-02-10T08:41:05.033566+0000","last_became_active":"2024-02-10T08:41:05.032606+0000","last_became_peered":"2024-02-10T08:41:05.032606+0000","last_unstale":"2024-02-10T08:41:05.033566+0000","last_undegraded":"2024-02-10T08:41:05.033566+0000","last_fullsized":"2024-02-10T08:41:05.033566+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"204.1b","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.037894+0000","last_change":"2024-02-10T08:41:05.036765+0000","last_active":"2024-02-10T08:41:05.037894+0000","last_peered":"2024-02-10T08:41:05.037894+0000","last_clean":"2024-02-10T08:41:05.037894+0000","last_became_active":"2024-02-10T08:41:05.036424+0000","last_became_peered":"2024-02-10T08:41:05.036424+0000","last_unstale":"2024-02-10T08:41:05.037894+0000","last_undegraded":"2024-02-10T08:41:05.037894+0000","last_fullsized":"2024-02-10T08:41:05.037894+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.1a","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.036753+0000","last_change":"2024-02-10T08:41:05.035884+0000","last_active":"2024-02-10T08:41:05.036753+0000","last_peered":"2024-02-10T08:41:05.036753+0000","last_clean":"2024-02-10T08:41:05.036753+0000","last_became_active":"2024-02-10T08:41:05.035591+0000","last_became_peered":"2024-02-10T08:41:05.035591+0000","last_unstale":"2024-02-10T08:41:05.036753+0000","last_undegraded":"2024-02-10T08:41:05.036753+0000","last_fullsized":"2024-02-10T08:41:05.036753+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"204.19","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.032534+0000","last_change":"2024-02-10T08:41:05.032534+0000","last_active":"2024-02-10T08:41:05.032534+0000","last_peered":"2024-02-10T08:41:05.032534+0000","last_clean":"2024-02-10T08:41:05.032534+0000","last_became_active":"2024-02-10T08:41:05.032228+0000","last_became_peered":"2024-02-10T08:41:05.032228+0000","last_unstale":"2024-02-10T08:41:05.032534+0000","last_undegraded":"2024-02-10T08:41:05.032534+0000","last_fullsized":"2024-02-10T08:41:05.032534+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"204.18","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.027517+0000","last_change":"2024-02-10T08:41:05.027517+0000","last_active":"2024-02-10T08:41:05.027517+0000","last_peered":"2024-02-10T08:41:05.027517+0000","last_clean":"2024-02-10T08:41:05.027517+0000","last_became_active":"2024-02-10T08:41:05.027157+0000","last_became_peered":"2024-02-10T08:41:05.027157+0000","last_unstale":"2024-02-10T08:41:05.027517+0000","last_undegraded":"2024-02-10T08:41:05.027517+0000","last_fullsized":"2024-02-10T08:41:05.027517+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.17","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.035521+0000","last_change":"2024-02-10T08:41:05.034319+0000","last_active":"2024-02-10T08:41:05.035521+0000","last_peered":"2024-02-10T08:41:05.035521+0000","last_clean":"2024-02-10T08:41:05.035521+0000","last_became_active":"2024-02-10T08:41:05.034041+0000","last_became_peered":"2024-02-10T08:41:05.034041+0000","last_unstale":"2024-02-10T08:41:05.035521+0000","last_undegraded":"2024-02-10T08:41:05.035521+0000","last_fullsized":"2024-02-10T08:41:05.035521+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"204.16","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029598+0000","last_change":"2024-02-10T08:41:05.029598+0000","last_active":"2024-02-10T08:41:05.029598+0000","last_peered":"2024-02-10T08:41:05.029598+0000","last_clean":"2024-02-10T08:41:05.029598+0000","last_became_active":"2024-02-10T08:41:05.029319+0000","last_became_peered":"2024-02-10T08:41:05.029319+0000","last_unstale":"2024-02-10T08:41:05.029598+0000","last_undegraded":"2024-02-10T08:41:05.029598+0000","last_fullsized":"2024-02-10T08:41:05.029598+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"204.15","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.036414+0000","last_change":"2024-02-10T08:41:05.034880+0000","last_active":"2024-02-10T08:41:05.036414+0000","last_peered":"2024-02-10T08:41:05.036414+0000","last_clean":"2024-02-10T08:41:05.036414+0000","last_became_active":"2024-02-10T08:41:05.034564+0000","last_became_peered":"2024-02-10T08:41:05.034564+0000","last_unstale":"2024-02-10T08:41:05.036414+0000","last_undegraded":"2024-02-10T08:41:05.036414+0000","last_fullsized":"2024-02-10T08:41:05.036414+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.14","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.037785+0000","last_change":"2024-02-10T08:41:05.036965+0000","last_active":"2024-02-10T08:41:05.037785+0000","last_peered":"2024-02-10T08:41:05.037785+0000","last_clean":"2024-02-10T08:41:05.037785+0000","last_became_active":"2024-02-10T08:41:05.036650+0000","last_became_peered":"2024-02-10T08:41:05.036650+0000","last_unstale":"2024-02-10T08:41:05.037785+0000","last_undegraded":"2024-02-10T08:41:05.037785+0000","last_fullsized":"2024-02-10T08:41:05.037785+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.13","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.033499+0000","last_change":"2024-02-10T08:41:05.032502+0000","last_active":"2024-02-10T08:41:05.033499+0000","last_peered":"2024-02-10T08:41:05.033499+0000","last_clean":"2024-02-10T08:41:05.033499+0000","last_became_active":"2024-02-10T08:41:05.032206+0000","last_became_peered":"2024-02-10T08:41:05.032206+0000","last_unstale":"2024-02-10T08:41:05.033499+0000","last_undegraded":"2024-02-10T08:41:05.033499+0000","last_fullsized":"2024-02-10T08:41:05.033499+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.12","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.025376+0000","last_change":"2024-02-10T08:41:05.025019+0000","last_active":"2024-02-10T08:41:05.025376+0000","last_peered":"2024-02-10T08:41:05.025376+0000","last_clean":"2024-02-10T08:41:05.025376+0000","last_became_active":"2024-02-10T08:41:05.024730+0000","last_became_peered":"2024-02-10T08:41:05.024730+0000","last_unstale":"2024-02-10T08:41:05.025376+0000","last_undegraded":"2024-02-10T08:41:05.025376+0000","last_fullsized":"2024-02-10T08:41:05.025376+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.11","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029622+0000","last_change":"2024-02-10T08:41:05.029622+0000","last_active":"2024-02-10T08:41:05.029622+0000","last_peered":"2024-02-10T08:41:05.029622+0000","last_clean":"2024-02-10T08:41:05.029622+0000","last_became_active":"2024-02-10T08:41:05.029176+0000","last_became_peered":"2024-02-10T08:41:05.029176+0000","last_unstale":"2024-02-10T08:41:05.029622+0000","last_undegraded":"2024-02-10T08:41:05.029622+0000","last_fullsized":"2024-02-10T08:41:05.029622+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"204.10","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.032694+0000","last_change":"2024-02-10T08:41:05.032694+0000","last_active":"2024-02-10T08:41:05.032694+0000","last_peered":"2024-02-10T08:41:05.032694+0000","last_clean":"2024-02-10T08:41:05.032694+0000","last_became_active":"2024-02-10T08:41:05.031957+0000","last_became_peered":"2024-02-10T08:41:05.031957+0000","last_unstale":"2024-02-10T08:41:05.032694+0000","last_undegraded":"2024-02-10T08:41:05.032694+0000","last_fullsized":"2024-02-10T08:41:05.032694+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"204.f","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.035392+0000","last_change":"2024-02-10T08:41:05.035392+0000","last_active":"2024-02-10T08:41:05.035392+0000","last_peered":"2024-02-10T08:41:05.035392+0000","last_clean":"2024-02-10T08:41:05.035392+0000","last_became_active":"2024-02-10T08:41:05.035088+0000","last_became_peered":"2024-02-10T08:41:05.035088+0000","last_unstale":"2024-02-10T08:41:05.035392+0000","last_undegraded":"2024-02-10T08:41:05.035392+0000","last_fullsized":"2024-02-10T08:41:05.035392+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.e","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.030701+0000","last_change":"2024-02-10T08:41:05.028740+0000","last_active":"2024-02-10T08:41:05.030701+0000","last_peered":"2024-02-10T08:41:05.030701+0000","last_clean":"2024-02-10T08:41:05.030701+0000","last_became_active":"2024-02-10T08:41:05.028497+0000","last_became_peered":"2024-02-10T08:41:05.028497+0000","last_unstale":"2024-02-10T08:41:05.030701+0000","last_undegraded":"2024-02-10T08:41:05.030701+0000","last_fullsized":"2024-02-10T08:41:05.030701+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.d","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.035137+0000","last_change":"2024-02-10T08:41:05.034766+0000","last_active":"2024-02-10T08:41:05.035137+0000","last_peered":"2024-02-10T08:41:05.035137+0000","last_clean":"2024-02-10T08:41:05.035137+0000","last_became_active":"2024-02-10T08:41:05.034457+0000","last_became_peered":"2024-02-10T08:41:05.034457+0000","last_unstale":"2024-02-10T08:41:05.035137+0000","last_undegraded":"2024-02-10T08:41:05.035137+0000","last_fullsized":"2024-02-10T08:41:05.035137+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.c","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.037806+0000","last_change":"2024-02-10T08:41:05.037323+0000","last_active":"2024-02-10T08:41:05.037806+0000","last_peered":"2024-02-10T08:41:05.037806+0000","last_clean":"2024-02-10T08:41:05.037806+0000","last_became_active":"2024-02-10T08:41:05.037015+0000","last_became_peered":"2024-02-10T08:41:05.037015+0000","last_unstale":"2024-02-10T08:41:05.037806+0000","last_undegraded":"2024-02-10T08:41:05.037806+0000","last_fullsized":"2024-02-10T08:41:05.037806+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.b","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.035720+0000","last_change":"2024-02-10T08:41:05.035158+0000","last_active":"2024-02-10T08:41:05.035720+0000","last_peered":"2024-02-10T08:41:05.035720+0000","last_clean":"2024-02-10T08:41:05.035720+0000","last_became_active":"2024-02-10T08:41:05.034956+0000","last_became_peered":"2024-02-10T08:41:05.034956+0000","last_unstale":"2024-02-10T08:41:05.035720+0000","last_undegraded":"2024-02-10T08:41:05.035720+0000","last_fullsized":"2024-02-10T08:41:05.035720+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"204.a","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.028547+0000","last_change":"2024-02-10T08:41:05.027562+0000","last_active":"2024-02-10T08:41:05.028547+0000","last_peered":"2024-02-10T08:41:05.028547+0000","last_clean":"2024-02-10T08:41:05.028547+0000","last_became_active":"2024-02-10T08:41:05.027252+0000","last_became_peered":"2024-02-10T08:41:05.027252+0000","last_unstale":"2024-02-10T08:41:05.028547+0000","last_undegraded":"2024-02-10T08:41:05.028547+0000","last_fullsized":"2024-02-10T08:41:05.028547+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.9","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.036246+0000","last_change":"2024-02-10T08:41:05.034461+0000","last_active":"2024-02-10T08:41:05.036246+0000","last_peered":"2024-02-10T08:41:05.036246+0000","last_clean":"2024-02-10T08:41:05.036246+0000","last_became_active":"2024-02-10T08:41:05.034170+0000","last_became_peered":"2024-02-10T08:41:05.034170+0000","last_unstale":"2024-02-10T08:41:05.036246+0000","last_undegraded":"2024-02-10T08:41:05.036246+0000","last_fullsized":"2024-02-10T08:41:05.036246+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.8","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.032738+0000","last_change":"2024-02-10T08:41:05.032738+0000","last_active":"2024-02-10T08:41:05.032738+0000","last_peered":"2024-02-10T08:41:05.032738+0000","last_clean":"2024-02-10T08:41:05.032738+0000","last_became_active":"2024-02-10T08:41:05.032402+0000","last_became_peered":"2024-02-10T08:41:05.032402+0000","last_unstale":"2024-02-10T08:41:05.032738+0000","last_undegraded":"2024-02-10T08:41:05.032738+0000","last_fullsized":"2024-02-10T08:41:05.032738+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"204.3","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.033129+0000","last_change":"2024-02-10T08:41:05.032272+0000","last_active":"2024-02-10T08:41:05.033129+0000","last_peered":"2024-02-10T08:41:05.033129+0000","last_clean":"2024-02-10T08:41:05.033129+0000","last_became_active":"2024-02-10T08:41:05.031934+0000","last_became_peered":"2024-02-10T08:41:05.031934+0000","last_unstale":"2024-02-10T08:41:05.033129+0000","last_undegraded":"2024-02-10T08:41:05.033129+0000","last_fullsized":"2024-02-10T08:41:05.033129+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.2","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.035785+0000","last_change":"2024-02-10T08:41:05.035785+0000","last_active":"2024-02-10T08:41:05.035785+0000","last_peered":"2024-02-10T08:41:05.035785+0000","last_clean":"2024-02-10T08:41:05.035785+0000","last_became_active":"2024-02-10T08:41:05.035434+0000","last_became_peered":"2024-02-10T08:41:05.035434+0000","last_unstale":"2024-02-10T08:41:05.035785+0000","last_undegraded":"2024-02-10T08:41:05.035785+0000","last_fullsized":"2024-02-10T08:41:05.035785+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.1","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.032286+0000","last_change":"2024-02-10T08:41:05.032286+0000","last_active":"2024-02-10T08:41:05.032286+0000","last_peered":"2024-02-10T08:41:05.032286+0000","last_clean":"2024-02-10T08:41:05.032286+0000","last_became_active":"2024-02-10T08:41:05.032000+0000","last_became_peered":"2024-02-10T08:41:05.032000+0000","last_unstale":"2024-02-10T08:41:05.032286+0000","last_undegraded":"2024-02-10T08:41:05.032286+0000","last_fullsized":"2024-02-10T08:41:05.032286+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"204.0","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.033921+0000","last_change":"2024-02-10T08:41:05.033424+0000","last_active":"2024-02-10T08:41:05.033921+0000","last_peered":"2024-02-10T08:41:05.033921+0000","last_clean":"2024-02-10T08:41:05.033921+0000","last_became_active":"2024-02-10T08:41:05.033144+0000","last_became_peered":"2024-02-10T08:41:05.033144+0000","last_unstale":"2024-02-10T08:41:05.033921+0000","last_undegraded":"2024-02-10T08:41:05.033921+0000","last_fullsized":"2024-02-10T08:41:05.033921+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.7","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029615+0000","last_change":"2024-02-10T08:41:05.027268+0000","last_active":"2024-02-10T08:41:05.029615+0000","last_peered":"2024-02-10T08:41:05.029615+0000","last_clean":"2024-02-10T08:41:05.029615+0000","last_became_active":"2024-02-10T08:41:05.026986+0000","last_became_peered":"2024-02-10T08:41:05.026986+0000","last_unstale":"2024-02-10T08:41:05.029615+0000","last_undegraded":"2024-02-10T08:41:05.029615+0000","last_fullsized":"2024-02-10T08:41:05.029615+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"204.6","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.029556+0000","last_change":"2024-02-10T08:41:05.027999+0000","last_active":"2024-02-10T08:41:05.029556+0000","last_peered":"2024-02-10T08:41:05.029556+0000","last_clean":"2024-02-10T08:41:05.029556+0000","last_became_active":"2024-02-10T08:41:05.027653+0000","last_became_peered":"2024-02-10T08:41:05.027653+0000","last_unstale":"2024-02-10T08:41:05.029556+0000","last_undegraded":"2024-02-10T08:41:05.029556+0000","last_fullsized":"2024-02-10T08:41:05.029556+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"204.5","version":"0'0","reported_seq":9,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.028787+0000","last_change":"2024-02-10T08:41:05.028787+0000","last_active":"2024-02-10T08:41:05.028787+0000","last_peered":"2024-02-10T08:41:05.028787+0000","last_clean":"2024-02-10T08:41:05.028787+0000","last_became_active":"2024-02-10T08:41:05.028024+0000","last_became_peered":"2024-02-10T08:41:05.028024+0000","last_unstale":"2024-02-10T08:41:05.028787+0000","last_undegraded":"2024-02-10T08:41:05.028787+0000","last_fullsized":"2024-02-10T08:41:05.028787+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.4","version":"0'0","reported_seq":10,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.036695+0000","last_change":"2024-02-10T08:41:05.035842+0000","last_active":"2024-02-10T08:41:05.036695+0000","last_peered":"2024-02-10T08:41:05.036695+0000","last_clean":"2024-02-10T08:41:05.036695+0000","last_became_active":"2024-02-10T08:41:05.035538+0000","last_became_peered":"2024-02-10T08:41:05.035538+0000","last_unstale":"2024-02-10T08:41:05.036695+0000","last_undegraded":"2024-02-10T08:41:05.036695+0000","last_fullsized":"2024-02-10T08:41:05.036695+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":370,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:04.002336+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.002336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":339,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015072+0000","last_change":"2024-02-10T08:40:32.184907+0000","last_active":"2024-02-10T08:41:05.015072+0000","last_peered":"2024-02-10T08:41:05.015072+0000","last_clean":"2024-02-10T08:41:05.015072+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:41:05.015072+0000","last_undegraded":"2024-02-10T08:41:05.015072+0000","last_fullsized":"2024-02-10T08:41:05.015072+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:40:32.184829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.184829+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"338'29","reported_seq":344,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021077+0000","last_change":"2024-02-10T08:39:38.573741+0000","last_active":"2024-02-10T08:41:04.021077+0000","last_peered":"2024-02-10T08:41:04.021077+0000","last_clean":"2024-02-10T08:41:04.021077+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:41:04.021077+0000","last_undegraded":"2024-02-10T08:41:04.021077+0000","last_fullsized":"2024-02-10T08:41:04.021077+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"250'39","reported_seq":341,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020235+0000","last_change":"2024-02-10T08:39:40.483224+0000","last_active":"2024-02-10T08:41:04.020235+0000","last_peered":"2024-02-10T08:41:04.020235+0000","last_clean":"2024-02-10T08:41:04.020235+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:41:04.020235+0000","last_undegraded":"2024-02-10T08:41:04.020235+0000","last_fullsized":"2024-02-10T08:41:04.020235+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"371'108","reported_seq":417,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.039894+0000","last_change":"2024-02-10T08:40:35.402698+0000","last_active":"2024-02-10T08:41:05.039894+0000","last_peered":"2024-02-10T08:41:05.039894+0000","last_clean":"2024-02-10T08:41:05.039894+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:41:05.039894+0000","last_undegraded":"2024-02-10T08:41:05.039894+0000","last_fullsized":"2024-02-10T08:41:05.039894+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"371'35","reported_seq":320,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.014702+0000","last_change":"2024-02-10T08:39:53.632526+0000","last_active":"2024-02-10T08:41:05.014702+0000","last_peered":"2024-02-10T08:41:05.014702+0000","last_clean":"2024-02-10T08:41:05.014702+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:41:05.014702+0000","last_undegraded":"2024-02-10T08:41:05.014702+0000","last_fullsized":"2024-02-10T08:41:05.014702+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"338'73","reported_seq":376,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019779+0000","last_change":"2024-02-10T08:39:49.450775+0000","last_active":"2024-02-10T08:41:05.019779+0000","last_peered":"2024-02-10T08:41:05.019779+0000","last_clean":"2024-02-10T08:41:05.019779+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:41:05.019779+0000","last_undegraded":"2024-02-10T08:41:05.019779+0000","last_fullsized":"2024-02-10T08:41:05.019779+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":358,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021867+0000","last_change":"2024-02-10T08:39:52.583051+0000","last_active":"2024-02-10T08:41:04.021867+0000","last_peered":"2024-02-10T08:41:04.021867+0000","last_clean":"2024-02-10T08:41:04.021867+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:41:04.021867+0000","last_undegraded":"2024-02-10T08:41:04.021867+0000","last_fullsized":"2024-02-10T08:41:04.021867+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"338'23","reported_seq":336,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.022371+0000","last_change":"2024-02-10T08:39:42.510684+0000","last_active":"2024-02-10T08:41:04.022371+0000","last_peered":"2024-02-10T08:41:04.022371+0000","last_clean":"2024-02-10T08:41:04.022371+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:41:04.022371+0000","last_undegraded":"2024-02-10T08:41:04.022371+0000","last_fullsized":"2024-02-10T08:41:04.022371+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"250'101","reported_seq":402,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017377+0000","last_change":"2024-02-10T08:40:28.463491+0000","last_active":"2024-02-10T08:41:05.017377+0000","last_peered":"2024-02-10T08:41:05.017377+0000","last_clean":"2024-02-10T08:41:05.017377+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:41:05.017377+0000","last_undegraded":"2024-02-10T08:41:05.017377+0000","last_fullsized":"2024-02-10T08:41:05.017377+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'101","last_scrub_stamp":"2024-02-10T08:40:28.463426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.463426+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"371'23","reported_seq":335,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.013530+0000","last_change":"2024-02-10T08:39:47.635717+0000","last_active":"2024-02-10T08:41:05.013530+0000","last_peered":"2024-02-10T08:41:05.013530+0000","last_clean":"2024-02-10T08:41:05.013530+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:41:05.013530+0000","last_undegraded":"2024-02-10T08:41:05.013530+0000","last_fullsized":"2024-02-10T08:41:05.013530+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"338'111","reported_seq":397,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017670+0000","last_change":"2024-02-10T08:39:53.048253+0000","last_active":"2024-02-10T08:41:05.017670+0000","last_peered":"2024-02-10T08:41:05.017670+0000","last_clean":"2024-02-10T08:41:05.017670+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:41:05.017670+0000","last_undegraded":"2024-02-10T08:41:05.017670+0000","last_fullsized":"2024-02-10T08:41:05.017670+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"338'86","reported_seq":390,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020546+0000","last_change":"2024-02-10T08:39:50.630114+0000","last_active":"2024-02-10T08:41:04.020546+0000","last_peered":"2024-02-10T08:41:04.020546+0000","last_clean":"2024-02-10T08:41:04.020546+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:41:04.020546+0000","last_undegraded":"2024-02-10T08:41:04.020546+0000","last_fullsized":"2024-02-10T08:41:04.020546+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"353'94","reported_seq":388,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.014911+0000","last_change":"2024-02-10T08:39:43.510896+0000","last_active":"2024-02-10T08:41:04.014911+0000","last_peered":"2024-02-10T08:41:04.014911+0000","last_clean":"2024-02-10T08:41:04.014911+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:41:04.014911+0000","last_undegraded":"2024-02-10T08:41:04.014911+0000","last_fullsized":"2024-02-10T08:41:04.014911+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'93","last_scrub_stamp":"2024-02-10T08:39:43.510789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.510789+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":80,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":341,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020979+0000","last_change":"2024-02-10T08:39:54.894186+0000","last_active":"2024-02-10T08:41:04.020979+0000","last_peered":"2024-02-10T08:41:04.020979+0000","last_clean":"2024-02-10T08:41:04.020979+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:41:04.020979+0000","last_undegraded":"2024-02-10T08:41:04.020979+0000","last_fullsized":"2024-02-10T08:41:04.020979+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"338'57","reported_seq":355,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013521+0000","last_change":"2024-02-10T08:39:44.469992+0000","last_active":"2024-02-10T08:41:04.013521+0000","last_peered":"2024-02-10T08:41:04.013521+0000","last_clean":"2024-02-10T08:41:04.013521+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:41:04.013521+0000","last_undegraded":"2024-02-10T08:41:04.013521+0000","last_fullsized":"2024-02-10T08:41:04.013521+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-02-10T08:39:44.469923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.469923+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"338'70","reported_seq":375,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019043+0000","last_change":"2024-02-10T08:40:36.430142+0000","last_active":"2024-02-10T08:41:05.019043+0000","last_peered":"2024-02-10T08:41:05.019043+0000","last_clean":"2024-02-10T08:41:05.019043+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:41:05.019043+0000","last_undegraded":"2024-02-10T08:41:05.019043+0000","last_fullsized":"2024-02-10T08:41:05.019043+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:40:36.430069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.430069+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"338'56","reported_seq":361,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.023080+0000","last_change":"2024-02-10T08:39:33.014018+0000","last_active":"2024-02-10T08:41:05.023080+0000","last_peered":"2024-02-10T08:41:05.023080+0000","last_clean":"2024-02-10T08:41:05.023080+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:41:05.023080+0000","last_undegraded":"2024-02-10T08:41:05.023080+0000","last_fullsized":"2024-02-10T08:41:05.023080+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'54","last_scrub_stamp":"2024-02-10T08:39:12.673905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.673905+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":407,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015158+0000","last_change":"2024-02-10T08:39:46.646522+0000","last_active":"2024-02-10T08:41:04.015158+0000","last_peered":"2024-02-10T08:41:04.015158+0000","last_clean":"2024-02-10T08:41:04.015158+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:41:04.015158+0000","last_undegraded":"2024-02-10T08:41:04.015158+0000","last_fullsized":"2024-02-10T08:41:04.015158+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:39:46.646454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.646454+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"258'103","reported_seq":396,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.027011+0000","last_change":"2024-02-10T08:39:41.578640+0000","last_active":"2024-02-10T08:41:05.027011+0000","last_peered":"2024-02-10T08:41:05.027011+0000","last_clean":"2024-02-10T08:41:05.027011+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:41:05.027011+0000","last_undegraded":"2024-02-10T08:41:05.027011+0000","last_fullsized":"2024-02-10T08:41:05.027011+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":382,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015138+0000","last_change":"2024-02-10T08:39:55.527916+0000","last_active":"2024-02-10T08:41:04.015138+0000","last_peered":"2024-02-10T08:41:04.015138+0000","last_clean":"2024-02-10T08:41:04.015138+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:41:04.015138+0000","last_undegraded":"2024-02-10T08:41:04.015138+0000","last_fullsized":"2024-02-10T08:41:04.015138+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:39:55.527854+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"215'85","reported_seq":384,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015534+0000","last_change":"2024-02-10T08:39:54.516959+0000","last_active":"2024-02-10T08:41:04.015534+0000","last_peered":"2024-02-10T08:41:04.015534+0000","last_clean":"2024-02-10T08:41:04.015534+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:41:04.015534+0000","last_undegraded":"2024-02-10T08:41:04.015534+0000","last_fullsized":"2024-02-10T08:41:04.015534+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'85","last_scrub_stamp":"2024-02-10T08:39:54.516887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.516887+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":386,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020920+0000","last_change":"2024-02-10T08:39:43.711787+0000","last_active":"2024-02-10T08:41:04.020920+0000","last_peered":"2024-02-10T08:41:04.020920+0000","last_clean":"2024-02-10T08:41:04.020920+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:41:04.020920+0000","last_undegraded":"2024-02-10T08:41:04.020920+0000","last_fullsized":"2024-02-10T08:41:04.020920+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"338'18","reported_seq":328,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021772+0000","last_change":"2024-02-10T08:39:56.545155+0000","last_active":"2024-02-10T08:41:04.021772+0000","last_peered":"2024-02-10T08:41:04.021772+0000","last_clean":"2024-02-10T08:41:04.021772+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:41:04.021772+0000","last_undegraded":"2024-02-10T08:41:04.021772+0000","last_fullsized":"2024-02-10T08:41:04.021772+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":339,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019393+0000","last_change":"2024-02-10T08:39:49.714905+0000","last_active":"2024-02-10T08:41:04.019393+0000","last_peered":"2024-02-10T08:41:04.019393+0000","last_clean":"2024-02-10T08:41:04.019393+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:41:04.019393+0000","last_undegraded":"2024-02-10T08:41:04.019393+0000","last_fullsized":"2024-02-10T08:41:04.019393+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"338'69","reported_seq":354,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021158+0000","last_change":"2024-02-10T08:40:24.583295+0000","last_active":"2024-02-10T08:41:04.021158+0000","last_peered":"2024-02-10T08:41:04.021158+0000","last_clean":"2024-02-10T08:41:04.021158+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:41:04.021158+0000","last_undegraded":"2024-02-10T08:41:04.021158+0000","last_fullsized":"2024-02-10T08:41:04.021158+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"338'69","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016329+0000","last_change":"2024-02-10T08:40:35.307685+0000","last_active":"2024-02-10T08:41:05.016329+0000","last_peered":"2024-02-10T08:41:05.016329+0000","last_clean":"2024-02-10T08:41:05.016329+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:41:05.016329+0000","last_undegraded":"2024-02-10T08:41:05.016329+0000","last_fullsized":"2024-02-10T08:41:05.016329+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:40:35.307608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.307608+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":352,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019104+0000","last_change":"2024-02-10T08:40:28.186561+0000","last_active":"2024-02-10T08:41:05.019104+0000","last_peered":"2024-02-10T08:41:05.019104+0000","last_clean":"2024-02-10T08:41:05.019104+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:41:05.019104+0000","last_undegraded":"2024-02-10T08:41:05.019104+0000","last_fullsized":"2024-02-10T08:41:05.019104+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'34","last_scrub_stamp":"2024-02-10T08:40:28.186484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.186484+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"338'35","reported_seq":323,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020471+0000","last_change":"2024-02-10T08:39:54.602656+0000","last_active":"2024-02-10T08:41:04.020471+0000","last_peered":"2024-02-10T08:41:04.020471+0000","last_clean":"2024-02-10T08:41:04.020471+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:41:04.020471+0000","last_undegraded":"2024-02-10T08:41:04.020471+0000","last_fullsized":"2024-02-10T08:41:04.020471+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"338'36","reported_seq":352,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.021869+0000","last_change":"2024-02-10T08:39:33.014676+0000","last_active":"2024-02-10T08:41:05.021869+0000","last_peered":"2024-02-10T08:41:05.021869+0000","last_clean":"2024-02-10T08:41:05.021869+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:41:05.021869+0000","last_undegraded":"2024-02-10T08:41:05.021869+0000","last_fullsized":"2024-02-10T08:41:05.021869+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'34","last_scrub_stamp":"2024-02-10T08:39:04.785239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.785239+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":309,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015360+0000","last_change":"2024-02-10T08:39:38.041298+0000","last_active":"2024-02-10T08:41:05.015360+0000","last_peered":"2024-02-10T08:41:05.015360+0000","last_clean":"2024-02-10T08:41:05.015360+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:41:05.015360+0000","last_undegraded":"2024-02-10T08:41:05.015360+0000","last_fullsized":"2024-02-10T08:41:05.015360+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:39:38.041185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.041185+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"338'19","reported_seq":317,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019452+0000","last_change":"2024-02-10T08:40:30.882955+0000","last_active":"2024-02-10T08:41:04.019452+0000","last_peered":"2024-02-10T08:41:04.019452+0000","last_clean":"2024-02-10T08:41:04.019452+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:41:04.019452+0000","last_undegraded":"2024-02-10T08:41:04.019452+0000","last_fullsized":"2024-02-10T08:41:04.019452+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"371'35","reported_seq":344,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.041040+0000","last_change":"2024-02-10T08:40:23.345616+0000","last_active":"2024-02-10T08:41:05.041040+0000","last_peered":"2024-02-10T08:41:05.041040+0000","last_clean":"2024-02-10T08:41:05.041040+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:41:05.041040+0000","last_undegraded":"2024-02-10T08:41:05.041040+0000","last_fullsized":"2024-02-10T08:41:05.041040+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:40:23.345538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.345538+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":455,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016929+0000","last_change":"2024-02-10T08:38:21.488646+0000","last_active":"2024-02-10T08:41:04.016929+0000","last_peered":"2024-02-10T08:41:04.016929+0000","last_clean":"2024-02-10T08:41:04.016929+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:41:04.016929+0000","last_undegraded":"2024-02-10T08:41:04.016929+0000","last_fullsized":"2024-02-10T08:41:04.016929+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:38:21.488563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:21.488563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":368,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018178+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:41:05.018178+0000","last_peered":"2024-02-10T08:41:05.018178+0000","last_clean":"2024-02-10T08:41:05.018178+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:41:05.018178+0000","last_undegraded":"2024-02-10T08:41:05.018178+0000","last_fullsized":"2024-02-10T08:41:05.018178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":340,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019921+0000","last_change":"2024-02-10T08:40:42.232974+0000","last_active":"2024-02-10T08:41:05.019921+0000","last_peered":"2024-02-10T08:41:05.019921+0000","last_clean":"2024-02-10T08:41:05.019921+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:41:05.019921+0000","last_undegraded":"2024-02-10T08:41:05.019921+0000","last_fullsized":"2024-02-10T08:41:05.019921+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.232905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.232905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":346,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017448+0000","last_change":"2024-02-10T08:40:26.859788+0000","last_active":"2024-02-10T08:41:05.017448+0000","last_peered":"2024-02-10T08:41:05.017448+0000","last_clean":"2024-02-10T08:41:05.017448+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:41:05.017448+0000","last_undegraded":"2024-02-10T08:41:05.017448+0000","last_fullsized":"2024-02-10T08:41:05.017448+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.859697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.859697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":283,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019127+0000","last_change":"2024-02-10T08:38:33.920702+0000","last_active":"2024-02-10T08:41:05.019127+0000","last_peered":"2024-02-10T08:41:05.019127+0000","last_clean":"2024-02-10T08:41:05.019127+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:41:05.019127+0000","last_undegraded":"2024-02-10T08:41:05.019127+0000","last_fullsized":"2024-02-10T08:41:05.019127+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.920630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.920630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":312,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017643+0000","last_change":"2024-02-10T08:37:40.896760+0000","last_active":"2024-02-10T08:41:05.017643+0000","last_peered":"2024-02-10T08:41:05.017643+0000","last_clean":"2024-02-10T08:41:05.017643+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:41:05.017643+0000","last_undegraded":"2024-02-10T08:41:05.017643+0000","last_fullsized":"2024-02-10T08:41:05.017643+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:37:40.896694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:40.896694+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":345,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.559532+0000","last_change":"2024-02-10T08:41:05.559497+0000","last_active":"2024-02-10T08:41:05.559532+0000","last_peered":"2024-02-10T08:41:05.559532+0000","last_clean":"2024-02-10T08:41:05.559532+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:41:05.559532+0000","last_undegraded":"2024-02-10T08:41:05.559532+0000","last_fullsized":"2024-02-10T08:41:05.559532+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.559402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.559402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":354,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018786+0000","last_change":"2024-02-10T08:37:42.929238+0000","last_active":"2024-02-10T08:41:05.018786+0000","last_peered":"2024-02-10T08:41:05.018786+0000","last_clean":"2024-02-10T08:41:05.018786+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:41:05.018786+0000","last_undegraded":"2024-02-10T08:41:05.018786+0000","last_fullsized":"2024-02-10T08:41:05.018786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.929167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.929167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":256,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016712+0000","last_change":"2024-02-10T08:40:39.350744+0000","last_active":"2024-02-10T08:41:04.016712+0000","last_peered":"2024-02-10T08:41:04.016712+0000","last_clean":"2024-02-10T08:41:04.016712+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:41:04.016712+0000","last_undegraded":"2024-02-10T08:41:04.016712+0000","last_fullsized":"2024-02-10T08:41:04.016712+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.350663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":281,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.022654+0000","last_change":"2024-02-10T08:38:26.677780+0000","last_active":"2024-02-10T08:41:05.022654+0000","last_peered":"2024-02-10T08:41:05.022654+0000","last_clean":"2024-02-10T08:41:05.022654+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:41:05.022654+0000","last_undegraded":"2024-02-10T08:41:05.022654+0000","last_fullsized":"2024-02-10T08:41:05.022654+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.677701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.677701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":371,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013731+0000","last_change":"2024-02-10T08:40:24.395617+0000","last_active":"2024-02-10T08:41:04.013731+0000","last_peered":"2024-02-10T08:41:04.013731+0000","last_clean":"2024-02-10T08:41:04.013731+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:41:04.013731+0000","last_undegraded":"2024-02-10T08:41:04.013731+0000","last_fullsized":"2024-02-10T08:41:04.013731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:40:24.395541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.395541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":349,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019507+0000","last_change":"2024-02-10T08:40:40.314002+0000","last_active":"2024-02-10T08:41:05.019507+0000","last_peered":"2024-02-10T08:41:05.019507+0000","last_clean":"2024-02-10T08:41:05.019507+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:41:05.019507+0000","last_undegraded":"2024-02-10T08:41:05.019507+0000","last_fullsized":"2024-02-10T08:41:05.019507+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.313935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017511+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:41:05.017511+0000","last_peered":"2024-02-10T08:41:05.017511+0000","last_clean":"2024-02-10T08:41:05.017511+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:41:05.017511+0000","last_undegraded":"2024-02-10T08:41:05.017511+0000","last_fullsized":"2024-02-10T08:41:05.017511+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015293+0000","last_change":"2024-02-10T08:40:28.426111+0000","last_active":"2024-02-10T08:41:04.015293+0000","last_peered":"2024-02-10T08:41:04.015293+0000","last_clean":"2024-02-10T08:41:04.015293+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:41:04.015293+0000","last_undegraded":"2024-02-10T08:41:04.015293+0000","last_fullsized":"2024-02-10T08:41:04.015293+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:28.425994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.425994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":281,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016978+0000","last_change":"2024-02-10T08:39:44.585775+0000","last_active":"2024-02-10T08:41:04.016978+0000","last_peered":"2024-02-10T08:41:04.016978+0000","last_clean":"2024-02-10T08:41:04.016978+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:41:04.016978+0000","last_undegraded":"2024-02-10T08:41:04.016978+0000","last_fullsized":"2024-02-10T08:41:04.016978+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:44.585674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.585674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":365,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.022298+0000","last_change":"2024-02-10T08:37:50.586954+0000","last_active":"2024-02-10T08:41:04.022298+0000","last_peered":"2024-02-10T08:41:04.022298+0000","last_clean":"2024-02-10T08:41:04.022298+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:41:04.022298+0000","last_undegraded":"2024-02-10T08:41:04.022298+0000","last_fullsized":"2024-02-10T08:41:04.022298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:37:50.586882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:50.586882+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":173,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018830+0000","last_change":"2024-02-10T08:38:22.013062+0000","last_active":"2024-02-10T08:41:04.018830+0000","last_peered":"2024-02-10T08:41:04.018830+0000","last_clean":"2024-02-10T08:41:04.018830+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:41:04.018830+0000","last_undegraded":"2024-02-10T08:41:04.018830+0000","last_fullsized":"2024-02-10T08:41:04.018830+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.545778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.545778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017519+0000","last_change":"2024-02-10T08:40:40.501961+0000","last_active":"2024-02-10T08:41:05.017519+0000","last_peered":"2024-02-10T08:41:05.017519+0000","last_clean":"2024-02-10T08:41:05.017519+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:41:05.017519+0000","last_undegraded":"2024-02-10T08:41:05.017519+0000","last_fullsized":"2024-02-10T08:41:05.017519+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.501871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.501871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":334,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018935+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:41:05.018935+0000","last_peered":"2024-02-10T08:41:05.018935+0000","last_clean":"2024-02-10T08:41:05.018935+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:41:05.018935+0000","last_undegraded":"2024-02-10T08:41:05.018935+0000","last_fullsized":"2024-02-10T08:41:05.018935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":282,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020289+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:41:04.020289+0000","last_peered":"2024-02-10T08:41:04.020289+0000","last_clean":"2024-02-10T08:41:04.020289+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:41:04.020289+0000","last_undegraded":"2024-02-10T08:41:04.020289+0000","last_fullsized":"2024-02-10T08:41:04.020289+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":352,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.022767+0000","last_change":"2024-02-10T08:38:17.727465+0000","last_active":"2024-02-10T08:41:04.022767+0000","last_peered":"2024-02-10T08:41:04.022767+0000","last_clean":"2024-02-10T08:41:04.022767+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:41:04.022767+0000","last_undegraded":"2024-02-10T08:41:04.022767+0000","last_fullsized":"2024-02-10T08:41:04.022767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:17.727387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:17.727387+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":340,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016701+0000","last_change":"2024-02-10T08:37:38.680684+0000","last_active":"2024-02-10T08:41:05.016701+0000","last_peered":"2024-02-10T08:41:05.016701+0000","last_clean":"2024-02-10T08:41:05.016701+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:41:05.016701+0000","last_undegraded":"2024-02-10T08:41:05.016701+0000","last_fullsized":"2024-02-10T08:41:05.016701+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:38.680608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:38.680608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":457,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016170+0000","last_change":"2024-02-10T08:40:41.276981+0000","last_active":"2024-02-10T08:41:05.016170+0000","last_peered":"2024-02-10T08:41:05.016170+0000","last_clean":"2024-02-10T08:41:05.016170+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:41:05.016170+0000","last_undegraded":"2024-02-10T08:41:05.016170+0000","last_fullsized":"2024-02-10T08:41:05.016170+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:40:41.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.276901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023201+0000","last_change":"2024-02-10T08:38:25.851680+0000","last_active":"2024-02-10T08:41:04.023201+0000","last_peered":"2024-02-10T08:41:04.023201+0000","last_clean":"2024-02-10T08:41:04.023201+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:41:04.023201+0000","last_undegraded":"2024-02-10T08:41:04.023201+0000","last_fullsized":"2024-02-10T08:41:04.023201+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.851576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.851576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":252,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018632+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:41:04.018632+0000","last_peered":"2024-02-10T08:41:04.018632+0000","last_clean":"2024-02-10T08:41:04.018632+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:41:04.018632+0000","last_undegraded":"2024-02-10T08:41:04.018632+0000","last_fullsized":"2024-02-10T08:41:04.018632+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016154+0000","last_change":"2024-02-10T08:40:42.486085+0000","last_active":"2024-02-10T08:41:05.016154+0000","last_peered":"2024-02-10T08:41:05.016154+0000","last_clean":"2024-02-10T08:41:05.016154+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:41:05.016154+0000","last_undegraded":"2024-02-10T08:41:05.016154+0000","last_fullsized":"2024-02-10T08:41:05.016154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:40:42.486021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.486021+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":348,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019424+0000","last_change":"2024-02-10T08:38:18.008913+0000","last_active":"2024-02-10T08:41:05.019424+0000","last_peered":"2024-02-10T08:41:05.019424+0000","last_clean":"2024-02-10T08:41:05.019424+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:41:05.019424+0000","last_undegraded":"2024-02-10T08:41:05.019424+0000","last_fullsized":"2024-02-10T08:41:05.019424+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:18.008846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.008846+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":481,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023138+0000","last_change":"2024-02-10T08:40:25.631962+0000","last_active":"2024-02-10T08:41:04.023138+0000","last_peered":"2024-02-10T08:41:04.023138+0000","last_clean":"2024-02-10T08:41:04.023138+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:41:04.023138+0000","last_undegraded":"2024-02-10T08:41:04.023138+0000","last_fullsized":"2024-02-10T08:41:04.023138+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:40:25.631854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:25.631854+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":341,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019585+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:41:05.019585+0000","last_peered":"2024-02-10T08:41:05.019585+0000","last_clean":"2024-02-10T08:41:05.019585+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:41:05.019585+0000","last_undegraded":"2024-02-10T08:41:05.019585+0000","last_fullsized":"2024-02-10T08:41:05.019585+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":290,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.023245+0000","last_change":"2024-02-10T08:40:42.029093+0000","last_active":"2024-02-10T08:41:05.023245+0000","last_peered":"2024-02-10T08:41:05.023245+0000","last_clean":"2024-02-10T08:41:05.023245+0000","last_became_active":"2024-02-10T08:40:42.028734+0000","last_became_peered":"2024-02-10T08:40:42.028734+0000","last_unstale":"2024-02-10T08:41:05.023245+0000","last_undegraded":"2024-02-10T08:41:05.023245+0000","last_fullsized":"2024-02-10T08:41:05.023245+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.788118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.788118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":356,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023055+0000","last_change":"2024-02-10T08:38:59.745198+0000","last_active":"2024-02-10T08:41:04.023055+0000","last_peered":"2024-02-10T08:41:04.023055+0000","last_clean":"2024-02-10T08:41:04.023055+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:41:04.023055+0000","last_undegraded":"2024-02-10T08:41:04.023055+0000","last_fullsized":"2024-02-10T08:41:04.023055+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:38:59.745114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.745114+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.017196+0000","last_change":"2024-02-10T08:38:33.363322+0000","last_active":"2024-02-10T08:41:04.017196+0000","last_peered":"2024-02-10T08:41:04.017196+0000","last_clean":"2024-02-10T08:41:04.017196+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:41:04.017196+0000","last_undegraded":"2024-02-10T08:41:04.017196+0000","last_fullsized":"2024-02-10T08:41:04.017196+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.363195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":408,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.022363+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:41:05.022363+0000","last_peered":"2024-02-10T08:41:05.022363+0000","last_clean":"2024-02-10T08:41:05.022363+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:41:05.022363+0000","last_undegraded":"2024-02-10T08:41:05.022363+0000","last_fullsized":"2024-02-10T08:41:05.022363+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020948+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:41:04.020948+0000","last_peered":"2024-02-10T08:41:04.020948+0000","last_clean":"2024-02-10T08:41:04.020948+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:41:04.020948+0000","last_undegraded":"2024-02-10T08:41:04.020948+0000","last_fullsized":"2024-02-10T08:41:04.020948+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":282,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019098+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:41:04.019098+0000","last_peered":"2024-02-10T08:41:04.019098+0000","last_clean":"2024-02-10T08:41:04.019098+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:41:04.019098+0000","last_undegraded":"2024-02-10T08:41:04.019098+0000","last_fullsized":"2024-02-10T08:41:04.019098+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":438,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018845+0000","last_change":"2024-02-10T08:38:18.475575+0000","last_active":"2024-02-10T08:41:04.018845+0000","last_peered":"2024-02-10T08:41:04.018845+0000","last_clean":"2024-02-10T08:41:04.018845+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:41:04.018845+0000","last_undegraded":"2024-02-10T08:41:04.018845+0000","last_fullsized":"2024-02-10T08:41:04.018845+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:38:18.475509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.475509+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":358,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020876+0000","last_change":"2024-02-10T08:38:31.840285+0000","last_active":"2024-02-10T08:41:04.020876+0000","last_peered":"2024-02-10T08:41:04.020876+0000","last_clean":"2024-02-10T08:41:04.020876+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:41:04.020876+0000","last_undegraded":"2024-02-10T08:41:04.020876+0000","last_fullsized":"2024-02-10T08:41:04.020876+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:38:31.840218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.840218+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015117+0000","last_change":"2024-02-10T08:39:48.521060+0000","last_active":"2024-02-10T08:41:04.015117+0000","last_peered":"2024-02-10T08:41:04.015117+0000","last_clean":"2024-02-10T08:41:04.015117+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:41:04.015117+0000","last_undegraded":"2024-02-10T08:41:04.015117+0000","last_fullsized":"2024-02-10T08:41:04.015117+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:48.520941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:48.520941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016059+0000","last_change":"2024-02-10T08:39:56.568250+0000","last_active":"2024-02-10T08:41:04.016059+0000","last_peered":"2024-02-10T08:41:04.016059+0000","last_clean":"2024-02-10T08:41:04.016059+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:41:04.016059+0000","last_undegraded":"2024-02-10T08:41:04.016059+0000","last_fullsized":"2024-02-10T08:41:04.016059+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:56.568189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.568189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":253,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017157+0000","last_change":"2024-02-10T08:39:12.085361+0000","last_active":"2024-02-10T08:41:05.017157+0000","last_peered":"2024-02-10T08:41:05.017157+0000","last_clean":"2024-02-10T08:41:05.017157+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:41:05.017157+0000","last_undegraded":"2024-02-10T08:41:05.017157+0000","last_fullsized":"2024-02-10T08:41:05.017157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.085261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.085261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020993+0000","last_change":"2024-02-10T08:40:34.622947+0000","last_active":"2024-02-10T08:41:04.020993+0000","last_peered":"2024-02-10T08:41:04.020993+0000","last_clean":"2024-02-10T08:41:04.020993+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:41:04.020993+0000","last_undegraded":"2024-02-10T08:41:04.020993+0000","last_fullsized":"2024-02-10T08:41:04.020993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:34.622869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":425,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018823+0000","last_change":"2024-02-10T08:40:42.026419+0000","last_active":"2024-02-10T08:41:04.018823+0000","last_peered":"2024-02-10T08:41:04.018823+0000","last_clean":"2024-02-10T08:41:04.018823+0000","last_became_active":"2024-02-10T08:40:42.015369+0000","last_became_peered":"2024-02-10T08:40:42.015369+0000","last_unstale":"2024-02-10T08:41:04.018823+0000","last_undegraded":"2024-02-10T08:41:04.018823+0000","last_fullsized":"2024-02-10T08:41:04.018823+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:37:39.542439+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:37:39.542439+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":347,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015484+0000","last_change":"2024-02-10T08:38:57.920239+0000","last_active":"2024-02-10T08:41:05.015484+0000","last_peered":"2024-02-10T08:41:05.015484+0000","last_clean":"2024-02-10T08:41:05.015484+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:41:05.015484+0000","last_undegraded":"2024-02-10T08:41:05.015484+0000","last_fullsized":"2024-02-10T08:41:05.015484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:38:57.920144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.920144+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":328,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021547+0000","last_change":"2024-02-10T08:37:42.525144+0000","last_active":"2024-02-10T08:41:04.021547+0000","last_peered":"2024-02-10T08:41:04.021547+0000","last_clean":"2024-02-10T08:41:04.021547+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:41:04.021547+0000","last_undegraded":"2024-02-10T08:41:04.021547+0000","last_fullsized":"2024-02-10T08:41:04.021547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:42.525050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:42.525050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":261,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019633+0000","last_change":"2024-02-10T08:40:42.028088+0000","last_active":"2024-02-10T08:41:04.019633+0000","last_peered":"2024-02-10T08:41:04.019633+0000","last_clean":"2024-02-10T08:41:04.019633+0000","last_became_active":"2024-02-10T08:40:42.027869+0000","last_became_peered":"2024-02-10T08:40:42.027869+0000","last_unstale":"2024-02-10T08:41:04.019633+0000","last_undegraded":"2024-02-10T08:41:04.019633+0000","last_fullsized":"2024-02-10T08:41:04.019633+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013819+0000","last_change":"2024-02-10T08:40:32.412832+0000","last_active":"2024-02-10T08:41:04.013819+0000","last_peered":"2024-02-10T08:41:04.013819+0000","last_clean":"2024-02-10T08:41:04.013819+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:41:04.013819+0000","last_undegraded":"2024-02-10T08:41:04.013819+0000","last_fullsized":"2024-02-10T08:41:04.013819+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:32.412763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.412763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":317,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.023456+0000","last_change":"2024-02-10T08:38:18.588026+0000","last_active":"2024-02-10T08:41:05.023456+0000","last_peered":"2024-02-10T08:41:05.023456+0000","last_clean":"2024-02-10T08:41:05.023456+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:41:05.023456+0000","last_undegraded":"2024-02-10T08:41:05.023456+0000","last_fullsized":"2024-02-10T08:41:05.023456+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:38:18.587949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.587949+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":352,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.369663+0000","last_change":"2024-02-10T08:41:04.369613+0000","last_active":"2024-02-10T08:41:04.369663+0000","last_peered":"2024-02-10T08:41:04.369663+0000","last_clean":"2024-02-10T08:41:04.369663+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:41:04.369663+0000","last_undegraded":"2024-02-10T08:41:04.369663+0000","last_fullsized":"2024-02-10T08:41:04.369663+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.369534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.369534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":334,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019570+0000","last_change":"2024-02-10T08:38:26.946622+0000","last_active":"2024-02-10T08:41:05.019570+0000","last_peered":"2024-02-10T08:41:05.019570+0000","last_clean":"2024-02-10T08:41:05.019570+0000","last_became_active":"2024-02-10T08:33:24.126840+0000","last_became_peered":"2024-02-10T08:33:24.126840+0000","last_unstale":"2024-02-10T08:41:05.019570+0000","last_undegraded":"2024-02-10T08:41:05.019570+0000","last_fullsized":"2024-02-10T08:41:05.019570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:26.946531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.946531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018786+0000","last_change":"2024-02-10T08:38:25.903816+0000","last_active":"2024-02-10T08:41:05.018786+0000","last_peered":"2024-02-10T08:41:05.018786+0000","last_clean":"2024-02-10T08:41:05.018786+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:41:05.018786+0000","last_undegraded":"2024-02-10T08:41:05.018786+0000","last_fullsized":"2024-02-10T08:41:05.018786+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.903753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.903753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":361,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.022475+0000","last_change":"2024-02-10T08:38:31.745438+0000","last_active":"2024-02-10T08:41:05.022475+0000","last_peered":"2024-02-10T08:41:05.022475+0000","last_clean":"2024-02-10T08:41:05.022475+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:41:05.022475+0000","last_undegraded":"2024-02-10T08:41:05.022475+0000","last_fullsized":"2024-02-10T08:41:05.022475+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.745362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.745362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":297,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015640+0000","last_change":"2024-02-10T08:37:41.752968+0000","last_active":"2024-02-10T08:41:05.015640+0000","last_peered":"2024-02-10T08:41:05.015640+0000","last_clean":"2024-02-10T08:41:05.015640+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:41:05.015640+0000","last_undegraded":"2024-02-10T08:41:05.015640+0000","last_fullsized":"2024-02-10T08:41:05.015640+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:37:41.752902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.752902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":352,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016988+0000","last_change":"2024-02-10T08:40:23.406297+0000","last_active":"2024-02-10T08:41:04.016988+0000","last_peered":"2024-02-10T08:41:04.016988+0000","last_clean":"2024-02-10T08:41:04.016988+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:41:04.016988+0000","last_undegraded":"2024-02-10T08:41:04.016988+0000","last_fullsized":"2024-02-10T08:41:04.016988+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.406227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.406227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":252,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020411+0000","last_change":"2024-02-10T08:38:18.758911+0000","last_active":"2024-02-10T08:41:04.020411+0000","last_peered":"2024-02-10T08:41:04.020411+0000","last_clean":"2024-02-10T08:41:04.020411+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:41:04.020411+0000","last_undegraded":"2024-02-10T08:41:04.020411+0000","last_fullsized":"2024-02-10T08:41:04.020411+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:18.758843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:18.758843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":335,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017565+0000","last_change":"2024-02-10T08:37:47.814654+0000","last_active":"2024-02-10T08:41:05.017565+0000","last_peered":"2024-02-10T08:41:05.017565+0000","last_clean":"2024-02-10T08:41:05.017565+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:41:05.017565+0000","last_undegraded":"2024-02-10T08:41:05.017565+0000","last_fullsized":"2024-02-10T08:41:05.017565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:37:47.814587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:47.814587+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":289,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013639+0000","last_change":"2024-02-10T08:39:06.770922+0000","last_active":"2024-02-10T08:41:04.013639+0000","last_peered":"2024-02-10T08:41:04.013639+0000","last_clean":"2024-02-10T08:41:04.013639+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:41:04.013639+0000","last_undegraded":"2024-02-10T08:41:04.013639+0000","last_fullsized":"2024-02-10T08:41:04.013639+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.770850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.770850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":334,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016571+0000","last_change":"2024-02-10T08:39:38.645588+0000","last_active":"2024-02-10T08:41:04.016571+0000","last_peered":"2024-02-10T08:41:04.016571+0000","last_clean":"2024-02-10T08:41:04.016571+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:41:04.016571+0000","last_undegraded":"2024-02-10T08:41:04.016571+0000","last_fullsized":"2024-02-10T08:41:04.016571+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.645527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.645527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":362,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013710+0000","last_change":"2024-02-10T08:40:35.376945+0000","last_active":"2024-02-10T08:41:04.013710+0000","last_peered":"2024-02-10T08:41:04.013710+0000","last_clean":"2024-02-10T08:41:04.013710+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:41:04.013710+0000","last_undegraded":"2024-02-10T08:41:04.013710+0000","last_fullsized":"2024-02-10T08:41:04.013710+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:40:35.376859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.376859+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":355,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.017112+0000","last_change":"2024-02-10T08:39:36.639528+0000","last_active":"2024-02-10T08:41:04.017112+0000","last_peered":"2024-02-10T08:41:04.017112+0000","last_clean":"2024-02-10T08:41:04.017112+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:41:04.017112+0000","last_undegraded":"2024-02-10T08:41:04.017112+0000","last_fullsized":"2024-02-10T08:41:04.017112+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.639461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.639461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":338,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.023409+0000","last_change":"2024-02-10T08:38:25.633733+0000","last_active":"2024-02-10T08:41:05.023409+0000","last_peered":"2024-02-10T08:41:05.023409+0000","last_clean":"2024-02-10T08:41:05.023409+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:41:05.023409+0000","last_undegraded":"2024-02-10T08:41:05.023409+0000","last_fullsized":"2024-02-10T08:41:05.023409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:25.633608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:25.633608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":338,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021181+0000","last_change":"2024-02-10T08:40:39.891622+0000","last_active":"2024-02-10T08:41:04.021181+0000","last_peered":"2024-02-10T08:41:04.021181+0000","last_clean":"2024-02-10T08:41:04.021181+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:41:04.021181+0000","last_undegraded":"2024-02-10T08:41:04.021181+0000","last_fullsized":"2024-02-10T08:41:04.021181+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.891558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.891558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.022564+0000","last_change":"2024-02-10T08:39:00.751654+0000","last_active":"2024-02-10T08:41:05.022564+0000","last_peered":"2024-02-10T08:41:05.022564+0000","last_clean":"2024-02-10T08:41:05.022564+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:41:05.022564+0000","last_undegraded":"2024-02-10T08:41:05.022564+0000","last_fullsized":"2024-02-10T08:41:05.022564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:00.751584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.751584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.027098+0000","last_change":"2024-02-10T08:39:11.660136+0000","last_active":"2024-02-10T08:41:05.027098+0000","last_peered":"2024-02-10T08:41:05.027098+0000","last_clean":"2024-02-10T08:41:05.027098+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:41:05.027098+0000","last_undegraded":"2024-02-10T08:41:05.027098+0000","last_fullsized":"2024-02-10T08:41:05.027098+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:11.660049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.660049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":354,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.020099+0000","last_change":"2024-02-10T08:41:03.238150+0000","last_active":"2024-02-10T08:41:05.020099+0000","last_peered":"2024-02-10T08:41:05.020099+0000","last_clean":"2024-02-10T08:41:05.020099+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:41:05.020099+0000","last_undegraded":"2024-02-10T08:41:05.020099+0000","last_fullsized":"2024-02-10T08:41:05.020099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.238035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.238035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":386,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016086+0000","last_change":"2024-02-10T08:37:49.889209+0000","last_active":"2024-02-10T08:41:05.016086+0000","last_peered":"2024-02-10T08:41:05.016086+0000","last_clean":"2024-02-10T08:41:05.016086+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:41:05.016086+0000","last_undegraded":"2024-02-10T08:41:05.016086+0000","last_fullsized":"2024-02-10T08:41:05.016086+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:37:49.889145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:49.889145+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":351,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015416+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:41:05.015416+0000","last_peered":"2024-02-10T08:41:05.015416+0000","last_clean":"2024-02-10T08:41:05.015416+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:41:05.015416+0000","last_undegraded":"2024-02-10T08:41:05.015416+0000","last_fullsized":"2024-02-10T08:41:05.015416+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":805,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.915949+0000","last_change":"2024-02-10T08:39:05.948676+0000","last_active":"2024-02-10T08:41:05.915949+0000","last_peered":"2024-02-10T08:41:05.915949+0000","last_clean":"2024-02-10T08:41:05.915949+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:41:05.915949+0000","last_undegraded":"2024-02-10T08:41:05.915949+0000","last_fullsized":"2024-02-10T08:41:05.915949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:39:05.948579+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.948579+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016544+0000","last_change":"2024-02-10T08:39:03.251683+0000","last_active":"2024-02-10T08:41:04.016544+0000","last_peered":"2024-02-10T08:41:04.016544+0000","last_clean":"2024-02-10T08:41:04.016544+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:41:04.016544+0000","last_undegraded":"2024-02-10T08:41:04.016544+0000","last_fullsized":"2024-02-10T08:41:04.016544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.251585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.251585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015428+0000","last_change":"2024-02-10T08:40:22.234338+0000","last_active":"2024-02-10T08:41:05.015428+0000","last_peered":"2024-02-10T08:41:05.015428+0000","last_clean":"2024-02-10T08:41:05.015428+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:41:05.015428+0000","last_undegraded":"2024-02-10T08:41:05.015428+0000","last_fullsized":"2024-02-10T08:41:05.015428+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.234270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.234270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":253,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016905+0000","last_change":"2024-02-10T08:39:06.965218+0000","last_active":"2024-02-10T08:41:05.016905+0000","last_peered":"2024-02-10T08:41:05.016905+0000","last_clean":"2024-02-10T08:41:05.016905+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:41:05.016905+0000","last_undegraded":"2024-02-10T08:41:05.016905+0000","last_fullsized":"2024-02-10T08:41:05.016905+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:06.965152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:06.965152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":421,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.021286+0000","last_change":"2024-02-10T08:40:40.098585+0000","last_active":"2024-02-10T08:41:05.021286+0000","last_peered":"2024-02-10T08:41:05.021286+0000","last_clean":"2024-02-10T08:41:05.021286+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:41:05.021286+0000","last_undegraded":"2024-02-10T08:41:05.021286+0000","last_fullsized":"2024-02-10T08:41:05.021286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:40:40.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.098515+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017608+0000","last_change":"2024-02-10T08:40:23.818554+0000","last_active":"2024-02-10T08:41:05.017608+0000","last_peered":"2024-02-10T08:41:05.017608+0000","last_clean":"2024-02-10T08:41:05.017608+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:41:05.017608+0000","last_undegraded":"2024-02-10T08:41:05.017608+0000","last_fullsized":"2024-02-10T08:41:05.017608+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.818434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.818434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":327,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021213+0000","last_change":"2024-02-10T08:37:21.366405+0000","last_active":"2024-02-10T08:41:04.021213+0000","last_peered":"2024-02-10T08:41:04.021213+0000","last_clean":"2024-02-10T08:41:04.021213+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:41:04.021213+0000","last_undegraded":"2024-02-10T08:41:04.021213+0000","last_fullsized":"2024-02-10T08:41:04.021213+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:21.366288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:37:21.366288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015046+0000","last_change":"2024-02-10T08:38:57.168613+0000","last_active":"2024-02-10T08:41:05.015046+0000","last_peered":"2024-02-10T08:41:05.015046+0000","last_clean":"2024-02-10T08:41:05.015046+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:41:05.015046+0000","last_undegraded":"2024-02-10T08:41:05.015046+0000","last_fullsized":"2024-02-10T08:41:05.015046+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:57.168523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:57.168523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":325,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013363+0000","last_change":"2024-02-10T08:39:52.548918+0000","last_active":"2024-02-10T08:41:04.013363+0000","last_peered":"2024-02-10T08:41:04.013363+0000","last_clean":"2024-02-10T08:41:04.013363+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:41:04.013363+0000","last_undegraded":"2024-02-10T08:41:04.013363+0000","last_fullsized":"2024-02-10T08:41:04.013363+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:52.548839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.548839+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016988+0000","last_change":"2024-02-10T08:41:03.584639+0000","last_active":"2024-02-10T08:41:05.016988+0000","last_peered":"2024-02-10T08:41:05.016988+0000","last_clean":"2024-02-10T08:41:05.016988+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:41:05.016988+0000","last_undegraded":"2024-02-10T08:41:05.016988+0000","last_fullsized":"2024-02-10T08:41:05.016988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:41:03.584573+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.584573+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":349,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017416+0000","last_change":"2024-02-10T08:38:59.955336+0000","last_active":"2024-02-10T08:41:05.017416+0000","last_peered":"2024-02-10T08:41:05.017416+0000","last_clean":"2024-02-10T08:41:05.017416+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:41:05.017416+0000","last_undegraded":"2024-02-10T08:41:05.017416+0000","last_fullsized":"2024-02-10T08:41:05.017416+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.955272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.955272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":252,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018732+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:41:04.018732+0000","last_peered":"2024-02-10T08:41:04.018732+0000","last_clean":"2024-02-10T08:41:04.018732+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:41:04.018732+0000","last_undegraded":"2024-02-10T08:41:04.018732+0000","last_fullsized":"2024-02-10T08:41:04.018732+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":282,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021011+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:41:04.021011+0000","last_peered":"2024-02-10T08:41:04.021011+0000","last_clean":"2024-02-10T08:41:04.021011+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:41:04.021011+0000","last_undegraded":"2024-02-10T08:41:04.021011+0000","last_fullsized":"2024-02-10T08:41:04.021011+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":363,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015375+0000","last_change":"2024-02-10T08:37:41.943952+0000","last_active":"2024-02-10T08:41:05.015375+0000","last_peered":"2024-02-10T08:41:05.015375+0000","last_clean":"2024-02-10T08:41:05.015375+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:41:05.015375+0000","last_undegraded":"2024-02-10T08:41:05.015375+0000","last_fullsized":"2024-02-10T08:41:05.015375+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:41.943889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:37:41.943889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":365,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020990+0000","last_change":"2024-02-10T08:38:32.886182+0000","last_active":"2024-02-10T08:41:04.020990+0000","last_peered":"2024-02-10T08:41:04.020990+0000","last_clean":"2024-02-10T08:41:04.020990+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:41:04.020990+0000","last_undegraded":"2024-02-10T08:41:04.020990+0000","last_fullsized":"2024-02-10T08:41:04.020990+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:38:32.886090+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.886090+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":367,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023291+0000","last_change":"2024-02-10T08:40:38.667994+0000","last_active":"2024-02-10T08:41:04.023291+0000","last_peered":"2024-02-10T08:41:04.023291+0000","last_clean":"2024-02-10T08:41:04.023291+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:41:04.023291+0000","last_undegraded":"2024-02-10T08:41:04.023291+0000","last_fullsized":"2024-02-10T08:41:04.023291+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:40:38.667923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:38.667923+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020997+0000","last_change":"2024-02-10T08:38:22.407565+0000","last_active":"2024-02-10T08:41:04.020997+0000","last_peered":"2024-02-10T08:41:04.020997+0000","last_clean":"2024-02-10T08:41:04.020997+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:41:04.020997+0000","last_undegraded":"2024-02-10T08:41:04.020997+0000","last_fullsized":"2024-02-10T08:41:04.020997+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:22.407476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:22.407476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":283,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016085+0000","last_change":"2024-02-10T08:37:43.917341+0000","last_active":"2024-02-10T08:41:05.016085+0000","last_peered":"2024-02-10T08:41:05.016085+0000","last_clean":"2024-02-10T08:41:05.016085+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:41:05.016085+0000","last_undegraded":"2024-02-10T08:41:05.016085+0000","last_fullsized":"2024-02-10T08:41:05.016085+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:43.917281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:37:43.917281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":318,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021235+0000","last_change":"2024-02-10T08:37:48.469852+0000","last_active":"2024-02-10T08:41:04.021235+0000","last_peered":"2024-02-10T08:41:04.021235+0000","last_clean":"2024-02-10T08:41:04.021235+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:41:04.021235+0000","last_undegraded":"2024-02-10T08:41:04.021235+0000","last_fullsized":"2024-02-10T08:41:04.021235+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:37:48.469781+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:35:33.677377+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.469781+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":349,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015293+0000","last_change":"2024-02-10T08:37:44.870644+0000","last_active":"2024-02-10T08:41:05.015293+0000","last_peered":"2024-02-10T08:41:05.015293+0000","last_clean":"2024-02-10T08:41:05.015293+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:41:05.015293+0000","last_undegraded":"2024-02-10T08:41:05.015293+0000","last_fullsized":"2024-02-10T08:41:05.015293+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:37:44.870571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.870571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":334,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019285+0000","last_change":"2024-02-10T08:40:37.417370+0000","last_active":"2024-02-10T08:41:05.019285+0000","last_peered":"2024-02-10T08:41:05.019285+0000","last_clean":"2024-02-10T08:41:05.019285+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:41:05.019285+0000","last_undegraded":"2024-02-10T08:41:05.019285+0000","last_fullsized":"2024-02-10T08:41:05.019285+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:37.417303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.417303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":282,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021075+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:41:04.021075+0000","last_peered":"2024-02-10T08:41:04.021075+0000","last_clean":"2024-02-10T08:41:04.021075+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:41:04.021075+0000","last_undegraded":"2024-02-10T08:41:04.021075+0000","last_fullsized":"2024-02-10T08:41:04.021075+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":345,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023844+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:41:04.023844+0000","last_peered":"2024-02-10T08:41:04.023844+0000","last_clean":"2024-02-10T08:41:04.023844+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:41:04.023844+0000","last_undegraded":"2024-02-10T08:41:04.023844+0000","last_fullsized":"2024-02-10T08:41:04.023844+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013035+0000","last_change":"2024-02-10T08:40:43.494246+0000","last_active":"2024-02-10T08:41:04.013035+0000","last_peered":"2024-02-10T08:41:04.013035+0000","last_clean":"2024-02-10T08:41:04.013035+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:41:04.013035+0000","last_undegraded":"2024-02-10T08:41:04.013035+0000","last_fullsized":"2024-02-10T08:41:04.013035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:43.494171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:43.494171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016332+0000","last_change":"2024-02-10T08:40:36.384024+0000","last_active":"2024-02-10T08:41:04.016332+0000","last_peered":"2024-02-10T08:41:04.016332+0000","last_clean":"2024-02-10T08:41:04.016332+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:41:04.016332+0000","last_undegraded":"2024-02-10T08:41:04.016332+0000","last_fullsized":"2024-02-10T08:41:04.016332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.383958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.383958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":281,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.023359+0000","last_change":"2024-02-10T08:38:32.745050+0000","last_active":"2024-02-10T08:41:05.023359+0000","last_peered":"2024-02-10T08:41:05.023359+0000","last_clean":"2024-02-10T08:41:05.023359+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:41:05.023359+0000","last_undegraded":"2024-02-10T08:41:05.023359+0000","last_fullsized":"2024-02-10T08:41:05.023359+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.744972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.744972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":351,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021037+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:41:04.021037+0000","last_peered":"2024-02-10T08:41:04.021037+0000","last_clean":"2024-02-10T08:41:04.021037+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:41:04.021037+0000","last_undegraded":"2024-02-10T08:41:04.021037+0000","last_fullsized":"2024-02-10T08:41:04.021037+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017526+0000","last_change":"2024-02-10T08:40:35.021264+0000","last_active":"2024-02-10T08:41:05.017526+0000","last_peered":"2024-02-10T08:41:05.017526+0000","last_clean":"2024-02-10T08:41:05.017526+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:41:05.017526+0000","last_undegraded":"2024-02-10T08:41:05.017526+0000","last_fullsized":"2024-02-10T08:41:05.017526+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.021169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.021169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":448,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019510+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:41:05.019510+0000","last_peered":"2024-02-10T08:41:05.019510+0000","last_clean":"2024-02-10T08:41:05.019510+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:41:05.019510+0000","last_undegraded":"2024-02-10T08:41:05.019510+0000","last_fullsized":"2024-02-10T08:41:05.019510+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":251,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.017063+0000","last_change":"2024-02-10T08:39:43.632113+0000","last_active":"2024-02-10T08:41:04.017063+0000","last_peered":"2024-02-10T08:41:04.017063+0000","last_clean":"2024-02-10T08:41:04.017063+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:41:04.017063+0000","last_undegraded":"2024-02-10T08:41:04.017063+0000","last_fullsized":"2024-02-10T08:41:04.017063+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:43.632048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.632048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":250,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.012820+0000","last_change":"2024-02-10T08:39:01.740757+0000","last_active":"2024-02-10T08:41:04.012820+0000","last_peered":"2024-02-10T08:41:04.012820+0000","last_clean":"2024-02-10T08:41:04.012820+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:41:04.012820+0000","last_undegraded":"2024-02-10T08:41:04.012820+0000","last_fullsized":"2024-02-10T08:41:04.012820+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:01.740686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:01.740686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":352,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019412+0000","last_change":"2024-02-10T08:39:46.641751+0000","last_active":"2024-02-10T08:41:05.019412+0000","last_peered":"2024-02-10T08:41:05.019412+0000","last_clean":"2024-02-10T08:41:05.019412+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:41:05.019412+0000","last_undegraded":"2024-02-10T08:41:05.019412+0000","last_fullsized":"2024-02-10T08:41:05.019412+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:39:46.641683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.641683+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":252,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021376+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:41:04.021376+0000","last_peered":"2024-02-10T08:41:04.021376+0000","last_clean":"2024-02-10T08:41:04.021376+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:41:04.021376+0000","last_undegraded":"2024-02-10T08:41:04.021376+0000","last_fullsized":"2024-02-10T08:41:04.021376+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":435,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021277+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:41:04.021277+0000","last_peered":"2024-02-10T08:41:04.021277+0000","last_clean":"2024-02-10T08:41:04.021277+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:41:04.021277+0000","last_undegraded":"2024-02-10T08:41:04.021277+0000","last_fullsized":"2024-02-10T08:41:04.021277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023410+0000","last_change":"2024-02-10T08:40:41.538200+0000","last_active":"2024-02-10T08:41:04.023410+0000","last_peered":"2024-02-10T08:41:04.023410+0000","last_clean":"2024-02-10T08:41:04.023410+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:41:04.023410+0000","last_undegraded":"2024-02-10T08:41:04.023410+0000","last_fullsized":"2024-02-10T08:41:04.023410+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:41.538139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.538139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":296,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015212+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:41:05.015212+0000","last_peered":"2024-02-10T08:41:05.015212+0000","last_clean":"2024-02-10T08:41:05.015212+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:41:05.015212+0000","last_undegraded":"2024-02-10T08:41:05.015212+0000","last_fullsized":"2024-02-10T08:41:05.015212+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":360,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016790+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:41:05.016790+0000","last_peered":"2024-02-10T08:41:05.016790+0000","last_clean":"2024-02-10T08:41:05.016790+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:41:05.016790+0000","last_undegraded":"2024-02-10T08:41:05.016790+0000","last_fullsized":"2024-02-10T08:41:05.016790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.012821+0000","last_change":"2024-02-10T08:39:46.475770+0000","last_active":"2024-02-10T08:41:04.012821+0000","last_peered":"2024-02-10T08:41:04.012821+0000","last_clean":"2024-02-10T08:41:04.012821+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:41:04.012821+0000","last_undegraded":"2024-02-10T08:41:04.012821+0000","last_fullsized":"2024-02-10T08:41:04.012821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.475678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.475678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":343,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020912+0000","last_change":"2024-02-10T08:38:13.463715+0000","last_active":"2024-02-10T08:41:04.020912+0000","last_peered":"2024-02-10T08:41:04.020912+0000","last_clean":"2024-02-10T08:41:04.020912+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:41:04.020912+0000","last_undegraded":"2024-02-10T08:41:04.020912+0000","last_fullsized":"2024-02-10T08:41:04.020912+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:13.463644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:38:13.463644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":251,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019269+0000","last_change":"2024-02-10T08:38:20.592140+0000","last_active":"2024-02-10T08:41:05.019269+0000","last_peered":"2024-02-10T08:41:05.019269+0000","last_clean":"2024-02-10T08:41:05.019269+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:41:05.019269+0000","last_undegraded":"2024-02-10T08:41:05.019269+0000","last_fullsized":"2024-02-10T08:41:05.019269+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:20.592065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.592065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":289,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.014141+0000","last_change":"2024-02-10T08:39:12.687989+0000","last_active":"2024-02-10T08:41:04.014141+0000","last_peered":"2024-02-10T08:41:04.014141+0000","last_clean":"2024-02-10T08:41:04.014141+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:41:04.014141+0000","last_undegraded":"2024-02-10T08:41:04.014141+0000","last_fullsized":"2024-02-10T08:41:04.014141+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:12.687918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:12.687918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":356,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.682672+0000","last_change":"2024-02-10T08:41:05.682672+0000","last_active":"2024-02-10T08:41:05.682672+0000","last_peered":"2024-02-10T08:41:05.682672+0000","last_clean":"2024-02-10T08:41:05.682672+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:41:05.682672+0000","last_undegraded":"2024-02-10T08:41:05.682672+0000","last_fullsized":"2024-02-10T08:41:05.682672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:05.682577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.682577+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":351,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.017025+0000","last_change":"2024-02-10T08:39:04.936255+0000","last_active":"2024-02-10T08:41:05.017025+0000","last_peered":"2024-02-10T08:41:05.017025+0000","last_clean":"2024-02-10T08:41:05.017025+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:41:05.017025+0000","last_undegraded":"2024-02-10T08:41:05.017025+0000","last_fullsized":"2024-02-10T08:41:05.017025+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:04.936174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:04.936174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":336,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016903+0000","last_change":"2024-02-10T08:40:36.354179+0000","last_active":"2024-02-10T08:41:05.016903+0000","last_peered":"2024-02-10T08:41:05.016903+0000","last_clean":"2024-02-10T08:41:05.016903+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:41:05.016903+0000","last_undegraded":"2024-02-10T08:41:05.016903+0000","last_fullsized":"2024-02-10T08:41:05.016903+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.354073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":341,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018796+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:41:04.018796+0000","last_peered":"2024-02-10T08:41:04.018796+0000","last_clean":"2024-02-10T08:41:04.018796+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:41:04.018796+0000","last_undegraded":"2024-02-10T08:41:04.018796+0000","last_fullsized":"2024-02-10T08:41:04.018796+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":291,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019723+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:41:04.019723+0000","last_peered":"2024-02-10T08:41:04.019723+0000","last_clean":"2024-02-10T08:41:04.019723+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:41:04.019723+0000","last_undegraded":"2024-02-10T08:41:04.019723+0000","last_fullsized":"2024-02-10T08:41:04.019723+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018648+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:41:04.018648+0000","last_peered":"2024-02-10T08:41:04.018648+0000","last_clean":"2024-02-10T08:41:04.018648+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:41:04.018648+0000","last_undegraded":"2024-02-10T08:41:04.018648+0000","last_fullsized":"2024-02-10T08:41:04.018648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.909509+0000","last_change":"2024-02-10T08:41:05.909459+0000","last_active":"2024-02-10T08:41:05.909509+0000","last_peered":"2024-02-10T08:41:05.909509+0000","last_clean":"2024-02-10T08:41:05.909509+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:41:05.909509+0000","last_undegraded":"2024-02-10T08:41:05.909509+0000","last_fullsized":"2024-02-10T08:41:05.909509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.909358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":335,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.012973+0000","last_change":"2024-02-10T08:39:41.555411+0000","last_active":"2024-02-10T08:41:04.012973+0000","last_peered":"2024-02-10T08:41:04.012973+0000","last_clean":"2024-02-10T08:41:04.012973+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:41:04.012973+0000","last_undegraded":"2024-02-10T08:41:04.012973+0000","last_fullsized":"2024-02-10T08:41:04.012973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.555339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.555339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015568+0000","last_change":"2024-02-10T08:41:03.331622+0000","last_active":"2024-02-10T08:41:04.015568+0000","last_peered":"2024-02-10T08:41:04.015568+0000","last_clean":"2024-02-10T08:41:04.015568+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:41:04.015568+0000","last_undegraded":"2024-02-10T08:41:04.015568+0000","last_fullsized":"2024-02-10T08:41:04.015568+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":282,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019719+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:41:04.019719+0000","last_peered":"2024-02-10T08:41:04.019719+0000","last_clean":"2024-02-10T08:41:04.019719+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:41:04.019719+0000","last_undegraded":"2024-02-10T08:41:04.019719+0000","last_fullsized":"2024-02-10T08:41:04.019719+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":355,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019076+0000","last_change":"2024-02-10T08:41:03.770782+0000","last_active":"2024-02-10T08:41:04.019076+0000","last_peered":"2024-02-10T08:41:04.019076+0000","last_clean":"2024-02-10T08:41:04.019076+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:41:04.019076+0000","last_undegraded":"2024-02-10T08:41:04.019076+0000","last_fullsized":"2024-02-10T08:41:04.019076+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015043+0000","last_change":"2024-02-10T08:39:49.529787+0000","last_active":"2024-02-10T08:41:04.015043+0000","last_peered":"2024-02-10T08:41:04.015043+0000","last_clean":"2024-02-10T08:41:04.015043+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:41:04.015043+0000","last_undegraded":"2024-02-10T08:41:04.015043+0000","last_fullsized":"2024-02-10T08:41:04.015043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:49.529724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.529724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":334,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.017363+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:41:04.017363+0000","last_peered":"2024-02-10T08:41:04.017363+0000","last_clean":"2024-02-10T08:41:04.017363+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:41:04.017363+0000","last_undegraded":"2024-02-10T08:41:04.017363+0000","last_fullsized":"2024-02-10T08:41:04.017363+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":341,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021456+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:41:04.021456+0000","last_peered":"2024-02-10T08:41:04.021456+0000","last_clean":"2024-02-10T08:41:04.021456+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:41:04.021456+0000","last_undegraded":"2024-02-10T08:41:04.021456+0000","last_fullsized":"2024-02-10T08:41:04.021456+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":250,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.014015+0000","last_change":"2024-02-10T08:38:59.747834+0000","last_active":"2024-02-10T08:41:04.014015+0000","last_peered":"2024-02-10T08:41:04.014015+0000","last_clean":"2024-02-10T08:41:04.014015+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:41:04.014015+0000","last_undegraded":"2024-02-10T08:41:04.014015+0000","last_fullsized":"2024-02-10T08:41:04.014015+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:59.747762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:59.747762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":290,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.017275+0000","last_change":"2024-02-10T08:39:41.691153+0000","last_active":"2024-02-10T08:41:04.017275+0000","last_peered":"2024-02-10T08:41:04.017275+0000","last_clean":"2024-02-10T08:41:04.017275+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:41:04.017275+0000","last_undegraded":"2024-02-10T08:41:04.017275+0000","last_fullsized":"2024-02-10T08:41:04.017275+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:39:41.691086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.691086+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":352,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018271+0000","last_change":"2024-02-10T08:40:27.813479+0000","last_active":"2024-02-10T08:41:05.018271+0000","last_peered":"2024-02-10T08:41:05.018271+0000","last_clean":"2024-02-10T08:41:05.018271+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:41:05.018271+0000","last_undegraded":"2024-02-10T08:41:05.018271+0000","last_fullsized":"2024-02-10T08:41:05.018271+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.813396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.813396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016742+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:41:04.016742+0000","last_peered":"2024-02-10T08:41:04.016742+0000","last_clean":"2024-02-10T08:41:04.016742+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:41:04.016742+0000","last_undegraded":"2024-02-10T08:41:04.016742+0000","last_fullsized":"2024-02-10T08:41:04.016742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":283,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.018714+0000","last_change":"2024-02-10T08:38:14.837278+0000","last_active":"2024-02-10T08:41:05.018714+0000","last_peered":"2024-02-10T08:41:05.018714+0000","last_clean":"2024-02-10T08:41:05.018714+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:41:05.018714+0000","last_undegraded":"2024-02-10T08:41:05.018714+0000","last_fullsized":"2024-02-10T08:41:05.018714+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:14.837154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:14.837154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":359,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015199+0000","last_change":"2024-02-10T08:39:47.473081+0000","last_active":"2024-02-10T08:41:04.015199+0000","last_peered":"2024-02-10T08:41:04.015199+0000","last_clean":"2024-02-10T08:41:04.015199+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:41:04.015199+0000","last_undegraded":"2024-02-10T08:41:04.015199+0000","last_fullsized":"2024-02-10T08:41:04.015199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:39:47.473007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.473007+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.018893+0000","last_change":"2024-02-10T08:38:31.436092+0000","last_active":"2024-02-10T08:41:04.018893+0000","last_peered":"2024-02-10T08:41:04.018893+0000","last_clean":"2024-02-10T08:41:04.018893+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:41:04.018893+0000","last_undegraded":"2024-02-10T08:41:04.018893+0000","last_fullsized":"2024-02-10T08:41:04.018893+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:31.435994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:38:31.435994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":334,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.016861+0000","last_change":"2024-02-10T08:38:58.917062+0000","last_active":"2024-02-10T08:41:05.016861+0000","last_peered":"2024-02-10T08:41:05.016861+0000","last_clean":"2024-02-10T08:41:05.016861+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:41:05.016861+0000","last_undegraded":"2024-02-10T08:41:05.016861+0000","last_fullsized":"2024-02-10T08:41:05.016861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:58.916996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:58.916996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":283,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.015358+0000","last_change":"2024-02-10T08:38:28.906569+0000","last_active":"2024-02-10T08:41:05.015358+0000","last_peered":"2024-02-10T08:41:05.015358+0000","last_clean":"2024-02-10T08:41:05.015358+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:41:05.015358+0000","last_undegraded":"2024-02-10T08:41:05.015358+0000","last_fullsized":"2024-02-10T08:41:05.015358+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:38:28.906474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":370,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019744+0000","last_change":"2024-02-10T08:37:48.902052+0000","last_active":"2024-02-10T08:41:05.019744+0000","last_peered":"2024-02-10T08:41:05.019744+0000","last_clean":"2024-02-10T08:41:05.019744+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:41:05.019744+0000","last_undegraded":"2024-02-10T08:41:05.019744+0000","last_fullsized":"2024-02-10T08:41:05.019744+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:37:48.901981+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013912+0000","last_change":"2024-02-10T08:38:32.342305+0000","last_active":"2024-02-10T08:41:04.013912+0000","last_peered":"2024-02-10T08:41:04.013912+0000","last_clean":"2024-02-10T08:41:04.013912+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:41:04.013912+0000","last_undegraded":"2024-02-10T08:41:04.013912+0000","last_fullsized":"2024-02-10T08:41:04.013912+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:32.342200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:32.342200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023060+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:41:04.023060+0000","last_peered":"2024-02-10T08:41:04.023060+0000","last_clean":"2024-02-10T08:41:04.023060+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:41:04.023060+0000","last_undegraded":"2024-02-10T08:41:04.023060+0000","last_fullsized":"2024-02-10T08:41:04.023060+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":255,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.012734+0000","last_change":"2024-02-10T08:40:39.439343+0000","last_active":"2024-02-10T08:41:04.012734+0000","last_peered":"2024-02-10T08:41:04.012734+0000","last_clean":"2024-02-10T08:41:04.012734+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:41:04.012734+0000","last_undegraded":"2024-02-10T08:41:04.012734+0000","last_fullsized":"2024-02-10T08:41:04.012734+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.439244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.439244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":380,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.019646+0000","last_change":"2024-02-10T08:40:42.021634+0000","last_active":"2024-02-10T08:41:04.019646+0000","last_peered":"2024-02-10T08:41:04.019646+0000","last_clean":"2024-02-10T08:41:04.019646+0000","last_became_active":"2024-02-10T08:40:42.014594+0000","last_became_peered":"2024-02-10T08:40:42.014594+0000","last_unstale":"2024-02-10T08:41:04.019646+0000","last_undegraded":"2024-02-10T08:41:04.019646+0000","last_fullsized":"2024-02-10T08:41:04.019646+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:38:19.453060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:19.453060+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":338,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020821+0000","last_change":"2024-02-10T08:38:16.426049+0000","last_active":"2024-02-10T08:41:04.020821+0000","last_peered":"2024-02-10T08:41:04.020821+0000","last_clean":"2024-02-10T08:41:04.020821+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:41:04.020821+0000","last_undegraded":"2024-02-10T08:41:04.020821+0000","last_fullsized":"2024-02-10T08:41:04.020821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:16.425945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:38:16.425945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.023914+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:41:04.023914+0000","last_peered":"2024-02-10T08:41:04.023914+0000","last_clean":"2024-02-10T08:41:04.023914+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:41:04.023914+0000","last_undegraded":"2024-02-10T08:41:04.023914+0000","last_fullsized":"2024-02-10T08:41:04.023914+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":283,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019708+0000","last_change":"2024-02-10T08:39:46.928020+0000","last_active":"2024-02-10T08:41:05.019708+0000","last_peered":"2024-02-10T08:41:05.019708+0000","last_clean":"2024-02-10T08:41:05.019708+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:41:05.019708+0000","last_undegraded":"2024-02-10T08:41:05.019708+0000","last_fullsized":"2024-02-10T08:41:05.019708+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:46.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:46.927953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":372,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021334+0000","last_change":"2024-02-10T08:38:26.871174+0000","last_active":"2024-02-10T08:41:04.021334+0000","last_peered":"2024-02-10T08:41:04.021334+0000","last_clean":"2024-02-10T08:41:04.021334+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:41:04.021334+0000","last_undegraded":"2024-02-10T08:41:04.021334+0000","last_fullsized":"2024-02-10T08:41:04.021334+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:38:26.871100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:26.871100+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":338,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016327+0000","last_change":"2024-02-10T08:39:02.250833+0000","last_active":"2024-02-10T08:41:04.016327+0000","last_peered":"2024-02-10T08:41:04.016327+0000","last_clean":"2024-02-10T08:41:04.016327+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:41:04.016327+0000","last_undegraded":"2024-02-10T08:41:04.016327+0000","last_fullsized":"2024-02-10T08:41:04.016327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:02.250719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:02.250719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":253,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.013890+0000","last_change":"2024-02-10T08:38:27.952601+0000","last_active":"2024-02-10T08:41:05.013890+0000","last_peered":"2024-02-10T08:41:05.013890+0000","last_clean":"2024-02-10T08:41:05.013890+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:41:05.013890+0000","last_undegraded":"2024-02-10T08:41:05.013890+0000","last_fullsized":"2024-02-10T08:41:05.013890+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:27.952518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:27.952518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.019854+0000","last_change":"2024-02-10T08:40:22.359771+0000","last_active":"2024-02-10T08:41:05.019854+0000","last_peered":"2024-02-10T08:41:05.019854+0000","last_clean":"2024-02-10T08:41:05.019854+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:41:05.019854+0000","last_undegraded":"2024-02-10T08:41:05.019854+0000","last_fullsized":"2024-02-10T08:41:05.019854+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.359703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.359703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":349,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021460+0000","last_change":"2024-02-10T08:39:11.790943+0000","last_active":"2024-02-10T08:41:04.021460+0000","last_peered":"2024-02-10T08:41:04.021460+0000","last_clean":"2024-02-10T08:41:04.021460+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:41:04.021460+0000","last_undegraded":"2024-02-10T08:41:04.021460+0000","last_fullsized":"2024-02-10T08:41:04.021460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:39:11.790866+0000","last_clean_scrub_stamp":"2024-02-10T08:39:11.790866+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.016408+0000","last_change":"2024-02-10T08:38:30.389134+0000","last_active":"2024-02-10T08:41:04.016408+0000","last_peered":"2024-02-10T08:41:04.016408+0000","last_clean":"2024-02-10T08:41:04.016408+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:41:04.016408+0000","last_undegraded":"2024-02-10T08:41:04.016408+0000","last_fullsized":"2024-02-10T08:41:04.016408+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.389076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.027223+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:41:05.027223+0000","last_peered":"2024-02-10T08:41:05.027223+0000","last_clean":"2024-02-10T08:41:05.027223+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:41:05.027223+0000","last_undegraded":"2024-02-10T08:41:05.027223+0000","last_fullsized":"2024-02-10T08:41:05.027223+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":265,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.021961+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:41:05.021961+0000","last_peered":"2024-02-10T08:41:05.021961+0000","last_clean":"2024-02-10T08:41:05.021961+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:41:05.021961+0000","last_undegraded":"2024-02-10T08:41:05.021961+0000","last_fullsized":"2024-02-10T08:41:05.021961+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":372,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020638+0000","last_change":"2024-02-10T08:40:37.664677+0000","last_active":"2024-02-10T08:41:04.020638+0000","last_peered":"2024-02-10T08:41:04.020638+0000","last_clean":"2024-02-10T08:41:04.020638+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:41:04.020638+0000","last_undegraded":"2024-02-10T08:41:04.020638+0000","last_fullsized":"2024-02-10T08:41:04.020638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:40:37.664586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.664586+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":347,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021143+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:41:04.021143+0000","last_peered":"2024-02-10T08:41:04.021143+0000","last_clean":"2024-02-10T08:41:04.021143+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:41:04.021143+0000","last_undegraded":"2024-02-10T08:41:04.021143+0000","last_fullsized":"2024-02-10T08:41:04.021143+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":333,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.021244+0000","last_change":"2024-02-10T08:39:05.790083+0000","last_active":"2024-02-10T08:41:04.021244+0000","last_peered":"2024-02-10T08:41:04.021244+0000","last_clean":"2024-02-10T08:41:04.021244+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:41:04.021244+0000","last_undegraded":"2024-02-10T08:41:04.021244+0000","last_fullsized":"2024-02-10T08:41:04.021244+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:05.790005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:05.790005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":290,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-02-10T08:41:05.021238+0000","last_change":"2024-02-10T08:38:30.695748+0000","last_active":"2024-02-10T08:41:05.021238+0000","last_peered":"2024-02-10T08:41:05.021238+0000","last_clean":"2024-02-10T08:41:05.021238+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:41:05.021238+0000","last_undegraded":"2024-02-10T08:41:05.021238+0000","last_fullsized":"2024-02-10T08:41:05.021238+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:30.695684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:38:30.695684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":381,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.024043+0000","last_change":"2024-02-10T08:37:44.533293+0000","last_active":"2024-02-10T08:41:04.024043+0000","last_peered":"2024-02-10T08:41:04.024043+0000","last_clean":"2024-02-10T08:41:04.024043+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:41:04.024043+0000","last_undegraded":"2024-02-10T08:41:04.024043+0000","last_fullsized":"2024-02-10T08:41:04.024043+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:37:44.533195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:37:44.533195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":256,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.015914+0000","last_change":"2024-02-10T08:40:42.336046+0000","last_active":"2024-02-10T08:41:04.015914+0000","last_peered":"2024-02-10T08:41:04.015914+0000","last_clean":"2024-02-10T08:41:04.015914+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:41:04.015914+0000","last_undegraded":"2024-02-10T08:41:04.015914+0000","last_fullsized":"2024-02-10T08:41:04.015914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.335958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.335958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":332,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.020607+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:41:04.020607+0000","last_peered":"2024-02-10T08:41:04.020607+0000","last_clean":"2024-02-10T08:41:04.020607+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:41:04.020607+0000","last_undegraded":"2024-02-10T08:41:04.020607+0000","last_fullsized":"2024-02-10T08:41:04.020607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":285,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-02-10T08:41:04.013608+0000","last_change":"2024-02-10T08:40:36.368467+0000","last_active":"2024-02-10T08:41:04.013608+0000","last_peered":"2024-02-10T08:41:04.013608+0000","last_clean":"2024-02-10T08:41:04.013608+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:41:04.013608+0000","last_undegraded":"2024-02-10T08:41:04.013608+0000","last_fullsized":"2024-02-10T08:41:04.013608+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.368365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.368365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":204,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001420,"num_objects":148,"num_object_clones":2,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":102,"num_read_kb":7,"num_write":573,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13803520,"data_stored":12004268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1830,"ondisk_log_size":1830,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299488,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308024,"kb_used_data":2608,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437128,"statfs":{"total":95995035648,"available":95679619072,"internally_reserved":0,"allocated":2670592,"data_stored":1013797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.76000000000000001,"15min":0.60199999999999998},"min":{"1min":0.36799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.66500000000000004,"15min":0.56799999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66200000000000003,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.61399999999999999,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68799999999999994,"5min":7.593,"15min":7.593},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.64000000000000001,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.58499999999999996,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.63500000000000001,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.84899999999999998,"15min":0.64900000000000002},"min":{"1min":0.308,"5min":0.251,"15min":0.251},"max":{"1min":0.75,"5min":19.369,"15min":19.369},"last":0.75700000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.68500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.83999999999999997,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33100000000000002,"15min":0.27900000000000003},"max":{"1min":0.72599999999999998,"5min":22.102,"15min":22.102},"last":0.72699999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.48299999999999998,"15min":0.44700000000000001},"min":{"1min":0.247,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.495,"5min":8.827,"15min":8.827},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.55600000000000005,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67700000000000005,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.65700000000000003,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64000000000000001,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65200000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.68799999999999994,"15min":0.55900000000000005},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60399999999999998,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.502},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.35099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71499999999999997,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.58799999999999997}]}]},{"osd":6,"up_from":44,"seq":188978561126,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303644,"kb_used_data":6420,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441508,"statfs":{"total":95995035648,"available":95684104192,"internally_reserved":0,"allocated":6574080,"data_stored":5013556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.625,"15min":0.51700000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.19800000000000001},"max":{"1min":0.65900000000000003,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64900000000000002,"15min":0.53200000000000003},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.66700000000000004,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.51200000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72199999999999998,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.626},{"interface":"front","average":{"1min":0.497,"5min":0.57399999999999995,"15min":0.497},"min":{"1min":0.312,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.69599999999999995,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57899999999999996,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":4.798,"15min":4.798},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.58799999999999997,"15min":0.53000000000000003},"min":{"1min":0.33400000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.59999999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.82099999999999995,"15min":0.59399999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.70499999999999996,"5min":16.823,"15min":16.823},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.51500000000000001},"min":{"1min":0.318,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.75800000000000001,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.43099999999999999,"15min":0.39800000000000002},"min":{"1min":0.29399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52500000000000002,"5min":3.657,"15min":3.657},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.30499999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67700000000000005,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46600000000000003,"15min":0.40799999999999997},"min":{"1min":0.33200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.441},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.42999999999999999},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.45900000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.501,"15min":0.47199999999999998},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.63500000000000001,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50600000000000001,"15min":0.49199999999999999},"min":{"1min":0.40200000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.47699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299800,"kb_used_data":2576,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445352,"statfs":{"total":95995035648,"available":95688040448,"internally_reserved":0,"allocated":2637824,"data_stored":1014810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.56999999999999995,"15min":0.40899999999999997},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.436,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.64800000000000002,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.41499999999999998,"5min":16.166,"15min":16.166},"last":0.29899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":1.101,"15min":0.61499999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54200000000000004,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.86399999999999999,"15min":0.53100000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42999999999999999,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.746,"15min":0.52600000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52500000000000002,"5min":24.773,"15min":24.773},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.71499999999999997,"15min":0.53100000000000003},"min":{"1min":0.26000000000000001,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.56599999999999995,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.75700000000000001,"15min":0.61899999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.67600000000000005,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.67600000000000005,"15min":0.60799999999999998},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:40:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67300000000000004,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.33200000000000002,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.77500000000000002,"15min":0.66700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.69099999999999995,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.68000000000000005,"15min":0.60799999999999998},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.71599999999999997,"5min":9.532,"15min":9.532},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.79600000000000004,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.74299999999999999,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.72399999999999998,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.83799999999999997,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.754,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.63400000000000001}]}]},{"osd":0,"up_from":0,"seq":34359738499,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316188,"kb_used_data":2580,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93428964,"statfs":{"total":95995035648,"available":95671259136,"internally_reserved":0,"allocated":2641920,"data_stored":1013551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.41099999999999998},"min":{"1min":0.22800000000000001,"5min":0.193,"15min":0.187},"max":{"1min":3.0640000000000001,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.67300000000000004,"15min":0.46400000000000002},"min":{"1min":0.27100000000000002,"5min":0.192,"15min":0.192},"max":{"1min":1.4950000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.70999999999999996,"15min":0.47699999999999998},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.63400000000000001,"5min":12.512,"15min":12.512},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64900000000000002,"15min":0.47099999999999997},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":3.0960000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.374}]},{"osd":3,"last update":"Sat Feb 10 08:40:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.52400000000000002,"15min":0.44800000000000001},"min":{"1min":0.28000000000000003,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":0.55800000000000005,"5min":4.617,"15min":4.617},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.63100000000000001,"15min":0.49199999999999999},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.2930000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.40799999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.878,"15min":0.68000000000000005},"min":{"1min":0.32800000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.7370000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.437},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.77700000000000002,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":0.79000000000000004,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.42899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.86099999999999999,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.764,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.72699999999999998,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.78900000000000003,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:41:00 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.81899999999999995,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97599999999999998,"5min":9.266,"15min":9.266},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.73799999999999999,"15min":0.64900000000000002},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91300000000000003,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.81999999999999995,"15min":0.69399999999999995},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.708,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.495},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.77200000000000002,"15min":0.66500000000000004},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.54100000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182517,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320096,"kb_used_data":6488,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425056,"statfs":{"total":95995035648,"available":95667257344,"internally_reserved":0,"allocated":6643712,"data_stored":5013694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.81899999999999995,"15min":0.51200000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.55200000000000005,"5min":14.744,"15min":14.744},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.90900000000000003,"15min":0.55300000000000005},"min":{"1min":0.252,"5min":0.188,"15min":0.188},"max":{"1min":3.5579999999999998,"5min":11.808,"15min":11.808},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.91100000000000003,"15min":0.57199999999999995},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":9.3100000000000005,"5min":16.634,"15min":16.634},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.872,"15min":0.56999999999999995},"min":{"1min":0.26000000000000001,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":5.4370000000000003,"5min":12.438000000000001,"15min":12.438000000000001},"last":0.312}]},{"osd":2,"last update":"Sat Feb 10 08:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.89300000000000002,"15min":0.59299999999999997},"min":{"1min":0.28000000000000003,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":3.1549999999999998,"5min":16.536000000000001,"15min":16.536000000000001},"last":0.375},{"interface":"front","average":{"1min":0.35799999999999998,"5min":1.0800000000000001,"15min":0.63},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.495,"5min":23.294,"15min":23.294},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.84799999999999998,"15min":0.68400000000000005},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":0.70799999999999996,"5min":12.371,"15min":12.371},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.0089999999999999,"15min":0.71699999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":14.821999999999999,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.6040000000000001,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.625},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.80500000000000005,"15min":0.65400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.9909999999999997,"5min":11.448,"15min":11.448},"last":0.67900000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.0089999999999999,"15min":0.69599999999999995},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.48,"5min":16.091000000000001,"15min":16.091000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.94399999999999995,"15min":0.68500000000000005},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.2469999999999999,"5min":16.475000000000001,"15min":16.475000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.876,"15min":0.65300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.079,"15min":0.72599999999999998},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.52,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.66900000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986287,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308000,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437152,"statfs":{"total":95995035648,"available":95679643648,"internally_reserved":0,"allocated":2646016,"data_stored":1013812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.84799999999999998,"15min":0.64100000000000001},"min":{"1min":0.35899999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84999999999999998,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.65800000000000003,"15min":0.56100000000000005},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.91800000000000004,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.46999999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.73099999999999998,"15min":0.59199999999999997},"min":{"1min":0.35599999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93799999999999994,"5min":5.774,"15min":5.774},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.73799999999999999,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.432}]},{"osd":2,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.76100000000000001,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.251,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.77600000000000002,"15min":0.61099999999999999},"min":{"1min":0.34599999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":10.01,"15min":10.01},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.91800000000000004,"15min":0.66100000000000003},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.92700000000000005,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.0980000000000001,"15min":0.71999999999999997},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.82099999999999995,"5min":25.27,"15min":25.27},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.495,"15min":0.42099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.64100000000000001,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47299999999999998,"15min":0.41799999999999998},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.66900000000000004,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:40:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.64800000000000002,"15min":0.48499999999999999},"min":{"1min":0.314,"5min":0.247,"15min":0.247},"max":{"1min":0.59599999999999997,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.63700000000000001,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":13.753,"15min":13.753},"last":0.44400000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.623,"15min":0.51100000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.68100000000000005,"15min":0.57899999999999996},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78300000000000003,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.60099999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790059,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307752,"kb_used_data":6432,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437400,"statfs":{"total":95995035648,"available":95679897600,"internally_reserved":0,"allocated":6586368,"data_stored":5013343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.968,"15min":0.996},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67900000000000005,"5min":107.551,"15min":107.551},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.498,"5min":1.843,"15min":0.96199999999999997},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":0.76900000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":1.6930000000000001,"15min":0.93799999999999994},"min":{"1min":0.39300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67100000000000004,"5min":107.205,"15min":107.205},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.359,"15min":0.82799999999999996},"min":{"1min":0.38500000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":1.839,"15min":0.95299999999999996},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.623,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.9219999999999999,"15min":0.995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1379999999999999,"15min":0.71899999999999997},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":2.5169999999999999,"15min":1.206},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77400000000000002,"5min":114.526,"15min":114.526},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":1.151,"15min":0.65700000000000003},"min":{"1min":0.30499999999999999,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.88700000000000001,"15min":0.56299999999999994},"min":{"1min":0.317,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.58999999999999997,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.54700000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.1910000000000001,"15min":0.67300000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64800000000000002,"5min":72.241,"15min":72.241},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.70299999999999996,"15min":0.53900000000000003},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.56899999999999995,"5min":22.469999999999999,"15min":22.469999999999999},"last":0.623}]},{"osd":7,"last update":"Sat Feb 10 08:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.1699999999999999,"15min":0.70999999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.2370000000000001,"15min":0.74099999999999999},"min":{"1min":0.39200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72899999999999998,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4000615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4000491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4227072,"data_stored":4000555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:41:08.259 INFO:tasks.thrashosds.thrasher:Setting 6.1 to [2, 1, 7] 2024-02-10T08:41:08.259 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.1', '2', '1', '7'] 2024-02-10T08:41:08.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 6.1 2 1 7 2024-02-10T08:41:08.276 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:08.295 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:08.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:08.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:08.501 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:08.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:07.172781+0000 mgr.y (mgr.14553) 338 : audit [DBG] from='client.27933 v1:172.21.15.107:0/1247562334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:41:08.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.426197+0000 mgr.y (mgr.14553) 339 : cluster [DBG] pgmap v536: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:08.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.015020+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:41:08.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:08.254446+0000 mon.a (mon.0) 2338 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]': finished 2024-02-10T08:41:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:08.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:08.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:07.172781+0000 mgr.y (mgr.14553) 338 : audit [DBG] from='client.27933 v1:172.21.15.107:0/1247562334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:41:08.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.426197+0000 mgr.y (mgr.14553) 339 : cluster [DBG] pgmap v536: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:08.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.015020+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:41:08.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:08.254446+0000 mon.a (mon.0) 2338 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]': finished 2024-02-10T08:41:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:07.172781+0000 mgr.y (mgr.14553) 338 : audit [DBG] from='client.27933 v1:172.21.15.107:0/1247562334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:41:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.426197+0000 mgr.y (mgr.14553) 339 : cluster [DBG] pgmap v536: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.015020+0000 mon.a (mon.0) 2337 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:41:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:08.254446+0000 mon.a (mon.0) 2338 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-49", "mode": "writeback"}]': finished 2024-02-10T08:41:08.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:08.733 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:08.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:08.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:09.313 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:09.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:09.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.580608+0000 osd.1 (osd.1) 165 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:41:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.584118+0000 osd.1 (osd.1) 166 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:41:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.710239+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:41:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.712074+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.947087+0000 osd.5 (osd.5) 163 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:07.950592+0000 osd.5 (osd.5) 164 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.254523+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:08.294500+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:08.294875+0000 mon.a (mon.0) 2340 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.387825+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.390717+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:09.020180+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:09.020266+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:09.020309+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:09.022375+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:09.022872+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:09.066567+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? v1:172.21.15.107:0/1436844771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:09.553 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:09.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.580608+0000 osd.1 (osd.1) 165 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:41:09.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.584118+0000 osd.1 (osd.1) 166 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:41:09.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.710239+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:41:09.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.712074+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:41:09.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.947087+0000 osd.5 (osd.5) 163 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:07.950592+0000 osd.5 (osd.5) 164 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.254523+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:08.294500+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:08.294875+0000 mon.a (mon.0) 2340 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.387825+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.390717+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:09.020180+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:09.020266+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:09.020309+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:09.022375+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:09.022872+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:09.066567+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? v1:172.21.15.107:0/1436844771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.580608+0000 osd.1 (osd.1) 165 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.584118+0000 osd.1 (osd.1) 166 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.710239+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.712074+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.947087+0000 osd.5 (osd.5) 163 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:07.950592+0000 osd.5 (osd.5) 164 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.254523+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:08.294500+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:08.294875+0000 mon.a (mon.0) 2340 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.387825+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.390717+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:09.020180+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi107-112411-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:09.020266+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:41:09.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:09.020309+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-10T08:41:09.558 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:09.022375+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.558 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:09.022872+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:41:09.558 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:09.066567+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? v1:172.21.15.107:0/1436844771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:09.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:09.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.062+0000 7f6d92345700 1 -- 172.21.15.107:0/3243937638 >> v1:172.21.15.107:6790/0 conn(0x7f6d8c1007f0 legacy=0x7f6d8c102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.062+0000 7f6d92345700 1 -- 172.21.15.107:0/3243937638 shutdown_connections 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.062+0000 7f6d92345700 1 -- 172.21.15.107:0/3243937638 >> 172.21.15.107:0/3243937638 conn(0x7f6d8c0f3640 msgr2=0x7f6d8c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.062+0000 7f6d92345700 1 -- 172.21.15.107:0/3243937638 shutdown_connections 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.062+0000 7f6d92345700 1 -- 172.21.15.107:0/3243937638 wait complete. 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 librados: starting msgr at 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 librados: starting objecter 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 Processor -- start 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 -- start start 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 librados: setting wanted keys 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 librados: calling monclient init 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c117310 con 0x7f6d8c104330 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c1184f0 con 0x7f6d8c0fda50 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d92345700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c1196f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d8b7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6d8c1007f0 0x7f6d8c0fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50404/0 (socket says 172.21.15.107:50404) 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d8b7fe700 1 -- 172.21.15.107:0/1436844771 learned_addr learned my addr 172.21.15.107:0/1436844771 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter ms_handle_connect 0x7f6d8c104330 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter ms_handle_connect 0x7f6d8c1007f0 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter ms_handle_connect 0x7f6d8c0fda50 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400940727 0 0) 0x7f6d8c1196f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74003800 con 0x7f6d8c1007f0 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590688205 0 0) 0x7f6d8c117310 con 0x7f6d8c104330 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c1196f0 con 0x7f6d8c104330 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.063+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407252775 0 0) 0x7f6d8c1184f0 con 0x7f6d8c0fda50 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c117310 con 0x7f6d8c0fda50 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821790825 0 0) 0x7f6d74003800 con 0x7f6d8c1007f0 2024-02-10T08:41:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c1184f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d80002f60 con 0x7f6d8c1007f0 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142884519 0 0) 0x7f6d8c1196f0 con 0x7f6d8c104330 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74003800 con 0x7f6d8c104330 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d84001b40 con 0x7f6d8c104330 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531729165 0 0) 0x7f6d8c117310 con 0x7f6d8c0fda50 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c1196f0 con 0x7f6d8c0fda50 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d7c001620 con 0x7f6d8c0fda50 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549052550 0 0) 0x7f6d8c1184f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.195:6789/0 conn(0x7f6d8c0fda50 legacy=0x7f6d8c0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6789/0 conn(0x7f6d8c104330 legacy=0x7f6d8c1159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c11a8f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c1198e0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d92345700 10 client.53519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c119dd0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d92345700 1 librados: init done 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6d800024f0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.064+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6d80003760 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.065+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6d78004f00 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.065+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6d80002660 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(375..375 src has 275..375) v4 ==== 8941+0+0 (unknown 516084016 0 0) 0x7f6d800792e0 con 0x7f6d8c1007f0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 osd_map(375..375 src has 275..375) v4 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 3 client.53519.objecter handle_osd_map got epochs [375,375] > 0 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 3 client.53519.objecter handle_osd_map decoding full epoch 375 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 20 client.53519.objecter dump_active .. 0 homeless 2024-02-10T08:41:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v375) v1 ==== 54+0+16 (unknown 3259245576 0 2420000192) 0x7f6d8007b730 con 0x7f6d8c1007f0 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d74063e20 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 10 client.53519.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 1 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 1 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 0 1 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 4 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 10 client.53519.objecter _send_command 1 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78004950 con 0x7f6d78004530 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.066+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78004530 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.067+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d78004950 con 0x7f6d78004530 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.067+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.067+0000 7f6d897fa700 10 client.53519.objecter _finish_command 1 = system:0 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.067+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 4 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.067+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 0 1 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 10 client.53519.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 2 2024-02-10T08:41:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 2 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d780051a0 osd=0 3 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 0 2 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 4 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 10 client.53519.objecter _send_command 2 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.082+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d78004d30 con 0x7f6d78004530 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d78004d30 con 0x7f6d78004530 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 10 client.53519.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 4 2024-02-10T08:41:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 0 2 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 10 client.53519.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 1 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 4 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 10 client.53519.objecter _send_command 3 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d7800c000 con 0x7f6d78004150 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.084+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78004150 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.085+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d7800c000 con 0x7f6d78004150 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.085+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:41:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.085+0000 7f6d897fa700 10 client.53519.objecter _finish_command 3 = system:0 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.085+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.085+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 1 3 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 10 client.53519.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78003fe0 osd=1 3 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 1 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 10 client.53519.objecter _send_command 4 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.100+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d780018b0 con 0x7f6d78004150 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d780018b0 con 0x7f6d78004150 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d897fa700 10 client.53519.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 4 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 1 4 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 10 client.53519.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 5 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 5 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 2 5 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 4 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 10 client.53519.objecter _send_command 5 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.101+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d7800de90 con 0x7f6d7800d2f0 2024-02-10T08:41:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d7800d2f0 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d7800de90 con 0x7f6d7800d2f0 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 10 client.53519.objecter _finish_command 5 = system:0 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 4 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.102+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 2 5 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 10 client.53519.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 6 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 6 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78003630 osd=2 3 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 2 6 2024-02-10T08:41:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 4 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 10 client.53519.objecter _send_command 6 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.116+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d780030a0 con 0x7f6d7800d2f0 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d780030a0 con 0x7f6d7800d2f0 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 10 client.53519.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 4 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 2 6 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:41:09.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 10 client.53519.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 7 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 7 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 3 7 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 4 2024-02-10T08:41:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 10 client.53519.objecter _send_command 7 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78015b00 con 0x7f6d78011ff0 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.118+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78011ff0 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.119+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d78015b00 con 0x7f6d78011ff0 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.119+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.119+0000 7f6d897fa700 10 client.53519.objecter _finish_command 7 = system:0 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.119+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 4 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.119+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 3 7 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 10 client.53519.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 8 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 8 2024-02-10T08:41:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78011970 osd=3 3 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 3 8 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 4 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 10 client.53519.objecter _send_command 8 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.133+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d7800e240 con 0x7f6d78011ff0 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d7800e240 con 0x7f6d78011ff0 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d897fa700 10 client.53519.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 4 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 3 8 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 10 client.53519.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 9 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 9 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 4 9 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 4 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 10 client.53519.objecter _send_command 9 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.135+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d7800e1b0 con 0x7f6d7800ce50 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d7800ce50 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d7800e1b0 con 0x7f6d7800ce50 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 10 client.53519.objecter _finish_command 9 = system:0 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 4 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.136+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 4 9 2024-02-10T08:41:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 10 client.53519.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 10 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 10 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d7800e000 osd=4 3 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 4 10 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 4 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 10 client.53519.objecter _send_command 10 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.151+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d7800e1b0 con 0x7f6d7800ce50 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d7800e1b0 con 0x7f6d7800ce50 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d897fa700 10 client.53519.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 4 2024-02-10T08:41:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 4 10 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 10 client.53519.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 11 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 11 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 5 11 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 4 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 10 client.53519.objecter _send_command 11 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.153+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78018040 con 0x7f6d78017a60 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78017a60 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d78018040 con 0x7f6d78017a60 2024-02-10T08:41:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 10 client.53519.objecter _finish_command 11 = system:0 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 4 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.154+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 5 11 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 10 client.53519.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 12 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 12 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d7800e1b0 osd=5 3 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 5 12 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 4 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 10 client.53519.objecter _send_command 12 2024-02-10T08:41:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.169+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d78018040 con 0x7f6d78017a60 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d78018040 con 0x7f6d78017a60 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d897fa700 10 client.53519.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 4 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 5 12 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 10 client.53519.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 13 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 13 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 6 13 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 4 2024-02-10T08:41:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 10 client.53519.objecter _send_command 13 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.171+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78027040 con 0x7f6d78023490 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78023490 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d78027040 con 0x7f6d78023490 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 10 client.53519.objecter _finish_command 13 = system:0 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 4 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.172+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 6 13 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 10 client.53519.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 14 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 14 2024-02-10T08:41:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78018040 osd=6 3 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 6 14 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 4 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 10 client.53519.objecter _send_command 14 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.187+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d78027040 con 0x7f6d78023490 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:41:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.188+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d78027040 con 0x7f6d78023490 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.188+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.188+0000 7f6d897fa700 10 client.53519.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.188+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 4 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.188+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 6 14 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 10 client.53519.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 15 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 15 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 7 15 2024-02-10T08:41:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 4 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 10 client.53519.objecter _send_command 15 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78028260 con 0x7f6d78028d00 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.189+0000 7f6d897fa700 10 client.53519.objecter ms_handle_connect 0x7f6d78028d00 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.190+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d78028260 con 0x7f6d78028d00 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.190+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.190+0000 7f6d897fa700 10 client.53519.objecter _finish_command 15 = system:0 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.190+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 4 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.190+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 7 15 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 10 client.53519.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign -1 16 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 20 client.53519.objecter _get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 15 client.53519.objecter _session_command_op_remove -1 16 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 20 client.53519.objecter get_session s=0x7f6d78027040 osd=7 3 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 15 client.53519.objecter _session_command_op_assign 7 16 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 4 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 10 client.53519.objecter _send_command 16 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.204+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6d78028260 con 0x7f6d78028d00 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d897fa700 1 -- 172.21.15.107:0/1436844771 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6d78028260 con 0x7f6d78028d00 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d897fa700 10 client.53519.objecter ms_dispatch 0x7f6d8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d897fa700 10 client.53519.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d897fa700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 4 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d897fa700 15 client.53519.objecter _session_command_op_remove 7 16 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 librados: watch_flush enter 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 librados: watch_flush exit 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.0 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6801/3321596972 conn(0x7f6d78004530 legacy=0x7f6d78002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d780051a0 osd=0 1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6805/133516126 conn(0x7f6d78004150 legacy=0x7f6d7800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003fe0 osd=1 1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.2 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6809/3575881728 conn(0x7f6d7800d2f0 legacy=0x7f6d7800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78003630 osd=2 1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.3 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6813/2871825691 conn(0x7f6d78011ff0 legacy=0x7f6d780143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78011970 osd=3 1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.4 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.195:6800/2499671285 conn(0x7f6d7800ce50 legacy=0x7f6d78016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d908e2700 1 -- 172.21.15.107:0/1436844771 reap_dead start 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e000 osd=4 1 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.5 2024-02-10T08:41:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.195:6804/4178321486 conn(0x7f6d78017a60 legacy=0x7f6d78021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d908e2700 1 -- 172.21.15.107:0/1436844771 reap_dead start 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d908e2700 1 -- 172.21.15.107:0/1436844771 reap_dead start 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d7800e1b0 osd=5 1 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.6 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.195:6808/1417479017 conn(0x7f6d78023490 legacy=0x7f6d78025880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78018040 osd=6 1 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter close_session for osd.7 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.195:6812/1934176928 conn(0x7f6d78028d00 legacy=0x7f6d78029180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter put_session s=0x7f6d78027040 osd=7 1 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 20 client.53519.objecter shutdown clearing up homeless session... 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.206+0000 7f6d92345700 10 client.53519.objecter successfully canceled tick 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6d74063e20 legacy=0x7f6d740662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> v1:172.21.15.107:6790/0 conn(0x7f6d8c1007f0 legacy=0x7f6d8c0fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 shutdown_connections 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 >> 172.21.15.107:0/1436844771 conn(0x7f6d8c0f3640 msgr2=0x7f6d8c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 shutdown_connections 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 -- 172.21.15.107:0/1436844771 wait complete. 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:09.207+0000 7f6d92345700 1 librados: shutdown 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:41:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:09.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:09.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:10.024 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:10.049 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:10.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:10.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:10.204 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:10.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.530354+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:08.533246+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:09.349620+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:09.353084+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:09.426542+0000 mgr.y (mgr.14553) 340 : cluster [DBG] pgmap v539: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:10.024343+0000 mon.a (mon.0) 2345 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.024468+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-10T08:41:10.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:10.029237+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:10.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:10.029819+0000 mon.a (mon.0) 2347 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:10.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:10.334 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:10.352 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:10.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:10.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.530354+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:08.533246+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:09.349620+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:09.353084+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:09.426542+0000 mgr.y (mgr.14553) 340 : cluster [DBG] pgmap v539: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:10.024343+0000 mon.a (mon.0) 2345 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.024468+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:10.029237+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:10.029819+0000 mon.a (mon.0) 2347 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.530354+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:41:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:08.533246+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:41:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:09.349620+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:41:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:09.353084+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:41:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:09.426542+0000 mgr.y (mgr.14553) 340 : cluster [DBG] pgmap v539: 226 pgs: 8 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:10.024343+0000 mon.a (mon.0) 2345 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:41:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.024468+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-10T08:41:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:10.029237+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:10.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:10.029819+0000 mon.a (mon.0) 2347 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:41:11.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.709838+0000 osd.4 (osd.4) 164 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:41:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.710728+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:41:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.999912+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:11.005159+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:41:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.005254+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-10T08:41:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:11.006102+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:11.010270+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:11.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:11.010682+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.709838+0000 osd.4 (osd.4) 164 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.710728+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.999912+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:11.005159+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.005254+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:11.006102+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:11.010270+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:11.010682+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.709838+0000 osd.4 (osd.4) 164 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.710728+0000 osd.4 (osd.4) 165 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.999912+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:11.005159+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.005254+0000 mon.a (mon.0) 2350 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:11.006102+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:11.010270+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:11.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:11.010682+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:41:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:11.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:11.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:11.768 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:11.789 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:11.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:11.851 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:41:11.852 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:41:11.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:11.922 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:11.942 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:11.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:12.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:12.073 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:12.092 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:12.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:12.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:12.223 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:12.242 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:12.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:12.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.518662+0000 osd.6 (osd.6) 143 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.521559+0000 osd.6 (osd.6) 144 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.622997+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.a scrub starts 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:10.625884+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.a scrub ok 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.400532+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.401418+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:11.411125+0000 mgr.y (mgr.14553) 341 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.427418+0000 mgr.y (mgr.14553) 342 : cluster [DBG] pgmap v542: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.701307+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.702668+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:12.012200+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:12.012258+0000 mon.a (mon.0) 2354 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.012316+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:12.015274+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:12.015488+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:12.015753+0000 mon.a (mon.0) 2357 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.372 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:12.390 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:12.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:12.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:12.522 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.518662+0000 osd.6 (osd.6) 143 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.521559+0000 osd.6 (osd.6) 144 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.622997+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.a scrub starts 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:10.625884+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.a scrub ok 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.400532+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.401418+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:11.411125+0000 mgr.y (mgr.14553) 341 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:12.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.427418+0000 mgr.y (mgr.14553) 342 : cluster [DBG] pgmap v542: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.701307+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.702668+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:12.012200+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:12.012258+0000 mon.a (mon.0) 2354 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.012316+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:12.015274+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:12.015488+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:12.015753+0000 mon.a (mon.0) 2357 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:12.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:12.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.518662+0000 osd.6 (osd.6) 143 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:41:12.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.521559+0000 osd.6 (osd.6) 144 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:41:12.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.622997+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.a scrub starts 2024-02-10T08:41:12.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:10.625884+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.a scrub ok 2024-02-10T08:41:12.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.400532+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.401418+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:11.411125+0000 mgr.y (mgr.14553) 341 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.427418+0000 mgr.y (mgr.14553) 342 : cluster [DBG] pgmap v542: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.701307+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.702668+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:12.012200+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:12.012258+0000 mon.a (mon.0) 2354 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.012316+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:12.015274+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]: dispatch 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:12.015488+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:12.015753+0000 mon.a (mon.0) 2357 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:41:12.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:13.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 -- 172.21.15.107:0/1750073417 >> v1:172.21.15.107:6789/0 conn(0x7ff174104330 legacy=0x7ff1741067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 -- 172.21.15.107:0/1750073417 shutdown_connections 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 -- 172.21.15.107:0/1750073417 >> 172.21.15.107:0/1750073417 conn(0x7ff1740f3640 msgr2=0x7ff1740f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 -- 172.21.15.107:0/1750073417 shutdown_connections 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 -- 172.21.15.107:0/1750073417 wait complete. 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 librados: starting msgr at 2024-02-10T08:41:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 librados: starting objecter 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.867+0000 7ff179c9b700 1 Processor -- start 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 -- start start 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 librados: setting wanted keys 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 librados: calling monclient init 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1740fd000 con 0x7ff1741007f0 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1740fd1f0 con 0x7ff174104330 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff179c9b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1740fd3e0 con 0x7ff1740fda50 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff1737fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff1740fda50 0x7ff1740fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48542/0 (socket says 172.21.15.107:48542) 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff1737fe700 1 -- 172.21.15.107:0/1157006713 learned_addr learned my addr 172.21.15.107:0/1157006713 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff172ffd700 1 --1- 172.21.15.107:0/1157006713 >> v1:172.21.15.107:6789/0 conn(0x7ff1741007f0 0x7ff174114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53658/0 (socket says 172.21.15.107:53658) 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter ms_handle_connect 0x7ff1740fda50 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter ms_handle_connect 0x7ff1741007f0 2024-02-10T08:41:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter ms_handle_connect 0x7ff174104330 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702349451 0 0) 0x7ff1740fd000 con 0x7ff1741007f0 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff15c003030 con 0x7ff1741007f0 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980759671 0 0) 0x7ff1740fd3e0 con 0x7ff1740fda50 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1740fd000 con 0x7ff1740fda50 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522503888 0 0) 0x7ff1740fd1f0 con 0x7ff174104330 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1740fd3e0 con 0x7ff174104330 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3523699492 0 0) 0x7ff15c003030 con 0x7ff1741007f0 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1740fd1f0 con 0x7ff1741007f0 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1640018c0 con 0x7ff1741007f0 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112890260 0 0) 0x7ff1740fd3e0 con 0x7ff174104330 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff15c003030 con 0x7ff174104330 2024-02-10T08:41:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1979241284 0 0) 0x7ff1740fd000 con 0x7ff1740fda50 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1740fd3e0 con 0x7ff1740fda50 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff16c001890 con 0x7ff174104330 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1680019f0 con 0x7ff1740fda50 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.868+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1931497264 0 0) 0x7ff1740fd1f0 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 >> v1:172.21.15.107:6790/0 conn(0x7ff1740fda50 legacy=0x7ff1740fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 >> v1:172.21.15.195:6789/0 conn(0x7ff174104330 legacy=0x7ff1741184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17411cc00 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff174119c30 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff179c9b700 10 client.27945.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17411a0a0 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff179c9b700 1 librados: init done 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff164002460 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff164005a00 con 0x7ff1741007f0 2024-02-10T08:41:13.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.869+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff160004f20 con 0x7ff1741007f0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff164001fc0 con 0x7ff1741007f0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(378..378 src has 275..378) v4 ==== 8281+0+0 (unknown 1235593707 0 0) 0x7ff164078cd0 con 0x7ff1741007f0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 10 client.27945.objecter ms_dispatch 0x7ff174001550 osd_map(378..378 src has 275..378) v4 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 3 client.27945.objecter handle_osd_map got epochs [378,378] > 0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 3 client.27945.objecter handle_osd_map decoding full epoch 378 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.870+0000 7ff170ff9700 20 client.27945.objecter dump_active .. 0 homeless 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.871+0000 7ff170ff9700 10 client.27945.objecter ms_handle_connect 0x7ff15c063cd0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:12.875+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff16404c000 con 0x7ff1741007f0 2024-02-10T08:41:13.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:13.064+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff1600051d0 con 0x7ff1741007f0 2024-02-10T08:41:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:13.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.898441+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:41:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:11.901482+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:41:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.524914+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.525891+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.013294+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.013357+0000 mon.a (mon.0) 2359 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.013413+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.036722+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.037086+0000 mon.a (mon.0) 2361 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.038125+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.038386+0000 mon.a (mon.0) 2362 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.038801+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.039059+0000 mon.a (mon.0) 2363 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:13.065874+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.898441+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:41:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:11.901482+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.524914+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.525891+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.013294+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.013357+0000 mon.a (mon.0) 2359 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.013413+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.036722+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.037086+0000 mon.a (mon.0) 2361 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.038125+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.038386+0000 mon.a (mon.0) 2362 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.038801+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.039059+0000 mon.a (mon.0) 2363 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:13.065874+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:13.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:13.484 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:13.504 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:13.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.898441+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:11.901482+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.524914+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.525891+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.013294+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? v1:172.21.15.107:0/1728049522' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi107-112411-64"}]': finished 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.013357+0000 mon.a (mon.0) 2359 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-49","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.013413+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.036722+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.037086+0000 mon.a (mon.0) 2361 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.038125+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.038386+0000 mon.a (mon.0) 2362 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.038801+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.039059+0000 mon.a (mon.0) 2363 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:41:13.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:13.065874+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:13.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:13.640 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:13.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:13.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:14.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.015+0000 7ff170ff9700 1 -- 172.21.15.107:0/1157006713 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v380) v1 ==== 101+0+0 (unknown 1891652347 0 0) 0x7ff16404a410 con 0x7ff1741007f0 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.018+0000 7ff179c9b700 10 librados: watch_flush enter 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.018+0000 7ff179c9b700 10 librados: watch_flush exit 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.018+0000 7ff179c9b700 20 client.27945.objecter shutdown clearing up homeless session... 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.018+0000 7ff179c9b700 10 client.27945.objecter successfully canceled tick 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff15c063cd0 legacy=0x7ff15c066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 >> v1:172.21.15.107:6789/0 conn(0x7ff1741007f0 legacy=0x7ff174114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 shutdown_connections 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 >> 172.21.15.107:0/1157006713 conn(0x7ff1740f3640 msgr2=0x7ff1741077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 shutdown_connections 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 -- 172.21.15.107:0/1157006713 wait complete. 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:14.019+0000 7ff179c9b700 1 librados: shutdown 2024-02-10T08:41:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:41:14.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:41:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:14.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.550726+0000 osd.1 (osd.1) 169 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.554228+0000 osd.1 (osd.1) 170 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.926682+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:12.929585+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.427816+0000 mgr.y (mgr.14553) 343 : cluster [DBG] pgmap v545: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.451043+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:41:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.451984+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.769579+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub starts 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.773220+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.d scrub ok 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:14.010699+0000 mon.a (mon.0) 2365 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:14.016317+0000 mon.a (mon.0) 2366 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:14.016400+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:14.016447+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:14.018198+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:14.018580+0000 mon.a (mon.0) 2369 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:14.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.550726+0000 osd.1 (osd.1) 169 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:41:14.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.554228+0000 osd.1 (osd.1) 170 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:41:14.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.926682+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:41:14.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:12.929585+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:41:14.624 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.427816+0000 mgr.y (mgr.14553) 343 : cluster [DBG] pgmap v545: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.451043+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.451984+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.769579+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub starts 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.773220+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.d scrub ok 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:14.010699+0000 mon.a (mon.0) 2365 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:14.016317+0000 mon.a (mon.0) 2366 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:14.016400+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:14.016447+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:14.018198+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:14.018580+0000 mon.a (mon.0) 2369 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.550726+0000 osd.1 (osd.1) 169 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.554228+0000 osd.1 (osd.1) 170 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.926682+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:12.929585+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.427816+0000 mgr.y (mgr.14553) 343 : cluster [DBG] pgmap v545: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.451043+0000 osd.7 (osd.7) 157 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.451984+0000 osd.7 (osd.7) 158 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.769579+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub starts 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.773220+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.d scrub ok 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:14.010699+0000 mon.a (mon.0) 2365 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:14.016317+0000 mon.a (mon.0) 2366 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi107-112411-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:14.016400+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.107:0/1157006713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:14.016447+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-10T08:41:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:14.018198+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:14.018580+0000 mon.a (mon.0) 2369 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:14.879 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:14.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:14.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:15.009 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:15.521 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.975025+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:41:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:13.978008+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:41:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:14.277625+0000 mon.a (mon.0) 2370 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:41:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.255740+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.975025+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:13.978008+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:14.277625+0000 mon.a (mon.0) 2370 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.255740+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.975025+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:41:15.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:13.978008+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:41:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:14.277625+0000 mon.a (mon.0) 2370 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:41:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.255740+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-10T08:41:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:15] "GET /metrics HTTP/1.1" 200 189585 "" "Prometheus/2.33.4" 2024-02-10T08:41:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.301227+0000 osd.2 (osd.2) 131 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.304101+0000 osd.2 (osd.2) 132 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.428306+0000 mgr.y (mgr.14553) 344 : cluster [DBG] pgmap v548: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.514786+0000 osd.7 (osd.7) 159 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:15.515725+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:16.019005+0000 mon.a (mon.0) 2372 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.019112+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.301227+0000 osd.2 (osd.2) 131 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.304101+0000 osd.2 (osd.2) 132 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.428306+0000 mgr.y (mgr.14553) 344 : cluster [DBG] pgmap v548: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.514786+0000 osd.7 (osd.7) 159 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:41:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:15.515725+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:41:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:16.019005+0000 mon.a (mon.0) 2372 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.019112+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.301227+0000 osd.2 (osd.2) 131 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.304101+0000 osd.2 (osd.2) 132 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.428306+0000 mgr.y (mgr.14553) 344 : cluster [DBG] pgmap v548: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.514786+0000 osd.7 (osd.7) 159 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:15.515725+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:16.019005+0000 mon.a (mon.0) 2372 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi107-112411-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.019112+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-10T08:41:16.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:41:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.001185+0000 osd.5 (osd.5) 171 : cluster [DBG] 16.c scrub starts 2024-02-10T08:41:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.004162+0000 osd.5 (osd.5) 172 : cluster [DBG] 16.c scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.347228+0000 osd.2 (osd.2) 133 : cluster [DBG] 4.b scrub starts 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.350718+0000 osd.2 (osd.2) 134 : cluster [DBG] 4.b scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.495952+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.496991+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.637537+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.b scrub starts 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:16.638473+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.b scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:17.020433+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.001185+0000 osd.5 (osd.5) 171 : cluster [DBG] 16.c scrub starts 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.004162+0000 osd.5 (osd.5) 172 : cluster [DBG] 16.c scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.347228+0000 osd.2 (osd.2) 133 : cluster [DBG] 4.b scrub starts 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.350718+0000 osd.2 (osd.2) 134 : cluster [DBG] 4.b scrub ok 2024-02-10T08:41:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.495952+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:41:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.496991+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:41:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.637537+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.b scrub starts 2024-02-10T08:41:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:16.638473+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.b scrub ok 2024-02-10T08:41:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:17.020433+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.001185+0000 osd.5 (osd.5) 171 : cluster [DBG] 16.c scrub starts 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.004162+0000 osd.5 (osd.5) 172 : cluster [DBG] 16.c scrub ok 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.347228+0000 osd.2 (osd.2) 133 : cluster [DBG] 4.b scrub starts 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.350718+0000 osd.2 (osd.2) 134 : cluster [DBG] 4.b scrub ok 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.495952+0000 osd.7 (osd.7) 161 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.496991+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.637537+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.b scrub starts 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:16.638473+0000 osd.3 (osd.3) 180 : cluster [DBG] 2.b scrub ok 2024-02-10T08:41:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:17.020433+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-10T08:41:18.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:17.428705+0000 mgr.y (mgr.14553) 345 : cluster [DBG] pgmap v551: 226 pgs: 7 creating+peering, 1 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:18.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:18.023622+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:18.027830+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:18.028135+0000 mon.a (mon.0) 2376 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:17.428705+0000 mgr.y (mgr.14553) 345 : cluster [DBG] pgmap v551: 226 pgs: 7 creating+peering, 1 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:18.023622+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:18.027830+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:18.028135+0000 mon.a (mon.0) 2376 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.492 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:18.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:18.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:18.626 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:17.428705+0000 mgr.y (mgr.14553) 345 : cluster [DBG] pgmap v551: 226 pgs: 7 creating+peering, 1 unknown, 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:18.626 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:18.023622+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-10T08:41:18.626 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:18.027830+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.626 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:18.028135+0000 mon.a (mon.0) 2376 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:18.628 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:18.650 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:18.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:18.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:18.783 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- 172.21.15.107:0/1785850023 >> v1:172.21.15.107:6789/0 conn(0x7fde900f8a20 legacy=0x7fde900f8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- 172.21.15.107:0/1785850023 shutdown_connections 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- 172.21.15.107:0/1785850023 >> 172.21.15.107:0/1785850023 conn(0x7fde900f2510 msgr2=0x7fde900f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- 172.21.15.107:0/1785850023 shutdown_connections 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- 172.21.15.107:0/1785850023 wait complete. 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 librados: starting msgr at 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 librados: starting objecter 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 Processor -- start 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 -- start start 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 librados: setting wanted keys 2024-02-10T08:41:18.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.982+0000 7fde95649700 1 librados: calling monclient init 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde95649700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde901195e0 con 0x7fde900f95d0 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde95649700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde9011a7c0 con 0x7fde900f8a20 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde95649700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde9011b9a0 con 0x7fde90104350 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde8f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fde90104350 0x7fde90117dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48612/0 (socket says 172.21.15.107:48612) 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde8f7fe700 1 -- 172.21.15.107:0/3622467053 learned_addr learned my addr 172.21.15.107:0/3622467053 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter ms_handle_connect 0x7fde90104350 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter ms_handle_connect 0x7fde900f95d0 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700975970 0 0) 0x7fde9011b9a0 con 0x7fde90104350 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003030 con 0x7fde90104350 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754510701 0 0) 0x7fde901195e0 con 0x7fde900f95d0 2024-02-10T08:41:18.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde9011b9a0 con 0x7fde900f95d0 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter ms_handle_connect 0x7fde900f8a20 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916224868 0 0) 0x7fde9011a7c0 con 0x7fde900f8a20 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde901195e0 con 0x7fde900f8a20 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330100230 0 0) 0x7fde78003030 con 0x7fde90104350 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde9011a7c0 con 0x7fde90104350 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 925433283 0 0) 0x7fde9011b9a0 con 0x7fde900f95d0 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde78003030 con 0x7fde900f95d0 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fde88001c60 con 0x7fde90104350 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.983+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fde84001840 con 0x7fde900f95d0 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1249546193 0 0) 0x7fde9011a7c0 con 0x7fde90104350 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.195:6789/0 conn(0x7fde900f8a20 legacy=0x7fde900fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:18.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6789/0 conn(0x7fde900f95d0 legacy=0x7fde900fda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9011cba0 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde901197d0 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde95649700 10 client.53543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde90119c40 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde95649700 1 librados: init done 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fde880023a0 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fde88005df0 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.984+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fde7c004f00 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.985+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fde88005f50 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(384..384 src has 275..384) v4 ==== 8576+0+0 (unknown 2348026606 0 0) 0x7fde880791a0 con 0x7fde90104350 2024-02-10T08:41:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 osd_map(384..384 src has 275..384) v4 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 3 client.53543.objecter handle_osd_map got epochs [384,384] > 0 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 3 client.53543.objecter handle_osd_map decoding full epoch 384 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 20 client.53543.objecter dump_active .. 0 homeless 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde78063ba0 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v384) v1 ==== 54+0+16 (unknown 3564524033 0 2420000192) 0x7fde8807b590 con 0x7fde90104350 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 10 client.53543.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 1 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 1 2024-02-10T08:41:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 0 1 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 4 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 10 client.53543.objecter _send_command 1 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c004950 con 0x7fde7c004530 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.986+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c004530 2024-02-10T08:41:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.987+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c004950 con 0x7fde7c004530 2024-02-10T08:41:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.987+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:41:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.987+0000 7fde77fff700 10 client.53543.objecter _finish_command 1 = system:0 2024-02-10T08:41:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.987+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 4 2024-02-10T08:41:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.987+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 0 1 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 10 client.53543.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 2 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 2 2024-02-10T08:41:19.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0051a0 osd=0 3 2024-02-10T08:41:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 0 2 2024-02-10T08:41:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 4 2024-02-10T08:41:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 10 client.53543.objecter _send_command 2 2024-02-10T08:41:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.002+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c004d30 con 0x7fde7c004530 2024-02-10T08:41:19.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.003+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c004d30 con 0x7fde7c004530 2024-02-10T08:41:19.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.003+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.003+0000 7fde77fff700 10 client.53543.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.003+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 4 2024-02-10T08:41:19.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.003+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 0 2 2024-02-10T08:41:19.005 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 10 client.53543.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 1 3 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 4 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 10 client.53543.objecter _send_command 3 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c00c000 con 0x7fde7c004150 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c004150 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c00c000 con 0x7fde7c004150 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 10 client.53543.objecter _finish_command 3 = system:0 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 4 2024-02-10T08:41:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.004+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 1 3 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 10 client.53543.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 4 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 4 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c003fe0 osd=1 3 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 1 4 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 4 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 10 client.53543.objecter _send_command 4 2024-02-10T08:41:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.021+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c0018b0 con 0x7fde7c004150 2024-02-10T08:41:19.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.026+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c0018b0 con 0x7fde7c004150 2024-02-10T08:41:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.026+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.026+0000 7fde77fff700 10 client.53543.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.026+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 4 2024-02-10T08:41:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.026+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 1 4 2024-02-10T08:41:19.028 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 10 client.53543.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 5 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 5 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 2 5 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 4 2024-02-10T08:41:19.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 10 client.53543.objecter _send_command 5 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c003e40 con 0x7fde7c00d2f0 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c00d2f0 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c003e40 con 0x7fde7c00d2f0 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 10 client.53543.objecter _finish_command 5 = system:0 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 4 2024-02-10T08:41:19.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.027+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 2 5 2024-02-10T08:41:19.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 10 client.53543.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 6 2024-02-10T08:41:19.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 6 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c003630 osd=2 3 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 2 6 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 4 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 10 client.53543.objecter _send_command 6 2024-02-10T08:41:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.042+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c00e720 con 0x7fde7c00d2f0 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.043+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c00e720 con 0x7fde7c00d2f0 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.043+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde77fff700 10 client.53543.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 4 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 2 6 2024-02-10T08:41:19.045 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 10 client.53543.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 7 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 7 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 3 7 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 4 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 10 client.53543.objecter _send_command 7 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c015a10 con 0x7fde7c011e50 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.044+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c011e50 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.045+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c015a10 con 0x7fde7c011e50 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.045+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.045+0000 7fde77fff700 10 client.53543.objecter _finish_command 7 = system:0 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.045+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 4 2024-02-10T08:41:19.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.045+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 3 7 2024-02-10T08:41:19.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 10 client.53543.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 8 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 8 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0033c0 osd=3 3 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 3 8 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 4 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 10 client.53543.objecter _send_command 8 2024-02-10T08:41:19.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.059+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c00e190 con 0x7fde7c011e50 2024-02-10T08:41:19.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.060+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c00e190 con 0x7fde7c011e50 2024-02-10T08:41:19.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.060+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.060+0000 7fde77fff700 10 client.53543.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.060+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 4 2024-02-10T08:41:19.062 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.060+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 3 8 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 10 client.53543.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 9 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 9 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 4 9 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 4 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 10 client.53543.objecter _send_command 9 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c0162c0 con 0x7fde7c00ce50 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.061+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c00ce50 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.062+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c0162c0 con 0x7fde7c00ce50 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.062+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.062+0000 7fde77fff700 10 client.53543.objecter _finish_command 9 = system:0 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.062+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 4 2024-02-10T08:41:19.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.062+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 4 9 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 10 client.53543.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 10 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 10 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c00e190 osd=4 3 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 4 10 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 4 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 10 client.53543.objecter _send_command 10 2024-02-10T08:41:19.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.076+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c0162c0 con 0x7fde7c00ce50 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c0162c0 con 0x7fde7c00ce50 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde77fff700 10 client.53543.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 4 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 4 10 2024-02-10T08:41:19.080 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde95649700 10 client.53543.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.078+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 11 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 11 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 5 11 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 4 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 10 client.53543.objecter _send_command 11 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c017020 con 0x7fde7c016a50 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.079+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c016a50 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.080+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c017020 con 0x7fde7c016a50 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.080+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.080+0000 7fde77fff700 10 client.53543.objecter _finish_command 11 = system:0 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.080+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 4 2024-02-10T08:41:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.080+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 5 11 2024-02-10T08:41:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 10 client.53543.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 12 2024-02-10T08:41:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 12 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c0162c0 osd=5 3 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 5 12 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 4 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 10 client.53543.objecter _send_command 12 2024-02-10T08:41:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.094+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c017020 con 0x7fde7c016a50 2024-02-10T08:41:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c017020 con 0x7fde7c016a50 2024-02-10T08:41:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde77fff700 10 client.53543.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 4 2024-02-10T08:41:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 5 12 2024-02-10T08:41:19.098 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 10 client.53543.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 13 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 13 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 6 13 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 4 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 10 client.53543.objecter _send_command 13 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.096+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c01fbb0 con 0x7fde7c020650 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.097+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c020650 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.097+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c01fbb0 con 0x7fde7c020650 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.098+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.098+0000 7fde77fff700 10 client.53543.objecter _finish_command 13 = system:0 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.098+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 4 2024-02-10T08:41:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.098+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 6 13 2024-02-10T08:41:19.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.883+0000 7f777c7cf700 1 -- 172.21.15.107:0/622577742 >> v1:172.21.15.107:6790/0 conn(0x7f77740f8a60 legacy=0x7f77740f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.883+0000 7f777c7cf700 1 -- 172.21.15.107:0/622577742 shutdown_connections 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.883+0000 7f777c7cf700 1 -- 172.21.15.107:0/622577742 >> 172.21.15.107:0/622577742 conn(0x7f77740f2510 msgr2=0x7f77740f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.883+0000 7f777c7cf700 1 -- 172.21.15.107:0/622577742 shutdown_connections 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.883+0000 7f777c7cf700 1 -- 172.21.15.107:0/622577742 wait complete. 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 librados: starting msgr at 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 librados: starting objecter 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 Processor -- start 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 -- start start 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 librados: setting wanted keys 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 librados: calling monclient init 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7774102bd0 con 0x7f7774104330 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7774102da0 con 0x7f77740f9550 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777c7cf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7774102f90 con 0x7f77740f8a60 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777a56b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f77740f8a60 0x7f7774102430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48580/0 (socket says 172.21.15.107:48580) 2024-02-10T08:41:19.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f777a56b700 1 -- 172.21.15.107:0/1769860740 learned_addr learned my addr 172.21.15.107:0/1769860740 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter ms_handle_connect 0x7f7774104330 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter ms_handle_connect 0x7f77740f8a60 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24261861 0 0) 0x7f7774102f90 con 0x7f77740f8a60 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77600038e0 con 0x7f77740f8a60 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter ms_handle_connect 0x7f77740f9550 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618024072 0 0) 0x7f7774102bd0 con 0x7f7774104330 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774102f90 con 0x7f7774104330 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548821368 0 0) 0x7f7774102da0 con 0x7f77740f9550 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.884+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774102bd0 con 0x7f77740f9550 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293839107 0 0) 0x7f77600038e0 con 0x7f77740f8a60 2024-02-10T08:41:19.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7774102da0 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7764002050 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036223080 0 0) 0x7f7774102f90 con 0x7f7774104330 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77600038e0 con 0x7f7774104330 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f776c001c50 con 0x7f7774104330 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3145843929 0 0) 0x7f7774102da0 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 >> v1:172.21.15.195:6789/0 conn(0x7f77740f9550 legacy=0x7f7774114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 >> v1:172.21.15.107:6789/0 conn(0x7f7774104330 legacy=0x7f77741184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777411cc00 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7774119c30 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f777c7cf700 10 client.53537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f777411a0a0 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f777c7cf700 1 librados: init done 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7764002690 con 0x7f77740f8a60 2024-02-10T08:41:19.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.885+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f77640036f0 con 0x7f77740f8a60 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.886+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7758004f20 con 0x7f77740f8a60 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.886+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f77640021f0 con 0x7f77740f8a60 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.886+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(384..384 src has 275..384) v4 ==== 8576+0+0 (unknown 2348026606 0 0) 0x7f7764079260 con 0x7f77740f8a60 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.886+0000 7f776b7fe700 10 client.53537.objecter ms_dispatch 0x7f7774001550 osd_map(384..384 src has 275..384) v4 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.887+0000 7f776b7fe700 3 client.53537.objecter handle_osd_map got epochs [384,384] > 0 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.887+0000 7f776b7fe700 3 client.53537.objecter handle_osd_map decoding full epoch 384 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.887+0000 7f776b7fe700 20 client.53537.objecter dump_active .. 0 homeless 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.887+0000 7f776b7fe700 10 client.53537.objecter ms_handle_connect 0x7f7760063970 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:18.891+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f776404c530 con 0x7f77740f8a60 2024-02-10T08:41:19.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.081+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]} v 0) v1 -- 0x7f77580059e0 con 0x7f77740f8a60 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 10 client.53543.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 14 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 14 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c017020 osd=6 3 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 6 14 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 4 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 10 client.53543.objecter _send_command 14 2024-02-10T08:41:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.112+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c01fbb0 con 0x7fde7c020650 2024-02-10T08:41:19.115 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c01fbb0 con 0x7fde7c020650 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde77fff700 10 client.53543.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 4 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 6 14 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 10 client.53543.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:19.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 15 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 15 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 7 15 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 4 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 10 client.53543.objecter _send_command 15 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.114+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde7c027360 con 0x7fde7c023850 2024-02-10T08:41:19.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 10 client.53543.objecter ms_handle_connect 0x7fde7c023850 2024-02-10T08:41:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fde7c027360 con 0x7fde7c023850 2024-02-10T08:41:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:41:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 10 client.53543.objecter _finish_command 15 = system:0 2024-02-10T08:41:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 4 2024-02-10T08:41:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.115+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 7 15 2024-02-10T08:41:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 10 client.53543.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign -1 16 2024-02-10T08:41:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 20 client.53543.objecter _get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 15 client.53543.objecter _session_command_op_remove -1 16 2024-02-10T08:41:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 20 client.53543.objecter get_session s=0x7fde7c01fbb0 osd=7 3 2024-02-10T08:41:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 15 client.53543.objecter _session_command_op_assign 7 16 2024-02-10T08:41:19.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 4 2024-02-10T08:41:19.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 10 client.53543.objecter _send_command 16 2024-02-10T08:41:19.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.130+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fde7c027360 con 0x7fde7c023850 2024-02-10T08:41:19.133 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:41:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde77fff700 1 -- 172.21.15.107:0/3622467053 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fde7c027360 con 0x7fde7c023850 2024-02-10T08:41:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde77fff700 10 client.53543.objecter ms_dispatch 0x7fde90001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde77fff700 10 client.53543.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde77fff700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 4 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde77fff700 15 client.53543.objecter _session_command_op_remove 7 16 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 librados: watch_flush enter 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 librados: watch_flush exit 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.0 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6801/3321596972 conn(0x7fde7c004530 legacy=0x7fde7c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0051a0 osd=0 1 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.1 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6805/133516126 conn(0x7fde7c004150 legacy=0x7fde7c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003fe0 osd=1 1 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.2 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6809/3575881728 conn(0x7fde7c00d2f0 legacy=0x7fde7c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c003630 osd=2 1 2024-02-10T08:41:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.3 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6813/2871825691 conn(0x7fde7c011e50 legacy=0x7fde7c014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0033c0 osd=3 1 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.4 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.195:6800/2499671285 conn(0x7fde7c00ce50 legacy=0x7fde7c015bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde8f7fe700 1 -- 172.21.15.107:0/3622467053 reap_dead start 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde8f7fe700 1 -- 172.21.15.107:0/3622467053 reap_dead start 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c00e190 osd=4 1 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.5 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.195:6804/4178321486 conn(0x7fde7c016a50 legacy=0x7fde7c01d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c0162c0 osd=5 1 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.6 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.195:6808/1417479017 conn(0x7fde7c020650 legacy=0x7fde7c020ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c017020 osd=6 1 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter close_session for osd.7 2024-02-10T08:41:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.195:6812/1934176928 conn(0x7fde7c023850 legacy=0x7fde7c025c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter put_session s=0x7fde7c01fbb0 osd=7 1 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 20 client.53543.objecter shutdown clearing up homeless session... 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 10 client.53543.objecter successfully canceled tick 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6800/2634842009 conn(0x7fde78063ba0 legacy=0x7fde78066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> v1:172.21.15.107:6790/0 conn(0x7fde90104350 legacy=0x7fde90117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde8f7fe700 1 -- 172.21.15.107:0/3622467053 reap_dead start 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 shutdown_connections 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 >> 172.21.15.107:0/3622467053 conn(0x7fde900f2510 msgr2=0x7fde901077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 shutdown_connections 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 -- 172.21.15.107:0/3622467053 wait complete. 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:19.132+0000 7fde95649700 1 librados: shutdown 2024-02-10T08:41:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:41:19.235 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:41:19.235 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:41:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:18.754074+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.c scrub starts 2024-02-10T08:41:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:18.755510+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.c scrub ok 2024-02-10T08:41:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:18.987218+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/3622467053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:19.023462+0000 mon.a (mon.0) 2377 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.023518+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:19.025217+0000 mon.c (mon.2) 382 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:19.025515+0000 mon.a (mon.0) 2379 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:19.082963+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.107:0/1769860740' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:19.083262+0000 mon.a (mon.0) 2380 : audit [INF] from='client.53537 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:18.754074+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.c scrub starts 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:18.755510+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.c scrub ok 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:18.987218+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/3622467053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:19.023462+0000 mon.a (mon.0) 2377 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.023518+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:19.025217+0000 mon.c (mon.2) 382 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:19.025515+0000 mon.a (mon.0) 2379 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:19.082963+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.107:0/1769860740' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:19.083262+0000 mon.a (mon.0) 2380 : audit [INF] from='client.53537 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:18.754074+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.c scrub starts 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:18.755510+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.c scrub ok 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:18.987218+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? v1:172.21.15.107:0/3622467053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:19.023462+0000 mon.a (mon.0) 2377 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.023518+0000 mon.a (mon.0) 2378 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:19.025217+0000 mon.c (mon.2) 382 : audit [INF] from='client.? v1:172.21.15.107:0/443105020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:19.025515+0000 mon.a (mon.0) 2379 : audit [INF] from='client.53531 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]: dispatch 2024-02-10T08:41:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:19.082963+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.107:0/1769860740' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:19.083262+0000 mon.a (mon.0) 2380 : audit [INF] from='client.53537 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]: dispatch 2024-02-10T08:41:19.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:41:20.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.024+0000 7f776b7fe700 1 -- 172.21.15.107:0/1769860740 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]=0 set 6.1 pg_upmap mapping to [2,1,7] v386) v1 ==== 127+0+0 (unknown 3991813993 0 0) 0x7f776404a9a0 con 0x7f77740f8a60 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 10 librados: watch_flush enter 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 10 librados: watch_flush exit 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 20 client.53537.objecter shutdown clearing up homeless session... 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 10 client.53537.objecter successfully canceled tick 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7760063970 legacy=0x7f7760065e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 >> v1:172.21.15.107:6790/0 conn(0x7f77740f8a60 legacy=0x7f7774102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 shutdown_connections 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 >> 172.21.15.107:0/1769860740 conn(0x7f77740f2510 msgr2=0x7f77741077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 shutdown_connections 2024-02-10T08:41:20.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 -- 172.21.15.107:0/1769860740 wait complete. 2024-02-10T08:41:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:20.030+0000 7f777c7cf700 1 librados: shutdown 2024-02-10T08:41:20.033 INFO:teuthology.orchestra.run.smithi107.stderr:set 6.1 pg_upmap mapping to [2,1,7] 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6991 ms) 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7018 ms) 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6995 ms) 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6995 ms) 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-10T08:41:20.035 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7015 ms) 2024-02-10T08:41:20.036 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139065 ms total) 2024-02-10T08:41:20.036 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: 2024-02-10T08:41:20.036 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-10T08:41:20.036 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [==========] 56 tests from 4 test suites ran. (289886 ms total) 2024-02-10T08:41:20.036 INFO:tasks.workunit.client.0.smithi107.stdout: api_aio_pp: [ PASSED ] 56 tests. 2024-02-10T08:41:20.039 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.039 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112422 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112422 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112495 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112495 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112473 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112473 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112446 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112446 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112565 2024-02-10T08:41:20.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112565 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112414 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112414 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112582 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112582 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112531 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112531 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112620 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112620 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112574 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112574 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112636 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112636 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112398 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112398 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112628 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112628 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112590 2024-02-10T08:41:20.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112590 2024-02-10T08:41:20.043 INFO:tasks.workunit.client.0.smithi107.stderr:+ for t in "${!pids[@]}" 2024-02-10T08:41:20.043 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=112484 2024-02-10T08:41:20.043 INFO:tasks.workunit.client.0.smithi107.stderr:+ wait 112484 2024-02-10T08:41:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:18.575912+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:41:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:18.579528+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:41:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.429099+0000 mgr.y (mgr.14553) 346 : cluster [DBG] pgmap v554: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.562550+0000 osd.7 (osd.7) 163 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.563696+0000 osd.7 (osd.7) 164 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.651177+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.653214+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.763377+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:19.765189+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:41:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:20.024879+0000 mon.a (mon.0) 2381 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:20.024952+0000 mon.a (mon.0) 2382 : audit [INF] from='client.53537 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]': finished 2024-02-10T08:41:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:20.025007+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-10T08:41:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:18.575912+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:41:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:18.579528+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:41:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.429099+0000 mgr.y (mgr.14553) 346 : cluster [DBG] pgmap v554: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.562550+0000 osd.7 (osd.7) 163 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.563696+0000 osd.7 (osd.7) 164 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.651177+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.653214+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.763377+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:19.765189+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:20.024879+0000 mon.a (mon.0) 2381 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:20.024952+0000 mon.a (mon.0) 2382 : audit [INF] from='client.53537 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]': finished 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:20.025007+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:18.575912+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:18.579528+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.429099+0000 mgr.y (mgr.14553) 346 : cluster [DBG] pgmap v554: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:41:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.562550+0000 osd.7 (osd.7) 163 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.563696+0000 osd.7 (osd.7) 164 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.651177+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.653214+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.763377+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:19.765189+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:20.024879+0000 mon.a (mon.0) 2381 : audit [INF] from='client.53531 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi107-112411-65"}]': finished 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:20.024952+0000 mon.a (mon.0) 2382 : audit [INF] from='client.53537 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1", "id": [2, 1, 7]}]': finished 2024-02-10T08:41:20.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:20.025007+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-10T08:41:20.962 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:20.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:21.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:21.094 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:21.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:21.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:21.246 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.267 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:21.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:21.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:21.398 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:21.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:21.627 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:21.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:21.677 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:21.679 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.699 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:21.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:21.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:21.831 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.850 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:21.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:21.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:21.980 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:21.999 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:22.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:22.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:22.130 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:20.505034+0000 osd.6 (osd.6) 147 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:41:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:20.506028+0000 osd.6 (osd.6) 148 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:41:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:20.534889+0000 osd.7 (osd.7) 165 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:41:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:20.535879+0000 osd.7 (osd.7) 166 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:41:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.009118+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.383511+0000 osd.2 (osd.2) 135 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.384530+0000 osd.2 (osd.2) 136 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:21.422004+0000 mgr.y (mgr.14553) 347 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.429924+0000 mgr.y (mgr.14553) 348 : cluster [DBG] pgmap v557: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:20.505034+0000 osd.6 (osd.6) 147 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:20.506028+0000 osd.6 (osd.6) 148 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:20.534889+0000 osd.7 (osd.7) 165 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:20.535879+0000 osd.7 (osd.7) 166 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.009118+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-10T08:41:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.383511+0000 osd.2 (osd.2) 135 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:41:22.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.384530+0000 osd.2 (osd.2) 136 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:41:22.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:21.422004+0000 mgr.y (mgr.14553) 347 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:22.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.429924+0000 mgr.y (mgr.14553) 348 : cluster [DBG] pgmap v557: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:20.505034+0000 osd.6 (osd.6) 147 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:41:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:20.506028+0000 osd.6 (osd.6) 148 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:41:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:20.534889+0000 osd.7 (osd.7) 165 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:41:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:20.535879+0000 osd.7 (osd.7) 166 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:41:22.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.009118+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-10T08:41:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.383511+0000 osd.2 (osd.2) 135 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:41:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.384530+0000 osd.2 (osd.2) 136 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:41:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:21.422004+0000 mgr.y (mgr.14553) 347 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:22.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.429924+0000 mgr.y (mgr.14553) 348 : cluster [DBG] pgmap v557: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:41:22.327 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:22.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:22.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:22.458 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.460640+0000 osd.6 (osd.6) 149 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.463522+0000 osd.6 (osd.6) 150 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.595553+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.599030+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.838820+0000 osd.4 (osd.4) 172 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:21.839891+0000 osd.4 (osd.4) 173 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:22.011130+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-10T08:41:23.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:22.395001+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:41:23.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:22.395864+0000 osd.2 (osd.2) 138 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:41:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.460640+0000 osd.6 (osd.6) 149 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:41:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.463522+0000 osd.6 (osd.6) 150 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:41:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.595553+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:41:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.599030+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:41:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.838820+0000 osd.4 (osd.4) 172 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:21.839891+0000 osd.4 (osd.4) 173 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:22.011130+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:22.395001+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:22.395864+0000 osd.2 (osd.2) 138 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.460640+0000 osd.6 (osd.6) 149 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.463522+0000 osd.6 (osd.6) 150 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.595553+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.599030+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.838820+0000 osd.4 (osd.4) 172 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:21.839891+0000 osd.4 (osd.4) 173 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:22.011130+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-10T08:41:23.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:22.395001+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:41:23.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:22.395864+0000 osd.2 (osd.2) 138 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:41:23.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:41:23.807 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:23.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:23.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:23.946 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:23.967 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:23.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:24.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:22.617362+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:22.620251+0000 osd.1 (osd.1) 176 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:23.026618+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:23.026873+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.029698+0000 osd.5 (osd.5) 173 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:41:24.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.032697+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.430413+0000 mgr.y (mgr.14553) 349 : cluster [DBG] pgmap v558: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:22.617362+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:22.620251+0000 osd.1 (osd.1) 176 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:23.026618+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:23.026873+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.029698+0000 osd.5 (osd.5) 173 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.032697+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:41:24.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.430413+0000 mgr.y (mgr.14553) 349 : cluster [DBG] pgmap v558: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:24.100 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:22.617362+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:41:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:22.620251+0000 osd.1 (osd.1) 176 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:41:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:23.026618+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:23.026873+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:24.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.029698+0000 osd.5 (osd.5) 173 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:41:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.032697+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:41:24.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.430413+0000 mgr.y (mgr.14553) 349 : cluster [DBG] pgmap v558: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:24.449 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:24.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:24.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:24.728 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:24.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:24.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:24.860 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:24.879 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:24.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:24.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:24.994 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:41:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:41:24.994 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:41:24.994 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:41:25.015 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:25.037 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:25.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:25.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 -- 172.21.15.107:0/3237398652 >> v1:172.21.15.107:6789/0 conn(0x7f4ad80f9570 legacy=0x7f4ad8102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 -- 172.21.15.107:0/3237398652 shutdown_connections 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 -- 172.21.15.107:0/3237398652 >> 172.21.15.107:0/3237398652 conn(0x7f4ad80f2510 msgr2=0x7f4ad80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 -- 172.21.15.107:0/3237398652 shutdown_connections 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 -- 172.21.15.107:0/3237398652 wait complete. 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 librados: starting msgr at 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.162+0000 7f4adc81b700 1 librados: starting objecter 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 Processor -- start 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 -- start start 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 librados: setting wanted keys 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 librados: calling monclient init 2024-02-10T08:41:25.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ad8117420 con 0x7f4ad80f89c0 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ad8118600 con 0x7f4ad8104350 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4adc81b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ad8119800 con 0x7f4ad80f9570 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ad659c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4ad80f89c0 0x7f4ad80ff720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52292/0 (socket says 172.21.15.107:52292) 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ad659c700 1 -- 172.21.15.107:0/2591084435 learned_addr learned my addr 172.21.15.107:0/2591084435 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ad5d9b700 1 --1- 172.21.15.107:0/2591084435 >> v1:172.21.15.107:6790/0 conn(0x7f4ad80f9570 0x7f4ad80ffe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52824/0 (socket says 172.21.15.107:52824) 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter ms_handle_connect 0x7f4ad80f89c0 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter ms_handle_connect 0x7f4ad80f9570 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter ms_handle_connect 0x7f4ad8104350 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985395038 0 0) 0x7f4ad8117420 con 0x7f4ad80f89c0 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac0003800 con 0x7f4ad80f89c0 2024-02-10T08:41:25.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197432425 0 0) 0x7f4ad8119800 con 0x7f4ad80f9570 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad8117420 con 0x7f4ad80f9570 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18372564 0 0) 0x7f4ad8118600 con 0x7f4ad8104350 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.163+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad8119800 con 0x7f4ad8104350 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865234546 0 0) 0x7f4ac0003800 con 0x7f4ad80f89c0 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad8118600 con 0x7f4ad80f89c0 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4ad0001d70 con 0x7f4ad80f89c0 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2442859651 0 0) 0x7f4ad8117420 con 0x7f4ad80f9570 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac0003800 con 0x7f4ad80f9570 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4acc002e50 con 0x7f4ad80f9570 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3975828065 0 0) 0x7f4ad8119800 con 0x7f4ad8104350 2024-02-10T08:41:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad8117420 con 0x7f4ad8104350 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4ac8001880 con 0x7f4ad8104350 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1689185593 0 0) 0x7f4ad8118600 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 >> v1:172.21.15.107:6790/0 conn(0x7f4ad80f9570 legacy=0x7f4ad80ffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 >> v1:172.21.15.195:6789/0 conn(0x7f4ad8104350 legacy=0x7f4ad8115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad811aa00 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad81199f0 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4adc81b700 10 client.27966.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad8119e60 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4adc81b700 1 librados: init done 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4ad0002810 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.164+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4ad0005ea0 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.165+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab8004f20 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.165+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4ad0002370 con 0x7f4ad80f89c0 2024-02-10T08:41:25.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(388..388 src has 275..388) v4 ==== 8212+0+0 (unknown 2922984393 0 0) 0x7f4ad00790d0 con 0x7f4ad80f89c0 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 10 client.27966.objecter ms_dispatch 0x7f4ad8001550 osd_map(388..388 src has 275..388) v4 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 3 client.27966.objecter handle_osd_map got epochs [388,388] > 0 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 3 client.27966.objecter handle_osd_map decoding full epoch 388 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 20 client.27966.objecter dump_active .. 0 homeless 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.166+0000 7f4ac77fe700 10 client.27966.objecter ms_handle_connect 0x7f4ac0063c70 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.170+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4ad004c450 con 0x7f4ad80f89c0 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:24.349+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f4ab80051d0 con 0x7f4ad80f89c0 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.036+0000 7f4ac77fe700 1 -- 172.21.15.107:0/2591084435 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v389) v1 ==== 87+0+0 (unknown 4012567547 0 0) 0x7f4ad004a860 con 0x7f4ad80f89c0 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 10 librados: watch_flush enter 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 10 librados: watch_flush exit 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 20 client.27966.objecter shutdown clearing up homeless session... 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 10 client.27966.objecter successfully canceled tick 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4ac0063c70 legacy=0x7f4ac0066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:25.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.039+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 >> v1:172.21.15.107:6789/0 conn(0x7f4ad80f89c0 legacy=0x7f4ad80ff720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.040+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 shutdown_connections 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.040+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 >> 172.21.15.107:0/2591084435 conn(0x7f4ad80f2510 msgr2=0x7f4ad81077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.040+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 shutdown_connections 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.040+0000 7f4adc81b700 1 -- 172.21.15.107:0/2591084435 wait complete. 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:25.040+0000 7f4adc81b700 1 librados: shutdown 2024-02-10T08:41:25.135 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:41:25.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:25.173 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.659080+0000 osd.1 (osd.1) 177 : cluster [DBG] 16.b scrub starts 2024-02-10T08:41:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.662117+0000 osd.1 (osd.1) 178 : cluster [DBG] 16.b scrub ok 2024-02-10T08:41:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.735000+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:41:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:23.736918+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:41:25.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:24.033119+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.033186+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:24.034294+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:24.034639+0000 mon.a (mon.0) 2389 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.125924+0000 osd.5 (osd.5) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.126079+0000 osd.5 (osd.5) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:25.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:24.350890+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:41:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.659080+0000 osd.1 (osd.1) 177 : cluster [DBG] 16.b scrub starts 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.662117+0000 osd.1 (osd.1) 178 : cluster [DBG] 16.b scrub ok 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.735000+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:23.736918+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:24.033119+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:25.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.033186+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:24.034294+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:24.034639+0000 mon.a (mon.0) 2389 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.125924+0000 osd.5 (osd.5) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.126079+0000 osd.5 (osd.5) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:24.350890+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.659080+0000 osd.1 (osd.1) 177 : cluster [DBG] 16.b scrub starts 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.662117+0000 osd.1 (osd.1) 178 : cluster [DBG] 16.b scrub ok 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.735000+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:23.736918+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:24.033119+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.033186+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-10T08:41:25.373 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:24.034294+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.374 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:24.034639+0000 mon.a (mon.0) 2389 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:25.374 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.125924+0000 osd.5 (osd.5) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:25.374 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.126079+0000 osd.5 (osd.5) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:25.374 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:24.350890+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:41:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:25.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:25.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:25.503 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:25.909 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:41:25.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T08:41:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:25] "GET /metrics HTTP/1.1" 200 187294 "" "Prometheus/2.33.4" 2024-02-10T08:41:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:26.052 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:26.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:26.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:26.185 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:26.204 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:26.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.466181+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.469110+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.516154+0000 osd.7 (osd.7) 167 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.519139+0000 osd.7 (osd.7) 168 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.619782+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.622688+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.845977+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:24.847985+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:25.031391+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:26.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.035412+0000 mon.a (mon.0) 2392 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.035477+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:25.035509+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.065123+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.065468+0000 mon.a (mon.0) 2395 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.066118+0000 mon.b (mon.1) 557 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.066322+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:25.156279+0000 mon.a (mon.0) 2397 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:25.430866+0000 mgr.y (mgr.14553) 350 : cluster [DBG] pgmap v561: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.466181+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.469110+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:41:26.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.516154+0000 osd.7 (osd.7) 167 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.519139+0000 osd.7 (osd.7) 168 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.619782+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.622688+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.845977+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:24.847985+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:25.031391+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.035412+0000 mon.a (mon.0) 2392 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.035477+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:25.035509+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.065123+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.065468+0000 mon.a (mon.0) 2395 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.066118+0000 mon.b (mon.1) 557 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.066322+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:25.156279+0000 mon.a (mon.0) 2397 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:26.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:25.430866+0000 mgr.y (mgr.14553) 350 : cluster [DBG] pgmap v561: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:26.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:26.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.466181+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:41:26.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.469110+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:41:26.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.516154+0000 osd.7 (osd.7) 167 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:41:26.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.519139+0000 osd.7 (osd.7) 168 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:41:26.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.619782+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.622688+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.845977+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:24.847985+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:25.031391+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.035412+0000 mon.a (mon.0) 2392 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]': finished 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.035477+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? v1:172.21.15.107:0/2591084435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:25.035509+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.065123+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.065468+0000 mon.a (mon.0) 2395 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.066118+0000 mon.b (mon.1) 557 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.066322+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-49"}]: dispatch 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:25.156279+0000 mon.a (mon.0) 2397 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:26.334 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:25.430866+0000 mgr.y (mgr.14553) 350 : cluster [DBG] pgmap v561: 218 pgs: 1 activating, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 946 B/s rd, 0 op/s 2024-02-10T08:41:26.335 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:26.354 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:26.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:26.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:26.489 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:26.508 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:26.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:26.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:26.639 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:26.091959+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-10T08:41:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:26.344642+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:41:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:26.091959+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-10T08:41:27.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:26.344642+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:41:27.287 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:27.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:27.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:27.424 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:26.091959+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-10T08:41:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:26.344642+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:41:27.623 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:27.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:27.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:27.759 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:27.780 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:27.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:27.913 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:27.933 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:27.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:28.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:28.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:41:28.065 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:28.139 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:28.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:28.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:27.099839+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-10T08:41:28.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:27.105835+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:27.106220+0000 mon.a (mon.0) 2401 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:27.431260+0000 mgr.y (mgr.14553) 351 : cluster [DBG] pgmap v564: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:27.099839+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-10T08:41:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:27.105835+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:27.106220+0000 mon.a (mon.0) 2401 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:27.431260+0000 mgr.y (mgr.14553) 351 : cluster [DBG] pgmap v564: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 -- 172.21.15.107:0/3833625014 >> v1:172.21.15.107:6790/0 conn(0x7fc6f00fda30 legacy=0x7fc6f00fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 -- 172.21.15.107:0/3833625014 shutdown_connections 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 -- 172.21.15.107:0/3833625014 >> 172.21.15.107:0/3833625014 conn(0x7fc6f00f3630 msgr2=0x7fc6f00f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 -- 172.21.15.107:0/3833625014 shutdown_connections 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 -- 172.21.15.107:0/3833625014 wait complete. 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 librados: starting msgr at 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 librados: starting objecter 2024-02-10T08:41:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.236+0000 7fc6f54df700 1 Processor -- start 2024-02-10T08:41:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 -- start start 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 librados: setting wanted keys 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 librados: calling monclient init 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f0117430 con 0x7fc6f0104320 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f0118630 con 0x7fc6f0100810 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6f54df700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6f0119830 con 0x7fc6f00fda30 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6eeffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc6f00fda30 0x7fc6f0100bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52842/0 (socket says 172.21.15.107:52842) 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6eeffd700 1 -- 172.21.15.107:0/3316421142 learned_addr learned my addr 172.21.15.107:0/3316421142 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter ms_handle_connect 0x7fc6f00fda30 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter ms_handle_connect 0x7fc6f0100810 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter ms_handle_connect 0x7fc6f0104320 2024-02-10T08:41:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936053081 0 0) 0x7fc6f0119830 con 0x7fc6f00fda30 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d80033f0 con 0x7fc6f00fda30 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350676872 0 0) 0x7fc6f0118630 con 0x7fc6f0100810 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0119830 con 0x7fc6f0100810 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809584394 0 0) 0x7fc6f0117430 con 0x7fc6f0104320 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.237+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0118630 con 0x7fc6f0104320 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2660123410 0 0) 0x7fc6f0119830 con 0x7fc6f0100810 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f0117430 con 0x7fc6f0100810 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880673341 0 0) 0x7fc6f0118630 con 0x7fc6f0104320 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f0119830 con 0x7fc6f0104320 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536631647 0 0) 0x7fc6d80033f0 con 0x7fc6f00fda30 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f0118630 con 0x7fc6f00fda30 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc6e00028f0 con 0x7fc6f0100810 2024-02-10T08:41:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc6e80019a0 con 0x7fc6f0104320 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc6e4002050 con 0x7fc6f00fda30 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194001354 0 0) 0x7fc6f0117430 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6790/0 conn(0x7fc6f00fda30 legacy=0x7fc6f0100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6789/0 conn(0x7fc6f0104320 legacy=0x7fc6f0115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f011aa30 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f0118820 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6f54df700 10 client.48268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f0118c90 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6f54df700 1 librados: init done 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc6e00017d0 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.238+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc6e0003310 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.239+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc6dc004f00 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.239+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc6e0016240 con 0x7fc6f0100810 2024-02-10T08:41:28.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(392..392 src has 275..392) v4 ==== 8178+0+0 (unknown 3522239429 0 0) 0x7fc6e0076670 con 0x7fc6f0100810 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 osd_map(392..392 src has 275..392) v4 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 3 client.48268.objecter handle_osd_map got epochs [392,392] > 0 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 3 client.48268.objecter handle_osd_map decoding full epoch 392 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 20 client.48268.objecter dump_active .. 0 homeless 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v392) v1 ==== 54+0+16 (unknown 2199643977 0 2420000192) 0x7fc6e0078a60 con 0x7fc6f0100810 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6d8063d60 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 10 client.48268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 1 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 1 2024-02-10T08:41:28.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 0 1 2024-02-10T08:41:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 4 2024-02-10T08:41:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 10 client.48268.objecter _send_command 1 2024-02-10T08:41:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc004950 con 0x7fc6dc004530 2024-02-10T08:41:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.240+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc004530 2024-02-10T08:41:28.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.241+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc004950 con 0x7fc6dc004530 2024-02-10T08:41:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.241+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:41:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.241+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 1 = system:0 2024-02-10T08:41:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.241+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 4 2024-02-10T08:41:28.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.241+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 0 1 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 10 client.48268.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 2 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 2 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc0051a0 osd=0 3 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 0 2 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 4 2024-02-10T08:41:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 10 client.48268.objecter _send_command 2 2024-02-10T08:41:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.257+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc004d30 con 0x7fc6dc004530 2024-02-10T08:41:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc004d30 con 0x7fc6dc004530 2024-02-10T08:41:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 4 2024-02-10T08:41:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 0 2 2024-02-10T08:41:28.261 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 10 client.48268.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 1 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 4 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 10 client.48268.objecter _send_command 3 2024-02-10T08:41:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.259+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc00c000 con 0x7fc6dc004150 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc004150 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc00c000 con 0x7fc6dc004150 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 3 = system:0 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 4 2024-02-10T08:41:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.260+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 1 3 2024-02-10T08:41:28.275 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 10 client.48268.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 4 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 4 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc003fe0 osd=1 3 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 1 4 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 4 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 10 client.48268.objecter _send_command 4 2024-02-10T08:41:28.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.276+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc0018b0 con 0x7fc6dc004150 2024-02-10T08:41:28.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc0018b0 con 0x7fc6dc004150 2024-02-10T08:41:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 4 2024-02-10T08:41:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 1 4 2024-02-10T08:41:28.280 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 10 client.48268.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 5 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 5 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 2 5 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 4 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 10 client.48268.objecter _send_command 5 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.278+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc00de60 con 0x7fc6dc00d2f0 2024-02-10T08:41:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.279+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc00d2f0 2024-02-10T08:41:28.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.280+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc00de60 con 0x7fc6dc00d2f0 2024-02-10T08:41:28.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.280+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:41:28.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.280+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 5 = system:0 2024-02-10T08:41:28.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.280+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 4 2024-02-10T08:41:28.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.280+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 2 5 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 10 client.48268.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 6 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 6 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc003630 osd=2 3 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 2 6 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 4 2024-02-10T08:41:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 10 client.48268.objecter _send_command 6 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.296+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc0030a0 con 0x7fc6dc00d2f0 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc0030a0 con 0x7fc6dc00d2f0 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 4 2024-02-10T08:41:28.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 2 6 2024-02-10T08:41:28.300 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:41:28.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 10 client.48268.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 7 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 7 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 3 7 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 4 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 10 client.48268.objecter _send_command 7 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc015ad0 con 0x7fc6dc011e80 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.298+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc011e80 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.299+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc015ad0 con 0x7fc6dc011e80 2024-02-10T08:41:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.299+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:41:28.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.299+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 7 = system:0 2024-02-10T08:41:28.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.299+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 4 2024-02-10T08:41:28.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.299+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 3 7 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 10 client.48268.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 8 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 8 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc0034b0 osd=3 3 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 3 8 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 4 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 10 client.48268.objecter _send_command 8 2024-02-10T08:41:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.313+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc00e240 con 0x7fc6dc011e80 2024-02-10T08:41:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc00e240 con 0x7fc6dc011e80 2024-02-10T08:41:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 4 2024-02-10T08:41:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 3 8 2024-02-10T08:41:28.317 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 10 client.48268.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 9 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 9 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 4 9 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 4 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 10 client.48268.objecter _send_command 9 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.315+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc00e210 con 0x7fc6dc00ce50 2024-02-10T08:41:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc00ce50 2024-02-10T08:41:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc00e210 con 0x7fc6dc00ce50 2024-02-10T08:41:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:41:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 9 = system:0 2024-02-10T08:41:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 4 2024-02-10T08:41:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.316+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 4 9 2024-02-10T08:41:28.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 10 client.48268.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 10 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 10 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc00e000 osd=4 3 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 4 10 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 4 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 10 client.48268.objecter _send_command 10 2024-02-10T08:41:28.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.331+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc00e210 con 0x7fc6dc00ce50 2024-02-10T08:41:28.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc00e210 con 0x7fc6dc00ce50 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 4 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 4 10 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:41:28.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 10 client.48268.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 11 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 11 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 5 11 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 4 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 10 client.48268.objecter _send_command 11 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.333+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc018090 con 0x7fc6dc017af0 2024-02-10T08:41:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.334+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc017af0 2024-02-10T08:41:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.335+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc018090 con 0x7fc6dc017af0 2024-02-10T08:41:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.335+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:41:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.335+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 11 = system:0 2024-02-10T08:41:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.335+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 4 2024-02-10T08:41:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.335+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 5 11 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 10 client.48268.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 12 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 12 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc00e210 osd=5 3 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 5 12 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 4 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 10 client.48268.objecter _send_command 12 2024-02-10T08:41:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.348+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc018090 con 0x7fc6dc017af0 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.350+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc018090 con 0x7fc6dc017af0 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.350+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.350+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.350+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 4 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.350+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 5 12 2024-02-10T08:41:28.352 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 10 client.48268.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 13 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 13 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 6 13 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 4 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 10 client.48268.objecter _send_command 13 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc022ca0 con 0x7fc6dc01f0d0 2024-02-10T08:41:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.351+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc01f0d0 2024-02-10T08:41:28.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.352+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc022ca0 con 0x7fc6dc01f0d0 2024-02-10T08:41:28.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.352+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:41:28.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.352+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 13 = system:0 2024-02-10T08:41:28.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.352+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 4 2024-02-10T08:41:28.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.352+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 6 13 2024-02-10T08:41:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 10 client.48268.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 14 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 14 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc018090 osd=6 3 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 6 14 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 4 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 10 client.48268.objecter _send_command 14 2024-02-10T08:41:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.366+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc024450 con 0x7fc6dc01f0d0 2024-02-10T08:41:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc024450 con 0x7fc6dc01f0d0 2024-02-10T08:41:28.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 4 2024-02-10T08:41:28.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 6 14 2024-02-10T08:41:28.370 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 10 client.48268.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 15 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 15 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 7 15 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 4 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 10 client.48268.objecter _send_command 15 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.368+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6dc024f60 con 0x7fc6dc024ad0 2024-02-10T08:41:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.369+0000 7fc6d7fff700 10 client.48268.objecter ms_handle_connect 0x7fc6dc024ad0 2024-02-10T08:41:28.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.370+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc6dc024f60 con 0x7fc6dc024ad0 2024-02-10T08:41:28.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.370+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:41:28.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.370+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 15 = system:0 2024-02-10T08:41:28.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.370+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 4 2024-02-10T08:41:28.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.370+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 7 15 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 10 client.48268.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign -1 16 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 20 client.48268.objecter _get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_remove -1 16 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 20 client.48268.objecter get_session s=0x7fc6dc024450 osd=7 3 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 15 client.48268.objecter _session_command_op_assign 7 16 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 4 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 10 client.48268.objecter _send_command 16 2024-02-10T08:41:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.385+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6dc024f60 con 0x7fc6dc024ad0 2024-02-10T08:41:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.386+0000 7fc6d7fff700 1 -- 172.21.15.107:0/3316421142 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc6dc024f60 con 0x7fc6dc024ad0 2024-02-10T08:41:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.386+0000 7fc6d7fff700 10 client.48268.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.386+0000 7fc6d7fff700 10 client.48268.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.386+0000 7fc6d7fff700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 4 2024-02-10T08:41:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.386+0000 7fc6d7fff700 15 client.48268.objecter _session_command_op_remove 7 16 2024-02-10T08:41:28.389 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 librados: watch_flush enter 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 librados: watch_flush exit 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.0 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6801/3321596972 conn(0x7fc6dc004530 legacy=0x7fc6dc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0051a0 osd=0 1 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.1 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6805/133516126 conn(0x7fc6dc004150 legacy=0x7fc6dc00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003fe0 osd=1 1 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.2 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6809/3575881728 conn(0x7fc6dc00d2f0 legacy=0x7fc6dc00d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc003630 osd=2 1 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.3 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6813/2871825691 conn(0x7fc6dc011e80 legacy=0x7fc6dc014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc0034b0 osd=3 1 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.4 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.195:6800/2499671285 conn(0x7fc6dc00ce50 legacy=0x7fc6dc015c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e000 osd=4 1 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.5 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.195:6804/4178321486 conn(0x7fc6dc017af0 legacy=0x7fc6dc01d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc00e210 osd=5 1 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.6 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.195:6808/1417479017 conn(0x7fc6dc01f0d0 legacy=0x7fc6dc0214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc018090 osd=6 1 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter close_session for osd.7 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.195:6812/1934176928 conn(0x7fc6dc024ad0 legacy=0x7fc6dc022ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter put_session s=0x7fc6dc024450 osd=7 1 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 20 client.48268.objecter shutdown clearing up homeless session... 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6f54df700 10 client.48268.objecter successfully canceled tick 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6ef7fe700 1 -- 172.21.15.107:0/3316421142 reap_dead start 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6ef7fe700 1 -- 172.21.15.107:0/3316421142 reap_dead start 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6ef7fe700 1 -- 172.21.15.107:0/3316421142 reap_dead start 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6ef7fe700 1 -- 172.21.15.107:0/3316421142 reap_dead start 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.387+0000 7fc6ef7fe700 1 -- 172.21.15.107:0/3316421142 reap_dead start 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.388+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc6d8063d60 legacy=0x7fc6d8066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.388+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> v1:172.21.15.195:6789/0 conn(0x7fc6f0100810 legacy=0x7fc6f0101400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.388+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 shutdown_connections 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.388+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 >> 172.21.15.107:0/3316421142 conn(0x7fc6f00f3630 msgr2=0x7fc6f0107890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.388+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 shutdown_connections 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.389+0000 7fc6f54df700 1 -- 172.21.15.107:0/3316421142 wait complete. 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:28.389+0000 7fc6f54df700 1 librados: shutdown 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:41:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:28.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:28.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:41:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:28.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:28.535 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:27.099839+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-10T08:41:28.535 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:27.105835+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.535 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:27.106220+0000 mon.a (mon.0) 2401 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:28.535 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:27.431260+0000 mgr.y (mgr.14553) 351 : cluster [DBG] pgmap v564: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:28.537 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:28.557 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:28.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:28.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:28.688 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:28.702 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:28.707 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:28.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:28.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:28.838 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:28.857 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:28.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:28.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:29.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:28.094210+0000 mon.a (mon.0) 2402 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.103881+0000 mon.a (mon.0) 2403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:28.104000+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.139831+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.140165+0000 mon.a (mon.0) 2405 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.140784+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.140981+0000 mon.a (mon.0) 2406 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:28.241694+0000 mon.b (mon.1) 561 : audit [DBG] from='client.? v1:172.21.15.107:0/3316421142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:29.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:28.094210+0000 mon.a (mon.0) 2402 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.103881+0000 mon.a (mon.0) 2403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:28.104000+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.139831+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.140165+0000 mon.a (mon.0) 2405 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.140784+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.140981+0000 mon.a (mon.0) 2406 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.166 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:28.241694+0000 mon.b (mon.1) 561 : audit [DBG] from='client.? v1:172.21.15.107:0/3316421142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:29.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:28.094210+0000 mon.a (mon.0) 2402 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.103881+0000 mon.a (mon.0) 2403 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:28.104000+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.139831+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.140165+0000 mon.a (mon.0) 2405 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.140784+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.140981+0000 mon.a (mon.0) 2406 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-51"}]: dispatch 2024-02-10T08:41:29.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:28.241694+0000 mon.b (mon.1) 561 : audit [DBG] from='client.? v1:172.21.15.107:0/3316421142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:29.838 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:29.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:29.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:29.985 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:30.006 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:30.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:30.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:30.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:29.023008+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:30.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:29.023162+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:30.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:29.107609+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-10T08:41:30.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:29.431683+0000 mgr.y (mgr.14553) 352 : cluster [DBG] pgmap v567: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:30.145 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:30.165 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:30.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:30.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:30.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:30.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:29.023008+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:30.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:29.023162+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:30.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:29.107609+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-10T08:41:30.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:29.431683+0000 mgr.y (mgr.14553) 352 : cluster [DBG] pgmap v567: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:29.023008+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:29.023162+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:29.107609+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-10T08:41:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:29.431683+0000 mgr.y (mgr.14553) 352 : cluster [DBG] pgmap v567: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T08:41:30.844 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:30.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:30.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:30.999 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:31.038 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:31.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:31.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:31.169 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:31.188 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:31.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:30.118745+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:30.123136+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:30.125641+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.013193+0000 mon.a (mon.0) 2410 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:31.013314+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-10T08:41:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.015579+0000 mon.b (mon.1) 563 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:31.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.061168+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.061575+0000 mon.a (mon.0) 2412 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.062019+0000 mon.b (mon.1) 565 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.062177+0000 mon.a (mon.0) 2413 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:31.319 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:31.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:30.118745+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-10T08:41:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:30.123136+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:30.125641+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.013193+0000 mon.a (mon.0) 2410 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:31.013314+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.015579+0000 mon.b (mon.1) 563 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.061168+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.061575+0000 mon.a (mon.0) 2412 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.062019+0000 mon.b (mon.1) 565 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.062177+0000 mon.a (mon.0) 2413 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:30.118745+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:30.123136+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:30.125641+0000 mon.a (mon.0) 2409 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.013193+0000 mon.a (mon.0) 2410 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:31.013314+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.015579+0000 mon.b (mon.1) 563 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.061168+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.061575+0000 mon.a (mon.0) 2412 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.062019+0000 mon.b (mon.1) 565 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.062177+0000 mon.a (mon.0) 2413 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-53"}]: dispatch 2024-02-10T08:41:31.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:31.759 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:31.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:31.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:31.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:31.890 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.974+0000 7fe1de937700 1 -- 172.21.15.107:0/745235457 >> v1:172.21.15.107:6790/0 conn(0x7fe1d80f67b0 legacy=0x7fe1d80f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 -- 172.21.15.107:0/745235457 shutdown_connections 2024-02-10T08:41:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 -- 172.21.15.107:0/745235457 >> 172.21.15.107:0/745235457 conn(0x7fe1d80f2510 msgr2=0x7fe1d80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 -- 172.21.15.107:0/745235457 shutdown_connections 2024-02-10T08:41:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 -- 172.21.15.107:0/745235457 wait complete. 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 librados: starting msgr at 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.975+0000 7fe1de937700 1 librados: starting objecter 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 Processor -- start 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 -- start start 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 librados: setting wanted keys 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 librados: calling monclient init 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1d8119630 con 0x7fe1d80f67b0 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1d811a7d0 con 0x7fe1d80f72a0 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1de937700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1d811b9d0 con 0x7fe1d8106680 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1dced4700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe1d8106680 0x7fe1d8117cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39760/0 (socket says 172.21.15.107:39760) 2024-02-10T08:41:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1dced4700 1 -- 172.21.15.107:0/1760123166 learned_addr learned my addr 172.21.15.107:0/1760123166 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d7fff700 1 --1- 172.21.15.107:0/1760123166 >> v1:172.21.15.107:6789/0 conn(0x7fe1d80f67b0 0x7fe1d8101af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48440/0 (socket says 172.21.15.107:48440) 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter ms_handle_connect 0x7fe1d8106680 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter ms_handle_connect 0x7fe1d80f67b0 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214187055 0 0) 0x7fe1d811b9d0 con 0x7fe1d8106680 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c00033f0 con 0x7fe1d8106680 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter ms_handle_connect 0x7fe1d80f72a0 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.976+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130021168 0 0) 0x7fe1d8119630 con 0x7fe1d80f67b0 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d811b9d0 con 0x7fe1d80f67b0 2024-02-10T08:41:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423889244 0 0) 0x7fe1d811a7d0 con 0x7fe1d80f72a0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d8119630 con 0x7fe1d80f72a0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436373412 0 0) 0x7fe1c00033f0 con 0x7fe1d8106680 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d811a7d0 con 0x7fe1d8106680 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe1d0002c90 con 0x7fe1d8106680 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 795015725 0 0) 0x7fe1d811b9d0 con 0x7fe1d80f67b0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c00033f0 con 0x7fe1d80f67b0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289270139 0 0) 0x7fe1d8119630 con 0x7fe1d80f72a0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d811b9d0 con 0x7fe1d80f72a0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe1c8002050 con 0x7fe1d80f67b0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe1cc0015b0 con 0x7fe1d80f72a0 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1669889816 0 0) 0x7fe1d811a7d0 con 0x7fe1d8106680 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 >> v1:172.21.15.195:6789/0 conn(0x7fe1d80f72a0 legacy=0x7fe1d8102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 >> v1:172.21.15.107:6789/0 conn(0x7fe1d80f67b0 legacy=0x7fe1d8101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d811cbd0 con 0x7fe1d8106680 2024-02-10T08:41:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d8119820 con 0x7fe1d8106680 2024-02-10T08:41:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1de937700 10 client.53561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d8119c90 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1de937700 1 librados: init done 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe1d0001b50 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.977+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe1d0005e50 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.978+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c4004f20 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.978+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe1d0005fb0 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(395..395 src has 275..395) v4 ==== 8178+0+0 (unknown 2824162046 0 0) 0x7fe1d0078fe0 con 0x7fe1d8106680 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 10 client.53561.objecter ms_dispatch 0x7fe1d8001550 osd_map(395..395 src has 275..395) v4 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 3 client.53561.objecter handle_osd_map got epochs [395,395] > 0 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 3 client.53561.objecter handle_osd_map decoding full epoch 395 2024-02-10T08:41:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 20 client.53561.objecter dump_active .. 0 homeless 2024-02-10T08:41:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.979+0000 7fe1d57fa700 10 client.53561.objecter ms_handle_connect 0x7fe1c0063d50 2024-02-10T08:41:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:31.983+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe1d004c3c0 con 0x7fe1d8106680 2024-02-10T08:41:31.988 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:31.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:32.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:32.125 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.185+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fe1c40051d0 con 0x7fe1d8106680 2024-02-10T08:41:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:32.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:32.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:32.333 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:32.360 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:32.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:31.432492+0000 mgr.y (mgr.14553) 353 : cluster [DBG] pgmap v570: 218 pgs: 6 creating+activating, 212 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:41:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.432626+0000 mgr.y (mgr.14553) 354 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:31.678012+0000 mon.c (mon.2) 384 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:41:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:32.026744+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-10T08:41:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:31.432492+0000 mgr.y (mgr.14553) 353 : cluster [DBG] pgmap v570: 218 pgs: 6 creating+activating, 212 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:41:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.432626+0000 mgr.y (mgr.14553) 354 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:31.678012+0000 mon.c (mon.2) 384 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:41:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:32.026744+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-10T08:41:32.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:32.491 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:32.509 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:32.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:32.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:32.637 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:31.432492+0000 mgr.y (mgr.14553) 353 : cluster [DBG] pgmap v570: 218 pgs: 6 creating+activating, 212 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T08:41:32.637 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.432626+0000 mgr.y (mgr.14553) 354 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:32.637 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:31.678012+0000 mon.c (mon.2) 384 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:41:32.638 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:32.026744+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-10T08:41:32.639 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:32.659 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:32.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.675+0000 7f602efad700 1 -- 172.21.15.107:0/2338231927 >> v1:172.21.15.107:6790/0 conn(0x7f6028100840 legacy=0x7f6028102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.675+0000 7f602efad700 1 -- 172.21.15.107:0/2338231927 shutdown_connections 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.675+0000 7f602efad700 1 -- 172.21.15.107:0/2338231927 >> 172.21.15.107:0/2338231927 conn(0x7f602806c090 msgr2=0x7f602806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.675+0000 7f602efad700 1 -- 172.21.15.107:0/2338231927 shutdown_connections 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.675+0000 7f602efad700 1 -- 172.21.15.107:0/2338231927 wait complete. 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 librados: starting msgr at 2024-02-10T08:41:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 librados: starting objecter 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 Processor -- start 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 -- start start 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 librados: setting wanted keys 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 librados: calling monclient init 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60281174f0 con 0x7f6028104350 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60281186f0 con 0x7f60280fd9f0 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f602efad700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60281198f0 con 0x7f6028100840 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6027fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6028100840 0x7f60280fa540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39778/0 (socket says 172.21.15.107:39778) 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6027fff700 1 -- 172.21.15.107:0/549519310 learned_addr learned my addr 172.21.15.107:0/549519310 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter ms_handle_connect 0x7f6028104350 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter ms_handle_connect 0x7f6028100840 2024-02-10T08:41:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter ms_handle_connect 0x7f60280fd9f0 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865999924 0 0) 0x7f60281174f0 con 0x7f6028104350 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60180038e0 con 0x7f6028104350 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161862248 0 0) 0x7f60281198f0 con 0x7f6028100840 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.676+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60281174f0 con 0x7f6028100840 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348322993 0 0) 0x7f60281186f0 con 0x7f60280fd9f0 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60281198f0 con 0x7f60280fd9f0 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230976061 0 0) 0x7f60180038e0 con 0x7f6028104350 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60281186f0 con 0x7f6028104350 2024-02-10T08:41:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6020001b80 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644772537 0 0) 0x7f60281174f0 con 0x7f6028100840 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60180038e0 con 0x7f6028100840 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f60140015d0 con 0x7f6028100840 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736994029 0 0) 0x7f60281198f0 con 0x7f60280fd9f0 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60281174f0 con 0x7f60280fd9f0 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f601c0018d0 con 0x7f60280fd9f0 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2265339621 0 0) 0x7f60281186f0 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 >> v1:172.21.15.107:6790/0 conn(0x7f6028100840 legacy=0x7f60280fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 >> v1:172.21.15.195:6789/0 conn(0x7f60280fd9f0 legacy=0x7f60280f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602811aaf0 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6028119ae0 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6020002290 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6020003620 con 0x7f6028104350 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f602efad700 10 client.27984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:32.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6028119f50 con 0x7f6028104350 2024-02-10T08:41:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.677+0000 7f602efad700 1 librados: init done 2024-02-10T08:41:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.678+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602805f5a0 con 0x7f6028104350 2024-02-10T08:41:32.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6020001df0 con 0x7f6028104350 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(396..396 src has 275..396) v4 ==== 7792+0+0 (unknown 2076434593 0 0) 0x7f60200766e0 con 0x7f6028104350 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 10 client.27984.objecter ms_dispatch 0x7f6028001550 osd_map(396..396 src has 275..396) v4 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 3 client.27984.objecter handle_osd_map got epochs [396,396] > 0 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 3 client.27984.objecter handle_osd_map decoding full epoch 396 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 20 client.27984.objecter dump_active .. 0 homeless 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.679+0000 7f6025ffb700 10 client.27984.objecter ms_handle_connect 0x7f6018063ca0 2024-02-10T08:41:32.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.683+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6020049c40 con 0x7f6028104350 2024-02-10T08:41:32.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:32.791 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:32.866+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f60280475e0 con 0x7f6028104350 2024-02-10T08:41:33.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.030+0000 7f6025ffb700 1 -- 172.21.15.107:0/549519310 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v397) v1 ==== 97+0+0 (unknown 3531858607 0 0) 0x7f6020048050 con 0x7f6028104350 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.033+0000 7f602efad700 10 librados: watch_flush enter 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.033+0000 7f602efad700 10 librados: watch_flush exit 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.033+0000 7f602efad700 20 client.27984.objecter shutdown clearing up homeless session... 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.033+0000 7f602efad700 10 client.27984.objecter successfully canceled tick 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.034+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6018063ca0 legacy=0x7f6018066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.034+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 >> v1:172.21.15.107:6789/0 conn(0x7f6028104350 legacy=0x7f6028115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.034+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 shutdown_connections 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.034+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 >> 172.21.15.107:0/549519310 conn(0x7f602806c090 msgr2=0x7f6028106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:33.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.034+0000 7fe1d57fa700 1 -- 172.21.15.107:0/1760123166 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v397) v1 ==== 113+0+0 (unknown 3161392884 0 0) 0x7fe1d004a7d0 con 0x7fe1d8106680 2024-02-10T08:41:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 shutdown_connections 2024-02-10T08:41:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7f602efad700 1 -- 172.21.15.107:0/549519310 wait complete. 2024-02-10T08:41:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7f602efad700 1 librados: shutdown 2024-02-10T08:41:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 10 librados: watch_flush enter 2024-02-10T08:41:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 10 librados: watch_flush exit 2024-02-10T08:41:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 20 client.53561.objecter shutdown clearing up homeless session... 2024-02-10T08:41:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 10 client.53561.objecter successfully canceled tick 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe1c0063d50 legacy=0x7fe1c0066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.037+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 >> v1:172.21.15.107:6790/0 conn(0x7fe1d8106680 legacy=0x7fe1d8117cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.038+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 shutdown_connections 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.038+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 >> 172.21.15.107:0/1760123166 conn(0x7fe1d80f2510 msgr2=0x7fe1d8109ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.038+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 shutdown_connections 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.038+0000 7fe1de937700 1 -- 172.21.15.107:0/1760123166 wait complete. 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:33.038+0000 7fe1de937700 1 librados: shutdown 2024-02-10T08:41:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:41:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T08:41:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.118597+0000 mon.c (mon.2) 385 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.119320+0000 mon.c (mon.2) 386 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.187641+0000 mon.c (mon.2) 387 : audit [INF] from='client.? v1:172.21.15.107:0/1760123166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.295145+0000 mon.a (mon.0) 2415 : audit [INF] from='client.53561 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.301338+0000 mon.a (mon.0) 2416 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:33.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:32.868078+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:33.025345+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:33.030537+0000 mon.a (mon.0) 2419 : audit [INF] from='client.53561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:33.030618+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:33.030688+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:33.036742+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:33.037950+0000 mon.a (mon.0) 2422 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.118597+0000 mon.c (mon.2) 385 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.119320+0000 mon.c (mon.2) 386 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.187641+0000 mon.c (mon.2) 387 : audit [INF] from='client.? v1:172.21.15.107:0/1760123166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.295145+0000 mon.a (mon.0) 2415 : audit [INF] from='client.53561 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.301338+0000 mon.a (mon.0) 2416 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:33.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:32.868078+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:33.025345+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:33.030537+0000 mon.a (mon.0) 2419 : audit [INF] from='client.53561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:33.030618+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:33.030688+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:33.036742+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:33.037950+0000 mon.a (mon.0) 2422 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:33.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:33.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:33.478 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:33.499 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:33.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:33.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:33.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.118597+0000 mon.c (mon.2) 385 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.119320+0000 mon.c (mon.2) 386 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.187641+0000 mon.c (mon.2) 387 : audit [INF] from='client.? v1:172.21.15.107:0/1760123166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.295145+0000 mon.a (mon.0) 2415 : audit [INF] from='client.53561 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.301338+0000 mon.a (mon.0) 2416 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:32.868078+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:33.025345+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:33.030537+0000 mon.a (mon.0) 2419 : audit [INF] from='client.53561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:33.030618+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? v1:172.21.15.107:0/549519310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:33.030688+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-10T08:41:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:33.036742+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:33.037950+0000 mon.a (mon.0) 2422 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:33.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:33.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:34.015 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:34.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T08:41:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:34.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:33.433105+0000 mgr.y (mgr.14553) 355 : cluster [DBG] pgmap v573: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:34.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:34.031411+0000 mon.a (mon.0) 2423 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:34.031487+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:34.073081+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:34.073383+0000 mon.a (mon.0) 2425 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:34.073878+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:34.074050+0000 mon.a (mon.0) 2426 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:33.433105+0000 mgr.y (mgr.14553) 355 : cluster [DBG] pgmap v573: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:34.031411+0000 mon.a (mon.0) 2423 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:34.031487+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:34.073081+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:34.073383+0000 mon.a (mon.0) 2425 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.671 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:34.073878+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.672 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:34.074050+0000 mon.a (mon.0) 2426 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:34.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:34.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:33.433105+0000 mgr.y (mgr.14553) 355 : cluster [DBG] pgmap v573: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:34.031411+0000 mon.a (mon.0) 2423 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:34.031487+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:34.073081+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:34.073383+0000 mon.a (mon.0) 2425 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:34.073878+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:34.074050+0000 mon.a (mon.0) 2426 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-55"}]: dispatch 2024-02-10T08:41:34.800 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:35] "GET /metrics HTTP/1.1" 200 187294 "" "Prometheus/2.33.4" 2024-02-10T08:41:36.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:35.037899+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-10T08:41:36.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:35.433499+0000 mgr.y (mgr.14553) 356 : cluster [DBG] pgmap v576: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:35.037899+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-10T08:41:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:35.433499+0000 mgr.y (mgr.14553) 356 : cluster [DBG] pgmap v576: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:35.037899+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-10T08:41:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:35.433499+0000 mgr.y (mgr.14553) 356 : cluster [DBG] pgmap v576: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:37.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:41:37.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:36.090075+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-10T08:41:37.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:36.098755+0000 mon.b (mon.1) 569 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:36.099686+0000 mon.a (mon.0) 2429 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:36.090075+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-10T08:41:37.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:36.098755+0000 mon.b (mon.1) 569 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:36.099686+0000 mon.a (mon.0) 2429 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:36.090075+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-10T08:41:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:36.098755+0000 mon.b (mon.1) 569 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:36.099686+0000 mon.a (mon.0) 2429 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:37.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:37.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.349+0000 7f02429c6700 1 -- 172.21.15.107:0/1173077787 >> v1:172.21.15.107:6790/0 conn(0x7f023c060a80 legacy=0x7f023c060f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.349+0000 7f02429c6700 1 -- 172.21.15.107:0/1173077787 shutdown_connections 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.349+0000 7f02429c6700 1 -- 172.21.15.107:0/1173077787 >> 172.21.15.107:0/1173077787 conn(0x7f023c0f2510 msgr2=0x7f023c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.349+0000 7f02429c6700 1 -- 172.21.15.107:0/1173077787 shutdown_connections 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.349+0000 7f02429c6700 1 -- 172.21.15.107:0/1173077787 wait complete. 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 librados: starting msgr at 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 librados: starting objecter 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 Processor -- start 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 -- start start 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 librados: setting wanted keys 2024-02-10T08:41:37.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 librados: calling monclient init 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f023c10fa60 con 0x7f023c060a80 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f023c107b10 con 0x7f023c0fc910 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f02429c6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f023c107ca0 con 0x7f023c061610 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f0233fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f023c061610 0x7f023c10c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39798/0 (socket says 172.21.15.107:39798) 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f0233fff700 1 -- 172.21.15.107:0/822706082 learned_addr learned my addr 172.21.15.107:0/822706082 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f0239ffb700 10 client.?.objecter ms_handle_connect 0x7f023c061610 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.350+0000 7f0239ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 10 client.?.objecter ms_handle_connect 0x7f023c060a80 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210216126 0 0) 0x7f023c107ca0 con 0x7f023c061610 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228003630 con 0x7f023c061610 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 10 client.?.objecter ms_handle_connect 0x7f023c0fc910 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570463767 0 0) 0x7f023c107b10 con 0x7f023c0fc910 2024-02-10T08:41:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c107ca0 con 0x7f023c0fc910 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819868607 0 0) 0x7f023c10fa60 con 0x7f023c060a80 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c107b10 con 0x7f023c060a80 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815943211 0 0) 0x7f0228003630 con 0x7f023c061610 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f023c10fa60 con 0x7f023c061610 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684077064 0 0) 0x7f023c107ca0 con 0x7f023c0fc910 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0228003630 con 0x7f023c0fc910 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549492144 0 0) 0x7f023c107b10 con 0x7f023c060a80 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f023c107ca0 con 0x7f023c060a80 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0224002e10 con 0x7f023c061610 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0234001b40 con 0x7f023c0fc910 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f022c0019b0 con 0x7f023c060a80 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4276036588 0 0) 0x7f023c10fa60 con 0x7f023c061610 2024-02-10T08:41:37.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 >> v1:172.21.15.195:6789/0 conn(0x7f023c0fc910 legacy=0x7f023c107400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 >> v1:172.21.15.107:6789/0 conn(0x7f023c060a80 legacy=0x7f023c10bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c107e30 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c11e5d0 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f02429c6700 10 client.53573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c11e980 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.351+0000 7f02429c6700 1 librados: init done 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.352+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0224001b20 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.352+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f02240038b0 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.352+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c004f20 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.353+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0224003ab0 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(401..401 src has 330..401) v4 ==== 8178+0+0 (unknown 71479126 0 0) 0x7f0224079070 con 0x7f023c061610 2024-02-10T08:41:37.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 10 client.53573.objecter ms_dispatch 0x7f023c001550 osd_map(401..401 src has 330..401) v4 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 3 client.53573.objecter handle_osd_map got epochs [401,401] > 0 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 3 client.53573.objecter handle_osd_map decoding full epoch 401 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 20 client.53573.objecter dump_active .. 0 homeless 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.354+0000 7f0239ffb700 10 client.53573.objecter ms_handle_connect 0x7f0228063d40 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.358+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f022404c450 con 0x7f023c061610 2024-02-10T08:41:37.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:37.550+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f021c0051d0 con 0x7f023c061610 2024-02-10T08:41:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.070+0000 7f0239ffb700 1 -- 172.21.15.107:0/822706082 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v402) v1 ==== 113+0+0 (unknown 88548534 0 0) 0x7f022404a8c0 con 0x7f023c061610 2024-02-10T08:41:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.073+0000 7f02429c6700 10 librados: watch_flush enter 2024-02-10T08:41:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.073+0000 7f02429c6700 10 librados: watch_flush exit 2024-02-10T08:41:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.073+0000 7f02429c6700 20 client.53573.objecter shutdown clearing up homeless session... 2024-02-10T08:41:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.073+0000 7f02429c6700 10 client.53573.objecter successfully canceled tick 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0228063d40 legacy=0x7f0228066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 >> v1:172.21.15.107:6790/0 conn(0x7f023c061610 legacy=0x7f023c10c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 shutdown_connections 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 >> 172.21.15.107:0/822706082 conn(0x7f023c0f2510 msgr2=0x7f023c0f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 shutdown_connections 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 -- 172.21.15.107:0/822706082 wait complete. 2024-02-10T08:41:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:38.074+0000 7f02429c6700 1 librados: shutdown 2024-02-10T08:41:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T08:41:38.244 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:41:38.244 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.067517+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:37.067617+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.084482+0000 mon.b (mon.1) 570 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.119997+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.120314+0000 mon.a (mon.0) 2432 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.120842+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.121002+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:37.433913+0000 mgr.y (mgr.14553) 357 : cluster [DBG] pgmap v579: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.551051+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.107:0/822706082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:37.551304+0000 mon.a (mon.0) 2434 : audit [INF] from='client.53573 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:38.065518+0000 mon.a (mon.0) 2435 : audit [INF] from='client.53573 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:38.065584+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.067517+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:37.067617+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.084482+0000 mon.b (mon.1) 570 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.119997+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.120314+0000 mon.a (mon.0) 2432 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.120842+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.121002+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:37.433913+0000 mgr.y (mgr.14553) 357 : cluster [DBG] pgmap v579: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:38.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.551051+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.107:0/822706082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:37.551304+0000 mon.a (mon.0) 2434 : audit [INF] from='client.53573 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:38.065518+0000 mon.a (mon.0) 2435 : audit [INF] from='client.53573 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:41:38.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:38.065584+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-10T08:41:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.067517+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:37.067617+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-10T08:41:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.084482+0000 mon.b (mon.1) 570 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.119997+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.120314+0000 mon.a (mon.0) 2432 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.120842+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.121002+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-57"}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:37.433913+0000 mgr.y (mgr.14553) 357 : cluster [DBG] pgmap v579: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.551051+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.107:0/822706082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:37.551304+0000 mon.a (mon.0) 2434 : audit [INF] from='client.53573 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:38.065518+0000 mon.a (mon.0) 2435 : audit [INF] from='client.53573 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:41:38.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:38.065584+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-10T08:41:38.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T08:41:39.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:41:39.732 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:39.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:39.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:39.861 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:39.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:39.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:40.023 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:40.044 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:40.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:40.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:40.174 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:40.193 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:40.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:40.244 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:38.240743+0000 osd.3 (osd.3) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:38.240871+0000 osd.3 (osd.3) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:39.068421+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:39.071722+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:39.073071+0000 mon.a (mon.0) 2438 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:39.434437+0000 mgr.y (mgr.14553) 358 : cluster [DBG] pgmap v582: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:40.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:40.322 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:40.340 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:40.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:38.240743+0000 osd.3 (osd.3) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:38.240871+0000 osd.3 (osd.3) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:39.068421+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-10T08:41:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:39.071722+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:39.073071+0000 mon.a (mon.0) 2438 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:39.434437+0000 mgr.y (mgr.14553) 358 : cluster [DBG] pgmap v582: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:38.240743+0000 osd.3 (osd.3) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:38.240871+0000 osd.3 (osd.3) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:39.068421+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:39.071722+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:39.073071+0000 mon.a (mon.0) 2438 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:40.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:39.434437+0000 mgr.y (mgr.14553) 358 : cluster [DBG] pgmap v582: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T08:41:40.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:40.470 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:40.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:40.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:40.619 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:40.637 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:40.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:40.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:40.768 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 -- 172.21.15.107:0/2345642257 >> v1:172.21.15.107:6790/0 conn(0x7f946c0f89e0 legacy=0x7f946c0f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 -- 172.21.15.107:0/2345642257 shutdown_connections 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 -- 172.21.15.107:0/2345642257 >> 172.21.15.107:0/2345642257 conn(0x7f946c0f2550 msgr2=0x7f946c0f4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 -- 172.21.15.107:0/2345642257 shutdown_connections 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 -- 172.21.15.107:0/2345642257 wait complete. 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 librados: starting msgr at 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 librados: starting objecter 2024-02-10T08:41:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.228+0000 7f9471bef700 1 Processor -- start 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 -- start start 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 librados: setting wanted keys 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 librados: calling monclient init 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946c117360 con 0x7f946c0f94d0 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946c118540 con 0x7f946c0f89e0 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9471bef700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946c119740 con 0x7f946c104340 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f946bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f946c104340 0x7f946c115a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52310/0 (socket says 172.21.15.107:52310) 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f946bfff700 1 -- 172.21.15.107:0/44431661 learned_addr learned my addr 172.21.15.107:0/44431661 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f946affd700 1 --1- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6789/0 conn(0x7f946c0f94d0 0x7f946c0ffbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39600/0 (socket says 172.21.15.107:39600) 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter ms_handle_connect 0x7f946c104340 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter ms_handle_connect 0x7f946c0f94d0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter ms_handle_connect 0x7f946c0f89e0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857622322 0 0) 0x7f946c119740 con 0x7f946c104340 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454003800 con 0x7f946c104340 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941136553 0 0) 0x7f946c117360 con 0x7f946c0f94d0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c119740 con 0x7f946c0f94d0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249905270 0 0) 0x7f946c118540 con 0x7f946c0f89e0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.229+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c117360 con 0x7f946c0f89e0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664211892 0 0) 0x7f9454003800 con 0x7f946c104340 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c118540 con 0x7f946c104340 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9464001d50 con 0x7f946c104340 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798450991 0 0) 0x7f946c117360 con 0x7f946c0f89e0 2024-02-10T08:41:41.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9454003800 con 0x7f946c0f89e0 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172168904 0 0) 0x7f946c119740 con 0x7f946c0f94d0 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c117360 con 0x7f946c0f94d0 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9460002bd0 con 0x7f946c0f89e0 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f945c0019b0 con 0x7f946c0f94d0 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 268330153 0 0) 0x7f946c118540 con 0x7f946c104340 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.195:6789/0 conn(0x7f946c0f89e0 legacy=0x7f946c0ff4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6789/0 conn(0x7f946c0f94d0 legacy=0x7f946c0ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c11a940 con 0x7f946c104340 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c117550 con 0x7f946c104340 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9471bef700 10 client.53579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c1179c0 con 0x7f946c104340 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9471bef700 1 librados: init done 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9464003040 con 0x7f946c104340 2024-02-10T08:41:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.230+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9464005e60 con 0x7f946c104340 2024-02-10T08:41:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.231+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9458004f00 con 0x7f946c104340 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.231+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f94640022e0 con 0x7f946c104340 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(405..405 src has 330..405) v4 ==== 8194+0+0 (unknown 217130706 0 0) 0x7f94640791e0 con 0x7f946c104340 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 osd_map(405..405 src has 330..405) v4 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 3 client.53579.objecter handle_osd_map got epochs [405,405] > 0 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 3 client.53579.objecter handle_osd_map decoding full epoch 405 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 20 client.53579.objecter dump_active .. 0 homeless 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v405) v1 ==== 54+0+16 (unknown 1542870172 0 2420000192) 0x7f946407b5f0 con 0x7f946c104340 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9454068140 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 10 client.53579.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 1 2024-02-10T08:41:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 1 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 0 1 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 4 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 10 client.53579.objecter _send_command 1 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458004950 con 0x7f9458004530 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.232+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9458004530 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.233+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458004950 con 0x7f9458004530 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.233+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.233+0000 7f9468ff9700 10 client.53579.objecter _finish_command 1 = system:0 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.233+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 4 2024-02-10T08:41:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.233+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 0 1 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 10 client.53579.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 2 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 2 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f94580051a0 osd=0 3 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 0 2 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 4 2024-02-10T08:41:41.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 10 client.53579.objecter _send_command 2 2024-02-10T08:41:41.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.248+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9458004d30 con 0x7f9458004530 2024-02-10T08:41:41.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9458004d30 con 0x7f9458004530 2024-02-10T08:41:41.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9468ff9700 10 client.53579.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 4 2024-02-10T08:41:41.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 0 2 2024-02-10T08:41:41.250 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 10 client.53579.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 1 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 4 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 10 client.53579.objecter _send_command 3 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.249+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f945800c000 con 0x7f9458004150 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9458004150 2024-02-10T08:41:41.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f945800c000 con 0x7f9458004150 2024-02-10T08:41:41.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:41:41.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 10 client.53579.objecter _finish_command 3 = system:0 2024-02-10T08:41:41.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 4 2024-02-10T08:41:41.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.250+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 1 3 2024-02-10T08:41:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 10 client.53579.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 4 2024-02-10T08:41:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 4 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003fe0 osd=1 3 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 1 4 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 4 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 10 client.53579.objecter _send_command 4 2024-02-10T08:41:41.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.266+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f94580018b0 con 0x7f9458004150 2024-02-10T08:41:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f94580018b0 con 0x7f9458004150 2024-02-10T08:41:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9468ff9700 10 client.53579.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 4 2024-02-10T08:41:41.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 1 4 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 10 client.53579.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 5 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 5 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 2 5 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 4 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 10 client.53579.objecter _send_command 5 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.268+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458003e70 con 0x7f945800d2f0 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f945800d2f0 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458003e70 con 0x7f945800d2f0 2024-02-10T08:41:41.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:41:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 10 client.53579.objecter _finish_command 5 = system:0 2024-02-10T08:41:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 4 2024-02-10T08:41:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.269+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 2 5 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 10 client.53579.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 6 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 6 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003630 osd=2 3 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 2 6 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 4 2024-02-10T08:41:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 10 client.53579.objecter _send_command 6 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.285+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f945800e730 con 0x7f945800d2f0 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f945800e730 con 0x7f945800d2f0 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 10 client.53579.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 4 2024-02-10T08:41:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 2 6 2024-02-10T08:41:41.288 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:41:41.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 10 client.53579.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 7 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 7 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 3 7 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 4 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 10 client.53579.objecter _send_command 7 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458015ac0 con 0x7f9458011e80 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.287+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9458011e80 2024-02-10T08:41:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.288+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458015ac0 con 0x7f9458011e80 2024-02-10T08:41:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.288+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:41:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.288+0000 7f9468ff9700 10 client.53579.objecter _finish_command 7 = system:0 2024-02-10T08:41:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.288+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 4 2024-02-10T08:41:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.288+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 3 7 2024-02-10T08:41:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 10 client.53579.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 8 2024-02-10T08:41:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 8 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458003410 osd=3 3 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 3 8 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 4 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 10 client.53579.objecter _send_command 8 2024-02-10T08:41:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.303+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f945800e1a0 con 0x7f9458011e80 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f945800e1a0 con 0x7f9458011e80 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 10 client.53579.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 4 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 3 8 2024-02-10T08:41:41.305 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 10 client.53579.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 9 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 9 2024-02-10T08:41:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 4 9 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 4 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 10 client.53579.objecter _send_command 9 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f945800e100 con 0x7f945800ce50 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.305+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f945800ce50 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.306+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f945800e100 con 0x7f945800ce50 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.306+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.306+0000 7f9468ff9700 10 client.53579.objecter _finish_command 9 = system:0 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.306+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 4 2024-02-10T08:41:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.306+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 4 9 2024-02-10T08:41:41.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 10 client.53579.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 10 2024-02-10T08:41:41.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 10 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f945800df60 osd=4 3 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 4 10 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 4 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 10 client.53579.objecter _send_command 10 2024-02-10T08:41:41.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.321+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f945800e100 con 0x7f945800ce50 2024-02-10T08:41:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f945800e100 con 0x7f945800ce50 2024-02-10T08:41:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9468ff9700 10 client.53579.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 4 2024-02-10T08:41:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 4 10 2024-02-10T08:41:41.324 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:41:41.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 10 client.53579.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 11 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 11 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 5 11 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 4 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 10 client.53579.objecter _send_command 11 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.323+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458017ff0 con 0x7f9458017a50 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9458017a50 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458017ff0 con 0x7f9458017a50 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:41:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 10 client.53579.objecter _finish_command 11 = system:0 2024-02-10T08:41:41.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 4 2024-02-10T08:41:41.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.324+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 5 11 2024-02-10T08:41:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 10 client.53579.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 12 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 12 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f945800e100 osd=5 3 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 5 12 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 4 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 10 client.53579.objecter _send_command 12 2024-02-10T08:41:41.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.339+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9458017ff0 con 0x7f9458017a50 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9458017ff0 con 0x7f9458017a50 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 10 client.53579.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 4 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 5 12 2024-02-10T08:41:41.341 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 10 client.53579.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 13 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 13 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 6 13 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 4 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 10 client.53579.objecter _send_command 13 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458022c00 con 0x7f945801f0c0 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.341+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f945801f0c0 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.342+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458022c00 con 0x7f945801f0c0 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.342+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.342+0000 7f9468ff9700 10 client.53579.objecter _finish_command 13 = system:0 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.342+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 4 2024-02-10T08:41:41.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.342+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 6 13 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 10 client.53579.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 14 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 14 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f9458017ff0 osd=6 3 2024-02-10T08:41:41.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 6 14 2024-02-10T08:41:41.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 4 2024-02-10T08:41:41.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 10 client.53579.objecter _send_command 14 2024-02-10T08:41:41.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.357+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f94580243b0 con 0x7f945801f0c0 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.358+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f94580243b0 con 0x7f945801f0c0 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.358+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.358+0000 7f9468ff9700 10 client.53579.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.358+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 4 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.358+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 6 14 2024-02-10T08:41:41.359 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 10 client.53579.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 15 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 15 2024-02-10T08:41:41.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 7 15 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 4 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 10 client.53579.objecter _send_command 15 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9458024fe0 con 0x7f9458024a30 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.359+0000 7f9468ff9700 10 client.53579.objecter ms_handle_connect 0x7f9458024a30 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.360+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9458024fe0 con 0x7f9458024a30 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.360+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.360+0000 7f9468ff9700 10 client.53579.objecter _finish_command 15 = system:0 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.360+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 4 2024-02-10T08:41:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.360+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 7 15 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 10 client.53579.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign -1 16 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 20 client.53579.objecter _get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 15 client.53579.objecter _session_command_op_remove -1 16 2024-02-10T08:41:41.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 20 client.53579.objecter get_session s=0x7f94580243b0 osd=7 3 2024-02-10T08:41:41.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 15 client.53579.objecter _session_command_op_assign 7 16 2024-02-10T08:41:41.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 4 2024-02-10T08:41:41.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 10 client.53579.objecter _send_command 16 2024-02-10T08:41:41.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.374+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9458024fe0 con 0x7f9458024a30 2024-02-10T08:41:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9468ff9700 1 -- 172.21.15.107:0/44431661 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9458024fe0 con 0x7f9458024a30 2024-02-10T08:41:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9468ff9700 10 client.53579.objecter ms_dispatch 0x7f946c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:41:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9468ff9700 10 client.53579.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9468ff9700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 4 2024-02-10T08:41:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9468ff9700 15 client.53579.objecter _session_command_op_remove 7 16 2024-02-10T08:41:41.377 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 librados: watch_flush enter 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 librados: watch_flush exit 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.0 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6801/3321596972 conn(0x7f9458004530 legacy=0x7f9458002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580051a0 osd=0 1 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.1 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6805/133516126 conn(0x7f9458004150 legacy=0x7f945800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003fe0 osd=1 1 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.2 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6809/3575881728 conn(0x7f945800d2f0 legacy=0x7f945800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003630 osd=2 1 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.3 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6813/2871825691 conn(0x7f9458011e80 legacy=0x7f9458014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458003410 osd=3 1 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.4 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.195:6800/2499671285 conn(0x7f945800ce50 legacy=0x7f9458015c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f946bfff700 1 -- 172.21.15.107:0/44431661 reap_dead start 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800df60 osd=4 1 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.5 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.195:6804/4178321486 conn(0x7f9458017a50 legacy=0x7f945801d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f946bfff700 1 -- 172.21.15.107:0/44431661 reap_dead start 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f945800e100 osd=5 1 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.6 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.195:6808/1417479017 conn(0x7f945801f0c0 legacy=0x7f94580214b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f9458017ff0 osd=6 1 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter close_session for osd.7 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.195:6812/1934176928 conn(0x7f9458024a30 legacy=0x7f9458022c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter put_session s=0x7f94580243b0 osd=7 1 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 20 client.53579.objecter shutdown clearing up homeless session... 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 10 client.53579.objecter successfully canceled tick 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f946bfff700 1 -- 172.21.15.107:0/44431661 reap_dead start 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9454068140 legacy=0x7f945406a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> v1:172.21.15.107:6790/0 conn(0x7f946c104340 legacy=0x7f946c115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 shutdown_connections 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 >> 172.21.15.107:0/44431661 conn(0x7f946c0f2550 msgr2=0x7f946c1077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.376+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 shutdown_connections 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.377+0000 7f9471bef700 1 -- 172.21.15.107:0/44431661 wait complete. 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:41.377+0000 7f9471bef700 1 librados: shutdown 2024-02-10T08:41:41.380 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:40.073711+0000 mon.a (mon.0) 2439 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:40.073781+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:40.079161+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:40.082525+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:40.083265+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:40.155978+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.016768+0000 mon.a (mon.0) 2443 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:41.016846+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.047974+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.048302+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.048834+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.049010+0000 mon.a (mon.0) 2446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:40.073711+0000 mon.a (mon.0) 2439 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:40.073781+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:40.079161+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:40.082525+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:40.083265+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:40.155978+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.016768+0000 mon.a (mon.0) 2443 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:41.016846+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.047974+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.048302+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.048834+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.398 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.049010+0000 mon.a (mon.0) 2446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.416 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:41.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:41.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:40.073711+0000 mon.a (mon.0) 2439 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:41.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:40.073781+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-10T08:41:41.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:40.079161+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:41.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:40.082525+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:40.083265+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:40.155978+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.016768+0000 mon.a (mon.0) 2443 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:41.016846+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.047974+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.048302+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.048834+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.049010+0000 mon.a (mon.0) 2446 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-59"}]: dispatch 2024-02-10T08:41:41.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:41.554 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:41.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:41.802 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:41.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:41.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:41.939 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:41.960 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:41.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:42.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:42.093 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.117 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:42.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:42.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.232444+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.107:0/44431661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:41.435278+0000 mgr.y (mgr.14553) 359 : cluster [DBG] pgmap v585: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:41:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:41.437415+0000 mgr.y (mgr.14553) 360 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:42.249 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.267 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:42.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:42.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:42.399 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.418 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:42.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.232444+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.107:0/44431661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:41.435278+0000 mgr.y (mgr.14553) 359 : cluster [DBG] pgmap v585: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:41:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:41.437415+0000 mgr.y (mgr.14553) 360 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:42.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.232444+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.107:0/44431661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:42.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:41.435278+0000 mgr.y (mgr.14553) 359 : cluster [DBG] pgmap v585: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:41:42.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:41.437415+0000 mgr.y (mgr.14553) 360 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:42.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:42.549 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.568 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:42.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:42.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:42.699 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.718 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:42.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 -- 172.21.15.107:0/1153486304 >> v1:172.21.15.107:6790/0 conn(0x7f21f4103250 legacy=0x7f21f41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 -- 172.21.15.107:0/1153486304 shutdown_connections 2024-02-10T08:41:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 -- 172.21.15.107:0/1153486304 >> 172.21.15.107:0/1153486304 conn(0x7f21f40f2510 msgr2=0x7f21f40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 -- 172.21.15.107:0/1153486304 shutdown_connections 2024-02-10T08:41:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 -- 172.21.15.107:0/1153486304 wait complete. 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 librados: starting msgr at 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.809+0000 7f21fa47d700 1 librados: starting objecter 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 Processor -- start 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 -- start start 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 librados: setting wanted keys 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 librados: calling monclient init 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21f41162e0 con 0x7f21f40ff710 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21f41174c0 con 0x7f21f40fc910 2024-02-10T08:41:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21fa47d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21f41186c0 con 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f8a1a700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f21f4103250 0x7f21f41149a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52320/0 (socket says 172.21.15.107:52320) 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f8a1a700 1 -- 172.21.15.107:0/3491535085 learned_addr learned my addr 172.21.15.107:0/3491535085 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f37fe700 1 --1- 172.21.15.107:0/3491535085 >> v1:172.21.15.107:6789/0 conn(0x7f21f40ff710 0x7f21f40f9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39624/0 (socket says 172.21.15.107:39624) 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f17fa700 10 client.?.objecter ms_handle_connect 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f17fa700 10 client.?.objecter ms_handle_connect 0x7f21f40ff710 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.810+0000 7f21f17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121193896 0 0) 0x7f21f41186c0 con 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21dc003030 con 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816425647 0 0) 0x7f21f41162e0 con 0x7f21f40ff710 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f41186c0 con 0x7f21f40ff710 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 10 client.?.objecter ms_handle_connect 0x7f21f40fc910 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835471105 0 0) 0x7f21dc003030 con 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21f41162e0 con 0x7f21f4103250 2024-02-10T08:41:42.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f21ec0017b0 con 0x7f21f4103250 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696788399 0 0) 0x7f21f41174c0 con 0x7f21f40fc910 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21dc003030 con 0x7f21f40fc910 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 75000940 0 0) 0x7f21f41186c0 con 0x7f21f40ff710 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21f41174c0 con 0x7f21f40ff710 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f21e8001930 con 0x7f21f40ff710 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 976183222 0 0) 0x7f21f41162e0 con 0x7f21f4103250 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 >> v1:172.21.15.195:6789/0 conn(0x7f21f40fc910 legacy=0x7f21f40f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 >> v1:172.21.15.107:6789/0 conn(0x7f21f40ff710 legacy=0x7f21f40f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f41198c0 con 0x7f21f4103250 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f41164d0 con 0x7f21f4103250 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21fa47d700 10 client.53585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f41169c0 con 0x7f21f4103250 2024-02-10T08:41:42.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21fa47d700 1 librados: init done 2024-02-10T08:41:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f21ec002780 con 0x7f21f4103250 2024-02-10T08:41:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.811+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f21ec003c30 con 0x7f21f4103250 2024-02-10T08:41:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.812+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e0004f20 con 0x7f21f4103250 2024-02-10T08:41:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.812+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f21ec001bc0 con 0x7f21f4103250 2024-02-10T08:41:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(406..406 src has 330..406) v4 ==== 7808+0+0 (unknown 2596698896 0 0) 0x7f21ec079270 con 0x7f21f4103250 2024-02-10T08:41:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 10 client.53585.objecter ms_dispatch 0x7f21f4001550 osd_map(406..406 src has 330..406) v4 2024-02-10T08:41:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 3 client.53585.objecter handle_osd_map got epochs [406,406] > 0 2024-02-10T08:41:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 3 client.53585.objecter handle_osd_map decoding full epoch 406 2024-02-10T08:41:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 20 client.53585.objecter dump_active .. 0 homeless 2024-02-10T08:41:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.813+0000 7f21f17fa700 10 client.53585.objecter ms_handle_connect 0x7f21dc063c60 2024-02-10T08:41:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.817+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f21ec04c820 con 0x7f21f4103250 2024-02-10T08:41:42.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:42.851 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:42.995+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f21e00051d0 con 0x7f21f4103250 2024-02-10T08:41:43.105 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:42.103992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-10T08:41:43.105 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:42.996751+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.107:0/3491535085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:42.997076+0000 mon.a (mon.0) 2448 : audit [INF] from='client.53585 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.114+0000 7f21f17fa700 1 -- 172.21.15.107:0/3491535085 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v407) v1 ==== 91+0+0 (unknown 3587057794 0 0) 0x7f21ec04ac90 con 0x7f21f4103250 2024-02-10T08:41:43.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.118+0000 7f21fa47d700 10 librados: watch_flush enter 2024-02-10T08:41:43.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 10 librados: watch_flush exit 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 20 client.53585.objecter shutdown clearing up homeless session... 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 10 client.53585.objecter successfully canceled tick 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 >> v1:172.21.15.107:6800/2634842009 conn(0x7f21dc063c60 legacy=0x7f21dc066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 >> v1:172.21.15.107:6790/0 conn(0x7f21f4103250 legacy=0x7f21f41149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 shutdown_connections 2024-02-10T08:41:43.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.119+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 >> 172.21.15.107:0/3491535085 conn(0x7f21f40f2510 msgr2=0x7f21f41066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:43.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.121+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 shutdown_connections 2024-02-10T08:41:43.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.122+0000 7f21fa47d700 1 -- 172.21.15.107:0/3491535085 wait complete. 2024-02-10T08:41:43.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:43.122+0000 7f21fa47d700 1 librados: shutdown 2024-02-10T08:41:43.124 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:41:43.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:42.103992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-10T08:41:43.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:42.996751+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.107:0/3491535085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:42.997076+0000 mon.a (mon.0) 2448 : audit [INF] from='client.53585 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:43.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:42.103992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-10T08:41:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:42.996751+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.107:0/3491535085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:42.997076+0000 mon.a (mon.0) 2448 : audit [INF] from='client.53585 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:41:43.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:43.633 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.102423+0000 mon.a (mon.0) 2449 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:43.114490+0000 mon.a (mon.0) 2450 : audit [INF] from='client.53585 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.114566+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:43.119341+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:43.120379+0000 mon.a (mon.0) 2452 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.192424+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.a scrub starts 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.195593+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.a scrub ok 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.435685+0000 mgr.y (mgr.14553) 361 : cluster [DBG] pgmap v588: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.680483+0000 osd.4 (osd.4) 176 : cluster [DBG] 6.e scrub starts 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:43.683360+0000 osd.4 (osd.4) 177 : cluster [DBG] 6.e scrub ok 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.102423+0000 mon.a (mon.0) 2449 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:43.114490+0000 mon.a (mon.0) 2450 : audit [INF] from='client.53585 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.114566+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:43.119341+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:43.120379+0000 mon.a (mon.0) 2452 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.192424+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.a scrub starts 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.195593+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.a scrub ok 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.435685+0000 mgr.y (mgr.14553) 361 : cluster [DBG] pgmap v588: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.680483+0000 osd.4 (osd.4) 176 : cluster [DBG] 6.e scrub starts 2024-02-10T08:41:44.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:43.683360+0000 osd.4 (osd.4) 177 : cluster [DBG] 6.e scrub ok 2024-02-10T08:41:44.182 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:44.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:44.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:44.337 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:44.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:44.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:44.467 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.102423+0000 mon.a (mon.0) 2449 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:41:44.467 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:43.114490+0000 mon.a (mon.0) 2450 : audit [INF] from='client.53585 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.114566+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:43.119341+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:43.120379+0000 mon.a (mon.0) 2452 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.192424+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.a scrub starts 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.195593+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.a scrub ok 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.435685+0000 mgr.y (mgr.14553) 361 : cluster [DBG] pgmap v588: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.680483+0000 osd.4 (osd.4) 176 : cluster [DBG] 6.e scrub starts 2024-02-10T08:41:44.468 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:43.683360+0000 osd.4 (osd.4) 177 : cluster [DBG] 6.e scrub ok 2024-02-10T08:41:44.469 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:44.488 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:44.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:44.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:44.623 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:44.871 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:44.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:44.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:45.005 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:45.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:41:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:44.118234+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.118335+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-10T08:41:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:44.132884+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:44.136108+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:44.138717+0000 mon.a (mon.0) 2455 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.631579+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.633195+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:44.118234+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.118335+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:44.132884+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:44.136108+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:44.138717+0000 mon.a (mon.0) 2455 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.631579+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:41:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.633195+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:44.118234+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.118335+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:44.132884+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:44.136108+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:44.138717+0000 mon.a (mon.0) 2455 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:41:45.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.631579+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:41:45.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.633195+0000 osd.4 (osd.4) 179 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:41:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:45] "GET /metrics HTTP/1.1" 200 187389 "" "Prometheus/2.33.4" 2024-02-10T08:41:45.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:41:46.255 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.354121+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.354940+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.747858+0000 osd.1 (osd.1) 181 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.751366+0000 osd.1 (osd.1) 182 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.771628+0000 osd.3 (osd.3) 187 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:44.772934+0000 osd.3 (osd.3) 188 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.089441+0000 osd.5 (osd.5) 177 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.092611+0000 osd.5 (osd.5) 178 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:41:46.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:45.119008+0000 mon.a (mon.0) 2456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:46.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.119077+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-10T08:41:46.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.256563+0000 osd.2 (osd.2) 143 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:41:46.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.260149+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:41:46.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.436067+0000 mgr.y (mgr.14553) 362 : cluster [DBG] pgmap v591: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:46.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:46.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.354121+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:41:46.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.354940+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:41:46.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.747858+0000 osd.1 (osd.1) 181 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:41:46.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.751366+0000 osd.1 (osd.1) 182 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.771628+0000 osd.3 (osd.3) 187 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:44.772934+0000 osd.3 (osd.3) 188 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.089441+0000 osd.5 (osd.5) 177 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.092611+0000 osd.5 (osd.5) 178 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:45.119008+0000 mon.a (mon.0) 2456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.119077+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.256563+0000 osd.2 (osd.2) 143 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.260149+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:41:46.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.436067+0000 mgr.y (mgr.14553) 362 : cluster [DBG] pgmap v591: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:46.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:46.427 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:46.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.354121+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.354940+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.747858+0000 osd.1 (osd.1) 181 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.751366+0000 osd.1 (osd.1) 182 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.771628+0000 osd.3 (osd.3) 187 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:44.772934+0000 osd.3 (osd.3) 188 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.089441+0000 osd.5 (osd.5) 177 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.092611+0000 osd.5 (osd.5) 178 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:41:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:45.119008+0000 mon.a (mon.0) 2456 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:41:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.119077+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-10T08:41:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.256563+0000 osd.2 (osd.2) 143 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:41:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.260149+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:41:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.436067+0000 mgr.y (mgr.14553) 362 : cluster [DBG] pgmap v591: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:41:46.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:46.587 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:46.609 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:46.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:46.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:46.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:46.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:46.949 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.651+0000 7f736d4ab700 1 -- 172.21.15.107:0/3087437820 >> v1:172.21.15.107:6789/0 conn(0x7f73680fc920 legacy=0x7f73680fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.651+0000 7f736d4ab700 1 -- 172.21.15.107:0/3087437820 shutdown_connections 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.651+0000 7f736d4ab700 1 -- 172.21.15.107:0/3087437820 >> 172.21.15.107:0/3087437820 conn(0x7f73680f2540 msgr2=0x7f73680f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.651+0000 7f736d4ab700 1 -- 172.21.15.107:0/3087437820 shutdown_connections 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.651+0000 7f736d4ab700 1 -- 172.21.15.107:0/3087437820 wait complete. 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 librados: starting msgr at 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 librados: starting objecter 2024-02-10T08:41:47.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 Processor -- start 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 -- start start 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 librados: setting wanted keys 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 librados: calling monclient init 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73681160d0 con 0x7f73680ff720 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73681172b0 con 0x7f7368103260 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.652+0000 7f736d4ab700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73681184b0 con 0x7f73680fc920 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f7366ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f73680fc920 0x7f73680f86d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52340/0 (socket says 172.21.15.107:52340) 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f7366ffd700 1 -- 172.21.15.107:0/2586856804 learned_addr learned my addr 172.21.15.107:0/2586856804 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter ms_handle_connect 0x7f73680fc920 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter ms_handle_connect 0x7f73680ff720 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter ms_handle_connect 0x7f7368103260 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089695519 0 0) 0x7f73681184b0 con 0x7f73680fc920 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7350003030 con 0x7f73680fc920 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797059345 0 0) 0x7f73681172b0 con 0x7f7368103260 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73681184b0 con 0x7f7368103260 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381377353 0 0) 0x7f73681160d0 con 0x7f73680ff720 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73681172b0 con 0x7f73680ff720 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206819851 0 0) 0x7f7350003030 con 0x7f73680fc920 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73681160d0 con 0x7f73680fc920 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f735c0020f0 con 0x7f73680fc920 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975422808 0 0) 0x7f73681172b0 con 0x7f73680ff720 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7350003030 con 0x7f73680ff720 2024-02-10T08:41:47.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2380207806 0 0) 0x7f73681184b0 con 0x7f7368103260 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73681172b0 con 0x7f7368103260 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f73580019f0 con 0x7f73680ff720 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7360001b40 con 0x7f7368103260 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2260737311 0 0) 0x7f73681160d0 con 0x7f73680fc920 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 >> v1:172.21.15.195:6789/0 conn(0x7f7368103260 legacy=0x7f73681148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 >> v1:172.21.15.107:6789/0 conn(0x7f73680ff720 legacy=0x7f73680f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73681196b0 con 0x7f73680fc920 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73681174a0 con 0x7f73680fc920 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f736d4ab700 10 client.53591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7368117990 con 0x7f73680fc920 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.653+0000 7f736d4ab700 1 librados: init done 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.654+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f735c002730 con 0x7f73680fc920 2024-02-10T08:41:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.654+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f735c003900 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.654+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7354004f20 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f735c002290 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(410..410 src has 330..410) v4 ==== 8194+0+0 (unknown 601198664 0 0) 0x7f735c079160 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 10 client.53591.objecter ms_dispatch 0x7f7368001550 osd_map(410..410 src has 330..410) v4 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 3 client.53591.objecter handle_osd_map got epochs [410,410] > 0 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 3 client.53591.objecter handle_osd_map decoding full epoch 410 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 20 client.53591.objecter dump_active .. 0 homeless 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.655+0000 7f734ffff700 10 client.53591.objecter ms_handle_connect 0x7f7350063e00 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.659+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f735c04c590 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:46.851+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f73540051d0 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.139+0000 7f734ffff700 1 -- 172.21.15.107:0/2586856804 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v411) v1 ==== 113+0+0 (unknown 3491309899 0 0) 0x7f735c04aa00 con 0x7f73680fc920 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 10 librados: watch_flush enter 2024-02-10T08:41:47.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 10 librados: watch_flush exit 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 20 client.53591.objecter shutdown clearing up homeless session... 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 10 client.53591.objecter successfully canceled tick 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7350063e00 legacy=0x7f73500662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 >> v1:172.21.15.107:6790/0 conn(0x7f73680fc920 legacy=0x7f73680f86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 shutdown_connections 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 >> 172.21.15.107:0/2586856804 conn(0x7f73680f2540 msgr2=0x7f73681066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 shutdown_connections 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 -- 172.21.15.107:0/2586856804 wait complete. 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:47.143+0000 7f736d4ab700 1 librados: shutdown 2024-02-10T08:41:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T08:41:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.371183+0000 osd.6 (osd.6) 155 : cluster [DBG] 60.f scrub starts 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:45.373405+0000 osd.6 (osd.6) 156 : cluster [DBG] 60.f scrub ok 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.082295+0000 osd.5 (osd.5) 179 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.085759+0000 osd.5 (osd.5) 180 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.135900+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:46.852547+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.107:0/2586856804' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:47.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:46.852891+0000 mon.a (mon.0) 2459 : audit [INF] from='client.53591 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.371183+0000 osd.6 (osd.6) 155 : cluster [DBG] 60.f scrub starts 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:45.373405+0000 osd.6 (osd.6) 156 : cluster [DBG] 60.f scrub ok 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.082295+0000 osd.5 (osd.5) 179 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.085759+0000 osd.5 (osd.5) 180 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.135900+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:46.852547+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.107:0/2586856804' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:46.852891+0000 mon.a (mon.0) 2459 : audit [INF] from='client.53591 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.371183+0000 osd.6 (osd.6) 155 : cluster [DBG] 60.f scrub starts 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:45.373405+0000 osd.6 (osd.6) 156 : cluster [DBG] 60.f scrub ok 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.082295+0000 osd.5 (osd.5) 179 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.085759+0000 osd.5 (osd.5) 180 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.135900+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-10T08:41:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:46.852547+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.107:0/2586856804' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:46.852891+0000 mon.a (mon.0) 2459 : audit [INF] from='client.53591 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25276.0:2 52.14 52:28e9c4a8:test-rados-api-smithi107-112562-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.84962321900002, 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999, 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25276", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935029+0000", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935027+0000", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935035+0000", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0684e-05 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935046+0000", 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7470000000000005e-06 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.935054+0000", 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020156359999999999 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937070+0000", 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4700000000000002e-06 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.937078+0000", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020263032 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957341+0000", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4965999999999999e-05 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957426+0000", 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699000000000001e-05 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.957524+0000", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062096199999999997 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958145+0000", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0729999999999994e-06 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958154+0000", 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048278610000000001 2024-02-10T08:41:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962982+0000", 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0270999999999999e-05 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963042+0000", 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6792000000000002e-05 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.963089+0000", 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028059338999999999 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25291.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.857916536, 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003, 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25291", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2248243388", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926736+0000", 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926734+0000", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8688000000000001e-05 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926753+0000", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8120000000000002e-06 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926762+0000", 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6359999999999994e-06 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926770+0000", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018594099999999999 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926956+0000", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4220000000000003e-06 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.926963+0000", 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025476736 2024-02-10T08:41:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952440+0000", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103766 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952544+0000", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013851099999999999 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.952682+0000", 2024-02-10T08:41:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047535930000000004 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957436+0000", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065080549999999996 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963944+0000", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004709319 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968653+0000", 2024-02-10T08:41:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4684999999999995e-05 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968718+0000", 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6301999999999999e-05 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968754+0000", 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042018336000000003 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35219.0:9 51.1s0 51:967778f1:test-rados-api-smithi107-112451-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.83951202200001, 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237, 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35219", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945140+0000", 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945138+0000", 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7849999999999992e-06 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945146+0000", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5050000000000007e-06 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945155+0000", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0849999999999999e-06 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945162+0000", 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001228563 2024-02-10T08:41:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946390+0000", 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8227999999999996e-05 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946479+0000", 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382076999999998 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.962861+0000", 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017720237 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.14907.0:2 55.15 55:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.83889680800002, 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724, 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.14907", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4285222104", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945756+0000", 2024-02-10T08:41:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945754+0000", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9549999999999997e-06 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945760+0000", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3450000000000006e-06 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945769+0000", 2024-02-10T08:41:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2589999999999997e-06 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945775+0000", 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01446833 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960243+0000", 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118359 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960362+0000", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016521099999999999 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.960527+0000", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010993870000000001 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.961626+0000", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060821130000000001 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967708+0000", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002001966 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969710+0000", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3856999999999998e-05 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969744+0000", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1362e-05 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.969765+0000", 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024009724 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.24889.0:19 23.d e114/102 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head v 114'3, mlcod=0'0)", 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.83880623200002, 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503, 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945846+0000", 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5839999999999995e-06 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945854+0000", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3709999999999999e-06 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945860+0000", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7637999999999999e-05 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945918+0000", 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014029747 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959947+0000", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2229999999999999e-05 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.959969+0000", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076815219999999997 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967651+0000", 2024-02-10T08:41:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5574000000000001e-05 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967697+0000", 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0218503 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(50.4s1 epoch 114/111 rep_tid 1204 entries 114'2 (0'0) error 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head by client.35222.0:9 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 299.838780391, 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002, 2024-02-10T08:41:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945872+0000", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9130000000000003e-06 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.945880+0000", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056525589999999997 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951532+0000", 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6671e-05 2024-02-10T08:41:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.951549+0000", 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056510120000000004 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.957200+0000", 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067084960000000004 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.963908+0000", 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018036142000000002 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.5.0:1233 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'1, mlcod=0'0)", 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 296.800573681, 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001, 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984079+0000", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984078+0000", 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7410000000000004e-06 2024-02-10T08:41:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984087+0000", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.772e-06 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984096+0000", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8090000000000004e-06 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984103+0000", 2024-02-10T08:41:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000327542 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984430+0000", 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8281999999999999e-05 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984449+0000", 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017470151999999999 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001919+0000", 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4270999999999999e-05 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.001943+0000", 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017864357000000001 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:8 56.1 e117/114 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head v 117'2, mlcod=0'0)", 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 296.800540356, 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001, 2024-02-10T08:41:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984112+0000", 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8529999999999997e-06 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984117+0000", 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9869999999999999e-06 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984122+0000", 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.625e-06 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984126+0000", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046452199999999998 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984591+0000", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6535999999999999e-05 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.984607+0000", 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023819407000000001 2024-02-10T08:41:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008427+0000", 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7294e-05 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.008454+0000", 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024341713000000001 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:4 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-10T08:41:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 294.71772053000001, 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002, 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066932+0000", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066929+0000", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8779999999999993e-06 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066939+0000", 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.431e-05 2024-02-10T08:41:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066953+0000", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3720000000000005e-06 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.066962+0000", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040994600000000001 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071061+0000", 2024-02-10T08:41:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0001153 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071177+0000", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014616299999999999 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.071323+0000", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020771520000000001 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.073400+0000", 2024-02-10T08:41:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00123847 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.074638+0000", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082236259999999995 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082862+0000", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7891e-05 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082900+0000", 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6568e-05 2024-02-10T08:41:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:53.082926+0000", 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015994408000000002 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:5 63.f 63:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e121)", 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 292.74491793700003, 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002, 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:41:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039735+0000", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039732+0000", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1054e-05 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039743+0000", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2641e-05 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039755+0000", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9560000000000003e-06 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039764+0000", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9907999999999997e-05 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039814+0000", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6459999999999993e-06 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:41:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.039822+0000", 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030549936 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070372+0000", 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4599999999999996e-05 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.070456+0000", 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019660001 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.090116+0000", 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.050381751000000002 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26977.0:63 66.1b 66:d9f0651a:test-rados-api-smithi107-112525-13::foo:head [write 0~128] snapc 16=[16] ondisk+write+known_if_redirected e141)", 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 272.74792589700002, 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635, 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26977", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 63 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036727+0000", 2024-02-10T08:41:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036723+0000", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4849e-05 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036738+0000", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1770000000000001e-05 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036749+0000", 2024-02-10T08:41:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1699999999999997e-06 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036758+0000", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000209371 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036967+0000", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0199999999999997e-06 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.036974+0000", 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020105096999999999 2024-02-10T08:41:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057079+0000", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120828 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057200+0000", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015114300000000001 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.057351+0000", 2024-02-10T08:41:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021251266000000001 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.078602+0000", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032546469999999998 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081857+0000", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6645e-05 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081884+0000", 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8924000000000001e-05 2024-02-10T08:41:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081933+0000", 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7667999999999998e-05 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:15.081980+0000", 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045253635 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47371.0:2 97.6 97:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 265.85089210000001, 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020953844999999999, 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47371", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3974906254", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:41:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933760+0000", 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933758+0000", 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9530000000000002e-06 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933765+0000", 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7169999999999997e-06 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933774+0000", 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0809999999999996e-06 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933781+0000", 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8953999999999999e-05 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933830+0000", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4800000000000004e-06 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.933838+0000", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014440671 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948278+0000", 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011152 2024-02-10T08:41:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948390+0000", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139682 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948530+0000", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001290404 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.949820+0000", 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4445e-05 2024-02-10T08:41:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.949854+0000", 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047572279999999996 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954612+0000", 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9101999999999998e-05 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954671+0000", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3550000000000001e-05 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.954714+0000", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020953844999999999 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52871.0:1 130.6s0 130:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:41:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 224.82542869299999, 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0506345260000001, 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52871", 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1154190771", 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959224+0000", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959222+0000", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959230+0000", 2024-02-10T08:41:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0638e-05 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959241+0000", 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3639999999999999e-06 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959249+0000", 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6088999999999999e-05 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959295+0000", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.23e-06 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:02.959298+0000", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.042640617 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.001939+0000", 2024-02-10T08:41:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8879e-05 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.002038+0000", 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024591899999999999 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.002284+0000", 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001147835 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:41:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.003432+0000", 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063665730000000004 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.009798+0000", 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0049000000000001e-05 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:04.009858+0000", 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0506345260000001 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.1.0:395 60.1e e191/135 60:7a4cebd2:test-rados-api-smithi107-112490-14::foo69:4 v 191'11, mlcod=149'7)", 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 221.78152112399999, 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017016453000000001, 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003131+0000", 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2029999999999992e-06 2024-02-10T08:41:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003139+0000", 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0289999999999994e-06 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003148+0000", 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3269999999999997e-06 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.003155+0000", 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01388257 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017037+0000", 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8848e-05 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017056+0000", 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030742999999999999 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.020130+0000", 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7609e-05 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.020148+0000", 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017016453000000001 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.2.0:345 60.c e191/135 60:37d01833:test-rados-api-smithi107-112490-14::foo42:4 v 191'44, mlcod=190'38)", 2024-02-10T08:41:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 221.776665627, 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013120199000000001, 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:41:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007987+0000", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0859999999999996e-06 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007994+0000", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.007999+0000", 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.019e-06 2024-02-10T08:41:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.008003+0000", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099050619999999992 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017908+0000", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8536e-05 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.017927+0000", 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031526549999999999 2024-02-10T08:41:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.021080+0000", 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7427999999999999e-05 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:06.021107+0000", 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013120199000000001 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47632.0:1 134.5s0 134:a12dce25:test-rados-api-smithi107-112480-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected e194)", 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.76539482999999, 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.020938202, 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47632", 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4023540388", 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019258+0000", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019256+0000", 2024-02-10T08:41:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8900000000000001e-06 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019263+0000", 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019270+0000", 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4020000000000001e-06 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019276+0000", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9276000000000001e-05 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019315+0000", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2820000000000001e-06 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:08.019318+0000", 2024-02-10T08:41:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.010518837 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.029837+0000", 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012234799999999999 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.029959+0000", 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032286800000000003 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.030282+0000", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077056499999999997 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.031053+0000", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009073434 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.040126+0000", 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9586999999999996e-05 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:09.040196+0000", 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.020938202 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52931.0:2 137.a 137:53c7e431:::oid_AioUnlockPP_:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e198)", 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.837727202, 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022407323, 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52931", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4202333394", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946925+0000", 2024-02-10T08:41:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946923+0000", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.064e-06 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946932+0000", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0265000000000001e-05 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946942+0000", 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.328e-06 2024-02-10T08:41:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.946950+0000", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000488782 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.947438+0000", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1780000000000003e-06 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.947447+0000", 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013418744 2024-02-10T08:41:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.960865+0000", 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000128928 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.960994+0000", 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000137465 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:41:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.961132+0000", 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029011100000000001 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.961422+0000", 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074843169999999999 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.968906+0000", 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000371484 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969278+0000", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1267999999999999e-05 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969309+0000", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3768999999999999e-05 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:11.969333+0000", 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022407323 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52997.0:2 146.6 146:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 200.828586876, 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015476029000000001, 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52997", 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1556895985", 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956066+0000", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956064+0000", 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0999999999999998e-06 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956071+0000", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4700000000000008e-06 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956081+0000", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5719999999999999e-06 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956087+0000", 2024-02-10T08:41:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102502 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956190+0000", 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6790000000000002e-06 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.956196+0000", 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099999990000000007 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966196+0000", 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114827 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966310+0000", 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014374799999999999 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.966454+0000", 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00066427700000000001 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.967118+0000", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025337910000000001 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.969652+0000", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017909740000000001 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971443+0000", 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0241000000000002e-05 2024-02-10T08:41:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971493+0000", 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8279999999999999e-05 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:26.971542+0000", 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015476029000000001 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47884.0:1 171.6s0 171:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e259)", 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 154.811350756, 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0029141699999999, 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47884", 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4001272799", 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973302+0000", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973300+0000", 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973307+0000", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.827e-06 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973314+0000", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1540000000000001e-06 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973319+0000", 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4014999999999999e-05 2024-02-10T08:41:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973353+0000", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5909999999999998e-06 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:12.973355+0000", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.999621396 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.972977+0000", 2024-02-10T08:41:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9870999999999994e-05 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973067+0000", 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027885300000000002 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973345+0000", 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035942999999999999 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:41:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.973705+0000", 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024675819999999998 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.976172+0000", 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3427999999999998e-05 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:13.976216+0000", 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0029141699999999 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:4742 197.c 197:30000000:test-rados-api-smithi107-112490-46:::head [pgnls start_epoch 347] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e347)", 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.770701587000005, 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004231037, 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4742 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:41:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:41:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013947+0000", 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5713999999999999e-05 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013963+0000", 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.173e-05 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013985+0000", 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3070000000000003e-06 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013993+0000", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5900999999999999e-05 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.014049+0000", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1820000000000006e-06 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.014057+0000", 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.003927923 2024-02-10T08:41:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.017985+0000", 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5151000000000003e-05 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.018020+0000", 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016174900000000001 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:41:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.018182+0000", 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004231037 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:41:48.400 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:41:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.401225+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:41:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.403274+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:41:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.725868+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:41:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:46.729392+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:41:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:47.138707+0000 mon.a (mon.0) 2460 : audit [INF] from='client.53591 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:47.138767+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:47.436458+0000 mgr.y (mgr.14553) 363 : cluster [DBG] pgmap v594: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.401225+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.403274+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.725868+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:46.729392+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:47.138707+0000 mon.a (mon.0) 2460 : audit [INF] from='client.53591 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:47.138767+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-10T08:41:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:47.436458+0000 mgr.y (mgr.14553) 363 : cluster [DBG] pgmap v594: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:48.495 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:48.497 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:48.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.401225+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.403274+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.725868+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:46.729392+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:47.138707+0000 mon.a (mon.0) 2460 : audit [INF] from='client.53591 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:47.138767+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-10T08:41:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:47.436458+0000 mgr.y (mgr.14553) 363 : cluster [DBG] pgmap v594: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:48.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:48.632 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:48.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:48.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T08:41:48.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:41:48.868 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:48.887 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:41:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:48.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:41:48.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:49.022 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:49.044 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:49.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:49.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:49.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.062027+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.065255+0000 osd.5 (osd.5) 182 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.140314+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.211845+0000 osd.2 (osd.2) 145 : cluster [DBG] 16.a scrub starts 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.214819+0000 osd.2 (osd.2) 146 : cluster [DBG] 16.a scrub ok 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.721084+0000 osd.3 (osd.3) 189 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:41:49.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.722063+0000 osd.3 (osd.3) 190 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:41:49.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:49.328 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:49.346 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:49.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.062027+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:41:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.065255+0000 osd.5 (osd.5) 182 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.140314+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.211845+0000 osd.2 (osd.2) 145 : cluster [DBG] 16.a scrub starts 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.214819+0000 osd.2 (osd.2) 146 : cluster [DBG] 16.a scrub ok 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.721084+0000 osd.3 (osd.3) 189 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.722063+0000 osd.3 (osd.3) 190 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.062027+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.065255+0000 osd.5 (osd.5) 182 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:41:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.140314+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-10T08:41:49.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.211845+0000 osd.2 (osd.2) 145 : cluster [DBG] 16.a scrub starts 2024-02-10T08:41:49.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.214819+0000 osd.2 (osd.2) 146 : cluster [DBG] 16.a scrub ok 2024-02-10T08:41:49.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.721084+0000 osd.3 (osd.3) 189 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:41:49.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.722063+0000 osd.3 (osd.3) 190 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:41:49.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:49.480 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.360424+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:48.361353+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.070044+0000 osd.5 (osd.5) 183 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.073505+0000 osd.5 (osd.5) 184 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.151470+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.436935+0000 mgr.y (mgr.14553) 364 : cluster [DBG] pgmap v597: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.360424+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:48.361353+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.070044+0000 osd.5 (osd.5) 183 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.073505+0000 osd.5 (osd.5) 184 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.151470+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-10T08:41:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.436935+0000 mgr.y (mgr.14553) 364 : cluster [DBG] pgmap v597: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:50.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:41:50.511 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:50.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:50.532 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:41:50.532 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.360424+0000 osd.6 (osd.6) 159 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:41:50.533 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:48.361353+0000 osd.6 (osd.6) 160 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:41:50.533 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.070044+0000 osd.5 (osd.5) 183 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:41:50.533 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.073505+0000 osd.5 (osd.5) 184 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:41:50.533 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.151470+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-10T08:41:50.533 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.436935+0000 mgr.y (mgr.14553) 364 : cluster [DBG] pgmap v597: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T08:41:50.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:50.642 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:50.662 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:50.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:50.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:50.779 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:41:50.793 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:50.892 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:50.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:50.910 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:50.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:51.026 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:51.225 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:51.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.641753+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:49.644715+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:50.103127+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:50.103193+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:50.165031+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:50.199114+0000 osd.2 (osd.2) 147 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:41:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:50.202087+0000 osd.2 (osd.2) 148 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:41:51.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:51.021288+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-10T08:41:51.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.641753+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:49.644715+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:50.103127+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:50.103193+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:50.165031+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:50.199114+0000 osd.2 (osd.2) 147 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:41:51.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:50.202087+0000 osd.2 (osd.2) 148 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:51.021288+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.641753+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:49.644715+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:50.103127+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:50.103193+0000 osd.0 (osd.0) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:50.165031+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:50.199114+0000 osd.2 (osd.2) 147 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:50.202087+0000 osd.2 (osd.2) 148 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:41:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:51.021288+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-10T08:41:51.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:41:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:41:52.201 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:51.437779+0000 mgr.y (mgr.14553) 365 : cluster [DBG] pgmap v600: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T08:41:52.201 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:51.447829+0000 mgr.y (mgr.14553) 366 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:52.428 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:41:52.428 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:41:52.428 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:41:52.428 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:41:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:51.437779+0000 mgr.y (mgr.14553) 365 : cluster [DBG] pgmap v600: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T08:41:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:51.447829+0000 mgr.y (mgr.14553) 366 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:51.437779+0000 mgr.y (mgr.14553) 365 : cluster [DBG] pgmap v600: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T08:41:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:51.447829+0000 mgr.y (mgr.14553) 366 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:41:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:52.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:52.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:52.969 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:52.990 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:52.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:53.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T08:41:53.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:41:53.122 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:53.142 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:41:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:53.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:41:53.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:53.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:51.534753+0000 osd.7 (osd.7) 169 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:41:53.271 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:51.538525+0000 osd.7 (osd.7) 170 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:41:53.275 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:53.293 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:53.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:53.325 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.377+0000 7f782020e700 1 -- 172.21.15.107:0/1697618984 >> v1:172.21.15.107:6790/0 conn(0x7f78180fc910 legacy=0x7f78180fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.377+0000 7f782020e700 1 -- 172.21.15.107:0/1697618984 shutdown_connections 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.377+0000 7f782020e700 1 -- 172.21.15.107:0/1697618984 >> 172.21.15.107:0/1697618984 conn(0x7f78180f2510 msgr2=0x7f78180f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.377+0000 7f782020e700 1 -- 172.21.15.107:0/1697618984 shutdown_connections 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.377+0000 7f782020e700 1 -- 172.21.15.107:0/1697618984 wait complete. 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 librados: starting msgr at 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 librados: starting objecter 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 Processor -- start 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 -- start start 2024-02-10T08:41:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 librados: setting wanted keys 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 librados: calling monclient init 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7818117360 con 0x7f78180ff710 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7818118540 con 0x7f7818103250 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f782020e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7818119740 con 0x7f78180fc910 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f781dfaa700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f78180fc910 0x7f78180f9af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50890/0 (socket says 172.21.15.107:50890) 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f781dfaa700 1 -- 172.21.15.107:0/1843645449 learned_addr learned my addr 172.21.15.107:0/1843645449 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f781d7a9700 1 --1- 172.21.15.107:0/1843645449 >> v1:172.21.15.107:6789/0 conn(0x7f78180ff710 0x7f78180fa200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36718/0 (socket says 172.21.15.107:36718) 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter ms_handle_connect 0x7f78180ff710 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter ms_handle_connect 0x7f78180fc910 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter ms_handle_connect 0x7f7818103250 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005780687 0 0) 0x7f7818117360 con 0x7f78180ff710 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7804003030 con 0x7f78180ff710 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242130815 0 0) 0x7f7818119740 con 0x7f78180fc910 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.378+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818117360 con 0x7f78180fc910 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367040264 0 0) 0x7f7818118540 con 0x7f7818103250 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818119740 con 0x7f7818103250 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3229958895 0 0) 0x7f7818117360 con 0x7f78180fc910 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7818118540 con 0x7f78180fc910 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78080015d0 con 0x7f78180fc910 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842804002 0 0) 0x7f7804003030 con 0x7f78180ff710 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7818117360 con 0x7f78180ff710 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7814001840 con 0x7f78180ff710 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3546043430 0 0) 0x7f7818119740 con 0x7f7818103250 2024-02-10T08:41:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7804003030 con 0x7f7818103250 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78100019a0 con 0x7f7818103250 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2623997584 0 0) 0x7f7818118540 con 0x7f78180fc910 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 >> v1:172.21.15.195:6789/0 conn(0x7f7818103250 legacy=0x7f7818115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 >> v1:172.21.15.107:6789/0 conn(0x7f78180ff710 legacy=0x7f78180fa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781811a940 con 0x7f78180fc910 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7818119930 con 0x7f78180fc910 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f782020e700 10 client.53597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7818119e20 con 0x7f78180fc910 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f782020e700 1 librados: init done 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f78080023a0 con 0x7f78180fc910 2024-02-10T08:41:53.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.379+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78080037d0 con 0x7f78180fc910 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.380+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc004f20 con 0x7f78180fc910 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.380+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7808003a10 con 0x7f78180fc910 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(415..415 src has 330..415) v4 ==== 8222+0+0 (unknown 2434258624 0 0) 0x7f7808079100 con 0x7f78180fc910 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 10 client.53597.objecter ms_dispatch 0x7f7818001550 osd_map(415..415 src has 330..415) v4 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 3 client.53597.objecter handle_osd_map got epochs [415,415] > 0 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 3 client.53597.objecter handle_osd_map decoding full epoch 415 2024-02-10T08:41:53.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 20 client.53597.objecter dump_active .. 0 homeless 2024-02-10T08:41:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.381+0000 7f780effd700 10 client.53597.objecter ms_handle_connect 0x7f7804063d10 2024-02-10T08:41:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.385+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f780804c680 con 0x7f78180fc910 2024-02-10T08:41:53.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:53.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:51.534753+0000 osd.7 (osd.7) 169 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:41:53.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:51.538525+0000 osd.7 (osd.7) 170 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:41:53.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:51.534753+0000 osd.7 (osd.7) 169 : cluster [DBG] 60.12 scrub starts 2024-02-10T08:41:53.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:51.538525+0000 osd.7 (osd.7) 170 : cluster [DBG] 60.12 scrub ok 2024-02-10T08:41:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:53.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.571+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f77fc0051d0 con 0x7f78180fc910 2024-02-10T08:41:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:53.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:53.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:53.811 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:53.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 -- 172.21.15.107:0/1762534144 >> v1:172.21.15.107:6790/0 conn(0x7fa6c41007f0 legacy=0x7fa6c4102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 -- 172.21.15.107:0/1762534144 shutdown_connections 2024-02-10T08:41:53.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 -- 172.21.15.107:0/1762534144 >> 172.21.15.107:0/1762534144 conn(0x7fa6c40f3640 msgr2=0x7fa6c40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:53.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 -- 172.21.15.107:0/1762534144 shutdown_connections 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 -- 172.21.15.107:0/1762534144 wait complete. 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 librados: starting msgr at 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.930+0000 7fa6c90d8700 1 librados: starting objecter 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 Processor -- start 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 -- start start 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 librados: setting wanted keys 2024-02-10T08:41:53.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 librados: calling monclient init 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6c4117390 con 0x7fa6c4104330 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6c4118590 con 0x7fa6c41007f0 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c90d8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6c4119790 con 0x7fa6c40fda50 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c2d9d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa6c40fda50 0x7fa6c40f9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50906/0 (socket says 172.21.15.107:50906) 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6c2d9d700 1 -- 172.21.15.107:0/479393855 learned_addr learned my addr 172.21.15.107:0/479393855 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6abfff700 10 client.?.objecter ms_handle_connect 0x7fa6c4104330 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6abfff700 10 client.?.objecter ms_handle_connect 0x7fa6c40fda50 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.931+0000 7fa6abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 10 client.?.objecter ms_handle_connect 0x7fa6c41007f0 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493759804 0 0) 0x7fa6c4117390 con 0x7fa6c4104330 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac003800 con 0x7fa6c4104330 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262783238 0 0) 0x7fa6c4119790 con 0x7fa6c40fda50 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4117390 con 0x7fa6c40fda50 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284093325 0 0) 0x7fa6c4118590 con 0x7fa6c41007f0 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4119790 con 0x7fa6c41007f0 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916859658 0 0) 0x7fa6ac003800 con 0x7fa6c4104330 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6c4118590 con 0x7fa6c4104330 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159334263 0 0) 0x7fa6c4117390 con 0x7fa6c40fda50 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6ac003800 con 0x7fa6c40fda50 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669273191 0 0) 0x7fa6c4119790 con 0x7fa6c41007f0 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6c4117390 con 0x7fa6c41007f0 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6bc001d70 con 0x7fa6c4104330 2024-02-10T08:41:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6b40018c0 con 0x7fa6c40fda50 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6b8002f20 con 0x7fa6c41007f0 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127506219 0 0) 0x7fa6c4118590 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 >> v1:172.21.15.107:6790/0 conn(0x7fa6c40fda50 legacy=0x7fa6c40f9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 >> v1:172.21.15.195:6789/0 conn(0x7fa6c41007f0 legacy=0x7fa6c40fa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c411a990 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c4119980 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6c90d8700 10 client.28017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c4119df0 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6c90d8700 1 librados: init done 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa6bc0026a0 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.932+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6bc0036d0 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.933+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b0004f20 con 0x7fa6c4104330 2024-02-10T08:41:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa6bc002200 con 0x7fa6c4104330 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(415..415 src has 330..415) v4 ==== 8222+0+0 (unknown 2434258624 0 0) 0x7fa6bc0767a0 con 0x7fa6c4104330 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 10 client.28017.objecter ms_dispatch 0x7fa6c4001550 osd_map(415..415 src has 330..415) v4 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 3 client.28017.objecter handle_osd_map got epochs [415,415] > 0 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 3 client.28017.objecter handle_osd_map decoding full epoch 415 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 20 client.28017.objecter dump_active .. 0 homeless 2024-02-10T08:41:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.934+0000 7fa6abfff700 10 client.28017.objecter ms_handle_connect 0x7fa6ac063c80 2024-02-10T08:41:53.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:53.938+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa6bc049b10 con 0x7fa6c4104330 2024-02-10T08:41:54.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.118+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fa6b00051d0 con 0x7fa6c4104330 2024-02-10T08:41:54.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.131+0000 7f2a1a35d700 1 -- 172.21.15.107:0/2093102809 >> v1:172.21.15.107:6790/0 conn(0x7f2a14104330 legacy=0x7f2a141067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.131+0000 7f2a1a35d700 1 -- 172.21.15.107:0/2093102809 shutdown_connections 2024-02-10T08:41:54.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.131+0000 7f2a1a35d700 1 -- 172.21.15.107:0/2093102809 >> 172.21.15.107:0/2093102809 conn(0x7f2a140f3640 msgr2=0x7f2a140f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:54.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.131+0000 7f2a1a35d700 1 -- 172.21.15.107:0/2093102809 shutdown_connections 2024-02-10T08:41:54.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.131+0000 7f2a1a35d700 1 -- 172.21.15.107:0/2093102809 wait complete. 2024-02-10T08:41:54.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 librados: starting msgr at 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 librados: starting objecter 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 Processor -- start 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 -- start start 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 librados: setting wanted keys 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 librados: calling monclient init 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a1406cc00 con 0x7f2a140fda50 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a1406cd90 con 0x7f2a141007f0 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a1a35d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a1406cf80 con 0x7f2a14104330 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a13fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2a140fda50 0x7f2a140fc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36766/0 (socket says 172.21.15.107:36766) 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a13fff700 1 -- 172.21.15.107:0/3968855078 learned_addr learned my addr 172.21.15.107:0/3968855078 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a188fa700 1 --1- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6790/0 conn(0x7f2a14104330 0x7f2a141184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50918/0 (socket says 172.21.15.107:50918) 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.132+0000 7f2a117fa700 10 client.?.objecter ms_handle_connect 0x7f2a140fda50 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:54.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 10 client.?.objecter ms_handle_connect 0x7f2a14104330 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 10 client.?.objecter ms_handle_connect 0x7f2a141007f0 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137333271 0 0) 0x7f2a1406cc00 con 0x7f2a140fda50 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29fc0038e0 con 0x7f2a140fda50 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736762186 0 0) 0x7f2a1406cf80 con 0x7f2a14104330 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1406cc00 con 0x7f2a14104330 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028516496 0 0) 0x7f2a1406cd90 con 0x7f2a141007f0 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1406cf80 con 0x7f2a141007f0 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1419577730 0 0) 0x7f29fc0038e0 con 0x7f2a140fda50 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a1406cd90 con 0x7f2a140fda50 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2a040019f0 con 0x7f2a140fda50 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214355164 0 0) 0x7f2a1406cf80 con 0x7f2a141007f0 2024-02-10T08:41:54.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29fc0038e0 con 0x7f2a141007f0 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 457265755 0 0) 0x7f2a1406cc00 con 0x7f2a14104330 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a1406cf80 con 0x7f2a14104330 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2a080017c0 con 0x7f2a141007f0 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2a0c003110 con 0x7f2a14104330 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1426338805 0 0) 0x7f2a1406cd90 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6790/0 conn(0x7f2a14104330 legacy=0x7f2a141184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.195:6789/0 conn(0x7f2a141007f0 legacy=0x7f2a14114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1411cc00 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a14119c30 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a1a35d700 10 client.28023.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1411a0a0 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a1a35d700 1 librados: init done 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2a04002480 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.133+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2a04005af0 con 0x7f2a140fda50 2024-02-10T08:41:54.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.134+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2a00004f00 con 0x7f2a140fda50 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2a04001fe0 con 0x7f2a140fda50 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(415..415 src has 330..415) v4 ==== 8222+0+0 (unknown 2434258624 0 0) 0x7f2a04078d60 con 0x7f2a140fda50 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 osd_map(415..415 src has 330..415) v4 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 3 client.28023.objecter handle_osd_map got epochs [415,415] > 0 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 3 client.28023.objecter handle_osd_map decoding full epoch 415 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 20 client.28023.objecter dump_active .. 0 homeless 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v415) v1 ==== 54+0+16 (unknown 423660678 0 2420000192) 0x7f2a040790d0 con 0x7f2a140fda50 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.135+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f29fc063be0 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 1 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 1 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 0 1 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 4 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 10 client.28023.objecter _send_command 1 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00004950 con 0x7f2a00004530 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a00004530 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a00004950 con 0x7f2a00004530 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 10 client.28023.objecter _finish_command 1 = system:0 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 4 2024-02-10T08:41:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.136+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 0 1 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 2 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 2 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a000051a0 osd=0 3 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 0 2 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 4 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 10 client.28023.objecter _send_command 2 2024-02-10T08:41:54.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.151+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a00004d30 con 0x7f2a00004530 2024-02-10T08:41:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a00004d30 con 0x7f2a00004530 2024-02-10T08:41:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 10 client.28023.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 4 2024-02-10T08:41:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 0 2 2024-02-10T08:41:54.154 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:41:54.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 1 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 4 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 10 client.28023.objecter _send_command 3 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a0000bef0 con 0x7f2a00004150 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.153+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a00004150 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.154+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a0000bef0 con 0x7f2a00004150 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.154+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:41:54.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.154+0000 7f2a117fa700 10 client.28023.objecter _finish_command 3 = system:0 2024-02-10T08:41:54.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.154+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 4 2024-02-10T08:41:54.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.154+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 1 3 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 4 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 4 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003fe0 osd=1 3 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 1 4 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 4 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 10 client.28023.objecter _send_command 4 2024-02-10T08:41:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.168+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a000018d0 con 0x7f2a00004150 2024-02-10T08:41:54.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a000018d0 con 0x7f2a00004150 2024-02-10T08:41:54.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 10 client.28023.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 4 2024-02-10T08:41:54.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 1 4 2024-02-10T08:41:54.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:41:54.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 5 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 5 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 2 5 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 4 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 10 client.28023.objecter _send_command 5 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00003e40 con 0x7f2a0000d1e0 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.170+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a0000d1e0 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.171+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a00003e40 con 0x7f2a0000d1e0 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.171+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.171+0000 7f2a117fa700 10 client.28023.objecter _finish_command 5 = system:0 2024-02-10T08:41:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.171+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 4 2024-02-10T08:41:54.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.171+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 2 5 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.183+0000 7fa6abfff700 1 -- 172.21.15.107:0/479393855 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v416) v1 ==== 113+0+0 (unknown 847095790 0 0) 0x7fa6bc047f20 con 0x7fa6c4104330 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.185+0000 7fa6c90d8700 10 librados: watch_flush enter 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.185+0000 7fa6c90d8700 10 librados: watch_flush exit 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.185+0000 7fa6c90d8700 20 client.28017.objecter shutdown clearing up homeless session... 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.185+0000 7fa6c90d8700 10 client.28017.objecter successfully canceled tick 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.185+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa6ac063c80 legacy=0x7fa6ac066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 >> v1:172.21.15.107:6789/0 conn(0x7fa6c4104330 legacy=0x7fa6c4115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.183+0000 7f780effd700 1 -- 172.21.15.107:0/1843645449 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v416) v1 ==== 101+0+0 (unknown 601592064 0 0) 0x7f780804aaf0 con 0x7f78180fc910 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 10 librados: watch_flush enter 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 10 librados: watch_flush exit 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 20 client.53597.objecter shutdown clearing up homeless session... 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 10 client.53597.objecter successfully canceled tick 2024-02-10T08:41:54.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7804063d10 legacy=0x7f78040661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 >> v1:172.21.15.107:6790/0 conn(0x7f78180fc910 legacy=0x7f78180f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 shutdown_connections 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.186+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 >> 172.21.15.107:0/1843645449 conn(0x7f78180f2510 msgr2=0x7f78181066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 shutdown_connections 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7f782020e700 1 -- 172.21.15.107:0/1843645449 wait complete. 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7f782020e700 1 librados: shutdown 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 shutdown_connections 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 >> 172.21.15.107:0/479393855 conn(0x7fa6c40f3640 msgr2=0x7fa6c41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 shutdown_connections 2024-02-10T08:41:54.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7fa6c90d8700 1 -- 172.21.15.107:0/479393855 wait complete. 2024-02-10T08:41:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.187+0000 7fa6c90d8700 1 librados: shutdown 2024-02-10T08:41:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:41:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T08:41:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 6 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 6 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003630 osd=2 3 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 2 6 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 4 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 10 client.28023.objecter _send_command 6 2024-02-10T08:41:54.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.190+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a0000e5f0 con 0x7f2a0000d1e0 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.192+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a0000e5f0 con 0x7f2a0000d1e0 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.192+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.192+0000 7f2a117fa700 10 client.28023.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.192+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 4 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.192+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 2 6 2024-02-10T08:41:54.193 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 7 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 7 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 3 7 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 4 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 10 client.28023.objecter _send_command 7 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a000159b0 con 0x7f2a00011d40 2024-02-10T08:41:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.193+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a00011d40 2024-02-10T08:41:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.194+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a000159b0 con 0x7f2a00011d40 2024-02-10T08:41:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.194+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:41:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.194+0000 7f2a117fa700 10 client.28023.objecter _finish_command 7 = system:0 2024-02-10T08:41:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.194+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 4 2024-02-10T08:41:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.194+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 3 7 2024-02-10T08:41:54.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 8 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 8 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00003420 osd=3 3 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 3 8 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 4 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 10 client.28023.objecter _send_command 8 2024-02-10T08:41:54.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.212+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a0000e060 con 0x7f2a00011d40 2024-02-10T08:41:54.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a0000e060 con 0x7f2a00011d40 2024-02-10T08:41:54.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a117fa700 10 client.28023.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 4 2024-02-10T08:41:54.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 3 8 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 9 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 9 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 4 9 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 4 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 10 client.28023.objecter _send_command 9 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.214+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a0000e350 con 0x7f2a0000cd40 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a0000cd40 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a0000e350 con 0x7f2a0000cd40 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:41:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 10 client.28023.objecter _finish_command 9 = system:0 2024-02-10T08:41:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 4 2024-02-10T08:41:54.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.215+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 4 9 2024-02-10T08:41:54.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 10 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 10 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000e060 osd=4 3 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 4 10 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 4 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 10 client.28023.objecter _send_command 10 2024-02-10T08:41:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.231+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a0000e350 con 0x7f2a0000cd40 2024-02-10T08:41:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a0000e350 con 0x7f2a0000cd40 2024-02-10T08:41:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a117fa700 10 client.28023.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 4 2024-02-10T08:41:54.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 4 10 2024-02-10T08:41:54.234 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:41:54.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 11 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 11 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 5 11 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 4 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 10 client.28023.objecter _send_command 11 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.233+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00017e50 con 0x7f2a000178f0 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a000178f0 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a00017e50 con 0x7f2a000178f0 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 10 client.28023.objecter _finish_command 11 = system:0 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 4 2024-02-10T08:41:54.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.234+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 5 11 2024-02-10T08:41:54.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 12 2024-02-10T08:41:54.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 12 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000e350 osd=5 3 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 5 12 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 4 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 10 client.28023.objecter _send_command 12 2024-02-10T08:41:54.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.249+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a00017e50 con 0x7f2a000178f0 2024-02-10T08:41:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a00017e50 con 0x7f2a000178f0 2024-02-10T08:41:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 10 client.28023.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 4 2024-02-10T08:41:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 5 12 2024-02-10T08:41:54.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:41:54.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 13 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 13 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 6 13 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 4 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 10 client.28023.objecter _send_command 13 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00022ce0 con 0x7f2a0001eff0 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.251+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a0001eff0 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.252+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a00022ce0 con 0x7f2a0001eff0 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.252+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.252+0000 7f2a117fa700 10 client.28023.objecter _finish_command 13 = system:0 2024-02-10T08:41:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.252+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 4 2024-02-10T08:41:54.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.252+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 6 13 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.266+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 14 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 14 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a00017e50 osd=6 3 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 6 14 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 4 2024-02-10T08:41:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 10 client.28023.objecter _send_command 14 2024-02-10T08:41:54.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.267+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a0000c2a0 con 0x7f2a0001eff0 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.268+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a0000c2a0 con 0x7f2a0001eff0 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.268+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.268+0000 7f2a117fa700 10 client.28023.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.268+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 4 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.268+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 6 14 2024-02-10T08:41:54.269 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 15 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 15 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 7 15 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 4 2024-02-10T08:41:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 10 client.28023.objecter _send_command 15 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a00016f70 con 0x7f2a0000c410 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.269+0000 7f2a117fa700 10 client.28023.objecter ms_handle_connect 0x7f2a0000c410 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.270+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2a00016f70 con 0x7f2a0000c410 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.270+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.270+0000 7f2a117fa700 10 client.28023.objecter _finish_command 15 = system:0 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.270+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 4 2024-02-10T08:41:54.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.270+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 7 15 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 10 client.28023.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign -1 16 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 20 client.28023.objecter _get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_remove -1 16 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 20 client.28023.objecter get_session s=0x7f2a0000c2a0 osd=7 3 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 15 client.28023.objecter _session_command_op_assign 7 16 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 4 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 10 client.28023.objecter _send_command 16 2024-02-10T08:41:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.290+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2a00016f70 con 0x7f2a0000c410 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.292+0000 7f2a117fa700 1 -- 172.21.15.107:0/3968855078 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2a00016f70 con 0x7f2a0000c410 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.292+0000 7f2a117fa700 10 client.28023.objecter ms_dispatch 0x7f2a14001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.292+0000 7f2a117fa700 10 client.28023.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.292+0000 7f2a117fa700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 4 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.292+0000 7f2a117fa700 15 client.28023.objecter _session_command_op_remove 7 16 2024-02-10T08:41:54.293 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 librados: watch_flush enter 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 librados: watch_flush exit 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.0 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6801/3321596972 conn(0x7f2a00004530 legacy=0x7f2a00002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a000051a0 osd=0 1 2024-02-10T08:41:54.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.1 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6805/133516126 conn(0x7f2a00004150 legacy=0x7f2a0000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003fe0 osd=1 1 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.2 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6809/3575881728 conn(0x7f2a0000d1e0 legacy=0x7f2a0000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003630 osd=2 1 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.3 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6813/2871825691 conn(0x7f2a00011d40 legacy=0x7f2a00014130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00003420 osd=3 1 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.4 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.195:6800/2499671285 conn(0x7f2a0000cd40 legacy=0x7f2a00015b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e060 osd=4 1 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.5 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.195:6804/4178321486 conn(0x7f2a000178f0 legacy=0x7f2a0001d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000e350 osd=5 1 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a188fa700 1 -- 172.21.15.107:0/3968855078 reap_dead start 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.6 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.195:6808/1417479017 conn(0x7f2a0001eff0 legacy=0x7f2a00021460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a188fa700 1 -- 172.21.15.107:0/3968855078 reap_dead start 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a00017e50 osd=6 1 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter close_session for osd.7 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.195:6812/1934176928 conn(0x7f2a0000c410 legacy=0x7f2a00022ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter put_session s=0x7f2a0000c2a0 osd=7 1 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 20 client.28023.objecter shutdown clearing up homeless session... 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 10 client.28023.objecter successfully canceled tick 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a188fa700 1 -- 172.21.15.107:0/3968855078 reap_dead start 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a188fa700 1 -- 172.21.15.107:0/3968855078 reap_dead start 2024-02-10T08:41:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6800/2634842009 conn(0x7f29fc063be0 legacy=0x7f29fc0660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> v1:172.21.15.107:6789/0 conn(0x7f2a140fda50 legacy=0x7f2a140fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 shutdown_connections 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 >> 172.21.15.107:0/3968855078 conn(0x7f2a140f3640 msgr2=0x7f2a141077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 shutdown_connections 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 -- 172.21.15.107:0/3968855078 wait complete. 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:54.293+0000 7f2a1a35d700 1 librados: shutdown 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.298 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.298 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.298 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:41:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:52.362977+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:41:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:52.364040+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:41:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.438125+0000 mgr.y (mgr.14553) 367 : cluster [DBG] pgmap v601: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:53.572775+0000 mon.c (mon.2) 392 : audit [INF] from='client.? v1:172.21.15.107:0/1843645449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:53.573056+0000 mon.a (mon.0) 2466 : audit [INF] from='client.53597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.826900+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.828603+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:54.119709+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:54.135517+0000 mon.a (mon.0) 2468 : audit [DBG] from='client.? v1:172.21.15.107:0/3968855078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:52.362977+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:52.364040+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.438125+0000 mgr.y (mgr.14553) 367 : cluster [DBG] pgmap v601: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:53.572775+0000 mon.c (mon.2) 392 : audit [INF] from='client.? v1:172.21.15.107:0/1843645449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:53.573056+0000 mon.a (mon.0) 2466 : audit [INF] from='client.53597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.826900+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:41:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.828603+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:41:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:54.119709+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:41:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:54.135517+0000 mon.a (mon.0) 2468 : audit [DBG] from='client.? v1:172.21.15.107:0/3968855078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:54.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:52.362977+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:41:54.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:52.364040+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:41:54.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.438125+0000 mgr.y (mgr.14553) 367 : cluster [DBG] pgmap v601: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:53.572775+0000 mon.c (mon.2) 392 : audit [INF] from='client.? v1:172.21.15.107:0/1843645449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:53.573056+0000 mon.a (mon.0) 2466 : audit [INF] from='client.53597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.826900+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.828603+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:54.119709+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:41:54.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:54.135517+0000 mon.a (mon.0) 2468 : audit [DBG] from='client.? v1:172.21.15.107:0/3968855078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:41:54.761 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:54.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:54.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:54.894 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:54.992 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:41:54.992 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:41:54.992 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:54.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:55.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:55.126 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.571455+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:41:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.573121+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.591767+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:53.594647+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.060929+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.063835+0000 osd.5 (osd.5) 186 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.172747+0000 mon.a (mon.0) 2469 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:54.182209+0000 mon.a (mon.0) 2470 : audit [INF] from='client.53597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:54.182280+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.182320+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-10T08:41:55.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:55.159078+0000 mon.a (mon.0) 2473 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:55.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.571455+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:41:55.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.573121+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:41:55.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.591767+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:41:55.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:53.594647+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:41:55.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.060929+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.063835+0000 osd.5 (osd.5) 186 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.172747+0000 mon.a (mon.0) 2469 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:54.182209+0000 mon.a (mon.0) 2470 : audit [INF] from='client.53597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:54.182280+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.182320+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:55.159078+0000 mon.a (mon.0) 2473 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.571455+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.573121+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.591767+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:53.594647+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.060929+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.063835+0000 osd.5 (osd.5) 186 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:41:55.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.172747+0000 mon.a (mon.0) 2469 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:41:55.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:54.182209+0000 mon.a (mon.0) 2470 : audit [INF] from='client.53597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:41:55.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:54.182280+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? v1:172.21.15.107:0/479393855' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:41:55.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.182320+0000 mon.a (mon.0) 2472 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-10T08:41:55.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:55.159078+0000 mon.a (mon.0) 2473 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:41:55.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T08:41:55.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:41:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:41:55] "GET /metrics HTTP/1.1" 200 187534 "" "Prometheus/2.33.4" 2024-02-10T08:41:55.875 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:41:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:55.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:41:55.922 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:41:55.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:56.005 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:56.154 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:56.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:56.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T08:41:56.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.312492+0000 osd.6 (osd.6) 163 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:54.315979+0000 osd.6 (osd.6) 164 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.145504+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.145667+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.192949+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.270554+0000 osd.6 (osd.6) 167 : cluster [DBG] 60.d scrub starts 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.273150+0000 osd.6 (osd.6) 168 : cluster [DBG] 60.d scrub ok 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.438534+0000 mgr.y (mgr.14553) 368 : cluster [DBG] pgmap v604: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.668365+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:41:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:55.669795+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:41:56.310 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:56.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:56.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:56.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:56.469 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.312492+0000 osd.6 (osd.6) 163 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:41:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:54.315979+0000 osd.6 (osd.6) 164 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:41:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.145504+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.145667+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.192949+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.270554+0000 osd.6 (osd.6) 167 : cluster [DBG] 60.d scrub starts 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.273150+0000 osd.6 (osd.6) 168 : cluster [DBG] 60.d scrub ok 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.438534+0000 mgr.y (mgr.14553) 368 : cluster [DBG] pgmap v604: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.668365+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:55.669795+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:41:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.312492+0000 osd.6 (osd.6) 163 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:54.315979+0000 osd.6 (osd.6) 164 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.145504+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.145667+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.192949+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.270554+0000 osd.6 (osd.6) 167 : cluster [DBG] 60.d scrub starts 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.273150+0000 osd.6 (osd.6) 168 : cluster [DBG] 60.d scrub ok 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.438534+0000 mgr.y (mgr.14553) 368 : cluster [DBG] pgmap v604: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.668365+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:41:56.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:55.669795+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:41:57.318 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:57.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.262644+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.265554+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.710473+0000 osd.4 (osd.4) 182 : cluster [DBG] 4.e scrub starts 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.714111+0000 osd.4 (osd.4) 183 : cluster [DBG] 4.e scrub ok 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.262644+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.265554+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:41:57.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.710473+0000 osd.4 (osd.4) 182 : cluster [DBG] 4.e scrub starts 2024-02-10T08:41:57.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.714111+0000 osd.4 (osd.4) 183 : cluster [DBG] 4.e scrub ok 2024-02-10T08:41:57.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:41:57.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:57.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.262644+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:41:57.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.265554+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:41:57.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.710473+0000 osd.4 (osd.4) 182 : cluster [DBG] 4.e scrub starts 2024-02-10T08:41:57.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.714111+0000 osd.4 (osd.4) 183 : cluster [DBG] 4.e scrub ok 2024-02-10T08:41:57.449 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:57.547 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:57.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:57.559 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:41:57.559 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:41:57.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:57.678 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:57.776 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:57.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:57.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:41:57.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:57.939 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:57.996 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:58.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:58.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:58.129 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:58.151 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:58.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:58.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:41:58.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.645113+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:56.647989+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.439467+0000 mgr.y (mgr.14553) 369 : cluster [DBG] pgmap v605: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 957 B/s wr, 4 op/s 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.956379+0000 osd.3 (osd.3) 193 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.957322+0000 osd.3 (osd.3) 194 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:41:58.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.645113+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:41:58.283 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:56.647989+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:41:58.283 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.439467+0000 mgr.y (mgr.14553) 369 : cluster [DBG] pgmap v605: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 957 B/s wr, 4 op/s 2024-02-10T08:41:58.283 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.956379+0000 osd.3 (osd.3) 193 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:41:58.283 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.957322+0000 osd.3 (osd.3) 194 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:41:58.285 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.297+0000 7fb615a83700 1 -- 172.21.15.107:0/1091026953 >> v1:172.21.15.107:6790/0 conn(0x7fb6100f8a60 legacy=0x7fb6100f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.297+0000 7fb615a83700 1 -- 172.21.15.107:0/1091026953 shutdown_connections 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.297+0000 7fb615a83700 1 -- 172.21.15.107:0/1091026953 >> 172.21.15.107:0/1091026953 conn(0x7fb6100f2510 msgr2=0x7fb6100f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.298+0000 7fb615a83700 1 -- 172.21.15.107:0/1091026953 shutdown_connections 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.299+0000 7fb615a83700 1 -- 172.21.15.107:0/1091026953 wait complete. 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.299+0000 7fb615a83700 1 librados: starting msgr at 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.299+0000 7fb615a83700 1 librados: starting objecter 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 Processor -- start 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 -- start start 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 librados: setting wanted keys 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 librados: calling monclient init 2024-02-10T08:41:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb610069270 con 0x7fb610104330 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb610110080 con 0x7fb6100f8a60 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb615a83700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb610111220 con 0x7fb6100f9550 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb60f7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb6100f8a60 0x7fb610068710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:34824/0 (socket says 172.21.15.107:34824) 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.300+0000 7fb60f7fe700 1 -- 172.21.15.107:0/3489490356 learned_addr learned my addr 172.21.15.107:0/3489490356 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 10 client.?.objecter ms_handle_connect 0x7fb6100f8a60 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 10 client.?.objecter ms_handle_connect 0x7fb610104330 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211092766 0 0) 0x7fb610110080 con 0x7fb6100f8a60 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f80033f0 con 0x7fb6100f8a60 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938283714 0 0) 0x7fb610069270 con 0x7fb610104330 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb610110080 con 0x7fb610104330 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888603136 0 0) 0x7fb5f80033f0 con 0x7fb6100f8a60 2024-02-10T08:41:58.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb610069270 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb600002cb0 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127090354 0 0) 0x7fb610110080 con 0x7fb610104330 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f80033f0 con 0x7fb610104330 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb608001bc0 con 0x7fb610104330 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480110848 0 0) 0x7fb610069270 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 >> v1:172.21.15.107:6790/0 conn(0x7fb6100f9550 legacy=0x7fb610101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 >> v1:172.21.15.107:6789/0 conn(0x7fb610104330 legacy=0x7fb61010d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6101123c0 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61010f0d0 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb615a83700 10 client.48328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61010f540 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb615a83700 1 librados: init done 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb600001b60 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.301+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6000037f0 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.302+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc004f20 con 0x7fb6100f8a60 2024-02-10T08:41:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb600003970 con 0x7fb6100f8a60 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(417..417 src has 330..417) v4 ==== 8222+0+0 (unknown 2996338447 0 0) 0x7fb600079070 con 0x7fb6100f8a60 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 10 client.48328.objecter ms_dispatch 0x7fb610001550 osd_map(417..417 src has 330..417) v4 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 3 client.48328.objecter handle_osd_map got epochs [417,417] > 0 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 3 client.48328.objecter handle_osd_map decoding full epoch 417 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 20 client.48328.objecter dump_active .. 0 homeless 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.303+0000 7fb60cf74700 10 client.48328.objecter ms_handle_connect 0x7fb5f8063d10 2024-02-10T08:41:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.308+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb60004c4a0 con 0x7fb6100f8a60 2024-02-10T08:41:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:41:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:58.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:41:58.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:41:58.443 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.645113+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:41:58.443 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:56.647989+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:41:58.444 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.439467+0000 mgr.y (mgr.14553) 369 : cluster [DBG] pgmap v605: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 957 B/s wr, 4 op/s 2024-02-10T08:41:58.444 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.956379+0000 osd.3 (osd.3) 193 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:41:58.444 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.957322+0000 osd.3 (osd.3) 194 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:41:58.446 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:58.464 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:41:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:58.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:41:58.480 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:41:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:58.491+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fb5fc0051d0 con 0x7fb6100f8a60 2024-02-10T08:41:58.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:58.594 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:58.722 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:41:58.742 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:58.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:58.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:41:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:59.031 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:41:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:59.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:41:59.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:41:59.163 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.209+0000 7fb60cf74700 1 -- 172.21.15.107:0/3489490356 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v418) v1 ==== 113+0+0 (unknown 903473277 0 0) 0x7fb60004a910 con 0x7fb6100f8a60 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 10 librados: watch_flush enter 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 10 librados: watch_flush exit 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 20 client.48328.objecter shutdown clearing up homeless session... 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 10 client.48328.objecter successfully canceled tick 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb5f8063d10 legacy=0x7fb5f80661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.213+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 >> v1:172.21.15.195:6789/0 conn(0x7fb6100f8a60 legacy=0x7fb610068710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:41:59.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.214+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 shutdown_connections 2024-02-10T08:41:59.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.214+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 >> 172.21.15.107:0/3489490356 conn(0x7fb6100f2510 msgr2=0x7fb6101077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:41:59.215 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:41:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:59.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:41:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.218+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 shutdown_connections 2024-02-10T08:41:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.218+0000 7fb615a83700 1 -- 172.21.15.107:0/3489490356 wait complete. 2024-02-10T08:41:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:41:59.218+0000 7fb615a83700 1 librados: shutdown 2024-02-10T08:41:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T08:41:59.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:41:59.346 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:41:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.280972+0000 osd.6 (osd.6) 171 : cluster [DBG] 4.a scrub starts 2024-02-10T08:41:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.284802+0000 osd.6 (osd.6) 172 : cluster [DBG] 4.a scrub ok 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.563150+0000 osd.7 (osd.7) 173 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.565092+0000 osd.7 (osd.7) 174 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.596680+0000 osd.1 (osd.1) 191 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:57.599653+0000 osd.1 (osd.1) 192 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.268847+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.271724+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.298990+0000 osd.2 (osd.2) 149 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.300175+0000 osd.2 (osd.2) 150 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:58.492519+0000 mon.b (mon.1) 581 : audit [INF] from='client.? v1:172.21.15.107:0/3489490356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:58.492866+0000 mon.a (mon.0) 2475 : audit [INF] from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.956093+0000 osd.3 (osd.3) 195 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:41:59.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:58.956898+0000 osd.3 (osd.3) 196 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.280972+0000 osd.6 (osd.6) 171 : cluster [DBG] 4.a scrub starts 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.284802+0000 osd.6 (osd.6) 172 : cluster [DBG] 4.a scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.563150+0000 osd.7 (osd.7) 173 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.565092+0000 osd.7 (osd.7) 174 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.596680+0000 osd.1 (osd.1) 191 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:57.599653+0000 osd.1 (osd.1) 192 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.268847+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.271724+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.298990+0000 osd.2 (osd.2) 149 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.300175+0000 osd.2 (osd.2) 150 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:58.492519+0000 mon.b (mon.1) 581 : audit [INF] from='client.? v1:172.21.15.107:0/3489490356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:58.492866+0000 mon.a (mon.0) 2475 : audit [INF] from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.956093+0000 osd.3 (osd.3) 195 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:41:59.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:41:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:58.956898+0000 osd.3 (osd.3) 196 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:41:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.280972+0000 osd.6 (osd.6) 171 : cluster [DBG] 4.a scrub starts 2024-02-10T08:41:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.284802+0000 osd.6 (osd.6) 172 : cluster [DBG] 4.a scrub ok 2024-02-10T08:41:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.563150+0000 osd.7 (osd.7) 173 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:41:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.565092+0000 osd.7 (osd.7) 174 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:41:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.596680+0000 osd.1 (osd.1) 191 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:57.599653+0000 osd.1 (osd.1) 192 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.268847+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.271724+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.298990+0000 osd.2 (osd.2) 149 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.300175+0000 osd.2 (osd.2) 150 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:58.492519+0000 mon.b (mon.1) 581 : audit [INF] from='client.? v1:172.21.15.107:0/3489490356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:58.492866+0000 mon.a (mon.0) 2475 : audit [INF] from='client.48328 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.956093+0000 osd.3 (osd.3) 195 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:41:59.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:41:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:58.956898+0000 osd.3 (osd.3) 196 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:41:59.693 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:41:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:41:59.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:41:59.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:41:59.826 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:00.072 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:00.072 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:00.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:00.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:00.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:41:59.208702+0000 mon.a (mon.0) 2476 : audit [INF] from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:42:00.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.208802+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.345303+0000 osd.2 (osd.2) 151 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.348207+0000 osd.2 (osd.2) 152 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.440061+0000 mgr.y (mgr.14553) 370 : cluster [DBG] pgmap v607: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:41:59.208702+0000 mon.a (mon.0) 2476 : audit [INF] from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.208802+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.345303+0000 osd.2 (osd.2) 151 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.348207+0000 osd.2 (osd.2) 152 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:42:00.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.440061+0000 mgr.y (mgr.14553) 370 : cluster [DBG] pgmap v607: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:00.304 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:00.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:41:59.208702+0000 mon.a (mon.0) 2476 : audit [INF] from='client.48328 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.208802+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.345303+0000 osd.2 (osd.2) 151 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.348207+0000 osd.2 (osd.2) 152 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:42:00.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.440061+0000 mgr.y (mgr.14553) 370 : cluster [DBG] pgmap v607: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:00.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:00.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:42:00.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.711+0000 7fa67c5e4700 1 -- 172.21.15.107:0/526276521 >> v1:172.21.15.107:6789/0 conn(0x7fa674104350 legacy=0x7fa6741067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.711+0000 7fa67c5e4700 1 -- 172.21.15.107:0/526276521 shutdown_connections 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.711+0000 7fa67c5e4700 1 -- 172.21.15.107:0/526276521 >> 172.21.15.107:0/526276521 conn(0x7fa6740f2510 msgr2=0x7fa6740f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- 172.21.15.107:0/526276521 shutdown_connections 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- 172.21.15.107:0/526276521 wait complete. 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 librados: starting msgr at 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 librados: starting objecter 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 Processor -- start 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- start start 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 librados: setting wanted keys 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 librados: calling monclient init 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6741173c0 con 0x7fa6740f95d0 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6741185a0 con 0x7fa674104350 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67c5e4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6741197a0 con 0x7fa6740f8a20 2024-02-10T08:42:00.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67a380700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa6740f8a20 0x7fa6740ff560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34628/0 (socket says 172.21.15.107:34628) 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.712+0000 7fa67a380700 1 -- 172.21.15.107:0/3597658374 learned_addr learned my addr 172.21.15.107:0/3597658374 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter ms_handle_connect 0x7fa6740f8a20 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter ms_handle_connect 0x7fa6740f95d0 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter ms_handle_connect 0x7fa674104350 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947778127 0 0) 0x7fa6741197a0 con 0x7fa6740f8a20 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6600033f0 con 0x7fa6740f8a20 2024-02-10T08:42:00.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816631195 0 0) 0x7fa6741173c0 con 0x7fa6740f95d0 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6741197a0 con 0x7fa6740f95d0 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536263169 0 0) 0x7fa6741185a0 con 0x7fa674104350 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6741173c0 con 0x7fa674104350 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2874090761 0 0) 0x7fa6600033f0 con 0x7fa6740f8a20 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6741185a0 con 0x7fa6740f8a20 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa6640019c0 con 0x7fa6740f8a20 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988240545 0 0) 0x7fa6741197a0 con 0x7fa6740f95d0 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6600033f0 con 0x7fa6740f95d0 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779137604 0 0) 0x7fa6741173c0 con 0x7fa674104350 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6741197a0 con 0x7fa674104350 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa670001840 con 0x7fa6740f95d0 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa66c0023d0 con 0x7fa674104350 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 240867516 0 0) 0x7fa6741185a0 con 0x7fa6740f8a20 2024-02-10T08:42:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.195:6789/0 conn(0x7fa674104350 legacy=0x7fa674115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6789/0 conn(0x7fa6740f95d0 legacy=0x7fa6740ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67411a9a0 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6741175b0 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa67c5e4700 10 client.53618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa674117a20 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa67c5e4700 1 librados: init done 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa664002410 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.713+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa664005b90 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.714+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa658004f00 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa664001f70 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(418..418 src has 330..418) v4 ==== 8266+0+0 (unknown 433853089 0 0) 0x7fa664078e50 con 0x7fa6740f8a20 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 osd_map(418..418 src has 330..418) v4 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 3 client.53618.objecter handle_osd_map got epochs [418,418] > 0 2024-02-10T08:42:00.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 3 client.53618.objecter handle_osd_map decoding full epoch 418 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 20 client.53618.objecter dump_active .. 0 homeless 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v418) v1 ==== 54+0+16 (unknown 2583141506 0 2420000192) 0x7fa66407b260 con 0x7fa6740f8a20 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa660068100 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 1 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 1 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 0 1 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 4 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 10 client.53618.objecter _send_command 1 2024-02-10T08:42:00.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.715+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658004950 con 0x7fa658004530 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa658004530 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658004950 con 0x7fa658004530 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 1 = system:0 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 4 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.716+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 0 1 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:00.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 2 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 2 2024-02-10T08:42:00.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580051a0 osd=0 3 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 0 2 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 4 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 10 client.53618.objecter _send_command 2 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.731+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa658004d30 con 0x7fa658004530 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa658004d30 con 0x7fa658004530 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 4 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 0 2 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 3 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 1 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 4 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 10 client.53618.objecter _send_command 3 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.732+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa65800c000 con 0x7fa658004150 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa658004150 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa65800c000 con 0x7fa658004150 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 3 = system:0 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 4 2024-02-10T08:42:00.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.733+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 1 3 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 4 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 4 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658003fe0 osd=1 3 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 1 4 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 4 2024-02-10T08:42:00.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 10 client.53618.objecter _send_command 4 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.748+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa658001850 con 0x7fa658004150 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.749+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa658001850 con 0x7fa658004150 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.749+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.749+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.749+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 4 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.749+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 1 4 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 5 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 5 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 2 5 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 4 2024-02-10T08:42:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 10 client.53618.objecter _send_command 5 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658003e70 con 0x7fa65800d2f0 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa65800d2f0 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658003e70 con 0x7fa65800d2f0 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 5 = system:0 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 4 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.750+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 2 5 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 6 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 6 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658003630 osd=2 3 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 2 6 2024-02-10T08:42:00.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 4 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 10 client.53618.objecter _send_command 6 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.765+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa65800e730 con 0x7fa65800d2f0 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.766+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa65800e730 con 0x7fa65800d2f0 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.766+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.766+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.766+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 4 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.766+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 2 6 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:00.867 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 7 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 7 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 3 7 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 4 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 10 client.53618.objecter _send_command 7 2024-02-10T08:42:00.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658015aa0 con 0x7fa658011e80 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa658011e80 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658015aa0 con 0x7fa658011e80 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 7 = system:0 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 4 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.767+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 3 7 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 8 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 8 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580033e0 osd=3 3 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 3 8 2024-02-10T08:42:00.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 4 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 10 client.53618.objecter _send_command 8 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.782+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa65800e1a0 con 0x7fa658011e80 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa65800e1a0 con 0x7fa658011e80 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 4 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 3 8 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 9 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.783+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 9 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 4 9 2024-02-10T08:42:00.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 4 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 10 client.53618.objecter _send_command 9 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa65800e100 con 0x7fa65800ce50 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.784+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa65800ce50 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.785+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa65800e100 con 0x7fa65800ce50 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.785+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.785+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 9 = system:0 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.785+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 4 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.785+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 4 9 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 10 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 10 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa65800df60 osd=4 3 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 4 10 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 4 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 10 client.53618.objecter _send_command 10 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.799+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa65800e100 con 0x7fa65800ce50 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:00.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa65800e100 con 0x7fa65800ce50 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 4 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 4 10 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 11 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 11 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 5 11 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 4 2024-02-10T08:42:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 10 client.53618.objecter _send_command 11 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.801+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658017ff0 con 0x7fa658017a30 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa658017a30 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658017ff0 con 0x7fa658017a30 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 11 = system:0 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 4 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.802+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 5 11 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 12 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 12 2024-02-10T08:42:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa65800e100 osd=5 3 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 5 12 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 4 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 10 client.53618.objecter _send_command 12 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.816+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa658017ff0 con 0x7fa658017a30 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa658017ff0 con 0x7fa658017a30 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 4 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 5 12 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 13 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 13 2024-02-10T08:42:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 6 13 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 4 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 10 client.53618.objecter _send_command 13 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658022c30 con 0x7fa65801f080 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.818+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa65801f080 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.819+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658022c30 con 0x7fa65801f080 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.819+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.819+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 13 = system:0 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.819+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 4 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.819+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 6 13 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 14 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 14 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa658017ff0 osd=6 3 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 6 14 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 4 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 10 client.53618.objecter _send_command 14 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.833+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6580243e0 con 0x7fa65801f080 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.834+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa6580243e0 con 0x7fa65801f080 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.834+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.834+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.834+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 4 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.834+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 6 14 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 15 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 15 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 7 15 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 4 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 10 client.53618.objecter _send_command 15 2024-02-10T08:42:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa658025020 con 0x7fa658024a90 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.835+0000 7fa66b7fe700 10 client.53618.objecter ms_handle_connect 0x7fa658024a90 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.836+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa658025020 con 0x7fa658024a90 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.836+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.836+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 15 = system:0 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.836+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 4 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.836+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 7 15 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 10 client.53618.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign -1 16 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 20 client.53618.objecter _get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_remove -1 16 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 20 client.53618.objecter get_session s=0x7fa6580243e0 osd=7 3 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 15 client.53618.objecter _session_command_op_assign 7 16 2024-02-10T08:42:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 4 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 10 client.53618.objecter _send_command 16 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.850+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa658025020 con 0x7fa658024a90 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa66b7fe700 1 -- 172.21.15.107:0/3597658374 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa658025020 con 0x7fa658024a90 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa66b7fe700 10 client.53618.objecter ms_dispatch 0x7fa674001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa66b7fe700 10 client.53618.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa66b7fe700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 4 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa66b7fe700 15 client.53618.objecter _session_command_op_remove 7 16 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 librados: watch_flush enter 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 librados: watch_flush exit 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.0 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6801/3321596972 conn(0x7fa658004530 legacy=0x7fa658002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580051a0 osd=0 1 2024-02-10T08:42:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.1 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6805/133516126 conn(0x7fa658004150 legacy=0x7fa65800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003fe0 osd=1 1 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.2 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6809/3575881728 conn(0x7fa65800d2f0 legacy=0x7fa65800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658003630 osd=2 1 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.3 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6813/2871825691 conn(0x7fa658011e80 legacy=0x7fa658014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580033e0 osd=3 1 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.4 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.195:6800/2499671285 conn(0x7fa65800ce50 legacy=0x7fa658015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67ab81700 1 -- 172.21.15.107:0/3597658374 reap_dead start 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800df60 osd=4 1 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.5 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.851+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.195:6804/4178321486 conn(0x7fa658017a30 legacy=0x7fa65801d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67ab81700 1 -- 172.21.15.107:0/3597658374 reap_dead start 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67ab81700 1 -- 172.21.15.107:0/3597658374 reap_dead start 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa65800e100 osd=5 1 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.6 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.195:6808/1417479017 conn(0x7fa65801f080 legacy=0x7fa658021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa658017ff0 osd=6 1 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 10 client.53618.objecter close_session for osd.7 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.195:6812/1934176928 conn(0x7fa658024a90 legacy=0x7fa658022c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 20 client.53618.objecter put_session s=0x7fa6580243e0 osd=7 1 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 20 client.53618.objecter shutdown clearing up homeless session... 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 10 client.53618.objecter successfully canceled tick 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa660068100 legacy=0x7fa66006a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> v1:172.21.15.107:6790/0 conn(0x7fa6740f8a20 legacy=0x7fa6740ff560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67ab81700 1 -- 172.21.15.107:0/3597658374 reap_dead start 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 shutdown_connections 2024-02-10T08:42:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 >> 172.21.15.107:0/3597658374 conn(0x7fa6740f2510 msgr2=0x7fa674103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 shutdown_connections 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 -- 172.21.15.107:0/3597658374 wait complete. 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:00.852+0000 7fa67c5e4700 1 librados: shutdown 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:42:00.935 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:01.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.566312+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:41:59.569190+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:00.247852+0000 osd.6 (osd.6) 175 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:00.249562+0000 osd.6 (osd.6) 176 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:00.715790+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/3597658374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:01.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.023584+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.566312+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:41:59.569190+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:00.247852+0000 osd.6 (osd.6) 175 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:00.249562+0000 osd.6 (osd.6) 176 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:00.715790+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/3597658374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.023584+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-10T08:42:01.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.566312+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:42:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:41:59.569190+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:42:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:00.247852+0000 osd.6 (osd.6) 175 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:42:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:00.249562+0000 osd.6 (osd.6) 176 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:42:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:00.715790+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.107:0/3597658374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:01.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.023584+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-10T08:42:01.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:01.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:01.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:02.021 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:02.049 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:02.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:02.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:02.179 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:02.325 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.440931+0000 mgr.y (mgr.14553) 371 : cluster [DBG] pgmap v609: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T08:42:02.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:01.458714+0000 mgr.y (mgr.14553) 372 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:02.327 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:02.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:02.394 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:02.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:02.459 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:02.585 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:42:02.585 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:42:02.585 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:42:02.585 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:42:02.585 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:42:02.606 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.440931+0000 mgr.y (mgr.14553) 371 : cluster [DBG] pgmap v609: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T08:42:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:01.458714+0000 mgr.y (mgr.14553) 372 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.440931+0000 mgr.y (mgr.14553) 371 : cluster [DBG] pgmap v609: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T08:42:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:01.458714+0000 mgr.y (mgr.14553) 372 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:02.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:02.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:02.739 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:02.791 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:02.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:02.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:02.921 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:03.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:42:03.269 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:03.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:03.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:03.401 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:03.526 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.484625+0000 osd.7 (osd.7) 177 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.488094+0000 osd.7 (osd.7) 178 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.670944+0000 osd.1 (osd.1) 193 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:01.673929+0000 osd.1 (osd.1) 194 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.121522+0000 osd.5 (osd.5) 187 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.124398+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.180706+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.183617+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.262968+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:02.263135+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:03.600 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.484625+0000 osd.7 (osd.7) 177 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.488094+0000 osd.7 (osd.7) 178 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.670944+0000 osd.1 (osd.1) 193 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:01.673929+0000 osd.1 (osd.1) 194 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.121522+0000 osd.5 (osd.5) 187 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.124398+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.180706+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.183617+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.262968+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:03.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:02.263135+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.484625+0000 osd.7 (osd.7) 177 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.488094+0000 osd.7 (osd.7) 178 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.670944+0000 osd.1 (osd.1) 193 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:01.673929+0000 osd.1 (osd.1) 194 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.121522+0000 osd.5 (osd.5) 187 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.124398+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.180706+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.183617+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.262968+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:03.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:02.263135+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:03.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:03.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:03.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:42:04.281 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:04.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:04.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:03.230682+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-10T08:42:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:03.441328+0000 mgr.y (mgr.14553) 373 : cluster [DBG] pgmap v611: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:03.644691+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:42:04.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:03.646370+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- 172.21.15.107:0/2228132380 >> v1:172.21.15.107:6790/0 conn(0x7f8f90103250 legacy=0x7f8f901056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- 172.21.15.107:0/2228132380 shutdown_connections 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- 172.21.15.107:0/2228132380 >> 172.21.15.107:0/2228132380 conn(0x7f8f900f2510 msgr2=0x7f8f900f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- 172.21.15.107:0/2228132380 shutdown_connections 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- 172.21.15.107:0/2228132380 wait complete. 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 librados: starting msgr at 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 librados: starting objecter 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 Processor -- start 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 -- start start 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 librados: setting wanted keys 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.595+0000 7f8f96648700 1 librados: calling monclient init 2024-02-10T08:42:04.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f96648700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f90117320 con 0x7f8f90103250 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f96648700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f90118500 con 0x7f8f900fc910 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f96648700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f90119700 con 0x7f8f900ff710 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8f900ff710 0x7f8f900f8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34644/0 (socket says 172.21.15.107:34644) 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8f7fe700 1 -- 172.21.15.107:0/1275280930 learned_addr learned my addr 172.21.15.107:0/1275280930 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f8f900ff710 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f8f90103250 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f8f900fc910 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053505068 0 0) 0x7f8f90117320 con 0x7f8f90103250 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f78003800 con 0x7f8f90103250 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055661539 0 0) 0x7f8f90119700 con 0x7f8f900ff710 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f90117320 con 0x7f8f900ff710 2024-02-10T08:42:04.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200423471 0 0) 0x7f8f90118500 con 0x7f8f900fc910 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f90119700 con 0x7f8f900fc910 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366624769 0 0) 0x7f8f78003800 con 0x7f8f90103250 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f90118500 con 0x7f8f90103250 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8f880032a0 con 0x7f8f90103250 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.596+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007236535 0 0) 0x7f8f90117320 con 0x7f8f900ff710 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f78003800 con 0x7f8f900ff710 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751335515 0 0) 0x7f8f90119700 con 0x7f8f900fc910 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f90117320 con 0x7f8f900fc910 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8f84001840 con 0x7f8f900ff710 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8f80001620 con 0x7f8f900fc910 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2170140518 0 0) 0x7f8f90118500 con 0x7f8f90103250 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 >> v1:172.21.15.107:6790/0 conn(0x7f8f900ff710 legacy=0x7f8f900f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 >> v1:172.21.15.195:6789/0 conn(0x7f8f900fc910 legacy=0x7f8f900f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9011a900 con 0x7f8f90103250 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f90117510 con 0x7f8f90103250 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f96648700 10 client.28041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f90117a00 con 0x7f8f90103250 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f96648700 1 librados: init done 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8f88002670 con 0x7f8f90103250 2024-02-10T08:42:04.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.597+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8f88003a70 con 0x7f8f90103250 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c004f20 con 0x7f8f90103250 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8f880169c0 con 0x7f8f90103250 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(420..420 src has 330..420) v4 ==== 8222+0+0 (unknown 3936225350 0 0) 0x7f8f88076c00 con 0x7f8f90103250 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f8d7fa700 10 client.28041.objecter ms_dispatch 0x7f8f90001550 osd_map(420..420 src has 330..420) v4 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f8d7fa700 3 client.28041.objecter handle_osd_map got epochs [420,420] > 0 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.598+0000 7f8f8d7fa700 3 client.28041.objecter handle_osd_map decoding full epoch 420 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.599+0000 7f8f8d7fa700 20 client.28041.objecter dump_active .. 0 homeless 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.599+0000 7f8f8d7fa700 10 client.28041.objecter ms_handle_connect 0x7f8f78063d30 2024-02-10T08:42:04.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.602+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8f8804aec0 con 0x7f8f90103250 2024-02-10T08:42:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:03.230682+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-10T08:42:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:03.441328+0000 mgr.y (mgr.14553) 373 : cluster [DBG] pgmap v611: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:03.644691+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:42:04.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:03.646370+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:03.230682+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-10T08:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:03.441328+0000 mgr.y (mgr.14553) 373 : cluster [DBG] pgmap v611: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:03.644691+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.1b deep-scrub starts 2024-02-10T08:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:03.646370+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b deep-scrub ok 2024-02-10T08:42:04.760 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:04.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:04.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.797+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f8f7c0051d0 con 0x7f8f90103250 2024-02-10T08:42:04.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:04.895 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:04.922 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:04.925 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:04.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:04.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 -- 172.21.15.107:0/3347858799 >> v1:172.21.15.107:6790/0 conn(0x7fc54c10ccc0 legacy=0x7fc54c10f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 -- 172.21.15.107:0/3347858799 shutdown_connections 2024-02-10T08:42:04.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 -- 172.21.15.107:0/3347858799 >> 172.21.15.107:0/3347858799 conn(0x7fc54c063bc0 msgr2=0x7fc54c0672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:04.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 -- 172.21.15.107:0/3347858799 shutdown_connections 2024-02-10T08:42:04.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 -- 172.21.15.107:0/3347858799 wait complete. 2024-02-10T08:42:04.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 librados: starting msgr at 2024-02-10T08:42:04.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.952+0000 7fc55110a700 1 librados: starting objecter 2024-02-10T08:42:04.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 Processor -- start 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 -- start start 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 librados: setting wanted keys 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 librados: calling monclient init 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc54c11fba0 con 0x7fc54c069650 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc54c120d80 con 0x7fc54c1091b0 2024-02-10T08:42:04.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc55110a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc54c121f80 con 0x7fc54c10ccc0 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc54b59e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc54c10ccc0 0x7fc54c11e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34672/0 (socket says 172.21.15.107:34672) 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc54b59e700 1 -- 172.21.15.107:0/4231963011 learned_addr learned my addr 172.21.15.107:0/4231963011 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc533fff700 10 client.?.objecter ms_handle_connect 0x7fc54c10ccc0 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc533fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc533fff700 10 client.?.objecter ms_handle_connect 0x7fc54c069650 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.953+0000 7fc533fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418580218 0 0) 0x7fc54c11fba0 con 0x7fc54c069650 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5340033f0 con 0x7fc54c069650 2024-02-10T08:42:04.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586832125 0 0) 0x7fc54c121f80 con 0x7fc54c10ccc0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c11fba0 con 0x7fc54c10ccc0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 10 client.?.objecter ms_handle_connect 0x7fc54c1091b0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312830521 0 0) 0x7fc5340033f0 con 0x7fc54c069650 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc54c121f80 con 0x7fc54c069650 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820508461 0 0) 0x7fc54c120d80 con 0x7fc54c1091b0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5340033f0 con 0x7fc54c1091b0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806403578 0 0) 0x7fc54c11fba0 con 0x7fc54c10ccc0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc54c120d80 con 0x7fc54c10ccc0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc53c0019d0 con 0x7fc54c069650 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc544001360 con 0x7fc54c10ccc0 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3909672838 0 0) 0x7fc54c121f80 con 0x7fc54c069650 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 >> v1:172.21.15.107:6790/0 conn(0x7fc54c10ccc0 legacy=0x7fc54c11e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 >> v1:172.21.15.195:6789/0 conn(0x7fc54c1091b0 legacy=0x7fc54c105bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c123180 con 0x7fc54c069650 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c11fd90 con 0x7fc54c069650 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc55110a700 10 client.28047.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c120200 con 0x7fc54c069650 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.954+0000 7fc55110a700 1 librados: init done 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.955+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc53c002300 con 0x7fc54c069650 2024-02-10T08:42:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.955+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc53c003420 con 0x7fc54c069650 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.955+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc538004f20 con 0x7fc54c069650 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.956+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc53c001e60 con 0x7fc54c069650 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.957+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(420..420 src has 330..420) v4 ==== 8222+0+0 (unknown 3936225350 0 0) 0x7fc53c076490 con 0x7fc54c069650 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.957+0000 7fc533fff700 10 client.28047.objecter ms_dispatch 0x7fc54c001550 osd_map(420..420 src has 330..420) v4 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.957+0000 7fc533fff700 3 client.28047.objecter handle_osd_map got epochs [420,420] > 0 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.957+0000 7fc533fff700 3 client.28047.objecter handle_osd_map decoding full epoch 420 2024-02-10T08:42:04.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.957+0000 7fc533fff700 20 client.28047.objecter dump_active .. 0 homeless 2024-02-10T08:42:04.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.959+0000 7fc533fff700 10 client.28047.objecter ms_handle_connect 0x7fc5340639c0 2024-02-10T08:42:04.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:04.961+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc53c049800 con 0x7fc54c069650 2024-02-10T08:42:05.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:545 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94547991872512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e77)", 2024-02-10T08:42:05.090 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 443.58119166099999, 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19536709899999999, 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 545 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506555+0000", 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506554+0000", 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8359999999999992e-06 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506562+0000", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.31e-06 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506567+0000", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5569999999999997e-06 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506572+0000", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4697999999999998e-05 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506617+0000", 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2550000000000003e-06 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.506623+0000", 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19260776499999999 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.699231+0000", 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011743699999999999 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.699348+0000", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011991020000000001 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700547+0000", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116805 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700664+0000", 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029738300000000002 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.700961+0000", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0784000000000004e-05 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.701042+0000", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00087991499999999999 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:34:41.701922+0000", 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19536709899999999 2024-02-10T08:42:05.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:735 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 338.56090125200001, 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999, 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 735 2024-02-10T08:42:05.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526842+0000", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2619999999999999e-05 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526855+0000", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5109e-05 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526880+0000", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1497e-05 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526891+0000", 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017202499999999999 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527063+0000", 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3579999999999997e-06 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527071+0000", 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62625176299999996 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153322+0000", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0579e-05 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153353+0000", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:736 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:42:05.101 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 338.56083625299999, 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998, 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 736 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526909+0000", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.87e-06 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526914+0000", 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7660000000000004e-06 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526923+0000", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7569999999999998e-06 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526930+0000", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016634899999999999 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527096+0000", 2024-02-10T08:42:05.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3100000000000001e-06 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527104+0000", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62646392500000003 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153568+0000", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1849999999999998e-06 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153575+0000", 2024-02-10T08:42:05.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:737 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 3] snapc 0=[] ondisk+write+known_if_redirected e99)", 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 336.93601701900002, 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172, 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:42:05.106 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 737 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151728+0000", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.536e-06 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151734+0000", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.661e-06 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151741+0000", 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0990000000000004e-06 2024-02-10T08:42:05.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151747+0000", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069385099999999999 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152441+0000", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7069999999999998e-06 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152450+0000", 2024-02-10T08:42:05.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0056702749999999 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158120+0000", 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000171347 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158292+0000", 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000874083 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.159166+0000", 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35219.0:6 51.6s0 51:6ab6d0da:test-rados-api-smithi107-112451-13::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 318.19205263100002, 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344, 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.111 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35219", 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 6 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895691+0000", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0546e-05 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895702+0000", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0954999999999999e-05 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895713+0000", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0730000000000005e-06 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895721+0000", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012145484999999999 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907866+0000", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101523 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907968+0000", 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015504442 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.923472+0000", 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35222.0:9 50.4s0 50:25cf229d:test-rados-api-smithi107-112459-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:42:05.115 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 317.14279613299999, 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020488565, 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35222", 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4111016630", 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944950+0000", 2024-02-10T08:42:05.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944948+0000", 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0909999999999993e-06 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944956+0000", 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4697e-05 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944971+0000", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2599999999999999e-06 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.944978+0000", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4249000000000001e-05 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.945042+0000", 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011398 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.945156+0000", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020282664999999998 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:47.965439+0000", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020488565 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25276.0:8 56.1 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:42:05.122 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 314.12099486099999, 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999, 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25276", 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 8 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:42:05.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966750+0000", 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4189999999999999e-06 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966756+0000", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4120000000000004e-06 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966764+0000", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5940000000000004e-06 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966768+0000", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9002999999999999e-05 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966807+0000", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016647400000000001 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966974+0000", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012392504 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979366+0000", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6769999999999999e-05 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979443+0000", 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110282 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979553+0000", 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036667200000000002 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979920+0000", 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013358821 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.993279+0000", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025924506 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019203+0000", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9867999999999998e-05 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019253+0000", 2024-02-10T08:42:05.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0630999999999998e-05 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019294+0000", 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.43309.0:2 90.13 90:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 290.03232038599998, 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514, 2024-02-10T08:42:05.129 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.43309", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4276406080", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:42:05.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055424+0000", 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2330000000000001e-05 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055437+0000", 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5120000000000002e-06 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055444+0000", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3789999999999997e-06 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055450+0000", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2941000000000002e-05 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055482+0000", 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5740000000000003e-06 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055488+0000", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016882637999999998 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072371+0000", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010682 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072477+0000", 2024-02-10T08:42:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115103 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072593+0000", 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031995 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072913+0000", 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013010588 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.085923+0000", 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012255667 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098179+0000", 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1485e-05 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098220+0000", 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6390999999999998e-05 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098257+0000", 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:4 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e146)", 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.14118818499998, 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001, 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.136 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 4 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946556+0000", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7836e-05 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946574+0000", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7283999999999999e-05 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946591+0000", 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.516e-06 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946601+0000", 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8421000000000001e-05 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946669+0000", 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5779999999999999e-06 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946678+0000", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098724250000000006 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956550+0000", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012298100000000001 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956673+0000", 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00064271 2024-02-10T08:42:05.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.957316+0000", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011186949999999999 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.968503+0000", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01058679 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979089+0000", 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3553000000000005e-05 2024-02-10T08:42:05.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979173+0000", 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:9 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 283.16391126799999, 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001, 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.142 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923833+0000", 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7350000000000002e-06 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923842+0000", 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6348999999999999e-05 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923858+0000", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8599999999999999e-06 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923867+0000", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7614000000000001e-05 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923925+0000", 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0160000000000008e-06 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923933+0000", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088728850000000005 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932805+0000", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9782000000000001e-05 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932905+0000", 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040048699999999999 2024-02-10T08:42:05.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.933306+0000", 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075809029999999999 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.940887+0000", 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011264712 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952151+0000", 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8347000000000001e-05 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952220+0000", 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27168.0:2 103.6 103:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-02-10T08:42:05.148 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 274.14418948399998, 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017574042000000002, 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27168", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/709535540", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:42:05.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943557+0000", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943554+0000", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1824000000000001e-05 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943566+0000", 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2244e-05 2024-02-10T08:42:05.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943578+0000", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2460000000000003e-06 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943587+0000", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7071999999999998e-05 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943654+0000", 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0369e-05 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.943664+0000", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084003250000000002 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952064+0000", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115461 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952180+0000", 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014554699999999999 2024-02-10T08:42:05.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.952325+0000", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003713872 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.956039+0000", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046460440000000002 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.960685+0000", 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035895300000000002 2024-02-10T08:42:05.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961044+0000", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7293000000000003e-05 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961092+0000", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9542000000000002e-05 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:30.961131+0000", 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017574042000000002 2024-02-10T08:42:05.154 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47650.0:2 135.1f 135:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 236.055087383, 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394, 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47650", 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1937376491", 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:05.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032657+0000", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2010000000000002e-06 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032664+0000", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1249999999999999e-06 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032673+0000", 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4060000000000004e-06 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032679+0000", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0767999999999999e-05 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032730+0000", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4550000000000001e-06 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032739+0000", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23991668399999999 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272656+0000", 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124196 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272780+0000", 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152177 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272932+0000", 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001504109 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.274436+0000", 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001049901 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.275486+0000", 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027108150000000001 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278197+0000", 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3831e-05 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278251+0000", 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0742999999999999e-05 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278292+0000", 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47734.0:2 147.1a 147:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 215.13098474500001, 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332, 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47734", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/29749952", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956760+0000", 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956768+0000", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956776+0000", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0839999999999999e-06 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956781+0000", 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7848000000000001e-05 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956819+0000", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.19e-06 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956825+0000", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090413761999999995 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047239+0000", 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112608 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047352+0000", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014200000000000001 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047494+0000", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012910860000000001 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.048785+0000", 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063636400000000003 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.055149+0000", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023764580000000001 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057525+0000", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8439000000000001e-05 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057583+0000", 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1635999999999998e-05 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057635+0000", 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3265 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e295)", 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.024858+0000", 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.06288873700001, 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022765114999999999, 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.168 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3265 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024858+0000", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024858+0000", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024857+0000", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3829999999999999e-06 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024860+0000", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8840000000000004e-06 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024864+0000", 2024-02-10T08:42:05.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5890000000000001e-06 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.024866+0000", 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010898947000000001 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035765+0000", 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0448999999999998e-05 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035816+0000", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011667752 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.047483+0000", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2244000000000007e-05 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.047576+0000", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7355999999999999e-05 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.047623+0000", 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022765114999999999 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3362 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [stat] snapc 0=[] ondisk+read+known_if_redirected e295)", 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.035477+0000", 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.052269948, 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017540694999999999, 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3362 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035477+0000", 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035477+0000", 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035476+0000", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2919999999999998e-06 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035478+0000", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6330000000000004e-06 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035483+0000", 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5500000000000001e-06 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035485+0000", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017426447000000001 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.052912+0000", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0569000000000002e-05 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.052973+0000", 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4784999999999999e-05 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053017+0000", 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017540694999999999 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3363 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [stat] snapc 0=[] ondisk+read+known_if_redirected e295)", 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.035590+0000", 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.052156164, 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017615885000000001, 2024-02-10T08:42:05.177 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3363 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035590+0000", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035590+0000", 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035590+0000", 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.362e-06 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035592+0000", 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4610000000000002e-06 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035600+0000", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4470000000000002e-06 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035603+0000", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017448495000000001 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053052+0000", 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1720999999999999e-05 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053133+0000", 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3036999999999999e-05 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053206+0000", 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017615885000000001 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.181 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3364 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [stat] snapc 0=[] ondisk+read+known_if_redirected e295)", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.035700+0000", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.052046582, 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017620790000000001, 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3364 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035700+0000", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035700+0000", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035699+0000", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5110000000000002e-06 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035702+0000", 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8809999999999998e-06 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035706+0000", 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5059999999999998e-06 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035708+0000", 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017525635000000001 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053234+0000", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5751999999999998e-05 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053280+0000", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1146999999999998e-05 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053321+0000", 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017620790000000001 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.185 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3365 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [stat] snapc 0=[] ondisk+read+known_if_redirected e295)", 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.035838+0000", 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.05190826500001, 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017665862000000001, 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3365 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035838+0000", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035838+0000", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035838+0000", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4200000000000001e-06 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035840+0000", 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8980000000000004e-06 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035845+0000", 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5799999999999999e-06 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.035848+0000", 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017503417 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053351+0000", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9060000000000005e-05 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053430+0000", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4128999999999994e-05 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053504+0000", 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017665862000000001 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.189 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3367 186.13 186:ca51e39d:test-rados-api-smithi107-112490-38::foo:head [stat] snapc 0=[] ondisk+read+known_if_redirected e295)", 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:49.036254+0000", 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 136.05149267100001, 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017539803, 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3367 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.190 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036254+0000", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036254+0000", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036253+0000", 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2460000000000002e-06 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036256+0000", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3799999999999998e-06 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036259+0000", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2979999999999999e-06 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.036261+0000", 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017377647999999999 2024-02-10T08:42:05.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053639+0000", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3120000000000004e-05 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053722+0000", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1526999999999995e-05 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:49.053794+0000", 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017539803 2024-02-10T08:42:05.193 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53393.0:1 195.1s0 195:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e330)", 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 101.06064891600001, 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441, 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53393", 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4089636740", 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:42:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027094+0000", 2024-02-10T08:42:05.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1878999999999999e-05 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027106+0000", 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9205999999999999e-05 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027125+0000", 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0699999999999999e-05 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:05.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027136+0000", 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017503530000000001 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028886+0000", 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7790000000000003e-06 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028891+0000", 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.008200303 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037091+0000", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014759500000000001 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037239+0000", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048123099999999999 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037720+0000", 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015191219999999999 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.039239+0000", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001215224 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040455+0000", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0549e-05 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040505+0000", 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.199 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.200 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.200 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:05.200 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:05.200 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:42:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.157+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7fc5380051d0 con 0x7fc54c069650 2024-02-10T08:42:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:05.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.248+0000 7f8f8d7fa700 1 -- 172.21.15.107:0/1275280930 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v421) v1 ==== 87+0+0 (unknown 1671751658 0 0) 0x7f8f88049330 con 0x7f8f90103250 2024-02-10T08:42:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.248+0000 7fc533fff700 1 -- 172.21.15.107:0/4231963011 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v421) v1 ==== 113+0+0 (unknown 234661989 0 0) 0x7fc53c047c10 con 0x7fc54c069650 2024-02-10T08:42:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 10 librados: watch_flush enter 2024-02-10T08:42:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 10 librados: watch_flush exit 2024-02-10T08:42:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 20 client.28047.objecter shutdown clearing up homeless session... 2024-02-10T08:42:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 10 client.28047.objecter successfully canceled tick 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc5340639c0 legacy=0x7fc534065e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 >> v1:172.21.15.107:6789/0 conn(0x7fc54c069650 legacy=0x7fc54c1054c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 shutdown_connections 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.251+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 >> 172.21.15.107:0/4231963011 conn(0x7fc54c063bc0 msgr2=0x7fc54c0682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.252+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 shutdown_connections 2024-02-10T08:42:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.252+0000 7fc55110a700 1 -- 172.21.15.107:0/4231963011 wait complete. 2024-02-10T08:42:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 10 librados: watch_flush enter 2024-02-10T08:42:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 10 librados: watch_flush exit 2024-02-10T08:42:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 20 client.28041.objecter shutdown clearing up homeless session... 2024-02-10T08:42:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 10 client.28041.objecter successfully canceled tick 2024-02-10T08:42:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8f78063d30 legacy=0x7f8f780661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 >> v1:172.21.15.107:6789/0 conn(0x7f8f90103250 legacy=0x7f8f90115ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7fc55110a700 1 librados: shutdown 2024-02-10T08:42:05.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 shutdown_connections 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 >> 172.21.15.107:0/1275280930 conn(0x7f8f900f2510 msgr2=0x7f8f90060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 shutdown_connections 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 -- 172.21.15.107:0/1275280930 wait complete. 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:05.254+0000 7f8f96648700 1 librados: shutdown 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T08:42:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:42:05.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:05.332 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:05.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:04.360280+0000 osd.2 (osd.2) 153 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:42:05.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:04.361422+0000 osd.2 (osd.2) 154 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:42:05.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:04.798651+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:05.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.066345+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-02-10T08:42:05.385 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.067402+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-02-10T08:42:05.385 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:05.158308+0000 mon.a (mon.0) 2481 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:42:05.385 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:05.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:04.360280+0000 osd.2 (osd.2) 153 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:04.361422+0000 osd.2 (osd.2) 154 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:04.798651+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.066345+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.067402+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-02-10T08:42:05.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:05.158308+0000 mon.a (mon.0) 2481 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:04.360280+0000 osd.2 (osd.2) 153 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:04.361422+0000 osd.2 (osd.2) 154 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:04.798651+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.066345+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.067402+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-02-10T08:42:05.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:05.158308+0000 mon.a (mon.0) 2481 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:42:05.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:05.528 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:05.654 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:05.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:05.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:42:05.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:05] "GET /metrics HTTP/1.1" 200 187534 "" "Prometheus/2.33.4" 2024-02-10T08:42:05.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:05.785 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:05.962 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:06.434 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:06.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:04.639415+0000 osd.1 (osd.1) 195 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:04.642900+0000 osd.1 (osd.1) 196 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.102977+0000 osd.5 (osd.5) 189 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.106457+0000 osd.5 (osd.5) 190 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.200451+0000 osd.6 (osd.6) 179 : cluster [DBG] 16.d scrub starts 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.201310+0000 osd.6 (osd.6) 180 : cluster [DBG] 16.d scrub ok 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.242332+0000 mon.a (mon.0) 2482 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:05.248792+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:06.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:05.248851+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:42:06.456 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.248884+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-10T08:42:06.456 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:05.441890+0000 mgr.y (mgr.14553) 374 : cluster [DBG] pgmap v613: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:06.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:06.566 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:06.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:04.639415+0000 osd.1 (osd.1) 195 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:42:06.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:04.642900+0000 osd.1 (osd.1) 196 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:42:06.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.102977+0000 osd.5 (osd.5) 189 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:42:06.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.106457+0000 osd.5 (osd.5) 190 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.200451+0000 osd.6 (osd.6) 179 : cluster [DBG] 16.d scrub starts 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.201310+0000 osd.6 (osd.6) 180 : cluster [DBG] 16.d scrub ok 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.242332+0000 mon.a (mon.0) 2482 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:05.248792+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:05.248851+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.248884+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:05.441890+0000 mgr.y (mgr.14553) 374 : cluster [DBG] pgmap v613: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:04.639415+0000 osd.1 (osd.1) 195 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:04.642900+0000 osd.1 (osd.1) 196 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.102977+0000 osd.5 (osd.5) 189 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.106457+0000 osd.5 (osd.5) 190 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.200451+0000 osd.6 (osd.6) 179 : cluster [DBG] 16.d scrub starts 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.201310+0000 osd.6 (osd.6) 180 : cluster [DBG] 16.d scrub ok 2024-02-10T08:42:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.242332+0000 mon.a (mon.0) 2482 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:06.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:05.248792+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? v1:172.21.15.107:0/1275280930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:06.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:05.248851+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? v1:172.21.15.107:0/4231963011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:42:06.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.248884+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-10T08:42:06.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:05.441890+0000 mgr.y (mgr.14553) 374 : cluster [DBG] pgmap v613: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:42:07.115 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:07.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:07.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:07.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:07.288 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:07.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:07.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:07.474 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:07.494 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:07.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:07.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:07.609 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:42:07.609 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:42:07.609 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:42:07.609 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:42:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:07.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:07.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T08:42:07.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:07.796 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:07.815 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:07.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:07.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:08.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.125+0000 7ff98071b700 1 -- 172.21.15.107:0/305772617 >> v1:172.21.15.107:6790/0 conn(0x7ff9780f62f0 legacy=0x7ff9780f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- 172.21.15.107:0/305772617 shutdown_connections 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- 172.21.15.107:0/305772617 >> 172.21.15.107:0/305772617 conn(0x7ff97806c090 msgr2=0x7ff97806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- 172.21.15.107:0/305772617 shutdown_connections 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- 172.21.15.107:0/305772617 wait complete. 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 librados: starting msgr at 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 librados: starting objecter 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 Processor -- start 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- start start 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 librados: setting wanted keys 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 librados: calling monclient init 2024-02-10T08:42:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff97811da60 con 0x7ff978107080 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff97811ec40 con 0x7ff97810ab90 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff98071b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff97811fe40 con 0x7ff9780f62f0 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff97e4b7700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff9780f62f0 0x7ff9780f7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34688/0 (socket says 172.21.15.107:34688) 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.126+0000 7ff97e4b7700 1 -- 172.21.15.107:0/1656876198 learned_addr learned my addr 172.21.15.107:0/1656876198 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter ms_handle_connect 0x7ff978107080 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter ms_handle_connect 0x7ff9780f62f0 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter ms_handle_connect 0x7ff97810ab90 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604714710 0 0) 0x7ff97811da60 con 0x7ff978107080 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9640038e0 con 0x7ff978107080 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429364834 0 0) 0x7ff97811fe40 con 0x7ff9780f62f0 2024-02-10T08:42:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff97811da60 con 0x7ff9780f62f0 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727678999 0 0) 0x7ff97811ec40 con 0x7ff97810ab90 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff97811fe40 con 0x7ff97810ab90 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507873178 0 0) 0x7ff9640038e0 con 0x7ff978107080 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff97811ec40 con 0x7ff978107080 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff968001880 con 0x7ff978107080 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369909805 0 0) 0x7ff97811da60 con 0x7ff9780f62f0 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9640038e0 con 0x7ff9780f62f0 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9740015d0 con 0x7ff9780f62f0 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1808509954 0 0) 0x7ff97811fe40 con 0x7ff97810ab90 2024-02-10T08:42:08.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff97811da60 con 0x7ff97810ab90 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff970001d70 con 0x7ff97810ab90 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3396841327 0 0) 0x7ff97811ec40 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6790/0 conn(0x7ff9780f62f0 legacy=0x7ff9780f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.195:6789/0 conn(0x7ff97810ab90 legacy=0x7ff97811c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978121040 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff978120030 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff98071b700 10 client.28053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9781204a0 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff98071b700 1 librados: init done 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff968001f40 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.127+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff968003310 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.128+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff95c004f00 con 0x7ff978107080 2024-02-10T08:42:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff968001aa0 con 0x7ff978107080 2024-02-10T08:42:08.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(422..422 src has 330..422) v4 ==== 8222+0+0 (unknown 1729060342 0 0) 0x7ff9680764c0 con 0x7ff978107080 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 osd_map(422..422 src has 330..422) v4 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 3 client.28053.objecter handle_osd_map got epochs [422,422] > 0 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 3 client.28053.objecter handle_osd_map decoding full epoch 422 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 20 client.28053.objecter dump_active .. 0 homeless 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.129+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v422) v1 ==== 54+0+16 (unknown 2990622246 0 2420000192) 0x7ff9680788d0 con 0x7ff978107080 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff964063e20 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 10 client.28053.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 1 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 1 2024-02-10T08:42:08.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 0 1 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 4 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 10 client.28053.objecter _send_command 1 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c004950 con 0x7ff95c004530 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.130+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c004530 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.131+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c004950 con 0x7ff95c004530 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.131+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.131+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 1 = system:0 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.131+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 4 2024-02-10T08:42:08.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.131+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 0 1 2024-02-10T08:42:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 10 client.28053.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 2 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 2 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c0051a0 osd=0 3 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 0 2 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 4 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 10 client.28053.objecter _send_command 2 2024-02-10T08:42:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.148+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c004d30 con 0x7ff95c004530 2024-02-10T08:42:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c004d30 con 0x7ff95c004530 2024-02-10T08:42:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 4 2024-02-10T08:42:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 0 2 2024-02-10T08:42:08.151 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 10 client.28053.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 1 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 4 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 10 client.28053.objecter _send_command 3 2024-02-10T08:42:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c00c000 con 0x7ff95c004150 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.150+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c004150 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.151+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c00c000 con 0x7ff95c004150 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.151+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.151+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 3 = system:0 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.151+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 4 2024-02-10T08:42:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.151+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 1 3 2024-02-10T08:42:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:08.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:08.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:42:08.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 10 client.28053.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 4 2024-02-10T08:42:08.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 4 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003fe0 osd=1 3 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 1 4 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 4 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 10 client.28053.objecter _send_command 4 2024-02-10T08:42:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.167+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c0018b0 con 0x7ff95c004150 2024-02-10T08:42:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.169+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c0018b0 con 0x7ff95c004150 2024-02-10T08:42:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.169+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.169+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.169+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 4 2024-02-10T08:42:08.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.169+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 1 4 2024-02-10T08:42:08.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 10 client.28053.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 5 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 5 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 2 5 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 4 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 10 client.28053.objecter _send_command 5 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c003e40 con 0x7ff95c00d2f0 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.170+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c00d2f0 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.171+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c003e40 con 0x7ff95c00d2f0 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.171+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.171+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 5 = system:0 2024-02-10T08:42:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.171+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 4 2024-02-10T08:42:08.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.171+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 2 5 2024-02-10T08:42:08.174 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 10 client.28053.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 6 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 6 2024-02-10T08:42:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003630 osd=2 3 2024-02-10T08:42:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 2 6 2024-02-10T08:42:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 4 2024-02-10T08:42:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 10 client.28053.objecter _send_command 6 2024-02-10T08:42:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.186+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c00e720 con 0x7ff95c00d2f0 2024-02-10T08:42:08.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c00e720 con 0x7ff95c00d2f0 2024-02-10T08:42:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 4 2024-02-10T08:42:08.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 2 6 2024-02-10T08:42:08.189 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 10 client.28053.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 7 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 7 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 3 7 2024-02-10T08:42:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 4 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 10 client.28053.objecter _send_command 7 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.188+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c015a90 con 0x7ff95c011e50 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c011e50 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c015a90 con 0x7ff95c011e50 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 7 = system:0 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 4 2024-02-10T08:42:08.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.189+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 3 7 2024-02-10T08:42:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 10 client.28053.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 8 2024-02-10T08:42:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 8 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c003440 osd=3 3 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 3 8 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 4 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 10 client.28053.objecter _send_command 8 2024-02-10T08:42:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.205+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c00e190 con 0x7ff95c011e50 2024-02-10T08:42:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.206+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c00e190 con 0x7ff95c011e50 2024-02-10T08:42:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.206+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.206+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.206+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 4 2024-02-10T08:42:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.206+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 3 8 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 10 client.28053.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 9 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 9 2024-02-10T08:42:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 4 9 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 4 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 10 client.28053.objecter _send_command 9 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c016340 con 0x7ff95c00ce50 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.207+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c00ce50 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.208+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c016340 con 0x7ff95c00ce50 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.208+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.208+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 9 = system:0 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.208+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 4 2024-02-10T08:42:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.208+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 4 9 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 10 client.28053.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 10 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 10 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c00e190 osd=4 3 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 4 10 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 4 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 10 client.28053.objecter _send_command 10 2024-02-10T08:42:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.222+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c016340 con 0x7ff95c00ce50 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c016340 con 0x7ff95c00ce50 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 4 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 4 10 2024-02-10T08:42:08.225 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff98071b700 10 client.28053.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.224+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 11 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 11 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 5 11 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 4 2024-02-10T08:42:08.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 10 client.28053.objecter _send_command 11 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c0181a0 con 0x7ff95c017bc0 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.225+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c017bc0 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.226+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c0181a0 con 0x7ff95c017bc0 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.226+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.226+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 11 = system:0 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.226+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 4 2024-02-10T08:42:08.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.226+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 5 11 2024-02-10T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 10 client.28053.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 12 2024-02-10T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 12 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c016340 osd=5 3 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 5 12 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 4 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 10 client.28053.objecter _send_command 12 2024-02-10T08:42:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.240+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c0181a0 con 0x7ff95c017bc0 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.241+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c0181a0 con 0x7ff95c017bc0 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.241+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.241+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.241+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 4 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 5 12 2024-02-10T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 10 client.28053.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 13 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 13 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 6 13 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 4 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 10 client.28053.objecter _send_command 13 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c022d80 con 0x7ff95c01f090 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.242+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c01f090 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.243+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c022d80 con 0x7ff95c01f090 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.243+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.243+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 13 = system:0 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.243+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 4 2024-02-10T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.243+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 6 13 2024-02-10T08:42:08.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:08.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 10 client.28053.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 14 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 14 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c0181a0 osd=6 3 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 6 14 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 4 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 10 client.28053.objecter _send_command 14 2024-02-10T08:42:08.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.258+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c022d80 con 0x7ff95c01f090 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c022d80 con 0x7ff95c01f090 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 4 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 6 14 2024-02-10T08:42:08.260 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:08.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 10 client.28053.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:08.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 15 2024-02-10T08:42:08.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 15 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 7 15 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 4 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 10 client.28053.objecter _send_command 15 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff95c023fa0 con 0x7ff95c024a80 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.260+0000 7ff96f7fe700 10 client.28053.objecter ms_handle_connect 0x7ff95c024a80 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.261+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff95c023fa0 con 0x7ff95c024a80 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.261+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.261+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 15 = system:0 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.261+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 4 2024-02-10T08:42:08.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.261+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 7 15 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 10 client.28053.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign -1 16 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 20 client.28053.objecter _get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 15 client.28053.objecter _session_command_op_remove -1 16 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 20 client.28053.objecter get_session s=0x7ff95c022d80 osd=7 3 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 15 client.28053.objecter _session_command_op_assign 7 16 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 4 2024-02-10T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 10 client.28053.objecter _send_command 16 2024-02-10T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.275+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff95c023fa0 con 0x7ff95c024a80 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff96f7fe700 1 -- 172.21.15.107:0/1656876198 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff95c023fa0 con 0x7ff95c024a80 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff96f7fe700 10 client.28053.objecter ms_dispatch 0x7ff978001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff96f7fe700 10 client.28053.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff96f7fe700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 4 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff96f7fe700 15 client.28053.objecter _session_command_op_remove 7 16 2024-02-10T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 librados: watch_flush enter 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 librados: watch_flush exit 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.0 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6801/3321596972 conn(0x7ff95c004530 legacy=0x7ff95c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0051a0 osd=0 1 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.1 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6805/133516126 conn(0x7ff95c004150 legacy=0x7ff95c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003fe0 osd=1 1 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.2 2024-02-10T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6809/3575881728 conn(0x7ff95c00d2f0 legacy=0x7ff95c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003630 osd=2 1 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.3 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6813/2871825691 conn(0x7ff95c011e50 legacy=0x7ff95c014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c003440 osd=3 1 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.4 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.195:6800/2499671285 conn(0x7ff95c00ce50 legacy=0x7ff95c015c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c00e190 osd=4 1 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 10 client.28053.objecter close_session for osd.5 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.195:6804/4178321486 conn(0x7ff95c017bc0 legacy=0x7ff95c01d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.277+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c016340 osd=5 1 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 10 client.28053.objecter close_session for osd.6 2024-02-10T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.195:6808/1417479017 conn(0x7ff95c01f090 legacy=0x7ff95c021500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff97ecb8700 1 -- 172.21.15.107:0/1656876198 reap_dead start 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff97ecb8700 1 -- 172.21.15.107:0/1656876198 reap_dead start 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c0181a0 osd=6 1 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 10 client.28053.objecter close_session for osd.7 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.195:6812/1934176928 conn(0x7ff95c024a80 legacy=0x7ff95c024ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 20 client.28053.objecter put_session s=0x7ff95c022d80 osd=7 1 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 20 client.28053.objecter shutdown clearing up homeless session... 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 10 client.28053.objecter successfully canceled tick 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff97ecb8700 1 -- 172.21.15.107:0/1656876198 reap_dead start 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff964063e20 legacy=0x7ff9640662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> v1:172.21.15.107:6789/0 conn(0x7ff978107080 legacy=0x7ff9780f8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 shutdown_connections 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 >> 172.21.15.107:0/1656876198 conn(0x7ff97806c090 msgr2=0x7ff97806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 shutdown_connections 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 -- 172.21.15.107:0/1656876198 wait complete. 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:08.278+0000 7ff98071b700 1 librados: shutdown 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:42:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:08.423 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:06.901875+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:06.902036+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:07.252450+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:07.442225+0000 mgr.y (mgr.14553) 375 : cluster [DBG] pgmap v615: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:08.130177+0000 mon.a (mon.0) 2487 : audit [DBG] from='client.? v1:172.21.15.107:0/1656876198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:06.901875+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:06.902036+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:07.252450+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-10T08:42:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:07.442225+0000 mgr.y (mgr.14553) 375 : cluster [DBG] pgmap v615: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:08.130177+0000 mon.a (mon.0) 2487 : audit [DBG] from='client.? v1:172.21.15.107:0/1656876198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:08.529 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:08.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:06.901875+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:06.902036+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:07.252450+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-10T08:42:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:07.442225+0000 mgr.y (mgr.14553) 375 : cluster [DBG] pgmap v615: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:08.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:08.130177+0000 mon.a (mon.0) 2487 : audit [DBG] from='client.? v1:172.21.15.107:0/1656876198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:08.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:08.662 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:09.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:42:09.389 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:09.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:09.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:09.525 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:09.648 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:09.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:09.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:09.876 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:09.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:09.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:10.008 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:10.028 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:10.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:10.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:10.145 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:10.145 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:10.162 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:10.182 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:10.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:10.250 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:10.271 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:10.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:10.311 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:09.259905+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:09.442595+0000 mgr.y (mgr.14553) 376 : cluster [DBG] pgmap v617: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:10.156688+0000 mon.a (mon.0) 2489 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:09.259905+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:09.442595+0000 mgr.y (mgr.14553) 376 : cluster [DBG] pgmap v617: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:10.156688+0000 mon.a (mon.0) 2489 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:10.508 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:10.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:10.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:09.259905+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-10T08:42:10.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:09.442595+0000 mgr.y (mgr.14553) 376 : cluster [DBG] pgmap v617: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:10.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:10.156688+0000 mon.a (mon.0) 2489 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:10.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:10.640 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:10.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:42:10.838 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:10.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:10.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:10.968 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:10.987 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:10.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:11.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:11.110 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:11.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:11.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:11.280 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:11.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:42:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 -- 172.21.15.107:0/1447106142 >> v1:172.21.15.107:6790/0 conn(0x7f49dc0fda50 legacy=0x7f49dc0fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 -- 172.21.15.107:0/1447106142 shutdown_connections 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 -- 172.21.15.107:0/1447106142 >> 172.21.15.107:0/1447106142 conn(0x7f49dc0f3640 msgr2=0x7f49dc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 -- 172.21.15.107:0/1447106142 shutdown_connections 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 -- 172.21.15.107:0/1447106142 wait complete. 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 librados: starting msgr at 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.428+0000 7f49e25ea700 1 librados: starting objecter 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 Processor -- start 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 -- start start 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 librados: setting wanted keys 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 librados: calling monclient init 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49dc117570 con 0x7f49dc1007f0 2024-02-10T08:42:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49dc118750 con 0x7f49dc104330 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49e25ea700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49dc119950 con 0x7f49dc0fda50 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49db7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f49dc1007f0 0x7f49dc0fa4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49356/0 (socket says 172.21.15.107:49356) 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49db7fe700 1 -- 172.21.15.107:0/2882442056 learned_addr learned my addr 172.21.15.107:0/2882442056 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter ms_handle_connect 0x7f49dc0fda50 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter ms_handle_connect 0x7f49dc1007f0 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter ms_handle_connect 0x7f49dc104330 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.429+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134059174 0 0) 0x7f49dc117570 con 0x7f49dc1007f0 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c4003630 con 0x7f49dc1007f0 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038283886 0 0) 0x7f49dc119950 con 0x7f49dc0fda50 2024-02-10T08:42:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc117570 con 0x7f49dc0fda50 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443047970 0 0) 0x7f49dc118750 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc119950 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136284207 0 0) 0x7f49dc119950 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49dc118750 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1590313250 0 0) 0x7f49dc117570 con 0x7f49dc0fda50 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49dc119950 con 0x7f49dc0fda50 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753252721 0 0) 0x7f49c4003630 con 0x7f49dc1007f0 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49dc117570 con 0x7f49dc1007f0 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f49d4001b80 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f49cc001ec0 con 0x7f49dc0fda50 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f49d0001840 con 0x7f49dc1007f0 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3090102200 0 0) 0x7f49dc118750 con 0x7f49dc104330 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 >> v1:172.21.15.107:6790/0 conn(0x7f49dc0fda50 legacy=0x7f49dc0f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 >> v1:172.21.15.107:6789/0 conn(0x7f49dc1007f0 legacy=0x7f49dc0fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc11ab50 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc117760 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49e25ea700 10 client.48358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc117bd0 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49e25ea700 1 librados: init done 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f49d4002240 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.430+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f49d4003620 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.431+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c8004f20 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.431+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f49d4016530 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(424..424 src has 330..424) v4 ==== 8250+0+0 (unknown 43797043 0 0) 0x7f49d4076970 con 0x7f49dc104330 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 10 client.48358.objecter ms_dispatch 0x7f49dc001550 osd_map(424..424 src has 330..424) v4 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 3 client.48358.objecter handle_osd_map got epochs [424,424] > 0 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 3 client.48358.objecter handle_osd_map decoding full epoch 424 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 20 client.48358.objecter dump_active .. 0 homeless 2024-02-10T08:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.432+0000 7f49d97fa700 10 client.48358.objecter ms_handle_connect 0x7f49c4063ca0 2024-02-10T08:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.437+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f49d4049ca0 con 0x7f49dc104330 2024-02-10T08:42:11.481 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:11.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:11.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:11.613 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:11.631+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f49c80051d0 con 0x7f49dc104330 2024-02-10T08:42:11.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:12.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.038+0000 7f49d97fa700 1 -- 172.21.15.107:0/2882442056 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v425) v1 ==== 113+0+0 (unknown 726497153 0 0) 0x7f49d4048110 con 0x7f49dc104330 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 10 librados: watch_flush enter 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 10 librados: watch_flush exit 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 20 client.48358.objecter shutdown clearing up homeless session... 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 10 client.48358.objecter successfully canceled tick 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 >> v1:172.21.15.107:6800/2634842009 conn(0x7f49c4063ca0 legacy=0x7f49c4066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 >> v1:172.21.15.195:6789/0 conn(0x7f49dc104330 legacy=0x7f49dc115c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 shutdown_connections 2024-02-10T08:42:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.043+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 >> 172.21.15.107:0/2882442056 conn(0x7f49dc0f3640 msgr2=0x7f49dc1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.044+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 shutdown_connections 2024-02-10T08:42:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.044+0000 7f49e25ea700 1 -- 172.21.15.107:0/2882442056 wait complete. 2024-02-10T08:42:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:12.044+0000 7f49e25ea700 1 librados: shutdown 2024-02-10T08:42:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:10.661368+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:10.661537+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:11.026074+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:11.443409+0000 mgr.y (mgr.14553) 377 : cluster [DBG] pgmap v619: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:11.469398+0000 mgr.y (mgr.14553) 378 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:11.632458+0000 mon.b (mon.1) 582 : audit [INF] from='client.? v1:172.21.15.107:0/2882442056' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.210 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:11.632788+0000 mon.a (mon.0) 2491 : audit [INF] from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:10.661368+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:10.661537+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:11.026074+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:11.443409+0000 mgr.y (mgr.14553) 377 : cluster [DBG] pgmap v619: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:11.469398+0000 mgr.y (mgr.14553) 378 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:11.632458+0000 mon.b (mon.1) 582 : audit [INF] from='client.? v1:172.21.15.107:0/2882442056' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:11.632788+0000 mon.a (mon.0) 2491 : audit [INF] from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:10.661368+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:10.661537+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:11.026074+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-10T08:42:12.293 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:11.443409+0000 mgr.y (mgr.14553) 377 : cluster [DBG] pgmap v619: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:42:12.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:11.469398+0000 mgr.y (mgr.14553) 378 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:12.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:11.632458+0000 mon.b (mon.1) 582 : audit [INF] from='client.? v1:172.21.15.107:0/2882442056' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:11.632788+0000 mon.a (mon.0) 2491 : audit [INF] from='client.48358 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:42:12.707 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:12.707 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:12.960 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T08:42:12.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:42:12.964 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:12.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:13.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:13.096 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:13.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.183+0000 7ff63cdcb700 1 -- 172.21.15.107:0/4214421168 >> v1:172.21.15.107:6790/0 conn(0x7ff6381007f0 legacy=0x7ff638102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.183+0000 7ff63cdcb700 1 -- 172.21.15.107:0/4214421168 shutdown_connections 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.183+0000 7ff63cdcb700 1 -- 172.21.15.107:0/4214421168 >> 172.21.15.107:0/4214421168 conn(0x7ff6380f3640 msgr2=0x7ff6380f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.183+0000 7ff63cdcb700 1 -- 172.21.15.107:0/4214421168 shutdown_connections 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.183+0000 7ff63cdcb700 1 -- 172.21.15.107:0/4214421168 wait complete. 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 librados: starting msgr at 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 librados: starting objecter 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 Processor -- start 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 -- start start 2024-02-10T08:42:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 librados: setting wanted keys 2024-02-10T08:42:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.184+0000 7ff63cdcb700 1 librados: calling monclient init 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff63cdcb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff638117330 con 0x7ff6381007f0 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff63cdcb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff638118510 con 0x7ff638104330 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff63cdcb700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff638119710 con 0x7ff6380fda50 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff635d9b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff6381007f0 0x7ff6380fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49376/0 (socket says 172.21.15.107:49376) 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff635d9b700 1 -- 172.21.15.107:0/2674459581 learned_addr learned my addr 172.21.15.107:0/2674459581 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter ms_handle_connect 0x7ff6381007f0 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter ms_handle_connect 0x7ff6380fda50 2024-02-10T08:42:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter ms_handle_connect 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105956981 0 0) 0x7ff638117330 con 0x7ff6381007f0 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6200033f0 con 0x7ff6381007f0 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780241526 0 0) 0x7ff638119710 con 0x7ff6380fda50 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638117330 con 0x7ff6380fda50 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119042849 0 0) 0x7ff638118510 con 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638119710 con 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 627137290 0 0) 0x7ff638119710 con 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff638118510 con 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff630001c30 con 0x7ff638104330 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457850453 0 0) 0x7ff638117330 con 0x7ff6380fda50 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff638119710 con 0x7ff6380fda50 2024-02-10T08:42:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106299926 0 0) 0x7ff6200033f0 con 0x7ff6381007f0 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff638117330 con 0x7ff6381007f0 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff628001840 con 0x7ff6380fda50 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff62c002050 con 0x7ff6381007f0 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2451465608 0 0) 0x7ff638118510 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 >> v1:172.21.15.107:6790/0 conn(0x7ff6380fda50 legacy=0x7ff6380f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 >> v1:172.21.15.107:6789/0 conn(0x7ff6381007f0 legacy=0x7ff6380fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63811a910 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.185+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff638119900 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff63cdcb700 10 client.48364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff638119d70 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff63cdcb700 1 librados: init done 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff6300024e0 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff630003540 con 0x7ff638104330 2024-02-10T08:42:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.186+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff618004f20 con 0x7ff638104330 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff6300036a0 con 0x7ff638104330 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(425..425 src has 330..425) v4 ==== 8222+0+0 (unknown 2399519789 0 0) 0x7ff630076660 con 0x7ff638104330 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 10 client.48364.objecter ms_dispatch 0x7ff638001550 osd_map(425..425 src has 330..425) v4 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 3 client.48364.objecter handle_osd_map got epochs [425,425] > 0 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 3 client.48364.objecter handle_osd_map decoding full epoch 425 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.187+0000 7ff6277fe700 20 client.48364.objecter dump_active .. 0 homeless 2024-02-10T08:42:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.188+0000 7ff6277fe700 10 client.48364.objecter ms_handle_connect 0x7ff620063ec0 2024-02-10T08:42:13.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.191+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff6300499b0 con 0x7ff638104330 2024-02-10T08:42:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:12.038296+0000 mon.a (mon.0) 2492 : audit [INF] from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:42:13.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:12.038367+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-10T08:42:13.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:42:13.345 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:13.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:13.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:13.389+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff6180051d0 con 0x7ff638104330 2024-02-10T08:42:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:12.038296+0000 mon.a (mon.0) 2492 : audit [INF] from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:42:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:12.038367+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-10T08:42:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:12.038296+0000 mon.a (mon.0) 2492 : audit [INF] from='client.48358 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:42:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:12.038367+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-10T08:42:13.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:13.477 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:13.495 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:13.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:13.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:13.613 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:13.628 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.050+0000 7ff6277fe700 1 -- 172.21.15.107:0/2674459581 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v426) v1 ==== 97+0+0 (unknown 778904088 0 0) 0x7ff630047dc0 con 0x7ff638104330 2024-02-10T08:42:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.052+0000 7ff63cdcb700 10 librados: watch_flush enter 2024-02-10T08:42:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.052+0000 7ff63cdcb700 10 librados: watch_flush exit 2024-02-10T08:42:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.052+0000 7ff63cdcb700 20 client.48364.objecter shutdown clearing up homeless session... 2024-02-10T08:42:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.052+0000 7ff63cdcb700 10 client.48364.objecter successfully canceled tick 2024-02-10T08:42:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff620063ec0 legacy=0x7ff620066380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 >> v1:172.21.15.195:6789/0 conn(0x7ff638104330 legacy=0x7ff6381159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 shutdown_connections 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 >> 172.21.15.107:0/2674459581 conn(0x7ff6380f3640 msgr2=0x7ff6381077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 shutdown_connections 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 -- 172.21.15.107:0/2674459581 wait complete. 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:14.053+0000 7ff63cdcb700 1 librados: shutdown 2024-02-10T08:42:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:42:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:13.390710+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.107:0/2674459581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:13.391042+0000 mon.a (mon.0) 2494 : audit [INF] from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:13.444011+0000 mgr.y (mgr.14553) 379 : cluster [DBG] pgmap v621: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:14.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:13.390710+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.107:0/2674459581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:13.391042+0000 mon.a (mon.0) 2494 : audit [INF] from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:13.444011+0000 mgr.y (mgr.14553) 379 : cluster [DBG] pgmap v621: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:14.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:13.390710+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.107:0/2674459581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:13.391042+0000 mon.a (mon.0) 2494 : audit [INF] from='client.48364 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:13.444011+0000 mgr.y (mgr.14553) 379 : cluster [DBG] pgmap v621: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:15.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:14.037831+0000 mon.a (mon.0) 2495 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:15.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:14.049549+0000 mon.a (mon.0) 2496 : audit [INF] from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:15.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:14.049604+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-10T08:42:15.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:14.037831+0000 mon.a (mon.0) 2495 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:15.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:14.049549+0000 mon.a (mon.0) 2496 : audit [INF] from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:15.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:14.049604+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-10T08:42:15.393 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:14.037831+0000 mon.a (mon.0) 2495 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:15.394 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:14.049549+0000 mon.a (mon.0) 2496 : audit [INF] from='client.48364 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:15.394 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:14.049604+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-10T08:42:15.897 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:15] "GET /metrics HTTP/1.1" 200 187544 "" "Prometheus/2.33.4" 2024-02-10T08:42:15.905 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:15.905 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:15.907 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:15.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:16.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:16.043 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:16.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:15.444380+0000 mgr.y (mgr.14553) 380 : cluster [DBG] pgmap v623: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:16.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:16.031460+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-10T08:42:16.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:15.444380+0000 mgr.y (mgr.14553) 380 : cluster [DBG] pgmap v623: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:16.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:16.031460+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-10T08:42:16.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:15.444380+0000 mgr.y (mgr.14553) 380 : cluster [DBG] pgmap v623: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:42:16.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:16.031460+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-10T08:42:16.244 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:16.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:16.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:16.475 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:16.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:16.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:42:16.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:16.606 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:16.624 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:16.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:16.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:16.755 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:16.776 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:16.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:16.805 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:16.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:16.906 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:17.029 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:17.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:17.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:17.162 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:17.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:17.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:17.424 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:17.522 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:17.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:17.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:17.654 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:17.729 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:17.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:17.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:17.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.840+0000 7efe95a88700 1 -- 172.21.15.107:0/3820347758 >> v1:172.21.15.107:6789/0 conn(0x7efe900fc950 legacy=0x7efe900fcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:17.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.840+0000 7efe95a88700 1 -- 172.21.15.107:0/3820347758 shutdown_connections 2024-02-10T08:42:17.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.840+0000 7efe95a88700 1 -- 172.21.15.107:0/3820347758 >> 172.21.15.107:0/3820347758 conn(0x7efe9006fe70 msgr2=0x7efe900f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:17.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.840+0000 7efe95a88700 1 -- 172.21.15.107:0/3820347758 shutdown_connections 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.840+0000 7efe95a88700 1 -- 172.21.15.107:0/3820347758 wait complete. 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 librados: starting msgr at 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 librados: starting objecter 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 Processor -- start 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 -- start start 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 librados: setting wanted keys 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 librados: calling monclient init 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe90116240 con 0x7efe900fc950 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe90117420 con 0x7efe901032b0 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe95a88700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe90118620 con 0x7efe900ff740 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe8f7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efe900fc950 0x7efe900f89b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49416/0 (socket says 172.21.15.107:49416) 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe8f7fe700 1 -- 172.21.15.107:0/2693081390 learned_addr learned my addr 172.21.15.107:0/2693081390 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:17.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.841+0000 7efe8effd700 1 --1- 172.21.15.107:0/2693081390 >> v1:172.21.15.107:6790/0 conn(0x7efe900ff740 0x7efe900f90c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50584/0 (socket says 172.21.15.107:50584) 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter ms_handle_connect 0x7efe900fc950 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter ms_handle_connect 0x7efe900ff740 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter ms_handle_connect 0x7efe901032b0 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693302653 0 0) 0x7efe90116240 con 0x7efe900fc950 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe78003030 con 0x7efe900fc950 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73282266 0 0) 0x7efe90118620 con 0x7efe900ff740 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe90116240 con 0x7efe900ff740 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642270666 0 0) 0x7efe78003030 con 0x7efe900fc950 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe90118620 con 0x7efe900fc950 2024-02-10T08:42:17.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 991049811 0 0) 0x7efe90116240 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe78003030 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efe84001a60 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3358905704 0 0) 0x7efe78003030 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 >> v1:172.21.15.195:6789/0 conn(0x7efe901032b0 legacy=0x7efe90114950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 >> v1:172.21.15.107:6789/0 conn(0x7efe900fc950 legacy=0x7efe900f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe90119820 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe90117610 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe95a88700 10 client.53660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe90117ac0 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.842+0000 7efe95a88700 1 librados: init done 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.843+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7efe840024a0 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.843+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efe84005a70 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.844+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c004f50 con 0x7efe900ff740 2024-02-10T08:42:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.844+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7efe84018900 con 0x7efe900ff740 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(427..427 src has 330..427) v4 ==== 8222+0+0 (unknown 43291037 0 0) 0x7efe84078ca0 con 0x7efe900ff740 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 10 client.53660.objecter ms_dispatch 0x7efe90001550 osd_map(427..427 src has 330..427) v4 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 3 client.53660.objecter handle_osd_map got epochs [427,427] > 0 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 3 client.53660.objecter handle_osd_map decoding full epoch 427 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 20 client.53660.objecter dump_active .. 0 homeless 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.845+0000 7efe8cf74700 10 client.53660.objecter ms_handle_connect 0x7efe78065d50 2024-02-10T08:42:17.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.850+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7efe84044f60 con 0x7efe900ff740 2024-02-10T08:42:17.863 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:17.892 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:17.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:17.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:18.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.758+0000 7ff1c97b7700 1 -- 172.21.15.107:0/3894351897 >> v1:172.21.15.107:6790/0 conn(0x7ff1c4104350 legacy=0x7ff1c41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.758+0000 7ff1c97b7700 1 -- 172.21.15.107:0/3894351897 shutdown_connections 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.758+0000 7ff1c97b7700 1 -- 172.21.15.107:0/3894351897 >> 172.21.15.107:0/3894351897 conn(0x7ff1c40f2510 msgr2=0x7ff1c40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.758+0000 7ff1c97b7700 1 -- 172.21.15.107:0/3894351897 shutdown_connections 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.758+0000 7ff1c97b7700 1 -- 172.21.15.107:0/3894351897 wait complete. 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 librados: starting msgr at 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 librados: starting objecter 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 Processor -- start 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 -- start start 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 librados: setting wanted keys 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 librados: calling monclient init 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1c4102bd0 con 0x7ff1c40f8a20 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1c4102da0 con 0x7ff1c40f95d0 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c97b7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff1c4102f90 con 0x7ff1c4104350 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c37fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff1c4104350 0x7ff1c41184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50554/0 (socket says 172.21.15.107:50554) 2024-02-10T08:42:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 learned_addr learned my addr 172.21.15.107:0/1169222440 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 10 client.?.objecter ms_handle_connect 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 10 client.?.objecter ms_handle_connect 0x7ff1c40f95d0 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405268124 0 0) 0x7ff1c4102f90 con 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ac0033f0 con 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.759+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219090432 0 0) 0x7ff1c4102da0 con 0x7ff1c40f95d0 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4102f90 con 0x7ff1c40f95d0 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 776791628 0 0) 0x7ff1ac0033f0 con 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c4102da0 con 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1bc001800 con 0x7ff1c4104350 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 10 client.?.objecter ms_handle_connect 0x7ff1c40f8a20 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327768529 0 0) 0x7ff1c4102f90 con 0x7ff1c40f95d0 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1ac0033f0 con 0x7ff1c40f95d0 2024-02-10T08:42:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1b80019d0 con 0x7ff1c40f95d0 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206576495 0 0) 0x7ff1c4102bd0 con 0x7ff1c40f8a20 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4102f90 con 0x7ff1c40f8a20 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2414168627 0 0) 0x7ff1c4102da0 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.195:6789/0 conn(0x7ff1c40f95d0 legacy=0x7ff1c4114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6789/0 conn(0x7ff1c40f8a20 legacy=0x7ff1c4102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c411cc20 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4119c50 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1c97b7700 10 client.53654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c411a0c0 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1c97b7700 1 librados: init done 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff1bc0030f0 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.760+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1bc003c50 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.761+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff1b0004f00 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.761+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff1bc0034b0 con 0x7ff1c4104350 2024-02-10T08:42:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(427..427 src has 330..427) v4 ==== 8222+0+0 (unknown 43291037 0 0) 0x7ff1bc079490 con 0x7ff1c4104350 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 osd_map(427..427 src has 330..427) v4 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 3 client.53654.objecter handle_osd_map got epochs [427,427] > 0 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 3 client.53654.objecter handle_osd_map decoding full epoch 427 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 20 client.53654.objecter dump_active .. 0 homeless 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v427) v1 ==== 54+0+16 (unknown 3296173923 0 2420000192) 0x7ff1bc07b850 con 0x7ff1c4104350 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1ac063c30 2024-02-10T08:42:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 1 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 1 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 0 1 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 4 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 10 client.53654.objecter _send_command 1 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.762+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b0004950 con 0x7ff1b0004530 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b0004530 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b0004950 con 0x7ff1b0004530 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 10 client.53654.objecter _finish_command 1 = system:0 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 4 2024-02-10T08:42:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.763+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 0 1 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 2 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 2 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b00051a0 osd=0 3 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 0 2 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 4 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 10 client.53654.objecter _send_command 2 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.778+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b0004d30 con 0x7ff1b0004530 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.779+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b0004d30 con 0x7ff1b0004530 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1abfff700 10 client.53654.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 4 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 0 2 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 1 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 4 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 10 client.53654.objecter _send_command 3 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b000c000 con 0x7ff1b0004150 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.780+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b0004150 2024-02-10T08:42:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.781+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b000c000 con 0x7ff1b0004150 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.781+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.781+0000 7ff1abfff700 10 client.53654.objecter _finish_command 3 = system:0 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.781+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.781+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 1 3 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0003fe0 osd=1 3 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 1 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 10 client.53654.objecter _send_command 4 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.795+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b00018b0 con 0x7ff1b0004150 2024-02-10T08:42:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b00018b0 con 0x7ff1b0004150 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1abfff700 10 client.53654.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 4 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 1 4 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 5 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 5 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 2 5 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 4 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 10 client.53654.objecter _send_command 5 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.797+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b000de90 con 0x7ff1b000d2f0 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b000d2f0 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b000de90 con 0x7ff1b000d2f0 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 10 client.53654.objecter _finish_command 5 = system:0 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 4 2024-02-10T08:42:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.798+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 2 5 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.812+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 6 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 6 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0003630 osd=2 3 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 2 6 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 4 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 10 client.53654.objecter _send_command 6 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.813+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b00030a0 con 0x7ff1b000d2f0 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b00030a0 con 0x7ff1b000d2f0 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1abfff700 10 client.53654.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 4 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 2 6 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 7 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 7 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 3 7 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 4 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 10 client.53654.objecter _send_command 7 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.814+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b0015b00 con 0x7ff1b0011ff0 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b0011ff0 2024-02-10T08:42:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b0015b00 con 0x7ff1b0011ff0 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 10 client.53654.objecter _finish_command 7 = system:0 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 4 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.815+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 3 7 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 8 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 8 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0011970 osd=3 3 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 3 8 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 4 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 10 client.53654.objecter _send_command 8 2024-02-10T08:42:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.830+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b000e240 con 0x7ff1b0011ff0 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.832+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b000e240 con 0x7ff1b0011ff0 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1abfff700 10 client.53654.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 4 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 3 8 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 9 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:18.022 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 9 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 4 9 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 4 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 10 client.53654.objecter _send_command 9 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b000e1b0 con 0x7ff1b000ce50 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.833+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b000ce50 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.834+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b000e1b0 con 0x7ff1b000ce50 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.834+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.834+0000 7ff1abfff700 10 client.53654.objecter _finish_command 9 = system:0 2024-02-10T08:42:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.834+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 4 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.834+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 4 9 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 10 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 10 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b000e000 osd=4 3 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 4 10 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 4 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 10 client.53654.objecter _send_command 10 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.852+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b000e1b0 con 0x7ff1b000ce50 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b000e1b0 con 0x7ff1b000ce50 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1abfff700 10 client.53654.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 4 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 4 10 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 11 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 11 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 5 11 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 4 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 10 client.53654.objecter _send_command 11 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.854+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b0018090 con 0x7ff1b0017aa0 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b0017aa0 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b0018090 con 0x7ff1b0017aa0 2024-02-10T08:42:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 10 client.53654.objecter _finish_command 11 = system:0 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 4 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.855+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 5 11 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 12 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 12 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b000e1b0 osd=5 3 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 5 12 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 4 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 10 client.53654.objecter _send_command 12 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.871+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b0018090 con 0x7ff1b0017aa0 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b0018090 con 0x7ff1b0017aa0 2024-02-10T08:42:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 10 client.53654.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 4 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 5 12 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 13 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 13 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 6 13 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 4 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 10 client.53654.objecter _send_command 13 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b0022c30 con 0x7ff1b001f080 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.873+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b001f080 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.874+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b0022c30 con 0x7ff1b001f080 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.874+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.874+0000 7ff1abfff700 10 client.53654.objecter _finish_command 13 = system:0 2024-02-10T08:42:18.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.874+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 4 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.874+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 6 13 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 14 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 14 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b0018090 osd=6 3 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 6 14 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 4 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 10 client.53654.objecter _send_command 14 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.894+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b00243e0 con 0x7ff1b001f080 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b00243e0 con 0x7ff1b001f080 2024-02-10T08:42:18.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1abfff700 10 client.53654.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 4 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 6 14 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 15 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.895+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 15 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 7 15 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 4 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 10 client.53654.objecter _send_command 15 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b0024fe0 con 0x7ff1b0024a60 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.896+0000 7ff1abfff700 10 client.53654.objecter ms_handle_connect 0x7ff1b0024a60 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.897+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff1b0024fe0 con 0x7ff1b0024a60 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.897+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.897+0000 7ff1abfff700 10 client.53654.objecter _finish_command 15 = system:0 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.897+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 4 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.897+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 7 15 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 10 client.53654.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign -1 16 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 20 client.53654.objecter _get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_remove -1 16 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 20 client.53654.objecter get_session s=0x7ff1b00243e0 osd=7 3 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 15 client.53654.objecter _session_command_op_assign 7 16 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 4 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 10 client.53654.objecter _send_command 16 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.911+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1b0024fe0 con 0x7ff1b0024a60 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1abfff700 1 -- 172.21.15.107:0/1169222440 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff1b0024fe0 con 0x7ff1b0024a60 2024-02-10T08:42:18.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1abfff700 10 client.53654.objecter ms_dispatch 0x7ff1c4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1abfff700 10 client.53654.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1abfff700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 4 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1abfff700 15 client.53654.objecter _session_command_op_remove 7 16 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 librados: watch_flush enter 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 librados: watch_flush exit 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.0 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6801/3321596972 conn(0x7ff1b0004530 legacy=0x7ff1b0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00051a0 osd=0 1 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.1 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6805/133516126 conn(0x7ff1b0004150 legacy=0x7ff1b000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003fe0 osd=1 1 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.2 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6809/3575881728 conn(0x7ff1b000d2f0 legacy=0x7ff1b000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0003630 osd=2 1 2024-02-10T08:42:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.3 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6813/2871825691 conn(0x7ff1b0011ff0 legacy=0x7ff1b00143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0011970 osd=3 1 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.4 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.195:6800/2499671285 conn(0x7ff1b000ce50 legacy=0x7ff1b0016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e000 osd=4 1 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.5 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.195:6804/4178321486 conn(0x7ff1b0017aa0 legacy=0x7ff1b001d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b000e1b0 osd=5 1 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.6 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.195:6808/1417479017 conn(0x7ff1b001f080 legacy=0x7ff1b0021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b0018090 osd=6 1 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter close_session for osd.7 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.195:6812/1934176928 conn(0x7ff1b0024a60 legacy=0x7ff1b0022c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter put_session s=0x7ff1b00243e0 osd=7 1 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 20 client.53654.objecter shutdown clearing up homeless session... 2024-02-10T08:42:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c97b7700 10 client.53654.objecter successfully canceled tick 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.913+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 reap_dead start 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 reap_dead start 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 reap_dead start 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 reap_dead start 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c37fe700 1 -- 172.21.15.107:0/1169222440 reap_dead start 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff1ac063c30 legacy=0x7ff1ac0660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> v1:172.21.15.107:6790/0 conn(0x7ff1c4104350 legacy=0x7ff1c41184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 shutdown_connections 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 >> 172.21.15.107:0/1169222440 conn(0x7ff1c40f2510 msgr2=0x7ff1c41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 shutdown_connections 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 -- 172.21.15.107:0/1169222440 wait complete. 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:17.914+0000 7ff1c97b7700 1 librados: shutdown 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:42:18.037 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:18.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.038+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe7c000bc0 con 0x7efe78065d50 2024-02-10T08:42:18.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.053+0000 7efe8cf74700 1 -- 172.21.15.107:0/2693081390 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+554386 (unknown 2965378022 0 3319037090) 0x7efe7c000bc0 con 0x7efe78065d50 2024-02-10T08:42:18.062 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 10 librados: watch_flush enter 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 10 librados: watch_flush exit 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 20 client.53660.objecter shutdown clearing up homeless session... 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 10 client.53660.objecter successfully canceled tick 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 >> v1:172.21.15.107:6800/2634842009 conn(0x7efe78065d50 legacy=0x7efe78068210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 >> v1:172.21.15.107:6790/0 conn(0x7efe900ff740 legacy=0x7efe900f90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 shutdown_connections 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 >> 172.21.15.107:0/2693081390 conn(0x7efe9006fe70 msgr2=0x7efe900f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 shutdown_connections 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 -- 172.21.15.107:0/2693081390 wait complete. 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:18.057+0000 7efe95a88700 1 librados: shutdown 2024-02-10T08:42:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:42:18.399 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:42:18.399 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:42:18.400 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:42:18.400 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:42:18.400 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:42:18.674 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:17.445455+0000 mgr.y (mgr.14553) 381 : cluster [DBG] pgmap v625: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:18.674 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:17.763719+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/1169222440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:18.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:17.445455+0000 mgr.y (mgr.14553) 381 : cluster [DBG] pgmap v625: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:18.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:17.763719+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/1169222440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:18.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:17.445455+0000 mgr.y (mgr.14553) 381 : cluster [DBG] pgmap v625: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:42:18.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:17.763719+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.107:0/1169222440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:18.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:42:18.973 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2024-02-10T08:42:17.444595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007292,"num_objects":354,"num_object_clones":2,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":1376,"num_read_kb":1026,"num_write":1043,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2384,"ondisk_log_size":2384,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":654,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2142764,"kb_used_data":29428,"kb_used_omap":0,"kb_used_meta":2113280,"kb_avail":654073300,"statfs":{"total":671965249536,"available":669771059200,"internally_reserved":0,"allocated":30134272,"data_stored":18826100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2163998720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001867"},"pg_stats":[{"pgid":"212.1f","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043895+0000","last_change":"2024-02-10T08:41:44.138746+0000","last_active":"2024-02-10T08:42:14.043895+0000","last_peered":"2024-02-10T08:42:14.043895+0000","last_clean":"2024-02-10T08:42:14.043895+0000","last_became_active":"2024-02-10T08:41:44.138031+0000","last_became_peered":"2024-02-10T08:41:44.138031+0000","last_unstale":"2024-02-10T08:42:14.043895+0000","last_undegraded":"2024-02-10T08:42:14.043895+0000","last_fullsized":"2024-02-10T08:42:14.043895+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.1e","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367575+0000","last_change":"2024-02-10T08:41:44.138094+0000","last_active":"2024-02-10T08:42:12.367575+0000","last_peered":"2024-02-10T08:42:12.367575+0000","last_clean":"2024-02-10T08:42:12.367575+0000","last_became_active":"2024-02-10T08:41:44.137527+0000","last_became_peered":"2024-02-10T08:41:44.137527+0000","last_unstale":"2024-02-10T08:42:12.367575+0000","last_undegraded":"2024-02-10T08:42:12.367575+0000","last_fullsized":"2024-02-10T08:42:12.367575+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1d","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050265+0000","last_change":"2024-02-10T08:41:44.124316+0000","last_active":"2024-02-10T08:42:14.050265+0000","last_peered":"2024-02-10T08:42:14.050265+0000","last_clean":"2024-02-10T08:42:14.050265+0000","last_became_active":"2024-02-10T08:41:44.124029+0000","last_became_peered":"2024-02-10T08:41:44.124029+0000","last_unstale":"2024-02-10T08:42:14.050265+0000","last_undegraded":"2024-02-10T08:42:14.050265+0000","last_fullsized":"2024-02-10T08:42:14.050265+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.1c","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397498+0000","last_change":"2024-02-10T08:41:44.134803+0000","last_active":"2024-02-10T08:42:12.397498+0000","last_peered":"2024-02-10T08:42:12.397498+0000","last_clean":"2024-02-10T08:42:12.397498+0000","last_became_active":"2024-02-10T08:41:44.134378+0000","last_became_peered":"2024-02-10T08:41:44.134378+0000","last_unstale":"2024-02-10T08:42:12.397498+0000","last_undegraded":"2024-02-10T08:42:12.397498+0000","last_fullsized":"2024-02-10T08:42:12.397498+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1b","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398354+0000","last_change":"2024-02-10T08:41:44.137857+0000","last_active":"2024-02-10T08:42:12.398354+0000","last_peered":"2024-02-10T08:42:12.398354+0000","last_clean":"2024-02-10T08:42:12.398354+0000","last_became_active":"2024-02-10T08:41:44.137554+0000","last_became_peered":"2024-02-10T08:41:44.137554+0000","last_unstale":"2024-02-10T08:42:12.398354+0000","last_undegraded":"2024-02-10T08:42:12.398354+0000","last_fullsized":"2024-02-10T08:42:12.398354+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1a","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397934+0000","last_change":"2024-02-10T08:41:44.135527+0000","last_active":"2024-02-10T08:42:12.397934+0000","last_peered":"2024-02-10T08:42:12.397934+0000","last_clean":"2024-02-10T08:42:12.397934+0000","last_became_active":"2024-02-10T08:41:44.135015+0000","last_became_peered":"2024-02-10T08:41:44.135015+0000","last_unstale":"2024-02-10T08:42:12.397934+0000","last_undegraded":"2024-02-10T08:42:12.397934+0000","last_fullsized":"2024-02-10T08:42:12.397934+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.19","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398751+0000","last_change":"2024-02-10T08:41:44.134842+0000","last_active":"2024-02-10T08:42:12.398751+0000","last_peered":"2024-02-10T08:42:12.398751+0000","last_clean":"2024-02-10T08:42:12.398751+0000","last_became_active":"2024-02-10T08:41:44.134526+0000","last_became_peered":"2024-02-10T08:41:44.134526+0000","last_unstale":"2024-02-10T08:42:12.398751+0000","last_undegraded":"2024-02-10T08:42:12.398751+0000","last_fullsized":"2024-02-10T08:42:12.398751+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.18","version":"409'1","reported_seq":29,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050137+0000","last_change":"2024-02-10T08:41:44.136731+0000","last_active":"2024-02-10T08:42:14.050137+0000","last_peered":"2024-02-10T08:42:14.050137+0000","last_clean":"2024-02-10T08:42:14.050137+0000","last_became_active":"2024-02-10T08:41:44.136236+0000","last_became_peered":"2024-02-10T08:41:44.136236+0000","last_unstale":"2024-02-10T08:42:14.050137+0000","last_undegraded":"2024-02-10T08:42:14.050137+0000","last_fullsized":"2024-02-10T08:42:14.050137+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.17","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049750+0000","last_change":"2024-02-10T08:41:44.136193+0000","last_active":"2024-02-10T08:42:14.049750+0000","last_peered":"2024-02-10T08:42:14.049750+0000","last_clean":"2024-02-10T08:42:14.049750+0000","last_became_active":"2024-02-10T08:41:44.135650+0000","last_became_peered":"2024-02-10T08:41:44.135650+0000","last_unstale":"2024-02-10T08:42:14.049750+0000","last_undegraded":"2024-02-10T08:42:14.049750+0000","last_fullsized":"2024-02-10T08:42:14.049750+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.16","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400012+0000","last_change":"2024-02-10T08:41:44.134155+0000","last_active":"2024-02-10T08:42:12.400012+0000","last_peered":"2024-02-10T08:42:12.400012+0000","last_clean":"2024-02-10T08:42:12.400012+0000","last_became_active":"2024-02-10T08:41:44.133721+0000","last_became_peered":"2024-02-10T08:41:44.133721+0000","last_unstale":"2024-02-10T08:42:12.400012+0000","last_undegraded":"2024-02-10T08:42:12.400012+0000","last_fullsized":"2024-02-10T08:42:12.400012+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.15","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399526+0000","last_change":"2024-02-10T08:41:44.137825+0000","last_active":"2024-02-10T08:42:12.399526+0000","last_peered":"2024-02-10T08:42:12.399526+0000","last_clean":"2024-02-10T08:42:12.399526+0000","last_became_active":"2024-02-10T08:41:44.137505+0000","last_became_peered":"2024-02-10T08:41:44.137505+0000","last_unstale":"2024-02-10T08:42:12.399526+0000","last_undegraded":"2024-02-10T08:42:12.399526+0000","last_fullsized":"2024-02-10T08:42:12.399526+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.14","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043495+0000","last_change":"2024-02-10T08:41:44.142108+0000","last_active":"2024-02-10T08:42:14.043495+0000","last_peered":"2024-02-10T08:42:14.043495+0000","last_clean":"2024-02-10T08:42:14.043495+0000","last_became_active":"2024-02-10T08:41:44.141679+0000","last_became_peered":"2024-02-10T08:41:44.141679+0000","last_unstale":"2024-02-10T08:42:14.043495+0000","last_undegraded":"2024-02-10T08:42:14.043495+0000","last_fullsized":"2024-02-10T08:42:14.043495+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.13","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043338+0000","last_change":"2024-02-10T08:41:44.128346+0000","last_active":"2024-02-10T08:42:14.043338+0000","last_peered":"2024-02-10T08:42:14.043338+0000","last_clean":"2024-02-10T08:42:14.043338+0000","last_became_active":"2024-02-10T08:41:44.128035+0000","last_became_peered":"2024-02-10T08:41:44.128035+0000","last_unstale":"2024-02-10T08:42:14.043338+0000","last_undegraded":"2024-02-10T08:42:14.043338+0000","last_fullsized":"2024-02-10T08:42:14.043338+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.12","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398002+0000","last_change":"2024-02-10T08:41:44.135351+0000","last_active":"2024-02-10T08:42:12.398002+0000","last_peered":"2024-02-10T08:42:12.398002+0000","last_clean":"2024-02-10T08:42:12.398002+0000","last_became_active":"2024-02-10T08:41:44.134596+0000","last_became_peered":"2024-02-10T08:41:44.134596+0000","last_unstale":"2024-02-10T08:42:12.398002+0000","last_undegraded":"2024-02-10T08:42:12.398002+0000","last_fullsized":"2024-02-10T08:42:12.398002+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.11","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367027+0000","last_change":"2024-02-10T08:41:44.135299+0000","last_active":"2024-02-10T08:42:12.367027+0000","last_peered":"2024-02-10T08:42:12.367027+0000","last_clean":"2024-02-10T08:42:12.367027+0000","last_became_active":"2024-02-10T08:41:44.135003+0000","last_became_peered":"2024-02-10T08:41:44.135003+0000","last_unstale":"2024-02-10T08:42:12.367027+0000","last_undegraded":"2024-02-10T08:42:12.367027+0000","last_fullsized":"2024-02-10T08:42:12.367027+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.10","version":"409'2","reported_seq":32,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050769+0000","last_change":"2024-02-10T08:41:44.134989+0000","last_active":"2024-02-10T08:42:14.050769+0000","last_peered":"2024-02-10T08:42:14.050769+0000","last_clean":"2024-02-10T08:42:14.050769+0000","last_became_active":"2024-02-10T08:41:44.134762+0000","last_became_peered":"2024-02-10T08:41:44.134762+0000","last_unstale":"2024-02-10T08:42:14.050769+0000","last_undegraded":"2024-02-10T08:42:14.050769+0000","last_fullsized":"2024-02-10T08:42:14.050769+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.f","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049827+0000","last_change":"2024-02-10T08:41:44.124216+0000","last_active":"2024-02-10T08:42:14.049827+0000","last_peered":"2024-02-10T08:42:14.049827+0000","last_clean":"2024-02-10T08:42:14.049827+0000","last_became_active":"2024-02-10T08:41:44.123955+0000","last_became_peered":"2024-02-10T08:41:44.123955+0000","last_unstale":"2024-02-10T08:42:14.049827+0000","last_undegraded":"2024-02-10T08:42:14.049827+0000","last_fullsized":"2024-02-10T08:42:14.049827+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.e","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367681+0000","last_change":"2024-02-10T08:41:44.137860+0000","last_active":"2024-02-10T08:42:12.367681+0000","last_peered":"2024-02-10T08:42:12.367681+0000","last_clean":"2024-02-10T08:42:12.367681+0000","last_became_active":"2024-02-10T08:41:44.137135+0000","last_became_peered":"2024-02-10T08:41:44.137135+0000","last_unstale":"2024-02-10T08:42:12.367681+0000","last_undegraded":"2024-02-10T08:42:12.367681+0000","last_fullsized":"2024-02-10T08:42:12.367681+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.d","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397710+0000","last_change":"2024-02-10T08:41:44.132523+0000","last_active":"2024-02-10T08:42:12.397710+0000","last_peered":"2024-02-10T08:42:12.397710+0000","last_clean":"2024-02-10T08:42:12.397710+0000","last_became_active":"2024-02-10T08:41:44.132242+0000","last_became_peered":"2024-02-10T08:41:44.132242+0000","last_unstale":"2024-02-10T08:42:12.397710+0000","last_undegraded":"2024-02-10T08:42:12.397710+0000","last_fullsized":"2024-02-10T08:42:12.397710+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.c","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043530+0000","last_change":"2024-02-10T08:41:44.141984+0000","last_active":"2024-02-10T08:42:14.043530+0000","last_peered":"2024-02-10T08:42:14.043530+0000","last_clean":"2024-02-10T08:42:14.043530+0000","last_became_active":"2024-02-10T08:41:44.141211+0000","last_became_peered":"2024-02-10T08:41:44.141211+0000","last_unstale":"2024-02-10T08:42:14.043530+0000","last_undegraded":"2024-02-10T08:42:14.043530+0000","last_fullsized":"2024-02-10T08:42:14.043530+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.b","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367278+0000","last_change":"2024-02-10T08:41:44.141379+0000","last_active":"2024-02-10T08:42:12.367278+0000","last_peered":"2024-02-10T08:42:12.367278+0000","last_clean":"2024-02-10T08:42:12.367278+0000","last_became_active":"2024-02-10T08:41:44.141093+0000","last_became_peered":"2024-02-10T08:41:44.141093+0000","last_unstale":"2024-02-10T08:42:12.367278+0000","last_undegraded":"2024-02-10T08:42:12.367278+0000","last_fullsized":"2024-02-10T08:42:12.367278+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.a","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049727+0000","last_change":"2024-02-10T08:41:44.135965+0000","last_active":"2024-02-10T08:42:14.049727+0000","last_peered":"2024-02-10T08:42:14.049727+0000","last_clean":"2024-02-10T08:42:14.049727+0000","last_became_active":"2024-02-10T08:41:44.135676+0000","last_became_peered":"2024-02-10T08:41:44.135676+0000","last_unstale":"2024-02-10T08:42:14.049727+0000","last_undegraded":"2024-02-10T08:42:14.049727+0000","last_fullsized":"2024-02-10T08:42:14.049727+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.9","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397321+0000","last_change":"2024-02-10T08:41:44.137260+0000","last_active":"2024-02-10T08:42:12.397321+0000","last_peered":"2024-02-10T08:42:12.397321+0000","last_clean":"2024-02-10T08:42:12.397321+0000","last_became_active":"2024-02-10T08:41:44.134738+0000","last_became_peered":"2024-02-10T08:41:44.134738+0000","last_unstale":"2024-02-10T08:42:12.397321+0000","last_undegraded":"2024-02-10T08:42:12.397321+0000","last_fullsized":"2024-02-10T08:42:12.397321+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.8","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397032+0000","last_change":"2024-02-10T08:41:44.137263+0000","last_active":"2024-02-10T08:42:12.397032+0000","last_peered":"2024-02-10T08:42:12.397032+0000","last_clean":"2024-02-10T08:42:12.397032+0000","last_became_active":"2024-02-10T08:41:44.136951+0000","last_became_peered":"2024-02-10T08:41:44.136951+0000","last_unstale":"2024-02-10T08:42:12.397032+0000","last_undegraded":"2024-02-10T08:42:12.397032+0000","last_fullsized":"2024-02-10T08:42:12.397032+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.7","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043967+0000","last_change":"2024-02-10T08:41:44.138924+0000","last_active":"2024-02-10T08:42:14.043967+0000","last_peered":"2024-02-10T08:42:14.043967+0000","last_clean":"2024-02-10T08:42:14.043967+0000","last_became_active":"2024-02-10T08:41:44.138442+0000","last_became_peered":"2024-02-10T08:41:44.138442+0000","last_unstale":"2024-02-10T08:42:14.043967+0000","last_undegraded":"2024-02-10T08:42:14.043967+0000","last_fullsized":"2024-02-10T08:42:14.043967+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.6","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367726+0000","last_change":"2024-02-10T08:41:44.138239+0000","last_active":"2024-02-10T08:42:12.367726+0000","last_peered":"2024-02-10T08:42:12.367726+0000","last_clean":"2024-02-10T08:42:12.367726+0000","last_became_active":"2024-02-10T08:41:44.137918+0000","last_became_peered":"2024-02-10T08:41:44.137918+0000","last_unstale":"2024-02-10T08:42:12.367726+0000","last_undegraded":"2024-02-10T08:42:12.367726+0000","last_fullsized":"2024-02-10T08:42:12.367726+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.5","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049032+0000","last_change":"2024-02-10T08:41:44.133541+0000","last_active":"2024-02-10T08:42:14.049032+0000","last_peered":"2024-02-10T08:42:14.049032+0000","last_clean":"2024-02-10T08:42:14.049032+0000","last_became_active":"2024-02-10T08:41:44.133219+0000","last_became_peered":"2024-02-10T08:41:44.133219+0000","last_unstale":"2024-02-10T08:42:14.049032+0000","last_undegraded":"2024-02-10T08:42:14.049032+0000","last_fullsized":"2024-02-10T08:42:14.049032+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.4","version":"0'0","reported_seq":27,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397680+0000","last_change":"2024-02-10T08:41:44.132635+0000","last_active":"2024-02-10T08:42:12.397680+0000","last_peered":"2024-02-10T08:42:12.397680+0000","last_clean":"2024-02-10T08:42:12.397680+0000","last_became_active":"2024-02-10T08:41:44.132332+0000","last_became_peered":"2024-02-10T08:41:44.132332+0000","last_unstale":"2024-02-10T08:42:12.397680+0000","last_undegraded":"2024-02-10T08:42:12.397680+0000","last_fullsized":"2024-02-10T08:42:12.397680+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.3","version":"418'5","reported_seq":35,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397401+0000","last_change":"2024-02-10T08:41:44.134017+0000","last_active":"2024-02-10T08:42:12.397401+0000","last_peered":"2024-02-10T08:42:12.397401+0000","last_clean":"2024-02-10T08:42:12.397401+0000","last_became_active":"2024-02-10T08:41:44.133698+0000","last_became_peered":"2024-02-10T08:41:44.133698+0000","last_unstale":"2024-02-10T08:42:12.397401+0000","last_undegraded":"2024-02-10T08:42:12.397401+0000","last_fullsized":"2024-02-10T08:42:12.397401+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.2","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048051+0000","last_change":"2024-02-10T08:41:44.135803+0000","last_active":"2024-02-10T08:42:14.048051+0000","last_peered":"2024-02-10T08:42:14.048051+0000","last_clean":"2024-02-10T08:42:14.048051+0000","last_became_active":"2024-02-10T08:41:44.135440+0000","last_became_peered":"2024-02-10T08:41:44.135440+0000","last_unstale":"2024-02-10T08:42:14.048051+0000","last_undegraded":"2024-02-10T08:42:14.048051+0000","last_fullsized":"2024-02-10T08:42:14.048051+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.1","version":"418'4","reported_seq":32,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367107+0000","last_change":"2024-02-10T08:41:44.141608+0000","last_active":"2024-02-10T08:42:12.367107+0000","last_peered":"2024-02-10T08:42:12.367107+0000","last_clean":"2024-02-10T08:42:12.367107+0000","last_became_active":"2024-02-10T08:41:44.141296+0000","last_became_peered":"2024-02-10T08:41:44.141296+0000","last_unstale":"2024-02-10T08:42:12.367107+0000","last_undegraded":"2024-02-10T08:42:12.367107+0000","last_fullsized":"2024-02-10T08:42:12.367107+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.0","version":"0'0","reported_seq":28,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.042883+0000","last_change":"2024-02-10T08:41:44.136327+0000","last_active":"2024-02-10T08:42:14.042883+0000","last_peered":"2024-02-10T08:42:14.042883+0000","last_clean":"2024-02-10T08:42:14.042883+0000","last_became_active":"2024-02-10T08:41:44.136003+0000","last_became_peered":"2024-02-10T08:41:44.136003+0000","last_unstale":"2024-02-10T08:42:14.042883+0000","last_undegraded":"2024-02-10T08:42:14.042883+0000","last_fullsized":"2024-02-10T08:42:14.042883+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:43.101875+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.101875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048538+0000","last_change":"2024-02-10T08:42:09.261204+0000","last_active":"2024-02-10T08:42:14.048538+0000","last_peered":"2024-02-10T08:42:14.048538+0000","last_clean":"2024-02-10T08:42:14.048538+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:42:14.048538+0000","last_undegraded":"2024-02-10T08:42:14.048538+0000","last_fullsized":"2024-02-10T08:42:14.048538+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:40:32.184829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.184829+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.2","version":"338'29","reported_seq":413,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043383+0000","last_change":"2024-02-10T08:42:09.271741+0000","last_active":"2024-02-10T08:42:14.043383+0000","last_peered":"2024-02-10T08:42:14.043383+0000","last_clean":"2024-02-10T08:42:14.043383+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:42:14.043383+0000","last_undegraded":"2024-02-10T08:42:14.043383+0000","last_fullsized":"2024-02-10T08:42:14.043383+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1","version":"401'42","reported_seq":413,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043124+0000","last_change":"2024-02-10T08:42:09.270997+0000","last_active":"2024-02-10T08:42:14.043124+0000","last_peered":"2024-02-10T08:42:14.043124+0000","last_clean":"2024-02-10T08:42:14.043124+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:42:14.043124+0000","last_undegraded":"2024-02-10T08:42:14.043124+0000","last_fullsized":"2024-02-10T08:42:14.043124+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.0","version":"371'108","reported_seq":488,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051019+0000","last_change":"2024-02-10T08:42:09.259389+0000","last_active":"2024-02-10T08:42:14.051019+0000","last_peered":"2024-02-10T08:42:14.051019+0000","last_clean":"2024-02-10T08:42:14.051019+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:42:14.051019+0000","last_undegraded":"2024-02-10T08:42:14.051019+0000","last_fullsized":"2024-02-10T08:42:14.051019+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.7","version":"371'35","reported_seq":391,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044122+0000","last_change":"2024-02-10T08:42:09.273008+0000","last_active":"2024-02-10T08:42:14.044122+0000","last_peered":"2024-02-10T08:42:14.044122+0000","last_clean":"2024-02-10T08:42:14.044122+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:42:14.044122+0000","last_undegraded":"2024-02-10T08:42:14.044122+0000","last_fullsized":"2024-02-10T08:42:14.044122+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.6","version":"338'73","reported_seq":445,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050535+0000","last_change":"2024-02-10T08:42:09.262392+0000","last_active":"2024-02-10T08:42:14.050535+0000","last_peered":"2024-02-10T08:42:14.050535+0000","last_clean":"2024-02-10T08:42:14.050535+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:42:14.050535+0000","last_undegraded":"2024-02-10T08:42:14.050535+0000","last_fullsized":"2024-02-10T08:42:14.050535+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.5","version":"279'42","reported_seq":427,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043732+0000","last_change":"2024-02-10T08:42:09.272549+0000","last_active":"2024-02-10T08:42:14.043732+0000","last_peered":"2024-02-10T08:42:14.043732+0000","last_clean":"2024-02-10T08:42:14.043732+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:42:14.043732+0000","last_undegraded":"2024-02-10T08:42:14.043732+0000","last_fullsized":"2024-02-10T08:42:14.043732+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.4","version":"338'23","reported_seq":405,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043626+0000","last_change":"2024-02-10T08:42:09.272043+0000","last_active":"2024-02-10T08:42:14.043626+0000","last_peered":"2024-02-10T08:42:14.043626+0000","last_clean":"2024-02-10T08:42:14.043626+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:42:14.043626+0000","last_undegraded":"2024-02-10T08:42:14.043626+0000","last_fullsized":"2024-02-10T08:42:14.043626+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.b","version":"395'105","reported_seq":476,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050255+0000","last_change":"2024-02-10T08:42:09.260818+0000","last_active":"2024-02-10T08:42:14.050255+0000","last_peered":"2024-02-10T08:42:14.050255+0000","last_clean":"2024-02-10T08:42:14.050255+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:42:14.050255+0000","last_undegraded":"2024-02-10T08:42:14.050255+0000","last_fullsized":"2024-02-10T08:42:14.050255+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'101","last_scrub_stamp":"2024-02-10T08:40:28.463426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.463426+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.a","version":"371'23","reported_seq":406,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043340+0000","last_change":"2024-02-10T08:42:09.271173+0000","last_active":"2024-02-10T08:42:14.043340+0000","last_peered":"2024-02-10T08:42:14.043340+0000","last_clean":"2024-02-10T08:42:14.043340+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:42:14.043340+0000","last_undegraded":"2024-02-10T08:42:14.043340+0000","last_fullsized":"2024-02-10T08:42:14.043340+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.9","version":"395'115","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048010+0000","last_change":"2024-02-10T08:42:09.263795+0000","last_active":"2024-02-10T08:42:14.048010+0000","last_peered":"2024-02-10T08:42:14.048010+0000","last_clean":"2024-02-10T08:42:14.048010+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:42:14.048010+0000","last_undegraded":"2024-02-10T08:42:14.048010+0000","last_fullsized":"2024-02-10T08:42:14.048010+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":115,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":35,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.8","version":"338'86","reported_seq":459,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043047+0000","last_change":"2024-02-10T08:42:09.270554+0000","last_active":"2024-02-10T08:42:14.043047+0000","last_peered":"2024-02-10T08:42:14.043047+0000","last_clean":"2024-02-10T08:42:14.043047+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:42:14.043047+0000","last_undegraded":"2024-02-10T08:42:14.043047+0000","last_fullsized":"2024-02-10T08:42:14.043047+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.f","version":"405'96","reported_seq":464,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399002+0000","last_change":"2024-02-10T08:42:09.599263+0000","last_active":"2024-02-10T08:42:12.399002+0000","last_peered":"2024-02-10T08:42:12.399002+0000","last_clean":"2024-02-10T08:42:12.399002+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:42:12.399002+0000","last_undegraded":"2024-02-10T08:42:12.399002+0000","last_fullsized":"2024-02-10T08:42:12.399002+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:41:45.373410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.373410+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.e","version":"250'56","reported_seq":410,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397912+0000","last_change":"2024-02-10T08:42:09.598551+0000","last_active":"2024-02-10T08:42:12.397912+0000","last_peered":"2024-02-10T08:42:12.397912+0000","last_clean":"2024-02-10T08:42:12.397912+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:42:12.397912+0000","last_undegraded":"2024-02-10T08:42:12.397912+0000","last_fullsized":"2024-02-10T08:42:12.397912+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.d","version":"423'78","reported_seq":450,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399721+0000","last_change":"2024-02-10T08:42:11.601220+0000","last_active":"2024-02-10T08:42:12.399721+0000","last_peered":"2024-02-10T08:42:12.399721+0000","last_clean":"2024-02-10T08:42:12.399721+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:42:12.399721+0000","last_undegraded":"2024-02-10T08:42:12.399721+0000","last_fullsized":"2024-02-10T08:42:12.399721+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'74","last_scrub_stamp":"2024-02-10T08:41:55.273156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.273156+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.c","version":"338'70","reported_seq":444,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051265+0000","last_change":"2024-02-10T08:42:09.262189+0000","last_active":"2024-02-10T08:42:14.051265+0000","last_peered":"2024-02-10T08:42:14.051265+0000","last_clean":"2024-02-10T08:42:14.051265+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:42:14.051265+0000","last_undegraded":"2024-02-10T08:42:14.051265+0000","last_fullsized":"2024-02-10T08:42:14.051265+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:40:36.430069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.430069+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.13","version":"338'56","reported_seq":433,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400723+0000","last_change":"2024-02-10T08:42:09.598603+0000","last_active":"2024-02-10T08:42:12.400723+0000","last_peered":"2024-02-10T08:42:12.400723+0000","last_clean":"2024-02-10T08:42:12.400723+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:42:12.400723+0000","last_undegraded":"2024-02-10T08:42:12.400723+0000","last_fullsized":"2024-02-10T08:42:12.400723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'56","last_scrub_stamp":"2024-02-10T08:41:49.073509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.073509+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.12","version":"214'102","reported_seq":479,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367117+0000","last_change":"2024-02-10T08:42:09.258181+0000","last_active":"2024-02-10T08:42:12.367117+0000","last_peered":"2024-02-10T08:42:12.367117+0000","last_clean":"2024-02-10T08:42:12.367117+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:42:12.367117+0000","last_undegraded":"2024-02-10T08:42:12.367117+0000","last_fullsized":"2024-02-10T08:42:12.367117+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.11","version":"398'105","reported_seq":465,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399018+0000","last_change":"2024-02-10T08:42:09.598422+0000","last_active":"2024-02-10T08:42:12.399018+0000","last_peered":"2024-02-10T08:42:12.399018+0000","last_clean":"2024-02-10T08:42:12.399018+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:42:12.399018+0000","last_undegraded":"2024-02-10T08:42:12.399018+0000","last_fullsized":"2024-02-10T08:42:12.399018+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.10","version":"301'72","reported_seq":454,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.366998+0000","last_change":"2024-02-10T08:42:09.258477+0000","last_active":"2024-02-10T08:42:12.366998+0000","last_peered":"2024-02-10T08:42:12.366998+0000","last_clean":"2024-02-10T08:42:12.366998+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:42:12.366998+0000","last_undegraded":"2024-02-10T08:42:12.366998+0000","last_fullsized":"2024-02-10T08:42:12.366998+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:42:01.488098+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.488098+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.17","version":"392'87","reported_seq":458,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.368173+0000","last_change":"2024-02-10T08:42:09.258979+0000","last_active":"2024-02-10T08:42:12.368173+0000","last_peered":"2024-02-10T08:42:12.368173+0000","last_clean":"2024-02-10T08:42:12.368173+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:42:12.368173+0000","last_undegraded":"2024-02-10T08:42:12.368173+0000","last_fullsized":"2024-02-10T08:42:12.368173+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:41:57.565097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.565097+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.16","version":"231'62","reported_seq":455,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397851+0000","last_change":"2024-02-10T08:42:09.598636+0000","last_active":"2024-02-10T08:42:12.397851+0000","last_peered":"2024-02-10T08:42:12.397851+0000","last_clean":"2024-02-10T08:42:12.397851+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:42:12.397851+0000","last_undegraded":"2024-02-10T08:42:12.397851+0000","last_fullsized":"2024-02-10T08:42:12.397851+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.15","version":"338'18","reported_seq":397,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043727+0000","last_change":"2024-02-10T08:42:09.272540+0000","last_active":"2024-02-10T08:42:14.043727+0000","last_peered":"2024-02-10T08:42:14.043727+0000","last_clean":"2024-02-10T08:42:14.043727+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:42:14.043727+0000","last_undegraded":"2024-02-10T08:42:14.043727+0000","last_fullsized":"2024-02-10T08:42:14.043727+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.14","version":"295'36","reported_seq":408,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397926+0000","last_change":"2024-02-10T08:42:09.598112+0000","last_active":"2024-02-10T08:42:12.397926+0000","last_peered":"2024-02-10T08:42:12.397926+0000","last_clean":"2024-02-10T08:42:12.397926+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:42:12.397926+0000","last_undegraded":"2024-02-10T08:42:12.397926+0000","last_fullsized":"2024-02-10T08:42:12.397926+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1b","version":"338'69","reported_seq":423,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043194+0000","last_change":"2024-02-10T08:42:09.271571+0000","last_active":"2024-02-10T08:42:14.043194+0000","last_peered":"2024-02-10T08:42:14.043194+0000","last_clean":"2024-02-10T08:42:14.043194+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:42:14.043194+0000","last_undegraded":"2024-02-10T08:42:14.043194+0000","last_fullsized":"2024-02-10T08:42:14.043194+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1a","version":"398'71","reported_seq":434,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048164+0000","last_change":"2024-02-10T08:42:09.264276+0000","last_active":"2024-02-10T08:42:14.048164+0000","last_peered":"2024-02-10T08:42:14.048164+0000","last_clean":"2024-02-10T08:42:14.048164+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:42:14.048164+0000","last_undegraded":"2024-02-10T08:42:14.048164+0000","last_fullsized":"2024-02-10T08:42:14.048164+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:40:35.307608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.307608+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.19","version":"338'35","reported_seq":421,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048041+0000","last_change":"2024-02-10T08:42:09.264072+0000","last_active":"2024-02-10T08:42:14.048041+0000","last_peered":"2024-02-10T08:42:14.048041+0000","last_clean":"2024-02-10T08:42:14.048041+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:42:14.048041+0000","last_undegraded":"2024-02-10T08:42:14.048041+0000","last_fullsized":"2024-02-10T08:42:14.048041+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'34","last_scrub_stamp":"2024-02-10T08:40:28.186484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.186484+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.18","version":"401'38","reported_seq":395,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043039+0000","last_change":"2024-02-10T08:42:09.270546+0000","last_active":"2024-02-10T08:42:14.043039+0000","last_peered":"2024-02-10T08:42:14.043039+0000","last_clean":"2024-02-10T08:42:14.043039+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:42:14.043039+0000","last_undegraded":"2024-02-10T08:42:14.043039+0000","last_fullsized":"2024-02-10T08:42:14.043039+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1f","version":"338'36","reported_seq":424,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400829+0000","last_change":"2024-02-10T08:42:09.600510+0000","last_active":"2024-02-10T08:42:12.400829+0000","last_peered":"2024-02-10T08:42:12.400829+0000","last_clean":"2024-02-10T08:42:12.400829+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:42:12.400829+0000","last_undegraded":"2024-02-10T08:42:12.400829+0000","last_fullsized":"2024-02-10T08:42:12.400829+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'36","last_scrub_stamp":"2024-02-10T08:41:46.085763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.085763+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1e","version":"210'17","reported_seq":383,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049456+0000","last_change":"2024-02-10T08:42:09.261550+0000","last_active":"2024-02-10T08:42:14.049456+0000","last_peered":"2024-02-10T08:42:14.049456+0000","last_clean":"2024-02-10T08:42:14.049456+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:42:14.049456+0000","last_undegraded":"2024-02-10T08:42:14.049456+0000","last_fullsized":"2024-02-10T08:42:14.049456+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:42:04.642908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.642908+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1d","version":"405'21","reported_seq":388,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397807+0000","last_change":"2024-02-10T08:42:09.598220+0000","last_active":"2024-02-10T08:42:12.397807+0000","last_peered":"2024-02-10T08:42:12.397807+0000","last_clean":"2024-02-10T08:42:12.397807+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:42:12.397807+0000","last_undegraded":"2024-02-10T08:42:12.397807+0000","last_fullsized":"2024-02-10T08:42:12.397807+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1c","version":"371'35","reported_seq":415,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051101+0000","last_change":"2024-02-10T08:42:09.262014+0000","last_active":"2024-02-10T08:42:14.051101+0000","last_peered":"2024-02-10T08:42:14.051101+0000","last_clean":"2024-02-10T08:42:14.051101+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:42:14.051101+0000","last_undegraded":"2024-02-10T08:42:14.051101+0000","last_fullsized":"2024-02-10T08:42:14.051101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:40:23.345538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.345538+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"5.9","version":"124'10","reported_seq":527,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367670+0000","last_change":"2024-02-10T08:41:09.353173+0000","last_active":"2024-02-10T08:42:12.367670+0000","last_peered":"2024-02-10T08:42:12.367670+0000","last_clean":"2024-02-10T08:42:12.367670+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:42:12.367670+0000","last_undegraded":"2024-02-10T08:42:12.367670+0000","last_fullsized":"2024-02-10T08:42:12.367670+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:41:09.353088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:09.353088+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":421,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397544+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:42:12.397544+0000","last_peered":"2024-02-10T08:42:12.397544+0000","last_clean":"2024-02-10T08:42:12.397544+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:42:12.397544+0000","last_undegraded":"2024-02-10T08:42:12.397544+0000","last_fullsized":"2024-02-10T08:42:12.397544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":395,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050625+0000","last_change":"2024-02-10T08:40:42.232974+0000","last_active":"2024-02-10T08:42:14.050625+0000","last_peered":"2024-02-10T08:42:14.050625+0000","last_clean":"2024-02-10T08:42:14.050625+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:42:14.050625+0000","last_undegraded":"2024-02-10T08:42:14.050625+0000","last_fullsized":"2024-02-10T08:42:14.050625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.232905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.232905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":399,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398526+0000","last_change":"2024-02-10T08:40:26.859788+0000","last_active":"2024-02-10T08:42:12.398526+0000","last_peered":"2024-02-10T08:42:12.398526+0000","last_clean":"2024-02-10T08:42:12.398526+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:42:12.398526+0000","last_undegraded":"2024-02-10T08:42:12.398526+0000","last_fullsized":"2024-02-10T08:42:12.398526+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.859697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.859697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":343,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051341+0000","last_change":"2024-02-10T08:41:59.348271+0000","last_active":"2024-02-10T08:42:14.051341+0000","last_peered":"2024-02-10T08:42:14.051341+0000","last_clean":"2024-02-10T08:42:14.051341+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:42:14.051341+0000","last_undegraded":"2024-02-10T08:42:14.051341+0000","last_fullsized":"2024-02-10T08:42:14.051341+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.348211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.348211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":372,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050659+0000","last_change":"2024-02-10T08:41:16.350785+0000","last_active":"2024-02-10T08:42:14.050659+0000","last_peered":"2024-02-10T08:42:14.050659+0000","last_clean":"2024-02-10T08:42:14.050659+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:42:14.050659+0000","last_undegraded":"2024-02-10T08:42:14.050659+0000","last_fullsized":"2024-02-10T08:42:14.050659+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":400,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049449+0000","last_change":"2024-02-10T08:41:05.559497+0000","last_active":"2024-02-10T08:42:14.049449+0000","last_peered":"2024-02-10T08:42:14.049449+0000","last_clean":"2024-02-10T08:42:14.049449+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:42:14.049449+0000","last_undegraded":"2024-02-10T08:42:14.049449+0000","last_fullsized":"2024-02-10T08:42:14.049449+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.559402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.559402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":414,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049947+0000","last_change":"2024-02-10T08:41:43.195685+0000","last_active":"2024-02-10T08:42:14.049947+0000","last_peered":"2024-02-10T08:42:14.049947+0000","last_clean":"2024-02-10T08:42:14.049947+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:42:14.049947+0000","last_undegraded":"2024-02-10T08:42:14.049947+0000","last_fullsized":"2024-02-10T08:42:14.049947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":309,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367293+0000","last_change":"2024-02-10T08:40:39.350744+0000","last_active":"2024-02-10T08:42:12.367293+0000","last_peered":"2024-02-10T08:42:12.367293+0000","last_clean":"2024-02-10T08:42:12.367293+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:42:12.367293+0000","last_undegraded":"2024-02-10T08:42:12.367293+0000","last_fullsized":"2024-02-10T08:42:12.367293+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.350663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":339,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.401127+0000","last_change":"2024-02-10T08:41:45.092740+0000","last_active":"2024-02-10T08:42:12.401127+0000","last_peered":"2024-02-10T08:42:12.401127+0000","last_clean":"2024-02-10T08:42:12.401127+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:42:12.401127+0000","last_undegraded":"2024-02-10T08:42:12.401127+0000","last_fullsized":"2024-02-10T08:42:12.401127+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:45.092620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.092620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":431,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398246+0000","last_change":"2024-02-10T08:41:57.284919+0000","last_active":"2024-02-10T08:42:12.398246+0000","last_peered":"2024-02-10T08:42:12.398246+0000","last_clean":"2024-02-10T08:42:12.398246+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:42:12.398246+0000","last_undegraded":"2024-02-10T08:42:12.398246+0000","last_fullsized":"2024-02-10T08:42:12.398246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:57.284811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.284811+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":404,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051528+0000","last_change":"2024-02-10T08:40:40.314002+0000","last_active":"2024-02-10T08:42:14.051528+0000","last_peered":"2024-02-10T08:42:14.051528+0000","last_clean":"2024-02-10T08:42:14.051528+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:42:14.051528+0000","last_undegraded":"2024-02-10T08:42:14.051528+0000","last_fullsized":"2024-02-10T08:42:14.051528+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.313935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":418,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050494+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:42:14.050494+0000","last_peered":"2024-02-10T08:42:14.050494+0000","last_clean":"2024-02-10T08:42:14.050494+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:42:14.050494+0000","last_undegraded":"2024-02-10T08:42:14.050494+0000","last_fullsized":"2024-02-10T08:42:14.050494+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.366800+0000","last_change":"2024-02-10T08:40:28.426111+0000","last_active":"2024-02-10T08:42:12.366800+0000","last_peered":"2024-02-10T08:42:12.366800+0000","last_clean":"2024-02-10T08:42:12.366800+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:42:12.366800+0000","last_undegraded":"2024-02-10T08:42:12.366800+0000","last_fullsized":"2024-02-10T08:42:12.366800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:28.425994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.425994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":339,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367898+0000","last_change":"2024-02-10T08:41:20.535962+0000","last_active":"2024-02-10T08:42:12.367898+0000","last_peered":"2024-02-10T08:42:12.367898+0000","last_clean":"2024-02-10T08:42:12.367898+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:42:12.367898+0000","last_undegraded":"2024-02-10T08:42:12.367898+0000","last_fullsized":"2024-02-10T08:42:12.367898+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":425,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044073+0000","last_change":"2024-02-10T08:41:13.773323+0000","last_active":"2024-02-10T08:42:14.044073+0000","last_peered":"2024-02-10T08:42:14.044073+0000","last_clean":"2024-02-10T08:42:14.044073+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:42:14.044073+0000","last_undegraded":"2024-02-10T08:42:14.044073+0000","last_fullsized":"2024-02-10T08:42:14.044073+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:41:13.773224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.773224+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":233,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397493+0000","last_change":"2024-02-10T08:41:16.638540+0000","last_active":"2024-02-10T08:42:12.397493+0000","last_peered":"2024-02-10T08:42:12.397493+0000","last_clean":"2024-02-10T08:42:12.397493+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:42:12.397493+0000","last_undegraded":"2024-02-10T08:42:12.397493+0000","last_fullsized":"2024-02-10T08:42:12.397493+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.638477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.638477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":418,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048836+0000","last_change":"2024-02-10T08:40:40.501961+0000","last_active":"2024-02-10T08:42:14.048836+0000","last_peered":"2024-02-10T08:42:14.048836+0000","last_clean":"2024-02-10T08:42:14.048836+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:42:14.048836+0000","last_undegraded":"2024-02-10T08:42:14.048836+0000","last_fullsized":"2024-02-10T08:42:14.048836+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.501871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.501871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":389,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050432+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:42:14.050432+0000","last_peered":"2024-02-10T08:42:14.050432+0000","last_clean":"2024-02-10T08:42:14.050432+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:42:14.050432+0000","last_undegraded":"2024-02-10T08:42:14.050432+0000","last_fullsized":"2024-02-10T08:42:14.050432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":337,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043118+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:42:14.043118+0000","last_peered":"2024-02-10T08:42:14.043118+0000","last_clean":"2024-02-10T08:42:14.043118+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:42:14.043118+0000","last_undegraded":"2024-02-10T08:42:14.043118+0000","last_fullsized":"2024-02-10T08:42:14.043118+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":412,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043814+0000","last_change":"2024-02-10T08:41:18.755591+0000","last_active":"2024-02-10T08:42:14.043814+0000","last_peered":"2024-02-10T08:42:14.043814+0000","last_clean":"2024-02-10T08:42:14.043814+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:42:14.043814+0000","last_undegraded":"2024-02-10T08:42:14.043814+0000","last_fullsized":"2024-02-10T08:42:14.043814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":400,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048452+0000","last_change":"2024-02-10T08:41:10.625950+0000","last_active":"2024-02-10T08:42:14.048452+0000","last_peered":"2024-02-10T08:42:14.048452+0000","last_clean":"2024-02-10T08:42:14.048452+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:42:14.048452+0000","last_undegraded":"2024-02-10T08:42:14.048452+0000","last_fullsized":"2024-02-10T08:42:14.048452+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.625888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.625888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":526,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050923+0000","last_change":"2024-02-10T08:40:41.276981+0000","last_active":"2024-02-10T08:42:14.050923+0000","last_peered":"2024-02-10T08:42:14.050923+0000","last_clean":"2024-02-10T08:42:14.050923+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:42:14.050923+0000","last_undegraded":"2024-02-10T08:42:14.050923+0000","last_fullsized":"2024-02-10T08:42:14.050923+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:40:41.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.276901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":393,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044477+0000","last_change":"2024-02-10T08:41:43.683458+0000","last_active":"2024-02-10T08:42:14.044477+0000","last_peered":"2024-02-10T08:42:14.044477+0000","last_clean":"2024-02-10T08:42:14.044477+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:42:14.044477+0000","last_undegraded":"2024-02-10T08:42:14.044477+0000","last_fullsized":"2024-02-10T08:42:14.044477+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":307,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397122+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:42:12.397122+0000","last_peered":"2024-02-10T08:42:12.397122+0000","last_clean":"2024-02-10T08:42:12.397122+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:42:12.397122+0000","last_undegraded":"2024-02-10T08:42:12.397122+0000","last_fullsized":"2024-02-10T08:42:12.397122+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":418,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050231+0000","last_change":"2024-02-10T08:40:42.486085+0000","last_active":"2024-02-10T08:42:14.050231+0000","last_peered":"2024-02-10T08:42:14.050231+0000","last_clean":"2024-02-10T08:42:14.050231+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:42:14.050231+0000","last_undegraded":"2024-02-10T08:42:14.050231+0000","last_fullsized":"2024-02-10T08:42:14.050231+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:40:42.486021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.486021+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048241+0000","last_change":"2024-02-10T08:41:21.599108+0000","last_active":"2024-02-10T08:42:14.048241+0000","last_peered":"2024-02-10T08:42:14.048241+0000","last_clean":"2024-02-10T08:42:14.048241+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:42:14.048241+0000","last_undegraded":"2024-02-10T08:42:14.048241+0000","last_fullsized":"2024-02-10T08:42:14.048241+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:21.599034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.599034+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":550,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044464+0000","last_change":"2024-02-10T08:40:25.631962+0000","last_active":"2024-02-10T08:42:14.044464+0000","last_peered":"2024-02-10T08:42:14.044464+0000","last_clean":"2024-02-10T08:42:14.044464+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:42:14.044464+0000","last_undegraded":"2024-02-10T08:42:14.044464+0000","last_fullsized":"2024-02-10T08:42:14.044464+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:40:25.631854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:25.631854+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":394,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400782+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:42:12.400782+0000","last_peered":"2024-02-10T08:42:12.400782+0000","last_clean":"2024-02-10T08:42:12.400782+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:42:12.400782+0000","last_undegraded":"2024-02-10T08:42:12.400782+0000","last_fullsized":"2024-02-10T08:42:12.400782+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":348,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400807+0000","last_change":"2024-02-10T08:41:54.063932+0000","last_active":"2024-02-10T08:42:12.400807+0000","last_peered":"2024-02-10T08:42:12.400807+0000","last_clean":"2024-02-10T08:42:12.400807+0000","last_became_active":"2024-02-10T08:40:42.028734+0000","last_became_peered":"2024-02-10T08:40:42.028734+0000","last_unstale":"2024-02-10T08:42:12.400807+0000","last_undegraded":"2024-02-10T08:42:12.400807+0000","last_fullsized":"2024-02-10T08:42:12.400807+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:54.063839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.063839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":416,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044377+0000","last_change":"2024-02-10T08:41:56.714215+0000","last_active":"2024-02-10T08:42:14.044377+0000","last_peered":"2024-02-10T08:42:14.044377+0000","last_clean":"2024-02-10T08:42:14.044377+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:42:14.044377+0000","last_undegraded":"2024-02-10T08:42:14.044377+0000","last_fullsized":"2024-02-10T08:42:14.044377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367236+0000","last_change":"2024-02-10T08:41:08.390794+0000","last_active":"2024-02-10T08:42:12.367236+0000","last_peered":"2024-02-10T08:42:12.367236+0000","last_clean":"2024-02-10T08:42:12.367236+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:42:12.367236+0000","last_undegraded":"2024-02-10T08:42:12.367236+0000","last_fullsized":"2024-02-10T08:42:12.367236+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.390722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.390722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":475,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.401059+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:42:12.401059+0000","last_peered":"2024-02-10T08:42:12.401059+0000","last_clean":"2024-02-10T08:42:12.401059+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:42:12.401059+0000","last_undegraded":"2024-02-10T08:42:12.401059+0000","last_fullsized":"2024-02-10T08:42:12.401059+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398195+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:42:12.398195+0000","last_peered":"2024-02-10T08:42:12.398195+0000","last_clean":"2024-02-10T08:42:12.398195+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:42:12.398195+0000","last_undegraded":"2024-02-10T08:42:12.398195+0000","last_fullsized":"2024-02-10T08:42:12.398195+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":337,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397368+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:42:12.397368+0000","last_peered":"2024-02-10T08:42:12.397368+0000","last_clean":"2024-02-10T08:42:12.397368+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:42:12.397368+0000","last_undegraded":"2024-02-10T08:42:12.397368+0000","last_fullsized":"2024-02-10T08:42:12.397368+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":512,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397331+0000","last_change":"2024-02-10T08:41:44.773002+0000","last_active":"2024-02-10T08:42:12.397331+0000","last_peered":"2024-02-10T08:42:12.397331+0000","last_clean":"2024-02-10T08:42:12.397331+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:42:12.397331+0000","last_undegraded":"2024-02-10T08:42:12.397331+0000","last_fullsized":"2024-02-10T08:42:12.397331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":418,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043362+0000","last_change":"2024-02-10T08:41:24.848074+0000","last_active":"2024-02-10T08:42:14.043362+0000","last_peered":"2024-02-10T08:42:14.043362+0000","last_clean":"2024-02-10T08:42:14.043362+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:42:14.043362+0000","last_undegraded":"2024-02-10T08:42:14.043362+0000","last_fullsized":"2024-02-10T08:42:14.043362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:41:24.847990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.847990+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399200+0000","last_change":"2024-02-10T08:41:48.361432+0000","last_active":"2024-02-10T08:42:12.399200+0000","last_peered":"2024-02-10T08:42:12.399200+0000","last_clean":"2024-02-10T08:42:12.399200+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:42:12.399200+0000","last_undegraded":"2024-02-10T08:42:12.399200+0000","last_fullsized":"2024-02-10T08:42:12.399200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.361357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.361357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.366795+0000","last_change":"2024-02-10T08:41:59.569272+0000","last_active":"2024-02-10T08:42:12.366795+0000","last_peered":"2024-02-10T08:42:12.366795+0000","last_clean":"2024-02-10T08:42:12.366795+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:42:12.366795+0000","last_undegraded":"2024-02-10T08:42:12.366795+0000","last_fullsized":"2024-02-10T08:42:12.366795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.569193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.569193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":313,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049463+0000","last_change":"2024-02-10T08:41:57.599784+0000","last_active":"2024-02-10T08:42:14.049463+0000","last_peered":"2024-02-10T08:42:14.049463+0000","last_clean":"2024-02-10T08:42:14.049463+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:42:14.049463+0000","last_undegraded":"2024-02-10T08:42:14.049463+0000","last_fullsized":"2024-02-10T08:42:14.049463+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":392,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043562+0000","last_change":"2024-02-10T08:40:34.622947+0000","last_active":"2024-02-10T08:42:14.043562+0000","last_peered":"2024-02-10T08:42:14.043562+0000","last_clean":"2024-02-10T08:42:14.043562+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:42:14.043562+0000","last_undegraded":"2024-02-10T08:42:14.043562+0000","last_fullsized":"2024-02-10T08:42:14.043562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:34.622869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":485,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397376+0000","last_change":"2024-02-10T08:41:07.712140+0000","last_active":"2024-02-10T08:42:12.397376+0000","last_peered":"2024-02-10T08:42:12.397376+0000","last_clean":"2024-02-10T08:42:12.397376+0000","last_became_active":"2024-02-10T08:40:42.015369+0000","last_became_peered":"2024-02-10T08:40:42.015369+0000","last_unstale":"2024-02-10T08:42:12.397376+0000","last_undegraded":"2024-02-10T08:42:12.397376+0000","last_fullsized":"2024-02-10T08:42:12.397376+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:41:07.712078+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.712078+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":407,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049612+0000","last_change":"2024-02-10T08:41:46.729506+0000","last_active":"2024-02-10T08:42:14.049612+0000","last_peered":"2024-02-10T08:42:14.049612+0000","last_clean":"2024-02-10T08:42:14.049612+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:42:14.049612+0000","last_undegraded":"2024-02-10T08:42:14.049612+0000","last_fullsized":"2024-02-10T08:42:14.049612+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043890+0000","last_change":"2024-02-10T08:41:10.710795+0000","last_active":"2024-02-10T08:42:14.043890+0000","last_peered":"2024-02-10T08:42:14.043890+0000","last_clean":"2024-02-10T08:42:14.043890+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:42:14.043890+0000","last_undegraded":"2024-02-10T08:42:14.043890+0000","last_fullsized":"2024-02-10T08:42:14.043890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":316,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398073+0000","last_change":"2024-02-10T08:40:42.028088+0000","last_active":"2024-02-10T08:42:12.398073+0000","last_peered":"2024-02-10T08:42:12.398073+0000","last_clean":"2024-02-10T08:42:12.398073+0000","last_became_active":"2024-02-10T08:40:42.027869+0000","last_became_peered":"2024-02-10T08:40:42.027869+0000","last_unstale":"2024-02-10T08:42:12.398073+0000","last_undegraded":"2024-02-10T08:42:12.398073+0000","last_fullsized":"2024-02-10T08:42:12.398073+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":407,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398318+0000","last_change":"2024-02-10T08:41:58.271798+0000","last_active":"2024-02-10T08:42:12.398318+0000","last_peered":"2024-02-10T08:42:12.398318+0000","last_clean":"2024-02-10T08:42:12.398318+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:42:12.398318+0000","last_undegraded":"2024-02-10T08:42:12.398318+0000","last_fullsized":"2024-02-10T08:42:12.398318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.271728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.271728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":375,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400958+0000","last_change":"2024-02-10T08:41:07.950673+0000","last_active":"2024-02-10T08:42:12.400958+0000","last_peered":"2024-02-10T08:42:12.400958+0000","last_clean":"2024-02-10T08:42:12.400958+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:42:12.400958+0000","last_undegraded":"2024-02-10T08:42:12.400958+0000","last_fullsized":"2024-02-10T08:42:12.400958+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:07.950597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.950597+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.368130+0000","last_change":"2024-02-10T08:41:04.369613+0000","last_active":"2024-02-10T08:42:12.368130+0000","last_peered":"2024-02-10T08:42:12.368130+0000","last_clean":"2024-02-10T08:42:12.368130+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:42:12.368130+0000","last_undegraded":"2024-02-10T08:42:12.368130+0000","last_fullsized":"2024-02-10T08:42:12.368130+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.369534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.369534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":57,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050827+0000","last_change":"2024-02-10T08:41:58.300293+0000","last_active":"2024-02-10T08:42:14.050827+0000","last_peered":"2024-02-10T08:42:14.050827+0000","last_clean":"2024-02-10T08:42:14.050827+0000","last_became_active":"2024-02-10T08:41:21.266720+0000","last_became_peered":"2024-02-10T08:41:21.266720+0000","last_unstale":"2024-02-10T08:42:14.050827+0000","last_undegraded":"2024-02-10T08:42:14.050827+0000","last_fullsized":"2024-02-10T08:42:14.050827+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.300183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.300183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":352,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050173+0000","last_change":"2024-02-10T08:41:50.202162+0000","last_active":"2024-02-10T08:42:14.050173+0000","last_peered":"2024-02-10T08:42:14.050173+0000","last_clean":"2024-02-10T08:42:14.050173+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:42:14.050173+0000","last_undegraded":"2024-02-10T08:42:14.050173+0000","last_fullsized":"2024-02-10T08:42:14.050173+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:50.202092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:50.202092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":419,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.401074+0000","last_change":"2024-02-10T08:41:11.901601+0000","last_active":"2024-02-10T08:42:12.401074+0000","last_peered":"2024-02-10T08:42:12.401074+0000","last_clean":"2024-02-10T08:42:12.401074+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:42:12.401074+0000","last_undegraded":"2024-02-10T08:42:12.401074+0000","last_fullsized":"2024-02-10T08:42:12.401074+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.901502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.901502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":357,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049774+0000","last_change":"2024-02-10T08:41:07.584245+0000","last_active":"2024-02-10T08:42:14.049774+0000","last_peered":"2024-02-10T08:42:14.049774+0000","last_clean":"2024-02-10T08:42:14.049774+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:42:14.049774+0000","last_undegraded":"2024-02-10T08:42:14.049774+0000","last_fullsized":"2024-02-10T08:42:14.049774+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:07.584127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.584127+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367062+0000","last_change":"2024-02-10T08:40:23.406297+0000","last_active":"2024-02-10T08:42:12.367062+0000","last_peered":"2024-02-10T08:42:12.367062+0000","last_clean":"2024-02-10T08:42:12.367062+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:42:12.367062+0000","last_undegraded":"2024-02-10T08:42:12.367062+0000","last_fullsized":"2024-02-10T08:42:12.367062+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.406227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.406227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":312,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043504+0000","last_change":"2024-02-10T08:41:21.839964+0000","last_active":"2024-02-10T08:42:14.043504+0000","last_peered":"2024-02-10T08:42:14.043504+0000","last_clean":"2024-02-10T08:42:14.043504+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:42:14.043504+0000","last_undegraded":"2024-02-10T08:42:14.043504+0000","last_fullsized":"2024-02-10T08:42:14.043504+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":395,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048890+0000","last_change":"2024-02-10T08:41:12.554311+0000","last_active":"2024-02-10T08:42:14.048890+0000","last_peered":"2024-02-10T08:42:14.048890+0000","last_clean":"2024-02-10T08:42:14.048890+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:42:14.048890+0000","last_undegraded":"2024-02-10T08:42:14.048890+0000","last_fullsized":"2024-02-10T08:42:14.048890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:41:12.554232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.554232+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":349,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398069+0000","last_change":"2024-02-10T08:41:10.521635+0000","last_active":"2024-02-10T08:42:12.398069+0000","last_peered":"2024-02-10T08:42:12.398069+0000","last_clean":"2024-02-10T08:42:12.398069+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:42:12.398069+0000","last_undegraded":"2024-02-10T08:42:12.398069+0000","last_fullsized":"2024-02-10T08:42:12.398069+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.521563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.521563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":392,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367662+0000","last_change":"2024-02-10T08:41:19.563777+0000","last_active":"2024-02-10T08:42:12.367662+0000","last_peered":"2024-02-10T08:42:12.367662+0000","last_clean":"2024-02-10T08:42:12.367662+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:42:12.367662+0000","last_undegraded":"2024-02-10T08:42:12.367662+0000","last_fullsized":"2024-02-10T08:42:12.367662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:19.563700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.563700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":422,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399873+0000","last_change":"2024-02-10T08:42:00.249642+0000","last_active":"2024-02-10T08:42:12.399873+0000","last_peered":"2024-02-10T08:42:12.399873+0000","last_clean":"2024-02-10T08:42:12.399873+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:42:12.399873+0000","last_undegraded":"2024-02-10T08:42:12.399873+0000","last_fullsized":"2024-02-10T08:42:12.399873+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":413,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367114+0000","last_change":"2024-02-10T08:41:24.519217+0000","last_active":"2024-02-10T08:42:12.367114+0000","last_peered":"2024-02-10T08:42:12.367114+0000","last_clean":"2024-02-10T08:42:12.367114+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:42:12.367114+0000","last_undegraded":"2024-02-10T08:42:12.367114+0000","last_fullsized":"2024-02-10T08:42:12.367114+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.519144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.519144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400892+0000","last_change":"2024-02-10T08:41:13.978089+0000","last_active":"2024-02-10T08:42:12.400892+0000","last_peered":"2024-02-10T08:42:12.400892+0000","last_clean":"2024-02-10T08:42:12.400892+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:42:12.400892+0000","last_undegraded":"2024-02-10T08:42:12.400892+0000","last_fullsized":"2024-02-10T08:42:12.400892+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.978013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.978013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":393,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397581+0000","last_change":"2024-02-10T08:40:39.891622+0000","last_active":"2024-02-10T08:42:12.397581+0000","last_peered":"2024-02-10T08:42:12.397581+0000","last_clean":"2024-02-10T08:42:12.397581+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:42:12.397581+0000","last_undegraded":"2024-02-10T08:42:12.397581+0000","last_fullsized":"2024-02-10T08:42:12.397581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.891558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.891558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.401148+0000","last_change":"2024-02-10T08:41:48.065347+0000","last_active":"2024-02-10T08:42:12.401148+0000","last_peered":"2024-02-10T08:42:12.401148+0000","last_clean":"2024-02-10T08:42:12.401148+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:42:12.401148+0000","last_undegraded":"2024-02-10T08:42:12.401148+0000","last_fullsized":"2024-02-10T08:42:12.401148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":350,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399050+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:42:12.399050+0000","last_peered":"2024-02-10T08:42:12.399050+0000","last_clean":"2024-02-10T08:42:12.399050+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:42:12.399050+0000","last_undegraded":"2024-02-10T08:42:12.399050+0000","last_fullsized":"2024-02-10T08:42:12.399050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":409,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050728+0000","last_change":"2024-02-10T08:41:03.238150+0000","last_active":"2024-02-10T08:42:14.050728+0000","last_peered":"2024-02-10T08:42:14.050728+0000","last_clean":"2024-02-10T08:42:14.050728+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:42:14.050728+0000","last_undegraded":"2024-02-10T08:42:14.050728+0000","last_fullsized":"2024-02-10T08:42:14.050728+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.238035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.238035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":446,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050144+0000","last_change":"2024-02-10T08:41:18.579612+0000","last_active":"2024-02-10T08:42:14.050144+0000","last_peered":"2024-02-10T08:42:14.050144+0000","last_clean":"2024-02-10T08:42:14.050144+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:42:14.050144+0000","last_undegraded":"2024-02-10T08:42:14.050144+0000","last_fullsized":"2024-02-10T08:42:14.050144+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:41:18.579532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.579532+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":406,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050766+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:42:14.050766+0000","last_peered":"2024-02-10T08:42:14.050766+0000","last_clean":"2024-02-10T08:42:14.050766+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:42:14.050766+0000","last_undegraded":"2024-02-10T08:42:14.050766+0000","last_fullsized":"2024-02-10T08:42:14.050766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":935,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-02-10T08:42:16.030360+0000","last_change":"2024-02-10T08:41:44.751436+0000","last_active":"2024-02-10T08:42:16.030360+0000","last_peered":"2024-02-10T08:42:16.030360+0000","last_clean":"2024-02-10T08:42:16.030360+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:42:16.030360+0000","last_undegraded":"2024-02-10T08:42:16.030360+0000","last_fullsized":"2024-02-10T08:42:16.030360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:41:44.751371+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.751371+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":531,"num_read_kb":526,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367944+0000","last_change":"2024-02-10T08:41:15.515804+0000","last_active":"2024-02-10T08:42:12.367944+0000","last_peered":"2024-02-10T08:42:12.367944+0000","last_clean":"2024-02-10T08:42:12.367944+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:42:12.367944+0000","last_undegraded":"2024-02-10T08:42:12.367944+0000","last_fullsized":"2024-02-10T08:42:12.367944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.515729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.515729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":347,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048959+0000","last_change":"2024-02-10T08:40:22.234338+0000","last_active":"2024-02-10T08:42:14.048959+0000","last_peered":"2024-02-10T08:42:14.048959+0000","last_clean":"2024-02-10T08:42:14.048959+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:42:14.048959+0000","last_undegraded":"2024-02-10T08:42:14.048959+0000","last_fullsized":"2024-02-10T08:42:14.048959+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.234270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.234270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":313,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049294+0000","last_change":"2024-02-10T08:41:53.594734+0000","last_active":"2024-02-10T08:42:14.049294+0000","last_peered":"2024-02-10T08:42:14.049294+0000","last_clean":"2024-02-10T08:42:14.049294+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:42:14.049294+0000","last_undegraded":"2024-02-10T08:42:14.049294+0000","last_fullsized":"2024-02-10T08:42:14.049294+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":488,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399269+0000","last_change":"2024-02-10T08:40:40.098585+0000","last_active":"2024-02-10T08:42:12.399269+0000","last_peered":"2024-02-10T08:42:12.399269+0000","last_clean":"2024-02-10T08:42:12.399269+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:42:12.399269+0000","last_undegraded":"2024-02-10T08:42:12.399269+0000","last_fullsized":"2024-02-10T08:42:12.399269+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:40:40.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.098515+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398673+0000","last_change":"2024-02-10T08:40:23.818554+0000","last_active":"2024-02-10T08:42:12.398673+0000","last_peered":"2024-02-10T08:42:12.398673+0000","last_clean":"2024-02-10T08:42:12.398673+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:42:12.398673+0000","last_undegraded":"2024-02-10T08:42:12.398673+0000","last_fullsized":"2024-02-10T08:42:12.398673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.818434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.818434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398137+0000","last_change":"2024-02-10T08:41:11.702739+0000","last_active":"2024-02-10T08:42:12.398137+0000","last_peered":"2024-02-10T08:42:12.398137+0000","last_clean":"2024-02-10T08:42:12.398137+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:42:12.398137+0000","last_undegraded":"2024-02-10T08:42:12.398137+0000","last_fullsized":"2024-02-10T08:42:12.398137+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":352,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051047+0000","last_change":"2024-02-10T08:42:04.361526+0000","last_active":"2024-02-10T08:42:14.051047+0000","last_peered":"2024-02-10T08:42:14.051047+0000","last_clean":"2024-02-10T08:42:14.051047+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:42:14.051047+0000","last_undegraded":"2024-02-10T08:42:14.051047+0000","last_fullsized":"2024-02-10T08:42:14.051047+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:04.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.361430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":399,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398776+0000","last_change":"2024-02-10T08:41:54.316045+0000","last_active":"2024-02-10T08:42:12.398776+0000","last_peered":"2024-02-10T08:42:12.398776+0000","last_clean":"2024-02-10T08:42:12.398776+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:42:12.398776+0000","last_undegraded":"2024-02-10T08:42:12.398776+0000","last_fullsized":"2024-02-10T08:42:12.398776+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:41:54.315983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.315983+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":418,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048627+0000","last_change":"2024-02-10T08:41:03.584639+0000","last_active":"2024-02-10T08:42:14.048627+0000","last_peered":"2024-02-10T08:42:14.048627+0000","last_clean":"2024-02-10T08:42:14.048627+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:42:14.048627+0000","last_undegraded":"2024-02-10T08:42:14.048627+0000","last_fullsized":"2024-02-10T08:42:14.048627+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:41:03.584573+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.584573+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048756+0000","last_change":"2024-02-10T08:41:24.622753+0000","last_active":"2024-02-10T08:42:14.048756+0000","last_peered":"2024-02-10T08:42:14.048756+0000","last_clean":"2024-02-10T08:42:14.048756+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:42:14.048756+0000","last_undegraded":"2024-02-10T08:42:14.048756+0000","last_fullsized":"2024-02-10T08:42:14.048756+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.622691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.622691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":307,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397236+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:42:12.397236+0000","last_peered":"2024-02-10T08:42:12.397236+0000","last_clean":"2024-02-10T08:42:12.397236+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:42:12.397236+0000","last_undegraded":"2024-02-10T08:42:12.397236+0000","last_fullsized":"2024-02-10T08:42:12.397236+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":337,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397934+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:42:12.397934+0000","last_peered":"2024-02-10T08:42:12.397934+0000","last_clean":"2024-02-10T08:42:12.397934+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:42:12.397934+0000","last_undegraded":"2024-02-10T08:42:12.397934+0000","last_fullsized":"2024-02-10T08:42:12.397934+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":423,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051231+0000","last_change":"2024-02-10T08:41:22.395964+0000","last_active":"2024-02-10T08:42:14.051231+0000","last_peered":"2024-02-10T08:42:14.051231+0000","last_clean":"2024-02-10T08:42:14.051231+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:42:14.051231+0000","last_undegraded":"2024-02-10T08:42:14.051231+0000","last_fullsized":"2024-02-10T08:42:14.051231+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.395872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.395872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":425,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043465+0000","last_change":"2024-02-10T08:41:44.633272+0000","last_active":"2024-02-10T08:42:14.043465+0000","last_peered":"2024-02-10T08:42:14.043465+0000","last_clean":"2024-02-10T08:42:14.043465+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:42:14.043465+0000","last_undegraded":"2024-02-10T08:42:14.043465+0000","last_fullsized":"2024-02-10T08:42:14.043465+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":422,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044256+0000","last_change":"2024-02-10T08:40:38.667994+0000","last_active":"2024-02-10T08:42:14.044256+0000","last_peered":"2024-02-10T08:42:14.044256+0000","last_clean":"2024-02-10T08:42:14.044256+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:42:14.044256+0000","last_undegraded":"2024-02-10T08:42:14.044256+0000","last_fullsized":"2024-02-10T08:42:14.044256+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:40:38.667923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:38.667923+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":392,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.396941+0000","last_change":"2024-02-10T08:42:05.067474+0000","last_active":"2024-02-10T08:42:12.396941+0000","last_peered":"2024-02-10T08:42:12.396941+0000","last_clean":"2024-02-10T08:42:12.396941+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:42:12.396941+0000","last_undegraded":"2024-02-10T08:42:12.396941+0000","last_fullsized":"2024-02-10T08:42:12.396941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":343,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050888+0000","last_change":"2024-02-10T08:41:15.304181+0000","last_active":"2024-02-10T08:42:14.050888+0000","last_peered":"2024-02-10T08:42:14.050888+0000","last_clean":"2024-02-10T08:42:14.050888+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:42:14.050888+0000","last_undegraded":"2024-02-10T08:42:14.050888+0000","last_fullsized":"2024-02-10T08:42:14.050888+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.304104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.304104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":378,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398206+0000","last_change":"2024-02-10T08:41:23.736997+0000","last_active":"2024-02-10T08:42:12.398206+0000","last_peered":"2024-02-10T08:42:12.398206+0000","last_clean":"2024-02-10T08:42:12.398206+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:42:12.398206+0000","last_undegraded":"2024-02-10T08:42:12.398206+0000","last_fullsized":"2024-02-10T08:42:12.398206+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.736922+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":409,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051177+0000","last_change":"2024-02-10T08:41:21.384611+0000","last_active":"2024-02-10T08:42:14.051177+0000","last_peered":"2024-02-10T08:42:14.051177+0000","last_clean":"2024-02-10T08:42:14.051177+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:42:14.051177+0000","last_undegraded":"2024-02-10T08:42:14.051177+0000","last_fullsized":"2024-02-10T08:42:14.051177+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":389,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051412+0000","last_change":"2024-02-10T08:40:37.417370+0000","last_active":"2024-02-10T08:42:14.051412+0000","last_peered":"2024-02-10T08:42:14.051412+0000","last_clean":"2024-02-10T08:42:14.051412+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:42:14.051412+0000","last_undegraded":"2024-02-10T08:42:14.051412+0000","last_fullsized":"2024-02-10T08:42:14.051412+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:37.417303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.417303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":337,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043424+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:42:14.043424+0000","last_peered":"2024-02-10T08:42:14.043424+0000","last_clean":"2024-02-10T08:42:14.043424+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:42:14.043424+0000","last_undegraded":"2024-02-10T08:42:14.043424+0000","last_fullsized":"2024-02-10T08:42:14.043424+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":400,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398093+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:42:12.398093+0000","last_peered":"2024-02-10T08:42:12.398093+0000","last_clean":"2024-02-10T08:42:12.398093+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:42:12.398093+0000","last_undegraded":"2024-02-10T08:42:12.398093+0000","last_fullsized":"2024-02-10T08:42:12.398093+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397649+0000","last_change":"2024-02-10T08:42:02.183760+0000","last_active":"2024-02-10T08:42:12.397649+0000","last_peered":"2024-02-10T08:42:12.397649+0000","last_clean":"2024-02-10T08:42:12.397649+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:42:12.397649+0000","last_undegraded":"2024-02-10T08:42:12.397649+0000","last_fullsized":"2024-02-10T08:42:12.397649+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367826+0000","last_change":"2024-02-10T08:40:36.384024+0000","last_active":"2024-02-10T08:42:12.367826+0000","last_peered":"2024-02-10T08:42:12.367826+0000","last_clean":"2024-02-10T08:42:12.367826+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:42:12.367826+0000","last_undegraded":"2024-02-10T08:42:12.367826+0000","last_fullsized":"2024-02-10T08:42:12.367826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.383958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.383958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":339,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400850+0000","last_change":"2024-02-10T08:41:23.032812+0000","last_active":"2024-02-10T08:42:12.400850+0000","last_peered":"2024-02-10T08:42:12.400850+0000","last_clean":"2024-02-10T08:42:12.400850+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:42:12.400850+0000","last_undegraded":"2024-02-10T08:42:12.400850+0000","last_fullsized":"2024-02-10T08:42:12.400850+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":406,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398244+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:42:12.398244+0000","last_peered":"2024-02-10T08:42:12.398244+0000","last_clean":"2024-02-10T08:42:12.398244+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:42:12.398244+0000","last_undegraded":"2024-02-10T08:42:12.398244+0000","last_fullsized":"2024-02-10T08:42:12.398244+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398598+0000","last_change":"2024-02-10T08:40:35.021264+0000","last_active":"2024-02-10T08:42:12.398598+0000","last_peered":"2024-02-10T08:42:12.398598+0000","last_clean":"2024-02-10T08:42:12.398598+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:42:12.398598+0000","last_undegraded":"2024-02-10T08:42:12.398598+0000","last_fullsized":"2024-02-10T08:42:12.398598+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.021169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.021169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":515,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400642+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:42:12.400642+0000","last_peered":"2024-02-10T08:42:12.400642+0000","last_clean":"2024-02-10T08:42:12.400642+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:42:12.400642+0000","last_undegraded":"2024-02-10T08:42:12.400642+0000","last_fullsized":"2024-02-10T08:42:12.400642+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":309,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367932+0000","last_change":"2024-02-10T08:41:16.497083+0000","last_active":"2024-02-10T08:42:12.367932+0000","last_peered":"2024-02-10T08:42:12.367932+0000","last_clean":"2024-02-10T08:42:12.367932+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:42:12.367932+0000","last_undegraded":"2024-02-10T08:42:12.367932+0000","last_fullsized":"2024-02-10T08:42:12.367932+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.496995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.496995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":310,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397584+0000","last_change":"2024-02-10T08:41:21.463600+0000","last_active":"2024-02-10T08:42:12.397584+0000","last_peered":"2024-02-10T08:42:12.397584+0000","last_clean":"2024-02-10T08:42:12.397584+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:42:12.397584+0000","last_undegraded":"2024-02-10T08:42:12.397584+0000","last_fullsized":"2024-02-10T08:42:12.397584+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.463527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.463527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":410,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400503+0000","last_change":"2024-02-10T08:42:05.106561+0000","last_active":"2024-02-10T08:42:12.400503+0000","last_peered":"2024-02-10T08:42:12.400503+0000","last_clean":"2024-02-10T08:42:12.400503+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:42:12.400503+0000","last_undegraded":"2024-02-10T08:42:12.400503+0000","last_fullsized":"2024-02-10T08:42:12.400503+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":307,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043422+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:42:14.043422+0000","last_peered":"2024-02-10T08:42:14.043422+0000","last_clean":"2024-02-10T08:42:14.043422+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:42:14.043422+0000","last_undegraded":"2024-02-10T08:42:14.043422+0000","last_fullsized":"2024-02-10T08:42:14.043422+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":504,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398346+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:42:12.398346+0000","last_peered":"2024-02-10T08:42:12.398346+0000","last_clean":"2024-02-10T08:42:12.398346+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:42:12.398346+0000","last_undegraded":"2024-02-10T08:42:12.398346+0000","last_fullsized":"2024-02-10T08:42:12.398346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044403+0000","last_change":"2024-02-10T08:40:41.538200+0000","last_active":"2024-02-10T08:42:14.044403+0000","last_peered":"2024-02-10T08:42:14.044403+0000","last_clean":"2024-02-10T08:42:14.044403+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:42:14.044403+0000","last_undegraded":"2024-02-10T08:42:14.044403+0000","last_fullsized":"2024-02-10T08:42:14.044403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:41.538139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.538139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":351,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050596+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:42:14.050596+0000","last_peered":"2024-02-10T08:42:14.050596+0000","last_clean":"2024-02-10T08:42:14.050596+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:42:14.050596+0000","last_undegraded":"2024-02-10T08:42:14.050596+0000","last_fullsized":"2024-02-10T08:42:14.050596+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":415,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048538+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:42:14.048538+0000","last_peered":"2024-02-10T08:42:14.048538+0000","last_clean":"2024-02-10T08:42:14.048538+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:42:14.048538+0000","last_undegraded":"2024-02-10T08:42:14.048538+0000","last_fullsized":"2024-02-10T08:42:14.048538+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397773+0000","last_change":"2024-02-10T08:41:44.355018+0000","last_active":"2024-02-10T08:42:12.397773+0000","last_peered":"2024-02-10T08:42:12.397773+0000","last_clean":"2024-02-10T08:42:12.397773+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:42:12.397773+0000","last_undegraded":"2024-02-10T08:42:12.397773+0000","last_fullsized":"2024-02-10T08:42:12.397773+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:44.354945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.354945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":403,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397993+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:42:12.397993+0000","last_peered":"2024-02-10T08:42:12.397993+0000","last_clean":"2024-02-10T08:42:12.397993+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:42:12.397993+0000","last_undegraded":"2024-02-10T08:42:12.397993+0000","last_fullsized":"2024-02-10T08:42:12.397993+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":309,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397717+0000","last_change":"2024-02-10T08:41:12.929709+0000","last_active":"2024-02-10T08:42:12.397717+0000","last_peered":"2024-02-10T08:42:12.397717+0000","last_clean":"2024-02-10T08:42:12.397717+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:42:12.397717+0000","last_undegraded":"2024-02-10T08:42:12.397717+0000","last_fullsized":"2024-02-10T08:42:12.397717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.929592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.929592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":349,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398268+0000","last_change":"2024-02-10T08:41:20.506111+0000","last_active":"2024-02-10T08:42:12.398268+0000","last_peered":"2024-02-10T08:42:12.398268+0000","last_clean":"2024-02-10T08:42:12.398268+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:42:12.398268+0000","last_undegraded":"2024-02-10T08:42:12.398268+0000","last_fullsized":"2024-02-10T08:42:12.398268+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.506032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.506032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":411,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043332+0000","last_change":"2024-02-10T08:41:05.682672+0000","last_active":"2024-02-10T08:42:14.043332+0000","last_peered":"2024-02-10T08:42:14.043332+0000","last_clean":"2024-02-10T08:42:14.043332+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:42:14.043332+0000","last_undegraded":"2024-02-10T08:42:14.043332+0000","last_fullsized":"2024-02-10T08:42:14.043332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:05.682577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.682577+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":411,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049373+0000","last_change":"2024-02-10T08:41:56.648067+0000","last_active":"2024-02-10T08:42:14.049373+0000","last_peered":"2024-02-10T08:42:14.049373+0000","last_clean":"2024-02-10T08:42:14.049373+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:42:14.049373+0000","last_undegraded":"2024-02-10T08:42:14.049373+0000","last_fullsized":"2024-02-10T08:42:14.049373+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.647993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":391,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048586+0000","last_change":"2024-02-10T08:40:36.354179+0000","last_active":"2024-02-10T08:42:14.048586+0000","last_peered":"2024-02-10T08:42:14.048586+0000","last_clean":"2024-02-10T08:42:14.048586+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:42:14.048586+0000","last_undegraded":"2024-02-10T08:42:14.048586+0000","last_fullsized":"2024-02-10T08:42:14.048586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.354073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397324+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:42:12.397324+0000","last_peered":"2024-02-10T08:42:12.397324+0000","last_clean":"2024-02-10T08:42:12.397324+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:42:12.397324+0000","last_undegraded":"2024-02-10T08:42:12.397324+0000","last_fullsized":"2024-02-10T08:42:12.397324+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":346,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397978+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:42:12.397978+0000","last_peered":"2024-02-10T08:42:12.397978+0000","last_clean":"2024-02-10T08:42:12.397978+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:42:12.397978+0000","last_undegraded":"2024-02-10T08:42:12.397978+0000","last_fullsized":"2024-02-10T08:42:12.397978+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":397,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397208+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:42:12.397208+0000","last_peered":"2024-02-10T08:42:12.397208+0000","last_clean":"2024-02-10T08:42:12.397208+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:42:12.397208+0000","last_undegraded":"2024-02-10T08:42:12.397208+0000","last_fullsized":"2024-02-10T08:42:12.397208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400401+0000","last_change":"2024-02-10T08:41:05.909459+0000","last_active":"2024-02-10T08:42:12.400401+0000","last_peered":"2024-02-10T08:42:12.400401+0000","last_clean":"2024-02-10T08:42:12.400401+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:42:12.400401+0000","last_undegraded":"2024-02-10T08:42:12.400401+0000","last_fullsized":"2024-02-10T08:42:12.400401+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.909358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":395,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397570+0000","last_change":"2024-02-10T08:41:24.469189+0000","last_active":"2024-02-10T08:42:12.397570+0000","last_peered":"2024-02-10T08:42:12.397570+0000","last_clean":"2024-02-10T08:42:12.397570+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:42:12.397570+0000","last_undegraded":"2024-02-10T08:42:12.397570+0000","last_fullsized":"2024-02-10T08:42:12.397570+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.469114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.469114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.366812+0000","last_change":"2024-02-10T08:41:03.331622+0000","last_active":"2024-02-10T08:42:12.366812+0000","last_peered":"2024-02-10T08:42:12.366812+0000","last_clean":"2024-02-10T08:42:12.366812+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:42:12.366812+0000","last_undegraded":"2024-02-10T08:42:12.366812+0000","last_fullsized":"2024-02-10T08:42:12.366812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":337,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398466+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:42:12.398466+0000","last_peered":"2024-02-10T08:42:12.398466+0000","last_clean":"2024-02-10T08:42:12.398466+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:42:12.398466+0000","last_undegraded":"2024-02-10T08:42:12.398466+0000","last_fullsized":"2024-02-10T08:42:12.398466+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":410,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397546+0000","last_change":"2024-02-10T08:41:03.770782+0000","last_active":"2024-02-10T08:42:12.397546+0000","last_peered":"2024-02-10T08:42:12.397546+0000","last_clean":"2024-02-10T08:42:12.397546+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:42:12.397546+0000","last_undegraded":"2024-02-10T08:42:12.397546+0000","last_fullsized":"2024-02-10T08:42:12.397546+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399128+0000","last_change":"2024-02-10T08:41:52.364114+0000","last_active":"2024-02-10T08:42:12.399128+0000","last_peered":"2024-02-10T08:42:12.399128+0000","last_clean":"2024-02-10T08:42:12.399128+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:42:12.399128+0000","last_undegraded":"2024-02-10T08:42:12.399128+0000","last_fullsized":"2024-02-10T08:42:12.399128+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:52.364044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:52.364044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367299+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:42:12.367299+0000","last_peered":"2024-02-10T08:42:12.367299+0000","last_clean":"2024-02-10T08:42:12.367299+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:42:12.367299+0000","last_undegraded":"2024-02-10T08:42:12.367299+0000","last_fullsized":"2024-02-10T08:42:12.367299+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398086+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:42:12.398086+0000","last_peered":"2024-02-10T08:42:12.398086+0000","last_clean":"2024-02-10T08:42:12.398086+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:42:12.398086+0000","last_undegraded":"2024-02-10T08:42:12.398086+0000","last_fullsized":"2024-02-10T08:42:12.398086+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":310,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400108+0000","last_change":"2024-02-10T08:41:12.525969+0000","last_active":"2024-02-10T08:42:12.400108+0000","last_peered":"2024-02-10T08:42:12.400108+0000","last_clean":"2024-02-10T08:42:12.400108+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:42:12.400108+0000","last_undegraded":"2024-02-10T08:42:12.400108+0000","last_fullsized":"2024-02-10T08:42:12.400108+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.525896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.525896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":348,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.368156+0000","last_change":"2024-02-10T08:41:53.573192+0000","last_active":"2024-02-10T08:42:12.368156+0000","last_peered":"2024-02-10T08:42:12.368156+0000","last_clean":"2024-02-10T08:42:12.368156+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:42:12.368156+0000","last_undegraded":"2024-02-10T08:42:12.368156+0000","last_fullsized":"2024-02-10T08:42:12.368156+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:53.573125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.573125+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397620+0000","last_change":"2024-02-10T08:40:27.813479+0000","last_active":"2024-02-10T08:42:12.397620+0000","last_peered":"2024-02-10T08:42:12.397620+0000","last_clean":"2024-02-10T08:42:12.397620+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:42:12.397620+0000","last_undegraded":"2024-02-10T08:42:12.397620+0000","last_fullsized":"2024-02-10T08:42:12.397620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.813396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.813396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":386,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.366843+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:42:12.366843+0000","last_peered":"2024-02-10T08:42:12.366843+0000","last_clean":"2024-02-10T08:42:12.366843+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:42:12.366843+0000","last_undegraded":"2024-02-10T08:42:12.366843+0000","last_fullsized":"2024-02-10T08:42:12.366843+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":343,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049905+0000","last_change":"2024-02-10T08:41:48.214886+0000","last_active":"2024-02-10T08:42:14.049905+0000","last_peered":"2024-02-10T08:42:14.049905+0000","last_clean":"2024-02-10T08:42:14.049905+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:42:14.049905+0000","last_undegraded":"2024-02-10T08:42:14.049905+0000","last_fullsized":"2024-02-10T08:42:14.049905+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.214822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.214822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":419,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399277+0000","last_change":"2024-02-10T08:41:46.403355+0000","last_active":"2024-02-10T08:42:12.399277+0000","last_peered":"2024-02-10T08:42:12.399277+0000","last_clean":"2024-02-10T08:42:12.399277+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:42:12.399277+0000","last_undegraded":"2024-02-10T08:42:12.399277+0000","last_fullsized":"2024-02-10T08:42:12.399277+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:46.403281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.403281+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":407,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397405+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:42:12.397405+0000","last_peered":"2024-02-10T08:42:12.397405+0000","last_clean":"2024-02-10T08:42:12.397405+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:42:12.397405+0000","last_undegraded":"2024-02-10T08:42:12.397405+0000","last_fullsized":"2024-02-10T08:42:12.397405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":394,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.049265+0000","last_change":"2024-02-10T08:41:49.644781+0000","last_active":"2024-02-10T08:42:14.049265+0000","last_peered":"2024-02-10T08:42:14.049265+0000","last_clean":"2024-02-10T08:42:14.049265+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:42:14.049265+0000","last_undegraded":"2024-02-10T08:42:14.049265+0000","last_fullsized":"2024-02-10T08:42:14.049265+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":343,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048894+0000","last_change":"2024-02-10T08:41:23.662181+0000","last_active":"2024-02-10T08:42:14.048894+0000","last_peered":"2024-02-10T08:42:14.048894+0000","last_clean":"2024-02-10T08:42:14.048894+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:42:14.048894+0000","last_undegraded":"2024-02-10T08:42:14.048894+0000","last_fullsized":"2024-02-10T08:42:14.048894+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":430,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.051603+0000","last_change":"2024-02-10T08:41:45.260263+0000","last_active":"2024-02-10T08:42:14.051603+0000","last_peered":"2024-02-10T08:42:14.051603+0000","last_clean":"2024-02-10T08:42:14.051603+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:42:14.051603+0000","last_undegraded":"2024-02-10T08:42:14.051603+0000","last_fullsized":"2024-02-10T08:42:14.051603+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":397,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398386+0000","last_change":"2024-02-10T08:41:08.533312+0000","last_active":"2024-02-10T08:42:12.398386+0000","last_peered":"2024-02-10T08:42:12.398386+0000","last_clean":"2024-02-10T08:42:12.398386+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:42:12.398386+0000","last_undegraded":"2024-02-10T08:42:12.398386+0000","last_fullsized":"2024-02-10T08:42:12.398386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.533249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.533249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.044403+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:42:14.044403+0000","last_peered":"2024-02-10T08:42:14.044403+0000","last_clean":"2024-02-10T08:42:14.044403+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:42:14.044403+0000","last_undegraded":"2024-02-10T08:42:14.044403+0000","last_fullsized":"2024-02-10T08:42:14.044403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":315,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397504+0000","last_change":"2024-02-10T08:41:56.265678+0000","last_active":"2024-02-10T08:42:12.397504+0000","last_peered":"2024-02-10T08:42:12.397504+0000","last_clean":"2024-02-10T08:42:12.397504+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:42:12.397504+0000","last_undegraded":"2024-02-10T08:42:12.397504+0000","last_fullsized":"2024-02-10T08:42:12.397504+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.265562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.265562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":440,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398100+0000","last_change":"2024-02-10T08:41:53.828692+0000","last_active":"2024-02-10T08:42:12.398100+0000","last_peered":"2024-02-10T08:42:12.398100+0000","last_clean":"2024-02-10T08:42:12.398100+0000","last_became_active":"2024-02-10T08:40:42.014594+0000","last_became_peered":"2024-02-10T08:40:42.014594+0000","last_unstale":"2024-02-10T08:42:12.398100+0000","last_undegraded":"2024-02-10T08:42:12.398100+0000","last_fullsized":"2024-02-10T08:42:12.398100+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":398,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397988+0000","last_change":"2024-02-10T08:41:57.957397+0000","last_active":"2024-02-10T08:42:12.397988+0000","last_peered":"2024-02-10T08:42:12.397988+0000","last_clean":"2024-02-10T08:42:12.397988+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:42:12.397988+0000","last_undegraded":"2024-02-10T08:42:12.397988+0000","last_fullsized":"2024-02-10T08:42:12.397988+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398243+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:42:12.398243+0000","last_peered":"2024-02-10T08:42:12.398243+0000","last_clean":"2024-02-10T08:42:12.398243+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:42:12.398243+0000","last_undegraded":"2024-02-10T08:42:12.398243+0000","last_fullsized":"2024-02-10T08:42:12.398243+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":343,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.048482+0000","last_change":"2024-02-10T08:42:01.674000+0000","last_active":"2024-02-10T08:42:14.048482+0000","last_peered":"2024-02-10T08:42:14.048482+0000","last_clean":"2024-02-10T08:42:14.048482+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:42:14.048482+0000","last_undegraded":"2024-02-10T08:42:14.048482+0000","last_fullsized":"2024-02-10T08:42:14.048482+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:01.673932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.673932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":432,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043747+0000","last_change":"2024-02-10T08:41:19.765286+0000","last_active":"2024-02-10T08:42:14.043747+0000","last_peered":"2024-02-10T08:42:14.043747+0000","last_clean":"2024-02-10T08:42:14.043747+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:42:14.043747+0000","last_undegraded":"2024-02-10T08:42:14.043747+0000","last_fullsized":"2024-02-10T08:42:14.043747+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367638+0000","last_change":"2024-02-10T08:41:13.452076+0000","last_active":"2024-02-10T08:42:12.367638+0000","last_peered":"2024-02-10T08:42:12.367638+0000","last_clean":"2024-02-10T08:42:12.367638+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:42:12.367638+0000","last_undegraded":"2024-02-10T08:42:12.367638+0000","last_fullsized":"2024-02-10T08:42:12.367638+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.451989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.451989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":313,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.047925+0000","last_change":"2024-02-10T08:41:22.620317+0000","last_active":"2024-02-10T08:42:14.047925+0000","last_peered":"2024-02-10T08:42:14.047925+0000","last_clean":"2024-02-10T08:42:14.047925+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:42:14.047925+0000","last_undegraded":"2024-02-10T08:42:14.047925+0000","last_fullsized":"2024-02-10T08:42:14.047925+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.620255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.620255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":347,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.050551+0000","last_change":"2024-02-10T08:40:22.359771+0000","last_active":"2024-02-10T08:42:14.050551+0000","last_peered":"2024-02-10T08:42:14.050551+0000","last_clean":"2024-02-10T08:42:14.050551+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:42:14.050551+0000","last_undegraded":"2024-02-10T08:42:14.050551+0000","last_fullsized":"2024-02-10T08:42:14.050551+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.359703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.359703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":409,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043433+0000","last_change":"2024-02-10T08:42:03.646444+0000","last_active":"2024-02-10T08:42:14.043433+0000","last_peered":"2024-02-10T08:42:14.043433+0000","last_clean":"2024-02-10T08:42:14.043433+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:42:14.043433+0000","last_undegraded":"2024-02-10T08:42:14.043433+0000","last_fullsized":"2024-02-10T08:42:14.043433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.367871+0000","last_change":"2024-02-10T08:41:11.401522+0000","last_active":"2024-02-10T08:42:12.367871+0000","last_peered":"2024-02-10T08:42:12.367871+0000","last_clean":"2024-02-10T08:42:12.367871+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:42:12.367871+0000","last_undegraded":"2024-02-10T08:42:12.367871+0000","last_fullsized":"2024-02-10T08:42:12.367871+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.401433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.401433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397129+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:42:12.397129+0000","last_peered":"2024-02-10T08:42:12.397129+0000","last_clean":"2024-02-10T08:42:12.397129+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:42:12.397129+0000","last_undegraded":"2024-02-10T08:42:12.397129+0000","last_fullsized":"2024-02-10T08:42:12.397129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":318,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.400890+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:42:12.400890+0000","last_peered":"2024-02-10T08:42:12.400890+0000","last_clean":"2024-02-10T08:42:12.400890+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:42:12.400890+0000","last_undegraded":"2024-02-10T08:42:12.400890+0000","last_fullsized":"2024-02-10T08:42:12.400890+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":427,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043154+0000","last_change":"2024-02-10T08:40:37.664677+0000","last_active":"2024-02-10T08:42:14.043154+0000","last_peered":"2024-02-10T08:42:14.043154+0000","last_clean":"2024-02-10T08:42:14.043154+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:42:14.043154+0000","last_undegraded":"2024-02-10T08:42:14.043154+0000","last_fullsized":"2024-02-10T08:42:14.043154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:40:37.664586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.664586+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.398105+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:42:12.398105+0000","last_peered":"2024-02-10T08:42:12.398105+0000","last_clean":"2024-02-10T08:42:12.398105+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:42:12.398105+0000","last_undegraded":"2024-02-10T08:42:12.398105+0000","last_fullsized":"2024-02-10T08:42:12.398105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":393,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-10T08:42:14.043609+0000","last_change":"2024-02-10T08:41:55.669868+0000","last_active":"2024-02-10T08:42:14.043609+0000","last_peered":"2024-02-10T08:42:14.043609+0000","last_clean":"2024-02-10T08:42:14.043609+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:42:14.043609+0000","last_undegraded":"2024-02-10T08:42:14.043609+0000","last_fullsized":"2024-02-10T08:42:14.043609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":348,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399187+0000","last_change":"2024-02-10T08:41:16.004246+0000","last_active":"2024-02-10T08:42:12.399187+0000","last_peered":"2024-02-10T08:42:12.399187+0000","last_clean":"2024-02-10T08:42:12.399187+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:42:12.399187+0000","last_undegraded":"2024-02-10T08:42:12.399187+0000","last_fullsized":"2024-02-10T08:42:12.399187+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":441,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397616+0000","last_change":"2024-02-10T08:41:19.653285+0000","last_active":"2024-02-10T08:42:12.397616+0000","last_peered":"2024-02-10T08:42:12.397616+0000","last_clean":"2024-02-10T08:42:12.397616+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:42:12.397616+0000","last_undegraded":"2024-02-10T08:42:12.397616+0000","last_fullsized":"2024-02-10T08:42:12.397616+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:41:19.653219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.653219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":309,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.368301+0000","last_change":"2024-02-10T08:40:42.336046+0000","last_active":"2024-02-10T08:42:12.368301+0000","last_peered":"2024-02-10T08:42:12.368301+0000","last_clean":"2024-02-10T08:42:12.368301+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:42:12.368301+0000","last_undegraded":"2024-02-10T08:42:12.368301+0000","last_fullsized":"2024-02-10T08:42:12.368301+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.335958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.335958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.397836+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:42:12.397836+0000","last_peered":"2024-02-10T08:42:12.397836+0000","last_clean":"2024-02-10T08:42:12.397836+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:42:12.397836+0000","last_undegraded":"2024-02-10T08:42:12.397836+0000","last_fullsized":"2024-02-10T08:42:12.397836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":345,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-10T08:42:12.399796+0000","last_change":"2024-02-10T08:42:05.201384+0000","last_active":"2024-02-10T08:42:12.399796+0000","last_peered":"2024-02-10T08:42:12.399796+0000","last_clean":"2024-02-10T08:42:12.399796+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:42:12.399796+0000","last_undegraded":"2024-02-10T08:42:12.399796+0000","last_fullsized":"2024-02-10T08:42:12.399796+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":212,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":11,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001479,"num_objects":158,"num_object_clones":2,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13901824,"data_stored":12004445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1875,"ondisk_log_size":1875,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":531,"num_read_kb":526,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299502,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307984,"kb_used_data":2568,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437168,"statfs":{"total":95995035648,"available":95679660032,"internally_reserved":0,"allocated":2629632,"data_stored":975154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.78300000000000003,"15min":0.60499999999999998},"min":{"1min":0.374,"5min":0.246,"15min":0.246},"max":{"1min":0.99299999999999999,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.66700000000000004,"15min":0.56899999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93100000000000005,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.627,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.314,"15min":0.24299999999999999},"max":{"1min":1.0509999999999999,"5min":7.593,"15min":7.593},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65400000000000003,"15min":0.58299999999999996},"min":{"1min":0.35899999999999999,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":1.0089999999999999,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.377}]},{"osd":2,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.61699999999999999,"15min":0.54500000000000004},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27100000000000002},"max":{"1min":0.73199999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.83299999999999996,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.251,"15min":0.251},"max":{"1min":1.0740000000000001,"5min":19.369,"15min":19.369},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67300000000000004,"15min":0.56499999999999995},"min":{"1min":0.38400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.083,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.84399999999999997,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.33100000000000002,"15min":0.27900000000000003},"max":{"1min":1.1459999999999999,"5min":22.102,"15min":22.102},"last":0.52900000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.45000000000000001},"min":{"1min":0.34899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":8.827,"15min":8.827},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56200000000000006,"15min":0.47799999999999998},"min":{"1min":0.28799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72399999999999998,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.66600000000000004,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.77600000000000002,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.496},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77200000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.69399999999999995,"15min":0.56499999999999995},"min":{"1min":0.436,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76300000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.435},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.5,"15min":0.503},"min":{"1min":0.39300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72599999999999998,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.52100000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561140,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307708,"kb_used_data":6388,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437444,"statfs":{"total":95995035648,"available":95679942656,"internally_reserved":0,"allocated":6541312,"data_stored":4974916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.19800000000000001},"max":{"1min":0.68200000000000005,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.64600000000000002,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.65000000000000002,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.57799999999999996,"15min":0.52600000000000002},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.501},{"interface":"front","average":{"1min":0.501,"5min":0.56599999999999995,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.69499999999999995,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.439}]},{"osd":2,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58999999999999997,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72299999999999998,"5min":4.798,"15min":4.798},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59699999999999998,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68999999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.78100000000000003,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":16.823,"15min":16.823},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.51800000000000002},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":0.67400000000000004,"5min":3.657,"15min":3.657},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45200000000000001,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":0.71099999999999997,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.33900000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71199999999999997,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46700000000000003,"15min":0.436},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.38300000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.47699999999999998},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70099999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.41599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303860,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441292,"statfs":{"total":95995035648,"available":95683883008,"internally_reserved":0,"allocated":2600960,"data_stored":976173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.48299999999999998,"15min":0.40699999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.36799999999999999,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.63,"15min":0.45500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.45600000000000002,"5min":16.166,"15min":16.166},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.103,"15min":0.61299999999999999},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47599999999999998,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.255},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.85699999999999998,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.496,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.318}]},{"osd":3,"last update":"Sat Feb 10 08:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.73299999999999998,"15min":0.51700000000000002},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.39800000000000002,"5min":24.773,"15min":24.773},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.69599999999999995,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.499,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.32500000000000001,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63700000000000001,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.76800000000000002,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.35399999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.65700000000000003,"15min":0.60699999999999998},"min":{"1min":0.41799999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":9.532,"15min":9.532},"last":0.372},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.73399999999999999,"15min":0.65500000000000003},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.71499999999999997,"15min":0.60199999999999998},"min":{"1min":0.44400000000000001,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.74399999999999999,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.83399999999999996,"15min":0.66800000000000004},"min":{"1min":0.44,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.71999999999999997,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.432}]}]},{"osd":0,"up_from":0,"seq":34359738513,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299768,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445384,"statfs":{"total":95995035648,"available":95688073216,"internally_reserved":0,"allocated":2605056,"data_stored":974921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.502,"15min":0.40899999999999997},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.42499999999999999,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.311,"5min":0.64700000000000002,"15min":0.46000000000000002},"min":{"1min":0.20599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.44800000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.373}]},{"osd":1,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.68500000000000005,"15min":0.47399999999999998},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":12.512,"15min":12.512},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.623,"15min":0.46899999999999997},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":0.46600000000000003,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:41:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.48999999999999999,"15min":0.441},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47199999999999998,"5min":4.617,"15min":4.617},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.57099999999999995,"15min":0.48699999999999999},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.22600000000000001},"max":{"1min":0.503,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:41:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.87,"15min":0.67900000000000005},"min":{"1min":0.36399999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91500000000000004,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.77100000000000002,"15min":0.65700000000000003},"min":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":0.87,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.84199999999999997,"15min":0.66000000000000003},"min":{"1min":0.40200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.80000000000000004,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.69699999999999995,"15min":0.622},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.753,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.67000000000000004,"15min":0.66200000000000003},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84099999999999997,"5min":4.3090000000000002,"15min":9.266},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.71299999999999997,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81999999999999995,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:41:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.81299999999999994,"15min":0.68999999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.75900000000000001,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81799999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.49099999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182531,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303672,"kb_used_data":6448,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441480,"statfs":{"total":95995035648,"available":95684075520,"internally_reserved":0,"allocated":6602752,"data_stored":4975056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.67000000000000004,"15min":0.50800000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.52100000000000002,"5min":11.550000000000001,"15min":14.744},"last":0.214},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.85299999999999998,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":0.504,"5min":11.808,"15min":11.808},"last":0.24099999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.747,"15min":0.56599999999999995},"min":{"1min":0.245,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":0.435,"5min":10.444000000000001,"15min":16.634},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.66900000000000004,"15min":0.56499999999999995},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.754,"5min":11.494999999999999,"15min":12.438000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.65300000000000002,"15min":0.58699999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.51000000000000001,"5min":11.516999999999999,"15min":16.536000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.85299999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.48399999999999999,"5min":17.722000000000001,"15min":23.294},"last":0.29399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:42:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.67900000000000005},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":1.9370000000000001,"5min":2.589,"15min":12.371},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.79700000000000004,"15min":0.71899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.008,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.004,"15min":0.71199999999999997},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.72699999999999998,"15min":0.65500000000000003},"min":{"1min":0.45100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.873,"5min":11.448,"15min":11.448},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.749,"15min":0.69799999999999995},"min":{"1min":0.41099999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77900000000000003,"5min":10.381,"15min":16.091000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":11.472,"15min":16.475000000000001},"last":0.45200000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.88,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81699999999999995,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.439},{"interface":"front","average":{"1min":0.60599999999999998,"5min":1.083,"15min":0.72999999999999998},"min":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75800000000000001,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.52600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986301,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307968,"kb_used_data":2552,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437184,"statfs":{"total":95995035648,"available":95679676416,"internally_reserved":0,"allocated":2613248,"data_stored":975185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.85199999999999998,"15min":0.63600000000000001},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73799999999999999,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.66300000000000003,"15min":0.56000000000000005},"min":{"1min":0.33000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.754,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.71299999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.73699999999999999,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65500000000000003,"5min":5.774,"15min":5.774},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.74399999999999999,"15min":0.60499999999999998},"min":{"1min":0.42199999999999999,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73199999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.67900000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.75800000000000001,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72799999999999998,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.76200000000000001,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72099999999999997,"5min":10.01,"15min":10.01},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.92100000000000004,"15min":0.66000000000000003},"min":{"1min":0.34999999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.1000000000000001,"15min":0.71899999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":25.27,"15min":25.27},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.752,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46999999999999997,"15min":0.42399999999999999},"min":{"1min":0.33000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.76400000000000001,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.78000000000000003,"5min":8.3119999999999994,"15min":13.342000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.51100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":13.753,"15min":13.753},"last":0.45400000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.61799999999999999,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67700000000000005,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.68000000000000005,"15min":0.58199999999999996},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74199999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.55000000000000004}]}]},{"osd":5,"up_from":37,"seq":158913790073,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311804,"kb_used_data":6388,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433348,"statfs":{"total":95995035648,"available":95675748352,"internally_reserved":0,"allocated":6541312,"data_stored":4974695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":1.968,"15min":0.995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.78200000000000003,"5min":107.551,"15min":107.551},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.853,"15min":0.96099999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":0.77000000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.7050000000000001,"15min":0.93799999999999994},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71399999999999997,"5min":107.205,"15min":107.205},"last":0.311},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.3700000000000001,"15min":0.82799999999999996},"min":{"1min":0.33600000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.74299999999999999,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.48199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.8460000000000001,"15min":0.95399999999999996},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.9259999999999999,"15min":0.995},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.151,"15min":0.72299999999999998},"min":{"1min":0.374,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.008,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":2.5139999999999998,"15min":1.206},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":114.526,"15min":114.526},"last":0.441}]},{"osd":4,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":1.149,"15min":0.66200000000000003},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.72399999999999998,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.88500000000000001,"15min":0.57099999999999995},"min":{"1min":0.32800000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.68000000000000005,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.68600000000000005},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":72.241},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.46300000000000002,"15min":0.54800000000000004},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.175,"15min":0.71699999999999997},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86199999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.2370000000000001,"15min":0.748},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.90700000000000003,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.51100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4000643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:42:18.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:42:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:18.985 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:18.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:19.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:42:19.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:19.114 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:19.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:19.234 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:42:19.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:19.283 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:19.433 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:19.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:19.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:18.040022+0000 mgr.y (mgr.14553) 382 : audit [DBG] from='client.53660 v1:172.21.15.107:0/2693081390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:19.276563+0000 mon.b (mon.1) 584 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:19.276913+0000 mon.a (mon.0) 2499 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:19.277438+0000 mon.b (mon.1) 585 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:19.277691+0000 mon.a (mon.0) 2500 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:19.565 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:19.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:18.040022+0000 mgr.y (mgr.14553) 382 : audit [DBG] from='client.53660 v1:172.21.15.107:0/2693081390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:19.870 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:19.276563+0000 mon.b (mon.1) 584 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:19.276913+0000 mon.a (mon.0) 2499 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:19.277438+0000 mon.b (mon.1) 585 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:19.277691+0000 mon.a (mon.0) 2500 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:18.040022+0000 mgr.y (mgr.14553) 382 : audit [DBG] from='client.53660 v1:172.21.15.107:0/2693081390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:19.276563+0000 mon.b (mon.1) 584 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:19.276913+0000 mon.a (mon.0) 2499 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:19.277438+0000 mon.b (mon.1) 585 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:19.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:19.277691+0000 mon.a (mon.0) 2500 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-61"}]: dispatch 2024-02-10T08:42:20.387 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:20.445 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:20.699 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:19.445855+0000 mgr.y (mgr.14553) 383 : cluster [DBG] pgmap v626: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-02-10T08:42:20.699 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:19.506826+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-10T08:42:20.719 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:20.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:20.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:19.445855+0000 mgr.y (mgr.14553) 383 : cluster [DBG] pgmap v626: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-02-10T08:42:20.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:19.506826+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-10T08:42:20.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:19.445855+0000 mgr.y (mgr.14553) 383 : cluster [DBG] pgmap v626: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 691 B/s rd, 0 op/s 2024-02-10T08:42:20.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:19.506826+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-10T08:42:20.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:20.854 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:20.914 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:20.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:20.940 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:42:20.940 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:42:20.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:42:20.940 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:3 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e117)", 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 330.03045589999999, 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003, 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.941 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907503+0000", 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1326e-05 2024-02-10T08:42:20.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907514+0000", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0261999999999999e-05 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907524+0000", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5769999999999999e-06 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907531+0000", 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046026000000000001 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912134+0000", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131586 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912265+0000", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156537 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912422+0000", 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046267600000000003 2024-02-10T08:42:20.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912884+0000", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097266900000000001 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.913857+0000", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035933647999999999 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949791+0000", 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0361e-05 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949851+0000", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0658999999999998e-05 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949892+0000", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:5 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected e118)", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:42:20.946 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 328.99233842799998, 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006, 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 5 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945621+0000", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3675e-05 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945634+0000", 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1625e-05 2024-02-10T08:42:20.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945656+0000", 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2959999999999999e-06 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945664+0000", 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027177170000000001 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948382+0000", 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9475000000000006e-05 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948472+0000", 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202852 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948674+0000", 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010601493 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959276+0000", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075265439999999996 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.966802+0000", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054018193999999999 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020821+0000", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5893999999999999e-05 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020857+0000", 2024-02-10T08:42:20.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9193999999999999e-05 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020886+0000", 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:787 5.9 5.84ada7c9 (undecoded) ondisk+write+known_if_redirected e121)", 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.915047663, 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001, 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.952 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 787 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022911+0000", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4569999999999994e-06 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022920+0000", 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3113e-05 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022933+0000", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9019999999999999e-06 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022941+0000", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011509515999999999 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034451+0000", 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3540000000000006e-06 2024-02-10T08:42:20.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034460+0000", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.018581161 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053041+0000", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2311e-05 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053064+0000", 2024-02-10T08:42:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:10 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e121)", 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.90520998400001, 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002, 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:42:20.957 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 10 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032749+0000", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1640000000000004e-06 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032756+0000", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.67e-06 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032765+0000", 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6840000000000002e-06 2024-02-10T08:42:20.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032774+0000", 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073123999999999997 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033505+0000", 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4190000000000005e-05 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033599+0000", 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016687399999999999 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033766+0000", 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01101619 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.044782+0000", 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011076585 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055859+0000", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01553619 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071395+0000", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8683999999999994e-05 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071474+0000", 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7649999999999999e-05 2024-02-10T08:42:20.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071521+0000", 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:12 57.10 e122/115 57:0d24be56:test-rados-api-smithi107-112513-8::foo:head v 122'1, mlcod=0'0)", 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 324.89756568500002, 2024-02-10T08:42:20.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013632729, 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040395+0000", 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040393+0000", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2047e-05 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040405+0000", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6800000000000005e-06 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040415+0000", 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2280000000000001e-06 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.040422+0000", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012024386 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.052447+0000", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3753999999999998e-05 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.052530+0000", 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001465528 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053996+0000", 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2224999999999997e-05 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.054028+0000", 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013632729 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:256 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:42:20.967 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 307.95297534100001, 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001, 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 256 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984983+0000", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0989999999999999e-06 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984991+0000", 2024-02-10T08:42:20.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6600000000000007e-06 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985001+0000", 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5549999999999997e-06 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985008+0000", 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2148999999999998e-05 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985060+0000", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023027400000000001 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985291+0000", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033679382000000001 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.018970+0000", 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9383999999999997e-05 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.019059+0000", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061927570000000001 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025252+0000", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6524000000000003e-05 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025309+0000", 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104486 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025413+0000", 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:57 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 307.94866805800001, 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001, 2024-02-10T08:42:20.973 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 57 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989292+0000", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989298+0000", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5689999999999994e-06 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989307+0000", 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8519999999999998e-06 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989314+0000", 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2061000000000003e-05 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989356+0000", 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8673999999999999e-05 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989435+0000", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010247 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989537+0000", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000341224 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989878+0000", 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017592000000000001 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.990054+0000", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026858147999999998 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016912+0000", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4705000000000001e-05 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016947+0000", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3204999999999997e-05 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016990+0000", 2024-02-10T08:42:20.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:257 83.b 83:d4dba154:test-rados-api-smithi107-112490-12::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 307.90950700399998, 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017712418000000001, 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 257 2024-02-10T08:42:20.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028454+0000", 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028452+0000", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0000000000000002e-06 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028458+0000", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1627000000000001e-05 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028470+0000", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.899e-06 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028475+0000", 2024-02-10T08:42:20.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1108e-05 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028516+0000", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014721600000000001 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.028663+0000", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010976935 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.039640+0000", 2024-02-10T08:42:20.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9827000000000003e-05 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.039690+0000", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063247499999999996 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046014+0000", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2419000000000001e-05 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046067+0000", 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9597000000000002e-05 2024-02-10T08:42:20.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.046167+0000", 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017712418000000001 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:258 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 307.88990181000003, 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999, 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.984 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 258 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048057+0000", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4479999999999999e-06 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048064+0000", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0129000000000001e-05 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048074+0000", 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3009999999999997e-06 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048079+0000", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9587e-05 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048109+0000", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012509199999999999 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048234+0000", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010028115000000001 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058262+0000", 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9953e-05 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058322+0000", 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019508307999999999 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077830+0000", 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2973999999999995e-05 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077913+0000", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102942 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.078016+0000", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:59 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 15=[15,14] ondisk+write+ordersnap+known_if_redirected e140)", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:42:20.989 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 306.93490554200002, 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998, 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 59 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003053+0000", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1612e-05 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003064+0000", 2024-02-10T08:42:20.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4388e-05 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003079+0000", 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0430000000000008e-06 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003087+0000", 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011723245 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014810+0000", 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4276000000000003e-05 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014904+0000", 2024-02-10T08:42:20.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002265094 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.017169+0000", 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057871249999999997 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.022957+0000", 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060097370000000002 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.028966+0000", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013099348 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042066+0000", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7976000000000002e-05 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042124+0000", 2024-02-10T08:42:20.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4863000000000001e-05 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042159+0000", 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.51281.0:2 92.18 92:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 302.99579750200002, 2024-02-10T08:42:20.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998, 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.51281", 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4152427963", 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942161+0000", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3089999999999996e-06 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942170+0000", 2024-02-10T08:42:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3520000000000007e-06 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942178+0000", 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5319999999999997e-06 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942184+0000", 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001344175 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943528+0000", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5959999999999998e-06 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943535+0000", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038514623999999997 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982050+0000", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112311 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982162+0000", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014368000000000001 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982306+0000", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010079780000000001 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983314+0000", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036102999999999998 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:21.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983675+0000", 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001564005 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985239+0000", 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8597999999999998e-05 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985288+0000", 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7169999999999998e-05 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985325+0000", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52646.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 291.00182179199999, 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018858733999999999, 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52646", 2024-02-10T08:42:21.003 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4246505494", 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936139+0000", 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936137+0000", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1719999999999996e-06 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936146+0000", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3692999999999999e-05 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936160+0000", 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2989999999999997e-06 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.936168+0000", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010850230000000001 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.937253+0000", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4284999999999999e-05 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.937278+0000", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099907429999999998 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.947268+0000", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2969999999999998e-06 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.947276+0000", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061726589999999996 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953448+0000", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8418000000000006e-05 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953537+0000", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4968000000000003e-05 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.953632+0000", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00059628800000000005 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954228+0000", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8599999999999993e-06 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:21.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954236+0000", 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00071013100000000002 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954946+0000", 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6857999999999999e-05 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954973+0000", 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5301000000000001e-05 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:29.954998+0000", 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018858733999999999 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52769.0:2 114.13 114:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 275.985644751, 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01324033, 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.010 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52769", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1173755769", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952316+0000", 2024-02-10T08:42:21.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952314+0000", 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7787000000000001e-05 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952332+0000", 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0054000000000001e-05 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952342+0000", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2800000000000003e-06 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952350+0000", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00064678100000000003 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.952997+0000", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9650000000000008e-06 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.953005+0000", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084007990000000005 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961406+0000", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012829000000000001 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961534+0000", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017568099999999999 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:42:21.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.961710+0000", 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016440770000000001 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.963354+0000", 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6365999999999999e-05 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.963370+0000", 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002110373 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965481+0000", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9601000000000003e-05 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965520+0000", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6538999999999997e-05 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:44.965557+0000", 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01324033 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.016 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.2.0:352 60.0 e196/135 60:01eb2b79:test-rados-api-smithi107-112490-14::foo33:4 v 196'72, mlcod=194'68)", 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 250.90427481899999, 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010767107, 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033686+0000", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033685+0000", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1747e-05 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033697+0000", 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1090000000000003e-06 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033703+0000", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.561e-06 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033708+0000", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021149799999999999 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033919+0000", 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6986000000000001e-05 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033956+0000", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010464941 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.044421+0000", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2191999999999999e-05 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.044453+0000", 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010767107 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.020 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.2.0:353 60.0 e196/135 60:053d865c:test-rados-api-smithi107-112490-14::foo38:4 v 196'74, mlcod=194'68)", 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 250.904248881, 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011387497999999999, 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033712+0000", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.023e-06 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033717+0000", 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7820000000000001e-06 2024-02-10T08:42:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033722+0000", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4669999999999998e-06 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.033726+0000", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073303079999999998 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.041056+0000", 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3583000000000001e-05 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.041080+0000", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039962649999999997 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.045076+0000", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3532000000000001e-05 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.045100+0000", 2024-02-10T08:42:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011387497999999999 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47647.0:1 136.6s0 136:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e196)", 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 250.90204415700001, 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997, 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47647", 2024-02-10T08:42:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/993932140", 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:42:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035914+0000", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6919999999999997e-06 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035924+0000", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3081000000000001e-05 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035937+0000", 2024-02-10T08:42:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0079999999999996e-06 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035944+0000", 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1489000000000001e-05 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035995+0000", 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2339999999999999e-06 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035999+0000", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91614844299999998 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952147+0000", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116906 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952264+0000", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048505899999999998 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952749+0000", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069718000000000004 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.953446+0000", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013142239999999999 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954760+0000", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9851999999999995e-05 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954830+0000", 2024-02-10T08:42:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:1847 149.f 149:f30fcc83:test-rados-api-smithi107-112490-22::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e221)", 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 225.97850468799999, 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021778952000000001, 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1847 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959456+0000", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959454+0000", 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3935e-05 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959477+0000", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3018e-05 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959491+0000", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4710000000000007e-06 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959499+0000", 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9802999999999996e-05 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959569+0000", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8009999999999998e-06 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.959576+0000", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015099468 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.974675+0000", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001121443 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.975796+0000", 2024-02-10T08:42:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043233430000000003 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.980120+0000", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0450000000000003e-05 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.980210+0000", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00085321500000000005 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981063+0000", 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9449e-05 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981113+0000", 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122416 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:34.981235+0000", 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021778952000000001 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.3.0:478 60.16 e231/135 60:6ce498fc:test-rados-api-smithi107-112490-24::bar:e v 231'62, mlcod=226'58)", 2024-02-10T08:42:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 215.97855838300001, 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013769422999999999, 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959403+0000", 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959402+0000", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6172999999999999e-05 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959418+0000", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4239999999999998e-06 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959425+0000", 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.622e-06 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959432+0000", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104139 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959536+0000", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6367999999999999e-05 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.959563+0000", 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013555905 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.973119+0000", 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3390000000000002e-05 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:44.973172+0000", 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013769422999999999 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53375.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-10T08:42:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 123.93586740000001, 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999, 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53375", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4067602482", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002090+0000", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2568000000000001e-05 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002103+0000", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6772000000000001e-05 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002120+0000", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4579999999999999e-06 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002129+0000", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2737e-05 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002182+0000", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.963e-06 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002186+0000", 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.005764858 2024-02-10T08:42:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.007951+0000", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014669699999999999 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008097+0000", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045664099999999998 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008554+0000", 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038511399999999999 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008939+0000", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023091299999999999 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009170+0000", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3881999999999996e-05 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009234+0000", 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27867.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 99.931686369999994, 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001, 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27867", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/10211293", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:42:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006272+0000", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2092e-05 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:42:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006284+0000", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1758e-05 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006296+0000", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5979999999999997e-06 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006304+0000", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015616800000000001 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006460+0000", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4299999999999999e-06 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006465+0000", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0093833510000001 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.015848+0000", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017003600000000001 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:42:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016018+0000", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00054150900000000002 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016560+0000", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013759289999999999 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.017936+0000", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045079799999999998 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:42:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018386+0000", 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0971999999999997e-05 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018457+0000", 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:42:21.051 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:42:21.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:21.079 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:21.202 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:21.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:21.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:21.332 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:21.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:20.511655+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-10T08:42:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:20.514456+0000 mon.b (mon.1) 586 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:20.514884+0000 mon.a (mon.0) 2503 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:21.032793+0000 mon.a (mon.0) 2504 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:21.032896+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:21.054065+0000 mon.b (mon.1) 587 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:21.056277+0000 mon.b (mon.1) 588 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.612 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:21.056525+0000 mon.a (mon.0) 2506 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:42:21.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:42:21.680 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:21.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:21.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:20.511655+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:20.514456+0000 mon.b (mon.1) 586 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:20.514884+0000 mon.a (mon.0) 2503 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:21.032793+0000 mon.a (mon.0) 2504 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:21.032896+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:21.054065+0000 mon.b (mon.1) 587 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:21.056277+0000 mon.b (mon.1) 588 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.809 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:21.056525+0000 mon.a (mon.0) 2506 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:20.511655+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:20.514456+0000 mon.b (mon.1) 586 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:20.514884+0000 mon.a (mon.0) 2503 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:21.032793+0000 mon.a (mon.0) 2504 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:21.032896+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:21.054065+0000 mon.b (mon.1) 587 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:21.056277+0000 mon.b (mon.1) 588 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:21.056525+0000 mon.a (mon.0) 2506 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:21.865 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:21.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:21.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:21.997 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 -- 172.21.15.107:0/1851440945 >> v1:172.21.15.107:6790/0 conn(0x7fc5180fda50 legacy=0x7fc5180fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 -- 172.21.15.107:0/1851440945 shutdown_connections 2024-02-10T08:42:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 -- 172.21.15.107:0/1851440945 >> 172.21.15.107:0/1851440945 conn(0x7fc5180f3640 msgr2=0x7fc5180f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 -- 172.21.15.107:0/1851440945 shutdown_connections 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 -- 172.21.15.107:0/1851440945 wait complete. 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 librados: starting msgr at 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.973+0000 7fc51dafd700 1 librados: starting objecter 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 Processor -- start 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 -- start start 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 librados: setting wanted keys 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 librados: calling monclient init 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc51806cc00 con 0x7fc5180fda50 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc51806cd90 con 0x7fc5181007f0 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc51dafd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc51806cf80 con 0x7fc518104330 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc5177fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc5180fda50 0x7fc5180fc9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51368/0 (socket says 172.21.15.107:51368) 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc5177fe700 1 -- 172.21.15.107:0/3087484404 learned_addr learned my addr 172.21.15.107:0/3087484404 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc517fff700 1 --1- 172.21.15.107:0/3087484404 >> v1:172.21.15.107:6790/0 conn(0x7fc518104330 0x7fc5181184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54186/0 (socket says 172.21.15.107:54186) 2024-02-10T08:42:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter ms_handle_connect 0x7fc5180fda50 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter ms_handle_connect 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter ms_handle_connect 0x7fc5181007f0 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160005297 0 0) 0x7fc51806cf80 con 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5000033f0 con 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617649020 0 0) 0x7fc51806cc00 con 0x7fc5180fda50 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51806cf80 con 0x7fc5180fda50 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579564693 0 0) 0x7fc51806cd90 con 0x7fc5181007f0 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.974+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51806cc00 con 0x7fc5181007f0 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776459429 0 0) 0x7fc5000033f0 con 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc51806cd90 con 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc510001c80 con 0x7fc518104330 2024-02-10T08:42:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 404982892 0 0) 0x7fc51806cf80 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5000033f0 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 373931886 0 0) 0x7fc51806cc00 con 0x7fc5181007f0 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc51806cf80 con 0x7fc5181007f0 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc50c0021f0 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc508001840 con 0x7fc5181007f0 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3137282312 0 0) 0x7fc5000033f0 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 >> v1:172.21.15.107:6790/0 conn(0x7fc518104330 legacy=0x7fc5181184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 >> v1:172.21.15.195:6789/0 conn(0x7fc5181007f0 legacy=0x7fc518114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51811cc00 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc518119c30 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc51dafd700 10 client.28083.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51811a0a0 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc51dafd700 1 librados: init done 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc50c002390 con 0x7fc5180fda50 2024-02-10T08:42:22.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.975+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc50c003860 con 0x7fc5180fda50 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.976+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc504004f20 con 0x7fc5180fda50 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.976+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc50c001880 con 0x7fc5180fda50 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(430..430 src has 330..430) v4 ==== 8222+0+0 (unknown 1519576788 0 0) 0x7fc50c079280 con 0x7fc5180fda50 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 10 client.28083.objecter ms_dispatch 0x7fc518001550 osd_map(430..430 src has 330..430) v4 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 3 client.28083.objecter handle_osd_map got epochs [430,430] > 0 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 3 client.28083.objecter handle_osd_map decoding full epoch 430 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 20 client.28083.objecter dump_active .. 0 homeless 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.977+0000 7fc514ff9700 10 client.28083.objecter ms_handle_connect 0x7fc500063cf0 2024-02-10T08:42:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:21.981+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc50c04c690 con 0x7fc5180fda50 2024-02-10T08:42:22.016 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:22.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:22.148 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.168 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:22.179+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fc5040051d0 con 0x7fc5180fda50 2024-02-10T08:42:22.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:22.299 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.317 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:22.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:22.449 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.621 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:22.622 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:22.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:22.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:21.446707+0000 mgr.y (mgr.14553) 384 : cluster [DBG] pgmap v630: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:22.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:21.475636+0000 mgr.y (mgr.14553) 385 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:22.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:22.047526+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:22.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:22.047601+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-10T08:42:22.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:22.181816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:42:22.781 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.799 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:21.446707+0000 mgr.y (mgr.14553) 384 : cluster [DBG] pgmap v630: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:21.475636+0000 mgr.y (mgr.14553) 385 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:22.047526+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:22.047601+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:22.181816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:21.446707+0000 mgr.y (mgr.14553) 384 : cluster [DBG] pgmap v630: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:21.475636+0000 mgr.y (mgr.14553) 385 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:22.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:22.047526+0000 mon.a (mon.0) 2507 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:22.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:22.047601+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-10T08:42:22.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:22.181816+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:42:22.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:22.929 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:22.948 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:22.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.047+0000 7fc514ff9700 1 -- 172.21.15.107:0/3087484404 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v432) v1 ==== 91+0+0 (unknown 805793529 0 0) 0x7fc50c04ab00 con 0x7fc5180fda50 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 10 librados: watch_flush enter 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 10 librados: watch_flush exit 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 20 client.28083.objecter shutdown clearing up homeless session... 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 10 client.28083.objecter successfully canceled tick 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 >> v1:172.21.15.107:6800/2634842009 conn(0x7fc500063cf0 legacy=0x7fc5000661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 >> v1:172.21.15.107:6789/0 conn(0x7fc5180fda50 legacy=0x7fc5180fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 shutdown_connections 2024-02-10T08:42:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.049+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 >> 172.21.15.107:0/3087484404 conn(0x7fc5180f3640 msgr2=0x7fc5181077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.050+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 shutdown_connections 2024-02-10T08:42:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.050+0000 7fc51dafd700 1 -- 172.21.15.107:0/3087484404 wait complete. 2024-02-10T08:42:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.050+0000 7fc51dafd700 1 librados: shutdown 2024-02-10T08:42:23.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:23.053 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:42:23.082 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:23.281 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:23.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:23.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:23.415 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.697+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1219805619 >> v1:172.21.15.107:6789/0 conn(0x7f25dc1007f0 legacy=0x7f25dc102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.697+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1219805619 shutdown_connections 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.697+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1219805619 >> 172.21.15.107:0/1219805619 conn(0x7f25dc0f3640 msgr2=0x7f25dc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.697+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1219805619 shutdown_connections 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.697+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1219805619 wait complete. 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 librados: starting msgr at 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 librados: starting objecter 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 Processor -- start 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 -- start start 2024-02-10T08:42:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 librados: setting wanted keys 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 librados: calling monclient init 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25dc117400 con 0x7f25dc104330 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25dc1185e0 con 0x7f25dc1007f0 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e2d1a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25dc1197e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e0ab6700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f25dc0fda50 0x7f25dc0f9bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54204/0 (socket says 172.21.15.107:54204) 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25e0ab6700 1 -- 172.21.15.107:0/1010540718 learned_addr learned my addr 172.21.15.107:0/1010540718 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter ms_handle_connect 0x7f25dc104330 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter ms_handle_connect 0x7f25dc0fda50 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter ms_handle_connect 0x7f25dc1007f0 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757361484 0 0) 0x7f25dc117400 con 0x7f25dc104330 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc0038e0 con 0x7f25dc104330 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666301455 0 0) 0x7f25dc1197e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc117400 con 0x7f25dc0fda50 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501498925 0 0) 0x7f25dc1185e0 con 0x7f25dc1007f0 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.698+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc1197e0 con 0x7f25dc1007f0 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476758621 0 0) 0x7f25dc117400 con 0x7f25dc0fda50 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc1185e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25d0001880 con 0x7f25dc0fda50 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523862605 0 0) 0x7f25cc0038e0 con 0x7f25dc104330 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc117400 con 0x7f25dc104330 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25d4001c50 con 0x7f25dc104330 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1460855589 0 0) 0x7f25dc1197e0 con 0x7f25dc1007f0 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25cc0038e0 con 0x7f25dc1007f0 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25c8000fe0 con 0x7f25dc1007f0 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2396094531 0 0) 0x7f25dc1185e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 >> v1:172.21.15.195:6789/0 conn(0x7f25dc1007f0 legacy=0x7f25dc0fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 >> v1:172.21.15.107:6789/0 conn(0x7f25dc104330 legacy=0x7f25dc115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc11a9e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc1199d0 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f25d00019e0 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25d0005a70 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25e2d1a700 10 client.53672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc119e40 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.699+0000 7f25e2d1a700 1 librados: init done 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.700+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc05f700 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.700+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f25d0001f80 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(432..432 src has 330..432) v4 ==== 8194+0+0 (unknown 2926265857 0 0) 0x7f25d0078cc0 con 0x7f25dc0fda50 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 10 client.53672.objecter ms_dispatch 0x7f25dc001550 osd_map(432..432 src has 330..432) v4 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 3 client.53672.objecter handle_osd_map got epochs [432,432] > 0 2024-02-10T08:42:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 3 client.53672.objecter handle_osd_map decoding full epoch 432 2024-02-10T08:42:23.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 20 client.53672.objecter dump_active .. 0 homeless 2024-02-10T08:42:23.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.701+0000 7f25d9ffb700 10 client.53672.objecter ms_handle_connect 0x7f25cc063cd0 2024-02-10T08:42:23.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.705+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f25d004c030 con 0x7f25dc0fda50 2024-02-10T08:42:23.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.043695+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:23.047491+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:42:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.047544+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-10T08:42:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.865+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25dc0f91e0 con 0x7f25cc063cd0 2024-02-10T08:42:23.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.879+0000 7f25d9ffb700 1 -- 172.21.15.107:0/1010540718 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553111 (unknown 2965378022 0 1145843535) 0x7f25dc0f91e0 con 0x7f25cc063cd0 2024-02-10T08:42:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:42:23.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 10 librados: watch_flush enter 2024-02-10T08:42:23.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 10 librados: watch_flush exit 2024-02-10T08:42:23.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 20 client.53672.objecter shutdown clearing up homeless session... 2024-02-10T08:42:23.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 10 client.53672.objecter successfully canceled tick 2024-02-10T08:42:23.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 >> v1:172.21.15.107:6800/2634842009 conn(0x7f25cc063cd0 legacy=0x7f25cc066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 >> v1:172.21.15.107:6790/0 conn(0x7f25dc0fda50 legacy=0x7f25dc0f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 shutdown_connections 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 >> 172.21.15.107:0/1010540718 conn(0x7f25dc0f3640 msgr2=0x7f25dc1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 shutdown_connections 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 -- 172.21.15.107:0/1010540718 wait complete. 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:23.883+0000 7f25e2d1a700 1 librados: shutdown 2024-02-10T08:42:23.893 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:42:23.904 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.043695+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:23.904 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:23.047491+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:42:23.904 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.047544+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-10T08:42:23.904 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.043695+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:23.904 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:23.047491+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? v1:172.21.15.107:0/3087484404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:42:23.905 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.047544+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-10T08:42:24.264 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:24.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:24.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:24.402 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:24.423 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:24.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:24.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:24.557 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:24.576 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:24.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:24.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:24.725 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:24.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:24.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.330158+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:42:24.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.333063+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:42:24.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.447059+0000 mgr.y (mgr.14553) 386 : cluster [DBG] pgmap v633: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.527404+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:23.529003+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:23.867136+0000 mgr.y (mgr.14553) 387 : audit [DBG] from='client.53672 v1:172.21.15.107:0/1010540718' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:24.051265+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:24.085858+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:24.087514+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.330158+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.333063+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.447059+0000 mgr.y (mgr.14553) 386 : cluster [DBG] pgmap v633: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:42:24.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.527404+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:23.529003+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:23.867136+0000 mgr.y (mgr.14553) 387 : audit [DBG] from='client.53672 v1:172.21.15.107:0/1010540718' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:24.051265+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:24.085858+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:24.087514+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.330158+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.333063+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:42:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.447059+0000 mgr.y (mgr.14553) 386 : cluster [DBG] pgmap v633: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.527404+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:23.529003+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:23.867136+0000 mgr.y (mgr.14553) 387 : audit [DBG] from='client.53672 v1:172.21.15.107:0/1010540718' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:24.051265+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:24.085858+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:42:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:24.087514+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:42:24.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:25] "GET /metrics HTTP/1.1" 200 187633 "" "Prometheus/2.33.4" 2024-02-10T08:42:26.068 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2024-02-10T08:42:23.446843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007280,"num_objects":351,"num_object_clones":2,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":1367,"num_read_kb":1020,"num_write":1031,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2372,"ondisk_log_size":2372,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":624,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2142932,"kb_used_data":29596,"kb_used_omap":0,"kb_used_meta":2113280,"kb_avail":654073132,"statfs":{"total":671965249536,"available":669770887168,"internally_reserved":0,"allocated":30306304,"data_stored":18952970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2163998720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.941938"},"pg_stats":[{"pgid":"213.1f","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.048767+0000","last_change":"2024-02-10T08:42:21.048767+0000","last_active":"2024-02-10T08:42:21.048767+0000","last_peered":"2024-02-10T08:42:21.048767+0000","last_clean":"2024-02-10T08:42:21.048767+0000","last_became_active":"2024-02-10T08:42:21.048416+0000","last_became_peered":"2024-02-10T08:42:21.048416+0000","last_unstale":"2024-02-10T08:42:21.048767+0000","last_undegraded":"2024-02-10T08:42:21.048767+0000","last_fullsized":"2024-02-10T08:42:21.048767+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.1e","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.044517+0000","last_change":"2024-02-10T08:42:21.044517+0000","last_active":"2024-02-10T08:42:21.044517+0000","last_peered":"2024-02-10T08:42:21.044517+0000","last_clean":"2024-02-10T08:42:21.044517+0000","last_became_active":"2024-02-10T08:42:21.044164+0000","last_became_peered":"2024-02-10T08:42:21.044164+0000","last_unstale":"2024-02-10T08:42:21.044517+0000","last_undegraded":"2024-02-10T08:42:21.044517+0000","last_fullsized":"2024-02-10T08:42:21.044517+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.1d","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.038469+0000","last_change":"2024-02-10T08:42:21.038643+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.038469+0000","last_undegraded":"2024-02-10T08:42:21.038469+0000","last_fullsized":"2024-02-10T08:42:21.038469+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.1c","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.047526+0000","last_change":"2024-02-10T08:42:21.047526+0000","last_active":"2024-02-10T08:42:21.047526+0000","last_peered":"2024-02-10T08:42:21.047526+0000","last_clean":"2024-02-10T08:42:21.047526+0000","last_became_active":"2024-02-10T08:42:21.047271+0000","last_became_peered":"2024-02-10T08:42:21.047271+0000","last_unstale":"2024-02-10T08:42:21.047526+0000","last_undegraded":"2024-02-10T08:42:21.047526+0000","last_fullsized":"2024-02-10T08:42:21.047526+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.1b","version":"0'0","reported_seq":3,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.035061+0000","last_change":"2024-02-10T08:42:21.035061+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.035061+0000","last_undegraded":"2024-02-10T08:42:21.035061+0000","last_fullsized":"2024-02-10T08:42:21.035061+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.1a","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.053591+0000","last_change":"2024-02-10T08:42:21.053591+0000","last_active":"2024-02-10T08:42:21.053591+0000","last_peered":"2024-02-10T08:42:21.053591+0000","last_clean":"2024-02-10T08:42:21.053591+0000","last_became_active":"2024-02-10T08:42:21.053318+0000","last_became_peered":"2024-02-10T08:42:21.053318+0000","last_unstale":"2024-02-10T08:42:21.053591+0000","last_undegraded":"2024-02-10T08:42:21.053591+0000","last_fullsized":"2024-02-10T08:42:21.053591+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"213.19","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.037497+0000","last_change":"2024-02-10T08:42:21.037677+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.037497+0000","last_undegraded":"2024-02-10T08:42:21.037497+0000","last_fullsized":"2024-02-10T08:42:21.037497+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.18","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042278+0000","last_change":"2024-02-10T08:42:21.042454+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042278+0000","last_undegraded":"2024-02-10T08:42:21.042278+0000","last_fullsized":"2024-02-10T08:42:21.042278+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":["7","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"213.17","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.048624+0000","last_change":"2024-02-10T08:42:21.048624+0000","last_active":"2024-02-10T08:42:21.048624+0000","last_peered":"2024-02-10T08:42:21.048624+0000","last_clean":"2024-02-10T08:42:21.048624+0000","last_became_active":"2024-02-10T08:42:21.048031+0000","last_became_peered":"2024-02-10T08:42:21.048031+0000","last_unstale":"2024-02-10T08:42:21.048624+0000","last_undegraded":"2024-02-10T08:42:21.048624+0000","last_fullsized":"2024-02-10T08:42:21.048624+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.16","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.047150+0000","last_change":"2024-02-10T08:42:21.047150+0000","last_active":"2024-02-10T08:42:21.047150+0000","last_peered":"2024-02-10T08:42:21.047150+0000","last_clean":"2024-02-10T08:42:21.047150+0000","last_became_active":"2024-02-10T08:42:21.046877+0000","last_became_peered":"2024-02-10T08:42:21.046877+0000","last_unstale":"2024-02-10T08:42:21.047150+0000","last_undegraded":"2024-02-10T08:42:21.047150+0000","last_fullsized":"2024-02-10T08:42:21.047150+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"213.15","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.028871+0000","last_change":"2024-02-10T08:42:21.029089+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.028871+0000","last_undegraded":"2024-02-10T08:42:21.028871+0000","last_fullsized":"2024-02-10T08:42:21.028871+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"213.14","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042463+0000","last_change":"2024-02-10T08:42:21.042640+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042463+0000","last_undegraded":"2024-02-10T08:42:21.042463+0000","last_fullsized":"2024-02-10T08:42:21.042463+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":["7","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"213.13","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.036550+0000","last_change":"2024-02-10T08:42:21.036750+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.036550+0000","last_undegraded":"2024-02-10T08:42:21.036550+0000","last_fullsized":"2024-02-10T08:42:21.036550+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.12","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042438+0000","last_change":"2024-02-10T08:42:21.042603+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042438+0000","last_undegraded":"2024-02-10T08:42:21.042438+0000","last_fullsized":"2024-02-10T08:42:21.042438+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":["7","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"213.11","version":"0'0","reported_seq":3,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.029112+0000","last_change":"2024-02-10T08:42:21.029112+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.029112+0000","last_undegraded":"2024-02-10T08:42:21.029112+0000","last_fullsized":"2024-02-10T08:42:21.029112+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"213.10","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.034667+0000","last_change":"2024-02-10T08:42:21.034854+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.034667+0000","last_undegraded":"2024-02-10T08:42:21.034667+0000","last_fullsized":"2024-02-10T08:42:21.034667+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"213.f","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.034913+0000","last_change":"2024-02-10T08:42:21.035120+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.034913+0000","last_undegraded":"2024-02-10T08:42:21.034913+0000","last_fullsized":"2024-02-10T08:42:21.034913+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.e","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.028898+0000","last_change":"2024-02-10T08:42:21.029072+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.028898+0000","last_undegraded":"2024-02-10T08:42:21.028898+0000","last_fullsized":"2024-02-10T08:42:21.028898+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"213.d","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.035669+0000","last_change":"2024-02-10T08:42:21.035806+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.035669+0000","last_undegraded":"2024-02-10T08:42:21.035669+0000","last_fullsized":"2024-02-10T08:42:21.035669+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"213.c","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.041898+0000","last_change":"2024-02-10T08:42:21.041898+0000","last_active":"2024-02-10T08:42:21.041898+0000","last_peered":"2024-02-10T08:42:21.041898+0000","last_clean":"2024-02-10T08:42:21.041898+0000","last_became_active":"2024-02-10T08:42:21.041564+0000","last_became_peered":"2024-02-10T08:42:21.041564+0000","last_unstale":"2024-02-10T08:42:21.041898+0000","last_undegraded":"2024-02-10T08:42:21.041898+0000","last_fullsized":"2024-02-10T08:42:21.041898+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.b","version":"0'0","reported_seq":3,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.036810+0000","last_change":"2024-02-10T08:42:21.036810+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.036810+0000","last_undegraded":"2024-02-10T08:42:21.036810+0000","last_fullsized":"2024-02-10T08:42:21.036810+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.a","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.028877+0000","last_change":"2024-02-10T08:42:21.029259+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.028877+0000","last_undegraded":"2024-02-10T08:42:21.028877+0000","last_fullsized":"2024-02-10T08:42:21.028877+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":["5","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"213.9","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.048648+0000","last_change":"2024-02-10T08:42:21.048648+0000","last_active":"2024-02-10T08:42:21.048648+0000","last_peered":"2024-02-10T08:42:21.048648+0000","last_clean":"2024-02-10T08:42:21.048648+0000","last_became_active":"2024-02-10T08:42:21.048359+0000","last_became_peered":"2024-02-10T08:42:21.048359+0000","last_unstale":"2024-02-10T08:42:21.048648+0000","last_undegraded":"2024-02-10T08:42:21.048648+0000","last_fullsized":"2024-02-10T08:42:21.048648+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.8","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042548+0000","last_change":"2024-02-10T08:42:21.042701+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042548+0000","last_undegraded":"2024-02-10T08:42:21.042548+0000","last_fullsized":"2024-02-10T08:42:21.042548+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.7","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.045601+0000","last_change":"2024-02-10T08:42:21.045601+0000","last_active":"2024-02-10T08:42:21.045601+0000","last_peered":"2024-02-10T08:42:21.045601+0000","last_clean":"2024-02-10T08:42:21.045601+0000","last_became_active":"2024-02-10T08:42:21.045193+0000","last_became_peered":"2024-02-10T08:42:21.045193+0000","last_unstale":"2024-02-10T08:42:21.045601+0000","last_undegraded":"2024-02-10T08:42:21.045601+0000","last_fullsized":"2024-02-10T08:42:21.045601+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.6","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.043192+0000","last_change":"2024-02-10T08:42:21.043376+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.043192+0000","last_undegraded":"2024-02-10T08:42:21.043192+0000","last_fullsized":"2024-02-10T08:42:21.043192+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"213.5","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.041233+0000","last_change":"2024-02-10T08:42:21.041233+0000","last_active":"2024-02-10T08:42:21.041233+0000","last_peered":"2024-02-10T08:42:21.041233+0000","last_clean":"2024-02-10T08:42:21.041233+0000","last_became_active":"2024-02-10T08:42:21.040989+0000","last_became_peered":"2024-02-10T08:42:21.040989+0000","last_unstale":"2024-02-10T08:42:21.041233+0000","last_undegraded":"2024-02-10T08:42:21.041233+0000","last_fullsized":"2024-02-10T08:42:21.041233+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"213.4","version":"0'0","reported_seq":3,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.036825+0000","last_change":"2024-02-10T08:42:21.036825+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.036825+0000","last_undegraded":"2024-02-10T08:42:21.036825+0000","last_fullsized":"2024-02-10T08:42:21.036825+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.3","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.053818+0000","last_change":"2024-02-10T08:42:21.053818+0000","last_active":"2024-02-10T08:42:21.053818+0000","last_peered":"2024-02-10T08:42:21.053818+0000","last_clean":"2024-02-10T08:42:21.053818+0000","last_became_active":"2024-02-10T08:42:21.050365+0000","last_became_peered":"2024-02-10T08:42:21.050365+0000","last_unstale":"2024-02-10T08:42:21.053818+0000","last_undegraded":"2024-02-10T08:42:21.053818+0000","last_fullsized":"2024-02-10T08:42:21.053818+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"213.2","version":"0'0","reported_seq":9,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.049785+0000","last_change":"2024-02-10T08:42:21.049785+0000","last_active":"2024-02-10T08:42:21.049785+0000","last_peered":"2024-02-10T08:42:21.049785+0000","last_clean":"2024-02-10T08:42:21.049785+0000","last_became_active":"2024-02-10T08:42:21.049479+0000","last_became_peered":"2024-02-10T08:42:21.049479+0000","last_unstale":"2024-02-10T08:42:21.049785+0000","last_undegraded":"2024-02-10T08:42:21.049785+0000","last_fullsized":"2024-02-10T08:42:21.049785+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"213.1","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042777+0000","last_change":"2024-02-10T08:42:21.042951+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042777+0000","last_undegraded":"2024-02-10T08:42:21.042777+0000","last_fullsized":"2024-02-10T08:42:21.042777+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"213.0","version":"0'0","reported_seq":2,"reported_epoch":430,"state":"creating+activating","last_fresh":"2024-02-10T08:42:21.042216+0000","last_change":"2024-02-10T08:42:21.042379+0000","last_active":"2024-02-10T08:42:20.504283+0000","last_peered":"2024-02-10T08:42:20.504283+0000","last_clean":"2024-02-10T08:42:20.504283+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:42:21.042216+0000","last_undegraded":"2024-02-10T08:42:21.042216+0000","last_fullsized":"2024-02-10T08:42:21.042216+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:20.504283+0000","last_clean_scrub_stamp":"2024-02-10T08:42:20.504283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":412,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.038232+0000","last_change":"2024-02-10T08:42:09.261204+0000","last_active":"2024-02-10T08:42:21.038232+0000","last_peered":"2024-02-10T08:42:21.038232+0000","last_clean":"2024-02-10T08:42:21.038232+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:42:21.038232+0000","last_undegraded":"2024-02-10T08:42:21.038232+0000","last_fullsized":"2024-02-10T08:42:21.038232+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:40:32.184829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:32.184829+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"338'29","reported_seq":415,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139000+0000","last_change":"2024-02-10T08:42:09.271741+0000","last_active":"2024-02-10T08:42:20.139000+0000","last_peered":"2024-02-10T08:42:20.139000+0000","last_clean":"2024-02-10T08:42:20.139000+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:42:20.139000+0000","last_undegraded":"2024-02-10T08:42:20.139000+0000","last_fullsized":"2024-02-10T08:42:20.139000+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-10T08:39:38.573672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.573672+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1","version":"401'42","reported_seq":415,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.137693+0000","last_change":"2024-02-10T08:42:09.270997+0000","last_active":"2024-02-10T08:42:20.137693+0000","last_peered":"2024-02-10T08:42:20.137693+0000","last_clean":"2024-02-10T08:42:20.137693+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:42:20.137693+0000","last_undegraded":"2024-02-10T08:42:20.137693+0000","last_fullsized":"2024-02-10T08:42:20.137693+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T08:39:40.483156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.483156+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.0","version":"371'108","reported_seq":492,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.039717+0000","last_change":"2024-02-10T08:42:09.259389+0000","last_active":"2024-02-10T08:42:21.039717+0000","last_peered":"2024-02-10T08:42:21.039717+0000","last_clean":"2024-02-10T08:42:21.039717+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:42:21.039717+0000","last_undegraded":"2024-02-10T08:42:21.039717+0000","last_fullsized":"2024-02-10T08:42:21.039717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"371'35","reported_seq":393,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.142382+0000","last_change":"2024-02-10T08:42:09.273008+0000","last_active":"2024-02-10T08:42:20.142382+0000","last_peered":"2024-02-10T08:42:20.142382+0000","last_clean":"2024-02-10T08:42:20.142382+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:42:20.142382+0000","last_undegraded":"2024-02-10T08:42:20.142382+0000","last_fullsized":"2024-02-10T08:42:20.142382+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:39:53.632416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.632416+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.6","version":"338'73","reported_seq":449,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.046142+0000","last_change":"2024-02-10T08:42:09.262392+0000","last_active":"2024-02-10T08:42:21.046142+0000","last_peered":"2024-02-10T08:42:21.046142+0000","last_clean":"2024-02-10T08:42:21.046142+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:42:21.046142+0000","last_undegraded":"2024-02-10T08:42:21.046142+0000","last_fullsized":"2024-02-10T08:42:21.046142+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'71","last_scrub_stamp":"2024-02-10T08:39:49.450649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.450649+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":429,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135009+0000","last_change":"2024-02-10T08:42:09.272549+0000","last_active":"2024-02-10T08:42:20.135009+0000","last_peered":"2024-02-10T08:42:20.135009+0000","last_clean":"2024-02-10T08:42:20.135009+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:42:20.135009+0000","last_undegraded":"2024-02-10T08:42:20.135009+0000","last_fullsized":"2024-02-10T08:42:20.135009+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.4","version":"338'23","reported_seq":407,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139730+0000","last_change":"2024-02-10T08:42:09.272043+0000","last_active":"2024-02-10T08:42:20.139730+0000","last_peered":"2024-02-10T08:42:20.139730+0000","last_clean":"2024-02-10T08:42:20.139730+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:42:20.139730+0000","last_undegraded":"2024-02-10T08:42:20.139730+0000","last_fullsized":"2024-02-10T08:42:20.139730+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.b","version":"395'105","reported_seq":480,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.045770+0000","last_change":"2024-02-10T08:42:09.260818+0000","last_active":"2024-02-10T08:42:21.045770+0000","last_peered":"2024-02-10T08:42:21.045770+0000","last_clean":"2024-02-10T08:42:21.045770+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:42:21.045770+0000","last_undegraded":"2024-02-10T08:42:21.045770+0000","last_fullsized":"2024-02-10T08:42:21.045770+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'101","last_scrub_stamp":"2024-02-10T08:40:28.463426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.463426+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"371'23","reported_seq":408,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138910+0000","last_change":"2024-02-10T08:42:09.271173+0000","last_active":"2024-02-10T08:42:20.138910+0000","last_peered":"2024-02-10T08:42:20.138910+0000","last_clean":"2024-02-10T08:42:20.138910+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:42:20.138910+0000","last_undegraded":"2024-02-10T08:42:20.138910+0000","last_fullsized":"2024-02-10T08:42:20.138910+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-02-10T08:39:47.635601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.635601+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.9","version":"395'115","reported_seq":475,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.038757+0000","last_change":"2024-02-10T08:42:09.263795+0000","last_active":"2024-02-10T08:42:21.038757+0000","last_peered":"2024-02-10T08:42:21.038757+0000","last_clean":"2024-02-10T08:42:21.038757+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:42:21.038757+0000","last_undegraded":"2024-02-10T08:42:21.038757+0000","last_fullsized":"2024-02-10T08:42:21.038757+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"299'110","last_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.048153+0000","log_size":115,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":35,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"338'86","reported_seq":463,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.035168+0000","last_change":"2024-02-10T08:42:09.270554+0000","last_active":"2024-02-10T08:42:21.035168+0000","last_peered":"2024-02-10T08:42:21.035168+0000","last_clean":"2024-02-10T08:42:21.035168+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:42:21.035168+0000","last_undegraded":"2024-02-10T08:42:21.035168+0000","last_fullsized":"2024-02-10T08:42:21.035168+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'85","last_scrub_stamp":"2024-02-10T08:39:50.630038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:50.630038+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"405'96","reported_seq":469,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.037881+0000","last_change":"2024-02-10T08:42:09.599263+0000","last_active":"2024-02-10T08:42:21.037881+0000","last_peered":"2024-02-10T08:42:21.037881+0000","last_clean":"2024-02-10T08:42:21.037881+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:42:21.037881+0000","last_undegraded":"2024-02-10T08:42:21.037881+0000","last_fullsized":"2024-02-10T08:42:21.037881+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:41:45.373410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.373410+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":415,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.035753+0000","last_change":"2024-02-10T08:42:09.598551+0000","last_active":"2024-02-10T08:42:21.035753+0000","last_peered":"2024-02-10T08:42:21.035753+0000","last_clean":"2024-02-10T08:42:21.035753+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:42:21.035753+0000","last_undegraded":"2024-02-10T08:42:21.035753+0000","last_fullsized":"2024-02-10T08:42:21.035753+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"423'78","reported_seq":457,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.038506+0000","last_change":"2024-02-10T08:42:11.601220+0000","last_active":"2024-02-10T08:42:21.038506+0000","last_peered":"2024-02-10T08:42:21.038506+0000","last_clean":"2024-02-10T08:42:21.038506+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:42:21.038506+0000","last_undegraded":"2024-02-10T08:42:21.038506+0000","last_fullsized":"2024-02-10T08:42:21.038506+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'74","last_scrub_stamp":"2024-02-10T08:41:55.273156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.273156+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"338'70","reported_seq":448,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.046080+0000","last_change":"2024-02-10T08:42:09.262189+0000","last_active":"2024-02-10T08:42:21.046080+0000","last_peered":"2024-02-10T08:42:21.046080+0000","last_clean":"2024-02-10T08:42:21.046080+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:42:21.046080+0000","last_undegraded":"2024-02-10T08:42:21.046080+0000","last_fullsized":"2024-02-10T08:42:21.046080+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:40:36.430069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.430069+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"338'56","reported_seq":438,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.029308+0000","last_change":"2024-02-10T08:42:09.598603+0000","last_active":"2024-02-10T08:42:21.029308+0000","last_peered":"2024-02-10T08:42:21.029308+0000","last_clean":"2024-02-10T08:42:21.029308+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:42:21.029308+0000","last_undegraded":"2024-02-10T08:42:21.029308+0000","last_fullsized":"2024-02-10T08:42:21.029308+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'56","last_scrub_stamp":"2024-02-10T08:41:49.073509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.073509+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":484,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.042308+0000","last_change":"2024-02-10T08:42:09.258181+0000","last_active":"2024-02-10T08:42:21.042308+0000","last_peered":"2024-02-10T08:42:21.042308+0000","last_clean":"2024-02-10T08:42:21.042308+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:42:21.042308+0000","last_undegraded":"2024-02-10T08:42:21.042308+0000","last_fullsized":"2024-02-10T08:42:21.042308+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"398'105","reported_seq":470,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.029485+0000","last_change":"2024-02-10T08:42:09.598422+0000","last_active":"2024-02-10T08:42:21.029485+0000","last_peered":"2024-02-10T08:42:21.029485+0000","last_clean":"2024-02-10T08:42:21.029485+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:42:21.029485+0000","last_undegraded":"2024-02-10T08:42:21.029485+0000","last_fullsized":"2024-02-10T08:42:21.029485+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'103","last_scrub_stamp":"2024-02-10T08:39:41.578533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.578533+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":459,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.042712+0000","last_change":"2024-02-10T08:42:09.258477+0000","last_active":"2024-02-10T08:42:21.042712+0000","last_peered":"2024-02-10T08:42:21.042712+0000","last_clean":"2024-02-10T08:42:21.042712+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:42:21.042712+0000","last_undegraded":"2024-02-10T08:42:21.042712+0000","last_fullsized":"2024-02-10T08:42:21.042712+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:42:01.488098+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.488098+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"392'87","reported_seq":463,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.043738+0000","last_change":"2024-02-10T08:42:09.258979+0000","last_active":"2024-02-10T08:42:21.043738+0000","last_peered":"2024-02-10T08:42:21.043738+0000","last_clean":"2024-02-10T08:42:21.043738+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:42:21.043738+0000","last_undegraded":"2024-02-10T08:42:21.043738+0000","last_fullsized":"2024-02-10T08:42:21.043738+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:41:57.565097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.565097+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":460,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.035711+0000","last_change":"2024-02-10T08:42:09.598636+0000","last_active":"2024-02-10T08:42:21.035711+0000","last_peered":"2024-02-10T08:42:21.035711+0000","last_clean":"2024-02-10T08:42:21.035711+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:42:21.035711+0000","last_undegraded":"2024-02-10T08:42:21.035711+0000","last_fullsized":"2024-02-10T08:42:21.035711+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"338'18","reported_seq":399,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.134956+0000","last_change":"2024-02-10T08:42:09.272540+0000","last_active":"2024-02-10T08:42:20.134956+0000","last_peered":"2024-02-10T08:42:20.134956+0000","last_clean":"2024-02-10T08:42:20.134956+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:42:20.134956+0000","last_undegraded":"2024-02-10T08:42:20.134956+0000","last_fullsized":"2024-02-10T08:42:20.134956+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.14","version":"295'36","reported_seq":413,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.034700+0000","last_change":"2024-02-10T08:42:09.598112+0000","last_active":"2024-02-10T08:42:21.034700+0000","last_peered":"2024-02-10T08:42:21.034700+0000","last_clean":"2024-02-10T08:42:21.034700+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:42:21.034700+0000","last_undegraded":"2024-02-10T08:42:21.034700+0000","last_fullsized":"2024-02-10T08:42:21.034700+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"338'69","reported_seq":425,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135292+0000","last_change":"2024-02-10T08:42:09.271571+0000","last_active":"2024-02-10T08:42:20.135292+0000","last_peered":"2024-02-10T08:42:20.135292+0000","last_clean":"2024-02-10T08:42:20.135292+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:42:20.135292+0000","last_undegraded":"2024-02-10T08:42:20.135292+0000","last_fullsized":"2024-02-10T08:42:20.135292+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"60.1a","version":"398'71","reported_seq":438,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.040502+0000","last_change":"2024-02-10T08:42:09.264276+0000","last_active":"2024-02-10T08:42:21.040502+0000","last_peered":"2024-02-10T08:42:21.040502+0000","last_clean":"2024-02-10T08:42:21.040502+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:42:21.040502+0000","last_undegraded":"2024-02-10T08:42:21.040502+0000","last_fullsized":"2024-02-10T08:42:21.040502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:40:35.307608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.307608+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":425,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.041327+0000","last_change":"2024-02-10T08:42:09.264072+0000","last_active":"2024-02-10T08:42:21.041327+0000","last_peered":"2024-02-10T08:42:21.041327+0000","last_clean":"2024-02-10T08:42:21.041327+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:42:21.041327+0000","last_undegraded":"2024-02-10T08:42:21.041327+0000","last_fullsized":"2024-02-10T08:42:21.041327+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'34","last_scrub_stamp":"2024-02-10T08:40:28.186484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.186484+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"401'38","reported_seq":399,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.035152+0000","last_change":"2024-02-10T08:42:09.270546+0000","last_active":"2024-02-10T08:42:21.035152+0000","last_peered":"2024-02-10T08:42:21.035152+0000","last_clean":"2024-02-10T08:42:21.035152+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:42:21.035152+0000","last_undegraded":"2024-02-10T08:42:21.035152+0000","last_fullsized":"2024-02-10T08:42:21.035152+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"338'36","reported_seq":429,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.030287+0000","last_change":"2024-02-10T08:42:09.600510+0000","last_active":"2024-02-10T08:42:21.030287+0000","last_peered":"2024-02-10T08:42:21.030287+0000","last_clean":"2024-02-10T08:42:21.030287+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:42:21.030287+0000","last_undegraded":"2024-02-10T08:42:21.030287+0000","last_fullsized":"2024-02-10T08:42:21.030287+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'36","last_scrub_stamp":"2024-02-10T08:41:46.085763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.085763+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":387,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.040572+0000","last_change":"2024-02-10T08:42:09.261550+0000","last_active":"2024-02-10T08:42:21.040572+0000","last_peered":"2024-02-10T08:42:21.040572+0000","last_clean":"2024-02-10T08:42:21.040572+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:42:21.040572+0000","last_undegraded":"2024-02-10T08:42:21.040572+0000","last_fullsized":"2024-02-10T08:42:21.040572+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:42:04.642908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.642908+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"405'21","reported_seq":393,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.035063+0000","last_change":"2024-02-10T08:42:09.598220+0000","last_active":"2024-02-10T08:42:21.035063+0000","last_peered":"2024-02-10T08:42:21.035063+0000","last_clean":"2024-02-10T08:42:21.035063+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:42:21.035063+0000","last_undegraded":"2024-02-10T08:42:21.035063+0000","last_fullsized":"2024-02-10T08:42:21.035063+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"371'35","reported_seq":419,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.045872+0000","last_change":"2024-02-10T08:42:09.262014+0000","last_active":"2024-02-10T08:42:21.045872+0000","last_peered":"2024-02-10T08:42:21.045872+0000","last_clean":"2024-02-10T08:42:21.045872+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:42:21.045872+0000","last_undegraded":"2024-02-10T08:42:21.045872+0000","last_fullsized":"2024-02-10T08:42:21.045872+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:40:23.345538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.345538+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":532,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515825+0000","last_change":"2024-02-10T08:41:09.353173+0000","last_active":"2024-02-10T08:42:20.515825+0000","last_peered":"2024-02-10T08:42:20.515825+0000","last_clean":"2024-02-10T08:42:20.515825+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:42:20.515825+0000","last_undegraded":"2024-02-10T08:42:20.515825+0000","last_fullsized":"2024-02-10T08:42:20.515825+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:41:09.353088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:09.353088+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":425,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517770+0000","last_change":"2024-02-10T08:39:42.594007+0000","last_active":"2024-02-10T08:42:20.517770+0000","last_peered":"2024-02-10T08:42:20.517770+0000","last_clean":"2024-02-10T08:42:20.517770+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:42:20.517770+0000","last_undegraded":"2024-02-10T08:42:20.517770+0000","last_fullsized":"2024-02-10T08:42:20.517770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:39:42.593941+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.593941+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":398,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516899+0000","last_change":"2024-02-10T08:40:42.232974+0000","last_active":"2024-02-10T08:42:20.516899+0000","last_peered":"2024-02-10T08:42:20.516899+0000","last_clean":"2024-02-10T08:42:20.516899+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:42:20.516899+0000","last_undegraded":"2024-02-10T08:42:20.516899+0000","last_fullsized":"2024-02-10T08:42:20.516899+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.232905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.232905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":403,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518073+0000","last_change":"2024-02-10T08:40:26.859788+0000","last_active":"2024-02-10T08:42:20.518073+0000","last_peered":"2024-02-10T08:42:20.518073+0000","last_clean":"2024-02-10T08:42:20.518073+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:42:20.518073+0000","last_undegraded":"2024-02-10T08:42:20.518073+0000","last_fullsized":"2024-02-10T08:42:20.518073+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.859697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.859697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":346,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516672+0000","last_change":"2024-02-10T08:41:59.348271+0000","last_active":"2024-02-10T08:42:20.516672+0000","last_peered":"2024-02-10T08:42:20.516672+0000","last_clean":"2024-02-10T08:42:20.516672+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:42:20.516672+0000","last_undegraded":"2024-02-10T08:42:20.516672+0000","last_fullsized":"2024-02-10T08:42:20.516672+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.348211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.348211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":375,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516415+0000","last_change":"2024-02-10T08:41:16.350785+0000","last_active":"2024-02-10T08:42:20.516415+0000","last_peered":"2024-02-10T08:42:20.516415+0000","last_clean":"2024-02-10T08:42:20.516415+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:42:20.516415+0000","last_undegraded":"2024-02-10T08:42:20.516415+0000","last_fullsized":"2024-02-10T08:42:20.516415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":403,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513594+0000","last_change":"2024-02-10T08:41:05.559497+0000","last_active":"2024-02-10T08:42:20.513594+0000","last_peered":"2024-02-10T08:42:20.513594+0000","last_clean":"2024-02-10T08:42:20.513594+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:42:20.513594+0000","last_undegraded":"2024-02-10T08:42:20.513594+0000","last_fullsized":"2024-02-10T08:42:20.513594+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.559402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.559402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":417,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516522+0000","last_change":"2024-02-10T08:41:43.195685+0000","last_active":"2024-02-10T08:42:20.516522+0000","last_peered":"2024-02-10T08:42:20.516522+0000","last_clean":"2024-02-10T08:42:20.516522+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:42:20.516522+0000","last_undegraded":"2024-02-10T08:42:20.516522+0000","last_fullsized":"2024-02-10T08:42:20.516522+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":313,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515497+0000","last_change":"2024-02-10T08:40:39.350744+0000","last_active":"2024-02-10T08:42:20.515497+0000","last_peered":"2024-02-10T08:42:20.515497+0000","last_clean":"2024-02-10T08:42:20.515497+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:42:20.515497+0000","last_undegraded":"2024-02-10T08:42:20.515497+0000","last_fullsized":"2024-02-10T08:42:20.515497+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.350663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":343,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519471+0000","last_change":"2024-02-10T08:41:45.092740+0000","last_active":"2024-02-10T08:42:20.519471+0000","last_peered":"2024-02-10T08:42:20.519471+0000","last_clean":"2024-02-10T08:42:20.519471+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:42:20.519471+0000","last_undegraded":"2024-02-10T08:42:20.519471+0000","last_fullsized":"2024-02-10T08:42:20.519471+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:45.092620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.092620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":435,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518024+0000","last_change":"2024-02-10T08:41:57.284919+0000","last_active":"2024-02-10T08:42:20.518024+0000","last_peered":"2024-02-10T08:42:20.518024+0000","last_clean":"2024-02-10T08:42:20.518024+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:42:20.518024+0000","last_undegraded":"2024-02-10T08:42:20.518024+0000","last_fullsized":"2024-02-10T08:42:20.518024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:57.284811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.284811+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":407,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517992+0000","last_change":"2024-02-10T08:40:40.314002+0000","last_active":"2024-02-10T08:42:20.517992+0000","last_peered":"2024-02-10T08:42:20.517992+0000","last_clean":"2024-02-10T08:42:20.517992+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:42:20.517992+0000","last_undegraded":"2024-02-10T08:42:20.517992+0000","last_fullsized":"2024-02-10T08:42:20.517992+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.313935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":421,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516331+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:42:20.516331+0000","last_peered":"2024-02-10T08:42:20.516331+0000","last_clean":"2024-02-10T08:42:20.516331+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:42:20.516331+0000","last_undegraded":"2024-02-10T08:42:20.516331+0000","last_fullsized":"2024-02-10T08:42:20.516331+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":389,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514765+0000","last_change":"2024-02-10T08:40:28.426111+0000","last_active":"2024-02-10T08:42:20.514765+0000","last_peered":"2024-02-10T08:42:20.514765+0000","last_clean":"2024-02-10T08:42:20.514765+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:42:20.514765+0000","last_undegraded":"2024-02-10T08:42:20.514765+0000","last_fullsized":"2024-02-10T08:42:20.514765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:28.425994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.425994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":343,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513966+0000","last_change":"2024-02-10T08:41:20.535962+0000","last_active":"2024-02-10T08:42:20.513966+0000","last_peered":"2024-02-10T08:42:20.513966+0000","last_clean":"2024-02-10T08:42:20.513966+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:42:20.513966+0000","last_undegraded":"2024-02-10T08:42:20.513966+0000","last_fullsized":"2024-02-10T08:42:20.513966+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":427,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135308+0000","last_change":"2024-02-10T08:41:13.773323+0000","last_active":"2024-02-10T08:42:20.135308+0000","last_peered":"2024-02-10T08:42:20.135308+0000","last_clean":"2024-02-10T08:42:20.135308+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:42:20.135308+0000","last_undegraded":"2024-02-10T08:42:20.135308+0000","last_fullsized":"2024-02-10T08:42:20.135308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:41:13.773224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.773224+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":237,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515477+0000","last_change":"2024-02-10T08:41:16.638540+0000","last_active":"2024-02-10T08:42:20.515477+0000","last_peered":"2024-02-10T08:42:20.515477+0000","last_clean":"2024-02-10T08:42:20.515477+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:42:20.515477+0000","last_undegraded":"2024-02-10T08:42:20.515477+0000","last_fullsized":"2024-02-10T08:42:20.515477+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.638477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.638477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":421,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512496+0000","last_change":"2024-02-10T08:40:40.501961+0000","last_active":"2024-02-10T08:42:20.512496+0000","last_peered":"2024-02-10T08:42:20.512496+0000","last_clean":"2024-02-10T08:42:20.512496+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:42:20.512496+0000","last_undegraded":"2024-02-10T08:42:20.512496+0000","last_fullsized":"2024-02-10T08:42:20.512496+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.501871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.501871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":392,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517364+0000","last_change":"2024-02-10T08:39:51.451546+0000","last_active":"2024-02-10T08:42:20.517364+0000","last_peered":"2024-02-10T08:42:20.517364+0000","last_clean":"2024-02-10T08:42:20.517364+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:42:20.517364+0000","last_undegraded":"2024-02-10T08:42:20.517364+0000","last_fullsized":"2024-02-10T08:42:20.517364+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:51.451479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:39:51.451479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":339,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.137773+0000","last_change":"2024-02-10T08:39:14.683342+0000","last_active":"2024-02-10T08:42:20.137773+0000","last_peered":"2024-02-10T08:42:20.137773+0000","last_clean":"2024-02-10T08:42:20.137773+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:42:20.137773+0000","last_undegraded":"2024-02-10T08:42:20.137773+0000","last_fullsized":"2024-02-10T08:42:20.137773+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:39:14.683242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":414,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139964+0000","last_change":"2024-02-10T08:41:18.755591+0000","last_active":"2024-02-10T08:42:20.139964+0000","last_peered":"2024-02-10T08:42:20.139964+0000","last_clean":"2024-02-10T08:42:20.139964+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:42:20.139964+0000","last_undegraded":"2024-02-10T08:42:20.139964+0000","last_fullsized":"2024-02-10T08:42:20.139964+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":403,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511580+0000","last_change":"2024-02-10T08:41:10.625950+0000","last_active":"2024-02-10T08:42:20.511580+0000","last_peered":"2024-02-10T08:42:20.511580+0000","last_clean":"2024-02-10T08:42:20.511580+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:42:20.511580+0000","last_undegraded":"2024-02-10T08:42:20.511580+0000","last_fullsized":"2024-02-10T08:42:20.511580+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.625888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.625888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":530,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517531+0000","last_change":"2024-02-10T08:40:41.276981+0000","last_active":"2024-02-10T08:42:20.517531+0000","last_peered":"2024-02-10T08:42:20.517531+0000","last_clean":"2024-02-10T08:42:20.517531+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:42:20.517531+0000","last_undegraded":"2024-02-10T08:42:20.517531+0000","last_fullsized":"2024-02-10T08:42:20.517531+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:40:41.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.276901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":395,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138363+0000","last_change":"2024-02-10T08:41:43.683458+0000","last_active":"2024-02-10T08:42:20.138363+0000","last_peered":"2024-02-10T08:42:20.138363+0000","last_clean":"2024-02-10T08:42:20.138363+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:42:20.138363+0000","last_undegraded":"2024-02-10T08:42:20.138363+0000","last_fullsized":"2024-02-10T08:42:20.138363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":311,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515979+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:42:20.515979+0000","last_peered":"2024-02-10T08:42:20.515979+0000","last_clean":"2024-02-10T08:42:20.515979+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:42:20.515979+0000","last_undegraded":"2024-02-10T08:42:20.515979+0000","last_fullsized":"2024-02-10T08:42:20.515979+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":421,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512624+0000","last_change":"2024-02-10T08:40:42.486085+0000","last_active":"2024-02-10T08:42:20.512624+0000","last_peered":"2024-02-10T08:42:20.512624+0000","last_clean":"2024-02-10T08:42:20.512624+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:42:20.512624+0000","last_undegraded":"2024-02-10T08:42:20.512624+0000","last_fullsized":"2024-02-10T08:42:20.512624+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:40:42.486021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.486021+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":411,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511930+0000","last_change":"2024-02-10T08:41:21.599108+0000","last_active":"2024-02-10T08:42:20.511930+0000","last_peered":"2024-02-10T08:42:20.511930+0000","last_clean":"2024-02-10T08:42:20.511930+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:42:20.511930+0000","last_undegraded":"2024-02-10T08:42:20.511930+0000","last_fullsized":"2024-02-10T08:42:20.511930+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:21.599034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.599034+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":553,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138274+0000","last_change":"2024-02-10T08:40:25.631962+0000","last_active":"2024-02-10T08:42:20.138274+0000","last_peered":"2024-02-10T08:42:20.138274+0000","last_clean":"2024-02-10T08:42:20.138274+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:42:20.138274+0000","last_undegraded":"2024-02-10T08:42:20.138274+0000","last_fullsized":"2024-02-10T08:42:20.138274+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:40:25.631854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:25.631854+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":398,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518916+0000","last_change":"2024-02-10T08:39:54.672375+0000","last_active":"2024-02-10T08:42:20.518916+0000","last_peered":"2024-02-10T08:42:20.518916+0000","last_clean":"2024-02-10T08:42:20.518916+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:42:20.518916+0000","last_undegraded":"2024-02-10T08:42:20.518916+0000","last_fullsized":"2024-02-10T08:42:20.518916+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:54.672295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.672295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":352,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518497+0000","last_change":"2024-02-10T08:41:54.063932+0000","last_active":"2024-02-10T08:42:20.518497+0000","last_peered":"2024-02-10T08:42:20.518497+0000","last_clean":"2024-02-10T08:42:20.518497+0000","last_became_active":"2024-02-10T08:40:42.028734+0000","last_became_peered":"2024-02-10T08:40:42.028734+0000","last_unstale":"2024-02-10T08:42:20.518497+0000","last_undegraded":"2024-02-10T08:42:20.518497+0000","last_fullsized":"2024-02-10T08:42:20.518497+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:54.063839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.063839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":418,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.137144+0000","last_change":"2024-02-10T08:41:56.714215+0000","last_active":"2024-02-10T08:42:20.137144+0000","last_peered":"2024-02-10T08:42:20.137144+0000","last_clean":"2024-02-10T08:42:20.137144+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:42:20.137144+0000","last_undegraded":"2024-02-10T08:42:20.137144+0000","last_fullsized":"2024-02-10T08:42:20.137144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514860+0000","last_change":"2024-02-10T08:41:08.390794+0000","last_active":"2024-02-10T08:42:20.514860+0000","last_peered":"2024-02-10T08:42:20.514860+0000","last_clean":"2024-02-10T08:42:20.514860+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:42:20.514860+0000","last_undegraded":"2024-02-10T08:42:20.514860+0000","last_fullsized":"2024-02-10T08:42:20.514860+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.390722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.390722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":480,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519348+0000","last_change":"2024-02-10T08:39:44.685972+0000","last_active":"2024-02-10T08:42:20.519348+0000","last_peered":"2024-02-10T08:42:20.519348+0000","last_clean":"2024-02-10T08:42:20.519348+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:42:20.519348+0000","last_undegraded":"2024-02-10T08:42:20.519348+0000","last_fullsized":"2024-02-10T08:42:20.519348+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:39:44.685900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:44.685900+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":391,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517182+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:42:20.517182+0000","last_peered":"2024-02-10T08:42:20.517182+0000","last_clean":"2024-02-10T08:42:20.517182+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:42:20.517182+0000","last_undegraded":"2024-02-10T08:42:20.517182+0000","last_fullsized":"2024-02-10T08:42:20.517182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":341,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519392+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:42:20.519392+0000","last_peered":"2024-02-10T08:42:20.519392+0000","last_clean":"2024-02-10T08:42:20.519392+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:42:20.519392+0000","last_undegraded":"2024-02-10T08:42:20.519392+0000","last_fullsized":"2024-02-10T08:42:20.519392+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":517,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516046+0000","last_change":"2024-02-10T08:41:44.773002+0000","last_active":"2024-02-10T08:42:20.516046+0000","last_peered":"2024-02-10T08:42:20.516046+0000","last_clean":"2024-02-10T08:42:20.516046+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:42:20.516046+0000","last_undegraded":"2024-02-10T08:42:20.516046+0000","last_fullsized":"2024-02-10T08:42:20.516046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":420,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138594+0000","last_change":"2024-02-10T08:41:24.848074+0000","last_active":"2024-02-10T08:42:20.138594+0000","last_peered":"2024-02-10T08:42:20.138594+0000","last_clean":"2024-02-10T08:42:20.138594+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:42:20.138594+0000","last_undegraded":"2024-02-10T08:42:20.138594+0000","last_fullsized":"2024-02-10T08:42:20.138594+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:41:24.847990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.847990+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":406,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516674+0000","last_change":"2024-02-10T08:41:48.361432+0000","last_active":"2024-02-10T08:42:20.516674+0000","last_peered":"2024-02-10T08:42:20.516674+0000","last_clean":"2024-02-10T08:42:20.516674+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:42:20.516674+0000","last_undegraded":"2024-02-10T08:42:20.516674+0000","last_fullsized":"2024-02-10T08:42:20.516674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.361357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.361357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":394,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514496+0000","last_change":"2024-02-10T08:41:59.569272+0000","last_active":"2024-02-10T08:42:20.514496+0000","last_peered":"2024-02-10T08:42:20.514496+0000","last_clean":"2024-02-10T08:42:20.514496+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:42:20.514496+0000","last_undegraded":"2024-02-10T08:42:20.514496+0000","last_fullsized":"2024-02-10T08:42:20.514496+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.569193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.569193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":316,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513674+0000","last_change":"2024-02-10T08:41:57.599784+0000","last_active":"2024-02-10T08:42:20.513674+0000","last_peered":"2024-02-10T08:42:20.513674+0000","last_clean":"2024-02-10T08:42:20.513674+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:42:20.513674+0000","last_undegraded":"2024-02-10T08:42:20.513674+0000","last_fullsized":"2024-02-10T08:42:20.513674+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":394,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138760+0000","last_change":"2024-02-10T08:40:34.622947+0000","last_active":"2024-02-10T08:42:20.138760+0000","last_peered":"2024-02-10T08:42:20.138760+0000","last_clean":"2024-02-10T08:42:20.138760+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:42:20.138760+0000","last_undegraded":"2024-02-10T08:42:20.138760+0000","last_fullsized":"2024-02-10T08:42:20.138760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:34.622869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":489,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517599+0000","last_change":"2024-02-10T08:41:07.712140+0000","last_active":"2024-02-10T08:42:20.517599+0000","last_peered":"2024-02-10T08:42:20.517599+0000","last_clean":"2024-02-10T08:42:20.517599+0000","last_became_active":"2024-02-10T08:40:42.015369+0000","last_became_peered":"2024-02-10T08:40:42.015369+0000","last_unstale":"2024-02-10T08:42:20.517599+0000","last_undegraded":"2024-02-10T08:42:20.517599+0000","last_fullsized":"2024-02-10T08:42:20.517599+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:41:07.712078+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.712078+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":410,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511474+0000","last_change":"2024-02-10T08:41:46.729506+0000","last_active":"2024-02-10T08:42:20.511474+0000","last_peered":"2024-02-10T08:42:20.511474+0000","last_clean":"2024-02-10T08:42:20.511474+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:42:20.511474+0000","last_undegraded":"2024-02-10T08:42:20.511474+0000","last_fullsized":"2024-02-10T08:42:20.511474+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":390,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139520+0000","last_change":"2024-02-10T08:41:10.710795+0000","last_active":"2024-02-10T08:42:20.139520+0000","last_peered":"2024-02-10T08:42:20.139520+0000","last_clean":"2024-02-10T08:42:20.139520+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:42:20.139520+0000","last_undegraded":"2024-02-10T08:42:20.139520+0000","last_fullsized":"2024-02-10T08:42:20.139520+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":320,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516973+0000","last_change":"2024-02-10T08:40:42.028088+0000","last_active":"2024-02-10T08:42:20.516973+0000","last_peered":"2024-02-10T08:42:20.516973+0000","last_clean":"2024-02-10T08:42:20.516973+0000","last_became_active":"2024-02-10T08:40:42.027869+0000","last_became_peered":"2024-02-10T08:40:42.027869+0000","last_unstale":"2024-02-10T08:42:20.516973+0000","last_undegraded":"2024-02-10T08:42:20.516973+0000","last_fullsized":"2024-02-10T08:42:20.516973+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:08.629124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:08.629124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":411,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518061+0000","last_change":"2024-02-10T08:41:58.271798+0000","last_active":"2024-02-10T08:42:20.518061+0000","last_peered":"2024-02-10T08:42:20.518061+0000","last_clean":"2024-02-10T08:42:20.518061+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:42:20.518061+0000","last_undegraded":"2024-02-10T08:42:20.518061+0000","last_fullsized":"2024-02-10T08:42:20.518061+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.271728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.271728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":379,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518905+0000","last_change":"2024-02-10T08:41:07.950673+0000","last_active":"2024-02-10T08:42:20.518905+0000","last_peered":"2024-02-10T08:42:20.518905+0000","last_clean":"2024-02-10T08:42:20.518905+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:42:20.518905+0000","last_undegraded":"2024-02-10T08:42:20.518905+0000","last_fullsized":"2024-02-10T08:42:20.518905+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:07.950597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.950597+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513672+0000","last_change":"2024-02-10T08:41:04.369613+0000","last_active":"2024-02-10T08:42:20.513672+0000","last_peered":"2024-02-10T08:42:20.513672+0000","last_clean":"2024-02-10T08:42:20.513672+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:42:20.513672+0000","last_undegraded":"2024-02-10T08:42:20.513672+0000","last_fullsized":"2024-02-10T08:42:20.513672+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.369534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.369534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":60,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515904+0000","last_change":"2024-02-10T08:41:58.300293+0000","last_active":"2024-02-10T08:42:20.515904+0000","last_peered":"2024-02-10T08:42:20.515904+0000","last_clean":"2024-02-10T08:42:20.515904+0000","last_became_active":"2024-02-10T08:41:21.266720+0000","last_became_peered":"2024-02-10T08:41:21.266720+0000","last_unstale":"2024-02-10T08:42:20.515904+0000","last_undegraded":"2024-02-10T08:42:20.515904+0000","last_fullsized":"2024-02-10T08:42:20.515904+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.300183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.300183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":355,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517159+0000","last_change":"2024-02-10T08:41:50.202162+0000","last_active":"2024-02-10T08:42:20.517159+0000","last_peered":"2024-02-10T08:42:20.517159+0000","last_clean":"2024-02-10T08:42:20.517159+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:42:20.517159+0000","last_undegraded":"2024-02-10T08:42:20.517159+0000","last_fullsized":"2024-02-10T08:42:20.517159+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:50.202092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:50.202092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":423,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519398+0000","last_change":"2024-02-10T08:41:11.901601+0000","last_active":"2024-02-10T08:42:20.519398+0000","last_peered":"2024-02-10T08:42:20.519398+0000","last_clean":"2024-02-10T08:42:20.519398+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:42:20.519398+0000","last_undegraded":"2024-02-10T08:42:20.519398+0000","last_fullsized":"2024-02-10T08:42:20.519398+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.901502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.901502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":360,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511652+0000","last_change":"2024-02-10T08:41:07.584245+0000","last_active":"2024-02-10T08:42:20.511652+0000","last_peered":"2024-02-10T08:42:20.511652+0000","last_clean":"2024-02-10T08:42:20.511652+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:42:20.511652+0000","last_undegraded":"2024-02-10T08:42:20.511652+0000","last_fullsized":"2024-02-10T08:42:20.511652+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:07.584127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.584127+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514774+0000","last_change":"2024-02-10T08:40:23.406297+0000","last_active":"2024-02-10T08:42:20.514774+0000","last_peered":"2024-02-10T08:42:20.514774+0000","last_clean":"2024-02-10T08:42:20.514774+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:42:20.514774+0000","last_undegraded":"2024-02-10T08:42:20.514774+0000","last_fullsized":"2024-02-10T08:42:20.514774+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.406227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.406227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":314,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138454+0000","last_change":"2024-02-10T08:41:21.839964+0000","last_active":"2024-02-10T08:42:20.138454+0000","last_peered":"2024-02-10T08:42:20.138454+0000","last_clean":"2024-02-10T08:42:20.138454+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:42:20.138454+0000","last_undegraded":"2024-02-10T08:42:20.138454+0000","last_fullsized":"2024-02-10T08:42:20.138454+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":398,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512583+0000","last_change":"2024-02-10T08:41:12.554311+0000","last_active":"2024-02-10T08:42:20.512583+0000","last_peered":"2024-02-10T08:42:20.512583+0000","last_clean":"2024-02-10T08:42:20.512583+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:42:20.512583+0000","last_undegraded":"2024-02-10T08:42:20.512583+0000","last_fullsized":"2024-02-10T08:42:20.512583+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:41:12.554232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.554232+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":353,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517944+0000","last_change":"2024-02-10T08:41:10.521635+0000","last_active":"2024-02-10T08:42:20.517944+0000","last_peered":"2024-02-10T08:42:20.517944+0000","last_clean":"2024-02-10T08:42:20.517944+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:42:20.517944+0000","last_undegraded":"2024-02-10T08:42:20.517944+0000","last_fullsized":"2024-02-10T08:42:20.517944+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.521563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.521563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":396,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513195+0000","last_change":"2024-02-10T08:41:19.563777+0000","last_active":"2024-02-10T08:42:20.513195+0000","last_peered":"2024-02-10T08:42:20.513195+0000","last_clean":"2024-02-10T08:42:20.513195+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:42:20.513195+0000","last_undegraded":"2024-02-10T08:42:20.513195+0000","last_fullsized":"2024-02-10T08:42:20.513195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:19.563700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.563700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":426,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516743+0000","last_change":"2024-02-10T08:42:00.249642+0000","last_active":"2024-02-10T08:42:20.516743+0000","last_peered":"2024-02-10T08:42:20.516743+0000","last_clean":"2024-02-10T08:42:20.516743+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:42:20.516743+0000","last_undegraded":"2024-02-10T08:42:20.516743+0000","last_fullsized":"2024-02-10T08:42:20.516743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":417,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515117+0000","last_change":"2024-02-10T08:41:24.519217+0000","last_active":"2024-02-10T08:42:20.515117+0000","last_peered":"2024-02-10T08:42:20.515117+0000","last_clean":"2024-02-10T08:42:20.515117+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:42:20.515117+0000","last_undegraded":"2024-02-10T08:42:20.515117+0000","last_fullsized":"2024-02-10T08:42:20.515117+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.519144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.519144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518584+0000","last_change":"2024-02-10T08:41:13.978089+0000","last_active":"2024-02-10T08:42:20.518584+0000","last_peered":"2024-02-10T08:42:20.518584+0000","last_clean":"2024-02-10T08:42:20.518584+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:42:20.518584+0000","last_undegraded":"2024-02-10T08:42:20.518584+0000","last_fullsized":"2024-02-10T08:42:20.518584+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.978013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.978013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":397,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519565+0000","last_change":"2024-02-10T08:40:39.891622+0000","last_active":"2024-02-10T08:42:20.519565+0000","last_peered":"2024-02-10T08:42:20.519565+0000","last_clean":"2024-02-10T08:42:20.519565+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:42:20.519565+0000","last_undegraded":"2024-02-10T08:42:20.519565+0000","last_fullsized":"2024-02-10T08:42:20.519565+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.891558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.891558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":394,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519399+0000","last_change":"2024-02-10T08:41:48.065347+0000","last_active":"2024-02-10T08:42:20.519399+0000","last_peered":"2024-02-10T08:42:20.519399+0000","last_clean":"2024-02-10T08:42:20.519399+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:42:20.519399+0000","last_undegraded":"2024-02-10T08:42:20.519399+0000","last_fullsized":"2024-02-10T08:42:20.519399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":354,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517962+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:42:20.517962+0000","last_peered":"2024-02-10T08:42:20.517962+0000","last_clean":"2024-02-10T08:42:20.517962+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:42:20.517962+0000","last_undegraded":"2024-02-10T08:42:20.517962+0000","last_fullsized":"2024-02-10T08:42:20.517962+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":412,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517069+0000","last_change":"2024-02-10T08:41:03.238150+0000","last_active":"2024-02-10T08:42:20.517069+0000","last_peered":"2024-02-10T08:42:20.517069+0000","last_clean":"2024-02-10T08:42:20.517069+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:42:20.517069+0000","last_undegraded":"2024-02-10T08:42:20.517069+0000","last_fullsized":"2024-02-10T08:42:20.517069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.238035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.238035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":449,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512541+0000","last_change":"2024-02-10T08:41:18.579612+0000","last_active":"2024-02-10T08:42:20.512541+0000","last_peered":"2024-02-10T08:42:20.512541+0000","last_clean":"2024-02-10T08:42:20.512541+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:42:20.512541+0000","last_undegraded":"2024-02-10T08:42:20.512541+0000","last_fullsized":"2024-02-10T08:42:20.512541+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:41:18.579532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.579532+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515829+0000","last_change":"2024-02-10T08:39:41.404843+0000","last_active":"2024-02-10T08:42:20.515829+0000","last_peered":"2024-02-10T08:42:20.515829+0000","last_clean":"2024-02-10T08:42:20.515829+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:42:20.515829+0000","last_undegraded":"2024-02-10T08:42:20.515829+0000","last_fullsized":"2024-02-10T08:42:20.515829+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.404728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.404728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":943,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-10T08:42:21.043440+0000","last_change":"2024-02-10T08:41:44.751436+0000","last_active":"2024-02-10T08:42:21.043440+0000","last_peered":"2024-02-10T08:42:21.043440+0000","last_clean":"2024-02-10T08:42:21.043440+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:42:21.043440+0000","last_undegraded":"2024-02-10T08:42:21.043440+0000","last_fullsized":"2024-02-10T08:42:21.043440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:41:44.751371+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.751371+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":536,"num_read_kb":531,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":394,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513593+0000","last_change":"2024-02-10T08:41:15.515804+0000","last_active":"2024-02-10T08:42:20.513593+0000","last_peered":"2024-02-10T08:42:20.513593+0000","last_clean":"2024-02-10T08:42:20.513593+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:42:20.513593+0000","last_undegraded":"2024-02-10T08:42:20.513593+0000","last_fullsized":"2024-02-10T08:42:20.513593+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.515729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.515729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":350,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511425+0000","last_change":"2024-02-10T08:40:22.234338+0000","last_active":"2024-02-10T08:42:20.511425+0000","last_peered":"2024-02-10T08:42:20.511425+0000","last_clean":"2024-02-10T08:42:20.511425+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:42:20.511425+0000","last_undegraded":"2024-02-10T08:42:20.511425+0000","last_fullsized":"2024-02-10T08:42:20.511425+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.234270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.234270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":316,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513335+0000","last_change":"2024-02-10T08:41:53.594734+0000","last_active":"2024-02-10T08:42:20.513335+0000","last_peered":"2024-02-10T08:42:20.513335+0000","last_clean":"2024-02-10T08:42:20.513335+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:42:20.513335+0000","last_undegraded":"2024-02-10T08:42:20.513335+0000","last_fullsized":"2024-02-10T08:42:20.513335+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":493,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518271+0000","last_change":"2024-02-10T08:40:40.098585+0000","last_active":"2024-02-10T08:42:20.518271+0000","last_peered":"2024-02-10T08:42:20.518271+0000","last_clean":"2024-02-10T08:42:20.518271+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:42:20.518271+0000","last_undegraded":"2024-02-10T08:42:20.518271+0000","last_fullsized":"2024-02-10T08:42:20.518271+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:40:40.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.098515+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518213+0000","last_change":"2024-02-10T08:40:23.818554+0000","last_active":"2024-02-10T08:42:20.518213+0000","last_peered":"2024-02-10T08:42:20.518213+0000","last_clean":"2024-02-10T08:42:20.518213+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:42:20.518213+0000","last_undegraded":"2024-02-10T08:42:20.518213+0000","last_fullsized":"2024-02-10T08:42:20.518213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.818434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.818434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":391,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518220+0000","last_change":"2024-02-10T08:41:11.702739+0000","last_active":"2024-02-10T08:42:20.518220+0000","last_peered":"2024-02-10T08:42:20.518220+0000","last_clean":"2024-02-10T08:42:20.518220+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:42:20.518220+0000","last_undegraded":"2024-02-10T08:42:20.518220+0000","last_fullsized":"2024-02-10T08:42:20.518220+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":355,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518843+0000","last_change":"2024-02-10T08:42:04.361526+0000","last_active":"2024-02-10T08:42:20.518843+0000","last_peered":"2024-02-10T08:42:20.518843+0000","last_clean":"2024-02-10T08:42:20.518843+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:42:20.518843+0000","last_undegraded":"2024-02-10T08:42:20.518843+0000","last_fullsized":"2024-02-10T08:42:20.518843+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:04.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.361430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":404,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516405+0000","last_change":"2024-02-10T08:41:54.316045+0000","last_active":"2024-02-10T08:42:20.516405+0000","last_peered":"2024-02-10T08:42:20.516405+0000","last_clean":"2024-02-10T08:42:20.516405+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:42:20.516405+0000","last_undegraded":"2024-02-10T08:42:20.516405+0000","last_fullsized":"2024-02-10T08:42:20.516405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:41:54.315983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.315983+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":421,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511867+0000","last_change":"2024-02-10T08:41:03.584639+0000","last_active":"2024-02-10T08:42:20.511867+0000","last_peered":"2024-02-10T08:42:20.511867+0000","last_clean":"2024-02-10T08:42:20.511867+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:42:20.511867+0000","last_undegraded":"2024-02-10T08:42:20.511867+0000","last_fullsized":"2024-02-10T08:42:20.511867+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:41:03.584573+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.584573+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":412,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512325+0000","last_change":"2024-02-10T08:41:24.622753+0000","last_active":"2024-02-10T08:42:20.512325+0000","last_peered":"2024-02-10T08:42:20.512325+0000","last_clean":"2024-02-10T08:42:20.512325+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:42:20.512325+0000","last_undegraded":"2024-02-10T08:42:20.512325+0000","last_fullsized":"2024-02-10T08:42:20.512325+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.622691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.622691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":311,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515882+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:42:20.515882+0000","last_peered":"2024-02-10T08:42:20.515882+0000","last_clean":"2024-02-10T08:42:20.515882+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:42:20.515882+0000","last_undegraded":"2024-02-10T08:42:20.515882+0000","last_fullsized":"2024-02-10T08:42:20.515882+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":341,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518030+0000","last_change":"2024-02-10T08:39:13.495463+0000","last_active":"2024-02-10T08:42:20.518030+0000","last_peered":"2024-02-10T08:42:20.518030+0000","last_clean":"2024-02-10T08:42:20.518030+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:42:20.518030+0000","last_undegraded":"2024-02-10T08:42:20.518030+0000","last_fullsized":"2024-02-10T08:42:20.518030+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:13.495348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:13.495348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":426,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519163+0000","last_change":"2024-02-10T08:41:22.395964+0000","last_active":"2024-02-10T08:42:20.519163+0000","last_peered":"2024-02-10T08:42:20.519163+0000","last_clean":"2024-02-10T08:42:20.519163+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:42:20.519163+0000","last_undegraded":"2024-02-10T08:42:20.519163+0000","last_fullsized":"2024-02-10T08:42:20.519163+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.395872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.395872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":427,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.138681+0000","last_change":"2024-02-10T08:41:44.633272+0000","last_active":"2024-02-10T08:42:20.138681+0000","last_peered":"2024-02-10T08:42:20.138681+0000","last_clean":"2024-02-10T08:42:20.138681+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:42:20.138681+0000","last_undegraded":"2024-02-10T08:42:20.138681+0000","last_fullsized":"2024-02-10T08:42:20.138681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":424,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.142214+0000","last_change":"2024-02-10T08:40:38.667994+0000","last_active":"2024-02-10T08:42:20.142214+0000","last_peered":"2024-02-10T08:42:20.142214+0000","last_clean":"2024-02-10T08:42:20.142214+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:42:20.142214+0000","last_undegraded":"2024-02-10T08:42:20.142214+0000","last_fullsized":"2024-02-10T08:42:20.142214+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:40:38.667923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:38.667923+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":396,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516632+0000","last_change":"2024-02-10T08:42:05.067474+0000","last_active":"2024-02-10T08:42:20.516632+0000","last_peered":"2024-02-10T08:42:20.516632+0000","last_clean":"2024-02-10T08:42:20.516632+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:42:20.516632+0000","last_undegraded":"2024-02-10T08:42:20.516632+0000","last_fullsized":"2024-02-10T08:42:20.516632+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":346,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517483+0000","last_change":"2024-02-10T08:41:15.304181+0000","last_active":"2024-02-10T08:42:20.517483+0000","last_peered":"2024-02-10T08:42:20.517483+0000","last_clean":"2024-02-10T08:42:20.517483+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:42:20.517483+0000","last_undegraded":"2024-02-10T08:42:20.517483+0000","last_fullsized":"2024-02-10T08:42:20.517483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.304104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.304104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":382,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518205+0000","last_change":"2024-02-10T08:41:23.736997+0000","last_active":"2024-02-10T08:42:20.518205+0000","last_peered":"2024-02-10T08:42:20.518205+0000","last_clean":"2024-02-10T08:42:20.518205+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:42:20.518205+0000","last_undegraded":"2024-02-10T08:42:20.518205+0000","last_fullsized":"2024-02-10T08:42:20.518205+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.736922+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":412,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519087+0000","last_change":"2024-02-10T08:41:21.384611+0000","last_active":"2024-02-10T08:42:20.519087+0000","last_peered":"2024-02-10T08:42:20.519087+0000","last_clean":"2024-02-10T08:42:20.519087+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:42:20.519087+0000","last_undegraded":"2024-02-10T08:42:20.519087+0000","last_fullsized":"2024-02-10T08:42:20.519087+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":392,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517814+0000","last_change":"2024-02-10T08:40:37.417370+0000","last_active":"2024-02-10T08:42:20.517814+0000","last_peered":"2024-02-10T08:42:20.517814+0000","last_clean":"2024-02-10T08:42:20.517814+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:42:20.517814+0000","last_undegraded":"2024-02-10T08:42:20.517814+0000","last_fullsized":"2024-02-10T08:42:20.517814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:37.417303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.417303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":339,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139077+0000","last_change":"2024-02-10T08:39:41.482736+0000","last_active":"2024-02-10T08:42:20.139077+0000","last_peered":"2024-02-10T08:42:20.139077+0000","last_clean":"2024-02-10T08:42:20.139077+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:42:20.139077+0000","last_undegraded":"2024-02-10T08:42:20.139077+0000","last_fullsized":"2024-02-10T08:42:20.139077+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.482673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.482673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":404,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.520530+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:42:20.520530+0000","last_peered":"2024-02-10T08:42:20.520530+0000","last_clean":"2024-02-10T08:42:20.520530+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:42:20.520530+0000","last_undegraded":"2024-02-10T08:42:20.520530+0000","last_fullsized":"2024-02-10T08:42:20.520530+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":406,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515409+0000","last_change":"2024-02-10T08:42:02.183760+0000","last_active":"2024-02-10T08:42:20.515409+0000","last_peered":"2024-02-10T08:42:20.515409+0000","last_clean":"2024-02-10T08:42:20.515409+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:42:20.515409+0000","last_undegraded":"2024-02-10T08:42:20.515409+0000","last_fullsized":"2024-02-10T08:42:20.515409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":389,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513471+0000","last_change":"2024-02-10T08:40:36.384024+0000","last_active":"2024-02-10T08:42:20.513471+0000","last_peered":"2024-02-10T08:42:20.513471+0000","last_clean":"2024-02-10T08:42:20.513471+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:42:20.513471+0000","last_undegraded":"2024-02-10T08:42:20.513471+0000","last_fullsized":"2024-02-10T08:42:20.513471+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.383958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.383958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":343,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518592+0000","last_change":"2024-02-10T08:41:23.032812+0000","last_active":"2024-02-10T08:42:20.518592+0000","last_peered":"2024-02-10T08:42:20.518592+0000","last_clean":"2024-02-10T08:42:20.518592+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:42:20.518592+0000","last_undegraded":"2024-02-10T08:42:20.518592+0000","last_fullsized":"2024-02-10T08:42:20.518592+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":410,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517271+0000","last_change":"2024-02-10T08:38:20.446034+0000","last_active":"2024-02-10T08:42:20.517271+0000","last_peered":"2024-02-10T08:42:20.517271+0000","last_clean":"2024-02-10T08:42:20.517271+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:42:20.517271+0000","last_undegraded":"2024-02-10T08:42:20.517271+0000","last_fullsized":"2024-02-10T08:42:20.517271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:38:20.445970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:38:20.445970+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518151+0000","last_change":"2024-02-10T08:40:35.021264+0000","last_active":"2024-02-10T08:42:20.518151+0000","last_peered":"2024-02-10T08:42:20.518151+0000","last_clean":"2024-02-10T08:42:20.518151+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:42:20.518151+0000","last_undegraded":"2024-02-10T08:42:20.518151+0000","last_fullsized":"2024-02-10T08:42:20.518151+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.021169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.021169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":520,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518736+0000","last_change":"2024-02-10T08:39:43.636652+0000","last_active":"2024-02-10T08:42:20.518736+0000","last_peered":"2024-02-10T08:42:20.518736+0000","last_clean":"2024-02-10T08:42:20.518736+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:42:20.518736+0000","last_undegraded":"2024-02-10T08:42:20.518736+0000","last_fullsized":"2024-02-10T08:42:20.518736+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:39:43.636554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.636554+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":313,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514049+0000","last_change":"2024-02-10T08:41:16.497083+0000","last_active":"2024-02-10T08:42:20.514049+0000","last_peered":"2024-02-10T08:42:20.514049+0000","last_clean":"2024-02-10T08:42:20.514049+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:42:20.514049+0000","last_undegraded":"2024-02-10T08:42:20.514049+0000","last_fullsized":"2024-02-10T08:42:20.514049+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.496995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.496995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":314,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514808+0000","last_change":"2024-02-10T08:41:21.463600+0000","last_active":"2024-02-10T08:42:20.514808+0000","last_peered":"2024-02-10T08:42:20.514808+0000","last_clean":"2024-02-10T08:42:20.514808+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:42:20.514808+0000","last_undegraded":"2024-02-10T08:42:20.514808+0000","last_fullsized":"2024-02-10T08:42:20.514808+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.463527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.463527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":414,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518745+0000","last_change":"2024-02-10T08:42:05.106561+0000","last_active":"2024-02-10T08:42:20.518745+0000","last_peered":"2024-02-10T08:42:20.518745+0000","last_clean":"2024-02-10T08:42:20.518745+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:42:20.518745+0000","last_undegraded":"2024-02-10T08:42:20.518745+0000","last_fullsized":"2024-02-10T08:42:20.518745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":309,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135484+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:42:20.135484+0000","last_peered":"2024-02-10T08:42:20.135484+0000","last_clean":"2024-02-10T08:42:20.135484+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:42:20.135484+0000","last_undegraded":"2024-02-10T08:42:20.135484+0000","last_fullsized":"2024-02-10T08:42:20.135484+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":509,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517420+0000","last_change":"2024-02-10T08:39:00.645428+0000","last_active":"2024-02-10T08:42:20.517420+0000","last_peered":"2024-02-10T08:42:20.517420+0000","last_clean":"2024-02-10T08:42:20.517420+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:42:20.517420+0000","last_undegraded":"2024-02-10T08:42:20.517420+0000","last_fullsized":"2024-02-10T08:42:20.517420+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:39:00.645358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:39:00.645358+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":390,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.142467+0000","last_change":"2024-02-10T08:40:41.538200+0000","last_active":"2024-02-10T08:42:20.142467+0000","last_peered":"2024-02-10T08:42:20.142467+0000","last_clean":"2024-02-10T08:42:20.142467+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:42:20.142467+0000","last_undegraded":"2024-02-10T08:42:20.142467+0000","last_fullsized":"2024-02-10T08:42:20.142467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:41.538139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.538139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":354,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515626+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:42:20.515626+0000","last_peered":"2024-02-10T08:42:20.515626+0000","last_clean":"2024-02-10T08:42:20.515626+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:42:20.515626+0000","last_undegraded":"2024-02-10T08:42:20.515626+0000","last_fullsized":"2024-02-10T08:42:20.515626+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":418,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511686+0000","last_change":"2024-02-10T08:39:43.954609+0000","last_active":"2024-02-10T08:42:20.511686+0000","last_peered":"2024-02-10T08:42:20.511686+0000","last_clean":"2024-02-10T08:42:20.511686+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:42:20.511686+0000","last_undegraded":"2024-02-10T08:42:20.511686+0000","last_fullsized":"2024-02-10T08:42:20.511686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:39:43.954538+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.954538+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":406,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515961+0000","last_change":"2024-02-10T08:41:44.355018+0000","last_active":"2024-02-10T08:42:20.515961+0000","last_peered":"2024-02-10T08:42:20.515961+0000","last_clean":"2024-02-10T08:42:20.515961+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:42:20.515961+0000","last_undegraded":"2024-02-10T08:42:20.515961+0000","last_fullsized":"2024-02-10T08:42:20.515961+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:44.354945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.354945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":407,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516559+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:42:20.516559+0000","last_peered":"2024-02-10T08:42:20.516559+0000","last_clean":"2024-02-10T08:42:20.516559+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:42:20.516559+0000","last_undegraded":"2024-02-10T08:42:20.516559+0000","last_fullsized":"2024-02-10T08:42:20.516559+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":313,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517959+0000","last_change":"2024-02-10T08:41:12.929709+0000","last_active":"2024-02-10T08:42:20.517959+0000","last_peered":"2024-02-10T08:42:20.517959+0000","last_clean":"2024-02-10T08:42:20.517959+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:42:20.517959+0000","last_undegraded":"2024-02-10T08:42:20.517959+0000","last_fullsized":"2024-02-10T08:42:20.517959+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.929592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.929592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":353,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517309+0000","last_change":"2024-02-10T08:41:20.506111+0000","last_active":"2024-02-10T08:42:20.517309+0000","last_peered":"2024-02-10T08:42:20.517309+0000","last_clean":"2024-02-10T08:42:20.517309+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:42:20.517309+0000","last_undegraded":"2024-02-10T08:42:20.517309+0000","last_fullsized":"2024-02-10T08:42:20.517309+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.506032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.506032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":413,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135399+0000","last_change":"2024-02-10T08:41:05.682672+0000","last_active":"2024-02-10T08:42:20.135399+0000","last_peered":"2024-02-10T08:42:20.135399+0000","last_clean":"2024-02-10T08:42:20.135399+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:42:20.135399+0000","last_undegraded":"2024-02-10T08:42:20.135399+0000","last_fullsized":"2024-02-10T08:42:20.135399+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:05.682577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.682577+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":414,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513465+0000","last_change":"2024-02-10T08:41:56.648067+0000","last_active":"2024-02-10T08:42:20.513465+0000","last_peered":"2024-02-10T08:42:20.513465+0000","last_clean":"2024-02-10T08:42:20.513465+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:42:20.513465+0000","last_undegraded":"2024-02-10T08:42:20.513465+0000","last_fullsized":"2024-02-10T08:42:20.513465+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.647993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":394,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511770+0000","last_change":"2024-02-10T08:40:36.354179+0000","last_active":"2024-02-10T08:42:20.511770+0000","last_peered":"2024-02-10T08:42:20.511770+0000","last_clean":"2024-02-10T08:42:20.511770+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:42:20.511770+0000","last_undegraded":"2024-02-10T08:42:20.511770+0000","last_fullsized":"2024-02-10T08:42:20.511770+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.354073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515358+0000","last_change":"2024-02-10T08:38:33.472521+0000","last_active":"2024-02-10T08:42:20.515358+0000","last_peered":"2024-02-10T08:42:20.515358+0000","last_clean":"2024-02-10T08:42:20.515358+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:42:20.515358+0000","last_undegraded":"2024-02-10T08:42:20.515358+0000","last_fullsized":"2024-02-10T08:42:20.515358+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:38:33.472455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:38:33.472455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":350,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.520339+0000","last_change":"2024-02-10T08:39:03.545921+0000","last_active":"2024-02-10T08:42:20.520339+0000","last_peered":"2024-02-10T08:42:20.520339+0000","last_clean":"2024-02-10T08:42:20.520339+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:42:20.520339+0000","last_undegraded":"2024-02-10T08:42:20.520339+0000","last_fullsized":"2024-02-10T08:42:20.520339+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:03.545855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:03.545855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":401,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515788+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:42:20.515788+0000","last_peered":"2024-02-10T08:42:20.515788+0000","last_clean":"2024-02-10T08:42:20.515788+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:42:20.515788+0000","last_undegraded":"2024-02-10T08:42:20.515788+0000","last_fullsized":"2024-02-10T08:42:20.515788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518982+0000","last_change":"2024-02-10T08:41:05.909459+0000","last_active":"2024-02-10T08:42:20.518982+0000","last_peered":"2024-02-10T08:42:20.518982+0000","last_clean":"2024-02-10T08:42:20.518982+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:42:20.518982+0000","last_undegraded":"2024-02-10T08:42:20.518982+0000","last_fullsized":"2024-02-10T08:42:20.518982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.909358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":399,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515347+0000","last_change":"2024-02-10T08:41:24.469189+0000","last_active":"2024-02-10T08:42:20.515347+0000","last_peered":"2024-02-10T08:42:20.515347+0000","last_clean":"2024-02-10T08:42:20.515347+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:42:20.515347+0000","last_undegraded":"2024-02-10T08:42:20.515347+0000","last_fullsized":"2024-02-10T08:42:20.515347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.469114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.469114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":394,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513275+0000","last_change":"2024-02-10T08:41:03.331622+0000","last_active":"2024-02-10T08:42:20.513275+0000","last_peered":"2024-02-10T08:42:20.513275+0000","last_clean":"2024-02-10T08:42:20.513275+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:42:20.513275+0000","last_undegraded":"2024-02-10T08:42:20.513275+0000","last_fullsized":"2024-02-10T08:42:20.513275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":341,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517043+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:42:20.517043+0000","last_peered":"2024-02-10T08:42:20.517043+0000","last_clean":"2024-02-10T08:42:20.517043+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:42:20.517043+0000","last_undegraded":"2024-02-10T08:42:20.517043+0000","last_fullsized":"2024-02-10T08:42:20.517043+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":414,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515793+0000","last_change":"2024-02-10T08:41:03.770782+0000","last_active":"2024-02-10T08:42:20.515793+0000","last_peered":"2024-02-10T08:42:20.515793+0000","last_clean":"2024-02-10T08:42:20.515793+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:42:20.515793+0000","last_undegraded":"2024-02-10T08:42:20.515793+0000","last_fullsized":"2024-02-10T08:42:20.515793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":406,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516486+0000","last_change":"2024-02-10T08:41:52.364114+0000","last_active":"2024-02-10T08:42:20.516486+0000","last_peered":"2024-02-10T08:42:20.516486+0000","last_clean":"2024-02-10T08:42:20.516486+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:42:20.516486+0000","last_undegraded":"2024-02-10T08:42:20.516486+0000","last_fullsized":"2024-02-10T08:42:20.516486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:52.364044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:52.364044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":391,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514996+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:42:20.514996+0000","last_peered":"2024-02-10T08:42:20.514996+0000","last_clean":"2024-02-10T08:42:20.514996+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:42:20.514996+0000","last_undegraded":"2024-02-10T08:42:20.514996+0000","last_fullsized":"2024-02-10T08:42:20.514996+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516704+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:42:20.516704+0000","last_peered":"2024-02-10T08:42:20.516704+0000","last_clean":"2024-02-10T08:42:20.516704+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:42:20.516704+0000","last_undegraded":"2024-02-10T08:42:20.516704+0000","last_fullsized":"2024-02-10T08:42:20.516704+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":314,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516931+0000","last_change":"2024-02-10T08:41:12.525969+0000","last_active":"2024-02-10T08:42:20.516931+0000","last_peered":"2024-02-10T08:42:20.516931+0000","last_clean":"2024-02-10T08:42:20.516931+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:42:20.516931+0000","last_undegraded":"2024-02-10T08:42:20.516931+0000","last_fullsized":"2024-02-10T08:42:20.516931+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.525896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.525896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":352,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514196+0000","last_change":"2024-02-10T08:41:53.573192+0000","last_active":"2024-02-10T08:42:20.514196+0000","last_peered":"2024-02-10T08:42:20.514196+0000","last_clean":"2024-02-10T08:42:20.514196+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:42:20.514196+0000","last_undegraded":"2024-02-10T08:42:20.514196+0000","last_fullsized":"2024-02-10T08:42:20.514196+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:53.573125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.573125+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517864+0000","last_change":"2024-02-10T08:40:27.813479+0000","last_active":"2024-02-10T08:42:20.517864+0000","last_peered":"2024-02-10T08:42:20.517864+0000","last_clean":"2024-02-10T08:42:20.517864+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:42:20.517864+0000","last_undegraded":"2024-02-10T08:42:20.517864+0000","last_fullsized":"2024-02-10T08:42:20.517864+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.813396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.813396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":390,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513313+0000","last_change":"2024-02-10T08:39:47.649049+0000","last_active":"2024-02-10T08:42:20.513313+0000","last_peered":"2024-02-10T08:42:20.513313+0000","last_clean":"2024-02-10T08:42:20.513313+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:42:20.513313+0000","last_undegraded":"2024-02-10T08:42:20.513313+0000","last_fullsized":"2024-02-10T08:42:20.513313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:39:47.648978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:47.648978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":346,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516499+0000","last_change":"2024-02-10T08:41:48.214886+0000","last_active":"2024-02-10T08:42:20.516499+0000","last_peered":"2024-02-10T08:42:20.516499+0000","last_clean":"2024-02-10T08:42:20.516499+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:42:20.516499+0000","last_undegraded":"2024-02-10T08:42:20.516499+0000","last_fullsized":"2024-02-10T08:42:20.516499+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.214822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.214822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":423,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.517784+0000","last_change":"2024-02-10T08:41:46.403355+0000","last_active":"2024-02-10T08:42:20.517784+0000","last_peered":"2024-02-10T08:42:20.517784+0000","last_clean":"2024-02-10T08:42:20.517784+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:42:20.517784+0000","last_undegraded":"2024-02-10T08:42:20.517784+0000","last_fullsized":"2024-02-10T08:42:20.517784+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:46.403281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.403281+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":411,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515565+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:42:20.515565+0000","last_peered":"2024-02-10T08:42:20.515565+0000","last_clean":"2024-02-10T08:42:20.515565+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:42:20.515565+0000","last_undegraded":"2024-02-10T08:42:20.515565+0000","last_fullsized":"2024-02-10T08:42:20.515565+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":397,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513262+0000","last_change":"2024-02-10T08:41:49.644781+0000","last_active":"2024-02-10T08:42:20.513262+0000","last_peered":"2024-02-10T08:42:20.513262+0000","last_clean":"2024-02-10T08:42:20.513262+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:42:20.513262+0000","last_undegraded":"2024-02-10T08:42:20.513262+0000","last_fullsized":"2024-02-10T08:42:20.513262+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":346,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511344+0000","last_change":"2024-02-10T08:41:23.662181+0000","last_active":"2024-02-10T08:42:20.511344+0000","last_peered":"2024-02-10T08:42:20.511344+0000","last_clean":"2024-02-10T08:42:20.511344+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:42:20.511344+0000","last_undegraded":"2024-02-10T08:42:20.511344+0000","last_fullsized":"2024-02-10T08:42:20.511344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":433,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518189+0000","last_change":"2024-02-10T08:41:45.260263+0000","last_active":"2024-02-10T08:42:20.518189+0000","last_peered":"2024-02-10T08:42:20.518189+0000","last_clean":"2024-02-10T08:42:20.518189+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:42:20.518189+0000","last_undegraded":"2024-02-10T08:42:20.518189+0000","last_fullsized":"2024-02-10T08:42:20.518189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":401,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518096+0000","last_change":"2024-02-10T08:41:08.533312+0000","last_active":"2024-02-10T08:42:20.518096+0000","last_peered":"2024-02-10T08:42:20.518096+0000","last_clean":"2024-02-10T08:42:20.518096+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:42:20.518096+0000","last_undegraded":"2024-02-10T08:42:20.518096+0000","last_fullsized":"2024-02-10T08:42:20.518096+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.533249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.533249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":390,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.137240+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:42:20.137240+0000","last_peered":"2024-02-10T08:42:20.137240+0000","last_clean":"2024-02-10T08:42:20.137240+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:42:20.137240+0000","last_undegraded":"2024-02-10T08:42:20.137240+0000","last_fullsized":"2024-02-10T08:42:20.137240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":319,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.514713+0000","last_change":"2024-02-10T08:41:56.265678+0000","last_active":"2024-02-10T08:42:20.514713+0000","last_peered":"2024-02-10T08:42:20.514713+0000","last_clean":"2024-02-10T08:42:20.514713+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:42:20.514713+0000","last_undegraded":"2024-02-10T08:42:20.514713+0000","last_fullsized":"2024-02-10T08:42:20.514713+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.265562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.265562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":444,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518060+0000","last_change":"2024-02-10T08:41:53.828692+0000","last_active":"2024-02-10T08:42:20.518060+0000","last_peered":"2024-02-10T08:42:20.518060+0000","last_clean":"2024-02-10T08:42:20.518060+0000","last_became_active":"2024-02-10T08:40:42.014594+0000","last_became_peered":"2024-02-10T08:40:42.014594+0000","last_unstale":"2024-02-10T08:42:20.518060+0000","last_undegraded":"2024-02-10T08:42:20.518060+0000","last_fullsized":"2024-02-10T08:42:20.518060+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":402,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516470+0000","last_change":"2024-02-10T08:41:57.957397+0000","last_active":"2024-02-10T08:42:20.516470+0000","last_peered":"2024-02-10T08:42:20.516470+0000","last_clean":"2024-02-10T08:42:20.516470+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:42:20.516470+0000","last_undegraded":"2024-02-10T08:42:20.516470+0000","last_fullsized":"2024-02-10T08:42:20.516470+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":391,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.520741+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:42:20.520741+0000","last_peered":"2024-02-10T08:42:20.520741+0000","last_clean":"2024-02-10T08:42:20.520741+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:42:20.520741+0000","last_undegraded":"2024-02-10T08:42:20.520741+0000","last_fullsized":"2024-02-10T08:42:20.520741+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":346,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512101+0000","last_change":"2024-02-10T08:42:01.674000+0000","last_active":"2024-02-10T08:42:20.512101+0000","last_peered":"2024-02-10T08:42:20.512101+0000","last_clean":"2024-02-10T08:42:20.512101+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:42:20.512101+0000","last_undegraded":"2024-02-10T08:42:20.512101+0000","last_fullsized":"2024-02-10T08:42:20.512101+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:01.673932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.673932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":434,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139289+0000","last_change":"2024-02-10T08:41:19.765286+0000","last_active":"2024-02-10T08:42:20.139289+0000","last_peered":"2024-02-10T08:42:20.139289+0000","last_clean":"2024-02-10T08:42:20.139289+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:42:20.139289+0000","last_undegraded":"2024-02-10T08:42:20.139289+0000","last_fullsized":"2024-02-10T08:42:20.139289+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":400,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513037+0000","last_change":"2024-02-10T08:41:13.452076+0000","last_active":"2024-02-10T08:42:20.513037+0000","last_peered":"2024-02-10T08:42:20.513037+0000","last_clean":"2024-02-10T08:42:20.513037+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:42:20.513037+0000","last_undegraded":"2024-02-10T08:42:20.513037+0000","last_fullsized":"2024-02-10T08:42:20.513037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.451989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.451989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":316,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511132+0000","last_change":"2024-02-10T08:41:22.620317+0000","last_active":"2024-02-10T08:42:20.511132+0000","last_peered":"2024-02-10T08:42:20.511132+0000","last_clean":"2024-02-10T08:42:20.511132+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:42:20.511132+0000","last_undegraded":"2024-02-10T08:42:20.511132+0000","last_fullsized":"2024-02-10T08:42:20.511132+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.620255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.620255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":350,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516832+0000","last_change":"2024-02-10T08:40:22.359771+0000","last_active":"2024-02-10T08:42:20.516832+0000","last_peered":"2024-02-10T08:42:20.516832+0000","last_clean":"2024-02-10T08:42:20.516832+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:42:20.516832+0000","last_undegraded":"2024-02-10T08:42:20.516832+0000","last_fullsized":"2024-02-10T08:42:20.516832+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.359703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.359703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":411,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.135571+0000","last_change":"2024-02-10T08:42:03.646444+0000","last_active":"2024-02-10T08:42:20.135571+0000","last_peered":"2024-02-10T08:42:20.135571+0000","last_clean":"2024-02-10T08:42:20.135571+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:42:20.135571+0000","last_undegraded":"2024-02-10T08:42:20.135571+0000","last_fullsized":"2024-02-10T08:42:20.135571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":409,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.513512+0000","last_change":"2024-02-10T08:41:11.401522+0000","last_active":"2024-02-10T08:42:20.513512+0000","last_peered":"2024-02-10T08:42:20.513512+0000","last_clean":"2024-02-10T08:42:20.513512+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:42:20.513512+0000","last_undegraded":"2024-02-10T08:42:20.513512+0000","last_fullsized":"2024-02-10T08:42:20.513512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.401433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.401433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":389,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518189+0000","last_change":"2024-02-10T08:39:52.708474+0000","last_active":"2024-02-10T08:42:20.518189+0000","last_peered":"2024-02-10T08:42:20.518189+0000","last_clean":"2024-02-10T08:42:20.518189+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:42:20.518189+0000","last_undegraded":"2024-02-10T08:42:20.518189+0000","last_fullsized":"2024-02-10T08:42:20.518189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:52.708411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.708411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":322,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.519105+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:42:20.519105+0000","last_peered":"2024-02-10T08:42:20.519105+0000","last_clean":"2024-02-10T08:42:20.519105+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:42:20.519105+0000","last_undegraded":"2024-02-10T08:42:20.519105+0000","last_fullsized":"2024-02-10T08:42:20.519105+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":430,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.511385+0000","last_change":"2024-02-10T08:40:37.664677+0000","last_active":"2024-02-10T08:42:20.511385+0000","last_peered":"2024-02-10T08:42:20.511385+0000","last_clean":"2024-02-10T08:42:20.511385+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:42:20.511385+0000","last_undegraded":"2024-02-10T08:42:20.511385+0000","last_fullsized":"2024-02-10T08:42:20.511385+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:40:37.664586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.664586+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":406,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518157+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:42:20.518157+0000","last_peered":"2024-02-10T08:42:20.518157+0000","last_clean":"2024-02-10T08:42:20.518157+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:42:20.518157+0000","last_undegraded":"2024-02-10T08:42:20.518157+0000","last_fullsized":"2024-02-10T08:42:20.518157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":395,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.139205+0000","last_change":"2024-02-10T08:41:55.669868+0000","last_active":"2024-02-10T08:42:20.139205+0000","last_peered":"2024-02-10T08:42:20.139205+0000","last_clean":"2024-02-10T08:42:20.139205+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:42:20.139205+0000","last_undegraded":"2024-02-10T08:42:20.139205+0000","last_fullsized":"2024-02-10T08:42:20.139205+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":352,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.518190+0000","last_change":"2024-02-10T08:41:16.004246+0000","last_active":"2024-02-10T08:42:20.518190+0000","last_peered":"2024-02-10T08:42:20.518190+0000","last_clean":"2024-02-10T08:42:20.518190+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:42:20.518190+0000","last_undegraded":"2024-02-10T08:42:20.518190+0000","last_fullsized":"2024-02-10T08:42:20.518190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":445,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515887+0000","last_change":"2024-02-10T08:41:19.653285+0000","last_active":"2024-02-10T08:42:20.515887+0000","last_peered":"2024-02-10T08:42:20.515887+0000","last_clean":"2024-02-10T08:42:20.515887+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:42:20.515887+0000","last_undegraded":"2024-02-10T08:42:20.515887+0000","last_fullsized":"2024-02-10T08:42:20.515887+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:41:19.653219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.653219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":313,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.512853+0000","last_change":"2024-02-10T08:40:42.336046+0000","last_active":"2024-02-10T08:42:20.512853+0000","last_peered":"2024-02-10T08:42:20.512853+0000","last_clean":"2024-02-10T08:42:20.512853+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:42:20.512853+0000","last_undegraded":"2024-02-10T08:42:20.512853+0000","last_fullsized":"2024-02-10T08:42:20.512853+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.335958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.335958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":391,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.515869+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:42:20.515869+0000","last_peered":"2024-02-10T08:42:20.515869+0000","last_clean":"2024-02-10T08:42:20.515869+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:42:20.515869+0000","last_undegraded":"2024-02-10T08:42:20.515869+0000","last_fullsized":"2024-02-10T08:42:20.515869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":349,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-02-10T08:42:20.516543+0000","last_change":"2024-02-10T08:42:05.201384+0000","last_active":"2024-02-10T08:42:20.516543+0000","last_peered":"2024-02-10T08:42:20.516543+0000","last_clean":"2024-02-10T08:42:20.516543+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:42:20.516543+0000","last_undegraded":"2024-02-10T08:42:20.516543+0000","last_fullsized":"2024-02-10T08:42:20.516543+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":213,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001479,"num_objects":158,"num_object_clones":2,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13901824,"data_stored":12004445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1875,"ondisk_log_size":1875,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":536,"num_read_kb":531,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299503,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308008,"kb_used_data":2592,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437144,"statfs":{"total":95995035648,"available":95679635456,"internally_reserved":0,"allocated":2654208,"data_stored":991976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.78300000000000003,"15min":0.60499999999999998},"min":{"1min":0.374,"5min":0.246,"15min":0.246},"max":{"1min":0.99299999999999999,"5min":22.143000000000001,"15min":22.143000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.66700000000000004,"15min":0.56899999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93100000000000005,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.627,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.314,"15min":0.24299999999999999},"max":{"1min":1.0509999999999999,"5min":7.593,"15min":7.593},"last":0.308},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65400000000000003,"15min":0.58299999999999996},"min":{"1min":0.35899999999999999,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":1.0089999999999999,"5min":7.7759999999999998,"15min":7.7759999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.61699999999999999,"15min":0.54500000000000004},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27100000000000002},"max":{"1min":0.73199999999999998,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.83299999999999996,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.251,"15min":0.251},"max":{"1min":1.0740000000000001,"5min":19.369,"15min":19.369},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.67300000000000004,"15min":0.56499999999999995},"min":{"1min":0.38400000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.083,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.84399999999999997,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.33100000000000002,"15min":0.27900000000000003},"max":{"1min":1.1459999999999999,"5min":22.102,"15min":22.102},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.45000000000000001},"min":{"1min":0.34899999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":8.827,"15min":8.827},"last":0.438},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56200000000000006,"15min":0.47799999999999998},"min":{"1min":0.28799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72399999999999998,"5min":13.646000000000001,"15min":13.646000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.66600000000000004,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.77600000000000002,"5min":19.326000000000001,"15min":19.326000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77200000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.69399999999999995,"15min":0.56499999999999995},"min":{"1min":0.436,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76300000000000001,"5min":22.687000000000001,"15min":22.687000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.5,"15min":0.503},"min":{"1min":0.39300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72599999999999998,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.56200000000000006}]}]},{"osd":6,"up_from":44,"seq":188978561141,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307724,"kb_used_data":6404,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437428,"statfs":{"total":95995035648,"available":95679926272,"internally_reserved":0,"allocated":6557696,"data_stored":4991728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.19800000000000001},"max":{"1min":0.68200000000000005,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.64600000000000002,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.65000000000000002,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.34200000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.57799999999999996,"15min":0.52600000000000002},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.56599999999999995,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.69499999999999995,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58999999999999997,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72299999999999998,"5min":4.798,"15min":4.798},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59699999999999998,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68999999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.78100000000000003,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":16.823,"15min":16.823},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.51800000000000002},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.372}]},{"osd":4,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":0.67400000000000004,"5min":3.657,"15min":3.657},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45200000000000001,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":0.71099999999999997,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.307}]},{"osd":5,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71199999999999997,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.316},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46700000000000003,"15min":0.436},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.39000000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.47699999999999998},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70099999999999996,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.42299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303888,"kb_used_data":2568,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441264,"statfs":{"total":95995035648,"available":95683854336,"internally_reserved":0,"allocated":2629632,"data_stored":992997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.48299999999999998,"15min":0.40699999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.36799999999999999,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.63,"15min":0.45500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.45600000000000002,"5min":16.166,"15min":16.166},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.103,"15min":0.61299999999999999},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47599999999999998,"5min":34.985999999999997,"15min":34.985999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.85699999999999998,"15min":0.52800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.496,"5min":21.614999999999998,"15min":21.614999999999998},"last":0.378}]},{"osd":3,"last update":"Sat Feb 10 08:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.73299999999999998,"15min":0.51700000000000002},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.39800000000000002,"5min":24.773,"15min":24.773},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.69599999999999995,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.499,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.32500000000000001,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63700000000000001,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.76800000000000002,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.65700000000000003,"15min":0.60699999999999998},"min":{"1min":0.41799999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":9.532,"15min":9.532},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.73399999999999999,"15min":0.65500000000000003},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.53000000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.71499999999999997,"15min":0.60199999999999998},"min":{"1min":0.44400000000000001,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.74399999999999999,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.83399999999999996,"15min":0.66800000000000004},"min":{"1min":0.44,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.71999999999999997,"5min":7.9199999999999999,"15min":7.9199999999999999},"last":0.59099999999999997}]}]},{"osd":0,"up_from":0,"seq":34359738514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411464,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299788,"kb_used_data":2564,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445364,"statfs":{"total":95995035648,"available":95688052736,"internally_reserved":0,"allocated":2625536,"data_stored":991735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.502,"15min":0.40899999999999997},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.42499999999999999,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.311,"5min":0.64700000000000002,"15min":0.46000000000000002},"min":{"1min":0.20599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.44800000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.34899999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.68500000000000005,"15min":0.47399999999999998},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":12.512,"15min":12.512},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.623,"15min":0.46899999999999997},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":0.46600000000000003,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.31}]},{"osd":3,"last update":"Sat Feb 10 08:41:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.48999999999999999,"15min":0.441},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47199999999999998,"5min":4.617,"15min":4.617},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.57099999999999995,"15min":0.48699999999999999},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.22600000000000001},"max":{"1min":0.503,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.68899999999999995,"15min":0.67500000000000004},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76600000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.625},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.751,"15min":0.65300000000000002},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":1.014,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.62}]},{"osd":5,"last update":"Sat Feb 10 08:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.84199999999999997,"15min":0.66000000000000003},"min":{"1min":0.40200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.80000000000000004,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.69699999999999995,"15min":0.622},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.753,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.67000000000000004,"15min":0.66200000000000003},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84099999999999997,"5min":4.3090000000000002,"15min":9.266},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.71299999999999997,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81999999999999995,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:41:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.81299999999999994,"15min":0.68999999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.75900000000000001,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81799999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.59499999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182532,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303708,"kb_used_data":6484,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441444,"statfs":{"total":95995035648,"available":95684038656,"internally_reserved":0,"allocated":6639616,"data_stored":5000652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.67000000000000004,"15min":0.50800000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.52100000000000002,"5min":11.550000000000001,"15min":14.744},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.85299999999999998,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":0.504,"5min":11.808,"15min":11.808},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.747,"15min":0.56599999999999995},"min":{"1min":0.245,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":0.435,"5min":10.444000000000001,"15min":16.634},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.66900000000000004,"15min":0.56499999999999995},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.754,"5min":11.494999999999999,"15min":12.438000000000001},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.65300000000000002,"15min":0.58699999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.51000000000000001,"5min":11.516999999999999,"15min":16.536000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.85299999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.48399999999999999,"5min":17.722000000000001,"15min":23.294},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:42:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.67900000000000005},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":1.9370000000000001,"5min":2.589,"15min":12.371},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.79700000000000004,"15min":0.71899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.008,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.004,"15min":0.71199999999999997},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.73299999999999998,"5min":12.579000000000001,"15min":12.579000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.72699999999999998,"15min":0.65500000000000003},"min":{"1min":0.45100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.873,"5min":11.448,"15min":11.448},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.749,"15min":0.69799999999999995},"min":{"1min":0.41099999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77900000000000003,"5min":10.381,"15min":16.091000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":11.472,"15min":16.475000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.88,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81699999999999995,"5min":14.773999999999999,"15min":14.773999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":1.083,"15min":0.72999999999999998},"min":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75800000000000001,"5min":12.606999999999999,"15min":12.606999999999999},"last":0.55900000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986302,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308004,"kb_used_data":2588,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437148,"statfs":{"total":95995035648,"available":95679639552,"internally_reserved":0,"allocated":2650112,"data_stored":1000775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.85199999999999998,"15min":0.63600000000000001},"min":{"1min":0.26600000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73799999999999999,"5min":19.940999999999999,"15min":19.940999999999999},"last":0.377},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.66300000000000003,"15min":0.56000000000000005},"min":{"1min":0.33000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.754,"5min":9.1760000000000002,"15min":9.1760000000000002},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.73699999999999999,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65500000000000003,"5min":5.774,"15min":5.774},"last":0.434},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.74399999999999999,"15min":0.60499999999999998},"min":{"1min":0.42199999999999999,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73199999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.307}]},{"osd":2,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.75800000000000001,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72799999999999998,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.76200000000000001,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72099999999999997,"5min":10.01,"15min":10.01},"last":0.39800000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.92100000000000004,"15min":0.66000000000000003},"min":{"1min":0.34999999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.1000000000000001,"15min":0.71899999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.72199999999999998,"5min":25.27,"15min":25.27},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.752,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46999999999999997,"15min":0.42399999999999999},"min":{"1min":0.33000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.76400000000000001,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.38400000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.78000000000000003,"5min":8.3119999999999994,"15min":13.342000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.51100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":13.753,"15min":13.753},"last":0.317}]},{"osd":7,"last update":"Sat Feb 10 08:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.61799999999999999,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67700000000000005,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.68000000000000005,"15min":0.58199999999999996},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74199999999999999,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.41499999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790074,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311812,"kb_used_data":6396,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433340,"statfs":{"total":95995035648,"available":95675740160,"internally_reserved":0,"allocated":6549504,"data_stored":4983107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":1.968,"15min":0.995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.78200000000000003,"5min":107.551,"15min":107.551},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.853,"15min":0.96099999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":0.77000000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.7050000000000001,"15min":0.93799999999999994},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71399999999999997,"5min":107.205,"15min":107.205},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.3700000000000001,"15min":0.82799999999999996},"min":{"1min":0.33600000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.74299999999999999,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.8460000000000001,"15min":0.95399999999999996},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.436},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.9259999999999999,"15min":0.995},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.151,"15min":0.72299999999999998},"min":{"1min":0.374,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.008,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":2.5139999999999998,"15min":1.206},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":114.526,"15min":114.526},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":1.149,"15min":0.66200000000000003},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.72399999999999998,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.88500000000000001,"15min":0.57099999999999995},"min":{"1min":0.32800000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.68000000000000005,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.68600000000000005},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":72.241},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.46300000000000002,"15min":0.54800000000000004},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.497}]},{"osd":7,"last update":"Sat Feb 10 08:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.175,"15min":0.71699999999999997},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86199999999999999,"5min":66.844999999999999,"15min":66.844999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.2370000000000001,"15min":0.748},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.90700000000000003,"5min":72.257000000000005,"15min":72.257000000000005},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4000643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:42:26.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph status --format=json 2024-02-10T08:42:26.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:42:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.053120+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.088059+0000 osd.5 (osd.5) 191 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.091634+0000 osd.5 (osd.5) 192 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.447542+0000 mgr.y (mgr.14553) 388 : cluster [DBG] pgmap v636: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:26.032791+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.053120+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.088059+0000 osd.5 (osd.5) 191 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.091634+0000 osd.5 (osd.5) 192 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.447542+0000 mgr.y (mgr.14553) 388 : cluster [DBG] pgmap v636: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:26.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:26.032791+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-02-10T08:42:26.506 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:26.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.053120+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-02-10T08:42:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.088059+0000 osd.5 (osd.5) 191 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:42:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.091634+0000 osd.5 (osd.5) 192 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:42:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.447542+0000 mgr.y (mgr.14553) 388 : cluster [DBG] pgmap v636: 218 pgs: 20 creating+activating, 198 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:26.032791+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-02-10T08:42:26.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:26.643 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:26.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:26.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:26.798 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:26.818 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:26.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:26.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:26.952 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:26.972 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:26.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:27.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:27.104 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.123 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.070+0000 7fa984b5b700 1 -- 172.21.15.107:0/1921237128 >> v1:172.21.15.107:6790/0 conn(0x7fa980104330 legacy=0x7fa9801067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.070+0000 7fa984b5b700 1 -- 172.21.15.107:0/1921237128 shutdown_connections 2024-02-10T08:42:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.070+0000 7fa984b5b700 1 -- 172.21.15.107:0/1921237128 >> 172.21.15.107:0/1921237128 conn(0x7fa9800f3640 msgr2=0x7fa9800f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:27.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.070+0000 7fa984b5b700 1 -- 172.21.15.107:0/1921237128 shutdown_connections 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.070+0000 7fa984b5b700 1 -- 172.21.15.107:0/1921237128 wait complete. 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 librados: starting msgr at 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 librados: starting objecter 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 Processor -- start 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 -- start start 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 librados: setting wanted keys 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.071+0000 7fa984b5b700 1 librados: calling monclient init 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9800fa980 con 0x7fa980104330 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9801187f0 con 0x7fa9801007f0 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9801199d0 con 0x7fa9800fda50 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa97e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa9800fda50 0x7fa9800f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54230/0 (socket says 172.21.15.107:54230) 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa97e59c700 1 -- 172.21.15.107:0/1726237017 learned_addr learned my addr 172.21.15.107:0/1726237017 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:27.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter ms_handle_connect 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter ms_handle_connect 0x7fa980104330 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter ms_handle_connect 0x7fa9801007f0 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117830143 0 0) 0x7fa9801199d0 con 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9680033f0 con 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480765954 0 0) 0x7fa9800fa980 con 0x7fa980104330 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9801199d0 con 0x7fa980104330 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414800866 0 0) 0x7fa9801187f0 con 0x7fa9801007f0 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9800fa980 con 0x7fa9801007f0 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3796168372 0 0) 0x7fa9680033f0 con 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9801187f0 con 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa9740029b0 con 0x7fa9800fda50 2024-02-10T08:42:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2388694547 0 0) 0x7fa9801199d0 con 0x7fa980104330 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9680033f0 con 0x7fa980104330 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048028770 0 0) 0x7fa9800fa980 con 0x7fa9801007f0 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9801199d0 con 0x7fa9801007f0 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa978002f70 con 0x7fa980104330 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa970001840 con 0x7fa9801007f0 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243231236 0 0) 0x7fa9801187f0 con 0x7fa9800fda50 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.195:6789/0 conn(0x7fa9801007f0 legacy=0x7fa980112990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6789/0 conn(0x7fa980104330 legacy=0x7fa9801160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98011abb0 con 0x7fa9800fda50 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa980117840 con 0x7fa9800fda50 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 10 client.53678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa980117cb0 con 0x7fa9800fda50 2024-02-10T08:42:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa984b5b700 1 librados: init done 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa9740018d0 con 0x7fa9800fda50 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.072+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa974005ba0 con 0x7fa9800fda50 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.073+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa960004f00 con 0x7fa9800fda50 2024-02-10T08:42:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa974001430 con 0x7fa9800fda50 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(435..435 src has 330..435) v4 ==== 8222+0+0 (unknown 1144151291 0 0) 0x7fa974078ed0 con 0x7fa9800fda50 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 osd_map(435..435 src has 330..435) v4 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 3 client.53678.objecter handle_osd_map got epochs [435,435] > 0 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 3 client.53678.objecter handle_osd_map decoding full epoch 435 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 20 client.53678.objecter dump_active .. 0 homeless 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v435) v1 ==== 54+0+16 (unknown 1036075195 0 2420000192) 0x7fa97407b270 con 0x7fa9800fda50 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa968063c80 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa984b5b700 10 client.53678.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.074+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 1 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 1 2024-02-10T08:42:27.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 0 1 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 4 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 10 client.53678.objecter _send_command 1 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960004950 con 0x7fa960004530 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa960004530 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960004950 con 0x7fa960004530 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 1 = system:0 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 4 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.075+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 0 1 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 10 client.53678.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 2 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 2 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa9600051a0 osd=0 3 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 0 2 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 4 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 10 client.53678.objecter _send_command 2 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.090+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa960004d30 con 0x7fa960004530 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa960004d30 con 0x7fa960004530 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 4 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 0 2 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 10 client.53678.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 3 2024-02-10T08:42:27.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 1 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 4 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 10 client.53678.objecter _send_command 3 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa96000c000 con 0x7fa960004150 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.092+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa960004150 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.093+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa96000c000 con 0x7fa960004150 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.093+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.093+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 3 = system:0 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.093+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.093+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 1 3 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 10 client.53678.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960003fe0 osd=1 3 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 1 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 10 client.53678.objecter _send_command 4 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.107+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa960001850 con 0x7fa960004150 2024-02-10T08:42:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa960001850 con 0x7fa960004150 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 4 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 1 4 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 10 client.53678.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 5 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 5 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 2 5 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 4 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 10 client.53678.objecter _send_command 5 2024-02-10T08:42:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960003de0 con 0x7fa96000d2f0 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.109+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa96000d2f0 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.110+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960003de0 con 0x7fa96000d2f0 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.110+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.110+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 5 = system:0 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.110+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 4 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.110+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 2 5 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 10 client.53678.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 6 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 6 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960003630 osd=2 3 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 2 6 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 4 2024-02-10T08:42:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 10 client.53678.objecter _send_command 6 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.124+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa96000e690 con 0x7fa96000d2f0 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa96000e690 con 0x7fa96000d2f0 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 4 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 2 6 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 10 client.53678.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 7 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 7 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 3 7 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 4 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 10 client.53678.objecter _send_command 7 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.125+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960015a30 con 0x7fa960011eb0 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa960011eb0 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960015a30 con 0x7fa960011eb0 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 7 = system:0 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 4 2024-02-10T08:42:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.126+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 3 7 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 10 client.53678.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 8 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 8 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa9600033a0 osd=3 3 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 3 8 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 4 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 10 client.53678.objecter _send_command 8 2024-02-10T08:42:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.141+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa96000e100 con 0x7fa960011eb0 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa96000e100 con 0x7fa960011eb0 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 4 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 3 8 2024-02-10T08:42:27.146 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 10 client.53678.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 9 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 9 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 4 9 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 4 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 10 client.53678.objecter _send_command 9 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.142+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa96000e400 con 0x7fa96000ce50 2024-02-10T08:42:27.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa96000ce50 2024-02-10T08:42:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa96000e400 con 0x7fa96000ce50 2024-02-10T08:42:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 9 = system:0 2024-02-10T08:42:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 4 2024-02-10T08:42:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.143+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 4 9 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 10 client.53678.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 10 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 10 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa96000e100 osd=4 3 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 4 10 2024-02-10T08:42:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 4 2024-02-10T08:42:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 10 client.53678.objecter _send_command 10 2024-02-10T08:42:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.158+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa96000e400 con 0x7fa96000ce50 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.159+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa96000e400 con 0x7fa96000ce50 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.159+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.159+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.159+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 4 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.159+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 4 10 2024-02-10T08:42:27.161 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 10 client.53678.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 11 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 11 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 5 11 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 4 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 10 client.53678.objecter _send_command 11 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960016d40 con 0x7fa9600167f0 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.160+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa9600167f0 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.161+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960016d40 con 0x7fa9600167f0 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.161+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.161+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 11 = system:0 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.161+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 4 2024-02-10T08:42:27.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.161+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 5 11 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 10 client.53678.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 12 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 12 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa96000e400 osd=5 3 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 5 12 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 4 2024-02-10T08:42:27.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 10 client.53678.objecter _send_command 12 2024-02-10T08:42:27.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.175+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa960016d40 con 0x7fa9600167f0 2024-02-10T08:42:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.176+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa960016d40 con 0x7fa9600167f0 2024-02-10T08:42:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.176+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.176+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.176+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 4 2024-02-10T08:42:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.176+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 5 12 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 10 client.53678.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 13 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 13 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 6 13 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 4 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 10 client.53678.objecter _send_command 13 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960026f50 con 0x7fa960023440 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.177+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa960023440 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.178+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960026f50 con 0x7fa960023440 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.178+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.178+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 13 = system:0 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.178+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 4 2024-02-10T08:42:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.178+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 6 13 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 10 client.53678.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 14 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 14 2024-02-10T08:42:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960016d40 osd=6 3 2024-02-10T08:42:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 6 14 2024-02-10T08:42:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 4 2024-02-10T08:42:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 10 client.53678.objecter _send_command 14 2024-02-10T08:42:27.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.192+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa960026f50 con 0x7fa960023440 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.193+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa960026f50 con 0x7fa960023440 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.193+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.193+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.193+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 4 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.193+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 6 14 2024-02-10T08:42:27.195 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 10 client.53678.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 15 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 15 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 7 15 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 4 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 10 client.53678.objecter _send_command 15 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa960028170 con 0x7fa960028c10 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.194+0000 7fa96f7fe700 10 client.53678.objecter ms_handle_connect 0x7fa960028c10 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.195+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa960028170 con 0x7fa960028c10 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.195+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.195+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 15 = system:0 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.195+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 4 2024-02-10T08:42:27.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.195+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 7 15 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 10 client.53678.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign -1 16 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 20 client.53678.objecter _get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_remove -1 16 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 20 client.53678.objecter get_session s=0x7fa960026f50 osd=7 3 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 15 client.53678.objecter _session_command_op_assign 7 16 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 4 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 10 client.53678.objecter _send_command 16 2024-02-10T08:42:27.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.211+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa960028170 con 0x7fa960028c10 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.213+0000 7fa96f7fe700 1 -- 172.21.15.107:0/1726237017 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa960028170 con 0x7fa960028c10 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.213+0000 7fa96f7fe700 10 client.53678.objecter ms_dispatch 0x7fa980001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.213+0000 7fa96f7fe700 10 client.53678.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.213+0000 7fa96f7fe700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 4 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.213+0000 7fa96f7fe700 15 client.53678.objecter _session_command_op_remove 7 16 2024-02-10T08:42:27.215 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 librados: watch_flush enter 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 librados: watch_flush exit 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.0 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6801/3321596972 conn(0x7fa960004530 legacy=0x7fa960002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600051a0 osd=0 1 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.1 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6805/133516126 conn(0x7fa960004150 legacy=0x7fa96000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003fe0 osd=1 1 2024-02-10T08:42:27.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.2 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6809/3575881728 conn(0x7fa96000d2f0 legacy=0x7fa96000d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960003630 osd=2 1 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.3 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6813/2871825691 conn(0x7fa960011eb0 legacy=0x7fa9600142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa9600033a0 osd=3 1 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.4 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa97ed9d700 1 -- 172.21.15.107:0/1726237017 reap_dead start 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.195:6800/2499671285 conn(0x7fa96000ce50 legacy=0x7fa960015bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa97ed9d700 1 -- 172.21.15.107:0/1726237017 reap_dead start 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e100 osd=4 1 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.5 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.195:6804/4178321486 conn(0x7fa9600167f0 legacy=0x7fa960021810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa96000e400 osd=5 1 2024-02-10T08:42:27.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.6 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.195:6808/1417479017 conn(0x7fa960023440 legacy=0x7fa960025830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960016d40 osd=6 1 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter close_session for osd.7 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.195:6812/1934176928 conn(0x7fa960028c10 legacy=0x7fa960029090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter put_session s=0x7fa960026f50 osd=7 1 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 20 client.53678.objecter shutdown clearing up homeless session... 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 10 client.53678.objecter successfully canceled tick 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa968063c80 legacy=0x7fa968066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> v1:172.21.15.107:6790/0 conn(0x7fa9800fda50 legacy=0x7fa9800f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa97ed9d700 1 -- 172.21.15.107:0/1726237017 reap_dead start 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa97ed9d700 1 -- 172.21.15.107:0/1726237017 reap_dead start 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 shutdown_connections 2024-02-10T08:42:27.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 >> 172.21.15.107:0/1726237017 conn(0x7fa9800f3640 msgr2=0x7fa9801077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 shutdown_connections 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 -- 172.21.15.107:0/1726237017 wait complete. 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:27.214+0000 7fa984b5b700 1 librados: shutdown 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.220 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:42:27.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:27.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.479087+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:42:27.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.482779+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:42:27.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.594441+0000 osd.4 (osd.4) 188 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:42:27.252 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:25.595372+0000 osd.4 (osd.4) 189 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:42:27.253 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.271 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:27.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:27.402 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.420 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.479087+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:42:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.482779+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:42:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.594441+0000 osd.4 (osd.4) 188 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:42:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:25.595372+0000 osd.4 (osd.4) 189 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:42:27.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.479087+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:42:27.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.482779+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:42:27.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.594441+0000 osd.4 (osd.4) 188 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:42:27.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:25.595372+0000 osd.4 (osd.4) 189 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:42:27.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:27.555 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.577 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:27.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.725 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:27.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:27.855 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:27.873 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:27.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:27.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:28.057 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:28.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:28.067 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:28.067 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:28.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:28.280 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:28.289 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:28.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:28.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:26.386704+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:26.387994+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:26.613853+0000 osd.4 (osd.4) 190 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:26.614956+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:27.075965+0000 mon.c (mon.2) 395 : audit [DBG] from='client.? v1:172.21.15.107:0/1726237017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:27.080206+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:27.081059+0000 osd.3 (osd.3) 202 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:42:28.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:27.448410+0000 mgr.y (mgr.14553) 389 : cluster [DBG] pgmap v638: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:26.386704+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:26.387994+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:26.613853+0000 osd.4 (osd.4) 190 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:26.614956+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:27.075965+0000 mon.c (mon.2) 395 : audit [DBG] from='client.? v1:172.21.15.107:0/1726237017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:27.080206+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:27.081059+0000 osd.3 (osd.3) 202 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:42:28.420 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:27.448410+0000 mgr.y (mgr.14553) 389 : cluster [DBG] pgmap v638: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-02-10T08:42:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:28.518 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:28.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:26.386704+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:26.387994+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:26.613853+0000 osd.4 (osd.4) 190 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:26.614956+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:27.075965+0000 mon.c (mon.2) 395 : audit [DBG] from='client.? v1:172.21.15.107:0/1726237017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:27.080206+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:27.081059+0000 osd.3 (osd.3) 202 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:42:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:27.448410+0000 mgr.y (mgr.14553) 389 : cluster [DBG] pgmap v638: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-02-10T08:42:28.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:28.651 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:27.505943+0000 osd.1 (osd.1) 201 : cluster [DBG] 60.9 scrub starts 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:27.509418+0000 osd.1 (osd.1) 202 : cluster [DBG] 60.9 scrub ok 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.049994+0000 osd.3 (osd.3) 203 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.051416+0000 osd.3 (osd.3) 204 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.086664+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:42:29.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.090107+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.649132+0000 osd.4 (osd.4) 192 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.650870+0000 osd.4 (osd.4) 193 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:27.505943+0000 osd.1 (osd.1) 201 : cluster [DBG] 60.9 scrub starts 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:27.509418+0000 osd.1 (osd.1) 202 : cluster [DBG] 60.9 scrub ok 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.049994+0000 osd.3 (osd.3) 203 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.051416+0000 osd.3 (osd.3) 204 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.086664+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.090107+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.649132+0000 osd.4 (osd.4) 192 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:42:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.650870+0000 osd.4 (osd.4) 193 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:42:29.500 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:29.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:29.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:27.505943+0000 osd.1 (osd.1) 201 : cluster [DBG] 60.9 scrub starts 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:27.509418+0000 osd.1 (osd.1) 202 : cluster [DBG] 60.9 scrub ok 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.049994+0000 osd.3 (osd.3) 203 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.051416+0000 osd.3 (osd.3) 204 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.086664+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.090107+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.649132+0000 osd.4 (osd.4) 192 : cluster [DBG] 60.2 scrub starts 2024-02-10T08:42:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.650870+0000 osd.4 (osd.4) 193 : cluster [DBG] 60.2 scrub ok 2024-02-10T08:42:29.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:29.631 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:29.782 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:29.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:29.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:29.917 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.550261+0000 osd.1 (osd.1) 203 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:42:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:28.553561+0000 osd.1 (osd.1) 204 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:42:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.047961+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.f scrub starts 2024-02-10T08:42:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.051442+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.f scrub ok 2024-02-10T08:42:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.249686+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.250455+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.448995+0000 mgr.y (mgr.14553) 390 : cluster [DBG] pgmap v639: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.550261+0000 osd.1 (osd.1) 203 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:28.553561+0000 osd.1 (osd.1) 204 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.047961+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.f scrub starts 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.051442+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.f scrub ok 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.249686+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.250455+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:42:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.448995+0000 mgr.y (mgr.14553) 390 : cluster [DBG] pgmap v639: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T08:42:30.466 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:30.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.550261+0000 osd.1 (osd.1) 203 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:28.553561+0000 osd.1 (osd.1) 204 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.047961+0000 osd.5 (osd.5) 195 : cluster [DBG] 5.f scrub starts 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.051442+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.f scrub ok 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.249686+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.250455+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:42:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.448995+0000 mgr.y (mgr.14553) 390 : cluster [DBG] pgmap v639: 218 pgs: 1 active+clean+snaptrim, 217 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-10T08:42:30.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:30.596 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:30.843 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:30.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:30.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:42:30.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:30.979 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.000 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:31.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:31.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.593024+0000 osd.1 (osd.1) 205 : cluster [DBG] 5.c scrub starts 2024-02-10T08:42:31.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:29.596093+0000 osd.1 (osd.1) 206 : cluster [DBG] 5.c scrub ok 2024-02-10T08:42:31.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.096287+0000 osd.5 (osd.5) 197 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:42:31.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.099758+0000 osd.5 (osd.5) 198 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:42:31.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.717438+0000 osd.4 (osd.4) 194 : cluster [DBG] 60.a scrub starts 2024-02-10T08:42:31.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.719052+0000 osd.4 (osd.4) 195 : cluster [DBG] 60.a scrub ok 2024-02-10T08:42:31.132 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.150 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:31.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.593024+0000 osd.1 (osd.1) 205 : cluster [DBG] 5.c scrub starts 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:29.596093+0000 osd.1 (osd.1) 206 : cluster [DBG] 5.c scrub ok 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.096287+0000 osd.5 (osd.5) 197 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.099758+0000 osd.5 (osd.5) 198 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.717438+0000 osd.4 (osd.4) 194 : cluster [DBG] 60.a scrub starts 2024-02-10T08:42:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.719052+0000 osd.4 (osd.4) 195 : cluster [DBG] 60.a scrub ok 2024-02-10T08:42:31.299 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.593024+0000 osd.1 (osd.1) 205 : cluster [DBG] 5.c scrub starts 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:29.596093+0000 osd.1 (osd.1) 206 : cluster [DBG] 5.c scrub ok 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.096287+0000 osd.5 (osd.5) 197 : cluster [DBG] 60.11 scrub starts 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.099758+0000 osd.5 (osd.5) 198 : cluster [DBG] 60.11 scrub ok 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.717438+0000 osd.4 (osd.4) 194 : cluster [DBG] 60.a scrub starts 2024-02-10T08:42:31.397 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.719052+0000 osd.4 (osd.4) 195 : cluster [DBG] 60.a scrub ok 2024-02-10T08:42:31.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:31.429 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.448 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:31.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:31.578 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.596 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:31.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:31.721 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:31.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:31.726 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.745 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:31.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:31.875 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:31.893 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:31.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:31.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:32.023 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:32.041 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:32.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.491553+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:30.494585+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.197246+0000 osd.2 (osd.2) 157 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.200818+0000 osd.2 (osd.2) 158 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.449837+0000 mgr.y (mgr.14553) 391 : cluster [DBG] pgmap v640: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:31.479902+0000 mgr.y (mgr.14553) 392 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:32.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.733831+0000 osd.4 (osd.4) 196 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:42:32.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.735507+0000 osd.4 (osd.4) 197 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:42:32.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:32.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.491553+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:30.494585+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.197246+0000 osd.2 (osd.2) 157 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.200818+0000 osd.2 (osd.2) 158 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.449837+0000 mgr.y (mgr.14553) 391 : cluster [DBG] pgmap v640: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:31.479902+0000 mgr.y (mgr.14553) 392 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.733831+0000 osd.4 (osd.4) 196 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:42:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.735507+0000 osd.4 (osd.4) 197 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:42:32.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:32.332 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:32.352 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:32.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.491553+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:42:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:30.494585+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:42:32.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.197246+0000 osd.2 (osd.2) 157 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:42:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.200818+0000 osd.2 (osd.2) 158 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:42:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.449837+0000 mgr.y (mgr.14553) 391 : cluster [DBG] pgmap v640: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T08:42:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:31.479902+0000 mgr.y (mgr.14553) 392 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.733831+0000 osd.4 (osd.4) 196 : cluster [DBG] 60.1 scrub starts 2024-02-10T08:42:32.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.735507+0000 osd.4 (osd.4) 197 : cluster [DBG] 60.1 scrub ok 2024-02-10T08:42:32.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:32.483 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:32.735 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:32.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:32.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:32.866 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:32.919 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:32.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:33.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.619778+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:42:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:31.622660+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:42:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.011863+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:42:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.012851+0000 osd.3 (osd.3) 206 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:42:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.246998+0000 osd.2 (osd.2) 159 : cluster [DBG] 6.f scrub starts 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.249997+0000 osd.2 (osd.2) 160 : cluster [DBG] 6.f scrub ok 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:32.305717+0000 mon.c (mon.2) 396 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.700681+0000 osd.4 (osd.4) 198 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.704162+0000 osd.4 (osd.4) 199 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:32.794851+0000 mon.b (mon.1) 589 : audit [DBG] from='client.? v1:172.21.15.107:0/2574481547' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:32.837364+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:32.838055+0000 mon.c (mon.2) 398 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:32.842645+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.619778+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:42:33.307 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:31.622660+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.011863+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.012851+0000 osd.3 (osd.3) 206 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.246998+0000 osd.2 (osd.2) 159 : cluster [DBG] 6.f scrub starts 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.249997+0000 osd.2 (osd.2) 160 : cluster [DBG] 6.f scrub ok 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:32.305717+0000 mon.c (mon.2) 396 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.700681+0000 osd.4 (osd.4) 198 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.704162+0000 osd.4 (osd.4) 199 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:32.794851+0000 mon.b (mon.1) 589 : audit [DBG] from='client.? v1:172.21.15.107:0/2574481547' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:32.837364+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:32.838055+0000 mon.c (mon.2) 398 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:42:33.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:32.842645+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:33.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:33.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:33.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.619778+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:31.622660+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.011863+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.012851+0000 osd.3 (osd.3) 206 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.246998+0000 osd.2 (osd.2) 159 : cluster [DBG] 6.f scrub starts 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.249997+0000 osd.2 (osd.2) 160 : cluster [DBG] 6.f scrub ok 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:32.305717+0000 mon.c (mon.2) 396 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.700681+0000 osd.4 (osd.4) 198 : cluster [DBG] 60.7 scrub starts 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.704162+0000 osd.4 (osd.4) 199 : cluster [DBG] 60.7 scrub ok 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:32.794851+0000 mon.b (mon.1) 589 : audit [DBG] from='client.? v1:172.21.15.107:0/2574481547' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:32.837364+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:32.838055+0000 mon.c (mon.2) 398 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:42:33.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:32.842645+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:33.444 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:33.464 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:33.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:33.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:33.597 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.534+0000 7fe5075e9700 1 -- 172.21.15.107:0/2910127946 >> v1:172.21.15.107:6789/0 conn(0x7fe5000fda50 legacy=0x7fe5000fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- 172.21.15.107:0/2910127946 shutdown_connections 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- 172.21.15.107:0/2910127946 >> 172.21.15.107:0/2910127946 conn(0x7fe5000f3640 msgr2=0x7fe5000f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- 172.21.15.107:0/2910127946 shutdown_connections 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- 172.21.15.107:0/2910127946 wait complete. 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 librados: starting msgr at 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 librados: starting objecter 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 Processor -- start 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- start start 2024-02-10T08:42:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 librados: setting wanted keys 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 librados: calling monclient init 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5000fd000 con 0x7fe5000fda50 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5000fd1f0 con 0x7fe500104330 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe5075e9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5000fd3e0 con 0x7fe5001007f0 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe505385700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5000fda50 0x7fe5000fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60532/0 (socket says 172.21.15.107:60532) 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe505385700 1 -- 172.21.15.107:0/2574481547 learned_addr learned my addr 172.21.15.107:0/2574481547 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.535+0000 7fe504b84700 1 --1- 172.21.15.107:0/2574481547 >> v1:172.21.15.107:6790/0 conn(0x7fe5001007f0 0x7fe500114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37008/0 (socket says 172.21.15.107:37008) 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter ms_handle_connect 0x7fe5000fda50 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter ms_handle_connect 0x7fe5001007f0 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter ms_handle_connect 0x7fe500104330 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156571224 0 0) 0x7fe5000fd000 con 0x7fe5000fda50 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ec003030 con 0x7fe5000fda50 2024-02-10T08:42:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095074538 0 0) 0x7fe5000fd3e0 con 0x7fe5001007f0 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5000fd000 con 0x7fe5001007f0 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556701371 0 0) 0x7fe5000fd1f0 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5000fd3e0 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3435702454 0 0) 0x7fe5000fd3e0 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5000fd1f0 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 657802107 0 0) 0x7fe4ec003030 con 0x7fe5000fda50 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5000fd3e0 con 0x7fe5000fda50 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3382504748 0 0) 0x7fe5000fd000 con 0x7fe5001007f0 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ec003030 con 0x7fe5001007f0 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe4f8001b80 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe4fc002070 con 0x7fe5000fda50 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe4f0001800 con 0x7fe5001007f0 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1653754081 0 0) 0x7fe5000fd1f0 con 0x7fe500104330 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 >> v1:172.21.15.107:6790/0 conn(0x7fe5001007f0 legacy=0x7fe500114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 >> v1:172.21.15.107:6789/0 conn(0x7fe5000fda50 legacy=0x7fe5000fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50011cc00 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe500119c30 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe5075e9700 10 client.48400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50011a0a0 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe5075e9700 1 librados: init done 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe4f80024f0 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.536+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe4f8003540 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.537+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e4004f20 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe4f8002050 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(435..435 src has 330..435) v4 ==== 8222+0+0 (unknown 1144151291 0 0) 0x7fe4f80766a0 con 0x7fe500104330 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 10 client.48400.objecter ms_dispatch 0x7fe500001550 osd_map(435..435 src has 330..435) v4 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 3 client.48400.objecter handle_osd_map got epochs [435,435] > 0 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 3 client.48400.objecter handle_osd_map decoding full epoch 435 2024-02-10T08:42:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 20 client.48400.objecter dump_active .. 0 homeless 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":767,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":435,"num_osds":8,"num_up_osds":8,"osd_up_since":1707553990,"num_in_osds":7,"osd_in_since":1707554228,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":218}],"num_pgs":218,"num_pools":9,"num_objects":357,"data_bytes":4007314,"bytes_used":2195058688,"bytes_avail":669770190848,"bytes_total":671965249536,"read_bytes_sec":2629,"write_bytes_sec":1383,"read_op_per_sec":3,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"}},"servicemap":{"epoch":20,"modified":"2024-02-10T08:37:56.474042+0000","services":{"rgw":{"daemons":{"summary":"","24391":{"start_epoch":12,"start_stamp":"2024-02-10T08:33:25.211076+0000","gid":24391,"addr":"172.21.15.107:0/4187249279","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi107","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi107","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32537332","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"51e7526a-257b-41e2-b2a3-cd68464b4484","zone_name":"default","zonegroup_id":"5a6ddad7-fa42-479d-91fc-f2f04864c218","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.538+0000 7fe4f67fc700 10 client.48400.objecter ms_handle_connect 0x7fe4ec063e20 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.543+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe4f80499f0 con 0x7fe500104330 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.792+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe4e40051d0 con 0x7fe500104330 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.793+0000 7fe4f67fc700 1 -- 172.21.15.107:0/2574481547 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2474 (unknown 3226119948 0 1196088734) 0x7fe4f8047e00 con 0x7fe500104330 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 10 librados: watch_flush enter 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 10 librados: watch_flush exit 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 20 client.48400.objecter shutdown clearing up homeless session... 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 10 client.48400.objecter successfully canceled tick 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe4ec063e20 legacy=0x7fe4ec0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 >> v1:172.21.15.195:6789/0 conn(0x7fe500104330 legacy=0x7fe5001184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 shutdown_connections 2024-02-10T08:42:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 >> 172.21.15.107:0/2574481547 conn(0x7fe5000f3640 msgr2=0x7fe5001077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 shutdown_connections 2024-02-10T08:42:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 -- 172.21.15.107:0/2574481547 wait complete. 2024-02-10T08:42:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:32.795+0000 7fe5075e9700 1 librados: shutdown 2024-02-10T08:42:34.046 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:34.051 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:34.051 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:34.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:34.184 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:34.204 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:34.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.387236+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.390142+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.657765+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.f scrub starts 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:32.661262+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.f scrub ok 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:33.450331+0000 mgr.y (mgr.14553) 393 : cluster [DBG] pgmap v641: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.387236+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.390142+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.657765+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.f scrub starts 2024-02-10T08:42:34.336 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:32.661262+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.f scrub ok 2024-02-10T08:42:34.337 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:33.450331+0000 mgr.y (mgr.14553) 393 : cluster [DBG] pgmap v641: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T08:42:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:34.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:34.522 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.387236+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:42:34.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.390142+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:42:34.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.657765+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.f scrub starts 2024-02-10T08:42:34.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:32.661262+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.f scrub ok 2024-02-10T08:42:34.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:33.450331+0000 mgr.y (mgr.14553) 393 : cluster [DBG] pgmap v641: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T08:42:34.525 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:34.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:34.678 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:34.698 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:34.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:34.831 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:34.847 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-10T08:42:34.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:42:34.907 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:34.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:35.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:35.046 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:35.247 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:35.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:35.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:42:35.285 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:35.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:35.380 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:33.701378+0000 osd.1 (osd.1) 211 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:33.705059+0000 osd.1 (osd.1) 212 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.005145+0000 osd.3 (osd.3) 207 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.006173+0000 osd.3 (osd.3) 208 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.136596+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.d scrub starts 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.139520+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.d scrub ok 2024-02-10T08:42:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.732856+0000 osd.4 (osd.4) 200 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:34.734688+0000 osd.4 (osd.4) 201 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:35.073152+0000 mon.b (mon.1) 590 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:35.073434+0000 mon.a (mon.0) 2517 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:35.074021+0000 mon.b (mon.1) 591 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:35.074210+0000 mon.a (mon.0) 2518 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:33.701378+0000 osd.1 (osd.1) 211 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:33.705059+0000 osd.1 (osd.1) 212 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.005145+0000 osd.3 (osd.3) 207 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.006173+0000 osd.3 (osd.3) 208 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.136596+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.d scrub starts 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.139520+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.d scrub ok 2024-02-10T08:42:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.732856+0000 osd.4 (osd.4) 200 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:42:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:34.734688+0000 osd.4 (osd.4) 201 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:42:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:35.073152+0000 mon.b (mon.1) 590 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:35.073434+0000 mon.a (mon.0) 2517 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:35.074021+0000 mon.b (mon.1) 591 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:35.074210+0000 mon.a (mon.0) 2518 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:33.701378+0000 osd.1 (osd.1) 211 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:33.705059+0000 osd.1 (osd.1) 212 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.005145+0000 osd.3 (osd.3) 207 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.006173+0000 osd.3 (osd.3) 208 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.136596+0000 osd.5 (osd.5) 199 : cluster [DBG] 6.d scrub starts 2024-02-10T08:42:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.139520+0000 osd.5 (osd.5) 200 : cluster [DBG] 6.d scrub ok 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.732856+0000 osd.4 (osd.4) 200 : cluster [DBG] 60.8 scrub starts 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:34.734688+0000 osd.4 (osd.4) 201 : cluster [DBG] 60.8 scrub ok 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:35.073152+0000 mon.b (mon.1) 590 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:35.073434+0000 mon.a (mon.0) 2517 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:35.074021+0000 mon.b (mon.1) 591 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:35.074210+0000 mon.a (mon.0) 2518 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-63"}]: dispatch 2024-02-10T08:42:35.580 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:35.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:35.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.806+0000 7f0aae790700 1 -- 172.21.15.107:0/4188407963 >> v1:172.21.15.107:6789/0 conn(0x7f0aa80f9530 legacy=0x7f0aa8102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.806+0000 7f0aae790700 1 -- 172.21.15.107:0/4188407963 shutdown_connections 2024-02-10T08:42:35.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.806+0000 7f0aae790700 1 -- 172.21.15.107:0/4188407963 >> 172.21.15.107:0/4188407963 conn(0x7f0aa80f2510 msgr2=0x7f0aa80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.806+0000 7f0aae790700 1 -- 172.21.15.107:0/4188407963 shutdown_connections 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.806+0000 7f0aae790700 1 -- 172.21.15.107:0/4188407963 wait complete. 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 librados: starting msgr at 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 librados: starting objecter 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 Processor -- start 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 -- start start 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 librados: setting wanted keys 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 librados: calling monclient init 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0aa8102b80 con 0x7f0aa80f9530 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0aa8102d50 con 0x7f0aa80f8a40 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aae790700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0aa8102f40 con 0x7f0aa8104340 2024-02-10T08:42:35.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aa77fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0aa80f9530 0x7f0aa8114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60552/0 (socket says 172.21.15.107:60552) 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.807+0000 7f0aa77fe700 1 -- 172.21.15.107:0/2217873822 learned_addr learned my addr 172.21.15.107:0/2217873822 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter ms_handle_connect 0x7f0aa8104340 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter ms_handle_connect 0x7f0aa80f9530 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter ms_handle_connect 0x7f0aa80f8a40 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326031255 0 0) 0x7f0aa8102f40 con 0x7f0aa8104340 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a900038e0 con 0x7f0aa8104340 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252350806 0 0) 0x7f0aa8102b80 con 0x7f0aa80f9530 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa8102f40 con 0x7f0aa80f9530 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596476666 0 0) 0x7f0aa8102d50 con 0x7f0aa80f8a40 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa8102b80 con 0x7f0aa80f8a40 2024-02-10T08:42:35.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030153680 0 0) 0x7f0a900038e0 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0aa8102d50 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0aa0001d70 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2900005030 0 0) 0x7f0aa8102f40 con 0x7f0aa80f9530 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a900038e0 con 0x7f0aa80f9530 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a9c0020f0 con 0x7f0aa80f9530 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185671264 0 0) 0x7f0aa8102b80 con 0x7f0aa80f8a40 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0aa8102f40 con 0x7f0aa80f8a40 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a98001840 con 0x7f0aa80f8a40 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3486388375 0 0) 0x7f0aa8102d50 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 >> v1:172.21.15.195:6789/0 conn(0x7f0aa80f8a40 legacy=0x7f0aa81023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 >> v1:172.21.15.107:6789/0 conn(0x7f0aa80f9530 legacy=0x7f0aa8114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa811cc00 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa8119c30 con 0x7f0aa8104340 2024-02-10T08:42:35.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aae790700 10 client.53690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa811a0a0 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aae790700 1 librados: init done 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0aa00026e0 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.808+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0aa0003770 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.809+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94004f20 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0aa0002240 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(436..436 src has 330..436) v4 ==== 7836+0+0 (unknown 1444639480 0 0) 0x7f0aa00766c0 con 0x7f0aa8104340 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 10 client.53690.objecter ms_dispatch 0x7f0aa8001550 osd_map(436..436 src has 330..436) v4 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 3 client.53690.objecter handle_osd_map got epochs [436,436] > 0 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 3 client.53690.objecter handle_osd_map decoding full epoch 436 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 20 client.53690.objecter dump_active .. 0 homeless 2024-02-10T08:42:35.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.810+0000 7f0aa57fa700 10 client.53690.objecter ms_handle_connect 0x7f0a90063c70 2024-02-10T08:42:35.816 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:35.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:35.815+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0aa0049b90 con 0x7f0aa8104340 2024-02-10T08:42:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:35.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:35.855 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:35.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:35] "GET /metrics HTTP/1.1" 200 187633 "" "Prometheus/2.33.4" 2024-02-10T08:42:35.947 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.012+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0a940051d0 con 0x7f0aa8104340 2024-02-10T08:42:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.038+0000 7f0aa57fa700 1 -- 172.21.15.107:0/2217873822 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v437) v1 ==== 101+0+0 (unknown 902869449 0 0) 0x7f0aa0047fa0 con 0x7f0aa8104340 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.041+0000 7f0aae790700 10 librados: watch_flush enter 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 10 librados: watch_flush exit 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 20 client.53690.objecter shutdown clearing up homeless session... 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 10 client.53690.objecter successfully canceled tick 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0a90063c70 legacy=0x7f0a90066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 >> v1:172.21.15.107:6790/0 conn(0x7f0aa8104340 legacy=0x7f0aa81184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 shutdown_connections 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 >> 172.21.15.107:0/2217873822 conn(0x7f0aa80f2510 msgr2=0x7f0aa81077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 shutdown_connections 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 -- 172.21.15.107:0/2217873822 wait complete. 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:36.042+0000 7f0aae790700 1 librados: shutdown 2024-02-10T08:42:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:42:36.098 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:36.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:36.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:36.233 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:36.254 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:36.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:36.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.023109+0000 osd.3 (osd.3) 209 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.024094+0000 osd.3 (osd.3) 210 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.139541+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.155119+0000 osd.5 (osd.5) 201 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.160231+0000 osd.5 (osd.5) 202 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.450726+0000 mgr.y (mgr.14553) 394 : cluster [DBG] pgmap v643: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 978 B/s rd, 543 B/s wr, 1 op/s 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:36.013163+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.107:0/2217873822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:36.013463+0000 mon.a (mon.0) 2520 : audit [INF] from='client.53690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:36.032525+0000 mon.a (mon.0) 2521 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:42:36.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:36.037797+0000 mon.a (mon.0) 2522 : audit [INF] from='client.53690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:36.037908+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:36.041164+0000 mon.b (mon.1) 592 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:36.047788+0000 mon.a (mon.0) 2524 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.023109+0000 osd.3 (osd.3) 209 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.024094+0000 osd.3 (osd.3) 210 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.139541+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.155119+0000 osd.5 (osd.5) 201 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.160231+0000 osd.5 (osd.5) 202 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.450726+0000 mgr.y (mgr.14553) 394 : cluster [DBG] pgmap v643: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 978 B/s rd, 543 B/s wr, 1 op/s 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:36.013163+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.107:0/2217873822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:36.013463+0000 mon.a (mon.0) 2520 : audit [INF] from='client.53690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:36.032525+0000 mon.a (mon.0) 2521 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:42:36.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:36.037797+0000 mon.a (mon.0) 2522 : audit [INF] from='client.53690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:36.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:36.037908+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-02-10T08:42:36.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:36.041164+0000 mon.b (mon.1) 592 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:36.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:36.047788+0000 mon.a (mon.0) 2524 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.023109+0000 osd.3 (osd.3) 209 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.024094+0000 osd.3 (osd.3) 210 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.139541+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.155119+0000 osd.5 (osd.5) 201 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.160231+0000 osd.5 (osd.5) 202 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.450726+0000 mgr.y (mgr.14553) 394 : cluster [DBG] pgmap v643: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 978 B/s rd, 543 B/s wr, 1 op/s 2024-02-10T08:42:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:36.013163+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.107:0/2217873822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:36.013463+0000 mon.a (mon.0) 2520 : audit [INF] from='client.53690 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:36.032525+0000 mon.a (mon.0) 2521 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:36.037797+0000 mon.a (mon.0) 2522 : audit [INF] from='client.53690 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:36.037908+0000 mon.a (mon.0) 2523 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:36.041164+0000 mon.b (mon.1) 592 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:36.047788+0000 mon.a (mon.0) 2524 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:37.035 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:37.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:37.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:37.170 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:37.190 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:37.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:37.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:37.321 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:37.340 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:37.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.400562+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:42:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.401877+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.740807+0000 osd.1 (osd.1) 213 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:35.744576+0000 osd.1 (osd.1) 214 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:37.039938+0000 mon.a (mon.0) 2525 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.040036+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:37.050682+0000 mon.b (mon.1) 593 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.091311+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.092513+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.400562+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.401877+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.740807+0000 osd.1 (osd.1) 213 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:42:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:35.744576+0000 osd.1 (osd.1) 214 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:42:37.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:37.039938+0000 mon.a (mon.0) 2525 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:37.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.040036+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-02-10T08:42:37.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:37.050682+0000 mon.b (mon.1) 593 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:37.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.091311+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:42:37.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.092513+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:42:37.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.400562+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.401877+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.740807+0000 osd.1 (osd.1) 213 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:35.744576+0000 osd.1 (osd.1) 214 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:37.039938+0000 mon.a (mon.0) 2525 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.040036+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:37.050682+0000 mon.b (mon.1) 593 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.091311+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:42:37.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.092513+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:42:37.470 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:37.489 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:37.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:37.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:37.622 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:38.419 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:36.722430+0000 osd.1 (osd.1) 215 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:36.726087+0000 osd.1 (osd.1) 216 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.142295+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.145176+0000 osd.5 (osd.5) 204 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.451225+0000 mgr.y (mgr.14553) 395 : cluster [DBG] pgmap v646: 218 pgs: 28 creating+peering, 4 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.796202+0000 osd.4 (osd.4) 202 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:42:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:37.799721+0000 osd.4 (osd.4) 203 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:38.040736+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:38.071143+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:38.071490+0000 mon.a (mon.0) 2528 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:38.072061+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:38.072214+0000 mon.a (mon.0) 2529 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:36.722430+0000 osd.1 (osd.1) 215 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:36.726087+0000 osd.1 (osd.1) 216 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.142295+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.145176+0000 osd.5 (osd.5) 204 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.451225+0000 mgr.y (mgr.14553) 395 : cluster [DBG] pgmap v646: 218 pgs: 28 creating+peering, 4 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.796202+0000 osd.4 (osd.4) 202 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:37.799721+0000 osd.4 (osd.4) 203 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:42:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:38.040736+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-02-10T08:42:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:38.071143+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:38.071490+0000 mon.a (mon.0) 2528 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:38.072061+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:38.072214+0000 mon.a (mon.0) 2529 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:36.722430+0000 osd.1 (osd.1) 215 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:36.726087+0000 osd.1 (osd.1) 216 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.142295+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.145176+0000 osd.5 (osd.5) 204 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.451225+0000 mgr.y (mgr.14553) 395 : cluster [DBG] pgmap v646: 218 pgs: 28 creating+peering, 4 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.796202+0000 osd.4 (osd.4) 202 : cluster [DBG] 60.5 scrub starts 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:37.799721+0000 osd.4 (osd.4) 203 : cluster [DBG] 60.5 scrub ok 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:38.040736+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-02-10T08:42:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:38.071143+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:38.071490+0000 mon.a (mon.0) 2528 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:38.072061+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:38.072214+0000 mon.a (mon.0) 2529 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-65"}]: dispatch 2024-02-10T08:42:39.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:42:39.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 -- 172.21.15.107:0/943594116 >> v1:172.21.15.107:6789/0 conn(0x7f87e40fda50 legacy=0x7f87e40fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 -- 172.21.15.107:0/943594116 shutdown_connections 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 -- 172.21.15.107:0/943594116 >> 172.21.15.107:0/943594116 conn(0x7f87e40f3640 msgr2=0x7f87e40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 -- 172.21.15.107:0/943594116 shutdown_connections 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 -- 172.21.15.107:0/943594116 wait complete. 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 librados: starting msgr at 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.700+0000 7f87ebdda700 1 librados: starting objecter 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 Processor -- start 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 -- start start 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 librados: setting wanted keys 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 librados: calling monclient init 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87e406cbc0 con 0x7f87e41007f0 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87e406cd90 con 0x7f87e4104330 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87ebdda700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87e406cf80 con 0x7f87e40fda50 2024-02-10T08:42:39.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87e9b76700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f87e40fda50 0x7f87e40fc9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52616/0 (socket says 172.21.15.107:52616) 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87e9b76700 1 -- 172.21.15.107:0/994377385 learned_addr learned my addr 172.21.15.107:0/994377385 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter ms_handle_connect 0x7f87e41007f0 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter ms_handle_connect 0x7f87e40fda50 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter ms_handle_connect 0x7f87e4104330 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016154349 0 0) 0x7f87e406cf80 con 0x7f87e40fda50 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d0003030 con 0x7f87e40fda50 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815466992 0 0) 0x7f87e406cbc0 con 0x7f87e41007f0 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e406cf80 con 0x7f87e41007f0 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727057405 0 0) 0x7f87e406cd90 con 0x7f87e4104330 2024-02-10T08:42:39.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e406cbc0 con 0x7f87e4104330 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 525354080 0 0) 0x7f87d0003030 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87e406cd90 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f87e0001430 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067900091 0 0) 0x7f87e406cf80 con 0x7f87e41007f0 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d0003030 con 0x7f87e41007f0 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.701+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f87d40018c0 con 0x7f87e41007f0 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035397070 0 0) 0x7f87e406cbc0 con 0x7f87e4104330 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87e406cf80 con 0x7f87e4104330 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f87dc0018d0 con 0x7f87e4104330 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1774328335 0 0) 0x7f87e406cd90 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 >> v1:172.21.15.195:6789/0 conn(0x7f87e4104330 legacy=0x7f87e41184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 >> v1:172.21.15.107:6789/0 conn(0x7f87e41007f0 legacy=0x7f87e4114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e411cc00 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e4119c30 con 0x7f87e40fda50 2024-02-10T08:42:39.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87ebdda700 10 client.53696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e411a120 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87ebdda700 1 librados: init done 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f87e0001a70 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.702+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f87e0003880 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.703+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c8004f20 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.703+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f87e0003a80 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(440..440 src has 330..440) v4 ==== 7808+0+0 (unknown 1685873684 0 0) 0x7f87e0078eb0 con 0x7f87e40fda50 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 10 client.53696.objecter ms_dispatch 0x7f87e4001550 osd_map(440..440 src has 330..440) v4 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 3 client.53696.objecter handle_osd_map got epochs [440,440] > 0 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 3 client.53696.objecter handle_osd_map decoding full epoch 440 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 20 client.53696.objecter dump_active .. 0 homeless 2024-02-10T08:42:39.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.704+0000 7f87daffd700 10 client.53696.objecter ms_handle_connect 0x7f87d0063cc0 2024-02-10T08:42:39.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.712+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f87e004c4a0 con 0x7f87e40fda50 2024-02-10T08:42:39.798 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:42:39.798 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:42:39.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.906+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87c8000b90 con 0x7f87d0063cc0 2024-02-10T08:42:39.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.919+0000 7f87daffd700 1 -- 172.21.15.107:0/994377385 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+479803 (unknown 2965378022 0 849596495) 0x7f87c8000b90 con 0x7f87d0063cc0 2024-02-10T08:42:39.925 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 10 librados: watch_flush enter 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 10 librados: watch_flush exit 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 20 client.53696.objecter shutdown clearing up homeless session... 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 10 client.53696.objecter successfully canceled tick 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 >> v1:172.21.15.107:6800/2634842009 conn(0x7f87d0063cc0 legacy=0x7f87d0066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 >> v1:172.21.15.107:6790/0 conn(0x7f87e40fda50 legacy=0x7f87e40fc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 shutdown_connections 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 >> 172.21.15.107:0/994377385 conn(0x7f87e40f3640 msgr2=0x7f87e41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 shutdown_connections 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 -- 172.21.15.107:0/994377385 wait complete. 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:39.923+0000 7f87ebdda700 1 librados: shutdown 2024-02-10T08:42:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:38.784786+0000 osd.4 (osd.4) 204 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:38.786492+0000 osd.4 (osd.4) 205 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:39.041917+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-02-10T08:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:39.451563+0000 mgr.y (mgr.14553) 396 : cluster [DBG] pgmap v649: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:42:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:38.784786+0000 osd.4 (osd.4) 204 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:42:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:38.786492+0000 osd.4 (osd.4) 205 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:42:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:39.041917+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-02-10T08:42:40.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:39.451563+0000 mgr.y (mgr.14553) 396 : cluster [DBG] pgmap v649: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:42:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:38.784786+0000 osd.4 (osd.4) 204 : cluster [DBG] 60.4 scrub starts 2024-02-10T08:42:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:38.786492+0000 osd.4 (osd.4) 205 : cluster [DBG] 60.4 scrub ok 2024-02-10T08:42:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:39.041917+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-02-10T08:42:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:39.451563+0000 mgr.y (mgr.14553) 396 : cluster [DBG] pgmap v649: 186 pgs: 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:42:40.573 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:40.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:40.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:40.710 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:40.731 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:40.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:40.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:40.864 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:39.752820+0000 osd.4 (osd.4) 206 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:42:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:39.756501+0000 osd.4 (osd.4) 207 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:39.907835+0000 mgr.y (mgr.14553) 397 : audit [DBG] from='client.53696 v1:172.21.15.107:0/994377385' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.049701+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:40.050978+0000 mon.b (mon.1) 596 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:40.055031+0000 mon.a (mon.0) 2532 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.128453+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.131446+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:40.157235+0000 mon.a (mon.0) 2533 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.369999+0000 osd.2 (osd.2) 161 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.371651+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:39.752820+0000 osd.4 (osd.4) 206 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:42:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:39.756501+0000 osd.4 (osd.4) 207 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:39.907835+0000 mgr.y (mgr.14553) 397 : audit [DBG] from='client.53696 v1:172.21.15.107:0/994377385' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.049701+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:40.050978+0000 mon.b (mon.1) 596 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:40.055031+0000 mon.a (mon.0) 2532 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.128453+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.131446+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:40.157235+0000 mon.a (mon.0) 2533 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.369999+0000 osd.2 (osd.2) 161 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:42:41.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.371651+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:42:41.314 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2024-02-10T08:42:39.451368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007298,"num_objects":353,"num_object_clones":3,"num_object_copies":1059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1381,"num_read_kb":1034,"num_write":1036,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2388,"ondisk_log_size":2388,"up":558,"acting":558,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":644,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2143640,"kb_used_data":30304,"kb_used_omap":0,"kb_used_meta":2113280,"kb_avail":654072424,"statfs":{"total":671965249536,"available":669770162176,"internally_reserved":0,"allocated":31031296,"data_stored":19416260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2163998720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000844"},"pg_stats":[{"pgid":"60.3","version":"206'34","reported_seq":429,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.037131+0000","last_change":"2024-02-10T08:42:33.705136+0000","last_active":"2024-02-10T08:42:36.037131+0000","last_peered":"2024-02-10T08:42:36.037131+0000","last_clean":"2024-02-10T08:42:36.037131+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:42:36.037131+0000","last_undegraded":"2024-02-10T08:42:36.037131+0000","last_fullsized":"2024-02-10T08:42:36.037131+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:42:33.705063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:33.705063+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"338'29","reported_seq":432,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:28.650986+0000","last_change":"2024-02-10T08:42:28.650952+0000","last_active":"2024-02-10T08:42:28.650986+0000","last_peered":"2024-02-10T08:42:28.650986+0000","last_clean":"2024-02-10T08:42:28.650986+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:42:28.650986+0000","last_undegraded":"2024-02-10T08:42:28.650986+0000","last_fullsized":"2024-02-10T08:42:28.650986+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'29","last_scrub_stamp":"2024-02-10T08:42:28.650876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.650876+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.1","version":"401'42","reported_seq":432,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.735610+0000","last_change":"2024-02-10T08:42:31.735581+0000","last_active":"2024-02-10T08:42:31.735610+0000","last_peered":"2024-02-10T08:42:31.735610+0000","last_clean":"2024-02-10T08:42:31.735610+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:42:31.735610+0000","last_undegraded":"2024-02-10T08:42:31.735610+0000","last_fullsized":"2024-02-10T08:42:31.735610+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'42","last_scrub_stamp":"2024-02-10T08:42:31.735511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.735511+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.0","version":"371'108","reported_seq":504,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.046311+0000","last_change":"2024-02-10T08:42:25.056269+0000","last_active":"2024-02-10T08:42:36.046311+0000","last_peered":"2024-02-10T08:42:36.046311+0000","last_clean":"2024-02-10T08:42:36.046311+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:42:36.046311+0000","last_undegraded":"2024-02-10T08:42:36.046311+0000","last_fullsized":"2024-02-10T08:42:36.046311+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"371'35","reported_seq":410,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:32.704258+0000","last_change":"2024-02-10T08:42:32.704233+0000","last_active":"2024-02-10T08:42:32.704258+0000","last_peered":"2024-02-10T08:42:32.704258+0000","last_clean":"2024-02-10T08:42:32.704258+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:42:32.704258+0000","last_undegraded":"2024-02-10T08:42:32.704258+0000","last_fullsized":"2024-02-10T08:42:32.704258+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:32.704166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.704166+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.6","version":"338'73","reported_seq":466,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.040423+0000","last_change":"2024-02-10T08:42:31.200907+0000","last_active":"2024-02-10T08:42:36.040423+0000","last_peered":"2024-02-10T08:42:36.040423+0000","last_clean":"2024-02-10T08:42:36.040423+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:42:36.040423+0000","last_undegraded":"2024-02-10T08:42:36.040423+0000","last_fullsized":"2024-02-10T08:42:36.040423+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'73","last_scrub_stamp":"2024-02-10T08:42:31.200824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.200824+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"279'42","reported_seq":441,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033182+0000","last_change":"2024-02-10T08:42:25.055341+0000","last_active":"2024-02-10T08:42:26.033182+0000","last_peered":"2024-02-10T08:42:26.033182+0000","last_clean":"2024-02-10T08:42:26.033182+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:42:26.033182+0000","last_undegraded":"2024-02-10T08:42:26.033182+0000","last_fullsized":"2024-02-10T08:42:26.033182+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:39:52.582974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:52.582974+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.4","version":"338'23","reported_seq":419,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032694+0000","last_change":"2024-02-10T08:42:25.054598+0000","last_active":"2024-02-10T08:42:26.032694+0000","last_peered":"2024-02-10T08:42:26.032694+0000","last_clean":"2024-02-10T08:42:26.032694+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:42:26.032694+0000","last_undegraded":"2024-02-10T08:42:26.032694+0000","last_fullsized":"2024-02-10T08:42:26.032694+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'22","last_scrub_stamp":"2024-02-10T08:39:42.510605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:42.510605+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.b","version":"395'105","reported_seq":492,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036539+0000","last_change":"2024-02-10T08:42:25.056267+0000","last_active":"2024-02-10T08:42:36.036539+0000","last_peered":"2024-02-10T08:42:36.036539+0000","last_clean":"2024-02-10T08:42:36.036539+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:42:36.036539+0000","last_undegraded":"2024-02-10T08:42:36.036539+0000","last_fullsized":"2024-02-10T08:42:36.036539+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'101","last_scrub_stamp":"2024-02-10T08:40:28.463426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.463426+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"371'23","reported_seq":425,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:30.719162+0000","last_change":"2024-02-10T08:42:30.719134+0000","last_active":"2024-02-10T08:42:30.719162+0000","last_peered":"2024-02-10T08:42:30.719162+0000","last_clean":"2024-02-10T08:42:30.719162+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:42:30.719162+0000","last_undegraded":"2024-02-10T08:42:30.719162+0000","last_fullsized":"2024-02-10T08:42:30.719162+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'23","last_scrub_stamp":"2024-02-10T08:42:30.719056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.719056+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.9","version":"395'115","reported_seq":492,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036042+0000","last_change":"2024-02-10T08:42:27.509494+0000","last_active":"2024-02-10T08:42:36.036042+0000","last_peered":"2024-02-10T08:42:36.036042+0000","last_clean":"2024-02-10T08:42:36.036042+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:42:36.036042+0000","last_undegraded":"2024-02-10T08:42:36.036042+0000","last_fullsized":"2024-02-10T08:42:36.036042+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'115","last_scrub_stamp":"2024-02-10T08:42:27.509423+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.509423+0000","log_size":115,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":35,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"338'86","reported_seq":478,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:34.734796+0000","last_change":"2024-02-10T08:42:34.734765+0000","last_active":"2024-02-10T08:42:34.734796+0000","last_peered":"2024-02-10T08:42:34.734796+0000","last_clean":"2024-02-10T08:42:34.734796+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:42:34.734796+0000","last_undegraded":"2024-02-10T08:42:34.734796+0000","last_fullsized":"2024-02-10T08:42:34.734796+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'86","last_scrub_stamp":"2024-02-10T08:42:34.734692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.734692+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.f","version":"405'96","reported_seq":479,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388792+0000","last_change":"2024-02-10T08:42:25.088908+0000","last_active":"2024-02-10T08:42:26.388792+0000","last_peered":"2024-02-10T08:42:26.388792+0000","last_clean":"2024-02-10T08:42:26.388792+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:42:26.388792+0000","last_undegraded":"2024-02-10T08:42:26.388792+0000","last_fullsized":"2024-02-10T08:42:26.388792+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:41:45.373410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.373410+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.e","version":"250'56","reported_seq":427,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.045892+0000","last_change":"2024-02-10T08:42:25.061573+0000","last_active":"2024-02-10T08:42:36.045892+0000","last_peered":"2024-02-10T08:42:36.045892+0000","last_clean":"2024-02-10T08:42:36.045892+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:42:36.045892+0000","last_undegraded":"2024-02-10T08:42:36.045892+0000","last_fullsized":"2024-02-10T08:42:36.045892+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"423'78","reported_seq":467,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388407+0000","last_change":"2024-02-10T08:42:25.088786+0000","last_active":"2024-02-10T08:42:26.388407+0000","last_peered":"2024-02-10T08:42:26.388407+0000","last_clean":"2024-02-10T08:42:26.388407+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:42:26.388407+0000","last_undegraded":"2024-02-10T08:42:26.388407+0000","last_fullsized":"2024-02-10T08:42:26.388407+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'74","last_scrub_stamp":"2024-02-10T08:41:55.273156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.273156+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.c","version":"338'70","reported_seq":460,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036909+0000","last_change":"2024-02-10T08:42:25.056562+0000","last_active":"2024-02-10T08:42:36.036909+0000","last_peered":"2024-02-10T08:42:36.036909+0000","last_clean":"2024-02-10T08:42:36.036909+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:42:36.036909+0000","last_undegraded":"2024-02-10T08:42:36.036909+0000","last_fullsized":"2024-02-10T08:42:36.036909+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:40:36.430069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.430069+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"338'56","reported_seq":448,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390077+0000","last_change":"2024-02-10T08:42:25.057952+0000","last_active":"2024-02-10T08:42:26.390077+0000","last_peered":"2024-02-10T08:42:26.390077+0000","last_clean":"2024-02-10T08:42:26.390077+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:42:26.390077+0000","last_undegraded":"2024-02-10T08:42:26.390077+0000","last_fullsized":"2024-02-10T08:42:26.390077+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'56","last_scrub_stamp":"2024-02-10T08:41:49.073509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.073509+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.12","version":"214'102","reported_seq":496,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.043104+0000","last_change":"2024-02-10T08:42:25.060202+0000","last_active":"2024-02-10T08:42:36.043104+0000","last_peered":"2024-02-10T08:42:36.043104+0000","last_clean":"2024-02-10T08:42:36.043104+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:42:36.043104+0000","last_undegraded":"2024-02-10T08:42:36.043104+0000","last_fullsized":"2024-02-10T08:42:36.043104+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"398'105","reported_seq":485,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:30.099883+0000","last_change":"2024-02-10T08:42:30.099849+0000","last_active":"2024-02-10T08:42:30.099883+0000","last_peered":"2024-02-10T08:42:30.099883+0000","last_clean":"2024-02-10T08:42:30.099883+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:42:30.099883+0000","last_undegraded":"2024-02-10T08:42:30.099883+0000","last_fullsized":"2024-02-10T08:42:30.099883+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'105","last_scrub_stamp":"2024-02-10T08:42:30.099763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.099763+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.10","version":"301'72","reported_seq":471,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.045474+0000","last_change":"2024-02-10T08:42:25.060207+0000","last_active":"2024-02-10T08:42:36.045474+0000","last_peered":"2024-02-10T08:42:36.045474+0000","last_clean":"2024-02-10T08:42:36.045474+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:42:36.045474+0000","last_undegraded":"2024-02-10T08:42:36.045474+0000","last_fullsized":"2024-02-10T08:42:36.045474+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:42:01.488098+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.488098+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"392'87","reported_seq":475,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.044221+0000","last_change":"2024-02-10T08:42:25.062302+0000","last_active":"2024-02-10T08:42:36.044221+0000","last_peered":"2024-02-10T08:42:36.044221+0000","last_clean":"2024-02-10T08:42:36.044221+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:42:36.044221+0000","last_undegraded":"2024-02-10T08:42:36.044221+0000","last_fullsized":"2024-02-10T08:42:36.044221+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:41:57.565097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.565097+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":472,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.045474+0000","last_change":"2024-02-10T08:42:25.061409+0000","last_active":"2024-02-10T08:42:36.045474+0000","last_peered":"2024-02-10T08:42:36.045474+0000","last_clean":"2024-02-10T08:42:36.045474+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:42:36.045474+0000","last_undegraded":"2024-02-10T08:42:36.045474+0000","last_fullsized":"2024-02-10T08:42:36.045474+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"434'34","reported_seq":425,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:27.058262+0000","last_change":"2024-02-10T08:42:27.058262+0000","last_active":"2024-02-10T08:42:27.058262+0000","last_peered":"2024-02-10T08:42:27.058262+0000","last_clean":"2024-02-10T08:42:27.058262+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:42:27.058262+0000","last_undegraded":"2024-02-10T08:42:27.058262+0000","last_fullsized":"2024-02-10T08:42:27.058262+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-02-10T08:39:56.545042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:56.545042+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.14","version":"295'36","reported_seq":425,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.045907+0000","last_change":"2024-02-10T08:42:25.061224+0000","last_active":"2024-02-10T08:42:36.045907+0000","last_peered":"2024-02-10T08:42:36.045907+0000","last_clean":"2024-02-10T08:42:36.045907+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:42:36.045907+0000","last_undegraded":"2024-02-10T08:42:36.045907+0000","last_fullsized":"2024-02-10T08:42:36.045907+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"338'69","reported_seq":437,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032561+0000","last_change":"2024-02-10T08:42:25.054206+0000","last_active":"2024-02-10T08:42:26.032561+0000","last_peered":"2024-02-10T08:42:26.032561+0000","last_clean":"2024-02-10T08:42:26.032561+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:42:26.032561+0000","last_undegraded":"2024-02-10T08:42:26.032561+0000","last_fullsized":"2024-02-10T08:42:26.032561+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.1a","version":"398'71","reported_seq":450,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.038506+0000","last_change":"2024-02-10T08:42:25.054795+0000","last_active":"2024-02-10T08:42:36.038506+0000","last_peered":"2024-02-10T08:42:36.038506+0000","last_clean":"2024-02-10T08:42:36.038506+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:42:36.038506+0000","last_undegraded":"2024-02-10T08:42:36.038506+0000","last_fullsized":"2024-02-10T08:42:36.038506+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:40:35.307608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.307608+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":442,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036258+0000","last_change":"2024-02-10T08:42:35.744653+0000","last_active":"2024-02-10T08:42:36.036258+0000","last_peered":"2024-02-10T08:42:36.036258+0000","last_clean":"2024-02-10T08:42:36.036258+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:42:36.036258+0000","last_undegraded":"2024-02-10T08:42:36.036258+0000","last_fullsized":"2024-02-10T08:42:36.036258+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-10T08:42:35.744580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.744580+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"401'38","reported_seq":409,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032193+0000","last_change":"2024-02-10T08:42:25.052977+0000","last_active":"2024-02-10T08:42:26.032193+0000","last_peered":"2024-02-10T08:42:26.032193+0000","last_clean":"2024-02-10T08:42:26.032193+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:42:26.032193+0000","last_undegraded":"2024-02-10T08:42:26.032193+0000","last_fullsized":"2024-02-10T08:42:26.032193+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-10T08:39:54.602589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.602589+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.1f","version":"338'36","reported_seq":439,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390663+0000","last_change":"2024-02-10T08:42:25.058674+0000","last_active":"2024-02-10T08:42:26.390663+0000","last_peered":"2024-02-10T08:42:26.390663+0000","last_clean":"2024-02-10T08:42:26.390663+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:42:26.390663+0000","last_undegraded":"2024-02-10T08:42:26.390663+0000","last_fullsized":"2024-02-10T08:42:26.390663+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'36","last_scrub_stamp":"2024-02-10T08:41:46.085763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.085763+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"60.1e","version":"210'17","reported_seq":399,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036544+0000","last_change":"2024-02-10T08:42:25.056249+0000","last_active":"2024-02-10T08:42:36.036544+0000","last_peered":"2024-02-10T08:42:36.036544+0000","last_clean":"2024-02-10T08:42:36.036544+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:42:36.036544+0000","last_undegraded":"2024-02-10T08:42:36.036544+0000","last_fullsized":"2024-02-10T08:42:36.036544+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:42:04.642908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.642908+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"405'21","reported_seq":405,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.041844+0000","last_change":"2024-02-10T08:42:25.060687+0000","last_active":"2024-02-10T08:42:36.041844+0000","last_peered":"2024-02-10T08:42:36.041844+0000","last_clean":"2024-02-10T08:42:36.041844+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:42:36.041844+0000","last_undegraded":"2024-02-10T08:42:36.041844+0000","last_fullsized":"2024-02-10T08:42:36.041844+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"371'35","reported_seq":431,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-10T08:42:36.036765+0000","last_change":"2024-02-10T08:42:25.056522+0000","last_active":"2024-02-10T08:42:36.036765+0000","last_peered":"2024-02-10T08:42:36.036765+0000","last_clean":"2024-02-10T08:42:36.036765+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:42:36.036765+0000","last_undegraded":"2024-02-10T08:42:36.036765+0000","last_fullsized":"2024-02-10T08:42:36.036765+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-02-10T08:40:23.345538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.345538+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":542,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.402808+0000","last_change":"2024-02-10T08:41:09.353173+0000","last_active":"2024-02-10T08:42:35.402808+0000","last_peered":"2024-02-10T08:42:35.402808+0000","last_clean":"2024-02-10T08:42:35.402808+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:42:35.402808+0000","last_undegraded":"2024-02-10T08:42:35.402808+0000","last_fullsized":"2024-02-10T08:42:35.402808+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:41:09.353088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:09.353088+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":436,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.389647+0000","last_change":"2024-02-10T08:42:25.091712+0000","last_active":"2024-02-10T08:42:26.389647+0000","last_peered":"2024-02-10T08:42:26.389647+0000","last_clean":"2024-02-10T08:42:26.389647+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:42:26.389647+0000","last_undegraded":"2024-02-10T08:42:26.389647+0000","last_fullsized":"2024-02-10T08:42:26.389647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:42:25.091640+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.091640+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":405,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145127+0000","last_change":"2024-02-10T08:40:42.232974+0000","last_active":"2024-02-10T08:42:35.145127+0000","last_peered":"2024-02-10T08:42:35.145127+0000","last_clean":"2024-02-10T08:42:35.145127+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:42:35.145127+0000","last_undegraded":"2024-02-10T08:42:35.145127+0000","last_fullsized":"2024-02-10T08:42:35.145127+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.232905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.232905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":409,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390316+0000","last_change":"2024-02-10T08:40:26.859788+0000","last_active":"2024-02-10T08:42:26.390316+0000","last_peered":"2024-02-10T08:42:26.390316+0000","last_clean":"2024-02-10T08:42:26.390316+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:42:26.390316+0000","last_undegraded":"2024-02-10T08:42:26.390316+0000","last_fullsized":"2024-02-10T08:42:26.390316+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.859697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.859697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140108+0000","last_change":"2024-02-10T08:41:59.348271+0000","last_active":"2024-02-10T08:42:35.140108+0000","last_peered":"2024-02-10T08:42:35.140108+0000","last_clean":"2024-02-10T08:42:35.140108+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:42:35.140108+0000","last_undegraded":"2024-02-10T08:42:35.140108+0000","last_fullsized":"2024-02-10T08:42:35.140108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.348211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.348211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":382,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.139797+0000","last_change":"2024-02-10T08:41:16.350785+0000","last_active":"2024-02-10T08:42:35.139797+0000","last_peered":"2024-02-10T08:42:35.139797+0000","last_clean":"2024-02-10T08:42:35.139797+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:42:35.139797+0000","last_undegraded":"2024-02-10T08:42:35.139797+0000","last_fullsized":"2024-02-10T08:42:35.139797+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":410,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146469+0000","last_change":"2024-02-10T08:41:05.559497+0000","last_active":"2024-02-10T08:42:35.146469+0000","last_peered":"2024-02-10T08:42:35.146469+0000","last_clean":"2024-02-10T08:42:35.146469+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:42:35.146469+0000","last_undegraded":"2024-02-10T08:42:35.146469+0000","last_fullsized":"2024-02-10T08:42:35.146469+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.559402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.559402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":424,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.141993+0000","last_change":"2024-02-10T08:41:43.195685+0000","last_active":"2024-02-10T08:42:35.141993+0000","last_peered":"2024-02-10T08:42:35.141993+0000","last_clean":"2024-02-10T08:42:35.141993+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:42:35.141993+0000","last_undegraded":"2024-02-10T08:42:35.141993+0000","last_fullsized":"2024-02-10T08:42:35.141993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":320,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.402393+0000","last_change":"2024-02-10T08:40:39.350744+0000","last_active":"2024-02-10T08:42:35.402393+0000","last_peered":"2024-02-10T08:42:35.402393+0000","last_clean":"2024-02-10T08:42:35.402393+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:42:35.402393+0000","last_undegraded":"2024-02-10T08:42:35.402393+0000","last_fullsized":"2024-02-10T08:42:35.402393+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.350663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":349,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.391107+0000","last_change":"2024-02-10T08:41:45.092740+0000","last_active":"2024-02-10T08:42:26.391107+0000","last_peered":"2024-02-10T08:42:26.391107+0000","last_clean":"2024-02-10T08:42:26.391107+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:42:26.391107+0000","last_undegraded":"2024-02-10T08:42:26.391107+0000","last_fullsized":"2024-02-10T08:42:26.391107+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:45.092620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.092620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":441,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.387791+0000","last_change":"2024-02-10T08:41:57.284919+0000","last_active":"2024-02-10T08:42:26.387791+0000","last_peered":"2024-02-10T08:42:26.387791+0000","last_clean":"2024-02-10T08:42:26.387791+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:42:26.387791+0000","last_undegraded":"2024-02-10T08:42:26.387791+0000","last_fullsized":"2024-02-10T08:42:26.387791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:57.284811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.284811+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":414,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140450+0000","last_change":"2024-02-10T08:40:40.314002+0000","last_active":"2024-02-10T08:42:35.140450+0000","last_peered":"2024-02-10T08:42:35.140450+0000","last_clean":"2024-02-10T08:42:35.140450+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:42:35.140450+0000","last_undegraded":"2024-02-10T08:42:35.140450+0000","last_fullsized":"2024-02-10T08:42:35.140450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.313935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":428,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.139702+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:42:35.139702+0000","last_peered":"2024-02-10T08:42:35.139702+0000","last_clean":"2024-02-10T08:42:35.139702+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:42:35.139702+0000","last_undegraded":"2024-02-10T08:42:35.139702+0000","last_fullsized":"2024-02-10T08:42:35.139702+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":396,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404270+0000","last_change":"2024-02-10T08:40:28.426111+0000","last_active":"2024-02-10T08:42:35.404270+0000","last_peered":"2024-02-10T08:42:35.404270+0000","last_clean":"2024-02-10T08:42:35.404270+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:42:35.404270+0000","last_undegraded":"2024-02-10T08:42:35.404270+0000","last_fullsized":"2024-02-10T08:42:35.404270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:28.425994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:28.425994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":350,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404350+0000","last_change":"2024-02-10T08:41:20.535962+0000","last_active":"2024-02-10T08:42:35.404350+0000","last_peered":"2024-02-10T08:42:35.404350+0000","last_clean":"2024-02-10T08:42:35.404350+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:42:35.404350+0000","last_undegraded":"2024-02-10T08:42:35.404350+0000","last_fullsized":"2024-02-10T08:42:35.404350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":434,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033691+0000","last_change":"2024-02-10T08:41:13.773323+0000","last_active":"2024-02-10T08:42:26.033691+0000","last_peered":"2024-02-10T08:42:26.033691+0000","last_clean":"2024-02-10T08:42:26.033691+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:42:26.033691+0000","last_undegraded":"2024-02-10T08:42:26.033691+0000","last_fullsized":"2024-02-10T08:42:26.033691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:41:13.773224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.773224+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":244,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.159822+0000","last_change":"2024-02-10T08:41:16.638540+0000","last_active":"2024-02-10T08:42:35.159822+0000","last_peered":"2024-02-10T08:42:35.159822+0000","last_clean":"2024-02-10T08:42:35.159822+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:42:35.159822+0000","last_undegraded":"2024-02-10T08:42:35.159822+0000","last_fullsized":"2024-02-10T08:42:35.159822+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.638477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.638477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":433,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145125+0000","last_change":"2024-02-10T08:42:29.596160+0000","last_active":"2024-02-10T08:42:35.145125+0000","last_peered":"2024-02-10T08:42:35.145125+0000","last_clean":"2024-02-10T08:42:35.145125+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:42:35.145125+0000","last_undegraded":"2024-02-10T08:42:35.145125+0000","last_fullsized":"2024-02-10T08:42:35.145125+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":404,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144544+0000","last_change":"2024-02-10T08:42:32.250073+0000","last_active":"2024-02-10T08:42:35.144544+0000","last_peered":"2024-02-10T08:42:35.144544+0000","last_clean":"2024-02-10T08:42:35.144544+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:42:35.144544+0000","last_undegraded":"2024-02-10T08:42:35.144544+0000","last_fullsized":"2024-02-10T08:42:35.144544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":351,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032313+0000","last_change":"2024-02-10T08:42:25.595454+0000","last_active":"2024-02-10T08:42:26.032313+0000","last_peered":"2024-02-10T08:42:26.032313+0000","last_clean":"2024-02-10T08:42:26.032313+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:42:26.032313+0000","last_undegraded":"2024-02-10T08:42:26.032313+0000","last_fullsized":"2024-02-10T08:42:26.032313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":421,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032853+0000","last_change":"2024-02-10T08:41:18.755591+0000","last_active":"2024-02-10T08:42:26.032853+0000","last_peered":"2024-02-10T08:42:26.032853+0000","last_clean":"2024-02-10T08:42:26.032853+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:42:26.032853+0000","last_undegraded":"2024-02-10T08:42:26.032853+0000","last_fullsized":"2024-02-10T08:42:26.032853+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":410,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144228+0000","last_change":"2024-02-10T08:41:10.625950+0000","last_active":"2024-02-10T08:42:35.144228+0000","last_peered":"2024-02-10T08:42:35.144228+0000","last_clean":"2024-02-10T08:42:35.144228+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:42:35.144228+0000","last_undegraded":"2024-02-10T08:42:35.144228+0000","last_fullsized":"2024-02-10T08:42:35.144228+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.625888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.625888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":540,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.143625+0000","last_change":"2024-02-10T08:40:41.276981+0000","last_active":"2024-02-10T08:42:35.143625+0000","last_peered":"2024-02-10T08:42:35.143625+0000","last_clean":"2024-02-10T08:42:35.143625+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:42:35.143625+0000","last_undegraded":"2024-02-10T08:42:35.143625+0000","last_fullsized":"2024-02-10T08:42:35.143625+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:40:41.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.276901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":402,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033720+0000","last_change":"2024-02-10T08:41:43.683458+0000","last_active":"2024-02-10T08:42:26.033720+0000","last_peered":"2024-02-10T08:42:26.033720+0000","last_clean":"2024-02-10T08:42:26.033720+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:42:26.033720+0000","last_undegraded":"2024-02-10T08:42:26.033720+0000","last_fullsized":"2024-02-10T08:42:26.033720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":318,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160361+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:42:35.160361+0000","last_peered":"2024-02-10T08:42:35.160361+0000","last_clean":"2024-02-10T08:42:35.160361+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:42:35.160361+0000","last_undegraded":"2024-02-10T08:42:35.160361+0000","last_fullsized":"2024-02-10T08:42:35.160361+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":433,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146832+0000","last_change":"2024-02-10T08:42:32.661333+0000","last_active":"2024-02-10T08:42:35.146832+0000","last_peered":"2024-02-10T08:42:35.146832+0000","last_clean":"2024-02-10T08:42:35.146832+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:42:35.146832+0000","last_undegraded":"2024-02-10T08:42:35.146832+0000","last_fullsized":"2024-02-10T08:42:35.146832+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":418,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.139497+0000","last_change":"2024-02-10T08:41:21.599108+0000","last_active":"2024-02-10T08:42:35.139497+0000","last_peered":"2024-02-10T08:42:35.139497+0000","last_clean":"2024-02-10T08:42:35.139497+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:42:35.139497+0000","last_undegraded":"2024-02-10T08:42:35.139497+0000","last_fullsized":"2024-02-10T08:42:35.139497+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:21.599034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.599034+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":563,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.599362+0000","last_change":"2024-02-10T08:40:25.631962+0000","last_active":"2024-02-10T08:42:31.599362+0000","last_peered":"2024-02-10T08:42:31.599362+0000","last_clean":"2024-02-10T08:42:31.599362+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:42:31.599362+0000","last_undegraded":"2024-02-10T08:42:31.599362+0000","last_fullsized":"2024-02-10T08:42:31.599362+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:40:25.631854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:25.631854+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":409,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:34.139708+0000","last_change":"2024-02-10T08:42:34.139662+0000","last_active":"2024-02-10T08:42:34.139708+0000","last_peered":"2024-02-10T08:42:34.139708+0000","last_clean":"2024-02-10T08:42:34.139708+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:42:34.139708+0000","last_undegraded":"2024-02-10T08:42:34.139708+0000","last_fullsized":"2024-02-10T08:42:34.139708+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.139528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.139528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":358,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390298+0000","last_change":"2024-02-10T08:41:54.063932+0000","last_active":"2024-02-10T08:42:26.390298+0000","last_peered":"2024-02-10T08:42:26.390298+0000","last_clean":"2024-02-10T08:42:26.390298+0000","last_became_active":"2024-02-10T08:40:42.028734+0000","last_became_peered":"2024-02-10T08:40:42.028734+0000","last_unstale":"2024-02-10T08:42:26.390298+0000","last_undegraded":"2024-02-10T08:42:26.390298+0000","last_fullsized":"2024-02-10T08:42:26.390298+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:54.063839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.063839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":425,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033496+0000","last_change":"2024-02-10T08:41:56.714215+0000","last_active":"2024-02-10T08:42:26.033496+0000","last_peered":"2024-02-10T08:42:26.033496+0000","last_clean":"2024-02-10T08:42:26.033496+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:42:26.033496+0000","last_undegraded":"2024-02-10T08:42:26.033496+0000","last_fullsized":"2024-02-10T08:42:26.033496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":416,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.405161+0000","last_change":"2024-02-10T08:41:08.390794+0000","last_active":"2024-02-10T08:42:35.405161+0000","last_peered":"2024-02-10T08:42:35.405161+0000","last_clean":"2024-02-10T08:42:35.405161+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:42:35.405161+0000","last_undegraded":"2024-02-10T08:42:35.405161+0000","last_fullsized":"2024-02-10T08:42:35.405161+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.390722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.390722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":494,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.599358+0000","last_change":"2024-02-10T08:42:29.051548+0000","last_active":"2024-02-10T08:42:31.599358+0000","last_peered":"2024-02-10T08:42:31.599358+0000","last_clean":"2024-02-10T08:42:31.599358+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:42:31.599358+0000","last_undegraded":"2024-02-10T08:42:31.599358+0000","last_fullsized":"2024-02-10T08:42:31.599358+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":398,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162662+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:42:35.162662+0000","last_peered":"2024-02-10T08:42:35.162662+0000","last_clean":"2024-02-10T08:42:35.162662+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:42:35.162662+0000","last_undegraded":"2024-02-10T08:42:35.162662+0000","last_fullsized":"2024-02-10T08:42:35.162662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":348,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160056+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:42:35.160056+0000","last_peered":"2024-02-10T08:42:35.160056+0000","last_clean":"2024-02-10T08:42:35.160056+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:42:35.160056+0000","last_undegraded":"2024-02-10T08:42:35.160056+0000","last_fullsized":"2024-02-10T08:42:35.160056+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":527,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160562+0000","last_change":"2024-02-10T08:41:44.773002+0000","last_active":"2024-02-10T08:42:35.160562+0000","last_peered":"2024-02-10T08:42:35.160562+0000","last_clean":"2024-02-10T08:42:35.160562+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:42:35.160562+0000","last_undegraded":"2024-02-10T08:42:35.160562+0000","last_fullsized":"2024-02-10T08:42:35.160562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":427,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032888+0000","last_change":"2024-02-10T08:41:24.848074+0000","last_active":"2024-02-10T08:42:26.032888+0000","last_peered":"2024-02-10T08:42:26.032888+0000","last_clean":"2024-02-10T08:42:26.032888+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:42:26.032888+0000","last_undegraded":"2024-02-10T08:42:26.032888+0000","last_fullsized":"2024-02-10T08:42:26.032888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:41:24.847990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.847990+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":412,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388902+0000","last_change":"2024-02-10T08:41:48.361432+0000","last_active":"2024-02-10T08:42:26.388902+0000","last_peered":"2024-02-10T08:42:26.388902+0000","last_clean":"2024-02-10T08:42:26.388902+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:42:26.388902+0000","last_undegraded":"2024-02-10T08:42:26.388902+0000","last_fullsized":"2024-02-10T08:42:26.388902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.361357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.361357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":401,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.403769+0000","last_change":"2024-02-10T08:41:59.569272+0000","last_active":"2024-02-10T08:42:35.403769+0000","last_peered":"2024-02-10T08:42:35.403769+0000","last_clean":"2024-02-10T08:42:35.403769+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:42:35.403769+0000","last_undegraded":"2024-02-10T08:42:35.403769+0000","last_fullsized":"2024-02-10T08:42:35.403769+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.569193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.569193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":323,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146485+0000","last_change":"2024-02-10T08:41:57.599784+0000","last_active":"2024-02-10T08:42:35.146485+0000","last_peered":"2024-02-10T08:42:35.146485+0000","last_clean":"2024-02-10T08:42:35.146485+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:42:35.146485+0000","last_undegraded":"2024-02-10T08:42:35.146485+0000","last_fullsized":"2024-02-10T08:42:35.146485+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":401,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033071+0000","last_change":"2024-02-10T08:40:34.622947+0000","last_active":"2024-02-10T08:42:26.033071+0000","last_peered":"2024-02-10T08:42:26.033071+0000","last_clean":"2024-02-10T08:42:26.033071+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:42:26.033071+0000","last_undegraded":"2024-02-10T08:42:26.033071+0000","last_fullsized":"2024-02-10T08:42:26.033071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:34.622869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":496,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160535+0000","last_change":"2024-02-10T08:41:07.712140+0000","last_active":"2024-02-10T08:42:35.160535+0000","last_peered":"2024-02-10T08:42:35.160535+0000","last_clean":"2024-02-10T08:42:35.160535+0000","last_became_active":"2024-02-10T08:40:42.015369+0000","last_became_peered":"2024-02-10T08:40:42.015369+0000","last_unstale":"2024-02-10T08:42:35.160535+0000","last_undegraded":"2024-02-10T08:42:35.160535+0000","last_fullsized":"2024-02-10T08:42:35.160535+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:41:07.712078+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.712078+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":417,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145785+0000","last_change":"2024-02-10T08:41:46.729506+0000","last_active":"2024-02-10T08:42:35.145785+0000","last_peered":"2024-02-10T08:42:35.145785+0000","last_clean":"2024-02-10T08:42:35.145785+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:42:35.145785+0000","last_undegraded":"2024-02-10T08:42:35.145785+0000","last_fullsized":"2024-02-10T08:42:35.145785+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":397,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033318+0000","last_change":"2024-02-10T08:41:10.710795+0000","last_active":"2024-02-10T08:42:26.033318+0000","last_peered":"2024-02-10T08:42:26.033318+0000","last_clean":"2024-02-10T08:42:26.033318+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:42:26.033318+0000","last_undegraded":"2024-02-10T08:42:26.033318+0000","last_fullsized":"2024-02-10T08:42:26.033318+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":332,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162489+0000","last_change":"2024-02-10T08:42:35.024210+0000","last_active":"2024-02-10T08:42:35.162489+0000","last_peered":"2024-02-10T08:42:35.162489+0000","last_clean":"2024-02-10T08:42:35.162489+0000","last_became_active":"2024-02-10T08:40:42.027869+0000","last_became_peered":"2024-02-10T08:40:42.027869+0000","last_unstale":"2024-02-10T08:42:35.162489+0000","last_undegraded":"2024-02-10T08:42:35.162489+0000","last_fullsized":"2024-02-10T08:42:35.162489+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":417,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.387883+0000","last_change":"2024-02-10T08:41:58.271798+0000","last_active":"2024-02-10T08:42:26.387883+0000","last_peered":"2024-02-10T08:42:26.387883+0000","last_clean":"2024-02-10T08:42:26.387883+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:42:26.387883+0000","last_undegraded":"2024-02-10T08:42:26.387883+0000","last_fullsized":"2024-02-10T08:42:26.387883+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.271728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.271728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":385,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390723+0000","last_change":"2024-02-10T08:41:07.950673+0000","last_active":"2024-02-10T08:42:26.390723+0000","last_peered":"2024-02-10T08:42:26.390723+0000","last_clean":"2024-02-10T08:42:26.390723+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:42:26.390723+0000","last_undegraded":"2024-02-10T08:42:26.390723+0000","last_fullsized":"2024-02-10T08:42:26.390723+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:07.950597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.950597+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":416,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404075+0000","last_change":"2024-02-10T08:41:04.369613+0000","last_active":"2024-02-10T08:42:35.404075+0000","last_peered":"2024-02-10T08:42:35.404075+0000","last_clean":"2024-02-10T08:42:35.404075+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:42:35.404075+0000","last_undegraded":"2024-02-10T08:42:35.404075+0000","last_fullsized":"2024-02-10T08:42:35.404075+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.369534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.369534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":67,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140609+0000","last_change":"2024-02-10T08:41:58.300293+0000","last_active":"2024-02-10T08:42:35.140609+0000","last_peered":"2024-02-10T08:42:35.140609+0000","last_clean":"2024-02-10T08:42:35.140609+0000","last_became_active":"2024-02-10T08:41:21.266720+0000","last_became_peered":"2024-02-10T08:41:21.266720+0000","last_unstale":"2024-02-10T08:42:35.140609+0000","last_undegraded":"2024-02-10T08:42:35.140609+0000","last_fullsized":"2024-02-10T08:42:35.140609+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.300183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.300183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":362,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144313+0000","last_change":"2024-02-10T08:41:50.202162+0000","last_active":"2024-02-10T08:42:35.144313+0000","last_peered":"2024-02-10T08:42:35.144313+0000","last_clean":"2024-02-10T08:42:35.144313+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:42:35.144313+0000","last_undegraded":"2024-02-10T08:42:35.144313+0000","last_fullsized":"2024-02-10T08:42:35.144313+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:50.202092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:50.202092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":429,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.391034+0000","last_change":"2024-02-10T08:41:11.901601+0000","last_active":"2024-02-10T08:42:26.391034+0000","last_peered":"2024-02-10T08:42:26.391034+0000","last_clean":"2024-02-10T08:42:26.391034+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:42:26.391034+0000","last_undegraded":"2024-02-10T08:42:26.391034+0000","last_fullsized":"2024-02-10T08:42:26.391034+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.901502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.901502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":367,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145877+0000","last_change":"2024-02-10T08:41:07.584245+0000","last_active":"2024-02-10T08:42:35.145877+0000","last_peered":"2024-02-10T08:42:35.145877+0000","last_clean":"2024-02-10T08:42:35.145877+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:42:35.145877+0000","last_undegraded":"2024-02-10T08:42:35.145877+0000","last_fullsized":"2024-02-10T08:42:35.145877+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:07.584127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.584127+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404460+0000","last_change":"2024-02-10T08:42:30.494713+0000","last_active":"2024-02-10T08:42:35.404460+0000","last_peered":"2024-02-10T08:42:35.404460+0000","last_clean":"2024-02-10T08:42:35.404460+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:42:35.404460+0000","last_undegraded":"2024-02-10T08:42:35.404460+0000","last_fullsized":"2024-02-10T08:42:35.404460+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:30.494591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.494591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":321,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032577+0000","last_change":"2024-02-10T08:41:21.839964+0000","last_active":"2024-02-10T08:42:26.032577+0000","last_peered":"2024-02-10T08:42:26.032577+0000","last_clean":"2024-02-10T08:42:26.032577+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:42:26.032577+0000","last_undegraded":"2024-02-10T08:42:26.032577+0000","last_fullsized":"2024-02-10T08:42:26.032577+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":405,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145222+0000","last_change":"2024-02-10T08:41:12.554311+0000","last_active":"2024-02-10T08:42:35.145222+0000","last_peered":"2024-02-10T08:42:35.145222+0000","last_clean":"2024-02-10T08:42:35.145222+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:42:35.145222+0000","last_undegraded":"2024-02-10T08:42:35.145222+0000","last_fullsized":"2024-02-10T08:42:35.145222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:41:12.554232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.554232+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":364,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:32.390240+0000","last_change":"2024-02-10T08:42:32.390215+0000","last_active":"2024-02-10T08:42:32.390240+0000","last_peered":"2024-02-10T08:42:32.390240+0000","last_clean":"2024-02-10T08:42:32.390240+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:42:32.390240+0000","last_undegraded":"2024-02-10T08:42:32.390240+0000","last_fullsized":"2024-02-10T08:42:32.390240+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.390146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.390146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":403,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404121+0000","last_change":"2024-02-10T08:41:19.563777+0000","last_active":"2024-02-10T08:42:35.404121+0000","last_peered":"2024-02-10T08:42:35.404121+0000","last_clean":"2024-02-10T08:42:35.404121+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:42:35.404121+0000","last_undegraded":"2024-02-10T08:42:35.404121+0000","last_fullsized":"2024-02-10T08:42:35.404121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:19.563700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.563700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":432,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388534+0000","last_change":"2024-02-10T08:42:00.249642+0000","last_active":"2024-02-10T08:42:26.388534+0000","last_peered":"2024-02-10T08:42:26.388534+0000","last_clean":"2024-02-10T08:42:26.388534+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:42:26.388534+0000","last_undegraded":"2024-02-10T08:42:26.388534+0000","last_fullsized":"2024-02-10T08:42:26.388534+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":424,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404468+0000","last_change":"2024-02-10T08:41:24.519217+0000","last_active":"2024-02-10T08:42:35.404468+0000","last_peered":"2024-02-10T08:42:35.404468+0000","last_clean":"2024-02-10T08:42:35.404468+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:42:35.404468+0000","last_undegraded":"2024-02-10T08:42:35.404468+0000","last_fullsized":"2024-02-10T08:42:35.404468+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.519144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.519144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":406,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390491+0000","last_change":"2024-02-10T08:41:13.978089+0000","last_active":"2024-02-10T08:42:26.390491+0000","last_peered":"2024-02-10T08:42:26.390491+0000","last_clean":"2024-02-10T08:42:26.390491+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:42:26.390491+0000","last_undegraded":"2024-02-10T08:42:26.390491+0000","last_fullsized":"2024-02-10T08:42:26.390491+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.978013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.978013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":404,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160341+0000","last_change":"2024-02-10T08:40:39.891622+0000","last_active":"2024-02-10T08:42:35.160341+0000","last_peered":"2024-02-10T08:42:35.160341+0000","last_clean":"2024-02-10T08:42:35.160341+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:42:35.160341+0000","last_undegraded":"2024-02-10T08:42:35.160341+0000","last_fullsized":"2024-02-10T08:42:35.160341+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.891558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.891558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":400,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.391108+0000","last_change":"2024-02-10T08:41:48.065347+0000","last_active":"2024-02-10T08:42:26.391108+0000","last_peered":"2024-02-10T08:42:26.391108+0000","last_clean":"2024-02-10T08:42:26.391108+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:42:26.391108+0000","last_undegraded":"2024-02-10T08:42:26.391108+0000","last_fullsized":"2024-02-10T08:42:26.391108+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":360,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.389915+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:42:26.389915+0000","last_peered":"2024-02-10T08:42:26.389915+0000","last_clean":"2024-02-10T08:42:26.389915+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:42:26.389915+0000","last_undegraded":"2024-02-10T08:42:26.389915+0000","last_fullsized":"2024-02-10T08:42:26.389915+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":419,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145209+0000","last_change":"2024-02-10T08:41:03.238150+0000","last_active":"2024-02-10T08:42:35.145209+0000","last_peered":"2024-02-10T08:42:35.145209+0000","last_clean":"2024-02-10T08:42:35.145209+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:42:35.145209+0000","last_undegraded":"2024-02-10T08:42:35.145209+0000","last_fullsized":"2024-02-10T08:42:35.145209+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.238035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.238035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":456,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146749+0000","last_change":"2024-02-10T08:41:18.579612+0000","last_active":"2024-02-10T08:42:35.146749+0000","last_peered":"2024-02-10T08:42:35.146749+0000","last_clean":"2024-02-10T08:42:35.146749+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:42:35.146749+0000","last_undegraded":"2024-02-10T08:42:35.146749+0000","last_fullsized":"2024-02-10T08:42:35.146749+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:41:18.579532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.579532+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140568+0000","last_change":"2024-02-10T08:42:29.250521+0000","last_active":"2024-02-10T08:42:35.140568+0000","last_peered":"2024-02-10T08:42:35.140568+0000","last_clean":"2024-02-10T08:42:35.140568+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:42:35.140568+0000","last_undegraded":"2024-02-10T08:42:35.140568+0000","last_fullsized":"2024-02-10T08:42:35.140568+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.250458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.250458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":964,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.141173+0000","last_change":"2024-02-10T08:41:44.751436+0000","last_active":"2024-02-10T08:42:35.141173+0000","last_peered":"2024-02-10T08:42:35.141173+0000","last_clean":"2024-02-10T08:42:35.141173+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:42:35.141173+0000","last_undegraded":"2024-02-10T08:42:35.141173+0000","last_fullsized":"2024-02-10T08:42:35.141173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:41:44.751371+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.751371+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":550,"num_read_kb":545,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":401,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.403957+0000","last_change":"2024-02-10T08:41:15.515804+0000","last_active":"2024-02-10T08:42:35.403957+0000","last_peered":"2024-02-10T08:42:35.403957+0000","last_clean":"2024-02-10T08:42:35.403957+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:42:35.403957+0000","last_undegraded":"2024-02-10T08:42:35.403957+0000","last_fullsized":"2024-02-10T08:42:35.403957+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.515729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.515729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":362,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145911+0000","last_change":"2024-02-10T08:42:28.553646+0000","last_active":"2024-02-10T08:42:35.145911+0000","last_peered":"2024-02-10T08:42:35.145911+0000","last_clean":"2024-02-10T08:42:35.145911+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:42:35.145911+0000","last_undegraded":"2024-02-10T08:42:35.145911+0000","last_fullsized":"2024-02-10T08:42:35.145911+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:28.553564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.553564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":323,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146298+0000","last_change":"2024-02-10T08:41:53.594734+0000","last_active":"2024-02-10T08:42:35.146298+0000","last_peered":"2024-02-10T08:42:35.146298+0000","last_clean":"2024-02-10T08:42:35.146298+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:42:35.146298+0000","last_undegraded":"2024-02-10T08:42:35.146298+0000","last_fullsized":"2024-02-10T08:42:35.146298+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":502,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.599424+0000","last_change":"2024-02-10T08:40:40.098585+0000","last_active":"2024-02-10T08:42:31.599424+0000","last_peered":"2024-02-10T08:42:31.599424+0000","last_clean":"2024-02-10T08:42:31.599424+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:42:31.599424+0000","last_undegraded":"2024-02-10T08:42:31.599424+0000","last_fullsized":"2024-02-10T08:42:31.599424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:40:40.098515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.098515+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":406,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390586+0000","last_change":"2024-02-10T08:40:23.818554+0000","last_active":"2024-02-10T08:42:26.390586+0000","last_peered":"2024-02-10T08:42:26.390586+0000","last_clean":"2024-02-10T08:42:26.390586+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:42:26.390586+0000","last_undegraded":"2024-02-10T08:42:26.390586+0000","last_fullsized":"2024-02-10T08:42:26.390586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:23.818434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.818434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":398,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164324+0000","last_change":"2024-02-10T08:41:11.702739+0000","last_active":"2024-02-10T08:42:35.164324+0000","last_peered":"2024-02-10T08:42:35.164324+0000","last_clean":"2024-02-10T08:42:35.164324+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:42:35.164324+0000","last_undegraded":"2024-02-10T08:42:35.164324+0000","last_fullsized":"2024-02-10T08:42:35.164324+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":362,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.142147+0000","last_change":"2024-02-10T08:42:04.361526+0000","last_active":"2024-02-10T08:42:35.142147+0000","last_peered":"2024-02-10T08:42:35.142147+0000","last_clean":"2024-02-10T08:42:35.142147+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:42:35.142147+0000","last_undegraded":"2024-02-10T08:42:35.142147+0000","last_fullsized":"2024-02-10T08:42:35.142147+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:04.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.361430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":413,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.599357+0000","last_change":"2024-02-10T08:41:54.316045+0000","last_active":"2024-02-10T08:42:31.599357+0000","last_peered":"2024-02-10T08:42:31.599357+0000","last_clean":"2024-02-10T08:42:31.599357+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:42:31.599357+0000","last_undegraded":"2024-02-10T08:42:31.599357+0000","last_fullsized":"2024-02-10T08:42:31.599357+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:41:54.315983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.315983+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":428,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144109+0000","last_change":"2024-02-10T08:41:03.584639+0000","last_active":"2024-02-10T08:42:35.144109+0000","last_peered":"2024-02-10T08:42:35.144109+0000","last_clean":"2024-02-10T08:42:35.144109+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:42:35.144109+0000","last_undegraded":"2024-02-10T08:42:35.144109+0000","last_fullsized":"2024-02-10T08:42:35.144109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:41:03.584573+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.584573+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":419,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144922+0000","last_change":"2024-02-10T08:41:24.622753+0000","last_active":"2024-02-10T08:42:35.144922+0000","last_peered":"2024-02-10T08:42:35.144922+0000","last_clean":"2024-02-10T08:42:35.144922+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:42:35.144922+0000","last_undegraded":"2024-02-10T08:42:35.144922+0000","last_fullsized":"2024-02-10T08:42:35.144922+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.622691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.622691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":318,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160454+0000","last_change":"2024-02-10T08:39:36.871588+0000","last_active":"2024-02-10T08:42:35.160454+0000","last_peered":"2024-02-10T08:42:35.160454+0000","last_clean":"2024-02-10T08:42:35.160454+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:42:35.160454+0000","last_undegraded":"2024-02-10T08:42:35.160454+0000","last_fullsized":"2024-02-10T08:42:35.160454+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:36.871524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:36.871524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164017+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:42:35.164017+0000","last_peered":"2024-02-10T08:42:35.164017+0000","last_clean":"2024-02-10T08:42:35.164017+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:42:35.164017+0000","last_undegraded":"2024-02-10T08:42:35.164017+0000","last_fullsized":"2024-02-10T08:42:35.164017+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":433,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.142457+0000","last_change":"2024-02-10T08:41:22.395964+0000","last_active":"2024-02-10T08:42:35.142457+0000","last_peered":"2024-02-10T08:42:35.142457+0000","last_clean":"2024-02-10T08:42:35.142457+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:42:35.142457+0000","last_undegraded":"2024-02-10T08:42:35.142457+0000","last_fullsized":"2024-02-10T08:42:35.142457+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.395872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.395872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":434,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032997+0000","last_change":"2024-02-10T08:41:44.633272+0000","last_active":"2024-02-10T08:42:26.032997+0000","last_peered":"2024-02-10T08:42:26.032997+0000","last_clean":"2024-02-10T08:42:26.032997+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:42:26.032997+0000","last_undegraded":"2024-02-10T08:42:26.032997+0000","last_fullsized":"2024-02-10T08:42:26.032997+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":431,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033766+0000","last_change":"2024-02-10T08:40:38.667994+0000","last_active":"2024-02-10T08:42:26.033766+0000","last_peered":"2024-02-10T08:42:26.033766+0000","last_clean":"2024-02-10T08:42:26.033766+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:42:26.033766+0000","last_undegraded":"2024-02-10T08:42:26.033766+0000","last_fullsized":"2024-02-10T08:42:26.033766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:40:38.667923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:38.667923+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":403,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160758+0000","last_change":"2024-02-10T08:42:05.067474+0000","last_active":"2024-02-10T08:42:35.160758+0000","last_peered":"2024-02-10T08:42:35.160758+0000","last_clean":"2024-02-10T08:42:35.160758+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:42:35.160758+0000","last_undegraded":"2024-02-10T08:42:35.160758+0000","last_fullsized":"2024-02-10T08:42:35.160758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.143543+0000","last_change":"2024-02-10T08:41:15.304181+0000","last_active":"2024-02-10T08:42:35.143543+0000","last_peered":"2024-02-10T08:42:35.143543+0000","last_clean":"2024-02-10T08:42:35.143543+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:42:35.143543+0000","last_undegraded":"2024-02-10T08:42:35.143543+0000","last_fullsized":"2024-02-10T08:42:35.143543+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.304104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.304104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":389,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164407+0000","last_change":"2024-02-10T08:41:23.736997+0000","last_active":"2024-02-10T08:42:35.164407+0000","last_peered":"2024-02-10T08:42:35.164407+0000","last_clean":"2024-02-10T08:42:35.164407+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:42:35.164407+0000","last_undegraded":"2024-02-10T08:42:35.164407+0000","last_fullsized":"2024-02-10T08:42:35.164407+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.736922+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":419,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.142380+0000","last_change":"2024-02-10T08:41:21.384611+0000","last_active":"2024-02-10T08:42:35.142380+0000","last_peered":"2024-02-10T08:42:35.142380+0000","last_clean":"2024-02-10T08:42:35.142380+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:42:35.142380+0000","last_undegraded":"2024-02-10T08:42:35.142380+0000","last_fullsized":"2024-02-10T08:42:35.142380+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":399,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140261+0000","last_change":"2024-02-10T08:40:37.417370+0000","last_active":"2024-02-10T08:42:35.140261+0000","last_peered":"2024-02-10T08:42:35.140261+0000","last_clean":"2024-02-10T08:42:35.140261+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:42:35.140261+0000","last_undegraded":"2024-02-10T08:42:35.140261+0000","last_fullsized":"2024-02-10T08:42:35.140261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:37.417303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.417303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":351,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.615128+0000","last_change":"2024-02-10T08:42:26.615082+0000","last_active":"2024-02-10T08:42:26.615128+0000","last_peered":"2024-02-10T08:42:26.615128+0000","last_clean":"2024-02-10T08:42:26.615128+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:42:26.615128+0000","last_undegraded":"2024-02-10T08:42:26.615128+0000","last_fullsized":"2024-02-10T08:42:26.615128+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":411,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164086+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:42:35.164086+0000","last_peered":"2024-02-10T08:42:35.164086+0000","last_clean":"2024-02-10T08:42:35.164086+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:42:35.164086+0000","last_undegraded":"2024-02-10T08:42:35.164086+0000","last_fullsized":"2024-02-10T08:42:35.164086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":413,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.406999+0000","last_change":"2024-02-10T08:42:02.183760+0000","last_active":"2024-02-10T08:42:35.406999+0000","last_peered":"2024-02-10T08:42:35.406999+0000","last_clean":"2024-02-10T08:42:35.406999+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:42:35.406999+0000","last_undegraded":"2024-02-10T08:42:35.406999+0000","last_fullsized":"2024-02-10T08:42:35.406999+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":401,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.403524+0000","last_change":"2024-02-10T08:42:35.402000+0000","last_active":"2024-02-10T08:42:35.403524+0000","last_peered":"2024-02-10T08:42:35.403524+0000","last_clean":"2024-02-10T08:42:35.403524+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:42:35.403524+0000","last_undegraded":"2024-02-10T08:42:35.403524+0000","last_fullsized":"2024-02-10T08:42:35.403524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":349,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390412+0000","last_change":"2024-02-10T08:41:23.032812+0000","last_active":"2024-02-10T08:42:26.390412+0000","last_peered":"2024-02-10T08:42:26.390412+0000","last_clean":"2024-02-10T08:42:26.390412+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:42:26.390412+0000","last_undegraded":"2024-02-10T08:42:26.390412+0000","last_fullsized":"2024-02-10T08:42:26.390412+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":422,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162708+0000","last_change":"2024-02-10T08:42:24.087582+0000","last_active":"2024-02-10T08:42:35.162708+0000","last_peered":"2024-02-10T08:42:35.162708+0000","last_clean":"2024-02-10T08:42:35.162708+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:42:35.162708+0000","last_undegraded":"2024-02-10T08:42:35.162708+0000","last_fullsized":"2024-02-10T08:42:35.162708+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":406,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390419+0000","last_change":"2024-02-10T08:40:35.021264+0000","last_active":"2024-02-10T08:42:26.390419+0000","last_peered":"2024-02-10T08:42:26.390419+0000","last_clean":"2024-02-10T08:42:26.390419+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:42:26.390419+0000","last_undegraded":"2024-02-10T08:42:26.390419+0000","last_fullsized":"2024-02-10T08:42:26.390419+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.021169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.021169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":534,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:31.599370+0000","last_change":"2024-02-10T08:42:28.090195+0000","last_active":"2024-02-10T08:42:31.599370+0000","last_peered":"2024-02-10T08:42:31.599370+0000","last_clean":"2024-02-10T08:42:31.599370+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:42:31.599370+0000","last_undegraded":"2024-02-10T08:42:31.599370+0000","last_fullsized":"2024-02-10T08:42:31.599370+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:42:28.090111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.090111+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":320,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404420+0000","last_change":"2024-02-10T08:41:16.497083+0000","last_active":"2024-02-10T08:42:35.404420+0000","last_peered":"2024-02-10T08:42:35.404420+0000","last_clean":"2024-02-10T08:42:35.404420+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:42:35.404420+0000","last_undegraded":"2024-02-10T08:42:35.404420+0000","last_fullsized":"2024-02-10T08:42:35.404420+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.496995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.496995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":321,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.408847+0000","last_change":"2024-02-10T08:41:21.463600+0000","last_active":"2024-02-10T08:42:35.408847+0000","last_peered":"2024-02-10T08:42:35.408847+0000","last_clean":"2024-02-10T08:42:35.408847+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:42:35.408847+0000","last_undegraded":"2024-02-10T08:42:35.408847+0000","last_fullsized":"2024-02-10T08:42:35.408847+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.463527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.463527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":420,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390474+0000","last_change":"2024-02-10T08:42:05.106561+0000","last_active":"2024-02-10T08:42:26.390474+0000","last_peered":"2024-02-10T08:42:26.390474+0000","last_clean":"2024-02-10T08:42:26.390474+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:42:26.390474+0000","last_undegraded":"2024-02-10T08:42:26.390474+0000","last_fullsized":"2024-02-10T08:42:26.390474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":316,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032677+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:42:26.032677+0000","last_peered":"2024-02-10T08:42:26.032677+0000","last_clean":"2024-02-10T08:42:26.032677+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:42:26.032677+0000","last_undegraded":"2024-02-10T08:42:26.032677+0000","last_fullsized":"2024-02-10T08:42:26.032677+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":524,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.163753+0000","last_change":"2024-02-10T08:42:28.051493+0000","last_active":"2024-02-10T08:42:35.163753+0000","last_peered":"2024-02-10T08:42:35.163753+0000","last_clean":"2024-02-10T08:42:35.163753+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:42:35.163753+0000","last_undegraded":"2024-02-10T08:42:35.163753+0000","last_fullsized":"2024-02-10T08:42:35.163753+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":397,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033918+0000","last_change":"2024-02-10T08:40:41.538200+0000","last_active":"2024-02-10T08:42:26.033918+0000","last_peered":"2024-02-10T08:42:26.033918+0000","last_clean":"2024-02-10T08:42:26.033918+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:42:26.033918+0000","last_undegraded":"2024-02-10T08:42:26.033918+0000","last_fullsized":"2024-02-10T08:42:26.033918+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:41.538139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.538139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":361,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140461+0000","last_change":"2024-02-10T08:39:54.437376+0000","last_active":"2024-02-10T08:42:35.140461+0000","last_peered":"2024-02-10T08:42:35.140461+0000","last_clean":"2024-02-10T08:42:35.140461+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:42:35.140461+0000","last_undegraded":"2024-02-10T08:42:35.140461+0000","last_fullsized":"2024-02-10T08:42:35.140461+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:39:54.437272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.437272+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":430,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.143968+0000","last_change":"2024-02-10T08:42:25.482887+0000","last_active":"2024-02-10T08:42:35.143968+0000","last_peered":"2024-02-10T08:42:35.143968+0000","last_clean":"2024-02-10T08:42:35.143968+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:42:35.143968+0000","last_undegraded":"2024-02-10T08:42:35.143968+0000","last_fullsized":"2024-02-10T08:42:35.143968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:42:25.482788+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.482788+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":412,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388666+0000","last_change":"2024-02-10T08:41:44.355018+0000","last_active":"2024-02-10T08:42:26.388666+0000","last_peered":"2024-02-10T08:42:26.388666+0000","last_clean":"2024-02-10T08:42:26.388666+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:42:26.388666+0000","last_undegraded":"2024-02-10T08:42:26.388666+0000","last_fullsized":"2024-02-10T08:42:26.388666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:44.354945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.354945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":414,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160714+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:42:35.160714+0000","last_peered":"2024-02-10T08:42:35.160714+0000","last_clean":"2024-02-10T08:42:35.160714+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:42:35.160714+0000","last_undegraded":"2024-02-10T08:42:35.160714+0000","last_fullsized":"2024-02-10T08:42:35.160714+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":319,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.389759+0000","last_change":"2024-02-10T08:41:12.929709+0000","last_active":"2024-02-10T08:42:26.389759+0000","last_peered":"2024-02-10T08:42:26.389759+0000","last_clean":"2024-02-10T08:42:26.389759+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:42:26.389759+0000","last_undegraded":"2024-02-10T08:42:26.389759+0000","last_fullsized":"2024-02-10T08:42:26.389759+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.929592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.929592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":359,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388653+0000","last_change":"2024-02-10T08:41:20.506111+0000","last_active":"2024-02-10T08:42:26.388653+0000","last_peered":"2024-02-10T08:42:26.388653+0000","last_clean":"2024-02-10T08:42:26.388653+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:42:26.388653+0000","last_undegraded":"2024-02-10T08:42:26.388653+0000","last_fullsized":"2024-02-10T08:42:26.388653+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.506032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.506032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":420,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032901+0000","last_change":"2024-02-10T08:41:05.682672+0000","last_active":"2024-02-10T08:42:26.032901+0000","last_peered":"2024-02-10T08:42:26.032901+0000","last_clean":"2024-02-10T08:42:26.032901+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:42:26.032901+0000","last_undegraded":"2024-02-10T08:42:26.032901+0000","last_fullsized":"2024-02-10T08:42:26.032901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:05.682577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.682577+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":421,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146403+0000","last_change":"2024-02-10T08:41:56.648067+0000","last_active":"2024-02-10T08:42:35.146403+0000","last_peered":"2024-02-10T08:42:35.146403+0000","last_clean":"2024-02-10T08:42:35.146403+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:42:35.146403+0000","last_undegraded":"2024-02-10T08:42:35.146403+0000","last_fullsized":"2024-02-10T08:42:35.146403+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.647993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":406,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.144039+0000","last_change":"2024-02-10T08:42:31.622734+0000","last_active":"2024-02-10T08:42:35.144039+0000","last_peered":"2024-02-10T08:42:35.144039+0000","last_clean":"2024-02-10T08:42:35.144039+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:42:35.144039+0000","last_undegraded":"2024-02-10T08:42:35.144039+0000","last_fullsized":"2024-02-10T08:42:35.144039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:31.622670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.622670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":412,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.161312+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:42:35.161312+0000","last_peered":"2024-02-10T08:42:35.161312+0000","last_clean":"2024-02-10T08:42:35.161312+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:42:35.161312+0000","last_undegraded":"2024-02-10T08:42:35.161312+0000","last_fullsized":"2024-02-10T08:42:35.161312+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":362,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.163965+0000","last_change":"2024-02-10T08:42:32.012914+0000","last_active":"2024-02-10T08:42:35.163965+0000","last_peered":"2024-02-10T08:42:35.163965+0000","last_clean":"2024-02-10T08:42:35.163965+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:42:35.163965+0000","last_undegraded":"2024-02-10T08:42:35.163965+0000","last_fullsized":"2024-02-10T08:42:35.163965+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":408,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160472+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:42:35.160472+0000","last_peered":"2024-02-10T08:42:35.160472+0000","last_clean":"2024-02-10T08:42:35.160472+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:42:35.160472+0000","last_undegraded":"2024-02-10T08:42:35.160472+0000","last_fullsized":"2024-02-10T08:42:35.160472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":406,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390880+0000","last_change":"2024-02-10T08:41:05.909459+0000","last_active":"2024-02-10T08:42:26.390880+0000","last_peered":"2024-02-10T08:42:26.390880+0000","last_clean":"2024-02-10T08:42:26.390880+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:42:26.390880+0000","last_undegraded":"2024-02-10T08:42:26.390880+0000","last_fullsized":"2024-02-10T08:42:26.390880+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.909358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":406,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.407110+0000","last_change":"2024-02-10T08:41:24.469189+0000","last_active":"2024-02-10T08:42:35.407110+0000","last_peered":"2024-02-10T08:42:35.407110+0000","last_clean":"2024-02-10T08:42:35.407110+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:42:35.407110+0000","last_undegraded":"2024-02-10T08:42:35.407110+0000","last_fullsized":"2024-02-10T08:42:35.407110+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.469114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.469114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":401,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.402986+0000","last_change":"2024-02-10T08:41:03.331622+0000","last_active":"2024-02-10T08:42:35.402986+0000","last_peered":"2024-02-10T08:42:35.402986+0000","last_clean":"2024-02-10T08:42:35.402986+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:42:35.402986+0000","last_undegraded":"2024-02-10T08:42:35.402986+0000","last_fullsized":"2024-02-10T08:42:35.402986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":348,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162567+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:42:35.162567+0000","last_peered":"2024-02-10T08:42:35.162567+0000","last_clean":"2024-02-10T08:42:35.162567+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:42:35.162567+0000","last_undegraded":"2024-02-10T08:42:35.162567+0000","last_fullsized":"2024-02-10T08:42:35.162567+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":421,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162199+0000","last_change":"2024-02-10T08:41:03.770782+0000","last_active":"2024-02-10T08:42:35.162199+0000","last_peered":"2024-02-10T08:42:35.162199+0000","last_clean":"2024-02-10T08:42:35.162199+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:42:35.162199+0000","last_undegraded":"2024-02-10T08:42:35.162199+0000","last_fullsized":"2024-02-10T08:42:35.162199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":412,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388852+0000","last_change":"2024-02-10T08:41:52.364114+0000","last_active":"2024-02-10T08:42:26.388852+0000","last_peered":"2024-02-10T08:42:26.388852+0000","last_clean":"2024-02-10T08:42:26.388852+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:42:26.388852+0000","last_undegraded":"2024-02-10T08:42:26.388852+0000","last_fullsized":"2024-02-10T08:42:26.388852+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:52.364044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:52.364044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":398,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.406401+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:42:35.406401+0000","last_peered":"2024-02-10T08:42:35.406401+0000","last_clean":"2024-02-10T08:42:35.406401+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:42:35.406401+0000","last_undegraded":"2024-02-10T08:42:35.406401+0000","last_fullsized":"2024-02-10T08:42:35.406401+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":407,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160788+0000","last_change":"2024-02-10T08:39:40.832009+0000","last_active":"2024-02-10T08:42:35.160788+0000","last_peered":"2024-02-10T08:42:35.160788+0000","last_clean":"2024-02-10T08:42:35.160788+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:42:35.160788+0000","last_undegraded":"2024-02-10T08:42:35.160788+0000","last_fullsized":"2024-02-10T08:42:35.160788+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:40.831894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:40.831894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":325,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388865+0000","last_change":"2024-02-10T08:42:26.388122+0000","last_active":"2024-02-10T08:42:26.388865+0000","last_peered":"2024-02-10T08:42:26.388865+0000","last_clean":"2024-02-10T08:42:26.388865+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:42:26.388865+0000","last_undegraded":"2024-02-10T08:42:26.388865+0000","last_fullsized":"2024-02-10T08:42:26.388865+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.388002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.388002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":359,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.405394+0000","last_change":"2024-02-10T08:41:53.573192+0000","last_active":"2024-02-10T08:42:35.405394+0000","last_peered":"2024-02-10T08:42:35.405394+0000","last_clean":"2024-02-10T08:42:35.405394+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:42:35.405394+0000","last_undegraded":"2024-02-10T08:42:35.405394+0000","last_fullsized":"2024-02-10T08:42:35.405394+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:53.573125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.573125+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":415,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.389709+0000","last_change":"2024-02-10T08:40:27.813479+0000","last_active":"2024-02-10T08:42:26.389709+0000","last_peered":"2024-02-10T08:42:26.389709+0000","last_clean":"2024-02-10T08:42:26.389709+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:42:26.389709+0000","last_undegraded":"2024-02-10T08:42:26.389709+0000","last_fullsized":"2024-02-10T08:42:26.389709+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.813396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.813396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":402,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.404269+0000","last_change":"2024-02-10T08:42:23.529119+0000","last_active":"2024-02-10T08:42:35.404269+0000","last_peered":"2024-02-10T08:42:35.404269+0000","last_clean":"2024-02-10T08:42:35.404269+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:42:35.404269+0000","last_undegraded":"2024-02-10T08:42:35.404269+0000","last_fullsized":"2024-02-10T08:42:35.404269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:42:23.529013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.529013+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.141917+0000","last_change":"2024-02-10T08:41:48.214886+0000","last_active":"2024-02-10T08:42:35.141917+0000","last_peered":"2024-02-10T08:42:35.141917+0000","last_clean":"2024-02-10T08:42:35.141917+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:42:35.141917+0000","last_undegraded":"2024-02-10T08:42:35.141917+0000","last_fullsized":"2024-02-10T08:42:35.141917+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.214822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.214822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":429,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388911+0000","last_change":"2024-02-10T08:41:46.403355+0000","last_active":"2024-02-10T08:42:26.388911+0000","last_peered":"2024-02-10T08:42:26.388911+0000","last_clean":"2024-02-10T08:42:26.388911+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:42:26.388911+0000","last_undegraded":"2024-02-10T08:42:26.388911+0000","last_fullsized":"2024-02-10T08:42:26.388911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:46.403281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.403281+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":418,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.161515+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:42:35.161515+0000","last_peered":"2024-02-10T08:42:35.161515+0000","last_clean":"2024-02-10T08:42:35.161515+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:42:35.161515+0000","last_undegraded":"2024-02-10T08:42:35.161515+0000","last_fullsized":"2024-02-10T08:42:35.161515+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":404,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.146266+0000","last_change":"2024-02-10T08:41:49.644781+0000","last_active":"2024-02-10T08:42:35.146266+0000","last_peered":"2024-02-10T08:42:35.146266+0000","last_clean":"2024-02-10T08:42:35.146266+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:42:35.146266+0000","last_undegraded":"2024-02-10T08:42:35.146266+0000","last_fullsized":"2024-02-10T08:42:35.146266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145867+0000","last_change":"2024-02-10T08:41:23.662181+0000","last_active":"2024-02-10T08:42:35.145867+0000","last_peered":"2024-02-10T08:42:35.145867+0000","last_clean":"2024-02-10T08:42:35.145867+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:42:35.145867+0000","last_undegraded":"2024-02-10T08:42:35.145867+0000","last_fullsized":"2024-02-10T08:42:35.145867+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":440,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.142277+0000","last_change":"2024-02-10T08:41:45.260263+0000","last_active":"2024-02-10T08:42:35.142277+0000","last_peered":"2024-02-10T08:42:35.142277+0000","last_clean":"2024-02-10T08:42:35.142277+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:42:35.142277+0000","last_undegraded":"2024-02-10T08:42:35.142277+0000","last_fullsized":"2024-02-10T08:42:35.142277+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":412,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.387964+0000","last_change":"2024-02-10T08:42:23.333128+0000","last_active":"2024-02-10T08:42:26.387964+0000","last_peered":"2024-02-10T08:42:26.387964+0000","last_clean":"2024-02-10T08:42:26.387964+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:42:26.387964+0000","last_undegraded":"2024-02-10T08:42:26.387964+0000","last_fullsized":"2024-02-10T08:42:26.387964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":397,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033509+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:42:26.033509+0000","last_peered":"2024-02-10T08:42:26.033509+0000","last_clean":"2024-02-10T08:42:26.033509+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:42:26.033509+0000","last_undegraded":"2024-02-10T08:42:26.033509+0000","last_fullsized":"2024-02-10T08:42:26.033509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":326,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.406126+0000","last_change":"2024-02-10T08:41:56.265678+0000","last_active":"2024-02-10T08:42:35.406126+0000","last_peered":"2024-02-10T08:42:35.406126+0000","last_clean":"2024-02-10T08:42:35.406126+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:42:35.406126+0000","last_undegraded":"2024-02-10T08:42:35.406126+0000","last_fullsized":"2024-02-10T08:42:35.406126+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.265562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.265562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":451,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162135+0000","last_change":"2024-02-10T08:41:53.828692+0000","last_active":"2024-02-10T08:42:35.162135+0000","last_peered":"2024-02-10T08:42:35.162135+0000","last_clean":"2024-02-10T08:42:35.162135+0000","last_became_active":"2024-02-10T08:40:42.014594+0000","last_became_peered":"2024-02-10T08:40:42.014594+0000","last_unstale":"2024-02-10T08:42:35.162135+0000","last_undegraded":"2024-02-10T08:42:35.162135+0000","last_fullsized":"2024-02-10T08:42:35.162135+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":409,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160647+0000","last_change":"2024-02-10T08:41:57.957397+0000","last_active":"2024-02-10T08:42:35.160647+0000","last_peered":"2024-02-10T08:42:35.160647+0000","last_clean":"2024-02-10T08:42:35.160647+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:42:35.160647+0000","last_undegraded":"2024-02-10T08:42:35.160647+0000","last_fullsized":"2024-02-10T08:42:35.160647+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":398,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164180+0000","last_change":"2024-02-10T08:39:37.851437+0000","last_active":"2024-02-10T08:42:35.164180+0000","last_peered":"2024-02-10T08:42:35.164180+0000","last_clean":"2024-02-10T08:42:35.164180+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:42:35.164180+0000","last_undegraded":"2024-02-10T08:42:35.164180+0000","last_fullsized":"2024-02-10T08:42:35.164180+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:37.851370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:39:37.851370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":353,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.139709+0000","last_change":"2024-02-10T08:42:01.674000+0000","last_active":"2024-02-10T08:42:35.139709+0000","last_peered":"2024-02-10T08:42:35.139709+0000","last_clean":"2024-02-10T08:42:35.139709+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:42:35.139709+0000","last_undegraded":"2024-02-10T08:42:35.139709+0000","last_fullsized":"2024-02-10T08:42:35.139709+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:01.673932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.673932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":441,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.033046+0000","last_change":"2024-02-10T08:41:19.765286+0000","last_active":"2024-02-10T08:42:26.033046+0000","last_peered":"2024-02-10T08:42:26.033046+0000","last_clean":"2024-02-10T08:42:26.033046+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:42:26.033046+0000","last_undegraded":"2024-02-10T08:42:26.033046+0000","last_fullsized":"2024-02-10T08:42:26.033046+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":407,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.403409+0000","last_change":"2024-02-10T08:41:13.452076+0000","last_active":"2024-02-10T08:42:35.403409+0000","last_peered":"2024-02-10T08:42:35.403409+0000","last_clean":"2024-02-10T08:42:35.403409+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:42:35.403409+0000","last_undegraded":"2024-02-10T08:42:35.403409+0000","last_fullsized":"2024-02-10T08:42:35.403409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.451989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.451989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":323,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.140988+0000","last_change":"2024-02-10T08:41:22.620317+0000","last_active":"2024-02-10T08:42:35.140988+0000","last_peered":"2024-02-10T08:42:35.140988+0000","last_clean":"2024-02-10T08:42:35.140988+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:42:35.140988+0000","last_undegraded":"2024-02-10T08:42:35.140988+0000","last_fullsized":"2024-02-10T08:42:35.140988+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.620255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.620255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":357,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.145048+0000","last_change":"2024-02-10T08:40:22.359771+0000","last_active":"2024-02-10T08:42:35.145048+0000","last_peered":"2024-02-10T08:42:35.145048+0000","last_clean":"2024-02-10T08:42:35.145048+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:42:35.145048+0000","last_undegraded":"2024-02-10T08:42:35.145048+0000","last_fullsized":"2024-02-10T08:42:35.145048+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:22.359703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:40:22.359703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":418,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032966+0000","last_change":"2024-02-10T08:42:03.646444+0000","last_active":"2024-02-10T08:42:26.032966+0000","last_peered":"2024-02-10T08:42:26.032966+0000","last_clean":"2024-02-10T08:42:26.032966+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:42:26.032966+0000","last_undegraded":"2024-02-10T08:42:26.032966+0000","last_fullsized":"2024-02-10T08:42:26.032966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":416,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.403656+0000","last_change":"2024-02-10T08:41:11.401522+0000","last_active":"2024-02-10T08:42:35.403656+0000","last_peered":"2024-02-10T08:42:35.403656+0000","last_clean":"2024-02-10T08:42:35.403656+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:42:35.403656+0000","last_undegraded":"2024-02-10T08:42:35.403656+0000","last_fullsized":"2024-02-10T08:42:35.403656+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.401433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.401433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":400,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160383+0000","last_change":"2024-02-10T08:42:35.160383+0000","last_active":"2024-02-10T08:42:35.160383+0000","last_peered":"2024-02-10T08:42:35.160383+0000","last_clean":"2024-02-10T08:42:35.160383+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:42:35.160383+0000","last_undegraded":"2024-02-10T08:42:35.160383+0000","last_fullsized":"2024-02-10T08:42:35.160383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":328,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.390760+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:42:26.390760+0000","last_peered":"2024-02-10T08:42:26.390760+0000","last_clean":"2024-02-10T08:42:26.390760+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:42:26.390760+0000","last_undegraded":"2024-02-10T08:42:26.390760+0000","last_fullsized":"2024-02-10T08:42:26.390760+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":436,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032299+0000","last_change":"2024-02-10T08:40:37.664677+0000","last_active":"2024-02-10T08:42:26.032299+0000","last_peered":"2024-02-10T08:42:26.032299+0000","last_clean":"2024-02-10T08:42:26.032299+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:42:26.032299+0000","last_undegraded":"2024-02-10T08:42:26.032299+0000","last_fullsized":"2024-02-10T08:42:26.032299+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:40:37.664586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.664586+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":413,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.164250+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:42:35.164250+0000","last_peered":"2024-02-10T08:42:35.164250+0000","last_clean":"2024-02-10T08:42:35.164250+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:42:35.164250+0000","last_undegraded":"2024-02-10T08:42:35.164250+0000","last_fullsized":"2024-02-10T08:42:35.164250+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":402,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.032927+0000","last_change":"2024-02-10T08:41:55.669868+0000","last_active":"2024-02-10T08:42:26.032927+0000","last_peered":"2024-02-10T08:42:26.032927+0000","last_clean":"2024-02-10T08:42:26.032927+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:42:26.032927+0000","last_undegraded":"2024-02-10T08:42:26.032927+0000","last_fullsized":"2024-02-10T08:42:26.032927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":358,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.389944+0000","last_change":"2024-02-10T08:41:16.004246+0000","last_active":"2024-02-10T08:42:26.389944+0000","last_peered":"2024-02-10T08:42:26.389944+0000","last_clean":"2024-02-10T08:42:26.389944+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:42:26.389944+0000","last_undegraded":"2024-02-10T08:42:26.389944+0000","last_fullsized":"2024-02-10T08:42:26.389944+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":452,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.162257+0000","last_change":"2024-02-10T08:41:19.653285+0000","last_active":"2024-02-10T08:42:35.162257+0000","last_peered":"2024-02-10T08:42:35.162257+0000","last_clean":"2024-02-10T08:42:35.162257+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:42:35.162257+0000","last_undegraded":"2024-02-10T08:42:35.162257+0000","last_fullsized":"2024-02-10T08:42:35.162257+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:41:19.653219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.653219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":320,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.405488+0000","last_change":"2024-02-10T08:40:42.336046+0000","last_active":"2024-02-10T08:42:35.405488+0000","last_peered":"2024-02-10T08:42:35.405488+0000","last_clean":"2024-02-10T08:42:35.405488+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:42:35.405488+0000","last_undegraded":"2024-02-10T08:42:35.405488+0000","last_fullsized":"2024-02-10T08:42:35.405488+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.335958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.335958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":398,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-02-10T08:42:35.160539+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:42:35.160539+0000","last_peered":"2024-02-10T08:42:35.160539+0000","last_clean":"2024-02-10T08:42:35.160539+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:42:35.160539+0000","last_undegraded":"2024-02-10T08:42:35.160539+0000","last_fullsized":"2024-02-10T08:42:35.160539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":355,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-02-10T08:42:26.388561+0000","last_change":"2024-02-10T08:42:05.201384+0000","last_active":"2024-02-10T08:42:26.388561+0000","last_peered":"2024-02-10T08:42:26.388561+0000","last_clean":"2024-02-10T08:42:26.388561+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:42:26.388561+0000","last_undegraded":"2024-02-10T08:42:26.388561+0000","last_fullsized":"2024-02-10T08:42:26.388561+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001497,"num_objects":160,"num_object_clones":3,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":108,"num_read_kb":9,"num_write":601,"num_write_kb":4500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13926400,"data_stored":12004505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1891,"ondisk_log_size":1891,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":550,"num_read_kb":545,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299506,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308096,"kb_used_data":2680,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437056,"statfs":{"total":95995035648,"available":95679545344,"internally_reserved":0,"allocated":2744320,"data_stored":1053394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51900000000000002,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67900000000000005,"5min":3.8809999999999998,"15min":22.143000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.56599999999999995},"min":{"1min":0.33300000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71399999999999997,"5min":0.93100000000000005,"15min":8.7850000000000001},"last":0.83899999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":3.2389999999999999,"5min":3.8559999999999999,"15min":7.593},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":0.69499999999999995,"5min":1.0089999999999999,"15min":7.7759999999999998},"last":0.80900000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.497,"15min":0.55000000000000004},"min":{"1min":0.318,"5min":0.27700000000000002,"15min":0.27100000000000002},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":6.0300000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55800000000000005,"15min":0.65100000000000002},"min":{"1min":0.38,"5min":0.251,"15min":0.251},"max":{"1min":1.0469999999999999,"5min":3.8109999999999999,"15min":19.369},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.51800000000000002,"15min":0.57199999999999995},"min":{"1min":0.33500000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":6.9829999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.64000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.84299999999999997,"5min":3.903,"15min":22.102},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.62,"5min":0.79200000000000004,"15min":8.827},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.46100000000000002,"15min":0.48499999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.26200000000000001},"max":{"1min":1.903,"5min":1.903,"15min":13.646000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49399999999999999,"15min":0.52400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.81200000000000006,"5min":3.9300000000000002,"15min":19.326000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":0.66100000000000003,"5min":0.77200000000000002,"15min":1.6220000000000001},"last":0.99199999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49199999999999999,"15min":0.56399999999999995},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.627,"5min":0.89100000000000001,"15min":22.687000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.36499999999999999,"5min":0.30399999999999999,"15min":0.29699999999999999},"max":{"1min":0.66700000000000004,"5min":0.72599999999999998,"15min":1.7450000000000001},"last":0.52100000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561144,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307812,"kb_used_data":6492,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437340,"statfs":{"total":95995035648,"available":95679836160,"internally_reserved":0,"allocated":6647808,"data_stored":5053146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.19800000000000001},"max":{"1min":0.68200000000000005,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.255},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.64600000000000002,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.65000000000000002,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.57799999999999996,"15min":0.52600000000000002},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.56599999999999995,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.69499999999999995,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58999999999999997,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72299999999999998,"5min":4.798,"15min":4.798},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59699999999999998,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68999999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.78100000000000003,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":16.823,"15min":16.823},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.51800000000000002},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":0.67400000000000004,"5min":3.657,"15min":3.657},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45200000000000001,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":0.71099999999999997,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.48099999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71199999999999997,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.501},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46700000000000003,"15min":0.436},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.41799999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51200000000000001,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.33800000000000002,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.499},"min":{"1min":0.39700000000000002,"5min":0.32500000000000001,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.51200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304000,"kb_used_data":2680,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441152,"statfs":{"total":95995035648,"available":95683739648,"internally_reserved":0,"allocated":2744320,"data_stored":1062845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.48299999999999998,"15min":0.40699999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.36799999999999999,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.63,"15min":0.45500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.45600000000000002,"5min":16.166,"15min":16.166},"last":0.308}]},{"osd":2,"last update":"Sat Feb 10 08:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.60899999999999999},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.38100000000000001,"5min":0.79600000000000004,"15min":34.985999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.378,"5min":0.66800000000000004,"15min":21.614999999999998},"last":0.25700000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.73299999999999998,"15min":0.51700000000000002},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.39800000000000002,"5min":24.773,"15min":24.773},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.69599999999999995,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.499,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.32500000000000001,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63700000000000001,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.67500000000000004,"15min":0.60599999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.76800000000000002,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":7.8479999999999999,"15min":7.8479999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.65700000000000003,"15min":0.60699999999999998},"min":{"1min":0.41799999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":9.532,"15min":9.532},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.73399999999999999,"15min":0.65500000000000003},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":0.65500000000000003,"5min":0.88100000000000001,"15min":9.4640000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57499999999999996,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.39400000000000002},"max":{"1min":0.66500000000000004,"5min":1.02,"15min":7.9199999999999999},"last":0.504}]}]},{"osd":0,"up_from":0,"seq":34359738517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299892,"kb_used_data":2668,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445260,"statfs":{"total":95995035648,"available":95687946240,"internally_reserved":0,"allocated":2732032,"data_stored":1061563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.502,"15min":0.40899999999999997},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.187},"max":{"1min":0.42499999999999999,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.311,"5min":0.64700000000000002,"15min":0.46000000000000002},"min":{"1min":0.20599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.44800000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.29999999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.68500000000000005,"15min":0.47399999999999998},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.443,"5min":12.512,"15min":12.512},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.623,"15min":0.46899999999999997},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":0.46600000000000003,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.31900000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:41:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.48999999999999999,"15min":0.441},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47199999999999998,"5min":4.617,"15min":4.617},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.57099999999999995,"15min":0.48699999999999999},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.22600000000000001},"max":{"1min":0.503,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.68899999999999995,"15min":0.67500000000000004},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76600000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.751,"15min":0.65300000000000002},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":1.014,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.48399999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.84199999999999997,"15min":0.66000000000000003},"min":{"1min":0.40200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.80000000000000004,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.69699999999999995,"15min":0.622},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.753,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.503}]},{"osd":6,"last update":"Sat Feb 10 08:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.67000000000000004,"15min":0.66200000000000003},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84099999999999997,"5min":4.3090000000000002,"15min":9.266},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.71299999999999997,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81999999999999995,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:42:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60999999999999999,"15min":0.68200000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76800000000000002,"5min":2.9809999999999999,"15min":9.2940000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72099999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.52500000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182535,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303812,"kb_used_data":6588,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441340,"statfs":{"total":95995035648,"available":95683932160,"internally_reserved":0,"allocated":6746112,"data_stored":5070508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.67000000000000004,"15min":0.50800000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.52100000000000002,"5min":11.550000000000001,"15min":14.744},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.85299999999999998,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":0.504,"5min":11.808,"15min":11.808},"last":0.22800000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.747,"15min":0.56599999999999995},"min":{"1min":0.245,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":0.435,"5min":10.444000000000001,"15min":16.634},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.66900000000000004,"15min":0.56499999999999995},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.754,"5min":11.494999999999999,"15min":12.438000000000001},"last":0.245}]},{"osd":2,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.65300000000000002,"15min":0.58699999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.51000000000000001,"5min":11.516999999999999,"15min":16.536000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.85299999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.48399999999999999,"5min":17.722000000000001,"15min":23.294},"last":0.316}]},{"osd":4,"last update":"Sat Feb 10 08:42:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.67900000000000005},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":1.9370000000000001,"5min":2.589,"15min":12.371},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.79700000000000004,"15min":0.71899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.008,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57799999999999996,"15min":0.70699999999999996},"min":{"1min":0.373,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":3.6040000000000001,"15min":12.579000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.65600000000000003},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1970000000000001,"5min":4.9909999999999997,"15min":11.448},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.749,"15min":0.69799999999999995},"min":{"1min":0.41099999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77900000000000003,"5min":10.381,"15min":16.091000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":11.472,"15min":16.475000000000001},"last":0.42799999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55300000000000005,"15min":0.65900000000000003},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":14.773999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.69699999999999995,"15min":0.73599999999999999},"min":{"1min":0.436,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.1339999999999999,"5min":6.1289999999999996,"15min":12.606999999999999},"last":0.45400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986305,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308104,"kb_used_data":2688,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437048,"statfs":{"total":95995035648,"available":95679537152,"internally_reserved":0,"allocated":2752512,"data_stored":1061849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54100000000000004,"15min":0.63100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.61399999999999999,"5min":4.1449999999999996,"15min":19.940999999999999},"last":0.505},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53100000000000003,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.251,"15min":0.221},"max":{"1min":0.83599999999999997,"5min":3.8860000000000001,"15min":9.1760000000000002},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.54500000000000004,"15min":0.58399999999999996},"min":{"1min":0.29399999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.628,"5min":5.774,"15min":5.774},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.60299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.29399999999999998},"max":{"1min":0.88300000000000001,"5min":3.8620000000000001,"15min":8.6989999999999998},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.60399999999999998},"min":{"1min":0.30399999999999999,"5min":0.29399999999999998,"15min":0.251},"max":{"1min":0.80500000000000005,"5min":3.9289999999999998,"15min":6.2329999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.58099999999999996,"15min":0.60399999999999998},"min":{"1min":0.36599999999999999,"5min":0.28299999999999997,"15min":0.24299999999999999},"max":{"1min":0.70499999999999996,"5min":10.01,"15min":10.01},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.57899999999999996,"15min":0.65600000000000003},"min":{"1min":0.33000000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81399999999999995,"5min":8.3480000000000008,"15min":9.5909999999999993},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57399999999999995,"15min":0.71699999999999997},"min":{"1min":0.38900000000000001,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.66300000000000003,"5min":4.21,"15min":25.27},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Feb 10 08:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.752,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.621},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46999999999999997,"15min":0.42399999999999999},"min":{"1min":0.33000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.76400000000000001,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.78000000000000003,"5min":8.3119999999999994,"15min":13.342000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.51100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":13.753,"15min":13.753},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.627,"15min":0.51800000000000002},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":0.90400000000000003,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.67100000000000004,"15min":0.57899999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.76700000000000002,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.65800000000000003}]}]},{"osd":5,"up_from":37,"seq":158913790077,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311924,"kb_used_data":6508,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433228,"statfs":{"total":95995035648,"available":95675625472,"internally_reserved":0,"allocated":6664192,"data_stored":5052955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":1.968,"15min":0.995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.78200000000000003,"5min":107.551,"15min":107.551},"last":0.44},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.853,"15min":0.96099999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":0.77000000000000002,"5min":88.792000000000002,"15min":88.792000000000002},"last":0.42599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.7050000000000001,"15min":0.93799999999999994},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71399999999999997,"5min":107.205,"15min":107.205},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.3700000000000001,"15min":0.82799999999999996},"min":{"1min":0.33600000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.74299999999999999,"5min":72.272000000000006,"15min":72.272000000000006},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.8460000000000001,"15min":0.95399999999999996},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":54.174999999999997,"15min":54.174999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.9259999999999999,"15min":0.995},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":86.183000000000007,"15min":86.183000000000007},"last":0.36699999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.151,"15min":0.72299999999999998},"min":{"1min":0.374,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.008,"5min":44.395000000000003,"15min":44.395000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":2.5139999999999998,"15min":1.206},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":114.526,"15min":114.526},"last":0.41099999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:41:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":1.149,"15min":0.66200000000000003},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.72399999999999998,"5min":72.216999999999999,"15min":72.216999999999999},"last":0.307},{"interface":"front","average":{"1min":0.495,"5min":0.88500000000000001,"15min":0.57099999999999995},"min":{"1min":0.32800000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.68000000000000005,"5min":44.345999999999997,"15min":44.345999999999997},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.68600000000000005},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":72.241},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.46300000000000002,"15min":0.54800000000000004},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.434}]},{"osd":7,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.71799999999999997},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.86199999999999999,"15min":66.844999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.75},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72199999999999998,"5min":0.90700000000000003,"15min":72.257000000000005},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4000643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4000529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4000571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:42:41.323 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:41.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:41.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:39.752820+0000 osd.4 (osd.4) 206 : cluster [DBG] 60.15 scrub starts 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:39.756501+0000 osd.4 (osd.4) 207 : cluster [DBG] 60.15 scrub ok 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:39.907835+0000 mgr.y (mgr.14553) 397 : audit [DBG] from='client.53696 v1:172.21.15.107:0/994377385' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.049701+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:40.050978+0000 mon.b (mon.1) 596 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:40.055031+0000 mon.a (mon.0) 2532 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.128453+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.131446+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:40.157235+0000 mon.a (mon.0) 2533 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:41.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.369999+0000 osd.2 (osd.2) 161 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:42:41.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.371651+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:42:41.456 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:41.477 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:41.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:41.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:41.610 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:41.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:41.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:41.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:41.763 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:41.782 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:41.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:41.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:41.913 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:41.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:42.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:42.074 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.143045+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:42:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.143783+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:42:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.683532+0000 osd.1 (osd.1) 217 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:40.687010+0000 osd.1 (osd.1) 218 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.048693+0000 mon.a (mon.0) 2534 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.048854+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.067131+0000 mon.b (mon.1) 597 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.095582+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.095870+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.096351+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.096508+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.105488+0000 osd.5 (osd.5) 207 : cluster [DBG] 6.a scrub starts 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.108396+0000 osd.5 (osd.5) 208 : cluster [DBG] 6.a scrub ok 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.411590+0000 osd.2 (osd.2) 163 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.415068+0000 osd.2 (osd.2) 164 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.452527+0000 mgr.y (mgr.14553) 398 : cluster [DBG] pgmap v652: 218 pgs: 7 creating+activating, 22 creating+peering, 189 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:41.489996+0000 mgr.y (mgr.14553) 399 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.143045+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.143783+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.683532+0000 osd.1 (osd.1) 217 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:40.687010+0000 osd.1 (osd.1) 218 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.048693+0000 mon.a (mon.0) 2534 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.048854+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.067131+0000 mon.b (mon.1) 597 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.095582+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.095870+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.096351+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.096508+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.105488+0000 osd.5 (osd.5) 207 : cluster [DBG] 6.a scrub starts 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.108396+0000 osd.5 (osd.5) 208 : cluster [DBG] 6.a scrub ok 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.411590+0000 osd.2 (osd.2) 163 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.415068+0000 osd.2 (osd.2) 164 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:42:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.452527+0000 mgr.y (mgr.14553) 398 : cluster [DBG] pgmap v652: 218 pgs: 7 creating+activating, 22 creating+peering, 189 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:42.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:41.489996+0000 mgr.y (mgr.14553) 399 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.143045+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:42:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.143783+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.683532+0000 osd.1 (osd.1) 217 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:40.687010+0000 osd.1 (osd.1) 218 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.048693+0000 mon.a (mon.0) 2534 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.048854+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.067131+0000 mon.b (mon.1) 597 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.095582+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.095870+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.096351+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.096508+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-67"}]: dispatch 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.105488+0000 osd.5 (osd.5) 207 : cluster [DBG] 6.a scrub starts 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.108396+0000 osd.5 (osd.5) 208 : cluster [DBG] 6.a scrub ok 2024-02-10T08:42:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.411590+0000 osd.2 (osd.2) 163 : cluster [DBG] 60.1c scrub starts 2024-02-10T08:42:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.415068+0000 osd.2 (osd.2) 164 : cluster [DBG] 60.1c scrub ok 2024-02-10T08:42:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.452527+0000 mgr.y (mgr.14553) 398 : cluster [DBG] pgmap v652: 218 pgs: 7 creating+activating, 22 creating+peering, 189 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:42.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:41.489996+0000 mgr.y (mgr.14553) 399 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:42.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:42:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:42.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:42.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:42.856 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:42.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:43.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:43.120 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:43.139 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:43.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:43.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:43.271 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 -- 172.21.15.107:0/3535802896 >> v1:172.21.15.107:6790/0 conn(0x7fefc4103250 legacy=0x7fefc41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 -- 172.21.15.107:0/3535802896 shutdown_connections 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 -- 172.21.15.107:0/3535802896 >> 172.21.15.107:0/3535802896 conn(0x7fefc40f2510 msgr2=0x7fefc40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 -- 172.21.15.107:0/3535802896 shutdown_connections 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 -- 172.21.15.107:0/3535802896 wait complete. 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 librados: starting msgr at 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.286+0000 7fefcc68c700 1 librados: starting objecter 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 Processor -- start 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 -- start start 2024-02-10T08:42:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 librados: setting wanted keys 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 librados: calling monclient init 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefc4119520 con 0x7fefc40fc910 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefc411a700 con 0x7fefc40ff710 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcc68c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefc411b900 con 0x7fefc4103250 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcac29700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fefc4103250 0x7fefc4117cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52644/0 (socket says 172.21.15.107:52644) 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefcac29700 1 -- 172.21.15.107:0/345689696 learned_addr learned my addr 172.21.15.107:0/345689696 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefca428700 1 --1- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6789/0 conn(0x7fefc40fc910 0x7fefc4060a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60322/0 (socket says 172.21.15.107:60322) 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter ms_handle_connect 0x7fefc40fc910 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter ms_handle_connect 0x7fefc4103250 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter ms_handle_connect 0x7fefc40ff710 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801487019 0 0) 0x7fefc4119520 con 0x7fefc40fc910 2024-02-10T08:42:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb00033f0 con 0x7fefc40fc910 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111659475 0 0) 0x7fefc411a700 con 0x7fefc40ff710 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4119520 con 0x7fefc40ff710 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034528341 0 0) 0x7fefc411b900 con 0x7fefc4103250 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.287+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc411a700 con 0x7fefc4103250 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1313452520 0 0) 0x7fefb00033f0 con 0x7fefc40fc910 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc411b900 con 0x7fefc40fc910 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefc0001a60 con 0x7fefc40fc910 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222281383 0 0) 0x7fefc4119520 con 0x7fefc40ff710 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb00033f0 con 0x7fefc40ff710 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729840264 0 0) 0x7fefc411a700 con 0x7fefc4103250 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc4119520 con 0x7fefc4103250 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefb4001840 con 0x7fefc40ff710 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefbc0023d0 con 0x7fefc4103250 2024-02-10T08:42:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1532310170 0 0) 0x7fefc411b900 con 0x7fefc40fc910 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6790/0 conn(0x7fefc4103250 legacy=0x7fefc4117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.195:6789/0 conn(0x7fefc40ff710 legacy=0x7fefc4061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc411cb00 con 0x7fefc40fc910 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc411a8f0 con 0x7fefc40fc910 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefcc68c700 10 client.28119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc411ade0 con 0x7fefc40fc910 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefcc68c700 1 librados: init done 2024-02-10T08:42:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fefc00023d0 con 0x7fefc40fc910 2024-02-10T08:42:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:43.294 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:43.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.288+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefc0005a50 con 0x7fefc40fc910 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.289+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fefa8004f00 con 0x7fefc40fc910 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.289+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fefc00189c0 con 0x7fefc40fc910 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(444..444 src has 330..444) v4 ==== 8181+0+0 (unknown 3230531242 0 0) 0x7fefc0078cd0 con 0x7fefc40fc910 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 osd_map(444..444 src has 330..444) v4 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 3 client.28119.objecter handle_osd_map got epochs [444,444] > 0 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 3 client.28119.objecter handle_osd_map decoding full epoch 444 2024-02-10T08:42:43.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 20 client.28119.objecter dump_active .. 0 homeless 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v444) v1 ==== 54+0+16 (unknown 1580004524 0 2420000192) 0x7fefc007b070 con 0x7fefc40fc910 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefb0063e20 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefcc68c700 10 client.28119.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 1 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.290+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 1 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 0 1 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 4 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 10 client.28119.objecter _send_command 1 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8004950 con 0x7fefa8004530 2024-02-10T08:42:43.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa8004530 2024-02-10T08:42:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8004950 con 0x7fefa8004530 2024-02-10T08:42:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 1 = system:0 2024-02-10T08:42:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 4 2024-02-10T08:42:43.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.291+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 0 1 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 10 client.28119.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 2 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 2 2024-02-10T08:42:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa80051a0 osd=0 3 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 0 2 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 4 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 10 client.28119.objecter _send_command 2 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.307+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa8004d30 con 0x7fefa8004530 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa8004d30 con 0x7fefa8004530 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 4 2024-02-10T08:42:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 0 2 2024-02-10T08:42:43.310 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 10 client.28119.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 1 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 4 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 10 client.28119.objecter _send_command 3 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.309+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa800c000 con 0x7fefa8004150 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa8004150 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa800c000 con 0x7fefa8004150 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 3 = system:0 2024-02-10T08:42:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 4 2024-02-10T08:42:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.310+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 1 3 2024-02-10T08:42:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 10 client.28119.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 4 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 4 2024-02-10T08:42:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8003fe0 osd=1 3 2024-02-10T08:42:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 1 4 2024-02-10T08:42:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 4 2024-02-10T08:42:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 10 client.28119.objecter _send_command 4 2024-02-10T08:42:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.336+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa8001850 con 0x7fefa8004150 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa8001850 con 0x7fefa8004150 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 4 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 1 4 2024-02-10T08:42:43.339 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 10 client.28119.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 5 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 5 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 2 5 2024-02-10T08:42:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 4 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 10 client.28119.objecter _send_command 5 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.338+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8003e30 con 0x7fefa800d2f0 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.339+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa800d2f0 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.340+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8003e30 con 0x7fefa800d2f0 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.340+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.340+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 5 = system:0 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.340+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 4 2024-02-10T08:42:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.340+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 2 5 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 10 client.28119.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 6 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 6 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8003630 osd=2 3 2024-02-10T08:42:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 2 6 2024-02-10T08:42:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 4 2024-02-10T08:42:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 10 client.28119.objecter _send_command 6 2024-02-10T08:42:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.354+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa800e700 con 0x7fefa800d2f0 2024-02-10T08:42:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa800e700 con 0x7fefa800d2f0 2024-02-10T08:42:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 4 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 2 6 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 10 client.28119.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 7 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 7 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 3 7 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 4 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 10 client.28119.objecter _send_command 7 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.357+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8015a50 con 0x7fefa8011eb0 2024-02-10T08:42:43.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.358+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa8011eb0 2024-02-10T08:42:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.359+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8015a50 con 0x7fefa8011eb0 2024-02-10T08:42:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.359+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.359+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 7 = system:0 2024-02-10T08:42:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.359+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 4 2024-02-10T08:42:43.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.359+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 3 7 2024-02-10T08:42:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 10 client.28119.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 8 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 8 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa80033b0 osd=3 3 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 3 8 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 4 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 10 client.28119.objecter _send_command 8 2024-02-10T08:42:43.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.384+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa800e170 con 0x7fefa8011eb0 2024-02-10T08:42:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa800e170 con 0x7fefa8011eb0 2024-02-10T08:42:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 4 2024-02-10T08:42:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 3 8 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 10 client.28119.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 9 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 9 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 4 9 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 4 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 10 client.28119.objecter _send_command 9 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.386+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa800e470 con 0x7fefa800ce50 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa800ce50 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa800e470 con 0x7fefa800ce50 2024-02-10T08:42:43.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:43.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 9 = system:0 2024-02-10T08:42:43.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 4 2024-02-10T08:42:43.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.387+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 4 9 2024-02-10T08:42:43.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 10 client.28119.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 10 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 10 2024-02-10T08:42:43.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa800e1d0 osd=4 3 2024-02-10T08:42:43.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 4 10 2024-02-10T08:42:43.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 4 2024-02-10T08:42:43.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 10 client.28119.objecter _send_command 10 2024-02-10T08:42:43.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.403+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa800e470 con 0x7fefa800ce50 2024-02-10T08:42:43.406 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.405+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa800e470 con 0x7fefa800ce50 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.405+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.405+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.405+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 4 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.405+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 4 10 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 10 client.28119.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 11 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 11 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 5 11 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 4 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 10 client.28119.objecter _send_command 11 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8016d30 con 0x7fefa80167e0 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.406+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa80167e0 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.407+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8016d30 con 0x7fefa80167e0 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.407+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:43.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.407+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 11 = system:0 2024-02-10T08:42:43.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.407+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 4 2024-02-10T08:42:43.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.407+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 5 11 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.707140+0000 osd.4 (osd.4) 208 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:41.708914+0000 osd.4 (osd.4) 209 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:42.056746+0000 osd.5 (osd.5) 209 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:42.059699+0000 osd.5 (osd.5) 210 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:42.064714+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:42.386707+0000 osd.2 (osd.2) 165 : cluster [DBG] 60.b scrub starts 2024-02-10T08:42:43.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:42.390284+0000 osd.2 (osd.2) 166 : cluster [DBG] 60.b scrub ok 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.707140+0000 osd.4 (osd.4) 208 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:41.708914+0000 osd.4 (osd.4) 209 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:42.056746+0000 osd.5 (osd.5) 209 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:42.059699+0000 osd.5 (osd.5) 210 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:42.064714+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:42.386707+0000 osd.2 (osd.2) 165 : cluster [DBG] 60.b scrub starts 2024-02-10T08:42:43.422 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:42.390284+0000 osd.2 (osd.2) 166 : cluster [DBG] 60.b scrub ok 2024-02-10T08:42:43.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 10 client.28119.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 12 2024-02-10T08:42:43.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 12 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa800e470 osd=5 3 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 5 12 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 4 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 10 client.28119.objecter _send_command 12 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.422+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa8016d30 con 0x7fefa80167e0 2024-02-10T08:42:43.423 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa8016d30 con 0x7fefa80167e0 2024-02-10T08:42:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 4 2024-02-10T08:42:43.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 5 12 2024-02-10T08:42:43.425 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:43.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 10 client.28119.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 13 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 13 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 6 13 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 4 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 10 client.28119.objecter _send_command 13 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8022b60 con 0x7fefa801f050 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.424+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa801f050 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.425+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8022b60 con 0x7fefa801f050 2024-02-10T08:42:43.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.425+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.425+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 13 = system:0 2024-02-10T08:42:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.425+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 4 2024-02-10T08:42:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.425+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 6 13 2024-02-10T08:42:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 10 client.28119.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 14 2024-02-10T08:42:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 14 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8016d30 osd=6 3 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 6 14 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 4 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 10 client.28119.objecter _send_command 14 2024-02-10T08:42:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.441+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa8024310 con 0x7fefa801f050 2024-02-10T08:42:43.443 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa8024310 con 0x7fefa801f050 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 4 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 6 14 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 10 client.28119.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 15 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 15 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 7 15 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 4 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 10 client.28119.objecter _send_command 15 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8024ef0 con 0x7fefa8024990 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.443+0000 7fefbb7fe700 10 client.28119.objecter ms_handle_connect 0x7fefa8024990 2024-02-10T08:42:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.444+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefa8024ef0 con 0x7fefa8024990 2024-02-10T08:42:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.444+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.444+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 15 = system:0 2024-02-10T08:42:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.444+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 4 2024-02-10T08:42:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.444+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 7 15 2024-02-10T08:42:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 10 client.28119.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:42:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign -1 16 2024-02-10T08:42:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 20 client.28119.objecter _get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_remove -1 16 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 20 client.28119.objecter get_session s=0x7fefa8024310 osd=7 3 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 15 client.28119.objecter _session_command_op_assign 7 16 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 4 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 10 client.28119.objecter _send_command 16 2024-02-10T08:42:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.459+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fefa8024ef0 con 0x7fefa8024990 2024-02-10T08:42:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefbb7fe700 1 -- 172.21.15.107:0/345689696 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fefa8024ef0 con 0x7fefa8024990 2024-02-10T08:42:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefbb7fe700 10 client.28119.objecter ms_dispatch 0x7fefc4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:42:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefbb7fe700 10 client.28119.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefbb7fe700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 4 2024-02-10T08:42:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefbb7fe700 15 client.28119.objecter _session_command_op_remove 7 16 2024-02-10T08:42:43.462 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 librados: watch_flush enter 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 librados: watch_flush exit 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.0 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6801/3321596972 conn(0x7fefa8004530 legacy=0x7fefa8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80051a0 osd=0 1 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.1 2024-02-10T08:42:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6805/133516126 conn(0x7fefa8004150 legacy=0x7fefa800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003fe0 osd=1 1 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.2 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6809/3575881728 conn(0x7fefa800d2f0 legacy=0x7fefa800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8003630 osd=2 1 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.3 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6813/2871825691 conn(0x7fefa8011eb0 legacy=0x7fefa80142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa80033b0 osd=3 1 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.4 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.195:6800/2499671285 conn(0x7fefa800ce50 legacy=0x7fefa8015bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e1d0 osd=4 1 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.5 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.195:6804/4178321486 conn(0x7fefa80167e0 legacy=0x7fefa801d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa800e470 osd=5 1 2024-02-10T08:42:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.6 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.195:6808/1417479017 conn(0x7fefa801f050 legacy=0x7fefa8021440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcac29700 1 -- 172.21.15.107:0/345689696 reap_dead start 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8016d30 osd=6 1 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter close_session for osd.7 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.195:6812/1934176928 conn(0x7fefa8024990 legacy=0x7fefa8022b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcac29700 1 -- 172.21.15.107:0/345689696 reap_dead start 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter put_session s=0x7fefa8024310 osd=7 1 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 20 client.28119.objecter shutdown clearing up homeless session... 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 10 client.28119.objecter successfully canceled tick 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcac29700 1 -- 172.21.15.107:0/345689696 reap_dead start 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcac29700 1 -- 172.21.15.107:0/345689696 reap_dead start 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6800/2634842009 conn(0x7fefb0063e20 legacy=0x7fefb00662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> v1:172.21.15.107:6789/0 conn(0x7fefc40fc910 legacy=0x7fefc4060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 shutdown_connections 2024-02-10T08:42:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.461+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 >> 172.21.15.107:0/345689696 conn(0x7fefc40f2510 msgr2=0x7fefc41066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.462+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 shutdown_connections 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.462+0000 7fefcc68c700 1 -- 172.21.15.107:0/345689696 wait complete. 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:43.462+0000 7fefcc68c700 1 librados: shutdown 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.707140+0000 osd.4 (osd.4) 208 : cluster [DBG] 60.18 scrub starts 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:41.708914+0000 osd.4 (osd.4) 209 : cluster [DBG] 60.18 scrub ok 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:42.056746+0000 osd.5 (osd.5) 209 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:42.059699+0000 osd.5 (osd.5) 210 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:42.064714+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:42.386707+0000 osd.2 (osd.2) 165 : cluster [DBG] 60.b scrub starts 2024-02-10T08:42:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:42.390284+0000 osd.2 (osd.2) 166 : cluster [DBG] 60.b scrub ok 2024-02-10T08:42:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.069711+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-02-10T08:42:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:43.074743+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:43.075276+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.105521+0000 osd.5 (osd.5) 211 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.109116+0000 osd.5 (osd.5) 212 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.110537+0000 osd.3 (osd.3) 215 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.111306+0000 osd.3 (osd.3) 216 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:43.290711+0000 mon.a (mon.0) 2541 : audit [DBG] from='client.? v1:172.21.15.107:0/345689696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.396539+0000 osd.2 (osd.2) 167 : cluster [DBG] 16.e scrub starts 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.399700+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub ok 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.452901+0000 mgr.y (mgr.14553) 400 : cluster [DBG] pgmap v655: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.069711+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:43.074743+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:43.075276+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.105521+0000 osd.5 (osd.5) 211 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.109116+0000 osd.5 (osd.5) 212 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.110537+0000 osd.3 (osd.3) 215 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.111306+0000 osd.3 (osd.3) 216 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:43.290711+0000 mon.a (mon.0) 2541 : audit [DBG] from='client.? v1:172.21.15.107:0/345689696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.396539+0000 osd.2 (osd.2) 167 : cluster [DBG] 16.e scrub starts 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.399700+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub ok 2024-02-10T08:42:44.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.452901+0000 mgr.y (mgr.14553) 400 : cluster [DBG] pgmap v655: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:44.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:44.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:42:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.069711+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-02-10T08:42:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:43.074743+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:43.075276+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.105521+0000 osd.5 (osd.5) 211 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.109116+0000 osd.5 (osd.5) 212 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.110537+0000 osd.3 (osd.3) 215 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.111306+0000 osd.3 (osd.3) 216 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:42:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:43.290711+0000 mon.a (mon.0) 2541 : audit [DBG] from='client.? v1:172.21.15.107:0/345689696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.396539+0000 osd.2 (osd.2) 167 : cluster [DBG] 16.e scrub starts 2024-02-10T08:42:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.399700+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub ok 2024-02-10T08:42:44.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.452901+0000 mgr.y (mgr.14553) 400 : cluster [DBG] pgmap v655: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:44.873 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:44.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:44.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:45.009 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:45.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.505671+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.508623+0000 osd.6 (osd.6) 188 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.693784+0000 osd.1 (osd.1) 219 : cluster [DBG] 2.d scrub starts 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:43.696701+0000 osd.1 (osd.1) 220 : cluster [DBG] 2.d scrub ok 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:44.076231+0000 mon.a (mon.0) 2542 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.076330+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:44.091014+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:44.094677+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.120 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:44.094986+0000 mon.a (mon.0) 2544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.643851+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.e scrub starts 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.647459+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.e scrub ok 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.505671+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.508623+0000 osd.6 (osd.6) 188 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.693784+0000 osd.1 (osd.1) 219 : cluster [DBG] 2.d scrub starts 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:43.696701+0000 osd.1 (osd.1) 220 : cluster [DBG] 2.d scrub ok 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:44.076231+0000 mon.a (mon.0) 2542 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.076330+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:44.091014+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:44.094677+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:44.094986+0000 mon.a (mon.0) 2544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.643851+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.e scrub starts 2024-02-10T08:42:45.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.647459+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.e scrub ok 2024-02-10T08:42:45.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:45.162 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:45.181 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:45.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:45.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.505671+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.508623+0000 osd.6 (osd.6) 188 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.693784+0000 osd.1 (osd.1) 219 : cluster [DBG] 2.d scrub starts 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:43.696701+0000 osd.1 (osd.1) 220 : cluster [DBG] 2.d scrub ok 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:44.076231+0000 mon.a (mon.0) 2542 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.076330+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:44.091014+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:42:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:44.094677+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:44.094986+0000 mon.a (mon.0) 2544 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:42:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.643851+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.e scrub starts 2024-02-10T08:42:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.647459+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.e scrub ok 2024-02-10T08:42:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:45.436 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:45.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:45.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:45] "GET /metrics HTTP/1.1" 200 187764 "" "Prometheus/2.33.4" 2024-02-10T08:42:46.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.416887+0000 osd.7 (osd.7) 187 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.419903+0000 osd.7 (osd.7) 188 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.700643+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:44.704127+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:45.084594+0000 mon.a (mon.0) 2545 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.084675+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.344644+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.347681+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.453269+0000 mgr.y (mgr.14553) 401 : cluster [DBG] pgmap v658: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.618304+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:45.619188+0000 osd.4 (osd.4) 213 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:42:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.041101+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.416887+0000 osd.7 (osd.7) 187 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.419903+0000 osd.7 (osd.7) 188 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.700643+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:44.704127+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:45.084594+0000 mon.a (mon.0) 2545 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.084675+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.344644+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.347681+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.453269+0000 mgr.y (mgr.14553) 401 : cluster [DBG] pgmap v658: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.618304+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:45.619188+0000 osd.4 (osd.4) 213 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:42:46.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.041101+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-02-10T08:42:46.450 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:46.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.416887+0000 osd.7 (osd.7) 187 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:42:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.419903+0000 osd.7 (osd.7) 188 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:42:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.700643+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:44.704127+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:45.084594+0000 mon.a (mon.0) 2545 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.084675+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.344644+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.347681+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.453269+0000 mgr.y (mgr.14553) 401 : cluster [DBG] pgmap v658: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.618304+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:45.619188+0000 osd.4 (osd.4) 213 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:42:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.041101+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-02-10T08:42:46.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:46.585 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:46.607 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:46.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:46.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:46.739 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:46.758 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:46.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:46.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:46.891 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:46.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:47.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:47.040 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.325948+0000 osd.2 (osd.2) 171 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:42:47.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.327934+0000 osd.2 (osd.2) 172 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:42:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.325948+0000 osd.2 (osd.2) 171 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:42:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.327934+0000 osd.2 (osd.2) 172 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:42:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.325948+0000 osd.2 (osd.2) 171 : cluster [DBG] 60.0 scrub starts 2024-02-10T08:42:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.327934+0000 osd.2 (osd.2) 172 : cluster [DBG] 60.0 scrub ok 2024-02-10T08:42:47.768 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:42:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:42:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:42:47.769 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.797+0000 7f1aef898700 1 -- 172.21.15.107:0/1640349494 >> v1:172.21.15.107:6789/0 conn(0x7f1ae8104350 legacy=0x7f1ae81067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.797+0000 7f1aef898700 1 -- 172.21.15.107:0/1640349494 shutdown_connections 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.797+0000 7f1aef898700 1 -- 172.21.15.107:0/1640349494 >> 172.21.15.107:0/1640349494 conn(0x7f1ae80f2510 msgr2=0x7f1ae80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.797+0000 7f1aef898700 1 -- 172.21.15.107:0/1640349494 shutdown_connections 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.797+0000 7f1aef898700 1 -- 172.21.15.107:0/1640349494 wait complete. 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 librados: starting msgr at 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 librados: starting objecter 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 Processor -- start 2024-02-10T08:42:48.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 -- start start 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 librados: setting wanted keys 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 librados: calling monclient init 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1ae8117410 con 0x7f1ae80f8a20 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1ae81185f0 con 0x7f1ae8104350 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aef898700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1ae81197f0 con 0x7f1ae80f95d0 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aed634700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ae80f8a20 0x7f1ae80ff770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60336/0 (socket says 172.21.15.107:60336) 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1aed634700 1 -- 172.21.15.107:0/3038945093 learned_addr learned my addr 172.21.15.107:0/3038945093 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter ms_handle_connect 0x7f1ae80f95d0 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter ms_handle_connect 0x7f1ae80f8a20 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602115208 0 0) 0x7f1ae81197f0 con 0x7f1ae80f95d0 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003030 con 0x7f1ae80f95d0 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144641487 0 0) 0x7f1ae8117410 con 0x7f1ae80f8a20 2024-02-10T08:42:48.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae81197f0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter ms_handle_connect 0x7f1ae8104350 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.798+0000 7f1ade7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247264437 0 0) 0x7f1ae81185f0 con 0x7f1ae8104350 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae8117410 con 0x7f1ae8104350 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 530510207 0 0) 0x7f1ae81197f0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ae81185f0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1ad8001940 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894052018 0 0) 0x7f1ad4003030 con 0x7f1ae80f95d0 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ae81197f0 con 0x7f1ae80f95d0 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1ae4001920 con 0x7f1ae80f95d0 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 739340358 0 0) 0x7f1ae81185f0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 >> v1:172.21.15.107:6790/0 conn(0x7f1ae80f95d0 legacy=0x7f1ae80ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 >> v1:172.21.15.195:6789/0 conn(0x7f1ae8104350 legacy=0x7f1ae8115ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae811a9f0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae8117600 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1aef898700 10 client.28125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae8117a70 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1aef898700 1 librados: init done 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1ad8001cd0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.799+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1ad8005bc0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.800+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc004f20 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.800+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1ad8002270 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(447..447 src has 330..447) v4 ==== 8194+0+0 (unknown 2169499803 0 0) 0x7f1ad8078e70 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 10 client.28125.objecter ms_dispatch 0x7f1ae8001550 osd_map(447..447 src has 330..447) v4 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 3 client.28125.objecter handle_osd_map got epochs [447,447] > 0 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 3 client.28125.objecter handle_osd_map decoding full epoch 447 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 20 client.28125.objecter dump_active .. 0 homeless 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.801+0000 7f1ade7fc700 10 client.28125.objecter ms_handle_connect 0x7f1ad4063bc0 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.805+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1ad804c200 con 0x7f1ae80f8a20 2024-02-10T08:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:47.980+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f1acc0051d0 con 0x7f1ae80f8a20 2024-02-10T08:42:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:48.019 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:48.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:48.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:48.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.139+0000 7f1ade7fc700 1 -- 172.21.15.107:0/3038945093 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v448) v1 ==== 87+0+0 (unknown 3468548515 0 0) 0x7f1ad804a610 con 0x7f1ae80f8a20 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 10 librados: watch_flush enter 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 10 librados: watch_flush exit 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 20 client.28125.objecter shutdown clearing up homeless session... 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 10 client.28125.objecter successfully canceled tick 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1ad4063bc0 legacy=0x7f1ad4066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:48.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 >> v1:172.21.15.107:6789/0 conn(0x7f1ae80f8a20 legacy=0x7f1ae80ff770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 shutdown_connections 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 >> 172.21.15.107:0/3038945093 conn(0x7f1ae80f2510 msgr2=0x7f1ae81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 shutdown_connections 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.141+0000 7f1aef898700 1 -- 172.21.15.107:0/3038945093 wait complete. 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:48.142+0000 7f1aef898700 1 librados: shutdown 2024-02-10T08:42:48.143 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:42:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.218606+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:42:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.219430+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:42:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.339084+0000 osd.7 (osd.7) 189 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:42:48.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.341974+0000 osd.7 (osd.7) 190 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.709957+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:46.713408+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.454067+0000 mgr.y (mgr.14553) 402 : cluster [DBG] pgmap v660: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.606911+0000 osd.4 (osd.4) 214 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.608961+0000 osd.4 (osd.4) 215 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:47.981464+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.218606+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.219430+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.339084+0000 osd.7 (osd.7) 189 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.341974+0000 osd.7 (osd.7) 190 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.709957+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:46.713408+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:42:48.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.454067+0000 mgr.y (mgr.14553) 402 : cluster [DBG] pgmap v660: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:48.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.606911+0000 osd.4 (osd.4) 214 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:42:48.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.608961+0000 osd.4 (osd.4) 215 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:42:48.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:47.981464+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.218606+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.219430+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.339084+0000 osd.7 (osd.7) 189 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.341974+0000 osd.7 (osd.7) 190 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.709957+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:46.713408+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.454067+0000 mgr.y (mgr.14553) 402 : cluster [DBG] pgmap v660: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.606911+0000 osd.4 (osd.4) 214 : cluster [DBG] 60.1b scrub starts 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.608961+0000 osd.4 (osd.4) 215 : cluster [DBG] 60.1b scrub ok 2024-02-10T08:42:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:47.981464+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:42:48.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:42:48.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:42:49.109 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:49.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:49.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.239369+0000 osd.3 (osd.3) 219 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.240616+0000 osd.3 (osd.3) 220 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.294389+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.297338+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.455319+0000 osd.6 (osd.6) 189 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:47.456221+0000 osd.6 (osd.6) 190 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.129515+0000 mon.a (mon.0) 2549 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:48.139703+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.139757+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.250251+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.250467+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:49.238 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.540048+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.540176+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.239369+0000 osd.3 (osd.3) 219 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.240616+0000 osd.3 (osd.3) 220 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.294389+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.297338+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.455319+0000 osd.6 (osd.6) 189 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:47.456221+0000 osd.6 (osd.6) 190 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.129515+0000 mon.a (mon.0) 2549 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:48.139703+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:49.239 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.139757+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-02-10T08:42:49.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.250251+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.250467+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:49.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.540048+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.240 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.540176+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:49.240 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.239369+0000 osd.3 (osd.3) 219 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:42:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.240616+0000 osd.3 (osd.3) 220 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:42:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.294389+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:42:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.297338+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:42:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.455319+0000 osd.6 (osd.6) 189 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:47.456221+0000 osd.6 (osd.6) 190 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.129515+0000 mon.a (mon.0) 2549 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:48.139703+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? v1:172.21.15.107:0/3038945093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.139757+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.250251+0000 osd.2 (osd.2) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.250467+0000 osd.2 (osd.2) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.540048+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:49.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.540176+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.324214+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:42:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:48.325424+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:42:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:49.125770+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:49.125945+0000 osd.5 (osd.5) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:49.142284+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:49.454461+0000 mgr.y (mgr.14553) 403 : cluster [DBG] pgmap v663: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.324214+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:48.325424+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:49.125770+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:49.125945+0000 osd.5 (osd.5) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:49.142284+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-02-10T08:42:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:49.454461+0000 mgr.y (mgr.14553) 403 : cluster [DBG] pgmap v663: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:42:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.324214+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:42:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:48.325424+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:42:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:49.125770+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:42:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:49.125945+0000 osd.5 (osd.5) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:42:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:49.142284+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-02-10T08:42:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:49.454461+0000 mgr.y (mgr.14553) 403 : cluster [DBG] pgmap v663: 218 pgs: 218 active+clean; 3.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-02-10T08:42:50.890 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:50.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 -- 172.21.15.107:0/2041293695 >> v1:172.21.15.107:6790/0 conn(0x7ff63c103250 legacy=0x7ff63c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:50.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 -- 172.21.15.107:0/2041293695 shutdown_connections 2024-02-10T08:42:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 -- 172.21.15.107:0/2041293695 >> 172.21.15.107:0/2041293695 conn(0x7ff63c0f2510 msgr2=0x7ff63c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 -- 172.21.15.107:0/2041293695 shutdown_connections 2024-02-10T08:42:50.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 -- 172.21.15.107:0/2041293695 wait complete. 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 librados: starting msgr at 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.983+0000 7ff640991700 1 librados: starting objecter 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 Processor -- start 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 -- start start 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 librados: setting wanted keys 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 librados: calling monclient init 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff63c117420 con 0x7ff63c0fc910 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff63c1185e0 con 0x7ff63c103250 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff640991700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff63c1197e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff639d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff63c0ff710 0x7ff63c061460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42550/0 (socket says 172.21.15.107:42550) 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff639d9b700 1 -- 172.21.15.107:0/3873417695 learned_addr learned my addr 172.21.15.107:0/3873417695 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff6337fe700 10 client.?.objecter ms_handle_connect 0x7ff63c0ff710 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff6337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff6337fe700 10 client.?.objecter ms_handle_connect 0x7ff63c103250 2024-02-10T08:42:50.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff6337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.984+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662546589 0 0) 0x7ff63c1197e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280038e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842047001 0 0) 0x7ff63c1185e0 con 0x7ff63c103250 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c1197e0 con 0x7ff63c103250 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 807798348 0 0) 0x7ff6280038e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff63c1185e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff62c0019f0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364374362 0 0) 0x7ff63c1197e0 con 0x7ff63c103250 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6280038e0 con 0x7ff63c103250 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff634001c10 con 0x7ff63c103250 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 10 client.?.objecter ms_handle_connect 0x7ff63c0fc910 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2872705906 0 0) 0x7ff63c1185e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 >> v1:172.21.15.195:6789/0 conn(0x7ff63c103250 legacy=0x7ff63c115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 >> v1:172.21.15.107:6789/0 conn(0x7ff63c0fc910 legacy=0x7ff63c060d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:50.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.985+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c11a9e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c1187d0 con 0x7ff63c0ff710 2024-02-10T08:42:50.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff640991700 10 client.53714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c118c40 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff640991700 1 librados: init done 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff62c002510 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.986+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff62c005b50 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.987+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c004f20 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.987+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff62c0189e0 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(449..449 src has 330..449) v4 ==== 8194+0+0 (unknown 308346033 0 0) 0x7ff62c078dc0 con 0x7ff63c0ff710 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 10 client.53714.objecter ms_dispatch 0x7ff63c001550 osd_map(449..449 src has 330..449) v4 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 3 client.53714.objecter handle_osd_map got epochs [449,449] > 0 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 3 client.53714.objecter handle_osd_map decoding full epoch 449 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 20 client.53714.objecter dump_active .. 0 homeless 2024-02-10T08:42:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.988+0000 7ff6337fe700 10 client.53714.objecter ms_handle_connect 0x7ff628063af0 2024-02-10T08:42:50.991 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:50.992 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:50.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:50.993+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff62c04c130 con 0x7ff63c0ff710 2024-02-10T08:42:51.025 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:51.047 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:51.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:51.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:42:51.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.162+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff61c000b90 con 0x7ff628063af0 2024-02-10T08:42:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.177+0000 7ff6337fe700 1 -- 172.21.15.107:0/3873417695 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+553665 (unknown 2965378022 0 950453503) 0x7ff61c000b90 con 0x7ff628063af0 2024-02-10T08:42:51.184 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:51.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 10 librados: watch_flush enter 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 10 librados: watch_flush exit 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 20 client.53714.objecter shutdown clearing up homeless session... 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 10 client.53714.objecter successfully canceled tick 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff628063af0 legacy=0x7ff628065fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 >> v1:172.21.15.107:6790/0 conn(0x7ff63c0ff710 legacy=0x7ff63c061460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 shutdown_connections 2024-02-10T08:42:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 >> 172.21.15.107:0/3873417695 conn(0x7ff63c0f2510 msgr2=0x7ff63c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 shutdown_connections 2024-02-10T08:42:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.181+0000 7ff640991700 1 -- 172.21.15.107:0/3873417695 wait complete. 2024-02-10T08:42:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:51.182+0000 7ff640991700 1 librados: shutdown 2024-02-10T08:42:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:42:51.201 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:42:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:51.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:42:51.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:51.333 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:51.352 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:51.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:51.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:51.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:42:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:42:52.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:42:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:51.162903+0000 mgr.y (mgr.14553) 404 : audit [DBG] from='client.53714 v1:172.21.15.107:0/3873417695' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:51.455262+0000 mgr.y (mgr.14553) 405 : cluster [DBG] pgmap v664: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-10T08:42:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:51.490973+0000 mgr.y (mgr.14553) 406 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:51.162903+0000 mgr.y (mgr.14553) 404 : audit [DBG] from='client.53714 v1:172.21.15.107:0/3873417695' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:51.455262+0000 mgr.y (mgr.14553) 405 : cluster [DBG] pgmap v664: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-10T08:42:52.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:51.490973+0000 mgr.y (mgr.14553) 406 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:51.162903+0000 mgr.y (mgr.14553) 404 : audit [DBG] from='client.53714 v1:172.21.15.107:0/3873417695' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:42:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:51.455262+0000 mgr.y (mgr.14553) 405 : cluster [DBG] pgmap v664: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-10T08:42:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:51.490973+0000 mgr.y (mgr.14553) 406 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:42:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:52.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:52.544 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-02-10T08:42:49.454236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007345,"num_objects":365,"num_object_clones":4,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":1396,"num_read_kb":1045,"num_write":1053,"num_write_kb":4562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2414,"ondisk_log_size":2414,"up":654,"acting":654,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":615,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2148812,"kb_used_data":31380,"kb_used_omap":0,"kb_used_meta":2117376,"kb_avail":654067252,"statfs":{"total":671965249536,"available":669764866048,"internally_reserved":0,"allocated":32133120,"data_stored":20045832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2168193024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":15,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.372263"},"pg_stats":[{"pgid":"216.1f","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084714+0000","last_change":"2024-02-10T08:42:44.087594+0000","last_active":"2024-02-10T08:42:45.084714+0000","last_peered":"2024-02-10T08:42:45.084714+0000","last_clean":"2024-02-10T08:42:45.084714+0000","last_became_active":"2024-02-10T08:42:44.087169+0000","last_became_peered":"2024-02-10T08:42:44.087169+0000","last_unstale":"2024-02-10T08:42:45.084714+0000","last_undegraded":"2024-02-10T08:42:45.084714+0000","last_fullsized":"2024-02-10T08:42:45.084714+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.1e","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083835+0000","last_change":"2024-02-10T08:42:44.087371+0000","last_active":"2024-02-10T08:42:45.083835+0000","last_peered":"2024-02-10T08:42:45.083835+0000","last_clean":"2024-02-10T08:42:45.083835+0000","last_became_active":"2024-02-10T08:42:44.087103+0000","last_became_peered":"2024-02-10T08:42:44.087103+0000","last_unstale":"2024-02-10T08:42:45.083835+0000","last_undegraded":"2024-02-10T08:42:45.083835+0000","last_fullsized":"2024-02-10T08:42:45.083835+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.1d","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.092309+0000","last_change":"2024-02-10T08:42:44.091925+0000","last_active":"2024-02-10T08:42:44.092309+0000","last_peered":"2024-02-10T08:42:44.092309+0000","last_clean":"2024-02-10T08:42:44.092309+0000","last_became_active":"2024-02-10T08:42:44.091611+0000","last_became_peered":"2024-02-10T08:42:44.091611+0000","last_unstale":"2024-02-10T08:42:44.092309+0000","last_undegraded":"2024-02-10T08:42:44.092309+0000","last_fullsized":"2024-02-10T08:42:44.092309+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"216.1c","version":"446'2","reported_seq":12,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.102707+0000","last_change":"2024-02-10T08:42:44.089185+0000","last_active":"2024-02-10T08:42:45.102707+0000","last_peered":"2024-02-10T08:42:45.102707+0000","last_clean":"2024-02-10T08:42:45.102707+0000","last_became_active":"2024-02-10T08:42:44.088871+0000","last_became_peered":"2024-02-10T08:42:44.088871+0000","last_unstale":"2024-02-10T08:42:45.102707+0000","last_undegraded":"2024-02-10T08:42:45.102707+0000","last_fullsized":"2024-02-10T08:42:45.102707+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.1b","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097352+0000","last_change":"2024-02-10T08:42:44.091245+0000","last_active":"2024-02-10T08:42:45.097352+0000","last_peered":"2024-02-10T08:42:45.097352+0000","last_clean":"2024-02-10T08:42:45.097352+0000","last_became_active":"2024-02-10T08:42:44.090933+0000","last_became_peered":"2024-02-10T08:42:44.090933+0000","last_unstale":"2024-02-10T08:42:45.097352+0000","last_undegraded":"2024-02-10T08:42:45.097352+0000","last_fullsized":"2024-02-10T08:42:45.097352+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.1a","version":"446'1","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.100869+0000","last_change":"2024-02-10T08:42:44.089277+0000","last_active":"2024-02-10T08:42:45.100869+0000","last_peered":"2024-02-10T08:42:45.100869+0000","last_clean":"2024-02-10T08:42:45.100869+0000","last_became_active":"2024-02-10T08:42:44.088955+0000","last_became_peered":"2024-02-10T08:42:44.088955+0000","last_unstale":"2024-02-10T08:42:45.100869+0000","last_undegraded":"2024-02-10T08:42:45.100869+0000","last_fullsized":"2024-02-10T08:42:45.100869+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.19","version":"446'1","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.100759+0000","last_change":"2024-02-10T08:42:44.088329+0000","last_active":"2024-02-10T08:42:45.100759+0000","last_peered":"2024-02-10T08:42:45.100759+0000","last_clean":"2024-02-10T08:42:45.100759+0000","last_became_active":"2024-02-10T08:42:44.088016+0000","last_became_peered":"2024-02-10T08:42:44.088016+0000","last_unstale":"2024-02-10T08:42:45.100759+0000","last_undegraded":"2024-02-10T08:42:45.100759+0000","last_fullsized":"2024-02-10T08:42:45.100759+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.18","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083015+0000","last_change":"2024-02-10T08:42:44.089267+0000","last_active":"2024-02-10T08:42:45.083015+0000","last_peered":"2024-02-10T08:42:45.083015+0000","last_clean":"2024-02-10T08:42:45.083015+0000","last_became_active":"2024-02-10T08:42:44.088982+0000","last_became_peered":"2024-02-10T08:42:44.088982+0000","last_unstale":"2024-02-10T08:42:45.083015+0000","last_undegraded":"2024-02-10T08:42:45.083015+0000","last_fullsized":"2024-02-10T08:42:45.083015+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.17","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097443+0000","last_change":"2024-02-10T08:42:44.091527+0000","last_active":"2024-02-10T08:42:45.097443+0000","last_peered":"2024-02-10T08:42:45.097443+0000","last_clean":"2024-02-10T08:42:45.097443+0000","last_became_active":"2024-02-10T08:42:44.091136+0000","last_became_peered":"2024-02-10T08:42:44.091136+0000","last_unstale":"2024-02-10T08:42:45.097443+0000","last_undegraded":"2024-02-10T08:42:45.097443+0000","last_fullsized":"2024-02-10T08:42:45.097443+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.16","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.095232+0000","last_change":"2024-02-10T08:42:44.092175+0000","last_active":"2024-02-10T08:42:44.095232+0000","last_peered":"2024-02-10T08:42:44.095232+0000","last_clean":"2024-02-10T08:42:44.095232+0000","last_became_active":"2024-02-10T08:42:44.091436+0000","last_became_peered":"2024-02-10T08:42:44.091436+0000","last_unstale":"2024-02-10T08:42:44.095232+0000","last_undegraded":"2024-02-10T08:42:44.095232+0000","last_fullsized":"2024-02-10T08:42:44.095232+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"216.15","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097019+0000","last_change":"2024-02-10T08:42:44.089657+0000","last_active":"2024-02-10T08:42:45.097019+0000","last_peered":"2024-02-10T08:42:45.097019+0000","last_clean":"2024-02-10T08:42:45.097019+0000","last_became_active":"2024-02-10T08:42:44.089370+0000","last_became_peered":"2024-02-10T08:42:44.089370+0000","last_unstale":"2024-02-10T08:42:45.097019+0000","last_undegraded":"2024-02-10T08:42:45.097019+0000","last_fullsized":"2024-02-10T08:42:45.097019+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.14","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.093846+0000","last_change":"2024-02-10T08:42:44.089148+0000","last_active":"2024-02-10T08:42:44.093846+0000","last_peered":"2024-02-10T08:42:44.093846+0000","last_clean":"2024-02-10T08:42:44.093846+0000","last_became_active":"2024-02-10T08:42:44.088826+0000","last_became_peered":"2024-02-10T08:42:44.088826+0000","last_unstale":"2024-02-10T08:42:44.093846+0000","last_undegraded":"2024-02-10T08:42:44.093846+0000","last_fullsized":"2024-02-10T08:42:44.093846+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.13","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097456+0000","last_change":"2024-02-10T08:42:44.087057+0000","last_active":"2024-02-10T08:42:45.097456+0000","last_peered":"2024-02-10T08:42:45.097456+0000","last_clean":"2024-02-10T08:42:45.097456+0000","last_became_active":"2024-02-10T08:42:44.086772+0000","last_became_peered":"2024-02-10T08:42:44.086772+0000","last_unstale":"2024-02-10T08:42:45.097456+0000","last_undegraded":"2024-02-10T08:42:45.097456+0000","last_fullsized":"2024-02-10T08:42:45.097456+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.12","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.094252+0000","last_change":"2024-02-10T08:42:44.093532+0000","last_active":"2024-02-10T08:42:44.094252+0000","last_peered":"2024-02-10T08:42:44.094252+0000","last_clean":"2024-02-10T08:42:44.094252+0000","last_became_active":"2024-02-10T08:42:44.093196+0000","last_became_peered":"2024-02-10T08:42:44.093196+0000","last_unstale":"2024-02-10T08:42:44.094252+0000","last_undegraded":"2024-02-10T08:42:44.094252+0000","last_fullsized":"2024-02-10T08:42:44.094252+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.11","version":"446'2","reported_seq":12,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096126+0000","last_change":"2024-02-10T08:42:44.093521+0000","last_active":"2024-02-10T08:42:45.096126+0000","last_peered":"2024-02-10T08:42:45.096126+0000","last_clean":"2024-02-10T08:42:45.096126+0000","last_became_active":"2024-02-10T08:42:44.093257+0000","last_became_peered":"2024-02-10T08:42:44.093257+0000","last_unstale":"2024-02-10T08:42:45.096126+0000","last_undegraded":"2024-02-10T08:42:45.096126+0000","last_fullsized":"2024-02-10T08:42:45.096126+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"216.10","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096504+0000","last_change":"2024-02-10T08:42:44.089659+0000","last_active":"2024-02-10T08:42:45.096504+0000","last_peered":"2024-02-10T08:42:45.096504+0000","last_clean":"2024-02-10T08:42:45.096504+0000","last_became_active":"2024-02-10T08:42:44.089360+0000","last_became_peered":"2024-02-10T08:42:44.089360+0000","last_unstale":"2024-02-10T08:42:45.096504+0000","last_undegraded":"2024-02-10T08:42:45.096504+0000","last_fullsized":"2024-02-10T08:42:45.096504+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.f","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097547+0000","last_change":"2024-02-10T08:42:44.091767+0000","last_active":"2024-02-10T08:42:45.097547+0000","last_peered":"2024-02-10T08:42:45.097547+0000","last_clean":"2024-02-10T08:42:45.097547+0000","last_became_active":"2024-02-10T08:42:44.091454+0000","last_became_peered":"2024-02-10T08:42:44.091454+0000","last_unstale":"2024-02-10T08:42:45.097547+0000","last_undegraded":"2024-02-10T08:42:45.097547+0000","last_fullsized":"2024-02-10T08:42:45.097547+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.e","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097350+0000","last_change":"2024-02-10T08:42:44.094552+0000","last_active":"2024-02-10T08:42:45.097350+0000","last_peered":"2024-02-10T08:42:45.097350+0000","last_clean":"2024-02-10T08:42:45.097350+0000","last_became_active":"2024-02-10T08:42:44.094352+0000","last_became_peered":"2024-02-10T08:42:44.094352+0000","last_unstale":"2024-02-10T08:42:45.097350+0000","last_undegraded":"2024-02-10T08:42:45.097350+0000","last_fullsized":"2024-02-10T08:42:45.097350+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"216.d","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.092535+0000","last_change":"2024-02-10T08:42:44.092046+0000","last_active":"2024-02-10T08:42:44.092535+0000","last_peered":"2024-02-10T08:42:44.092535+0000","last_clean":"2024-02-10T08:42:44.092535+0000","last_became_active":"2024-02-10T08:42:44.091737+0000","last_became_peered":"2024-02-10T08:42:44.091737+0000","last_unstale":"2024-02-10T08:42:44.092535+0000","last_undegraded":"2024-02-10T08:42:44.092535+0000","last_fullsized":"2024-02-10T08:42:44.092535+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"216.c","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085547+0000","last_change":"2024-02-10T08:42:44.091545+0000","last_active":"2024-02-10T08:42:45.085547+0000","last_peered":"2024-02-10T08:42:45.085547+0000","last_clean":"2024-02-10T08:42:45.085547+0000","last_became_active":"2024-02-10T08:42:44.091259+0000","last_became_peered":"2024-02-10T08:42:44.091259+0000","last_unstale":"2024-02-10T08:42:45.085547+0000","last_undegraded":"2024-02-10T08:42:45.085547+0000","last_fullsized":"2024-02-10T08:42:45.085547+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"216.b","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.095252+0000","last_change":"2024-02-10T08:42:44.091522+0000","last_active":"2024-02-10T08:42:44.095252+0000","last_peered":"2024-02-10T08:42:44.095252+0000","last_clean":"2024-02-10T08:42:44.095252+0000","last_became_active":"2024-02-10T08:42:44.091217+0000","last_became_peered":"2024-02-10T08:42:44.091217+0000","last_unstale":"2024-02-10T08:42:44.095252+0000","last_undegraded":"2024-02-10T08:42:44.095252+0000","last_fullsized":"2024-02-10T08:42:44.095252+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"216.a","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.093939+0000","last_change":"2024-02-10T08:42:44.093298+0000","last_active":"2024-02-10T08:42:44.093939+0000","last_peered":"2024-02-10T08:42:44.093939+0000","last_clean":"2024-02-10T08:42:44.093939+0000","last_became_active":"2024-02-10T08:42:44.092769+0000","last_became_peered":"2024-02-10T08:42:44.092769+0000","last_unstale":"2024-02-10T08:42:44.093939+0000","last_undegraded":"2024-02-10T08:42:44.093939+0000","last_fullsized":"2024-02-10T08:42:44.093939+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"216.9","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084373+0000","last_change":"2024-02-10T08:42:44.090724+0000","last_active":"2024-02-10T08:42:45.084373+0000","last_peered":"2024-02-10T08:42:45.084373+0000","last_clean":"2024-02-10T08:42:45.084373+0000","last_became_active":"2024-02-10T08:42:44.090408+0000","last_became_peered":"2024-02-10T08:42:44.090408+0000","last_unstale":"2024-02-10T08:42:45.084373+0000","last_undegraded":"2024-02-10T08:42:45.084373+0000","last_fullsized":"2024-02-10T08:42:45.084373+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"216.8","version":"0'0","reported_seq":10,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.095373+0000","last_change":"2024-02-10T08:42:44.092199+0000","last_active":"2024-02-10T08:42:44.095373+0000","last_peered":"2024-02-10T08:42:44.095373+0000","last_clean":"2024-02-10T08:42:44.095373+0000","last_became_active":"2024-02-10T08:42:44.091904+0000","last_became_peered":"2024-02-10T08:42:44.091904+0000","last_unstale":"2024-02-10T08:42:44.095373+0000","last_undegraded":"2024-02-10T08:42:44.095373+0000","last_fullsized":"2024-02-10T08:42:44.095373+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"216.7","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084797+0000","last_change":"2024-02-10T08:42:44.092577+0000","last_active":"2024-02-10T08:42:45.084797+0000","last_peered":"2024-02-10T08:42:45.084797+0000","last_clean":"2024-02-10T08:42:45.084797+0000","last_became_active":"2024-02-10T08:42:44.092219+0000","last_became_peered":"2024-02-10T08:42:44.092219+0000","last_unstale":"2024-02-10T08:42:45.084797+0000","last_undegraded":"2024-02-10T08:42:45.084797+0000","last_fullsized":"2024-02-10T08:42:45.084797+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.6","version":"446'1","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.079388+0000","last_change":"2024-02-10T08:42:44.092376+0000","last_active":"2024-02-10T08:42:45.079388+0000","last_peered":"2024-02-10T08:42:45.079388+0000","last_clean":"2024-02-10T08:42:45.079388+0000","last_became_active":"2024-02-10T08:42:44.091855+0000","last_became_peered":"2024-02-10T08:42:44.091855+0000","last_unstale":"2024-02-10T08:42:45.079388+0000","last_undegraded":"2024-02-10T08:42:45.079388+0000","last_fullsized":"2024-02-10T08:42:45.079388+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"216.5","version":"446'1","reported_seq":12,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.106198+0000","last_change":"2024-02-10T08:42:44.087405+0000","last_active":"2024-02-10T08:42:45.106198+0000","last_peered":"2024-02-10T08:42:45.106198+0000","last_clean":"2024-02-10T08:42:45.106198+0000","last_became_active":"2024-02-10T08:42:44.087129+0000","last_became_peered":"2024-02-10T08:42:44.087129+0000","last_unstale":"2024-02-10T08:42:45.106198+0000","last_undegraded":"2024-02-10T08:42:45.106198+0000","last_fullsized":"2024-02-10T08:42:45.106198+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.4","version":"446'1","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.078728+0000","last_change":"2024-02-10T08:42:44.094570+0000","last_active":"2024-02-10T08:42:45.078728+0000","last_peered":"2024-02-10T08:42:45.078728+0000","last_clean":"2024-02-10T08:42:45.078728+0000","last_became_active":"2024-02-10T08:42:44.094275+0000","last_became_peered":"2024-02-10T08:42:44.094275+0000","last_unstale":"2024-02-10T08:42:45.078728+0000","last_undegraded":"2024-02-10T08:42:45.078728+0000","last_fullsized":"2024-02-10T08:42:45.078728+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"216.3","version":"446'2","reported_seq":12,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.099208+0000","last_change":"2024-02-10T08:42:44.088176+0000","last_active":"2024-02-10T08:42:45.099208+0000","last_peered":"2024-02-10T08:42:45.099208+0000","last_clean":"2024-02-10T08:42:45.099208+0000","last_became_active":"2024-02-10T08:42:44.087877+0000","last_became_peered":"2024-02-10T08:42:44.087877+0000","last_unstale":"2024-02-10T08:42:45.099208+0000","last_undegraded":"2024-02-10T08:42:45.099208+0000","last_fullsized":"2024-02-10T08:42:45.099208+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"216.2","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083370+0000","last_change":"2024-02-10T08:42:44.087256+0000","last_active":"2024-02-10T08:42:45.083370+0000","last_peered":"2024-02-10T08:42:45.083370+0000","last_clean":"2024-02-10T08:42:45.083370+0000","last_became_active":"2024-02-10T08:42:44.086969+0000","last_became_peered":"2024-02-10T08:42:44.086969+0000","last_unstale":"2024-02-10T08:42:45.083370+0000","last_undegraded":"2024-02-10T08:42:45.083370+0000","last_fullsized":"2024-02-10T08:42:45.083370+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.1","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084599+0000","last_change":"2024-02-10T08:42:44.087901+0000","last_active":"2024-02-10T08:42:45.084599+0000","last_peered":"2024-02-10T08:42:45.084599+0000","last_clean":"2024-02-10T08:42:45.084599+0000","last_became_active":"2024-02-10T08:42:44.087616+0000","last_became_peered":"2024-02-10T08:42:44.087616+0000","last_unstale":"2024-02-10T08:42:45.084599+0000","last_undegraded":"2024-02-10T08:42:45.084599+0000","last_fullsized":"2024-02-10T08:42:45.084599+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"216.0","version":"0'0","reported_seq":11,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096519+0000","last_change":"2024-02-10T08:42:44.091281+0000","last_active":"2024-02-10T08:42:45.096519+0000","last_peered":"2024-02-10T08:42:45.096519+0000","last_clean":"2024-02-10T08:42:45.096519+0000","last_became_active":"2024-02-10T08:42:44.090911+0000","last_became_peered":"2024-02-10T08:42:44.090911+0000","last_unstale":"2024-02-10T08:42:45.096519+0000","last_undegraded":"2024-02-10T08:42:45.096519+0000","last_fullsized":"2024-02-10T08:42:45.096519+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:43.059701+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.059701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.3","version":"206'34","reported_seq":439,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084480+0000","last_change":"2024-02-10T08:42:33.705136+0000","last_active":"2024-02-10T08:42:45.084480+0000","last_peered":"2024-02-10T08:42:45.084480+0000","last_clean":"2024-02-10T08:42:45.084480+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:42:45.084480+0000","last_undegraded":"2024-02-10T08:42:45.084480+0000","last_fullsized":"2024-02-10T08:42:45.084480+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:42:33.705063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:33.705063+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"338'29","reported_seq":443,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088289+0000","last_change":"2024-02-10T08:42:28.650952+0000","last_active":"2024-02-10T08:42:44.088289+0000","last_peered":"2024-02-10T08:42:44.088289+0000","last_clean":"2024-02-10T08:42:44.088289+0000","last_became_active":"2024-02-10T08:36:50.946752+0000","last_became_peered":"2024-02-10T08:36:50.946752+0000","last_unstale":"2024-02-10T08:42:44.088289+0000","last_undegraded":"2024-02-10T08:42:44.088289+0000","last_fullsized":"2024-02-10T08:42:44.088289+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'29","last_scrub_stamp":"2024-02-10T08:42:28.650876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.650876+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1","version":"401'42","reported_seq":443,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.086396+0000","last_change":"2024-02-10T08:42:31.735581+0000","last_active":"2024-02-10T08:42:44.086396+0000","last_peered":"2024-02-10T08:42:44.086396+0000","last_clean":"2024-02-10T08:42:44.086396+0000","last_became_active":"2024-02-10T08:36:51.012295+0000","last_became_peered":"2024-02-10T08:36:51.012295+0000","last_unstale":"2024-02-10T08:42:44.086396+0000","last_undegraded":"2024-02-10T08:42:44.086396+0000","last_fullsized":"2024-02-10T08:42:44.086396+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'42","last_scrub_stamp":"2024-02-10T08:42:31.735511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.735511+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.0","version":"371'108","reported_seq":514,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084704+0000","last_change":"2024-02-10T08:42:25.056269+0000","last_active":"2024-02-10T08:42:45.084704+0000","last_peered":"2024-02-10T08:42:45.084704+0000","last_clean":"2024-02-10T08:42:45.084704+0000","last_became_active":"2024-02-10T08:37:10.196395+0000","last_became_peered":"2024-02-10T08:37:10.196395+0000","last_unstale":"2024-02-10T08:42:45.084704+0000","last_undegraded":"2024-02-10T08:42:45.084704+0000","last_fullsized":"2024-02-10T08:42:45.084704+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'107","last_scrub_stamp":"2024-02-10T08:40:35.402562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.402562+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"371'35","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080186+0000","last_change":"2024-02-10T08:42:32.704233+0000","last_active":"2024-02-10T08:42:44.080186+0000","last_peered":"2024-02-10T08:42:44.080186+0000","last_clean":"2024-02-10T08:42:44.080186+0000","last_became_active":"2024-02-10T08:37:10.212764+0000","last_became_peered":"2024-02-10T08:37:10.212764+0000","last_unstale":"2024-02-10T08:42:44.080186+0000","last_undegraded":"2024-02-10T08:42:44.080186+0000","last_fullsized":"2024-02-10T08:42:44.080186+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:32.704166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.704166+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.6","version":"338'73","reported_seq":476,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086148+0000","last_change":"2024-02-10T08:42:31.200907+0000","last_active":"2024-02-10T08:42:45.086148+0000","last_peered":"2024-02-10T08:42:45.086148+0000","last_clean":"2024-02-10T08:42:45.086148+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:42:45.086148+0000","last_undegraded":"2024-02-10T08:42:45.086148+0000","last_fullsized":"2024-02-10T08:42:45.086148+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'73","last_scrub_stamp":"2024-02-10T08:42:31.200824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.200824+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"446'46","reported_seq":461,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.112701+0000","last_change":"2024-02-10T08:42:37.799793+0000","last_active":"2024-02-10T08:42:45.112701+0000","last_peered":"2024-02-10T08:42:45.112701+0000","last_clean":"2024-02-10T08:42:45.112701+0000","last_became_active":"2024-02-10T08:37:10.145733+0000","last_became_peered":"2024-02-10T08:37:10.145733+0000","last_unstale":"2024-02-10T08:42:45.112701+0000","last_undegraded":"2024-02-10T08:42:45.112701+0000","last_fullsized":"2024-02-10T08:42:45.112701+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:42:37.799725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.799725+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.4","version":"338'23","reported_seq":435,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083349+0000","last_change":"2024-02-10T08:42:38.786583+0000","last_active":"2024-02-10T08:42:44.083349+0000","last_peered":"2024-02-10T08:42:44.083349+0000","last_clean":"2024-02-10T08:42:44.083349+0000","last_became_active":"2024-02-10T08:36:50.946231+0000","last_became_peered":"2024-02-10T08:36:50.946231+0000","last_unstale":"2024-02-10T08:42:44.083349+0000","last_undegraded":"2024-02-10T08:42:44.083349+0000","last_fullsized":"2024-02-10T08:42:44.083349+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'23","last_scrub_stamp":"2024-02-10T08:42:38.786497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:38.786497+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.b","version":"395'105","reported_seq":507,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085531+0000","last_change":"2024-02-10T08:42:42.390360+0000","last_active":"2024-02-10T08:42:45.085531+0000","last_peered":"2024-02-10T08:42:45.085531+0000","last_clean":"2024-02-10T08:42:45.085531+0000","last_became_active":"2024-02-10T08:36:51.026807+0000","last_became_peered":"2024-02-10T08:36:51.026807+0000","last_unstale":"2024-02-10T08:42:45.085531+0000","last_undegraded":"2024-02-10T08:42:45.085531+0000","last_fullsized":"2024-02-10T08:42:45.085531+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'105","last_scrub_stamp":"2024-02-10T08:42:42.390288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.390288+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"442'28","reported_seq":442,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088195+0000","last_change":"2024-02-10T08:42:30.719134+0000","last_active":"2024-02-10T08:42:44.088195+0000","last_peered":"2024-02-10T08:42:44.088195+0000","last_clean":"2024-02-10T08:42:44.088195+0000","last_became_active":"2024-02-10T08:36:50.947144+0000","last_became_peered":"2024-02-10T08:36:50.947144+0000","last_unstale":"2024-02-10T08:42:44.088195+0000","last_undegraded":"2024-02-10T08:42:44.088195+0000","last_fullsized":"2024-02-10T08:42:44.088195+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'23","last_scrub_stamp":"2024-02-10T08:42:30.719056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.719056+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.9","version":"395'115","reported_seq":502,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084792+0000","last_change":"2024-02-10T08:42:27.509494+0000","last_active":"2024-02-10T08:42:45.084792+0000","last_peered":"2024-02-10T08:42:45.084792+0000","last_clean":"2024-02-10T08:42:45.084792+0000","last_became_active":"2024-02-10T08:37:10.153851+0000","last_became_peered":"2024-02-10T08:37:10.153851+0000","last_unstale":"2024-02-10T08:42:45.084792+0000","last_undegraded":"2024-02-10T08:42:45.084792+0000","last_fullsized":"2024-02-10T08:42:45.084792+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'115","last_scrub_stamp":"2024-02-10T08:42:27.509423+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.509423+0000","log_size":115,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":35,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"338'86","reported_seq":489,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080999+0000","last_change":"2024-02-10T08:42:34.734765+0000","last_active":"2024-02-10T08:42:44.080999+0000","last_peered":"2024-02-10T08:42:44.080999+0000","last_clean":"2024-02-10T08:42:44.080999+0000","last_became_active":"2024-02-10T08:36:50.944919+0000","last_became_peered":"2024-02-10T08:36:50.944919+0000","last_unstale":"2024-02-10T08:42:44.080999+0000","last_undegraded":"2024-02-10T08:42:44.080999+0000","last_fullsized":"2024-02-10T08:42:44.080999+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'86","last_scrub_stamp":"2024-02-10T08:42:34.734692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.734692+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.f","version":"405'96","reported_seq":489,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082359+0000","last_change":"2024-02-10T08:42:25.088908+0000","last_active":"2024-02-10T08:42:44.082359+0000","last_peered":"2024-02-10T08:42:44.082359+0000","last_clean":"2024-02-10T08:42:44.082359+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:42:44.082359+0000","last_undegraded":"2024-02-10T08:42:44.082359+0000","last_fullsized":"2024-02-10T08:42:44.082359+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:41:45.373410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.373410+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"250'56","reported_seq":437,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097505+0000","last_change":"2024-02-10T08:42:25.061573+0000","last_active":"2024-02-10T08:42:45.097505+0000","last_peered":"2024-02-10T08:42:45.097505+0000","last_clean":"2024-02-10T08:42:45.097505+0000","last_became_active":"2024-02-10T08:37:10.285018+0000","last_became_peered":"2024-02-10T08:37:10.285018+0000","last_unstale":"2024-02-10T08:42:45.097505+0000","last_undegraded":"2024-02-10T08:42:45.097505+0000","last_fullsized":"2024-02-10T08:42:45.097505+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:39:54.894092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:54.894092+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"423'78","reported_seq":477,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079513+0000","last_change":"2024-02-10T08:42:25.088786+0000","last_active":"2024-02-10T08:42:44.079513+0000","last_peered":"2024-02-10T08:42:44.079513+0000","last_clean":"2024-02-10T08:42:44.079513+0000","last_became_active":"2024-02-10T08:36:50.930592+0000","last_became_peered":"2024-02-10T08:36:50.930592+0000","last_unstale":"2024-02-10T08:42:44.079513+0000","last_undegraded":"2024-02-10T08:42:44.079513+0000","last_fullsized":"2024-02-10T08:42:44.079513+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'74","last_scrub_stamp":"2024-02-10T08:41:55.273156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.273156+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"338'70","reported_seq":470,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085790+0000","last_change":"2024-02-10T08:42:25.056562+0000","last_active":"2024-02-10T08:42:45.085790+0000","last_peered":"2024-02-10T08:42:45.085790+0000","last_clean":"2024-02-10T08:42:45.085790+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:42:45.085790+0000","last_undegraded":"2024-02-10T08:42:45.085790+0000","last_fullsized":"2024-02-10T08:42:45.085790+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:40:36.430069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:36.430069+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"338'56","reported_seq":458,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075335+0000","last_change":"2024-02-10T08:42:25.057952+0000","last_active":"2024-02-10T08:42:44.075335+0000","last_peered":"2024-02-10T08:42:44.075335+0000","last_clean":"2024-02-10T08:42:44.075335+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:42:44.075335+0000","last_undegraded":"2024-02-10T08:42:44.075335+0000","last_fullsized":"2024-02-10T08:42:44.075335+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'56","last_scrub_stamp":"2024-02-10T08:41:49.073509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.073509+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":3925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"214'102","reported_seq":504,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080254+0000","last_change":"2024-02-10T08:42:25.060202+0000","last_active":"2024-02-10T08:42:44.080254+0000","last_peered":"2024-02-10T08:42:44.080254+0000","last_clean":"2024-02-10T08:42:44.080254+0000","last_became_active":"2024-02-10T08:37:10.147216+0000","last_became_peered":"2024-02-10T08:37:10.147216+0000","last_unstale":"2024-02-10T08:42:44.080254+0000","last_undegraded":"2024-02-10T08:42:44.080254+0000","last_fullsized":"2024-02-10T08:42:44.080254+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"398'105","reported_seq":495,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075452+0000","last_change":"2024-02-10T08:42:30.099849+0000","last_active":"2024-02-10T08:42:44.075452+0000","last_peered":"2024-02-10T08:42:44.075452+0000","last_clean":"2024-02-10T08:42:44.075452+0000","last_became_active":"2024-02-10T08:36:50.986980+0000","last_became_peered":"2024-02-10T08:36:50.986980+0000","last_unstale":"2024-02-10T08:42:44.075452+0000","last_undegraded":"2024-02-10T08:42:44.075452+0000","last_fullsized":"2024-02-10T08:42:44.075452+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'105","last_scrub_stamp":"2024-02-10T08:42:30.099763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.099763+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"301'72","reported_seq":479,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.084585+0000","last_change":"2024-02-10T08:42:25.060207+0000","last_active":"2024-02-10T08:42:44.084585+0000","last_peered":"2024-02-10T08:42:44.084585+0000","last_clean":"2024-02-10T08:42:44.084585+0000","last_became_active":"2024-02-10T08:37:10.149594+0000","last_became_peered":"2024-02-10T08:37:10.149594+0000","last_unstale":"2024-02-10T08:42:44.084585+0000","last_undegraded":"2024-02-10T08:42:44.084585+0000","last_fullsized":"2024-02-10T08:42:44.084585+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:42:01.488098+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.488098+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"392'87","reported_seq":483,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083855+0000","last_change":"2024-02-10T08:42:25.062302+0000","last_active":"2024-02-10T08:42:44.083855+0000","last_peered":"2024-02-10T08:42:44.083855+0000","last_clean":"2024-02-10T08:42:44.083855+0000","last_became_active":"2024-02-10T08:37:10.213618+0000","last_became_peered":"2024-02-10T08:37:10.213618+0000","last_unstale":"2024-02-10T08:42:44.083855+0000","last_undegraded":"2024-02-10T08:42:44.083855+0000","last_fullsized":"2024-02-10T08:42:44.083855+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:41:57.565097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.565097+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"231'62","reported_seq":482,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097456+0000","last_change":"2024-02-10T08:42:25.061409+0000","last_active":"2024-02-10T08:42:45.097456+0000","last_peered":"2024-02-10T08:42:45.097456+0000","last_clean":"2024-02-10T08:42:45.097456+0000","last_became_active":"2024-02-10T08:37:10.313981+0000","last_became_peered":"2024-02-10T08:37:10.313981+0000","last_unstale":"2024-02-10T08:42:45.097456+0000","last_undegraded":"2024-02-10T08:42:45.097456+0000","last_fullsized":"2024-02-10T08:42:45.097456+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:39:43.711720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:39:43.711720+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"434'34","reported_seq":441,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080217+0000","last_change":"2024-02-10T08:42:39.756580+0000","last_active":"2024-02-10T08:42:44.080217+0000","last_peered":"2024-02-10T08:42:44.080217+0000","last_clean":"2024-02-10T08:42:44.080217+0000","last_became_active":"2024-02-10T08:36:50.946330+0000","last_became_peered":"2024-02-10T08:36:50.946330+0000","last_unstale":"2024-02-10T08:42:44.080217+0000","last_undegraded":"2024-02-10T08:42:44.080217+0000","last_fullsized":"2024-02-10T08:42:44.080217+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'34","last_scrub_stamp":"2024-02-10T08:42:39.756505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:39.756505+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.14","version":"295'36","reported_seq":435,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098020+0000","last_change":"2024-02-10T08:42:25.061224+0000","last_active":"2024-02-10T08:42:45.098020+0000","last_peered":"2024-02-10T08:42:45.098020+0000","last_clean":"2024-02-10T08:42:45.098020+0000","last_became_active":"2024-02-10T08:36:51.020524+0000","last_became_peered":"2024-02-10T08:36:51.020524+0000","last_unstale":"2024-02-10T08:42:45.098020+0000","last_undegraded":"2024-02-10T08:42:45.098020+0000","last_fullsized":"2024-02-10T08:42:45.098020+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:39:49.714835+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"338'69","reported_seq":448,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083544+0000","last_change":"2024-02-10T08:42:25.054206+0000","last_active":"2024-02-10T08:42:44.083544+0000","last_peered":"2024-02-10T08:42:44.083544+0000","last_clean":"2024-02-10T08:42:44.083544+0000","last_became_active":"2024-02-10T08:37:10.215949+0000","last_became_peered":"2024-02-10T08:37:10.215949+0000","last_unstale":"2024-02-10T08:42:44.083544+0000","last_undegraded":"2024-02-10T08:42:44.083544+0000","last_fullsized":"2024-02-10T08:42:44.083544+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-10T08:40:24.583223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.583223+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1a","version":"398'71","reported_seq":465,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083364+0000","last_change":"2024-02-10T08:42:36.726157+0000","last_active":"2024-02-10T08:42:45.083364+0000","last_peered":"2024-02-10T08:42:45.083364+0000","last_clean":"2024-02-10T08:42:45.083364+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:42:45.083364+0000","last_undegraded":"2024-02-10T08:42:45.083364+0000","last_fullsized":"2024-02-10T08:42:45.083364+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'71","last_scrub_stamp":"2024-02-10T08:42:36.726091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:36.726091+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":452,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084899+0000","last_change":"2024-02-10T08:42:35.744653+0000","last_active":"2024-02-10T08:42:45.084899+0000","last_peered":"2024-02-10T08:42:45.084899+0000","last_clean":"2024-02-10T08:42:45.084899+0000","last_became_active":"2024-02-10T08:37:10.153427+0000","last_became_peered":"2024-02-10T08:37:10.153427+0000","last_unstale":"2024-02-10T08:42:45.084899+0000","last_undegraded":"2024-02-10T08:42:45.084899+0000","last_fullsized":"2024-02-10T08:42:45.084899+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-10T08:42:35.744580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.744580+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"401'38","reported_seq":425,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081431+0000","last_change":"2024-02-10T08:42:41.708987+0000","last_active":"2024-02-10T08:42:44.081431+0000","last_peered":"2024-02-10T08:42:44.081431+0000","last_clean":"2024-02-10T08:42:44.081431+0000","last_became_active":"2024-02-10T08:37:10.218250+0000","last_became_peered":"2024-02-10T08:37:10.218250+0000","last_unstale":"2024-02-10T08:42:44.081431+0000","last_undegraded":"2024-02-10T08:42:44.081431+0000","last_fullsized":"2024-02-10T08:42:44.081431+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'38","last_scrub_stamp":"2024-02-10T08:42:41.708918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.708918+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"60.1f","version":"439'42","reported_seq":455,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076623+0000","last_change":"2024-02-10T08:42:25.058674+0000","last_active":"2024-02-10T08:42:44.076623+0000","last_peered":"2024-02-10T08:42:44.076623+0000","last_clean":"2024-02-10T08:42:44.076623+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:42:44.076623+0000","last_undegraded":"2024-02-10T08:42:44.076623+0000","last_fullsized":"2024-02-10T08:42:44.076623+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'36","last_scrub_stamp":"2024-02-10T08:41:46.085763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.085763+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"210'17","reported_seq":409,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084117+0000","last_change":"2024-02-10T08:42:25.056249+0000","last_active":"2024-02-10T08:42:45.084117+0000","last_peered":"2024-02-10T08:42:45.084117+0000","last_clean":"2024-02-10T08:42:45.084117+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:42:45.084117+0000","last_undegraded":"2024-02-10T08:42:45.084117+0000","last_fullsized":"2024-02-10T08:42:45.084117+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:42:04.642908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.642908+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"405'21","reported_seq":415,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097253+0000","last_change":"2024-02-10T08:42:25.060687+0000","last_active":"2024-02-10T08:42:45.097253+0000","last_peered":"2024-02-10T08:42:45.097253+0000","last_clean":"2024-02-10T08:42:45.097253+0000","last_became_active":"2024-02-10T08:37:10.249169+0000","last_became_peered":"2024-02-10T08:37:10.249169+0000","last_unstale":"2024-02-10T08:42:45.097253+0000","last_undegraded":"2024-02-10T08:42:45.097253+0000","last_fullsized":"2024-02-10T08:42:45.097253+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'17","last_scrub_stamp":"2024-02-10T08:40:30.882874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.882874+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"371'35","reported_seq":446,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085684+0000","last_change":"2024-02-10T08:42:41.415137+0000","last_active":"2024-02-10T08:42:45.085684+0000","last_peered":"2024-02-10T08:42:45.085684+0000","last_clean":"2024-02-10T08:42:45.085684+0000","last_became_active":"2024-02-10T08:36:51.024277+0000","last_became_peered":"2024-02-10T08:36:51.024277+0000","last_unstale":"2024-02-10T08:42:45.085684+0000","last_undegraded":"2024-02-10T08:42:45.085684+0000","last_fullsized":"2024-02-10T08:42:45.085684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:41.415071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.415071+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"124'10","reported_seq":552,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.085917+0000","last_change":"2024-02-10T08:41:09.353173+0000","last_active":"2024-02-10T08:42:44.085917+0000","last_peered":"2024-02-10T08:42:44.085917+0000","last_clean":"2024-02-10T08:42:44.085917+0000","last_became_active":"2024-02-10T08:36:58.128990+0000","last_became_peered":"2024-02-10T08:36:58.128990+0000","last_unstale":"2024-02-10T08:42:44.085917+0000","last_undegraded":"2024-02-10T08:42:44.085917+0000","last_fullsized":"2024-02-10T08:42:44.085917+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'10","last_scrub_stamp":"2024-02-10T08:41:09.353088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:09.353088+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'22","reported_seq":445,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075199+0000","last_change":"2024-02-10T08:42:25.091712+0000","last_active":"2024-02-10T08:42:44.075199+0000","last_peered":"2024-02-10T08:42:44.075199+0000","last_clean":"2024-02-10T08:42:44.075199+0000","last_became_active":"2024-02-10T08:33:20.256218+0000","last_became_peered":"2024-02-10T08:33:20.256218+0000","last_unstale":"2024-02-10T08:42:44.075199+0000","last_undegraded":"2024-02-10T08:42:44.075199+0000","last_fullsized":"2024-02-10T08:42:44.075199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:42:25.091640+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.091640+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":415,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086232+0000","last_change":"2024-02-10T08:40:42.232974+0000","last_active":"2024-02-10T08:42:45.086232+0000","last_peered":"2024-02-10T08:42:45.086232+0000","last_clean":"2024-02-10T08:42:45.086232+0000","last_became_active":"2024-02-10T08:33:18.107192+0000","last_became_peered":"2024-02-10T08:33:18.107192+0000","last_unstale":"2024-02-10T08:42:45.086232+0000","last_undegraded":"2024-02-10T08:42:45.086232+0000","last_fullsized":"2024-02-10T08:42:45.086232+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.232905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.232905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":423,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076757+0000","last_change":"2024-02-10T08:42:41.108507+0000","last_active":"2024-02-10T08:42:44.076757+0000","last_peered":"2024-02-10T08:42:44.076757+0000","last_clean":"2024-02-10T08:42:44.076757+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:42:44.076757+0000","last_undegraded":"2024-02-10T08:42:44.076757+0000","last_fullsized":"2024-02-10T08:42:44.076757+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:41.108418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.108418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085918+0000","last_change":"2024-02-10T08:41:59.348271+0000","last_active":"2024-02-10T08:42:45.085918+0000","last_peered":"2024-02-10T08:42:45.085918+0000","last_clean":"2024-02-10T08:42:45.085918+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:42:45.085918+0000","last_undegraded":"2024-02-10T08:42:45.085918+0000","last_fullsized":"2024-02-10T08:42:45.085918+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.348211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.348211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'9","reported_seq":392,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085753+0000","last_change":"2024-02-10T08:41:16.350785+0000","last_active":"2024-02-10T08:42:45.085753+0000","last_peered":"2024-02-10T08:42:45.085753+0000","last_clean":"2024-02-10T08:42:45.085753+0000","last_became_active":"2024-02-10T08:37:10.231211+0000","last_became_peered":"2024-02-10T08:37:10.231211+0000","last_unstale":"2024-02-10T08:42:45.085753+0000","last_undegraded":"2024-02-10T08:42:45.085753+0000","last_fullsized":"2024-02-10T08:42:45.085753+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":425,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085218+0000","last_change":"2024-02-10T08:42:43.696768+0000","last_active":"2024-02-10T08:42:45.085218+0000","last_peered":"2024-02-10T08:42:45.085218+0000","last_clean":"2024-02-10T08:42:45.085218+0000","last_became_active":"2024-02-10T08:33:18.113593+0000","last_became_peered":"2024-02-10T08:33:18.113593+0000","last_unstale":"2024-02-10T08:42:45.085218+0000","last_undegraded":"2024-02-10T08:42:45.085218+0000","last_fullsized":"2024-02-10T08:42:45.085218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.696704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.696704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":434,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084199+0000","last_change":"2024-02-10T08:41:43.195685+0000","last_active":"2024-02-10T08:42:45.084199+0000","last_peered":"2024-02-10T08:42:45.084199+0000","last_clean":"2024-02-10T08:42:45.084199+0000","last_became_active":"2024-02-10T08:36:55.097028+0000","last_became_peered":"2024-02-10T08:36:55.097028+0000","last_unstale":"2024-02-10T08:42:45.084199+0000","last_undegraded":"2024-02-10T08:42:45.084199+0000","last_fullsized":"2024-02-10T08:42:45.084199+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":328,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.085634+0000","last_change":"2024-02-10T08:40:39.350744+0000","last_active":"2024-02-10T08:42:44.085634+0000","last_peered":"2024-02-10T08:42:44.085634+0000","last_clean":"2024-02-10T08:42:44.085634+0000","last_became_active":"2024-02-10T08:37:10.214691+0000","last_became_peered":"2024-02-10T08:37:10.214691+0000","last_unstale":"2024-02-10T08:42:44.085634+0000","last_undegraded":"2024-02-10T08:42:44.085634+0000","last_fullsized":"2024-02-10T08:42:44.085634+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.350663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.350663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":358,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.077065+0000","last_change":"2024-02-10T08:41:45.092740+0000","last_active":"2024-02-10T08:42:44.077065+0000","last_peered":"2024-02-10T08:42:44.077065+0000","last_clean":"2024-02-10T08:42:44.077065+0000","last_became_active":"2024-02-10T08:36:37.032974+0000","last_became_peered":"2024-02-10T08:36:37.032974+0000","last_unstale":"2024-02-10T08:42:44.077065+0000","last_undegraded":"2024-02-10T08:42:44.077065+0000","last_fullsized":"2024-02-10T08:42:44.077065+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:45.092620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.092620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"63'19","reported_seq":450,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079301+0000","last_change":"2024-02-10T08:41:57.284919+0000","last_active":"2024-02-10T08:42:44.079301+0000","last_peered":"2024-02-10T08:42:44.079301+0000","last_clean":"2024-02-10T08:42:44.079301+0000","last_became_active":"2024-02-10T08:33:20.257503+0000","last_became_peered":"2024-02-10T08:33:20.257503+0000","last_unstale":"2024-02-10T08:42:44.079301+0000","last_undegraded":"2024-02-10T08:42:44.079301+0000","last_fullsized":"2024-02-10T08:42:44.079301+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:57.284811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.284811+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":424,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086204+0000","last_change":"2024-02-10T08:40:40.314002+0000","last_active":"2024-02-10T08:42:45.086204+0000","last_peered":"2024-02-10T08:42:45.086204+0000","last_clean":"2024-02-10T08:42:45.086204+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:42:45.086204+0000","last_undegraded":"2024-02-10T08:42:45.086204+0000","last_fullsized":"2024-02-10T08:42:45.086204+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.313935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":438,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085669+0000","last_change":"2024-02-10T08:40:26.423248+0000","last_active":"2024-02-10T08:42:45.085669+0000","last_peered":"2024-02-10T08:42:45.085669+0000","last_clean":"2024-02-10T08:42:45.085669+0000","last_became_active":"2024-02-10T08:37:10.181703+0000","last_became_peered":"2024-02-10T08:37:10.181703+0000","last_unstale":"2024-02-10T08:42:45.085669+0000","last_undegraded":"2024-02-10T08:42:45.085669+0000","last_fullsized":"2024-02-10T08:42:45.085669+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:26.423190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:26.423190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":411,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.420010+0000","last_change":"2024-02-10T08:42:44.419981+0000","last_active":"2024-02-10T08:42:44.420010+0000","last_peered":"2024-02-10T08:42:44.420010+0000","last_clean":"2024-02-10T08:42:44.420010+0000","last_became_active":"2024-02-10T08:33:24.121422+0000","last_became_peered":"2024-02-10T08:33:24.121422+0000","last_unstale":"2024-02-10T08:42:44.420010+0000","last_undegraded":"2024-02-10T08:42:44.420010+0000","last_fullsized":"2024-02-10T08:42:44.420010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:44.419908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.419908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":358,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083386+0000","last_change":"2024-02-10T08:41:20.535962+0000","last_active":"2024-02-10T08:42:44.083386+0000","last_peered":"2024-02-10T08:42:44.083386+0000","last_clean":"2024-02-10T08:42:44.083386+0000","last_became_active":"2024-02-10T08:36:37.016798+0000","last_became_peered":"2024-02-10T08:36:37.016798+0000","last_unstale":"2024-02-10T08:42:44.083386+0000","last_undegraded":"2024-02-10T08:42:44.083386+0000","last_fullsized":"2024-02-10T08:42:44.083386+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"63'17","reported_seq":444,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080886+0000","last_change":"2024-02-10T08:41:13.773323+0000","last_active":"2024-02-10T08:42:44.080886+0000","last_peered":"2024-02-10T08:42:44.080886+0000","last_clean":"2024-02-10T08:42:44.080886+0000","last_became_active":"2024-02-10T08:33:20.116681+0000","last_became_peered":"2024-02-10T08:33:20.116681+0000","last_unstale":"2024-02-10T08:42:44.080886+0000","last_undegraded":"2024-02-10T08:42:44.080886+0000","last_fullsized":"2024-02-10T08:42:44.080886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-10T08:41:13.773224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.773224+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":254,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097498+0000","last_change":"2024-02-10T08:41:16.638540+0000","last_active":"2024-02-10T08:42:45.097498+0000","last_peered":"2024-02-10T08:42:45.097498+0000","last_clean":"2024-02-10T08:42:45.097498+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:42:45.097498+0000","last_undegraded":"2024-02-10T08:42:45.097498+0000","last_fullsized":"2024-02-10T08:42:45.097498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.638477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.638477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":443,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084098+0000","last_change":"2024-02-10T08:42:29.596160+0000","last_active":"2024-02-10T08:42:45.084098+0000","last_peered":"2024-02-10T08:42:45.084098+0000","last_clean":"2024-02-10T08:42:45.084098+0000","last_became_active":"2024-02-10T08:37:10.148683+0000","last_became_peered":"2024-02-10T08:37:10.148683+0000","last_unstale":"2024-02-10T08:42:45.084098+0000","last_undegraded":"2024-02-10T08:42:45.084098+0000","last_fullsized":"2024-02-10T08:42:45.084098+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":414,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084205+0000","last_change":"2024-02-10T08:42:32.250073+0000","last_active":"2024-02-10T08:42:45.084205+0000","last_peered":"2024-02-10T08:42:45.084205+0000","last_clean":"2024-02-10T08:42:45.084205+0000","last_became_active":"2024-02-10T08:33:24.125145+0000","last_became_peered":"2024-02-10T08:33:24.125145+0000","last_unstale":"2024-02-10T08:42:45.084205+0000","last_undegraded":"2024-02-10T08:42:45.084205+0000","last_fullsized":"2024-02-10T08:42:45.084205+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":361,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.086483+0000","last_change":"2024-02-10T08:42:25.595454+0000","last_active":"2024-02-10T08:42:44.086483+0000","last_peered":"2024-02-10T08:42:44.086483+0000","last_clean":"2024-02-10T08:42:44.086483+0000","last_became_active":"2024-02-10T08:36:37.071966+0000","last_became_peered":"2024-02-10T08:36:37.071966+0000","last_unstale":"2024-02-10T08:42:44.086483+0000","last_undegraded":"2024-02-10T08:42:44.086483+0000","last_fullsized":"2024-02-10T08:42:44.086483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"63'10","reported_seq":431,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083578+0000","last_change":"2024-02-10T08:41:18.755591+0000","last_active":"2024-02-10T08:42:44.083578+0000","last_peered":"2024-02-10T08:42:44.083578+0000","last_clean":"2024-02-10T08:42:44.083578+0000","last_became_active":"2024-02-10T08:33:20.115789+0000","last_became_peered":"2024-02-10T08:33:20.115789+0000","last_unstale":"2024-02-10T08:42:44.083578+0000","last_undegraded":"2024-02-10T08:42:44.083578+0000","last_fullsized":"2024-02-10T08:42:44.083578+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":420,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083736+0000","last_change":"2024-02-10T08:41:10.625950+0000","last_active":"2024-02-10T08:42:45.083736+0000","last_peered":"2024-02-10T08:42:45.083736+0000","last_clean":"2024-02-10T08:42:45.083736+0000","last_became_active":"2024-02-10T08:33:18.108710+0000","last_became_peered":"2024-02-10T08:33:18.108710+0000","last_unstale":"2024-02-10T08:42:45.083736+0000","last_undegraded":"2024-02-10T08:42:45.083736+0000","last_fullsized":"2024-02-10T08:42:45.083736+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.625888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.625888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"108'10","reported_seq":552,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085739+0000","last_change":"2024-02-10T08:40:41.276981+0000","last_active":"2024-02-10T08:42:45.085739+0000","last_peered":"2024-02-10T08:42:45.085739+0000","last_clean":"2024-02-10T08:42:45.085739+0000","last_became_active":"2024-02-10T08:36:42.178326+0000","last_became_peered":"2024-02-10T08:36:42.178326+0000","last_unstale":"2024-02-10T08:42:45.085739+0000","last_undegraded":"2024-02-10T08:42:45.085739+0000","last_fullsized":"2024-02-10T08:42:45.085739+0000","mapping_epoch":59,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'10","last_scrub_stamp":"2024-02-10T08:40:41.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.276901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":412,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081893+0000","last_change":"2024-02-10T08:41:43.683458+0000","last_active":"2024-02-10T08:42:44.081893+0000","last_peered":"2024-02-10T08:42:44.081893+0000","last_clean":"2024-02-10T08:42:44.081893+0000","last_became_active":"2024-02-10T08:33:24.121519+0000","last_became_peered":"2024-02-10T08:33:24.121519+0000","last_unstale":"2024-02-10T08:42:44.081893+0000","last_undegraded":"2024-02-10T08:42:44.081893+0000","last_fullsized":"2024-02-10T08:42:44.081893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":328,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096647+0000","last_change":"2024-02-10T08:39:53.850831+0000","last_active":"2024-02-10T08:42:45.096647+0000","last_peered":"2024-02-10T08:42:45.096647+0000","last_clean":"2024-02-10T08:42:45.096647+0000","last_became_active":"2024-02-10T08:37:10.250166+0000","last_became_peered":"2024-02-10T08:37:10.250166+0000","last_unstale":"2024-02-10T08:42:45.096647+0000","last_undegraded":"2024-02-10T08:42:45.096647+0000","last_fullsized":"2024-02-10T08:42:45.096647+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:53.850764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:53.850764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"63'15","reported_seq":443,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085031+0000","last_change":"2024-02-10T08:42:32.661333+0000","last_active":"2024-02-10T08:42:45.085031+0000","last_peered":"2024-02-10T08:42:45.085031+0000","last_clean":"2024-02-10T08:42:45.085031+0000","last_became_active":"2024-02-10T08:33:20.110483+0000","last_became_peered":"2024-02-10T08:33:20.110483+0000","last_unstale":"2024-02-10T08:42:45.085031+0000","last_undegraded":"2024-02-10T08:42:45.085031+0000","last_fullsized":"2024-02-10T08:42:45.085031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":428,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085178+0000","last_change":"2024-02-10T08:41:21.599108+0000","last_active":"2024-02-10T08:42:45.085178+0000","last_peered":"2024-02-10T08:42:45.085178+0000","last_clean":"2024-02-10T08:42:45.085178+0000","last_became_active":"2024-02-10T08:33:18.105803+0000","last_became_peered":"2024-02-10T08:33:18.105803+0000","last_unstale":"2024-02-10T08:42:45.085178+0000","last_undegraded":"2024-02-10T08:42:45.085178+0000","last_fullsized":"2024-02-10T08:42:45.085178+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:21.599034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.599034+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"142'13","reported_seq":580,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.647602+0000","last_change":"2024-02-10T08:42:44.647564+0000","last_active":"2024-02-10T08:42:44.647602+0000","last_peered":"2024-02-10T08:42:44.647602+0000","last_clean":"2024-02-10T08:42:44.647602+0000","last_became_active":"2024-02-10T08:37:13.019032+0000","last_became_peered":"2024-02-10T08:37:13.019032+0000","last_unstale":"2024-02-10T08:42:44.647602+0000","last_undegraded":"2024-02-10T08:42:44.647602+0000","last_fullsized":"2024-02-10T08:42:44.647602+0000","mapping_epoch":138,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:42:44.647464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.647464+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":418,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076281+0000","last_change":"2024-02-10T08:42:34.139662+0000","last_active":"2024-02-10T08:42:44.076281+0000","last_peered":"2024-02-10T08:42:44.076281+0000","last_clean":"2024-02-10T08:42:44.076281+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:42:44.076281+0000","last_undegraded":"2024-02-10T08:42:44.076281+0000","last_fullsized":"2024-02-10T08:42:44.076281+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.139528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.139528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":367,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075512+0000","last_change":"2024-02-10T08:41:54.063932+0000","last_active":"2024-02-10T08:42:44.075512+0000","last_peered":"2024-02-10T08:42:44.075512+0000","last_clean":"2024-02-10T08:42:44.075512+0000","last_became_active":"2024-02-10T08:40:42.028734+0000","last_became_peered":"2024-02-10T08:40:42.028734+0000","last_unstale":"2024-02-10T08:42:44.075512+0000","last_undegraded":"2024-02-10T08:42:44.075512+0000","last_fullsized":"2024-02-10T08:42:44.075512+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:54.063839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.063839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"63'11","reported_seq":435,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081606+0000","last_change":"2024-02-10T08:41:56.714215+0000","last_active":"2024-02-10T08:42:44.081606+0000","last_peered":"2024-02-10T08:42:44.081606+0000","last_clean":"2024-02-10T08:42:44.081606+0000","last_became_active":"2024-02-10T08:33:20.116696+0000","last_became_peered":"2024-02-10T08:33:20.116696+0000","last_unstale":"2024-02-10T08:42:44.081606+0000","last_undegraded":"2024-02-10T08:42:44.081606+0000","last_fullsized":"2024-02-10T08:42:44.081606+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":424,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.085036+0000","last_change":"2024-02-10T08:41:08.390794+0000","last_active":"2024-02-10T08:42:44.085036+0000","last_peered":"2024-02-10T08:42:44.085036+0000","last_clean":"2024-02-10T08:42:44.085036+0000","last_became_active":"2024-02-10T08:37:10.149190+0000","last_became_peered":"2024-02-10T08:37:10.149190+0000","last_unstale":"2024-02-10T08:42:44.085036+0000","last_undegraded":"2024-02-10T08:42:44.085036+0000","last_fullsized":"2024-02-10T08:42:44.085036+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:08.390722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:08.390722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"100'10","reported_seq":505,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076973+0000","last_change":"2024-02-10T08:42:29.051548+0000","last_active":"2024-02-10T08:42:44.076973+0000","last_peered":"2024-02-10T08:42:44.076973+0000","last_clean":"2024-02-10T08:42:44.076973+0000","last_became_active":"2024-02-10T08:36:29.157807+0000","last_became_peered":"2024-02-10T08:36:29.157807+0000","last_unstale":"2024-02-10T08:42:44.076973+0000","last_undegraded":"2024-02-10T08:42:44.076973+0000","last_fullsized":"2024-02-10T08:42:44.076973+0000","mapping_epoch":59,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":408,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098380+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:42:45.098380+0000","last_peered":"2024-02-10T08:42:45.098380+0000","last_clean":"2024-02-10T08:42:45.098380+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:42:45.098380+0000","last_undegraded":"2024-02-10T08:42:45.098380+0000","last_fullsized":"2024-02-10T08:42:45.098380+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":358,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097134+0000","last_change":"2024-02-10T08:39:38.852396+0000","last_active":"2024-02-10T08:42:45.097134+0000","last_peered":"2024-02-10T08:42:45.097134+0000","last_clean":"2024-02-10T08:42:45.097134+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:42:45.097134+0000","last_undegraded":"2024-02-10T08:42:45.097134+0000","last_fullsized":"2024-02-10T08:42:45.097134+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:38.852332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:38.852332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"63'8","reported_seq":539,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096849+0000","last_change":"2024-02-10T08:41:44.773002+0000","last_active":"2024-02-10T08:42:45.096849+0000","last_peered":"2024-02-10T08:42:45.096849+0000","last_clean":"2024-02-10T08:42:45.096849+0000","last_became_active":"2024-02-10T08:33:22.121394+0000","last_became_peered":"2024-02-10T08:33:22.121394+0000","last_unstale":"2024-02-10T08:42:45.096849+0000","last_undegraded":"2024-02-10T08:42:45.096849+0000","last_fullsized":"2024-02-10T08:42:45.096849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'14","reported_seq":437,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.086818+0000","last_change":"2024-02-10T08:41:24.848074+0000","last_active":"2024-02-10T08:42:44.086818+0000","last_peered":"2024-02-10T08:42:44.086818+0000","last_clean":"2024-02-10T08:42:44.086818+0000","last_became_active":"2024-02-10T08:33:20.109684+0000","last_became_peered":"2024-02-10T08:33:20.109684+0000","last_unstale":"2024-02-10T08:42:44.086818+0000","last_undegraded":"2024-02-10T08:42:44.086818+0000","last_fullsized":"2024-02-10T08:42:44.086818+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:41:24.847990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.847990+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082598+0000","last_change":"2024-02-10T08:41:48.361432+0000","last_active":"2024-02-10T08:42:44.082598+0000","last_peered":"2024-02-10T08:42:44.082598+0000","last_clean":"2024-02-10T08:42:44.082598+0000","last_became_active":"2024-02-10T08:33:18.106997+0000","last_became_peered":"2024-02-10T08:33:18.106997+0000","last_unstale":"2024-02-10T08:42:44.082598+0000","last_undegraded":"2024-02-10T08:42:44.082598+0000","last_fullsized":"2024-02-10T08:42:44.082598+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.361357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.361357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080492+0000","last_change":"2024-02-10T08:41:59.569272+0000","last_active":"2024-02-10T08:42:44.080492+0000","last_peered":"2024-02-10T08:42:44.080492+0000","last_clean":"2024-02-10T08:42:44.080492+0000","last_became_active":"2024-02-10T08:33:24.127219+0000","last_became_peered":"2024-02-10T08:33:24.127219+0000","last_unstale":"2024-02-10T08:42:44.080492+0000","last_undegraded":"2024-02-10T08:42:44.080492+0000","last_fullsized":"2024-02-10T08:42:44.080492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.569193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.569193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":333,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085287+0000","last_change":"2024-02-10T08:41:57.599784+0000","last_active":"2024-02-10T08:42:45.085287+0000","last_peered":"2024-02-10T08:42:45.085287+0000","last_clean":"2024-02-10T08:42:45.085287+0000","last_became_active":"2024-02-10T08:37:10.155465+0000","last_became_peered":"2024-02-10T08:37:10.155465+0000","last_unstale":"2024-02-10T08:42:45.085287+0000","last_undegraded":"2024-02-10T08:42:45.085287+0000","last_fullsized":"2024-02-10T08:42:45.085287+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":416,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.619266+0000","last_change":"2024-02-10T08:42:45.619266+0000","last_active":"2024-02-10T08:42:45.619266+0000","last_peered":"2024-02-10T08:42:45.619266+0000","last_clean":"2024-02-10T08:42:45.619266+0000","last_became_active":"2024-02-10T08:33:22.120064+0000","last_became_peered":"2024-02-10T08:33:22.120064+0000","last_unstale":"2024-02-10T08:42:45.619266+0000","last_undegraded":"2024-02-10T08:42:45.619266+0000","last_fullsized":"2024-02-10T08:42:45.619266+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.619193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.619193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"80'26","reported_seq":506,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096761+0000","last_change":"2024-02-10T08:41:07.712140+0000","last_active":"2024-02-10T08:42:45.096761+0000","last_peered":"2024-02-10T08:42:45.096761+0000","last_clean":"2024-02-10T08:42:45.096761+0000","last_became_active":"2024-02-10T08:40:42.015369+0000","last_became_peered":"2024-02-10T08:40:42.015369+0000","last_unstale":"2024-02-10T08:42:45.096761+0000","last_undegraded":"2024-02-10T08:42:45.096761+0000","last_fullsized":"2024-02-10T08:42:45.096761+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:41:07.712078+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.712078+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":427,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084248+0000","last_change":"2024-02-10T08:41:46.729506+0000","last_active":"2024-02-10T08:42:45.084248+0000","last_peered":"2024-02-10T08:42:45.084248+0000","last_clean":"2024-02-10T08:42:45.084248+0000","last_became_active":"2024-02-10T08:33:18.107201+0000","last_became_peered":"2024-02-10T08:33:18.107201+0000","last_unstale":"2024-02-10T08:42:45.084248+0000","last_undegraded":"2024-02-10T08:42:45.084248+0000","last_fullsized":"2024-02-10T08:42:45.084248+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":407,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088726+0000","last_change":"2024-02-10T08:41:10.710795+0000","last_active":"2024-02-10T08:42:44.088726+0000","last_peered":"2024-02-10T08:42:44.088726+0000","last_clean":"2024-02-10T08:42:44.088726+0000","last_became_active":"2024-02-10T08:33:24.119593+0000","last_became_peered":"2024-02-10T08:33:24.119593+0000","last_unstale":"2024-02-10T08:42:44.088726+0000","last_undegraded":"2024-02-10T08:42:44.088726+0000","last_fullsized":"2024-02-10T08:42:44.088726+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":342,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098170+0000","last_change":"2024-02-10T08:42:35.024210+0000","last_active":"2024-02-10T08:42:45.098170+0000","last_peered":"2024-02-10T08:42:45.098170+0000","last_clean":"2024-02-10T08:42:45.098170+0000","last_became_active":"2024-02-10T08:40:42.027869+0000","last_became_peered":"2024-02-10T08:40:42.027869+0000","last_unstale":"2024-02-10T08:42:45.098170+0000","last_undegraded":"2024-02-10T08:42:45.098170+0000","last_fullsized":"2024-02-10T08:42:45.098170+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":426,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079391+0000","last_change":"2024-02-10T08:41:58.271798+0000","last_active":"2024-02-10T08:42:44.079391+0000","last_peered":"2024-02-10T08:42:44.079391+0000","last_clean":"2024-02-10T08:42:44.079391+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:42:44.079391+0000","last_undegraded":"2024-02-10T08:42:44.079391+0000","last_fullsized":"2024-02-10T08:42:44.079391+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.271728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.271728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'19","reported_seq":394,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075833+0000","last_change":"2024-02-10T08:41:07.950673+0000","last_active":"2024-02-10T08:42:44.075833+0000","last_peered":"2024-02-10T08:42:44.075833+0000","last_clean":"2024-02-10T08:42:44.075833+0000","last_became_active":"2024-02-10T08:37:10.149080+0000","last_became_peered":"2024-02-10T08:37:10.149080+0000","last_unstale":"2024-02-10T08:42:44.075833+0000","last_undegraded":"2024-02-10T08:42:44.075833+0000","last_fullsized":"2024-02-10T08:42:44.075833+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-10T08:41:07.950597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.950597+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":424,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082879+0000","last_change":"2024-02-10T08:41:04.369613+0000","last_active":"2024-02-10T08:42:44.082879+0000","last_peered":"2024-02-10T08:42:44.082879+0000","last_clean":"2024-02-10T08:42:44.082879+0000","last_became_active":"2024-02-10T08:37:10.148083+0000","last_became_peered":"2024-02-10T08:37:10.148083+0000","last_unstale":"2024-02-10T08:42:44.082879+0000","last_undegraded":"2024-02-10T08:42:44.082879+0000","last_fullsized":"2024-02-10T08:42:44.082879+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:04.369534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:04.369534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":77,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085043+0000","last_change":"2024-02-10T08:41:58.300293+0000","last_active":"2024-02-10T08:42:45.085043+0000","last_peered":"2024-02-10T08:42:45.085043+0000","last_clean":"2024-02-10T08:42:45.085043+0000","last_became_active":"2024-02-10T08:41:21.266720+0000","last_became_peered":"2024-02-10T08:41:21.266720+0000","last_unstale":"2024-02-10T08:42:45.085043+0000","last_undegraded":"2024-02-10T08:42:45.085043+0000","last_fullsized":"2024-02-10T08:42:45.085043+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.300183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.300183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":372,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083977+0000","last_change":"2024-02-10T08:41:50.202162+0000","last_active":"2024-02-10T08:42:45.083977+0000","last_peered":"2024-02-10T08:42:45.083977+0000","last_clean":"2024-02-10T08:42:45.083977+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:42:45.083977+0000","last_undegraded":"2024-02-10T08:42:45.083977+0000","last_fullsized":"2024-02-10T08:42:45.083977+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:50.202092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:50.202092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":438,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076994+0000","last_change":"2024-02-10T08:41:11.901601+0000","last_active":"2024-02-10T08:42:44.076994+0000","last_peered":"2024-02-10T08:42:44.076994+0000","last_clean":"2024-02-10T08:42:44.076994+0000","last_became_active":"2024-02-10T08:37:10.148606+0000","last_became_peered":"2024-02-10T08:37:10.148606+0000","last_unstale":"2024-02-10T08:42:44.076994+0000","last_undegraded":"2024-02-10T08:42:44.076994+0000","last_fullsized":"2024-02-10T08:42:44.076994+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.901502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.901502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'12","reported_seq":377,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084397+0000","last_change":"2024-02-10T08:41:07.584245+0000","last_active":"2024-02-10T08:42:45.084397+0000","last_peered":"2024-02-10T08:42:45.084397+0000","last_clean":"2024-02-10T08:42:45.084397+0000","last_became_active":"2024-02-10T08:37:10.192320+0000","last_became_peered":"2024-02-10T08:37:10.192320+0000","last_unstale":"2024-02-10T08:42:45.084397+0000","last_undegraded":"2024-02-10T08:42:45.084397+0000","last_fullsized":"2024-02-10T08:42:45.084397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:07.584127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.584127+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":429,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.084857+0000","last_change":"2024-02-10T08:42:30.494713+0000","last_active":"2024-02-10T08:42:44.084857+0000","last_peered":"2024-02-10T08:42:44.084857+0000","last_clean":"2024-02-10T08:42:44.084857+0000","last_became_active":"2024-02-10T08:37:10.213294+0000","last_became_peered":"2024-02-10T08:37:10.213294+0000","last_unstale":"2024-02-10T08:42:44.084857+0000","last_undegraded":"2024-02-10T08:42:44.084857+0000","last_fullsized":"2024-02-10T08:42:44.084857+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:30.494591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.494591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":331,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.087336+0000","last_change":"2024-02-10T08:41:21.839964+0000","last_active":"2024-02-10T08:42:44.087336+0000","last_peered":"2024-02-10T08:42:44.087336+0000","last_clean":"2024-02-10T08:42:44.087336+0000","last_became_active":"2024-02-10T08:37:10.214571+0000","last_became_peered":"2024-02-10T08:37:10.214571+0000","last_unstale":"2024-02-10T08:42:44.087336+0000","last_undegraded":"2024-02-10T08:42:44.087336+0000","last_fullsized":"2024-02-10T08:42:44.087336+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":420,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084180+0000","last_change":"2024-02-10T08:42:44.704196+0000","last_active":"2024-02-10T08:42:45.084180+0000","last_peered":"2024-02-10T08:42:45.084180+0000","last_clean":"2024-02-10T08:42:45.084180+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:42:45.084180+0000","last_undegraded":"2024-02-10T08:42:45.084180+0000","last_fullsized":"2024-02-10T08:42:45.084180+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:42:44.704131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.704131+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":373,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079067+0000","last_change":"2024-02-10T08:42:32.390215+0000","last_active":"2024-02-10T08:42:44.079067+0000","last_peered":"2024-02-10T08:42:44.079067+0000","last_clean":"2024-02-10T08:42:44.079067+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:42:44.079067+0000","last_undegraded":"2024-02-10T08:42:44.079067+0000","last_fullsized":"2024-02-10T08:42:44.079067+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.390146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.390146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":411,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081335+0000","last_change":"2024-02-10T08:41:19.563777+0000","last_active":"2024-02-10T08:42:44.081335+0000","last_peered":"2024-02-10T08:42:44.081335+0000","last_clean":"2024-02-10T08:42:44.081335+0000","last_became_active":"2024-02-10T08:33:22.120374+0000","last_became_peered":"2024-02-10T08:33:22.120374+0000","last_unstale":"2024-02-10T08:42:44.081335+0000","last_undegraded":"2024-02-10T08:42:44.081335+0000","last_fullsized":"2024-02-10T08:42:44.081335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:19.563700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.563700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"63'14","reported_seq":441,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079732+0000","last_change":"2024-02-10T08:42:00.249642+0000","last_active":"2024-02-10T08:42:44.079732+0000","last_peered":"2024-02-10T08:42:44.079732+0000","last_clean":"2024-02-10T08:42:44.079732+0000","last_became_active":"2024-02-10T08:33:20.257731+0000","last_became_peered":"2024-02-10T08:33:20.257731+0000","last_unstale":"2024-02-10T08:42:44.079732+0000","last_undegraded":"2024-02-10T08:42:44.079732+0000","last_fullsized":"2024-02-10T08:42:44.079732+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":432,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.084949+0000","last_change":"2024-02-10T08:41:24.519217+0000","last_active":"2024-02-10T08:42:44.084949+0000","last_peered":"2024-02-10T08:42:44.084949+0000","last_clean":"2024-02-10T08:42:44.084949+0000","last_became_active":"2024-02-10T08:37:10.148784+0000","last_became_peered":"2024-02-10T08:37:10.148784+0000","last_unstale":"2024-02-10T08:42:44.084949+0000","last_undegraded":"2024-02-10T08:42:44.084949+0000","last_fullsized":"2024-02-10T08:42:44.084949+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.519144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.519144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":415,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075706+0000","last_change":"2024-02-10T08:41:13.978089+0000","last_active":"2024-02-10T08:42:44.075706+0000","last_peered":"2024-02-10T08:42:44.075706+0000","last_clean":"2024-02-10T08:42:44.075706+0000","last_became_active":"2024-02-10T08:33:18.107736+0000","last_became_peered":"2024-02-10T08:33:18.107736+0000","last_unstale":"2024-02-10T08:42:44.075706+0000","last_undegraded":"2024-02-10T08:42:44.075706+0000","last_fullsized":"2024-02-10T08:42:44.075706+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.978013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.978013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":414,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097316+0000","last_change":"2024-02-10T08:40:39.891622+0000","last_active":"2024-02-10T08:42:45.097316+0000","last_peered":"2024-02-10T08:42:45.097316+0000","last_clean":"2024-02-10T08:42:45.097316+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:42:45.097316+0000","last_undegraded":"2024-02-10T08:42:45.097316+0000","last_fullsized":"2024-02-10T08:42:45.097316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:39.891558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:39.891558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.077068+0000","last_change":"2024-02-10T08:41:48.065347+0000","last_active":"2024-02-10T08:42:44.077068+0000","last_peered":"2024-02-10T08:42:44.077068+0000","last_clean":"2024-02-10T08:42:44.077068+0000","last_became_active":"2024-02-10T08:33:24.118602+0000","last_became_peered":"2024-02-10T08:33:24.118602+0000","last_unstale":"2024-02-10T08:42:44.077068+0000","last_undegraded":"2024-02-10T08:42:44.077068+0000","last_fullsized":"2024-02-10T08:42:44.077068+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":369,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075556+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:42:44.075556+0000","last_peered":"2024-02-10T08:42:44.075556+0000","last_clean":"2024-02-10T08:42:44.075556+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:42:44.075556+0000","last_undegraded":"2024-02-10T08:42:44.075556+0000","last_fullsized":"2024-02-10T08:42:44.075556+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":429,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086334+0000","last_change":"2024-02-10T08:41:03.238150+0000","last_active":"2024-02-10T08:42:45.086334+0000","last_peered":"2024-02-10T08:42:45.086334+0000","last_clean":"2024-02-10T08:42:45.086334+0000","last_became_active":"2024-02-10T08:37:10.187269+0000","last_became_peered":"2024-02-10T08:37:10.187269+0000","last_unstale":"2024-02-10T08:42:45.086334+0000","last_undegraded":"2024-02-10T08:42:45.086334+0000","last_fullsized":"2024-02-10T08:42:45.086334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.238035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.238035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'13","reported_seq":466,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084951+0000","last_change":"2024-02-10T08:41:18.579612+0000","last_active":"2024-02-10T08:42:45.084951+0000","last_peered":"2024-02-10T08:42:45.084951+0000","last_clean":"2024-02-10T08:42:45.084951+0000","last_became_active":"2024-02-10T08:37:10.174010+0000","last_became_peered":"2024-02-10T08:37:10.174010+0000","last_unstale":"2024-02-10T08:42:45.084951+0000","last_undegraded":"2024-02-10T08:42:45.084951+0000","last_fullsized":"2024-02-10T08:42:45.084951+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:41:18.579532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.579532+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":431,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084948+0000","last_change":"2024-02-10T08:42:29.250521+0000","last_active":"2024-02-10T08:42:45.084948+0000","last_peered":"2024-02-10T08:42:45.084948+0000","last_clean":"2024-02-10T08:42:45.084948+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:42:45.084948+0000","last_undegraded":"2024-02-10T08:42:45.084948+0000","last_fullsized":"2024-02-10T08:42:45.084948+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.250458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.250458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":984,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086029+0000","last_change":"2024-02-10T08:41:44.751436+0000","last_active":"2024-02-10T08:42:45.086029+0000","last_peered":"2024-02-10T08:42:45.086029+0000","last_clean":"2024-02-10T08:42:45.086029+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:42:45.086029+0000","last_undegraded":"2024-02-10T08:42:45.086029+0000","last_fullsized":"2024-02-10T08:42:45.086029+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:41:44.751371+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.751371+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":560,"num_read_kb":555,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082790+0000","last_change":"2024-02-10T08:41:15.515804+0000","last_active":"2024-02-10T08:42:44.082790+0000","last_peered":"2024-02-10T08:42:44.082790+0000","last_clean":"2024-02-10T08:42:44.082790+0000","last_became_active":"2024-02-10T08:33:24.127470+0000","last_became_peered":"2024-02-10T08:33:24.127470+0000","last_unstale":"2024-02-10T08:42:44.082790+0000","last_undegraded":"2024-02-10T08:42:44.082790+0000","last_fullsized":"2024-02-10T08:42:44.082790+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.515729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.515729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":372,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085062+0000","last_change":"2024-02-10T08:42:28.553646+0000","last_active":"2024-02-10T08:42:45.085062+0000","last_peered":"2024-02-10T08:42:45.085062+0000","last_clean":"2024-02-10T08:42:45.085062+0000","last_became_active":"2024-02-10T08:37:10.196721+0000","last_became_peered":"2024-02-10T08:37:10.196721+0000","last_unstale":"2024-02-10T08:42:45.085062+0000","last_undegraded":"2024-02-10T08:42:45.085062+0000","last_fullsized":"2024-02-10T08:42:45.085062+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:28.553564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.553564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":333,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084956+0000","last_change":"2024-02-10T08:41:53.594734+0000","last_active":"2024-02-10T08:42:45.084956+0000","last_peered":"2024-02-10T08:42:45.084956+0000","last_clean":"2024-02-10T08:42:45.084956+0000","last_became_active":"2024-02-10T08:37:10.155031+0000","last_became_peered":"2024-02-10T08:37:10.155031+0000","last_unstale":"2024-02-10T08:42:45.084956+0000","last_undegraded":"2024-02-10T08:42:45.084956+0000","last_fullsized":"2024-02-10T08:42:45.084956+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":518,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075298+0000","last_change":"2024-02-10T08:42:43.109193+0000","last_active":"2024-02-10T08:42:44.075298+0000","last_peered":"2024-02-10T08:42:44.075298+0000","last_clean":"2024-02-10T08:42:44.075298+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:42:44.075298+0000","last_undegraded":"2024-02-10T08:42:44.075298+0000","last_fullsized":"2024-02-10T08:42:44.075298+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:42:43.109121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.109121+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":420,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076826+0000","last_change":"2024-02-10T08:42:37.145252+0000","last_active":"2024-02-10T08:42:44.076826+0000","last_peered":"2024-02-10T08:42:44.076826+0000","last_clean":"2024-02-10T08:42:44.076826+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:42:44.076826+0000","last_undegraded":"2024-02-10T08:42:44.076826+0000","last_fullsized":"2024-02-10T08:42:44.076826+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.145180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.145180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":408,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097732+0000","last_change":"2024-02-10T08:41:11.702739+0000","last_active":"2024-02-10T08:42:45.097732+0000","last_peered":"2024-02-10T08:42:45.097732+0000","last_clean":"2024-02-10T08:42:45.097732+0000","last_became_active":"2024-02-10T08:33:24.116592+0000","last_became_peered":"2024-02-10T08:33:24.116592+0000","last_unstale":"2024-02-10T08:42:45.097732+0000","last_undegraded":"2024-02-10T08:42:45.097732+0000","last_fullsized":"2024-02-10T08:42:45.097732+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":372,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084843+0000","last_change":"2024-02-10T08:42:04.361526+0000","last_active":"2024-02-10T08:42:45.084843+0000","last_peered":"2024-02-10T08:42:45.084843+0000","last_clean":"2024-02-10T08:42:45.084843+0000","last_became_active":"2024-02-10T08:37:10.174764+0000","last_became_peered":"2024-02-10T08:37:10.174764+0000","last_unstale":"2024-02-10T08:42:45.084843+0000","last_undegraded":"2024-02-10T08:42:45.084843+0000","last_fullsized":"2024-02-10T08:42:45.084843+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:04.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.361430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":424,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.084766+0000","last_change":"2024-02-10T08:41:54.316045+0000","last_active":"2024-02-10T08:42:44.084766+0000","last_peered":"2024-02-10T08:42:44.084766+0000","last_clean":"2024-02-10T08:42:44.084766+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:42:44.084766+0000","last_undegraded":"2024-02-10T08:42:44.084766+0000","last_fullsized":"2024-02-10T08:42:44.084766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:41:54.315983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.315983+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"84'15","reported_seq":445,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083971+0000","last_change":"2024-02-10T08:42:40.687091+0000","last_active":"2024-02-10T08:42:45.083971+0000","last_peered":"2024-02-10T08:42:45.083971+0000","last_clean":"2024-02-10T08:42:45.083971+0000","last_became_active":"2024-02-10T08:33:20.110314+0000","last_became_peered":"2024-02-10T08:33:20.110314+0000","last_unstale":"2024-02-10T08:42:45.083971+0000","last_undegraded":"2024-02-10T08:42:45.083971+0000","last_fullsized":"2024-02-10T08:42:45.083971+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:42:40.687013+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.687013+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":429,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083914+0000","last_change":"2024-02-10T08:41:24.622753+0000","last_active":"2024-02-10T08:42:45.083914+0000","last_peered":"2024-02-10T08:42:45.083914+0000","last_clean":"2024-02-10T08:42:45.083914+0000","last_became_active":"2024-02-10T08:37:10.182906+0000","last_became_peered":"2024-02-10T08:37:10.182906+0000","last_unstale":"2024-02-10T08:42:45.083914+0000","last_undegraded":"2024-02-10T08:42:45.083914+0000","last_fullsized":"2024-02-10T08:42:45.083914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.622691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.622691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":333,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097100+0000","last_change":"2024-02-10T08:42:40.143847+0000","last_active":"2024-02-10T08:42:45.097100+0000","last_peered":"2024-02-10T08:42:45.097100+0000","last_clean":"2024-02-10T08:42:45.097100+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:42:45.097100+0000","last_undegraded":"2024-02-10T08:42:45.097100+0000","last_fullsized":"2024-02-10T08:42:45.097100+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.143787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.143787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097513+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:42:45.097513+0000","last_peered":"2024-02-10T08:42:45.097513+0000","last_clean":"2024-02-10T08:42:45.097513+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:42:45.097513+0000","last_undegraded":"2024-02-10T08:42:45.097513+0000","last_fullsized":"2024-02-10T08:42:45.097513+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":443,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085146+0000","last_change":"2024-02-10T08:41:22.395964+0000","last_active":"2024-02-10T08:42:45.085146+0000","last_peered":"2024-02-10T08:42:45.085146+0000","last_clean":"2024-02-10T08:42:45.085146+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:42:45.085146+0000","last_undegraded":"2024-02-10T08:42:45.085146+0000","last_fullsized":"2024-02-10T08:42:45.085146+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.395872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.395872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"78'17","reported_seq":444,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.086909+0000","last_change":"2024-02-10T08:41:44.633272+0000","last_active":"2024-02-10T08:42:44.086909+0000","last_peered":"2024-02-10T08:42:44.086909+0000","last_clean":"2024-02-10T08:42:44.086909+0000","last_became_active":"2024-02-10T08:33:20.117162+0000","last_became_peered":"2024-02-10T08:33:20.117162+0000","last_unstale":"2024-02-10T08:42:44.086909+0000","last_undegraded":"2024-02-10T08:42:44.086909+0000","last_fullsized":"2024-02-10T08:42:44.086909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":441,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080353+0000","last_change":"2024-02-10T08:40:38.667994+0000","last_active":"2024-02-10T08:42:44.080353+0000","last_peered":"2024-02-10T08:42:44.080353+0000","last_clean":"2024-02-10T08:42:44.080353+0000","last_became_active":"2024-02-10T08:37:10.131260+0000","last_became_peered":"2024-02-10T08:37:10.131260+0000","last_unstale":"2024-02-10T08:42:44.080353+0000","last_undegraded":"2024-02-10T08:42:44.080353+0000","last_fullsized":"2024-02-10T08:42:44.080353+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:40:38.667923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:38.667923+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":413,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097923+0000","last_change":"2024-02-10T08:42:05.067474+0000","last_active":"2024-02-10T08:42:45.097923+0000","last_peered":"2024-02-10T08:42:45.097923+0000","last_clean":"2024-02-10T08:42:45.097923+0000","last_became_active":"2024-02-10T08:33:24.115859+0000","last_became_peered":"2024-02-10T08:33:24.115859+0000","last_unstale":"2024-02-10T08:42:45.097923+0000","last_undegraded":"2024-02-10T08:42:45.097923+0000","last_fullsized":"2024-02-10T08:42:45.097923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085649+0000","last_change":"2024-02-10T08:41:15.304181+0000","last_active":"2024-02-10T08:42:45.085649+0000","last_peered":"2024-02-10T08:42:45.085649+0000","last_clean":"2024-02-10T08:42:45.085649+0000","last_became_active":"2024-02-10T08:36:37.053096+0000","last_became_peered":"2024-02-10T08:36:37.053096+0000","last_unstale":"2024-02-10T08:42:45.085649+0000","last_undegraded":"2024-02-10T08:42:45.085649+0000","last_fullsized":"2024-02-10T08:42:45.085649+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:15.304104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:15.304104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"84'13","reported_seq":399,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097780+0000","last_change":"2024-02-10T08:41:23.736997+0000","last_active":"2024-02-10T08:42:45.097780+0000","last_peered":"2024-02-10T08:42:45.097780+0000","last_clean":"2024-02-10T08:42:45.097780+0000","last_became_active":"2024-02-10T08:37:10.286374+0000","last_became_peered":"2024-02-10T08:37:10.286374+0000","last_unstale":"2024-02-10T08:42:45.097780+0000","last_undegraded":"2024-02-10T08:42:45.097780+0000","last_fullsized":"2024-02-10T08:42:45.097780+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.736922+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":429,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085094+0000","last_change":"2024-02-10T08:41:21.384611+0000","last_active":"2024-02-10T08:42:45.085094+0000","last_peered":"2024-02-10T08:42:45.085094+0000","last_clean":"2024-02-10T08:42:45.085094+0000","last_became_active":"2024-02-10T08:37:10.212595+0000","last_became_peered":"2024-02-10T08:37:10.212595+0000","last_unstale":"2024-02-10T08:42:45.085094+0000","last_undegraded":"2024-02-10T08:42:45.085094+0000","last_fullsized":"2024-02-10T08:42:45.085094+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":414,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.347767+0000","last_change":"2024-02-10T08:42:45.347746+0000","last_active":"2024-02-10T08:42:45.347767+0000","last_peered":"2024-02-10T08:42:45.347767+0000","last_clean":"2024-02-10T08:42:45.347767+0000","last_became_active":"2024-02-10T08:33:24.122939+0000","last_became_peered":"2024-02-10T08:33:24.122939+0000","last_unstale":"2024-02-10T08:42:45.347767+0000","last_undegraded":"2024-02-10T08:42:45.347767+0000","last_fullsized":"2024-02-10T08:42:45.347767+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.347685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.347685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":361,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088374+0000","last_change":"2024-02-10T08:42:26.615082+0000","last_active":"2024-02-10T08:42:44.088374+0000","last_peered":"2024-02-10T08:42:44.088374+0000","last_clean":"2024-02-10T08:42:44.088374+0000","last_became_active":"2024-02-10T08:36:37.040862+0000","last_became_peered":"2024-02-10T08:36:37.040862+0000","last_unstale":"2024-02-10T08:42:44.088374+0000","last_undegraded":"2024-02-10T08:42:44.088374+0000","last_fullsized":"2024-02-10T08:42:44.088374+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"63'6","reported_seq":421,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097818+0000","last_change":"2024-02-10T08:40:23.860591+0000","last_active":"2024-02-10T08:42:45.097818+0000","last_peered":"2024-02-10T08:42:45.097818+0000","last_clean":"2024-02-10T08:42:45.097818+0000","last_became_active":"2024-02-10T08:33:20.257254+0000","last_became_peered":"2024-02-10T08:33:20.257254+0000","last_unstale":"2024-02-10T08:42:45.097818+0000","last_undegraded":"2024-02-10T08:42:45.097818+0000","last_fullsized":"2024-02-10T08:42:45.097818+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'6","last_scrub_stamp":"2024-02-10T08:40:23.860526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:23.860526+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083498+0000","last_change":"2024-02-10T08:42:02.183760+0000","last_active":"2024-02-10T08:42:44.083498+0000","last_peered":"2024-02-10T08:42:44.083498+0000","last_clean":"2024-02-10T08:42:44.083498+0000","last_became_active":"2024-02-10T08:33:18.105676+0000","last_became_peered":"2024-02-10T08:33:18.105676+0000","last_unstale":"2024-02-10T08:42:44.083498+0000","last_undegraded":"2024-02-10T08:42:44.083498+0000","last_fullsized":"2024-02-10T08:42:44.083498+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082531+0000","last_change":"2024-02-10T08:42:35.402000+0000","last_active":"2024-02-10T08:42:44.082531+0000","last_peered":"2024-02-10T08:42:44.082531+0000","last_clean":"2024-02-10T08:42:44.082531+0000","last_became_active":"2024-02-10T08:33:24.127130+0000","last_became_peered":"2024-02-10T08:33:24.127130+0000","last_unstale":"2024-02-10T08:42:44.082531+0000","last_undegraded":"2024-02-10T08:42:44.082531+0000","last_fullsized":"2024-02-10T08:42:44.082531+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":358,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075598+0000","last_change":"2024-02-10T08:41:23.032812+0000","last_active":"2024-02-10T08:42:44.075598+0000","last_peered":"2024-02-10T08:42:44.075598+0000","last_clean":"2024-02-10T08:42:44.075598+0000","last_became_active":"2024-02-10T08:36:37.051128+0000","last_became_peered":"2024-02-10T08:36:37.051128+0000","last_unstale":"2024-02-10T08:42:44.075598+0000","last_undegraded":"2024-02-10T08:42:44.075598+0000","last_fullsized":"2024-02-10T08:42:44.075598+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'10","reported_seq":432,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098458+0000","last_change":"2024-02-10T08:42:24.087582+0000","last_active":"2024-02-10T08:42:45.098458+0000","last_peered":"2024-02-10T08:42:45.098458+0000","last_clean":"2024-02-10T08:42:45.098458+0000","last_became_active":"2024-02-10T08:33:20.256090+0000","last_became_peered":"2024-02-10T08:33:20.256090+0000","last_unstale":"2024-02-10T08:42:45.098458+0000","last_undegraded":"2024-02-10T08:42:45.098458+0000","last_fullsized":"2024-02-10T08:42:45.098458+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":420,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076783+0000","last_change":"2024-02-10T08:42:42.059823+0000","last_active":"2024-02-10T08:42:44.076783+0000","last_peered":"2024-02-10T08:42:44.076783+0000","last_clean":"2024-02-10T08:42:44.076783+0000","last_became_active":"2024-02-10T08:33:18.105613+0000","last_became_peered":"2024-02-10T08:33:18.105613+0000","last_unstale":"2024-02-10T08:42:44.076783+0000","last_undegraded":"2024-02-10T08:42:44.076783+0000","last_fullsized":"2024-02-10T08:42:44.076783+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:42.059708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.059708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":545,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075582+0000","last_change":"2024-02-10T08:42:28.090195+0000","last_active":"2024-02-10T08:42:44.075582+0000","last_peered":"2024-02-10T08:42:44.075582+0000","last_clean":"2024-02-10T08:42:44.075582+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:42:44.075582+0000","last_undegraded":"2024-02-10T08:42:44.075582+0000","last_fullsized":"2024-02-10T08:42:44.075582+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:42:28.090111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.090111+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":328,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083477+0000","last_change":"2024-02-10T08:41:16.497083+0000","last_active":"2024-02-10T08:42:44.083477+0000","last_peered":"2024-02-10T08:42:44.083477+0000","last_clean":"2024-02-10T08:42:44.083477+0000","last_became_active":"2024-02-10T08:37:10.241743+0000","last_became_peered":"2024-02-10T08:37:10.241743+0000","last_unstale":"2024-02-10T08:42:44.083477+0000","last_undegraded":"2024-02-10T08:42:44.083477+0000","last_fullsized":"2024-02-10T08:42:44.083477+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.496995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.496995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":329,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079387+0000","last_change":"2024-02-10T08:41:21.463600+0000","last_active":"2024-02-10T08:42:44.079387+0000","last_peered":"2024-02-10T08:42:44.079387+0000","last_clean":"2024-02-10T08:42:44.079387+0000","last_became_active":"2024-02-10T08:37:10.232026+0000","last_became_peered":"2024-02-10T08:37:10.232026+0000","last_unstale":"2024-02-10T08:42:44.079387+0000","last_undegraded":"2024-02-10T08:42:44.079387+0000","last_fullsized":"2024-02-10T08:42:44.079387+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.463527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.463527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"63'9","reported_seq":429,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075436+0000","last_change":"2024-02-10T08:42:05.106561+0000","last_active":"2024-02-10T08:42:44.075436+0000","last_peered":"2024-02-10T08:42:44.075436+0000","last_clean":"2024-02-10T08:42:44.075436+0000","last_became_active":"2024-02-10T08:33:20.256105+0000","last_became_peered":"2024-02-10T08:33:20.256105+0000","last_unstale":"2024-02-10T08:42:44.075436+0000","last_undegraded":"2024-02-10T08:42:44.075436+0000","last_fullsized":"2024-02-10T08:42:44.075436+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":326,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083812+0000","last_change":"2024-02-10T08:40:27.638382+0000","last_active":"2024-02-10T08:42:44.083812+0000","last_peered":"2024-02-10T08:42:44.083812+0000","last_clean":"2024-02-10T08:42:44.083812+0000","last_became_active":"2024-02-10T08:37:10.216575+0000","last_became_peered":"2024-02-10T08:37:10.216575+0000","last_unstale":"2024-02-10T08:42:44.083812+0000","last_undegraded":"2024-02-10T08:42:44.083812+0000","last_fullsized":"2024-02-10T08:42:44.083812+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:27.638304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:40:27.638304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"63'8","reported_seq":536,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098644+0000","last_change":"2024-02-10T08:42:28.051493+0000","last_active":"2024-02-10T08:42:45.098644+0000","last_peered":"2024-02-10T08:42:45.098644+0000","last_clean":"2024-02-10T08:42:45.098644+0000","last_became_active":"2024-02-10T08:33:22.119261+0000","last_became_peered":"2024-02-10T08:33:22.119261+0000","last_unstale":"2024-02-10T08:42:45.098644+0000","last_undegraded":"2024-02-10T08:42:45.098644+0000","last_fullsized":"2024-02-10T08:42:45.098644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":407,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080514+0000","last_change":"2024-02-10T08:40:41.538200+0000","last_active":"2024-02-10T08:42:44.080514+0000","last_peered":"2024-02-10T08:42:44.080514+0000","last_clean":"2024-02-10T08:42:44.080514+0000","last_became_active":"2024-02-10T08:33:24.119272+0000","last_became_peered":"2024-02-10T08:33:24.119272+0000","last_unstale":"2024-02-10T08:42:44.080514+0000","last_undegraded":"2024-02-10T08:42:44.080514+0000","last_fullsized":"2024-02-10T08:42:44.080514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:41.538139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:41.538139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":376,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084726+0000","last_change":"2024-02-10T08:42:40.371733+0000","last_active":"2024-02-10T08:42:45.084726+0000","last_peered":"2024-02-10T08:42:45.084726+0000","last_clean":"2024-02-10T08:42:45.084726+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:42:45.084726+0000","last_undegraded":"2024-02-10T08:42:45.084726+0000","last_fullsized":"2024-02-10T08:42:45.084726+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:42:40.371655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.371655+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"78'13","reported_seq":440,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083815+0000","last_change":"2024-02-10T08:42:25.482887+0000","last_active":"2024-02-10T08:42:45.083815+0000","last_peered":"2024-02-10T08:42:45.083815+0000","last_clean":"2024-02-10T08:42:45.083815+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:42:45.083815+0000","last_undegraded":"2024-02-10T08:42:45.083815+0000","last_fullsized":"2024-02-10T08:42:45.083815+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:42:25.482788+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.482788+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081568+0000","last_change":"2024-02-10T08:41:44.355018+0000","last_active":"2024-02-10T08:42:44.081568+0000","last_peered":"2024-02-10T08:42:44.081568+0000","last_clean":"2024-02-10T08:42:44.081568+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:42:44.081568+0000","last_undegraded":"2024-02-10T08:42:44.081568+0000","last_fullsized":"2024-02-10T08:42:44.081568+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:44.354945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.354945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":424,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097840+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:42:45.097840+0000","last_peered":"2024-02-10T08:42:45.097840+0000","last_clean":"2024-02-10T08:42:45.097840+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:42:45.097840+0000","last_undegraded":"2024-02-10T08:42:45.097840+0000","last_fullsized":"2024-02-10T08:42:45.097840+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":328,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075866+0000","last_change":"2024-02-10T08:41:12.929709+0000","last_active":"2024-02-10T08:42:44.075866+0000","last_peered":"2024-02-10T08:42:44.075866+0000","last_clean":"2024-02-10T08:42:44.075866+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:42:44.075866+0000","last_undegraded":"2024-02-10T08:42:44.075866+0000","last_fullsized":"2024-02-10T08:42:44.075866+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:12.929592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:12.929592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":368,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080872+0000","last_change":"2024-02-10T08:41:20.506111+0000","last_active":"2024-02-10T08:42:44.080872+0000","last_peered":"2024-02-10T08:42:44.080872+0000","last_clean":"2024-02-10T08:42:44.080872+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:42:44.080872+0000","last_undegraded":"2024-02-10T08:42:44.080872+0000","last_fullsized":"2024-02-10T08:42:44.080872+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.506032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.506032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"63'11","reported_seq":430,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083723+0000","last_change":"2024-02-10T08:41:05.682672+0000","last_active":"2024-02-10T08:42:44.083723+0000","last_peered":"2024-02-10T08:42:44.083723+0000","last_clean":"2024-02-10T08:42:44.083723+0000","last_became_active":"2024-02-10T08:33:20.117985+0000","last_became_peered":"2024-02-10T08:33:20.117985+0000","last_unstale":"2024-02-10T08:42:44.083723+0000","last_undegraded":"2024-02-10T08:42:44.083723+0000","last_fullsized":"2024-02-10T08:42:44.083723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:05.682577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.682577+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":431,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085092+0000","last_change":"2024-02-10T08:41:56.648067+0000","last_active":"2024-02-10T08:42:45.085092+0000","last_peered":"2024-02-10T08:42:45.085092+0000","last_clean":"2024-02-10T08:42:45.085092+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:42:45.085092+0000","last_undegraded":"2024-02-10T08:42:45.085092+0000","last_fullsized":"2024-02-10T08:42:45.085092+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.647993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":416,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083893+0000","last_change":"2024-02-10T08:42:31.622734+0000","last_active":"2024-02-10T08:42:45.083893+0000","last_peered":"2024-02-10T08:42:45.083893+0000","last_clean":"2024-02-10T08:42:45.083893+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:42:45.083893+0000","last_undegraded":"2024-02-10T08:42:45.083893+0000","last_fullsized":"2024-02-10T08:42:45.083893+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:31.622670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.622670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":422,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096784+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:42:45.096784+0000","last_peered":"2024-02-10T08:42:45.096784+0000","last_clean":"2024-02-10T08:42:45.096784+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:42:45.096784+0000","last_undegraded":"2024-02-10T08:42:45.096784+0000","last_fullsized":"2024-02-10T08:42:45.096784+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":372,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097723+0000","last_change":"2024-02-10T08:42:32.012914+0000","last_active":"2024-02-10T08:42:45.097723+0000","last_peered":"2024-02-10T08:42:45.097723+0000","last_clean":"2024-02-10T08:42:45.097723+0000","last_became_active":"2024-02-10T08:37:10.250880+0000","last_became_peered":"2024-02-10T08:37:10.250880+0000","last_unstale":"2024-02-10T08:42:45.097723+0000","last_undegraded":"2024-02-10T08:42:45.097723+0000","last_fullsized":"2024-02-10T08:42:45.097723+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"63'4","reported_seq":418,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096674+0000","last_change":"2024-02-10T08:40:31.849161+0000","last_active":"2024-02-10T08:42:45.096674+0000","last_peered":"2024-02-10T08:42:45.096674+0000","last_clean":"2024-02-10T08:42:45.096674+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:42:45.096674+0000","last_undegraded":"2024-02-10T08:42:45.096674+0000","last_fullsized":"2024-02-10T08:42:45.096674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'4","last_scrub_stamp":"2024-02-10T08:40:31.849092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:31.849092+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":415,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076681+0000","last_change":"2024-02-10T08:41:05.909459+0000","last_active":"2024-02-10T08:42:44.076681+0000","last_peered":"2024-02-10T08:42:44.076681+0000","last_clean":"2024-02-10T08:42:44.076681+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:42:44.076681+0000","last_undegraded":"2024-02-10T08:42:44.076681+0000","last_fullsized":"2024-02-10T08:42:44.076681+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:05.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:05.909358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":419,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083409+0000","last_change":"2024-02-10T08:42:43.508710+0000","last_active":"2024-02-10T08:42:44.083409+0000","last_peered":"2024-02-10T08:42:44.083409+0000","last_clean":"2024-02-10T08:42:44.083409+0000","last_became_active":"2024-02-10T08:33:22.123177+0000","last_became_peered":"2024-02-10T08:33:22.123177+0000","last_unstale":"2024-02-10T08:42:44.083409+0000","last_undegraded":"2024-02-10T08:42:44.083409+0000","last_fullsized":"2024-02-10T08:42:44.083409+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.508636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.508636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080591+0000","last_change":"2024-02-10T08:41:03.331622+0000","last_active":"2024-02-10T08:42:44.080591+0000","last_peered":"2024-02-10T08:42:44.080591+0000","last_clean":"2024-02-10T08:42:44.080591+0000","last_became_active":"2024-02-10T08:33:24.119679+0000","last_became_peered":"2024-02-10T08:33:24.119679+0000","last_unstale":"2024-02-10T08:42:44.080591+0000","last_undegraded":"2024-02-10T08:42:44.080591+0000","last_fullsized":"2024-02-10T08:42:44.080591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":358,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.098242+0000","last_change":"2024-02-10T08:39:41.795320+0000","last_active":"2024-02-10T08:42:45.098242+0000","last_peered":"2024-02-10T08:42:45.098242+0000","last_clean":"2024-02-10T08:42:45.098242+0000","last_became_active":"2024-02-10T08:36:37.164084+0000","last_became_peered":"2024-02-10T08:36:37.164084+0000","last_unstale":"2024-02-10T08:42:45.098242+0000","last_undegraded":"2024-02-10T08:42:45.098242+0000","last_fullsized":"2024-02-10T08:42:45.098242+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:41.795227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:39:41.795227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"63'11","reported_seq":431,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097185+0000","last_change":"2024-02-10T08:41:03.770782+0000","last_active":"2024-02-10T08:42:45.097185+0000","last_peered":"2024-02-10T08:42:45.097185+0000","last_clean":"2024-02-10T08:42:45.097185+0000","last_became_active":"2024-02-10T08:33:20.256286+0000","last_became_peered":"2024-02-10T08:33:20.256286+0000","last_unstale":"2024-02-10T08:42:45.097185+0000","last_undegraded":"2024-02-10T08:42:45.097185+0000","last_fullsized":"2024-02-10T08:42:45.097185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082517+0000","last_change":"2024-02-10T08:41:52.364114+0000","last_active":"2024-02-10T08:42:44.082517+0000","last_peered":"2024-02-10T08:42:44.082517+0000","last_clean":"2024-02-10T08:42:44.082517+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:42:44.082517+0000","last_undegraded":"2024-02-10T08:42:44.082517+0000","last_fullsized":"2024-02-10T08:42:44.082517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:52.364044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:52.364044+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":406,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.085216+0000","last_change":"2024-02-10T08:40:30.446435+0000","last_active":"2024-02-10T08:42:44.085216+0000","last_peered":"2024-02-10T08:42:44.085216+0000","last_clean":"2024-02-10T08:42:44.085216+0000","last_became_active":"2024-02-10T08:33:22.122255+0000","last_became_peered":"2024-02-10T08:33:22.122255+0000","last_unstale":"2024-02-10T08:42:44.085216+0000","last_undegraded":"2024-02-10T08:42:44.085216+0000","last_fullsized":"2024-02-10T08:42:44.085216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:30.446350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:40:30.446350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":422,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097922+0000","last_change":"2024-02-10T08:42:43.111366+0000","last_active":"2024-02-10T08:42:45.097922+0000","last_peered":"2024-02-10T08:42:45.097922+0000","last_clean":"2024-02-10T08:42:45.097922+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:42:45.097922+0000","last_undegraded":"2024-02-10T08:42:45.097922+0000","last_fullsized":"2024-02-10T08:42:45.097922+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.111309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.111309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":334,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.080011+0000","last_change":"2024-02-10T08:42:26.388122+0000","last_active":"2024-02-10T08:42:44.080011+0000","last_peered":"2024-02-10T08:42:44.080011+0000","last_clean":"2024-02-10T08:42:44.080011+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:42:44.080011+0000","last_undegraded":"2024-02-10T08:42:44.080011+0000","last_fullsized":"2024-02-10T08:42:44.080011+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.388002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.388002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'10","reported_seq":367,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083716+0000","last_change":"2024-02-10T08:41:53.573192+0000","last_active":"2024-02-10T08:42:44.083716+0000","last_peered":"2024-02-10T08:42:44.083716+0000","last_clean":"2024-02-10T08:42:44.083716+0000","last_became_active":"2024-02-10T08:37:10.242048+0000","last_became_peered":"2024-02-10T08:37:10.242048+0000","last_unstale":"2024-02-10T08:42:44.083716+0000","last_undegraded":"2024-02-10T08:42:44.083716+0000","last_fullsized":"2024-02-10T08:42:44.083716+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:53.573125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.573125+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":429,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075626+0000","last_change":"2024-02-10T08:42:40.131559+0000","last_active":"2024-02-10T08:42:44.075626+0000","last_peered":"2024-02-10T08:42:44.075626+0000","last_clean":"2024-02-10T08:42:44.075626+0000","last_became_active":"2024-02-10T08:34:41.695029+0000","last_became_peered":"2024-02-10T08:34:41.695029+0000","last_unstale":"2024-02-10T08:42:44.075626+0000","last_undegraded":"2024-02-10T08:42:44.075626+0000","last_fullsized":"2024-02-10T08:42:44.075626+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.131457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.131457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":410,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081513+0000","last_change":"2024-02-10T08:42:23.529119+0000","last_active":"2024-02-10T08:42:44.081513+0000","last_peered":"2024-02-10T08:42:44.081513+0000","last_clean":"2024-02-10T08:42:44.081513+0000","last_became_active":"2024-02-10T08:33:24.123579+0000","last_became_peered":"2024-02-10T08:33:24.123579+0000","last_unstale":"2024-02-10T08:42:44.081513+0000","last_undegraded":"2024-02-10T08:42:44.081513+0000","last_fullsized":"2024-02-10T08:42:44.081513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:42:23.529013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.529013+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084121+0000","last_change":"2024-02-10T08:41:48.214886+0000","last_active":"2024-02-10T08:42:45.084121+0000","last_peered":"2024-02-10T08:42:45.084121+0000","last_clean":"2024-02-10T08:42:45.084121+0000","last_became_active":"2024-02-10T08:36:37.047591+0000","last_became_peered":"2024-02-10T08:36:37.047591+0000","last_unstale":"2024-02-10T08:42:45.084121+0000","last_undegraded":"2024-02-10T08:42:45.084121+0000","last_fullsized":"2024-02-10T08:42:45.084121+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.214822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.214822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"63'11","reported_seq":438,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082700+0000","last_change":"2024-02-10T08:41:46.403355+0000","last_active":"2024-02-10T08:42:44.082700+0000","last_peered":"2024-02-10T08:42:44.082700+0000","last_clean":"2024-02-10T08:42:44.082700+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:42:44.082700+0000","last_undegraded":"2024-02-10T08:42:44.082700+0000","last_fullsized":"2024-02-10T08:42:44.082700+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:46.403281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.403281+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":428,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.096973+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:42:45.096973+0000","last_peered":"2024-02-10T08:42:45.096973+0000","last_clean":"2024-02-10T08:42:45.096973+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:42:45.096973+0000","last_undegraded":"2024-02-10T08:42:45.096973+0000","last_fullsized":"2024-02-10T08:42:45.096973+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":414,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084886+0000","last_change":"2024-02-10T08:41:49.644781+0000","last_active":"2024-02-10T08:42:45.084886+0000","last_peered":"2024-02-10T08:42:45.084886+0000","last_clean":"2024-02-10T08:42:45.084886+0000","last_became_active":"2024-02-10T08:33:24.113014+0000","last_became_peered":"2024-02-10T08:33:24.113014+0000","last_unstale":"2024-02-10T08:42:45.084886+0000","last_undegraded":"2024-02-10T08:42:45.084886+0000","last_fullsized":"2024-02-10T08:42:45.084886+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.084984+0000","last_change":"2024-02-10T08:41:23.662181+0000","last_active":"2024-02-10T08:42:45.084984+0000","last_peered":"2024-02-10T08:42:45.084984+0000","last_clean":"2024-02-10T08:42:45.084984+0000","last_became_active":"2024-02-10T08:36:37.123728+0000","last_became_peered":"2024-02-10T08:36:37.123728+0000","last_unstale":"2024-02-10T08:42:45.084984+0000","last_undegraded":"2024-02-10T08:42:45.084984+0000","last_fullsized":"2024-02-10T08:42:45.084984+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"174'22","reported_seq":450,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086359+0000","last_change":"2024-02-10T08:41:45.260263+0000","last_active":"2024-02-10T08:42:45.086359+0000","last_peered":"2024-02-10T08:42:45.086359+0000","last_clean":"2024-02-10T08:42:45.086359+0000","last_became_active":"2024-02-10T08:33:20.112304+0000","last_became_peered":"2024-02-10T08:33:20.112304+0000","last_unstale":"2024-02-10T08:42:45.086359+0000","last_undegraded":"2024-02-10T08:42:45.086359+0000","last_fullsized":"2024-02-10T08:42:45.086359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":421,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079479+0000","last_change":"2024-02-10T08:42:23.333128+0000","last_active":"2024-02-10T08:42:44.079479+0000","last_peered":"2024-02-10T08:42:44.079479+0000","last_clean":"2024-02-10T08:42:44.079479+0000","last_became_active":"2024-02-10T08:33:18.109992+0000","last_became_peered":"2024-02-10T08:33:18.109992+0000","last_unstale":"2024-02-10T08:42:44.079479+0000","last_undegraded":"2024-02-10T08:42:44.079479+0000","last_fullsized":"2024-02-10T08:42:44.079479+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":407,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081708+0000","last_change":"2024-02-10T08:40:35.620440+0000","last_active":"2024-02-10T08:42:44.081708+0000","last_peered":"2024-02-10T08:42:44.081708+0000","last_clean":"2024-02-10T08:42:44.081708+0000","last_became_active":"2024-02-10T08:33:24.125669+0000","last_became_peered":"2024-02-10T08:33:24.125669+0000","last_unstale":"2024-02-10T08:42:44.081708+0000","last_undegraded":"2024-02-10T08:42:44.081708+0000","last_fullsized":"2024-02-10T08:42:44.081708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:35.620358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:35.620358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":334,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079297+0000","last_change":"2024-02-10T08:41:56.265678+0000","last_active":"2024-02-10T08:42:44.079297+0000","last_peered":"2024-02-10T08:42:44.079297+0000","last_clean":"2024-02-10T08:42:44.079297+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:42:44.079297+0000","last_undegraded":"2024-02-10T08:42:44.079297+0000","last_fullsized":"2024-02-10T08:42:44.079297+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:56.265562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.265562+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"63'12","reported_seq":461,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097391+0000","last_change":"2024-02-10T08:41:53.828692+0000","last_active":"2024-02-10T08:42:45.097391+0000","last_peered":"2024-02-10T08:42:45.097391+0000","last_clean":"2024-02-10T08:42:45.097391+0000","last_became_active":"2024-02-10T08:40:42.014594+0000","last_became_peered":"2024-02-10T08:40:42.014594+0000","last_unstale":"2024-02-10T08:42:45.097391+0000","last_undegraded":"2024-02-10T08:42:45.097391+0000","last_fullsized":"2024-02-10T08:42:45.097391+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":419,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097829+0000","last_change":"2024-02-10T08:41:57.957397+0000","last_active":"2024-02-10T08:42:45.097829+0000","last_peered":"2024-02-10T08:42:45.097829+0000","last_clean":"2024-02-10T08:42:45.097829+0000","last_became_active":"2024-02-10T08:33:18.106975+0000","last_became_peered":"2024-02-10T08:33:18.106975+0000","last_unstale":"2024-02-10T08:42:45.097829+0000","last_undegraded":"2024-02-10T08:42:45.097829+0000","last_fullsized":"2024-02-10T08:42:45.097829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":413,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097872+0000","last_change":"2024-02-10T08:42:37.092622+0000","last_active":"2024-02-10T08:42:45.097872+0000","last_peered":"2024-02-10T08:42:45.097872+0000","last_clean":"2024-02-10T08:42:45.097872+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:42:45.097872+0000","last_undegraded":"2024-02-10T08:42:45.097872+0000","last_fullsized":"2024-02-10T08:42:45.097872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.092522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.092522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":363,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.085429+0000","last_change":"2024-02-10T08:42:01.674000+0000","last_active":"2024-02-10T08:42:45.085429+0000","last_peered":"2024-02-10T08:42:45.085429+0000","last_clean":"2024-02-10T08:42:45.085429+0000","last_became_active":"2024-02-10T08:36:37.037980+0000","last_became_peered":"2024-02-10T08:36:37.037980+0000","last_unstale":"2024-02-10T08:42:45.085429+0000","last_undegraded":"2024-02-10T08:42:45.085429+0000","last_fullsized":"2024-02-10T08:42:45.085429+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:01.673932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.673932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'9","reported_seq":451,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088588+0000","last_change":"2024-02-10T08:41:19.765286+0000","last_active":"2024-02-10T08:42:44.088588+0000","last_peered":"2024-02-10T08:42:44.088588+0000","last_clean":"2024-02-10T08:42:44.088588+0000","last_became_active":"2024-02-10T08:37:10.313190+0000","last_became_peered":"2024-02-10T08:37:10.313190+0000","last_unstale":"2024-02-10T08:42:44.088588+0000","last_undegraded":"2024-02-10T08:42:44.088588+0000","last_fullsized":"2024-02-10T08:42:44.088588+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":415,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081132+0000","last_change":"2024-02-10T08:41:13.452076+0000","last_active":"2024-02-10T08:42:44.081132+0000","last_peered":"2024-02-10T08:42:44.081132+0000","last_clean":"2024-02-10T08:42:44.081132+0000","last_became_active":"2024-02-10T08:33:18.105260+0000","last_became_peered":"2024-02-10T08:33:18.105260+0000","last_unstale":"2024-02-10T08:42:44.081132+0000","last_undegraded":"2024-02-10T08:42:44.081132+0000","last_fullsized":"2024-02-10T08:42:44.081132+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.451989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.451989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":333,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.083277+0000","last_change":"2024-02-10T08:41:22.620317+0000","last_active":"2024-02-10T08:42:45.083277+0000","last_peered":"2024-02-10T08:42:45.083277+0000","last_clean":"2024-02-10T08:42:45.083277+0000","last_became_active":"2024-02-10T08:37:10.185285+0000","last_became_peered":"2024-02-10T08:37:10.185285+0000","last_unstale":"2024-02-10T08:42:45.083277+0000","last_undegraded":"2024-02-10T08:42:45.083277+0000","last_fullsized":"2024-02-10T08:42:45.083277+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:22.620255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:22.620255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":372,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.086202+0000","last_change":"2024-02-10T08:42:43.399829+0000","last_active":"2024-02-10T08:42:45.086202+0000","last_peered":"2024-02-10T08:42:45.086202+0000","last_clean":"2024-02-10T08:42:45.086202+0000","last_became_active":"2024-02-10T08:37:10.217028+0000","last_became_peered":"2024-02-10T08:37:10.217028+0000","last_unstale":"2024-02-10T08:42:45.086202+0000","last_undegraded":"2024-02-10T08:42:45.086202+0000","last_fullsized":"2024-02-10T08:42:45.086202+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.399709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.399709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"257'7","reported_seq":428,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.083911+0000","last_change":"2024-02-10T08:42:03.646444+0000","last_active":"2024-02-10T08:42:44.083911+0000","last_peered":"2024-02-10T08:42:44.083911+0000","last_clean":"2024-02-10T08:42:44.083911+0000","last_became_active":"2024-02-10T08:33:20.118373+0000","last_became_peered":"2024-02-10T08:33:20.118373+0000","last_unstale":"2024-02-10T08:42:44.083911+0000","last_undegraded":"2024-02-10T08:42:44.083911+0000","last_fullsized":"2024-02-10T08:42:44.083911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":424,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.082616+0000","last_change":"2024-02-10T08:41:11.401522+0000","last_active":"2024-02-10T08:42:44.082616+0000","last_peered":"2024-02-10T08:42:44.082616+0000","last_clean":"2024-02-10T08:42:44.082616+0000","last_became_active":"2024-02-10T08:37:10.243172+0000","last_became_peered":"2024-02-10T08:37:10.243172+0000","last_unstale":"2024-02-10T08:42:44.082616+0000","last_undegraded":"2024-02-10T08:42:44.082616+0000","last_fullsized":"2024-02-10T08:42:44.082616+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.401433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.401433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":409,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075779+0000","last_change":"2024-02-10T08:42:35.160383+0000","last_active":"2024-02-10T08:42:44.075779+0000","last_peered":"2024-02-10T08:42:44.075779+0000","last_clean":"2024-02-10T08:42:44.075779+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:42:44.075779+0000","last_undegraded":"2024-02-10T08:42:44.075779+0000","last_fullsized":"2024-02-10T08:42:44.075779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":337,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.076646+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:42:44.076646+0000","last_peered":"2024-02-10T08:42:44.076646+0000","last_clean":"2024-02-10T08:42:44.076646+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:42:44.076646+0000","last_undegraded":"2024-02-10T08:42:44.076646+0000","last_fullsized":"2024-02-10T08:42:44.076646+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"63'9","reported_seq":446,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.081101+0000","last_change":"2024-02-10T08:40:37.664677+0000","last_active":"2024-02-10T08:42:44.081101+0000","last_peered":"2024-02-10T08:42:44.081101+0000","last_clean":"2024-02-10T08:42:44.081101+0000","last_became_active":"2024-02-10T08:34:41.695440+0000","last_became_peered":"2024-02-10T08:34:41.695440+0000","last_unstale":"2024-02-10T08:42:44.081101+0000","last_undegraded":"2024-02-10T08:42:44.081101+0000","last_fullsized":"2024-02-10T08:42:44.081101+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:40:37.664586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:40:37.664586+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":423,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097675+0000","last_change":"2024-02-10T08:39:39.822368+0000","last_active":"2024-02-10T08:42:45.097675+0000","last_peered":"2024-02-10T08:42:45.097675+0000","last_clean":"2024-02-10T08:42:45.097675+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:42:45.097675+0000","last_undegraded":"2024-02-10T08:42:45.097675+0000","last_fullsized":"2024-02-10T08:42:45.097675+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:39:39.822302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:39:39.822302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":412,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.088511+0000","last_change":"2024-02-10T08:41:55.669868+0000","last_active":"2024-02-10T08:42:44.088511+0000","last_peered":"2024-02-10T08:42:44.088511+0000","last_clean":"2024-02-10T08:42:44.088511+0000","last_became_active":"2024-02-10T08:33:24.116098+0000","last_became_peered":"2024-02-10T08:33:24.116098+0000","last_unstale":"2024-02-10T08:42:44.088511+0000","last_undegraded":"2024-02-10T08:42:44.088511+0000","last_fullsized":"2024-02-10T08:42:44.088511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":367,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.075198+0000","last_change":"2024-02-10T08:41:16.004246+0000","last_active":"2024-02-10T08:42:44.075198+0000","last_peered":"2024-02-10T08:42:44.075198+0000","last_clean":"2024-02-10T08:42:44.075198+0000","last_became_active":"2024-02-10T08:37:10.144659+0000","last_became_peered":"2024-02-10T08:37:10.144659+0000","last_unstale":"2024-02-10T08:42:44.075198+0000","last_undegraded":"2024-02-10T08:42:44.075198+0000","last_fullsized":"2024-02-10T08:42:44.075198+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"63'15","reported_seq":462,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097292+0000","last_change":"2024-02-10T08:41:19.653285+0000","last_active":"2024-02-10T08:42:45.097292+0000","last_peered":"2024-02-10T08:42:45.097292+0000","last_clean":"2024-02-10T08:42:45.097292+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:42:45.097292+0000","last_undegraded":"2024-02-10T08:42:45.097292+0000","last_fullsized":"2024-02-10T08:42:45.097292+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:41:19.653219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.653219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":328,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.084210+0000","last_change":"2024-02-10T08:40:42.336046+0000","last_active":"2024-02-10T08:42:44.084210+0000","last_peered":"2024-02-10T08:42:44.084210+0000","last_clean":"2024-02-10T08:42:44.084210+0000","last_became_active":"2024-02-10T08:37:10.111822+0000","last_became_peered":"2024-02-10T08:37:10.111822+0000","last_unstale":"2024-02-10T08:42:44.084210+0000","last_undegraded":"2024-02-10T08:42:44.084210+0000","last_fullsized":"2024-02-10T08:42:44.084210+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:42.335958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:40:42.335958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":408,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-02-10T08:42:45.097699+0000","last_change":"2024-02-10T08:40:24.826360+0000","last_active":"2024-02-10T08:42:45.097699+0000","last_peered":"2024-02-10T08:42:45.097699+0000","last_clean":"2024-02-10T08:42:45.097699+0000","last_became_active":"2024-02-10T08:33:24.126009+0000","last_became_peered":"2024-02-10T08:33:24.126009+0000","last_unstale":"2024-02-10T08:42:45.097699+0000","last_undegraded":"2024-02-10T08:42:45.097699+0000","last_fullsized":"2024-02-10T08:42:45.097699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:24.826225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:40:24.826225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":364,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-02-10T08:42:44.079620+0000","last_change":"2024-02-10T08:42:05.201384+0000","last_active":"2024-02-10T08:42:44.079620+0000","last_peered":"2024-02-10T08:42:44.079620+0000","last_clean":"2024-02-10T08:42:44.079620+0000","last_became_active":"2024-02-10T08:36:37.082024+0000","last_became_peered":"2024-02-10T08:36:37.082024+0000","last_unstale":"2024-02-10T08:42:44.079620+0000","last_undegraded":"2024-02-10T08:42:44.079620+0000","last_fullsized":"2024-02-10T08:42:44.079620+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":216,"num_pg":32,"stat_sum":{"num_bytes":16,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":48,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":7},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4001528,"num_objects":164,"num_object_clones":4,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":110,"num_read_kb":10,"num_write":607,"num_write_kb":4506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13979648,"data_stored":12004635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1906,"ondisk_log_size":1906,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":560,"num_read_kb":555,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299508,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308260,"kb_used_data":2844,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436892,"statfs":{"total":95995035648,"available":95679377408,"internally_reserved":0,"allocated":2912256,"data_stored":1149390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51900000000000002,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67900000000000005,"5min":3.8809999999999998,"15min":22.143000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.56599999999999995},"min":{"1min":0.33300000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71399999999999997,"5min":0.93100000000000005,"15min":8.7850000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":3.2389999999999999,"5min":3.8559999999999999,"15min":7.593},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.22500000000000001,"15min":0.19900000000000001},"max":{"1min":0.69499999999999995,"5min":1.0089999999999999,"15min":7.7759999999999998},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.497,"15min":0.55000000000000004},"min":{"1min":0.318,"5min":0.27700000000000002,"15min":0.27100000000000002},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":6.0300000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55800000000000005,"15min":0.65100000000000002},"min":{"1min":0.38,"5min":0.251,"15min":0.251},"max":{"1min":1.0469999999999999,"5min":3.8109999999999999,"15min":19.369},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.51800000000000002,"15min":0.57199999999999995},"min":{"1min":0.33500000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":6.9829999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.64000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.84299999999999997,"5min":3.903,"15min":22.102},"last":0.47199999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.62,"5min":0.79200000000000004,"15min":8.827},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.46100000000000002,"15min":0.48499999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.26200000000000001},"max":{"1min":1.903,"5min":1.903,"15min":13.646000000000001},"last":0.377}]},{"osd":5,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49399999999999999,"15min":0.52400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":0.81200000000000006,"5min":3.9300000000000002,"15min":19.326000000000001},"last":0.443},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":0.66100000000000003,"5min":0.77200000000000002,"15min":1.6220000000000001},"last":0.40899999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49199999999999999,"15min":0.56399999999999995},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.627,"5min":0.89100000000000001,"15min":22.687000000000001},"last":0.432},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.36499999999999999,"5min":0.30399999999999999,"15min":0.29699999999999999},"max":{"1min":0.66700000000000004,"5min":0.72599999999999998,"15min":1.7450000000000001},"last":0.41699999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561146,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307976,"kb_used_data":6656,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437176,"statfs":{"total":95995035648,"available":95679668224,"internally_reserved":0,"allocated":6815744,"data_stored":5149150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.63500000000000001,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.19800000000000001},"max":{"1min":0.68200000000000005,"5min":8.5250000000000004,"15min":8.5250000000000004},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.64600000000000002,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.65000000000000002,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.29999999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.57799999999999996,"15min":0.52600000000000002},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":8.3680000000000003,"15min":8.3680000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.56599999999999995,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.69499999999999995,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58999999999999997,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72299999999999998,"5min":4.798,"15min":4.798},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59699999999999998,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68999999999999995,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.439}]},{"osd":3,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.78100000000000003,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":16.823,"15min":16.823},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.51800000000000002},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.22500000000000001},"max":{"1min":0.67400000000000004,"5min":3.657,"15min":3.657},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45200000000000001,"15min":0.39900000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":0.71099999999999997,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.317}]},{"osd":5,"last update":"Sat Feb 10 08:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71199999999999997,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.373},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46700000000000003,"15min":0.436},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":2.1339999999999999,"15min":2.2029999999999998},"last":0.36799999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51200000000000001,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.33800000000000002,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.499},"min":{"1min":0.39700000000000002,"5min":0.32500000000000001,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.76200000000000001},"last":0.40400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304140,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441012,"statfs":{"total":95995035648,"available":95683596288,"internally_reserved":0,"allocated":2887680,"data_stored":1150419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.48299999999999998,"15min":0.40699999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.36799999999999999,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.214},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.63,"15min":0.45500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.45600000000000002,"5min":16.166,"15min":16.166},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.60899999999999999},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.38100000000000001,"5min":0.79600000000000004,"15min":34.985999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.52400000000000002},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.378,"5min":0.66800000000000004,"15min":21.614999999999998},"last":0.22700000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.73299999999999998,"15min":0.51700000000000002},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.39800000000000002,"5min":24.773,"15min":24.773},"last":0.246},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.69599999999999995,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.251},"max":{"1min":0.499,"5min":16.120999999999999,"15min":16.120999999999999},"last":0.26900000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.32500000000000001,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":16.074000000000002,"15min":16.074000000000002},"last":0.371},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63700000000000001,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.33500000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.55400000000000005,"15min":0.60699999999999998},"min":{"1min":0.37,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":9.5600000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55300000000000005,"15min":0.65800000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.31900000000000001},"max":{"1min":0.69599999999999995,"5min":0.876,"15min":7.8479999999999999},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.65700000000000003,"15min":0.60699999999999998},"min":{"1min":0.41799999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.80000000000000004,"5min":9.532,"15min":9.532},"last":0.379},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.73399999999999999,"15min":0.65500000000000003},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.45900000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":0.65500000000000003,"5min":0.88100000000000001,"15min":9.4640000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57499999999999996,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.41599999999999998,"15min":0.39400000000000002},"max":{"1min":0.66500000000000004,"5min":1.02,"15min":7.9199999999999999},"last":0.51100000000000001}]}]},{"osd":0,"up_from":0,"seq":34359738519,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411469,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300048,"kb_used_data":2824,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445104,"statfs":{"total":95995035648,"available":95687786496,"internally_reserved":0,"allocated":2891776,"data_stored":1149177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.187},"max":{"1min":0.40999999999999998,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.41799999999999998,"15min":0.45600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.373,"5min":8.1940000000000008,"15min":9.1440000000000001},"last":0.28499999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.505,"15min":0.47099999999999997},"min":{"1min":0.216,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.434,"5min":12.512,"15min":12.512},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.50700000000000001,"15min":0.46400000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":0.38300000000000001,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:41:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.48999999999999999,"15min":0.441},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47199999999999998,"5min":4.617,"15min":4.617},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.57099999999999995,"15min":0.48699999999999999},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.22600000000000001},"max":{"1min":0.503,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.32000000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.68899999999999995,"15min":0.67500000000000004},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76600000000000001,"5min":10.920999999999999,"15min":10.920999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.751,"15min":0.65300000000000002},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.309},"max":{"1min":1.014,"5min":12.292999999999999,"15min":12.292999999999999},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.63600000000000001,"15min":0.65600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69299999999999995,"5min":7.4859999999999998,"15min":9.2029999999999994},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.66100000000000003,"5min":7.6799999999999997,"15min":7.6799999999999997},"last":0.73999999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.67000000000000004,"15min":0.66200000000000003},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84099999999999997,"5min":4.3090000000000002,"15min":9.266},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.71299999999999997,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81999999999999995,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:42:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60999999999999999,"15min":0.68200000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76800000000000002,"5min":2.9809999999999999,"15min":9.2940000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72099999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"last":0.64000000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182537,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303944,"kb_used_data":6720,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441208,"statfs":{"total":95995035648,"available":95683796992,"internally_reserved":0,"allocated":6881280,"data_stored":5149282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.67000000000000004,"15min":0.50800000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.52100000000000002,"5min":11.550000000000001,"15min":14.744},"last":0.245},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.85299999999999998,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":0.504,"5min":11.808,"15min":11.808},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.747,"15min":0.56599999999999995},"min":{"1min":0.245,"5min":0.224,"15min":0.19700000000000001},"max":{"1min":0.435,"5min":10.444000000000001,"15min":16.634},"last":0.318},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.66900000000000004,"15min":0.56499999999999995},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.754,"5min":11.494999999999999,"15min":12.438000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.65300000000000002,"15min":0.58699999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.51000000000000001,"5min":11.516999999999999,"15min":16.536000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.85299999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.48399999999999999,"5min":17.722000000000001,"15min":23.294},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:42:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.67900000000000005},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32300000000000001},"max":{"1min":1.9370000000000001,"5min":2.589,"15min":12.371},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.79700000000000004,"15min":0.71899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.008,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57799999999999996,"15min":0.70699999999999996},"min":{"1min":0.373,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":3.6040000000000001,"15min":12.579000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.65600000000000003},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1970000000000001,"5min":4.9909999999999997,"15min":11.448},"last":0.60699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:42:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.749,"15min":0.69799999999999995},"min":{"1min":0.41099999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77900000000000003,"5min":10.381,"15min":16.091000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.69599999999999995,"15min":0.68600000000000005},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":11.472,"15min":16.475000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55300000000000005,"15min":0.65900000000000003},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":14.773999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.69699999999999995,"15min":0.73599999999999999},"min":{"1min":0.436,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.1339999999999999,"5min":6.1289999999999996,"15min":12.606999999999999},"last":0.65000000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986307,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312344,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432808,"statfs":{"total":95995035648,"available":95675195392,"internally_reserved":0,"allocated":2899968,"data_stored":1149441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54100000000000004,"15min":0.63100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.61399999999999999,"5min":4.1449999999999996,"15min":19.940999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53100000000000003,"15min":0.55800000000000005},"min":{"1min":0.33000000000000002,"5min":0.251,"15min":0.221},"max":{"1min":0.83599999999999997,"5min":3.8860000000000001,"15min":9.1760000000000002},"last":0.433}]},{"osd":1,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.54500000000000004,"15min":0.58399999999999996},"min":{"1min":0.29399999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.628,"5min":5.774,"15min":5.774},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.60299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.29399999999999998},"max":{"1min":0.88300000000000001,"5min":3.8620000000000001,"15min":8.6989999999999998},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.60399999999999998},"min":{"1min":0.30399999999999999,"5min":0.29399999999999998,"15min":0.251},"max":{"1min":0.80500000000000005,"5min":3.9289999999999998,"15min":6.2329999999999997},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.58099999999999996,"15min":0.60399999999999998},"min":{"1min":0.36599999999999999,"5min":0.28299999999999997,"15min":0.24299999999999999},"max":{"1min":0.70499999999999996,"5min":10.01,"15min":10.01},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.57899999999999996,"15min":0.65600000000000003},"min":{"1min":0.33000000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81399999999999995,"5min":8.3480000000000008,"15min":9.5909999999999993},"last":0.504},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57399999999999995,"15min":0.71699999999999997},"min":{"1min":0.38900000000000001,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.66300000000000003,"5min":4.21,"15min":25.27},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.752,"5min":6.8730000000000002,"15min":6.8730000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46999999999999997,"15min":0.42399999999999999},"min":{"1min":0.33000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.76400000000000001,"5min":3.8290000000000002,"15min":3.8290000000000002},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.78000000000000003,"5min":8.3119999999999994,"15min":13.342000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.51100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":13.753,"15min":13.753},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.627,"15min":0.51800000000000002},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":0.90400000000000003,"5min":12.835000000000001,"15min":12.835000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.67100000000000004,"15min":0.57899999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.76700000000000002,"5min":12.875999999999999,"15min":12.875999999999999},"last":0.58899999999999997}]}]},{"osd":5,"up_from":37,"seq":158913790079,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312100,"kb_used_data":6684,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433052,"statfs":{"total":95995035648,"available":95675445248,"internally_reserved":0,"allocated":6844416,"data_stored":5148973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46100000000000002,"15min":0.99099999999999999},"min":{"1min":0.24399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.68899999999999995,"5min":0.78200000000000003,"15min":107.551},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.499,"15min":0.96599999999999997},"min":{"1min":0.314,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":3.875,"5min":3.875,"15min":88.792000000000002},"last":0.69499999999999995}]},{"osd":1,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.55000000000000004,"15min":0.94399999999999995},"min":{"1min":0.29999999999999999,"5min":0.28399999999999997,"15min":0.27600000000000002},"max":{"1min":4.4349999999999996,"5min":4.4349999999999996,"15min":107.205},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.55100000000000005,"15min":0.83599999999999997},"min":{"1min":0.36199999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":72.272000000000006},"last":0.47399999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48599999999999999,"15min":0.95199999999999996},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":54.174999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.501,"15min":0.99399999999999999},"min":{"1min":0.35399999999999998,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":86.183000000000007},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.72399999999999998},"min":{"1min":0.34000000000000002,"5min":0.33200000000000002,"15min":0.30399999999999999},"max":{"1min":1.782,"5min":2.2669999999999999,"15min":44.395000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":1.2070000000000001},"min":{"1min":0.374,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":114.526},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Feb 10 08:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.68600000000000005},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":72.241},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.46300000000000002,"15min":0.54800000000000004},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.71799999999999997},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.86199999999999999,"15min":66.844999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.75},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72199999999999998,"5min":0.90700000000000003,"15min":72.257000000000005},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4000643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":4000559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4247552,"data_stored":4000591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":216,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:42:52.552 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T08:42:52.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:42:52.553 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:42:52.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd out 7 2024-02-10T08:42:52.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:52.668 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:53.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:53.047 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:53.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:53.148 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:53.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:53.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:53.322 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:53.343 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:53.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:53.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:53.479 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:53.499 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:53.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:53.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:53.630 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:54.380 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:54.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:54.474 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:42:54.474 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:42:54.474 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:42:54.474 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:42:54.474 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:42:54.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:42:54.516 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:54.714 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:42:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:54.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:42:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:53.455818+0000 mgr.y (mgr.14553) 407 : cluster [DBG] pgmap v665: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-10T08:42:54.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:53.455818+0000 mgr.y (mgr.14553) 407 : cluster [DBG] pgmap v665: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-10T08:42:54.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:53.455818+0000 mgr.y (mgr.14553) 407 : cluster [DBG] pgmap v665: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-10T08:42:54.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:54.850 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:54.872 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:54.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:54.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:55.076 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:55.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:55.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:42:55.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:55.207 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:55.227 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:55.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:55.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:55.361 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:42:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:42:55] "GET /metrics HTTP/1.1" 200 187910 "" "Prometheus/2.33.4" 2024-02-10T08:42:56.113 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:56.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:56.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:56.246 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:56.369 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:56.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:55.159785+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:56.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:55.456218+0000 mgr.y (mgr.14553) 408 : cluster [DBG] pgmap v666: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:56.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:55.159785+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:56.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:55.456218+0000 mgr.y (mgr.14553) 408 : cluster [DBG] pgmap v666: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:56.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:56.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:56.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:42:56.505 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:55.159785+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:42:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:55.456218+0000 mgr.y (mgr.14553) 408 : cluster [DBG] pgmap v666: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:42:57.855 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:42:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:57.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:42:57.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:42:57.991 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:58.012 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:42:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:58.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:42:58.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:58.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.429+0000 7fbdb04e7700 1 -- 172.21.15.107:0/4153091048 >> v1:172.21.15.107:6790/0 conn(0x7fbda80fda50 legacy=0x7fbda80fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.429+0000 7fbdb04e7700 1 -- 172.21.15.107:0/4153091048 shutdown_connections 2024-02-10T08:42:58.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.429+0000 7fbdb04e7700 1 -- 172.21.15.107:0/4153091048 >> 172.21.15.107:0/4153091048 conn(0x7fbda80f3640 msgr2=0x7fbda80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:58.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.429+0000 7fbdb04e7700 1 -- 172.21.15.107:0/4153091048 shutdown_connections 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.429+0000 7fbdb04e7700 1 -- 172.21.15.107:0/4153091048 wait complete. 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 librados: starting msgr at 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 librados: starting objecter 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 Processor -- start 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 -- start start 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 librados: setting wanted keys 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 librados: calling monclient init 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbda81194c0 con 0x7fbda80fda50 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbda811a660 con 0x7fbda8104330 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdb04e7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbda811b800 con 0x7fbda81007f0 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdaea84700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbda8104330 0x7fbda8117da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:51248/0 (socket says 172.21.15.107:51248) 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.430+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 learned_addr learned my addr 172.21.15.107:0/2925053852 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter ms_handle_connect 0x7fbda8104330 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter ms_handle_connect 0x7fbda80fda50 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter ms_handle_connect 0x7fbda81007f0 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174360836 0 0) 0x7fbda811a660 con 0x7fbda8104330 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd940038e0 con 0x7fbda8104330 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726912829 0 0) 0x7fbda81194c0 con 0x7fbda80fda50 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda811a660 con 0x7fbda80fda50 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085198009 0 0) 0x7fbda811b800 con 0x7fbda81007f0 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda81194c0 con 0x7fbda81007f0 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84115621 0 0) 0x7fbd940038e0 con 0x7fbda8104330 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbda811b800 con 0x7fbda8104330 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbda0001cd0 con 0x7fbda8104330 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753553444 0 0) 0x7fbda811a660 con 0x7fbda80fda50 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd940038e0 con 0x7fbda80fda50 2024-02-10T08:42:58.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177253198 0 0) 0x7fbda81194c0 con 0x7fbda81007f0 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbda811a660 con 0x7fbda81007f0 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbda4000fe0 con 0x7fbda80fda50 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbd98001620 con 0x7fbda81007f0 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849213494 0 0) 0x7fbda811b800 con 0x7fbda8104330 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6790/0 conn(0x7fbda81007f0 legacy=0x7fbda80fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6789/0 conn(0x7fbda80fda50 legacy=0x7fbda80fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda811ca00 con 0x7fbda8104330 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda81196b0 con 0x7fbda8104330 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbdb04e7700 10 client.48442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda8119ba0 con 0x7fbda8104330 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbdb04e7700 1 librados: init done 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbda00027e0 con 0x7fbda8104330 2024-02-10T08:42:58.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.431+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbda0005e70 con 0x7fbda8104330 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.432+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbd8c004f00 con 0x7fbda8104330 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbda0002340 con 0x7fbda8104330 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(449..449 src has 330..449) v4 ==== 8194+0+0 (unknown 308346033 0 0) 0x7fbda0079150 con 0x7fbda8104330 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 osd_map(449..449 src has 330..449) v4 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 3 client.48442.objecter handle_osd_map got epochs [449,449] > 0 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 3 client.48442.objecter handle_osd_map decoding full epoch 449 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 20 client.48442.objecter dump_active .. 0 homeless 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v449) v1 ==== 54+0+16 (unknown 1848550154 0 2420000192) 0x7fbda0079530 con 0x7fbda8104330 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.433+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd94063f20 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 1 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 1 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 0 1 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 4 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 10 client.48442.objecter _send_command 1 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c004950 con 0x7fbd8c004530 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c004530 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c004950 con 0x7fbd8c004530 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 1 = system:0 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 4 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.434+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 0 1 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 2 2024-02-10T08:42:58.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 2 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c0051a0 osd=0 3 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 0 2 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 4 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 10 client.48442.objecter _send_command 2 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.449+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c004d30 con 0x7fbd8c004530 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c004d30 con 0x7fbd8c004530 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 4 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 0 2 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 1 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 4 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 10 client.48442.objecter _send_command 3 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c00bef0 con 0x7fbd8c004150 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.451+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c004150 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.452+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c00bef0 con 0x7fbd8c004150 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.452+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.452+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 3 = system:0 2024-02-10T08:42:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.452+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 4 2024-02-10T08:42:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.452+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 1 3 2024-02-10T08:42:58.465 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:42:58.465 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 4 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 4 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003fe0 osd=1 3 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 1 4 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 4 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 10 client.48442.objecter _send_command 4 2024-02-10T08:42:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.470+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c0018d0 con 0x7fbd8c004150 2024-02-10T08:42:58.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c0018d0 con 0x7fbd8c004150 2024-02-10T08:42:58.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 4 2024-02-10T08:42:58.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 1 4 2024-02-10T08:42:58.474 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:42:58.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 5 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 5 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 2 5 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 4 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 10 client.48442.objecter _send_command 5 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c00dda0 con 0x7fbd8c00d1e0 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.473+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c00d1e0 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.474+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c00dda0 con 0x7fbd8c00d1e0 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.474+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.474+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 5 = system:0 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.474+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 4 2024-02-10T08:42:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.474+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 2 5 2024-02-10T08:42:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 6 2024-02-10T08:42:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 6 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003630 osd=2 3 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 2 6 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 4 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 10 client.48442.objecter _send_command 6 2024-02-10T08:42:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.490+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c0030a0 con 0x7fbd8c00d1e0 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.491+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c0030a0 con 0x7fbd8c00d1e0 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.491+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.491+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.491+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 4 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.491+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 2 6 2024-02-10T08:42:58.492 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 7 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 7 2024-02-10T08:42:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 3 7 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 4 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 10 client.48442.objecter _send_command 7 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c015970 con 0x7fbd8c011d40 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.492+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c011d40 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.493+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c015970 con 0x7fbd8c011d40 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.493+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.493+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 7 = system:0 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.493+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 4 2024-02-10T08:42:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.493+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 3 7 2024-02-10T08:42:58.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 8 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 8 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c003450 osd=3 3 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 3 8 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 4 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 10 client.48442.objecter _send_command 8 2024-02-10T08:42:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.507+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c015970 con 0x7fbd8c011d40 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.508+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c015970 con 0x7fbd8c011d40 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.508+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.508+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.508+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 4 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.508+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 3 8 2024-02-10T08:42:58.509 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 9 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 9 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 4 9 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 4 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 10 client.48442.objecter _send_command 9 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c00ce60 con 0x7fbd8c00c870 2024-02-10T08:42:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.509+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c00c870 2024-02-10T08:42:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.510+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c00ce60 con 0x7fbd8c00c870 2024-02-10T08:42:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.510+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:42:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.510+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 9 = system:0 2024-02-10T08:42:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.510+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 4 2024-02-10T08:42:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.510+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 4 9 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 10 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 10 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c015970 osd=4 3 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 4 10 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 4 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 10 client.48442.objecter _send_command 10 2024-02-10T08:42:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.526+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c00ce60 con 0x7fbd8c00c870 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.528+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c00ce60 con 0x7fbd8c00c870 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.528+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.528+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.528+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 4 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.528+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 4 10 2024-02-10T08:42:58.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 11 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 11 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 5 11 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 4 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 10 client.48442.objecter _send_command 11 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c017d60 con 0x7fbd8c0178b0 2024-02-10T08:42:58.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.529+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c0178b0 2024-02-10T08:42:58.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.530+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c017d60 con 0x7fbd8c0178b0 2024-02-10T08:42:58.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.530+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:42:58.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.530+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 11 = system:0 2024-02-10T08:42:58.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.530+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 4 2024-02-10T08:42:58.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.530+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 5 11 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 12 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 12 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c00ce60 osd=5 3 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 5 12 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 4 2024-02-10T08:42:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 10 client.48442.objecter _send_command 12 2024-02-10T08:42:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.544+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c017d60 con 0x7fbd8c0178b0 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.546+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c017d60 con 0x7fbd8c0178b0 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.546+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.546+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.546+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 4 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.546+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 5 12 2024-02-10T08:42:58.547 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 13 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 13 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 6 13 2024-02-10T08:42:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 4 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 10 client.48442.objecter _send_command 13 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c022a80 con 0x7fbd8c01ef70 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.547+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c01ef70 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.548+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c022a80 con 0x7fbd8c01ef70 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.548+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.548+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 13 = system:0 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.548+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 4 2024-02-10T08:42:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.548+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 6 13 2024-02-10T08:42:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 14 2024-02-10T08:42:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 14 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c017d60 osd=6 3 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 6 14 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 4 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 10 client.48442.objecter _send_command 14 2024-02-10T08:42:58.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.562+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c022a80 con 0x7fbd8c01ef70 2024-02-10T08:42:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c022a80 con 0x7fbd8c01ef70 2024-02-10T08:42:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 4 2024-02-10T08:42:58.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 6 14 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 15 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 15 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 7 15 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 4 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 10 client.48442.objecter _send_command 15 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd8c026dd0 con 0x7fbd8c023100 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.564+0000 7fbd9f7fe700 10 client.48442.objecter ms_handle_connect 0x7fbd8c023100 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.565+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd8c026dd0 con 0x7fbd8c023100 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.565+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.565+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 15 = system:0 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.565+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 4 2024-02-10T08:42:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.565+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 7 15 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 10 client.48442.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign -1 16 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 20 client.48442.objecter _get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_remove -1 16 2024-02-10T08:42:58.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 20 client.48442.objecter get_session s=0x7fbd8c022a80 osd=7 3 2024-02-10T08:42:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 15 client.48442.objecter _session_command_op_assign 7 16 2024-02-10T08:42:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 4 2024-02-10T08:42:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 10 client.48442.objecter _send_command 16 2024-02-10T08:42:58.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.579+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbd8c028580 con 0x7fbd8c023100 2024-02-10T08:42:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbd9f7fe700 1 -- 172.21.15.107:0/2925053852 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fbd8c028580 con 0x7fbd8c023100 2024-02-10T08:42:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbd9f7fe700 10 client.48442.objecter ms_dispatch 0x7fbda8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:42:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbd9f7fe700 10 client.48442.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbd9f7fe700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 4 2024-02-10T08:42:58.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbd9f7fe700 15 client.48442.objecter _session_command_op_remove 7 16 2024-02-10T08:42:58.582 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 librados: watch_flush enter 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 librados: watch_flush exit 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.0 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6801/3321596972 conn(0x7fbd8c004530 legacy=0x7fbd8c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c0051a0 osd=0 1 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.1 2024-02-10T08:42:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6805/133516126 conn(0x7fbd8c004150 legacy=0x7fbd8c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003fe0 osd=1 1 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.2 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6809/3575881728 conn(0x7fbd8c00d1e0 legacy=0x7fbd8c00d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003630 osd=2 1 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.3 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6813/2871825691 conn(0x7fbd8c011d40 legacy=0x7fbd8c014150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c003450 osd=3 1 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.4 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.195:6800/2499671285 conn(0x7fbd8c00c870 legacy=0x7fbd8c015b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c015970 osd=4 1 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.5 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.195:6804/4178321486 conn(0x7fbd8c0178b0 legacy=0x7fbd8c01d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c00ce60 osd=5 1 2024-02-10T08:42:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.6 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.195:6808/1417479017 conn(0x7fbd8c01ef70 legacy=0x7fbd8c021360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c017d60 osd=6 1 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter close_session for osd.7 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.195:6812/1934176928 conn(0x7fbd8c023100 legacy=0x7fbd8c025550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 reap_dead start 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter put_session s=0x7fbd8c022a80 osd=7 1 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 20 client.48442.objecter shutdown clearing up homeless session... 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 10 client.48442.objecter successfully canceled tick 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 reap_dead start 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 reap_dead start 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 reap_dead start 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdaea84700 1 -- 172.21.15.107:0/2925053852 reap_dead start 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbd94063f20 legacy=0x7fbd940663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> v1:172.21.15.195:6789/0 conn(0x7fbda8104330 legacy=0x7fbda8117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 shutdown_connections 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 >> 172.21.15.107:0/2925053852 conn(0x7fbda80f3640 msgr2=0x7fbda81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 shutdown_connections 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.581+0000 7fbdb04e7700 1 -- 172.21.15.107:0/2925053852 wait complete. 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.582+0000 7fbdb04e7700 1 librados: shutdown 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.344+0000 7f1e7603e700 1 -- 172.21.15.107:0/961077319 >> v1:172.21.15.107:6790/0 conn(0x7f1e70104350 legacy=0x7f1e701067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.344+0000 7f1e7603e700 1 -- 172.21.15.107:0/961077319 shutdown_connections 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.344+0000 7f1e7603e700 1 -- 172.21.15.107:0/961077319 >> 172.21.15.107:0/961077319 conn(0x7f1e700f2510 msgr2=0x7f1e700f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.344+0000 7f1e7603e700 1 -- 172.21.15.107:0/961077319 shutdown_connections 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.344+0000 7f1e7603e700 1 -- 172.21.15.107:0/961077319 wait complete. 2024-02-10T08:42:58.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 librados: starting msgr at 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 librados: starting objecter 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 Processor -- start 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 -- start start 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 librados: setting wanted keys 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 librados: calling monclient init 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e70102bd0 con 0x7f1e70104350 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e70102da0 con 0x7f1e700f89c0 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.345+0000 7f1e7603e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e70102f90 con 0x7f1e700f9570 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6effd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1e700f9570 0x7f1e70114da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42570/0 (socket says 172.21.15.107:42570) 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6ffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e70104350 0x7f1e701184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53320/0 (socket says 172.21.15.107:53320) 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6effd700 1 -- 172.21.15.107:0/4221255318 learned_addr learned my addr 172.21.15.107:0/4221255318 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter ms_handle_connect 0x7f1e700f9570 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter ms_handle_connect 0x7f1e70104350 2024-02-10T08:42:58.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719701059 0 0) 0x7f1e70102bd0 con 0x7f1e70104350 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e580038e0 con 0x7f1e70104350 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter ms_handle_connect 0x7f1e700f89c0 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809870140 0 0) 0x7f1e70102f90 con 0x7f1e700f9570 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e70102bd0 con 0x7f1e700f9570 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404404233 0 0) 0x7f1e70102da0 con 0x7f1e700f89c0 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e70102f90 con 0x7f1e700f89c0 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374337625 0 0) 0x7f1e70102bd0 con 0x7f1e700f9570 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e70102da0 con 0x7f1e700f9570 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e640018c0 con 0x7f1e700f9570 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066285003 0 0) 0x7f1e580038e0 con 0x7f1e70104350 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e70102bd0 con 0x7f1e70104350 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e68002f80 con 0x7f1e70104350 2024-02-10T08:42:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1507413652 0 0) 0x7f1e70102da0 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 >> v1:172.21.15.195:6789/0 conn(0x7f1e700f89c0 legacy=0x7f1e70102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 >> v1:172.21.15.107:6789/0 conn(0x7f1e70104350 legacy=0x7f1e701184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7011cc60 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e70119c50 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e7603e700 10 client.53720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7011a0c0 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e7603e700 1 librados: init done 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1e64002420 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.346+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e64005a00 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.347+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c004f20 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.348+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1e64002bc0 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.348+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(449..449 src has 330..449) v4 ==== 8194+0+0 (unknown 308346033 0 0) 0x7f1e64078e00 con 0x7f1e700f9570 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.348+0000 7f1e6cff9700 10 client.53720.objecter ms_dispatch 0x7f1e70001550 osd_map(449..449 src has 330..449) v4 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.348+0000 7f1e6cff9700 3 client.53720.objecter handle_osd_map got epochs [449,449] > 0 2024-02-10T08:42:58.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.348+0000 7f1e6cff9700 3 client.53720.objecter handle_osd_map decoding full epoch 449 2024-02-10T08:42:58.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.349+0000 7f1e6cff9700 20 client.53720.objecter dump_active .. 0 homeless 2024-02-10T08:42:58.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.349+0000 7f1e6cff9700 10 client.53720.objecter ms_handle_connect 0x7f1e58063c10 2024-02-10T08:42:58.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.352+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1e6404c170 con 0x7f1e700f9570 2024-02-10T08:42:58.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.538+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1e5c0051d0 con 0x7f1e700f9570 2024-02-10T08:42:58.596 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:58.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:58.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:58.730 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:57.457045+0000 mgr.y (mgr.14553) 409 : cluster [DBG] pgmap v667: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 439 B/s wr, 2 op/s 2024-02-10T08:42:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:58.433647+0000 mon.b (mon.1) 603 : audit [DBG] from='client.? v1:172.21.15.107:0/2925053852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.820+0000 7f34c13d3700 1 -- 172.21.15.107:0/1998383351 >> v1:172.21.15.107:6790/0 conn(0x7f34bc0ff710 legacy=0x7f34bc101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.820+0000 7f34c13d3700 1 -- 172.21.15.107:0/1998383351 shutdown_connections 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.820+0000 7f34c13d3700 1 -- 172.21.15.107:0/1998383351 >> 172.21.15.107:0/1998383351 conn(0x7f34bc0f2510 msgr2=0x7f34bc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.820+0000 7f34c13d3700 1 -- 172.21.15.107:0/1998383351 shutdown_connections 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.820+0000 7f34c13d3700 1 -- 172.21.15.107:0/1998383351 wait complete. 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 librados: starting msgr at 2024-02-10T08:42:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 librados: starting objecter 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 Processor -- start 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 -- start start 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 librados: setting wanted keys 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 librados: calling monclient init 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f34bc0fbfe0 con 0x7f34bc0fc910 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f34bc0fc1b0 con 0x7f34bc103250 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34c13d3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f34bc11bb20 con 0x7f34bc0ff710 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34baffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f34bc0fc910 0x7f34bc0fb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53348/0 (socket says 172.21.15.107:53348) 2024-02-10T08:42:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.821+0000 7f34baffd700 1 -- 172.21.15.107:0/3199507182 learned_addr learned my addr 172.21.15.107:0/3199507182 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter ms_handle_connect 0x7f34bc0ff710 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter ms_handle_connect 0x7f34bc0fc910 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391076506 0 0) 0x7f34bc11bb20 con 0x7f34bc0ff710 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a40038e0 con 0x7f34bc0ff710 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter ms_handle_connect 0x7f34bc103250 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474158371 0 0) 0x7f34bc0fbfe0 con 0x7f34bc0fc910 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc11bb20 con 0x7f34bc0fc910 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268362786 0 0) 0x7f34bc0fc1b0 con 0x7f34bc103250 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc0fbfe0 con 0x7f34bc103250 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178447003 0 0) 0x7f34a40038e0 con 0x7f34bc0ff710 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34bc0fc1b0 con 0x7f34bc0ff710 2024-02-10T08:42:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f34b0002c10 con 0x7f34bc0ff710 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 319236456 0 0) 0x7f34bc11bb20 con 0x7f34bc0fc910 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34a40038e0 con 0x7f34bc0fc910 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f34ac0019f0 con 0x7f34bc0fc910 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268898351 0 0) 0x7f34bc0fbfe0 con 0x7f34bc103250 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34bc11bb20 con 0x7f34bc103250 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f34b4001b80 con 0x7f34bc103250 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2714030374 0 0) 0x7f34bc0fc1b0 con 0x7f34bc0ff710 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 >> v1:172.21.15.195:6789/0 conn(0x7f34bc103250 legacy=0x7f34bc1183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 >> v1:172.21.15.107:6789/0 conn(0x7f34bc0fc910 legacy=0x7f34bc0fb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc11ccc0 con 0x7f34bc0ff710 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc11bd10 con 0x7f34bc0ff710 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34c13d3700 10 client.53732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc11c180 con 0x7f34bc0ff710 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34c13d3700 1 librados: init done 2024-02-10T08:42:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f34b00019d0 con 0x7f34bc0ff710 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.822+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f34b00035b0 con 0x7f34bc0ff710 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f34b0001e00 con 0x7f34bc0ff710 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8004f20 con 0x7f34bc0ff710 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(449..449 src has 330..449) v4 ==== 8194+0+0 (unknown 308346033 0 0) 0x7f34b0076780 con 0x7f34bc0ff710 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 10 client.53732.objecter ms_dispatch 0x7f34bc001550 osd_map(449..449 src has 330..449) v4 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 3 client.53732.objecter handle_osd_map got epochs [449,449] > 0 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 3 client.53732.objecter handle_osd_map decoding full epoch 449 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 20 client.53732.objecter dump_active .. 0 homeless 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.824+0000 7f34a3fff700 10 client.53732.objecter ms_handle_connect 0x7f34a4063c20 2024-02-10T08:42:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:58.828+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f34b0049bd0 con 0x7f34bc0ff710 2024-02-10T08:42:58.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:57.457045+0000 mgr.y (mgr.14553) 409 : cluster [DBG] pgmap v667: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 439 B/s wr, 2 op/s 2024-02-10T08:42:58.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:58.433647+0000 mon.b (mon.1) 603 : audit [DBG] from='client.? v1:172.21.15.107:0/2925053852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:58.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:57.457045+0000 mgr.y (mgr.14553) 409 : cluster [DBG] pgmap v667: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 439 B/s wr, 2 op/s 2024-02-10T08:42:58.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:58.433647+0000 mon.b (mon.1) 603 : audit [DBG] from='client.? v1:172.21.15.107:0/2925053852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:42:59.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.004+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd out", "ids": ["7"]} v 0) v1 -- 0x7f34a80051d0 con 0x7f34bc0ff710 2024-02-10T08:42:59.078 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:59.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:42:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:42:59.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:42:59.005+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e449 definitely_dead 0 2024-02-10T08:42:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:42:59.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:42:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:42:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:42:59.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.529+0000 7f34a3fff700 1 -- 172.21.15.107:0/3199507182 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["7"]}]=0 marked out osd.7. v450) v1 ==== 87+0+0 (unknown 1143223574 0 0) 0x7f34b0048040 con 0x7f34bc0ff710 2024-02-10T08:42:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.531+0000 7f1e6cff9700 1 -- 172.21.15.107:0/4221255318 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v450) v1 ==== 97+0+0 (unknown 2608496118 0 0) 0x7f1e6404a580 con 0x7f1e700f9570 2024-02-10T08:42:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 10 librados: watch_flush enter 2024-02-10T08:42:59.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 10 librados: watch_flush exit 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 20 client.53720.objecter shutdown clearing up homeless session... 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 10 client.53720.objecter successfully canceled tick 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.532+0000 7f34c13d3700 10 librados: watch_flush enter 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f34c13d3700 10 librados: watch_flush exit 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f34c13d3700 20 client.53732.objecter shutdown clearing up homeless session... 2024-02-10T08:42:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f34c13d3700 10 client.53732.objecter successfully canceled tick 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1e58063c10 legacy=0x7f1e580660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.534+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 >> v1:172.21.15.107:6790/0 conn(0x7f1e700f9570 legacy=0x7f1e70114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.535+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 shutdown_connections 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.535+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 >> 172.21.15.107:0/4221255318 conn(0x7f1e700f2510 msgr2=0x7f1e701077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.535+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 shutdown_connections 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.535+0000 7f1e7603e700 1 -- 172.21.15.107:0/4221255318 wait complete. 2024-02-10T08:42:59.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.535+0000 7f1e7603e700 1 librados: shutdown 2024-02-10T08:42:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.537+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 >> v1:172.21.15.107:6800/2634842009 conn(0x7f34a4063c20 legacy=0x7f34a40660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:59.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.537+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 >> v1:172.21.15.107:6790/0 conn(0x7f34bc0ff710 legacy=0x7f34bc114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:42:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.538+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 shutdown_connections 2024-02-10T08:42:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.538+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 >> 172.21.15.107:0/3199507182 conn(0x7f34bc0f2510 msgr2=0x7f34bc1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:42:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.538+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 shutdown_connections 2024-02-10T08:42:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.538+0000 7f34c13d3700 1 -- 172.21.15.107:0/3199507182 wait complete. 2024-02-10T08:42:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:42:59.538+0000 7f34c13d3700 1 librados: shutdown 2024-02-10T08:42:59.540 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.7. 2024-02-10T08:42:59.541 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:58.539495+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.107:0/4221255318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:58.539775+0000 mon.a (mon.0) 2554 : audit [INF] from='client.53720 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:59.005808+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.107:0/3199507182' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:59.006115+0000 mon.a (mon.0) 2555 : audit [INF] from='client.53732 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:59.006195+0000 mon.a (mon.0) 2556 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:58.539495+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.107:0/4221255318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:58.539775+0000 mon.a (mon.0) 2554 : audit [INF] from='client.53720 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:59.005808+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.107:0/3199507182' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:59.006115+0000 mon.a (mon.0) 2555 : audit [INF] from='client.53732 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:42:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:59.006195+0000 mon.a (mon.0) 2556 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-10T08:42:59.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:42:59.619 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:42:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:42:59.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:59.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:58.539495+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.107:0/4221255318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:58.539775+0000 mon.a (mon.0) 2554 : audit [INF] from='client.53720 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:42:59.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:59.005808+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.107:0/3199507182' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:59.006115+0000 mon.a (mon.0) 2555 : audit [INF] from='client.53732 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-10T08:42:59.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:42:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:59.006195+0000 mon.a (mon.0) 2556 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-10T08:42:59.770 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:59.789 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:42:59.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:42:59.920 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:42:59.938 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:42:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:42:59.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:00.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:00.069 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:00.317 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:00.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:00.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:00.447 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:00.513 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:00.646 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:59.457442+0000 mgr.y (mgr.14553) 410 : cluster [DBG] pgmap v668: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 397 B/s wr, 1 op/s 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:59.511897+0000 mon.a (mon.0) 2557 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:59.519837+0000 mon.a (mon.0) 2558 : audit [INF] from='client.53720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:42:59.519910+0000 mon.a (mon.0) 2559 : audit [INF] from='client.53732 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:42:59.519961+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e450: 8 total, 8 up, 6 in 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:59.457442+0000 mgr.y (mgr.14553) 410 : cluster [DBG] pgmap v668: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 397 B/s wr, 1 op/s 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:59.511897+0000 mon.a (mon.0) 2557 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:59.519837+0000 mon.a (mon.0) 2558 : audit [INF] from='client.53720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:42:59.519910+0000 mon.a (mon.0) 2559 : audit [INF] from='client.53732 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-10T08:43:00.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:42:59.519961+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e450: 8 total, 8 up, 6 in 2024-02-10T08:43:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:00.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:00.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:00.773 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:59.457442+0000 mgr.y (mgr.14553) 410 : cluster [DBG] pgmap v668: 218 pgs: 218 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 397 B/s wr, 1 op/s 2024-02-10T08:43:00.773 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:59.511897+0000 mon.a (mon.0) 2557 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:00.773 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:59.519837+0000 mon.a (mon.0) 2558 : audit [INF] from='client.53720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:00.773 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:42:59.519910+0000 mon.a (mon.0) 2559 : audit [INF] from='client.53732 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-10T08:43:00.773 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:42:59.519961+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e450: 8 total, 8 up, 6 in 2024-02-10T08:43:00.775 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:00.798 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:00.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:00.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:00.932 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:00.957 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:00.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:01.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:01.091 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:01.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:01.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:01.265 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:01.284 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:01.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:01.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:01.414 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:01.433 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:01.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:01.530 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:01.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:01.720 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:01.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:01.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:00.522873+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e451: 8 total, 8 up, 6 in 2024-02-10T08:43:01.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:00.537296+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:01.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:00.537519+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:01.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:01.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:00.522873+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e451: 8 total, 8 up, 6 in 2024-02-10T08:43:01.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:00.537296+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:01.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:00.537519+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:01.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:00.522873+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e451: 8 total, 8 up, 6 in 2024-02-10T08:43:01.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:00.537296+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:01.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:00.537519+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:01.853 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:01.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:43:02.702 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:02.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:02.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:01.458402+0000 mgr.y (mgr.14553) 411 : cluster [DBG] pgmap v671: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 167 B/s, 9 objects/s recovering 2024-02-10T08:43:02.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:01.500872+0000 mgr.y (mgr.14553) 412 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:01.522610+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e452: 8 total, 8 up, 6 in 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:01.525897+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 124'10 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:01.526414+0000 osd.2 (osd.2) 175 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 108'10 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:02.047789+0000 mon.a (mon.0) 2563 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:02.047817+0000 mon.a (mon.0) 2564 : cluster [WRN] Health check failed: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:01.458402+0000 mgr.y (mgr.14553) 411 : cluster [DBG] pgmap v671: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 167 B/s, 9 objects/s recovering 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:01.500872+0000 mgr.y (mgr.14553) 412 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:02.707 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:01.522610+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e452: 8 total, 8 up, 6 in 2024-02-10T08:43:02.708 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:01.525897+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 124'10 2024-02-10T08:43:02.708 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:01.526414+0000 osd.2 (osd.2) 175 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 108'10 2024-02-10T08:43:02.708 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:02.047789+0000 mon.a (mon.0) 2563 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:02.708 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:02.047817+0000 mon.a (mon.0) 2564 : cluster [WRN] Health check failed: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:02.709 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:02.726 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:02.726 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:01.458402+0000 mgr.y (mgr.14553) 411 : cluster [DBG] pgmap v671: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 167 B/s, 9 objects/s recovering 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:01.500872+0000 mgr.y (mgr.14553) 412 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:01.522610+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e452: 8 total, 8 up, 6 in 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:01.525897+0000 osd.6 (osd.6) 193 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 124'10 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:01.526414+0000 osd.2 (osd.2) 175 : cluster [DBG] 5.d starting backfill to osd.3 from (0'0,0'0] MAX to 108'10 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:02.047789+0000 mon.a (mon.0) 2563 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:02.047817+0000 mon.a (mon.0) 2564 : cluster [WRN] Health check failed: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:02.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:02.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:02.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:02.981 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:03.001 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:03.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:03.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:03.130 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:03.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:43:03.149 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:03.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:03.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:03.278 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:03.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:03.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:03.427 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:03.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:02.536453+0000 mon.a (mon.0) 2565 : cluster [DBG] osdmap e453: 8 total, 8 up, 6 in 2024-02-10T08:43:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:02.536453+0000 mon.a (mon.0) 2565 : cluster [DBG] osdmap e453: 8 total, 8 up, 6 in 2024-02-10T08:43:03.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:43:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:02.536453+0000 mon.a (mon.0) 2565 : cluster [DBG] osdmap e453: 8 total, 8 up, 6 in 2024-02-10T08:43:04.276 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:04.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:04.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:04.412 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:04.433 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:04.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:04.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:03.458861+0000 mgr.y (mgr.14553) 413 : cluster [DBG] pgmap v674: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 5 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 334 B/s, 19 objects/s recovering 2024-02-10T08:43:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:03.543315+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e454: 8 total, 8 up, 6 in 2024-02-10T08:43:04.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:03.458861+0000 mgr.y (mgr.14553) 413 : cluster [DBG] pgmap v674: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 5 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 334 B/s, 19 objects/s recovering 2024-02-10T08:43:04.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:03.543315+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e454: 8 total, 8 up, 6 in 2024-02-10T08:43:04.802 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:03.458861+0000 mgr.y (mgr.14553) 413 : cluster [DBG] pgmap v674: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 5 op/s; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%); 334 B/s, 19 objects/s recovering 2024-02-10T08:43:04.802 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:03.543315+0000 mon.a (mon.0) 2566 : cluster [DBG] osdmap e454: 8 total, 8 up, 6 in 2024-02-10T08:43:05.611 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:05.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:05.615 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:05.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:05.745 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:05.767 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:05.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:05.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:04.547953+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e455: 8 total, 8 up, 6 in 2024-02-10T08:43:05.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:04.547953+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e455: 8 total, 8 up, 6 in 2024-02-10T08:43:05.842 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:05] "GET /metrics HTTP/1.1" 200 187910 "" "Prometheus/2.33.4" 2024-02-10T08:43:05.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:05.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:04.547953+0000 mon.a (mon.0) 2567 : cluster [DBG] osdmap e455: 8 total, 8 up, 6 in 2024-02-10T08:43:05.900 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:05.920 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:05.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:06.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:06.054 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:06.074 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:06.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:06.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:06.205 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:06.223 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:06.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:06.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:06.358 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:06.379 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:06.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:06.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:05.459245+0000 mgr.y (mgr.14553) 414 : cluster [DBG] pgmap v677: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%) 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:05.566290+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e456: 8 total, 8 up, 6 in 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:06.047906+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e457: 8 total, 8 up, 6 in 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:06.097141+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:06.097457+0000 mon.a (mon.0) 2570 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:06.097907+0000 mon.b (mon.1) 605 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:06.098090+0000 mon.a (mon.0) 2571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:05.459245+0000 mgr.y (mgr.14553) 414 : cluster [DBG] pgmap v677: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%) 2024-02-10T08:43:06.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:05.566290+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e456: 8 total, 8 up, 6 in 2024-02-10T08:43:06.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:06.047906+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e457: 8 total, 8 up, 6 in 2024-02-10T08:43:06.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:06.097141+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:06.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:06.097457+0000 mon.a (mon.0) 2570 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:06.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:06.097907+0000 mon.b (mon.1) 605 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:06.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:06.098090+0000 mon.a (mon.0) 2571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:05.459245+0000 mgr.y (mgr.14553) 414 : cluster [DBG] pgmap v677: 218 pgs: 6 remapped+peering, 1 active+recovery_wait, 6 active+recovering, 13 active+recovery_wait+degraded, 192 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 126/1083 objects degraded (11.634%); 18/1083 objects misplaced (1.662%) 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:05.566290+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e456: 8 total, 8 up, 6 in 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:06.047906+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e457: 8 total, 8 up, 6 in 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:06.097141+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:06.097457+0000 mon.a (mon.0) 2570 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:06.097907+0000 mon.b (mon.1) 605 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:07.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:06.098090+0000 mon.a (mon.0) 2571 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-69"}]: dispatch 2024-02-10T08:43:07.152 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:43:07.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:43:07.152 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-02-10T08:43:07.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd in 7 2024-02-10T08:43:07.259 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:07.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:07.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:07.395 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:07.417 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:07.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 -- 172.21.15.107:0/3238526996 >> v1:172.21.15.107:6790/0 conn(0x7f6a780ff720 legacy=0x7f6a78101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 -- 172.21.15.107:0/3238526996 shutdown_connections 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 -- 172.21.15.107:0/3238526996 >> 172.21.15.107:0/3238526996 conn(0x7f6a780f2540 msgr2=0x7f6a780f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 -- 172.21.15.107:0/3238526996 shutdown_connections 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 -- 172.21.15.107:0/3238526996 wait complete. 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 librados: starting msgr at 2024-02-10T08:43:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 librados: starting objecter 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.839+0000 7f6a7d852700 1 Processor -- start 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 -- start start 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 librados: setting wanted keys 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 librados: calling monclient init 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a78119580 con 0x7f6a780ff720 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a7811a760 con 0x7f6a78103260 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a7d852700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a7811b960 con 0x7f6a780fc920 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a767fc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6a780ff720 0x7f6a780612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46278/0 (socket says 172.21.15.107:46278) 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a767fc700 1 -- 172.21.15.107:0/2044710760 learned_addr learned my addr 172.21.15.107:0/2044710760 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter ms_handle_connect 0x7f6a780ff720 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter ms_handle_connect 0x7f6a780fc920 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter ms_handle_connect 0x7f6a78103260 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709107003 0 0) 0x7f6a78119580 con 0x7f6a780ff720 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a600038e0 con 0x7f6a780ff720 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414424737 0 0) 0x7f6a7811b960 con 0x7f6a780fc920 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78119580 con 0x7f6a780fc920 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696440903 0 0) 0x7f6a7811a760 con 0x7f6a78103260 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7811b960 con 0x7f6a78103260 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338108357 0 0) 0x7f6a78119580 con 0x7f6a780fc920 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a7811a760 con 0x7f6a780fc920 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159243610 0 0) 0x7f6a600038e0 con 0x7f6a780ff720 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a78119580 con 0x7f6a780ff720 2024-02-10T08:43:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6a6c001920 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6a68002d60 con 0x7f6a780ff720 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973478982 0 0) 0x7f6a7811b960 con 0x7f6a78103260 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a600038e0 con 0x7f6a78103260 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6a70001b80 con 0x7f6a78103260 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.840+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 894615054 0 0) 0x7f6a7811a760 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.195:6789/0 conn(0x7f6a78103260 legacy=0x7f6a78117ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6789/0 conn(0x7f6a780ff720 legacy=0x7f6a780612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7811cb60 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7811bb50 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a7d852700 10 client.53738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7811c040 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a7d852700 1 librados: init done 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6a6c002250 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6a6c003310 con 0x7f6a780fc920 2024-02-10T08:43:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.841+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6a64004f00 con 0x7f6a780fc920 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6a6c003510 con 0x7f6a780fc920 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(457..457 src has 330..457) v4 ==== 8107+0+0 (unknown 2668703221 0 0) 0x7f6a6c076510 con 0x7f6a780fc920 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 osd_map(457..457 src has 330..457) v4 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 3 client.53738.objecter handle_osd_map got epochs [457,457] > 0 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 3 client.53738.objecter handle_osd_map decoding full epoch 457 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 20 client.53738.objecter dump_active .. 0 homeless 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v457) v1 ==== 54+0+16 (unknown 960697922 0 2420000192) 0x7f6a6c0788b0 con 0x7f6a780fc920 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.842+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a60063cb0 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 10 client.53738.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 1 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 1 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 0 1 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 4 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 10 client.53738.objecter _send_command 1 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64004950 con 0x7f6a64004530 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a64004530 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64004950 con 0x7f6a64004530 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 1 = system:0 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 4 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.843+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 0 1 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 10 client.53738.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 2 2024-02-10T08:43:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 2 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a640051a0 osd=0 3 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 0 2 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 4 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 10 client.53738.objecter _send_command 2 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.858+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a64004d30 con 0x7f6a64004530 2024-02-10T08:43:07.500 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a64004d30 con 0x7f6a64004530 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 4 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 0 2 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 10 client.53738.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 3 2024-02-10T08:43:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 1 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 4 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 10 client.53738.objecter _send_command 3 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a6400c000 con 0x7f6a64004150 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.860+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a64004150 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.861+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a6400c000 con 0x7f6a64004150 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.861+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.861+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 3 = system:0 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.861+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 4 2024-02-10T08:43:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.861+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 1 3 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 10 client.53738.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 4 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 4 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003fe0 osd=1 3 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 1 4 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 4 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 10 client.53738.objecter _send_command 4 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.875+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a64001850 con 0x7f6a64004150 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a64001850 con 0x7f6a64004150 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 4 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 1 4 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 10 client.53738.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 5 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 5 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 2 5 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 4 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 10 client.53738.objecter _send_command 5 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64003e70 con 0x7f6a6400d2f0 2024-02-10T08:43:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.877+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a6400d2f0 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.878+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64003e70 con 0x7f6a6400d2f0 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.878+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.878+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 5 = system:0 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.878+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 4 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.878+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 2 5 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 10 client.53738.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 6 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 6 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003630 osd=2 3 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 2 6 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 4 2024-02-10T08:43:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 10 client.53738.objecter _send_command 6 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.892+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a6400e750 con 0x7f6a6400d2f0 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a6400e750 con 0x7f6a6400d2f0 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 4 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 2 6 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 10 client.53738.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 7 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 7 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 3 7 2024-02-10T08:43:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 4 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 10 client.53738.objecter _send_command 7 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64015b60 con 0x7f6a64011e80 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.894+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a64011e80 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.895+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64015b60 con 0x7f6a64011e80 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.895+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.895+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 7 = system:0 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.895+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 4 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.895+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 3 7 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 10 client.53738.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 8 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 8 2024-02-10T08:43:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64003470 osd=3 3 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 3 8 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 4 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 10 client.53738.objecter _send_command 8 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.909+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a6400e1c0 con 0x7f6a64011e80 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:43:07.508 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a6400e1c0 con 0x7f6a64011e80 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 4 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 3 8 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 10 client.53738.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 9 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 9 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 4 9 2024-02-10T08:43:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 4 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 10 client.53738.objecter _send_command 9 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a6400e4f0 con 0x7f6a6400ce50 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.911+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a6400ce50 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.912+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a6400e4f0 con 0x7f6a6400ce50 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.912+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.912+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 9 = system:0 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.912+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 4 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.912+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 4 9 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 10 client.53738.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 10 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 10 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a6400e1c0 osd=4 3 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 4 10 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 4 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 10 client.53738.objecter _send_command 10 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.927+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a6400e4f0 con 0x7f6a6400ce50 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a6400e4f0 con 0x7f6a6400ce50 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 4 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 4 10 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 10 client.53738.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 11 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 11 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 5 11 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 4 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 10 client.53738.objecter _send_command 11 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.929+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64018060 con 0x7f6a64017aa0 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a64017aa0 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64018060 con 0x7f6a64017aa0 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 11 = system:0 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 4 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.930+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 5 11 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 10 client.53738.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 12 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 12 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a6400e4f0 osd=5 3 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 5 12 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 4 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 10 client.53738.objecter _send_command 12 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.945+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a64018060 con 0x7f6a64017aa0 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a64018060 con 0x7f6a64017aa0 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 4 2024-02-10T08:43:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 5 12 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 10 client.53738.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 13 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 13 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 6 13 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 4 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 10 client.53738.objecter _send_command 13 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.947+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64022cb0 con 0x7f6a6401f160 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.948+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a6401f160 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.949+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64022cb0 con 0x7f6a6401f160 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.949+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.949+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 13 = system:0 2024-02-10T08:43:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.949+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 4 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.949+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 6 13 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 10 client.53738.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 14 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 14 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64018060 osd=6 3 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 6 14 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 4 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 10 client.53738.objecter _send_command 14 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.963+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a64022cb0 con 0x7f6a6401f160 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a64022cb0 con 0x7f6a6401f160 2024-02-10T08:43:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 4 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 6 14 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 10 client.53738.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 15 2024-02-10T08:43:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 15 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 7 15 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 4 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 10 client.53738.objecter _send_command 15 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a64023f00 con 0x7f6a640249b0 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.965+0000 7f6a5ffff700 10 client.53738.objecter ms_handle_connect 0x7f6a640249b0 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.966+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6a64023f00 con 0x7f6a640249b0 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.966+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.966+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 15 = system:0 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.966+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 4 2024-02-10T08:43:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.966+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 7 15 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 10 client.53738.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign -1 16 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 20 client.53738.objecter _get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_remove -1 16 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 20 client.53738.objecter get_session s=0x7f6a64022cb0 osd=7 3 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 15 client.53738.objecter _session_command_op_assign 7 16 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 4 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 10 client.53738.objecter _send_command 16 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.981+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6a64023f00 con 0x7f6a640249b0 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.983+0000 7f6a5ffff700 1 -- 172.21.15.107:0/2044710760 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6a64023f00 con 0x7f6a640249b0 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.983+0000 7f6a5ffff700 10 client.53738.objecter ms_dispatch 0x7f6a78001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.983+0000 7f6a5ffff700 10 client.53738.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.983+0000 7f6a5ffff700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 4 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.983+0000 7f6a5ffff700 15 client.53738.objecter _session_command_op_remove 7 16 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 librados: watch_flush enter 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 librados: watch_flush exit 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.0 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6801/3321596972 conn(0x7f6a64004530 legacy=0x7f6a64002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a640051a0 osd=0 1 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.1 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6805/133516126 conn(0x7f6a64004150 legacy=0x7f6a6400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003fe0 osd=1 1 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.2 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6809/3575881728 conn(0x7f6a6400d2f0 legacy=0x7f6a6400d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003630 osd=2 1 2024-02-10T08:43:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.3 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6813/2871825691 conn(0x7f6a64011e80 legacy=0x7f6a640142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64003470 osd=3 1 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.4 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.195:6800/2499671285 conn(0x7f6a6400ce50 legacy=0x7f6a64015d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e1c0 osd=4 1 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.5 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.195:6804/4178321486 conn(0x7f6a64017aa0 legacy=0x7f6a6401d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a6400e4f0 osd=5 1 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.6 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.195:6808/1417479017 conn(0x7f6a6401f160 legacy=0x7f6a64021550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64018060 osd=6 1 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter close_session for osd.7 2024-02-10T08:43:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.195:6812/1934176928 conn(0x7f6a640249b0 legacy=0x7f6a64024e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter put_session s=0x7f6a64022cb0 osd=7 1 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 20 client.53738.objecter shutdown clearing up homeless session... 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 10 client.53738.objecter successfully canceled tick 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a777fe700 1 -- 172.21.15.107:0/2044710760 reap_dead start 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a777fe700 1 -- 172.21.15.107:0/2044710760 reap_dead start 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a777fe700 1 -- 172.21.15.107:0/2044710760 reap_dead start 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a777fe700 1 -- 172.21.15.107:0/2044710760 reap_dead start 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a777fe700 1 -- 172.21.15.107:0/2044710760 reap_dead start 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6a60063cb0 legacy=0x7f6a60066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> v1:172.21.15.107:6790/0 conn(0x7f6a780fc920 legacy=0x7f6a78060bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 shutdown_connections 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 >> 172.21.15.107:0/2044710760 conn(0x7f6a780f2540 msgr2=0x7f6a781066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 shutdown_connections 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 -- 172.21.15.107:0/2044710760 wait complete. 2024-02-10T08:43:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:06.984+0000 7f6a7d852700 1 librados: shutdown 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:43:07.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:07.551 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:07.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:07.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:07.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:06.843102+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2044710760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:07.055651+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e458: 8 total, 8 up, 6 in 2024-02-10T08:43:07.802 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:07.823 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:07.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:07.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:06.843102+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2044710760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:07.055651+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e458: 8 total, 8 up, 6 in 2024-02-10T08:43:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:06.843102+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.107:0/2044710760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:07.055651+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e458: 8 total, 8 up, 6 in 2024-02-10T08:43:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:08.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:08.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 -- 172.21.15.107:0/1231720337 >> v1:172.21.15.107:6790/0 conn(0x7fce501007f0 legacy=0x7fce50102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 -- 172.21.15.107:0/1231720337 shutdown_connections 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 -- 172.21.15.107:0/1231720337 >> 172.21.15.107:0/1231720337 conn(0x7fce500f3640 msgr2=0x7fce500f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 -- 172.21.15.107:0/1231720337 shutdown_connections 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 -- 172.21.15.107:0/1231720337 wait complete. 2024-02-10T08:43:08.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 librados: starting msgr at 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 librados: starting objecter 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.741+0000 7fce56046700 1 Processor -- start 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 -- start start 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 librados: setting wanted keys 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 librados: calling monclient init 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce50117420 con 0x7fce501007f0 2024-02-10T08:43:08.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce50118620 con 0x7fce500fda50 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce56046700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce50119820 con 0x7fce50104330 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce46ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fce501007f0 0x7fce500fa1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46298/0 (socket says 172.21.15.107:46298) 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce46ffd700 1 -- 172.21.15.107:0/3348846078 learned_addr learned my addr 172.21.15.107:0/3348846078 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4ffff700 1 --1- 172.21.15.107:0/3348846078 >> v1:172.21.15.107:6790/0 conn(0x7fce50104330 0x7fce50115ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43994/0 (socket says 172.21.15.107:43994) 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter ms_handle_connect 0x7fce501007f0 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter ms_handle_connect 0x7fce50104330 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:08.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter ms_handle_connect 0x7fce500fda50 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660166461 0 0) 0x7fce50117420 con 0x7fce501007f0 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c0033f0 con 0x7fce501007f0 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794173079 0 0) 0x7fce50119820 con 0x7fce50104330 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50117420 con 0x7fce50104330 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813637758 0 0) 0x7fce50118620 con 0x7fce500fda50 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.742+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50119820 con 0x7fce500fda50 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306218688 0 0) 0x7fce3c0033f0 con 0x7fce501007f0 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce50118620 con 0x7fce501007f0 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce38002c60 con 0x7fce501007f0 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318269023 0 0) 0x7fce50117420 con 0x7fce50104330 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce3c0033f0 con 0x7fce50104330 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce48002c10 con 0x7fce50104330 2024-02-10T08:43:08.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199800349 0 0) 0x7fce50119820 con 0x7fce500fda50 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce50117420 con 0x7fce500fda50 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce40001600 con 0x7fce500fda50 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1298165922 0 0) 0x7fce50118620 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 >> v1:172.21.15.107:6790/0 conn(0x7fce50104330 legacy=0x7fce50115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 >> v1:172.21.15.195:6789/0 conn(0x7fce500fda50 legacy=0x7fce500f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5011aa20 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce50119a10 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce56046700 10 client.28158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce50119e80 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce56046700 1 librados: init done 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fce38001b60 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.743+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce380038c0 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.744+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce30004f20 con 0x7fce501007f0 2024-02-10T08:43:08.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.744+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fce38003a20 con 0x7fce501007f0 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(459..459 src has 330..459) v4 ==== 8094+0+0 (unknown 1961029846 0 0) 0x7fce38078ee0 con 0x7fce501007f0 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 10 client.28158.objecter ms_dispatch 0x7fce50001550 osd_map(459..459 src has 330..459) v4 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 3 client.28158.objecter handle_osd_map got epochs [459,459] > 0 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 3 client.28158.objecter handle_osd_map decoding full epoch 459 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 20 client.28158.objecter dump_active .. 0 homeless 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.745+0000 7fce4d7fa700 10 client.28158.objecter ms_handle_connect 0x7fce3c063d00 2024-02-10T08:43:08.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.749+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fce3804c3d0 con 0x7fce501007f0 2024-02-10T08:43:08.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:08.924+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fce300051d0 con 0x7fce501007f0 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:07.077735+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:07.077772+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:07.459645+0000 mgr.y (mgr.14553) 415 : cluster [DBG] pgmap v681: 186 pgs: 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 459 B/s, 75 objects/s recovering 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:08.052006+0000 mon.a (mon.0) 2573 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:08.052032+0000 mon.a (mon.0) 2574 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded) 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:08.058530+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e459: 8 total, 8 up, 6 in 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:08.071455+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:08.071753+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:07.077735+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:07.077772+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:07.459645+0000 mgr.y (mgr.14553) 415 : cluster [DBG] pgmap v681: 186 pgs: 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 459 B/s, 75 objects/s recovering 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:08.052006+0000 mon.a (mon.0) 2573 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:08.052032+0000 mon.a (mon.0) 2574 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded) 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:08.058530+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e459: 8 total, 8 up, 6 in 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:08.071455+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:08.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:08.071753+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:07.077735+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:07.077772+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:07.459645+0000 mgr.y (mgr.14553) 415 : cluster [DBG] pgmap v681: 186 pgs: 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 459 B/s, 75 objects/s recovering 2024-02-10T08:43:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:08.052006+0000 mon.a (mon.0) 2573 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-02-10T08:43:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:08.052032+0000 mon.a (mon.0) 2574 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 126/1083 objects degraded (11.634%), 13 pgs degraded) 2024-02-10T08:43:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:08.058530+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e459: 8 total, 8 up, 6 in 2024-02-10T08:43:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:08.071455+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:08.071753+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:09.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.063+0000 7fce4d7fa700 1 -- 172.21.15.107:0/3348846078 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v460) v1 ==== 91+0+0 (unknown 317966960 0 0) 0x7fce38002ec0 con 0x7fce501007f0 2024-02-10T08:43:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 10 librados: watch_flush enter 2024-02-10T08:43:09.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 10 librados: watch_flush exit 2024-02-10T08:43:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 20 client.28158.objecter shutdown clearing up homeless session... 2024-02-10T08:43:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 10 client.28158.objecter successfully canceled tick 2024-02-10T08:43:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 >> v1:172.21.15.107:6800/2634842009 conn(0x7fce3c063d00 legacy=0x7fce3c0661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:09.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.065+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 >> v1:172.21.15.107:6789/0 conn(0x7fce501007f0 legacy=0x7fce500fa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:09.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.067+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 shutdown_connections 2024-02-10T08:43:09.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.067+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 >> 172.21.15.107:0/3348846078 conn(0x7fce500f3640 msgr2=0x7fce501077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:09.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.068+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 shutdown_connections 2024-02-10T08:43:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.068+0000 7fce56046700 1 -- 172.21.15.107:0/3348846078 wait complete. 2024-02-10T08:43:09.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:09.069+0000 7fce56046700 1 librados: shutdown 2024-02-10T08:43:09.078 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:43:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:08.925137+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:09.056880+0000 mon.a (mon.0) 2578 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:09.063377+0000 mon.a (mon.0) 2579 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:09.063488+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:09.063529+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e460: 8 total, 8 up, 6 in 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:09.065266+0000 mon.b (mon.1) 607 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:08.925137+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:09.056880+0000 mon.a (mon.0) 2578 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:09.063377+0000 mon.a (mon.0) 2579 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:09.063488+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:09.063529+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e460: 8 total, 8 up, 6 in 2024-02-10T08:43:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:09.065266+0000 mon.b (mon.1) 607 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:08.925137+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:09.056880+0000 mon.a (mon.0) 2578 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:09.063377+0000 mon.a (mon.0) 2579 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:09.063488+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? v1:172.21.15.107:0/3348846078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:09.063529+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e460: 8 total, 8 up, 6 in 2024-02-10T08:43:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:09.065266+0000 mon.b (mon.1) 607 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:10.187 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:10.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:10.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:43:10.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:10.328 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:10.407 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:10.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:10.449 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:43:10.449 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:43:10.449 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:43:10.449 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:43:10.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:10.539 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:09.460052+0000 mgr.y (mgr.14553) 416 : cluster [DBG] pgmap v684: 218 pgs: 32 unknown, 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 462 B/s, 75 objects/s recovering 2024-02-10T08:43:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:09.798640+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:10.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:09.798883+0000 osd.1 (osd.1) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:10.065482+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e461: 8 total, 8 up, 6 in 2024-02-10T08:43:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:09.460052+0000 mgr.y (mgr.14553) 416 : cluster [DBG] pgmap v684: 218 pgs: 32 unknown, 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 462 B/s, 75 objects/s recovering 2024-02-10T08:43:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:09.798640+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:09.798883+0000 osd.1 (osd.1) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:10.065482+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e461: 8 total, 8 up, 6 in 2024-02-10T08:43:10.737 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:10.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:10.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:10.868 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:09.460052+0000 mgr.y (mgr.14553) 416 : cluster [DBG] pgmap v684: 218 pgs: 32 unknown, 1 activating, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 462 B/s, 75 objects/s recovering 2024-02-10T08:43:11.168 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:09.798640+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:11.168 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:09.798883+0000 osd.1 (osd.1) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:11.168 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:10.065482+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e461: 8 total, 8 up, 6 in 2024-02-10T08:43:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:11.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:11.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:43:11.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:11.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:12.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:43:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:10.386995+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:10.390461+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:10.800515+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:10.801468+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.068159+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e462: 8 total, 8 up, 6 in 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.116127+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.116855+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.460877+0000 mgr.y (mgr.14553) 417 : cluster [DBG] pgmap v687: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.501840+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub starts 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.504910+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.e scrub ok 2024-02-10T08:43:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:11.511500+0000 mgr.y (mgr.14553) 418 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:10.386995+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:10.390461+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:10.800515+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:10.801468+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.068159+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e462: 8 total, 8 up, 6 in 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.116127+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.116855+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.460877+0000 mgr.y (mgr.14553) 417 : cluster [DBG] pgmap v687: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.501840+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub starts 2024-02-10T08:43:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.504910+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.e scrub ok 2024-02-10T08:43:12.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:11.511500+0000 mgr.y (mgr.14553) 418 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:10.386995+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:10.390461+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:10.800515+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:10.801468+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.068159+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e462: 8 total, 8 up, 6 in 2024-02-10T08:43:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.116127+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:43:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.116855+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:43:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.460877+0000 mgr.y (mgr.14553) 417 : cluster [DBG] pgmap v687: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:43:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.501840+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub starts 2024-02-10T08:43:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.504910+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.e scrub ok 2024-02-10T08:43:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:11.511500+0000 mgr.y (mgr.14553) 418 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:13.001 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:13.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:13.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:13.139 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:13.160 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:13.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:13.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.807149+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub starts 2024-02-10T08:43:13.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:11.810626+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.f scrub ok 2024-02-10T08:43:13.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.807149+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub starts 2024-02-10T08:43:13.255 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:11.810626+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.f scrub ok 2024-02-10T08:43:13.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:13.295 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:13.316 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:13.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:13.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:13.448 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.807149+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub starts 2024-02-10T08:43:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:11.810626+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.f scrub ok 2024-02-10T08:43:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:13.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.746+0000 7f45546f7700 1 -- 172.21.15.107:0/3021834462 >> v1:172.21.15.107:6789/0 conn(0x7f454c104330 legacy=0x7f454c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.746+0000 7f45546f7700 1 -- 172.21.15.107:0/3021834462 shutdown_connections 2024-02-10T08:43:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.746+0000 7f45546f7700 1 -- 172.21.15.107:0/3021834462 >> 172.21.15.107:0/3021834462 conn(0x7f454c0f3640 msgr2=0x7f454c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.746+0000 7f45546f7700 1 -- 172.21.15.107:0/3021834462 shutdown_connections 2024-02-10T08:43:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.746+0000 7f45546f7700 1 -- 172.21.15.107:0/3021834462 wait complete. 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 librados: starting msgr at 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 librados: starting objecter 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 Processor -- start 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 -- start start 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 librados: setting wanted keys 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 librados: calling monclient init 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f454c117230 con 0x7f454c0fda50 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f454c118410 con 0x7f454c1007f0 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f45546f7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f454c1195f0 con 0x7f454c104330 2024-02-10T08:43:13.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f4552493700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f454c0fda50 0x7f454c0f9800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39032/0 (socket says 172.21.15.107:39032) 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.747+0000 7f4552493700 1 -- 172.21.15.107:0/4226046706 learned_addr learned my addr 172.21.15.107:0/4226046706 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter ms_handle_connect 0x7f454c0fda50 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter ms_handle_connect 0x7f454c104330 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter ms_handle_connect 0x7f454c1007f0 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151365841 0 0) 0x7f454c117230 con 0x7f454c0fda50 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4538003030 con 0x7f454c0fda50 2024-02-10T08:43:13.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425276626 0 0) 0x7f454c1195f0 con 0x7f454c104330 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c117230 con 0x7f454c104330 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083467190 0 0) 0x7f454c118410 con 0x7f454c1007f0 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c1195f0 con 0x7f454c1007f0 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1426654172 0 0) 0x7f4538003030 con 0x7f454c0fda50 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c118410 con 0x7f454c0fda50 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4548001990 con 0x7f454c0fda50 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341548122 0 0) 0x7f454c117230 con 0x7f454c104330 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4538003030 con 0x7f454c104330 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4544002310 con 0x7f454c104330 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215609909 0 0) 0x7f454c1195f0 con 0x7f454c1007f0 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c117230 con 0x7f454c1007f0 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f453c0019d0 con 0x7f454c1007f0 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2095214309 0 0) 0x7f454c118410 con 0x7f454c0fda50 2024-02-10T08:43:13.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 >> v1:172.21.15.107:6790/0 conn(0x7f454c104330 legacy=0x7f454c1159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 >> v1:172.21.15.195:6789/0 conn(0x7f454c1007f0 legacy=0x7f454c0f9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c11a7f0 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c1197e0 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45546f7700 10 client.28164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c119cd0 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45546f7700 1 librados: init done 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4548001c50 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.748+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4548005bc0 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.749+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530004f20 con 0x7f454c0fda50 2024-02-10T08:43:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4548002b50 con 0x7f454c0fda50 2024-02-10T08:43:13.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(462..462 src has 330..462) v4 ==== 8107+0+0 (unknown 121972050 0 0) 0x7f4548078dc0 con 0x7f454c0fda50 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 10 client.28164.objecter ms_dispatch 0x7f454c001550 osd_map(462..462 src has 330..462) v4 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 3 client.28164.objecter handle_osd_map got epochs [462,462] > 0 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 3 client.28164.objecter handle_osd_map decoding full epoch 462 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 20 client.28164.objecter dump_active .. 0 homeless 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.750+0000 7f45437fe700 10 client.28164.objecter ms_handle_connect 0x7f4538063d20 2024-02-10T08:43:13.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.754+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f454804c1c0 con 0x7f454c0fda50 2024-02-10T08:43:13.782 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:13.879 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:13.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:13.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:13.935+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd in", "ids": ["7"]} v 0) v1 -- 0x7f45300051d0 con 0x7f454c0fda50 2024-02-10T08:43:13.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:14.012 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.085+0000 7f45437fe700 1 -- 172.21.15.107:0/4226046706 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["7"]}]=0 marked in osd.7. v463) v1 ==== 85+0+0 (unknown 2860102806 0 0) 0x7f454804a5d0 con 0x7f454c0fda50 2024-02-10T08:43:14.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.089+0000 7f45546f7700 10 librados: watch_flush enter 2024-02-10T08:43:14.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.092+0000 7f45546f7700 10 librados: watch_flush exit 2024-02-10T08:43:14.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.092+0000 7f45546f7700 20 client.28164.objecter shutdown clearing up homeless session... 2024-02-10T08:43:14.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.092+0000 7f45546f7700 10 client.28164.objecter successfully canceled tick 2024-02-10T08:43:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.093+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4538063d20 legacy=0x7f45380661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.093+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 >> v1:172.21.15.107:6789/0 conn(0x7f454c0fda50 legacy=0x7f454c0f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.095+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 shutdown_connections 2024-02-10T08:43:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.095+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 >> 172.21.15.107:0/4226046706 conn(0x7f454c0f3640 msgr2=0x7f454c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:14.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.101+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 shutdown_connections 2024-02-10T08:43:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.104+0000 7f45546f7700 1 -- 172.21.15.107:0/4226046706 wait complete. 2024-02-10T08:43:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:14.104+0000 7f45546f7700 1 librados: shutdown 2024-02-10T08:43:14.109 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.7. 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:43:13.935+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e462 definitely_dead 0 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:12.377244+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:12.378209+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:12.733686+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:12.736733+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.461226+0000 mgr.y (mgr.14553) 419 : cluster [DBG] pgmap v688: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 947 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.783084+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.786125+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:43:14.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:13.936491+0000 mon.a (mon.0) 2584 : audit [DBG] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:12.377244+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:12.378209+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:12.733686+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:12.736733+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.461226+0000 mgr.y (mgr.14553) 419 : cluster [DBG] pgmap v688: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 947 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.783084+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.786125+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:43:14.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:13.936491+0000 mon.a (mon.0) 2584 : audit [DBG] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-10T08:43:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:12.377244+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:43:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:12.378209+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:43:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:12.733686+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:43:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:12.736733+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:43:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.461226+0000 mgr.y (mgr.14553) 419 : cluster [DBG] pgmap v688: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 947 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.783084+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:43:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.786125+0000 osd.4 (osd.4) 221 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:43:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:13.936491+0000 mon.a (mon.0) 2584 : audit [DBG] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-10T08:43:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.329006+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:43:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.329908+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:43:15.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.741045+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a scrub starts 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:13.744034+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a scrub ok 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:14.085884+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.085917+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.429453+0000 osd.2 (osd.2) 178 : cluster [DBG] 60.c scrub starts 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.432947+0000 osd.2 (osd.2) 179 : cluster [DBG] 60.c scrub ok 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.329006+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.329908+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:43:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.741045+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a scrub starts 2024-02-10T08:43:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:13.744034+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a scrub ok 2024-02-10T08:43:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:14.085884+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-10T08:43:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.085917+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-02-10T08:43:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.429453+0000 osd.2 (osd.2) 178 : cluster [DBG] 60.c scrub starts 2024-02-10T08:43:15.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.432947+0000 osd.2 (osd.2) 179 : cluster [DBG] 60.c scrub ok 2024-02-10T08:43:15.462 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:15.463 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-02-10T08:43:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:15.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.329006+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.329908+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.741045+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a scrub starts 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:13.744034+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a scrub ok 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:14.085884+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? v1:172.21.15.107:0/4226046706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.085917+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.429453+0000 osd.2 (osd.2) 178 : cluster [DBG] 60.c scrub starts 2024-02-10T08:43:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.432947+0000 osd.2 (osd.2) 179 : cluster [DBG] 60.c scrub ok 2024-02-10T08:43:15.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:15.591 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:15.613 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:15.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:15.690 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:15] "GET /metrics HTTP/1.1" 200 188026 "" "Prometheus/2.33.4" 2024-02-10T08:43:15.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:15.743 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:15.761 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:15.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:15.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:15.890 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:15.911 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:15.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:16.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:16.252 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:16.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.143752+0000 osd.3 (osd.3) 225 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.144534+0000 osd.3 (osd.3) 226 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.354134+0000 osd.6 (osd.6) 200 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.357147+0000 osd.6 (osd.6) 201 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.588398+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:14.588661+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:16.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:15.090046+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-02-10T08:43:16.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:15.461588+0000 mgr.y (mgr.14553) 420 : cluster [DBG] pgmap v691: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 948 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:16.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:15.467589+0000 mon.a (mon.0) 2588 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:16.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:16.050736+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-02-10T08:43:16.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.143752+0000 osd.3 (osd.3) 225 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:43:16.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.144534+0000 osd.3 (osd.3) 226 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:43:16.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.354134+0000 osd.6 (osd.6) 200 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:43:16.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.357147+0000 osd.6 (osd.6) 201 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:43:16.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.588398+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:14.588661+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:15.090046+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:15.461588+0000 mgr.y (mgr.14553) 420 : cluster [DBG] pgmap v691: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 948 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:15.467589+0000 mon.a (mon.0) 2588 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:16.050736+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.143752+0000 osd.3 (osd.3) 225 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.144534+0000 osd.3 (osd.3) 226 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.354134+0000 osd.6 (osd.6) 200 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.357147+0000 osd.6 (osd.6) 201 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.588398+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:14.588661+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:16.353 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:15.090046+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-02-10T08:43:16.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:15.461588+0000 mgr.y (mgr.14553) 420 : cluster [DBG] pgmap v691: 218 pgs: 218 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 948 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T08:43:16.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:15.467589+0000 mon.a (mon.0) 2588 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:16.354 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:16.050736+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-02-10T08:43:16.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:16.733 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:16.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:16.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:16.863 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:16.960 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:16.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:17.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:17.093 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:17.119 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:17.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:17.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:17.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:16.049698+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 455'13 2024-02-10T08:43:17.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:16.055273+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 454'13 2024-02-10T08:43:17.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:16.049698+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 455'13 2024-02-10T08:43:17.249 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:16.055273+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 454'13 2024-02-10T08:43:17.252 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:43:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:43:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:43:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:43:17.464 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:43:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:16.049698+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 455'13 2024-02-10T08:43:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:16.055273+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 454'13 2024-02-10T08:43:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:18.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:18.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:18.142 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:18.241 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:18.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:18.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.276+0000 7f22957b8700 1 -- 172.21.15.107:0/2524798972 >> v1:172.21.15.107:6790/0 conn(0x7f22901007f0 legacy=0x7f2290102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 -- 172.21.15.107:0/2524798972 shutdown_connections 2024-02-10T08:43:18.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 -- 172.21.15.107:0/2524798972 >> 172.21.15.107:0/2524798972 conn(0x7f22900f3640 msgr2=0x7f22900f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:18.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 -- 172.21.15.107:0/2524798972 shutdown_connections 2024-02-10T08:43:18.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 -- 172.21.15.107:0/2524798972 wait complete. 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 librados: starting msgr at 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 librados: starting objecter 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.277+0000 7f22957b8700 1 Processor -- start 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 -- start start 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 librados: setting wanted keys 2024-02-10T08:43:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 librados: calling monclient init 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2290115370 con 0x7f22900fda50 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2290116550 con 0x7f2290104330 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f22957b8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2290117750 con 0x7f22901007f0 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f228effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f22900fda50 0x7f2290102670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39046/0 (socket says 172.21.15.107:39046) 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f228effd700 1 -- 172.21.15.107:0/964075665 learned_addr learned my addr 172.21.15.107:0/964075665 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter ms_handle_connect 0x7f2290104330 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter ms_handle_connect 0x7f22900fda50 2024-02-10T08:43:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter ms_handle_connect 0x7f22901007f0 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.278+0000 7f2287fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107471298 0 0) 0x7f2290117750 con 0x7f22901007f0 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f227c0033f0 con 0x7f22901007f0 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239881680 0 0) 0x7f2290115370 con 0x7f22900fda50 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290117750 con 0x7f22900fda50 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367896450 0 0) 0x7f2290116550 con 0x7f2290104330 2024-02-10T08:43:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290115370 con 0x7f2290104330 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 330214161 0 0) 0x7f2290115370 con 0x7f2290104330 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2290116550 con 0x7f2290104330 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2288001c40 con 0x7f2290104330 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048087201 0 0) 0x7f227c0033f0 con 0x7f22901007f0 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2290115370 con 0x7f22901007f0 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2280002ca0 con 0x7f22901007f0 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864529202 0 0) 0x7f2290117750 con 0x7f22900fda50 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f227c0033f0 con 0x7f22900fda50 2024-02-10T08:43:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2278001880 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1862321654 0 0) 0x7f227c0033f0 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6790/0 conn(0x7f22901007f0 legacy=0x7f2290102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.195:6789/0 conn(0x7f2290104330 legacy=0x7f2290113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290117940 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2290117bf0 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f22957b8700 10 client.28170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2290118060 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.279+0000 7f22957b8700 1 librados: init done 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.280+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2278001f40 con 0x7f22900fda50 2024-02-10T08:43:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.280+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2278005a60 con 0x7f22900fda50 2024-02-10T08:43:18.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.281+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2270004f00 con 0x7f22900fda50 2024-02-10T08:43:18.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.281+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f22780188f0 con 0x7f22900fda50 2024-02-10T08:43:18.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(467..467 src has 330..467) v4 ==== 8260+0+0 (unknown 3600705626 0 0) 0x7f22780791f0 con 0x7f22900fda50 2024-02-10T08:43:18.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 osd_map(467..467 src has 330..467) v4 2024-02-10T08:43:18.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 3 client.28170.objecter handle_osd_map got epochs [467,467] > 0 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 3 client.28170.objecter handle_osd_map decoding full epoch 467 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 20 client.28170.objecter dump_active .. 0 homeless 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v467) v1 ==== 54+0+16 (unknown 3576603848 0 2420000192) 0x7f2278001b00 con 0x7f22900fda50 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 10 client.28170.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 1 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 1 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 0 1 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 4 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 10 client.28170.objecter _send_command 1 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2270004950 con 0x7f2270004530 2024-02-10T08:43:18.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.282+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f227c063e40 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.284+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f2270004530 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.285+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2270004950 con 0x7f2270004530 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.285+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.285+0000 7f2287fff700 10 client.28170.objecter _finish_command 1 = system:0 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.285+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 4 2024-02-10T08:43:18.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.285+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 0 1 2024-02-10T08:43:18.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 10 client.28170.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 2 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 2 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f22700051a0 osd=0 3 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 0 2 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 4 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 10 client.28170.objecter _send_command 2 2024-02-10T08:43:18.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.312+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2270004d30 con 0x7f2270004530 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.314+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2270004d30 con 0x7f2270004530 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.314+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.314+0000 7f2287fff700 10 client.28170.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.314+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 4 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.314+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 0 2 2024-02-10T08:43:18.316 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 10 client.28170.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 1 3 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 4 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 10 client.28170.objecter _send_command 3 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.315+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f227000c000 con 0x7f2270004150 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.316+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f2270004150 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.317+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f227000c000 con 0x7f2270004150 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.317+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.317+0000 7f2287fff700 10 client.28170.objecter _finish_command 3 = system:0 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.317+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 4 2024-02-10T08:43:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.317+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 1 3 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 10 client.28170.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 4 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 4 2024-02-10T08:43:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270003fe0 osd=1 3 2024-02-10T08:43:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 1 4 2024-02-10T08:43:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 4 2024-02-10T08:43:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 10 client.28170.objecter _send_command 4 2024-02-10T08:43:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.344+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f22700018b0 con 0x7f2270004150 2024-02-10T08:43:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f22700018b0 con 0x7f2270004150 2024-02-10T08:43:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f2287fff700 10 client.28170.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 4 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 1 4 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 10 client.28170.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 5 2024-02-10T08:43:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 5 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 2 5 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 4 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 10 client.28170.objecter _send_command 5 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.346+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f227000de60 con 0x7f227000d2f0 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f227000d2f0 2024-02-10T08:43:18.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f227000de60 con 0x7f227000d2f0 2024-02-10T08:43:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 10 client.28170.objecter _finish_command 5 = system:0 2024-02-10T08:43:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 4 2024-02-10T08:43:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.347+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 2 5 2024-02-10T08:43:18.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:18.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 10 client.28170.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 6 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 6 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270003630 osd=2 3 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 2 6 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 4 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 10 client.28170.objecter _send_command 6 2024-02-10T08:43:18.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.374+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f22700030a0 con 0x7f227000d2f0 2024-02-10T08:43:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f22700030a0 con 0x7f227000d2f0 2024-02-10T08:43:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f2287fff700 10 client.28170.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 4 2024-02-10T08:43:18.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 2 6 2024-02-10T08:43:18.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:17.105180+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-02-10T08:43:18.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:17.461976+0000 mgr.y (mgr.14553) 421 : cluster [DBG] pgmap v694: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 263 B/s, 11 objects/s recovering 2024-02-10T08:43:18.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:17.105180+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-02-10T08:43:18.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:17.461976+0000 mgr.y (mgr.14553) 421 : cluster [DBG] pgmap v694: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 263 B/s, 11 objects/s recovering 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.377+0000 7f22957b8700 10 client.28170.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 7 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 7 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 3 7 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 4 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 10 client.28170.objecter _send_command 7 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2270015ad0 con 0x7f2270011e80 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.378+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f2270011e80 2024-02-10T08:43:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:18.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.379+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2270015ad0 con 0x7f2270011e80 2024-02-10T08:43:18.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.379+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:18.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.379+0000 7f2287fff700 10 client.28170.objecter _finish_command 7 = system:0 2024-02-10T08:43:18.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.379+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 4 2024-02-10T08:43:18.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.379+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 3 7 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 10 client.28170.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 8 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 8 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f22700034b0 osd=3 3 2024-02-10T08:43:18.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 3 8 2024-02-10T08:43:18.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 4 2024-02-10T08:43:18.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 10 client.28170.objecter _send_command 8 2024-02-10T08:43:18.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.405+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f227000e240 con 0x7f2270011e80 2024-02-10T08:43:18.408 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:18.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f227000e240 con 0x7f2270011e80 2024-02-10T08:43:18.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f2287fff700 10 client.28170.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 4 2024-02-10T08:43:18.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 3 8 2024-02-10T08:43:18.410 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 10 client.28170.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 9 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 9 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 4 9 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 4 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 10 client.28170.objecter _send_command 9 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.408+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f227000e1b0 con 0x7f227000ce50 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.409+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f227000ce50 2024-02-10T08:43:18.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.409+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f227000e1b0 con 0x7f227000ce50 2024-02-10T08:43:18.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.410+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:43:18.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.410+0000 7f2287fff700 10 client.28170.objecter _finish_command 9 = system:0 2024-02-10T08:43:18.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.410+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 4 2024-02-10T08:43:18.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.410+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 4 9 2024-02-10T08:43:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:18.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 10 client.28170.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 10 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 10 2024-02-10T08:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f227000e000 osd=4 3 2024-02-10T08:43:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 4 10 2024-02-10T08:43:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 4 2024-02-10T08:43:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 10 client.28170.objecter _send_command 10 2024-02-10T08:43:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.436+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f227000e1b0 con 0x7f227000ce50 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.437+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f227000e1b0 con 0x7f227000ce50 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.437+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.437+0000 7f2287fff700 10 client.28170.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.437+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 4 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.437+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 4 10 2024-02-10T08:43:18.439 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 10 client.28170.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 11 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 11 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 5 11 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 4 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 10 client.28170.objecter _send_command 11 2024-02-10T08:43:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2270018090 con 0x7f2270017a60 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.438+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f2270017a60 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.439+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2270018090 con 0x7f2270017a60 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.439+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.439+0000 7f2287fff700 10 client.28170.objecter _finish_command 11 = system:0 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.439+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 4 2024-02-10T08:43:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.439+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 5 11 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 10 client.28170.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 12 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 12 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f227000e1b0 osd=5 3 2024-02-10T08:43:18.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 5 12 2024-02-10T08:43:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 4 2024-02-10T08:43:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 10 client.28170.objecter _send_command 12 2024-02-10T08:43:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.456+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2270018260 con 0x7f2270017a60 2024-02-10T08:43:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2270018260 con 0x7f2270017a60 2024-02-10T08:43:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f2287fff700 10 client.28170.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 4 2024-02-10T08:43:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 5 12 2024-02-10T08:43:18.459 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 10 client.28170.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 13 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 13 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 6 13 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 4 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 10 client.28170.objecter _send_command 13 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.457+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2270022ca0 con 0x7f227001f0d0 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.458+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f227001f0d0 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.459+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2270022ca0 con 0x7f227001f0d0 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.459+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.459+0000 7f2287fff700 10 client.28170.objecter _finish_command 13 = system:0 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.459+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 4 2024-02-10T08:43:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.459+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 6 13 2024-02-10T08:43:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 10 client.28170.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 14 2024-02-10T08:43:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 14 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270018050 osd=6 3 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 6 14 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 4 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 10 client.28170.objecter _send_command 14 2024-02-10T08:43:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.473+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2270022ca0 con 0x7f227001f0d0 2024-02-10T08:43:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2270022ca0 con 0x7f227001f0d0 2024-02-10T08:43:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 10 client.28170.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 4 2024-02-10T08:43:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 6 14 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 10 client.28170.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 15 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 15 2024-02-10T08:43:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 7 15 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 4 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 10 client.28170.objecter _send_command 15 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2270023e90 con 0x7f2270024960 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.475+0000 7f2287fff700 10 client.28170.objecter ms_handle_connect 0x7f2270024960 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.476+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2270023e90 con 0x7f2270024960 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.476+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.476+0000 7f2287fff700 10 client.28170.objecter _finish_command 15 = system:0 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.476+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 4 2024-02-10T08:43:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.476+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 7 15 2024-02-10T08:43:18.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 10 client.28170.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:18.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign -1 16 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 20 client.28170.objecter _get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 15 client.28170.objecter _session_command_op_remove -1 16 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 20 client.28170.objecter get_session s=0x7f2270022ca0 osd=7 3 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 15 client.28170.objecter _session_command_op_assign 7 16 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 4 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 10 client.28170.objecter _send_command 16 2024-02-10T08:43:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.503+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2270023e50 con 0x7f2270024960 2024-02-10T08:43:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f2287fff700 1 -- 172.21.15.107:0/964075665 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2270023e50 con 0x7f2270024960 2024-02-10T08:43:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f2287fff700 10 client.28170.objecter ms_dispatch 0x7f2290001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f2287fff700 10 client.28170.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.506 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f2287fff700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 4 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f2287fff700 15 client.28170.objecter _session_command_op_remove 7 16 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 librados: watch_flush enter 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 librados: watch_flush exit 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.0 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6801/3321596972 conn(0x7f2270004530 legacy=0x7f2270002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700051a0 osd=0 1 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.1 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6805/133516126 conn(0x7f2270004150 legacy=0x7f227000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003fe0 osd=1 1 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.2 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6809/3575881728 conn(0x7f227000d2f0 legacy=0x7f227000d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270003630 osd=2 1 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.3 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6813/2871825691 conn(0x7f2270011e80 legacy=0x7f2270014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f228f7fe700 1 -- 172.21.15.107:0/964075665 reap_dead start 2024-02-10T08:43:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f22700034b0 osd=3 1 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.4 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.195:6800/2499671285 conn(0x7f227000ce50 legacy=0x7f2270015c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f228f7fe700 1 -- 172.21.15.107:0/964075665 reap_dead start 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e000 osd=4 1 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.5 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.195:6804/4178321486 conn(0x7f2270017a60 legacy=0x7f227001d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f227000e1b0 osd=5 1 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.6 2024-02-10T08:43:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.195:6808/1417479017 conn(0x7f227001f0d0 legacy=0x7f22700214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270018050 osd=6 1 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter close_session for osd.7 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.195:6812/1934176928 conn(0x7f2270024960 legacy=0x7f2270024dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter put_session s=0x7f2270022ca0 osd=7 1 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 20 client.28170.objecter shutdown clearing up homeless session... 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 10 client.28170.objecter successfully canceled tick 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6800/2634842009 conn(0x7f227c063e40 legacy=0x7f227c066300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> v1:172.21.15.107:6789/0 conn(0x7f22900fda50 legacy=0x7f2290102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.505+0000 7f228f7fe700 1 -- 172.21.15.107:0/964075665 reap_dead start 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.506+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 shutdown_connections 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.506+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 >> 172.21.15.107:0/964075665 conn(0x7f22900f3640 msgr2=0x7f22901077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.506+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 shutdown_connections 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.506+0000 7f22957b8700 1 -- 172.21.15.107:0/964075665 wait complete. 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:18.506+0000 7f22957b8700 1 librados: shutdown 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:43:18.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:17.105180+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-02-10T08:43:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:17.461976+0000 mgr.y (mgr.14553) 421 : cluster [DBG] pgmap v694: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 263 B/s, 11 objects/s recovering 2024-02-10T08:43:18.549 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:18.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:43:19.129 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:19.133 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:18.099966+0000 mon.a (mon.0) 2591 : cluster [WRN] Health check failed: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:19.133 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:18.107697+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-02-10T08:43:19.133 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:18.283502+0000 mon.a (mon.0) 2593 : audit [DBG] from='client.? v1:172.21.15.107:0/964075665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:19.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:18.099966+0000 mon.a (mon.0) 2591 : cluster [WRN] Health check failed: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:19.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:18.107697+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-02-10T08:43:19.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:18.283502+0000 mon.a (mon.0) 2593 : audit [DBG] from='client.? v1:172.21.15.107:0/964075665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:19.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:19.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:19.264 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:19.283 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:19.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:19.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:18.099966+0000 mon.a (mon.0) 2591 : cluster [WRN] Health check failed: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:18.107697+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-02-10T08:43:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:18.283502+0000 mon.a (mon.0) 2593 : audit [DBG] from='client.? v1:172.21.15.107:0/964075665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:19.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:19.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:19.747 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:19.765 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:19.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:19.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:19.895 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:19.914 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:19.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:20.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:20.043 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:20.062 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:20.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:20.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:20.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:19.462522+0000 mgr.y (mgr.14553) 422 : cluster [DBG] pgmap v696: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 234 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 240 B/s, 10 objects/s recovering 2024-02-10T08:43:20.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:19.462522+0000 mgr.y (mgr.14553) 422 : cluster [DBG] pgmap v696: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 234 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 240 B/s, 10 objects/s recovering 2024-02-10T08:43:20.192 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:20.210 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:20.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:20.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:20.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:19.462522+0000 mgr.y (mgr.14553) 422 : cluster [DBG] pgmap v696: 218 pgs: 1 activating+undersized+remapped, 3 activating+undersized+degraded+remapped, 2 activating+remapped, 13 active+recovery_wait+degraded, 4 active+recovering, 195 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 234 B/s wr, 4 op/s; 257/1095 objects degraded (23.470%); 19/1095 objects misplaced (1.735%); 240 B/s, 10 objects/s recovering 2024-02-10T08:43:20.338 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:20.340 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:20.357 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:20.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:20.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:20.464 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:43:20.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:43:20.464 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.9418217701173248 2024-02-10T08:43:20.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 2 0.9418217701173248 2024-02-10T08:43:20.465 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:20.493 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:20.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:20.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:21.377 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:20.115508+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:20.490181+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.028482+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.031371+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.051997+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-02-10T08:43:21.379 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:20.115508+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-02-10T08:43:21.380 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:20.490181+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:21.380 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.028482+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:43:21.380 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.031371+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:43:21.380 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.051997+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-02-10T08:43:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:21.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:21.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:20.115508+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-02-10T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:20.490181+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.028482+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.031371+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.051997+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-02-10T08:43:21.509 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:21.527 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:21.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:21.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:43:21.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:21.658 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:21.676 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:21.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:21.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:21.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:21.805 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:21.823 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:21.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:21.954 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:21.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:22.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.048060+0000 osd.3 (osd.3) 227 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.049818+0000 osd.3 (osd.3) 228 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:21.149530+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:21.149832+0000 mon.a (mon.0) 2597 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:21.150339+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:21.150595+0000 mon.a (mon.0) 2598 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.463264+0000 mgr.y (mgr.14553) 423 : cluster [DBG] pgmap v699: 218 pgs: 2 peering, 1 activating+remapped, 215 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/1095 objects misplaced (0.091%); 365 B/s, 57 objects/s recovering 2024-02-10T08:43:22.309 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:21.521862+0000 mgr.y (mgr.14553) 424 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:22.309 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:22.055166+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.048060+0000 osd.3 (osd.3) 227 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.049818+0000 osd.3 (osd.3) 228 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:21.149530+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:21.149832+0000 mon.a (mon.0) 2597 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:21.150339+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:21.150595+0000 mon.a (mon.0) 2598 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.463264+0000 mgr.y (mgr.14553) 423 : cluster [DBG] pgmap v699: 218 pgs: 2 peering, 1 activating+remapped, 215 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/1095 objects misplaced (0.091%); 365 B/s, 57 objects/s recovering 2024-02-10T08:43:22.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:21.521862+0000 mgr.y (mgr.14553) 424 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:22.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:22.055166+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-02-10T08:43:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.048060+0000 osd.3 (osd.3) 227 : cluster [DBG] 60.16 scrub starts 2024-02-10T08:43:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.049818+0000 osd.3 (osd.3) 228 : cluster [DBG] 60.16 scrub ok 2024-02-10T08:43:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:21.149530+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:21.149832+0000 mon.a (mon.0) 2597 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:21.150339+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:21.150595+0000 mon.a (mon.0) 2598 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-71"}]: dispatch 2024-02-10T08:43:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.463264+0000 mgr.y (mgr.14553) 423 : cluster [DBG] pgmap v699: 218 pgs: 2 peering, 1 activating+remapped, 215 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/1095 objects misplaced (0.091%); 365 B/s, 57 objects/s recovering 2024-02-10T08:43:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:21.521862+0000 mgr.y (mgr.14553) 424 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:22.055166+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-02-10T08:43:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- 172.21.15.107:0/785466652 >> v1:172.21.15.107:6790/0 conn(0x7fadc80f67b0 legacy=0x7fadc80f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:23.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- 172.21.15.107:0/785466652 shutdown_connections 2024-02-10T08:43:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- 172.21.15.107:0/785466652 >> 172.21.15.107:0/785466652 conn(0x7fadc80f2510 msgr2=0x7fadc80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- 172.21.15.107:0/785466652 shutdown_connections 2024-02-10T08:43:23.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- 172.21.15.107:0/785466652 wait complete. 2024-02-10T08:43:23.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 librados: starting msgr at 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 librados: starting objecter 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 Processor -- start 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 -- start start 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 librados: setting wanted keys 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.294+0000 7fadcdfdb700 1 librados: calling monclient init 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadcdfdb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc8119560 con 0x7fadc80f72a0 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadcdfdb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc811a740 con 0x7fadc80f67b0 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadcdfdb700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc811b940 con 0x7fadc8106680 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fadc8106680 0x7fadc8117c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47402/0 (socket says 172.21.15.107:47402) 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadbeffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fadc80f72a0 0x7fadc8101fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42662/0 (socket says 172.21.15.107:42662) 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc7fff700 1 -- 172.21.15.107:0/266893806 learned_addr learned my addr 172.21.15.107:0/266893806 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter ms_handle_connect 0x7fadc80f72a0 2024-02-10T08:43:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter ms_handle_connect 0x7fadc8106680 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter ms_handle_connect 0x7fadc80f67b0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158371749 0 0) 0x7fadc8119560 con 0x7fadc80f72a0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb4003030 con 0x7fadc80f72a0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856469881 0 0) 0x7fadc811b940 con 0x7fadc8106680 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc8119560 con 0x7fadc8106680 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801874581 0 0) 0x7fadc811a740 con 0x7fadc80f67b0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc811b940 con 0x7fadc80f67b0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3669031717 0 0) 0x7fadb4003030 con 0x7fadc80f72a0 2024-02-10T08:43:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc811a740 con 0x7fadc80f72a0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.295+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fadb00018c0 con 0x7fadc80f72a0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238617759 0 0) 0x7fadc8119560 con 0x7fadc8106680 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadb4003030 con 0x7fadc8106680 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fadc0001c50 con 0x7fadc8106680 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2907409692 0 0) 0x7fadc811b940 con 0x7fadc80f67b0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc8119560 con 0x7fadc80f67b0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fadb80015d0 con 0x7fadc80f67b0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1159452419 0 0) 0x7fadc811a740 con 0x7fadc80f72a0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 >> v1:172.21.15.107:6790/0 conn(0x7fadc8106680 legacy=0x7fadc8117c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 >> v1:172.21.15.195:6789/0 conn(0x7fadc80f67b0 legacy=0x7fadc8101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc811cb40 con 0x7fadc80f72a0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc811bb30 con 0x7fadc80f72a0 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadcdfdb700 10 client.28176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc811c020 con 0x7fadc80f72a0 2024-02-10T08:43:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadcdfdb700 1 librados: init done 2024-02-10T08:43:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fadb0002460 con 0x7fadc80f72a0 2024-02-10T08:43:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.296+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fadb0005ae0 con 0x7fadc80f72a0 2024-02-10T08:43:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.297+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8004f20 con 0x7fadc80f72a0 2024-02-10T08:43:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.297+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fadb0001fc0 con 0x7fadc80f72a0 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(471..471 src has 330..471) v4 ==== 8094+0+0 (unknown 2376715338 0 0) 0x7fadb0078bd0 con 0x7fadc80f72a0 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 10 client.28176.objecter ms_dispatch 0x7fadc8001550 osd_map(471..471 src has 330..471) v4 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 3 client.28176.objecter handle_osd_map got epochs [471,471] > 0 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 3 client.28176.objecter handle_osd_map decoding full epoch 471 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 20 client.28176.objecter dump_active .. 0 homeless 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.298+0000 7fadc57fa700 10 client.28176.objecter ms_handle_connect 0x7fadb4063d00 2024-02-10T08:43:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.301+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fadb004c000 con 0x7fadc80f72a0 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.355409+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:21.356189+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:22.034528+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:22.036210+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:22.299914+0000 mon.a (mon.0) 2600 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:43:23.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:22.299980+0000 mon.a (mon.0) 2601 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded) 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.026421+0000 osd.3 (osd.3) 229 : cluster [DBG] 60.14 scrub starts 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.027836+0000 osd.3 (osd.3) 230 : cluster [DBG] 60.14 scrub ok 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.056033+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:23.062865+0000 mon.b (mon.1) 610 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:23.063378+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.270309+0000 osd.6 (osd.6) 204 : cluster [DBG] 60.d scrub starts 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.272225+0000 osd.6 (osd.6) 205 : cluster [DBG] 60.d scrub ok 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.355409+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:21.356189+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:22.034528+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:43:23.418 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:22.036210+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:22.299914+0000 mon.a (mon.0) 2600 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:22.299980+0000 mon.a (mon.0) 2601 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded) 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.026421+0000 osd.3 (osd.3) 229 : cluster [DBG] 60.14 scrub starts 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.027836+0000 osd.3 (osd.3) 230 : cluster [DBG] 60.14 scrub ok 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.056033+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:23.062865+0000 mon.b (mon.1) 610 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:23.063378+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.270309+0000 osd.6 (osd.6) 204 : cluster [DBG] 60.d scrub starts 2024-02-10T08:43:23.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.272225+0000 osd.6 (osd.6) 205 : cluster [DBG] 60.d scrub ok 2024-02-10T08:43:23.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:23.481+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fada80051d0 con 0x7fadc80f72a0 2024-02-10T08:43:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:23.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.355409+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:43:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:21.356189+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:43:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:22.034528+0000 osd.4 (osd.4) 224 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:43:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:22.036210+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:22.299914+0000 mon.a (mon.0) 2600 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:22.299980+0000 mon.a (mon.0) 2601 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 257/1095 objects degraded (23.470%), 16 pgs degraded) 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.026421+0000 osd.3 (osd.3) 229 : cluster [DBG] 60.14 scrub starts 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.027836+0000 osd.3 (osd.3) 230 : cluster [DBG] 60.14 scrub ok 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.056033+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:23.062865+0000 mon.b (mon.1) 610 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:23.063378+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.270309+0000 osd.6 (osd.6) 204 : cluster [DBG] 60.d scrub starts 2024-02-10T08:43:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.272225+0000 osd.6 (osd.6) 205 : cluster [DBG] 60.d scrub ok 2024-02-10T08:43:23.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:23.888 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:23.908 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:23.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:23.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:24.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:24.017 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:24.017 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:24.043 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:24.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.055+0000 7fadc57fa700 1 -- 172.21.15.107:0/266893806 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v472) v1 ==== 101+0+0 (unknown 3650257690 0 0) 0x7fadb004a410 con 0x7fadc80f72a0 2024-02-10T08:43:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 10 librados: watch_flush enter 2024-02-10T08:43:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 10 librados: watch_flush exit 2024-02-10T08:43:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 20 client.28176.objecter shutdown clearing up homeless session... 2024-02-10T08:43:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 10 client.28176.objecter successfully canceled tick 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 >> v1:172.21.15.107:6800/2634842009 conn(0x7fadb4063d00 legacy=0x7fadb40661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 >> v1:172.21.15.107:6789/0 conn(0x7fadc80f72a0 legacy=0x7fadc8101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 shutdown_connections 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 >> 172.21.15.107:0/266893806 conn(0x7fadc80f2510 msgr2=0x7fadc8105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 shutdown_connections 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 -- 172.21.15.107:0/266893806 wait complete. 2024-02-10T08:43:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:24.058+0000 7fadcdfdb700 1 librados: shutdown 2024-02-10T08:43:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:43:24.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.143197+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:24.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.143434+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.419300+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.422339+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.463729+0000 mgr.y (mgr.14553) 425 : cluster [DBG] pgmap v702: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%); 392 B/s, 61 objects/s recovering 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:23.483693+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.518107+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:23.521222+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:24.052408+0000 mon.a (mon.0) 2605 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:24.056512+0000 mon.a (mon.0) 2606 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:24.056590+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:24.056633+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-02-10T08:43:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:24.059095+0000 mon.b (mon.1) 611 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.143197+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.143434+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.419300+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.422339+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.463729+0000 mgr.y (mgr.14553) 425 : cluster [DBG] pgmap v702: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%); 392 B/s, 61 objects/s recovering 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:23.483693+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.518107+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:23.521222+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:24.052408+0000 mon.a (mon.0) 2605 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:24.056512+0000 mon.a (mon.0) 2606 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:24.056590+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:24.056633+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-02-10T08:43:24.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:24.059095+0000 mon.b (mon.1) 611 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:24.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.143197+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:43:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.143434+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:43:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.419300+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-02-10T08:43:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.422339+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-02-10T08:43:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.463729+0000 mgr.y (mgr.14553) 425 : cluster [DBG] pgmap v702: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%); 392 B/s, 61 objects/s recovering 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:23.483693+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.518107+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:23.521222+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:24.052408+0000 mon.a (mon.0) 2605 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:24.056512+0000 mon.a (mon.0) 2606 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:24.056590+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.107:0/266893806' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:24.056633+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-02-10T08:43:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:24.059095+0000 mon.b (mon.1) 611 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:24.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:24.859 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:24.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:24.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:24.993 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.013 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:25.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:25.144 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.164 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:25.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:25.295 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:25.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:43:25.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:25.499 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.520 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:25.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:25.635 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:25] "GET /metrics HTTP/1.1" 200 188144 "" "Prometheus/2.33.4" 2024-02-10T08:43:25.650 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.670 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:25.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:25.799 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:25.818 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:25.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:25.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 -- 172.21.15.107:0/800868800 >> v1:172.21.15.107:6789/0 conn(0x7f48d01007f0 legacy=0x7f48d0102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:25.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 -- 172.21.15.107:0/800868800 shutdown_connections 2024-02-10T08:43:25.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 -- 172.21.15.107:0/800868800 >> 172.21.15.107:0/800868800 conn(0x7f48d00f3640 msgr2=0x7f48d00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 -- 172.21.15.107:0/800868800 shutdown_connections 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 -- 172.21.15.107:0/800868800 wait complete. 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 librados: starting msgr at 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 librados: starting objecter 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.884+0000 7f48d5c68700 1 Processor -- start 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 -- start start 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 librados: setting wanted keys 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 librados: calling monclient init 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48d00fcf80 con 0x7f48d00fda50 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48d00fd110 con 0x7f48d0104330 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48d5c68700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48d00fd300 con 0x7f48d01007f0 2024-02-10T08:43:25.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48cf7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f48d00fda50 0x7f48d00fc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42688/0 (socket says 172.21.15.107:42688) 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ceffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f48d01007f0 0x7f48d0114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47416/0 (socket says 172.21.15.107:47416) 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48cf7fe700 1 -- 172.21.15.107:0/2322450137 learned_addr learned my addr 172.21.15.107:0/2322450137 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter ms_handle_connect 0x7f48d00fda50 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter ms_handle_connect 0x7f48d01007f0 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter ms_handle_connect 0x7f48d0104330 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061211420 0 0) 0x7f48d00fcf80 con 0x7f48d00fda50 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b8003030 con 0x7f48d00fda50 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382226330 0 0) 0x7f48d00fd110 con 0x7f48d0104330 2024-02-10T08:43:25.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d00fcf80 con 0x7f48d0104330 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474688815 0 0) 0x7f48d00fd300 con 0x7f48d01007f0 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.885+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d00fd110 con 0x7f48d01007f0 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665950099 0 0) 0x7f48d00fcf80 con 0x7f48d0104330 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48d00fd300 con 0x7f48d0104330 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f48c8001a20 con 0x7f48d0104330 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492338738 0 0) 0x7f48b8003030 con 0x7f48d00fda50 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48d00fcf80 con 0x7f48d00fda50 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422568786 0 0) 0x7f48d00fd110 con 0x7f48d01007f0 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48b8003030 con 0x7f48d01007f0 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f48c4001900 con 0x7f48d00fda50 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f48c0001ed0 con 0x7f48d01007f0 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760083532 0 0) 0x7f48d00fd300 con 0x7f48d0104330 2024-02-10T08:43:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 >> v1:172.21.15.107:6790/0 conn(0x7f48d01007f0 legacy=0x7f48d0114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 >> v1:172.21.15.107:6789/0 conn(0x7f48d00fda50 legacy=0x7f48d00fc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d011cc00 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d0119c30 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48d5c68700 10 client.48484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d011a120 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48d5c68700 1 librados: init done 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f48c8002640 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.886+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f48c80036a0 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc004f20 con 0x7f48d0104330 2024-02-10T08:43:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f48c80021a0 con 0x7f48d0104330 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(473..473 src has 330..473) v4 ==== 8107+0+0 (unknown 317680427 0 0) 0x7f48c80767d0 con 0x7f48d0104330 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48ccff9700 10 client.48484.objecter ms_dispatch 0x7f48d0001550 osd_map(473..473 src has 330..473) v4 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48ccff9700 3 client.48484.objecter handle_osd_map got epochs [473,473] > 0 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.887+0000 7f48ccff9700 3 client.48484.objecter handle_osd_map decoding full epoch 473 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.888+0000 7f48ccff9700 20 client.48484.objecter dump_active .. 0 homeless 2024-02-10T08:43:25.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.888+0000 7f48ccff9700 10 client.48484.objecter ms_handle_connect 0x7f48b8063d60 2024-02-10T08:43:25.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:25.892+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f48c8049b90 con 0x7f48d0104330 2024-02-10T08:43:25.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:26.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:26.090+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248} v 0) v1 -- 0x7f48bc0059e0 con 0x7f48d0104330 2024-02-10T08:43:26.153 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:26.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:26.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:26.284 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:43:26.148+0000 7f4099e6b700 -1 snap_mapper.add_oid found existing snaps mapped on 60:a8690e93:test-rados-api-smithi107-112490-74::foo:24, removing 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:24.670855+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:24.674527+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.060076+0000 mon.a (mon.0) 2609 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:25.157961+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.249379+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.251376+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.464109+0000 mgr.y (mgr.14553) 426 : cluster [DBG] pgmap v705: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%) 2024-02-10T08:43:26.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.476979+0000 osd.5 (osd.5) 217 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.480616+0000 osd.5 (osd.5) 218 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:24.670855+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:24.674527+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.060076+0000 mon.a (mon.0) 2609 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:25.157961+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.249379+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.251376+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.464109+0000 mgr.y (mgr.14553) 426 : cluster [DBG] pgmap v705: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%) 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.476979+0000 osd.5 (osd.5) 217 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:43:26.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.480616+0000 osd.5 (osd.5) 218 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:43:26.287 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:26.384 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:26.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:26.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:26.515 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:26.537 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:43:26.149+0000 7fdb66803700 -1 snap_mapper.add_oid found existing snaps mapped on 60:a8690e93:test-rados-api-smithi107-112490-74::foo:24, removing 2024-02-10T08:43:26.537 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:43:26.149+0000 7fa48ecd7700 -1 snap_mapper.add_oid found existing snaps mapped on 60:a8690e93:test-rados-api-smithi107-112490-74::foo:24, removing 2024-02-10T08:43:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:24.670855+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:43:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:24.674527+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.060076+0000 mon.a (mon.0) 2609 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:25.157961+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.249379+0000 osd.6 (osd.6) 206 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.251376+0000 osd.6 (osd.6) 207 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.464109+0000 mgr.y (mgr.14553) 426 : cluster [DBG] pgmap v705: 218 pgs: 32 unknown, 2 peering, 1 activating+remapped, 183 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/1083 objects misplaced (0.092%) 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.476979+0000 osd.5 (osd.5) 217 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:43:26.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.480616+0000 osd.5 (osd.5) 218 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:43:26.662 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:26.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:26.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:26.793 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:26.812 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:26.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:26.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:26.942 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:26.960 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:26.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:27.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:27.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.070+0000 7f48ccff9700 1 -- 172.21.15.107:0/2322450137 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]=0 reweighted osd.2 to 0.941822 (f11b) v475) v1 ==== 134+0+0 (unknown 2642294749 0 0) 0x7f48c8047fa0 con 0x7f48d0104330 2024-02-10T08:43:27.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.075+0000 7f48d5c68700 10 librados: watch_flush enter 2024-02-10T08:43:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.075+0000 7f48d5c68700 10 librados: watch_flush exit 2024-02-10T08:43:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.075+0000 7f48d5c68700 20 client.48484.objecter shutdown clearing up homeless session... 2024-02-10T08:43:27.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.075+0000 7f48d5c68700 10 client.48484.objecter successfully canceled tick 2024-02-10T08:43:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.078+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 >> v1:172.21.15.107:6800/2634842009 conn(0x7f48b8063d60 legacy=0x7f48b8066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.078+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 >> v1:172.21.15.195:6789/0 conn(0x7f48d0104330 legacy=0x7f48d01184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.078+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 shutdown_connections 2024-02-10T08:43:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.078+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 >> 172.21.15.107:0/2322450137 conn(0x7f48d00f3640 msgr2=0x7f48d01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.080+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 shutdown_connections 2024-02-10T08:43:27.082 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.648611+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:43:27.082 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:25.651861+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:43:27.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.082+0000 7f48d5c68700 1 -- 172.21.15.107:0/2322450137 wait complete. 2024-02-10T08:43:27.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:27.082+0000 7f48d5c68700 1 librados: shutdown 2024-02-10T08:43:27.086 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 0.941822 (f11b) 2024-02-10T08:43:27.095 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:27.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:27.244 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:27.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:27.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.648611+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:43:27.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:25.651861+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:43:27.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:26.069507+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.092468+0000 mon.b (mon.1) 612 : audit [INF] from='client.? v1:172.21.15.107:0/2322450137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.092845+0000 mon.a (mon.0) 2612 : audit [INF] from='client.48484 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.166907+0000 mon.b (mon.1) 613 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.167252+0000 mon.a (mon.0) 2613 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.167733+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:26.167915+0000 mon.a (mon.0) 2614 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.648611+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:25.651861+0000 osd.1 (osd.1) 235 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:26.069507+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.092468+0000 mon.b (mon.1) 612 : audit [INF] from='client.? v1:172.21.15.107:0/2322450137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.092845+0000 mon.a (mon.0) 2612 : audit [INF] from='client.48484 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.166907+0000 mon.b (mon.1) 613 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.167252+0000 mon.a (mon.0) 2613 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.167733+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:26.167915+0000 mon.a (mon.0) 2614 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.393 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:27.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:26.069507+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.092468+0000 mon.b (mon.1) 612 : audit [INF] from='client.? v1:172.21.15.107:0/2322450137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.092845+0000 mon.a (mon.0) 2612 : audit [INF] from='client.48484 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]: dispatch 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.166907+0000 mon.b (mon.1) 613 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.167252+0000 mon.a (mon.0) 2613 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.167733+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:26.167915+0000 mon.a (mon.0) 2614 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-73"}]: dispatch 2024-02-10T08:43:27.544 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.562 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:27.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:27.693 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.817 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:27.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:27.949 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:27.969 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:27.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:28.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:27.069197+0000 mon.a (mon.0) 2615 : audit [INF] from='client.48484 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]': finished 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.069299+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.254154+0000 osd.6 (osd.6) 208 : cluster [DBG] 60.f scrub starts 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.256665+0000 osd.6 (osd.6) 209 : cluster [DBG] 60.f scrub ok 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.377593+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.d scrub starts 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.381242+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.d scrub ok 2024-02-10T08:43:28.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.464673+0000 mgr.y (mgr.14553) 427 : cluster [DBG] pgmap v708: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:28.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:28.070326+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-02-10T08:43:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:28.131 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:28.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:28.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:27.069197+0000 mon.a (mon.0) 2615 : audit [INF] from='client.48484 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]': finished 2024-02-10T08:43:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.069299+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-02-10T08:43:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.254154+0000 osd.6 (osd.6) 208 : cluster [DBG] 60.f scrub starts 2024-02-10T08:43:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.256665+0000 osd.6 (osd.6) 209 : cluster [DBG] 60.f scrub ok 2024-02-10T08:43:28.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.377593+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.d scrub starts 2024-02-10T08:43:28.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.381242+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.d scrub ok 2024-02-10T08:43:28.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.464673+0000 mgr.y (mgr.14553) 427 : cluster [DBG] pgmap v708: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:28.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:28.070326+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-02-10T08:43:28.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:28.076677+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:28.077568+0000 mon.a (mon.0) 2618 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:28.076677+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:28.077568+0000 mon.a (mon.0) 2618 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:27.069197+0000 mon.a (mon.0) 2615 : audit [INF] from='client.48484 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.9418217701173248}]': finished 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.069299+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.254154+0000 osd.6 (osd.6) 208 : cluster [DBG] 60.f scrub starts 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.256665+0000 osd.6 (osd.6) 209 : cluster [DBG] 60.f scrub ok 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.377593+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.d scrub starts 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.381242+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.d scrub ok 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.464673+0000 mgr.y (mgr.14553) 427 : cluster [DBG] pgmap v708: 186 pgs: 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:28.070326+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:28.076677+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:28.077568+0000 mon.a (mon.0) 2618 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:28.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:29.013 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:29.013 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:29.013 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:29.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:29.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:43:29.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:29.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:29.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 -- 172.21.15.107:0/1170693655 >> v1:172.21.15.107:6789/0 conn(0x7f5b300f7300 legacy=0x7f5b30104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 -- 172.21.15.107:0/1170693655 shutdown_connections 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 -- 172.21.15.107:0/1170693655 >> 172.21.15.107:0/1170693655 conn(0x7f5b300f2510 msgr2=0x7f5b300f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 -- 172.21.15.107:0/1170693655 shutdown_connections 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 -- 172.21.15.107:0/1170693655 wait complete. 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 librados: starting msgr at 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.473+0000 7f5b366e0700 1 librados: starting objecter 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 Processor -- start 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 -- start start 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 librados: setting wanted keys 2024-02-10T08:43:29.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 librados: calling monclient init 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b30119540 con 0x7f5b300f6810 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b3011a720 con 0x7f5b30106640 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b366e0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b3011b920 con 0x7f5b300f7300 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5b300f7300 0x7f5b30102fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47430/0 (socket says 172.21.15.107:47430) 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2f7fe700 1 -- 172.21.15.107:0/1240189366 learned_addr learned my addr 172.21.15.107:0/1240189366 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter ms_handle_connect 0x7f5b300f7300 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter ms_handle_connect 0x7f5b300f6810 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604465695 0 0) 0x7f5b30119540 con 0x7f5b300f6810 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003800 con 0x7f5b300f6810 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069693138 0 0) 0x7f5b3011b920 con 0x7f5b300f7300 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30119540 con 0x7f5b300f7300 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter ms_handle_connect 0x7f5b30106640 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.474+0000 7f5b2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351754784 0 0) 0x7f5b3011a720 con 0x7f5b30106640 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b3011b920 con 0x7f5b30106640 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 111658812 0 0) 0x7f5b30119540 con 0x7f5b300f7300 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b3011a720 con 0x7f5b300f7300 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5b24002ea0 con 0x7f5b300f7300 2024-02-10T08:43:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500459404 0 0) 0x7f5b18003800 con 0x7f5b300f6810 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b30119540 con 0x7f5b300f6810 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5b200018c0 con 0x7f5b300f6810 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736007019 0 0) 0x7f5b3011b920 con 0x7f5b30106640 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b18003800 con 0x7f5b30106640 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5b28001d70 con 0x7f5b30106640 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2184843516 0 0) 0x7f5b3011a720 con 0x7f5b300f7300 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.195:6789/0 conn(0x7f5b30106640 legacy=0x7f5b30117c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6789/0 conn(0x7f5b300f6810 legacy=0x7f5b301028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3011cb20 con 0x7f5b300f7300 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b30119730 con 0x7f5b300f7300 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b366e0700 10 client.53774.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b30119c20 con 0x7f5b300f7300 2024-02-10T08:43:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b366e0700 1 librados: init done 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5b24001210 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.475+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5b24003800 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.476+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5b1c004f00 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5b24001820 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(476..476 src has 330..476) v4 ==== 8127+0+0 (unknown 296163125 0 0) 0x7f5b24078fc0 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 osd_map(476..476 src has 330..476) v4 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 3 client.53774.objecter handle_osd_map got epochs [476,476] > 0 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 3 client.53774.objecter handle_osd_map decoding full epoch 476 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 20 client.53774.objecter dump_active .. 0 homeless 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.477+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v476) v1 ==== 54+0+16 (unknown 3066237151 0 2420000192) 0x7f5b2407b3c0 con 0x7f5b300f7300 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b18063cb0 2024-02-10T08:43:29.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 10 client.53774.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 1 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 1 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 0 1 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 4 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 10 client.53774.objecter _send_command 1 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c004950 con 0x7f5b1c004530 2024-02-10T08:43:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c004530 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c004950 con 0x7f5b1c004530 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 1 = system:0 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 4 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.478+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 0 1 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 10 client.53774.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 2 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 2 2024-02-10T08:43:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c0051a0 osd=0 3 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 0 2 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 4 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 10 client.53774.objecter _send_command 2 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.493+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c004d30 con 0x7f5b1c004530 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c004d30 con 0x7f5b1c004530 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 4 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 0 2 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 10 client.53774.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 3 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 1 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 4 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 10 client.53774.objecter _send_command 3 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c00c000 con 0x7f5b1c004150 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.495+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c004150 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.496+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c00c000 con 0x7f5b1c004150 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.496+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.496+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 3 = system:0 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.496+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 4 2024-02-10T08:43:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.496+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 1 3 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 10 client.53774.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 4 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 4 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c003fe0 osd=1 3 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 1 4 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 4 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 10 client.53774.objecter _send_command 4 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.510+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c0018b0 con 0x7f5b1c004150 2024-02-10T08:43:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.511+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c0018b0 con 0x7f5b1c004150 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.511+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.511+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 4 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 1 4 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 10 client.53774.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 5 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 5 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 2 5 2024-02-10T08:43:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 4 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 10 client.53774.objecter _send_command 5 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c00de90 con 0x7f5b1c00d2f0 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c00d2f0 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c00de90 con 0x7f5b1c00d2f0 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 5 = system:0 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 4 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.512+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 2 5 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 10 client.53774.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 6 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 6 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c003630 osd=2 3 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 2 6 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 4 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 10 client.53774.objecter _send_command 6 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.527+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c003160 con 0x7f5b1c00d2f0 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c003160 con 0x7f5b1c00d2f0 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 4 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 2 6 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 10 client.53774.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 7 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 7 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 3 7 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 4 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 10 client.53774.objecter _send_command 7 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.528+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c015b90 con 0x7f5b1c011ff0 2024-02-10T08:43:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c011ff0 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c015b90 con 0x7f5b1c011ff0 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 7 = system:0 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 4 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.529+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 3 7 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 10 client.53774.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 8 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 8 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c011970 osd=3 3 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 3 8 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 4 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 10 client.53774.objecter _send_command 8 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.544+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c00e240 con 0x7f5b1c011ff0 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c00e240 con 0x7f5b1c011ff0 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 4 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 3 8 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 10 client.53774.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 9 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 9 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 4 9 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 4 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 10 client.53774.objecter _send_command 9 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.545+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6800/2499671285 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c00e530 con 0x7f5b1c00ce50 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c00ce50 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.4 v1:172.21.15.195:6800/2499671285 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c00e530 con 0x7f5b1c00ce50 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:43:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 9 = system:0 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 4 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.546+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 4 9 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 10 client.53774.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 10 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 10 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c00e240 osd=4 3 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 4 10 2024-02-10T08:43:29.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 4 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 10 client.53774.objecter _send_command 10 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.561+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6800/2499671285 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c00e530 con 0x7f5b1c00ce50 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.4 v1:172.21.15.195:6800/2499671285 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c00e530 con 0x7f5b1c00ce50 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 4 2024-02-10T08:43:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 4 10 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 10 client.53774.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 11 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 11 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 5 11 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 4 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 10 client.53774.objecter _send_command 11 2024-02-10T08:43:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.563+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c016ec0 con 0x7f5b1c016950 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c016950 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c016ec0 con 0x7f5b1c016950 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 11 = system:0 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 4 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.564+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 5 11 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 10 client.53774.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 12 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 12 2024-02-10T08:43:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c00e530 osd=5 3 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 5 12 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 4 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 10 client.53774.objecter _send_command 12 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.579+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c016ec0 con 0x7f5b1c016950 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c016ec0 con 0x7f5b1c016950 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 4 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 5 12 2024-02-10T08:43:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 10 client.53774.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 13 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 13 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 6 13 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 4 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 10 client.53774.objecter _send_command 13 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c022ca0 con 0x7f5b1c01f190 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.581+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c01f190 2024-02-10T08:43:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.582+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c022ca0 con 0x7f5b1c01f190 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.582+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.582+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 13 = system:0 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.582+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 4 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.582+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 6 13 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 10 client.53774.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 14 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 14 2024-02-10T08:43:29.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c016ec0 osd=6 3 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 6 14 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 4 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 10 client.53774.objecter _send_command 14 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.596+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c024420 con 0x7f5b1c01f190 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c024420 con 0x7f5b1c01f190 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 4 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 6 14 2024-02-10T08:43:29.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 10 client.53774.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 15 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 15 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 7 15 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 4 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 10 client.53774.objecter _send_command 15 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.598+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1c025030 con 0x7f5b1c024aa0 2024-02-10T08:43:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.599+0000 7f5b2d7fa700 10 client.53774.objecter ms_handle_connect 0x7f5b1c024aa0 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.599+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1c025030 con 0x7f5b1c024aa0 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.599+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.599+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 15 = system:0 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.600+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 4 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.600+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 7 15 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 10 client.53774.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign -1 16 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 20 client.53774.objecter _get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_remove -1 16 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 20 client.53774.objecter get_session s=0x7f5b1c024420 osd=7 3 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 15 client.53774.objecter _session_command_op_assign 7 16 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 4 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 10 client.53774.objecter _send_command 16 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.614+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1c025030 con 0x7f5b1c024aa0 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b2d7fa700 1 -- 172.21.15.107:0/1240189366 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1c025030 con 0x7f5b1c024aa0 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b2d7fa700 10 client.53774.objecter ms_dispatch 0x7f5b30001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b2d7fa700 10 client.53774.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b2d7fa700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 4 2024-02-10T08:43:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b2d7fa700 15 client.53774.objecter _session_command_op_remove 7 16 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 librados: watch_flush enter 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 librados: watch_flush exit 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.0 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6801/3321596972 conn(0x7f5b1c004530 legacy=0x7f5b1c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c0051a0 osd=0 1 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.1 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6805/133516126 conn(0x7f5b1c004150 legacy=0x7f5b1c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003fe0 osd=1 1 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.2 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6809/3575881728 conn(0x7f5b1c00d2f0 legacy=0x7f5b1c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c003630 osd=2 1 2024-02-10T08:43:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.3 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6813/2871825691 conn(0x7f5b1c011ff0 legacy=0x7f5b1c0143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c011970 osd=3 1 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.4 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.195:6800/2499671285 conn(0x7f5b1c00ce50 legacy=0x7f5b1c015d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e240 osd=4 1 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.5 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.195:6804/4178321486 conn(0x7f5b1c016950 legacy=0x7f5b1c01d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c00e530 osd=5 1 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.6 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.195:6808/1417479017 conn(0x7f5b1c01f190 legacy=0x7f5b1c021580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b34c7d700 1 -- 172.21.15.107:0/1240189366 reap_dead start 2024-02-10T08:43:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c016ec0 osd=6 1 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter close_session for osd.7 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.195:6812/1934176928 conn(0x7f5b1c024aa0 legacy=0x7f5b1c022ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b34c7d700 1 -- 172.21.15.107:0/1240189366 reap_dead start 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter put_session s=0x7f5b1c024420 osd=7 1 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 20 client.53774.objecter shutdown clearing up homeless session... 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 10 client.53774.objecter successfully canceled tick 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b34c7d700 1 -- 172.21.15.107:0/1240189366 reap_dead start 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b34c7d700 1 -- 172.21.15.107:0/1240189366 reap_dead start 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5b18063cb0 legacy=0x7f5b18066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> v1:172.21.15.107:6790/0 conn(0x7f5b300f7300 legacy=0x7f5b30102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 shutdown_connections 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 >> 172.21.15.107:0/1240189366 conn(0x7f5b300f2510 msgr2=0x7f5b30109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 shutdown_connections 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 -- 172.21.15.107:0/1240189366 wait complete. 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:28.616+0000 7f5b366e0700 1 librados: shutdown 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:43:29.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:29.084 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:29.169 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.297167+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:43:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.297945+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:43:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.602757+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:43:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:27.605827+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:28.079566+0000 mon.a (mon.0) 2619 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:28.479089+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.107:0/1240189366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:28.878942+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:28.880846+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:29.073874+0000 mon.a (mon.0) 2620 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.073972+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:29.083571+0000 mon.b (mon.1) 616 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.297167+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.297945+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.602757+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:27.605827+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:43:29.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:28.079566+0000 mon.a (mon.0) 2619 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:28.479089+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.107:0/1240189366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:28.878942+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:28.880846+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:29.073874+0000 mon.a (mon.0) 2620 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.073972+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-02-10T08:43:29.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:29.083571+0000 mon.b (mon.1) 616 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.297167+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.297945+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.602757+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:27.605827+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:28.079566+0000 mon.a (mon.0) 2619 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T08:43:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:28.479089+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.107:0/1240189366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:28.878942+0000 osd.4 (osd.4) 228 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:43:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:28.880846+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:43:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:29.073874+0000 mon.a (mon.0) 2620 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.073972+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-02-10T08:43:29.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:29.083571+0000 mon.b (mon.1) 616 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:29.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:43:29.837 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:29.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:29.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:29.969 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:30.068 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:30.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:30.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.080144+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.e starting backfill to osd.3 from (0'0,0'0] MAX to 142'13 2024-02-10T08:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.283159+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-10T08:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.286732+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-10T08:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.465125+0000 mgr.y (mgr.14553) 428 : cluster [DBG] pgmap v711: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.840789+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.d scrub starts 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:29.844450+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.d scrub ok 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.074729+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.080144+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.e starting backfill to osd.3 from (0'0,0'0] MAX to 142'13 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.283159+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.286732+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.465125+0000 mgr.y (mgr.14553) 428 : cluster [DBG] pgmap v711: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.840789+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.d scrub starts 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:29.844450+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.d scrub ok 2024-02-10T08:43:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.074729+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.080144+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.e starting backfill to osd.3 from (0'0,0'0] MAX to 142'13 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.283159+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.286732+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.465125+0000 mgr.y (mgr.14553) 428 : cluster [DBG] pgmap v711: 218 pgs: 32 unknown, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 108 KiB/s rd, 511 B/s wr, 180 op/s; 16 B/s, 7 objects/s recovering 2024-02-10T08:43:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.840789+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.d scrub starts 2024-02-10T08:43:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:29.844450+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.d scrub ok 2024-02-10T08:43:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.074729+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-02-10T08:43:30.847 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:30.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:30.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:30.983 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.004 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:31.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:31.136 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.156 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:31.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.089321+0000 osd.3 (osd.3) 231 : cluster [DBG] 60.e scrub starts 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.090934+0000 osd.3 (osd.3) 232 : cluster [DBG] 60.e scrub ok 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:30.510421+0000 mon.a (mon.0) 2623 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.558288+0000 osd.5 (osd.5) 219 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.561291+0000 osd.5 (osd.5) 220 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.055722+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-02-10T08:43:31.285 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:31.074798+0000 mon.b (mon.1) 617 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:31.286 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.305 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:31.375 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:31.379 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:43:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:43:31.379 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:43:31.379 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:43:31.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.089321+0000 osd.3 (osd.3) 231 : cluster [DBG] 60.e scrub starts 2024-02-10T08:43:31.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.090934+0000 osd.3 (osd.3) 232 : cluster [DBG] 60.e scrub ok 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:30.510421+0000 mon.a (mon.0) 2623 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.558288+0000 osd.5 (osd.5) 219 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.561291+0000 osd.5 (osd.5) 220 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.055722+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:31.074798+0000 mon.b (mon.1) 617 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.089321+0000 osd.3 (osd.3) 231 : cluster [DBG] 60.e scrub starts 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.090934+0000 osd.3 (osd.3) 232 : cluster [DBG] 60.e scrub ok 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:30.510421+0000 mon.a (mon.0) 2623 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.558288+0000 osd.5 (osd.5) 219 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:43:31.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.561291+0000 osd.5 (osd.5) 220 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:43:31.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.055722+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-02-10T08:43:31.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:31.074798+0000 mon.b (mon.1) 617 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:31.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:43:31.435 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.454 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:43:31.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:31.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:31.587 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:43:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:31.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:31.812 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:31.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:31.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:32.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:32.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:32.147 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:32.347 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:32.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.240602+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:43:32.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:30.241401+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:31.074997+0000 mgr.y (mgr.14553) 429 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.260203+0000 osd.6 (osd.6) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.263670+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.465872+0000 mgr.y (mgr.14553) 430 : cluster [DBG] pgmap v714: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:31.532546+0000 mgr.y (mgr.14553) 431 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.240602+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:30.241401+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:31.074997+0000 mgr.y (mgr.14553) 429 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.260203+0000 osd.6 (osd.6) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:43:32.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.263670+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:43:32.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.465872+0000 mgr.y (mgr.14553) 430 : cluster [DBG] pgmap v714: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:43:32.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:31.532546+0000 mgr.y (mgr.14553) 431 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:32.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:32.353 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:32.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.240602+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:30.241401+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:31.074997+0000 mgr.y (mgr.14553) 429 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1"}]: dispatch 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.260203+0000 osd.6 (osd.6) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.263670+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.465872+0000 mgr.y (mgr.14553) 430 : cluster [DBG] pgmap v714: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.5 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T08:43:32.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:31.532546+0000 mgr.y (mgr.14553) 431 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:32.481 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:32.501 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:32.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:32.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:32.632 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:32.651 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:32.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:32.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:32.800 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:32.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:32.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:32.936 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:33.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.280402+0000 osd.7 (osd.7) 201 : cluster [DBG] 219.1 deep-scrub starts 2024-02-10T08:43:33.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:31.284163+0000 osd.7 (osd.7) 202 : cluster [DBG] 219.1 deep-scrub ok 2024-02-10T08:43:33.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:32.418451+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:32.421381+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:32.847081+0000 mon.c (mon.2) 404 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:33.114026+0000 osd.3 (osd.3) 233 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:33.115533+0000 osd.3 (osd.3) 234 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.280402+0000 osd.7 (osd.7) 201 : cluster [DBG] 219.1 deep-scrub starts 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:31.284163+0000 osd.7 (osd.7) 202 : cluster [DBG] 219.1 deep-scrub ok 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:32.418451+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:32.421381+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:32.847081+0000 mon.c (mon.2) 404 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:33.114026+0000 osd.3 (osd.3) 233 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:43:33.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:33.115533+0000 osd.3 (osd.3) 234 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:43:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.280402+0000 osd.7 (osd.7) 201 : cluster [DBG] 219.1 deep-scrub starts 2024-02-10T08:43:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:31.284163+0000 osd.7 (osd.7) 202 : cluster [DBG] 219.1 deep-scrub ok 2024-02-10T08:43:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:32.418451+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:43:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:32.421381+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:43:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:32.847081+0000 mon.c (mon.2) 404 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:43:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:33.114026+0000 osd.3 (osd.3) 233 : cluster [DBG] 60.1d scrub starts 2024-02-10T08:43:33.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:33.115533+0000 osd.3 (osd.3) 234 : cluster [DBG] 60.1d scrub ok 2024-02-10T08:43:33.590 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:33.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:33.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:33.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:43:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:33.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:33.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:33.889 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:43:33.889 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:43:33.889 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T08:43:33.889 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T08:43:33.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4 2024-02-10T08:43:34.205 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:33 smithi195 systemd[1]: Stopping Ceph osd.4 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:43:34.206 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:33 smithi195 bash[125371]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:33.466234+0000 mgr.y (mgr.14553) 432 : cluster [DBG] pgmap v715: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 1.9 KiB/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:33.672840+0000 mon.c (mon.2) 405 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:33.673473+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:33.677868+0000 mon.a (mon.0) 2625 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:34.094678+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:43:34.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:34.096203+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:43:34.321 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:34.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:34.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:33.466234+0000 mgr.y (mgr.14553) 432 : cluster [DBG] pgmap v715: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 1.9 KiB/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:33.672840+0000 mon.c (mon.2) 405 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:33.673473+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:33.677868+0000 mon.a (mon.0) 2625 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:34.094678+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:34.096203+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:43:34.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:33.466234+0000 mgr.y (mgr.14553) 432 : cluster [DBG] pgmap v715: 218 pgs: 1 peering, 217 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 1.9 KiB/s wr, 3 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T08:43:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:33.672840+0000 mon.c (mon.2) 405 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:43:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:33.673473+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:43:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:33.677868+0000 mon.a (mon.0) 2625 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:34.094678+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:43:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:34.096203+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:43:34.451 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:34.537 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:43:34.204+0000 7fdb7f969700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:43:34.537 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:43:34.204+0000 7fdb7f969700 -1 osd.4 479 *** Got signal Terminated *** 2024-02-10T08:43:34.537 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[73798]: debug 2024-02-10T08:43:34.204+0000 7fdb7f969700 -1 osd.4 479 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:43:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:34.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:34.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- 172.21.15.107:0/3005645466 >> v1:172.21.15.107:6789/0 conn(0x7f066c0ff710 legacy=0x7f066c101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- 172.21.15.107:0/3005645466 shutdown_connections 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- 172.21.15.107:0/3005645466 >> 172.21.15.107:0/3005645466 conn(0x7f066c0f2510 msgr2=0x7f066c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- 172.21.15.107:0/3005645466 shutdown_connections 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- 172.21.15.107:0/3005645466 wait complete. 2024-02-10T08:43:34.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 librados: starting msgr at 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 librados: starting objecter 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 Processor -- start 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 -- start start 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 librados: setting wanted keys 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.894+0000 7f06734a1700 1 librados: calling monclient init 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f06734a1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f066c119480 con 0x7f066c0ff710 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f06734a1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f066c11a620 con 0x7f066c103250 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f06734a1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f066c11b820 con 0x7f066c0fc910 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f0670a3c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f066c0ff710 0x7f066c0f8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33734/0 (socket says 172.21.15.107:33734) 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f067123d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f066c0fc910 0x7f066c0f84a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60394/0 (socket says 172.21.15.107:60394) 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f0670a3c700 1 -- 172.21.15.107:0/2544591097 learned_addr learned my addr 172.21.15.107:0/2544591097 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter ms_handle_connect 0x7f066c0fc910 2024-02-10T08:43:34.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter ms_handle_connect 0x7f066c0ff710 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter ms_handle_connect 0x7f066c103250 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377708308 0 0) 0x7f066c11b820 con 0x7f066c0fc910 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0658003b70 con 0x7f066c0fc910 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457726473 0 0) 0x7f066c11a620 con 0x7f066c103250 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c11b820 con 0x7f066c103250 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760401098 0 0) 0x7f066c119480 con 0x7f066c0ff710 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c11a620 con 0x7f066c0ff710 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244213009 0 0) 0x7f0658003b70 con 0x7f066c0fc910 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f066c119480 con 0x7f066c0fc910 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0664001e00 con 0x7f066c0fc910 2024-02-10T08:43:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 15418966 0 0) 0x7f066c11a620 con 0x7f066c0ff710 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0658003b70 con 0x7f066c0ff710 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333498148 0 0) 0x7f066c11b820 con 0x7f066c103250 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f066c11a620 con 0x7f066c103250 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.895+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0668001370 con 0x7f066c0ff710 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f06600017c0 con 0x7f066c103250 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2668619763 0 0) 0x7f066c119480 con 0x7f066c0fc910 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 >> v1:172.21.15.195:6789/0 conn(0x7f066c103250 legacy=0x7f066c117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 >> v1:172.21.15.107:6789/0 conn(0x7f066c0ff710 legacy=0x7f066c0f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c11ca20 con 0x7f066c0fc910 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c119670 con 0x7f066c0fc910 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f06734a1700 10 client.53780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c119ae0 con 0x7f066c0fc910 2024-02-10T08:43:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f06734a1700 1 librados: init done 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0664002840 con 0x7f066c0fc910 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.896+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0664005de0 con 0x7f066c0fc910 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.897+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650004f20 con 0x7f066c0fc910 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.897+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0664018d50 con 0x7f066c0fc910 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(479..479 src has 330..479) v4 ==== 8107+0+0 (unknown 1632907259 0 0) 0x7f0664079050 con 0x7f066c0fc910 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 10 client.53780.objecter ms_dispatch 0x7f066c001550 osd_map(479..479 src has 330..479) v4 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 3 client.53780.objecter handle_osd_map got epochs [479,479] > 0 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 3 client.53780.objecter handle_osd_map decoding full epoch 479 2024-02-10T08:43:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 20 client.53780.objecter dump_active .. 0 homeless 2024-02-10T08:43:34.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.898+0000 7f065e7fc700 10 client.53780.objecter ms_handle_connect 0x7f0658063db0 2024-02-10T08:43:34.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:34.902+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f066404c410 con 0x7f066c0fc910 2024-02-10T08:43:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:34.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:34.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:35.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:35.048 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:35.069 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:35.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:35.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.086+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f06500051d0 con 0x7f066c0fc910 2024-02-10T08:43:35.154 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:34.206038+0000 mon.a (mon.0) 2626 : cluster [INF] osd.4 marked itself down and dead 2024-02-10T08:43:35.154 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:34.881443+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:43:35.154 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:34.883358+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:43:35.154 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:35.088348+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2544591097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.154 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:35.088596+0000 mon.a (mon.0) 2627 : audit [INF] from='client.53780 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.160+0000 7f065e7fc700 1 -- 172.21.15.107:0/2544591097 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v480) v1 ==== 87+0+0 (unknown 1110693298 0 0) 0x7f066404a820 con 0x7f066c0fc910 2024-02-10T08:43:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.163+0000 7f06734a1700 10 librados: watch_flush enter 2024-02-10T08:43:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.163+0000 7f06734a1700 10 librados: watch_flush exit 2024-02-10T08:43:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.163+0000 7f06734a1700 20 client.53780.objecter shutdown clearing up homeless session... 2024-02-10T08:43:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.163+0000 7f06734a1700 10 client.53780.objecter successfully canceled tick 2024-02-10T08:43:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.166+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0658063db0 legacy=0x7f0658066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.166+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 >> v1:172.21.15.107:6790/0 conn(0x7f066c0fc910 legacy=0x7f066c0f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.170+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 shutdown_connections 2024-02-10T08:43:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.170+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 >> 172.21.15.107:0/2544591097 conn(0x7f066c0f2510 msgr2=0x7f066c106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:35.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.172+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 shutdown_connections 2024-02-10T08:43:35.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.175+0000 7f06734a1700 1 -- 172.21.15.107:0/2544591097 wait complete. 2024-02-10T08:43:35.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.175+0000 7f06734a1700 1 librados: shutdown 2024-02-10T08:43:35.179 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:43:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:35.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:34.206038+0000 mon.a (mon.0) 2626 : cluster [INF] osd.4 marked itself down and dead 2024-02-10T08:43:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:34.881443+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:43:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:34.883358+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:43:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:35.088348+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2544591097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:35.088596+0000 mon.a (mon.0) 2627 : audit [INF] from='client.53780 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:35.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:34.206038+0000 mon.a (mon.0) 2626 : cluster [INF] osd.4 marked itself down and dead 2024-02-10T08:43:35.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:34.881443+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:43:35.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:34.883358+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:43:35.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:35.088348+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.107:0/2544591097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:35.088596+0000 mon.a (mon.0) 2627 : audit [INF] from='client.53780 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:43:35.487 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:35.508 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:35.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:35.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:35.644 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:35.787 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:35 smithi195 bash[125412]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4 2024-02-10T08:43:35.842 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:35] "GET /metrics HTTP/1.1" 200 188144 "" "Prometheus/2.33.4" 2024-02-10T08:43:35.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:35.973 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:35.992 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:35.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.994+0000 7fc8be628700 1 -- 172.21.15.107:0/1244617382 >> v1:172.21.15.107:6790/0 conn(0x7fc8b80fc910 legacy=0x7fc8b80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 -- 172.21.15.107:0/1244617382 shutdown_connections 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 -- 172.21.15.107:0/1244617382 >> 172.21.15.107:0/1244617382 conn(0x7fc8b80f2510 msgr2=0x7fc8b80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 -- 172.21.15.107:0/1244617382 shutdown_connections 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 -- 172.21.15.107:0/1244617382 wait complete. 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 librados: starting msgr at 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 librados: starting objecter 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 Processor -- start 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 -- start start 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 librados: setting wanted keys 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.995+0000 7fc8be628700 1 librados: calling monclient init 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8be628700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc8b8061820 con 0x7fc8b80ff710 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8be628700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc8b80619f0 con 0x7fc8b80fc910 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8be628700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc8b8119790 con 0x7fc8b8103250 2024-02-10T08:43:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8bcbc5700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc8b8103250 0x7fc8b8115ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60414/0 (socket says 172.21.15.107:60414) 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b7fff700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc8b80fc910 0x7fc8b8060d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:53878/0 (socket says 172.21.15.107:53878) 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8bcbc5700 1 -- 172.21.15.107:0/386640316 learned_addr learned my addr 172.21.15.107:0/386640316 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b77fe700 1 --1- 172.21.15.107:0/386640316 >> v1:172.21.15.107:6789/0 conn(0x7fc8b80ff710 0x7fc8b81128c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33762/0 (socket says 172.21.15.107:33762) 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter ms_handle_connect 0x7fc8b80ff710 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter ms_handle_connect 0x7fc8b8103250 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter ms_handle_connect 0x7fc8b80fc910 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478726117 0 0) 0x7fc8b8061820 con 0x7fc8b80ff710 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0003800 con 0x7fc8b80ff710 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199613247 0 0) 0x7fc8b8119790 con 0x7fc8b8103250 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8061820 con 0x7fc8b8103250 2024-02-10T08:43:36.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461582014 0 0) 0x7fc8b810a610 con 0x7fc8b80fc910 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8119790 con 0x7fc8b80fc910 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.996+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3176700433 0 0) 0x7fc8a0003800 con 0x7fc8b80ff710 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b810a610 con 0x7fc8b80ff710 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915585205 0 0) 0x7fc8b8061820 con 0x7fc8b8103250 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a0003800 con 0x7fc8b8103250 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682344691 0 0) 0x7fc8b8119790 con 0x7fc8b80fc910 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b8061820 con 0x7fc8b80fc910 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc8ac0018c0 con 0x7fc8b80ff710 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc8b0001b80 con 0x7fc8b8103250 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc8a8002ca0 con 0x7fc8b80fc910 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3660420232 0 0) 0x7fc8b810a610 con 0x7fc8b80ff710 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 >> v1:172.21.15.107:6790/0 conn(0x7fc8b8103250 legacy=0x7fc8b8115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 >> v1:172.21.15.195:6789/0 conn(0x7fc8b80fc910 legacy=0x7fc8b8060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b811a990 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8119980 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8be628700 10 client.28200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8119e70 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8be628700 1 librados: init done 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fc8ac002000 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.997+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fc8ac0059d0 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.998+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc8a4004f00 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.998+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fc8ac001b60 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(480..480 src has 330..480) v4 ==== 8107+0+0 (unknown 739359136 0 0) 0x7fc8ac078c90 con 0x7fc8b80ff710 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 osd_map(480..480 src has 330..480) v4 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 3 client.28200.objecter handle_osd_map got epochs [480,480] > 0 2024-02-10T08:43:36.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 3 client.28200.objecter handle_osd_map decoding full epoch 480 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 20 client.28200.objecter dump_active .. 0 homeless 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v480) v1 ==== 54+0+16 (unknown 1016706674 0 2420000192) 0x7fc8ac07b090 con 0x7fc8b80ff710 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a0063e60 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 10 client.28200.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 1 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 1 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 0 1 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 4 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 10 client.28200.objecter _send_command 1 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a4004950 con 0x7fc8a4004530 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:35.999+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a4004530 2024-02-10T08:43:36.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.000+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a4004950 con 0x7fc8a4004530 2024-02-10T08:43:36.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.000+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:36.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.000+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 1 = system:0 2024-02-10T08:43:36.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.000+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 4 2024-02-10T08:43:36.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.000+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 0 1 2024-02-10T08:43:36.014 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:36.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 10 client.28200.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 2 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 2 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a40051a0 osd=0 3 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 0 2 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 4 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 10 client.28200.objecter _send_command 2 2024-02-10T08:43:36.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.016+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a4004d30 con 0x7fc8a4004530 2024-02-10T08:43:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a4004d30 con 0x7fc8a4004530 2024-02-10T08:43:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a40051a0 osd=0 4 2024-02-10T08:43:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 0 2 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 10 client.28200.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 3 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 3 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 1 3 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 4 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 10 client.28200.objecter _send_command 3 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a400c000 con 0x7fc8a4004150 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.017+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a4004150 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.018+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a400c000 con 0x7fc8a4004150 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.018+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.018+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 3 = system:0 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.018+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 4 2024-02-10T08:43:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.018+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 1 3 2024-02-10T08:43:36.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 10 client.28200.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 4 2024-02-10T08:43:36.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 4 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a4003fe0 osd=1 3 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 1 4 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 4 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 10 client.28200.objecter _send_command 4 2024-02-10T08:43:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.045+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a40018b0 con 0x7fc8a4004150 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a40018b0 con 0x7fc8a4004150 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a4003fe0 osd=1 4 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 1 4 2024-02-10T08:43:36.048 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 10 client.28200.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 5 2024-02-10T08:43:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 5 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 2 5 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 4 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 10 client.28200.objecter _send_command 5 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a400de60 con 0x7fc8a400d2f0 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.047+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a400d2f0 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.048+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a400de60 con 0x7fc8a400d2f0 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.048+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.048+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 5 = system:0 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.048+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 4 2024-02-10T08:43:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.048+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 2 5 2024-02-10T08:43:36.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 10 client.28200.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 6 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 6 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a4003630 osd=2 3 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 2 6 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 4 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 10 client.28200.objecter _send_command 6 2024-02-10T08:43:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.076+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a40030a0 con 0x7fc8a400d2f0 2024-02-10T08:43:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a40030a0 con 0x7fc8a400d2f0 2024-02-10T08:43:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a4003630 osd=2 4 2024-02-10T08:43:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 2 6 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 10 client.28200.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 7 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 7 2024-02-10T08:43:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 3 7 2024-02-10T08:43:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 4 2024-02-10T08:43:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 10 client.28200.objecter _send_command 7 2024-02-10T08:43:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.082+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a4015aa0 con 0x7fc8a4011e80 2024-02-10T08:43:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.086+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a4011e80 2024-02-10T08:43:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.090+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a4015aa0 con 0x7fc8a4011e80 2024-02-10T08:43:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.090+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.090+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 7 = system:0 2024-02-10T08:43:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.090+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 4 2024-02-10T08:43:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.090+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 3 7 2024-02-10T08:43:36.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 10 client.28200.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 8 2024-02-10T08:43:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 8 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a40034b0 osd=3 3 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 3 8 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 4 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 10 client.28200.objecter _send_command 8 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.116+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a400e210 con 0x7fc8a4011e80 2024-02-10T08:43:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a400e210 con 0x7fc8a4011e80 2024-02-10T08:43:36.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a40034b0 osd=3 4 2024-02-10T08:43:36.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 3 8 2024-02-10T08:43:36.118 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 10 client.28200.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 9 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter _get_session osd=-1 returning homeless 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 10 client.28200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=481}) v3 -- 0x7fc8a400dfd0 con 0x7fc8b80ff710 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8a400c940 con 0x7fc8b80ff710 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 10 ==== osd_map(481..481 src has 330..481) v4 ==== 328+0+0 (unknown 4004032896 0 0) 0x7fc8ac051c30 con 0x7fc8b80ff710 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 osd_map(481..481 src has 330..481) v4 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 3 client.28200.objecter handle_osd_map got epochs [481,481] > 480 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 3 client.28200.objecter handle_osd_map decoding incremental epoch 481 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter checking command 9 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter _check_command_map_dne tid 9 current 481 map_dne_bound 0 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 20 client.28200.objecter dump_active .. 1 homeless 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== mon.0 v1:172.21.15.107:6789/0 11 ==== mon_get_version_reply(handle=1 version=481) v2 ==== 24+0+0 (unknown 1214815589 0 0) 0x7fc8ac04bc40 con 0x7fc8b80ff710 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b67fc700 10 client.28200.objecter _check_command_map_dne tid 9 current 481 map_dne_bound 481 2024-02-10T08:43:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b67fc700 10 client.28200.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b67fc700 15 client.28200.objecter _session_command_op_remove -1 9 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 10 client.28200.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 10 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 10 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 5 10 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 4 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 10 client.28200.objecter _send_command 10 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a400cf20 con 0x7fc8a4016150 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.117+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a4016150 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.118+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a400cf20 con 0x7fc8a4016150 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.118+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:43:36.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.118+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 10 = system:0 2024-02-10T08:43:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.118+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 4 2024-02-10T08:43:36.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.118+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 5 10 2024-02-10T08:43:36.126 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 10 client.28200.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 11 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 11 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a400cdb0 osd=5 3 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 5 11 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 4 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 10 client.28200.objecter _send_command 11 2024-02-10T08:43:36.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.134+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a400cf20 con 0x7fc8a4016150 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a400cf20 con 0x7fc8a4016150 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a400cdb0 osd=5 4 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 5 11 2024-02-10T08:43:36.137 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 10 client.28200.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 12 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 12 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 6 12 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 4 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 10 client.28200.objecter _send_command 12 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.136+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a401ef70 con 0x7fc8a401b460 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a401b460 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a401ef70 con 0x7fc8a401b460 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 12 = system:0 2024-02-10T08:43:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 4 2024-02-10T08:43:36.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.137+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 6 12 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 10 client.28200.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 13 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 13 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a400cf20 osd=6 3 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 6 13 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 4 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 10 client.28200.objecter _send_command 13 2024-02-10T08:43:36.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.154+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a401ef70 con 0x7fc8a401b460 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a401ef70 con 0x7fc8a401b460 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a400cf20 osd=6 4 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 6 13 2024-02-10T08:43:36.158 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 10 client.28200.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 14 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 14 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 7 14 2024-02-10T08:43:36.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 4 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 10 client.28200.objecter _send_command 14 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.156+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8a40211f0 con 0x7fc8a4021da0 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.157+0000 7fc8b57fa700 10 client.28200.objecter ms_handle_connect 0x7fc8a4021da0 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.158+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fc8a40211f0 con 0x7fc8a4021da0 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.158+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.158+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 14 = system:0 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.158+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 4 2024-02-10T08:43:36.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.158+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 7 14 2024-02-10T08:43:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 10 client.28200.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign -1 15 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 20 client.28200.objecter _get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 15 client.28200.objecter _session_command_op_remove -1 15 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 20 client.28200.objecter get_session s=0x7fc8a401ef70 osd=7 3 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 15 client.28200.objecter _session_command_op_assign 7 15 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 4 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 10 client.28200.objecter _send_command 15 2024-02-10T08:43:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.172+0000 7fc8be628700 1 -- 172.21.15.107:0/386640316 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc8a40211f0 con 0x7fc8a4021da0 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.173+0000 7fc8b57fa700 1 -- 172.21.15.107:0/386640316 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fc8a40211f0 con 0x7fc8a4021da0 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.174+0000 7fc8b57fa700 10 client.28200.objecter ms_dispatch 0x7fc8b8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.174+0000 7fc8b57fa700 10 client.28200.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.174+0000 7fc8b57fa700 20 client.28200.objecter put_session s=0x7fc8a401ef70 osd=7 4 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:36.174+0000 7fc8b57fa700 15 client.28200.objecter _session_command_op_remove 7 15 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:43:36.273 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:36.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:36.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:35.151119+0000 mon.a (mon.0) 2628 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:35.151136+0000 mon.a (mon.0) 2629 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:35.158229+0000 mon.a (mon.0) 2630 : audit [INF] from='client.53780 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:35.158291+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:35.466768+0000 mgr.y (mgr.14553) 433 : cluster [DBG] pgmap v717: 218 pgs: 1 stale+peering, 31 stale+active+clean, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 1.7 KiB/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:36.000567+0000 mon.a (mon.0) 2632 : audit [DBG] from='client.? v1:172.21.15.107:0/386640316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:36.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:36.057856+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-10T08:43:36.409 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:36.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:35.151119+0000 mon.a (mon.0) 2628 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:36.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:35.151136+0000 mon.a (mon.0) 2629 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T08:43:36.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:35.158229+0000 mon.a (mon.0) 2630 : audit [INF] from='client.53780 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:43:36.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:35.158291+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:35.466768+0000 mgr.y (mgr.14553) 433 : cluster [DBG] pgmap v717: 218 pgs: 1 stale+peering, 31 stale+active+clean, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 1.7 KiB/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:36.000567+0000 mon.a (mon.0) 2632 : audit [DBG] from='client.? v1:172.21.15.107:0/386640316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:36.057856+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:35.151119+0000 mon.a (mon.0) 2628 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:35.151136+0000 mon.a (mon.0) 2629 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:35.158229+0000 mon.a (mon.0) 2630 : audit [INF] from='client.53780 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:35.158291+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:35.466768+0000 mgr.y (mgr.14553) 433 : cluster [DBG] pgmap v717: 218 pgs: 1 stale+peering, 31 stale+active+clean, 186 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 1.7 KiB/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:36.000567+0000 mon.a (mon.0) 2632 : audit [DBG] from='client.? v1:172.21.15.107:0/386640316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:36.057856+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-10T08:43:36.557 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:36.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:36.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:37.040 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:37.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:37.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:37.170 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:37.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T08:43:37.350 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:37 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4.service: Succeeded. 2024-02-10T08:43:37.350 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:43:37 smithi195 systemd[1]: Stopped Ceph osd.4 for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:43:37.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:43:37.371 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T08:43:37.371 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-02-10T08:43:37.371 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi195.front.sepia.ceph.com 2024-02-10T08:43:37.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.4 --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-02-10T08:43:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:37.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:37.616 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/osd.4/config 2024-02-10T08:43:37.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:37.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:37.518705+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check failed: Reduced data availability: 6 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:37.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:37.518737+0000 mon.a (mon.0) 2635 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1107 objects degraded (8.582%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:37.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:37.518705+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check failed: Reduced data availability: 6 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:37.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:37.518737+0000 mon.a (mon.0) 2635 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1107 objects degraded (8.582%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:37.749 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:37.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:37.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:37.518705+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check failed: Reduced data availability: 6 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-10T08:43:37.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:37.518737+0000 mon.a (mon.0) 2635 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1107 objects degraded (8.582%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:37.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:37.881 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:38.279 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:38.279 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:37.467527+0000 mgr.y (mgr.14553) 434 : cluster [DBG] pgmap v719: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 159 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:38.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:37.467527+0000 mgr.y (mgr.14553) 434 : cluster [DBG] pgmap v719: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 159 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:38.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:37.467527+0000 mgr.y (mgr.14553) 434 : cluster [DBG] pgmap v719: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 159 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:38.930 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:38.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:38.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:43:39.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:39.061 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:43:39.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:43:39.490 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:39.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:39.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:39.738 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:39.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:39.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:39.869 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:39.966 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:39.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:40.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:43:40.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:40.101 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:40.119 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:40.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:40.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:40.251 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:40.271 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:40.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:40.370 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:40.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:40.402 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:40.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:39.468008+0000 mgr.y (mgr.14553) 435 : cluster [DBG] pgmap v720: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:40.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:40.159002+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:39.468008+0000 mgr.y (mgr.14553) 435 : cluster [DBG] pgmap v720: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:40.159002+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:39.468008+0000 mgr.y (mgr.14553) 435 : cluster [DBG] pgmap v720: 218 pgs: 4 active+undersized+degraded, 2 active+undersized, 36 activating+undersized, 50 peering, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 95/1107 objects degraded (8.582%) 2024-02-10T08:43:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:40.159002+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:41.150 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:41.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:41.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:41.281 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:41.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:43:41.830 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:41.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:41.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.a 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.9 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.8 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.2 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.1c 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.1 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:60.15 2024-02-10T08:43:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:219.e 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.8 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.16 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.2 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.1 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.4 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:4.9 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.15 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:60.d 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:219.0 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:5.e 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:2.f 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:60.5 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:2.d 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:5.c 2024-02-10T08:43:41.939 INFO:teuthology.orchestra.run.smithi195.stdout:2.8 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:5.5 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:4.e 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:219.10 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:219.1a 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:4.13 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:16.0 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:60.1b 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:16.3 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:16.1e 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:2.6 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:5.a 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:16.19 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:4.1 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:4.b 2024-02-10T08:43:41.940 INFO:teuthology.orchestra.run.smithi195.stdout:3.1 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:2.5 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:16.2 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:4.18 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:6.14 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:60.7 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:4.f 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:2.18 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:2.1f 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:6.1a 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:2.1a 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:5.7 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:5.0 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:16.15 2024-02-10T08:43:41.941 INFO:teuthology.orchestra.run.smithi195.stdout:16.b 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:2.10 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:60.e 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:16.7 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:16.d 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:219.d 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:219.12 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.1a 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.1b 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.1c 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:60.18 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.5 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:6.17 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.c 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:4.1d 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:5.10 2024-02-10T08:43:41.942 INFO:teuthology.orchestra.run.smithi195.stdout:2.11 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:60.11 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:4.d 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:4.2 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:5.11 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:5.1 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:2.13 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:5.9 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:5.f 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:6.15 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:60.14 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:16.18 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:6.16 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:6.1d 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:6.1e 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:60.4 2024-02-10T08:43:41.943 INFO:teuthology.orchestra.run.smithi195.stdout:16.e 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.2 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.f 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.6 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.12 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.7 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:16.c 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:6.e 2024-02-10T08:43:41.944 INFO:teuthology.orchestra.run.smithi195.stdout:60.12 2024-02-10T08:43:41.961 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:42.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:41.468962+0000 mgr.y (mgr.14553) 436 : cluster [DBG] pgmap v721: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:42.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:41.543218+0000 mgr.y (mgr.14553) 437 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:41.468962+0000 mgr.y (mgr.14553) 436 : cluster [DBG] pgmap v721: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:41.543218+0000 mgr.y (mgr.14553) 437 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:42.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:41.468962+0000 mgr.y (mgr.14553) 436 : cluster [DBG] pgmap v721: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 127 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:42.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:41.543218+0000 mgr.y (mgr.14553) 437 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:42.780 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:42.781 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:43.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.4 --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 4.1 --file /var/log/ceph/exp.4.1.4 2024-02-10T08:43:43.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:42.056215+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 4 pgs peering) 2024-02-10T08:43:43.339 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:42.056215+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 4 pgs peering) 2024-02-10T08:43:43.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:43:43.378 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/osd.4/config 2024-02-10T08:43:43.384 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:42.056215+0000 mon.a (mon.0) 2637 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 4 pgs peering) 2024-02-10T08:43:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.585+0000 7fbe9b724700 1 -- 172.21.15.107:0/2840842121 >> v1:172.21.15.107:6789/0 conn(0x7fbe941007f0 legacy=0x7fbe94102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.585+0000 7fbe9b724700 1 -- 172.21.15.107:0/2840842121 shutdown_connections 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.585+0000 7fbe9b724700 1 -- 172.21.15.107:0/2840842121 >> 172.21.15.107:0/2840842121 conn(0x7fbe940f3640 msgr2=0x7fbe940f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.585+0000 7fbe9b724700 1 -- 172.21.15.107:0/2840842121 shutdown_connections 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.585+0000 7fbe9b724700 1 -- 172.21.15.107:0/2840842121 wait complete. 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 librados: starting msgr at 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 librados: starting objecter 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 Processor -- start 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 -- start start 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 librados: setting wanted keys 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 librados: calling monclient init 2024-02-10T08:43:43.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe941194c0 con 0x7fbe94104330 2024-02-10T08:43:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe9411a660 con 0x7fbe941007f0 2024-02-10T08:43:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe9b724700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe9411b840 con 0x7fbe940fda50 2024-02-10T08:43:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe98cbf700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbe941007f0 0x7fbe940fcdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:35248/0 (socket says 172.21.15.107:35248) 2024-02-10T08:43:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe98cbf700 1 -- 172.21.15.107:0/1213225315 learned_addr learned my addr 172.21.15.107:0/1213225315 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe8a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe941007f0 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe8a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe94104330 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.586+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232077245 0 0) 0x7fbe9411a660 con 0x7fbe941007f0 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe80003030 con 0x7fbe941007f0 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe940fda50 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931324934 0 0) 0x7fbe9411b840 con 0x7fbe940fda50 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9411a660 con 0x7fbe940fda50 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327407365 0 0) 0x7fbe941194c0 con 0x7fbe94104330 2024-02-10T08:43:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9411b840 con 0x7fbe94104330 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1544891245 0 0) 0x7fbe80003030 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe941194c0 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbe84002c20 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3382574838 0 0) 0x7fbe941194c0 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 >> v1:172.21.15.107:6790/0 conn(0x7fbe940fda50 legacy=0x7fbe940fc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 >> v1:172.21.15.107:6789/0 conn(0x7fbe94104330 legacy=0x7fbe94117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9411ca20 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe941196b0 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe9b724700 10 client.48508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe94119b20 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe9b724700 1 librados: init done 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbe84001aa0 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.587+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbe84003880 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.588+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe78004f50 con 0x7fbe941007f0 2024-02-10T08:43:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbe84002050 con 0x7fbe941007f0 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(481..481 src has 330..481) v4 ==== 8107+0+0 (unknown 3355375063 0 0) 0x7fbe84079060 con 0x7fbe941007f0 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 10 client.48508.objecter ms_dispatch 0x7fbe94001550 osd_map(481..481 src has 330..481) v4 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 3 client.48508.objecter handle_osd_map got epochs [481,481] > 0 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 3 client.48508.objecter handle_osd_map decoding full epoch 481 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 20 client.48508.objecter dump_active .. 0 homeless 2024-02-10T08:43:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.589+0000 7fbe8a7fc700 10 client.48508.objecter ms_handle_connect 0x7fbe80063aa0 2024-02-10T08:43:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.593+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbe8404c4e0 con 0x7fbe941007f0 2024-02-10T08:43:43.607 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:43.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:43.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:43.744 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:43.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:43.780+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fbe78005200 con 0x7fbe941007f0 2024-02-10T08:43:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.071+0000 7fbe8a7fc700 1 -- 172.21.15.107:0/1213225315 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v482) v1 ==== 97+0+0 (unknown 1624849408 0 0) 0x7fbe8404a950 con 0x7fbe941007f0 2024-02-10T08:43:44.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.075+0000 7fbe9b724700 10 librados: watch_flush enter 2024-02-10T08:43:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.086+0000 7fbe9b724700 10 librados: watch_flush exit 2024-02-10T08:43:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.086+0000 7fbe9b724700 20 client.48508.objecter shutdown clearing up homeless session... 2024-02-10T08:43:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.086+0000 7fbe9b724700 10 client.48508.objecter successfully canceled tick 2024-02-10T08:43:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.087+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbe80063aa0 legacy=0x7fbe80065f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:44.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.087+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 >> v1:172.21.15.195:6789/0 conn(0x7fbe941007f0 legacy=0x7fbe940fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:44.092 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:44.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.090+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 shutdown_connections 2024-02-10T08:43:44.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.090+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 >> 172.21.15.107:0/1213225315 conn(0x7fbe940f3640 msgr2=0x7fbe940f6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:44.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:44.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.097+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 shutdown_connections 2024-02-10T08:43:44.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.102+0000 7fbe9b724700 1 -- 172.21.15.107:0/1213225315 wait complete. 2024-02-10T08:43:44.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:44.102+0000 7fbe9b724700 1 librados: shutdown 2024-02-10T08:43:44.107 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:43:44.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:44.231 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:44.357 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:43.469620+0000 mgr.y (mgr.14553) 438 : cluster [DBG] pgmap v722: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:43.781538+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.107:0/1213225315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:43.781766+0000 mon.a (mon.0) 2638 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:43.469620+0000 mgr.y (mgr.14553) 438 : cluster [DBG] pgmap v722: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:44.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:43.781538+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.107:0/1213225315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:43.781766+0000 mon.a (mon.0) 2638 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:44.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:44.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:44.490 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:44.518 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:43.469620+0000 mgr.y (mgr.14553) 438 : cluster [DBG] pgmap v722: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:44.519 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:43.781538+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.107:0/1213225315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.519 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:43.781766+0000 mon.a (mon.0) 2638 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:44.519 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:43:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:44.063352+0000 mon.a (mon.0) 2639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:44.071191+0000 mon.a (mon.0) 2640 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:45.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:44.071257+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-10T08:43:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:44.063352+0000 mon.a (mon.0) 2639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:44.071191+0000 mon.a (mon.0) 2640 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:45.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:44.071257+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-10T08:43:45.244 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:45.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:44.063352+0000 mon.a (mon.0) 2639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:44.071191+0000 mon.a (mon.0) 2640 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:44.071257+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-10T08:43:45.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:45.378 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:45.630 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:45.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:45.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:45.813 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:45.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:45.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:45] "GET /metrics HTTP/1.1" 200 188444 "" "Prometheus/2.33.4" 2024-02-10T08:43:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:46.067 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:46.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:46.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:46.234 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 -- 172.21.15.107:0/2057680518 >> v1:172.21.15.107:6790/0 conn(0x7f844c0fda50 legacy=0x7f844c0fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 -- 172.21.15.107:0/2057680518 shutdown_connections 2024-02-10T08:43:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 -- 172.21.15.107:0/2057680518 >> 172.21.15.107:0/2057680518 conn(0x7f844c0f3640 msgr2=0x7f844c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 -- 172.21.15.107:0/2057680518 shutdown_connections 2024-02-10T08:43:46.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 -- 172.21.15.107:0/2057680518 wait complete. 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 librados: starting msgr at 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 librados: starting objecter 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.382+0000 7f8451ce2700 1 Processor -- start 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 -- start start 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 librados: setting wanted keys 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 librados: calling monclient init 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c0fd000 con 0x7f844c0fda50 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c0fd1d0 con 0x7f844c1007f0 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8451ce2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c0fd3e0 con 0x7f844c104330 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f844bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f844c104330 0x7f844c1184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38942/0 (socket says 172.21.15.107:38942) 2024-02-10T08:43:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f844bfff700 1 -- 172.21.15.107:0/2671901168 learned_addr learned my addr 172.21.15.107:0/2671901168 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f844b7fe700 1 --1- 172.21.15.107:0/2671901168 >> v1:172.21.15.107:6789/0 conn(0x7f844c0fda50 0x7f844c0fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39846/0 (socket says 172.21.15.107:39846) 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter ms_handle_connect 0x7f844c104330 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter ms_handle_connect 0x7f844c1007f0 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter ms_handle_connect 0x7f844c0fda50 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795562898 0 0) 0x7f844c0fd3e0 con 0x7f844c104330 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8434003800 con 0x7f844c104330 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272079646 0 0) 0x7f844c0fd000 con 0x7f844c0fda50 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c0fd3e0 con 0x7f844c0fda50 2024-02-10T08:43:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938265531 0 0) 0x7f844c0fd1d0 con 0x7f844c1007f0 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c0fd000 con 0x7f844c1007f0 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28014485 0 0) 0x7f844c0fd3e0 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c0fd1d0 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f843c0015b0 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431315593 0 0) 0x7f8434003800 con 0x7f844c104330 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c0fd3e0 con 0x7f844c104330 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.383+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8444002c90 con 0x7f844c104330 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2877866802 0 0) 0x7f844c0fd1d0 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 >> v1:172.21.15.107:6790/0 conn(0x7f844c104330 legacy=0x7f844c1184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 >> v1:172.21.15.195:6789/0 conn(0x7f844c1007f0 legacy=0x7f844c114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c11cc00 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c119c30 con 0x7f844c0fda50 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8451ce2700 10 client.28212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c11a0a0 con 0x7f844c0fda50 2024-02-10T08:43:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8451ce2700 1 librados: init done 2024-02-10T08:43:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f843c002560 con 0x7f844c0fda50 2024-02-10T08:43:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.384+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f843c003920 con 0x7f844c0fda50 2024-02-10T08:43:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.385+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8438004f00 con 0x7f844c0fda50 2024-02-10T08:43:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.385+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f843c003b60 con 0x7f844c0fda50 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(483..483 src has 330..483) v4 ==== 8107+0+0 (unknown 365697992 0 0) 0x7f843c079140 con 0x7f844c0fda50 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 osd_map(483..483 src has 330..483) v4 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 3 client.28212.objecter handle_osd_map got epochs [483,483] > 0 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 3 client.28212.objecter handle_osd_map decoding full epoch 483 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 20 client.28212.objecter dump_active .. 0 homeless 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v483) v1 ==== 54+0+16 (unknown 598699401 0 2420000192) 0x7f843c07b540 con 0x7f844c0fda50 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f84340639a0 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 10 client.28212.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 1 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 1 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 0 1 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 4 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 10 client.28212.objecter _send_command 1 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8438004950 con 0x7f8438004530 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.386+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f8438004530 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.387+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8438004950 con 0x7f8438004530 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.387+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.387+0000 7f8448ff9700 10 client.28212.objecter _finish_command 1 = system:0 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.387+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 4 2024-02-10T08:43:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.387+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 0 1 2024-02-10T08:43:46.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 10 client.28212.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 2 2024-02-10T08:43:46.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 2 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f84380051a0 osd=0 3 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 0 2 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 4 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 10 client.28212.objecter _send_command 2 2024-02-10T08:43:46.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.402+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8438004d30 con 0x7f8438004530 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.404+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f8438004d30 con 0x7f8438004530 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.404+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.404+0000 7f8448ff9700 10 client.28212.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.404+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f84380051a0 osd=0 4 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.404+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 0 2 2024-02-10T08:43:46.405 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 10 client.28212.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 3 2024-02-10T08:43:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 1 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 4 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 10 client.28212.objecter _send_command 3 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f843800c000 con 0x7f8438004150 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f8438004150 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f843800c000 con 0x7f8438004150 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 10 client.28212.objecter _finish_command 3 = system:0 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 4 2024-02-10T08:43:46.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.405+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 1 3 2024-02-10T08:43:46.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 10 client.28212.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 4 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 4 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003fe0 osd=1 3 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 1 4 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 4 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 10 client.28212.objecter _send_command 4 2024-02-10T08:43:46.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.420+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8438001850 con 0x7f8438004150 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f8438001850 con 0x7f8438004150 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 10 client.28212.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003fe0 osd=1 4 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 1 4 2024-02-10T08:43:46.422 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:46.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 10 client.28212.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 5 2024-02-10T08:43:46.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 5 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 2 5 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 4 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 10 client.28212.objecter _send_command 5 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8438003e70 con 0x7f843800d2f0 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f843800d2f0 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8438003e70 con 0x7f843800d2f0 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.422+0000 7f8448ff9700 10 client.28212.objecter _finish_command 5 = system:0 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.423+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 4 2024-02-10T08:43:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.423+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 2 5 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:45.470151+0000 mgr.y (mgr.14553) 439 : cluster [DBG] pgmap v724: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:46.056085+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: Degraded data redundancy: 177/1107 objects degraded (15.989%), 39 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:46.061729+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:45.470151+0000 mgr.y (mgr.14553) 439 : cluster [DBG] pgmap v724: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:46.056085+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: Degraded data redundancy: 177/1107 objects degraded (15.989%), 39 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:46.061729+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 10 client.28212.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 6 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 6 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003630 osd=2 3 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 2 6 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 4 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 10 client.28212.objecter _send_command 6 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.437+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f843800e6e0 con 0x7f843800d2f0 2024-02-10T08:43:46.438 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.438+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f843800e6e0 con 0x7f843800d2f0 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.438+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.438+0000 7f8448ff9700 10 client.28212.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.438+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003630 osd=2 4 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.438+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 2 6 2024-02-10T08:43:46.439 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 10 client.28212.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 7 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 7 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 3 7 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 4 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 10 client.28212.objecter _send_command 7 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8438015b40 con 0x7f8438011e80 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f8438011e80 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8438015b40 con 0x7f8438011e80 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 10 client.28212.objecter _finish_command 7 = system:0 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 4 2024-02-10T08:43:46.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.439+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 3 7 2024-02-10T08:43:46.442 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:45.470151+0000 mgr.y (mgr.14553) 439 : cluster [DBG] pgmap v724: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 B/s wr, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:46.442 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:46.056085+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: Degraded data redundancy: 177/1107 objects degraded (15.989%), 39 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:46.442 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:46.061729+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-10T08:43:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:46.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 10 client.28212.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 8 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 8 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438003440 osd=3 3 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 3 8 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 4 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 10 client.28212.objecter _send_command 8 2024-02-10T08:43:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.453+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f843800e150 con 0x7f8438011e80 2024-02-10T08:43:46.455 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:46.455 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f843800e150 con 0x7f8438011e80 2024-02-10T08:43:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 10 client.28212.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438003440 osd=3 4 2024-02-10T08:43:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 3 8 2024-02-10T08:43:46.456 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 10 client.28212.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 9 2024-02-10T08:43:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 20 client.28212.objecter _get_session osd=-1 returning homeless 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 10 client.28212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=484}) v3 -- 0x7f843800e310 con 0x7f844c0fda50 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f843800c940 con 0x7f844c0fda50 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_get_version_reply(handle=1 version=483) v2 ==== 24+0+0 (unknown 2845634664 0 0) 0x7f843c04d050 con 0x7f844c0fda50 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8449ffb700 10 client.28212.objecter _check_command_map_dne tid 9 current 483 map_dne_bound 483 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8449ffb700 10 client.28212.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.455+0000 7f8449ffb700 15 client.28212.objecter _session_command_op_remove -1 9 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 10 client.28212.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 10 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 10 2024-02-10T08:43:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 5 10 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 4 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 10 client.28212.objecter _send_command 10 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f843800cf20 con 0x7f84380161f0 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.456+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f84380161f0 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.457+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f843800cf20 con 0x7f84380161f0 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.457+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.457+0000 7f8448ff9700 10 client.28212.objecter _finish_command 10 = system:0 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.457+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 4 2024-02-10T08:43:46.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.457+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 5 10 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 10 client.28212.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 11 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 11 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f843800cdb0 osd=5 3 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 5 11 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 4 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 10 client.28212.objecter _send_command 11 2024-02-10T08:43:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.471+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f843800d110 con 0x7f84380161f0 2024-02-10T08:43:46.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f843800d110 con 0x7f84380161f0 2024-02-10T08:43:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8448ff9700 10 client.28212.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f843800cdb0 osd=5 4 2024-02-10T08:43:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 5 11 2024-02-10T08:43:46.474 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 10 client.28212.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 12 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 12 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 6 12 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 4 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 10 client.28212.objecter _send_command 12 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.473+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f843801f0a0 con 0x7f843801b500 2024-02-10T08:43:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f843801b500 2024-02-10T08:43:46.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f843801f0a0 con 0x7f843801b500 2024-02-10T08:43:46.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:43:46.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 10 client.28212.objecter _finish_command 12 = system:0 2024-02-10T08:43:46.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 4 2024-02-10T08:43:46.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.474+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 6 12 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 10 client.28212.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 13 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 13 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f843800d110 osd=6 3 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 6 13 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 4 2024-02-10T08:43:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 10 client.28212.objecter _send_command 13 2024-02-10T08:43:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.489+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f843801f0a0 con 0x7f843801b500 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f843801f0a0 con 0x7f843801b500 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8448ff9700 10 client.28212.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f843800d110 osd=6 4 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 6 13 2024-02-10T08:43:46.491 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 10 client.28212.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 14 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 14 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 7 14 2024-02-10T08:43:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 4 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 10 client.28212.objecter _send_command 14 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.490+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8438021380 con 0x7f8438020ed0 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.491+0000 7f8448ff9700 10 client.28212.objecter ms_handle_connect 0x7f8438020ed0 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.492+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8438021380 con 0x7f8438020ed0 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.492+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.492+0000 7f8448ff9700 10 client.28212.objecter _finish_command 14 = system:0 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.492+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 4 2024-02-10T08:43:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.492+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 7 14 2024-02-10T08:43:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 10 client.28212.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign -1 15 2024-02-10T08:43:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 20 client.28212.objecter _get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_remove -1 15 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 20 client.28212.objecter get_session s=0x7f8438020850 osd=7 3 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 15 client.28212.objecter _session_command_op_assign 7 15 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 4 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 10 client.28212.objecter _send_command 15 2024-02-10T08:43:46.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.506+0000 7f8451ce2700 1 -- 172.21.15.107:0/2671901168 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8438021380 con 0x7f8438020ed0 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.508+0000 7f8448ff9700 1 -- 172.21.15.107:0/2671901168 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f8438021380 con 0x7f8438020ed0 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.508+0000 7f8448ff9700 10 client.28212.objecter ms_dispatch 0x7f844c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.508+0000 7f8448ff9700 10 client.28212.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.508+0000 7f8448ff9700 20 client.28212.objecter put_session s=0x7f8438020850 osd=7 4 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:46.508+0000 7f8448ff9700 15 client.28212.objecter _session_command_op_remove 7 15 2024-02-10T08:43:46.509 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:43:46.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:46.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:46.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:46.723 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:46.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:46.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:46.955 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:46.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:47.083 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:46.386431+0000 mon.a (mon.0) 2644 : audit [DBG] from='client.? v1:172.21.15.107:0/2671901168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:47.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Exporting 4.1 info 4.1( v 473'28 (0'0,473'28] local-lis/les=57/58 n=7 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:80542eaa:::obj_delete_at_hint.0000000068:head# 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:80b6fe23:reshard::reshard.0000000011:head# 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:80d23011:::obj_delete_at_hint.0000000085:head# 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:8411e274:::obj_delete_at_hint.0000000073:head# 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:84d19476:::obj_delete_at_hint.0000000038:head# 2024-02-10T08:43:47.115 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:856b9b26:reshard::reshard.0000000002:head# 2024-02-10T08:43:47.116 INFO:teuthology.orchestra.run.smithi195.stderr:Read #4:86cc8a85:::obj_delete_at_hint.0000000118:head# 2024-02-10T08:43:47.116 INFO:teuthology.orchestra.run.smithi195.stderr:Export successful 2024-02-10T08:43:47.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:43:47.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:43:47.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:46.386431+0000 mon.a (mon.0) 2644 : audit [DBG] from='client.? v1:172.21.15.107:0/2671901168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:43:47.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:43:47.453 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:43:47.458 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:47.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:47.482 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:46.386431+0000 mon.a (mon.0) 2644 : audit [DBG] from='client.? v1:172.21.15.107:0/2671901168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:47.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:47.589 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:48.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:47.471168+0000 mgr.y (mgr.14553) 440 : cluster [DBG] pgmap v726: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:48.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:47.471168+0000 mgr.y (mgr.14553) 440 : cluster [DBG] pgmap v726: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:48.347 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:47.471168+0000 mgr.y (mgr.14553) 440 : cluster [DBG] pgmap v726: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:48.938 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:48.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:49.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:49.077 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:49.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.4 --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 4.1 2024-02-10T08:43:49.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:43:49.579 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:43:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:43:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:43:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:43:49.579 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:43:49.628 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:49.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:49.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:49.761 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:49.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.859+0000 7f9a51ba9700 1 -- 172.21.15.107:0/4208844137 >> v1:172.21.15.107:6790/0 conn(0x7f9a4c0ff710 legacy=0x7f9a4c101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:49.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.859+0000 7f9a51ba9700 1 -- 172.21.15.107:0/4208844137 shutdown_connections 2024-02-10T08:43:49.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.859+0000 7f9a51ba9700 1 -- 172.21.15.107:0/4208844137 >> 172.21.15.107:0/4208844137 conn(0x7f9a4c0f2510 msgr2=0x7f9a4c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.859+0000 7f9a51ba9700 1 -- 172.21.15.107:0/4208844137 shutdown_connections 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.859+0000 7f9a51ba9700 1 -- 172.21.15.107:0/4208844137 wait complete. 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 librados: starting msgr at 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 librados: starting objecter 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 Processor -- start 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 -- start start 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 librados: setting wanted keys 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 librados: calling monclient init 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c116150 con 0x7f9a4c0fc910 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c117330 con 0x7f9a4c0ff710 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a51ba9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c118510 con 0x7f9a4c103250 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a4b7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9a4c0fc910 0x7f9a4c0f8700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51252/0 (socket says 172.21.15.107:51252) 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.860+0000 7f9a4b7fe700 1 -- 172.21.15.107:0/2446623200 learned_addr learned my addr 172.21.15.107:0/2446623200 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter ms_handle_connect 0x7f9a4c103250 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:49.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter ms_handle_connect 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter ms_handle_connect 0x7f9a4c0ff710 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599251345 0 0) 0x7f9a4c116150 con 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a380038e0 con 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244569376 0 0) 0x7f9a4c118510 con 0x7f9a4c103250 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c116150 con 0x7f9a4c103250 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480620395 0 0) 0x7f9a4c117330 con 0x7f9a4c0ff710 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c118510 con 0x7f9a4c0ff710 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911773473 0 0) 0x7f9a380038e0 con 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a4c117330 con 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a3c001d70 con 0x7f9a4c0fc910 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2971697696 0 0) 0x7f9a4c116150 con 0x7f9a4c103250 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a380038e0 con 0x7f9a4c103250 2024-02-10T08:43:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084124235 0 0) 0x7f9a4c118510 con 0x7f9a4c0ff710 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a4c116150 con 0x7f9a4c0ff710 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a440017c0 con 0x7f9a4c103250 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a340012f0 con 0x7f9a4c0ff710 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 436196460 0 0) 0x7f9a4c117330 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 >> v1:172.21.15.107:6790/0 conn(0x7f9a4c103250 legacy=0x7f9a4c1148d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 >> v1:172.21.15.195:6789/0 conn(0x7f9a4c0ff710 legacy=0x7f9a4c0f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c119710 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c116340 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a51ba9700 10 client.28218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c1167b0 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a51ba9700 1 librados: init done 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9a3c0028d0 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.861+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a3c005fa0 con 0x7f9a4c0fc910 2024-02-10T08:43:49.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.862+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c004f20 con 0x7f9a4c0fc910 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9a3c002430 con 0x7f9a4c0fc910 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(483..483 src has 330..483) v4 ==== 8107+0+0 (unknown 365697992 0 0) 0x7f9a3c079160 con 0x7f9a4c0fc910 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 10 client.28218.objecter ms_dispatch 0x7f9a4c001550 osd_map(483..483 src has 330..483) v4 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 3 client.28218.objecter handle_osd_map got epochs [483,483] > 0 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 3 client.28218.objecter handle_osd_map decoding full epoch 483 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 20 client.28218.objecter dump_active .. 0 homeless 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.863+0000 7f9a48ff9700 10 client.28218.objecter ms_handle_connect 0x7f9a38065ef0 2024-02-10T08:43:49.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:49.867+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9a3c04c520 con 0x7f9a4c0fc910 2024-02-10T08:43:49.912 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:49.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:49.967 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:50.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:50.046 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.067+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f9a2c0051d0 con 0x7f9a4c0fc910 2024-02-10T08:43:50.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:43:50.158 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/osd.4/config 2024-02-10T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:50.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:50.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:50.374 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:50.401 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:43:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:43:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.533+0000 7f9a48ff9700 1 -- 172.21.15.107:0/2446623200 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v484) v1 ==== 91+0+0 (unknown 353045805 0 0) 0x7f9a3c002a40 con 0x7f9a4c0fc910 2024-02-10T08:43:50.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.536+0000 7f9a51ba9700 10 librados: watch_flush enter 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.543+0000 7f9a51ba9700 10 librados: watch_flush exit 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.543+0000 7f9a51ba9700 20 client.28218.objecter shutdown clearing up homeless session... 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.543+0000 7f9a51ba9700 10 client.28218.objecter successfully canceled tick 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.543+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9a38065ef0 legacy=0x7f9a380683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.543+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 >> v1:172.21.15.107:6789/0 conn(0x7f9a4c0fc910 legacy=0x7f9a4c0f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.550+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 shutdown_connections 2024-02-10T08:43:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.550+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 >> 172.21.15.107:0/2446623200 conn(0x7f9a4c0f2510 msgr2=0x7f9a4c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:50.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.562+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 shutdown_connections 2024-02-10T08:43:50.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.573+0000 7f9a51ba9700 1 -- 172.21.15.107:0/2446623200 wait complete. 2024-02-10T08:43:50.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:50.577+0000 7f9a51ba9700 1 librados: shutdown 2024-02-10T08:43:50.621 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:43:50.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:49.471749+0000 mgr.y (mgr.14553) 441 : cluster [DBG] pgmap v727: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:50.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:50.068650+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:50.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:49.471749+0000 mgr.y (mgr.14553) 441 : cluster [DBG] pgmap v727: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:50.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:50.068650+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:49.471749+0000 mgr.y (mgr.14553) 441 : cluster [DBG] pgmap v727: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:50.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:50.068650+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:43:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:50.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:50.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:50.931 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:50.970 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:50.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:51.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:51.125 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:51.180 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:43:51.171+0000 7f409be6f700 -1 snap_mapper.add_oid found existing snaps mapped on 219:81388422:test-rados-api-smithi107-112490-76::foo:2, removing 2024-02-10T08:43:51.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:43:51.171+0000 7faf2346d700 -1 snap_mapper.add_oid found existing snaps mapped on 219:81388422:test-rados-api-smithi107-112490-76::foo:2, removing 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:50.524878+0000 mon.a (mon.0) 2646 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:50.533657+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:50.533712+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:50.686025+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:50.689504+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:51.065685+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:51.173687+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:51.180992+0000 mon.a (mon.0) 2650 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:51.181572+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:51.583 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:51.181864+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:51.583 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:43:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:43:51.801 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:43:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:50.524878+0000 mon.a (mon.0) 2646 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:50.533657+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:50.533712+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-10T08:43:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:50.686025+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:43:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:50.689504+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:51.065685+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:51.173687+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:51.180992+0000 mon.a (mon.0) 2650 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:51.181572+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:51.181864+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:50.524878+0000 mon.a (mon.0) 2646 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:50.533657+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? v1:172.21.15.107:0/2446623200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:50.533712+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:50.686025+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:50.689504+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:51.065685+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-10T08:43:51.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:51.173687+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:51.180992+0000 mon.a (mon.0) 2650 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:51.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:51.181572+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:51.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:51.181864+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-75"}]: dispatch 2024-02-10T08:43:52.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.547+0000 7f92c488b700 1 -- 172.21.15.107:0/3626806318 >> v1:172.21.15.107:6790/0 conn(0x7f92c00f89c0 legacy=0x7f92c00f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.547+0000 7f92c488b700 1 -- 172.21.15.107:0/3626806318 shutdown_connections 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.547+0000 7f92c488b700 1 -- 172.21.15.107:0/3626806318 >> 172.21.15.107:0/3626806318 conn(0x7f92c00f2510 msgr2=0x7f92c00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.547+0000 7f92c488b700 1 -- 172.21.15.107:0/3626806318 shutdown_connections 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.547+0000 7f92c488b700 1 -- 172.21.15.107:0/3626806318 wait complete. 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 librados: starting msgr at 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 librados: starting objecter 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 Processor -- start 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 -- start start 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 librados: setting wanted keys 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 librados: calling monclient init 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92c01173e0 con 0x7f92c0104350 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92c01185c0 con 0x7f92c00f89c0 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92c488b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92c01197c0 con 0x7f92c00f9570 2024-02-10T08:43:52.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92bdd9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f92c00f9570 0x7f92c00ffc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37544/0 (socket says 172.21.15.107:37544) 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.548+0000 7f92bdd9b700 1 -- 172.21.15.107:0/396331367 learned_addr learned my addr 172.21.15.107:0/396331367 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter ms_handle_connect 0x7f92c00f9570 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter ms_handle_connect 0x7f92c0104350 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter ms_handle_connect 0x7f92c00f89c0 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055371080 0 0) 0x7f92c01197c0 con 0x7f92c00f9570 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a80033f0 con 0x7f92c00f9570 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165420137 0 0) 0x7f92c01173e0 con 0x7f92c0104350 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c01197c0 con 0x7f92c0104350 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849419876 0 0) 0x7f92c01185c0 con 0x7f92c00f89c0 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c01173e0 con 0x7f92c00f89c0 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24012648 0 0) 0x7f92a80033f0 con 0x7f92c00f9570 2024-02-10T08:43:52.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c01185c0 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92b4002910 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607310979 0 0) 0x7f92c01197c0 con 0x7f92c0104350 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a80033f0 con 0x7f92c0104350 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92b8001bb0 con 0x7f92c0104350 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576523523 0 0) 0x7f92c01173e0 con 0x7f92c00f89c0 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c01197c0 con 0x7f92c00f89c0 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92b0002050 con 0x7f92c00f89c0 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2531652806 0 0) 0x7f92c01185c0 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 >> v1:172.21.15.195:6789/0 conn(0x7f92c00f89c0 legacy=0x7f92c00ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 >> v1:172.21.15.107:6789/0 conn(0x7f92c0104350 legacy=0x7f92c0115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c011a9c0 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c01175d0 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92c488b700 10 client.53810.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c0117a40 con 0x7f92c00f9570 2024-02-10T08:43:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92c488b700 1 librados: init done 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f92b40017d0 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.549+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92b4005ab0 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.550+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f92a0004f00 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f92b4001330 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(486..486 src has 330..486) v4 ==== 7721+0+0 (unknown 2468039707 0 0) 0x7f92b4078b40 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 osd_map(486..486 src has 330..486) v4 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 3 client.53810.objecter handle_osd_map got epochs [486,486] > 0 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 3 client.53810.objecter handle_osd_map decoding full epoch 486 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 20 client.53810.objecter dump_active .. 0 homeless 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v486) v1 ==== 54+0+16 (unknown 49545604 0 2420000192) 0x7f92b4078f30 con 0x7f92c00f9570 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a8063d60 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 10 client.53810.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 1 2024-02-10T08:43:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 1 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 0 1 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 4 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 10 client.53810.objecter _send_command 1 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.551+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a0004950 con 0x7f92a0004530 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a0004530 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a0004950 con 0x7f92a0004530 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 10 client.53810.objecter _finish_command 1 = system:0 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 4 2024-02-10T08:43:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.552+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 0 1 2024-02-10T08:43:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 10 client.53810.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 2 2024-02-10T08:43:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 2 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a00051a0 osd=0 3 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 0 2 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 4 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 10 client.53810.objecter _send_command 2 2024-02-10T08:43:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.567+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a0004d30 con 0x7f92a0004530 2024-02-10T08:43:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.568+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a0004d30 con 0x7f92a0004530 2024-02-10T08:43:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.568+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.568+0000 7f92af7fe700 10 client.53810.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.568+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a00051a0 osd=0 4 2024-02-10T08:43:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.568+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 0 2 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 10 client.53810.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 3 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 3 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 1 3 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 4 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 10 client.53810.objecter _send_command 3 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a000c000 con 0x7f92a0004150 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a0004150 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a000c000 con 0x7f92a0004150 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 10 client.53810.objecter _finish_command 3 = system:0 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 4 2024-02-10T08:43:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.569+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 1 3 2024-02-10T08:43:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:52.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:51.338325+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:43:52.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:51.341230+0000 osd.6 (osd.6) 215 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:51.472542+0000 mgr.y (mgr.14553) 442 : cluster [DBG] pgmap v730: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:51.553879+0000 mgr.y (mgr.14553) 443 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:52.079150+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:51.338325+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:51.341230+0000 osd.6 (osd.6) 215 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:51.472542+0000 mgr.y (mgr.14553) 442 : cluster [DBG] pgmap v730: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:51.553879+0000 mgr.y (mgr.14553) 443 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:52.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:52.079150+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-10T08:43:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:52.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 10 client.53810.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 4 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 4 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0003fe0 osd=1 3 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 1 4 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 4 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 10 client.53810.objecter _send_command 4 2024-02-10T08:43:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.585+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a00018b0 con 0x7f92a0004150 2024-02-10T08:43:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a00018b0 con 0x7f92a0004150 2024-02-10T08:43:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 10 client.53810.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0003fe0 osd=1 4 2024-02-10T08:43:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 1 4 2024-02-10T08:43:52.588 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 10 client.53810.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 5 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 5 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 2 5 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 4 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 10 client.53810.objecter _send_command 5 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a000ccd0 con 0x7f92a000c170 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.587+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a000c170 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.588+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a000ccd0 con 0x7f92a000c170 2024-02-10T08:43:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.588+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.588+0000 7f92af7fe700 10 client.53810.objecter _finish_command 5 = system:0 2024-02-10T08:43:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.588+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 4 2024-02-10T08:43:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.588+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 2 5 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 10 client.53810.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 6 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 6 2024-02-10T08:43:52.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0003630 osd=2 3 2024-02-10T08:43:52.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 2 6 2024-02-10T08:43:52.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 4 2024-02-10T08:43:52.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 10 client.53810.objecter _send_command 6 2024-02-10T08:43:52.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.602+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a00030a0 con 0x7f92a000c170 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.603+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a00030a0 con 0x7f92a000c170 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.603+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.603+0000 7f92af7fe700 10 client.53810.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.603+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0003630 osd=2 4 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.603+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 2 6 2024-02-10T08:43:52.604 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 10 client.53810.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 7 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 7 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 3 7 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 4 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 10 client.53810.objecter _send_command 7 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a0019ee0 con 0x7f92a0016290 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a0016290 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a0019ee0 con 0x7f92a0016290 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 10 client.53810.objecter _finish_command 7 = system:0 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 4 2024-02-10T08:43:52.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.604+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 3 7 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 10 client.53810.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 8 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 8 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a00034b0 osd=3 3 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 3 8 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 4 2024-02-10T08:43:52.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 10 client.53810.objecter _send_command 8 2024-02-10T08:43:52.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.619+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a001b200 con 0x7f92a0016290 2024-02-10T08:43:52.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a001b200 con 0x7f92a0016290 2024-02-10T08:43:52.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 10 client.53810.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a00034b0 osd=3 4 2024-02-10T08:43:52.621 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 3 8 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 10 client.53810.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 9 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 20 client.53810.objecter _get_session osd=-1 returning homeless 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 10 client.53810.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=487}) v3 -- 0x7f92a001b4d0 con 0x7f92c00f9570 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92a000e640 con 0x7f92c00f9570 2024-02-10T08:43:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=486) v2 ==== 24+0+0 (unknown 2643743101 0 0) 0x7f92b404bc50 con 0x7f92c00f9570 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92bcd99700 10 client.53810.objecter _check_command_map_dne tid 9 current 486 map_dne_bound 486 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92bcd99700 10 client.53810.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.620+0000 7f92bcd99700 15 client.53810.objecter _session_command_op_remove -1 9 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 10 client.53810.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 10 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 10 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 5 10 2024-02-10T08:43:52.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 4 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 10 client.53810.objecter _send_command 10 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a000db50 con 0x7f92a001a6d0 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.621+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a001a6d0 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.622+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a000db50 con 0x7f92a001a6d0 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.622+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.622+0000 7f92af7fe700 10 client.53810.objecter _finish_command 10 = system:0 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.622+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 4 2024-02-10T08:43:52.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.622+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 5 10 2024-02-10T08:43:52.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 10 client.53810.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 11 2024-02-10T08:43:52.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 11 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a001a560 osd=5 3 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 5 11 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 4 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 10 client.53810.objecter _send_command 11 2024-02-10T08:43:52.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.636+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a000dd70 con 0x7f92a001a6d0 2024-02-10T08:43:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a000dd70 con 0x7f92a001a6d0 2024-02-10T08:43:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 10 client.53810.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a001a560 osd=5 4 2024-02-10T08:43:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 5 11 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 10 client.53810.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 12 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 12 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 6 12 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 4 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 10 client.53810.objecter _send_command 12 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a0023490 con 0x7f92a001f8a0 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.638+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a001f8a0 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.639+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a0023490 con 0x7f92a001f8a0 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.639+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.639+0000 7f92af7fe700 10 client.53810.objecter _finish_command 12 = system:0 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.639+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 4 2024-02-10T08:43:52.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.639+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 6 12 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 10 client.53810.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 13 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 13 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a000dd70 osd=6 3 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 6 13 2024-02-10T08:43:52.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 4 2024-02-10T08:43:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 10 client.53810.objecter _send_command 13 2024-02-10T08:43:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.653+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a0024c40 con 0x7f92a001f8a0 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.654+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a0024c40 con 0x7f92a001f8a0 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.654+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.654+0000 7f92af7fe700 10 client.53810.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.654+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a000dd70 osd=6 4 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.654+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 6 13 2024-02-10T08:43:52.655 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 10 client.53810.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 14 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 14 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 7 14 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 4 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 10 client.53810.objecter _send_command 14 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a0025860 con 0x7f92a0025300 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.655+0000 7f92af7fe700 10 client.53810.objecter ms_handle_connect 0x7f92a0025300 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.656+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92a0025860 con 0x7f92a0025300 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.656+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.656+0000 7f92af7fe700 10 client.53810.objecter _finish_command 14 = system:0 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.656+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 4 2024-02-10T08:43:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.656+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 7 14 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 10 client.53810.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign -1 15 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 20 client.53810.objecter _get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 15 client.53810.objecter _session_command_op_remove -1 15 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 20 client.53810.objecter get_session s=0x7f92a0024c40 osd=7 3 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 15 client.53810.objecter _session_command_op_assign 7 15 2024-02-10T08:43:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 4 2024-02-10T08:43:52.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 10 client.53810.objecter _send_command 15 2024-02-10T08:43:52.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.670+0000 7f92c488b700 1 -- 172.21.15.107:0/396331367 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f92a0025860 con 0x7f92a0025300 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.671+0000 7f92af7fe700 1 -- 172.21.15.107:0/396331367 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f92a0025860 con 0x7f92a0025300 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.671+0000 7f92af7fe700 10 client.53810.objecter ms_dispatch 0x7f92c0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.671+0000 7f92af7fe700 10 client.53810.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.671+0000 7f92af7fe700 20 client.53810.objecter put_session s=0x7f92a0024c40 osd=7 4 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:52.671+0000 7f92af7fe700 15 client.53810.objecter _session_command_op_remove 7 15 2024-02-10T08:43:52.672 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.673 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:43:52.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:43:52.715 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:52.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:51.338325+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:43:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:51.341230+0000 osd.6 (osd.6) 215 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:43:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:51.472542+0000 mgr.y (mgr.14553) 442 : cluster [DBG] pgmap v730: 218 pgs: 39 active+undersized+degraded, 53 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 177/1107 objects degraded (15.989%) 2024-02-10T08:43:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:51.553879+0000 mgr.y (mgr.14553) 443 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:43:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:52.079150+0000 mon.a (mon.0) 2652 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-10T08:43:52.964 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:43:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:52.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:43:53.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:53.472 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:53.473 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:43:53.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:43:53.516 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:53.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:53.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:43:53.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:43:53.650 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:53.671 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:43:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:53.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:43:53.702 INFO:teuthology.orchestra.run.smithi195.stdout: marking collection for removal 2024-02-10T08:43:53.702 INFO:teuthology.orchestra.run.smithi195.stdout:setting '_remove' omap key 2024-02-10T08:43:53.702 INFO:teuthology.orchestra.run.smithi195.stdout:finish_remove_pgs 4.1_head removing 4.1 2024-02-10T08:43:53.702 INFO:teuthology.orchestra.run.smithi195.stdout:Remove successful 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:52.172096+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:52.173789+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:52.551919+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.107:0/396331367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.087017+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:53.112802+0000 mon.b (mon.1) 621 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.758 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:53.120055+0000 mon.a (mon.0) 2654 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:53.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:52.172096+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:43:53.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:52.173789+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:52.551919+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.107:0/396331367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.087017+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:53.112802+0000 mon.b (mon.1) 621 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:53.120055+0000 mon.a (mon.0) 2654 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:52.172096+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:52.173789+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:52.551919+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.107:0/396331367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.087017+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:53.112802+0000 mon.b (mon.1) 621 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:53.120055+0000 mon.a (mon.0) 2654 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:43:53.806 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:53.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:53.830 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:53.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:53.958 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:54.056 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:54.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:54.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:43:54.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:54.196 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:54.222 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:54.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:54.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:54.353 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.319008+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.319906+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.469547+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.c scrub starts 2024-02-10T08:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.472442+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.c scrub ok 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:53.473172+0000 mgr.y (mgr.14553) 444 : cluster [DBG] pgmap v733: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:54.078791+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:54.085326+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:54.085394+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:54.112842+0000 mon.b (mon.1) 622 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:54.130118+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:54.148873+0000 mon.a (mon.0) 2658 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.319008+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.319906+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.469547+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.c scrub starts 2024-02-10T08:43:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.472442+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.c scrub ok 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:53.473172+0000 mgr.y (mgr.14553) 444 : cluster [DBG] pgmap v733: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:54.078791+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:54.085326+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:54.085394+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:54.112842+0000 mon.b (mon.1) 622 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:54.130118+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:54.148873+0000 mon.a (mon.0) 2658 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.319008+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.319906+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.469547+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.c scrub starts 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.472442+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.c scrub ok 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:53.473172+0000 mgr.y (mgr.14553) 444 : cluster [DBG] pgmap v733: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:54.078791+0000 mon.a (mon.0) 2655 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:54.085326+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:43:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:54.085394+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-10T08:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:54.112842+0000 mon.b (mon.1) 622 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:54.130118+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:54.148873+0000 mon.a (mon.0) 2658 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:43:54.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.4 --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1.4 2024-02-10T08:43:54.875 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:43:55.090 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/osd.4/config 2024-02-10T08:43:55.203 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:55.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:55.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:55.340 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:55.472 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:55.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:55.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:43:55.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:55.627 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:55.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.730+0000 7f4a7998c700 1 -- 172.21.15.107:0/1862776078 >> v1:172.21.15.107:6790/0 conn(0x7f4a74104360 legacy=0x7f4a74106800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.730+0000 7f4a7998c700 1 -- 172.21.15.107:0/1862776078 shutdown_connections 2024-02-10T08:43:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.730+0000 7f4a7998c700 1 -- 172.21.15.107:0/1862776078 >> 172.21.15.107:0/1862776078 conn(0x7f4a740f2510 msgr2=0x7f4a740f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.730+0000 7f4a7998c700 1 -- 172.21.15.107:0/1862776078 shutdown_connections 2024-02-10T08:43:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.730+0000 7f4a7998c700 1 -- 172.21.15.107:0/1862776078 wait complete. 2024-02-10T08:43:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 librados: starting msgr at 2024-02-10T08:43:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 librados: starting objecter 2024-02-10T08:43:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 Processor -- start 2024-02-10T08:43:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 -- start start 2024-02-10T08:43:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 librados: setting wanted keys 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 librados: calling monclient init 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.731+0000 7f4a7998c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a74119570 con 0x7f4a740f8a60 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a7998c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a7411a750 con 0x7f4a74104360 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a7998c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a7411b950 con 0x7f4a740f9550 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a737fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4a74104360 0x7f4a74117dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:35432/0 (socket says 172.21.15.107:35432) 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a72ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4a740f8a60 0x7f4a741023d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51288/0 (socket says 172.21.15.107:51288) 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a737fe700 1 -- 172.21.15.107:0/130477707 learned_addr learned my addr 172.21.15.107:0/130477707 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 10 client.?.objecter ms_handle_connect 0x7f4a740f8a60 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 10 client.?.objecter ms_handle_connect 0x7f4a74104360 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115118703 0 0) 0x7f4a74119570 con 0x7f4a740f8a60 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c0033f0 con 0x7f4a740f8a60 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989067714 0 0) 0x7f4a7411a750 con 0x7f4a74104360 2024-02-10T08:43:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74119570 con 0x7f4a74104360 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 980989157 0 0) 0x7f4a5c0033f0 con 0x7f4a740f8a60 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a7411a750 con 0x7f4a740f8a60 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a640018c0 con 0x7f4a740f8a60 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876415511 0 0) 0x7f4a74119570 con 0x7f4a74104360 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a5c0033f0 con 0x7f4a74104360 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a6c003170 con 0x7f4a74104360 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 725848893 0 0) 0x7f4a7411a750 con 0x7f4a740f8a60 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 >> v1:172.21.15.107:6790/0 conn(0x7f4a740f9550 legacy=0x7f4a74102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 >> v1:172.21.15.195:6789/0 conn(0x7f4a74104360 legacy=0x7f4a74117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7411cb50 con 0x7f4a740f8a60 2024-02-10T08:43:55.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.732+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7411a940 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.733+0000 7f4a7998c700 10 client.28230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.733+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7411adb0 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.733+0000 7f4a7998c700 1 librados: init done 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.733+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4a64002c80 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.733+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a64005a70 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.734+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a60004f20 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.734+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4a640189c0 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(489..489 src has 330..489) v4 ==== 8123+0+0 (unknown 2537138446 0 0) 0x7f4a64078cc0 con 0x7f4a740f8a60 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 10 client.28230.objecter ms_dispatch 0x7f4a74001550 osd_map(489..489 src has 330..489) v4 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 3 client.28230.objecter handle_osd_map got epochs [489,489] > 0 2024-02-10T08:43:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 3 client.28230.objecter handle_osd_map decoding full epoch 489 2024-02-10T08:43:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 20 client.28230.objecter dump_active .. 0 homeless 2024-02-10T08:43:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.735+0000 7f4a5bfff700 10 client.28230.objecter ms_handle_connect 0x7f4a5c063bd0 2024-02-10T08:43:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.739+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4a6404c0d0 con 0x7f4a740f8a60 2024-02-10T08:43:55.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:54.506613+0000 osd.2 (osd.2) 189 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:43:55.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:54.509642+0000 osd.2 (osd.2) 190 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:43:55.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:55.087355+0000 mon.a (mon.0) 2659 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:43:55.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:55.087434+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-10T08:43:55.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:55.088736+0000 mon.b (mon.1) 624 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:55.089016+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:55.181562+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:54.506613+0000 osd.2 (osd.2) 189 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:54.509642+0000 osd.2 (osd.2) 190 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:55.087355+0000 mon.a (mon.0) 2659 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:55.087434+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:55.088736+0000 mon.b (mon.1) 624 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:55.089016+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:55.181562+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:55.771 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:43:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:43:55] "GET /metrics HTTP/1.1" 200 188545 "" "Prometheus/2.33.4" 2024-02-10T08:43:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:54.506613+0000 osd.2 (osd.2) 189 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:54.509642+0000 osd.2 (osd.2) 190 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:55.087355+0000 mon.a (mon.0) 2659 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:55.087434+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:55.088736+0000 mon.b (mon.1) 624 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:55.089016+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:43:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:55.181562+0000 mon.a (mon.0) 2662 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:43:55.788 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:55.793 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:55.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:55.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:43:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:55.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:55.930+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f4a600051d0 con 0x7f4a740f8a60 2024-02-10T08:43:56.021 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:43:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:56.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:43:56.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.088+0000 7f4a5bfff700 1 -- 172.21.15.107:0/130477707 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v490) v1 ==== 101+0+0 (unknown 3348327898 0 0) 0x7f4a6404a4e0 con 0x7f4a740f8a60 2024-02-10T08:43:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.092+0000 7f4a7998c700 10 librados: watch_flush enter 2024-02-10T08:43:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.092+0000 7f4a7998c700 10 librados: watch_flush exit 2024-02-10T08:43:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.092+0000 7f4a7998c700 20 client.28230.objecter shutdown clearing up homeless session... 2024-02-10T08:43:56.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.092+0000 7f4a7998c700 10 client.28230.objecter successfully canceled tick 2024-02-10T08:43:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.104+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4a5c063bd0 legacy=0x7f4a5c066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:56.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.104+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 >> v1:172.21.15.107:6789/0 conn(0x7f4a740f8a60 legacy=0x7f4a741023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:56.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:43:56.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.126+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 shutdown_connections 2024-02-10T08:43:56.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.126+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 >> 172.21.15.107:0/130477707 conn(0x7f4a740f2510 msgr2=0x7f4a741077f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.147+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 shutdown_connections 2024-02-10T08:43:56.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.157+0000 7f4a7998c700 1 -- 172.21.15.107:0/130477707 wait complete. 2024-02-10T08:43:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:56.176+0000 7f4a7998c700 1 librados: shutdown 2024-02-10T08:43:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:43:56.207 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:56.616 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:55.173909+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:55.185033+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:55.473925+0000 mgr.y (mgr.14553) 445 : cluster [DBG] pgmap v736: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:55.931707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:56.084442+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:56.088860+0000 mon.a (mon.0) 2665 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:56.088949+0000 mon.a (mon.0) 2666 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:56.088990+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:56.091092+0000 mon.b (mon.1) 625 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:56.091543+0000 mon.a (mon.0) 2668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:55.173909+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:43:56.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:55.185033+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:55.473925+0000 mgr.y (mgr.14553) 445 : cluster [DBG] pgmap v736: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:55.931707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:56.084442+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:56.088860+0000 mon.a (mon.0) 2665 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:56.088949+0000 mon.a (mon.0) 2666 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:56.088990+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:56.091092+0000 mon.b (mon.1) 625 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:56.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:56.091543+0000 mon.a (mon.0) 2668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:56.891 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:55.173909+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:43:56.891 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:55.185033+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:55.473925+0000 mgr.y (mgr.14553) 445 : cluster [DBG] pgmap v736: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:55.931707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:56.084442+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:56.088860+0000 mon.a (mon.0) 2665 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:56.088949+0000 mon.a (mon.0) 2666 : audit [INF] from='client.? v1:172.21.15.107:0/130477707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:56.088990+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:56.091092+0000 mon.b (mon.1) 625 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:56.091543+0000 mon.a (mon.0) 2668 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:43:57.364 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:43:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:57.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:43:57.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:57.499 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:57.559 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:57.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:57.592 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:43:57.593 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:43:57.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:57.689 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:58.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:56.337435+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:43:58.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:56.338370+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:43:58.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:57.090096+0000 mon.a (mon.0) 2669 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:43:58.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:57.090174+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:57.100884+0000 mon.b (mon.1) 626 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:57.101329+0000 mon.a (mon.0) 2671 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:57.474528+0000 mgr.y (mgr.14553) 446 : cluster [DBG] pgmap v739: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:56.337435+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:56.338370+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:57.090096+0000 mon.a (mon.0) 2669 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:57.090174+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:57.100884+0000 mon.b (mon.1) 626 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:57.101329+0000 mon.a (mon.0) 2671 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:57.474528+0000 mgr.y (mgr.14553) 446 : cluster [DBG] pgmap v739: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:56.337435+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:56.338370+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:57.090096+0000 mon.a (mon.0) 2669 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:57.090174+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:57.100884+0000 mon.b (mon.1) 626 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:57.101329+0000 mon.a (mon.0) 2671 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:43:58.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:57.474528+0000 mgr.y (mgr.14553) 446 : cluster [DBG] pgmap v739: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:43:58.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:43:58.339 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:58.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:58.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 -- 172.21.15.107:0/2065481155 >> v1:172.21.15.107:6790/0 conn(0x7f71340fc910 legacy=0x7f71340fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 -- 172.21.15.107:0/2065481155 shutdown_connections 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 -- 172.21.15.107:0/2065481155 >> 172.21.15.107:0/2065481155 conn(0x7f71340f2510 msgr2=0x7f71340f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 -- 172.21.15.107:0/2065481155 shutdown_connections 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 -- 172.21.15.107:0/2065481155 wait complete. 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 librados: starting msgr at 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 librados: starting objecter 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.460+0000 7f713aab4700 1 Processor -- start 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 -- start start 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 librados: setting wanted keys 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 librados: calling monclient init 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71341173f0 con 0x7f7134103250 2024-02-10T08:43:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71341185f0 con 0x7f71340ff710 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f713aab4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71341197f0 con 0x7f71340fc910 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7138850700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f71340fc910 0x7f7134060e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37580/0 (socket says 172.21.15.107:37580) 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7138850700 1 -- 172.21.15.107:0/2743334197 learned_addr learned my addr 172.21.15.107:0/2743334197 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter ms_handle_connect 0x7f71340fc910 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter ms_handle_connect 0x7f7134103250 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter ms_handle_connect 0x7f71340ff710 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 10 client.?.objecter resend_mon_ops 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552989062 0 0) 0x7f71341173f0 con 0x7f7134103250 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7124003030 con 0x7f7134103250 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259250034 0 0) 0x7f71341197f0 con 0x7f71340fc910 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71341173f0 con 0x7f71340fc910 2024-02-10T08:43:58.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241381129 0 0) 0x7f71341185f0 con 0x7f71340ff710 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71341197f0 con 0x7f71340ff710 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1975205794 0 0) 0x7f71341173f0 con 0x7f71340fc910 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71341185f0 con 0x7f71340fc910 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71280015d0 con 0x7f71340fc910 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475559829 0 0) 0x7f7124003030 con 0x7f7134103250 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71341173f0 con 0x7f7134103250 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f712c001b00 con 0x7f7134103250 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975407911 0 0) 0x7f71341197f0 con 0x7f71340ff710 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7124003030 con 0x7f71340ff710 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7120001920 con 0x7f71340ff710 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2368522844 0 0) 0x7f71341185f0 con 0x7f71340fc910 2024-02-10T08:43:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 >> v1:172.21.15.195:6789/0 conn(0x7f71340ff710 legacy=0x7f7134061560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 >> v1:172.21.15.107:6789/0 conn(0x7f7134103250 legacy=0x7f7134115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.461+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713411a9f0 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71341175e0 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7128002400 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71280038f0 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f713aab4700 10 client.53822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7134117a50 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f713aab4700 1 librados: init done 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.462+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7134060410 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7128001840 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(492..492 src has 330..492) v4 ==== 8174+0+0 (unknown 1659814963 0 0) 0x7f71280791e0 con 0x7f71340fc910 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 osd_map(492..492 src has 330..492) v4 2024-02-10T08:43:58.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 3 client.53822.objecter handle_osd_map got epochs [492,492] > 0 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 3 client.53822.objecter handle_osd_map decoding full epoch 492 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 20 client.53822.objecter dump_active .. 0 homeless 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.463+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v492) v1 ==== 54+0+16 (unknown 1078122910 0 2420000192) 0x7f712807b5c0 con 0x7f71340fc910 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f7124063de0 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 10 client.53822.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 1 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 1 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 0 1 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 4 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 10 client.53822.objecter _send_command 1 2024-02-10T08:43:58.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7134108f50 con 0x7f7134118780 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.464+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f7134118780 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.465+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7134108f50 con 0x7f7134118780 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.465+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.465+0000 7f7131f76700 10 client.53822.objecter _finish_command 1 = system:0 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.465+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 4 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.465+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 0 1 2024-02-10T08:43:58.471 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 10 client.53822.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 2 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 2 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71340f8180 osd=0 3 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 0 2 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 4 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 10 client.53822.objecter _send_command 2 2024-02-10T08:43:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.479+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f71341194b0 con 0x7f7134118780 2024-02-10T08:43:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f71341194b0 con 0x7f7134118780 2024-02-10T08:43:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 10 client.53822.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71340f8180 osd=0 4 2024-02-10T08:43:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 0 2 2024-02-10T08:43:58.482 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:43:58.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 10 client.53822.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 1 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 4 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 10 client.53822.objecter _send_command 3 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71340fa740 con 0x7f7134059990 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.481+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f7134059990 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.482+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f71340fa740 con 0x7f7134059990 2024-02-10T08:43:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.482+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:43:58.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.482+0000 7f7131f76700 10 client.53822.objecter _finish_command 3 = system:0 2024-02-10T08:43:58.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.482+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 4 2024-02-10T08:43:58.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.482+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 1 3 2024-02-10T08:43:58.487 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:43:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:58.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 10 client.53822.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 4 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 4 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134119620 osd=1 3 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 1 4 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 4 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 10 client.53822.objecter _send_command 4 2024-02-10T08:43:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.496+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f713405a030 con 0x7f7134059990 2024-02-10T08:43:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f713405a030 con 0x7f7134059990 2024-02-10T08:43:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 10 client.53822.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134119620 osd=1 4 2024-02-10T08:43:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 1 4 2024-02-10T08:43:58.499 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:43:58.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 10 client.53822.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 5 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 5 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 2 5 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 4 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 10 client.53822.objecter _send_command 5 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7134028800 con 0x7f7134059570 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.498+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f7134059570 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.499+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7134028800 con 0x7f7134059570 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.499+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:43:58.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.499+0000 7f7131f76700 10 client.53822.objecter _finish_command 5 = system:0 2024-02-10T08:43:58.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.499+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 4 2024-02-10T08:43:58.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.499+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 2 5 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 10 client.53822.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 6 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 6 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134059400 osd=2 3 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 2 6 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 4 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 10 client.53822.objecter _send_command 6 2024-02-10T08:43:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.514+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f713405a1a0 con 0x7f7134059570 2024-02-10T08:43:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f713405a1a0 con 0x7f7134059570 2024-02-10T08:43:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f7131f76700 10 client.53822.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134059400 osd=2 4 2024-02-10T08:43:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 2 6 2024-02-10T08:43:58.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 10 client.53822.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 7 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 7 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 3 7 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 4 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 10 client.53822.objecter _send_command 7 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.515+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7134028110 con 0x7f71340f35d0 2024-02-10T08:43:58.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f71340f35d0 2024-02-10T08:43:58.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7134028110 con 0x7f71340f35d0 2024-02-10T08:43:58.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:43:58.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 10 client.53822.objecter _finish_command 7 = system:0 2024-02-10T08:43:58.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 4 2024-02-10T08:43:58.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.516+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 3 7 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 10 client.53822.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 8 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 8 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134027fa0 osd=3 3 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 3 8 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 4 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 10 client.53822.objecter _send_command 8 2024-02-10T08:43:58.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.530+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f71340f2b30 con 0x7f71340f35d0 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f71340f2b30 con 0x7f71340f35d0 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 10 client.53822.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134027fa0 osd=3 4 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 3 8 2024-02-10T08:43:58.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:43:58.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 10 client.53822.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 9 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter _get_session osd=-1 returning homeless 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 10 client.53822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=493}) v3 -- 0x7f71340f28f0 con 0x7f71340fc910 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7134125f10 con 0x7f71340fc910 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=492) v2 ==== 24+0+0 (unknown 4119139159 0 0) 0x7f712804c150 con 0x7f71340fc910 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7132ffd700 10 client.53822.objecter _check_command_map_dne tid 9 current 492 map_dne_bound 492 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7132ffd700 10 client.53822.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f7132ffd700 15 client.53822.objecter _session_command_op_remove -1 9 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 10 client.53822.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 10 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 10 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 5 10 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 4 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 10 client.53822.objecter _send_command 10 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.532+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71341264f0 con 0x7f713412b680 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.533+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f713412b680 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.534+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f71341264f0 con 0x7f713412b680 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.534+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.534+0000 7f7131f76700 10 client.53822.objecter _finish_command 10 = system:0 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.534+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 4 2024-02-10T08:43:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.534+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 5 10 2024-02-10T08:43:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 10 client.53822.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 11 2024-02-10T08:43:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 11 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f7134126380 osd=5 3 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 5 11 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 4 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 10 client.53822.objecter _send_command 11 2024-02-10T08:43:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.548+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f71341264f0 con 0x7f713412b680 2024-02-10T08:43:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f71341264f0 con 0x7f713412b680 2024-02-10T08:43:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f7131f76700 10 client.53822.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f7134126380 osd=5 4 2024-02-10T08:43:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 5 11 2024-02-10T08:43:58.551 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 10 client.53822.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 12 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 12 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 6 12 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 4 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 10 client.53822.objecter _send_command 12 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.550+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71341332d0 con 0x7f713412f7c0 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f713412f7c0 2024-02-10T08:43:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f71341332d0 con 0x7f713412f7c0 2024-02-10T08:43:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:43:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 10 client.53822.objecter _finish_command 12 = system:0 2024-02-10T08:43:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 4 2024-02-10T08:43:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.551+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 6 12 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 10 client.53822.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 13 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 13 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71341264f0 osd=6 3 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 6 13 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 4 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 10 client.53822.objecter _send_command 13 2024-02-10T08:43:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.565+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f71341332d0 con 0x7f713412f7c0 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.567+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f71341332d0 con 0x7f713412f7c0 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.567+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.567+0000 7f7131f76700 10 client.53822.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.567+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71341264f0 osd=6 4 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.567+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 6 13 2024-02-10T08:43:58.568 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 10 client.53822.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 14 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 14 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 7 14 2024-02-10T08:43:58.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 4 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 10 client.53822.objecter _send_command 14 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71341375e0 con 0x7f7134133950 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.568+0000 7f7131f76700 10 client.53822.objecter ms_handle_connect 0x7f7134133950 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.569+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f71341375e0 con 0x7f7134133950 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.569+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.569+0000 7f7131f76700 10 client.53822.objecter _finish_command 14 = system:0 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.569+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 4 2024-02-10T08:43:58.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.569+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 7 14 2024-02-10T08:43:58.580 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 10 client.53822.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign -1 15 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 20 client.53822.objecter _get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 15 client.53822.objecter _session_command_op_remove -1 15 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 20 client.53822.objecter get_session s=0x7f71341332d0 osd=7 3 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 15 client.53822.objecter _session_command_op_assign 7 15 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 4 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 10 client.53822.objecter _send_command 15 2024-02-10T08:43:58.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.582+0000 7f713aab4700 1 -- 172.21.15.107:0/2743334197 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f71341375e0 con 0x7f7134133950 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.585+0000 7f7131f76700 1 -- 172.21.15.107:0/2743334197 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f71341375e0 con 0x7f7134133950 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.585+0000 7f7131f76700 10 client.53822.objecter ms_dispatch 0x7f7134001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.585+0000 7f7131f76700 10 client.53822.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.585+0000 7f7131f76700 20 client.53822.objecter put_session s=0x7f71341332d0 osd=7 4 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:43:58.585+0000 7f7131f76700 15 client.53822.objecter _session_command_op_remove 7 15 2024-02-10T08:43:58.585 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:43:58.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:43:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:58.870 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:43:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:58.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:43:58.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:59.001 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:59.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:58.096589+0000 mon.a (mon.0) 2672 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.096660+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:58.238072+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:58.239879+0000 mon.a (mon.0) 2674 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:58.464159+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2743334197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.504636+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:43:59.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.505428+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:43:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:43:59.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:43:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:58.096589+0000 mon.a (mon.0) 2672 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:43:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.096660+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-10T08:43:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:58.238072+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:58.239879+0000 mon.a (mon.0) 2674 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:58.464159+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2743334197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.504636+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:43:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:43:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.505428+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:43:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:43:59.449 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:59.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:58.096589+0000 mon.a (mon.0) 2672 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.096660+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:58.238072+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:58.239879+0000 mon.a (mon.0) 2674 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:58.464159+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.107:0/2743334197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.504636+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:43:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:43:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.505428+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:43:59.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:43:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:43:59.679 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:43:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:43:59.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:43:59.687 INFO:teuthology.orchestra.run.smithi195.stdout:get_pg_num_history pg_num_history pg_num_history(e479 pg_nums {1={19=1},2={55=32},3={55=3},4={57=32},5={135=22},6={61=32},16={102=32},60={116=32},193={316=32},195={330=8},196={337=8},197={337=32},198={342=32},199={347=8},200={352=32},201={354=8},202={361=8},203={368=8},204={370=32},205={375=8},206={382=8},207={391=32},208={394=32},209={397=32},210={400=32},211={403=32},212={407=32},213={429=32},214={437=32},215={441=32},216={444=32},217={459=32},218={471=32},219={476=32}} deleted_pools 332,195,336,193,339,196,344,198,349,199,351,197,356,201,363,202,369,200,370,203,377,205,384,206,390,204,393,207,396,208,399,209,402,210,406,211,428,212,436,213,440,214,443,215,458,216,470,217,475,218) 2024-02-10T08:43:59.687 INFO:teuthology.orchestra.run.smithi195.stdout:Importing pgid 4.1 2024-02-10T08:43:59.687 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:80542eaa:::obj_delete_at_hint.0000000068:head# 2024-02-10T08:43:59.687 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.688 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:80b6fe23:reshard::reshard.0000000011:head# 2024-02-10T08:43:59.688 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.688 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:80d23011:::obj_delete_at_hint.0000000085:head# 2024-02-10T08:43:59.688 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:8411e274:::obj_delete_at_hint.0000000073:head# 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:84d19476:::obj_delete_at_hint.0000000038:head# 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:856b9b26:reshard::reshard.0000000002:head# 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:Write #4:86cc8a85:::obj_delete_at_hint.0000000118:head# 2024-02-10T08:43:59.689 INFO:teuthology.orchestra.run.smithi195.stdout:snapset 0=[]:{} 2024-02-10T08:43:59.690 INFO:teuthology.orchestra.run.smithi195.stdout:write_pg epoch 479 info 4.1( v 473'28 (0'0,473'28] local-lis/les=57/58 n=7 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) 2024-02-10T08:43:59.690 INFO:teuthology.orchestra.run.smithi195.stdout:Import successful 2024-02-10T08:43:59.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:43:59.814 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:00.012 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:00.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:00.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.345986+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.346898+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.731874+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:58.735342+0000 osd.1 (osd.1) 241 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:59.096884+0000 mon.a (mon.0) 2675 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T08:44:00.153 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:00.175 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:00.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:00.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.345986+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:44:00.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.346898+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:44:00.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.731874+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:44:00.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:58.735342+0000 osd.1 (osd.1) 241 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:59.096884+0000 mon.a (mon.0) 2675 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.096929+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:59.237664+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:43:59.239020+0000 mon.a (mon.0) 2677 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.466958+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.469847+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.475094+0000 mgr.y (mgr.14553) 447 : cluster [DBG] pgmap v742: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.549676+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:44:00.184 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.552597+0000 osd.2 (osd.2) 194 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.345986+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.346898+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.731874+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:58.735342+0000 osd.1 (osd.1) 241 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:59.096884+0000 mon.a (mon.0) 2675 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.096929+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:59.237664+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:43:59.239020+0000 mon.a (mon.0) 2677 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.466958+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:44:00.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.469847+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:44:00.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.475094+0000 mgr.y (mgr.14553) 447 : cluster [DBG] pgmap v742: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:00.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.549676+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:44:00.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.552597+0000 osd.2 (osd.2) 194 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:44:00.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:00.343 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:44:00.343 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:44:00.343 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:44:00.343 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:44:00.343 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.096929+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:59.237664+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:43:59.239020+0000 mon.a (mon.0) 2677 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.466958+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.469847+0000 osd.5 (osd.5) 222 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:44:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.475094+0000 mgr.y (mgr.14553) 447 : cluster [DBG] pgmap v742: 218 pgs: 38 active+undersized+degraded, 59 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.549676+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:44:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.552597+0000 osd.2 (osd.2) 194 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:44:00.764 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:00.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:00.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:00.898 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:01.152 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:01.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:44:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:01.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:01.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:01.307 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:01.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:44:01.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.346+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/1612292956 >> v1:172.21.15.107:6790/0 conn(0x7f2b9c104350 legacy=0x7f2b9c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/1612292956 shutdown_connections 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/1612292956 >> 172.21.15.107:0/1612292956 conn(0x7f2b9c0f2510 msgr2=0x7f2b9c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/1612292956 shutdown_connections 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/1612292956 wait complete. 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 librados: starting msgr at 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 librados: starting objecter 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 Processor -- start 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 -- start start 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 librados: setting wanted keys 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.347+0000 7f2ba0f5d700 1 librados: calling monclient init 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c102b60 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c102d30 con 0x7f2b9c104350 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b9c102f20 con 0x7f2b9c0f9570 2024-02-10T08:44:01.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b9a59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2b9c0f89c0 0x7f2b9c1023c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60368/0 (socket says 172.21.15.107:60368) 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b9a59c700 1 -- 172.21.15.107:0/2532965536 learned_addr learned my addr 172.21.15.107:0/2532965536 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b99d9b700 1 --1- 172.21.15.107:0/2532965536 >> v1:172.21.15.107:6790/0 conn(0x7f2b9c0f9570 0x7f2b9c114da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51334/0 (socket says 172.21.15.107:51334) 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f2b9c0f9570 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f2b9c0f89c0 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f2b9c104350 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866095085 0 0) 0x7f2b9c102f20 con 0x7f2b9c0f9570 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b84003800 con 0x7f2b9c0f9570 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544075269 0 0) 0x7f2b9c102b60 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c102f20 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726012546 0 0) 0x7f2b9c102d30 con 0x7f2b9c104350 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c102b60 con 0x7f2b9c104350 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 655462780 0 0) 0x7f2b84003800 con 0x7f2b9c0f9570 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c102d30 con 0x7f2b9c0f9570 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2b900018c0 con 0x7f2b9c0f9570 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2813170069 0 0) 0x7f2b9c102f20 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b84003800 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955388992 0 0) 0x7f2b9c102b60 con 0x7f2b9c104350 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c102f20 con 0x7f2b9c104350 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2b8c0019f0 con 0x7f2b9c0f89c0 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2b94001930 con 0x7f2b9c104350 2024-02-10T08:44:01.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3909961147 0 0) 0x7f2b9c102d30 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 >> v1:172.21.15.195:6789/0 conn(0x7f2b9c104350 legacy=0x7f2b9c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 >> v1:172.21.15.107:6789/0 conn(0x7f2b9c0f89c0 legacy=0x7f2b9c1023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c11cc20 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c119c50 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 10 client.53828.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c11a0c0 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.348+0000 7f2ba0f5d700 1 librados: init done 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.349+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2b90002000 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.349+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2b90005a00 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.349+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c004f20 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2b90001b60 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(495..495 src has 330..495) v4 ==== 8174+0+0 (unknown 936465340 0 0) 0x7f2b90078c20 con 0x7f2b9c0f9570 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 10 client.53828.objecter ms_dispatch 0x7f2b9c001550 osd_map(495..495 src has 330..495) v4 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 3 client.53828.objecter handle_osd_map got epochs [495,495] > 0 2024-02-10T08:44:01.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 3 client.53828.objecter handle_osd_map decoding full epoch 495 2024-02-10T08:44:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 20 client.53828.objecter dump_active .. 0 homeless 2024-02-10T08:44:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.350+0000 7f2b8b7fe700 10 client.53828.objecter ms_handle_connect 0x7f2b84063de0 2024-02-10T08:44:01.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.354+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2b9004bfa0 con 0x7f2b9c0f9570 2024-02-10T08:44:01.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.371942+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:44:01.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.374853+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.378300+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c deep-scrub starts 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.379142+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c deep-scrub ok 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.757753+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:43:59.760718+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:00.140075+0000 mon.a (mon.0) 2678 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:00.140151+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:00.291001+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:00.292034+0000 mon.a (mon.0) 2680 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:01.061336+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.068437+0000 mon.a (mon.0) 2682 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:01.068507+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.371942+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:44:01.407 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.374853+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.378300+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c deep-scrub starts 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.379142+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c deep-scrub ok 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.757753+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:43:59.760718+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:00.140075+0000 mon.a (mon.0) 2678 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:00.140151+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:00.291001+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:00.292034+0000 mon.a (mon.0) 2680 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:01.061336+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.068437+0000 mon.a (mon.0) 2682 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:01.408 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:01.068507+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-10T08:44:01.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:01.535+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2b7c0051d0 con 0x7f2b9c0f9570 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.371942+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.374853+0000 osd.6 (osd.6) 221 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.378300+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c deep-scrub starts 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.379142+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c deep-scrub ok 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.757753+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:43:59.760718+0000 osd.1 (osd.1) 243 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:00.140075+0000 mon.a (mon.0) 2678 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:00.140151+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:00.291001+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:00.292034+0000 mon.a (mon.0) 2680 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:01.061336+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.068437+0000 mon.a (mon.0) 2682 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:01.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:01.068507+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-10T08:44:01.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/exp.4.1.4 2024-02-10T08:44:01.626 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:44:01.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:01.655 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:01.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:01.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:01.787 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:01.877 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:01.911 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:01.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:02.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:02.042 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:02.178 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:00.270031+0000 osd.3 (osd.3) 241 : cluster [DBG] 3.2 deep-scrub starts 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:00.274543+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 deep-scrub ok 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:00.748740+0000 osd.1 (osd.1) 244 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:00.751691+0000 osd.1 (osd.1) 245 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.203757+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.204065+0000 mon.a (mon.0) 2684 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.204565+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.204774+0000 mon.a (mon.0) 2685 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:01.476192+0000 mgr.y (mgr.14553) 448 : cluster [DBG] pgmap v745: 218 pgs: 39 active+undersized+degraded, 58 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s; 175/1101 objects degraded (15.895%) 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.536773+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.107:0/2532965536' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.537068+0000 mon.a (mon.0) 2686 : audit [INF] from='client.53828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.179 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:01.564570+0000 mgr.y (mgr.14553) 449 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:02.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.191+0000 7f2b8b7fe700 1 -- 172.21.15.107:0/2532965536 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v496) v1 ==== 87+0+0 (unknown 2256726210 0 0) 0x7f2b9004a3b0 con 0x7f2b9c0f9570 2024-02-10T08:44:02.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.208+0000 7f2ba0f5d700 10 librados: watch_flush enter 2024-02-10T08:44:02.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.209+0000 7f2ba0f5d700 10 librados: watch_flush exit 2024-02-10T08:44:02.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.209+0000 7f2ba0f5d700 20 client.53828.objecter shutdown clearing up homeless session... 2024-02-10T08:44:02.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.209+0000 7f2ba0f5d700 10 client.53828.objecter successfully canceled tick 2024-02-10T08:44:02.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.213+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2b84063de0 legacy=0x7f2b840662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:02.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.213+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 >> v1:172.21.15.107:6790/0 conn(0x7f2b9c0f9570 legacy=0x7f2b9c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:02.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.213+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 shutdown_connections 2024-02-10T08:44:02.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.213+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 >> 172.21.15.107:0/2532965536 conn(0x7f2b9c0f2510 msgr2=0x7f2b9c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:02.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.225+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 shutdown_connections 2024-02-10T08:44:02.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.238+0000 7f2ba0f5d700 1 -- 172.21.15.107:0/2532965536 wait complete. 2024-02-10T08:44:02.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:02.241+0000 7f2ba0f5d700 1 librados: shutdown 2024-02-10T08:44:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:44:02.390 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:02.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:00.270031+0000 osd.3 (osd.3) 241 : cluster [DBG] 3.2 deep-scrub starts 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:00.274543+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 deep-scrub ok 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:00.748740+0000 osd.1 (osd.1) 244 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:00.751691+0000 osd.1 (osd.1) 245 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.203757+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.204065+0000 mon.a (mon.0) 2684 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.204565+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.204774+0000 mon.a (mon.0) 2685 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:01.476192+0000 mgr.y (mgr.14553) 448 : cluster [DBG] pgmap v745: 218 pgs: 39 active+undersized+degraded, 58 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s; 175/1101 objects degraded (15.895%) 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.536773+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.107:0/2532965536' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.537068+0000 mon.a (mon.0) 2686 : audit [INF] from='client.53828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:01.564570+0000 mgr.y (mgr.14553) 449 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:00.270031+0000 osd.3 (osd.3) 241 : cluster [DBG] 3.2 deep-scrub starts 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:00.274543+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 deep-scrub ok 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:00.748740+0000 osd.1 (osd.1) 244 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:00.751691+0000 osd.1 (osd.1) 245 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.203757+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.204065+0000 mon.a (mon.0) 2684 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.204565+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.204774+0000 mon.a (mon.0) 2685 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-77"}]: dispatch 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:01.476192+0000 mgr.y (mgr.14553) 448 : cluster [DBG] pgmap v745: 218 pgs: 39 active+undersized+degraded, 58 active+undersized, 121 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s; 175/1101 objects degraded (15.895%) 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.536773+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.107:0/2532965536' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.537068+0000 mon.a (mon.0) 2686 : audit [INF] from='client.53828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:02.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:01.564570+0000 mgr.y (mgr.14553) 449 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:02.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:02.522 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:02.548 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:02.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:02.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:02.683 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:02.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:02.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:03.011 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:03.109 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:03.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:03.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:03.299 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:03.327 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:03.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35273.0:1 58.1s0 58:8fcb97c6:test-rados-api-smithi107-112553-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e116)", 2024-02-10T08:44:03.391 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 432.99738653700001, 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.58733886400000002, 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35273", 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2054498539", 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391665+0000", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391663+0000", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4849999999999997e-06 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391670+0000", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4370000000000007e-06 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391678+0000", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9359999999999998e-06 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391683+0000", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6999999999999997e-05 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391730+0000", 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7719999999999999e-06 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.391733+0000", 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.56671075800000004 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.958444+0000", 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014385299999999999 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.958587+0000", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043469100000000002 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.959022+0000", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043462299999999998 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.959457+0000", 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019472974000000001 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.978930+0000", 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4040000000000003e-05 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979004+0000", 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.58733886400000002 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.397 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35273.0:2 58.1s0 58:8fcb97c6:test-rados-api-smithi107-112553-13::foo:head [watch watch cookie 94609606614144] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 432.40532678699998, 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014181251000000001, 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35273", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2054498539", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:44:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983725+0000", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983723+0000", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2670000000000003e-06 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983729+0000", 2024-02-10T08:44:03.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7109999999999995e-06 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983737+0000", 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0190000000000003e-06 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.983743+0000", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039794899999999998 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984141+0000", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4467000000000003e-05 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984225+0000", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039063500000000002 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984616+0000", 2024-02-10T08:44:03.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028579099999999997 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.984902+0000", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012934038 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.997836+0000", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0287e-05 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.997906+0000", 2024-02-10T08:44:03.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014181251000000001 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:3 60.f 60:f44ff5b3:test-rados-api-smithi107-112490-8::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 430.36263045499999, 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026465336999999999, 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:44:03.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026421+0000", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026419+0000", 2024-02-10T08:44:03.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6020000000000006e-06 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026428+0000", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6636000000000001e-05 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026445+0000", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6056e-05 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026461+0000", 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-02-10T08:44:03.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026510+0000", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013219600000000001 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026642+0000", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014158500000000001 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.026784+0000", 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000910372 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.027694+0000", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034724220000000002 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.031166+0000", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021647165999999999 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052814+0000", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1364e-05 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052845+0000", 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1374000000000001e-05 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:53.052886+0000", 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026465336999999999 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35279.0:10 57.1a e121/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head v 121'6, mlcod=118'3)", 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:44:03.408 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 428.34574898099999, 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012434328, 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043302+0000", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3654000000000001e-05 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043326+0000", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4459999999999998e-06 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043332+0000", 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4690000000000002e-06 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043339+0000", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032279800000000001 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043661+0000", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455999999999999e-05 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.043682+0000", 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012017435 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055699+0000", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7508000000000001e-05 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055737+0000", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012434328 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:606 60.0 e148/135 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head v 148'20, mlcod=147'16)", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:44:03.412 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 401.45444198000001, 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013072240000000001, 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934609+0000", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9150000000000002e-06 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934618+0000", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1688999999999999e-05 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934640+0000", 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1930000000000004e-06 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.934647+0000", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082309480000000001 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942878+0000", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0771000000000002e-05 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.942899+0000", 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047591389999999999 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.947658+0000", 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3985000000000002e-05 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.947682+0000", 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013072240000000001 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52877.0:2 129.c 129:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e189)", 2024-02-10T08:44:03.416 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:02.963026+0000", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 360.42602509900001, 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018132216, 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52877", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2466810138", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963026+0000", 2024-02-10T08:44:03.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963026+0000", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963023+0000", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0328e-05 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963034+0000", 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.365e-05 2024-02-10T08:44:03.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963047+0000", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2669999999999995e-06 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963057+0000", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2174000000000005e-05 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963119+0000", 2024-02-10T08:44:03.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2959999999999999e-06 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.963127+0000", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012323445000000001 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.975451+0000", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147887 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.975598+0000", 2024-02-10T08:44:03.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019302300000000001 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.975792+0000", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001523539 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.977315+0000", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000570132 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.977885+0000", 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031740729999999999 2024-02-10T08:44:03.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.981059+0000", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4854e-05 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.981114+0000", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4434000000000002e-05 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:02.981159+0000", 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018132216 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.422 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.1.0:391 60.19 e190/135 60:9b647aab:test-rados-api-smithi107-112490-14::foo4:4 v 190'22, mlcod=150'16)", 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:03.973402+0000", 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 359.41564959999999, 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012960839, 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973402+0000", 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973402+0000", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973400+0000", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.609e-05 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973416+0000", 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1938e-05 2024-02-10T08:44:03.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973428+0000", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4229999999999999e-06 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.973435+0000", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028114870000000001 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.976247+0000", 2024-02-10T08:44:03.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0398e-05 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.976267+0000", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010059722 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986327+0000", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5960000000000001e-05 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986363+0000", 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012960839 2024-02-10T08:44:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.5.0:1426 60.1f e190/135 60:fe60b27e:test-rados-api-smithi107-112490-14::foo30:4 v 190'20, mlcod=150'16)", 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:03.986147+0000", 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 359.402904847, 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017599825999999999, 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986147+0000", 2024-02-10T08:44:03.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986147+0000", 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986146+0000", 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.015e-06 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986152+0000", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3890000000000005e-06 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986161+0000", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6139999999999997e-06 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986167+0000", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5699000000000006e-05 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986263+0000", 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9568999999999998e-05 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986282+0000", 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017438398000000001 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003721+0000", 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5627999999999999e-05 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003746+0000", 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017599825999999999 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.5.0:1427 60.1f e190/135 60:fb639e18:test-rados-api-smithi107-112490-14::foo41:4 v 190'22, mlcod=150'16)", 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:03.986183+0000", 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 359.40286855800002, 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020528434000000002, 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986183+0000", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986183+0000", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986182+0000", 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2599999999999999e-06 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986187+0000", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6859999999999999e-06 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986190+0000", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0000000000000001e-06 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:03.986193+0000", 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015399153 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.001592+0000", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9402000000000002e-05 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.001612+0000", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050782919999999999 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.006690+0000", 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1209999999999999e-05 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.006711+0000", 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020528434000000002 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.7.0:324 60.10 e190/135 60:0eb4e82d:test-rados-api-smithi107-112490-14::foo53:4 v 190'48, mlcod=186'42)", 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:04.003155+0000", 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 359.38589627599998, 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010172588, 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003155+0000", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003155+0000", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003155+0000", 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9670000000000003e-06 2024-02-10T08:44:03.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003159+0000", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7720000000000001e-06 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003163+0000", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8899999999999999e-06 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.003165+0000", 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063908489999999997 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.009556+0000", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1350000000000001e-05 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.009578+0000", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037150690000000001 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.013293+0000", 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5077000000000003e-05 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.013328+0000", 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010172588 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52844.0:26 124.7s0 124:e1b0e034:test-rados-api-smithi107-112525-23::foo:head [write 0~8192] snapc a=[a] ondisk+write+known_if_redirected e191)", 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:04.963249+0000", 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.42580198799999, 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018758302000000001, 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.439 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52844", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2993270293", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 26 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963249+0000", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963249+0000", 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:44:03.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963246+0000", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0239000000000001e-05 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963267+0000", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9000000000000001e-05 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963286+0000", 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9388000000000001e-05 2024-02-10T08:44:03.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.963305+0000", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011305806999999999 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.974611+0000", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134277 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.974745+0000", 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012868389999999999 2024-02-10T08:44:03.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.976032+0000", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021693089999999999 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.978201+0000", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037330599999999999 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.981934+0000", 2024-02-10T08:44:03.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3456999999999998e-05 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:04.982008+0000", 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018758302000000001 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.52844.0:28 124.7s0 124:e1b0e034:test-rados-api-smithi107-112525-23::foo:head [stat,delete] snapc b=[b,a] ondisk+write+known_if_redirected e192)", 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:05.999917+0000", 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 357.38913464299998, 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024249910999999999, 2024-02-10T08:44:03.444 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.52844", 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2993270293", 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 28 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999917+0000", 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999917+0000", 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999915+0000", 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6969999999999999e-06 2024-02-10T08:44:03.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999923+0000", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1304999999999999e-05 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999935+0000", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9079999999999995e-06 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999943+0000", 2024-02-10T08:44:03.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2479999999999999e-05 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:05.999995+0000", 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1769999999999998e-06 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.000003+0000", 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078160769999999994 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.007819+0000", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000133592 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.007953+0000", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066695399999999998 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.008620+0000", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040050909999999997 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.012625+0000", 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011468377 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.024093+0000", 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3415000000000001e-05 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.024167+0000", 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024249910999999999 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.2.0:342 60.6 e191/116 60:60ec3952:test-rados-api-smithi107-112490-14::foo62:4 v 191'38, mlcod=150'34)", 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:06.001874+0000", 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 357.38717734400001, 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010026675000000001, 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001874+0000", 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001874+0000", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001873+0000", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5449999999999992e-06 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001882+0000", 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6660000000000008e-06 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001890+0000", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5769999999999999e-06 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.001896+0000", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022167390000000001 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.004113+0000", 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1251999999999999e-05 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.004134+0000", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077455600000000003 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.011880+0000", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0750999999999998e-05 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:06.011901+0000", 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010026675000000001 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:1507 132.8 132:10000000::::head [pgnls start_epoch 199] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e199)", 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:12.946709+0000", 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 350.44234230199999, 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30258517400000001, 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1507 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946709+0000", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946709+0000", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946705+0000", 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4572999999999999e-05 2024-02-10T08:44:03.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946720+0000", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9310999999999999e-05 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946739+0000", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9080000000000007e-06 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946749+0000", 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5660999999999994e-05 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946814+0000", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4819999999999997e-06 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:12.946822+0000", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30105117599999998 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:13.247873+0000", 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6007000000000002e-05 2024-02-10T08:44:03.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:13.247919+0000", 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001375168 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:13.249294+0000", 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30258517400000001 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.460 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47731.0:1 148.6s0 148:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e217)", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:30.964686+0000", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 332.42436591199998, 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0064210689999999, 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47731", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/241311821", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964686+0000", 2024-02-10T08:44:03.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964686+0000", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964683+0000", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9809999999999992e-06 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964692+0000", 2024-02-10T08:44:03.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0634e-05 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964703+0000", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8329999999999994e-06 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964711+0000", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0258e-05 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964751+0000", 2024-02-10T08:44:03.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2720000000000002e-06 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.964754+0000", 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99753115699999995 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.962285+0000", 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136439 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.962422+0000", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045019999999999999 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.962872+0000", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069190639999999999 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.969791+0000", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012494019999999999 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.971040+0000", 2024-02-10T08:44:03.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6248999999999995e-05 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:31.971107+0000", 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0064210689999999 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47857.0:1 167.6s0 167:68f45781:::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:39:03.972536+0000", 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 299.41651500900002, 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.251592292, 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.466 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47857", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3596812208", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972536+0000", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972536+0000", 2024-02-10T08:44:03.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972535+0000", 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1890000000000007e-06 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972543+0000", 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2536e-05 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972555+0000", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0740000000000004e-06 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972562+0000", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1715000000000002e-05 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972614+0000", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.914e-06 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:44:03.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:03.972617+0000", 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2495008249999999 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.222118+0000", 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136577 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.222254+0000", 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00049508199999999999 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.222749+0000", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000438973 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.223188+0000", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00089406399999999997 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.224082+0000", 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6238e-05 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:39:05.224129+0000", 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.251592292 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.7.52:69 60.13 e493/116 60:ce87a4eb:test-rados-api-smithi107-112490-78::5a02e6184a18760c5f8050d0e5b120c3d7a79519:head v 493'60, mlcod=492'57)", 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.158610+0000", 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.2304411399999999, 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013764500000000001, 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158610+0000", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158610+0000", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158610+0000", 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1704e-05 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158622+0000", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0117e-05 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158632+0000", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7270000000000001e-06 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158635+0000", 2024-02-10T08:44:03.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025413200000000001 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158890+0000", 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6286e-05 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158906+0000", 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013439537999999999 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.172345+0000", 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9450000000000001e-05 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.172375+0000", 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013764500000000001 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.7.52:62 60.1f e493/135 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head v 493'43, mlcod=439'41)", 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.222444029, 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014157032999999999, 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166606+0000", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1691999999999999e-05 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166618+0000", 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5758000000000001e-05 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166633+0000", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1700000000000002e-06 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166640+0000", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9800000000000002e-05 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166719+0000", 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2801000000000001e-05 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166742+0000", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013993217000000001 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180735+0000", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8994999999999999e-05 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180764+0000", 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014157032999999999 2024-02-10T08:44:03.480 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(3.2 REQUEST e494)", 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.237735+0000", 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 3.1513162000000001, 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097252889999999998, 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237735+0000", 2024-02-10T08:44:03.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237735+0000", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237735+0000", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9419999999999997e-06 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237739+0000", 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7986999999999999e-05 2024-02-10T08:44:03.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237757+0000", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5520000000000005e-06 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.237765+0000", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063270700000000004 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.238398+0000", 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.165e-06 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.238405+0000", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090175949999999998 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.247423+0000", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6656000000000001e-05 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.247439+0000", 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1304000000000001e-05 2024-02-10T08:44:03.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.247461+0000", 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097252889999999998 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.25339.0:7077 60.6 e495/116 60:66fd1095:test-rados-api-smithi107-112490-78::bar-chunk:head v 495'77, mlcod=492'75)", 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.138673+0000", 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.2503784910000002, 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012154666999999999, 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:03.485 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138673+0000", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138673+0000", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138673+0000", 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7800000000000003e-06 2024-02-10T08:44:03.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138679+0000", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8819999999999999e-05 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138698+0000", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1590000000000001e-05 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138710+0000", 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1489999999999995e-05 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138781+0000", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7628000000000001e-05 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.138799+0000", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011978900000000001 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.150778+0000", 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9839000000000002e-05 2024-02-10T08:44:03.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.150828+0000", 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012154666999999999 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:44:03.489 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:44:03.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:02.172678+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:02.180297+0000 mon.a (mon.0) 2688 : audit [INF] from='client.53828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:02.180328+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:02.172678+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:02.180297+0000 mon.a (mon.0) 2688 : audit [INF] from='client.53828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:02.180328+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:02.172678+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:02.180297+0000 mon.a (mon.0) 2688 : audit [INF] from='client.53828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:03.491 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:02.180328+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-10T08:44:03.517 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:03.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:03.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:03.993 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:03.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:04.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:44:04.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:04.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.281+0000 7fce49221700 1 -- 172.21.15.107:0/2212377864 >> v1:172.21.15.107:6789/0 conn(0x7fce44069490 legacy=0x7fce44069870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.282+0000 7fce49221700 1 -- 172.21.15.107:0/2212377864 shutdown_connections 2024-02-10T08:44:04.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.282+0000 7fce49221700 1 -- 172.21.15.107:0/2212377864 >> 172.21.15.107:0/2212377864 conn(0x7fce44063bc0 msgr2=0x7fce440672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.285+0000 7fce49221700 1 -- 172.21.15.107:0/2212377864 shutdown_connections 2024-02-10T08:44:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.285+0000 7fce49221700 1 -- 172.21.15.107:0/2212377864 wait complete. 2024-02-10T08:44:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.285+0000 7fce49221700 1 librados: starting msgr at 2024-02-10T08:44:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.285+0000 7fce49221700 1 librados: starting objecter 2024-02-10T08:44:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.285+0000 7fce49221700 1 Processor -- start 2024-02-10T08:44:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 -- start start 2024-02-10T08:44:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 librados: setting wanted keys 2024-02-10T08:44:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 librados: calling monclient init 2024-02-10T08:44:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce441261f0 con 0x7fce44069490 2024-02-10T08:44:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce441273d0 con 0x7fce44069f80 2024-02-10T08:44:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.286+0000 7fce49221700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce441285d0 con 0x7fce441132f0 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:02.383213+0000 osd.6 (osd.6) 222 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:02.384201+0000 osd.6 (osd.6) 223 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:03.189972+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:03.206869+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:03.209001+0000 mon.a (mon.0) 2691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:03.476756+0000 mgr.y (mgr.14553) 450 : cluster [DBG] pgmap v748: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce4359e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fce441132f0 0x7fce441248b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51348/0 (socket says 172.21.15.107:51348) 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce4359e700 1 -- 172.21.15.107:0/978484061 learned_addr learned my addr 172.21.15.107:0/978484061 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter ms_handle_connect 0x7fce441132f0 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter ms_handle_connect 0x7fce44069f80 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632717578 0 0) 0x7fce441285d0 con 0x7fce441132f0 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce2c003030 con 0x7fce441132f0 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256424429 0 0) 0x7fce441273d0 con 0x7fce44069f80 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce441285d0 con 0x7fce44069f80 2024-02-10T08:44:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter ms_handle_connect 0x7fce44069490 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3691111409 0 0) 0x7fce2c003030 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce441273d0 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.287+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce3c001c40 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 373107757 0 0) 0x7fce441273d0 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 >> v1:172.21.15.195:6789/0 conn(0x7fce44069f80 legacy=0x7fce44101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 >> v1:172.21.15.107:6789/0 conn(0x7fce44069490 legacy=0x7fce44101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce441297d0 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce441275c0 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce49221700 10 client.53834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce44127a70 con 0x7fce441132f0 2024-02-10T08:44:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce49221700 1 librados: init done 2024-02-10T08:44:04.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fce3c002720 con 0x7fce441132f0 2024-02-10T08:44:04.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fce3c005e90 con 0x7fce441132f0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.288+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fce4405ffc0 con 0x7fce441132f0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.290+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fce3c002280 con 0x7fce441132f0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.290+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(498..498 src has 330..498) v4 ==== 8107+0+0 (unknown 1943867745 0 0) 0x7fce3c078ff0 con 0x7fce441132f0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.290+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 osd_map(498..498 src has 330..498) v4 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.290+0000 7fce2bfff700 3 client.53834.objecter handle_osd_map got epochs [498,498] > 0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.290+0000 7fce2bfff700 3 client.53834.objecter handle_osd_map decoding full epoch 498 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce2bfff700 20 client.53834.objecter dump_active .. 0 homeless 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v498) v1 ==== 54+0+16 (unknown 2274996656 0 2420000192) 0x7fce3c0793b0 con 0x7fce441132f0 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce2c063900 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 10 client.53834.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 1 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 1 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 0 1 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 4 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 10 client.53834.objecter _send_command 1 2024-02-10T08:44:04.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.291+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce440475e0 con 0x7fce44128f30 2024-02-10T08:44:04.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce44128f30 2024-02-10T08:44:04.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce440475e0 con 0x7fce44128f30 2024-02-10T08:44:04.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:04.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 10 client.53834.objecter _finish_command 1 = system:0 2024-02-10T08:44:04.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 4 2024-02-10T08:44:04.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.297+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 0 1 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 10 client.53834.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 2 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 2 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce4406a900 osd=0 3 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 0 2 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 4 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 10 client.53834.objecter _send_command 2 2024-02-10T08:44:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.323+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce44118e70 con 0x7fce44128f30 2024-02-10T08:44:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce44118e70 con 0x7fce44128f30 2024-02-10T08:44:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 10 client.53834.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce4406a900 osd=0 4 2024-02-10T08:44:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 0 2 2024-02-10T08:44:04.326 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 10 client.53834.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 3 2024-02-10T08:44:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 1 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 4 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 10 client.53834.objecter _send_command 3 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce44116e00 con 0x7fce44126fa0 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.325+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce44126fa0 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.326+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce44116e00 con 0x7fce44126fa0 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.326+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.326+0000 7fce2bfff700 10 client.53834.objecter _finish_command 3 = system:0 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.326+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 4 2024-02-10T08:44:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.326+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 1 3 2024-02-10T08:44:04.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:04.336 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 10 client.53834.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 4 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 4 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44129550 osd=1 3 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 1 4 2024-02-10T08:44:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 4 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 10 client.53834.objecter _send_command 4 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.340+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce4412da40 con 0x7fce44126fa0 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce4412da40 con 0x7fce44126fa0 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 10 client.53834.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44129550 osd=1 4 2024-02-10T08:44:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 1 4 2024-02-10T08:44:04.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 10 client.53834.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 5 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 5 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 2 5 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 4 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 10 client.53834.objecter _send_command 5 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce44059ea0 con 0x7fce44128b10 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.342+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce44128b10 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.343+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce44059ea0 con 0x7fce44128b10 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.343+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.343+0000 7fce2bfff700 10 client.53834.objecter _finish_command 5 = system:0 2024-02-10T08:44:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.343+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 4 2024-02-10T08:44:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.343+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 2 5 2024-02-10T08:44:04.355 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:04.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 10 client.53834.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 6 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 6 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce441289a0 osd=2 3 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 2 6 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 4 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 10 client.53834.objecter _send_command 6 2024-02-10T08:44:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.357+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce441324c0 con 0x7fce44128b10 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce441324c0 con 0x7fce44128b10 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 10 client.53834.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce441289a0 osd=2 4 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 2 6 2024-02-10T08:44:04.359 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 10 client.53834.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 7 2024-02-10T08:44:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 7 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 3 7 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 4 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 10 client.53834.objecter _send_command 7 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce4412e930 con 0x7fce44059570 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.359+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce44059570 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.360+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce4412e930 con 0x7fce44059570 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.360+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.360+0000 7fce2bfff700 10 client.53834.objecter _finish_command 7 = system:0 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.360+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 4 2024-02-10T08:44:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.360+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 3 7 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 10 client.53834.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 8 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 8 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44059400 osd=3 3 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 3 8 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 4 2024-02-10T08:44:04.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 10 client.53834.objecter _send_command 8 2024-02-10T08:44:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.374+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce44136f10 con 0x7fce44059570 2024-02-10T08:44:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce44136f10 con 0x7fce44059570 2024-02-10T08:44:04.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 10 client.53834.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44059400 osd=3 4 2024-02-10T08:44:04.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 3 8 2024-02-10T08:44:04.376 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:04.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 10 client.53834.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 9 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 20 client.53834.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 10 client.53834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=499}) v3 -- 0x7fce441398c0 con 0x7fce441132f0 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce44139b10 con 0x7fce441132f0 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=498) v2 ==== 24+0+0 (unknown 1303827753 0 0) 0x7fce3c04a7c0 con 0x7fce441132f0 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce4159a700 10 client.53834.objecter _check_command_map_dne tid 9 current 498 map_dne_bound 498 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce4159a700 10 client.53834.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.375+0000 7fce4159a700 15 client.53834.objecter _session_command_op_remove -1 9 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 10 client.53834.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 10 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 10 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 5 10 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 4 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 10 client.53834.objecter _send_command 10 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce44133430 con 0x7fce4412df60 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.376+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce4412df60 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.377+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce44133430 con 0x7fce4412df60 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.377+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.377+0000 7fce2bfff700 10 client.53834.objecter _finish_command 10 = system:0 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.377+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 4 2024-02-10T08:44:04.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.377+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 5 10 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 10 client.53834.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 11 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 11 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce4412ddf0 osd=5 3 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 5 11 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 4 2024-02-10T08:44:04.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 10 client.53834.objecter _send_command 11 2024-02-10T08:44:04.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.392+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce44133430 con 0x7fce4412df60 2024-02-10T08:44:04.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce44133430 con 0x7fce4412df60 2024-02-10T08:44:04.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce2bfff700 10 client.53834.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce4412ddf0 osd=5 4 2024-02-10T08:44:04.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 5 11 2024-02-10T08:44:04.394 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 10 client.53834.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 12 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 12 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 6 12 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 4 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 10 client.53834.objecter _send_command 12 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.393+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce44133670 con 0x7fce44138f50 2024-02-10T08:44:04.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.394+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce44138f50 2024-02-10T08:44:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.395+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce44133670 con 0x7fce44138f50 2024-02-10T08:44:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.395+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.395+0000 7fce2bfff700 10 client.53834.objecter _finish_command 12 = system:0 2024-02-10T08:44:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.395+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 4 2024-02-10T08:44:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.395+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 6 12 2024-02-10T08:44:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 10 client.53834.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 13 2024-02-10T08:44:04.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 13 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44133430 osd=6 3 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 6 13 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 4 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 10 client.53834.objecter _send_command 13 2024-02-10T08:44:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.409+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce44133670 con 0x7fce44138f50 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.411+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce44133670 con 0x7fce44138f50 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.411+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.411+0000 7fce2bfff700 10 client.53834.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.411+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44133430 osd=6 4 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.411+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 6 13 2024-02-10T08:44:04.412 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 10 client.53834.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 14 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 14 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 7 14 2024-02-10T08:44:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 4 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 10 client.53834.objecter _send_command 14 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce441444c0 con 0x7fce441409b0 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.412+0000 7fce2bfff700 10 client.53834.objecter ms_handle_connect 0x7fce441409b0 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.413+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fce441444c0 con 0x7fce441409b0 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.413+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.413+0000 7fce2bfff700 10 client.53834.objecter _finish_command 14 = system:0 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.413+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 4 2024-02-10T08:44:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.413+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 7 14 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 10 client.53834.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign -1 15 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 20 client.53834.objecter _get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 15 client.53834.objecter _session_command_op_remove -1 15 2024-02-10T08:44:04.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 20 client.53834.objecter get_session s=0x7fce44133670 osd=7 3 2024-02-10T08:44:04.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 15 client.53834.objecter _session_command_op_assign 7 15 2024-02-10T08:44:04.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 4 2024-02-10T08:44:04.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 10 client.53834.objecter _send_command 15 2024-02-10T08:44:04.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.427+0000 7fce49221700 1 -- 172.21.15.107:0/978484061 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fce441444c0 con 0x7fce441409b0 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.428+0000 7fce2bfff700 1 -- 172.21.15.107:0/978484061 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fce441444c0 con 0x7fce441409b0 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.428+0000 7fce2bfff700 10 client.53834.objecter ms_dispatch 0x7fce44001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.428+0000 7fce2bfff700 10 client.53834.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.428+0000 7fce2bfff700 20 client.53834.objecter put_session s=0x7fce44133670 osd=7 4 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:04.428+0000 7fce2bfff700 15 client.53834.objecter _session_command_op_remove 7 15 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:44:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:02.383213+0000 osd.6 (osd.6) 222 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:44:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:02.384201+0000 osd.6 (osd.6) 223 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:03.189972+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:03.206869+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:03.209001+0000 mon.a (mon.0) 2691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:03.476756+0000 mgr.y (mgr.14553) 450 : cluster [DBG] pgmap v748: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:02.383213+0000 osd.6 (osd.6) 222 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:02.384201+0000 osd.6 (osd.6) 223 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:03.189972+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:03.206869+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:03.209001+0000 mon.a (mon.0) 2691 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:03.476756+0000 mgr.y (mgr.14553) 450 : cluster [DBG] pgmap v748: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:04.603 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:04.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:04.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:05.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:44:05.184 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:05.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:05.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:05.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:05.325 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:05.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:04.195878+0000 mon.a (mon.0) 2692 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:04.195944+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:04.213084+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:04.224730+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:04.226137+0000 mon.a (mon.0) 2694 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:04.290233+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/978484061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:04.195878+0000 mon.a (mon.0) 2692 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:04.195944+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:04.213084+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:04.224730+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:04.226137+0000 mon.a (mon.0) 2694 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:04.290233+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/978484061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:05.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:04.195878+0000 mon.a (mon.0) 2692 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:04.195944+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:04.213084+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:04.224730+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:04.226137+0000 mon.a (mon.0) 2694 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:05.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:04.290233+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? v1:172.21.15.107:0/978484061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:05.682 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:05.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:05.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:05.814 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:05.894 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:05] "GET /metrics HTTP/1.1" 200 188545 "" "Prometheus/2.33.4" 2024-02-10T08:44:06.013 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:06.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:06.017 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:44:06.017 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:44:06.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:06.147 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:06.270 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:05.199687+0000 mon.a (mon.0) 2695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:05.199760+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:05.210022+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:05.210602+0000 mon.a (mon.0) 2697 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:05.477424+0000 mgr.y (mgr.14553) 451 : cluster [DBG] pgmap v751: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:06.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:06.063083+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:06.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:06.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:06.401 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:06.654 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:44:06.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:44:06.654 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:44:06.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:05.199687+0000 mon.a (mon.0) 2695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:05.199760+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:05.210022+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:05.210602+0000 mon.a (mon.0) 2697 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:05.477424+0000 mgr.y (mgr.14553) 451 : cluster [DBG] pgmap v751: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:06.063083+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:05.199687+0000 mon.a (mon.0) 2695 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:06.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:05.199760+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-10T08:44:06.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:05.210022+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:05.210602+0000 mon.a (mon.0) 2697 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:06.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:05.477424+0000 mgr.y (mgr.14553) 451 : cluster [DBG] pgmap v751: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 174/1098 objects degraded (15.847%) 2024-02-10T08:44:06.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:06.063083+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: Degraded data redundancy: 174/1098 objects degraded (15.847%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:06.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:44:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:06.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:06.981 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:44:07.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.181+0000 7fedfeb31700 1 -- 172.21.15.107:0/172611848 >> v1:172.21.15.107:6789/0 conn(0x7fedf8104330 legacy=0x7fedf81067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.181+0000 7fedfeb31700 1 -- 172.21.15.107:0/172611848 shutdown_connections 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.181+0000 7fedfeb31700 1 -- 172.21.15.107:0/172611848 >> 172.21.15.107:0/172611848 conn(0x7fedf80f3640 msgr2=0x7fedf80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.181+0000 7fedfeb31700 1 -- 172.21.15.107:0/172611848 shutdown_connections 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.181+0000 7fedfeb31700 1 -- 172.21.15.107:0/172611848 wait complete. 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 librados: starting msgr at 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 librados: starting objecter 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 Processor -- start 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 -- start start 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 librados: setting wanted keys 2024-02-10T08:44:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 librados: calling monclient init 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fedf81172a0 con 0x7fedf81007f0 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fedf8118480 con 0x7fedf8104330 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedfeb31700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fedf8119680 con 0x7fedf80fda50 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf7fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fedf81007f0 0x7fedf80fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60396/0 (socket says 172.21.15.107:60396) 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf7fff700 1 -- 172.21.15.107:0/1025627854 learned_addr learned my addr 172.21.15.107:0/1025627854 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter ms_handle_connect 0x7fedf80fda50 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter ms_handle_connect 0x7fedf81007f0 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter ms_handle_connect 0x7fedf8104330 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041581437 0 0) 0x7fedf8119680 con 0x7fedf80fda50 2024-02-10T08:44:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede80038e0 con 0x7fedf80fda50 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970489550 0 0) 0x7fedf81172a0 con 0x7fedf81007f0 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf8119680 con 0x7fedf81007f0 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362687599 0 0) 0x7fedf8118480 con 0x7fedf8104330 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.182+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf81172a0 con 0x7fedf8104330 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 49909502 0 0) 0x7fedf8119680 con 0x7fedf81007f0 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedf8118480 con 0x7fedf81007f0 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fede4001940 con 0x7fedf81007f0 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305350124 0 0) 0x7fede80038e0 con 0x7fedf80fda50 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedf8119680 con 0x7fedf80fda50 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fedec001620 con 0x7fedf80fda50 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012699919 0 0) 0x7fedf81172a0 con 0x7fedf8104330 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fede80038e0 con 0x7fedf8104330 2024-02-10T08:44:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fedf0002100 con 0x7fedf8104330 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168761865 0 0) 0x7fedf8118480 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 >> v1:172.21.15.107:6790/0 conn(0x7fedf80fda50 legacy=0x7fedf80f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 >> v1:172.21.15.195:6789/0 conn(0x7fedf8104330 legacy=0x7fedf81159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf811a880 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf8117490 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fede4001cf0 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fede4005c60 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedfeb31700 10 client.28254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf8117980 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.183+0000 7fedfeb31700 1 librados: init done 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.184+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf805f700 con 0x7fedf81007f0 2024-02-10T08:44:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.184+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fede4018b90 con 0x7fedf81007f0 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(500..500 src has 330..500) v4 ==== 8139+0+0 (unknown 2109859737 0 0) 0x7fede4078ea0 con 0x7fedf81007f0 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 10 client.28254.objecter ms_dispatch 0x7fedf8001550 osd_map(500..500 src has 330..500) v4 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 3 client.28254.objecter handle_osd_map got epochs [500,500] > 0 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 3 client.28254.objecter handle_osd_map decoding full epoch 500 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 20 client.28254.objecter dump_active .. 0 homeless 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.185+0000 7fedf5ffb700 10 client.28254.objecter ms_handle_connect 0x7fede8063d10 2024-02-10T08:44:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.189+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fede404c3f0 con 0x7fedf81007f0 2024-02-10T08:44:07.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:44:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:07.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:07.285 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:07.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:07.385 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:07.443+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fedf80475e0 con 0x7fedf81007f0 2024-02-10T08:44:07.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:06.208245+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:07.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:06.208300+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-10T08:44:07.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:06.209651+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.469 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:06.210574+0000 mon.a (mon.0) 2701 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:06.208245+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:06.208300+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:06.209651+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:06.210574+0000 mon.a (mon.0) 2701 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:06.208245+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:07.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:06.208300+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-10T08:44:07.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:06.209651+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:06.210574+0000 mon.a (mon.0) 2701 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:07.535 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:07.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:07.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:08.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.261+0000 7fedf5ffb700 1 -- 172.21.15.107:0/1025627854 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v502) v1 ==== 97+0+0 (unknown 3253850625 0 0) 0x7fede404a800 con 0x7fedf81007f0 2024-02-10T08:44:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.270+0000 7fedfeb31700 10 librados: watch_flush enter 2024-02-10T08:44:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.270+0000 7fedfeb31700 10 librados: watch_flush exit 2024-02-10T08:44:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.270+0000 7fedfeb31700 20 client.28254.objecter shutdown clearing up homeless session... 2024-02-10T08:44:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.270+0000 7fedfeb31700 10 client.28254.objecter successfully canceled tick 2024-02-10T08:44:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.273+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 >> v1:172.21.15.107:6800/2634842009 conn(0x7fede8063d10 legacy=0x7fede80661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.273+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 >> v1:172.21.15.107:6789/0 conn(0x7fedf81007f0 legacy=0x7fedf80fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.275+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 shutdown_connections 2024-02-10T08:44:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.275+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 >> 172.21.15.107:0/1025627854 conn(0x7fedf80f3640 msgr2=0x7fedf81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.299+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 shutdown_connections 2024-02-10T08:44:08.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.310+0000 7fedfeb31700 1 -- 172.21.15.107:0/1025627854 wait complete. 2024-02-10T08:44:08.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:08.322+0000 7fedfeb31700 1 librados: shutdown 2024-02-10T08:44:08.351 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:44:08.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:07.210347+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:08.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:07.210423+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-10T08:44:08.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:07.221037+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:07.290024+0000 mon.a (mon.0) 2704 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:07.444771+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:08.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:07.477955+0000 mgr.y (mgr.14553) 452 : cluster [DBG] pgmap v754: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:07.210347+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:07.210423+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:07.221037+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:07.290024+0000 mon.a (mon.0) 2704 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:07.444771+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:08.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:07.477955+0000 mgr.y (mgr.14553) 452 : cluster [DBG] pgmap v754: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:07.210347+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:07.210423+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:07.221037+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:07.290024+0000 mon.a (mon.0) 2704 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:07.444771+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:08.643 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:07.477955+0000 mgr.y (mgr.14553) 452 : cluster [DBG] pgmap v754: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:08.707 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:44:08.708 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:44:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:08.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:09.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:09.053 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:09.159 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:09.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:08.252487+0000 mon.a (mon.0) 2706 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:09.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:08.261411+0000 mon.a (mon.0) 2707 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:09.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:08.261478+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:09.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:08.261522+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-10T08:44:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:09.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:09.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:08.252487+0000 mon.a (mon.0) 2706 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:09.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:08.261411+0000 mon.a (mon.0) 2707 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:09.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:08.261478+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:09.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:08.261522+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-10T08:44:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:08.252487+0000 mon.a (mon.0) 2706 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:08.261411+0000 mon.a (mon.0) 2707 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:08.261478+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? v1:172.21.15.107:0/1025627854' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:08.261522+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-10T08:44:09.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:09.640 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:09.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:44:09.988 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:09.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:10.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:10.120 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:10.228 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:44:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:09.265862+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-10T08:44:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:09.478411+0000 mgr.y (mgr.14553) 453 : cluster [DBG] pgmap v757: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:09.596655+0000 mon.b (mon.1) 638 : audit [DBG] from='client.? v1:172.21.15.107:0/2261257447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:44:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:09.265862+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-10T08:44:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:09.478411+0000 mgr.y (mgr.14553) 453 : cluster [DBG] pgmap v757: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:09.596655+0000 mon.b (mon.1) 638 : audit [DBG] from='client.? v1:172.21.15.107:0/2261257447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:44:10.481 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:10.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:09.265862+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-10T08:44:10.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:09.478411+0000 mgr.y (mgr.14553) 453 : cluster [DBG] pgmap v757: 218 pgs: 38 active+undersized+degraded, 55 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 173/1095 objects degraded (15.799%) 2024-02-10T08:44:10.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:09.596655+0000 mon.b (mon.1) 638 : audit [DBG] from='client.? v1:172.21.15.107:0/2261257447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:44:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.405+0000 7f77bd7a2700 1 -- 172.21.15.107:0/404000476 >> v1:172.21.15.107:6789/0 conn(0x7f77b80ff590 legacy=0x7f77b8101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.405+0000 7f77bd7a2700 1 -- 172.21.15.107:0/404000476 shutdown_connections 2024-02-10T08:44:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.405+0000 7f77bd7a2700 1 -- 172.21.15.107:0/404000476 >> 172.21.15.107:0/404000476 conn(0x7f77b806cb60 msgr2=0x7f77b80f3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.405+0000 7f77bd7a2700 1 -- 172.21.15.107:0/404000476 shutdown_connections 2024-02-10T08:44:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.405+0000 7f77bd7a2700 1 -- 172.21.15.107:0/404000476 wait complete. 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 librados: starting msgr at 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 librados: starting objecter 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 Processor -- start 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 -- start start 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 librados: setting wanted keys 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 librados: calling monclient init 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77b80f9560 con 0x7f77b81030a0 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77b8117550 con 0x7f77b80ff590 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77bd7a2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f77b81186f0 con 0x7f77b80fc740 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f77b80fc740 0x7f77b80f8970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59832/0 (socket says 172.21.15.107:59832) 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b7fff700 1 -- 172.21.15.107:0/2261257447 learned_addr learned my addr 172.21.15.107:0/2261257447 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter ms_handle_connect 0x7f77b80fc740 2024-02-10T08:44:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter ms_handle_connect 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter ms_handle_connect 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.406+0000 7f77b57fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683608891 0 0) 0x7f77b8117550 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a00038e0 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291370580 0 0) 0x7f77b80f9560 con 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b8117550 con 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877941344 0 0) 0x7f77a00038e0 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b80f9560 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f77ac0014e0 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3279291345 0 0) 0x7f77b8117550 con 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77a00038e0 con 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f77b0001d70 con 0x7f77b81030a0 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655068013 0 0) 0x7f77b80f9560 con 0x7f77b80ff590 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 >> v1:172.21.15.107:6790/0 conn(0x7f77b80fc740 legacy=0x7f77b80f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 >> v1:172.21.15.107:6789/0 conn(0x7f77b81030a0 legacy=0x7f77b8114e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b81198f0 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b81165a0 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77bd7a2700 10 client.48565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b8116a90 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77bd7a2700 1 librados: init done 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f77ac001a80 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.407+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f77ac0036d0 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.408+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a4004f20 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.409+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f77ac001ee0 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(503..503 src has 330..503) v4 ==== 8174+0+0 (unknown 2515862054 0 0) 0x7f77ac0767f0 con 0x7f77b80ff590 2024-02-10T08:44:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 10 client.48565.objecter ms_dispatch 0x7f77b8001550 osd_map(503..503 src has 330..503) v4 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 503, 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:44:09.256146+0000", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:43:35.150516+0000", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:43:14.076553+0000", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:44:10.637 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 102, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 221, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:44:10.638 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.639 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.640 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "67", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.641 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.642 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:44:10.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.645 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.647 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:44:10.649 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:44:10.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.652 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.653 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.655 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.657 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.662 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.663 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.665 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.666 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.667 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 3 client.48565.objecter handle_osd_map got epochs [503,503] > 0 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 3 client.48565.objecter handle_osd_map decoding full epoch 503 2024-02-10T08:44:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 20 client.48565.objecter dump_active .. 0 homeless 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.410+0000 7f77b57fa700 10 client.48565.objecter ms_handle_connect 0x7f77a0067fa0 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.414+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f77ac049bd0 con 0x7f77b80ff590 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.595+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f77a40051d0 con 0x7f77b80ff590 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.597+0000 7f77b57fa700 1 -- 172.21.15.107:0/2261257447 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v503) v1 ==== 81+0+48563 (unknown 4108743201 0 2704415381) 0x7f77ac048040 con 0x7f77b80ff590 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.670 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.671 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:44:10.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.673 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.674 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 60, 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-6", 2024-02-10T08:44:10.675 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:49.742959+0000", 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.676 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "474", 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "388", 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "388", 2024-02-10T08:44:10.677 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "388", 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 36, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 474, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.678 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.679 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "fingerprint_algorithm": "sha1" 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 221, 2024-02-10T08:44:10.680 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-79", 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:44:02.462638+0000", 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:44:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "503", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:44:10.682 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 2, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 503, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:44:10.683 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:44:10.684 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "dedup_cdc_chunk_size": 1024, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "dedup_tier": 60, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "fingerprint_algorithm": "sha1" 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:44:10.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:44:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.689 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:44:10.690 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:44:10.691 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.692 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.9418182373046875, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.694 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.697 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.698 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.699 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.701 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 0, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:44:10.702 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 478, 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 480, 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 10 librados: watch_flush enter 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 10 librados: watch_flush exit 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 20 client.48565.objecter shutdown clearing up homeless session... 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 10 client.48565.objecter successfully canceled tick 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 >> v1:172.21.15.107:6800/2634842009 conn(0x7f77a0067fa0 legacy=0x7f77a006a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 >> v1:172.21.15.195:6789/0 conn(0x7f77b80ff590 legacy=0x7f77b81116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 shutdown_connections 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 >> 172.21.15.107:0/2261257447 conn(0x7f77b806cb60 msgr2=0x7f77b8106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 shutdown_connections 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 -- 172.21.15.107:0/2261257447 wait complete. 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:09.599+0000 7f77bd7a2700 1 librados: shutdown 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "exists" 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:44:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:44:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:44:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 497, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:44:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:44:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:07.077774+0000", 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:09.798885+0000", 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:42:48.250470+0000", 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:44:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:42:48.540179+0000", 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:23.143436+0000", 2024-02-10T08:44:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 479 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:42:49.125948+0000", 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:44:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:00.537522+0000", 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:44:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:14.588663+0000", 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "6.1", 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.0", 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1d", 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:44:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.f", 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 3 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:44:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:44:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:44:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:44:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:44:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:44:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:10.744 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:44:11.402 INFO:tasks.thrashosds.thrasher:j is {'epoch': 503, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:44:09.256146+0000', 'last_up_change': '2024-02-10T08:43:35.150516+0000', 'last_in_change': '2024-02-10T08:43:14.076553+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 102, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 221, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 60, 'pool_name': 'test-rados-api-smithi107-112490-6', 'create_time': '2024-02-10T08:36:49.742959+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '474', 'last_force_op_resend': '388', 'last_force_op_resend_prenautilus': '388', 'last_force_op_resend_preluminous': '388', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 36, 'snap_epoch': 474, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}, {'pool': 221, 'pool_name': 'test-rados-api-smithi107-112490-79', 'create_time': '2024-02-10T08:44:02.462638+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '503', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 503, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 1024, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 60, 'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 0.9418182373046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 478, 'down_at': 480, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:43:07.077774+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:09.798885+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:42:48.250470+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:42:48.540179+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:23.143436+0000', 'dead_epoch': 479}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:42:49.125948+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:00.537522+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:14.588663+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '4.1', 'osds': [4, 0, 7]}, {'pgid': '6.1', 'osds': [2, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.0', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '16.f', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:44:11.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:44:11.470 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:11.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:11.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:11.614 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:10.268742+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-10T08:44:11.614 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:10.448035+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.614 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:10.448385+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:10.448952+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:10.449315+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.065178+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:11.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.071057+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-10T08:44:11.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:10.268742+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-10T08:44:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:10.448035+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:10.448385+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:10.448952+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:10.449315+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.065178+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.071057+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:10.268742+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:10.448035+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:10.448385+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:10.448952+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:10.449315+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-79"}]: dispatch 2024-02-10T08:44:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.065178+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: Degraded data redundancy: 173/1095 objects degraded (15.799%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:11.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.071057+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-10T08:44:11.854 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:44:11.854 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:44:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:11.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:12.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:12.083 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:12.240 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:12.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:12.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:12.374 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.199870+0000 osd.3 (osd.3) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.199978+0000 osd.3 (osd.3) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.479383+0000 mgr.y (mgr.14553) 454 : cluster [DBG] pgmap v760: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:11.570241+0000 mgr.y (mgr.14553) 455 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.644324+0000 osd.6 (osd.6) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:11.644480+0000 osd.6 (osd.6) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:12.099188+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:12.105192+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.440 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:12.106630+0000 mon.a (mon.0) 2717 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.448 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:12.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:12.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:44:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.482+0000 7f18fbaa2700 1 -- 172.21.15.107:0/2074084330 >> v1:172.21.15.107:6790/0 conn(0x7f18f40f89c0 legacy=0x7f18f40f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:12.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.482+0000 7f18fbaa2700 1 -- 172.21.15.107:0/2074084330 shutdown_connections 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.482+0000 7f18fbaa2700 1 -- 172.21.15.107:0/2074084330 >> 172.21.15.107:0/2074084330 conn(0x7f18f40f2510 msgr2=0x7f18f40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.482+0000 7f18fbaa2700 1 -- 172.21.15.107:0/2074084330 shutdown_connections 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.482+0000 7f18fbaa2700 1 -- 172.21.15.107:0/2074084330 wait complete. 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 librados: starting msgr at 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 librados: starting objecter 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 Processor -- start 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 -- start start 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 librados: setting wanted keys 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 librados: calling monclient init 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18f4102cf0 con 0x7f18f4104350 2024-02-10T08:44:12.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18f4102ec0 con 0x7f18f40f9570 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18fbaa2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18f411bc10 con 0x7f18f40f89c0 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18f983e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f18f40f89c0 0x7f18f4102550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59852/0 (socket says 172.21.15.107:59852) 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18f983e700 1 -- 172.21.15.107:0/1971624023 learned_addr learned my addr 172.21.15.107:0/1971624023 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.483+0000 7f18eaf78700 10 client.?.objecter ms_handle_connect 0x7f18f40f89c0 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 10 client.?.objecter ms_handle_connect 0x7f18f40f9570 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 10 client.?.objecter ms_handle_connect 0x7f18f4104350 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127385196 0 0) 0x7f18f411bc10 con 0x7f18f40f89c0 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0003630 con 0x7f18f40f89c0 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945854167 0 0) 0x7f18f4102ec0 con 0x7f18f40f9570 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f411bc10 con 0x7f18f40f9570 2024-02-10T08:44:12.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091654440 0 0) 0x7f18f4102cf0 con 0x7f18f4104350 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f4102ec0 con 0x7f18f4104350 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85813885 0 0) 0x7f18e0003630 con 0x7f18f40f89c0 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18f4102cf0 con 0x7f18f40f89c0 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 530403147 0 0) 0x7f18f411bc10 con 0x7f18f40f9570 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e0003630 con 0x7f18f40f9570 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18f0002d60 con 0x7f18f40f89c0 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18e40018c0 con 0x7f18f40f9570 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2166886123 0 0) 0x7f18f4102ec0 con 0x7f18f4104350 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18f411bc10 con 0x7f18f4104350 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18ec001a30 con 0x7f18f4104350 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1853615735 0 0) 0x7f18f4102cf0 con 0x7f18f40f89c0 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 >> v1:172.21.15.195:6789/0 conn(0x7f18f40f9570 legacy=0x7f18f4114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 >> v1:172.21.15.107:6789/0 conn(0x7f18f4104350 legacy=0x7f18f41184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:12.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f411cdb0 con 0x7f18f40f89c0 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f411be00 con 0x7f18f40f89c0 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18fbaa2700 10 client.53846.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f411c270 con 0x7f18f40f89c0 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18fbaa2700 1 librados: init done 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f18f0001bb0 con 0x7f18f40f89c0 2024-02-10T08:44:12.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.484+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18f0003870 con 0x7f18f40f89c0 2024-02-10T08:44:12.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:44:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.485+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f18d8004f00 con 0x7f18f40f89c0 2024-02-10T08:44:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.486+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f18f0003a70 con 0x7f18f40f89c0 2024-02-10T08:44:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.486+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(506..506 src has 330..506) v4 ==== 8094+0+0 (unknown 3245321846 0 0) 0x7f18f0079080 con 0x7f18f40f89c0 2024-02-10T08:44:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.486+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 osd_map(506..506 src has 330..506) v4 2024-02-10T08:44:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.486+0000 7f18eaf78700 3 client.53846.objecter handle_osd_map got epochs [506,506] > 0 2024-02-10T08:44:12.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.486+0000 7f18eaf78700 3 client.53846.objecter handle_osd_map decoding full epoch 506 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.199870+0000 osd.3 (osd.3) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.199978+0000 osd.3 (osd.3) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.479383+0000 mgr.y (mgr.14553) 454 : cluster [DBG] pgmap v760: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:11.570241+0000 mgr.y (mgr.14553) 455 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.644324+0000 osd.6 (osd.6) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:11.644480+0000 osd.6 (osd.6) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:12.099188+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:12.105192+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:12.106630+0000 mon.a (mon.0) 2717 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.199870+0000 osd.3 (osd.3) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.199978+0000 osd.3 (osd.3) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.479383+0000 mgr.y (mgr.14553) 454 : cluster [DBG] pgmap v760: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:11.570241+0000 mgr.y (mgr.14553) 455 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.644324+0000 osd.6 (osd.6) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:11.644480+0000 osd.6 (osd.6) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:12.099188+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:12.105192+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:12.106630+0000 mon.a (mon.0) 2717 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.487+0000 7f18eaf78700 20 client.53846.objecter dump_active .. 0 homeless 2024-02-10T08:44:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.487+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v506) v1 ==== 54+0+16 (unknown 3505652984 0 2420000192) 0x7f18f007b480 con 0x7f18f40f89c0 2024-02-10T08:44:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.487+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18e0063d10 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 1 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 1 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 0 1 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 4 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 10 client.53846.objecter _send_command 1 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8004950 con 0x7f18d8004530 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.488+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d8004530 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.489+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d8004950 con 0x7f18d8004530 2024-02-10T08:44:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.489+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.489+0000 7f18eaf78700 10 client.53846.objecter _finish_command 1 = system:0 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.489+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 4 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.489+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 0 1 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.504+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 2 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 2 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d80051a0 osd=0 3 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 0 2 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 4 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 10 client.53846.objecter _send_command 2 2024-02-10T08:44:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.505+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d8004d30 con 0x7f18d8004530 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d8004d30 con 0x7f18d8004530 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18eaf78700 10 client.53846.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d80051a0 osd=0 4 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 0 2 2024-02-10T08:44:12.542 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 1 3 2024-02-10T08:44:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 4 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 10 client.53846.objecter _send_command 3 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.506+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d800c000 con 0x7f18d8004150 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d8004150 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d800c000 con 0x7f18d8004150 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 10 client.53846.objecter _finish_command 3 = system:0 2024-02-10T08:44:12.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 4 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.507+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 1 3 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 4 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 4 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003fe0 osd=1 3 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 1 4 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 4 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 10 client.53846.objecter _send_command 4 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.523+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d8001850 con 0x7f18d8004150 2024-02-10T08:44:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d8001850 con 0x7f18d8004150 2024-02-10T08:44:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18eaf78700 10 client.53846.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003fe0 osd=1 4 2024-02-10T08:44:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 1 4 2024-02-10T08:44:12.549 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 5 2024-02-10T08:44:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 5 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 2 5 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 4 2024-02-10T08:44:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 10 client.53846.objecter _send_command 5 2024-02-10T08:44:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.524+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8003e70 con 0x7f18d800d2f0 2024-02-10T08:44:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d800d2f0 2024-02-10T08:44:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d8003e70 con 0x7f18d800d2f0 2024-02-10T08:44:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 10 client.53846.objecter _finish_command 5 = system:0 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 4 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.525+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 2 5 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 6 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 6 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003630 osd=2 3 2024-02-10T08:44:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 2 6 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 4 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 10 client.53846.objecter _send_command 6 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.546+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d800e6c0 con 0x7f18d800d2f0 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.547+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d800e6c0 con 0x7f18d800d2f0 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.547+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.547+0000 7f18eaf78700 10 client.53846.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.547+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003630 osd=2 4 2024-02-10T08:44:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.547+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 2 6 2024-02-10T08:44:12.556 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 7 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 7 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 3 7 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 4 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 10 client.53846.objecter _send_command 7 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8015b00 con 0x7f18d8011e80 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.548+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d8011e80 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.549+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d8015b00 con 0x7f18d8011e80 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.549+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.549+0000 7f18eaf78700 10 client.53846.objecter _finish_command 7 = system:0 2024-02-10T08:44:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.549+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 4 2024-02-10T08:44:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.549+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 3 7 2024-02-10T08:44:12.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:12.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 8 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 8 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8003430 osd=3 3 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 3 8 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 4 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 10 client.53846.objecter _send_command 8 2024-02-10T08:44:12.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.565+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d800e100 con 0x7f18d8011e80 2024-02-10T08:44:12.569 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:12.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d800e100 con 0x7f18d8011e80 2024-02-10T08:44:12.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18eaf78700 10 client.53846.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8003430 osd=3 4 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 3 8 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 9 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 20 client.53846.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 10 client.53846.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=507}) v3 -- 0x7f18d800e300 con 0x7f18f40f89c0 2024-02-10T08:44:12.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.568+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18d800c940 con 0x7f18f40f89c0 2024-02-10T08:44:12.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.571+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=506) v2 ==== 24+0+0 (unknown 3293837838 0 0) 0x7f18f004c0a0 con 0x7f18f40f89c0 2024-02-10T08:44:12.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.571+0000 7f18ebfff700 10 client.53846.objecter _check_command_map_dne tid 9 current 506 map_dne_bound 506 2024-02-10T08:44:12.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.571+0000 7f18ebfff700 10 client.53846.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:12.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.571+0000 7f18ebfff700 15 client.53846.objecter _session_command_op_remove -1 9 2024-02-10T08:44:12.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.573+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.573+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 10 2024-02-10T08:44:12.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.573+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.573+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.573+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 10 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 5 10 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 4 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 10 client.53846.objecter _send_command 10 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d80168c0 con 0x7f18d800cee0 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.574+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d800cee0 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.575+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d80168c0 con 0x7f18d800cee0 2024-02-10T08:44:12.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.575+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:12.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.575+0000 7f18eaf78700 10 client.53846.objecter _finish_command 10 = system:0 2024-02-10T08:44:12.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.575+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 4 2024-02-10T08:44:12.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.575+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 5 10 2024-02-10T08:44:12.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 11 2024-02-10T08:44:12.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 11 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d800cd70 osd=5 3 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 5 11 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 4 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 10 client.53846.objecter _send_command 11 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.598+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d8016b00 con 0x7f18d800cee0 2024-02-10T08:44:12.599 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.599+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d8016b00 con 0x7f18d800cee0 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.599+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.599+0000 7f18eaf78700 10 client.53846.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.599+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d800cd70 osd=5 4 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.599+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 5 11 2024-02-10T08:44:12.600 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 12 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 12 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 6 12 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 4 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 10 client.53846.objecter _send_command 12 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d801f000 con 0x7f18d801b4c0 2024-02-10T08:44:12.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.600+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d801b4c0 2024-02-10T08:44:12.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.601+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d801f000 con 0x7f18d801b4c0 2024-02-10T08:44:12.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.601+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:12.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.601+0000 7f18eaf78700 10 client.53846.objecter _finish_command 12 = system:0 2024-02-10T08:44:12.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.601+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 4 2024-02-10T08:44:12.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.601+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 6 12 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 13 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 13 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d8016b00 osd=6 3 2024-02-10T08:44:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 6 13 2024-02-10T08:44:12.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 4 2024-02-10T08:44:12.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 10 client.53846.objecter _send_command 13 2024-02-10T08:44:12.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.622+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d801f000 con 0x7f18d801b4c0 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d801f000 con 0x7f18d801b4c0 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 10 client.53846.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d8016b00 osd=6 4 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 6 13 2024-02-10T08:44:12.624 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 14 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 14 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 7 14 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 4 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 10 client.53846.objecter _send_command 14 2024-02-10T08:44:12.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8020270 con 0x7f18d8020d20 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.624+0000 7f18eaf78700 10 client.53846.objecter ms_handle_connect 0x7f18d8020d20 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.625+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18d8020270 con 0x7f18d8020d20 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.625+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.625+0000 7f18eaf78700 10 client.53846.objecter _finish_command 14 = system:0 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.625+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 4 2024-02-10T08:44:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.625+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 7 14 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 10 client.53846.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign -1 15 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 20 client.53846.objecter _get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_remove -1 15 2024-02-10T08:44:12.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 20 client.53846.objecter get_session s=0x7f18d801f000 osd=7 3 2024-02-10T08:44:12.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 15 client.53846.objecter _session_command_op_assign 7 15 2024-02-10T08:44:12.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 4 2024-02-10T08:44:12.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 10 client.53846.objecter _send_command 15 2024-02-10T08:44:12.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.640+0000 7f18fbaa2700 1 -- 172.21.15.107:0/1971624023 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18d8020270 con 0x7f18d8020d20 2024-02-10T08:44:12.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.642+0000 7f18eaf78700 1 -- 172.21.15.107:0/1971624023 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18d8020270 con 0x7f18d8020d20 2024-02-10T08:44:12.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.642+0000 7f18eaf78700 10 client.53846.objecter ms_dispatch 0x7f18f4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:12.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.642+0000 7f18eaf78700 10 client.53846.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.642+0000 7f18eaf78700 20 client.53846.objecter put_session s=0x7f18d801f000 osd=7 4 2024-02-10T08:44:12.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:12.642+0000 7f18eaf78700 15 client.53846.objecter _session_command_op_remove 7 15 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:44:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:44:13.459 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:13.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:13.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:13.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:13.591 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:13.609 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:13.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:12.339811+0000 osd.2 (osd.2) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:12.339960+0000 osd.2 (osd.2) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:12.487203+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.107:0/1971624023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:13.104301+0000 mon.a (mon.0) 2718 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:13.104427+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:13.109423+0000 mon.b (mon.1) 642 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:13.138997+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:13.139735+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:12.339811+0000 osd.2 (osd.2) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:12.339960+0000 osd.2 (osd.2) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:12.487203+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.107:0/1971624023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:13.104301+0000 mon.a (mon.0) 2718 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:13.104427+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:13.109423+0000 mon.b (mon.1) 642 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:13.138997+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:13.139735+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:12.339811+0000 osd.2 (osd.2) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:12.339960+0000 osd.2 (osd.2) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:12.487203+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.107:0/1971624023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:13.104301+0000 mon.a (mon.0) 2718 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:13.104427+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-10T08:44:13.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:13.109423+0000 mon.b (mon.1) 642 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:13.739 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:13.138997+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.739 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:13.139735+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:13.740 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:13.758 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:13.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:13.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:13.888 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:13.906 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:13.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:14.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:14.393 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:14.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:14.425 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:14.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:14.524 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:14.647 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:13.479960+0000 mgr.y (mgr.14553) 456 : cluster [DBG] pgmap v763: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:14.104612+0000 mon.a (mon.0) 2721 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:14.104692+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:14.116949+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:14.132413+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:13.479960+0000 mgr.y (mgr.14553) 456 : cluster [DBG] pgmap v763: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:14.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:14.104612+0000 mon.a (mon.0) 2721 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:14.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:14.104692+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-10T08:44:14.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:14.116949+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:14.132413+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:14.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:14.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:14.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:13.479960+0000 mgr.y (mgr.14553) 456 : cluster [DBG] pgmap v763: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:14.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:14.104612+0000 mon.a (mon.0) 2721 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:14.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:14.104692+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-10T08:44:14.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:14.116949+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.779 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:14.132413+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:14.781 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:14.799 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:14.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:14.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:14.937 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:15.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:44:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:15.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:15.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:15.715 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:15.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:15] "GET /metrics HTTP/1.1" 200 188676 "" "Prometheus/2.33.4" 2024-02-10T08:44:15.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:15.006446+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:15.006591+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:15.105793+0000 mon.a (mon.0) 2724 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:15.105866+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:15.106901+0000 mon.b (mon.1) 645 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:15.107156+0000 mon.a (mon.0) 2726 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:15.480517+0000 mgr.y (mgr.14553) 457 : cluster [DBG] pgmap v766: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:16.067296+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 180/1119 objects degraded (16.086%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:15.006446+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:15.006591+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:15.105793+0000 mon.a (mon.0) 2724 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:15.105866+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:15.106901+0000 mon.b (mon.1) 645 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:15.107156+0000 mon.a (mon.0) 2726 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:15.480517+0000 mgr.y (mgr.14553) 457 : cluster [DBG] pgmap v766: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:16.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:16.067296+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 180/1119 objects degraded (16.086%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:15.006446+0000 osd.5 (osd.5) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:15.006591+0000 osd.5 (osd.5) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:15.105793+0000 mon.a (mon.0) 2724 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_tier","val": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:15.105866+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:15.106901+0000 mon.b (mon.1) 645 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:15.107156+0000 mon.a (mon.0) 2726 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:15.480517+0000 mgr.y (mgr.14553) 457 : cluster [DBG] pgmap v766: 218 pgs: 32 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:16.067296+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 180/1119 objects degraded (16.086%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:16.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.628+0000 7f938d6a2700 1 -- 172.21.15.107:0/2520892890 >> v1:172.21.15.107:6790/0 conn(0x7f93880f9570 legacy=0x7f9388102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.628+0000 7f938d6a2700 1 -- 172.21.15.107:0/2520892890 shutdown_connections 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.628+0000 7f938d6a2700 1 -- 172.21.15.107:0/2520892890 >> 172.21.15.107:0/2520892890 conn(0x7f93880f2510 msgr2=0x7f93880f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.628+0000 7f938d6a2700 1 -- 172.21.15.107:0/2520892890 shutdown_connections 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.628+0000 7f938d6a2700 1 -- 172.21.15.107:0/2520892890 wait complete. 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 librados: starting msgr at 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 librados: starting objecter 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 Processor -- start 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 -- start start 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 librados: setting wanted keys 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 librados: calling monclient init 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f93881194e0 con 0x7f93880f9570 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f938811a680 con 0x7f9388104350 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f938d6a2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f938811b820 con 0x7f93880f89c0 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f93867fc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f93880f9570 0x7f93881028f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39472/0 (socket says 172.21.15.107:39472) 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f9386ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f93880f89c0 0x7f93881021e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59870/0 (socket says 172.21.15.107:59870) 2024-02-10T08:44:16.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f93867fc700 1 -- 172.21.15.107:0/72023657 learned_addr learned my addr 172.21.15.107:0/72023657 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 10 client.?.objecter ms_handle_connect 0x7f93880f9570 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 10 client.?.objecter ms_handle_connect 0x7f93880f89c0 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927751941 0 0) 0x7f938811b820 con 0x7f93880f89c0 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9370003030 con 0x7f93880f89c0 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948234066 0 0) 0x7f93881194e0 con 0x7f93880f9570 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.629+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f938811b820 con 0x7f93880f9570 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 10 client.?.objecter ms_handle_connect 0x7f9388104350 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29761168 0 0) 0x7f938811a680 con 0x7f9388104350 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93881194e0 con 0x7f9388104350 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789695276 0 0) 0x7f9370003030 con 0x7f93880f89c0 2024-02-10T08:44:16.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f938811a680 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9378001a80 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1833771394 0 0) 0x7f938811b820 con 0x7f93880f9570 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9370003030 con 0x7f93880f9570 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f937c000fe0 con 0x7f93880f9570 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2363660200 0 0) 0x7f93881194e0 con 0x7f9388104350 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f938811b820 con 0x7f9388104350 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9380001930 con 0x7f9388104350 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2411539308 0 0) 0x7f938811a680 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 >> v1:172.21.15.195:6789/0 conn(0x7f9388104350 legacy=0x7f9388117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 >> v1:172.21.15.107:6789/0 conn(0x7f93880f9570 legacy=0x7f93881028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938811ca00 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938811ba10 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f938d6a2700 10 client.53852.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f938811be80 con 0x7f93880f89c0 2024-02-10T08:44:16.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f938d6a2700 1 librados: init done 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9378002440 con 0x7f93880f89c0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.630+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9378005a50 con 0x7f93880f89c0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.631+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9374004f20 con 0x7f93880f89c0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.631+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9378001fa0 con 0x7f93880f89c0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(510..510 src has 330..510) v4 ==== 8158+0+0 (unknown 2634264085 0 0) 0x7f9378078d20 con 0x7f93880f89c0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 10 client.53852.objecter ms_dispatch 0x7f9388001550 osd_map(510..510 src has 330..510) v4 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 3 client.53852.objecter handle_osd_map got epochs [510,510] > 0 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 3 client.53852.objecter handle_osd_map decoding full epoch 510 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 20 client.53852.objecter dump_active .. 0 homeless 2024-02-10T08:44:16.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.632+0000 7f936ffff700 10 client.53852.objecter ms_handle_connect 0x7f9370063bf0 2024-02-10T08:44:16.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.636+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f937804c110 con 0x7f93880f89c0 2024-02-10T08:44:16.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.798+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9374000b90 con 0x7f9370063bf0 2024-02-10T08:44:16.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.813+0000 7f936ffff700 1 -- 172.21.15.107:0/72023657 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+556105 (unknown 2965378022 0 179604204) 0x7f9374000b90 con 0x7f9370063bf0 2024-02-10T08:44:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 10 librados: watch_flush enter 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 10 librados: watch_flush exit 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 20 client.53852.objecter shutdown clearing up homeless session... 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 10 client.53852.objecter successfully canceled tick 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9370063bf0 legacy=0x7f93700660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:16.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 >> v1:172.21.15.107:6790/0 conn(0x7f93880f89c0 legacy=0x7f93881021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 shutdown_connections 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 >> 172.21.15.107:0/72023657 conn(0x7f93880f2510 msgr2=0x7f93880f4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 shutdown_connections 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 -- 172.21.15.107:0/72023657 wait complete. 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:16.817+0000 7f938d6a2700 1 librados: shutdown 2024-02-10T08:44:16.828 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:44:17.157 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:16.105481+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:16.105553+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:16.111364+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:16.115283+0000 mon.a (mon.0) 2730 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:16.186663+0000 osd.7 (osd.7) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:16.186875+0000 osd.7 (osd.7) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:17.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:16.799504+0000 mgr.y (mgr.14553) 458 : audit [DBG] from='client.53852 v1:172.21.15.107:0/72023657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:17.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:17.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:17.308 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:44:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:44:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:44:17.309 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:44:17.309 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:16.105481+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:16.105553+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-10T08:44:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:16.111364+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:16.115283+0000 mon.a (mon.0) 2730 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:16.186663+0000 osd.7 (osd.7) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:16.186875+0000 osd.7 (osd.7) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:16.799504+0000 mgr.y (mgr.14553) 458 : audit [DBG] from='client.53852 v1:172.21.15.107:0/72023657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:16.105481+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:16.105553+0000 mon.a (mon.0) 2729 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:16.111364+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:16.115283+0000 mon.a (mon.0) 2730 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:16.186663+0000 osd.7 (osd.7) 207 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:16.186875+0000 osd.7 (osd.7) 208 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:16.799504+0000 mgr.y (mgr.14553) 458 : audit [DBG] from='client.53852 v1:172.21.15.107:0/72023657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:17.657 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:17.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:17.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:17.786 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:17.810 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:17.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:17.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:17.943 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:18.065 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:18.065 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:18.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:18.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:18.195 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:18.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:18.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:18.345 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:18.364 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:18.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:17.116053+0000 mon.a (mon.0) 2731 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:17.116125+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2024-02-10T08:44:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:17.480995+0000 mgr.y (mgr.14553) 459 : cluster [DBG] pgmap v769: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:17.116053+0000 mon.a (mon.0) 2731 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:17.116125+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2024-02-10T08:44:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:17.480995+0000 mgr.y (mgr.14553) 459 : cluster [DBG] pgmap v769: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:18.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:18.495 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:18.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:18.536 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:17.116053+0000 mon.a (mon.0) 2731 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:17.116125+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2024-02-10T08:44:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:17.480995+0000 mgr.y (mgr.14553) 459 : cluster [DBG] pgmap v769: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:18.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:18.648 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:18.721 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:18.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:18.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:18.853 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:18.161888+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2024-02-10T08:44:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:18.161888+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2024-02-10T08:44:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:18.161888+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2024-02-10T08:44:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:19.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:44:19.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:19.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:19.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:19.863 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:19.881 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:19.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:19.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:20.012 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:20.133 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":767,"stamp":"2024-02-10T08:44:16.103435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4016816,"num_objects":373,"num_object_clones":9,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":1982,"num_read_kb":1594,"num_write":1463,"num_write_kb":4640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":727,"num_bytes_recovered":7992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2830,"ondisk_log_size":2830,"up":562,"acting":562,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":633,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2170472,"kb_used_data":36656,"kb_used_omap":0,"kb_used_meta":2133760,"kb_avail":654045592,"statfs":{"total":671965249536,"available":669742686208,"internally_reserved":0,"allocated":37535744,"data_stored":23586360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184970240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006223"},"pg_stats":[{"pgid":"222.1f","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102239+0000","last_change":"2024-02-10T08:44:13.128652+0000","last_active":"2024-02-10T08:44:14.102239+0000","last_peered":"2024-02-10T08:44:14.102239+0000","last_clean":"2024-02-10T08:44:14.102239+0000","last_became_active":"2024-02-10T08:44:13.128364+0000","last_became_peered":"2024-02-10T08:44:13.128364+0000","last_unstale":"2024-02-10T08:44:14.102239+0000","last_undegraded":"2024-02-10T08:44:14.102239+0000","last_fullsized":"2024-02-10T08:44:14.102239+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.1e","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104493+0000","last_change":"2024-02-10T08:44:13.125108+0000","last_active":"2024-02-10T08:44:14.104493+0000","last_peered":"2024-02-10T08:44:14.104493+0000","last_clean":"2024-02-10T08:44:14.104493+0000","last_became_active":"2024-02-10T08:44:13.124808+0000","last_became_peered":"2024-02-10T08:44:13.124808+0000","last_unstale":"2024-02-10T08:44:14.104493+0000","last_undegraded":"2024-02-10T08:44:14.104493+0000","last_fullsized":"2024-02-10T08:44:14.104493+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"222.1d","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.328361+0000","last_change":"2024-02-10T08:44:13.129947+0000","last_active":"2024-02-10T08:44:14.328361+0000","last_peered":"2024-02-10T08:44:14.328361+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.129947+0000","last_became_peered":"2024-02-10T08:44:13.129947+0000","last_unstale":"2024-02-10T08:44:14.328361+0000","last_undegraded":"2024-02-10T08:44:14.328361+0000","last_fullsized":"2024-02-10T08:44:13.106633+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.1c","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.405183+0000","last_change":"2024-02-10T08:44:13.153667+0000","last_active":"2024-02-10T08:44:14.405183+0000","last_peered":"2024-02-10T08:44:14.405183+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.153667+0000","last_became_peered":"2024-02-10T08:44:13.153667+0000","last_unstale":"2024-02-10T08:44:14.405183+0000","last_undegraded":"2024-02-10T08:44:14.405183+0000","last_fullsized":"2024-02-10T08:44:13.106552+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"222.1b","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.101159+0000","last_change":"2024-02-10T08:44:13.105595+0000","last_active":"2024-02-10T08:44:14.101159+0000","last_peered":"2024-02-10T08:44:14.101159+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.105595+0000","last_became_peered":"2024-02-10T08:44:13.105595+0000","last_unstale":"2024-02-10T08:44:14.101159+0000","last_undegraded":"2024-02-10T08:44:14.101159+0000","last_fullsized":"2024-02-10T08:44:13.099655+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.1a","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327565+0000","last_change":"2024-02-10T08:44:13.113276+0000","last_active":"2024-02-10T08:44:14.327565+0000","last_peered":"2024-02-10T08:44:14.327565+0000","last_clean":"2024-02-10T08:44:14.327565+0000","last_became_active":"2024-02-10T08:44:13.112887+0000","last_became_peered":"2024-02-10T08:44:13.112887+0000","last_unstale":"2024-02-10T08:44:14.327565+0000","last_undegraded":"2024-02-10T08:44:14.327565+0000","last_fullsized":"2024-02-10T08:44:14.327565+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.19","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104653+0000","last_change":"2024-02-10T08:44:13.141566+0000","last_active":"2024-02-10T08:44:14.104653+0000","last_peered":"2024-02-10T08:44:14.104653+0000","last_clean":"2024-02-10T08:44:14.104653+0000","last_became_active":"2024-02-10T08:44:13.141301+0000","last_became_peered":"2024-02-10T08:44:13.141301+0000","last_unstale":"2024-02-10T08:44:14.104653+0000","last_undegraded":"2024-02-10T08:44:14.104653+0000","last_fullsized":"2024-02-10T08:44:14.104653+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.18","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.100743+0000","last_change":"2024-02-10T08:44:13.112396+0000","last_active":"2024-02-10T08:44:14.100743+0000","last_peered":"2024-02-10T08:44:14.100743+0000","last_clean":"2024-02-10T08:44:14.100743+0000","last_became_active":"2024-02-10T08:44:13.112124+0000","last_became_peered":"2024-02-10T08:44:13.112124+0000","last_unstale":"2024-02-10T08:44:14.100743+0000","last_undegraded":"2024-02-10T08:44:14.100743+0000","last_fullsized":"2024-02-10T08:44:14.100743+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.17","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.101754+0000","last_change":"2024-02-10T08:44:13.117993+0000","last_active":"2024-02-10T08:44:14.101754+0000","last_peered":"2024-02-10T08:44:14.101754+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.117993+0000","last_became_peered":"2024-02-10T08:44:13.117993+0000","last_unstale":"2024-02-10T08:44:14.101754+0000","last_undegraded":"2024-02-10T08:44:14.101754+0000","last_fullsized":"2024-02-10T08:44:13.099908+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.16","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101417+0000","last_change":"2024-02-10T08:44:13.118318+0000","last_active":"2024-02-10T08:44:14.101417+0000","last_peered":"2024-02-10T08:44:14.101417+0000","last_clean":"2024-02-10T08:44:14.101417+0000","last_became_active":"2024-02-10T08:44:13.118033+0000","last_became_peered":"2024-02-10T08:44:13.118033+0000","last_unstale":"2024-02-10T08:44:14.101417+0000","last_undegraded":"2024-02-10T08:44:14.101417+0000","last_fullsized":"2024-02-10T08:44:14.101417+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.15","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106008+0000","last_change":"2024-02-10T08:44:13.154163+0000","last_active":"2024-02-10T08:44:14.106008+0000","last_peered":"2024-02-10T08:44:14.106008+0000","last_clean":"2024-02-10T08:44:14.106008+0000","last_became_active":"2024-02-10T08:44:13.153889+0000","last_became_peered":"2024-02-10T08:44:13.153889+0000","last_unstale":"2024-02-10T08:44:14.106008+0000","last_undegraded":"2024-02-10T08:44:14.106008+0000","last_fullsized":"2024-02-10T08:44:14.106008+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.14","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.328180+0000","last_change":"2024-02-10T08:44:13.135652+0000","last_active":"2024-02-10T08:44:14.328180+0000","last_peered":"2024-02-10T08:44:14.328180+0000","last_clean":"2024-02-10T08:44:14.328180+0000","last_became_active":"2024-02-10T08:44:13.135374+0000","last_became_peered":"2024-02-10T08:44:13.135374+0000","last_unstale":"2024-02-10T08:44:14.328180+0000","last_undegraded":"2024-02-10T08:44:14.328180+0000","last_fullsized":"2024-02-10T08:44:14.328180+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.13","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327887+0000","last_change":"2024-02-10T08:44:13.148646+0000","last_active":"2024-02-10T08:44:14.327887+0000","last_peered":"2024-02-10T08:44:14.327887+0000","last_clean":"2024-02-10T08:44:14.327887+0000","last_became_active":"2024-02-10T08:44:13.148365+0000","last_became_peered":"2024-02-10T08:44:13.148365+0000","last_unstale":"2024-02-10T08:44:14.327887+0000","last_undegraded":"2024-02-10T08:44:14.327887+0000","last_fullsized":"2024-02-10T08:44:14.327887+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.12","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105435+0000","last_change":"2024-02-10T08:44:13.130348+0000","last_active":"2024-02-10T08:44:14.105435+0000","last_peered":"2024-02-10T08:44:14.105435+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.130348+0000","last_became_peered":"2024-02-10T08:44:13.130348+0000","last_unstale":"2024-02-10T08:44:14.105435+0000","last_undegraded":"2024-02-10T08:44:14.105435+0000","last_fullsized":"2024-02-10T08:44:13.103100+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"222.11","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327408+0000","last_change":"2024-02-10T08:44:13.130797+0000","last_active":"2024-02-10T08:44:14.327408+0000","last_peered":"2024-02-10T08:44:14.327408+0000","last_clean":"2024-02-10T08:44:14.327408+0000","last_became_active":"2024-02-10T08:44:13.130383+0000","last_became_peered":"2024-02-10T08:44:13.130383+0000","last_unstale":"2024-02-10T08:44:14.327408+0000","last_undegraded":"2024-02-10T08:44:14.327408+0000","last_fullsized":"2024-02-10T08:44:14.327408+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.10","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.394216+0000","last_change":"2024-02-10T08:44:13.136371+0000","last_active":"2024-02-10T08:44:14.394216+0000","last_peered":"2024-02-10T08:44:14.394216+0000","last_clean":"2024-02-10T08:44:14.394216+0000","last_became_active":"2024-02-10T08:44:13.136095+0000","last_became_peered":"2024-02-10T08:44:13.136095+0000","last_unstale":"2024-02-10T08:44:14.394216+0000","last_undegraded":"2024-02-10T08:44:14.394216+0000","last_fullsized":"2024-02-10T08:44:14.394216+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"222.f","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.328879+0000","last_change":"2024-02-10T08:44:13.136991+0000","last_active":"2024-02-10T08:44:14.328879+0000","last_peered":"2024-02-10T08:44:14.328879+0000","last_clean":"2024-02-10T08:44:14.328879+0000","last_became_active":"2024-02-10T08:44:13.136794+0000","last_became_peered":"2024-02-10T08:44:13.136794+0000","last_unstale":"2024-02-10T08:44:14.328879+0000","last_undegraded":"2024-02-10T08:44:14.328879+0000","last_fullsized":"2024-02-10T08:44:14.328879+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.e","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105038+0000","last_change":"2024-02-10T08:44:13.141337+0000","last_active":"2024-02-10T08:44:14.105038+0000","last_peered":"2024-02-10T08:44:14.105038+0000","last_clean":"2024-02-10T08:44:14.105038+0000","last_became_active":"2024-02-10T08:44:13.140852+0000","last_became_peered":"2024-02-10T08:44:13.140852+0000","last_unstale":"2024-02-10T08:44:14.105038+0000","last_undegraded":"2024-02-10T08:44:14.105038+0000","last_fullsized":"2024-02-10T08:44:14.105038+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.d","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405397+0000","last_change":"2024-02-10T08:44:13.127871+0000","last_active":"2024-02-10T08:44:14.405397+0000","last_peered":"2024-02-10T08:44:14.405397+0000","last_clean":"2024-02-10T08:44:14.405397+0000","last_became_active":"2024-02-10T08:44:13.127599+0000","last_became_peered":"2024-02-10T08:44:13.127599+0000","last_unstale":"2024-02-10T08:44:14.405397+0000","last_undegraded":"2024-02-10T08:44:14.405397+0000","last_fullsized":"2024-02-10T08:44:14.405397+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"222.c","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101583+0000","last_change":"2024-02-10T08:44:13.130725+0000","last_active":"2024-02-10T08:44:14.101583+0000","last_peered":"2024-02-10T08:44:14.101583+0000","last_clean":"2024-02-10T08:44:14.101583+0000","last_became_active":"2024-02-10T08:44:13.130449+0000","last_became_peered":"2024-02-10T08:44:13.130449+0000","last_unstale":"2024-02-10T08:44:14.101583+0000","last_undegraded":"2024-02-10T08:44:14.101583+0000","last_fullsized":"2024-02-10T08:44:14.101583+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"222.b","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105739+0000","last_change":"2024-02-10T08:44:13.112603+0000","last_active":"2024-02-10T08:44:14.105739+0000","last_peered":"2024-02-10T08:44:14.105739+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.112603+0000","last_became_peered":"2024-02-10T08:44:13.112603+0000","last_unstale":"2024-02-10T08:44:14.105739+0000","last_undegraded":"2024-02-10T08:44:14.105739+0000","last_fullsized":"2024-02-10T08:44:13.111040+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"222.a","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105523+0000","last_change":"2024-02-10T08:44:13.155111+0000","last_active":"2024-02-10T08:44:14.105523+0000","last_peered":"2024-02-10T08:44:14.105523+0000","last_clean":"2024-02-10T08:44:14.105523+0000","last_became_active":"2024-02-10T08:44:13.153487+0000","last_became_peered":"2024-02-10T08:44:13.153487+0000","last_unstale":"2024-02-10T08:44:14.105523+0000","last_undegraded":"2024-02-10T08:44:14.105523+0000","last_fullsized":"2024-02-10T08:44:14.105523+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.9","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.402072+0000","last_change":"2024-02-10T08:44:13.128171+0000","last_active":"2024-02-10T08:44:14.402072+0000","last_peered":"2024-02-10T08:44:14.402072+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.128171+0000","last_became_peered":"2024-02-10T08:44:13.128171+0000","last_unstale":"2024-02-10T08:44:14.402072+0000","last_undegraded":"2024-02-10T08:44:14.402072+0000","last_fullsized":"2024-02-10T08:44:13.106484+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"222.8","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.103558+0000","last_change":"2024-02-10T08:44:13.153970+0000","last_active":"2024-02-10T08:44:14.103558+0000","last_peered":"2024-02-10T08:44:14.103558+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.153970+0000","last_became_peered":"2024-02-10T08:44:13.153970+0000","last_unstale":"2024-02-10T08:44:14.103558+0000","last_undegraded":"2024-02-10T08:44:14.103558+0000","last_fullsized":"2024-02-10T08:44:13.102599+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.7","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106233+0000","last_change":"2024-02-10T08:44:13.137384+0000","last_active":"2024-02-10T08:44:14.106233+0000","last_peered":"2024-02-10T08:44:14.106233+0000","last_clean":"2024-02-10T08:44:14.106233+0000","last_became_active":"2024-02-10T08:44:13.137115+0000","last_became_peered":"2024-02-10T08:44:13.137115+0000","last_unstale":"2024-02-10T08:44:14.106233+0000","last_undegraded":"2024-02-10T08:44:14.106233+0000","last_fullsized":"2024-02-10T08:44:14.106233+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.6","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405799+0000","last_change":"2024-02-10T08:44:13.127325+0000","last_active":"2024-02-10T08:44:14.405799+0000","last_peered":"2024-02-10T08:44:14.405799+0000","last_clean":"2024-02-10T08:44:14.405799+0000","last_became_active":"2024-02-10T08:44:13.127072+0000","last_became_peered":"2024-02-10T08:44:13.127072+0000","last_unstale":"2024-02-10T08:44:14.405799+0000","last_undegraded":"2024-02-10T08:44:14.405799+0000","last_fullsized":"2024-02-10T08:44:14.405799+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"222.5","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.406136+0000","last_change":"2024-02-10T08:44:13.149028+0000","last_active":"2024-02-10T08:44:14.406136+0000","last_peered":"2024-02-10T08:44:14.406136+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.149028+0000","last_became_peered":"2024-02-10T08:44:13.149028+0000","last_unstale":"2024-02-10T08:44:14.406136+0000","last_undegraded":"2024-02-10T08:44:14.406136+0000","last_fullsized":"2024-02-10T08:44:13.106794+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"222.4","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.395029+0000","last_change":"2024-02-10T08:44:13.119596+0000","last_active":"2024-02-10T08:44:14.395029+0000","last_peered":"2024-02-10T08:44:14.395029+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.119596+0000","last_became_peered":"2024-02-10T08:44:13.119596+0000","last_unstale":"2024-02-10T08:44:14.395029+0000","last_undegraded":"2024-02-10T08:44:14.395029+0000","last_fullsized":"2024-02-10T08:44:13.111265+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"222.3","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.104979+0000","last_change":"2024-02-10T08:44:13.124979+0000","last_active":"2024-02-10T08:44:14.104979+0000","last_peered":"2024-02-10T08:44:14.104979+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.124979+0000","last_became_peered":"2024-02-10T08:44:13.124979+0000","last_unstale":"2024-02-10T08:44:14.104979+0000","last_undegraded":"2024-02-10T08:44:14.104979+0000","last_fullsized":"2024-02-10T08:44:13.111246+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"222.2","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105668+0000","last_change":"2024-02-10T08:44:13.141323+0000","last_active":"2024-02-10T08:44:14.105668+0000","last_peered":"2024-02-10T08:44:14.105668+0000","last_clean":"2024-02-10T08:44:14.105668+0000","last_became_active":"2024-02-10T08:44:13.141034+0000","last_became_peered":"2024-02-10T08:44:13.141034+0000","last_unstale":"2024-02-10T08:44:14.105668+0000","last_undegraded":"2024-02-10T08:44:14.105668+0000","last_fullsized":"2024-02-10T08:44:14.105668+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"222.1","version":"0'0","reported_seq":8,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.327478+0000","last_change":"2024-02-10T08:44:13.129995+0000","last_active":"2024-02-10T08:44:14.327478+0000","last_peered":"2024-02-10T08:44:14.327478+0000","last_clean":"2024-02-10T08:44:12.094850+0000","last_became_active":"2024-02-10T08:44:13.129995+0000","last_became_peered":"2024-02-10T08:44:13.129995+0000","last_unstale":"2024-02-10T08:44:14.327478+0000","last_undegraded":"2024-02-10T08:44:14.327478+0000","last_fullsized":"2024-02-10T08:44:13.108194+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"222.0","version":"0'0","reported_seq":11,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.401018+0000","last_change":"2024-02-10T08:44:13.142306+0000","last_active":"2024-02-10T08:44:14.401018+0000","last_peered":"2024-02-10T08:44:14.401018+0000","last_clean":"2024-02-10T08:44:14.401018+0000","last_became_active":"2024-02-10T08:44:13.141910+0000","last_became_peered":"2024-02-10T08:44:13.141910+0000","last_unstale":"2024-02-10T08:44:14.401018+0000","last_undegraded":"2024-02-10T08:44:14.401018+0000","last_fullsized":"2024-02-10T08:44:14.401018+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":506,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:12.094850+0000","last_clean_scrub_stamp":"2024-02-10T08:44:12.094850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.3","version":"493'36","reported_seq":503,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105047+0000","last_change":"2024-02-10T08:42:33.705136+0000","last_active":"2024-02-10T08:44:14.105047+0000","last_peered":"2024-02-10T08:44:14.105047+0000","last_clean":"2024-02-10T08:44:14.105047+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:44:14.105047+0000","last_undegraded":"2024-02-10T08:44:14.105047+0000","last_fullsized":"2024-02-10T08:44:14.105047+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'34","last_scrub_stamp":"2024-02-10T08:42:33.705063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:33.705063+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"504'37","reported_seq":586,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.105762+0000","last_change":"2024-02-10T08:43:36.076342+0000","last_active":"2024-02-10T08:44:14.105762+0000","last_peered":"2024-02-10T08:44:14.105762+0000","last_clean":"2024-02-10T08:43:31.057691+0000","last_became_active":"2024-02-10T08:43:36.076342+0000","last_became_peered":"2024-02-10T08:43:36.076342+0000","last_unstale":"2024-02-10T08:44:14.105762+0000","last_undegraded":"2024-02-10T08:43:36.061670+0000","last_fullsized":"2024-02-10T08:43:36.061548+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'29","last_scrub_stamp":"2024-02-10T08:42:28.650876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.650876+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":22,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.1","version":"401'42","reported_seq":408,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.402275+0000","last_change":"2024-02-10T08:43:36.073243+0000","last_active":"2024-02-10T08:44:14.402275+0000","last_peered":"2024-02-10T08:44:14.402275+0000","last_clean":"2024-02-10T08:41:37.100674+0000","last_became_active":"2024-02-10T08:43:36.073243+0000","last_became_peered":"2024-02-10T08:43:36.073243+0000","last_unstale":"2024-02-10T08:44:14.402275+0000","last_undegraded":"2024-02-10T08:43:36.066452+0000","last_fullsized":"2024-02-10T08:43:36.066344+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'42","last_scrub_stamp":"2024-02-10T08:42:31.735511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.735511+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.0","version":"371'108","reported_seq":751,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.103832+0000","last_change":"2024-02-10T08:43:18.114463+0000","last_active":"2024-02-10T08:44:14.103832+0000","last_peered":"2024-02-10T08:44:14.103832+0000","last_clean":"2024-02-10T08:44:14.103832+0000","last_became_active":"2024-02-10T08:43:18.113727+0000","last_became_peered":"2024-02-10T08:43:18.113727+0000","last_unstale":"2024-02-10T08:44:14.103832+0000","last_undegraded":"2024-02-10T08:44:14.103832+0000","last_fullsized":"2024-02-10T08:44:14.103832+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'108","last_scrub_stamp":"2024-02-10T08:42:46.327938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.327938+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"504'39","reported_seq":359,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.102177+0000","last_change":"2024-02-10T08:43:36.078057+0000","last_active":"2024-02-10T08:44:14.102177+0000","last_peered":"2024-02-10T08:44:14.102177+0000","last_clean":"2024-02-10T08:41:05.014702+0000","last_became_active":"2024-02-10T08:43:36.078057+0000","last_became_peered":"2024-02-10T08:43:36.078057+0000","last_unstale":"2024-02-10T08:44:14.102177+0000","last_undegraded":"2024-02-10T08:43:36.057295+0000","last_fullsized":"2024-02-10T08:43:36.057186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:32.704166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.704166+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2787,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.6","version":"495'79","reported_seq":544,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105382+0000","last_change":"2024-02-10T08:42:31.200907+0000","last_active":"2024-02-10T08:44:14.105382+0000","last_peered":"2024-02-10T08:44:14.105382+0000","last_clean":"2024-02-10T08:44:14.105382+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:44:14.105382+0000","last_undegraded":"2024-02-10T08:44:14.105382+0000","last_fullsized":"2024-02-10T08:44:14.105382+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'73","last_scrub_stamp":"2024-02-10T08:42:31.200824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.200824+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":31,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"495'63","reported_seq":573,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.395137+0000","last_change":"2024-02-10T08:43:36.065069+0000","last_active":"2024-02-10T08:44:14.395137+0000","last_peered":"2024-02-10T08:44:14.395137+0000","last_clean":"2024-02-10T08:43:11.067979+0000","last_became_active":"2024-02-10T08:43:36.065069+0000","last_became_peered":"2024-02-10T08:43:36.065069+0000","last_unstale":"2024-02-10T08:44:14.395137+0000","last_undegraded":"2024-02-10T08:43:36.060893+0000","last_fullsized":"2024-02-10T08:43:36.060775+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:42:37.799725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.799725+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":27,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.4","version":"504'27","reported_seq":342,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.106784+0000","last_change":"2024-02-10T08:43:36.075301+0000","last_active":"2024-02-10T08:44:14.106784+0000","last_peered":"2024-02-10T08:44:14.106784+0000","last_clean":"2024-02-10T08:40:31.992410+0000","last_became_active":"2024-02-10T08:43:36.075301+0000","last_became_peered":"2024-02-10T08:43:36.075301+0000","last_unstale":"2024-02-10T08:44:14.106784+0000","last_undegraded":"2024-02-10T08:43:36.063465+0000","last_fullsized":"2024-02-10T08:43:36.063358+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'23","last_scrub_stamp":"2024-02-10T08:42:38.786497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:38.786497+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":545,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.b","version":"395'105","reported_seq":739,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106056+0000","last_change":"2024-02-10T08:43:21.062588+0000","last_active":"2024-02-10T08:44:14.106056+0000","last_peered":"2024-02-10T08:44:14.106056+0000","last_clean":"2024-02-10T08:44:14.106056+0000","last_became_active":"2024-02-10T08:43:21.061839+0000","last_became_peered":"2024-02-10T08:43:21.061839+0000","last_unstale":"2024-02-10T08:44:14.106056+0000","last_undegraded":"2024-02-10T08:44:14.106056+0000","last_fullsized":"2024-02-10T08:44:14.106056+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'105","last_scrub_stamp":"2024-02-10T08:42:42.390288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.390288+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"494'30","reported_seq":474,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.101009+0000","last_change":"2024-02-10T08:43:36.069735+0000","last_active":"2024-02-10T08:44:14.101009+0000","last_peered":"2024-02-10T08:44:14.101009+0000","last_clean":"2024-02-10T08:42:41.079071+0000","last_became_active":"2024-02-10T08:43:36.069735+0000","last_became_peered":"2024-02-10T08:43:36.069735+0000","last_unstale":"2024-02-10T08:44:14.101009+0000","last_undegraded":"2024-02-10T08:43:36.056543+0000","last_fullsized":"2024-02-10T08:43:36.056433+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'23","last_scrub_stamp":"2024-02-10T08:42:30.719056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.719056+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.9","version":"494'117","reported_seq":742,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.104814+0000","last_change":"2024-02-10T08:43:36.076377+0000","last_active":"2024-02-10T08:44:14.104814+0000","last_peered":"2024-02-10T08:44:14.104814+0000","last_clean":"2024-02-10T08:43:31.055757+0000","last_became_active":"2024-02-10T08:43:36.076377+0000","last_became_peered":"2024-02-10T08:43:36.076377+0000","last_unstale":"2024-02-10T08:44:14.104814+0000","last_undegraded":"2024-02-10T08:43:36.060586+0000","last_fullsized":"2024-02-10T08:43:36.060472+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'115","last_scrub_stamp":"2024-02-10T08:42:27.509423+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.509423+0000","log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":38,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"494'100","reported_seq":615,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.103168+0000","last_change":"2024-02-10T08:43:36.065933+0000","last_active":"2024-02-10T08:44:14.103168+0000","last_peered":"2024-02-10T08:44:14.103168+0000","last_clean":"2024-02-10T08:43:11.133528+0000","last_became_active":"2024-02-10T08:43:36.065933+0000","last_became_peered":"2024-02-10T08:43:36.065933+0000","last_unstale":"2024-02-10T08:44:14.103168+0000","last_undegraded":"2024-02-10T08:43:36.056709+0000","last_fullsized":"2024-02-10T08:43:36.056582+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'86","last_scrub_stamp":"2024-02-10T08:42:34.734692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.734692+0000","log_size":100,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":32,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":63,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.f","version":"405'96","reported_seq":556,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.328926+0000","last_change":"2024-02-10T08:43:27.256781+0000","last_active":"2024-02-10T08:44:14.328926+0000","last_peered":"2024-02-10T08:44:14.328926+0000","last_clean":"2024-02-10T08:44:14.328926+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:44:14.328926+0000","last_undegraded":"2024-02-10T08:44:14.328926+0000","last_fullsized":"2024-02-10T08:44:14.328926+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:43:27.256674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.256674+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"495'58","reported_seq":512,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.406213+0000","last_change":"2024-02-10T08:43:36.074273+0000","last_active":"2024-02-10T08:44:14.406213+0000","last_peered":"2024-02-10T08:44:14.406213+0000","last_clean":"2024-02-10T08:43:31.058727+0000","last_became_active":"2024-02-10T08:43:36.074273+0000","last_became_peered":"2024-02-10T08:43:36.074273+0000","last_unstale":"2024-02-10T08:44:14.406213+0000","last_undegraded":"2024-02-10T08:43:36.069012+0000","last_fullsized":"2024-02-10T08:43:36.067912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:43:30.090939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.090939+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":20,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"423'78","reported_seq":550,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.328464+0000","last_change":"2024-02-10T08:43:36.072717+0000","last_active":"2024-02-10T08:44:14.328464+0000","last_peered":"2024-02-10T08:44:14.328464+0000","last_clean":"2024-02-10T08:43:31.156895+0000","last_became_active":"2024-02-10T08:43:36.072717+0000","last_became_peered":"2024-02-10T08:43:36.072717+0000","last_unstale":"2024-02-10T08:44:14.328464+0000","last_undegraded":"2024-02-10T08:43:36.064470+0000","last_fullsized":"2024-02-10T08:43:36.064396+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'78","last_scrub_stamp":"2024-02-10T08:43:23.272230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.272230+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"494'72","reported_seq":539,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106928+0000","last_change":"2024-02-10T08:43:14.433052+0000","last_active":"2024-02-10T08:44:14.106928+0000","last_peered":"2024-02-10T08:44:14.106928+0000","last_clean":"2024-02-10T08:44:14.106928+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:44:14.106928+0000","last_undegraded":"2024-02-10T08:44:14.106928+0000","last_fullsized":"2024-02-10T08:44:14.106928+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:43:14.432955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.432955+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":25,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"495'68","reported_seq":535,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101385+0000","last_change":"2024-02-10T08:42:25.057952+0000","last_active":"2024-02-10T08:44:14.101385+0000","last_peered":"2024-02-10T08:44:14.101385+0000","last_clean":"2024-02-10T08:44:14.101385+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:44:14.101385+0000","last_undegraded":"2024-02-10T08:44:14.101385+0000","last_fullsized":"2024-02-10T08:44:14.101385+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'56","last_scrub_stamp":"2024-02-10T08:41:49.073509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.073509+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":9,"num_write":36,"num_write_kb":3935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"495'104","reported_seq":285,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.396175+0000","last_change":"2024-02-10T08:43:36.073079+0000","last_active":"2024-02-10T08:44:14.396175+0000","last_peered":"2024-02-10T08:44:14.396175+0000","last_clean":"2024-02-10T08:43:31.060665+0000","last_became_active":"2024-02-10T08:43:36.073079+0000","last_became_peered":"2024-02-10T08:43:36.073079+0000","last_unstale":"2024-02-10T08:44:14.396175+0000","last_undegraded":"2024-02-10T08:43:36.062346+0000","last_fullsized":"2024-02-10T08:43:36.062237+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":33,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"495'109","reported_seq":568,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.100966+0000","last_change":"2024-02-10T08:43:36.063661+0000","last_active":"2024-02-10T08:44:14.100966+0000","last_peered":"2024-02-10T08:44:14.100966+0000","last_clean":"2024-02-10T08:43:31.061494+0000","last_became_active":"2024-02-10T08:43:36.063661+0000","last_became_peered":"2024-02-10T08:43:36.063661+0000","last_unstale":"2024-02-10T08:44:14.100966+0000","last_undegraded":"2024-02-10T08:43:36.056704+0000","last_fullsized":"2024-02-10T08:43:36.056581+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'105","last_scrub_stamp":"2024-02-10T08:42:30.099763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.099763+0000","log_size":109,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":38,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"504'73","reported_seq":98,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.394765+0000","last_change":"2024-02-10T08:43:17.247458+0000","last_active":"2024-02-10T08:44:14.394765+0000","last_peered":"2024-02-10T08:44:14.394765+0000","last_clean":"2024-02-10T08:44:14.394765+0000","last_became_active":"2024-02-10T08:43:15.134958+0000","last_became_peered":"2024-02-10T08:43:15.134958+0000","last_unstale":"2024-02-10T08:44:14.394765+0000","last_undegraded":"2024-02-10T08:44:14.394765+0000","last_fullsized":"2024-02-10T08:44:14.394765+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'72","last_scrub_stamp":"2024-02-10T08:42:01.488098+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:42:01.488098+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":611,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"392'87","reported_seq":109,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.396728+0000","last_change":"2024-02-10T08:43:15.184580+0000","last_active":"2024-02-10T08:44:14.396728+0000","last_peered":"2024-02-10T08:44:14.396728+0000","last_clean":"2024-02-10T08:44:14.396728+0000","last_became_active":"2024-02-10T08:43:15.138604+0000","last_became_peered":"2024-02-10T08:43:15.138604+0000","last_unstale":"2024-02-10T08:44:14.396728+0000","last_undegraded":"2024-02-10T08:44:14.396728+0000","last_fullsized":"2024-02-10T08:44:14.396728+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:41:57.565097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.565097+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"494'66","reported_seq":655,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.406041+0000","last_change":"2024-02-10T08:43:21.049927+0000","last_active":"2024-02-10T08:44:14.406041+0000","last_peered":"2024-02-10T08:44:14.406041+0000","last_clean":"2024-02-10T08:44:14.406041+0000","last_became_active":"2024-02-10T08:43:15.137295+0000","last_became_peered":"2024-02-10T08:43:15.137295+0000","last_unstale":"2024-02-10T08:44:14.406041+0000","last_undegraded":"2024-02-10T08:44:14.406041+0000","last_fullsized":"2024-02-10T08:44:14.406041+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:43:21.049825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.049825+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":26,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"494'51","reported_seq":519,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.101336+0000","last_change":"2024-02-10T08:43:36.069660+0000","last_active":"2024-02-10T08:44:14.101336+0000","last_peered":"2024-02-10T08:44:14.101336+0000","last_clean":"2024-02-10T08:43:26.149056+0000","last_became_active":"2024-02-10T08:43:36.069660+0000","last_became_peered":"2024-02-10T08:43:36.069660+0000","last_unstale":"2024-02-10T08:44:14.101336+0000","last_undegraded":"2024-02-10T08:43:36.057367+0000","last_fullsized":"2024-02-10T08:43:36.056883+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'34","last_scrub_stamp":"2024-02-10T08:42:39.756505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:39.756505+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.14","version":"504'39","reported_seq":511,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.405207+0000","last_change":"2024-02-10T08:43:36.089221+0000","last_active":"2024-02-10T08:44:14.405207+0000","last_peered":"2024-02-10T08:44:14.405207+0000","last_clean":"2024-02-10T08:43:31.058392+0000","last_became_active":"2024-02-10T08:43:36.089221+0000","last_became_peered":"2024-02-10T08:43:36.089221+0000","last_unstale":"2024-02-10T08:44:14.405207+0000","last_undegraded":"2024-02-10T08:43:36.071737+0000","last_fullsized":"2024-02-10T08:43:36.071615+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:43:23.027840+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.027840+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1359,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"504'72","reported_seq":632,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.402618+0000","last_change":"2024-02-10T08:43:36.082574+0000","last_active":"2024-02-10T08:44:14.402618+0000","last_peered":"2024-02-10T08:44:14.402618+0000","last_clean":"2024-02-10T08:43:31.057526+0000","last_became_active":"2024-02-10T08:43:36.082574+0000","last_became_peered":"2024-02-10T08:43:36.082574+0000","last_unstale":"2024-02-10T08:44:14.402618+0000","last_undegraded":"2024-02-10T08:43:36.072001+0000","last_fullsized":"2024-02-10T08:43:36.071880+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:42:47.608965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.608965+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2372,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1a","version":"494'73","reported_seq":529,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105601+0000","last_change":"2024-02-10T08:42:36.726157+0000","last_active":"2024-02-10T08:44:14.105601+0000","last_peered":"2024-02-10T08:44:14.105601+0000","last_clean":"2024-02-10T08:44:14.105601+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:44:14.105601+0000","last_undegraded":"2024-02-10T08:44:14.105601+0000","last_fullsized":"2024-02-10T08:44:14.105601+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'71","last_scrub_stamp":"2024-02-10T08:42:36.726091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:36.726091+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":580,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104969+0000","last_change":"2024-02-10T08:43:16.354493+0000","last_active":"2024-02-10T08:44:14.104969+0000","last_peered":"2024-02-10T08:44:14.104969+0000","last_clean":"2024-02-10T08:44:14.104969+0000","last_became_active":"2024-02-10T08:43:15.130144+0000","last_became_peered":"2024-02-10T08:43:15.130144+0000","last_unstale":"2024-02-10T08:44:14.104969+0000","last_undegraded":"2024-02-10T08:44:14.104969+0000","last_fullsized":"2024-02-10T08:44:14.104969+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-10T08:42:35.744580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.744580+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"401'38","reported_seq":390,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.401192+0000","last_change":"2024-02-10T08:43:36.081808+0000","last_active":"2024-02-10T08:44:14.401192+0000","last_peered":"2024-02-10T08:44:14.401192+0000","last_clean":"2024-02-10T08:41:37.103139+0000","last_became_active":"2024-02-10T08:43:36.081808+0000","last_became_peered":"2024-02-10T08:43:36.081808+0000","last_unstale":"2024-02-10T08:44:14.401192+0000","last_undegraded":"2024-02-10T08:43:36.066395+0000","last_fullsized":"2024-02-10T08:43:36.066281+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'38","last_scrub_stamp":"2024-02-10T08:42:41.708918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.708918+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1f","version":"494'44","reported_seq":521,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102331+0000","last_change":"2024-02-10T08:42:25.058674+0000","last_active":"2024-02-10T08:44:14.102331+0000","last_peered":"2024-02-10T08:44:14.102331+0000","last_clean":"2024-02-10T08:44:14.102331+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:44:14.102331+0000","last_undegraded":"2024-02-10T08:44:14.102331+0000","last_fullsized":"2024-02-10T08:44:14.102331+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'36","last_scrub_stamp":"2024-02-10T08:41:46.085763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.085763+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"495'21","reported_seq":475,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104589+0000","last_change":"2024-02-10T08:42:25.056249+0000","last_active":"2024-02-10T08:44:14.104589+0000","last_peered":"2024-02-10T08:44:14.104589+0000","last_clean":"2024-02-10T08:44:14.104589+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:44:14.104589+0000","last_undegraded":"2024-02-10T08:44:14.104589+0000","last_fullsized":"2024-02-10T08:44:14.104589+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'17","last_scrub_stamp":"2024-02-10T08:42:04.642908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.642908+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"504'23","reported_seq":538,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405587+0000","last_change":"2024-02-10T08:43:33.115605+0000","last_active":"2024-02-10T08:44:14.405587+0000","last_peered":"2024-02-10T08:44:14.405587+0000","last_clean":"2024-02-10T08:44:14.405587+0000","last_became_active":"2024-02-10T08:43:15.136520+0000","last_became_peered":"2024-02-10T08:43:15.136520+0000","last_unstale":"2024-02-10T08:44:14.405587+0000","last_undegraded":"2024-02-10T08:44:14.405587+0000","last_fullsized":"2024-02-10T08:44:14.405587+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-02-10T08:43:33.115538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:33.115538+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"504'36","reported_seq":515,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.106616+0000","last_change":"2024-02-10T08:43:36.076024+0000","last_active":"2024-02-10T08:44:14.106616+0000","last_peered":"2024-02-10T08:44:14.106616+0000","last_clean":"2024-02-10T08:43:31.066314+0000","last_became_active":"2024-02-10T08:43:36.076024+0000","last_became_peered":"2024-02-10T08:43:36.076024+0000","last_unstale":"2024-02-10T08:44:14.106616+0000","last_undegraded":"2024-02-10T08:43:36.063160+0000","last_fullsized":"2024-02-10T08:43:36.063048+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:41.415071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.415071+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1245,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"481'17","reported_seq":126,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.395584+0000","last_change":"2024-02-10T08:43:36.074524+0000","last_active":"2024-02-10T08:44:14.395584+0000","last_peered":"2024-02-10T08:44:14.395584+0000","last_clean":"2024-02-10T08:43:31.610321+0000","last_became_active":"2024-02-10T08:43:36.074524+0000","last_became_peered":"2024-02-10T08:43:36.074524+0000","last_unstale":"2024-02-10T08:44:14.395584+0000","last_undegraded":"2024-02-10T08:43:36.061132+0000","last_fullsized":"2024-02-10T08:43:36.061013+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'13","last_scrub_stamp":"2024-02-10T08:43:10.390465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.390465+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":576,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.100955+0000","last_change":"2024-02-10T08:43:15.731491+0000","last_active":"2024-02-10T08:44:14.100955+0000","last_peered":"2024-02-10T08:44:14.100955+0000","last_clean":"2024-02-10T08:44:14.100955+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:44:14.100955+0000","last_undegraded":"2024-02-10T08:44:14.100955+0000","last_fullsized":"2024-02-10T08:44:14.100955+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:42:25.091640+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.091640+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":500,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105585+0000","last_change":"2024-02-10T08:43:15.119398+0000","last_active":"2024-02-10T08:44:14.105585+0000","last_peered":"2024-02-10T08:44:14.105585+0000","last_clean":"2024-02-10T08:44:14.105585+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:44:14.105585+0000","last_undegraded":"2024-02-10T08:44:14.105585+0000","last_fullsized":"2024-02-10T08:44:14.105585+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.504915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.504915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":486,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101310+0000","last_change":"2024-02-10T08:42:41.108507+0000","last_active":"2024-02-10T08:44:14.101310+0000","last_peered":"2024-02-10T08:44:14.101310+0000","last_clean":"2024-02-10T08:44:14.101310+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:44:14.101310+0000","last_undegraded":"2024-02-10T08:44:14.101310+0000","last_fullsized":"2024-02-10T08:44:14.101310+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:41.108418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.108418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.107062+0000","last_change":"2024-02-10T08:41:59.348271+0000","last_active":"2024-02-10T08:44:14.107062+0000","last_peered":"2024-02-10T08:44:14.107062+0000","last_clean":"2024-02-10T08:44:14.107062+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:44:14.107062+0000","last_undegraded":"2024-02-10T08:44:14.107062+0000","last_fullsized":"2024-02-10T08:44:14.107062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.348211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.348211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":469,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.106230+0000","last_change":"2024-02-10T08:43:36.079191+0000","last_active":"2024-02-10T08:44:14.106230+0000","last_peered":"2024-02-10T08:44:14.106230+0000","last_clean":"2024-02-10T08:43:31.066295+0000","last_became_active":"2024-02-10T08:43:36.079191+0000","last_became_peered":"2024-02-10T08:43:36.079191+0000","last_unstale":"2024-02-10T08:44:14.106230+0000","last_undegraded":"2024-02-10T08:43:36.059738+0000","last_fullsized":"2024-02-10T08:43:36.059623+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":493,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.104225+0000","last_change":"2024-02-10T08:43:36.085621+0000","last_active":"2024-02-10T08:44:14.104225+0000","last_peered":"2024-02-10T08:44:14.104225+0000","last_clean":"2024-02-10T08:43:31.059000+0000","last_became_active":"2024-02-10T08:43:36.085621+0000","last_became_peered":"2024-02-10T08:43:36.085621+0000","last_unstale":"2024-02-10T08:44:14.104225+0000","last_undegraded":"2024-02-10T08:44:14.104225+0000","last_fullsized":"2024-02-10T08:43:36.063451+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.696704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.696704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":502,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105975+0000","last_change":"2024-02-10T08:43:36.079793+0000","last_active":"2024-02-10T08:44:14.105975+0000","last_peered":"2024-02-10T08:44:14.105975+0000","last_clean":"2024-02-10T08:43:31.065630+0000","last_became_active":"2024-02-10T08:43:36.079793+0000","last_became_peered":"2024-02-10T08:43:36.079793+0000","last_unstale":"2024-02-10T08:44:14.105975+0000","last_undegraded":"2024-02-10T08:44:14.105975+0000","last_fullsized":"2024-02-10T08:43:36.061952+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395492+0000","last_change":"2024-02-10T08:43:15.129434+0000","last_active":"2024-02-10T08:44:14.395492+0000","last_peered":"2024-02-10T08:44:14.395492+0000","last_clean":"2024-02-10T08:44:14.395492+0000","last_became_active":"2024-02-10T08:43:15.128618+0000","last_became_peered":"2024-02-10T08:43:15.128618+0000","last_unstale":"2024-02-10T08:44:14.395492+0000","last_undegraded":"2024-02-10T08:44:14.395492+0000","last_fullsized":"2024-02-10T08:44:14.395492+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.297347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.297347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":439,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102740+0000","last_change":"2024-02-10T08:43:15.126531+0000","last_active":"2024-02-10T08:44:14.102740+0000","last_peered":"2024-02-10T08:44:14.102740+0000","last_clean":"2024-02-10T08:44:14.102740+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:44:14.102740+0000","last_undegraded":"2024-02-10T08:44:14.102740+0000","last_fullsized":"2024-02-10T08:44:14.102740+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:45.092620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.092620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"473'35","reported_seq":599,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327642+0000","last_change":"2024-02-10T08:43:29.286798+0000","last_active":"2024-02-10T08:44:14.327642+0000","last_peered":"2024-02-10T08:44:14.327642+0000","last_clean":"2024-02-10T08:44:14.327642+0000","last_became_active":"2024-02-10T08:43:15.136753+0000","last_became_peered":"2024-02-10T08:43:15.136753+0000","last_unstale":"2024-02-10T08:44:14.327642+0000","last_undegraded":"2024-02-10T08:44:14.327642+0000","last_fullsized":"2024-02-10T08:44:14.327642+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'35","last_scrub_stamp":"2024-02-10T08:43:29.286737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:29.286737+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":491,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.107439+0000","last_change":"2024-02-10T08:43:53.472550+0000","last_active":"2024-02-10T08:44:14.107439+0000","last_peered":"2024-02-10T08:44:14.107439+0000","last_clean":"2024-02-10T08:44:14.107439+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:44:14.107439+0000","last_undegraded":"2024-02-10T08:44:14.107439+0000","last_fullsized":"2024-02-10T08:44:14.107439+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:53.472449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:43:53.472449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":525,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.107032+0000","last_change":"2024-02-10T08:43:23.422453+0000","last_active":"2024-02-10T08:44:14.107032+0000","last_peered":"2024-02-10T08:44:14.107032+0000","last_clean":"2024-02-10T08:44:14.107032+0000","last_became_active":"2024-02-10T08:43:15.119008+0000","last_became_peered":"2024-02-10T08:43:15.119008+0000","last_unstale":"2024-02-10T08:44:14.107032+0000","last_undegraded":"2024-02-10T08:44:14.107032+0000","last_fullsized":"2024-02-10T08:44:14.107032+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.422346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.422346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.394656+0000","last_change":"2024-02-10T08:43:15.127270+0000","last_active":"2024-02-10T08:44:14.394656+0000","last_peered":"2024-02-10T08:44:14.394656+0000","last_clean":"2024-02-10T08:44:14.394656+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:44:14.394656+0000","last_undegraded":"2024-02-10T08:44:14.394656+0000","last_fullsized":"2024-02-10T08:44:14.394656+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:44.419908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.419908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.396535+0000","last_change":"2024-02-10T08:43:36.072003+0000","last_active":"2024-02-10T08:44:14.396535+0000","last_peered":"2024-02-10T08:44:14.396535+0000","last_clean":"2024-02-10T08:43:31.061070+0000","last_became_active":"2024-02-10T08:43:36.072003+0000","last_became_peered":"2024-02-10T08:43:36.072003+0000","last_unstale":"2024-02-10T08:44:14.396535+0000","last_undegraded":"2024-02-10T08:44:14.396535+0000","last_fullsized":"2024-02-10T08:43:36.062752+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":527,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.106525+0000","last_change":"2024-02-10T08:43:36.076710+0000","last_active":"2024-02-10T08:44:14.106525+0000","last_peered":"2024-02-10T08:44:14.106525+0000","last_clean":"2024-02-10T08:43:25.756727+0000","last_became_active":"2024-02-10T08:43:36.076710+0000","last_became_peered":"2024-02-10T08:43:36.076710+0000","last_unstale":"2024-02-10T08:44:14.106525+0000","last_undegraded":"2024-02-10T08:43:36.060349+0000","last_fullsized":"2024-02-10T08:43:36.060248+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:43:29.844454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:29.844454+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":316,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.401286+0000","last_change":"2024-02-10T08:41:16.638540+0000","last_active":"2024-02-10T08:44:14.401286+0000","last_peered":"2024-02-10T08:44:14.401286+0000","last_clean":"2024-02-10T08:44:14.401286+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:44:14.401286+0000","last_undegraded":"2024-02-10T08:44:14.401286+0000","last_fullsized":"2024-02-10T08:44:14.401286+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.638477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.638477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.104487+0000","last_change":"2024-02-10T08:43:36.076112+0000","last_active":"2024-02-10T08:44:14.104487+0000","last_peered":"2024-02-10T08:44:14.104487+0000","last_clean":"2024-02-10T08:43:31.060641+0000","last_became_active":"2024-02-10T08:43:36.076112+0000","last_became_peered":"2024-02-10T08:43:36.076112+0000","last_unstale":"2024-02-10T08:44:14.104487+0000","last_undegraded":"2024-02-10T08:44:14.104487+0000","last_fullsized":"2024-02-10T08:43:36.063764+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":482,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.106874+0000","last_change":"2024-02-10T08:43:36.075565+0000","last_active":"2024-02-10T08:44:14.106874+0000","last_peered":"2024-02-10T08:44:14.106874+0000","last_clean":"2024-02-10T08:43:31.066268+0000","last_became_active":"2024-02-10T08:43:36.075565+0000","last_became_peered":"2024-02-10T08:43:36.075565+0000","last_unstale":"2024-02-10T08:44:14.106874+0000","last_undegraded":"2024-02-10T08:44:14.106874+0000","last_fullsized":"2024-02-10T08:43:36.061936+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":37,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.403013+0000","last_change":"2024-02-10T08:43:36.089962+0000","last_active":"2024-02-10T08:44:14.403013+0000","last_peered":"2024-02-10T08:44:14.403013+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"2024-02-10T08:43:36.089962+0000","last_became_peered":"2024-02-10T08:43:36.089962+0000","last_unstale":"2024-02-10T08:44:14.403013+0000","last_undegraded":"2024-02-10T08:44:14.403013+0000","last_fullsized":"2024-02-10T08:43:36.068401+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":508,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.405607+0000","last_change":"2024-02-10T08:43:36.073845+0000","last_active":"2024-02-10T08:44:14.405607+0000","last_peered":"2024-02-10T08:44:14.405607+0000","last_clean":"2024-02-10T08:43:25.718471+0000","last_became_active":"2024-02-10T08:43:36.073845+0000","last_became_peered":"2024-02-10T08:43:36.073845+0000","last_unstale":"2024-02-10T08:44:14.405607+0000","last_undegraded":"2024-02-10T08:43:36.068245+0000","last_fullsized":"2024-02-10T08:43:36.068123+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":505,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105990+0000","last_change":"2024-02-10T08:43:15.130789+0000","last_active":"2024-02-10T08:44:14.105990+0000","last_peered":"2024-02-10T08:44:14.105990+0000","last_clean":"2024-02-10T08:44:14.105990+0000","last_became_active":"2024-02-10T08:43:15.130483+0000","last_became_peered":"2024-02-10T08:43:15.130483+0000","last_unstale":"2024-02-10T08:44:14.105990+0000","last_undegraded":"2024-02-10T08:44:14.105990+0000","last_fullsized":"2024-02-10T08:44:14.105990+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.744038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.744038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"469'16","reported_seq":691,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106422+0000","last_change":"2024-02-10T08:43:27.381317+0000","last_active":"2024-02-10T08:44:14.106422+0000","last_peered":"2024-02-10T08:44:14.106422+0000","last_clean":"2024-02-10T08:44:14.106422+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:44:14.106422+0000","last_undegraded":"2024-02-10T08:44:14.106422+0000","last_fullsized":"2024-02-10T08:44:14.106422+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'16","last_scrub_stamp":"2024-02-10T08:43:27.381246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.381246+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":37,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.106730+0000","last_change":"2024-02-10T08:43:36.084792+0000","last_active":"2024-02-10T08:44:14.106730+0000","last_peered":"2024-02-10T08:44:14.106730+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.084792+0000","last_became_peered":"2024-02-10T08:43:36.084792+0000","last_unstale":"2024-02-10T08:44:14.106730+0000","last_undegraded":"2024-02-10T08:44:14.106730+0000","last_fullsized":"2024-02-10T08:43:36.060560+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":419,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.401343+0000","last_change":"2024-02-10T08:43:36.082315+0000","last_active":"2024-02-10T08:44:14.401343+0000","last_peered":"2024-02-10T08:44:14.401343+0000","last_clean":"2024-02-10T08:43:31.057840+0000","last_became_active":"2024-02-10T08:43:36.082315+0000","last_became_peered":"2024-02-10T08:43:36.082315+0000","last_unstale":"2024-02-10T08:44:14.401343+0000","last_undegraded":"2024-02-10T08:44:14.401343+0000","last_fullsized":"2024-02-10T08:43:36.070820+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.219434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.219434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":529,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.103744+0000","last_change":"2024-02-10T08:43:36.088120+0000","last_active":"2024-02-10T08:44:14.103744+0000","last_peered":"2024-02-10T08:44:14.103744+0000","last_clean":"2024-02-10T08:43:31.056937+0000","last_became_active":"2024-02-10T08:43:36.088120+0000","last_became_peered":"2024-02-10T08:43:36.088120+0000","last_unstale":"2024-02-10T08:44:14.103744+0000","last_undegraded":"2024-02-10T08:43:36.061412+0000","last_fullsized":"2024-02-10T08:43:36.061296+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":521,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105280+0000","last_change":"2024-02-10T08:43:50.689633+0000","last_active":"2024-02-10T08:44:14.105280+0000","last_peered":"2024-02-10T08:44:14.105280+0000","last_clean":"2024-02-10T08:44:14.105280+0000","last_became_active":"2024-02-10T08:43:15.130693+0000","last_became_peered":"2024-02-10T08:43:15.130693+0000","last_unstale":"2024-02-10T08:44:14.105280+0000","last_undegraded":"2024-02-10T08:44:14.105280+0000","last_fullsized":"2024-02-10T08:44:14.105280+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:43:50.689512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:50.689512+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"481'17","reported_seq":65,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.101651+0000","last_change":"2024-02-10T08:43:36.076136+0000","last_active":"2024-02-10T08:44:14.101651+0000","last_peered":"2024-02-10T08:44:14.101651+0000","last_clean":"2024-02-10T08:33:25.224233+0000","last_became_active":"2024-02-10T08:43:36.076136+0000","last_became_peered":"2024-02-10T08:43:36.076136+0000","last_unstale":"2024-02-10T08:44:14.101651+0000","last_undegraded":"2024-02-10T08:43:36.056738+0000","last_fullsized":"2024-02-10T08:43:36.056588+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:42:44.647464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.647464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":481,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101718+0000","last_change":"2024-02-10T08:42:34.139662+0000","last_active":"2024-02-10T08:44:14.101718+0000","last_peered":"2024-02-10T08:44:14.101718+0000","last_clean":"2024-02-10T08:44:14.101718+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:44:14.101718+0000","last_undegraded":"2024-02-10T08:44:14.101718+0000","last_fullsized":"2024-02-10T08:44:14.101718+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.139528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.139528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":439,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101547+0000","last_change":"2024-02-10T08:43:00.552619+0000","last_active":"2024-02-10T08:44:14.101547+0000","last_peered":"2024-02-10T08:44:14.101547+0000","last_clean":"2024-02-10T08:44:14.101547+0000","last_became_active":"2024-02-10T08:43:00.552233+0000","last_became_peered":"2024-02-10T08:43:00.552233+0000","last_unstale":"2024-02-10T08:44:14.101547+0000","last_undegraded":"2024-02-10T08:44:14.101547+0000","last_fullsized":"2024-02-10T08:44:14.101547+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:54.063839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:54.063839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":508,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.328737+0000","last_change":"2024-02-10T08:43:36.068792+0000","last_active":"2024-02-10T08:44:14.328737+0000","last_peered":"2024-02-10T08:44:14.328737+0000","last_clean":"2024-02-10T08:43:25.758929+0000","last_became_active":"2024-02-10T08:43:36.068792+0000","last_became_peered":"2024-02-10T08:43:36.068792+0000","last_unstale":"2024-02-10T08:44:14.328737+0000","last_undegraded":"2024-02-10T08:43:36.064019+0000","last_fullsized":"2024-02-10T08:43:36.063905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.394527+0000","last_change":"2024-02-10T08:43:36.068338+0000","last_active":"2024-02-10T08:44:14.394527+0000","last_peered":"2024-02-10T08:44:14.394527+0000","last_clean":"2024-02-10T08:43:31.059980+0000","last_became_active":"2024-02-10T08:43:36.068338+0000","last_became_peered":"2024-02-10T08:43:36.068338+0000","last_unstale":"2024-02-10T08:44:14.394527+0000","last_undegraded":"2024-02-10T08:44:14.394527+0000","last_fullsized":"2024-02-10T08:43:36.060130+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.736740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.736740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"481'11","reported_seq":593,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.102575+0000","last_change":"2024-02-10T08:43:36.068155+0000","last_active":"2024-02-10T08:44:14.102575+0000","last_peered":"2024-02-10T08:44:14.102575+0000","last_clean":"2024-02-10T08:43:31.610349+0000","last_became_active":"2024-02-10T08:43:36.068155+0000","last_became_peered":"2024-02-10T08:43:36.068155+0000","last_unstale":"2024-02-10T08:44:14.102575+0000","last_undegraded":"2024-02-10T08:43:36.058153+0000","last_fullsized":"2024-02-10T08:43:36.058069+0000","mapping_epoch":480,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":470,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405389+0000","last_change":"2024-02-10T08:40:40.893385+0000","last_active":"2024-02-10T08:44:14.405389+0000","last_peered":"2024-02-10T08:44:14.405389+0000","last_clean":"2024-02-10T08:44:14.405389+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:44:14.405389+0000","last_undegraded":"2024-02-10T08:44:14.405389+0000","last_fullsized":"2024-02-10T08:44:14.405389+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:40:40.893261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.401468+0000","last_change":"2024-02-10T08:42:47.240729+0000","last_active":"2024-02-10T08:44:14.401468+0000","last_peered":"2024-02-10T08:44:14.401468+0000","last_clean":"2024-02-10T08:44:14.401468+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:44:14.401468+0000","last_undegraded":"2024-02-10T08:44:14.401468+0000","last_fullsized":"2024-02-10T08:44:14.401468+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.240624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.240624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"481'9","reported_seq":626,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.401658+0000","last_change":"2024-02-10T08:43:36.077627+0000","last_active":"2024-02-10T08:44:14.401658+0000","last_peered":"2024-02-10T08:44:14.401658+0000","last_clean":"2024-02-10T08:43:31.610208+0000","last_became_active":"2024-02-10T08:43:36.077627+0000","last_became_peered":"2024-02-10T08:43:36.077627+0000","last_unstale":"2024-02-10T08:44:14.401658+0000","last_undegraded":"2024-02-10T08:43:36.066022+0000","last_fullsized":"2024-02-10T08:43:36.065899+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":520,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.101117+0000","last_change":"2024-02-10T08:43:36.064143+0000","last_active":"2024-02-10T08:44:14.101117+0000","last_peered":"2024-02-10T08:44:14.101117+0000","last_clean":"2024-02-10T08:43:25.763333+0000","last_became_active":"2024-02-10T08:43:36.064143+0000","last_became_peered":"2024-02-10T08:43:36.064143+0000","last_unstale":"2024-02-10T08:44:14.101117+0000","last_undegraded":"2024-02-10T08:43:36.057979+0000","last_fullsized":"2024-02-10T08:43:36.057862+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:43:34.883363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.883363+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":510,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.329014+0000","last_change":"2024-02-10T08:43:21.356259+0000","last_active":"2024-02-10T08:44:14.329014+0000","last_peered":"2024-02-10T08:44:14.329014+0000","last_clean":"2024-02-10T08:44:14.329014+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:44:14.329014+0000","last_undegraded":"2024-02-10T08:44:14.329014+0000","last_fullsized":"2024-02-10T08:44:14.329014+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.356193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.356193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395422+0000","last_change":"2024-02-10T08:43:15.134249+0000","last_active":"2024-02-10T08:44:14.395422+0000","last_peered":"2024-02-10T08:44:14.395422+0000","last_clean":"2024-02-10T08:44:14.395422+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:44:14.395422+0000","last_undegraded":"2024-02-10T08:44:14.395422+0000","last_fullsized":"2024-02-10T08:44:14.395422+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:59.569193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:59.569193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":401,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.104298+0000","last_change":"2024-02-10T08:43:36.073331+0000","last_active":"2024-02-10T08:44:14.104298+0000","last_peered":"2024-02-10T08:44:14.104298+0000","last_clean":"2024-02-10T08:43:31.059113+0000","last_became_active":"2024-02-10T08:43:36.073331+0000","last_became_peered":"2024-02-10T08:43:36.073331+0000","last_unstale":"2024-02-10T08:44:14.104298+0000","last_undegraded":"2024-02-10T08:44:14.104298+0000","last_fullsized":"2024-02-10T08:43:36.063737+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":480,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.402779+0000","last_change":"2024-02-10T08:43:36.082324+0000","last_active":"2024-02-10T08:44:14.402779+0000","last_peered":"2024-02-10T08:44:14.402779+0000","last_clean":"2024-02-10T08:43:11.068543+0000","last_became_active":"2024-02-10T08:43:36.082324+0000","last_became_peered":"2024-02-10T08:43:36.082324+0000","last_unstale":"2024-02-10T08:44:14.402779+0000","last_undegraded":"2024-02-10T08:44:14.402779+0000","last_fullsized":"2024-02-10T08:43:36.067905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.619193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.619193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":640,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.401562+0000","last_change":"2024-02-10T08:43:16.451265+0000","last_active":"2024-02-10T08:44:14.401562+0000","last_peered":"2024-02-10T08:44:14.401562+0000","last_clean":"2024-02-10T08:44:14.401562+0000","last_became_active":"2024-02-10T08:43:15.136661+0000","last_became_peered":"2024-02-10T08:43:15.136661+0000","last_unstale":"2024-02-10T08:44:14.401562+0000","last_undegraded":"2024-02-10T08:44:14.401562+0000","last_fullsized":"2024-02-10T08:44:14.401562+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'26","last_scrub_stamp":"2024-02-10T08:41:07.712078+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:41:07.712078+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":495,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.104797+0000","last_change":"2024-02-10T08:43:36.068713+0000","last_active":"2024-02-10T08:44:14.104797+0000","last_peered":"2024-02-10T08:44:14.104797+0000","last_clean":"2024-02-10T08:43:31.057619+0000","last_became_active":"2024-02-10T08:43:36.068713+0000","last_became_peered":"2024-02-10T08:43:36.068713+0000","last_unstale":"2024-02-10T08:44:14.104797+0000","last_undegraded":"2024-02-10T08:43:36.060168+0000","last_fullsized":"2024-02-10T08:43:36.060055+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":37,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.401561+0000","last_change":"2024-02-10T08:43:36.080856+0000","last_active":"2024-02-10T08:44:14.401561+0000","last_peered":"2024-02-10T08:44:14.401561+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.080856+0000","last_became_peered":"2024-02-10T08:43:36.080856+0000","last_unstale":"2024-02-10T08:44:14.401561+0000","last_undegraded":"2024-02-10T08:44:14.401561+0000","last_fullsized":"2024-02-10T08:43:36.066562+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":413,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405299+0000","last_change":"2024-02-10T08:43:00.547238+0000","last_active":"2024-02-10T08:44:14.405299+0000","last_peered":"2024-02-10T08:44:14.405299+0000","last_clean":"2024-02-10T08:44:14.405299+0000","last_became_active":"2024-02-10T08:43:00.546139+0000","last_became_peered":"2024-02-10T08:43:00.546139+0000","last_unstale":"2024-02-10T08:44:14.405299+0000","last_undegraded":"2024-02-10T08:44:14.405299+0000","last_fullsized":"2024-02-10T08:44:14.405299+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":493,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326877+0000","last_change":"2024-02-10T08:43:59.374953+0000","last_active":"2024-02-10T08:44:14.326877+0000","last_peered":"2024-02-10T08:44:14.326877+0000","last_clean":"2024-02-10T08:44:14.326877+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:44:14.326877+0000","last_undegraded":"2024-02-10T08:44:14.326877+0000","last_fullsized":"2024-02-10T08:44:14.326877+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.374859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.374859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":531,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102284+0000","last_change":"2024-02-10T08:43:25.480698+0000","last_active":"2024-02-10T08:44:14.102284+0000","last_peered":"2024-02-10T08:44:14.102284+0000","last_clean":"2024-02-10T08:44:14.102284+0000","last_became_active":"2024-02-10T08:43:15.126404+0000","last_became_peered":"2024-02-10T08:43:15.126404+0000","last_unstale":"2024-02-10T08:44:14.102284+0000","last_undegraded":"2024-02-10T08:44:14.102284+0000","last_fullsized":"2024-02-10T08:44:14.102284+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'21","last_scrub_stamp":"2024-02-10T08:43:25.480620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.480620+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.396081+0000","last_change":"2024-02-10T08:43:36.073770+0000","last_active":"2024-02-10T08:44:14.396081+0000","last_peered":"2024-02-10T08:44:14.396081+0000","last_clean":"2024-02-10T08:43:31.060583+0000","last_became_active":"2024-02-10T08:43:36.073770+0000","last_became_peered":"2024-02-10T08:43:36.073770+0000","last_unstale":"2024-02-10T08:44:14.396081+0000","last_undegraded":"2024-02-10T08:44:14.396081+0000","last_fullsized":"2024-02-10T08:43:36.061914+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.297949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.297949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":406,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105217+0000","last_change":"2024-02-10T08:43:15.107144+0000","last_active":"2024-02-10T08:44:14.105217+0000","last_peered":"2024-02-10T08:44:14.105217+0000","last_clean":"2024-02-10T08:44:14.105217+0000","last_became_active":"2024-02-10T08:43:15.106310+0000","last_became_peered":"2024-02-10T08:43:15.106310+0000","last_unstale":"2024-02-10T08:44:14.105217+0000","last_undegraded":"2024-02-10T08:44:14.105217+0000","last_fullsized":"2024-02-10T08:44:14.105217+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.300183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.300183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":439,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106538+0000","last_change":"2024-02-10T08:43:59.552708+0000","last_active":"2024-02-10T08:44:14.106538+0000","last_peered":"2024-02-10T08:44:14.106538+0000","last_clean":"2024-02-10T08:44:14.106538+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:44:14.106538+0000","last_undegraded":"2024-02-10T08:44:14.106538+0000","last_fullsized":"2024-02-10T08:44:14.106538+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.552604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.552604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":512,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.102610+0000","last_change":"2024-02-10T08:43:36.069190+0000","last_active":"2024-02-10T08:44:14.102610+0000","last_peered":"2024-02-10T08:44:14.102610+0000","last_clean":"2024-02-10T08:43:31.062577+0000","last_became_active":"2024-02-10T08:43:36.069190+0000","last_became_peered":"2024-02-10T08:43:36.069190+0000","last_unstale":"2024-02-10T08:44:14.102610+0000","last_undegraded":"2024-02-10T08:44:14.102610+0000","last_fullsized":"2024-02-10T08:43:36.058346+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.561295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"473'24","reported_seq":508,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104844+0000","last_change":"2024-02-10T08:43:16.907675+0000","last_active":"2024-02-10T08:44:14.104844+0000","last_peered":"2024-02-10T08:44:14.104844+0000","last_clean":"2024-02-10T08:44:14.104844+0000","last_became_active":"2024-02-10T08:43:15.129639+0000","last_became_peered":"2024-02-10T08:43:15.129639+0000","last_unstale":"2024-02-10T08:44:14.104844+0000","last_undegraded":"2024-02-10T08:44:14.104844+0000","last_fullsized":"2024-02-10T08:44:14.104844+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:42:46.713412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.713412+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.394469+0000","last_change":"2024-02-10T08:43:15.133101+0000","last_active":"2024-02-10T08:44:14.394469+0000","last_peered":"2024-02-10T08:44:14.394469+0000","last_clean":"2024-02-10T08:44:14.394469+0000","last_became_active":"2024-02-10T08:43:15.132762+0000","last_became_peered":"2024-02-10T08:43:15.132762+0000","last_unstale":"2024-02-10T08:44:14.394469+0000","last_undegraded":"2024-02-10T08:44:14.394469+0000","last_fullsized":"2024-02-10T08:44:14.394469+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:30.494591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.494591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.402648+0000","last_change":"2024-02-10T08:43:36.072974+0000","last_active":"2024-02-10T08:44:14.402648+0000","last_peered":"2024-02-10T08:44:14.402648+0000","last_clean":"2024-02-10T08:33:17.081938+0000","last_became_active":"2024-02-10T08:43:36.072974+0000","last_became_peered":"2024-02-10T08:43:36.072974+0000","last_unstale":"2024-02-10T08:44:14.402648+0000","last_undegraded":"2024-02-10T08:44:14.402648+0000","last_fullsized":"2024-02-10T08:43:36.065921+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":482,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104614+0000","last_change":"2024-02-10T08:42:44.704196+0000","last_active":"2024-02-10T08:44:14.104614+0000","last_peered":"2024-02-10T08:44:14.104614+0000","last_clean":"2024-02-10T08:44:14.104614+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:44:14.104614+0000","last_undegraded":"2024-02-10T08:44:14.104614+0000","last_fullsized":"2024-02-10T08:44:14.104614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:42:44.704131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.704131+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":435,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327313+0000","last_change":"2024-02-10T08:42:32.390215+0000","last_active":"2024-02-10T08:44:14.327313+0000","last_peered":"2024-02-10T08:44:14.327313+0000","last_clean":"2024-02-10T08:44:14.327313+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:44:14.327313+0000","last_undegraded":"2024-02-10T08:44:14.327313+0000","last_fullsized":"2024-02-10T08:44:14.327313+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.390146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.390146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":62,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395660+0000","last_change":"2024-02-10T08:43:56.338478+0000","last_active":"2024-02-10T08:44:14.395660+0000","last_peered":"2024-02-10T08:44:14.395660+0000","last_clean":"2024-02-10T08:44:14.395660+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:44:14.395660+0000","last_undegraded":"2024-02-10T08:44:14.395660+0000","last_fullsized":"2024-02-10T08:44:14.395660+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:56.338378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:56.338378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":577,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.328854+0000","last_change":"2024-02-10T08:43:36.076013+0000","last_active":"2024-02-10T08:44:14.328854+0000","last_peered":"2024-02-10T08:44:14.328854+0000","last_clean":"2024-02-10T08:43:31.155643+0000","last_became_active":"2024-02-10T08:43:36.076013+0000","last_became_peered":"2024-02-10T08:43:36.076013+0000","last_unstale":"2024-02-10T08:44:14.328854+0000","last_undegraded":"2024-02-10T08:43:36.064979+0000","last_fullsized":"2024-02-10T08:43:36.064906+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.394688+0000","last_change":"2024-02-10T08:43:15.136009+0000","last_active":"2024-02-10T08:44:14.394688+0000","last_peered":"2024-02-10T08:44:14.394688+0000","last_clean":"2024-02-10T08:44:14.394688+0000","last_became_active":"2024-02-10T08:43:15.135366+0000","last_became_peered":"2024-02-10T08:43:15.135366+0000","last_unstale":"2024-02-10T08:44:14.394688+0000","last_undegraded":"2024-02-10T08:44:14.394688+0000","last_fullsized":"2024-02-10T08:44:14.394688+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:24.519144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:41:24.519144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":496,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102236+0000","last_change":"2024-02-10T08:43:15.127237+0000","last_active":"2024-02-10T08:44:14.102236+0000","last_peered":"2024-02-10T08:44:14.102236+0000","last_clean":"2024-02-10T08:44:14.102236+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:44:14.102236+0000","last_undegraded":"2024-02-10T08:44:14.102236+0000","last_fullsized":"2024-02-10T08:44:14.102236+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:13.978013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:13.978013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":481,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398941+0000","last_change":"2024-02-10T08:44:00.274641+0000","last_active":"2024-02-10T08:44:14.398941+0000","last_peered":"2024-02-10T08:44:14.398941+0000","last_clean":"2024-02-10T08:44:14.398941+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:44:14.398941+0000","last_undegraded":"2024-02-10T08:44:14.398941+0000","last_fullsized":"2024-02-10T08:44:14.398941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.274550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":478,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.102725+0000","last_change":"2024-02-10T08:43:36.077071+0000","last_active":"2024-02-10T08:44:14.102725+0000","last_peered":"2024-02-10T08:44:14.102725+0000","last_clean":"2024-02-10T08:43:31.062622+0000","last_became_active":"2024-02-10T08:43:36.077071+0000","last_became_peered":"2024-02-10T08:43:36.077071+0000","last_unstale":"2024-02-10T08:44:14.102725+0000","last_undegraded":"2024-02-10T08:44:14.102725+0000","last_fullsized":"2024-02-10T08:43:36.058617+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":432,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.100995+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:44:14.100995+0000","last_peered":"2024-02-10T08:44:14.100995+0000","last_clean":"2024-02-10T08:44:14.100995+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:44:14.100995+0000","last_undegraded":"2024-02-10T08:44:14.100995+0000","last_fullsized":"2024-02-10T08:44:14.100995+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":514,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105765+0000","last_change":"2024-02-10T08:43:32.421461+0000","last_active":"2024-02-10T08:44:14.105765+0000","last_peered":"2024-02-10T08:44:14.105765+0000","last_clean":"2024-02-10T08:44:14.105765+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:44:14.105765+0000","last_undegraded":"2024-02-10T08:44:14.105765+0000","last_fullsized":"2024-02-10T08:44:14.105765+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:32.421384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:32.421384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":581,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.103627+0000","last_change":"2024-02-10T08:43:24.674606+0000","last_active":"2024-02-10T08:44:14.103627+0000","last_peered":"2024-02-10T08:44:14.103627+0000","last_clean":"2024-02-10T08:44:14.103627+0000","last_became_active":"2024-02-10T08:43:15.129755+0000","last_became_peered":"2024-02-10T08:43:15.129755+0000","last_unstale":"2024-02-10T08:44:14.103627+0000","last_undegraded":"2024-02-10T08:44:14.103627+0000","last_fullsized":"2024-02-10T08:44:14.103627+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:43:24.674531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:24.674531+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":493,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105147+0000","last_change":"2024-02-10T08:42:29.250521+0000","last_active":"2024-02-10T08:44:14.105147+0000","last_peered":"2024-02-10T08:44:14.105147+0000","last_clean":"2024-02-10T08:44:14.105147+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:44:14.105147+0000","last_undegraded":"2024-02-10T08:44:14.105147+0000","last_fullsized":"2024-02-10T08:44:14.105147+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.250458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.250458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":1141,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-10T08:44:15.105190+0000","last_change":"2024-02-10T08:43:58.735413+0000","last_active":"2024-02-10T08:44:15.105190+0000","last_peered":"2024-02-10T08:44:15.105190+0000","last_clean":"2024-02-10T08:44:15.105190+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:44:15.105190+0000","last_undegraded":"2024-02-10T08:44:15.105190+0000","last_fullsized":"2024-02-10T08:44:15.105190+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:43:58.735346+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.735346+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":650,"num_read_kb":645,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395996+0000","last_change":"2024-02-10T08:43:15.134639+0000","last_active":"2024-02-10T08:44:14.395996+0000","last_peered":"2024-02-10T08:44:14.395996+0000","last_clean":"2024-02-10T08:44:14.395996+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:44:14.395996+0000","last_undegraded":"2024-02-10T08:44:14.395996+0000","last_fullsized":"2024-02-10T08:44:14.395996+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.329911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.329911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":452,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105357+0000","last_change":"2024-02-10T08:43:15.130234+0000","last_active":"2024-02-10T08:44:14.105357+0000","last_peered":"2024-02-10T08:44:14.105357+0000","last_clean":"2024-02-10T08:44:14.105357+0000","last_became_active":"2024-02-10T08:43:15.129900+0000","last_became_peered":"2024-02-10T08:43:15.129900+0000","last_unstale":"2024-02-10T08:44:14.105357+0000","last_undegraded":"2024-02-10T08:44:14.105357+0000","last_fullsized":"2024-02-10T08:44:14.105357+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:28.553564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.553564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":401,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.103975+0000","last_change":"2024-02-10T08:43:36.073644+0000","last_active":"2024-02-10T08:44:14.103975+0000","last_peered":"2024-02-10T08:44:14.103975+0000","last_clean":"2024-02-10T08:43:31.058736+0000","last_became_active":"2024-02-10T08:43:36.073644+0000","last_became_peered":"2024-02-10T08:43:36.073644+0000","last_unstale":"2024-02-10T08:44:14.103975+0000","last_undegraded":"2024-02-10T08:44:14.103975+0000","last_fullsized":"2024-02-10T08:43:36.062919+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"473'41","reported_seq":617,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101441+0000","last_change":"2024-02-10T08:42:43.109193+0000","last_active":"2024-02-10T08:44:14.101441+0000","last_peered":"2024-02-10T08:44:14.101441+0000","last_clean":"2024-02-10T08:44:14.101441+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:44:14.101441+0000","last_undegraded":"2024-02-10T08:44:14.101441+0000","last_fullsized":"2024-02-10T08:44:14.101441+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:42:43.109121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.109121+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":483,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101707+0000","last_change":"2024-02-10T08:42:37.145252+0000","last_active":"2024-02-10T08:44:14.101707+0000","last_peered":"2024-02-10T08:44:14.101707+0000","last_clean":"2024-02-10T08:44:14.101707+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:44:14.101707+0000","last_undegraded":"2024-02-10T08:44:14.101707+0000","last_fullsized":"2024-02-10T08:44:14.101707+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.145180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.145180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":494,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.406361+0000","last_change":"2024-02-10T08:43:36.083021+0000","last_active":"2024-02-10T08:44:14.406361+0000","last_peered":"2024-02-10T08:44:14.406361+0000","last_clean":"2024-02-10T08:43:31.058994+0000","last_became_active":"2024-02-10T08:43:36.083021+0000","last_became_peered":"2024-02-10T08:43:36.083021+0000","last_unstale":"2024-02-10T08:44:14.406361+0000","last_undegraded":"2024-02-10T08:44:14.406361+0000","last_fullsized":"2024-02-10T08:43:36.072420+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":452,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104015+0000","last_change":"2024-02-10T08:43:15.119736+0000","last_active":"2024-02-10T08:44:14.104015+0000","last_peered":"2024-02-10T08:44:14.104015+0000","last_clean":"2024-02-10T08:44:14.104015+0000","last_became_active":"2024-02-10T08:43:15.119340+0000","last_became_peered":"2024-02-10T08:43:15.119340+0000","last_unstale":"2024-02-10T08:44:14.104015+0000","last_undegraded":"2024-02-10T08:44:14.104015+0000","last_fullsized":"2024-02-10T08:44:14.104015+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:04.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:04.361430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":509,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326916+0000","last_change":"2024-02-10T08:43:31.263741+0000","last_active":"2024-02-10T08:44:14.326916+0000","last_peered":"2024-02-10T08:44:14.326916+0000","last_clean":"2024-02-10T08:44:14.326916+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:44:14.326916+0000","last_undegraded":"2024-02-10T08:44:14.326916+0000","last_fullsized":"2024-02-10T08:44:14.326916+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:43:31.263675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:31.263675+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":528,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.106239+0000","last_change":"2024-02-10T08:43:36.071773+0000","last_active":"2024-02-10T08:44:14.106239+0000","last_peered":"2024-02-10T08:44:14.106239+0000","last_clean":"2024-02-10T08:43:31.056280+0000","last_became_active":"2024-02-10T08:43:36.071773+0000","last_became_peered":"2024-02-10T08:43:36.071773+0000","last_unstale":"2024-02-10T08:44:14.106239+0000","last_undegraded":"2024-02-10T08:43:36.059969+0000","last_fullsized":"2024-02-10T08:43:36.059851+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:42:40.687013+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.687013+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":514,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104191+0000","last_change":"2024-02-10T08:43:27.605903+0000","last_active":"2024-02-10T08:44:14.104191+0000","last_peered":"2024-02-10T08:44:14.104191+0000","last_clean":"2024-02-10T08:44:14.104191+0000","last_became_active":"2024-02-10T08:43:15.129787+0000","last_became_peered":"2024-02-10T08:43:15.129787+0000","last_unstale":"2024-02-10T08:44:14.104191+0000","last_undegraded":"2024-02-10T08:44:14.104191+0000","last_fullsized":"2024-02-10T08:44:14.104191+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.605831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.605831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":395,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398102+0000","last_change":"2024-02-10T08:42:40.143847+0000","last_active":"2024-02-10T08:44:14.398102+0000","last_peered":"2024-02-10T08:44:14.398102+0000","last_clean":"2024-02-10T08:44:14.398102+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:44:14.398102+0000","last_undegraded":"2024-02-10T08:44:14.398102+0000","last_fullsized":"2024-02-10T08:44:14.398102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.143787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.143787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.406221+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:44:14.406221+0000","last_peered":"2024-02-10T08:44:14.406221+0000","last_clean":"2024-02-10T08:44:14.406221+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:44:14.406221+0000","last_undegraded":"2024-02-10T08:44:14.406221+0000","last_fullsized":"2024-02-10T08:44:14.406221+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":510,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104336+0000","last_change":"2024-02-10T08:43:58.505496+0000","last_active":"2024-02-10T08:44:14.104336+0000","last_peered":"2024-02-10T08:44:14.104336+0000","last_clean":"2024-02-10T08:44:14.104336+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:44:14.104336+0000","last_undegraded":"2024-02-10T08:44:14.104336+0000","last_fullsized":"2024-02-10T08:44:14.104336+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.505431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.505431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":515,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.105389+0000","last_change":"2024-02-10T08:43:36.079165+0000","last_active":"2024-02-10T08:44:14.105389+0000","last_peered":"2024-02-10T08:44:14.105389+0000","last_clean":"2024-02-10T08:43:25.579489+0000","last_became_active":"2024-02-10T08:43:36.079165+0000","last_became_peered":"2024-02-10T08:43:36.079165+0000","last_unstale":"2024-02-10T08:44:14.105389+0000","last_undegraded":"2024-02-10T08:43:36.061039+0000","last_fullsized":"2024-02-10T08:43:36.060910+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":516,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.396739+0000","last_change":"2024-02-10T08:43:36.072147+0000","last_active":"2024-02-10T08:44:14.396739+0000","last_peered":"2024-02-10T08:44:14.396739+0000","last_clean":"2024-02-10T08:43:11.810746+0000","last_became_active":"2024-02-10T08:43:36.072147+0000","last_became_peered":"2024-02-10T08:43:36.072147+0000","last_unstale":"2024-02-10T08:44:14.396739+0000","last_undegraded":"2024-02-10T08:43:36.061852+0000","last_fullsized":"2024-02-10T08:43:36.061736+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:43:11.810639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.810639+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":493,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.403237+0000","last_change":"2024-02-10T08:43:15.126259+0000","last_active":"2024-02-10T08:44:14.403237+0000","last_peered":"2024-02-10T08:44:14.403237+0000","last_clean":"2024-02-10T08:44:14.403237+0000","last_became_active":"2024-02-10T08:43:15.125756+0000","last_became_peered":"2024-02-10T08:43:15.125756+0000","last_unstale":"2024-02-10T08:44:14.403237+0000","last_undegraded":"2024-02-10T08:44:14.403237+0000","last_fullsized":"2024-02-10T08:44:14.403237+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":448,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106340+0000","last_change":"2024-02-10T08:43:54.509761+0000","last_active":"2024-02-10T08:44:14.106340+0000","last_peered":"2024-02-10T08:44:14.106340+0000","last_clean":"2024-02-10T08:44:14.106340+0000","last_became_active":"2024-02-10T08:43:15.119419+0000","last_became_peered":"2024-02-10T08:43:15.119419+0000","last_unstale":"2024-02-10T08:44:14.106340+0000","last_undegraded":"2024-02-10T08:44:14.106340+0000","last_fullsized":"2024-02-10T08:44:14.106340+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:54.509649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:54.509649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":508,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.406430+0000","last_change":"2024-02-10T08:43:19.456744+0000","last_active":"2024-02-10T08:44:14.406430+0000","last_peered":"2024-02-10T08:44:14.406430+0000","last_clean":"2024-02-10T08:44:14.406430+0000","last_became_active":"2024-02-10T08:43:15.137834+0000","last_became_peered":"2024-02-10T08:43:15.137834+0000","last_unstale":"2024-02-10T08:44:14.406430+0000","last_undegraded":"2024-02-10T08:44:14.406430+0000","last_fullsized":"2024-02-10T08:44:14.406430+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_deep_scrub":"84'13","last_deep_scrub_stamp":"2024-02-10T08:41:23.736922+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.736922+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":497,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.104219+0000","last_change":"2024-02-10T08:43:36.075991+0000","last_active":"2024-02-10T08:44:14.104219+0000","last_peered":"2024-02-10T08:44:14.104219+0000","last_clean":"2024-02-10T08:43:31.065892+0000","last_became_active":"2024-02-10T08:43:36.075991+0000","last_became_peered":"2024-02-10T08:43:36.075991+0000","last_unstale":"2024-02-10T08:44:14.104219+0000","last_undegraded":"2024-02-10T08:44:14.104219+0000","last_fullsized":"2024-02-10T08:43:36.062400+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":500,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.107216+0000","last_change":"2024-02-10T08:43:36.075699+0000","last_active":"2024-02-10T08:44:14.107216+0000","last_peered":"2024-02-10T08:44:14.107216+0000","last_clean":"2024-02-10T08:43:31.066554+0000","last_became_active":"2024-02-10T08:43:36.075699+0000","last_became_peered":"2024-02-10T08:43:36.075699+0000","last_unstale":"2024-02-10T08:44:14.107216+0000","last_undegraded":"2024-02-10T08:44:14.107216+0000","last_fullsized":"2024-02-10T08:43:36.063912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.347685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.347685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105703+0000","last_change":"2024-02-10T08:43:36.075546+0000","last_active":"2024-02-10T08:44:14.105703+0000","last_peered":"2024-02-10T08:44:14.105703+0000","last_clean":"2024-02-10T08:43:11.068374+0000","last_became_active":"2024-02-10T08:43:36.075546+0000","last_became_peered":"2024-02-10T08:43:36.075546+0000","last_unstale":"2024-02-10T08:44:14.105703+0000","last_undegraded":"2024-02-10T08:44:14.105703+0000","last_fullsized":"2024-02-10T08:43:36.057136+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":531,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405935+0000","last_change":"2024-02-10T08:43:52.173893+0000","last_active":"2024-02-10T08:44:14.405935+0000","last_peered":"2024-02-10T08:44:14.405935+0000","last_clean":"2024-02-10T08:44:14.405935+0000","last_became_active":"2024-02-10T08:43:15.125753+0000","last_became_peered":"2024-02-10T08:43:15.125753+0000","last_unstale":"2024-02-10T08:44:14.405935+0000","last_undegraded":"2024-02-10T08:44:14.405935+0000","last_fullsized":"2024-02-10T08:44:14.405935+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:43:52.173797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:52.173797+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":489,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.327986+0000","last_change":"2024-02-10T08:43:36.068896+0000","last_active":"2024-02-10T08:44:14.327986+0000","last_peered":"2024-02-10T08:44:14.327986+0000","last_clean":"2024-02-10T08:43:31.156388+0000","last_became_active":"2024-02-10T08:43:36.068896+0000","last_became_peered":"2024-02-10T08:43:36.068896+0000","last_unstale":"2024-02-10T08:44:14.327986+0000","last_undegraded":"2024-02-10T08:44:14.327986+0000","last_fullsized":"2024-02-10T08:43:36.062533+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.395410+0000","last_change":"2024-02-10T08:43:36.074707+0000","last_active":"2024-02-10T08:44:14.395410+0000","last_peered":"2024-02-10T08:44:14.395410+0000","last_clean":"2024-02-10T08:43:31.060201+0000","last_became_active":"2024-02-10T08:43:36.074707+0000","last_became_peered":"2024-02-10T08:43:36.074707+0000","last_unstale":"2024-02-10T08:44:14.395410+0000","last_undegraded":"2024-02-10T08:44:14.395410+0000","last_fullsized":"2024-02-10T08:43:36.061288+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":427,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.102166+0000","last_change":"2024-02-10T08:43:36.065698+0000","last_active":"2024-02-10T08:44:14.102166+0000","last_peered":"2024-02-10T08:44:14.102166+0000","last_clean":"2024-02-10T08:43:31.061677+0000","last_became_active":"2024-02-10T08:43:36.065698+0000","last_became_peered":"2024-02-10T08:43:36.065698+0000","last_unstale":"2024-02-10T08:44:14.102166+0000","last_undegraded":"2024-02-10T08:44:14.102166+0000","last_fullsized":"2024-02-10T08:43:36.058144+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":551,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405442+0000","last_change":"2024-02-10T08:43:19.438856+0000","last_active":"2024-02-10T08:44:14.405442+0000","last_peered":"2024-02-10T08:44:14.405442+0000","last_clean":"2024-02-10T08:44:14.405442+0000","last_became_active":"2024-02-10T08:43:15.137251+0000","last_became_peered":"2024-02-10T08:43:15.137251+0000","last_unstale":"2024-02-10T08:44:14.405442+0000","last_undegraded":"2024-02-10T08:44:14.405442+0000","last_fullsized":"2024-02-10T08:44:14.405442+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":501,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101566+0000","last_change":"2024-02-10T08:43:15.112162+0000","last_active":"2024-02-10T08:44:14.101566+0000","last_peered":"2024-02-10T08:44:14.101566+0000","last_clean":"2024-02-10T08:44:14.101566+0000","last_became_active":"2024-02-10T08:43:15.111824+0000","last_became_peered":"2024-02-10T08:43:15.111824+0000","last_unstale":"2024-02-10T08:44:14.101566+0000","last_undegraded":"2024-02-10T08:44:14.101566+0000","last_fullsized":"2024-02-10T08:44:14.101566+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:42.059708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.059708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":626,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101559+0000","last_change":"2024-02-10T08:42:28.090195+0000","last_active":"2024-02-10T08:44:14.101559+0000","last_peered":"2024-02-10T08:44:14.101559+0000","last_clean":"2024-02-10T08:44:14.101559+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:44:14.101559+0000","last_undegraded":"2024-02-10T08:44:14.101559+0000","last_fullsized":"2024-02-10T08:44:14.101559+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:42:28.090111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.090111+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":66,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.396427+0000","last_change":"2024-02-10T08:43:36.078925+0000","last_active":"2024-02-10T08:44:14.396427+0000","last_peered":"2024-02-10T08:44:14.396427+0000","last_clean":"2024-02-10T08:43:31.060910+0000","last_became_active":"2024-02-10T08:43:36.078925+0000","last_became_peered":"2024-02-10T08:43:36.078925+0000","last_unstale":"2024-02-10T08:44:14.396427+0000","last_undegraded":"2024-02-10T08:44:14.396427+0000","last_fullsized":"2024-02-10T08:43:36.063718+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.241405+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":402,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.327516+0000","last_change":"2024-02-10T08:43:36.067846+0000","last_active":"2024-02-10T08:44:14.327516+0000","last_peered":"2024-02-10T08:44:14.327516+0000","last_clean":"2024-02-10T08:43:31.156305+0000","last_became_active":"2024-02-10T08:43:36.067846+0000","last_became_peered":"2024-02-10T08:43:36.067846+0000","last_unstale":"2024-02-10T08:44:14.327516+0000","last_undegraded":"2024-02-10T08:44:14.327516+0000","last_fullsized":"2024-02-10T08:43:36.063242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.357151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.357151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":539,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101451+0000","last_change":"2024-02-10T08:43:15.941418+0000","last_active":"2024-02-10T08:44:14.101451+0000","last_peered":"2024-02-10T08:44:14.101451+0000","last_clean":"2024-02-10T08:44:14.101451+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:44:14.101451+0000","last_undegraded":"2024-02-10T08:44:14.101451+0000","last_fullsized":"2024-02-10T08:44:14.101451+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":395,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.402765+0000","last_change":"2024-02-10T08:43:36.082360+0000","last_active":"2024-02-10T08:44:14.402765+0000","last_peered":"2024-02-10T08:44:14.402765+0000","last_clean":"2024-02-10T08:43:11.067893+0000","last_became_active":"2024-02-10T08:43:36.082360+0000","last_became_peered":"2024-02-10T08:43:36.082360+0000","last_unstale":"2024-02-10T08:44:14.402765+0000","last_undegraded":"2024-02-10T08:44:14.402765+0000","last_fullsized":"2024-02-10T08:43:36.067825+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:10.801476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.801476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":643,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405581+0000","last_change":"2024-02-10T08:43:19.433673+0000","last_active":"2024-02-10T08:44:14.405581+0000","last_peered":"2024-02-10T08:44:14.405581+0000","last_clean":"2024-02-10T08:44:14.405581+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:44:14.405581+0000","last_undegraded":"2024-02-10T08:44:14.405581+0000","last_fullsized":"2024-02-10T08:44:14.405581+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":37,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.106741+0000","last_change":"2024-02-10T08:43:36.075017+0000","last_active":"2024-02-10T08:44:14.106741+0000","last_peered":"2024-02-10T08:44:14.106741+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.075017+0000","last_became_peered":"2024-02-10T08:43:36.075017+0000","last_unstale":"2024-02-10T08:44:14.106741+0000","last_undegraded":"2024-02-10T08:44:14.106741+0000","last_fullsized":"2024-02-10T08:43:36.058808+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.786130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.786130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":438,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104959+0000","last_change":"2024-02-10T08:42:40.371733+0000","last_active":"2024-02-10T08:44:14.104959+0000","last_peered":"2024-02-10T08:44:14.104959+0000","last_clean":"2024-02-10T08:44:14.104959+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:44:14.104959+0000","last_undegraded":"2024-02-10T08:44:14.104959+0000","last_fullsized":"2024-02-10T08:44:14.104959+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:42:40.371655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.371655+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106069+0000","last_change":"2024-02-10T08:42:25.482887+0000","last_active":"2024-02-10T08:44:14.106069+0000","last_peered":"2024-02-10T08:44:14.106069+0000","last_clean":"2024-02-10T08:44:14.106069+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:44:14.106069+0000","last_undegraded":"2024-02-10T08:44:14.106069+0000","last_fullsized":"2024-02-10T08:44:14.106069+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'13","last_scrub_stamp":"2024-02-10T08:42:25.482788+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.482788+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":488,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326915+0000","last_change":"2024-02-10T08:43:53.319974+0000","last_active":"2024-02-10T08:44:14.326915+0000","last_peered":"2024-02-10T08:44:14.326915+0000","last_clean":"2024-02-10T08:44:14.326915+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:44:14.326915+0000","last_undegraded":"2024-02-10T08:44:14.326915+0000","last_fullsized":"2024-02-10T08:44:14.326915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:53.319909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:53.319909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":486,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398271+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:44:14.398271+0000","last_peered":"2024-02-10T08:44:14.398271+0000","last_clean":"2024-02-10T08:44:14.398271+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:44:14.398271+0000","last_undegraded":"2024-02-10T08:44:14.398271+0000","last_fullsized":"2024-02-10T08:44:14.398271+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":396,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101048+0000","last_change":"2024-02-10T08:43:59.469912+0000","last_active":"2024-02-10T08:44:14.101048+0000","last_peered":"2024-02-10T08:44:14.101048+0000","last_clean":"2024-02-10T08:44:14.101048+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:44:14.101048+0000","last_undegraded":"2024-02-10T08:44:14.101048+0000","last_fullsized":"2024-02-10T08:44:14.101048+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.469850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.469850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":435,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326983+0000","last_change":"2024-02-10T08:42:47.456298+0000","last_active":"2024-02-10T08:44:14.326983+0000","last_peered":"2024-02-10T08:44:14.326983+0000","last_clean":"2024-02-10T08:44:14.326983+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:44:14.326983+0000","last_undegraded":"2024-02-10T08:44:14.326983+0000","last_fullsized":"2024-02-10T08:44:14.326983+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.456225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.456225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":503,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.328122+0000","last_change":"2024-02-10T08:43:36.069761+0000","last_active":"2024-02-10T08:44:14.328122+0000","last_peered":"2024-02-10T08:44:14.328122+0000","last_clean":"2024-02-10T08:43:25.738370+0000","last_became_active":"2024-02-10T08:43:36.069761+0000","last_became_peered":"2024-02-10T08:43:36.069761+0000","last_unstale":"2024-02-10T08:44:14.328122+0000","last_undegraded":"2024-02-10T08:43:36.063371+0000","last_fullsized":"2024-02-10T08:43:36.063247+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:43:28.880851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:28.880851+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":498,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.104141+0000","last_change":"2024-02-10T08:43:59.760782+0000","last_active":"2024-02-10T08:44:14.104141+0000","last_peered":"2024-02-10T08:44:14.104141+0000","last_clean":"2024-02-10T08:44:14.104141+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:44:14.104141+0000","last_undegraded":"2024-02-10T08:44:14.104141+0000","last_fullsized":"2024-02-10T08:44:14.104141+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.760721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.760721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":478,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.106147+0000","last_change":"2024-02-10T08:42:31.622734+0000","last_active":"2024-02-10T08:44:14.106147+0000","last_peered":"2024-02-10T08:44:14.106147+0000","last_clean":"2024-02-10T08:44:14.106147+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:44:14.106147+0000","last_undegraded":"2024-02-10T08:44:14.106147+0000","last_fullsized":"2024-02-10T08:44:14.106147+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:31.622670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.622670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":484,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398435+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:44:14.398435+0000","last_peered":"2024-02-10T08:44:14.398435+0000","last_clean":"2024-02-10T08:44:14.398435+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:44:14.398435+0000","last_undegraded":"2024-02-10T08:44:14.398435+0000","last_fullsized":"2024-02-10T08:44:14.398435+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":440,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.405730+0000","last_change":"2024-02-10T08:43:36.081224+0000","last_active":"2024-02-10T08:44:14.405730+0000","last_peered":"2024-02-10T08:44:14.405730+0000","last_clean":"2024-02-10T08:43:31.059324+0000","last_became_active":"2024-02-10T08:43:36.081224+0000","last_became_peered":"2024-02-10T08:43:36.081224+0000","last_unstale":"2024-02-10T08:44:14.405730+0000","last_undegraded":"2024-02-10T08:44:14.405730+0000","last_fullsized":"2024-02-10T08:43:36.068534+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":491,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.401468+0000","last_change":"2024-02-10T08:43:34.096276+0000","last_active":"2024-02-10T08:44:14.401468+0000","last_peered":"2024-02-10T08:44:14.401468+0000","last_clean":"2024-02-10T08:44:14.401468+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:44:14.401468+0000","last_undegraded":"2024-02-10T08:44:14.401468+0000","last_fullsized":"2024-02-10T08:44:14.401468+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:43:34.096208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.096208+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":487,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101921+0000","last_change":"2024-02-10T08:43:23.521301+0000","last_active":"2024-02-10T08:44:14.101921+0000","last_peered":"2024-02-10T08:44:14.101921+0000","last_clean":"2024-02-10T08:44:14.101921+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:44:14.101921+0000","last_undegraded":"2024-02-10T08:44:14.101921+0000","last_fullsized":"2024-02-10T08:44:14.101921+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.521227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.521227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":505,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.327797+0000","last_change":"2024-02-10T08:43:36.072045+0000","last_active":"2024-02-10T08:44:14.327797+0000","last_peered":"2024-02-10T08:44:14.327797+0000","last_clean":"2024-02-10T08:43:31.155301+0000","last_became_active":"2024-02-10T08:43:36.072045+0000","last_became_peered":"2024-02-10T08:43:36.072045+0000","last_unstale":"2024-02-10T08:44:14.327797+0000","last_undegraded":"2024-02-10T08:44:14.327797+0000","last_fullsized":"2024-02-10T08:43:36.062267+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.508636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.508636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.396039+0000","last_change":"2024-02-10T08:43:36.072813+0000","last_active":"2024-02-10T08:44:14.396039+0000","last_peered":"2024-02-10T08:44:14.396039+0000","last_clean":"2024-02-10T08:43:31.060566+0000","last_became_active":"2024-02-10T08:43:36.072813+0000","last_became_peered":"2024-02-10T08:43:36.072813+0000","last_unstale":"2024-02-10T08:44:14.396039+0000","last_undegraded":"2024-02-10T08:44:14.396039+0000","last_fullsized":"2024-02-10T08:43:36.061933+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":443,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.405489+0000","last_change":"2024-02-10T08:43:15.137758+0000","last_active":"2024-02-10T08:44:14.405489+0000","last_peered":"2024-02-10T08:44:14.405489+0000","last_clean":"2024-02-10T08:44:14.405489+0000","last_became_active":"2024-02-10T08:43:15.137437+0000","last_became_peered":"2024-02-10T08:43:15.137437+0000","last_unstale":"2024-02-10T08:44:14.405489+0000","last_undegraded":"2024-02-10T08:44:14.405489+0000","last_fullsized":"2024-02-10T08:44:14.405489+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.116858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":547,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.402908+0000","last_change":"2024-02-10T08:43:16.806279+0000","last_active":"2024-02-10T08:44:14.402908+0000","last_peered":"2024-02-10T08:44:14.402908+0000","last_clean":"2024-02-10T08:44:14.402908+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:44:14.402908+0000","last_undegraded":"2024-02-10T08:44:14.402908+0000","last_fullsized":"2024-02-10T08:44:14.402908+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:03.770710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.770710+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":488,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.327015+0000","last_change":"2024-02-10T08:43:58.346958+0000","last_active":"2024-02-10T08:44:14.327015+0000","last_peered":"2024-02-10T08:44:14.327015+0000","last_clean":"2024-02-10T08:44:14.327015+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:44:14.327015+0000","last_undegraded":"2024-02-10T08:44:14.327015+0000","last_fullsized":"2024-02-10T08:44:14.327015+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.346901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.346901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.394560+0000","last_change":"2024-02-10T08:43:36.074002+0000","last_active":"2024-02-10T08:44:14.394560+0000","last_peered":"2024-02-10T08:44:14.394560+0000","last_clean":"2024-02-10T08:43:31.060062+0000","last_became_active":"2024-02-10T08:43:36.074002+0000","last_became_peered":"2024-02-10T08:43:36.074002+0000","last_unstale":"2024-02-10T08:44:14.394560+0000","last_undegraded":"2024-02-10T08:44:14.394560+0000","last_fullsized":"2024-02-10T08:43:36.060407+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.341979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":484,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398212+0000","last_change":"2024-02-10T08:42:43.111366+0000","last_active":"2024-02-10T08:44:14.398212+0000","last_peered":"2024-02-10T08:44:14.398212+0000","last_clean":"2024-02-10T08:44:14.398212+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:44:14.398212+0000","last_undegraded":"2024-02-10T08:44:14.398212+0000","last_fullsized":"2024-02-10T08:44:14.398212+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.111309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.111309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":401,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326978+0000","last_change":"2024-02-10T08:44:02.384309+0000","last_active":"2024-02-10T08:44:14.326978+0000","last_peered":"2024-02-10T08:44:14.326978+0000","last_clean":"2024-02-10T08:44:14.326978+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:44:14.326978+0000","last_undegraded":"2024-02-10T08:44:14.326978+0000","last_fullsized":"2024-02-10T08:44:14.326978+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:02.384207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:02.384207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":82,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.396444+0000","last_change":"2024-02-10T08:43:16.441297+0000","last_active":"2024-02-10T08:44:14.396444+0000","last_peered":"2024-02-10T08:44:14.396444+0000","last_clean":"2024-02-10T08:44:14.396444+0000","last_became_active":"2024-02-10T08:43:15.139755+0000","last_became_peered":"2024-02-10T08:43:15.139755+0000","last_unstale":"2024-02-10T08:44:14.396444+0000","last_undegraded":"2024-02-10T08:44:14.396444+0000","last_fullsized":"2024-02-10T08:44:14.396444+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:53.573125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.573125+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":516,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.100986+0000","last_change":"2024-02-10T08:43:36.066663+0000","last_active":"2024-02-10T08:44:14.100986+0000","last_peered":"2024-02-10T08:44:14.100986+0000","last_clean":"2024-02-10T08:43:31.060592+0000","last_became_active":"2024-02-10T08:43:36.066663+0000","last_became_peered":"2024-02-10T08:43:36.066663+0000","last_unstale":"2024-02-10T08:44:14.100986+0000","last_undegraded":"2024-02-10T08:44:14.100986+0000","last_fullsized":"2024-02-10T08:43:36.056257+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.131457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.131457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":59,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395505+0000","last_change":"2024-02-10T08:43:15.145074+0000","last_active":"2024-02-10T08:44:14.395505+0000","last_peered":"2024-02-10T08:44:14.395505+0000","last_clean":"2024-02-10T08:44:14.395505+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:44:14.395505+0000","last_undegraded":"2024-02-10T08:44:14.395505+0000","last_fullsized":"2024-02-10T08:44:14.395505+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:42:23.529013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.529013+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":443,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105891+0000","last_change":"2024-02-10T08:43:15.119540+0000","last_active":"2024-02-10T08:44:14.105891+0000","last_peered":"2024-02-10T08:44:14.105891+0000","last_clean":"2024-02-10T08:44:14.105891+0000","last_became_active":"2024-02-10T08:43:15.119127+0000","last_became_peered":"2024-02-10T08:43:15.119127+0000","last_unstale":"2024-02-10T08:44:14.105891+0000","last_undegraded":"2024-02-10T08:44:14.105891+0000","last_fullsized":"2024-02-10T08:44:14.105891+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.214822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.214822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":517,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.329054+0000","last_change":"2024-02-10T08:43:25.251448+0000","last_active":"2024-02-10T08:44:14.329054+0000","last_peered":"2024-02-10T08:44:14.329054+0000","last_clean":"2024-02-10T08:44:14.329054+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:44:14.329054+0000","last_undegraded":"2024-02-10T08:44:14.329054+0000","last_fullsized":"2024-02-10T08:44:14.329054+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:43:25.251382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.251382+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":490,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398361+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:44:14.398361+0000","last_peered":"2024-02-10T08:44:14.398361+0000","last_clean":"2024-02-10T08:44:14.398361+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:44:14.398361+0000","last_undegraded":"2024-02-10T08:44:14.398361+0000","last_fullsized":"2024-02-10T08:44:14.398361+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":482,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.103851+0000","last_change":"2024-02-10T08:43:36.073017+0000","last_active":"2024-02-10T08:44:14.103851+0000","last_peered":"2024-02-10T08:44:14.103851+0000","last_clean":"2024-02-10T08:43:31.058620+0000","last_became_active":"2024-02-10T08:43:36.073017+0000","last_became_peered":"2024-02-10T08:43:36.073017+0000","last_unstale":"2024-02-10T08:44:14.103851+0000","last_undegraded":"2024-02-10T08:44:14.103851+0000","last_fullsized":"2024-02-10T08:43:36.062639+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":431,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105266+0000","last_change":"2024-02-10T08:43:36.085258+0000","last_active":"2024-02-10T08:44:14.105266+0000","last_peered":"2024-02-10T08:44:14.105266+0000","last_clean":"2024-02-10T08:43:31.059986+0000","last_became_active":"2024-02-10T08:43:36.085258+0000","last_became_peered":"2024-02-10T08:43:36.085258+0000","last_unstale":"2024-02-10T08:44:14.105266+0000","last_undegraded":"2024-02-10T08:44:14.105266+0000","last_fullsized":"2024-02-10T08:43:36.058326+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":559,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.104295+0000","last_change":"2024-02-10T08:43:36.088112+0000","last_active":"2024-02-10T08:44:14.104295+0000","last_peered":"2024-02-10T08:44:14.104295+0000","last_clean":"2024-02-10T08:43:31.060512+0000","last_became_active":"2024-02-10T08:43:36.088112+0000","last_became_peered":"2024-02-10T08:43:36.088112+0000","last_unstale":"2024-02-10T08:44:14.104295+0000","last_undegraded":"2024-02-10T08:43:36.063487+0000","last_fullsized":"2024-02-10T08:43:36.063380+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":507,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.327937+0000","last_change":"2024-02-10T08:43:36.078681+0000","last_active":"2024-02-10T08:44:14.327937+0000","last_peered":"2024-02-10T08:44:14.327937+0000","last_clean":"2024-02-10T08:43:31.155537+0000","last_became_active":"2024-02-10T08:43:36.078681+0000","last_became_peered":"2024-02-10T08:43:36.078681+0000","last_unstale":"2024-02-10T08:44:14.327937+0000","last_undegraded":"2024-02-10T08:44:14.327937+0000","last_fullsized":"2024-02-10T08:43:36.069413+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.328692+0000","last_change":"2024-02-10T08:43:36.071762+0000","last_active":"2024-02-10T08:44:14.328692+0000","last_peered":"2024-02-10T08:44:14.328692+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.071762+0000","last_became_peered":"2024-02-10T08:43:36.071762+0000","last_unstale":"2024-02-10T08:44:14.328692+0000","last_undegraded":"2024-02-10T08:44:14.328692+0000","last_fullsized":"2024-02-10T08:43:36.062825+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.031375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.031375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":401,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.326876+0000","last_change":"2024-02-10T08:43:51.341302+0000","last_active":"2024-02-10T08:44:14.326876+0000","last_peered":"2024-02-10T08:44:14.326876+0000","last_clean":"2024-02-10T08:44:14.326876+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:44:14.326876+0000","last_undegraded":"2024-02-10T08:44:14.326876+0000","last_fullsized":"2024-02-10T08:44:14.326876+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:51.341234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:51.341234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":538,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.405799+0000","last_change":"2024-02-10T08:43:36.081271+0000","last_active":"2024-02-10T08:44:14.405799+0000","last_peered":"2024-02-10T08:44:14.405799+0000","last_clean":"2024-02-10T08:43:31.058844+0000","last_became_active":"2024-02-10T08:43:36.081271+0000","last_became_peered":"2024-02-10T08:43:36.081271+0000","last_unstale":"2024-02-10T08:44:14.405799+0000","last_undegraded":"2024-02-10T08:43:36.068229+0000","last_fullsized":"2024-02-10T08:43:36.068095+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":499,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.402934+0000","last_change":"2024-02-10T08:43:15.126069+0000","last_active":"2024-02-10T08:44:14.402934+0000","last_peered":"2024-02-10T08:44:14.402934+0000","last_clean":"2024-02-10T08:44:14.402934+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:44:14.402934+0000","last_undegraded":"2024-02-10T08:44:14.402934+0000","last_fullsized":"2024-02-10T08:44:14.402934+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":475,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.406064+0000","last_change":"2024-02-10T08:42:37.092622+0000","last_active":"2024-02-10T08:44:14.406064+0000","last_peered":"2024-02-10T08:44:14.406064+0000","last_clean":"2024-02-10T08:44:14.406064+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:44:14.406064+0000","last_undegraded":"2024-02-10T08:44:14.406064+0000","last_fullsized":"2024-02-10T08:44:14.406064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.092522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.092522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":439,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.105640+0000","last_change":"2024-02-10T08:44:00.751759+0000","last_active":"2024-02-10T08:44:14.105640+0000","last_peered":"2024-02-10T08:44:14.105640+0000","last_clean":"2024-02-10T08:44:14.105640+0000","last_became_active":"2024-02-10T08:43:28.080931+0000","last_became_peered":"2024-02-10T08:43:28.080931+0000","last_unstale":"2024-02-10T08:44:14.105640+0000","last_undegraded":"2024-02-10T08:44:14.105640+0000","last_fullsized":"2024-02-10T08:44:14.105640+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.751695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":522,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.401752+0000","last_change":"2024-02-10T08:43:36.080654+0000","last_active":"2024-02-10T08:44:14.401752+0000","last_peered":"2024-02-10T08:44:14.401752+0000","last_clean":"2024-02-10T08:43:25.677981+0000","last_became_active":"2024-02-10T08:43:36.080654+0000","last_became_peered":"2024-02-10T08:43:36.080654+0000","last_unstale":"2024-02-10T08:44:14.401752+0000","last_undegraded":"2024-02-10T08:43:36.066356+0000","last_fullsized":"2024-02-10T08:43:36.066242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":60,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395581+0000","last_change":"2024-02-10T08:43:59.379201+0000","last_active":"2024-02-10T08:44:14.395581+0000","last_peered":"2024-02-10T08:44:14.395581+0000","last_clean":"2024-02-10T08:44:14.395581+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:44:14.395581+0000","last_undegraded":"2024-02-10T08:44:14.395581+0000","last_fullsized":"2024-02-10T08:44:14.395581+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.379146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":418,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.103306+0000","last_change":"2024-02-10T08:43:25.651939+0000","last_active":"2024-02-10T08:44:14.103306+0000","last_peered":"2024-02-10T08:44:14.103306+0000","last_clean":"2024-02-10T08:44:14.103306+0000","last_became_active":"2024-02-10T08:43:15.131035+0000","last_became_peered":"2024-02-10T08:43:15.131035+0000","last_unstale":"2024-02-10T08:44:14.103306+0000","last_undegraded":"2024-02-10T08:44:14.103306+0000","last_fullsized":"2024-02-10T08:44:14.103306+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:25.651865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.651865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":440,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.105470+0000","last_change":"2024-02-10T08:43:36.078496+0000","last_active":"2024-02-10T08:44:14.105470+0000","last_peered":"2024-02-10T08:44:14.105470+0000","last_clean":"2024-02-10T08:43:31.066991+0000","last_became_active":"2024-02-10T08:43:36.078496+0000","last_became_peered":"2024-02-10T08:43:36.078496+0000","last_unstale":"2024-02-10T08:44:14.105470+0000","last_undegraded":"2024-02-10T08:44:14.105470+0000","last_fullsized":"2024-02-10T08:43:36.061038+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.399709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.399709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":493,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.402857+0000","last_change":"2024-02-10T08:43:36.081968+0000","last_active":"2024-02-10T08:44:14.402857+0000","last_peered":"2024-02-10T08:44:14.402857+0000","last_clean":"2024-02-10T08:43:25.647108+0000","last_became_active":"2024-02-10T08:43:36.081968+0000","last_became_peered":"2024-02-10T08:43:36.081968+0000","last_unstale":"2024-02-10T08:44:14.402857+0000","last_undegraded":"2024-02-10T08:43:36.066991+0000","last_fullsized":"2024-02-10T08:43:36.066884+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":55,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.395916+0000","last_change":"2024-02-10T08:43:15.134662+0000","last_active":"2024-02-10T08:44:14.395916+0000","last_peered":"2024-02-10T08:44:14.395916+0000","last_clean":"2024-02-10T08:44:14.395916+0000","last_became_active":"2024-02-10T08:43:15.134426+0000","last_became_peered":"2024-02-10T08:43:15.134426+0000","last_unstale":"2024-02-10T08:44:14.395916+0000","last_undegraded":"2024-02-10T08:44:14.395916+0000","last_fullsized":"2024-02-10T08:44:14.395916+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.378214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.378214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":472,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.101131+0000","last_change":"2024-02-10T08:42:35.160383+0000","last_active":"2024-02-10T08:44:14.101131+0000","last_peered":"2024-02-10T08:44:14.101131+0000","last_clean":"2024-02-10T08:44:14.101131+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:44:14.101131+0000","last_undegraded":"2024-02-10T08:44:14.101131+0000","last_fullsized":"2024-02-10T08:44:14.101131+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":400,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.102333+0000","last_change":"2024-02-10T08:41:02.860849+0000","last_active":"2024-02-10T08:44:14.102333+0000","last_peered":"2024-02-10T08:44:14.102333+0000","last_clean":"2024-02-10T08:44:14.102333+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:44:14.102333+0000","last_undegraded":"2024-02-10T08:44:14.102333+0000","last_fullsized":"2024-02-10T08:44:14.102333+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:02.860772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:02.860772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":521,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:14.327831+0000","last_change":"2024-02-10T08:43:36.080498+0000","last_active":"2024-02-10T08:44:14.327831+0000","last_peered":"2024-02-10T08:44:14.327831+0000","last_clean":"2024-02-10T08:43:25.563172+0000","last_became_active":"2024-02-10T08:43:36.080498+0000","last_became_peered":"2024-02-10T08:43:36.080498+0000","last_unstale":"2024-02-10T08:44:14.327831+0000","last_undegraded":"2024-02-10T08:43:36.063844+0000","last_fullsized":"2024-02-10T08:43:36.063724+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:43:22.036213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:22.036213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":490,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398682+0000","last_change":"2024-02-10T08:43:14.144599+0000","last_active":"2024-02-10T08:44:14.398682+0000","last_peered":"2024-02-10T08:44:14.398682+0000","last_clean":"2024-02-10T08:44:14.398682+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:44:14.398682+0000","last_undegraded":"2024-02-10T08:44:14.398682+0000","last_fullsized":"2024-02-10T08:44:14.398682+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.144537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.144537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":37,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.101914+0000","last_change":"2024-02-10T08:43:36.070306+0000","last_active":"2024-02-10T08:44:14.101914+0000","last_peered":"2024-02-10T08:44:14.101914+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.070306+0000","last_became_peered":"2024-02-10T08:43:36.070306+0000","last_unstale":"2024-02-10T08:44:14.101914+0000","last_undegraded":"2024-02-10T08:44:14.101914+0000","last_fullsized":"2024-02-10T08:43:36.056977+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":436,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.101216+0000","last_change":"2024-02-10T08:43:36.069163+0000","last_active":"2024-02-10T08:44:14.101216+0000","last_peered":"2024-02-10T08:44:14.101216+0000","last_clean":"2024-02-10T08:43:31.063295+0000","last_became_active":"2024-02-10T08:43:36.069163+0000","last_became_peered":"2024-02-10T08:43:36.069163+0000","last_unstale":"2024-02-10T08:44:14.101216+0000","last_undegraded":"2024-02-10T08:44:14.101216+0000","last_fullsized":"2024-02-10T08:43:36.056950+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":542,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.398132+0000","last_change":"2024-02-10T08:41:19.653285+0000","last_active":"2024-02-10T08:44:14.398132+0000","last_peered":"2024-02-10T08:44:14.398132+0000","last_clean":"2024-02-10T08:44:14.398132+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:44:14.398132+0000","last_undegraded":"2024-02-10T08:44:14.398132+0000","last_fullsized":"2024-02-10T08:44:14.398132+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:41:19.653219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.653219+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":61,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.396852+0000","last_change":"2024-02-10T08:43:36.074560+0000","last_active":"2024-02-10T08:44:14.396852+0000","last_peered":"2024-02-10T08:44:14.396852+0000","last_clean":"2024-02-10T08:43:31.060871+0000","last_became_active":"2024-02-10T08:43:36.074560+0000","last_became_peered":"2024-02-10T08:43:36.074560+0000","last_unstale":"2024-02-10T08:44:14.396852+0000","last_undegraded":"2024-02-10T08:44:14.396852+0000","last_fullsized":"2024-02-10T08:43:36.062458+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:48.325433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:48.325433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":493,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-10T08:44:14.402711+0000","last_change":"2024-02-10T08:43:55.185139+0000","last_active":"2024-02-10T08:44:14.402711+0000","last_peered":"2024-02-10T08:44:14.402711+0000","last_clean":"2024-02-10T08:44:14.402711+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:44:14.402711+0000","last_undegraded":"2024-02-10T08:44:14.402711+0000","last_fullsized":"2024-02-10T08:44:14.402711+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:55.185039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:55.185039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":441,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-02-10T08:44:14.328487+0000","last_change":"2024-02-10T08:43:36.085682+0000","last_active":"2024-02-10T08:44:14.328487+0000","last_peered":"2024-02-10T08:44:14.328487+0000","last_clean":"2024-02-10T08:43:31.156976+0000","last_became_active":"2024-02-10T08:43:36.085682+0000","last_became_peered":"2024-02-10T08:43:36.085682+0000","last_unstale":"2024-02-10T08:44:14.328487+0000","last_undegraded":"2024-02-10T08:44:14.328487+0000","last_fullsized":"2024-02-10T08:43:36.064612+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":222,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":6},{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4011015,"num_objects":180,"num_object_clones":9,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":190,"num_read_kb":50,"num_write":742,"num_write_kb":4592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":1217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14147584,"data_stored":12024256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2026,"ondisk_log_size":2026,"up":79,"acting":79,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":55,"acting":55,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":5506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":720,"ondisk_log_size":720,"up":81,"acting":81,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":650,"num_read_kb":645,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":85,"acting":85,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299526,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300880,"kb_used_data":3656,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444272,"statfs":{"total":95995035648,"available":95686934528,"internally_reserved":0,"allocated":3743744,"data_stored":1691161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.499,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.63500000000000001,"5min":0.99299999999999999,"15min":22.143000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48899999999999999,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.83899999999999997,"5min":0.93100000000000005,"15min":8.7850000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52800000000000002,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":0.79300000000000004,"5min":3.2389999999999999,"15min":7.593},"last":0.5},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51300000000000001,"15min":0.57699999999999996},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.19900000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":7.7759999999999998},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.49199999999999999,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":2.5009999999999999,"15min":6.0300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.64700000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":0.67800000000000005,"5min":1.0740000000000001,"15min":19.369},"last":0.54200000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51800000000000002,"15min":0.56999999999999995},"min":{"1min":0.32800000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63400000000000001,"5min":2.7029999999999998,"15min":6.9829999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.53200000000000003,"15min":0.63800000000000001},"min":{"1min":0.34100000000000003,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.70699999999999996,"5min":1.1459999999999999,"15min":22.102},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":8.827},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":1.903,"15min":13.646000000000001},"last":0.35199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48099999999999998,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":19.326000000000001},"last":0.38},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.6220000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.56799999999999995},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.86899999999999999,"5min":0.89100000000000001,"15min":22.687000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.30399999999999999,"15min":0.29699999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.7450000000000001},"last":0.55300000000000005}]}]},{"osd":6,"up_from":44,"seq":188978561164,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312856,"kb_used_data":7440,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432296,"statfs":{"total":95995035648,"available":95674671104,"internally_reserved":0,"allocated":7618560,"data_stored":5687342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.51100000000000001,"15min":0.53400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.19800000000000001},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.54000000000000004},"min":{"1min":0.28100000000000003,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.86399999999999999,"5min":1.694,"15min":8.4629999999999992},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":0.63,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":18.358000000000001,"5min":18.358000000000001,"15min":18.358000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.63300000000000001,"15min":0.55400000000000005},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":18.402999999999999,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.65100000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":4.798},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":4.8070000000000004},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.59899999999999998},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.878,"5min":0.878,"15min":16.823},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":3.7389999999999999},"last":0.73899999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45300000000000001,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":3.657,"15min":3.657},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.46500000000000002,"15min":0.40500000000000003},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.24399999999999999},"max":{"1min":0.61799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.44,"15min":0.42299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":3.7429999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46100000000000002,"15min":0.44400000000000001},"min":{"1min":0.26900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61099999999999999,"5min":0.68500000000000005,"15min":2.2029999999999998},"last":0.624}]},{"osd":7,"last update":"Sat Feb 10 08:43:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.28499999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":3.7149999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53200000000000003,"15min":0.505},"min":{"1min":0.40400000000000003,"5min":0.36599999999999999,"15min":0.29999999999999999},"max":{"1min":0.72799999999999998,"5min":0.72999999999999998,"15min":0.76200000000000001},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309060,"kb_used_data":3644,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436092,"statfs":{"total":95995035648,"available":95678558208,"internally_reserved":0,"allocated":3731456,"data_stored":1698761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.40799999999999997},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.58199999999999996,"5min":0.66200000000000003,"15min":7.8780000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.45700000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":16.166},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.60699999999999998},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":34.985999999999997},"last":0.375},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.34000000000000002,"15min":0.52600000000000002},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.913,"5min":1.913,"15min":21.614999999999998},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:43:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.502},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55200000000000005,"5min":0.627,"15min":24.773},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.371,"15min":0.51100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":16.120999999999999},"last":0.436}]},{"osd":4,"last update":"Sat Feb 10 08:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.60999999999999999},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.89000000000000001,"15min":16.074000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.60399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":9.5090000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55300000000000005,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":2.0270000000000001,"15min":9.5600000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.65300000000000002},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.31900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":7.8479999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Sat Feb 10 08:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.312},"max":{"1min":0.92700000000000005,"5min":1.996,"15min":9.532},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.64900000000000002},"min":{"1min":0.42199999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":7.9000000000000004},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.60599999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.34499999999999997},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":9.4640000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.66400000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71199999999999997,"5min":0.81000000000000005,"15min":7.9199999999999999},"last":0.59899999999999998}]}]},{"osd":0,"up_from":0,"seq":34359738537,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411487,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304948,"kb_used_data":3628,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440204,"statfs":{"total":95995035648,"available":95682768896,"internally_reserved":0,"allocated":3715072,"data_stored":1697891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34200000000000003,"15min":0.40300000000000002},"min":{"1min":0.20399999999999999,"5min":0.20200000000000001,"15min":0.187},"max":{"1min":0.54700000000000004,"5min":3.0640000000000001,"15min":4.3310000000000004},"last":0.311},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32900000000000001,"15min":0.45200000000000001},"min":{"1min":0.224,"5min":0.187,"15min":0.187},"max":{"1min":0.52800000000000002,"5min":1.4950000000000001,"15min":9.1440000000000001},"last":0.29299999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.32500000000000001,"15min":0.46700000000000003},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.495,"5min":0.63400000000000001,"15min":12.512},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.36599999999999999,"15min":0.45900000000000002},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":0.50800000000000001,"5min":3.0960000000000001,"15min":9.6769999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.497,"15min":0.47199999999999998},"min":{"1min":0.245,"5min":0.217,"15min":0.217},"max":{"1min":11.612,"5min":11.612,"15min":11.612},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.22600000000000001},"max":{"1min":11.710000000000001,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.33900000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.67200000000000004},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76200000000000001,"5min":1.7370000000000001,"15min":10.920999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":1.014,"15min":12.292999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.65600000000000003},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":1.764,"15min":9.2029999999999994},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78900000000000003,"15min":7.6799999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.66000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75700000000000001,"5min":1.6759999999999999,"15min":9.266},"last":0.377},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.64600000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":5.4630000000000001},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57099999999999995,"15min":0.67800000000000005},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":1.708,"15min":9.2940000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.38400000000000001,"15min":0.38200000000000001},"max":{"1min":0.75600000000000001,"5min":0.83599999999999997,"15min":7.4219999999999997},"last":0.36599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312936,"kb_used_data":7520,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432216,"statfs":{"total":95995035648,"available":95674589184,"internally_reserved":0,"allocated":7700480,"data_stored":5694534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.313,"15min":0.501},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.56100000000000005,"5min":0.878,"15min":14.744},"last":0.314},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.36499999999999999,"15min":0.54100000000000004},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.188},"max":{"1min":0.39600000000000002,"5min":3.5579999999999998,"15min":11.808},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.42799999999999999,"15min":0.55400000000000005},"min":{"1min":0.24299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46700000000000003,"5min":9.3100000000000005,"15min":16.634},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.55800000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":2.472,"5min":5.4370000000000003,"15min":12.438000000000001},"last":0.374}]},{"osd":2,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41899999999999998,"15min":0.58199999999999996},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":2.4340000000000002,"5min":3.1549999999999998,"15min":16.536000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.625},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.5230000000000001,"5min":2.5230000000000001,"15min":23.294},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54500000000000004,"15min":0.66900000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":2.589,"15min":12.371},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68700000000000006,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57099999999999995,"15min":0.70599999999999996},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.96699999999999997,"5min":3.6040000000000001,"15min":12.579000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59799999999999998,"15min":0.65200000000000002},"min":{"1min":0.34699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95899999999999996,"5min":4.9909999999999997,"15min":11.448},"last":0.436}]},{"osd":6,"last update":"Sat Feb 10 08:44:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57499999999999996,"15min":0.69799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":1.0269999999999999,"5min":3.48,"15min":16.091000000000001},"last":0.496},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59699999999999998,"15min":0.68999999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.33800000000000002},"max":{"1min":2.383,"5min":3.2469999999999999,"15min":16.475000000000001},"last":0.46800000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:43:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55000000000000004,"15min":0.65800000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88500000000000001,"5min":1.024,"15min":14.773999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.62,"15min":0.73899999999999999},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0069999999999999,"5min":3.52,"15min":12.606999999999999},"last":0.56999999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986316,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312804,"kb_used_data":3292,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432348,"statfs":{"total":95995035648,"available":95674724352,"internally_reserved":0,"allocated":3371008,"data_stored":1427290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.627},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":19.940999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.55300000000000005},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.91800000000000004,"15min":9.1760000000000002},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46899999999999997,"15min":0.57899999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.59699999999999998,"5min":0.93799999999999994,"15min":5.774},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.59799999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":8.6989999999999998},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.59799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.83999999999999997,"15min":6.2329999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.59799999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.64600000000000002,"5min":0.77100000000000002,"15min":10.01},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48399999999999999,"15min":0.65000000000000002},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.27700000000000002},"max":{"1min":0.58299999999999996,"5min":0.92700000000000005,"15min":9.5909999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51900000000000002,"15min":0.71799999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":25.27},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.621,"5min":0.876,"15min":6.8730000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.223},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":3.8290000000000002},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.247},"max":{"1min":0.626,"5min":0.78000000000000003,"15min":13.342000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":1.085,"15min":13.753},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.52000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.90400000000000003,"15min":12.835000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.72599999999999998,"5min":1.0509999999999999,"15min":12.875999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790097,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316988,"kb_used_data":7476,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428164,"statfs":{"total":95995035648,"available":95670439936,"internally_reserved":0,"allocated":7655424,"data_stored":5689381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.98799999999999999},"min":{"1min":0.24099999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.63300000000000001,"5min":0.78200000000000003,"15min":107.551},"last":0.377},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.501,"15min":0.96099999999999997},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":0.69499999999999995,"5min":3.875,"15min":88.792000000000002},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54800000000000004,"15min":0.93899999999999995},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70999999999999996,"5min":4.4349999999999996,"15min":107.205},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.82999999999999996},"min":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.246},"max":{"1min":0.64000000000000001,"5min":4.4580000000000002,"15min":72.272000000000006},"last":0.435}]},{"osd":2,"last update":"Sat Feb 10 08:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47599999999999998,"15min":0.94699999999999995},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":54.174999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.504,"15min":0.99099999999999999},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.65200000000000002,"5min":1.2230000000000001,"15min":86.183000000000007},"last":0.40699999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54600000000000004,"15min":0.72799999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30399999999999999},"max":{"1min":2.0379999999999998,"5min":2.2669999999999999,"15min":44.395000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.53000000000000003,"15min":1.204},"min":{"1min":0.35699999999999998,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":0.72399999999999998,"5min":1.9390000000000001,"15min":114.526},"last":0.42999999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Feb 10 08:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.69199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.492,"5min":1.492,"15min":72.241},"last":0.317},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.47099999999999997,"15min":0.54600000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54200000000000004,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.34799999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.71999999999999997},"min":{"1min":0.32800000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81200000000000006,"5min":0.86199999999999999,"15min":66.844999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.753},"min":{"1min":0.33300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":72.257000000000005},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":2617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4008272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4255744,"data_stored":4003346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":4001158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":4875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":222,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:44:20.142 INFO:tasks.thrashosds.thrasher:Setting 2.13 to [4, 2, 7] 2024-02-10T08:44:20.143 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.13', '4', '2', '7'] 2024-02-10T08:44:20.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 2.13 4 2 7 2024-02-10T08:44:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:20.144 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:20.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:20.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:20.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:19.164151+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2024-02-10T08:44:20.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:19.296875+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:19.300002+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:19.300582+0000 mon.b (mon.1) 648 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:19.300819+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:19.481445+0000 mgr.y (mgr.14553) 460 : cluster [DBG] pgmap v772: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:19.164151+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:19.296875+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:19.300002+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:19.300582+0000 mon.b (mon.1) 648 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:19.300819+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:19.481445+0000 mgr.y (mgr.14553) 460 : cluster [DBG] pgmap v772: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:20.294 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:19.164151+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2024-02-10T08:44:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:19.296875+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:19.300002+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:19.300582+0000 mon.b (mon.1) 648 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:19.300819+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-81"}]: dispatch 2024-02-10T08:44:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:19.481445+0000 mgr.y (mgr.14553) 460 : cluster [DBG] pgmap v772: 218 pgs: 38 active+undersized+degraded, 54 active+undersized, 126 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 180/1119 objects degraded (16.086%) 2024-02-10T08:44:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.978+0000 7f292f36e700 1 -- 172.21.15.107:0/1771501141 >> v1:172.21.15.107:6790/0 conn(0x7f29281007f0 legacy=0x7f2928102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- 172.21.15.107:0/1771501141 shutdown_connections 2024-02-10T08:44:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- 172.21.15.107:0/1771501141 >> 172.21.15.107:0/1771501141 conn(0x7f29280f3640 msgr2=0x7f29280f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- 172.21.15.107:0/1771501141 shutdown_connections 2024-02-10T08:44:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- 172.21.15.107:0/1771501141 wait complete. 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 librados: starting msgr at 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 librados: starting objecter 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 Processor -- start 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- start start 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 librados: setting wanted keys 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 librados: calling monclient init 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29280fd000 con 0x7f29280fda50 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29280fd190 con 0x7f2928104330 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292f36e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29280fd360 con 0x7f29281007f0 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292c909700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f29281007f0 0x7f2928114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46680/0 (socket says 172.21.15.107:46680) 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f292c909700 1 -- 172.21.15.107:0/220150238 learned_addr learned my addr 172.21.15.107:0/220150238 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.979+0000 7f292d10a700 1 --1- 172.21.15.107:0/220150238 >> v1:172.21.15.107:6789/0 conn(0x7f29280fda50 0x7f29280fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45054/0 (socket says 172.21.15.107:45054) 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 10 client.?.objecter ms_handle_connect 0x7f29281007f0 2024-02-10T08:44:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 10 client.?.objecter ms_handle_connect 0x7f29280fda50 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367364536 0 0) 0x7f29280fd360 con 0x7f29281007f0 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914003630 con 0x7f29281007f0 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728310007 0 0) 0x7f29280fd000 con 0x7f29280fda50 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280fd360 con 0x7f29280fda50 2024-02-10T08:44:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653813763 0 0) 0x7f2914003630 con 0x7f29281007f0 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29280fd000 con 0x7f29281007f0 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2918002ee0 con 0x7f29281007f0 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567825507 0 0) 0x7f29280fd360 con 0x7f29280fda50 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2914003630 con 0x7f29280fda50 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f29240016b0 con 0x7f29280fda50 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141030586 0 0) 0x7f29280fd000 con 0x7f29281007f0 2024-02-10T08:44:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 >> v1:172.21.15.195:6789/0 conn(0x7f2928104330 legacy=0x7f29281184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 >> v1:172.21.15.107:6789/0 conn(0x7f29280fda50 legacy=0x7f29280fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292811cc00 con 0x7f29281007f0 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928119c30 con 0x7f29281007f0 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f292f36e700 10 client.53858.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f292811a120 con 0x7f29281007f0 2024-02-10T08:44:20.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f292f36e700 1 librados: init done 2024-02-10T08:44:20.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2918002560 con 0x7f29281007f0 2024-02-10T08:44:20.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.980+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2918003b10 con 0x7f29281007f0 2024-02-10T08:44:20.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.981+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290c004f20 con 0x7f29281007f0 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f29180032a0 con 0x7f29281007f0 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(514..514 src has 330..514) v4 ==== 7721+0+0 (unknown 2012752355 0 0) 0x7f2918078f40 con 0x7f29281007f0 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 10 client.53858.objecter ms_dispatch 0x7f2928001550 osd_map(514..514 src has 330..514) v4 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 3 client.53858.objecter handle_osd_map got epochs [514,514] > 0 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 3 client.53858.objecter handle_osd_map decoding full epoch 514 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 20 client.53858.objecter dump_active .. 0 homeless 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.982+0000 7f291e7fc700 10 client.53858.objecter ms_handle_connect 0x7f2914063c30 2024-02-10T08:44:20.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:20.986+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f291804c4e0 con 0x7f29281007f0 2024-02-10T08:44:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:21.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:21.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:21.216+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f290c0051d0 con 0x7f29281007f0 2024-02-10T08:44:21.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:21.294 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:21.313 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:21.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:21.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:20.180195+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2024-02-10T08:44:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:20.194970+0000 osd.0 (osd.0) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:20.195033+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:21.077030+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e515: 8 total, 7 up, 7 in 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:21.088889+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:21.125754+0000 mon.a (mon.0) 2739 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:20.180195+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:20.194970+0000 osd.0 (osd.0) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:20.195033+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:21.077030+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e515: 8 total, 7 up, 7 in 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:21.088889+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:21.125754+0000 mon.a (mon.0) 2739 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:21.497 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:21.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:20.180195+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:20.194970+0000 osd.0 (osd.0) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:20.195033+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:21.077030+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e515: 8 total, 7 up, 7 in 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:21.088889+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:21.125754+0000 mon.a (mon.0) 2739 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:21.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:21.627 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:22.009 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:22.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:22.052 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:22.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.084+0000 7f291e7fc700 1 -- 172.21.15.107:0/220150238 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v516) v1 ==== 91+0+0 (unknown 121790356 0 0) 0x7f291804a950 con 0x7f29281007f0 2024-02-10T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.092+0000 7f292f36e700 10 librados: watch_flush enter 2024-02-10T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.093+0000 7f292f36e700 10 librados: watch_flush exit 2024-02-10T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.093+0000 7f292f36e700 20 client.53858.objecter shutdown clearing up homeless session... 2024-02-10T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.093+0000 7f292f36e700 10 client.53858.objecter successfully canceled tick 2024-02-10T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.093+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2914063c30 legacy=0x7f29140660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.093+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 >> v1:172.21.15.107:6790/0 conn(0x7f29281007f0 legacy=0x7f2928114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.101+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 shutdown_connections 2024-02-10T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.101+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 >> 172.21.15.107:0/220150238 conn(0x7f29280f3640 msgr2=0x7f29281077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.113+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 shutdown_connections 2024-02-10T08:44:22.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.118+0000 7f292f36e700 1 -- 172.21.15.107:0/220150238 wait complete. 2024-02-10T08:44:22.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:22.125+0000 7f292f36e700 1 librados: shutdown 2024-02-10T08:44:22.133 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:44:22.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:20.626464+0000 osd.1 (osd.1) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:22.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:20.626617+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:21.218400+0000 mon.c (mon.2) 413 : audit [INF] from='client.? v1:172.21.15.107:0/220150238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:21.224677+0000 mon.a (mon.0) 2740 : audit [INF] from='client.53858 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:21.482367+0000 mgr.y (mgr.14553) 461 : cluster [DBG] pgmap v775: 218 pgs: 8 creating+peering, 24 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:21.581106+0000 mgr.y (mgr.14553) 462 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:22.072886+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:22.079027+0000 mon.a (mon.0) 2742 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:22.079105+0000 mon.a (mon.0) 2743 : audit [INF] from='client.53858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:22.079156+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e516: 8 total, 7 up, 7 in 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:22.087759+0000 mon.b (mon.1) 650 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:20.626464+0000 osd.1 (osd.1) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:22.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:20.626617+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:21.218400+0000 mon.c (mon.2) 413 : audit [INF] from='client.? v1:172.21.15.107:0/220150238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:21.224677+0000 mon.a (mon.0) 2740 : audit [INF] from='client.53858 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:21.482367+0000 mgr.y (mgr.14553) 461 : cluster [DBG] pgmap v775: 218 pgs: 8 creating+peering, 24 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:21.581106+0000 mgr.y (mgr.14553) 462 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:22.072886+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:22.079027+0000 mon.a (mon.0) 2742 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:22.079105+0000 mon.a (mon.0) 2743 : audit [INF] from='client.53858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:22.079156+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e516: 8 total, 7 up, 7 in 2024-02-10T08:44:22.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:22.087759+0000 mon.b (mon.1) 650 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:20.626464+0000 osd.1 (osd.1) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:20.626617+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:21.218400+0000 mon.c (mon.2) 413 : audit [INF] from='client.? v1:172.21.15.107:0/220150238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:21.224677+0000 mon.a (mon.0) 2740 : audit [INF] from='client.53858 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:21.482367+0000 mgr.y (mgr.14553) 461 : cluster [DBG] pgmap v775: 218 pgs: 8 creating+peering, 24 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:21.581106+0000 mgr.y (mgr.14553) 462 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:22.072886+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:22.079027+0000 mon.a (mon.0) 2742 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:22.079105+0000 mon.a (mon.0) 2743 : audit [INF] from='client.53858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:44:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:22.079156+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e516: 8 total, 7 up, 7 in 2024-02-10T08:44:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:22.087759+0000 mon.b (mon.1) 650 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:22.386851+0000 mon.a (mon.0) 2745 : cluster [WRN] Health check update: Degraded data redundancy: 185/1143 objects degraded (16.185%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.078705+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2024-02-10T08:44:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:23.085527+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:23.086348+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:22.386851+0000 mon.a (mon.0) 2745 : cluster [WRN] Health check update: Degraded data redundancy: 185/1143 objects degraded (16.185%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.078705+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2024-02-10T08:44:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:23.085527+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:23.086348+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:22.386851+0000 mon.a (mon.0) 2745 : cluster [WRN] Health check update: Degraded data redundancy: 185/1143 objects degraded (16.185%), 38 pgs degraded (PG_DEGRADED) 2024-02-10T08:44:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.078705+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2024-02-10T08:44:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:23.085527+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:23.086348+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:24.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.459751+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:44:24.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.462667+0000 osd.6 (osd.6) 227 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.482895+0000 mgr.y (mgr.14553) 463 : cluster [DBG] pgmap v778: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:24.081041+0000 mon.a (mon.0) 2748 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:24.081167+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:24.100199+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:24.102476+0000 mon.a (mon.0) 2750 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:24.230110+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? v1:172.21.15.107:0/4195600178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.459751+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.462667+0000 osd.6 (osd.6) 227 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.482895+0000 mgr.y (mgr.14553) 463 : cluster [DBG] pgmap v778: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:24.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:24.081041+0000 mon.a (mon.0) 2748 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:24.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:24.081167+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2024-02-10T08:44:24.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:24.100199+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:24.102476+0000 mon.a (mon.0) 2750 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:24.230110+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? v1:172.21.15.107:0/4195600178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.223+0000 7fa37cf19700 1 -- 172.21.15.107:0/69717359 >> v1:172.21.15.107:6790/0 conn(0x7fa37806cfc0 legacy=0x7fa37806f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 -- 172.21.15.107:0/69717359 shutdown_connections 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 -- 172.21.15.107:0/69717359 >> 172.21.15.107:0/69717359 conn(0x7fa378063bc0 msgr2=0x7fa3780672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 -- 172.21.15.107:0/69717359 shutdown_connections 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 -- 172.21.15.107:0/69717359 wait complete. 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 librados: starting msgr at 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.224+0000 7fa37cf19700 1 librados: starting objecter 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 Processor -- start 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 -- start start 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 librados: setting wanted keys 2024-02-10T08:44:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 librados: calling monclient init 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa37807bbe0 con 0x7fa37806cfc0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa37807bd70 con 0x7fa378069540 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa37cf19700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa378139770 con 0x7fa378070ba0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa375d9b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa37806cfc0 0x7fa378103e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45066/0 (socket says 172.21.15.107:45066) 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa376d9d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa378070ba0 0x7fa378104540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46694/0 (socket says 172.21.15.107:46694) 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa375d9b700 1 -- 172.21.15.107:0/4195600178 learned_addr learned my addr 172.21.15.107:0/4195600178 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 10 client.?.objecter ms_handle_connect 0x7fa378070ba0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 10 client.?.objecter ms_handle_connect 0x7fa37806cfc0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900109121 0 0) 0x7fa37807bbe0 con 0x7fa37806cfc0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.225+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3640038a0 con 0x7fa37806cfc0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925319941 0 0) 0x7fa378139770 con 0x7fa378070ba0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37807bbe0 con 0x7fa378070ba0 2024-02-10T08:44:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292839660 0 0) 0x7fa37807bbe0 con 0x7fa378070ba0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa378139770 con 0x7fa378070ba0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa370001bc0 con 0x7fa378070ba0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 10 client.?.objecter ms_handle_connect 0x7fa378069540 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283684032 0 0) 0x7fa3640038a0 con 0x7fa37806cfc0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa37807bbe0 con 0x7fa37806cfc0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa368002d60 con 0x7fa37806cfc0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2434752310 0 0) 0x7fa378139770 con 0x7fa378070ba0 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 >> v1:172.21.15.195:6789/0 conn(0x7fa378069540 legacy=0x7fa37807b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 >> v1:172.21.15.107:6789/0 conn(0x7fa37806cfc0 legacy=0x7fa378103e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:24.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37813a970 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa378139960 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa37cf19700 10 client.53864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa378139dd0 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.226+0000 7fa37cf19700 1 librados: init done 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.228+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa37807ad90 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.228+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa370002720 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.228+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa370005d80 con 0x7fa378070ba0 2024-02-10T08:44:24.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa370002280 con 0x7fa378070ba0 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(518..518 src has 330..518) v4 ==== 8507+0+0 (unknown 1315754766 0 0) 0x7fa370079140 con 0x7fa378070ba0 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 osd_map(518..518 src has 330..518) v4 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 3 client.53864.objecter handle_osd_map got epochs [518,518] > 0 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 3 client.53864.objecter handle_osd_map decoding full epoch 518 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 20 client.53864.objecter dump_active .. 0 homeless 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v518) v1 ==== 54+0+16 (unknown 3496443661 0 2420000192) 0x7fa370079500 con 0x7fa378070ba0 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa37cf19700 10 client.53864.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 1 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.230+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 1 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 0 1 2024-02-10T08:44:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 4 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 10 client.53864.objecter _send_command 1 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa37806aa70 con 0x7fa378028590 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.231+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa364063ce0 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.233+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa378028590 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.234+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa37806aa70 con 0x7fa378028590 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.234+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.234+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 1 = system:0 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.234+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 4 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.234+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 0 1 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 10 client.53864.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 2 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 2 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37805ffc0 osd=0 3 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 0 2 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 4 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 10 client.53864.objecter _send_command 2 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.249+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3780475e0 con 0x7fa378028590 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa3780475e0 con 0x7fa378028590 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37805ffc0 osd=0 4 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 0 2 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 10 client.53864.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 3 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 4 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 10 client.53864.objecter _send_command 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.251+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa378076780 con 0x7fa37813abc0 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.252+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa37813abc0 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.253+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa378076780 con 0x7fa37813abc0 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.253+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.253+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 3 = system:0 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.253+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 4 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.253+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 1 3 2024-02-10T08:44:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 10 client.53864.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 4 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 4 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37806fc60 osd=1 3 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 1 4 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 4 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 10 client.53864.objecter _send_command 4 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.269+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa37806f6d0 con 0x7fa37813abc0 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.271+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa37806f6d0 con 0x7fa37813abc0 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.271+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.271+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.271+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37806fc60 osd=1 4 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.271+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 1 4 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 10 client.53864.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 5 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 5 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 2 5 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 4 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 10 client.53864.objecter _send_command 5 2024-02-10T08:44:24.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3781430b0 con 0x7fa3780280f0 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.272+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa3780280f0 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.273+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa3781430b0 con 0x7fa3780280f0 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.273+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.273+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 5 = system:0 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.273+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 4 2024-02-10T08:44:24.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.273+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 2 5 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 10 client.53864.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 6 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 6 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37806fa80 osd=2 3 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 2 6 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 4 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 10 client.53864.objecter _send_command 6 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.287+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa378027650 con 0x7fa3780280f0 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa378027650 con 0x7fa3780280f0 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37806fa80 osd=2 4 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 2 6 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 10 client.53864.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 7 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 7 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 3 7 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 4 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 10 client.53864.objecter _send_command 7 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3781439f0 con 0x7fa37813f3b0 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa37813f3b0 2024-02-10T08:44:24.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa3781439f0 con 0x7fa37813f3b0 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 7 = system:0 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 4 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.289+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 3 7 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 10 client.53864.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 8 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 8 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa378027a00 osd=3 3 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 3 8 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 4 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 10 client.53864.objecter _send_command 8 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.304+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3781443b0 con 0x7fa37813f3b0 2024-02-10T08:44:24.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.305+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa3781443b0 con 0x7fa37813f3b0 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.305+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.305+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.305+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa378027a00 osd=3 4 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.305+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 3 8 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 10 client.53864.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 9 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 10 client.53864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=519}) v3 -- 0x7fa378143be0 con 0x7fa378070ba0 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa378143f20 con 0x7fa378070ba0 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=518) v2 ==== 24+0+0 (unknown 4041400324 0 0) 0x7fa37004c3d0 con 0x7fa378070ba0 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa374d99700 10 client.53864.objecter _check_command_map_dne tid 9 current 518 map_dne_bound 518 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa374d99700 10 client.53864.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa374d99700 15 client.53864.objecter _session_command_op_remove -1 9 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 10 client.53864.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 10 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 10 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 5 10 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 4 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 10 client.53864.objecter _send_command 10 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa37814cbb0 con 0x7fa3781490a0 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.306+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa3781490a0 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.307+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa37814cbb0 con 0x7fa3781490a0 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.307+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.307+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 10 = system:0 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.307+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 4 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.307+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 5 10 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 10 client.53864.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 11 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 11 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa378148a20 osd=5 3 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 5 11 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 4 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 10 client.53864.objecter _send_command 11 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.322+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa37814cbb0 con 0x7fa3781490a0 2024-02-10T08:44:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.323+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa37814cbb0 con 0x7fa3781490a0 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.323+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.323+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.323+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa378148a20 osd=5 4 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.323+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 5 11 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 10 client.53864.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 12 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 12 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 6 12 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 4 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 10 client.53864.objecter _send_command 12 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa37814ef60 con 0x7fa37814d290 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.324+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa37814d290 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.325+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa37814ef60 con 0x7fa37814d290 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.325+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.325+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 12 = system:0 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.325+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 4 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.325+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 6 12 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 10 client.53864.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 13 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 13 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37814cbb0 osd=6 3 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 6 13 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 4 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 10 client.53864.objecter _send_command 13 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.340+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa37814ef60 con 0x7fa37814d290 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa37814ef60 con 0x7fa37814d290 2024-02-10T08:44:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37814cbb0 osd=6 4 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 6 13 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 10 client.53864.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 14 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 14 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 7 14 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 4 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 10 client.53864.objecter _send_command 14 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.342+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa378155500 con 0x7fa3781519f0 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.343+0000 7fa36f7fe700 10 client.53864.objecter ms_handle_connect 0x7fa3781519f0 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.344+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa378155500 con 0x7fa3781519f0 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.344+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.344+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 14 = system:0 2024-02-10T08:44:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.344+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 4 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.344+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 7 14 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 10 client.53864.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign -1 15 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 20 client.53864.objecter _get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_remove -1 15 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 20 client.53864.objecter get_session s=0x7fa37814ef60 osd=7 3 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 15 client.53864.objecter _session_command_op_assign 7 15 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 4 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 10 client.53864.objecter _send_command 15 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.358+0000 7fa37cf19700 1 -- 172.21.15.107:0/4195600178 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa378155500 con 0x7fa3781519f0 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.360+0000 7fa36f7fe700 1 -- 172.21.15.107:0/4195600178 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa378155500 con 0x7fa3781519f0 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.360+0000 7fa36f7fe700 10 client.53864.objecter ms_dispatch 0x7fa378001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.360+0000 7fa36f7fe700 10 client.53864.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.360+0000 7fa36f7fe700 20 client.53864.objecter put_session s=0x7fa37814ef60 osd=7 4 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:24.360+0000 7fa36f7fe700 15 client.53864.objecter _session_command_op_remove 7 15 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.459751+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.462667+0000 osd.6 (osd.6) 227 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.482895+0000 mgr.y (mgr.14553) 463 : cluster [DBG] pgmap v778: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:24.081041+0000 mon.a (mon.0) 2748 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:24.081167+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:24.100199+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:24.102476+0000 mon.a (mon.0) 2750 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]: dispatch 2024-02-10T08:44:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:24.230110+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? v1:172.21.15.107:0/4195600178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.766612+0000 osd.1 (osd.1) 248 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:23.770080+0000 osd.1 (osd.1) 249 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:24.394536+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:24.395740+0000 osd.2 (osd.2) 198 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:24.597031+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:24.600072+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:25.080521+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]': finished 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.080582+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:25.082154+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:25.083957+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.766612+0000 osd.1 (osd.1) 248 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:23.770080+0000 osd.1 (osd.1) 249 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:24.394536+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:24.395740+0000 osd.2 (osd.2) 198 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:24.597031+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:24.600072+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:25.080521+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]': finished 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.080582+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:25.082154+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:25.083957+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.766612+0000 osd.1 (osd.1) 248 : cluster [DBG] 60.1e scrub starts 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:23.770080+0000 osd.1 (osd.1) 249 : cluster [DBG] 60.1e scrub ok 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:24.394536+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:24.395740+0000 osd.2 (osd.2) 198 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:24.597031+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:24.600072+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:25.080521+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_tier","val": "test-rados-api-smithi107-112490-85-test-flush"}]': finished 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.080582+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2024-02-10T08:44:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:25.082154+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:25.083957+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:44:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:25.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:25] "GET /metrics HTTP/1.1" 200 191360 "" "Prometheus/2.33.4" 2024-02-10T08:44:26.239 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:26.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:26.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:26.388 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:26.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.266720+0000 osd.5 (osd.5) 225 : cluster [DBG] 16.f scrub starts 2024-02-10T08:44:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.269752+0000 osd.5 (osd.5) 226 : cluster [DBG] 16.f scrub ok 2024-02-10T08:44:26.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.446006+0000 osd.6 (osd.6) 228 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.447290+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.483327+0000 mgr.y (mgr.14553) 464 : cluster [DBG] pgmap v781: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.554476+0000 osd.7 (osd.7) 211 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:25.556043+0000 osd.7 (osd.7) 212 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:26.087740+0000 mon.a (mon.0) 2754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.087824+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e520: 8 total, 7 up, 7 in 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:26.092092+0000 mon.b (mon.1) 654 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:26.092533+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.266720+0000 osd.5 (osd.5) 225 : cluster [DBG] 16.f scrub starts 2024-02-10T08:44:26.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.269752+0000 osd.5 (osd.5) 226 : cluster [DBG] 16.f scrub ok 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.446006+0000 osd.6 (osd.6) 228 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.447290+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.483327+0000 mgr.y (mgr.14553) 464 : cluster [DBG] pgmap v781: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.554476+0000 osd.7 (osd.7) 211 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:25.556043+0000 osd.7 (osd.7) 212 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:26.087740+0000 mon.a (mon.0) 2754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.087824+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e520: 8 total, 7 up, 7 in 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:26.092092+0000 mon.b (mon.1) 654 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:26.092533+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:26.543 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:26.562 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:26.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:26.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:26.693 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.266720+0000 osd.5 (osd.5) 225 : cluster [DBG] 16.f scrub starts 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.269752+0000 osd.5 (osd.5) 226 : cluster [DBG] 16.f scrub ok 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.446006+0000 osd.6 (osd.6) 228 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.447290+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.483327+0000 mgr.y (mgr.14553) 464 : cluster [DBG] pgmap v781: 250 pgs: 8 creating+peering, 56 unknown, 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.554476+0000 osd.7 (osd.7) 211 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:25.556043+0000 osd.7 (osd.7) 212 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:26.087740+0000 mon.a (mon.0) 2754 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.087824+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e520: 8 total, 7 up, 7 in 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:26.092092+0000 mon.b (mon.1) 654 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:26.092533+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:44:26.840 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:26.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:26.853 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:44:26.853 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:44:26.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:26.971 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:26.990 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:26.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:27.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:27.161 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:27.202 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:27.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:27.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:27.333 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:27.354 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:27.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:27.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:44:27.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:27.486 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.256071+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:44:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.266158+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:44:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.396346+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-02-10T08:44:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.397683+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.785522+0000 osd.1 (osd.1) 250 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:26.789623+0000 osd.1 (osd.1) 251 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:27.086743+0000 mon.a (mon.0) 2757 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:27.086809+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.256071+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.266158+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.396346+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.397683+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.785522+0000 osd.1 (osd.1) 250 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:26.789623+0000 osd.1 (osd.1) 251 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:44:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:27.086743+0000 mon.a (mon.0) 2757 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:27.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:27.086809+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2024-02-10T08:44:27.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.256071+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.266158+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.396346+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.397683+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.785522+0000 osd.1 (osd.1) 250 : cluster [DBG] 60.3 scrub starts 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:26.789623+0000 osd.1 (osd.1) 251 : cluster [DBG] 60.3 scrub ok 2024-02-10T08:44:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:27.086743+0000 mon.a (mon.0) 2757 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:44:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:27.086809+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2024-02-10T08:44:27.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:44:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.154+0000 7f3f6627e700 1 -- 172.21.15.107:0/2155604178 >> v1:172.21.15.107:6789/0 conn(0x7f3f600ff710 legacy=0x7f3f60101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.157+0000 7f3f6627e700 1 -- 172.21.15.107:0/2155604178 shutdown_connections 2024-02-10T08:44:28.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.157+0000 7f3f6627e700 1 -- 172.21.15.107:0/2155604178 >> 172.21.15.107:0/2155604178 conn(0x7f3f600f2510 msgr2=0x7f3f600f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:28.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.170+0000 7f3f6627e700 1 -- 172.21.15.107:0/2155604178 shutdown_connections 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.172+0000 7f3f6627e700 1 -- 172.21.15.107:0/2155604178 wait complete. 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 librados: starting msgr at 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 librados: starting objecter 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 Processor -- start 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 -- start start 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 librados: setting wanted keys 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.173+0000 7f3f6627e700 1 librados: calling monclient init 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.174+0000 7f3f6627e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f600670d0 con 0x7f3f600fc910 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.174+0000 7f3f6627e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f600682b0 con 0x7f3f60110990 2024-02-10T08:44:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.174+0000 7f3f6627e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f600694b0 con 0x7f3f60103250 2024-02-10T08:44:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.174+0000 7f3f5ffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3f600fc910 0x7f3f600f8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45076/0 (socket says 172.21.15.107:45076) 2024-02-10T08:44:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.174+0000 7f3f5ffff700 1 -- 172.21.15.107:0/2000250266 learned_addr learned my addr 172.21.15.107:0/2000250266 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.175+0000 7f3f5d775700 10 client.?.objecter ms_handle_connect 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.175+0000 7f3f5d775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.175+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358769037 0 0) 0x7f3f600670d0 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.175+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f48002220 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347786272 0 0) 0x7f3f48002220 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f600670d0 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3f50001980 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2597725334 0 0) 0x7f3f600670d0 con 0x7f3f600fc910 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 >> v1:172.21.15.107:6790/0 conn(0x7f3f60103250 legacy=0x7f3f600f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:28.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 >> v1:172.21.15.195:6789/0 conn(0x7f3f60110990 legacy=0x7f3f600657c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T08:44:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6006a6b0 con 0x7f3f600fc910 2024-02-10T08:44:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6006a960 con 0x7f3f600fc910 2024-02-10T08:44:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f6627e700 10 client.28290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.177+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f60069640 con 0x7f3f600fc910 2024-02-10T08:44:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.178+0000 7f3f6627e700 1 librados: init done 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.181+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3f50001d00 con 0x7f3f600fc910 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.181+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3f50005be0 con 0x7f3f600fc910 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.181+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3f50018b70 con 0x7f3f600fc910 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.182+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(522..522 src has 330..522) v4 ==== 8174+0+0 (unknown 3100978138 0 0) 0x7f3f50079e50 con 0x7f3f600fc910 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.182+0000 7f3f5d775700 10 client.28290.objecter ms_dispatch 0x7f3f60001550 osd_map(522..522 src has 330..522) v4 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.182+0000 7f3f5d775700 3 client.28290.objecter handle_osd_map got epochs [522,522] > 0 2024-02-10T08:44:28.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.182+0000 7f3f5d775700 3 client.28290.objecter handle_osd_map decoding full epoch 522 2024-02-10T08:44:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.182+0000 7f3f5d775700 20 client.28290.objecter dump_active .. 0 homeless 2024-02-10T08:44:28.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.192+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c004f50 con 0x7f3f600fc910 2024-02-10T08:44:28.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.206+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3f5004d250 con 0x7f3f600fc910 2024-02-10T08:44:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.221+0000 7f3f5d775700 10 client.28290.objecter ms_handle_connect 0x7f3f48063cd0 2024-02-10T08:44:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:28.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:28.289 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:28.438+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]} v 0) v1 -- 0x7f3f4c005a10 con 0x7f3f600fc910 2024-02-10T08:44:28.562 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:28.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:28.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:27.484017+0000 mgr.y (mgr.14553) 465 : cluster [DBG] pgmap v784: 250 pgs: 38 active+undersized+degraded, 71 active+undersized, 141 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.088526+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:28.300029+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:28.300423+0000 mon.a (mon.0) 2760 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:28.300973+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:28.301189+0000 mon.a (mon.0) 2761 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:27.484017+0000 mgr.y (mgr.14553) 465 : cluster [DBG] pgmap v784: 250 pgs: 38 active+undersized+degraded, 71 active+undersized, 141 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.088526+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:28.300029+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:28.300423+0000 mon.a (mon.0) 2760 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:28.300973+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:28.301189+0000 mon.a (mon.0) 2761 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.692 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:27.484017+0000 mgr.y (mgr.14553) 465 : cluster [DBG] pgmap v784: 250 pgs: 38 active+undersized+degraded, 71 active+undersized, 141 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:28.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.088526+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2024-02-10T08:44:28.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:28.300029+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:28.300423+0000 mon.a (mon.0) 2760 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:28.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:28.300973+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:28.301189+0000 mon.a (mon.0) 2761 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-83"}]: dispatch 2024-02-10T08:44:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:28.713 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:28.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:28.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:29.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.093+0000 7f3f5d775700 1 -- 172.21.15.107:0/2000250266 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]=0 set 2.13 pg_upmap mapping to [4,2,7] v523) v1 ==== 129+0+0 (unknown 186075884 0 0) 0x7f3f5004b660 con 0x7f3f600fc910 2024-02-10T08:44:29.102 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (17992 ms) 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23085 ms) 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3016 ms) 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2924 ms) 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3008 ms) 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-10T08:44:29.103 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3030 ms) 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4034 ms) 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (37405 ms) 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (15631 ms) 2024-02-10T08:44:29.104 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3901 ms) 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3024 ms) 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24991 ms) 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14998 ms) 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.105 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5024 ms) 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: waiting for scrubs... 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: done waiting 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25006 ms) 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10099 ms) 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8896 ms) 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9110 ms) 2024-02-10T08:44:29.106 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-10T08:44:29.107 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:44:29.107 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8912 ms) 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.100+0000 7f3f6627e700 10 librados: watch_flush enter 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.101+0000 7f3f6627e700 10 librados: watch_flush exit 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.101+0000 7f3f6627e700 20 client.28290.objecter shutdown clearing up homeless session... 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.101+0000 7f3f6627e700 10 client.28290.objecter successfully canceled tick 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.101+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3f48063cd0 legacy=0x7f3f48066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:29.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.101+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 >> v1:172.21.15.107:6789/0 conn(0x7f3f600fc910 legacy=0x7f3f600f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.117+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 shutdown_connections 2024-02-10T08:44:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.117+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 >> 172.21.15.107:0/2000250266 conn(0x7f3f600f2510 msgr2=0x7f3f60060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.123+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 shutdown_connections 2024-02-10T08:44:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.123+0000 7f3f6627e700 1 -- 172.21.15.107:0/2000250266 wait complete. 2024-02-10T08:44:29.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:29.128+0000 7f3f6627e700 1 librados: shutdown 2024-02-10T08:44:29.146 INFO:teuthology.orchestra.run.smithi107.stderr:set 2.13 pg_upmap mapping to [4,2,7] 2024-02-10T08:44:29.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.328841+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:44:29.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.331594+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:44:29.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:28.440185+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]: dispatch 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.454656+0000 osd.2 (osd.2) 199 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.457734+0000 osd.2 (osd.2) 200 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.729923+0000 osd.1 (osd.1) 252 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:28.733581+0000 osd.1 (osd.1) 253 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:29.092939+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]': finished 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.092997+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.328841+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.331594+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:28.440185+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]: dispatch 2024-02-10T08:44:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.454656+0000 osd.2 (osd.2) 199 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:44:29.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.457734+0000 osd.2 (osd.2) 200 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:44:29.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.729923+0000 osd.1 (osd.1) 252 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:44:29.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:28.733581+0000 osd.1 (osd.1) 253 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:44:29.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:29.092939+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]': finished 2024-02-10T08:44:29.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.092997+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.328841+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.331594+0000 osd.3 (osd.3) 248 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:28.440185+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]: dispatch 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.454656+0000 osd.2 (osd.2) 199 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.457734+0000 osd.2 (osd.2) 200 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.729923+0000 osd.1 (osd.1) 252 : cluster [DBG] 60.19 scrub starts 2024-02-10T08:44:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:28.733581+0000 osd.1 (osd.1) 253 : cluster [DBG] 60.19 scrub ok 2024-02-10T08:44:29.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:29.092939+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? v1:172.21.15.107:0/2000250266' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [4, 2, 7]}]': finished 2024-02-10T08:44:29.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.092997+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.438734+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.439469+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.457974+0000 osd.2 (osd.2) 201 : cluster [DBG] 16.a scrub starts 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.460859+0000 osd.2 (osd.2) 202 : cluster [DBG] 16.a scrub ok 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:29.484693+0000 mgr.y (mgr.14553) 466 : cluster [DBG] pgmap v787: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.092256+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2024-02-10T08:44:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:30.100370+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:30.101917+0000 mon.a (mon.0) 2766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.438734+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.439469+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.457974+0000 osd.2 (osd.2) 201 : cluster [DBG] 16.a scrub starts 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.460859+0000 osd.2 (osd.2) 202 : cluster [DBG] 16.a scrub ok 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:29.484693+0000 mgr.y (mgr.14553) 466 : cluster [DBG] pgmap v787: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.092256+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:30.100370+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:30.101917+0000 mon.a (mon.0) 2766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.438734+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.439469+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.457974+0000 osd.2 (osd.2) 201 : cluster [DBG] 16.a scrub starts 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.460859+0000 osd.2 (osd.2) 202 : cluster [DBG] 16.a scrub ok 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:29.484693+0000 mgr.y (mgr.14553) 466 : cluster [DBG] pgmap v787: 186 pgs: 38 active+undersized+degraded, 42 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.092256+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:30.100370+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:30.101917+0000 mon.a (mon.0) 2766 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:30.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:30.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:30.829 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:30.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:30.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:30.982 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:31.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:31.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:31.473 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.441+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/3923860312 >> v1:172.21.15.107:6790/0 conn(0x7fd0c8103270 legacy=0x7fd0c81056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.441+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/3923860312 shutdown_connections 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.441+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/3923860312 >> 172.21.15.107:0/3923860312 conn(0x7fd0c80f2510 msgr2=0x7fd0c80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.441+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/3923860312 shutdown_connections 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.441+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/3923860312 wait complete. 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 librados: starting msgr at 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 librados: starting objecter 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 Processor -- start 2024-02-10T08:44:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 -- start start 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 librados: setting wanted keys 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 librados: calling monclient init 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0c81173d0 con 0x7fd0c8103270 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0c81185b0 con 0x7fd0c80ff730 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0cd6fe700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0c81197b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0c6ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd0c80fc930 0x7fd0c80f9b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44796/0 (socket says 172.21.15.107:44796) 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0c6ffd700 1 -- 172.21.15.107:0/16806957 learned_addr learned my addr 172.21.15.107:0/16806957 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter ms_handle_connect 0x7fd0c80fc930 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter ms_handle_connect 0x7fd0c8103270 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter ms_handle_connect 0x7fd0c80ff730 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155114245 0 0) 0x7fd0c81197b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b0003030 con 0x7fd0c80fc930 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736885313 0 0) 0x7fd0c81173d0 con 0x7fd0c8103270 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c81197b0 con 0x7fd0c8103270 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068509589 0 0) 0x7fd0c81185b0 con 0x7fd0c80ff730 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.442+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c81173d0 con 0x7fd0c80ff730 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835620180 0 0) 0x7fd0b0003030 con 0x7fd0c80fc930 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0c81185b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184965396 0 0) 0x7fd0c81197b0 con 0x7fd0c8103270 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0b0003030 con 0x7fd0c8103270 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd0b80019a0 con 0x7fd0c80fc930 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1080606437 0 0) 0x7fd0c81173d0 con 0x7fd0c80ff730 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0c81197b0 con 0x7fd0c80ff730 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd0c00021d0 con 0x7fd0c8103270 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd0bc0017c0 con 0x7fd0c80ff730 2024-02-10T08:44:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2275937909 0 0) 0x7fd0c81185b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 >> v1:172.21.15.195:6789/0 conn(0x7fd0c80ff730 legacy=0x7fd0c80fa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 >> v1:172.21.15.107:6789/0 conn(0x7fd0c8103270 legacy=0x7fd0c8115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c811a9b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c81175c0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0cd6fe700 10 client.53873.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c8117ab0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0cd6fe700 1 librados: init done 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd0b80024b0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.443+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd0b8005ae0 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.444+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b4004f20 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.444+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd0b8002010 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(525..525 src has 330..525) v4 ==== 8173+0+0 (unknown 112089743 0 0) 0x7fd0b8078e50 con 0x7fd0c80fc930 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 10 client.53873.objecter ms_dispatch 0x7fd0c8001550 osd_map(525..525 src has 330..525) v4 2024-02-10T08:44:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 3 client.53873.objecter handle_osd_map got epochs [525,525] > 0 2024-02-10T08:44:31.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 3 client.53873.objecter handle_osd_map decoding full epoch 525 2024-02-10T08:44:31.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 20 client.53873.objecter dump_active .. 0 homeless 2024-02-10T08:44:31.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.445+0000 7fd0affff700 10 client.53873.objecter ms_handle_connect 0x7fd0b0063c70 2024-02-10T08:44:31.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.449+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd0b804c230 con 0x7fd0c80fc930 2024-02-10T08:44:31.491 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:31.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:31.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:31.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.367463+0000 osd.5 (osd.5) 229 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:44:31.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.371904+0000 osd.5 (osd.5) 230 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:44:31.622 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.424225+0000 osd.2 (osd.2) 203 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.426338+0000 osd.2 (osd.2) 204 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.582251+0000 osd.7 (osd.7) 213 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.585727+0000 osd.7 (osd.7) 214 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.723748+0000 osd.1 (osd.1) 254 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:30.727650+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.079020+0000 mon.a (mon.0) 2767 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.079096+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.086536+0000 mon.b (mon.1) 658 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.103941+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.107252+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.367463+0000 osd.5 (osd.5) 229 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:44:31.623 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.371904+0000 osd.5 (osd.5) 230 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.424225+0000 osd.2 (osd.2) 203 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.426338+0000 osd.2 (osd.2) 204 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.582251+0000 osd.7 (osd.7) 213 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.585727+0000 osd.7 (osd.7) 214 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.723748+0000 osd.1 (osd.1) 254 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:30.727650+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.079020+0000 mon.a (mon.0) 2767 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.079096+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.086536+0000 mon.b (mon.1) 658 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.103941+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.107252+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.632+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd0b40051d0 con 0x7fd0c80fc930 2024-02-10T08:44:31.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.687+0000 7f9bb9a00700 1 -- 172.21.15.107:0/2854740809 >> v1:172.21.15.107:6789/0 conn(0x7f9bb40fc7b0 legacy=0x7f9bb40fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.687+0000 7f9bb9a00700 1 -- 172.21.15.107:0/2854740809 shutdown_connections 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.687+0000 7f9bb9a00700 1 -- 172.21.15.107:0/2854740809 >> 172.21.15.107:0/2854740809 conn(0x7f9bb40f2390 msgr2=0x7f9bb40f47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.687+0000 7f9bb9a00700 1 -- 172.21.15.107:0/2854740809 shutdown_connections 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.687+0000 7f9bb9a00700 1 -- 172.21.15.107:0/2854740809 wait complete. 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 librados: starting msgr at 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 librados: starting objecter 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 Processor -- start 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 -- start start 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 librados: setting wanted keys 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 librados: calling monclient init 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bb40fbf40 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bb40fc110 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.688+0000 7f9bb9a00700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bb411b9c0 con 0x7f9bb41030f0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb37fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9bb41030f0 0x7f9bb4118280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44826/0 (socket says 172.21.15.107:44826) 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb37fe700 1 -- 172.21.15.107:0/931309929 learned_addr learned my addr 172.21.15.107:0/931309929 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb27fc700 1 --1- 172.21.15.107:0/931309929 >> v1:172.21.15.195:6789/0 conn(0x7f9bb40ff5b0 0x7f9bb4114b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:38826/0 (socket says 172.21.15.107:38826) 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter ms_handle_connect 0x7f9bb41030f0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter ms_handle_connect 0x7f9bb40fc7b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter ms_handle_connect 0x7f9bb40ff5b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307678423 0 0) 0x7f9bb40fc110 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b9c0033f0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169519677 0 0) 0x7f9bb40fbf40 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb40fc110 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732281578 0 0) 0x7f9bb411b9c0 con 0x7f9bb41030f0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb40fbf40 con 0x7f9bb41030f0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194718881 0 0) 0x7f9b9c0033f0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb411b9c0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9ba80018c0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14918984 0 0) 0x7f9bb40fc110 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b9c0033f0 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121297697 0 0) 0x7f9bb40fbf40 con 0x7f9bb41030f0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb40fc110 con 0x7f9bb41030f0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9ba4001f90 con 0x7f9bb40fc7b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9bac001d80 con 0x7f9bb41030f0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3094240460 0 0) 0x7f9bb411b9c0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 >> v1:172.21.15.107:6790/0 conn(0x7f9bb41030f0 legacy=0x7f9bb4118280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 >> v1:172.21.15.107:6789/0 conn(0x7f9bb40fc7b0 legacy=0x7f9bb40fb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb411cb60 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb411bbb0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb9a00700 10 client.48595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb411c0a0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9bb9a00700 1 librados: init done 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9ba8001f80 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.689+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9ba8005990 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.690+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9ba0004f00 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9ba8001ae0 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(525..525 src has 330..525) v4 ==== 8173+0+0 (unknown 112089743 0 0) 0x7f9ba8078c10 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 osd_map(525..525 src has 330..525) v4 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 3 client.48595.objecter handle_osd_map got epochs [525,525] > 0 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 3 client.48595.objecter handle_osd_map decoding full epoch 525 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 20 client.48595.objecter dump_active .. 0 homeless 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v525) v1 ==== 54+0+16 (unknown 2554073534 0 2420000192) 0x7f9ba8079060 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9b9c063fb0 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 1 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 1 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 0 1 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 4 2024-02-10T08:44:31.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 10 client.48595.objecter _send_command 1 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.691+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba0004950 con 0x7f9ba0004530 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0004530 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba0004950 con 0x7f9ba0004530 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 1 = system:0 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 4 2024-02-10T08:44:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.692+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 0 1 2024-02-10T08:44:31.703 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:31.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:31.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 2 2024-02-10T08:44:31.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 2 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00051a0 osd=0 3 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 0 2 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 4 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 10 client.48595.objecter _send_command 2 2024-02-10T08:44:31.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.708+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba0004d30 con 0x7f9ba0004530 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba0004d30 con 0x7f9ba0004530 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00051a0 osd=0 4 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 0 2 2024-02-10T08:44:31.711 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 1 3 2024-02-10T08:44:31.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 4 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 10 client.48595.objecter _send_command 3 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba000bf60 con 0x7f9ba0003b40 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.710+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0003b40 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.711+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba000bf60 con 0x7f9ba0003b40 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.711+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.711+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 3 = system:0 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.711+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 4 2024-02-10T08:44:31.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.711+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 1 3 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 4 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 4 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba0004270 osd=1 3 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 1 4 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 4 2024-02-10T08:44:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 10 client.48595.objecter _send_command 4 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.725+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba000d280 con 0x7f9ba0003b40 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba000d280 con 0x7f9ba0003b40 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba0004270 osd=1 4 2024-02-10T08:44:31.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 1 4 2024-02-10T08:44:31.729 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 5 2024-02-10T08:44:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 5 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 2 5 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 4 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 10 client.48595.objecter _send_command 5 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba0010d30 con 0x7f9ba0002070 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.727+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0002070 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.728+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba0010d30 con 0x7f9ba0002070 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.728+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.728+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 5 = system:0 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.728+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 4 2024-02-10T08:44:31.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.728+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 2 5 2024-02-10T08:44:31.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 6 2024-02-10T08:44:31.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 6 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00033a0 osd=2 3 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 2 6 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 4 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 10 client.48595.objecter _send_command 6 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.742+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba0001630 con 0x7f9ba0002070 2024-02-10T08:44:31.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba0001630 con 0x7f9ba0002070 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00033a0 osd=2 4 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 2 6 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:31.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 7 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 7 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 3 7 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 4 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 10 client.48595.objecter _send_command 7 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.743+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba000d010 con 0x7f9ba0010ed0 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0010ed0 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba000d010 con 0x7f9ba0010ed0 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:31.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 7 = system:0 2024-02-10T08:44:31.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 4 2024-02-10T08:44:31.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.744+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 3 7 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 8 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 8 2024-02-10T08:44:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba000c8d0 osd=3 3 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 3 8 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 4 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 10 client.48595.objecter _send_command 8 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.758+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba0012210 con 0x7f9ba0010ed0 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba0012210 con 0x7f9ba0010ed0 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba000c8d0 osd=3 4 2024-02-10T08:44:31.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 3 8 2024-02-10T08:44:31.762 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 9 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 20 client.48595.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 10 client.48595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=526}) v3 -- 0x7f9ba000c340 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ba0012210 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_get_version_reply(handle=1 version=525) v2 ==== 24+0+0 (unknown 1795141584 0 0) 0x7f9ba804bb20 con 0x7f9bb40ff5b0 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb17fa700 10 client.48595.objecter _check_command_map_dne tid 9 current 525 map_dne_bound 525 2024-02-10T08:44:31.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb17fa700 10 client.48595.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.760+0000 7f9bb17fa700 15 client.48595.objecter _session_command_op_remove -1 9 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 10 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 10 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 5 10 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 4 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 10 client.48595.objecter _send_command 10 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba00131c0 con 0x7f9ba0011df0 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.761+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0011df0 2024-02-10T08:44:31.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.762+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba00131c0 con 0x7f9ba0011df0 2024-02-10T08:44:31.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.762+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:31.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.762+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 10 = system:0 2024-02-10T08:44:31.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.762+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 4 2024-02-10T08:44:31.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.762+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 5 10 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 11 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 11 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba0011c80 osd=5 3 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 5 11 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 4 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 10 client.48595.objecter _send_command 11 2024-02-10T08:44:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.776+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba00133a0 con 0x7f9ba0011df0 2024-02-10T08:44:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba00133a0 con 0x7f9ba0011df0 2024-02-10T08:44:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba0011c80 osd=5 4 2024-02-10T08:44:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 5 11 2024-02-10T08:44:31.780 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 12 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 12 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 6 12 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 4 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 10 client.48595.objecter _send_command 12 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba001efc0 con 0x7f9ba001b4b0 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.778+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba001b4b0 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.779+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba001efc0 con 0x7f9ba001b4b0 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.779+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.779+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 12 = system:0 2024-02-10T08:44:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.779+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 4 2024-02-10T08:44:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.779+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 6 12 2024-02-10T08:44:31.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.367463+0000 osd.5 (osd.5) 229 : cluster [DBG] 60.1f scrub starts 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.371904+0000 osd.5 (osd.5) 230 : cluster [DBG] 60.1f scrub ok 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.424225+0000 osd.2 (osd.2) 203 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.426338+0000 osd.2 (osd.2) 204 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.582251+0000 osd.7 (osd.7) 213 : cluster [DBG] 60.10 scrub starts 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.585727+0000 osd.7 (osd.7) 214 : cluster [DBG] 60.10 scrub ok 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.723748+0000 osd.1 (osd.1) 254 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:30.727650+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:44:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.079020+0000 mon.a (mon.0) 2767 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.079096+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2024-02-10T08:44:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.086536+0000 mon.b (mon.1) 658 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:44:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.103941+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.107252+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:44:31.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 13 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 13 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba00133a0 osd=6 3 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 6 13 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 4 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 10 client.48595.objecter _send_command 13 2024-02-10T08:44:31.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.793+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba001efc0 con 0x7f9ba001b4b0 2024-02-10T08:44:31.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba001efc0 con 0x7f9ba001b4b0 2024-02-10T08:44:31.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba00133a0 osd=6 4 2024-02-10T08:44:31.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 6 13 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 14 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 14 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 7 14 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 4 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 10 client.48595.objecter _send_command 14 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba00201b0 con 0x7f9ba0020c80 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.795+0000 7f9b9bfff700 10 client.48595.objecter ms_handle_connect 0x7f9ba0020c80 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.796+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9ba00201b0 con 0x7f9ba0020c80 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.796+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.796+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 14 = system:0 2024-02-10T08:44:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.796+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 4 2024-02-10T08:44:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.796+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 7 14 2024-02-10T08:44:31.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 10 client.48595.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign -1 15 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 20 client.48595.objecter _get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_remove -1 15 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 20 client.48595.objecter get_session s=0x7f9ba001efc0 osd=7 3 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 15 client.48595.objecter _session_command_op_assign 7 15 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 4 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 10 client.48595.objecter _send_command 15 2024-02-10T08:44:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.811+0000 7f9bb9a00700 1 -- 172.21.15.107:0/931309929 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ba00201b0 con 0x7f9ba0020c80 2024-02-10T08:44:31.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.813+0000 7f9b9bfff700 1 -- 172.21.15.107:0/931309929 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9ba00201b0 con 0x7f9ba0020c80 2024-02-10T08:44:31.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.813+0000 7f9b9bfff700 10 client.48595.objecter ms_dispatch 0x7f9bb4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.813+0000 7f9b9bfff700 10 client.48595.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.813+0000 7f9b9bfff700 20 client.48595.objecter put_session s=0x7f9ba001efc0 osd=7 4 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:31.813+0000 7f9b9bfff700 15 client.48595.objecter _session_command_op_remove 7 15 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.816 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:31.816 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.816 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.816 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:44:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:32.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.100+0000 7fd0affff700 1 -- 172.21.15.107:0/16806957 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v526) v1 ==== 101+0+0 (unknown 1131256706 0 0) 0x7fd0b804a6a0 con 0x7fd0c80fc930 2024-02-10T08:44:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.114+0000 7fd0cd6fe700 10 librados: watch_flush enter 2024-02-10T08:44:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.121+0000 7fd0cd6fe700 10 librados: watch_flush exit 2024-02-10T08:44:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.121+0000 7fd0cd6fe700 20 client.53873.objecter shutdown clearing up homeless session... 2024-02-10T08:44:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.121+0000 7fd0cd6fe700 10 client.53873.objecter successfully canceled tick 2024-02-10T08:44:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.132+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd0b0063c70 legacy=0x7fd0b0066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.132+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 >> v1:172.21.15.107:6790/0 conn(0x7fd0c80fc930 legacy=0x7fd0c80f9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.141+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 shutdown_connections 2024-02-10T08:44:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.141+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 >> 172.21.15.107:0/16806957 conn(0x7fd0c80f2510 msgr2=0x7fd0c80613e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.142+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 shutdown_connections 2024-02-10T08:44:32.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.154+0000 7fd0cd6fe700 1 -- 172.21.15.107:0/16806957 wait complete. 2024-02-10T08:44:32.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:32.160+0000 7fd0cd6fe700 1 librados: shutdown 2024-02-10T08:44:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:44:32.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.265467+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.267256+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.485475+0000 mgr.y (mgr.14553) 467 : cluster [DBG] pgmap v790: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.571779+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:31.574677+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.590021+0000 mgr.y (mgr.14553) 468 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.634197+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.107:0/16806957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.634507+0000 mon.a (mon.0) 2770 : audit [INF] from='client.53873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:31.692521+0000 mon.b (mon.1) 660 : audit [DBG] from='client.? v1:172.21.15.107:0/931309929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:32.089091+0000 mon.a (mon.0) 2771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:32.093141+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:32.093219+0000 mon.a (mon.0) 2773 : audit [INF] from='client.53873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:32.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:32.093265+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.265467+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.267256+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.485475+0000 mgr.y (mgr.14553) 467 : cluster [DBG] pgmap v790: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.571779+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:31.574677+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.590021+0000 mgr.y (mgr.14553) 468 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.634197+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.107:0/16806957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.634507+0000 mon.a (mon.0) 2770 : audit [INF] from='client.53873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:31.692521+0000 mon.b (mon.1) 660 : audit [DBG] from='client.? v1:172.21.15.107:0/931309929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:32.089091+0000 mon.a (mon.0) 2771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:32.093141+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:32.093219+0000 mon.a (mon.0) 2773 : audit [INF] from='client.53873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:32.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:32.093265+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.265467+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.267256+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.485475+0000 mgr.y (mgr.14553) 467 : cluster [DBG] pgmap v790: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.571779+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:31.574677+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.590021+0000 mgr.y (mgr.14553) 468 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.634197+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.107:0/16806957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.634507+0000 mon.a (mon.0) 2770 : audit [INF] from='client.53873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:31.692521+0000 mon.b (mon.1) 660 : audit [DBG] from='client.? v1:172.21.15.107:0/931309929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:32.089091+0000 mon.a (mon.0) 2771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:44:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:32.093141+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:44:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:32.093219+0000 mon.a (mon.0) 2773 : audit [INF] from='client.53873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:32.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:32.093265+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2024-02-10T08:44:32.883 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:32.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:32.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:33.016 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:33.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:33.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:33.501 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:33.521 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:33.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:33.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:34.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:34.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:34.236 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:34.263 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:34.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:34.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:32.563818+0000 osd.7 (osd.7) 217 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:32.565772+0000 osd.7 (osd.7) 218 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.095569+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.480289+0000 osd.2 (osd.2) 205 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.484189+0000 osd.2 (osd.2) 206 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.485985+0000 mgr.y (mgr.14553) 469 : cluster [DBG] pgmap v793: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.682215+0000 mon.c (mon.2) 416 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.953918+0000 mon.c (mon.2) 417 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.954187+0000 mon.a (mon.0) 2776 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.954882+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.955104+0000 mon.a (mon.0) 2777 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.955722+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.955932+0000 mon.a (mon.0) 2778 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.956516+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.956730+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:33.961181+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:32.563818+0000 osd.7 (osd.7) 217 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:32.565772+0000 osd.7 (osd.7) 218 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.095569+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.480289+0000 osd.2 (osd.2) 205 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:44:34.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.484189+0000 osd.2 (osd.2) 206 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.485985+0000 mgr.y (mgr.14553) 469 : cluster [DBG] pgmap v793: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.682215+0000 mon.c (mon.2) 416 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.953918+0000 mon.c (mon.2) 417 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.954187+0000 mon.a (mon.0) 2776 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.954882+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.955104+0000 mon.a (mon.0) 2777 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.955722+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.955932+0000 mon.a (mon.0) 2778 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.956516+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.956730+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:33.961181+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:32.563818+0000 osd.7 (osd.7) 217 : cluster [DBG] 60.17 scrub starts 2024-02-10T08:44:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:32.565772+0000 osd.7 (osd.7) 218 : cluster [DBG] 60.17 scrub ok 2024-02-10T08:44:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.095569+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2024-02-10T08:44:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.480289+0000 osd.2 (osd.2) 205 : cluster [DBG] 60.6 scrub starts 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.484189+0000 osd.2 (osd.2) 206 : cluster [DBG] 60.6 scrub ok 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.485985+0000 mgr.y (mgr.14553) 469 : cluster [DBG] pgmap v793: 218 pgs: 1 creating+activating, 1 active, 1 active+clean+remapped, 15 creating+peering, 38 active+undersized+degraded, 2 creating+activating+undersized, 49 active+undersized, 111 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.682215+0000 mon.c (mon.2) 416 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.953918+0000 mon.c (mon.2) 417 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.954187+0000 mon.a (mon.0) 2776 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.954882+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.955104+0000 mon.a (mon.0) 2777 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.955722+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.955932+0000 mon.a (mon.0) 2778 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.956516+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.956730+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:34.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:33.961181+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:34.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:34.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:34.700 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:34.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:34.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:35.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.296294+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.b deep-scrub starts 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:33.297420+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.b deep-scrub ok 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:44:33.957237+0000 mgr.y (mgr.14553) 470 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 3256M 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:34.107395+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.244659+0000 mon.b (mon.1) 661 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.245937+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.246435+0000 mon.b (mon.1) 662 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.246613+0000 mon.a (mon.0) 2783 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.277891+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.278360+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.279392+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.279685+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.280761+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.281048+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.281801+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.282094+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cephadm 2024-02-10T08:44:34.282693+0000 mgr.y (mgr.14553) 471 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3256M 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.287893+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.290414+0000 mon.c (mon.2) 425 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.290942+0000 mon.c (mon.2) 426 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:34.295136+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.296294+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.b deep-scrub starts 2024-02-10T08:44:35.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:33.297420+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.b deep-scrub ok 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:44:33.957237+0000 mgr.y (mgr.14553) 470 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 3256M 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:34.107395+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.244659+0000 mon.b (mon.1) 661 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.245937+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.246435+0000 mon.b (mon.1) 662 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.246613+0000 mon.a (mon.0) 2783 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.277891+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.278360+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.279392+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.279685+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.280761+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.281048+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.281801+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.282094+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cephadm 2024-02-10T08:44:34.282693+0000 mgr.y (mgr.14553) 471 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3256M 2024-02-10T08:44:35.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.287893+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.290414+0000 mon.c (mon.2) 425 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:44:35.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.290942+0000 mon.c (mon.2) 426 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:44:35.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:34.295136+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:35.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:17 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [write 0~3] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 468.638409497, 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057249742999999999, 2024-02-10T08:44:35.534 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893949+0000", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893947+0000", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9770000000000006e-06 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893956+0000", 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2445e-05 2024-02-10T08:44:35.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893968+0000", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8250000000000005e-06 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.893976+0000", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0333000000000001e-05 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.894026+0000", 2024-02-10T08:44:35.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6769999999999998e-06 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.894033+0000", 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047954573 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.941987+0000", 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014102099999999999 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.942129+0000", 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016330500000000001 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.942292+0000", 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005302813 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.947595+0000", 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033408940000000001 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.950936+0000", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016446899999999999 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951100+0000", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3541000000000001e-05 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951154+0000", 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.511e-05 2024-02-10T08:44:35.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.951199+0000", 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057249742999999999 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25276.0:2 52.14 e113/112 52:28e9c4a8:test-rados-api-smithi107-112562-5::foo:head v 113'1, mlcod=0'0)", 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 468.57360855299999, 2024-02-10T08:44:35.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040456570000000002, 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958750+0000", 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958749+0000", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022999999999999e-05 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958760+0000", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4339999999999997e-06 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958767+0000", 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1820000000000003e-06 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.958773+0000", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011023809999999999 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.959875+0000", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1713e-05 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.959897+0000", 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028442839999999999 2024-02-10T08:44:35.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962741+0000", 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4208e-05 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.962795+0000", 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040456570000000002 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.545 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(51.6s1 113/111 ECSubWrite(tid=1200, reqid=client.35219.0:7, at_version=113'3, trim_to=0'0, roll_forward_to=112'1))", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 468.50385839, 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054912240000000003, 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028500+0000", 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028497+0000", 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3481e-05 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028530+0000", 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0550000000000001e-05 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028551+0000", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7580000000000004e-06 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028561+0000", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3418999999999998e-05 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028614+0000", 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:44:35.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.028623+0000", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053399140000000003 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.033963+0000", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8592e-05 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.033991+0000", 2024-02-10T08:44:35.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054912240000000003 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:19 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e114)", 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 467.59479042800001, 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030457504999999999, 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:44:35.550 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 19 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937568+0000", 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937566+0000", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0020000000000003e-06 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937572+0000", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5529999999999993e-06 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937581+0000", 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2579999999999998e-06 2024-02-10T08:44:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.937588+0000", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062451119999999997 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.943833+0000", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4054000000000004e-05 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.943927+0000", 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021535700000000001 2024-02-10T08:44:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.944143+0000", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028311209999999998 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.946974+0000", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067872269999999998 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953761+0000", 2024-02-10T08:44:35.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01420008 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967961+0000", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3223999999999999e-05 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.967994+0000", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1270999999999999e-05 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.968025+0000", 2024-02-10T08:44:35.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030457504999999999 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.24889.0:19 23.d e114/102 ondisk, result = 0)", 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 467.58303185199998, 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044369580000000004, 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949327+0000", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949326+0000", 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9810000000000003e-06 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949331+0000", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1840000000000005e-06 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949340+0000", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0509999999999999e-06 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.949347+0000", 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043975309999999997 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953744+0000", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8240000000000006e-06 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953753+0000", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0251999999999999e-05 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:47.953763+0000", 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044369580000000004 2024-02-10T08:44:35.559 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(50.7s2 114/111 ECSubWrite(tid=148, reqid=client.35222.0:14, at_version=114'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 467.46028512300001, 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019889280000000001, 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:44:35.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072073+0000", 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072072+0000", 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7529000000000001e-05 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072089+0000", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2250000000000006e-06 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072099+0000", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6750000000000001e-06 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072103+0000", 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4253999999999999e-05 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072158+0000", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1949999999999997e-06 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.072165+0000", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001880217 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.074045+0000", 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7237000000000001e-05 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:48.074062+0000", 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019889280000000001 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.24889.0:21 23.d 23:b264a8b5:test-rados-api-smithi107-112525-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 466.38379273300001, 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022309029999999998, 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.564 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.24889", 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110400822", 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 21 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148566+0000", 2024-02-10T08:44:35.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148562+0000", 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3216e-05 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148575+0000", 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0112999999999999e-05 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148595+0000", 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0395e-05 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148606+0000", 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0525999999999995e-05 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148676+0000", 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010333299999999999 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148780+0000", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000200175 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.148980+0000", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091076999999999998 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.149891+0000", 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00075057699999999999 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150641+0000", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6660999999999998e-05 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150698+0000", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.291e-05 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150751+0000", 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5816999999999999e-05 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.150797+0000", 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022309029999999998 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25153.0:8 41.5 e115/107 41:a378f8be:::oid_PoolQuotaPP_5:head v 115'2, mlcod=0'0)", 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 465.74379993500003, 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013390228000000001, 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788558+0000", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788556+0000", 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-02-10T08:44:35.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788573+0000", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0271e-05 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788593+0000", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1259999999999998e-06 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788602+0000", 2024-02-10T08:44:35.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6830999999999995e-05 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788698+0000", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3570999999999998e-05 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.788722+0000", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013167371000000001 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.801889+0000", 2024-02-10T08:44:35.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9398000000000003e-05 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:49.801949+0000", 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013390228000000001 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(osd.0.0:283 23.d e116/102 ondisk, result = 0)", 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 465.14271411800001, 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021669789999999999, 2024-02-10T08:44:35.575 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389644+0000", 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:44:35.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389643+0000", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2249999999999999e-06 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389649+0000", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8700000000000003e-06 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389656+0000", 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9599999999999997e-06 2024-02-10T08:44:35.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.389661+0000", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021301509999999998 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391792+0000", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3749999999999992e-06 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391801+0000", 2024-02-10T08:44:35.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0264e-05 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.391811+0000", 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021669789999999999 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=251, reqid=client.35273.0:1, at_version=117'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.571624955, 2024-02-10T08:44:35.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012908147999999999, 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960733+0000", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.402e-05 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960757+0000", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6120000000000004e-06 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960764+0000", 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.968e-06 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.960771+0000", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00065651399999999997 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.961427+0000", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6250000000000002e-06 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.961437+0000", 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012188005999999999 2024-02-10T08:44:35.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.973625+0000", 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6701000000000002e-05 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.973642+0000", 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012908147999999999 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.583 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:4 52.1d 52:bb1f5efa:test-rados-api-smithi107-112562-7::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e117)", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.56050826799998, 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035486110000000001, 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971850+0000", 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971847+0000", 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4769999999999999e-05 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971862+0000", 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4911e-05 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971877+0000", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.893e-06 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971887+0000", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034160739999999998 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975303+0000", 2024-02-10T08:44:35.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5656999999999996e-05 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975399+0000", 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035486110000000001 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:5 52.1d 52:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e117)", 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.56045666300002, 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035821059999999998, 2024-02-10T08:44:35.587 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971902+0000", 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971901+0000", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4009999999999996e-06 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971906+0000", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2639999999999997e-06 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971913+0000", 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0470000000000004e-06 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.971917+0000", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035043299999999999 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975421+0000", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2872000000000004e-05 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.975484+0000", 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035821059999999998 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.590 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=252, reqid=client.35273.0:1, at_version=0'0, trim_to=0'0, roll_forward_to=117'1))", 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.55152979600001, 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016757220999999999, 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980829+0000", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980827+0000", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2146e-05 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980839+0000", 2024-02-10T08:44:35.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1671000000000001e-05 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980851+0000", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1720000000000001e-06 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980858+0000", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4196999999999995e-05 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980922+0000", 2024-02-10T08:44:35.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1598e-05 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.980934+0000", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016631714999999998 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997566+0000", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0282e-05 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997586+0000", 2024-02-10T08:44:35.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016757220999999999 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(58.1s2 117/116 ECSubWrite(tid=253, reqid=client.35273.0:2, at_version=117'2, trim_to=0'0, roll_forward_to=117'1))", 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.546622786, 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011863251, 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985736+0000", 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985734+0000", 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3638e-05 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985758+0000", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5441e-05 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985773+0000", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7770000000000002e-06 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.985781+0000", 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051802100000000002 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986299+0000", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0242e-05 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.986309+0000", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011282307 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997592+0000", 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1210000000000001e-06 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:50.997599+0000", 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011863251 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26977.0:3 66.17 66:ef3fc2a1:test-rados-api-smithi107-112525-8::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e121)", 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 460.45854621799998, 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013570198, 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.599 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26977", 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073812+0000", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073810+0000", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9479999999999997e-06 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073819+0000", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0499999999999999e-05 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073830+0000", 2024-02-10T08:44:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2830000000000004e-06 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.073837+0000", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001023461 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.074860+0000", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012824000000000001 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.074989+0000", 2024-02-10T08:44:35.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000145761 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.075134+0000", 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012714499999999999 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.076406+0000", 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010855439999999999 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:35.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087261+0000", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0412999999999998e-05 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087292+0000", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0078999999999999e-05 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087342+0000", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0708999999999999e-05 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087382+0000", 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013570198 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.26977.0:3 66.17 e121/119 ondisk, result = 0)", 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 460.45149923899999, 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065275589999999996, 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080859+0000", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080858+0000", 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.896e-06 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080864+0000", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3210000000000005e-06 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080872+0000", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2320000000000004e-06 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.080879+0000", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063623159999999998 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087242+0000", 2024-02-10T08:44:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0676000000000001e-05 2024-02-10T08:44:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087252+0000", 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013427 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087387+0000", 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065275589999999996 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.611 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.15969.0:2 67.6 e121/120 67:602f83fe:::foo:head v 121'1, mlcod=0'0)", 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 460.44475617299997, 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042885639999999999, 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:44:35.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087602+0000", 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7630000000000003e-06 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:35.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087608+0000", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0780000000000001e-06 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087615+0000", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3210000000000001e-06 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.087621+0000", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055538500000000004 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:35.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.088176+0000", 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6487e-05 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.088193+0000", 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003665031 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.091858+0000", 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3052999999999997e-05 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:35.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:55.091891+0000", 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042885639999999999 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:35.616 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:44:35.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:35.617 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.296294+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.b deep-scrub starts 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:33.297420+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.b deep-scrub ok 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:44:33.957237+0000 mgr.y (mgr.14553) 470 : cephadm [INF] Adjusting osd_memory_target on smithi195 to 3256M 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:34.107395+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.244659+0000 mon.b (mon.1) 661 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.245937+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.246435+0000 mon.b (mon.1) 662 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.246613+0000 mon.a (mon.0) 2783 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-86"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.277891+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.278360+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.279392+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.279685+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.280761+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.281048+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.281801+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.282094+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cephadm 2024-02-10T08:44:34.282693+0000 mgr.y (mgr.14553) 471 : cephadm [INF] Adjusting osd_memory_target on smithi107 to 3256M 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.287893+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.290414+0000 mon.c (mon.2) 425 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.290942+0000 mon.c (mon.2) 426 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:44:35.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:34.295136+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:35.646 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:35.664 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:35.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:35.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:35.795 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:35.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:35.864 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:35] "GET /metrics HTTP/1.1" 200 191360 "" "Prometheus/2.33.4" 2024-02-10T08:44:35.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:35.944 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:35.961 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:35.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:36.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:36.093 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:36.113 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:36.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:36.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:44:36.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:36.261 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:36.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:36.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:36.397 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:36.416 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:36.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:36.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:36.525 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:36.536 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:34.560955+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:44:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:34.562365+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:44:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.290659+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2024-02-10T08:44:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.374061+0000 osd.5 (osd.5) 231 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:44:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.382965+0000 osd.5 (osd.5) 232 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:44:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.486616+0000 mgr.y (mgr.14553) 472 : cluster [DBG] pgmap v796: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:36.544 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:36.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:44:36.563 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:36.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:36.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:34.560955+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:34.562365+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.290659+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.374061+0000 osd.5 (osd.5) 231 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.382965+0000 osd.5 (osd.5) 232 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.486616+0000 mgr.y (mgr.14553) 472 : cluster [DBG] pgmap v796: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:34.560955+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:34.562365+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.290659+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.374061+0000 osd.5 (osd.5) 231 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.382965+0000 osd.5 (osd.5) 232 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.486616+0000 mgr.y (mgr.14553) 472 : cluster [DBG] pgmap v796: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 185/1143 objects degraded (16.185%) 2024-02-10T08:44:36.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:36.793 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:36.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:36.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:36.924 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:37.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:44:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:37.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:37.413 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:37.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:37.510 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:37.529 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:37.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:37.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:37.657 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.781771+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:44:37.659 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:35.784709+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:44:37.659 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.296553+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2024-02-10T08:44:37.659 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:36.310160+0000 mon.b (mon.1) 663 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.659 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:36.312423+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.659 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.341357+0000 osd.5 (osd.5) 233 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:44:37.660 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.345938+0000 osd.5 (osd.5) 234 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:44:37.660 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:37.677 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:37.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.781771+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:35.784709+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.296553+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:36.310160+0000 mon.b (mon.1) 663 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:36.312423+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.341357+0000 osd.5 (osd.5) 233 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.345938+0000 osd.5 (osd.5) 234 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.781771+0000 osd.1 (osd.1) 256 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:35.784709+0000 osd.1 (osd.1) 257 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.296553+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:36.310160+0000 mon.b (mon.1) 663 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:36.312423+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.341357+0000 osd.5 (osd.5) 233 : cluster [DBG] 60.13 scrub starts 2024-02-10T08:44:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.345938+0000 osd.5 (osd.5) 234 : cluster [DBG] 60.13 scrub ok 2024-02-10T08:44:37.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:37.809 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:37.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:37.990 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:38.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:38.088 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:38.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.567803+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.570693+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.774197+0000 osd.1 (osd.1) 258 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:36.777525+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:37.298063+0000 mon.a (mon.0) 2793 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.298153+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.373784+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:37.375787+0000 mon.b (mon.1) 664 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:37.380006+0000 mon.a (mon.0) 2795 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.388013+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.468939+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.469793+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.487510+0000 mgr.y (mgr.14553) 473 : cluster [DBG] pgmap v799: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.567803+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.570693+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:44:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.774197+0000 osd.1 (osd.1) 258 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:36.777525+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:37.298063+0000 mon.a (mon.0) 2793 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.298153+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.373784+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:37.375787+0000 mon.b (mon.1) 664 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:37.380006+0000 mon.a (mon.0) 2795 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.388013+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:44:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.468939+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:44:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.469793+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:44:38.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.487510+0000 mgr.y (mgr.14553) 473 : cluster [DBG] pgmap v799: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:38.433 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:38.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:38.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.567803+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:44:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.570693+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:44:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.774197+0000 osd.1 (osd.1) 258 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:44:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:36.777525+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:44:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:37.298063+0000 mon.a (mon.0) 2793 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.298153+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.373784+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:37.375787+0000 mon.b (mon.1) 664 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:37.380006+0000 mon.a (mon.0) 2795 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.388013+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.468939+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.469793+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:44:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.487510+0000 mgr.y (mgr.14553) 473 : cluster [DBG] pgmap v799: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:38.559 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:38.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:38.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:38.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:39.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:39.081 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:44:39.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:44:39.082 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-10T08:44:39.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:44:39.095 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:39.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:39.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:39.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:39.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.421982+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.a scrub starts 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:37.425665+0000 osd.6 (osd.6) 233 : cluster [DBG] 4.a scrub ok 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.302223+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: Degraded data redundancy: 188/1152 objects degraded (16.319%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:38.316871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.316933+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e532: 8 total, 7 up, 7 in 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:38.327978+0000 mon.b (mon.1) 665 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:38.342975+0000 mon.a (mon.0) 2799 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.403054+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:44:39.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.409479+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.421982+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.a scrub starts 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:37.425665+0000 osd.6 (osd.6) 233 : cluster [DBG] 4.a scrub ok 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.302223+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: Degraded data redundancy: 188/1152 objects degraded (16.319%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:38.316871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.316933+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e532: 8 total, 7 up, 7 in 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:38.327978+0000 mon.b (mon.1) 665 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:38.342975+0000 mon.a (mon.0) 2799 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.403054+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:44:39.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.409479+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.421982+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.a scrub starts 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:37.425665+0000 osd.6 (osd.6) 233 : cluster [DBG] 4.a scrub ok 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.302223+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: Degraded data redundancy: 188/1152 objects degraded (16.319%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:38.316871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.316933+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e532: 8 total, 7 up, 7 in 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:38.327978+0000 mon.b (mon.1) 665 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:38.342975+0000 mon.a (mon.0) 2799 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]: dispatch 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.403054+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 deep-scrub starts 2024-02-10T08:44:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.409479+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 deep-scrub ok 2024-02-10T08:44:40.102 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:44:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:44:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:44:40.102 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:44:40.409 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.730380+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:38.734152+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.307610+0000 mon.a (mon.0) 2800 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:39.314262+0000 mon.a (mon.0) 2801 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]': finished 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.314326+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:39.317949+0000 mon.b (mon.1) 666 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:39.318544+0000 mon.a (mon.0) 2803 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.335121+0000 osd.5 (osd.5) 235 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.356495+0000 osd.5 (osd.5) 236 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.405290+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.424463+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.488122+0000 mgr.y (mgr.14553) 474 : cluster [DBG] pgmap v802: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.595075+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.597962+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:44:40.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:40.167661+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:40.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:40.314022+0000 mon.a (mon.0) 2805 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:40.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:40.314102+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-10T08:44:40.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:40.316154+0000 mon.b (mon.1) 667 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:40.316466+0000 mon.a (mon.0) 2807 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.730380+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:38.734152+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.307610+0000 mon.a (mon.0) 2800 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:39.314262+0000 mon.a (mon.0) 2801 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]': finished 2024-02-10T08:44:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.314326+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:39.317949+0000 mon.b (mon.1) 666 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:39.318544+0000 mon.a (mon.0) 2803 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.335121+0000 osd.5 (osd.5) 235 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.356495+0000 osd.5 (osd.5) 236 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.405290+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.424463+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.488122+0000 mgr.y (mgr.14553) 474 : cluster [DBG] pgmap v802: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.595075+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.597962+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:40.167661+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:40.314022+0000 mon.a (mon.0) 2805 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:40.314102+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:40.316154+0000 mon.b (mon.1) 667 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:40.316466+0000 mon.a (mon.0) 2807 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.730380+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:44:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:38.734152+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:44:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.307610+0000 mon.a (mon.0) 2800 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:39.314262+0000 mon.a (mon.0) 2801 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-88", "mode": "writeback"}]': finished 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.314326+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:39.317949+0000 mon.b (mon.1) 666 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:39.318544+0000 mon.a (mon.0) 2803 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.335121+0000 osd.5 (osd.5) 235 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.356495+0000 osd.5 (osd.5) 236 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.405290+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.424463+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.488122+0000 mgr.y (mgr.14553) 474 : cluster [DBG] pgmap v802: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 188/1152 objects degraded (16.319%) 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.595075+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.597962+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:40.167661+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:40.314022+0000 mon.a (mon.0) 2805 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:40.314102+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2024-02-10T08:44:40.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:40.316154+0000 mon.b (mon.1) 667 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:40.316466+0000 mon.a (mon.0) 2807 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 -- 172.21.15.107:0/1614874643 >> v1:172.21.15.107:6790/0 conn(0x7f48080ff5b0 legacy=0x7f48081019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 -- 172.21.15.107:0/1614874643 shutdown_connections 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 -- 172.21.15.107:0/1614874643 >> 172.21.15.107:0/1614874643 conn(0x7f48080f2390 msgr2=0x7f48080f47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 -- 172.21.15.107:0/1614874643 shutdown_connections 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 -- 172.21.15.107:0/1614874643 wait complete. 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 librados: starting msgr at 2024-02-10T08:44:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.942+0000 7f480d001700 1 librados: starting objecter 2024-02-10T08:44:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 Processor -- start 2024-02-10T08:44:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 -- start start 2024-02-10T08:44:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 librados: setting wanted keys 2024-02-10T08:44:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 librados: calling monclient init 2024-02-10T08:44:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48081175b0 con 0x7f48081030f0 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4808118750 con 0x7f48080fc7b0 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480d001700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48081198f0 con 0x7f48080ff5b0 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480759e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f48081030f0 0x7f4808115e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43040/0 (socket says 172.21.15.107:43040) 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480759e700 1 -- 172.21.15.107:0/3568663198 learned_addr learned my addr 172.21.15.107:0/3568663198 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f480659c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f48080ff5b0 0x7f4808112760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35112/0 (socket says 172.21.15.107:35112) 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f47fffff700 10 client.?.objecter ms_handle_connect 0x7f48080ff5b0 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f47fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f47fffff700 10 client.?.objecter ms_handle_connect 0x7f48081030f0 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.943+0000 7f47fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:40.945 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582220205 0 0) 0x7f48081198f0 con 0x7f48080ff5b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f4003800 con 0x7f48080ff5b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 10 client.?.objecter ms_handle_connect 0x7f48080fc7b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137522906 0 0) 0x7f48081175b0 con 0x7f48081030f0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48081198f0 con 0x7f48081030f0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476838789 0 0) 0x7f4808118750 con 0x7f48080fc7b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48081175b0 con 0x7f48080fc7b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160911804 0 0) 0x7f47f4003800 con 0x7f48080ff5b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4808118750 con 0x7f48080ff5b0 2024-02-10T08:44:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47f0002f10 con 0x7f48080ff5b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715896581 0 0) 0x7f48081198f0 con 0x7f48081030f0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f4003800 con 0x7f48081030f0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4800001d70 con 0x7f48081030f0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036351834 0 0) 0x7f48081175b0 con 0x7f48080fc7b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48081198f0 con 0x7f48080fc7b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47f80018c0 con 0x7f48080fc7b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3625771285 0 0) 0x7f4808118750 con 0x7f48080ff5b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 >> v1:172.21.15.195:6789/0 conn(0x7f48080fc7b0 legacy=0x7f4808060ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 >> v1:172.21.15.107:6789/0 conn(0x7f48081030f0 legacy=0x7f4808115e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480811aa90 con 0x7f48080ff5b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48081177a0 con 0x7f48080ff5b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f480d001700 10 client.53885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4808117c10 con 0x7f48080ff5b0 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f480d001700 1 librados: init done 2024-02-10T08:44:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f47f0001a80 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.944+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47f0003770 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.945+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f47e8004f00 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.946+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f47f0002c50 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(534..534 src has 420..534) v4 ==== 8181+0+0 (unknown 2169367034 0 0) 0x7f47f0078ec0 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 osd_map(534..534 src has 420..534) v4 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 3 client.53885.objecter handle_osd_map got epochs [534,534] > 0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 3 client.53885.objecter handle_osd_map decoding full epoch 534 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 20 client.53885.objecter dump_active .. 0 homeless 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v534) v1 ==== 54+0+16 (unknown 2126255517 0 2420000192) 0x7f47f00792c0 con 0x7f48080ff5b0 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47f4063c80 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 10 client.53885.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 1 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 1 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 0 1 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 4 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 10 client.53885.objecter _send_command 1 2024-02-10T08:44:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e8004950 con 0x7f47e8004530 2024-02-10T08:44:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:40.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.947+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e8004530 2024-02-10T08:44:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.948+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e8004950 con 0x7f47e8004530 2024-02-10T08:44:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.948+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.948+0000 7f47fffff700 10 client.53885.objecter _finish_command 1 = system:0 2024-02-10T08:44:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.948+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 4 2024-02-10T08:44:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.948+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 0 1 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 10 client.53885.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 2 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 2 2024-02-10T08:44:40.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e80051a0 osd=0 3 2024-02-10T08:44:40.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 0 2 2024-02-10T08:44:40.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 4 2024-02-10T08:44:40.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 10 client.53885.objecter _send_command 2 2024-02-10T08:44:40.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.963+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e8004d30 con 0x7f47e8004530 2024-02-10T08:44:40.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.964+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e8004d30 con 0x7f47e8004530 2024-02-10T08:44:40.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.964+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:40.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.964+0000 7f47fffff700 10 client.53885.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:40.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.964+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e80051a0 osd=0 4 2024-02-10T08:44:40.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.964+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 0 2 2024-02-10T08:44:40.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 10 client.53885.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 1 3 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 4 2024-02-10T08:44:40.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 10 client.53885.objecter _send_command 3 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e8010380 con 0x7f47e8003b40 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e8003b40 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e8010380 con 0x7f47e8003b40 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 10 client.53885.objecter _finish_command 3 = system:0 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 4 2024-02-10T08:44:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.965+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 1 3 2024-02-10T08:44:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 10 client.53885.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 4 2024-02-10T08:44:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 4 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8004270 osd=1 3 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 1 4 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 4 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 10 client.53885.objecter _send_command 4 2024-02-10T08:44:40.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.980+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e80116a0 con 0x7f47e8003b40 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.981+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e80116a0 con 0x7f47e8003b40 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.981+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.981+0000 7f47fffff700 10 client.53885.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.981+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8004270 osd=1 4 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.981+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 1 4 2024-02-10T08:44:40.982 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 10 client.53885.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 5 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 5 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 2 5 2024-02-10T08:44:40.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 4 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 10 client.53885.objecter _send_command 5 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e8015260 con 0x7f47e8002070 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e8002070 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e8015260 con 0x7f47e8002070 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 10 client.53885.objecter _finish_command 5 = system:0 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 4 2024-02-10T08:44:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.982+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 2 5 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 10 client.53885.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 6 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 6 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e80118f0 osd=2 3 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 2 6 2024-02-10T08:44:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 4 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 10 client.53885.objecter _send_command 6 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.997+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e8001690 con 0x7f47e8002070 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e8001690 con 0x7f47e8002070 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f47fffff700 10 client.53885.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e80118f0 osd=2 4 2024-02-10T08:44:40.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 2 6 2024-02-10T08:44:41.000 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:41.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 10 client.53885.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:41.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 7 2024-02-10T08:44:41.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 7 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 3 7 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 4 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 10 client.53885.objecter _send_command 7 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.998+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e80114f0 con 0x7f47e8016550 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e8016550 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e80114f0 con 0x7f47e8016550 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 10 client.53885.objecter _finish_command 7 = system:0 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 4 2024-02-10T08:44:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:40.999+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 3 7 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 10 client.53885.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 8 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 8 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8010cf0 osd=3 3 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 3 8 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 4 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 10 client.53885.objecter _send_command 8 2024-02-10T08:44:41.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.014+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e8010760 con 0x7f47e8016550 2024-02-10T08:44:41.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e8010760 con 0x7f47e8016550 2024-02-10T08:44:41.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:41.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f47fffff700 10 client.53885.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8010cf0 osd=3 4 2024-02-10T08:44:41.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 3 8 2024-02-10T08:44:41.016 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 10 client.53885.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 9 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 20 client.53885.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 10 client.53885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=535}) v3 -- 0x7f47e8010790 con 0x7f48080ff5b0 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.015+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47e8010ad0 con 0x7f48080ff5b0 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=534) v2 ==== 24+0+0 (unknown 882609293 0 0) 0x7f47f004c340 con 0x7f48080ff5b0 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480559a700 10 client.53885.objecter _check_command_map_dne tid 9 current 534 map_dne_bound 534 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480559a700 10 client.53885.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480559a700 15 client.53885.objecter _session_command_op_remove -1 9 2024-02-10T08:44:41.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 10 client.53885.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 10 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 10 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 5 10 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 4 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 10 client.53885.objecter _send_command 10 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e801abe0 con 0x7f47e801a060 2024-02-10T08:44:41.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.016+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e801a060 2024-02-10T08:44:41.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.017+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e801abe0 con 0x7f47e801a060 2024-02-10T08:44:41.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.017+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:41.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.017+0000 7f47fffff700 10 client.53885.objecter _finish_command 10 = system:0 2024-02-10T08:44:41.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.017+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 4 2024-02-10T08:44:41.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.017+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 5 10 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 10 client.53885.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 11 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 11 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8015e40 osd=5 3 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 5 11 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 4 2024-02-10T08:44:41.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 10 client.53885.objecter _send_command 11 2024-02-10T08:44:41.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.032+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e8015c00 con 0x7f47e801a060 2024-02-10T08:44:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e8015c00 con 0x7f47e801a060 2024-02-10T08:44:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 10 client.53885.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8015e40 osd=5 4 2024-02-10T08:44:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 5 11 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 10 client.53885.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 12 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 12 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 6 12 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 4 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 10 client.53885.objecter _send_command 12 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e8023390 con 0x7f47e801f880 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.034+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e801f880 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.035+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e8023390 con 0x7f47e801f880 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.035+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.035+0000 7f47fffff700 10 client.53885.objecter _finish_command 12 = system:0 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.035+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 4 2024-02-10T08:44:41.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.035+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 6 12 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 10 client.53885.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 13 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 13 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8015c00 osd=6 3 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 6 13 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 4 2024-02-10T08:44:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 10 client.53885.objecter _send_command 13 2024-02-10T08:44:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.050+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e8023390 con 0x7f47e801f880 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e8023390 con 0x7f47e801f880 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 10 client.53885.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8015c00 osd=6 4 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 6 13 2024-02-10T08:44:41.052 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 10 client.53885.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 14 2024-02-10T08:44:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 14 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 7 14 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 4 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 10 client.53885.objecter _send_command 14 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47e80245b0 con 0x7f47e8025090 2024-02-10T08:44:41.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.052+0000 7f47fffff700 10 client.53885.objecter ms_handle_connect 0x7f47e8025090 2024-02-10T08:44:41.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.053+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47e80245b0 con 0x7f47e8025090 2024-02-10T08:44:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.053+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.053+0000 7f47fffff700 10 client.53885.objecter _finish_command 14 = system:0 2024-02-10T08:44:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.053+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 4 2024-02-10T08:44:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.053+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 7 14 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 10 client.53885.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign -1 15 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 20 client.53885.objecter _get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 15 client.53885.objecter _session_command_op_remove -1 15 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 20 client.53885.objecter get_session s=0x7f47e8023390 osd=7 3 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 15 client.53885.objecter _session_command_op_assign 7 15 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 4 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 10 client.53885.objecter _send_command 15 2024-02-10T08:44:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.068+0000 7f480d001700 1 -- 172.21.15.107:0/3568663198 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47e80245b0 con 0x7f47e8025090 2024-02-10T08:44:41.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.070+0000 7f47fffff700 1 -- 172.21.15.107:0/3568663198 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47e80245b0 con 0x7f47e8025090 2024-02-10T08:44:41.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.070+0000 7f47fffff700 10 client.53885.objecter ms_dispatch 0x7f4808001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:44:41.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.070+0000 7f47fffff700 10 client.53885.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.070+0000 7f47fffff700 20 client.53885.objecter put_session s=0x7f47e8023390 osd=7 4 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:41.070+0000 7f47fffff700 15 client.53885.objecter _session_command_op_remove 7 15 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:44:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:41.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.695371+0000 osd.1 (osd.1) 262 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:44:41.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:39.698908+0000 osd.1 (osd.1) 263 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:44:41.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:40.547294+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:40.551659+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:40.946628+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.107:0/3568663198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.079028+0000 mon.a (mon.0) 2808 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:41.083358+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.083409+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:41.085057+0000 mon.b (mon.1) 668 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.406 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:41.092608+0000 mon.a (mon.0) 2811 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.695371+0000 osd.1 (osd.1) 262 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:44:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:39.698908+0000 osd.1 (osd.1) 263 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:44:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:40.547294+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:44:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:40.551659+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:40.946628+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.107:0/3568663198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.079028+0000 mon.a (mon.0) 2808 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:41.083358+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.083409+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:41.085057+0000 mon.b (mon.1) 668 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:41.092608+0000 mon.a (mon.0) 2811 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.695371+0000 osd.1 (osd.1) 262 : cluster [DBG] 60.1a scrub starts 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:39.698908+0000 osd.1 (osd.1) 263 : cluster [DBG] 60.1a scrub ok 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:40.547294+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:40.551659+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:40.946628+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.107:0/3568663198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.079028+0000 mon.a (mon.0) 2808 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:41.083358+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:44:41.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.083409+0000 mon.a (mon.0) 2810 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2024-02-10T08:44:41.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:41.085057+0000 mon.b (mon.1) 668 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:41.092608+0000 mon.a (mon.0) 2811 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T08:44:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:42.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:42.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:42.193 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:42.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:40.719625+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:44:42.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:40.723092+0000 osd.1 (osd.1) 265 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:44:42.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.384129+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:44:42.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.392405+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.489052+0000 mgr.y (mgr.14553) 475 : cluster [DBG] pgmap v805: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:41.594673+0000 mgr.y (mgr.14553) 476 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.615801+0000 osd.2 (osd.2) 209 : cluster [DBG] 60.b scrub starts 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.619361+0000 osd.2 (osd.2) 210 : cluster [DBG] 60.b scrub ok 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:42.084126+0000 mon.a (mon.0) 2812 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]': finished 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:42.084193+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:42.088847+0000 mon.b (mon.1) 669 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:42.094901+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:40.719625+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:40.723092+0000 osd.1 (osd.1) 265 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.384129+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:44:42.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.392405+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.489052+0000 mgr.y (mgr.14553) 475 : cluster [DBG] pgmap v805: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:41.594673+0000 mgr.y (mgr.14553) 476 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.615801+0000 osd.2 (osd.2) 209 : cluster [DBG] 60.b scrub starts 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.619361+0000 osd.2 (osd.2) 210 : cluster [DBG] 60.b scrub ok 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:42.084126+0000 mon.a (mon.0) 2812 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]': finished 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:42.084193+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:42.088847+0000 mon.b (mon.1) 669 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:42.094901+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:42.675 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:40.719625+0000 osd.1 (osd.1) 264 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:40.723092+0000 osd.1 (osd.1) 265 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.384129+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.392405+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.489052+0000 mgr.y (mgr.14553) 475 : cluster [DBG] pgmap v805: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:41.594673+0000 mgr.y (mgr.14553) 476 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.615801+0000 osd.2 (osd.2) 209 : cluster [DBG] 60.b scrub starts 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.619361+0000 osd.2 (osd.2) 210 : cluster [DBG] 60.b scrub ok 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:42.084126+0000 mon.a (mon.0) 2812 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_count","val": "1"}]': finished 2024-02-10T08:44:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:42.084193+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2024-02-10T08:44:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:42.088847+0000 mon.b (mon.1) 669 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:42.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:42.094901+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:43.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:44:43.379 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:43.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:43.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:43.586 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:43.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.727584+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.a scrub starts 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:41.730473+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.a scrub ok 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:43.085903+0000 mon.a (mon.0) 2815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:43.085975+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:43.097043+0000 mon.b (mon.1) 670 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:43.097466+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.727584+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.a scrub starts 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:41.730473+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.a scrub ok 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:43.085903+0000 mon.a (mon.0) 2815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:43.085975+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:43.097043+0000 mon.b (mon.1) 670 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:43.097466+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.727584+0000 osd.1 (osd.1) 266 : cluster [DBG] 2.a scrub starts 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:41.730473+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.a scrub ok 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:43.085903+0000 mon.a (mon.0) 2815 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:43.085975+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:43.097043+0000 mon.b (mon.1) 670 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:43.097466+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T08:44:43.718 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:43.736 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:43.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:43.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.816+0000 7f812f808700 1 -- 172.21.15.107:0/616440229 >> v1:172.21.15.107:6789/0 conn(0x7f8128104340 legacy=0x7f81281067e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.816+0000 7f812f808700 1 -- 172.21.15.107:0/616440229 shutdown_connections 2024-02-10T08:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.816+0000 7f812f808700 1 -- 172.21.15.107:0/616440229 >> 172.21.15.107:0/616440229 conn(0x7f81280f2510 msgr2=0x7f81280f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- 172.21.15.107:0/616440229 shutdown_connections 2024-02-10T08:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- 172.21.15.107:0/616440229 wait complete. 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 librados: starting msgr at 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 librados: starting objecter 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 Processor -- start 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- start start 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 librados: setting wanted keys 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 librados: calling monclient init 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f81281107e0 con 0x7f81280f8a40 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f812811a870 con 0x7f8128104340 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812f808700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f812811ba50 con 0x7f81280f9530 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812d5a4700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f81280f8a40 0x7f812810fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43052/0 (socket says 172.21.15.107:43052) 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f812d5a4700 1 -- 172.21.15.107:0/722229720 learned_addr learned my addr 172.21.15.107:0/722229720 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.817+0000 7f811e7fc700 10 client.?.objecter ms_handle_connect 0x7f81280f8a40 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 10 client.?.objecter ms_handle_connect 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 10 client.?.objecter ms_handle_connect 0x7f81280f9530 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177094970 0 0) 0x7f81281107e0 con 0x7f81280f8a40 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81140038e0 con 0x7f81280f8a40 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142166239 0 0) 0x7f812811a870 con 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81281107e0 con 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058030909 0 0) 0x7f812811ba50 con 0x7f81280f9530 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812811a870 con 0x7f81280f9530 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 258219253 0 0) 0x7f81281107e0 con 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f812811ba50 con 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f81200021d0 con 0x7f8128104340 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015901650 0 0) 0x7f81140038e0 con 0x7f81280f8a40 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81281107e0 con 0x7f81280f8a40 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f81240019a0 con 0x7f81280f8a40 2024-02-10T08:44:43.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 949854196 0 0) 0x7f812811a870 con 0x7f81280f9530 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81140038e0 con 0x7f81280f9530 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f81180018c0 con 0x7f81280f9530 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 489183011 0 0) 0x7f812811ba50 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 >> v1:172.21.15.107:6790/0 conn(0x7f81280f9530 legacy=0x7f81281149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 >> v1:172.21.15.107:6789/0 conn(0x7f81280f8a40 legacy=0x7f812810fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812811cc50 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812811aa60 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8120002510 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.818+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8120003a70 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.819+0000 7f812f808700 10 client.48607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.819+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f812811aed0 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.819+0000 7f812f808700 1 librados: init done 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.819+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812805f7a0 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8120001ac0 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(537..537 src has 420..537) v4 ==== 8207+0+0 (unknown 1015580157 0 0) 0x7f8120076ad0 con 0x7f8128104340 2024-02-10T08:44:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 10 client.48607.objecter ms_dispatch 0x7f8128001550 osd_map(537..537 src has 420..537) v4 2024-02-10T08:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 3 client.48607.objecter handle_osd_map got epochs [537,537] > 0 2024-02-10T08:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 3 client.48607.objecter handle_osd_map decoding full epoch 537 2024-02-10T08:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 20 client.48607.objecter dump_active .. 0 homeless 2024-02-10T08:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.820+0000 7f811e7fc700 10 client.48607.objecter ms_handle_connect 0x7f8114063d60 2024-02-10T08:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:43.824+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8120049e30 con 0x7f8128104340 2024-02-10T08:44:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.006+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f81280475e0 con 0x7f8128104340 2024-02-10T08:44:44.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.088+0000 7f811e7fc700 1 -- 172.21.15.107:0/722229720 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v538) v1 ==== 87+0+0 (unknown 3247448932 0 0) 0x7f81200482a0 con 0x7f8128104340 2024-02-10T08:44:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.121+0000 7f812f808700 10 librados: watch_flush enter 2024-02-10T08:44:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.121+0000 7f812f808700 10 librados: watch_flush exit 2024-02-10T08:44:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.121+0000 7f812f808700 20 client.48607.objecter shutdown clearing up homeless session... 2024-02-10T08:44:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.121+0000 7f812f808700 10 client.48607.objecter successfully canceled tick 2024-02-10T08:44:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.133+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8114063d60 legacy=0x7f8114066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.133+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 >> v1:172.21.15.195:6789/0 conn(0x7f8128104340 legacy=0x7f8128118100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.155+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 shutdown_connections 2024-02-10T08:44:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.155+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 >> 172.21.15.107:0/722229720 conn(0x7f81280f2510 msgr2=0x7f81281077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.171+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 shutdown_connections 2024-02-10T08:44:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.172+0000 7f812f808700 1 -- 172.21.15.107:0/722229720 wait complete. 2024-02-10T08:44:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:44.174+0000 7f812f808700 1 librados: shutdown 2024-02-10T08:44:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:43.489717+0000 mgr.y (mgr.14553) 477 : cluster [DBG] pgmap v808: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.007911+0000 mon.b (mon.1) 671 : audit [INF] from='client.? v1:172.21.15.107:0/722229720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.008175+0000 mon.a (mon.0) 2818 : audit [INF] from='client.48607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:44.083132+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.086880+0000 mon.a (mon.0) 2820 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]': finished 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.086959+0000 mon.a (mon.0) 2821 : audit [INF] from='client.48607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:44.087015+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.258799+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:44.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:44.259962+0000 mon.a (mon.0) 2823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:43.489717+0000 mgr.y (mgr.14553) 477 : cluster [DBG] pgmap v808: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.007911+0000 mon.b (mon.1) 671 : audit [INF] from='client.? v1:172.21.15.107:0/722229720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.008175+0000 mon.a (mon.0) 2818 : audit [INF] from='client.48607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:44.083132+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.086880+0000 mon.a (mon.0) 2820 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]': finished 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.086959+0000 mon.a (mon.0) 2821 : audit [INF] from='client.48607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:44.087015+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.258799+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:44.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:44.259962+0000 mon.a (mon.0) 2823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:44.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:43.489717+0000 mgr.y (mgr.14553) 477 : cluster [DBG] pgmap v808: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.007911+0000 mon.b (mon.1) 671 : audit [INF] from='client.? v1:172.21.15.107:0/722229720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.008175+0000 mon.a (mon.0) 2818 : audit [INF] from='client.48607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:44.083132+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.086880+0000 mon.a (mon.0) 2820 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-88","var": "target_max_objects","val": "250"}]': finished 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.086959+0000 mon.a (mon.0) 2821 : audit [INF] from='client.48607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:44:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:44.087015+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2024-02-10T08:44:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.258799+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:44.259962+0000 mon.a (mon.0) 2823 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:45.427 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:45.674 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:45] "GET /metrics HTTP/1.1" 200 189027 "" "Prometheus/2.33.4" 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.182+0000 7f9614dcf700 1 -- 172.21.15.107:0/4216294853 >> v1:172.21.15.195:6789/0 conn(0x7f9610069490 legacy=0x7f9610069870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.185+0000 7f9614dcf700 1 -- 172.21.15.107:0/4216294853 shutdown_connections 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.185+0000 7f9614dcf700 1 -- 172.21.15.107:0/4216294853 >> 172.21.15.107:0/4216294853 conn(0x7f9610063bc0 msgr2=0x7f96100672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.187+0000 7f9614dcf700 1 -- 172.21.15.107:0/4216294853 shutdown_connections 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.187+0000 7f9614dcf700 1 -- 172.21.15.107:0/4216294853 wait complete. 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.187+0000 7f9614dcf700 1 librados: starting msgr at 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.187+0000 7f9614dcf700 1 librados: starting objecter 2024-02-10T08:44:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 Processor -- start 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 -- start start 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 librados: setting wanted keys 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 librados: calling monclient init 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9610112d60 con 0x7f9610069490 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9610112ef0 con 0x7f96101132f0 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f9614dcf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9610113080 con 0x7f9610069f80 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f960dd9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9610069f80 0x7f9610111640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35144/0 (socket says 172.21.15.107:35144) 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f960dd9b700 1 -- 172.21.15.107:0/3443509985 learned_addr learned my addr 172.21.15.107:0/3443509985 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f95ff7fe700 10 client.?.objecter ms_handle_connect 0x7f9610069f80 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f95ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.188+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685298853 0 0) 0x7f9610113080 con 0x7f9610069f80 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8003030 con 0x7f9610069f80 2024-02-10T08:44:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 10 client.?.objecter ms_handle_connect 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328849071 0 0) 0x7f9610112d60 con 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610113080 con 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162555706 0 0) 0x7f95f8003030 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9610112d60 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f96000029d0 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348061116 0 0) 0x7f9610113080 con 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f8003030 con 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9604002d80 con 0x7f9610069490 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2109803770 0 0) 0x7f9610112d60 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 >> v1:172.21.15.195:6789/0 conn(0x7f96101132f0 legacy=0x7f961011c790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 >> v1:172.21.15.107:6789/0 conn(0x7f9610069490 legacy=0x7f961010b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610120ee0 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961011deb0 con 0x7f9610069f80 2024-02-10T08:44:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f9614dcf700 10 client.53897.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f961011e320 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.189+0000 7f9614dcf700 1 librados: init done 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.190+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9600002c90 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.190+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9600005b50 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.191+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f0004f20 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9600001700 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(540..540 src has 420..540) v4 ==== 8173+0+0 (unknown 3344133405 0 0) 0x7f9600078c90 con 0x7f9610069f80 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 10 client.53897.objecter ms_dispatch 0x7f9610001550 osd_map(540..540 src has 420..540) v4 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 3 client.53897.objecter handle_osd_map got epochs [540,540] > 0 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 3 client.53897.objecter handle_osd_map decoding full epoch 540 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.192+0000 7f95ff7fe700 20 client.53897.objecter dump_active .. 0 homeless 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.194+0000 7f95ff7fe700 10 client.53897.objecter ms_handle_connect 0x7f95f8063ca0 2024-02-10T08:44:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.197+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f960004c070 con 0x7f9610069f80 2024-02-10T08:44:46.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.371+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95f0000b90 con 0x7f95f8063ca0 2024-02-10T08:44:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.386+0000 7f95ff7fe700 1 -- 172.21.15.107:0/3443509985 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+556048 (unknown 2965378022 0 1440390546) 0x7f95f0000b90 con 0x7f95f8063ca0 2024-02-10T08:44:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 10 librados: watch_flush enter 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 10 librados: watch_flush exit 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 20 client.53897.objecter shutdown clearing up homeless session... 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 10 client.53897.objecter successfully canceled tick 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 >> v1:172.21.15.107:6800/2634842009 conn(0x7f95f8063ca0 legacy=0x7f95f8066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 >> v1:172.21.15.107:6790/0 conn(0x7f9610069f80 legacy=0x7f9610111640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 shutdown_connections 2024-02-10T08:44:46.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 >> 172.21.15.107:0/3443509985 conn(0x7f9610063bc0 msgr2=0x7f96101136d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 shutdown_connections 2024-02-10T08:44:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 -- 172.21.15.107:0/3443509985 wait complete. 2024-02-10T08:44:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:46.390+0000 7f9614dcf700 1 librados: shutdown 2024-02-10T08:44:46.401 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:45.088595+0000 mon.a (mon.0) 2824 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:45.088661+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:45.090210+0000 mon.b (mon.1) 673 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:45.098132+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:45.490367+0000 mgr.y (mgr.14553) 478 : cluster [DBG] pgmap v811: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:46.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:46.080047+0000 mon.a (mon.0) 2827 : cluster [WRN] Health check update: Degraded data redundancy: 189/1155 objects degraded (16.364%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:45.088595+0000 mon.a (mon.0) 2824 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:45.088661+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:45.090210+0000 mon.b (mon.1) 673 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:45.098132+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:45.490367+0000 mgr.y (mgr.14553) 478 : cluster [DBG] pgmap v811: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:46.412 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:46.080047+0000 mon.a (mon.0) 2827 : cluster [WRN] Health check update: Degraded data redundancy: 189/1155 objects degraded (16.364%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:45.088595+0000 mon.a (mon.0) 2824 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:45.088661+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:45.090210+0000 mon.b (mon.1) 673 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:45.098132+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]: dispatch 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:45.490367+0000 mgr.y (mgr.14553) 478 : cluster [DBG] pgmap v811: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 54 active+undersized, 125 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:46.080047+0000 mon.a (mon.0) 2827 : cluster [WRN] Health check update: Degraded data redundancy: 189/1155 objects degraded (16.364%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:47.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:44:47.350 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:47.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:47.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:47.489 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:47.509 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:47.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:47.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:47.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:46.098297+0000 mon.a (mon.0) 2828 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:47.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:46.098387+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-10T08:44:47.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:46.372273+0000 mgr.y (mgr.14553) 479 : audit [DBG] from='client.53897 v1:172.21.15.107:0/3443509985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:47.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:46.098297+0000 mon.a (mon.0) 2828 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:47.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:46.098387+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-10T08:44:47.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:46.372273+0000 mgr.y (mgr.14553) 479 : audit [DBG] from='client.53897 v1:172.21.15.107:0/3443509985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:47.669 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:47.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:47.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:47.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:46.098297+0000 mon.a (mon.0) 2828 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-88"}]': finished 2024-02-10T08:44:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:46.098387+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e540: 8 total, 7 up, 7 in 2024-02-10T08:44:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:46.372273+0000 mgr.y (mgr.14553) 479 : audit [DBG] from='client.53897 v1:172.21.15.107:0/3443509985' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:44:47.804 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:47.825 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:47.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:47.875 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:44:47.875 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:44:47.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:47.959 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:47.978 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:47.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:48.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.195 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:48.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:48.324 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.346 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:48.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:48.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:47.096417+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-10T08:44:48.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:47.491015+0000 mgr.y (mgr.14553) 480 : cluster [DBG] pgmap v814: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:48.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:48.097670+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-10T08:44:48.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:48.122174+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:48.126956+0000 mon.a (mon.0) 2832 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.494 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:48.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:48.624 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.642 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:48.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:47.096417+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-10T08:44:48.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:47.491015+0000 mgr.y (mgr.14553) 480 : cluster [DBG] pgmap v814: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:48.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:48.097670+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:48.122174+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:48.126956+0000 mon.a (mon.0) 2832 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:47.096417+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e541: 8 total, 7 up, 7 in 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:47.491015+0000 mgr.y (mgr.14553) 480 : cluster [DBG] pgmap v814: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:48.097670+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e542: 8 total, 7 up, 7 in 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:48.122174+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:48.126956+0000 mon.a (mon.0) 2832 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:44:48.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:48.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:44:48.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:48.927 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:48.946 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:48.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:49.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:49.078 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.097 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:49.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:49.242 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:49.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:49.424 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.442 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:49.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:49.573 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.626 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:49.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:49.757 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.775 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:49.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:49.905 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:49.923 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:49.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:50.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:50.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2024-02-10T08:44:46.095054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4026236,"num_objects":385,"num_object_clones":11,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":2025,"num_read_kb":1625,"num_write":1499,"num_write_kb":4667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":727,"num_bytes_recovered":7992,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2865,"ondisk_log_size":2865,"up":561,"acting":562,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":654,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2170372,"kb_used_data":31344,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":654045692,"statfs":{"total":671965249536,"available":669742788608,"internally_reserved":0,"allocated":32096256,"data_stored":20187249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012050"},"pg_stats":[{"pgid":"226.1f","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.183404+0000","last_change":"2024-02-10T08:44:37.338552+0000","last_active":"2024-02-10T08:44:44.183404+0000","last_peered":"2024-02-10T08:44:44.183404+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.338552+0000","last_became_peered":"2024-02-10T08:44:37.338552+0000","last_unstale":"2024-02-10T08:44:44.183404+0000","last_undegraded":"2024-02-10T08:44:44.183404+0000","last_fullsized":"2024-02-10T08:44:37.302857+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.1e","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.181885+0000","last_change":"2024-02-10T08:44:37.320761+0000","last_active":"2024-02-10T08:44:44.181885+0000","last_peered":"2024-02-10T08:44:44.181885+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.320761+0000","last_became_peered":"2024-02-10T08:44:37.320761+0000","last_unstale":"2024-02-10T08:44:44.181885+0000","last_undegraded":"2024-02-10T08:44:44.181885+0000","last_fullsized":"2024-02-10T08:44:37.302764+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.1d","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.182124+0000","last_change":"2024-02-10T08:44:37.328376+0000","last_active":"2024-02-10T08:44:44.182124+0000","last_peered":"2024-02-10T08:44:44.182124+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.328376+0000","last_became_peered":"2024-02-10T08:44:37.328376+0000","last_unstale":"2024-02-10T08:44:44.182124+0000","last_undegraded":"2024-02-10T08:44:44.182124+0000","last_fullsized":"2024-02-10T08:44:37.301687+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.1c","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.086191+0000","last_change":"2024-02-10T08:44:37.307048+0000","last_active":"2024-02-10T08:44:44.086191+0000","last_peered":"2024-02-10T08:44:44.086191+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.307048+0000","last_became_peered":"2024-02-10T08:44:37.307048+0000","last_unstale":"2024-02-10T08:44:44.086191+0000","last_undegraded":"2024-02-10T08:44:44.086191+0000","last_fullsized":"2024-02-10T08:44:37.297754+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.1b","version":"538'2","reported_seq":20,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.258082+0000","last_change":"2024-02-10T08:44:37.325494+0000","last_active":"2024-02-10T08:44:44.258082+0000","last_peered":"2024-02-10T08:44:44.258082+0000","last_clean":"2024-02-10T08:44:44.258082+0000","last_became_active":"2024-02-10T08:44:37.325209+0000","last_became_peered":"2024-02-10T08:44:37.325209+0000","last_unstale":"2024-02-10T08:44:44.258082+0000","last_undegraded":"2024-02-10T08:44:44.258082+0000","last_fullsized":"2024-02-10T08:44:44.258082+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.1a","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.136061+0000","last_change":"2024-02-10T08:44:37.378843+0000","last_active":"2024-02-10T08:44:44.136061+0000","last_peered":"2024-02-10T08:44:44.136061+0000","last_clean":"2024-02-10T08:44:44.136061+0000","last_became_active":"2024-02-10T08:44:37.378542+0000","last_became_peered":"2024-02-10T08:44:37.378542+0000","last_unstale":"2024-02-10T08:44:44.136061+0000","last_undegraded":"2024-02-10T08:44:44.136061+0000","last_fullsized":"2024-02-10T08:44:44.136061+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.19","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.135213+0000","last_change":"2024-02-10T08:44:37.333985+0000","last_active":"2024-02-10T08:44:44.135213+0000","last_peered":"2024-02-10T08:44:44.135213+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.333985+0000","last_became_peered":"2024-02-10T08:44:37.333985+0000","last_unstale":"2024-02-10T08:44:44.135213+0000","last_undegraded":"2024-02-10T08:44:44.135213+0000","last_fullsized":"2024-02-10T08:44:37.312702+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.18","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.085915+0000","last_change":"2024-02-10T08:44:37.325890+0000","last_active":"2024-02-10T08:44:44.085915+0000","last_peered":"2024-02-10T08:44:44.085915+0000","last_clean":"2024-02-10T08:44:44.085915+0000","last_became_active":"2024-02-10T08:44:37.325129+0000","last_became_peered":"2024-02-10T08:44:37.325129+0000","last_unstale":"2024-02-10T08:44:44.085915+0000","last_undegraded":"2024-02-10T08:44:44.085915+0000","last_fullsized":"2024-02-10T08:44:44.085915+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.17","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.183650+0000","last_change":"2024-02-10T08:44:37.327223+0000","last_active":"2024-02-10T08:44:44.183650+0000","last_peered":"2024-02-10T08:44:44.183650+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.327223+0000","last_became_peered":"2024-02-10T08:44:37.327223+0000","last_unstale":"2024-02-10T08:44:44.183650+0000","last_undegraded":"2024-02-10T08:44:44.183650+0000","last_fullsized":"2024-02-10T08:44:37.302519+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.16","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.315108+0000","last_change":"2024-02-10T08:44:37.317842+0000","last_active":"2024-02-10T08:44:40.315108+0000","last_peered":"2024-02-10T08:44:40.315108+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.317842+0000","last_became_peered":"2024-02-10T08:44:37.317842+0000","last_unstale":"2024-02-10T08:44:40.315108+0000","last_undegraded":"2024-02-10T08:44:40.315108+0000","last_fullsized":"2024-02-10T08:44:37.298328+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"226.15","version":"0'0","reported_seq":13,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315052+0000","last_change":"2024-02-10T08:44:37.324294+0000","last_active":"2024-02-10T08:44:40.315052+0000","last_peered":"2024-02-10T08:44:40.315052+0000","last_clean":"2024-02-10T08:44:40.315052+0000","last_became_active":"2024-02-10T08:44:37.324029+0000","last_became_peered":"2024-02-10T08:44:37.324029+0000","last_unstale":"2024-02-10T08:44:40.315052+0000","last_undegraded":"2024-02-10T08:44:40.315052+0000","last_fullsized":"2024-02-10T08:44:40.315052+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"226.14","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086258+0000","last_change":"2024-02-10T08:44:37.326122+0000","last_active":"2024-02-10T08:44:44.086258+0000","last_peered":"2024-02-10T08:44:44.086258+0000","last_clean":"2024-02-10T08:44:44.086258+0000","last_became_active":"2024-02-10T08:44:37.325828+0000","last_became_peered":"2024-02-10T08:44:37.325828+0000","last_unstale":"2024-02-10T08:44:44.086258+0000","last_undegraded":"2024-02-10T08:44:44.086258+0000","last_fullsized":"2024-02-10T08:44:44.086258+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.13","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.184624+0000","last_change":"2024-02-10T08:44:37.336203+0000","last_active":"2024-02-10T08:44:44.184624+0000","last_peered":"2024-02-10T08:44:44.184624+0000","last_clean":"2024-02-10T08:44:44.184624+0000","last_became_active":"2024-02-10T08:44:37.335931+0000","last_became_peered":"2024-02-10T08:44:37.335931+0000","last_unstale":"2024-02-10T08:44:44.184624+0000","last_undegraded":"2024-02-10T08:44:44.184624+0000","last_fullsized":"2024-02-10T08:44:44.184624+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.12","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.086037+0000","last_change":"2024-02-10T08:44:37.315296+0000","last_active":"2024-02-10T08:44:44.086037+0000","last_peered":"2024-02-10T08:44:44.086037+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.315296+0000","last_became_peered":"2024-02-10T08:44:37.315296+0000","last_unstale":"2024-02-10T08:44:44.086037+0000","last_undegraded":"2024-02-10T08:44:44.086037+0000","last_fullsized":"2024-02-10T08:44:37.297142+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.11","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.180370+0000","last_change":"2024-02-10T08:44:37.333880+0000","last_active":"2024-02-10T08:44:44.180370+0000","last_peered":"2024-02-10T08:44:44.180370+0000","last_clean":"2024-02-10T08:44:44.180370+0000","last_became_active":"2024-02-10T08:44:37.333578+0000","last_became_peered":"2024-02-10T08:44:37.333578+0000","last_unstale":"2024-02-10T08:44:44.180370+0000","last_undegraded":"2024-02-10T08:44:44.180370+0000","last_fullsized":"2024-02-10T08:44:44.180370+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.10","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134065+0000","last_change":"2024-02-10T08:44:37.334186+0000","last_active":"2024-02-10T08:44:44.134065+0000","last_peered":"2024-02-10T08:44:44.134065+0000","last_clean":"2024-02-10T08:44:44.134065+0000","last_became_active":"2024-02-10T08:44:37.333870+0000","last_became_peered":"2024-02-10T08:44:37.333870+0000","last_unstale":"2024-02-10T08:44:44.134065+0000","last_undegraded":"2024-02-10T08:44:44.134065+0000","last_fullsized":"2024-02-10T08:44:44.134065+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.f","version":"0'0","reported_seq":12,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405706+0000","last_change":"2024-02-10T08:44:37.346361+0000","last_active":"2024-02-10T08:44:39.405706+0000","last_peered":"2024-02-10T08:44:39.405706+0000","last_clean":"2024-02-10T08:44:39.405706+0000","last_became_active":"2024-02-10T08:44:37.346080+0000","last_became_peered":"2024-02-10T08:44:37.346080+0000","last_unstale":"2024-02-10T08:44:39.405706+0000","last_undegraded":"2024-02-10T08:44:39.405706+0000","last_fullsized":"2024-02-10T08:44:39.405706+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"226.e","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.315637+0000","last_change":"2024-02-10T08:44:37.318524+0000","last_active":"2024-02-10T08:44:40.315637+0000","last_peered":"2024-02-10T08:44:40.315637+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.318524+0000","last_became_peered":"2024-02-10T08:44:37.318524+0000","last_unstale":"2024-02-10T08:44:40.315637+0000","last_undegraded":"2024-02-10T08:44:40.315637+0000","last_fullsized":"2024-02-10T08:44:37.298008+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"226.d","version":"0'0","reported_seq":12,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405232+0000","last_change":"2024-02-10T08:44:37.350356+0000","last_active":"2024-02-10T08:44:39.405232+0000","last_peered":"2024-02-10T08:44:39.405232+0000","last_clean":"2024-02-10T08:44:39.405232+0000","last_became_active":"2024-02-10T08:44:37.350081+0000","last_became_peered":"2024-02-10T08:44:37.350081+0000","last_unstale":"2024-02-10T08:44:39.405232+0000","last_undegraded":"2024-02-10T08:44:39.405232+0000","last_fullsized":"2024-02-10T08:44:39.405232+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"226.c","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087280+0000","last_change":"2024-02-10T08:44:37.325807+0000","last_active":"2024-02-10T08:44:44.087280+0000","last_peered":"2024-02-10T08:44:44.087280+0000","last_clean":"2024-02-10T08:44:44.087280+0000","last_became_active":"2024-02-10T08:44:37.325509+0000","last_became_peered":"2024-02-10T08:44:37.325509+0000","last_unstale":"2024-02-10T08:44:44.087280+0000","last_undegraded":"2024-02-10T08:44:44.087280+0000","last_fullsized":"2024-02-10T08:44:44.087280+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.b","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.184835+0000","last_change":"2024-02-10T08:44:37.339422+0000","last_active":"2024-02-10T08:44:44.184835+0000","last_peered":"2024-02-10T08:44:44.184835+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.339422+0000","last_became_peered":"2024-02-10T08:44:37.339422+0000","last_unstale":"2024-02-10T08:44:44.184835+0000","last_undegraded":"2024-02-10T08:44:44.184835+0000","last_fullsized":"2024-02-10T08:44:37.302347+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.a","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.136265+0000","last_change":"2024-02-10T08:44:37.320614+0000","last_active":"2024-02-10T08:44:44.136265+0000","last_peered":"2024-02-10T08:44:44.136265+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.320614+0000","last_became_peered":"2024-02-10T08:44:37.320614+0000","last_unstale":"2024-02-10T08:44:44.136265+0000","last_undegraded":"2024-02-10T08:44:44.136265+0000","last_fullsized":"2024-02-10T08:44:37.311890+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.9","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.085965+0000","last_change":"2024-02-10T08:44:37.342353+0000","last_active":"2024-02-10T08:44:44.085965+0000","last_peered":"2024-02-10T08:44:44.085965+0000","last_clean":"2024-02-10T08:44:44.085965+0000","last_became_active":"2024-02-10T08:44:37.339432+0000","last_became_peered":"2024-02-10T08:44:37.339432+0000","last_unstale":"2024-02-10T08:44:44.085965+0000","last_undegraded":"2024-02-10T08:44:44.085965+0000","last_fullsized":"2024-02-10T08:44:44.085965+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.8","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.185832+0000","last_change":"2024-02-10T08:44:37.327808+0000","last_active":"2024-02-10T08:44:44.185832+0000","last_peered":"2024-02-10T08:44:44.185832+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.327808+0000","last_became_peered":"2024-02-10T08:44:37.327808+0000","last_unstale":"2024-02-10T08:44:44.185832+0000","last_undegraded":"2024-02-10T08:44:44.185832+0000","last_fullsized":"2024-02-10T08:44:37.301405+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.7","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087248+0000","last_change":"2024-02-10T08:44:37.340204+0000","last_active":"2024-02-10T08:44:44.087248+0000","last_peered":"2024-02-10T08:44:44.087248+0000","last_clean":"2024-02-10T08:44:44.087248+0000","last_became_active":"2024-02-10T08:44:37.339895+0000","last_became_peered":"2024-02-10T08:44:37.339895+0000","last_unstale":"2024-02-10T08:44:44.087248+0000","last_undegraded":"2024-02-10T08:44:44.087248+0000","last_fullsized":"2024-02-10T08:44:44.087248+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"226.6","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.136187+0000","last_change":"2024-02-10T08:44:37.350346+0000","last_active":"2024-02-10T08:44:44.136187+0000","last_peered":"2024-02-10T08:44:44.136187+0000","last_clean":"2024-02-10T08:44:44.136187+0000","last_became_active":"2024-02-10T08:44:37.350057+0000","last_became_peered":"2024-02-10T08:44:37.350057+0000","last_unstale":"2024-02-10T08:44:44.136187+0000","last_undegraded":"2024-02-10T08:44:44.136187+0000","last_fullsized":"2024-02-10T08:44:44.136187+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.5","version":"0'0","reported_seq":13,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.314959+0000","last_change":"2024-02-10T08:44:37.317112+0000","last_active":"2024-02-10T08:44:40.314959+0000","last_peered":"2024-02-10T08:44:40.314959+0000","last_clean":"2024-02-10T08:44:40.314959+0000","last_became_active":"2024-02-10T08:44:37.316471+0000","last_became_peered":"2024-02-10T08:44:37.316471+0000","last_unstale":"2024-02-10T08:44:40.314959+0000","last_undegraded":"2024-02-10T08:44:40.314959+0000","last_fullsized":"2024-02-10T08:44:40.314959+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"226.4","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.181564+0000","last_change":"2024-02-10T08:44:37.339000+0000","last_active":"2024-02-10T08:44:44.181564+0000","last_peered":"2024-02-10T08:44:44.181564+0000","last_clean":"2024-02-10T08:44:44.181564+0000","last_became_active":"2024-02-10T08:44:37.338708+0000","last_became_peered":"2024-02-10T08:44:37.338708+0000","last_unstale":"2024-02-10T08:44:44.181564+0000","last_undegraded":"2024-02-10T08:44:44.181564+0000","last_fullsized":"2024-02-10T08:44:44.181564+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.3","version":"0'0","reported_seq":17,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134761+0000","last_change":"2024-02-10T08:44:37.350462+0000","last_active":"2024-02-10T08:44:44.134761+0000","last_peered":"2024-02-10T08:44:44.134761+0000","last_clean":"2024-02-10T08:44:44.134761+0000","last_became_active":"2024-02-10T08:44:37.350140+0000","last_became_peered":"2024-02-10T08:44:37.350140+0000","last_unstale":"2024-02-10T08:44:44.134761+0000","last_undegraded":"2024-02-10T08:44:44.134761+0000","last_fullsized":"2024-02-10T08:44:44.134761+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"226.2","version":"0'0","reported_seq":14,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.183354+0000","last_change":"2024-02-10T08:44:37.329112+0000","last_active":"2024-02-10T08:44:44.183354+0000","last_peered":"2024-02-10T08:44:44.183354+0000","last_clean":"2024-02-10T08:44:36.288217+0000","last_became_active":"2024-02-10T08:44:37.329112+0000","last_became_peered":"2024-02-10T08:44:37.329112+0000","last_unstale":"2024-02-10T08:44:44.183354+0000","last_undegraded":"2024-02-10T08:44:44.183354+0000","last_fullsized":"2024-02-10T08:44:37.301446+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"226.1","version":"0'0","reported_seq":13,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313044+0000","last_change":"2024-02-10T08:44:37.324841+0000","last_active":"2024-02-10T08:44:40.313044+0000","last_peered":"2024-02-10T08:44:40.313044+0000","last_clean":"2024-02-10T08:44:40.313044+0000","last_became_active":"2024-02-10T08:44:37.324582+0000","last_became_peered":"2024-02-10T08:44:37.324582+0000","last_unstale":"2024-02-10T08:44:40.313044+0000","last_undegraded":"2024-02-10T08:44:40.313044+0000","last_fullsized":"2024-02-10T08:44:40.313044+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"226.0","version":"0'0","reported_seq":13,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.312775+0000","last_change":"2024-02-10T08:44:37.326454+0000","last_active":"2024-02-10T08:44:40.312775+0000","last_peered":"2024-02-10T08:44:40.312775+0000","last_clean":"2024-02-10T08:44:40.312775+0000","last_became_active":"2024-02-10T08:44:37.326181+0000","last_became_peered":"2024-02-10T08:44:37.326181+0000","last_unstale":"2024-02-10T08:44:40.312775+0000","last_undegraded":"2024-02-10T08:44:40.312775+0000","last_fullsized":"2024-02-10T08:44:40.312775+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":530,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.288217+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.288217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.3","version":"493'36","reported_seq":534,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.327011+0000","last_change":"2024-02-10T08:44:26.789707+0000","last_active":"2024-02-10T08:44:40.327011+0000","last_peered":"2024-02-10T08:44:40.327011+0000","last_clean":"2024-02-10T08:44:40.327011+0000","last_became_active":"2024-02-10T08:36:50.945392+0000","last_became_peered":"2024-02-10T08:36:50.945392+0000","last_unstale":"2024-02-10T08:44:40.327011+0000","last_undegraded":"2024-02-10T08:44:40.327011+0000","last_fullsized":"2024-02-10T08:44:40.327011+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'36","last_scrub_stamp":"2024-02-10T08:44:26.789628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.789628+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.2","version":"504'37","reported_seq":612,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.314972+0000","last_change":"2024-02-10T08:43:36.076342+0000","last_active":"2024-02-10T08:44:40.314972+0000","last_peered":"2024-02-10T08:44:40.314972+0000","last_clean":"2024-02-10T08:43:31.057691+0000","last_became_active":"2024-02-10T08:43:36.076342+0000","last_became_peered":"2024-02-10T08:43:36.076342+0000","last_unstale":"2024-02-10T08:44:40.314972+0000","last_undegraded":"2024-02-10T08:43:36.061670+0000","last_fullsized":"2024-02-10T08:43:36.061548+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'29","last_scrub_stamp":"2024-02-10T08:42:28.650876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.650876+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":22,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.1","version":"401'42","reported_seq":433,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.401938+0000","last_change":"2024-02-10T08:43:36.073243+0000","last_active":"2024-02-10T08:44:39.401938+0000","last_peered":"2024-02-10T08:44:39.401938+0000","last_clean":"2024-02-10T08:41:37.100674+0000","last_became_active":"2024-02-10T08:43:36.073243+0000","last_became_peered":"2024-02-10T08:43:36.073243+0000","last_unstale":"2024-02-10T08:44:39.401938+0000","last_undegraded":"2024-02-10T08:43:36.066452+0000","last_fullsized":"2024-02-10T08:43:36.066344+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'42","last_scrub_stamp":"2024-02-10T08:42:31.735511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:31.735511+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.0","version":"513'109","reported_seq":778,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313692+0000","last_change":"2024-02-10T08:43:18.114463+0000","last_active":"2024-02-10T08:44:40.313692+0000","last_peered":"2024-02-10T08:44:40.313692+0000","last_clean":"2024-02-10T08:44:40.313692+0000","last_became_active":"2024-02-10T08:43:18.113727+0000","last_became_peered":"2024-02-10T08:43:18.113727+0000","last_unstale":"2024-02-10T08:44:40.313692+0000","last_undegraded":"2024-02-10T08:44:40.313692+0000","last_fullsized":"2024-02-10T08:44:40.313692+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'108","last_scrub_stamp":"2024-02-10T08:42:46.327938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.327938+0000","log_size":109,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":595,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":35,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.7","version":"513'40","reported_seq":390,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.087233+0000","last_change":"2024-02-10T08:43:36.078057+0000","last_active":"2024-02-10T08:44:44.087233+0000","last_peered":"2024-02-10T08:44:44.087233+0000","last_clean":"2024-02-10T08:41:05.014702+0000","last_became_active":"2024-02-10T08:43:36.078057+0000","last_became_peered":"2024-02-10T08:43:36.078057+0000","last_unstale":"2024-02-10T08:44:44.087233+0000","last_undegraded":"2024-02-10T08:43:36.057295+0000","last_fullsized":"2024-02-10T08:43:36.057186+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:32.704166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.704166+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4124,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.6","version":"495'79","reported_seq":575,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.316014+0000","last_change":"2024-02-10T08:44:33.484260+0000","last_active":"2024-02-10T08:44:40.316014+0000","last_peered":"2024-02-10T08:44:40.316014+0000","last_clean":"2024-02-10T08:44:40.316014+0000","last_became_active":"2024-02-10T08:36:50.985897+0000","last_became_peered":"2024-02-10T08:36:50.985897+0000","last_unstale":"2024-02-10T08:44:40.316014+0000","last_undegraded":"2024-02-10T08:44:40.316014+0000","last_fullsized":"2024-02-10T08:44:40.316014+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'79","last_scrub_stamp":"2024-02-10T08:44:33.484193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:33.484193+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":31,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.5","version":"495'63","reported_seq":602,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.135462+0000","last_change":"2024-02-10T08:43:36.065069+0000","last_active":"2024-02-10T08:44:44.135462+0000","last_peered":"2024-02-10T08:44:44.135462+0000","last_clean":"2024-02-10T08:43:11.067979+0000","last_became_active":"2024-02-10T08:43:36.065069+0000","last_became_peered":"2024-02-10T08:43:36.065069+0000","last_unstale":"2024-02-10T08:44:44.135462+0000","last_undegraded":"2024-02-10T08:43:36.060893+0000","last_fullsized":"2024-02-10T08:43:36.060775+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'42","last_scrub_stamp":"2024-02-10T08:42:37.799725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.799725+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":27,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.4","version":"504'27","reported_seq":368,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.315350+0000","last_change":"2024-02-10T08:43:36.075301+0000","last_active":"2024-02-10T08:44:40.315350+0000","last_peered":"2024-02-10T08:44:40.315350+0000","last_clean":"2024-02-10T08:40:31.992410+0000","last_became_active":"2024-02-10T08:43:36.075301+0000","last_became_peered":"2024-02-10T08:43:36.075301+0000","last_unstale":"2024-02-10T08:44:40.315350+0000","last_undegraded":"2024-02-10T08:43:36.063465+0000","last_fullsized":"2024-02-10T08:43:36.063358+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'23","last_scrub_stamp":"2024-02-10T08:42:38.786497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:38.786497+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":545,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.b","version":"513'107","reported_seq":767,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315222+0000","last_change":"2024-02-10T08:43:21.062588+0000","last_active":"2024-02-10T08:44:40.315222+0000","last_peered":"2024-02-10T08:44:40.315222+0000","last_clean":"2024-02-10T08:44:40.315222+0000","last_became_active":"2024-02-10T08:43:21.061839+0000","last_became_peered":"2024-02-10T08:43:21.061839+0000","last_unstale":"2024-02-10T08:44:40.315222+0000","last_undegraded":"2024-02-10T08:44:40.315222+0000","last_fullsized":"2024-02-10T08:44:40.315222+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'105","last_scrub_stamp":"2024-02-10T08:42:42.390288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.390288+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.a","version":"513'31","reported_seq":505,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.086475+0000","last_change":"2024-02-10T08:43:36.069735+0000","last_active":"2024-02-10T08:44:44.086475+0000","last_peered":"2024-02-10T08:44:44.086475+0000","last_clean":"2024-02-10T08:42:41.079071+0000","last_became_active":"2024-02-10T08:43:36.069735+0000","last_became_peered":"2024-02-10T08:43:36.069735+0000","last_unstale":"2024-02-10T08:44:44.086475+0000","last_undegraded":"2024-02-10T08:43:36.056543+0000","last_fullsized":"2024-02-10T08:43:36.056433+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'23","last_scrub_stamp":"2024-02-10T08:42:30.719056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.719056+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2339,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":16,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.9","version":"513'120","reported_seq":771,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.313026+0000","last_change":"2024-02-10T08:43:36.076377+0000","last_active":"2024-02-10T08:44:40.313026+0000","last_peered":"2024-02-10T08:44:40.313026+0000","last_clean":"2024-02-10T08:43:31.055757+0000","last_became_active":"2024-02-10T08:43:36.076377+0000","last_became_peered":"2024-02-10T08:43:36.076377+0000","last_unstale":"2024-02-10T08:44:40.313026+0000","last_undegraded":"2024-02-10T08:43:36.060586+0000","last_fullsized":"2024-02-10T08:43:36.060472+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'115","last_scrub_stamp":"2024-02-10T08:42:27.509423+0000","last_deep_scrub":"299'110","last_deep_scrub_stamp":"2024-02-10T08:39:53.048153+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.509423+0000","log_size":120,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":3,"num_write":41,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.8","version":"513'101","reported_seq":643,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.313108+0000","last_change":"2024-02-10T08:43:36.065933+0000","last_active":"2024-02-10T08:44:40.313108+0000","last_peered":"2024-02-10T08:44:40.313108+0000","last_clean":"2024-02-10T08:43:11.133528+0000","last_became_active":"2024-02-10T08:43:36.065933+0000","last_became_peered":"2024-02-10T08:43:36.065933+0000","last_unstale":"2024-02-10T08:44:40.313108+0000","last_undegraded":"2024-02-10T08:43:36.056709+0000","last_fullsized":"2024-02-10T08:43:36.056582+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'86","last_scrub_stamp":"2024-02-10T08:42:34.734692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.734692+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1308,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":63,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.f","version":"513'97","reported_seq":586,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.183934+0000","last_change":"2024-02-10T08:43:27.256781+0000","last_active":"2024-02-10T08:44:44.183934+0000","last_peered":"2024-02-10T08:44:44.183934+0000","last_clean":"2024-02-10T08:44:44.183934+0000","last_became_active":"2024-02-10T08:36:50.930535+0000","last_became_peered":"2024-02-10T08:36:50.930535+0000","last_unstale":"2024-02-10T08:44:44.183934+0000","last_undegraded":"2024-02-10T08:44:44.183934+0000","last_fullsized":"2024-02-10T08:44:44.183934+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'96","last_scrub_stamp":"2024-02-10T08:43:27.256674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.256674+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1303,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.e","version":"513'60","reported_seq":539,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.405708+0000","last_change":"2024-02-10T08:43:36.074273+0000","last_active":"2024-02-10T08:44:39.405708+0000","last_peered":"2024-02-10T08:44:39.405708+0000","last_clean":"2024-02-10T08:43:31.058727+0000","last_became_active":"2024-02-10T08:43:36.074273+0000","last_became_peered":"2024-02-10T08:43:36.074273+0000","last_unstale":"2024-02-10T08:44:39.405708+0000","last_undegraded":"2024-02-10T08:43:36.069012+0000","last_fullsized":"2024-02-10T08:43:36.067912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'56","last_scrub_stamp":"2024-02-10T08:43:30.090939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.090939+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.d","version":"528'92","reported_seq":595,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.182352+0000","last_change":"2024-02-10T08:43:36.072717+0000","last_active":"2024-02-10T08:44:44.182352+0000","last_peered":"2024-02-10T08:44:44.182352+0000","last_clean":"2024-02-10T08:43:31.156895+0000","last_became_active":"2024-02-10T08:43:36.072717+0000","last_became_peered":"2024-02-10T08:43:36.072717+0000","last_unstale":"2024-02-10T08:44:44.182352+0000","last_undegraded":"2024-02-10T08:43:36.064470+0000","last_fullsized":"2024-02-10T08:43:36.064396+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'78","last_scrub_stamp":"2024-02-10T08:43:23.272230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.272230+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"60.c","version":"494'72","reported_seq":565,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315190+0000","last_change":"2024-02-10T08:43:14.433052+0000","last_active":"2024-02-10T08:44:40.315190+0000","last_peered":"2024-02-10T08:44:40.315190+0000","last_clean":"2024-02-10T08:44:40.315190+0000","last_became_active":"2024-02-10T08:37:10.232337+0000","last_became_peered":"2024-02-10T08:37:10.232337+0000","last_unstale":"2024-02-10T08:44:40.315190+0000","last_undegraded":"2024-02-10T08:44:40.315190+0000","last_fullsized":"2024-02-10T08:44:40.315190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'70","last_scrub_stamp":"2024-02-10T08:43:14.432955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.432955+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":25,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"60.13","version":"495'68","reported_seq":570,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086210+0000","last_change":"2024-02-10T08:44:36.346036+0000","last_active":"2024-02-10T08:44:44.086210+0000","last_peered":"2024-02-10T08:44:44.086210+0000","last_clean":"2024-02-10T08:44:44.086210+0000","last_became_active":"2024-02-10T08:36:51.019301+0000","last_became_peered":"2024-02-10T08:36:51.019301+0000","last_unstale":"2024-02-10T08:44:44.086210+0000","last_undegraded":"2024-02-10T08:44:44.086210+0000","last_fullsized":"2024-02-10T08:44:44.086210+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'68","last_scrub_stamp":"2024-02-10T08:44:36.345945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.345945+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000062,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":9,"num_write":36,"num_write_kb":3935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.12","version":"513'105","reported_seq":316,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.136904+0000","last_change":"2024-02-10T08:43:36.073079+0000","last_active":"2024-02-10T08:44:44.136904+0000","last_peered":"2024-02-10T08:44:44.136904+0000","last_clean":"2024-02-10T08:43:31.060665+0000","last_became_active":"2024-02-10T08:43:36.073079+0000","last_became_peered":"2024-02-10T08:43:36.073079+0000","last_unstale":"2024-02-10T08:44:44.136904+0000","last_undegraded":"2024-02-10T08:43:36.062346+0000","last_fullsized":"2024-02-10T08:43:36.062237+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-10T08:41:51.538529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:41:51.538529+0000","log_size":105,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":766,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":3,"num_write":35,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.11","version":"513'110","reported_seq":599,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.086375+0000","last_change":"2024-02-10T08:43:36.063661+0000","last_active":"2024-02-10T08:44:44.086375+0000","last_peered":"2024-02-10T08:44:44.086375+0000","last_clean":"2024-02-10T08:43:31.061494+0000","last_became_active":"2024-02-10T08:43:36.063661+0000","last_became_peered":"2024-02-10T08:43:36.063661+0000","last_unstale":"2024-02-10T08:44:44.086375+0000","last_undegraded":"2024-02-10T08:43:36.056704+0000","last_fullsized":"2024-02-10T08:43:36.056581+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'105","last_scrub_stamp":"2024-02-10T08:42:30.099763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:30.099763+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":651,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":40,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.10","version":"522'76","reported_seq":135,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134830+0000","last_change":"2024-02-10T08:44:30.585792+0000","last_active":"2024-02-10T08:44:44.134830+0000","last_peered":"2024-02-10T08:44:44.134830+0000","last_clean":"2024-02-10T08:44:44.134830+0000","last_became_active":"2024-02-10T08:43:15.134958+0000","last_became_peered":"2024-02-10T08:43:15.134958+0000","last_unstale":"2024-02-10T08:44:44.134830+0000","last_undegraded":"2024-02-10T08:44:44.134830+0000","last_fullsized":"2024-02-10T08:44:44.134830+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"522'76","last_scrub_stamp":"2024-02-10T08:44:30.585730+0000","last_deep_scrub":"301'72","last_deep_scrub_stamp":"2024-02-10T08:39:55.527854+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.585730+0000","log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":611,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":25,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.17","version":"392'87","reported_seq":143,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.137216+0000","last_change":"2024-02-10T08:44:32.565847+0000","last_active":"2024-02-10T08:44:44.137216+0000","last_peered":"2024-02-10T08:44:44.137216+0000","last_clean":"2024-02-10T08:44:44.137216+0000","last_became_active":"2024-02-10T08:43:15.138604+0000","last_became_peered":"2024-02-10T08:43:15.138604+0000","last_unstale":"2024-02-10T08:44:44.137216+0000","last_undegraded":"2024-02-10T08:44:44.137216+0000","last_fullsized":"2024-02-10T08:44:44.137216+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'87","last_scrub_stamp":"2024-02-10T08:44:32.565777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:32.565777+0000","log_size":87,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"60.16","version":"494'66","reported_seq":680,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405598+0000","last_change":"2024-02-10T08:43:21.049927+0000","last_active":"2024-02-10T08:44:39.405598+0000","last_peered":"2024-02-10T08:44:39.405598+0000","last_clean":"2024-02-10T08:44:39.405598+0000","last_became_active":"2024-02-10T08:43:15.137295+0000","last_became_peered":"2024-02-10T08:43:15.137295+0000","last_unstale":"2024-02-10T08:44:39.405598+0000","last_undegraded":"2024-02-10T08:44:39.405598+0000","last_fullsized":"2024-02-10T08:44:39.405598+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'62","last_scrub_stamp":"2024-02-10T08:43:21.049825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.049825+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":26,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.15","version":"494'51","reported_seq":549,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.087070+0000","last_change":"2024-02-10T08:43:36.069660+0000","last_active":"2024-02-10T08:44:44.087070+0000","last_peered":"2024-02-10T08:44:44.087070+0000","last_clean":"2024-02-10T08:43:26.149056+0000","last_became_active":"2024-02-10T08:43:36.069660+0000","last_became_peered":"2024-02-10T08:43:36.069660+0000","last_unstale":"2024-02-10T08:44:44.087070+0000","last_undegraded":"2024-02-10T08:43:36.057367+0000","last_fullsized":"2024-02-10T08:43:36.056883+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'34","last_scrub_stamp":"2024-02-10T08:42:39.756505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:39.756505+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.14","version":"504'39","reported_seq":536,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.402777+0000","last_change":"2024-02-10T08:43:36.089221+0000","last_active":"2024-02-10T08:44:39.402777+0000","last_peered":"2024-02-10T08:44:39.402777+0000","last_clean":"2024-02-10T08:43:31.058392+0000","last_became_active":"2024-02-10T08:43:36.089221+0000","last_became_peered":"2024-02-10T08:43:36.089221+0000","last_unstale":"2024-02-10T08:44:39.402777+0000","last_undegraded":"2024-02-10T08:43:36.071737+0000","last_fullsized":"2024-02-10T08:43:36.071615+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'36","last_scrub_stamp":"2024-02-10T08:43:23.027840+0000","last_deep_scrub":"295'36","last_deep_scrub_stamp":"2024-02-10T08:39:49.714835+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.027840+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1359,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1b","version":"531'73","reported_seq":658,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.404307+0000","last_change":"2024-02-10T08:43:36.082574+0000","last_active":"2024-02-10T08:44:39.404307+0000","last_peered":"2024-02-10T08:44:39.404307+0000","last_clean":"2024-02-10T08:43:31.057526+0000","last_became_active":"2024-02-10T08:43:36.082574+0000","last_became_peered":"2024-02-10T08:43:36.082574+0000","last_unstale":"2024-02-10T08:44:39.404307+0000","last_undegraded":"2024-02-10T08:43:36.072001+0000","last_fullsized":"2024-02-10T08:43:36.071880+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-10T08:42:47.608965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.608965+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2372,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1a","version":"513'74","reported_seq":561,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.312890+0000","last_change":"2024-02-10T08:44:39.698978+0000","last_active":"2024-02-10T08:44:40.312890+0000","last_peered":"2024-02-10T08:44:40.312890+0000","last_clean":"2024-02-10T08:44:40.312890+0000","last_became_active":"2024-02-10T08:36:51.014367+0000","last_became_peered":"2024-02-10T08:36:51.014367+0000","last_unstale":"2024-02-10T08:44:40.312890+0000","last_undegraded":"2024-02-10T08:44:40.312890+0000","last_fullsized":"2024-02-10T08:44:40.312890+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'74","last_scrub_stamp":"2024-02-10T08:44:39.698913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.698913+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1594,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.19","version":"338'35","reported_seq":611,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313194+0000","last_change":"2024-02-10T08:44:28.733671+0000","last_active":"2024-02-10T08:44:40.313194+0000","last_peered":"2024-02-10T08:44:40.313194+0000","last_clean":"2024-02-10T08:44:40.313194+0000","last_became_active":"2024-02-10T08:43:15.130144+0000","last_became_peered":"2024-02-10T08:43:15.130144+0000","last_unstale":"2024-02-10T08:44:40.313194+0000","last_undegraded":"2024-02-10T08:44:40.313194+0000","last_fullsized":"2024-02-10T08:44:40.313194+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-10T08:44:28.733585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.733585+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.18","version":"401'38","reported_seq":415,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.401540+0000","last_change":"2024-02-10T08:43:36.081808+0000","last_active":"2024-02-10T08:44:39.401540+0000","last_peered":"2024-02-10T08:44:39.401540+0000","last_clean":"2024-02-10T08:41:37.103139+0000","last_became_active":"2024-02-10T08:43:36.081808+0000","last_became_peered":"2024-02-10T08:43:36.081808+0000","last_unstale":"2024-02-10T08:44:39.401540+0000","last_undegraded":"2024-02-10T08:43:36.066395+0000","last_fullsized":"2024-02-10T08:43:36.066281+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'38","last_scrub_stamp":"2024-02-10T08:42:41.708918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.708918+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1f","version":"494'44","reported_seq":558,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087378+0000","last_change":"2024-02-10T08:44:30.371985+0000","last_active":"2024-02-10T08:44:44.087378+0000","last_peered":"2024-02-10T08:44:44.087378+0000","last_clean":"2024-02-10T08:44:44.087378+0000","last_became_active":"2024-02-10T08:37:10.167611+0000","last_became_peered":"2024-02-10T08:37:10.167611+0000","last_unstale":"2024-02-10T08:44:44.087378+0000","last_undegraded":"2024-02-10T08:44:44.087378+0000","last_fullsized":"2024-02-10T08:44:44.087378+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'44","last_scrub_stamp":"2024-02-10T08:44:30.371909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.371909+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"60.1e","version":"495'21","reported_seq":506,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.327804+0000","last_change":"2024-02-10T08:44:23.770151+0000","last_active":"2024-02-10T08:44:40.327804+0000","last_peered":"2024-02-10T08:44:40.327804+0000","last_clean":"2024-02-10T08:44:40.327804+0000","last_became_active":"2024-02-10T08:37:10.218671+0000","last_became_peered":"2024-02-10T08:37:10.218671+0000","last_unstale":"2024-02-10T08:44:40.327804+0000","last_undegraded":"2024-02-10T08:44:40.327804+0000","last_fullsized":"2024-02-10T08:44:40.327804+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'21","last_scrub_stamp":"2024-02-10T08:44:23.770085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:44:23.770085+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"60.1d","version":"504'23","reported_seq":563,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405348+0000","last_change":"2024-02-10T08:43:33.115605+0000","last_active":"2024-02-10T08:44:39.405348+0000","last_peered":"2024-02-10T08:44:39.405348+0000","last_clean":"2024-02-10T08:44:39.405348+0000","last_became_active":"2024-02-10T08:43:15.136520+0000","last_became_peered":"2024-02-10T08:43:15.136520+0000","last_unstale":"2024-02-10T08:44:39.405348+0000","last_undegraded":"2024-02-10T08:44:39.405348+0000","last_fullsized":"2024-02-10T08:44:39.405348+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'21","last_scrub_stamp":"2024-02-10T08:43:33.115538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:43:33.115538+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"60.1c","version":"504'36","reported_seq":541,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.314703+0000","last_change":"2024-02-10T08:43:36.076024+0000","last_active":"2024-02-10T08:44:40.314703+0000","last_peered":"2024-02-10T08:44:40.314703+0000","last_clean":"2024-02-10T08:43:31.066314+0000","last_became_active":"2024-02-10T08:43:36.076024+0000","last_became_peered":"2024-02-10T08:43:36.076024+0000","last_unstale":"2024-02-10T08:44:40.314703+0000","last_undegraded":"2024-02-10T08:43:36.063160+0000","last_fullsized":"2024-02-10T08:43:36.063048+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'35","last_scrub_stamp":"2024-02-10T08:42:41.415071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:50.313439+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.415071+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1245,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"481'17","reported_seq":161,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.137171+0000","last_change":"2024-02-10T08:43:36.074524+0000","last_active":"2024-02-10T08:44:44.137171+0000","last_peered":"2024-02-10T08:44:44.137171+0000","last_clean":"2024-02-10T08:43:31.610321+0000","last_became_active":"2024-02-10T08:43:36.074524+0000","last_became_peered":"2024-02-10T08:43:36.074524+0000","last_unstale":"2024-02-10T08:44:44.137171+0000","last_undegraded":"2024-02-10T08:43:36.061132+0000","last_fullsized":"2024-02-10T08:43:36.061013+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'13","last_scrub_stamp":"2024-02-10T08:43:10.390465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.390465+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":606,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.085869+0000","last_change":"2024-02-10T08:43:15.731491+0000","last_active":"2024-02-10T08:44:44.085869+0000","last_peered":"2024-02-10T08:44:44.085869+0000","last_clean":"2024-02-10T08:44:44.085869+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:44:44.085869+0000","last_undegraded":"2024-02-10T08:44:44.085869+0000","last_fullsized":"2024-02-10T08:44:44.085869+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:42:25.091640+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.091640+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":526,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.316161+0000","last_change":"2024-02-10T08:43:15.119398+0000","last_active":"2024-02-10T08:44:40.316161+0000","last_peered":"2024-02-10T08:44:40.316161+0000","last_clean":"2024-02-10T08:44:40.316161+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:44:40.316161+0000","last_undegraded":"2024-02-10T08:44:40.316161+0000","last_fullsized":"2024-02-10T08:44:40.316161+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.504915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.504915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":516,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086684+0000","last_change":"2024-02-10T08:42:41.108507+0000","last_active":"2024-02-10T08:44:44.086684+0000","last_peered":"2024-02-10T08:44:44.086684+0000","last_clean":"2024-02-10T08:44:44.086684+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:44:44.086684+0000","last_undegraded":"2024-02-10T08:44:44.086684+0000","last_fullsized":"2024-02-10T08:44:44.086684+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:41.108418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.108418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":456,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315331+0000","last_change":"2024-02-10T08:44:28.457808+0000","last_active":"2024-02-10T08:44:40.315331+0000","last_peered":"2024-02-10T08:44:40.315331+0000","last_clean":"2024-02-10T08:44:40.315331+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:44:40.315331+0000","last_undegraded":"2024-02-10T08:44:40.315331+0000","last_fullsized":"2024-02-10T08:44:40.315331+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:28.457738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.457738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":495,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.315591+0000","last_change":"2024-02-10T08:43:36.079191+0000","last_active":"2024-02-10T08:44:40.315591+0000","last_peered":"2024-02-10T08:44:40.315591+0000","last_clean":"2024-02-10T08:43:31.066295+0000","last_became_active":"2024-02-10T08:43:36.079191+0000","last_became_peered":"2024-02-10T08:43:36.079191+0000","last_unstale":"2024-02-10T08:44:40.315591+0000","last_undegraded":"2024-02-10T08:43:36.059738+0000","last_fullsized":"2024-02-10T08:43:36.059623+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":519,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.335646+0000","last_change":"2024-02-10T08:43:36.085621+0000","last_active":"2024-02-10T08:44:40.335646+0000","last_peered":"2024-02-10T08:44:40.335646+0000","last_clean":"2024-02-10T08:43:31.059000+0000","last_became_active":"2024-02-10T08:43:36.085621+0000","last_became_peered":"2024-02-10T08:43:36.085621+0000","last_unstale":"2024-02-10T08:44:40.335646+0000","last_undegraded":"2024-02-10T08:44:40.335646+0000","last_fullsized":"2024-02-10T08:43:36.063451+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.696704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.696704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":528,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.315208+0000","last_change":"2024-02-10T08:43:36.079793+0000","last_active":"2024-02-10T08:44:40.315208+0000","last_peered":"2024-02-10T08:44:40.315208+0000","last_clean":"2024-02-10T08:43:31.065630+0000","last_became_active":"2024-02-10T08:43:36.079793+0000","last_became_peered":"2024-02-10T08:43:36.079793+0000","last_unstale":"2024-02-10T08:44:40.315208+0000","last_undegraded":"2024-02-10T08:44:40.315208+0000","last_fullsized":"2024-02-10T08:43:36.061952+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135732+0000","last_change":"2024-02-10T08:44:39.598037+0000","last_active":"2024-02-10T08:44:44.135732+0000","last_peered":"2024-02-10T08:44:44.135732+0000","last_clean":"2024-02-10T08:44:44.135732+0000","last_became_active":"2024-02-10T08:43:15.128618+0000","last_became_peered":"2024-02-10T08:43:15.128618+0000","last_unstale":"2024-02-10T08:44:44.135732+0000","last_undegraded":"2024-02-10T08:44:44.135732+0000","last_fullsized":"2024-02-10T08:44:44.135732+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.597966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.597966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":474,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087816+0000","last_change":"2024-02-10T08:44:39.356595+0000","last_active":"2024-02-10T08:44:44.087816+0000","last_peered":"2024-02-10T08:44:44.087816+0000","last_clean":"2024-02-10T08:44:44.087816+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:44:44.087816+0000","last_undegraded":"2024-02-10T08:44:44.087816+0000","last_fullsized":"2024-02-10T08:44:44.087816+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.356502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.356502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"473'35","reported_seq":633,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.183667+0000","last_change":"2024-02-10T08:44:37.425734+0000","last_active":"2024-02-10T08:44:44.183667+0000","last_peered":"2024-02-10T08:44:44.183667+0000","last_clean":"2024-02-10T08:44:44.183667+0000","last_became_active":"2024-02-10T08:43:15.136753+0000","last_became_peered":"2024-02-10T08:43:15.136753+0000","last_unstale":"2024-02-10T08:44:44.183667+0000","last_undegraded":"2024-02-10T08:44:44.183667+0000","last_fullsized":"2024-02-10T08:44:44.183667+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'35","last_scrub_stamp":"2024-02-10T08:44:37.425670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.425670+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":517,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315783+0000","last_change":"2024-02-10T08:43:53.472550+0000","last_active":"2024-02-10T08:44:40.315783+0000","last_peered":"2024-02-10T08:44:40.315783+0000","last_clean":"2024-02-10T08:44:40.315783+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:44:40.315783+0000","last_undegraded":"2024-02-10T08:44:40.315783+0000","last_fullsized":"2024-02-10T08:44:40.315783+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:53.472449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:43:53.472449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":551,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315398+0000","last_change":"2024-02-10T08:43:23.422453+0000","last_active":"2024-02-10T08:44:40.315398+0000","last_peered":"2024-02-10T08:44:40.315398+0000","last_clean":"2024-02-10T08:44:40.315398+0000","last_became_active":"2024-02-10T08:43:15.119008+0000","last_became_peered":"2024-02-10T08:43:15.119008+0000","last_unstale":"2024-02-10T08:44:40.315398+0000","last_undegraded":"2024-02-10T08:44:40.315398+0000","last_fullsized":"2024-02-10T08:44:40.315398+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.422346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.422346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134666+0000","last_change":"2024-02-10T08:44:40.551722+0000","last_active":"2024-02-10T08:44:44.134666+0000","last_peered":"2024-02-10T08:44:44.134666+0000","last_clean":"2024-02-10T08:44:44.134666+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:44:44.134666+0000","last_undegraded":"2024-02-10T08:44:44.134666+0000","last_fullsized":"2024-02-10T08:44:44.134666+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.551662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.136825+0000","last_change":"2024-02-10T08:43:36.072003+0000","last_active":"2024-02-10T08:44:44.136825+0000","last_peered":"2024-02-10T08:44:44.136825+0000","last_clean":"2024-02-10T08:43:31.061070+0000","last_became_active":"2024-02-10T08:43:36.072003+0000","last_became_peered":"2024-02-10T08:43:36.072003+0000","last_unstale":"2024-02-10T08:44:44.136825+0000","last_undegraded":"2024-02-10T08:44:44.136825+0000","last_fullsized":"2024-02-10T08:43:36.062752+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":553,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.315372+0000","last_change":"2024-02-10T08:43:36.076710+0000","last_active":"2024-02-10T08:44:40.315372+0000","last_peered":"2024-02-10T08:44:40.315372+0000","last_clean":"2024-02-10T08:43:25.756727+0000","last_became_active":"2024-02-10T08:43:36.076710+0000","last_became_peered":"2024-02-10T08:43:36.076710+0000","last_unstale":"2024-02-10T08:44:40.315372+0000","last_undegraded":"2024-02-10T08:43:36.060349+0000","last_fullsized":"2024-02-10T08:43:36.060248+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:43:29.844454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:29.844454+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":346,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.403958+0000","last_change":"2024-02-10T08:44:33.297486+0000","last_active":"2024-02-10T08:44:39.403958+0000","last_peered":"2024-02-10T08:44:39.403958+0000","last_clean":"2024-02-10T08:44:39.403958+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:44:39.403958+0000","last_undegraded":"2024-02-10T08:44:39.403958+0000","last_fullsized":"2024-02-10T08:44:39.403958+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:44:33.297424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":537,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.331389+0000","last_change":"2024-02-10T08:43:36.076112+0000","last_active":"2024-02-10T08:44:40.331389+0000","last_peered":"2024-02-10T08:44:40.331389+0000","last_clean":"2024-02-10T08:43:31.060641+0000","last_became_active":"2024-02-10T08:43:36.076112+0000","last_became_peered":"2024-02-10T08:43:36.076112+0000","last_unstale":"2024-02-10T08:44:40.331389+0000","last_undegraded":"2024-02-10T08:44:40.331389+0000","last_fullsized":"2024-02-10T08:43:36.063764+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":508,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.316344+0000","last_change":"2024-02-10T08:43:36.075565+0000","last_active":"2024-02-10T08:44:40.316344+0000","last_peered":"2024-02-10T08:44:40.316344+0000","last_clean":"2024-02-10T08:43:31.066268+0000","last_became_active":"2024-02-10T08:43:36.075565+0000","last_became_peered":"2024-02-10T08:43:36.075565+0000","last_unstale":"2024-02-10T08:44:40.316344+0000","last_undegraded":"2024-02-10T08:44:40.316344+0000","last_fullsized":"2024-02-10T08:43:36.061936+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":62,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.402866+0000","last_change":"2024-02-10T08:43:36.089962+0000","last_active":"2024-02-10T08:44:39.402866+0000","last_peered":"2024-02-10T08:44:39.402866+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"2024-02-10T08:43:36.089962+0000","last_became_peered":"2024-02-10T08:43:36.089962+0000","last_unstale":"2024-02-10T08:44:39.402866+0000","last_undegraded":"2024-02-10T08:44:39.402866+0000","last_fullsized":"2024-02-10T08:43:36.068401+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":533,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.403643+0000","last_change":"2024-02-10T08:43:36.073845+0000","last_active":"2024-02-10T08:44:39.403643+0000","last_peered":"2024-02-10T08:44:39.403643+0000","last_clean":"2024-02-10T08:43:25.718471+0000","last_became_active":"2024-02-10T08:43:36.073845+0000","last_became_peered":"2024-02-10T08:43:36.073845+0000","last_unstale":"2024-02-10T08:44:39.403643+0000","last_undegraded":"2024-02-10T08:43:36.068245+0000","last_fullsized":"2024-02-10T08:43:36.068123+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":531,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.321392+0000","last_change":"2024-02-10T08:43:15.130789+0000","last_active":"2024-02-10T08:44:40.321392+0000","last_peered":"2024-02-10T08:44:40.321392+0000","last_clean":"2024-02-10T08:44:40.321392+0000","last_became_active":"2024-02-10T08:43:15.130483+0000","last_became_peered":"2024-02-10T08:43:15.130483+0000","last_unstale":"2024-02-10T08:44:40.321392+0000","last_undegraded":"2024-02-10T08:44:40.321392+0000","last_fullsized":"2024-02-10T08:44:40.321392+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.744038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.744038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"469'16","reported_seq":722,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315275+0000","last_change":"2024-02-10T08:43:27.381317+0000","last_active":"2024-02-10T08:44:40.315275+0000","last_peered":"2024-02-10T08:44:40.315275+0000","last_clean":"2024-02-10T08:44:40.315275+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:44:40.315275+0000","last_undegraded":"2024-02-10T08:44:40.315275+0000","last_fullsized":"2024-02-10T08:44:40.315275+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'16","last_scrub_stamp":"2024-02-10T08:43:27.381246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.381246+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":63,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.328057+0000","last_change":"2024-02-10T08:43:36.084792+0000","last_active":"2024-02-10T08:44:40.328057+0000","last_peered":"2024-02-10T08:44:40.328057+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.084792+0000","last_became_peered":"2024-02-10T08:43:36.084792+0000","last_unstale":"2024-02-10T08:44:40.328057+0000","last_undegraded":"2024-02-10T08:44:40.328057+0000","last_fullsized":"2024-02-10T08:43:36.060560+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":444,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.401534+0000","last_change":"2024-02-10T08:43:36.082315+0000","last_active":"2024-02-10T08:44:39.401534+0000","last_peered":"2024-02-10T08:44:39.401534+0000","last_clean":"2024-02-10T08:43:31.057840+0000","last_became_active":"2024-02-10T08:43:36.082315+0000","last_became_peered":"2024-02-10T08:43:36.082315+0000","last_unstale":"2024-02-10T08:44:39.401534+0000","last_undegraded":"2024-02-10T08:44:39.401534+0000","last_fullsized":"2024-02-10T08:43:36.070820+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.219434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.219434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":555,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.327270+0000","last_change":"2024-02-10T08:43:36.088120+0000","last_active":"2024-02-10T08:44:40.327270+0000","last_peered":"2024-02-10T08:44:40.327270+0000","last_clean":"2024-02-10T08:43:31.056937+0000","last_became_active":"2024-02-10T08:43:36.088120+0000","last_became_peered":"2024-02-10T08:43:36.088120+0000","last_unstale":"2024-02-10T08:44:40.327270+0000","last_undegraded":"2024-02-10T08:43:36.061412+0000","last_fullsized":"2024-02-10T08:43:36.061296+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":547,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313501+0000","last_change":"2024-02-10T08:43:50.689633+0000","last_active":"2024-02-10T08:44:40.313501+0000","last_peered":"2024-02-10T08:44:40.313501+0000","last_clean":"2024-02-10T08:44:40.313501+0000","last_became_active":"2024-02-10T08:43:15.130693+0000","last_became_peered":"2024-02-10T08:43:15.130693+0000","last_unstale":"2024-02-10T08:44:40.313501+0000","last_undegraded":"2024-02-10T08:44:40.313501+0000","last_fullsized":"2024-02-10T08:44:40.313501+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:43:50.689512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:50.689512+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"481'17","reported_seq":101,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.087321+0000","last_change":"2024-02-10T08:43:36.076136+0000","last_active":"2024-02-10T08:44:44.087321+0000","last_peered":"2024-02-10T08:44:44.087321+0000","last_clean":"2024-02-10T08:33:25.224233+0000","last_became_active":"2024-02-10T08:43:36.076136+0000","last_became_peered":"2024-02-10T08:43:36.076136+0000","last_unstale":"2024-02-10T08:44:44.087321+0000","last_undegraded":"2024-02-10T08:43:36.056738+0000","last_fullsized":"2024-02-10T08:43:36.056588+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:42:44.647464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.647464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":511,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087046+0000","last_change":"2024-02-10T08:42:34.139662+0000","last_active":"2024-02-10T08:44:44.087046+0000","last_peered":"2024-02-10T08:44:44.087046+0000","last_clean":"2024-02-10T08:44:44.087046+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:44:44.087046+0000","last_undegraded":"2024-02-10T08:44:44.087046+0000","last_fullsized":"2024-02-10T08:44:44.087046+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.139528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.139528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":474,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086320+0000","last_change":"2024-02-10T08:44:35.383065+0000","last_active":"2024-02-10T08:44:44.086320+0000","last_peered":"2024-02-10T08:44:44.086320+0000","last_clean":"2024-02-10T08:44:44.086320+0000","last_became_active":"2024-02-10T08:43:00.552233+0000","last_became_peered":"2024-02-10T08:43:00.552233+0000","last_unstale":"2024-02-10T08:44:44.086320+0000","last_undegraded":"2024-02-10T08:44:44.086320+0000","last_fullsized":"2024-02-10T08:44:44.086320+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.382972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.382972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":537,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.182300+0000","last_change":"2024-02-10T08:43:36.068792+0000","last_active":"2024-02-10T08:44:44.182300+0000","last_peered":"2024-02-10T08:44:44.182300+0000","last_clean":"2024-02-10T08:43:25.758929+0000","last_became_active":"2024-02-10T08:43:36.068792+0000","last_became_peered":"2024-02-10T08:43:36.068792+0000","last_unstale":"2024-02-10T08:44:44.182300+0000","last_undegraded":"2024-02-10T08:43:36.064019+0000","last_fullsized":"2024-02-10T08:43:36.063905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.134433+0000","last_change":"2024-02-10T08:43:36.068338+0000","last_active":"2024-02-10T08:44:44.134433+0000","last_peered":"2024-02-10T08:44:44.134433+0000","last_clean":"2024-02-10T08:43:31.059980+0000","last_became_active":"2024-02-10T08:43:36.068338+0000","last_became_peered":"2024-02-10T08:43:36.068338+0000","last_unstale":"2024-02-10T08:44:44.134433+0000","last_undegraded":"2024-02-10T08:44:44.134433+0000","last_fullsized":"2024-02-10T08:43:36.060130+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.736740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.736740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"481'11","reported_seq":629,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.087633+0000","last_change":"2024-02-10T08:43:36.068155+0000","last_active":"2024-02-10T08:44:44.087633+0000","last_peered":"2024-02-10T08:44:44.087633+0000","last_clean":"2024-02-10T08:43:31.610349+0000","last_became_active":"2024-02-10T08:43:36.068155+0000","last_became_peered":"2024-02-10T08:43:36.068155+0000","last_unstale":"2024-02-10T08:44:44.087633+0000","last_undegraded":"2024-02-10T08:43:36.058153+0000","last_fullsized":"2024-02-10T08:43:36.058069+0000","mapping_epoch":480,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":502,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.403222+0000","last_change":"2024-02-10T08:44:26.397785+0000","last_active":"2024-02-10T08:44:39.403222+0000","last_peered":"2024-02-10T08:44:39.403222+0000","last_clean":"2024-02-10T08:44:39.403222+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:44:39.403222+0000","last_undegraded":"2024-02-10T08:44:39.403222+0000","last_fullsized":"2024-02-10T08:44:39.403222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.397690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.397690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":450,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402030+0000","last_change":"2024-02-10T08:42:47.240729+0000","last_active":"2024-02-10T08:44:39.402030+0000","last_peered":"2024-02-10T08:44:39.402030+0000","last_clean":"2024-02-10T08:44:39.402030+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:44:39.402030+0000","last_undegraded":"2024-02-10T08:44:39.402030+0000","last_fullsized":"2024-02-10T08:44:39.402030+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.240624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.240624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"481'9","reported_seq":656,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.401874+0000","last_change":"2024-02-10T08:43:36.077627+0000","last_active":"2024-02-10T08:44:39.401874+0000","last_peered":"2024-02-10T08:44:39.401874+0000","last_clean":"2024-02-10T08:43:31.610208+0000","last_became_active":"2024-02-10T08:43:36.077627+0000","last_became_peered":"2024-02-10T08:43:36.077627+0000","last_unstale":"2024-02-10T08:44:39.401874+0000","last_undegraded":"2024-02-10T08:43:36.066022+0000","last_fullsized":"2024-02-10T08:43:36.065899+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":550,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.086624+0000","last_change":"2024-02-10T08:43:36.064143+0000","last_active":"2024-02-10T08:44:44.086624+0000","last_peered":"2024-02-10T08:44:44.086624+0000","last_clean":"2024-02-10T08:43:25.763333+0000","last_became_active":"2024-02-10T08:43:36.064143+0000","last_became_peered":"2024-02-10T08:43:36.064143+0000","last_unstale":"2024-02-10T08:44:44.086624+0000","last_undegraded":"2024-02-10T08:43:36.057979+0000","last_fullsized":"2024-02-10T08:43:36.057862+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:43:34.883363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.883363+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":544,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.184124+0000","last_change":"2024-02-10T08:44:29.439534+0000","last_active":"2024-02-10T08:44:44.184124+0000","last_peered":"2024-02-10T08:44:44.184124+0000","last_clean":"2024-02-10T08:44:44.184124+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:44:44.184124+0000","last_undegraded":"2024-02-10T08:44:44.184124+0000","last_fullsized":"2024-02-10T08:44:44.184124+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.439473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.439473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135395+0000","last_change":"2024-02-10T08:44:31.574745+0000","last_active":"2024-02-10T08:44:44.135395+0000","last_peered":"2024-02-10T08:44:44.135395+0000","last_clean":"2024-02-10T08:44:44.135395+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:44:44.135395+0000","last_undegraded":"2024-02-10T08:44:44.135395+0000","last_fullsized":"2024-02-10T08:44:44.135395+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:31.574681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.574681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":427,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.335762+0000","last_change":"2024-02-10T08:43:36.073331+0000","last_active":"2024-02-10T08:44:40.335762+0000","last_peered":"2024-02-10T08:44:40.335762+0000","last_clean":"2024-02-10T08:43:31.059113+0000","last_became_active":"2024-02-10T08:43:36.073331+0000","last_became_peered":"2024-02-10T08:43:36.073331+0000","last_unstale":"2024-02-10T08:44:40.335762+0000","last_undegraded":"2024-02-10T08:44:40.335762+0000","last_fullsized":"2024-02-10T08:43:36.063737+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":505,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.402578+0000","last_change":"2024-02-10T08:43:36.082324+0000","last_active":"2024-02-10T08:44:39.402578+0000","last_peered":"2024-02-10T08:44:39.402578+0000","last_clean":"2024-02-10T08:43:11.068543+0000","last_became_active":"2024-02-10T08:43:36.082324+0000","last_became_peered":"2024-02-10T08:43:36.082324+0000","last_unstale":"2024-02-10T08:44:39.402578+0000","last_undegraded":"2024-02-10T08:44:39.402578+0000","last_fullsized":"2024-02-10T08:43:36.067905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.619193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.619193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":670,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.401781+0000","last_change":"2024-02-10T08:44:37.388097+0000","last_active":"2024-02-10T08:44:39.401781+0000","last_peered":"2024-02-10T08:44:39.401781+0000","last_clean":"2024-02-10T08:44:39.401781+0000","last_became_active":"2024-02-10T08:43:15.136661+0000","last_became_peered":"2024-02-10T08:43:15.136661+0000","last_unstale":"2024-02-10T08:44:39.401781+0000","last_undegraded":"2024-02-10T08:44:39.401781+0000","last_fullsized":"2024-02-10T08:44:39.401781+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:44:37.388018+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.388018+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":521,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.327888+0000","last_change":"2024-02-10T08:43:36.068713+0000","last_active":"2024-02-10T08:44:40.327888+0000","last_peered":"2024-02-10T08:44:40.327888+0000","last_clean":"2024-02-10T08:43:31.057619+0000","last_became_active":"2024-02-10T08:43:36.068713+0000","last_became_peered":"2024-02-10T08:43:36.068713+0000","last_unstale":"2024-02-10T08:44:40.327888+0000","last_undegraded":"2024-02-10T08:43:36.060168+0000","last_fullsized":"2024-02-10T08:43:36.060055+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":62,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.402255+0000","last_change":"2024-02-10T08:43:36.080856+0000","last_active":"2024-02-10T08:44:39.402255+0000","last_peered":"2024-02-10T08:44:39.402255+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.080856+0000","last_became_peered":"2024-02-10T08:43:36.080856+0000","last_unstale":"2024-02-10T08:44:39.402255+0000","last_undegraded":"2024-02-10T08:44:39.402255+0000","last_fullsized":"2024-02-10T08:43:36.066562+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":438,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402967+0000","last_change":"2024-02-10T08:43:00.547238+0000","last_active":"2024-02-10T08:44:39.402967+0000","last_peered":"2024-02-10T08:44:39.402967+0000","last_clean":"2024-02-10T08:44:39.402967+0000","last_became_active":"2024-02-10T08:43:00.546139+0000","last_became_peered":"2024-02-10T08:43:00.546139+0000","last_unstale":"2024-02-10T08:44:39.402967+0000","last_undegraded":"2024-02-10T08:44:39.402967+0000","last_fullsized":"2024-02-10T08:44:39.402967+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":522,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.183742+0000","last_change":"2024-02-10T08:43:59.374953+0000","last_active":"2024-02-10T08:44:44.183742+0000","last_peered":"2024-02-10T08:44:44.183742+0000","last_clean":"2024-02-10T08:44:44.183742+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:44:44.183742+0000","last_undegraded":"2024-02-10T08:44:44.183742+0000","last_fullsized":"2024-02-10T08:44:44.183742+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.374859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.374859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":561,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086496+0000","last_change":"2024-02-10T08:43:25.480698+0000","last_active":"2024-02-10T08:44:44.086496+0000","last_peered":"2024-02-10T08:44:44.086496+0000","last_clean":"2024-02-10T08:44:44.086496+0000","last_became_active":"2024-02-10T08:43:15.126404+0000","last_became_peered":"2024-02-10T08:43:15.126404+0000","last_unstale":"2024-02-10T08:44:44.086496+0000","last_undegraded":"2024-02-10T08:44:44.086496+0000","last_fullsized":"2024-02-10T08:44:44.086496+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'21","last_scrub_stamp":"2024-02-10T08:43:25.480620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.480620+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.135828+0000","last_change":"2024-02-10T08:43:36.073770+0000","last_active":"2024-02-10T08:44:44.135828+0000","last_peered":"2024-02-10T08:44:44.135828+0000","last_clean":"2024-02-10T08:43:31.060583+0000","last_became_active":"2024-02-10T08:43:36.073770+0000","last_became_peered":"2024-02-10T08:43:36.073770+0000","last_unstale":"2024-02-10T08:44:44.135828+0000","last_undegraded":"2024-02-10T08:44:44.135828+0000","last_fullsized":"2024-02-10T08:43:36.061914+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.297949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.297949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":437,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313577+0000","last_change":"2024-02-10T08:44:24.395820+0000","last_active":"2024-02-10T08:44:40.313577+0000","last_peered":"2024-02-10T08:44:40.313577+0000","last_clean":"2024-02-10T08:44:40.313577+0000","last_became_active":"2024-02-10T08:43:15.106310+0000","last_became_peered":"2024-02-10T08:43:15.106310+0000","last_unstale":"2024-02-10T08:44:40.313577+0000","last_undegraded":"2024-02-10T08:44:40.313577+0000","last_fullsized":"2024-02-10T08:44:40.313577+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.395744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.395744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":465,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315827+0000","last_change":"2024-02-10T08:43:59.552708+0000","last_active":"2024-02-10T08:44:40.315827+0000","last_peered":"2024-02-10T08:44:40.315827+0000","last_clean":"2024-02-10T08:44:40.315827+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:44:40.315827+0000","last_undegraded":"2024-02-10T08:44:40.315827+0000","last_fullsized":"2024-02-10T08:44:40.315827+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.552604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.552604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":542,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.087693+0000","last_change":"2024-02-10T08:43:36.069190+0000","last_active":"2024-02-10T08:44:44.087693+0000","last_peered":"2024-02-10T08:44:44.087693+0000","last_clean":"2024-02-10T08:43:31.062577+0000","last_became_active":"2024-02-10T08:43:36.069190+0000","last_became_peered":"2024-02-10T08:43:36.069190+0000","last_unstale":"2024-02-10T08:44:44.087693+0000","last_undegraded":"2024-02-10T08:44:44.087693+0000","last_fullsized":"2024-02-10T08:43:36.058346+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.561295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"473'24","reported_seq":539,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.723182+0000","last_change":"2024-02-10T08:44:40.723157+0000","last_active":"2024-02-10T08:44:40.723182+0000","last_peered":"2024-02-10T08:44:40.723182+0000","last_clean":"2024-02-10T08:44:40.723182+0000","last_became_active":"2024-02-10T08:43:15.129639+0000","last_became_peered":"2024-02-10T08:43:15.129639+0000","last_unstale":"2024-02-10T08:44:40.723182+0000","last_undegraded":"2024-02-10T08:44:40.723182+0000","last_fullsized":"2024-02-10T08:44:40.723182+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'24","last_scrub_stamp":"2024-02-10T08:44:40.723096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.723096+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134413+0000","last_change":"2024-02-10T08:44:36.570760+0000","last_active":"2024-02-10T08:44:44.134413+0000","last_peered":"2024-02-10T08:44:44.134413+0000","last_clean":"2024-02-10T08:44:44.134413+0000","last_became_active":"2024-02-10T08:43:15.132762+0000","last_became_peered":"2024-02-10T08:43:15.132762+0000","last_unstale":"2024-02-10T08:44:44.134413+0000","last_undegraded":"2024-02-10T08:44:44.134413+0000","last_fullsized":"2024-02-10T08:44:44.134413+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.570697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.570697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.402454+0000","last_change":"2024-02-10T08:43:36.072974+0000","last_active":"2024-02-10T08:44:39.402454+0000","last_peered":"2024-02-10T08:44:39.402454+0000","last_clean":"2024-02-10T08:33:17.081938+0000","last_became_active":"2024-02-10T08:43:36.072974+0000","last_became_peered":"2024-02-10T08:43:36.072974+0000","last_unstale":"2024-02-10T08:44:39.402454+0000","last_undegraded":"2024-02-10T08:44:39.402454+0000","last_fullsized":"2024-02-10T08:43:36.065921+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":513,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.331511+0000","last_change":"2024-02-10T08:44:38.734244+0000","last_active":"2024-02-10T08:44:40.331511+0000","last_peered":"2024-02-10T08:44:40.331511+0000","last_clean":"2024-02-10T08:44:40.331511+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:44:40.331511+0000","last_undegraded":"2024-02-10T08:44:40.331511+0000","last_fullsized":"2024-02-10T08:44:40.331511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:38.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.734157+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":469,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.183473+0000","last_change":"2024-02-10T08:44:23.462760+0000","last_active":"2024-02-10T08:44:44.183473+0000","last_peered":"2024-02-10T08:44:44.183473+0000","last_clean":"2024-02-10T08:44:44.183473+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:44:44.183473+0000","last_undegraded":"2024-02-10T08:44:44.183473+0000","last_fullsized":"2024-02-10T08:44:44.183473+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:23.462674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:23.462674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":91,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135138+0000","last_change":"2024-02-10T08:43:56.338478+0000","last_active":"2024-02-10T08:44:44.135138+0000","last_peered":"2024-02-10T08:44:44.135138+0000","last_clean":"2024-02-10T08:44:44.135138+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:44:44.135138+0000","last_undegraded":"2024-02-10T08:44:44.135138+0000","last_fullsized":"2024-02-10T08:44:44.135138+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:56.338378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:56.338378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":606,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.182611+0000","last_change":"2024-02-10T08:43:36.076013+0000","last_active":"2024-02-10T08:44:44.182611+0000","last_peered":"2024-02-10T08:44:44.182611+0000","last_clean":"2024-02-10T08:43:31.155643+0000","last_became_active":"2024-02-10T08:43:36.076013+0000","last_became_peered":"2024-02-10T08:43:36.076013+0000","last_unstale":"2024-02-10T08:44:44.182611+0000","last_undegraded":"2024-02-10T08:43:36.064979+0000","last_fullsized":"2024-02-10T08:43:36.064906+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134743+0000","last_change":"2024-02-10T08:44:24.600149+0000","last_active":"2024-02-10T08:44:44.134743+0000","last_peered":"2024-02-10T08:44:44.134743+0000","last_clean":"2024-02-10T08:44:44.134743+0000","last_became_active":"2024-02-10T08:43:15.135366+0000","last_became_peered":"2024-02-10T08:43:15.135366+0000","last_unstale":"2024-02-10T08:44:44.134743+0000","last_undegraded":"2024-02-10T08:44:44.134743+0000","last_fullsized":"2024-02-10T08:44:44.134743+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.600075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.600075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":531,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086405+0000","last_change":"2024-02-10T08:44:26.266252+0000","last_active":"2024-02-10T08:44:44.086405+0000","last_peered":"2024-02-10T08:44:44.086405+0000","last_clean":"2024-02-10T08:44:44.086405+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:44:44.086405+0000","last_undegraded":"2024-02-10T08:44:44.086405+0000","last_fullsized":"2024-02-10T08:44:44.086405+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.266165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.266165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":506,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402411+0000","last_change":"2024-02-10T08:44:00.274641+0000","last_active":"2024-02-10T08:44:39.402411+0000","last_peered":"2024-02-10T08:44:39.402411+0000","last_clean":"2024-02-10T08:44:39.402411+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:44:39.402411+0000","last_undegraded":"2024-02-10T08:44:39.402411+0000","last_fullsized":"2024-02-10T08:44:39.402411+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.274550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":508,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.087724+0000","last_change":"2024-02-10T08:43:36.077071+0000","last_active":"2024-02-10T08:44:44.087724+0000","last_peered":"2024-02-10T08:44:44.087724+0000","last_clean":"2024-02-10T08:43:31.062622+0000","last_became_active":"2024-02-10T08:43:36.077071+0000","last_became_peered":"2024-02-10T08:43:36.077071+0000","last_unstale":"2024-02-10T08:44:44.087724+0000","last_undegraded":"2024-02-10T08:44:44.087724+0000","last_fullsized":"2024-02-10T08:43:36.058617+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":462,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086375+0000","last_change":"2024-02-10T08:42:02.124479+0000","last_active":"2024-02-10T08:44:44.086375+0000","last_peered":"2024-02-10T08:44:44.086375+0000","last_clean":"2024-02-10T08:44:44.086375+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:44:44.086375+0000","last_undegraded":"2024-02-10T08:44:44.086375+0000","last_fullsized":"2024-02-10T08:44:44.086375+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.124401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.124401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":540,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.316278+0000","last_change":"2024-02-10T08:43:32.421461+0000","last_active":"2024-02-10T08:44:40.316278+0000","last_peered":"2024-02-10T08:44:40.316278+0000","last_clean":"2024-02-10T08:44:40.316278+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:44:40.316278+0000","last_undegraded":"2024-02-10T08:44:40.316278+0000","last_fullsized":"2024-02-10T08:44:40.316278+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:32.421384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:32.421384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":607,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.327162+0000","last_change":"2024-02-10T08:43:24.674606+0000","last_active":"2024-02-10T08:44:40.327162+0000","last_peered":"2024-02-10T08:44:40.327162+0000","last_clean":"2024-02-10T08:44:40.327162+0000","last_became_active":"2024-02-10T08:43:15.129755+0000","last_became_peered":"2024-02-10T08:43:15.129755+0000","last_unstale":"2024-02-10T08:44:40.327162+0000","last_undegraded":"2024-02-10T08:44:40.327162+0000","last_fullsized":"2024-02-10T08:44:40.327162+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2024-02-10T08:43:24.674531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:24.674531+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":524,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313505+0000","last_change":"2024-02-10T08:44:37.469860+0000","last_active":"2024-02-10T08:44:40.313505+0000","last_peered":"2024-02-10T08:44:40.313505+0000","last_clean":"2024-02-10T08:44:40.313505+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:44:40.313505+0000","last_undegraded":"2024-02-10T08:44:40.313505+0000","last_fullsized":"2024-02-10T08:44:40.313505+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:37.469796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.469796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":1192,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.410100+0000","last_change":"2024-02-10T08:43:58.735413+0000","last_active":"2024-02-10T08:44:40.410100+0000","last_peered":"2024-02-10T08:44:40.410100+0000","last_clean":"2024-02-10T08:44:40.410100+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:44:40.410100+0000","last_undegraded":"2024-02-10T08:44:40.410100+0000","last_fullsized":"2024-02-10T08:44:40.410100+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:43:58.735346+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.735346+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":675,"num_read_kb":670,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":84,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135736+0000","last_change":"2024-02-10T08:43:15.134639+0000","last_active":"2024-02-10T08:44:44.135736+0000","last_peered":"2024-02-10T08:44:44.135736+0000","last_clean":"2024-02-10T08:44:44.135736+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:44:44.135736+0000","last_undegraded":"2024-02-10T08:44:44.135736+0000","last_fullsized":"2024-02-10T08:44:44.135736+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.329911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.329911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":483,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.327424+0000","last_change":"2024-02-10T08:44:36.777644+0000","last_active":"2024-02-10T08:44:40.327424+0000","last_peered":"2024-02-10T08:44:40.327424+0000","last_clean":"2024-02-10T08:44:40.327424+0000","last_became_active":"2024-02-10T08:43:15.129900+0000","last_became_peered":"2024-02-10T08:43:15.129900+0000","last_unstale":"2024-02-10T08:44:40.327424+0000","last_undegraded":"2024-02-10T08:44:40.327424+0000","last_fullsized":"2024-02-10T08:44:40.327424+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.777533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":427,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.335297+0000","last_change":"2024-02-10T08:43:36.073644+0000","last_active":"2024-02-10T08:44:40.335297+0000","last_peered":"2024-02-10T08:44:40.335297+0000","last_clean":"2024-02-10T08:43:31.058736+0000","last_became_active":"2024-02-10T08:43:36.073644+0000","last_became_peered":"2024-02-10T08:43:36.073644+0000","last_unstale":"2024-02-10T08:44:40.335297+0000","last_undegraded":"2024-02-10T08:44:40.335297+0000","last_fullsized":"2024-02-10T08:43:36.062919+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"473'41","reported_seq":653,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087034+0000","last_change":"2024-02-10T08:42:43.109193+0000","last_active":"2024-02-10T08:44:44.087034+0000","last_peered":"2024-02-10T08:44:44.087034+0000","last_clean":"2024-02-10T08:44:44.087034+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:44:44.087034+0000","last_undegraded":"2024-02-10T08:44:44.087034+0000","last_fullsized":"2024-02-10T08:44:44.087034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:42:43.109121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.109121+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":513,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086838+0000","last_change":"2024-02-10T08:42:37.145252+0000","last_active":"2024-02-10T08:44:44.086838+0000","last_peered":"2024-02-10T08:44:44.086838+0000","last_clean":"2024-02-10T08:44:44.086838+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:44:44.086838+0000","last_undegraded":"2024-02-10T08:44:44.086838+0000","last_fullsized":"2024-02-10T08:44:44.086838+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.145180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.145180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":519,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.405921+0000","last_change":"2024-02-10T08:43:36.083021+0000","last_active":"2024-02-10T08:44:39.405921+0000","last_peered":"2024-02-10T08:44:39.405921+0000","last_clean":"2024-02-10T08:43:31.058994+0000","last_became_active":"2024-02-10T08:43:36.083021+0000","last_became_peered":"2024-02-10T08:43:36.083021+0000","last_unstale":"2024-02-10T08:44:39.405921+0000","last_undegraded":"2024-02-10T08:44:39.405921+0000","last_fullsized":"2024-02-10T08:43:36.072420+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":483,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313830+0000","last_change":"2024-02-10T08:44:30.426450+0000","last_active":"2024-02-10T08:44:40.313830+0000","last_peered":"2024-02-10T08:44:40.313830+0000","last_clean":"2024-02-10T08:44:40.313830+0000","last_became_active":"2024-02-10T08:43:15.119340+0000","last_became_peered":"2024-02-10T08:43:15.119340+0000","last_unstale":"2024-02-10T08:44:40.313830+0000","last_undegraded":"2024-02-10T08:44:40.313830+0000","last_fullsized":"2024-02-10T08:44:40.313830+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:30.426345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.426345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":549,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.185415+0000","last_change":"2024-02-10T08:44:39.424567+0000","last_active":"2024-02-10T08:44:44.185415+0000","last_peered":"2024-02-10T08:44:44.185415+0000","last_clean":"2024-02-10T08:44:44.185415+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:44:44.185415+0000","last_undegraded":"2024-02-10T08:44:44.185415+0000","last_fullsized":"2024-02-10T08:44:44.185415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:44:39.424472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.424472+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":554,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.321678+0000","last_change":"2024-02-10T08:43:36.071773+0000","last_active":"2024-02-10T08:44:40.321678+0000","last_peered":"2024-02-10T08:44:40.321678+0000","last_clean":"2024-02-10T08:43:31.056280+0000","last_became_active":"2024-02-10T08:43:36.071773+0000","last_became_peered":"2024-02-10T08:43:36.071773+0000","last_unstale":"2024-02-10T08:44:40.321678+0000","last_undegraded":"2024-02-10T08:43:36.059969+0000","last_fullsized":"2024-02-10T08:43:36.059851+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:42:40.687013+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.687013+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":540,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.331100+0000","last_change":"2024-02-10T08:43:27.605903+0000","last_active":"2024-02-10T08:44:40.331100+0000","last_peered":"2024-02-10T08:44:40.331100+0000","last_clean":"2024-02-10T08:44:40.331100+0000","last_became_active":"2024-02-10T08:43:15.129787+0000","last_became_peered":"2024-02-10T08:43:15.129787+0000","last_unstale":"2024-02-10T08:44:40.331100+0000","last_undegraded":"2024-02-10T08:44:40.331100+0000","last_fullsized":"2024-02-10T08:44:40.331100+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.605831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.605831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":420,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.401696+0000","last_change":"2024-02-10T08:42:40.143847+0000","last_active":"2024-02-10T08:44:39.401696+0000","last_peered":"2024-02-10T08:44:39.401696+0000","last_clean":"2024-02-10T08:44:39.401696+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:44:39.401696+0000","last_undegraded":"2024-02-10T08:44:39.401696+0000","last_fullsized":"2024-02-10T08:44:39.401696+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.143787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.143787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":450,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405704+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:44:39.405704+0000","last_peered":"2024-02-10T08:44:39.405704+0000","last_clean":"2024-02-10T08:44:39.405704+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:44:39.405704+0000","last_undegraded":"2024-02-10T08:44:39.405704+0000","last_fullsized":"2024-02-10T08:44:39.405704+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":536,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.314161+0000","last_change":"2024-02-10T08:43:58.505496+0000","last_active":"2024-02-10T08:44:40.314161+0000","last_peered":"2024-02-10T08:44:40.314161+0000","last_clean":"2024-02-10T08:44:40.314161+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:44:40.314161+0000","last_undegraded":"2024-02-10T08:44:40.314161+0000","last_fullsized":"2024-02-10T08:44:40.314161+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.505431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.505431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":541,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.313605+0000","last_change":"2024-02-10T08:43:36.079165+0000","last_active":"2024-02-10T08:44:40.313605+0000","last_peered":"2024-02-10T08:44:40.313605+0000","last_clean":"2024-02-10T08:43:25.579489+0000","last_became_active":"2024-02-10T08:43:36.079165+0000","last_became_peered":"2024-02-10T08:43:36.079165+0000","last_unstale":"2024-02-10T08:44:40.313605+0000","last_undegraded":"2024-02-10T08:43:36.061039+0000","last_fullsized":"2024-02-10T08:43:36.060910+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":545,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.137337+0000","last_change":"2024-02-10T08:43:36.072147+0000","last_active":"2024-02-10T08:44:44.137337+0000","last_peered":"2024-02-10T08:44:44.137337+0000","last_clean":"2024-02-10T08:43:11.810746+0000","last_became_active":"2024-02-10T08:43:36.072147+0000","last_became_peered":"2024-02-10T08:43:36.072147+0000","last_unstale":"2024-02-10T08:44:44.137337+0000","last_undegraded":"2024-02-10T08:43:36.061852+0000","last_fullsized":"2024-02-10T08:43:36.061736+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:43:11.810639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.810639+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":518,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.404900+0000","last_change":"2024-02-10T08:43:15.126259+0000","last_active":"2024-02-10T08:44:39.404900+0000","last_peered":"2024-02-10T08:44:39.404900+0000","last_clean":"2024-02-10T08:44:39.404900+0000","last_became_active":"2024-02-10T08:43:15.125756+0000","last_became_peered":"2024-02-10T08:43:15.125756+0000","last_unstale":"2024-02-10T08:44:39.404900+0000","last_undegraded":"2024-02-10T08:44:39.404900+0000","last_fullsized":"2024-02-10T08:44:39.404900+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":474,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315220+0000","last_change":"2024-02-10T08:43:54.509761+0000","last_active":"2024-02-10T08:44:40.315220+0000","last_peered":"2024-02-10T08:44:40.315220+0000","last_clean":"2024-02-10T08:44:40.315220+0000","last_became_active":"2024-02-10T08:43:15.119419+0000","last_became_peered":"2024-02-10T08:43:15.119419+0000","last_unstale":"2024-02-10T08:44:40.315220+0000","last_undegraded":"2024-02-10T08:44:40.315220+0000","last_fullsized":"2024-02-10T08:44:40.315220+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:54.509649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:54.509649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":538,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405948+0000","last_change":"2024-02-10T08:44:38.409588+0000","last_active":"2024-02-10T08:44:39.405948+0000","last_peered":"2024-02-10T08:44:39.405948+0000","last_clean":"2024-02-10T08:44:39.405948+0000","last_became_active":"2024-02-10T08:43:15.137834+0000","last_became_peered":"2024-02-10T08:43:15.137834+0000","last_unstale":"2024-02-10T08:44:39.405948+0000","last_undegraded":"2024-02-10T08:44:39.405948+0000","last_fullsized":"2024-02-10T08:44:39.405948+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.409486+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":523,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.314037+0000","last_change":"2024-02-10T08:43:36.075991+0000","last_active":"2024-02-10T08:44:40.314037+0000","last_peered":"2024-02-10T08:44:40.314037+0000","last_clean":"2024-02-10T08:43:31.065892+0000","last_became_active":"2024-02-10T08:43:36.075991+0000","last_became_peered":"2024-02-10T08:43:36.075991+0000","last_unstale":"2024-02-10T08:44:40.314037+0000","last_undegraded":"2024-02-10T08:44:40.314037+0000","last_fullsized":"2024-02-10T08:43:36.062400+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":526,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.315515+0000","last_change":"2024-02-10T08:43:36.075699+0000","last_active":"2024-02-10T08:44:40.315515+0000","last_peered":"2024-02-10T08:44:40.315515+0000","last_clean":"2024-02-10T08:43:31.066554+0000","last_became_active":"2024-02-10T08:43:36.075699+0000","last_became_peered":"2024-02-10T08:43:36.075699+0000","last_unstale":"2024-02-10T08:44:40.315515+0000","last_undegraded":"2024-02-10T08:44:40.315515+0000","last_fullsized":"2024-02-10T08:43:36.063912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.347685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.347685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":451,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.321084+0000","last_change":"2024-02-10T08:43:36.075546+0000","last_active":"2024-02-10T08:44:40.321084+0000","last_peered":"2024-02-10T08:44:40.321084+0000","last_clean":"2024-02-10T08:43:11.068374+0000","last_became_active":"2024-02-10T08:43:36.075546+0000","last_became_peered":"2024-02-10T08:43:36.075546+0000","last_unstale":"2024-02-10T08:44:40.321084+0000","last_undegraded":"2024-02-10T08:44:40.321084+0000","last_fullsized":"2024-02-10T08:43:36.057136+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":556,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.406049+0000","last_change":"2024-02-10T08:43:52.173893+0000","last_active":"2024-02-10T08:44:39.406049+0000","last_peered":"2024-02-10T08:44:39.406049+0000","last_clean":"2024-02-10T08:44:39.406049+0000","last_became_active":"2024-02-10T08:43:15.125753+0000","last_became_peered":"2024-02-10T08:43:15.125753+0000","last_unstale":"2024-02-10T08:44:39.406049+0000","last_undegraded":"2024-02-10T08:44:39.406049+0000","last_fullsized":"2024-02-10T08:44:39.406049+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:43:52.173797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:52.173797+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":518,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.180761+0000","last_change":"2024-02-10T08:43:36.068896+0000","last_active":"2024-02-10T08:44:44.180761+0000","last_peered":"2024-02-10T08:44:44.180761+0000","last_clean":"2024-02-10T08:43:31.156388+0000","last_became_active":"2024-02-10T08:43:36.068896+0000","last_became_peered":"2024-02-10T08:43:36.068896+0000","last_unstale":"2024-02-10T08:44:44.180761+0000","last_undegraded":"2024-02-10T08:44:44.180761+0000","last_fullsized":"2024-02-10T08:43:36.062533+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.135531+0000","last_change":"2024-02-10T08:43:36.074707+0000","last_active":"2024-02-10T08:44:44.135531+0000","last_peered":"2024-02-10T08:44:44.135531+0000","last_clean":"2024-02-10T08:43:31.060201+0000","last_became_active":"2024-02-10T08:43:36.074707+0000","last_became_peered":"2024-02-10T08:43:36.074707+0000","last_unstale":"2024-02-10T08:44:44.135531+0000","last_undegraded":"2024-02-10T08:44:44.135531+0000","last_fullsized":"2024-02-10T08:43:36.061288+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":457,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.086364+0000","last_change":"2024-02-10T08:43:36.065698+0000","last_active":"2024-02-10T08:44:44.086364+0000","last_peered":"2024-02-10T08:44:44.086364+0000","last_clean":"2024-02-10T08:43:31.061677+0000","last_became_active":"2024-02-10T08:43:36.065698+0000","last_became_peered":"2024-02-10T08:43:36.065698+0000","last_unstale":"2024-02-10T08:44:44.086364+0000","last_undegraded":"2024-02-10T08:44:44.086364+0000","last_fullsized":"2024-02-10T08:43:36.058144+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":576,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.403302+0000","last_change":"2024-02-10T08:43:19.438856+0000","last_active":"2024-02-10T08:44:39.403302+0000","last_peered":"2024-02-10T08:44:39.403302+0000","last_clean":"2024-02-10T08:44:39.403302+0000","last_became_active":"2024-02-10T08:43:15.137251+0000","last_became_peered":"2024-02-10T08:43:15.137251+0000","last_unstale":"2024-02-10T08:44:39.403302+0000","last_undegraded":"2024-02-10T08:44:39.403302+0000","last_fullsized":"2024-02-10T08:44:39.403302+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":531,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086763+0000","last_change":"2024-02-10T08:43:15.112162+0000","last_active":"2024-02-10T08:44:44.086763+0000","last_peered":"2024-02-10T08:44:44.086763+0000","last_clean":"2024-02-10T08:44:44.086763+0000","last_became_active":"2024-02-10T08:43:15.111824+0000","last_became_peered":"2024-02-10T08:43:15.111824+0000","last_unstale":"2024-02-10T08:44:44.086763+0000","last_undegraded":"2024-02-10T08:44:44.086763+0000","last_fullsized":"2024-02-10T08:44:44.086763+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:42.059708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.059708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":662,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086835+0000","last_change":"2024-02-10T08:42:28.090195+0000","last_active":"2024-02-10T08:44:44.086835+0000","last_peered":"2024-02-10T08:44:44.086835+0000","last_clean":"2024-02-10T08:44:44.086835+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:44:44.086835+0000","last_undegraded":"2024-02-10T08:44:44.086835+0000","last_fullsized":"2024-02-10T08:44:44.086835+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:42:28.090111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.090111+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":95,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.136606+0000","last_change":"2024-02-10T08:43:36.078925+0000","last_active":"2024-02-10T08:44:44.136606+0000","last_peered":"2024-02-10T08:44:44.136606+0000","last_clean":"2024-02-10T08:43:31.060910+0000","last_became_active":"2024-02-10T08:43:36.078925+0000","last_became_peered":"2024-02-10T08:43:36.078925+0000","last_unstale":"2024-02-10T08:44:44.136606+0000","last_undegraded":"2024-02-10T08:44:44.136606+0000","last_fullsized":"2024-02-10T08:43:36.063718+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.241405+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":431,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.180411+0000","last_change":"2024-02-10T08:43:36.067846+0000","last_active":"2024-02-10T08:44:44.180411+0000","last_peered":"2024-02-10T08:44:44.180411+0000","last_clean":"2024-02-10T08:43:31.156305+0000","last_became_active":"2024-02-10T08:43:36.067846+0000","last_became_peered":"2024-02-10T08:43:36.067846+0000","last_unstale":"2024-02-10T08:44:44.180411+0000","last_undegraded":"2024-02-10T08:44:44.180411+0000","last_fullsized":"2024-02-10T08:43:36.063242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.357151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.357151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":569,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086682+0000","last_change":"2024-02-10T08:43:15.941418+0000","last_active":"2024-02-10T08:44:44.086682+0000","last_peered":"2024-02-10T08:44:44.086682+0000","last_clean":"2024-02-10T08:44:44.086682+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:44:44.086682+0000","last_undegraded":"2024-02-10T08:44:44.086682+0000","last_fullsized":"2024-02-10T08:44:44.086682+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:42:05.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.106466+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":22,"reported_epoch":534,"state":"active+clean+remapped","last_fresh":"2024-02-10T08:44:40.314700+0000","last_change":"2024-02-10T08:44:31.113549+0000","last_active":"2024-02-10T08:44:40.314700+0000","last_peered":"2024-02-10T08:44:40.314700+0000","last_clean":"2024-02-10T08:44:40.314700+0000","last_became_active":"2024-02-10T08:44:31.113168+0000","last_became_peered":"2024-02-10T08:44:31.113168+0000","last_unstale":"2024-02-10T08:44:40.314700+0000","last_undegraded":"2024-02-10T08:44:40.314700+0000","last_fullsized":"2024-02-10T08:44:40.314700+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:10.801476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.801476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":673,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.403491+0000","last_change":"2024-02-10T08:43:19.433673+0000","last_active":"2024-02-10T08:44:39.403491+0000","last_peered":"2024-02-10T08:44:39.403491+0000","last_clean":"2024-02-10T08:44:39.403491+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:44:39.403491+0000","last_undegraded":"2024-02-10T08:44:39.403491+0000","last_fullsized":"2024-02-10T08:44:39.403491+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":63,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.316066+0000","last_change":"2024-02-10T08:43:36.075017+0000","last_active":"2024-02-10T08:44:40.316066+0000","last_peered":"2024-02-10T08:44:40.316066+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.075017+0000","last_became_peered":"2024-02-10T08:43:36.075017+0000","last_unstale":"2024-02-10T08:44:40.316066+0000","last_undegraded":"2024-02-10T08:44:40.316066+0000","last_fullsized":"2024-02-10T08:43:36.058808+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.786130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.786130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":464,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313310+0000","last_change":"2024-02-10T08:42:40.371733+0000","last_active":"2024-02-10T08:44:40.313310+0000","last_peered":"2024-02-10T08:44:40.313310+0000","last_clean":"2024-02-10T08:44:40.313310+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:44:40.313310+0000","last_undegraded":"2024-02-10T08:44:40.313310+0000","last_fullsized":"2024-02-10T08:44:40.313310+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:42:40.371655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.371655+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":542,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.321479+0000","last_change":"2024-02-10T08:44:30.727741+0000","last_active":"2024-02-10T08:44:40.321479+0000","last_peered":"2024-02-10T08:44:40.321479+0000","last_clean":"2024-02-10T08:44:40.321479+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:44:40.321479+0000","last_undegraded":"2024-02-10T08:44:40.321479+0000","last_fullsized":"2024-02-10T08:44:40.321479+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:30.727654+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.727654+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":517,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.181798+0000","last_change":"2024-02-10T08:43:53.319974+0000","last_active":"2024-02-10T08:44:44.181798+0000","last_peered":"2024-02-10T08:44:44.181798+0000","last_clean":"2024-02-10T08:44:44.181798+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:44:44.181798+0000","last_undegraded":"2024-02-10T08:44:44.181798+0000","last_fullsized":"2024-02-10T08:44:44.181798+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:53.319909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:53.319909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":511,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.404814+0000","last_change":"2024-02-10T08:41:48.722133+0000","last_active":"2024-02-10T08:44:39.404814+0000","last_peered":"2024-02-10T08:44:39.404814+0000","last_clean":"2024-02-10T08:44:39.404814+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:44:39.404814+0000","last_undegraded":"2024-02-10T08:44:39.404814+0000","last_fullsized":"2024-02-10T08:44:39.404814+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.722068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.722068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":426,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086025+0000","last_change":"2024-02-10T08:43:59.469912+0000","last_active":"2024-02-10T08:44:44.086025+0000","last_peered":"2024-02-10T08:44:44.086025+0000","last_clean":"2024-02-10T08:44:44.086025+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:44:44.086025+0000","last_undegraded":"2024-02-10T08:44:44.086025+0000","last_fullsized":"2024-02-10T08:44:44.086025+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.469850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.469850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":471,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.182530+0000","last_change":"2024-02-10T08:44:25.447359+0000","last_active":"2024-02-10T08:44:44.182530+0000","last_peered":"2024-02-10T08:44:44.182530+0000","last_clean":"2024-02-10T08:44:44.182530+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:44:44.182530+0000","last_undegraded":"2024-02-10T08:44:44.182530+0000","last_fullsized":"2024-02-10T08:44:44.182530+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.447293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.447293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":532,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.185269+0000","last_change":"2024-02-10T08:43:36.069761+0000","last_active":"2024-02-10T08:44:44.185269+0000","last_peered":"2024-02-10T08:44:44.185269+0000","last_clean":"2024-02-10T08:43:25.738370+0000","last_became_active":"2024-02-10T08:43:36.069761+0000","last_became_peered":"2024-02-10T08:43:36.069761+0000","last_unstale":"2024-02-10T08:44:44.185269+0000","last_undegraded":"2024-02-10T08:43:36.063371+0000","last_fullsized":"2024-02-10T08:43:36.063247+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:43:28.880851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:28.880851+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":524,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.335465+0000","last_change":"2024-02-10T08:43:59.760782+0000","last_active":"2024-02-10T08:44:40.335465+0000","last_peered":"2024-02-10T08:44:40.335465+0000","last_clean":"2024-02-10T08:44:40.335465+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:44:40.335465+0000","last_undegraded":"2024-02-10T08:44:40.335465+0000","last_fullsized":"2024-02-10T08:44:40.335465+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.760721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.760721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":509,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.321568+0000","last_change":"2024-02-10T08:44:35.784832+0000","last_active":"2024-02-10T08:44:40.321568+0000","last_peered":"2024-02-10T08:44:40.321568+0000","last_clean":"2024-02-10T08:44:40.321568+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:44:40.321568+0000","last_undegraded":"2024-02-10T08:44:40.321568+0000","last_fullsized":"2024-02-10T08:44:40.321568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.784716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":509,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402125+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:44:39.402125+0000","last_peered":"2024-02-10T08:44:39.402125+0000","last_clean":"2024-02-10T08:44:39.402125+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:44:39.402125+0000","last_undegraded":"2024-02-10T08:44:39.402125+0000","last_fullsized":"2024-02-10T08:44:39.402125+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":465,"reported_epoch":533,"state":"active+undersized","last_fresh":"2024-02-10T08:44:39.405960+0000","last_change":"2024-02-10T08:43:36.081224+0000","last_active":"2024-02-10T08:44:39.405960+0000","last_peered":"2024-02-10T08:44:39.405960+0000","last_clean":"2024-02-10T08:43:31.059324+0000","last_became_active":"2024-02-10T08:43:36.081224+0000","last_became_peered":"2024-02-10T08:43:36.081224+0000","last_unstale":"2024-02-10T08:44:39.405960+0000","last_undegraded":"2024-02-10T08:44:39.405960+0000","last_fullsized":"2024-02-10T08:43:36.068534+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":516,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.401685+0000","last_change":"2024-02-10T08:43:34.096276+0000","last_active":"2024-02-10T08:44:39.401685+0000","last_peered":"2024-02-10T08:44:39.401685+0000","last_clean":"2024-02-10T08:44:39.401685+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:44:39.401685+0000","last_undegraded":"2024-02-10T08:44:39.401685+0000","last_fullsized":"2024-02-10T08:44:39.401685+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:43:34.096208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.096208+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":517,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087592+0000","last_change":"2024-02-10T08:43:23.521301+0000","last_active":"2024-02-10T08:44:44.087592+0000","last_peered":"2024-02-10T08:44:44.087592+0000","last_clean":"2024-02-10T08:44:44.087592+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:44:44.087592+0000","last_undegraded":"2024-02-10T08:44:44.087592+0000","last_fullsized":"2024-02-10T08:44:44.087592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.521227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.521227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":534,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.180625+0000","last_change":"2024-02-10T08:43:36.072045+0000","last_active":"2024-02-10T08:44:44.180625+0000","last_peered":"2024-02-10T08:44:44.180625+0000","last_clean":"2024-02-10T08:43:31.155301+0000","last_became_active":"2024-02-10T08:43:36.072045+0000","last_became_peered":"2024-02-10T08:43:36.072045+0000","last_unstale":"2024-02-10T08:44:44.180625+0000","last_undegraded":"2024-02-10T08:44:44.180625+0000","last_fullsized":"2024-02-10T08:43:36.062267+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.508636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.508636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.136766+0000","last_change":"2024-02-10T08:43:36.072813+0000","last_active":"2024-02-10T08:44:44.136766+0000","last_peered":"2024-02-10T08:44:44.136766+0000","last_clean":"2024-02-10T08:43:31.060566+0000","last_became_active":"2024-02-10T08:43:36.072813+0000","last_became_peered":"2024-02-10T08:43:36.072813+0000","last_unstale":"2024-02-10T08:44:44.136766+0000","last_undegraded":"2024-02-10T08:44:44.136766+0000","last_fullsized":"2024-02-10T08:43:36.061933+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":468,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.403081+0000","last_change":"2024-02-10T08:43:15.137758+0000","last_active":"2024-02-10T08:44:39.403081+0000","last_peered":"2024-02-10T08:44:39.403081+0000","last_clean":"2024-02-10T08:44:39.403081+0000","last_became_active":"2024-02-10T08:43:15.137437+0000","last_became_peered":"2024-02-10T08:43:15.137437+0000","last_unstale":"2024-02-10T08:44:39.403081+0000","last_undegraded":"2024-02-10T08:44:39.403081+0000","last_fullsized":"2024-02-10T08:44:39.403081+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.116858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":577,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402702+0000","last_change":"2024-02-10T08:44:28.331732+0000","last_active":"2024-02-10T08:44:39.402702+0000","last_peered":"2024-02-10T08:44:39.402702+0000","last_clean":"2024-02-10T08:44:39.402702+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:44:39.402702+0000","last_undegraded":"2024-02-10T08:44:39.402702+0000","last_fullsized":"2024-02-10T08:44:39.402702+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:28.331601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.331601+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":517,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.184056+0000","last_change":"2024-02-10T08:43:58.346958+0000","last_active":"2024-02-10T08:44:44.184056+0000","last_peered":"2024-02-10T08:44:44.184056+0000","last_clean":"2024-02-10T08:44:44.184056+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:44:44.184056+0000","last_undegraded":"2024-02-10T08:44:44.184056+0000","last_fullsized":"2024-02-10T08:44:44.184056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.346901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.346901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.134532+0000","last_change":"2024-02-10T08:43:36.074002+0000","last_active":"2024-02-10T08:44:44.134532+0000","last_peered":"2024-02-10T08:44:44.134532+0000","last_clean":"2024-02-10T08:43:31.060062+0000","last_became_active":"2024-02-10T08:43:36.074002+0000","last_became_peered":"2024-02-10T08:43:36.074002+0000","last_unstale":"2024-02-10T08:44:44.134532+0000","last_undegraded":"2024-02-10T08:44:44.134532+0000","last_fullsized":"2024-02-10T08:43:36.060407+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.341979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":509,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.404984+0000","last_change":"2024-02-10T08:42:43.111366+0000","last_active":"2024-02-10T08:44:39.404984+0000","last_peered":"2024-02-10T08:44:39.404984+0000","last_clean":"2024-02-10T08:44:39.404984+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:44:39.404984+0000","last_undegraded":"2024-02-10T08:44:39.404984+0000","last_fullsized":"2024-02-10T08:44:39.404984+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.111309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.111309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":430,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.182889+0000","last_change":"2024-02-10T08:44:02.384309+0000","last_active":"2024-02-10T08:44:44.182889+0000","last_peered":"2024-02-10T08:44:44.182889+0000","last_clean":"2024-02-10T08:44:44.182889+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:44:44.182889+0000","last_undegraded":"2024-02-10T08:44:44.182889+0000","last_fullsized":"2024-02-10T08:44:44.182889+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:02.384207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:02.384207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":116,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.136738+0000","last_change":"2024-02-10T08:44:25.556111+0000","last_active":"2024-02-10T08:44:44.136738+0000","last_peered":"2024-02-10T08:44:44.136738+0000","last_clean":"2024-02-10T08:44:44.136738+0000","last_became_active":"2024-02-10T08:43:15.139755+0000","last_became_peered":"2024-02-10T08:43:15.139755+0000","last_unstale":"2024-02-10T08:44:44.136738+0000","last_undegraded":"2024-02-10T08:44:44.136738+0000","last_fullsized":"2024-02-10T08:44:44.136738+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:44:25.556047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.556047+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":546,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.085974+0000","last_change":"2024-02-10T08:43:36.066663+0000","last_active":"2024-02-10T08:44:44.085974+0000","last_peered":"2024-02-10T08:44:44.085974+0000","last_clean":"2024-02-10T08:43:31.060592+0000","last_became_active":"2024-02-10T08:43:36.066663+0000","last_became_peered":"2024-02-10T08:43:36.066663+0000","last_unstale":"2024-02-10T08:44:44.085974+0000","last_undegraded":"2024-02-10T08:44:44.085974+0000","last_fullsized":"2024-02-10T08:43:36.056257+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.131457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.131457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":93,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.134984+0000","last_change":"2024-02-10T08:44:34.562436+0000","last_active":"2024-02-10T08:44:44.134984+0000","last_peered":"2024-02-10T08:44:44.134984+0000","last_clean":"2024-02-10T08:44:44.134984+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:44:44.134984+0000","last_undegraded":"2024-02-10T08:44:44.134984+0000","last_fullsized":"2024-02-10T08:44:44.134984+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:34.562370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:34.562370+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":474,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.315113+0000","last_change":"2024-02-10T08:44:29.460927+0000","last_active":"2024-02-10T08:44:40.315113+0000","last_peered":"2024-02-10T08:44:40.315113+0000","last_clean":"2024-02-10T08:44:40.315113+0000","last_became_active":"2024-02-10T08:43:15.119127+0000","last_became_peered":"2024-02-10T08:43:15.119127+0000","last_unstale":"2024-02-10T08:44:40.315113+0000","last_undegraded":"2024-02-10T08:44:40.315113+0000","last_fullsized":"2024-02-10T08:44:40.315113+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.460863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.460863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":551,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.184196+0000","last_change":"2024-02-10T08:44:41.392510+0000","last_active":"2024-02-10T08:44:44.184196+0000","last_peered":"2024-02-10T08:44:44.184196+0000","last_clean":"2024-02-10T08:44:44.184196+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:44:44.184196+0000","last_undegraded":"2024-02-10T08:44:44.184196+0000","last_fullsized":"2024-02-10T08:44:44.184196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:41.392413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:41.392413+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":515,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402319+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:44:39.402319+0000","last_peered":"2024-02-10T08:44:39.402319+0000","last_clean":"2024-02-10T08:44:39.402319+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:44:39.402319+0000","last_undegraded":"2024-02-10T08:44:39.402319+0000","last_fullsized":"2024-02-10T08:44:39.402319+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":508,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.335171+0000","last_change":"2024-02-10T08:43:36.073017+0000","last_active":"2024-02-10T08:44:40.335171+0000","last_peered":"2024-02-10T08:44:40.335171+0000","last_clean":"2024-02-10T08:43:31.058620+0000","last_became_active":"2024-02-10T08:43:36.073017+0000","last_became_peered":"2024-02-10T08:43:36.073017+0000","last_unstale":"2024-02-10T08:44:40.335171+0000","last_undegraded":"2024-02-10T08:44:40.335171+0000","last_fullsized":"2024-02-10T08:43:36.062639+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":457,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.327312+0000","last_change":"2024-02-10T08:43:36.085258+0000","last_active":"2024-02-10T08:44:40.327312+0000","last_peered":"2024-02-10T08:44:40.327312+0000","last_clean":"2024-02-10T08:43:31.059986+0000","last_became_active":"2024-02-10T08:43:36.085258+0000","last_became_peered":"2024-02-10T08:43:36.085258+0000","last_unstale":"2024-02-10T08:44:40.327312+0000","last_undegraded":"2024-02-10T08:44:40.327312+0000","last_fullsized":"2024-02-10T08:43:36.058326+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":585,"reported_epoch":534,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:40.331197+0000","last_change":"2024-02-10T08:43:36.088112+0000","last_active":"2024-02-10T08:44:40.331197+0000","last_peered":"2024-02-10T08:44:40.331197+0000","last_clean":"2024-02-10T08:43:31.060512+0000","last_became_active":"2024-02-10T08:43:36.088112+0000","last_became_peered":"2024-02-10T08:43:36.088112+0000","last_unstale":"2024-02-10T08:44:40.331197+0000","last_undegraded":"2024-02-10T08:43:36.063487+0000","last_fullsized":"2024-02-10T08:43:36.063380+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":536,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.183827+0000","last_change":"2024-02-10T08:43:36.078681+0000","last_active":"2024-02-10T08:44:44.183827+0000","last_peered":"2024-02-10T08:44:44.183827+0000","last_clean":"2024-02-10T08:43:31.155537+0000","last_became_active":"2024-02-10T08:43:36.078681+0000","last_became_peered":"2024-02-10T08:43:36.078681+0000","last_unstale":"2024-02-10T08:44:44.183827+0000","last_undegraded":"2024-02-10T08:44:44.183827+0000","last_fullsized":"2024-02-10T08:43:36.069413+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":65,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.182171+0000","last_change":"2024-02-10T08:43:36.071762+0000","last_active":"2024-02-10T08:44:44.182171+0000","last_peered":"2024-02-10T08:44:44.182171+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.071762+0000","last_became_peered":"2024-02-10T08:43:36.071762+0000","last_unstale":"2024-02-10T08:44:44.182171+0000","last_undegraded":"2024-02-10T08:44:44.182171+0000","last_fullsized":"2024-02-10T08:43:36.062825+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.031375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.031375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":430,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.180315+0000","last_change":"2024-02-10T08:43:51.341302+0000","last_active":"2024-02-10T08:44:44.180315+0000","last_peered":"2024-02-10T08:44:44.180315+0000","last_clean":"2024-02-10T08:44:44.180315+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:44:44.180315+0000","last_undegraded":"2024-02-10T08:44:44.180315+0000","last_fullsized":"2024-02-10T08:44:44.180315+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:51.341234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:51.341234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":563,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.405387+0000","last_change":"2024-02-10T08:43:36.081271+0000","last_active":"2024-02-10T08:44:39.405387+0000","last_peered":"2024-02-10T08:44:39.405387+0000","last_clean":"2024-02-10T08:43:31.058844+0000","last_became_active":"2024-02-10T08:43:36.081271+0000","last_became_peered":"2024-02-10T08:43:36.081271+0000","last_unstale":"2024-02-10T08:44:39.405387+0000","last_undegraded":"2024-02-10T08:43:36.068229+0000","last_fullsized":"2024-02-10T08:43:36.068095+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":524,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.404738+0000","last_change":"2024-02-10T08:43:15.126069+0000","last_active":"2024-02-10T08:44:39.404738+0000","last_peered":"2024-02-10T08:44:39.404738+0000","last_clean":"2024-02-10T08:44:39.404738+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:44:39.404738+0000","last_undegraded":"2024-02-10T08:44:39.404738+0000","last_fullsized":"2024-02-10T08:44:39.404738+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.957326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.957326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":500,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.406083+0000","last_change":"2024-02-10T08:42:37.092622+0000","last_active":"2024-02-10T08:44:39.406083+0000","last_peered":"2024-02-10T08:44:39.406083+0000","last_clean":"2024-02-10T08:44:39.406083+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:44:39.406083+0000","last_undegraded":"2024-02-10T08:44:39.406083+0000","last_fullsized":"2024-02-10T08:44:39.406083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.092522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.092522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":465,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313854+0000","last_change":"2024-02-10T08:44:00.751759+0000","last_active":"2024-02-10T08:44:40.313854+0000","last_peered":"2024-02-10T08:44:40.313854+0000","last_clean":"2024-02-10T08:44:40.313854+0000","last_became_active":"2024-02-10T08:43:28.080931+0000","last_became_peered":"2024-02-10T08:43:28.080931+0000","last_unstale":"2024-02-10T08:44:40.313854+0000","last_undegraded":"2024-02-10T08:44:40.313854+0000","last_fullsized":"2024-02-10T08:44:40.313854+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.751695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":547,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.402272+0000","last_change":"2024-02-10T08:43:36.080654+0000","last_active":"2024-02-10T08:44:39.402272+0000","last_peered":"2024-02-10T08:44:39.402272+0000","last_clean":"2024-02-10T08:43:25.677981+0000","last_became_active":"2024-02-10T08:43:36.080654+0000","last_became_peered":"2024-02-10T08:43:36.080654+0000","last_unstale":"2024-02-10T08:44:39.402272+0000","last_undegraded":"2024-02-10T08:43:36.066356+0000","last_fullsized":"2024-02-10T08:43:36.066242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":89,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135063+0000","last_change":"2024-02-10T08:43:59.379201+0000","last_active":"2024-02-10T08:44:44.135063+0000","last_peered":"2024-02-10T08:44:44.135063+0000","last_clean":"2024-02-10T08:44:44.135063+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:44:44.135063+0000","last_undegraded":"2024-02-10T08:44:44.135063+0000","last_fullsized":"2024-02-10T08:44:44.135063+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.379146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":444,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-02-10T08:44:40.313318+0000","last_change":"2024-02-10T08:43:25.651939+0000","last_active":"2024-02-10T08:44:40.313318+0000","last_peered":"2024-02-10T08:44:40.313318+0000","last_clean":"2024-02-10T08:44:40.313318+0000","last_became_active":"2024-02-10T08:43:15.131035+0000","last_became_peered":"2024-02-10T08:43:15.131035+0000","last_unstale":"2024-02-10T08:44:40.313318+0000","last_undegraded":"2024-02-10T08:44:40.313318+0000","last_fullsized":"2024-02-10T08:44:40.313318+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:25.651865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.651865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":466,"reported_epoch":534,"state":"active+undersized","last_fresh":"2024-02-10T08:44:40.316093+0000","last_change":"2024-02-10T08:43:36.078496+0000","last_active":"2024-02-10T08:44:40.316093+0000","last_peered":"2024-02-10T08:44:40.316093+0000","last_clean":"2024-02-10T08:43:31.066991+0000","last_became_active":"2024-02-10T08:43:36.078496+0000","last_became_peered":"2024-02-10T08:43:36.078496+0000","last_unstale":"2024-02-10T08:44:40.316093+0000","last_undegraded":"2024-02-10T08:44:40.316093+0000","last_fullsized":"2024-02-10T08:43:36.061038+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.399709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.399709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":518,"reported_epoch":533,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:39.404600+0000","last_change":"2024-02-10T08:43:36.081968+0000","last_active":"2024-02-10T08:44:39.404600+0000","last_peered":"2024-02-10T08:44:39.404600+0000","last_clean":"2024-02-10T08:43:25.647108+0000","last_became_active":"2024-02-10T08:43:36.081968+0000","last_became_peered":"2024-02-10T08:43:36.081968+0000","last_unstale":"2024-02-10T08:44:39.404600+0000","last_undegraded":"2024-02-10T08:43:36.066991+0000","last_fullsized":"2024-02-10T08:43:36.066884+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":84,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.135659+0000","last_change":"2024-02-10T08:43:15.134662+0000","last_active":"2024-02-10T08:44:44.135659+0000","last_peered":"2024-02-10T08:44:44.135659+0000","last_clean":"2024-02-10T08:44:44.135659+0000","last_became_active":"2024-02-10T08:43:15.134426+0000","last_became_peered":"2024-02-10T08:43:15.134426+0000","last_unstale":"2024-02-10T08:44:44.135659+0000","last_undegraded":"2024-02-10T08:44:44.135659+0000","last_fullsized":"2024-02-10T08:44:44.135659+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.378214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.378214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":502,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.086557+0000","last_change":"2024-02-10T08:42:35.160383+0000","last_active":"2024-02-10T08:44:44.086557+0000","last_peered":"2024-02-10T08:44:44.086557+0000","last_clean":"2024-02-10T08:44:44.086557+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:44:44.086557+0000","last_undegraded":"2024-02-10T08:44:44.086557+0000","last_fullsized":"2024-02-10T08:44:44.086557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":435,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-02-10T08:44:44.087393+0000","last_change":"2024-02-10T08:44:25.269848+0000","last_active":"2024-02-10T08:44:44.087393+0000","last_peered":"2024-02-10T08:44:44.087393+0000","last_clean":"2024-02-10T08:44:44.087393+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:44:44.087393+0000","last_undegraded":"2024-02-10T08:44:44.087393+0000","last_fullsized":"2024-02-10T08:44:44.087393+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.269759+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":550,"reported_epoch":538,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:44:44.180674+0000","last_change":"2024-02-10T08:43:36.080498+0000","last_active":"2024-02-10T08:44:44.180674+0000","last_peered":"2024-02-10T08:44:44.180674+0000","last_clean":"2024-02-10T08:43:25.563172+0000","last_became_active":"2024-02-10T08:43:36.080498+0000","last_became_peered":"2024-02-10T08:43:36.080498+0000","last_unstale":"2024-02-10T08:44:44.180674+0000","last_undegraded":"2024-02-10T08:43:36.063844+0000","last_fullsized":"2024-02-10T08:43:36.063724+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:43:22.036213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:22.036213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":515,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.405864+0000","last_change":"2024-02-10T08:43:14.144599+0000","last_active":"2024-02-10T08:44:39.405864+0000","last_peered":"2024-02-10T08:44:39.405864+0000","last_clean":"2024-02-10T08:44:39.405864+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:44:39.405864+0000","last_undegraded":"2024-02-10T08:44:39.405864+0000","last_fullsized":"2024-02-10T08:44:39.405864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.144537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.144537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":67,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.086640+0000","last_change":"2024-02-10T08:43:36.070306+0000","last_active":"2024-02-10T08:44:44.086640+0000","last_peered":"2024-02-10T08:44:44.086640+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.070306+0000","last_became_peered":"2024-02-10T08:43:36.070306+0000","last_unstale":"2024-02-10T08:44:44.086640+0000","last_undegraded":"2024-02-10T08:44:44.086640+0000","last_fullsized":"2024-02-10T08:43:36.056977+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":466,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.086754+0000","last_change":"2024-02-10T08:43:36.069163+0000","last_active":"2024-02-10T08:44:44.086754+0000","last_peered":"2024-02-10T08:44:44.086754+0000","last_clean":"2024-02-10T08:43:31.063295+0000","last_became_active":"2024-02-10T08:43:36.069163+0000","last_became_peered":"2024-02-10T08:43:36.069163+0000","last_unstale":"2024-02-10T08:44:44.086754+0000","last_undegraded":"2024-02-10T08:44:44.086754+0000","last_fullsized":"2024-02-10T08:43:36.056950+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":572,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.402785+0000","last_change":"2024-02-10T08:44:31.267329+0000","last_active":"2024-02-10T08:44:39.402785+0000","last_peered":"2024-02-10T08:44:39.402785+0000","last_clean":"2024-02-10T08:44:39.402785+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:44:39.402785+0000","last_undegraded":"2024-02-10T08:44:39.402785+0000","last_fullsized":"2024-02-10T08:44:39.402785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:44:31.267261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.267261+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":90,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.137306+0000","last_change":"2024-02-10T08:43:36.074560+0000","last_active":"2024-02-10T08:44:44.137306+0000","last_peered":"2024-02-10T08:44:44.137306+0000","last_clean":"2024-02-10T08:43:31.060871+0000","last_became_active":"2024-02-10T08:43:36.074560+0000","last_became_peered":"2024-02-10T08:43:36.074560+0000","last_unstale":"2024-02-10T08:44:44.137306+0000","last_undegraded":"2024-02-10T08:44:44.137306+0000","last_fullsized":"2024-02-10T08:43:36.062458+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:48.325433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:48.325433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":518,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-10T08:44:39.404448+0000","last_change":"2024-02-10T08:43:55.185139+0000","last_active":"2024-02-10T08:44:39.404448+0000","last_peered":"2024-02-10T08:44:39.404448+0000","last_clean":"2024-02-10T08:44:39.404448+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:44:39.404448+0000","last_undegraded":"2024-02-10T08:44:39.404448+0000","last_fullsized":"2024-02-10T08:44:39.404448+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:55.185039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:55.185039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":470,"reported_epoch":538,"state":"active+undersized","last_fresh":"2024-02-10T08:44:44.182481+0000","last_change":"2024-02-10T08:43:36.085682+0000","last_active":"2024-02-10T08:44:44.182481+0000","last_peered":"2024-02-10T08:44:44.182481+0000","last_clean":"2024-02-10T08:43:31.156976+0000","last_became_active":"2024-02-10T08:43:36.085682+0000","last_became_peered":"2024-02-10T08:43:36.085682+0000","last_unstale":"2024-02-10T08:44:44.182481+0000","last_undegraded":"2024-02-10T08:44:44.182481+0000","last_fullsized":"2024-02-10T08:43:36.064612+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":60,"num_pg":32,"stat_sum":{"num_bytes":4020435,"num_objects":192,"num_object_clones":11,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":208,"num_read_kb":56,"num_write":777,"num_write_kb":4619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":1217,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14249984,"data_stored":12046408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2059,"ondisk_log_size":2059,"up":79,"acting":79,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":55,"acting":55,"num_store_stats":8},{"poolid":226,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":5506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":720,"ondisk_log_size":720,"up":81,"acting":81,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":675,"num_read_kb":670,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":85,"acting":86,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299532,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300028,"kb_used_data":2804,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445124,"statfs":{"total":95995035648,"available":95687806976,"internally_reserved":0,"allocated":2871296,"data_stored":1151433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.59299999999999997},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.623,"5min":0.99299999999999999,"15min":22.143000000000001},"last":0.499},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.56100000000000005},"min":{"1min":0.35999999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.623,"5min":1.623,"15min":8.7850000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":1.359,"5min":3.2389999999999999,"15min":7.593},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51300000000000001,"15min":0.57299999999999995},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.19900000000000001},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":7.7759999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49299999999999999,"15min":0.54400000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":1.504,"5min":2.5009999999999999,"15min":6.0300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.64500000000000002},"min":{"1min":0.34100000000000003,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":19.369},"last":0.629}]},{"osd":3,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.52200000000000002,"15min":0.56999999999999995},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.6419999999999999,"5min":2.7029999999999998,"15min":6.9829999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52900000000000003,"15min":0.63700000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":1.79,"5min":1.79,"15min":22.102},"last":0.59799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":8.827},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":1.903,"15min":13.646000000000001},"last":0.35199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.45500000000000002,"15min":0.52200000000000002},"min":{"1min":0.26000000000000001,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.66000000000000003,"5min":0.81200000000000006,"15min":19.326000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66600000000000004,"5min":0.99199999999999999,"15min":1.6220000000000001},"last":0.36499999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48899999999999999,"15min":0.56200000000000006},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26600000000000001},"max":{"1min":0.68600000000000005,"5min":0.86899999999999999,"15min":22.687000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":0.77700000000000002,"15min":1.7450000000000001},"last":0.41199999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561170,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312000,"kb_used_data":6584,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433152,"statfs":{"total":95995035648,"available":95675547648,"internally_reserved":0,"allocated":6742016,"data_stored":5137815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.51100000000000001,"15min":0.53400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.19800000000000001},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.54000000000000004},"min":{"1min":0.28100000000000003,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":0.86399999999999999,"5min":1.694,"15min":8.4629999999999992},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":0.63,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":18.358000000000001,"5min":18.358000000000001,"15min":18.358000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.63300000000000001,"15min":0.55400000000000005},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":18.402999999999999,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.625}]},{"osd":2,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":4.798},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":4.8070000000000004},"last":0.61499999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.59899999999999998},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.878,"5min":0.878,"15min":16.823},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":3.7389999999999999},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45300000000000001,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":3.657,"15min":3.657},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.46500000000000002,"15min":0.40500000000000003},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.24399999999999999},"max":{"1min":0.61799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.44,"15min":0.42299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":3.7429999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46100000000000002,"15min":0.44400000000000001},"min":{"1min":0.26900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61099999999999999,"5min":0.68500000000000005,"15min":2.2029999999999998},"last":0.41799999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.54600000000000004,"15min":0.503},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.5650000000000004,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.52800000000000002,"15min":0.50600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308520,"kb_used_data":2732,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436632,"statfs":{"total":95995035648,"available":95679111168,"internally_reserved":0,"allocated":2797568,"data_stored":1107657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.40799999999999997},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.58199999999999996,"5min":0.66200000000000003,"15min":7.8780000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.45700000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":16.166},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.378,"15min":0.61799999999999999},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":34.985999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.36199999999999999,"15min":0.53200000000000003},"min":{"1min":0.224,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":21.614999999999998},"last":0.23400000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:43:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.502},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55200000000000005,"5min":0.627,"15min":24.773},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.371,"15min":0.51100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":16.120999999999999},"last":0.254}]},{"osd":4,"last update":"Sat Feb 10 08:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.60999999999999999},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.89000000000000001,"15min":16.074000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.60399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":9.5090000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55300000000000005,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":2.0270000000000001,"15min":9.5600000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.65300000000000002},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.31900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":7.8479999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.312},"max":{"1min":0.92700000000000005,"5min":1.996,"15min":9.532},"last":0.438},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.64900000000000002},"min":{"1min":0.42199999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":7.9000000000000004},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.60499999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.95099999999999996,"5min":2.0939999999999999,"15min":9.4640000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57499999999999996,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":7.9199999999999999},"last":0.5}]}]},{"osd":0,"up_from":0,"seq":34359738543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308496,"kb_used_data":2708,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436656,"statfs":{"total":95995035648,"available":95679135744,"internally_reserved":0,"allocated":2772992,"data_stored":1106883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.45900000000000002,"15min":0.438},"min":{"1min":0.192,"5min":0.192,"15min":0.187},"max":{"1min":10.342000000000001,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.204,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.20499999999999999,"5min":0.187,"15min":0.187},"max":{"1min":14.858000000000001,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.45800000000000002,"15min":0.50800000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":12.512},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.41199999999999998,"15min":0.47099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.0760000000000001,"5min":3.0960000000000001,"15min":9.6769999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.497,"15min":0.47199999999999998},"min":{"1min":0.245,"5min":0.217,"15min":0.217},"max":{"1min":11.612,"5min":11.612,"15min":11.612},"last":0.249},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.22600000000000001},"max":{"1min":11.710000000000001,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.37}]},{"osd":4,"last update":"Sat Feb 10 08:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.67200000000000004},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76200000000000001,"5min":1.7370000000000001,"15min":10.920999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":1.014,"15min":12.292999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.65600000000000003},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":1.764,"15min":9.2029999999999994},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.33100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78900000000000003,"15min":7.6799999999999997},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.66000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75700000000000001,"5min":1.6759999999999999,"15min":9.266},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.64600000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":5.4630000000000001},"last":0.45100000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.708,"15min":9.2940000000000005},"last":0.497},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.65500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":7.4219999999999997},"last":0.40600000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182560,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312380,"kb_used_data":6592,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432772,"statfs":{"total":95995035648,"available":95675158528,"internally_reserved":0,"allocated":6750208,"data_stored":5101983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.313,"15min":0.501},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.56100000000000005,"5min":0.878,"15min":14.744},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.36499999999999999,"15min":0.54100000000000004},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.188},"max":{"1min":0.39600000000000002,"5min":3.5579999999999998,"15min":11.808},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.42799999999999999,"15min":0.55400000000000005},"min":{"1min":0.24299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46700000000000003,"5min":9.3100000000000005,"15min":16.634},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.55800000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":2.472,"5min":5.4370000000000003,"15min":12.438000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41899999999999998,"15min":0.58199999999999996},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":2.4340000000000002,"5min":3.1549999999999998,"15min":16.536000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.625},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":2.5230000000000001,"5min":2.5230000000000001,"15min":23.294},"last":0.52300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54500000000000004,"15min":0.66900000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":2.589,"15min":12.371},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68700000000000006,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:44:36 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.57299999999999995,"15min":0.70099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.4019999999999999,"5min":3.6040000000000001,"15min":12.579000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.65000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.4129999999999998,"5min":4.9909999999999997,"15min":11.448},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Feb 10 08:44:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57499999999999996,"15min":0.69799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":1.0269999999999999,"5min":3.48,"15min":16.091000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59699999999999998,"15min":0.68999999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.33800000000000002},"max":{"1min":2.383,"5min":3.2469999999999999,"15min":16.475000000000001},"last":0.66600000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:44:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.63500000000000001,"15min":0.68400000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.6140000000000008,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.621},{"interface":"front","average":{"1min":0.503,"5min":0.61699999999999999,"15min":0.73599999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85599999999999998,"5min":3.52,"15min":12.606999999999999},"last":0.58999999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986316,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312804,"kb_used_data":3292,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432348,"statfs":{"total":95995035648,"available":95674724352,"internally_reserved":0,"allocated":3371008,"data_stored":1427290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.627},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":19.940999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.55300000000000005},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.91800000000000004,"15min":9.1760000000000002},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46899999999999997,"15min":0.57899999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.59699999999999998,"5min":0.93799999999999994,"15min":5.774},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.59799999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":8.6989999999999998},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.59799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.83999999999999997,"15min":6.2329999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.59799999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.64600000000000002,"5min":0.77100000000000002,"15min":10.01},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48399999999999999,"15min":0.65000000000000002},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.27700000000000002},"max":{"1min":0.58299999999999996,"5min":0.92700000000000005,"15min":9.5909999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51900000000000002,"15min":0.71799999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":25.27},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.621,"5min":0.876,"15min":6.8730000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.223},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":3.8290000000000002},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.247},"max":{"1min":0.626,"5min":0.78000000000000003,"15min":13.342000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":1.085,"15min":13.753},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.52000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.90400000000000003,"15min":12.835000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.72599999999999998,"5min":1.0509999999999999,"15min":12.875999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790103,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316144,"kb_used_data":6632,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429008,"statfs":{"total":95995035648,"available":95671304192,"internally_reserved":0,"allocated":6791168,"data_stored":5154188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":0.59799999999999998,"15min":1.028},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":107.551},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.55000000000000004,"15min":0.97199999999999998},"min":{"1min":0.29299999999999998,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":5.375,"5min":5.375,"15min":88.792000000000002},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.96499999999999997},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":8.7200000000000006,"5min":8.7200000000000006,"15min":107.205},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.60799999999999998,"15min":0.84399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":4.7629999999999999,"5min":4.7629999999999999,"15min":72.272000000000006},"last":0.59599999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.58999999999999997,"15min":0.98199999999999998},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":10.794,"5min":10.794,"15min":54.174999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.486,"5min":0.70899999999999996,"15min":1.0549999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":8.6890000000000001,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":0.77700000000000002,"15min":0.81000000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":13.691000000000001,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.4359999999999999,"5min":0.71599999999999997,"15min":1.2629999999999999},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":8.7690000000000001,"5min":8.7690000000000001,"15min":114.526},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Feb 10 08:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.69199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.492,"5min":1.492,"15min":72.241},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.47099999999999997,"15min":0.54600000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54200000000000004,"5min":0.68500000000000005,"15min":22.469999999999999},"last":0.34599999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.495,"15min":0.71699999999999997},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":66.844999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.50900000000000001,"15min":0.746},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56699999999999995,"5min":0.90700000000000003,"15min":72.257000000000005},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":6619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":7242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4304896,"data_stored":4011151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4276224,"data_stored":4010317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4002924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":60,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":7311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":226,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:44:50.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 1 2024-02-10T08:44:50.053 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.072 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:50.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:50.202 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.221 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:50.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:50.358 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:44:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:49.100082+0000 mon.a (mon.0) 2833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:49.100161+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-10T08:44:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:49.144505+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.359 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:49.152116+0000 mon.a (mon.0) 2835 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.359 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:49.491640+0000 mgr.y (mgr.14553) 481 : cluster [DBG] pgmap v817: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.383 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:49.100082+0000 mon.a (mon.0) 2833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:49.100161+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-10T08:44:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:49.144505+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:49.152116+0000 mon.a (mon.0) 2835 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:49.491640+0000 mgr.y (mgr.14553) 481 : cluster [DBG] pgmap v817: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:49.100082+0000 mon.a (mon.0) 2833 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:44:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:49.100161+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e543: 8 total, 7 up, 7 in 2024-02-10T08:44:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:49.144505+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:49.152116+0000 mon.a (mon.0) 2835 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:44:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:49.491640+0000 mgr.y (mgr.14553) 481 : cluster [DBG] pgmap v817: 218 pgs: 32 unknown, 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:50.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:50.512 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.530 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:50.542 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:50.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:50.659 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.677 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:50.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:44:50.807 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.825 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:44:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:44:50.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:50.973 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:50.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:51.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:44:51.107 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:51.131 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:44:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:51.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:44:51.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:50.103760+0000 mon.a (mon.0) 2836 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:50.103819+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:50.110305+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:50.124971+0000 mon.a (mon.0) 2838 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:51.088575+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:51.088626+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-10T08:44:51.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:51.091328+0000 mon.a (mon.0) 2841 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:50.103760+0000 mon.a (mon.0) 2836 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:50.103819+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:50.110305+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:50.124971+0000 mon.a (mon.0) 2838 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:51.088575+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:51.088626+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-10T08:44:51.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:51.091328+0000 mon.a (mon.0) 2841 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:50.103760+0000 mon.a (mon.0) 2836 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:50.103819+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e544: 8 total, 7 up, 7 in 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:50.110305+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:50.124971+0000 mon.a (mon.0) 2838 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:51.088575+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-6", "overlaypool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:51.088626+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e545: 8 total, 7 up, 7 in 2024-02-10T08:44:51.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:51.091328+0000 mon.a (mon.0) 2841 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:51.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.295+0000 7f9d94753700 1 -- 172.21.15.107:0/952012327 >> v1:172.21.15.195:6789/0 conn(0x7f9d8c1042d0 legacy=0x7f9d8c106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.295+0000 7f9d94753700 1 -- 172.21.15.107:0/952012327 shutdown_connections 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.295+0000 7f9d94753700 1 -- 172.21.15.107:0/952012327 >> 172.21.15.107:0/952012327 conn(0x7f9d8c06fdf0 msgr2=0x7f9d8c0f40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.295+0000 7f9d94753700 1 -- 172.21.15.107:0/952012327 shutdown_connections 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.295+0000 7f9d94753700 1 -- 172.21.15.107:0/952012327 wait complete. 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 librados: starting msgr at 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 librados: starting objecter 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 Processor -- start 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 -- start start 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 librados: setting wanted keys 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 librados: calling monclient init 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d8c0fd2a0 con 0x7f9d8c1007c0 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d8c0fd470 con 0x7f9d8c1042d0 2024-02-10T08:44:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d94753700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d8c0f8b70 con 0x7f9d8c0fd970 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d91cee700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9d8c1007c0 0x7f9d8c119520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60194/0 (socket says 172.21.15.107:60194) 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d91cee700 1 -- 172.21.15.107:0/1213903127 learned_addr learned my addr 172.21.15.107:0/1213903127 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter ms_handle_connect 0x7f9d8c1007c0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter ms_handle_connect 0x7f9d8c0fd970 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter ms_handle_connect 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.296+0000 7f9d837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394875621 0 0) 0x7f9d8c0fd2a0 con 0x7f9d8c1007c0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d78003110 con 0x7f9d8c1007c0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271315370 0 0) 0x7f9d8c0fd470 con 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0fd2a0 con 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4163250647 0 0) 0x7f9d8c0fd2a0 con 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c0fd470 con 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9d840030c0 con 0x7f9d8c1042d0 2024-02-10T08:44:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587900981 0 0) 0x7f9d78003110 con 0x7f9d8c1007c0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c0fd2a0 con 0x7f9d8c1007c0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9d7c001800 con 0x7f9d8c1007c0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3608047114 0 0) 0x7f9d8c0fd470 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 >> v1:172.21.15.107:6790/0 conn(0x7f9d8c0fd970 legacy=0x7f9d8c0fca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 >> v1:172.21.15.107:6789/0 conn(0x7f9d8c1007c0 legacy=0x7f9d8c119520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0f8d00 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c0f8fb0 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d94753700 10 client.48616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c11e650 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d94753700 1 librados: init done 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9d840014d0 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.297+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9d84003990 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.299+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9d70004f00 con 0x7f9d8c1042d0 2024-02-10T08:44:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.299+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9d84002310 con 0x7f9d8c1042d0 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(545..545 src has 420..545) v4 ==== 8181+0+0 (unknown 659656176 0 0) 0x7f9d84076b40 con 0x7f9d8c1042d0 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 osd_map(545..545 src has 420..545) v4 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 3 client.48616.objecter handle_osd_map got epochs [545,545] > 0 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 3 client.48616.objecter handle_osd_map decoding full epoch 545 2024-02-10T08:44:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 20 client.48616.objecter dump_active .. 0 homeless 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.300+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v545) v1 ==== 54+0+16 (unknown 3166559619 0 2420000192) 0x7f9d84049f20 con 0x7f9d8c1042d0 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 10 client.48616.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 1 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 1 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 0 1 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 4 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 10 client.48616.objecter _send_command 1 2024-02-10T08:44:51.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70004950 con 0x7f9d70004530 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.301+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d70004530 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.302+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d70004950 con 0x7f9d70004530 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.302+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.302+0000 7f9d837fe700 10 client.48616.objecter _finish_command 1 = system:0 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.302+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 4 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.302+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 0 1 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.307+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d78063d20 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 10 client.48616.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 2 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 2 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d700051a0 osd=0 3 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 0 2 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 4 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 10 client.48616.objecter _send_command 2 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.327+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d70004d30 con 0x7f9d70004530 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d70004d30 con 0x7f9d70004530 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d837fe700 10 client.48616.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d700051a0 osd=0 4 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 0 2 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 10 client.48616.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 3 2024-02-10T08:44:51.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 1 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 4 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 10 client.48616.objecter _send_command 3 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.328+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7000c000 con 0x7f9d70004150 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.333+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d70004150 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.334+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d7000c000 con 0x7f9d70004150 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.334+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.334+0000 7f9d837fe700 10 client.48616.objecter _finish_command 3 = system:0 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.334+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 4 2024-02-10T08:44:51.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.334+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 1 3 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 10 client.48616.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 4 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 4 2024-02-10T08:44:51.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70003fe0 osd=1 3 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 1 4 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 4 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 10 client.48616.objecter _send_command 4 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.349+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d700018b0 con 0x7f9d70004150 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d700018b0 con 0x7f9d70004150 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:44:51.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d837fe700 10 client.48616.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70003fe0 osd=1 4 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 1 4 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 10 client.48616.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 5 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 5 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 2 5 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 4 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 10 client.48616.objecter _send_command 5 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.350+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7000de90 con 0x7f9d7000d2f0 2024-02-10T08:44:51.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d7000d2f0 2024-02-10T08:44:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d7000de90 con 0x7f9d7000d2f0 2024-02-10T08:44:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:44:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 10 client.48616.objecter _finish_command 5 = system:0 2024-02-10T08:44:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 4 2024-02-10T08:44:51.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.351+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 2 5 2024-02-10T08:44:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 10 client.48616.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 6 2024-02-10T08:44:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 6 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70003630 osd=2 3 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 2 6 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 4 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 10 client.48616.objecter _send_command 6 2024-02-10T08:44:51.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.366+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d70003160 con 0x7f9d7000d2f0 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.367+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d70003160 con 0x7f9d7000d2f0 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.367+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.367+0000 7f9d837fe700 10 client.48616.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.367+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70003630 osd=2 4 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.367+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 2 6 2024-02-10T08:44:51.368 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 10 client.48616.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 7 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 7 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 3 7 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 4 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 10 client.48616.objecter _send_command 7 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70015bb0 con 0x7f9d70011ff0 2024-02-10T08:44:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.368+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d70011ff0 2024-02-10T08:44:51.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.369+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d70015bb0 con 0x7f9d70011ff0 2024-02-10T08:44:51.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.369+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:44:51.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.369+0000 7f9d837fe700 10 client.48616.objecter _finish_command 7 = system:0 2024-02-10T08:44:51.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.369+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 4 2024-02-10T08:44:51.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.369+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 3 7 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 10 client.48616.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 8 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 8 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70011970 osd=3 3 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 3 8 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 4 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 10 client.48616.objecter _send_command 8 2024-02-10T08:44:51.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.383+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7000e270 con 0x7f9d70011ff0 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d7000e270 con 0x7f9d70011ff0 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 10 client.48616.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70011970 osd=3 4 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 3 8 2024-02-10T08:44:51.385 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:44:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 10 client.48616.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 9 2024-02-10T08:44:51.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter _get_session osd=-1 returning homeless 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 10 client.48616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=546}) v3 -- 0x7f9d7000e030 con 0x7f9d8c1042d0 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d7000c940 con 0x7f9d8c1042d0 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_get_version_reply(handle=1 version=545) v2 ==== 24+0+0 (unknown 3675076948 0 0) 0x7f9d84001e00 con 0x7f9d8c1042d0 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d90cec700 10 client.48616.objecter _check_command_map_dne tid 9 current 545 map_dne_bound 545 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d90cec700 10 client.48616.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d90cec700 15 client.48616.objecter _session_command_op_remove -1 9 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 10 client.48616.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 10 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 10 2024-02-10T08:44:51.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 5 10 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 4 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 10 client.48616.objecter _send_command 10 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.385+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70017040 con 0x7f9d7000ce20 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.386+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d7000ce20 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.387+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d70017040 con 0x7f9d7000ce20 2024-02-10T08:44:51.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.387+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:44:51.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.387+0000 7f9d837fe700 10 client.48616.objecter _finish_command 10 = system:0 2024-02-10T08:44:51.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.387+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 4 2024-02-10T08:44:51.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.387+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 5 10 2024-02-10T08:44:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 10 client.48616.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 11 2024-02-10T08:44:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 11 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d70016ed0 osd=5 3 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 5 11 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 4 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 10 client.48616.objecter _send_command 11 2024-02-10T08:44:51.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.401+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7000cc40 con 0x7f9d7000ce20 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d7000cc40 con 0x7f9d7000ce20 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d837fe700 10 client.48616.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d70016ed0 osd=5 4 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 5 11 2024-02-10T08:44:51.404 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 10 client.48616.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 12 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 12 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 6 12 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 4 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 10 client.48616.objecter _send_command 12 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.403+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7001f0d0 con 0x7f9d7001b4b0 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.404+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d7001b4b0 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.405+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d7001f0d0 con 0x7f9d7001b4b0 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.405+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.405+0000 7f9d837fe700 10 client.48616.objecter _finish_command 12 = system:0 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.405+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 4 2024-02-10T08:44:51.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.405+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 6 12 2024-02-10T08:44:51.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 10 client.48616.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 13 2024-02-10T08:44:51.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 13 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d7000cc40 osd=6 3 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 6 13 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 4 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 10 client.48616.objecter _send_command 13 2024-02-10T08:44:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.419+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7001f0d0 con 0x7f9d7001b4b0 2024-02-10T08:44:51.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d7001f0d0 con 0x7f9d7001b4b0 2024-02-10T08:44:51.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d837fe700 10 client.48616.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d7000cc40 osd=6 4 2024-02-10T08:44:51.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 6 13 2024-02-10T08:44:51.422 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 10 client.48616.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 14 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 14 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 7 14 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 4 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 10 client.48616.objecter _send_command 14 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.421+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d700203b0 con 0x7f9d70020e00 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.422+0000 7f9d837fe700 10 client.48616.objecter ms_handle_connect 0x7f9d70020e00 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.423+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9d700203b0 con 0x7f9d70020e00 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.423+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.423+0000 7f9d837fe700 10 client.48616.objecter _finish_command 14 = system:0 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.423+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 4 2024-02-10T08:44:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.423+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 7 14 2024-02-10T08:44:51.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 10 client.48616.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign -1 15 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 20 client.48616.objecter _get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 15 client.48616.objecter _session_command_op_remove -1 15 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 20 client.48616.objecter get_session s=0x7f9d7001f0d0 osd=7 3 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 15 client.48616.objecter _session_command_op_assign 7 15 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 4 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 10 client.48616.objecter _send_command 15 2024-02-10T08:44:51.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.437+0000 7f9d94753700 1 -- 172.21.15.107:0/1213903127 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d70020370 con 0x7f9d70020e00 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.439+0000 7f9d837fe700 1 -- 172.21.15.107:0/1213903127 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f9d70020370 con 0x7f9d70020e00 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.439+0000 7f9d837fe700 10 client.48616.objecter ms_dispatch 0x7f9d8c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.439+0000 7f9d837fe700 10 client.48616.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.439+0000 7f9d837fe700 20 client.48616.objecter put_session s=0x7f9d7001f0d0 osd=7 4 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:51.439+0000 7f9d837fe700 15 client.48616.objecter _session_command_op_remove 7 15 2024-02-10T08:44:51.440 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:44:51.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:44:52.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:44:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:51.089967+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:51.300803+0000 mon.b (mon.1) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1213903127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:51.492538+0000 mgr.y (mgr.14553) 482 : cluster [DBG] pgmap v820: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:51.605475+0000 mgr.y (mgr.14553) 483 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:52.084270+0000 mon.a (mon.0) 2842 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:52.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:52.091601+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]': finished 2024-02-10T08:44:52.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:52.091673+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-10T08:44:52.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:52.092651+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:52.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:52.093092+0000 mon.a (mon.0) 2845 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:51.089967+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:51.300803+0000 mon.b (mon.1) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1213903127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:51.492538+0000 mgr.y (mgr.14553) 482 : cluster [DBG] pgmap v820: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:51.605475+0000 mgr.y (mgr.14553) 483 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:52.084270+0000 mon.a (mon.0) 2842 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:52.091601+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]': finished 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:52.091673+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:52.092651+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:52.093092+0000 mon.a (mon.0) 2845 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:51.089967+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:51.300803+0000 mon.b (mon.1) 678 : audit [DBG] from='client.? v1:172.21.15.107:0/1213903127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:44:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:51.492538+0000 mgr.y (mgr.14553) 482 : cluster [DBG] pgmap v820: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:51.605475+0000 mgr.y (mgr.14553) 483 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:52.084270+0000 mon.a (mon.0) 2842 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:52.091601+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-90", "mode": "writeback"}]': finished 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:52.091673+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e546: 8 total, 7 up, 7 in 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:52.092651+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:52.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:52.093092+0000 mon.a (mon.0) 2845 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:44:53.708 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:44:53.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:44:53.746 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:53.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:53.751 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:53.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:53.879 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:53.978 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:53.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:54.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:54.119 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:53.096015+0000 mon.a (mon.0) 2846 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:53.096094+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:53.097320+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:53.097689+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:53.493443+0000 mgr.y (mgr.14553) 484 : cluster [DBG] pgmap v823: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:54.270 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:53.096015+0000 mon.a (mon.0) 2846 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:44:54.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:53.096094+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-10T08:44:54.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:53.097320+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:53.097689+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:53.493443+0000 mgr.y (mgr.14553) 484 : cluster [DBG] pgmap v823: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:54.279 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:54.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:54.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:54.411 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:53.096015+0000 mon.a (mon.0) 2846 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:44:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:53.096094+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e547: 8 total, 7 up, 7 in 2024-02-10T08:44:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:53.097320+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:53.097689+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:44:54.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:53.493443+0000 mgr.y (mgr.14553) 484 : cluster [DBG] pgmap v823: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:54.859 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:54.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:54.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:44:54.990 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:44:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:44:55.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:55.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:54.102831+0000 mon.a (mon.0) 2849 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:55.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:54.102896+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-10T08:44:55.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:54.111137+0000 mon.b (mon.1) 681 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:54.111485+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:54.102831+0000 mon.a (mon.0) 2849 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:55.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:54.102896+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-10T08:44:55.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:54.111137+0000 mon.b (mon.1) 681 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:54.111485+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.230 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:54.102831+0000 mon.a (mon.0) 2849 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:44:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:54.102896+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e548: 8 total, 7 up, 7 in 2024-02-10T08:44:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:54.111137+0000 mon.b (mon.1) 681 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:54.111485+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:44:55.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:44:55.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:44:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:44:55] "GET /metrics HTTP/1.1" 200 189137 "" "Prometheus/2.33.4" 2024-02-10T08:44:56.380 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:55.099780+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:56.380 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:55.103603+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:55.103677+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:55.105620+0000 mon.b (mon.1) 682 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:55.106021+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:55.201575+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:55.494161+0000 mgr.y (mgr.14553) 485 : cluster [DBG] pgmap v826: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:56.381 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:55.099780+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:55.103603+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:55.103677+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:55.105620+0000 mon.b (mon.1) 682 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:55.106021+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:55.201575+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:56.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:55.494161+0000 mgr.y (mgr.14553) 485 : cluster [DBG] pgmap v826: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:56.390 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:56.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:56.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:44:56.523 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:55.099780+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:55.103603+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:55.103677+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e549: 8 total, 7 up, 7 in 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:55.105620+0000 mon.b (mon.1) 682 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:55.106021+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:55.201575+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:44:56.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:55.494161+0000 mgr.y (mgr.14553) 485 : cluster [DBG] pgmap v826: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 189/1155 objects degraded (16.364%) 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 -- 172.21.15.107:0/3775438356 >> v1:172.21.15.107:6790/0 conn(0x7f22d0104330 legacy=0x7f22d01067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 -- 172.21.15.107:0/3775438356 shutdown_connections 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 -- 172.21.15.107:0/3775438356 >> 172.21.15.107:0/3775438356 conn(0x7f22d00f3640 msgr2=0x7f22d00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 -- 172.21.15.107:0/3775438356 shutdown_connections 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 -- 172.21.15.107:0/3775438356 wait complete. 2024-02-10T08:44:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 librados: starting msgr at 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 librados: starting objecter 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.604+0000 7f22d5978700 1 Processor -- start 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 -- start start 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 librados: setting wanted keys 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 librados: calling monclient init 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22d01173d0 con 0x7f22d0104330 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22d01185b0 con 0x7f22d00fda50 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22d5978700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f22d01197b0 con 0x7f22d01007f0 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22ce7fc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f22d01007f0 0x7f22d00fa290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57890/0 (socket says 172.21.15.107:57890) 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22ce7fc700 1 -- 172.21.15.107:0/2889659784 learned_addr learned my addr 172.21.15.107:0/2889659784 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter ms_handle_connect 0x7f22d01007f0 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter ms_handle_connect 0x7f22d0104330 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter ms_handle_connect 0x7f22d00fda50 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001765624 0 0) 0x7f22d01197b0 con 0x7f22d01007f0 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b80033f0 con 0x7f22d01007f0 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181637231 0 0) 0x7f22d01173d0 con 0x7f22d0104330 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d01197b0 con 0x7f22d0104330 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267804662 0 0) 0x7f22d01185b0 con 0x7f22d00fda50 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.605+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d01173d0 con 0x7f22d00fda50 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433205568 0 0) 0x7f22d01197b0 con 0x7f22d0104330 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22d01185b0 con 0x7f22d0104330 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3712317982 0 0) 0x7f22b80033f0 con 0x7f22d01007f0 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22d01197b0 con 0x7f22d01007f0 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f22c0000fe0 con 0x7f22d0104330 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1079241702 0 0) 0x7f22d01173d0 con 0x7f22d00fda50 2024-02-10T08:44:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b80033f0 con 0x7f22d00fda50 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f22c40028f0 con 0x7f22d01007f0 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f22c8001c00 con 0x7f22d00fda50 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 96757406 0 0) 0x7f22d01185b0 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 >> v1:172.21.15.107:6790/0 conn(0x7f22d01007f0 legacy=0x7f22d00fa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 >> v1:172.21.15.195:6789/0 conn(0x7f22d00fda50 legacy=0x7f22d00f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d011a9b0 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d01175c0 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22d5978700 10 client.28332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d0117a30 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22d5978700 1 librados: init done 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f22c0002390 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.606+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f22c0003710 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.607+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc004f20 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.607+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f22c0001880 con 0x7f22d0104330 2024-02-10T08:44:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(550..550 src has 420..550) v4 ==== 8181+0+0 (unknown 2346968211 0 0) 0x7f22c0076810 con 0x7f22d0104330 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 10 client.28332.objecter ms_dispatch 0x7f22d0001550 osd_map(550..550 src has 420..550) v4 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 3 client.28332.objecter handle_osd_map got epochs [550,550] > 0 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 3 client.28332.objecter handle_osd_map decoding full epoch 550 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 20 client.28332.objecter dump_active .. 0 homeless 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.608+0000 7f22b7fff700 10 client.28332.objecter ms_handle_connect 0x7f22b8063ca0 2024-02-10T08:44:56.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.612+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f22c0049c50 con 0x7f22d0104330 2024-02-10T08:44:56.646 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:44:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:56.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:44:56.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:56.779 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:56.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:56.800+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f22bc0051d0 con 0x7f22d0104330 2024-02-10T08:44:56.850 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:56.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:56.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:44:56.983 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:57.009 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:44:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:57.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:44:57.103 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:44:57.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:57.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.127+0000 7f2ae3547700 1 -- 172.21.15.107:0/3541726777 >> v1:172.21.15.107:6789/0 conn(0x7f2adc0feea0 legacy=0x7f2adc101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:57.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 -- 172.21.15.107:0/3541726777 shutdown_connections 2024-02-10T08:44:57.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 -- 172.21.15.107:0/3541726777 >> 172.21.15.107:0/3541726777 conn(0x7f2adc06fe70 msgr2=0x7f2adc0f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:57.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 -- 172.21.15.107:0/3541726777 shutdown_connections 2024-02-10T08:44:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 -- 172.21.15.107:0/3541726777 wait complete. 2024-02-10T08:44:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 librados: starting msgr at 2024-02-10T08:44:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.128+0000 7f2ae3547700 1 librados: starting objecter 2024-02-10T08:44:57.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 Processor -- start 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 -- start start 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 librados: setting wanted keys 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 librados: calling monclient init 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2adc10e170 con 0x7f2adc0feea0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2adc104a00 con 0x7f2adc1029b0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae3547700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2adc104b90 con 0x7f2adc0fc050 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae0ae2700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2adc0feea0 0x7f2adc103700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60218/0 (socket says 172.21.15.107:60218) 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ae0ae2700 1 -- 172.21.15.107:0/370707961 learned_addr learned my addr 172.21.15.107:0/370707961 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 10 client.?.objecter ms_handle_connect 0x7f2adc0feea0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322939969 0 0) 0x7f2adc10e170 con 0x7f2adc0feea0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac8003630 con 0x7f2adc0feea0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 10 client.?.objecter ms_handle_connect 0x7f2adc1029b0 2024-02-10T08:44:57.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.129+0000 7f2ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 10 client.?.objecter ms_handle_connect 0x7f2adc0fc050 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1173425979 0 0) 0x7f2ac8003630 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2adc10e170 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2acc002f70 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034559340 0 0) 0x7f2adc104b90 con 0x7f2adc0fc050 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac8003630 con 0x7f2adc0fc050 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 358287449 0 0) 0x7f2adc10e170 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 >> v1:172.21.15.107:6790/0 conn(0x7f2adc0fc050 legacy=0x7f2adc102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 >> v1:172.21.15.195:6789/0 conn(0x7f2adc1029b0 legacy=0x7f2adc11cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc104d60 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc11e330 con 0x7f2adc0feea0 2024-02-10T08:44:57.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ae3547700 10 client.28338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc11e7a0 con 0x7f2adc0feea0 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.130+0000 7f2ae3547700 1 librados: init done 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.131+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2acc002540 con 0x7f2adc0feea0 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.131+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2adc05f780 con 0x7f2adc0feea0 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.132+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2acc003960 con 0x7f2adc0feea0 2024-02-10T08:44:57.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.133+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2acc003ac0 con 0x7f2adc0feea0 2024-02-10T08:44:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.133+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(550..550 src has 420..550) v4 ==== 8181+0+0 (unknown 2346968211 0 0) 0x7f2acc079270 con 0x7f2adc0feea0 2024-02-10T08:44:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.133+0000 7f2ad27fc700 10 client.28338.objecter ms_dispatch 0x7f2adc001550 osd_map(550..550 src has 420..550) v4 2024-02-10T08:44:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.133+0000 7f2ad27fc700 3 client.28338.objecter handle_osd_map got epochs [550,550] > 0 2024-02-10T08:44:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.133+0000 7f2ad27fc700 3 client.28338.objecter handle_osd_map decoding full epoch 550 2024-02-10T08:44:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.134+0000 7f2ad27fc700 20 client.28338.objecter dump_active .. 0 homeless 2024-02-10T08:44:57.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.134+0000 7f2ad27fc700 10 client.28338.objecter ms_handle_connect 0x7f2ac8063ab0 2024-02-10T08:44:57.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.138+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2acc04c7a0 con 0x7f2adc0feea0 2024-02-10T08:44:57.149 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:57.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.213+0000 7f22b7fff700 1 -- 172.21.15.107:0/2889659784 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v551) v1 ==== 97+0+0 (unknown 1404490930 0 0) 0x7f22c00480c0 con 0x7f22d0104330 2024-02-10T08:44:57.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.228+0000 7f22d5978700 10 librados: watch_flush enter 2024-02-10T08:44:57.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.236+0000 7f22d5978700 10 librados: watch_flush exit 2024-02-10T08:44:57.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.236+0000 7f22d5978700 20 client.28332.objecter shutdown clearing up homeless session... 2024-02-10T08:44:57.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.236+0000 7f22d5978700 10 client.28332.objecter successfully canceled tick 2024-02-10T08:44:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.252+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 >> v1:172.21.15.107:6800/2634842009 conn(0x7f22b8063ca0 legacy=0x7f22b8066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.252+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 >> v1:172.21.15.107:6789/0 conn(0x7f22d0104330 legacy=0x7f22d0115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:57.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.274+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 shutdown_connections 2024-02-10T08:44:57.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.274+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 >> 172.21.15.107:0/2889659784 conn(0x7f22d00f3640 msgr2=0x7f22d01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.287+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 shutdown_connections 2024-02-10T08:44:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.290+0000 7f22d5978700 1 -- 172.21.15.107:0/2889659784 wait complete. 2024-02-10T08:44:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.290+0000 7f22d5978700 1 librados: shutdown 2024-02-10T08:44:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:44:57.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:56.107024+0000 mon.a (mon.0) 2857 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T08:44:57.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:56.107094+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-10T08:44:57.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:56.226133+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:56.226432+0000 mon.a (mon.0) 2859 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:56.801579+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:57.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:56.107024+0000 mon.a (mon.0) 2857 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T08:44:57.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:56.107094+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-10T08:44:57.316 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:56.226133+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.316 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:56.226432+0000 mon.a (mon.0) 2859 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.316 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:56.801579+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:57.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:57.362+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7f2adc0475e0 con 0x7f2adc0feea0 2024-02-10T08:44:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:56.107024+0000 mon.a (mon.0) 2857 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T08:44:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:56.107094+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e550: 8 total, 7 up, 7 in 2024-02-10T08:44:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:56.226133+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:56.226432+0000 mon.a (mon.0) 2859 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:44:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:56.801579+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:44:57.779 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:44:57.779 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(50.2s1 112/111 ECSubWrite(tid=333, reqid=client.35222.0:1, at_version=112'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:45.916178+0000", 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 491.86096661200003, 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024985265999999999, 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916178+0000", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916178+0000", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916178+0000", 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1461e-05 2024-02-10T08:44:57.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916199+0000", 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6869999999999994e-06 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916207+0000", 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9530000000000002e-06 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.916214+0000", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032273380000000002 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.919441+0000", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0132000000000001e-05 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.919451+0000", 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021689099999999999 2024-02-10T08:44:57.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.941141+0000", 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3079999999999999e-05 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.941164+0000", 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024985265999999999 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(50.2s1 112/111 ECSubWrite(tid=334, reqid=client.35222.0:1, at_version=0'0, trim_to=0'0, roll_forward_to=112'1))", 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:45.944019+0000", 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 491.83312586099998, 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020696697999999999, 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944019+0000", 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944019+0000", 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944016+0000", 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8201e-05 2024-02-10T08:44:57.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944044+0000", 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3212e-05 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944058+0000", 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7569999999999993e-06 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.944066+0000", 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001033843 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.945100+0000", 2024-02-10T08:44:57.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2311e-05 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.945113+0000", 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019578142999999999 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.964691+0000", 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5114e-05 2024-02-10T08:44:57.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:45.964716+0000", 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020696697999999999 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.35279.0:5 57.1a e118/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head v 118'3, mlcod=0'0)", 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:51.959390+0000", 2024-02-10T08:44:57.790 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 485.81775476500002, 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.053622594000000003, 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959390+0000", 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959390+0000", 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959389+0000", 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1867e-05 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959400+0000", 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0020999999999999e-05 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959410+0000", 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7290000000000003e-06 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:51.959417+0000", 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045816928999999999 2024-02-10T08:44:57.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:52.005234+0000", 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2819999999999998e-05 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:52.005257+0000", 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077052220000000003 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:52.012962+0000", 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0732000000000001e-05 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:52.013013+0000", 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.053622594000000003 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.7.0:168 57.1a e120/115 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:3 v 120'5, mlcod=117'1)", 2024-02-10T08:44:57.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:54.031105+0000", 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 483.74603999800001, 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.027895364999999998, 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031105+0000", 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031105+0000", 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031105+0000", 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1130000000000003e-06 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031112+0000", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9569999999999996e-06 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031119+0000", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1289999999999998e-06 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031126+0000", 2024-02-10T08:44:57.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5760000000000005e-05 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031222+0000", 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1732000000000001e-05 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.031243+0000", 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.027728824999999999 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.058972+0000", 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8294e-05 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.059000+0000", 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.027895364999999998 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.41800.0:2 89.6 89:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:13.006509+0000", 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 464.77063613500002, 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021355539999999999, 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.801 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.41800", 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/787475483", 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006509+0000", 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006509+0000", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006506+0000", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1130000000000007e-06 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006515+0000", 2024-02-10T08:44:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5369000000000001e-05 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006531+0000", 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3780000000000001e-06 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006539+0000", 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3987000000000003e-05 2024-02-10T08:44:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006593+0000", 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014339100000000001 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006737+0000", 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000128255 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.006865+0000", 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051528440000000002 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.012018+0000", 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005107019 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.017125+0000", 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010640266000000001 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.027765+0000", 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3575000000000001e-05 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.027819+0000", 2024-02-10T08:44:57.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5865999999999998e-05 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.027864+0000", 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021355539999999999 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(111.1s1 172/167 ECSubWrite(tid=490, reqid=client.52739.0:7, at_version=172'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:44:57.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:45.966754+0000", 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 431.81039054299998, 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049379487, 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966754+0000", 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966754+0000", 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966754+0000", 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9734e-05 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966784+0000", 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0299999999999994e-06 2024-02-10T08:44:57.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966792+0000", 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4719999999999998e-06 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.966799+0000", 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039139029999999998 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.970713+0000", 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0224000000000001e-05 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.970724+0000", 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045393907999999997 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.016118+0000", 2024-02-10T08:44:57.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6456999999999998e-05 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.016134+0000", 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049379487 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52895.0:2 131.1 131:84bbdd76:test-rados-api-smithi107-112480-34::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected e190)", 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:03.972393+0000", 2024-02-10T08:44:57.813 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 413.80475180100001, 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026181527999999999, 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52895", 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3225455625", 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972393+0000", 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972393+0000", 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972391+0000", 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8090000000000004e-06 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972401+0000", 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6569999999999992e-06 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972411+0000", 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6499999999999996e-06 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972418+0000", 2024-02-10T08:44:57.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8192000000000002e-05 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972466+0000", 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6100000000000006e-06 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.972475+0000", 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022955328000000001 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.995430+0000", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3492999999999994e-05 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.995524+0000", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3874000000000003e-05 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.995618+0000", 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012696420000000001 2024-02-10T08:44:57.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.996887+0000", 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083612699999999999 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.997723+0000", 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000789056 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.998513+0000", 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7620000000000002e-05 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.998550+0000", 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4601e-05 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:03.998575+0000", 2024-02-10T08:44:57.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026181527999999999 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.53144.0:1 168.6s0 168:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e252)", 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:05.979091+0000", 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 351.79805422200002, 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.019819756, 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.821 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.53144", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2407674176", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979091+0000", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979091+0000", 2024-02-10T08:44:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979088+0000", 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0740000000000001e-05 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979099+0000", 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1673e-05 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979110+0000", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7329999999999992e-06 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979119+0000", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3455999999999997e-05 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979183+0000", 2024-02-10T08:44:57.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2279999999999998e-06 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:05.979186+0000", 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0139350579999999 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.993121+0000", 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011120099999999999 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.993232+0000", 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383173 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.993615+0000", 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040522699999999997 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.994020+0000", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048211779999999997 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.998842+0000", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8979999999999993e-05 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:06.998911+0000", 2024-02-10T08:44:57.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.019819756 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.27678.0:1 180.5s0 180:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:31.967542+0000", 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 325.80960327700001, 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.051445797, 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.828 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.27678", 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1617103425", 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967542+0000", 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967542+0000", 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967539+0000", 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5800000000000003e-06 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967547+0000", 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7999999999999999e-06 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967555+0000", 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2360000000000004e-06 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.967560+0000", 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093036200000000001 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.968490+0000", 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9009999999999998e-06 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:31.968493+0000", 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0486733420000001 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.017167+0000", 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012816600000000001 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.017295+0000", 2024-02-10T08:44:57.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046933400000000002 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.017764+0000", 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0006022 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.018366+0000", 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056881900000000003 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.018935+0000", 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2333999999999997e-05 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.018988+0000", 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.051445797 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47971.0:1 181.6s0 181:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e280)", 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:33.979044+0000", 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 323.798101256, 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0103645750000001, 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47971", 2024-02-10T08:44:57.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3424795470", 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979044+0000", 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979044+0000", 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:57.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979041+0000", 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1200000000000008e-06 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979050+0000", 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9049999999999992e-06 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979060+0000", 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7440000000000007e-06 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979068+0000", 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8826999999999997e-05 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979137+0000", 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1499999999999999e-06 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:33.979140+0000", 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0089060080000001 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.988046+0000", 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119814 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.988166+0000", 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032628799999999999 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.988492+0000", 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057385800000000001 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.989066+0000", 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027699700000000002 2024-02-10T08:44:57.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.989343+0000", 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5382999999999994e-05 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:34.989408+0000", 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0103645750000001 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.53297.0:1 187.2s0 187:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:39:49.004581+0000", 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 308.77256371800001, 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.412742779, 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.53297", 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3193245591", 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004581+0000", 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004581+0000", 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004579+0000", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1444e-05 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004601+0000", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9429999999999995e-06 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004610+0000", 2024-02-10T08:44:57.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0500000000000003e-06 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004618+0000", 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6845999999999997e-05 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004674+0000", 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7129999999999999e-06 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:49.004678+0000", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4106301569999999 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.415308+0000", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013551599999999999 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.415444+0000", 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035326800000000001 2024-02-10T08:44:57.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.415797+0000", 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094143500000000001 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.416738+0000", 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050891300000000005 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.417247+0000", 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6724999999999995e-05 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:39:50.417324+0000", 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.412742779 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.849 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.27909.0:1 203.4s0 203:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e368)", 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:41:02.008801+0000", 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 235.76834446300001, 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.012417718, 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.27909", 2024-02-10T08:44:57.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/901028778", 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008801+0000", 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008801+0000", 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:44:57.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008799+0000", 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2459999999999997e-06 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008806+0000", 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5370000000000003e-06 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008815+0000", 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.66e-06 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008821+0000", 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8718999999999999e-05 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008870+0000", 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.362e-06 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:02.008872+0000", 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0080660749999999 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.016938+0000", 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012859699999999999 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.017067+0000", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047038100000000002 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.017537+0000", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00085728900000000001 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.018394+0000", 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027581089999999999 2024-02-10T08:44:57.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.021152+0000", 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5778999999999995e-05 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:41:03.021218+0000", 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.012417718 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25339.0:6699 60.8 e462/450 60:16ef700a:test-rados-api-smithi107-112490-72::foo:head v 462'91, mlcod=460'87)", 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:11.072715+0000", 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.704430488, 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045980884999999999, 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072715+0000", 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072715+0000", 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072714+0000", 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1780000000000003e-06 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072722+0000", 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3860000000000004e-06 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072729+0000", 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7630000000000003e-06 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072735+0000", 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9393e-05 2024-02-10T08:44:57.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072785+0000", 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5420000000000001e-05 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.072800+0000", 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045839837000000001 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.118640+0000", 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5325999999999998e-05 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:11.118695+0000", 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045980884999999999 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.7.52:67 60.8 60:13a5c189:test-rados-api-smithi107-112490-78::c5740e5a8abdfdd24173d1f0746ed7b8f004fd8c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:59.148150+0000", 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.628994531000004, 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047220100000000001, 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148150+0000", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148150+0000", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148149+0000", 2024-02-10T08:44:57.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7239999999999992e-06 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148157+0000", 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1293e-05 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148168+0000", 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6780000000000003e-06 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148175+0000", 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013684600000000001 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148311+0000", 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013975000000000001 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148451+0000", 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020761900000000001 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148659+0000", 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055242199999999998 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.154183+0000", 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041053055999999997 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195236+0000", 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7840000000000003e-05 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195284+0000", 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6670999999999997e-05 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195371+0000", 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047220100000000001 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.7.52:68 60.9 60:95f6ba29:test-rados-api-smithi107-112490-78::50b85ce808c4eb5f4ae987b89988ff8efd7370ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:59.148192+0000", 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.628953371000001, 2024-02-10T08:44:57.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046894844999999998, 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148192+0000", 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148192+0000", 2024-02-10T08:44:57.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148191+0000", 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7510000000000002e-06 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148195+0000", 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7189999999999998e-06 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148200+0000", 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1420000000000003e-06 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148205+0000", 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147684 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148352+0000", 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011524899999999999 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148468+0000", 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000212232 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.148680+0000", 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054527350000000002 2024-02-10T08:44:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.154133+0000", 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040808609000000003 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.194941+0000", 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2652999999999999e-05 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.194994+0000", 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2701999999999994e-05 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195086+0000", 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046894844999999998 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.7.52:66 60.15 e493/480 60:ab1b3caf:test-rados-api-smithi107-112490-78::4b02657dec822853c72fefcaa266fff97efe7078:head v 493'50, mlcod=0'0)", 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:59.173643+0000", 2024-02-10T08:44:57.874 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.603502286999998, 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022356205000000001, 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173643+0000", 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173643+0000", 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:57.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173641+0000", 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4673e-05 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173656+0000", 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.893e-05 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173675+0000", 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0202e-05 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173685+0000", 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021113285999999998 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.194798+0000", 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1143000000000001e-05 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.194820+0000", 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001145124 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195965+0000", 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4193000000000003e-05 2024-02-10T08:44:57.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195999+0000", 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022356205000000001 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.25339.0:7073 220.1e e494/487 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head v 494'6, mlcod=493'4)", 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:00.185066+0000", 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 57.592078682999997, 2024-02-10T08:44:57.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032678365000000001, 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185066+0000", 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185066+0000", 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185066+0000", 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.558e-05 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185082+0000", 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7799999999999995e-06 2024-02-10T08:44:57.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185091+0000", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1239999999999999e-06 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185098+0000", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5929000000000001e-05 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185134+0000", 2024-02-10T08:44:57.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4632000000000001e-05 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.185159+0000", 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032547695000000001 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.217707+0000", 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7969999999999997e-05 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:00.217745+0000", 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032678365000000001 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7299 60.9 60:908a7d88:test-rados-api-smithi107-112490-82::bar:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e510)", 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:17.142948+0000", 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.634197491000002, 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034401090000000002, 2024-02-10T08:44:57.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7299 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142948+0000", 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142948+0000", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142944+0000", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7907999999999999e-05 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142961+0000", 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3224999999999999e-05 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142985+0000", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7180000000000008e-06 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.142994+0000", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107729 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.143102+0000", 2024-02-10T08:44:57.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000138694 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.143241+0000", 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016502899999999999 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.143406+0000", 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082705799999999998 2024-02-10T08:44:57.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.144233+0000", 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032960133000000003 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177193+0000", 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9239e-05 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177252+0000", 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6304999999999994e-05 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177349+0000", 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034401090000000002 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.25339.0:7299 60.9 e511/480 ondisk, result = 0)", 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:17.150903+0000", 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.626242206999997, 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026451921, 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150903+0000", 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150903+0000", 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150901+0000", 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8339999999999994e-06 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150911+0000", 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.257e-05 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150924+0000", 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9739999999999999e-06 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.150930+0000", 2024-02-10T08:44:57.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026242884000000001 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177172+0000", 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1138000000000001e-05 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177184+0000", 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000171161 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.177355+0000", 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026451921 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.3.25:68 60.9 60:908a7d88:test-rados-api-smithi107-112490-82::bar:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e511)", 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:17.179373+0000", 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.597772032000002, 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.033538680000000001, 2024-02-10T08:44:57.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179373+0000", 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179373+0000", 2024-02-10T08:44:57.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179371+0000", 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1919999999999994e-06 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179380+0000", 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5818e-05 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179396+0000", 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9290000000000001e-06 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179403+0000", 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015305533 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:44:57.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.194708+0000", 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6118000000000004e-05 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.194804+0000", 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021058000000000001 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.195015+0000", 2024-02-10T08:44:57.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063137449999999999 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.201329+0000", 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011413929 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.212743+0000", 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2959000000000005e-05 2024-02-10T08:44:57.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.212806+0000", 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106075 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.212912+0000", 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.033538680000000001 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:44:57.902 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:44:57.902 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:44:58.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.214+0000 7f2ad27fc700 1 -- 172.21.15.107:0/370707961 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v552) v1 ==== 146+0+0 (unknown 344012716 0 0) 0x7f2acc04ac10 con 0x7f2adc0feea0 2024-02-10T08:44:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.219+0000 7f2ae3547700 10 librados: watch_flush enter 2024-02-10T08:44:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.226+0000 7f2ae3547700 10 librados: watch_flush exit 2024-02-10T08:44:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.226+0000 7f2ae3547700 20 client.28338.objecter shutdown clearing up homeless session... 2024-02-10T08:44:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.226+0000 7f2ae3547700 10 client.28338.objecter successfully canceled tick 2024-02-10T08:44:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.231+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2ac8063ab0 legacy=0x7f2ac8065f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.231+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 >> v1:172.21.15.107:6789/0 conn(0x7f2adc0feea0 legacy=0x7f2adc103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:44:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.235+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 shutdown_connections 2024-02-10T08:44:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.235+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 >> 172.21.15.107:0/370707961 conn(0x7f2adc06fe70 msgr2=0x7f2adc0f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:44:58.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.256+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 shutdown_connections 2024-02-10T08:44:58.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.264+0000 7f2ae3547700 1 -- 172.21.15.107:0/370707961 wait complete. 2024-02-10T08:44:58.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:44:58.275+0000 7f2ae3547700 1 librados: shutdown 2024-02-10T08:44:58.297 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pgp_num to 1 2024-02-10T08:44:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:44:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:58.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:44:58.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:44:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:57.200348+0000 mon.a (mon.0) 2861 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:57.205497+0000 mon.a (mon.0) 2862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:57.205563+0000 mon.a (mon.0) 2863 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:58.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:57.205597+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:57.215640+0000 mon.b (mon.1) 684 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:57.216057+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:57.364292+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:57.494855+0000 mgr.y (mgr.14553) 486 : cluster [DBG] pgmap v829: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1158 objects degraded (16.321%) 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:57.200348+0000 mon.a (mon.0) 2861 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:57.205497+0000 mon.a (mon.0) 2862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:57.205563+0000 mon.a (mon.0) 2863 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:57.205597+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-10T08:44:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:57.215640+0000 mon.b (mon.1) 684 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:57.216057+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:57.364292+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-10T08:44:58.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:57.494855+0000 mgr.y (mgr.14553) 486 : cluster [DBG] pgmap v829: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1158 objects degraded (16.321%) 2024-02-10T08:44:58.454 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:57.200348+0000 mon.a (mon.0) 2861 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:57.205497+0000 mon.a (mon.0) 2862 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]': finished 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:57.205563+0000 mon.a (mon.0) 2863 : audit [INF] from='client.? v1:172.21.15.107:0/2889659784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:57.205597+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e551: 8 total, 7 up, 7 in 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:57.215640+0000 mon.b (mon.1) 684 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:57.216057+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]: dispatch 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:57.364292+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-10T08:44:58.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:57.494855+0000 mgr.y (mgr.14553) 486 : cluster [DBG] pgmap v829: 218 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 55 active+undersized, 124 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1158 objects degraded (16.321%) 2024-02-10T08:44:59.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:58.201312+0000 mon.a (mon.0) 2867 : cluster [WRN] Health check update: Degraded data redundancy: 189/1158 objects degraded (16.321%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:59.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:58.209087+0000 mon.a (mon.0) 2868 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:59.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:44:58.209160+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-10T08:44:59.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:58.209209+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-10T08:44:59.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:58.201312+0000 mon.a (mon.0) 2867 : cluster [WRN] Health check update: Degraded data redundancy: 189/1158 objects degraded (16.321%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:58.209087+0000 mon.a (mon.0) 2868 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:44:58.209160+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-10T08:44:59.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:44:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:58.209209+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-10T08:44:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:44:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:44:59.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:44:59.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:44:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:58.201312+0000 mon.a (mon.0) 2867 : cluster [WRN] Health check update: Degraded data redundancy: 189/1158 objects degraded (16.321%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:44:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:58.209087+0000 mon.a (mon.0) 2868 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-90"}]': finished 2024-02-10T08:44:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:44:58.209160+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.107:0/370707961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-10T08:44:59.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:44:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:58.209209+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e552: 8 total, 7 up, 7 in 2024-02-10T08:44:59.539 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:44:59.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:45:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:59.213018+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-10T08:45:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:44:59.495652+0000 mgr.y (mgr.14553) 487 : cluster [DBG] pgmap v832: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:45:00.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:59.213018+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-10T08:45:00.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:44:59.495652+0000 mgr.y (mgr.14553) 487 : cluster [DBG] pgmap v832: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:45:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:59.213018+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e553: 8 total, 7 up, 7 in 2024-02-10T08:45:00.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:44:59.495652+0000 mgr.y (mgr.14553) 487 : cluster [DBG] pgmap v832: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 189/1155 objects degraded (16.364%) 2024-02-10T08:45:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:00.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:00.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:00.920 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:01.121 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:01.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:01.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:01.262 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:01.360 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:01.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:45:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:01.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:01.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:00.221056+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:00.231526+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:00.231897+0000 mon.a (mon.0) 2873 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.090533+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:01.090611+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.134017+0000 mon.b (mon.1) 686 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.139757+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.146526+0000 mon.a (mon.0) 2876 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:00.221056+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:00.231526+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:00.231897+0000 mon.a (mon.0) 2873 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.090533+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:01.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:01.090611+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-10T08:45:01.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.134017+0000 mon.b (mon.1) 686 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:45:01.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.139757+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.146526+0000 mon.a (mon.0) 2876 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.465+0000 7f66ba5f1700 1 -- 172.21.15.107:0/2765298221 >> v1:172.21.15.107:6790/0 conn(0x7f66b40f9530 legacy=0x7f66b4102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 -- 172.21.15.107:0/2765298221 shutdown_connections 2024-02-10T08:45:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 -- 172.21.15.107:0/2765298221 >> 172.21.15.107:0/2765298221 conn(0x7f66b40f2510 msgr2=0x7f66b40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 -- 172.21.15.107:0/2765298221 shutdown_connections 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 -- 172.21.15.107:0/2765298221 wait complete. 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 librados: starting msgr at 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 librados: starting objecter 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 Processor -- start 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 -- start start 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 librados: setting wanted keys 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.466+0000 7f66ba5f1700 1 librados: calling monclient init 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66ba5f1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b4117330 con 0x7f66b40f8a40 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66ba5f1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b4118510 con 0x7f66b4104340 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66ba5f1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b4119710 con 0x7f66b40f9530 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b37fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f66b40f9530 0x7f66b40ffc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49878/0 (socket says 172.21.15.107:49878) 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b37fe700 1 -- 172.21.15.107:0/1429327942 learned_addr learned my addr 172.21.15.107:0/1429327942 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter ms_handle_connect 0x7f66b40f9530 2024-02-10T08:45:01.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter ms_handle_connect 0x7f66b40f8a40 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter ms_handle_connect 0x7f66b4104340 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638119074 0 0) 0x7f66b4117330 con 0x7f66b40f8a40 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c0033f0 con 0x7f66b40f8a40 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954466645 0 0) 0x7f66b4118510 con 0x7f66b4104340 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b4117330 con 0x7f66b4104340 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54046697 0 0) 0x7f66b4119710 con 0x7f66b40f9530 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b4118510 con 0x7f66b40f9530 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364035361 0 0) 0x7f669c0033f0 con 0x7f66b40f8a40 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b4119710 con 0x7f66b40f8a40 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515990775 0 0) 0x7f66b4117330 con 0x7f66b4104340 2024-02-10T08:45:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f669c0033f0 con 0x7f66b4104340 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66a4001680 con 0x7f66b40f8a40 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335453542 0 0) 0x7f66b4118510 con 0x7f66b40f9530 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b4117330 con 0x7f66b40f9530 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66ac001bc0 con 0x7f66b4104340 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66a8002050 con 0x7f66b40f9530 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 489227993 0 0) 0x7f66b4119710 con 0x7f66b40f8a40 2024-02-10T08:45:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 >> v1:172.21.15.107:6790/0 conn(0x7f66b40f9530 legacy=0x7f66b40ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 >> v1:172.21.15.195:6789/0 conn(0x7f66b4104340 legacy=0x7f66b41159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b411a910 con 0x7f66b40f8a40 2024-02-10T08:45:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.467+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b4117520 con 0x7f66b40f8a40 2024-02-10T08:45:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.468+0000 7f66ba5f1700 10 client.28344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.468+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b4117990 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.468+0000 7f66ba5f1700 1 librados: init done 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.468+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f66a40022b0 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.468+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f66a4003320 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.469+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f66a0004f00 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.469+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f66a4003560 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(555..555 src has 420..555) v4 ==== 8173+0+0 (unknown 883365101 0 0) 0x7f66a4076510 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 osd_map(555..555 src has 420..555) v4 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 3 client.28344.objecter handle_osd_map got epochs [555,555] > 0 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 3 client.28344.objecter handle_osd_map decoding full epoch 555 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 20 client.28344.objecter dump_active .. 0 homeless 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v555) v1 ==== 54+0+16 (unknown 4262135193 0 2420000192) 0x7f66a4078960 con 0x7f66b40f8a40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f669c063c40 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 1 2024-02-10T08:45:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 1 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 0 1 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 4 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 10 client.28344.objecter _send_command 1 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a0004950 con 0x7f66a0004530 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.470+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a0004530 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.471+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a0004950 con 0x7f66a0004530 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.471+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.471+0000 7f66b17fa700 10 client.28344.objecter _finish_command 1 = system:0 2024-02-10T08:45:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.471+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 4 2024-02-10T08:45:01.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.471+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 0 1 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:00.221056+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e554: 8 total, 7 up, 7 in 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:00.231526+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:00.231897+0000 mon.a (mon.0) 2873 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.090533+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:01.090611+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e555: 8 total, 7 up, 7 in 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.134017+0000 mon.b (mon.1) 686 : audit [DBG] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.139757+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.146526+0000 mon.a (mon.0) 2876 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T08:45:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 2 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 2 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a00051a0 osd=0 3 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 0 2 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 4 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 10 client.28344.objecter _send_command 2 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.490+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a0004d30 con 0x7f66a0004530 2024-02-10T08:45:01.491 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a0004d30 con 0x7f66a0004530 2024-02-10T08:45:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66b17fa700 10 client.28344.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a00051a0 osd=0 4 2024-02-10T08:45:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 0 2 2024-02-10T08:45:01.493 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 1 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 4 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 10 client.28344.objecter _send_command 3 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.492+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a000c000 con 0x7f66a0004150 2024-02-10T08:45:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.493+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a0004150 2024-02-10T08:45:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.494+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a000c000 con 0x7f66a0004150 2024-02-10T08:45:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.494+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.494+0000 7f66b17fa700 10 client.28344.objecter _finish_command 3 = system:0 2024-02-10T08:45:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.494+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 4 2024-02-10T08:45:01.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.494+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 1 3 2024-02-10T08:45:01.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 4 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 4 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003fe0 osd=1 3 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 1 4 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 4 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 10 client.28344.objecter _send_command 4 2024-02-10T08:45:01.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.509+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a0001850 con 0x7f66a0004150 2024-02-10T08:45:01.511 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.511+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a0001850 con 0x7f66a0004150 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.511+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.511+0000 7f66b17fa700 10 client.28344.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.511+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003fe0 osd=1 4 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.511+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 1 4 2024-02-10T08:45:01.512 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 5 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 5 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 2 5 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 4 2024-02-10T08:45:01.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 10 client.28344.objecter _send_command 5 2024-02-10T08:45:01.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a0003e70 con 0x7f66a000d2f0 2024-02-10T08:45:01.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.512+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a000d2f0 2024-02-10T08:45:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.513+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a0003e70 con 0x7f66a000d2f0 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.513+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.513+0000 7f66b17fa700 10 client.28344.objecter _finish_command 5 = system:0 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.513+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 4 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.513+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 2 5 2024-02-10T08:45:01.515 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:01.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:01.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 6 2024-02-10T08:45:01.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 6 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003630 osd=2 3 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 2 6 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 4 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 10 client.28344.objecter _send_command 6 2024-02-10T08:45:01.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.530+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a000e6e0 con 0x7f66a000d2f0 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.532+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a000e6e0 con 0x7f66a000d2f0 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.532+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.532+0000 7f66b17fa700 10 client.28344.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.532+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003630 osd=2 4 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.532+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 2 6 2024-02-10T08:45:01.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 7 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 7 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 3 7 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 4 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 10 client.28344.objecter _send_command 7 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a0013b40 con 0x7f66a0011e80 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.533+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a0011e80 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.534+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a0013b40 con 0x7f66a0011e80 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.534+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.534+0000 7f66b17fa700 10 client.28344.objecter _finish_command 7 = system:0 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.534+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 4 2024-02-10T08:45:01.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.534+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 3 7 2024-02-10T08:45:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 8 2024-02-10T08:45:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 8 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a0003440 osd=3 3 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 3 8 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 4 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 10 client.28344.objecter _send_command 8 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.560+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a001a4f0 con 0x7f66a0011e80 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a001a4f0 con 0x7f66a0011e80 2024-02-10T08:45:01.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66b17fa700 10 client.28344.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a0003440 osd=3 4 2024-02-10T08:45:01.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 3 8 2024-02-10T08:45:01.562 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 9 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 20 client.28344.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 10 client.28344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=556}) v3 -- 0x7f66a001cc50 con 0x7f66b40f8a40 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.561+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66a001cf40 con 0x7f66b40f8a40 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_get_version_reply(handle=1 version=555) v2 ==== 24+0+0 (unknown 3004988286 0 0) 0x7f66a4047eb0 con 0x7f66b40f8a40 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66b27fc700 10 client.28344.objecter _check_command_map_dne tid 9 current 555 map_dne_bound 555 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66b27fc700 10 client.28344.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:01.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66b27fc700 15 client.28344.objecter _session_command_op_remove -1 9 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 10 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 10 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 5 10 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 4 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 10 client.28344.objecter _send_command 10 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a000e420 con 0x7f66a000ce50 2024-02-10T08:45:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.562+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a000ce50 2024-02-10T08:45:01.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.563+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a000e420 con 0x7f66a000ce50 2024-02-10T08:45:01.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.563+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:01.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.563+0000 7f66b17fa700 10 client.28344.objecter _finish_command 10 = system:0 2024-02-10T08:45:01.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.563+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 4 2024-02-10T08:45:01.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.563+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 5 10 2024-02-10T08:45:01.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 11 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 11 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a000e1b0 osd=5 3 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 5 11 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 4 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 10 client.28344.objecter _send_command 11 2024-02-10T08:45:01.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.578+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a000e420 con 0x7f66a000ce50 2024-02-10T08:45:01.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a000e420 con 0x7f66a000ce50 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66b17fa700 10 client.28344.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a000e1b0 osd=5 4 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 5 11 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:01.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 12 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 12 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 6 12 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 4 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 10 client.28344.objecter _send_command 12 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.580+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a001c8e0 con 0x7f66a001c400 2024-02-10T08:45:01.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.581+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a001c400 2024-02-10T08:45:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.582+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a001c8e0 con 0x7f66a001c400 2024-02-10T08:45:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.582+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.582+0000 7f66b17fa700 10 client.28344.objecter _finish_command 12 = system:0 2024-02-10T08:45:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.582+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 4 2024-02-10T08:45:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.582+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 6 12 2024-02-10T08:45:01.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 13 2024-02-10T08:45:01.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 13 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a001bd80 osd=6 3 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 6 13 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 4 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 10 client.28344.objecter _send_command 13 2024-02-10T08:45:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.597+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a001c8e0 con 0x7f66a001c400 2024-02-10T08:45:01.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a001c8e0 con 0x7f66a001c400 2024-02-10T08:45:01.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66b17fa700 10 client.28344.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a001bd80 osd=6 4 2024-02-10T08:45:01.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 6 13 2024-02-10T08:45:01.601 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:01.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:01.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 14 2024-02-10T08:45:01.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 14 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 7 14 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 4 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 10 client.28344.objecter _send_command 14 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.599+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a00278c0 con 0x7f66a0023d10 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.600+0000 7f66b17fa700 10 client.28344.objecter ms_handle_connect 0x7f66a0023d10 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.601+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f66a00278c0 con 0x7f66a0023d10 2024-02-10T08:45:01.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.601+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:01.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.601+0000 7f66b17fa700 10 client.28344.objecter _finish_command 14 = system:0 2024-02-10T08:45:01.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.601+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 4 2024-02-10T08:45:01.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.601+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 7 14 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 10 client.28344.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign -1 15 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 20 client.28344.objecter _get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_remove -1 15 2024-02-10T08:45:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 20 client.28344.objecter get_session s=0x7f66a001c8e0 osd=7 3 2024-02-10T08:45:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 15 client.28344.objecter _session_command_op_assign 7 15 2024-02-10T08:45:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 4 2024-02-10T08:45:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 10 client.28344.objecter _send_command 15 2024-02-10T08:45:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.615+0000 7f66ba5f1700 1 -- 172.21.15.107:0/1429327942 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a0029070 con 0x7f66a0023d10 2024-02-10T08:45:01.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.617+0000 7f66b17fa700 1 -- 172.21.15.107:0/1429327942 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f66a0029070 con 0x7f66a0023d10 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.617+0000 7f66b17fa700 10 client.28344.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.617+0000 7f66b17fa700 10 client.28344.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.617+0000 7f66b17fa700 20 client.28344.objecter put_session s=0x7f66a001c8e0 osd=7 4 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:01.617+0000 7f66b17fa700 15 client.28344.objecter _session_command_op_remove 7 15 2024-02-10T08:45:01.617 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:45:01.644 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:01.662 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:01.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:01.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:01.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:01.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:01.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:02.022 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:02.040 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:02.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:02.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:02.225 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.470377+0000 mon.a (mon.0) 2877 : audit [DBG] from='client.? v1:172.21.15.107:0/1429327942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:01.496543+0000 mgr.y (mgr.14553) 488 : cluster [DBG] pgmap v835: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:01.616156+0000 mgr.y (mgr.14553) 489 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:02.111750+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:02.111819+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:02.126006+0000 mon.b (mon.1) 688 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:02.126909+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:02.414 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:02.437 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:02.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:02.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.470377+0000 mon.a (mon.0) 2877 : audit [DBG] from='client.? v1:172.21.15.107:0/1429327942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:01.496543+0000 mgr.y (mgr.14553) 488 : cluster [DBG] pgmap v835: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:01.616156+0000 mgr.y (mgr.14553) 489 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:02.111750+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:02.111819+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:02.126006+0000 mon.b (mon.1) 688 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:02.126909+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.470377+0000 mon.a (mon.0) 2877 : audit [DBG] from='client.? v1:172.21.15.107:0/1429327942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:01.496543+0000 mgr.y (mgr.14553) 488 : cluster [DBG] pgmap v835: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:01.616156+0000 mgr.y (mgr.14553) 489 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:02.111750+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:02.111819+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e556: 8 total, 7 up, 7 in 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:02.126006+0000 mon.b (mon.1) 688 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:02.126909+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T08:45:02.568 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:02.590 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:02.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:02.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:02.721 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:02.740 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:02.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:02.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:03.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:03.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:03.258 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:03.277 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:03.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:03.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:03.426 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:03.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:03.530 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:03.556 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:03.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:03.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:03.904 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:03.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:04.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:04.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:04.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:03.115213+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:45:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:03.115288+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-10T08:45:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:03.125641+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:03.126938+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:03.497171+0000 mgr.y (mgr.14553) 490 : cluster [DBG] pgmap v838: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:04.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:03.115213+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:03.115288+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:03.125641+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:03.126938+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:03.497171+0000 mgr.y (mgr.14553) 490 : cluster [DBG] pgmap v838: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:03.115213+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T08:45:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:03.115288+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e557: 8 total, 7 up, 7 in 2024-02-10T08:45:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:03.125641+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:03.126938+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T08:45:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:03.497171+0000 mgr.y (mgr.14553) 490 : cluster [DBG] pgmap v838: 218 pgs: 2 creating+activating, 1 active+clean+remapped, 14 creating+peering, 38 active+undersized+degraded, 1 creating+activating+undersized, 49 active+undersized, 113 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:04.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:04.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:04.856 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:04.875 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:04.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:04.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:05.005 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:05.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:05.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:05.267 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:05.288 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:05.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:05.363 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:45:05.364 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:04.117771+0000 mon.a (mon.0) 2884 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:04.117850+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:04.296208+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:04.296552+0000 mon.a (mon.0) 2886 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:04.297135+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:04.297374+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.384 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:04.117771+0000 mon.a (mon.0) 2884 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:45:05.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:04.117850+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-10T08:45:05.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:04.296208+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:04.296552+0000 mon.a (mon.0) 2886 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:04.297135+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:04.297374+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:05.419 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:04.117771+0000 mon.a (mon.0) 2884 : audit [INF] from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:04.117850+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e558: 8 total, 7 up, 7 in 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:04.296208+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:04.296552+0000 mon.a (mon.0) 2886 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-6"}]: dispatch 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:04.297135+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.107:0/949878047' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:04.297374+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-6", "tierpool": "test-rados-api-smithi107-112490-92"}]: dispatch 2024-02-10T08:45:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:05.669 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:05] "GET /metrics HTTP/1.1" 200 189137 "" "Prometheus/2.33.4" 2024-02-10T08:45:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:05.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:05.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:05.798 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:05.851 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:45:05.851 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:45:05.851 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:45:05.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:45:06.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:45:06.248 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:06.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:06.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:06.377 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:05.124243+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-10T08:45:06.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:05.497947+0000 mgr.y (mgr.14553) 491 : cluster [DBG] pgmap v841: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:06.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:06.087237+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check update: Degraded data redundancy: 190/1158 objects degraded (16.408%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:06.378 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:06.397 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:06.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:05.124243+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-10T08:45:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:05.497947+0000 mgr.y (mgr.14553) 491 : cluster [DBG] pgmap v841: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:06.087237+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check update: Degraded data redundancy: 190/1158 objects degraded (16.408%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:05.124243+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e559: 8 total, 7 up, 7 in 2024-02-10T08:45:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:05.497947+0000 mgr.y (mgr.14553) 491 : cluster [DBG] pgmap v841: 186 pgs: 1 active+clean+remapped, 38 active+undersized+degraded, 41 active+undersized, 106 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 190/1158 objects degraded (16.408%) 2024-02-10T08:45:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:06.087237+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check update: Degraded data redundancy: 190/1158 objects degraded (16.408%), 38 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:06.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.503+0000 7f686a2f0700 1 -- 172.21.15.107:0/1027435218 >> v1:172.21.15.107:6790/0 conn(0x7f68640f8a60 legacy=0x7f68640f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.503+0000 7f686a2f0700 1 -- 172.21.15.107:0/1027435218 shutdown_connections 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.503+0000 7f686a2f0700 1 -- 172.21.15.107:0/1027435218 >> 172.21.15.107:0/1027435218 conn(0x7f68640f2510 msgr2=0x7f68640f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.503+0000 7f686a2f0700 1 -- 172.21.15.107:0/1027435218 shutdown_connections 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.503+0000 7f686a2f0700 1 -- 172.21.15.107:0/1027435218 wait complete. 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 librados: starting msgr at 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 librados: starting objecter 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 Processor -- start 2024-02-10T08:45:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 -- start start 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 librados: setting wanted keys 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 librados: calling monclient init 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6864117540 con 0x7f68640f9550 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6864118720 con 0x7f68640f8a60 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686a2f0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6864119920 con 0x7f6864104360 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686888d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6864104360 0x7f6864115c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49910/0 (socket says 172.21.15.107:49910) 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f686888d700 1 -- 172.21.15.107:0/1282765791 learned_addr learned my addr 172.21.15.107:0/1282765791 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f68617fa700 10 client.?.objecter ms_handle_connect 0x7f68640f9550 2024-02-10T08:45:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f68617fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.504+0000 7f68617fa700 10 client.?.objecter ms_handle_connect 0x7f6864104360 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 10 client.?.objecter ms_handle_connect 0x7f68640f8a60 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545545174 0 0) 0x7f6864117540 con 0x7f68640f9550 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c003630 con 0x7f68640f9550 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969978814 0 0) 0x7f6864119920 con 0x7f6864104360 2024-02-10T08:45:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6864117540 con 0x7f6864104360 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193117748 0 0) 0x7f6864118720 con 0x7f68640f8a60 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6864119920 con 0x7f68640f8a60 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575042652 0 0) 0x7f684c003630 con 0x7f68640f9550 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6864118720 con 0x7f68640f9550 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6854001880 con 0x7f68640f9550 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838939805 0 0) 0x7f6864117540 con 0x7f6864104360 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f684c003630 con 0x7f6864104360 2024-02-10T08:45:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f685c001c40 con 0x7f6864104360 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1232113167 0 0) 0x7f6864119920 con 0x7f68640f8a60 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6864117540 con 0x7f68640f8a60 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6858001ec0 con 0x7f68640f8a60 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2473510196 0 0) 0x7f684c003630 con 0x7f6864104360 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 >> v1:172.21.15.195:6789/0 conn(0x7f68640f8a60 legacy=0x7f68640ff8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 >> v1:172.21.15.107:6789/0 conn(0x7f68640f9550 legacy=0x7f68640ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686411ab20 con 0x7f6864104360 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864119b10 con 0x7f6864104360 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f686a2f0700 10 client.53924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6864119f80 con 0x7f6864104360 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f686a2f0700 1 librados: init done 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f685c002300 con 0x7f6864104360 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.505+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f685c005df0 con 0x7f6864104360 2024-02-10T08:45:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.506+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6850004f20 con 0x7f6864104360 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.506+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f685c001e60 con 0x7f6864104360 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(560..560 src has 420..560) v4 ==== 7386+0+0 (unknown 3575636949 0 0) 0x7f685c078d80 con 0x7f6864104360 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 10 client.53924.objecter ms_dispatch 0x7f6864001550 osd_map(560..560 src has 420..560) v4 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 3 client.53924.objecter handle_osd_map got epochs [560,560] > 0 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 3 client.53924.objecter handle_osd_map decoding full epoch 560 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 20 client.53924.objecter dump_active .. 0 homeless 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.507+0000 7f68617fa700 10 client.53924.objecter ms_handle_connect 0x7f684c063c10 2024-02-10T08:45:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.511+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f685c04c410 con 0x7f6864104360 2024-02-10T08:45:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:06.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:06.560 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:06.602 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:06.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:06.676 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:06.694 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:06.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:06.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:06.707+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f68500051d0 con 0x7f6864104360 2024-02-10T08:45:06.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:06.827 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:07.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:07.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:07.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.138+0000 7f68617fa700 1 -- 172.21.15.107:0/1282765791 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v561) v1 ==== 91+0+0 (unknown 1543658208 0 0) 0x7f685c04a820 con 0x7f6864104360 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 10 librados: watch_flush enter 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 10 librados: watch_flush exit 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 20 client.53924.objecter shutdown clearing up homeless session... 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 10 client.53924.objecter successfully canceled tick 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 >> v1:172.21.15.107:6800/2634842009 conn(0x7f684c063c10 legacy=0x7f684c0660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 >> v1:172.21.15.107:6790/0 conn(0x7f6864104360 legacy=0x7f6864115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 shutdown_connections 2024-02-10T08:45:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.144+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 >> 172.21.15.107:0/1282765791 conn(0x7f68640f2510 msgr2=0x7f68641077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.146+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 shutdown_connections 2024-02-10T08:45:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.151+0000 7f686a2f0700 1 -- 172.21.15.107:0/1282765791 wait complete. 2024-02-10T08:45:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:07.158+0000 7f686a2f0700 1 librados: shutdown 2024-02-10T08:45:07.168 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:45:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:07.408 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:06.130263+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:06.169011+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:06.171115+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:06.171429+0000 mon.a (mon.0) 2893 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:06.707949+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.107:0/1282765791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:06.708246+0000 mon.a (mon.0) 2894 : audit [INF] from='client.53924 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:06.130263+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:06.169011+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:06.171115+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.410 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:06.171429+0000 mon.a (mon.0) 2893 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:07.411 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:06.707949+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.107:0/1282765791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.411 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:06.708246+0000 mon.a (mon.0) 2894 : audit [INF] from='client.53924 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:07.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:07.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:06.130263+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e560: 8 total, 7 up, 7 in 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:06.169011+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:06.171115+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:06.171429+0000 mon.a (mon.0) 2893 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:06.707949+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.107:0/1282765791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:06.708246+0000 mon.a (mon.0) 2894 : audit [INF] from='client.53924 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:07.966 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:07.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:08.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:08.099 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:08.118 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:08.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:08.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:08.247 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:08.266 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:08.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:08.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:08.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.126906+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:07.134016+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:07.134092+0000 mon.a (mon.0) 2897 : audit [INF] from='client.53924 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.134144+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:07.140460+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.498497+0000 mgr.y (mgr.14553) 492 : cluster [DBG] pgmap v844: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.126906+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:07.134016+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:08.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:07.134092+0000 mon.a (mon.0) 2897 : audit [INF] from='client.53924 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:08.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.134144+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-10T08:45:08.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:07.140460+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:08.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.498497+0000 mgr.y (mgr.14553) 492 : cluster [DBG] pgmap v844: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.126906+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:07.134016+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:07.134092+0000 mon.a (mon.0) 2897 : audit [INF] from='client.53924 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.134144+0000 mon.a (mon.0) 2898 : cluster [DBG] osdmap e561: 8 total, 7 up, 7 in 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:07.140460+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.498497+0000 mgr.y (mgr.14553) 492 : cluster [DBG] pgmap v844: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:09.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:09.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:09.477 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:09.495 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:09.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.458932+0000 osd.3 (osd.3) 257 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.459657+0000 osd.3 (osd.3) 258 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.681275+0000 osd.1 (osd.1) 268 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:07.684761+0000 osd.1 (osd.1) 269 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:08.374508+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-10T08:45:09.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:09.134426+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.134510+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.458932+0000 osd.3 (osd.3) 257 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.459657+0000 osd.3 (osd.3) 258 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.681275+0000 osd.1 (osd.1) 268 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:07.684761+0000 osd.1 (osd.1) 269 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:08.374508+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:09.134426+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:09.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.134510+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-10T08:45:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:09.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.458932+0000 osd.3 (osd.3) 257 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.459657+0000 osd.3 (osd.3) 258 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.681275+0000 osd.1 (osd.1) 268 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:07.684761+0000 osd.1 (osd.1) 269 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:08.374508+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e562: 8 total, 7 up, 7 in 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:09.134426+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi107-112490-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.134510+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e563: 8 total, 7 up, 7 in 2024-02-10T08:45:09.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:09.866 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:09.884 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:09.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:09.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:10.016 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.453286+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:45:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.454289+0000 osd.7 (osd.7) 228 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:45:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.466000+0000 osd.2 (osd.2) 211 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:45:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.467609+0000 osd.2 (osd.2) 212 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:45:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:09.499169+0000 mgr.y (mgr.14553) 493 : cluster [DBG] pgmap v847: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:10.116433+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.107:0/3224362507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:10.140932+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-10T08:45:10.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.453286+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:45:10.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.454289+0000 osd.7 (osd.7) 228 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.466000+0000 osd.2 (osd.2) 211 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.467609+0000 osd.2 (osd.2) 212 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:09.499169+0000 mgr.y (mgr.14553) 493 : cluster [DBG] pgmap v847: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:10.116433+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.107:0/3224362507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:10.140932+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.453286+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.454289+0000 osd.7 (osd.7) 228 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.466000+0000 osd.2 (osd.2) 211 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.467609+0000 osd.2 (osd.2) 212 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:45:10.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:09.499169+0000 mgr.y (mgr.14553) 493 : cluster [DBG] pgmap v847: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:10.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:10.116433+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.107:0/3224362507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:10.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:10.140932+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e564: 8 total, 7 up, 7 in 2024-02-10T08:45:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:10.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:10.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.935+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3759487913 >> v1:172.21.15.107:6789/0 conn(0x7f70b4103320 legacy=0x7f70b41057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.935+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3759487913 shutdown_connections 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.935+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3759487913 >> 172.21.15.107:0/3759487913 conn(0x7f70b406c090 msgr2=0x7f70b406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.935+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3759487913 shutdown_connections 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3759487913 wait complete. 2024-02-10T08:45:10.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 librados: starting msgr at 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 librados: starting objecter 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 Processor -- start 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 -- start start 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 librados: setting wanted keys 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 librados: calling monclient init 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70b4117390 con 0x7f70b40ff720 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70b4118570 con 0x7f70b4103320 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.936+0000 7f70ba5aa700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f70b4119770 con 0x7f70b40fc960 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b3fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f70b40fc960 0x7f70b40f9980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45666/0 (socket says 172.21.15.107:45666) 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b3fff700 1 -- 172.21.15.107:0/3224362507 learned_addr learned my addr 172.21.15.107:0/3224362507 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter ms_handle_connect 0x7f70b40ff720 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter ms_handle_connect 0x7f70b40fc960 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter ms_handle_connect 0x7f70b4103320 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152246341 0 0) 0x7f70b4119770 con 0x7f70b40fc960 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c0033f0 con 0x7f70b40fc960 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335786357 0 0) 0x7f70b4117390 con 0x7f70b40ff720 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4119770 con 0x7f70b40ff720 2024-02-10T08:45:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686174217 0 0) 0x7f70b4118570 con 0x7f70b4103320 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4117390 con 0x7f70b4103320 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818269311 0 0) 0x7f709c0033f0 con 0x7f70b40fc960 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b4118570 con 0x7f70b40fc960 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70a40019d0 con 0x7f70b40fc960 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197328539 0 0) 0x7f70b4117390 con 0x7f70b4103320 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f709c0033f0 con 0x7f70b4103320 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1845298815 0 0) 0x7f70b4119770 con 0x7f70b40ff720 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b4117390 con 0x7f70b40ff720 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70ac002310 con 0x7f70b4103320 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70a8001840 con 0x7f70b40ff720 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1701486272 0 0) 0x7f70b4118570 con 0x7f70b40fc960 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 >> v1:172.21.15.195:6789/0 conn(0x7f70b4103320 legacy=0x7f70b4115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 >> v1:172.21.15.107:6789/0 conn(0x7f70b40ff720 legacy=0x7f70b40fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b411a970 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b4119960 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70ba5aa700 10 client.53930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b4119dd0 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.937+0000 7f70ba5aa700 1 librados: init done 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.938+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f70a4002530 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.938+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f70a4005bc0 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.938+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0004f20 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f70a4018b10 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(563..563 src has 420..563) v4 ==== 8214+0+0 (unknown 243004123 0 0) 0x7f70a4078e10 con 0x7f70b40fc960 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 10 client.53930.objecter ms_dispatch 0x7f70b4001550 osd_map(563..563 src has 420..563) v4 2024-02-10T08:45:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 3 client.53930.objecter handle_osd_map got epochs [563,563] > 0 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 3 client.53930.objecter handle_osd_map decoding full epoch 563 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 20 client.53930.objecter dump_active .. 0 homeless 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.939+0000 7f70b17fa700 10 client.53930.objecter ms_handle_connect 0x7f709c063c80 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:09.943+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f70a404c2e0 con 0x7f70b40fc960 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.115+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f70a00051d0 con 0x7f70b40fc960 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.116+0000 7f70b17fa700 1 -- 172.21.15.107:0/3224362507 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v563) v1 ==== 81+0+46369 (unknown 1485764266 0 2292180758) 0x7f70a404a6f0 con 0x7f70b40fc960 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 10 librados: watch_flush enter 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 10 librados: watch_flush exit 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 20 client.53930.objecter shutdown clearing up homeless session... 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 10 client.53930.objecter successfully canceled tick 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 >> v1:172.21.15.107:6800/2634842009 conn(0x7f709c063c80 legacy=0x7f709c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 >> v1:172.21.15.107:6790/0 conn(0x7f70b40fc960 legacy=0x7f70b40f9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.118+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 shutdown_connections 2024-02-10T08:45:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.119+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 >> 172.21.15.107:0/3224362507 conn(0x7f70b406c090 msgr2=0x7f70b406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.119+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 shutdown_connections 2024-02-10T08:45:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.119+0000 7f70ba5aa700 1 -- 172.21.15.107:0/3224362507 wait complete. 2024-02-10T08:45:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:10.119+0000 7f70ba5aa700 1 librados: shutdown 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 563, 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:45:09.129987+0000", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:43:35.150516+0000", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:43:14.076553+0000", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5804032, 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:45:10.705 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 103, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 229, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:45:10.706 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.707 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "552", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.709 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.710 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.711 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:45:10.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.713 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.714 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.715 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.717 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:45:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:45:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.720 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.721 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.722 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.723 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:10.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.726 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.727 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.728 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.729 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:45:10.730 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.731 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.733 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.734 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.735 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.736 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:10.737 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.738 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.739 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.740 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.741 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.742 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.743 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.744 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.745 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.746 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 229, 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "LibRadosTierECPP_smithi107-112490-94", 2024-02-10T08:45:10.747 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:45:08.374442+0000", 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:10.748 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:10.749 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "563", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:10.750 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:10.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-LibRadosTierECPP_smithi107-112490-94", 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:10.752 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:45:10.753 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:45:10.754 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.755 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.758 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.759 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.760 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.762 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.9418182373046875, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.763 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:10.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:45:10.766 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.767 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.771 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 0, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.772 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 478, 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 480, 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.774 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:45:10.775 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: "exists" 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:45:10.777 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.778 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:45:10.779 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:45:10.781 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.782 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:45:10.783 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:45:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 554, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.787 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:10.789 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:45:10.790 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.791 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:20.195035+0000", 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.792 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:20.626620+0000", 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:12.339962+0000", 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:11.199980+0000", 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:23.143436+0000", 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 479 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:45:10.795 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:15.006595+0000", 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.796 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:11.644483+0000", 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:10.797 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:16.186878+0000", 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:45:10.799 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "6.1", 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.0", 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:45:10.801 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.802 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1d", 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.f", 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 3 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [ 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: 7, 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: 3 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.805 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:45:10.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-LibRadosTierECPP_smithi107-112490-94": { 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:45:10.807 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:45:10.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:45:10.809 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:45:10.812 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.813 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:45:10.816 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.817 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:45:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:10.823 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:45:10.824 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:11.573 INFO:tasks.thrashosds.thrasher:j is {'epoch': 563, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:45:09.129987+0000', 'last_up_change': '2024-02-10T08:43:35.150516+0000', 'last_in_change': '2024-02-10T08:43:14.076553+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 103, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 229, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '552', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 229, 'pool_name': 'LibRadosTierECPP_smithi107-112490-94', 'create_time': '2024-02-10T08:45:08.374442+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '563', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosTierECPP_smithi107-112490-94', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 0.9418182373046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 478, 'down_at': 480, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 554, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:44:20.195035+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:20.626620+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:12.339962+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:11.199980+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:23.143436+0000', 'dead_epoch': 479}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:15.006595+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:11.644483+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:16.186878+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '2.13', 'osds': [4, 2, 7]}, {'pgid': '4.1', 'osds': [4, 0, 7]}, {'pgid': '6.1', 'osds': [2, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.0', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '16.f', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_temp': [{'pgid': '2.13', 'osds': [2, 7, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosTierECPP_smithi107-112490-94': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:45:11.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:45:11.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:10.419489+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:45:11.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:10.422956+0000 osd.6 (osd.6) 239 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:45:11.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:11.089182+0000 mon.a (mon.0) 2904 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:11.095817+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:11.102152+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:11.110389+0000 mon.a (mon.0) 2906 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:10.419489+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:10.422956+0000 osd.6 (osd.6) 239 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:11.089182+0000 mon.a (mon.0) 2904 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:11.095817+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:11.102152+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:11.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:11.110389+0000 mon.a (mon.0) 2906 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:11.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:10.419489+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:45:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:10.422956+0000 osd.6 (osd.6) 239 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:45:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:11.089182+0000 mon.a (mon.0) 2904 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:11.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:11.095817+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e565: 8 total, 7 up, 7 in 2024-02-10T08:45:11.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:11.102152+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:11.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:11.110389+0000 mon.a (mon.0) 2906 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:12.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:10.598801+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:10.601754+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:11.455033+0000 osd.3 (osd.3) 259 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:11.455801+0000 osd.3 (osd.3) 260 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:11.500059+0000 mgr.y (mgr.14553) 494 : cluster [DBG] pgmap v850: 194 pgs: 1 active+clean+remapped, 20 creating+peering, 12 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:11.626756+0000 mgr.y (mgr.14553) 495 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:12.099520+0000 mon.a (mon.0) 2907 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:12.099597+0000 mon.a (mon.0) 2908 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:10.598801+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:10.601754+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:45:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:11.455033+0000 osd.3 (osd.3) 259 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:45:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:11.455801+0000 osd.3 (osd.3) 260 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:45:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:11.500059+0000 mgr.y (mgr.14553) 494 : cluster [DBG] pgmap v850: 194 pgs: 1 active+clean+remapped, 20 creating+peering, 12 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:11.626756+0000 mgr.y (mgr.14553) 495 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:12.099520+0000 mon.a (mon.0) 2907 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:12.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:12.099597+0000 mon.a (mon.0) 2908 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:10.598801+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:10.601754+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:11.455033+0000 osd.3 (osd.3) 259 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:11.455801+0000 osd.3 (osd.3) 260 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:11.500059+0000 mgr.y (mgr.14553) 494 : cluster [DBG] pgmap v850: 194 pgs: 1 active+clean+remapped, 20 creating+peering, 12 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:11.626756+0000 mgr.y (mgr.14553) 495 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:12.099520+0000 mon.a (mon.0) 2907 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:12.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:12.099597+0000 mon.a (mon.0) 2908 : cluster [DBG] osdmap e566: 8 total, 7 up, 7 in 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.064+0000 7f68f3b23700 1 -- 172.21.15.107:0/41722575 >> v1:172.21.15.107:6790/0 conn(0x7f68ec1007f0 legacy=0x7f68ec102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- 172.21.15.107:0/41722575 shutdown_connections 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- 172.21.15.107:0/41722575 >> 172.21.15.107:0/41722575 conn(0x7f68ec0f3640 msgr2=0x7f68ec0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- 172.21.15.107:0/41722575 shutdown_connections 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- 172.21.15.107:0/41722575 wait complete. 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 librados: starting msgr at 2024-02-10T08:45:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 librados: starting objecter 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 Processor -- start 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- start start 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 librados: setting wanted keys 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 librados: calling monclient init 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.065+0000 7f68f3b23700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f68ec117550 con 0x7f68ec0fda50 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f3b23700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f68ec118750 con 0x7f68ec1007f0 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f3b23700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f68ec119950 con 0x7f68ec104330 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f20c0700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f68ec104330 0x7f68ec115c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45706/0 (socket says 172.21.15.107:45706) 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f20c0700 1 -- 172.21.15.107:0/4134407553 learned_addr learned my addr 172.21.15.107:0/4134407553 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f18bf700 1 --1- 172.21.15.107:0/4134407553 >> v1:172.21.15.107:6789/0 conn(0x7f68ec0fda50 0x7f68ec0f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43778/0 (socket says 172.21.15.107:43778) 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter ms_handle_connect 0x7f68ec104330 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter ms_handle_connect 0x7f68ec0fda50 2024-02-10T08:45:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter ms_handle_connect 0x7f68ec1007f0 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327624945 0 0) 0x7f68ec119950 con 0x7f68ec104330 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d80033f0 con 0x7f68ec104330 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062990186 0 0) 0x7f68ec117550 con 0x7f68ec0fda50 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec119950 con 0x7f68ec0fda50 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052012054 0 0) 0x7f68ec118750 con 0x7f68ec1007f0 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec117550 con 0x7f68ec1007f0 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809973622 0 0) 0x7f68ec117550 con 0x7f68ec1007f0 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68ec118750 con 0x7f68ec1007f0 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2890998819 0 0) 0x7f68d80033f0 con 0x7f68ec104330 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68ec117550 con 0x7f68ec104330 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2630204412 0 0) 0x7f68ec119950 con 0x7f68ec0fda50 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68d80033f0 con 0x7f68ec0fda50 2024-02-10T08:45:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f68dc001ec0 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f68e40028b0 con 0x7f68ec104330 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f68e8002c10 con 0x7f68ec0fda50 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 582712428 0 0) 0x7f68ec118750 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 >> v1:172.21.15.107:6790/0 conn(0x7f68ec104330 legacy=0x7f68ec115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 >> v1:172.21.15.107:6789/0 conn(0x7f68ec0fda50 legacy=0x7f68ec0f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec11ab50 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec117740 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.066+0000 7f68f3b23700 10 client.48664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.067+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec117bb0 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.067+0000 7f68f3b23700 1 librados: init done 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.067+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f68dc001bb0 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.067+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f68dc0036a0 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.067+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f68d0004f00 con 0x7f68ec1007f0 2024-02-10T08:45:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f68dc0020c0 con 0x7f68ec1007f0 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(566..566 src has 420..566) v4 ==== 8600+0+0 (unknown 1632680895 0 0) 0x7f68dc076b00 con 0x7f68ec1007f0 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 osd_map(566..566 src has 420..566) v4 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 3 client.48664.objecter handle_osd_map got epochs [566,566] > 0 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 3 client.48664.objecter handle_osd_map decoding full epoch 566 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 20 client.48664.objecter dump_active .. 0 homeless 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.068+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v566) v1 ==== 54+0+16 (unknown 652654156 0 2420000192) 0x7f68dc076e60 con 0x7f68ec1007f0 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d8063cf0 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 10 client.48664.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 1 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 1 2024-02-10T08:45:13.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 0 1 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 4 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 10 client.48664.objecter _send_command 1 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d0004950 con 0x7f68d0004530 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d0004530 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d0004950 con 0x7f68d0004530 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 1 = system:0 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 4 2024-02-10T08:45:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.069+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 0 1 2024-02-10T08:45:13.079 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:13.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 10 client.48664.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 2 2024-02-10T08:45:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 2 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d00051a0 osd=0 3 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 0 2 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 4 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 10 client.48664.objecter _send_command 2 2024-02-10T08:45:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.086+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d0004d30 con 0x7f68d0004530 2024-02-10T08:45:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d0004d30 con 0x7f68d0004530 2024-02-10T08:45:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d00051a0 osd=0 4 2024-02-10T08:45:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 0 2 2024-02-10T08:45:13.089 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 10 client.48664.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 3 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 3 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 1 3 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 4 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 10 client.48664.objecter _send_command 3 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.087+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d000c000 con 0x7f68d0004150 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d0004150 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d000c000 con 0x7f68d0004150 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 3 = system:0 2024-02-10T08:45:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 4 2024-02-10T08:45:13.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.088+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 1 3 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 10 client.48664.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 4 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 4 2024-02-10T08:45:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0003fe0 osd=1 3 2024-02-10T08:45:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 1 4 2024-02-10T08:45:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 4 2024-02-10T08:45:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 10 client.48664.objecter _send_command 4 2024-02-10T08:45:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.109+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d00018b0 con 0x7f68d0004150 2024-02-10T08:45:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.117+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d00018b0 con 0x7f68d0004150 2024-02-10T08:45:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.117+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.117+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.117+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0003fe0 osd=1 4 2024-02-10T08:45:13.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.117+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 1 4 2024-02-10T08:45:13.118 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 10 client.48664.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 5 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 5 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 2 5 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 4 2024-02-10T08:45:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 10 client.48664.objecter _send_command 5 2024-02-10T08:45:13.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.118+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d000de90 con 0x7f68d000d2f0 2024-02-10T08:45:13.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.128+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d000d2f0 2024-02-10T08:45:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.139+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d000de90 con 0x7f68d000d2f0 2024-02-10T08:45:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.139+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.139+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 5 = system:0 2024-02-10T08:45:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.139+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 4 2024-02-10T08:45:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.139+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 2 5 2024-02-10T08:45:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:45:13.169 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:45:13.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 10 client.48664.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 6 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 6 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0003630 osd=2 3 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 2 6 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 4 2024-02-10T08:45:13.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 10 client.48664.objecter _send_command 6 2024-02-10T08:45:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.194+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d0003160 con 0x7f68d000d2f0 2024-02-10T08:45:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d0003160 con 0x7f68d000d2f0 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0003630 osd=2 4 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 2 6 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68f3b23700 10 client.48664.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.196+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 7 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 7 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 3 7 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 4 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 10 client.48664.objecter _send_command 7 2024-02-10T08:45:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.197+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d0019fb0 con 0x7f68d0016400 2024-02-10T08:45:13.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.201+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d0016400 2024-02-10T08:45:13.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.202+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d0019fb0 con 0x7f68d0016400 2024-02-10T08:45:13.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.202+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:13.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.202+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 7 = system:0 2024-02-10T08:45:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.202+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 4 2024-02-10T08:45:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.202+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 3 7 2024-02-10T08:45:13.217 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 10 client.48664.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 8 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 8 2024-02-10T08:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d0015d80 osd=3 3 2024-02-10T08:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 3 8 2024-02-10T08:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 4 2024-02-10T08:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 10 client.48664.objecter _send_command 8 2024-02-10T08:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.231+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d0019fb0 con 0x7f68d0016400 2024-02-10T08:45:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d0019fb0 con 0x7f68d0016400 2024-02-10T08:45:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d0015d80 osd=3 4 2024-02-10T08:45:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 3 8 2024-02-10T08:45:13.234 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.233+0000 7f68f3b23700 10 client.48664.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 9 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 10 client.48664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=567}) v3 -- 0x7f68d000e2f0 con 0x7f68ec1007f0 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68d000e630 con 0x7f68ec1007f0 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 10 ==== osd_map(567..567 src has 420..567) v4 ==== 999+0+0 (unknown 1498620834 0 0) 0x7f68dc002f60 con 0x7f68ec1007f0 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 osd_map(567..567 src has 420..567) v4 2024-02-10T08:45:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 3 client.48664.objecter handle_osd_map got epochs [567,567] > 566 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 3 client.48664.objecter handle_osd_map decoding incremental epoch 567 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 10 client.48664.objecter checking command 9 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 10 client.48664.objecter _check_command_map_dne tid 9 current 567 map_dne_bound 0 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 20 client.48664.objecter dump_active .. 1 homeless 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== mon.1 v1:172.21.15.195:6789/0 11 ==== mon_get_version_reply(handle=1 version=567) v2 ==== 24+0+0 (unknown 3940274189 0 0) 0x7f68dc078ea0 con 0x7f68ec1007f0 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e3fff700 10 client.48664.objecter _check_command_map_dne tid 9 current 567 map_dne_bound 567 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e3fff700 10 client.48664.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68e3fff700 15 client.48664.objecter _session_command_op_remove -1 9 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 10 client.48664.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 10 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 10 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 5 10 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 4 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 10 client.48664.objecter _send_command 10 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.234+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d001aca0 con 0x7f68d001a150 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.235+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d001a150 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.236+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d001aca0 con 0x7f68d001a150 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.236+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.236+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 10 = system:0 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.236+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 4 2024-02-10T08:45:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.236+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 5 10 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 10 client.48664.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 11 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 11 2024-02-10T08:45:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d000cb80 osd=5 3 2024-02-10T08:45:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 5 11 2024-02-10T08:45:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 4 2024-02-10T08:45:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 10 client.48664.objecter _send_command 11 2024-02-10T08:45:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.263+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d000c940 con 0x7f68d001a150 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.264+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d000c940 con 0x7f68d001a150 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.264+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.264+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.264+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d000cb80 osd=5 4 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.264+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 5 11 2024-02-10T08:45:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 10 client.48664.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 12 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 12 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 6 12 2024-02-10T08:45:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 4 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 10 client.48664.objecter _send_command 12 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d001b5c0 con 0x7f68d001f970 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.265+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d001f970 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.266+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d001b5c0 con 0x7f68d001f970 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.266+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.266+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 12 = system:0 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.266+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 4 2024-02-10T08:45:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.266+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 6 12 2024-02-10T08:45:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 10 client.48664.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 13 2024-02-10T08:45:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 13 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d000c940 osd=6 3 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 6 13 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 4 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 10 client.48664.objecter _send_command 13 2024-02-10T08:45:13.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.280+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d001b5c0 con 0x7f68d001f970 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.281+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d001b5c0 con 0x7f68d001f970 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.281+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.281+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.281+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d000c940 osd=6 4 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.281+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 6 13 2024-02-10T08:45:13.282 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 10 client.48664.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 14 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 14 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 7 14 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 4 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 10 client.48664.objecter _send_command 14 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68d0024510 con 0x7f68d0024fd0 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.282+0000 7f68e2ffd700 10 client.48664.objecter ms_handle_connect 0x7f68d0024fd0 2024-02-10T08:45:13.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.283+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f68d0024510 con 0x7f68d0024fd0 2024-02-10T08:45:13.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.283+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:13.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.283+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 14 = system:0 2024-02-10T08:45:13.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.283+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 4 2024-02-10T08:45:13.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.283+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 7 14 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 10 client.48664.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign -1 15 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 20 client.48664.objecter _get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_remove -1 15 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 20 client.48664.objecter get_session s=0x7f68d001b5c0 osd=7 3 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 15 client.48664.objecter _session_command_op_assign 7 15 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 4 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 10 client.48664.objecter _send_command 15 2024-02-10T08:45:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.297+0000 7f68f3b23700 1 -- 172.21.15.107:0/4134407553 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f68d00244d0 con 0x7f68d0024fd0 2024-02-10T08:45:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.298+0000 7f68e2ffd700 1 -- 172.21.15.107:0/4134407553 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f68d00244d0 con 0x7f68d0024fd0 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.298+0000 7f68e2ffd700 10 client.48664.objecter ms_dispatch 0x7f68ec001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.298+0000 7f68e2ffd700 10 client.48664.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.298+0000 7f68e2ffd700 20 client.48664.objecter put_session s=0x7f68d001b5c0 osd=7 4 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:13.298+0000 7f68e2ffd700 15 client.48664.objecter _session_command_op_remove 7 15 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.300 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:45:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:12.388731+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:12.463979+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:12.464718+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:13.069255+0000 mon.b (mon.1) 692 : audit [DBG] from='client.? v1:172.21.15.107:0/4134407553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.098936+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:13.110838+0000 mon.c (mon.2) 431 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:13.117100+0000 mon.a (mon.0) 2911 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:12.388731+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:12.463979+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:12.464718+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:45:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:13.069255+0000 mon.b (mon.1) 692 : audit [DBG] from='client.? v1:172.21.15.107:0/4134407553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.098936+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-10T08:45:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:13.110838+0000 mon.c (mon.2) 431 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:13.117100+0000 mon.a (mon.0) 2911 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:12.388731+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:12.463979+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:45:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:12.464718+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:45:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:13.069255+0000 mon.b (mon.1) 692 : audit [DBG] from='client.? v1:172.21.15.107:0/4134407553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.098936+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e567: 8 total, 7 up, 7 in 2024-02-10T08:45:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:13.110838+0000 mon.c (mon.2) 431 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:13.117100+0000 mon.a (mon.0) 2911 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:13.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:45:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:13.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:14.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:14.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:45:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.213 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:14.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:14.344 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.362 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:14.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:14.492 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.510 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:14.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.157004+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:45:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.165562+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:45:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.431097+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:45:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.431918+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:45:14.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:13.500781+0000 mgr.y (mgr.14553) 496 : cluster [DBG] pgmap v853: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:14.101059+0000 mon.a (mon.0) 2912 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:14.101153+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:14.158152+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:14.161743+0000 mon.a (mon.0) 2914 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.157004+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.165562+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.431097+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.431918+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:45:14.626 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:13.500781+0000 mgr.y (mgr.14553) 496 : cluster [DBG] pgmap v853: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:14.101059+0000 mon.a (mon.0) 2912 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:14.101153+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-10T08:45:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:14.158152+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.627 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:14.161743+0000 mon.a (mon.0) 2914 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:14.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:14.727 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:14.763 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:14.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.157004+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.165562+0000 osd.5 (osd.5) 238 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.431097+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.431918+0000 osd.7 (osd.7) 230 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:13.500781+0000 mgr.y (mgr.14553) 496 : cluster [DBG] pgmap v853: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:14.101059+0000 mon.a (mon.0) 2912 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:14.101153+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e568: 8 total, 7 up, 7 in 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:14.158152+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:14.161743+0000 mon.a (mon.0) 2914 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.812 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:14.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:14.960 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:14.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:15.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:15.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:15.114 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:15.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:15.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:15.242 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:15.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:15.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:15.400 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:14.447116+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:14.449990+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:14.480179+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:14.480991+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:15.106186+0000 mon.a (mon.0) 2915 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:15.106257+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:15.107947+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:15.112905+0000 mon.a (mon.0) 2917 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:14.447116+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:14.449990+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:14.480179+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:14.480991+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:15.106186+0000 mon.a (mon.0) 2915 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:15.106257+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:15.107947+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:15.112905+0000 mon.a (mon.0) 2917 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.640 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:15.658 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:15.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:15.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:14.447116+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:14.449990+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:14.480179+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:14.480991+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:15.106186+0000 mon.a (mon.0) 2915 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:15.106257+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e569: 8 total, 7 up, 7 in 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:15.107947+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:15.112905+0000 mon.a (mon.0) 2917 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:15] "GET /metrics HTTP/1.1" 200 186830 "" "Prometheus/2.33.4" 2024-02-10T08:45:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:16.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:16.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:16.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:45:16.668 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:15.501522+0000 mgr.y (mgr.14553) 497 : cluster [DBG] pgmap v856: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:16.668 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:16.091778+0000 mon.a (mon.0) 2918 : cluster [WRN] Health check update: Degraded data redundancy: 87/582 objects degraded (14.948%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:16.668 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:16.107947+0000 mon.a (mon.0) 2919 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:16.668 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:16.108020+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-10T08:45:16.668 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:16.111997+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.669 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:16.121222+0000 mon.a (mon.0) 2921 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.670 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:15.501522+0000 mgr.y (mgr.14553) 497 : cluster [DBG] pgmap v856: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:16.091778+0000 mon.a (mon.0) 2918 : cluster [WRN] Health check update: Degraded data redundancy: 87/582 objects degraded (14.948%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:16.107947+0000 mon.a (mon.0) 2919 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:16.108020+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:16.111997+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:16.121222+0000 mon.a (mon.0) 2921 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:15.501522+0000 mgr.y (mgr.14553) 497 : cluster [DBG] pgmap v856: 226 pgs: 1 active+clean+remapped, 20 creating+peering, 44 unknown, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:16.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:16.091778+0000 mon.a (mon.0) 2918 : cluster [WRN] Health check update: Degraded data redundancy: 87/582 objects degraded (14.948%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:16.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:16.107947+0000 mon.a (mon.0) 2919 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-97", "overlaypool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:16.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:16.108020+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e570: 8 total, 7 up, 7 in 2024-02-10T08:45:16.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:16.111997+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:16.121222+0000 mon.a (mon.0) 2921 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:16.689 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:16.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:16.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:16.820 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:17.400 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.35222.0:6 50.3s0 50:c1c2e34e:test-rados-api-smithi107-112459-13::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:45:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 510.45578347700001, 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049033968999999997, 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.35222", 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/4111016630", 2024-02-10T08:45:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890935+0000", 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890932+0000", 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0301e-05 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890943+0000", 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2490999999999999e-05 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890955+0000", 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.926e-06 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.890963+0000", 2024-02-10T08:45:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025738898999999999 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.916702+0000", 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6725000000000007e-05 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.916799+0000", 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023170130000000001 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:46.939969+0000", 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049033968999999997 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.26956.0:2 65.15 65:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:36:54.045025+0000", 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 503.30169346700001, 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044327159999999997, 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.26956", 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/110610435", 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045025+0000", 2024-02-10T08:45:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045025+0000", 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045023+0000", 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5120000000000002e-06 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045030+0000", 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5089999999999999e-06 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045040+0000", 2024-02-10T08:45:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4890000000000003e-06 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.045045+0000", 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085118680000000006 2024-02-10T08:45:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.053557+0000", 2024-02-10T08:45:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0149999999999992e-06 2024-02-10T08:45:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.053566+0000", 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011474371000000001 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.065041+0000", 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3467000000000007e-05 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.065114+0000", 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105888 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,7", 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.065220+0000", 2024-02-10T08:45:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098607590000000002 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.075081+0000", 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044997939999999997 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.079581+0000", 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096745300000000006 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.089255+0000", 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5242999999999999e-05 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.089310+0000", 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1708000000000001e-05 2024-02-10T08:45:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:36:54.089352+0000", 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.044327159999999997 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:259 83.16 83:6a183d1d:test-rados-api-smithi107-112490-12::bar:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:13.081740+0000", 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 484.26497859300002, 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034516705000000002, 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 259 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081740+0000", 2024-02-10T08:45:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081740+0000", 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999952 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081735+0000", 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7612e-05 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081752+0000", 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5086999999999999e-05 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081778+0000", 2024-02-10T08:45:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0325e-05 2024-02-10T08:45:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081788+0000", 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9766999999999999e-05 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.081848+0000", 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022230500000000001 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.082070+0000", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018323259000000001 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.100393+0000", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019259100000000001 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.100586+0000", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057602560000000001 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.106346+0000", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8505999999999997e-05 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.106425+0000", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094946570000000001 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.115919+0000", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5055000000000004e-05 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.116014+0000", 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024228800000000001 2024-02-10T08:45:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.116257+0000", 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034516705000000002 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:260 83.a 83:556e16fc:test-rados-api-smithi107-112490-12::baz:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:13.125751+0000", 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 484.22096722499998, 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.039807161000000001, 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 260 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125751+0000", 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125751+0000", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125748+0000", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.216e-05 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125760+0000", 2024-02-10T08:45:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0672000000000001e-05 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125781+0000", 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.85e-06 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125789+0000", 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0046999999999997e-05 2024-02-10T08:45:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.125840+0000", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000252048 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.126092+0000", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036131561999999999 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.162223+0000", 2024-02-10T08:45:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018829 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.162411+0000", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010774300000000001 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.163489+0000", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7046000000000007e-05 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.163576+0000", 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00081247399999999999 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.164388+0000", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5873999999999999e-05 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.164464+0000", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010940780000000001 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:13.165558+0000", 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.039807161000000001 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:505 60.0 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head [writefull 0~11] snapc 4=[4] ondisk+write+known_if_redirected e147)", 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:20.921247+0000", 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 476.42547162099999, 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022401913999999998, 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 505 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921247+0000", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921247+0000", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921245+0000", 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1729999999999995e-06 2024-02-10T08:45:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921253+0000", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8769999999999993e-06 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921261+0000", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1900000000000003e-06 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921266+0000", 2024-02-10T08:45:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5556999999999999e-05 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921301+0000", 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0911e-05 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921362+0000", 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016082 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.921523+0000", 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042631750000000001 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.925786+0000", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011711638 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.937498+0000", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060541359999999999 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.943552+0000", 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3063999999999999e-05 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.943605+0000", 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3603999999999998e-05 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:20.943649+0000", 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022401913999999998 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:606 60.0 60:033d636e:test-rados-api-smithi107-112490-14::foo0:head [writefull 0~11] snapc 5=[5,4] ondisk+write+known_if_redirected e148)", 2024-02-10T08:45:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:21.927459+0000", 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 475.41925964699999, 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02098904, 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 606 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.927459+0000", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.927459+0000", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.927456+0000", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6500000000000008e-06 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.927466+0000", 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096036400000000001 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.928426+0000", 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7389999999999994e-06 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.928436+0000", 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041297740000000001 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.932566+0000", 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4079000000000005e-05 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.932660+0000", 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017321299999999999 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:45:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.934392+0000", 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062506530000000001 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.940643+0000", 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00075031000000000002 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.941393+0000", 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069399809999999996 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948333+0000", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0457000000000007e-05 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948403+0000", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4313000000000001e-05 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:21.948448+0000", 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02098904 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.27252.0:2 112.16 112:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:41.929816+0000", 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 455.416902064, 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030530713000000001, 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.27252", 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1908017200", 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929816+0000", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929816+0000", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929814+0000", 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3700000000000001e-06 2024-02-10T08:45:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929823+0000", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1667e-05 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929835+0000", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1489999999999994e-06 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929843+0000", 2024-02-10T08:45:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6889000000000003e-05 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929910+0000", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.973e-06 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.929917+0000", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018539170000000001 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.931771+0000", 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5820000000000004e-06 2024-02-10T08:45:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.931778+0000", 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019525819999999999 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.951304+0000", 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126518 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.951431+0000", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016206200000000001 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.951593+0000", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032585420000000001 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.954851+0000", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014921300000000001 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.955001+0000", 2024-02-10T08:45:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052409789999999998 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.960242+0000", 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1029999999999997e-05 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.960303+0000", 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4401999999999999e-05 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:41.960347+0000", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030530713000000001 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52775.0:2 115.6 115:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:45.977537+0000", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 451.369181443, 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045690254999999999, 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52775", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1963374674", 2024-02-10T08:45:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977537+0000", 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977537+0000", 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977534+0000", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6829999999999997e-06 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977543+0000", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0104e-05 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977553+0000", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8090000000000004e-06 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977560+0000", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2504999999999999e-05 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977623+0000", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8490000000000006e-06 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977630+0000", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041174871000000002 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.018805+0000", 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110034 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.018915+0000", 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000148467 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.019064+0000", 2024-02-10T08:45:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0032713659999999999 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.022335+0000", 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0435000000000001e-05 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.022386+0000", 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000736475 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023122+0000", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1143000000000001e-05 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023173+0000", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4078e-05 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023227+0000", 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045690254999999999 2024-02-10T08:45:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.52775.0:3 115.6 115:602f83fe:::foo:head [call] snapc 0=[] ondisk+read+known_if_redirected e172)", 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:37:45.977579+0000", 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 451.36913931300001, 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045839294000000003, 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.52775", 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/1963374674", 2024-02-10T08:45:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977579+0000", 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977579+0000", 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977578+0000", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.702e-06 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977583+0000", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3059999999999997e-06 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977588+0000", 2024-02-10T08:45:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6380000000000002e-06 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977592+0000", 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1321999999999993e-05 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977653+0000", 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8210000000000001e-06 2024-02-10T08:45:17.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:45.977658+0000", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041651122999999998 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.019309+0000", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9127000000000004e-05 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.019378+0000", 2024-02-10T08:45:17.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00390391 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023282+0000", 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9048999999999996e-05 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023351+0000", 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7156000000000006e-05 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:37:46.023418+0000", 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045839294000000003 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.53066.0:1 157.5s0 157:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e231)", 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:38:44.962510+0000", 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 392.38420834900001, 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.068891204, 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.53066", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/2713829649", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962510+0000", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962510+0000", 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962508+0000", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3280000000000002e-06 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962515+0000", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.016e-05 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962535+0000", 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4530000000000002e-06 2024-02-10T08:45:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962543+0000", 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8783999999999999e-05 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962592+0000", 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.323e-06 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:44.962594+0000", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.068703527 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:46.031298+0000", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010327699999999999 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:38:46.031401+0000", 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.068891204 2024-02-10T08:45:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:6965 219.10 219:08000000:%01:::head [pgnls start_epoch 485] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e485)", 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:51.112036+0000", 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.234682383000006, 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041076909000000002, 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6965 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112036+0000", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112036+0000", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112033+0000", 2024-02-10T08:45:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4127e-05 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112048+0000", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.359e-05 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112061+0000", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5699999999999998e-06 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112068+0000", 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.465e-05 2024-02-10T08:45:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112122+0000", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5939999999999996e-06 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.112130+0000", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040883656999999997 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.153014+0000", 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6080000000000001e-05 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.153040+0000", 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3238000000000006e-05 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:51.153113+0000", 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041076909000000002 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.7.52:64 60.c 60:35842c8e:test-rados-api-smithi107-112490-78::060e72eaff4fb3562782e1de4fe785e9648c08af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:59.149202+0000", 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 78.197516390999994, 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046410226999999998, 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149202+0000", 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149202+0000", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149199+0000", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1347e-05 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149210+0000", 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8369999999999999e-05 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149229+0000", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4800000000000001e-06 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149237+0000", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6839000000000002e-05 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149294+0000", 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9550000000000003e-06 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.149301+0000", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016236428000000001 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.165537+0000", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000133986 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.165671+0000", 2024-02-10T08:45:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020721300000000001 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.165879+0000", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079480690000000003 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173827+0000", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010945005000000001 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.184772+0000", 2024-02-10T08:45:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010748399 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195520+0000", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2686999999999999e-05 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195573+0000", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9422000000000003e-05 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195612+0000", 2024-02-10T08:45:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046410226999999998 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.7.52:62 60.1f e493/135 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head v 493'43, mlcod=439'41)", 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:59.173675+0000", 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 78.173043492000005, 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022038516000000001, 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173675+0000", 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173675+0000", 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173673+0000", 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7899999999999994e-06 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.173683+0000", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045028860000000002 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.178186+0000", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.178192+0000", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2432000000000002e-05 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.178245+0000", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9961000000000001e-05 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.178265+0000", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017417585999999999 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195682+0000", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1072999999999999e-05 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:59.195713+0000", 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022038516000000001 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.7.52:92 60.12 e495/480 60:49b62e28:test-rados-api-smithi107-112490-78::bfb5db8bf32239a5b4437a6edddde56f73a76483:head v 495'104, mlcod=214'102)", 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:01.114697+0000", 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 76.232021304, 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021169229000000001, 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114697+0000", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114697+0000", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114697+0000", 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6370000000000004e-06 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114702+0000", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3011e-05 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114715+0000", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5769999999999999e-06 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114722+0000", 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8460000000000001e-05 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114760+0000", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3408999999999999e-05 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.114784+0000", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021048076999999998 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135832+0000", 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4554000000000002e-05 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135866+0000", 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021169229000000001 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7077 60.6 60:66fd1095:test-rados-api-smithi107-112490-78::bar-chunk:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:01.135123+0000", 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 76.211595658999997, 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022797951, 2024-02-10T08:45:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7077 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135123+0000", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135123+0000", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135121+0000", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0639999999999994e-06 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135129+0000", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3995999999999999e-05 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135143+0000", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4840000000000001e-06 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135149+0000", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6314999999999999e-05 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135205+0000", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4559000000000001e-05 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135290+0000", 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015332900000000001 2024-02-10T08:45:17.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.135443+0000", 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00065462599999999995 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.136098+0000", 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051797880000000003 2024-02-10T08:45:17.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.141278+0000", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016588446 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.157866+0000", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0720999999999999e-05 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.157897+0000", 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3855000000000001e-05 2024-02-10T08:45:17.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.157921+0000", 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022797951 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.7.52:93 60.6 60:66fd1095:test-rados-api-smithi107-112490-78::bar-chunk:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e495)", 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:01.158654+0000", 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 76.188063987000007, 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022219985000000001, 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158654+0000", 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158654+0000", 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158653+0000", 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3950000000000001e-06 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158661+0000", 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6756999999999999e-05 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158678+0000", 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8820000000000003e-06 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158685+0000", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2688999999999998e-05 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158717+0000", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6214000000000001e-05 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158774+0000", 2024-02-10T08:45:17.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119449 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.158893+0000", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028045799999999998 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.159173+0000", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024572140000000001 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.161631+0000", 2024-02-10T08:45:17.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019180296999999999 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.180811+0000", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4938000000000003e-05 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.180846+0000", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8504e-05 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:01.180874+0000", 2024-02-10T08:45:17.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022219985000000001 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(16.1f RELEASE e533)", 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:39.360113+0000", 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 37.986605318999999, 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021614417, 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360113+0000", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360113+0000", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360112+0000", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8729999999999998e-06 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360117+0000", 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0848e-05 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360128+0000", 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.360133+0000", 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021562998999999999 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.381696+0000", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2829e-05 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.381709+0000", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8723e-05 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:39.381728+0000", 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021614417 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.515 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7773 227.5 227:a4ab4d39:test-rados-api-smithi107-112490-91::foo:head [call] snapc 0=[] ondisk+write+known_if_redirected+returnvec e550)", 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:56.119726+0000", 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.226992065000001, 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022278279000000002, 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:45:17.516 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7773 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119726+0000", 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119726+0000", 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119724+0000", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0559e-05 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119734+0000", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5004000000000001e-05 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119749+0000", 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7370000000000001e-06 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.119759+0000", 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001246715 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.121006+0000", 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.776e-06 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.121013+0000", 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010541352 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.131554+0000", 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010447764999999999 2024-02-10T08:45:17.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.142002+0000", 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9170000000000002e-06 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:56.142005+0000", 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022278279000000002 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(osd.6.0:625 231.6 e571/567 231:602f83fe:::foo:head v 571'1, mlcod=0'0)", 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:45:17.137696+0000", 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.20902272599999999, 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028304268, 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137696+0000", 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137696+0000", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137695+0000", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4739999999999999e-05 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137710+0000", 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1075e-05 2024-02-10T08:45:17.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137721+0000", 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0380000000000002e-06 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.137728+0000", 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059791339999999997 2024-02-10T08:45:17.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143707+0000", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8773000000000001e-05 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143726+0000", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022243659999999998 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.165970+0000", 2024-02-10T08:45:17.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0307e-05 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.166000+0000", 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028304268 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.53936.0:3 231.6 e571/567 231:602f83fe:::foo:head v 571'2, mlcod=0'0)", 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:45:17.143690+0000", 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.20302821400000001, 2024-02-10T08:45:17.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043036024999999999, 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143690+0000", 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143690+0000", 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:45:17.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143689+0000", 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8000000000000001e-06 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143696+0000", 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8869999999999992e-06 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:45:17.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143704+0000", 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1959999999999999e-06 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.143709+0000", 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022017668000000001 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.165727+0000", 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7065999999999999e-05 2024-02-10T08:45:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.165744+0000", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020954747999999999 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.186698+0000", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.773e-05 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:17.186726+0000", 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043036024999999999 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.530 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.531 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.531 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:17.531 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:17.531 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:45:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:15.553747+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:45:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:15.556632+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:45:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:16.052805+0000 osd.5 (osd.5) 239 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:45:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:16.055980+0000 osd.5 (osd.5) 240 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:17.104422+0000 mon.a (mon.0) 2922 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:17.108498+0000 mon.a (mon.0) 2923 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:17.108567+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:17.234174+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:17.234517+0000 mon.a (mon.0) 2925 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:15.553747+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:15.556632+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:16.052805+0000 osd.5 (osd.5) 239 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:16.055980+0000 osd.5 (osd.5) 240 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:45:17.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:17.104422+0000 mon.a (mon.0) 2922 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:17.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:17.108498+0000 mon.a (mon.0) 2923 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:17.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:17.108567+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-10T08:45:17.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:17.234174+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:17.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:17.234517+0000 mon.a (mon.0) 2925 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:15.553747+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:15.556632+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:16.052805+0000 osd.5 (osd.5) 239 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:16.055980+0000 osd.5 (osd.5) 240 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:17.104422+0000 mon.a (mon.0) 2922 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:17.108498+0000 mon.a (mon.0) 2923 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-97-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:17.108567+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e571: 8 total, 7 up, 7 in 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:17.234174+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:17.234517+0000 mon.a (mon.0) 2925 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]: dispatch 2024-02-10T08:45:18.414 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:17.502180+0000 mgr.y (mgr.14553) 498 : cluster [DBG] pgmap v859: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:17.813897+0000 osd.6 (osd.6) 244 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:17.814098+0000 osd.6 (osd.6) 245 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:18.110749+0000 mon.a (mon.0) 2926 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]': finished 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:18.110811+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:18.117374+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:18.117686+0000 mon.a (mon.0) 2928 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:18.136906+0000 osd.7 (osd.7) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:18.137153+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:17.502180+0000 mgr.y (mgr.14553) 498 : cluster [DBG] pgmap v859: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:17.813897+0000 osd.6 (osd.6) 244 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:17.814098+0000 osd.6 (osd.6) 245 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:18.110749+0000 mon.a (mon.0) 2926 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]': finished 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:18.110811+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-10T08:45:18.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:18.117374+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:18.117686+0000 mon.a (mon.0) 2928 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:18.136906+0000 osd.7 (osd.7) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:18.137153+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:17.502180+0000 mgr.y (mgr.14553) 498 : cluster [DBG] pgmap v859: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:17.813897+0000 osd.6 (osd.6) 244 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:17.814098+0000 osd.6 (osd.6) 245 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:18.110749+0000 mon.a (mon.0) 2926 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-97"}]': finished 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:18.110811+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e572: 8 total, 7 up, 7 in 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:18.117374+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.107:0/130121399' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:18.117686+0000 mon.a (mon.0) 2928 : audit [INF] from='client.53936 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]: dispatch 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:18.136906+0000 osd.7 (osd.7) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:18.137153+0000 osd.7 (osd.7) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:19.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:45:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:19.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:19.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:19.540 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:19.558 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:19.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:19.107547+0000 mon.a (mon.0) 2929 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:19.111469+0000 mon.a (mon.0) 2930 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:19.111515+0000 mon.a (mon.0) 2931 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:19.107547+0000 mon.a (mon.0) 2929 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:19.111469+0000 mon.a (mon.0) 2930 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:19.111515+0000 mon.a (mon.0) 2931 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-10T08:45:19.690 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:19.107547+0000 mon.a (mon.0) 2929 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:19.111469+0000 mon.a (mon.0) 2930 : audit [INF] from='client.53936 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-97", "tierpool": "test-rados-api-smithi107-112490-97-cache"}]': finished 2024-02-10T08:45:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:19.111515+0000 mon.a (mon.0) 2931 : cluster [DBG] osdmap e573: 8 total, 7 up, 7 in 2024-02-10T08:45:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.107+0000 7f6503582700 1 -- 172.21.15.107:0/1318008339 >> v1:172.21.15.107:6790/0 conn(0x7f64fc104350 legacy=0x7f64fc106810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.107+0000 7f6503582700 1 -- 172.21.15.107:0/1318008339 shutdown_connections 2024-02-10T08:45:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.107+0000 7f6503582700 1 -- 172.21.15.107:0/1318008339 >> 172.21.15.107:0/1318008339 conn(0x7f64fc06c090 msgr2=0x7f64fc06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.107+0000 7f6503582700 1 -- 172.21.15.107:0/1318008339 shutdown_connections 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.107+0000 7f6503582700 1 -- 172.21.15.107:0/1318008339 wait complete. 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 librados: starting msgr at 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 librados: starting objecter 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 Processor -- start 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 -- start start 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 librados: setting wanted keys 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 librados: calling monclient init 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64fc0fa9e0 con 0x7f64fc104350 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64fc118810 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6503582700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64fc1199b0 con 0x7f64fc100840 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6500b1d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f64fc100840 0x7f64fc1129b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51688/0 (socket says 172.21.15.107:51688) 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f6500b1d700 1 -- 172.21.15.107:0/3914790522 learned_addr learned my addr 172.21.15.107:0/3914790522 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter ms_handle_connect 0x7f64fc100840 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter ms_handle_connect 0x7f64fc104350 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter ms_handle_connect 0x7f64fc0fd9f0 2024-02-10T08:45:20.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.108+0000 7f64f27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:20.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.113+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015494032 0 0) 0x7f64fc118810 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.113+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8003030 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831069669 0 0) 0x7f64e8003030 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64fc118810 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f64ec001880 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801706590 0 0) 0x7f64fc1199b0 con 0x7f64fc100840 2024-02-10T08:45:20.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8003030 con 0x7f64fc100840 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2366361171 0 0) 0x7f64fc118810 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 >> v1:172.21.15.107:6790/0 conn(0x7f64fc100840 legacy=0x7f64fc1129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 >> v1:172.21.15.107:6789/0 conn(0x7f64fc104350 legacy=0x7f64fc1160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc11ab50 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc117860 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f6503582700 10 client.48670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc117d10 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f6503582700 1 librados: init done 2024-02-10T08:45:20.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f64ec001f40 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.114+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f64ec003350 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.115+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0004f50 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f64ec0026b0 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(574..574 src has 420..574) v4 ==== 8600+0+0 (unknown 1999549650 0 0) 0x7f64ec076680 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 10 client.48670.objecter ms_dispatch 0x7f64fc001550 osd_map(574..574 src has 420..574) v4 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 3 client.48670.objecter handle_osd_map got epochs [574,574] > 0 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 3 client.48670.objecter handle_osd_map decoding full epoch 574 2024-02-10T08:45:20.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.117+0000 7f64f27fc700 20 client.48670.objecter dump_active .. 0 homeless 2024-02-10T08:45:20.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.120+0000 7f64f27fc700 10 client.48670.objecter ms_handle_connect 0x7f64e8063b40 2024-02-10T08:45:20.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.126+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f64ec049860 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.241 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:20.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:20.292 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:20.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.389+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f64e0005200 con 0x7f64fc0fd9f0 2024-02-10T08:45:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:18.555517+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:45:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:18.558984+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:45:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:19.464652+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.b scrub starts 2024-02-10T08:45:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:19.465630+0000 osd.3 (osd.3) 262 : cluster [DBG] 6.b scrub ok 2024-02-10T08:45:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:19.502818+0000 mgr.y (mgr.14553) 499 : cluster [DBG] pgmap v862: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:20.114947+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-10T08:45:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:20.392108+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.107:0/3914790522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:20.392634+0000 mon.a (mon.0) 2933 : audit [INF] from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:18.555517+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:18.558984+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:19.464652+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.b scrub starts 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:19.465630+0000 osd.3 (osd.3) 262 : cluster [DBG] 6.b scrub ok 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:19.502818+0000 mgr.y (mgr.14553) 499 : cluster [DBG] pgmap v862: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:20.114947+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:20.392108+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.107:0/3914790522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:20.392634+0000 mon.a (mon.0) 2933 : audit [INF] from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:18.555517+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:18.558984+0000 osd.1 (osd.1) 275 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:19.464652+0000 osd.3 (osd.3) 261 : cluster [DBG] 6.b scrub starts 2024-02-10T08:45:20.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:19.465630+0000 osd.3 (osd.3) 262 : cluster [DBG] 6.b scrub ok 2024-02-10T08:45:20.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:19.502818+0000 mgr.y (mgr.14553) 499 : cluster [DBG] pgmap v862: 226 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 75 active+undersized, 129 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:20.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:20.114947+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e574: 8 total, 7 up, 7 in 2024-02-10T08:45:20.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:20.392108+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.107:0/3914790522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:20.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:20.392634+0000 mon.a (mon.0) 2933 : audit [INF] from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.104+0000 7f64f27fc700 1 -- 172.21.15.107:0/3914790522 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v575) v1 ==== 101+0+0 (unknown 441818388 0 0) 0x7f64ec047c70 con 0x7f64fc0fd9f0 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 10 librados: watch_flush enter 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 10 librados: watch_flush exit 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 20 client.48670.objecter shutdown clearing up homeless session... 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 10 client.48670.objecter successfully canceled tick 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 >> v1:172.21.15.107:6800/2634842009 conn(0x7f64e8063b40 legacy=0x7f64e8066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.107+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 >> v1:172.21.15.195:6789/0 conn(0x7f64fc0fd9f0 legacy=0x7f64fc0f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.110+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 shutdown_connections 2024-02-10T08:45:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.110+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 >> 172.21.15.107:0/3914790522 conn(0x7f64fc06c090 msgr2=0x7f64fc1067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:21.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.118+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 shutdown_connections 2024-02-10T08:45:21.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.127+0000 7f6503582700 1 -- 172.21.15.107:0/3914790522 wait complete. 2024-02-10T08:45:21.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.127+0000 7f6503582700 1 librados: shutdown 2024-02-10T08:45:21.164 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:45:21.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 -- 172.21.15.107:0/1564469830 >> v1:172.21.15.107:6789/0 conn(0x7ff11c103250 legacy=0x7ff11c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 -- 172.21.15.107:0/1564469830 shutdown_connections 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 -- 172.21.15.107:0/1564469830 >> 172.21.15.107:0/1564469830 conn(0x7ff11c0f2510 msgr2=0x7ff11c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 -- 172.21.15.107:0/1564469830 shutdown_connections 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 -- 172.21.15.107:0/1564469830 wait complete. 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 librados: starting msgr at 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 librados: starting objecter 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.968+0000 7ff124480700 1 Processor -- start 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 -- start start 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 librados: setting wanted keys 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 librados: calling monclient init 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff11c119590 con 0x7ff11c103250 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff11c11a770 con 0x7ff11c0fc910 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff124480700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff11c11b970 con 0x7ff11c0ff710 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff122a1d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff11c103250 0x7ff11c117cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40324/0 (socket says 172.21.15.107:40324) 2024-02-10T08:45:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff121a1b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff11c0ff710 0x7ff11c0f8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51712/0 (socket says 172.21.15.107:51712) 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff122a1d700 1 -- 172.21.15.107:0/1443799658 learned_addr learned my addr 172.21.15.107:0/1443799658 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter ms_handle_connect 0x7ff11c103250 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter ms_handle_connect 0x7ff11c0ff710 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter ms_handle_connect 0x7ff11c0fc910 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711035011 0 0) 0x7ff11c119590 con 0x7ff11c103250 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff108003030 con 0x7ff11c103250 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202143917 0 0) 0x7ff11c11b970 con 0x7ff11c0ff710 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c119590 con 0x7ff11c0ff710 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205864824 0 0) 0x7ff11c11a770 con 0x7ff11c0fc910 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c11b970 con 0x7ff11c0fc910 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2594377606 0 0) 0x7ff108003030 con 0x7ff11c103250 2024-02-10T08:45:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff11c11a770 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff118001950 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3867708049 0 0) 0x7ff11c119590 con 0x7ff11c0ff710 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff108003030 con 0x7ff11c0ff710 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff10c001920 con 0x7ff11c0ff710 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868067249 0 0) 0x7ff11c11b970 con 0x7ff11c0fc910 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff11c119590 con 0x7ff11c0fc910 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.969+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff1140017c0 con 0x7ff11c0fc910 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3082145591 0 0) 0x7ff11c11a770 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 >> v1:172.21.15.107:6790/0 conn(0x7ff11c0ff710 legacy=0x7ff11c0f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 >> v1:172.21.15.195:6789/0 conn(0x7ff11c0fc910 legacy=0x7ff11c0f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c11cb70 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c11bb60 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff124480700 10 client.28383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c11bfd0 con 0x7ff11c103250 2024-02-10T08:45:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff124480700 1 librados: init done 2024-02-10T08:45:21.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff118001480 con 0x7ff11c103250 2024-02-10T08:45:21.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.970+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff118003b70 con 0x7ff11c103250 2024-02-10T08:45:21.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff100004f20 con 0x7ff11c103250 2024-02-10T08:45:21.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff118001bc0 con 0x7ff11c103250 2024-02-10T08:45:21.656 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(574..574 src has 420..574) v4 ==== 8600+0+0 (unknown 1999549650 0 0) 0x7ff118079530 con 0x7ff11c103250 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 10 client.28383.objecter ms_dispatch 0x7ff11c001550 osd_map(574..574 src has 420..574) v4 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 3 client.28383.objecter handle_osd_map got epochs [574,574] > 0 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 3 client.28383.objecter handle_osd_map decoding full epoch 574 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.971+0000 7ff1137fe700 20 client.28383.objecter dump_active .. 0 homeless 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.972+0000 7ff1137fe700 10 client.28383.objecter ms_handle_connect 0x7ff108063df0 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:20.976+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff11804c790 con 0x7ff11c103250 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.191+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff100000b90 con 0x7ff108063df0 2024-02-10T08:45:21.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.203+0000 7ff1137fe700 1 -- 172.21.15.107:0/1443799658 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426554 (unknown 2965378022 0 3847768694) 0x7ff100000b90 con 0x7ff108063df0 2024-02-10T08:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.653+0000 7ff124480700 10 librados: watch_flush enter 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.653+0000 7ff124480700 10 librados: watch_flush exit 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.653+0000 7ff124480700 20 client.28383.objecter shutdown clearing up homeless session... 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.653+0000 7ff124480700 10 client.28383.objecter successfully canceled tick 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff108063df0 legacy=0x7ff1080662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 >> v1:172.21.15.107:6789/0 conn(0x7ff11c103250 legacy=0x7ff11c117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 shutdown_connections 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 >> 172.21.15.107:0/1443799658 conn(0x7ff11c0f2510 msgr2=0x7ff11c0f5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 shutdown_connections 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 -- 172.21.15.107:0/1443799658 wait complete. 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:21.654+0000 7ff124480700 1 librados: shutdown 2024-02-10T08:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:45:21.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:20.442284+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:20.443073+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:20.610543+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:20.614010+0000 osd.1 (osd.1) 277 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:21.093963+0000 mon.a (mon.0) 2934 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.100169+0000 mon.a (mon.0) 2935 : audit [INF] from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:21.100232+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.143699+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.149067+0000 mon.a (mon.0) 2937 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.149651+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.153497+0000 mon.a (mon.0) 2938 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.154009+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.158629+0000 mon.a (mon.0) 2939 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:20.442284+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:20.443073+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:20.610543+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:20.614010+0000 osd.1 (osd.1) 277 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:21.093963+0000 mon.a (mon.0) 2934 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.100169+0000 mon.a (mon.0) 2935 : audit [INF] from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:21.100232+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.143699+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.149067+0000 mon.a (mon.0) 2937 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.149651+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.153497+0000 mon.a (mon.0) 2938 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.154009+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.158629+0000 mon.a (mon.0) 2939 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:20.442284+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:20.443073+0000 osd.7 (osd.7) 234 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:20.610543+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:20.614010+0000 osd.1 (osd.1) 277 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:21.093963+0000 mon.a (mon.0) 2934 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.100169+0000 mon.a (mon.0) 2935 : audit [INF] from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:21.100232+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e575: 8 total, 7 up, 7 in 2024-02-10T08:45:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.143699+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.149067+0000 mon.a (mon.0) 2937 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.149651+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.153497+0000 mon.a (mon.0) 2938 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:21.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.154009+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.158629+0000 mon.a (mon.0) 2939 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:21.826 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:21.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:21.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:21.961 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:21.979 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:21.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:22.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:22.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:22.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:22.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:22.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:22.550 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:22.569 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:22.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:22.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:21.037428+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:45:22.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:21.040982+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:45:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.192870+0000 mgr.y (mgr.14553) 500 : audit [DBG] from='client.28383 v1:172.21.15.107:0/1443799658' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:21.503629+0000 mgr.y (mgr.14553) 501 : cluster [DBG] pgmap v865: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:21.637511+0000 mgr.y (mgr.14553) 502 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:22.111613+0000 mon.a (mon.0) 2940 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:22.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.111713+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:22.112623+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:22.115884+0000 mon.a (mon.0) 2942 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:21.037428+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:21.040982+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.192870+0000 mgr.y (mgr.14553) 500 : audit [DBG] from='client.28383 v1:172.21.15.107:0/1443799658' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:21.503629+0000 mgr.y (mgr.14553) 501 : cluster [DBG] pgmap v865: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:21.637511+0000 mgr.y (mgr.14553) 502 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:22.111613+0000 mon.a (mon.0) 2940 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.111713+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-10T08:45:22.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:22.112623+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:22.115884+0000 mon.a (mon.0) 2942 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.700 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:21.037428+0000 osd.5 (osd.5) 241 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:21.040982+0000 osd.5 (osd.5) 242 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.192870+0000 mgr.y (mgr.14553) 500 : audit [DBG] from='client.28383 v1:172.21.15.107:0/1443799658' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:21.503629+0000 mgr.y (mgr.14553) 501 : cluster [DBG] pgmap v865: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:21.637511+0000 mgr.y (mgr.14553) 502 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:22.111613+0000 mon.a (mon.0) 2940 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.111713+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 7 up, 7 in 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:22.112623+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:22.115884+0000 mon.a (mon.0) 2942 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:22.948 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:22.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:23.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:23.080 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:23.098 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:23.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:23.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:23.227 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:23.244 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:23.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:23.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:23.377 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:23.397 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:23.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:23.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:23.526 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:23.544 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:23.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:23.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.516+0000 7f7130733700 1 -- 172.21.15.107:0/2677748631 >> v1:172.21.15.107:6790/0 conn(0x7f7128103250 legacy=0x7f71281056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- 172.21.15.107:0/2677748631 shutdown_connections 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- 172.21.15.107:0/2677748631 >> 172.21.15.107:0/2677748631 conn(0x7f71280f2510 msgr2=0x7f71280f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- 172.21.15.107:0/2677748631 shutdown_connections 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- 172.21.15.107:0/2677748631 wait complete. 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 librados: starting msgr at 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 librados: starting objecter 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 Processor -- start 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- start start 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 librados: setting wanted keys 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 librados: calling monclient init 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71280fbf40 con 0x7f71280ff710 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71280fc130 con 0x7f7128103250 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.517+0000 7f7130733700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71280fc320 con 0x7f71280fc910 2024-02-10T08:45:23.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f712e4cf700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f71280fc910 0x7f71280fb740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51754/0 (socket says 172.21.15.107:51754) 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f712e4cf700 1 -- 172.21.15.107:0/3806212038 learned_addr learned my addr 172.21.15.107:0/3806212038 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter ms_handle_connect 0x7f71280fc910 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter ms_handle_connect 0x7f71280ff710 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter ms_handle_connect 0x7f7128103250 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611939856 0 0) 0x7f71280fbf40 con 0x7f71280ff710 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71140033f0 con 0x7f71280ff710 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127158180 0 0) 0x7f71280fc320 con 0x7f71280fc910 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71280fbf40 con 0x7f71280fc910 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816637348 0 0) 0x7f71280fc130 con 0x7f7128103250 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71280fc320 con 0x7f7128103250 2024-02-10T08:45:23.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474730751 0 0) 0x7f71140033f0 con 0x7f71280ff710 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71280fc130 con 0x7f71280ff710 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f711c0019d0 con 0x7f71280ff710 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4213712517 0 0) 0x7f71280fc320 con 0x7f7128103250 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71140033f0 con 0x7f7128103250 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142907376 0 0) 0x7f71280fbf40 con 0x7f71280fc910 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71280fc320 con 0x7f71280fc910 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71200023d0 con 0x7f7128103250 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7124001900 con 0x7f71280fc910 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1974973875 0 0) 0x7f71280fc130 con 0x7f71280ff710 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 >> v1:172.21.15.107:6790/0 conn(0x7f71280fc910 legacy=0x7f71280fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 >> v1:172.21.15.195:6789/0 conn(0x7f7128103250 legacy=0x7f71281183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712811cb70 con 0x7f71280ff710 2024-02-10T08:45:23.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7128119b60 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f7130733700 10 client.28389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f712811a050 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f7130733700 1 librados: init done 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f711c002280 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.518+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f711c003140 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.519+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f710c004f00 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f711c001de0 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(577..577 src has 420..577) v4 ==== 8615+0+0 (unknown 2028430681 0 0) 0x7f711c076500 con 0x7f71280ff710 2024-02-10T08:45:23.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 osd_map(577..577 src has 420..577) v4 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 3 client.28389.objecter handle_osd_map got epochs [577,577] > 0 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 3 client.28389.objecter handle_osd_map decoding full epoch 577 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 20 client.28389.objecter dump_active .. 0 homeless 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v577) v1 ==== 54+0+16 (unknown 1414674928 0 2420000192) 0x7f711c0768e0 con 0x7f71280ff710 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.520+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f7114063c80 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 10 client.28389.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 1 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 1 2024-02-10T08:45:23.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 0 1 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 4 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 10 client.28389.objecter _send_command 1 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c004950 con 0x7f710c004530 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c004530 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c004950 con 0x7f710c004530 2024-02-10T08:45:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 10 client.28389.objecter _finish_command 1 = system:0 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 4 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.521+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 0 1 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 10 client.28389.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 2 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 2 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c0051a0 osd=0 3 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 0 2 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 4 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 10 client.28389.objecter _send_command 2 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.536+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c004d30 con 0x7f710c004530 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.537+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c004d30 con 0x7f710c004530 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.537+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.537+0000 7f711b7fe700 10 client.28389.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.537+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c0051a0 osd=0 4 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.537+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 0 2 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 10 client.28389.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 3 2024-02-10T08:45:23.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 1 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 4 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 10 client.28389.objecter _send_command 3 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c00bf60 con 0x7f710c003b40 2024-02-10T08:45:23.594 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c003b40 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c00bf60 con 0x7f710c003b40 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.538+0000 7f711b7fe700 10 client.28389.objecter _finish_command 3 = system:0 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.539+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 4 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.539+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 1 3 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 10 client.28389.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 4 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c004270 osd=1 3 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 1 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 10 client.28389.objecter _send_command 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.553+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c00d280 con 0x7f710c003b40 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.554+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c00d280 con 0x7f710c003b40 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.554+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.554+0000 7f711b7fe700 10 client.28389.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.554+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c004270 osd=1 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.554+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 1 4 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 10 client.28389.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 5 2024-02-10T08:45:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 5 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 2 5 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 4 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 10 client.28389.objecter _send_command 5 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c010d30 con 0x7f710c002070 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c002070 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c010d30 con 0x7f710c002070 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 10 client.28389.objecter _finish_command 5 = system:0 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 4 2024-02-10T08:45:23.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.555+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 2 5 2024-02-10T08:45:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 10 client.28389.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 6 2024-02-10T08:45:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.598 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 6 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c0033a0 osd=2 3 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 2 6 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 4 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 10 client.28389.objecter _send_command 6 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.569+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c001690 con 0x7f710c002070 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c001690 con 0x7f710c002070 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 10 client.28389.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c0033a0 osd=2 4 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 2 6 2024-02-10T08:45:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 10 client.28389.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 7 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 7 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 3 7 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 4 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 10 client.28389.objecter _send_command 7 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c00d0e0 con 0x7f710c010ed0 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.571+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c010ed0 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.572+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c00d0e0 con 0x7f710c010ed0 2024-02-10T08:45:23.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.572+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.572+0000 7f711b7fe700 10 client.28389.objecter _finish_command 7 = system:0 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.572+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 4 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.572+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 3 7 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 10 client.28389.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 8 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 8 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c001920 osd=3 3 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 3 8 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 4 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 10 client.28389.objecter _send_command 8 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.586+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c012260 con 0x7f710c010ed0 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.587+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c012260 con 0x7f710c010ed0 2024-02-10T08:45:23.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.587+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.587+0000 7f711b7fe700 10 client.28389.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.587+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c001920 osd=3 4 2024-02-10T08:45:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.587+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 3 8 2024-02-10T08:45:23.602 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 10 client.28389.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 9 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 10 client.28389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=578}) v3 -- 0x7f710c00c580 con 0x7f71280ff710 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f710c012260 con 0x7f71280ff710 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_get_version_reply(handle=1 version=577) v2 ==== 24+0+0 (unknown 2950286038 0 0) 0x7f711c0494c0 con 0x7f71280ff710 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f712cccc700 10 client.28389.objecter _check_command_map_dne tid 9 current 577 map_dne_bound 577 2024-02-10T08:45:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f712cccc700 10 client.28389.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f712cccc700 15 client.28389.objecter _session_command_op_remove -1 9 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 10 client.28389.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 10 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 10 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 5 10 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 4 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 10 client.28389.objecter _send_command 10 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c0131c0 con 0x7f710c011e40 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.588+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c011e40 2024-02-10T08:45:23.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.589+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c0131c0 con 0x7f710c011e40 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.589+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.589+0000 7f711b7fe700 10 client.28389.objecter _finish_command 10 = system:0 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.589+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 4 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.589+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 5 10 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 10 client.28389.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 11 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 11 2024-02-10T08:45:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c011cd0 osd=5 3 2024-02-10T08:45:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 5 11 2024-02-10T08:45:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 4 2024-02-10T08:45:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 10 client.28389.objecter _send_command 11 2024-02-10T08:45:23.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.603+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c013360 con 0x7f710c011e40 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.605+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c013360 con 0x7f710c011e40 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.605+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.605+0000 7f711b7fe700 10 client.28389.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.605+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c011cd0 osd=5 4 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.605+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 5 11 2024-02-10T08:45:23.607 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 10 client.28389.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 12 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 12 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 6 12 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 4 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 10 client.28389.objecter _send_command 12 2024-02-10T08:45:23.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c01efc0 con 0x7f710c01b4b0 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.606+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c01b4b0 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.607+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c01efc0 con 0x7f710c01b4b0 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.607+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.607+0000 7f711b7fe700 10 client.28389.objecter _finish_command 12 = system:0 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.607+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 4 2024-02-10T08:45:23.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.607+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 6 12 2024-02-10T08:45:23.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 10 client.28389.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 13 2024-02-10T08:45:23.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 13 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c013360 osd=6 3 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 6 13 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 4 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 10 client.28389.objecter _send_command 13 2024-02-10T08:45:23.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.621+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c020740 con 0x7f710c01b4b0 2024-02-10T08:45:23.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c020740 con 0x7f710c01b4b0 2024-02-10T08:45:23.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 10 client.28389.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c013360 osd=6 4 2024-02-10T08:45:23.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 6 13 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 10 client.28389.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 14 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 14 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 7 14 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 4 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 10 client.28389.objecter _send_command 14 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c021370 con 0x7f710c020e20 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.623+0000 7f711b7fe700 10 client.28389.objecter ms_handle_connect 0x7f710c020e20 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.624+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f710c021370 con 0x7f710c020e20 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.624+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.624+0000 7f711b7fe700 10 client.28389.objecter _finish_command 14 = system:0 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.624+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 4 2024-02-10T08:45:23.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.624+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 7 14 2024-02-10T08:45:23.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 10 client.28389.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:23.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign -1 15 2024-02-10T08:45:23.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 20 client.28389.objecter _get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 15 client.28389.objecter _session_command_op_remove -1 15 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 20 client.28389.objecter get_session s=0x7f710c020740 osd=7 3 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 15 client.28389.objecter _session_command_op_assign 7 15 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 4 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 10 client.28389.objecter _send_command 15 2024-02-10T08:45:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.638+0000 7f7130733700 1 -- 172.21.15.107:0/3806212038 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f710c021370 con 0x7f710c020e20 2024-02-10T08:45:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.640+0000 7f711b7fe700 1 -- 172.21.15.107:0/3806212038 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f710c021370 con 0x7f710c020e20 2024-02-10T08:45:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.640+0000 7f711b7fe700 10 client.28389.objecter ms_dispatch 0x7f7128001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.640+0000 7f711b7fe700 10 client.28389.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.640+0000 7f711b7fe700 20 client.28389.objecter put_session s=0x7f710c020740 osd=7 4 2024-02-10T08:45:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:23.640+0000 7f711b7fe700 15 client.28389.objecter _session_command_op_remove 7 15 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.643 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:45:23.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:23.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.033518+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.037062+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.419473+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.e scrub starts 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.422378+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.e scrub ok 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.561449+0000 osd.6 (osd.6) 246 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.562834+0000 osd.6 (osd.6) 247 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.582082+0000 osd.1 (osd.1) 278 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:22.585211+0000 osd.1 (osd.1) 279 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.376097+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.033518+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.037062+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:45:23.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.419473+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.e scrub starts 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.422378+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.e scrub ok 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.561449+0000 osd.6 (osd.6) 246 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.562834+0000 osd.6 (osd.6) 247 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.582082+0000 osd.1 (osd.1) 278 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:22.585211+0000 osd.1 (osd.1) 279 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:45:23.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.376097+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-10T08:45:23.675 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.033518+0000 osd.5 (osd.5) 243 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.037062+0000 osd.5 (osd.5) 244 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.419473+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.e scrub starts 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.422378+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.e scrub ok 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.561449+0000 osd.6 (osd.6) 246 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.562834+0000 osd.6 (osd.6) 247 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:45:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.582082+0000 osd.1 (osd.1) 278 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:45:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:22.585211+0000 osd.1 (osd.1) 279 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:45:23.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.376097+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e577: 8 total, 7 up, 7 in 2024-02-10T08:45:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:24.590 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-02-10T08:45:21.097755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1854,"num_read_kb":1604,"num_write":722,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":6775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":808,"ondisk_log_size":808,"up":423,"acting":424,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":628,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2170784,"kb_used_data":20584,"kb_used_omap":0,"kb_used_meta":2150144,"kb_avail":654045280,"statfs":{"total":671965249536,"available":669742366720,"internally_reserved":0,"allocated":21078016,"data_stored":10016126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2201747456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.991653"},"pg_stats":[{"pgid":"229.7","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.108648+0000","last_change":"2024-02-10T08:45:10.421219+0000","last_active":"2024-02-10T08:45:15.108648+0000","last_peered":"2024-02-10T08:45:15.108648+0000","last_clean":"2024-02-10T08:45:15.108648+0000","last_became_active":"2024-02-10T08:45:10.420818+0000","last_became_peered":"2024-02-10T08:45:10.420818+0000","last_unstale":"2024-02-10T08:45:15.108648+0000","last_undegraded":"2024-02-10T08:45:15.108648+0000","last_fullsized":"2024-02-10T08:45:15.108648+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.108620+0000","last_change":"2024-02-10T08:45:10.145555+0000","last_active":"2024-02-10T08:45:15.108620+0000","last_peered":"2024-02-10T08:45:15.108620+0000","last_clean":"2024-02-10T08:45:15.108620+0000","last_became_active":"2024-02-10T08:45:10.145228+0000","last_became_peered":"2024-02-10T08:45:10.145228+0000","last_unstale":"2024-02-10T08:45:15.108620+0000","last_undegraded":"2024-02-10T08:45:15.108620+0000","last_fullsized":"2024-02-10T08:45:15.108620+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.5","version":"0'0","reported_seq":19,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148345+0000","last_change":"2024-02-10T08:45:10.143042+0000","last_active":"2024-02-10T08:45:19.148345+0000","last_peered":"2024-02-10T08:45:19.148345+0000","last_clean":"2024-02-10T08:45:19.148345+0000","last_became_active":"2024-02-10T08:45:10.142687+0000","last_became_peered":"2024-02-10T08:45:10.142687+0000","last_unstale":"2024-02-10T08:45:19.148345+0000","last_undegraded":"2024-02-10T08:45:19.148345+0000","last_fullsized":"2024-02-10T08:45:19.148345+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104941+0000","last_change":"2024-02-10T08:45:10.152357+0000","last_active":"2024-02-10T08:45:15.104941+0000","last_peered":"2024-02-10T08:45:15.104941+0000","last_clean":"2024-02-10T08:45:15.104941+0000","last_became_active":"2024-02-10T08:45:10.152036+0000","last_became_peered":"2024-02-10T08:45:10.152036+0000","last_unstale":"2024-02-10T08:45:15.104941+0000","last_undegraded":"2024-02-10T08:45:15.104941+0000","last_fullsized":"2024-02-10T08:45:15.104941+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":18,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431468+0000","last_change":"2024-02-10T08:45:10.150379+0000","last_active":"2024-02-10T08:45:19.431468+0000","last_peered":"2024-02-10T08:45:19.431468+0000","last_clean":"2024-02-10T08:45:19.431468+0000","last_became_active":"2024-02-10T08:45:10.150078+0000","last_became_peered":"2024-02-10T08:45:10.150078+0000","last_unstale":"2024-02-10T08:45:19.431468+0000","last_undegraded":"2024-02-10T08:45:19.431468+0000","last_fullsized":"2024-02-10T08:45:19.431468+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"229.2","version":"0'0","reported_seq":19,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148132+0000","last_change":"2024-02-10T08:45:10.145215+0000","last_active":"2024-02-10T08:45:19.148132+0000","last_peered":"2024-02-10T08:45:19.148132+0000","last_clean":"2024-02-10T08:45:19.148132+0000","last_became_active":"2024-02-10T08:45:10.144871+0000","last_became_peered":"2024-02-10T08:45:10.144871+0000","last_unstale":"2024-02-10T08:45:19.148132+0000","last_undegraded":"2024-02-10T08:45:19.148132+0000","last_fullsized":"2024-02-10T08:45:19.148132+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"229.1","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104420+0000","last_change":"2024-02-10T08:45:10.421086+0000","last_active":"2024-02-10T08:45:15.104420+0000","last_peered":"2024-02-10T08:45:15.104420+0000","last_clean":"2024-02-10T08:45:15.104420+0000","last_became_active":"2024-02-10T08:45:10.420742+0000","last_became_peered":"2024-02-10T08:45:10.420742+0000","last_unstale":"2024-02-10T08:45:15.104420+0000","last_undegraded":"2024-02-10T08:45:15.104420+0000","last_fullsized":"2024-02-10T08:45:15.104420+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"229.0","version":"564'4","reported_seq":20,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.117971+0000","last_change":"2024-02-10T08:45:10.422379+0000","last_active":"2024-02-10T08:45:14.117971+0000","last_peered":"2024-02-10T08:45:14.117971+0000","last_clean":"2024-02-10T08:45:14.117971+0000","last_became_active":"2024-02-10T08:45:10.421713+0000","last_became_peered":"2024-02-10T08:45:10.421713+0000","last_unstale":"2024-02-10T08:45:14.117971+0000","last_undegraded":"2024-02-10T08:45:14.117971+0000","last_fullsized":"2024-02-10T08:45:14.117971+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:09.129987+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.129987+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"481'17","reported_seq":201,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.432152+0000","last_change":"2024-02-10T08:43:36.074524+0000","last_active":"2024-02-10T08:45:19.432152+0000","last_peered":"2024-02-10T08:45:19.432152+0000","last_clean":"2024-02-10T08:43:31.610321+0000","last_became_active":"2024-02-10T08:43:36.074524+0000","last_became_peered":"2024-02-10T08:43:36.074524+0000","last_unstale":"2024-02-10T08:45:19.432152+0000","last_undegraded":"2024-02-10T08:43:36.061132+0000","last_fullsized":"2024-02-10T08:43:36.061013+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'13","last_scrub_stamp":"2024-02-10T08:43:10.390465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.390465+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":639,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146236+0000","last_change":"2024-02-10T08:43:15.731491+0000","last_active":"2024-02-10T08:45:19.146236+0000","last_peered":"2024-02-10T08:45:19.146236+0000","last_clean":"2024-02-10T08:45:19.146236+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:45:19.146236+0000","last_undegraded":"2024-02-10T08:45:19.146236+0000","last_fullsized":"2024-02-10T08:45:19.146236+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'22","last_scrub_stamp":"2024-02-10T08:42:25.091640+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.091640+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":560,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.106230+0000","last_change":"2024-02-10T08:43:15.119398+0000","last_active":"2024-02-10T08:45:15.106230+0000","last_peered":"2024-02-10T08:45:15.106230+0000","last_clean":"2024-02-10T08:45:15.106230+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:45:15.106230+0000","last_undegraded":"2024-02-10T08:45:15.106230+0000","last_fullsized":"2024-02-10T08:45:15.106230+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.504915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.504915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":549,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148442+0000","last_change":"2024-02-10T08:42:41.108507+0000","last_active":"2024-02-10T08:45:19.148442+0000","last_peered":"2024-02-10T08:45:19.148442+0000","last_clean":"2024-02-10T08:45:19.148442+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:45:19.148442+0000","last_undegraded":"2024-02-10T08:45:19.148442+0000","last_fullsized":"2024-02-10T08:45:19.148442+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:41.108418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.108418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":490,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.105101+0000","last_change":"2024-02-10T08:44:28.457808+0000","last_active":"2024-02-10T08:45:15.105101+0000","last_peered":"2024-02-10T08:45:15.105101+0000","last_clean":"2024-02-10T08:45:15.105101+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:45:15.105101+0000","last_undegraded":"2024-02-10T08:45:15.105101+0000","last_fullsized":"2024-02-10T08:45:15.105101+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:28.457738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.457738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":529,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.105160+0000","last_change":"2024-02-10T08:43:36.079191+0000","last_active":"2024-02-10T08:45:15.105160+0000","last_peered":"2024-02-10T08:45:15.105160+0000","last_clean":"2024-02-10T08:43:31.066295+0000","last_became_active":"2024-02-10T08:43:36.079191+0000","last_became_peered":"2024-02-10T08:43:36.079191+0000","last_unstale":"2024-02-10T08:45:15.105160+0000","last_undegraded":"2024-02-10T08:43:36.059738+0000","last_fullsized":"2024-02-10T08:43:36.059623+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":554,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.109123+0000","last_change":"2024-02-10T08:43:36.085621+0000","last_active":"2024-02-10T08:45:15.109123+0000","last_peered":"2024-02-10T08:45:15.109123+0000","last_clean":"2024-02-10T08:43:31.059000+0000","last_became_active":"2024-02-10T08:43:36.085621+0000","last_became_peered":"2024-02-10T08:43:36.085621+0000","last_unstale":"2024-02-10T08:45:15.109123+0000","last_undegraded":"2024-02-10T08:45:15.109123+0000","last_fullsized":"2024-02-10T08:43:36.063451+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.696704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.696704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":562,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.104981+0000","last_change":"2024-02-10T08:43:36.079793+0000","last_active":"2024-02-10T08:45:15.104981+0000","last_peered":"2024-02-10T08:45:15.104981+0000","last_clean":"2024-02-10T08:43:31.065630+0000","last_became_active":"2024-02-10T08:43:36.079793+0000","last_became_peered":"2024-02-10T08:43:36.079793+0000","last_unstale":"2024-02-10T08:45:15.104981+0000","last_undegraded":"2024-02-10T08:45:15.104981+0000","last_fullsized":"2024-02-10T08:43:36.061952+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.432052+0000","last_change":"2024-02-10T08:44:39.598037+0000","last_active":"2024-02-10T08:45:19.432052+0000","last_peered":"2024-02-10T08:45:19.432052+0000","last_clean":"2024-02-10T08:45:19.432052+0000","last_became_active":"2024-02-10T08:43:15.128618+0000","last_became_peered":"2024-02-10T08:43:15.128618+0000","last_unstale":"2024-02-10T08:45:19.432052+0000","last_undegraded":"2024-02-10T08:45:19.432052+0000","last_fullsized":"2024-02-10T08:45:19.432052+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.597966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.597966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":507,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.147568+0000","last_change":"2024-02-10T08:44:39.356595+0000","last_active":"2024-02-10T08:45:19.147568+0000","last_peered":"2024-02-10T08:45:19.147568+0000","last_clean":"2024-02-10T08:45:19.147568+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:45:19.147568+0000","last_undegraded":"2024-02-10T08:45:19.147568+0000","last_fullsized":"2024-02-10T08:45:19.147568+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.356502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.356502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"473'35","reported_seq":664,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.111217+0000","last_change":"2024-02-10T08:44:37.425734+0000","last_active":"2024-02-10T08:45:19.111217+0000","last_peered":"2024-02-10T08:45:19.111217+0000","last_clean":"2024-02-10T08:45:19.111217+0000","last_became_active":"2024-02-10T08:43:15.136753+0000","last_became_peered":"2024-02-10T08:43:15.136753+0000","last_unstale":"2024-02-10T08:45:19.111217+0000","last_undegraded":"2024-02-10T08:45:19.111217+0000","last_fullsized":"2024-02-10T08:45:19.111217+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'35","last_scrub_stamp":"2024-02-10T08:44:37.425670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.425670+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":551,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.105343+0000","last_change":"2024-02-10T08:43:53.472550+0000","last_active":"2024-02-10T08:45:15.105343+0000","last_peered":"2024-02-10T08:45:15.105343+0000","last_clean":"2024-02-10T08:45:15.105343+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:45:15.105343+0000","last_undegraded":"2024-02-10T08:45:15.105343+0000","last_fullsized":"2024-02-10T08:45:15.105343+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:53.472449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:43:53.472449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":585,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.105079+0000","last_change":"2024-02-10T08:43:23.422453+0000","last_active":"2024-02-10T08:45:15.105079+0000","last_peered":"2024-02-10T08:45:15.105079+0000","last_clean":"2024-02-10T08:45:15.105079+0000","last_became_active":"2024-02-10T08:43:15.119008+0000","last_became_peered":"2024-02-10T08:43:15.119008+0000","last_unstale":"2024-02-10T08:45:15.105079+0000","last_undegraded":"2024-02-10T08:45:15.105079+0000","last_fullsized":"2024-02-10T08:45:15.105079+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.422346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.422346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431390+0000","last_change":"2024-02-10T08:44:40.551722+0000","last_active":"2024-02-10T08:45:19.431390+0000","last_peered":"2024-02-10T08:45:19.431390+0000","last_clean":"2024-02-10T08:45:19.431390+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:45:19.431390+0000","last_undegraded":"2024-02-10T08:45:19.431390+0000","last_fullsized":"2024-02-10T08:45:19.431390+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.551662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.432709+0000","last_change":"2024-02-10T08:43:36.072003+0000","last_active":"2024-02-10T08:45:19.432709+0000","last_peered":"2024-02-10T08:45:19.432709+0000","last_clean":"2024-02-10T08:43:31.061070+0000","last_became_active":"2024-02-10T08:43:36.072003+0000","last_became_peered":"2024-02-10T08:43:36.072003+0000","last_unstale":"2024-02-10T08:45:19.432709+0000","last_undegraded":"2024-02-10T08:45:19.432709+0000","last_fullsized":"2024-02-10T08:43:36.062752+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":587,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.106003+0000","last_change":"2024-02-10T08:43:36.076710+0000","last_active":"2024-02-10T08:45:15.106003+0000","last_peered":"2024-02-10T08:45:15.106003+0000","last_clean":"2024-02-10T08:43:25.756727+0000","last_became_active":"2024-02-10T08:43:36.076710+0000","last_became_peered":"2024-02-10T08:43:36.076710+0000","last_unstale":"2024-02-10T08:45:15.106003+0000","last_undegraded":"2024-02-10T08:43:36.060349+0000","last_fullsized":"2024-02-10T08:43:36.060248+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:43:29.844454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:29.844454+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":380,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.127360+0000","last_change":"2024-02-10T08:44:33.297486+0000","last_active":"2024-02-10T08:45:14.127360+0000","last_peered":"2024-02-10T08:45:14.127360+0000","last_clean":"2024-02-10T08:45:14.127360+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:45:14.127360+0000","last_undegraded":"2024-02-10T08:45:14.127360+0000","last_fullsized":"2024-02-10T08:45:14.127360+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:44:33.297424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":572,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.109661+0000","last_change":"2024-02-10T08:43:36.076112+0000","last_active":"2024-02-10T08:45:15.109661+0000","last_peered":"2024-02-10T08:45:15.109661+0000","last_clean":"2024-02-10T08:43:31.060641+0000","last_became_active":"2024-02-10T08:43:36.076112+0000","last_became_peered":"2024-02-10T08:43:36.076112+0000","last_unstale":"2024-02-10T08:45:15.109661+0000","last_undegraded":"2024-02-10T08:45:15.109661+0000","last_fullsized":"2024-02-10T08:43:36.063764+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":542,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.107498+0000","last_change":"2024-02-10T08:43:36.075565+0000","last_active":"2024-02-10T08:45:15.107498+0000","last_peered":"2024-02-10T08:45:15.107498+0000","last_clean":"2024-02-10T08:43:31.066268+0000","last_became_active":"2024-02-10T08:43:36.075565+0000","last_became_peered":"2024-02-10T08:43:36.075565+0000","last_unstale":"2024-02-10T08:45:15.107498+0000","last_undegraded":"2024-02-10T08:45:15.107498+0000","last_fullsized":"2024-02-10T08:43:36.061936+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":96,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.119511+0000","last_change":"2024-02-10T08:43:36.089962+0000","last_active":"2024-02-10T08:45:14.119511+0000","last_peered":"2024-02-10T08:45:14.119511+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"2024-02-10T08:43:36.089962+0000","last_became_peered":"2024-02-10T08:43:36.089962+0000","last_unstale":"2024-02-10T08:45:14.119511+0000","last_undegraded":"2024-02-10T08:45:14.119511+0000","last_fullsized":"2024-02-10T08:43:36.068401+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":567,"reported_epoch":568,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:14.121546+0000","last_change":"2024-02-10T08:43:36.073845+0000","last_active":"2024-02-10T08:45:14.121546+0000","last_peered":"2024-02-10T08:45:14.121546+0000","last_clean":"2024-02-10T08:43:25.718471+0000","last_became_active":"2024-02-10T08:43:36.073845+0000","last_became_peered":"2024-02-10T08:43:36.073845+0000","last_unstale":"2024-02-10T08:45:14.121546+0000","last_undegraded":"2024-02-10T08:43:36.068245+0000","last_fullsized":"2024-02-10T08:43:36.068123+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":571,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.107244+0000","last_change":"2024-02-10T08:44:41.730549+0000","last_active":"2024-02-10T08:45:15.107244+0000","last_peered":"2024-02-10T08:45:15.107244+0000","last_clean":"2024-02-10T08:45:15.107244+0000","last_became_active":"2024-02-10T08:43:15.130483+0000","last_became_peered":"2024-02-10T08:43:15.130483+0000","last_unstale":"2024-02-10T08:45:15.107244+0000","last_undegraded":"2024-02-10T08:45:15.107244+0000","last_fullsized":"2024-02-10T08:45:15.107244+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:41.730476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:41.730476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"469'16","reported_seq":763,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.105951+0000","last_change":"2024-02-10T08:43:27.381317+0000","last_active":"2024-02-10T08:45:15.105951+0000","last_peered":"2024-02-10T08:45:15.105951+0000","last_clean":"2024-02-10T08:45:15.105951+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:45:15.105951+0000","last_undegraded":"2024-02-10T08:45:15.105951+0000","last_fullsized":"2024-02-10T08:45:15.105951+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'16","last_scrub_stamp":"2024-02-10T08:43:27.381246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.381246+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.108957+0000","last_change":"2024-02-10T08:43:36.084792+0000","last_active":"2024-02-10T08:45:15.108957+0000","last_peered":"2024-02-10T08:45:15.108957+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.084792+0000","last_became_peered":"2024-02-10T08:43:36.084792+0000","last_unstale":"2024-02-10T08:45:15.108957+0000","last_undegraded":"2024-02-10T08:45:15.108957+0000","last_fullsized":"2024-02-10T08:43:36.060560+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":478,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.118117+0000","last_change":"2024-02-10T08:43:36.082315+0000","last_active":"2024-02-10T08:45:14.118117+0000","last_peered":"2024-02-10T08:45:14.118117+0000","last_clean":"2024-02-10T08:43:31.057840+0000","last_became_active":"2024-02-10T08:43:36.082315+0000","last_became_peered":"2024-02-10T08:43:36.082315+0000","last_unstale":"2024-02-10T08:45:14.118117+0000","last_undegraded":"2024-02-10T08:45:14.118117+0000","last_fullsized":"2024-02-10T08:43:36.070820+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.219434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.219434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":590,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.108934+0000","last_change":"2024-02-10T08:43:36.088120+0000","last_active":"2024-02-10T08:45:15.108934+0000","last_peered":"2024-02-10T08:45:15.108934+0000","last_clean":"2024-02-10T08:43:31.056937+0000","last_became_active":"2024-02-10T08:43:36.088120+0000","last_became_peered":"2024-02-10T08:43:36.088120+0000","last_unstale":"2024-02-10T08:45:15.108934+0000","last_undegraded":"2024-02-10T08:43:36.061412+0000","last_fullsized":"2024-02-10T08:43:36.061296+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":582,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.106344+0000","last_change":"2024-02-10T08:43:50.689633+0000","last_active":"2024-02-10T08:45:15.106344+0000","last_peered":"2024-02-10T08:45:15.106344+0000","last_clean":"2024-02-10T08:45:15.106344+0000","last_became_active":"2024-02-10T08:43:15.130693+0000","last_became_peered":"2024-02-10T08:43:15.130693+0000","last_unstale":"2024-02-10T08:45:15.106344+0000","last_undegraded":"2024-02-10T08:45:15.106344+0000","last_fullsized":"2024-02-10T08:45:15.106344+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:43:50.689512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:50.689512+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"481'17","reported_seq":141,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.147791+0000","last_change":"2024-02-10T08:43:36.076136+0000","last_active":"2024-02-10T08:45:19.147791+0000","last_peered":"2024-02-10T08:45:19.147791+0000","last_clean":"2024-02-10T08:33:25.224233+0000","last_became_active":"2024-02-10T08:43:36.076136+0000","last_became_peered":"2024-02-10T08:43:36.076136+0000","last_unstale":"2024-02-10T08:45:19.147791+0000","last_undegraded":"2024-02-10T08:43:36.056738+0000","last_fullsized":"2024-02-10T08:43:36.056588+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:42:44.647464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.647464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":544,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148617+0000","last_change":"2024-02-10T08:42:34.139662+0000","last_active":"2024-02-10T08:45:19.148617+0000","last_peered":"2024-02-10T08:45:19.148617+0000","last_clean":"2024-02-10T08:45:19.148617+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:45:19.148617+0000","last_undegraded":"2024-02-10T08:45:19.148617+0000","last_fullsized":"2024-02-10T08:45:19.148617+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.139528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.139528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":507,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146652+0000","last_change":"2024-02-10T08:44:35.383065+0000","last_active":"2024-02-10T08:45:19.146652+0000","last_peered":"2024-02-10T08:45:19.146652+0000","last_clean":"2024-02-10T08:45:19.146652+0000","last_became_active":"2024-02-10T08:43:00.552233+0000","last_became_peered":"2024-02-10T08:43:00.552233+0000","last_unstale":"2024-02-10T08:45:19.146652+0000","last_undegraded":"2024-02-10T08:45:19.146652+0000","last_fullsized":"2024-02-10T08:45:19.146652+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.382972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.382972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":568,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.114348+0000","last_change":"2024-02-10T08:43:36.068792+0000","last_active":"2024-02-10T08:45:19.114348+0000","last_peered":"2024-02-10T08:45:19.114348+0000","last_clean":"2024-02-10T08:43:25.758929+0000","last_became_active":"2024-02-10T08:43:36.068792+0000","last_became_peered":"2024-02-10T08:43:36.068792+0000","last_unstale":"2024-02-10T08:45:19.114348+0000","last_undegraded":"2024-02-10T08:43:36.064019+0000","last_fullsized":"2024-02-10T08:43:36.063905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.431245+0000","last_change":"2024-02-10T08:43:36.068338+0000","last_active":"2024-02-10T08:45:19.431245+0000","last_peered":"2024-02-10T08:45:19.431245+0000","last_clean":"2024-02-10T08:43:31.059980+0000","last_became_active":"2024-02-10T08:43:36.068338+0000","last_became_peered":"2024-02-10T08:43:36.068338+0000","last_unstale":"2024-02-10T08:45:19.431245+0000","last_undegraded":"2024-02-10T08:45:19.431245+0000","last_fullsized":"2024-02-10T08:43:36.060130+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.736740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.736740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"481'11","reported_seq":669,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.147354+0000","last_change":"2024-02-10T08:43:36.068155+0000","last_active":"2024-02-10T08:45:19.147354+0000","last_peered":"2024-02-10T08:45:19.147354+0000","last_clean":"2024-02-10T08:43:31.610349+0000","last_became_active":"2024-02-10T08:43:36.068155+0000","last_became_peered":"2024-02-10T08:43:36.068155+0000","last_unstale":"2024-02-10T08:45:19.147354+0000","last_undegraded":"2024-02-10T08:43:36.058153+0000","last_fullsized":"2024-02-10T08:43:36.058069+0000","mapping_epoch":480,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":536,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.121090+0000","last_change":"2024-02-10T08:44:26.397785+0000","last_active":"2024-02-10T08:45:14.121090+0000","last_peered":"2024-02-10T08:45:14.121090+0000","last_clean":"2024-02-10T08:45:14.121090+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:45:14.121090+0000","last_undegraded":"2024-02-10T08:45:14.121090+0000","last_fullsized":"2024-02-10T08:45:14.121090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.397690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.397690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":484,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.124847+0000","last_change":"2024-02-10T08:42:47.240729+0000","last_active":"2024-02-10T08:45:14.124847+0000","last_peered":"2024-02-10T08:45:14.124847+0000","last_clean":"2024-02-10T08:45:14.124847+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:45:14.124847+0000","last_undegraded":"2024-02-10T08:45:14.124847+0000","last_fullsized":"2024-02-10T08:45:14.124847+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.240624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.240624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"481'9","reported_seq":697,"reported_epoch":568,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:14.118522+0000","last_change":"2024-02-10T08:43:36.077627+0000","last_active":"2024-02-10T08:45:14.118522+0000","last_peered":"2024-02-10T08:45:14.118522+0000","last_clean":"2024-02-10T08:43:31.610208+0000","last_became_active":"2024-02-10T08:43:36.077627+0000","last_became_peered":"2024-02-10T08:43:36.077627+0000","last_unstale":"2024-02-10T08:45:14.118522+0000","last_undegraded":"2024-02-10T08:43:36.066022+0000","last_fullsized":"2024-02-10T08:43:36.065899+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":583,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.146918+0000","last_change":"2024-02-10T08:43:36.064143+0000","last_active":"2024-02-10T08:45:19.146918+0000","last_peered":"2024-02-10T08:45:19.146918+0000","last_clean":"2024-02-10T08:43:25.763333+0000","last_became_active":"2024-02-10T08:43:36.064143+0000","last_became_peered":"2024-02-10T08:43:36.064143+0000","last_unstale":"2024-02-10T08:45:19.146918+0000","last_undegraded":"2024-02-10T08:43:36.057979+0000","last_fullsized":"2024-02-10T08:43:36.057862+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:43:34.883363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.883363+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":575,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.113219+0000","last_change":"2024-02-10T08:44:29.439534+0000","last_active":"2024-02-10T08:45:19.113219+0000","last_peered":"2024-02-10T08:45:19.113219+0000","last_clean":"2024-02-10T08:45:19.113219+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:45:19.113219+0000","last_undegraded":"2024-02-10T08:45:19.113219+0000","last_fullsized":"2024-02-10T08:45:19.113219+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.439473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.439473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.432283+0000","last_change":"2024-02-10T08:44:31.574745+0000","last_active":"2024-02-10T08:45:19.432283+0000","last_peered":"2024-02-10T08:45:19.432283+0000","last_clean":"2024-02-10T08:45:19.432283+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:45:19.432283+0000","last_undegraded":"2024-02-10T08:45:19.432283+0000","last_fullsized":"2024-02-10T08:45:19.432283+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:31.574681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.574681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":462,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.109242+0000","last_change":"2024-02-10T08:43:36.073331+0000","last_active":"2024-02-10T08:45:15.109242+0000","last_peered":"2024-02-10T08:45:15.109242+0000","last_clean":"2024-02-10T08:43:31.059113+0000","last_became_active":"2024-02-10T08:43:36.073331+0000","last_became_peered":"2024-02-10T08:43:36.073331+0000","last_unstale":"2024-02-10T08:45:15.109242+0000","last_undegraded":"2024-02-10T08:45:15.109242+0000","last_fullsized":"2024-02-10T08:43:36.063737+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":539,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.119221+0000","last_change":"2024-02-10T08:43:36.082324+0000","last_active":"2024-02-10T08:45:14.119221+0000","last_peered":"2024-02-10T08:45:14.119221+0000","last_clean":"2024-02-10T08:43:11.068543+0000","last_became_active":"2024-02-10T08:43:36.082324+0000","last_became_peered":"2024-02-10T08:43:36.082324+0000","last_unstale":"2024-02-10T08:45:14.119221+0000","last_undegraded":"2024-02-10T08:45:14.119221+0000","last_fullsized":"2024-02-10T08:43:36.067905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.619193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.619193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":704,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.118423+0000","last_change":"2024-02-10T08:44:37.388097+0000","last_active":"2024-02-10T08:45:14.118423+0000","last_peered":"2024-02-10T08:45:14.118423+0000","last_clean":"2024-02-10T08:45:14.118423+0000","last_became_active":"2024-02-10T08:43:15.136661+0000","last_became_peered":"2024-02-10T08:43:15.136661+0000","last_unstale":"2024-02-10T08:45:14.118423+0000","last_undegraded":"2024-02-10T08:45:14.118423+0000","last_fullsized":"2024-02-10T08:45:14.118423+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:44:37.388018+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.388018+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":556,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.108815+0000","last_change":"2024-02-10T08:43:36.068713+0000","last_active":"2024-02-10T08:45:15.108815+0000","last_peered":"2024-02-10T08:45:15.108815+0000","last_clean":"2024-02-10T08:43:31.057619+0000","last_became_active":"2024-02-10T08:43:36.068713+0000","last_became_peered":"2024-02-10T08:43:36.068713+0000","last_unstale":"2024-02-10T08:45:15.108815+0000","last_undegraded":"2024-02-10T08:43:36.060168+0000","last_fullsized":"2024-02-10T08:43:36.060055+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":96,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.124938+0000","last_change":"2024-02-10T08:43:36.080856+0000","last_active":"2024-02-10T08:45:14.124938+0000","last_peered":"2024-02-10T08:45:14.124938+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.080856+0000","last_became_peered":"2024-02-10T08:43:36.080856+0000","last_unstale":"2024-02-10T08:45:14.124938+0000","last_undegraded":"2024-02-10T08:45:14.124938+0000","last_fullsized":"2024-02-10T08:43:36.066562+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":472,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.120892+0000","last_change":"2024-02-10T08:43:00.547238+0000","last_active":"2024-02-10T08:45:14.120892+0000","last_peered":"2024-02-10T08:45:14.120892+0000","last_clean":"2024-02-10T08:45:14.120892+0000","last_became_active":"2024-02-10T08:43:00.546139+0000","last_became_peered":"2024-02-10T08:43:00.546139+0000","last_unstale":"2024-02-10T08:45:14.120892+0000","last_undegraded":"2024-02-10T08:45:14.120892+0000","last_fullsized":"2024-02-10T08:45:14.120892+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":553,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.111318+0000","last_change":"2024-02-10T08:43:59.374953+0000","last_active":"2024-02-10T08:45:19.111318+0000","last_peered":"2024-02-10T08:45:19.111318+0000","last_clean":"2024-02-10T08:45:19.111318+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:45:19.111318+0000","last_undegraded":"2024-02-10T08:45:19.111318+0000","last_fullsized":"2024-02-10T08:45:19.111318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.374859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.374859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":594,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146844+0000","last_change":"2024-02-10T08:43:25.480698+0000","last_active":"2024-02-10T08:45:19.146844+0000","last_peered":"2024-02-10T08:45:19.146844+0000","last_clean":"2024-02-10T08:45:19.146844+0000","last_became_active":"2024-02-10T08:43:15.126404+0000","last_became_peered":"2024-02-10T08:43:15.126404+0000","last_unstale":"2024-02-10T08:45:19.146844+0000","last_undegraded":"2024-02-10T08:45:19.146844+0000","last_fullsized":"2024-02-10T08:45:19.146844+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'21","last_scrub_stamp":"2024-02-10T08:43:25.480620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.480620+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.432298+0000","last_change":"2024-02-10T08:43:36.073770+0000","last_active":"2024-02-10T08:45:19.432298+0000","last_peered":"2024-02-10T08:45:19.432298+0000","last_clean":"2024-02-10T08:43:31.060583+0000","last_became_active":"2024-02-10T08:43:36.073770+0000","last_became_peered":"2024-02-10T08:43:36.073770+0000","last_unstale":"2024-02-10T08:45:19.432298+0000","last_undegraded":"2024-02-10T08:45:19.432298+0000","last_fullsized":"2024-02-10T08:43:36.061914+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.297949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.297949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":471,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104948+0000","last_change":"2024-02-10T08:44:24.395820+0000","last_active":"2024-02-10T08:45:15.104948+0000","last_peered":"2024-02-10T08:45:15.104948+0000","last_clean":"2024-02-10T08:45:15.104948+0000","last_became_active":"2024-02-10T08:43:15.106310+0000","last_became_peered":"2024-02-10T08:43:15.106310+0000","last_unstale":"2024-02-10T08:45:15.104948+0000","last_undegraded":"2024-02-10T08:45:15.104948+0000","last_fullsized":"2024-02-10T08:45:15.104948+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.395744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.395744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":499,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.107183+0000","last_change":"2024-02-10T08:43:59.552708+0000","last_active":"2024-02-10T08:45:15.107183+0000","last_peered":"2024-02-10T08:45:15.107183+0000","last_clean":"2024-02-10T08:45:15.107183+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:45:15.107183+0000","last_undegraded":"2024-02-10T08:45:15.107183+0000","last_fullsized":"2024-02-10T08:45:15.107183+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.552604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.552604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":575,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.147431+0000","last_change":"2024-02-10T08:43:36.069190+0000","last_active":"2024-02-10T08:45:19.147431+0000","last_peered":"2024-02-10T08:45:19.147431+0000","last_clean":"2024-02-10T08:43:31.062577+0000","last_became_active":"2024-02-10T08:43:36.069190+0000","last_became_peered":"2024-02-10T08:43:36.069190+0000","last_unstale":"2024-02-10T08:45:19.147431+0000","last_undegraded":"2024-02-10T08:45:19.147431+0000","last_fullsized":"2024-02-10T08:43:36.058346+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.561295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"473'24","reported_seq":574,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.108931+0000","last_change":"2024-02-10T08:44:40.723157+0000","last_active":"2024-02-10T08:45:15.108931+0000","last_peered":"2024-02-10T08:45:15.108931+0000","last_clean":"2024-02-10T08:45:15.108931+0000","last_became_active":"2024-02-10T08:43:15.129639+0000","last_became_peered":"2024-02-10T08:43:15.129639+0000","last_unstale":"2024-02-10T08:45:15.108931+0000","last_undegraded":"2024-02-10T08:45:15.108931+0000","last_fullsized":"2024-02-10T08:45:15.108931+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'24","last_scrub_stamp":"2024-02-10T08:44:40.723096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.723096+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431227+0000","last_change":"2024-02-10T08:44:36.570760+0000","last_active":"2024-02-10T08:45:19.431227+0000","last_peered":"2024-02-10T08:45:19.431227+0000","last_clean":"2024-02-10T08:45:19.431227+0000","last_became_active":"2024-02-10T08:43:15.132762+0000","last_became_peered":"2024-02-10T08:43:15.132762+0000","last_unstale":"2024-02-10T08:45:19.431227+0000","last_undegraded":"2024-02-10T08:45:19.431227+0000","last_fullsized":"2024-02-10T08:45:19.431227+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.570697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.570697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.119101+0000","last_change":"2024-02-10T08:43:36.072974+0000","last_active":"2024-02-10T08:45:14.119101+0000","last_peered":"2024-02-10T08:45:14.119101+0000","last_clean":"2024-02-10T08:33:17.081938+0000","last_became_active":"2024-02-10T08:43:36.072974+0000","last_became_peered":"2024-02-10T08:43:36.072974+0000","last_unstale":"2024-02-10T08:45:14.119101+0000","last_undegraded":"2024-02-10T08:45:14.119101+0000","last_fullsized":"2024-02-10T08:43:36.065921+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":548,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.109789+0000","last_change":"2024-02-10T08:44:38.734244+0000","last_active":"2024-02-10T08:45:15.109789+0000","last_peered":"2024-02-10T08:45:15.109789+0000","last_clean":"2024-02-10T08:45:15.109789+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:45:15.109789+0000","last_undegraded":"2024-02-10T08:45:15.109789+0000","last_fullsized":"2024-02-10T08:45:15.109789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:38.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.734157+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":500,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.110914+0000","last_change":"2024-02-10T08:44:23.462760+0000","last_active":"2024-02-10T08:45:19.110914+0000","last_peered":"2024-02-10T08:45:19.110914+0000","last_clean":"2024-02-10T08:45:19.110914+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:45:19.110914+0000","last_undegraded":"2024-02-10T08:45:19.110914+0000","last_fullsized":"2024-02-10T08:45:19.110914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:23.462674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:23.462674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":124,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.432346+0000","last_change":"2024-02-10T08:43:56.338478+0000","last_active":"2024-02-10T08:45:19.432346+0000","last_peered":"2024-02-10T08:45:19.432346+0000","last_clean":"2024-02-10T08:45:19.432346+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:45:19.432346+0000","last_undegraded":"2024-02-10T08:45:19.432346+0000","last_fullsized":"2024-02-10T08:45:19.432346+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:56.338378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:56.338378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":637,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.113452+0000","last_change":"2024-02-10T08:43:36.076013+0000","last_active":"2024-02-10T08:45:19.113452+0000","last_peered":"2024-02-10T08:45:19.113452+0000","last_clean":"2024-02-10T08:43:31.155643+0000","last_became_active":"2024-02-10T08:43:36.076013+0000","last_became_peered":"2024-02-10T08:43:36.076013+0000","last_unstale":"2024-02-10T08:45:19.113452+0000","last_undegraded":"2024-02-10T08:43:36.064979+0000","last_fullsized":"2024-02-10T08:43:36.064906+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431541+0000","last_change":"2024-02-10T08:44:24.600149+0000","last_active":"2024-02-10T08:45:19.431541+0000","last_peered":"2024-02-10T08:45:19.431541+0000","last_clean":"2024-02-10T08:45:19.431541+0000","last_became_active":"2024-02-10T08:43:15.135366+0000","last_became_peered":"2024-02-10T08:43:15.135366+0000","last_unstale":"2024-02-10T08:45:19.431541+0000","last_undegraded":"2024-02-10T08:45:19.431541+0000","last_fullsized":"2024-02-10T08:45:19.431541+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.600075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.600075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":564,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146765+0000","last_change":"2024-02-10T08:44:26.266252+0000","last_active":"2024-02-10T08:45:19.146765+0000","last_peered":"2024-02-10T08:45:19.146765+0000","last_clean":"2024-02-10T08:45:19.146765+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:45:19.146765+0000","last_undegraded":"2024-02-10T08:45:19.146765+0000","last_fullsized":"2024-02-10T08:45:19.146765+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.266165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.266165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":540,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.125291+0000","last_change":"2024-02-10T08:44:00.274641+0000","last_active":"2024-02-10T08:45:14.125291+0000","last_peered":"2024-02-10T08:45:14.125291+0000","last_clean":"2024-02-10T08:45:14.125291+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:45:14.125291+0000","last_undegraded":"2024-02-10T08:45:14.125291+0000","last_fullsized":"2024-02-10T08:45:14.125291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.274550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":541,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.147503+0000","last_change":"2024-02-10T08:43:36.077071+0000","last_active":"2024-02-10T08:45:19.147503+0000","last_peered":"2024-02-10T08:45:19.147503+0000","last_clean":"2024-02-10T08:43:31.062622+0000","last_became_active":"2024-02-10T08:43:36.077071+0000","last_became_peered":"2024-02-10T08:43:36.077071+0000","last_unstale":"2024-02-10T08:45:19.147503+0000","last_undegraded":"2024-02-10T08:45:19.147503+0000","last_fullsized":"2024-02-10T08:43:36.058617+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":500,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146820+0000","last_change":"2024-02-10T08:45:16.056076+0000","last_active":"2024-02-10T08:45:19.146820+0000","last_peered":"2024-02-10T08:45:19.146820+0000","last_clean":"2024-02-10T08:45:19.146820+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:45:19.146820+0000","last_undegraded":"2024-02-10T08:45:19.146820+0000","last_fullsized":"2024-02-10T08:45:19.146820+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:16.055988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:16.055988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":574,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.106415+0000","last_change":"2024-02-10T08:43:32.421461+0000","last_active":"2024-02-10T08:45:15.106415+0000","last_peered":"2024-02-10T08:45:15.106415+0000","last_clean":"2024-02-10T08:45:15.106415+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:45:15.106415+0000","last_undegraded":"2024-02-10T08:45:15.106415+0000","last_fullsized":"2024-02-10T08:45:15.106415+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:32.421384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:32.421384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":647,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.108841+0000","last_change":"2024-02-10T08:45:07.684836+0000","last_active":"2024-02-10T08:45:15.108841+0000","last_peered":"2024-02-10T08:45:15.108841+0000","last_clean":"2024-02-10T08:45:15.108841+0000","last_became_active":"2024-02-10T08:43:15.129755+0000","last_became_peered":"2024-02-10T08:43:15.129755+0000","last_unstale":"2024-02-10T08:45:15.108841+0000","last_undegraded":"2024-02-10T08:45:15.108841+0000","last_fullsized":"2024-02-10T08:45:15.108841+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:45:07.684765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.684765+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":558,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104891+0000","last_change":"2024-02-10T08:44:37.469860+0000","last_active":"2024-02-10T08:45:15.104891+0000","last_peered":"2024-02-10T08:45:15.104891+0000","last_clean":"2024-02-10T08:45:15.104891+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:45:15.104891+0000","last_undegraded":"2024-02-10T08:45:15.104891+0000","last_fullsized":"2024-02-10T08:45:15.104891+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:37.469796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.469796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":1262,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.460171+0000","last_change":"2024-02-10T08:43:58.735413+0000","last_active":"2024-02-10T08:45:15.460171+0000","last_peered":"2024-02-10T08:45:15.460171+0000","last_clean":"2024-02-10T08:45:15.460171+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:45:15.460171+0000","last_undegraded":"2024-02-10T08:45:15.460171+0000","last_fullsized":"2024-02-10T08:45:15.460171+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:43:58.735346+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.735346+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":710,"num_read_kb":705,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431965+0000","last_change":"2024-02-10T08:45:09.454368+0000","last_active":"2024-02-10T08:45:19.431965+0000","last_peered":"2024-02-10T08:45:19.431965+0000","last_clean":"2024-02-10T08:45:19.431965+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:45:19.431965+0000","last_undegraded":"2024-02-10T08:45:19.431965+0000","last_fullsized":"2024-02-10T08:45:19.431965+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.454293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.454293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":518,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.107970+0000","last_change":"2024-02-10T08:44:36.777644+0000","last_active":"2024-02-10T08:45:15.107970+0000","last_peered":"2024-02-10T08:45:15.107970+0000","last_clean":"2024-02-10T08:45:15.107970+0000","last_became_active":"2024-02-10T08:43:15.129900+0000","last_became_peered":"2024-02-10T08:43:15.129900+0000","last_unstale":"2024-02-10T08:45:15.107970+0000","last_undegraded":"2024-02-10T08:45:15.107970+0000","last_fullsized":"2024-02-10T08:45:15.107970+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.777533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":462,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.108312+0000","last_change":"2024-02-10T08:43:36.073644+0000","last_active":"2024-02-10T08:45:15.108312+0000","last_peered":"2024-02-10T08:45:15.108312+0000","last_clean":"2024-02-10T08:43:31.058736+0000","last_became_active":"2024-02-10T08:43:36.073644+0000","last_became_peered":"2024-02-10T08:43:36.073644+0000","last_unstale":"2024-02-10T08:45:15.108312+0000","last_undegraded":"2024-02-10T08:45:15.108312+0000","last_fullsized":"2024-02-10T08:43:36.062919+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"473'41","reported_seq":693,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.147475+0000","last_change":"2024-02-10T08:42:43.109193+0000","last_active":"2024-02-10T08:45:19.147475+0000","last_peered":"2024-02-10T08:45:19.147475+0000","last_clean":"2024-02-10T08:45:19.147475+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:45:19.147475+0000","last_undegraded":"2024-02-10T08:45:19.147475+0000","last_fullsized":"2024-02-10T08:45:19.147475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:42:43.109121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.109121+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":546,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148581+0000","last_change":"2024-02-10T08:42:37.145252+0000","last_active":"2024-02-10T08:45:19.148581+0000","last_peered":"2024-02-10T08:45:19.148581+0000","last_clean":"2024-02-10T08:45:19.148581+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:45:19.148581+0000","last_undegraded":"2024-02-10T08:45:19.148581+0000","last_fullsized":"2024-02-10T08:45:19.148581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.145180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.145180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":553,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.120828+0000","last_change":"2024-02-10T08:43:36.083021+0000","last_active":"2024-02-10T08:45:14.120828+0000","last_peered":"2024-02-10T08:45:14.120828+0000","last_clean":"2024-02-10T08:43:31.058994+0000","last_became_active":"2024-02-10T08:43:36.083021+0000","last_became_peered":"2024-02-10T08:43:36.083021+0000","last_unstale":"2024-02-10T08:45:14.120828+0000","last_undegraded":"2024-02-10T08:45:14.120828+0000","last_fullsized":"2024-02-10T08:43:36.072420+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":517,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104428+0000","last_change":"2024-02-10T08:44:30.426450+0000","last_active":"2024-02-10T08:45:15.104428+0000","last_peered":"2024-02-10T08:45:15.104428+0000","last_clean":"2024-02-10T08:45:15.104428+0000","last_became_active":"2024-02-10T08:43:15.119340+0000","last_became_peered":"2024-02-10T08:43:15.119340+0000","last_unstale":"2024-02-10T08:45:15.104428+0000","last_undegraded":"2024-02-10T08:45:15.104428+0000","last_fullsized":"2024-02-10T08:45:15.104428+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:30.426345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.426345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":587,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.111424+0000","last_change":"2024-02-10T08:44:39.424567+0000","last_active":"2024-02-10T08:45:19.111424+0000","last_peered":"2024-02-10T08:45:19.111424+0000","last_clean":"2024-02-10T08:45:19.111424+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:45:19.111424+0000","last_undegraded":"2024-02-10T08:45:19.111424+0000","last_fullsized":"2024-02-10T08:45:19.111424+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:44:39.424472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.424472+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":589,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.107443+0000","last_change":"2024-02-10T08:43:36.071773+0000","last_active":"2024-02-10T08:45:15.107443+0000","last_peered":"2024-02-10T08:45:15.107443+0000","last_clean":"2024-02-10T08:43:31.056280+0000","last_became_active":"2024-02-10T08:43:36.071773+0000","last_became_peered":"2024-02-10T08:43:36.071773+0000","last_unstale":"2024-02-10T08:45:15.107443+0000","last_undegraded":"2024-02-10T08:43:36.059969+0000","last_fullsized":"2024-02-10T08:43:36.059851+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:42:40.687013+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.687013+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":580,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.109388+0000","last_change":"2024-02-10T08:45:10.601821+0000","last_active":"2024-02-10T08:45:15.109388+0000","last_peered":"2024-02-10T08:45:15.109388+0000","last_clean":"2024-02-10T08:45:15.109388+0000","last_became_active":"2024-02-10T08:43:15.129787+0000","last_became_peered":"2024-02-10T08:43:15.129787+0000","last_unstale":"2024-02-10T08:45:15.109388+0000","last_undegraded":"2024-02-10T08:45:15.109388+0000","last_fullsized":"2024-02-10T08:45:15.109388+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.601758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.601758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":454,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.118335+0000","last_change":"2024-02-10T08:42:40.143847+0000","last_active":"2024-02-10T08:45:14.118335+0000","last_peered":"2024-02-10T08:45:14.118335+0000","last_clean":"2024-02-10T08:45:14.118335+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:45:14.118335+0000","last_undegraded":"2024-02-10T08:45:14.118335+0000","last_fullsized":"2024-02-10T08:45:14.118335+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.143787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.143787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":484,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.120496+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:45:14.120496+0000","last_peered":"2024-02-10T08:45:14.120496+0000","last_clean":"2024-02-10T08:45:14.120496+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:45:14.120496+0000","last_undegraded":"2024-02-10T08:45:14.120496+0000","last_fullsized":"2024-02-10T08:45:14.120496+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":570,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104598+0000","last_change":"2024-02-10T08:43:58.505496+0000","last_active":"2024-02-10T08:45:15.104598+0000","last_peered":"2024-02-10T08:45:15.104598+0000","last_clean":"2024-02-10T08:45:15.104598+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:45:15.104598+0000","last_undegraded":"2024-02-10T08:45:15.104598+0000","last_fullsized":"2024-02-10T08:45:15.104598+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.505431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.505431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":576,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.106443+0000","last_change":"2024-02-10T08:43:36.079165+0000","last_active":"2024-02-10T08:45:15.106443+0000","last_peered":"2024-02-10T08:45:15.106443+0000","last_clean":"2024-02-10T08:43:25.579489+0000","last_became_active":"2024-02-10T08:43:36.079165+0000","last_became_peered":"2024-02-10T08:43:36.079165+0000","last_unstale":"2024-02-10T08:45:15.106443+0000","last_undegraded":"2024-02-10T08:43:36.061039+0000","last_fullsized":"2024-02-10T08:43:36.060910+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":578,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.432196+0000","last_change":"2024-02-10T08:43:36.072147+0000","last_active":"2024-02-10T08:45:19.432196+0000","last_peered":"2024-02-10T08:45:19.432196+0000","last_clean":"2024-02-10T08:43:11.810746+0000","last_became_active":"2024-02-10T08:43:36.072147+0000","last_became_peered":"2024-02-10T08:43:36.072147+0000","last_unstale":"2024-02-10T08:45:19.432196+0000","last_undegraded":"2024-02-10T08:43:36.061852+0000","last_fullsized":"2024-02-10T08:43:36.061736+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:43:11.810639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.810639+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":552,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.139368+0000","last_change":"2024-02-10T08:43:15.126259+0000","last_active":"2024-02-10T08:45:14.139368+0000","last_peered":"2024-02-10T08:45:14.139368+0000","last_clean":"2024-02-10T08:45:14.139368+0000","last_became_active":"2024-02-10T08:43:15.125756+0000","last_became_peered":"2024-02-10T08:43:15.125756+0000","last_unstale":"2024-02-10T08:45:14.139368+0000","last_undegraded":"2024-02-10T08:45:14.139368+0000","last_fullsized":"2024-02-10T08:45:14.139368+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.067409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.067409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":508,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.105890+0000","last_change":"2024-02-10T08:43:54.509761+0000","last_active":"2024-02-10T08:45:15.105890+0000","last_peered":"2024-02-10T08:45:15.105890+0000","last_clean":"2024-02-10T08:45:15.105890+0000","last_became_active":"2024-02-10T08:43:15.119419+0000","last_became_peered":"2024-02-10T08:43:15.119419+0000","last_unstale":"2024-02-10T08:45:15.105890+0000","last_undegraded":"2024-02-10T08:45:15.105890+0000","last_fullsized":"2024-02-10T08:45:15.105890+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:54.509649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:54.509649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":572,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.120951+0000","last_change":"2024-02-10T08:44:38.409588+0000","last_active":"2024-02-10T08:45:14.120951+0000","last_peered":"2024-02-10T08:45:14.120951+0000","last_clean":"2024-02-10T08:45:14.120951+0000","last_became_active":"2024-02-10T08:43:15.137834+0000","last_became_peered":"2024-02-10T08:43:15.137834+0000","last_unstale":"2024-02-10T08:45:14.120951+0000","last_undegraded":"2024-02-10T08:45:14.120951+0000","last_fullsized":"2024-02-10T08:45:14.120951+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.409486+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":557,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.104526+0000","last_change":"2024-02-10T08:43:36.075991+0000","last_active":"2024-02-10T08:45:15.104526+0000","last_peered":"2024-02-10T08:45:15.104526+0000","last_clean":"2024-02-10T08:43:31.065892+0000","last_became_active":"2024-02-10T08:43:36.075991+0000","last_became_peered":"2024-02-10T08:43:36.075991+0000","last_unstale":"2024-02-10T08:45:15.104526+0000","last_undegraded":"2024-02-10T08:45:15.104526+0000","last_fullsized":"2024-02-10T08:43:36.062400+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":560,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.105205+0000","last_change":"2024-02-10T08:43:36.075699+0000","last_active":"2024-02-10T08:45:15.105205+0000","last_peered":"2024-02-10T08:45:15.105205+0000","last_clean":"2024-02-10T08:43:31.066554+0000","last_became_active":"2024-02-10T08:43:36.075699+0000","last_became_peered":"2024-02-10T08:43:36.075699+0000","last_unstale":"2024-02-10T08:45:15.105205+0000","last_undegraded":"2024-02-10T08:45:15.105205+0000","last_fullsized":"2024-02-10T08:43:36.063912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.347685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.347685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":486,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.106723+0000","last_change":"2024-02-10T08:43:36.075546+0000","last_active":"2024-02-10T08:45:15.106723+0000","last_peered":"2024-02-10T08:45:15.106723+0000","last_clean":"2024-02-10T08:43:11.068374+0000","last_became_active":"2024-02-10T08:43:36.075546+0000","last_became_peered":"2024-02-10T08:43:36.075546+0000","last_unstale":"2024-02-10T08:45:15.106723+0000","last_undegraded":"2024-02-10T08:45:15.106723+0000","last_fullsized":"2024-02-10T08:43:36.057136+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":590,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.126533+0000","last_change":"2024-02-10T08:43:52.173893+0000","last_active":"2024-02-10T08:45:14.126533+0000","last_peered":"2024-02-10T08:45:14.126533+0000","last_clean":"2024-02-10T08:45:14.126533+0000","last_became_active":"2024-02-10T08:43:15.125753+0000","last_became_peered":"2024-02-10T08:43:15.125753+0000","last_unstale":"2024-02-10T08:45:14.126533+0000","last_undegraded":"2024-02-10T08:45:14.126533+0000","last_fullsized":"2024-02-10T08:45:14.126533+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:43:52.173797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:52.173797+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":549,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.111080+0000","last_change":"2024-02-10T08:43:36.068896+0000","last_active":"2024-02-10T08:45:19.111080+0000","last_peered":"2024-02-10T08:45:19.111080+0000","last_clean":"2024-02-10T08:43:31.156388+0000","last_became_active":"2024-02-10T08:43:36.068896+0000","last_became_peered":"2024-02-10T08:43:36.068896+0000","last_unstale":"2024-02-10T08:45:19.111080+0000","last_undegraded":"2024-02-10T08:45:19.111080+0000","last_fullsized":"2024-02-10T08:43:36.062533+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.431741+0000","last_change":"2024-02-10T08:43:36.074707+0000","last_active":"2024-02-10T08:45:19.431741+0000","last_peered":"2024-02-10T08:45:19.431741+0000","last_clean":"2024-02-10T08:43:31.060201+0000","last_became_active":"2024-02-10T08:43:36.074707+0000","last_became_peered":"2024-02-10T08:43:36.074707+0000","last_unstale":"2024-02-10T08:45:19.431741+0000","last_undegraded":"2024-02-10T08:45:19.431741+0000","last_fullsized":"2024-02-10T08:43:36.061288+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":490,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.146700+0000","last_change":"2024-02-10T08:43:36.065698+0000","last_active":"2024-02-10T08:45:19.146700+0000","last_peered":"2024-02-10T08:45:19.146700+0000","last_clean":"2024-02-10T08:43:31.061677+0000","last_became_active":"2024-02-10T08:43:36.065698+0000","last_became_peered":"2024-02-10T08:43:36.065698+0000","last_unstale":"2024-02-10T08:45:19.146700+0000","last_undegraded":"2024-02-10T08:45:19.146700+0000","last_fullsized":"2024-02-10T08:43:36.058144+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":610,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.121171+0000","last_change":"2024-02-10T08:43:19.438856+0000","last_active":"2024-02-10T08:45:14.121171+0000","last_peered":"2024-02-10T08:45:14.121171+0000","last_clean":"2024-02-10T08:45:14.121171+0000","last_became_active":"2024-02-10T08:43:15.137251+0000","last_became_peered":"2024-02-10T08:43:15.137251+0000","last_unstale":"2024-02-10T08:45:14.121171+0000","last_undegraded":"2024-02-10T08:45:14.121171+0000","last_fullsized":"2024-02-10T08:45:14.121171+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":564,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148512+0000","last_change":"2024-02-10T08:43:15.112162+0000","last_active":"2024-02-10T08:45:19.148512+0000","last_peered":"2024-02-10T08:45:19.148512+0000","last_clean":"2024-02-10T08:45:19.148512+0000","last_became_active":"2024-02-10T08:43:15.111824+0000","last_became_peered":"2024-02-10T08:43:15.111824+0000","last_unstale":"2024-02-10T08:45:19.148512+0000","last_undegraded":"2024-02-10T08:45:19.148512+0000","last_fullsized":"2024-02-10T08:45:19.148512+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:42.059708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.059708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":707,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-02-10T08:45:21.041083+0000","last_change":"2024-02-10T08:45:21.041083+0000","last_active":"2024-02-10T08:45:21.041083+0000","last_peered":"2024-02-10T08:45:21.041083+0000","last_clean":"2024-02-10T08:45:21.041083+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:45:21.041083+0000","last_undegraded":"2024-02-10T08:45:21.041083+0000","last_fullsized":"2024-02-10T08:45:21.041083+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:45:21.040990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:21.040990+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":128,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.432481+0000","last_change":"2024-02-10T08:43:36.078925+0000","last_active":"2024-02-10T08:45:19.432481+0000","last_peered":"2024-02-10T08:45:19.432481+0000","last_clean":"2024-02-10T08:43:31.060910+0000","last_became_active":"2024-02-10T08:43:36.078925+0000","last_became_peered":"2024-02-10T08:43:36.078925+0000","last_unstale":"2024-02-10T08:45:19.432481+0000","last_undegraded":"2024-02-10T08:45:19.432481+0000","last_fullsized":"2024-02-10T08:43:36.063718+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.241405+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":462,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.110842+0000","last_change":"2024-02-10T08:43:36.067846+0000","last_active":"2024-02-10T08:45:19.110842+0000","last_peered":"2024-02-10T08:45:19.110842+0000","last_clean":"2024-02-10T08:43:31.156305+0000","last_became_active":"2024-02-10T08:43:36.067846+0000","last_became_peered":"2024-02-10T08:43:36.067846+0000","last_unstale":"2024-02-10T08:45:19.110842+0000","last_undegraded":"2024-02-10T08:45:19.110842+0000","last_fullsized":"2024-02-10T08:43:36.063242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.357151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.357151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":607,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.148468+0000","last_change":"2024-02-10T08:45:13.165679+0000","last_active":"2024-02-10T08:45:19.148468+0000","last_peered":"2024-02-10T08:45:19.148468+0000","last_clean":"2024-02-10T08:45:19.148468+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:45:19.148468+0000","last_undegraded":"2024-02-10T08:45:19.148468+0000","last_fullsized":"2024-02-10T08:45:19.148468+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:45:13.165571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.165571+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":61,"reported_epoch":569,"state":"active+clean+remapped","last_fresh":"2024-02-10T08:45:15.104555+0000","last_change":"2024-02-10T08:45:14.450083+0000","last_active":"2024-02-10T08:45:15.104555+0000","last_peered":"2024-02-10T08:45:15.104555+0000","last_clean":"2024-02-10T08:45:15.104555+0000","last_became_active":"2024-02-10T08:44:31.113168+0000","last_became_peered":"2024-02-10T08:44:31.113168+0000","last_unstale":"2024-02-10T08:45:15.104555+0000","last_undegraded":"2024-02-10T08:45:15.104555+0000","last_fullsized":"2024-02-10T08:45:15.104555+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.449994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.449994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":714,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.121376+0000","last_change":"2024-02-10T08:43:19.433673+0000","last_active":"2024-02-10T08:45:14.121376+0000","last_peered":"2024-02-10T08:45:14.121376+0000","last_clean":"2024-02-10T08:45:14.121376+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:45:14.121376+0000","last_undegraded":"2024-02-10T08:45:14.121376+0000","last_fullsized":"2024-02-10T08:45:14.121376+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":97,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.107367+0000","last_change":"2024-02-10T08:43:36.075017+0000","last_active":"2024-02-10T08:45:15.107367+0000","last_peered":"2024-02-10T08:45:15.107367+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.075017+0000","last_became_peered":"2024-02-10T08:43:36.075017+0000","last_unstale":"2024-02-10T08:45:15.107367+0000","last_undegraded":"2024-02-10T08:45:15.107367+0000","last_fullsized":"2024-02-10T08:43:36.058808+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.786130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.786130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":503,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104860+0000","last_change":"2024-02-10T08:45:09.467738+0000","last_active":"2024-02-10T08:45:15.104860+0000","last_peered":"2024-02-10T08:45:15.104860+0000","last_clean":"2024-02-10T08:45:15.104860+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:45:15.104860+0000","last_undegraded":"2024-02-10T08:45:15.104860+0000","last_fullsized":"2024-02-10T08:45:15.104860+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:45:09.467617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.467617+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":577,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.107332+0000","last_change":"2024-02-10T08:44:30.727741+0000","last_active":"2024-02-10T08:45:15.107332+0000","last_peered":"2024-02-10T08:45:15.107332+0000","last_clean":"2024-02-10T08:45:15.107332+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:45:15.107332+0000","last_undegraded":"2024-02-10T08:45:15.107332+0000","last_fullsized":"2024-02-10T08:45:15.107332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:30.727654+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.727654+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":553,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.112866+0000","last_change":"2024-02-10T08:45:14.481063+0000","last_active":"2024-02-10T08:45:19.112866+0000","last_peered":"2024-02-10T08:45:19.112866+0000","last_clean":"2024-02-10T08:45:19.112866+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:45:19.112866+0000","last_undegraded":"2024-02-10T08:45:19.112866+0000","last_fullsized":"2024-02-10T08:45:19.112866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.480995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.480995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":550,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.139279+0000","last_change":"2024-02-10T08:45:07.459730+0000","last_active":"2024-02-10T08:45:14.139279+0000","last_peered":"2024-02-10T08:45:14.139279+0000","last_clean":"2024-02-10T08:45:14.139279+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:45:14.139279+0000","last_undegraded":"2024-02-10T08:45:14.139279+0000","last_fullsized":"2024-02-10T08:45:14.139279+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:07.459661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.459661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":459,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.146322+0000","last_change":"2024-02-10T08:43:59.469912+0000","last_active":"2024-02-10T08:45:19.146322+0000","last_peered":"2024-02-10T08:45:19.146322+0000","last_clean":"2024-02-10T08:45:19.146322+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:45:19.146322+0000","last_undegraded":"2024-02-10T08:45:19.146322+0000","last_fullsized":"2024-02-10T08:45:19.146322+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.469850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.469850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":502,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.114581+0000","last_change":"2024-02-10T08:44:25.447359+0000","last_active":"2024-02-10T08:45:19.114581+0000","last_peered":"2024-02-10T08:45:19.114581+0000","last_clean":"2024-02-10T08:45:19.114581+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:45:19.114581+0000","last_undegraded":"2024-02-10T08:45:19.114581+0000","last_fullsized":"2024-02-10T08:45:19.114581+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.447293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.447293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":563,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.111447+0000","last_change":"2024-02-10T08:43:36.069761+0000","last_active":"2024-02-10T08:45:19.111447+0000","last_peered":"2024-02-10T08:45:19.111447+0000","last_clean":"2024-02-10T08:43:25.738370+0000","last_became_active":"2024-02-10T08:43:36.069761+0000","last_became_peered":"2024-02-10T08:43:36.069761+0000","last_unstale":"2024-02-10T08:45:19.111447+0000","last_undegraded":"2024-02-10T08:43:36.063371+0000","last_fullsized":"2024-02-10T08:43:36.063247+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:43:28.880851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:28.880851+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":559,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.108990+0000","last_change":"2024-02-10T08:43:59.760782+0000","last_active":"2024-02-10T08:45:15.108990+0000","last_peered":"2024-02-10T08:45:15.108990+0000","last_clean":"2024-02-10T08:45:15.108990+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:45:15.108990+0000","last_undegraded":"2024-02-10T08:45:15.108990+0000","last_fullsized":"2024-02-10T08:45:15.108990+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.760721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.760721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":544,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.107379+0000","last_change":"2024-02-10T08:44:35.784832+0000","last_active":"2024-02-10T08:45:15.107379+0000","last_peered":"2024-02-10T08:45:15.107379+0000","last_clean":"2024-02-10T08:45:15.107379+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:45:15.107379+0000","last_undegraded":"2024-02-10T08:45:15.107379+0000","last_fullsized":"2024-02-10T08:45:15.107379+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.784716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":543,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.118791+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:45:14.118791+0000","last_peered":"2024-02-10T08:45:14.118791+0000","last_clean":"2024-02-10T08:45:14.118791+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:45:14.118791+0000","last_undegraded":"2024-02-10T08:45:14.118791+0000","last_fullsized":"2024-02-10T08:45:14.118791+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":499,"reported_epoch":568,"state":"active+undersized","last_fresh":"2024-02-10T08:45:14.126291+0000","last_change":"2024-02-10T08:43:36.081224+0000","last_active":"2024-02-10T08:45:14.126291+0000","last_peered":"2024-02-10T08:45:14.126291+0000","last_clean":"2024-02-10T08:43:31.059324+0000","last_became_active":"2024-02-10T08:43:36.081224+0000","last_became_peered":"2024-02-10T08:43:36.081224+0000","last_unstale":"2024-02-10T08:45:14.126291+0000","last_undegraded":"2024-02-10T08:45:14.126291+0000","last_fullsized":"2024-02-10T08:43:36.068534+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":550,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.118248+0000","last_change":"2024-02-10T08:43:34.096276+0000","last_active":"2024-02-10T08:45:14.118248+0000","last_peered":"2024-02-10T08:45:14.118248+0000","last_clean":"2024-02-10T08:45:14.118248+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:45:14.118248+0000","last_undegraded":"2024-02-10T08:45:14.118248+0000","last_fullsized":"2024-02-10T08:45:14.118248+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:43:34.096208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.096208+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":550,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.147994+0000","last_change":"2024-02-10T08:43:23.521301+0000","last_active":"2024-02-10T08:45:19.147994+0000","last_peered":"2024-02-10T08:45:19.147994+0000","last_clean":"2024-02-10T08:45:19.147994+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:45:19.147994+0000","last_undegraded":"2024-02-10T08:45:19.147994+0000","last_fullsized":"2024-02-10T08:45:19.147994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.521227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.521227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":565,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.111038+0000","last_change":"2024-02-10T08:43:36.072045+0000","last_active":"2024-02-10T08:45:19.111038+0000","last_peered":"2024-02-10T08:45:19.111038+0000","last_clean":"2024-02-10T08:43:31.155301+0000","last_became_active":"2024-02-10T08:43:36.072045+0000","last_became_peered":"2024-02-10T08:43:36.072045+0000","last_unstale":"2024-02-10T08:45:19.111038+0000","last_undegraded":"2024-02-10T08:45:19.111038+0000","last_fullsized":"2024-02-10T08:43:36.062267+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.508636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.508636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.431323+0000","last_change":"2024-02-10T08:43:36.072813+0000","last_active":"2024-02-10T08:45:19.431323+0000","last_peered":"2024-02-10T08:45:19.431323+0000","last_clean":"2024-02-10T08:43:31.060566+0000","last_became_active":"2024-02-10T08:43:36.072813+0000","last_became_peered":"2024-02-10T08:43:36.072813+0000","last_unstale":"2024-02-10T08:45:19.431323+0000","last_undegraded":"2024-02-10T08:45:19.431323+0000","last_fullsized":"2024-02-10T08:43:36.061933+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":502,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.120959+0000","last_change":"2024-02-10T08:43:15.137758+0000","last_active":"2024-02-10T08:45:14.120959+0000","last_peered":"2024-02-10T08:45:14.120959+0000","last_clean":"2024-02-10T08:45:14.120959+0000","last_became_active":"2024-02-10T08:43:15.137437+0000","last_became_peered":"2024-02-10T08:43:15.137437+0000","last_unstale":"2024-02-10T08:45:14.120959+0000","last_undegraded":"2024-02-10T08:45:14.120959+0000","last_fullsized":"2024-02-10T08:45:14.120959+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.116858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":611,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.119338+0000","last_change":"2024-02-10T08:44:28.331732+0000","last_active":"2024-02-10T08:45:14.119338+0000","last_peered":"2024-02-10T08:45:14.119338+0000","last_clean":"2024-02-10T08:45:14.119338+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:45:14.119338+0000","last_undegraded":"2024-02-10T08:45:14.119338+0000","last_fullsized":"2024-02-10T08:45:14.119338+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:28.331601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.331601+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":553,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.113176+0000","last_change":"2024-02-10T08:45:12.464783+0000","last_active":"2024-02-10T08:45:19.113176+0000","last_peered":"2024-02-10T08:45:19.113176+0000","last_clean":"2024-02-10T08:45:19.113176+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:45:19.113176+0000","last_undegraded":"2024-02-10T08:45:19.113176+0000","last_fullsized":"2024-02-10T08:45:19.113176+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:12.464721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:12.464721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.431333+0000","last_change":"2024-02-10T08:43:36.074002+0000","last_active":"2024-02-10T08:45:19.431333+0000","last_peered":"2024-02-10T08:45:19.431333+0000","last_clean":"2024-02-10T08:43:31.060062+0000","last_became_active":"2024-02-10T08:43:36.074002+0000","last_became_peered":"2024-02-10T08:43:36.074002+0000","last_unstale":"2024-02-10T08:45:19.431333+0000","last_undegraded":"2024-02-10T08:45:19.431333+0000","last_fullsized":"2024-02-10T08:43:36.060407+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.341979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":543,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.139441+0000","last_change":"2024-02-10T08:42:43.111366+0000","last_active":"2024-02-10T08:45:14.139441+0000","last_peered":"2024-02-10T08:45:14.139441+0000","last_clean":"2024-02-10T08:45:14.139441+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:45:14.139441+0000","last_undegraded":"2024-02-10T08:45:14.139441+0000","last_fullsized":"2024-02-10T08:45:14.139441+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.111309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.111309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":461,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.113733+0000","last_change":"2024-02-10T08:44:02.384309+0000","last_active":"2024-02-10T08:45:19.113733+0000","last_peered":"2024-02-10T08:45:19.113733+0000","last_clean":"2024-02-10T08:45:19.113733+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:45:19.113733+0000","last_undegraded":"2024-02-10T08:45:19.113733+0000","last_fullsized":"2024-02-10T08:45:19.113733+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:02.384207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:02.384207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":149,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.432614+0000","last_change":"2024-02-10T08:44:25.556111+0000","last_active":"2024-02-10T08:45:19.432614+0000","last_peered":"2024-02-10T08:45:19.432614+0000","last_clean":"2024-02-10T08:45:19.432614+0000","last_became_active":"2024-02-10T08:43:15.139755+0000","last_became_peered":"2024-02-10T08:43:15.139755+0000","last_unstale":"2024-02-10T08:45:19.432614+0000","last_undegraded":"2024-02-10T08:45:19.432614+0000","last_fullsized":"2024-02-10T08:45:19.432614+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:44:25.556047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.556047+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":579,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.146296+0000","last_change":"2024-02-10T08:43:36.066663+0000","last_active":"2024-02-10T08:45:19.146296+0000","last_peered":"2024-02-10T08:45:19.146296+0000","last_clean":"2024-02-10T08:43:31.060592+0000","last_became_active":"2024-02-10T08:43:36.066663+0000","last_became_peered":"2024-02-10T08:43:36.066663+0000","last_unstale":"2024-02-10T08:45:19.146296+0000","last_undegraded":"2024-02-10T08:45:19.146296+0000","last_fullsized":"2024-02-10T08:43:36.056257+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.131457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.131457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":126,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.432075+0000","last_change":"2024-02-10T08:44:34.562436+0000","last_active":"2024-02-10T08:45:19.432075+0000","last_peered":"2024-02-10T08:45:19.432075+0000","last_clean":"2024-02-10T08:45:19.432075+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:45:19.432075+0000","last_undegraded":"2024-02-10T08:45:19.432075+0000","last_fullsized":"2024-02-10T08:45:19.432075+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:34.562370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:34.562370+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":508,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.104759+0000","last_change":"2024-02-10T08:44:29.460927+0000","last_active":"2024-02-10T08:45:15.104759+0000","last_peered":"2024-02-10T08:45:15.104759+0000","last_clean":"2024-02-10T08:45:15.104759+0000","last_became_active":"2024-02-10T08:43:15.119127+0000","last_became_peered":"2024-02-10T08:43:15.119127+0000","last_unstale":"2024-02-10T08:45:15.104759+0000","last_undegraded":"2024-02-10T08:45:15.104759+0000","last_fullsized":"2024-02-10T08:45:15.104759+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.460863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.460863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":582,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.113263+0000","last_change":"2024-02-10T08:44:41.392510+0000","last_active":"2024-02-10T08:45:19.113263+0000","last_peered":"2024-02-10T08:45:19.113263+0000","last_clean":"2024-02-10T08:45:19.113263+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:45:19.113263+0000","last_undegraded":"2024-02-10T08:45:19.113263+0000","last_fullsized":"2024-02-10T08:45:19.113263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:41.392413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:41.392413+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":549,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.118971+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:45:14.118971+0000","last_peered":"2024-02-10T08:45:14.118971+0000","last_clean":"2024-02-10T08:45:14.118971+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:45:14.118971+0000","last_undegraded":"2024-02-10T08:45:14.118971+0000","last_fullsized":"2024-02-10T08:45:14.118971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":543,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.108813+0000","last_change":"2024-02-10T08:43:36.073017+0000","last_active":"2024-02-10T08:45:15.108813+0000","last_peered":"2024-02-10T08:45:15.108813+0000","last_clean":"2024-02-10T08:43:31.058620+0000","last_became_active":"2024-02-10T08:43:36.073017+0000","last_became_peered":"2024-02-10T08:43:36.073017+0000","last_unstale":"2024-02-10T08:45:15.108813+0000","last_undegraded":"2024-02-10T08:45:15.108813+0000","last_fullsized":"2024-02-10T08:43:36.062639+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":492,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.107943+0000","last_change":"2024-02-10T08:43:36.085258+0000","last_active":"2024-02-10T08:45:15.107943+0000","last_peered":"2024-02-10T08:45:15.107943+0000","last_clean":"2024-02-10T08:43:31.059986+0000","last_became_active":"2024-02-10T08:43:36.085258+0000","last_became_peered":"2024-02-10T08:43:36.085258+0000","last_unstale":"2024-02-10T08:45:15.107943+0000","last_undegraded":"2024-02-10T08:45:15.107943+0000","last_fullsized":"2024-02-10T08:43:36.058326+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":620,"reported_epoch":569,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:15.109477+0000","last_change":"2024-02-10T08:43:36.088112+0000","last_active":"2024-02-10T08:45:15.109477+0000","last_peered":"2024-02-10T08:45:15.109477+0000","last_clean":"2024-02-10T08:43:31.060512+0000","last_became_active":"2024-02-10T08:43:36.088112+0000","last_became_peered":"2024-02-10T08:43:36.088112+0000","last_unstale":"2024-02-10T08:45:15.109477+0000","last_undegraded":"2024-02-10T08:43:36.063487+0000","last_fullsized":"2024-02-10T08:43:36.063380+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":567,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.111442+0000","last_change":"2024-02-10T08:43:36.078681+0000","last_active":"2024-02-10T08:45:19.111442+0000","last_peered":"2024-02-10T08:45:19.111442+0000","last_clean":"2024-02-10T08:43:31.155537+0000","last_became_active":"2024-02-10T08:43:36.078681+0000","last_became_peered":"2024-02-10T08:43:36.078681+0000","last_unstale":"2024-02-10T08:45:19.111442+0000","last_undegraded":"2024-02-10T08:45:19.111442+0000","last_fullsized":"2024-02-10T08:43:36.069413+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":96,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.114218+0000","last_change":"2024-02-10T08:43:36.071762+0000","last_active":"2024-02-10T08:45:19.114218+0000","last_peered":"2024-02-10T08:45:19.114218+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.071762+0000","last_became_peered":"2024-02-10T08:43:36.071762+0000","last_unstale":"2024-02-10T08:45:19.114218+0000","last_undegraded":"2024-02-10T08:45:19.114218+0000","last_fullsized":"2024-02-10T08:43:36.062825+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.031375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.031375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":466,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.110807+0000","last_change":"2024-02-10T08:45:10.423028+0000","last_active":"2024-02-10T08:45:19.110807+0000","last_peered":"2024-02-10T08:45:19.110807+0000","last_clean":"2024-02-10T08:45:19.110807+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:45:19.110807+0000","last_undegraded":"2024-02-10T08:45:19.110807+0000","last_fullsized":"2024-02-10T08:45:19.110807+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.422960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.422960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":597,"reported_epoch":568,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:14.118834+0000","last_change":"2024-02-10T08:43:36.081271+0000","last_active":"2024-02-10T08:45:14.118834+0000","last_peered":"2024-02-10T08:45:14.118834+0000","last_clean":"2024-02-10T08:43:31.058844+0000","last_became_active":"2024-02-10T08:43:36.081271+0000","last_became_peered":"2024-02-10T08:43:36.081271+0000","last_unstale":"2024-02-10T08:45:14.118834+0000","last_undegraded":"2024-02-10T08:43:36.068229+0000","last_fullsized":"2024-02-10T08:43:36.068095+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":563,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.127874+0000","last_change":"2024-02-10T08:45:11.455867+0000","last_active":"2024-02-10T08:45:14.127874+0000","last_peered":"2024-02-10T08:45:14.127874+0000","last_clean":"2024-02-10T08:45:14.127874+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:45:14.127874+0000","last_undegraded":"2024-02-10T08:45:14.127874+0000","last_fullsized":"2024-02-10T08:45:14.127874+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:11.455805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:11.455805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":534,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.126609+0000","last_change":"2024-02-10T08:42:37.092622+0000","last_active":"2024-02-10T08:45:14.126609+0000","last_peered":"2024-02-10T08:45:14.126609+0000","last_clean":"2024-02-10T08:45:14.126609+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:45:14.126609+0000","last_undegraded":"2024-02-10T08:45:14.126609+0000","last_fullsized":"2024-02-10T08:45:14.126609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.092522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.092522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":500,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.106742+0000","last_change":"2024-02-10T08:44:00.751759+0000","last_active":"2024-02-10T08:45:15.106742+0000","last_peered":"2024-02-10T08:45:15.106742+0000","last_clean":"2024-02-10T08:45:15.106742+0000","last_became_active":"2024-02-10T08:43:28.080931+0000","last_became_peered":"2024-02-10T08:43:28.080931+0000","last_unstale":"2024-02-10T08:45:15.106742+0000","last_undegraded":"2024-02-10T08:45:15.106742+0000","last_fullsized":"2024-02-10T08:45:15.106742+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.751695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":581,"reported_epoch":568,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:14.125122+0000","last_change":"2024-02-10T08:43:36.080654+0000","last_active":"2024-02-10T08:45:14.125122+0000","last_peered":"2024-02-10T08:45:14.125122+0000","last_clean":"2024-02-10T08:43:25.677981+0000","last_became_active":"2024-02-10T08:43:36.080654+0000","last_became_peered":"2024-02-10T08:43:36.080654+0000","last_unstale":"2024-02-10T08:45:14.125122+0000","last_undegraded":"2024-02-10T08:43:36.066356+0000","last_fullsized":"2024-02-10T08:43:36.066242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":127,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-02-10T08:45:20.443139+0000","last_change":"2024-02-10T08:45:20.443139+0000","last_active":"2024-02-10T08:45:20.443139+0000","last_peered":"2024-02-10T08:45:20.443139+0000","last_clean":"2024-02-10T08:45:20.443139+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:45:20.443139+0000","last_undegraded":"2024-02-10T08:45:20.443139+0000","last_fullsized":"2024-02-10T08:45:20.443139+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:20.443077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:45:20.443077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":484,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-10T08:45:15.556747+0000","last_change":"2024-02-10T08:45:15.556725+0000","last_active":"2024-02-10T08:45:15.556747+0000","last_peered":"2024-02-10T08:45:15.556747+0000","last_clean":"2024-02-10T08:45:15.556747+0000","last_became_active":"2024-02-10T08:43:15.131035+0000","last_became_peered":"2024-02-10T08:43:15.131035+0000","last_unstale":"2024-02-10T08:45:15.556747+0000","last_undegraded":"2024-02-10T08:45:15.556747+0000","last_fullsized":"2024-02-10T08:45:15.556747+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:15.556636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:15.556636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":500,"reported_epoch":569,"state":"active+undersized","last_fresh":"2024-02-10T08:45:15.106112+0000","last_change":"2024-02-10T08:43:36.078496+0000","last_active":"2024-02-10T08:45:15.106112+0000","last_peered":"2024-02-10T08:45:15.106112+0000","last_clean":"2024-02-10T08:43:31.066991+0000","last_became_active":"2024-02-10T08:43:36.078496+0000","last_became_peered":"2024-02-10T08:43:36.078496+0000","last_unstale":"2024-02-10T08:45:15.106112+0000","last_undegraded":"2024-02-10T08:45:15.106112+0000","last_fullsized":"2024-02-10T08:43:36.061038+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.399709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.399709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":552,"reported_epoch":568,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:14.127749+0000","last_change":"2024-02-10T08:43:36.081968+0000","last_active":"2024-02-10T08:45:14.127749+0000","last_peered":"2024-02-10T08:45:14.127749+0000","last_clean":"2024-02-10T08:43:25.647108+0000","last_became_active":"2024-02-10T08:43:36.081968+0000","last_became_peered":"2024-02-10T08:43:36.081968+0000","last_unstale":"2024-02-10T08:45:14.127749+0000","last_undegraded":"2024-02-10T08:43:36.066991+0000","last_fullsized":"2024-02-10T08:43:36.066884+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":122,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.431799+0000","last_change":"2024-02-10T08:45:13.432001+0000","last_active":"2024-02-10T08:45:19.431799+0000","last_peered":"2024-02-10T08:45:19.431799+0000","last_clean":"2024-02-10T08:45:19.431799+0000","last_became_active":"2024-02-10T08:43:15.134426+0000","last_became_peered":"2024-02-10T08:43:15.134426+0000","last_unstale":"2024-02-10T08:45:19.431799+0000","last_undegraded":"2024-02-10T08:45:19.431799+0000","last_fullsized":"2024-02-10T08:45:19.431799+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:13.431922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.431922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":535,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.147055+0000","last_change":"2024-02-10T08:42:35.160383+0000","last_active":"2024-02-10T08:45:19.147055+0000","last_peered":"2024-02-10T08:45:19.147055+0000","last_clean":"2024-02-10T08:45:19.147055+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:45:19.147055+0000","last_undegraded":"2024-02-10T08:45:19.147055+0000","last_fullsized":"2024-02-10T08:45:19.147055+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":468,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-02-10T08:45:19.147111+0000","last_change":"2024-02-10T08:44:25.269848+0000","last_active":"2024-02-10T08:45:19.147111+0000","last_peered":"2024-02-10T08:45:19.147111+0000","last_clean":"2024-02-10T08:45:19.147111+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:45:19.147111+0000","last_undegraded":"2024-02-10T08:45:19.147111+0000","last_fullsized":"2024-02-10T08:45:19.147111+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.269759+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":581,"reported_epoch":573,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:19.111153+0000","last_change":"2024-02-10T08:43:36.080498+0000","last_active":"2024-02-10T08:45:19.111153+0000","last_peered":"2024-02-10T08:45:19.111153+0000","last_clean":"2024-02-10T08:43:25.563172+0000","last_became_active":"2024-02-10T08:43:36.080498+0000","last_became_peered":"2024-02-10T08:43:36.080498+0000","last_unstale":"2024-02-10T08:45:19.111153+0000","last_undegraded":"2024-02-10T08:43:36.063844+0000","last_fullsized":"2024-02-10T08:43:36.063724+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:43:22.036213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:22.036213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":549,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.120744+0000","last_change":"2024-02-10T08:43:14.144599+0000","last_active":"2024-02-10T08:45:14.120744+0000","last_peered":"2024-02-10T08:45:14.120744+0000","last_clean":"2024-02-10T08:45:14.120744+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:45:14.120744+0000","last_undegraded":"2024-02-10T08:45:14.120744+0000","last_fullsized":"2024-02-10T08:45:14.120744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.144537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.144537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.148322+0000","last_change":"2024-02-10T08:43:36.070306+0000","last_active":"2024-02-10T08:45:19.148322+0000","last_peered":"2024-02-10T08:45:19.148322+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.070306+0000","last_became_peered":"2024-02-10T08:43:36.070306+0000","last_unstale":"2024-02-10T08:45:19.148322+0000","last_undegraded":"2024-02-10T08:45:19.148322+0000","last_fullsized":"2024-02-10T08:43:36.056977+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":499,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.147351+0000","last_change":"2024-02-10T08:43:36.069163+0000","last_active":"2024-02-10T08:45:19.147351+0000","last_peered":"2024-02-10T08:45:19.147351+0000","last_clean":"2024-02-10T08:43:31.063295+0000","last_became_active":"2024-02-10T08:43:36.069163+0000","last_became_peered":"2024-02-10T08:43:36.069163+0000","last_unstale":"2024-02-10T08:45:19.147351+0000","last_undegraded":"2024-02-10T08:45:19.147351+0000","last_fullsized":"2024-02-10T08:43:36.056950+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":606,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.119426+0000","last_change":"2024-02-10T08:44:31.267329+0000","last_active":"2024-02-10T08:45:14.119426+0000","last_peered":"2024-02-10T08:45:14.119426+0000","last_clean":"2024-02-10T08:45:14.119426+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:45:14.119426+0000","last_undegraded":"2024-02-10T08:45:14.119426+0000","last_fullsized":"2024-02-10T08:45:14.119426+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:44:31.267261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.267261+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":123,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.432218+0000","last_change":"2024-02-10T08:43:36.074560+0000","last_active":"2024-02-10T08:45:19.432218+0000","last_peered":"2024-02-10T08:45:19.432218+0000","last_clean":"2024-02-10T08:43:31.060871+0000","last_became_active":"2024-02-10T08:43:36.074560+0000","last_became_peered":"2024-02-10T08:43:36.074560+0000","last_unstale":"2024-02-10T08:45:19.432218+0000","last_undegraded":"2024-02-10T08:45:19.432218+0000","last_fullsized":"2024-02-10T08:43:36.062458+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:48.325433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:48.325433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":552,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-10T08:45:14.127589+0000","last_change":"2024-02-10T08:43:55.185139+0000","last_active":"2024-02-10T08:45:14.127589+0000","last_peered":"2024-02-10T08:45:14.127589+0000","last_clean":"2024-02-10T08:45:14.127589+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:45:14.127589+0000","last_undegraded":"2024-02-10T08:45:14.127589+0000","last_fullsized":"2024-02-10T08:45:14.127589+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:55.185039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:55.185039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":501,"reported_epoch":573,"state":"active+undersized","last_fresh":"2024-02-10T08:45:19.113319+0000","last_change":"2024-02-10T08:43:36.085682+0000","last_active":"2024-02-10T08:45:19.113319+0000","last_peered":"2024-02-10T08:45:19.113319+0000","last_clean":"2024-02-10T08:43:31.156976+0000","last_became_active":"2024-02-10T08:43:36.085682+0000","last_became_peered":"2024-02-10T08:43:36.085682+0000","last_unstale":"2024-02-10T08:45:19.113319+0000","last_undegraded":"2024-02-10T08:45:19.113319+0000","last_fullsized":"2024-02-10T08:43:36.064612+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":55,"acting":55,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":5506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":720,"ondisk_log_size":720,"up":81,"acting":81,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":710,"num_read_kb":705,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":85,"acting":86,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299539,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304252,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440900,"statfs":{"total":95995035648,"available":95683481600,"internally_reserved":0,"allocated":3002368,"data_stored":1458800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.59299999999999997},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.623,"5min":0.99299999999999999,"15min":22.143000000000001},"last":4.6849999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.56100000000000005},"min":{"1min":0.35999999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.623,"5min":1.623,"15min":8.7850000000000001},"last":0.25800000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":1.359,"5min":3.2389999999999999,"15min":7.593},"last":4.7320000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51300000000000001,"15min":0.57299999999999995},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.19900000000000001},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":7.7759999999999998},"last":4.5339999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49299999999999999,"15min":0.54400000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":1.504,"5min":2.5009999999999999,"15min":6.0300000000000002},"last":7.3949999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.64500000000000002},"min":{"1min":0.34100000000000003,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":19.369},"last":7.5099999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.52200000000000002,"15min":0.56999999999999995},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.6419999999999999,"5min":2.7029999999999998,"15min":6.9829999999999997},"last":1.371},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52900000000000003,"15min":0.63700000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":1.79,"5min":1.79,"15min":22.102},"last":7.6840000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":8.827},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":1.903,"15min":13.646000000000001},"last":0.35199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.45500000000000002,"15min":0.52200000000000002},"min":{"1min":0.26000000000000001,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.66000000000000003,"5min":0.81200000000000006,"15min":19.326000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66600000000000004,"5min":0.99199999999999999,"15min":1.6220000000000001},"last":0.39200000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48899999999999999,"15min":0.56200000000000006},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26600000000000001},"max":{"1min":0.68600000000000005,"5min":0.86899999999999999,"15min":22.687000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":0.77700000000000002,"15min":1.7450000000000001},"last":0.31}]}]},{"osd":6,"up_from":44,"seq":188978561177,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308304,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436848,"statfs":{"total":95995035648,"available":95679332352,"internally_reserved":0,"allocated":2957312,"data_stored":1449518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.25900000000000001,"5min":0.23400000000000001,"15min":0.19800000000000001},"max":{"1min":3.9399999999999999,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.798,"5min":0.76500000000000001,"15min":0.628},"min":{"1min":0.254,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":18.777999999999999,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.83899999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":0.89700000000000002,"15min":0.65900000000000003},"min":{"1min":0.35399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":18.425999999999998,"5min":18.425999999999998,"15min":18.425999999999998},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.74299999999999999,"15min":0.58999999999999997},"min":{"1min":0.29099999999999998,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":8.4399999999999995,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":0.66200000000000003,"15min":0.57999999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":10.772,"5min":10.772,"15min":10.772},"last":0.747},{"interface":"front","average":{"1min":1.6539999999999999,"5min":0.749,"15min":0.61099999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.246},"max":{"1min":18.829000000000001,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.88}]},{"osd":3,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.0990000000000002,"5min":7.0990000000000002,"15min":16.823},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.318,"15min":0.28699999999999998},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":3.7389999999999999},"last":0.76500000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45300000000000001,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":3.657,"15min":3.657},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.46500000000000002,"15min":0.40500000000000003},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.24399999999999999},"max":{"1min":0.61799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.64300000000000002,"15min":0.496},"min":{"1min":0.28000000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":18.853999999999999,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.48699999999999999,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":0.97699999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.54600000000000004,"15min":0.503},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.5650000000000004,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.52800000000000002,"15min":0.50600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":1.0229999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312372,"kb_used_data":2860,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432780,"statfs":{"total":95995035648,"available":95675166720,"internally_reserved":0,"allocated":2928640,"data_stored":1411461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.38300000000000001,"15min":0.42999999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.8710000000000004,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.46100000000000002},"min":{"1min":0.247,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.4910000000000001,"5min":2.1890000000000001,"15min":16.166},"last":0.26100000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.378,"15min":0.61799999999999999},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":34.985999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.36199999999999999,"15min":0.53200000000000003},"min":{"1min":0.224,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":21.614999999999998},"last":0.27400000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.35199999999999998,"15min":0.505},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":24.773},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.38,"15min":0.51100000000000001},"min":{"1min":0.254,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":16.120999999999999},"last":0.311}]},{"osd":4,"last update":"Sat Feb 10 08:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.60999999999999999},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.89000000000000001,"15min":16.074000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.60399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":9.5090000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:44:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54100000000000004,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":0.89800000000000002,"5min":2.0270000000000001,"15min":9.5600000000000005},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55100000000000005,"15min":0.64900000000000002},"min":{"1min":0.371,"5min":0.35399999999999998,"15min":0.31900000000000001},"max":{"1min":1.036,"5min":1.036,"15min":7.8479999999999999},"last":0.38300000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":1.5489999999999999,"5min":1.996,"15min":9.532},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.41399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.8520000000000003,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.35699999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.60499999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.95099999999999996,"5min":2.0939999999999999,"15min":9.4640000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57499999999999996,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":7.9199999999999999},"last":0.45000000000000001}]}]},{"osd":0,"up_from":0,"seq":34359738550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411499,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308252,"kb_used_data":2836,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436900,"statfs":{"total":95995035648,"available":95679385600,"internally_reserved":0,"allocated":2904064,"data_stored":1410064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.45900000000000002,"15min":0.438},"min":{"1min":0.192,"5min":0.192,"15min":0.187},"max":{"1min":10.342000000000001,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.204,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.20499999999999999,"5min":0.187,"15min":0.187},"max":{"1min":14.858000000000001,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.379}]},{"osd":1,"last update":"Sat Feb 10 08:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.45800000000000002,"15min":0.50800000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":12.512},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.41199999999999998,"15min":0.47099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.0760000000000001,"5min":3.0960000000000001,"15min":9.6769999999999996},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:45:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.73399999999999999,"15min":0.55400000000000005},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":10.994999999999999,"5min":11.612,"15min":11.612},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.66800000000000004,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.252,"15min":0.22600000000000001},"max":{"1min":7.8280000000000003,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.32100000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.67200000000000004},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76200000000000001,"5min":1.7370000000000001,"15min":10.920999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":1.014,"15min":12.292999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:44:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57499999999999996,"15min":0.66000000000000003},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":9.2029999999999994},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.64800000000000002,"15min":0.64900000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":9.6690000000000005,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.44700000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:45:08 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":0.77100000000000002,"15min":0.72399999999999998},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.714,"5min":0.82499999999999996,"15min":0.71899999999999997},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.708,"15min":9.2940000000000005},"last":0.5},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.65500000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":7.4219999999999997},"last":0.48099999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182567,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308256,"kb_used_data":2840,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436896,"statfs":{"total":95995035648,"available":95679381504,"internally_reserved":0,"allocated":2908160,"data_stored":1400452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32100000000000001,"15min":0.498},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.52800000000000002,"5min":0.878,"15min":14.744},"last":0.218},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.36799999999999999,"15min":0.53700000000000003},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.188},"max":{"1min":0.48999999999999999,"5min":3.5579999999999998,"15min":11.808},"last":0.29899999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.42199999999999999,"15min":0.54800000000000004},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.53500000000000003,"5min":9.3100000000000005,"15min":16.634},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.41599999999999998,"15min":0.55000000000000004},"min":{"1min":0.222,"5min":0.222,"15min":0.215},"max":{"1min":0.54200000000000004,"5min":5.4370000000000003,"15min":12.438000000000001},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.57599999999999996},"min":{"1min":0.26000000000000001,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":0.94899999999999995,"5min":3.1549999999999998,"15min":16.536000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.38300000000000001,"15min":0.622},"min":{"1min":0.247,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.59799999999999998,"5min":2.5230000000000001,"15min":23.294},"last":0.311}]},{"osd":4,"last update":"Sat Feb 10 08:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54500000000000004,"15min":0.66900000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":2.589,"15min":12.371},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68700000000000006,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:44:36 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.57299999999999995,"15min":0.70099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.4019999999999999,"5min":3.6040000000000001,"15min":12.579000000000001},"last":0.435},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.65000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.4129999999999998,"5min":4.9909999999999997,"15min":11.448},"last":0.34999999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.57299999999999995,"15min":0.69499999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":3.48,"15min":16.091000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.58799999999999997,"15min":0.68600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":3.2469999999999999,"15min":16.475000000000001},"last":0.36699999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:44:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.63500000000000001,"15min":0.68400000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.6140000000000008,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.61699999999999999,"15min":0.73599999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85599999999999998,"5min":3.52,"15min":12.606999999999999},"last":0.41499999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986316,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312804,"kb_used_data":3292,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432348,"statfs":{"total":95995035648,"available":95674724352,"internally_reserved":0,"allocated":3371008,"data_stored":1427290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.627},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":19.940999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.55300000000000005},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.91800000000000004,"15min":9.1760000000000002},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46899999999999997,"15min":0.57899999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.59699999999999998,"5min":0.93799999999999994,"15min":5.774},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.59799999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":8.6989999999999998},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.59799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.83999999999999997,"15min":6.2329999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.59799999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.64600000000000002,"5min":0.77100000000000002,"15min":10.01},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48399999999999999,"15min":0.65000000000000002},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.27700000000000002},"max":{"1min":0.58299999999999996,"5min":0.92700000000000005,"15min":9.5909999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51900000000000002,"15min":0.71799999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":25.27},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.621,"5min":0.876,"15min":6.8730000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.223},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":3.8290000000000002},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.247},"max":{"1min":0.626,"5min":0.78000000000000003,"15min":13.342000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":1.085,"15min":13.753},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.52000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.90400000000000003,"15min":12.835000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.72599999999999998,"5min":1.0509999999999999,"15min":12.875999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790110,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316544,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93428608,"statfs":{"total":95995035648,"available":95670894592,"internally_reserved":0,"allocated":3006464,"data_stored":1458541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":0.59799999999999998,"15min":1.028},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":107.551},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.55000000000000004,"15min":0.97199999999999998},"min":{"1min":0.29299999999999998,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":5.375,"5min":5.375,"15min":88.792000000000002},"last":0.32400000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.96499999999999997},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":8.7200000000000006,"5min":8.7200000000000006,"15min":107.205},"last":0.318},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.60799999999999998,"15min":0.84399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":4.7629999999999999,"5min":4.7629999999999999,"15min":72.272000000000006},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.58999999999999997,"15min":0.98199999999999998},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":10.794,"5min":10.794,"15min":54.174999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.486,"5min":0.70899999999999996,"15min":1.0549999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":8.6890000000000001,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.442}]},{"osd":3,"last update":"Sat Feb 10 08:44:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":0.77700000000000002,"15min":0.81000000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":13.691000000000001,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.38},{"interface":"front","average":{"1min":1.4359999999999999,"5min":0.71599999999999997,"15min":1.2629999999999999},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":8.7690000000000001,"5min":8.7690000000000001,"15min":114.526},"last":0.42299999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Feb 10 08:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.55900000000000005,"15min":0.72199999999999998},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":10.023999999999999,"5min":10.023999999999999,"15min":72.241},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.013,"5min":10.013,"15min":22.469999999999999},"last":0.30499999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.495,"15min":0.71699999999999997},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":66.844999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.50900000000000001,"15min":0.746},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56699999999999995,"5min":0.90700000000000003,"15min":72.257000000000005},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:45:24.597 INFO:tasks.thrashosds.thrasher:Setting 2.12 to [3, 7, 1] 2024-02-10T08:45:24.597 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.12', '3', '7', '1'] 2024-02-10T08:45:24.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 2.12 3 7 1 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.000057+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.d deep-scrub starts 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.003135+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.d deep-scrub ok 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.375082+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.377972+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:23.504153+0000 mgr.y (mgr.14553) 503 : cluster [DBG] pgmap v868: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:23.521561+0000 mon.a (mon.0) 2944 : audit [DBG] from='client.? v1:172.21.15.107:0/3806212038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:24.116840+0000 mon.a (mon.0) 2945 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:24.116902+0000 mon.a (mon.0) 2946 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-10T08:45:24.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.000057+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.d deep-scrub starts 2024-02-10T08:45:24.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.003135+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.d deep-scrub ok 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.375082+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.377972+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:23.504153+0000 mgr.y (mgr.14553) 503 : cluster [DBG] pgmap v868: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:23.521561+0000 mon.a (mon.0) 2944 : audit [DBG] from='client.? v1:172.21.15.107:0/3806212038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:24.116840+0000 mon.a (mon.0) 2945 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:24.116902+0000 mon.a (mon.0) 2946 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.000057+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.d deep-scrub starts 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.003135+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.d deep-scrub ok 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.375082+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.377972+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:23.504153+0000 mgr.y (mgr.14553) 503 : cluster [DBG] pgmap v868: 162 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:23.521561+0000 mon.a (mon.0) 2944 : audit [DBG] from='client.? v1:172.21.15.107:0/3806212038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:24.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:24.116840+0000 mon.a (mon.0) 2945 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:24.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:24.116902+0000 mon.a (mon.0) 2946 : cluster [DBG] osdmap e578: 8 total, 7 up, 7 in 2024-02-10T08:45:24.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:45:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:25.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:25.425 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:25.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:25.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:25.556 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:25.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:25.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:25] "GET /metrics HTTP/1.1" 200 191908 "" "Prometheus/2.33.4" 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:24.424897+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.b scrub starts 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:24.427788+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.b scrub ok 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.119332+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:25.129172+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:25.129555+0000 mon.a (mon.0) 2948 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:25.168433+0000 mon.a (mon.0) 2949 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:24.424897+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.b scrub starts 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:24.427788+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.b scrub ok 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.119332+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:25.129172+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:25.129555+0000 mon.a (mon.0) 2948 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:25.168433+0000 mon.a (mon.0) 2949 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:24.424897+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.b scrub starts 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:24.427788+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.b scrub ok 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.119332+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 7 up, 7 in 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:25.129172+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:25.129555+0000 mon.a (mon.0) 2948 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:25.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:25.168433+0000 mon.a (mon.0) 2949 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:25.704 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:25.722 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:25.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:25.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:25.851 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:25.869 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:25.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:25.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:26.000 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:26.352 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:26.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:26.356 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:26.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:26.483 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:26.510 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:26.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:26.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.019132+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.022133+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.449739+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.c scrub starts 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.452716+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.c scrub ok 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:25.504735+0000 mgr.y (mgr.14553) 504 : cluster [DBG] pgmap v871: 202 pgs: 40 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:26.118718+0000 mon.a (mon.0) 2950 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:26.118789+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-10T08:45:26.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:26.135044+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:26.143409+0000 mon.a (mon.0) 2952 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.019132+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.022133+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.449739+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.c scrub starts 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.452716+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.c scrub ok 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:25.504735+0000 mgr.y (mgr.14553) 504 : cluster [DBG] pgmap v871: 202 pgs: 40 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:26.118718+0000 mon.a (mon.0) 2950 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:26.118789+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:26.135044+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:26.143409+0000 mon.a (mon.0) 2952 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.642 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:26.661 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:26.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:26.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.019132+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.022133+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.449739+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.c scrub starts 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.452716+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.c scrub ok 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:25.504735+0000 mgr.y (mgr.14553) 504 : cluster [DBG] pgmap v871: 202 pgs: 40 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 99 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:26.118718+0000 mon.a (mon.0) 2950 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:26.118789+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e580: 8 total, 7 up, 7 in 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:26.135044+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:26.143409+0000 mon.a (mon.0) 2952 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:26.790 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:26.808 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:26.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:26.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:26.941 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:27.262 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:45:27.262 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:45:27.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:45:28.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:45:28.101 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:28.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:28.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:28.246 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:27.119817+0000 mon.a (mon.0) 2953 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:28.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:27.119886+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:27.121437+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:27.121817+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:27.505339+0000 mgr.y (mgr.14553) 505 : cluster [DBG] pgmap v874: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:27.119817+0000 mon.a (mon.0) 2953 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:27.119886+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:27.121437+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:27.121817+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:27.505339+0000 mgr.y (mgr.14553) 505 : cluster [DBG] pgmap v874: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:27.119817+0000 mon.a (mon.0) 2953 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:27.119886+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e581: 8 total, 7 up, 7 in 2024-02-10T08:45:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:27.121437+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:27.121817+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:27.505339+0000 mgr.y (mgr.14553) 505 : cluster [DBG] pgmap v874: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:29.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:28.130054+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:29.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:28.130129+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-10T08:45:29.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:28.132871+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:28.133359+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.192 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:28.130054+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:29.192 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:28.130129+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-10T08:45:29.192 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:28.132871+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.192 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:28.133359+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:29.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:29.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:29.345 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:29.368 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:29.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:29.460 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:29.461 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:29.463 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:29.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.491+0000 7fad6677e700 1 -- 172.21.15.107:0/513049255 >> v1:172.21.15.107:6789/0 conn(0x7fad601007f0 legacy=0x7fad60102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.491+0000 7fad6677e700 1 -- 172.21.15.107:0/513049255 shutdown_connections 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.491+0000 7fad6677e700 1 -- 172.21.15.107:0/513049255 >> 172.21.15.107:0/513049255 conn(0x7fad600f3640 msgr2=0x7fad600f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.491+0000 7fad6677e700 1 -- 172.21.15.107:0/513049255 shutdown_connections 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.491+0000 7fad6677e700 1 -- 172.21.15.107:0/513049255 wait complete. 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 librados: starting msgr at 2024-02-10T08:45:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 librados: starting objecter 2024-02-10T08:45:29.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 Processor -- start 2024-02-10T08:45:29.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 -- start start 2024-02-10T08:45:29.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 librados: setting wanted keys 2024-02-10T08:45:29.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 librados: calling monclient init 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad600fcf70 con 0x7fad600fda50 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad600fd160 con 0x7fad601007f0 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad6677e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad600fd350 con 0x7fad60104330 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad5f7fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fad601007f0 0x7fad60114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:41626/0 (socket says 172.21.15.107:41626) 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad5f7fe700 1 -- 172.21.15.107:0/443484998 learned_addr learned my addr 172.21.15.107:0/443484998 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.492+0000 7fad64d1b700 1 --1- 172.21.15.107:0/443484998 >> v1:172.21.15.107:6790/0 conn(0x7fad60104330 0x7fad601184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44668/0 (socket says 172.21.15.107:44668) 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 10 client.?.objecter ms_handle_connect 0x7fad601007f0 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 10 client.?.objecter ms_handle_connect 0x7fad60104330 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:29.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548143516 0 0) 0x7fad600fd160 con 0x7fad601007f0 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad48003030 con 0x7fad601007f0 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972126389 0 0) 0x7fad600fd350 con 0x7fad60104330 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600fd160 con 0x7fad60104330 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058031196 0 0) 0x7fad48003030 con 0x7fad601007f0 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad600fd350 con 0x7fad601007f0 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3443299484 0 0) 0x7fad600fd160 con 0x7fad60104330 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad48003030 con 0x7fad60104330 2024-02-10T08:45:29.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fad54002e20 con 0x7fad601007f0 2024-02-10T08:45:29.496 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:28.130054+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-99", "overlaypool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:29.496 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:28.130129+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e582: 8 total, 7 up, 7 in 2024-02-10T08:45:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:28.132871+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:28.133359+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]: dispatch 2024-02-10T08:45:29.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fad58002ac0 con 0x7fad60104330 2024-02-10T08:45:29.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703215881 0 0) 0x7fad600fd350 con 0x7fad601007f0 2024-02-10T08:45:29.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 >> v1:172.21.15.107:6790/0 conn(0x7fad60104330 legacy=0x7fad601184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:29.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 >> v1:172.21.15.107:6789/0 conn(0x7fad600fda50 legacy=0x7fad600fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.493+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6011cc00 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fad540024a0 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fad540039f0 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad60119c30 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad6677e700 10 client.48691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6011a120 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.494+0000 7fad6677e700 1 librados: init done 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.496+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6005f740 con 0x7fad601007f0 2024-02-10T08:45:29.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.496+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fad54002000 con 0x7fad601007f0 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(583..583 src has 420..583) v4 ==== 9421+0+0 (unknown 3728522111 0 0) 0x7fad540796a0 con 0x7fad601007f0 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 10 client.48691.objecter ms_dispatch 0x7fad60001550 osd_map(583..583 src has 420..583) v4 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 3 client.48691.objecter handle_osd_map got epochs [583,583] > 0 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 3 client.48691.objecter handle_osd_map decoding full epoch 583 2024-02-10T08:45:29.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 20 client.48691.objecter dump_active .. 0 homeless 2024-02-10T08:45:29.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.497+0000 7fad5d7fa700 10 client.48691.objecter ms_handle_connect 0x7fad48063b60 2024-02-10T08:45:29.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.501+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fad5404c5a0 con 0x7fad601007f0 2024-02-10T08:45:29.516 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:29.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:29.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:29.652 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:29.676 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:29.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:29.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:29.681+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fad600475e0 con 0x7fad601007f0 2024-02-10T08:45:29.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:29.806 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:29.824 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:29.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:29.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:29.957 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.187+0000 7fad5d7fa700 1 -- 172.21.15.107:0/443484998 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v584) v1 ==== 87+0+0 (unknown 21452649 0 0) 0x7fad5404aa10 con 0x7fad601007f0 2024-02-10T08:45:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.191+0000 7fad6677e700 10 librados: watch_flush enter 2024-02-10T08:45:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.195+0000 7fad6677e700 10 librados: watch_flush exit 2024-02-10T08:45:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.195+0000 7fad6677e700 20 client.48691.objecter shutdown clearing up homeless session... 2024-02-10T08:45:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.195+0000 7fad6677e700 10 client.48691.objecter successfully canceled tick 2024-02-10T08:45:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.198+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 >> v1:172.21.15.107:6800/2634842009 conn(0x7fad48063b60 legacy=0x7fad48066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.198+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 >> v1:172.21.15.195:6789/0 conn(0x7fad601007f0 legacy=0x7fad60114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:30.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.209+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 shutdown_connections 2024-02-10T08:45:30.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.209+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 >> 172.21.15.107:0/443484998 conn(0x7fad600f3640 msgr2=0x7fad601077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.222+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 shutdown_connections 2024-02-10T08:45:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.224+0000 7fad6677e700 1 -- 172.21.15.107:0/443484998 wait complete. 2024-02-10T08:45:30.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:30.228+0000 7fad6677e700 1 librados: shutdown 2024-02-10T08:45:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:28.637982+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:28.640895+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.056737+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.056892+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.124404+0000 mon.a (mon.0) 2959 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:29.132641+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.132715+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:29.146086+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:29.176956+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.293252+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.297287+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.505937+0000 mgr.y (mgr.14553) 506 : cluster [DBG] pgmap v877: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:29.683943+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.107:0/443484998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:29.684293+0000 mon.a (mon.0) 2963 : audit [INF] from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:28.637982+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:28.640895+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.056737+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.056892+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:30.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.124404+0000 mon.a (mon.0) 2959 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:29.132641+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.132715+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:29.146086+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:29.176956+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.293252+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.297287+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.505937+0000 mgr.y (mgr.14553) 506 : cluster [DBG] pgmap v877: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:29.683943+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.107:0/443484998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:30.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:29.684293+0000 mon.a (mon.0) 2963 : audit [INF] from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:28.637982+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:28.640895+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.056737+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.056892+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.124404+0000 mon.a (mon.0) 2959 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:29.132641+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-99-cache", "mode": "writeback"}]': finished 2024-02-10T08:45:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.132715+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e583: 8 total, 7 up, 7 in 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:29.146086+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:29.176956+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.293252+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.297287+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.505937+0000 mgr.y (mgr.14553) 506 : cluster [DBG] pgmap v877: 202 pgs: 1 active+clean+remapped, 22 creating+peering, 21 active+undersized+degraded, 49 active+undersized, 109 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:29.683943+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.107:0/443484998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:29.684293+0000 mon.a (mon.0) 2963 : audit [INF] from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:45:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.561997+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:45:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:29.566470+0000 osd.6 (osd.6) 249 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:45:31.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:30.169741+0000 mon.a (mon.0) 2964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:30.181797+0000 mon.a (mon.0) 2965 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:30.181858+0000 mon.a (mon.0) 2966 : audit [INF] from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:30.181899+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:30.183709+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:30.184552+0000 mon.a (mon.0) 2968 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:31.104291+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:31.104364+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:31.108486+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:31.111725+0000 mon.a (mon.0) 2971 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.561997+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:29.566470+0000 osd.6 (osd.6) 249 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:30.169741+0000 mon.a (mon.0) 2964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:31.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:30.181797+0000 mon.a (mon.0) 2965 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:30.181858+0000 mon.a (mon.0) 2966 : audit [INF] from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:30.181899+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:30.183709+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:30.184552+0000 mon.a (mon.0) 2968 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:31.104291+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:31.104364+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:31.108486+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:31.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:31.111725+0000 mon.a (mon.0) 2971 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.561997+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:29.566470+0000 osd.6 (osd.6) 249 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:30.169741+0000 mon.a (mon.0) 2964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:30.181797+0000 mon.a (mon.0) 2965 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:30.181858+0000 mon.a (mon.0) 2966 : audit [INF] from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:30.181899+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e584: 8 total, 7 up, 7 in 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:30.183709+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:30.184552+0000 mon.a (mon.0) 2968 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:31.104291+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:31.104364+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e585: 8 total, 7 up, 7 in 2024-02-10T08:45:31.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:31.108486+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:31.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:31.111725+0000 mon.a (mon.0) 2971 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:45:32.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:32.409 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:32.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:32.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:31.506961+0000 mgr.y (mgr.14553) 507 : cluster [DBG] pgmap v880: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:31.647882+0000 mgr.y (mgr.14553) 508 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:32.109483+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:32.115281+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:32.115327+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:32.117142+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:32.119700+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:31.506961+0000 mgr.y (mgr.14553) 507 : cluster [DBG] pgmap v880: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:32.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:31.647882+0000 mgr.y (mgr.14553) 508 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:32.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:32.109483+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:32.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:32.115281+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:32.115327+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:32.117142+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:32.119700+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:31.506961+0000 mgr.y (mgr.14553) 507 : cluster [DBG] pgmap v880: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:31.647882+0000 mgr.y (mgr.14553) 508 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:32.109483+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:32.115281+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:32.115327+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e586: 8 total, 7 up, 7 in 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:32.117142+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:32.119700+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T08:45:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:32.664 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:32.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:32.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:32.793 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:32.811 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:32.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:32.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:32.942 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:32.960 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:32.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:33.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:33.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:33.108 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:33.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:33.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:33.241 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:33.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.736+0000 7f3e70753700 1 -- 172.21.15.107:0/2455348701 >> v1:172.21.15.107:6790/0 conn(0x7f3e680fda50 legacy=0x7f3e680fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.736+0000 7f3e70753700 1 -- 172.21.15.107:0/2455348701 shutdown_connections 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.736+0000 7f3e70753700 1 -- 172.21.15.107:0/2455348701 >> 172.21.15.107:0/2455348701 conn(0x7f3e680f3640 msgr2=0x7f3e680f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.736+0000 7f3e70753700 1 -- 172.21.15.107:0/2455348701 shutdown_connections 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.736+0000 7f3e70753700 1 -- 172.21.15.107:0/2455348701 wait complete. 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 librados: starting msgr at 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 librados: starting objecter 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 Processor -- start 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 -- start start 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 librados: setting wanted keys 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 librados: calling monclient init 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e6806cbc0 con 0x7f3e68104330 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e6806cd90 con 0x7f3e681007f0 2024-02-10T08:45:33.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e70753700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e6806cf80 con 0x7f3e680fda50 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e6e4ef700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3e680fda50 0x7f3e680fc9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44676/0 (socket says 172.21.15.107:44676) 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e6e4ef700 1 -- 172.21.15.107:0/1641124368 learned_addr learned my addr 172.21.15.107:0/1641124368 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter ms_handle_connect 0x7f3e680fda50 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter ms_handle_connect 0x7f3e68104330 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter ms_handle_connect 0x7f3e681007f0 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.737+0000 7f3e5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815845970 0 0) 0x7f3e6806cbc0 con 0x7f3e68104330 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e54003030 con 0x7f3e68104330 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070735897 0 0) 0x7f3e6806cd90 con 0x7f3e681007f0 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e6806cbc0 con 0x7f3e681007f0 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052713017 0 0) 0x7f3e6806cf80 con 0x7f3e680fda50 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e6806cd90 con 0x7f3e680fda50 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029642358 0 0) 0x7f3e54003030 con 0x7f3e68104330 2024-02-10T08:45:33.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6806cf80 con 0x7f3e68104330 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3e60001ca0 con 0x7f3e68104330 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1891183953 0 0) 0x7f3e6806cd90 con 0x7f3e680fda50 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:33.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:33.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e54003030 con 0x7f3e680fda50 2024-02-10T08:45:33.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632356294 0 0) 0x7f3e6806cbc0 con 0x7f3e681007f0 2024-02-10T08:45:33.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6806cd90 con 0x7f3e681007f0 2024-02-10T08:45:33.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3e58001430 con 0x7f3e680fda50 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3e64001800 con 0x7f3e681007f0 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1942834108 0 0) 0x7f3e6806cf80 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 >> v1:172.21.15.107:6790/0 conn(0x7f3e680fda50 legacy=0x7f3e680fc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 >> v1:172.21.15.195:6789/0 conn(0x7f3e681007f0 legacy=0x7f3e68114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6811cc00 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e68119c30 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e70753700 10 client.28398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6811a120 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.738+0000 7f3e70753700 1 librados: init done 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.739+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3e60002610 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.739+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3e600036a0 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.739+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3e6805f700 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.740+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3e600038f0 con 0x7f3e68104330 2024-02-10T08:45:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.740+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(587..587 src has 420..587) v4 ==== 9421+0+0 (unknown 500663176 0 0) 0x7f3e60076da0 con 0x7f3e68104330 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.740+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 osd_map(587..587 src has 420..587) v4 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.740+0000 7f3e5f7fe700 3 client.28398.objecter handle_osd_map got epochs [587,587] > 0 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.740+0000 7f3e5f7fe700 3 client.28398.objecter handle_osd_map decoding full epoch 587 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e5f7fe700 20 client.28398.objecter dump_active .. 0 homeless 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v587) v1 ==== 54+0+16 (unknown 384098794 0 2420000192) 0x7f3e60077130 con 0x7f3e68104330 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e54063cf0 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 10 client.28398.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 1 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 1 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 0 1 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 4 2024-02-10T08:45:33.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 10 client.28398.objecter _send_command 1 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e680f91e0 con 0x7f3e68059990 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.741+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e68059990 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.742+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e680f91e0 con 0x7f3e68059990 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.742+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.742+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 1 = system:0 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.742+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 4 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.742+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 0 1 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 10 client.28398.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 2 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 2 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e680475e0 osd=0 3 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 0 2 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 4 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 10 client.28398.objecter _send_command 2 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.757+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e6810a070 con 0x7f3e68059990 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e6810a070 con 0x7f3e68059990 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e680475e0 osd=0 4 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 0 2 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 10 client.28398.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 3 2024-02-10T08:45:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 1 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 4 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 10 client.28398.objecter _send_command 3 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e68120a90 con 0x7f3e68059ee0 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.759+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e68059ee0 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.760+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e68120a90 con 0x7f3e68059ee0 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.760+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:33.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.760+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 3 = system:0 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.760+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.760+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 1 3 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 10 client.28398.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68059d70 osd=1 3 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 1 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 10 client.28398.objecter _send_command 4 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.775+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e68059400 con 0x7f3e68059ee0 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e68059400 con 0x7f3e68059ee0 2024-02-10T08:45:33.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68059d70 osd=1 4 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 1 4 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 10 client.28398.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 5 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 5 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 2 5 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 4 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 10 client.28398.objecter _send_command 5 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e681254b0 con 0x7f3e68105480 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.777+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e68105480 2024-02-10T08:45:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.778+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e681254b0 con 0x7f3e68105480 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.778+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.778+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 5 = system:0 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.778+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 4 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.778+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 2 5 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 10 client.28398.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 6 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 6 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68059810 osd=2 3 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 2 6 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 4 2024-02-10T08:45:33.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 10 client.28398.objecter _send_command 6 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.792+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e681049e0 con 0x7f3e68105480 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e681049e0 con 0x7f3e68105480 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68059810 osd=2 4 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 2 6 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 10 client.28398.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 7 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 7 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 3 7 2024-02-10T08:45:33.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 4 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 10 client.28398.objecter _send_command 7 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.793+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e68129fd0 con 0x7f3e681218e0 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e681218e0 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e68129fd0 con 0x7f3e681218e0 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 7 = system:0 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 4 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.794+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 3 7 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 10 client.28398.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 8 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 8 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.808+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e68104df0 osd=3 3 2024-02-10T08:45:33.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.809+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 3 8 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.809+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 4 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.809+0000 7f3e70753700 10 client.28398.objecter _send_command 8 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.809+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e68120e40 con 0x7f3e681218e0 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e68120e40 con 0x7f3e681218e0 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e68104df0 osd=3 4 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 3 8 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 10 client.28398.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 9 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 20 client.28398.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 10 client.28398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=588}) v3 -- 0x7f3e68120c00 con 0x7f3e68104330 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e68125df0 con 0x7f3e68104330 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_get_version_reply(handle=1 version=587) v2 ==== 24+0+0 (unknown 3351808252 0 0) 0x7f3e6004ab60 con 0x7f3e68104330 2024-02-10T08:45:33.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e6ccec700 10 client.28398.objecter _check_command_map_dne tid 9 current 587 map_dne_bound 587 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e6ccec700 10 client.28398.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.810+0000 7f3e6ccec700 15 client.28398.objecter _session_command_op_remove -1 9 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 10 client.28398.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 10 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 10 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 5 10 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 4 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 10 client.28398.objecter _send_command 10 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e6812b460 con 0x7f3e681262b0 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.811+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e681262b0 2024-02-10T08:45:33.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.812+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e6812b460 con 0x7f3e681262b0 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.812+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.812+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 10 = system:0 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.812+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 4 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.812+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 5 10 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 10 client.28398.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 11 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 11 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e6812b2f0 osd=5 3 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 5 11 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 4 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 10 client.28398.objecter _send_command 11 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.826+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e6812b460 con 0x7f3e681262b0 2024-02-10T08:45:33.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e6812b460 con 0x7f3e681262b0 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e6812b2f0 osd=5 4 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 5 11 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 10 client.28398.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 12 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 12 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 6 12 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 4 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 10 client.28398.objecter _send_command 12 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e681333e0 con 0x7f3e6812f8d0 2024-02-10T08:45:33.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.828+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e6812f8d0 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.829+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e681333e0 con 0x7f3e6812f8d0 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.829+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.829+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 12 = system:0 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.829+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 4 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.829+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 6 12 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 10 client.28398.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 13 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 13 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e6812b460 osd=6 3 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 6 13 2024-02-10T08:45:33.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 4 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 10 client.28398.objecter _send_command 13 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.843+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e681333e0 con 0x7f3e6812f8d0 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e681333e0 con 0x7f3e6812f8d0 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e6812b460 osd=6 4 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 6 13 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 10 client.28398.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 14 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 14 2024-02-10T08:45:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 7 14 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 4 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 10 client.28398.objecter _send_command 14 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.845+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e68134600 con 0x7f3e681350a0 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 10 client.28398.objecter ms_handle_connect 0x7f3e681350a0 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3e68134600 con 0x7f3e681350a0 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 14 = system:0 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 4 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.846+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 7 14 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 10 client.28398.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign -1 15 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 20 client.28398.objecter _get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 15 client.28398.objecter _session_command_op_remove -1 15 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 20 client.28398.objecter get_session s=0x7f3e681333e0 osd=7 3 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 15 client.28398.objecter _session_command_op_assign 7 15 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 4 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 10 client.28398.objecter _send_command 15 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.860+0000 7f3e70753700 1 -- 172.21.15.107:0/1641124368 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3e68134600 con 0x7f3e681350a0 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.862+0000 7f3e5f7fe700 1 -- 172.21.15.107:0/1641124368 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3e68134600 con 0x7f3e681350a0 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.862+0000 7f3e5f7fe700 10 client.28398.objecter ms_dispatch 0x7f3e68001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.862+0000 7f3e5f7fe700 10 client.28398.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.862+0000 7f3e5f7fe700 20 client.28398.objecter put_session s=0x7f3e681333e0 osd=7 4 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:33.862+0000 7f3e5f7fe700 15 client.28398.objecter _session_command_op_remove 7 15 2024-02-10T08:45:33.882 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:45:33.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:45:34.193 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:34.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.060+0000 7f4b7a120700 1 -- 172.21.15.107:0/3061808511 >> v1:172.21.15.107:6790/0 conn(0x7f4b74104330 legacy=0x7f4b741067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.060+0000 7f4b7a120700 1 -- 172.21.15.107:0/3061808511 shutdown_connections 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.060+0000 7f4b7a120700 1 -- 172.21.15.107:0/3061808511 >> 172.21.15.107:0/3061808511 conn(0x7f4b740f3640 msgr2=0x7f4b740f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.060+0000 7f4b7a120700 1 -- 172.21.15.107:0/3061808511 shutdown_connections 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.060+0000 7f4b7a120700 1 -- 172.21.15.107:0/3061808511 wait complete. 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 librados: starting msgr at 2024-02-10T08:45:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 librados: starting objecter 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 Processor -- start 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 -- start start 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 librados: setting wanted keys 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 librados: calling monclient init 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b740fd000 con 0x7f4b740fda50 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b740fd190 con 0x7f4b74104330 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b7a120700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b740fd360 con 0x7f4b741007f0 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b72ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4b741007f0 0x7f4b74114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44690/0 (socket says 172.21.15.107:44690) 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b72ffd700 1 -- 172.21.15.107:0/3074758529 learned_addr learned my addr 172.21.15.107:0/3074758529 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b737fe700 1 --1- 172.21.15.107:0/3074758529 >> v1:172.21.15.107:6789/0 conn(0x7f4b740fda50 0x7f4b740fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46110/0 (socket says 172.21.15.107:46110) 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter ms_handle_connect 0x7f4b741007f0 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter ms_handle_connect 0x7f4b740fda50 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter ms_handle_connect 0x7f4b74104330 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816258587 0 0) 0x7f4b740fd360 con 0x7f4b741007f0 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.061+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c0038e0 con 0x7f4b741007f0 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014917609 0 0) 0x7f4b740fd000 con 0x7f4b740fda50 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b740fd360 con 0x7f4b740fda50 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507041223 0 0) 0x7f4b740fd190 con 0x7f4b74104330 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b740fd000 con 0x7f4b74104330 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270387572 0 0) 0x7f4b5c0038e0 con 0x7f4b741007f0 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b740fd190 con 0x7f4b741007f0 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b68001a70 con 0x7f4b741007f0 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298932459 0 0) 0x7f4b740fd360 con 0x7f4b740fda50 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b5c0038e0 con 0x7f4b740fda50 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b640018c0 con 0x7f4b740fda50 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209827934 0 0) 0x7f4b740fd000 con 0x7f4b74104330 2024-02-10T08:45:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b740fd360 con 0x7f4b74104330 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b6c001950 con 0x7f4b74104330 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328147316 0 0) 0x7f4b740fd190 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 >> v1:172.21.15.195:6789/0 conn(0x7f4b74104330 legacy=0x7f4b741184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 >> v1:172.21.15.107:6789/0 conn(0x7f4b740fda50 legacy=0x7f4b740fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7411cc00 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b74119c30 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b7a120700 10 client.53978.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7411a120 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b7a120700 1 librados: init done 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4b68002430 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.062+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4b68005a50 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.063+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b60004f20 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.063+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4b68001f90 con 0x7f4b741007f0 2024-02-10T08:45:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(587..587 src has 420..587) v4 ==== 9421+0+0 (unknown 500663176 0 0) 0x7f4b68079250 con 0x7f4b741007f0 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 10 client.53978.objecter ms_dispatch 0x7f4b74001550 osd_map(587..587 src has 420..587) v4 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 3 client.53978.objecter handle_osd_map got epochs [587,587] > 0 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 3 client.53978.objecter handle_osd_map decoding full epoch 587 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 20 client.53978.objecter dump_active .. 0 homeless 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.064+0000 7f4b70ff9700 10 client.53978.objecter ms_handle_connect 0x7f4b5c063c70 2024-02-10T08:45:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.068+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4b6804c0f0 con 0x7f4b741007f0 2024-02-10T08:45:34.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:34.332 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:34.366+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]} v 0) v1 -- 0x7f4b600059e0 con 0x7f4b741007f0 2024-02-10T08:45:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:33.116767+0000 mon.a (mon.0) 2976 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T08:45:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:33.116844+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-10T08:45:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:33.271828+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:33.278315+0000 mon.a (mon.0) 2978 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:33.507622+0000 mgr.y (mgr.14553) 509 : cluster [DBG] pgmap v883: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:33.741714+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? v1:172.21.15.107:0/1641124368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:33.116767+0000 mon.a (mon.0) 2976 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:33.116844+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:33.271828+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:33.278315+0000 mon.a (mon.0) 2978 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:33.507622+0000 mgr.y (mgr.14553) 509 : cluster [DBG] pgmap v883: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:34.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:33.741714+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? v1:172.21.15.107:0/1641124368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:34.480 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:34.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:33.116767+0000 mon.a (mon.0) 2976 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:33.116844+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e587: 8 total, 7 up, 7 in 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:33.271828+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:33.278315+0000 mon.a (mon.0) 2978 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:33.507622+0000 mgr.y (mgr.14553) 509 : cluster [DBG] pgmap v883: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:33.741714+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? v1:172.21.15.107:0/1641124368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:34.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:34.618 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:34.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.126+0000 7f4b70ff9700 1 -- 172.21.15.107:0/3074758529 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]=0 set 2.12 pg_upmap mapping to [3,7,1] v589) v1 ==== 129+0+0 (unknown 754288220 0 0) 0x7f4b6804a500 con 0x7f4b741007f0 2024-02-10T08:45:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.128+0000 7f4b7a120700 10 librados: watch_flush enter 2024-02-10T08:45:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.128+0000 7f4b7a120700 10 librados: watch_flush exit 2024-02-10T08:45:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.128+0000 7f4b7a120700 20 client.53978.objecter shutdown clearing up homeless session... 2024-02-10T08:45:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.128+0000 7f4b7a120700 10 client.53978.objecter successfully canceled tick 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.124874+0000 mon.a (mon.0) 2980 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:34.124940+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.125668+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.126135+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:34.131730+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:35.135 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:34.131793+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:35.136 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.299427+0000 mon.c (mon.2) 437 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:45:35.136 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.368028+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.107:0/3074758529' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.136 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:34.368375+0000 mon.a (mon.0) 2983 : audit [INF] from='client.53978 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.132+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4b5c063c70 legacy=0x7f4b5c066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.132+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 >> v1:172.21.15.107:6790/0 conn(0x7f4b741007f0 legacy=0x7f4b74114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.136+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 shutdown_connections 2024-02-10T08:45:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.136+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 >> 172.21.15.107:0/3074758529 conn(0x7f4b740f3640 msgr2=0x7f4b741077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:35.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.137+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 shutdown_connections 2024-02-10T08:45:35.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.137+0000 7f4b7a120700 1 -- 172.21.15.107:0/3074758529 wait complete. 2024-02-10T08:45:35.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:35.137+0000 7f4b7a120700 1 librados: shutdown 2024-02-10T08:45:35.139 INFO:teuthology.orchestra.run.smithi107.stderr:set 2.12 pg_upmap mapping to [3,7,1] 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.124874+0000 mon.a (mon.0) 2980 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:34.124940+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.125668+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.126135+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:34.131730+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:34.131793+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.299427+0000 mon.c (mon.2) 437 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:45:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.368028+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.107:0/3074758529' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:34.368375+0000 mon.a (mon.0) 2983 : audit [INF] from='client.53978 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.124874+0000 mon.a (mon.0) 2980 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:34.124940+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e588: 8 total, 7 up, 7 in 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.125668+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.126135+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]: dispatch 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:34.131730+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:34.131793+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.299427+0000 mon.c (mon.2) 437 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.368028+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.107:0/3074758529' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:34.368375+0000 mon.a (mon.0) 2983 : audit [INF] from='client.53978 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]: dispatch 2024-02-10T08:45:35.467 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:35.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:35.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:35.598 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:35.839 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:35] "GET /metrics HTTP/1.1" 200 191908 "" "Prometheus/2.33.4" 2024-02-10T08:45:35.845 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:35.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:35.978 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:36.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:35.125560+0000 mon.a (mon.0) 2984 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:36.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:35.125637+0000 mon.a (mon.0) 2985 : audit [INF] from='client.53978 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]': finished 2024-02-10T08:45:36.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:35.125696+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-10T08:45:36.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:35.508477+0000 mgr.y (mgr.14553) 510 : cluster [DBG] pgmap v886: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:36.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:36.125556+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e590: 8 total, 7 up, 7 in 2024-02-10T08:45:36.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:35.125560+0000 mon.a (mon.0) 2984 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:36.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:35.125637+0000 mon.a (mon.0) 2985 : audit [INF] from='client.53978 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]': finished 2024-02-10T08:45:36.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:35.125696+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-10T08:45:36.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:35.508477+0000 mgr.y (mgr.14553) 510 : cluster [DBG] pgmap v886: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:36.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:36.125556+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e590: 8 total, 7 up, 7 in 2024-02-10T08:45:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:36.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:36.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:36.323 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:36.477 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:36.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:35.125560+0000 mon.a (mon.0) 2984 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-99", "tierpool": "test-rados-api-smithi107-112490-99-cache"}]': finished 2024-02-10T08:45:36.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:35.125637+0000 mon.a (mon.0) 2985 : audit [INF] from='client.53978 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.12", "id": [3, 7, 1]}]': finished 2024-02-10T08:45:36.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:35.125696+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e589: 8 total, 7 up, 7 in 2024-02-10T08:45:36.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:35.508477+0000 mgr.y (mgr.14553) 510 : cluster [DBG] pgmap v886: 202 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 60 active+undersized, 120 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:36.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:36.125556+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e590: 8 total, 7 up, 7 in 2024-02-10T08:45:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:36.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:36.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:36.611 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:36.685 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:45:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:45:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:45:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:45:36.685 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:45:36.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:45:37.019 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:37.914 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:37.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:37.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:45:37.947 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:38.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:38.045 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:37.146770+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e591: 8 total, 7 up, 7 in 2024-02-10T08:45:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:37.150533+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:37.150861+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:37.509144+0000 mgr.y (mgr.14553) 511 : cluster [DBG] pgmap v889: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:37.146770+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e591: 8 total, 7 up, 7 in 2024-02-10T08:45:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:37.150533+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:37.150861+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:37.509144+0000 mgr.y (mgr.14553) 511 : cluster [DBG] pgmap v889: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:38.196 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:38.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:38.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:38.336 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:38.466 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:38.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:38.530 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:37.146770+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e591: 8 total, 7 up, 7 in 2024-02-10T08:45:38.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:37.150533+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:37.150861+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:38.531 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:37.509144+0000 mgr.y (mgr.14553) 511 : cluster [DBG] pgmap v889: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:38.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:38.598 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:38.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:38.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:38.979 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:39.005 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:39.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:39.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:38.141554+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:38.148426+0000 mon.a (mon.0) 2991 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:38.148483+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e592: 8 total, 7 up, 7 in 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:38.149075+0000 mon.b (mon.1) 710 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:38.149697+0000 mon.a (mon.0) 2993 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:38.946241+0000 mon.a (mon.0) 2994 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:38.141554+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:45:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:38.148426+0000 mon.a (mon.0) 2991 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:38.148483+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e592: 8 total, 7 up, 7 in 2024-02-10T08:45:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:38.149075+0000 mon.b (mon.1) 710 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:38.149697+0000 mon.a (mon.0) 2993 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:38.946241+0000 mon.a (mon.0) 2994 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:38.141554+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:38.148426+0000 mon.a (mon.0) 2991 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:38.148483+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e592: 8 total, 7 up, 7 in 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:38.149075+0000 mon.b (mon.1) 710 : audit [INF] from='client.? v1:172.21.15.107:0/1456595452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:38.149697+0000 mon.a (mon.0) 2993 : audit [INF] from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]: dispatch 2024-02-10T08:45:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:38.946241+0000 mon.a (mon.0) 2994 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- 172.21.15.107:0/1994590512 >> v1:172.21.15.107:6790/0 conn(0x7fe28c0fc910 legacy=0x7fe28c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- 172.21.15.107:0/1994590512 shutdown_connections 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- 172.21.15.107:0/1994590512 >> 172.21.15.107:0/1994590512 conn(0x7fe28c0f2510 msgr2=0x7fe28c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- 172.21.15.107:0/1994590512 shutdown_connections 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- 172.21.15.107:0/1994590512 wait complete. 2024-02-10T08:45:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 librados: starting msgr at 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 librados: starting objecter 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 Processor -- start 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- start start 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 librados: setting wanted keys 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 librados: calling monclient init 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe28c117420 con 0x7fe28c103250 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe28c1185e0 con 0x7fe28c0fc910 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.039+0000 7fe293621700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe28c1197e0 con 0x7fe28c0ff710 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe290bbc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe28c0ff710 0x7fe28c061460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53074/0 (socket says 172.21.15.107:53074) 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe290bbc700 1 -- 172.21.15.107:0/2052362244 learned_addr learned my addr 172.21.15.107:0/2052362244 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe291bbe700 1 --1- 172.21.15.107:0/2052362244 >> v1:172.21.15.107:6789/0 conn(0x7fe28c103250 0x7fe28c115b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34648/0 (socket says 172.21.15.107:34648) 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter ms_handle_connect 0x7fe28c0ff710 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter ms_handle_connect 0x7fe28c0fc910 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter ms_handle_connect 0x7fe28c103250 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930926167 0 0) 0x7fe28c1197e0 con 0x7fe28c0ff710 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2780033f0 con 0x7fe28c0ff710 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210150434 0 0) 0x7fe28c1185e0 con 0x7fe28c0fc910 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c1197e0 con 0x7fe28c0fc910 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564028240 0 0) 0x7fe28c117420 con 0x7fe28c103250 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c1185e0 con 0x7fe28c103250 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204869694 0 0) 0x7fe2780033f0 con 0x7fe28c0ff710 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28c117420 con 0x7fe28c0ff710 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500405300 0 0) 0x7fe28c1197e0 con 0x7fe28c0fc910 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2780033f0 con 0x7fe28c0fc910 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe27c002910 con 0x7fe28c0ff710 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589312003 0 0) 0x7fe28c1185e0 con 0x7fe28c103250 2024-02-10T08:45:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28c1197e0 con 0x7fe28c103250 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe288000fe0 con 0x7fe28c0fc910 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe284001c40 con 0x7fe28c103250 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2408165492 0 0) 0x7fe28c117420 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 >> v1:172.21.15.195:6789/0 conn(0x7fe28c0fc910 legacy=0x7fe28c060d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 >> v1:172.21.15.107:6789/0 conn(0x7fe28c103250 legacy=0x7fe28c115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c11a9e0 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c1187d0 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe293621700 10 client.53984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c118c40 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe293621700 1 librados: init done 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe27c002b00 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.040+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe27c005ae0 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.041+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe270004f20 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe27c018a70 con 0x7fe28c0ff710 2024-02-10T08:45:40.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(593..593 src has 420..593) v4 ==== 8247+0+0 (unknown 690181454 0 0) 0x7fe27c078dc0 con 0x7fe28c0ff710 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 10 client.53984.objecter ms_dispatch 0x7fe28c001550 osd_map(593..593 src has 420..593) v4 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 3 client.53984.objecter handle_osd_map got epochs [593,593] > 0 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 3 client.53984.objecter handle_osd_map decoding full epoch 593 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 20 client.53984.objecter dump_active .. 0 homeless 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.042+0000 7fe2827fc700 10 client.53984.objecter ms_handle_connect 0x7fe278063cb0 2024-02-10T08:45:40.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.046+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe27c04c270 con 0x7fe28c0ff710 2024-02-10T08:45:40.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:40.265+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe2700051d0 con 0x7fe28c0ff710 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:39.157388+0000 mon.a (mon.0) 2995 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:39.157433+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e593: 8 total, 7 up, 7 in 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:39.509854+0000 mgr.y (mgr.14553) 512 : cluster [DBG] pgmap v892: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:39.157388+0000 mon.a (mon.0) 2995 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:39.157433+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e593: 8 total, 7 up, 7 in 2024-02-10T08:45:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:39.509854+0000 mgr.y (mgr.14553) 512 : cluster [DBG] pgmap v892: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:39.157388+0000 mon.a (mon.0) 2995 : audit [INF] from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-99"}]': finished 2024-02-10T08:45:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:39.157433+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e593: 8 total, 7 up, 7 in 2024-02-10T08:45:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:39.509854+0000 mgr.y (mgr.14553) 512 : cluster [DBG] pgmap v892: 162 pgs: 1 active+clean+remapped, 1 peering, 21 active+undersized+degraded, 41 active+undersized, 98 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:40.864 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:45:40.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:45:40.865 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T08:45:40.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:45:40.894 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:40.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:41.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:41.031 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:41.053 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:41.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.114+0000 7fe2827fc700 1 -- 172.21.15.107:0/2052362244 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v595) v1 ==== 97+0+0 (unknown 4221013592 0 0) 0x7fe27c04a680 con 0x7fe28c0ff710 2024-02-10T08:45:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 10 librados: watch_flush enter 2024-02-10T08:45:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 10 librados: watch_flush exit 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 20 client.53984.objecter shutdown clearing up homeless session... 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 10 client.53984.objecter successfully canceled tick 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe278063cb0 legacy=0x7fe278066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.117+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 >> v1:172.21.15.107:6790/0 conn(0x7fe28c0ff710 legacy=0x7fe28c061460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.118+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 shutdown_connections 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.118+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 >> 172.21.15.107:0/2052362244 conn(0x7fe28c0f2510 msgr2=0x7fe28c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.118+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 shutdown_connections 2024-02-10T08:45:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.118+0000 7fe293621700 1 -- 172.21.15.107:0/2052362244 wait complete. 2024-02-10T08:45:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.118+0000 7fe293621700 1 librados: shutdown 2024-02-10T08:45:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:45:41.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:41.225 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:40.110145+0000 osd.3 (osd.3) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:40.110322+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:40.160864+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e594: 8 total, 7 up, 7 in 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:40.162438+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:40.183141+0000 mon.a (mon.0) 2999 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:40.266762+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2052362244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:40.267036+0000 mon.a (mon.0) 3000 : audit [INF] from='client.53984 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:41.100648+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:41.105399+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:41.105467+0000 mon.a (mon.0) 3003 : audit [INF] from='client.53984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:41.105517+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e595: 8 total, 7 up, 7 in 2024-02-10T08:45:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:41.108031+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.376 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:41.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:40.110145+0000 osd.3 (osd.3) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:40.110322+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:40.160864+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e594: 8 total, 7 up, 7 in 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:40.162438+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:40.183141+0000 mon.a (mon.0) 2999 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:40.266762+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2052362244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:40.267036+0000 mon.a (mon.0) 3000 : audit [INF] from='client.53984 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:41.100648+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:41.105399+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:41.105467+0000 mon.a (mon.0) 3003 : audit [INF] from='client.53984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:41.105517+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e595: 8 total, 7 up, 7 in 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:41.108031+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:40.110145+0000 osd.3 (osd.3) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:40.110322+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:40.160864+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e594: 8 total, 7 up, 7 in 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:40.162438+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:40.183141+0000 mon.a (mon.0) 2999 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:40.266762+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.107:0/2052362244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:40.267036+0000 mon.a (mon.0) 3000 : audit [INF] from='client.53984 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:41.100648+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:41.105399+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:41.105467+0000 mon.a (mon.0) 3003 : audit [INF] from='client.53984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:41.105517+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e595: 8 total, 7 up, 7 in 2024-02-10T08:45:41.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:41.108031+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]: dispatch 2024-02-10T08:45:41.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:41.629 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:41.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:41.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:41.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- 172.21.15.107:0/192329947 >> v1:172.21.15.107:6789/0 conn(0x7f22141007f0 legacy=0x7f2214102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:41.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- 172.21.15.107:0/192329947 shutdown_connections 2024-02-10T08:45:41.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- 172.21.15.107:0/192329947 >> 172.21.15.107:0/192329947 conn(0x7f22140f3640 msgr2=0x7f22140f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- 172.21.15.107:0/192329947 shutdown_connections 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- 172.21.15.107:0/192329947 wait complete. 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 librados: starting msgr at 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 librados: starting objecter 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 Processor -- start 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 -- start start 2024-02-10T08:45:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 librados: setting wanted keys 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.856+0000 7f221ab05700 1 librados: calling monclient init 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f221ab05700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2214117330 con 0x7f22140fda50 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f221ab05700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2214118510 con 0x7f22141007f0 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f221ab05700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2214119710 con 0x7f2214104330 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f22188a1700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f22140fda50 0x7f22140f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34654/0 (socket says 172.21.15.107:34654) 2024-02-10T08:45:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f22188a1700 1 -- 172.21.15.107:0/1870421193 learned_addr learned my addr 172.21.15.107:0/1870421193 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f22190a2700 1 --1- 172.21.15.107:0/1870421193 >> v1:172.21.15.107:6790/0 conn(0x7f2214104330 0x7f22141159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53078/0 (socket says 172.21.15.107:53078) 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter ms_handle_connect 0x7f2214104330 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter ms_handle_connect 0x7f22140fda50 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046576278 0 0) 0x7f2214119710 con 0x7f2214104330 2024-02-10T08:45:41.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204003800 con 0x7f2214104330 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter ms_handle_connect 0x7f22141007f0 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22317989 0 0) 0x7f2214117330 con 0x7f22140fda50 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2214119710 con 0x7f22140fda50 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102236790 0 0) 0x7f2204003800 con 0x7f2214104330 2024-02-10T08:45:41.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2214117330 con 0x7f2214104330 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f220c001c40 con 0x7f2214104330 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4224579046 0 0) 0x7f2214119710 con 0x7f22140fda50 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2204003800 con 0x7f22140fda50 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440152554 0 0) 0x7f2214118510 con 0x7f22141007f0 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2214119710 con 0x7f22141007f0 2024-02-10T08:45:41.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2208001850 con 0x7f22140fda50 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3777579090 0 0) 0x7f2214117330 con 0x7f2214104330 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 >> v1:172.21.15.195:6789/0 conn(0x7f22141007f0 legacy=0x7f22140fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 >> v1:172.21.15.107:6789/0 conn(0x7f22140fda50 legacy=0x7f22140f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.857+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221411a910 con 0x7f2214104330 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2214117520 con 0x7f2214104330 2024-02-10T08:45:41.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f220c002240 con 0x7f2214104330 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f220c005df0 con 0x7f2214104330 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f221ab05700 10 client.53990.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2214117a10 con 0x7f2214104330 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f221ab05700 1 librados: init done 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.858+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f221405f700 con 0x7f2214104330 2024-02-10T08:45:41.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f220c001da0 con 0x7f2214104330 2024-02-10T08:45:41.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(595..595 src has 420..595) v4 ==== 7531+0+0 (unknown 2109553434 0 0) 0x7f220c078d60 con 0x7f2214104330 2024-02-10T08:45:41.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 osd_map(595..595 src has 420..595) v4 2024-02-10T08:45:41.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 3 client.53990.objecter handle_osd_map got epochs [595,595] > 0 2024-02-10T08:45:41.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 3 client.53990.objecter handle_osd_map decoding full epoch 595 2024-02-10T08:45:41.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 20 client.53990.objecter dump_active .. 0 homeless 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.859+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v595) v1 ==== 54+0+16 (unknown 4015144498 0 2420000192) 0x7f220c07b1b0 con 0x7f2214104330 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f2204063ce0 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 10 client.53990.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 1 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 1 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 0 1 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 4 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 10 client.53990.objecter _send_command 1 2024-02-10T08:45:41.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22140475e0 con 0x7f22141186a0 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f22141186a0 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f22140475e0 con 0x7f22141186a0 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 10 client.53990.objecter _finish_command 1 = system:0 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 4 2024-02-10T08:45:41.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.860+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 0 1 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 10 client.53990.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 2 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 2 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f91e0 osd=0 3 2024-02-10T08:45:41.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 0 2 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 4 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 10 client.53990.objecter _send_command 2 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.875+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2214109f40 con 0x7f22141186a0 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2214109f40 con 0x7f22141186a0 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 10 client.53990.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f91e0 osd=0 4 2024-02-10T08:45:41.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 0 2 2024-02-10T08:45:41.942 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:41.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 10 client.53990.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 3 2024-02-10T08:45:41.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 1 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 4 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 10 client.53990.objecter _send_command 3 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22140f44c0 con 0x7f22140f40a0 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.877+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f22140f40a0 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.878+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f22140f44c0 con 0x7f22140f40a0 2024-02-10T08:45:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.878+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.878+0000 7f2211ffb700 10 client.53990.objecter _finish_command 3 = system:0 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.878+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 4 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.878+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 1 3 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 10 client.53990.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 4 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 4 2024-02-10T08:45:41.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f3a20 osd=1 3 2024-02-10T08:45:41.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 1 4 2024-02-10T08:45:41.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 4 2024-02-10T08:45:41.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 10 client.53990.objecter _send_command 4 2024-02-10T08:45:41.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.893+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f22140f48a0 con 0x7f22140f40a0 2024-02-10T08:45:41.945 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.894+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f22140f48a0 con 0x7f22140f40a0 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.894+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.894+0000 7f2211ffb700 10 client.53990.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.894+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f3a20 osd=1 4 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.894+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 1 4 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 10 client.53990.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 5 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 5 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 2 5 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 4 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 10 client.53990.objecter _send_command 5 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2214028780 con 0x7f22140282f0 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f22140282f0 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2214028780 con 0x7f22140282f0 2024-02-10T08:45:41.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 10 client.53990.objecter _finish_command 5 = system:0 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 4 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.895+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 2 5 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 10 client.53990.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 6 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.948 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:41.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 6 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22140f4a10 osd=2 3 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 2 6 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 4 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 10 client.53990.objecter _send_command 6 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.910+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2214027dc0 con 0x7f22140282f0 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.911+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2214027dc0 con 0x7f22140282f0 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.911+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.911+0000 7f2211ffb700 10 client.53990.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.911+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22140f4a10 osd=2 4 2024-02-10T08:45:41.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.911+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 2 6 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 10 client.53990.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 7 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 7 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 3 7 2024-02-10T08:45:41.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 4 2024-02-10T08:45:41.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 10 client.53990.objecter _send_command 7 2024-02-10T08:45:41.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f221405a080 con 0x7f2214124440 2024-02-10T08:45:41.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f2214124440 2024-02-10T08:45:41.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f221405a080 con 0x7f2214124440 2024-02-10T08:45:41.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:41.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 10 client.53990.objecter _finish_command 7 = system:0 2024-02-10T08:45:41.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 4 2024-02-10T08:45:41.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.912+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 3 7 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 10 client.53990.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 8 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 8 2024-02-10T08:45:41.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f2214059990 osd=3 3 2024-02-10T08:45:41.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 3 8 2024-02-10T08:45:41.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 4 2024-02-10T08:45:41.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 10 client.53990.objecter _send_command 8 2024-02-10T08:45:41.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.927+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2214059400 con 0x7f2214124440 2024-02-10T08:45:41.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2214059400 con 0x7f2214124440 2024-02-10T08:45:41.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 10 client.53990.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f2214059990 osd=3 4 2024-02-10T08:45:41.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 3 8 2024-02-10T08:45:41.959 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:41.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 10 client.53990.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 9 2024-02-10T08:45:41.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:41.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 10 client.53990.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=596}) v3 -- 0x7f22140595d0 con 0x7f2214104330 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2214059400 con 0x7f2214104330 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=595) v2 ==== 24+0+0 (unknown 1477152612 0 0) 0x7f220c04bf50 con 0x7f2214104330 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2212ffd700 10 client.53990.objecter _check_command_map_dne tid 9 current 595 map_dne_bound 595 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2212ffd700 10 client.53990.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f2212ffd700 15 client.53990.objecter _session_command_op_remove -1 9 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 10 client.53990.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 10 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 10 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 5 10 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 4 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 10 client.53990.objecter _send_command 10 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.929+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f221412cc80 con 0x7f22141290d0 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.930+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f22141290d0 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.931+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f221412cc80 con 0x7f22141290d0 2024-02-10T08:45:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.931+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.931+0000 7f2211ffb700 10 client.53990.objecter _finish_command 10 = system:0 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.931+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 4 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.931+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 5 10 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 10 client.53990.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 11 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 11 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f22141242b0 osd=5 3 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 5 11 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 4 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 10 client.53990.objecter _send_command 11 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.946+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f22141339b0 con 0x7f22141290d0 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f22141339b0 con 0x7f22141290d0 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 10 client.53990.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f22141242b0 osd=5 4 2024-02-10T08:45:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 5 11 2024-02-10T08:45:41.968 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 10 client.53990.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 12 2024-02-10T08:45:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 12 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 6 12 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 4 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 10 client.53990.objecter _send_command 12 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f221406caa0 con 0x7f221406b630 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.948+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f221406b630 2024-02-10T08:45:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.949+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f221406caa0 con 0x7f221406b630 2024-02-10T08:45:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.949+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.949+0000 7f2211ffb700 10 client.53990.objecter _finish_command 12 = system:0 2024-02-10T08:45:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.949+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 4 2024-02-10T08:45:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.949+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 6 12 2024-02-10T08:45:41.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:41.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 10 client.53990.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 13 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 13 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f221406afb0 osd=6 3 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 6 13 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 4 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 10 client.53990.objecter _send_command 13 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.964+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f221406caa0 con 0x7f221406b630 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f221406caa0 con 0x7f221406b630 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f2211ffb700 10 client.53990.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f221406afb0 osd=6 4 2024-02-10T08:45:41.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 6 13 2024-02-10T08:45:41.974 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 10 client.53990.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 14 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 14 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 7 14 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 4 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 10 client.53990.objecter _send_command 14 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.966+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2214137770 con 0x7f2214133b20 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 10 client.53990.objecter ms_handle_connect 0x7f2214133b20 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2214137770 con 0x7f2214133b20 2024-02-10T08:45:41.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:41.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 10 client.53990.objecter _finish_command 14 = system:0 2024-02-10T08:45:41.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 4 2024-02-10T08:45:41.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.967+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 7 14 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 10 client.53990.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign -1 15 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 20 client.53990.objecter _get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 15 client.53990.objecter _session_command_op_remove -1 15 2024-02-10T08:45:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 20 client.53990.objecter get_session s=0x7f221406caa0 osd=7 3 2024-02-10T08:45:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 15 client.53990.objecter _session_command_op_assign 7 15 2024-02-10T08:45:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 4 2024-02-10T08:45:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 10 client.53990.objecter _send_command 15 2024-02-10T08:45:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.984+0000 7f221ab05700 1 -- 172.21.15.107:0/1870421193 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2214138f20 con 0x7f2214133b20 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.986+0000 7f2211ffb700 1 -- 172.21.15.107:0/1870421193 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2214138f20 con 0x7f2214133b20 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.986+0000 7f2211ffb700 10 client.53990.objecter ms_dispatch 0x7f2214001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.986+0000 7f2211ffb700 10 client.53990.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.986+0000 7f2211ffb700 20 client.53990.objecter put_session s=0x7f221406caa0 osd=7 4 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:41.986+0000 7f2211ffb700 15 client.53990.objecter _session_command_op_remove 7 15 2024-02-10T08:45:41.986 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:45:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:45:42.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:42.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:42.101 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:41.127785+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:41.128118+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:41.510683+0000 mgr.y (mgr.14553) 513 : cluster [DBG] pgmap v895: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:41.658578+0000 mgr.y (mgr.14553) 514 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:41.860118+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? v1:172.21.15.107:0/1870421193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.107206+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:42.107253+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e596: 8 total, 7 up, 7 in 2024-02-10T08:45:42.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.130946+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.132016+0000 mon.a (mon.0) 3008 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.132577+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.132928+0000 mon.a (mon.0) 3009 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.133392+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:42.133651+0000 mon.a (mon.0) 3010 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:42.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:41.127785+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:41.128118+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:41.510683+0000 mgr.y (mgr.14553) 513 : cluster [DBG] pgmap v895: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:41.658578+0000 mgr.y (mgr.14553) 514 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:41.860118+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? v1:172.21.15.107:0/1870421193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:42.366 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.107206+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:42.107253+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e596: 8 total, 7 up, 7 in 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.130946+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.132016+0000 mon.a (mon.0) 3008 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.132577+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.132928+0000 mon.a (mon.0) 3009 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.133392+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.367 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:42.133651+0000 mon.a (mon.0) 3010 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:41.127785+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:41.128118+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:41.510683+0000 mgr.y (mgr.14553) 513 : cluster [DBG] pgmap v895: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:41.658578+0000 mgr.y (mgr.14553) 514 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:41.860118+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? v1:172.21.15.107:0/1870421193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.107206+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.107:0/1119880345' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi107-112490-94"}]': finished 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:42.107253+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e596: 8 total, 7 up, 7 in 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.130946+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.132016+0000 mon.a (mon.0) 3008 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.132577+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.493 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.132928+0000 mon.a (mon.0) 3009 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:42.493 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.133392+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.493 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:42.133651+0000 mon.a (mon.0) 3010 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T08:45:42.494 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:42.512 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:42.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:42.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:42.641 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:42.659 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:42.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:42.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:42.788 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:42.806 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:42.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:42.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:42.935 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:42.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:43.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:43.083 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:43.102 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:43.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:43.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:43.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:43.232 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:43.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:42.185198+0000 mon.a (mon.0) 3011 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:42.185227+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:42.185245+0000 mon.a (mon.0) 3013 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:43.111299+0000 mon.a (mon.0) 3014 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:43.111381+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e597: 8 total, 7 up, 7 in 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:43.112565+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:43.120032+0000 mon.a (mon.0) 3016 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:42.185198+0000 mon.a (mon.0) 3011 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:42.185227+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:42.185245+0000 mon.a (mon.0) 3013 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:45:43.402 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:43.111299+0000 mon.a (mon.0) 3014 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:43.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:43.111381+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e597: 8 total, 7 up, 7 in 2024-02-10T08:45:43.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:43.112565+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:43.403 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:43.120032+0000 mon.a (mon.0) 3016 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:42.185198+0000 mon.a (mon.0) 3011 : cluster [WRN] Health check update: Degraded data redundancy: 87/579 objects degraded (15.026%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:42.185227+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:42.185245+0000 mon.a (mon.0) 3013 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:43.111299+0000 mon.a (mon.0) 3014 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi107-112490-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:43.111381+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e597: 8 total, 7 up, 7 in 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:43.112565+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:43.120032+0000 mon.a (mon.0) 3016 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:44.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:43.511224+0000 mgr.y (mgr.14553) 515 : cluster [DBG] pgmap v898: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:43.511224+0000 mgr.y (mgr.14553) 515 : cluster [DBG] pgmap v898: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:43.511224+0000 mgr.y (mgr.14553) 515 : cluster [DBG] pgmap v898: 154 pgs: 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:45.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:45:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:44.375736+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e598: 8 total, 7 up, 7 in 2024-02-10T08:45:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:44.386295+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:44.938114+0000 mon.c (mon.2) 441 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:45:45.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:44.938832+0000 mon.c (mon.2) 442 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:44.945587+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:45.116854+0000 mon.a (mon.0) 3020 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:45.116944+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e599: 8 total, 7 up, 7 in 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:44.375736+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e598: 8 total, 7 up, 7 in 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:44.386295+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:44.938114+0000 mon.c (mon.2) 441 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:44.938832+0000 mon.c (mon.2) 442 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:44.945587+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:45.116854+0000 mon.a (mon.0) 3020 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:45.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:45.116944+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e599: 8 total, 7 up, 7 in 2024-02-10T08:45:45.782 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:44.375736+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e598: 8 total, 7 up, 7 in 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:44.386295+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:44.938114+0000 mon.c (mon.2) 441 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:44.938832+0000 mon.c (mon.2) 442 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:44.945587+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:45.116854+0000 mon.a (mon.0) 3020 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi107-112490-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:45.116944+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e599: 8 total, 7 up, 7 in 2024-02-10T08:45:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:45.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:45.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:45.916 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:45] "GET /metrics HTTP/1.1" 200 184641 "" "Prometheus/2.33.4" 2024-02-10T08:45:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:46.167 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:46.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:46.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:46.298 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:45.511840+0000 mgr.y (mgr.14553) 516 : cluster [DBG] pgmap v901: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:46.114890+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e600: 8 total, 7 up, 7 in 2024-02-10T08:45:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:46.134901+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:46.137123+0000 mon.a (mon.0) 3023 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:45.511840+0000 mgr.y (mgr.14553) 516 : cluster [DBG] pgmap v901: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:46.114890+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e600: 8 total, 7 up, 7 in 2024-02-10T08:45:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:46.134901+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:46.137123+0000 mon.a (mon.0) 3023 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:45.511840+0000 mgr.y (mgr.14553) 516 : cluster [DBG] pgmap v901: 162 pgs: 8 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/579 objects degraded (15.026%) 2024-02-10T08:45:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:46.114890+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e600: 8 total, 7 up, 7 in 2024-02-10T08:45:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:46.134901+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:46.137123+0000 mon.a (mon.0) 3023 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:47.494 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:45:47.494 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.377+0000 7f875b64c700 1 -- 172.21.15.107:0/308855931 >> v1:172.21.15.195:6789/0 conn(0x7f8754104330 legacy=0x7f87541067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.377+0000 7f875b64c700 1 -- 172.21.15.107:0/308855931 shutdown_connections 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.377+0000 7f875b64c700 1 -- 172.21.15.107:0/308855931 >> 172.21.15.107:0/308855931 conn(0x7f87540f3640 msgr2=0x7f87540f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.377+0000 7f875b64c700 1 -- 172.21.15.107:0/308855931 shutdown_connections 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.377+0000 7f875b64c700 1 -- 172.21.15.107:0/308855931 wait complete. 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 librados: starting msgr at 2024-02-10T08:45:47.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 librados: starting objecter 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 Processor -- start 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 -- start start 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 librados: setting wanted keys 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 librados: calling monclient init 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87540fcf70 con 0x7f8754104330 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87540fd140 con 0x7f87540fda50 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f875b64c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87540fd310 con 0x7f87541007f0 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f8758be7700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f87541007f0 0x7f8754114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53126/0 (socket says 172.21.15.107:53126) 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f8758be7700 1 -- 172.21.15.107:0/3171995154 learned_addr learned my addr 172.21.15.107:0/3171995154 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter ms_handle_connect 0x7f8754104330 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter ms_handle_connect 0x7f87541007f0 2024-02-10T08:45:47.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857930034 0 0) 0x7f87540fd310 con 0x7f87541007f0 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8740003800 con 0x7f87541007f0 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter ms_handle_connect 0x7f87540fda50 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020101027 0 0) 0x7f87540fcf70 con 0x7f8754104330 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87540fd310 con 0x7f8754104330 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.378+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941526117 0 0) 0x7f87540fd140 con 0x7f87540fda50 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87540fcf70 con 0x7f87540fda50 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411673419 0 0) 0x7f8740003800 con 0x7f87541007f0 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87540fd140 con 0x7f87541007f0 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8744001910 con 0x7f87541007f0 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 678992020 0 0) 0x7f87540fd310 con 0x7f8754104330 2024-02-10T08:45:47.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8740003800 con 0x7f8754104330 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f874c002f20 con 0x7f8754104330 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966390950 0 0) 0x7f87540fcf70 con 0x7f87540fda50 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87540fd310 con 0x7f87540fda50 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f87500019f0 con 0x7f87540fda50 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4200899378 0 0) 0x7f87540fd140 con 0x7f87541007f0 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 >> v1:172.21.15.195:6789/0 conn(0x7f87540fda50 legacy=0x7f87540fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 >> v1:172.21.15.107:6789/0 conn(0x7f8754104330 legacy=0x7f87541184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875411cc00 con 0x7f87541007f0 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8754119c30 con 0x7f87541007f0 2024-02-10T08:45:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 601, 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:45:47.113189+0000", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:43:35.150516+0000", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:43:14.076553+0000", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2024-02-10T08:45:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 107, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 235, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:45:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "552", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:45:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:45:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f875b64c700 10 client.53999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f875411a0a0 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f875b64c700 1 librados: init done 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8744002470 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.379+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8744005b40 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.380+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8738004f20 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.380+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8744005ca0 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(601..601 src has 420..601) v4 ==== 8630+0+0 (unknown 3651154553 0 0) 0x7f8744078da0 con 0x7f87541007f0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 10 client.53999.objecter ms_dispatch 0x7f8754001550 osd_map(601..601 src has 420..601) v4 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 3 client.53999.objecter handle_osd_map got epochs [601,601] > 0 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 3 client.53999.objecter handle_osd_map decoding full epoch 601 2024-02-10T08:45:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 20 client.53999.objecter dump_active .. 0 homeless 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.381+0000 7f874a7fc700 10 client.53999.objecter ms_handle_connect 0x7f8740063d10 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.385+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f874404bf80 con 0x7f87541007f0 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.558+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f87380051d0 con 0x7f87541007f0 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.559+0000 7f874a7fc700 1 -- 172.21.15.107:0/3171995154 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v601) v1 ==== 81+0+49133 (unknown 3790378520 0 592883124) 0x7f87440025e0 con 0x7f87541007f0 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.561+0000 7f875b64c700 10 librados: watch_flush enter 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.561+0000 7f875b64c700 10 librados: watch_flush exit 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.561+0000 7f875b64c700 20 client.53999.objecter shutdown clearing up homeless session... 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.561+0000 7f875b64c700 10 client.53999.objecter successfully canceled tick 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:45:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 234, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-101", 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:45:44.375681+0000", 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "601", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi107-112490-101", 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 235, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-102", 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:45:45.394904+0000", 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:45:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "601", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:45:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:45:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 134, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.9418182373046875, 2024-02-10T08:45:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:45:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 0, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 31, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 478, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 480, 2024-02-10T08:45:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:45:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:45:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8740063d10 legacy=0x7f87400661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 >> v1:172.21.15.107:6790/0 conn(0x7f87541007f0 legacy=0x7f8754114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 shutdown_connections 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 >> 172.21.15.107:0/3171995154 conn(0x7f87540f3640 msgr2=0x7f87541077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 shutdown_connections 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 -- 172.21.15.107:0/3171995154 wait complete. 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:47.562+0000 7f875b64c700 1 librados: shutdown 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2499671285 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/2499671285", 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/2499671285", 2024-02-10T08:45:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/2499671285", 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/2499671285", 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "exists" 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:45:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:45:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:45:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:45:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 600, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:45:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:45:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:34.131796+0000", 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:44:20.626620+0000", 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:41.128121+0000", 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:40.110324+0000", 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:43:23.143436+0000", 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 479 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:29.056895+0000", 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:17.814100+0000", 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:45:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:18.137156+0000", 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:45:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.12", 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: 7, 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: 1 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "6.1", 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.0", 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1d", 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:45:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.f", 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 3 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [ 2024-02-10T08:45:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: 7, 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: 3 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:45:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-test-rados-api-smithi107-112490-101": { 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:45:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:45:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:45:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:45:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:45:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:45:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:45:47.727 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:45:48.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:45:48.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:47.119113+0000 mon.a (mon.0) 3024 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:48.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:47.119190+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e601: 8 total, 7 up, 7 in 2024-02-10T08:45:48.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:47.205231+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:47.206464+0000 mon.a (mon.0) 3026 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:47.512443+0000 mgr.y (mgr.14553) 517 : cluster [DBG] pgmap v904: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:47.559359+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.107:0/3171995154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:47.119113+0000 mon.a (mon.0) 3024 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:47.119190+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e601: 8 total, 7 up, 7 in 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:47.205231+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:47.206464+0000 mon.a (mon.0) 3026 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:47.512443+0000 mgr.y (mgr.14553) 517 : cluster [DBG] pgmap v904: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:48.234 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:47.559359+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.107:0/3171995154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:48.390 INFO:tasks.thrashosds.thrasher:j is {'epoch': 601, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:45:47.113189+0000', 'last_up_change': '2024-02-10T08:43:35.150516+0000', 'last_in_change': '2024-02-10T08:43:14.076553+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 107, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 235, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '552', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi107-112490-101', 'create_time': '2024-02-10T08:45:44.375681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '601', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi107-112490-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 235, 'pool_name': 'test-rados-api-smithi107-112490-102', 'create_time': '2024-02-10T08:45:45.394904+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '601', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 0.9418182373046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 478, 'down_at': 480, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 2499671285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 2499671285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 2499671285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 2499671285}]}, 'public_addr': '172.21.15.195:6800/2499671285', 'cluster_addr': '172.21.15.195:6801/2499671285', 'heartbeat_back_addr': '172.21.15.195:6803/2499671285', 'heartbeat_front_addr': '172.21.15.195:6802/2499671285', 'state': ['exists']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:45:34.131796+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:44:20.626620+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:41.128121+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:40.110324+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:43:23.143436+0000', 'dead_epoch': 479}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:29.056895+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:17.814100+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:18.137156+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '2.12', 'osds': [3, 7, 1]}, {'pgid': '2.13', 'osds': [4, 2, 7]}, {'pgid': '4.1', 'osds': [4, 0, 7]}, {'pgid': '6.1', 'osds': [2, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.0', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '16.f', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_temp': [{'pgid': '2.13', 'osds': [2, 7, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi107-112490-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:45:48.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:47.119113+0000 mon.a (mon.0) 3024 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:47.119190+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e601: 8 total, 7 up, 7 in 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:47.205231+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:47.206464+0000 mon.a (mon.0) 3026 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:47.512443+0000 mgr.y (mgr.14553) 517 : cluster [DBG] pgmap v904: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:47.559359+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.107:0/3171995154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:45:48.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:45:48.940 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:48.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:49.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:49.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:49.113 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:49.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:49.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:49.255 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:45:49.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:48.123100+0000 mon.a (mon.0) 3027 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:49.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:48.123184+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e602: 8 total, 7 up, 7 in 2024-02-10T08:45:49.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:48.124899+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:48.125217+0000 mon.a (mon.0) 3029 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.274 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:49.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:48.123100+0000 mon.a (mon.0) 3027 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:49.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:48.123184+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e602: 8 total, 7 up, 7 in 2024-02-10T08:45:49.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:48.124899+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:48.125217+0000 mon.a (mon.0) 3029 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.277 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:48.123100+0000 mon.a (mon.0) 3027 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:49.277 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:48.123184+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e602: 8 total, 7 up, 7 in 2024-02-10T08:45:49.277 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:48.124899+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.277 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:48.125217+0000 mon.a (mon.0) 3029 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:49.296 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:49.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:49.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:49.627 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:49.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:49.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:49.762 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:49.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:49.969 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:50.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:49.124104+0000 mon.a (mon.0) 3030 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:49.124174+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-10T08:45:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:49.259809+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:49.260208+0000 mon.a (mon.0) 3032 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:49.513268+0000 mgr.y (mgr.14553) 518 : cluster [DBG] pgmap v907: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:49.124104+0000 mon.a (mon.0) 3030 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:49.124174+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:49.259809+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:49.260208+0000 mon.a (mon.0) 3032 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:49.513268+0000 mgr.y (mgr.14553) 518 : cluster [DBG] pgmap v907: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:50.536 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:45:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:49.124104+0000 mon.a (mon.0) 3030 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:49.124174+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-10T08:45:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:49.259809+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:49.260208+0000 mon.a (mon.0) 3032 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:49.513268+0000 mgr.y (mgr.14553) 518 : cluster [DBG] pgmap v907: 194 pgs: 1 creating+activating, 1 active+clean+remapped, 22 creating+peering, 15 unknown, 21 active+undersized+degraded, 1 creating+activating+undersized, 42 active+undersized, 91 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/579 objects degraded (15.026%) 2024-02-10T08:45:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:50.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:50.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:50.785 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:51.083 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:45:51.083 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:45:51.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:50.048250+0000 osd.1 (osd.1) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:51.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:50.048445+0000 osd.1 (osd.1) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:51.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:50.147892+0000 mon.a (mon.0) 3033 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:51.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:50.147951+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-10T08:45:51.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:50.152549+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:50.154897+0000 mon.a (mon.0) 3035 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:51.108564+0000 mon.a (mon.0) 3036 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:51.446 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:51.108637+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:50.048250+0000 osd.1 (osd.1) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:50.048445+0000 osd.1 (osd.1) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:50.147892+0000 mon.a (mon.0) 3033 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:50.147951+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:50.152549+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:50.154897+0000 mon.a (mon.0) 3035 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:51.108564+0000 mon.a (mon.0) 3036 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:51.108637+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:50.048250+0000 osd.1 (osd.1) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:50.048445+0000 osd.1 (osd.1) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:50.147892+0000 mon.a (mon.0) 3033 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:50.147951+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:50.152549+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:50.154897+0000 mon.a (mon.0) 3035 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]: dispatch 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:51.108564+0000 mon.a (mon.0) 3036 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-102"}]': finished 2024-02-10T08:45:51.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:51.108637+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-10T08:45:51.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:45:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:45:51.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:45:51.835 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:51.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:51.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:51.966 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:52.001 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:45:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:52.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:52.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:52.327 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:51.514116+0000 mgr.y (mgr.14553) 519 : cluster [DBG] pgmap v910: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 58 active+undersized, 113 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:52.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:51.669271+0000 mgr.y (mgr.14553) 520 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:52.531 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:52.533 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:51.514116+0000 mgr.y (mgr.14553) 519 : cluster [DBG] pgmap v910: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 58 active+undersized, 113 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:52.533 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:51.669271+0000 mgr.y (mgr.14553) 520 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:52.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:51.514116+0000 mgr.y (mgr.14553) 519 : cluster [DBG] pgmap v910: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 58 active+undersized, 113 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:52.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:51.669271+0000 mgr.y (mgr.14553) 520 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:45:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:52.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:52.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:52.662 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:52.681 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:52.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:52.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:53.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:53.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:53.293 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:53.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:52.196343+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:53.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:52.196379+0000 mon.a (mon.0) 3039 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:53.405 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:52.206711+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-10T08:45:53.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:52.196343+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:53.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:52.196379+0000 mon.a (mon.0) 3039 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:53.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:52.206711+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-10T08:45:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:52.196343+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:52.196379+0000 mon.a (mon.0) 3039 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:45:53.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:52.206711+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-10T08:45:53.469 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:53.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:53.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/1942143574 >> v1:172.21.15.107:6790/0 conn(0x7fbb240fc050 legacy=0x7fbb240fc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/1942143574 shutdown_connections 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/1942143574 >> 172.21.15.107:0/1942143574 conn(0x7fbb2406fe70 msgr2=0x7fbb240f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/1942143574 shutdown_connections 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/1942143574 wait complete. 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 librados: starting msgr at 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.594+0000 7fbb2b7d1700 1 librados: starting objecter 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 Processor -- start 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 -- start start 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 librados: setting wanted keys 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 librados: calling monclient init 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb24115150 con 0x7fbb241029b0 2024-02-10T08:45:53.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb24116350 con 0x7fbb240feea0 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2b7d1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbb24117550 con 0x7fbb240fc050 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2956d700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fbb240fc050 0x7fbb240fb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54070/0 (socket says 172.21.15.107:54070) 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb2956d700 1 -- 172.21.15.107:0/3108807769 learned_addr learned my addr 172.21.15.107:0/3108807769 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb240fc050 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb241029b0 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter ms_handle_connect 0x7fbb240feea0 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689798272 0 0) 0x7fbb24117550 con 0x7fbb240fc050 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb10004500 con 0x7fbb240fc050 2024-02-10T08:45:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501187970 0 0) 0x7fbb24115150 con 0x7fbb241029b0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb24117550 con 0x7fbb241029b0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928623501 0 0) 0x7fbb24116350 con 0x7fbb240feea0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb24115150 con 0x7fbb240feea0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326236758 0 0) 0x7fbb10004500 con 0x7fbb240fc050 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb24116350 con 0x7fbb240fc050 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb20002060 con 0x7fbb240fc050 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.595+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4255947123 0 0) 0x7fbb24117550 con 0x7fbb241029b0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb10004500 con 0x7fbb241029b0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb1c001c40 con 0x7fbb241029b0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262466311 0 0) 0x7fbb24115150 con 0x7fbb240feea0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb24117550 con 0x7fbb240feea0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb140019d0 con 0x7fbb240feea0 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2974983433 0 0) 0x7fbb24116350 con 0x7fbb240fc050 2024-02-10T08:45:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 >> v1:172.21.15.195:6789/0 conn(0x7fbb240feea0 legacy=0x7fbb2410fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 >> v1:172.21.15.107:6789/0 conn(0x7fbb241029b0 legacy=0x7fbb24113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24118750 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb24115340 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb2b7d1700 10 client.54005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb241157b0 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb2b7d1700 1 librados: init done 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.596+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbb2405f780 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.597+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbb20002300 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.597+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbb20003750 con 0x7fbb240fc050 2024-02-10T08:45:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.597+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbb20003a10 con 0x7fbb240fc050 2024-02-10T08:45:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(607..607 src has 420..607) v4 ==== 8617+0+0 (unknown 2004016381 0 0) 0x7fbb200793a0 con 0x7fbb240fc050 2024-02-10T08:45:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 osd_map(607..607 src has 420..607) v4 2024-02-10T08:45:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 3 client.54005.objecter handle_osd_map got epochs [607,607] > 0 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 3 client.54005.objecter handle_osd_map decoding full epoch 607 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 20 client.54005.objecter dump_active .. 0 homeless 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v607) v1 ==== 54+0+16 (unknown 2475250142 0 2420000192) 0x7fbb20079730 con 0x7fbb240fc050 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb10064d90 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 1 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 1 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 0 1 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 4 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 1 2024-02-10T08:45:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb240f78d0 con 0x7fbb241164e0 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.598+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb241164e0 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.599+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb240f78d0 con 0x7fbb241164e0 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.599+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.599+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 1 = system:0 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.599+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 4 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.599+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 0 1 2024-02-10T08:45:53.604 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 2 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 2 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb240475e0 osd=0 3 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 0 2 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 4 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 2 2024-02-10T08:45:53.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.613+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb24107d50 con 0x7fbb241164e0 2024-02-10T08:45:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb24107d50 con 0x7fbb241164e0 2024-02-10T08:45:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb240475e0 osd=0 4 2024-02-10T08:45:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 0 2 2024-02-10T08:45:53.616 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 3 2024-02-10T08:45:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 1 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 4 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 3 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb241038b0 con 0x7fbb241034b0 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.615+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb241034b0 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.616+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb241038b0 con 0x7fbb241034b0 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.616+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.616+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 3 = system:0 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.616+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 4 2024-02-10T08:45:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.616+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 1 3 2024-02-10T08:45:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.630+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 4 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 4 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24102e30 osd=1 3 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 1 4 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 4 2024-02-10T08:45:53.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 4 2024-02-10T08:45:53.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.631+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb24103c60 con 0x7fbb241034b0 2024-02-10T08:45:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:53.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb24103c60 con 0x7fbb241034b0 2024-02-10T08:45:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24102e30 osd=1 4 2024-02-10T08:45:53.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 1 4 2024-02-10T08:45:53.634 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 5 2024-02-10T08:45:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 5 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 2 5 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 4 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 5 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb2411c9c0 con 0x7fbb240282f0 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.632+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb240282f0 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.633+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb2411c9c0 con 0x7fbb240282f0 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.633+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.633+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 5 = system:0 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.633+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 4 2024-02-10T08:45:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.633+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 2 5 2024-02-10T08:45:53.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 6 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 6 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24103e30 osd=2 3 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 2 6 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 4 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 6 2024-02-10T08:45:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.647+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb24125710 con 0x7fbb240282f0 2024-02-10T08:45:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb24125710 con 0x7fbb240282f0 2024-02-10T08:45:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24103e30 osd=2 4 2024-02-10T08:45:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 2 6 2024-02-10T08:45:53.650 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 7 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 7 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 3 7 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 4 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 7 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb24059f40 con 0x7fbb24027ed0 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.649+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb24027ed0 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.650+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb24059f40 con 0x7fbb24027ed0 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.650+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.650+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 7 = system:0 2024-02-10T08:45:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.650+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 4 2024-02-10T08:45:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.650+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 3 7 2024-02-10T08:45:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 8 2024-02-10T08:45:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.668 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 8 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24027d60 osd=3 3 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 3 8 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 4 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 8 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.664+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb2405a180 con 0x7fbb24027ed0 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb2405a180 con 0x7fbb24027ed0 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24027d60 osd=3 4 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 3 8 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 9 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 10 client.54005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=608}) v3 -- 0x7fbb24059400 con 0x7fbb240fc050 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb24059780 con 0x7fbb240fc050 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=607) v2 ==== 24+0+0 (unknown 401214632 0 0) 0x7fbb2004b7f0 con 0x7fbb240fc050 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1b7fe700 10 client.54005.objecter _check_command_map_dne tid 9 current 607 map_dne_bound 607 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1b7fe700 10 client.54005.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb1b7fe700 15 client.54005.objecter _session_command_op_remove -1 9 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 10 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 10 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 5 10 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 4 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 10 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.666+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb2412edc0 con 0x7fbb2412b2b0 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb2412b2b0 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb2412edc0 con 0x7fbb2412b2b0 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 10 = system:0 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 4 2024-02-10T08:45:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.667+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 5 10 2024-02-10T08:45:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 11 2024-02-10T08:45:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 11 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb24126290 osd=5 3 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 5 11 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 4 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 11 2024-02-10T08:45:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.682+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb2412edc0 con 0x7fbb2412b2b0 2024-02-10T08:45:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb2412edc0 con 0x7fbb2412b2b0 2024-02-10T08:45:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb24126290 osd=5 4 2024-02-10T08:45:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 5 11 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 12 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 12 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 6 12 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 4 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 12 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb241330f0 con 0x7fbb2412f4a0 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.684+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb2412f4a0 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.685+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb241330f0 con 0x7fbb2412f4a0 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.685+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.685+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 12 = system:0 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.685+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 4 2024-02-10T08:45:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.685+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 6 12 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 13 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 13 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb2412edc0 osd=6 3 2024-02-10T08:45:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 6 13 2024-02-10T08:45:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 4 2024-02-10T08:45:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 13 2024-02-10T08:45:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.700+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb241330f0 con 0x7fbb2412f4a0 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.701+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb241330f0 con 0x7fbb2412f4a0 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.701+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.701+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.701+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb2412edc0 osd=6 4 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.701+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 6 13 2024-02-10T08:45:53.702 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 14 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 14 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 7 14 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 4 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 14 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb24134340 con 0x7fbb24134df0 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.702+0000 7fbb1a7fc700 10 client.54005.objecter ms_handle_connect 0x7fbb24134df0 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.703+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbb24134340 con 0x7fbb24134df0 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.703+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.703+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 14 = system:0 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.703+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 4 2024-02-10T08:45:53.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.703+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 7 14 2024-02-10T08:45:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 10 client.54005.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:45:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign -1 15 2024-02-10T08:45:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 20 client.54005.objecter _get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_remove -1 15 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 20 client.54005.objecter get_session s=0x7fbb241330f0 osd=7 3 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 15 client.54005.objecter _session_command_op_assign 7 15 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 4 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 10 client.54005.objecter _send_command 15 2024-02-10T08:45:53.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.717+0000 7fbb2b7d1700 1 -- 172.21.15.107:0/3108807769 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb24134340 con 0x7fbb24134df0 2024-02-10T08:45:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.719+0000 7fbb1a7fc700 1 -- 172.21.15.107:0/3108807769 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbb24134340 con 0x7fbb24134df0 2024-02-10T08:45:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.719+0000 7fbb1a7fc700 10 client.54005.objecter ms_dispatch 0x7fbb24001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:45:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.719+0000 7fbb1a7fc700 10 client.54005.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.719+0000 7fbb1a7fc700 20 client.54005.objecter put_session s=0x7fbb241330f0 osd=7 4 2024-02-10T08:45:53.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.719+0000 7fbb1a7fc700 15 client.54005.objecter _session_command_op_remove 7 15 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:45:53.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:45:53.742 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:45:53.742 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:45:53.742 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:45:53.742 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:45:53.742 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:45:53.760 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:53.779 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:45:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:53.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:45:53.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:53.912 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:53.950 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:53.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- 172.21.15.107:0/1307562196 >> v1:172.21.15.107:6789/0 conn(0x7f48540ff710 legacy=0x7f4854101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- 172.21.15.107:0/1307562196 shutdown_connections 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- 172.21.15.107:0/1307562196 >> 172.21.15.107:0/1307562196 conn(0x7f48540f2510 msgr2=0x7f48540f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- 172.21.15.107:0/1307562196 shutdown_connections 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- 172.21.15.107:0/1307562196 wait complete. 2024-02-10T08:45:54.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 librados: starting msgr at 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 librados: starting objecter 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 Processor -- start 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 -- start start 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 librados: setting wanted keys 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.865+0000 7f485b232700 1 librados: calling monclient init 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f485b232700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4854061860 con 0x7f48540fc910 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f485b232700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4854118720 con 0x7f48540ff710 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f485b232700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48541198c0 con 0x7f4854103250 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4858fce700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f48540fc910 0x7f4854060e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60114/0 (socket says 172.21.15.107:60114) 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4858fce700 1 -- 172.21.15.107:0/186759458 learned_addr learned my addr 172.21.15.107:0/186759458 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter ms_handle_connect 0x7f48540fc910 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter ms_handle_connect 0x7f4854103250 2024-02-10T08:45:54.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664985494 0 0) 0x7f4854061860 con 0x7f48540fc910 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4844003800 con 0x7f48540fc910 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861001185 0 0) 0x7f48541198c0 con 0x7f4854103250 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854061860 con 0x7f4854103250 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter ms_handle_connect 0x7f48540ff710 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126688840 0 0) 0x7f4844003800 con 0x7f48540fc910 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48541198c0 con 0x7f48540fc910 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4848002a10 con 0x7f48540fc910 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878546502 0 0) 0x7f4854061860 con 0x7f4854103250 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4844003800 con 0x7f4854103250 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143123877 0 0) 0x7f4854118720 con 0x7f48540ff710 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854061860 con 0x7f48540ff710 2024-02-10T08:45:54.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f484c0019a0 con 0x7f4854103250 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3376782723 0 0) 0x7f48541198c0 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 >> v1:172.21.15.107:6790/0 conn(0x7f4854103250 legacy=0x7f4854115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 >> v1:172.21.15.195:6789/0 conn(0x7f48540ff710 legacy=0x7f48541128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485411aa60 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.866+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854119ab0 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.867+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4848002d60 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.867+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4848005cb0 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.867+0000 7f485b232700 10 client.28437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.867+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4854119f20 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.867+0000 7f485b232700 1 librados: init done 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485405f7a0 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4848001720 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(607..607 src has 420..607) v4 ==== 8617+0+0 (unknown 2004016381 0 0) 0x7f48480791a0 con 0x7f48540fc910 2024-02-10T08:45:54.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f4851ffb700 10 client.28437.objecter ms_dispatch 0x7f4854001550 osd_map(607..607 src has 420..607) v4 2024-02-10T08:45:54.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f4851ffb700 3 client.28437.objecter handle_osd_map got epochs [607,607] > 0 2024-02-10T08:45:54.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.868+0000 7f4851ffb700 3 client.28437.objecter handle_osd_map decoding full epoch 607 2024-02-10T08:45:54.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.869+0000 7f4851ffb700 20 client.28437.objecter dump_active .. 0 homeless 2024-02-10T08:45:54.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.869+0000 7f4851ffb700 10 client.28437.objecter ms_handle_connect 0x7f4844063c80 2024-02-10T08:45:54.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:53.873+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f484804c3a0 con 0x7f48540fc910 2024-02-10T08:45:54.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:45:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.054+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f4854108e30 con 0x7f48540fc910 2024-02-10T08:45:54.082 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:54.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.217+0000 7f4851ffb700 1 -- 172.21.15.107:0/186759458 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v608) v1 ==== 91+0+0 (unknown 3686438009 0 0) 0x7f484804a810 con 0x7f48540fc910 2024-02-10T08:45:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.219+0000 7f485b232700 10 librados: watch_flush enter 2024-02-10T08:45:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.219+0000 7f485b232700 10 librados: watch_flush exit 2024-02-10T08:45:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.219+0000 7f485b232700 20 client.28437.objecter shutdown clearing up homeless session... 2024-02-10T08:45:54.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.219+0000 7f485b232700 10 client.28437.objecter successfully canceled tick 2024-02-10T08:45:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:45:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.226+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4844063c80 legacy=0x7f4844066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.226+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 >> v1:172.21.15.107:6789/0 conn(0x7f48540fc910 legacy=0x7f4854060e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.228+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 shutdown_connections 2024-02-10T08:45:54.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.228+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 >> 172.21.15.107:0/186759458 conn(0x7f48540f2510 msgr2=0x7f48541066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:45:54.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.249+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 shutdown_connections 2024-02-10T08:45:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.255+0000 7f485b232700 1 -- 172.21.15.107:0/186759458 wait complete. 2024-02-10T08:45:54.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:54.261+0000 7f485b232700 1 librados: shutdown 2024-02-10T08:45:54.283 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:53.210768+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:53.240638+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:53.241004+0000 mon.a (mon.0) 3042 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:53.515034+0000 mgr.y (mgr.14553) 521 : cluster [DBG] pgmap v913: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:53.597948+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.107:0/3108807769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:54.333 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:54.055521+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:54.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:54.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:45:54.373 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:54.392 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:54.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:53.210768+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-10T08:45:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:53.240638+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:53.241004+0000 mon.a (mon.0) 3042 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:53.515034+0000 mgr.y (mgr.14553) 521 : cluster [DBG] pgmap v913: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:54.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:53.597948+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.107:0/3108807769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:54.055521+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:53.210768+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:53.240638+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:53.241004+0000 mon.a (mon.0) 3042 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:53.515034+0000 mgr.y (mgr.14553) 521 : cluster [DBG] pgmap v913: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 87/582 objects degraded (14.948%) 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:53.597948+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.107:0/3108807769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:45:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:54.055521+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:45:54.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:54.522 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:54.543 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:54.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:54.642 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:45:54.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:55.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:45:55.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:45:55.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:55.134 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:55.153 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:45:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:55.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:45:55.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:55.304 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:55.382 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:55.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.208672+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:54.217888+0000 mon.a (mon.0) 3045 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:54.217979+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.218019+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:54.275026+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:54.281941+0000 mon.a (mon.0) 3048 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:55.168730+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.208672+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:54.217888+0000 mon.a (mon.0) 3045 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:54.217979+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.218019+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:54.275026+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:54.281941+0000 mon.a (mon.0) 3048 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:55.168730+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:55.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:45:55.526 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:55.526 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.208672+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:45:55.526 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:54.217888+0000 mon.a (mon.0) 3045 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:45:55.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:54.217979+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? v1:172.21.15.107:0/186759458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:45:55.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.218019+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-10T08:45:55.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:54.275026+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:54.281941+0000 mon.a (mon.0) 3048 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:45:55.527 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:55.168730+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:45:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:45:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:45:55] "GET /metrics HTTP/1.1" 200 189766 "" "Prometheus/2.33.4" 2024-02-10T08:45:56.326 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:45:56.326 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:45:56.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:45:56.326 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:45:56.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:735 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 569.79690491700001, 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999, 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 735 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526845+0000", 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526842+0000", 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2619999999999999e-05 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526855+0000", 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5109e-05 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526880+0000", 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1497e-05 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526891+0000", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017202499999999999 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527063+0000", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3579999999999997e-06 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527071+0000", 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62625176299999996 2024-02-10T08:45:56.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153322+0000", 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0579e-05 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153353+0000", 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62650757099999999 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.331 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:736 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected e97)", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 569.79683991800005, 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998, 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 736 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526910+0000", 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526909+0000", 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.87e-06 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526914+0000", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7660000000000004e-06 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526923+0000", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7569999999999998e-06 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.526930+0000", 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016634899999999999 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527096+0000", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3100000000000001e-06 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:26.527104+0000", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62646392500000003 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153568+0000", 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1849999999999998e-06 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:27.153575+0000", 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62666498199999998 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:737 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 3] snapc 0=[] ondisk+write+known_if_redirected e99)", 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.17202068400002, 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172, 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 737 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151730+0000", 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151728+0000", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.536e-06 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151734+0000", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.661e-06 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151741+0000", 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0990000000000004e-06 2024-02-10T08:45:56.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.151747+0000", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069385099999999999 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152441+0000", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7069999999999998e-06 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:28.152450+0000", 2024-02-10T08:45:56.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0056702749999999 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158120+0000", 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000171347 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.158292+0000", 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000874083 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:29.159166+0000", 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.007436172 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35219.0:6 51.6s0 51:6ab6d0da:test-rados-api-smithi107-112451-13::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 549.42805629600002, 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344, 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35219", 2024-02-10T08:45:56.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2138383223", 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 6 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895694+0000", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895691+0000", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0546e-05 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895702+0000", 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0954999999999999e-05 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895713+0000", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0730000000000005e-06 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.895721+0000", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012145484999999999 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907866+0000", 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101523 2024-02-10T08:45:56.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.907968+0000", 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015504442 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:46.923472+0000", 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027778344 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.345 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25276.0:8 56.1 56:bb1f5efa:test-rados-api-smithi107-112562-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 545.35699852599998, 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999, 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25276", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2938278666", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 8 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966752+0000", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966750+0000", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4189999999999999e-06 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966756+0000", 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4120000000000004e-06 2024-02-10T08:45:56.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966764+0000", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5940000000000004e-06 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966768+0000", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9002999999999999e-05 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966807+0000", 2024-02-10T08:45:56.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016647400000000001 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.966974+0000", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012392504 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979366+0000", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6769999999999999e-05 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979443+0000", 2024-02-10T08:45:56.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110282 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979553+0000", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036667200000000002 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.979920+0000", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013358821 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.993279+0000", 2024-02-10T08:45:56.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025924506 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019203+0000", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9867999999999998e-05 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019253+0000", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0630999999999998e-05 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.019294+0000", 2024-02-10T08:45:56.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052542287999999999 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.43309.0:2 90.13 90:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 521.26832405100004, 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514, 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.43309", 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4276406080", 2024-02-10T08:45:56.352 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055426+0000", 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055424+0000", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2330000000000001e-05 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055437+0000", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5120000000000002e-06 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055444+0000", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3789999999999997e-06 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055450+0000", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2941000000000002e-05 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055482+0000", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5740000000000003e-06 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.055488+0000", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016882637999999998 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072371+0000", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010682 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072477+0000", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115103 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072593+0000", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031995 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.072913+0000", 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013010588 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.085923+0000", 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012255667 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098179+0000", 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1485e-05 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098220+0000", 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6390999999999998e-05 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:15.098257+0000", 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042830514 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.358 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:4 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e146)", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 516.37719185000003, 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001, 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 4 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:45:56.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946558+0000", 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946556+0000", 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7836e-05 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946574+0000", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7283999999999999e-05 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946591+0000", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.516e-06 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946601+0000", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8421000000000001e-05 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946669+0000", 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5779999999999999e-06 2024-02-10T08:45:56.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.946678+0000", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098724250000000006 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956550+0000", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012298100000000001 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.956673+0000", 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00064271 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.957316+0000", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011186949999999999 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.968503+0000", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01058679 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979089+0000", 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3553000000000005e-05 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:19.979173+0000", 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032614625000000001 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44944.0:9 93.6s0 93:62aeb327:test-rados-api-smithi107-112513-15::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 514.39991493299999, 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001, 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44944", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/21513973", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923835+0000", 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923833+0000", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7350000000000002e-06 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923842+0000", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6348999999999999e-05 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923858+0000", 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8599999999999999e-06 2024-02-10T08:45:56.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923867+0000", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7614000000000001e-05 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923925+0000", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0160000000000008e-06 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.923933+0000", 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088728850000000005 2024-02-10T08:45:56.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932805+0000", 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9782000000000001e-05 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.932905+0000", 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040048699999999999 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.933306+0000", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075809029999999999 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.940887+0000", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011264712 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952151+0000", 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8347000000000001e-05 2024-02-10T08:45:56.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:21.952220+0000", 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028384406000000001 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47650.0:2 135.1f 135:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-10T08:45:56.370 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 467.291091048, 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394, 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47650", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1937376491", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:45:56.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032657+0000", 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2010000000000002e-06 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032664+0000", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1249999999999999e-06 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032673+0000", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4060000000000004e-06 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032679+0000", 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0767999999999999e-05 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032730+0000", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4550000000000001e-06 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032739+0000", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23991668399999999 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272656+0000", 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124196 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272780+0000", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152177 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272932+0000", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001504109 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.274436+0000", 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001049901 2024-02-10T08:45:56.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.275486+0000", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027108150000000001 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278197+0000", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3831e-05 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278251+0000", 2024-02-10T08:45:56.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0742999999999999e-05 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278292+0000", 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47734.0:2 147.1a 147:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 446.36698840999998, 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332, 2024-02-10T08:45:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47734", 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/29749952", 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956760+0000", 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956768+0000", 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956776+0000", 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0839999999999999e-06 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956781+0000", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7848000000000001e-05 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956819+0000", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.19e-06 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956825+0000", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090413761999999995 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047239+0000", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112608 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047352+0000", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014200000000000001 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047494+0000", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012910860000000001 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.048785+0000", 2024-02-10T08:45:56.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063636400000000003 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.055149+0000", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023764580000000001 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057525+0000", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8439000000000001e-05 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057583+0000", 2024-02-10T08:45:56.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1635999999999998e-05 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057635+0000", 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53393.0:1 195.1s0 195:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e330)", 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 332.29665258099999, 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441, 2024-02-10T08:45:56.384 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53393", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4089636740", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027094+0000", 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1878999999999999e-05 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027106+0000", 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9205999999999999e-05 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027125+0000", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0699999999999999e-05 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027136+0000", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017503530000000001 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028886+0000", 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7790000000000003e-06 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028891+0000", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.008200303 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037091+0000", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014759500000000001 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037239+0000", 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048123099999999999 2024-02-10T08:45:56.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037720+0000", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015191219999999999 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.039239+0000", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001215224 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040455+0000", 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0549e-05 2024-02-10T08:45:56.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040505+0000", 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1660 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94547991866752 gen 10] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 141.13882001499999, 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001, 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.390 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1660 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:45:56.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184927+0000", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.077e-05 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184938+0000", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3421000000000001e-05 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184951+0000", 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6079999999999998e-06 2024-02-10T08:45:56.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184959+0000", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1104000000000003e-05 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185020+0000", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0170000000000002e-06 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89132679699999995 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076356+0000", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013946 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076495+0000", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011721326000000001 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.088217+0000", 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.394 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1661 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 4] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 141.13877173200001, 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999, 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1661 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.395 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184981+0000", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1167e-05 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184993+0000", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9099999999999996e-06 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184998+0000", 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2206999999999997e-05 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185050+0000", 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1770000000000004e-06 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185059+0000", 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88333866299999997 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068398+0000", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172676 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068570+0000", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044926389999999997 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.073063+0000", 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999 2024-02-10T08:45:56.399 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:61 60.13 60:ca6ce4f9:test-rados-api-smithi107-112490-78::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.17503447999999, 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999, 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148715+0000", 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1810000000000003e-06 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148719+0000", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9329999999999998e-06 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148725+0000", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1559999999999997e-06 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148729+0000", 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4011000000000006e-05 2024-02-10T08:45:56.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148813+0000", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0549999999999996e-06 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148819+0000", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080372289999999999 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156857+0000", 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8814999999999997e-05 2024-02-10T08:45:56.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156935+0000", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017242200000000001 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157108+0000", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016798900000000001 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158788+0000", 2024-02-10T08:45:56.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072491250000000004 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166037+0000", 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013801147 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179838+0000", 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0494000000000001e-05 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179889+0000", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8433999999999998e-05 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179927+0000", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:62 60.1f 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:45:56.406 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.175010065, 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006, 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:45:56.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4249999999999998e-06 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148743+0000", 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2950000000000001e-06 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148748+0000", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0949999999999998e-06 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148752+0000", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015047300000000001 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148903+0000", 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.639e-06 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015726237000000001 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164634+0000", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1116e-05 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164725+0000", 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016155299999999999 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164886+0000", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016114943 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.181001+0000", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099756480000000002 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190977+0000", 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033742560999999997 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224720+0000", 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1842999999999998e-05 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224771+0000", 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8983999999999998e-05 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224810+0000", 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:65 60.7 60:e0e5fb82:test-rados-api-smithi107-112490-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.174984487, 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999, 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148765+0000", 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.311e-06 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148773+0000", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5339999999999999e-06 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148778+0000", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0620000000000002e-06 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148782+0000", 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014662 2024-02-10T08:45:56.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148929+0000", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8380000000000001e-06 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148934+0000", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015063796000000001 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.163997+0000", 2024-02-10T08:45:56.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164103+0000", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017560199999999999 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164279+0000", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016715187999999999 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180994+0000", 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097823279999999999 2024-02-10T08:45:56.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190776+0000", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8204999999999997e-05 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190825+0000", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5817999999999997e-05 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190911+0000", 2024-02-10T08:45:56.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:66 60.15 60:ab1b3caf:test-rados-api-smithi107-112490-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.17496394, 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002, 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2550000000000002e-06 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148789+0000", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1379999999999998e-06 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148794+0000", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8199999999999998e-06 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148798+0000", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109837 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5809999999999996e-06 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148913+0000", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013380609999999999 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162294+0000", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012214800000000001 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162416+0000", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020241599999999999 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162619+0000", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004775708 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.167394+0000", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057395455999999997 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224790+0000", 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4917999999999999e-05 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224835+0000", 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7835000000000002e-05 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224923+0000", 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:69 60.13 60:ce87a4eb:test-rados-api-smithi107-112490-78::5a02e6184a18760c5f8050d0e5b120c3d7a79519:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.174937988, 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999, 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6509999999999999e-06 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148815+0000", 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5979999999999996e-06 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148821+0000", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5329999999999996e-06 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148825+0000", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7842999999999997e-05 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148863+0000", 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3610000000000003e-06 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148868+0000", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084767599999999999 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157344+0000", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010973599999999999 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157454+0000", 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020398000000000001 2024-02-10T08:45:56.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157658+0000", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004485096 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162143+0000", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017667062000000001 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179810+0000", 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147684 2024-02-10T08:45:56.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179958+0000", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6878000000000002e-05 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179995+0000", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9448e-05 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180024+0000", 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.430 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(3.2 REQUEST e494)", 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 116.078977798, 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998, 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.517e-06 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244776+0000", 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5489999999999999e-05 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244791+0000", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9300000000000003e-06 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244799+0000", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025046212000000002 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269845+0000", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455999999999999e-05 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269866+0000", 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3003999999999998e-05 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269899+0000", 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1857 5.e 5:7759931f:::notify.3:head [watch ping cookie 94547991866752 gen 10] snapc 0=[] ondisk+write+known_if_redirected e570)", 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:45:21.629435+0000", 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.694315109000001, 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063327399999999997, 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:45:56.434 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1857 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629435+0000", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629435+0000", 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629434+0000", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4209999999999999e-06 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629438+0000", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6119999999999998e-06 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629443+0000", 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1110000000000001e-06 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629447+0000", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048949200000000001 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629937+0000", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.291e-05 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.629990+0000", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8553000000000005e-05 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:45:56.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:21.630068+0000", 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063327399999999997 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:45:56.438 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.473809+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.474800+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.690258+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:54.693149+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:55.225366+0000 mon.a (mon.0) 3050 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.225437+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:55.228379+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:55.228695+0000 mon.a (mon.0) 3052 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.356754+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.358051+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.515677+0000 mgr.y (mgr.14553) 522 : cluster [DBG] pgmap v916: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:56.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:55.590797+0000 mgr.y (mgr.14553) 523 : audit [DBG] from='client.54017 v1:172.21.15.107:0/3122379938' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- 172.21.15.107:0/1561024497 >> v1:172.21.15.107:6789/0 conn(0x7f1e100f9570 legacy=0x7f1e10102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- 172.21.15.107:0/1561024497 shutdown_connections 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- 172.21.15.107:0/1561024497 >> 172.21.15.107:0/1561024497 conn(0x7f1e100f2510 msgr2=0x7f1e100f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- 172.21.15.107:0/1561024497 shutdown_connections 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- 172.21.15.107:0/1561024497 wait complete. 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 librados: starting msgr at 2024-02-10T08:45:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 librados: starting objecter 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 Processor -- start 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- start start 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 librados: setting wanted keys 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 librados: calling monclient init 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e10126260 con 0x7f1e100f89c0 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e10127440 con 0x7f1e100f9570 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.404+0000 7f1e170bf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1e10128640 con 0x7f1e10104350 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e1565c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1e10104350 0x7f1e10102a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54112/0 (socket says 172.21.15.107:54112) 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e1565c700 1 -- 172.21.15.107:0/3122379938 learned_addr learned my addr 172.21.15.107:0/3122379938 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter ms_handle_connect 0x7f1e10104350 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter ms_handle_connect 0x7f1e100f89c0 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter ms_handle_connect 0x7f1e100f9570 2024-02-10T08:45:56.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944529199 0 0) 0x7f1e10128640 con 0x7f1e10104350 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e000033f0 con 0x7f1e10104350 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344369534 0 0) 0x7f1e10126260 con 0x7f1e100f89c0 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e10128640 con 0x7f1e100f89c0 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736477006 0 0) 0x7f1e10127440 con 0x7f1e100f9570 2024-02-10T08:45:56.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e10126260 con 0x7f1e100f9570 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042419402 0 0) 0x7f1e000033f0 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e10127440 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e08001d50 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596340566 0 0) 0x7f1e10128640 con 0x7f1e100f89c0 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e000033f0 con 0x7f1e100f89c0 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e040018c0 con 0x7f1e100f89c0 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685938485 0 0) 0x7f1e10127440 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 >> v1:172.21.15.195:6789/0 conn(0x7f1e100f9570 legacy=0x7f1e1010fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 >> v1:172.21.15.107:6789/0 conn(0x7f1e100f89c0 legacy=0x7f1e1010f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e10129840 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.405+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e10126450 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.406+0000 7f1e170bf700 10 client.54017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.406+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101268c0 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.406+0000 7f1e170bf700 1 librados: init done 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.406+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1e080026e0 con 0x7f1e10104350 2024-02-10T08:45:56.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.406+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1e08005e60 con 0x7f1e10104350 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1005f7a0 con 0x7f1e10104350 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1e080070f0 con 0x7f1e10104350 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(609..609 src has 420..609) v4 ==== 8625+0+0 (unknown 404438320 0 0) 0x7f1e080792f0 con 0x7f1e10104350 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e0dffb700 10 client.54017.objecter ms_dispatch 0x7f1e10001550 osd_map(609..609 src has 420..609) v4 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e0dffb700 3 client.54017.objecter handle_osd_map got epochs [609,609] > 0 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.407+0000 7f1e0dffb700 3 client.54017.objecter handle_osd_map decoding full epoch 609 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.408+0000 7f1e0dffb700 20 client.54017.objecter dump_active .. 0 homeless 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.408+0000 7f1e0dffb700 10 client.54017.objecter ms_handle_connect 0x7f1e00063be0 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.412+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1e0804d2d0 con 0x7f1e10104350 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.589+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e10109f60 con 0x7f1e00063be0 2024-02-10T08:45:56.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:55.603+0000 7f1e0dffb700 1 -- 172.21.15.107:0/3122379938 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+496887 (unknown 2965378022 0 1242845369) 0x7f1e10109f60 con 0x7f1e00063be0 2024-02-10T08:45:56.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 10 librados: watch_flush enter 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 10 librados: watch_flush exit 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 20 client.54017.objecter shutdown clearing up homeless session... 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 10 client.54017.objecter successfully canceled tick 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1e00063be0 legacy=0x7f1e000660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 >> v1:172.21.15.107:6790/0 conn(0x7f1e10104350 legacy=0x7f1e10102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 shutdown_connections 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 >> 172.21.15.107:0/3122379938 conn(0x7f1e100f2510 msgr2=0x7f1e101077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 shutdown_connections 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 -- 172.21.15.107:0/3122379938 wait complete. 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:56.455+0000 7f1e170bf700 1 librados: shutdown 2024-02-10T08:45:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:45:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.473809+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:45:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.474800+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:45:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.690258+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:54.693149+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:55.225366+0000 mon.a (mon.0) 3050 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.225437+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:55.228379+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:55.228695+0000 mon.a (mon.0) 3052 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.356754+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.358051+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.515677+0000 mgr.y (mgr.14553) 522 : cluster [DBG] pgmap v916: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:55.590797+0000 mgr.y (mgr.14553) 523 : audit [DBG] from='client.54017 v1:172.21.15.107:0/3122379938' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.473809+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.474800+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.690258+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:54.693149+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:55.225366+0000 mon.a (mon.0) 3050 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.225437+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:55.228379+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:55.228695+0000 mon.a (mon.0) 3052 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.356754+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.358051+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.515677+0000 mgr.y (mgr.14553) 522 : cluster [DBG] pgmap v916: 194 pgs: 32 unknown, 1 active+clean+remapped, 21 active+undersized+degraded, 44 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 87/582 objects degraded (14.948%) 2024-02-10T08:45:56.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:55.590797+0000 mgr.y (mgr.14553) 523 : audit [DBG] from='client.54017 v1:172.21.15.107:0/3122379938' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:45:57.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:45:57.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:45:57.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.476975+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:45:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.477888+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:45:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.829794+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:45:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:55.833259+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:45:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:56.231973+0000 mon.a (mon.0) 3053 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:45:57.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:56.232047+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-10T08:45:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:56.235099+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.289 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:56.235553+0000 mon.a (mon.0) 3055 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.299 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:45:57.313 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:45:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":916,"stamp":"2024-02-10T08:45:55.515218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5805,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":1888,"num_read_kb":1640,"num_write":722,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":6775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":806,"ondisk_log_size":806,"up":423,"acting":424,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2161596,"kb_used_data":23684,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":654054468,"statfs":{"total":671965249536,"available":669751775232,"internally_reserved":0,"allocated":24252416,"data_stored":11883048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001781"},"pg_stats":[{"pgid":"236.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":13,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.230995+0000","last_change":"2024-02-10T08:45:46.132662+0000","last_active":"2024-02-10T08:45:49.230995+0000","last_peered":"2024-02-10T08:45:49.230995+0000","last_clean":"2024-02-10T08:45:49.230995+0000","last_became_active":"2024-02-10T08:45:46.132344+0000","last_became_peered":"2024-02-10T08:45:46.132344+0000","last_unstale":"2024-02-10T08:45:49.230995+0000","last_undegraded":"2024-02-10T08:45:49.230995+0000","last_fullsized":"2024-02-10T08:45:49.230995+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"236.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"601'1","reported_seq":15,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.198389+0000","last_change":"2024-02-10T08:45:46.132419+0000","last_active":"2024-02-10T08:45:49.198389+0000","last_peered":"2024-02-10T08:45:49.198389+0000","last_clean":"2024-02-10T08:45:49.198389+0000","last_became_active":"2024-02-10T08:45:46.132095+0000","last_became_peered":"2024-02-10T08:45:46.132095+0000","last_unstale":"2024-02-10T08:45:49.198389+0000","last_undegraded":"2024-02-10T08:45:49.198389+0000","last_fullsized":"2024-02-10T08:45:49.198389+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"0'0","reported_seq":11,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.149225+0000","last_change":"2024-02-10T08:45:46.135704+0000","last_active":"2024-02-10T08:45:50.149225+0000","last_peered":"2024-02-10T08:45:50.149225+0000","last_clean":"2024-02-10T08:45:45.107430+0000","last_became_active":"2024-02-10T08:45:46.135704+0000","last_became_peered":"2024-02-10T08:45:46.135704+0000","last_unstale":"2024-02-10T08:45:50.149225+0000","last_undegraded":"2024-02-10T08:45:50.149225+0000","last_fullsized":"2024-02-10T08:45:46.122277+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,5],"acting":[2,2147483647,5],"avail_no_missing":["2(0)","5(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"236.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"0'0","reported_seq":13,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123375+0000","last_change":"2024-02-10T08:45:46.128155+0000","last_active":"2024-02-10T08:45:49.123375+0000","last_peered":"2024-02-10T08:45:49.123375+0000","last_clean":"2024-02-10T08:45:49.123375+0000","last_became_active":"2024-02-10T08:45:46.127941+0000","last_became_peered":"2024-02-10T08:45:46.127941+0000","last_unstale":"2024-02-10T08:45:49.123375+0000","last_undegraded":"2024-02-10T08:45:49.123375+0000","last_fullsized":"2024-02-10T08:45:49.123375+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"236.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":13,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123517+0000","last_change":"2024-02-10T08:45:46.129799+0000","last_active":"2024-02-10T08:45:49.123517+0000","last_peered":"2024-02-10T08:45:49.123517+0000","last_clean":"2024-02-10T08:45:49.123517+0000","last_became_active":"2024-02-10T08:45:46.129572+0000","last_became_peered":"2024-02-10T08:45:46.129572+0000","last_unstale":"2024-02-10T08:45:49.123517+0000","last_undegraded":"2024-02-10T08:45:49.123517+0000","last_fullsized":"2024-02-10T08:45:49.123517+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"236.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":13,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.175894+0000","last_change":"2024-02-10T08:45:46.129466+0000","last_active":"2024-02-10T08:45:49.175894+0000","last_peered":"2024-02-10T08:45:49.175894+0000","last_clean":"2024-02-10T08:45:49.175894+0000","last_became_active":"2024-02-10T08:45:46.129152+0000","last_became_peered":"2024-02-10T08:45:46.129152+0000","last_unstale":"2024-02-10T08:45:49.175894+0000","last_undegraded":"2024-02-10T08:45:49.175894+0000","last_fullsized":"2024-02-10T08:45:49.175894+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":10,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.127460+0000","last_change":"2024-02-10T08:45:46.125773+0000","last_active":"2024-02-10T08:45:49.127460+0000","last_peered":"2024-02-10T08:45:49.127460+0000","last_clean":"2024-02-10T08:45:45.107430+0000","last_became_active":"2024-02-10T08:45:46.125773+0000","last_became_peered":"2024-02-10T08:45:46.125773+0000","last_unstale":"2024-02-10T08:45:49.127460+0000","last_undegraded":"2024-02-10T08:45:49.127460+0000","last_fullsized":"2024-02-10T08:45:46.117804+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2147483647],"acting":[3,7,2147483647],"avail_no_missing":["3(0)","7(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"236.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:45:53.202973+0000","last_change":"2024-02-10T08:45:53.202973+0000","last_active":"2024-02-10T08:45:53.202973+0000","last_peered":"2024-02-10T08:45:53.202973+0000","last_clean":"2024-02-10T08:45:53.202973+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:45:53.202973+0000","last_undegraded":"2024-02-10T08:45:53.202973+0000","last_fullsized":"2024-02-10T08:45:53.202973+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:53.202973+0000","last_clean_scrub_stamp":"2024-02-10T08:45:53.202973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"0'0","reported_seq":10,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.175510+0000","last_change":"2024-02-10T08:45:46.122705+0000","last_active":"2024-02-10T08:45:49.175510+0000","last_peered":"2024-02-10T08:45:49.175510+0000","last_clean":"2024-02-10T08:45:45.107430+0000","last_became_active":"2024-02-10T08:45:46.122705+0000","last_became_peered":"2024-02-10T08:45:46.122705+0000","last_unstale":"2024-02-10T08:45:49.175510+0000","last_undegraded":"2024-02-10T08:45:49.175510+0000","last_fullsized":"2024-02-10T08:45:46.117649+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,3],"acting":[2147483647,6,3],"avail_no_missing":["6(1)","3(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"481'17","reported_seq":236,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.227708+0000","last_change":"2024-02-10T08:43:36.074524+0000","last_active":"2024-02-10T08:45:49.227708+0000","last_peered":"2024-02-10T08:45:49.227708+0000","last_clean":"2024-02-10T08:43:31.610321+0000","last_became_active":"2024-02-10T08:43:36.074524+0000","last_became_peered":"2024-02-10T08:43:36.074524+0000","last_unstale":"2024-02-10T08:45:49.227708+0000","last_undegraded":"2024-02-10T08:43:36.061132+0000","last_fullsized":"2024-02-10T08:43:36.061013+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'13","last_scrub_stamp":"2024-02-10T08:43:10.390465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:10.390465+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":674,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123213+0000","last_change":"2024-02-10T08:45:22.037140+0000","last_active":"2024-02-10T08:45:49.123213+0000","last_peered":"2024-02-10T08:45:49.123213+0000","last_clean":"2024-02-10T08:45:49.123213+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:45:49.123213+0000","last_undegraded":"2024-02-10T08:45:49.123213+0000","last_fullsized":"2024-02-10T08:45:49.123213+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:45:22.037068+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.037068+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":602,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.153154+0000","last_change":"2024-02-10T08:45:22.422467+0000","last_active":"2024-02-10T08:45:50.153154+0000","last_peered":"2024-02-10T08:45:50.153154+0000","last_clean":"2024-02-10T08:45:50.153154+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:45:50.153154+0000","last_undegraded":"2024-02-10T08:45:50.153154+0000","last_fullsized":"2024-02-10T08:45:50.153154+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.422382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.422382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":579,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123822+0000","last_change":"2024-02-10T08:42:41.108507+0000","last_active":"2024-02-10T08:45:49.123822+0000","last_peered":"2024-02-10T08:45:49.123822+0000","last_clean":"2024-02-10T08:45:49.123822+0000","last_became_active":"2024-02-10T08:37:10.168460+0000","last_became_peered":"2024-02-10T08:37:10.168460+0000","last_unstale":"2024-02-10T08:45:49.123822+0000","last_undegraded":"2024-02-10T08:45:49.123822+0000","last_fullsized":"2024-02-10T08:45:49.123822+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:41.108418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:41.108418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":525,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.151922+0000","last_change":"2024-02-10T08:44:28.457808+0000","last_active":"2024-02-10T08:45:50.151922+0000","last_peered":"2024-02-10T08:45:50.151922+0000","last_clean":"2024-02-10T08:45:50.151922+0000","last_became_active":"2024-02-10T08:36:37.046476+0000","last_became_peered":"2024-02-10T08:36:37.046476+0000","last_unstale":"2024-02-10T08:45:50.151922+0000","last_undegraded":"2024-02-10T08:45:50.151922+0000","last_fullsized":"2024-02-10T08:45:50.151922+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:28.457738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.457738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":564,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.148814+0000","last_change":"2024-02-10T08:43:36.079191+0000","last_active":"2024-02-10T08:45:50.148814+0000","last_peered":"2024-02-10T08:45:50.148814+0000","last_clean":"2024-02-10T08:43:31.066295+0000","last_became_active":"2024-02-10T08:43:36.079191+0000","last_became_peered":"2024-02-10T08:43:36.079191+0000","last_unstale":"2024-02-10T08:45:50.148814+0000","last_undegraded":"2024-02-10T08:43:36.059738+0000","last_fullsized":"2024-02-10T08:43:36.059623+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:16.350722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.350722+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":589,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.148504+0000","last_change":"2024-02-10T08:43:36.085621+0000","last_active":"2024-02-10T08:45:50.148504+0000","last_peered":"2024-02-10T08:45:50.148504+0000","last_clean":"2024-02-10T08:43:31.059000+0000","last_became_active":"2024-02-10T08:43:36.085621+0000","last_became_peered":"2024-02-10T08:43:36.085621+0000","last_unstale":"2024-02-10T08:45:50.148504+0000","last_undegraded":"2024-02-10T08:45:50.148504+0000","last_fullsized":"2024-02-10T08:43:36.063451+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.696704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.696704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":597,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.150770+0000","last_change":"2024-02-10T08:43:36.079793+0000","last_active":"2024-02-10T08:45:50.150770+0000","last_peered":"2024-02-10T08:45:50.150770+0000","last_clean":"2024-02-10T08:43:31.065630+0000","last_became_active":"2024-02-10T08:43:36.079793+0000","last_became_peered":"2024-02-10T08:43:36.079793+0000","last_unstale":"2024-02-10T08:45:50.150770+0000","last_undegraded":"2024-02-10T08:45:50.150770+0000","last_fullsized":"2024-02-10T08:43:36.061952+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.195597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.195597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.227607+0000","last_change":"2024-02-10T08:44:39.598037+0000","last_active":"2024-02-10T08:45:49.227607+0000","last_peered":"2024-02-10T08:45:49.227607+0000","last_clean":"2024-02-10T08:45:49.227607+0000","last_became_active":"2024-02-10T08:43:15.128618+0000","last_became_peered":"2024-02-10T08:43:15.128618+0000","last_unstale":"2024-02-10T08:45:49.227607+0000","last_undegraded":"2024-02-10T08:45:49.227607+0000","last_fullsized":"2024-02-10T08:45:49.227607+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.597966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.597966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":537,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126012+0000","last_change":"2024-02-10T08:44:39.356595+0000","last_active":"2024-02-10T08:45:49.126012+0000","last_peered":"2024-02-10T08:45:49.126012+0000","last_clean":"2024-02-10T08:45:49.126012+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:45:49.126012+0000","last_undegraded":"2024-02-10T08:45:49.126012+0000","last_fullsized":"2024-02-10T08:45:49.126012+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:39.356502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.356502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"473'35","reported_seq":693,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.176177+0000","last_change":"2024-02-10T08:44:37.425734+0000","last_active":"2024-02-10T08:45:49.176177+0000","last_peered":"2024-02-10T08:45:49.176177+0000","last_clean":"2024-02-10T08:45:49.176177+0000","last_became_active":"2024-02-10T08:43:15.136753+0000","last_became_peered":"2024-02-10T08:43:15.136753+0000","last_unstale":"2024-02-10T08:45:49.176177+0000","last_undegraded":"2024-02-10T08:45:49.176177+0000","last_fullsized":"2024-02-10T08:45:49.176177+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'35","last_scrub_stamp":"2024-02-10T08:44:37.425670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.425670+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":591,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.152321+0000","last_change":"2024-02-10T08:45:25.452785+0000","last_active":"2024-02-10T08:45:50.152321+0000","last_peered":"2024-02-10T08:45:50.152321+0000","last_clean":"2024-02-10T08:45:50.152321+0000","last_became_active":"2024-02-10T08:37:10.205165+0000","last_became_peered":"2024-02-10T08:37:10.205165+0000","last_unstale":"2024-02-10T08:45:50.152321+0000","last_undegraded":"2024-02-10T08:45:50.152321+0000","last_fullsized":"2024-02-10T08:45:50.152321+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.452719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":625,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148659+0000","last_change":"2024-02-10T08:45:24.427857+0000","last_active":"2024-02-10T08:45:50.148659+0000","last_peered":"2024-02-10T08:45:50.148659+0000","last_clean":"2024-02-10T08:45:50.148659+0000","last_became_active":"2024-02-10T08:43:15.119008+0000","last_became_peered":"2024-02-10T08:43:15.119008+0000","last_unstale":"2024-02-10T08:45:50.148659+0000","last_undegraded":"2024-02-10T08:45:50.148659+0000","last_fullsized":"2024-02-10T08:45:50.148659+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:24.427791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:24.427791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.230801+0000","last_change":"2024-02-10T08:44:40.551722+0000","last_active":"2024-02-10T08:45:49.230801+0000","last_peered":"2024-02-10T08:45:49.230801+0000","last_clean":"2024-02-10T08:45:49.230801+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:45:49.230801+0000","last_undegraded":"2024-02-10T08:45:49.230801+0000","last_fullsized":"2024-02-10T08:45:49.230801+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:40.551662+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.551662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.230640+0000","last_change":"2024-02-10T08:43:36.072003+0000","last_active":"2024-02-10T08:45:49.230640+0000","last_peered":"2024-02-10T08:45:49.230640+0000","last_clean":"2024-02-10T08:43:31.061070+0000","last_became_active":"2024-02-10T08:43:36.072003+0000","last_became_peered":"2024-02-10T08:43:36.072003+0000","last_unstale":"2024-02-10T08:45:49.230640+0000","last_undegraded":"2024-02-10T08:45:49.230640+0000","last_fullsized":"2024-02-10T08:43:36.062752+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:20.535884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:20.535884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":622,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.150221+0000","last_change":"2024-02-10T08:43:36.076710+0000","last_active":"2024-02-10T08:45:50.150221+0000","last_peered":"2024-02-10T08:45:50.150221+0000","last_clean":"2024-02-10T08:43:25.756727+0000","last_became_active":"2024-02-10T08:43:36.076710+0000","last_became_peered":"2024-02-10T08:43:36.076710+0000","last_unstale":"2024-02-10T08:45:50.150221+0000","last_undegraded":"2024-02-10T08:43:36.060349+0000","last_fullsized":"2024-02-10T08:43:36.060248+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:43:29.844454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:29.844454+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":415,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126394+0000","last_change":"2024-02-10T08:44:33.297486+0000","last_active":"2024-02-10T08:45:49.126394+0000","last_peered":"2024-02-10T08:45:49.126394+0000","last_clean":"2024-02-10T08:45:49.126394+0000","last_became_active":"2024-02-10T08:38:22.012706+0000","last_became_peered":"2024-02-10T08:38:22.012706+0000","last_unstale":"2024-02-10T08:45:49.126394+0000","last_undegraded":"2024-02-10T08:45:49.126394+0000","last_fullsized":"2024-02-10T08:45:49.126394+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:44:33.297424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":607,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.149961+0000","last_change":"2024-02-10T08:43:36.076112+0000","last_active":"2024-02-10T08:45:50.149961+0000","last_peered":"2024-02-10T08:45:50.149961+0000","last_clean":"2024-02-10T08:43:31.060641+0000","last_became_active":"2024-02-10T08:43:36.076112+0000","last_became_peered":"2024-02-10T08:43:36.076112+0000","last_unstale":"2024-02-10T08:45:50.149961+0000","last_undegraded":"2024-02-10T08:45:50.149961+0000","last_fullsized":"2024-02-10T08:43:36.063764+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:29.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.596097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":577,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.151573+0000","last_change":"2024-02-10T08:43:36.075565+0000","last_active":"2024-02-10T08:45:50.151573+0000","last_peered":"2024-02-10T08:45:50.151573+0000","last_clean":"2024-02-10T08:43:31.066268+0000","last_became_active":"2024-02-10T08:43:36.075565+0000","last_became_peered":"2024-02-10T08:43:36.075565+0000","last_unstale":"2024-02-10T08:45:50.151573+0000","last_undegraded":"2024-02-10T08:45:50.151573+0000","last_fullsized":"2024-02-10T08:43:36.061936+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.250001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":131,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.128021+0000","last_change":"2024-02-10T08:43:36.089962+0000","last_active":"2024-02-10T08:45:49.128021+0000","last_peered":"2024-02-10T08:45:49.128021+0000","last_clean":"2024-02-10T08:36:35.870864+0000","last_became_active":"2024-02-10T08:43:36.089962+0000","last_became_peered":"2024-02-10T08:43:36.089962+0000","last_unstale":"2024-02-10T08:45:49.128021+0000","last_undegraded":"2024-02-10T08:45:49.128021+0000","last_fullsized":"2024-02-10T08:43:36.068401+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:25.595376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:42:25.595376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":602,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.124449+0000","last_change":"2024-02-10T08:43:36.073845+0000","last_active":"2024-02-10T08:45:49.124449+0000","last_peered":"2024-02-10T08:45:49.124449+0000","last_clean":"2024-02-10T08:43:25.718471+0000","last_became_active":"2024-02-10T08:43:36.073845+0000","last_became_peered":"2024-02-10T08:43:36.073845+0000","last_unstale":"2024-02-10T08:45:49.124449+0000","last_undegraded":"2024-02-10T08:43:36.068245+0000","last_fullsized":"2024-02-10T08:43:36.068123+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:41:18.755515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:18.755515+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":606,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148734+0000","last_change":"2024-02-10T08:44:41.730549+0000","last_active":"2024-02-10T08:45:50.148734+0000","last_peered":"2024-02-10T08:45:50.148734+0000","last_clean":"2024-02-10T08:45:50.148734+0000","last_became_active":"2024-02-10T08:43:15.130483+0000","last_became_peered":"2024-02-10T08:43:15.130483+0000","last_unstale":"2024-02-10T08:45:50.148734+0000","last_undegraded":"2024-02-10T08:45:50.148734+0000","last_fullsized":"2024-02-10T08:45:50.148734+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:41.730476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:41.730476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":811,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.149577+0000","last_change":"2024-02-10T08:45:29.297349+0000","last_active":"2024-02-10T08:45:50.149577+0000","last_peered":"2024-02-10T08:45:50.149577+0000","last_clean":"2024-02-10T08:45:50.149577+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:45:50.149577+0000","last_undegraded":"2024-02-10T08:45:50.149577+0000","last_fullsized":"2024-02-10T08:45:50.149577+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:45:29.297292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":133,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.149570+0000","last_change":"2024-02-10T08:43:36.084792+0000","last_active":"2024-02-10T08:45:50.149570+0000","last_peered":"2024-02-10T08:45:50.149570+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.084792+0000","last_became_peered":"2024-02-10T08:43:36.084792+0000","last_unstale":"2024-02-10T08:45:50.149570+0000","last_undegraded":"2024-02-10T08:45:50.149570+0000","last_fullsized":"2024-02-10T08:43:36.060560+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:43.683364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:43.683364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":513,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.123375+0000","last_change":"2024-02-10T08:43:36.082315+0000","last_active":"2024-02-10T08:45:49.123375+0000","last_peered":"2024-02-10T08:45:49.123375+0000","last_clean":"2024-02-10T08:43:31.057840+0000","last_became_active":"2024-02-10T08:43:36.082315+0000","last_became_peered":"2024-02-10T08:43:36.082315+0000","last_unstale":"2024-02-10T08:45:49.123375+0000","last_undegraded":"2024-02-10T08:45:49.123375+0000","last_fullsized":"2024-02-10T08:43:36.070820+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.219434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.219434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":625,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.147430+0000","last_change":"2024-02-10T08:43:36.088120+0000","last_active":"2024-02-10T08:45:50.147430+0000","last_peered":"2024-02-10T08:45:50.147430+0000","last_clean":"2024-02-10T08:43:31.056937+0000","last_became_active":"2024-02-10T08:43:36.088120+0000","last_became_peered":"2024-02-10T08:43:36.088120+0000","last_unstale":"2024-02-10T08:45:50.147430+0000","last_undegraded":"2024-02-10T08:43:36.061412+0000","last_fullsized":"2024-02-10T08:43:36.061296+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-10T08:42:32.661266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.661266+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":622,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.147145+0000","last_change":"2024-02-10T08:45:18.559052+0000","last_active":"2024-02-10T08:45:50.147145+0000","last_peered":"2024-02-10T08:45:50.147145+0000","last_clean":"2024-02-10T08:45:50.147145+0000","last_became_active":"2024-02-10T08:43:15.130693+0000","last_became_peered":"2024-02-10T08:43:15.130693+0000","last_unstale":"2024-02-10T08:45:50.147145+0000","last_undegraded":"2024-02-10T08:45:50.147145+0000","last_fullsized":"2024-02-10T08:45:50.147145+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:45:18.558991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:18.558991+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"481'17","reported_seq":177,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.124454+0000","last_change":"2024-02-10T08:43:36.076136+0000","last_active":"2024-02-10T08:45:49.124454+0000","last_peered":"2024-02-10T08:45:49.124454+0000","last_clean":"2024-02-10T08:33:25.224233+0000","last_became_active":"2024-02-10T08:43:36.076136+0000","last_became_peered":"2024-02-10T08:43:36.076136+0000","last_unstale":"2024-02-10T08:45:49.124454+0000","last_undegraded":"2024-02-10T08:43:36.056738+0000","last_fullsized":"2024-02-10T08:43:36.056588+0000","mapping_epoch":480,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'13","last_scrub_stamp":"2024-02-10T08:42:44.647464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:44.647464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":579,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.125129+0000","last_change":"2024-02-10T08:45:23.003219+0000","last_active":"2024-02-10T08:45:49.125129+0000","last_peered":"2024-02-10T08:45:49.125129+0000","last_clean":"2024-02-10T08:45:49.125129+0000","last_became_active":"2024-02-10T08:37:10.148878+0000","last_became_peered":"2024-02-10T08:37:10.148878+0000","last_unstale":"2024-02-10T08:45:49.125129+0000","last_undegraded":"2024-02-10T08:45:49.125129+0000","last_fullsized":"2024-02-10T08:45:49.125129+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:45:23.003138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":537,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123690+0000","last_change":"2024-02-10T08:44:35.383065+0000","last_active":"2024-02-10T08:45:49.123690+0000","last_peered":"2024-02-10T08:45:49.123690+0000","last_clean":"2024-02-10T08:45:49.123690+0000","last_became_active":"2024-02-10T08:43:00.552233+0000","last_became_peered":"2024-02-10T08:43:00.552233+0000","last_unstale":"2024-02-10T08:45:49.123690+0000","last_undegraded":"2024-02-10T08:45:49.123690+0000","last_fullsized":"2024-02-10T08:45:49.123690+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.382972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.382972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":597,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.192606+0000","last_change":"2024-02-10T08:43:36.068792+0000","last_active":"2024-02-10T08:45:49.192606+0000","last_peered":"2024-02-10T08:45:49.192606+0000","last_clean":"2024-02-10T08:43:25.758929+0000","last_became_active":"2024-02-10T08:43:36.068792+0000","last_became_peered":"2024-02-10T08:43:36.068792+0000","last_unstale":"2024-02-10T08:45:49.192606+0000","last_undegraded":"2024-02-10T08:43:36.064019+0000","last_fullsized":"2024-02-10T08:43:36.063905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'11","last_scrub_stamp":"2024-02-10T08:41:56.714119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:56.714119+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.230481+0000","last_change":"2024-02-10T08:43:36.068338+0000","last_active":"2024-02-10T08:45:49.230481+0000","last_peered":"2024-02-10T08:45:49.230481+0000","last_clean":"2024-02-10T08:43:31.059980+0000","last_became_active":"2024-02-10T08:43:36.068338+0000","last_became_peered":"2024-02-10T08:43:36.068338+0000","last_unstale":"2024-02-10T08:45:49.230481+0000","last_undegraded":"2024-02-10T08:45:49.230481+0000","last_fullsized":"2024-02-10T08:43:36.060130+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:12.736740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:12.736740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"481'11","reported_seq":705,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.125622+0000","last_change":"2024-02-10T08:43:36.068155+0000","last_active":"2024-02-10T08:45:49.125622+0000","last_peered":"2024-02-10T08:45:49.125622+0000","last_clean":"2024-02-10T08:43:31.610349+0000","last_became_active":"2024-02-10T08:43:36.068155+0000","last_became_peered":"2024-02-10T08:43:36.068155+0000","last_unstale":"2024-02-10T08:45:49.125622+0000","last_undegraded":"2024-02-10T08:43:36.058153+0000","last_fullsized":"2024-02-10T08:43:36.058069+0000","mapping_epoch":480,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'10","last_scrub_stamp":"2024-02-10T08:42:29.051452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:29.051452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":571,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124026+0000","last_change":"2024-02-10T08:44:26.397785+0000","last_active":"2024-02-10T08:45:49.124026+0000","last_peered":"2024-02-10T08:45:49.124026+0000","last_clean":"2024-02-10T08:45:49.124026+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:45:49.124026+0000","last_undegraded":"2024-02-10T08:45:49.124026+0000","last_fullsized":"2024-02-10T08:45:49.124026+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.397690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.397690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":519,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123200+0000","last_change":"2024-02-10T08:42:47.240729+0000","last_active":"2024-02-10T08:45:49.123200+0000","last_peered":"2024-02-10T08:45:49.123200+0000","last_clean":"2024-02-10T08:45:49.123200+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:45:49.123200+0000","last_undegraded":"2024-02-10T08:45:49.123200+0000","last_fullsized":"2024-02-10T08:45:49.123200+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:47.240624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:47.240624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"481'9","reported_seq":739,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.123715+0000","last_change":"2024-02-10T08:43:36.077627+0000","last_active":"2024-02-10T08:45:49.123715+0000","last_peered":"2024-02-10T08:45:49.123715+0000","last_clean":"2024-02-10T08:43:31.610208+0000","last_became_active":"2024-02-10T08:43:36.077627+0000","last_became_peered":"2024-02-10T08:43:36.077627+0000","last_unstale":"2024-02-10T08:45:49.123715+0000","last_undegraded":"2024-02-10T08:43:36.066022+0000","last_fullsized":"2024-02-10T08:43:36.065899+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:41:44.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.772939+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":613,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.123339+0000","last_change":"2024-02-10T08:43:36.064143+0000","last_active":"2024-02-10T08:45:49.123339+0000","last_peered":"2024-02-10T08:45:49.123339+0000","last_clean":"2024-02-10T08:43:25.763333+0000","last_became_active":"2024-02-10T08:43:36.064143+0000","last_became_peered":"2024-02-10T08:43:36.064143+0000","last_unstale":"2024-02-10T08:45:49.123339+0000","last_undegraded":"2024-02-10T08:43:36.057979+0000","last_fullsized":"2024-02-10T08:43:36.057862+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:43:34.883363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.883363+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":604,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.191987+0000","last_change":"2024-02-10T08:44:29.439534+0000","last_active":"2024-02-10T08:45:49.191987+0000","last_peered":"2024-02-10T08:45:49.191987+0000","last_clean":"2024-02-10T08:45:49.191987+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:45:49.191987+0000","last_undegraded":"2024-02-10T08:45:49.191987+0000","last_fullsized":"2024-02-10T08:45:49.191987+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.439473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.439473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.229197+0000","last_change":"2024-02-10T08:44:31.574745+0000","last_active":"2024-02-10T08:45:49.229197+0000","last_peered":"2024-02-10T08:45:49.229197+0000","last_clean":"2024-02-10T08:45:49.229197+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:45:49.229197+0000","last_undegraded":"2024-02-10T08:45:49.229197+0000","last_fullsized":"2024-02-10T08:45:49.229197+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:31.574681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.574681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":497,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.148596+0000","last_change":"2024-02-10T08:43:36.073331+0000","last_active":"2024-02-10T08:45:50.148596+0000","last_peered":"2024-02-10T08:45:50.148596+0000","last_clean":"2024-02-10T08:43:31.059113+0000","last_became_active":"2024-02-10T08:43:36.073331+0000","last_became_peered":"2024-02-10T08:43:36.073331+0000","last_unstale":"2024-02-10T08:45:50.148596+0000","last_undegraded":"2024-02-10T08:45:50.148596+0000","last_fullsized":"2024-02-10T08:43:36.063737+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:57.599660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:57.599660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":574,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.127824+0000","last_change":"2024-02-10T08:43:36.082324+0000","last_active":"2024-02-10T08:45:49.127824+0000","last_peered":"2024-02-10T08:45:49.127824+0000","last_clean":"2024-02-10T08:43:11.068543+0000","last_became_active":"2024-02-10T08:43:36.082324+0000","last_became_peered":"2024-02-10T08:43:36.082324+0000","last_unstale":"2024-02-10T08:45:49.127824+0000","last_undegraded":"2024-02-10T08:45:49.127824+0000","last_fullsized":"2024-02-10T08:43:36.067905+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.619193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.619193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":739,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126052+0000","last_change":"2024-02-10T08:44:37.388097+0000","last_active":"2024-02-10T08:45:49.126052+0000","last_peered":"2024-02-10T08:45:49.126052+0000","last_clean":"2024-02-10T08:45:49.126052+0000","last_became_active":"2024-02-10T08:43:15.136661+0000","last_became_peered":"2024-02-10T08:43:15.136661+0000","last_unstale":"2024-02-10T08:45:49.126052+0000","last_undegraded":"2024-02-10T08:45:49.126052+0000","last_fullsized":"2024-02-10T08:45:49.126052+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:44:37.388018+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.388018+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":591,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.149215+0000","last_change":"2024-02-10T08:43:36.068713+0000","last_active":"2024-02-10T08:45:50.149215+0000","last_peered":"2024-02-10T08:45:50.149215+0000","last_clean":"2024-02-10T08:43:31.057619+0000","last_became_active":"2024-02-10T08:43:36.068713+0000","last_became_peered":"2024-02-10T08:43:36.068713+0000","last_unstale":"2024-02-10T08:45:50.149215+0000","last_undegraded":"2024-02-10T08:43:36.060168+0000","last_fullsized":"2024-02-10T08:43:36.060055+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:41:46.729401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:46.729401+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":131,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.125908+0000","last_change":"2024-02-10T08:43:36.080856+0000","last_active":"2024-02-10T08:45:49.125908+0000","last_peered":"2024-02-10T08:45:49.125908+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.080856+0000","last_became_peered":"2024-02-10T08:43:36.080856+0000","last_unstale":"2024-02-10T08:45:49.125908+0000","last_undegraded":"2024-02-10T08:45:49.125908+0000","last_fullsized":"2024-02-10T08:43:36.066562+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:10.710732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:10.710732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":507,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123458+0000","last_change":"2024-02-10T08:43:00.547238+0000","last_active":"2024-02-10T08:45:49.123458+0000","last_peered":"2024-02-10T08:45:49.123458+0000","last_clean":"2024-02-10T08:45:49.123458+0000","last_became_active":"2024-02-10T08:43:00.546139+0000","last_became_peered":"2024-02-10T08:43:00.546139+0000","last_unstale":"2024-02-10T08:45:49.123458+0000","last_undegraded":"2024-02-10T08:45:49.123458+0000","last_fullsized":"2024-02-10T08:45:49.123458+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.024104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.024104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":587,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.176255+0000","last_change":"2024-02-10T08:45:29.566541+0000","last_active":"2024-02-10T08:45:49.176255+0000","last_peered":"2024-02-10T08:45:49.176255+0000","last_clean":"2024-02-10T08:45:49.176255+0000","last_became_active":"2024-02-10T08:37:10.231068+0000","last_became_peered":"2024-02-10T08:37:10.231068+0000","last_unstale":"2024-02-10T08:45:49.176255+0000","last_undegraded":"2024-02-10T08:45:49.176255+0000","last_fullsized":"2024-02-10T08:45:49.176255+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:29.566473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:29.566473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":624,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123837+0000","last_change":"2024-02-10T08:43:25.480698+0000","last_active":"2024-02-10T08:45:49.123837+0000","last_peered":"2024-02-10T08:45:49.123837+0000","last_clean":"2024-02-10T08:45:49.123837+0000","last_became_active":"2024-02-10T08:43:15.126404+0000","last_became_peered":"2024-02-10T08:43:15.126404+0000","last_unstale":"2024-02-10T08:45:49.123837+0000","last_undegraded":"2024-02-10T08:45:49.123837+0000","last_fullsized":"2024-02-10T08:45:49.123837+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'21","last_scrub_stamp":"2024-02-10T08:43:25.480620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:25.480620+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.229812+0000","last_change":"2024-02-10T08:43:36.073770+0000","last_active":"2024-02-10T08:45:49.229812+0000","last_peered":"2024-02-10T08:45:49.229812+0000","last_clean":"2024-02-10T08:43:31.060583+0000","last_became_active":"2024-02-10T08:43:36.073770+0000","last_became_peered":"2024-02-10T08:43:36.073770+0000","last_unstale":"2024-02-10T08:45:49.229812+0000","last_undegraded":"2024-02-10T08:45:49.229812+0000","last_fullsized":"2024-02-10T08:43:36.061914+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:27.297949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:27.297949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":506,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.151050+0000","last_change":"2024-02-10T08:44:24.395820+0000","last_active":"2024-02-10T08:45:50.151050+0000","last_peered":"2024-02-10T08:45:50.151050+0000","last_clean":"2024-02-10T08:45:50.151050+0000","last_became_active":"2024-02-10T08:43:15.106310+0000","last_became_peered":"2024-02-10T08:43:15.106310+0000","last_unstale":"2024-02-10T08:45:50.151050+0000","last_undegraded":"2024-02-10T08:45:50.151050+0000","last_fullsized":"2024-02-10T08:45:50.151050+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.395744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.395744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":534,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.151237+0000","last_change":"2024-02-10T08:43:59.552708+0000","last_active":"2024-02-10T08:45:50.151237+0000","last_peered":"2024-02-10T08:45:50.151237+0000","last_clean":"2024-02-10T08:45:50.151237+0000","last_became_active":"2024-02-10T08:37:10.233637+0000","last_became_peered":"2024-02-10T08:37:10.233637+0000","last_unstale":"2024-02-10T08:45:50.151237+0000","last_undegraded":"2024-02-10T08:45:50.151237+0000","last_fullsized":"2024-02-10T08:45:50.151237+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.552604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.552604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":605,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.125771+0000","last_change":"2024-02-10T08:43:36.069190+0000","last_active":"2024-02-10T08:45:49.125771+0000","last_peered":"2024-02-10T08:45:49.125771+0000","last_clean":"2024-02-10T08:43:31.062577+0000","last_became_active":"2024-02-10T08:43:36.069190+0000","last_became_peered":"2024-02-10T08:43:36.069190+0000","last_unstale":"2024-02-10T08:45:49.125771+0000","last_undegraded":"2024-02-10T08:45:49.125771+0000","last_fullsized":"2024-02-10T08:43:36.058346+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.561295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"473'24","reported_seq":609,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.149472+0000","last_change":"2024-02-10T08:44:40.723157+0000","last_active":"2024-02-10T08:45:50.149472+0000","last_peered":"2024-02-10T08:45:50.149472+0000","last_clean":"2024-02-10T08:45:50.149472+0000","last_became_active":"2024-02-10T08:43:15.129639+0000","last_became_peered":"2024-02-10T08:43:15.129639+0000","last_unstale":"2024-02-10T08:45:50.149472+0000","last_undegraded":"2024-02-10T08:45:50.149472+0000","last_fullsized":"2024-02-10T08:45:50.149472+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'24","last_scrub_stamp":"2024-02-10T08:44:40.723096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:40.723096+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.230394+0000","last_change":"2024-02-10T08:44:36.570760+0000","last_active":"2024-02-10T08:45:49.230394+0000","last_peered":"2024-02-10T08:45:49.230394+0000","last_clean":"2024-02-10T08:45:49.230394+0000","last_became_active":"2024-02-10T08:43:15.132762+0000","last_became_peered":"2024-02-10T08:43:15.132762+0000","last_unstale":"2024-02-10T08:45:49.230394+0000","last_undegraded":"2024-02-10T08:45:49.230394+0000","last_fullsized":"2024-02-10T08:45:49.230394+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.570697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.570697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":134,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.127801+0000","last_change":"2024-02-10T08:43:36.072974+0000","last_active":"2024-02-10T08:45:49.127801+0000","last_peered":"2024-02-10T08:45:49.127801+0000","last_clean":"2024-02-10T08:33:17.081938+0000","last_became_active":"2024-02-10T08:43:36.072974+0000","last_became_peered":"2024-02-10T08:43:36.072974+0000","last_unstale":"2024-02-10T08:45:49.127801+0000","last_undegraded":"2024-02-10T08:45:49.127801+0000","last_fullsized":"2024-02-10T08:43:36.065921+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.839896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.839896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":583,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.150200+0000","last_change":"2024-02-10T08:44:38.734244+0000","last_active":"2024-02-10T08:45:50.150200+0000","last_peered":"2024-02-10T08:45:50.150200+0000","last_clean":"2024-02-10T08:45:50.150200+0000","last_became_active":"2024-02-10T08:33:24.121718+0000","last_became_peered":"2024-02-10T08:33:24.121718+0000","last_unstale":"2024-02-10T08:45:50.150200+0000","last_undegraded":"2024-02-10T08:45:50.150200+0000","last_fullsized":"2024-02-10T08:45:50.150200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:38.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.734157+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":529,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.175978+0000","last_change":"2024-02-10T08:44:23.462760+0000","last_active":"2024-02-10T08:45:49.175978+0000","last_peered":"2024-02-10T08:45:49.175978+0000","last_clean":"2024-02-10T08:45:49.175978+0000","last_became_active":"2024-02-10T08:37:10.153565+0000","last_became_peered":"2024-02-10T08:37:10.153565+0000","last_unstale":"2024-02-10T08:45:49.175978+0000","last_undegraded":"2024-02-10T08:45:49.175978+0000","last_fullsized":"2024-02-10T08:45:49.175978+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:23.462674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:23.462674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":153,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.228559+0000","last_change":"2024-02-10T08:43:56.338478+0000","last_active":"2024-02-10T08:45:49.228559+0000","last_peered":"2024-02-10T08:45:49.228559+0000","last_clean":"2024-02-10T08:45:49.228559+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:45:49.228559+0000","last_undegraded":"2024-02-10T08:45:49.228559+0000","last_fullsized":"2024-02-10T08:45:49.228559+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:56.338378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:56.338378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":666,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.177284+0000","last_change":"2024-02-10T08:43:36.076013+0000","last_active":"2024-02-10T08:45:49.177284+0000","last_peered":"2024-02-10T08:45:49.177284+0000","last_clean":"2024-02-10T08:43:31.155643+0000","last_became_active":"2024-02-10T08:43:36.076013+0000","last_became_peered":"2024-02-10T08:43:36.076013+0000","last_unstale":"2024-02-10T08:45:49.177284+0000","last_undegraded":"2024-02-10T08:43:36.064979+0000","last_fullsized":"2024-02-10T08:43:36.064906+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'14","last_scrub_stamp":"2024-02-10T08:42:00.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:00.249567+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.230877+0000","last_change":"2024-02-10T08:44:24.600149+0000","last_active":"2024-02-10T08:45:49.230877+0000","last_peered":"2024-02-10T08:45:49.230877+0000","last_clean":"2024-02-10T08:45:49.230877+0000","last_became_active":"2024-02-10T08:43:15.135366+0000","last_became_peered":"2024-02-10T08:43:15.135366+0000","last_unstale":"2024-02-10T08:45:49.230877+0000","last_undegraded":"2024-02-10T08:45:49.230877+0000","last_fullsized":"2024-02-10T08:45:49.230877+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:24.600075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:44:24.600075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":594,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123773+0000","last_change":"2024-02-10T08:44:26.266252+0000","last_active":"2024-02-10T08:45:49.123773+0000","last_peered":"2024-02-10T08:45:49.123773+0000","last_clean":"2024-02-10T08:45:49.123773+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:45:49.123773+0000","last_undegraded":"2024-02-10T08:45:49.123773+0000","last_fullsized":"2024-02-10T08:45:49.123773+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:26.266165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:26.266165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":575,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123574+0000","last_change":"2024-02-10T08:44:00.274641+0000","last_active":"2024-02-10T08:45:49.123574+0000","last_peered":"2024-02-10T08:45:49.123574+0000","last_clean":"2024-02-10T08:45:49.123574+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:45:49.123574+0000","last_undegraded":"2024-02-10T08:45:49.123574+0000","last_fullsized":"2024-02-10T08:45:49.123574+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:44:00.274550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":571,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.125902+0000","last_change":"2024-02-10T08:43:36.077071+0000","last_active":"2024-02-10T08:45:49.125902+0000","last_peered":"2024-02-10T08:45:49.125902+0000","last_clean":"2024-02-10T08:43:31.062622+0000","last_became_active":"2024-02-10T08:43:36.077071+0000","last_became_peered":"2024-02-10T08:43:36.077071+0000","last_unstale":"2024-02-10T08:45:49.125902+0000","last_undegraded":"2024-02-10T08:45:49.125902+0000","last_fullsized":"2024-02-10T08:43:36.058617+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:48.065260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:48.065260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":530,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123241+0000","last_change":"2024-02-10T08:45:16.056076+0000","last_active":"2024-02-10T08:45:49.123241+0000","last_peered":"2024-02-10T08:45:49.123241+0000","last_clean":"2024-02-10T08:45:49.123241+0000","last_became_active":"2024-02-10T08:37:10.148294+0000","last_became_peered":"2024-02-10T08:37:10.148294+0000","last_unstale":"2024-02-10T08:45:49.123241+0000","last_undegraded":"2024-02-10T08:45:49.123241+0000","last_fullsized":"2024-02-10T08:45:49.123241+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:16.055988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:16.055988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":614,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.153324+0000","last_change":"2024-02-10T08:45:23.378056+0000","last_active":"2024-02-10T08:45:50.153324+0000","last_peered":"2024-02-10T08:45:50.153324+0000","last_clean":"2024-02-10T08:45:50.153324+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:45:50.153324+0000","last_undegraded":"2024-02-10T08:45:50.153324+0000","last_fullsized":"2024-02-10T08:45:50.153324+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:23.377976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:23.377976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":682,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.147092+0000","last_change":"2024-02-10T08:45:07.684836+0000","last_active":"2024-02-10T08:45:50.147092+0000","last_peered":"2024-02-10T08:45:50.147092+0000","last_clean":"2024-02-10T08:45:50.147092+0000","last_became_active":"2024-02-10T08:43:15.129755+0000","last_became_peered":"2024-02-10T08:43:15.129755+0000","last_unstale":"2024-02-10T08:45:50.147092+0000","last_undegraded":"2024-02-10T08:45:50.147092+0000","last_fullsized":"2024-02-10T08:45:50.147092+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:45:07.684765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.684765+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":593,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.150975+0000","last_change":"2024-02-10T08:44:37.469860+0000","last_active":"2024-02-10T08:45:50.150975+0000","last_peered":"2024-02-10T08:45:50.150975+0000","last_clean":"2024-02-10T08:45:50.150975+0000","last_became_active":"2024-02-10T08:37:10.234436+0000","last_became_peered":"2024-02-10T08:37:10.234436+0000","last_unstale":"2024-02-10T08:45:50.150975+0000","last_undegraded":"2024-02-10T08:45:50.150975+0000","last_fullsized":"2024-02-10T08:45:50.150975+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:37.469796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.469796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":1337,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.515727+0000","last_change":"2024-02-10T08:45:20.614091+0000","last_active":"2024-02-10T08:45:50.515727+0000","last_peered":"2024-02-10T08:45:50.515727+0000","last_clean":"2024-02-10T08:45:50.515727+0000","last_became_active":"2024-02-10T08:33:18.109968+0000","last_became_peered":"2024-02-10T08:33:18.109968+0000","last_unstale":"2024-02-10T08:45:50.515727+0000","last_undegraded":"2024-02-10T08:45:50.515727+0000","last_fullsized":"2024-02-10T08:45:50.515727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:45:20.614013+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:36:33.452367+0000","last_clean_scrub_stamp":"2024-02-10T08:45:20.614013+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":745,"num_read_kb":740,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.229732+0000","last_change":"2024-02-10T08:45:09.454368+0000","last_active":"2024-02-10T08:45:49.229732+0000","last_peered":"2024-02-10T08:45:49.229732+0000","last_clean":"2024-02-10T08:45:49.229732+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:45:49.229732+0000","last_undegraded":"2024-02-10T08:45:49.229732+0000","last_fullsized":"2024-02-10T08:45:49.229732+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:09.454293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.454293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":553,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148613+0000","last_change":"2024-02-10T08:44:36.777644+0000","last_active":"2024-02-10T08:45:50.148613+0000","last_peered":"2024-02-10T08:45:50.148613+0000","last_clean":"2024-02-10T08:45:50.148613+0000","last_became_active":"2024-02-10T08:43:15.129900+0000","last_became_peered":"2024-02-10T08:43:15.129900+0000","last_unstale":"2024-02-10T08:45:50.148613+0000","last_undegraded":"2024-02-10T08:45:50.148613+0000","last_fullsized":"2024-02-10T08:45:50.148613+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_clean_scrub_stamp":"2024-02-10T08:44:36.777533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":497,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.148177+0000","last_change":"2024-02-10T08:43:36.073644+0000","last_active":"2024-02-10T08:45:50.148177+0000","last_peered":"2024-02-10T08:45:50.148177+0000","last_clean":"2024-02-10T08:43:31.058736+0000","last_became_active":"2024-02-10T08:43:36.073644+0000","last_became_peered":"2024-02-10T08:43:36.073644+0000","last_unstale":"2024-02-10T08:45:50.148177+0000","last_undegraded":"2024-02-10T08:45:50.148177+0000","last_fullsized":"2024-02-10T08:43:36.062919+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:53.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.594651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"473'41","reported_seq":729,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124791+0000","last_change":"2024-02-10T08:42:43.109193+0000","last_active":"2024-02-10T08:45:49.124791+0000","last_peered":"2024-02-10T08:45:49.124791+0000","last_clean":"2024-02-10T08:45:49.124791+0000","last_became_active":"2024-02-10T08:34:41.698934+0000","last_became_peered":"2024-02-10T08:34:41.698934+0000","last_unstale":"2024-02-10T08:45:49.124791+0000","last_undegraded":"2024-02-10T08:45:49.124791+0000","last_fullsized":"2024-02-10T08:45:49.124791+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-10T08:42:43.109121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.109121+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":576,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123901+0000","last_change":"2024-02-10T08:42:37.145252+0000","last_active":"2024-02-10T08:45:49.123901+0000","last_peered":"2024-02-10T08:45:49.123901+0000","last_clean":"2024-02-10T08:45:49.123901+0000","last_became_active":"2024-02-10T08:33:18.106236+0000","last_became_peered":"2024-02-10T08:33:18.106236+0000","last_unstale":"2024-02-10T08:45:49.123901+0000","last_undegraded":"2024-02-10T08:45:49.123901+0000","last_fullsized":"2024-02-10T08:45:49.123901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.145180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.145180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":588,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.126658+0000","last_change":"2024-02-10T08:43:36.083021+0000","last_active":"2024-02-10T08:45:49.126658+0000","last_peered":"2024-02-10T08:45:49.126658+0000","last_clean":"2024-02-10T08:43:31.058994+0000","last_became_active":"2024-02-10T08:43:36.083021+0000","last_became_peered":"2024-02-10T08:43:36.083021+0000","last_unstale":"2024-02-10T08:45:49.126658+0000","last_undegraded":"2024-02-10T08:45:49.126658+0000","last_fullsized":"2024-02-10T08:43:36.072420+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:11.702672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:11.702672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":552,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.147537+0000","last_change":"2024-02-10T08:44:30.426450+0000","last_active":"2024-02-10T08:45:50.147537+0000","last_peered":"2024-02-10T08:45:50.147537+0000","last_clean":"2024-02-10T08:45:50.147537+0000","last_became_active":"2024-02-10T08:43:15.119340+0000","last_became_peered":"2024-02-10T08:43:15.119340+0000","last_unstale":"2024-02-10T08:45:50.147537+0000","last_undegraded":"2024-02-10T08:45:50.147537+0000","last_fullsized":"2024-02-10T08:45:50.147537+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:30.426345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.426345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"137'9","reported_seq":622,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.176922+0000","last_change":"2024-02-10T08:44:39.424567+0000","last_active":"2024-02-10T08:45:49.176922+0000","last_peered":"2024-02-10T08:45:49.176922+0000","last_clean":"2024-02-10T08:45:49.176922+0000","last_became_active":"2024-02-10T08:37:10.150101+0000","last_became_peered":"2024-02-10T08:37:10.150101+0000","last_unstale":"2024-02-10T08:45:49.176922+0000","last_undegraded":"2024-02-10T08:45:49.176922+0000","last_fullsized":"2024-02-10T08:45:49.176922+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:44:39.424472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:44:39.424472+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":624,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.148991+0000","last_change":"2024-02-10T08:43:36.071773+0000","last_active":"2024-02-10T08:45:50.148991+0000","last_peered":"2024-02-10T08:45:50.148991+0000","last_clean":"2024-02-10T08:43:31.056280+0000","last_became_active":"2024-02-10T08:43:36.071773+0000","last_became_peered":"2024-02-10T08:43:36.071773+0000","last_unstale":"2024-02-10T08:45:50.148991+0000","last_undegraded":"2024-02-10T08:43:36.059969+0000","last_fullsized":"2024-02-10T08:43:36.059851+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-02-10T08:42:40.687013+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.687013+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":615,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.149609+0000","last_change":"2024-02-10T08:45:10.601821+0000","last_active":"2024-02-10T08:45:50.149609+0000","last_peered":"2024-02-10T08:45:50.149609+0000","last_clean":"2024-02-10T08:45:50.149609+0000","last_became_active":"2024-02-10T08:43:15.129787+0000","last_became_peered":"2024-02-10T08:43:15.129787+0000","last_unstale":"2024-02-10T08:45:50.149609+0000","last_undegraded":"2024-02-10T08:45:50.149609+0000","last_fullsized":"2024-02-10T08:45:50.149609+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.601758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.601758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":489,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123585+0000","last_change":"2024-02-10T08:42:40.143847+0000","last_active":"2024-02-10T08:45:49.123585+0000","last_peered":"2024-02-10T08:45:49.123585+0000","last_clean":"2024-02-10T08:45:49.123585+0000","last_became_active":"2024-02-10T08:37:10.249091+0000","last_became_peered":"2024-02-10T08:37:10.249091+0000","last_unstale":"2024-02-10T08:45:49.123585+0000","last_undegraded":"2024-02-10T08:45:49.123585+0000","last_fullsized":"2024-02-10T08:45:49.123585+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.143787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.143787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":519,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126348+0000","last_change":"2024-02-10T08:42:34.006247+0000","last_active":"2024-02-10T08:45:49.126348+0000","last_peered":"2024-02-10T08:45:49.126348+0000","last_clean":"2024-02-10T08:45:49.126348+0000","last_became_active":"2024-02-10T08:36:37.155308+0000","last_became_peered":"2024-02-10T08:36:37.155308+0000","last_unstale":"2024-02-10T08:45:49.126348+0000","last_undegraded":"2024-02-10T08:45:49.126348+0000","last_fullsized":"2024-02-10T08:45:49.126348+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:34.006177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:34.006177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":605,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.147925+0000","last_change":"2024-02-10T08:43:58.505496+0000","last_active":"2024-02-10T08:45:50.147925+0000","last_peered":"2024-02-10T08:45:50.147925+0000","last_clean":"2024-02-10T08:45:50.147925+0000","last_became_active":"2024-02-10T08:37:10.212992+0000","last_became_peered":"2024-02-10T08:37:10.212992+0000","last_unstale":"2024-02-10T08:45:50.147925+0000","last_undegraded":"2024-02-10T08:45:50.147925+0000","last_fullsized":"2024-02-10T08:45:50.147925+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:58.505431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:43:58.505431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":611,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.147516+0000","last_change":"2024-02-10T08:43:36.079165+0000","last_active":"2024-02-10T08:45:50.147516+0000","last_peered":"2024-02-10T08:45:50.147516+0000","last_clean":"2024-02-10T08:43:25.579489+0000","last_became_active":"2024-02-10T08:43:36.079165+0000","last_became_peered":"2024-02-10T08:43:36.079165+0000","last_unstale":"2024-02-10T08:45:50.147516+0000","last_undegraded":"2024-02-10T08:43:36.061039+0000","last_fullsized":"2024-02-10T08:43:36.060910+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'17","last_scrub_stamp":"2024-02-10T08:41:44.633201+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:41:44.633201+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":607,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.231419+0000","last_change":"2024-02-10T08:43:36.072147+0000","last_active":"2024-02-10T08:45:49.231419+0000","last_peered":"2024-02-10T08:45:49.231419+0000","last_clean":"2024-02-10T08:43:11.810746+0000","last_became_active":"2024-02-10T08:43:36.072147+0000","last_became_peered":"2024-02-10T08:43:36.072147+0000","last_unstale":"2024-02-10T08:45:49.231419+0000","last_undegraded":"2024-02-10T08:43:36.061852+0000","last_fullsized":"2024-02-10T08:43:36.061736+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:43:11.810639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.810639+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":592,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126925+0000","last_change":"2024-02-10T08:45:19.465741+0000","last_active":"2024-02-10T08:45:49.126925+0000","last_peered":"2024-02-10T08:45:49.126925+0000","last_clean":"2024-02-10T08:45:49.126925+0000","last_became_active":"2024-02-10T08:43:15.125756+0000","last_became_peered":"2024-02-10T08:43:15.125756+0000","last_unstale":"2024-02-10T08:45:49.126925+0000","last_undegraded":"2024-02-10T08:45:49.126925+0000","last_fullsized":"2024-02-10T08:45:49.126925+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:19.465638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:19.465638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":543,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.149497+0000","last_change":"2024-02-10T08:43:54.509761+0000","last_active":"2024-02-10T08:45:50.149497+0000","last_peered":"2024-02-10T08:45:50.149497+0000","last_clean":"2024-02-10T08:45:50.149497+0000","last_became_active":"2024-02-10T08:43:15.119419+0000","last_became_peered":"2024-02-10T08:43:15.119419+0000","last_unstale":"2024-02-10T08:45:50.149497+0000","last_undegraded":"2024-02-10T08:45:50.149497+0000","last_fullsized":"2024-02-10T08:45:50.149497+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:54.509649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:54.509649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":607,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126786+0000","last_change":"2024-02-10T08:44:38.409588+0000","last_active":"2024-02-10T08:45:49.126786+0000","last_peered":"2024-02-10T08:45:49.126786+0000","last_clean":"2024-02-10T08:45:49.126786+0000","last_became_active":"2024-02-10T08:43:15.137834+0000","last_became_peered":"2024-02-10T08:43:15.137834+0000","last_unstale":"2024-02-10T08:45:49.126786+0000","last_undegraded":"2024-02-10T08:45:49.126786+0000","last_fullsized":"2024-02-10T08:45:49.126786+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:44:38.409486+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":592,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.147779+0000","last_change":"2024-02-10T08:43:36.075991+0000","last_active":"2024-02-10T08:45:50.147779+0000","last_peered":"2024-02-10T08:45:50.147779+0000","last_clean":"2024-02-10T08:43:31.065892+0000","last_became_active":"2024-02-10T08:43:36.075991+0000","last_became_peered":"2024-02-10T08:43:36.075991+0000","last_unstale":"2024-02-10T08:45:50.147779+0000","last_undegraded":"2024-02-10T08:45:50.147779+0000","last_fullsized":"2024-02-10T08:43:36.062400+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:21.384536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:41:21.384536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":595,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.152089+0000","last_change":"2024-02-10T08:43:36.075699+0000","last_active":"2024-02-10T08:45:50.152089+0000","last_peered":"2024-02-10T08:45:50.152089+0000","last_clean":"2024-02-10T08:43:31.066554+0000","last_became_active":"2024-02-10T08:43:36.075699+0000","last_became_peered":"2024-02-10T08:43:36.075699+0000","last_unstale":"2024-02-10T08:45:50.152089+0000","last_undegraded":"2024-02-10T08:45:50.152089+0000","last_fullsized":"2024-02-10T08:43:36.063912+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:45.347685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:45.347685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":521,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.147359+0000","last_change":"2024-02-10T08:43:36.075546+0000","last_active":"2024-02-10T08:45:50.147359+0000","last_peered":"2024-02-10T08:45:50.147359+0000","last_clean":"2024-02-10T08:43:11.068374+0000","last_became_active":"2024-02-10T08:43:36.075546+0000","last_became_peered":"2024-02-10T08:43:36.075546+0000","last_unstale":"2024-02-10T08:45:50.147359+0000","last_undegraded":"2024-02-10T08:45:50.147359+0000","last_fullsized":"2024-02-10T08:43:36.057136+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:26.614964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:26.614964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":625,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127225+0000","last_change":"2024-02-10T08:43:52.173893+0000","last_active":"2024-02-10T08:45:49.127225+0000","last_peered":"2024-02-10T08:45:49.127225+0000","last_clean":"2024-02-10T08:45:49.127225+0000","last_became_active":"2024-02-10T08:43:15.125753+0000","last_became_peered":"2024-02-10T08:43:15.125753+0000","last_unstale":"2024-02-10T08:45:49.127225+0000","last_undegraded":"2024-02-10T08:45:49.127225+0000","last_fullsized":"2024-02-10T08:45:49.127225+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:43:52.173797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:52.173797+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":578,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.175732+0000","last_change":"2024-02-10T08:43:36.068896+0000","last_active":"2024-02-10T08:45:49.175732+0000","last_peered":"2024-02-10T08:45:49.175732+0000","last_clean":"2024-02-10T08:43:31.156388+0000","last_became_active":"2024-02-10T08:43:36.068896+0000","last_became_peered":"2024-02-10T08:43:36.068896+0000","last_unstale":"2024-02-10T08:45:49.175732+0000","last_undegraded":"2024-02-10T08:45:49.175732+0000","last_fullsized":"2024-02-10T08:43:36.062533+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:02.183626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:02.183626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.229483+0000","last_change":"2024-02-10T08:43:36.074707+0000","last_active":"2024-02-10T08:45:49.229483+0000","last_peered":"2024-02-10T08:45:49.229483+0000","last_clean":"2024-02-10T08:43:31.060201+0000","last_became_active":"2024-02-10T08:43:36.074707+0000","last_became_peered":"2024-02-10T08:43:36.074707+0000","last_unstale":"2024-02-10T08:45:49.229483+0000","last_undegraded":"2024-02-10T08:45:49.229483+0000","last_fullsized":"2024-02-10T08:43:36.061288+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:35.401884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:35.401884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":520,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.123707+0000","last_change":"2024-02-10T08:43:36.065698+0000","last_active":"2024-02-10T08:45:49.123707+0000","last_peered":"2024-02-10T08:45:49.123707+0000","last_clean":"2024-02-10T08:43:31.061677+0000","last_became_active":"2024-02-10T08:43:36.065698+0000","last_became_peered":"2024-02-10T08:43:36.065698+0000","last_unstale":"2024-02-10T08:45:49.123707+0000","last_undegraded":"2024-02-10T08:45:49.123707+0000","last_fullsized":"2024-02-10T08:43:36.058144+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.032707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.032707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":645,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124107+0000","last_change":"2024-02-10T08:43:19.438856+0000","last_active":"2024-02-10T08:45:49.124107+0000","last_peered":"2024-02-10T08:45:49.124107+0000","last_clean":"2024-02-10T08:45:49.124107+0000","last_became_active":"2024-02-10T08:43:15.137251+0000","last_became_peered":"2024-02-10T08:43:15.137251+0000","last_unstale":"2024-02-10T08:45:49.124107+0000","last_undegraded":"2024-02-10T08:45:49.124107+0000","last_fullsized":"2024-02-10T08:45:49.124107+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'10","last_scrub_stamp":"2024-02-10T08:42:24.087519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:42:24.087519+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":473,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123277+0000","last_change":"2024-02-10T08:45:36.300011+0000","last_active":"2024-02-10T08:45:49.123277+0000","last_peered":"2024-02-10T08:45:49.123277+0000","last_clean":"2024-02-10T08:45:49.123277+0000","last_became_active":"2024-02-10T08:45:36.299657+0000","last_became_peered":"2024-02-10T08:45:36.299657+0000","last_unstale":"2024-02-10T08:45:49.123277+0000","last_undegraded":"2024-02-10T08:45:49.123277+0000","last_fullsized":"2024-02-10T08:45:49.123277+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:42.059708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:42.059708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"137'9","reported_seq":743,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.125051+0000","last_change":"2024-02-10T08:45:21.041083+0000","last_active":"2024-02-10T08:45:49.125051+0000","last_peered":"2024-02-10T08:45:49.125051+0000","last_clean":"2024-02-10T08:45:49.125051+0000","last_became_active":"2024-02-10T08:37:10.240946+0000","last_became_peered":"2024-02-10T08:37:10.240946+0000","last_unstale":"2024-02-10T08:45:49.125051+0000","last_undegraded":"2024-02-10T08:45:49.125051+0000","last_fullsized":"2024-02-10T08:45:49.125051+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:45:21.040990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:21.040990+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":157,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.230397+0000","last_change":"2024-02-10T08:43:36.078925+0000","last_active":"2024-02-10T08:45:49.230397+0000","last_peered":"2024-02-10T08:45:49.230397+0000","last_clean":"2024-02-10T08:43:31.060910+0000","last_became_active":"2024-02-10T08:43:36.078925+0000","last_became_peered":"2024-02-10T08:43:36.078925+0000","last_unstale":"2024-02-10T08:45:49.230397+0000","last_undegraded":"2024-02-10T08:45:49.230397+0000","last_fullsized":"2024-02-10T08:43:36.063718+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:43:30.241405+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":491,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.175804+0000","last_change":"2024-02-10T08:43:36.067846+0000","last_active":"2024-02-10T08:45:49.175804+0000","last_peered":"2024-02-10T08:45:49.175804+0000","last_clean":"2024-02-10T08:43:31.156305+0000","last_became_active":"2024-02-10T08:43:36.067846+0000","last_became_peered":"2024-02-10T08:43:36.067846+0000","last_unstale":"2024-02-10T08:45:49.175804+0000","last_undegraded":"2024-02-10T08:45:49.175804+0000","last_fullsized":"2024-02-10T08:43:36.063242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.357151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.357151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":637,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124909+0000","last_change":"2024-02-10T08:45:13.165679+0000","last_active":"2024-02-10T08:45:49.124909+0000","last_peered":"2024-02-10T08:45:49.124909+0000","last_clean":"2024-02-10T08:45:49.124909+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:45:49.124909+0000","last_undegraded":"2024-02-10T08:45:49.124909+0000","last_fullsized":"2024-02-10T08:45:49.124909+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:45:13.165571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.165571+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":96,"reported_epoch":604,"state":"active+clean+remapped","last_fresh":"2024-02-10T08:45:50.148281+0000","last_change":"2024-02-10T08:45:14.450083+0000","last_active":"2024-02-10T08:45:50.148281+0000","last_peered":"2024-02-10T08:45:50.148281+0000","last_clean":"2024-02-10T08:45:50.148281+0000","last_became_active":"2024-02-10T08:44:31.113168+0000","last_became_peered":"2024-02-10T08:44:31.113168+0000","last_unstale":"2024-02-10T08:45:50.148281+0000","last_undegraded":"2024-02-10T08:45:50.148281+0000","last_fullsized":"2024-02-10T08:45:50.148281+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.449994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.449994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":756,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124296+0000","last_change":"2024-02-10T08:43:19.433673+0000","last_active":"2024-02-10T08:45:49.124296+0000","last_peered":"2024-02-10T08:45:49.124296+0000","last_clean":"2024-02-10T08:45:49.124296+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:45:49.124296+0000","last_undegraded":"2024-02-10T08:45:49.124296+0000","last_fullsized":"2024-02-10T08:45:49.124296+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'8","last_scrub_stamp":"2024-02-10T08:42:28.051420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:28.051420+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":132,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.151421+0000","last_change":"2024-02-10T08:43:36.075017+0000","last_active":"2024-02-10T08:45:50.151421+0000","last_peered":"2024-02-10T08:45:50.151421+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.075017+0000","last_became_peered":"2024-02-10T08:43:36.075017+0000","last_unstale":"2024-02-10T08:45:50.151421+0000","last_undegraded":"2024-02-10T08:45:50.151421+0000","last_fullsized":"2024-02-10T08:43:36.058808+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:13.786130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:13.786130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":538,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.150739+0000","last_change":"2024-02-10T08:45:09.467738+0000","last_active":"2024-02-10T08:45:50.150739+0000","last_peered":"2024-02-10T08:45:50.150739+0000","last_clean":"2024-02-10T08:45:50.150739+0000","last_became_active":"2024-02-10T08:36:37.040534+0000","last_became_peered":"2024-02-10T08:36:37.040534+0000","last_unstale":"2024-02-10T08:45:50.150739+0000","last_undegraded":"2024-02-10T08:45:50.150739+0000","last_fullsized":"2024-02-10T08:45:50.150739+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:45:09.467617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:09.467617+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":612,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148816+0000","last_change":"2024-02-10T08:44:30.727741+0000","last_active":"2024-02-10T08:45:50.148816+0000","last_peered":"2024-02-10T08:45:50.148816+0000","last_clean":"2024-02-10T08:45:50.148816+0000","last_became_active":"2024-02-10T08:33:20.104620+0000","last_became_peered":"2024-02-10T08:33:20.104620+0000","last_unstale":"2024-02-10T08:45:50.148816+0000","last_undegraded":"2024-02-10T08:45:50.148816+0000","last_fullsized":"2024-02-10T08:45:50.148816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:30.727654+0000","last_deep_scrub":"78'13","last_deep_scrub_stamp":"2024-02-10T08:36:31.434047+0000","last_clean_scrub_stamp":"2024-02-10T08:44:30.727654+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":582,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.176982+0000","last_change":"2024-02-10T08:45:14.481063+0000","last_active":"2024-02-10T08:45:49.176982+0000","last_peered":"2024-02-10T08:45:49.176982+0000","last_clean":"2024-02-10T08:45:49.176982+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:45:49.176982+0000","last_undegraded":"2024-02-10T08:45:49.176982+0000","last_fullsized":"2024-02-10T08:45:49.176982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.480995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.480995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":585,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126885+0000","last_change":"2024-02-10T08:45:07.459730+0000","last_active":"2024-02-10T08:45:49.126885+0000","last_peered":"2024-02-10T08:45:49.126885+0000","last_clean":"2024-02-10T08:45:49.126885+0000","last_became_active":"2024-02-10T08:37:10.315382+0000","last_became_peered":"2024-02-10T08:37:10.315382+0000","last_unstale":"2024-02-10T08:45:49.126885+0000","last_undegraded":"2024-02-10T08:45:49.126885+0000","last_fullsized":"2024-02-10T08:45:49.126885+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:07.459661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.459661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":489,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123300+0000","last_change":"2024-02-10T08:43:59.469912+0000","last_active":"2024-02-10T08:45:49.123300+0000","last_peered":"2024-02-10T08:45:49.123300+0000","last_clean":"2024-02-10T08:45:49.123300+0000","last_became_active":"2024-02-10T08:37:10.149864+0000","last_became_peered":"2024-02-10T08:37:10.149864+0000","last_unstale":"2024-02-10T08:45:49.123300+0000","last_undegraded":"2024-02-10T08:45:49.123300+0000","last_fullsized":"2024-02-10T08:45:49.123300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:59.469850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:59.469850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":531,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.192825+0000","last_change":"2024-02-10T08:44:25.447359+0000","last_active":"2024-02-10T08:45:49.192825+0000","last_peered":"2024-02-10T08:45:49.192825+0000","last_clean":"2024-02-10T08:45:49.192825+0000","last_became_active":"2024-02-10T08:37:10.202409+0000","last_became_peered":"2024-02-10T08:37:10.202409+0000","last_unstale":"2024-02-10T08:45:49.192825+0000","last_undegraded":"2024-02-10T08:45:49.192825+0000","last_fullsized":"2024-02-10T08:45:49.192825+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.447293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.447293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":592,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.176789+0000","last_change":"2024-02-10T08:43:36.069761+0000","last_active":"2024-02-10T08:45:49.176789+0000","last_peered":"2024-02-10T08:45:49.176789+0000","last_clean":"2024-02-10T08:43:25.738370+0000","last_became_active":"2024-02-10T08:43:36.069761+0000","last_became_peered":"2024-02-10T08:43:36.069761+0000","last_unstale":"2024-02-10T08:45:49.176789+0000","last_undegraded":"2024-02-10T08:43:36.063371+0000","last_fullsized":"2024-02-10T08:43:36.063247+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:43:28.880851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:28.880851+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":599,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148354+0000","last_change":"2024-02-10T08:45:28.640965+0000","last_active":"2024-02-10T08:45:50.148354+0000","last_peered":"2024-02-10T08:45:50.148354+0000","last_clean":"2024-02-10T08:45:50.148354+0000","last_became_active":"2024-02-10T08:37:10.150248+0000","last_became_peered":"2024-02-10T08:37:10.150248+0000","last_unstale":"2024-02-10T08:45:50.148354+0000","last_undegraded":"2024-02-10T08:45:50.148354+0000","last_fullsized":"2024-02-10T08:45:50.148354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:28.640899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:45:28.640899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":579,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.148893+0000","last_change":"2024-02-10T08:44:35.784832+0000","last_active":"2024-02-10T08:45:50.148893+0000","last_peered":"2024-02-10T08:45:50.148893+0000","last_clean":"2024-02-10T08:45:50.148893+0000","last_became_active":"2024-02-10T08:33:22.119878+0000","last_became_peered":"2024-02-10T08:33:22.119878+0000","last_unstale":"2024-02-10T08:45:50.148893+0000","last_undegraded":"2024-02-10T08:45:50.148893+0000","last_fullsized":"2024-02-10T08:45:50.148893+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:35.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:44:35.784716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":578,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127582+0000","last_change":"2024-02-10T08:42:27.081166+0000","last_active":"2024-02-10T08:45:49.127582+0000","last_peered":"2024-02-10T08:45:49.127582+0000","last_clean":"2024-02-10T08:45:49.127582+0000","last_became_active":"2024-02-10T08:37:10.264945+0000","last_became_peered":"2024-02-10T08:37:10.264945+0000","last_unstale":"2024-02-10T08:45:49.127582+0000","last_undegraded":"2024-02-10T08:45:49.127582+0000","last_fullsized":"2024-02-10T08:45:49.127582+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:27.081067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:27.081067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":534,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.127068+0000","last_change":"2024-02-10T08:43:36.081224+0000","last_active":"2024-02-10T08:45:49.127068+0000","last_peered":"2024-02-10T08:45:49.127068+0000","last_clean":"2024-02-10T08:43:31.059324+0000","last_became_active":"2024-02-10T08:43:36.081224+0000","last_became_peered":"2024-02-10T08:43:36.081224+0000","last_unstale":"2024-02-10T08:45:49.127068+0000","last_undegraded":"2024-02-10T08:45:49.127068+0000","last_fullsized":"2024-02-10T08:43:36.068534+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:32.012854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:32.012854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":585,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123502+0000","last_change":"2024-02-10T08:43:34.096276+0000","last_active":"2024-02-10T08:45:49.123502+0000","last_peered":"2024-02-10T08:45:49.123502+0000","last_clean":"2024-02-10T08:45:49.123502+0000","last_became_active":"2024-02-10T08:33:20.113907+0000","last_became_peered":"2024-02-10T08:33:20.113907+0000","last_unstale":"2024-02-10T08:45:49.123502+0000","last_undegraded":"2024-02-10T08:45:49.123502+0000","last_fullsized":"2024-02-10T08:45:49.123502+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:43:34.096208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.096208+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":580,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.124688+0000","last_change":"2024-02-10T08:43:23.521301+0000","last_active":"2024-02-10T08:45:49.124688+0000","last_peered":"2024-02-10T08:45:49.124688+0000","last_clean":"2024-02-10T08:45:49.124688+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:45:49.124688+0000","last_undegraded":"2024-02-10T08:45:49.124688+0000","last_fullsized":"2024-02-10T08:45:49.124688+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:23.521227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:23.521227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":594,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.175601+0000","last_change":"2024-02-10T08:43:36.072045+0000","last_active":"2024-02-10T08:45:49.175601+0000","last_peered":"2024-02-10T08:45:49.175601+0000","last_clean":"2024-02-10T08:43:31.155301+0000","last_became_active":"2024-02-10T08:43:36.072045+0000","last_became_peered":"2024-02-10T08:43:36.072045+0000","last_unstale":"2024-02-10T08:45:49.175601+0000","last_undegraded":"2024-02-10T08:45:49.175601+0000","last_fullsized":"2024-02-10T08:43:36.062267+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.508636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.508636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.228341+0000","last_change":"2024-02-10T08:43:36.072813+0000","last_active":"2024-02-10T08:45:49.228341+0000","last_peered":"2024-02-10T08:45:49.228341+0000","last_clean":"2024-02-10T08:43:31.060566+0000","last_became_active":"2024-02-10T08:43:36.072813+0000","last_became_peered":"2024-02-10T08:43:36.072813+0000","last_unstale":"2024-02-10T08:45:49.228341+0000","last_undegraded":"2024-02-10T08:45:49.228341+0000","last_fullsized":"2024-02-10T08:43:36.061933+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:03.331550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:03.331550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":537,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123531+0000","last_change":"2024-02-10T08:43:15.137758+0000","last_active":"2024-02-10T08:45:49.123531+0000","last_peered":"2024-02-10T08:45:49.123531+0000","last_clean":"2024-02-10T08:45:49.123531+0000","last_became_active":"2024-02-10T08:43:15.137437+0000","last_became_peered":"2024-02-10T08:43:15.137437+0000","last_unstale":"2024-02-10T08:45:49.123531+0000","last_undegraded":"2024-02-10T08:45:49.123531+0000","last_fullsized":"2024-02-10T08:45:49.123531+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:11.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:43:11.116858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":646,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127925+0000","last_change":"2024-02-10T08:44:28.331732+0000","last_active":"2024-02-10T08:45:49.127925+0000","last_peered":"2024-02-10T08:45:49.127925+0000","last_clean":"2024-02-10T08:45:49.127925+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:45:49.127925+0000","last_undegraded":"2024-02-10T08:45:49.127925+0000","last_fullsized":"2024-02-10T08:45:49.127925+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:28.331601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:28.331601+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":582,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.191886+0000","last_change":"2024-02-10T08:45:12.464783+0000","last_active":"2024-02-10T08:45:49.191886+0000","last_peered":"2024-02-10T08:45:49.191886+0000","last_clean":"2024-02-10T08:45:49.191886+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:45:49.191886+0000","last_undegraded":"2024-02-10T08:45:49.191886+0000","last_fullsized":"2024-02-10T08:45:49.191886+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:12.464721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:12.464721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.230606+0000","last_change":"2024-02-10T08:43:36.074002+0000","last_active":"2024-02-10T08:45:49.230606+0000","last_peered":"2024-02-10T08:45:49.230606+0000","last_clean":"2024-02-10T08:43:31.060062+0000","last_became_active":"2024-02-10T08:43:36.074002+0000","last_became_peered":"2024-02-10T08:43:36.074002+0000","last_unstale":"2024-02-10T08:45:49.230606+0000","last_undegraded":"2024-02-10T08:45:49.230606+0000","last_fullsized":"2024-02-10T08:43:36.060407+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:42:46.341979+0000","last_clean_scrub_stamp":"2024-02-10T08:42:46.341979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":578,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126965+0000","last_change":"2024-02-10T08:42:43.111366+0000","last_active":"2024-02-10T08:45:49.126965+0000","last_peered":"2024-02-10T08:45:49.126965+0000","last_clean":"2024-02-10T08:45:49.126965+0000","last_became_active":"2024-02-10T08:37:10.259965+0000","last_became_peered":"2024-02-10T08:37:10.259965+0000","last_unstale":"2024-02-10T08:45:49.126965+0000","last_undegraded":"2024-02-10T08:45:49.126965+0000","last_fullsized":"2024-02-10T08:45:49.126965+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.111309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.111309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":495,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.177488+0000","last_change":"2024-02-10T08:45:22.562907+0000","last_active":"2024-02-10T08:45:49.177488+0000","last_peered":"2024-02-10T08:45:49.177488+0000","last_clean":"2024-02-10T08:45:49.177488+0000","last_became_active":"2024-02-10T08:37:10.147862+0000","last_became_peered":"2024-02-10T08:37:10.147862+0000","last_unstale":"2024-02-10T08:45:49.177488+0000","last_undegraded":"2024-02-10T08:45:49.177488+0000","last_fullsized":"2024-02-10T08:45:49.177488+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.562838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.562838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":178,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.230529+0000","last_change":"2024-02-10T08:44:25.556111+0000","last_active":"2024-02-10T08:45:49.230529+0000","last_peered":"2024-02-10T08:45:49.230529+0000","last_clean":"2024-02-10T08:45:49.230529+0000","last_became_active":"2024-02-10T08:43:15.139755+0000","last_became_peered":"2024-02-10T08:43:15.139755+0000","last_unstale":"2024-02-10T08:45:49.230529+0000","last_undegraded":"2024-02-10T08:45:49.230529+0000","last_fullsized":"2024-02-10T08:45:49.230529+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:44:25.556047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.556047+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":609,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.123226+0000","last_change":"2024-02-10T08:43:36.066663+0000","last_active":"2024-02-10T08:45:49.123226+0000","last_peered":"2024-02-10T08:45:49.123226+0000","last_clean":"2024-02-10T08:43:31.060592+0000","last_became_active":"2024-02-10T08:43:36.066663+0000","last_became_peered":"2024-02-10T08:43:36.066663+0000","last_unstale":"2024-02-10T08:45:49.123226+0000","last_undegraded":"2024-02-10T08:45:49.123226+0000","last_fullsized":"2024-02-10T08:43:36.056257+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:40.131457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:40.131457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":155,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.228415+0000","last_change":"2024-02-10T08:44:34.562436+0000","last_active":"2024-02-10T08:45:49.228415+0000","last_peered":"2024-02-10T08:45:49.228415+0000","last_clean":"2024-02-10T08:45:49.228415+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:45:49.228415+0000","last_undegraded":"2024-02-10T08:45:49.228415+0000","last_fullsized":"2024-02-10T08:45:49.228415+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:44:34.562370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:44:34.562370+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":543,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.150682+0000","last_change":"2024-02-10T08:44:29.460927+0000","last_active":"2024-02-10T08:45:50.150682+0000","last_peered":"2024-02-10T08:45:50.150682+0000","last_clean":"2024-02-10T08:45:50.150682+0000","last_became_active":"2024-02-10T08:43:15.119127+0000","last_became_peered":"2024-02-10T08:43:15.119127+0000","last_unstale":"2024-02-10T08:45:50.150682+0000","last_undegraded":"2024-02-10T08:45:50.150682+0000","last_fullsized":"2024-02-10T08:45:50.150682+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:29.460863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:29.460863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":611,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.192062+0000","last_change":"2024-02-10T08:44:41.392510+0000","last_active":"2024-02-10T08:45:49.192062+0000","last_peered":"2024-02-10T08:45:49.192062+0000","last_clean":"2024-02-10T08:45:49.192062+0000","last_became_active":"2024-02-10T08:33:20.109373+0000","last_became_peered":"2024-02-10T08:33:20.109373+0000","last_unstale":"2024-02-10T08:45:49.192062+0000","last_undegraded":"2024-02-10T08:45:49.192062+0000","last_fullsized":"2024-02-10T08:45:49.192062+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:44:41.392413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:41.392413+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":584,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127733+0000","last_change":"2024-02-10T08:41:58.956970+0000","last_active":"2024-02-10T08:45:49.127733+0000","last_peered":"2024-02-10T08:45:49.127733+0000","last_clean":"2024-02-10T08:45:49.127733+0000","last_became_active":"2024-02-10T08:37:10.265420+0000","last_became_peered":"2024-02-10T08:37:10.265420+0000","last_unstale":"2024-02-10T08:45:49.127733+0000","last_undegraded":"2024-02-10T08:45:49.127733+0000","last_fullsized":"2024-02-10T08:45:49.127733+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:58.956903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:41:58.956903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":578,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.148017+0000","last_change":"2024-02-10T08:43:36.073017+0000","last_active":"2024-02-10T08:45:50.148017+0000","last_peered":"2024-02-10T08:45:50.148017+0000","last_clean":"2024-02-10T08:43:31.058620+0000","last_became_active":"2024-02-10T08:43:36.073017+0000","last_became_peered":"2024-02-10T08:43:36.073017+0000","last_unstale":"2024-02-10T08:45:50.148017+0000","last_undegraded":"2024-02-10T08:45:50.148017+0000","last_fullsized":"2024-02-10T08:43:36.062639+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:49.644718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:49.644718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":527,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.148465+0000","last_change":"2024-02-10T08:43:36.085258+0000","last_active":"2024-02-10T08:45:50.148465+0000","last_peered":"2024-02-10T08:45:50.148465+0000","last_clean":"2024-02-10T08:43:31.059986+0000","last_became_active":"2024-02-10T08:43:36.085258+0000","last_became_peered":"2024-02-10T08:43:36.085258+0000","last_unstale":"2024-02-10T08:45:50.148465+0000","last_undegraded":"2024-02-10T08:45:50.148465+0000","last_fullsized":"2024-02-10T08:43:36.058326+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:23.662121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:41:23.662121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":655,"reported_epoch":604,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:50.149722+0000","last_change":"2024-02-10T08:43:36.088112+0000","last_active":"2024-02-10T08:45:50.149722+0000","last_peered":"2024-02-10T08:45:50.149722+0000","last_clean":"2024-02-10T08:43:31.060512+0000","last_became_active":"2024-02-10T08:43:36.088112+0000","last_became_peered":"2024-02-10T08:43:36.088112+0000","last_unstale":"2024-02-10T08:45:50.149722+0000","last_undegraded":"2024-02-10T08:43:36.063487+0000","last_fullsized":"2024-02-10T08:43:36.063380+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'22","last_scrub_stamp":"2024-02-10T08:41:45.260157+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:41:45.260157+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":596,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.176344+0000","last_change":"2024-02-10T08:43:36.078681+0000","last_active":"2024-02-10T08:45:49.176344+0000","last_peered":"2024-02-10T08:45:49.176344+0000","last_clean":"2024-02-10T08:43:31.155537+0000","last_became_active":"2024-02-10T08:43:36.078681+0000","last_became_peered":"2024-02-10T08:43:36.078681+0000","last_unstale":"2024-02-10T08:45:49.176344+0000","last_undegraded":"2024-02-10T08:45:49.176344+0000","last_fullsized":"2024-02-10T08:43:36.069413+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:23.333067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:23.333067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":125,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.192481+0000","last_change":"2024-02-10T08:43:36.071762+0000","last_active":"2024-02-10T08:45:49.192481+0000","last_peered":"2024-02-10T08:45:49.192481+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.071762+0000","last_became_peered":"2024-02-10T08:43:36.071762+0000","last_unstale":"2024-02-10T08:45:49.192481+0000","last_undegraded":"2024-02-10T08:45:49.192481+0000","last_fullsized":"2024-02-10T08:43:36.062825+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.031375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.031375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":495,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.175712+0000","last_change":"2024-02-10T08:45:10.423028+0000","last_active":"2024-02-10T08:45:49.175712+0000","last_peered":"2024-02-10T08:45:49.175712+0000","last_clean":"2024-02-10T08:45:49.175712+0000","last_became_active":"2024-02-10T08:37:10.231761+0000","last_became_peered":"2024-02-10T08:37:10.231761+0000","last_unstale":"2024-02-10T08:45:49.175712+0000","last_undegraded":"2024-02-10T08:45:49.175712+0000","last_fullsized":"2024-02-10T08:45:49.175712+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.422960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.422960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":632,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.125807+0000","last_change":"2024-02-10T08:43:36.081271+0000","last_active":"2024-02-10T08:45:49.125807+0000","last_peered":"2024-02-10T08:45:49.125807+0000","last_clean":"2024-02-10T08:43:31.058844+0000","last_became_active":"2024-02-10T08:43:36.081271+0000","last_became_peered":"2024-02-10T08:43:36.081271+0000","last_unstale":"2024-02-10T08:45:49.125807+0000","last_undegraded":"2024-02-10T08:43:36.068229+0000","last_fullsized":"2024-02-10T08:43:36.068095+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-10T08:41:53.828608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:53.828608+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":598,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126843+0000","last_change":"2024-02-10T08:45:11.455867+0000","last_active":"2024-02-10T08:45:49.126843+0000","last_peered":"2024-02-10T08:45:49.126843+0000","last_clean":"2024-02-10T08:45:49.126843+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:45:49.126843+0000","last_undegraded":"2024-02-10T08:45:49.126843+0000","last_fullsized":"2024-02-10T08:45:49.126843+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:11.455805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:11.455805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":569,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127295+0000","last_change":"2024-02-10T08:42:37.092622+0000","last_active":"2024-02-10T08:45:49.127295+0000","last_peered":"2024-02-10T08:45:49.127295+0000","last_clean":"2024-02-10T08:45:49.127295+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:45:49.127295+0000","last_undegraded":"2024-02-10T08:45:49.127295+0000","last_fullsized":"2024-02-10T08:45:49.127295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:37.092522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:42:37.092522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":542,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.147437+0000","last_change":"2024-02-10T08:45:22.585286+0000","last_active":"2024-02-10T08:45:50.147437+0000","last_peered":"2024-02-10T08:45:50.147437+0000","last_clean":"2024-02-10T08:45:50.147437+0000","last_became_active":"2024-02-10T08:43:28.080931+0000","last_became_peered":"2024-02-10T08:43:28.080931+0000","last_unstale":"2024-02-10T08:45:50.147437+0000","last_undegraded":"2024-02-10T08:45:50.147437+0000","last_fullsized":"2024-02-10T08:45:50.147437+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.585215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.585215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":616,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.123385+0000","last_change":"2024-02-10T08:43:36.080654+0000","last_active":"2024-02-10T08:45:49.123385+0000","last_peered":"2024-02-10T08:45:49.123385+0000","last_clean":"2024-02-10T08:43:25.677981+0000","last_became_active":"2024-02-10T08:43:36.080654+0000","last_became_peered":"2024-02-10T08:43:36.080654+0000","last_unstale":"2024-02-10T08:45:49.123385+0000","last_undegraded":"2024-02-10T08:43:36.066356+0000","last_fullsized":"2024-02-10T08:43:36.066242+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:41:19.765193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:41:19.765193+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":156,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.228489+0000","last_change":"2024-02-10T08:45:20.443139+0000","last_active":"2024-02-10T08:45:49.228489+0000","last_peered":"2024-02-10T08:45:49.228489+0000","last_clean":"2024-02-10T08:45:49.228489+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:45:49.228489+0000","last_undegraded":"2024-02-10T08:45:49.228489+0000","last_fullsized":"2024-02-10T08:45:49.228489+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:20.443077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:45:20.443077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":519,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-10T08:45:50.146938+0000","last_change":"2024-02-10T08:45:15.556725+0000","last_active":"2024-02-10T08:45:50.146938+0000","last_peered":"2024-02-10T08:45:50.146938+0000","last_clean":"2024-02-10T08:45:50.146938+0000","last_became_active":"2024-02-10T08:43:15.131035+0000","last_became_peered":"2024-02-10T08:43:15.131035+0000","last_unstale":"2024-02-10T08:45:50.146938+0000","last_undegraded":"2024-02-10T08:45:50.146938+0000","last_fullsized":"2024-02-10T08:45:50.146938+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:15.556636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:15.556636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":535,"reported_epoch":604,"state":"active+undersized","last_fresh":"2024-02-10T08:45:50.153014+0000","last_change":"2024-02-10T08:43:36.078496+0000","last_active":"2024-02-10T08:45:50.153014+0000","last_peered":"2024-02-10T08:45:50.153014+0000","last_clean":"2024-02-10T08:43:31.066991+0000","last_became_active":"2024-02-10T08:43:36.078496+0000","last_became_peered":"2024-02-10T08:43:36.078496+0000","last_unstale":"2024-02-10T08:45:50.153014+0000","last_undegraded":"2024-02-10T08:45:50.153014+0000","last_fullsized":"2024-02-10T08:43:36.061038+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:43.399709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:43.399709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":587,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.126725+0000","last_change":"2024-02-10T08:43:36.081968+0000","last_active":"2024-02-10T08:45:49.126725+0000","last_peered":"2024-02-10T08:45:49.126725+0000","last_clean":"2024-02-10T08:43:25.647108+0000","last_became_active":"2024-02-10T08:43:36.081968+0000","last_became_peered":"2024-02-10T08:43:36.081968+0000","last_unstale":"2024-02-10T08:45:49.126725+0000","last_undegraded":"2024-02-10T08:43:36.066991+0000","last_fullsized":"2024-02-10T08:43:36.066884+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'7","last_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:42:03.646375+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":151,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.229658+0000","last_change":"2024-02-10T08:45:13.432001+0000","last_active":"2024-02-10T08:45:49.229658+0000","last_peered":"2024-02-10T08:45:49.229658+0000","last_clean":"2024-02-10T08:45:49.229658+0000","last_became_active":"2024-02-10T08:43:15.134426+0000","last_became_peered":"2024-02-10T08:43:15.134426+0000","last_unstale":"2024-02-10T08:45:49.229658+0000","last_undegraded":"2024-02-10T08:45:49.229658+0000","last_fullsized":"2024-02-10T08:45:49.229658+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:13.431922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.431922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":570,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.123403+0000","last_change":"2024-02-10T08:45:25.022215+0000","last_active":"2024-02-10T08:45:49.123403+0000","last_peered":"2024-02-10T08:45:49.123403+0000","last_clean":"2024-02-10T08:45:49.123403+0000","last_became_active":"2024-02-10T08:33:24.118639+0000","last_became_peered":"2024-02-10T08:33:24.118639+0000","last_unstale":"2024-02-10T08:45:49.123403+0000","last_undegraded":"2024-02-10T08:45:49.123403+0000","last_fullsized":"2024-02-10T08:45:49.123403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.022136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.022136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":498,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.125256+0000","last_change":"2024-02-10T08:44:25.269848+0000","last_active":"2024-02-10T08:45:49.125256+0000","last_peered":"2024-02-10T08:45:49.125256+0000","last_clean":"2024-02-10T08:45:49.125256+0000","last_became_active":"2024-02-10T08:40:42.028166+0000","last_became_peered":"2024-02-10T08:40:42.028166+0000","last_unstale":"2024-02-10T08:45:49.125256+0000","last_undegraded":"2024-02-10T08:45:49.125256+0000","last_fullsized":"2024-02-10T08:45:49.125256+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:44:25.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:44:25.269759+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":610,"reported_epoch":603,"state":"active+undersized+degraded","last_fresh":"2024-02-10T08:45:49.176030+0000","last_change":"2024-02-10T08:43:36.080498+0000","last_active":"2024-02-10T08:45:49.176030+0000","last_peered":"2024-02-10T08:45:49.176030+0000","last_clean":"2024-02-10T08:43:25.563172+0000","last_became_active":"2024-02-10T08:43:36.080498+0000","last_became_peered":"2024-02-10T08:43:36.080498+0000","last_unstale":"2024-02-10T08:45:49.176030+0000","last_undegraded":"2024-02-10T08:43:36.063844+0000","last_fullsized":"2024-02-10T08:43:36.063724+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'9","last_scrub_stamp":"2024-02-10T08:43:22.036213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:22.036213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":584,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126575+0000","last_change":"2024-02-10T08:43:14.144599+0000","last_active":"2024-02-10T08:45:49.126575+0000","last_peered":"2024-02-10T08:45:49.126575+0000","last_clean":"2024-02-10T08:45:49.126575+0000","last_became_active":"2024-02-10T08:37:10.279283+0000","last_became_peered":"2024-02-10T08:37:10.279283+0000","last_unstale":"2024-02-10T08:45:49.126575+0000","last_undegraded":"2024-02-10T08:45:49.126575+0000","last_fullsized":"2024-02-10T08:45:49.126575+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:14.144537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:43:14.144537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":130,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.123557+0000","last_change":"2024-02-10T08:43:36.070306+0000","last_active":"2024-02-10T08:45:49.123557+0000","last_peered":"2024-02-10T08:45:49.123557+0000","last_clean":"2024-02-10T08:33:23.102654+0000","last_became_active":"2024-02-10T08:43:36.070306+0000","last_became_peered":"2024-02-10T08:43:36.070306+0000","last_unstale":"2024-02-10T08:45:49.123557+0000","last_undegraded":"2024-02-10T08:45:49.123557+0000","last_fullsized":"2024-02-10T08:43:36.056977+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:55.669798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:41:55.669798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":529,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.124718+0000","last_change":"2024-02-10T08:43:36.069163+0000","last_active":"2024-02-10T08:45:49.124718+0000","last_peered":"2024-02-10T08:45:49.124718+0000","last_clean":"2024-02-10T08:43:31.063295+0000","last_became_active":"2024-02-10T08:43:36.069163+0000","last_became_peered":"2024-02-10T08:43:36.069163+0000","last_unstale":"2024-02-10T08:45:49.124718+0000","last_undegraded":"2024-02-10T08:45:49.124718+0000","last_fullsized":"2024-02-10T08:43:36.056950+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:41:16.004167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:41:16.004167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":641,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.127955+0000","last_change":"2024-02-10T08:44:31.267329+0000","last_active":"2024-02-10T08:45:49.127955+0000","last_peered":"2024-02-10T08:45:49.127955+0000","last_clean":"2024-02-10T08:45:49.127955+0000","last_became_active":"2024-02-10T08:37:10.265888+0000","last_became_peered":"2024-02-10T08:37:10.265888+0000","last_unstale":"2024-02-10T08:45:49.127955+0000","last_undegraded":"2024-02-10T08:45:49.127955+0000","last_fullsized":"2024-02-10T08:45:49.127955+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:44:31.267261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:44:31.267261+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":152,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.231554+0000","last_change":"2024-02-10T08:43:36.074560+0000","last_active":"2024-02-10T08:45:49.231554+0000","last_peered":"2024-02-10T08:45:49.231554+0000","last_clean":"2024-02-10T08:43:31.060871+0000","last_became_active":"2024-02-10T08:43:36.074560+0000","last_became_peered":"2024-02-10T08:43:36.074560+0000","last_unstale":"2024-02-10T08:45:49.231554+0000","last_undegraded":"2024-02-10T08:45:49.231554+0000","last_fullsized":"2024-02-10T08:43:36.062458+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:48.325433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:42:48.325433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":587,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-10T08:45:49.126577+0000","last_change":"2024-02-10T08:43:55.185139+0000","last_active":"2024-02-10T08:45:49.126577+0000","last_peered":"2024-02-10T08:45:49.126577+0000","last_clean":"2024-02-10T08:45:49.126577+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:45:49.126577+0000","last_undegraded":"2024-02-10T08:45:49.126577+0000","last_fullsized":"2024-02-10T08:45:49.126577+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:55.185039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:55.185039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":530,"reported_epoch":603,"state":"active+undersized","last_fresh":"2024-02-10T08:45:49.177208+0000","last_change":"2024-02-10T08:43:36.085682+0000","last_active":"2024-02-10T08:45:49.177208+0000","last_peered":"2024-02-10T08:45:49.177208+0000","last_clean":"2024-02-10T08:43:31.156976+0000","last_became_active":"2024-02-10T08:43:36.085682+0000","last_became_peered":"2024-02-10T08:43:36.085682+0000","last_unstale":"2024-02-10T08:45:49.177208+0000","last_undegraded":"2024-02-10T08:45:49.177208+0000","last_fullsized":"2024-02-10T08:43:36.064612+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:42:05.201314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:42:05.201314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":236,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":24,"acting":24,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":55,"acting":55,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":5506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":720,"ondisk_log_size":720,"up":81,"acting":81,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":745,"num_read_kb":740,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":85,"acting":86,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299545,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304704,"kb_used_data":3384,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440448,"statfs":{"total":95995035648,"available":95683018752,"internally_reserved":0,"allocated":3465216,"data_stored":1734120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.55000000000000004,"15min":0.61099999999999999},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":5.3200000000000003,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.57599999999999996,"15min":0.58699999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":11.000999999999999,"5min":11.000999999999999,"15min":11.000999999999999},"last":1.429}]},{"osd":1,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":0.69899999999999995,"15min":0.60899999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":9.2560000000000002,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.371,"5min":0.69099999999999995,"15min":0.627},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.19900000000000001},"max":{"1min":10.978,"5min":10.978,"15min":10.978},"last":0.93100000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.68400000000000005,"15min":0.60499999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.61599999999999999,"15min":0.67100000000000004},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":19.369},"last":0.51000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":9.3439999999999994,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.68300000000000005,"15min":0.68600000000000005},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":22.102},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":8.827},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":1.903,"15min":13.646000000000001},"last":0.35199999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46000000000000002,"15min":0.52300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":19.326000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.54800000000000004,"15min":0.503},"min":{"1min":0.25600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":10.946999999999999,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.90300000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.47799999999999998,"15min":0.55500000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":22.687000000000001},"last":3.9060000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.484,"5min":5.484,"15min":5.484},"last":0.38800000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561183,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312892,"kb_used_data":3380,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432260,"statfs":{"total":95995035648,"available":95674634240,"internally_reserved":0,"allocated":3461120,"data_stored":1747690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.25900000000000001,"5min":0.23400000000000001,"15min":0.19800000000000001},"max":{"1min":3.9399999999999999,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.798,"5min":0.76500000000000001,"15min":0.628},"min":{"1min":0.254,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":18.777999999999999,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.307}]},{"osd":1,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":0.89700000000000002,"15min":0.65900000000000003},"min":{"1min":0.35399999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":18.425999999999998,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.74299999999999999,"15min":0.58999999999999997},"min":{"1min":0.29099999999999998,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":8.4399999999999995,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":0.66200000000000003,"15min":0.57999999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":10.772,"5min":10.772,"15min":10.772},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.6539999999999999,"5min":0.749,"15min":0.61099999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.246},"max":{"1min":18.829000000000001,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.377}]},{"osd":3,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.63800000000000001,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.0990000000000002,"5min":7.0990000000000002,"15min":16.823},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.318,"15min":0.28699999999999998},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":3.7389999999999999},"last":0.36199999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45300000000000001,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":3.657,"15min":3.657},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.46500000000000002,"15min":0.40500000000000003},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.24399999999999999},"max":{"1min":0.61799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:45:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.64300000000000002,"15min":0.496},"min":{"1min":0.28000000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":18.853999999999999,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.48699999999999999,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":0.32300000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:45:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53500000000000003,"15min":0.5},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.91300000000000003,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.39800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312932,"kb_used_data":3420,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432220,"statfs":{"total":95995035648,"available":95674593280,"internally_reserved":0,"allocated":3502080,"data_stored":1748897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.38300000000000001,"15min":0.42999999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.8710000000000004,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":1.423},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.46100000000000002},"min":{"1min":0.247,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.4910000000000001,"5min":2.1890000000000001,"15min":16.166},"last":1.0940000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.47799999999999998,"15min":0.64900000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.4820000000000002,"5min":7.4820000000000002,"15min":34.985999999999997},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.442,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":21.614999999999998},"last":1.056}]},{"osd":3,"last update":"Sat Feb 10 08:45:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.45200000000000001,"15min":0.53200000000000003},"min":{"1min":0.23100000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.8940000000000001,"5min":4.8940000000000001,"15min":24.773},"last":1.542},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.42499999999999999,"15min":0.52100000000000002},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":2.222,"5min":2.222,"15min":16.120999999999999},"last":1.1699999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.60999999999999999},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.89000000000000001,"15min":16.074000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.60399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":9.5090000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.496,"5min":3.496,"15min":9.5600000000000005},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.31900000000000001},"max":{"1min":4.8310000000000004,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":1.4790000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":1.5489999999999999,"5min":1.996,"15min":9.532},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.41399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.8520000000000003,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":1.4530000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:45:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.70099999999999996,"15min":0.65400000000000003},"min":{"1min":0.376,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":12.147,"5min":12.147,"15min":12.147},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.64500000000000002,"15min":0.68799999999999994},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.8019999999999996,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":1.51}]}]},{"osd":0,"up_from":0,"seq":34359738556,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308816,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436336,"statfs":{"total":95995035648,"available":95678808064,"internally_reserved":0,"allocated":3481600,"data_stored":1747505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":0.63900000000000001,"15min":0.50600000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":16.170000000000002,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":0.68799999999999994,"15min":0.57199999999999995},"min":{"1min":0.217,"5min":0.187,"15min":0.187},"max":{"1min":10.567,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.252}]},{"osd":1,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.626,"15min":0.56399999999999995},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.9720000000000004,"5min":9.6980000000000004,"15min":12.512},"last":0.216},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.59499999999999997,"15min":0.54200000000000004},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":12.875,"5min":12.875,"15min":12.875},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:45:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.73399999999999999,"15min":0.55400000000000005},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":10.994999999999999,"5min":11.612,"15min":11.612},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.66800000000000004,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.252,"15min":0.22600000000000001},"max":{"1min":7.8280000000000003,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.67200000000000004},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76200000000000001,"5min":1.7370000000000001,"15min":10.920999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":1.014,"15min":12.292999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:45:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.71999999999999997,"15min":0.67300000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.8970000000000002,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:45:08 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":0.77100000000000002,"15min":0.72399999999999998},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.714,"5min":0.82499999999999996,"15min":0.71899999999999997},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.36899999999999999},"max":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Feb 10 08:45:30 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":0.747,"15min":0.72899999999999998},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":10.539,"5min":10.539,"15min":10.539},"last":0.434},{"interface":"front","average":{"1min":1.615,"5min":0.78700000000000003,"15min":0.72199999999999998},"min":{"1min":0.371,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.6319999999999997,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.46400000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182574,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308812,"kb_used_data":3396,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436340,"statfs":{"total":95995035648,"available":95678812160,"internally_reserved":0,"allocated":3477504,"data_stored":1734096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32100000000000001,"15min":0.498},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.52800000000000002,"5min":0.878,"15min":14.744},"last":0.221},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.36799999999999999,"15min":0.53700000000000003},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.188},"max":{"1min":0.48999999999999999,"5min":3.5579999999999998,"15min":11.808},"last":0.19600000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.42199999999999999,"15min":0.54800000000000004},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.53500000000000003,"5min":9.3100000000000005,"15min":16.634},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.41599999999999998,"15min":0.55000000000000004},"min":{"1min":0.222,"5min":0.222,"15min":0.215},"max":{"1min":0.54200000000000004,"5min":5.4370000000000003,"15min":12.438000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.57599999999999996},"min":{"1min":0.26000000000000001,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":0.94899999999999995,"5min":3.1549999999999998,"15min":16.536000000000001},"last":0.317},{"interface":"front","average":{"1min":0.377,"5min":0.38300000000000001,"15min":0.622},"min":{"1min":0.247,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.59799999999999998,"5min":2.5230000000000001,"15min":23.294},"last":0.27200000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54500000000000004,"15min":0.66900000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":2.589,"15min":12.371},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68700000000000006,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55000000000000004,"15min":0.70399999999999996},"min":{"1min":0.35499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":0.73899999999999999,"15min":0.70499999999999996},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":17.292000000000002,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.375}]},{"osd":6,"last update":"Sat Feb 10 08:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.57299999999999995,"15min":0.69499999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":3.48,"15min":16.091000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.58799999999999997,"15min":0.68600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":3.2469999999999999,"15min":16.475000000000001},"last":0.33800000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.67400000000000004,"15min":0.69799999999999995},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.056,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56899999999999995,"15min":0.73499999999999999},"min":{"1min":0.41399999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.877,"5min":2.1339999999999999,"15min":12.606999999999999},"last":0.42199999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986316,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312804,"kb_used_data":3292,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432348,"statfs":{"total":95995035648,"available":95674724352,"internally_reserved":0,"allocated":3371008,"data_stored":1427290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.627},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":19.940999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.55300000000000005},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.221},"max":{"1min":0.61599999999999999,"5min":0.91800000000000004,"15min":9.1760000000000002},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46899999999999997,"15min":0.57899999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.59699999999999998,"5min":0.93799999999999994,"15min":5.774},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49399999999999999,"15min":0.59799999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":8.6989999999999998},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.59799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.83999999999999997,"15min":6.2329999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.59799999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":0.64600000000000002,"5min":0.77100000000000002,"15min":10.01},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48399999999999999,"15min":0.65000000000000002},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.27700000000000002},"max":{"1min":0.58299999999999996,"5min":0.92700000000000005,"15min":9.5909999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51900000000000002,"15min":0.71799999999999997},"min":{"1min":0.316,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":25.27},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":0.621,"5min":0.876,"15min":6.8730000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.223},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":3.8290000000000002},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.247},"max":{"1min":0.626,"5min":0.78000000000000003,"15min":13.342000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":1.085,"15min":13.753},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.52000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.69599999999999995,"5min":0.90400000000000003,"15min":12.835000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53800000000000003,"15min":0.57899999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.72599999999999998,"5min":1.0509999999999999,"15min":12.875999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790116,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300636,"kb_used_data":3412,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444516,"statfs":{"total":95995035648,"available":95687184384,"internally_reserved":0,"allocated":3493888,"data_stored":1743450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.59999999999999998,"15min":1.0249999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.1299999999999999,"5min":8.7400000000000002,"15min":107.551},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54400000000000004,"15min":0.96799999999999997},"min":{"1min":0.317,"5min":0.255,"15min":0.22900000000000001},"max":{"1min":1.1479999999999999,"5min":5.375,"15min":88.792000000000002},"last":2.8860000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.63200000000000001,"15min":0.95999999999999996},"min":{"1min":0.311,"5min":0.216,"15min":0.216},"max":{"1min":1.107,"5min":8.7200000000000006,"15min":107.205},"last":1.8049999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.59999999999999998,"15min":0.83999999999999997},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":1.125,"5min":4.7629999999999999,"15min":72.272000000000006},"last":1.4830000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.67000000000000004,"15min":1.0069999999999999},"min":{"1min":0.32200000000000001,"5min":0.247,"15min":0.247},"max":{"1min":9.9600000000000009,"5min":10.794,"15min":54.174999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":1.0549999999999999},"min":{"1min":0.26900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.2230000000000001,"5min":8.6890000000000001,"15min":86.183000000000007},"last":2.157}]},{"osd":3,"last update":"Sat Feb 10 08:45:40 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.29999999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":9.9900000000000002,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.71299999999999997,"15min":1.2609999999999999},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":1.198,"5min":8.7690000000000001,"15min":114.526},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Feb 10 08:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.55900000000000005,"15min":0.72199999999999998},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":10.023999999999999,"5min":10.023999999999999,"15min":72.241},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.55900000000000005,"15min":0.57399999999999995},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.013,"5min":10.013,"15min":22.469999999999999},"last":0.373}]},{"osd":7,"last update":"Sat Feb 10 08:45:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.48399999999999999,"15min":0.71299999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":66.844999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.49199999999999999,"15min":0.74299999999999999},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":72.257000000000005},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:45:57.331 INFO:tasks.thrashosds.thrasher:Setting 2.1d to [3, 4, 2] 2024-02-10T08:45:57.331 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1d', '3', '4', '2'] 2024-02-10T08:45:57.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 2.1d 3 4 2 2024-02-10T08:45:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:57.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:45:57.414 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:57.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:57.462 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:57.534 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:57.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:57.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:45:57.664 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.476975+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.477888+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.829794+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:55.833259+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:56.231973+0000 mon.a (mon.0) 3053 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:56.232047+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:56.235099+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:56.235553+0000 mon.a (mon.0) 3055 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.476975+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.477888+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.829794+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:55.833259+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:56.231973+0000 mon.a (mon.0) 3053 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:56.232047+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:56.235099+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:56.235553+0000 mon.a (mon.0) 3055 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]: dispatch 2024-02-10T08:45:57.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:56.503693+0000 osd.6 (osd.6) 250 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:56.515030+0000 osd.6 (osd.6) 251 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.227298+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:57.231126+0000 mon.a (mon.0) 3057 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]': finished 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.231190+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:57.401830+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:57.402399+0000 mon.a (mon.0) 3059 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.516254+0000 mgr.y (mgr.14553) 524 : cluster [DBG] pgmap v919: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:56.503693+0000 osd.6 (osd.6) 250 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:56.515030+0000 osd.6 (osd.6) 251 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.227298+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:57.231126+0000 mon.a (mon.0) 3057 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]': finished 2024-02-10T08:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.231190+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:57.401830+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:57.402399+0000 mon.a (mon.0) 3059 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.516254+0000 mgr.y (mgr.14553) 524 : cluster [DBG] pgmap v919: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:56.503693+0000 osd.6 (osd.6) 250 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:56.515030+0000 osd.6 (osd.6) 251 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.227298+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:57.231126+0000 mon.a (mon.0) 3057 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-104", "mode": "writeback"}]': finished 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.231190+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:57.401830+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:57.402399+0000 mon.a (mon.0) 3059 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:45:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.516254+0000 mgr.y (mgr.14553) 524 : cluster [DBG] pgmap v919: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:45:58.865 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:58.905 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:45:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:45:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:45:58.905 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:45:58.912 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:45:58.914 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:45:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:58.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:45:59.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:45:59.047 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:59.145 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:45:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:59.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:45:59.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.463282+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.464138+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.863682+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:57.867129+0000 osd.1 (osd.1) 287 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.234505+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:58.242363+0000 mon.a (mon.0) 3061 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.242427+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-10T08:45:59.259 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:58.243393+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.260 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:45:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:58.243684+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.351 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:59.412 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:45:59.414 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.463282+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.464138+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.863682+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:57.867129+0000 osd.1 (osd.1) 287 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.234505+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:58.242363+0000 mon.a (mon.0) 3061 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.242427+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:58.243393+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:58.243684+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.415 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.463282+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.464138+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.863682+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:57.867129+0000 osd.1 (osd.1) 287 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.234505+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 88/585 objects degraded (15.043%), 22 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:58.242363+0000 mon.a (mon.0) 3061 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.242427+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:58.243393+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.416 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:45:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:58.243684+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]: dispatch 2024-02-10T08:45:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:45:59.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:45:59.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:45:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:45:59.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- 172.21.15.107:0/1284044005 >> v1:172.21.15.107:6790/0 conn(0x7fd4a80ff710 legacy=0x7fd4a8101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- 172.21.15.107:0/1284044005 shutdown_connections 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- 172.21.15.107:0/1284044005 >> 172.21.15.107:0/1284044005 conn(0x7fd4a80f2510 msgr2=0x7fd4a80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- 172.21.15.107:0/1284044005 shutdown_connections 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- 172.21.15.107:0/1284044005 wait complete. 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 librados: starting msgr at 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 librados: starting objecter 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 Processor -- start 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 -- start start 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 librados: setting wanted keys 2024-02-10T08:45:59.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.749+0000 7fd4aed02700 1 librados: calling monclient init 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aed02700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4a8117350 con 0x7fd4a80ff710 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aed02700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4a8118530 con 0x7fd4a8103250 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aed02700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4a8119730 con 0x7fd4a80fc910 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aca9e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd4a80fc910 0x7fd4a80f9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37956/0 (socket says 172.21.15.107:37956) 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aca9e700 1 -- 172.21.15.107:0/2564037768 learned_addr learned my addr 172.21.15.107:0/2564037768 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd4a80fc910 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd4a80ff710 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter ms_handle_connect 0x7fd4a8103250 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140612276 0 0) 0x7fd4a8119730 con 0x7fd4a80fc910 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4940033f0 con 0x7fd4a80fc910 2024-02-10T08:45:59.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59490168 0 0) 0x7fd4a8118530 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8119730 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215499602 0 0) 0x7fd4a8117350 con 0x7fd4a80ff710 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8118530 con 0x7fd4a80ff710 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452891463 0 0) 0x7fd4a8119730 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4a8117350 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4980018c0 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2934704120 0 0) 0x7fd4940033f0 con 0x7fd4a80fc910 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4a8119730 con 0x7fd4a80fc910 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4a0001d50 con 0x7fd4a80fc910 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099483045 0 0) 0x7fd4a8118530 con 0x7fd4a80ff710 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4940033f0 con 0x7fd4a80ff710 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd490002050 con 0x7fd4a80ff710 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3051113274 0 0) 0x7fd4a8117350 con 0x7fd4a8103250 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 >> v1:172.21.15.107:6790/0 conn(0x7fd4a80fc910 legacy=0x7fd4a80f9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 >> v1:172.21.15.107:6789/0 conn(0x7fd4a80ff710 legacy=0x7fd4a80fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:45:59.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a811a930 con 0x7fd4a8103250 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.750+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a8117540 con 0x7fd4a8103250 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd498001f80 con 0x7fd4a8103250 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4980033b0 con 0x7fd4a8103250 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4aed02700 10 client.48751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a81179b0 con 0x7fd4a8103250 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4aed02700 1 librados: init done 2024-02-10T08:45:59.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.751+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd4a805f7a0 con 0x7fd4a8103250 2024-02-10T08:45:59.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd498001ae0 con 0x7fd4a8103250 2024-02-10T08:45:59.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(613..613 src has 420..613) v4 ==== 8617+0+0 (unknown 1828248053 0 0) 0x7fd4980767f0 con 0x7fd4a8103250 2024-02-10T08:45:59.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 osd_map(613..613 src has 420..613) v4 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 3 client.48751.objecter handle_osd_map got epochs [613,613] > 0 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 3 client.48751.objecter handle_osd_map decoding full epoch 613 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 20 client.48751.objecter dump_active .. 0 homeless 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v613) v1 ==== 54+0+16 (unknown 666779781 0 2420000192) 0x7fd498078be0 con 0x7fd4a8103250 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.752+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd494063d40 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 10 client.48751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 1 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 1 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 0 1 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 4 2024-02-10T08:45:59.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 10 client.48751.objecter _send_command 1 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a81183b0 con 0x7fd4a8118e90 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a8118e90 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd490001950 con 0x7fd4a8118e90 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 1 = system:0 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 4 2024-02-10T08:45:59.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.753+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 0 1 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 10 client.48751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 2 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 2 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a80475e0 osd=0 3 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 0 2 2024-02-10T08:45:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 4 2024-02-10T08:45:59.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 10 client.48751.objecter _send_command 2 2024-02-10T08:45:59.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.769+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a8108e30 con 0x7fd4a8118e90 2024-02-10T08:45:59.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a8108e30 con 0x7fd4a8118e90 2024-02-10T08:45:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a80475e0 osd=0 4 2024-02-10T08:45:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 0 2 2024-02-10T08:45:59.771 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 10 client.48751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 1 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 4 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 10 client.48751.objecter _send_command 3 2024-02-10T08:45:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.770+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a81056d0 con 0x7fd4a811a4e0 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a811a4e0 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a81056d0 con 0x7fd4a811a4e0 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 3 = system:0 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 4 2024-02-10T08:45:59.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.771+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 1 3 2024-02-10T08:45:59.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 10 client.48751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 4 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 4 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a81194b0 osd=1 3 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 1 4 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 4 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 10 client.48751.objecter _send_command 4 2024-02-10T08:45:59.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.786+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a80f3900 con 0x7fd4a811a4e0 2024-02-10T08:45:59.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a80f3900 con 0x7fd4a811a4e0 2024-02-10T08:45:59.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.788 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:45:59.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a81194b0 osd=1 4 2024-02-10T08:45:59.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 1 4 2024-02-10T08:45:59.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 10 client.48751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 5 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 5 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 2 5 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 4 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 10 client.48751.objecter _send_command 5 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a80f3e80 con 0x7fd4a811d7b0 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.788+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a811d7b0 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.789+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a80f3e80 con 0x7fd4a811d7b0 2024-02-10T08:45:59.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.789+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:45:59.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.789+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 5 = system:0 2024-02-10T08:45:59.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.789+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 4 2024-02-10T08:45:59.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.789+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 2 5 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 10 client.48751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 6 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 6 2024-02-10T08:45:59.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a80f3d10 osd=2 3 2024-02-10T08:45:59.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 2 6 2024-02-10T08:45:59.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 4 2024-02-10T08:45:59.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 10 client.48751.objecter _send_command 6 2024-02-10T08:45:59.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.803+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a8125850 con 0x7fd4a811d7b0 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.804+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a8125850 con 0x7fd4a811d7b0 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.804+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.804+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.804+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a80f3d10 osd=2 4 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.804+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 2 6 2024-02-10T08:45:59.805 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 10 client.48751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 7 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 7 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 3 7 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 4 2024-02-10T08:45:59.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 10 client.48751.objecter _send_command 7 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a8059ea0 con 0x7fd4a8118a70 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a8118a70 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a8059ea0 con 0x7fd4a8118a70 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 7 = system:0 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 4 2024-02-10T08:45:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.805+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 3 7 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.819+0000 7fd4aed02700 10 client.48751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.819+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 8 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.819+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.819+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.819+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 8 2024-02-10T08:45:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a8118900 osd=3 3 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 3 8 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 4 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 10 client.48751.objecter _send_command 8 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.820+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a812a2d0 con 0x7fd4a8118a70 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a812a2d0 con 0x7fd4a8118a70 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a8118900 osd=3 4 2024-02-10T08:45:59.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 3 8 2024-02-10T08:45:59.822 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 10 client.48751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 9 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 20 client.48751.objecter _get_session osd=-1 returning homeless 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 10 client.48751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=614}) v3 -- 0x7fd4a8059430 con 0x7fd4a8103250 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4a8059730 con 0x7fd4a8103250 2024-02-10T08:45:59.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_get_version_reply(handle=1 version=613) v2 ==== 24+0+0 (unknown 1167144771 0 0) 0x7fd498049540 con 0x7fd4a8103250 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a6ffd700 10 client.48751.objecter _check_command_map_dne tid 9 current 613 map_dne_bound 613 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a6ffd700 10 client.48751.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.821+0000 7fd4a6ffd700 15 client.48751.objecter _session_command_op_remove -1 9 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 10 client.48751.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 10 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 10 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 5 10 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 4 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 10 client.48751.objecter _send_command 10 2024-02-10T08:45:59.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a812ef90 con 0x7fd4a812b450 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.822+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a812b450 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.823+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a812ef90 con 0x7fd4a812b450 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.823+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.823+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 10 = system:0 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.823+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 4 2024-02-10T08:45:59.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.823+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 5 10 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 10 client.48751.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 11 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 11 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a81263d0 osd=5 3 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 5 11 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 4 2024-02-10T08:45:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 10 client.48751.objecter _send_command 11 2024-02-10T08:45:59.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.838+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a812ef90 con 0x7fd4a812b450 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.839+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a812ef90 con 0x7fd4a812b450 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.839+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.839+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.839+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a81263d0 osd=5 4 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.839+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 5 11 2024-02-10T08:45:59.840 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 10 client.48751.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 12 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 12 2024-02-10T08:45:59.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 6 12 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 4 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 10 client.48751.objecter _send_command 12 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a8130210 con 0x7fd4a8130c90 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.840+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a8130c90 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.841+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a8130210 con 0x7fd4a8130c90 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.841+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.841+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 12 = system:0 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.841+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 4 2024-02-10T08:45:59.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.841+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 6 12 2024-02-10T08:45:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 10 client.48751.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 13 2024-02-10T08:45:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 13 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a812ef90 osd=6 3 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 6 13 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 4 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 10 client.48751.objecter _send_command 13 2024-02-10T08:45:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.855+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a8130210 con 0x7fd4a8130c90 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a8130210 con 0x7fd4a8130c90 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a812ef90 osd=6 4 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 6 13 2024-02-10T08:45:59.857 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 10 client.48751.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 14 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 14 2024-02-10T08:45:59.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 7 14 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 4 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 10 client.48751.objecter _send_command 14 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4a81379b0 con 0x7fd4a8133ea0 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.857+0000 7fd4a5ffb700 10 client.48751.objecter ms_handle_connect 0x7fd4a8133ea0 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.858+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4a81379b0 con 0x7fd4a8133ea0 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.858+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.858+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 14 = system:0 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.858+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 4 2024-02-10T08:45:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.858+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 7 14 2024-02-10T08:45:59.866 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 10 client.48751.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign -1 15 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 20 client.48751.objecter _get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_remove -1 15 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 20 client.48751.objecter get_session s=0x7fd4a8130210 osd=7 3 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 15 client.48751.objecter _session_command_op_assign 7 15 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 4 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 10 client.48751.objecter _send_command 15 2024-02-10T08:45:59.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.872+0000 7fd4aed02700 1 -- 172.21.15.107:0/2564037768 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd4a81379b0 con 0x7fd4a8133ea0 2024-02-10T08:45:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.874+0000 7fd4a5ffb700 1 -- 172.21.15.107:0/2564037768 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fd4a81379b0 con 0x7fd4a8133ea0 2024-02-10T08:45:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.874+0000 7fd4a5ffb700 10 client.48751.objecter ms_dispatch 0x7fd4a8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:45:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.874+0000 7fd4a5ffb700 10 client.48751.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.874+0000 7fd4a5ffb700 20 client.48751.objecter put_session s=0x7fd4a8130210 osd=7 4 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:45:59.874+0000 7fd4a5ffb700 15 client.48751.objecter _session_command_op_remove 7 15 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:45:59.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.876 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:45:59.876 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.375891+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.376844+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.414233+0000 osd.6 (osd.6) 252 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.414945+0000 osd.6 (osd.6) 253 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.479373+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:46:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:58.480948+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:46:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:59.238895+0000 mon.a (mon.0) 3064 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:59.244153+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:46:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:59.244228+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-10T08:46:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:59.517020+0000 mgr.y (mgr.14553) 525 : cluster [DBG] pgmap v922: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:45:59.753109+0000 mon.b (mon.1) 726 : audit [DBG] from='client.? v1:172.21.15.107:0/2564037768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.375891+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.376844+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.414233+0000 osd.6 (osd.6) 252 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.414945+0000 osd.6 (osd.6) 253 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.479373+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:58.480948+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:59.238895+0000 mon.a (mon.0) 3064 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:59.244153+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:59.244228+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:59.517020+0000 mgr.y (mgr.14553) 525 : cluster [DBG] pgmap v922: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:45:59.753109+0000 mon.b (mon.1) 726 : audit [DBG] from='client.? v1:172.21.15.107:0/2564037768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.375891+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.376844+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.414233+0000 osd.6 (osd.6) 252 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.414945+0000 osd.6 (osd.6) 253 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.479373+0000 osd.3 (osd.3) 271 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:58.480948+0000 osd.3 (osd.3) 272 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:59.238895+0000 mon.a (mon.0) 3064 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:00.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:59.244153+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-104"}]': finished 2024-02-10T08:46:00.544 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:59.244228+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-10T08:46:00.544 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:59.517020+0000 mgr.y (mgr.14553) 525 : cluster [DBG] pgmap v922: 194 pgs: 1 active+clean+remapped, 22 active+undersized+degraded, 59 active+undersized, 112 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:00.544 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:45:59.753109+0000 mon.b (mon.1) 726 : audit [DBG] from='client.? v1:172.21.15.107:0/2564037768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:46:01.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:46:01.468 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:01.468 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:59.657637+0000 osd.5 (osd.5) 253 : cluster [DBG] 6.a scrub starts 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:45:59.660544+0000 osd.5 (osd.5) 254 : cluster [DBG] 6.a scrub ok 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:00.264030+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:01.107870+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:01.117027+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-10T08:46:01.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:01.127590+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:01.129090+0000 mon.a (mon.0) 3070 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:59.657637+0000 osd.5 (osd.5) 253 : cluster [DBG] 6.a scrub starts 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:45:59.660544+0000 osd.5 (osd.5) 254 : cluster [DBG] 6.a scrub ok 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:00.264030+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:01.107870+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:01.117027+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:01.127590+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:01.129090+0000 mon.a (mon.0) 3070 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:59.657637+0000 osd.5 (osd.5) 253 : cluster [DBG] 6.a scrub starts 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:45:59.660544+0000 osd.5 (osd.5) 254 : cluster [DBG] 6.a scrub ok 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:00.264030+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-10T08:46:01.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:01.107870+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:01.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:01.117027+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-10T08:46:01.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:01.127590+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:01.129090+0000 mon.a (mon.0) 3070 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:01.696 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:01.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:01.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:01.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:01.828 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:02.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:46:02.255 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:02.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:02.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:02.387 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:02.394 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:00.429638+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:00.432058+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:00.875378+0000 osd.1 (osd.1) 288 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:00.878277+0000 osd.1 (osd.1) 289 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:01.517940+0000 mgr.y (mgr.14553) 526 : cluster [DBG] pgmap v925: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:01.680108+0000 mgr.y (mgr.14553) 527 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:02.115123+0000 mon.a (mon.0) 3071 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:02.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:02.115223+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-10T08:46:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:00.429638+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:46:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:00.432058+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:00.875378+0000 osd.1 (osd.1) 288 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:00.878277+0000 osd.1 (osd.1) 289 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:01.517940+0000 mgr.y (mgr.14553) 526 : cluster [DBG] pgmap v925: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:01.680108+0000 mgr.y (mgr.14553) 527 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:02.115123+0000 mon.a (mon.0) 3071 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:02.115223+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:00.429638+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:00.432058+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:00.875378+0000 osd.1 (osd.1) 288 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:00.878277+0000 osd.1 (osd.1) 289 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:01.517940+0000 mgr.y (mgr.14553) 526 : cluster [DBG] pgmap v925: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:01.680108+0000 mgr.y (mgr.14553) 527 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:02.115123+0000 mon.a (mon.0) 3071 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:02.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:02.115223+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-10T08:46:02.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.968+0000 7f2536063700 1 -- 172.21.15.107:0/2444231794 >> v1:172.21.15.107:6790/0 conn(0x7f25300fc910 legacy=0x7f25300fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 -- 172.21.15.107:0/2444231794 shutdown_connections 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 -- 172.21.15.107:0/2444231794 >> 172.21.15.107:0/2444231794 conn(0x7f25300f2510 msgr2=0x7f25300f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 -- 172.21.15.107:0/2444231794 shutdown_connections 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 -- 172.21.15.107:0/2444231794 wait complete. 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 librados: starting msgr at 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 librados: starting objecter 2024-02-10T08:46:02.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 Processor -- start 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 -- start start 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 librados: setting wanted keys 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.969+0000 7f2536063700 1 librados: calling monclient init 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f2536063700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25301196f0 con 0x7f2530103250 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f2536063700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f253011a8d0 con 0x7f25300ff710 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f2536063700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f253011bad0 con 0x7f25300fc910 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f25300fc910 0x7f2530060c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37964/0 (socket says 172.21.15.107:37964) 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252f7fe700 1 -- 172.21.15.107:0/2581904124 learned_addr learned my addr 172.21.15.107:0/2581904124 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter ms_handle_connect 0x7f2530103250 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter ms_handle_connect 0x7f25300fc910 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter ms_handle_connect 0x7f25300ff710 2024-02-10T08:46:02.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199868450 0 0) 0x7f253011a8d0 con 0x7f25300ff710 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25180033f0 con 0x7f25300ff710 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324391120 0 0) 0x7f25301196f0 con 0x7f2530103250 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f253011a8d0 con 0x7f2530103250 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062286564 0 0) 0x7f253011bad0 con 0x7f25300fc910 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25301196f0 con 0x7f25300fc910 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159023835 0 0) 0x7f25180033f0 con 0x7f25300ff710 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f253011bad0 con 0x7f25300ff710 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25240018c0 con 0x7f25300ff710 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505465869 0 0) 0x7f253011a8d0 con 0x7f2530103250 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25180033f0 con 0x7f2530103250 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2528001bc0 con 0x7f2530103250 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2884601629 0 0) 0x7f25301196f0 con 0x7f25300fc910 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f253011a8d0 con 0x7f25300fc910 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2520002ca0 con 0x7f25300fc910 2024-02-10T08:46:02.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3843095948 0 0) 0x7f253011bad0 con 0x7f25300ff710 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 >> v1:172.21.15.107:6790/0 conn(0x7f25300fc910 legacy=0x7f2530060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 >> v1:172.21.15.107:6789/0 conn(0x7f2530103250 legacy=0x7f2530117db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253011ccd0 con 0x7f25300ff710 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.970+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253011bcc0 con 0x7f25300ff710 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f2536063700 10 client.48757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f253011c130 con 0x7f25300ff710 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f2536063700 1 librados: init done 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2524002280 con 0x7f25300ff710 2024-02-10T08:46:02.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f25240032c0 con 0x7f25300ff710 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.971+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c004f20 con 0x7f25300ff710 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2524001de0 con 0x7f25300ff710 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(616..616 src has 420..616) v4 ==== 8630+0+0 (unknown 3659482407 0 0) 0x7f2524076730 con 0x7f25300ff710 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 10 client.48757.objecter ms_dispatch 0x7f2530001550 osd_map(616..616 src has 420..616) v4 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 3 client.48757.objecter handle_osd_map got epochs [616,616] > 0 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 3 client.48757.objecter handle_osd_map decoding full epoch 616 2024-02-10T08:46:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.972+0000 7f252cff9700 20 client.48757.objecter dump_active .. 0 homeless 2024-02-10T08:46:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.973+0000 7f252cff9700 10 client.48757.objecter ms_handle_connect 0x7f2518063ce0 2024-02-10T08:46:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:02.977+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f25240498f0 con 0x7f25300ff710 2024-02-10T08:46:03.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.203+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]} v 0) v1 -- 0x7f251c0059e0 con 0x7f25300ff710 2024-02-10T08:46:03.237 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:03.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:03.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:46:03.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:03.373 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:03.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.490+0000 7f868853d700 1 -- 172.21.15.107:0/1643979022 >> v1:172.21.15.107:6790/0 conn(0x7f86800f6830 legacy=0x7f86800f6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:03.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.490+0000 7f868853d700 1 -- 172.21.15.107:0/1643979022 shutdown_connections 2024-02-10T08:46:03.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.490+0000 7f868853d700 1 -- 172.21.15.107:0/1643979022 >> 172.21.15.107:0/1643979022 conn(0x7f86800f2510 msgr2=0x7f86800f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:03.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.490+0000 7f868853d700 1 -- 172.21.15.107:0/1643979022 shutdown_connections 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.490+0000 7f868853d700 1 -- 172.21.15.107:0/1643979022 wait complete. 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 librados: starting msgr at 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 librados: starting objecter 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 Processor -- start 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 -- start start 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 librados: setting wanted keys 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 librados: calling monclient init 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86801193f0 con 0x7f86800f6830 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f868011a5d0 con 0x7f86800f7320 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f868853d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f868011b7d0 con 0x7f8680106660 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f8686ada700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8680106660 0x7f8680117ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37986/0 (socket says 172.21.15.107:37986) 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86862d9700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86800f6830 0x7f86801027e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58408/0 (socket says 172.21.15.107:58408) 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f8686ada700 1 -- 172.21.15.107:0/3864907327 learned_addr learned my addr 172.21.15.107:0/3864907327 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter ms_handle_connect 0x7f8680106660 2024-02-10T08:46:03.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter ms_handle_connect 0x7f86800f6830 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter ms_handle_connect 0x7f86800f7320 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.491+0000 7f86777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926847033 0 0) 0x7f86801193f0 con 0x7f86800f6830 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866c0038e0 con 0x7f86800f6830 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645123288 0 0) 0x7f868011b7d0 con 0x7f8680106660 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86801193f0 con 0x7f8680106660 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512434315 0 0) 0x7f868011a5d0 con 0x7f86800f7320 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868011b7d0 con 0x7f86800f7320 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014118412 0 0) 0x7f86801193f0 con 0x7f8680106660 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f868011a5d0 con 0x7f8680106660 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8678001bc0 con 0x7f8680106660 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351241629 0 0) 0x7f866c0038e0 con 0x7f86800f6830 2024-02-10T08:46:03.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86801193f0 con 0x7f86800f6830 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425331303 0 0) 0x7f868011b7d0 con 0x7f86800f7320 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866c0038e0 con 0x7f86800f7320 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f867c001360 con 0x7f86800f6830 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86700018d0 con 0x7f86800f7320 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3484216863 0 0) 0x7f868011a5d0 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 >> v1:172.21.15.195:6789/0 conn(0x7f86800f7320 legacy=0x7f8680102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 >> v1:172.21.15.107:6789/0 conn(0x7f86800f6830 legacy=0x7f86801027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868011c9d0 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86801195e0 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f868853d700 10 client.54035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8680119ad0 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f868853d700 1 librados: init done 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8678002700 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.492+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8678005d80 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.493+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8664004f20 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8678002260 con 0x7f8680106660 2024-02-10T08:46:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(617..617 src has 420..617) v4 ==== 8630+0+0 (unknown 3308466139 0 0) 0x7f8678079220 con 0x7f8680106660 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 10 client.54035.objecter ms_dispatch 0x7f8680001550 osd_map(617..617 src has 420..617) v4 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 3 client.54035.objecter handle_osd_map got epochs [617,617] > 0 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 3 client.54035.objecter handle_osd_map decoding full epoch 617 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 20 client.54035.objecter dump_active .. 0 homeless 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.494+0000 7f86777fe700 10 client.54035.objecter ms_handle_connect 0x7f866c063e20 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.498+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f867804c400 con 0x7f8680106660 2024-02-10T08:46:03.524 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:01.401209+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:01.404251+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.116483+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.151792+0000 mon.b (mon.1) 728 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.152365+0000 mon.a (mon.0) 3074 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.204853+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.107:0/2581904124' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.525 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.205167+0000 mon.a (mon.0) 3075 : audit [INF] from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:03.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:03.618 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:03.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:03.653 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:01.401209+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:46:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:01.404251+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:46:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.116483+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-10T08:46:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.151792+0000 mon.b (mon.1) 728 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.152365+0000 mon.a (mon.0) 3074 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.204853+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.107:0/2581904124' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.205167+0000 mon.a (mon.0) 3075 : audit [INF] from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:01.401209+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:01.404251+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.116483+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.151792+0000 mon.b (mon.1) 728 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.152365+0000 mon.a (mon.0) 3074 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.204853+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.107:0/2581904124' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.205167+0000 mon.a (mon.0) 3075 : audit [INF] from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]: dispatch 2024-02-10T08:46:03.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:03.705+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f86640051d0 con 0x7f8680106660 2024-02-10T08:46:03.851 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:03.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:03.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:03.985 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:04.005 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:04.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:04.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:04.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.117+0000 7f86777fe700 1 -- 172.21.15.107:0/3864907327 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v618) v1 ==== 101+0+0 (unknown 2137933314 0 0) 0x7f867804a810 con 0x7f8680106660 2024-02-10T08:46:04.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.125+0000 7f868853d700 10 librados: watch_flush enter 2024-02-10T08:46:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.127+0000 7f868853d700 10 librados: watch_flush exit 2024-02-10T08:46:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.127+0000 7f868853d700 20 client.54035.objecter shutdown clearing up homeless session... 2024-02-10T08:46:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.127+0000 7f868853d700 10 client.54035.objecter successfully canceled tick 2024-02-10T08:46:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.127+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 >> v1:172.21.15.107:6800/2634842009 conn(0x7f866c063e20 legacy=0x7f866c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:04.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.127+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 >> v1:172.21.15.107:6790/0 conn(0x7f8680106660 legacy=0x7f8680117ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.129+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 shutdown_connections 2024-02-10T08:46:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.129+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 >> 172.21.15.107:0/3864907327 conn(0x7f86800f2510 msgr2=0x7f8680108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.129+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 shutdown_connections 2024-02-10T08:46:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.129+0000 7f868853d700 1 -- 172.21.15.107:0/3864907327 wait complete. 2024-02-10T08:46:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.129+0000 7f868853d700 1 librados: shutdown 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.118+0000 7f252cff9700 1 -- 172.21.15.107:0/2581904124 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]=0 set 2.1d pg_upmap mapping to [3,4,2] v618) v1 ==== 129+0+0 (unknown 2800592724 0 0) 0x7f2524047d00 con 0x7f25300ff710 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.122+0000 7f2536063700 10 librados: watch_flush enter 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 10 librados: watch_flush exit 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 20 client.48757.objecter shutdown clearing up homeless session... 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 10 client.48757.objecter successfully canceled tick 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2518063ce0 legacy=0x7f25180661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 >> v1:172.21.15.195:6789/0 conn(0x7f25300ff710 legacy=0x7f2530061350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 shutdown_connections 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.128+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 >> 172.21.15.107:0/2581904124 conn(0x7f25300f2510 msgr2=0x7f25301066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:04.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.130+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 shutdown_connections 2024-02-10T08:46:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.132+0000 7f2536063700 1 -- 172.21.15.107:0/2581904124 wait complete. 2024-02-10T08:46:04.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:04.139+0000 7f2536063700 1 librados: shutdown 2024-02-10T08:46:04.145 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:set 2.1d pg_upmap mapping to [3,4,2] 2024-02-10T08:46:04.165 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:04.168 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:04.169 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:04.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:04.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:04.296 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:04.369 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:04.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:02.383385+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:02.384901+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.518851+0000 mgr.y (mgr.14553) 528 : cluster [DBG] pgmap v928: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.706137+0000 mon.c (mon.2) 445 : audit [INF] from='client.? v1:172.21.15.107:0/3864907327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:03.706425+0000 mon.a (mon.0) 3076 : audit [INF] from='client.54035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:04.113520+0000 mon.a (mon.0) 3077 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:04.117176+0000 mon.a (mon.0) 3078 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:04.117262+0000 mon.a (mon.0) 3079 : audit [INF] from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]': finished 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:04.117320+0000 mon.a (mon.0) 3080 : audit [INF] from='client.54035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:04.117364+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:04.145836+0000 mon.b (mon.1) 730 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:04.154498+0000 mon.a (mon.0) 3082 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:02.383385+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:02.384901+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.518851+0000 mgr.y (mgr.14553) 528 : cluster [DBG] pgmap v928: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.706137+0000 mon.c (mon.2) 445 : audit [INF] from='client.? v1:172.21.15.107:0/3864907327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:03.706425+0000 mon.a (mon.0) 3076 : audit [INF] from='client.54035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:04.113520+0000 mon.a (mon.0) 3077 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:04.117176+0000 mon.a (mon.0) 3078 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:04.117262+0000 mon.a (mon.0) 3079 : audit [INF] from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]': finished 2024-02-10T08:46:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:04.117320+0000 mon.a (mon.0) 3080 : audit [INF] from='client.54035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:04.117364+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-10T08:46:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:04.145836+0000 mon.b (mon.1) 730 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:04.154498+0000 mon.a (mon.0) 3082 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:02.383385+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:02.384901+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.518851+0000 mgr.y (mgr.14553) 528 : cluster [DBG] pgmap v928: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 88/585 objects degraded (15.043%) 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.706137+0000 mon.c (mon.2) 445 : audit [INF] from='client.? v1:172.21.15.107:0/3864907327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:03.706425+0000 mon.a (mon.0) 3076 : audit [INF] from='client.54035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:04.113520+0000 mon.a (mon.0) 3077 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:04.117176+0000 mon.a (mon.0) 3078 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:04.117262+0000 mon.a (mon.0) 3079 : audit [INF] from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1d", "id": [3, 4, 2]}]': finished 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:04.117320+0000 mon.a (mon.0) 3080 : audit [INF] from='client.54035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:04.117364+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-10T08:46:04.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:04.145836+0000 mon.b (mon.1) 730 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.493 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:04.154498+0000 mon.a (mon.0) 3082 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:04.501 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:04.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:46:04.848 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:04.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:04.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:04.982 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:05.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:05.100 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:05.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:05.143 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:05.239 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:05.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:05.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:05.368 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:05.388 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:05.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:05.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:05.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.675487+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:46:05.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.679197+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:46:05.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.825988+0000 osd.1 (osd.1) 290 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:46:05.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:03.829732+0000 osd.1 (osd.1) 291 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:04.535269+0000 osd.3 (osd.3) 273 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:04.536311+0000 osd.3 (osd.3) 274 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:05.120885+0000 mon.a (mon.0) 3083 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.120952+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:05.135807+0000 mon.b (mon.1) 731 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:05.142374+0000 mon.a (mon.0) 3085 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.675487+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.679197+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.825988+0000 osd.1 (osd.1) 290 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:03.829732+0000 osd.1 (osd.1) 291 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:04.535269+0000 osd.3 (osd.3) 273 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:04.536311+0000 osd.3 (osd.3) 274 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:46:05.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:05.120885+0000 mon.a (mon.0) 3083 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:05.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.120952+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-10T08:46:05.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:05.135807+0000 mon.b (mon.1) 731 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:05.142374+0000 mon.a (mon.0) 3085 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.675487+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.679197+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.825988+0000 osd.1 (osd.1) 290 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:03.829732+0000 osd.1 (osd.1) 291 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:04.535269+0000 osd.3 (osd.3) 273 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:04.536311+0000 osd.3 (osd.3) 274 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:05.120885+0000 mon.a (mon.0) 3083 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.120952+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:05.135807+0000 mon.b (mon.1) 731 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:05.142374+0000 mon.a (mon.0) 3085 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]: dispatch 2024-02-10T08:46:05.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:05] "GET /metrics HTTP/1.1" 200 189766 "" "Prometheus/2.33.4" 2024-02-10T08:46:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.367376+0000 osd.2 (osd.2) 229 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:46:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.370993+0000 osd.2 (osd.2) 230 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:46:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.519450+0000 mgr.y (mgr.14553) 529 : cluster [DBG] pgmap v931: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88/585 objects degraded (15.043%) 2024-02-10T08:46:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.553470+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:46:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:05.554596+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:06.115932+0000 mon.a (mon.0) 3086 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:06.125604+0000 mon.a (mon.0) 3087 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]': finished 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:06.125714+0000 mon.a (mon.0) 3088 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:06.220989+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.367376+0000 osd.2 (osd.2) 229 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.370993+0000 osd.2 (osd.2) 230 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.519450+0000 mgr.y (mgr.14553) 529 : cluster [DBG] pgmap v931: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88/585 objects degraded (15.043%) 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.553470+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:05.554596+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:06.115932+0000 mon.a (mon.0) 3086 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:06.125604+0000 mon.a (mon.0) 3087 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]': finished 2024-02-10T08:46:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:06.125714+0000 mon.a (mon.0) 3088 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-10T08:46:06.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:06.220989+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.367376+0000 osd.2 (osd.2) 229 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.370993+0000 osd.2 (osd.2) 230 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.519450+0000 mgr.y (mgr.14553) 529 : cluster [DBG] pgmap v931: 194 pgs: 1 active+clean+remapped, 16 creating+peering, 16 unknown, 22 active+undersized+degraded, 43 active+undersized, 96 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88/585 objects degraded (15.043%) 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.553470+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:05.554596+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:06.115932+0000 mon.a (mon.0) 3086 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:06.125604+0000 mon.a (mon.0) 3087 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-106", "mode": "writeback"}]': finished 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:06.125714+0000 mon.a (mon.0) 3088 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-10T08:46:06.479 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:06.220989+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:06.599 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:06.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:06.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:06.720 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:06.720 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:06.733 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:06.752 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:06.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:06.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:06.885 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- 172.21.15.107:0/661435587 >> v1:172.21.15.107:6789/0 conn(0x7fd4e00fda50 legacy=0x7fd4e00fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- 172.21.15.107:0/661435587 shutdown_connections 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- 172.21.15.107:0/661435587 >> 172.21.15.107:0/661435587 conn(0x7fd4e00f3640 msgr2=0x7fd4e00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- 172.21.15.107:0/661435587 shutdown_connections 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- 172.21.15.107:0/661435587 wait complete. 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 librados: starting msgr at 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 librados: starting objecter 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 Processor -- start 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 -- start start 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 librados: setting wanted keys 2024-02-10T08:46:07.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.036+0000 7fd4e6967700 1 librados: calling monclient init 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4e6967700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4e0117200 con 0x7fd4e0104330 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4e6967700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4e01183a0 con 0x7fd4e01007f0 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4e6967700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd4e01195a0 con 0x7fd4e00fda50 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dffff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd4e00fda50 0x7fd4e00f9800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37994/0 (socket says 172.21.15.107:37994) 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dffff700 1 -- 172.21.15.107:0/676855250 learned_addr learned my addr 172.21.15.107:0/676855250 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fd4e00fda50 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fd4e0104330 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859807285 0 0) 0x7fd4e01195a0 con 0x7fd4e00fda50 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c80038e0 con 0x7fd4e00fda50 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088242547 0 0) 0x7fd4e0117200 con 0x7fd4e0104330 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e01195a0 con 0x7fd4e0104330 2024-02-10T08:46:07.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fd4e01007f0 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489937610 0 0) 0x7fd4c80038e0 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e0117200 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4d00012d0 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412087393 0 0) 0x7fd4e01195a0 con 0x7fd4e0104330 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4c80038e0 con 0x7fd4e0104330 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4d80019a0 con 0x7fd4e0104330 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825977637 0 0) 0x7fd4e01183a0 con 0x7fd4e01007f0 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.037+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e01195a0 con 0x7fd4e01007f0 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3936345299 0 0) 0x7fd4e0117200 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 >> v1:172.21.15.195:6789/0 conn(0x7fd4e01007f0 legacy=0x7fd4e00f9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 >> v1:172.21.15.107:6789/0 conn(0x7fd4e0104330 legacy=0x7fd4e01159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e011a7a0 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e01173f0 con 0x7fd4e00fda50 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4e6967700 10 client.54041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:07.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01178e0 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4e6967700 1 librados: init done 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd4d00010f0 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.038+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd4d0003800 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd4cc004f00 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd4d0003a40 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(620..620 src has 420..620) v4 ==== 8691+0+0 (unknown 2119243106 0 0) 0x7fd4d0079360 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 osd_map(620..620 src has 420..620) v4 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 3 client.54041.objecter handle_osd_map got epochs [620,620] > 0 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 3 client.54041.objecter handle_osd_map decoding full epoch 620 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 20 client.54041.objecter dump_active .. 0 homeless 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.039+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v620) v1 ==== 54+0+16 (unknown 2050900836 0 2420000192) 0x7fd4d007b750 con 0x7fd4e00fda50 2024-02-10T08:46:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4c8063c00 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 10 client.54041.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 1 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 1 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 0 1 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 4 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 10 client.54041.objecter _send_command 1 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc004950 con 0x7fd4cc004530 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc004530 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc004950 con 0x7fd4cc004530 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 1 = system:0 2024-02-10T08:46:07.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 4 2024-02-10T08:46:07.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.040+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 0 1 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 10 client.54041.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 2 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 2 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc0051a0 osd=0 3 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 0 2 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 4 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 10 client.54041.objecter _send_command 2 2024-02-10T08:46:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.055+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc004d30 con 0x7fd4cc004530 2024-02-10T08:46:07.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc004d30 con 0x7fd4cc004530 2024-02-10T08:46:07.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc0051a0 osd=0 4 2024-02-10T08:46:07.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 0 2 2024-02-10T08:46:07.058 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 10 client.54041.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 1 3 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 4 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 10 client.54041.objecter _send_command 3 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc00c000 con 0x7fd4cc004110 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.057+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc004110 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.058+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc00c000 con 0x7fd4cc004110 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.058+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.058+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 3 = system:0 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.058+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 4 2024-02-10T08:46:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.058+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 1 3 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 10 client.54041.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 4 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 4 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc003fa0 osd=1 3 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 1 4 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 4 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 10 client.54041.objecter _send_command 4 2024-02-10T08:46:07.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.072+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc003160 con 0x7fd4cc004110 2024-02-10T08:46:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc003160 con 0x7fd4cc004110 2024-02-10T08:46:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc003fa0 osd=1 4 2024-02-10T08:46:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 1 4 2024-02-10T08:46:07.075 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 10 client.54041.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 5 2024-02-10T08:46:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 5 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 2 5 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 4 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 10 client.54041.objecter _send_command 5 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc00de90 con 0x7fd4cc00d2f0 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.074+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc00d2f0 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.075+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc00de90 con 0x7fd4cc00d2f0 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.075+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.075+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 5 = system:0 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.075+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 4 2024-02-10T08:46:07.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.075+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 2 5 2024-02-10T08:46:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:07.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 10 client.54041.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 6 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 6 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc001b40 osd=2 3 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 2 6 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 4 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 10 client.54041.objecter _send_command 6 2024-02-10T08:46:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.089+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc0015b0 con 0x7fd4cc00d2f0 2024-02-10T08:46:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc0015b0 con 0x7fd4cc00d2f0 2024-02-10T08:46:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc001b40 osd=2 4 2024-02-10T08:46:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 2 6 2024-02-10T08:46:07.092 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 10 client.54041.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 7 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 7 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 3 7 2024-02-10T08:46:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 4 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 10 client.54041.objecter _send_command 7 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc015b90 con 0x7fd4cc011ed0 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.091+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc011ed0 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.092+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc015b90 con 0x7fd4cc011ed0 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.092+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.092+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 7 = system:0 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.092+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 4 2024-02-10T08:46:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.092+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 3 7 2024-02-10T08:46:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 10 client.54041.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 8 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 8 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc0019c0 osd=3 3 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 3 8 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 4 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 10 client.54041.objecter _send_command 8 2024-02-10T08:46:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.106+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc00e270 con 0x7fd4cc011ed0 2024-02-10T08:46:07.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc00e270 con 0x7fd4cc011ed0 2024-02-10T08:46:07.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc0019c0 osd=3 4 2024-02-10T08:46:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 3 8 2024-02-10T08:46:07.109 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 10 client.54041.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 9 2024-02-10T08:46:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter _get_session osd=-1 returning homeless 2024-02-10T08:46:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 10 client.54041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=621}) v3 -- 0x7fd4cc00e410 con 0x7fd4e00fda50 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4cc00c940 con 0x7fd4e00fda50 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=620) v2 ==== 24+0+0 (unknown 1048057242 0 0) 0x7fd4d004aa10 con 0x7fd4e00fda50 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4de7fc700 10 client.54041.objecter _check_command_map_dne tid 9 current 620 map_dne_bound 620 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4de7fc700 10 client.54041.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4de7fc700 15 client.54041.objecter _session_command_op_remove -1 9 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 10 client.54041.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 10 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 10 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 5 10 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 4 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 10 client.54041.objecter _send_command 10 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.108+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc00cf20 con 0x7fd4cc016240 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.109+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc016240 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.110+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc00cf20 con 0x7fd4cc016240 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.110+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.110+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 10 = system:0 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.110+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 4 2024-02-10T08:46:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.110+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 5 10 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 10 client.54041.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 11 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 11 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc00cdb0 osd=5 3 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 5 11 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 4 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 10 client.54041.objecter _send_command 11 2024-02-10T08:46:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.124+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc00cc30 con 0x7fd4cc016240 2024-02-10T08:46:07.126 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc00c940 con 0x7fd4cc016240 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc00cdb0 osd=5 4 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 5 11 2024-02-10T08:46:07.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 10 client.54041.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 12 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 12 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 6 12 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 4 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 10 client.54041.objecter _send_command 12 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.126+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc01f0d0 con 0x7fd4cc01b550 2024-02-10T08:46:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.127+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc01b550 2024-02-10T08:46:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.127+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc01f0d0 con 0x7fd4cc01b550 2024-02-10T08:46:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.127+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:46:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.127+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 12 = system:0 2024-02-10T08:46:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.128+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 4 2024-02-10T08:46:07.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.128+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 6 12 2024-02-10T08:46:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 10 client.54041.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 13 2024-02-10T08:46:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 13 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc00c940 osd=6 3 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 6 13 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 4 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 10 client.54041.objecter _send_command 13 2024-02-10T08:46:07.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.142+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc01f0d0 con 0x7fd4cc01b550 2024-02-10T08:46:07.144 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc01f0d0 con 0x7fd4cc01b550 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc00c940 osd=6 4 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 6 13 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 10 client.54041.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 14 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 14 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 7 14 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 4 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 10 client.54041.objecter _send_command 14 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc020320 con 0x7fd4cc020dd0 2024-02-10T08:46:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.144+0000 7fd4dd7fa700 10 client.54041.objecter ms_handle_connect 0x7fd4cc020dd0 2024-02-10T08:46:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.145+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd4cc020320 con 0x7fd4cc020dd0 2024-02-10T08:46:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.145+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:46:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.145+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 14 = system:0 2024-02-10T08:46:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.145+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 4 2024-02-10T08:46:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.145+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 7 14 2024-02-10T08:46:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 10 client.54041.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign -1 15 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 20 client.54041.objecter _get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_remove -1 15 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 20 client.54041.objecter get_session s=0x7fd4cc01f0d0 osd=7 3 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 15 client.54041.objecter _session_command_op_assign 7 15 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 4 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 10 client.54041.objecter _send_command 15 2024-02-10T08:46:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.159+0000 7fd4e6967700 1 -- 172.21.15.107:0/676855250 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4cc020320 con 0x7fd4cc020dd0 2024-02-10T08:46:07.161 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.161+0000 7fd4dd7fa700 1 -- 172.21.15.107:0/676855250 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fd4cc020320 con 0x7fd4cc020dd0 2024-02-10T08:46:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.161+0000 7fd4dd7fa700 10 client.54041.objecter ms_dispatch 0x7fd4e0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.161+0000 7fd4dd7fa700 10 client.54041.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.161+0000 7fd4dd7fa700 20 client.54041.objecter put_session s=0x7fd4cc01f0d0 osd=7 4 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:07.161+0000 7fd4dd7fa700 15 client.54041.objecter _session_command_op_remove 7 15 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:46:07.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:07.216 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:07.336 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:06.221255+0000 mgr.y (mgr.14553) 530 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:07.336 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:07.040180+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? v1:172.21.15.107:0/676855250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:07.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:46:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:06.221255+0000 mgr.y (mgr.14553) 530 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:07.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:07.040180+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? v1:172.21.15.107:0/676855250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:06.221255+0000 mgr.y (mgr.14553) 530 : audit [DBG] from='mon.? v1:172.21.15.195:0/967713239' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.0"}]: dispatch 2024-02-10T08:46:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:07.040180+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? v1:172.21.15.107:0/676855250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:07.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:07.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:07.617 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:07.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:46:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:46:08.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:46:08.041 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:08.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:08.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:08.172 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:08.321 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:08.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:08.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:08.453 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:06.673767+0000 osd.5 (osd.5) 257 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:06.677287+0000 osd.5 (osd.5) 258 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:07.492096+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:07.493167+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:07.520362+0000 mgr.y (mgr.14553) 531 : cluster [DBG] pgmap v933: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:06.673767+0000 osd.5 (osd.5) 257 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:06.677287+0000 osd.5 (osd.5) 258 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:07.492096+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:07.493167+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:46:08.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:07.520362+0000 mgr.y (mgr.14553) 531 : cluster [DBG] pgmap v933: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:06.673767+0000 osd.5 (osd.5) 257 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:46:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:06.677287+0000 osd.5 (osd.5) 258 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:46:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:07.492096+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:46:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:07.493167+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:46:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:07.520362+0000 mgr.y (mgr.14553) 531 : cluster [DBG] pgmap v933: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:09.216 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:09.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:09.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:09.401 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:09.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:09.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.284705+0000 mon.a (mon.0) 3089 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:46:09.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.284755+0000 mon.a (mon.0) 3090 : cluster [WRN] Health check update: Degraded data redundancy: 94/609 objects degraded (15.435%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:09.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.401032+0000 osd.2 (osd.2) 231 : cluster [DBG] 237.0 scrub starts 2024-02-10T08:46:09.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.402574+0000 osd.2 (osd.2) 232 : cluster [DBG] 237.0 scrub ok 2024-02-10T08:46:09.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.455881+0000 osd.3 (osd.3) 279 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:46:09.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.457266+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:46:09.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:09.535 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.284705+0000 mon.a (mon.0) 3089 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.284755+0000 mon.a (mon.0) 3090 : cluster [WRN] Health check update: Degraded data redundancy: 94/609 objects degraded (15.435%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.401032+0000 osd.2 (osd.2) 231 : cluster [DBG] 237.0 scrub starts 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.402574+0000 osd.2 (osd.2) 232 : cluster [DBG] 237.0 scrub ok 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.455881+0000 osd.3 (osd.3) 279 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.457266+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:46:09.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.284705+0000 mon.a (mon.0) 3089 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T08:46:09.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.284755+0000 mon.a (mon.0) 3090 : cluster [WRN] Health check update: Degraded data redundancy: 94/609 objects degraded (15.435%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:09.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.401032+0000 osd.2 (osd.2) 231 : cluster [DBG] 237.0 scrub starts 2024-02-10T08:46:09.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.402574+0000 osd.2 (osd.2) 232 : cluster [DBG] 237.0 scrub ok 2024-02-10T08:46:09.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.455881+0000 osd.3 (osd.3) 279 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:46:09.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.457266+0000 osd.3 (osd.3) 280 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:46:09.684 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:09.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:09.691 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:46:09.691 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:46:09.691 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:46:09.691 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:46:09.691 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:46:09.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:09.817 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:09.837 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:09.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:09.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.938+0000 7ff2f5475700 1 -- 172.21.15.107:0/787034382 >> v1:172.21.15.107:6789/0 conn(0x7ff2f0100840 legacy=0x7ff2f0102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.938+0000 7ff2f5475700 1 -- 172.21.15.107:0/787034382 shutdown_connections 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.938+0000 7ff2f5475700 1 -- 172.21.15.107:0/787034382 >> 172.21.15.107:0/787034382 conn(0x7ff2f006c090 msgr2=0x7ff2f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 -- 172.21.15.107:0/787034382 shutdown_connections 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 -- 172.21.15.107:0/787034382 wait complete. 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 librados: starting msgr at 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 librados: starting objecter 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 Processor -- start 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 -- start start 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 librados: setting wanted keys 2024-02-10T08:46:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.939+0000 7ff2f5475700 1 librados: calling monclient init 2024-02-10T08:46:09.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:09.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2f5475700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f01173e0 con 0x7ff2f0100840 2024-02-10T08:46:09.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2f5475700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f01185c0 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2f5475700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2f01197c0 con 0x7ff2f0104350 2024-02-10T08:46:09.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2ef7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff2f0104350 0x7ff2f0115aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36298/0 (socket says 172.21.15.107:36298) 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2ef7fe700 1 -- 172.21.15.107:0/501707245 learned_addr learned my addr 172.21.15.107:0/501707245 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter ms_handle_connect 0x7ff2f0100840 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter ms_handle_connect 0x7ff2f0104350 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter ms_handle_connect 0x7ff2f00fd9f0 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927065598 0 0) 0x7ff2f01173e0 con 0x7ff2f0100840 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d8003800 con 0x7ff2f0100840 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745753077 0 0) 0x7ff2f01197c0 con 0x7ff2f0104350 2024-02-10T08:46:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f01173e0 con 0x7ff2f0104350 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937342204 0 0) 0x7ff2f01185c0 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f01197c0 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369980285 0 0) 0x7ff2d8003800 con 0x7ff2f0100840 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2f01185c0 con 0x7ff2f0100840 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e4002e50 con 0x7ff2f0100840 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 849751508 0 0) 0x7ff2f01173e0 con 0x7ff2f0104350 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2d8003800 con 0x7ff2f0104350 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2967415509 0 0) 0x7ff2f01197c0 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2f01173e0 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e8001cd0 con 0x7ff2f0104350 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e0001620 con 0x7ff2f00fd9f0 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2883254088 0 0) 0x7ff2f01185c0 con 0x7ff2f0100840 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 >> v1:172.21.15.107:6790/0 conn(0x7ff2f0104350 legacy=0x7ff2f0115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 >> v1:172.21.15.195:6789/0 conn(0x7ff2f00fd9f0 legacy=0x7ff2f00f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f011a9c0 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.940+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f01199b0 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.941+0000 7ff2f5475700 10 client.28473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.941+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f0119e20 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.941+0000 7ff2f5475700 1 librados: init done 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.941+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff2e40026c0 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.941+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff2e40037c0 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc004f20 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff2e4003920 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(620..620 src has 420..620) v4 ==== 8691+0+0 (unknown 2119243106 0 0) 0x7ff2e4076a30 con 0x7ff2f0100840 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 10 client.28473.objecter ms_dispatch 0x7ff2f0001550 osd_map(620..620 src has 420..620) v4 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 3 client.28473.objecter handle_osd_map got epochs [620,620] > 0 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 3 client.28473.objecter handle_osd_map decoding full epoch 620 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.942+0000 7ff2d7fff700 20 client.28473.objecter dump_active .. 0 homeless 2024-02-10T08:46:09.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.943+0000 7ff2d7fff700 10 client.28473.objecter ms_handle_connect 0x7ff2d8063e00 2024-02-10T08:46:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:09.947+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff2e4049bd0 con 0x7ff2f0100840 2024-02-10T08:46:09.975 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:09.980 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:46:09.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:46:09.980 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-02-10T08:46:09.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd in 0 2024-02-10T08:46:10.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:46:10.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.128+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff2dc0051d0 con 0x7ff2f0100840 2024-02-10T08:46:10.282 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:10.300 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:10.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.425285+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.428185+0000 osd.7 (osd.7) 242 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.729181+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.732070+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.789305+0000 osd.1 (osd.1) 292 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:08.794754+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.429794+0000 osd.2 (osd.2) 233 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.433142+0000 osd.2 (osd.2) 234 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.489673+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.490551+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.521037+0000 mgr.y (mgr.14553) 532 : cluster [DBG] pgmap v934: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 1.2 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:10.129284+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:10.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:10.169077+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:10.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.306+0000 7ff2d7fff700 1 -- 172.21.15.107:0/501707245 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v621) v1 ==== 87+0+0 (unknown 2964195315 0 0) 0x7ff2e4048040 con 0x7ff2f0100840 2024-02-10T08:46:10.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.318+0000 7ff2f5475700 10 librados: watch_flush enter 2024-02-10T08:46:10.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.322+0000 7ff2f5475700 10 librados: watch_flush exit 2024-02-10T08:46:10.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.322+0000 7ff2f5475700 20 client.28473.objecter shutdown clearing up homeless session... 2024-02-10T08:46:10.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.322+0000 7ff2f5475700 10 client.28473.objecter successfully canceled tick 2024-02-10T08:46:10.323 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:10.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:10.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.328+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff2d8063e00 legacy=0x7ff2d80662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:10.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.328+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 >> v1:172.21.15.107:6789/0 conn(0x7ff2f0100840 legacy=0x7ff2f00fa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:10.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:46:10.333 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:10.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.335+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 shutdown_connections 2024-02-10T08:46:10.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.335+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 >> 172.21.15.107:0/501707245 conn(0x7ff2f006c090 msgr2=0x7ff2f0106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:10.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.336+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 shutdown_connections 2024-02-10T08:46:10.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.347+0000 7ff2f5475700 1 -- 172.21.15.107:0/501707245 wait complete. 2024-02-10T08:46:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:10.349+0000 7ff2f5475700 1 librados: shutdown 2024-02-10T08:46:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:46:10.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:10.583 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:10.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.425285+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:46:10.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.428185+0000 osd.7 (osd.7) 242 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:46:10.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.729181+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:46:10.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.732070+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:46:10.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.789305+0000 osd.1 (osd.1) 292 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:46:10.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:08.794754+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.429794+0000 osd.2 (osd.2) 233 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.433142+0000 osd.2 (osd.2) 234 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.489673+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.490551+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.521037+0000 mgr.y (mgr.14553) 532 : cluster [DBG] pgmap v934: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 1.2 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:10.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:10.129284+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:10.588 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:10.169077+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:10.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:10.657 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:10.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:10.717 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.425285+0000 osd.7 (osd.7) 241 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.428185+0000 osd.7 (osd.7) 242 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.729181+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.732070+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.789305+0000 osd.1 (osd.1) 292 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:08.794754+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.429794+0000 osd.2 (osd.2) 233 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:46:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.433142+0000 osd.2 (osd.2) 234 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:46:10.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.489673+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:46:10.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.490551+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:46:10.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.521037+0000 mgr.y (mgr.14553) 532 : cluster [DBG] pgmap v934: 194 pgs: 1 remapped+peering, 1 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 1.2 KiB/s wr, 2 op/s; 94/609 objects degraded (15.435%) 2024-02-10T08:46:10.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:10.129284+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:10.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:10.169077+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:11.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:11.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:11.349 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:11.402 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:11.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:11.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:11.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.423312+0000 osd.7 (osd.7) 243 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:09.425159+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:10.291584+0000 mon.a (mon.0) 3093 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:10.306880+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:10.306920+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.423312+0000 osd.7 (osd.7) 243 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:09.425159+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:10.291584+0000 mon.a (mon.0) 3093 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:10.306880+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:11.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:10.306920+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-10T08:46:11.533 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:11.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.423312+0000 osd.7 (osd.7) 243 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:46:11.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:09.425159+0000 osd.7 (osd.7) 244 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:46:11.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:10.291584+0000 mon.a (mon.0) 3093 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:11.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:10.306880+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? v1:172.21.15.107:0/501707245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:11.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:10.306920+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-10T08:46:11.782 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:11.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:11.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:11.913 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:11.932 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:11.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:11.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:12.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:12.065 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:12.138 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:12.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:12.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:12.265 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:46:12.265 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:46:12.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:3 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e117)", 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 561.35523108300004, 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003, 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907505+0000", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907503+0000", 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1326e-05 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907514+0000", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0261999999999999e-05 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907524+0000", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5769999999999999e-06 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.907531+0000", 2024-02-10T08:46:12.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046026000000000001 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912134+0000", 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131586 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912265+0000", 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156537 2024-02-10T08:46:12.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912422+0000", 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046267600000000003 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.912884+0000", 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097266900000000001 2024-02-10T08:46:12.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.913857+0000", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035933647999999999 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949791+0000", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0361e-05 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949851+0000", 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0658999999999998e-05 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:50.949892+0000", 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042386549000000003 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:5 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected e118)", 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 560.31711361099997, 2024-02-10T08:46:12.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006, 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 5 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945623+0000", 2024-02-10T08:46:12.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945621+0000", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3675e-05 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945634+0000", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1625e-05 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945656+0000", 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2959999999999999e-06 2024-02-10T08:46:12.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.945664+0000", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027177170000000001 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948382+0000", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9475000000000006e-05 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948472+0000", 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202852 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.948674+0000", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010601493 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.959276+0000", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075265439999999996 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:51.966802+0000", 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054018193999999999 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020821+0000", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5893999999999999e-05 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020857+0000", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9193999999999999e-05 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:52.020886+0000", 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075263087000000006 2024-02-10T08:46:12.277 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:787 5.9 5.84ada7c9 (undecoded) ondisk+write+known_if_redirected e121)", 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 557.23982284600004, 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001, 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 787 2024-02-10T08:46:12.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022913+0000", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022911+0000", 2024-02-10T08:46:12.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4569999999999994e-06 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022920+0000", 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3113e-05 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022933+0000", 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9019999999999999e-06 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.022941+0000", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011509515999999999 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034451+0000", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3540000000000006e-06 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.034460+0000", 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.018581161 2024-02-10T08:46:12.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053041+0000", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2311e-05 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:56.053064+0000", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301502950000001 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35279.0:10 57.1a 57:5f47bad3:test-rados-api-smithi107-112513-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e121)", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:46:12.282 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 557.229985167, 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002, 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35279", 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/2637540386", 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 10 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032751+0000", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032749+0000", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1640000000000004e-06 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032756+0000", 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.67e-06 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032765+0000", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6840000000000002e-06 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.032774+0000", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073123999999999997 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033505+0000", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4190000000000005e-05 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033599+0000", 2024-02-10T08:46:12.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016687399999999999 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,6", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.033766+0000", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01101619 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.044782+0000", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011076585 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.055859+0000", 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01553619 2024-02-10T08:46:12.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071395+0000", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8683999999999994e-05 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071474+0000", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7649999999999999e-05 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:36:55.071521+0000", 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038770152000000002 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.287 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:256 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 539.277750524, 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001, 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 256 2024-02-10T08:46:12.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984986+0000", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984983+0000", 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0989999999999999e-06 2024-02-10T08:46:12.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.984991+0000", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6600000000000007e-06 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985001+0000", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5549999999999997e-06 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985008+0000", 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2148999999999998e-05 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985060+0000", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023027400000000001 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.985291+0000", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033679382000000001 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.018970+0000", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9383999999999997e-05 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:46:12.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.019059+0000", 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061927570000000001 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025252+0000", 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6524000000000003e-05 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025309+0000", 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104486 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.025413+0000", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.040427469000000001 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:57 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 539.273443241, 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001, 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:46:12.293 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 57 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989293+0000", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989292+0000", 2024-02-10T08:46:12.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989298+0000", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5689999999999994e-06 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989307+0000", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8519999999999998e-06 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989314+0000", 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2061000000000003e-05 2024-02-10T08:46:12.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989356+0000", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8673999999999999e-05 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989435+0000", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010247 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989537+0000", 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000341224 2024-02-10T08:46:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.989878+0000", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017592000000000001 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:12.990054+0000", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026858147999999998 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016912+0000", 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4705000000000001e-05 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016947+0000", 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3204999999999997e-05 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.016990+0000", 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027697327000000001 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.298 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:258 83.b 83:d481a176:test-rados-api-smithi107-112490-12::foo:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e139)", 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 539.21467699300001, 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999, 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.299 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 258 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048059+0000", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048057+0000", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4479999999999999e-06 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048064+0000", 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0129000000000001e-05 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048074+0000", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3009999999999997e-06 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048079+0000", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9587e-05 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048109+0000", 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012509199999999999 2024-02-10T08:46:12.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.048234+0000", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010028115000000001 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058262+0000", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9953e-05 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.058322+0000", 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019508307999999999 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077830+0000", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2973999999999995e-05 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.077913+0000", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102942 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:13.078016+0000", 2024-02-10T08:46:12.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029956858999999999 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.26977.0:59 66.15 66:ae45314e:test-rados-api-smithi107-112525-12::foo:head [write 0~128] snapc 15=[15,14] ondisk+write+ordersnap+known_if_redirected e140)", 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 538.25968072499995, 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998, 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.26977", 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1465655171", 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 59 2024-02-10T08:46:12.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003056+0000", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003053+0000", 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1612e-05 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003064+0000", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4388e-05 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003079+0000", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0430000000000008e-06 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.003087+0000", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011723245 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014810+0000", 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4276000000000003e-05 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.014904+0000", 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002265094 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.017169+0000", 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057871249999999997 2024-02-10T08:46:12.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.022957+0000", 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060097370000000002 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.028966+0000", 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013099348 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042066+0000", 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7976000000000002e-05 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042124+0000", 2024-02-10T08:46:12.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4863000000000001e-05 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:14.042159+0000", 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039102962999999998 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.51281.0:2 92.18 92:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-10T08:46:12.311 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 534.320572685, 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998, 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.51281", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4152427963", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:46:12.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942164+0000", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942161+0000", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3089999999999996e-06 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942170+0000", 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3520000000000007e-06 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942178+0000", 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5319999999999997e-06 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.942184+0000", 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001344175 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943528+0000", 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5959999999999998e-06 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.943535+0000", 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038514623999999997 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982050+0000", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112311 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982162+0000", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014368000000000001 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.982306+0000", 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010079780000000001 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983314+0000", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036102999999999998 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.983675+0000", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001564005 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985239+0000", 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8597999999999998e-05 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985288+0000", 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7169999999999998e-05 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:37:17.985325+0000", 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043161224999999998 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47647.0:1 136.6s0 136:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e196)", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 482.22681934000002, 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997, 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47647", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/993932140", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035914+0000", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6919999999999997e-06 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035924+0000", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3081000000000001e-05 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035937+0000", 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0079999999999996e-06 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035944+0000", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1489000000000001e-05 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035995+0000", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2339999999999999e-06 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035999+0000", 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91614844299999998 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952147+0000", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116906 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952264+0000", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048505899999999998 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952749+0000", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069718000000000004 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:46:12.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.953446+0000", 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013142239999999999 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954760+0000", 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9851999999999995e-05 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954830+0000", 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997 2024-02-10T08:46:12.324 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53375.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.26064258299999, 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999, 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53375", 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4067602482", 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.325 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002090+0000", 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2568000000000001e-05 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002103+0000", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6772000000000001e-05 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002120+0000", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4579999999999999e-06 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002129+0000", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2737e-05 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002182+0000", 2024-02-10T08:46:12.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.963e-06 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002186+0000", 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.005764858 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.007951+0000", 2024-02-10T08:46:12.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014669699999999999 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008097+0000", 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045664099999999998 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:46:12.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008554+0000", 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038511399999999999 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008939+0000", 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023091299999999999 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009170+0000", 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3881999999999996e-05 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009234+0000", 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27867.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 331.25646155300001, 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001, 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.332 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27867", 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/10211293", 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:46:12.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006272+0000", 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2092e-05 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006284+0000", 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1758e-05 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006296+0000", 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5979999999999997e-06 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006304+0000", 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015616800000000001 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006460+0000", 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4299999999999999e-06 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006465+0000", 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0093833510000001 2024-02-10T08:46:12.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.015848+0000", 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017003600000000001 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016018+0000", 2024-02-10T08:46:12.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00054150900000000002 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016560+0000", 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013759289999999999 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:46:12.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.017936+0000", 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045079799999999998 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018386+0000", 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0971999999999997e-05 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018457+0000", 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1662 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94547991871360 gen 9] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:46:12.341 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 157.07773357900001, 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997, 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1662 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:46:12.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184999+0000", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8840000000000006e-06 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185009+0000", 2024-02-10T08:46:12.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3593e-05 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185023+0000", 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8629999999999996e-06 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0421999999999999e-05 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185079+0000", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185086+0000", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88982775599999997 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.074914+0000", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016209899999999999 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.075076+0000", 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005269442 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.080345+0000", 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7071 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e493)", 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:46:12.346 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 133.11597089, 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994, 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7071 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146764+0000", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.878e-06 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146771+0000", 2024-02-10T08:46:12.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0343000000000001e-05 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146781+0000", 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4099999999999999e-06 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146786+0000", 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5929000000000001e-05 2024-02-10T08:46:12.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146822+0000", 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1459999999999995e-05 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146914+0000", 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080156775999999999 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.227071+0000", 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7073 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e494)", 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 132.08893686499999, 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997, 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.351 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7073 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173794+0000", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2813e-05 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173817+0000", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0312e-05 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173848+0000", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1270999999999999e-05 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173859+0000", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4023999999999996e-05 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173923+0000", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.131e-06 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173930+0000", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079391889999999993 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.181869+0000", 2024-02-10T08:46:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014406799999999999 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182013+0000", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018625099999999999 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182199+0000", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017831909999999999 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.183983+0000", 2024-02-10T08:46:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037849075000000003 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.221832+0000", 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000174136 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222006+0000", 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6679000000000001e-05 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222103+0000", 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7074 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 132.02798090499999, 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998, 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7074 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999943 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234750+0000", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0859000000000001e-05 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234771+0000", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8118000000000001e-05 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234799+0000", 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234811+0000", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038360999999999999 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235195+0000", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839000000000001 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235323+0000", 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038646028999999998 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.273969+0000", 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:79 60.12 60:49b62e28:test-rados-api-smithi107-112490-78::bfb5db8bf32239a5b4437a6edddde56f73a76483:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e494)", 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.235934+0000", 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 132.02680196399999, 2024-02-10T08:46:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022917797, 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235934+0000", 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235934+0000", 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235933+0000", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3519999999999999e-05 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235946+0000", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.393e-05 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235960+0000", 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4320000000000004e-06 2024-02-10T08:46:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235967+0000", 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4527000000000001e-05 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.236031+0000", 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131526 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.236163+0000", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025101700000000003 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.236414+0000", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015920909999999999 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.238006+0000", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020681752000000001 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.258688+0000", 2024-02-10T08:46:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4993999999999995e-05 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.258753+0000", 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9563999999999997e-05 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.258852+0000", 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022917797 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7076 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 131.17994905099999, 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999, 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7076 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082784+0000", 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1815e-05 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082796+0000", 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4978e-05 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082811+0000", 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8789999999999992e-06 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082820+0000", 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6817000000000002e-05 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082877+0000", 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8079999999999994e-06 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082886+0000", 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013785067 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096671+0000", 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109211 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096780+0000", 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000188168 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096968+0000", 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01095494 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.107923+0000", 2024-02-10T08:46:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175246 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108098+0000", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011893200000000001 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108217+0000", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2880999999999995e-05 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108300+0000", 2024-02-10T08:46:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:92 60.12 60:49b62e28:test-rados-api-smithi107-112490-78::bfb5db8bf32239a5b4437a6edddde56f73a76483:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e495)", 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 131.15436638700001, 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002, 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108368+0000", 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3590000000000008e-06 2024-02-10T08:46:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108376+0000", 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9339999999999995e-06 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108385+0000", 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1519999999999997e-06 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108391+0000", 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0077999999999997e-05 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108442+0000", 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0480000000000001e-05 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108532+0000", 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018637699999999999 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108718+0000", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036315000000000002 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.109082+0000", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049288513999999999 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158370+0000", 2024-02-10T08:46:12.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4228999999999998e-05 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158434+0000", 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2422000000000004e-05 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158527+0000", 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7078 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e495)", 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 131.10453745199999, 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001, 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7078 2024-02-10T08:46:12.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:46:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158195+0000", 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9330999999999998e-05 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158214+0000", 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4609000000000001e-05 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158239+0000", 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9750000000000002e-06 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158249+0000", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013279000000000001 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158381+0000", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4767999999999998e-05 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158476+0000", 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022729646999999999 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181206+0000", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000387876 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181594+0000", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203425 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181797+0000", 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6589999999999999e-05 2024-02-10T08:46:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181844+0000", 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8802999999999997e-05 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181933+0000", 2024-02-10T08:46:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:46:12.386 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:46:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:12.388 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:12.388 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:12.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:12.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:12.517 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:10.414266+0000 osd.7 (osd.7) 245 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:46:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:10.491125+0000 osd.6 (osd.6) 256 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:46:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:10.495150+0000 osd.6 (osd.6) 257 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:46:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:11.522284+0000 mgr.y (mgr.14553) 533 : cluster [DBG] pgmap v936: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:11.682002+0000 mgr.y (mgr.14553) 534 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:12.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:10.414266+0000 osd.7 (osd.7) 245 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:46:12.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:10.491125+0000 osd.6 (osd.6) 256 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:46:12.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:10.495150+0000 osd.6 (osd.6) 257 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:11.522284+0000 mgr.y (mgr.14553) 533 : cluster [DBG] pgmap v936: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:11.682002+0000 mgr.y (mgr.14553) 534 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:10.414266+0000 osd.7 (osd.7) 245 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:10.491125+0000 osd.6 (osd.6) 256 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:10.495150+0000 osd.6 (osd.6) 257 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:11.522284+0000 mgr.y (mgr.14553) 533 : cluster [DBG] pgmap v936: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:12.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:11.682002+0000 mgr.y (mgr.14553) 534 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:12.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:46:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:12.302017+0000 mon.a (mon.0) 3096 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:46:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:12.302017+0000 mon.a (mon.0) 3096 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:46:13.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:12.302017+0000 mon.a (mon.0) 3096 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T08:46:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 -- 172.21.15.107:0/413411699 >> v1:172.21.15.107:6790/0 conn(0x7f76f40fda50 legacy=0x7f76f40fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 -- 172.21.15.107:0/413411699 shutdown_connections 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 -- 172.21.15.107:0/413411699 >> 172.21.15.107:0/413411699 conn(0x7f76f40f3640 msgr2=0x7f76f40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 -- 172.21.15.107:0/413411699 shutdown_connections 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 -- 172.21.15.107:0/413411699 wait complete. 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 librados: starting msgr at 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.872+0000 7f76f9c18700 1 librados: starting objecter 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 Processor -- start 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 -- start start 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 librados: setting wanted keys 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 librados: calling monclient init 2024-02-10T08:46:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76f4119550 con 0x7f76f4104330 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76f411a6f0 con 0x7f76f41007f0 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f9c18700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76f411b8d0 con 0x7f76f40fda50 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f37fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f76f40fda50 0x7f76f4102460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36314/0 (socket says 172.21.15.107:36314) 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f37fe700 1 -- 172.21.15.107:0/3184311473 learned_addr learned my addr 172.21.15.107:0/3184311473 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter ms_handle_connect 0x7f76f40fda50 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter ms_handle_connect 0x7f76f4104330 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:13.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718534557 0 0) 0x7f76f411b8d0 con 0x7f76f40fda50 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e0003030 con 0x7f76f40fda50 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter ms_handle_connect 0x7f76f41007f0 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937636039 0 0) 0x7f76f4119550 con 0x7f76f4104330 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.873+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f411b8d0 con 0x7f76f4104330 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206961872 0 0) 0x7f76f411a6f0 con 0x7f76f41007f0 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f4119550 con 0x7f76f41007f0 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106050440 0 0) 0x7f76e0003030 con 0x7f76f40fda50 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76f411a6f0 con 0x7f76f40fda50 2024-02-10T08:46:13.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2819643329 0 0) 0x7f76f411b8d0 con 0x7f76f4104330 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e0003030 con 0x7f76f4104330 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f76dc001610 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f76ec001a30 con 0x7f76f4104330 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1375658464 0 0) 0x7f76f411a6f0 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 >> v1:172.21.15.195:6789/0 conn(0x7f76f41007f0 legacy=0x7f76f4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 >> v1:172.21.15.107:6789/0 conn(0x7f76f4104330 legacy=0x7f76f4117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f411cad0 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f4119740 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f9c18700 10 client.54053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f4119c30 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f9c18700 1 librados: init done 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f76dc002340 con 0x7f76f40fda50 2024-02-10T08:46:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.874+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f76dc0038a0 con 0x7f76f40fda50 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.875+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d4004f20 con 0x7f76f40fda50 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.875+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f76dc003ae0 con 0x7f76f40fda50 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(621..621 src has 420..621) v4 ==== 8691+0+0 (unknown 3184681732 0 0) 0x7f76dc0793c0 con 0x7f76f40fda50 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 10 client.54053.objecter ms_dispatch 0x7f76f4001550 osd_map(621..621 src has 420..621) v4 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 3 client.54053.objecter handle_osd_map got epochs [621,621] > 0 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 3 client.54053.objecter handle_osd_map decoding full epoch 621 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 20 client.54053.objecter dump_active .. 0 homeless 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.876+0000 7f76f0ff9700 10 client.54053.objecter ms_handle_connect 0x7f76e0063bd0 2024-02-10T08:46:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:13.880+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f76dc04c620 con 0x7f76f40fda50 2024-02-10T08:46:13.966 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:13.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/833260479 >> v1:172.21.15.107:6790/0 conn(0x7f3bb80fc910 legacy=0x7f3bb80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/833260479 shutdown_connections 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/833260479 >> 172.21.15.107:0/833260479 conn(0x7f3bb80f2510 msgr2=0x7f3bb80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/833260479 shutdown_connections 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/833260479 wait complete. 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 librados: starting msgr at 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.062+0000 7f3bbe2b0700 1 librados: starting objecter 2024-02-10T08:46:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 Processor -- start 2024-02-10T08:46:14.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 -- start start 2024-02-10T08:46:14.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 librados: setting wanted keys 2024-02-10T08:46:14.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 librados: calling monclient init 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb8116320 con 0x7f3bb80ff710 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb8117520 con 0x7f3bb8103250 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bbe2b0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb8118720 con 0x7f3bb80fc910 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3bb80fc910 0x7f3bb80f8c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36326/0 (socket says 172.21.15.107:36326) 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb7fff700 1 -- 172.21.15.107:0/2472031771 learned_addr learned my addr 172.21.15.107:0/2472031771 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb77fe700 1 --1- 172.21.15.107:0/2472031771 >> v1:172.21.15.107:6789/0 conn(0x7f3bb80ff710 0x7f3bb80f9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46770/0 (socket says 172.21.15.107:46770) 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb5775700 10 client.?.objecter ms_handle_connect 0x7f3bb80fc910 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb5775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb5775700 10 client.?.objecter ms_handle_connect 0x7f3bb80ff710 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.063+0000 7f3bb5775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 10 client.?.objecter ms_handle_connect 0x7f3bb8103250 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506617841 0 0) 0x7f3bb8118720 con 0x7f3bb80fc910 2024-02-10T08:46:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba0003630 con 0x7f3bb80fc910 2024-02-10T08:46:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 -- 0x7f76d40051d0 con 0x7f76f40fda50 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639020921 0 0) 0x7f3bb8116320 con 0x7f3bb80ff710 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8118720 con 0x7f3bb80ff710 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359873309 0 0) 0x7f3bb8117520 con 0x7f3bb8103250 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8116320 con 0x7f3bb8103250 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194412615 0 0) 0x7f3bb8118720 con 0x7f3bb80ff710 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bb8117520 con 0x7f3bb80ff710 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3bac001960 con 0x7f3bb80ff710 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3081653432 0 0) 0x7f3ba0003630 con 0x7f3bb80fc910 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bb8118720 con 0x7f3bb80fc910 2024-02-10T08:46:14.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3ba8002e30 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2857179158 0 0) 0x7f3bb8118720 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 >> v1:172.21.15.195:6789/0 conn(0x7f3bb8103250 legacy=0x7f3bb81149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 >> v1:172.21.15.107:6789/0 conn(0x7f3bb80ff710 legacy=0x7f3bb80f9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb8119920 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb8116510 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bbe2b0700 10 client.54059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb8116980 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.064+0000 7f3bbe2b0700 1 librados: init done 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.065+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3ba8001aa0 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.065+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3ba80037c0 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.065+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3ba4004f00 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.066+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3ba8002050 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(621..621 src has 420..621) v4 ==== 8691+0+0 (unknown 3184681732 0 0) 0x7f3ba80791e0 con 0x7f3bb80fc910 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 osd_map(621..621 src has 420..621) v4 2024-02-10T08:46:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 3 client.54059.objecter handle_osd_map got epochs [621,621] > 0 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 3 client.54059.objecter handle_osd_map decoding full epoch 621 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 20 client.54059.objecter dump_active .. 0 homeless 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v621) v1 ==== 54+0+16 (unknown 1892939213 0 2420000192) 0x7f3ba807b630 con 0x7f3bb80fc910 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba0068070 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 1 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 1 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 0 1 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 4 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 1 2024-02-10T08:46:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba4004950 con 0x7f3ba4004530 2024-02-10T08:46:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.067+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba4004530 2024-02-10T08:46:14.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.068+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba4004950 con 0x7f3ba4004530 2024-02-10T08:46:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.068+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.068+0000 7f3bb5775700 10 client.54059.objecter _finish_command 1 = system:0 2024-02-10T08:46:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.068+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 4 2024-02-10T08:46:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.068+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 0 1 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 2 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 2 2024-02-10T08:46:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba40051a0 osd=0 3 2024-02-10T08:46:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 0 2 2024-02-10T08:46:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 4 2024-02-10T08:46:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 2 2024-02-10T08:46:14.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.084+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba4004d30 con 0x7f3ba4004530 2024-02-10T08:46:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.085+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba4004d30 con 0x7f3ba4004530 2024-02-10T08:46:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.085+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.085+0000 7f3bb5775700 10 client.54059.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.085+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba40051a0 osd=0 4 2024-02-10T08:46:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.085+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 0 2 2024-02-10T08:46:14.087 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 3 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 3 2024-02-10T08:46:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 1 3 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 4 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 3 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba400c000 con 0x7f3ba4004150 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba4004150 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba400c000 con 0x7f3ba4004150 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 10 client.54059.objecter _finish_command 3 = system:0 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 4 2024-02-10T08:46:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.086+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 1 3 2024-02-10T08:46:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 4 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 4 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003fe0 osd=1 3 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 1 4 2024-02-10T08:46:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 4 2024-02-10T08:46:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 4 2024-02-10T08:46:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.102+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba4001850 con 0x7f3ba4004150 2024-02-10T08:46:14.104 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba4001850 con 0x7f3ba4004150 2024-02-10T08:46:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bb5775700 10 client.54059.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003fe0 osd=1 4 2024-02-10T08:46:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 1 4 2024-02-10T08:46:14.105 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 5 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 5 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 2 5 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 4 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 5 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.104+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba4003e70 con 0x7f3ba400d2f0 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba400d2f0 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba4003e70 con 0x7f3ba400d2f0 2024-02-10T08:46:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 10 client.54059.objecter _finish_command 5 = system:0 2024-02-10T08:46:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 4 2024-02-10T08:46:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.105+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 2 5 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 6 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 6 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003630 osd=2 3 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 2 6 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 4 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 6 2024-02-10T08:46:14.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.121+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba400e6e0 con 0x7f3ba400d2f0 2024-02-10T08:46:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba400e6e0 con 0x7f3ba400d2f0 2024-02-10T08:46:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 10 client.54059.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003630 osd=2 4 2024-02-10T08:46:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 2 6 2024-02-10T08:46:14.124 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 7 2024-02-10T08:46:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 7 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 3 7 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 4 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 7 2024-02-10T08:46:14.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba4015b40 con 0x7f3ba4011e80 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.123+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba4011e80 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.124+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba4015b40 con 0x7f3ba4011e80 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.124+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.124+0000 7f3bb5775700 10 client.54059.objecter _finish_command 7 = system:0 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.124+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 4 2024-02-10T08:46:14.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.124+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 3 7 2024-02-10T08:46:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:14.129 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:14.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 8 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 8 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba4003440 osd=3 3 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 3 8 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 4 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 8 2024-02-10T08:46:14.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.139+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba400e150 con 0x7f3ba4011e80 2024-02-10T08:46:14.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba400e150 con 0x7f3ba4011e80 2024-02-10T08:46:14.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 10 client.54059.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba4003440 osd=3 4 2024-02-10T08:46:14.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 3 8 2024-02-10T08:46:14.142 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 9 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter _get_session osd=-1 returning homeless 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 10 client.54059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=622}) v3 -- 0x7f3ba400df10 con 0x7f3bb80fc910 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ba400c940 con 0x7f3bb80fc910 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_get_version_reply(handle=1 version=621) v2 ==== 24+0+0 (unknown 3430160484 0 0) 0x7f3ba804c010 con 0x7f3bb80fc910 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb67fc700 10 client.54059.objecter _check_command_map_dne tid 9 current 621 map_dne_bound 621 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb67fc700 10 client.54059.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bb67fc700 15 client.54059.objecter _session_command_op_remove -1 9 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 10 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 10 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 5 10 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 4 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 10 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.141+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba400cf20 con 0x7f3ba40161f0 2024-02-10T08:46:14.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.142+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba40161f0 2024-02-10T08:46:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.143+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba400cf20 con 0x7f3ba40161f0 2024-02-10T08:46:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.143+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:46:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.143+0000 7f3bb5775700 10 client.54059.objecter _finish_command 10 = system:0 2024-02-10T08:46:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.143+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 4 2024-02-10T08:46:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.143+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 5 10 2024-02-10T08:46:14.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 11 2024-02-10T08:46:14.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 11 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba400cdb0 osd=5 3 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 5 11 2024-02-10T08:46:14.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 4 2024-02-10T08:46:14.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 11 2024-02-10T08:46:14.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.158+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba400d110 con 0x7f3ba40161f0 2024-02-10T08:46:14.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.159+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba400d110 con 0x7f3ba40161f0 2024-02-10T08:46:14.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.159+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.159+0000 7f3bb5775700 10 client.54059.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.159+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba400cdb0 osd=5 4 2024-02-10T08:46:14.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.159+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 5 11 2024-02-10T08:46:14.162 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:14.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 12 2024-02-10T08:46:14.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 12 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 6 12 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 4 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 12 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba401f0e0 con 0x7f3ba401b500 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.160+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba401b500 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.161+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba401f0e0 con 0x7f3ba401b500 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.161+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.161+0000 7f3bb5775700 10 client.54059.objecter _finish_command 12 = system:0 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.161+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 4 2024-02-10T08:46:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.161+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 6 12 2024-02-10T08:46:14.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 13 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 13 2024-02-10T08:46:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba400d110 osd=6 3 2024-02-10T08:46:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 6 13 2024-02-10T08:46:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 4 2024-02-10T08:46:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 13 2024-02-10T08:46:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.175+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba401f0e0 con 0x7f3ba401b500 2024-02-10T08:46:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba401f0e0 con 0x7f3ba401b500 2024-02-10T08:46:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bb5775700 10 client.54059.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba400d110 osd=6 4 2024-02-10T08:46:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 6 13 2024-02-10T08:46:14.179 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:14.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:14.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 14 2024-02-10T08:46:14.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 14 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 7 14 2024-02-10T08:46:14.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 4 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 14 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.177+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ba4020300 con 0x7f3ba4020de0 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 10 client.54059.objecter ms_handle_connect 0x7f3ba4020de0 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3ba4020300 con 0x7f3ba4020de0 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:46:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 10 client.54059.objecter _finish_command 14 = system:0 2024-02-10T08:46:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 4 2024-02-10T08:46:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.178+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 7 14 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 10 client.54059.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign -1 15 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 20 client.54059.objecter _get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_remove -1 15 2024-02-10T08:46:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 20 client.54059.objecter get_session s=0x7f3ba401f0e0 osd=7 3 2024-02-10T08:46:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 15 client.54059.objecter _session_command_op_assign 7 15 2024-02-10T08:46:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 4 2024-02-10T08:46:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 10 client.54059.objecter _send_command 15 2024-02-10T08:46:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.193+0000 7f3bbe2b0700 1 -- 172.21.15.107:0/2472031771 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3ba4020300 con 0x7f3ba4020de0 2024-02-10T08:46:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.195+0000 7f3bb5775700 1 -- 172.21.15.107:0/2472031771 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3ba4020300 con 0x7f3ba4020de0 2024-02-10T08:46:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.195+0000 7f3bb5775700 10 client.54059.objecter ms_dispatch 0x7f3bb8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.195+0000 7f3bb5775700 10 client.54059.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.195+0000 7f3bb5775700 20 client.54059.objecter put_session s=0x7f3ba401f0e0 osd=7 4 2024-02-10T08:46:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.195+0000 7f3bb5775700 15 client.54059.objecter _session_command_op_remove 7 15 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:46:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:46:14.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:46:14.064+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e621 definitely_dead 0 2024-02-10T08:46:14.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:14.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.327+0000 7f76f0ff9700 1 -- 172.21.15.107:0/3184311473 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["0"]}]=0 marked in osd.0. v622) v1 ==== 85+0+0 (unknown 655153566 0 0) 0x7f76dc04aa90 con 0x7f76f40fda50 2024-02-10T08:46:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.335+0000 7f76f9c18700 10 librados: watch_flush enter 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 10 librados: watch_flush exit 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 20 client.54053.objecter shutdown clearing up homeless session... 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 10 client.54053.objecter successfully canceled tick 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 >> v1:172.21.15.107:6800/2634842009 conn(0x7f76e0063bd0 legacy=0x7f76e0066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 >> v1:172.21.15.107:6790/0 conn(0x7f76f40fda50 legacy=0x7f76f4102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 shutdown_connections 2024-02-10T08:46:14.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.345+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 >> 172.21.15.107:0/3184311473 conn(0x7f76f40f3640 msgr2=0x7f76f41077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:14.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.346+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 shutdown_connections 2024-02-10T08:46:14.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.346+0000 7f76f9c18700 1 -- 172.21.15.107:0/3184311473 wait complete. 2024-02-10T08:46:14.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:14.346+0000 7f76f9c18700 1 librados: shutdown 2024-02-10T08:46:14.349 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.0. 2024-02-10T08:46:14.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:46:14.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:13.523080+0000 mgr.y (mgr.14553) 535 : cluster [DBG] pgmap v937: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:14.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:14.064940+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? v1:172.21.15.107:0/3184311473' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:14.065362+0000 mon.a (mon.0) 3097 : audit [DBG] from='client.54053 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:14.067505+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.107:0/2472031771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:14.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:13.523080+0000 mgr.y (mgr.14553) 535 : cluster [DBG] pgmap v937: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:14.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:14.064940+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? v1:172.21.15.107:0/3184311473' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:14.065362+0000 mon.a (mon.0) 3097 : audit [DBG] from='client.54053 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:14.067505+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.107:0/2472031771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:14.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:13.523080+0000 mgr.y (mgr.14553) 535 : cluster [DBG] pgmap v937: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:14.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:14.064940+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? v1:172.21.15.107:0/3184311473' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:14.065362+0000 mon.a (mon.0) 3097 : audit [DBG] from='client.54053 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-10T08:46:14.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:14.067505+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.107:0/2472031771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:15.160 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-02-10T08:46:15.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:14.316654+0000 mon.a (mon.0) 3098 : audit [INF] from='client.54053 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-10T08:46:15.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:14.316731+0000 mon.a (mon.0) 3099 : cluster [DBG] osdmap e622: 8 total, 7 up, 8 in 2024-02-10T08:46:15.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:14.316654+0000 mon.a (mon.0) 3098 : audit [INF] from='client.54053 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-10T08:46:15.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:14.316731+0000 mon.a (mon.0) 3099 : cluster [DBG] osdmap e622: 8 total, 7 up, 8 in 2024-02-10T08:46:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:46:15.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:46:15.716 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:15.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:14.316654+0000 mon.a (mon.0) 3098 : audit [INF] from='client.54053 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-10T08:46:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:14.316731+0000 mon.a (mon.0) 3099 : cluster [DBG] osdmap e622: 8 total, 7 up, 8 in 2024-02-10T08:46:15.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:15.847 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:15] "GET /metrics HTTP/1.1" 200 189927 "" "Prometheus/2.33.4" 2024-02-10T08:46:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:15.908 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:15.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:16.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:16.039 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:16.291 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:16.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:16.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:15.318166+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e623: 8 total, 7 up, 8 in 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:15.523723+0000 mgr.y (mgr.14553) 536 : cluster [DBG] pgmap v940: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:16.113589+0000 mon.a (mon.0) 3101 : cluster [WRN] Health check update: Degraded data redundancy: 94/615 objects degraded (15.285%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:16.320304+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e624: 8 total, 7 up, 8 in 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:15.318166+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e623: 8 total, 7 up, 8 in 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:15.523723+0000 mgr.y (mgr.14553) 536 : cluster [DBG] pgmap v940: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:16.113589+0000 mon.a (mon.0) 3101 : cluster [WRN] Health check update: Degraded data redundancy: 94/615 objects degraded (15.285%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:16.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:16.320304+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e624: 8 total, 7 up, 8 in 2024-02-10T08:46:16.555 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:16.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:16.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:16.686 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:15.318166+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e623: 8 total, 7 up, 8 in 2024-02-10T08:46:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:15.523723+0000 mgr.y (mgr.14553) 536 : cluster [DBG] pgmap v940: 194 pgs: 2 active+clean+remapped, 24 active+undersized+degraded, 52 active+undersized, 116 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 94/615 objects degraded (15.285%) 2024-02-10T08:46:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:16.113589+0000 mon.a (mon.0) 3101 : cluster [WRN] Health check update: Degraded data redundancy: 94/615 objects degraded (15.285%), 24 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:16.320304+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e624: 8 total, 7 up, 8 in 2024-02-10T08:46:17.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:46:17.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:16.318021+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 481'17 2024-02-10T08:46:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:16.318021+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 481'17 2024-02-10T08:46:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:16.318021+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 481'17 2024-02-10T08:46:18.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.519+0000 7f99902b8700 1 -- 172.21.15.107:0/923078332 >> v1:172.21.15.107:6790/0 conn(0x7f9988104350 legacy=0x7f99881067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 -- 172.21.15.107:0/923078332 shutdown_connections 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 -- 172.21.15.107:0/923078332 >> 172.21.15.107:0/923078332 conn(0x7f99880f2510 msgr2=0x7f99880f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 -- 172.21.15.107:0/923078332 shutdown_connections 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 -- 172.21.15.107:0/923078332 wait complete. 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 librados: starting msgr at 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 librados: starting objecter 2024-02-10T08:46:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 Processor -- start 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 -- start start 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 librados: setting wanted keys 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.520+0000 7f99902b8700 1 librados: calling monclient init 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f99902b8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99881174d0 con 0x7f99880f89c0 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f99902b8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99881186b0 con 0x7f9988104350 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f99902b8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99881198b0 con 0x7f99880f9570 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f998d853700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f99880f9570 0x7f99880ffed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36338/0 (socket says 172.21.15.107:36338) 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f998d853700 1 -- 172.21.15.107:0/1897601033 learned_addr learned my addr 172.21.15.107:0/1897601033 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter ms_handle_connect 0x7f99880f9570 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter ms_handle_connect 0x7f9988104350 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter ms_handle_connect 0x7f99880f89c0 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007246986 0 0) 0x7f99881198b0 con 0x7f99880f9570 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974003630 con 0x7f99880f9570 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306174582 0 0) 0x7f99881186b0 con 0x7f9988104350 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99881198b0 con 0x7f9988104350 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837562026 0 0) 0x7f99881174d0 con 0x7f99880f89c0 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99881186b0 con 0x7f99880f89c0 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2643349544 0 0) 0x7f99881198b0 con 0x7f9988104350 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99881174d0 con 0x7f9988104350 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239941218 0 0) 0x7f9974003630 con 0x7f99880f9570 2024-02-10T08:46:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99881198b0 con 0x7f99880f9570 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99800016f0 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f99780018c0 con 0x7f99880f9570 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434494935 0 0) 0x7f99881186b0 con 0x7f99880f89c0 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9974003630 con 0x7f99880f89c0 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.521+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9984001a10 con 0x7f99880f89c0 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 828868312 0 0) 0x7f99881174d0 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 >> v1:172.21.15.107:6790/0 conn(0x7f99880f9570 legacy=0x7f99880ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 >> v1:172.21.15.107:6789/0 conn(0x7f99880f89c0 legacy=0x7f99880ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998811aab0 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99881188a0 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f99902b8700 10 client.48790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9988118d10 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f99902b8700 1 librados: init done 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9980001960 con 0x7f9988104350 2024-02-10T08:46:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.522+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9980003b60 con 0x7f9988104350 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c004f20 con 0x7f9988104350 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9980004db0 con 0x7f9988104350 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(626..626 src has 420..626) v4 ==== 9227+0+0 (unknown 3585650688 0 0) 0x7f9980076f00 con 0x7f9988104350 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f997f779700 10 client.48790.objecter ms_dispatch 0x7f9988001550 osd_map(626..626 src has 420..626) v4 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f997f779700 3 client.48790.objecter handle_osd_map got epochs [626,626] > 0 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.523+0000 7f997f779700 3 client.48790.objecter handle_osd_map decoding full epoch 626 2024-02-10T08:46:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.524+0000 7f997f779700 20 client.48790.objecter dump_active .. 0 homeless 2024-02-10T08:46:18.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.524+0000 7f997f779700 10 client.48790.objecter ms_handle_connect 0x7f9974063cf0 2024-02-10T08:46:18.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.528+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f998004a010 con 0x7f9988104350 2024-02-10T08:46:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:18.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:18.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:17.358377+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e625: 8 total, 7 up, 8 in 2024-02-10T08:46:18.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:17.524297+0000 mgr.y (mgr.14553) 537 : cluster [DBG] pgmap v943: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:18.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:17.358377+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e625: 8 total, 7 up, 8 in 2024-02-10T08:46:18.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:17.524297+0000 mgr.y (mgr.14553) 537 : cluster [DBG] pgmap v943: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:18.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:18.720+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f996c0051d0 con 0x7f9988104350 2024-02-10T08:46:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:17.358377+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e625: 8 total, 7 up, 8 in 2024-02-10T08:46:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:17.524297+0000 mgr.y (mgr.14553) 537 : cluster [DBG] pgmap v943: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:18.801 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:18.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:18.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:18.933 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:19.058 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:19.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:19.060 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:19.061 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:19.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:19.189 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:19.311 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:19.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.367+0000 7f997f779700 1 -- 172.21.15.107:0/1897601033 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v627) v1 ==== 97+0+0 (unknown 3461038 0 0) 0x7f9980001550 con 0x7f9988104350 2024-02-10T08:46:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 10 librados: watch_flush enter 2024-02-10T08:46:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 10 librados: watch_flush exit 2024-02-10T08:46:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 20 client.48790.objecter shutdown clearing up homeless session... 2024-02-10T08:46:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 10 client.48790.objecter successfully canceled tick 2024-02-10T08:46:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9974063cf0 legacy=0x7f99740661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.376+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 >> v1:172.21.15.195:6789/0 conn(0x7f9988104350 legacy=0x7f9988115b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:19.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.377+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 shutdown_connections 2024-02-10T08:46:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.377+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 >> 172.21.15.107:0/1897601033 conn(0x7f99880f2510 msgr2=0x7f99881077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.377+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 shutdown_connections 2024-02-10T08:46:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.377+0000 7f99902b8700 1 -- 172.21.15.107:0/1897601033 wait complete. 2024-02-10T08:46:19.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:19.378+0000 7f99902b8700 1 librados: shutdown 2024-02-10T08:46:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:46:19.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:19.463 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:18.364118+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e626: 8 total, 7 up, 8 in 2024-02-10T08:46:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:18.721298+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.107:0/1897601033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:18.721731+0000 mon.a (mon.0) 3105 : audit [INF] from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:18.364118+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e626: 8 total, 7 up, 8 in 2024-02-10T08:46:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:18.721298+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.107:0/1897601033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:18.721731+0000 mon.a (mon.0) 3105 : audit [INF] from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:18.364118+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e626: 8 total, 7 up, 8 in 2024-02-10T08:46:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:18.721298+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.107:0/1897601033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:18.721731+0000 mon.a (mon.0) 3105 : audit [INF] from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:19.790 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:19.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:19.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:19.925 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:20.161 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-02-10T08:46:20.162 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:46:20.162 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-02-10T08:46:20.162 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-02-10T08:46:20.162 DEBUG:teuthology.orchestra.run.smithi195:osd.4> sudo journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T08:46:20.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl start ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4 2024-02-10T08:46:20.194 INFO:journalctl@ceph.osd.4.smithi195.stdout:-- Logs begin at Sat 2024-02-10 08:16:01 UTC. -- 2024-02-10T08:46:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:19.361436+0000 mon.a (mon.0) 3106 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:19.366100+0000 mon.a (mon.0) 3107 : audit [INF] from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:19.366165+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e627: 8 total, 7 up, 8 in 2024-02-10T08:46:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:19.524908+0000 mgr.y (mgr.14553) 538 : cluster [DBG] pgmap v946: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:20.538 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:20.538 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:20 smithi195 systemd[1]: Starting Ceph osd.4 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:46:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:19.361436+0000 mon.a (mon.0) 3106 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:19.366100+0000 mon.a (mon.0) 3107 : audit [INF] from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:19.366165+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e627: 8 total, 7 up, 8 in 2024-02-10T08:46:20.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:19.524908+0000 mgr.y (mgr.14553) 538 : cluster [DBG] pgmap v946: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:19.361436+0000 mon.a (mon.0) 3106 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:19.366100+0000 mon.a (mon.0) 3107 : audit [INF] from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:19.366165+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e627: 8 total, 7 up, 8 in 2024-02-10T08:46:20.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:19.524908+0000 mgr.y (mgr.14553) 538 : cluster [DBG] pgmap v946: 194 pgs: 18 remapped+peering, 2 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 69/576 objects degraded (11.979%); 232 B/s, 11 objects/s recovering 2024-02-10T08:46:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:21.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-4 --no-mon-config 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T08:46:21.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:21 smithi195 bash[137643]: --> ceph-volume lvm activate successful for osd ID: 4 2024-02-10T08:46:21.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:21.554 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:21.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:21.115362+0000 mon.a (mon.0) 3109 : cluster [WRN] Health check update: Degraded data redundancy: 69/576 objects degraded (11.979%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:21.555 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:21.115362+0000 mon.a (mon.0) 3109 : cluster [WRN] Health check update: Degraded data redundancy: 69/576 objects degraded (11.979%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:21.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:21.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:21.686 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:21.692 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:21.115362+0000 mon.a (mon.0) 3109 : cluster [WRN] Health check update: Degraded data redundancy: 69/576 objects degraded (11.979%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- 172.21.15.107:0/3691326480 >> v1:172.21.15.107:6789/0 conn(0x7f844c0fda50 legacy=0x7f844c0fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- 172.21.15.107:0/3691326480 shutdown_connections 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- 172.21.15.107:0/3691326480 >> 172.21.15.107:0/3691326480 conn(0x7f844c0f3640 msgr2=0x7f844c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- 172.21.15.107:0/3691326480 shutdown_connections 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- 172.21.15.107:0/3691326480 wait complete. 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 librados: starting msgr at 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 librados: starting objecter 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 Processor -- start 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 -- start start 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 librados: setting wanted keys 2024-02-10T08:46:21.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.778+0000 7f845191a700 1 librados: calling monclient init 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f845191a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c1172d0 con 0x7f844c1007f0 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f845191a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c1184b0 con 0x7f844c104330 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f845191a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f844c1196b0 con 0x7f844c0fda50 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f844affd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f844c0fda50 0x7f844c0f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38208/0 (socket says 172.21.15.107:38208) 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f844affd700 1 -- 172.21.15.107:0/2672882034 learned_addr learned my addr 172.21.15.107:0/2672882034 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f844a7fc700 1 --1- 172.21.15.107:0/2672882034 >> v1:172.21.15.107:6789/0 conn(0x7f844c1007f0 0x7f844c0fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50834/0 (socket says 172.21.15.107:50834) 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter ms_handle_connect 0x7f844c0fda50 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter ms_handle_connect 0x7f844c1007f0 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter ms_handle_connect 0x7f844c104330 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257845771 0 0) 0x7f844c1196b0 con 0x7f844c0fda50 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84380033f0 con 0x7f844c0fda50 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192011365 0 0) 0x7f844c1172d0 con 0x7f844c1007f0 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c1196b0 con 0x7f844c1007f0 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387338223 0 0) 0x7f844c1184b0 con 0x7f844c104330 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c1172d0 con 0x7f844c104330 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345693129 0 0) 0x7f844c1196b0 con 0x7f844c1007f0 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c1184b0 con 0x7f844c1007f0 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8434001a80 con 0x7f844c1007f0 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285315287 0 0) 0x7f84380033f0 con 0x7f844c0fda50 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c1196b0 con 0x7f844c0fda50 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2081417378 0 0) 0x7f844c1172d0 con 0x7f844c104330 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84380033f0 con 0x7f844c104330 2024-02-10T08:46:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f843c002b10 con 0x7f844c0fda50 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8444001bc0 con 0x7f844c104330 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 832025167 0 0) 0x7f844c1184b0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 >> v1:172.21.15.107:6790/0 conn(0x7f844c0fda50 legacy=0x7f844c0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 >> v1:172.21.15.195:6789/0 conn(0x7f844c104330 legacy=0x7f844c1159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c11a8b0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.779+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c1174c0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f845191a700 10 client.28494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1179b0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f845191a700 1 librados: init done 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8434004be0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f84340059b0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.780+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f842c004f00 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8434001f50 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(627..627 src has 420..627) v4 ==== 9227+0+0 (unknown 778631946 0 0) 0x7f8434078fe0 con 0x7f844c1007f0 2024-02-10T08:46:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 osd_map(627..627 src has 420..627) v4 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 3 client.28494.objecter handle_osd_map got epochs [627,627] > 0 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 3 client.28494.objecter handle_osd_map decoding full epoch 627 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 20 client.28494.objecter dump_active .. 0 homeless 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v627) v1 ==== 54+0+16 (unknown 3070795235 0 2420000192) 0x7f84340793c0 con 0x7f844c1007f0 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.781+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f8438063d70 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 10 client.28494.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 1 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 1 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 0 1 2024-02-10T08:46:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 4 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 10 client.28494.objecter _send_command 1 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c004950 con 0x7f842c004530 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c004530 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c004950 con 0x7f842c004530 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 10 client.28494.objecter _finish_command 1 = system:0 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 4 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.782+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 0 1 2024-02-10T08:46:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:21.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 10 client.28494.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 2 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 2 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0051a0 osd=0 3 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 0 2 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 4 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 10 client.28494.objecter _send_command 2 2024-02-10T08:46:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.797+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c004d30 con 0x7f842c004530 2024-02-10T08:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c004d30 con 0x7f842c004530 2024-02-10T08:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f8443fff700 10 client.28494.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0051a0 osd=0 4 2024-02-10T08:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 0 2 2024-02-10T08:46:21.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 10 client.28494.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 1 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 4 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 10 client.28494.objecter _send_command 3 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.799+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c00bf60 con 0x7f842c003b40 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c003b40 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c00bf60 con 0x7f842c003b40 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 10 client.28494.objecter _finish_command 3 = system:0 2024-02-10T08:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 4 2024-02-10T08:46:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.800+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 1 3 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 10 client.28494.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 4 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 4 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c004270 osd=1 3 2024-02-10T08:46:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 1 4 2024-02-10T08:46:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 4 2024-02-10T08:46:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 10 client.28494.objecter _send_command 4 2024-02-10T08:46:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.815+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c00d280 con 0x7f842c003b40 2024-02-10T08:46:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c00d280 con 0x7f842c003b40 2024-02-10T08:46:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f8443fff700 10 client.28494.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c004270 osd=1 4 2024-02-10T08:46:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 1 4 2024-02-10T08:46:21.818 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 10 client.28494.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 5 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 5 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 2 5 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 4 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 10 client.28494.objecter _send_command 5 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.816+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c010d30 con 0x7f842c002070 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c002070 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c010d30 con 0x7f842c002070 2024-02-10T08:46:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 10 client.28494.objecter _finish_command 5 = system:0 2024-02-10T08:46:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 4 2024-02-10T08:46:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.817+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 2 5 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 10 client.28494.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 6 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 6 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0033a0 osd=2 3 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 2 6 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 4 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 10 client.28494.objecter _send_command 6 2024-02-10T08:46:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.831+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c001630 con 0x7f842c002070 2024-02-10T08:46:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c001630 con 0x7f842c002070 2024-02-10T08:46:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 10 client.28494.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0033a0 osd=2 4 2024-02-10T08:46:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 2 6 2024-02-10T08:46:21.835 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 10 client.28494.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 7 2024-02-10T08:46:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 7 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 3 7 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 4 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 10 client.28494.objecter _send_command 7 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c00d050 con 0x7f842c010ed0 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.833+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c010ed0 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.834+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c00d050 con 0x7f842c010ed0 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.834+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.834+0000 7f8443fff700 10 client.28494.objecter _finish_command 7 = system:0 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.834+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 4 2024-02-10T08:46:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.834+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 3 7 2024-02-10T08:46:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 10 client.28494.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 8 2024-02-10T08:46:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 8 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c00c8d0 osd=3 3 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 3 8 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 4 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 10 client.28494.objecter _send_command 8 2024-02-10T08:46:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.848+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c012240 con 0x7f842c010ed0 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.849+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c012240 con 0x7f842c010ed0 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.849+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.849+0000 7f8443fff700 10 client.28494.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.849+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c00c8d0 osd=3 4 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.849+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 3 8 2024-02-10T08:46:21.851 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 10 client.28494.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 9 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter _get_session osd=-1 returning homeless 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 10 client.28494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=628}) v3 -- 0x7f842c011aa0 con 0x7f844c1007f0 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f842c011dc0 con 0x7f844c1007f0 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_get_version_reply(handle=1 version=627) v2 ==== 24+0+0 (unknown 1950866970 0 0) 0x7f843404a370 con 0x7f844c1007f0 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f84497fa700 10 client.28494.objecter _check_command_map_dne tid 9 current 627 map_dne_bound 627 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f84497fa700 10 client.28494.objecter _finish_command 9 = osd:6 osd down 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f84497fa700 15 client.28494.objecter _session_command_op_remove -1 9 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 10 client.28494.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 10 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 10 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 5 10 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 4 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 10 client.28494.objecter _send_command 10 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6804/4178321486 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c013110 con 0x7f842c012cb0 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.850+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c012cb0 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.851+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c013110 con 0x7f842c012cb0 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.851+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 10: 0 ) v1 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.851+0000 7f8443fff700 10 client.28494.objecter _finish_command 10 = system:0 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.851+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 4 2024-02-10T08:46:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.851+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 5 10 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 10 client.28494.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 11 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 11 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c012070 osd=5 3 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 5 11 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 4 2024-02-10T08:46:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 10 client.28494.objecter _send_command 11 2024-02-10T08:46:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.865+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c00c160 con 0x7f842c012cb0 2024-02-10T08:46:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c00c160 con 0x7f842c012cb0 2024-02-10T08:46:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f8443fff700 10 client.28494.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c012070 osd=5 4 2024-02-10T08:46:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 5 11 2024-02-10T08:46:21.869 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 10 client.28494.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 12 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 12 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 6 12 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 4 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 10 client.28494.objecter _send_command 12 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.867+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6808/1417479017 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c01efc0 con 0x7f842c01b4b0 2024-02-10T08:46:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.868+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c01b4b0 2024-02-10T08:46:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.869+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c01efc0 con 0x7f842c01b4b0 2024-02-10T08:46:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.869+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 12: 0 ) v1 2024-02-10T08:46:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.869+0000 7f8443fff700 10 client.28494.objecter _finish_command 12 = system:0 2024-02-10T08:46:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.869+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 4 2024-02-10T08:46:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.869+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 6 12 2024-02-10T08:46:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 10 client.28494.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 13 2024-02-10T08:46:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 13 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c00c160 osd=6 3 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 6 13 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 4 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 10 client.28494.objecter _send_command 13 2024-02-10T08:46:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.883+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c0233d0 con 0x7f842c01b4b0 2024-02-10T08:46:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.884+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c0233d0 con 0x7f842c01b4b0 2024-02-10T08:46:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f8443fff700 10 client.28494.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c00c160 osd=6 4 2024-02-10T08:46:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 6 13 2024-02-10T08:46:21.887 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 10 client.28494.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 14 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 14 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 7 14 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 4 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 10 client.28494.objecter _send_command 14 2024-02-10T08:46:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6812/1934176928 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c027790 con 0x7f842c023af0 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.885+0000 7f8443fff700 10 client.28494.objecter ms_handle_connect 0x7f842c023af0 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.886+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f842c027790 con 0x7f842c023af0 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.886+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 14: 0 ) v1 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.886+0000 7f8443fff700 10 client.28494.objecter _finish_command 14 = system:0 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.886+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 4 2024-02-10T08:46:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.886+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 7 14 2024-02-10T08:46:21.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 10 client.28494.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign -1 15 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 20 client.28494.objecter _get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 15 client.28494.objecter _session_command_op_remove -1 15 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 20 client.28494.objecter get_session s=0x7f842c0233d0 osd=7 3 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 15 client.28494.objecter _session_command_op_assign 7 15 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 4 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 10 client.28494.objecter _send_command 15 2024-02-10T08:46:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.900+0000 7f845191a700 1 -- 172.21.15.107:0/2672882034 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f842c027790 con 0x7f842c023af0 2024-02-10T08:46:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.902+0000 7f8443fff700 1 -- 172.21.15.107:0/2672882034 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f842c027790 con 0x7f842c023af0 2024-02-10T08:46:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.902+0000 7f8443fff700 10 client.28494.objecter ms_dispatch 0x7f844c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.902+0000 7f8443fff700 10 client.28494.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.902+0000 7f8443fff700 20 client.28494.objecter put_session s=0x7f842c0233d0 osd=7 4 2024-02-10T08:46:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:21.902+0000 7f8443fff700 15 client.28494.objecter _session_command_op_remove 7 15 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:46:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:21.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:22.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:46:22.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:22 smithi195 bash[137905]: b7593665db67b5e16e48c9f780ecaa49e2d2c15b86300dd0c650c559348015e2 2024-02-10T08:46:22.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:22 smithi195 systemd[1]: Started Ceph osd.4 for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:46:22.426 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:22.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:46:22.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:21.525913+0000 mgr.y (mgr.14553) 539 : cluster [DBG] pgmap v947: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 589 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:21.691968+0000 mgr.y (mgr.14553) 540 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:21.782600+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? v1:172.21.15.107:0/2672882034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:22.123771+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e628: 8 total, 7 up, 8 in 2024-02-10T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:22.170958+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:22.171299+0000 mon.a (mon.0) 3112 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:21.525913+0000 mgr.y (mgr.14553) 539 : cluster [DBG] pgmap v947: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 589 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:21.691968+0000 mgr.y (mgr.14553) 540 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:21.782600+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? v1:172.21.15.107:0/2672882034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:22.123771+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e628: 8 total, 7 up, 8 in 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:22.170958+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:22.171299+0000 mon.a (mon.0) 3112 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.675 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:46:22.585+0000 7f5696911380 -1 Falling back to public interface 2024-02-10T08:46:22.675 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:21.525913+0000 mgr.y (mgr.14553) 539 : cluster [DBG] pgmap v947: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 589 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:22.675 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:21.691968+0000 mgr.y (mgr.14553) 540 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:22.675 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:21.782600+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? v1:172.21.15.107:0/2672882034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:22.675 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:22.123771+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e628: 8 total, 7 up, 8 in 2024-02-10T08:46:22.675 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:22.170958+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.676 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:22.171299+0000 mon.a (mon.0) 3112 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:22.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:22.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:23.198 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:23.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:23.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T08:46:23.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T08:46:23.330 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:23.340 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:23.377 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:23.580 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:23.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:23.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:23.712 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:23.810 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:23.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:23.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:23.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:24.059 INFO:teuthology.orchestra.run.smithi195.stderr:no valid command found; 10 closest matches: 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:0 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:1 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:2 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:abort 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:assert 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:config diff 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:config diff get 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:config get 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:config help [] 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:config set ... 2024-02-10T08:46:24.060 INFO:teuthology.orchestra.run.smithi195.stderr:admin_socket: invalid command 2024-02-10T08:46:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:23.123108+0000 mon.a (mon.0) 3113 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:23.123167+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:23.123898+0000 mon.b (mon.1) 735 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:23.127939+0000 mon.a (mon.0) 3115 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:23.526571+0000 mgr.y (mgr.14553) 541 : cluster [DBG] pgmap v950: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 594 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:24.427 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:23.123108+0000 mon.a (mon.0) 3113 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:24.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:23.123167+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-02-10T08:46:24.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:23.123898+0000 mon.b (mon.1) 735 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:23.127939+0000 mon.a (mon.0) 3115 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:23.526571+0000 mgr.y (mgr.14553) 541 : cluster [DBG] pgmap v950: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 594 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:24.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:24.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:23.123108+0000 mon.a (mon.0) 3113 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:24.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:23.123167+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-02-10T08:46:24.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:23.123898+0000 mon.b (mon.1) 735 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:23.127939+0000 mon.a (mon.0) 3115 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]: dispatch 2024-02-10T08:46:24.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:23.526571+0000 mgr.y (mgr.14553) 541 : cluster [DBG] pgmap v950: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 594 B/s wr, 24 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 81 B/s, 4 objects/s recovering 2024-02-10T08:46:24.530 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:24.561 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T08:46:24.710 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-02-10T08:46:24.710 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:24.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:24.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:24.895 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:24.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:24.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:25.026 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:25.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:46:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:24.121160+0000 mon.a (mon.0) 3116 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:24.128985+0000 mon.a (mon.0) 3117 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:25.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:24.129057+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-02-10T08:46:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:24.121160+0000 mon.a (mon.0) 3116 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:24.128985+0000 mon.a (mon.0) 3117 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:25.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:24.129057+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-02-10T08:46:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:24.121160+0000 mon.a (mon.0) 3116 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:24.128985+0000 mon.a (mon.0) 3117 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-106"}]': finished 2024-02-10T08:46:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:24.129057+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-02-10T08:46:25.578 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:25.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:25.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:25.773 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:25.774 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:25] "GET /metrics HTTP/1.1" 200 189930 "" "Prometheus/2.33.4" 2024-02-10T08:46:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:25.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:25.791 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:46:25.791 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:46:25.791 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:46:25.791 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:46:25.792 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:46:25.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:26.254 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:26.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:26.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:26.386 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:26.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:25.171454+0000 mon.a (mon.0) 3119 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:25.527345+0000 mgr.y (mgr.14553) 542 : cluster [DBG] pgmap v952: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 21 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 70 B/s, 3 objects/s recovering 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:25.583466+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:26.116889+0000 mon.a (mon.0) 3121 : cluster [WRN] Health check update: Degraded data redundancy: 69/615 objects degraded (11.220%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:26.121985+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:25.171454+0000 mon.a (mon.0) 3119 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:25.527345+0000 mgr.y (mgr.14553) 542 : cluster [DBG] pgmap v952: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 21 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 70 B/s, 3 objects/s recovering 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:25.583466+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:26.116889+0000 mon.a (mon.0) 3121 : cluster [WRN] Health check update: Degraded data redundancy: 69/615 objects degraded (11.220%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:26.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:26.121985+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-02-10T08:46:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:25.171454+0000 mon.a (mon.0) 3119 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:25.527345+0000 mgr.y (mgr.14553) 542 : cluster [DBG] pgmap v952: 194 pgs: 20 active+clean+remapped, 18 active+undersized+degraded, 28 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 21 op/s; 69/615 objects degraded (11.220%); 18/615 objects misplaced (2.927%); 70 B/s, 3 objects/s recovering 2024-02-10T08:46:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:25.583466+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:26.116889+0000 mon.a (mon.0) 3121 : cluster [WRN] Health check update: Degraded data redundancy: 69/615 objects degraded (11.220%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:26.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:26.121985+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:26.999+0000 7f1588978700 1 -- 172.21.15.107:0/2507883483 >> v1:172.21.15.107:6790/0 conn(0x7f15840fc8b0 legacy=0x7f15840fcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:26.999+0000 7f1588978700 1 -- 172.21.15.107:0/2507883483 shutdown_connections 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 -- 172.21.15.107:0/2507883483 >> 172.21.15.107:0/2507883483 conn(0x7f15840f2470 msgr2=0x7f15840f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 -- 172.21.15.107:0/2507883483 shutdown_connections 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 -- 172.21.15.107:0/2507883483 wait complete. 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 librados: starting msgr at 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 librados: starting objecter 2024-02-10T08:46:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 Processor -- start 2024-02-10T08:46:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 -- start start 2024-02-10T08:46:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 librados: setting wanted keys 2024-02-10T08:46:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.000+0000 7f1588978700 1 librados: calling monclient init 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f1588978700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15841172f0 con 0x7f15840fc8b0 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f1588978700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15841184d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f1588978700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15841196d0 con 0x7f15841031f0 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f158259c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15840fc8b0 0x7f15840f9a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50846/0 (socket says 172.21.15.107:50846) 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f158259c700 1 -- 172.21.15.107:0/3626254828 learned_addr learned my addr 172.21.15.107:0/3626254828 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f1582d9d700 1 --1- 172.21.15.107:0/3626254828 >> v1:172.21.15.107:6790/0 conn(0x7f15841031f0 0x7f15841159b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38226/0 (socket says 172.21.15.107:38226) 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter ms_handle_connect 0x7f15841031f0 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter ms_handle_connect 0x7f15840ff6b0 2024-02-10T08:46:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter ms_handle_connect 0x7f15840fc8b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592680382 0 0) 0x7f15841196d0 con 0x7f15841031f0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c0033f0 con 0x7f15841031f0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137028702 0 0) 0x7f15841184d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15841196d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814893254 0 0) 0x7f15841172f0 con 0x7f15840fc8b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15841184d0 con 0x7f15840fc8b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486515585 0 0) 0x7f15841196d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15841172f0 con 0x7f15840ff6b0 2024-02-10T08:46:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f15780018c0 con 0x7f15840ff6b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1618977380 0 0) 0x7f156c0033f0 con 0x7f15841031f0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15841196d0 con 0x7f15841031f0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.001+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f157c001cc0 con 0x7f15841031f0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530972506 0 0) 0x7f15841184d0 con 0x7f15840fc8b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f156c0033f0 con 0x7f15840fc8b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1574002d70 con 0x7f15840fc8b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888994765 0 0) 0x7f15841172f0 con 0x7f15840ff6b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 >> v1:172.21.15.107:6790/0 conn(0x7f15841031f0 legacy=0x7f15841159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 >> v1:172.21.15.107:6789/0 conn(0x7f15840fc8b0 legacy=0x7f15840f9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158411a8d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15841186c0 con 0x7f15840ff6b0 2024-02-10T08:46:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f1588978700 10 client.48808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1584118bb0 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f1588978700 1 librados: init done 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1578002230 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.002+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1578003150 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1564004f20 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1578001d90 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(631..631 src has 420..631) v4 ==== 9275+0+0 (unknown 2816322821 0 0) 0x7f15780766d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 10 client.48808.objecter ms_dispatch 0x7f1584001550 osd_map(631..631 src has 420..631) v4 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 3 client.48808.objecter handle_osd_map got epochs [631,631] > 0 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 3 client.48808.objecter handle_osd_map decoding full epoch 631 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.003+0000 7f15737fe700 20 client.48808.objecter dump_active .. 0 homeless 2024-02-10T08:46:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.004+0000 7f15737fe700 10 client.48808.objecter ms_handle_connect 0x7f156c063d90 2024-02-10T08:46:27.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.008+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1578049600 con 0x7f15840ff6b0 2024-02-10T08:46:27.035 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:27.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:27.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:27.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:27.196+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f15640051d0 con 0x7f15840ff6b0 2024-02-10T08:46:27.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:46:26.954+0000 7f5696911380 -1 osd.4 479 log_to_monitors {default=true} 2024-02-10T08:46:27.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:26.169297+0000 mon.a (mon.0) 3123 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:27.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:26.958109+0000 mon.b (mon.1) 736 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:26.958460+0000 mon.a (mon.0) 3124 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:26.169297+0000 mon.a (mon.0) 3123 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:27.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:26.958109+0000 mon.b (mon.1) 736 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:26.958460+0000 mon.a (mon.0) 3124 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:26.169297+0000 mon.a (mon.0) 3123 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:26.958109+0000 mon.b (mon.1) 736 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:26.958460+0000 mon.a (mon.0) 3124 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T08:46:27.483 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:27.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:27.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:46:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:27.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:27.613 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:27.699 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:27.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:27.787 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:46:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:46:27.530+0000 7f56855f3700 -1 osd.4 632 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T08:46:27.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:27.831 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:27.851 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:27.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:27.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:27.981 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:27.999 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:28.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:28.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:28.129 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:28.147 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:28.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:28.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.191+0000 7f15737fe700 1 -- 172.21.15.107:0/3626254828 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v633) v1 ==== 91+0+0 (unknown 2919217982 0 0) 0x7f1578047a10 con 0x7f15840ff6b0 2024-02-10T08:46:28.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.196+0000 7f1588978700 10 librados: watch_flush enter 2024-02-10T08:46:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.196+0000 7f1588978700 10 librados: watch_flush exit 2024-02-10T08:46:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.196+0000 7f1588978700 20 client.48808.objecter shutdown clearing up homeless session... 2024-02-10T08:46:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.196+0000 7f1588978700 10 client.48808.objecter successfully canceled tick 2024-02-10T08:46:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 >> v1:172.21.15.107:6800/2634842009 conn(0x7f156c063d90 legacy=0x7f156c066250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 >> v1:172.21.15.195:6789/0 conn(0x7f15840ff6b0 legacy=0x7f15840fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 shutdown_connections 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 >> 172.21.15.107:0/3626254828 conn(0x7f15840f2470 msgr2=0x7f15840612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 shutdown_connections 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.204+0000 7f1588978700 1 -- 172.21.15.107:0/3626254828 wait complete. 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:28.205+0000 7f1588978700 1 librados: shutdown 2024-02-10T08:46:28.206 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:46:28.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:28.279 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:28.295 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:28.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:28.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:27.183688+0000 mon.a (mon.0) 3125 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:46:28.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:27.183767+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-02-10T08:46:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:27.184309+0000 mon.b (mon.1) 737 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:27.193754+0000 mon.a (mon.0) 3127 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:27.198923+0000 mon.b (mon.1) 738 : audit [INF] from='client.? v1:172.21.15.107:0/3626254828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:27.202065+0000 mon.a (mon.0) 3128 : audit [INF] from='client.48808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:27.528021+0000 mgr.y (mgr.14553) 543 : cluster [DBG] pgmap v955: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:27.183688+0000 mon.a (mon.0) 3125 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:46:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:27.183767+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-02-10T08:46:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:27.184309+0000 mon.b (mon.1) 737 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:27.193754+0000 mon.a (mon.0) 3127 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:27.198923+0000 mon.b (mon.1) 738 : audit [INF] from='client.? v1:172.21.15.107:0/3626254828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:27.202065+0000 mon.a (mon.0) 3128 : audit [INF] from='client.48808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:27.528021+0000 mgr.y (mgr.14553) 543 : cluster [DBG] pgmap v955: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:27.183688+0000 mon.a (mon.0) 3125 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:27.183767+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:27.184309+0000 mon.b (mon.1) 737 : audit [INF] from='osd.4 v1:172.21.15.195:6800/3310789251' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:27.193754+0000 mon.a (mon.0) 3127 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:27.198923+0000 mon.b (mon.1) 738 : audit [INF] from='client.? v1:172.21.15.107:0/3626254828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:27.202065+0000 mon.a (mon.0) 3128 : audit [INF] from='client.48808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:46:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:27.528021+0000 mgr.y (mgr.14553) 543 : cluster [DBG] pgmap v955: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.181795+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.181816+0000 mon.a (mon.0) 3130 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:28.186060+0000 mon.a (mon.0) 3131 : audit [INF] from='client.48808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.186134+0000 mon.a (mon.0) 3132 : cluster [INF] osd.4 v1:172.21.15.195:6800/3310789251 boot 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.186162+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:28.186805+0000 mon.c (mon.2) 449 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.377090+0000 osd.2 (osd.2) 235 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.377900+0000 osd.2 (osd.2) 236 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:46:29.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.190154+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-02-10T08:46:29.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.181795+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:29.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.181816+0000 mon.a (mon.0) 3130 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T08:46:29.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:28.186060+0000 mon.a (mon.0) 3131 : audit [INF] from='client.48808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:46:29.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.186134+0000 mon.a (mon.0) 3132 : cluster [INF] osd.4 v1:172.21.15.195:6800/3310789251 boot 2024-02-10T08:46:29.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.186162+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:28.186805+0000 mon.c (mon.2) 449 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.377090+0000 osd.2 (osd.2) 235 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.377900+0000 osd.2 (osd.2) 236 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.190154+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.181795+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.181816+0000 mon.a (mon.0) 3130 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:28.186060+0000 mon.a (mon.0) 3131 : audit [INF] from='client.48808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.186134+0000 mon.a (mon.0) 3132 : cluster [INF] osd.4 v1:172.21.15.195:6800/3310789251 boot 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.186162+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:28.186805+0000 mon.c (mon.2) 449 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.377090+0000 osd.2 (osd.2) 235 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:46:29.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.377900+0000 osd.2 (osd.2) 236 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:46:29.613 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.190154+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-02-10T08:46:29.679 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:29.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:29.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:46:29.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:29.811 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:29.836 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:29.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:29.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:29.967 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:29.990 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:29.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:30.001 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:46:30.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:27.908344+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:27.908576+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.369338+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:28.372319+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.528642+0000 mgr.y (mgr.14553) 544 : cluster [DBG] pgmap v958: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.904769+0000 osd.4 (osd.4) 3 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.907731+0000 osd.4 (osd.4) 4 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.188899+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-02-10T08:46:30.253 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:30.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:27.908344+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:30.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:27.908576+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:30.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.369338+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:46:30.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:28.372319+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:46:30.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.528642+0000 mgr.y (mgr.14553) 544 : cluster [DBG] pgmap v958: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:30.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.904769+0000 osd.4 (osd.4) 3 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:46:30.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.907731+0000 osd.4 (osd.4) 4 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:46:30.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.188899+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-02-10T08:46:30.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:27.908344+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:30.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:27.908576+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:30.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.369338+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:46:30.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:28.372319+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:46:30.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.528642+0000 mgr.y (mgr.14553) 544 : cluster [DBG] pgmap v958: 194 pgs: 20 active+clean+remapped, 19 active+undersized+degraded, 27 active+undersized, 128 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s; 71/612 objects degraded (11.601%); 18/612 objects misplaced (2.941%) 2024-02-10T08:46:30.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.904769+0000 osd.4 (osd.4) 3 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:46:30.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.907731+0000 osd.4 (osd.4) 4 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:46:30.393 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.188899+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-02-10T08:46:30.975 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:30.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:31.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:31.129 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:31.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:31.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:31.513 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:46:31.513 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:46:31.513 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:46:31.513 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:46:31.523 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:31.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:31.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:31.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:46:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:31.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.922218+0000 osd.0 (osd.0) 93 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:46:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:29.923088+0000 osd.0 (osd.0) 94 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:46:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.590966+0000 osd.6 (osd.6) 258 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:46:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.592663+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:46:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:30.594766+0000 mon.a (mon.0) 3136 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.713997+0000 osd.1 (osd.1) 294 : cluster [DBG] 16.b scrub starts 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.717288+0000 osd.1 (osd.1) 295 : cluster [DBG] 16.b scrub ok 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.947458+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.950915+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.118521+0000 mon.a (mon.0) 3137 : cluster [WRN] Health check update: Degraded data redundancy: 71/612 objects degraded (11.601%), 19 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.214567+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-10T08:46:31.823 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw-foo-a[94732]: debug 2024-02-10T08:46:31.641+0000 7fb5c4cc5700 -1 RGWWatcher::handle_error cookie 94547991864448 err (110) Connection timed out 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.922218+0000 osd.0 (osd.0) 93 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:29.923088+0000 osd.0 (osd.0) 94 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.590966+0000 osd.6 (osd.6) 258 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.592663+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:30.594766+0000 mon.a (mon.0) 3136 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.713997+0000 osd.1 (osd.1) 294 : cluster [DBG] 16.b scrub starts 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.717288+0000 osd.1 (osd.1) 295 : cluster [DBG] 16.b scrub ok 2024-02-10T08:46:31.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.947458+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.950915+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.118521+0000 mon.a (mon.0) 3137 : cluster [WRN] Health check update: Degraded data redundancy: 71/612 objects degraded (11.601%), 19 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.214567+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.922218+0000 osd.0 (osd.0) 93 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:29.923088+0000 osd.0 (osd.0) 94 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.590966+0000 osd.6 (osd.6) 258 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.592663+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:30.594766+0000 mon.a (mon.0) 3136 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.713997+0000 osd.1 (osd.1) 294 : cluster [DBG] 16.b scrub starts 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.717288+0000 osd.1 (osd.1) 295 : cluster [DBG] 16.b scrub ok 2024-02-10T08:46:31.824 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.947458+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:46:31.825 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.950915+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:46:31.825 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.118521+0000 mon.a (mon.0) 3137 : cluster [WRN] Health check update: Degraded data redundancy: 71/612 objects degraded (11.601%), 19 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-10T08:46:31.825 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.214567+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-10T08:46:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:31.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:31.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.070 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:32.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.070+0000 7f281f8bb700 1 -- 172.21.15.107:0/4237543280 >> v1:172.21.15.107:6790/0 conn(0x7f28180f8a60 legacy=0x7f28180f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- 172.21.15.107:0/4237543280 shutdown_connections 2024-02-10T08:46:32.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- 172.21.15.107:0/4237543280 >> 172.21.15.107:0/4237543280 conn(0x7f28180f2510 msgr2=0x7f28180f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:32.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- 172.21.15.107:0/4237543280 shutdown_connections 2024-02-10T08:46:32.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- 172.21.15.107:0/4237543280 wait complete. 2024-02-10T08:46:32.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 librados: starting msgr at 2024-02-10T08:46:32.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 librados: starting objecter 2024-02-10T08:46:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 Processor -- start 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- start start 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 librados: setting wanted keys 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 librados: calling monclient init 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.071+0000 7f281f8bb700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2818102c10 con 0x7f2818104330 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2818102de0 con 0x7f28180f8a60 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2818102fd0 con 0x7f28180f9550 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281ce56700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f28180f9550 0x7f2818114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36738/0 (socket says 172.21.15.107:36738) 2024-02-10T08:46:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281de58700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2818104330 0x7f28181184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49002/0 (socket says 172.21.15.107:49002) 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281ce56700 1 -- 172.21.15.107:0/2861595060 learned_addr learned my addr 172.21.15.107:0/2861595060 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f2818104330 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f28180f9550 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter ms_handle_connect 0x7f28180f8a60 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742922647 0 0) 0x7f2818102de0 con 0x7f28180f8a60 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28040033f0 con 0x7f28180f8a60 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272888422 0 0) 0x7f2818102c10 con 0x7f2818104330 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2818102de0 con 0x7f2818104330 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876696547 0 0) 0x7f2818102fd0 con 0x7f28180f9550 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2818102c10 con 0x7f28180f9550 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3420464185 0 0) 0x7f28040033f0 con 0x7f28180f8a60 2024-02-10T08:46:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2818102fd0 con 0x7f28180f8a60 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2814002050 con 0x7f28180f8a60 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143466740 0 0) 0x7f2818102de0 con 0x7f2818104330 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28040033f0 con 0x7f2818104330 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 167031305 0 0) 0x7f2818102c10 con 0x7f28180f9550 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2818102de0 con 0x7f28180f9550 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2810001bc0 con 0x7f2818104330 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2808001900 con 0x7f28180f9550 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2837054277 0 0) 0x7f2818102fd0 con 0x7f28180f8a60 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6790/0 conn(0x7f28180f9550 legacy=0x7f2818114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6789/0 conn(0x7f2818104330 legacy=0x7f28181184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281811cc00 con 0x7f28180f8a60 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2818119c30 con 0x7f28180f8a60 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 10 client.48814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f281811a0a0 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f281f8bb700 1 librados: init done 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2814002fd0 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.072+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2814003650 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.073+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f27fc004f00 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2814001880 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(636..636 src has 420..636) v4 ==== 8241+0+0 (unknown 2455563277 0 0) 0x7f2814076690 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 osd_map(636..636 src has 420..636) v4 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 3 client.48814.objecter handle_osd_map got epochs [636,636] > 0 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 3 client.48814.objecter handle_osd_map decoding full epoch 636 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 20 client.48814.objecter dump_active .. 0 homeless 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v636) v1 ==== 54+0+16 (unknown 3571788276 0 2420000192) 0x7f2814076a50 con 0x7f28180f8a60 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.074+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f2804063e00 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 10 client.48814.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 1 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 1 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 0 1 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 4 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 10 client.48814.objecter _send_command 1 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc004950 con 0x7f27fc004530 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc004530 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc004950 con 0x7f27fc004530 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 10 client.48814.objecter _finish_command 1 = system:0 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 4 2024-02-10T08:46:32.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.075+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 0 1 2024-02-10T08:46:32.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 10 client.48814.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 2 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 2 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc0051a0 osd=0 3 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 0 2 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 4 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 10 client.48814.objecter _send_command 2 2024-02-10T08:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.090+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc004d30 con 0x7f27fc004530 2024-02-10T08:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc004d30 con 0x7f27fc004530 2024-02-10T08:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 10 client.48814.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 4 2024-02-10T08:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 0 2 2024-02-10T08:46:32.094 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 10 client.48814.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 3 2024-02-10T08:46:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 1 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 4 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 10 client.48814.objecter _send_command 3 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc00bf60 con 0x7f27fc003b40 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.092+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc003b40 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.093+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc00bf60 con 0x7f27fc003b40 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.093+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.093+0000 7f280e7fc700 10 client.48814.objecter _finish_command 3 = system:0 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.093+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 4 2024-02-10T08:46:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.093+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 1 3 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 10 client.48814.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 4 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 4 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc004270 osd=1 3 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 1 4 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 4 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 10 client.48814.objecter _send_command 4 2024-02-10T08:46:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.107+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc00d280 con 0x7f27fc003b40 2024-02-10T08:46:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc00d280 con 0x7f27fc003b40 2024-02-10T08:46:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 10 client.48814.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 4 2024-02-10T08:46:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 1 4 2024-02-10T08:46:32.111 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 10 client.48814.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 5 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 5 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 2 5 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 4 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 10 client.48814.objecter _send_command 5 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc010d30 con 0x7f27fc002070 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.109+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc002070 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.110+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc010d30 con 0x7f27fc002070 2024-02-10T08:46:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.110+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:32.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.110+0000 7f280e7fc700 10 client.48814.objecter _finish_command 5 = system:0 2024-02-10T08:46:32.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.110+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 4 2024-02-10T08:46:32.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.110+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 2 5 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 10 client.48814.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 6 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 6 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc0033a0 osd=2 3 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 2 6 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 4 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 10 client.48814.objecter _send_command 6 2024-02-10T08:46:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.124+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc001630 con 0x7f27fc002070 2024-02-10T08:46:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc001630 con 0x7f27fc002070 2024-02-10T08:46:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 10 client.48814.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 4 2024-02-10T08:46:32.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 2 6 2024-02-10T08:46:32.128 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:32.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 10 client.48814.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 7 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 7 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 3 7 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 4 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 10 client.48814.objecter _send_command 7 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc00d0b0 con 0x7f27fc010ed0 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.126+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc010ed0 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.127+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc00d0b0 con 0x7f27fc010ed0 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.127+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.127+0000 7f280e7fc700 10 client.48814.objecter _finish_command 7 = system:0 2024-02-10T08:46:32.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.127+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 4 2024-02-10T08:46:32.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.127+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 3 7 2024-02-10T08:46:32.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 10 client.48814.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 8 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 8 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc00c8d0 osd=3 3 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 3 8 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 4 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 10 client.48814.objecter _send_command 8 2024-02-10T08:46:32.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.141+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc012260 con 0x7f27fc010ed0 2024-02-10T08:46:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc012260 con 0x7f27fc010ed0 2024-02-10T08:46:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f280e7fc700 10 client.48814.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 4 2024-02-10T08:46:32.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 3 8 2024-02-10T08:46:32.145 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 10 client.48814.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 9 2024-02-10T08:46:32.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 9 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 4 9 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 4 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 10 client.48814.objecter _send_command 9 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.143+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc01a2d0 con 0x7f27fc0166f0 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc0166f0 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc01a2d0 con 0x7f27fc0166f0 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 10 client.48814.objecter _finish_command 9 = system:0 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 4 2024-02-10T08:46:32.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.144+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 4 9 2024-02-10T08:46:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 10 client.48814.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 10 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 10 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc012260 osd=4 3 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 4 10 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 4 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 10 client.48814.objecter _send_command 10 2024-02-10T08:46:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.159+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc01ba80 con 0x7f27fc0166f0 2024-02-10T08:46:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc01ba80 con 0x7f27fc0166f0 2024-02-10T08:46:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f280e7fc700 10 client.48814.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 4 2024-02-10T08:46:32.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 4 10 2024-02-10T08:46:32.163 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 10 client.48814.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 11 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 11 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 5 11 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 4 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 10 client.48814.objecter _send_command 11 2024-02-10T08:46:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.161+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc01c6b0 con 0x7f27fc01c100 2024-02-10T08:46:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc01c100 2024-02-10T08:46:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc01c6b0 con 0x7f27fc01c100 2024-02-10T08:46:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:46:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 10 client.48814.objecter _finish_command 11 = system:0 2024-02-10T08:46:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 4 2024-02-10T08:46:32.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.162+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 5 11 2024-02-10T08:46:32.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 10 client.48814.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 12 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 12 2024-02-10T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc01ba80 osd=5 3 2024-02-10T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 5 12 2024-02-10T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 4 2024-02-10T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 10 client.48814.objecter _send_command 12 2024-02-10T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.177+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc01c6b0 con 0x7f27fc01c100 2024-02-10T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc01c6b0 con 0x7f27fc01c100 2024-02-10T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 10 client.48814.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 4 2024-02-10T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 5 12 2024-02-10T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 10 client.48814.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 13 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 13 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 6 13 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 4 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 10 client.48814.objecter _send_command 13 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc022ca0 con 0x7f27fc01f190 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.179+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc01f190 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.180+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc022ca0 con 0x7f27fc01f190 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.180+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:46:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.180+0000 7f280e7fc700 10 client.48814.objecter _finish_command 13 = system:0 2024-02-10T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.180+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 4 2024-02-10T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.180+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 6 13 2024-02-10T08:46:32.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 10 client.48814.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 14 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 14 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc01c6b0 osd=6 3 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 6 14 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 4 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 10 client.48814.objecter _send_command 14 2024-02-10T08:46:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.195+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc022ca0 con 0x7f27fc01f190 2024-02-10T08:46:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc022ca0 con 0x7f27fc01f190 2024-02-10T08:46:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f280e7fc700 10 client.48814.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 4 2024-02-10T08:46:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 6 14 2024-02-10T08:46:32.199 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 10 client.48814.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 15 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 15 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 7 15 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 4 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 10 client.48814.objecter _send_command 15 2024-02-10T08:46:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.197+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27fc027010 con 0x7f27fc023320 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 10 client.48814.objecter ms_handle_connect 0x7f27fc023320 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f27fc027010 con 0x7f27fc023320 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 10 client.48814.objecter _finish_command 15 = system:0 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 4 2024-02-10T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.198+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 7 15 2024-02-10T08:46:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 10 client.48814.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign -1 16 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 20 client.48814.objecter _get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_remove -1 16 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 20 client.48814.objecter get_session s=0x7f27fc022ca0 osd=7 3 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 15 client.48814.objecter _session_command_op_assign 7 16 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 4 2024-02-10T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 10 client.48814.objecter _send_command 16 2024-02-10T08:46:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.215+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f27fc0287c0 con 0x7f27fc023320 2024-02-10T08:46:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f280e7fc700 1 -- 172.21.15.107:0/2861595060 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f27fc0287c0 con 0x7f27fc023320 2024-02-10T08:46:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f280e7fc700 10 client.48814.objecter ms_dispatch 0x7f2818001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f280e7fc700 10 client.48814.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f280e7fc700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 4 2024-02-10T08:46:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f280e7fc700 15 client.48814.objecter _session_command_op_remove 7 16 2024-02-10T08:46:32.219 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 librados: watch_flush enter 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 librados: watch_flush exit 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.0 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6801/3321596972 conn(0x7f27fc004530 legacy=0x7f27fc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0051a0 osd=0 1 2024-02-10T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.1 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6805/133516126 conn(0x7f27fc003b40 legacy=0x7f27fc00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc004270 osd=1 1 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.2 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6809/3575881728 conn(0x7f27fc002070 legacy=0x7f27fc00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc0033a0 osd=2 1 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.3 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6813/2871825691 conn(0x7f27fc010ed0 legacy=0x7f27fc011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc00c8d0 osd=3 1 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.4 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.195:6800/3310789251 conn(0x7f27fc0166f0 legacy=0x7f27fc018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281de58700 1 -- 172.21.15.107:0/2861595060 reap_dead start 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc012260 osd=4 1 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.5 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.217+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.195:6804/4178321486 conn(0x7f27fc01c100 legacy=0x7f27fc01a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281de58700 1 -- 172.21.15.107:0/2861595060 reap_dead start 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01ba80 osd=5 1 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.6 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.195:6808/1417479017 conn(0x7f27fc01f190 legacy=0x7f27fc021580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc01c6b0 osd=6 1 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 10 client.48814.objecter close_session for osd.7 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.195:6812/1934176928 conn(0x7f27fc023320 legacy=0x7f27fc025790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 20 client.48814.objecter put_session s=0x7f27fc022ca0 osd=7 1 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 20 client.48814.objecter shutdown clearing up homeless session... 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 10 client.48814.objecter successfully canceled tick 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281de58700 1 -- 172.21.15.107:0/2861595060 reap_dead start 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2804063e00 legacy=0x7f28040662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> v1:172.21.15.195:6789/0 conn(0x7f28180f8a60 legacy=0x7f2818102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 shutdown_connections 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 >> 172.21.15.107:0/2861595060 conn(0x7f28180f2510 msgr2=0x7f28181077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 shutdown_connections 2024-02-10T08:46:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.218+0000 7f281f8bb700 1 -- 172.21.15.107:0/2861595060 wait complete. 2024-02-10T08:46:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:32.219+0000 7f281f8bb700 1 librados: shutdown 2024-02-10T08:46:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:46:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:32.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:32.367 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.386 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:32.402 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:32.402 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:32.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:32.517 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:32.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.822339+0000 osd.5 (osd.5) 262 : cluster [DBG] 16.c scrub starts 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:30.825225+0000 osd.5 (osd.5) 263 : cluster [DBG] 16.c scrub ok 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.350738+0000 osd.7 (osd.7) 248 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.350968+0000 osd.7 (osd.7) 249 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.352937+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.356660+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-02-10T08:46:32.664 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.452199+0000 osd.7 (osd.7) 250 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.456575+0000 osd.7 (osd.7) 251 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.529157+0000 mgr.y (mgr.14553) 545 : cluster [DBG] pgmap v961: 162 pgs: 1 active+clean+snaptrim, 5 active+clean+remapped, 8 active+undersized+degraded, 8 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 38/600 objects degraded (6.333%); 14/600 objects misplaced (2.333%); 3 B/s, 0 objects/s recovering 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.574335+0000 osd.6 (osd.6) 260 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.577208+0000 osd.6 (osd.6) 261 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:31.701958+0000 mgr.y (mgr.14553) 546 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:32.075465+0000 mon.b (mon.1) 739 : audit [DBG] from='client.? v1:172.21.15.107:0/2861595060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.211963+0000 mon.a (mon.0) 3139 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.224408+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:32.233945+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.665 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:32.235030+0000 mon.a (mon.0) 3141 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.666 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:32.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:32.818 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.822339+0000 osd.5 (osd.5) 262 : cluster [DBG] 16.c scrub starts 2024-02-10T08:46:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:30.825225+0000 osd.5 (osd.5) 263 : cluster [DBG] 16.c scrub ok 2024-02-10T08:46:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.350738+0000 osd.7 (osd.7) 248 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.350968+0000 osd.7 (osd.7) 249 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.352937+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.356660+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.452199+0000 osd.7 (osd.7) 250 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.456575+0000 osd.7 (osd.7) 251 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.529157+0000 mgr.y (mgr.14553) 545 : cluster [DBG] pgmap v961: 162 pgs: 1 active+clean+snaptrim, 5 active+clean+remapped, 8 active+undersized+degraded, 8 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 38/600 objects degraded (6.333%); 14/600 objects misplaced (2.333%); 3 B/s, 0 objects/s recovering 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.574335+0000 osd.6 (osd.6) 260 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.577208+0000 osd.6 (osd.6) 261 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:31.701958+0000 mgr.y (mgr.14553) 546 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:32.075465+0000 mon.b (mon.1) 739 : audit [DBG] from='client.? v1:172.21.15.107:0/2861595060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.211963+0000 mon.a (mon.0) 3139 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.224408+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-10T08:46:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:32.233945+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:32.235030+0000 mon.a (mon.0) 3141 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.822339+0000 osd.5 (osd.5) 262 : cluster [DBG] 16.c scrub starts 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:30.825225+0000 osd.5 (osd.5) 263 : cluster [DBG] 16.c scrub ok 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.350738+0000 osd.7 (osd.7) 248 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.350968+0000 osd.7 (osd.7) 249 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.352937+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.a scrub starts 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.356660+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.a scrub ok 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.452199+0000 osd.7 (osd.7) 250 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.456575+0000 osd.7 (osd.7) 251 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.529157+0000 mgr.y (mgr.14553) 545 : cluster [DBG] pgmap v961: 162 pgs: 1 active+clean+snaptrim, 5 active+clean+remapped, 8 active+undersized+degraded, 8 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 38/600 objects degraded (6.333%); 14/600 objects misplaced (2.333%); 3 B/s, 0 objects/s recovering 2024-02-10T08:46:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.574335+0000 osd.6 (osd.6) 260 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.577208+0000 osd.6 (osd.6) 261 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:31.701958+0000 mgr.y (mgr.14553) 546 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:32.075465+0000 mon.b (mon.1) 739 : audit [DBG] from='client.? v1:172.21.15.107:0/2861595060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.211963+0000 mon.a (mon.0) 3139 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.224408+0000 mon.a (mon.0) 3140 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:32.233945+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.933 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:32.235030+0000 mon.a (mon.0) 3141 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:32.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:33.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:33.104 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:33.351 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:33.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:33.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:33.481 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.820721+0000 osd.5 (osd.5) 264 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:31.823898+0000 osd.5 (osd.5) 265 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.368470+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.f scrub starts 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.371493+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.f scrub ok 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.416213+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.419094+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:33.226439+0000 mon.a (mon.0) 3142 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.226516+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-10T08:46:33.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.820721+0000 osd.5 (osd.5) 264 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:31.823898+0000 osd.5 (osd.5) 265 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.368470+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.f scrub starts 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.371493+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.f scrub ok 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.416213+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.419094+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:33.226439+0000 mon.a (mon.0) 3142 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.226516+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-10T08:46:33.859 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:33.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.820721+0000 osd.5 (osd.5) 264 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:31.823898+0000 osd.5 (osd.5) 265 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.368470+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.f scrub starts 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.371493+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.f scrub ok 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.416213+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.419094+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:33.226439+0000 mon.a (mon.0) 3142 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.226516+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-10T08:46:34.148 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:34.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:34.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:34.286 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:34.308 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:34.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:34.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.977998+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.b scrub starts 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:32.979932+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.b scrub ok 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.459736+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.460863+0000 osd.3 (osd.3) 284 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.529844+0000 mgr.y (mgr.14553) 547 : cluster [DBG] pgmap v964: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.630957+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:33.634646+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:34.220591+0000 mon.a (mon.0) 3144 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/600 objects degraded (6.333%), 8 pgs degraded, 15 pgs undersized) 2024-02-10T08:46:34.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:34.228480+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:34.236926+0000 mon.b (mon.1) 741 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:34.237340+0000 mon.a (mon.0) 3146 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.977998+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.b scrub starts 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:32.979932+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.b scrub ok 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.459736+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.460863+0000 osd.3 (osd.3) 284 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.529844+0000 mgr.y (mgr.14553) 547 : cluster [DBG] pgmap v964: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.630957+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:33.634646+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:34.220591+0000 mon.a (mon.0) 3144 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/600 objects degraded (6.333%), 8 pgs degraded, 15 pgs undersized) 2024-02-10T08:46:34.709 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:34.228480+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-10T08:46:34.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:34.236926+0000 mon.b (mon.1) 741 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:34.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:34.237340+0000 mon.a (mon.0) 3146 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.977998+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.b scrub starts 2024-02-10T08:46:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:32.979932+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.b scrub ok 2024-02-10T08:46:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.459736+0000 osd.3 (osd.3) 283 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:46:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.460863+0000 osd.3 (osd.3) 284 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:46:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.529844+0000 mgr.y (mgr.14553) 547 : cluster [DBG] pgmap v964: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.630957+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:33.634646+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:34.220591+0000 mon.a (mon.0) 3144 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/600 objects degraded (6.333%), 8 pgs degraded, 15 pgs undersized) 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:34.228480+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:34.236926+0000 mon.b (mon.1) 741 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:35.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:34.237340+0000 mon.a (mon.0) 3146 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:35.041 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:35.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:35.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:46:35.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:35.427 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:35.448 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:35.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:35.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:35.581 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:35] "GET /metrics HTTP/1.1" 200 189930 "" "Prometheus/2.33.4" 2024-02-10T08:46:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:35.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:36.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:36.068 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:36.089 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:36.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:36.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:36.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:34.784431+0000 osd.5 (osd.5) 266 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:46:36.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:34.787654+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:35.230259+0000 mon.a (mon.0) 3147 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:35.230328+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:35.231472+0000 mon.b (mon.1) 742 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:35.231735+0000 mon.a (mon.0) 3149 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:35.530199+0000 mgr.y (mgr.14553) 548 : cluster [DBG] pgmap v967: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:36.125794+0000 mon.a (mon.0) 3150 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.125867+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:36.130066+0000 mon.b (mon.1) 743 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:36.130316+0000 mon.a (mon.0) 3152 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:36.500 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:34.784431+0000 osd.5 (osd.5) 266 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:34.787654+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:35.230259+0000 mon.a (mon.0) 3147 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:35.230328+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:35.231472+0000 mon.b (mon.1) 742 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:35.231735+0000 mon.a (mon.0) 3149 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:35.530199+0000 mgr.y (mgr.14553) 548 : cluster [DBG] pgmap v967: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:36.125794+0000 mon.a (mon.0) 3150 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.125867+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:36.130066+0000 mon.b (mon.1) 743 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:36.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:36.130316+0000 mon.a (mon.0) 3152 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:34.784431+0000 osd.5 (osd.5) 266 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:46:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:34.787654+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:46:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:35.230259+0000 mon.a (mon.0) 3147 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:35.230328+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-10T08:46:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:35.231472+0000 mon.b (mon.1) 742 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:35.231735+0000 mon.a (mon.0) 3149 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:35.530199+0000 mgr.y (mgr.14553) 548 : cluster [DBG] pgmap v967: 194 pgs: 32 unknown, 1 active+clean+snaptrim, 161 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:36.125794+0000 mon.a (mon.0) 3150 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.125867+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:36.130066+0000 mon.b (mon.1) 743 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:36.130316+0000 mon.a (mon.0) 3152 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]: dispatch 2024-02-10T08:46:37.115 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:37.115 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:37.124 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:46:37.124 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:46:37.125 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-10T08:46:37.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:46:37.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:35.560824+0000 osd.6 (osd.6) 262 : cluster [DBG] 16.d scrub starts 2024-02-10T08:46:37.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:35.562987+0000 osd.6 (osd.6) 263 : cluster [DBG] 16.d scrub ok 2024-02-10T08:46:37.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.869623+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.c scrub starts 2024-02-10T08:46:37.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.873114+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.c scrub ok 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.128230+0000 mon.a (mon.0) 3153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:37.132803+0000 mon.a (mon.0) 3154 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]': finished 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.132874+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:35.560824+0000 osd.6 (osd.6) 262 : cluster [DBG] 16.d scrub starts 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:35.562987+0000 osd.6 (osd.6) 263 : cluster [DBG] 16.d scrub ok 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.869623+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.c scrub starts 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.873114+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.c scrub ok 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.128230+0000 mon.a (mon.0) 3153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:37.132803+0000 mon.a (mon.0) 3154 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]': finished 2024-02-10T08:46:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.132874+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.085+0000 7f3e09407700 1 -- 172.21.15.107:0/1010079416 >> v1:172.21.15.107:6790/0 conn(0x7f3e040f9570 legacy=0x7f3e04102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.085+0000 7f3e09407700 1 -- 172.21.15.107:0/1010079416 shutdown_connections 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.085+0000 7f3e09407700 1 -- 172.21.15.107:0/1010079416 >> 172.21.15.107:0/1010079416 conn(0x7f3e040f2510 msgr2=0x7f3e040f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 -- 172.21.15.107:0/1010079416 shutdown_connections 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 -- 172.21.15.107:0/1010079416 wait complete. 2024-02-10T08:46:37.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 librados: starting msgr at 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 librados: starting objecter 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 Processor -- start 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 -- start start 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 librados: setting wanted keys 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 librados: calling monclient init 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.086+0000 7f3e09407700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e04117470 con 0x7f3e04104350 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e04118650 con 0x7f3e040f9570 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e04119850 con 0x7f3e040f89c0 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e02ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3e040f89c0 0x7f3e040ff760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36744/0 (socket says 172.21.15.107:36744) 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e02ffd700 1 -- 172.21.15.107:0/505026250 learned_addr learned my addr 172.21.15.107:0/505026250 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter ms_handle_connect 0x7f3e04104350 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter ms_handle_connect 0x7f3e040f89c0 2024-02-10T08:46:37.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter ms_handle_connect 0x7f3e040f9570 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287323301 0 0) 0x7f3e04117470 con 0x7f3e04104350 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dec003030 con 0x7f3e04104350 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303887616 0 0) 0x7f3e04119850 con 0x7f3e040f89c0 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e04117470 con 0x7f3e040f89c0 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760741142 0 0) 0x7f3e04118650 con 0x7f3e040f9570 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e04119850 con 0x7f3e040f9570 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270220471 0 0) 0x7f3dec003030 con 0x7f3e04104350 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e04118650 con 0x7f3e04104350 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3dfc0018d0 con 0x7f3e04104350 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929931047 0 0) 0x7f3e04117470 con 0x7f3e040f89c0 2024-02-10T08:46:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dec003030 con 0x7f3e040f89c0 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3df4001880 con 0x7f3e040f89c0 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712065976 0 0) 0x7f3e04119850 con 0x7f3e040f9570 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e04117470 con 0x7f3e040f9570 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3df8001430 con 0x7f3e040f9570 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180779443 0 0) 0x7f3e04118650 con 0x7f3e04104350 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 >> v1:172.21.15.107:6790/0 conn(0x7f3e040f89c0 legacy=0x7f3e040ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 >> v1:172.21.15.195:6789/0 conn(0x7f3e040f9570 legacy=0x7f3e040ffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0411aa50 con 0x7f3e04104350 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e04119a40 con 0x7f3e04104350 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 10 client.28518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e04119eb0 con 0x7f3e04104350 2024-02-10T08:46:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3e09407700 1 librados: init done 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3dfc002570 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.087+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3dfc003550 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.088+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0004f20 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3dfc0020d0 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(641..641 src has 420..641) v4 ==== 8623+0+0 (unknown 3684246713 0 0) 0x7f3dfc076870 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 10 client.28518.objecter ms_dispatch 0x7f3e04001550 osd_map(641..641 src has 420..641) v4 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 3 client.28518.objecter handle_osd_map got epochs [641,641] > 0 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 3 client.28518.objecter handle_osd_map decoding full epoch 641 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 20 client.28518.objecter dump_active .. 0 homeless 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.089+0000 7f3debfff700 10 client.28518.objecter ms_handle_connect 0x7f3dec063d60 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.094+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3dfc049a30 con 0x7f3e04104350 2024-02-10T08:46:37.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:37.280+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f3df00051d0 con 0x7f3e04104350 2024-02-10T08:46:37.471 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:37.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:35.560824+0000 osd.6 (osd.6) 262 : cluster [DBG] 16.d scrub starts 2024-02-10T08:46:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:35.562987+0000 osd.6 (osd.6) 263 : cluster [DBG] 16.d scrub ok 2024-02-10T08:46:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.869623+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.c scrub starts 2024-02-10T08:46:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.873114+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.c scrub ok 2024-02-10T08:46:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.128230+0000 mon.a (mon.0) 3153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:37.132803+0000 mon.a (mon.0) 3154 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-108", "mode": "writeback"}]': finished 2024-02-10T08:46:37.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.132874+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-10T08:46:37.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:37.607 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:37.628 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:37.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:37.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:37.762 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:37.782 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:37.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:37.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:37.914 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:38.013 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:38.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:38.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:46:38.040 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:38.041 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:38.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:38.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.136+0000 7f3debfff700 1 -- 172.21.15.107:0/505026250 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v643) v1 ==== 101+0+0 (unknown 2165367719 0 0) 0x7f3dfc047e40 con 0x7f3e04104350 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 10 librados: watch_flush enter 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 10 librados: watch_flush exit 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 20 client.28518.objecter shutdown clearing up homeless session... 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 10 client.28518.objecter successfully canceled tick 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3dec063d60 legacy=0x7f3dec066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 >> v1:172.21.15.107:6789/0 conn(0x7f3e04104350 legacy=0x7f3e04115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 shutdown_connections 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 >> 172.21.15.107:0/505026250 conn(0x7f3e040f2510 msgr2=0x7f3e041077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:38.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 shutdown_connections 2024-02-10T08:46:38.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 -- 172.21.15.107:0/505026250 wait complete. 2024-02-10T08:46:38.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:38.138+0000 7f3e09407700 1 librados: shutdown 2024-02-10T08:46:38.141 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:46:38.156 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:38.177 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:38.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:38.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:38.309 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:38.328 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:38.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:38.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:38.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.561031+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:46:38.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.562319+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:46:38.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.684982+0000 osd.1 (osd.1) 298 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:46:38.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:36.688431+0000 osd.1 (osd.1) 299 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:46:38.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:37.282074+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.470364+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.e scrub starts 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.473390+0000 osd.2 (osd.2) 242 : cluster [DBG] 16.e scrub ok 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.530850+0000 mgr.y (mgr.14553) 549 : cluster [DBG] pgmap v970: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.820337+0000 osd.4 (osd.4) 9 : cluster [DBG] 6.e scrub starts 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:37.823464+0000 osd.4 (osd.4) 10 : cluster [DBG] 6.e scrub ok 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.129866+0000 mon.a (mon.0) 3157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:38.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:38.134430+0000 mon.a (mon.0) 3158 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:38.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.134474+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-10T08:46:38.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:38.166191+0000 mon.b (mon.1) 744 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:38.166530+0000 mon.a (mon.0) 3160 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.561031+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:46:38.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.562319+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.684982+0000 osd.1 (osd.1) 298 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:36.688431+0000 osd.1 (osd.1) 299 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:37.282074+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.470364+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.e scrub starts 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.473390+0000 osd.2 (osd.2) 242 : cluster [DBG] 16.e scrub ok 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.530850+0000 mgr.y (mgr.14553) 549 : cluster [DBG] pgmap v970: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.820337+0000 osd.4 (osd.4) 9 : cluster [DBG] 6.e scrub starts 2024-02-10T08:46:38.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:37.823464+0000 osd.4 (osd.4) 10 : cluster [DBG] 6.e scrub ok 2024-02-10T08:46:38.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.129866+0000 mon.a (mon.0) 3157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:38.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:38.134430+0000 mon.a (mon.0) 3158 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:38.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.134474+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-10T08:46:38.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:38.166191+0000 mon.b (mon.1) 744 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:38.166530+0000 mon.a (mon.0) 3160 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.512 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:38.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.561031+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.562319+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.684982+0000 osd.1 (osd.1) 298 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:36.688431+0000 osd.1 (osd.1) 299 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:37.282074+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.470364+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.e scrub starts 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.473390+0000 osd.2 (osd.2) 242 : cluster [DBG] 16.e scrub ok 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.530850+0000 mgr.y (mgr.14553) 549 : cluster [DBG] pgmap v970: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.820337+0000 osd.4 (osd.4) 9 : cluster [DBG] 6.e scrub starts 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:37.823464+0000 osd.4 (osd.4) 10 : cluster [DBG] 6.e scrub ok 2024-02-10T08:46:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.129866+0000 mon.a (mon.0) 3157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:46:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:38.134430+0000 mon.a (mon.0) 3158 : audit [INF] from='client.? v1:172.21.15.107:0/505026250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:46:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.134474+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-10T08:46:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:38.166191+0000 mon.b (mon.1) 744 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:38.166530+0000 mon.a (mon.0) 3160 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:38.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:38.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:38.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:39.144 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:39.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:39.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:39.284 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:39.306 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:39.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:39.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.037997+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.039060+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.446415+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.449604+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:39.137948+0000 mon.a (mon.0) 3161 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.138019+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:39.139542+0000 mon.b (mon.1) 745 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:39.139919+0000 mon.a (mon.0) 3163 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.037997+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.039060+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.446415+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.449604+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:39.137948+0000 mon.a (mon.0) 3161 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.138019+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:39.139542+0000 mon.b (mon.1) 745 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:39.139919+0000 mon.a (mon.0) 3163 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.437 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.037997+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.039060+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.446415+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.449604+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:39.137948+0000 mon.a (mon.0) 3161 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.138019+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-10T08:46:39.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:39.139542+0000 mon.b (mon.1) 745 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:39.139919+0000 mon.a (mon.0) 3163 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]: dispatch 2024-02-10T08:46:39.596 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:39.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:39.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:39.732 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:39.753 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:39.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:39.885 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:39.903 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:39.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:40.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:40.037 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:40.059 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:40.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:40.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.849675+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.f scrub starts 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:38.853137+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.f scrub ok 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.454241+0000 osd.2 (osd.2) 245 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.455165+0000 osd.2 (osd.2) 246 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:46:40.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.531296+0000 mgr.y (mgr.14553) 550 : cluster [DBG] pgmap v973: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.132895+0000 mon.a (mon.0) 3164 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:40.139460+0000 mon.a (mon.0) 3165 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.139516+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-10T08:46:40.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:40.170405+0000 mon.a (mon.0) 3167 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:40.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.849675+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.f scrub starts 2024-02-10T08:46:40.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:38.853137+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.f scrub ok 2024-02-10T08:46:40.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.454241+0000 osd.2 (osd.2) 245 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.455165+0000 osd.2 (osd.2) 246 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.531296+0000 mgr.y (mgr.14553) 550 : cluster [DBG] pgmap v973: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.132895+0000 mon.a (mon.0) 3164 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:40.139460+0000 mon.a (mon.0) 3165 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.139516+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:40.170405+0000 mon.a (mon.0) 3167 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.849675+0000 osd.5 (osd.5) 268 : cluster [DBG] 5.f scrub starts 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:38.853137+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.f scrub ok 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.454241+0000 osd.2 (osd.2) 245 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.455165+0000 osd.2 (osd.2) 246 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.531296+0000 mgr.y (mgr.14553) 550 : cluster [DBG] pgmap v973: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:46:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.132895+0000 mon.a (mon.0) 3164 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:40.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:40.139460+0000 mon.a (mon.0) 3165 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-108"}]': finished 2024-02-10T08:46:40.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.139516+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-10T08:46:40.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:40.170405+0000 mon.a (mon.0) 3167 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:41.271 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:41.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:41.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:41.408 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:41.429 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:41.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:41.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:41.563 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.497261+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:46:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.499027+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:46:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.564923+0000 osd.7 (osd.7) 256 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:46:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.567815+0000 osd.7 (osd.7) 257 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:46:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.658849+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:39.661739+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.891949+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.895765+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.013625+0000 mon.c (mon.2) 450 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.013817+0000 mon.c (mon.2) 451 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.013940+0000 mon.a (mon.0) 3168 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.014237+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.126792+0000 mon.a (mon.0) 3170 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.126876+0000 mon.a (mon.0) 3171 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]': finished 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.126932+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-10T08:46:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.497261+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.499027+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.564923+0000 osd.7 (osd.7) 256 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.567815+0000 osd.7 (osd.7) 257 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.658849+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:39.661739+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.891949+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.895765+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.013625+0000 mon.c (mon.2) 450 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.013817+0000 mon.c (mon.2) 451 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.013940+0000 mon.a (mon.0) 3168 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.014237+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.126792+0000 mon.a (mon.0) 3170 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.126876+0000 mon.a (mon.0) 3171 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]': finished 2024-02-10T08:46:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.126932+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-10T08:46:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:41.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:41.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.497261+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:46:41.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.499027+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:46:41.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.564923+0000 osd.7 (osd.7) 256 : cluster [DBG] 5.10 deep-scrub starts 2024-02-10T08:46:41.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.567815+0000 osd.7 (osd.7) 257 : cluster [DBG] 5.10 deep-scrub ok 2024-02-10T08:46:41.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.658849+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:39.661739+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.891949+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.895765+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.013625+0000 mon.c (mon.2) 450 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.013817+0000 mon.c (mon.2) 451 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.013940+0000 mon.a (mon.0) 3168 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]: dispatch 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.014237+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]: dispatch 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.126792+0000 mon.a (mon.0) 3170 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.126876+0000 mon.a (mon.0) 3171 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.d", "id": [0, 4]}]': finished 2024-02-10T08:46:41.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.126932+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-10T08:46:41.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.921+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/75669352 >> v1:172.21.15.107:6789/0 conn(0x7fdca40ff720 legacy=0x7fdca4101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.921+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/75669352 shutdown_connections 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.921+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/75669352 >> 172.21.15.107:0/75669352 conn(0x7fdca40f2540 msgr2=0x7fdca40f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.921+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/75669352 shutdown_connections 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.921+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/75669352 wait complete. 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 librados: starting msgr at 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 librados: starting objecter 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 Processor -- start 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 -- start start 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 librados: setting wanted keys 2024-02-10T08:46:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 librados: calling monclient init 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdca4119530 con 0x7fdca40ff720 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdca411a710 con 0x7fdca40fc920 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdcaa2e0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdca411b910 con 0x7fdca4103260 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca37fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdca40ff720 0x7fdca40613c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46104/0 (socket says 172.21.15.107:46104) 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca37fe700 1 -- 172.21.15.107:0/2787007467 learned_addr learned my addr 172.21.15.107:0/2787007467 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca1775700 10 client.?.objecter ms_handle_connect 0x7fdca4103260 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca1775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca1775700 10 client.?.objecter ms_handle_connect 0x7fdca40ff720 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.922+0000 7fdca1775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 10 client.?.objecter ms_handle_connect 0x7fdca40fc920 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226453911 0 0) 0x7fdca411b910 con 0x7fdca4103260 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc8c003800 con 0x7fdca4103260 2024-02-10T08:46:41.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499026966 0 0) 0x7fdca4119530 con 0x7fdca40ff720 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca411b910 con 0x7fdca40ff720 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596194690 0 0) 0x7fdca411a710 con 0x7fdca40fc920 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca4119530 con 0x7fdca40fc920 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704149150 0 0) 0x7fdc8c003800 con 0x7fdca4103260 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca411a710 con 0x7fdca4103260 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc9c001c40 con 0x7fdca4103260 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3196725313 0 0) 0x7fdca411b910 con 0x7fdca40ff720 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc8c003800 con 0x7fdca40ff720 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc94002f60 con 0x7fdca40ff720 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183765544 0 0) 0x7fdca4119530 con 0x7fdca40fc920 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca411b910 con 0x7fdca40fc920 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc980016a0 con 0x7fdca40fc920 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2428937199 0 0) 0x7fdca411a710 con 0x7fdca4103260 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.195:6789/0 conn(0x7fdca40fc920 legacy=0x7fdca4060cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:41.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6789/0 conn(0x7fdca40ff720 legacy=0x7fdca40613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca411cb10 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca4119720 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdcaa2e0700 10 client.54101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4119c10 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdcaa2e0700 1 librados: init done 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fdc9c0023d0 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.923+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdc9c003690 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.924+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdc90004f00 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.924+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fdc9c001f30 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(646..646 src has 420..646) v4 ==== 8334+0+0 (unknown 1878630520 0 0) 0x7fdc9c076940 con 0x7fdca4103260 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 osd_map(646..646 src has 420..646) v4 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 3 client.54101.objecter handle_osd_map got epochs [646,646] > 0 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 3 client.54101.objecter handle_osd_map decoding full epoch 646 2024-02-10T08:46:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 20 client.54101.objecter dump_active .. 0 homeless 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v646) v1 ==== 54+0+16 (unknown 3034678968 0 2420000192) 0x7fdc9c076ca0 con 0x7fdca4103260 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc8c063cb0 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 1 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 1 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 0 1 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 4 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 1 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90004950 con 0x7fdc90004530 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.925+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc90004530 2024-02-10T08:46:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.926+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc90004950 con 0x7fdc90004530 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.926+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.926+0000 7fdca1775700 10 client.54101.objecter _finish_command 1 = system:0 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.926+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 4 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.926+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 0 1 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 2 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 2 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc900051a0 osd=0 3 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 0 2 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 4 2024-02-10T08:46:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 2 2024-02-10T08:46:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.941+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc90004d30 con 0x7fdc90004530 2024-02-10T08:46:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc90004d30 con 0x7fdc90004530 2024-02-10T08:46:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:41.987 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdca1775700 10 client.54101.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 4 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 0 2 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 1 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 4 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 3 2024-02-10T08:46:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.942+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc9000c000 con 0x7fdc90004150 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc90004150 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc9000c000 con 0x7fdc90004150 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 10 client.54101.objecter _finish_command 3 = system:0 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 4 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.943+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 1 3 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 4 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 4 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003fe0 osd=1 3 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 1 4 2024-02-10T08:46:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 4 2024-02-10T08:46:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 4 2024-02-10T08:46:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.958+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc900018b0 con 0x7fdc90004150 2024-02-10T08:46:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc900018b0 con 0x7fdc90004150 2024-02-10T08:46:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 10 client.54101.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 4 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 1 4 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 5 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 5 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 2 5 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 4 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 5 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc9000de90 con 0x7fdc9000d2f0 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.959+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc9000d2f0 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.960+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc9000de90 con 0x7fdc9000d2f0 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.960+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.960+0000 7fdca1775700 10 client.54101.objecter _finish_command 5 = system:0 2024-02-10T08:46:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.960+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 4 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.960+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 2 5 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 6 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 6 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003630 osd=2 3 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 2 6 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 4 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 6 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.974+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc900030a0 con 0x7fdc9000d2f0 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc900030a0 con 0x7fdc9000d2f0 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 10 client.54101.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:41.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 4 2024-02-10T08:46:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 2 6 2024-02-10T08:46:41.995 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:41.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 7 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 7 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 3 7 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 4 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 7 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90015a90 con 0x7fdc90011e80 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.976+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc90011e80 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.977+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc90015a90 con 0x7fdc90011e80 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.977+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.977+0000 7fdca1775700 10 client.54101.objecter _finish_command 7 = system:0 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.977+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 4 2024-02-10T08:46:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.977+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 3 7 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 8 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 8 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90003480 osd=3 3 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 3 8 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 4 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 8 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.991+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc9000e270 con 0x7fdc90011e80 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc9000e270 con 0x7fdc90011e80 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 10 client.54101.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 4 2024-02-10T08:46:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 3 8 2024-02-10T08:46:41.998 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 9 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 9 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 4 9 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 4 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 9 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc9000e1e0 con 0x7fdc9000ce50 2024-02-10T08:46:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.993+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc9000ce50 2024-02-10T08:46:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.994+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc9000e1e0 con 0x7fdc9000ce50 2024-02-10T08:46:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.994+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:46:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.994+0000 7fdca1775700 10 client.54101.objecter _finish_command 9 = system:0 2024-02-10T08:46:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.994+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 4 2024-02-10T08:46:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:41.994+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 4 9 2024-02-10T08:46:42.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:42.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 10 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 10 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc9000e030 osd=4 3 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 4 10 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 4 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 10 2024-02-10T08:46:42.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.009+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc9000e1e0 con 0x7fdc9000ce50 2024-02-10T08:46:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc9000e1e0 con 0x7fdc9000ce50 2024-02-10T08:46:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 10 client.54101.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 4 2024-02-10T08:46:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 4 10 2024-02-10T08:46:42.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:46:42.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:42.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 11 2024-02-10T08:46:42.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 11 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 5 11 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 4 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 11 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90016ea0 con 0x7fdc900168e0 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.011+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc900168e0 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.012+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc90016ea0 con 0x7fdc900168e0 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.012+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.012+0000 7fdca1775700 10 client.54101.objecter _finish_command 11 = system:0 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.012+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 4 2024-02-10T08:46:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.012+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 5 11 2024-02-10T08:46:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 12 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 12 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc9000e1e0 osd=5 3 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 5 12 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 4 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 12 2024-02-10T08:46:42.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.027+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc90016ea0 con 0x7fdc900168e0 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.029+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc90016ea0 con 0x7fdc900168e0 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.029+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.029+0000 7fdca1775700 10 client.54101.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.029+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 4 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.029+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 5 12 2024-02-10T08:46:42.030 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 13 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 13 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 6 13 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 4 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 13 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90022c60 con 0x7fdc9001f090 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.030+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc9001f090 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.031+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc90022c60 con 0x7fdc9001f090 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.031+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.031+0000 7fdca1775700 10 client.54101.objecter _finish_command 13 = system:0 2024-02-10T08:46:42.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.031+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 4 2024-02-10T08:46:42.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.031+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 6 13 2024-02-10T08:46:42.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 14 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 14 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90016ea0 osd=6 3 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 6 14 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 4 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 14 2024-02-10T08:46:42.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.045+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc90022c60 con 0x7fdc9001f090 2024-02-10T08:46:42.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.047+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc90022c60 con 0x7fdc9001f090 2024-02-10T08:46:42.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.047+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:42.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.047+0000 7fdca1775700 10 client.54101.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.047+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 4 2024-02-10T08:46:42.048 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:42.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.047+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 6 14 2024-02-10T08:46:42.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:42.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 15 2024-02-10T08:46:42.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 15 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 7 15 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 4 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 15 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90023e80 con 0x7fdc90024920 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.048+0000 7fdca1775700 10 client.54101.objecter ms_handle_connect 0x7fdc90024920 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.049+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdc90023e80 con 0x7fdc90024920 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.049+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.049+0000 7fdca1775700 10 client.54101.objecter _finish_command 15 = system:0 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.049+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 4 2024-02-10T08:46:42.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.049+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 7 15 2024-02-10T08:46:42.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 10 client.54101.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign -1 16 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 20 client.54101.objecter _get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_remove -1 16 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 20 client.54101.objecter get_session s=0x7fdc90022c60 osd=7 3 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 15 client.54101.objecter _session_command_op_assign 7 16 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 4 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 10 client.54101.objecter _send_command 16 2024-02-10T08:46:42.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.063+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdc90023e80 con 0x7fdc90024920 2024-02-10T08:46:42.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdca1775700 1 -- 172.21.15.107:0/2787007467 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdc90023e80 con 0x7fdc90024920 2024-02-10T08:46:42.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdca1775700 10 client.54101.objecter ms_dispatch 0x7fdca4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:46:42.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdca1775700 10 client.54101.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdca1775700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 4 2024-02-10T08:46:42.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdca1775700 15 client.54101.objecter _session_command_op_remove 7 16 2024-02-10T08:46:42.066 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:42.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdcaa2e0700 10 librados: watch_flush enter 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdcaa2e0700 10 librados: watch_flush exit 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.0 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.065+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6801/3321596972 conn(0x7fdc90004530 legacy=0x7fdc90002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc900051a0 osd=0 1 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.1 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6805/133516126 conn(0x7fdc90004150 legacy=0x7fdc9000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003fe0 osd=1 1 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.2 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6809/3575881728 conn(0x7fdc9000d2f0 legacy=0x7fdc9000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003630 osd=2 1 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.3 2024-02-10T08:46:42.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6813/2871825691 conn(0x7fdc90011e80 legacy=0x7fdc90014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90003480 osd=3 1 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.4 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdca887d700 1 -- 172.21.15.107:0/2787007467 reap_dead start 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.195:6800/3310789251 conn(0x7fdc9000ce50 legacy=0x7fdc90015c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdca887d700 1 -- 172.21.15.107:0/2787007467 reap_dead start 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e030 osd=4 1 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.5 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.195:6804/4178321486 conn(0x7fdc900168e0 legacy=0x7fdc9001d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc9000e1e0 osd=5 1 2024-02-10T08:46:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.6 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.195:6808/1417479017 conn(0x7fdc9001f090 legacy=0x7fdc90021480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90016ea0 osd=6 1 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter close_session for osd.7 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.195:6812/1934176928 conn(0x7fdc90024920 legacy=0x7fdc90024da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter put_session s=0x7fdc90022c60 osd=7 1 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 20 client.54101.objecter shutdown clearing up homeless session... 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 10 client.54101.objecter successfully canceled tick 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6800/2634842009 conn(0x7fdc8c063cb0 legacy=0x7fdc8c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> v1:172.21.15.107:6790/0 conn(0x7fdca4103260 legacy=0x7fdca4117ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdca887d700 1 -- 172.21.15.107:0/2787007467 reap_dead start 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 shutdown_connections 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 >> 172.21.15.107:0/2787007467 conn(0x7fdca40f2540 msgr2=0x7fdca41066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 shutdown_connections 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 -- 172.21.15.107:0/2787007467 wait complete. 2024-02-10T08:46:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.066+0000 7fdcaa2e0700 1 librados: shutdown 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:46:42.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:42.157 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:42.177 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:42.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:42.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- 172.21.15.107:0/3167642920 >> v1:172.21.15.107:6790/0 conn(0x7fccd0104330 legacy=0x7fccd01067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- 172.21.15.107:0/3167642920 shutdown_connections 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- 172.21.15.107:0/3167642920 >> 172.21.15.107:0/3167642920 conn(0x7fccd00f3640 msgr2=0x7fccd00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- 172.21.15.107:0/3167642920 shutdown_connections 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- 172.21.15.107:0/3167642920 wait complete. 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 librados: starting msgr at 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 librados: starting objecter 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 Processor -- start 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 -- start start 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 librados: setting wanted keys 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.248+0000 7fccd7107700 1 librados: calling monclient init 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fccd7107700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccd01173c0 con 0x7fccd01007f0 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fccd7107700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccd01185a0 con 0x7fccd00fda50 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fccd7107700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccd01197a0 con 0x7fccd0104330 2024-02-10T08:46:42.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fccd01007f0 0x7fccd00fa140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46132/0 (socket says 172.21.15.107:46132) 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccffff700 1 -- 172.21.15.107:0/1929728581 learned_addr learned my addr 172.21.15.107:0/1929728581 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fccd4ea3700 1 --1- 172.21.15.107:0/1929728581 >> v1:172.21.15.195:6789/0 conn(0x7fccd00fda50 0x7fccd00f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55126/0 (socket says 172.21.15.107:55126) 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fccd56a4700 1 --1- 172.21.15.107:0/1929728581 >> v1:172.21.15.107:6790/0 conn(0x7fccd0104330 0x7fccd0115a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47554/0 (socket says 172.21.15.107:47554) 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter ms_handle_connect 0x7fccd01007f0 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter ms_handle_connect 0x7fccd0104330 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter ms_handle_connect 0x7fccd00fda50 2024-02-10T08:46:42.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457014061 0 0) 0x7fccd01197a0 con 0x7fccd0104330 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc00033f0 con 0x7fccd0104330 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990604778 0 0) 0x7fccd01185a0 con 0x7fccd00fda50 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd01197a0 con 0x7fccd00fda50 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675895693 0 0) 0x7fccd01173c0 con 0x7fccd01007f0 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd01185a0 con 0x7fccd01007f0 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1179340659 0 0) 0x7fccc00033f0 con 0x7fccd0104330 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccd01173c0 con 0x7fccd0104330 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fccc8003170 con 0x7fccd0104330 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743681026 0 0) 0x7fccd01197a0 con 0x7fccd00fda50 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc00033f0 con 0x7fccd00fda50 2024-02-10T08:46:42.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1133628360 0 0) 0x7fccd01185a0 con 0x7fccd01007f0 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccd01197a0 con 0x7fccd01007f0 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fccc4001840 con 0x7fccd00fda50 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fccbc0019d0 con 0x7fccd01007f0 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622468571 0 0) 0x7fccd01173c0 con 0x7fccd0104330 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 >> v1:172.21.15.195:6789/0 conn(0x7fccd00fda50 legacy=0x7fccd00f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 >> v1:172.21.15.107:6789/0 conn(0x7fccd01007f0 legacy=0x7fccd00fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.249+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd011a9a0 con 0x7fccd0104330 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd0119990 con 0x7fccd0104330 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fccc80032d0 con 0x7fccd0104330 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fccc8003b60 con 0x7fccd0104330 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fccd7107700 10 client.54107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:42.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd0119e00 con 0x7fccd0104330 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.250+0000 7fccd7107700 1 librados: init done 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd005f700 con 0x7fccd0104330 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fccc8001330 con 0x7fccd0104330 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(647..647 src has 420..647) v4 ==== 8680+0+0 (unknown 1943195712 0 0) 0x7fccc80794b0 con 0x7fccd0104330 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 10 client.54107.objecter ms_dispatch 0x7fccd0001550 osd_map(647..647 src has 420..647) v4 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 3 client.54107.objecter handle_osd_map got epochs [647,647] > 0 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 3 client.54107.objecter handle_osd_map decoding full epoch 647 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 20 client.54107.objecter dump_active .. 0 homeless 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.251+0000 7fcccdffb700 10 client.54107.objecter ms_handle_connect 0x7fccc0063e60 2024-02-10T08:46:42.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.255+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fccc804c700 con 0x7fccd0104330 2024-02-10T08:46:42.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.433+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fccd00475e0 con 0x7fccd0104330 2024-02-10T08:46:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.435+0000 7fcccdffb700 1 -- 172.21.15.107:0/1929728581 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v647) v1 ==== 81+0+49537 (unknown 2925112353 0 3161125963) 0x7fccc804ab70 con 0x7fccd0104330 2024-02-10T08:46:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:46:42.436 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 647, 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:46:42.124199+0000", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:46:28.181193+0000", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:46:14.308228+0000", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5799936, 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:46:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 108, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 239, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:46:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "552", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:46:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:46:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.454 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:46:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:46:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:46:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:46:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.476 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:46:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:46:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 234, 2024-02-10T08:46:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-101", 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:45:44.375681+0000", 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2024-02-10T08:46:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:46:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "645", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "644", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "644", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "644", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 5, 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 643, 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi107-112490-101", 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 239, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-110", 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:46:41.383990+0000", 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 32769, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,creating", 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:46:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:46:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "647", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:46:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:46:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:46:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:46:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:46:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.9418182373046875, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:46:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:46:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 31, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 479, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 633, 2024-02-10T08:46:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 480, 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/3310789251", 2024-02-10T08:46:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/3310789251", 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/3310789251", 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/3310789251", 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:46:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 646, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:46:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 646, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 637, 2024-02-10T08:46:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:46:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:46:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:34.131796+0000", 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:50.048446+0000", 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:41.128121+0000", 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:40.110324+0000", 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:27.908579+0000", 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 479 2024-02-10T08:46:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:29.056895+0000", 2024-02-10T08:46:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:45:17.814100+0000", 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:46:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:31.350970+0000", 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:46:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.12", 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: 7, 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: 1 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.1d", 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: 2 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:46:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "6.1", 2024-02-10T08:46:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.4", 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:46:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.b", 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:46:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.18", 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1d", 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 1, 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 2 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 3, 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 5 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.d", 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 0, 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 4 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:46:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:46:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-test-rados-api-smithi107-112490-101": { 2024-02-10T08:46:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:46:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:46:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:46:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 234, 2024-02-10T08:46:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 237, 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 238, 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:46:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:46:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:46:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:46:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:46:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:46:42.585 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:46:42.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.598025+0000 osd.7 (osd.7) 258 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.601506+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.682861+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:40.685827+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.054046+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.054852+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.499329+0000 osd.3 (osd.3) 287 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.501153+0000 osd.3 (osd.3) 288 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:46:42.587 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.531938+0000 mgr.y (mgr.14553) 551 : cluster [DBG] pgmap v976: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.549964+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.550848+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.651844+0000 osd.6 (osd.6) 266 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.654729+0000 osd.6 (osd.6) 267 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.711056+0000 mgr.y (mgr.14553) 552 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:41.925772+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.107:0/2787007467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:42.129395+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:42.137353+0000 mon.b (mon.1) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:42.138684+0000 mon.a (mon.0) 3174 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.436+0000 7fccd7107700 10 librados: watch_flush enter 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 10 librados: watch_flush exit 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 20 client.54107.objecter shutdown clearing up homeless session... 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 10 client.54107.objecter successfully canceled tick 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 >> v1:172.21.15.107:6800/2634842009 conn(0x7fccc0063e60 legacy=0x7fccc0066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 >> v1:172.21.15.107:6790/0 conn(0x7fccd0104330 legacy=0x7fccd0115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 shutdown_connections 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 >> 172.21.15.107:0/1929728581 conn(0x7fccd00f3640 msgr2=0x7fccd01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 shutdown_connections 2024-02-10T08:46:42.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 -- 172.21.15.107:0/1929728581 wait complete. 2024-02-10T08:46:42.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:42.437+0000 7fccd7107700 1 librados: shutdown 2024-02-10T08:46:42.614 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:42.631 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:42.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.598025+0000 osd.7 (osd.7) 258 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:46:42.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.601506+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.682861+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:40.685827+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.054046+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.054852+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.499329+0000 osd.3 (osd.3) 287 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.501153+0000 osd.3 (osd.3) 288 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.531938+0000 mgr.y (mgr.14553) 551 : cluster [DBG] pgmap v976: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.549964+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.550848+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.651844+0000 osd.6 (osd.6) 266 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.654729+0000 osd.6 (osd.6) 267 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.711056+0000 mgr.y (mgr.14553) 552 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:41.925772+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.107:0/2787007467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:42.129395+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:42.137353+0000 mon.b (mon.1) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:42.138684+0000 mon.a (mon.0) 3174 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.598025+0000 osd.7 (osd.7) 258 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.601506+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.682861+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:40.685827+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.054046+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.054852+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.499329+0000 osd.3 (osd.3) 287 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.501153+0000 osd.3 (osd.3) 288 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.531938+0000 mgr.y (mgr.14553) 551 : cluster [DBG] pgmap v976: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:42.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.549964+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.550848+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.651844+0000 osd.6 (osd.6) 266 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.654729+0000 osd.6 (osd.6) 267 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.711056+0000 mgr.y (mgr.14553) 552 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:41.925772+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.107:0/2787007467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:42.129395+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:42.137353+0000 mon.b (mon.1) 746 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:42.138684+0000 mon.a (mon.0) 3174 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:42.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:42.760 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:42.779 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:42.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:42.908 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:42.929 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:42.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:43.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:43.056 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:43.059 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:43.077 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:43.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:43.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.666573+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:41.670338+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:42.007438+0000 osd.0 (osd.0) 101 : cluster [DBG] 16.18 deep-scrub starts 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:42.008751+0000 osd.0 (osd.0) 102 : cluster [DBG] 16.18 deep-scrub ok 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:42.434711+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.107:0/1929728581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:43.133459+0000 mon.a (mon.0) 3175 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.133540+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:43.150958+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:43.151229+0000 mon.a (mon.0) 3177 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.666573+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-10T08:46:43.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:41.670338+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:42.007438+0000 osd.0 (osd.0) 101 : cluster [DBG] 16.18 deep-scrub starts 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:42.008751+0000 osd.0 (osd.0) 102 : cluster [DBG] 16.18 deep-scrub ok 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:42.434711+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.107:0/1929728581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:43.133459+0000 mon.a (mon.0) 3175 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.133540+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:43.150958+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:43.151229+0000 mon.a (mon.0) 3177 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:46:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.666573+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-10T08:46:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:41.670338+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-10T08:46:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:42.007438+0000 osd.0 (osd.0) 101 : cluster [DBG] 16.18 deep-scrub starts 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:42.008751+0000 osd.0 (osd.0) 102 : cluster [DBG] 16.18 deep-scrub ok 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:42.434711+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.107:0/1929728581' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:43.133459+0000 mon.a (mon.0) 3175 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.133540+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:43.150958+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:43.151229+0000 mon.a (mon.0) 3177 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:43.858 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:43.861 INFO:tasks.thrashosds.thrasher:j is {'epoch': 647, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:46:42.124199+0000', 'last_up_change': '2024-02-10T08:46:28.181193+0000', 'last_in_change': '2024-02-10T08:46:14.308228+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 108, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 239, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '552', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi107-112490-101', 'create_time': '2024-02-10T08:45:44.375681+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '645', 'last_force_op_resend': '644', 'last_force_op_resend_prenautilus': '644', 'last_force_op_resend_preluminous': '644', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 643, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi107-112490-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 239, 'pool_name': 'test-rados-api-smithi107-112490-110', 'create_time': '2024-02-10T08:46:41.383990+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '647', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 0.9418182373046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 479, 'up_from': 633, 'up_thru': 637, 'down_at': 480, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 3310789251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 3310789251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 3310789251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 3310789251}]}, 'public_addr': '172.21.15.195:6800/3310789251', 'cluster_addr': '172.21.15.195:6801/3310789251', 'heartbeat_back_addr': '172.21.15.195:6803/3310789251', 'heartbeat_front_addr': '172.21.15.195:6802/3310789251', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 646, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 646, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:34.131796+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:50.048446+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:41.128121+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:40.110324+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:27.908579+0000', 'dead_epoch': 479}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:29.056895+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:45:17.814100+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:31.350970+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '2.12', 'osds': [3, 7, 1]}, {'pgid': '2.13', 'osds': [4, 2, 7]}, {'pgid': '2.1d', 'osds': [3, 4, 2]}, {'pgid': '4.1', 'osds': [4, 0, 7]}, {'pgid': '6.1', 'osds': [2, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}, {'from': 3, 'to': 5}]}, {'pgid': '16.d', 'mappings': [{'from': 0, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi107-112490-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:46:43.861 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1d 2024-02-10T08:46:43.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd rm-pg-upmap-items 4.1d 2024-02-10T08:46:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:43.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:43.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:43.993 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:44.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:44.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:44.270 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:44.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:44.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:44.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:44.401 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.467641+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.468473+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.501626+0000 osd.3 (osd.3) 289 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.502408+0000 osd.3 (osd.3) 290 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.532496+0000 mgr.y (mgr.14553) 553 : cluster [DBG] pgmap v979: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.802968+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.805857+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:46:44.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.036889+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.037134+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:44.132170+0000 mon.a (mon.0) 3178 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.132243+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:44.133763+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:44.134077+0000 mon.a (mon.0) 3180 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.467641+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.468473+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.501626+0000 osd.3 (osd.3) 289 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.502408+0000 osd.3 (osd.3) 290 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.532496+0000 mgr.y (mgr.14553) 553 : cluster [DBG] pgmap v979: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.802968+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:46:44.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.805857+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.036889+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.037134+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:44.132170+0000 mon.a (mon.0) 3178 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.132243+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:44.133763+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:44.134077+0000 mon.a (mon.0) 3180 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.467641+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:46:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.468473+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:46:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.501626+0000 osd.3 (osd.3) 289 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.502408+0000 osd.3 (osd.3) 290 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.532496+0000 mgr.y (mgr.14553) 553 : cluster [DBG] pgmap v979: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.802968+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.805857+0000 osd.4 (osd.4) 14 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.036889+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.037134+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:44.132170+0000 mon.a (mon.0) 3178 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.132243+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:44.133763+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:44.134077+0000 mon.a (mon.0) 3180 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:44.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:46:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:45.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:45.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:45.290 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:45.392 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:45.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:45.422 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:46:45.422 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:46:45.422 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:46:45.422 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:46:45.423 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:46:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:45.435 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:45.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:45.527 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:45.675 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:45.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.669592+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.c scrub starts 2024-02-10T08:46:45.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.672464+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.c scrub ok 2024-02-10T08:46:45.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.845407+0000 osd.5 (osd.5) 270 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:46:45.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:43.848893+0000 osd.5 (osd.5) 271 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:46:45.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.046720+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.049629+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.527442+0000 osd.3 (osd.3) 291 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:44.529224+0000 osd.3 (osd.3) 292 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:44.952969+0000 mon.c (mon.2) 454 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.134190+0000 mon.a (mon.0) 3181 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.134259+0000 mon.a (mon.0) 3182 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.138410+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.138789+0000 mon.a (mon.0) 3183 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.262054+0000 mon.c (mon.2) 455 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.262658+0000 mon.c (mon.2) 456 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:46:45.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.267282+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:45] "GET /metrics HTTP/1.1" 200 189917 "" "Prometheus/2.33.4" 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.669592+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.c scrub starts 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.672464+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.c scrub ok 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.845407+0000 osd.5 (osd.5) 270 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:43.848893+0000 osd.5 (osd.5) 271 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.046720+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.049629+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.527442+0000 osd.3 (osd.3) 291 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:44.529224+0000 osd.3 (osd.3) 292 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:44.952969+0000 mon.c (mon.2) 454 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:46:45.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.134190+0000 mon.a (mon.0) 3181 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.134259+0000 mon.a (mon.0) 3182 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.138410+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.138789+0000 mon.a (mon.0) 3183 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.262054+0000 mon.c (mon.2) 455 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.262658+0000 mon.c (mon.2) 456 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.267282+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:45.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:45.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.669592+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.c scrub starts 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.672464+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.c scrub ok 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.845407+0000 osd.5 (osd.5) 270 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:43.848893+0000 osd.5 (osd.5) 271 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.046720+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.049629+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.527442+0000 osd.3 (osd.3) 291 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:44.529224+0000 osd.3 (osd.3) 292 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:44.952969+0000 mon.c (mon.2) 454 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.134190+0000 mon.a (mon.0) 3181 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.134259+0000 mon.a (mon.0) 3182 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-10T08:46:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.138410+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.138789+0000 mon.a (mon.0) 3183 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]: dispatch 2024-02-10T08:46:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.262054+0000 mon.c (mon.2) 455 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:46:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.262658+0000 mon.c (mon.2) 456 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:46:45.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.267282+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:45.810 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:45.829 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:45.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:45.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:45.960 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:45.978 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:45.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:46.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:46.128 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:46.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:46.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:46.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:46.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:46.700 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:46.846 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:46.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.002742+0000 osd.0 (osd.0) 105 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:46:46.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.003804+0000 osd.0 (osd.0) 106 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:46:46.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.532946+0000 mgr.y (mgr.14553) 554 : cluster [DBG] pgmap v982: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:46.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.592834+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:46:46.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.596427+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:45.607497+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.855828+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.e scrub starts 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:45.859389+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.e scrub ok 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.124342+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:46.129228+0000 mon.a (mon.0) 3187 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]': finished 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.129296+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:46.237963+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:46.238346+0000 mon.a (mon.0) 3189 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.002742+0000 osd.0 (osd.0) 105 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:46:46.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.003804+0000 osd.0 (osd.0) 106 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.532946+0000 mgr.y (mgr.14553) 554 : cluster [DBG] pgmap v982: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.592834+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.596427+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:45.607497+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.855828+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.e scrub starts 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:45.859389+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.e scrub ok 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.124342+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:46.129228+0000 mon.a (mon.0) 3187 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]': finished 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.129296+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:46.237963+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:46.238346+0000 mon.a (mon.0) 3189 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:46.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:46.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:46.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.002742+0000 osd.0 (osd.0) 105 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:46:46.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.003804+0000 osd.0 (osd.0) 106 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:46:46.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.532946+0000 mgr.y (mgr.14553) 554 : cluster [DBG] pgmap v982: 194 pgs: 2 creating+peering, 30 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:46.981 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.592834+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.596427+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:45.607497+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.855828+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.e scrub starts 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:45.859389+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.e scrub ok 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.124342+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:46.129228+0000 mon.a (mon.0) 3187 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-110", "mode": "writeback"}]': finished 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.129296+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:46.237963+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:46.238346+0000 mon.a (mon.0) 3189 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:46.983 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.005 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:47.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:47.141 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.161 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:47.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:47.293 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.391 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:47.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:47.526 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.547 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:47.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:47.681 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.701 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.045933+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:46:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.046773+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:46:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.481745+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:46:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.484651+0000 osd.7 (osd.7) 263 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:46:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.581147+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.d scrub starts 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.584147+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub ok 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.886270+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:46.889764+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:47.135319+0000 mon.a (mon.0) 3190 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.135385+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:47.137060+0000 mon.b (mon.1) 751 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:47.137311+0000 mon.a (mon.0) 3192 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:47.833 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:47.852 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:47.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.045933+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:46:47.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.046773+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.481745+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.484651+0000 osd.7 (osd.7) 263 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.581147+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.d scrub starts 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.584147+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub ok 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.886270+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:46.889764+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:47.135319+0000 mon.a (mon.0) 3190 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.135385+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:47.137060+0000 mon.b (mon.1) 751 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:47.137311+0000 mon.a (mon.0) 3192 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.045933+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.046773+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.481745+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.8 deep-scrub starts 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.484651+0000 osd.7 (osd.7) 263 : cluster [DBG] 6.8 deep-scrub ok 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.581147+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.d scrub starts 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.584147+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.d scrub ok 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.886270+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:46.889764+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:47.135319+0000 mon.a (mon.0) 3190 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.135385+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:47.137060+0000 mon.b (mon.1) 751 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:47.137311+0000 mon.a (mon.0) 3192 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]: dispatch 2024-02-10T08:46:47.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:48.231 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:48.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:48.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:48.366 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:48.386 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:48.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:48.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:48.521 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:48.541 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:48.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:48.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.469210+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.472083+0000 osd.2 (osd.2) 250 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.533430+0000 mgr.y (mgr.14553) 555 : cluster [DBG] pgmap v985: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.842486+0000 osd.4 (osd.4) 19 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:47.845651+0000 osd.4 (osd.4) 20 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:46:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.132902+0000 mon.a (mon.0) 3193 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:48.143654+0000 mon.a (mon.0) 3194 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.143713+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.469210+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.472083+0000 osd.2 (osd.2) 250 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.533430+0000 mgr.y (mgr.14553) 555 : cluster [DBG] pgmap v985: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.842486+0000 osd.4 (osd.4) 19 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:47.845651+0000 osd.4 (osd.4) 20 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.132902+0000 mon.a (mon.0) 3193 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:48.143654+0000 mon.a (mon.0) 3194 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.143713+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-10T08:46:48.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.469210+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.472083+0000 osd.2 (osd.2) 250 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.533430+0000 mgr.y (mgr.14553) 555 : cluster [DBG] pgmap v985: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.842486+0000 osd.4 (osd.4) 19 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:47.845651+0000 osd.4 (osd.4) 20 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.132902+0000 mon.a (mon.0) 3193 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:48.143654+0000 mon.a (mon.0) 3194 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-110"}]': finished 2024-02-10T08:46:48.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.143713+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-10T08:46:48.870 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:48.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:48.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:49.002 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:49.102 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:49.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:49.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:49.233 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:49.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.086110+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.087223+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.509938+0000 osd.3 (osd.3) 293 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.511483+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.839853+0000 osd.4 (osd.4) 21 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:48.843206+0000 osd.4 (osd.4) 22 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.143976+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.086110+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:46:49.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.087223+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:46:49.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.509938+0000 osd.3 (osd.3) 293 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:46:49.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.511483+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:46:49.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.839853+0000 osd.4 (osd.4) 21 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:46:49.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:48.843206+0000 osd.4 (osd.4) 22 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:46:49.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.143976+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.086110+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.087223+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.509938+0000 osd.3 (osd.3) 293 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.511483+0000 osd.3 (osd.3) 294 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.839853+0000 osd.4 (osd.4) 21 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:48.843206+0000 osd.4 (osd.4) 22 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:46:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.143976+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-10T08:46:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.135+0000 7f9a51301700 1 -- 172.21.15.107:0/4156110385 >> v1:172.21.15.107:6790/0 conn(0x7f9a4c0fc960 legacy=0x7f9a4c0fcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.135+0000 7f9a51301700 1 -- 172.21.15.107:0/4156110385 shutdown_connections 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.135+0000 7f9a51301700 1 -- 172.21.15.107:0/4156110385 >> 172.21.15.107:0/4156110385 conn(0x7f9a4c06c090 msgr2=0x7f9a4c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.135+0000 7f9a51301700 1 -- 172.21.15.107:0/4156110385 shutdown_connections 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.135+0000 7f9a51301700 1 -- 172.21.15.107:0/4156110385 wait complete. 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 librados: starting msgr at 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 librados: starting objecter 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 Processor -- start 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 -- start start 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 librados: setting wanted keys 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 librados: calling monclient init 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c117400 con 0x7f9a4c0fc960 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c118600 con 0x7f9a4c103320 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a51301700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a4c119800 con 0x7f9a4c0ff720 2024-02-10T08:46:50.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a4affd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9a4c0fc960 0x7f9a4c0f6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45220/0 (socket says 172.21.15.107:45220) 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a4affd700 1 -- 172.21.15.107:0/1103006242 learned_addr learned my addr 172.21.15.107:0/1103006242 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a4a7fc700 1 --1- 172.21.15.107:0/1103006242 >> v1:172.21.15.107:6790/0 conn(0x7f9a4c0ff720 0x7f9a4c0f69c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43686/0 (socket says 172.21.15.107:43686) 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter ms_handle_connect 0x7f9a4c0fc960 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter ms_handle_connect 0x7f9a4c0ff720 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter ms_handle_connect 0x7f9a4c103320 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.136+0000 7f9a33fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550033642 0 0) 0x7f9a4c117400 con 0x7f9a4c0fc960 2024-02-10T08:46:50.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a340033f0 con 0x7f9a4c0fc960 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701739745 0 0) 0x7f9a4c119800 con 0x7f9a4c0ff720 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c117400 con 0x7f9a4c0ff720 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163351067 0 0) 0x7f9a4c118600 con 0x7f9a4c103320 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c119800 con 0x7f9a4c103320 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305571035 0 0) 0x7f9a4c117400 con 0x7f9a4c0ff720 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a4c118600 con 0x7f9a4c0ff720 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a40001940 con 0x7f9a4c0ff720 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027414124 0 0) 0x7f9a340033f0 con 0x7f9a4c0fc960 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a4c117400 con 0x7f9a4c0fc960 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738946710 0 0) 0x7f9a4c119800 con 0x7f9a4c103320 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a340033f0 con 0x7f9a4c103320 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a3c002050 con 0x7f9a4c0fc960 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a44001bc0 con 0x7f9a4c103320 2024-02-10T08:46:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3566179380 0 0) 0x7f9a4c118600 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 >> v1:172.21.15.195:6789/0 conn(0x7f9a4c103320 legacy=0x7f9a4c115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 >> v1:172.21.15.107:6789/0 conn(0x7f9a4c0fc960 legacy=0x7f9a4c0f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c11aa00 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c1199f0 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a51301700 10 client.54113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c119e60 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a51301700 1 librados: init done 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9a400024e0 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.137+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a40005ba0 con 0x7f9a4c0ff720 2024-02-10T08:46:50.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.138+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a38004f20 con 0x7f9a4c0ff720 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.138+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9a40018a30 con 0x7f9a4c0ff720 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(654..654 src has 420..654) v4 ==== 8278+0+0 (unknown 2324874247 0 0) 0x7f9a40078e80 con 0x7f9a4c0ff720 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 10 client.54113.objecter ms_dispatch 0x7f9a4c001550 osd_map(654..654 src has 420..654) v4 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 3 client.54113.objecter handle_osd_map got epochs [654,654] > 0 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 3 client.54113.objecter handle_osd_map decoding full epoch 654 2024-02-10T08:46:50.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 20 client.54113.objecter dump_active .. 0 homeless 2024-02-10T08:46:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.139+0000 7f9a33fff700 10 client.54113.objecter ms_handle_connect 0x7f9a34063e60 2024-02-10T08:46:50.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.143+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9a4004c1c0 con 0x7f9a4c0ff720 2024-02-10T08:46:50.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.328+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f9a380051d0 con 0x7f9a4c0ff720 2024-02-10T08:46:50.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.341+0000 7f9a33f31700 1 -- 172.21.15.107:0/3912627127 >> v1:172.21.15.107:6790/0 conn(0x7f9a2c104330 legacy=0x7f9a2c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.341+0000 7f9a33f31700 1 -- 172.21.15.107:0/3912627127 shutdown_connections 2024-02-10T08:46:50.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.341+0000 7f9a33f31700 1 -- 172.21.15.107:0/3912627127 >> 172.21.15.107:0/3912627127 conn(0x7f9a2c0f3640 msgr2=0x7f9a2c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:50.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.341+0000 7f9a33f31700 1 -- 172.21.15.107:0/3912627127 shutdown_connections 2024-02-10T08:46:50.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.341+0000 7f9a33f31700 1 -- 172.21.15.107:0/3912627127 wait complete. 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 librados: starting msgr at 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 librados: starting objecter 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 Processor -- start 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 -- start start 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 librados: setting wanted keys 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 librados: calling monclient init 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a2c0fcfd0 con 0x7f9a2c104330 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a2c0fd1a0 con 0x7f9a2c0fda50 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a33f31700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a2c0fd390 con 0x7f9a2c1007f0 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a314cc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9a2c1007f0 0x7f9a2c114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43708/0 (socket says 172.21.15.107:43708) 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a314cc700 1 -- 172.21.15.107:0/3562774649 learned_addr learned my addr 172.21.15.107:0/3562774649 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter ms_handle_connect 0x7f9a2c1007f0 2024-02-10T08:46:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter ms_handle_connect 0x7f9a2c104330 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter ms_handle_connect 0x7f9a2c0fda50 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694919889 0 0) 0x7f9a2c0fd390 con 0x7f9a2c1007f0 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a18003030 con 0x7f9a2c1007f0 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022320286 0 0) 0x7f9a2c0fcfd0 con 0x7f9a2c104330 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c0fd390 con 0x7f9a2c104330 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709480838 0 0) 0x7f9a2c0fd1a0 con 0x7f9a2c0fda50 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.342+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c0fcfd0 con 0x7f9a2c0fda50 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887084656 0 0) 0x7f9a18003030 con 0x7f9a2c1007f0 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a2c0fd1a0 con 0x7f9a2c1007f0 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a1c001900 con 0x7f9a2c1007f0 2024-02-10T08:46:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2620252435 0 0) 0x7f9a2c0fd390 con 0x7f9a2c104330 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a18003030 con 0x7f9a2c104330 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a240017b0 con 0x7f9a2c104330 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4050908039 0 0) 0x7f9a2c0fcfd0 con 0x7f9a2c0fda50 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a2c0fd390 con 0x7f9a2c0fda50 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a280016a0 con 0x7f9a2c0fda50 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 387044396 0 0) 0x7f9a2c0fd1a0 con 0x7f9a2c1007f0 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.195:6789/0 conn(0x7f9a2c0fda50 legacy=0x7f9a2c0fc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6789/0 conn(0x7f9a2c104330 legacy=0x7f9a2c1184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c11cc00 con 0x7f9a2c1007f0 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c119c30 con 0x7f9a2c1007f0 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a33f31700 10 client.54119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c11a120 con 0x7f9a2c1007f0 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a33f31700 1 librados: init done 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9a1c002420 con 0x7f9a2c1007f0 2024-02-10T08:46:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.343+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9a1c005ae0 con 0x7f9a2c1007f0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.344+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9a10004f00 con 0x7f9a2c1007f0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.344+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9a1c001f80 con 0x7f9a2c1007f0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(655..655 src has 420..655) v4 ==== 8652+0+0 (unknown 2908142681 0 0) 0x7f9a1c078e20 con 0x7f9a2c1007f0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 osd_map(655..655 src has 420..655) v4 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 3 client.54119.objecter handle_osd_map got epochs [655,655] > 0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 3 client.54119.objecter handle_osd_map decoding full epoch 655 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 20 client.54119.objecter dump_active .. 0 homeless 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v655) v1 ==== 54+0+16 (unknown 3915482457 0 2420000192) 0x7f9a1c079200 con 0x7f9a2c1007f0 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a18063c30 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 10 client.54119.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 1 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 1 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 0 1 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 4 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 10 client.54119.objecter _send_command 1 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10004950 con 0x7f9a10004530 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.345+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a10004530 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.346+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a10004950 con 0x7f9a10004530 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.346+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:46:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.346+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 1 = system:0 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.346+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 4 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.346+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 0 1 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 10 client.54119.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 2 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 2 2024-02-10T08:46:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a100051a0 osd=0 3 2024-02-10T08:46:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 0 2 2024-02-10T08:46:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 4 2024-02-10T08:46:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 10 client.54119.objecter _send_command 2 2024-02-10T08:46:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.361+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a10004d30 con 0x7f9a10004530 2024-02-10T08:46:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a10004d30 con 0x7f9a10004530 2024-02-10T08:46:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 4 2024-02-10T08:46:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 0 2 2024-02-10T08:46:50.364 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 10 client.54119.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 1 3 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 4 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 10 client.54119.objecter _send_command 3 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.362+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a1000bf60 con 0x7f9a10003b40 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a10003b40 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a1000bf60 con 0x7f9a10003b40 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 3 = system:0 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 4 2024-02-10T08:46:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.363+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 1 3 2024-02-10T08:46:50.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 10 client.54119.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 4 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 4 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10004270 osd=1 3 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 1 4 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 4 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 10 client.54119.objecter _send_command 4 2024-02-10T08:46:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.377+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a1000d280 con 0x7f9a10003b40 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a1000d280 con 0x7f9a10003b40 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 4 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 1 4 2024-02-10T08:46:50.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 10 client.54119.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 5 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 5 2024-02-10T08:46:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 2 5 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 4 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 10 client.54119.objecter _send_command 5 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10010d30 con 0x7f9a10002070 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.379+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a10002070 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.380+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a10010d30 con 0x7f9a10002070 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.380+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.380+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 5 = system:0 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.380+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 4 2024-02-10T08:46:50.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.380+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 2 5 2024-02-10T08:46:50.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 10 client.54119.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 6 2024-02-10T08:46:50.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 6 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a100033a0 osd=2 3 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 2 6 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 4 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 10 client.54119.objecter _send_command 6 2024-02-10T08:46:50.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.394+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a10001630 con 0x7f9a10002070 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.395+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a10001630 con 0x7f9a10002070 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.395+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.395+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.395+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 4 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.395+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 2 6 2024-02-10T08:46:50.396 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 10 client.54119.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 7 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 7 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 3 7 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 4 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 10 client.54119.objecter _send_command 7 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a1000d080 con 0x7f9a10010ed0 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a10010ed0 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a1000d080 con 0x7f9a10010ed0 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 7 = system:0 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 4 2024-02-10T08:46:50.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.396+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 3 7 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 10 client.54119.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 8 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 8 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1000c8d0 osd=3 3 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 3 8 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 4 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 10 client.54119.objecter _send_command 8 2024-02-10T08:46:50.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.410+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a10012240 con 0x7f9a10010ed0 2024-02-10T08:46:50.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a10012240 con 0x7f9a10010ed0 2024-02-10T08:46:50.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 4 2024-02-10T08:46:50.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 3 8 2024-02-10T08:46:50.413 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 10 client.54119.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 9 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 9 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 4 9 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 4 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 10 client.54119.objecter _send_command 9 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.412+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a1001a2a0 con 0x7f9a100166f0 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a100166f0 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a1001a2a0 con 0x7f9a100166f0 2024-02-10T08:46:50.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:46:50.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 9 = system:0 2024-02-10T08:46:50.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 4 2024-02-10T08:46:50.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.413+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 4 9 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 10 client.54119.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 10 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 10 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10012240 osd=4 3 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 4 10 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 4 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 10 client.54119.objecter _send_command 10 2024-02-10T08:46:50.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.428+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a1001a2a0 con 0x7f9a100166f0 2024-02-10T08:46:50.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a1001a2a0 con 0x7f9a100166f0 2024-02-10T08:46:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 4 2024-02-10T08:46:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 4 10 2024-02-10T08:46:50.431 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 10 client.54119.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 11 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 11 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 5 11 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 4 2024-02-10T08:46:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 10 client.54119.objecter _send_command 11 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.430+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a1001b520 con 0x7f9a1001bfa0 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.431+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a1001bfa0 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.432+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a1001b520 con 0x7f9a1001bfa0 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.432+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.432+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 11 = system:0 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.432+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 4 2024-02-10T08:46:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.432+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 5 11 2024-02-10T08:46:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 10 client.54119.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 12 2024-02-10T08:46:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 12 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1001a2a0 osd=5 3 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 5 12 2024-02-10T08:46:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 4 2024-02-10T08:46:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 10 client.54119.objecter _send_command 12 2024-02-10T08:46:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.446+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a1001b520 con 0x7f9a1001bfa0 2024-02-10T08:46:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a1001b520 con 0x7f9a1001bfa0 2024-02-10T08:46:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 4 2024-02-10T08:46:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 5 12 2024-02-10T08:46:50.450 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:46:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 10 client.54119.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 13 2024-02-10T08:46:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 13 2024-02-10T08:46:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 6 13 2024-02-10T08:46:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 4 2024-02-10T08:46:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 10 client.54119.objecter _send_command 13 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.448+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10022c70 con 0x7f9a1001f160 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a1001f160 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a10022c70 con 0x7f9a1001f160 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 13 = system:0 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 4 2024-02-10T08:46:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.449+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 6 13 2024-02-10T08:46:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 10 client.54119.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 14 2024-02-10T08:46:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 14 2024-02-10T08:46:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a1001b520 osd=6 3 2024-02-10T08:46:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 6 14 2024-02-10T08:46:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 4 2024-02-10T08:46:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 10 client.54119.objecter _send_command 14 2024-02-10T08:46:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.463+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a10022c70 con 0x7f9a1001f160 2024-02-10T08:46:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.465+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a10022c70 con 0x7f9a1001f160 2024-02-10T08:46:50.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.465+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.465+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.465+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 4 2024-02-10T08:46:50.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.465+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 6 14 2024-02-10T08:46:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 10 client.54119.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 15 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 15 2024-02-10T08:46:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 7 15 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 4 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 10 client.54119.objecter _send_command 15 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10026fb0 con 0x7f9a100232f0 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.466+0000 7f9a22ffd700 10 client.54119.objecter ms_handle_connect 0x7f9a100232f0 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.467+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9a10026fb0 con 0x7f9a100232f0 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.467+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.467+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 15 = system:0 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.467+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 4 2024-02-10T08:46:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.467+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 7 15 2024-02-10T08:46:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 10 client.54119.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:46:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign -1 16 2024-02-10T08:46:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 20 client.54119.objecter _get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_remove -1 16 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 20 client.54119.objecter get_session s=0x7f9a10022c70 osd=7 3 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 15 client.54119.objecter _session_command_op_assign 7 16 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 4 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 10 client.54119.objecter _send_command 16 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.481+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9a10028760 con 0x7f9a100232f0 2024-02-10T08:46:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a22ffd700 1 -- 172.21.15.107:0/3562774649 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9a10028760 con 0x7f9a100232f0 2024-02-10T08:46:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a22ffd700 10 client.54119.objecter ms_dispatch 0x7f9a2c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:46:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a22ffd700 10 client.54119.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a22ffd700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 4 2024-02-10T08:46:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a22ffd700 15 client.54119.objecter _session_command_op_remove 7 16 2024-02-10T08:46:50.484 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:46:50.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 librados: watch_flush enter 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 librados: watch_flush exit 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.0 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6801/3321596972 conn(0x7f9a10004530 legacy=0x7f9a10002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100051a0 osd=0 1 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.1 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6805/133516126 conn(0x7f9a10003b40 legacy=0x7f9a1000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10004270 osd=1 1 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.2 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6809/3575881728 conn(0x7f9a10002070 legacy=0x7f9a1000f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a100033a0 osd=2 1 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.3 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6813/2871825691 conn(0x7f9a10010ed0 legacy=0x7f9a10011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1000c8d0 osd=3 1 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.4 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.195:6800/3310789251 conn(0x7f9a100166f0 legacy=0x7f9a10018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10012240 osd=4 1 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.5 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a324ce700 1 -- 172.21.15.107:0/3562774649 reap_dead start 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.195:6804/4178321486 conn(0x7f9a1001bfa0 legacy=0x7f9a1001c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a324ce700 1 -- 172.21.15.107:0/3562774649 reap_dead start 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a324ce700 1 -- 172.21.15.107:0/3562774649 reap_dead start 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001a2a0 osd=5 1 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.6 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.195:6808/1417479017 conn(0x7f9a1001f160 legacy=0x7f9a10021550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a1001b520 osd=6 1 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter close_session for osd.7 2024-02-10T08:46:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.195:6812/1934176928 conn(0x7f9a100232f0 legacy=0x7f9a10025760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter put_session s=0x7f9a10022c70 osd=7 1 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 20 client.54119.objecter shutdown clearing up homeless session... 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 10 client.54119.objecter successfully canceled tick 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9a18063c30 legacy=0x7f9a180660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.483+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> v1:172.21.15.107:6790/0 conn(0x7f9a2c1007f0 legacy=0x7f9a2c114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a324ce700 1 -- 172.21.15.107:0/3562774649 reap_dead start 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 shutdown_connections 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 >> 172.21.15.107:0/3562774649 conn(0x7f9a2c0f3640 msgr2=0x7f9a2c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 shutdown_connections 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a33f31700 1 -- 172.21.15.107:0/3562774649 wait complete. 2024-02-10T08:46:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.484+0000 7f9a33f31700 1 librados: shutdown 2024-02-10T08:46:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:50.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:50.488 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:46:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:46:50.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:50.623 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:50.644 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:50.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:50.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- 172.21.15.107:0/1348589000 >> v1:172.21.15.107:6790/0 conn(0x7fe3d00f9570 legacy=0x7fe3d0102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- 172.21.15.107:0/1348589000 shutdown_connections 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- 172.21.15.107:0/1348589000 >> 172.21.15.107:0/1348589000 conn(0x7fe3d00f2510 msgr2=0x7fe3d00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- 172.21.15.107:0/1348589000 shutdown_connections 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- 172.21.15.107:0/1348589000 wait complete. 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 librados: starting msgr at 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 librados: starting objecter 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 Processor -- start 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- start start 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 librados: setting wanted keys 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 librados: calling monclient init 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.592+0000 7fe3d840c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d0102b40 con 0x7fe3d00f89c0 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d0102d10 con 0x7fe3d0104350 2024-02-10T08:46:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d0102f00 con 0x7fe3d00f9570 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d61a8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe3d00f89c0 0x7fe3d01023a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45260/0 (socket says 172.21.15.107:45260) 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d61a8700 1 -- 172.21.15.107:0/590994275 learned_addr learned my addr 172.21.15.107:0/590994275 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter ms_handle_connect 0x7fe3d00f89c0 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter ms_handle_connect 0x7fe3d00f9570 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428265801 0 0) 0x7fe3d0102b40 con 0x7fe3d00f89c0 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc003030 con 0x7fe3d00f89c0 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter ms_handle_connect 0x7fe3d0104350 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033747170 0 0) 0x7fe3d0102f00 con 0x7fe3d00f9570 2024-02-10T08:46:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d0102b40 con 0x7fe3d00f9570 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865217912 0 0) 0x7fe3d0102d10 con 0x7fe3d0104350 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d0102f00 con 0x7fe3d0104350 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591819102 0 0) 0x7fe3bc003030 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3d0102d10 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3c0001940 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2002443697 0 0) 0x7fe3d0102b40 con 0x7fe3d00f9570 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3bc003030 con 0x7fe3d00f9570 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3cc000fe0 con 0x7fe3d00f9570 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3382372741 0 0) 0x7fe3d0102d10 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 >> v1:172.21.15.107:6790/0 conn(0x7fe3d00f9570 legacy=0x7fe3d0114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 >> v1:172.21.15.195:6789/0 conn(0x7fe3d0104350 legacy=0x7fe3d01184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d011cc20 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d0119c50 con 0x7fe3d00f89c0 2024-02-10T08:46:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 10 client.28548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d011a0c0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.593+0000 7fe3d840c700 1 librados: init done 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.594+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe3c0001cd0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.594+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe3c0005bc0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.594+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4004f20 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.595+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe3c0002ba0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.595+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(655..655 src has 420..655) v4 ==== 8652+0+0 (unknown 2908142681 0 0) 0x7fe3c0078e00 con 0x7fe3d00f89c0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.595+0000 7fe3c77fe700 10 client.28548.objecter ms_dispatch 0x7fe3d0001550 osd_map(655..655 src has 420..655) v4 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.596+0000 7fe3c77fe700 3 client.28548.objecter handle_osd_map got epochs [655,655] > 0 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.596+0000 7fe3c77fe700 3 client.28548.objecter handle_osd_map decoding full epoch 655 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.596+0000 7fe3c77fe700 20 client.28548.objecter dump_active .. 0 homeless 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.596+0000 7fe3c77fe700 10 client.28548.objecter ms_handle_connect 0x7fe3bc063990 2024-02-10T08:46:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.600+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe3c004bfc0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.512462+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.515518+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.533952+0000 mgr.y (mgr.14553) 556 : cluster [DBG] pgmap v988: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.579122+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.a scrub starts 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.582107+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.a scrub ok 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.654068+0000 osd.6 (osd.6) 270 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.661951+0000 osd.6 (osd.6) 271 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.867278+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:49.870274+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:46:50.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:50.151356+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.159882+0000 mon.b (mon.1) 752 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.160301+0000 mon.a (mon.0) 3198 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.329068+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.107:0/1103006242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.329385+0000 mon.a (mon.0) 3199 : audit [INF] from='client.54113 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.345008+0000 mon.c (mon.2) 458 : audit [DBG] from='client.? v1:172.21.15.107:0/3562774649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.512462+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.515518+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.533952+0000 mgr.y (mgr.14553) 556 : cluster [DBG] pgmap v988: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.579122+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.a scrub starts 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.582107+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.a scrub ok 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.654068+0000 osd.6 (osd.6) 270 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:46:50.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.661951+0000 osd.6 (osd.6) 271 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.867278+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:49.870274+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:50.151356+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.159882+0000 mon.b (mon.1) 752 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.160301+0000 mon.a (mon.0) 3198 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.329068+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.107:0/1103006242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.329385+0000 mon.a (mon.0) 3199 : audit [INF] from='client.54113 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:50.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.345008+0000 mon.c (mon.2) 458 : audit [DBG] from='client.? v1:172.21.15.107:0/3562774649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:50.776 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:50.786+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} v 0) v1 -- 0x7fe3b40051d0 con 0x7fe3d00f89c0 2024-02-10T08:46:50.974 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:50.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.512462+0000 osd.3 (osd.3) 295 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:46:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.515518+0000 osd.3 (osd.3) 296 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:46:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.533952+0000 mgr.y (mgr.14553) 556 : cluster [DBG] pgmap v988: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s; 54 B/s, 1 objects/s recovering 2024-02-10T08:46:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.579122+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.a scrub starts 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.582107+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.a scrub ok 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.654068+0000 osd.6 (osd.6) 270 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.661951+0000 osd.6 (osd.6) 271 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.867278+0000 osd.4 (osd.4) 23 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:49.870274+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:50.151356+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.159882+0000 mon.b (mon.1) 752 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.160301+0000 mon.a (mon.0) 3198 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.329068+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.107:0/1103006242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.329385+0000 mon.a (mon.0) 3199 : audit [INF] from='client.54113 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:46:51.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.345008+0000 mon.c (mon.2) 458 : audit [DBG] from='client.? v1:172.21.15.107:0/3562774649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:46:51.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:51.132 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:51.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:51.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.152+0000 7fe3c77fe700 1 -- 172.21.15.107:0/590994275 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]=0 clear 4.1d pg_upmap_items mapping v656) v1 ==== 118+0+0 (unknown 2314098500 0 0) 0x7fe3c004a3d0 con 0x7fe3d00f89c0 2024-02-10T08:46:51.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.153+0000 7f9a33fff700 1 -- 172.21.15.107:0/1103006242 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v656) v1 ==== 87+0+0 (unknown 2193508308 0 0) 0x7f9a4004a5d0 con 0x7f9a4c0ff720 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 10 librados: watch_flush enter 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 10 librados: watch_flush exit 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 20 client.28548.objecter shutdown clearing up homeless session... 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 10 client.28548.objecter successfully canceled tick 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe3bc063990 legacy=0x7fe3bc065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 >> v1:172.21.15.107:6789/0 conn(0x7fe3d00f89c0 legacy=0x7fe3d01023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 shutdown_connections 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 >> 172.21.15.107:0/590994275 conn(0x7fe3d00f2510 msgr2=0x7fe3d01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 shutdown_connections 2024-02-10T08:46:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 -- 172.21.15.107:0/590994275 wait complete. 2024-02-10T08:46:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.155+0000 7fe3d840c700 1 librados: shutdown 2024-02-10T08:46:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 10 librados: watch_flush enter 2024-02-10T08:46:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 10 librados: watch_flush exit 2024-02-10T08:46:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 20 client.54113.objecter shutdown clearing up homeless session... 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 10 client.54113.objecter successfully canceled tick 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9a34063e60 legacy=0x7f9a34066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 >> v1:172.21.15.107:6790/0 conn(0x7f9a4c0ff720 legacy=0x7f9a4c0f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 shutdown_connections 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 >> 172.21.15.107:0/1103006242 conn(0x7f9a4c06c090 msgr2=0x7f9a4c06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 shutdown_connections 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 -- 172.21.15.107:0/1103006242 wait complete. 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:51.156+0000 7f9a51301700 1 librados: shutdown 2024-02-10T08:46:51.158 INFO:teuthology.orchestra.run.smithi107.stderr:clear 4.1d pg_upmap_items mapping 2024-02-10T08:46:51.159 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:46:51.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:51.264 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:51.283 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:51.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:51.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:51.413 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:51.432 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:51.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:51.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:51.561 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:51.580 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:51.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:51.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:51.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:50.493190+0000 osd.7 (osd.7) 264 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:46:51.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:50.496085+0000 osd.7 (osd.7) 265 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:46:51.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:50.787273+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]: dispatch 2024-02-10T08:46:51.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:51.148182+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.153217+0000 mon.a (mon.0) 3202 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.153275+0000 mon.a (mon.0) 3203 : audit [INF] from='client.54113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.153309+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:51.153343+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.215158+0000 mon.b (mon.1) 753 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.215521+0000 mon.a (mon.0) 3206 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:50.493190+0000 osd.7 (osd.7) 264 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:50.496085+0000 osd.7 (osd.7) 265 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:50.787273+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]: dispatch 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:51.148182+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:51.710 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.153217+0000 mon.a (mon.0) 3202 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:51.711 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.153275+0000 mon.a (mon.0) 3203 : audit [INF] from='client.54113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:51.711 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.153309+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-02-10T08:46:51.711 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:51.153343+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-10T08:46:51.711 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.215158+0000 mon.b (mon.1) 753 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:51.711 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.215521+0000 mon.a (mon.0) 3206 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:51.711 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:50.493190+0000 osd.7 (osd.7) 264 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:50.496085+0000 osd.7 (osd.7) 265 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:50.787273+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]: dispatch 2024-02-10T08:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:51.148182+0000 mon.a (mon.0) 3201 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:46:51.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.153217+0000 mon.a (mon.0) 3202 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.153275+0000 mon.a (mon.0) 3203 : audit [INF] from='client.54113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.153309+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? v1:172.21.15.107:0/590994275' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-02-10T08:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:51.153343+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-10T08:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.215158+0000 mon.b (mon.1) 753 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:51.715 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.215521+0000 mon.a (mon.0) 3206 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:46:52.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:46:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:46:52.260 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:52.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:52.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:46:52.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:52.395 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:51.534602+0000 mgr.y (mgr.14553) 557 : cluster [DBG] pgmap v991: 194 pgs: 6 creating+activating, 18 creating+peering, 3 unknown, 167 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:51.714366+0000 mgr.y (mgr.14553) 558 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:51.821777+0000 osd.3 (osd.3) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:51.821952+0000 osd.3 (osd.3) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:52.158960+0000 mon.a (mon.0) 3207 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:52.159035+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:52.159606+0000 mon.b (mon.1) 754 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:52.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:52.160097+0000 mon.a (mon.0) 3209 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:51.534602+0000 mgr.y (mgr.14553) 557 : cluster [DBG] pgmap v991: 194 pgs: 6 creating+activating, 18 creating+peering, 3 unknown, 167 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:51.714366+0000 mgr.y (mgr.14553) 558 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:51.821777+0000 osd.3 (osd.3) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:51.821952+0000 osd.3 (osd.3) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:52.158960+0000 mon.a (mon.0) 3207 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:52.159035+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:52.159606+0000 mon.b (mon.1) 754 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:52.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:52.160097+0000 mon.a (mon.0) 3209 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:51.534602+0000 mgr.y (mgr.14553) 557 : cluster [DBG] pgmap v991: 194 pgs: 6 creating+activating, 18 creating+peering, 3 unknown, 167 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:51.714366+0000 mgr.y (mgr.14553) 558 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:51.821777+0000 osd.3 (osd.3) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:51.821952+0000 osd.3 (osd.3) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:52.158960+0000 mon.a (mon.0) 3207 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:52.159035+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:52.159606+0000 mon.b (mon.1) 754 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:53.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:52.160097+0000 mon.a (mon.0) 3209 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:53.243 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:53.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:53.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:53.380 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:53.401 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:53.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:53.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:46:53.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:53.549 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:53.569 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:53.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:53.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:53.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:52.281894+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:53.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:52.282082+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:53.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:53.159720+0000 mon.a (mon.0) 3210 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:53.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:53.159799+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-10T08:46:53.704 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:53.163133+0000 mon.b (mon.1) 755 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.704 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:53.163391+0000 mon.a (mon.0) 3212 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.705 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:53.727 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:53.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:53.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:53.861 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:53.881 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:53.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:52.281894+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:52.282082+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:53.159720+0000 mon.a (mon.0) 3210 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:53.159799+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-10T08:46:53.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:53.163133+0000 mon.b (mon.1) 755 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:53.163391+0000 mon.a (mon.0) 3212 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:52.281894+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:52.282082+0000 osd.5 (osd.5) 273 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:53.159720+0000 mon.a (mon.0) 3210 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:53.159799+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:53.163133+0000 mon.b (mon.1) 755 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:53.163391+0000 mon.a (mon.0) 3212 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]: dispatch 2024-02-10T08:46:53.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:54.015 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:54.035 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:54.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:54.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:54.168 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:54.268 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:54.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:54.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:54.399 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:54.418 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:54.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:54.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:54.549 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:54.567 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:54.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:54.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:54.702 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:52.941145+0000 osd.2 (osd.2) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:52.941338+0000 osd.2 (osd.2) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:53.534948+0000 mgr.y (mgr.14553) 559 : cluster [DBG] pgmap v994: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:54.157174+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:54.165130+0000 mon.a (mon.0) 3214 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]': finished 2024-02-10T08:46:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:54.165209+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-10T08:46:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:54.210893+0000 mon.b (mon.1) 756 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:54.211266+0000 mon.a (mon.0) 3216 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:54.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:54.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:52.941145+0000 osd.2 (osd.2) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:52.941338+0000 osd.2 (osd.2) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:53.534948+0000 mgr.y (mgr.14553) 559 : cluster [DBG] pgmap v994: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:54.157174+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:54.165130+0000 mon.a (mon.0) 3214 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]': finished 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:54.165209+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:54.210893+0000 mon.b (mon.1) 756 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:54.211266+0000 mon.a (mon.0) 3216 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:52.941145+0000 osd.2 (osd.2) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:52.941338+0000 osd.2 (osd.2) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:53.534948+0000 mgr.y (mgr.14553) 559 : cluster [DBG] pgmap v994: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:54.157174+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:54.165130+0000 mon.a (mon.0) 3214 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-112", "mode": "writeback"}]': finished 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:54.165209+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:54.210893+0000 mon.b (mon.1) 756 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:54.211266+0000 mon.a (mon.0) 3216 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:46:54.956 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.079 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:46:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:46:55.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:55.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:46:55.211 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.229 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:55.331 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:46:55.331 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:46:55.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:55.348 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:55.361 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.379 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:55.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:46:55.511 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:46:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:46:55.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:55.696 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:55.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:55.845 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:55.862 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:55.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:55.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:46:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:46:55] "GET /metrics HTTP/1.1" 200 190057 "" "Prometheus/2.33.4" 2024-02-10T08:46:55.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:46:55.993 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:56.011 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:46:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:56.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:46:56.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:46:56.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:56.143 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:56.163 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:56.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:56.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:46:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:55.166551+0000 mon.a (mon.0) 3217 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:55.166597+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-10T08:46:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:55.168689+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:55.170583+0000 mon.a (mon.0) 3219 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:55.173652+0000 mon.a (mon.0) 3220 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:55.535471+0000 mgr.y (mgr.14553) 560 : cluster [DBG] pgmap v997: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:56.128553+0000 mon.a (mon.0) 3221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:56.133757+0000 mon.a (mon.0) 3222 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:56.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:56.133850+0000 mon.a (mon.0) 3223 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-10T08:46:56.293 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:56.312 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:46:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:56.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:46:56.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:55.166551+0000 mon.a (mon.0) 3217 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:56.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:55.166597+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:55.168689+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:55.170583+0000 mon.a (mon.0) 3219 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:55.173652+0000 mon.a (mon.0) 3220 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:55.535471+0000 mgr.y (mgr.14553) 560 : cluster [DBG] pgmap v997: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:56.128553+0000 mon.a (mon.0) 3221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:56.133757+0000 mon.a (mon.0) 3222 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:56.133850+0000 mon.a (mon.0) 3223 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:55.166551+0000 mon.a (mon.0) 3217 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:55.166597+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:55.168689+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:55.170583+0000 mon.a (mon.0) 3219 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]: dispatch 2024-02-10T08:46:56.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:55.173652+0000 mon.a (mon.0) 3220 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:46:56.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:55.535471+0000 mgr.y (mgr.14553) 560 : cluster [DBG] pgmap v997: 194 pgs: 6 creating+activating, 18 creating+peering, 170 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:46:56.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:56.128553+0000 mon.a (mon.0) 3221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:46:56.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:56.133757+0000 mon.a (mon.0) 3222 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-112"}]': finished 2024-02-10T08:46:56.352 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:56.133850+0000 mon.a (mon.0) 3223 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-10T08:46:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:56.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:56.441 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:46:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:56.742342+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:56.742396+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:57.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:57.169831+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-10T08:46:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:56.742342+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:56.742396+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:57.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:57.169831+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-10T08:46:57.684 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:46:57.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:46:57.684 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-10T08:46:57.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd out 1 2024-02-10T08:46:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:56.742342+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:46:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:56.742396+0000 osd.0 (osd.0) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:46:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:57.169831+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-10T08:46:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:57.535978+0000 mgr.y (mgr.14553) 561 : cluster [DBG] pgmap v1000: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:46:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:58.172096+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-10T08:46:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:58.181602+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:58.183032+0000 mon.a (mon.0) 3226 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:57.535978+0000 mgr.y (mgr.14553) 561 : cluster [DBG] pgmap v1000: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:46:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:58.172096+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-10T08:46:58.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:58.181602+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:58.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:46:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:58.183032+0000 mon.a (mon.0) 3226 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:58.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:57.535978+0000 mgr.y (mgr.14553) 561 : cluster [DBG] pgmap v1000: 162 pgs: 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:46:58.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:58.172096+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-10T08:46:58.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:58.181602+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:58.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:46:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:58.183032+0000 mon.a (mon.0) 3226 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:46:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:46:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:59.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:46:59.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:59.225 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:59.246 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:59.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:59.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:46:59.379 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:59.399 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:46:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:59.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:46:59.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:46:59.532 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:46:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:59.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:46:59.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:46:59.740 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:46:59.759 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:46:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:46:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:46:59.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:46:59.890 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:58.758929+0000 osd.1 (osd.1) 312 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:58.759145+0000 osd.1 (osd.1) 313 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:59.173119+0000 mon.a (mon.0) 3227 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:59.173220+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:46:59.536461+0000 mgr.y (mgr.14553) 562 : cluster [DBG] pgmap v1003: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:59.603409+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.107:0/271176526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:46:59.603707+0000 mon.a (mon.0) 3229 : audit [INF] from='client.54131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:58.758929+0000 osd.1 (osd.1) 312 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:00.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:58.759145+0000 osd.1 (osd.1) 313 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:59.173119+0000 mon.a (mon.0) 3227 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:59.173220+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-10T08:47:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:46:59.536461+0000 mgr.y (mgr.14553) 562 : cluster [DBG] pgmap v1003: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:47:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:59.603409+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.107:0/271176526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:46:59.603707+0000 mon.a (mon.0) 3229 : audit [INF] from='client.54131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:58.758929+0000 osd.1 (osd.1) 312 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:58.759145+0000 osd.1 (osd.1) 313 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:59.173119+0000 mon.a (mon.0) 3227 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:59.173220+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-10T08:47:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:46:59.536461+0000 mgr.y (mgr.14553) 562 : cluster [DBG] pgmap v1003: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T08:47:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:59.603409+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.107:0/271176526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:46:59.603707+0000 mon.a (mon.0) 3229 : audit [INF] from='client.54131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- 172.21.15.107:0/3947029477 >> v1:172.21.15.107:6790/0 conn(0x7f16b0104350 legacy=0x7f16b0106810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- 172.21.15.107:0/3947029477 shutdown_connections 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- 172.21.15.107:0/3947029477 >> 172.21.15.107:0/3947029477 conn(0x7f16b006c090 msgr2=0x7f16b006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- 172.21.15.107:0/3947029477 shutdown_connections 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- 172.21.15.107:0/3947029477 wait complete. 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 librados: starting msgr at 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 librados: starting objecter 2024-02-10T08:47:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 Processor -- start 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- start start 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 librados: setting wanted keys 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 librados: calling monclient init 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16b0117800 con 0x7f16b0100840 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16b01189a0 con 0x7f16b00fd9f0 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.875+0000 7f16b6fa4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16b0119b40 con 0x7f16b0104350 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16affff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f16b0100840 0x7f16b01129b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34046/0 (socket says 172.21.15.107:34046) 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16affff700 1 -- 172.21.15.107:0/891339988 learned_addr learned my addr 172.21.15.107:0/891339988 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16b5541700 1 --1- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6790/0 conn(0x7f16b0104350 0x7f16b01160e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51040/0 (socket says 172.21.15.107:51040) 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter ms_handle_connect 0x7f16b0100840 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter ms_handle_connect 0x7f16b0104350 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter ms_handle_connect 0x7f16b00fd9f0 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698120305 0 0) 0x7f16b0119b40 con 0x7f16b0104350 2024-02-10T08:47:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0003030 con 0x7f16b0104350 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717997086 0 0) 0x7f16b0117800 con 0x7f16b0100840 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b0119b40 con 0x7f16b0100840 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229238 0 0) 0x7f16b01189a0 con 0x7f16b00fd9f0 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b0117800 con 0x7f16b00fd9f0 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031965439 0 0) 0x7f16a0003030 con 0x7f16b0104350 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b01189a0 con 0x7f16b0104350 2024-02-10T08:47:00.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f16a80021d0 con 0x7f16b0104350 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1171190814 0 0) 0x7f16b0119b40 con 0x7f16b0100840 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16a0003030 con 0x7f16b0100840 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f169c001890 con 0x7f16b0100840 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728403926 0 0) 0x7f16b0117800 con 0x7f16b00fd9f0 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b0119b40 con 0x7f16b00fd9f0 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f16a4001860 con 0x7f16b00fd9f0 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1955342530 0 0) 0x7f16b01189a0 con 0x7f16b0104350 2024-02-10T08:47:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.195:6789/0 conn(0x7f16b00fd9f0 legacy=0x7f16b00fc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6789/0 conn(0x7f16b0100840 legacy=0x7f16b01129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b011ace0 con 0x7f16b0104350 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b0119d30 con 0x7f16b0104350 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f16a8002710 con 0x7f16b0104350 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.876+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f16a8003b70 con 0x7f16b0104350 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.877+0000 7f16b6fa4700 10 client.54137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.877+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b011a1a0 con 0x7f16b0104350 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.877+0000 7f16b6fa4700 1 librados: init done 2024-02-10T08:47:00.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.877+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f16b00fb4c0 con 0x7f16b0104350 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f16a8001bc0 con 0x7f16b0104350 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(665..665 src has 420..665) v4 ==== 8628+0+0 (unknown 915647556 0 0) 0x7f16a80794d0 con 0x7f16b0104350 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 osd_map(665..665 src has 420..665) v4 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 3 client.54137.objecter handle_osd_map got epochs [665,665] > 0 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 3 client.54137.objecter handle_osd_map decoding full epoch 665 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 20 client.54137.objecter dump_active .. 0 homeless 2024-02-10T08:47:00.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v665) v1 ==== 54+0+16 (unknown 2044155967 0 2420000192) 0x7f16a807b890 con 0x7f16b0104350 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16a0063c30 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 1 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 1 2024-02-10T08:47:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 0 1 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.878+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 4 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16b6fa4700 10 client.54137.objecter _send_command 1 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b00f9450 con 0x7f16b0118b30 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b0118b30 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b00f9450 con 0x7f16b0118b30 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 10 client.54137.objecter _finish_command 1 = system:0 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 4 2024-02-10T08:47:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.879+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 0 1 2024-02-10T08:47:00.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 2 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 2 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b005f580 osd=0 3 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 0 2 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 4 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 10 client.54137.objecter _send_command 2 2024-02-10T08:47:00.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.894+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b00475e0 con 0x7f16b0118b30 2024-02-10T08:47:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b00475e0 con 0x7f16b0118b30 2024-02-10T08:47:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 10 client.54137.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 4 2024-02-10T08:47:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 0 2 2024-02-10T08:47:00.897 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:00.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 3 2024-02-10T08:47:00.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 1 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 4 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 10 client.54137.objecter _send_command 3 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b00f5120 con 0x7f16b00f4630 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.896+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b00f4630 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.897+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16a400a570 con 0x7f16b00f4630 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.897+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.897+0000 7f16adffb700 10 client.54137.objecter _finish_command 3 = system:0 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.897+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 4 2024-02-10T08:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.897+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 1 3 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 4 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 4 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b010a070 osd=1 3 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 1 4 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 4 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 10 client.54137.objecter _send_command 4 2024-02-10T08:47:00.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.911+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0028660 con 0x7f16b00f4630 2024-02-10T08:47:00.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b0028660 con 0x7f16b00f4630 2024-02-10T08:47:00.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 10 client.54137.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 4 2024-02-10T08:47:00.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 1 4 2024-02-10T08:47:00.914 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 5 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 5 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 2 5 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 4 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 10 client.54137.objecter _send_command 5 2024-02-10T08:47:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0059dd0 con 0x7f16b0059990 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.913+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b0059990 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.914+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b0059dd0 con 0x7f16b0059990 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.914+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.914+0000 7f16adffb700 10 client.54137.objecter _finish_command 5 = system:0 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.914+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 4 2024-02-10T08:47:00.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.914+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 2 5 2024-02-10T08:47:00.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 6 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 6 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b00287d0 osd=2 3 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 2 6 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 4 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 10 client.54137.objecter _send_command 6 2024-02-10T08:47:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.928+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b00f7240 con 0x7f16b0059990 2024-02-10T08:47:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b00f7240 con 0x7f16b0059990 2024-02-10T08:47:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 10 client.54137.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 4 2024-02-10T08:47:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 2 6 2024-02-10T08:47:00.931 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:00.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 7 2024-02-10T08:47:00.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 7 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 3 7 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 4 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 10 client.54137.objecter _send_command 7 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b011fd00 con 0x7f16b0059570 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.930+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b0059570 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.931+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b011fd00 con 0x7f16b0059570 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.931+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:00.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.931+0000 7f16adffb700 10 client.54137.objecter _finish_command 7 = system:0 2024-02-10T08:47:00.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.931+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 4 2024-02-10T08:47:00.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.931+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 3 7 2024-02-10T08:47:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 8 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 8 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b0059400 osd=3 3 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 3 8 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 4 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 10 client.54137.objecter _send_command 8 2024-02-10T08:47:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.945+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b011ff40 con 0x7f16b0059570 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.946+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b011ff40 con 0x7f16b0059570 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.946+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.946+0000 7f16adffb700 10 client.54137.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.946+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 4 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.946+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 3 8 2024-02-10T08:47:00.947 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 9 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 9 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 4 9 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 4 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 10 client.54137.objecter _send_command 9 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b012e770 con 0x7f16b012ac30 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.947+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b012ac30 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.948+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b012e770 con 0x7f16b012ac30 2024-02-10T08:47:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.948+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.948+0000 7f16adffb700 10 client.54137.objecter _finish_command 9 = system:0 2024-02-10T08:47:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.948+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 4 2024-02-10T08:47:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.948+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 4 9 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 10 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 10 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b011ff40 osd=4 3 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 4 10 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 4 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 10 client.54137.objecter _send_command 10 2024-02-10T08:47:00.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.962+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b012e770 con 0x7f16b012ac30 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b012e770 con 0x7f16b012ac30 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16adffb700 10 client.54137.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 4 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 4 10 2024-02-10T08:47:00.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.964+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 11 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 11 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 5 11 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 4 2024-02-10T08:47:00.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 10 client.54137.objecter _send_command 11 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b012f9c0 con 0x7f16b0130470 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.965+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b0130470 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.966+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b012f9c0 con 0x7f16b0130470 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.966+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.966+0000 7f16adffb700 10 client.54137.objecter _finish_command 11 = system:0 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.966+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 4 2024-02-10T08:47:00.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.966+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 5 11 2024-02-10T08:47:00.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 12 2024-02-10T08:47:00.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 12 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b012e770 osd=5 3 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 5 12 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 4 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 10 client.54137.objecter _send_command 12 2024-02-10T08:47:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.980+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b012f9c0 con 0x7f16b0130470 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.981+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b012f9c0 con 0x7f16b0130470 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.981+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16adffb700 10 client.54137.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 4 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 5 12 2024-02-10T08:47:00.982 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 13 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 13 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 6 13 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 4 2024-02-10T08:47:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 10 client.54137.objecter _send_command 13 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0137140 con 0x7f16b0133630 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.982+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b0133630 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.983+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b0137140 con 0x7f16b0133630 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.983+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.983+0000 7f16adffb700 10 client.54137.objecter _finish_command 13 = system:0 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.983+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 4 2024-02-10T08:47:00.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.983+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 6 13 2024-02-10T08:47:00.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 14 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 14 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b012f9c0 osd=6 3 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 6 14 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 4 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 10 client.54137.objecter _send_command 14 2024-02-10T08:47:00.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.998+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0137140 con 0x7f16b0133630 2024-02-10T08:47:01.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b0137140 con 0x7f16b0133630 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16adffb700 10 client.54137.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 4 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 6 14 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:01.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 15 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 15 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 7 15 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 4 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 10 client.54137.objecter _send_command 15 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.000+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0139450 con 0x7f16b01377c0 2024-02-10T08:47:01.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.001+0000 7f16adffb700 10 client.54137.objecter ms_handle_connect 0x7f16b01377c0 2024-02-10T08:47:01.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.002+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f16b0139450 con 0x7f16b01377c0 2024-02-10T08:47:01.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.002+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:01.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.002+0000 7f16adffb700 10 client.54137.objecter _finish_command 15 = system:0 2024-02-10T08:47:01.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.002+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 4 2024-02-10T08:47:01.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.002+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 7 15 2024-02-10T08:47:01.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 10 client.54137.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign -1 16 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 20 client.54137.objecter _get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_remove -1 16 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 20 client.54137.objecter get_session s=0x7f16b0137140 osd=7 3 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 15 client.54137.objecter _session_command_op_assign 7 16 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 4 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 10 client.54137.objecter _send_command 16 2024-02-10T08:47:01.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.016+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0139450 con 0x7f16b01377c0 2024-02-10T08:47:01.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16adffb700 1 -- 172.21.15.107:0/891339988 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f16b0139450 con 0x7f16b01377c0 2024-02-10T08:47:01.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16adffb700 10 client.54137.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:01.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16adffb700 10 client.54137.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16adffb700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 4 2024-02-10T08:47:01.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16adffb700 15 client.54137.objecter _session_command_op_remove 7 16 2024-02-10T08:47:01.019 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:01.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 10 librados: watch_flush enter 2024-02-10T08:47:01.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 10 librados: watch_flush exit 2024-02-10T08:47:01.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.0 2024-02-10T08:47:01.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6801/3321596972 conn(0x7f16b0118b30 legacy=0x7f16b0118f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b005f580 osd=0 1 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.1 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6805/133516126 conn(0x7f16b00f4630 legacy=0x7f16b00f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b010a070 osd=1 1 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.018+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.2 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6809/3575881728 conn(0x7f16b0059990 legacy=0x7f16b01220e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b00287d0 osd=2 1 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.3 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6813/2871825691 conn(0x7f16b0059570 legacy=0x7f16b00f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0059400 osd=3 1 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.4 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.195:6800/3310789251 conn(0x7f16b012ac30 legacy=0x7f16b012d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b011ff40 osd=4 1 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.5 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.195:6804/4178321486 conn(0x7f16b0130470 legacy=0x7f16b01308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012e770 osd=5 1 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.6 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.195:6808/1417479017 conn(0x7f16b0133630 legacy=0x7f16b0135a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b012f9c0 osd=6 1 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter close_session for osd.7 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.195:6812/1934176928 conn(0x7f16b01377c0 legacy=0x7f16b0137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter put_session s=0x7f16b0137140 osd=7 1 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 20 client.54137.objecter shutdown clearing up homeless session... 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 10 client.54137.objecter successfully canceled tick 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b5541700 1 -- 172.21.15.107:0/891339988 reap_dead start 2024-02-10T08:47:01.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b5541700 1 -- 172.21.15.107:0/891339988 reap_dead start 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b5541700 1 -- 172.21.15.107:0/891339988 reap_dead start 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b5541700 1 -- 172.21.15.107:0/891339988 reap_dead start 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b5541700 1 -- 172.21.15.107:0/891339988 reap_dead start 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6800/2634842009 conn(0x7f16a0063c30 legacy=0x7f16a00660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> v1:172.21.15.107:6790/0 conn(0x7f16b0104350 legacy=0x7f16b01160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 shutdown_connections 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 >> 172.21.15.107:0/891339988 conn(0x7f16b006c090 msgr2=0x7f16b01067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 shutdown_connections 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 -- 172.21.15.107:0/891339988 wait complete. 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:01.019+0000 7f16b6fa4700 1 librados: shutdown 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.023 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.024 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.024 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.024 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:47:01.139 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:47:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:47:01.140 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:47:01.140 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:47:01.338 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:01.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:00.170868+0000 mon.a (mon.0) 3230 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:01.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:00.181201+0000 mon.a (mon.0) 3231 : audit [INF] from='client.54131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:01.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:00.181250+0000 mon.a (mon.0) 3232 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:00.197670+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:00.198093+0000 mon.a (mon.0) 3233 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:00.878990+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.107:0/891339988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:00.170868+0000 mon.a (mon.0) 3230 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:00.181201+0000 mon.a (mon.0) 3231 : audit [INF] from='client.54131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:00.181250+0000 mon.a (mon.0) 3232 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:00.197670+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:00.198093+0000 mon.a (mon.0) 3233 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.341 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:00.878990+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.107:0/891339988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:01.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:01.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:01.492 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:01.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:00.170868+0000 mon.a (mon.0) 3230 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:00.181201+0000 mon.a (mon.0) 3231 : audit [INF] from='client.54131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:00.181250+0000 mon.a (mon.0) 3232 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:00.197670+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:00.198093+0000 mon.a (mon.0) 3233 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:01.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:00.878990+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.107:0/891339988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:01.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:01.625 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:02.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:02.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:02.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:47:02.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:02.210 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:02.232 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:02.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.416+0000 7fd18b443700 1 -- 172.21.15.107:0/391265548 >> v1:172.21.15.107:6790/0 conn(0x7fd184104330 legacy=0x7fd1841067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.416+0000 7fd18b443700 1 -- 172.21.15.107:0/391265548 shutdown_connections 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.416+0000 7fd18b443700 1 -- 172.21.15.107:0/391265548 >> 172.21.15.107:0/391265548 conn(0x7fd1840f3640 msgr2=0x7fd1840f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.416+0000 7fd18b443700 1 -- 172.21.15.107:0/391265548 shutdown_connections 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.416+0000 7fd18b443700 1 -- 172.21.15.107:0/391265548 wait complete. 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 librados: starting msgr at 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 librados: starting objecter 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 Processor -- start 2024-02-10T08:47:02.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 -- start start 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 librados: setting wanted keys 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 librados: calling monclient init 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd184117540 con 0x7fd1840fda50 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd184118720 con 0x7fd1841007f0 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd18b443700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd184119920 con 0x7fd184104330 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd1891df700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1840fda50 0x7fd1840f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34026/0 (socket says 172.21.15.107:34026) 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd1891df700 1 -- 172.21.15.107:0/271176526 learned_addr learned my addr 172.21.15.107:0/271176526 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd17a7fc700 10 client.?.objecter ms_handle_connect 0x7fd184104330 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd17a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd17a7fc700 10 client.?.objecter ms_handle_connect 0x7fd1840fda50 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd17a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:02.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.417+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250724295 0 0) 0x7fd184119920 con 0x7fd184104330 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1700033f0 con 0x7fd184104330 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217905828 0 0) 0x7fd184117540 con 0x7fd1840fda50 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184119920 con 0x7fd1840fda50 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 10 client.?.objecter ms_handle_connect 0x7fd1841007f0 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067102214 0 0) 0x7fd184118720 con 0x7fd1841007f0 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184117540 con 0x7fd1841007f0 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516101756 0 0) 0x7fd1700033f0 con 0x7fd184104330 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd184118720 con 0x7fd184104330 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd17c002240 con 0x7fd184104330 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795036922 0 0) 0x7fd184119920 con 0x7fd1840fda50 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1700033f0 con 0x7fd1840fda50 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd174002870 con 0x7fd1840fda50 2024-02-10T08:47:02.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387330311 0 0) 0x7fd184117540 con 0x7fd1841007f0 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd184119920 con 0x7fd1841007f0 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd1800018c0 con 0x7fd1841007f0 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025683050 0 0) 0x7fd184118720 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 >> v1:172.21.15.195:6789/0 conn(0x7fd1841007f0 legacy=0x7fd1840fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 >> v1:172.21.15.107:6789/0 conn(0x7fd1840fda50 legacy=0x7fd1840f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18411ab20 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd184117730 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd18b443700 10 client.54131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd184117ba0 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd18b443700 1 librados: init done 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd17c0032c0 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.418+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd17c0039a0 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.419+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd168004f20 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.419+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd17c001b30 con 0x7fd184104330 2024-02-10T08:47:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(664..664 src has 420..664) v4 ==== 8628+0+0 (unknown 2998658063 0 0) 0x7fd17c076d00 con 0x7fd184104330 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 10 client.54131.objecter ms_dispatch 0x7fd184001550 osd_map(664..664 src has 420..664) v4 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 3 client.54131.objecter handle_osd_map got epochs [664,664] > 0 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 3 client.54131.objecter handle_osd_map decoding full epoch 664 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 20 client.54131.objecter dump_active .. 0 homeless 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.420+0000 7fd17a7fc700 10 client.54131.objecter ms_handle_connect 0x7fd170063cf0 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.424+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd17c049f40 con 0x7fd184104330 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:46:59.602+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd1680051d0 con 0x7fd184104330 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.181+0000 7fd17a7fc700 1 -- 172.21.15.107:0/271176526 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v665) v1 ==== 97+0+0 (unknown 1004973046 0 0) 0x7fd17c002460 con 0x7fd184104330 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 10 librados: watch_flush enter 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 10 librados: watch_flush exit 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 20 client.54131.objecter shutdown clearing up homeless session... 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 10 client.54131.objecter successfully canceled tick 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd170063cf0 legacy=0x7fd1700661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 >> v1:172.21.15.107:6790/0 conn(0x7fd184104330 legacy=0x7fd184115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 shutdown_connections 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 >> 172.21.15.107:0/271176526 conn(0x7fd1840f3640 msgr2=0x7fd1841077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.184+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 shutdown_connections 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.185+0000 7fd18b443700 1 -- 172.21.15.107:0/271176526 wait complete. 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:00.185+0000 7fd18b443700 1 librados: shutdown 2024-02-10T08:47:02.268 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:47:02.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:02.365 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:02.384 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:02.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:01.240006+0000 mon.a (mon.0) 3234 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:01.240080+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:01.241607+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:01.241862+0000 mon.a (mon.0) 3236 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:01.537067+0000 mgr.y (mgr.14553) 563 : cluster [DBG] pgmap v1006: 194 pgs: 6 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:01.725192+0000 mgr.y (mgr.14553) 564 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:01.240006+0000 mon.a (mon.0) 3234 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:01.240080+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:01.241607+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:01.241862+0000 mon.a (mon.0) 3236 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:01.537067+0000 mgr.y (mgr.14553) 563 : cluster [DBG] pgmap v1006: 194 pgs: 6 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:47:02.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:01.725192+0000 mgr.y (mgr.14553) 564 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:02.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:01.240006+0000 mon.a (mon.0) 3234 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:01.240080+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:01.241607+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:01.241862+0000 mon.a (mon.0) 3236 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:01.537067+0000 mgr.y (mgr.14553) 563 : cluster [DBG] pgmap v1006: 194 pgs: 6 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T08:47:02.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:01.725192+0000 mgr.y (mgr.14553) 564 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:02.516 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:02.535 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:02.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:02.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:02.991 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:03.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:02.243490+0000 mon.a (mon.0) 3237 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:03.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:02.243554+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-10T08:47:03.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:02.244125+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:02.244348+0000 mon.a (mon.0) 3239 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:02.243490+0000 mon.a (mon.0) 3237 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:03.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:02.243554+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-10T08:47:03.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:02.244125+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:02.244348+0000 mon.a (mon.0) 3239 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.514 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:03.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:02.243490+0000 mon.a (mon.0) 3237 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:02.243554+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-10T08:47:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:02.244125+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:02.244348+0000 mon.a (mon.0) 3239 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]: dispatch 2024-02-10T08:47:03.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:03.650 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:03.671 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:03.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:03.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:03.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:03.802 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:03.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:03.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:03.950 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:03.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:04.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:04.104 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:04.126 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:04.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:04.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:04.285 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:04.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:04.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:03.237430+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:03.248766+0000 mon.a (mon.0) 3241 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]': finished 2024-02-10T08:47:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:03.248894+0000 mon.a (mon.0) 3242 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-10T08:47:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:03.537486+0000 mgr.y (mgr.14553) 565 : cluster [DBG] pgmap v1009: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:47:04.418 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:04.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:04.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:03.237430+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:04.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:03.248766+0000 mon.a (mon.0) 3241 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]': finished 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:03.248894+0000 mon.a (mon.0) 3242 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:03.537486+0000 mgr.y (mgr.14553) 565 : cluster [DBG] pgmap v1009: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:03.237430+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:03.248766+0000 mon.a (mon.0) 3241 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-114", "mode": "writeback"}]': finished 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:03.248894+0000 mon.a (mon.0) 3242 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-10T08:47:04.514 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:03.537486+0000 mgr.y (mgr.14553) 565 : cluster [DBG] pgmap v1009: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T08:47:04.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:04.569 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:04.587 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:04.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:04.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:04.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:47:05.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:47:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:04.265783+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-10T08:47:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:04.295258+0000 mon.b (mon.1) 762 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:04.295537+0000 mon.a (mon.0) 3244 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:04.265783+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-10T08:47:05.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:04.295258+0000 mon.b (mon.1) 762 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:04.295537+0000 mon.a (mon.0) 3244 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:04.265783+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-10T08:47:05.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:04.295258+0000 mon.b (mon.1) 762 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:04.295537+0000 mon.a (mon.0) 3244 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:05.824 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:05] "GET /metrics HTTP/1.1" 200 190057 "" "Prometheus/2.33.4" 2024-02-10T08:47:06.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 -- 172.21.15.107:0/188030160 >> v1:172.21.15.107:6789/0 conn(0x7fcd2c0fc910 legacy=0x7fcd2c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:06.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 -- 172.21.15.107:0/188030160 shutdown_connections 2024-02-10T08:47:06.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 -- 172.21.15.107:0/188030160 >> 172.21.15.107:0/188030160 conn(0x7fcd2c0f2510 msgr2=0x7fcd2c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:06.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 -- 172.21.15.107:0/188030160 shutdown_connections 2024-02-10T08:47:06.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 -- 172.21.15.107:0/188030160 wait complete. 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 librados: starting msgr at 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 librados: starting objecter 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.406+0000 7fcd3350a700 1 Processor -- start 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 -- start start 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 librados: setting wanted keys 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 librados: calling monclient init 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd2c116120 con 0x7fcd2c0ff710 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd2c1172c0 con 0x7fcd2c0fc910 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd3350a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd2c1184c0 con 0x7fcd2c103250 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd31aa7700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcd2c103250 0x7fcd2c1148d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51050/0 (socket says 172.21.15.107:51050) 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd31aa7700 1 -- 172.21.15.107:0/2304530045 learned_addr learned my addr 172.21.15.107:0/2304530045 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter ms_handle_connect 0x7fcd2c103250 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:06.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter ms_handle_connect 0x7fcd2c0ff710 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter ms_handle_connect 0x7fcd2c0fc910 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020031829 0 0) 0x7fcd2c1184c0 con 0x7fcd2c103250 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd18003030 con 0x7fcd2c103250 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364513557 0 0) 0x7fcd2c116120 con 0x7fcd2c0ff710 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd2c1184c0 con 0x7fcd2c0ff710 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527384818 0 0) 0x7fcd2c1172c0 con 0x7fcd2c0fc910 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd2c116120 con 0x7fcd2c0fc910 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660399420 0 0) 0x7fcd18003030 con 0x7fcd2c103250 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd2c1172c0 con 0x7fcd2c103250 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd24001cd0 con 0x7fcd2c103250 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929929835 0 0) 0x7fcd2c1184c0 con 0x7fcd2c0ff710 2024-02-10T08:47:06.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd18003030 con 0x7fcd2c0ff710 2024-02-10T08:47:06.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.407+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd1c001830 con 0x7fcd2c0ff710 2024-02-10T08:47:06.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481992995 0 0) 0x7fcd2c116120 con 0x7fcd2c0fc910 2024-02-10T08:47:06.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd2c1184c0 con 0x7fcd2c0fc910 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd280018d0 con 0x7fcd2c0fc910 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1020588879 0 0) 0x7fcd2c1172c0 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 >> v1:172.21.15.195:6789/0 conn(0x7fcd2c0fc910 legacy=0x7fcd2c0f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 >> v1:172.21.15.107:6789/0 conn(0x7fcd2c0ff710 legacy=0x7fcd2c0f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c1196c0 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c116310 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd3350a700 10 client.54143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c116800 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd3350a700 1 librados: init done 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fcd24002060 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.408+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd24005fd0 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd10004f20 con 0x7fcd2c103250 2024-02-10T08:47:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fcd240025f0 con 0x7fcd2c103250 2024-02-10T08:47:06.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(671..671 src has 420..671) v4 ==== 8671+0+0 (unknown 2411724374 0 0) 0x7fcd24079320 con 0x7fcd2c103250 2024-02-10T08:47:06.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 10 client.54143.objecter ms_dispatch 0x7fcd2c001550 osd_map(671..671 src has 420..671) v4 2024-02-10T08:47:06.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 3 client.54143.objecter handle_osd_map got epochs [671,671] > 0 2024-02-10T08:47:06.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 3 client.54143.objecter handle_osd_map decoding full epoch 671 2024-02-10T08:47:06.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.409+0000 7fcd227fc700 20 client.54143.objecter dump_active .. 0 homeless 2024-02-10T08:47:06.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.410+0000 7fcd227fc700 10 client.54143.objecter ms_handle_connect 0x7fcd18063c70 2024-02-10T08:47:06.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.413+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcd2404c4d0 con 0x7fcd2c103250 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:05.269178+0000 mon.a (mon.0) 3245 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:05.269248+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:05.273508+0000 mon.b (mon.1) 763 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:05.274095+0000 mon.a (mon.0) 3247 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:05.537954+0000 mgr.y (mgr.14553) 566 : cluster [DBG] pgmap v1012: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:06.132500+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:06.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:06.140229+0000 mon.a (mon.0) 3249 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:06.140300+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:05.269178+0000 mon.a (mon.0) 3245 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:05.269248+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:05.273508+0000 mon.b (mon.1) 763 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:05.274095+0000 mon.a (mon.0) 3247 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:05.537954+0000 mgr.y (mgr.14553) 566 : cluster [DBG] pgmap v1012: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:06.132500+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:06.140229+0000 mon.a (mon.0) 3249 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:06.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:06.140300+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:05.269178+0000 mon.a (mon.0) 3245 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:05.269248+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:05.273508+0000 mon.b (mon.1) 763 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:05.274095+0000 mon.a (mon.0) 3247 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]: dispatch 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:05.537954+0000 mgr.y (mgr.14553) 566 : cluster [DBG] pgmap v1012: 194 pgs: 194 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:06.132500+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:06.140229+0000 mon.a (mon.0) 3249 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-114"}]': finished 2024-02-10T08:47:06.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:06.140300+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-10T08:47:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:06.596+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd out", "ids": ["1"]} v 0) v1 -- 0x7fcd100051d0 con 0x7fcd2c103250 2024-02-10T08:47:06.901 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:47:06.901 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:47:06.901 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:47:06.901 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:47:06.901 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:47:06.902 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:47:06.596+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e671 definitely_dead 0 2024-02-10T08:47:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:07.130 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.162+0000 7fcd227fc700 1 -- 172.21.15.107:0/2304530045 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["1"]}]=0 marked out osd.1. v672) v1 ==== 87+0+0 (unknown 424058185 0 0) 0x7fcd2404a8e0 con 0x7fcd2c103250 2024-02-10T08:47:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.169+0000 7fcd3350a700 10 librados: watch_flush enter 2024-02-10T08:47:07.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.170+0000 7fcd3350a700 10 librados: watch_flush exit 2024-02-10T08:47:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.170+0000 7fcd3350a700 20 client.54143.objecter shutdown clearing up homeless session... 2024-02-10T08:47:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.170+0000 7fcd3350a700 10 client.54143.objecter successfully canceled tick 2024-02-10T08:47:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.170+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 >> v1:172.21.15.107:6800/2634842009 conn(0x7fcd18063c70 legacy=0x7fcd18066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.170+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 >> v1:172.21.15.107:6790/0 conn(0x7fcd2c103250 legacy=0x7fcd2c1148d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.174+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 shutdown_connections 2024-02-10T08:47:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.174+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 >> 172.21.15.107:0/2304530045 conn(0x7fcd2c0f2510 msgr2=0x7fcd2c0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:07.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.184+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 shutdown_connections 2024-02-10T08:47:07.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.185+0000 7fcd3350a700 1 -- 172.21.15.107:0/2304530045 wait complete. 2024-02-10T08:47:07.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:07.187+0000 7fcd3350a700 1 librados: shutdown 2024-02-10T08:47:07.188 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.1. 2024-02-10T08:47:07.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:06.596974+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.107:0/2304530045' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:06.597294+0000 mon.a (mon.0) 3251 : audit [INF] from='client.54143 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:06.597393+0000 mon.a (mon.0) 3252 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:07.152045+0000 mon.a (mon.0) 3253 : audit [INF] from='client.54143 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:07.152108+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:06.596974+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.107:0/2304530045' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:06.597294+0000 mon.a (mon.0) 3251 : audit [INF] from='client.54143 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:06.597393+0000 mon.a (mon.0) 3252 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:07.152045+0000 mon.a (mon.0) 3253 : audit [INF] from='client.54143 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-10T08:47:07.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:07.152108+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-10T08:47:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:07.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:07.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:07.734 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:06.596974+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.107:0/2304530045' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:06.597294+0000 mon.a (mon.0) 3251 : audit [INF] from='client.54143 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-02-10T08:47:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:06.597393+0000 mon.a (mon.0) 3252 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2024-02-10T08:47:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:07.152045+0000 mon.a (mon.0) 3253 : audit [INF] from='client.54143 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-02-10T08:47:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:07.152108+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-10T08:47:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:07.538510+0000 mgr.y (mgr.14553) 567 : cluster [DBG] pgmap v1015: 162 pgs: 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:08.150924+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-10T08:47:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:08.165657+0000 mon.b (mon.1) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:08.178542+0000 mon.a (mon.0) 3256 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:07.538510+0000 mgr.y (mgr.14553) 567 : cluster [DBG] pgmap v1015: 162 pgs: 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:08.150924+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-10T08:47:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:08.165657+0000 mon.b (mon.1) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:08.178542+0000 mon.a (mon.0) 3256 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:07.538510+0000 mgr.y (mgr.14553) 567 : cluster [DBG] pgmap v1015: 162 pgs: 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:08.150924+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-10T08:47:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:08.165657+0000 mon.b (mon.1) 764 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:08.178542+0000 mon.a (mon.0) 3256 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:09.213 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:09.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:09.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:09.343 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:09.362 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:09.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:09.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:09.514 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:09.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:09.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:09.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 -- 172.21.15.107:0/448169381 >> v1:172.21.15.107:6790/0 conn(0x7fb6e01007f0 legacy=0x7fb6e0102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 -- 172.21.15.107:0/448169381 shutdown_connections 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 -- 172.21.15.107:0/448169381 >> 172.21.15.107:0/448169381 conn(0x7fb6e00f3640 msgr2=0x7fb6e00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 -- 172.21.15.107:0/448169381 shutdown_connections 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 -- 172.21.15.107:0/448169381 wait complete. 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 librados: starting msgr at 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 librados: starting objecter 2024-02-10T08:47:09.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.845+0000 7fb6e6259700 1 Processor -- start 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 -- start start 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 librados: setting wanted keys 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 librados: calling monclient init 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e0119550 con 0x7fb6e00fda50 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e011a730 con 0x7fb6e0104330 2024-02-10T08:47:09.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6e6259700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e011b910 con 0x7fb6e01007f0 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6df7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6e00fda50 0x7fb6e01024a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48340/0 (socket says 172.21.15.107:48340) 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6df7fe700 1 -- 172.21.15.107:0/1742658284 learned_addr learned my addr 172.21.15.107:0/1742658284 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6deffd700 1 --1- 172.21.15.107:0/1742658284 >> v1:172.21.15.107:6790/0 conn(0x7fb6e01007f0 0x7fb6e0102bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37690/0 (socket says 172.21.15.107:37690) 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter ms_handle_connect 0x7fb6e00fda50 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter ms_handle_connect 0x7fb6e01007f0 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter ms_handle_connect 0x7fb6e0104330 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441273651 0 0) 0x7fb6e011b910 con 0x7fb6e01007f0 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003800 con 0x7fb6e01007f0 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437825945 0 0) 0x7fb6e0119550 con 0x7fb6e00fda50 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e011b910 con 0x7fb6e00fda50 2024-02-10T08:47:09.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52890016 0 0) 0x7fb6e011a730 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e0119550 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3499269181 0 0) 0x7fb6e0119550 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6e011a730 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6d0001620 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 570958790 0 0) 0x7fb6e011b910 con 0x7fb6e00fda50 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6e0119550 con 0x7fb6e00fda50 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3346066176 0 0) 0x7fb6c8003800 con 0x7fb6e01007f0 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6e011b910 con 0x7fb6e01007f0 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6d8001c80 con 0x7fb6e00fda50 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6d40015b0 con 0x7fb6e01007f0 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.846+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 989621584 0 0) 0x7fb6e011a730 con 0x7fb6e0104330 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 >> v1:172.21.15.107:6790/0 conn(0x7fb6e01007f0 legacy=0x7fb6e0102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 >> v1:172.21.15.107:6789/0 conn(0x7fb6e00fda50 legacy=0x7fb6e01024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e011cb10 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e011bb00 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6e6259700 10 client.48871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e011bff0 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6e6259700 1 librados: init done 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb6d0002200 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb6d0003220 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.847+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc004f20 con 0x7fb6e0104330 2024-02-10T08:47:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb6d0001d60 con 0x7fb6e0104330 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(674..674 src has 420..674) v4 ==== 8597+0+0 (unknown 2051202249 0 0) 0x7fb6d0076510 con 0x7fb6e0104330 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 10 client.48871.objecter ms_dispatch 0x7fb6e0001550 osd_map(674..674 src has 420..674) v4 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 3 client.48871.objecter handle_osd_map got epochs [674,674] > 0 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 3 client.48871.objecter handle_osd_map decoding full epoch 674 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.848+0000 7fb6dcff9700 20 client.48871.objecter dump_active .. 0 homeless 2024-02-10T08:47:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.849+0000 7fb6dcff9700 10 client.48871.objecter ms_handle_connect 0x7fb6c8063cb0 2024-02-10T08:47:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:09.853+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb6d0049710 con 0x7fb6e0104330 2024-02-10T08:47:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.029+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fb6cc0051d0 con 0x7fb6e0104330 2024-02-10T08:47:10.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.157+0000 7fb6dcff9700 1 -- 172.21.15.107:0/1742658284 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v675) v1 ==== 91+0+0 (unknown 2943031578 0 0) 0x7fb6d0047b20 con 0x7fb6e0104330 2024-02-10T08:47:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.158+0000 7fb6e6259700 10 librados: watch_flush enter 2024-02-10T08:47:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.158+0000 7fb6e6259700 10 librados: watch_flush exit 2024-02-10T08:47:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.158+0000 7fb6e6259700 20 client.48871.objecter shutdown clearing up homeless session... 2024-02-10T08:47:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.158+0000 7fb6e6259700 10 client.48871.objecter successfully canceled tick 2024-02-10T08:47:10.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.158+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb6c8063cb0 legacy=0x7fb6c8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 >> v1:172.21.15.195:6789/0 conn(0x7fb6e0104330 legacy=0x7fb6e0117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 shutdown_connections 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 >> 172.21.15.107:0/1742658284 conn(0x7fb6e00f3640 msgr2=0x7fb6e01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 shutdown_connections 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 -- 172.21.15.107:0/1742658284 wait complete. 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:10.159+0000 7fb6e6259700 1 librados: shutdown 2024-02-10T08:47:10.160 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:09.153868+0000 mon.a (mon.0) 3257 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:09.153954+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:09.165354+0000 mon.b (mon.1) 765 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:09.168818+0000 mon.a (mon.0) 3259 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:09.538897+0000 mgr.y (mgr.14553) 568 : cluster [DBG] pgmap v1018: 194 pgs: 32 unknown, 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:10.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.030942+0000 mon.b (mon.1) 766 : audit [INF] from='client.? v1:172.21.15.107:0/1742658284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.031235+0000 mon.a (mon.0) 3260 : audit [INF] from='client.48871 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:09.153868+0000 mon.a (mon.0) 3257 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:09.153954+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:09.165354+0000 mon.b (mon.1) 765 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:09.168818+0000 mon.a (mon.0) 3259 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:09.538897+0000 mgr.y (mgr.14553) 568 : cluster [DBG] pgmap v1018: 194 pgs: 32 unknown, 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.030942+0000 mon.b (mon.1) 766 : audit [INF] from='client.? v1:172.21.15.107:0/1742658284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.031235+0000 mon.a (mon.0) 3260 : audit [INF] from='client.48871 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:09.153868+0000 mon.a (mon.0) 3257 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:09.153954+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:09.165354+0000 mon.b (mon.1) 765 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:09.168818+0000 mon.a (mon.0) 3259 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:09.538897+0000 mgr.y (mgr.14553) 568 : cluster [DBG] pgmap v1018: 194 pgs: 32 unknown, 12 peering, 2 active+clean+snaptrim, 148 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.030942+0000 mon.b (mon.1) 766 : audit [INF] from='client.? v1:172.21.15.107:0/1742658284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.031235+0000 mon.a (mon.0) 3260 : audit [INF] from='client.48871 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:10.597 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:10.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:10.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:10.729 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:10.885 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:10.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:10.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:11.018 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:11.037 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:11.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 -- 172.21.15.107:0/2911556576 >> v1:172.21.15.107:6790/0 conn(0x7f81500f9570 legacy=0x7f8150102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 -- 172.21.15.107:0/2911556576 shutdown_connections 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 -- 172.21.15.107:0/2911556576 >> 172.21.15.107:0/2911556576 conn(0x7f81500f2510 msgr2=0x7f81500f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 -- 172.21.15.107:0/2911556576 shutdown_connections 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 -- 172.21.15.107:0/2911556576 wait complete. 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 librados: starting msgr at 2024-02-10T08:47:11.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.112+0000 7f8154f76700 1 librados: starting objecter 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 Processor -- start 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 -- start start 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 librados: setting wanted keys 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 librados: calling monclient init 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8150117290 con 0x7f8150104350 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8150118470 con 0x7f81500f9570 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f8154f76700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8150119670 con 0x7f81500f89c0 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f814e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f81500f89c0 0x7f81500ff3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37708/0 (socket says 172.21.15.107:37708) 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f814e59c700 1 -- 172.21.15.107:0/828690763 learned_addr learned my addr 172.21.15.107:0/828690763 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f814ed9d700 1 --1- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6789/0 conn(0x7f8150104350 0x7f81501159d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48360/0 (socket says 172.21.15.107:48360) 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter ms_handle_connect 0x7f81500f89c0 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter ms_handle_connect 0x7f8150104350 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter ms_handle_connect 0x7f81500f9570 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973386013 0 0) 0x7f8150119670 con 0x7f81500f89c0 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8138003030 con 0x7f81500f89c0 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571222542 0 0) 0x7f8150118470 con 0x7f81500f9570 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150119670 con 0x7f81500f9570 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906923614 0 0) 0x7f8150117290 con 0x7f8150104350 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.113+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150118470 con 0x7f8150104350 2024-02-10T08:47:11.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748919581 0 0) 0x7f8138003030 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8150117290 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f81440019f0 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358614851 0 0) 0x7f8150119670 con 0x7f81500f9570 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8138003030 con 0x7f81500f9570 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2737098296 0 0) 0x7f8150118470 con 0x7f8150104350 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8150119670 con 0x7f8150104350 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8140001890 con 0x7f81500f9570 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8148001b40 con 0x7f8150104350 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805115964 0 0) 0x7f8150117290 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.195:6789/0 conn(0x7f81500f9570 legacy=0x7f81500ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6789/0 conn(0x7f8150104350 legacy=0x7f81501159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815011a870 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8150117480 con 0x7f81500f89c0 2024-02-10T08:47:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f8154f76700 10 client.54155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81501178f0 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f8154f76700 1 librados: init done 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8144002400 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.114+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8144005b10 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.115+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8130004f00 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.115+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8144002570 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(675..675 src has 420..675) v4 ==== 8592+0+0 (unknown 2278971442 0 0) 0x7f8144078e80 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 osd_map(675..675 src has 420..675) v4 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 3 client.54155.objecter handle_osd_map got epochs [675,675] > 0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 3 client.54155.objecter handle_osd_map decoding full epoch 675 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 20 client.54155.objecter dump_active .. 0 homeless 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v675) v1 ==== 54+0+16 (unknown 3454028132 0 2420000192) 0x7f814407b270 con 0x7f81500f89c0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f8138063cd0 2024-02-10T08:47:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 10 client.54155.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 1 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 1 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 0 1 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 4 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 10 client.54155.objecter _send_command 1 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130004950 con 0x7f8130004530 2024-02-10T08:47:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.116+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f8130004530 2024-02-10T08:47:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.117+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130004950 con 0x7f8130004530 2024-02-10T08:47:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.117+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.117+0000 7f813f7fe700 10 client.54155.objecter _finish_command 1 = system:0 2024-02-10T08:47:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.117+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 4 2024-02-10T08:47:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.117+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 0 1 2024-02-10T08:47:11.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 10 client.54155.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 2 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 2 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f81300051a0 osd=0 3 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 0 2 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 4 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 10 client.54155.objecter _send_command 2 2024-02-10T08:47:11.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.131+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8130004d30 con 0x7f8130004530 2024-02-10T08:47:11.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8130004d30 con 0x7f8130004530 2024-02-10T08:47:11.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 10 client.54155.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 4 2024-02-10T08:47:11.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 0 2 2024-02-10T08:47:11.134 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:11.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 10 client.54155.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 3 2024-02-10T08:47:11.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 1 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 4 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 10 client.54155.objecter _send_command 3 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f813000c000 con 0x7f8130004150 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.133+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f8130004150 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.134+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f813000c000 con 0x7f8130004150 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.134+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.134+0000 7f813f7fe700 10 client.54155.objecter _finish_command 3 = system:0 2024-02-10T08:47:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.134+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 4 2024-02-10T08:47:11.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.134+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 1 3 2024-02-10T08:47:11.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 10 client.54155.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 4 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 4 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130003fe0 osd=1 3 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 1 4 2024-02-10T08:47:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 4 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 10 client.54155.objecter _send_command 4 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.150+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8130001850 con 0x7f8130004150 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8130001850 con 0x7f8130004150 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f813f7fe700 10 client.54155.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 4 2024-02-10T08:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 1 4 2024-02-10T08:47:11.153 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 10 client.54155.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 5 2024-02-10T08:47:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 5 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 2 5 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 4 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 10 client.54155.objecter _send_command 5 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.152+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130003e70 con 0x7f813000d2f0 2024-02-10T08:47:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.153+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f813000d2f0 2024-02-10T08:47:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.154+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130003e70 con 0x7f813000d2f0 2024-02-10T08:47:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.154+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.154+0000 7f813f7fe700 10 client.54155.objecter _finish_command 5 = system:0 2024-02-10T08:47:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.154+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 4 2024-02-10T08:47:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.154+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 2 5 2024-02-10T08:47:11.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 10 client.54155.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 6 2024-02-10T08:47:11.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 6 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130003630 osd=2 3 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 2 6 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 4 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 10 client.54155.objecter _send_command 6 2024-02-10T08:47:11.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.175+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f813000e6e0 con 0x7f813000d2f0 2024-02-10T08:47:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f813000e6e0 con 0x7f813000d2f0 2024-02-10T08:47:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 10 client.54155.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 4 2024-02-10T08:47:11.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 2 6 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 10 client.54155.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 7 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 7 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 3 7 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 4 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 10 client.54155.objecter _send_command 7 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130015aa0 con 0x7f8130011e80 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.177+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f8130011e80 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.178+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130015aa0 con 0x7f8130011e80 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.178+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.178+0000 7f813f7fe700 10 client.54155.objecter _finish_command 7 = system:0 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.178+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 4 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.178+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 3 7 2024-02-10T08:47:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 10 client.54155.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 8 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 8 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f81300033b0 osd=3 3 2024-02-10T08:47:11.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 3 8 2024-02-10T08:47:11.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 4 2024-02-10T08:47:11.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 10 client.54155.objecter _send_command 8 2024-02-10T08:47:11.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.193+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f813000e150 con 0x7f8130011e80 2024-02-10T08:47:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f813000e150 con 0x7f8130011e80 2024-02-10T08:47:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 10 client.54155.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 4 2024-02-10T08:47:11.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 3 8 2024-02-10T08:47:11.197 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 10 client.54155.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 9 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 9 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 4 9 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 4 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 10 client.54155.objecter _send_command 9 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f813000e080 con 0x7f813000ce50 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.196+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f813000ce50 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.197+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f813000e080 con 0x7f813000ce50 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.197+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.197+0000 7f813f7fe700 10 client.54155.objecter _finish_command 9 = system:0 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.197+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 4 2024-02-10T08:47:11.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.197+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 4 9 2024-02-10T08:47:11.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 10 client.54155.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 10 2024-02-10T08:47:11.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 10 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f813000df10 osd=4 3 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 4 10 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 4 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 10 client.54155.objecter _send_command 10 2024-02-10T08:47:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.215+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f813000e080 con 0x7f813000ce50 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.216+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f813000e080 con 0x7f813000ce50 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.216+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.216+0000 7f813f7fe700 10 client.54155.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.216+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 4 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.216+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 4 10 2024-02-10T08:47:11.217 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 10 client.54155.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 11 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 11 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 5 11 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 4 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 10 client.54155.objecter _send_command 11 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130017fd0 con 0x7f81300179e0 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.217+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f81300179e0 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.218+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130017fd0 con 0x7f81300179e0 2024-02-10T08:47:11.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.218+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:11.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.218+0000 7f813f7fe700 10 client.54155.objecter _finish_command 11 = system:0 2024-02-10T08:47:11.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.218+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 4 2024-02-10T08:47:11.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.218+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 5 11 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 10 client.54155.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 12 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 12 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f813000e080 osd=5 3 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 5 12 2024-02-10T08:47:11.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 4 2024-02-10T08:47:11.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 10 client.54155.objecter _send_command 12 2024-02-10T08:47:11.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.233+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8130017fd0 con 0x7f81300179e0 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8130017fd0 con 0x7f81300179e0 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f813f7fe700 10 client.54155.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 4 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 5 12 2024-02-10T08:47:11.236 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f8154f76700 10 client.54155.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 13 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.235+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 13 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 6 13 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 4 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 10 client.54155.objecter _send_command 13 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130022bc0 con 0x7f813001f080 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.236+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f813001f080 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.237+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130022bc0 con 0x7f813001f080 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.237+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.237+0000 7f813f7fe700 10 client.54155.objecter _finish_command 13 = system:0 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.237+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 4 2024-02-10T08:47:11.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.237+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 6 13 2024-02-10T08:47:11.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 10 client.54155.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 14 2024-02-10T08:47:11.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 14 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130017fd0 osd=6 3 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 6 14 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 4 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 10 client.54155.objecter _send_command 14 2024-02-10T08:47:11.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.251+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8130024370 con 0x7f813001f080 2024-02-10T08:47:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8130024370 con 0x7f813001f080 2024-02-10T08:47:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 10 client.54155.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 4 2024-02-10T08:47:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 6 14 2024-02-10T08:47:11.254 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:11.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 10 client.54155.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 15 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 15 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 7 15 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 4 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 10 client.54155.objecter _send_command 15 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8130024fe0 con 0x7f8130024a50 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.253+0000 7f813f7fe700 10 client.54155.objecter ms_handle_connect 0x7f8130024a50 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.254+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8130024fe0 con 0x7f8130024a50 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.254+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.254+0000 7f813f7fe700 10 client.54155.objecter _finish_command 15 = system:0 2024-02-10T08:47:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.254+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 4 2024-02-10T08:47:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.254+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 7 15 2024-02-10T08:47:11.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 10 client.54155.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:11.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign -1 16 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 20 client.54155.objecter _get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 15 client.54155.objecter _session_command_op_remove -1 16 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 20 client.54155.objecter get_session s=0x7f8130024370 osd=7 3 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 15 client.54155.objecter _session_command_op_assign 7 16 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 4 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 10 client.54155.objecter _send_command 16 2024-02-10T08:47:11.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.268+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8130024fe0 con 0x7f8130024a50 2024-02-10T08:47:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f813f7fe700 1 -- 172.21.15.107:0/828690763 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8130024fe0 con 0x7f8130024a50 2024-02-10T08:47:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f813f7fe700 10 client.54155.objecter ms_dispatch 0x7f8150001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f813f7fe700 10 client.54155.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f813f7fe700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 4 2024-02-10T08:47:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f813f7fe700 15 client.54155.objecter _session_command_op_remove 7 16 2024-02-10T08:47:11.271 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:11.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 librados: watch_flush enter 2024-02-10T08:47:11.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 librados: watch_flush exit 2024-02-10T08:47:11.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.0 2024-02-10T08:47:11.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6801/3321596972 conn(0x7f8130004530 legacy=0x7f8130002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300051a0 osd=0 1 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.1 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6805/133516126 conn(0x7f8130004150 legacy=0x7f813000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003fe0 osd=1 1 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.2 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6809/3575881728 conn(0x7f813000d2f0 legacy=0x7f813000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130003630 osd=2 1 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.3 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6813/2871825691 conn(0x7f8130011e80 legacy=0x7f8130014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f81300033b0 osd=3 1 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.4 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f814ed9d700 1 -- 172.21.15.107:0/828690763 reap_dead start 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.195:6800/3310789251 conn(0x7f813000ce50 legacy=0x7f8130015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f814ed9d700 1 -- 172.21.15.107:0/828690763 reap_dead start 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000df10 osd=4 1 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.5 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.195:6804/4178321486 conn(0x7f81300179e0 legacy=0x7f813001d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f813000e080 osd=5 1 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.6 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.195:6808/1417479017 conn(0x7f813001f080 legacy=0x7f8130021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130017fd0 osd=6 1 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter close_session for osd.7 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.195:6812/1934176928 conn(0x7f8130024a50 legacy=0x7f8130022bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter put_session s=0x7f8130024370 osd=7 1 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 20 client.54155.objecter shutdown clearing up homeless session... 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 10 client.54155.objecter successfully canceled tick 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8138063cd0 legacy=0x7f8138066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> v1:172.21.15.107:6790/0 conn(0x7f81500f89c0 legacy=0x7f81500ff3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f814ed9d700 1 -- 172.21.15.107:0/828690763 reap_dead start 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 shutdown_connections 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 >> 172.21.15.107:0/828690763 conn(0x7f81500f2510 msgr2=0x7f81501077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 shutdown_connections 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.270+0000 7f8154f76700 1 -- 172.21.15.107:0/828690763 wait complete. 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:11.271+0000 7f8154f76700 1 librados: shutdown 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.276 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.148788+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.157355+0000 mon.a (mon.0) 3262 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.157429+0000 mon.a (mon.0) 3263 : audit [INF] from='client.48871 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.157480+0000 mon.a (mon.0) 3264 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.157920+0000 mon.b (mon.1) 767 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.158210+0000 mon.a (mon.0) 3265 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:10.169669+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.259234+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.260053+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:47:11.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:11.116463+0000 mon.c (mon.2) 462 : audit [DBG] from='client.? v1:172.21.15.107:0/828690763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:11.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.148788+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:11.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.157355+0000 mon.a (mon.0) 3262 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:11.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.157429+0000 mon.a (mon.0) 3263 : audit [INF] from='client.48871 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:11.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.157480+0000 mon.a (mon.0) 3264 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-10T08:47:11.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.157920+0000 mon.b (mon.1) 767 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.158210+0000 mon.a (mon.0) 3265 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:10.169669+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.259234+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.260053+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:11.116463+0000 mon.c (mon.2) 462 : audit [DBG] from='client.? v1:172.21.15.107:0/828690763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.148788+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.157355+0000 mon.a (mon.0) 3262 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.157429+0000 mon.a (mon.0) 3263 : audit [INF] from='client.48871 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.157480+0000 mon.a (mon.0) 3264 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.157920+0000 mon.b (mon.1) 767 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.158210+0000 mon.a (mon.0) 3265 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:11.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:10.169669+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:11.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.259234+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:47:11.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.260053+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:47:11.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:11.116463+0000 mon.c (mon.2) 462 : audit [DBG] from='client.? v1:172.21.15.107:0/828690763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:11.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:47:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:11.778 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:11.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:11.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:11.913 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:11.934 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:11.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:12.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:12.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:12.163 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:12.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:12.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:12.292 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.557082+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:47:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.557834+0000 osd.6 (osd.6) 273 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:47:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.666524+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:47:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:10.668196+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:47:12.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:11.160184+0000 mon.a (mon.0) 3267 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:11.160267+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:11.165511+0000 mon.b (mon.1) 768 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:11.213242+0000 mon.a (mon.0) 3269 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:11.539521+0000 mgr.y (mgr.14553) 569 : cluster [DBG] pgmap v1021: 194 pgs: 7 unknown, 12 peering, 175 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 581 B/s, 17 objects/s recovering 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:11.728305+0000 mgr.y (mgr.14553) 570 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.557082+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.557834+0000 osd.6 (osd.6) 273 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.666524+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:10.668196+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:47:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:11.160184+0000 mon.a (mon.0) 3267 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:11.160267+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-10T08:47:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:11.165511+0000 mon.b (mon.1) 768 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:11.213242+0000 mon.a (mon.0) 3269 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:11.539521+0000 mgr.y (mgr.14553) 569 : cluster [DBG] pgmap v1021: 194 pgs: 7 unknown, 12 peering, 175 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 581 B/s, 17 objects/s recovering 2024-02-10T08:47:12.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:11.728305+0000 mgr.y (mgr.14553) 570 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.557082+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.557834+0000 osd.6 (osd.6) 273 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.666524+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:10.668196+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:11.160184+0000 mon.a (mon.0) 3267 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:11.160267+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-10T08:47:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:11.165511+0000 mon.b (mon.1) 768 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:11.213242+0000 mon.a (mon.0) 3269 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]: dispatch 2024-02-10T08:47:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:11.539521+0000 mgr.y (mgr.14553) 569 : cluster [DBG] pgmap v1021: 194 pgs: 7 unknown, 12 peering, 175 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 581 B/s, 17 objects/s recovering 2024-02-10T08:47:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:11.728305+0000 mgr.y (mgr.14553) 570 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:12.917 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:12.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:13.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:13.074 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:13.172 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:13.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:13.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:13.303 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:13.401 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:13.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:11.588083+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.b scrub starts 2024-02-10T08:47:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:11.590959+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.b scrub ok 2024-02-10T08:47:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.210313+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:13.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:12.222583+0000 mon.a (mon.0) 3271 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]': finished 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.222628+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.256611+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.257561+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:12.281242+0000 mon.b (mon.1) 769 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:12.281658+0000 mon.a (mon.0) 3273 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:11.588083+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.b scrub starts 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:11.590959+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.b scrub ok 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.210313+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:12.222583+0000 mon.a (mon.0) 3271 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]': finished 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.222628+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.256611+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:47:13.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.257561+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:47:13.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:12.281242+0000 mon.b (mon.1) 769 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:12.281658+0000 mon.a (mon.0) 3273 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:13.532 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:11.588083+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.b scrub starts 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:11.590959+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.b scrub ok 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.210313+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:12.222583+0000 mon.a (mon.0) 3271 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-116", "mode": "writeback"}]': finished 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.222628+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.256611+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.257561+0000 osd.0 (osd.0) 116 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:12.281242+0000 mon.b (mon.1) 769 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:12.281658+0000 mon.a (mon.0) 3273 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:13.680 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:13.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:13.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:13.811 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:13.989 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 0] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:47:13.989 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:47:13.989 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-10T08:47:13.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T08:47:14.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:47:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:14.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:14.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.547162+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.548147+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.564177+0000 osd.6 (osd.6) 274 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.565011+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.674421+0000 osd.3 (osd.3) 301 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.675232+0000 osd.3 (osd.3) 302 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:47:14.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.829213+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:12.832143+0000 osd.5 (osd.5) 275 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.004894+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.e scrub starts 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.008369+0000 osd.4 (osd.4) 26 : cluster [DBG] 5.e scrub ok 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:13.226719+0000 mon.a (mon.0) 3274 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.226789+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:13.227827+0000 mon.b (mon.1) 770 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:13.228086+0000 mon.a (mon.0) 3276 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.539924+0000 mgr.y (mgr.14553) 571 : cluster [DBG] pgmap v1024: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 774 B/s, 24 objects/s recovering 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.547162+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:47:14.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.548147+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.564177+0000 osd.6 (osd.6) 274 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.565011+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.674421+0000 osd.3 (osd.3) 301 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.675232+0000 osd.3 (osd.3) 302 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.829213+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:12.832143+0000 osd.5 (osd.5) 275 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.004894+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.e scrub starts 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.008369+0000 osd.4 (osd.4) 26 : cluster [DBG] 5.e scrub ok 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:13.226719+0000 mon.a (mon.0) 3274 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.226789+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:13.227827+0000 mon.b (mon.1) 770 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:13.228086+0000 mon.a (mon.0) 3276 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.539924+0000 mgr.y (mgr.14553) 571 : cluster [DBG] pgmap v1024: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 774 B/s, 24 objects/s recovering 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.547162+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.548147+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.564177+0000 osd.6 (osd.6) 274 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.565011+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.674421+0000 osd.3 (osd.3) 301 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.675232+0000 osd.3 (osd.3) 302 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.829213+0000 osd.5 (osd.5) 274 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:12.832143+0000 osd.5 (osd.5) 275 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:47:14.540 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.004894+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.e scrub starts 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.008369+0000 osd.4 (osd.4) 26 : cluster [DBG] 5.e scrub ok 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:13.226719+0000 mon.a (mon.0) 3274 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.226789+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:13.227827+0000 mon.b (mon.1) 770 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:13.228086+0000 mon.a (mon.0) 3276 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]: dispatch 2024-02-10T08:47:14.541 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.539924+0000 mgr.y (mgr.14553) 571 : cluster [DBG] pgmap v1024: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 774 B/s, 24 objects/s recovering 2024-02-10T08:47:14.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:14.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:14.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:14.679 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:14.700 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:14.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:14.788 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:14.789 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:14.801 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:14.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:14.834 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:14.855 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:14.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:14.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:47:14.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:14.988 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.007 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:15.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:15.138 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.158 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6190 ms) 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11808 ms) 2024-02-10T08:47:15.238 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 00000012 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12112 ms) 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: require_osd_release = pacific 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5913 ms) 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP (494183 ms total) 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1279 ms) 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-10T08:47:15.239 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10688 ms) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18047 ms) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30016 ms total) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7087 ms) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8058 ms) 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: waiting for scrub... 2024-02-10T08:47:15.240 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: done waiting 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi107-112490-107 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 19 more times... 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi107-112490-107 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 18 more times... 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi107-112490-107 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 17 more times... 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi107-112490-107 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 16 more times... 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (30951 ms) 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9913 ms) 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8014 ms) 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-10T08:47:15.241 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8029 ms) 2024-02-10T08:47:15.242 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-10T08:47:15.242 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10002 ms) 2024-02-10T08:47:15.242 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-10T08:47:15.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:15.289 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.307 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:15.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:15.437 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.455 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.509210+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.515129+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.688528+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.689967+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.823056+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:13.825971+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:14.224975+0000 mon.a (mon.0) 3277 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:14.236438+0000 mon.a (mon.0) 3278 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:15.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:14.236549+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.509210+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.515129+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.688528+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.689967+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.823056+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:13.825971+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:14.224975+0000 mon.a (mon.0) 3277 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:14.236438+0000 mon.a (mon.0) 3278 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:15.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:14.236549+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.509210+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.515129+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.688528+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.689967+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.823056+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:47:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:13.825971+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:47:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:14.224975+0000 mon.a (mon.0) 3277 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:14.236438+0000 mon.a (mon.0) 3278 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-116"}]': finished 2024-02-10T08:47:15.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:14.236549+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2024-02-10T08:47:15.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.786 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:15.787 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:15] "GET /metrics HTTP/1.1" 200 190176 "" "Prometheus/2.33.4" 2024-02-10T08:47:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:15.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:15.919 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:15.937 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:15.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:16.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:16.068 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:16.316 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:16.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:14.565318+0000 osd.6 (osd.6) 276 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:47:16.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:14.568438+0000 osd.6 (osd.6) 277 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:47:16.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.146842+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub starts 2024-02-10T08:47:16.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.149798+0000 osd.2 (osd.2) 254 : cluster [DBG] 16.a scrub ok 2024-02-10T08:47:16.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.237156+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.377817+0000 osd.0 (osd.0) 117 : cluster [DBG] 16.f scrub starts 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.381428+0000 osd.0 (osd.0) 118 : cluster [DBG] 16.f scrub ok 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:15.540300+0000 mgr.y (mgr.14553) 572 : cluster [DBG] pgmap v1027: 162 pgs: 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 192 B/s, 7 objects/s recovering 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.144217+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:16.153270+0000 mon.b (mon.1) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:16.156684+0000 mon.a (mon.0) 3282 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:16.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:16.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:14.565318+0000 osd.6 (osd.6) 276 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:47:16.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:14.568438+0000 osd.6 (osd.6) 277 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:47:16.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.146842+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub starts 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.149798+0000 osd.2 (osd.2) 254 : cluster [DBG] 16.a scrub ok 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.237156+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.377817+0000 osd.0 (osd.0) 117 : cluster [DBG] 16.f scrub starts 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.381428+0000 osd.0 (osd.0) 118 : cluster [DBG] 16.f scrub ok 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:15.540300+0000 mgr.y (mgr.14553) 572 : cluster [DBG] pgmap v1027: 162 pgs: 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 192 B/s, 7 objects/s recovering 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.144217+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:16.153270+0000 mon.b (mon.1) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.321 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:16.156684+0000 mon.a (mon.0) 3282 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:14.565318+0000 osd.6 (osd.6) 276 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:14.568438+0000 osd.6 (osd.6) 277 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.146842+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub starts 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.149798+0000 osd.2 (osd.2) 254 : cluster [DBG] 16.a scrub ok 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.237156+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.377817+0000 osd.0 (osd.0) 117 : cluster [DBG] 16.f scrub starts 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.381428+0000 osd.0 (osd.0) 118 : cluster [DBG] 16.f scrub ok 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:15.540300+0000 mgr.y (mgr.14553) 572 : cluster [DBG] pgmap v1027: 162 pgs: 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 192 B/s, 7 objects/s recovering 2024-02-10T08:47:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.144217+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2024-02-10T08:47:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:16.153270+0000 mon.b (mon.1) 771 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:16.156684+0000 mon.a (mon.0) 3282 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:16.589 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:16.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:16.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:16.723 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:16.742 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:16.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:16.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:16.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:16.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:47:17.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:17.113 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:17.133 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:17.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:17.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:17.267 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:17.286 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:17.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:17.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:17.417 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:17.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:17.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.368837+0000 osd.0 (osd.0) 119 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:47:17.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.370178+0000 osd.0 (osd.0) 120 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:17.142502+0000 mon.a (mon.0) 3283 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.142579+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:17.150946+0000 mon.b (mon.1) 772 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:17.152161+0000 mon.a (mon.0) 3285 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.368837+0000 osd.0 (osd.0) 119 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.370178+0000 osd.0 (osd.0) 120 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:17.142502+0000 mon.a (mon.0) 3283 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.142579+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:17.150946+0000 mon.b (mon.1) 772 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:17.152161+0000 mon.a (mon.0) 3285 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.368837+0000 osd.0 (osd.0) 119 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.370178+0000 osd.0 (osd.0) 120 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:17.142502+0000 mon.a (mon.0) 3283 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.142579+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:17.150946+0000 mon.b (mon.1) 772 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:17.152161+0000 mon.a (mon.0) 3285 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:17.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:17.568 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:17.586 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:17.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:17.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:17.717 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.772150+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:47:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.775174+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:47:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.814802+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.c scrub starts 2024-02-10T08:47:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:16.815637+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.c scrub ok 2024-02-10T08:47:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.395462+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.396692+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.541005+0000 mgr.y (mgr.14553) 573 : cluster [DBG] pgmap v1030: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:17.776628+0000 mon.a (mon.0) 3286 : audit [DBG] from='client.? v1:172.21.15.107:0/2174228272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:18.143111+0000 mon.a (mon.0) 3287 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.143161+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:18.145962+0000 mon.b (mon.1) 773 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:18.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:18.147873+0000 mon.a (mon.0) 3289 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:18.590 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:18.590 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:18.590 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:18.590 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:18.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:18.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:18.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:18.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:18.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.772+0000 7fb80ef3c700 1 -- 172.21.15.107:0/1428935183 >> v1:172.21.15.107:6789/0 conn(0x7fb8080f89c0 legacy=0x7fb8080f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.772+0000 7fb80ef3c700 1 -- 172.21.15.107:0/1428935183 shutdown_connections 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.772+0000 7fb80ef3c700 1 -- 172.21.15.107:0/1428935183 >> 172.21.15.107:0/1428935183 conn(0x7fb8080f24d0 msgr2=0x7fb8080f48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.772+0000 7fb80ef3c700 1 -- 172.21.15.107:0/1428935183 shutdown_connections 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.772+0000 7fb80ef3c700 1 -- 172.21.15.107:0/1428935183 wait complete. 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 librados: starting msgr at 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 librados: starting objecter 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 Processor -- start 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 -- start start 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 librados: setting wanted keys 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.773+0000 7fb80ef3c700 1 librados: calling monclient init 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb80ef3c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8081173b0 con 0x7fb808104350 2024-02-10T08:47:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb80ef3c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb808118590 con 0x7fb8080f89c0 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb80ef3c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb808119790 con 0x7fb8080f9570 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb807fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb8080f9570 0x7fb8080ffc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37724/0 (socket says 172.21.15.107:37724) 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb807fff700 1 -- 172.21.15.107:0/2174228272 learned_addr learned my addr 172.21.15.107:0/2174228272 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter ms_handle_connect 0x7fb8080f9570 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter ms_handle_connect 0x7fb808104350 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter ms_handle_connect 0x7fb8080f89c0 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533691940 0 0) 0x7fb8081173b0 con 0x7fb808104350 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f40038e0 con 0x7fb808104350 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833065852 0 0) 0x7fb808119790 con 0x7fb8080f9570 2024-02-10T08:47:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8081173b0 con 0x7fb8080f9570 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147312266 0 0) 0x7fb808118590 con 0x7fb8080f89c0 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808119790 con 0x7fb8080f89c0 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445099382 0 0) 0x7fb7f40038e0 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb808118590 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb800001d70 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4142878509 0 0) 0x7fb8081173b0 con 0x7fb8080f9570 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7f40038e0 con 0x7fb8080f9570 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb7f00018c0 con 0x7fb8080f9570 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4000313256 0 0) 0x7fb808118590 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6790/0 conn(0x7fb8080f9570 legacy=0x7fb8080ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.195:6789/0 conn(0x7fb8080f89c0 legacy=0x7fb8080ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80811a990 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.774+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8081175a0 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb8000026a0 con 0x7fb808104350 2024-02-10T08:47:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb800003610 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb80ef3c700 10 client.28584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb808117a10 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb80ef3c700 1 librados: init done 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.775+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb808060410 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb800002200 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(682..682 src has 420..682) v4 ==== 8569+0+0 (unknown 2775831096 0 0) 0x7fb800076ac0 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 osd_map(682..682 src has 420..682) v4 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 3 client.28584.objecter handle_osd_map got epochs [682,682] > 0 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 3 client.28584.objecter handle_osd_map decoding full epoch 682 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 20 client.28584.objecter dump_active .. 0 homeless 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.776+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v682) v1 ==== 54+0+16 (unknown 2422229637 0 2420000192) 0x7fb800076e60 con 0x7fb808104350 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb7f4063ba0 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 1 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 1 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 0 1 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 4 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 10 client.28584.objecter _send_command 1 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8080615c0 con 0x7fb808118720 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.777+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb808118720 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.778+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8080615c0 con 0x7fb808118720 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.778+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:18.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.778+0000 7fb805ffb700 10 client.28584.objecter _finish_command 1 = system:0 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.778+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 4 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.778+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 0 1 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 2 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 2 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080475e0 osd=0 3 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 0 2 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 4 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 10 client.28584.objecter _send_command 2 2024-02-10T08:47:18.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.793+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb80810a070 con 0x7fb808118720 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb80810a070 con 0x7fb808118720 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb805ffb700 10 client.28584.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 4 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 0 2 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 1 3 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 4 2024-02-10T08:47:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 10 client.28584.objecter _send_command 3 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.794+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb808028420 con 0x7fb8080fa050 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb8080fa050 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb808028420 con 0x7fb8080fa050 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 10 client.28584.objecter _finish_command 3 = system:0 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 4 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.795+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 1 3 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 4 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080f99d0 osd=1 3 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 1 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 10 client.28584.objecter _send_command 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.810+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb808028800 con 0x7fb8080fa050 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb808028800 con 0x7fb8080fa050 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb805ffb700 10 client.28584.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 1 4 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 5 2024-02-10T08:47:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 5 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 2 5 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 4 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 10 client.28584.objecter _send_command 5 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.811+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb808059f60 con 0x7fb808118070 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb808118070 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb808059f60 con 0x7fb808118070 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 10 client.28584.objecter _finish_command 5 = system:0 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 4 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.812+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 2 5 2024-02-10T08:47:18.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 6 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 6 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb808117f00 osd=2 3 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 2 6 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 4 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 10 client.28584.objecter _send_command 6 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.826+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb80805a180 con 0x7fb808118070 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb80805a180 con 0x7fb808118070 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 10 client.28584.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 4 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 2 6 2024-02-10T08:47:18.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 7 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 7 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 3 7 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 4 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 10 client.28584.objecter _send_command 7 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb808127d60 con 0x7fb80811f8e0 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.828+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb80811f8e0 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.829+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb808127d60 con 0x7fb80811f8e0 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.829+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:18.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.829+0000 7fb805ffb700 10 client.28584.objecter _finish_command 7 = system:0 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.829+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 4 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.829+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 3 7 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 8 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 8 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8080596a0 osd=3 3 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 3 8 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 4 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 10 client.28584.objecter _send_command 8 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.843+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb80811ee40 con 0x7fb80811f8e0 2024-02-10T08:47:18.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb80811ee40 con 0x7fb80811f8e0 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 10 client.28584.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 4 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 3 8 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 9 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 9 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 4 9 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 4 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 10 client.28584.objecter _send_command 9 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8081242f0 con 0x7fb80811efb0 2024-02-10T08:47:18.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.845+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb80811efb0 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.846+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8081242f0 con 0x7fb80811efb0 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.846+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.846+0000 7fb805ffb700 10 client.28584.objecter _finish_command 9 = system:0 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.846+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 4 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.846+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 4 9 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 10 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 10 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb80811ee40 osd=4 3 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 4 10 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 4 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 10 client.28584.objecter _send_command 10 2024-02-10T08:47:18.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.861+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb8081242f0 con 0x7fb80811efb0 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.862+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb8081242f0 con 0x7fb80811efb0 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.862+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.862+0000 7fb805ffb700 10 client.28584.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.862+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 4 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.862+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 4 10 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 11 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 11 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 5 11 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 4 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 10 client.28584.objecter _send_command 11 2024-02-10T08:47:18.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8081308e0 con 0x7fb80812cc90 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.863+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb80812cc90 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.864+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8081308e0 con 0x7fb80812cc90 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.864+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.864+0000 7fb805ffb700 10 client.28584.objecter _finish_command 11 = system:0 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.864+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 4 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.864+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 5 11 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 12 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 12 2024-02-10T08:47:18.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8081242f0 osd=5 3 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 5 12 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 4 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 10 client.28584.objecter _send_command 12 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.878+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb8081308e0 con 0x7fb80812cc90 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb8081308e0 con 0x7fb80812cc90 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb805ffb700 10 client.28584.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 4 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 5 12 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 13 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 13 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 6 13 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 4 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 10 client.28584.objecter _send_command 13 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.880+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb808131b00 con 0x7fb8081325a0 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.881+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb8081325a0 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.882+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb808131b00 con 0x7fb8081325a0 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.882+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.882+0000 7fb805ffb700 10 client.28584.objecter _finish_command 13 = system:0 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.882+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 4 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.882+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 6 13 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 14 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 14 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb8081308e0 osd=6 3 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 6 14 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 4 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 10 client.28584.objecter _send_command 14 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.896+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb808131b00 con 0x7fb8081325a0 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb808131b00 con 0x7fb8081325a0 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 10 client.28584.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 4 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 6 14 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 15 2024-02-10T08:47:18.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 15 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 7 15 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 4 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 10 client.28584.objecter _send_command 15 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8081392b0 con 0x7fb8081357a0 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.898+0000 7fb805ffb700 10 client.28584.objecter ms_handle_connect 0x7fb8081357a0 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.899+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8081392b0 con 0x7fb8081357a0 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.899+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.899+0000 7fb805ffb700 10 client.28584.objecter _finish_command 15 = system:0 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.899+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 4 2024-02-10T08:47:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.899+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 7 15 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 10 client.28584.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign -1 16 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 20 client.28584.objecter _get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_remove -1 16 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 20 client.28584.objecter get_session s=0x7fb808131b00 osd=7 3 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 15 client.28584.objecter _session_command_op_assign 7 16 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 4 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 10 client.28584.objecter _send_command 16 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.913+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb8081392b0 con 0x7fb8081357a0 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.915+0000 7fb805ffb700 1 -- 172.21.15.107:0/2174228272 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb8081392b0 con 0x7fb8081357a0 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.915+0000 7fb805ffb700 10 client.28584.objecter ms_dispatch 0x7fb808001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.915+0000 7fb805ffb700 10 client.28584.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.915+0000 7fb805ffb700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 4 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.915+0000 7fb805ffb700 15 client.28584.objecter _session_command_op_remove 7 16 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 librados: watch_flush enter 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 librados: watch_flush exit 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.0 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6801/3321596972 conn(0x7fb808118720 legacy=0x7fb808118b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080475e0 osd=0 1 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.1 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6805/133516126 conn(0x7fb8080fa050 legacy=0x7fb8080fa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080f99d0 osd=1 1 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.2 2024-02-10T08:47:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6809/3575881728 conn(0x7fb808118070 legacy=0x7fb808121d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808117f00 osd=2 1 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.3 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6813/2871825691 conn(0x7fb80811f8e0 legacy=0x7fb808126640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80d4d9700 1 -- 172.21.15.107:0/2174228272 reap_dead start 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8080596a0 osd=3 1 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.4 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.195:6800/3310789251 conn(0x7fb80811efb0 legacy=0x7fb80812b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80d4d9700 1 -- 172.21.15.107:0/2174228272 reap_dead start 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb80811ee40 osd=4 1 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.5 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.195:6804/4178321486 conn(0x7fb80812cc90 legacy=0x7fb80812f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081242f0 osd=5 1 2024-02-10T08:47:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.6 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.195:6808/1417479017 conn(0x7fb8081325a0 legacy=0x7fb808132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb8081308e0 osd=6 1 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter close_session for osd.7 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.195:6812/1934176928 conn(0x7fb8081357a0 legacy=0x7fb808137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter put_session s=0x7fb808131b00 osd=7 1 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 20 client.28584.objecter shutdown clearing up homeless session... 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 10 client.28584.objecter successfully canceled tick 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb7f4063ba0 legacy=0x7fb7f4066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> v1:172.21.15.107:6789/0 conn(0x7fb808104350 legacy=0x7fb808115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80d4d9700 1 -- 172.21.15.107:0/2174228272 reap_dead start 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 shutdown_connections 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 >> 172.21.15.107:0/2174228272 conn(0x7fb8080f24d0 msgr2=0x7fb8081077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 shutdown_connections 2024-02-10T08:47:18.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 -- 172.21.15.107:0/2174228272 wait complete. 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:17.916+0000 7fb80ef3c700 1 librados: shutdown 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.772150+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.775174+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.814802+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.c scrub starts 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:16.815637+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.c scrub ok 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.395462+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.396692+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.541005+0000 mgr.y (mgr.14553) 573 : cluster [DBG] pgmap v1030: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:17.776628+0000 mon.a (mon.0) 3286 : audit [DBG] from='client.? v1:172.21.15.107:0/2174228272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:18.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:18.143111+0000 mon.a (mon.0) 3287 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.143161+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:18.145962+0000 mon.b (mon.1) 773 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:18.147873+0000 mon.a (mon.0) 3289 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.772150+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.775174+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.814802+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.c scrub starts 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:16.815637+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.c scrub ok 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.395462+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.396692+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.541005+0000 mgr.y (mgr.14553) 573 : cluster [DBG] pgmap v1030: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:17.776628+0000 mon.a (mon.0) 3286 : audit [DBG] from='client.? v1:172.21.15.107:0/2174228272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:18.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:18.143111+0000 mon.a (mon.0) 3287 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:18.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.143161+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2024-02-10T08:47:18.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:18.145962+0000 mon.b (mon.1) 773 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:18.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:18.147873+0000 mon.a (mon.0) 3289 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:19.167 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:19.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:19.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:19.304 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:19.325 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:19.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.568180+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.571083+0000 osd.6 (osd.6) 279 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.782883+0000 osd.3 (osd.3) 307 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.784921+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.820763+0000 osd.5 (osd.5) 280 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.824245+0000 osd.5 (osd.5) 281 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.981754+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:47:19.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:17.985231+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.042975+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.043727+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.401662+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.403393+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:19.145888+0000 mon.a (mon.0) 3290 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.145978+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:19.146863+0000 mon.b (mon.1) 774 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:19.147430+0000 mon.a (mon.0) 3292 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.568180+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.571083+0000 osd.6 (osd.6) 279 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.782883+0000 osd.3 (osd.3) 307 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.784921+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.820763+0000 osd.5 (osd.5) 280 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.824245+0000 osd.5 (osd.5) 281 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.981754+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:17.985231+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.042975+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.043727+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:47:19.404 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.401662+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:47:19.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.403393+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:47:19.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:19.145888+0000 mon.a (mon.0) 3290 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:19.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.145978+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-10T08:47:19.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:19.146863+0000 mon.b (mon.1) 774 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.405 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:19.147430+0000 mon.a (mon.0) 3292 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:19.459 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:19.511 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:19.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.568180+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.571083+0000 osd.6 (osd.6) 279 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.782883+0000 osd.3 (osd.3) 307 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.784921+0000 osd.3 (osd.3) 308 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.820763+0000 osd.5 (osd.5) 280 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.824245+0000 osd.5 (osd.5) 281 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:47:19.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.981754+0000 osd.4 (osd.4) 27 : cluster [DBG] 2.f scrub starts 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:17.985231+0000 osd.4 (osd.4) 28 : cluster [DBG] 2.f scrub ok 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.042975+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.043727+0000 osd.2 (osd.2) 256 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.401662+0000 osd.0 (osd.0) 123 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.403393+0000 osd.0 (osd.0) 124 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:19.145888+0000 mon.a (mon.0) 3290 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.145978+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:19.146863+0000 mon.b (mon.1) 774 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:19.147430+0000 mon.a (mon.0) 3292 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]: dispatch 2024-02-10T08:47:19.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:19.658 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:19.678 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:19.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.703+0000 7f617815f700 1 -- 172.21.15.107:0/3692901317 >> v1:172.21.15.107:6789/0 conn(0x7f6170104330 legacy=0x7f61701067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.703+0000 7f617815f700 1 -- 172.21.15.107:0/3692901317 shutdown_connections 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.703+0000 7f617815f700 1 -- 172.21.15.107:0/3692901317 >> 172.21.15.107:0/3692901317 conn(0x7f61700f3640 msgr2=0x7f61700f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.703+0000 7f617815f700 1 -- 172.21.15.107:0/3692901317 shutdown_connections 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 -- 172.21.15.107:0/3692901317 wait complete. 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 librados: starting msgr at 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 librados: starting objecter 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 Processor -- start 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 -- start start 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 librados: setting wanted keys 2024-02-10T08:47:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 librados: calling monclient init 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f617006cd70 con 0x7f61700fda50 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f617006cf60 con 0x7f6170104330 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f617815f700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f617011bbf0 con 0x7f61701007f0 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f6175efb700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f61700fda50 0x7f61700fca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60160/0 (socket says 172.21.15.107:60160) 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f61756fa700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f61701007f0 0x7f6170114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60228/0 (socket says 172.21.15.107:60228) 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f6175efb700 1 -- 172.21.15.107:0/283827994 learned_addr learned my addr 172.21.15.107:0/283827994 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.704+0000 7f6162ffd700 10 client.?.objecter ms_handle_connect 0x7f61700fda50 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 10 client.?.objecter ms_handle_connect 0x7f61701007f0 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 10 client.?.objecter ms_handle_connect 0x7f6170104330 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194131243 0 0) 0x7f617006cd70 con 0x7f61700fda50 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f615c003030 con 0x7f61700fda50 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864091686 0 0) 0x7f617011bbf0 con 0x7f61701007f0 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617006cd70 con 0x7f61701007f0 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328906464 0 0) 0x7f617006cf60 con 0x7f6170104330 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617011bbf0 con 0x7f6170104330 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1379218592 0 0) 0x7f615c003030 con 0x7f61700fda50 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f617006cf60 con 0x7f61700fda50 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6164001980 con 0x7f61700fda50 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727070095 0 0) 0x7f617006cd70 con 0x7f61701007f0 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f615c003030 con 0x7f61701007f0 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f616c0018a0 con 0x7f61701007f0 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575783443 0 0) 0x7f617011bbf0 con 0x7f6170104330 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f617006cd70 con 0x7f6170104330 2024-02-10T08:47:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6168002310 con 0x7f6170104330 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1799537617 0 0) 0x7f617006cf60 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 >> v1:172.21.15.107:6790/0 conn(0x7f61701007f0 legacy=0x7f6170114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 >> v1:172.21.15.195:6789/0 conn(0x7f6170104330 legacy=0x7f61701184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617011cd90 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617011bde0 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f617815f700 10 client.28590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f617011c250 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f617815f700 1 librados: init done 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6164001d10 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.705+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6164005c30 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.706+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6154004f20 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6164002270 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(684..684 src has 420..684) v4 ==== 8564+0+0 (unknown 1901188973 0 0) 0x7f6164079130 con 0x7f61700fda50 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 10 client.28590.objecter ms_dispatch 0x7f6170001550 osd_map(684..684 src has 420..684) v4 2024-02-10T08:47:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 3 client.28590.objecter handle_osd_map got epochs [684,684] > 0 2024-02-10T08:47:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 3 client.28590.objecter handle_osd_map decoding full epoch 684 2024-02-10T08:47:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 20 client.28590.objecter dump_active .. 0 homeless 2024-02-10T08:47:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.707+0000 7f6162ffd700 10 client.28590.objecter ms_handle_connect 0x7f615c063c70 2024-02-10T08:47:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.712+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f616404c350 con 0x7f61700fda50 2024-02-10T08:47:19.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:19.811 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:19.894+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f61540051d0 con 0x7f61700fda50 2024-02-10T08:47:20.059 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:20.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:20.101 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:20.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.147+0000 7f6162ffd700 1 -- 172.21.15.107:0/283827994 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v685) v1 ==== 101+0+0 (unknown 1499185917 0 0) 0x7f616404a760 con 0x7f61700fda50 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 10 librados: watch_flush enter 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 10 librados: watch_flush exit 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 20 client.28590.objecter shutdown clearing up homeless session... 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 10 client.28590.objecter successfully canceled tick 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 >> v1:172.21.15.107:6800/2634842009 conn(0x7f615c063c70 legacy=0x7f615c066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 >> v1:172.21.15.107:6789/0 conn(0x7f61700fda50 legacy=0x7f61700fca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 shutdown_connections 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 >> 172.21.15.107:0/283827994 conn(0x7f61700f3640 msgr2=0x7f61701077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:20.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.149+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 shutdown_connections 2024-02-10T08:47:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.150+0000 7f617815f700 1 -- 172.21.15.107:0/283827994 wait complete. 2024-02-10T08:47:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:20.150+0000 7f617815f700 1 librados: shutdown 2024-02-10T08:47:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:47:20.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:20.194 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:20.214 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:20.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:20.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:20.346 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:20.365 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:20.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:20.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.547985+0000 osd.6 (osd.6) 280 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:47:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:18.549597+0000 osd.6 (osd.6) 281 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:47:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.052246+0000 osd.2 (osd.2) 257 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:47:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.053927+0000 osd.2 (osd.2) 258 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:47:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.541375+0000 mgr.y (mgr.14553) 574 : cluster [DBG] pgmap v1033: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:20.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:19.895521+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:20.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.942809+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:47:20.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.946398+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:47:20.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:20.143068+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:20.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:20.143090+0000 mon.a (mon.0) 3295 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:20.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:20.147771+0000 mon.a (mon.0) 3296 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]': finished 2024-02-10T08:47:20.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:20.147845+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:20.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:20.147888+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-10T08:47:20.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.547985+0000 osd.6 (osd.6) 280 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:47:20.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:18.549597+0000 osd.6 (osd.6) 281 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:47:20.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.052246+0000 osd.2 (osd.2) 257 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.053927+0000 osd.2 (osd.2) 258 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.541375+0000 mgr.y (mgr.14553) 574 : cluster [DBG] pgmap v1033: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:19.895521+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.942809+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.946398+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:47:20.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:20.143068+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:20.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:20.143090+0000 mon.a (mon.0) 3295 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:20.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:20.147771+0000 mon.a (mon.0) 3296 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]': finished 2024-02-10T08:47:20.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:20.147845+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:20.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:20.147888+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-10T08:47:20.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:20.496 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.547985+0000 osd.6 (osd.6) 280 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:18.549597+0000 osd.6 (osd.6) 281 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.052246+0000 osd.2 (osd.2) 257 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.053927+0000 osd.2 (osd.2) 258 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.541375+0000 mgr.y (mgr.14553) 574 : cluster [DBG] pgmap v1033: 194 pgs: 3 creating+peering, 26 unknown, 165 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:19.895521+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.942809+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.946398+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:20.143068+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:20.143090+0000 mon.a (mon.0) 3295 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:20.147771+0000 mon.a (mon.0) 3296 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-118", "mode": "writeback"}]': finished 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:20.147845+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.107:0/283827994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:20.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:20.147888+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2024-02-10T08:47:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:20.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:20.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:20.831 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:20.852 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:20.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:20.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:20.985 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.005 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:21.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:21.137 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.157 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:21.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:21.290 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.387 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:21.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.265+0000 7f3fd0451700 1 -- 172.21.15.107:0/2067236917 >> v1:172.21.15.107:6790/0 conn(0x7f3fc80fc910 legacy=0x7f3fc80fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.265+0000 7f3fd0451700 1 -- 172.21.15.107:0/2067236917 shutdown_connections 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.265+0000 7f3fd0451700 1 -- 172.21.15.107:0/2067236917 >> 172.21.15.107:0/2067236917 conn(0x7f3fc80f2510 msgr2=0x7f3fc80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.265+0000 7f3fd0451700 1 -- 172.21.15.107:0/2067236917 shutdown_connections 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.265+0000 7f3fd0451700 1 -- 172.21.15.107:0/2067236917 wait complete. 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 librados: starting msgr at 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 librados: starting objecter 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 Processor -- start 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 -- start start 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 librados: setting wanted keys 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 librados: calling monclient init 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fc80fc080 con 0x7f3fc80fc910 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fc80fc250 con 0x7f3fc80ff710 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fd0451700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3fc811bb20 con 0x7f3fc8103250 2024-02-10T08:47:21.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fce9ee700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3fc8103250 0x7f3fc81183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60256/0 (socket says 172.21.15.107:60256) 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fce9ee700 1 -- 172.21.15.107:0/3754312630 learned_addr learned my addr 172.21.15.107:0/3754312630 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fce1ed700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3fc80fc910 0x7f3fc80fb8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60168/0 (socket says 172.21.15.107:60168) 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f3fc80fc910 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f3fc8103250 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter ms_handle_connect 0x7f3fc80ff710 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.266+0000 7f3fbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030719785 0 0) 0x7f3fc80fc080 con 0x7f3fc80fc910 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb40033f0 con 0x7f3fc80fc910 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17378040 0 0) 0x7f3fc811bb20 con 0x7f3fc8103250 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc80fc080 con 0x7f3fc8103250 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40189898 0 0) 0x7f3fc80fc250 con 0x7f3fc80ff710 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc811bb20 con 0x7f3fc80ff710 2024-02-10T08:47:21.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055941169 0 0) 0x7f3fb40033f0 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fc80fc250 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3fc4002f00 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841240442 0 0) 0x7f3fc811bb20 con 0x7f3fc80ff710 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fb40033f0 con 0x7f3fc80ff710 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43830035 0 0) 0x7f3fc80fc080 con 0x7f3fc8103250 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fc811bb20 con 0x7f3fc8103250 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3fb80017c0 con 0x7f3fc80ff710 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3fc0001c00 con 0x7f3fc8103250 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1401141948 0 0) 0x7f3fc80fc250 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 >> v1:172.21.15.107:6790/0 conn(0x7f3fc8103250 legacy=0x7f3fc81183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 >> v1:172.21.15.195:6789/0 conn(0x7f3fc80ff710 legacy=0x7f3fc8114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc811ccc0 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc811bd10 con 0x7f3fc80fc910 2024-02-10T08:47:21.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fd0451700 10 client.28596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc811c200 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.267+0000 7f3fd0451700 1 librados: init done 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.268+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3fc4003060 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.268+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3fc4003890 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.268+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac004f20 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3fc40039f0 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(685..685 src has 420..685) v4 ==== 8564+0+0 (unknown 1580529007 0 0) 0x7f3fc40790d0 con 0x7f3fc80fc910 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 10 client.28596.objecter ms_dispatch 0x7f3fc8001550 osd_map(685..685 src has 420..685) v4 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 3 client.28596.objecter handle_osd_map got epochs [685,685] > 0 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 3 client.28596.objecter handle_osd_map decoding full epoch 685 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 20 client.28596.objecter dump_active .. 0 homeless 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.269+0000 7f3fbf7fe700 10 client.28596.objecter ms_handle_connect 0x7f3fb4063d70 2024-02-10T08:47:21.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.273+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3fc404d2e0 con 0x7f3fc80fc910 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.446+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3fac0051d0 con 0x7f3fc80fc910 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.448+0000 7f3fbf7fe700 1 -- 172.21.15.107:0/3754312630 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v685) v1 ==== 81+0+48786 (unknown 355817256 0 3253380827) 0x7f3fc404b750 con 0x7f3fc80fc910 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 685, 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "275c2754-c7ee-11ee-95b8-87774f69a715", 2024-02-10T08:47:21.481 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2024-02-10T08:27:44.316161+0000", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2024-02-10T08:47:20.142829+0000", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2024-02-10T08:46:28.181193+0000", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2024-02-10T08:47:07.143327+0000", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5799936, 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 108, 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T08:47:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 243, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "luminous", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "luminous", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "pacific", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:31:09.850435+0000", 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2024-02-10T08:47:21.484 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "552", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.485 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.486 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.487 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.488 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.489 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_max": 32, 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.490 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": ".rgw.root", 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.728206+0000", 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.492 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.493 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.494 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.495 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.496 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.498 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.499 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.499 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.499 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.499 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.499 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "iscsi", 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:16.960794+0000", 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 3, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 3, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 3, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 3, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 3, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.501 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "58", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.502 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.503 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.504 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 4, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.log", 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:18.379465+0000", 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.505 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:47:21.506 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "59", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.507 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.508 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.509 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:47:21.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 5, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.control", 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:20.354391+0000", 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.511 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 22, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 22, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 22, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 22, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 22, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "5.16", 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 135, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 134, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 134, 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.512 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "135", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "135", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "133", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.513 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.515 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 6, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "default.rgw.meta", 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:33:22.423222+0000", 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.516 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.517 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "63", 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.519 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.520 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.521 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_bias": 4 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2024-02-10T08:47:21.522 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw": {} 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 16, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:36:32.714443+0000", 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.523 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.524 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "104", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.525 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.526 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 234, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-101", 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:45:44.375681+0000", 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8193, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2024-02-10T08:47:21.528 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2024-02-10T08:47:21.529 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "684", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "684", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "684", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "684", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 7, 2024-02-10T08:47:21.530 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 683, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [ 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: 243 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": 243, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": 243, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.531 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi107-112490-101", 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.532 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 243, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "test-rados-api-smithi107-112490-118", 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2024-02-10T08:47:15.496669+0000", 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 8201, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2024-02-10T08:47:21.533 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_count": 0, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_target": 0, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 32, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 32, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 32, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 32, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 32, 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2024-02-10T08:47:21.534 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "685", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "684", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "684", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "684", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 7, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 683, 2024-02-10T08:47:21.535 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": 234, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "writeback", 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2024-02-10T08:47:21.536 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2024-02-10T08:47:21.537 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": {} 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e30a7307-1dfd-4dfa-b4c4-89b83eb24167", 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.538 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 8, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.539 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:47:21.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3321596972 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/3321596972", 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6802/3321596972", 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6804/3321596972", 2024-02-10T08:47:21.541 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6803/3321596972", 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c5fe7280-6de7-4c89-a9f2-22eebe4c1c80", 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 0, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.542 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 663, 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 133516126 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.545 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6805/133516126", 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6806/133516126", 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6808/133516126", 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6807/133516126", 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "3ae7131b-69d5-4cbd-b1ca-29556fac7fbf", 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.9418182373046875, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 18, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.547 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2024-02-10T08:47:21.548 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.549 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3575881728 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/3575881728", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6810/3575881728", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6812/3575881728", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6811/3575881728", 2024-02-10T08:47:21.550 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1ffb2633-18bb-4048-90eb-7130e0cc7d43", 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 25, 2024-02-10T08:47:21.551 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.552 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2871825691 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6813/2871825691", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6814/2871825691", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6816/2871825691", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6815/2871825691", 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.554 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "1b7529a6-19fc-4435-be7f-28876750838a", 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 31, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 479, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 633, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.555 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 480, 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6800", 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.556 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6801", 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.557 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6803", 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.449+0000 7f3fd0451700 10 librados: watch_flush enter 2024-02-10T08:47:21.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.449+0000 7f3fd0451700 10 librados: watch_flush exit 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.449+0000 7f3fd0451700 20 client.28596.objecter shutdown clearing up homeless session... 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.449+0000 7f3fd0451700 10 client.28596.objecter successfully canceled tick 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3fb4063d70 legacy=0x7f3fb4066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 >> v1:172.21.15.107:6789/0 conn(0x7f3fc80fc910 legacy=0x7f3fc80fb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 shutdown_connections 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 >> 172.21.15.107:0/3754312630 conn(0x7f3fc80f2510 msgr2=0x7f3fc80612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 shutdown_connections 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 -- 172.21.15.107:0/3754312630 wait complete. 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:21.450+0000 7f3fd0451700 1 librados: shutdown 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.559 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6802", 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3310789251 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6800/3310789251", 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6801/3310789251", 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6803/3310789251", 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6802/3310789251", 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.561 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "bdc1b91c-94ce-4287-8019-535b326da3e5", 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.562 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 37, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.563 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.564 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6804", 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6805", 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.566 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6807", 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6806", 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4178321486 2024-02-10T08:47:21.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6804/4178321486", 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6805/4178321486", 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6807/4178321486", 2024-02-10T08:47:21.569 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6806/4178321486", 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.570 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "217fdf91-33a7-40e2-a948-51b05ed63031", 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 44, 2024-02-10T08:47:21.571 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6808", 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.572 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6809", 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6811", 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:47:21.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6810", 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1417479017 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6808/1417479017", 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6809/1417479017", 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6811/1417479017", 2024-02-10T08:47:21.574 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6810/1417479017", 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f0cd42da-4080-4e6a-843b-d73e10ce0948", 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2024-02-10T08:47:21.575 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 52, 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 681, 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6812", 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2024-02-10T08:47:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6813", 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6815", 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:47:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.195:6814", 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1934176928 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.195:6812/1934176928", 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.195:6813/1934176928", 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.195:6815/1934176928", 2024-02-10T08:47:21.578 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.195:6814/1934176928", 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.579 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:56.742397+0000", 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 65536, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:58.759146+0000", 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:52.941341+0000", 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:51.821954+0000", 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:27.908579+0000", 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 479 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:52.282084+0000", 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.583 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:44.037149+0000", 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138314316775423, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2024-02-10T08:47:21.584 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2024-02-10T08:46:31.350970+0000", 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [ 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.b", 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: 6 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.12", 2024-02-10T08:47:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 7, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 1 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.13", 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "2.1d", 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 3, 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.1", 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 4, 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 0, 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "6.1", 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2, 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: 1, 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: 7 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [ 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "16.d", 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: "mappings": [ 2024-02-10T08:47:21.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "from": 0, 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "to": 4 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklist": { 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1675475667": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/682167559": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/3360068681": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/2916003802": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:47:21.589 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1214732392": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/810931708": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/1610921635": "2024-02-11T08:28:07.029499+0000", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:0/942960899": "2024-02-11T08:33:39.974787+0000", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/2081774553": "2024-02-11T08:28:29.992672+0000", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "172.21.15.107:6800/4244374259": "2024-02-11T08:28:07.029499+0000" 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "range_blocklist": {}, 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:47:21.590 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "testprofile-test-rados-api-smithi107-112490-101": { 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [ 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 23, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.592 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 9, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 27, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 57, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.594 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 77, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 3, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 83, 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 93, 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 95, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 4 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 106, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 111, 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 5, 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 7, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 124, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 10, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 2 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 132, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 12, 2024-02-10T08:47:21.601 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 154, 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 14, 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 164, 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 17, 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.603 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 176, 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 20, 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.604 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 237, 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 2, 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 238, 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 4, 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 241, 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: "snaps": [ 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: "begin": 6, 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: "length": 1 2024-02-10T08:47:21.607 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {}, 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode": { 2024-02-10T08:47:21.608 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_enabled": false, 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_bucket_count": 0, 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout: "degraded_stretch_mode": 0, 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout: "recovering_stretch_mode": 0, 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout: "stretch_mode_bucket": 0 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:47:21.609 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:47:21.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:21.610 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.521044+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:47:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.521841+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:47:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.865032+0000 osd.5 (osd.5) 282 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:47:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:19.868855+0000 osd.5 (osd.5) 283 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:47:21.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:20.641326+0000 mon.a (mon.0) 3299 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:21.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.521044+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:47:21.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.521841+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:47:21.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.865032+0000 osd.5 (osd.5) 282 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:47:21.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:19.868855+0000 osd.5 (osd.5) 283 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:47:21.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:20.641326+0000 mon.a (mon.0) 3299 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:21.630 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.521044+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:47:21.631 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.521841+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:47:21.631 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.865032+0000 osd.5 (osd.5) 282 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:47:21.631 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:19.868855+0000 osd.5 (osd.5) 283 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:47:21.631 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:20.641326+0000 mon.a (mon.0) 3299 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:21.643 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.664 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:21.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:21.799 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.819 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:21.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:21.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:21.972 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:21.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:22.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:22.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:47:22.104 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:22.122 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:22.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:22.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:22.275 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:22.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:22.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:21.448836+0000 mon.a (mon.0) 3300 : audit [DBG] from='client.? v1:172.21.15.107:0/3754312630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:47:22.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.542214+0000 mgr.y (mgr.14553) 575 : cluster [DBG] pgmap v1035: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 465 B/s wr, 1 op/s 2024-02-10T08:47:22.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:21.729539+0000 mgr.y (mgr.14553) 576 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:22.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.934730+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:47:22.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.938267+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:47:22.376 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:21.448836+0000 mon.a (mon.0) 3300 : audit [DBG] from='client.? v1:172.21.15.107:0/3754312630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:47:22.376 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.542214+0000 mgr.y (mgr.14553) 575 : cluster [DBG] pgmap v1035: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 465 B/s wr, 1 op/s 2024-02-10T08:47:22.376 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:21.729539+0000 mgr.y (mgr.14553) 576 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:22.376 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.934730+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:47:22.376 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.938267+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:47:22.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:22.408 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:22.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:21.448836+0000 mon.a (mon.0) 3300 : audit [DBG] from='client.? v1:172.21.15.107:0/3754312630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T08:47:22.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.542214+0000 mgr.y (mgr.14553) 575 : cluster [DBG] pgmap v1035: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 465 B/s wr, 1 op/s 2024-02-10T08:47:22.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:21.729539+0000 mgr.y (mgr.14553) 576 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:22.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.934730+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:47:22.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.938267+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:47:22.701 INFO:tasks.thrashosds.thrasher:j is {'epoch': 685, 'fsid': '275c2754-c7ee-11ee-95b8-87774f69a715', 'created': '2024-02-10T08:27:44.316161+0000', 'modified': '2024-02-10T08:47:20.142829+0000', 'last_up_change': '2024-02-10T08:46:28.181193+0000', 'last_in_change': '2024-02-10T08:47:07.143327+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 108, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 243, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-10T08:31:09.850435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '552', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T08:33:16.728206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-10T08:33:16.960794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T08:33:18.379465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T08:33:20.354391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 135, 'last_epoch_started': 134, 'last_epoch_clean': 134, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '135', 'last_force_op_resend_preluminous': '133', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T08:33:22.423222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-10T08:36:32.714443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi107-112490-101', 'create_time': '2024-02-10T08:45:44.375681+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '684', 'last_force_op_resend': '684', 'last_force_op_resend_prenautilus': '684', 'last_force_op_resend_preluminous': '684', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 683, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [243], 'tier_of': -1, 'read_tier': 243, 'write_tier': 243, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi107-112490-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 243, 'pool_name': 'test-rados-api-smithi107-112490-118', 'create_time': '2024-02-10T08:47:15.496669+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '685', 'last_force_op_resend': '684', 'last_force_op_resend_prenautilus': '684', 'last_force_op_resend_preluminous': '684', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 683, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e30a7307-1dfd-4dfa-b4c4-89b83eb24167', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 3321596972}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6802', 'nonce': 3321596972}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6804', 'nonce': 3321596972}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 3321596972}]}, 'public_addr': '172.21.15.107:6801/3321596972', 'cluster_addr': '172.21.15.107:6802/3321596972', 'heartbeat_back_addr': '172.21.15.107:6804/3321596972', 'heartbeat_front_addr': '172.21.15.107:6803/3321596972', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5fe7280-6de7-4c89-a9f2-22eebe4c1c80', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 663, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 133516126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6806', 'nonce': 133516126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6808', 'nonce': 133516126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 133516126}]}, 'public_addr': '172.21.15.107:6805/133516126', 'cluster_addr': '172.21.15.107:6806/133516126', 'heartbeat_back_addr': '172.21.15.107:6808/133516126', 'heartbeat_front_addr': '172.21.15.107:6807/133516126', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3ae7131b-69d5-4cbd-b1ca-29556fac7fbf', 'up': 1, 'in': 1, 'weight': 0.9418182373046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 3575881728}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6810', 'nonce': 3575881728}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6812', 'nonce': 3575881728}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 3575881728}]}, 'public_addr': '172.21.15.107:6809/3575881728', 'cluster_addr': '172.21.15.107:6810/3575881728', 'heartbeat_back_addr': '172.21.15.107:6812/3575881728', 'heartbeat_front_addr': '172.21.15.107:6811/3575881728', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1ffb2633-18bb-4048-90eb-7130e0cc7d43', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 2871825691}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6814', 'nonce': 2871825691}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6816', 'nonce': 2871825691}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 2871825691}]}, 'public_addr': '172.21.15.107:6813/2871825691', 'cluster_addr': '172.21.15.107:6814/2871825691', 'heartbeat_back_addr': '172.21.15.107:6816/2871825691', 'heartbeat_front_addr': '172.21.15.107:6815/2871825691', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b7529a6-19fc-4435-be7f-28876750838a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 479, 'up_from': 633, 'up_thru': 681, 'down_at': 480, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6800', 'nonce': 3310789251}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6801', 'nonce': 3310789251}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 3310789251}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6802', 'nonce': 3310789251}]}, 'public_addr': '172.21.15.195:6800/3310789251', 'cluster_addr': '172.21.15.195:6801/3310789251', 'heartbeat_back_addr': '172.21.15.195:6803/3310789251', 'heartbeat_front_addr': '172.21.15.195:6802/3310789251', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bdc1b91c-94ce-4287-8019-535b326da3e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6804', 'nonce': 4178321486}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 4178321486}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 4178321486}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6806', 'nonce': 4178321486}]}, 'public_addr': '172.21.15.195:6804/4178321486', 'cluster_addr': '172.21.15.195:6805/4178321486', 'heartbeat_back_addr': '172.21.15.195:6807/4178321486', 'heartbeat_front_addr': '172.21.15.195:6806/4178321486', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '217fdf91-33a7-40e2-a948-51b05ed63031', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6808', 'nonce': 1417479017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 1417479017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 1417479017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6810', 'nonce': 1417479017}]}, 'public_addr': '172.21.15.195:6808/1417479017', 'cluster_addr': '172.21.15.195:6809/1417479017', 'heartbeat_back_addr': '172.21.15.195:6811/1417479017', 'heartbeat_front_addr': '172.21.15.195:6810/1417479017', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f0cd42da-4080-4e6a-843b-d73e10ce0948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6812', 'nonce': 1934176928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 1934176928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 1934176928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.195:6814', 'nonce': 1934176928}]}, 'public_addr': '172.21.15.195:6812/1934176928', 'cluster_addr': '172.21.15.195:6813/1934176928', 'heartbeat_back_addr': '172.21.15.195:6815/1934176928', 'heartbeat_front_addr': '172.21.15.195:6814/1934176928', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:56.742397+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T08:46:58.759146+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:52.941341+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:51.821954+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:27.908579+0000', 'dead_epoch': 479}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:52.282084+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:44.037149+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T08:46:31.350970+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.b', 'osds': [3, 1, 6]}, {'pgid': '2.12', 'osds': [3, 7, 1]}, {'pgid': '2.13', 'osds': [4, 2, 7]}, {'pgid': '2.1d', 'osds': [3, 4, 2]}, {'pgid': '4.1', 'osds': [4, 0, 7]}, {'pgid': '6.1', 'osds': [2, 1, 7]}], 'pg_upmap_items': [{'pgid': '16.d', 'mappings': [{'from': 0, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.107:0/1675475667': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/682167559': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/3360068681': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:0/2916003802': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1214732392': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/810931708': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:0/1610921635': '2024-02-11T08:28:07.029499+0000', '172.21.15.107:0/942960899': '2024-02-11T08:33:39.974787+0000', '172.21.15.107:6800/2081774553': '2024-02-11T08:28:29.992672+0000', '172.21.15.107:6800/4244374259': '2024-02-11T08:28:07.029499+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi107-112490-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 23, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 27, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 77, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 83, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 106, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 111, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T08:47:22.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:47:22.957 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:22.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:23.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:23.094 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:23.115 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:23.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:23.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:23.247 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:23.444 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:23.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.544094+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:47:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.545073+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:47:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.616714+0000 osd.6 (osd.6) 282 : cluster [DBG] 2.7 deep-scrub starts 2024-02-10T08:47:23.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:21.617897+0000 osd.6 (osd.6) 283 : cluster [DBG] 2.7 deep-scrub ok 2024-02-10T08:47:23.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:23.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.544094+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.545073+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.616714+0000 osd.6 (osd.6) 282 : cluster [DBG] 2.7 deep-scrub starts 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:21.617897+0000 osd.6 (osd.6) 283 : cluster [DBG] 2.7 deep-scrub ok 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.544094+0000 osd.7 (osd.7) 274 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.545073+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.616714+0000 osd.6 (osd.6) 282 : cluster [DBG] 2.7 deep-scrub starts 2024-02-10T08:47:23.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:21.617897+0000 osd.6 (osd.6) 283 : cluster [DBG] 2.7 deep-scrub ok 2024-02-10T08:47:23.575 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:24.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:47:24.493 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:24.494 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:24.529 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:47:24.530 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:47:24.530 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:47:24.530 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:22.549720+0000 osd.7 (osd.7) 276 : cluster [DBG] 234.7 deep-scrub starts 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:22.551819+0000 osd.7 (osd.7) 277 : cluster [DBG] 234.7 deep-scrub ok 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:22.629344+0000 osd.6 (osd.6) 284 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:22.632344+0000 osd.6 (osd.6) 285 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.542794+0000 mgr.y (mgr.14553) 577 : cluster [DBG] pgmap v1036: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.916290+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:47:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.919466+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:47:24.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:22.549720+0000 osd.7 (osd.7) 276 : cluster [DBG] 234.7 deep-scrub starts 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:22.551819+0000 osd.7 (osd.7) 277 : cluster [DBG] 234.7 deep-scrub ok 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:22.629344+0000 osd.6 (osd.6) 284 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:22.632344+0000 osd.6 (osd.6) 285 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.542794+0000 mgr.y (mgr.14553) 577 : cluster [DBG] pgmap v1036: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.916290+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.919466+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:22.549720+0000 osd.7 (osd.7) 276 : cluster [DBG] 234.7 deep-scrub starts 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:22.551819+0000 osd.7 (osd.7) 277 : cluster [DBG] 234.7 deep-scrub ok 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:22.629344+0000 osd.6 (osd.6) 284 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:22.632344+0000 osd.6 (osd.6) 285 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:47:24.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.542794+0000 mgr.y (mgr.14553) 577 : cluster [DBG] pgmap v1036: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-10T08:47:24.550 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.916290+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:47:24.550 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.919466+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.597141+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.598778+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.667885+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:23.671537+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.041779+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:47:25.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.043508+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:47:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.364148+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:47:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.365329+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:47:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:25.169956+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:25.210732+0000 mon.b (mon.1) 775 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:25.211048+0000 mon.a (mon.0) 3302 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.597141+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:47:25.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.598778+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.667885+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:23.671537+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.041779+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.043508+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.364148+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.365329+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:25.169956+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:25.210732+0000 mon.b (mon.1) 775 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:25.211048+0000 mon.a (mon.0) 3302 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.597141+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.598778+0000 osd.7 (osd.7) 279 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.667885+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.12 deep-scrub starts 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:23.671537+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.12 deep-scrub ok 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.041779+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.0 deep-scrub starts 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.043508+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.0 deep-scrub ok 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.364148+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.365329+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:25.169956+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:25.210732+0000 mon.b (mon.1) 775 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:25.211048+0000 mon.a (mon.0) 3302 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:25.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:25.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:47:25.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:25] "GET /metrics HTTP/1.1" 200 190338 "" "Prometheus/2.33.4" 2024-02-10T08:47:25.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:25.964 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:25.984 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:25.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:26.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:26.118 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:26.137 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:26.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:26.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:26.267 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:26.285 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:26.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:26.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.804949+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:47:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:24.808419+0000 osd.5 (osd.5) 285 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:47:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:25.543372+0000 mgr.y (mgr.14553) 578 : cluster [DBG] pgmap v1037: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 830 B/s rd, 276 B/s wr, 1 op/s 2024-02-10T08:47:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:25.598521+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:47:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:25.601668+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:25.651074+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:25.753176+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:25.754224+0000 osd.6 (osd.6) 289 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:26.225151+0000 mon.a (mon.0) 3304 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.225235+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:26.225768+0000 mon.b (mon.1) 776 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:26.226035+0000 mon.a (mon.0) 3306 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.804949+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:24.808419+0000 osd.5 (osd.5) 285 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:25.543372+0000 mgr.y (mgr.14553) 578 : cluster [DBG] pgmap v1037: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 830 B/s rd, 276 B/s wr, 1 op/s 2024-02-10T08:47:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:25.598521+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:25.601668+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:25.651074+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:25.753176+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:25.754224+0000 osd.6 (osd.6) 289 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:26.225151+0000 mon.a (mon.0) 3304 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.225235+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:26.225768+0000 mon.b (mon.1) 776 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:26.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:26.226035+0000 mon.a (mon.0) 3306 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:26.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:26.882 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:26.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.804949+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:24.808419+0000 osd.5 (osd.5) 285 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:25.543372+0000 mgr.y (mgr.14553) 578 : cluster [DBG] pgmap v1037: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 830 B/s rd, 276 B/s wr, 1 op/s 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:25.598521+0000 osd.7 (osd.7) 280 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:25.601668+0000 osd.7 (osd.7) 281 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:47:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:25.651074+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:25.753176+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:25.754224+0000 osd.6 (osd.6) 289 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:26.225151+0000 mon.a (mon.0) 3304 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.225235+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:26.225768+0000 mon.b (mon.1) 776 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:27.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:26.226035+0000 mon.a (mon.0) 3306 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]: dispatch 2024-02-10T08:47:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:27.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:27.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:27.259 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:27.280 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:27.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:27.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:27.413 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.040638+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:47:27.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.043544+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.362312+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.363334+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.748648+0000 osd.6 (osd.6) 290 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.751528+0000 osd.6 (osd.6) 291 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.778295+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.780321+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.932266+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:26.935754+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.220184+0000 mon.a (mon.0) 3307 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:27.226576+0000 mon.a (mon.0) 3308 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:27.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.226645+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-10T08:47:27.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.040638+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:47:27.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.043544+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:47:27.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.362312+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:47:27.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.363334+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:47:27.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.748648+0000 osd.6 (osd.6) 290 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:47:27.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.751528+0000 osd.6 (osd.6) 291 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:47:27.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.778295+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:47:27.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.780321+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:47:27.935 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.932266+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-02-10T08:47:27.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:26.935754+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-02-10T08:47:27.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.220184+0000 mon.a (mon.0) 3307 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:27.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:27.226576+0000 mon.a (mon.0) 3308 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:27.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.226645+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.040638+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.043544+0000 osd.2 (osd.2) 262 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.362312+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.363334+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.748648+0000 osd.6 (osd.6) 290 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:47:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.751528+0000 osd.6 (osd.6) 291 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.778295+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.780321+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.932266+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.d scrub starts 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:26.935754+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.d scrub ok 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.220184+0000 mon.a (mon.0) 3307 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:27.226576+0000 mon.a (mon.0) 3308 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-118"}]': finished 2024-02-10T08:47:28.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.226645+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2024-02-10T08:47:28.063 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:28.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:28.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:28.198 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:28.296 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:28.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:28.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:28.430 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:28.450 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:28.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:28.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:28.582 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:28.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.336141+0000 osd.0 (osd.0) 129 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.336911+0000 osd.0 (osd.0) 130 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.543818+0000 mgr.y (mgr.14553) 579 : cluster [DBG] pgmap v1040: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 276 B/s wr, 2 op/s 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.939369+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:27.942856+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:47:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:28.227834+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-10T08:47:28.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.336141+0000 osd.0 (osd.0) 129 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.336911+0000 osd.0 (osd.0) 130 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.543818+0000 mgr.y (mgr.14553) 579 : cluster [DBG] pgmap v1040: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 276 B/s wr, 2 op/s 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.939369+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:27.942856+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:28.227834+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-10T08:47:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.336141+0000 osd.0 (osd.0) 129 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:47:28.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.336911+0000 osd.0 (osd.0) 130 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:47:28.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.543818+0000 mgr.y (mgr.14553) 579 : cluster [DBG] pgmap v1040: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 276 B/s wr, 2 op/s 2024-02-10T08:47:28.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.939369+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:47:28.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:27.942856+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:47:28.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:28.227834+0000 mon.a (mon.0) 3310 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2024-02-10T08:47:28.939 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:28.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:29.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:29.074 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:29.095 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:29.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:29.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:29.227 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.297+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2749610704 >> v1:172.21.15.107:6789/0 conn(0x7ff7b00f94d0 legacy=0x7ff7b0102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.297+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2749610704 shutdown_connections 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.297+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2749610704 >> 172.21.15.107:0/2749610704 conn(0x7ff7b00f2550 msgr2=0x7ff7b00f4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.297+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2749610704 shutdown_connections 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.297+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2749610704 wait complete. 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 librados: starting msgr at 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 librados: starting objecter 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 Processor -- start 2024-02-10T08:47:29.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 -- start start 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 librados: setting wanted keys 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 librados: calling monclient init 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b0117400 con 0x7ff7b00f89e0 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b01185e0 con 0x7ff7b0104340 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7b69f1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b01197e0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7affff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff7b00f89e0 0x7ff7b00ff730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60186/0 (socket says 172.21.15.107:60186) 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.298+0000 7ff7affff700 1 -- 172.21.15.107:0/2868614894 learned_addr learned my addr 172.21.15.107:0/2868614894 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter ms_handle_connect 0x7ff7b00f89e0 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter ms_handle_connect 0x7ff7b00f94d0 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter ms_handle_connect 0x7ff7b0104340 2024-02-10T08:47:29.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441381176 0 0) 0x7ff7b0117400 con 0x7ff7b00f89e0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff79c003030 con 0x7ff7b00f89e0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724477795 0 0) 0x7ff7b01197e0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b0117400 con 0x7ff7b00f94d0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031878244 0 0) 0x7ff7b01185e0 con 0x7ff7b0104340 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b01197e0 con 0x7ff7b0104340 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236466597 0 0) 0x7ff7b0117400 con 0x7ff7b00f94d0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7b01185e0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff798001510 con 0x7ff7b00f94d0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398650404 0 0) 0x7ff79c003030 con 0x7ff7b00f89e0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7b0117400 con 0x7ff7b00f89e0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff7a0001920 con 0x7ff7b00f89e0 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3256215384 0 0) 0x7ff7b01197e0 con 0x7ff7b0104340 2024-02-10T08:47:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff79c003030 con 0x7ff7b0104340 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff7a8001b80 con 0x7ff7b0104340 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1393378044 0 0) 0x7ff7b01185e0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 >> v1:172.21.15.195:6789/0 conn(0x7ff7b0104340 legacy=0x7ff7b0115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 >> v1:172.21.15.107:6789/0 conn(0x7ff7b00f89e0 legacy=0x7ff7b00ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b011a9e0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b01199d0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7b69f1700 10 client.54179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b0119e40 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7b69f1700 1 librados: init done 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff798002280 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.299+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff798003740 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.300+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790004f20 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff7980017f0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(689..689 src has 420..689) v4 ==== 8556+0+0 (unknown 961195005 0 0) 0x7ff798076ac0 con 0x7ff7b00f94d0 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 10 client.54179.objecter ms_dispatch 0x7ff7b0001550 osd_map(689..689 src has 420..689) v4 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 3 client.54179.objecter handle_osd_map got epochs [689,689] > 0 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 3 client.54179.objecter handle_osd_map decoding full epoch 689 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 20 client.54179.objecter dump_active .. 0 homeless 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.301+0000 7ff7ad7fa700 10 client.54179.objecter ms_handle_connect 0x7ff79c063c70 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.305+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff798049d40 con 0x7ff7b00f94d0 2024-02-10T08:47:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:29.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:29.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:29.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.474+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff790000b90 con 0x7ff79c063c70 2024-02-10T08:47:29.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.488+0000 7ff7ad7fa700 1 -- 172.21.15.107:0/2868614894 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+493408 (unknown 2965378022 0 1796591249) 0x7ff790000b90 con 0x7ff79c063c70 2024-02-10T08:47:29.496 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:47:29.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 10 librados: watch_flush enter 2024-02-10T08:47:29.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 10 librados: watch_flush exit 2024-02-10T08:47:29.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 20 client.54179.objecter shutdown clearing up homeless session... 2024-02-10T08:47:29.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 10 client.54179.objecter successfully canceled tick 2024-02-10T08:47:29.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff79c063c70 legacy=0x7ff79c066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:29.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 >> v1:172.21.15.107:6790/0 conn(0x7ff7b00f94d0 legacy=0x7ff7b00ffe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 shutdown_connections 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 >> 172.21.15.107:0/2868614894 conn(0x7ff7b00f2550 msgr2=0x7ff7b01077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 shutdown_connections 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 -- 172.21.15.107:0/2868614894 wait complete. 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:29.491+0000 7ff7b69f1700 1 librados: shutdown 2024-02-10T08:47:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:47:29.797 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:29.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:28.695194+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:29.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:28.695464+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:28.926468+0000 osd.4 (osd.4) 41 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:28.929443+0000 osd.4 (osd.4) 42 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.231931+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:29.240903+0000 mon.b (mon.1) 777 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:29.241475+0000 mon.a (mon.0) 3312 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:28.695194+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:28.695464+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:28.926468+0000 osd.4 (osd.4) 41 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:28.929443+0000 osd.4 (osd.4) 42 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.231931+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-10T08:47:29.914 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:29.240903+0000 mon.b (mon.1) 777 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:29.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:29.241475+0000 mon.a (mon.0) 3312 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:28.695194+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:28.695464+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:30.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:28.926468+0000 osd.4 (osd.4) 41 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:47:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:28.929443+0000 osd.4 (osd.4) 42 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:47:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.231931+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2024-02-10T08:47:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:29.240903+0000 mon.b (mon.1) 777 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:30.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:29.241475+0000 mon.a (mon.0) 3312 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:30.182 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1042,"stamp":"2024-02-10T08:47:29.229126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5872,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1253,"num_read_kb":996,"num_write":747,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":11379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":889,"ondisk_log_size":889,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":582,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2185320,"kb_used_data":33200,"kb_used_omap":59,"kb_used_meta":2152004,"kb_avail":654030744,"statfs":{"total":671965249536,"available":669727481856,"internally_reserved":0,"allocated":33996800,"data_stored":17713734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2203652715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-1,"num_flush_kb":-1,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.686640"},"pg_stats":[{"pgid":"244.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"244.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"669'16","reported_seq":134,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:22.551914+0000","last_change":"2024-02-10T08:47:22.551914+0000","last_active":"2024-02-10T08:47:22.551914+0000","last_peered":"2024-02-10T08:47:22.551914+0000","last_clean":"2024-02-10T08:47:22.551914+0000","last_became_active":"2024-02-10T08:45:46.132344+0000","last_became_peered":"2024-02-10T08:45:46.132344+0000","last_unstale":"2024-02-10T08:47:22.551914+0000","last_undegraded":"2024-02-10T08:47:22.551914+0000","last_fullsized":"2024-02-10T08:47:22.551914+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'16","last_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_deep_scrub":"669'16","last_deep_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_clean_scrub_stamp":"2024-02-10T08:47:22.551825+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"244.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"601'1","reported_seq":127,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414970+0000","last_change":"2024-02-10T08:47:18.549703+0000","last_active":"2024-02-10T08:47:19.414970+0000","last_peered":"2024-02-10T08:47:19.414970+0000","last_clean":"2024-02-10T08:47:19.414970+0000","last_became_active":"2024-02-10T08:47:08.172139+0000","last_became_peered":"2024-02-10T08:47:08.172139+0000","last_unstale":"2024-02-10T08:47:19.414970+0000","last_undegraded":"2024-02-10T08:47:19.414970+0000","last_fullsized":"2024-02-10T08:47:19.414970+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'1","last_scrub_stamp":"2024-02-10T08:47:18.549603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.549603+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"244.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"634'5","reported_seq":139,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146177+0000","last_change":"2024-02-10T08:47:04.278039+0000","last_active":"2024-02-10T08:47:19.146177+0000","last_peered":"2024-02-10T08:47:19.146177+0000","last_clean":"2024-02-10T08:47:19.146177+0000","last_became_active":"2024-02-10T08:46:29.211292+0000","last_became_peered":"2024-02-10T08:46:29.211292+0000","last_unstale":"2024-02-10T08:47:19.146177+0000","last_undegraded":"2024-02-10T08:47:19.146177+0000","last_fullsized":"2024-02-10T08:47:19.146177+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"244.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"677'15","reported_seq":150,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411411+0000","last_change":"2024-02-10T08:47:08.644492+0000","last_active":"2024-02-10T08:47:19.411411+0000","last_peered":"2024-02-10T08:47:19.411411+0000","last_clean":"2024-02-10T08:47:19.411411+0000","last_became_active":"2024-02-10T08:47:08.180514+0000","last_became_peered":"2024-02-10T08:47:08.180514+0000","last_unstale":"2024-02-10T08:47:19.411411+0000","last_undegraded":"2024-02-10T08:47:19.411411+0000","last_fullsized":"2024-02-10T08:47:19.411411+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"244.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":107,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413431+0000","last_change":"2024-02-10T08:47:04.274343+0000","last_active":"2024-02-10T08:47:19.413431+0000","last_peered":"2024-02-10T08:47:19.413431+0000","last_clean":"2024-02-10T08:47:19.413431+0000","last_became_active":"2024-02-10T08:45:46.129572+0000","last_became_peered":"2024-02-10T08:45:46.129572+0000","last_unstale":"2024-02-10T08:47:19.413431+0000","last_undegraded":"2024-02-10T08:47:19.413431+0000","last_fullsized":"2024-02-10T08:47:19.413431+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"244.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":121,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:22.632435+0000","last_change":"2024-02-10T08:47:22.632435+0000","last_active":"2024-02-10T08:47:22.632435+0000","last_peered":"2024-02-10T08:47:22.632435+0000","last_clean":"2024-02-10T08:47:22.632435+0000","last_became_active":"2024-02-10T08:46:15.351881+0000","last_became_peered":"2024-02-10T08:46:15.351881+0000","last_unstale":"2024-02-10T08:47:22.632435+0000","last_undegraded":"2024-02-10T08:47:22.632435+0000","last_fullsized":"2024-02-10T08:47:22.632435+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:22.632348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:47:22.632348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"244.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"608'1","reported_seq":130,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411041+0000","last_change":"2024-02-10T08:47:04.269661+0000","last_active":"2024-02-10T08:47:19.411041+0000","last_peered":"2024-02-10T08:47:19.411041+0000","last_clean":"2024-02-10T08:47:19.411041+0000","last_became_active":"2024-02-10T08:46:29.219928+0000","last_became_peered":"2024-02-10T08:46:29.219928+0000","last_unstale":"2024-02-10T08:47:19.411041+0000","last_undegraded":"2024-02-10T08:47:19.411041+0000","last_fullsized":"2024-02-10T08:47:19.411041+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"244.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:47:29.225756+0000","last_change":"2024-02-10T08:47:29.225756+0000","last_active":"2024-02-10T08:47:29.225756+0000","last_peered":"2024-02-10T08:47:29.225756+0000","last_clean":"2024-02-10T08:47:29.225756+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:29.225756+0000","last_undegraded":"2024-02-10T08:47:29.225756+0000","last_fullsized":"2024-02-10T08:47:29.225756+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.225756+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.225756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"685'16","reported_seq":93,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:25.185247+0000","last_change":"2024-02-10T08:47:06.269921+0000","last_active":"2024-02-10T08:47:25.185247+0000","last_peered":"2024-02-10T08:47:25.185247+0000","last_clean":"2024-02-10T08:47:25.185247+0000","last_became_active":"2024-02-10T08:46:29.210532+0000","last_became_peered":"2024-02-10T08:46:29.210532+0000","last_unstale":"2024-02-10T08:47:25.185247+0000","last_undegraded":"2024-02-10T08:47:25.185247+0000","last_fullsized":"2024-02-10T08:47:25.185247+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"636'18","reported_seq":351,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.165156+0000","last_change":"2024-02-10T08:46:40.601578+0000","last_active":"2024-02-10T08:47:20.165156+0000","last_peered":"2024-02-10T08:47:20.165156+0000","last_clean":"2024-02-10T08:47:20.165156+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:47:20.165156+0000","last_undegraded":"2024-02-10T08:47:20.165156+0000","last_fullsized":"2024-02-10T08:47:20.165156+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:46:40.601510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.601510+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":753,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412619+0000","last_change":"2024-02-10T08:45:22.037140+0000","last_active":"2024-02-10T08:47:19.412619+0000","last_peered":"2024-02-10T08:47:19.412619+0000","last_clean":"2024-02-10T08:47:19.412619+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:47:19.412619+0000","last_undegraded":"2024-02-10T08:47:19.412619+0000","last_fullsized":"2024-02-10T08:47:19.412619+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:45:22.037068+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.037068+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":681,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145567+0000","last_change":"2024-02-10T08:45:22.422467+0000","last_active":"2024-02-10T08:47:19.145567+0000","last_peered":"2024-02-10T08:47:19.145567+0000","last_clean":"2024-02-10T08:47:19.145567+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:47:19.145567+0000","last_undegraded":"2024-02-10T08:47:19.145567+0000","last_fullsized":"2024-02-10T08:47:19.145567+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.422382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.422382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":672,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416046+0000","last_change":"2024-02-10T08:46:15.332578+0000","last_active":"2024-02-10T08:47:19.416046+0000","last_peered":"2024-02-10T08:47:19.416046+0000","last_clean":"2024-02-10T08:47:19.416046+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:47:19.416046+0000","last_undegraded":"2024-02-10T08:47:19.416046+0000","last_fullsized":"2024-02-10T08:47:19.416046+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:59.660566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:59.660566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":618,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146581+0000","last_change":"2024-02-10T08:47:08.181764+0000","last_active":"2024-02-10T08:47:19.146581+0000","last_peered":"2024-02-10T08:47:19.146581+0000","last_clean":"2024-02-10T08:47:19.146581+0000","last_became_active":"2024-02-10T08:47:08.181395+0000","last_became_peered":"2024-02-10T08:47:08.181395+0000","last_unstale":"2024-02-10T08:47:19.146581+0000","last_undegraded":"2024-02-10T08:47:19.146581+0000","last_fullsized":"2024-02-10T08:47:19.146581+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:47.472086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:47.472086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":114,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148828+0000","last_change":"2024-02-10T08:47:08.352070+0000","last_active":"2024-02-10T08:47:19.148828+0000","last_peered":"2024-02-10T08:47:19.148828+0000","last_clean":"2024-02-10T08:47:19.148828+0000","last_became_active":"2024-02-10T08:47:08.168050+0000","last_became_peered":"2024-02-10T08:47:08.168050+0000","last_unstale":"2024-02-10T08:47:19.148828+0000","last_undegraded":"2024-02-10T08:47:19.148828+0000","last_fullsized":"2024-02-10T08:47:19.148828+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:46:32.979941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.979941+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147280+0000","last_change":"2024-02-10T08:47:08.170117+0000","last_active":"2024-02-10T08:47:20.147280+0000","last_peered":"2024-02-10T08:47:20.147280+0000","last_clean":"2024-02-10T08:47:20.147280+0000","last_became_active":"2024-02-10T08:47:08.169730+0000","last_became_peered":"2024-02-10T08:47:08.169730+0000","last_unstale":"2024-02-10T08:47:20.147280+0000","last_undegraded":"2024-02-10T08:47:20.147280+0000","last_fullsized":"2024-02-10T08:47:20.147280+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.584151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.584151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":692,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.144919+0000","last_change":"2024-02-10T08:46:31.356749+0000","last_active":"2024-02-10T08:47:19.144919+0000","last_peered":"2024-02-10T08:47:19.144919+0000","last_clean":"2024-02-10T08:47:19.144919+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:47:19.144919+0000","last_undegraded":"2024-02-10T08:47:19.144919+0000","last_fullsized":"2024-02-10T08:47:19.144919+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.356675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.356675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148024+0000","last_change":"2024-02-10T08:47:17.396762+0000","last_active":"2024-02-10T08:47:19.148024+0000","last_peered":"2024-02-10T08:47:19.148024+0000","last_clean":"2024-02-10T08:47:19.148024+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:47:19.148024+0000","last_undegraded":"2024-02-10T08:47:19.148024+0000","last_fullsized":"2024-02-10T08:47:19.148024+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.396696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.396696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":621,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.868966+0000","last_change":"2024-02-10T08:47:19.868939+0000","last_active":"2024-02-10T08:47:19.868966+0000","last_peered":"2024-02-10T08:47:19.868966+0000","last_clean":"2024-02-10T08:47:19.868966+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:47:19.868966+0000","last_undegraded":"2024-02-10T08:47:19.868966+0000","last_fullsized":"2024-02-10T08:47:19.868966+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:19.868859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.868859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":815,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411819+0000","last_change":"2024-02-10T08:47:09.412695+0000","last_active":"2024-02-10T08:47:19.411819+0000","last_peered":"2024-02-10T08:47:19.411819+0000","last_clean":"2024-02-10T08:47:19.411819+0000","last_became_active":"2024-02-10T08:47:08.188007+0000","last_became_peered":"2024-02-10T08:47:08.188007+0000","last_unstale":"2024-02-10T08:47:19.411819+0000","last_undegraded":"2024-02-10T08:47:19.411819+0000","last_fullsized":"2024-02-10T08:47:19.411819+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.661956+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":679,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146931+0000","last_change":"2024-02-10T08:46:15.374244+0000","last_active":"2024-02-10T08:47:19.146931+0000","last_peered":"2024-02-10T08:47:19.146931+0000","last_clean":"2024-02-10T08:47:19.146931+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:47:19.146931+0000","last_undegraded":"2024-02-10T08:47:19.146931+0000","last_fullsized":"2024-02-10T08:47:19.146931+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.452719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":713,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145308+0000","last_change":"2024-02-10T08:46:15.367673+0000","last_active":"2024-02-10T08:47:19.145308+0000","last_peered":"2024-02-10T08:47:19.145308+0000","last_clean":"2024-02-10T08:47:19.145308+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:47:19.145308+0000","last_undegraded":"2024-02-10T08:47:19.145308+0000","last_fullsized":"2024-02-10T08:47:19.145308+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:24.427791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:24.427791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":237,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412778+0000","last_change":"2024-02-10T08:46:46.484714+0000","last_active":"2024-02-10T08:47:19.412778+0000","last_peered":"2024-02-10T08:47:19.412778+0000","last_clean":"2024-02-10T08:47:19.412778+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:47:19.412778+0000","last_undegraded":"2024-02-10T08:47:19.412778+0000","last_fullsized":"2024-02-10T08:47:19.412778+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.484656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416362+0000","last_change":"2024-02-10T08:46:31.456642+0000","last_active":"2024-02-10T08:47:19.416362+0000","last_peered":"2024-02-10T08:47:19.416362+0000","last_clean":"2024-02-10T08:47:19.416362+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:47:19.416362+0000","last_undegraded":"2024-02-10T08:47:19.416362+0000","last_fullsized":"2024-02-10T08:47:19.416362+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.456579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.456579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":595,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-02-10T08:47:26.935834+0000","last_change":"2024-02-10T08:47:26.935834+0000","last_active":"2024-02-10T08:47:26.935834+0000","last_peered":"2024-02-10T08:47:26.935834+0000","last_clean":"2024-02-10T08:47:26.935834+0000","last_became_active":"2024-02-10T08:47:08.169556+0000","last_became_peered":"2024-02-10T08:47:08.169556+0000","last_unstale":"2024-02-10T08:47:26.935834+0000","last_undegraded":"2024-02-10T08:47:26.935834+0000","last_fullsized":"2024-02-10T08:47:26.935834+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:47:26.935760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.935760+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416626+0000","last_change":"2024-02-10T08:47:11.591033+0000","last_active":"2024-02-10T08:47:19.416626+0000","last_peered":"2024-02-10T08:47:19.416626+0000","last_clean":"2024-02-10T08:47:19.416626+0000","last_became_active":"2024-02-10T08:47:08.181184+0000","last_became_peered":"2024-02-10T08:47:08.181184+0000","last_unstale":"2024-02-10T08:47:19.416626+0000","last_undegraded":"2024-02-10T08:47:19.416626+0000","last_fullsized":"2024-02-10T08:47:19.416626+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":672,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147463+0000","last_change":"2024-02-10T08:47:08.172418+0000","last_active":"2024-02-10T08:47:20.147463+0000","last_peered":"2024-02-10T08:47:20.147463+0000","last_clean":"2024-02-10T08:47:20.147463+0000","last_became_active":"2024-02-10T08:47:08.171202+0000","last_became_peered":"2024-02-10T08:47:08.171202+0000","last_unstale":"2024-02-10T08:47:20.147463+0000","last_undegraded":"2024-02-10T08:47:20.147463+0000","last_fullsized":"2024-02-10T08:47:20.147463+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.672468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.672468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":670,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146158+0000","last_change":"2024-02-10T08:46:32.371559+0000","last_active":"2024-02-10T08:47:19.146158+0000","last_peered":"2024-02-10T08:47:19.146158+0000","last_clean":"2024-02-10T08:47:19.146158+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:47:19.146158+0000","last_undegraded":"2024-02-10T08:47:19.146158+0000","last_fullsized":"2024-02-10T08:47:19.146158+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:32.371497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.371497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":468,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147328+0000","last_change":"2024-02-10T08:47:08.177103+0000","last_active":"2024-02-10T08:47:20.147328+0000","last_peered":"2024-02-10T08:47:20.147328+0000","last_clean":"2024-02-10T08:47:20.147328+0000","last_became_active":"2024-02-10T08:47:08.176768+0000","last_became_peered":"2024-02-10T08:47:08.176768+0000","last_unstale":"2024-02-10T08:47:20.147328+0000","last_undegraded":"2024-02-10T08:47:20.147328+0000","last_fullsized":"2024-02-10T08:47:20.147328+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:48.843209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.843209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":541,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147558+0000","last_change":"2024-02-10T08:46:36.873208+0000","last_active":"2024-02-10T08:47:20.147558+0000","last_peered":"2024-02-10T08:47:20.147558+0000","last_clean":"2024-02-10T08:47:20.147558+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:47:20.147558+0000","last_undegraded":"2024-02-10T08:47:20.147558+0000","last_fullsized":"2024-02-10T08:47:20.147558+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:46:36.873122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.873122+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":475,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411994+0000","last_change":"2024-02-10T08:47:08.180643+0000","last_active":"2024-02-10T08:47:19.411994+0000","last_peered":"2024-02-10T08:47:19.411994+0000","last_clean":"2024-02-10T08:47:19.411994+0000","last_became_active":"2024-02-10T08:47:08.180316+0000","last_became_peered":"2024-02-10T08:47:08.180316+0000","last_unstale":"2024-02-10T08:47:19.411994+0000","last_undegraded":"2024-02-10T08:47:19.411994+0000","last_fullsized":"2024-02-10T08:47:19.411994+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:49.582110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.582110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":909,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147724+0000","last_change":"2024-02-10T08:45:29.297349+0000","last_active":"2024-02-10T08:47:20.147724+0000","last_peered":"2024-02-10T08:47:20.147724+0000","last_clean":"2024-02-10T08:47:20.147724+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:47:20.147724+0000","last_undegraded":"2024-02-10T08:47:20.147724+0000","last_fullsized":"2024-02-10T08:47:20.147724+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:45:29.297292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":492,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147624+0000","last_change":"2024-02-10T08:47:08.179859+0000","last_active":"2024-02-10T08:47:20.147624+0000","last_peered":"2024-02-10T08:47:20.147624+0000","last_clean":"2024-02-10T08:47:20.147624+0000","last_became_active":"2024-02-10T08:47:08.179407+0000","last_became_peered":"2024-02-10T08:47:08.179407+0000","last_unstale":"2024-02-10T08:47:20.147624+0000","last_undegraded":"2024-02-10T08:47:20.147624+0000","last_fullsized":"2024-02-10T08:47:20.147624+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:37.823468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:37.823468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148208+0000","last_change":"2024-02-10T08:46:42.008821+0000","last_active":"2024-02-10T08:47:19.148208+0000","last_peered":"2024-02-10T08:47:19.148208+0000","last_clean":"2024-02-10T08:47:19.148208+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:47:19.148208+0000","last_undegraded":"2024-02-10T08:47:19.148208+0000","last_fullsized":"2024-02-10T08:47:19.148208+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:46:42.008755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":525,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412323+0000","last_change":"2024-02-10T08:47:08.207001+0000","last_active":"2024-02-10T08:47:19.412323+0000","last_peered":"2024-02-10T08:47:19.412323+0000","last_clean":"2024-02-10T08:47:19.412323+0000","last_became_active":"2024-02-10T08:47:08.174207+0000","last_became_peered":"2024-02-10T08:47:08.174207+0000","last_unstale":"2024-02-10T08:47:19.412323+0000","last_undegraded":"2024-02-10T08:47:19.412323+0000","last_fullsized":"2024-02-10T08:47:19.412323+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:46:41.670342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.670342+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":491,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416903+0000","last_change":"2024-02-10T08:47:13.515209+0000","last_active":"2024-02-10T08:47:19.416903+0000","last_peered":"2024-02-10T08:47:19.416903+0000","last_clean":"2024-02-10T08:47:19.416903+0000","last_became_active":"2024-02-10T08:47:08.176339+0000","last_became_peered":"2024-02-10T08:47:08.176339+0000","last_unstale":"2024-02-10T08:47:19.416903+0000","last_undegraded":"2024-02-10T08:47:19.416903+0000","last_fullsized":"2024-02-10T08:47:19.416903+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:47:13.515134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.515134+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":726,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-02-10T08:47:26.467729+0000","last_change":"2024-02-10T08:47:13.008447+0000","last_active":"2024-02-10T08:47:26.467729+0000","last_peered":"2024-02-10T08:47:26.467729+0000","last_clean":"2024-02-10T08:47:26.467729+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:47:26.467729+0000","last_undegraded":"2024-02-10T08:47:26.467729+0000","last_fullsized":"2024-02-10T08:47:26.467729+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:47:13.008374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.008374+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":676,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413487+0000","last_change":"2024-02-10T08:47:08.189575+0000","last_active":"2024-02-10T08:47:19.413487+0000","last_peered":"2024-02-10T08:47:19.413487+0000","last_clean":"2024-02-10T08:47:19.413487+0000","last_became_active":"2024-02-10T08:47:08.189274+0000","last_became_peered":"2024-02-10T08:47:08.189274+0000","last_unstale":"2024-02-10T08:47:19.413487+0000","last_undegraded":"2024-02-10T08:47:19.413487+0000","last_fullsized":"2024-02-10T08:47:19.413487+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:45:23.003138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":630,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414794+0000","last_change":"2024-02-10T08:47:17.824311+0000","last_active":"2024-02-10T08:47:19.414794+0000","last_peered":"2024-02-10T08:47:19.414794+0000","last_clean":"2024-02-10T08:47:19.414794+0000","last_became_active":"2024-02-10T08:47:08.181458+0000","last_became_peered":"2024-02-10T08:47:08.181458+0000","last_unstale":"2024-02-10T08:47:19.414794+0000","last_undegraded":"2024-02-10T08:47:19.414794+0000","last_fullsized":"2024-02-10T08:47:19.414794+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.824249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.824249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":563,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.148040+0000","last_change":"2024-02-10T08:47:08.607355+0000","last_active":"2024-02-10T08:47:20.148040+0000","last_peered":"2024-02-10T08:47:20.148040+0000","last_clean":"2024-02-10T08:47:20.148040+0000","last_became_active":"2024-02-10T08:47:08.166968+0000","last_became_peered":"2024-02-10T08:47:08.166968+0000","last_unstale":"2024-02-10T08:47:20.148040+0000","last_undegraded":"2024-02-10T08:47:20.148040+0000","last_fullsized":"2024-02-10T08:47:20.148040+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:46:45.859396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.859396+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":256,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412517+0000","last_change":"2024-02-10T08:47:08.182267+0000","last_active":"2024-02-10T08:47:19.412517+0000","last_peered":"2024-02-10T08:47:19.412517+0000","last_clean":"2024-02-10T08:47:19.412517+0000","last_became_active":"2024-02-10T08:47:08.181205+0000","last_became_peered":"2024-02-10T08:47:08.181205+0000","last_unstale":"2024-02-10T08:47:19.412517+0000","last_undegraded":"2024-02-10T08:47:19.412517+0000","last_fullsized":"2024-02-10T08:47:19.412517+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:28.372323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:28.372323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":818,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.161464+0000","last_change":"2024-02-10T08:46:38.853235+0000","last_active":"2024-02-10T08:47:20.161464+0000","last_peered":"2024-02-10T08:47:20.161464+0000","last_clean":"2024-02-10T08:47:20.161464+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:47:20.161464+0000","last_undegraded":"2024-02-10T08:47:20.161464+0000","last_fullsized":"2024-02-10T08:47:20.161464+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:38.853141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.853141+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":656,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411718+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:47:19.411718+0000","last_peered":"2024-02-10T08:47:19.411718+0000","last_clean":"2024-02-10T08:47:19.411718+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:47:19.411718+0000","last_undegraded":"2024-02-10T08:47:19.411718+0000","last_fullsized":"2024-02-10T08:47:19.411718+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":602,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411080+0000","last_change":"2024-02-10T08:46:09.490629+0000","last_active":"2024-02-10T08:47:19.411080+0000","last_peered":"2024-02-10T08:47:19.411080+0000","last_clean":"2024-02-10T08:47:19.411080+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:47:19.411080+0000","last_undegraded":"2024-02-10T08:47:19.411080+0000","last_fullsized":"2024-02-10T08:47:19.411080+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:09.490555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.490555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"676'13","reported_seq":868,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.157363+0000","last_change":"2024-02-10T08:47:08.628476+0000","last_active":"2024-02-10T08:47:20.157363+0000","last_peered":"2024-02-10T08:47:20.157363+0000","last_clean":"2024-02-10T08:47:20.157363+0000","last_became_active":"2024-02-10T08:47:08.174824+0000","last_became_peered":"2024-02-10T08:47:08.174824+0000","last_unstale":"2024-02-10T08:47:20.157363+0000","last_undegraded":"2024-02-10T08:47:20.157363+0000","last_fullsized":"2024-02-10T08:47:20.157363+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:39.499038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.499038+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":555,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.146892+0000","last_change":"2024-02-10T08:46:30.199113+0000","last_active":"2024-02-10T08:47:20.146892+0000","last_peered":"2024-02-10T08:47:20.146892+0000","last_clean":"2024-02-10T08:47:20.146892+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:47:20.146892+0000","last_undegraded":"2024-02-10T08:47:20.146892+0000","last_fullsized":"2024-02-10T08:47:20.146892+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:43:34.883363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:43:34.883363+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":693,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:21.618028+0000","last_change":"2024-02-10T08:47:21.618028+0000","last_active":"2024-02-10T08:47:21.618028+0000","last_peered":"2024-02-10T08:47:21.618028+0000","last_clean":"2024-02-10T08:47:21.618028+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:47:21.618028+0000","last_undegraded":"2024-02-10T08:47:21.618028+0000","last_fullsized":"2024-02-10T08:47:21.618028+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.617907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":237,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.417179+0000","last_change":"2024-02-10T08:46:01.404326+0000","last_active":"2024-02-10T08:47:19.417179+0000","last_peered":"2024-02-10T08:47:19.417179+0000","last_clean":"2024-02-10T08:47:19.417179+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:47:19.417179+0000","last_undegraded":"2024-02-10T08:47:19.417179+0000","last_fullsized":"2024-02-10T08:47:19.417179+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:01.404256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:01.404256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":83,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149224+0000","last_change":"2024-02-10T08:47:08.183556+0000","last_active":"2024-02-10T08:47:19.149224+0000","last_peered":"2024-02-10T08:47:19.149224+0000","last_clean":"2024-02-10T08:47:19.149224+0000","last_became_active":"2024-02-10T08:47:08.183262+0000","last_became_peered":"2024-02-10T08:47:08.183262+0000","last_unstale":"2024-02-10T08:47:19.149224+0000","last_undegraded":"2024-02-10T08:47:19.149224+0000","last_fullsized":"2024-02-10T08:47:19.149224+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:29.923093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:29.923093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":520,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.146752+0000","last_change":"2024-02-10T08:46:49.870373+0000","last_active":"2024-02-10T08:47:20.146752+0000","last_peered":"2024-02-10T08:47:20.146752+0000","last_clean":"2024-02-10T08:47:20.146752+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:47:20.146752+0000","last_undegraded":"2024-02-10T08:47:20.146752+0000","last_fullsized":"2024-02-10T08:47:20.146752+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:49.870278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.870278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":844,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410437+0000","last_change":"2024-02-10T08:46:15.415194+0000","last_active":"2024-02-10T08:47:19.410437+0000","last_peered":"2024-02-10T08:47:19.410437+0000","last_clean":"2024-02-10T08:47:19.410437+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:47:19.410437+0000","last_undegraded":"2024-02-10T08:47:19.410437+0000","last_fullsized":"2024-02-10T08:47:19.410437+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:44:37.388018+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:44:37.388018+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":36,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.415317+0000","last_change":"2024-02-10T08:47:08.347471+0000","last_active":"2024-02-10T08:47:19.415317+0000","last_peered":"2024-02-10T08:47:19.415317+0000","last_clean":"2024-02-10T08:47:19.415317+0000","last_became_active":"2024-02-10T08:47:08.172036+0000","last_became_peered":"2024-02-10T08:47:08.172036+0000","last_unstale":"2024-02-10T08:47:19.415317+0000","last_undegraded":"2024-02-10T08:47:19.415317+0000","last_fullsized":"2024-02-10T08:47:19.415317+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:46:33.634650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:33.634650+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":488,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.146801+0000","last_change":"2024-02-10T08:46:29.907845+0000","last_active":"2024-02-10T08:47:20.146801+0000","last_peered":"2024-02-10T08:47:20.146801+0000","last_clean":"2024-02-10T08:47:20.146801+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:47:20.146801+0000","last_undegraded":"2024-02-10T08:47:20.146801+0000","last_fullsized":"2024-02-10T08:47:20.146801+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:29.907739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:29.907739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":83,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149234+0000","last_change":"2024-02-10T08:47:08.173892+0000","last_active":"2024-02-10T08:47:19.149234+0000","last_peered":"2024-02-10T08:47:19.149234+0000","last_clean":"2024-02-10T08:47:19.149234+0000","last_became_active":"2024-02-10T08:47:08.173506+0000","last_became_peered":"2024-02-10T08:47:08.173506+0000","last_unstale":"2024-02-10T08:47:19.149234+0000","last_undegraded":"2024-02-10T08:47:19.149234+0000","last_fullsized":"2024-02-10T08:47:19.149234+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:45.003808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.003808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":680,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410719+0000","last_change":"2024-02-10T08:47:17.571180+0000","last_active":"2024-02-10T08:47:19.410719+0000","last_peered":"2024-02-10T08:47:19.410719+0000","last_clean":"2024-02-10T08:47:19.410719+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:47:19.410719+0000","last_undegraded":"2024-02-10T08:47:19.410719+0000","last_fullsized":"2024-02-10T08:47:19.410719+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.571093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.571093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":87,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148916+0000","last_change":"2024-02-10T08:46:15.400357+0000","last_active":"2024-02-10T08:47:19.148916+0000","last_peered":"2024-02-10T08:47:19.148916+0000","last_clean":"2024-02-10T08:47:19.148916+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:47:19.148916+0000","last_undegraded":"2024-02-10T08:47:19.148916+0000","last_fullsized":"2024-02-10T08:47:19.148916+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:46:06.677291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:06.677291+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":262,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413040+0000","last_change":"2024-02-10T08:46:41.550928+0000","last_active":"2024-02-10T08:47:19.413040+0000","last_peered":"2024-02-10T08:47:19.413040+0000","last_clean":"2024-02-10T08:47:19.413040+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:47:19.413040+0000","last_undegraded":"2024-02-10T08:47:19.413040+0000","last_fullsized":"2024-02-10T08:47:19.413040+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.550852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.550852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":22,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411338+0000","last_change":"2024-02-10T08:47:08.186121+0000","last_active":"2024-02-10T08:47:19.411338+0000","last_peered":"2024-02-10T08:47:19.411338+0000","last_clean":"2024-02-10T08:47:19.411338+0000","last_became_active":"2024-02-10T08:47:08.185649+0000","last_became_peered":"2024-02-10T08:47:08.185649+0000","last_unstale":"2024-02-10T08:47:19.411338+0000","last_undegraded":"2024-02-10T08:47:19.411338+0000","last_fullsized":"2024-02-10T08:47:19.411338+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.455169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.455169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":636,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145932+0000","last_change":"2024-02-10T08:47:08.170876+0000","last_active":"2024-02-10T08:47:19.145932+0000","last_peered":"2024-02-10T08:47:19.145932+0000","last_clean":"2024-02-10T08:47:19.145932+0000","last_became_active":"2024-02-10T08:47:08.170537+0000","last_became_peered":"2024-02-10T08:47:08.170537+0000","last_unstale":"2024-02-10T08:47:19.145932+0000","last_undegraded":"2024-02-10T08:47:19.145932+0000","last_fullsized":"2024-02-10T08:47:19.145932+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:05.371003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:05.371003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":707,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414561+0000","last_change":"2024-02-10T08:47:13.826052+0000","last_active":"2024-02-10T08:47:19.414561+0000","last_peered":"2024-02-10T08:47:19.414561+0000","last_clean":"2024-02-10T08:47:19.414561+0000","last_became_active":"2024-02-10T08:47:08.183407+0000","last_became_peered":"2024-02-10T08:47:08.183407+0000","last_unstale":"2024-02-10T08:47:19.414561+0000","last_undegraded":"2024-02-10T08:47:19.414561+0000","last_fullsized":"2024-02-10T08:47:19.414561+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:13.825974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.825974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":106,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149432+0000","last_change":"2024-02-10T08:47:08.361367+0000","last_active":"2024-02-10T08:47:19.149432+0000","last_peered":"2024-02-10T08:47:19.149432+0000","last_clean":"2024-02-10T08:47:19.149432+0000","last_became_active":"2024-02-10T08:47:08.182324+0000","last_became_peered":"2024-02-10T08:47:08.182324+0000","last_unstale":"2024-02-10T08:47:19.149432+0000","last_undegraded":"2024-02-10T08:47:19.149432+0000","last_fullsized":"2024-02-10T08:47:19.149432+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:46:08.794758+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:46:08.794758+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.794758+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":260,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:25.601751+0000","last_change":"2024-02-10T08:47:25.601751+0000","last_active":"2024-02-10T08:47:25.601751+0000","last_peered":"2024-02-10T08:47:25.601751+0000","last_clean":"2024-02-10T08:47:25.601751+0000","last_became_active":"2024-02-10T08:47:08.181656+0000","last_became_peered":"2024-02-10T08:47:08.181656+0000","last_unstale":"2024-02-10T08:47:25.601751+0000","last_undegraded":"2024-02-10T08:47:25.601751+0000","last_fullsized":"2024-02-10T08:47:25.601751+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.601673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.601673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":85,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.147944+0000","last_change":"2024-02-10T08:46:38.039133+0000","last_active":"2024-02-10T08:47:19.147944+0000","last_peered":"2024-02-10T08:47:19.147944+0000","last_clean":"2024-02-10T08:47:19.147944+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:47:19.147944+0000","last_undegraded":"2024-02-10T08:47:19.147944+0000","last_fullsized":"2024-02-10T08:47:19.147944+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:38.039064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.039064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":37,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414388+0000","last_change":"2024-02-10T08:47:09.124842+0000","last_active":"2024-02-10T08:47:19.414388+0000","last_peered":"2024-02-10T08:47:19.414388+0000","last_clean":"2024-02-10T08:47:19.414388+0000","last_became_active":"2024-02-10T08:47:08.181566+0000","last_became_peered":"2024-02-10T08:47:08.181566+0000","last_unstale":"2024-02-10T08:47:19.414388+0000","last_undegraded":"2024-02-10T08:47:19.414388+0000","last_fullsized":"2024-02-10T08:47:19.414388+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:45:57.867134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:57.867134+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":622,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410829+0000","last_change":"2024-02-10T08:46:15.352833+0000","last_active":"2024-02-10T08:47:19.410829+0000","last_peered":"2024-02-10T08:47:19.410829+0000","last_clean":"2024-02-10T08:47:19.410829+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:47:19.410829+0000","last_undegraded":"2024-02-10T08:47:19.410829+0000","last_fullsized":"2024-02-10T08:47:19.410829+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:56.515037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:56.515037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":244,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:21.545206+0000","last_change":"2024-02-10T08:47:21.545206+0000","last_active":"2024-02-10T08:47:21.545206+0000","last_peered":"2024-02-10T08:47:21.545206+0000","last_clean":"2024-02-10T08:47:21.545206+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:47:21.545206+0000","last_undegraded":"2024-02-10T08:47:21.545206+0000","last_fullsized":"2024-02-10T08:47:21.545206+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:21.545082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.545082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":759,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414283+0000","last_change":"2024-02-10T08:46:30.592735+0000","last_active":"2024-02-10T08:47:19.414283+0000","last_peered":"2024-02-10T08:47:19.414283+0000","last_clean":"2024-02-10T08:47:19.414283+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:47:19.414283+0000","last_undegraded":"2024-02-10T08:47:19.414283+0000","last_fullsized":"2024-02-10T08:47:19.414283+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:46:30.592668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.592668+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":248,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411921+0000","last_change":"2024-02-10T08:46:50.496168+0000","last_active":"2024-02-10T08:47:19.411921+0000","last_peered":"2024-02-10T08:47:19.411921+0000","last_clean":"2024-02-10T08:47:19.411921+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:47:19.411921+0000","last_undegraded":"2024-02-10T08:47:19.411921+0000","last_fullsized":"2024-02-10T08:47:19.411921+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:50.496092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:46:50.496092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":678,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413686+0000","last_change":"2024-02-10T08:47:16.775257+0000","last_active":"2024-02-10T08:47:19.413686+0000","last_peered":"2024-02-10T08:47:19.413686+0000","last_clean":"2024-02-10T08:47:19.413686+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:47:19.413686+0000","last_undegraded":"2024-02-10T08:47:19.413686+0000","last_fullsized":"2024-02-10T08:47:19.413686+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.775179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.775179+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":658,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411925+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:47:19.411925+0000","last_peered":"2024-02-10T08:47:19.411925+0000","last_clean":"2024-02-10T08:47:19.411925+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:47:19.411925+0000","last_undegraded":"2024-02-10T08:47:19.411925+0000","last_fullsized":"2024-02-10T08:47:19.411925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":664,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414650+0000","last_change":"2024-02-10T08:46:34.787736+0000","last_active":"2024-02-10T08:47:19.414650+0000","last_peered":"2024-02-10T08:47:19.414650+0000","last_clean":"2024-02-10T08:47:19.414650+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:47:19.414650+0000","last_undegraded":"2024-02-10T08:47:19.414650+0000","last_fullsized":"2024-02-10T08:47:19.414650+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:34.787659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:34.787659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":627,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411701+0000","last_change":"2024-02-10T08:47:08.183194+0000","last_active":"2024-02-10T08:47:19.411701+0000","last_peered":"2024-02-10T08:47:19.411701+0000","last_clean":"2024-02-10T08:47:19.411701+0000","last_became_active":"2024-02-10T08:47:08.182884+0000","last_became_peered":"2024-02-10T08:47:08.182884+0000","last_unstale":"2024-02-10T08:47:19.411701+0000","last_undegraded":"2024-02-10T08:47:19.411701+0000","last_fullsized":"2024-02-10T08:47:19.411701+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:16.055988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:16.055988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":698,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:26.043612+0000","last_change":"2024-02-10T08:47:26.043612+0000","last_active":"2024-02-10T08:47:26.043612+0000","last_peered":"2024-02-10T08:47:26.043612+0000","last_clean":"2024-02-10T08:47:26.043612+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:47:26.043612+0000","last_undegraded":"2024-02-10T08:47:26.043612+0000","last_fullsized":"2024-02-10T08:47:26.043612+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.043548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.043548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":579,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414279+0000","last_change":"2024-02-10T08:47:08.622117+0000","last_active":"2024-02-10T08:47:19.414279+0000","last_peered":"2024-02-10T08:47:19.414279+0000","last_clean":"2024-02-10T08:47:19.414279+0000","last_became_active":"2024-02-10T08:47:08.182920+0000","last_became_peered":"2024-02-10T08:47:08.182920+0000","last_unstale":"2024-02-10T08:47:19.414279+0000","last_undegraded":"2024-02-10T08:47:19.414279+0000","last_fullsized":"2024-02-10T08:47:19.414279+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:45:07.684765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.684765+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":686,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145212+0000","last_change":"2024-02-10T08:47:18.043795+0000","last_active":"2024-02-10T08:47:19.145212+0000","last_peered":"2024-02-10T08:47:19.145212+0000","last_clean":"2024-02-10T08:47:19.145212+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:47:19.145212+0000","last_undegraded":"2024-02-10T08:47:19.145212+0000","last_fullsized":"2024-02-10T08:47:19.145212+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:18.043731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.043731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":191,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:25.205952+0000","last_change":"2024-02-10T08:47:24.043594+0000","last_active":"2024-02-10T08:47:25.205952+0000","last_peered":"2024-02-10T08:47:25.205952+0000","last_clean":"2024-02-10T08:47:25.205952+0000","last_became_active":"2024-02-10T08:47:08.176500+0000","last_became_peered":"2024-02-10T08:47:08.176500+0000","last_unstale":"2024-02-10T08:47:25.205952+0000","last_undegraded":"2024-02-10T08:47:25.205952+0000","last_fullsized":"2024-02-10T08:47:25.205952+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.043513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":236,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412963+0000","last_change":"2024-02-10T08:47:12.548298+0000","last_active":"2024-02-10T08:47:19.412963+0000","last_peered":"2024-02-10T08:47:19.412963+0000","last_clean":"2024-02-10T08:47:19.412963+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:47:19.412963+0000","last_undegraded":"2024-02-10T08:47:19.412963+0000","last_fullsized":"2024-02-10T08:47:19.412963+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.548156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.548156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":600,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416795+0000","last_change":"2024-02-10T08:47:08.181326+0000","last_active":"2024-02-10T08:47:19.416795+0000","last_peered":"2024-02-10T08:47:19.416795+0000","last_clean":"2024-02-10T08:47:19.416795+0000","last_became_active":"2024-02-10T08:47:08.180937+0000","last_became_peered":"2024-02-10T08:47:08.180937+0000","last_unstale":"2024-02-10T08:47:19.416795+0000","last_undegraded":"2024-02-10T08:47:19.416795+0000","last_fullsized":"2024-02-10T08:47:19.416795+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:03.829736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:36.777533+0000","last_clean_scrub_stamp":"2024-02-10T08:46:03.829736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149321+0000","last_change":"2024-02-10T08:47:08.167819+0000","last_active":"2024-02-10T08:47:19.149321+0000","last_peered":"2024-02-10T08:47:19.149321+0000","last_clean":"2024-02-10T08:47:19.149321+0000","last_became_active":"2024-02-10T08:47:08.167013+0000","last_became_peered":"2024-02-10T08:47:08.167013+0000","last_unstale":"2024-02-10T08:47:19.149321+0000","last_undegraded":"2024-02-10T08:47:19.149321+0000","last_fullsized":"2024-02-10T08:47:19.149321+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.054857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"676'42","reported_seq":618,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147606+0000","last_change":"2024-02-10T08:47:09.212149+0000","last_active":"2024-02-10T08:47:20.147606+0000","last_peered":"2024-02-10T08:47:20.147606+0000","last_clean":"2024-02-10T08:47:20.147606+0000","last_became_active":"2024-02-10T08:47:08.176582+0000","last_became_peered":"2024-02-10T08:47:08.176582+0000","last_unstale":"2024-02-10T08:47:20.147606+0000","last_undegraded":"2024-02-10T08:47:20.147606+0000","last_fullsized":"2024-02-10T08:47:20.147606+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'41","last_scrub_stamp":"2024-02-10T08:46:03.679201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:03.679201+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":669,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416423+0000","last_change":"2024-02-10T08:47:08.184187+0000","last_active":"2024-02-10T08:47:19.416423+0000","last_peered":"2024-02-10T08:47:19.416423+0000","last_clean":"2024-02-10T08:47:19.416423+0000","last_became_active":"2024-02-10T08:47:08.183578+0000","last_became_peered":"2024-02-10T08:47:08.183578+0000","last_unstale":"2024-02-10T08:47:19.416423+0000","last_undegraded":"2024-02-10T08:47:19.416423+0000","last_fullsized":"2024-02-10T08:47:19.416423+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:54.693153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:45:54.693153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":680,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412195+0000","last_change":"2024-02-10T08:46:33.460971+0000","last_active":"2024-02-10T08:47:19.412195+0000","last_peered":"2024-02-10T08:47:19.412195+0000","last_clean":"2024-02-10T08:47:19.412195+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:47:19.412195+0000","last_undegraded":"2024-02-10T08:47:19.412195+0000","last_fullsized":"2024-02-10T08:47:19.412195+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:33.460870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:33.460870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":645,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.144910+0000","last_change":"2024-02-10T08:46:43.468542+0000","last_active":"2024-02-10T08:47:19.144910+0000","last_peered":"2024-02-10T08:47:19.144910+0000","last_clean":"2024-02-10T08:47:19.144910+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:47:19.144910+0000","last_undegraded":"2024-02-10T08:47:19.144910+0000","last_fullsized":"2024-02-10T08:47:19.144910+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.468478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.468478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":92,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.155617+0000","last_change":"2024-02-10T08:47:18.403493+0000","last_active":"2024-02-10T08:47:20.155617+0000","last_peered":"2024-02-10T08:47:20.155617+0000","last_clean":"2024-02-10T08:47:20.155617+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:47:20.155617+0000","last_undegraded":"2024-02-10T08:47:20.155617+0000","last_fullsized":"2024-02-10T08:47:20.155617+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:47:18.403400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.403400+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":520,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416598+0000","last_change":"2024-02-10T08:47:08.646641+0000","last_active":"2024-02-10T08:47:19.416598+0000","last_peered":"2024-02-10T08:47:19.416598+0000","last_clean":"2024-02-10T08:47:19.416598+0000","last_became_active":"2024-02-10T08:47:08.187933+0000","last_became_peered":"2024-02-10T08:47:08.187933+0000","last_unstale":"2024-02-10T08:47:19.416598+0000","last_undegraded":"2024-02-10T08:47:19.416598+0000","last_fullsized":"2024-02-10T08:47:19.416598+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:46:45.596434+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.596434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":656,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149134+0000","last_change":"2024-02-10T08:47:08.176774+0000","last_active":"2024-02-10T08:47:19.149134+0000","last_peered":"2024-02-10T08:47:19.149134+0000","last_clean":"2024-02-10T08:47:19.149134+0000","last_became_active":"2024-02-10T08:47:08.176446+0000","last_became_peered":"2024-02-10T08:47:08.176446+0000","last_unstale":"2024-02-10T08:47:19.149134+0000","last_undegraded":"2024-02-10T08:47:19.149134+0000","last_fullsized":"2024-02-10T08:47:19.149134+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.601758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.601758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148052+0000","last_change":"2024-02-10T08:46:46.046842+0000","last_active":"2024-02-10T08:47:19.148052+0000","last_peered":"2024-02-10T08:47:19.148052+0000","last_clean":"2024-02-10T08:47:19.148052+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:47:19.148052+0000","last_undegraded":"2024-02-10T08:47:19.148052+0000","last_fullsized":"2024-02-10T08:47:19.148052+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.046777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.046777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":611,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411710+0000","last_change":"2024-02-10T08:47:08.172997+0000","last_active":"2024-02-10T08:47:19.411710+0000","last_peered":"2024-02-10T08:47:19.411710+0000","last_clean":"2024-02-10T08:47:19.411710+0000","last_became_active":"2024-02-10T08:47:08.172648+0000","last_became_peered":"2024-02-10T08:47:08.172648+0000","last_unstale":"2024-02-10T08:47:19.411710+0000","last_undegraded":"2024-02-10T08:47:19.411710+0000","last_fullsized":"2024-02-10T08:47:19.411710+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:57.464141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:57.464141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":707,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145221+0000","last_change":"2024-02-10T08:47:08.179099+0000","last_active":"2024-02-10T08:47:19.145221+0000","last_peered":"2024-02-10T08:47:19.145221+0000","last_clean":"2024-02-10T08:47:19.145221+0000","last_became_active":"2024-02-10T08:47:08.178749+0000","last_became_peered":"2024-02-10T08:47:08.178749+0000","last_unstale":"2024-02-10T08:47:19.145221+0000","last_undegraded":"2024-02-10T08:47:19.145221+0000","last_fullsized":"2024-02-10T08:47:19.145221+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:55.358058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:55.358058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":569,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147021+0000","last_change":"2024-02-10T08:47:09.554952+0000","last_active":"2024-02-10T08:47:20.147021+0000","last_peered":"2024-02-10T08:47:20.147021+0000","last_clean":"2024-02-10T08:47:20.147021+0000","last_became_active":"2024-02-10T08:47:08.176575+0000","last_became_peered":"2024-02-10T08:47:08.176575+0000","last_unstale":"2024-02-10T08:47:20.147021+0000","last_undegraded":"2024-02-10T08:47:20.147021+0000","last_fullsized":"2024-02-10T08:47:20.147021+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:46:40.895770+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.895770+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":560,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.148067+0000","last_change":"2024-02-10T08:47:17.985317+0000","last_active":"2024-02-10T08:47:20.148067+0000","last_peered":"2024-02-10T08:47:20.148067+0000","last_clean":"2024-02-10T08:47:20.148067+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:47:20.148067+0000","last_undegraded":"2024-02-10T08:47:20.148067+0000","last_fullsized":"2024-02-10T08:47:20.148067+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:47:17.985235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.985235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":679,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411384+0000","last_change":"2024-02-10T08:47:08.172656+0000","last_active":"2024-02-10T08:47:19.411384+0000","last_peered":"2024-02-10T08:47:19.411384+0000","last_clean":"2024-02-10T08:47:19.411384+0000","last_became_active":"2024-02-10T08:47:08.172143+0000","last_became_peered":"2024-02-10T08:47:08.172143+0000","last_unstale":"2024-02-10T08:47:19.411384+0000","last_undegraded":"2024-02-10T08:47:19.411384+0000","last_fullsized":"2024-02-10T08:47:19.411384+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:19.465638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:19.465638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":636,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146253+0000","last_change":"2024-02-10T08:47:08.168386+0000","last_active":"2024-02-10T08:47:19.146253+0000","last_peered":"2024-02-10T08:47:19.146253+0000","last_clean":"2024-02-10T08:47:19.146253+0000","last_became_active":"2024-02-10T08:47:08.167997+0000","last_became_peered":"2024-02-10T08:47:08.167997+0000","last_unstale":"2024-02-10T08:47:19.146253+0000","last_undegraded":"2024-02-10T08:47:19.146253+0000","last_fullsized":"2024-02-10T08:47:19.146253+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:09.433145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.433145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":83,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149542+0000","last_change":"2024-02-10T08:47:16.370246+0000","last_active":"2024-02-10T08:47:19.149542+0000","last_peered":"2024-02-10T08:47:19.149542+0000","last_clean":"2024-02-10T08:47:19.149542+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:47:19.149542+0000","last_undegraded":"2024-02-10T08:47:19.149542+0000","last_fullsized":"2024-02-10T08:47:19.149542+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:16.370183+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.370183+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":694,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145106+0000","last_change":"2024-02-10T08:47:08.170466+0000","last_active":"2024-02-10T08:47:19.145106+0000","last_peered":"2024-02-10T08:47:19.145106+0000","last_clean":"2024-02-10T08:47:19.145106+0000","last_became_active":"2024-02-10T08:47:08.170142+0000","last_became_peered":"2024-02-10T08:47:08.170142+0000","last_unstale":"2024-02-10T08:47:19.145106+0000","last_undegraded":"2024-02-10T08:47:19.145106+0000","last_fullsized":"2024-02-10T08:47:19.145106+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:28.377904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:28.377904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":688,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.146718+0000","last_change":"2024-02-10T08:46:38.449729+0000","last_active":"2024-02-10T08:47:19.146718+0000","last_peered":"2024-02-10T08:47:19.146718+0000","last_clean":"2024-02-10T08:47:19.146718+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:47:19.146718+0000","last_undegraded":"2024-02-10T08:47:19.146718+0000","last_fullsized":"2024-02-10T08:47:19.146718+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:38.449611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.449611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":474,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147115+0000","last_change":"2024-02-10T08:47:08.164891+0000","last_active":"2024-02-10T08:47:20.147115+0000","last_peered":"2024-02-10T08:47:20.147115+0000","last_clean":"2024-02-10T08:47:20.147115+0000","last_became_active":"2024-02-10T08:47:08.164512+0000","last_became_peered":"2024-02-10T08:47:08.164512+0000","last_unstale":"2024-02-10T08:47:20.147115+0000","last_undegraded":"2024-02-10T08:47:20.147115+0000","last_fullsized":"2024-02-10T08:47:20.147115+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:47.845659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:47.845659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":725,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412398+0000","last_change":"2024-02-10T08:47:08.348917+0000","last_active":"2024-02-10T08:47:19.412398+0000","last_peered":"2024-02-10T08:47:19.412398+0000","last_clean":"2024-02-10T08:47:19.412398+0000","last_became_active":"2024-02-10T08:47:08.174762+0000","last_became_peered":"2024-02-10T08:47:08.174762+0000","last_unstale":"2024-02-10T08:47:19.412398+0000","last_undegraded":"2024-02-10T08:47:19.412398+0000","last_fullsized":"2024-02-10T08:47:19.412398+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:46:49.515525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.515525+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":680,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411592+0000","last_change":"2024-02-10T08:47:08.185748+0000","last_active":"2024-02-10T08:47:19.411592+0000","last_peered":"2024-02-10T08:47:19.411592+0000","last_clean":"2024-02-10T08:47:19.411592+0000","last_became_active":"2024-02-10T08:47:08.185363+0000","last_became_peered":"2024-02-10T08:47:08.185363+0000","last_unstale":"2024-02-10T08:47:19.411592+0000","last_undegraded":"2024-02-10T08:47:19.411592+0000","last_fullsized":"2024-02-10T08:47:19.411592+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.577220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.577220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412812+0000","last_change":"2024-02-10T08:46:36.562444+0000","last_active":"2024-02-10T08:47:19.412812+0000","last_peered":"2024-02-10T08:47:19.412812+0000","last_clean":"2024-02-10T08:47:19.412812+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:47:19.412812+0000","last_undegraded":"2024-02-10T08:47:19.412812+0000","last_fullsized":"2024-02-10T08:47:19.412812+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:36.562328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.562328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":613,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413598+0000","last_change":"2024-02-10T08:46:31.823967+0000","last_active":"2024-02-10T08:47:19.413598+0000","last_peered":"2024-02-10T08:47:19.413598+0000","last_clean":"2024-02-10T08:47:19.413598+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:47:19.413598+0000","last_undegraded":"2024-02-10T08:47:19.413598+0000","last_fullsized":"2024-02-10T08:47:19.413598+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.823902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.823902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":749,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411797+0000","last_change":"2024-02-10T08:47:08.365918+0000","last_active":"2024-02-10T08:47:19.411797+0000","last_peered":"2024-02-10T08:47:19.411797+0000","last_clean":"2024-02-10T08:47:19.411797+0000","last_became_active":"2024-02-10T08:47:08.176028+0000","last_became_peered":"2024-02-10T08:47:08.176028+0000","last_unstale":"2024-02-10T08:47:19.411797+0000","last_undegraded":"2024-02-10T08:47:19.411797+0000","last_fullsized":"2024-02-10T08:47:19.411797+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:45:58.480952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:58.480952+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416267+0000","last_change":"2024-02-10T08:47:08.183007+0000","last_active":"2024-02-10T08:47:19.416267+0000","last_peered":"2024-02-10T08:47:19.416267+0000","last_clean":"2024-02-10T08:47:19.416267+0000","last_became_active":"2024-02-10T08:47:08.182174+0000","last_became_peered":"2024-02-10T08:47:08.182174+0000","last_unstale":"2024-02-10T08:47:19.416267+0000","last_undegraded":"2024-02-10T08:47:19.416267+0000","last_fullsized":"2024-02-10T08:47:19.416267+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:04.536314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:04.536314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"676'11","reported_seq":867,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.161008+0000","last_change":"2024-02-10T08:47:09.211327+0000","last_active":"2024-02-10T08:47:20.161008+0000","last_peered":"2024-02-10T08:47:20.161008+0000","last_clean":"2024-02-10T08:47:20.161008+0000","last_became_active":"2024-02-10T08:47:08.188638+0000","last_became_peered":"2024-02-10T08:47:08.188638+0000","last_unstale":"2024-02-10T08:47:20.161008+0000","last_undegraded":"2024-02-10T08:47:20.161008+0000","last_fullsized":"2024-02-10T08:47:20.161008+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'9","last_scrub_stamp":"2024-02-10T08:45:21.040990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:21.040990+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149551+0000","last_change":"2024-02-10T08:47:10.260148+0000","last_active":"2024-02-10T08:47:19.149551+0000","last_peered":"2024-02-10T08:47:19.149551+0000","last_clean":"2024-02-10T08:47:19.149551+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:47:19.149551+0000","last_undegraded":"2024-02-10T08:47:19.149551+0000","last_fullsized":"2024-02-10T08:47:19.149551+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.260061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.260061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148250+0000","last_change":"2024-02-10T08:46:48.087294+0000","last_active":"2024-02-10T08:47:19.148250+0000","last_peered":"2024-02-10T08:47:19.148250+0000","last_clean":"2024-02-10T08:47:19.148250+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:47:19.148250+0000","last_undegraded":"2024-02-10T08:47:19.148250+0000","last_fullsized":"2024-02-10T08:47:19.148250+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:48.087228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.087228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":721,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:24.808523+0000","last_change":"2024-02-10T08:47:24.808523+0000","last_active":"2024-02-10T08:47:24.808523+0000","last_peered":"2024-02-10T08:47:24.808523+0000","last_clean":"2024-02-10T08:47:24.808523+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:47:24.808523+0000","last_undegraded":"2024-02-10T08:47:24.808523+0000","last_fullsized":"2024-02-10T08:47:24.808523+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:24.808424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.808424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":428,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147474+0000","last_change":"2024-02-10T08:46:30.196752+0000","last_active":"2024-02-10T08:47:20.147474+0000","last_peered":"2024-02-10T08:47:20.147474+0000","last_clean":"2024-02-10T08:47:20.147474+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:47:20.147474+0000","last_undegraded":"2024-02-10T08:47:20.147474+0000","last_fullsized":"2024-02-10T08:47:20.147474+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.449994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.449994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":858,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.160008+0000","last_change":"2024-02-10T08:46:08.457332+0000","last_active":"2024-02-10T08:47:20.160008+0000","last_peered":"2024-02-10T08:47:20.160008+0000","last_clean":"2024-02-10T08:47:20.160008+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:47:20.160008+0000","last_undegraded":"2024-02-10T08:47:20.160008+0000","last_fullsized":"2024-02-10T08:47:20.160008+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:46:08.457270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.457270+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":495,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:23.919598+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:47:23.919598+0000","last_peered":"2024-02-10T08:47:23.919598+0000","last_clean":"2024-02-10T08:47:23.919598+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:47:23.919598+0000","last_undegraded":"2024-02-10T08:47:23.919598+0000","last_fullsized":"2024-02-10T08:47:23.919598+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":635,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145000+0000","last_change":"2024-02-10T08:47:19.054002+0000","last_active":"2024-02-10T08:47:19.145000+0000","last_peered":"2024-02-10T08:47:19.145000+0000","last_clean":"2024-02-10T08:47:19.145000+0000","last_became_active":"2024-02-10T08:47:08.180586+0000","last_became_peered":"2024-02-10T08:47:08.180586+0000","last_unstale":"2024-02-10T08:47:19.145000+0000","last_undegraded":"2024-02-10T08:47:19.145000+0000","last_fullsized":"2024-02-10T08:47:19.145000+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:47:19.053931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.053931+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":512,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:23.671637+0000","last_change":"2024-02-10T08:47:23.671637+0000","last_active":"2024-02-10T08:47:23.671637+0000","last_peered":"2024-02-10T08:47:23.671637+0000","last_clean":"2024-02-10T08:47:23.671637+0000","last_became_active":"2024-02-10T08:47:08.173228+0000","last_became_peered":"2024-02-10T08:47:08.173228+0000","last_unstale":"2024-02-10T08:47:23.671637+0000","last_undegraded":"2024-02-10T08:47:23.671637+0000","last_fullsized":"2024-02-10T08:47:23.671637+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.671541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":666,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414920+0000","last_change":"2024-02-10T08:47:12.565082+0000","last_active":"2024-02-10T08:47:19.414920+0000","last_peered":"2024-02-10T08:47:19.414920+0000","last_clean":"2024-02-10T08:47:19.414920+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:47:19.414920+0000","last_undegraded":"2024-02-10T08:47:19.414920+0000","last_fullsized":"2024-02-10T08:47:19.414920+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.565015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.565015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":681,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411342+0000","last_change":"2024-02-10T08:47:08.180615+0000","last_active":"2024-02-10T08:47:19.411342+0000","last_peered":"2024-02-10T08:47:19.411342+0000","last_clean":"2024-02-10T08:47:19.411342+0000","last_became_active":"2024-02-10T08:47:08.180296+0000","last_became_peered":"2024-02-10T08:47:08.180296+0000","last_unstale":"2024-02-10T08:47:19.411342+0000","last_undegraded":"2024-02-10T08:47:19.411342+0000","last_fullsized":"2024-02-10T08:47:19.411342+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:07.459661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:07.459661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148103+0000","last_change":"2024-02-10T08:47:12.257623+0000","last_active":"2024-02-10T08:47:19.148103+0000","last_peered":"2024-02-10T08:47:19.148103+0000","last_clean":"2024-02-10T08:47:19.148103+0000","last_became_active":"2024-02-10T08:47:08.183003+0000","last_became_peered":"2024-02-10T08:47:08.183003+0000","last_unstale":"2024-02-10T08:47:19.148103+0000","last_undegraded":"2024-02-10T08:47:19.148103+0000","last_fullsized":"2024-02-10T08:47:19.148103+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.257566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.257566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":629,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:25.754309+0000","last_change":"2024-02-10T08:47:25.754309+0000","last_active":"2024-02-10T08:47:25.754309+0000","last_peered":"2024-02-10T08:47:25.754309+0000","last_clean":"2024-02-10T08:47:25.754309+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:47:25.754309+0000","last_undegraded":"2024-02-10T08:47:25.754309+0000","last_fullsized":"2024-02-10T08:47:25.754309+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.754229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.754229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":565,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147674+0000","last_change":"2024-02-10T08:47:19.946477+0000","last_active":"2024-02-10T08:47:20.147674+0000","last_peered":"2024-02-10T08:47:20.147674+0000","last_clean":"2024-02-10T08:47:20.147674+0000","last_became_active":"2024-02-10T08:47:08.179021+0000","last_became_peered":"2024-02-10T08:47:08.179021+0000","last_unstale":"2024-02-10T08:47:20.147674+0000","last_undegraded":"2024-02-10T08:47:20.147674+0000","last_fullsized":"2024-02-10T08:47:20.147674+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:19.946402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.946402+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":641,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413211+0000","last_change":"2024-02-10T08:47:08.188308+0000","last_active":"2024-02-10T08:47:19.413211+0000","last_peered":"2024-02-10T08:47:19.413211+0000","last_clean":"2024-02-10T08:47:19.413211+0000","last_became_active":"2024-02-10T08:47:08.187929+0000","last_became_peered":"2024-02-10T08:47:08.187929+0000","last_unstale":"2024-02-10T08:47:19.413211+0000","last_undegraded":"2024-02-10T08:47:19.413211+0000","last_fullsized":"2024-02-10T08:47:19.413211+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:28.640899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:45:28.640899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":24,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.416528+0000","last_change":"2024-02-10T08:47:08.183939+0000","last_active":"2024-02-10T08:47:19.416528+0000","last_peered":"2024-02-10T08:47:19.416528+0000","last_clean":"2024-02-10T08:47:19.416528+0000","last_became_active":"2024-02-10T08:47:08.182587+0000","last_became_peered":"2024-02-10T08:47:08.182587+0000","last_unstale":"2024-02-10T08:47:19.416528+0000","last_undegraded":"2024-02-10T08:47:19.416528+0000","last_fullsized":"2024-02-10T08:47:19.416528+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:00.878280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:46:00.878280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":670,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410834+0000","last_change":"2024-02-10T08:46:15.350532+0000","last_active":"2024-02-10T08:47:19.410834+0000","last_peered":"2024-02-10T08:47:19.410834+0000","last_clean":"2024-02-10T08:47:19.410834+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:47:19.410834+0000","last_undegraded":"2024-02-10T08:47:19.410834+0000","last_fullsized":"2024-02-10T08:47:19.410834+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:55.477897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:55.477897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":626,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410563+0000","last_change":"2024-02-10T08:46:41.501224+0000","last_active":"2024-02-10T08:47:19.410563+0000","last_peered":"2024-02-10T08:47:19.410563+0000","last_clean":"2024-02-10T08:47:19.410563+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:47:19.410563+0000","last_undegraded":"2024-02-10T08:47:19.410563+0000","last_fullsized":"2024-02-10T08:47:19.410563+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.501157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.501157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":683,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410947+0000","last_change":"2024-02-10T08:47:08.224199+0000","last_active":"2024-02-10T08:47:19.410947+0000","last_peered":"2024-02-10T08:47:19.410947+0000","last_clean":"2024-02-10T08:47:19.410947+0000","last_became_active":"2024-02-10T08:47:08.175322+0000","last_became_peered":"2024-02-10T08:47:08.175322+0000","last_unstale":"2024-02-10T08:47:19.410947+0000","last_undegraded":"2024-02-10T08:47:19.410947+0000","last_fullsized":"2024-02-10T08:47:19.410947+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:46:48.511488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.511488+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":664,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412301+0000","last_change":"2024-02-10T08:46:08.732146+0000","last_active":"2024-02-10T08:47:19.412301+0000","last_peered":"2024-02-10T08:47:19.412301+0000","last_clean":"2024-02-10T08:47:19.412301+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:47:19.412301+0000","last_undegraded":"2024-02-10T08:47:19.412301+0000","last_fullsized":"2024-02-10T08:47:19.412301+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:08.732073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.732073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":687,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411563+0000","last_change":"2024-02-10T08:46:41.654802+0000","last_active":"2024-02-10T08:47:19.411563+0000","last_peered":"2024-02-10T08:47:19.411563+0000","last_clean":"2024-02-10T08:47:19.411563+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:47:19.411563+0000","last_undegraded":"2024-02-10T08:47:19.411563+0000","last_fullsized":"2024-02-10T08:47:19.411563+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.654732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.654732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414331+0000","last_change":"2024-02-10T08:46:32.419160+0000","last_active":"2024-02-10T08:47:19.414331+0000","last_peered":"2024-02-10T08:47:19.414331+0000","last_clean":"2024-02-10T08:47:19.414331+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:47:19.414331+0000","last_undegraded":"2024-02-10T08:47:19.414331+0000","last_fullsized":"2024-02-10T08:47:19.414331+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:32.419097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.419097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":629,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411535+0000","last_change":"2024-02-10T08:47:08.176393+0000","last_active":"2024-02-10T08:47:19.411535+0000","last_peered":"2024-02-10T08:47:19.411535+0000","last_clean":"2024-02-10T08:47:19.411535+0000","last_became_active":"2024-02-10T08:47:08.175540+0000","last_became_peered":"2024-02-10T08:47:08.175540+0000","last_unstale":"2024-02-10T08:47:19.411535+0000","last_undegraded":"2024-02-10T08:47:19.411535+0000","last_fullsized":"2024-02-10T08:47:19.411535+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.529230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.529230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":729,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411587+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:47:19.411587+0000","last_peered":"2024-02-10T08:47:19.411587+0000","last_clean":"2024-02-10T08:47:19.411587+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:47:19.411587+0000","last_undegraded":"2024-02-10T08:47:19.411587+0000","last_fullsized":"2024-02-10T08:47:19.411587+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":666,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411987+0000","last_change":"2024-02-10T08:47:10.557902+0000","last_active":"2024-02-10T08:47:19.411987+0000","last_peered":"2024-02-10T08:47:19.411987+0000","last_clean":"2024-02-10T08:47:19.411987+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:47:19.411987+0000","last_undegraded":"2024-02-10T08:47:19.411987+0000","last_fullsized":"2024-02-10T08:47:19.411987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.557838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.557838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412599+0000","last_change":"2024-02-10T08:46:39.567932+0000","last_active":"2024-02-10T08:47:19.412599+0000","last_peered":"2024-02-10T08:47:19.412599+0000","last_clean":"2024-02-10T08:47:19.412599+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:47:19.412599+0000","last_undegraded":"2024-02-10T08:47:19.412599+0000","last_fullsized":"2024-02-10T08:47:19.412599+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.567819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":670,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410503+0000","last_change":"2024-02-10T08:46:15.348134+0000","last_active":"2024-02-10T08:47:19.410503+0000","last_peered":"2024-02-10T08:47:19.410503+0000","last_clean":"2024-02-10T08:47:19.410503+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:47:19.410503+0000","last_undegraded":"2024-02-10T08:47:19.410503+0000","last_fullsized":"2024-02-10T08:47:19.410503+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:05.554604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:05.554604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":69,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149235+0000","last_change":"2024-02-10T08:46:15.361322+0000","last_active":"2024-02-10T08:47:19.149235+0000","last_peered":"2024-02-10T08:47:19.149235+0000","last_clean":"2024-02-10T08:47:19.149235+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:47:19.149235+0000","last_undegraded":"2024-02-10T08:47:19.149235+0000","last_fullsized":"2024-02-10T08:47:19.149235+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.562838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.562838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":81,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149452+0000","last_change":"2024-02-10T08:46:15.400959+0000","last_active":"2024-02-10T08:47:19.149452+0000","last_peered":"2024-02-10T08:47:19.149452+0000","last_clean":"2024-02-10T08:47:19.149452+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:47:19.149452+0000","last_undegraded":"2024-02-10T08:47:19.149452+0000","last_fullsized":"2024-02-10T08:47:19.149452+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:46:09.425164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.425164+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":702,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412706+0000","last_change":"2024-02-10T08:47:12.832224+0000","last_active":"2024-02-10T08:47:19.412706+0000","last_peered":"2024-02-10T08:47:19.412706+0000","last_clean":"2024-02-10T08:47:19.412706+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:47:19.412706+0000","last_undegraded":"2024-02-10T08:47:19.412706+0000","last_fullsized":"2024-02-10T08:47:19.412706+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.832146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.832146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":246,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:23.598866+0000","last_change":"2024-02-10T08:47:23.598866+0000","last_active":"2024-02-10T08:47:23.598866+0000","last_peered":"2024-02-10T08:47:23.598866+0000","last_clean":"2024-02-10T08:47:23.598866+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:47:23.598866+0000","last_undegraded":"2024-02-10T08:47:23.598866+0000","last_fullsized":"2024-02-10T08:47:23.598866+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:47:23.598783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.598783+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":636,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.144837+0000","last_change":"2024-02-10T08:47:15.149867+0000","last_active":"2024-02-10T08:47:19.144837+0000","last_peered":"2024-02-10T08:47:19.144837+0000","last_clean":"2024-02-10T08:47:19.144837+0000","last_became_active":"2024-02-10T08:47:08.166836+0000","last_became_peered":"2024-02-10T08:47:08.166836+0000","last_unstale":"2024-02-10T08:47:19.144837+0000","last_undegraded":"2024-02-10T08:47:19.144837+0000","last_fullsized":"2024-02-10T08:47:19.144837+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.149802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.149802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":716,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412143+0000","last_change":"2024-02-10T08:47:08.626781+0000","last_active":"2024-02-10T08:47:19.412143+0000","last_peered":"2024-02-10T08:47:19.412143+0000","last_clean":"2024-02-10T08:47:19.412143+0000","last_became_active":"2024-02-10T08:47:08.183474+0000","last_became_peered":"2024-02-10T08:47:08.183474+0000","last_unstale":"2024-02-10T08:47:19.412143+0000","last_undegraded":"2024-02-10T08:47:19.412143+0000","last_fullsized":"2024-02-10T08:47:19.412143+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:46:10.495154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:10.495154+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":676,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410718+0000","last_change":"2024-02-10T08:46:15.350704+0000","last_active":"2024-02-10T08:47:19.410718+0000","last_peered":"2024-02-10T08:47:19.410718+0000","last_clean":"2024-02-10T08:47:19.410718+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:47:19.410718+0000","last_undegraded":"2024-02-10T08:47:19.410718+0000","last_fullsized":"2024-02-10T08:47:19.410718+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:54.474804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:45:54.474804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":631,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413595+0000","last_change":"2024-02-10T08:47:08.185211+0000","last_active":"2024-02-10T08:47:19.413595+0000","last_peered":"2024-02-10T08:47:19.413595+0000","last_clean":"2024-02-10T08:47:19.413595+0000","last_became_active":"2024-02-10T08:47:08.184890+0000","last_became_peered":"2024-02-10T08:47:08.184890+0000","last_unstale":"2024-02-10T08:47:19.413595+0000","last_undegraded":"2024-02-10T08:47:19.413595+0000","last_fullsized":"2024-02-10T08:47:19.413595+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:40.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.685831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":580,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145222+0000","last_change":"2024-02-10T08:47:08.169230+0000","last_active":"2024-02-10T08:47:19.145222+0000","last_peered":"2024-02-10T08:47:19.145222+0000","last_clean":"2024-02-10T08:47:19.145222+0000","last_became_active":"2024-02-10T08:47:08.168900+0000","last_became_peered":"2024-02-10T08:47:08.168900+0000","last_unstale":"2024-02-10T08:47:19.145222+0000","last_undegraded":"2024-02-10T08:47:19.145222+0000","last_fullsized":"2024-02-10T08:47:19.145222+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:30.717295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.717295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":532,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411941+0000","last_change":"2024-02-10T08:47:08.634486+0000","last_active":"2024-02-10T08:47:19.411941+0000","last_peered":"2024-02-10T08:47:19.411941+0000","last_clean":"2024-02-10T08:47:19.411941+0000","last_became_active":"2024-02-10T08:47:08.176005+0000","last_became_peered":"2024-02-10T08:47:08.176005+0000","last_unstale":"2024-02-10T08:47:19.411941+0000","last_undegraded":"2024-02-10T08:47:19.411941+0000","last_fullsized":"2024-02-10T08:47:19.411941+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:46:36.688435+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.688435+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":689,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412466+0000","last_change":"2024-02-10T08:46:39.661813+0000","last_active":"2024-02-10T08:47:19.412466+0000","last_peered":"2024-02-10T08:47:19.412466+0000","last_clean":"2024-02-10T08:47:19.412466+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:47:19.412466+0000","last_undegraded":"2024-02-10T08:47:19.412466+0000","last_fullsized":"2024-02-10T08:47:19.412466+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.661744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.661744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":496,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147952+0000","last_change":"2024-02-10T08:46:29.220306+0000","last_active":"2024-02-10T08:47:20.147952+0000","last_peered":"2024-02-10T08:47:20.147952+0000","last_clean":"2024-02-10T08:47:20.147952+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:47:20.147952+0000","last_undegraded":"2024-02-10T08:47:20.147952+0000","last_fullsized":"2024-02-10T08:47:20.147952+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:43:21.031375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:43:21.031375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":78,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.148170+0000","last_change":"2024-02-10T08:47:08.175878+0000","last_active":"2024-02-10T08:47:19.148170+0000","last_peered":"2024-02-10T08:47:19.148170+0000","last_clean":"2024-02-10T08:47:19.148170+0000","last_became_active":"2024-02-10T08:47:08.175381+0000","last_became_peered":"2024-02-10T08:47:08.175381+0000","last_unstale":"2024-02-10T08:47:19.148170+0000","last_undegraded":"2024-02-10T08:47:19.148170+0000","last_fullsized":"2024-02-10T08:47:19.148170+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:10.422960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:10.422960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":74,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411503+0000","last_change":"2024-02-10T08:47:08.190978+0000","last_active":"2024-02-10T08:47:19.411503+0000","last_peered":"2024-02-10T08:47:19.411503+0000","last_clean":"2024-02-10T08:47:19.411503+0000","last_became_active":"2024-02-10T08:47:08.172938+0000","last_became_peered":"2024-02-10T08:47:08.172938+0000","last_unstale":"2024-02-10T08:47:19.411503+0000","last_undegraded":"2024-02-10T08:47:19.411503+0000","last_fullsized":"2024-02-10T08:47:19.411503+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:46:43.848897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.848897+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":676,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411309+0000","last_change":"2024-02-10T08:45:11.455867+0000","last_active":"2024-02-10T08:47:19.411309+0000","last_peered":"2024-02-10T08:47:19.411309+0000","last_clean":"2024-02-10T08:47:19.411309+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:47:19.411309+0000","last_undegraded":"2024-02-10T08:47:19.411309+0000","last_fullsized":"2024-02-10T08:47:19.411309+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:11.455805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:11.455805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":652,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.412406+0000","last_change":"2024-02-10T08:46:07.493242+0000","last_active":"2024-02-10T08:47:19.412406+0000","last_peered":"2024-02-10T08:47:19.412406+0000","last_clean":"2024-02-10T08:47:19.412406+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:47:19.412406+0000","last_undegraded":"2024-02-10T08:47:19.412406+0000","last_fullsized":"2024-02-10T08:47:19.412406+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:07.493170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:07.493170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":589,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411739+0000","last_change":"2024-02-10T08:47:14.568534+0000","last_active":"2024-02-10T08:47:19.411739+0000","last_peered":"2024-02-10T08:47:19.411739+0000","last_clean":"2024-02-10T08:47:19.411739+0000","last_became_active":"2024-02-10T08:47:08.186293+0000","last_became_peered":"2024-02-10T08:47:08.186293+0000","last_unstale":"2024-02-10T08:47:19.411739+0000","last_undegraded":"2024-02-10T08:47:19.411739+0000","last_fullsized":"2024-02-10T08:47:19.411739+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:14.568443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:47:14.568443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":553,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.147215+0000","last_change":"2024-02-10T08:46:30.950978+0000","last_active":"2024-02-10T08:47:20.147215+0000","last_peered":"2024-02-10T08:47:20.147215+0000","last_clean":"2024-02-10T08:47:20.147215+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:47:20.147215+0000","last_undegraded":"2024-02-10T08:47:20.147215+0000","last_fullsized":"2024-02-10T08:47:20.147215+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:46:30.950919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.950919+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":244,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.521952+0000","last_change":"2024-02-10T08:47:19.521917+0000","last_active":"2024-02-10T08:47:19.521952+0000","last_peered":"2024-02-10T08:47:19.521952+0000","last_clean":"2024-02-10T08:47:19.521952+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:47:19.521952+0000","last_undegraded":"2024-02-10T08:47:19.521952+0000","last_fullsized":"2024-02-10T08:47:19.521952+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:19.521845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.521845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:24.365404+0000","last_change":"2024-02-10T08:47:24.365404+0000","last_active":"2024-02-10T08:47:24.365404+0000","last_peered":"2024-02-10T08:47:24.365404+0000","last_clean":"2024-02-10T08:47:24.365404+0000","last_became_active":"2024-02-10T08:47:08.176148+0000","last_became_peered":"2024-02-10T08:47:08.176148+0000","last_unstale":"2024-02-10T08:47:24.365404+0000","last_undegraded":"2024-02-10T08:47:24.365404+0000","last_fullsized":"2024-02-10T08:47:24.365404+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:24.365335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.365335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":639,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.145503+0000","last_change":"2024-02-10T08:46:37.473469+0000","last_active":"2024-02-10T08:47:19.145503+0000","last_peered":"2024-02-10T08:47:19.145503+0000","last_clean":"2024-02-10T08:47:19.145503+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:47:19.145503+0000","last_undegraded":"2024-02-10T08:47:19.145503+0000","last_fullsized":"2024-02-10T08:47:19.145503+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:37.473394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:37.473394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":541,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.148092+0000","last_change":"2024-02-10T08:47:08.345164+0000","last_active":"2024-02-10T08:47:20.148092+0000","last_peered":"2024-02-10T08:47:20.148092+0000","last_clean":"2024-02-10T08:47:20.148092+0000","last_became_active":"2024-02-10T08:47:08.169446+0000","last_became_peered":"2024-02-10T08:47:08.169446+0000","last_unstale":"2024-02-10T08:47:20.148092+0000","last_undegraded":"2024-02-10T08:47:20.148092+0000","last_fullsized":"2024-02-10T08:47:20.148092+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:46:46.889769+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.889769+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":188,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411533+0000","last_change":"2024-02-10T08:46:30.198693+0000","last_active":"2024-02-10T08:47:19.411533+0000","last_peered":"2024-02-10T08:47:19.411533+0000","last_clean":"2024-02-10T08:47:19.411533+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:47:19.411533+0000","last_undegraded":"2024-02-10T08:47:19.411533+0000","last_fullsized":"2024-02-10T08:47:19.411533+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:13.431922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.431922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":658,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411786+0000","last_change":"2024-02-10T08:47:08.182544+0000","last_active":"2024-02-10T08:47:19.411786+0000","last_peered":"2024-02-10T08:47:19.411786+0000","last_clean":"2024-02-10T08:47:19.411786+0000","last_became_active":"2024-02-10T08:47:08.182199+0000","last_became_peered":"2024-02-10T08:47:08.182199+0000","last_unstale":"2024-02-10T08:47:19.411786+0000","last_undegraded":"2024-02-10T08:47:19.411786+0000","last_fullsized":"2024-02-10T08:47:19.411786+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.022136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.022136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":73,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149539+0000","last_change":"2024-02-10T08:47:15.381535+0000","last_active":"2024-02-10T08:47:19.149539+0000","last_peered":"2024-02-10T08:47:19.149539+0000","last_clean":"2024-02-10T08:47:19.149539+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:47:19.149539+0000","last_undegraded":"2024-02-10T08:47:19.149539+0000","last_fullsized":"2024-02-10T08:47:19.149539+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.381441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.381441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":574,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:21.938460+0000","last_change":"2024-02-10T08:47:21.938407+0000","last_active":"2024-02-10T08:47:21.938460+0000","last_peered":"2024-02-10T08:47:21.938460+0000","last_clean":"2024-02-10T08:47:21.938460+0000","last_became_active":"2024-02-10T08:47:08.166650+0000","last_became_peered":"2024-02-10T08:47:08.166650+0000","last_unstale":"2024-02-10T08:47:21.938460+0000","last_undegraded":"2024-02-10T08:47:21.938460+0000","last_fullsized":"2024-02-10T08:47:21.938460+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:21.938274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.938274+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":676,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410659+0000","last_change":"2024-02-10T08:46:43.502472+0000","last_active":"2024-02-10T08:47:19.410659+0000","last_peered":"2024-02-10T08:47:19.410659+0000","last_clean":"2024-02-10T08:47:19.410659+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:47:19.410659+0000","last_undegraded":"2024-02-10T08:47:19.410659+0000","last_fullsized":"2024-02-10T08:47:19.410659+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.502412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.502412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":502,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-02-10T08:47:20.146912+0000","last_change":"2024-02-10T08:47:08.179194+0000","last_active":"2024-02-10T08:47:20.146912+0000","last_peered":"2024-02-10T08:47:20.146912+0000","last_clean":"2024-02-10T08:47:20.146912+0000","last_became_active":"2024-02-10T08:47:08.178839+0000","last_became_peered":"2024-02-10T08:47:08.178839+0000","last_unstale":"2024-02-10T08:47:20.146912+0000","last_undegraded":"2024-02-10T08:47:20.146912+0000","last_fullsized":"2024-02-10T08:47:20.146912+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.805860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.805860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":633,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.413883+0000","last_change":"2024-02-10T08:46:30.825307+0000","last_active":"2024-02-10T08:47:19.413883+0000","last_peered":"2024-02-10T08:47:19.413883+0000","last_clean":"2024-02-10T08:47:19.413883+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:47:19.413883+0000","last_undegraded":"2024-02-10T08:47:19.413883+0000","last_fullsized":"2024-02-10T08:47:19.413883+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:30.825230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.825230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":749,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.410471+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:47:19.410471+0000","last_peered":"2024-02-10T08:47:19.410471+0000","last_clean":"2024-02-10T08:47:19.410471+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:47:19.410471+0000","last_undegraded":"2024-02-10T08:47:19.410471+0000","last_fullsized":"2024-02-10T08:47:19.410471+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":85,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.149548+0000","last_change":"2024-02-10T08:46:44.049731+0000","last_active":"2024-02-10T08:47:19.149548+0000","last_peered":"2024-02-10T08:47:19.149548+0000","last_clean":"2024-02-10T08:47:19.149548+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:47:19.149548+0000","last_undegraded":"2024-02-10T08:47:19.149548+0000","last_fullsized":"2024-02-10T08:47:19.149548+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.049633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":672,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.411203+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:47:19.411203+0000","last_peered":"2024-02-10T08:47:19.411203+0000","last_clean":"2024-02-10T08:47:19.411203+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:47:19.411203+0000","last_undegraded":"2024-02-10T08:47:19.411203+0000","last_fullsized":"2024-02-10T08:47:19.411203+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":632,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-02-10T08:47:19.414122+0000","last_change":"2024-02-10T08:46:42.142516+0000","last_active":"2024-02-10T08:47:19.414122+0000","last_peered":"2024-02-10T08:47:19.414122+0000","last_clean":"2024-02-10T08:47:19.414122+0000","last_became_active":"2024-02-10T08:46:42.142143+0000","last_became_peered":"2024-02-10T08:46:42.142143+0000","last_unstale":"2024-02-10T08:47:19.414122+0000","last_undegraded":"2024-02-10T08:47:19.414122+0000","last_fullsized":"2024-02-10T08:47:19.414122+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:35.562991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:35.562991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":244,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":71,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":26,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":8834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299564,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310152,"kb_used_data":4736,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435000,"statfs":{"total":95995035648,"available":95677440000,"internally_reserved":0,"allocated":4849664,"data_stored":2520431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53700000000000003,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.76500000000000001,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.2749999999999999,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.33700000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.68899999999999995,"15min":0.60899999999999999},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.76300000000000001,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.68200000000000005,"15min":0.625},"min":{"1min":0.39000000000000001,"5min":0.29799999999999999,"15min":0.19900000000000001},"max":{"1min":0.93100000000000005,"5min":10.978,"15min":10.978},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.67800000000000005,"15min":0.60399999999999998},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.68400000000000005,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.67500000000000004},"min":{"1min":0.375,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":2.5640000000000001,"5min":7.5099999999999998,"15min":19.369},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.30499999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.81000000000000005,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70199999999999996,"15min":0.69699999999999995},"min":{"1min":0.375,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":3.6339999999999999,"5min":7.8700000000000001,"15min":22.102},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":8.827},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":1.903,"15min":13.646000000000001},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.51800000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.66400000000000003,"5min":3.0630000000000002,"15min":19.326000000000001},"last":0.498},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.51500000000000001,"15min":0.498},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.90300000000000002,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.54500000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47799999999999998,"15min":0.56100000000000005},"min":{"1min":0.27500000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":22.687000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67900000000000005,"5min":5.484,"15min":5.484},"last":0.52000000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561202,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318368,"kb_used_data":4760,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426784,"statfs":{"total":95995035648,"available":95669026816,"internally_reserved":0,"allocated":4874240,"data_stored":2533428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.222,"5min":0.222,"15min":0.19800000000000001},"max":{"1min":0.68300000000000005,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.63400000000000001},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":2.649,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.33700000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.878,"15min":0.65300000000000002},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63200000000000001,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.74199999999999999,"15min":0.59499999999999997},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":2.4769999999999999,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.746,"15min":0.61299999999999999},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":2.4319999999999999,"5min":10.772,"15min":10.772},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.83599999999999997,"15min":0.64400000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":2.3570000000000002,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.79700000000000004,"15min":0.70299999999999996},"min":{"1min":0.34599999999999997,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":9.2650000000000006,"5min":9.2650000000000006,"15min":16.823},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.28699999999999998},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45300000000000001,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":3.657,"15min":3.657},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.46500000000000002,"15min":0.40500000000000003},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.24399999999999999},"max":{"1min":0.61799999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.628,"15min":0.504},"min":{"1min":0.27600000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.63300000000000001,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46500000000000002},"min":{"1min":0.311,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":3.915,"15min":3.915},"last":0.496}]},{"osd":7,"last update":"Sat Feb 10 08:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.54700000000000004,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.36899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54700000000000004,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.53300000000000003}]}]},{"osd":1,"up_from":0,"seq":55834575049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318228,"kb_used_data":4748,"kb_used_omap":0,"kb_used_meta":313472,"kb_avail":93426924,"statfs":{"total":95995035648,"available":95669170176,"internally_reserved":0,"allocated":4861952,"data_stored":2536433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":320995328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:47:14 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.46899999999999997,"15min":0.40799999999999997},"min":{"1min":0.24299999999999999,"5min":0.17799999999999999,"15min":0.156},"max":{"1min":0.371,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.70899999999999996,"15min":0.49099999999999999},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.51100000000000001,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:46:40 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.58699999999999997,"15min":0.46400000000000002},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.987,"5min":11.987,"15min":11.987},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.51700000000000002,"15min":0.48799999999999999},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.749,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.26200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.438,"15min":0.47199999999999998},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55700000000000005,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.51700000000000002,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.22600000000000001},"max":{"1min":4.0780000000000003,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:43:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":0.79900000000000004,"5min":0.89200000000000002,"15min":3.327},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.159,"5min":1.6080000000000001,"15min":8.4730000000000008},"last":0.63200000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:46:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.32400000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.82699999999999996,"5min":4.609,"15min":4.609},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":0.78500000000000003,"5min":1.7769999999999999,"15min":3.4079999999999999},"last":0.64900000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:47:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54500000000000004,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.747,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58399999999999996,"15min":0.61899999999999999},"min":{"1min":0.373,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59799999999999998,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.317,"15min":0.317},"max":{"1min":1.4370000000000001,"5min":1.849,"15min":3.363},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56499999999999995,"15min":0.65500000000000003},"min":{"1min":0.38300000000000001,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.82099999999999995,"5min":1.206,"15min":8.4299999999999997},"last":0.59899999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411525,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314212,"kb_used_data":4700,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430940,"statfs":{"total":95995035648,"available":95673282560,"internally_reserved":0,"allocated":4812800,"data_stored":2511645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:46:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.63800000000000001,"15min":0.504},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.67800000000000005,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.68600000000000005,"15min":0.56699999999999995},"min":{"1min":0.219,"5min":0.187,"15min":0.187},"max":{"1min":0.41799999999999998,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.317}]},{"osd":1,"last update":"Sat Feb 10 08:46:40 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.623,"15min":0.56100000000000005},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53200000000000003,"5min":9.6980000000000004,"15min":12.512},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.59599999999999997,"15min":0.54000000000000004},"min":{"1min":0.25600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.621,"5min":12.875,"15min":12.875},"last":0.28299999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.73499999999999999,"15min":0.54500000000000004},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47399999999999998,"5min":11.612,"15min":11.612},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.67300000000000004,"15min":0.56399999999999995},"min":{"1min":0.253,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.65600000000000003,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.35599999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.67200000000000004},"min":{"1min":0.34499999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76200000000000001,"5min":1.7370000000000001,"15min":10.920999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":1.014,"15min":12.292999999999999},"last":0.752}]},{"osd":5,"last update":"Sat Feb 10 08:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.69799999999999995,"15min":0.70699999999999996},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":10.34,"15min":10.34},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.71099999999999997,"15min":0.66800000000000004},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66800000000000004,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.747,"15min":0.71499999999999997},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69299999999999995,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.80400000000000005,"15min":0.71099999999999997},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69999999999999996,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.72799999999999998,"15min":0.71899999999999997},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":10.539,"15min":10.539},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.77500000000000002,"15min":0.71699999999999997},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0700000000000001,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.625}]}]},{"osd":3,"up_from":25,"seq":107374182593,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314288,"kb_used_data":4776,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430864,"statfs":{"total":95995035648,"available":95673204736,"internally_reserved":0,"allocated":4890624,"data_stored":2541870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.219,"5min":0.184,"15min":0.184},"max":{"1min":1.8879999999999999,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.53900000000000003,"15min":0.60099999999999998},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":6.6369999999999996,"5min":10.484999999999999,"15min":11.808},"last":0.29299999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76300000000000001,"15min":0.67700000000000005},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.5810000000000004,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":2.9950000000000001,"5min":17.343,"15min":17.343},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.78000000000000003,"15min":0.68999999999999995},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":12.664999999999999,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.45800000000000002,"15min":0.63900000000000001},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":1.206,"5min":5.0259999999999998,"15min":23.294},"last":0.38700000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54500000000000004,"15min":0.66900000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":2.589,"15min":12.371},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68700000000000006,"5min":14.821999999999999,"15min":14.821999999999999},"last":0.623}]},{"osd":5,"last update":"Sat Feb 10 08:46:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.70599999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.647,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.73299999999999998,"15min":0.70399999999999996},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5229999999999999,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.71399999999999997},"min":{"1min":0.36499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.244,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.73099999999999998,"15min":0.746},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79800000000000004,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:46:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.67400000000000004,"15min":0.70299999999999996},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.413,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57499999999999996,"15min":0.74099999999999999},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6779999999999999,"5min":2.1339999999999999,"15min":12.606999999999999},"last":0.55000000000000004}]}]},{"osd":4,"up_from":633,"seq":2718714298381,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304016,"kb_used_data":4744,"kb_used_omap":59,"kb_used_meta":299204,"kb_avail":93441136,"statfs":{"total":95995035648,"available":95683723264,"internally_reserved":0,"allocated":4857856,"data_stored":2529932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":306385515},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790135,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306056,"kb_used_data":4736,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439096,"statfs":{"total":95995035648,"available":95681634304,"internally_reserved":0,"allocated":4849664,"data_stored":2539995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:46:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.67800000000000005,"15min":1.05},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.923,"5min":8.7400000000000002,"15min":107.551},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.57999999999999996,"15min":0.97899999999999998},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.22900000000000001},"max":{"1min":2.9590000000000001,"5min":5.375,"15min":88.792000000000002},"last":0.29799999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:46:41 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.79300000000000004,"15min":1.0129999999999999},"min":{"1min":0.312,"5min":0.216,"15min":0.216},"max":{"1min":12.375999999999999,"5min":12.375999999999999,"15min":107.205},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":0.74199999999999999,"15min":0.88700000000000001},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":12.845000000000001,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:46:41 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73699999999999999,"15min":1.0309999999999999},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":5.5039999999999996,"5min":10.794,"15min":54.174999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.73399999999999999,"15min":1.0629999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.335,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:46:41 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.91200000000000003,"15min":0.85899999999999999},"min":{"1min":0.33300000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.5380000000000003,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.442},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.79100000000000004,"15min":1.2869999999999999},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":8.0120000000000005,"5min":8.7690000000000001,"15min":114.526},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":0.72399999999999998,"15min":72.216999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.57599999999999996},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":44.345999999999997},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.51000000000000001,"15min":0.71899999999999997},"min":{"1min":0.26800000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71699999999999997,"5min":10.023999999999999,"15min":72.241},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54700000000000004,"15min":0.57799999999999996},"min":{"1min":0.29399999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":10.013,"15min":22.469999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46300000000000002,"15min":0.71299999999999997},"min":{"1min":0.28399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.47599999999999998,"15min":0.745},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:47:30.191 INFO:tasks.thrashosds.thrasher:Setting 16.1d to [2, 5, 1, 3, 0, 4] 2024-02-10T08:47:30.191 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '16.1d', '2', '5', '1', '3', '0', '4'] 2024-02-10T08:47:30.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pg-upmap-items 16.1d 2 5 1 3 0 4 2024-02-10T08:47:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.046195+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.049680+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.347192+0000 osd.0 (osd.0) 131 : cluster [DBG] 16.5 deep-scrub starts 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.348709+0000 osd.0 (osd.0) 132 : cluster [DBG] 16.5 deep-scrub ok 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:29.476527+0000 mgr.y (mgr.14553) 580 : audit [DBG] from='client.54179 v1:172.21.15.107:0/2868614894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.544185+0000 mgr.y (mgr.14553) 581 : cluster [DBG] pgmap v1043: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.553484+0000 osd.7 (osd.7) 282 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.556595+0000 osd.7 (osd.7) 283 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.882653+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:29.886690+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:47:30.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:30.236242+0000 mon.a (mon.0) 3313 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:30.236301+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:30.244123+0000 mon.b (mon.1) 778 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:30.246130+0000 mon.a (mon.0) 3315 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.046195+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.049680+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.347192+0000 osd.0 (osd.0) 131 : cluster [DBG] 16.5 deep-scrub starts 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.348709+0000 osd.0 (osd.0) 132 : cluster [DBG] 16.5 deep-scrub ok 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:29.476527+0000 mgr.y (mgr.14553) 580 : audit [DBG] from='client.54179 v1:172.21.15.107:0/2868614894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.544185+0000 mgr.y (mgr.14553) 581 : cluster [DBG] pgmap v1043: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.553484+0000 osd.7 (osd.7) 282 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:47:30.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.556595+0000 osd.7 (osd.7) 283 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.882653+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:29.886690+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:30.236242+0000 mon.a (mon.0) 3313 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:30.236301+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:30.244123+0000 mon.b (mon.1) 778 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:30.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:30.246130+0000 mon.a (mon.0) 3315 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.046195+0000 osd.2 (osd.2) 263 : cluster [DBG] 5.d scrub starts 2024-02-10T08:47:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.049680+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub ok 2024-02-10T08:47:31.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.347192+0000 osd.0 (osd.0) 131 : cluster [DBG] 16.5 deep-scrub starts 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.348709+0000 osd.0 (osd.0) 132 : cluster [DBG] 16.5 deep-scrub ok 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:29.476527+0000 mgr.y (mgr.14553) 580 : audit [DBG] from='client.54179 v1:172.21.15.107:0/2868614894' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.544185+0000 mgr.y (mgr.14553) 581 : cluster [DBG] pgmap v1043: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.553484+0000 osd.7 (osd.7) 282 : cluster [DBG] 16.13 deep-scrub starts 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.556595+0000 osd.7 (osd.7) 283 : cluster [DBG] 16.13 deep-scrub ok 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.882653+0000 osd.5 (osd.5) 286 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:29.886690+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:30.236242+0000 mon.a (mon.0) 3313 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:30.236301+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:30.244123+0000 mon.b (mon.1) 778 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:31.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:30.246130+0000 mon.a (mon.0) 3315 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:30.368643+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:30.370518+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:30.666088+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:30.844052+0000 osd.5 (osd.5) 288 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:30.847587+0000 osd.5 (osd.5) 289 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.145407+0000 mon.a (mon.0) 3317 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:31.145507+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.147176+0000 mon.b (mon.1) 779 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.149836+0000 mon.a (mon.0) 3319 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.298932+0000 mon.c (mon.2) 463 : audit [INF] from='client.? v1:172.21.15.107:0/2910186396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.299227+0000 mon.a (mon.0) 3320 : audit [INF] from='client.54185 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:30.368643+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:30.370518+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:30.666088+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:30.844052+0000 osd.5 (osd.5) 288 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:30.847587+0000 osd.5 (osd.5) 289 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.145407+0000 mon.a (mon.0) 3317 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:31.145507+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.147176+0000 mon.b (mon.1) 779 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:31.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.149836+0000 mon.a (mon.0) 3319 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:31.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.298932+0000 mon.c (mon.2) 463 : audit [INF] from='client.? v1:172.21.15.107:0/2910186396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:31.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.299227+0000 mon.a (mon.0) 3320 : audit [INF] from='client.54185 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:32.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:30.368643+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 deep-scrub starts 2024-02-10T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:30.370518+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 deep-scrub ok 2024-02-10T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:30.666088+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:30.844052+0000 osd.5 (osd.5) 288 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:30.847587+0000 osd.5 (osd.5) 289 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.145407+0000 mon.a (mon.0) 3317 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:31.145507+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.147176+0000 mon.b (mon.1) 779 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.149836+0000 mon.a (mon.0) 3319 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.298932+0000 mon.c (mon.2) 463 : audit [INF] from='client.? v1:172.21.15.107:0/2910186396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.299227+0000 mon.a (mon.0) 3320 : audit [INF] from='client.54185 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:47:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:32.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:32.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 -- 172.21.15.107:0/1437229637 >> v1:172.21.15.107:6790/0 conn(0x7f794c1007f0 legacy=0x7f794c102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 -- 172.21.15.107:0/1437229637 shutdown_connections 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 -- 172.21.15.107:0/1437229637 >> 172.21.15.107:0/1437229637 conn(0x7f794c0f3640 msgr2=0x7f794c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 -- 172.21.15.107:0/1437229637 shutdown_connections 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 -- 172.21.15.107:0/1437229637 wait complete. 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 librados: starting msgr at 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 librados: starting objecter 2024-02-10T08:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.104+0000 7f7951e8e700 1 Processor -- start 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 -- start start 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 librados: setting wanted keys 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 librados: calling monclient init 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f794c1172d0 con 0x7f794c104330 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f794c1184d0 con 0x7f794c1007f0 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f794c1196d0 con 0x7f794c0fda50 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f794b7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f794c0fda50 0x7f794c0f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43276/0 (socket says 172.21.15.107:43276) 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f794b7fe700 1 -- 172.21.15.107:0/2910186396 learned_addr learned my addr 172.21.15.107:0/2910186396 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter ms_handle_connect 0x7f794c0fda50 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:32.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter ms_handle_connect 0x7f794c104330 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851222027 0 0) 0x7f794c1196d0 con 0x7f794c0fda50 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7934003630 con 0x7f794c0fda50 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82799204 0 0) 0x7f794c1172d0 con 0x7f794c104330 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f794c1196d0 con 0x7f794c104330 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter ms_handle_connect 0x7f794c1007f0 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847251570 0 0) 0x7f794c1184d0 con 0x7f794c1007f0 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f794c1172d0 con 0x7f794c1007f0 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944071629 0 0) 0x7f7934003630 con 0x7f794c0fda50 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f794c1184d0 con 0x7f794c0fda50 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191362087 0 0) 0x7f794c1196d0 con 0x7f794c104330 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7934003630 con 0x7f794c104330 2024-02-10T08:47:32.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7940001a20 con 0x7f794c0fda50 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7944001b00 con 0x7f794c104330 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2693311024 0 0) 0x7f794c1184d0 con 0x7f794c0fda50 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 >> v1:172.21.15.195:6789/0 conn(0x7f794c1007f0 legacy=0x7f794c0fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 >> v1:172.21.15.107:6789/0 conn(0x7f794c104330 legacy=0x7f794c1159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c11a8d0 con 0x7f794c0fda50 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.105+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c1174c0 con 0x7f794c0fda50 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.106+0000 7f7951e8e700 10 client.54185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.106+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c1179b0 con 0x7f794c0fda50 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.106+0000 7f7951e8e700 1 librados: init done 2024-02-10T08:47:32.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.106+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7940004c00 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.106+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7940005ae0 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7938004f20 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7940018a30 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(690..690 src has 420..690) v4 ==== 8569+0+0 (unknown 3855404305 0 0) 0x7f7940078f30 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 10 client.54185.objecter ms_dispatch 0x7f794c001550 osd_map(690..690 src has 420..690) v4 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 3 client.54185.objecter handle_osd_map got epochs [690,690] > 0 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 3 client.54185.objecter handle_osd_map decoding full epoch 690 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 20 client.54185.objecter dump_active .. 0 homeless 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.107+0000 7f7948ff9700 10 client.54185.objecter ms_handle_connect 0x7f7934063bb0 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.111+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f794004c120 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:31.297+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f79380051d0 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.154+0000 7f7948ff9700 1 -- 172.21.15.107:0/2910186396 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v692) v1 ==== 87+0+0 (unknown 1058800793 0 0) 0x7f794004a530 con 0x7f794c0fda50 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.156+0000 7f7951e8e700 10 librados: watch_flush enter 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.156+0000 7f7951e8e700 10 librados: watch_flush exit 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.156+0000 7f7951e8e700 20 client.54185.objecter shutdown clearing up homeless session... 2024-02-10T08:47:32.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.156+0000 7f7951e8e700 10 client.54185.objecter successfully canceled tick 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.156+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7934063bb0 legacy=0x7f7934066070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 >> v1:172.21.15.107:6790/0 conn(0x7f794c0fda50 legacy=0x7f794c0f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 shutdown_connections 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 >> 172.21.15.107:0/2910186396 conn(0x7f794c0f3640 msgr2=0x7f794c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 shutdown_connections 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 -- 172.21.15.107:0/2910186396 wait complete. 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:32.157+0000 7f7951e8e700 1 librados: shutdown 2024-02-10T08:47:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:47:32.799 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:32.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:32.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:32.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:31.544960+0000 mgr.y (mgr.14553) 582 : cluster [DBG] pgmap v1046: 194 pgs: 7 unknown, 187 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:31.735459+0000 mgr.y (mgr.14553) 583 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:31.834532+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.d scrub starts 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:31.837635+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.d scrub ok 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:32.147918+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:32.155908+0000 mon.a (mon.0) 3322 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:32.156025+0000 mon.a (mon.0) 3323 : audit [INF] from='client.54185 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:32.156086+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:32.156812+0000 mon.b (mon.1) 780 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:32.157128+0000 mon.a (mon.0) 3325 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:31.544960+0000 mgr.y (mgr.14553) 582 : cluster [DBG] pgmap v1046: 194 pgs: 7 unknown, 187 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:31.735459+0000 mgr.y (mgr.14553) 583 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:31.834532+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.d scrub starts 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:31.837635+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.d scrub ok 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:32.147918+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:32.155908+0000 mon.a (mon.0) 3322 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:32.156025+0000 mon.a (mon.0) 3323 : audit [INF] from='client.54185 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:32.156086+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:32.156812+0000 mon.b (mon.1) 780 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:32.157128+0000 mon.a (mon.0) 3325 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:32.931 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:31.544960+0000 mgr.y (mgr.14553) 582 : cluster [DBG] pgmap v1046: 194 pgs: 7 unknown, 187 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:31.735459+0000 mgr.y (mgr.14553) 583 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:31.834532+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.d scrub starts 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:31.837635+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.d scrub ok 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:32.147918+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:32.155908+0000 mon.a (mon.0) 3322 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:32.156025+0000 mon.a (mon.0) 3323 : audit [INF] from='client.54185 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:47:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:32.156086+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2024-02-10T08:47:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:32.156812+0000 mon.b (mon.1) 780 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:32.157128+0000 mon.a (mon.0) 3325 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]: dispatch 2024-02-10T08:47:33.768 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:33.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:33.152801+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:33.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:33.156608+0000 mon.a (mon.0) 3327 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]': finished 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:33.156681+0000 mon.a (mon.0) 3328 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:33.234901+0000 mon.b (mon.1) 781 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:33.235325+0000 mon.a (mon.0) 3329 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:33.152801+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:33.156608+0000 mon.a (mon.0) 3327 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]': finished 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:33.156681+0000 mon.a (mon.0) 3328 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:33.234901+0000 mon.b (mon.1) 781 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:33.235325+0000 mon.a (mon.0) 3329 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:33.152801+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:33.156608+0000 mon.a (mon.0) 3327 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-120", "mode": "writeback"}]': finished 2024-02-10T08:47:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:33.156681+0000 mon.a (mon.0) 3328 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2024-02-10T08:47:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:33.234901+0000 mon.b (mon.1) 781 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:33.235325+0000 mon.a (mon.0) 3329 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:34.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.423+0000 7f134e15a700 1 -- 172.21.15.107:0/4030627924 >> v1:172.21.15.107:6789/0 conn(0x7f1348104330 legacy=0x7f13481067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.423+0000 7f134e15a700 1 -- 172.21.15.107:0/4030627924 shutdown_connections 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.423+0000 7f134e15a700 1 -- 172.21.15.107:0/4030627924 >> 172.21.15.107:0/4030627924 conn(0x7f13480f3640 msgr2=0x7f13480f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.423+0000 7f134e15a700 1 -- 172.21.15.107:0/4030627924 shutdown_connections 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.423+0000 7f134e15a700 1 -- 172.21.15.107:0/4030627924 wait complete. 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 librados: starting msgr at 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 librados: starting objecter 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 Processor -- start 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 -- start start 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 librados: setting wanted keys 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 librados: calling monclient init 2024-02-10T08:47:34.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134806cbf0 con 0x7f1348104330 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134806cdc0 con 0x7f13480fda50 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f134e15a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134806cfb0 con 0x7f13481007f0 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1346ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f13481007f0 0x7f1348114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43286/0 (socket says 172.21.15.107:43286) 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1347fff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1348104330 0x7f13481184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58500/0 (socket says 172.21.15.107:58500) 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1346ffd700 1 -- 172.21.15.107:0/1248871181 learned_addr learned my addr 172.21.15.107:0/1248871181 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1344ff9700 10 client.?.objecter ms_handle_connect 0x7f13481007f0 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1344ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1344ff9700 10 client.?.objecter ms_handle_connect 0x7f1348104330 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.424+0000 7f1344ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 10 client.?.objecter ms_handle_connect 0x7f13480fda50 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180684464 0 0) 0x7f134806cbf0 con 0x7f1348104330 2024-02-10T08:47:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13340033f0 con 0x7f1348104330 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465516245 0 0) 0x7f134806cfb0 con 0x7f13481007f0 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134806cbf0 con 0x7f13481007f0 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180457659 0 0) 0x7f134806cdc0 con 0x7f13480fda50 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134806cfb0 con 0x7f13480fda50 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2342180267 0 0) 0x7f13340033f0 con 0x7f1348104330 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134806cdc0 con 0x7f1348104330 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1340002320 con 0x7f1348104330 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2871150600 0 0) 0x7f134806cbf0 con 0x7f13481007f0 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13340033f0 con 0x7f13481007f0 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1330001950 con 0x7f13481007f0 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181488741 0 0) 0x7f134806cfb0 con 0x7f13480fda50 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134806cbf0 con 0x7f13480fda50 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1338001980 con 0x7f13480fda50 2024-02-10T08:47:34.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560092258 0 0) 0x7f134806cdc0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6790/0 conn(0x7f13481007f0 legacy=0x7f1348114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.195:6789/0 conn(0x7f13480fda50 legacy=0x7f13480fc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134811cc00 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348119c30 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f134e15a700 10 client.28614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f134811a0a0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f134e15a700 1 librados: init done 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f13400024c0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.425+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1340003aa0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.426+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1328004f00 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1340001aa0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(694..694 src has 420..694) v4 ==== 8564+0+0 (unknown 2899589194 0 0) 0x7f13400792b0 con 0x7f1348104330 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 osd_map(694..694 src has 420..694) v4 2024-02-10T08:47:34.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 3 client.28614.objecter handle_osd_map got epochs [694,694] > 0 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 3 client.28614.objecter handle_osd_map decoding full epoch 694 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 20 client.28614.objecter dump_active .. 0 homeless 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v694) v1 ==== 54+0+16 (unknown 1113608185 0 2420000192) 0x7f1340079690 con 0x7f1348104330 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1334063da0 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 10 client.28614.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 1 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 1 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 0 1 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 4 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 10 client.28614.objecter _send_command 1 2024-02-10T08:47:34.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.427+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1328004950 con 0x7f1328004530 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1328004530 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1328004950 con 0x7f1328004530 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 10 client.28614.objecter _finish_command 1 = system:0 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 4 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.428+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 0 1 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 10 client.28614.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 2 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 2 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f13280051a0 osd=0 3 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 0 2 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 4 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 10 client.28614.objecter _send_command 2 2024-02-10T08:47:34.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.443+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1328004d30 con 0x7f1328004530 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1328004d30 con 0x7f1328004530 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f1344ff9700 10 client.28614.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 4 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 0 2 2024-02-10T08:47:34.446 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 10 client.28614.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 1 3 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 4 2024-02-10T08:47:34.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 10 client.28614.objecter _send_command 3 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.444+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f132800bf60 con 0x7f1328003b40 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1328003b40 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f132800bf60 con 0x7f1328003b40 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 10 client.28614.objecter _finish_command 3 = system:0 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 4 2024-02-10T08:47:34.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.445+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 1 3 2024-02-10T08:47:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 10 client.28614.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 4 2024-02-10T08:47:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 4 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f1328004270 osd=1 3 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 1 4 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 4 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 10 client.28614.objecter _send_command 4 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.460+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f132800d280 con 0x7f1328003b40 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f132800d280 con 0x7f1328003b40 2024-02-10T08:47:34.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f1344ff9700 10 client.28614.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 4 2024-02-10T08:47:34.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 1 4 2024-02-10T08:47:34.463 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 10 client.28614.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 5 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 5 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 2 5 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 4 2024-02-10T08:47:34.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 10 client.28614.objecter _send_command 5 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.461+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1328010d30 con 0x7f1328002070 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1328002070 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1328010d30 con 0x7f1328002070 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 10 client.28614.objecter _finish_command 5 = system:0 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 4 2024-02-10T08:47:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.462+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 2 5 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 10 client.28614.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 6 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 6 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f13280033a0 osd=2 3 2024-02-10T08:47:34.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 2 6 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 4 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 10 client.28614.objecter _send_command 6 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.477+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1328001630 con 0x7f1328002070 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1328001630 con 0x7f1328002070 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f1344ff9700 10 client.28614.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 4 2024-02-10T08:47:34.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 2 6 2024-02-10T08:47:34.480 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 10 client.28614.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 7 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 7 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 3 7 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 4 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 10 client.28614.objecter _send_command 7 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.478+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f132800d010 con 0x7f1328010ed0 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1328010ed0 2024-02-10T08:47:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f132800d010 con 0x7f1328010ed0 2024-02-10T08:47:34.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:34.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 10 client.28614.objecter _finish_command 7 = system:0 2024-02-10T08:47:34.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 4 2024-02-10T08:47:34.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.479+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 3 7 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 10 client.28614.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 8 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 8 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132800c8d0 osd=3 3 2024-02-10T08:47:34.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 3 8 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 4 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 10 client.28614.objecter _send_command 8 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.493+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f132800d010 con 0x7f1328010ed0 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f132800d010 con 0x7f1328010ed0 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f1344ff9700 10 client.28614.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.497 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 4 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 3 8 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 10 client.28614.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 9 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 9 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 4 9 2024-02-10T08:47:34.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 4 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 10 client.28614.objecter _send_command 9 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.495+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f132801a200 con 0x7f13280166f0 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f13280166f0 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f132801a200 con 0x7f13280166f0 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 10 client.28614.objecter _finish_command 9 = system:0 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 4 2024-02-10T08:47:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.496+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 4 9 2024-02-10T08:47:34.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 10 client.28614.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 10 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 10 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132800d010 osd=4 3 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 4 10 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 4 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 10 client.28614.objecter _send_command 10 2024-02-10T08:47:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.512+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f132801a200 con 0x7f13280166f0 2024-02-10T08:47:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f132801a200 con 0x7f13280166f0 2024-02-10T08:47:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 10 client.28614.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 4 2024-02-10T08:47:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 4 10 2024-02-10T08:47:34.517 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 10 client.28614.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 11 2024-02-10T08:47:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 11 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 5 11 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 4 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 10 client.28614.objecter _send_command 11 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f132801e5b0 con 0x7f132801a8e0 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.515+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f132801a8e0 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.516+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f132801e5b0 con 0x7f132801a8e0 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.516+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.516+0000 7f1344ff9700 10 client.28614.objecter _finish_command 11 = system:0 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.516+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 4 2024-02-10T08:47:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.516+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 5 11 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 10 client.28614.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 12 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 12 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801a200 osd=5 3 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 5 12 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 4 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 10 client.28614.objecter _send_command 12 2024-02-10T08:47:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.530+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f132801e5b0 con 0x7f132801a8e0 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.532+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f132801e5b0 con 0x7f132801a8e0 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.532+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.532+0000 7f1344ff9700 10 client.28614.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.532+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 4 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.532+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 5 12 2024-02-10T08:47:34.534 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 10 client.28614.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 13 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 13 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 6 13 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 4 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 10 client.28614.objecter _send_command 13 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f132801f800 con 0x7f13280202b0 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.533+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f13280202b0 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.534+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f132801f800 con 0x7f13280202b0 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.534+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.534+0000 7f1344ff9700 10 client.28614.objecter _finish_command 13 = system:0 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.534+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 4 2024-02-10T08:47:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.534+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 6 13 2024-02-10T08:47:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 10 client.28614.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 14 2024-02-10T08:47:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 14 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801e5b0 osd=6 3 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 6 14 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 4 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 10 client.28614.objecter _send_command 14 2024-02-10T08:47:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.548+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f132801f800 con 0x7f13280202b0 2024-02-10T08:47:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f132801f800 con 0x7f13280202b0 2024-02-10T08:47:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 10 client.28614.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 4 2024-02-10T08:47:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 6 14 2024-02-10T08:47:34.552 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 10 client.28614.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 15 2024-02-10T08:47:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 15 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 7 15 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 4 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 10 client.28614.objecter _send_command 15 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1328026f80 con 0x7f1328023470 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.550+0000 7f1344ff9700 10 client.28614.objecter ms_handle_connect 0x7f1328023470 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.551+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1328026f80 con 0x7f1328023470 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.551+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.551+0000 7f1344ff9700 10 client.28614.objecter _finish_command 15 = system:0 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.551+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 4 2024-02-10T08:47:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.551+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 7 15 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 10 client.28614.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign -1 16 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 20 client.28614.objecter _get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 15 client.28614.objecter _session_command_op_remove -1 16 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 20 client.28614.objecter get_session s=0x7f132801f800 osd=7 3 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 15 client.28614.objecter _session_command_op_assign 7 16 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 4 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 10 client.28614.objecter _send_command 16 2024-02-10T08:47:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.566+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1328026f80 con 0x7f1328023470 2024-02-10T08:47:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f1344ff9700 1 -- 172.21.15.107:0/1248871181 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1328026f80 con 0x7f1328023470 2024-02-10T08:47:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f1344ff9700 10 client.28614.objecter ms_dispatch 0x7f1348001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f1344ff9700 10 client.28614.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f1344ff9700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 4 2024-02-10T08:47:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f1344ff9700 15 client.28614.objecter _session_command_op_remove 7 16 2024-02-10T08:47:34.570 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 10 librados: watch_flush enter 2024-02-10T08:47:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 10 librados: watch_flush exit 2024-02-10T08:47:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 10 client.28614.objecter close_session for osd.0 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6801/3321596972 conn(0x7f1328004530 legacy=0x7f1328002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280051a0 osd=0 1 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 10 client.28614.objecter close_session for osd.1 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6805/133516126 conn(0x7f1328003b40 legacy=0x7f132800a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f1328004270 osd=1 1 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 10 client.28614.objecter close_session for osd.2 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.568+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6809/3575881728 conn(0x7f1328002070 legacy=0x7f132800f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f13280033a0 osd=2 1 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter close_session for osd.3 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6813/2871825691 conn(0x7f1328010ed0 legacy=0x7f1328011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800c8d0 osd=3 1 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter close_session for osd.4 2024-02-10T08:47:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f1347fff700 1 -- 172.21.15.107:0/1248871181 reap_dead start 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.195:6800/3310789251 conn(0x7f13280166f0 legacy=0x7f1328018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f1347fff700 1 -- 172.21.15.107:0/1248871181 reap_dead start 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132800d010 osd=4 1 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter close_session for osd.5 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.195:6804/4178321486 conn(0x7f132801a8e0 legacy=0x7f132801cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801a200 osd=5 1 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter close_session for osd.6 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.195:6808/1417479017 conn(0x7f13280202b0 legacy=0x7f1328020710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801e5b0 osd=6 1 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter close_session for osd.7 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.195:6812/1934176928 conn(0x7f1328023470 legacy=0x7f1328025860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter put_session s=0x7f132801f800 osd=7 1 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 20 client.28614.objecter shutdown clearing up homeless session... 2024-02-10T08:47:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 10 client.28614.objecter successfully canceled tick 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1334063da0 legacy=0x7f1334066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> v1:172.21.15.107:6789/0 conn(0x7f1348104330 legacy=0x7f13481184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f1347fff700 1 -- 172.21.15.107:0/1248871181 reap_dead start 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 shutdown_connections 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 >> 172.21.15.107:0/1248871181 conn(0x7f13480f3640 msgr2=0x7f13481077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 shutdown_connections 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 -- 172.21.15.107:0/1248871181 wait complete. 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:34.569+0000 7f134e15a700 1 librados: shutdown 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:47:34.885 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:33.545465+0000 mgr.y (mgr.14553) 584 : cluster [DBG] pgmap v1049: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:34.165671+0000 mon.a (mon.0) 3330 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:34.165736+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:34.167236+0000 mon.b (mon.1) 782 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:34.167577+0000 mon.a (mon.0) 3332 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:34.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:34.428254+0000 mon.a (mon.0) 3333 : audit [DBG] from='client.? v1:172.21.15.107:0/1248871181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:33.545465+0000 mgr.y (mgr.14553) 584 : cluster [DBG] pgmap v1049: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:34.165671+0000 mon.a (mon.0) 3330 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:34.165736+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:34.167236+0000 mon.b (mon.1) 782 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:34.167577+0000 mon.a (mon.0) 3332 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:34.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:34.428254+0000 mon.a (mon.0) 3333 : audit [DBG] from='client.? v1:172.21.15.107:0/1248871181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:34.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:34.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:35.021 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:33.545465+0000 mgr.y (mgr.14553) 584 : cluster [DBG] pgmap v1049: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:34.165671+0000 mon.a (mon.0) 3330 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:34.165736+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:34.167236+0000 mon.b (mon.1) 782 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:34.167577+0000 mon.a (mon.0) 3332 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]: dispatch 2024-02-10T08:47:35.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:34.428254+0000 mon.a (mon.0) 3333 : audit [DBG] from='client.? v1:172.21.15.107:0/1248871181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:35.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:47:35.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:35.161451+0000 mon.a (mon.0) 3334 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:35.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:35.164935+0000 mon.a (mon.0) 3335 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:35.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:35.164989+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-10T08:47:35.916 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:35] "GET /metrics HTTP/1.1" 200 190338 "" "Prometheus/2.33.4" 2024-02-10T08:47:35.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:35.161451+0000 mon.a (mon.0) 3334 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:35.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:35.164935+0000 mon.a (mon.0) 3335 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:35.916 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:35.164989+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-10T08:47:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:35.161451+0000 mon.a (mon.0) 3334 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:35.164935+0000 mon.a (mon.0) 3335 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-120"}]': finished 2024-02-10T08:47:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:35.164989+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2024-02-10T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.353+0000 7f91df03a700 1 -- 172.21.15.107:0/1644645222 >> v1:172.21.15.107:6790/0 conn(0x7f91d8104330 legacy=0x7f91d81067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.353+0000 7f91df03a700 1 -- 172.21.15.107:0/1644645222 shutdown_connections 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.353+0000 7f91df03a700 1 -- 172.21.15.107:0/1644645222 >> 172.21.15.107:0/1644645222 conn(0x7f91d80f3640 msgr2=0x7f91d80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.353+0000 7f91df03a700 1 -- 172.21.15.107:0/1644645222 shutdown_connections 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.353+0000 7f91df03a700 1 -- 172.21.15.107:0/1644645222 wait complete. 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 librados: starting msgr at 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 librados: starting objecter 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 Processor -- start 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 -- start start 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 librados: setting wanted keys 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 librados: calling monclient init 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91d806ce70 con 0x7f91d8104330 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91d811abe0 con 0x7f91d80fda50 2024-02-10T08:47:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91df03a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91d811bd80 con 0x7f91d81007f0 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d7fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f91d81007f0 0x7f91d8114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43304/0 (socket says 172.21.15.107:43304) 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d7fff700 1 -- 172.21.15.107:0/980444823 learned_addr learned my addr 172.21.15.107:0/980444823 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91dd5d7700 1 --1- 172.21.15.107:0/980444823 >> v1:172.21.15.107:6789/0 conn(0x7f91d8104330 0x7f91d81184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58510/0 (socket says 172.21.15.107:58510) 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter ms_handle_connect 0x7f91d81007f0 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter ms_handle_connect 0x7f91d8104330 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter ms_handle_connect 0x7f91d80fda50 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36068016 0 0) 0x7f91d811bd80 con 0x7f91d81007f0 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.354+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c8002220 con 0x7f91d81007f0 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229964527 0 0) 0x7f91d811abe0 con 0x7f91d80fda50 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d811bd80 con 0x7f91d80fda50 2024-02-10T08:47:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237984385 0 0) 0x7f91c8002220 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d811abe0 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91c4001880 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 436496378 0 0) 0x7f91d811bd80 con 0x7f91d80fda50 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c8002220 con 0x7f91d80fda50 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91cc001880 con 0x7f91d80fda50 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3916284056 0 0) 0x7f91d811abe0 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 >> v1:172.21.15.195:6789/0 conn(0x7f91d80fda50 legacy=0x7f91d80fcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 >> v1:172.21.15.107:6789/0 conn(0x7f91d8104330 legacy=0x7f91d81184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d811cf20 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d8119c30 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f91c4002360 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91c4005af0 con 0x7f91d81007f0 2024-02-10T08:47:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91df03a700 10 client.54197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d811a0a0 con 0x7f91d81007f0 2024-02-10T08:47:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.355+0000 7f91df03a700 1 librados: init done 2024-02-10T08:47:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.356+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d805f700 con 0x7f91d81007f0 2024-02-10T08:47:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.356+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f91c4018a20 con 0x7f91d81007f0 2024-02-10T08:47:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(696..696 src has 420..696) v4 ==== 8182+0+0 (unknown 4038001175 0 0) 0x7f91c4078d90 con 0x7f91d81007f0 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 10 client.54197.objecter ms_dispatch 0x7f91d8001550 osd_map(696..696 src has 420..696) v4 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 3 client.54197.objecter handle_osd_map got epochs [696,696] > 0 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 3 client.54197.objecter handle_osd_map decoding full epoch 696 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 20 client.54197.objecter dump_active .. 0 homeless 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.357+0000 7f91d5ffb700 10 client.54197.objecter ms_handle_connect 0x7f91c8063bb0 2024-02-10T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.361+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f91c404c150 con 0x7f91d81007f0 2024-02-10T08:47:36.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:36.542+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]} v 0) v1 -- 0x7f91d80475e0 con 0x7f91d81007f0 2024-02-10T08:47:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:35.545848+0000 mgr.y (mgr.14553) 585 : cluster [DBG] pgmap v1052: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:36.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:36.173606+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-10T08:47:36.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:36.544651+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.107:0/980444823' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:36.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:36.544982+0000 mon.a (mon.0) 3338 : audit [INF] from='client.54197 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:36.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:35.545848+0000 mgr.y (mgr.14553) 585 : cluster [DBG] pgmap v1052: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:36.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:36.173606+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-10T08:47:36.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:36.544651+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.107:0/980444823' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:36.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:36.544982+0000 mon.a (mon.0) 3338 : audit [INF] from='client.54197 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:35.545848+0000 mgr.y (mgr.14553) 585 : cluster [DBG] pgmap v1052: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:36.173606+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2024-02-10T08:47:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:36.544651+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.107:0/980444823' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:37.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:36.544982+0000 mon.a (mon.0) 3338 : audit [INF] from='client.54197 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]: dispatch 2024-02-10T08:47:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.175+0000 7f91d5ffb700 1 -- 172.21.15.107:0/980444823 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]=0 set 16.1d pg_upmap_items mapping to [2->5,1->3,0->4] v697) v1 ==== 161+0+0 (unknown 881381505 0 0) 0x7f91c404a560 con 0x7f91d81007f0 2024-02-10T08:47:37.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.178+0000 7f91df03a700 10 librados: watch_flush enter 2024-02-10T08:47:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.181+0000 7f91df03a700 10 librados: watch_flush exit 2024-02-10T08:47:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.181+0000 7f91df03a700 20 client.54197.objecter shutdown clearing up homeless session... 2024-02-10T08:47:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.181+0000 7f91df03a700 10 client.54197.objecter successfully canceled tick 2024-02-10T08:47:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.182+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 >> v1:172.21.15.107:6800/2634842009 conn(0x7f91c8063bb0 legacy=0x7f91c8066070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.182+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 >> v1:172.21.15.107:6790/0 conn(0x7f91d81007f0 legacy=0x7f91d8114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.185+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 shutdown_connections 2024-02-10T08:47:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.185+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 >> 172.21.15.107:0/980444823 conn(0x7f91d80f3640 msgr2=0x7f91d81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.188+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 shutdown_connections 2024-02-10T08:47:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.188+0000 7f91df03a700 1 -- 172.21.15.107:0/980444823 wait complete. 2024-02-10T08:47:37.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:37.189+0000 7f91df03a700 1 librados: shutdown 2024-02-10T08:47:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:set 16.1d pg_upmap_items mapping to [2->5,1->3,0->4] 2024-02-10T08:47:38.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:37.176331+0000 mon.a (mon.0) 3339 : audit [INF] from='client.54197 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]': finished 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:37.176441+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:37.186398+0000 mon.b (mon.1) 783 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:37.187537+0000 mon.a (mon.0) 3341 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:37.546303+0000 mgr.y (mgr.14553) 586 : cluster [DBG] pgmap v1055: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:37.176331+0000 mon.a (mon.0) 3339 : audit [INF] from='client.54197 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]': finished 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:37.176441+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:37.186398+0000 mon.b (mon.1) 783 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:37.187537+0000 mon.a (mon.0) 3341 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:37.546303+0000 mgr.y (mgr.14553) 586 : cluster [DBG] pgmap v1055: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:38.477 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:38.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:38.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:47:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:37.176331+0000 mon.a (mon.0) 3339 : audit [INF] from='client.54197 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "16.1d", "id": [2, 5, 1, 3, 0, 4]}]': finished 2024-02-10T08:47:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:37.176441+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2024-02-10T08:47:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:37.186398+0000 mon.b (mon.1) 783 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:37.187537+0000 mon.a (mon.0) 3341 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:37.546303+0000 mgr.y (mgr.14553) 586 : cluster [DBG] pgmap v1055: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:38.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:38.616 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:38.638 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:38.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:38.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:38.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:38.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:38.978 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:39.000 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:39.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:39.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:38.182240+0000 mon.a (mon.0) 3342 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:38.182334+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-10T08:47:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:38.192249+0000 mon.b (mon.1) 784 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:38.196113+0000 mon.a (mon.0) 3344 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:47:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:38.182240+0000 mon.a (mon.0) 3342 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:38.182334+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-10T08:47:39.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:38.192249+0000 mon.b (mon.1) 784 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:38.196113+0000 mon.a (mon.0) 3344 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:38.182240+0000 mon.a (mon.0) 3342 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:38.182334+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2024-02-10T08:47:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:38.192249+0000 mon.b (mon.1) 784 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:38.196113+0000 mon.a (mon.0) 3344 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:39.437 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:39.981 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:39.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:40.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:40.115 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:40.139 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:40.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:39.183081+0000 mon.a (mon.0) 3345 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:39.183136+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:39.184268+0000 mon.b (mon.1) 785 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:39.187589+0000 mon.a (mon.0) 3347 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:39.546681+0000 mgr.y (mgr.14553) 587 : cluster [DBG] pgmap v1058: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.170508+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:39.183081+0000 mon.a (mon.0) 3345 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:39.183136+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:39.184268+0000 mon.b (mon.1) 785 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:39.187589+0000 mon.a (mon.0) 3347 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:39.546681+0000 mgr.y (mgr.14553) 587 : cluster [DBG] pgmap v1058: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:40.236 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.170508+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:40.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:40.273 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:39.183081+0000 mon.a (mon.0) 3345 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:39.183136+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2024-02-10T08:47:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:39.184268+0000 mon.b (mon.1) 785 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:39.187589+0000 mon.a (mon.0) 3347 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:39.546681+0000 mgr.y (mgr.14553) 587 : cluster [DBG] pgmap v1058: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:47:40.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.170508+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:40.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.657+0000 7f18b4f5a700 1 -- 172.21.15.107:0/3497267974 >> v1:172.21.15.107:6790/0 conn(0x7f18b01007f0 legacy=0x7f18b0102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.657+0000 7f18b4f5a700 1 -- 172.21.15.107:0/3497267974 shutdown_connections 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.657+0000 7f18b4f5a700 1 -- 172.21.15.107:0/3497267974 >> 172.21.15.107:0/3497267974 conn(0x7f18b00f3640 msgr2=0x7f18b00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.657+0000 7f18b4f5a700 1 -- 172.21.15.107:0/3497267974 shutdown_connections 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.657+0000 7f18b4f5a700 1 -- 172.21.15.107:0/3497267974 wait complete. 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 librados: starting msgr at 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 librados: starting objecter 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 Processor -- start 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 -- start start 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 librados: setting wanted keys 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 librados: calling monclient init 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18b0117330 con 0x7f18b0104330 2024-02-10T08:47:40.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18b0118510 con 0x7f18b01007f0 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18b4f5a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18b0119710 con 0x7f18b00fda50 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18ae59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f18b00fda50 0x7f18b00f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47036/0 (socket says 172.21.15.107:47036) 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f18ae59c700 1 -- 172.21.15.107:0/2640067493 learned_addr learned my addr 172.21.15.107:0/2640067493 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter ms_handle_connect 0x7f18b0104330 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter ms_handle_connect 0x7f18b00fda50 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter ms_handle_connect 0x7f18b01007f0 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793605685 0 0) 0x7f18b0117330 con 0x7f18b0104330 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.658+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18980033f0 con 0x7f18b0104330 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921673553 0 0) 0x7f18b0119710 con 0x7f18b00fda50 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b0117330 con 0x7f18b00fda50 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475283093 0 0) 0x7f18b0118510 con 0x7f18b01007f0 2024-02-10T08:47:40.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b0119710 con 0x7f18b01007f0 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1055713411 0 0) 0x7f18b0117330 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18b0118510 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18a4001980 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1203803488 0 0) 0x7f18980033f0 con 0x7f18b0104330 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18b0117330 con 0x7f18b0104330 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18a8001bb0 con 0x7f18b0104330 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973743226 0 0) 0x7f18b0119710 con 0x7f18b01007f0 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18980033f0 con 0x7f18b01007f0 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18a0000fe0 con 0x7f18b01007f0 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333487000 0 0) 0x7f18b0118510 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 >> v1:172.21.15.195:6789/0 conn(0x7f18b01007f0 legacy=0x7f18b00fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 >> v1:172.21.15.107:6789/0 conn(0x7f18b0104330 legacy=0x7f18b01159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b011a910 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b0119900 con 0x7f18b00fda50 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f18b4f5a700 10 client.54203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:40.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b0119d70 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f18b4f5a700 1 librados: init done 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f18a4002d00 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.659+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f18a4005bc0 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.660+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1890004f20 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.660+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f18a4001fa0 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(700..700 src has 420..700) v4 ==== 8593+0+0 (unknown 1883505288 0 0) 0x7f18a4078f10 con 0x7f18b00fda50 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 10 client.54203.objecter ms_dispatch 0x7f18b0001550 osd_map(700..700 src has 420..700) v4 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 3 client.54203.objecter handle_osd_map got epochs [700,700] > 0 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 3 client.54203.objecter handle_osd_map decoding full epoch 700 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 20 client.54203.objecter dump_active .. 0 homeless 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.661+0000 7f189f7fe700 10 client.54203.objecter ms_handle_connect 0x7f1898063c30 2024-02-10T08:47:40.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.666+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f18a404c0f0 con 0x7f18b00fda50 2024-02-10T08:47:40.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:40.855+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f18900051d0 con 0x7f18b00fda50 2024-02-10T08:47:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:41.122 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:41.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:41.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.148+0000 7f189f7fe700 1 -- 172.21.15.107:0/2640067493 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v701) v1 ==== 97+0+0 (unknown 482125050 0 0) 0x7f18a404a500 con 0x7f18b00fda50 2024-02-10T08:47:41.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 10 librados: watch_flush enter 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 10 librados: watch_flush exit 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 20 client.54203.objecter shutdown clearing up homeless session... 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 10 client.54203.objecter successfully canceled tick 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1898063c30 legacy=0x7f18980660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 >> v1:172.21.15.107:6790/0 conn(0x7f18b00fda50 legacy=0x7f18b00f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 shutdown_connections 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 >> 172.21.15.107:0/2640067493 conn(0x7f18b00f3640 msgr2=0x7f18b01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 shutdown_connections 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 -- 172.21.15.107:0/2640067493 wait complete. 2024-02-10T08:47:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:41.150+0000 7f18b4f5a700 1 librados: shutdown 2024-02-10T08:47:41.153 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:47:41.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:41.256 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:39.984563+0000 osd.7 (osd.7) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:39.984922+0000 osd.7 (osd.7) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.194571+0000 mon.a (mon.0) 3349 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:41.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:40.194654+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.195873+0000 mon.b (mon.1) 786 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.196422+0000 mon.a (mon.0) 3351 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.857136+0000 mon.c (mon.2) 465 : audit [INF] from='client.? v1:172.21.15.107:0/2640067493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:40.857371+0000 mon.a (mon.0) 3352 : audit [INF] from='client.54203 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:41.144658+0000 mon.a (mon.0) 3353 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:41.144694+0000 mon.a (mon.0) 3354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:41.148361+0000 mon.a (mon.0) 3355 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]': finished 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:41.148437+0000 mon.a (mon.0) 3356 : audit [INF] from='client.54203 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:41.148488+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:39.984563+0000 osd.7 (osd.7) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:39.984922+0000 osd.7 (osd.7) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.194571+0000 mon.a (mon.0) 3349 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:40.194654+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.195873+0000 mon.b (mon.1) 786 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.196422+0000 mon.a (mon.0) 3351 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.857136+0000 mon.c (mon.2) 465 : audit [INF] from='client.? v1:172.21.15.107:0/2640067493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:40.857371+0000 mon.a (mon.0) 3352 : audit [INF] from='client.54203 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:41.144658+0000 mon.a (mon.0) 3353 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:41.144694+0000 mon.a (mon.0) 3354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:41.148361+0000 mon.a (mon.0) 3355 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]': finished 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:41.148437+0000 mon.a (mon.0) 3356 : audit [INF] from='client.54203 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:41.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:41.148488+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:39.984563+0000 osd.7 (osd.7) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:39.984922+0000 osd.7 (osd.7) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.194571+0000 mon.a (mon.0) 3349 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:40.194654+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.195873+0000 mon.b (mon.1) 786 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.196422+0000 mon.a (mon.0) 3351 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]: dispatch 2024-02-10T08:47:41.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.857136+0000 mon.c (mon.2) 465 : audit [INF] from='client.? v1:172.21.15.107:0/2640067493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:40.857371+0000 mon.a (mon.0) 3352 : audit [INF] from='client.54203 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:41.144658+0000 mon.a (mon.0) 3353 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:41.144694+0000 mon.a (mon.0) 3354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:41.148361+0000 mon.a (mon.0) 3355 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-122", "mode": "writeback"}]': finished 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:41.148437+0000 mon.a (mon.0) 3356 : audit [INF] from='client.54203 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:41.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:41.148488+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2024-02-10T08:47:42.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:42.405 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:42.622 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:41.547414+0000 mgr.y (mgr.14553) 588 : cluster [DBG] pgmap v1061: 194 pgs: 3 creating+peering, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:41.743533+0000 mgr.y (mgr.14553) 589 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:42.155764+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:41.547414+0000 mgr.y (mgr.14553) 588 : cluster [DBG] pgmap v1061: 194 pgs: 3 creating+peering, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:41.743533+0000 mgr.y (mgr.14553) 589 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:42.155764+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-10T08:47:42.623 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:41.547414+0000 mgr.y (mgr.14553) 588 : cluster [DBG] pgmap v1061: 194 pgs: 3 creating+peering, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:42.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:41.743533+0000 mgr.y (mgr.14553) 589 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:42.624 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:42.155764+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2024-02-10T08:47:42.705 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:42.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:42.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:42.840 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:42.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:42.979 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 0] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:47:42.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:47:42.979 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-02-10T08:47:42.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd in 1 2024-02-10T08:47:43.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:43.075 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:43.095 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:43.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:43.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:43.225 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:43.246 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:43.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:43.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:43.377 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:44.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:47:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:44.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:43.159099+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-10T08:47:44.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:43.194165+0000 mon.b (mon.1) 787 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:43.194495+0000 mon.a (mon.0) 3360 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:43.547872+0000 mgr.y (mgr.14553) 590 : cluster [DBG] pgmap v1064: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:43.159099+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-10T08:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:43.194165+0000 mon.b (mon.1) 787 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:43.194495+0000 mon.a (mon.0) 3360 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:43.547872+0000 mgr.y (mgr.14553) 590 : cluster [DBG] pgmap v1064: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:44.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:44.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:44.536 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:43.159099+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2024-02-10T08:47:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:43.194165+0000 mon.b (mon.1) 787 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:43.194495+0000 mon.a (mon.0) 3360 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:43.547872+0000 mgr.y (mgr.14553) 590 : cluster [DBG] pgmap v1064: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:47:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:44.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:47:44.852 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:45.108 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:45.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:45.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:45.241 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:45.365 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:45.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:45.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:44.164008+0000 mon.a (mon.0) 3361 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:44.164071+0000 mon.a (mon.0) 3362 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:44.168411+0000 mon.b (mon.1) 788 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:44.168702+0000 mon.a (mon.0) 3363 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:44.164008+0000 mon.a (mon.0) 3361 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:44.164071+0000 mon.a (mon.0) 3362 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:44.168411+0000 mon.b (mon.1) 788 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.370 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:44.168702+0000 mon.a (mon.0) 3363 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:45.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:44.164008+0000 mon.a (mon.0) 3361 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:45.487 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:44.164071+0000 mon.a (mon.0) 3362 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2024-02-10T08:47:45.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:44.168411+0000 mon.b (mon.1) 788 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:44.168702+0000 mon.a (mon.0) 3363 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:45] "GET /metrics HTTP/1.1" 200 190437 "" "Prometheus/2.33.4" 2024-02-10T08:47:46.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 -- 172.21.15.107:0/2774712853 >> v1:172.21.15.107:6789/0 conn(0x7f91c00f95d0 legacy=0x7f91c0102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 -- 172.21.15.107:0/2774712853 shutdown_connections 2024-02-10T08:47:46.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 -- 172.21.15.107:0/2774712853 >> 172.21.15.107:0/2774712853 conn(0x7f91c00f2510 msgr2=0x7f91c00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 -- 172.21.15.107:0/2774712853 shutdown_connections 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 -- 172.21.15.107:0/2774712853 wait complete. 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 librados: starting msgr at 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.890+0000 7f91c6631700 1 librados: starting objecter 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 Processor -- start 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 -- start start 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 librados: setting wanted keys 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 librados: calling monclient init 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c01194e0 con 0x7f91c0104350 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c011a6c0 con 0x7f91c00f95d0 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91c6631700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c011b8c0 con 0x7f91c00f8a20 2024-02-10T08:47:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bffff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f91c00f8a20 0x7f91c01022e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47058/0 (socket says 172.21.15.107:47058) 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bffff700 1 -- 172.21.15.107:0/1623783119 learned_addr learned my addr 172.21.15.107:0/1623783119 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter ms_handle_connect 0x7f91c00f8a20 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter ms_handle_connect 0x7f91c0104350 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter ms_handle_connect 0x7f91c00f95d0 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159398876 0 0) 0x7f91c011b8c0 con 0x7f91c00f8a20 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a80038e0 con 0x7f91c00f8a20 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312539257 0 0) 0x7f91c01194e0 con 0x7f91c0104350 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c011b8c0 con 0x7f91c0104350 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90019223 0 0) 0x7f91c011a6c0 con 0x7f91c00f95d0 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.891+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c01194e0 con 0x7f91c00f95d0 2024-02-10T08:47:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 986961190 0 0) 0x7f91a80038e0 con 0x7f91c00f8a20 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c011a6c0 con 0x7f91c00f8a20 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91b0001960 con 0x7f91c00f8a20 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782992651 0 0) 0x7f91c011b8c0 con 0x7f91c0104350 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a80038e0 con 0x7f91c0104350 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816714215 0 0) 0x7f91c01194e0 con 0x7f91c00f95d0 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c011b8c0 con 0x7f91c00f95d0 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91b8001b40 con 0x7f91c0104350 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91b4002c70 con 0x7f91c00f95d0 2024-02-10T08:47:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1785171185 0 0) 0x7f91c011a6c0 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.195:6789/0 conn(0x7f91c00f95d0 legacy=0x7f91c01029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6789/0 conn(0x7f91c0104350 legacy=0x7f91c0117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c011cac0 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c01196d0 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91c6631700 10 client.54209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c0119b40 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91c6631700 1 librados: init done 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f91b0002370 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.892+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91b0005ac0 con 0x7f91c00f8a20 2024-02-10T08:47:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.893+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f91ac004f00 con 0x7f91c00f8a20 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.893+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f91b0001ed0 con 0x7f91c00f8a20 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(705..705 src has 420..705) v4 ==== 8593+0+0 (unknown 1314505525 0 0) 0x7f91b0078f30 con 0x7f91c00f8a20 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 osd_map(705..705 src has 420..705) v4 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 3 client.54209.objecter handle_osd_map got epochs [705,705] > 0 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 3 client.54209.objecter handle_osd_map decoding full epoch 705 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 20 client.54209.objecter dump_active .. 0 homeless 2024-02-10T08:47:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v705) v1 ==== 54+0+16 (unknown 819249221 0 2420000192) 0x7f91b007b340 con 0x7f91c00f8a20 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91a8063cc0 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 10 client.54209.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 1 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 1 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 0 1 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 4 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 10 client.54209.objecter _send_command 1 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac004950 con 0x7f91ac004530 2024-02-10T08:47:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.894+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac004530 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.895+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac004950 con 0x7f91ac004530 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.895+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.895+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 1 = system:0 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.895+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 4 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.895+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 0 1 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 10 client.54209.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 2 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 2 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac0051a0 osd=0 3 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 0 2 2024-02-10T08:47:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 4 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 10 client.54209.objecter _send_command 2 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.910+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac004d30 con 0x7f91ac004530 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac004d30 con 0x7f91ac004530 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 4 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 0 2 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.911+0000 7f91c6631700 10 client.54209.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 3 2024-02-10T08:47:46.019 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 1 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 4 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 10 client.54209.objecter _send_command 3 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac00c000 con 0x7f91ac004150 2024-02-10T08:47:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac004150 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac00c000 con 0x7f91ac004150 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 3 = system:0 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 4 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.912+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 1 3 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 10 client.54209.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 4 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 4 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003fe0 osd=1 3 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 1 4 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 4 2024-02-10T08:47:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 10 client.54209.objecter _send_command 4 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.927+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac001850 con 0x7f91ac004150 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac001850 con 0x7f91ac004150 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 4 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 1 4 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 10 client.54209.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 5 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 5 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 2 5 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 4 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 10 client.54209.objecter _send_command 5 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.928+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac003e70 con 0x7f91ac00d2f0 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac00d2f0 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac003e70 con 0x7f91ac00d2f0 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 5 = system:0 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 4 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.929+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 2 5 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:46.023 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 10 client.54209.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 6 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 6 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003630 osd=2 3 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 2 6 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 4 2024-02-10T08:47:46.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 10 client.54209.objecter _send_command 6 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.943+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac00e6c0 con 0x7f91ac00d2f0 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac00e6c0 con 0x7f91ac00d2f0 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 4 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 2 6 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 10 client.54209.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 7 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 7 2024-02-10T08:47:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 3 7 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 4 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 10 client.54209.objecter _send_command 7 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac015a60 con 0x7f91ac011e80 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.945+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac011e80 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.946+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac015a60 con 0x7f91ac011e80 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.946+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.946+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 7 = system:0 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.946+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 4 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.946+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 3 7 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 10 client.54209.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 8 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 8 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac003390 osd=3 3 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 3 8 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 4 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 10 client.54209.objecter _send_command 8 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.960+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac00e100 con 0x7f91ac011e80 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac00e100 con 0x7f91ac011e80 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 4 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 3 8 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 10 client.54209.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 9 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 9 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 4 9 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 4 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 10 client.54209.objecter _send_command 9 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac00e400 con 0x7f91ac00ce50 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.962+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac00ce50 2024-02-10T08:47:46.028 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.963+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac00e400 con 0x7f91ac00ce50 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.963+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.963+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 9 = system:0 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.963+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 4 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.963+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 4 9 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 10 client.54209.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 10 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 10 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac00e100 osd=4 3 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 4 10 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 4 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 10 client.54209.objecter _send_command 10 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.977+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac00e400 con 0x7f91ac00ce50 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.979+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac00e400 con 0x7f91ac00ce50 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.979+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.979+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.979+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 4 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.979+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 4 10 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 10 client.54209.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 11 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 11 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 5 11 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 4 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 10 client.54209.objecter _send_command 11 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac017f00 con 0x7f91ac0179a0 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.980+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac0179a0 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.981+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac017f00 con 0x7f91ac0179a0 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.981+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.981+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 11 = system:0 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.981+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 4 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.981+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 5 11 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 10 client.54209.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 12 2024-02-10T08:47:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 12 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac00e400 osd=5 3 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 5 12 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 4 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 10 client.54209.objecter _send_command 12 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.995+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac017f00 con 0x7f91ac0179a0 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac017f00 con 0x7f91ac0179a0 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 4 2024-02-10T08:47:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 5 12 2024-02-10T08:47:46.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 10 client.54209.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 13 2024-02-10T08:47:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 13 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 6 13 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 4 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 10 client.54209.objecter _send_command 13 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac022b70 con 0x7f91ac01f060 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.997+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac01f060 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.998+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac022b70 con 0x7f91ac01f060 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.998+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.998+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 13 = system:0 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.998+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 4 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:45.998+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 6 13 2024-02-10T08:47:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 10 client.54209.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 14 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 14 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac017f00 osd=6 3 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 6 14 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 4 2024-02-10T08:47:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 10 client.54209.objecter _send_command 14 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.012+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac022b70 con 0x7f91ac01f060 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac022b70 con 0x7f91ac01f060 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 4 2024-02-10T08:47:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 6 14 2024-02-10T08:47:46.037 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 10 client.54209.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 15 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 15 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 7 15 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 4 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 10 client.54209.objecter _send_command 15 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ac024ed0 con 0x7f91ac0231f0 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.014+0000 7f91bd7fa700 10 client.54209.objecter ms_handle_connect 0x7f91ac0231f0 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.015+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f91ac024ed0 con 0x7f91ac0231f0 2024-02-10T08:47:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.015+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.015+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 15 = system:0 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.015+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 4 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.015+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 7 15 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 10 client.54209.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign -1 16 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 20 client.54209.objecter _get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 15 client.54209.objecter _session_command_op_remove -1 16 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 20 client.54209.objecter get_session s=0x7f91ac022b70 osd=7 3 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 15 client.54209.objecter _session_command_op_assign 7 16 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 4 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 10 client.54209.objecter _send_command 16 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.029+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f91ac024ed0 con 0x7f91ac0231f0 2024-02-10T08:47:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91bd7fa700 1 -- 172.21.15.107:0/1623783119 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f91ac024ed0 con 0x7f91ac0231f0 2024-02-10T08:47:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91bd7fa700 10 client.54209.objecter ms_dispatch 0x7f91c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:47:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91bd7fa700 10 client.54209.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91bd7fa700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 4 2024-02-10T08:47:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91bd7fa700 15 client.54209.objecter _session_command_op_remove 7 16 2024-02-10T08:47:46.040 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91c6631700 10 librados: watch_flush enter 2024-02-10T08:47:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91c6631700 10 librados: watch_flush exit 2024-02-10T08:47:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91c6631700 10 client.54209.objecter close_session for osd.0 2024-02-10T08:47:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.031+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6801/3321596972 conn(0x7f91ac004530 legacy=0x7f91ac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac0051a0 osd=0 1 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.1 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6805/133516126 conn(0x7f91ac004150 legacy=0x7f91ac00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003fe0 osd=1 1 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.2 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6809/3575881728 conn(0x7f91ac00d2f0 legacy=0x7f91ac00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003630 osd=2 1 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.3 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6813/2871825691 conn(0x7f91ac011e80 legacy=0x7f91ac014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c4bce700 1 -- 172.21.15.107:0/1623783119 reap_dead start 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac003390 osd=3 1 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.4 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.195:6800/3310789251 conn(0x7f91ac00ce50 legacy=0x7f91ac015c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e100 osd=4 1 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.5 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.195:6804/4178321486 conn(0x7f91ac0179a0 legacy=0x7f91ac01d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac00e400 osd=5 1 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.6 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.195:6808/1417479017 conn(0x7f91ac01f060 legacy=0x7f91ac021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac017f00 osd=6 1 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter close_session for osd.7 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.195:6812/1934176928 conn(0x7f91ac0231f0 legacy=0x7f91ac023650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter put_session s=0x7f91ac022b70 osd=7 1 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 20 client.54209.objecter shutdown clearing up homeless session... 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 10 client.54209.objecter successfully canceled tick 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c4bce700 1 -- 172.21.15.107:0/1623783119 reap_dead start 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6800/2634842009 conn(0x7f91a8063cc0 legacy=0x7f91a8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> v1:172.21.15.107:6790/0 conn(0x7f91c00f8a20 legacy=0x7f91c01022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 shutdown_connections 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 >> 172.21.15.107:0/1623783119 conn(0x7f91c00f2510 msgr2=0x7f91c01067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 shutdown_connections 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 -- 172.21.15.107:0/1623783119 wait complete. 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:46.032+0000 7f91c6631700 1 librados: shutdown 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:47:46.366 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:47:46.366 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:47:46.367 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:47:46.367 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:47:46.391 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:45.167937+0000 mon.a (mon.0) 3364 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:46.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:45.168005+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-10T08:47:46.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:45.272470+0000 mon.c (mon.2) 466 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:47:46.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:45.548214+0000 mgr.y (mgr.14553) 591 : cluster [DBG] pgmap v1067: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:46.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:45.894516+0000 mon.c (mon.2) 467 : audit [DBG] from='client.? v1:172.21.15.107:0/1623783119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:45.167937+0000 mon.a (mon.0) 3364 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:45.168005+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-10T08:47:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:45.272470+0000 mon.c (mon.2) 466 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:47:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:45.548214+0000 mgr.y (mgr.14553) 591 : cluster [DBG] pgmap v1067: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:45.894516+0000 mon.c (mon.2) 467 : audit [DBG] from='client.? v1:172.21.15.107:0/1623783119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:45.167937+0000 mon.a (mon.0) 3364 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:45.168005+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2024-02-10T08:47:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:45.272470+0000 mon.c (mon.2) 466 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:47:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:45.548214+0000 mgr.y (mgr.14553) 591 : cluster [DBG] pgmap v1067: 194 pgs: 194 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-10T08:47:46.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:45.894516+0000 mon.c (mon.2) 467 : audit [DBG] from='client.? v1:172.21.15.107:0/1623783119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:46.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:46.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:46.612 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:46.764 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:46.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:46.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:46.895 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:46.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:47:47.145 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:47.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:47.234 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:47.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:46.186324+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:46.210327+0000 mon.b (mon.1) 789 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:46.210686+0000 mon.a (mon.0) 3367 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:46.900345+0000 mon.c (mon.2) 468 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:46.901096+0000 mon.c (mon.2) 469 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:47:47.274 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:46.906724+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:47.276 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.011+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/3646578664 >> v1:172.21.15.107:6789/0 conn(0x7f4a94103250 legacy=0x7f4a941056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.011+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/3646578664 shutdown_connections 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.011+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/3646578664 >> 172.21.15.107:0/3646578664 conn(0x7f4a940f2510 msgr2=0x7f4a940f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.011+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/3646578664 shutdown_connections 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/3646578664 wait complete. 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 librados: starting msgr at 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 librados: starting objecter 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 Processor -- start 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 -- start start 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 librados: setting wanted keys 2024-02-10T08:47:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 librados: calling monclient init 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a94117410 con 0x7f4a940fc910 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a941185d0 con 0x7f4a94103250 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.012+0000 7f4a9a0cf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a941197d0 con 0x7f4a940ff710 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a92ffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4a940ff710 0x7f4a94061460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47068/0 (socket says 172.21.15.107:47068) 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a937fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4a940fc910 0x7f4a94060d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48580/0 (socket says 172.21.15.107:48580) 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a92ffd700 1 -- 172.21.15.107:0/593588613 learned_addr learned my addr 172.21.15.107:0/593588613 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter ms_handle_connect 0x7f4a940ff710 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter ms_handle_connect 0x7f4a940fc910 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter ms_handle_connect 0x7f4a94103250 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271143264 0 0) 0x7f4a941197d0 con 0x7f4a940ff710 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003800 con 0x7f4a940ff710 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895635258 0 0) 0x7f4a94117410 con 0x7f4a940fc910 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a941197d0 con 0x7f4a940fc910 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898350294 0 0) 0x7f4a941185d0 con 0x7f4a94103250 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94117410 con 0x7f4a94103250 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517061583 0 0) 0x7f4a941197d0 con 0x7f4a940fc910 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a941185d0 con 0x7f4a940fc910 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619650353 0 0) 0x7f4a7c003800 con 0x7f4a940ff710 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a941197d0 con 0x7f4a940ff710 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a84002930 con 0x7f4a940fc910 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a88001920 con 0x7f4a940ff710 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532949997 0 0) 0x7f4a94117410 con 0x7f4a94103250 2024-02-10T08:47:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a7c003800 con 0x7f4a94103250 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a8c002f60 con 0x7f4a94103250 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 932298485 0 0) 0x7f4a941185d0 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 >> v1:172.21.15.107:6790/0 conn(0x7f4a940ff710 legacy=0x7f4a94061460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 >> v1:172.21.15.195:6789/0 conn(0x7f4a94103250 legacy=0x7f4a94115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9411a9d0 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a94117600 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a9a0cf700 10 client.28632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a94117a70 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a9a0cf700 1 librados: init done 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4a84002cb0 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.013+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4a84005a90 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.014+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a80004f20 con 0x7f4a940fc910 2024-02-10T08:47:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4a840015a0 con 0x7f4a940fc910 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(706..706 src has 420..706) v4 ==== 8705+0+0 (unknown 933075505 0 0) 0x7f4a84078f70 con 0x7f4a940fc910 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 10 client.28632.objecter ms_dispatch 0x7f4a94001550 osd_map(706..706 src has 420..706) v4 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 3 client.28632.objecter handle_osd_map got epochs [706,706] > 0 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 3 client.28632.objecter handle_osd_map decoding full epoch 706 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 20 client.28632.objecter dump_active .. 0 homeless 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.015+0000 7f4a90ff9700 10 client.28632.objecter ms_handle_connect 0x7f4a7c063ca0 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.019+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4a8404c0e0 con 0x7f4a940fc910 2024-02-10T08:47:47.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:47.209+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd in", "ids": ["1"]} v 0) v1 -- 0x7f4a800051d0 con 0x7f4a940fc910 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:46.186324+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:46.210327+0000 mon.b (mon.1) 789 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:46.210686+0000 mon.a (mon.0) 3367 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:46.900345+0000 mon.c (mon.2) 468 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:46.901096+0000 mon.c (mon.2) 469 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:47:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:46.906724+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:46.186324+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:46.210327+0000 mon.b (mon.1) 789 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:46.210686+0000 mon.a (mon.0) 3367 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:46.900345+0000 mon.c (mon.2) 468 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:46.901096+0000 mon.c (mon.2) 469 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:46.906724+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:47.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:47:47.210+0000 7fe8a61aa700 -1 mon.a@0(leader).osd e707 definitely_dead 0 2024-02-10T08:47:47.473 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:47.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:47.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:47.607 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:47.705 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:47.705 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:47.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:47.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:47.841 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:47.913 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:47.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:48.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:48.048 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:48.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.200+0000 7f4a90ff9700 1 -- 172.21.15.107:0/593588613 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["1"]}]=0 marked in osd.1. v708) v1 ==== 85+0+0 (unknown 1145706930 0 0) 0x7f4a8404a4f0 con 0x7f4a940fc910 2024-02-10T08:47:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.207+0000 7f4a9a0cf700 10 librados: watch_flush enter 2024-02-10T08:47:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.210+0000 7f4a9a0cf700 10 librados: watch_flush exit 2024-02-10T08:47:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.210+0000 7f4a9a0cf700 20 client.28632.objecter shutdown clearing up homeless session... 2024-02-10T08:47:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.210+0000 7f4a9a0cf700 10 client.28632.objecter successfully canceled tick 2024-02-10T08:47:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.211+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4a7c063ca0 legacy=0x7f4a7c066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.211+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 >> v1:172.21.15.107:6789/0 conn(0x7f4a940fc910 legacy=0x7f4a94060d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.213+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 shutdown_connections 2024-02-10T08:47:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.213+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 >> 172.21.15.107:0/593588613 conn(0x7f4a940f2510 msgr2=0x7f4a941066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:48.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.218+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 shutdown_connections 2024-02-10T08:47:48.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.218+0000 7f4a9a0cf700 1 -- 172.21.15.107:0/593588613 wait complete. 2024-02-10T08:47:48.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:48.224+0000 7f4a9a0cf700 1 librados: shutdown 2024-02-10T08:47:48.230 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.1. 2024-02-10T08:47:48.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:47.185922+0000 mon.a (mon.0) 3369 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:48.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:47.185996+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-10T08:47:48.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:47.190273+0000 mon.b (mon.1) 790 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:47.193717+0000 mon.a (mon.0) 3371 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:47.210634+0000 mon.a (mon.0) 3372 : audit [DBG] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:47.548714+0000 mgr.y (mgr.14553) 592 : cluster [DBG] pgmap v1070: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:47.185922+0000 mon.a (mon.0) 3369 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:47.185996+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:47.190273+0000 mon.b (mon.1) 790 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:47.193717+0000 mon.a (mon.0) 3371 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:47.210634+0000 mon.a (mon.0) 3372 : audit [DBG] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-10T08:47:48.256 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:47.548714+0000 mgr.y (mgr.14553) 592 : cluster [DBG] pgmap v1070: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:47.185922+0000 mon.a (mon.0) 3369 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:47.185996+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:47.190273+0000 mon.b (mon.1) 790 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:47.193717+0000 mon.a (mon.0) 3371 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]: dispatch 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:47.210634+0000 mon.a (mon.0) 3372 : audit [DBG] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-02-10T08:47:48.283 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:47.548714+0000 mgr.y (mgr.14553) 592 : cluster [DBG] pgmap v1070: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:48.293 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:48.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:48.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:48.426 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:48.480 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:48.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:48.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:48.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:48.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:47:48.763 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:47:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:47:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:47:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:47:48.764 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:47:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:49.268 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-02-10T08:47:49.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:48.191140+0000 mon.a (mon.0) 3373 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:49.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:48.200060+0000 mon.a (mon.0) 3374 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:49.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:48.200139+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-10T08:47:49.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:48.200183+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-10T08:47:49.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:47:49.662 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:49.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:48.191140+0000 mon.a (mon.0) 3373 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:49.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:48.200060+0000 mon.a (mon.0) 3374 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:49.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:48.200139+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-10T08:47:49.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:48.200183+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-10T08:47:49.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:48.191140+0000 mon.a (mon.0) 3373 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:49.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:48.200060+0000 mon.a (mon.0) 3374 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-122"}]': finished 2024-02-10T08:47:49.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:48.200139+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? v1:172.21.15.107:0/593588613' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-02-10T08:47:49.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:48.200183+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-10T08:47:49.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:47:50.424 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:47:50.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:49.202124+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-10T08:47:50.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:49.549093+0000 mgr.y (mgr.14553) 593 : cluster [DBG] pgmap v1073: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:50.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:49.202124+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-10T08:47:50.426 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:49.549093+0000 mgr.y (mgr.14553) 593 : cluster [DBG] pgmap v1073: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:50.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:47:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.430+0000 7f7bb4857700 1 -- 172.21.15.107:0/1882167829 >> v1:172.21.15.195:6789/0 conn(0x7f7bb0103320 legacy=0x7f7bb01057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.430+0000 7f7bb4857700 1 -- 172.21.15.107:0/1882167829 shutdown_connections 2024-02-10T08:47:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.430+0000 7f7bb4857700 1 -- 172.21.15.107:0/1882167829 >> 172.21.15.107:0/1882167829 conn(0x7f7bb006c090 msgr2=0x7f7bb006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.431+0000 7f7bb4857700 1 -- 172.21.15.107:0/1882167829 shutdown_connections 2024-02-10T08:47:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.431+0000 7f7bb4857700 1 -- 172.21.15.107:0/1882167829 wait complete. 2024-02-10T08:47:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.431+0000 7f7bb4857700 1 librados: starting msgr at 2024-02-10T08:47:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.431+0000 7f7bb4857700 1 librados: starting objecter 2024-02-10T08:47:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.431+0000 7f7bb4857700 1 Processor -- start 2024-02-10T08:47:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 -- start start 2024-02-10T08:47:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 librados: setting wanted keys 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 librados: calling monclient init 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7bb010eaa0 con 0x7f7bb00ff720 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7bb00fc070 con 0x7f7bb0103320 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7bb4857700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7bb00fc200 con 0x7f7bb00fc960 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7baed9d700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7bb0103320 0x7f7bb011ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:43344/0 (socket says 172.21.15.107:43344) 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7baed9d700 1 -- 172.21.15.107:0/741356096 learned_addr learned my addr 172.21.15.107:0/741356096 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 10 client.?.objecter ms_handle_connect 0x7f7bb0103320 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 10 client.?.objecter ms_handle_connect 0x7f7bb00ff720 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853365477 0 0) 0x7f7bb00fc070 con 0x7f7bb0103320 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.432+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98003030 con 0x7f7bb0103320 2024-02-10T08:47:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2620830351 0 0) 0x7f7b98003030 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb00fc070 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 10 client.?.objecter ms_handle_connect 0x7f7bb00fc960 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7ba4002d90 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359650677 0 0) 0x7f7bb00fc070 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 >> v1:172.21.15.107:6790/0 conn(0x7f7bb00fc960 legacy=0x7f7bb00fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 >> v1:172.21.15.107:6789/0 conn(0x7f7bb00ff720 legacy=0x7f7bb01195a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb00fc390 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb011e450 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7bb4857700 10 client.48943.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb011e8c0 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7bb4857700 1 librados: init done 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7ba4001b20 con 0x7f7bb0103320 2024-02-10T08:47:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.433+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7ba4003870 con 0x7f7bb0103320 2024-02-10T08:47:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.434+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90004f50 con 0x7f7bb0103320 2024-02-10T08:47:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.434+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7ba4003b20 con 0x7f7bb0103320 2024-02-10T08:47:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(710..710 src has 420..710) v4 ==== 8267+0+0 (unknown 2783849483 0 0) 0x7f7ba4079060 con 0x7f7bb0103320 2024-02-10T08:47:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 10 client.48943.objecter ms_dispatch 0x7f7bb0001550 osd_map(710..710 src has 420..710) v4 2024-02-10T08:47:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 3 client.48943.objecter handle_osd_map got epochs [710,710] > 0 2024-02-10T08:47:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 3 client.48943.objecter handle_osd_map decoding full epoch 710 2024-02-10T08:47:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 20 client.48943.objecter dump_active .. 0 homeless 2024-02-10T08:47:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.435+0000 7f7b9f7fe700 10 client.48943.objecter ms_handle_connect 0x7f7b98063980 2024-02-10T08:47:50.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.443+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7ba404c3e0 con 0x7f7bb0103320 2024-02-10T08:47:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:50.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:47:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:47:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:49.202124+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-10T08:47:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:49.549093+0000 mgr.y (mgr.14553) 593 : cluster [DBG] pgmap v1073: 194 pgs: 1 active+clean+snaptrim, 193 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T08:47:50.561 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:50.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:50.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:50.660+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f7b90005200 con 0x7f7bb0103320 2024-02-10T08:47:50.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:50.750 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:50.856 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:50.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:50.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:50.986 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:51.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:51.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.155+0000 7f7b9f7fe700 1 -- 172.21.15.107:0/741356096 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v711) v1 ==== 91+0+0 (unknown 1939067639 0 0) 0x7f7ba4003e50 con 0x7f7bb0103320 2024-02-10T08:47:51.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.158+0000 7f7bb4857700 10 librados: watch_flush enter 2024-02-10T08:47:51.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.161+0000 7f7bb4857700 10 librados: watch_flush exit 2024-02-10T08:47:51.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.161+0000 7f7bb4857700 20 client.48943.objecter shutdown clearing up homeless session... 2024-02-10T08:47:51.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.161+0000 7f7bb4857700 10 client.48943.objecter successfully canceled tick 2024-02-10T08:47:51.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.162+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7b98063980 legacy=0x7f7b98065e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:51.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.162+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 >> v1:172.21.15.195:6789/0 conn(0x7f7bb0103320 legacy=0x7f7bb011ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:51.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.162+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 shutdown_connections 2024-02-10T08:47:51.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.162+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 >> 172.21.15.107:0/741356096 conn(0x7f7bb006c090 msgr2=0x7f7bb006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:51.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.167+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 shutdown_connections 2024-02-10T08:47:51.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.167+0000 7f7bb4857700 1 -- 172.21.15.107:0/741356096 wait complete. 2024-02-10T08:47:51.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:51.175+0000 7f7bb4857700 1 librados: shutdown 2024-02-10T08:47:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:47:51.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:51.244 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:51.367 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:51.369 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47650.0:2 135.1f 135:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 582.33478843800003, 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394, 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47650", 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/1937376491", 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:47:51.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032659+0000", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032657+0000", 2024-02-10T08:47:51.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2010000000000002e-06 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032664+0000", 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1249999999999999e-06 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032673+0000", 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4060000000000004e-06 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032679+0000", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0767999999999999e-05 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032730+0000", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4550000000000001e-06 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.032739+0000", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23991668399999999 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272656+0000", 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124196 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272780+0000", 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152177 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.272932+0000", 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001504109 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.274436+0000", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001049901 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.275486+0000", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027108150000000001 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278197+0000", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3831e-05 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278251+0000", 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0742999999999999e-05 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:09.278292+0000", 2024-02-10T08:47:51.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.245632394 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47734.0:2 147.1a 147:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:47:51.377 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 561.41068580000001, 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332, 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47734", 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/29749952", 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956762+0000", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956760+0000", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4909999999999999e-06 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956768+0000", 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199999999999995e-06 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956776+0000", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0839999999999999e-06 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956781+0000", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7848000000000001e-05 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956819+0000", 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.19e-06 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:29.956825+0000", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090413761999999995 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047239+0000", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112608 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047352+0000", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014200000000000001 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-02-10T08:47:51.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.047494+0000", 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012910860000000001 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.048785+0000", 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063636400000000003 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.055149+0000", 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023764580000000001 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057525+0000", 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8439000000000001e-05 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057583+0000", 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1635999999999998e-05 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:30.057635+0000", 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100873332 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53393.0:1 195.1s0 195:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e330)", 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:47:51.384 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 447.34034997100002, 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441, 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53393", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4089636740", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027094+0000", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1878999999999999e-05 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027106+0000", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9205999999999999e-05 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027125+0000", 2024-02-10T08:47:51.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0699999999999999e-05 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027136+0000", 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017503530000000001 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028886+0000", 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7790000000000003e-06 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:47:51.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028891+0000", 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.008200303 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037091+0000", 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014759500000000001 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037239+0000", 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048123099999999999 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037720+0000", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015191219999999999 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.039239+0000", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001215224 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040455+0000", 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0549e-05 2024-02-10T08:47:51.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040505+0000", 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1660 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94547991866752 gen 10] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.182517405, 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001, 2024-02-10T08:47:51.390 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1660 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:47:51.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184927+0000", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.077e-05 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184938+0000", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3421000000000001e-05 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184951+0000", 2024-02-10T08:47:51.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6079999999999998e-06 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184959+0000", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1104000000000003e-05 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185020+0000", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0170000000000002e-06 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:47:51.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89132679699999995 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076356+0000", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013946 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076495+0000", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011721326000000001 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.088217+0000", 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001 2024-02-10T08:47:51.394 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1661 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 4] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.18246912199999, 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999, 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1661 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184981+0000", 2024-02-10T08:47:51.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1167e-05 2024-02-10T08:47:51.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:50.199617+0000 mon.a (mon.0) 3378 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:51.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:50.215347+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-10T08:47:51.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:50.660985+0000 mon.b (mon.1) 791 : audit [INF] from='client.? v1:172.21.15.107:0/741356096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:50.661322+0000 mon.a (mon.0) 3380 : audit [INF] from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.397 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:51.148443+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:51.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:51.153989+0000 mon.a (mon.0) 3382 : audit [INF] from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:51.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:51.154053+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-10T08:47:51.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:51.157924+0000 mon.b (mon.1) 792 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:51.160118+0000 mon.a (mon.0) 3384 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184993+0000", 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9099999999999996e-06 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184998+0000", 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2206999999999997e-05 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185050+0000", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1770000000000004e-06 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185059+0000", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88333866299999997 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068398+0000", 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172676 2024-02-10T08:47:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068570+0000", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044926389999999997 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.073063+0000", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:61 60.13 60:ca6ce4f9:test-rados-api-smithi107-112490-78::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:47:51.401 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 232.21873187, 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999, 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148715+0000", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1810000000000003e-06 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148719+0000", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9329999999999998e-06 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148725+0000", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1559999999999997e-06 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148729+0000", 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4011000000000006e-05 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148813+0000", 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0549999999999996e-06 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148819+0000", 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080372289999999999 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156857+0000", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8814999999999997e-05 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156935+0000", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017242200000000001 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157108+0000", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016798900000000001 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158788+0000", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072491250000000004 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166037+0000", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013801147 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179838+0000", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0494000000000001e-05 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179889+0000", 2024-02-10T08:47:51.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8433999999999998e-05 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179927+0000", 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:62 60.1f 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 232.21870745499999, 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006, 2024-02-10T08:47:51.407 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:47:51.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4249999999999998e-06 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148743+0000", 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2950000000000001e-06 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148748+0000", 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0949999999999998e-06 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148752+0000", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015047300000000001 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148903+0000", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.639e-06 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015726237000000001 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164634+0000", 2024-02-10T08:47:51.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1116e-05 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164725+0000", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016155299999999999 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164886+0000", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016114943 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.181001+0000", 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099756480000000002 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190977+0000", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033742560999999997 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224720+0000", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1842999999999998e-05 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224771+0000", 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8983999999999998e-05 2024-02-10T08:47:51.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224810+0000", 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:65 60.7 60:e0e5fb82:test-rados-api-smithi107-112490-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 232.21868187699999, 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999, 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.413 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148765+0000", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.311e-06 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148773+0000", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5339999999999999e-06 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148778+0000", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0620000000000002e-06 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148782+0000", 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014662 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148929+0000", 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8380000000000001e-06 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148934+0000", 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015063796000000001 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.163997+0000", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164103+0000", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017560199999999999 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164279+0000", 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016715187999999999 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180994+0000", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097823279999999999 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190776+0000", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8204999999999997e-05 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190825+0000", 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5817999999999997e-05 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190911+0000", 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:66 60.15 60:ab1b3caf:test-rados-api-smithi107-112490-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 232.21866133, 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002, 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.419 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2550000000000002e-06 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148789+0000", 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1379999999999998e-06 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148794+0000", 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8199999999999998e-06 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148798+0000", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109837 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5809999999999996e-06 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148913+0000", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013380609999999999 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162294+0000", 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012214800000000001 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162416+0000", 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020241599999999999 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162619+0000", 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004775708 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.167394+0000", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057395455999999997 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224790+0000", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4917999999999999e-05 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224835+0000", 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7835000000000002e-05 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224923+0000", 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:69 60.13 60:ce87a4eb:test-rados-api-smithi107-112490-78::5a02e6184a18760c5f8050d0e5b120c3d7a79519:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 232.21863537799999, 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999, 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6509999999999999e-06 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148815+0000", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5979999999999996e-06 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148821+0000", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5329999999999996e-06 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148825+0000", 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7842999999999997e-05 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148863+0000", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3610000000000003e-06 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148868+0000", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084767599999999999 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157344+0000", 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010973599999999999 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157454+0000", 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020398000000000001 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157658+0000", 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004485096 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162143+0000", 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017667062000000001 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179810+0000", 2024-02-10T08:47:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147684 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179958+0000", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6878000000000002e-05 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179995+0000", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9448e-05 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180024+0000", 2024-02-10T08:47:51.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(3.2 REQUEST e494)", 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 231.12267518799999, 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998, 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.517e-06 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244776+0000", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5489999999999999e-05 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244791+0000", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9300000000000003e-06 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244799+0000", 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025046212000000002 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269845+0000", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455999999999999e-05 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269866+0000", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3003999999999998e-05 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269899+0000", 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.435 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:935 241.1c 241:3fdb4e31:test-rados-api-smithi107-112490-115::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e668)", 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:03.270981+0000", 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 48.096466313999997, 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002786471, 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:47:51.436 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 935 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.270981+0000", 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.270981+0000", 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.270978+0000", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0208e-05 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.270989+0000", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2801e-05 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.271001+0000", 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5899999999999996e-06 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.271008+0000", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1788e-05 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.271050+0000", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017175400000000001 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.271222+0000", 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001678035 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.272900+0000", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6843999999999996e-05 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.272996+0000", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00058969700000000003 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.273586+0000", 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7793999999999995e-05 2024-02-10T08:47:51.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.273654+0000", 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000113767 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.273768+0000", 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002786471 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:46cb1ec1:reshard::reshard.0000000003:head)])", 2024-02-10T08:47:51.441 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 43.172283921000002, 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003791034, 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195163+0000", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1680000000000003e-06 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195168+0000", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4689999999999997e-06 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195176+0000", 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.229e-06 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195183+0000", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036324640000000002 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.198815+0000", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000139739 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.198955+0000", 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003791034 2024-02-10T08:47:51.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:4798f7e9:::obj_delete_at_hint.0000000007:head)])", 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 43.082028708000003, 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040317240000000004, 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:47:51.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285418+0000", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0409999999999998e-06 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285426+0000", 2024-02-10T08:47:51.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2284e-05 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285438+0000", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6699999999999997e-06 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285445+0000", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037752670000000001 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.289220+0000", 2024-02-10T08:47:51.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023066100000000001 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.289451+0000", 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040317240000000004 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:4578209b:::obj_delete_at_hint.0000000014:head)])", 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:47:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.991597089000003, 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038931109999999999, 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375850+0000", 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.994e-06 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375855+0000", 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1009999999999998e-06 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375864+0000", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5399999999999997e-06 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375868+0000", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037501779999999998 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.379618+0000", 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000125291 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.379744+0000", 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038931109999999999 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:479ba34f:::obj_delete_at_hint.0000000021:head)])", 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.901119633, 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038335589999999998, 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.452 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466327+0000", 2024-02-10T08:47:51.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7039999999999999e-06 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466332+0000", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5619999999999993e-06 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466342+0000", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2559999999999996e-06 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466349+0000", 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003671058 2024-02-10T08:47:51.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.470020+0000", 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141342 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.470162+0000", 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038335589999999998 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.455 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:47403372:::obj_delete_at_hint.0000000112:head)])", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.810755810000003, 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038613470000000002, 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556691+0000", 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6399999999999999e-06 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556695+0000", 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7899999999999996e-06 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556701+0000", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5199999999999997e-06 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556706+0000", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037127570000000001 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.560419+0000", 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134046 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.560553+0000", 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038613470000000002 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1428 244.1c 244:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e693)", 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.209081347000001, 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046145960000000003, 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:47:51.459 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1428 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:47:51.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158363+0000", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4042e-05 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158377+0000", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3719e-05 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158391+0000", 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2459999999999997e-06 2024-02-10T08:47:51.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158398+0000", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4943e-05 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158443+0000", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000182914 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158626+0000", 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034397 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162066+0000", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9436000000000004e-05 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162135+0000", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010961 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162245+0000", 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043155400000000001 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162676+0000", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203284 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162879+0000", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8809000000000001e-05 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162908+0000", 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3383000000000002e-05 2024-02-10T08:47:51.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162942+0000", 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9164e-05 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162981+0000", 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046145960000000003 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1473 244.1c 244:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e693)", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.178919241999999, 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029607119999999999, 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1473 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188527+0000", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7519999999999998e-06 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188531+0000", 2024-02-10T08:47:51.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.498e-06 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188539+0000", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8809999999999998e-06 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188542+0000", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8841999999999999e-05 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188571+0000", 2024-02-10T08:47:51.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9135999999999997e-05 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188640+0000", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026846190000000001 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191325+0000", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010083 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191426+0000", 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3165999999999998e-05 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191489+0000", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029607119999999999 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1516 244.1c 244:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e693)", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.206141+0000", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.161306889999999, 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025862709999999998, 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:47:51.470 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1516 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206141+0000", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206141+0000", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:47:51.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206139+0000", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7170000000000002e-06 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206146+0000", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0710000000000001e-05 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206156+0000", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7100000000000004e-06 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:47:51.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206162+0000", 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6665999999999997e-05 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206209+0000", 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016041199999999999 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.206369+0000", 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001771494 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.208141+0000", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5993000000000003e-05 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.208207+0000", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029114499999999998 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.208498+0000", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8276999999999995e-05 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:47:51.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.208586+0000", 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014088600000000001 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.208727+0000", 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025862709999999998 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:47:51.475 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:47:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:51.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:51.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:47:51.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:50.199617+0000 mon.a (mon.0) 3378 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:51.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:50.215347+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-10T08:47:51.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:50.660985+0000 mon.b (mon.1) 791 : audit [INF] from='client.? v1:172.21.15.107:0/741356096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:50.661322+0000 mon.a (mon.0) 3380 : audit [INF] from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:51.148443+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:51.153989+0000 mon.a (mon.0) 3382 : audit [INF] from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:51.154053+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:51.157924+0000 mon.b (mon.1) 792 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:51.160118+0000 mon.a (mon.0) 3384 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:50.199617+0000 mon.a (mon.0) 3378 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:50.215347+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:50.660985+0000 mon.b (mon.1) 791 : audit [INF] from='client.? v1:172.21.15.107:0/741356096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:50.661322+0000 mon.a (mon.0) 3380 : audit [INF] from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:47:51.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:51.148443+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:47:51.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:51.153989+0000 mon.a (mon.0) 3382 : audit [INF] from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:47:51.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:51.154053+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-10T08:47:51.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:51.157924+0000 mon.b (mon.1) 792 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:51.160118+0000 mon.a (mon.0) 3384 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:51.604 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:51.946 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:47:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:47:51.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:47:52.241 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:52.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:51.549610+0000 mgr.y (mgr.14553) 594 : cluster [DBG] pgmap v1076: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 24 unknown, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 0 B/s, 12 objects/s recovering 2024-02-10T08:47:52.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:51.754341+0000 mgr.y (mgr.14553) 595 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:52.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:52.155487+0000 mon.a (mon.0) 3385 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:52.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:52.155594+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-10T08:47:52.514 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:52.165860+0000 mon.b (mon.1) 793 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.515 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:52.169144+0000 mon.a (mon.0) 3387 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:51.549610+0000 mgr.y (mgr.14553) 594 : cluster [DBG] pgmap v1076: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 24 unknown, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 0 B/s, 12 objects/s recovering 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:51.754341+0000 mgr.y (mgr.14553) 595 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:52.155487+0000 mon.a (mon.0) 3385 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:52.155594+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:52.165860+0000 mon.b (mon.1) 793 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:52.169144+0000 mon.a (mon.0) 3387 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:51.549610+0000 mgr.y (mgr.14553) 594 : cluster [DBG] pgmap v1076: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 24 unknown, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 0 B/s, 12 objects/s recovering 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:51.754341+0000 mgr.y (mgr.14553) 595 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:52.155487+0000 mon.a (mon.0) 3385 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:52.155594+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:52.165860+0000 mon.b (mon.1) 793 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:52.169144+0000 mon.a (mon.0) 3387 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:47:52.934 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:47:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:52.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:47:53.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:53.065 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- 172.21.15.107:0/1298274434 >> v1:172.21.15.107:6790/0 conn(0x7ff958104350 legacy=0x7ff9581067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- 172.21.15.107:0/1298274434 shutdown_connections 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- 172.21.15.107:0/1298274434 >> 172.21.15.107:0/1298274434 conn(0x7ff9580f2510 msgr2=0x7ff9580f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- 172.21.15.107:0/1298274434 shutdown_connections 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- 172.21.15.107:0/1298274434 wait complete. 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 librados: starting msgr at 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 librados: starting objecter 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 Processor -- start 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 -- start start 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 librados: setting wanted keys 2024-02-10T08:47:53.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.183+0000 7ff95cd5b700 1 librados: calling monclient init 2024-02-10T08:47:53.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff95cd5b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff958117440 con 0x7ff958104350 2024-02-10T08:47:53.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff95cd5b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff958118620 con 0x7ff9580f9570 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff95cd5b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff958119820 con 0x7ff9580f89c0 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff95659c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff9580f89c0 0x7ff9580ff740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37178/0 (socket says 172.21.15.107:37178) 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff95659c700 1 -- 172.21.15.107:0/3101162206 learned_addr learned my addr 172.21.15.107:0/3101162206 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff956d9d700 1 --1- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6789/0 conn(0x7ff958104350 0x7ff958115b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57448/0 (socket says 172.21.15.107:57448) 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter ms_handle_connect 0x7ff9580f89c0 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter ms_handle_connect 0x7ff958104350 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter ms_handle_connect 0x7ff9580f9570 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860954885 0 0) 0x7ff958119820 con 0x7ff9580f89c0 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9400033f0 con 0x7ff9580f89c0 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123406206 0 0) 0x7ff958118620 con 0x7ff9580f9570 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff958119820 con 0x7ff9580f9570 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820728934 0 0) 0x7ff958117440 con 0x7ff958104350 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff958118620 con 0x7ff958104350 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172815855 0 0) 0x7ff958119820 con 0x7ff9580f9570 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff958117440 con 0x7ff9580f9570 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189723660 0 0) 0x7ff9400033f0 con 0x7ff9580f89c0 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff958119820 con 0x7ff9580f89c0 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff94c0018c0 con 0x7ff9580f9570 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 677163969 0 0) 0x7ff958118620 con 0x7ff958104350 2024-02-10T08:47:53.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9400033f0 con 0x7ff958104350 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff948001960 con 0x7ff9580f89c0 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff9500013f0 con 0x7ff958104350 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1080515833 0 0) 0x7ff958117440 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6790/0 conn(0x7ff9580f89c0 legacy=0x7ff9580ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.184+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6789/0 conn(0x7ff958104350 legacy=0x7ff958115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95811aa20 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958117630 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff95cd5b700 10 client.48949.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958117aa0 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff95cd5b700 1 librados: init done 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff94c002000 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff94c003270 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.185+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff938004f00 con 0x7ff9580f9570 2024-02-10T08:47:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.186+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff94c001b60 con 0x7ff9580f9570 2024-02-10T08:47:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(713..713 src has 420..713) v4 ==== 8649+0+0 (unknown 3500081781 0 0) 0x7ff94c076610 con 0x7ff9580f9570 2024-02-10T08:47:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 osd_map(713..713 src has 420..713) v4 2024-02-10T08:47:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 3 client.48949.objecter handle_osd_map got epochs [713,713] > 0 2024-02-10T08:47:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 3 client.48949.objecter handle_osd_map decoding full epoch 713 2024-02-10T08:47:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 20 client.48949.objecter dump_active .. 0 homeless 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v713) v1 ==== 54+0+16 (unknown 1740207373 0 2420000192) 0x7ff94c078a00 con 0x7ff9580f9570 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff940063d10 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 1 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 1 2024-02-10T08:47:53.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 0 1 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 4 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 10 client.48949.objecter _send_command 1 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff938004950 con 0x7ff938004530 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.187+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff938004530 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.188+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff938004950 con 0x7ff938004530 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.188+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.188+0000 7ff9477fe700 10 client.48949.objecter _finish_command 1 = system:0 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.188+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 4 2024-02-10T08:47:53.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.188+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 0 1 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.202+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 2 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 2 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff9380051a0 osd=0 3 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 0 2 2024-02-10T08:47:53.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 4 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 10 client.48949.objecter _send_command 2 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.203+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff938004d30 con 0x7ff938004530 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff938004d30 con 0x7ff938004530 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 10 client.48949.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 4 2024-02-10T08:47:53.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 0 2 2024-02-10T08:47:53.205 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 1 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 4 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 10 client.48949.objecter _send_command 3 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff93800c000 con 0x7ff938004150 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.204+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff938004150 2024-02-10T08:47:53.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.205+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff93800c000 con 0x7ff938004150 2024-02-10T08:47:53.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.205+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:47:53.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.205+0000 7ff9477fe700 10 client.48949.objecter _finish_command 3 = system:0 2024-02-10T08:47:53.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.205+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 4 2024-02-10T08:47:53.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.205+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 1 3 2024-02-10T08:47:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 4 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 4 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938003fe0 osd=1 3 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 1 4 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 4 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 10 client.48949.objecter _send_command 4 2024-02-10T08:47:53.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.220+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9380018b0 con 0x7ff938004150 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.222+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff9380018b0 con 0x7ff938004150 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.222+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.222+0000 7ff9477fe700 10 client.48949.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.222+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 4 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.222+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 1 4 2024-02-10T08:47:53.223 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 5 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 5 2024-02-10T08:47:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 2 5 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 4 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 10 client.48949.objecter _send_command 5 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff93800de90 con 0x7ff93800d2f0 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff93800d2f0 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff93800de90 con 0x7ff93800d2f0 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 10 client.48949.objecter _finish_command 5 = system:0 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 4 2024-02-10T08:47:53.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.223+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 2 5 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 6 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 6 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938003630 osd=2 3 2024-02-10T08:47:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 2 6 2024-02-10T08:47:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 4 2024-02-10T08:47:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 10 client.48949.objecter _send_command 6 2024-02-10T08:47:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.238+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff9380030a0 con 0x7ff93800d2f0 2024-02-10T08:47:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff9380030a0 con 0x7ff93800d2f0 2024-02-10T08:47:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff9477fe700 10 client.48949.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 4 2024-02-10T08:47:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 2 6 2024-02-10T08:47:53.240 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 7 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 7 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 3 7 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 4 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 10 client.48949.objecter _send_command 7 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.239+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff938015b00 con 0x7ff938011ff0 2024-02-10T08:47:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff938011ff0 2024-02-10T08:47:53.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff938015b00 con 0x7ff938011ff0 2024-02-10T08:47:53.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:47:53.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 10 client.48949.objecter _finish_command 7 = system:0 2024-02-10T08:47:53.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 4 2024-02-10T08:47:53.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.240+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 3 7 2024-02-10T08:47:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 8 2024-02-10T08:47:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 8 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938011970 osd=3 3 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 3 8 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 4 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 10 client.48949.objecter _send_command 8 2024-02-10T08:47:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.255+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff93800e2a0 con 0x7ff938011ff0 2024-02-10T08:47:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff93800e2a0 con 0x7ff938011ff0 2024-02-10T08:47:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff9477fe700 10 client.48949.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 4 2024-02-10T08:47:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 3 8 2024-02-10T08:47:53.258 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 9 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 9 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 4 9 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 4 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 10 client.48949.objecter _send_command 9 2024-02-10T08:47:53.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.257+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff93801a520 con 0x7ff93800ce50 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.258+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff93800ce50 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.259+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff93801a520 con 0x7ff93800ce50 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.259+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.259+0000 7ff9477fe700 10 client.48949.objecter _finish_command 9 = system:0 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.259+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 4 2024-02-10T08:47:53.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.259+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 4 9 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 10 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 10 2024-02-10T08:47:53.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff93800e060 osd=4 3 2024-02-10T08:47:53.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 4 10 2024-02-10T08:47:53.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 4 2024-02-10T08:47:53.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 10 client.48949.objecter _send_command 10 2024-02-10T08:47:53.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.276+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff93801a520 con 0x7ff93800ce50 2024-02-10T08:47:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff93801a520 con 0x7ff93800ce50 2024-02-10T08:47:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 10 client.48949.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 4 2024-02-10T08:47:53.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 4 10 2024-02-10T08:47:53.279 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:47:53.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 11 2024-02-10T08:47:53.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 11 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 5 11 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 4 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 10 client.48949.objecter _send_command 11 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff93801e8b0 con 0x7ff93801ac30 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.278+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff93801ac30 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.279+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff93801e8b0 con 0x7ff93801ac30 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.279+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.279+0000 7ff9477fe700 10 client.48949.objecter _finish_command 11 = system:0 2024-02-10T08:47:53.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.279+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 4 2024-02-10T08:47:53.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.279+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 5 11 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 12 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 12 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff93801a520 osd=5 3 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 5 12 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 4 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 10 client.48949.objecter _send_command 12 2024-02-10T08:47:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.294+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff938020060 con 0x7ff93801ac30 2024-02-10T08:47:53.295 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:47:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff938020060 con 0x7ff93801ac30 2024-02-10T08:47:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff9477fe700 10 client.48949.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 4 2024-02-10T08:47:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 5 12 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 13 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 13 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 6 13 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 4 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 10 client.48949.objecter _send_command 13 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.295+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff938020cb0 con 0x7ff938020750 2024-02-10T08:47:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff938020750 2024-02-10T08:47:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff938020cb0 con 0x7ff938020750 2024-02-10T08:47:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:47:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 10 client.48949.objecter _finish_command 13 = system:0 2024-02-10T08:47:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 4 2024-02-10T08:47:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.296+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 6 13 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 14 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 14 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938020060 osd=6 3 2024-02-10T08:47:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 6 14 2024-02-10T08:47:53.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 4 2024-02-10T08:47:53.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 10 client.48949.objecter _send_command 14 2024-02-10T08:47:53.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.311+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff938020cb0 con 0x7ff938020750 2024-02-10T08:47:53.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff938020cb0 con 0x7ff938020750 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff9477fe700 10 client.48949.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 4 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 6 14 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:47:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 15 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 15 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 7 15 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 4 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 10 client.48949.objecter _send_command 15 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.312+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff938027260 con 0x7ff938023750 2024-02-10T08:47:53.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.313+0000 7ff9477fe700 10 client.48949.objecter ms_handle_connect 0x7ff938023750 2024-02-10T08:47:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.314+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff938027260 con 0x7ff938023750 2024-02-10T08:47:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.314+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:47:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.314+0000 7ff9477fe700 10 client.48949.objecter _finish_command 15 = system:0 2024-02-10T08:47:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.314+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 4 2024-02-10T08:47:53.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.314+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 7 15 2024-02-10T08:47:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 10 client.48949.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:47:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign -1 16 2024-02-10T08:47:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 20 client.48949.objecter _get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_remove -1 16 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 20 client.48949.objecter get_session s=0x7ff938020cb0 osd=7 3 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 15 client.48949.objecter _session_command_op_assign 7 16 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 4 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 10 client.48949.objecter _send_command 16 2024-02-10T08:47:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.328+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff938027260 con 0x7ff938023750 2024-02-10T08:47:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff9477fe700 1 -- 172.21.15.107:0/3101162206 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ff938027260 con 0x7ff938023750 2024-02-10T08:47:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff9477fe700 10 client.48949.objecter ms_dispatch 0x7ff958001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:47:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff9477fe700 10 client.48949.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff9477fe700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 4 2024-02-10T08:47:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff9477fe700 15 client.48949.objecter _session_command_op_remove 7 16 2024-02-10T08:47:53.331 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 librados: watch_flush enter 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 librados: watch_flush exit 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.0 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6801/3321596972 conn(0x7ff938004530 legacy=0x7ff938002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff9380051a0 osd=0 1 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.1 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6805/133516126 conn(0x7ff938004150 legacy=0x7ff93800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003fe0 osd=1 1 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.2 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6809/3575881728 conn(0x7ff93800d2f0 legacy=0x7ff93800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938003630 osd=2 1 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.3 2024-02-10T08:47:53.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6813/2871825691 conn(0x7ff938011ff0 legacy=0x7ff9380143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938011970 osd=3 1 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.4 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.195:6800/3310789251 conn(0x7ff93800ce50 legacy=0x7ff938018e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff956d9d700 1 -- 172.21.15.107:0/3101162206 reap_dead start 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff956d9d700 1 -- 172.21.15.107:0/3101162206 reap_dead start 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93800e060 osd=4 1 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.5 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.195:6804/4178321486 conn(0x7ff93801ac30 legacy=0x7ff93801d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff93801a520 osd=5 1 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.6 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.195:6808/1417479017 conn(0x7ff938020750 legacy=0x7ff93801e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020060 osd=6 1 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter close_session for osd.7 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.195:6812/1934176928 conn(0x7ff938023750 legacy=0x7ff938025b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter put_session s=0x7ff938020cb0 osd=7 1 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 20 client.48949.objecter shutdown clearing up homeless session... 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 10 client.48949.objecter successfully canceled tick 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff940063d10 legacy=0x7ff9400661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> v1:172.21.15.195:6789/0 conn(0x7ff9580f9570 legacy=0x7ff9580ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff956d9d700 1 -- 172.21.15.107:0/3101162206 reap_dead start 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff956d9d700 1 -- 172.21.15.107:0/3101162206 reap_dead start 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 shutdown_connections 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.330+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 >> 172.21.15.107:0/3101162206 conn(0x7ff9580f2510 msgr2=0x7ff9581077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.331+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 shutdown_connections 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.331+0000 7ff95cd5b700 1 -- 172.21.15.107:0/3101162206 wait complete. 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:53.331+0000 7ff95cd5b700 1 librados: shutdown 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.334 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.335 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.335 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.335 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:52.260170+0000 mon.a (mon.0) 3388 : cluster [WRN] Health check failed: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:53.155511+0000 mon.a (mon.0) 3389 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:53.155569+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:53.159270+0000 mon.b (mon.1) 794 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:53.159633+0000 mon.a (mon.0) 3391 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:53.187600+0000 mon.b (mon.1) 795 : audit [DBG] from='client.? v1:172.21.15.107:0/3101162206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:53.414 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:53.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:53.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:53.543 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:53.642 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:53.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:52.260170+0000 mon.a (mon.0) 3388 : cluster [WRN] Health check failed: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:53.155511+0000 mon.a (mon.0) 3389 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:53.155569+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:53.159270+0000 mon.b (mon.1) 794 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:53.159633+0000 mon.a (mon.0) 3391 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:53.187600+0000 mon.b (mon.1) 795 : audit [DBG] from='client.? v1:172.21.15.107:0/3101162206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:52.260170+0000 mon.a (mon.0) 3388 : cluster [WRN] Health check failed: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:53.155511+0000 mon.a (mon.0) 3389 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:53.155569+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:53.159270+0000 mon.b (mon.1) 794 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:53.159633+0000 mon.a (mon.0) 3391 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:53.187600+0000 mon.b (mon.1) 795 : audit [DBG] from='client.? v1:172.21.15.107:0/3101162206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:47:53.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:53.775 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:53.905 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:53.905 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:54.024 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:54.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:54.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:47:54.159 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:54.269 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 0, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:47:54.269 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:47:54.270 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-10T08:47:54.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:47:54.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:53.550025+0000 mgr.y (mgr.14553) 596 : cluster [DBG] pgmap v1079: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 153 B/s, 20 objects/s recovering 2024-02-10T08:47:54.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:54.159427+0000 mon.a (mon.0) 3392 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:54.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:54.159472+0000 mon.a (mon.0) 3393 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-10T08:47:54.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:54.160267+0000 mon.b (mon.1) 796 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:54.160724+0000 mon.a (mon.0) 3394 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.508 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:47:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:54.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:47:54.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:47:54.536 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:54.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:47:54.641 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:54.660 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:47:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:54.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:47:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:53.550025+0000 mgr.y (mgr.14553) 596 : cluster [DBG] pgmap v1079: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 153 B/s, 20 objects/s recovering 2024-02-10T08:47:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:54.159427+0000 mon.a (mon.0) 3392 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:54.159472+0000 mon.a (mon.0) 3393 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-10T08:47:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:54.160267+0000 mon.b (mon.1) 796 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:54.160724+0000 mon.a (mon.0) 3394 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:53.550025+0000 mgr.y (mgr.14553) 596 : cluster [DBG] pgmap v1079: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 153 B/s, 20 objects/s recovering 2024-02-10T08:47:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:54.159427+0000 mon.a (mon.0) 3392 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:54.159472+0000 mon.a (mon.0) 3393 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-10T08:47:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:54.160267+0000 mon.b (mon.1) 796 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:54.160724+0000 mon.a (mon.0) 3394 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]: dispatch 2024-02-10T08:47:54.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:47:54.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:54.794 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:54.813 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:54.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:54.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:54.942 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:55.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:55.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:55.176 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:55.251 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:55.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:55.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:55.379 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.155037+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:55.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:55.159674+0000 mon.a (mon.0) 3396 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]': finished 2024-02-10T08:47:55.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.159743+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-10T08:47:55.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:55.171196+0000 mon.a (mon.0) 3398 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:55.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:55.195804+0000 mon.b (mon.1) 797 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:55.196104+0000 mon.a (mon.0) 3399 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.382 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:55.414 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.155037+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:55.159674+0000 mon.a (mon.0) 3396 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]': finished 2024-02-10T08:47:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.159743+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-10T08:47:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:55.171196+0000 mon.a (mon.0) 3398 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:55.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:55.195804+0000 mon.b (mon.1) 797 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:55.196104+0000 mon.a (mon.0) 3399 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.155037+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:55.159674+0000 mon.a (mon.0) 3396 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-124", "mode": "writeback"}]': finished 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.159743+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:55.171196+0000 mon.a (mon.0) 3398 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:55.195804+0000 mon.b (mon.1) 797 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:55.196104+0000 mon.a (mon.0) 3399 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:47:55.579 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:55.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:55.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:47:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:47:55] "GET /metrics HTTP/1.1" 200 190578 "" "Prometheus/2.33.4" 2024-02-10T08:47:55.960 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:55.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:56.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:47:56.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:56.090 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:56.189 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:56.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:56.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:56.323 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:56.345 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:56.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:56.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:56.473 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:56.474 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:56.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:54.965821+0000 osd.5 (osd.5) 292 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:47:56.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:54.969033+0000 osd.5 (osd.5) 293 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:47:56.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.028409+0000 osd.4 (osd.4) 43 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:47:56.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.031298+0000 osd.4 (osd.4) 44 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.550671+0000 mgr.y (mgr.14553) 597 : cluster [DBG] pgmap v1082: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 192 B/s, 7 objects/s recovering 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.934026+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.934784+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.150131+0000 mon.a (mon.0) 3400 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:56.154779+0000 mon.a (mon.0) 3401 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.154851+0000 mon.a (mon.0) 3402 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:56.155859+0000 mon.b (mon.1) 798 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.475 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:56.156491+0000 mon.a (mon.0) 3403 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.476 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:54.965821+0000 osd.5 (osd.5) 292 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:54.969033+0000 osd.5 (osd.5) 293 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.028409+0000 osd.4 (osd.4) 43 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.031298+0000 osd.4 (osd.4) 44 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.550671+0000 mgr.y (mgr.14553) 597 : cluster [DBG] pgmap v1082: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 192 B/s, 7 objects/s recovering 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.934026+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.934784+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.150131+0000 mon.a (mon.0) 3400 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:56.154779+0000 mon.a (mon.0) 3401 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.154851+0000 mon.a (mon.0) 3402 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:56.155859+0000 mon.b (mon.1) 798 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:56.156491+0000 mon.a (mon.0) 3403 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:54.965821+0000 osd.5 (osd.5) 292 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:54.969033+0000 osd.5 (osd.5) 293 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.028409+0000 osd.4 (osd.4) 43 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.031298+0000 osd.4 (osd.4) 44 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.550671+0000 mgr.y (mgr.14553) 597 : cluster [DBG] pgmap v1082: 194 pgs: 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 19 unknown, 160 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14/624 objects degraded (2.244%); 9/624 objects misplaced (1.442%); 192 B/s, 7 objects/s recovering 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.934026+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.934784+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.150131+0000 mon.a (mon.0) 3400 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:56.154779+0000 mon.a (mon.0) 3401 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.154851+0000 mon.a (mon.0) 3402 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-10T08:47:56.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:56.155859+0000 mon.b (mon.1) 798 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:56.156491+0000 mon.a (mon.0) 3403 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]: dispatch 2024-02-10T08:47:56.496 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:56.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:56.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:47:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:57.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:47:57.384 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:47:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.563363+0000 osd.7 (osd.7) 286 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:47:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:55.566273+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:47:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.152382+0000 mon.a (mon.0) 3404 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:57.155887+0000 mon.a (mon.0) 3405 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:57.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.155954+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-10T08:47:57.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.563363+0000 osd.7 (osd.7) 286 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:47:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:55.566273+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:47:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.152382+0000 mon.a (mon.0) 3404 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:57.155887+0000 mon.a (mon.0) 3405 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:57.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.155954+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-10T08:47:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.563363+0000 osd.7 (osd.7) 286 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:47:57.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:55.566273+0000 osd.7 (osd.7) 287 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:47:57.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.152382+0000 mon.a (mon.0) 3404 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:47:57.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:57.155887+0000 mon.a (mon.0) 3405 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-124"}]': finished 2024-02-10T08:47:57.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.155954+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-10T08:47:58.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.325+0000 7f93645de700 1 -- 172.21.15.107:0/3446733896 >> v1:172.21.15.107:6790/0 conn(0x7f935c0f9570 legacy=0x7f935c102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- 172.21.15.107:0/3446733896 shutdown_connections 2024-02-10T08:47:58.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- 172.21.15.107:0/3446733896 >> 172.21.15.107:0/3446733896 conn(0x7f935c0f2510 msgr2=0x7f935c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:58.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- 172.21.15.107:0/3446733896 shutdown_connections 2024-02-10T08:47:58.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- 172.21.15.107:0/3446733896 wait complete. 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 librados: starting msgr at 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 librados: starting objecter 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 Processor -- start 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- start start 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 librados: setting wanted keys 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 librados: calling monclient init 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f935c117800 con 0x7f935c104350 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.326+0000 7f93645de700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f935c1189a0 con 0x7f935c0f9570 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93645de700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f935c119b40 con 0x7f935c0f89c0 2024-02-10T08:47:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f936237a700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f935c0f89c0 0x7f935c102310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37222/0 (socket says 172.21.15.107:37222) 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f9362b7b700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f935c104350 0x7f935c1160e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57484/0 (socket says 172.21.15.107:57484) 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f936237a700 1 -- 172.21.15.107:0/4263186383 learned_addr learned my addr 172.21.15.107:0/4263186383 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter ms_handle_connect 0x7f935c104350 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter ms_handle_connect 0x7f935c0f9570 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter ms_handle_connect 0x7f935c0f89c0 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787785124 0 0) 0x7f935c1189a0 con 0x7f935c0f9570 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9348003630 con 0x7f935c0f9570 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800287171 0 0) 0x7f935c119b40 con 0x7f935c0f89c0 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c1189a0 con 0x7f935c0f89c0 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71021305 0 0) 0x7f9348003630 con 0x7f935c0f9570 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f935c119b40 con 0x7f935c0f9570 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f934c002d60 con 0x7f935c0f9570 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732773838 0 0) 0x7f935c1189a0 con 0x7f935c0f89c0 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9348003630 con 0x7f935c0f89c0 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9358001a10 con 0x7f935c0f89c0 2024-02-10T08:47:58.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1584135025 0 0) 0x7f935c119b40 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 >> v1:172.21.15.107:6790/0 conn(0x7f935c0f89c0 legacy=0x7f935c102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 >> v1:172.21.15.107:6789/0 conn(0x7f935c104350 legacy=0x7f935c1160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c11ace0 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c118b90 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93645de700 10 client.48961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c119000 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93645de700 1 librados: init done 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f934c001230 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.327+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f934c0035d0 con 0x7f935c0f9570 2024-02-10T08:47:58.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.328+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340004f20 con 0x7f935c0f9570 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f934c0020c0 con 0x7f935c0f9570 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(718..718 src has 420..718) v4 ==== 8239+0+0 (unknown 622768124 0 0) 0x7f934c0767c0 con 0x7f935c0f9570 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 10 client.48961.objecter ms_dispatch 0x7f935c001550 osd_map(718..718 src has 420..718) v4 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 3 client.48961.objecter handle_osd_map got epochs [718,718] > 0 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 3 client.48961.objecter handle_osd_map decoding full epoch 718 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 20 client.48961.objecter dump_active .. 0 homeless 2024-02-10T08:47:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.329+0000 7f93537fe700 10 client.48961.objecter ms_handle_connect 0x7f9348063b70 2024-02-10T08:47:58.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.333+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f934c049bc0 con 0x7f935c0f9570 2024-02-10T08:47:58.380 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:47:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:58.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:47:58.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- 172.21.15.107:0/2327342629 >> v1:172.21.15.107:6790/0 conn(0x7f917c0fc910 legacy=0x7f917c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- 172.21.15.107:0/2327342629 shutdown_connections 2024-02-10T08:47:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- 172.21.15.107:0/2327342629 >> 172.21.15.107:0/2327342629 conn(0x7f917c0f2510 msgr2=0x7f917c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- 172.21.15.107:0/2327342629 shutdown_connections 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- 172.21.15.107:0/2327342629 wait complete. 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 librados: starting msgr at 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 librados: starting objecter 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 Processor -- start 2024-02-10T08:47:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 -- start start 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 librados: setting wanted keys 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.845+0000 7f9181fa6700 1 librados: calling monclient init 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9181fa6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f917c1195c0 con 0x7f917c0ff710 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9181fa6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f917c11a7a0 con 0x7f917c0fc910 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9181fa6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f917c11b9a0 con 0x7f917c103250 2024-02-10T08:47:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9172ffd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f917c0ff710 0x7f917c0f8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57474/0 (socket says 172.21.15.107:57474) 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f917bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f917c103250 0x7f917c117cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37200/0 (socket says 172.21.15.107:37200) 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9172ffd700 1 -- 172.21.15.107:0/1095536105 learned_addr learned my addr 172.21.15.107:0/1095536105 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter ms_handle_connect 0x7f917c103250 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter ms_handle_connect 0x7f917c0ff710 2024-02-10T08:47:58.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter ms_handle_connect 0x7f917c0fc910 2024-02-10T08:47:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:47:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225460223 0 0) 0x7f917c11b9a0 con 0x7f917c103250 2024-02-10T08:47:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91680033f0 con 0x7f917c103250 2024-02-10T08:47:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901654804 0 0) 0x7f917c1195c0 con 0x7f917c0ff710 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917c11b9a0 con 0x7f917c0ff710 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600274667 0 0) 0x7f917c11a7a0 con 0x7f917c0fc910 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917c1195c0 con 0x7f917c0fc910 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2702008842 0 0) 0x7f91680033f0 con 0x7f917c103250 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f917c11a7a0 con 0x7f917c103250 2024-02-10T08:47:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9174001c40 con 0x7f917c103250 2024-02-10T08:47:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588681223 0 0) 0x7f917c1195c0 con 0x7f917c0fc910 2024-02-10T08:47:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91680033f0 con 0x7f917c0fc910 2024-02-10T08:47:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338098706 0 0) 0x7f917c11b9a0 con 0x7f917c0ff710 2024-02-10T08:47:58.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f917c1195c0 con 0x7f917c0ff710 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f916c001430 con 0x7f917c0fc910 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f91640019d0 con 0x7f917c0ff710 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1812969510 0 0) 0x7f917c11a7a0 con 0x7f917c103250 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 >> v1:172.21.15.195:6789/0 conn(0x7f917c0fc910 legacy=0x7f917c0f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 >> v1:172.21.15.107:6789/0 conn(0x7f917c0ff710 legacy=0x7f917c0f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c11cba0 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.846+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917c1197b0 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.847+0000 7f9181fa6700 10 client.54233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.847+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c119c20 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.847+0000 7f9181fa6700 1 librados: init done 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.847+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9174002240 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.847+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9174005df0 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c004f20 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9174001da0 con 0x7f917c103250 2024-02-10T08:47:58.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(716..716 src has 420..716) v4 ==== 8649+0+0 (unknown 4200736945 0 0) 0x7f9174079100 con 0x7f917c103250 2024-02-10T08:47:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 10 client.54233.objecter ms_dispatch 0x7f917c001550 osd_map(716..716 src has 420..716) v4 2024-02-10T08:47:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 3 client.54233.objecter handle_osd_map got epochs [716,716] > 0 2024-02-10T08:47:58.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 3 client.54233.objecter handle_osd_map decoding full epoch 716 2024-02-10T08:47:58.497 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:47:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 20 client.54233.objecter dump_active .. 0 homeless 2024-02-10T08:47:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.848+0000 7f91797fa700 10 client.54233.objecter ms_handle_connect 0x7f9168063d70 2024-02-10T08:47:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:56.852+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f917404c2c0 con 0x7f917c103250 2024-02-10T08:47:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:57.019+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f915c000b90 con 0x7f9168063d70 2024-02-10T08:47:58.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:57.033+0000 7f91797fa700 1 -- 172.21.15.107:0/1095536105 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+505051 (unknown 2965378022 0 2089187348) 0x7f915c000b90 con 0x7f9168063d70 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.597230+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.600501+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.904014+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.907091+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.918368+0000 osd.1 (osd.1) 314 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:56.921282+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:57.020324+0000 mgr.y (mgr.14553) 598 : audit [DBG] from='client.54233 v1:172.21.15.107:0/1095536105' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:58.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.551225+0000 mgr.y (mgr.14553) 599 : cluster [DBG] pgmap v1085: 194 pgs: 194 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:47:58.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.558528+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:47:58.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.560580+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:47:58.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.157516+0000 mon.a (mon.0) 3407 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 10 librados: watch_flush enter 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 10 librados: watch_flush exit 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 20 client.54233.objecter shutdown clearing up homeless session... 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 10 client.54233.objecter successfully canceled tick 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9168063d70 legacy=0x7f9168066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 >> v1:172.21.15.107:6790/0 conn(0x7f917c103250 legacy=0x7f917c117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 shutdown_connections 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 >> 172.21.15.107:0/1095536105 conn(0x7f917c0f2510 msgr2=0x7f917c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 shutdown_connections 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 -- 172.21.15.107:0/1095536105 wait complete. 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.488+0000 7f9181fa6700 1 librados: shutdown 2024-02-10T08:47:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:47:58.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:58.520+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f93400051d0 con 0x7f935c0f9570 2024-02-10T08:47:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:47:58.536 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.597230+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:47:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.600501+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:47:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.904014+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:47:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.907091+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.918368+0000 osd.1 (osd.1) 314 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:56.921282+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:57.020324+0000 mgr.y (mgr.14553) 598 : audit [DBG] from='client.54233 v1:172.21.15.107:0/1095536105' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.551225+0000 mgr.y (mgr.14553) 599 : cluster [DBG] pgmap v1085: 194 pgs: 194 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.558528+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.560580+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.157516+0000 mon.a (mon.0) 3407 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.597230+0000 osd.7 (osd.7) 288 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.600501+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.904014+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.907091+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:47:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.918368+0000 osd.1 (osd.1) 314 : cluster [DBG] 2.15 scrub starts 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:56.921282+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.15 scrub ok 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:57.020324+0000 mgr.y (mgr.14553) 598 : audit [DBG] from='client.54233 v1:172.21.15.107:0/1095536105' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.551225+0000 mgr.y (mgr.14553) 599 : cluster [DBG] pgmap v1085: 194 pgs: 194 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.558528+0000 osd.0 (osd.0) 135 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.560580+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:47:58.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.157516+0000 mon.a (mon.0) 3407 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-10T08:47:58.887 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:47:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:58.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:47:58.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:59.020 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:59.144 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:59.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:59.148 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:47:59.148 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:47:59.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.164+0000 7f93537fe700 1 -- 172.21.15.107:0/4263186383 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v719) v1 ==== 101+0+0 (unknown 3945517009 0 0) 0x7f934c048030 con 0x7f935c0f9570 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 10 librados: watch_flush enter 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 10 librados: watch_flush exit 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 20 client.48961.objecter shutdown clearing up homeless session... 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 10 client.48961.objecter successfully canceled tick 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9348063b70 legacy=0x7f9348066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 >> v1:172.21.15.195:6789/0 conn(0x7f935c0f9570 legacy=0x7f935c1129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 shutdown_connections 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 >> 172.21.15.107:0/4263186383 conn(0x7f935c0f2510 msgr2=0x7f935c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 shutdown_connections 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 -- 172.21.15.107:0/4263186383 wait complete. 2024-02-10T08:47:59.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:47:59.168+0000 7f93645de700 1 librados: shutdown 2024-02-10T08:47:59.170 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:47:59.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:47:59.276 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:59.373 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:47:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:59.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:47:59.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:47:59.503 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.640111+0000 osd.7 (osd.7) 290 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.641175+0000 osd.7 (osd.7) 291 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.661658+0000 osd.6 (osd.6) 292 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.663287+0000 osd.6 (osd.6) 293 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.937235+0000 osd.3 (osd.3) 313 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:57.938092+0000 osd.3 (osd.3) 314 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.024903+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.025035+0000 osd.0 (osd.0) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:59.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.413940+0000 mon.a (mon.0) 3408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded) 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:58.521858+0000 mon.b (mon.1) 799 : audit [INF] from='client.? v1:172.21.15.107:0/4263186383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:58.522230+0000 mon.a (mon.0) 3409 : audit [INF] from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.528189+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.529472+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.641057+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.641890+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:59.155068+0000 mon.a (mon.0) 3410 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:59.162250+0000 mon.a (mon.0) 3411 : audit [INF] from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:59.162323+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:59.168297+0000 mon.b (mon.1) 800 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:47:59.168897+0000 mon.a (mon.0) 3413 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.640111+0000 osd.7 (osd.7) 290 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.641175+0000 osd.7 (osd.7) 291 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.661658+0000 osd.6 (osd.6) 292 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.663287+0000 osd.6 (osd.6) 293 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.937235+0000 osd.3 (osd.3) 313 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:57.938092+0000 osd.3 (osd.3) 314 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.024903+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.025035+0000 osd.0 (osd.0) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.413940+0000 mon.a (mon.0) 3408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded) 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:58.521858+0000 mon.b (mon.1) 799 : audit [INF] from='client.? v1:172.21.15.107:0/4263186383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.675 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:58.522230+0000 mon.a (mon.0) 3409 : audit [INF] from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.528189+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.529472+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.641057+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.641890+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:59.155068+0000 mon.a (mon.0) 3410 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:59.162250+0000 mon.a (mon.0) 3411 : audit [INF] from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:59.162323+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:59.168297+0000 mon.b (mon.1) 800 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:47:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:47:59.168897+0000 mon.a (mon.0) 3413 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.692 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1083,"stamp":"2024-02-10T08:47:56.153364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5882,"num_objects":209,"num_object_clones":1,"num_object_copies":627,"num_objects_missing_on_primary":23,"num_objects_missing":23,"num_objects_degraded":14,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1241,"num_read_kb":983,"num_write":762,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":11994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":5,"num_flush_kb":5,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":917,"ondisk_log_size":917,"up":582,"acting":582,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":582,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2462304,"kb_used_data":41496,"kb_used_omap":59,"kb_used_meta":2420676,"kb_avail":747498912,"statfs":{"total":767960285184,"available":765438885888,"internally_reserved":0,"allocated":42491904,"data_stored":22359500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2478772843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":5,"num_flush_kb":5,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999848"},"pg_stats":[{"pgid":"246.17","version":"0'0","reported_seq":11,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161192+0000","last_change":"2024-02-10T08:47:52.178112+0000","last_active":"2024-02-10T08:47:53.161192+0000","last_peered":"2024-02-10T08:47:53.161192+0000","last_clean":"2024-02-10T08:47:53.161192+0000","last_became_active":"2024-02-10T08:47:52.177846+0000","last_became_peered":"2024-02-10T08:47:52.177846+0000","last_unstale":"2024-02-10T08:47:53.161192+0000","last_undegraded":"2024-02-10T08:47:53.161192+0000","last_fullsized":"2024-02-10T08:47:53.161192+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.16","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969011+0000","last_change":"2024-02-10T08:47:52.180772+0000","last_active":"2024-02-10T08:47:54.969011+0000","last_peered":"2024-02-10T08:47:54.969011+0000","last_clean":"2024-02-10T08:47:54.969011+0000","last_became_active":"2024-02-10T08:47:52.180461+0000","last_became_peered":"2024-02-10T08:47:52.180461+0000","last_unstale":"2024-02-10T08:47:54.969011+0000","last_undegraded":"2024-02-10T08:47:54.969011+0000","last_fullsized":"2024-02-10T08:47:54.969011+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"246.15","version":"0'0","reported_seq":9,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-10T08:47:52.164053+0000","last_change":"2024-02-10T08:47:52.164053+0000","last_active":"2024-02-10T08:47:52.164053+0000","last_peered":"2024-02-10T08:47:52.164053+0000","last_clean":"2024-02-10T08:47:52.164053+0000","last_became_active":"2024-02-10T08:47:52.163831+0000","last_became_peered":"2024-02-10T08:47:52.163831+0000","last_unstale":"2024-02-10T08:47:52.164053+0000","last_undegraded":"2024-02-10T08:47:52.164053+0000","last_fullsized":"2024-02-10T08:47:52.164053+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.154875+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"246.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.157937+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.12","version":"0'0","reported_seq":9,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-10T08:47:52.175049+0000","last_change":"2024-02-10T08:47:52.175049+0000","last_active":"2024-02-10T08:47:52.175049+0000","last_peered":"2024-02-10T08:47:52.175049+0000","last_clean":"2024-02-10T08:47:52.175049+0000","last_became_active":"2024-02-10T08:47:52.174770+0000","last_became_peered":"2024-02-10T08:47:52.174770+0000","last_unstale":"2024-02-10T08:47:52.175049+0000","last_undegraded":"2024-02-10T08:47:52.175049+0000","last_fullsized":"2024-02-10T08:47:52.175049+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.157323+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.10","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200031+0000","last_change":"2024-02-10T08:47:52.171990+0000","last_active":"2024-02-10T08:47:54.200031+0000","last_peered":"2024-02-10T08:47:54.200031+0000","last_clean":"2024-02-10T08:47:54.200031+0000","last_became_active":"2024-02-10T08:47:52.171704+0000","last_became_peered":"2024-02-10T08:47:52.171704+0000","last_unstale":"2024-02-10T08:47:54.200031+0000","last_undegraded":"2024-02-10T08:47:54.200031+0000","last_fullsized":"2024-02-10T08:47:54.200031+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.f","version":"0'0","reported_seq":9,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-10T08:47:52.174578+0000","last_change":"2024-02-10T08:47:52.174578+0000","last_active":"2024-02-10T08:47:52.174578+0000","last_peered":"2024-02-10T08:47:52.174578+0000","last_clean":"2024-02-10T08:47:52.174578+0000","last_became_active":"2024-02-10T08:47:52.174306+0000","last_became_peered":"2024-02-10T08:47:52.174306+0000","last_unstale":"2024-02-10T08:47:52.174578+0000","last_undegraded":"2024-02-10T08:47:52.174578+0000","last_fullsized":"2024-02-10T08:47:52.174578+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.e","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967148+0000","last_change":"2024-02-10T08:47:52.178218+0000","last_active":"2024-02-10T08:47:54.967148+0000","last_peered":"2024-02-10T08:47:54.967148+0000","last_clean":"2024-02-10T08:47:54.967148+0000","last_became_active":"2024-02-10T08:47:52.177931+0000","last_became_peered":"2024-02-10T08:47:52.177931+0000","last_unstale":"2024-02-10T08:47:54.967148+0000","last_undegraded":"2024-02-10T08:47:54.967148+0000","last_fullsized":"2024-02-10T08:47:54.967148+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"246.d","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968830+0000","last_change":"2024-02-10T08:47:52.177883+0000","last_active":"2024-02-10T08:47:54.968830+0000","last_peered":"2024-02-10T08:47:54.968830+0000","last_clean":"2024-02-10T08:47:54.968830+0000","last_became_active":"2024-02-10T08:47:52.177613+0000","last_became_peered":"2024-02-10T08:47:52.177613+0000","last_unstale":"2024-02-10T08:47:54.968830+0000","last_undegraded":"2024-02-10T08:47:54.968830+0000","last_fullsized":"2024-02-10T08:47:54.968830+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"246.c","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200478+0000","last_change":"2024-02-10T08:47:52.172321+0000","last_active":"2024-02-10T08:47:54.200478+0000","last_peered":"2024-02-10T08:47:54.200478+0000","last_clean":"2024-02-10T08:47:54.200478+0000","last_became_active":"2024-02-10T08:47:52.171822+0000","last_became_peered":"2024-02-10T08:47:52.171822+0000","last_unstale":"2024-02-10T08:47:54.200478+0000","last_undegraded":"2024-02-10T08:47:54.200478+0000","last_fullsized":"2024-02-10T08:47:54.200478+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.157619+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.a","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200208+0000","last_change":"2024-02-10T08:47:52.167248+0000","last_active":"2024-02-10T08:47:54.200208+0000","last_peered":"2024-02-10T08:47:54.200208+0000","last_clean":"2024-02-10T08:47:54.200208+0000","last_became_active":"2024-02-10T08:47:52.166966+0000","last_became_peered":"2024-02-10T08:47:52.166966+0000","last_unstale":"2024-02-10T08:47:54.200208+0000","last_undegraded":"2024-02-10T08:47:54.200208+0000","last_fullsized":"2024-02-10T08:47:54.200208+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.9","version":"715'10","reported_seq":35,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-10T08:47:55.194168+0000","last_change":"2024-02-10T08:47:52.169478+0000","last_active":"2024-02-10T08:47:55.194168+0000","last_peered":"2024-02-10T08:47:55.194168+0000","last_clean":"2024-02-10T08:47:55.194168+0000","last_became_active":"2024-02-10T08:47:52.169000+0000","last_became_peered":"2024-02-10T08:47:52.169000+0000","last_unstale":"2024-02-10T08:47:55.194168+0000","last_undegraded":"2024-02-10T08:47:55.194168+0000","last_fullsized":"2024-02-10T08:47:55.194168+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":5,"num_flush_kb":5,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"246.8","version":"0'0","reported_seq":11,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.159810+0000","last_change":"2024-02-10T08:47:52.168237+0000","last_active":"2024-02-10T08:47:53.159810+0000","last_peered":"2024-02-10T08:47:53.159810+0000","last_clean":"2024-02-10T08:47:53.159810+0000","last_became_active":"2024-02-10T08:47:52.167953+0000","last_became_peered":"2024-02-10T08:47:52.167953+0000","last_unstale":"2024-02-10T08:47:53.159810+0000","last_undegraded":"2024-02-10T08:47:53.159810+0000","last_fullsized":"2024-02-10T08:47:53.159810+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.7","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200848+0000","last_change":"2024-02-10T08:47:52.171984+0000","last_active":"2024-02-10T08:47:54.200848+0000","last_peered":"2024-02-10T08:47:54.200848+0000","last_clean":"2024-02-10T08:47:54.200848+0000","last_became_active":"2024-02-10T08:47:52.171743+0000","last_became_peered":"2024-02-10T08:47:52.171743+0000","last_unstale":"2024-02-10T08:47:54.200848+0000","last_undegraded":"2024-02-10T08:47:54.200848+0000","last_fullsized":"2024-02-10T08:47:54.200848+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.6","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967253+0000","last_change":"2024-02-10T08:47:52.178188+0000","last_active":"2024-02-10T08:47:54.967253+0000","last_peered":"2024-02-10T08:47:54.967253+0000","last_clean":"2024-02-10T08:47:54.967253+0000","last_became_active":"2024-02-10T08:47:52.177883+0000","last_became_peered":"2024-02-10T08:47:52.177883+0000","last_unstale":"2024-02-10T08:47:54.967253+0000","last_undegraded":"2024-02-10T08:47:54.967253+0000","last_fullsized":"2024-02-10T08:47:54.967253+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"246.5","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391128+0000","last_change":"2024-02-10T08:47:52.174315+0000","last_active":"2024-02-10T08:47:54.391128+0000","last_peered":"2024-02-10T08:47:54.391128+0000","last_clean":"2024-02-10T08:47:54.391128+0000","last_became_active":"2024-02-10T08:47:52.174024+0000","last_became_peered":"2024-02-10T08:47:52.174024+0000","last_unstale":"2024-02-10T08:47:54.391128+0000","last_undegraded":"2024-02-10T08:47:54.391128+0000","last_fullsized":"2024-02-10T08:47:54.391128+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"246.4","version":"0'0","reported_seq":11,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161050+0000","last_change":"2024-02-10T08:47:52.166539+0000","last_active":"2024-02-10T08:47:53.161050+0000","last_peered":"2024-02-10T08:47:53.161050+0000","last_clean":"2024-02-10T08:47:53.161050+0000","last_became_active":"2024-02-10T08:47:52.166272+0000","last_became_peered":"2024-02-10T08:47:52.166272+0000","last_unstale":"2024-02-10T08:47:53.161050+0000","last_undegraded":"2024-02-10T08:47:53.161050+0000","last_fullsized":"2024-02-10T08:47:53.161050+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.155128+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"246.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.157121+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.1","version":"0'0","reported_seq":9,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-10T08:47:52.165434+0000","last_change":"2024-02-10T08:47:52.165434+0000","last_active":"2024-02-10T08:47:52.165434+0000","last_peered":"2024-02-10T08:47:52.165434+0000","last_clean":"2024-02-10T08:47:52.165434+0000","last_became_active":"2024-02-10T08:47:52.165166+0000","last_became_peered":"2024-02-10T08:47:52.165166+0000","last_unstale":"2024-02-10T08:47:52.165434+0000","last_undegraded":"2024-02-10T08:47:52.165434+0000","last_fullsized":"2024-02-10T08:47:52.165434+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.0","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200029+0000","last_change":"2024-02-10T08:47:52.171320+0000","last_active":"2024-02-10T08:47:54.200029+0000","last_peered":"2024-02-10T08:47:54.200029+0000","last_clean":"2024-02-10T08:47:54.200029+0000","last_became_active":"2024-02-10T08:47:52.171036+0000","last_became_peered":"2024-02-10T08:47:52.171036+0000","last_unstale":"2024-02-10T08:47:54.200029+0000","last_undegraded":"2024-02-10T08:47:54.200029+0000","last_fullsized":"2024-02-10T08:47:54.200029+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.1b","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966750+0000","last_change":"2024-02-10T08:47:52.171172+0000","last_active":"2024-02-10T08:47:54.966750+0000","last_peered":"2024-02-10T08:47:54.966750+0000","last_clean":"2024-02-10T08:47:54.966750+0000","last_became_active":"2024-02-10T08:47:52.170885+0000","last_became_peered":"2024-02-10T08:47:52.170885+0000","last_unstale":"2024-02-10T08:47:54.966750+0000","last_undegraded":"2024-02-10T08:47:54.966750+0000","last_fullsized":"2024-02-10T08:47:54.966750+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.7","version":"669'16","reported_seq":167,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200926+0000","last_change":"2024-02-10T08:47:46.185206+0000","last_active":"2024-02-10T08:47:54.200926+0000","last_peered":"2024-02-10T08:47:54.200926+0000","last_clean":"2024-02-10T08:47:54.200926+0000","last_became_active":"2024-02-10T08:45:46.132344+0000","last_became_peered":"2024-02-10T08:45:46.132344+0000","last_unstale":"2024-02-10T08:47:54.200926+0000","last_undegraded":"2024-02-10T08:47:54.200926+0000","last_fullsized":"2024-02-10T08:47:54.200926+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'16","last_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_deep_scrub":"669'16","last_deep_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_clean_scrub_stamp":"2024-02-10T08:47:22.551825+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1a","version":"0'0","reported_seq":9,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-02-10T08:47:52.174506+0000","last_change":"2024-02-10T08:47:52.174506+0000","last_active":"2024-02-10T08:47:52.174506+0000","last_peered":"2024-02-10T08:47:52.174506+0000","last_clean":"2024-02-10T08:47:52.174506+0000","last_became_active":"2024-02-10T08:47:52.174234+0000","last_became_peered":"2024-02-10T08:47:52.174234+0000","last_unstale":"2024-02-10T08:47:52.174506+0000","last_undegraded":"2024-02-10T08:47:52.174506+0000","last_fullsized":"2024-02-10T08:47:52.174506+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.6","version":"601'1","reported_seq":172,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391332+0000","last_change":"2024-02-10T08:47:49.276664+0000","last_active":"2024-02-10T08:47:54.391332+0000","last_peered":"2024-02-10T08:47:54.391332+0000","last_clean":"2024-02-10T08:47:54.391332+0000","last_became_active":"2024-02-10T08:47:49.222914+0000","last_became_peered":"2024-02-10T08:47:49.222914+0000","last_unstale":"2024-02-10T08:47:54.391332+0000","last_undegraded":"2024-02-10T08:47:54.391332+0000","last_fullsized":"2024-02-10T08:47:54.391332+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'1","last_scrub_stamp":"2024-02-10T08:47:18.549603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.549603+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.19","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200334+0000","last_change":"2024-02-10T08:47:52.167610+0000","last_active":"2024-02-10T08:47:54.200334+0000","last_peered":"2024-02-10T08:47:54.200334+0000","last_clean":"2024-02-10T08:47:54.200334+0000","last_became_active":"2024-02-10T08:47:52.167331+0000","last_became_peered":"2024-02-10T08:47:52.167331+0000","last_unstale":"2024-02-10T08:47:54.200334+0000","last_undegraded":"2024-02-10T08:47:54.200334+0000","last_fullsized":"2024-02-10T08:47:54.200334+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.5","version":"706'13","reported_seq":178,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.290588+0000","last_change":"2024-02-10T08:47:48.198518+0000","last_active":"2024-02-10T08:47:50.290588+0000","last_peered":"2024-02-10T08:47:50.290588+0000","last_clean":"2024-02-10T08:47:50.290588+0000","last_became_active":"2024-02-10T08:46:29.211292+0000","last_became_peered":"2024-02-10T08:46:29.211292+0000","last_unstale":"2024-02-10T08:47:50.290588+0000","last_undegraded":"2024-02-10T08:47:50.290588+0000","last_fullsized":"2024-02-10T08:47:50.290588+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.162637+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"693'17","reported_seq":212,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161169+0000","last_change":"2024-02-10T08:47:49.992426+0000","last_active":"2024-02-10T08:47:53.161169+0000","last_peered":"2024-02-10T08:47:53.161169+0000","last_clean":"2024-02-10T08:47:53.161169+0000","last_became_active":"2024-02-10T08:47:49.231271+0000","last_became_peered":"2024-02-10T08:47:49.231271+0000","last_unstale":"2024-02-10T08:47:53.161169+0000","last_undegraded":"2024-02-10T08:47:53.161169+0000","last_fullsized":"2024-02-10T08:47:53.161169+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1f","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967377+0000","last_change":"2024-02-10T08:47:52.171435+0000","last_active":"2024-02-10T08:47:54.967377+0000","last_peered":"2024-02-10T08:47:54.967377+0000","last_clean":"2024-02-10T08:47:54.967377+0000","last_became_active":"2024-02-10T08:47:52.171160+0000","last_became_peered":"2024-02-10T08:47:52.171160+0000","last_unstale":"2024-02-10T08:47:54.967377+0000","last_undegraded":"2024-02-10T08:47:54.967377+0000","last_fullsized":"2024-02-10T08:47:54.967377+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":138,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966811+0000","last_change":"2024-02-10T08:47:46.201481+0000","last_active":"2024-02-10T08:47:54.966811+0000","last_peered":"2024-02-10T08:47:54.966811+0000","last_clean":"2024-02-10T08:47:54.966811+0000","last_became_active":"2024-02-10T08:45:46.129572+0000","last_became_peered":"2024-02-10T08:45:46.129572+0000","last_unstale":"2024-02-10T08:47:54.966811+0000","last_undegraded":"2024-02-10T08:47:54.966811+0000","last_fullsized":"2024-02-10T08:47:54.966811+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1e","version":"0'0","reported_seq":12,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391439+0000","last_change":"2024-02-10T08:47:52.173843+0000","last_active":"2024-02-10T08:47:54.391439+0000","last_peered":"2024-02-10T08:47:54.391439+0000","last_clean":"2024-02-10T08:47:54.391439+0000","last_became_active":"2024-02-10T08:47:52.173569+0000","last_became_peered":"2024-02-10T08:47:52.173569+0000","last_unstale":"2024-02-10T08:47:54.391439+0000","last_undegraded":"2024-02-10T08:47:54.391439+0000","last_fullsized":"2024-02-10T08:47:54.391439+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":153,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.390725+0000","last_change":"2024-02-10T08:47:46.192141+0000","last_active":"2024-02-10T08:47:54.390725+0000","last_peered":"2024-02-10T08:47:54.390725+0000","last_clean":"2024-02-10T08:47:54.390725+0000","last_became_active":"2024-02-10T08:46:15.351881+0000","last_became_peered":"2024-02-10T08:46:15.351881+0000","last_unstale":"2024-02-10T08:47:54.390725+0000","last_undegraded":"2024-02-10T08:47:54.390725+0000","last_fullsized":"2024-02-10T08:47:54.390725+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:22.632348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:47:22.632348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1d","version":"0'0","reported_seq":11,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160900+0000","last_change":"2024-02-10T08:47:52.166601+0000","last_active":"2024-02-10T08:47:53.160900+0000","last_peered":"2024-02-10T08:47:53.160900+0000","last_clean":"2024-02-10T08:47:53.160900+0000","last_became_active":"2024-02-10T08:47:52.166337+0000","last_became_peered":"2024-02-10T08:47:52.166337+0000","last_unstale":"2024-02-10T08:47:53.160900+0000","last_undegraded":"2024-02-10T08:47:53.160900+0000","last_fullsized":"2024-02-10T08:47:53.160900+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.1","version":"715'6","reported_seq":166,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-10T08:47:55.176903+0000","last_change":"2024-02-10T08:47:46.186999+0000","last_active":"2024-02-10T08:47:55.176903+0000","last_peered":"2024-02-10T08:47:55.176903+0000","last_clean":"2024-02-10T08:47:55.176903+0000","last_became_active":"2024-02-10T08:46:29.219928+0000","last_became_peered":"2024-02-10T08:46:29.219928+0000","last_unstale":"2024-02-10T08:47:55.176903+0000","last_undegraded":"2024-02-10T08:47:55.176903+0000","last_fullsized":"2024-02-10T08:47:55.176903+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-10T08:47:51.147909+0000","last_change":"2024-02-10T08:47:51.157181+0000","last_active":"2024-02-10T08:47:51.147909+0000","last_peered":"2024-02-10T08:47:51.147909+0000","last_clean":"2024-02-10T08:47:51.147909+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:47:51.147909+0000","last_undegraded":"2024-02-10T08:47:51.147909+0000","last_fullsized":"2024-02-10T08:47:51.147909+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":711,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:51.147909+0000","last_clean_scrub_stamp":"2024-02-10T08:47:51.147909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.0","version":"685'16","reported_seq":122,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.154746+0000","last_change":"2024-02-10T08:47:46.187215+0000","last_active":"2024-02-10T08:47:51.154746+0000","last_peered":"2024-02-10T08:47:51.154746+0000","last_clean":"2024-02-10T08:47:51.154746+0000","last_became_active":"2024-02-10T08:46:29.210532+0000","last_became_peered":"2024-02-10T08:46:29.210532+0000","last_unstale":"2024-02-10T08:47:51.154746+0000","last_undegraded":"2024-02-10T08:47:51.154746+0000","last_fullsized":"2024-02-10T08:47:51.154746+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:45:45.107430+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"5.9","version":"636'18","reported_seq":386,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200582+0000","last_change":"2024-02-10T08:46:40.601578+0000","last_active":"2024-02-10T08:47:54.200582+0000","last_peered":"2024-02-10T08:47:54.200582+0000","last_clean":"2024-02-10T08:47:54.200582+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:47:54.200582+0000","last_undegraded":"2024-02-10T08:47:54.200582+0000","last_fullsized":"2024-02-10T08:47:54.200582+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:46:40.601510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.601510+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":780,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966487+0000","last_change":"2024-02-10T08:45:22.037140+0000","last_active":"2024-02-10T08:47:54.966487+0000","last_peered":"2024-02-10T08:47:54.966487+0000","last_clean":"2024-02-10T08:47:54.966487+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:47:54.966487+0000","last_undegraded":"2024-02-10T08:47:54.966487+0000","last_fullsized":"2024-02-10T08:47:54.966487+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:45:22.037068+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.037068+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":707,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291390+0000","last_change":"2024-02-10T08:45:22.422467+0000","last_active":"2024-02-10T08:47:50.291390+0000","last_peered":"2024-02-10T08:47:50.291390+0000","last_clean":"2024-02-10T08:47:50.291390+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:47:50.291390+0000","last_undegraded":"2024-02-10T08:47:50.291390+0000","last_fullsized":"2024-02-10T08:47:50.291390+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:22.422382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:45:22.422382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":699,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966730+0000","last_change":"2024-02-10T08:46:15.332578+0000","last_active":"2024-02-10T08:47:54.966730+0000","last_peered":"2024-02-10T08:47:54.966730+0000","last_clean":"2024-02-10T08:47:54.966730+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:47:54.966730+0000","last_undegraded":"2024-02-10T08:47:54.966730+0000","last_fullsized":"2024-02-10T08:47:54.966730+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:59.660566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:59.660566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":653,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291517+0000","last_change":"2024-02-10T08:47:49.237658+0000","last_active":"2024-02-10T08:47:50.291517+0000","last_peered":"2024-02-10T08:47:50.291517+0000","last_clean":"2024-02-10T08:47:50.291517+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:47:50.291517+0000","last_undegraded":"2024-02-10T08:47:50.291517+0000","last_fullsized":"2024-02-10T08:47:50.291517+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:47.472086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:47.472086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":161,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968608+0000","last_change":"2024-02-10T08:47:49.969280+0000","last_active":"2024-02-10T08:47:54.968608+0000","last_peered":"2024-02-10T08:47:54.968608+0000","last_clean":"2024-02-10T08:47:54.968608+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:47:54.968608+0000","last_undegraded":"2024-02-10T08:47:54.968608+0000","last_fullsized":"2024-02-10T08:47:54.968608+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:46:32.979941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.979941+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.218333+0000","last_change":"2024-02-10T08:47:49.242055+0000","last_active":"2024-02-10T08:47:50.218333+0000","last_peered":"2024-02-10T08:47:50.218333+0000","last_clean":"2024-02-10T08:47:50.218333+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:47:50.218333+0000","last_undegraded":"2024-02-10T08:47:50.218333+0000","last_fullsized":"2024-02-10T08:47:50.218333+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.584151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.584151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":718,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289811+0000","last_change":"2024-02-10T08:46:31.356749+0000","last_active":"2024-02-10T08:47:50.289811+0000","last_peered":"2024-02-10T08:47:50.289811+0000","last_clean":"2024-02-10T08:47:50.289811+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:47:50.289811+0000","last_undegraded":"2024-02-10T08:47:50.289811+0000","last_fullsized":"2024-02-10T08:47:50.289811+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.356675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.356675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968532+0000","last_change":"2024-02-10T08:47:17.396762+0000","last_active":"2024-02-10T08:47:54.968532+0000","last_peered":"2024-02-10T08:47:54.968532+0000","last_clean":"2024-02-10T08:47:54.968532+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:47:54.968532+0000","last_undegraded":"2024-02-10T08:47:54.968532+0000","last_fullsized":"2024-02-10T08:47:54.968532+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.396696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.396696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":648,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967705+0000","last_change":"2024-02-10T08:47:19.868939+0000","last_active":"2024-02-10T08:47:54.967705+0000","last_peered":"2024-02-10T08:47:54.967705+0000","last_clean":"2024-02-10T08:47:54.967705+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:47:54.967705+0000","last_undegraded":"2024-02-10T08:47:54.967705+0000","last_fullsized":"2024-02-10T08:47:54.967705+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:19.868859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.868859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":872,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.390891+0000","last_change":"2024-02-10T08:47:50.045869+0000","last_active":"2024-02-10T08:47:54.390891+0000","last_peered":"2024-02-10T08:47:54.390891+0000","last_clean":"2024-02-10T08:47:54.390891+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:47:54.390891+0000","last_undegraded":"2024-02-10T08:47:54.390891+0000","last_fullsized":"2024-02-10T08:47:54.390891+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.661956+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":705,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291737+0000","last_change":"2024-02-10T08:46:15.374244+0000","last_active":"2024-02-10T08:47:50.291737+0000","last_peered":"2024-02-10T08:47:50.291737+0000","last_clean":"2024-02-10T08:47:50.291737+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:47:50.291737+0000","last_undegraded":"2024-02-10T08:47:50.291737+0000","last_fullsized":"2024-02-10T08:47:50.291737+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.452719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":739,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289869+0000","last_change":"2024-02-10T08:46:15.367673+0000","last_active":"2024-02-10T08:47:50.289869+0000","last_peered":"2024-02-10T08:47:50.289869+0000","last_clean":"2024-02-10T08:47:50.289869+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:47:50.289869+0000","last_undegraded":"2024-02-10T08:47:50.289869+0000","last_fullsized":"2024-02-10T08:47:50.289869+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:24.427791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:24.427791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":266,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200407+0000","last_change":"2024-02-10T08:46:46.484714+0000","last_active":"2024-02-10T08:47:54.200407+0000","last_peered":"2024-02-10T08:47:54.200407+0000","last_clean":"2024-02-10T08:47:54.200407+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:47:54.200407+0000","last_undegraded":"2024-02-10T08:47:54.200407+0000","last_fullsized":"2024-02-10T08:47:54.200407+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.484656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":276,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.201116+0000","last_change":"2024-02-10T08:46:31.456642+0000","last_active":"2024-02-10T08:47:54.201116+0000","last_peered":"2024-02-10T08:47:54.201116+0000","last_clean":"2024-02-10T08:47:54.201116+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:47:54.201116+0000","last_undegraded":"2024-02-10T08:47:54.201116+0000","last_fullsized":"2024-02-10T08:47:54.201116+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.456579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.456579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":647,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.156386+0000","last_change":"2024-02-10T08:47:49.965414+0000","last_active":"2024-02-10T08:47:51.156386+0000","last_peered":"2024-02-10T08:47:51.156386+0000","last_clean":"2024-02-10T08:47:51.156386+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:47:51.156386+0000","last_undegraded":"2024-02-10T08:47:51.156386+0000","last_fullsized":"2024-02-10T08:47:51.156386+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:47:26.935760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.935760+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160580+0000","last_change":"2024-02-10T08:47:49.226548+0000","last_active":"2024-02-10T08:47:53.160580+0000","last_peered":"2024-02-10T08:47:53.160580+0000","last_clean":"2024-02-10T08:47:53.160580+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:47:53.160580+0000","last_undegraded":"2024-02-10T08:47:53.160580+0000","last_fullsized":"2024-02-10T08:47:53.160580+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.215980+0000","last_change":"2024-02-10T08:47:49.256765+0000","last_active":"2024-02-10T08:47:50.215980+0000","last_peered":"2024-02-10T08:47:50.215980+0000","last_clean":"2024-02-10T08:47:50.215980+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:47:50.215980+0000","last_undegraded":"2024-02-10T08:47:50.215980+0000","last_fullsized":"2024-02-10T08:47:50.215980+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.672468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.672468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":696,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291069+0000","last_change":"2024-02-10T08:46:32.371559+0000","last_active":"2024-02-10T08:47:50.291069+0000","last_peered":"2024-02-10T08:47:50.291069+0000","last_clean":"2024-02-10T08:47:50.291069+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:47:50.291069+0000","last_undegraded":"2024-02-10T08:47:50.291069+0000","last_fullsized":"2024-02-10T08:47:50.291069+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:32.371497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.371497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":502,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.156053+0000","last_change":"2024-02-10T08:47:49.242441+0000","last_active":"2024-02-10T08:47:51.156053+0000","last_peered":"2024-02-10T08:47:51.156053+0000","last_clean":"2024-02-10T08:47:51.156053+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:47:51.156053+0000","last_undegraded":"2024-02-10T08:47:51.156053+0000","last_fullsized":"2024-02-10T08:47:51.156053+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:48.843209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.843209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":566,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155094+0000","last_change":"2024-02-10T08:46:36.873208+0000","last_active":"2024-02-10T08:47:51.155094+0000","last_peered":"2024-02-10T08:47:51.155094+0000","last_clean":"2024-02-10T08:47:51.155094+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:47:51.155094+0000","last_undegraded":"2024-02-10T08:47:51.155094+0000","last_fullsized":"2024-02-10T08:47:51.155094+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:46:36.873122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.873122+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.214928+0000","last_change":"2024-02-10T08:47:49.236238+0000","last_active":"2024-02-10T08:47:50.214928+0000","last_peered":"2024-02-10T08:47:50.214928+0000","last_clean":"2024-02-10T08:47:50.214928+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:47:50.214928+0000","last_undegraded":"2024-02-10T08:47:50.214928+0000","last_fullsized":"2024-02-10T08:47:50.214928+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:49.582110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.582110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":945,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.290683+0000","last_change":"2024-02-10T08:47:29.049751+0000","last_active":"2024-02-10T08:47:50.290683+0000","last_peered":"2024-02-10T08:47:50.290683+0000","last_clean":"2024-02-10T08:47:50.290683+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:47:50.290683+0000","last_undegraded":"2024-02-10T08:47:50.290683+0000","last_fullsized":"2024-02-10T08:47:50.290683+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:47:29.049685+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.049685+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":526,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.160225+0000","last_change":"2024-02-10T08:47:49.242916+0000","last_active":"2024-02-10T08:47:51.160225+0000","last_peered":"2024-02-10T08:47:51.160225+0000","last_clean":"2024-02-10T08:47:51.160225+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:47:51.160225+0000","last_undegraded":"2024-02-10T08:47:51.160225+0000","last_fullsized":"2024-02-10T08:47:51.160225+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:37.823468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:37.823468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968533+0000","last_change":"2024-02-10T08:46:42.008821+0000","last_active":"2024-02-10T08:47:54.968533+0000","last_peered":"2024-02-10T08:47:54.968533+0000","last_clean":"2024-02-10T08:47:54.968533+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:47:54.968533+0000","last_undegraded":"2024-02-10T08:47:54.968533+0000","last_fullsized":"2024-02-10T08:47:54.968533+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:46:42.008755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":10,"reported_epoch":710,"state":"active+recovery_wait+degraded","last_fresh":"2024-02-10T08:47:50.219057+0000","last_change":"2024-02-10T08:47:49.244230+0000","last_active":"2024-02-10T08:47:50.219057+0000","last_peered":"2024-02-10T08:47:50.219057+0000","last_clean":"2024-02-10T08:47:47.446413+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:47:50.219057+0000","last_undegraded":"2024-02-10T08:47:49.204357+0000","last_fullsized":"2024-02-10T08:47:50.219057+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:46:41.670342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.670342+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":16,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.656744+0000","last_change":"2024-02-10T08:47:50.656406+0000","last_active":"2024-02-10T08:47:50.656744+0000","last_peered":"2024-02-10T08:47:50.656744+0000","last_clean":"2024-02-10T08:47:50.656744+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:47:50.656744+0000","last_undegraded":"2024-02-10T08:47:50.656744+0000","last_fullsized":"2024-02-10T08:47:50.656744+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:47:13.515134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.515134+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":756,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.657335+0000","last_change":"2024-02-10T08:47:13.008447+0000","last_active":"2024-02-10T08:47:51.657335+0000","last_peered":"2024-02-10T08:47:51.657335+0000","last_clean":"2024-02-10T08:47:51.657335+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:47:51.657335+0000","last_undegraded":"2024-02-10T08:47:51.657335+0000","last_fullsized":"2024-02-10T08:47:51.657335+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:47:13.008374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.008374+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":717,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967070+0000","last_change":"2024-02-10T08:47:49.238554+0000","last_active":"2024-02-10T08:47:54.967070+0000","last_peered":"2024-02-10T08:47:54.967070+0000","last_clean":"2024-02-10T08:47:54.967070+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:47:54.967070+0000","last_undegraded":"2024-02-10T08:47:54.967070+0000","last_fullsized":"2024-02-10T08:47:54.967070+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:31.837641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:47:31.837641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":666,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967235+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:47:54.967235+0000","last_peered":"2024-02-10T08:47:54.967235+0000","last_clean":"2024-02-10T08:47:54.967235+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:47:54.967235+0000","last_undegraded":"2024-02-10T08:47:54.967235+0000","last_fullsized":"2024-02-10T08:47:54.967235+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.824249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.824249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":609,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.160387+0000","last_change":"2024-02-10T08:47:50.476339+0000","last_active":"2024-02-10T08:47:51.160387+0000","last_peered":"2024-02-10T08:47:51.160387+0000","last_clean":"2024-02-10T08:47:51.160387+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:47:51.160387+0000","last_undegraded":"2024-02-10T08:47:51.160387+0000","last_fullsized":"2024-02-10T08:47:51.160387+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:46:45.859396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.859396+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":294,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200307+0000","last_change":"2024-02-10T08:47:49.253011+0000","last_active":"2024-02-10T08:47:54.200307+0000","last_peered":"2024-02-10T08:47:54.200307+0000","last_clean":"2024-02-10T08:47:54.200307+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:47:54.200307+0000","last_undegraded":"2024-02-10T08:47:54.200307+0000","last_fullsized":"2024-02-10T08:47:54.200307+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:28.372323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:28.372323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":851,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967596+0000","last_change":"2024-02-10T08:46:38.853235+0000","last_active":"2024-02-10T08:47:54.967596+0000","last_peered":"2024-02-10T08:47:54.967596+0000","last_clean":"2024-02-10T08:47:54.967596+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:47:54.967596+0000","last_undegraded":"2024-02-10T08:47:54.967596+0000","last_fullsized":"2024-02-10T08:47:54.967596+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:38.853141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.853141+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":683,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161425+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:47:53.161425+0000","last_peered":"2024-02-10T08:47:53.161425+0000","last_clean":"2024-02-10T08:47:53.161425+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:47:53.161425+0000","last_undegraded":"2024-02-10T08:47:53.161425+0000","last_fullsized":"2024-02-10T08:47:53.161425+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":629,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160295+0000","last_change":"2024-02-10T08:46:09.490629+0000","last_active":"2024-02-10T08:47:53.160295+0000","last_peered":"2024-02-10T08:47:53.160295+0000","last_clean":"2024-02-10T08:47:53.160295+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:47:53.160295+0000","last_undegraded":"2024-02-10T08:47:53.160295+0000","last_fullsized":"2024-02-10T08:47:53.160295+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:09.490555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.490555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":914,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160013+0000","last_change":"2024-02-10T08:47:49.260456+0000","last_active":"2024-02-10T08:47:53.160013+0000","last_peered":"2024-02-10T08:47:53.160013+0000","last_clean":"2024-02-10T08:47:53.160013+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:47:53.160013+0000","last_undegraded":"2024-02-10T08:47:53.160013+0000","last_fullsized":"2024-02-10T08:47:53.160013+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:39.499038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.499038+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":585,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155735+0000","last_change":"2024-02-10T08:47:27.942949+0000","last_active":"2024-02-10T08:47:51.155735+0000","last_peered":"2024-02-10T08:47:51.155735+0000","last_clean":"2024-02-10T08:47:51.155735+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:47:51.155735+0000","last_undegraded":"2024-02-10T08:47:51.155735+0000","last_fullsized":"2024-02-10T08:47:51.155735+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:47:27.942861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.942861+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":721,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391455+0000","last_change":"2024-02-10T08:47:21.618028+0000","last_active":"2024-02-10T08:47:54.391455+0000","last_peered":"2024-02-10T08:47:54.391455+0000","last_clean":"2024-02-10T08:47:54.391455+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:47:54.391455+0000","last_undegraded":"2024-02-10T08:47:54.391455+0000","last_fullsized":"2024-02-10T08:47:54.391455+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.617907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":273,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-10T08:47:55.566362+0000","last_change":"2024-02-10T08:47:55.566362+0000","last_active":"2024-02-10T08:47:55.566362+0000","last_peered":"2024-02-10T08:47:55.566362+0000","last_clean":"2024-02-10T08:47:55.566362+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:47:55.566362+0000","last_undegraded":"2024-02-10T08:47:55.566362+0000","last_fullsized":"2024-02-10T08:47:55.566362+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.566281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.566281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":120,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969023+0000","last_change":"2024-02-10T08:47:49.242386+0000","last_active":"2024-02-10T08:47:54.969023+0000","last_peered":"2024-02-10T08:47:54.969023+0000","last_clean":"2024-02-10T08:47:54.969023+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:47:54.969023+0000","last_undegraded":"2024-02-10T08:47:54.969023+0000","last_fullsized":"2024-02-10T08:47:54.969023+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:29.923093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:29.923093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":545,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155526+0000","last_change":"2024-02-10T08:46:49.870373+0000","last_active":"2024-02-10T08:47:51.155526+0000","last_peered":"2024-02-10T08:47:51.155526+0000","last_clean":"2024-02-10T08:47:51.155526+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:47:51.155526+0000","last_undegraded":"2024-02-10T08:47:51.155526+0000","last_fullsized":"2024-02-10T08:47:51.155526+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:49.870278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.870278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":876,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160004+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:47:53.160004+0000","last_peered":"2024-02-10T08:47:53.160004+0000","last_clean":"2024-02-10T08:47:53.160004+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:47:53.160004+0000","last_undegraded":"2024-02-10T08:47:53.160004+0000","last_fullsized":"2024-02-10T08:47:53.160004+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":10,"reported_epoch":710,"state":"active+recovery_wait+degraded","last_fresh":"2024-02-10T08:47:50.218472+0000","last_change":"2024-02-10T08:47:49.245171+0000","last_active":"2024-02-10T08:47:50.218472+0000","last_peered":"2024-02-10T08:47:50.218472+0000","last_clean":"2024-02-10T08:33:18.109579+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:47:50.218472+0000","last_undegraded":"2024-02-10T08:47:49.203851+0000","last_fullsized":"2024-02-10T08:47:50.218472+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:46:33.634650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:33.634650+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":513,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155861+0000","last_change":"2024-02-10T08:46:29.907845+0000","last_active":"2024-02-10T08:47:51.155861+0000","last_peered":"2024-02-10T08:47:51.155861+0000","last_clean":"2024-02-10T08:47:51.155861+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:47:51.155861+0000","last_undegraded":"2024-02-10T08:47:51.155861+0000","last_fullsized":"2024-02-10T08:47:51.155861+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:29.907739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:29.907739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":120,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968904+0000","last_change":"2024-02-10T08:47:49.240988+0000","last_active":"2024-02-10T08:47:54.968904+0000","last_peered":"2024-02-10T08:47:54.968904+0000","last_clean":"2024-02-10T08:47:54.968904+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:47:54.968904+0000","last_undegraded":"2024-02-10T08:47:54.968904+0000","last_fullsized":"2024-02-10T08:47:54.968904+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:45.003808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.003808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":708,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391059+0000","last_change":"2024-02-10T08:47:17.571180+0000","last_active":"2024-02-10T08:47:54.391059+0000","last_peered":"2024-02-10T08:47:54.391059+0000","last_clean":"2024-02-10T08:47:54.391059+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:47:54.391059+0000","last_undegraded":"2024-02-10T08:47:54.391059+0000","last_fullsized":"2024-02-10T08:47:54.391059+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.571093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.571093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":115,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968654+0000","last_change":"2024-02-10T08:46:15.400357+0000","last_active":"2024-02-10T08:47:54.968654+0000","last_peered":"2024-02-10T08:47:54.968654+0000","last_clean":"2024-02-10T08:47:54.968654+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:47:54.968654+0000","last_undegraded":"2024-02-10T08:47:54.968654+0000","last_fullsized":"2024-02-10T08:47:54.968654+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:46:06.677291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:06.677291+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.201006+0000","last_change":"2024-02-10T08:46:41.550928+0000","last_active":"2024-02-10T08:47:54.201006+0000","last_peered":"2024-02-10T08:47:54.201006+0000","last_clean":"2024-02-10T08:47:54.201006+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:47:54.201006+0000","last_undegraded":"2024-02-10T08:47:54.201006+0000","last_fullsized":"2024-02-10T08:47:54.201006+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.550852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.550852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":589,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.290590+0000","last_change":"2024-02-10T08:47:49.238746+0000","last_active":"2024-02-10T08:47:50.290590+0000","last_peered":"2024-02-10T08:47:50.290590+0000","last_clean":"2024-02-10T08:47:50.290590+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:47:50.290590+0000","last_undegraded":"2024-02-10T08:47:50.290590+0000","last_fullsized":"2024-02-10T08:47:50.290590+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.455169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.455169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":671,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.290836+0000","last_change":"2024-02-10T08:47:49.230315+0000","last_active":"2024-02-10T08:47:50.290836+0000","last_peered":"2024-02-10T08:47:50.290836+0000","last_clean":"2024-02-10T08:47:50.290836+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:47:50.290836+0000","last_undegraded":"2024-02-10T08:47:50.290836+0000","last_fullsized":"2024-02-10T08:47:50.290836+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:05.371003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:05.371003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":743,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967608+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:47:54.967608+0000","last_peered":"2024-02-10T08:47:54.967608+0000","last_clean":"2024-02-10T08:47:54.967608+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:47:54.967608+0000","last_undegraded":"2024-02-10T08:47:54.967608+0000","last_fullsized":"2024-02-10T08:47:54.967608+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:13.825974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.825974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":162,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969249+0000","last_change":"2024-02-10T08:47:49.975699+0000","last_active":"2024-02-10T08:47:54.969249+0000","last_peered":"2024-02-10T08:47:54.969249+0000","last_clean":"2024-02-10T08:47:54.969249+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:47:54.969249+0000","last_undegraded":"2024-02-10T08:47:54.969249+0000","last_fullsized":"2024-02-10T08:47:54.969249+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.370523+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":298,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200264+0000","last_change":"2024-02-10T08:47:49.252790+0000","last_active":"2024-02-10T08:47:54.200264+0000","last_peered":"2024-02-10T08:47:54.200264+0000","last_clean":"2024-02-10T08:47:54.200264+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:47:54.200264+0000","last_undegraded":"2024-02-10T08:47:54.200264+0000","last_fullsized":"2024-02-10T08:47:54.200264+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.601673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.601673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":113,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968429+0000","last_change":"2024-02-10T08:46:38.039133+0000","last_active":"2024-02-10T08:47:54.968429+0000","last_peered":"2024-02-10T08:47:54.968429+0000","last_clean":"2024-02-10T08:47:54.968429+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:47:54.968429+0000","last_undegraded":"2024-02-10T08:47:54.968429+0000","last_fullsized":"2024-02-10T08:47:54.968429+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:38.039064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.039064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"63'1","reported_seq":10,"reported_epoch":710,"state":"active+recovery_wait+degraded","last_fresh":"2024-02-10T08:47:50.216106+0000","last_change":"2024-02-10T08:47:49.249350+0000","last_active":"2024-02-10T08:47:50.216106+0000","last_peered":"2024-02-10T08:47:50.216106+0000","last_clean":"2024-02-10T08:33:25.110489+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:47:50.216106+0000","last_undegraded":"2024-02-10T08:47:49.205724+0000","last_fullsized":"2024-02-10T08:47:50.216106+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:45:57.867134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:57.867134+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":655,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.390753+0000","last_change":"2024-02-10T08:47:26.751598+0000","last_active":"2024-02-10T08:47:54.390753+0000","last_peered":"2024-02-10T08:47:54.390753+0000","last_clean":"2024-02-10T08:47:54.390753+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:47:54.390753+0000","last_undegraded":"2024-02-10T08:47:54.390753+0000","last_fullsized":"2024-02-10T08:47:54.390753+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.751532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.751532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":273,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200755+0000","last_change":"2024-02-10T08:47:21.545206+0000","last_active":"2024-02-10T08:47:54.200755+0000","last_peered":"2024-02-10T08:47:54.200755+0000","last_clean":"2024-02-10T08:47:54.200755+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:47:54.200755+0000","last_undegraded":"2024-02-10T08:47:54.200755+0000","last_fullsized":"2024-02-10T08:47:54.200755+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:21.545082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.545082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":787,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391253+0000","last_change":"2024-02-10T08:46:30.592735+0000","last_active":"2024-02-10T08:47:54.391253+0000","last_peered":"2024-02-10T08:47:54.391253+0000","last_clean":"2024-02-10T08:47:54.391253+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:47:54.391253+0000","last_undegraded":"2024-02-10T08:47:54.391253+0000","last_fullsized":"2024-02-10T08:47:54.391253+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:46:30.592668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.592668+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":277,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200790+0000","last_change":"2024-02-10T08:46:50.496168+0000","last_active":"2024-02-10T08:47:54.200790+0000","last_peered":"2024-02-10T08:47:54.200790+0000","last_clean":"2024-02-10T08:47:54.200790+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:47:54.200790+0000","last_undegraded":"2024-02-10T08:47:54.200790+0000","last_fullsized":"2024-02-10T08:47:54.200790+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:50.496092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:46:50.496092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":705,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967243+0000","last_change":"2024-02-10T08:47:16.775257+0000","last_active":"2024-02-10T08:47:54.967243+0000","last_peered":"2024-02-10T08:47:54.967243+0000","last_clean":"2024-02-10T08:47:54.967243+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:47:54.967243+0000","last_undegraded":"2024-02-10T08:47:54.967243+0000","last_fullsized":"2024-02-10T08:47:54.967243+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.775179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.775179+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":685,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160458+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:47:53.160458+0000","last_peered":"2024-02-10T08:47:53.160458+0000","last_clean":"2024-02-10T08:47:53.160458+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:47:53.160458+0000","last_undegraded":"2024-02-10T08:47:53.160458+0000","last_fullsized":"2024-02-10T08:47:53.160458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":691,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967683+0000","last_change":"2024-02-10T08:46:34.787736+0000","last_active":"2024-02-10T08:47:54.967683+0000","last_peered":"2024-02-10T08:47:54.967683+0000","last_clean":"2024-02-10T08:47:54.967683+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:47:54.967683+0000","last_undegraded":"2024-02-10T08:47:54.967683+0000","last_fullsized":"2024-02-10T08:47:54.967683+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:34.787659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:34.787659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":670,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969143+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:47:54.969143+0000","last_peered":"2024-02-10T08:47:54.969143+0000","last_clean":"2024-02-10T08:47:54.969143+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:47:54.969143+0000","last_undegraded":"2024-02-10T08:47:54.969143+0000","last_fullsized":"2024-02-10T08:47:54.969143+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":724,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291573+0000","last_change":"2024-02-10T08:47:26.043612+0000","last_active":"2024-02-10T08:47:50.291573+0000","last_peered":"2024-02-10T08:47:50.291573+0000","last_clean":"2024-02-10T08:47:50.291573+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:47:50.291573+0000","last_undegraded":"2024-02-10T08:47:50.291573+0000","last_fullsized":"2024-02-10T08:47:50.291573+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.043548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.043548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":10,"reported_epoch":710,"state":"active+recovery_wait+degraded","last_fresh":"2024-02-10T08:47:50.219206+0000","last_change":"2024-02-10T08:47:49.250586+0000","last_active":"2024-02-10T08:47:50.219206+0000","last_peered":"2024-02-10T08:47:50.219206+0000","last_clean":"2024-02-10T08:46:10.310024+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:47:50.219206+0000","last_undegraded":"2024-02-10T08:47:49.204858+0000","last_fullsized":"2024-02-10T08:47:50.219206+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:47:30.847592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.847592+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":5,"num_objects_missing":5,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":712,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.290018+0000","last_change":"2024-02-10T08:47:18.043795+0000","last_active":"2024-02-10T08:47:50.290018+0000","last_peered":"2024-02-10T08:47:50.290018+0000","last_clean":"2024-02-10T08:47:50.290018+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:47:50.290018+0000","last_undegraded":"2024-02-10T08:47:50.290018+0000","last_fullsized":"2024-02-10T08:47:50.290018+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:18.043731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.043731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":18,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.242008+0000","last_change":"2024-02-10T08:47:49.933829+0000","last_active":"2024-02-10T08:47:50.242008+0000","last_peered":"2024-02-10T08:47:50.242008+0000","last_clean":"2024-02-10T08:47:50.242008+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:47:50.242008+0000","last_undegraded":"2024-02-10T08:47:50.242008+0000","last_fullsized":"2024-02-10T08:47:50.242008+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.043513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":94,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":265,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.201004+0000","last_change":"2024-02-10T08:47:12.548298+0000","last_active":"2024-02-10T08:47:54.201004+0000","last_peered":"2024-02-10T08:47:54.201004+0000","last_clean":"2024-02-10T08:47:54.201004+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:47:54.201004+0000","last_undegraded":"2024-02-10T08:47:54.201004+0000","last_fullsized":"2024-02-10T08:47:54.201004+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.548156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:34:51.910184+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.548156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.215289+0000","last_change":"2024-02-10T08:47:49.236998+0000","last_active":"2024-02-10T08:47:50.215289+0000","last_peered":"2024-02-10T08:47:50.215289+0000","last_clean":"2024-02-10T08:47:50.215289+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:47:50.215289+0000","last_undegraded":"2024-02-10T08:47:50.215289+0000","last_fullsized":"2024-02-10T08:47:50.215289+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.556602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":131,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969102+0000","last_change":"2024-02-10T08:47:49.242555+0000","last_active":"2024-02-10T08:47:54.969102+0000","last_peered":"2024-02-10T08:47:54.969102+0000","last_clean":"2024-02-10T08:47:54.969102+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:47:54.969102+0000","last_undegraded":"2024-02-10T08:47:54.969102+0000","last_fullsized":"2024-02-10T08:47:54.969102+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.054857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"676'42","reported_seq":23,"reported_epoch":710,"state":"active+recovering","last_fresh":"2024-02-10T08:47:51.146851+0000","last_change":"2024-02-10T08:47:49.261420+0000","last_active":"2024-02-10T08:47:51.146851+0000","last_peered":"2024-02-10T08:47:51.146851+0000","last_clean":"2024-02-10T08:47:47.188585+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:47:51.146851+0000","last_undegraded":"2024-02-10T08:47:51.146851+0000","last_fullsized":"2024-02-10T08:47:51.146851+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'41","last_scrub_stamp":"2024-02-10T08:46:03.679201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:03.679201+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"1,2,3","objects":6},{"shards":"2,3,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":705,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966818+0000","last_change":"2024-02-10T08:47:49.238540+0000","last_active":"2024-02-10T08:47:54.966818+0000","last_peered":"2024-02-10T08:47:54.966818+0000","last_clean":"2024-02-10T08:47:54.966818+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:47:54.966818+0000","last_undegraded":"2024-02-10T08:47:54.966818+0000","last_fullsized":"2024-02-10T08:47:54.966818+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:54.693153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:45:54.693153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":707,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161491+0000","last_change":"2024-02-10T08:46:33.460971+0000","last_active":"2024-02-10T08:47:53.161491+0000","last_peered":"2024-02-10T08:47:53.161491+0000","last_clean":"2024-02-10T08:47:53.161491+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:47:53.161491+0000","last_undegraded":"2024-02-10T08:47:53.161491+0000","last_fullsized":"2024-02-10T08:47:53.161491+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:33.460870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:33.460870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":671,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289294+0000","last_change":"2024-02-10T08:46:43.468542+0000","last_active":"2024-02-10T08:47:50.289294+0000","last_peered":"2024-02-10T08:47:50.289294+0000","last_clean":"2024-02-10T08:47:50.289294+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:47:50.289294+0000","last_undegraded":"2024-02-10T08:47:50.289294+0000","last_fullsized":"2024-02-10T08:47:50.289294+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.468478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.468478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":126,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968626+0000","last_change":"2024-02-10T08:47:18.403493+0000","last_active":"2024-02-10T08:47:54.968626+0000","last_peered":"2024-02-10T08:47:54.968626+0000","last_clean":"2024-02-10T08:47:54.968626+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:47:54.968626+0000","last_undegraded":"2024-02-10T08:47:54.968626+0000","last_fullsized":"2024-02-10T08:47:54.968626+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:47:18.403400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.403400+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":15,"reported_epoch":711,"state":"active+recovering","last_fresh":"2024-02-10T08:47:51.152775+0000","last_change":"2024-02-10T08:47:50.897357+0000","last_active":"2024-02-10T08:47:51.152775+0000","last_peered":"2024-02-10T08:47:51.152775+0000","last_clean":"2024-02-10T08:43:25.750254+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:47:51.152775+0000","last_undegraded":"2024-02-10T08:47:51.152775+0000","last_fullsized":"2024-02-10T08:47:51.152775+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:46:45.596434+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.596434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3},{"shards":"1,4,5","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.216908+0000","last_change":"2024-02-10T08:47:49.256946+0000","last_active":"2024-02-10T08:47:50.216908+0000","last_peered":"2024-02-10T08:47:50.216908+0000","last_clean":"2024-02-10T08:47:50.216908+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:47:50.216908+0000","last_undegraded":"2024-02-10T08:47:50.216908+0000","last_fullsized":"2024-02-10T08:47:50.216908+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.363337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.363337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968329+0000","last_change":"2024-02-10T08:46:46.046842+0000","last_active":"2024-02-10T08:47:54.968329+0000","last_peered":"2024-02-10T08:47:54.968329+0000","last_clean":"2024-02-10T08:47:54.968329+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:47:54.968329+0000","last_undegraded":"2024-02-10T08:47:54.968329+0000","last_fullsized":"2024-02-10T08:47:54.968329+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:46.046777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.046777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":647,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161185+0000","last_change":"2024-02-10T08:47:49.225418+0000","last_active":"2024-02-10T08:47:53.161185+0000","last_peered":"2024-02-10T08:47:53.161185+0000","last_clean":"2024-02-10T08:47:53.161185+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:47:53.161185+0000","last_undegraded":"2024-02-10T08:47:53.161185+0000","last_fullsized":"2024-02-10T08:47:53.161185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:57.464141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:45:57.464141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":742,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289495+0000","last_change":"2024-02-10T08:47:49.237095+0000","last_active":"2024-02-10T08:47:50.289495+0000","last_peered":"2024-02-10T08:47:50.289495+0000","last_clean":"2024-02-10T08:47:50.289495+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:47:50.289495+0000","last_undegraded":"2024-02-10T08:47:50.289495+0000","last_fullsized":"2024-02-10T08:47:50.289495+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:55.358058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:45:55.358058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":615,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155945+0000","last_change":"2024-02-10T08:47:50.224240+0000","last_active":"2024-02-10T08:47:51.155945+0000","last_peered":"2024-02-10T08:47:51.155945+0000","last_clean":"2024-02-10T08:47:51.155945+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:47:51.155945+0000","last_undegraded":"2024-02-10T08:47:51.155945+0000","last_fullsized":"2024-02-10T08:47:51.155945+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:46:40.895770+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.895770+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":585,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.160536+0000","last_change":"2024-02-10T08:47:17.985317+0000","last_active":"2024-02-10T08:47:51.160536+0000","last_peered":"2024-02-10T08:47:51.160536+0000","last_clean":"2024-02-10T08:47:51.160536+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:47:51.160536+0000","last_undegraded":"2024-02-10T08:47:51.160536+0000","last_fullsized":"2024-02-10T08:47:51.160536+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:47:17.985235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.985235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":715,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160778+0000","last_change":"2024-02-10T08:47:49.226978+0000","last_active":"2024-02-10T08:47:53.160778+0000","last_peered":"2024-02-10T08:47:53.160778+0000","last_clean":"2024-02-10T08:47:53.160778+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:47:53.160778+0000","last_undegraded":"2024-02-10T08:47:53.160778+0000","last_fullsized":"2024-02-10T08:47:53.160778+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:19.465638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:19.465638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":34,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966769+0000","last_change":"2024-02-10T08:47:49.238699+0000","last_active":"2024-02-10T08:47:54.966769+0000","last_peered":"2024-02-10T08:47:54.966769+0000","last_clean":"2024-02-10T08:47:54.966769+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:47:54.966769+0000","last_undegraded":"2024-02-10T08:47:54.966769+0000","last_fullsized":"2024-02-10T08:47:54.966769+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:09.433145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.433145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":111,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969359+0000","last_change":"2024-02-10T08:47:16.370246+0000","last_active":"2024-02-10T08:47:54.969359+0000","last_peered":"2024-02-10T08:47:54.969359+0000","last_clean":"2024-02-10T08:47:54.969359+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:47:54.969359+0000","last_undegraded":"2024-02-10T08:47:54.969359+0000","last_fullsized":"2024-02-10T08:47:54.969359+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:16.370183+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.370183+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":729,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289425+0000","last_change":"2024-02-10T08:47:49.236909+0000","last_active":"2024-02-10T08:47:50.289425+0000","last_peered":"2024-02-10T08:47:50.289425+0000","last_clean":"2024-02-10T08:47:50.289425+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:47:50.289425+0000","last_undegraded":"2024-02-10T08:47:50.289425+0000","last_fullsized":"2024-02-10T08:47:50.289425+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:28.377904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:28.377904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":714,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291652+0000","last_change":"2024-02-10T08:46:38.449729+0000","last_active":"2024-02-10T08:47:50.291652+0000","last_peered":"2024-02-10T08:47:50.291652+0000","last_clean":"2024-02-10T08:47:50.291652+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:47:50.291652+0000","last_undegraded":"2024-02-10T08:47:50.291652+0000","last_fullsized":"2024-02-10T08:47:50.291652+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:38.449611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.449611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":508,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.156264+0000","last_change":"2024-02-10T08:47:49.241750+0000","last_active":"2024-02-10T08:47:51.156264+0000","last_peered":"2024-02-10T08:47:51.156264+0000","last_clean":"2024-02-10T08:47:51.156264+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:47:51.156264+0000","last_undegraded":"2024-02-10T08:47:51.156264+0000","last_fullsized":"2024-02-10T08:47:51.156264+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:47.845659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:47.845659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":769,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161402+0000","last_change":"2024-02-10T08:47:49.978588+0000","last_active":"2024-02-10T08:47:53.161402+0000","last_peered":"2024-02-10T08:47:53.161402+0000","last_clean":"2024-02-10T08:47:53.161402+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:47:53.161402+0000","last_undegraded":"2024-02-10T08:47:53.161402+0000","last_fullsized":"2024-02-10T08:47:53.161402+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:46:49.515525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.515525+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":717,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.390936+0000","last_change":"2024-02-10T08:47:49.242356+0000","last_active":"2024-02-10T08:47:54.390936+0000","last_peered":"2024-02-10T08:47:54.390936+0000","last_clean":"2024-02-10T08:47:54.390936+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:47:54.390936+0000","last_undegraded":"2024-02-10T08:47:54.390936+0000","last_fullsized":"2024-02-10T08:47:54.390936+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.577220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.577220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":276,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200878+0000","last_change":"2024-02-10T08:46:36.562444+0000","last_active":"2024-02-10T08:47:54.200878+0000","last_peered":"2024-02-10T08:47:54.200878+0000","last_clean":"2024-02-10T08:47:54.200878+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:47:54.200878+0000","last_undegraded":"2024-02-10T08:47:54.200878+0000","last_fullsized":"2024-02-10T08:47:54.200878+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:36.562328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.562328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":640,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967156+0000","last_change":"2024-02-10T08:46:31.823967+0000","last_active":"2024-02-10T08:47:54.967156+0000","last_peered":"2024-02-10T08:47:54.967156+0000","last_clean":"2024-02-10T08:47:54.967156+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:47:54.967156+0000","last_undegraded":"2024-02-10T08:47:54.967156+0000","last_fullsized":"2024-02-10T08:47:54.967156+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:31.823902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:31.823902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":797,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161500+0000","last_change":"2024-02-10T08:47:49.996933+0000","last_active":"2024-02-10T08:47:53.161500+0000","last_peered":"2024-02-10T08:47:53.161500+0000","last_clean":"2024-02-10T08:47:53.161500+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:47:53.161500+0000","last_undegraded":"2024-02-10T08:47:53.161500+0000","last_fullsized":"2024-02-10T08:47:53.161500+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:45:58.480952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:58.480952+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":558,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160364+0000","last_change":"2024-02-10T08:47:49.225808+0000","last_active":"2024-02-10T08:47:53.160364+0000","last_peered":"2024-02-10T08:47:53.160364+0000","last_clean":"2024-02-10T08:47:53.160364+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:47:53.160364+0000","last_undegraded":"2024-02-10T08:47:53.160364+0000","last_fullsized":"2024-02-10T08:47:53.160364+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:04.536314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:04.536314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":919,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966960+0000","last_change":"2024-02-10T08:47:49.363312+0000","last_active":"2024-02-10T08:47:54.966960+0000","last_peered":"2024-02-10T08:47:54.966960+0000","last_clean":"2024-02-10T08:47:54.966960+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:47:54.966960+0000","last_undegraded":"2024-02-10T08:47:54.966960+0000","last_fullsized":"2024-02-10T08:47:54.966960+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.886696+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969284+0000","last_change":"2024-02-10T08:47:10.260148+0000","last_active":"2024-02-10T08:47:54.969284+0000","last_peered":"2024-02-10T08:47:54.969284+0000","last_clean":"2024-02-10T08:47:54.969284+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:47:54.969284+0000","last_undegraded":"2024-02-10T08:47:54.969284+0000","last_fullsized":"2024-02-10T08:47:54.969284+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.260061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.260061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968557+0000","last_change":"2024-02-10T08:46:48.087294+0000","last_active":"2024-02-10T08:47:54.968557+0000","last_peered":"2024-02-10T08:47:54.968557+0000","last_clean":"2024-02-10T08:47:54.968557+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:47:54.968557+0000","last_undegraded":"2024-02-10T08:47:54.968557+0000","last_fullsized":"2024-02-10T08:47:54.968557+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:48.087228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.087228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":748,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966854+0000","last_change":"2024-02-10T08:47:24.808523+0000","last_active":"2024-02-10T08:47:54.966854+0000","last_peered":"2024-02-10T08:47:54.966854+0000","last_clean":"2024-02-10T08:47:54.966854+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:47:54.966854+0000","last_undegraded":"2024-02-10T08:47:54.966854+0000","last_fullsized":"2024-02-10T08:47:54.966854+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:24.808424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.808424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":453,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155170+0000","last_change":"2024-02-10T08:46:30.196752+0000","last_active":"2024-02-10T08:47:51.155170+0000","last_peered":"2024-02-10T08:47:51.155170+0000","last_clean":"2024-02-10T08:47:51.155170+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:47:51.155170+0000","last_undegraded":"2024-02-10T08:47:51.155170+0000","last_fullsized":"2024-02-10T08:47:51.155170+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:14.449994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:45:14.449994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":891,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161647+0000","last_change":"2024-02-10T08:46:08.457332+0000","last_active":"2024-02-10T08:47:53.161647+0000","last_peered":"2024-02-10T08:47:53.161647+0000","last_clean":"2024-02-10T08:47:53.161647+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:47:53.161647+0000","last_undegraded":"2024-02-10T08:47:53.161647+0000","last_fullsized":"2024-02-10T08:47:53.161647+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:46:08.457270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.457270+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":520,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.160521+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:47:51.160521+0000","last_peered":"2024-02-10T08:47:51.160521+0000","last_clean":"2024-02-10T08:47:51.160521+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:47:51.160521+0000","last_undegraded":"2024-02-10T08:47:51.160521+0000","last_fullsized":"2024-02-10T08:47:51.160521+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":674,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289361+0000","last_change":"2024-02-10T08:47:49.268597+0000","last_active":"2024-02-10T08:47:50.289361+0000","last_peered":"2024-02-10T08:47:50.289361+0000","last_clean":"2024-02-10T08:47:50.289361+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:47:50.289361+0000","last_undegraded":"2024-02-10T08:47:50.289361+0000","last_fullsized":"2024-02-10T08:47:50.289361+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:47:19.053931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.053931+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":22,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.215018+0000","last_change":"2024-02-10T08:47:49.932955+0000","last_active":"2024-02-10T08:47:50.215018+0000","last_peered":"2024-02-10T08:47:50.215018+0000","last_clean":"2024-02-10T08:47:50.215018+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:47:50.215018+0000","last_undegraded":"2024-02-10T08:47:50.215018+0000","last_fullsized":"2024-02-10T08:47:50.215018+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.671541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":694,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391058+0000","last_change":"2024-02-10T08:47:12.565082+0000","last_active":"2024-02-10T08:47:54.391058+0000","last_peered":"2024-02-10T08:47:54.391058+0000","last_clean":"2024-02-10T08:47:54.391058+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:47:54.391058+0000","last_undegraded":"2024-02-10T08:47:54.391058+0000","last_fullsized":"2024-02-10T08:47:54.391058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.565015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.565015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":722,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-10T08:47:55.934857+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:47:55.934857+0000","last_peered":"2024-02-10T08:47:55.934857+0000","last_clean":"2024-02-10T08:47:55.934857+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:47:55.934857+0000","last_undegraded":"2024-02-10T08:47:55.934857+0000","last_fullsized":"2024-02-10T08:47:55.934857+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":120,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968420+0000","last_change":"2024-02-10T08:47:49.241459+0000","last_active":"2024-02-10T08:47:54.968420+0000","last_peered":"2024-02-10T08:47:54.968420+0000","last_clean":"2024-02-10T08:47:54.968420+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:47:54.968420+0000","last_undegraded":"2024-02-10T08:47:54.968420+0000","last_fullsized":"2024-02-10T08:47:54.968420+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.257566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.257566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":657,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391535+0000","last_change":"2024-02-10T08:47:25.754309+0000","last_active":"2024-02-10T08:47:54.391535+0000","last_peered":"2024-02-10T08:47:54.391535+0000","last_clean":"2024-02-10T08:47:54.391535+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:47:54.391535+0000","last_undegraded":"2024-02-10T08:47:54.391535+0000","last_fullsized":"2024-02-10T08:47:54.391535+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.754229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.754229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":611,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155230+0000","last_change":"2024-02-10T08:47:49.793777+0000","last_active":"2024-02-10T08:47:51.155230+0000","last_peered":"2024-02-10T08:47:51.155230+0000","last_clean":"2024-02-10T08:47:51.155230+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:47:51.155230+0000","last_undegraded":"2024-02-10T08:47:51.155230+0000","last_fullsized":"2024-02-10T08:47:51.155230+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:19.946402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.946402+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.218497+0000","last_change":"2024-02-10T08:47:49.251648+0000","last_active":"2024-02-10T08:47:50.218497+0000","last_peered":"2024-02-10T08:47:50.218497+0000","last_clean":"2024-02-10T08:47:50.218497+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:47:50.218497+0000","last_undegraded":"2024-02-10T08:47:50.218497+0000","last_fullsized":"2024-02-10T08:47:50.218497+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:28.640899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:45:28.640899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.215104+0000","last_change":"2024-02-10T08:47:49.222892+0000","last_active":"2024-02-10T08:47:50.215104+0000","last_peered":"2024-02-10T08:47:50.215104+0000","last_clean":"2024-02-10T08:47:50.215104+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:47:50.215104+0000","last_undegraded":"2024-02-10T08:47:50.215104+0000","last_fullsized":"2024-02-10T08:47:50.215104+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:00.878280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:46:00.878280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":697,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160116+0000","last_change":"2024-02-10T08:46:15.350532+0000","last_active":"2024-02-10T08:47:53.160116+0000","last_peered":"2024-02-10T08:47:53.160116+0000","last_clean":"2024-02-10T08:47:53.160116+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:47:53.160116+0000","last_undegraded":"2024-02-10T08:47:53.160116+0000","last_fullsized":"2024-02-10T08:47:53.160116+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:55.477897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:55.477897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":653,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161266+0000","last_change":"2024-02-10T08:46:41.501224+0000","last_active":"2024-02-10T08:47:53.161266+0000","last_peered":"2024-02-10T08:47:53.161266+0000","last_clean":"2024-02-10T08:47:53.161266+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:47:53.161266+0000","last_undegraded":"2024-02-10T08:47:53.161266+0000","last_fullsized":"2024-02-10T08:47:53.161266+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.501157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.501157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":725,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.159918+0000","last_change":"2024-02-10T08:47:49.262275+0000","last_active":"2024-02-10T08:47:53.159918+0000","last_peered":"2024-02-10T08:47:53.159918+0000","last_clean":"2024-02-10T08:47:53.159918+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:47:53.159918+0000","last_undegraded":"2024-02-10T08:47:53.159918+0000","last_fullsized":"2024-02-10T08:47:53.159918+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:46:48.511488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.511488+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":691,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.967333+0000","last_change":"2024-02-10T08:46:08.732146+0000","last_active":"2024-02-10T08:47:54.967333+0000","last_peered":"2024-02-10T08:47:54.967333+0000","last_clean":"2024-02-10T08:47:54.967333+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:47:54.967333+0000","last_undegraded":"2024-02-10T08:47:54.967333+0000","last_fullsized":"2024-02-10T08:47:54.967333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:08.732073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.732073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":715,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.390755+0000","last_change":"2024-02-10T08:46:41.654802+0000","last_active":"2024-02-10T08:47:54.390755+0000","last_peered":"2024-02-10T08:47:54.390755+0000","last_clean":"2024-02-10T08:47:54.390755+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:47:54.390755+0000","last_undegraded":"2024-02-10T08:47:54.390755+0000","last_fullsized":"2024-02-10T08:47:54.390755+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.654732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.654732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":276,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200476+0000","last_change":"2024-02-10T08:46:32.419160+0000","last_active":"2024-02-10T08:47:54.200476+0000","last_peered":"2024-02-10T08:47:54.200476+0000","last_clean":"2024-02-10T08:47:54.200476+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:47:54.200476+0000","last_undegraded":"2024-02-10T08:47:54.200476+0000","last_fullsized":"2024-02-10T08:47:54.200476+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:32.419097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:32.419097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":665,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161295+0000","last_change":"2024-02-10T08:47:49.232286+0000","last_active":"2024-02-10T08:47:53.161295+0000","last_peered":"2024-02-10T08:47:53.161295+0000","last_clean":"2024-02-10T08:47:53.161295+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:47:53.161295+0000","last_undegraded":"2024-02-10T08:47:53.161295+0000","last_fullsized":"2024-02-10T08:47:53.161295+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.529230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.529230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":756,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160366+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:47:53.160366+0000","last_peered":"2024-02-10T08:47:53.160366+0000","last_clean":"2024-02-10T08:47:53.160366+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:47:53.160366+0000","last_undegraded":"2024-02-10T08:47:53.160366+0000","last_fullsized":"2024-02-10T08:47:53.160366+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":694,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391453+0000","last_change":"2024-02-10T08:47:10.557902+0000","last_active":"2024-02-10T08:47:54.391453+0000","last_peered":"2024-02-10T08:47:54.391453+0000","last_clean":"2024-02-10T08:47:54.391453+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:47:54.391453+0000","last_undegraded":"2024-02-10T08:47:54.391453+0000","last_fullsized":"2024-02-10T08:47:54.391453+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.557838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.557838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":276,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200356+0000","last_change":"2024-02-10T08:46:39.567932+0000","last_active":"2024-02-10T08:47:54.200356+0000","last_peered":"2024-02-10T08:47:54.200356+0000","last_clean":"2024-02-10T08:47:54.200356+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:47:54.200356+0000","last_undegraded":"2024-02-10T08:47:54.200356+0000","last_fullsized":"2024-02-10T08:47:54.200356+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.567819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":697,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160817+0000","last_change":"2024-02-10T08:46:15.348134+0000","last_active":"2024-02-10T08:47:53.160817+0000","last_peered":"2024-02-10T08:47:53.160817+0000","last_clean":"2024-02-10T08:47:53.160817+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:47:53.160817+0000","last_undegraded":"2024-02-10T08:47:53.160817+0000","last_fullsized":"2024-02-10T08:47:53.160817+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:05.554604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:05.554604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969024+0000","last_change":"2024-02-10T08:47:29.348776+0000","last_active":"2024-02-10T08:47:54.969024+0000","last_peered":"2024-02-10T08:47:54.969024+0000","last_clean":"2024-02-10T08:47:54.969024+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:47:54.969024+0000","last_undegraded":"2024-02-10T08:47:54.969024+0000","last_fullsized":"2024-02-10T08:47:54.969024+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.348714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":109,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969195+0000","last_change":"2024-02-10T08:46:15.400959+0000","last_active":"2024-02-10T08:47:54.969195+0000","last_peered":"2024-02-10T08:47:54.969195+0000","last_clean":"2024-02-10T08:47:54.969195+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:47:54.969195+0000","last_undegraded":"2024-02-10T08:47:54.969195+0000","last_fullsized":"2024-02-10T08:47:54.969195+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:46:09.425164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.425164+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":729,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966561+0000","last_change":"2024-02-10T08:47:12.832224+0000","last_active":"2024-02-10T08:47:54.966561+0000","last_peered":"2024-02-10T08:47:54.966561+0000","last_clean":"2024-02-10T08:47:54.966561+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:47:54.966561+0000","last_undegraded":"2024-02-10T08:47:54.966561+0000","last_fullsized":"2024-02-10T08:47:54.966561+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.832146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.832146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"63'1","reported_seq":275,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200684+0000","last_change":"2024-02-10T08:47:23.598866+0000","last_active":"2024-02-10T08:47:54.200684+0000","last_peered":"2024-02-10T08:47:54.200684+0000","last_clean":"2024-02-10T08:47:54.200684+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:47:54.200684+0000","last_undegraded":"2024-02-10T08:47:54.200684+0000","last_fullsized":"2024-02-10T08:47:54.200684+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'1","last_scrub_stamp":"2024-02-10T08:47:23.598783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.598783+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":671,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.289739+0000","last_change":"2024-02-10T08:47:49.229890+0000","last_active":"2024-02-10T08:47:50.289739+0000","last_peered":"2024-02-10T08:47:50.289739+0000","last_clean":"2024-02-10T08:47:50.289739+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:47:50.289739+0000","last_undegraded":"2024-02-10T08:47:50.289739+0000","last_fullsized":"2024-02-10T08:47:50.289739+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.149802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.149802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":765,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391536+0000","last_change":"2024-02-10T08:47:49.276612+0000","last_active":"2024-02-10T08:47:54.391536+0000","last_peered":"2024-02-10T08:47:54.391536+0000","last_clean":"2024-02-10T08:47:54.391536+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:47:54.391536+0000","last_undegraded":"2024-02-10T08:47:54.391536+0000","last_fullsized":"2024-02-10T08:47:54.391536+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:46:10.495154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:10.495154+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":703,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160212+0000","last_change":"2024-02-10T08:46:15.350704+0000","last_active":"2024-02-10T08:47:53.160212+0000","last_peered":"2024-02-10T08:47:53.160212+0000","last_clean":"2024-02-10T08:47:53.160212+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:47:53.160212+0000","last_undegraded":"2024-02-10T08:47:53.160212+0000","last_fullsized":"2024-02-10T08:47:53.160212+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:54.474804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:45:54.474804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.218418+0000","last_change":"2024-02-10T08:47:49.249250+0000","last_active":"2024-02-10T08:47:50.218418+0000","last_peered":"2024-02-10T08:47:50.218418+0000","last_clean":"2024-02-10T08:47:50.218418+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:47:50.218418+0000","last_undegraded":"2024-02-10T08:47:50.218418+0000","last_fullsized":"2024-02-10T08:47:50.218418+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:40.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.685831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.215367+0000","last_change":"2024-02-10T08:47:49.235376+0000","last_active":"2024-02-10T08:47:50.215367+0000","last_peered":"2024-02-10T08:47:50.215367+0000","last_clean":"2024-02-10T08:47:50.215367+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:47:50.215367+0000","last_undegraded":"2024-02-10T08:47:50.215367+0000","last_fullsized":"2024-02-10T08:47:50.215367+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:30.717295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.717295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":21,"reported_epoch":710,"state":"active+recovering","last_fresh":"2024-02-10T08:47:51.060919+0000","last_change":"2024-02-10T08:47:49.342317+0000","last_active":"2024-02-10T08:47:51.060919+0000","last_peered":"2024-02-10T08:47:51.060919+0000","last_clean":"2024-02-10T08:43:31.060512+0000","last_became_active":"2024-02-10T08:47:49.229342+0000","last_became_peered":"2024-02-10T08:47:49.229342+0000","last_unstale":"2024-02-10T08:47:51.060919+0000","last_undegraded":"2024-02-10T08:47:51.060919+0000","last_fullsized":"2024-02-10T08:47:51.060919+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:46:36.688435+0000","last_deep_scrub":"174'22","last_deep_scrub_stamp":"2024-02-10T08:37:48.901981+0000","last_clean_scrub_stamp":"2024-02-10T08:46:36.688435+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,1,3","objects":5},{"shards":"0,3,4","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":717,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391059+0000","last_change":"2024-02-10T08:46:39.661813+0000","last_active":"2024-02-10T08:47:54.391059+0000","last_peered":"2024-02-10T08:47:54.391059+0000","last_clean":"2024-02-10T08:47:54.391059+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:47:54.391059+0000","last_undegraded":"2024-02-10T08:47:54.391059+0000","last_fullsized":"2024-02-10T08:47:54.391059+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.661744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.661744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":526,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.160287+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:47:51.160287+0000","last_peered":"2024-02-10T08:47:51.160287+0000","last_clean":"2024-02-10T08:47:51.160287+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:47:51.160287+0000","last_undegraded":"2024-02-10T08:47:51.160287+0000","last_fullsized":"2024-02-10T08:47:51.160287+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":120,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968454+0000","last_change":"2024-02-10T08:47:49.241622+0000","last_active":"2024-02-10T08:47:54.968454+0000","last_peered":"2024-02-10T08:47:54.968454+0000","last_clean":"2024-02-10T08:47:54.968454+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:47:54.968454+0000","last_undegraded":"2024-02-10T08:47:54.968454+0000","last_fullsized":"2024-02-10T08:47:54.968454+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:27.336914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.336914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":122,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161010+0000","last_change":"2024-02-10T08:47:49.265858+0000","last_active":"2024-02-10T08:47:53.161010+0000","last_peered":"2024-02-10T08:47:53.161010+0000","last_clean":"2024-02-10T08:47:53.161010+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:47:53.161010+0000","last_undegraded":"2024-02-10T08:47:53.161010+0000","last_fullsized":"2024-02-10T08:47:53.161010+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:46:43.848897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.848897+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":703,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160703+0000","last_change":"2024-02-10T08:45:11.455867+0000","last_active":"2024-02-10T08:47:53.160703+0000","last_peered":"2024-02-10T08:47:53.160703+0000","last_clean":"2024-02-10T08:47:53.160703+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:47:53.160703+0000","last_undegraded":"2024-02-10T08:47:53.160703+0000","last_fullsized":"2024-02-10T08:47:53.160703+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:11.455805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:45:11.455805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":679,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161525+0000","last_change":"2024-02-10T08:46:07.493242+0000","last_active":"2024-02-10T08:47:53.161525+0000","last_peered":"2024-02-10T08:47:53.161525+0000","last_clean":"2024-02-10T08:47:53.161525+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:47:53.161525+0000","last_undegraded":"2024-02-10T08:47:53.161525+0000","last_fullsized":"2024-02-10T08:47:53.161525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:07.493170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:07.493170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.214296+0000","last_change":"2024-02-10T08:47:49.223545+0000","last_active":"2024-02-10T08:47:50.214296+0000","last_peered":"2024-02-10T08:47:50.214296+0000","last_clean":"2024-02-10T08:47:50.214296+0000","last_became_active":"2024-02-10T08:47:49.222511+0000","last_became_peered":"2024-02-10T08:47:49.222511+0000","last_unstale":"2024-02-10T08:47:50.214296+0000","last_undegraded":"2024-02-10T08:47:50.214296+0000","last_fullsized":"2024-02-10T08:47:50.214296+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:14.568443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:47:14.568443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":578,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.156448+0000","last_change":"2024-02-10T08:46:30.950978+0000","last_active":"2024-02-10T08:47:51.156448+0000","last_peered":"2024-02-10T08:47:51.156448+0000","last_clean":"2024-02-10T08:47:51.156448+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:47:51.156448+0000","last_undegraded":"2024-02-10T08:47:51.156448+0000","last_fullsized":"2024-02-10T08:47:51.156448+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:46:30.950919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.950919+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":273,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.200720+0000","last_change":"2024-02-10T08:47:19.521917+0000","last_active":"2024-02-10T08:47:54.200720+0000","last_peered":"2024-02-10T08:47:54.200720+0000","last_clean":"2024-02-10T08:47:54.200720+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:47:54.200720+0000","last_undegraded":"2024-02-10T08:47:54.200720+0000","last_fullsized":"2024-02-10T08:47:54.200720+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:19.521845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.521845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":120,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.968503+0000","last_change":"2024-02-10T08:47:49.241831+0000","last_active":"2024-02-10T08:47:54.968503+0000","last_peered":"2024-02-10T08:47:54.968503+0000","last_clean":"2024-02-10T08:47:54.968503+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:47:54.968503+0000","last_undegraded":"2024-02-10T08:47:54.968503+0000","last_fullsized":"2024-02-10T08:47:54.968503+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:24.365335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.365335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":665,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-02-10T08:47:50.291312+0000","last_change":"2024-02-10T08:46:37.473469+0000","last_active":"2024-02-10T08:47:50.291312+0000","last_peered":"2024-02-10T08:47:50.291312+0000","last_clean":"2024-02-10T08:47:50.291312+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:47:50.291312+0000","last_undegraded":"2024-02-10T08:47:50.291312+0000","last_fullsized":"2024-02-10T08:47:50.291312+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:37.473394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:37.473394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":581,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.155433+0000","last_change":"2024-02-10T08:47:50.829067+0000","last_active":"2024-02-10T08:47:51.155433+0000","last_peered":"2024-02-10T08:47:51.155433+0000","last_clean":"2024-02-10T08:47:51.155433+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:47:51.155433+0000","last_undegraded":"2024-02-10T08:47:51.155433+0000","last_fullsized":"2024-02-10T08:47:51.155433+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:46:46.889769+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:46:46.889769+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":215,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161009+0000","last_change":"2024-02-10T08:46:30.198693+0000","last_active":"2024-02-10T08:47:53.161009+0000","last_peered":"2024-02-10T08:47:53.161009+0000","last_clean":"2024-02-10T08:47:53.161009+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:47:53.161009+0000","last_undegraded":"2024-02-10T08:47:53.161009+0000","last_fullsized":"2024-02-10T08:47:53.161009+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:13.431922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:45:13.431922+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":694,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966983+0000","last_change":"2024-02-10T08:47:49.237408+0000","last_active":"2024-02-10T08:47:54.966983+0000","last_peered":"2024-02-10T08:47:54.966983+0000","last_clean":"2024-02-10T08:47:54.966983+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:47:54.966983+0000","last_undegraded":"2024-02-10T08:47:54.966983+0000","last_fullsized":"2024-02-10T08:47:54.966983+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:45:25.022136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:45:25.022136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":101,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969146+0000","last_change":"2024-02-10T08:47:15.381535+0000","last_active":"2024-02-10T08:47:54.969146+0000","last_peered":"2024-02-10T08:47:54.969146+0000","last_clean":"2024-02-10T08:47:54.969146+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:47:54.969146+0000","last_undegraded":"2024-02-10T08:47:54.969146+0000","last_fullsized":"2024-02-10T08:47:54.969146+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.381441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.381441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":618,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.154888+0000","last_change":"2024-02-10T08:47:50.341317+0000","last_active":"2024-02-10T08:47:51.154888+0000","last_peered":"2024-02-10T08:47:51.154888+0000","last_clean":"2024-02-10T08:47:51.154888+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:47:51.154888+0000","last_undegraded":"2024-02-10T08:47:51.154888+0000","last_fullsized":"2024-02-10T08:47:51.154888+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:21.938274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.938274+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":703,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.161415+0000","last_change":"2024-02-10T08:46:43.502472+0000","last_active":"2024-02-10T08:47:53.161415+0000","last_peered":"2024-02-10T08:47:53.161415+0000","last_clean":"2024-02-10T08:47:53.161415+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:47:53.161415+0000","last_undegraded":"2024-02-10T08:47:53.161415+0000","last_fullsized":"2024-02-10T08:47:53.161415+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.502412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.502412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":536,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-02-10T08:47:51.156042+0000","last_change":"2024-02-10T08:47:49.241641+0000","last_active":"2024-02-10T08:47:51.156042+0000","last_peered":"2024-02-10T08:47:51.156042+0000","last_clean":"2024-02-10T08:47:51.156042+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:47:51.156042+0000","last_undegraded":"2024-02-10T08:47:51.156042+0000","last_fullsized":"2024-02-10T08:47:51.156042+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.805860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.805860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":660,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.966847+0000","last_change":"2024-02-10T08:46:30.825307+0000","last_active":"2024-02-10T08:47:54.966847+0000","last_peered":"2024-02-10T08:47:54.966847+0000","last_clean":"2024-02-10T08:47:54.966847+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:47:54.966847+0000","last_undegraded":"2024-02-10T08:47:54.966847+0000","last_fullsized":"2024-02-10T08:47:54.966847+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:30.825230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:30.825230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":776,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160374+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:47:53.160374+0000","last_peered":"2024-02-10T08:47:53.160374+0000","last_clean":"2024-02-10T08:47:53.160374+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:47:53.160374+0000","last_undegraded":"2024-02-10T08:47:53.160374+0000","last_fullsized":"2024-02-10T08:47:53.160374+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":113,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.969237+0000","last_change":"2024-02-10T08:46:44.049731+0000","last_active":"2024-02-10T08:47:54.969237+0000","last_peered":"2024-02-10T08:47:54.969237+0000","last_clean":"2024-02-10T08:47:54.969237+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:47:54.969237+0000","last_undegraded":"2024-02-10T08:47:54.969237+0000","last_fullsized":"2024-02-10T08:47:54.969237+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.049633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":699,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-10T08:47:53.160624+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:47:53.160624+0000","last_peered":"2024-02-10T08:47:53.160624+0000","last_clean":"2024-02-10T08:47:53.160624+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:47:53.160624+0000","last_undegraded":"2024-02-10T08:47:53.160624+0000","last_fullsized":"2024-02-10T08:47:53.160624+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":660,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-10T08:47:54.391227+0000","last_change":"2024-02-10T08:46:42.142516+0000","last_active":"2024-02-10T08:47:54.391227+0000","last_peered":"2024-02-10T08:47:54.391227+0000","last_clean":"2024-02-10T08:47:54.391227+0000","last_became_active":"2024-02-10T08:46:42.142143+0000","last_became_peered":"2024-02-10T08:46:42.142143+0000","last_unstale":"2024-02-10T08:47:54.391227+0000","last_undegraded":"2024-02-10T08:47:54.391227+0000","last_fullsized":"2024-02-10T08:47:54.391227+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:35.562991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:35.562991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":246,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":5,"num_flush_kb":5,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":81,"num_objects":16,"num_object_clones":1,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":14,"num_read_kb":4,"num_write":36,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":12,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":10292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":10666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":94,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299570,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310624,"kb_used_data":5204,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434528,"statfs":{"total":95995035648,"available":95676956672,"internally_reserved":0,"allocated":5328896,"data_stored":2801118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54700000000000004,"15min":0.60899999999999999},"min":{"1min":0.35399999999999998,"5min":0.307,"15min":0.246},"max":{"1min":1.3049999999999999,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.32400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0740000000000001,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.40400000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.66600000000000004,"15min":0.60899999999999999},"min":{"1min":0.33800000000000002,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":1.333,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.68300000000000005,"15min":0.622},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.19900000000000001},"max":{"1min":1.1240000000000001,"5min":10.978,"15min":10.978},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.66000000000000003,"15min":0.60299999999999998},"min":{"1min":0.34899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.37},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.61899999999999999,"15min":0.67400000000000004},"min":{"1min":0.34599999999999997,"5min":0.28000000000000003,"15min":0.251},"max":{"1min":1.3200000000000001,"5min":7.5099999999999998,"15min":19.369},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.56299999999999994,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69399999999999995,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.69899999999999995},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":1.345,"5min":7.8700000000000001,"15min":22.102},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.47899999999999998,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.46,"5min":4.46,"15min":19.326000000000001},"last":0.317},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.8650000000000002,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.53000000000000003,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.718,"5min":4.718,"15min":22.687000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.505,"15min":0.504},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.8899999999999999,"5min":5.484,"15min":5.484},"last":0.443}]}]},{"osd":6,"up_from":44,"seq":188978561208,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302432,"kb_used_data":5204,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442720,"statfs":{"total":95995035648,"available":95685345280,"internally_reserved":0,"allocated":5328896,"data_stored":2813634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.222,"5min":0.222,"15min":0.19800000000000001},"max":{"1min":0.68300000000000005,"5min":5.5110000000000001,"15min":8.5250000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.63400000000000001},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":2.649,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.878,"15min":0.65300000000000002},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63200000000000001,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.497},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.74199999999999999,"15min":0.59499999999999997},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":2.4769999999999999,"5min":18.402999999999999,"15min":18.402999999999999},"last":0.38}]},{"osd":2,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.746,"15min":0.61299999999999999},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":2.4319999999999999,"5min":10.772,"15min":10.772},"last":0.504},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.83599999999999997,"15min":0.64400000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":2.3570000000000002,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.441}]},{"osd":3,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.79700000000000004,"15min":0.70299999999999996},"min":{"1min":0.34599999999999997,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":9.2650000000000006,"5min":9.2650000000000006,"15min":16.823},"last":0.432},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.28699999999999998},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.41899999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.628,"15min":0.504},"min":{"1min":0.27600000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.63300000000000001,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46500000000000002},"min":{"1min":0.311,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":3.915,"15min":3.915},"last":0.39500000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52100000000000002,"15min":0.501},"min":{"1min":0.34899999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.70499999999999996,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.746,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575054,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302388,"kb_used_data":5164,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442764,"statfs":{"total":95995035648,"available":95685390336,"internally_reserved":0,"allocated":5287936,"data_stored":2767259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.20899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.4810000000000001,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.189},{"interface":"front","average":{"1min":0.377,"5min":0.39700000000000002,"15min":0.46899999999999997},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.2669999999999999,"5min":2.1890000000000001,"15min":16.166},"last":0.23899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:47:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.503,"15min":0.65200000000000002},"min":{"1min":0.218,"5min":0.193,"15min":0.193},"max":{"1min":0.57699999999999996,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.47999999999999998,"15min":0.56100000000000005},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.55400000000000005,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.25600000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.46100000000000002,"15min":0.52700000000000002},"min":{"1min":0.26300000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":4.8940000000000001,"15min":24.773},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.433,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.79800000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.307}]},{"osd":4,"last update":"Sat Feb 10 08:47:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.31900000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59499999999999997,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.86899999999999999,"5min":3.496,"15min":9.5600000000000005},"last":0.372},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65800000000000003,"15min":0.66700000000000004},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.31900000000000001},"max":{"1min":0.94699999999999995,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.38}]},{"osd":6,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.79900000000000004,"5min":3.4710000000000001,"15min":9.532},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63700000000000001,"15min":0.67100000000000004},"min":{"1min":0.38300000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.36099999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:47:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.72699999999999998,"15min":0.66600000000000004},"min":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.84699999999999998,"5min":12.147,"15min":12.147},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68100000000000005,"15min":0.69899999999999995},"min":{"1min":0.44400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.90400000000000003,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.40400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302432,"kb_used_data":5208,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442720,"statfs":{"total":95995035648,"available":95685345280,"internally_reserved":0,"allocated":5332992,"data_stored":2804663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:47:14 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.46899999999999997,"15min":0.40799999999999997},"min":{"1min":0.24299999999999999,"5min":0.17799999999999999,"15min":0.156},"max":{"1min":0.371,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.70899999999999996,"15min":0.49099999999999999},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.51100000000000001,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.55500000000000005,"15min":0.46200000000000002},"min":{"1min":0.214,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.66700000000000004,"5min":11.987,"15min":11.987},"last":0.374},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.25,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.64000000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.309}]},{"osd":3,"last update":"Sat Feb 10 08:47:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.438,"15min":0.47199999999999998},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55700000000000005,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.51700000000000002,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.22600000000000001},"max":{"1min":4.0780000000000003,"5min":5.9779999999999998,"15min":5.9779999999999998},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.89600000000000002,"5min":4.609,"15min":4.609},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.57899999999999996},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":0.92300000000000004,"5min":1.7769999999999999,"15min":3.4079999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Sat Feb 10 08:47:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54500000000000004,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.747,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58399999999999996,"15min":0.61899999999999999},"min":{"1min":0.373,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73199999999999998,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:47:40 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60199999999999998,"15min":0.61899999999999999},"min":{"1min":0.39900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.871,"5min":1.849,"15min":3.363},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56799999999999995,"15min":0.65700000000000003},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.36299999999999999},"max":{"1min":0.89100000000000001,"5min":1.206,"15min":8.4299999999999997},"last":0.60199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411530,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314620,"kb_used_data":5108,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430532,"statfs":{"total":95995035648,"available":95672864768,"internally_reserved":0,"allocated":5230592,"data_stored":2753881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.64100000000000001,"15min":0.501},"min":{"1min":0.22700000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.433,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.70099999999999996,"15min":0.56799999999999995},"min":{"1min":0.254,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":1.458,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.627,"15min":0.55800000000000005},"min":{"1min":0.23499999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.38400000000000001,"5min":9.6980000000000004,"15min":12.512},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.318,"5min":0.59799999999999998,"15min":0.53500000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46200000000000002,"5min":12.875,"15min":12.875},"last":0.33600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.73499999999999999,"15min":0.54500000000000004},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47399999999999998,"5min":11.612,"15min":11.612},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.67300000000000004,"15min":0.56399999999999995},"min":{"1min":0.253,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.65600000000000003,"5min":11.710000000000001,"15min":11.710000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.69799999999999995,"15min":0.70699999999999996},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":10.34,"15min":10.34},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.71099999999999997,"15min":0.66800000000000004},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66800000000000004,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.747,"15min":0.71499999999999997},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69299999999999995,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.80400000000000005,"15min":0.71099999999999997},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69999999999999996,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.41999999999999998,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":10.539,"15min":10.539},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.71299999999999997},"min":{"1min":0.45000000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68799999999999994,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.55300000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182599,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314760,"kb_used_data":5248,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430392,"statfs":{"total":95995035648,"available":95672721408,"internally_reserved":0,"allocated":5373952,"data_stored":2825890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.219,"5min":0.184,"15min":0.184},"max":{"1min":1.8879999999999999,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.53900000000000003,"15min":0.60099999999999998},"min":{"1min":0.218,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":6.6369999999999996,"5min":10.484999999999999,"15min":11.808},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76300000000000001,"15min":0.67700000000000005},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.5810000000000004,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.215},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":2.9950000000000001,"5min":17.343,"15min":17.343},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:47:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.78000000000000003,"15min":0.68999999999999995},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":12.664999999999999,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.317},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.45800000000000002,"15min":0.63900000000000001},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":1.206,"5min":5.0259999999999998,"15min":23.294},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:47:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56000000000000005,"15min":0.70399999999999996},"min":{"1min":0.372,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80600000000000005,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.72999999999999998,"15min":0.70299999999999996},"min":{"1min":0.39100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78000000000000003,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.71399999999999997},"min":{"1min":0.36499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.244,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.73099999999999998,"15min":0.746},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79800000000000004,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.48199999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.38900000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86799999999999999,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.74299999999999999},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.498}]}]},{"osd":4,"up_from":633,"seq":2718714298386,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308520,"kb_used_data":5152,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93436632,"statfs":{"total":95995035648,"available":95679111168,"internally_reserved":0,"allocated":5275648,"data_stored":2772158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.375},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.44900000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.439}]},{"osd":3,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.499}]},{"osd":5,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.53600000000000003}]}]},{"osd":5,"up_from":37,"seq":158913790141,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306528,"kb_used_data":5208,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438624,"statfs":{"total":95995035648,"available":95681150976,"internally_reserved":0,"allocated":5332992,"data_stored":2820897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.68700000000000006,"15min":1.05},"min":{"1min":0.26100000000000001,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":1.151,"5min":8.7400000000000002,"15min":107.551},"last":0.495},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.97899999999999998},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.22900000000000001},"max":{"1min":2.2589999999999999,"5min":5.375,"15min":88.792000000000002},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.76300000000000001,"15min":1.01},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.94699999999999995,"5min":12.375999999999999,"15min":107.205},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.71199999999999997,"15min":0.88500000000000001},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":0.68100000000000005,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.441}]},{"osd":2,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.74299999999999999,"15min":1.0309999999999999},"min":{"1min":0.316,"5min":0.247,"15min":0.247},"max":{"1min":1.0549999999999999,"5min":10.794,"15min":54.174999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.748,"15min":1.0660000000000001},"min":{"1min":0.39000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.6559999999999999,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.51000000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.90900000000000003,"15min":0.85899999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73499999999999999,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79500000000000004,"15min":1.2889999999999999},"min":{"1min":0.35699999999999998,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":1.7010000000000001,"5min":8.7690000000000001,"15min":114.526},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46000000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.51000000000000001,"15min":0.71899999999999997},"min":{"1min":0.26800000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.71699999999999997,"5min":10.023999999999999,"15min":72.241},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54700000000000004,"15min":0.57799999999999996},"min":{"1min":0.29399999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":10.013,"15min":22.469999999999999},"last":0.39300000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.47399999999999998,"15min":0.71799999999999997},"min":{"1min":0.34999999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69699999999999995,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.379},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.48099999999999998,"15min":0.749},"min":{"1min":0.376,"5min":0.253,"15min":0.253},"max":{"1min":0.72799999999999998,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.640111+0000 osd.7 (osd.7) 290 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.641175+0000 osd.7 (osd.7) 291 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.661658+0000 osd.6 (osd.6) 292 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.663287+0000 osd.6 (osd.6) 293 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.937235+0000 osd.3 (osd.3) 313 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:47:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:57.938092+0000 osd.3 (osd.3) 314 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.024903+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.025035+0000 osd.0 (osd.0) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.413940+0000 mon.a (mon.0) 3408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/624 objects degraded (2.244%), 4 pgs degraded) 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:58.521858+0000 mon.b (mon.1) 799 : audit [INF] from='client.? v1:172.21.15.107:0/4263186383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:58.522230+0000 mon.a (mon.0) 3409 : audit [INF] from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.528189+0000 osd.0 (osd.0) 139 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.529472+0000 osd.0 (osd.0) 140 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:47:59.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.641057+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.641890+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:59.155068+0000 mon.a (mon.0) 3410 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:59.162250+0000 mon.a (mon.0) 3411 : audit [INF] from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:59.162323+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:59.168297+0000 mon.b (mon.1) 800 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:47:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:47:59.168897+0000 mon.a (mon.0) 3413 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:47:59.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:47:59.852 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:47:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:47:59.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:47:59.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:47:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:00.111 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:00.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.931711+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:48:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.934590+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:48:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.955736+0000 osd.5 (osd.5) 296 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:58.959267+0000 osd.5 (osd.5) 297 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:59.540189+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:59.540997+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:47:59.551641+0000 mgr.y (mgr.14553) 600 : cluster [DBG] pgmap v1088: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:00.159857+0000 mon.a (mon.0) 3414 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:00.159913+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:00.165538+0000 mon.b (mon.1) 801 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:00.166115+0000 mon.a (mon.0) 3416 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:00.570 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:00.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.931711+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:48:00.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.934590+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.955736+0000 osd.5 (osd.5) 296 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:58.959267+0000 osd.5 (osd.5) 297 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:59.540189+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:59.540997+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:47:59.551641+0000 mgr.y (mgr.14553) 600 : cluster [DBG] pgmap v1088: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:00.159857+0000 mon.a (mon.0) 3414 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:00.159913+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:00.165538+0000 mon.b (mon.1) 801 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:00.166115+0000 mon.a (mon.0) 3416 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.931711+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:48:00.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.934590+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.955736+0000 osd.5 (osd.5) 296 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:58.959267+0000 osd.5 (osd.5) 297 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:59.540189+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:59.540997+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:47:59.551641+0000 mgr.y (mgr.14553) 600 : cluster [DBG] pgmap v1088: 194 pgs: 32 unknown, 162 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 331 B/s, 5 objects/s recovering 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:00.159857+0000 mon.a (mon.0) 3414 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:00.159913+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:00.165538+0000 mon.b (mon.1) 801 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:00.166115+0000 mon.a (mon.0) 3416 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:00.918 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:00.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:01.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:01.051 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:01.200 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:01.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:01.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:01.331 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:01.404 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:01.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:01.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:01.538 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:01.576 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:01.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:01.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:01.701 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:01.701 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:01.708 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:01.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.892+0000 7fbfab99c700 1 -- 172.21.15.107:0/3784035821 >> v1:172.21.15.107:6789/0 conn(0x7fbfa40fda50 legacy=0x7fbfa40fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.892+0000 7fbfab99c700 1 -- 172.21.15.107:0/3784035821 shutdown_connections 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.892+0000 7fbfab99c700 1 -- 172.21.15.107:0/3784035821 >> 172.21.15.107:0/3784035821 conn(0x7fbfa40f3640 msgr2=0x7fbfa40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.892+0000 7fbfab99c700 1 -- 172.21.15.107:0/3784035821 shutdown_connections 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.892+0000 7fbfab99c700 1 -- 172.21.15.107:0/3784035821 wait complete. 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 librados: starting msgr at 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 librados: starting objecter 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 Processor -- start 2024-02-10T08:48:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 -- start start 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 librados: setting wanted keys 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 librados: calling monclient init 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbfa41194c0 con 0x7fbfa41007f0 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbfa411a660 con 0x7fbfa4104330 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfab99c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbfa411b800 con 0x7fbfa40fda50 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfa9738700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fbfa40fda50 0x7fbfa40fc670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36598/0 (socket says 172.21.15.107:36598) 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbfa9738700 1 -- 172.21.15.107:0/837888526 learned_addr learned my addr 172.21.15.107:0/837888526 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbf9a7fc700 10 client.?.objecter ms_handle_connect 0x7fbfa41007f0 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbf9a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbf9a7fc700 10 client.?.objecter ms_handle_connect 0x7fbfa40fda50 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbf9a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.893+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157008195 0 0) 0x7fbfa41194c0 con 0x7fbfa41007f0 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90003030 con 0x7fbfa41007f0 2024-02-10T08:48:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 10 client.?.objecter ms_handle_connect 0x7fbfa4104330 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351995361 0 0) 0x7fbfa411b800 con 0x7fbfa40fda50 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa41194c0 con 0x7fbfa40fda50 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771965731 0 0) 0x7fbfa411a660 con 0x7fbfa4104330 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa411b800 con 0x7fbfa4104330 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331461785 0 0) 0x7fbfa41194c0 con 0x7fbfa40fda50 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfa411a660 con 0x7fbfa40fda50 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbf94002e10 con 0x7fbfa40fda50 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3863757607 0 0) 0x7fbf90003030 con 0x7fbfa41007f0 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfa41194c0 con 0x7fbfa41007f0 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbfa00017e0 con 0x7fbfa41007f0 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007586139 0 0) 0x7fbfa411b800 con 0x7fbfa4104330 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf90003030 con 0x7fbfa4104330 2024-02-10T08:48:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbf9c001d70 con 0x7fbfa4104330 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 478265969 0 0) 0x7fbfa411a660 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.195:6789/0 conn(0x7fbfa4104330 legacy=0x7fbfa4117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6789/0 conn(0x7fbfa41007f0 legacy=0x7fbfa40fcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa411c9e0 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa411b9f0 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbfab99c700 10 client.54245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa411bee0 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbfab99c700 1 librados: init done 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fbf94002f70 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.894+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fbf94003640 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.895+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbf88004f00 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.895+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fbf94003880 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(721..721 src has 420..721) v4 ==== 8621+0+0 (unknown 3654380824 0 0) 0x7fbf94079080 con 0x7fbfa40fda50 2024-02-10T08:48:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 osd_map(721..721 src has 420..721) v4 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 3 client.54245.objecter handle_osd_map got epochs [721,721] > 0 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 3 client.54245.objecter handle_osd_map decoding full epoch 721 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 20 client.54245.objecter dump_active .. 0 homeless 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v721) v1 ==== 54+0+16 (unknown 2651826901 0 2420000192) 0x7fbf9407b470 con 0x7fbfa40fda50 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf90063d90 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 10 client.54245.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 1 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 1 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 0 1 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 4 2024-02-10T08:48:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 10 client.54245.objecter _send_command 1 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.896+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88004950 con 0x7fbf88004530 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf88004530 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88004950 con 0x7fbf88004530 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 1 = system:0 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 4 2024-02-10T08:48:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.897+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 0 1 2024-02-10T08:48:01.906 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:01.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 10 client.54245.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 2 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 2 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf880051a0 osd=0 3 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 0 2 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 4 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 10 client.54245.objecter _send_command 2 2024-02-10T08:48:01.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.912+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf88004d30 con 0x7fbf88004530 2024-02-10T08:48:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf88004d30 con 0x7fbf88004530 2024-02-10T08:48:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 4 2024-02-10T08:48:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 0 2 2024-02-10T08:48:01.915 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:01.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 10 client.54245.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 1 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 4 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 10 client.54245.objecter _send_command 3 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.914+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf8800c000 con 0x7fbf88004150 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf88004150 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf8800c000 con 0x7fbf88004150 2024-02-10T08:48:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 3 = system:0 2024-02-10T08:48:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 4 2024-02-10T08:48:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.915+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 1 3 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 10 client.54245.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 4 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 4 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.929+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88003fe0 osd=1 3 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.930+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 1 4 2024-02-10T08:48:01.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.930+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 4 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.930+0000 7fbfab99c700 10 client.54245.objecter _send_command 4 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.930+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf88001850 con 0x7fbf88004150 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf88001850 con 0x7fbf88004150 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:01.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 4 2024-02-10T08:48:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 1 4 2024-02-10T08:48:01.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 10 client.54245.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 5 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 5 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 2 5 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 4 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 10 client.54245.objecter _send_command 5 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88003e70 con 0x7fbf8800d2f0 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.931+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf8800d2f0 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.932+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88003e70 con 0x7fbf8800d2f0 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.932+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.932+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 5 = system:0 2024-02-10T08:48:01.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.932+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 4 2024-02-10T08:48:01.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.932+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 2 5 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 10 client.54245.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 6 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 6 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88003630 osd=2 3 2024-02-10T08:48:01.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 2 6 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 4 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 10 client.54245.objecter _send_command 6 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.946+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf8800e6c0 con 0x7fbf8800d2f0 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf8800e6c0 con 0x7fbf8800d2f0 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:01.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 4 2024-02-10T08:48:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 2 6 2024-02-10T08:48:01.949 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 10 client.54245.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 7 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 7 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 3 7 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 4 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 10 client.54245.objecter _send_command 7 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88015ad0 con 0x7fbf88011e80 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.948+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf88011e80 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.949+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88015ad0 con 0x7fbf88011e80 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.949+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.949+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 7 = system:0 2024-02-10T08:48:01.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.949+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 4 2024-02-10T08:48:01.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.949+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 3 7 2024-02-10T08:48:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 10 client.54245.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 8 2024-02-10T08:48:01.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 8 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf880033d0 osd=3 3 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 3 8 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 4 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 10 client.54245.objecter _send_command 8 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.963+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf8800e100 con 0x7fbf88011e80 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.964+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf8800e100 con 0x7fbf88011e80 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.964+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.964+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.964+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 4 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.964+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 3 8 2024-02-10T08:48:01.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 10 client.54245.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 9 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 9 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 4 9 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 4 2024-02-10T08:48:01.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 10 client.54245.objecter _send_command 9 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf8800e4b0 con 0x7fbf8800ce50 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.965+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf8800ce50 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.966+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf8800e4b0 con 0x7fbf8800ce50 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.966+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.966+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 9 = system:0 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.966+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 4 2024-02-10T08:48:01.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.966+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 4 9 2024-02-10T08:48:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 10 client.54245.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 10 2024-02-10T08:48:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 10 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf8800e300 osd=4 3 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 4 10 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 4 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 10 client.54245.objecter _send_command 10 2024-02-10T08:48:01.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.980+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf8800e4b0 con 0x7fbf8800ce50 2024-02-10T08:48:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf8800e4b0 con 0x7fbf8800ce50 2024-02-10T08:48:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 4 2024-02-10T08:48:01.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 4 10 2024-02-10T08:48:01.983 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 10 client.54245.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 11 2024-02-10T08:48:01.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 11 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 5 11 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 4 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 10 client.54245.objecter _send_command 11 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.982+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88018040 con 0x7fbf88017a60 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf88017a60 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88018040 con 0x7fbf88017a60 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 11 = system:0 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 4 2024-02-10T08:48:01.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.983+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 5 11 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 10 client.54245.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 12 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 12 2024-02-10T08:48:01.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf8800e4b0 osd=5 3 2024-02-10T08:48:01.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 5 12 2024-02-10T08:48:01.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 4 2024-02-10T08:48:01.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 10 client.54245.objecter _send_command 12 2024-02-10T08:48:01.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.997+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf88018040 con 0x7fbf88017a60 2024-02-10T08:48:02.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf88018040 con 0x7fbf88017a60 2024-02-10T08:48:02.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:02.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 4 2024-02-10T08:48:02.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 5 12 2024-02-10T08:48:02.001 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:02.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 10 client.54245.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:02.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 13 2024-02-10T08:48:02.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 13 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 6 13 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 4 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 10 client.54245.objecter _send_command 13 2024-02-10T08:48:02.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88022c80 con 0x7fbf8801f0d0 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:01.999+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf8801f0d0 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.000+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88022c80 con 0x7fbf8801f0d0 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.000+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.000+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 13 = system:0 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.000+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 4 2024-02-10T08:48:02.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.000+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 6 13 2024-02-10T08:48:02.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:02.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 10 client.54245.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 14 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 14 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88018040 osd=6 3 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 6 14 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 4 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 10 client.54245.objecter _send_command 14 2024-02-10T08:48:02.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.014+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf88024430 con 0x7fbf8801f0d0 2024-02-10T08:48:02.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf88024430 con 0x7fbf8801f0d0 2024-02-10T08:48:02.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:02.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 4 2024-02-10T08:48:02.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 6 14 2024-02-10T08:48:02.018 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 10 client.54245.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 15 2024-02-10T08:48:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 15 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 7 15 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 4 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 10 client.54245.objecter _send_command 15 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf88025060 con 0x7fbf88024ab0 2024-02-10T08:48:02.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.016+0000 7fbf9a7fc700 10 client.54245.objecter ms_handle_connect 0x7fbf88024ab0 2024-02-10T08:48:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.017+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf88025060 con 0x7fbf88024ab0 2024-02-10T08:48:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.017+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.017+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 15 = system:0 2024-02-10T08:48:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.017+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 4 2024-02-10T08:48:02.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.017+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 7 15 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 10 client.54245.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign -1 16 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 20 client.54245.objecter _get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_remove -1 16 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 20 client.54245.objecter get_session s=0x7fbf88024430 osd=7 3 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 15 client.54245.objecter _session_command_op_assign 7 16 2024-02-10T08:48:02.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 4 2024-02-10T08:48:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 10 client.54245.objecter _send_command 16 2024-02-10T08:48:02.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.031+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf88025060 con 0x7fbf88024ab0 2024-02-10T08:48:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbf9a7fc700 1 -- 172.21.15.107:0/837888526 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf88025060 con 0x7fbf88024ab0 2024-02-10T08:48:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbf9a7fc700 10 client.54245.objecter ms_dispatch 0x7fbfa4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbf9a7fc700 10 client.54245.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbf9a7fc700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 4 2024-02-10T08:48:02.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbf9a7fc700 15 client.54245.objecter _session_command_op_remove 7 16 2024-02-10T08:48:02.035 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 librados: watch_flush enter 2024-02-10T08:48:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 librados: watch_flush exit 2024-02-10T08:48:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.0 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6801/3321596972 conn(0x7fbf88004530 legacy=0x7fbf88002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880051a0 osd=0 1 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.1 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6805/133516126 conn(0x7fbf88004150 legacy=0x7fbf8800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003fe0 osd=1 1 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.2 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6809/3575881728 conn(0x7fbf8800d2f0 legacy=0x7fbf8800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88003630 osd=2 1 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.3 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6813/2871825691 conn(0x7fbf88011e80 legacy=0x7fbf88014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.033+0000 7fbfa9f39700 1 -- 172.21.15.107:0/837888526 reap_dead start 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf880033d0 osd=3 1 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.4 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.195:6800/3310789251 conn(0x7fbf8800ce50 legacy=0x7fbf88015c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e300 osd=4 1 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.5 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.195:6804/4178321486 conn(0x7fbf88017a60 legacy=0x7fbf8801d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfa9f39700 1 -- 172.21.15.107:0/837888526 reap_dead start 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf8800e4b0 osd=5 1 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.6 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.195:6808/1417479017 conn(0x7fbf8801f0d0 legacy=0x7fbf880214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88018040 osd=6 1 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 10 client.54245.objecter close_session for osd.7 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.195:6812/1934176928 conn(0x7fbf88024ab0 legacy=0x7fbf88022c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter put_session s=0x7fbf88024430 osd=7 1 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 20 client.54245.objecter shutdown clearing up homeless session... 2024-02-10T08:48:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 10 client.54245.objecter successfully canceled tick 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6800/2634842009 conn(0x7fbf90063d90 legacy=0x7fbf90066250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> v1:172.21.15.107:6790/0 conn(0x7fbfa40fda50 legacy=0x7fbfa40fc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfa9f39700 1 -- 172.21.15.107:0/837888526 reap_dead start 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 shutdown_connections 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 >> 172.21.15.107:0/837888526 conn(0x7fbfa40f3640 msgr2=0x7fbfa40f65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 shutdown_connections 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 -- 172.21.15.107:0/837888526 wait complete. 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:02.034+0000 7fbfab99c700 1 librados: shutdown 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:48:02.042 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:02.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:00.921038+0000 osd.3 (osd.3) 315 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:48:02.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:00.922097+0000 osd.3 (osd.3) 316 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.030573+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.033497+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.104689+0000 osd.5 (osd.5) 300 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.104932+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:01.162353+0000 mon.a (mon.0) 3417 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.162426+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:01.163442+0000 mon.b (mon.1) 802 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:01.164123+0000 mon.a (mon.0) 3419 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:01.552325+0000 mgr.y (mgr.14553) 601 : cluster [DBG] pgmap v1091: 194 pgs: 5 unknown, 189 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:01.764968+0000 mgr.y (mgr.14553) 602 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:02.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:01.896698+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? v1:172.21.15.107:0/837888526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:02.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:48:02.285 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:02.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:00.921038+0000 osd.3 (osd.3) 315 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:00.922097+0000 osd.3 (osd.3) 316 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.030573+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.033497+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.104689+0000 osd.5 (osd.5) 300 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.104932+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:01.162353+0000 mon.a (mon.0) 3417 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.162426+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-10T08:48:02.384 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:01.163442+0000 mon.b (mon.1) 802 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:01.164123+0000 mon.a (mon.0) 3419 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:01.552325+0000 mgr.y (mgr.14553) 601 : cluster [DBG] pgmap v1091: 194 pgs: 5 unknown, 189 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:01.764968+0000 mgr.y (mgr.14553) 602 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:01.896698+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? v1:172.21.15.107:0/837888526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:00.921038+0000 osd.3 (osd.3) 315 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:00.922097+0000 osd.3 (osd.3) 316 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.030573+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.033497+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-02-10T08:48:02.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.104689+0000 osd.5 (osd.5) 300 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.104932+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:01.162353+0000 mon.a (mon.0) 3417 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.162426+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:01.163442+0000 mon.b (mon.1) 802 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:01.164123+0000 mon.a (mon.0) 3419 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:01.552325+0000 mgr.y (mgr.14553) 601 : cluster [DBG] pgmap v1091: 194 pgs: 5 unknown, 189 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:01.764968+0000 mgr.y (mgr.14553) 602 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:02.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:01.896698+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? v1:172.21.15.107:0/837888526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:02.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:02.549 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:02.765 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:02.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:02.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:02.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:48:02.896 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:02.914 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:02.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:03.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:03.044 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:03.063 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:03.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:03.149 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:03.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:03.197 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:03.325 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:03.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:03.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:02.159832+0000 mon.a (mon.0) 3420 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.159905+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:02.161490+0000 mon.b (mon.1) 803 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:02.161725+0000 mon.a (mon.0) 3422 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.597801+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.600699+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.840357+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.b scrub starts 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.843263+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.b scrub ok 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.155949+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:03.159997+0000 mon.a (mon.0) 3424 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]': finished 2024-02-10T08:48:03.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.160067+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:02.159832+0000 mon.a (mon.0) 3420 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.159905+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:02.161490+0000 mon.b (mon.1) 803 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:02.161725+0000 mon.a (mon.0) 3422 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.597801+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.600699+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.840357+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.b scrub starts 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.843263+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.b scrub ok 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.155949+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:03.159997+0000 mon.a (mon.0) 3424 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]': finished 2024-02-10T08:48:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.160067+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:02.159832+0000 mon.a (mon.0) 3420 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.159905+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:02.161490+0000 mon.b (mon.1) 803 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:02.161725+0000 mon.a (mon.0) 3422 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]: dispatch 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.597801+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:48:03.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.600699+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:48:03.434 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.840357+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.b scrub starts 2024-02-10T08:48:03.434 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.843263+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.b scrub ok 2024-02-10T08:48:03.434 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.155949+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:03.434 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:03.159997+0000 mon.a (mon.0) 3424 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-126", "mode": "writeback"}]': finished 2024-02-10T08:48:03.434 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.160067+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-10T08:48:03.458 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:03.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:03.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:04.203 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:48:04.203 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:48:04.203 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:48:04.204 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:48:04.204 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:48:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:04.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:04.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.998498+0000 osd.6 (osd.6) 294 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:02.998692+0000 osd.6 (osd.6) 295 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:04.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.072514+0000 osd.5 (osd.5) 302 : cluster [DBG] 234.3 scrub starts 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.075410+0000 osd.5 (osd.5) 303 : cluster [DBG] 234.3 scrub ok 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:03.226863+0000 mon.b (mon.1) 804 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:03.227205+0000 mon.a (mon.0) 3426 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.552738+0000 mgr.y (mgr.14553) 603 : cluster [DBG] pgmap v1094: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.835170+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.e scrub starts 2024-02-10T08:48:04.418 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.838060+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.e scrub ok 2024-02-10T08:48:04.419 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.998498+0000 osd.6 (osd.6) 294 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:02.998692+0000 osd.6 (osd.6) 295 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.072514+0000 osd.5 (osd.5) 302 : cluster [DBG] 234.3 scrub starts 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.075410+0000 osd.5 (osd.5) 303 : cluster [DBG] 234.3 scrub ok 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:03.226863+0000 mon.b (mon.1) 804 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:03.227205+0000 mon.a (mon.0) 3426 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.552738+0000 mgr.y (mgr.14553) 603 : cluster [DBG] pgmap v1094: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.835170+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.e scrub starts 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.838060+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.e scrub ok 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.998498+0000 osd.6 (osd.6) 294 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:02.998692+0000 osd.6 (osd.6) 295 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.072514+0000 osd.5 (osd.5) 302 : cluster [DBG] 234.3 scrub starts 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.075410+0000 osd.5 (osd.5) 303 : cluster [DBG] 234.3 scrub ok 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:03.226863+0000 mon.b (mon.1) 804 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:03.227205+0000 mon.a (mon.0) 3426 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.552738+0000 mgr.y (mgr.14553) 603 : cluster [DBG] pgmap v1094: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.835170+0000 osd.2 (osd.2) 267 : cluster [DBG] 2.e scrub starts 2024-02-10T08:48:04.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.838060+0000 osd.2 (osd.2) 268 : cluster [DBG] 2.e scrub ok 2024-02-10T08:48:04.438 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:04.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:04.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:04.570 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:04.703 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 0, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-10T08:48:04.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T08:48:04.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:48:04.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:48:05.064 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:05.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:05.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.342+0000 7f26200cd700 1 -- 172.21.15.107:0/2047522940 >> v1:172.21.15.107:6789/0 conn(0x7f26180fc910 legacy=0x7f26180fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.342+0000 7f26200cd700 1 -- 172.21.15.107:0/2047522940 shutdown_connections 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.342+0000 7f26200cd700 1 -- 172.21.15.107:0/2047522940 >> 172.21.15.107:0/2047522940 conn(0x7f26180f2510 msgr2=0x7f26180f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.342+0000 7f26200cd700 1 -- 172.21.15.107:0/2047522940 shutdown_connections 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.342+0000 7f26200cd700 1 -- 172.21.15.107:0/2047522940 wait complete. 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 librados: starting msgr at 2024-02-10T08:48:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 librados: starting objecter 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 Processor -- start 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 -- start start 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 librados: setting wanted keys 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 librados: calling monclient init 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2618116250 con 0x7f26180ff710 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2618117430 con 0x7f2618103250 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f26200cd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2618118630 con 0x7f26180fc910 2024-02-10T08:48:05.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f261de69700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f26180fc910 0x7f26180f88f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36628/0 (socket says 172.21.15.107:36628) 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f261de69700 1 -- 172.21.15.107:0/2581161916 learned_addr learned my addr 172.21.15.107:0/2581161916 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter ms_handle_connect 0x7f26180ff710 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter ms_handle_connect 0x7f26180fc910 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter ms_handle_connect 0x7f2618103250 2024-02-10T08:48:05.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:05.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291675212 0 0) 0x7f2618118630 con 0x7f26180fc910 2024-02-10T08:48:05.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26000038e0 con 0x7f26180fc910 2024-02-10T08:48:05.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.343+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432459700 0 0) 0x7f2618116250 con 0x7f26180ff710 2024-02-10T08:48:05.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2618118630 con 0x7f26180ff710 2024-02-10T08:48:05.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642928492 0 0) 0x7f2618117430 con 0x7f2618103250 2024-02-10T08:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2618116250 con 0x7f2618103250 2024-02-10T08:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601519770 0 0) 0x7f26000038e0 con 0x7f26180fc910 2024-02-10T08:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2618117430 con 0x7f26180fc910 2024-02-10T08:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2610000fe0 con 0x7f26180fc910 2024-02-10T08:48:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826589936 0 0) 0x7f2618118630 con 0x7f26180ff710 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26000038e0 con 0x7f26180ff710 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f260c0018d0 con 0x7f26180ff710 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459590411 0 0) 0x7f2618116250 con 0x7f2618103250 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2618118630 con 0x7f2618103250 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2608001b00 con 0x7f2618103250 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 858148927 0 0) 0x7f2618117430 con 0x7f26180fc910 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 >> v1:172.21.15.195:6789/0 conn(0x7f2618103250 legacy=0x7f2618114910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 >> v1:172.21.15.107:6789/0 conn(0x7f26180ff710 legacy=0x7f26180f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2618119830 con 0x7f26180fc910 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2618116440 con 0x7f26180fc910 2024-02-10T08:48:05.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f26200cd700 10 client.54251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2618116930 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f26200cd700 1 librados: init done 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f26100026b0 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.344+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2610003860 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.345+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2604004f20 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.345+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2610001840 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(725..725 src has 420..725) v4 ==== 8613+0+0 (unknown 1326430954 0 0) 0x7f2610079350 con 0x7f26180fc910 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 10 client.54251.objecter ms_dispatch 0x7f2618001550 osd_map(725..725 src has 420..725) v4 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 3 client.54251.objecter handle_osd_map got epochs [725,725] > 0 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 3 client.54251.objecter handle_osd_map decoding full epoch 725 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 20 client.54251.objecter dump_active .. 0 homeless 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.346+0000 7f2616ffd700 10 client.54251.objecter ms_handle_connect 0x7f2600063d10 2024-02-10T08:48:05.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.350+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f261004c600 con 0x7f26180fc910 2024-02-10T08:48:05.387 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:05.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.700575+0000 osd.6 (osd.6) 296 : cluster [DBG] 16.d deep-scrub starts 2024-02-10T08:48:05.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:03.702584+0000 osd.6 (osd.6) 297 : cluster [DBG] 16.d deep-scrub ok 2024-02-10T08:48:05.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:04.170590+0000 mon.a (mon.0) 3427 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:05.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:04.170685+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:04.172379+0000 mon.b (mon.1) 805 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:04.172641+0000 mon.a (mon.0) 3429 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.700575+0000 osd.6 (osd.6) 296 : cluster [DBG] 16.d deep-scrub starts 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:03.702584+0000 osd.6 (osd.6) 297 : cluster [DBG] 16.d deep-scrub ok 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:04.170590+0000 mon.a (mon.0) 3427 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:04.170685+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-10T08:48:05.388 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:04.172379+0000 mon.b (mon.1) 805 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:04.172641+0000 mon.a (mon.0) 3429 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:05.433 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:05.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.700575+0000 osd.6 (osd.6) 296 : cluster [DBG] 16.d deep-scrub starts 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:03.702584+0000 osd.6 (osd.6) 297 : cluster [DBG] 16.d deep-scrub ok 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:04.170590+0000 mon.a (mon.0) 3427 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:04.170685+0000 mon.a (mon.0) 3428 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:04.172379+0000 mon.b (mon.1) 805 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.523 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:04.172641+0000 mon.a (mon.0) 3429 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]: dispatch 2024-02-10T08:48:05.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:05.549+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f26040051d0 con 0x7f26180fc910 2024-02-10T08:48:05.657 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:05] "GET /metrics HTTP/1.1" 200 190578 "" "Prometheus/2.33.4" 2024-02-10T08:48:06.186 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.168523+0000 mon.a (mon.0) 3430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:05.173343+0000 mon.a (mon.0) 3431 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.173417+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.522982+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.523994+0000 osd.0 (osd.0) 146 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:05.550787+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.107:0/2581161916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:05.551089+0000 mon.a (mon.0) 3433 : audit [INF] from='client.54251 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.553085+0000 mgr.y (mgr.14553) 604 : cluster [DBG] pgmap v1097: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:06.187 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:05.696993+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.189+0000 7f2616ffd700 1 -- 172.21.15.107:0/2581161916 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v726) v1 ==== 87+0+0 (unknown 3086238197 0 0) 0x7f261004aa70 con 0x7f26180fc910 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.191+0000 7f26200cd700 10 librados: watch_flush enter 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 10 librados: watch_flush exit 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 20 client.54251.objecter shutdown clearing up homeless session... 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 10 client.54251.objecter successfully canceled tick 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2600063d10 legacy=0x7f26000661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 >> v1:172.21.15.107:6790/0 conn(0x7f26180fc910 legacy=0x7f26180f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 shutdown_connections 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 >> 172.21.15.107:0/2581161916 conn(0x7f26180f2510 msgr2=0x7f26181066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 shutdown_connections 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 -- 172.21.15.107:0/2581161916 wait complete. 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:06.192+0000 7f26200cd700 1 librados: shutdown 2024-02-10T08:48:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:48:06.294 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.168523+0000 mon.a (mon.0) 3430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:05.173343+0000 mon.a (mon.0) 3431 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.173417+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.522982+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.523994+0000 osd.0 (osd.0) 146 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:05.550787+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.107:0/2581161916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:05.551089+0000 mon.a (mon.0) 3433 : audit [INF] from='client.54251 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.553085+0000 mgr.y (mgr.14553) 604 : cluster [DBG] pgmap v1097: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:06.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:05.696993+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.168523+0000 mon.a (mon.0) 3430 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:05.173343+0000 mon.a (mon.0) 3431 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-126"}]': finished 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.173417+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.522982+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.523994+0000 osd.0 (osd.0) 146 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:05.550787+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.107:0/2581161916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:05.551089+0000 mon.a (mon.0) 3433 : audit [INF] from='client.54251 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.553085+0000 mgr.y (mgr.14553) 604 : cluster [DBG] pgmap v1097: 194 pgs: 194 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:06.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:05.696993+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:06.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:06.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:06.430 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:06.655 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.47647.0:1 136.6s0 136:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e196)", 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 596.61697520099995, 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997, 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.47647", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/993932140", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035917+0000", 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:48:06.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035914+0000", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6919999999999997e-06 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035924+0000", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3081000000000001e-05 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035937+0000", 2024-02-10T08:48:06.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0079999999999996e-06 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035944+0000", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1489000000000001e-05 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035995+0000", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2339999999999999e-06 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.035999+0000", 2024-02-10T08:48:06.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91614844299999998 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952147+0000", 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116906 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952264+0000", 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048505899999999998 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.952749+0000", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069718000000000004 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.953446+0000", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013142239999999999 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954760+0000", 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9851999999999995e-05 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:38:10.954830+0000", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91891334199999997 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53375.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 469.65079844399997, 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999, 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53375", 2024-02-10T08:48:06.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4067602482", 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002090+0000", 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2568000000000001e-05 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002103+0000", 2024-02-10T08:48:06.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6772000000000001e-05 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002120+0000", 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4579999999999999e-06 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002129+0000", 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2737e-05 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002182+0000", 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.963e-06 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002186+0000", 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.005764858 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.007951+0000", 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014669699999999999 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008097+0000", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045664099999999998 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008554+0000", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038511399999999999 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008939+0000", 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023091299999999999 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009170+0000", 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3881999999999996e-05 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009234+0000", 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.668 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27867.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 445.64661741399999, 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001, 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27867", 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/10211293", 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006272+0000", 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2092e-05 2024-02-10T08:48:06.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006284+0000", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1758e-05 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006296+0000", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5979999999999997e-06 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006304+0000", 2024-02-10T08:48:06.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015616800000000001 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006460+0000", 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4299999999999999e-06 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006465+0000", 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0093833510000001 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.015848+0000", 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017003600000000001 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016018+0000", 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00054150900000000002 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016560+0000", 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013759289999999999 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.017936+0000", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045079799999999998 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018386+0000", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0971999999999997e-05 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018457+0000", 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.674 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1662 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94547991871360 gen 9] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 271.46788944000002, 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997, 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1662 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184999+0000", 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8840000000000006e-06 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185009+0000", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3593e-05 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185023+0000", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8629999999999996e-06 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0421999999999999e-05 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185079+0000", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185086+0000", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88982775599999997 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.074914+0000", 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016209899999999999 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.075076+0000", 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005269442 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.080345+0000", 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7071 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e493)", 2024-02-10T08:48:06.679 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 247.50612675100001, 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994, 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7071 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146764+0000", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.878e-06 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146771+0000", 2024-02-10T08:48:06.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0343000000000001e-05 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146781+0000", 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4099999999999999e-06 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146786+0000", 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5929000000000001e-05 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146822+0000", 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1459999999999995e-05 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146914+0000", 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080156775999999999 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.227071+0000", 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.683 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7073 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e494)", 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 246.479092726, 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997, 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7073 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.684 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173794+0000", 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2813e-05 2024-02-10T08:48:06.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173817+0000", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0312e-05 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173848+0000", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1270999999999999e-05 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173859+0000", 2024-02-10T08:48:06.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4023999999999996e-05 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173923+0000", 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.131e-06 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173930+0000", 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079391889999999993 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.181869+0000", 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014406799999999999 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182013+0000", 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018625099999999999 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182199+0000", 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017831909999999999 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.183983+0000", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037849075000000003 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.221832+0000", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000174136 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222006+0000", 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6679000000000001e-05 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222103+0000", 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7074 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 246.418136766, 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998, 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.690 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7074 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:48:06.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999943 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234750+0000", 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0859000000000001e-05 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234771+0000", 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8118000000000001e-05 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234799+0000", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234811+0000", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038360999999999999 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235195+0000", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839000000000001 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235323+0000", 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038646028999999998 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.273969+0000", 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7076 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:48:06.694 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 245.57010491200001, 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999, 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7076 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:48:06.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082784+0000", 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1815e-05 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082796+0000", 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4978e-05 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082811+0000", 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8789999999999992e-06 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082820+0000", 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6817000000000002e-05 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082877+0000", 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8079999999999994e-06 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082886+0000", 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013785067 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096671+0000", 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109211 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096780+0000", 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000188168 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096968+0000", 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01095494 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.107923+0000", 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175246 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108098+0000", 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011893200000000001 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108217+0000", 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2880999999999995e-05 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108300+0000", 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:92 60.12 60:49b62e28:test-rados-api-smithi107-112490-78::bfb5db8bf32239a5b4437a6edddde56f73a76483:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e495)", 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 245.54452224799999, 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002, 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108368+0000", 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3590000000000008e-06 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108376+0000", 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9339999999999995e-06 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108385+0000", 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1519999999999997e-06 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108391+0000", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0077999999999997e-05 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108442+0000", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0480000000000001e-05 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108532+0000", 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018637699999999999 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108718+0000", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036315000000000002 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.109082+0000", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049288513999999999 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158370+0000", 2024-02-10T08:48:06.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4228999999999998e-05 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158434+0000", 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2422000000000004e-05 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158527+0000", 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7078 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e495)", 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 245.494693313, 2024-02-10T08:48:06.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001, 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7078 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158195+0000", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9330999999999998e-05 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158214+0000", 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4609000000000001e-05 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158239+0000", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9750000000000002e-06 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158249+0000", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013279000000000001 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158381+0000", 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4767999999999998e-05 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158476+0000", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022729646999999999 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181206+0000", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000387876 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181594+0000", 2024-02-10T08:48:06.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203425 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181797+0000", 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6589999999999999e-05 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181844+0000", 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8802999999999997e-05 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181933+0000", 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:469 237.10 237.10 (undecoded) ondisk+read+ignore_overlay+known_if_redirected e633)", 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 98.396681290000004, 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.944453247, 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.714 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 469 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256208+0000", 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0508e-05 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256219+0000", 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3166999999999999e-05 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256232+0000", 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9609999999999999e-06 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256239+0000", 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8158000000000001e-05 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256297+0000", 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8319999999999995e-06 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256306+0000", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94434135299999999 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.200647+0000", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6847999999999999e-05 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.200664+0000", 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.944453247 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.718 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.0.8:1 234.7s0 234:ffb4a4cf:test-rados-api-smithi107-112490-115::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e668)", 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 63.389895494999998, 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030410150000000002, 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:48:06.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262993+0000", 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1848000000000001e-05 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263005+0000", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2612000000000001e-05 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263018+0000", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0649999999999999e-06 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263026+0000", 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8707999999999993e-05 2024-02-10T08:48:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263095+0000", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6760000000000004e-06 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263100+0000", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027235660000000002 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.265824+0000", 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011065000000000001 2024-02-10T08:48:06.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.265934+0000", 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010323399999999999 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266038+0000", 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030410150000000002 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.724 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.0.8:2 234.7s0 234:ffb4a4cf:test-rados-api-smithi107-112490-115::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e668)", 2024-02-10T08:48:06.724 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:48:06.724 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 63.389855980999997, 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003812757, 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.293e-06 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263041+0000", 2024-02-10T08:48:06.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9949999999999996e-06 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263046+0000", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9299999999999999e-06 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263049+0000", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107218 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263156+0000", 2024-02-10T08:48:06.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2320000000000001e-06 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263161+0000", 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002893077 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266054+0000", 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5282999999999999e-05 2024-02-10T08:48:06.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266110+0000", 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073932200000000005 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266849+0000", 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003812757 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:969 241.5 241:a0000000:%01:::head [pgnls start_epoch 669] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e669)", 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.366691279999998, 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028089130000000001, 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 969 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286198+0000", 2024-02-10T08:48:06.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.944e-06 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286208+0000", 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2311e-05 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286220+0000", 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4770000000000002e-06 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286226+0000", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0899000000000003e-05 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286267+0000", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9780000000000002e-06 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286273+0000", 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002599334 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.288872+0000", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8223000000000003e-05 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.288910+0000", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9705999999999997e-05 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.289010+0000", 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028089130000000001 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.734 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(6.16 REQUEST e675)", 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.396436285, 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026645219999999999, 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256455+0000", 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2129999999999999e-06 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256460+0000", 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9454e-05 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256480+0000", 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4569999999999999e-05 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256494+0000", 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105244 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256600+0000", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6909999999999999e-05 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256617+0000", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00246321 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259080+0000", 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.615e-05 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259096+0000", 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4400999999999999e-05 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259120+0000", 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026645219999999999 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1247 243.1a 243:58000000:test-rados-api-smithi107-112490-119:::head [pgnls start_epoch 685] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e685)", 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:20.163268+0000", 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.489624249000002, 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020218839999999998, 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1247 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163268+0000", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163268+0000", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163264+0000", 2024-02-10T08:48:06.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3380999999999999e-05 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163278+0000", 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8185999999999998e-05 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163296+0000", 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6000000000000001e-06 2024-02-10T08:48:06.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163303+0000", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8808999999999998e-05 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163372+0000", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9689999999999997e-06 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.163379+0000", 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017903890000000001 2024-02-10T08:48:06.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.165170+0000", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2850999999999999e-05 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.165192+0000", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7368999999999996e-05 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.165290+0000", 2024-02-10T08:48:06.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020218839999999998 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1373 243.1a 243:58000000::::head [pgnls start_epoch 687] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e687)", 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 39.413064659, 2024-02-10T08:48:06.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021910319999999999, 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1373 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239823+0000", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5954e-05 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239839+0000", 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5633e-05 2024-02-10T08:48:06.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239865+0000", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0718e-05 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239876+0000", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8201999999999995e-05 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239944+0000", 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1449999999999996e-06 2024-02-10T08:48:06.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239950+0000", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001918131 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.241868+0000", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7027000000000003e-05 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.241905+0000", 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011327099999999999 2024-02-10T08:48:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.242019+0000", 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021910319999999999 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1432 244.18 244:18000000:test-rados-api-smithi107-112490-121:::head [pgnls start_epoch 693] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e693)", 2024-02-10T08:48:06.752 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.168492+0000", 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.484399828000001, 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001788284, 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1432 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168492+0000", 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168492+0000", 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168488+0000", 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6847999999999999e-05 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168505+0000", 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1897999999999999e-05 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168527+0000", 2024-02-10T08:48:06.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1182999999999999e-05 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168538+0000", 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0994000000000005e-05 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168609+0000", 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3589999999999999e-06 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.168614+0000", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015135400000000001 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.170128+0000", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9743000000000002e-05 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.170168+0000", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011288100000000001 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.170281+0000", 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001788284 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1750 245.10 245:08000000::::head [pgnls start_epoch 708] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.432039933999999, 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0319844600000001, 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.760 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1750 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220849+0000", 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1374e-05 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220861+0000", 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7833999999999999e-05 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220879+0000", 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5169999999999999e-06 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220887+0000", 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4659e-05 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220942+0000", 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1242e-05 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:48:06.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220953+0000", 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.031753017 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252706+0000", 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5024000000000001e-05 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252751+0000", 2024-02-10T08:48:06.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5668999999999996e-05 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252837+0000", 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0319844600000001 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1965 247.10 247:08000000::::head [pgnls start_epoch 725] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e725)", 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:05.176817+0000", 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.4760755919999999, 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018418359999999999, 2024-02-10T08:48:06.766 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1965 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176817+0000", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:48:06.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176817+0000", 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176812+0000", 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6745e-05 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176829+0000", 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0701000000000001e-05 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176850+0000", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8090000000000004e-06 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176859+0000", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4444999999999998e-05 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176914+0000", 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3530000000000002e-06 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.176919+0000", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001591077 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.178510+0000", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4434999999999998e-05 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.178545+0000", 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000113656 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:48:06.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:05.178658+0000", 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018418359999999999 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:48:06.771 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:48:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:06.772 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:06.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:06.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:06.902 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:06.924 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:06.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:07.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:07.059 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.924620+0000 osd.3 (osd.3) 317 : cluster [DBG] 234.4 scrub starts 2024-02-10T08:48:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.926584+0000 osd.3 (osd.3) 318 : cluster [DBG] 234.4 scrub ok 2024-02-10T08:48:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.940887+0000 osd.1 (osd.1) 318 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:48:07.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:05.945152+0000 osd.1 (osd.1) 319 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:48:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:06.175913+0000 mon.a (mon.0) 3435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:06.188017+0000 mon.a (mon.0) 3436 : audit [INF] from='client.54251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:06.188076+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-10T08:48:07.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:06.489680+0000 osd.0 (osd.0) 147 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:07.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:48:07.408 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:07.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:07.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:07.537 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:07.558 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:07.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:07.640 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:07.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.924620+0000 osd.3 (osd.3) 317 : cluster [DBG] 234.4 scrub starts 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.926584+0000 osd.3 (osd.3) 318 : cluster [DBG] 234.4 scrub ok 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.940887+0000 osd.1 (osd.1) 318 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:05.945152+0000 osd.1 (osd.1) 319 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:06.175913+0000 mon.a (mon.0) 3435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:06.188017+0000 mon.a (mon.0) 3436 : audit [INF] from='client.54251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:07.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:06.188076+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:06.489680+0000 osd.0 (osd.0) 147 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.924620+0000 osd.3 (osd.3) 317 : cluster [DBG] 234.4 scrub starts 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.926584+0000 osd.3 (osd.3) 318 : cluster [DBG] 234.4 scrub ok 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.940887+0000 osd.1 (osd.1) 318 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:05.945152+0000 osd.1 (osd.1) 319 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:06.175913+0000 mon.a (mon.0) 3435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:06.188017+0000 mon.a (mon.0) 3436 : audit [INF] from='client.54251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:06.188076+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-10T08:48:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:06.489680+0000 osd.0 (osd.0) 147 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:07.688 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:07.915 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:07.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:08.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:08.047 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:08.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:08.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:08.233 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:08.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:07.192368+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:07.201087+0000 mon.b (mon.1) 806 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:07.203925+0000 mon.a (mon.0) 3439 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:07.553641+0000 mgr.y (mgr.14553) 605 : cluster [DBG] pgmap v1100: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:08.431 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:07.192368+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:07.201087+0000 mon.b (mon.1) 806 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:07.203925+0000 mon.a (mon.0) 3439 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:07.553641+0000 mgr.y (mgr.14553) 605 : cluster [DBG] pgmap v1100: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:08.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:07.192368+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-10T08:48:08.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:07.201087+0000 mon.b (mon.1) 806 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:07.203925+0000 mon.a (mon.0) 3439 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:08.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:07.553641+0000 mgr.y (mgr.14553) 605 : cluster [DBG] pgmap v1100: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:08.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:08.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.433+0000 7fdd61090700 1 -- 172.21.15.107:0/785845193 >> v1:172.21.15.107:6790/0 conn(0x7fdd5c104350 legacy=0x7fdd5c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 -- 172.21.15.107:0/785845193 shutdown_connections 2024-02-10T08:48:08.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 -- 172.21.15.107:0/785845193 >> 172.21.15.107:0/785845193 conn(0x7fdd5c0f2510 msgr2=0x7fdd5c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:08.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 -- 172.21.15.107:0/785845193 shutdown_connections 2024-02-10T08:48:08.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 -- 172.21.15.107:0/785845193 wait complete. 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 librados: starting msgr at 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 librados: starting objecter 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.434+0000 7fdd61090700 1 Processor -- start 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 -- start start 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 librados: setting wanted keys 2024-02-10T08:48:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 librados: calling monclient init 2024-02-10T08:48:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd5c102d20 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd5c102ef0 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd61090700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd5c11bc10 con 0x7fdd5c104350 2024-02-10T08:48:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd5ad9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdd5c0f8a20 0x7fdd5c102580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39190/0 (socket says 172.21.15.107:39190) 2024-02-10T08:48:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd5ad9d700 1 -- 172.21.15.107:0/3690560470 learned_addr learned my addr 172.21.15.107:0/3690560470 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd5b59e700 1 --1- 172.21.15.107:0/3690560470 >> v1:172.21.15.107:6790/0 conn(0x7fdd5c104350 0x7fdd5c1184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36654/0 (socket says 172.21.15.107:36654) 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c104350 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c0f8a20 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c0f95d0 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215119726 0 0) 0x7fdd5c11bc10 con 0x7fdd5c104350 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd440033f0 con 0x7fdd5c104350 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809793645 0 0) 0x7fdd5c102d20 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c11bc10 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824198504 0 0) 0x7fdd5c102ef0 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c102d20 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437294450 0 0) 0x7fdd440033f0 con 0x7fdd5c104350 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd5c102ef0 con 0x7fdd5c104350 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdd540030e0 con 0x7fdd5c104350 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377796085 0 0) 0x7fdd5c11bc10 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd440033f0 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240384183 0 0) 0x7fdd5c102d20 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.435+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd5c11bc10 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdd4c002870 con 0x7fdd5c0f8a20 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdd50001840 con 0x7fdd5c0f95d0 2024-02-10T08:48:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3901635178 0 0) 0x7fdd5c102ef0 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 >> v1:172.21.15.195:6789/0 conn(0x7fdd5c0f95d0 legacy=0x7fdd5c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 >> v1:172.21.15.107:6789/0 conn(0x7fdd5c0f8a20 legacy=0x7fdd5c102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c11cdb0 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c11be00 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd61090700 10 client.54257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c11c270 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd61090700 1 librados: init done 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fdd54003320 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.436+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdd54003b10 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48004f20 con 0x7fdd5c104350 2024-02-10T08:48:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fdd54003d50 con 0x7fdd5c104350 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(728..728 src has 420..728) v4 ==== 8626+0+0 (unknown 1069372685 0 0) 0x7fdd54079570 con 0x7fdd5c104350 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd43fff700 10 client.54257.objecter ms_dispatch 0x7fdd5c001550 osd_map(728..728 src has 420..728) v4 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd43fff700 3 client.54257.objecter handle_osd_map got epochs [728,728] > 0 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.437+0000 7fdd43fff700 3 client.54257.objecter handle_osd_map decoding full epoch 728 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.438+0000 7fdd43fff700 20 client.54257.objecter dump_active .. 0 homeless 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.438+0000 7fdd43fff700 10 client.54257.objecter ms_handle_connect 0x7fdd44063ec0 2024-02-10T08:48:08.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.441+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdd5404c900 con 0x7fdd5c104350 2024-02-10T08:48:08.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:08.562 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.622+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fdd480051d0 con 0x7fdd5c104350 2024-02-10T08:48:08.810 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:08.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:08.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- 172.21.15.107:0/2837691326 >> v1:172.21.15.107:6789/0 conn(0x7f30b00ff710 legacy=0x7f30b0101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- 172.21.15.107:0/2837691326 shutdown_connections 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- 172.21.15.107:0/2837691326 >> 172.21.15.107:0/2837691326 conn(0x7f30b00f2510 msgr2=0x7f30b00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- 172.21.15.107:0/2837691326 shutdown_connections 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- 172.21.15.107:0/2837691326 wait complete. 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 librados: starting msgr at 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 librados: starting objecter 2024-02-10T08:48:08.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 Processor -- start 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 -- start start 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 librados: setting wanted keys 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.895+0000 7f30b75ab700 1 librados: calling monclient init 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b75ab700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30b01160c0 con 0x7f30b00fc910 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b75ab700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30b01172a0 con 0x7f30b00ff710 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b75ab700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f30b01184a0 con 0x7f30b0103250 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b5347700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f30b00fc910 0x7f30b00f8690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39212/0 (socket says 172.21.15.107:39212) 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b5347700 1 -- 172.21.15.107:0/822580331 learned_addr learned my addr 172.21.15.107:0/822580331 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b5b48700 1 --1- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6790/0 conn(0x7f30b0103250 0x7f30b01148d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36686/0 (socket says 172.21.15.107:36686) 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter ms_handle_connect 0x7f30b00fc910 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter ms_handle_connect 0x7f30b0103250 2024-02-10T08:48:08.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215731913 0 0) 0x7f30b01184a0 con 0x7f30b0103250 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003030 con 0x7f30b0103250 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627135244 0 0) 0x7f30b01160c0 con 0x7f30b00fc910 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b01184a0 con 0x7f30b00fc910 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter ms_handle_connect 0x7f30b00ff710 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768887572 0 0) 0x7f30b01172a0 con 0x7f30b00ff710 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b01160c0 con 0x7f30b00ff710 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2550080456 0 0) 0x7f3098003030 con 0x7f30b0103250 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b01172a0 con 0x7f30b0103250 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 723427620 0 0) 0x7f30b01184a0 con 0x7f30b00fc910 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3098003030 con 0x7f30b00fc910 2024-02-10T08:48:08.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30a0001c20 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30a80019f0 con 0x7f30b00fc910 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2905620094 0 0) 0x7f30b01172a0 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.195:6789/0 conn(0x7f30b00ff710 legacy=0x7f30b00f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6789/0 conn(0x7f30b00fc910 legacy=0x7f30b00f8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b01196a0 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.896+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b0118690 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30b75ab700 10 client.54263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b0118b80 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30b75ab700 1 librados: init done 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f30a00027a0 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30a0005df0 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.897+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f309c004f00 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f30a0018c80 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(728..728 src has 420..728) v4 ==== 8626+0+0 (unknown 1069372685 0 0) 0x7f30a0079240 con 0x7f30b0103250 2024-02-10T08:48:08.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 osd_map(728..728 src has 420..728) v4 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 3 client.54263.objecter handle_osd_map got epochs [728,728] > 0 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 3 client.54263.objecter handle_osd_map decoding full epoch 728 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 20 client.54263.objecter dump_active .. 0 homeless 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v728) v1 ==== 54+0+16 (unknown 3280972084 0 2420000192) 0x7f30a007b600 con 0x7f30b0103250 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.898+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f3098063980 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 10 client.54263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 1 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 1 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 0 1 2024-02-10T08:48:08.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 4 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 10 client.54263.objecter _send_command 1 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c004950 con 0x7f309c004530 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c004530 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c004950 con 0x7f309c004530 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 1 = system:0 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 4 2024-02-10T08:48:08.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.899+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 0 1 2024-02-10T08:48:08.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 10 client.54263.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:08.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 2 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 2 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c0051a0 osd=0 3 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 0 2 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 4 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 10 client.54263.objecter _send_command 2 2024-02-10T08:48:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.914+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c004d30 con 0x7f309c004530 2024-02-10T08:48:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c004d30 con 0x7f309c004530 2024-02-10T08:48:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 4 2024-02-10T08:48:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 0 2 2024-02-10T08:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 10 client.54263.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 1 3 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 4 2024-02-10T08:48:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 10 client.54263.objecter _send_command 3 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c00c000 con 0x7f309c004150 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.916+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c004150 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.917+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c00c000 con 0x7f309c004150 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.917+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.917+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 3 = system:0 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.917+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 4 2024-02-10T08:48:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.917+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 1 3 2024-02-10T08:48:08.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 10 client.54263.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:08.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 4 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 4 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c003fe0 osd=1 3 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 1 4 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 4 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 10 client.54263.objecter _send_command 4 2024-02-10T08:48:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.932+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c0018b0 con 0x7f309c004150 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c0018b0 con 0x7f309c004150 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 4 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 1 4 2024-02-10T08:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 10 client.54263.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 5 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 5 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 2 5 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 4 2024-02-10T08:48:08.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 10 client.54263.objecter _send_command 5 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.933+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c003e40 con 0x7f309c00d2f0 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c00d2f0 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c003e40 con 0x7f309c00d2f0 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 5 = system:0 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 4 2024-02-10T08:48:08.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.934+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 2 5 2024-02-10T08:48:08.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 10 client.54263.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 6 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 6 2024-02-10T08:48:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c003630 osd=2 3 2024-02-10T08:48:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 2 6 2024-02-10T08:48:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 4 2024-02-10T08:48:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 10 client.54263.objecter _send_command 6 2024-02-10T08:48:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.950+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c00e720 con 0x7f309c00d2f0 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.951+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c00e720 con 0x7f309c00d2f0 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.951+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.951+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.951+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 4 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.951+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 2 6 2024-02-10T08:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 10 client.54263.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 7 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 7 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 3 7 2024-02-10T08:48:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 4 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 10 client.54263.objecter _send_command 7 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c017c20 con 0x7f309c014060 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c014060 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c017c20 con 0x7f309c014060 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 7 = system:0 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 4 2024-02-10T08:48:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.952+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 3 7 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 10 client.54263.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 8 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 8 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c0033c0 osd=3 3 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 3 8 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 4 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 10 client.54263.objecter _send_command 8 2024-02-10T08:48:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.972+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c00e190 con 0x7f309c014060 2024-02-10T08:48:08.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c00e190 con 0x7f309c014060 2024-02-10T08:48:08.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 4 2024-02-10T08:48:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 3 8 2024-02-10T08:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:08.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 10 client.54263.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 9 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 9 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 4 9 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 4 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 10 client.54263.objecter _send_command 9 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.974+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c0184d0 con 0x7f309c00ce50 2024-02-10T08:48:08.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.975+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c00ce50 2024-02-10T08:48:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.976+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c0184d0 con 0x7f309c00ce50 2024-02-10T08:48:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.976+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.976+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 9 = system:0 2024-02-10T08:48:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.976+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 4 2024-02-10T08:48:08.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.976+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 4 9 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 10 client.54263.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 10 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 10 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c00e190 osd=4 3 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 4 10 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 4 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 10 client.54263.objecter _send_command 10 2024-02-10T08:48:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.990+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c0184d0 con 0x7f309c00ce50 2024-02-10T08:48:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c0184d0 con 0x7f309c00ce50 2024-02-10T08:48:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 4 2024-02-10T08:48:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 4 10 2024-02-10T08:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 10 client.54263.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 11 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 11 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 5 11 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 4 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 10 client.54263.objecter _send_command 11 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.992+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c01a3a0 con 0x7f309c019ec0 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c019ec0 2024-02-10T08:48:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c01a3a0 con 0x7f309c019ec0 2024-02-10T08:48:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 11 = system:0 2024-02-10T08:48:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 4 2024-02-10T08:48:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:08.993+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 5 11 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 10 client.54263.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 12 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 12 2024-02-10T08:48:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c019840 osd=5 3 2024-02-10T08:48:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 5 12 2024-02-10T08:48:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 4 2024-02-10T08:48:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 10 client.54263.objecter _send_command 12 2024-02-10T08:48:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.008+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c01a3a0 con 0x7f309c019ec0 2024-02-10T08:48:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c01a3a0 con 0x7f309c019ec0 2024-02-10T08:48:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 4 2024-02-10T08:48:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 5 12 2024-02-10T08:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 10 client.54263.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 13 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 13 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 6 13 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 4 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 10 client.54263.objecter _send_command 13 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.010+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c021d90 con 0x7f309c022830 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c022830 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c021d90 con 0x7f309c022830 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 13 = system:0 2024-02-10T08:48:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 4 2024-02-10T08:48:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.011+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 6 13 2024-02-10T08:48:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 10 client.54263.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 14 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 14 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c01a3a0 osd=6 3 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 6 14 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 4 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 10 client.54263.objecter _send_command 14 2024-02-10T08:48:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.025+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c021d90 con 0x7f309c022830 2024-02-10T08:48:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c021d90 con 0x7f309c022830 2024-02-10T08:48:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 4 2024-02-10T08:48:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 6 14 2024-02-10T08:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 10 client.54263.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 15 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 15 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 7 15 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 4 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 10 client.54263.objecter _send_command 15 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.027+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f309c029540 con 0x7f309c025a30 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 10 client.54263.objecter ms_handle_connect 0x7f309c025a30 2024-02-10T08:48:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f309c029540 con 0x7f309c025a30 2024-02-10T08:48:09.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:09.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 15 = system:0 2024-02-10T08:48:09.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 4 2024-02-10T08:48:09.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.028+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 7 15 2024-02-10T08:48:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 10 client.54263.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign -1 16 2024-02-10T08:48:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 20 client.54263.objecter _get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_remove -1 16 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 20 client.54263.objecter get_session s=0x7f309c021d90 osd=7 3 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 15 client.54263.objecter _session_command_op_assign 7 16 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 4 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 10 client.54263.objecter _send_command 16 2024-02-10T08:48:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.042+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f309c029540 con 0x7f309c025a30 2024-02-10T08:48:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30ae7fc700 1 -- 172.21.15.107:0/822580331 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f309c029540 con 0x7f309c025a30 2024-02-10T08:48:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30ae7fc700 10 client.54263.objecter ms_dispatch 0x7f30b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30ae7fc700 10 client.54263.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30ae7fc700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 4 2024-02-10T08:48:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30ae7fc700 15 client.54263.objecter _session_command_op_remove 7 16 2024-02-10T08:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 librados: watch_flush enter 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 librados: watch_flush exit 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.0 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6801/3321596972 conn(0x7f309c004530 legacy=0x7f309c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0051a0 osd=0 1 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.1 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6805/133516126 conn(0x7f309c004150 legacy=0x7f309c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003fe0 osd=1 1 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.2 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6809/3575881728 conn(0x7f309c00d2f0 legacy=0x7f309c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c003630 osd=2 1 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.3 2024-02-10T08:48:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6813/2871825691 conn(0x7f309c014060 legacy=0x7f309c016450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c0033c0 osd=3 1 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.4 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.195:6800/3310789251 conn(0x7f309c00ce50 legacy=0x7f309c017dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c00e190 osd=4 1 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.5 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.195:6804/4178321486 conn(0x7f309c019ec0 legacy=0x7f309c01f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c019840 osd=5 1 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.6 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.195:6808/1417479017 conn(0x7f309c022830 legacy=0x7f309c022cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c01a3a0 osd=6 1 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter close_session for osd.7 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.195:6812/1934176928 conn(0x7f309c025a30 legacy=0x7f309c027e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter put_session s=0x7f309c021d90 osd=7 1 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b5b48700 1 -- 172.21.15.107:0/822580331 reap_dead start 2024-02-10T08:48:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 20 client.54263.objecter shutdown clearing up homeless session... 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 10 client.54263.objecter successfully canceled tick 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b5b48700 1 -- 172.21.15.107:0/822580331 reap_dead start 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b5b48700 1 -- 172.21.15.107:0/822580331 reap_dead start 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b5b48700 1 -- 172.21.15.107:0/822580331 reap_dead start 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b5b48700 1 -- 172.21.15.107:0/822580331 reap_dead start 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3098063980 legacy=0x7f3098065e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> v1:172.21.15.107:6790/0 conn(0x7f30b0103250 legacy=0x7f30b01148d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 shutdown_connections 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 >> 172.21.15.107:0/822580331 conn(0x7f30b00f2510 msgr2=0x7f30b01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 shutdown_connections 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 -- 172.21.15.107:0/822580331 wait complete. 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.044+0000 7f30b75ab700 1 librados: shutdown 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.049 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:48:09.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.205+0000 7fdd43fff700 1 -- 172.21.15.107:0/3690560470 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v729) v1 ==== 113+0+0 (unknown 2724201860 0 0) 0x7fdd5404ad70 con 0x7fdd5c104350 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 10 librados: watch_flush enter 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 10 librados: watch_flush exit 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 20 client.54257.objecter shutdown clearing up homeless session... 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 10 client.54257.objecter successfully canceled tick 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 >> v1:172.21.15.107:6800/2634842009 conn(0x7fdd44063ec0 legacy=0x7fdd44066380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 >> v1:172.21.15.107:6790/0 conn(0x7fdd5c104350 legacy=0x7fdd5c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 shutdown_connections 2024-02-10T08:48:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.208+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 >> 172.21.15.107:0/3690560470 conn(0x7fdd5c0f2510 msgr2=0x7fdd5c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.210+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 shutdown_connections 2024-02-10T08:48:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.211+0000 7fdd61090700 1 -- 172.21.15.107:0/3690560470 wait complete. 2024-02-10T08:48:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:09.211+0000 7fdd61090700 1 librados: shutdown 2024-02-10T08:48:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T08:48:09.263 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:09.263 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:09.519 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.193153+0000 mon.a (mon.0) 3440 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:08.193225+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.215494+0000 mon.b (mon.1) 807 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.215896+0000 mon.a (mon.0) 3442 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.623245+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.107:0/3690560470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.623522+0000 mon.a (mon.0) 3443 : audit [INF] from='client.54257 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:08.899089+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.107:0/822580331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.193153+0000 mon.a (mon.0) 3440 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:08.193225+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.215494+0000 mon.b (mon.1) 807 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.520 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.215896+0000 mon.a (mon.0) 3442 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.623245+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.107:0/3690560470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.623522+0000 mon.a (mon.0) 3443 : audit [INF] from='client.54257 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:08.899089+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.107:0/822580331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.193153+0000 mon.a (mon.0) 3440 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:08.193225+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.215494+0000 mon.b (mon.1) 807 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.215896+0000 mon.a (mon.0) 3442 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.623245+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.107:0/3690560470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.623522+0000 mon.a (mon.0) 3443 : audit [INF] from='client.54257 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:48:09.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:08.899089+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.107:0/822580331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:09.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T08:48:10.167 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:10.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:10.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:10.434 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:10.453 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:09.205807+0000 mon.a (mon.0) 3444 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:09.205881+0000 mon.a (mon.0) 3445 : audit [INF] from='client.54257 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:09.205931+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:09.209789+0000 mon.b (mon.1) 808 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:09.210346+0000 mon.a (mon.0) 3447 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:09.554131+0000 mgr.y (mgr.14553) 606 : cluster [DBG] pgmap v1103: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:10.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:10.170818+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:10.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:10.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:10.584 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:09.205807+0000 mon.a (mon.0) 3444 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:09.205881+0000 mon.a (mon.0) 3445 : audit [INF] from='client.54257 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:09.205931+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:09.209789+0000 mon.b (mon.1) 808 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:09.210346+0000 mon.a (mon.0) 3447 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:09.554131+0000 mgr.y (mgr.14553) 606 : cluster [DBG] pgmap v1103: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:10.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:10.170818+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:09.205807+0000 mon.a (mon.0) 3444 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:09.205881+0000 mon.a (mon.0) 3445 : audit [INF] from='client.54257 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:09.205931+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:09.209789+0000 mon.b (mon.1) 808 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:09.210346+0000 mon.a (mon.0) 3447 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:09.554131+0000 mgr.y (mgr.14553) 606 : cluster [DBG] pgmap v1103: 194 pgs: 3 creating+peering, 29 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T08:48:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:10.170818+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:10.832 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:10.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:10.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:10.962 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:10.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:10.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:11.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T08:48:11.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:11.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:11.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:10.210984+0000 mon.a (mon.0) 3449 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:11.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:10.211076+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-10T08:48:11.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:10.214622+0000 mon.b (mon.1) 809 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:10.214980+0000 mon.a (mon.0) 3451 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:11.165710+0000 mon.a (mon.0) 3452 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:11.165781+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:11.167385+0000 mon.b (mon.1) 810 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:11.167708+0000 mon.a (mon.0) 3454 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:10.210984+0000 mon.a (mon.0) 3449 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:10.211076+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:10.214622+0000 mon.b (mon.1) 809 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:10.214980+0000 mon.a (mon.0) 3451 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:11.280 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:11.300 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:11.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:11.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:11.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:11.431 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:11.504 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:10.210984+0000 mon.a (mon.0) 3449 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:10.211076+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:10.214622+0000 mon.b (mon.1) 809 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:10.214980+0000 mon.a (mon.0) 3451 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:11.165710+0000 mon.a (mon.0) 3452 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:11.165781+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:11.167385+0000 mon.b (mon.1) 810 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.506 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:11.167708+0000 mon.a (mon.0) 3454 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:11.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:11.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:48:11.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:11.165710+0000 mon.a (mon.0) 3452 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:11.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:11.165781+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-10T08:48:11.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:11.167385+0000 mon.b (mon.1) 810 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.563 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:11.167708+0000 mon.a (mon.0) 3454 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T08:48:11.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:11.639 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:11.712 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:11.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:11.754 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:11.754 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:11.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:11.861 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:11.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:11.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:12.001 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:12.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:12.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:48:12.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:11.554646+0000 mgr.y (mgr.14553) 607 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+recovering, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:12.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:11.775035+0000 mgr.y (mgr.14553) 608 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:12.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:12.162637+0000 mon.a (mon.0) 3455 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:48:12.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:12.162718+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-10T08:48:12.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T08:48:12.452 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:12.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:12.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:12.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:11.554646+0000 mgr.y (mgr.14553) 607 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+recovering, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:12.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:11.775035+0000 mgr.y (mgr.14553) 608 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:12.162637+0000 mon.a (mon.0) 3455 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:12.162718+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:11.554646+0000 mgr.y (mgr.14553) 607 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+recovering, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:11.775035+0000 mgr.y (mgr.14553) 608 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:12.162637+0000 mon.a (mon.0) 3455 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T08:48:12.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:12.162718+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-10T08:48:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:12.705 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:12.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:12.714 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:48:12.749 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:12.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 -- 172.21.15.107:0/745354245 >> v1:172.21.15.107:6789/0 conn(0x7fa304104330 legacy=0x7fa3041067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 -- 172.21.15.107:0/745354245 shutdown_connections 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 -- 172.21.15.107:0/745354245 >> 172.21.15.107:0/745354245 conn(0x7fa3040f3640 msgr2=0x7fa3040f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 -- 172.21.15.107:0/745354245 shutdown_connections 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 -- 172.21.15.107:0/745354245 wait complete. 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 librados: starting msgr at 2024-02-10T08:48:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.429+0000 7fa308ecc700 1 librados: starting objecter 2024-02-10T08:48:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 Processor -- start 2024-02-10T08:48:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 -- start start 2024-02-10T08:48:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 librados: setting wanted keys 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 librados: calling monclient init 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa304117430 con 0x7fa304104330 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa304118610 con 0x7fa3041007f0 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa308ecc700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa304119810 con 0x7fa3040fda50 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa30259c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa3040fda50 0x7fa3040f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40790/0 (socket says 172.21.15.107:40790) 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa30259c700 1 -- 172.21.15.107:0/1427290912 learned_addr learned my addr 172.21.15.107:0/1427290912 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa2fb7fe700 10 client.?.objecter ms_handle_connect 0x7fa3040fda50 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa2fb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa2fb7fe700 10 client.?.objecter ms_handle_connect 0x7fa304104330 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.430+0000 7fa2fb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181216923 0 0) 0x7fa304117430 con 0x7fa304104330 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0003800 con 0x7fa304104330 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 10 client.?.objecter ms_handle_connect 0x7fa3041007f0 2024-02-10T08:48:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265233311 0 0) 0x7fa304119810 con 0x7fa3040fda50 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304117430 con 0x7fa3040fda50 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620740782 0 0) 0x7fa304118610 con 0x7fa3041007f0 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304119810 con 0x7fa3041007f0 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309111187 0 0) 0x7fa2f0003800 con 0x7fa304104330 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa304118610 con 0x7fa304104330 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2fc002f60 con 0x7fa304104330 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2550582844 0 0) 0x7fa304117430 con 0x7fa3040fda50 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f0003800 con 0x7fa3040fda50 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2f40019a0 con 0x7fa3040fda50 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1145496914 0 0) 0x7fa304119810 con 0x7fa3041007f0 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa304117430 con 0x7fa3041007f0 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2ec0018c0 con 0x7fa3041007f0 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025120027 0 0) 0x7fa304118610 con 0x7fa304104330 2024-02-10T08:48:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 >> v1:172.21.15.107:6790/0 conn(0x7fa3040fda50 legacy=0x7fa3040f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 >> v1:172.21.15.195:6789/0 conn(0x7fa3041007f0 legacy=0x7fa3040fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30411aa10 con 0x7fa304104330 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa304117620 con 0x7fa304104330 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa308ecc700 10 client.28674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa304117a90 con 0x7fa304104330 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa308ecc700 1 librados: init done 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa2fc0032a0 con 0x7fa304104330 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.431+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2fc003920 con 0x7fa304104330 2024-02-10T08:48:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.432+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4004f20 con 0x7fa304104330 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa2fc001b80 con 0x7fa304104330 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(733..733 src has 420..733) v4 ==== 8584+0+0 (unknown 2179154940 0 0) 0x7fa2fc076bf0 con 0x7fa304104330 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 10 client.28674.objecter ms_dispatch 0x7fa304001550 osd_map(733..733 src has 420..733) v4 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 3 client.28674.objecter handle_osd_map got epochs [733,733] > 0 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 3 client.28674.objecter handle_osd_map decoding full epoch 733 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 20 client.28674.objecter dump_active .. 0 homeless 2024-02-10T08:48:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.433+0000 7fa2fb7fe700 10 client.28674.objecter ms_handle_connect 0x7fa2f0063db0 2024-02-10T08:48:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.437+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa2fc049e00 con 0x7fa304104330 2024-02-10T08:48:13.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.177195+0000 mon.b (mon.1) 811 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.177490+0000 mon.a (mon.0) 3457 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.177986+0000 mon.b (mon.1) 812 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.178144+0000 mon.a (mon.0) 3458 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.177195+0000 mon.b (mon.1) 811 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.177490+0000 mon.a (mon.0) 3457 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.177986+0000 mon.b (mon.1) 812 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.178144+0000 mon.a (mon.0) 3458 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.177195+0000 mon.b (mon.1) 811 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.177490+0000 mon.a (mon.0) 3457 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.177986+0000 mon.b (mon.1) 812 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.178144+0000 mon.a (mon.0) 3458 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]: dispatch 2024-02-10T08:48:13.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:13.618+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fa2e40051d0 con 0x7fa304104330 2024-02-10T08:48:13.905 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:13.906 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:13.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:14.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:14.040 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:14.059 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:14.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:14.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:14.192 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:14.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.270+0000 7fa2fb7fe700 1 -- 172.21.15.107:0/1427290912 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v734) v1 ==== 113+0+0 (unknown 1445697550 0 0) 0x7fa2fc048270 con 0x7fa304104330 2024-02-10T08:48:14.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.273+0000 7fa308ecc700 10 librados: watch_flush enter 2024-02-10T08:48:14.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.273+0000 7fa308ecc700 10 librados: watch_flush exit 2024-02-10T08:48:14.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.273+0000 7fa308ecc700 20 client.28674.objecter shutdown clearing up homeless session... 2024-02-10T08:48:14.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.273+0000 7fa308ecc700 10 client.28674.objecter successfully canceled tick 2024-02-10T08:48:14.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.275+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa2f0063db0 legacy=0x7fa2f0066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:14.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.275+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 >> v1:172.21.15.107:6789/0 conn(0x7fa304104330 legacy=0x7fa304115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:14.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.275+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 shutdown_connections 2024-02-10T08:48:14.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.275+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 >> 172.21.15.107:0/1427290912 conn(0x7fa3040f3640 msgr2=0x7fa3041077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:14.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.276+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 shutdown_connections 2024-02-10T08:48:14.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.276+0000 7fa308ecc700 1 -- 172.21.15.107:0/1427290912 wait complete. 2024-02-10T08:48:14.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:14.276+0000 7fa308ecc700 1 librados: shutdown 2024-02-10T08:48:14.277 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:12.477162+0000 osd.3 (osd.3) 319 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:12.477361+0000 osd.3 (osd.3) 320 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.134218+0000 osd.2 (osd.2) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.134368+0000 osd.2 (osd.2) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.263332+0000 mon.a (mon.0) 3459 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]': finished 2024-02-10T08:48:14.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.263390+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.555111+0000 mgr.y (mgr.14553) 609 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+recovering, 193 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:13.619465+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:12.477162+0000 osd.3 (osd.3) 319 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:12.477361+0000 osd.3 (osd.3) 320 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.134218+0000 osd.2 (osd.2) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.134368+0000 osd.2 (osd.2) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.263332+0000 mon.a (mon.0) 3459 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]': finished 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.263390+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.555111+0000 mgr.y (mgr.14553) 609 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+recovering, 193 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:14.464 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:13.619465+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:12.477162+0000 osd.3 (osd.3) 319 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:12.477361+0000 osd.3 (osd.3) 320 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.134218+0000 osd.2 (osd.2) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.134368+0000 osd.2 (osd.2) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.263332+0000 mon.a (mon.0) 3459 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-128"}]': finished 2024-02-10T08:48:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.263390+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-10T08:48:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.555111+0000 mgr.y (mgr.14553) 609 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+recovering, 193 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:13.619465+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:48:14.786 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:14.787 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:15.042 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:15.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:15.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T08:48:15.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.487268+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:13.487541+0000 osd.1 (osd.1) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:14.269358+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:14.269397+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.487268+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:15.314 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:13.487541+0000 osd.1 (osd.1) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:15.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:14.269358+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:48:15.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:14.269397+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-10T08:48:15.322 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:15.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:15.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.487268+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:15.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:13.487541+0000 osd.1 (osd.1) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:15.373 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:14.269358+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.107:0/1427290912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:48:15.373 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:14.269397+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-10T08:48:15.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T08:48:15.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:15.463 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:15.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:15] "GET /metrics HTTP/1.1" 200 190711 "" "Prometheus/2.33.4" 2024-02-10T08:48:16.015 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:16.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:16.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:16.156 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:16.181 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:16.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.232+0000 7fb50fa26700 1 -- 172.21.15.107:0/1331616327 >> v1:172.21.15.107:6790/0 conn(0x7fb508106640 legacy=0x7fb508108a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.232+0000 7fb50fa26700 1 -- 172.21.15.107:0/1331616327 shutdown_connections 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.232+0000 7fb50fa26700 1 -- 172.21.15.107:0/1331616327 >> 172.21.15.107:0/1331616327 conn(0x7fb5080f2510 msgr2=0x7fb5080f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.232+0000 7fb50fa26700 1 -- 172.21.15.107:0/1331616327 shutdown_connections 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.232+0000 7fb50fa26700 1 -- 172.21.15.107:0/1331616327 wait complete. 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 librados: starting msgr at 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 librados: starting objecter 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 Processor -- start 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 -- start start 2024-02-10T08:48:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 librados: setting wanted keys 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 librados: calling monclient init 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb508119600 con 0x7fb5080f6810 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb50811a7e0 con 0x7fb5080f7300 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.233+0000 7fb50fa26700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb50811b9e0 con 0x7fb508106640 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50dfc3700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb508106640 0x7fb508117cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40808/0 (socket says 172.21.15.107:40808) 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50dfc3700 1 -- 172.21.15.107:0/101783006 learned_addr learned my addr 172.21.15.107:0/101783006 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter ms_handle_connect 0x7fb508106640 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter ms_handle_connect 0x7fb5080f6810 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter ms_handle_connect 0x7fb5080f7300 2024-02-10T08:48:16.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983783813 0 0) 0x7fb50811b9e0 con 0x7fb508106640 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f40033f0 con 0x7fb508106640 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972224078 0 0) 0x7fb50811a7e0 con 0x7fb5080f7300 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb50811b9e0 con 0x7fb5080f7300 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428813255 0 0) 0x7fb508119600 con 0x7fb5080f6810 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb50811a7e0 con 0x7fb5080f6810 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562576781 0 0) 0x7fb4f40033f0 con 0x7fb508106640 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb508119600 con 0x7fb508106640 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb5000023e0 con 0x7fb508106640 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718511123 0 0) 0x7fb50811a7e0 con 0x7fb5080f6810 2024-02-10T08:48:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4f40033f0 con 0x7fb5080f6810 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687075177 0 0) 0x7fb50811b9e0 con 0x7fb5080f7300 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb50811a7e0 con 0x7fb5080f7300 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb504001840 con 0x7fb5080f6810 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb4f80019d0 con 0x7fb5080f7300 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3431097248 0 0) 0x7fb508119600 con 0x7fb508106640 2024-02-10T08:48:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 >> v1:172.21.15.195:6789/0 conn(0x7fb5080f7300 legacy=0x7fb508103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 >> v1:172.21.15.107:6789/0 conn(0x7fb5080f6810 legacy=0x7fb508102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50811cbe0 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5081197f0 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50fa26700 10 client.54275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb508119c60 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb50fa26700 1 librados: init done 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.234+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb500002a20 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.235+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb500003c40 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.235+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec004f20 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb500002580 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(735..735 src has 420..735) v4 ==== 8584+0+0 (unknown 3757444750 0 0) 0x7fb5000795c0 con 0x7fb508106640 2024-02-10T08:48:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 10 client.54275.objecter ms_dispatch 0x7fb508001550 osd_map(735..735 src has 420..735) v4 2024-02-10T08:48:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 3 client.54275.objecter handle_osd_map got epochs [735,735] > 0 2024-02-10T08:48:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 3 client.54275.objecter handle_osd_map decoding full epoch 735 2024-02-10T08:48:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 20 client.54275.objecter dump_active .. 0 homeless 2024-02-10T08:48:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.236+0000 7fb4fe7fc700 10 client.54275.objecter ms_handle_connect 0x7fb4f4063cd0 2024-02-10T08:48:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.240+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb50004c890 con 0x7fb508106640 2024-02-10T08:48:16.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:16.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:15.274430+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-10T08:48:16.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:15.284395+0000 mon.b (mon.1) 813 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:15.284752+0000 mon.a (mon.0) 3465 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.320 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:16.339 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:16.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:16.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:16.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:16.445+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb4ec0051d0 con 0x7fb508106640 2024-02-10T08:48:16.471 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:15.274430+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-10T08:48:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:15.284395+0000 mon.b (mon.1) 813 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:15.284752+0000 mon.a (mon.0) 3465 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:15.555555+0000 mgr.y (mgr.14553) 610 : cluster [DBG] pgmap v1112: 194 pgs: 32 unknown, 1 active+recovering, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:15.555555+0000 mgr.y (mgr.14553) 610 : cluster [DBG] pgmap v1112: 194 pgs: 32 unknown, 1 active+recovering, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:15.274430+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-10T08:48:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:15.284395+0000 mon.b (mon.1) 813 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:15.284752+0000 mon.a (mon.0) 3465 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:15.555555+0000 mgr.y (mgr.14553) 610 : cluster [DBG] pgmap v1112: 194 pgs: 32 unknown, 1 active+recovering, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/630 objects misplaced (0.159%) 2024-02-10T08:48:16.905 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:16.920 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:16.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:17.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:17.056 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:17.077 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:17.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:17.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:17.210 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:17.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.280+0000 7fb4fe7fc700 1 -- 172.21.15.107:0/101783006 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v737) v1 ==== 97+0+0 (unknown 4078982371 0 0) 0x7fb50004ad00 con 0x7fb508106640 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 10 librados: watch_flush enter 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 10 librados: watch_flush exit 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 20 client.54275.objecter shutdown clearing up homeless session... 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 10 client.54275.objecter successfully canceled tick 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb4f4063cd0 legacy=0x7fb4f4066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 >> v1:172.21.15.107:6790/0 conn(0x7fb508106640 legacy=0x7fb508117cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 shutdown_connections 2024-02-10T08:48:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 >> 172.21.15.107:0/101783006 conn(0x7fb5080f2510 msgr2=0x7fb508109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 shutdown_connections 2024-02-10T08:48:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.282+0000 7fb50fa26700 1 -- 172.21.15.107:0/101783006 wait complete. 2024-02-10T08:48:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:17.283+0000 7fb50fa26700 1 librados: shutdown 2024-02-10T08:48:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:48:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:17.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:17.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:16.278995+0000 mon.a (mon.0) 3466 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:16.279077+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:16.285700+0000 mon.b (mon.1) 814 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:16.296541+0000 mon.a (mon.0) 3468 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:16.446170+0000 mon.c (mon.2) 474 : audit [INF] from='client.? v1:172.21.15.107:0/101783006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:16.446511+0000 mon.a (mon.0) 3469 : audit [INF] from='client.54275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:17.273563+0000 mon.a (mon.0) 3470 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:17.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:17.280325+0000 mon.a (mon.0) 3471 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:17.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:17.280406+0000 mon.a (mon.0) 3472 : audit [INF] from='client.54275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:17.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:17.280461+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-10T08:48:17.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:17.282394+0000 mon.b (mon.1) 815 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:17.282773+0000 mon.a (mon.0) 3474 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.492 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:17.512 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:17.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:17.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:17.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:16.278995+0000 mon.a (mon.0) 3466 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:16.279077+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:16.285700+0000 mon.b (mon.1) 814 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:16.296541+0000 mon.a (mon.0) 3468 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:16.446170+0000 mon.c (mon.2) 474 : audit [INF] from='client.? v1:172.21.15.107:0/101783006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:16.446511+0000 mon.a (mon.0) 3469 : audit [INF] from='client.54275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:17.273563+0000 mon.a (mon.0) 3470 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:17.280325+0000 mon.a (mon.0) 3471 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:17.280406+0000 mon.a (mon.0) 3472 : audit [INF] from='client.54275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:17.280461+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:17.282394+0000 mon.b (mon.1) 815 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:17.282773+0000 mon.a (mon.0) 3474 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:16.278995+0000 mon.a (mon.0) 3466 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:16.279077+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:16.285700+0000 mon.b (mon.1) 814 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:16.296541+0000 mon.a (mon.0) 3468 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:16.446170+0000 mon.c (mon.2) 474 : audit [INF] from='client.? v1:172.21.15.107:0/101783006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:16.446511+0000 mon.a (mon.0) 3469 : audit [INF] from='client.54275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:17.273563+0000 mon.a (mon.0) 3470 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:17.280325+0000 mon.a (mon.0) 3471 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:17.280406+0000 mon.a (mon.0) 3472 : audit [INF] from='client.54275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:17.280461+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:17.282394+0000 mon.b (mon.1) 815 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.644 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:17.282773+0000 mon.a (mon.0) 3474 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T08:48:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:17.556104+0000 mgr.y (mgr.14553) 611 : cluster [DBG] pgmap v1115: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:18.279366+0000 mon.a (mon.0) 3475 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:48:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:18.279443+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-10T08:48:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:18.280326+0000 mon.b (mon.1) 816 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:18.280675+0000 mon.a (mon.0) 3477 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:17.556104+0000 mgr.y (mgr.14553) 611 : cluster [DBG] pgmap v1115: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:18.279366+0000 mon.a (mon.0) 3475 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:48:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:18.279443+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-10T08:48:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:18.280326+0000 mon.b (mon.1) 816 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:18.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:18.280675+0000 mon.a (mon.0) 3477 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:17.556104+0000 mgr.y (mgr.14553) 611 : cluster [DBG] pgmap v1115: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:18.279366+0000 mon.a (mon.0) 3475 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_count","val": "3"}]': finished 2024-02-10T08:48:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:18.279443+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-10T08:48:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:18.280326+0000 mon.b (mon.1) 816 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:18.280675+0000 mon.a (mon.0) 3477 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 -- 172.21.15.107:0/519055915 >> v1:172.21.15.107:6790/0 conn(0x7f4ca4104350 legacy=0x7f4ca41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 -- 172.21.15.107:0/519055915 shutdown_connections 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 -- 172.21.15.107:0/519055915 >> 172.21.15.107:0/519055915 conn(0x7f4ca40f2510 msgr2=0x7f4ca40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 -- 172.21.15.107:0/519055915 shutdown_connections 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 -- 172.21.15.107:0/519055915 wait complete. 2024-02-10T08:48:19.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 librados: starting msgr at 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.188+0000 7f4cab9b6700 1 librados: starting objecter 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 Processor -- start 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 -- start start 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 librados: setting wanted keys 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 librados: calling monclient init 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ca41172c0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ca41184a0 con 0x7f4ca40f9570 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4cab9b6700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ca41196a0 con 0x7f4ca4104350 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca9f53700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4ca4104350 0x7f4ca41159d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40822/0 (socket says 172.21.15.107:40822) 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca9752700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4ca40f89c0 0x7f4ca40ff400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33452/0 (socket says 172.21.15.107:33452) 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca9f53700 1 -- 172.21.15.107:0/2437839459 learned_addr learned my addr 172.21.15.107:0/2437839459 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter ms_handle_connect 0x7f4ca40f89c0 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter ms_handle_connect 0x7f4ca4104350 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785001298 0 0) 0x7f4ca41196a0 con 0x7f4ca4104350 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c003030 con 0x7f4ca4104350 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222206149 0 0) 0x7f4ca41172c0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca41196a0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter ms_handle_connect 0x7f4ca40f9570 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.189+0000 7f4ca27fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457449206 0 0) 0x7f4ca41184a0 con 0x7f4ca40f9570 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca41172c0 con 0x7f4ca40f9570 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200471620 0 0) 0x7f4ca41196a0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca41184a0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c94001c00 con 0x7f4ca40f89c0 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682171039 0 0) 0x7f4c8c003030 con 0x7f4ca4104350 2024-02-10T08:48:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca41196a0 con 0x7f4ca4104350 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c9c002050 con 0x7f4ca4104350 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 521829454 0 0) 0x7f4ca41184a0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6790/0 conn(0x7f4ca4104350 legacy=0x7f4ca41159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.195:6789/0 conn(0x7f4ca40f9570 legacy=0x7f4ca40ffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca411a8a0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca4119890 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4cab9b6700 10 client.28686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4119d00 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4cab9b6700 1 librados: init done 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4c94002720 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.190+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4c94005df0 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.191+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4c90004f00 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.191+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4c94018c80 con 0x7f4ca40f89c0 2024-02-10T08:48:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(738..738 src has 420..738) v4 ==== 8592+0+0 (unknown 4201489542 0 0) 0x7f4c94079170 con 0x7f4ca40f89c0 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 osd_map(738..738 src has 420..738) v4 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 3 client.28686.objecter handle_osd_map got epochs [738,738] > 0 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 3 client.28686.objecter handle_osd_map decoding full epoch 738 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 20 client.28686.objecter dump_active .. 0 homeless 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v738) v1 ==== 54+0+16 (unknown 2008594543 0 2420000192) 0x7f4c9407b570 con 0x7f4ca40f89c0 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c8c063a60 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 1 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 1 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 0 1 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 4 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 10 client.28686.objecter _send_command 1 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.192+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90004950 con 0x7f4c90004530 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.193+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c90004530 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.194+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90004950 con 0x7f4c90004530 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.194+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.194+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 1 = system:0 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.194+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 4 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.194+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 0 1 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 2 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 2 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900051a0 osd=0 3 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 0 2 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 4 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 10 client.28686.objecter _send_command 2 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.208+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90004d30 con 0x7f4c90004530 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90004d30 con 0x7f4c90004530 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 4 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 0 2 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 3 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 4 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 10 client.28686.objecter _send_command 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90010450 con 0x7f4c90004150 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.211+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c90004150 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.212+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90010450 con 0x7f4c90004150 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.212+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.212+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 3 = system:0 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.212+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 4 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.212+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 1 3 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 4 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 4 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90003fe0 osd=1 3 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 1 4 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 4 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 10 client.28686.objecter _send_command 4 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.228+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90001850 con 0x7f4c90004150 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90001850 con 0x7f4c90004150 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:19.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:19.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:19.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:19.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 4 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 1 4 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 5 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 5 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 2 5 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 4 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 10 client.28686.objecter _send_command 5 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.229+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90003e70 con 0x7f4c90011740 2024-02-10T08:48:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c90011740 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90003e70 con 0x7f4c90011740 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 5 = system:0 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 4 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.230+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 2 5 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 6 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 6 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90003630 osd=2 3 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 2 6 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 4 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 10 client.28686.objecter _send_command 6 2024-02-10T08:48:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.245+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90012b30 con 0x7f4c90011740 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90012b30 con 0x7f4c90011740 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 4 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 2 6 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 7 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 7 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 3 7 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 4 2024-02-10T08:48:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 10 client.28686.objecter _send_command 7 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.246+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90019ec0 con 0x7f4c900162a0 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c900162a0 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90019ec0 con 0x7f4c900162a0 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 7 = system:0 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 4 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.247+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 3 7 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 8 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 8 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900033c0 osd=3 3 2024-02-10T08:48:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 3 8 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 4 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 10 client.28686.objecter _send_command 8 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.262+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c900125a0 con 0x7f4c900162a0 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.263+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c900125a0 con 0x7f4c900162a0 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.263+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.263+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.263+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 4 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.263+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 3 8 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 9 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 9 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 4 9 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 4 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 10 client.28686.objecter _send_command 9 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90012870 con 0x7f4c900112a0 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.264+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c900112a0 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.265+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90012870 con 0x7f4c900112a0 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.265+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.265+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 9 = system:0 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.265+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 4 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.265+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 4 9 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 10 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 10 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c900125a0 osd=4 3 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 4 10 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 4 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 10 client.28686.objecter _send_command 10 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.283+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90012870 con 0x7f4c900112a0 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90012870 con 0x7f4c900112a0 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 4 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 4 10 2024-02-10T08:48:19.298 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 11 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 11 2024-02-10T08:48:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 5 11 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 4 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 10 client.28686.objecter _send_command 11 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c9001c390 con 0x7f4c9001be00 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.285+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c9001be00 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.286+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c9001c390 con 0x7f4c9001be00 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.286+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.286+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 11 = system:0 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.286+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 4 2024-02-10T08:48:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.286+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 5 11 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 12 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 12 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90012870 osd=5 3 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 5 12 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 4 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 10 client.28686.objecter _send_command 12 2024-02-10T08:48:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.301+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c9001c390 con 0x7f4c9001be00 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.303+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c9001c390 con 0x7f4c9001be00 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.303+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.303+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.303+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 4 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.303+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 5 12 2024-02-10T08:48:19.304 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 13 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 13 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 6 13 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 4 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 10 client.28686.objecter _send_command 13 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90026fd0 con 0x7f4c900234c0 2024-02-10T08:48:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.304+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c900234c0 2024-02-10T08:48:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.305+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90026fd0 con 0x7f4c900234c0 2024-02-10T08:48:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.305+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.305+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 13 = system:0 2024-02-10T08:48:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.305+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 4 2024-02-10T08:48:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.305+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 6 13 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 14 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 14 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c9001c390 osd=6 3 2024-02-10T08:48:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 6 14 2024-02-10T08:48:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 4 2024-02-10T08:48:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 10 client.28686.objecter _send_command 14 2024-02-10T08:48:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.322+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90028780 con 0x7f4c900234c0 2024-02-10T08:48:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90028780 con 0x7f4c900234c0 2024-02-10T08:48:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 4 2024-02-10T08:48:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 6 14 2024-02-10T08:48:19.325 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 15 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 15 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 7 15 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 4 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 10 client.28686.objecter _send_command 15 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.324+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90029340 con 0x7f4c90028e00 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.325+0000 7f4ca27fc700 10 client.28686.objecter ms_handle_connect 0x7f4c90028e00 2024-02-10T08:48:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.326+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4c90029340 con 0x7f4c90028e00 2024-02-10T08:48:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.326+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.326+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 15 = system:0 2024-02-10T08:48:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.326+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 4 2024-02-10T08:48:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.326+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 7 15 2024-02-10T08:48:19.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 10 client.28686.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign -1 16 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 20 client.28686.objecter _get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_remove -1 16 2024-02-10T08:48:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 20 client.28686.objecter get_session s=0x7f4c90028780 osd=7 3 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 15 client.28686.objecter _session_command_op_assign 7 16 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 4 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 10 client.28686.objecter _send_command 16 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.340+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4c90029340 con 0x7f4c90028e00 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca27fc700 1 -- 172.21.15.107:0/2437839459 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4c90029340 con 0x7f4c90028e00 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca27fc700 10 client.28686.objecter ms_dispatch 0x7f4ca4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca27fc700 10 client.28686.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca27fc700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 4 2024-02-10T08:48:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca27fc700 15 client.28686.objecter _session_command_op_remove 7 16 2024-02-10T08:48:19.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 librados: watch_flush enter 2024-02-10T08:48:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 librados: watch_flush exit 2024-02-10T08:48:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.0 2024-02-10T08:48:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6801/3321596972 conn(0x7f4c90004530 legacy=0x7f4c90002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900051a0 osd=0 1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6805/133516126 conn(0x7f4c90004150 legacy=0x7f4c9000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003fe0 osd=1 1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.2 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6809/3575881728 conn(0x7f4c90011740 legacy=0x7f4c90011b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90003630 osd=2 1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.3 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6813/2871825691 conn(0x7f4c900162a0 legacy=0x7f4c90018690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900033c0 osd=3 1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.4 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.195:6800/3310789251 conn(0x7f4c900112a0 legacy=0x7f4c9001a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c900125a0 osd=4 1 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.5 2024-02-10T08:48:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca9f53700 1 -- 172.21.15.107:0/2437839459 reap_dead start 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.195:6804/4178321486 conn(0x7f4c9001be00 legacy=0x7f4c90021890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca9f53700 1 -- 172.21.15.107:0/2437839459 reap_dead start 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.342+0000 7f4ca9f53700 1 -- 172.21.15.107:0/2437839459 reap_dead start 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90012870 osd=5 1 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.6 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.195:6808/1417479017 conn(0x7f4c900234c0 legacy=0x7f4c900258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c9001c390 osd=6 1 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 10 client.28686.objecter close_session for osd.7 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.195:6812/1934176928 conn(0x7f4c90028e00 legacy=0x7f4c90026fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 20 client.28686.objecter put_session s=0x7f4c90028780 osd=7 1 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 20 client.28686.objecter shutdown clearing up homeless session... 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 10 client.28686.objecter successfully canceled tick 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4c8c063a60 legacy=0x7f4c8c065f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> v1:172.21.15.107:6789/0 conn(0x7f4ca40f89c0 legacy=0x7f4ca40ff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4ca9f53700 1 -- 172.21.15.107:0/2437839459 reap_dead start 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 shutdown_connections 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 >> 172.21.15.107:0/2437839459 conn(0x7f4ca40f2510 msgr2=0x7f4ca40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 shutdown_connections 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 -- 172.21.15.107:0/2437839459 wait complete. 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:19.343+0000 7f4cab9b6700 1 librados: shutdown 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:48:19.398 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:19.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:19.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:19.192804+0000 mon.a (mon.0) 3478 : audit [DBG] from='client.? v1:172.21.15.107:0/2437839459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:19.280542+0000 mon.a (mon.0) 3479 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:19.280630+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:19.284006+0000 mon.b (mon.1) 817 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:19.284302+0000 mon.a (mon.0) 3481 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:19.192804+0000 mon.a (mon.0) 3478 : audit [DBG] from='client.? v1:172.21.15.107:0/2437839459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:19.280542+0000 mon.a (mon.0) 3479 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:19.280630+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:19.284006+0000 mon.b (mon.1) 817 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:19.284302+0000 mon.a (mon.0) 3481 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.731 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:19.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:19.192804+0000 mon.a (mon.0) 3478 : audit [DBG] from='client.? v1:172.21.15.107:0/2437839459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:19.280542+0000 mon.a (mon.0) 3479 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_period","val": "3"}]': finished 2024-02-10T08:48:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:19.280630+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-10T08:48:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:19.284006+0000 mon.b (mon.1) 817 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:19.284302+0000 mon.a (mon.0) 3481 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:19.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:19.901 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:48:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:48:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:48:19.901 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:48:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:19.556614+0000 mgr.y (mgr.14553) 612 : cluster [DBG] pgmap v1118: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:20.282328+0000 mon.a (mon.0) 3482 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:20.282409+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-10T08:48:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:20.283597+0000 mon.b (mon.1) 818 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:20.283920+0000 mon.a (mon.0) 3484 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:19.556614+0000 mgr.y (mgr.14553) 612 : cluster [DBG] pgmap v1118: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:20.282328+0000 mon.a (mon.0) 3482 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:20.282409+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:20.283597+0000 mon.b (mon.1) 818 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:20.283920+0000 mon.a (mon.0) 3484 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:19.556614+0000 mgr.y (mgr.14553) 612 : cluster [DBG] pgmap v1118: 194 pgs: 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:20.282328+0000 mon.a (mon.0) 3482 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:20.282409+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-10T08:48:20.628 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:20.283597+0000 mon.b (mon.1) 818 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:20.283920+0000 mon.a (mon.0) 3484 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T08:48:20.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T08:48:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:21.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:21.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:21.244 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:22.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:21.163749+0000 mon.a (mon.0) 3485 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:48:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:21.163821+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-10T08:48:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:21.557112+0000 mgr.y (mgr.14553) 613 : cluster [DBG] pgmap v1121: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:21.776909+0000 mgr.y (mgr.14553) 614 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:21.163749+0000 mon.a (mon.0) 3485 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:48:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:21.163821+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-10T08:48:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:21.557112+0000 mgr.y (mgr.14553) 613 : cluster [DBG] pgmap v1121: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:22.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:21.776909+0000 mgr.y (mgr.14553) 614 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:22.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:22.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:22.427 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:21.163749+0000 mon.a (mon.0) 3485 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T08:48:22.427 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:21.163821+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-10T08:48:22.427 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:21.557112+0000 mgr.y (mgr.14553) 613 : cluster [DBG] pgmap v1121: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:48:22.427 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:21.776909+0000 mgr.y (mgr.14553) 614 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:22.429 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:22.450 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:22.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:22.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:22.584 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:23.433 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:23.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.100+0000 7f72ec552700 1 -- 172.21.15.107:0/601023715 >> v1:172.21.15.107:6789/0 conn(0x7f72e40ff720 legacy=0x7f72e4101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- 172.21.15.107:0/601023715 shutdown_connections 2024-02-10T08:48:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- 172.21.15.107:0/601023715 >> 172.21.15.107:0/601023715 conn(0x7f72e406c090 msgr2=0x7f72e406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- 172.21.15.107:0/601023715 shutdown_connections 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- 172.21.15.107:0/601023715 wait complete. 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 librados: starting msgr at 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 librados: starting objecter 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 Processor -- start 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- start start 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 librados: setting wanted keys 2024-02-10T08:48:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 librados: calling monclient init 2024-02-10T08:48:23.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72e4117400 con 0x7f72e40fc960 2024-02-10T08:48:23.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72e41185e0 con 0x7f72e4103320 2024-02-10T08:48:23.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.101+0000 7f72ec552700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72e41197e0 con 0x7f72e40ff720 2024-02-10T08:48:23.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72e9aed700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f72e40ff720 0x7f72e40fa130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33696/0 (socket says 172.21.15.107:33696) 2024-02-10T08:48:23.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72ea2ee700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72e40fc960 0x7f72e40f99e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33198/0 (socket says 172.21.15.107:33198) 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72e9aed700 1 -- 172.21.15.107:0/4287583475 learned_addr learned my addr 172.21.15.107:0/4287583475 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter ms_handle_connect 0x7f72e40ff720 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter ms_handle_connect 0x7f72e40fc960 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter ms_handle_connect 0x7f72e4103320 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803497139 0 0) 0x7f72e41197e0 con 0x7f72e40ff720 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d00038e0 con 0x7f72e40ff720 2024-02-10T08:48:23.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101223150 0 0) 0x7f72e4117400 con 0x7f72e40fc960 2024-02-10T08:48:23.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e41197e0 con 0x7f72e40fc960 2024-02-10T08:48:23.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537179876 0 0) 0x7f72e41185e0 con 0x7f72e4103320 2024-02-10T08:48:23.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e4117400 con 0x7f72e4103320 2024-02-10T08:48:23.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139831615 0 0) 0x7f72d00038e0 con 0x7f72e40ff720 2024-02-10T08:48:23.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e41185e0 con 0x7f72e40ff720 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f72d40020f0 con 0x7f72e40ff720 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1670291371 0 0) 0x7f72e41197e0 con 0x7f72e40fc960 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d00038e0 con 0x7f72e40fc960 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f72e00019f0 con 0x7f72e40fc960 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917561584 0 0) 0x7f72e4117400 con 0x7f72e4103320 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e41197e0 con 0x7f72e4103320 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f72dc001b80 con 0x7f72e4103320 2024-02-10T08:48:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 405106284 0 0) 0x7f72e41185e0 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 >> v1:172.21.15.195:6789/0 conn(0x7f72e4103320 legacy=0x7f72e4115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 >> v1:172.21.15.107:6789/0 conn(0x7f72e40fc960 legacy=0x7f72e40f99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e411a9e0 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e41175f0 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72ec552700 10 client.54287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e4117a60 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72ec552700 1 librados: init done 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f72d4001af0 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.102+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f72d4003850 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.103+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c8004f20 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f72d4003a90 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(741..741 src has 420..741) v4 ==== 8618+0+0 (unknown 798111027 0 0) 0x7f72d4079480 con 0x7f72e40ff720 2024-02-10T08:48:23.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 10 client.54287.objecter ms_dispatch 0x7f72e4001550 osd_map(741..741 src has 420..741) v4 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 3 client.54287.objecter handle_osd_map got epochs [741,741] > 0 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 3 client.54287.objecter handle_osd_map decoding full epoch 741 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 20 client.54287.objecter dump_active .. 0 homeless 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.104+0000 7f72db7fe700 10 client.54287.objecter ms_handle_connect 0x7f72d0063c20 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.108+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f72d404c720 con 0x7f72e40ff720 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.287+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f72c80051d0 con 0x7f72e40ff720 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.343+0000 7f72db7fe700 1 -- 172.21.15.107:0/4287583475 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v742) v1 ==== 113+0+0 (unknown 3843401564 0 0) 0x7f72d404ab90 con 0x7f72e40ff720 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.345+0000 7f72ec552700 10 librados: watch_flush enter 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 10 librados: watch_flush exit 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 20 client.54287.objecter shutdown clearing up homeless session... 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 10 client.54287.objecter successfully canceled tick 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 >> v1:172.21.15.107:6800/2634842009 conn(0x7f72d0063c20 legacy=0x7f72d00660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 >> v1:172.21.15.107:6790/0 conn(0x7f72e40ff720 legacy=0x7f72e40fa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:23.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 shutdown_connections 2024-02-10T08:48:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 >> 172.21.15.107:0/4287583475 conn(0x7f72e406c090 msgr2=0x7f72e406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 shutdown_connections 2024-02-10T08:48:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 -- 172.21.15.107:0/4287583475 wait complete. 2024-02-10T08:48:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:23.346+0000 7f72ec552700 1 librados: shutdown 2024-02-10T08:48:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T08:48:23.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:23.568 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:23.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:48:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:23.289131+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.107:0/4287583475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:23.289423+0000 mon.a (mon.0) 3487 : audit [INF] from='client.54287 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:23.289131+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.107:0/4287583475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:23.289423+0000 mon.a (mon.0) 3487 : audit [INF] from='client.54287 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:23.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:23.289131+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.107:0/4287583475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:23.289423+0000 mon.a (mon.0) 3487 : audit [INF] from='client.54287 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:48:23.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:23.851 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:23.873 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:23.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:23.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:23.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:24.004 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:24.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:48:24.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T08:48:24.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:23.344602+0000 mon.a (mon.0) 3488 : audit [INF] from='client.54287 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:48:24.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:23.344686+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-10T08:48:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:23.557476+0000 mgr.y (mgr.14553) 615 : cluster [DBG] pgmap v1123: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 1 op/s 2024-02-10T08:48:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:23.344602+0000 mon.a (mon.0) 3488 : audit [INF] from='client.54287 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:48:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:23.344686+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-10T08:48:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:23.557476+0000 mgr.y (mgr.14553) 615 : cluster [DBG] pgmap v1123: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 1 op/s 2024-02-10T08:48:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:23.344602+0000 mon.a (mon.0) 3488 : audit [INF] from='client.54287 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:48:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:23.344686+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-10T08:48:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:23.557476+0000 mgr.y (mgr.14553) 615 : cluster [DBG] pgmap v1123: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 1 op/s 2024-02-10T08:48:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:24.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:24.922 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:24.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:24.984 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:25.132 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:25.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:25.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:25.264 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:25.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:25.448 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:48:25.449 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:48:25.449 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:48:25.449 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:48:25.449 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:48:25.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:25.548 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:25.569 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:25.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:25.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:25.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:25] "GET /metrics HTTP/1.1" 200 190869 "" "Prometheus/2.33.4" 2024-02-10T08:48:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:26.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:25.557929+0000 mgr.y (mgr.14553) 616 : cluster [DBG] pgmap v1124: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-02-10T08:48:26.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:26.167526+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-10T08:48:26.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:25.557929+0000 mgr.y (mgr.14553) 616 : cluster [DBG] pgmap v1124: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-02-10T08:48:26.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:26.167526+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-10T08:48:26.952 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:26.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:25.557929+0000 mgr.y (mgr.14553) 616 : cluster [DBG] pgmap v1124: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-02-10T08:48:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:26.167526+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-10T08:48:27.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:27.089 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:27.110 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:27.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:27.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:27.265 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:27.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:27.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:27.495 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:27.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:27.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.503+0000 7f42b737a700 1 -- 172.21.15.107:0/1938862135 >> v1:172.21.15.107:6789/0 conn(0x7f42b00fda50 legacy=0x7f42b00fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:27.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- 172.21.15.107:0/1938862135 shutdown_connections 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- 172.21.15.107:0/1938862135 >> 172.21.15.107:0/1938862135 conn(0x7f42b00f3640 msgr2=0x7f42b00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- 172.21.15.107:0/1938862135 shutdown_connections 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- 172.21.15.107:0/1938862135 wait complete. 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 librados: starting msgr at 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 librados: starting objecter 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 Processor -- start 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- start start 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 librados: setting wanted keys 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 librados: calling monclient init 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42b01174a0 con 0x7f42b00fda50 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42b0118680 con 0x7f42b01007f0 2024-02-10T08:48:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.504+0000 7f42b737a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42b0119880 con 0x7f42b0104330 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b5116700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f42b00fda50 0x7f42b00f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33206/0 (socket says 172.21.15.107:33206) 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b5116700 1 -- 172.21.15.107:0/1352808400 learned_addr learned my addr 172.21.15.107:0/1352808400 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter ms_handle_connect 0x7f42b00fda50 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter ms_handle_connect 0x7f42b0104330 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter ms_handle_connect 0x7f42b01007f0 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342429696 0 0) 0x7f42b01174a0 con 0x7f42b00fda50 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c003630 con 0x7f42b00fda50 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283738116 0 0) 0x7f42b0119880 con 0x7f42b0104330 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b01174a0 con 0x7f42b0104330 2024-02-10T08:48:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657973239 0 0) 0x7f42b0118680 con 0x7f42b01007f0 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b0119880 con 0x7f42b01007f0 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1702752310 0 0) 0x7f429c003630 con 0x7f42b00fda50 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b0118680 con 0x7f42b00fda50 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42ac002130 con 0x7f42b00fda50 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048001308 0 0) 0x7f42b01174a0 con 0x7f42b0104330 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f429c003630 con 0x7f42b0104330 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42a8001b00 con 0x7f42b0104330 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1533001238 0 0) 0x7f42b0119880 con 0x7f42b01007f0 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b01174a0 con 0x7f42b01007f0 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42a00018a0 con 0x7f42b01007f0 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 928522955 0 0) 0x7f42b0118680 con 0x7f42b00fda50 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 >> v1:172.21.15.107:6790/0 conn(0x7f42b0104330 legacy=0x7f42b0115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 >> v1:172.21.15.195:6789/0 conn(0x7f42b01007f0 legacy=0x7f42b00fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:27.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b011aa80 con 0x7f42b00fda50 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b0119a70 con 0x7f42b00fda50 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b737a700 10 client.28698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b0119f60 con 0x7f42b00fda50 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42b737a700 1 librados: init done 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f42ac002770 con 0x7f42b00fda50 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.505+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42ac0039c0 con 0x7f42b00fda50 2024-02-10T08:48:27.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.506+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4294004f20 con 0x7f42b00fda50 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f42ac0031c0 con 0x7f42b00fda50 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(743..743 src has 420..743) v4 ==== 8618+0+0 (unknown 1569670062 0 0) 0x7f42ac079190 con 0x7f42b00fda50 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 10 client.28698.objecter ms_dispatch 0x7f42b0001550 osd_map(743..743 src has 420..743) v4 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 3 client.28698.objecter handle_osd_map got epochs [743,743] > 0 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 3 client.28698.objecter handle_osd_map decoding full epoch 743 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 20 client.28698.objecter dump_active .. 0 homeless 2024-02-10T08:48:27.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.507+0000 7f42a67fc700 10 client.28698.objecter ms_handle_connect 0x7f429c063cd0 2024-02-10T08:48:27.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.511+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f42ac04c470 con 0x7f42b00fda50 2024-02-10T08:48:27.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:27.632 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:27.653 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:27.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:27.687+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f42940051d0 con 0x7f42b00fda50 2024-02-10T08:48:27.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:27.787 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:27.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:27.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:28.578 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:28.589 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:28.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.624+0000 7f42a67fc700 1 -- 172.21.15.107:0/1352808400 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v744) v1 ==== 91+0+0 (unknown 3282128464 0 0) 0x7f42ac04a8e0 con 0x7f42b00fda50 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 10 librados: watch_flush enter 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 10 librados: watch_flush exit 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 20 client.28698.objecter shutdown clearing up homeless session... 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 10 client.28698.objecter successfully canceled tick 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 >> v1:172.21.15.107:6800/2634842009 conn(0x7f429c063cd0 legacy=0x7f429c066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.625+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 >> v1:172.21.15.107:6789/0 conn(0x7f42b00fda50 legacy=0x7f42b00f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.626+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 shutdown_connections 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.626+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 >> 172.21.15.107:0/1352808400 conn(0x7f42b00f3640 msgr2=0x7f42b01077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.626+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 shutdown_connections 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.626+0000 7f42b737a700 1 -- 172.21.15.107:0/1352808400 wait complete. 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:28.626+0000 7f42b737a700 1 librados: shutdown 2024-02-10T08:48:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:48:28.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:28.872 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:28.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:27.558526+0000 mgr.y (mgr.14553) 617 : cluster [DBG] pgmap v1126: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.1 KiB/s wr, 2 op/s 2024-02-10T08:48:28.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:27.689467+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:48:28.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:27.558526+0000 mgr.y (mgr.14553) 617 : cluster [DBG] pgmap v1126: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.1 KiB/s wr, 2 op/s 2024-02-10T08:48:28.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:27.689467+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:48:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:28.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:28.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:27.558526+0000 mgr.y (mgr.14553) 617 : cluster [DBG] pgmap v1126: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 9.1 KiB/s wr, 2 op/s 2024-02-10T08:48:29.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:27.689467+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:48:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:29.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:29.259 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T08:48:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:29.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:29.414 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.436 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:29.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:29.570 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:29.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:29.721 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.615460+0000 mon.a (mon.0) 3492 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:29.721 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:28.624563+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:48:29.721 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.624619+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-10T08:48:29.721 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.749822+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:48:29.721 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.753285+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:48:29.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.808835+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:48:29.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.809924+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:48:29.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.964521+0000 osd.4 (osd.4) 45 : cluster [DBG] 234.0 scrub starts 2024-02-10T08:48:29.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:28.968027+0000 osd.4 (osd.4) 46 : cluster [DBG] 234.0 scrub ok 2024-02-10T08:48:29.723 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.742 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:29.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:29.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.615460+0000 mon.a (mon.0) 3492 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:28.624563+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.624619+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.749822+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.753285+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.808835+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.809924+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.964521+0000 osd.4 (osd.4) 45 : cluster [DBG] 234.0 scrub starts 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:28.968027+0000 osd.4 (osd.4) 46 : cluster [DBG] 234.0 scrub ok 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.615460+0000 mon.a (mon.0) 3492 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:29.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:28.624563+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? v1:172.21.15.107:0/1352808400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.624619+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.749822+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.753285+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.808835+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.809924+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.964521+0000 osd.4 (osd.4) 45 : cluster [DBG] 234.0 scrub starts 2024-02-10T08:48:29.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:28.968027+0000 osd.4 (osd.4) 46 : cluster [DBG] 234.0 scrub ok 2024-02-10T08:48:29.875 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:29.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:30.028 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.116+0000 7fdef5d35700 1 -- 172.21.15.107:0/1591129613 >> v1:172.21.15.107:6789/0 conn(0x7fdef00ff710 legacy=0x7fdef0101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.116+0000 7fdef5d35700 1 -- 172.21.15.107:0/1591129613 shutdown_connections 2024-02-10T08:48:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.116+0000 7fdef5d35700 1 -- 172.21.15.107:0/1591129613 >> 172.21.15.107:0/1591129613 conn(0x7fdef00f2510 msgr2=0x7fdef00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.116+0000 7fdef5d35700 1 -- 172.21.15.107:0/1591129613 shutdown_connections 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.116+0000 7fdef5d35700 1 -- 172.21.15.107:0/1591129613 wait complete. 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 librados: starting msgr at 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 librados: starting objecter 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 Processor -- start 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 -- start start 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 librados: setting wanted keys 2024-02-10T08:48:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 librados: calling monclient init 2024-02-10T08:48:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdef0119480 con 0x7fdef00ff710 2024-02-10T08:48:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdef011a660 con 0x7fdef0103250 2024-02-10T08:48:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdef5d35700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdef011b840 con 0x7fdef00fc910 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdeef7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fdef00fc910 0x7fdef00fb580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37906/0 (socket says 172.21.15.107:37906) 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdeef7fe700 1 -- 172.21.15.107:0/133260691 learned_addr learned my addr 172.21.15.107:0/133260691 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.117+0000 7fdeecff9700 10 client.?.objecter ms_handle_connect 0x7fdef00fc910 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627879156 0 0) 0x7fdef011b840 con 0x7fdef00fc910 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded80038e0 con 0x7fdef00fc910 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 10 client.?.objecter ms_handle_connect 0x7fdef0103250 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 10 client.?.objecter ms_handle_connect 0x7fdef00ff710 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913709606 0 0) 0x7fdef011a660 con 0x7fdef0103250 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef011b840 con 0x7fdef0103250 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264937824 0 0) 0x7fdef0119480 con 0x7fdef00ff710 2024-02-10T08:48:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef011a660 con 0x7fdef00ff710 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789884438 0 0) 0x7fded80038e0 con 0x7fdef00fc910 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdef0119480 con 0x7fdef00fc910 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdee0001960 con 0x7fdef00fc910 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325528069 0 0) 0x7fdef011a660 con 0x7fdef00ff710 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded80038e0 con 0x7fdef00ff710 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdee4002dc0 con 0x7fdef00ff710 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903856669 0 0) 0x7fdef011b840 con 0x7fdef0103250 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdef011a660 con 0x7fdef0103250 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdee8001a20 con 0x7fdef0103250 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 452640416 0 0) 0x7fdef0119480 con 0x7fdef00fc910 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 >> v1:172.21.15.195:6789/0 conn(0x7fdef0103250 legacy=0x7fdef0117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 >> v1:172.21.15.107:6789/0 conn(0x7fdef00ff710 legacy=0x7fdef00fbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef011ca60 con 0x7fdef00fc910 2024-02-10T08:48:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef011a850 con 0x7fdef00fc910 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdef5d35700 10 client.54299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef011ad40 con 0x7fdef00fc910 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdef5d35700 1 librados: init done 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fdee0002440 con 0x7fdef00fc910 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.118+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fdee0005b20 con 0x7fdef00fc910 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdedc004f20 con 0x7fdef00fc910 2024-02-10T08:48:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fdee0018a90 con 0x7fdef00fc910 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(744..744 src has 420..744) v4 ==== 8618+0+0 (unknown 3184129210 0 0) 0x7fdee0078f80 con 0x7fdef00fc910 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 10 client.54299.objecter ms_dispatch 0x7fdef0001550 osd_map(744..744 src has 420..744) v4 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 3 client.54299.objecter handle_osd_map got epochs [744,744] > 0 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 3 client.54299.objecter handle_osd_map decoding full epoch 744 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 20 client.54299.objecter dump_active .. 0 homeless 2024-02-10T08:48:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.120+0000 7fdeecff9700 10 client.54299.objecter ms_handle_connect 0x7fded8063cc0 2024-02-10T08:48:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.125+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdee004c160 con 0x7fdef00fc910 2024-02-10T08:48:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:30.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:30.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:30.261 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:30.280 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:30.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:30.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.318+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fdedc0051d0 con 0x7fdef00fc910 2024-02-10T08:48:30.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:30.412 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:30.485 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:30.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:30.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.625+0000 7fdeecff9700 1 -- 172.21.15.107:0/133260691 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v745) v1 ==== 113+0+0 (unknown 2466171215 0 0) 0x7fdee004a570 con 0x7fdef00fc910 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 10 librados: watch_flush enter 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 10 librados: watch_flush exit 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 20 client.54299.objecter shutdown clearing up homeless session... 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 10 client.54299.objecter successfully canceled tick 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 >> v1:172.21.15.107:6800/2634842009 conn(0x7fded8063cc0 legacy=0x7fded8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 >> v1:172.21.15.107:6790/0 conn(0x7fdef00fc910 legacy=0x7fdef00fb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 shutdown_connections 2024-02-10T08:48:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.628+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 >> 172.21.15.107:0/133260691 conn(0x7fdef00f2510 msgr2=0x7fdef01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.629+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 shutdown_connections 2024-02-10T08:48:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.629+0000 7fdef5d35700 1 -- 172.21.15.107:0/133260691 wait complete. 2024-02-10T08:48:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:30.629+0000 7fdef5d35700 1 librados: shutdown 2024-02-10T08:48:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T08:48:30.633 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:29.432098+0000 osd.0 (osd.0) 148 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:29.433216+0000 osd.0 (osd.0) 149 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:29.558981+0000 mgr.y (mgr.14553) 618 : cluster [DBG] pgmap v1128: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 823 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:29.844843+0000 osd.7 (osd.7) 296 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:29.848296+0000 osd.7 (osd.7) 297 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:30.320545+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.107:0/133260691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:30.654 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:30.320846+0000 mon.a (mon.0) 3495 : audit [INF] from='client.54299 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:29.432098+0000 osd.0 (osd.0) 148 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:29.433216+0000 osd.0 (osd.0) 149 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:29.558981+0000 mgr.y (mgr.14553) 618 : cluster [DBG] pgmap v1128: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 823 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:29.844843+0000 osd.7 (osd.7) 296 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:29.848296+0000 osd.7 (osd.7) 297 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:30.320545+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.107:0/133260691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:30.655 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:30.320846+0000 mon.a (mon.0) 3495 : audit [INF] from='client.54299 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:30.656 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:30.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:30.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:30.881 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:30.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:30.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:31.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:29.432098+0000 osd.0 (osd.0) 148 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:48:31.013 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:29.433216+0000 osd.0 (osd.0) 149 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:48:31.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:29.558981+0000 mgr.y (mgr.14553) 618 : cluster [DBG] pgmap v1128: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 823 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-02-10T08:48:31.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:29.844843+0000 osd.7 (osd.7) 296 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:48:31.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:29.848296+0000 osd.7 (osd.7) 297 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:48:31.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:30.320545+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.107:0/133260691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:31.014 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:30.320846+0000 mon.a (mon.0) 3495 : audit [INF] from='client.54299 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:48:31.015 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:31.036 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:31.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:31.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:31.170 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:31.190 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:31.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:31.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:31.320 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:31.338 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:31.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:31.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:31.469 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:31.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:31.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T08:48:31.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:31.748 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.432187+0000 osd.0 (osd.0) 150 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:48:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.433417+0000 osd.0 (osd.0) 151 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:48:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:30.626986+0000 mon.a (mon.0) 3496 : audit [INF] from='client.54299 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:48:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.627058+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.893426+0000 osd.7 (osd.7) 298 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.896302+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.432187+0000 osd.0 (osd.0) 150 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.433417+0000 osd.0 (osd.0) 151 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:30.626986+0000 mon.a (mon.0) 3496 : audit [INF] from='client.54299 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.627058+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.893426+0000 osd.7 (osd.7) 298 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:48:31.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.896302+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:48:31.996 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:31.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:32.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.432187+0000 osd.0 (osd.0) 150 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:48:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.433417+0000 osd.0 (osd.0) 151 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:48:32.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:30.626986+0000 mon.a (mon.0) 3496 : audit [INF] from='client.54299 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:48:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.627058+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-10T08:48:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.893426+0000 osd.7 (osd.7) 298 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:48:32.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.896302+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:48:32.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:32.132 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:32.152 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:32.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:32.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.848659+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:30.851853+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.131478+0000 osd.5 (osd.5) 304 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.134565+0000 osd.5 (osd.5) 305 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.559798+0000 mgr.y (mgr.14553) 619 : cluster [DBG] pgmap v1130: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 2 op/s 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:31.787846+0000 mgr.y (mgr.14553) 620 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.837482+0000 osd.6 (osd.6) 298 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.839254+0000 osd.6 (osd.6) 299 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.856423+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.859311+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:48:32.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.969490+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.848659+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:30.851853+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.131478+0000 osd.5 (osd.5) 304 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.134565+0000 osd.5 (osd.5) 305 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.559798+0000 mgr.y (mgr.14553) 619 : cluster [DBG] pgmap v1130: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 2 op/s 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:31.787846+0000 mgr.y (mgr.14553) 620 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.837482+0000 osd.6 (osd.6) 298 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.839254+0000 osd.6 (osd.6) 299 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.856423+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.859311+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:48:32.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.969490+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.848659+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:30.851853+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.131478+0000 osd.5 (osd.5) 304 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.134565+0000 osd.5 (osd.5) 305 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.559798+0000 mgr.y (mgr.14553) 619 : cluster [DBG] pgmap v1130: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 2 op/s 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:31.787846+0000 mgr.y (mgr.14553) 620 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.837482+0000 osd.6 (osd.6) 298 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:48:33.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.839254+0000 osd.6 (osd.6) 299 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:48:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.856423+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:48:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.859311+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:48:33.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.969490+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:48:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:31.973016+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:48:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.369788+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.b scrub starts 2024-02-10T08:48:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.371568+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub ok 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.719054+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.c scrub starts 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.722117+0000 osd.1 (osd.1) 325 : cluster [DBG] 5.c scrub ok 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.813289+0000 osd.6 (osd.6) 300 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.816207+0000 osd.6 (osd.6) 301 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.217352+0000 mon.b (mon.1) 819 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.217791+0000 mon.a (mon.0) 3498 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.218280+0000 mon.b (mon.1) 820 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.218496+0000 mon.a (mon.0) 3499 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.381609+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.107:0/408239632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:31.973016+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.369788+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.b scrub starts 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.371568+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub ok 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.719054+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.c scrub starts 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.722117+0000 osd.1 (osd.1) 325 : cluster [DBG] 5.c scrub ok 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.813289+0000 osd.6 (osd.6) 300 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:48:33.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.816207+0000 osd.6 (osd.6) 301 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:48:33.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.217352+0000 mon.b (mon.1) 819 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:33.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.217791+0000 mon.a (mon.0) 3498 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:33.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.218280+0000 mon.b (mon.1) 820 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:33.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.218496+0000 mon.a (mon.0) 3499 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:33.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.381609+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.107:0/408239632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:33.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:48:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:31.973016+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:48:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.369788+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.b scrub starts 2024-02-10T08:48:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.371568+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub ok 2024-02-10T08:48:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.719054+0000 osd.1 (osd.1) 324 : cluster [DBG] 5.c scrub starts 2024-02-10T08:48:34.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.722117+0000 osd.1 (osd.1) 325 : cluster [DBG] 5.c scrub ok 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.813289+0000 osd.6 (osd.6) 300 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.816207+0000 osd.6 (osd.6) 301 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.217352+0000 mon.b (mon.1) 819 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.217791+0000 mon.a (mon.0) 3498 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.218280+0000 mon.b (mon.1) 820 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.218496+0000 mon.a (mon.0) 3499 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]: dispatch 2024-02-10T08:48:34.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.381609+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.107:0/408239632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:34.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 -- 172.21.15.107:0/1314330067 >> v1:172.21.15.107:6790/0 conn(0x7f5d78104330 legacy=0x7f5d781067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 -- 172.21.15.107:0/1314330067 shutdown_connections 2024-02-10T08:48:34.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 -- 172.21.15.107:0/1314330067 >> 172.21.15.107:0/1314330067 conn(0x7f5d780f3640 msgr2=0x7f5d780f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 -- 172.21.15.107:0/1314330067 shutdown_connections 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 -- 172.21.15.107:0/1314330067 wait complete. 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 librados: starting msgr at 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.376+0000 7f5d7e407700 1 librados: starting objecter 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 Processor -- start 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 -- start start 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 librados: setting wanted keys 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 librados: calling monclient init 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5d78117550 con 0x7f5d78104330 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5d78118750 con 0x7f5d781007f0 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d7e407700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5d78119950 con 0x7f5d780fda50 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d77fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5d780fda50 0x7f5d780f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37932/0 (socket says 172.21.15.107:37932) 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d77fff700 1 -- 172.21.15.107:0/408239632 learned_addr learned my addr 172.21.15.107:0/408239632 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter ms_handle_connect 0x7f5d780fda50 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter ms_handle_connect 0x7f5d78104330 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter ms_handle_connect 0x7f5d781007f0 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.377+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998918634 0 0) 0x7f5d78119950 con 0x7f5d780fda50 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d600038e0 con 0x7f5d780fda50 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351968847 0 0) 0x7f5d78117550 con 0x7f5d78104330 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d78119950 con 0x7f5d78104330 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061623750 0 0) 0x7f5d78118750 con 0x7f5d781007f0 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d78117550 con 0x7f5d781007f0 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418168493 0 0) 0x7f5d600038e0 con 0x7f5d780fda50 2024-02-10T08:48:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d78118750 con 0x7f5d780fda50 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5d68001a30 con 0x7f5d780fda50 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3458330736 0 0) 0x7f5d78119950 con 0x7f5d78104330 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d600038e0 con 0x7f5d78104330 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5d700013d0 con 0x7f5d78104330 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926909396 0 0) 0x7f5d78117550 con 0x7f5d781007f0 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d78119950 con 0x7f5d781007f0 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5d6c0018c0 con 0x7f5d781007f0 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 122590439 0 0) 0x7f5d78118750 con 0x7f5d780fda50 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.195:6789/0 conn(0x7f5d781007f0 legacy=0x7f5d780fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6789/0 conn(0x7f5d78104330 legacy=0x7f5d78115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7811ab50 con 0x7f5d780fda50 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d78117740 con 0x7f5d780fda50 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d7e407700 10 client.54305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:34.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d78117bb0 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d7e407700 1 librados: init done 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f5d68002510 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.378+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f5d68005bd0 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.379+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5d64004f00 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.379+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f5d68018a60 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(745..745 src has 420..745) v4 ==== 8618+0+0 (unknown 103904396 0 0) 0x7f5d68079000 con 0x7f5d780fda50 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 osd_map(745..745 src has 420..745) v4 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 3 client.54305.objecter handle_osd_map got epochs [745,745] > 0 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 3 client.54305.objecter handle_osd_map decoding full epoch 745 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 20 client.54305.objecter dump_active .. 0 homeless 2024-02-10T08:48:34.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v745) v1 ==== 54+0+16 (unknown 1071896284 0 2420000192) 0x7f5d6807b410 con 0x7f5d780fda50 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d60063d50 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 10 client.54305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 1 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 1 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 0 1 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 4 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 10 client.54305.objecter _send_command 1 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d64004950 con 0x7f5d64004530 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.380+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d64004530 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.381+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d64004950 con 0x7f5d64004530 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.381+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.381+0000 7f5d757fa700 10 client.54305.objecter _finish_command 1 = system:0 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.381+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 4 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.381+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 0 1 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 10 client.54305.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 2 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 2 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d640051a0 osd=0 3 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 0 2 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 4 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 10 client.54305.objecter _send_command 2 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.396+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d64004d30 con 0x7f5d64004530 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.397+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d64004d30 con 0x7f5d64004530 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.397+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.397+0000 7f5d757fa700 10 client.54305.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.397+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 4 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.397+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 0 2 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 10 client.54305.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 3 2024-02-10T08:48:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 4 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 10 client.54305.objecter _send_command 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d6400c000 con 0x7f5d64004150 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d64004150 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d6400c000 con 0x7f5d64004150 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 10 client.54305.objecter _finish_command 3 = system:0 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 4 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.398+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 1 3 2024-02-10T08:48:34.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 10 client.54305.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 4 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003fe0 osd=1 3 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 1 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 10 client.54305.objecter _send_command 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.414+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d640018b0 con 0x7f5d64004150 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d640018b0 con 0x7f5d64004150 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 10 client.54305.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 1 4 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 10 client.54305.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 5 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 5 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 2 5 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 4 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 10 client.54305.objecter _send_command 5 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d64003e00 con 0x7f5d6400d2f0 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.416+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d6400d2f0 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.417+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d64003e00 con 0x7f5d6400d2f0 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.417+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.417+0000 7f5d757fa700 10 client.54305.objecter _finish_command 5 = system:0 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.417+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 4 2024-02-10T08:48:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.417+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 2 5 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 10 client.54305.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 6 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 6 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003630 osd=2 3 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 2 6 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 4 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 10 client.54305.objecter _send_command 6 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.431+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d6400e6b0 con 0x7f5d6400d2f0 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d6400e6b0 con 0x7f5d6400d2f0 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 10 client.54305.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 4 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 2 6 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 10 client.54305.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 7 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 7 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 3 7 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 4 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 10 client.54305.objecter _send_command 7 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d64015a30 con 0x7f5d64011e50 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.433+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d64011e50 2024-02-10T08:48:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.434+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d64015a30 con 0x7f5d64011e50 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.434+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.434+0000 7f5d757fa700 10 client.54305.objecter _finish_command 7 = system:0 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.434+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 4 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.434+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 3 7 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 10 client.54305.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 8 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:34.260 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 8 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64003390 osd=3 3 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 3 8 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 4 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 10 client.54305.objecter _send_command 8 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.448+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d6400e120 con 0x7f5d64011e50 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.449+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d6400e120 con 0x7f5d64011e50 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.449+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.449+0000 7f5d757fa700 10 client.54305.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.449+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 4 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.449+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 3 8 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 10 client.54305.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 9 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 9 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 4 9 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 4 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 10 client.54305.objecter _send_command 9 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d6400e050 con 0x7f5d6400ce50 2024-02-10T08:48:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.450+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d6400ce50 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.451+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d6400e050 con 0x7f5d6400ce50 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.451+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.451+0000 7f5d757fa700 10 client.54305.objecter _finish_command 9 = system:0 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.451+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 4 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.451+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 4 9 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 10 client.54305.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 10 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 10 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d6400dee0 osd=4 3 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 4 10 2024-02-10T08:48:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 4 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 10 client.54305.objecter _send_command 10 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.466+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d6400e050 con 0x7f5d6400ce50 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d6400e050 con 0x7f5d6400ce50 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d757fa700 10 client.54305.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 4 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 4 10 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 10 client.54305.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 11 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 11 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 5 11 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 4 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 10 client.54305.objecter _send_command 11 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.468+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d64017f30 con 0x7f5d64017980 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d64017980 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d64017f30 con 0x7f5d64017980 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:34.265 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 10 client.54305.objecter _finish_command 11 = system:0 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 4 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.469+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 5 11 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.484+0000 7f5d7e407700 10 client.54305.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 12 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 12 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d6400e050 osd=5 3 2024-02-10T08:48:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 5 12 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 4 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 10 client.54305.objecter _send_command 12 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.485+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d64017f30 con 0x7f5d64017980 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d64017f30 con 0x7f5d64017980 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 10 client.54305.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 4 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 5 12 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 10 client.54305.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 13 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 13 2024-02-10T08:48:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 6 13 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 4 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 10 client.54305.objecter _send_command 13 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d64020bb0 con 0x7f5d6401f030 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.487+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d6401f030 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.488+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d64020bb0 con 0x7f5d6401f030 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.488+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.488+0000 7f5d757fa700 10 client.54305.objecter _finish_command 13 = system:0 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.488+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 4 2024-02-10T08:48:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.488+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 6 13 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 10 client.54305.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 14 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 14 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64017f30 osd=6 3 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 6 14 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 4 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 10 client.54305.objecter _send_command 14 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.502+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d64020bb0 con 0x7f5d6401f030 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d64020bb0 con 0x7f5d6401f030 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d757fa700 10 client.54305.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 4 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 6 14 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 10 client.54305.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 15 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 15 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 7 15 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 4 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 10 client.54305.objecter _send_command 15 2024-02-10T08:48:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.504+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d6402b430 con 0x7f5d64020da0 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 10 client.54305.objecter ms_handle_connect 0x7f5d64020da0 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d6402b430 con 0x7f5d64020da0 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 10 client.54305.objecter _finish_command 15 = system:0 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 4 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.505+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 7 15 2024-02-10T08:48:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 10 client.54305.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign -1 16 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 20 client.54305.objecter _get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_remove -1 16 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 20 client.54305.objecter get_session s=0x7f5d64020bb0 osd=7 3 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 15 client.54305.objecter _session_command_op_assign 7 16 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 4 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 10 client.54305.objecter _send_command 16 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.520+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5d6402b3f0 con 0x7f5d64020da0 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d757fa700 1 -- 172.21.15.107:0/408239632 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5d6402b3f0 con 0x7f5d64020da0 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d757fa700 10 client.54305.objecter ms_dispatch 0x7f5d78001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d757fa700 10 client.54305.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d757fa700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 4 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d757fa700 15 client.54305.objecter _session_command_op_remove 7 16 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 librados: watch_flush enter 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 librados: watch_flush exit 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.0 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6801/3321596972 conn(0x7f5d64004530 legacy=0x7f5d64002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d640051a0 osd=0 1 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.1 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6805/133516126 conn(0x7f5d64004150 legacy=0x7f5d6400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003fe0 osd=1 1 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.2 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6809/3575881728 conn(0x7f5d6400d2f0 legacy=0x7f5d6400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003630 osd=2 1 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.3 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6813/2871825691 conn(0x7f5d64011e50 legacy=0x7f5d64014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64003390 osd=3 1 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.4 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.195:6800/3310789251 conn(0x7f5d6400ce50 legacy=0x7f5d64015ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7c9a4700 1 -- 172.21.15.107:0/408239632 reap_dead start 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400dee0 osd=4 1 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.5 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.195:6804/4178321486 conn(0x7f5d64017980 legacy=0x7f5d6401d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7c9a4700 1 -- 172.21.15.107:0/408239632 reap_dead start 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7c9a4700 1 -- 172.21.15.107:0/408239632 reap_dead start 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d6400e050 osd=5 1 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.6 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.195:6808/1417479017 conn(0x7f5d6401f030 legacy=0x7f5d6401f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64017f30 osd=6 1 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter close_session for osd.7 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.195:6812/1934176928 conn(0x7f5d64020da0 legacy=0x7f5d64029bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter put_session s=0x7f5d64020bb0 osd=7 1 2024-02-10T08:48:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 20 client.54305.objecter shutdown clearing up homeless session... 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 10 client.54305.objecter successfully canceled tick 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6800/2634842009 conn(0x7f5d60063d50 legacy=0x7f5d60066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.522+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> v1:172.21.15.107:6790/0 conn(0x7f5d780fda50 legacy=0x7f5d780f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.523+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 shutdown_connections 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.523+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 >> 172.21.15.107:0/408239632 conn(0x7f5d780f3640 msgr2=0x7f5d781077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.523+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 shutdown_connections 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.523+0000 7f5d7e407700 1 -- 172.21.15.107:0/408239632 wait complete. 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:33.523+0000 7f5d7e407700 1 librados: shutdown 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.276 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:48:34.436 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:34.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:34.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:34.574 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:34.596 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:34.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:34.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:34.729 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:34.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:34.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.986462+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:32.989966+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.155176+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.158659+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.411838+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.412767+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.560250+0000 mgr.y (mgr.14553) 621 : cluster [DBG] pgmap v1131: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-02-10T08:48:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:33.642289+0000 mon.a (mon.0) 3500 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]': finished 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:33.642360+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.986462+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:32.989966+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.155176+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.158659+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.411838+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.412767+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.560250+0000 mgr.y (mgr.14553) 621 : cluster [DBG] pgmap v1131: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-02-10T08:48:34.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:33.642289+0000 mon.a (mon.0) 3500 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]': finished 2024-02-10T08:48:34.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:33.642360+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-10T08:48:34.881 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:34.906 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:34.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:35.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.986462+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:32.989966+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.155176+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.158659+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.411838+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.412767+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.560250+0000 mgr.y (mgr.14553) 621 : cluster [DBG] pgmap v1131: 194 pgs: 194 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 KiB/s wr, 3 op/s 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:33.642289+0000 mon.a (mon.0) 3500 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-130"}]': finished 2024-02-10T08:48:35.036 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:33.642360+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-10T08:48:35.038 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:35.057 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:35.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:35.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:35.190 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:35.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:35.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:34.646448+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-10T08:48:35.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:35] "GET /metrics HTTP/1.1" 200 190869 "" "Prometheus/2.33.4" 2024-02-10T08:48:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:34.646448+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-10T08:48:36.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:34.646448+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-10T08:48:36.421 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:36.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:36.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:36.561 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:36.583 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:36.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:34.930844+0000 osd.2 (osd.2) 273 : cluster [DBG] 234.5 scrub starts 2024-02-10T08:48:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:34.935356+0000 osd.2 (osd.2) 274 : cluster [DBG] 234.5 scrub ok 2024-02-10T08:48:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:35.560633+0000 mgr.y (mgr.14553) 622 : cluster [DBG] pgmap v1134: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:35.652586+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:35.656524+0000 mon.b (mon.1) 821 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:35.657327+0000 mon.a (mon.0) 3504 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:36.172022+0000 mon.a (mon.0) 3505 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.172200+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:36.217096+0000 mon.b (mon.1) 822 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:36.217484+0000 mon.a (mon.0) 3507 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:34.930844+0000 osd.2 (osd.2) 273 : cluster [DBG] 234.5 scrub starts 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:34.935356+0000 osd.2 (osd.2) 274 : cluster [DBG] 234.5 scrub ok 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:35.560633+0000 mgr.y (mgr.14553) 622 : cluster [DBG] pgmap v1134: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:48:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:35.652586+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:35.656524+0000 mon.b (mon.1) 821 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:35.657327+0000 mon.a (mon.0) 3504 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:36.172022+0000 mon.a (mon.0) 3505 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.172200+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:36.217096+0000 mon.b (mon.1) 822 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:36.217484+0000 mon.a (mon.0) 3507 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:36.716 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:36.735 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:36.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:36.776 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:36.776 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:36.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:36.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:34.930844+0000 osd.2 (osd.2) 273 : cluster [DBG] 234.5 scrub starts 2024-02-10T08:48:36.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:34.935356+0000 osd.2 (osd.2) 274 : cluster [DBG] 234.5 scrub ok 2024-02-10T08:48:36.869 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:35.560633+0000 mgr.y (mgr.14553) 622 : cluster [DBG] pgmap v1134: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:35.652586+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:35.656524+0000 mon.b (mon.1) 821 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:35.657327+0000 mon.a (mon.0) 3504 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:36.172022+0000 mon.a (mon.0) 3505 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.172200+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:36.217096+0000 mon.b (mon.1) 822 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:36.217484+0000 mon.a (mon.0) 3507 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:36.871 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:36.890 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:36.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:36.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:48:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:36.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:37.021 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.040 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:37.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:37.171 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.189 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:37.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:37.319 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.338 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:37.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:37.468 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.486 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:37.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:37.616 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.635 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:37.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.150526+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.153580+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.451188+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.452175+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.649470+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.652973+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:48:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.963668+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:48:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:36.966556+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:48:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:37.186967+0000 mon.a (mon.0) 3508 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:37.187073+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-10T08:48:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:37.187789+0000 mon.b (mon.1) 823 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:37.190916+0000 mon.a (mon.0) 3510 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.150526+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.153580+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.451188+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.452175+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.649470+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:48:37.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.652973+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.963668+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:36.966556+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:37.186967+0000 mon.a (mon.0) 3508 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:37.187073+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:37.187789+0000 mon.b (mon.1) 823 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:37.190916+0000 mon.a (mon.0) 3510 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:37.766 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:37.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.150526+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:48:37.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.153580+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.451188+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.452175+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.649470+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.652973+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.963668+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:36.966556+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:37.186967+0000 mon.a (mon.0) 3508 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:37.187073+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:37.187789+0000 mon.b (mon.1) 823 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:37.190916+0000 mon.a (mon.0) 3510 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:37.863 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:37.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:37.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T08:48:37.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:37.994 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:38.014 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:38.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:38.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:38.146 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:38.243 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:38.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- 172.21.15.107:0/2780779217 >> v1:172.21.15.107:6789/0 conn(0x7f1d6c100840 legacy=0x7f1d6c102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- 172.21.15.107:0/2780779217 shutdown_connections 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- 172.21.15.107:0/2780779217 >> 172.21.15.107:0/2780779217 conn(0x7f1d6c06c090 msgr2=0x7f1d6c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- 172.21.15.107:0/2780779217 shutdown_connections 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- 172.21.15.107:0/2780779217 wait complete. 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 librados: starting msgr at 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 librados: starting objecter 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 Processor -- start 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 -- start start 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 librados: setting wanted keys 2024-02-10T08:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.344+0000 7f1d7414c700 1 librados: calling monclient init 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d7414c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c0f8ee0 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d7414c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c0f9070 con 0x7f1d6c100840 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d7414c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c0f9260 con 0x7f1d6c104350 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d71ee8700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1d6c0fd9f0 0x7f1d6c0fcae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40186/0 (socket says 172.21.15.107:40186) 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d71ee8700 1 -- 172.21.15.107:0/4182699554 learned_addr learned my addr 172.21.15.107:0/4182699554 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter ms_handle_connect 0x7f1d6c104350 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter ms_handle_connect 0x7f1d6c0fd9f0 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter ms_handle_connect 0x7f1d6c100840 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112171307 0 0) 0x7f1d6c0f8ee0 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d58003030 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230128962 0 0) 0x7f1d6c0f9260 con 0x7f1d6c104350 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c0f8ee0 con 0x7f1d6c104350 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626087243 0 0) 0x7f1d6c0f9070 con 0x7f1d6c100840 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c0f9260 con 0x7f1d6c100840 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179359391 0 0) 0x7f1d58003030 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6c0f9070 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d68001960 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345500017 0 0) 0x7f1d6c0f8ee0 con 0x7f1d6c104350 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d58003030 con 0x7f1d6c104350 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d64001d30 con 0x7f1d6c104350 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994470686 0 0) 0x7f1d6c0f9260 con 0x7f1d6c100840 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6c0f8ee0 con 0x7f1d6c100840 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d5c001fa0 con 0x7f1d6c100840 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1190215867 0 0) 0x7f1d6c0f9070 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 >> v1:172.21.15.107:6790/0 conn(0x7f1d6c104350 legacy=0x7f1d6c1184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.345+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 >> v1:172.21.15.195:6789/0 conn(0x7f1d6c100840 legacy=0x7f1d6c114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c11cc20 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c119c50 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d7414c700 10 client.28716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c11a0c0 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d7414c700 1 librados: init done 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1d68002460 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d68005b50 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.346+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50004f20 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.347+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1d68001fc0 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(751..751 src has 420..751) v4 ==== 8592+0+0 (unknown 2542885483 0 0) 0x7f1d68078e80 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 10 client.28716.objecter ms_dispatch 0x7f1d6c001550 osd_map(751..751 src has 420..751) v4 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 3 client.28716.objecter handle_osd_map got epochs [751,751] > 0 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 3 client.28716.objecter handle_osd_map decoding full epoch 751 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 20 client.28716.objecter dump_active .. 0 homeless 2024-02-10T08:48:38.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.348+0000 7f1d62ffd700 10 client.28716.objecter ms_handle_connect 0x7f1d58063c20 2024-02-10T08:48:38.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.352+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1d6804c080 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.383 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:38.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:38.539+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f1d500051d0 con 0x7f1d6c0fd9f0 2024-02-10T08:48:38.581 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:38.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:38.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:38.713 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:37.426245+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:37.427510+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:37.561074+0000 mgr.y (mgr.14553) 623 : cluster [DBG] pgmap v1138: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.001710+0000 osd.3 (osd.3) 321 : cluster [DBG] 6.b scrub starts 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.002993+0000 osd.3 (osd.3) 322 : cluster [DBG] 6.b scrub ok 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.013076+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.a scrub starts 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.016611+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub ok 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:38.187596+0000 mon.a (mon.0) 3511 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.187680+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:38.188550+0000 mon.b (mon.1) 824 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:38.189071+0000 mon.a (mon.0) 3513 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:38.541819+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:37.426245+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:37.427510+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:37.561074+0000 mgr.y (mgr.14553) 623 : cluster [DBG] pgmap v1138: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.001710+0000 osd.3 (osd.3) 321 : cluster [DBG] 6.b scrub starts 2024-02-10T08:48:38.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.002993+0000 osd.3 (osd.3) 322 : cluster [DBG] 6.b scrub ok 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.013076+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.a scrub starts 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.016611+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub ok 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:38.187596+0000 mon.a (mon.0) 3511 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.187680+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:38.188550+0000 mon.b (mon.1) 824 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:38.189071+0000 mon.a (mon.0) 3513 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:38.933 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:38.541819+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:37.426245+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:37.427510+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:37.561074+0000 mgr.y (mgr.14553) 623 : cluster [DBG] pgmap v1138: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.001710+0000 osd.3 (osd.3) 321 : cluster [DBG] 6.b scrub starts 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.002993+0000 osd.3 (osd.3) 322 : cluster [DBG] 6.b scrub ok 2024-02-10T08:48:39.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.013076+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.a scrub starts 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.016611+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub ok 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:38.187596+0000 mon.a (mon.0) 3511 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.187680+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:38.188550+0000 mon.b (mon.1) 824 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:38.189071+0000 mon.a (mon.0) 3513 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]: dispatch 2024-02-10T08:48:39.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:38.541819+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:48:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:39.168 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.188+0000 7f1d62ffd700 1 -- 172.21.15.107:0/4182699554 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v752) v1 ==== 113+0+0 (unknown 2305001699 0 0) 0x7f1d6804a490 con 0x7f1d6c0fd9f0 2024-02-10T08:48:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.190+0000 7f1d7414c700 10 librados: watch_flush enter 2024-02-10T08:48:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.190+0000 7f1d7414c700 10 librados: watch_flush exit 2024-02-10T08:48:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.190+0000 7f1d7414c700 20 client.28716.objecter shutdown clearing up homeless session... 2024-02-10T08:48:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.190+0000 7f1d7414c700 10 client.28716.objecter successfully canceled tick 2024-02-10T08:48:39.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.191+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1d58063c20 legacy=0x7f1d580660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.191+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 >> v1:172.21.15.107:6789/0 conn(0x7f1d6c0fd9f0 legacy=0x7f1d6c0fcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.192+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 shutdown_connections 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.192+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 >> 172.21.15.107:0/4182699554 conn(0x7f1d6c06c090 msgr2=0x7f1d6c1067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.192+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 shutdown_connections 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.192+0000 7f1d7414c700 1 -- 172.21.15.107:0/4182699554 wait complete. 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:39.192+0000 7f1d7414c700 1 librados: shutdown 2024-02-10T08:48:39.194 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T08:48:39.762 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:39.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:39.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:39.894 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.469784+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.471054+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.642424+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.d scrub starts 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:38.645298+0000 osd.1 (osd.1) 329 : cluster [DBG] 2.d scrub ok 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.003834+0000 osd.3 (osd.3) 323 : cluster [DBG] 234.1 scrub starts 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.006284+0000 osd.3 (osd.3) 324 : cluster [DBG] 234.1 scrub ok 2024-02-10T08:48:39.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.185520+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:39.189250+0000 mon.a (mon.0) 3516 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]': finished 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:39.189330+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.189377+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:39.191107+0000 mon.b (mon.1) 825 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:39.193061+0000 mon.a (mon.0) 3519 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.469784+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.471054+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.642424+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.d scrub starts 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:38.645298+0000 osd.1 (osd.1) 329 : cluster [DBG] 2.d scrub ok 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.003834+0000 osd.3 (osd.3) 323 : cluster [DBG] 234.1 scrub starts 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.006284+0000 osd.3 (osd.3) 324 : cluster [DBG] 234.1 scrub ok 2024-02-10T08:48:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.185520+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:39.189250+0000 mon.a (mon.0) 3516 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]': finished 2024-02-10T08:48:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:39.189330+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:48:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.189377+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-10T08:48:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:39.191107+0000 mon.b (mon.1) 825 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:39.193061+0000 mon.a (mon.0) 3519 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.469784+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub starts 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.471054+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.16 scrub ok 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.642424+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.d scrub starts 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:38.645298+0000 osd.1 (osd.1) 329 : cluster [DBG] 2.d scrub ok 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.003834+0000 osd.3 (osd.3) 323 : cluster [DBG] 234.1 scrub starts 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.006284+0000 osd.3 (osd.3) 324 : cluster [DBG] 234.1 scrub ok 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.185520+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:40.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:39.189250+0000 mon.a (mon.0) 3516 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-132", "mode": "writeback"}]': finished 2024-02-10T08:48:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:39.189330+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.107:0/4182699554' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:48:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.189377+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-10T08:48:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:39.191107+0000 mon.b (mon.1) 825 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:40.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:39.193061+0000 mon.a (mon.0) 3519 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:48:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:40.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:40.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:40.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:40.679 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:40.700 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:40.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:40.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.033133+0000 osd.2 (osd.2) 275 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.036692+0000 osd.2 (osd.2) 276 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:39.561672+0000 mgr.y (mgr.14553) 624 : cluster [DBG] pgmap v1141: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:40.047902+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:40.048942+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:40.171409+0000 mon.a (mon.0) 3520 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:40.195101+0000 mon.a (mon.0) 3521 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:40.195170+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-10T08:48:40.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:40.196173+0000 mon.b (mon.1) 826 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:40.196513+0000 mon.a (mon.0) 3523 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.033133+0000 osd.2 (osd.2) 275 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.036692+0000 osd.2 (osd.2) 276 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:39.561672+0000 mgr.y (mgr.14553) 624 : cluster [DBG] pgmap v1141: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:40.047902+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:40.048942+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:40.171409+0000 mon.a (mon.0) 3520 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:40.195101+0000 mon.a (mon.0) 3521 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:40.195170+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:40.196173+0000 mon.b (mon.1) 826 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:40.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:40.196513+0000 mon.a (mon.0) 3523 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:40.831 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.033133+0000 osd.2 (osd.2) 275 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.036692+0000 osd.2 (osd.2) 276 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:39.561672+0000 mgr.y (mgr.14553) 624 : cluster [DBG] pgmap v1141: 194 pgs: 19 creating+peering, 175 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:40.047902+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:40.048942+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:40.171409+0000 mon.a (mon.0) 3520 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:41.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:40.195101+0000 mon.a (mon.0) 3521 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:48:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:40.195170+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-10T08:48:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:40.196173+0000 mon.b (mon.1) 826 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:41.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:40.196513+0000 mon.a (mon.0) 3523 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:48:41.379 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:41.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:41.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:41.516 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:41.537 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:41.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:41.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:41.669 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:41.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:41.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:42.003 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:42.023 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:42.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:42.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:42.177 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:42.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:42.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:42.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:42.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:40.856918+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:40.860223+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.012627+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.013539+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.069330+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.070120+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.099052+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.102015+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:41.222135+0000 mon.a (mon.0) 3524 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.222186+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:41.223365+0000 mon.b (mon.1) 827 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:41.223688+0000 mon.a (mon.0) 3526 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.562057+0000 mgr.y (mgr.14553) 625 : cluster [DBG] pgmap v1144: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:42.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:41.798128+0000 mgr.y (mgr.14553) 626 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.940144+0000 osd.4 (osd.4) 51 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:41.943283+0000 osd.4 (osd.4) 52 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:40.856918+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:40.860223+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.012627+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.013539+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.069330+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.070120+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.099052+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.102015+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:48:42.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:41.222135+0000 mon.a (mon.0) 3524 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.222186+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:41.223365+0000 mon.b (mon.1) 827 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:41.223688+0000 mon.a (mon.0) 3526 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.562057+0000 mgr.y (mgr.14553) 625 : cluster [DBG] pgmap v1144: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:41.798128+0000 mgr.y (mgr.14553) 626 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.940144+0000 osd.4 (osd.4) 51 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:48:42.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:41.943283+0000 osd.4 (osd.4) 52 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:40.856918+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:40.860223+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.012627+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.013539+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.069330+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.070120+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.099052+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:48:42.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.102015+0000 osd.6 (osd.6) 307 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:41.222135+0000 mon.a (mon.0) 3524 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.222186+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:41.223365+0000 mon.b (mon.1) 827 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:41.223688+0000 mon.a (mon.0) 3526 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.562057+0000 mgr.y (mgr.14553) 625 : cluster [DBG] pgmap v1144: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:41.798128+0000 mgr.y (mgr.14553) 626 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.940144+0000 osd.4 (osd.4) 51 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:48:42.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:41.943283+0000 osd.4 (osd.4) 52 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:48:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:42.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:42.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T08:48:42.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:42.894 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:42.915 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:42.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:43.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:43.051 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:43.071 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:43.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:43.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:43.203 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:43.224 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:43.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.040096+0000 osd.5 (osd.5) 310 : cluster [DBG] 16.c scrub starts 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.043002+0000 osd.5 (osd.5) 311 : cluster [DBG] 16.c scrub ok 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.196227+0000 mon.a (mon.0) 3527 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:42.206089+0000 mon.a (mon.0) 3528 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.206144+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-10T08:48:43.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:42.207368+0000 mon.b (mon.1) 828 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:42.207938+0000 mon.a (mon.0) 3530 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.040096+0000 osd.5 (osd.5) 310 : cluster [DBG] 16.c scrub starts 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.043002+0000 osd.5 (osd.5) 311 : cluster [DBG] 16.c scrub ok 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.196227+0000 mon.a (mon.0) 3527 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:42.206089+0000 mon.a (mon.0) 3528 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.206144+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:42.207368+0000 mon.b (mon.1) 828 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.271 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:42.207938+0000 mon.a (mon.0) 3530 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:43.356 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:43.356 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.040096+0000 osd.5 (osd.5) 310 : cluster [DBG] 16.c scrub starts 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.043002+0000 osd.5 (osd.5) 311 : cluster [DBG] 16.c scrub ok 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.196227+0000 mon.a (mon.0) 3527 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:42.206089+0000 mon.a (mon.0) 3528 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.206144+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:42.207368+0000 mon.b (mon.1) 828 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:42.207938+0000 mon.a (mon.0) 3530 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:48:43.804 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:43.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:43.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.376+0000 7f0a1b873700 1 -- 172.21.15.107:0/3044124602 >> v1:172.21.15.107:6789/0 conn(0x7f0a140f7300 legacy=0x7f0a14104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.376+0000 7f0a1b873700 1 -- 172.21.15.107:0/3044124602 shutdown_connections 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.376+0000 7f0a1b873700 1 -- 172.21.15.107:0/3044124602 >> 172.21.15.107:0/3044124602 conn(0x7f0a140f2510 msgr2=0x7f0a140f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.376+0000 7f0a1b873700 1 -- 172.21.15.107:0/3044124602 shutdown_connections 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.376+0000 7f0a1b873700 1 -- 172.21.15.107:0/3044124602 wait complete. 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 librados: starting msgr at 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 librados: starting objecter 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 Processor -- start 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 -- start start 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 librados: setting wanted keys 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 librados: calling monclient init 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a14119560 con 0x7f0a14106640 2024-02-10T08:48:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a1411a740 con 0x7f0a140f7300 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1b873700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a1411b940 con 0x7f0a140f6810 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1960f700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0a140f6810 0x7f0a14102990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33414/0 (socket says 172.21.15.107:33414) 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a1960f700 1 -- 172.21.15.107:0/873596421 learned_addr learned my addr 172.21.15.107:0/873596421 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a19e10700 1 --1- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6789/0 conn(0x7f0a14106640 0x7f0a14117c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36022/0 (socket says 172.21.15.107:36022) 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a0a7fc700 10 client.?.objecter ms_handle_connect 0x7f0a140f6810 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a0a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a0a7fc700 10 client.?.objecter ms_handle_connect 0x7f0a14106640 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.377+0000 7f0a0a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 10 client.?.objecter ms_handle_connect 0x7f0a140f7300 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779721018 0 0) 0x7f0a1411b940 con 0x7f0a140f6810 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a00003030 con 0x7f0a140f6810 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948520365 0 0) 0x7f0a14119560 con 0x7f0a14106640 2024-02-10T08:48:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a1411b940 con 0x7f0a14106640 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635676545 0 0) 0x7f0a1411a740 con 0x7f0a140f7300 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a14119560 con 0x7f0a140f7300 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265983082 0 0) 0x7f0a1411b940 con 0x7f0a14106640 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a1411a740 con 0x7f0a14106640 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4218969498 0 0) 0x7f0a00003030 con 0x7f0a140f6810 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a1411b940 con 0x7f0a140f6810 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a0c001e00 con 0x7f0a14106640 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a040018c0 con 0x7f0a140f6810 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214244579 0 0) 0x7f0a14119560 con 0x7f0a140f7300 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a00003030 con 0x7f0a140f7300 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a10000fe0 con 0x7f0a140f7300 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 409086844 0 0) 0x7f0a1411a740 con 0x7f0a14106640 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6790/0 conn(0x7f0a140f6810 legacy=0x7f0a14102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.195:6789/0 conn(0x7f0a140f7300 legacy=0x7f0a141030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1411cb40 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a14119750 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a1b873700 10 client.28722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a14119c40 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a1b873700 1 librados: init done 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0a0c0027f0 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.378+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0a0c005de0 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.379+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f09f8004f00 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.379+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0a0c002350 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(756..756 src has 420..756) v4 ==== 8618+0+0 (unknown 179673854 0 0) 0x7f0a0c079270 con 0x7f0a14106640 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 osd_map(756..756 src has 420..756) v4 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 3 client.28722.objecter handle_osd_map got epochs [756,756] > 0 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 3 client.28722.objecter handle_osd_map decoding full epoch 756 2024-02-10T08:48:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 20 client.28722.objecter dump_active .. 0 homeless 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v756) v1 ==== 54+0+16 (unknown 3876480265 0 2420000192) 0x7f0a0c07b660 con 0x7f0a14106640 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f0a00063cf0 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 10 client.28722.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 1 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 1 2024-02-10T08:48:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 0 1 2024-02-10T08:48:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 4 2024-02-10T08:48:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 10 client.28722.objecter _send_command 1 2024-02-10T08:48:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f8004950 con 0x7f09f8004530 2024-02-10T08:48:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.380+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8004530 2024-02-10T08:48:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.381+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f8004950 con 0x7f09f8004530 2024-02-10T08:48:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.381+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.381+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 1 = system:0 2024-02-10T08:48:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.381+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 4 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.381+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 0 1 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 10 client.28722.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 2 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 2 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80051a0 osd=0 3 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 0 2 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 4 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 10 client.28722.objecter _send_command 2 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.396+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f8004d30 con 0x7f09f8004530 2024-02-10T08:48:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.397+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f8004d30 con 0x7f09f8004530 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.397+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.397+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.397+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 4 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.397+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 0 2 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 10 client.28722.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 1 3 2024-02-10T08:48:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 4 2024-02-10T08:48:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 10 client.28722.objecter _send_command 3 2024-02-10T08:48:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f800c000 con 0x7f09f8004150 2024-02-10T08:48:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8004150 2024-02-10T08:48:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f800c000 con 0x7f09f8004150 2024-02-10T08:48:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 3 = system:0 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 4 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.398+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 1 3 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 10 client.28722.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 4 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 4 2024-02-10T08:48:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8003fe0 osd=1 3 2024-02-10T08:48:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 1 4 2024-02-10T08:48:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 4 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 10 client.28722.objecter _send_command 4 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.413+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f80018b0 con 0x7f09f8004150 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f80018b0 con 0x7f09f8004150 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 4 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 1 4 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 10 client.28722.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 5 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 5 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 2 5 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 4 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 10 client.28722.objecter _send_command 5 2024-02-10T08:48:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.414+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f8003e40 con 0x7f09f800d2f0 2024-02-10T08:48:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:43.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f800d2f0 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f8003e40 con 0x7f09f800d2f0 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 5 = system:0 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 4 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.415+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 2 5 2024-02-10T08:48:43.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 10 client.28722.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 6 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 6 2024-02-10T08:48:43.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8003630 osd=2 3 2024-02-10T08:48:43.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 2 6 2024-02-10T08:48:43.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 4 2024-02-10T08:48:43.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 10 client.28722.objecter _send_command 6 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.430+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f800e720 con 0x7f09f800d2f0 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f800e720 con 0x7f09f800d2f0 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 4 2024-02-10T08:48:43.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 2 6 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 10 client.28722.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 7 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 7 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 3 7 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 4 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 10 client.28722.objecter _send_command 7 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f8015a10 con 0x7f09f8011e50 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.431+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8011e50 2024-02-10T08:48:43.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.432+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f8015a10 con 0x7f09f8011e50 2024-02-10T08:48:43.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.432+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:43.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.432+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 7 = system:0 2024-02-10T08:48:43.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.432+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 4 2024-02-10T08:48:43.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.432+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 3 7 2024-02-10T08:48:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 10 client.28722.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 8 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 8 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80033c0 osd=3 3 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 3 8 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 4 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 10 client.28722.objecter _send_command 8 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.447+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f800e190 con 0x7f09f8011e50 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.448+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f800e190 con 0x7f09f8011e50 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.448+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.448+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.448+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 4 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.448+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 3 8 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 10 client.28722.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 9 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 9 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 4 9 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 4 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 10 client.28722.objecter _send_command 9 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f800e5a0 con 0x7f09f800ce50 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.449+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f800ce50 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.451+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0a04006260 con 0x7f09f800ce50 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.451+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.451+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 9 = system:0 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.451+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 4 2024-02-10T08:48:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.451+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 4 9 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 10 client.28722.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 10 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 10 2024-02-10T08:48:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f800e190 osd=4 3 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 4 10 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 4 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 10 client.28722.objecter _send_command 10 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.464+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f80161c0 con 0x7f09f800ce50 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f80161c0 con 0x7f09f800ce50 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 4 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 4 10 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 10 client.28722.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 11 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 11 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 5 11 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 4 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 10 client.28722.objecter _send_command 11 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f8016e50 con 0x7f09f8016880 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8016880 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f8016e50 con 0x7f09f8016880 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.467+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 11 = system:0 2024-02-10T08:48:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.468+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 4 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.468+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 5 11 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 10 client.28722.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 12 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 12 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f80161c0 osd=5 3 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 5 12 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 4 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 10 client.28722.objecter _send_command 12 2024-02-10T08:48:43.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.482+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f8016e50 con 0x7f09f8016880 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f8016e50 con 0x7f09f8016880 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 4 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 5 12 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 10 client.28722.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 13 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 13 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 6 13 2024-02-10T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 4 2024-02-10T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 10 client.28722.objecter _send_command 13 2024-02-10T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f801fb80 con 0x7f09f8020620 2024-02-10T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.484+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8020620 2024-02-10T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.485+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f801fb80 con 0x7f09f8020620 2024-02-10T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.485+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.485+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 13 = system:0 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.485+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 4 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.485+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 6 13 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 10 client.28722.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 14 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 14 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f8016e50 osd=6 3 2024-02-10T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 6 14 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 4 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 10 client.28722.objecter _send_command 14 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.499+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f801fb80 con 0x7f09f8020620 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.501+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f801fb80 con 0x7f09f8020620 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.501+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.501+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.501+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 4 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.501+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 6 14 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 10 client.28722.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 15 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 15 2024-02-10T08:48:43.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 7 15 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 4 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 10 client.28722.objecter _send_command 15 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09f8027330 con 0x7f09f8023820 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.502+0000 7f0a0a7fc700 10 client.28722.objecter ms_handle_connect 0x7f09f8023820 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.504+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f09f8027330 con 0x7f09f8023820 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.504+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.504+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 15 = system:0 2024-02-10T08:48:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.504+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 4 2024-02-10T08:48:43.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.504+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 7 15 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 10 client.28722.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign -1 16 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 20 client.28722.objecter _get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_remove -1 16 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 20 client.28722.objecter get_session s=0x7f09f801fb80 osd=7 3 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 15 client.28722.objecter _session_command_op_assign 7 16 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 4 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 10 client.28722.objecter _send_command 16 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.518+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f09f8027330 con 0x7f09f8023820 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a0a7fc700 1 -- 172.21.15.107:0/873596421 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f09f8027330 con 0x7f09f8023820 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a0a7fc700 10 client.28722.objecter ms_dispatch 0x7f0a14001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:48:43.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a0a7fc700 10 client.28722.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a0a7fc700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 4 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a0a7fc700 15 client.28722.objecter _session_command_op_remove 7 16 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 librados: watch_flush enter 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 librados: watch_flush exit 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.0 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6801/3321596972 conn(0x7f09f8004530 legacy=0x7f09f8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80051a0 osd=0 1 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.1 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6805/133516126 conn(0x7f09f8004150 legacy=0x7f09f800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003fe0 osd=1 1 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.2 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6809/3575881728 conn(0x7f09f800d2f0 legacy=0x7f09f800d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8003630 osd=2 1 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.3 2024-02-10T08:48:43.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6813/2871825691 conn(0x7f09f8011e50 legacy=0x7f09f8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80033c0 osd=3 1 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.4 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.195:6800/3310789251 conn(0x7f09f800ce50 legacy=0x7f09f8015a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a19e10700 1 -- 172.21.15.107:0/873596421 reap_dead start 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f800e190 osd=4 1 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.5 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.195:6804/4178321486 conn(0x7f09f8016880 legacy=0x7f09f801d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a19e10700 1 -- 172.21.15.107:0/873596421 reap_dead start 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f80161c0 osd=5 1 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.6 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.195:6808/1417479017 conn(0x7f09f8020620 legacy=0x7f09f8020aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f8016e50 osd=6 1 2024-02-10T08:48:43.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter close_session for osd.7 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.195:6812/1934176928 conn(0x7f09f8023820 legacy=0x7f09f8025c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter put_session s=0x7f09f801fb80 osd=7 1 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 20 client.28722.objecter shutdown clearing up homeless session... 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 10 client.28722.objecter successfully canceled tick 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a19e10700 1 -- 172.21.15.107:0/873596421 reap_dead start 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0a00063cf0 legacy=0x7f0a000661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> v1:172.21.15.107:6789/0 conn(0x7f0a14106640 legacy=0x7f0a14117c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 shutdown_connections 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 >> 172.21.15.107:0/873596421 conn(0x7f0a140f2510 msgr2=0x7f0a14109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 shutdown_connections 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.519+0000 7f0a1b873700 1 -- 172.21.15.107:0/873596421 wait complete. 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:43.520+0000 7f0a1b873700 1 librados: shutdown 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:43.990 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:43.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:44.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:44.121 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:44.141 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:44.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:44.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:44.273 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:44.292 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:44.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:44.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.680595+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:48:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.684177+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:48:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.981890+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:48:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:42.984815+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:48:44.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.030013+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.032091+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.078104+0000 osd.6 (osd.6) 308 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.081144+0000 osd.6 (osd.6) 309 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:43.217636+0000 mon.a (mon.0) 3531 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.217748+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:43.218653+0000 mon.b (mon.1) 829 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:43.218994+0000 mon.a (mon.0) 3533 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:43.381754+0000 mon.a (mon.0) 3534 : audit [DBG] from='client.? v1:172.21.15.107:0/873596421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:44.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:43.562516+0000 mgr.y (mgr.14553) 627 : cluster [DBG] pgmap v1147: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:44.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.680595+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:48:44.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.684177+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:48:44.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.981890+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:48:44.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:42.984815+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:48:44.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.030013+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.032091+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.078104+0000 osd.6 (osd.6) 308 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.081144+0000 osd.6 (osd.6) 309 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:43.217636+0000 mon.a (mon.0) 3531 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.217748+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:43.218653+0000 mon.b (mon.1) 829 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:43.218994+0000 mon.a (mon.0) 3533 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:43.381754+0000 mon.a (mon.0) 3534 : audit [DBG] from='client.? v1:172.21.15.107:0/873596421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:43.562516+0000 mgr.y (mgr.14553) 627 : cluster [DBG] pgmap v1147: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.680595+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:48:44.638 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.684177+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.981890+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:42.984815+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.030013+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.032091+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.078104+0000 osd.6 (osd.6) 308 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.081144+0000 osd.6 (osd.6) 309 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:43.217636+0000 mon.a (mon.0) 3531 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.217748+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:43.218653+0000 mon.b (mon.1) 829 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:43.218994+0000 mon.a (mon.0) 3533 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:43.381754+0000 mon.a (mon.0) 3534 : audit [DBG] from='client.? v1:172.21.15.107:0/873596421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:44.639 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:43.562516+0000 mgr.y (mgr.14553) 627 : cluster [DBG] pgmap v1147: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-02-10T08:48:44.671 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:44.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:44.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:44.801 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:44.874 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:44.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:44.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:45.009 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:45.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:45.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:45.164 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:44.067220+0000 osd.6 (osd.6) 310 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:48:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:44.070722+0000 osd.6 (osd.6) 311 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:48:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:44.223466+0000 mon.a (mon.0) 3535 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:48:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:44.223541+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-10T08:48:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:44.224492+0000 mon.b (mon.1) 830 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:44.225157+0000 mon.a (mon.0) 3537 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:44.935717+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:48:45.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:44.939184+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:44.067220+0000 osd.6 (osd.6) 310 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:44.070722+0000 osd.6 (osd.6) 311 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:44.223466+0000 mon.a (mon.0) 3535 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:44.223541+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:44.224492+0000 mon.b (mon.1) 830 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:44.225157+0000 mon.a (mon.0) 3537 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:44.935717+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:48:45.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:44.939184+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:44.067220+0000 osd.6 (osd.6) 310 : cluster [DBG] 234.2 scrub starts 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:44.070722+0000 osd.6 (osd.6) 311 : cluster [DBG] 234.2 scrub ok 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:44.223466+0000 mon.a (mon.0) 3535 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:44.223541+0000 mon.a (mon.0) 3536 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:44.224492+0000 mon.b (mon.1) 830 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:44.225157+0000 mon.a (mon.0) 3537 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:44.935717+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:48:45.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:44.939184+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:48:45.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T08:48:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.206+0000 7fa546465700 1 -- 172.21.15.107:0/2259921435 >> v1:172.21.15.107:6790/0 conn(0x7fa5400fc910 legacy=0x7fa5400fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.206+0000 7fa546465700 1 -- 172.21.15.107:0/2259921435 shutdown_connections 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.206+0000 7fa546465700 1 -- 172.21.15.107:0/2259921435 >> 172.21.15.107:0/2259921435 conn(0x7fa5400f2510 msgr2=0x7fa5400f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.206+0000 7fa546465700 1 -- 172.21.15.107:0/2259921435 shutdown_connections 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.206+0000 7fa546465700 1 -- 172.21.15.107:0/2259921435 wait complete. 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 librados: starting msgr at 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 librados: starting objecter 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 Processor -- start 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 -- start start 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 librados: setting wanted keys 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 librados: calling monclient init 2024-02-10T08:48:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5401152d0 con 0x7fa5400fc910 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5401164b0 con 0x7fa5400ff710 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa546465700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5401176b0 con 0x7fa540103250 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa53ffff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa5400fc910 0x7fa5400f84a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36036/0 (socket says 172.21.15.107:36036) 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa53ffff700 1 -- 172.21.15.107:0/1111164301 learned_addr learned my addr 172.21.15.107:0/1111164301 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa53d7fa700 10 client.?.objecter ms_handle_connect 0x7fa540103250 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.207+0000 7fa53d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 10 client.?.objecter ms_handle_connect 0x7fa5400fc910 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 10 client.?.objecter ms_handle_connect 0x7fa5400ff710 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230793648 0 0) 0x7fa5401176b0 con 0x7fa540103250 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5280033f0 con 0x7fa540103250 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471249616 0 0) 0x7fa5401152d0 con 0x7fa5400fc910 2024-02-10T08:48:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5401176b0 con 0x7fa5400fc910 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023480841 0 0) 0x7fa5401164b0 con 0x7fa5400ff710 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5401152d0 con 0x7fa5400ff710 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2896531061 0 0) 0x7fa5280033f0 con 0x7fa540103250 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5401164b0 con 0x7fa540103250 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa538001cb0 con 0x7fa540103250 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513733917 0 0) 0x7fa5401176b0 con 0x7fa5400fc910 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5280033f0 con 0x7fa5400fc910 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa530002050 con 0x7fa5400fc910 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988517435 0 0) 0x7fa5401152d0 con 0x7fa5400ff710 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5401176b0 con 0x7fa5400ff710 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa534001880 con 0x7fa5400ff710 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 549192905 0 0) 0x7fa5401164b0 con 0x7fa540103250 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 >> v1:172.21.15.195:6789/0 conn(0x7fa5400ff710 legacy=0x7fa5400f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 >> v1:172.21.15.107:6789/0 conn(0x7fa5400fc910 legacy=0x7fa5400f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5401188b0 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5401154c0 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa546465700 10 client.54323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5401159b0 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.208+0000 7fa546465700 1 librados: init done 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.209+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa5380026e0 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.209+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa5380035f0 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.209+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c004f20 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa538016480 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(757..757 src has 420..757) v4 ==== 8618+0+0 (unknown 2974620360 0 0) 0x7fa538076b50 con 0x7fa540103250 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 10 client.54323.objecter ms_dispatch 0x7fa540001550 osd_map(757..757 src has 420..757) v4 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 3 client.54323.objecter handle_osd_map got epochs [757,757] > 0 2024-02-10T08:48:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 3 client.54323.objecter handle_osd_map decoding full epoch 757 2024-02-10T08:48:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 20 client.54323.objecter dump_active .. 0 homeless 2024-02-10T08:48:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.210+0000 7fa53d7fa700 10 client.54323.objecter ms_handle_connect 0x7fa528065f50 2024-02-10T08:48:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.215+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa538048f30 con 0x7fa540103250 2024-02-10T08:48:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:45.405+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa52c0051d0 con 0x7fa540103250 2024-02-10T08:48:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:45.612 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:45.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:45.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:45] "GET /metrics HTTP/1.1" 200 190952 "" "Prometheus/2.33.4" 2024-02-10T08:48:45.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:45.749 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:45.771 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:45.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:45.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:45.906 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:45.927 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:45.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:46.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:46.060 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:46.080 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:46.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.172+0000 7fa53d7fa700 1 -- 172.21.15.107:0/1111164301 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v759) v1 ==== 101+0+0 (unknown 1649550868 0 0) 0x7fa53804f8b0 con 0x7fa540103250 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.176+0000 7fa546465700 10 librados: watch_flush enter 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.176+0000 7fa546465700 10 librados: watch_flush exit 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.176+0000 7fa546465700 20 client.54323.objecter shutdown clearing up homeless session... 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.176+0000 7fa546465700 10 client.54323.objecter successfully canceled tick 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.176+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa528065f50 legacy=0x7fa528068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.177+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 >> v1:172.21.15.107:6790/0 conn(0x7fa540103250 legacy=0x7fa540113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:46.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.177+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 shutdown_connections 2024-02-10T08:48:46.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.177+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 >> 172.21.15.107:0/1111164301 conn(0x7fa5400f2510 msgr2=0x7fa540060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:46.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.178+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 shutdown_connections 2024-02-10T08:48:46.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.178+0000 7fa546465700 1 -- 172.21.15.107:0/1111164301 wait complete. 2024-02-10T08:48:46.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:46.178+0000 7fa546465700 1 librados: shutdown 2024-02-10T08:48:46.179 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:48:46.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:46.212 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:46.231 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:46.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:46.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:46.362 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:46.460 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:46.461 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.087999+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.088958+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:45.223259+0000 mon.a (mon.0) 3538 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.223328+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:45.271872+0000 mon.b (mon.1) 831 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:45.272214+0000 mon.a (mon.0) 3540 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:45.405929+0000 mon.c (mon.2) 478 : audit [INF] from='client.? v1:172.21.15.107:0/1111164301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:45.406214+0000 mon.a (mon.0) 3541 : audit [INF] from='client.54323 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.562867+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.562953+0000 mgr.y (mgr.14553) 628 : cluster [DBG] pgmap v1150: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.563120+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.966518+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.969413+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:46.168466+0000 mon.a (mon.0) 3542 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:48:46.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:46.172989+0000 mon.a (mon.0) 3543 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:46.173056+0000 mon.a (mon.0) 3544 : audit [INF] from='client.54323 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:46.173112+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.087999+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.088958+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:45.223259+0000 mon.a (mon.0) 3538 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.223328+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:45.271872+0000 mon.b (mon.1) 831 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:45.272214+0000 mon.a (mon.0) 3540 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:45.405929+0000 mon.c (mon.2) 478 : audit [INF] from='client.? v1:172.21.15.107:0/1111164301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:45.406214+0000 mon.a (mon.0) 3541 : audit [INF] from='client.54323 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.562867+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.562953+0000 mgr.y (mgr.14553) 628 : cluster [DBG] pgmap v1150: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.563120+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.966518+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.969413+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:46.168466+0000 mon.a (mon.0) 3542 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:46.172989+0000 mon.a (mon.0) 3543 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:46.173056+0000 mon.a (mon.0) 3544 : audit [INF] from='client.54323 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:46.173112+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:46.174368+0000 mon.b (mon.1) 832 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.464 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:46.175131+0000 mon.a (mon.0) 3546 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:46.174368+0000 mon.b (mon.1) 832 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.465 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:46.175131+0000 mon.a (mon.0) 3546 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:46.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.087999+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:48:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.088958+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:48:46.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:45.223259+0000 mon.a (mon.0) 3538 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.223328+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:45.271872+0000 mon.b (mon.1) 831 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:45.272214+0000 mon.a (mon.0) 3540 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:45.405929+0000 mon.c (mon.2) 478 : audit [INF] from='client.? v1:172.21.15.107:0/1111164301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:45.406214+0000 mon.a (mon.0) 3541 : audit [INF] from='client.54323 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.562867+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.562953+0000 mgr.y (mgr.14553) 628 : cluster [DBG] pgmap v1150: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.563120+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.966518+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.969413+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:46.168466+0000 mon.a (mon.0) 3542 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:46.172989+0000 mon.a (mon.0) 3543 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:46.173056+0000 mon.a (mon.0) 3544 : audit [INF] from='client.54323 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:48:46.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:46.173112+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-10T08:48:46.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:46.174368+0000 mon.b (mon.1) 832 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:46.175131+0000 mon.a (mon.0) 3546 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:46.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:46.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:48:46.664 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:46.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:46.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:46.798 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:46.817 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:46.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:46.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:46.947 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:47.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:47.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.868816+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.871768+0000 osd.7 (osd.7) 305 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.996619+0000 osd.6 (osd.6) 312 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:45.997361+0000 osd.6 (osd.6) 313 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:46.912071+0000 mon.c (mon.2) 479 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.175149+0000 mon.a (mon.0) 3547 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:47.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.175238+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.198741+0000 mon.b (mon.1) 833 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.199122+0000 mon.a (mon.0) 3549 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.199613+0000 mon.b (mon.1) 834 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.199889+0000 mon.a (mon.0) 3550 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.868816+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.871768+0000 osd.7 (osd.7) 305 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.996619+0000 osd.6 (osd.6) 312 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:45.997361+0000 osd.6 (osd.6) 313 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:46.912071+0000 mon.c (mon.2) 479 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:48:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.175149+0000 mon.a (mon.0) 3547 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.175238+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.198741+0000 mon.b (mon.1) 833 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.199122+0000 mon.a (mon.0) 3549 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.199613+0000 mon.b (mon.1) 834 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.199889+0000 mon.a (mon.0) 3550 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.868816+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.871768+0000 osd.7 (osd.7) 305 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.996619+0000 osd.6 (osd.6) 312 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:45.997361+0000 osd.6 (osd.6) 313 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:48:47.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:46.912071+0000 mon.c (mon.2) 479 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.175149+0000 mon.a (mon.0) 3547 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]': finished 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.175238+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.198741+0000 mon.b (mon.1) 833 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.199122+0000 mon.a (mon.0) 3549 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.199613+0000 mon.b (mon.1) 834 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.199889+0000 mon.a (mon.0) 3550 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-132"}]: dispatch 2024-02-10T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:47.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:47.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:47.682 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:47.700 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:47.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:47.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.078207+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.3 deep-scrub starts 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.081533+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.3 deep-scrub ok 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.159970+0000 osd.2 (osd.2) 279 : cluster [DBG] 6.f scrub starts 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.163007+0000 osd.2 (osd.2) 280 : cluster [DBG] 6.f scrub ok 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.285902+0000 mon.a (mon.0) 3551 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.563415+0000 mgr.y (mgr.14553) 629 : cluster [DBG] pgmap v1153: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.635260+0000 mon.c (mon.2) 480 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.635910+0000 mon.c (mon.2) 481 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:47.640215+0000 mon.a (mon.0) 3552 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.971425+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.974386+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.176076+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.078207+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.3 deep-scrub starts 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.081533+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.3 deep-scrub ok 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.159970+0000 osd.2 (osd.2) 279 : cluster [DBG] 6.f scrub starts 2024-02-10T08:48:48.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.163007+0000 osd.2 (osd.2) 280 : cluster [DBG] 6.f scrub ok 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.285902+0000 mon.a (mon.0) 3551 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.563415+0000 mgr.y (mgr.14553) 629 : cluster [DBG] pgmap v1153: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.635260+0000 mon.c (mon.2) 480 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.635910+0000 mon.c (mon.2) 481 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:47.640215+0000 mon.a (mon.0) 3552 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.971425+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.974386+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:48:48.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.176076+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.078207+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.3 deep-scrub starts 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.081533+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.3 deep-scrub ok 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.159970+0000 osd.2 (osd.2) 279 : cluster [DBG] 6.f scrub starts 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.163007+0000 osd.2 (osd.2) 280 : cluster [DBG] 6.f scrub ok 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.285902+0000 mon.a (mon.0) 3551 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.563415+0000 mgr.y (mgr.14553) 629 : cluster [DBG] pgmap v1153: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.635260+0000 mon.c (mon.2) 480 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:48:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.635910+0000 mon.c (mon.2) 481 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:48:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:47.640215+0000 mon.a (mon.0) 3552 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.971425+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:48:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.974386+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:48:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.176076+0000 mon.a (mon.0) 3553 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-10T08:48:48.892 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:49.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:48:49.386 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:49.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:49.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.786430+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:47.787347+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.075961+0000 osd.5 (osd.5) 314 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.079265+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.417315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.959827+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.e scrub starts 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:48.962816+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.e scrub ok 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:49.183100+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:49.186667+0000 mon.b (mon.1) 835 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:49.187187+0000 mon.a (mon.0) 3556 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.786430+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:48:49.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:47.787347+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.075961+0000 osd.5 (osd.5) 314 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.079265+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.417315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.959827+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.e scrub starts 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:48.962816+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.e scrub ok 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:49.183100+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:49.186667+0000 mon.b (mon.1) 835 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:49.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:49.187187+0000 mon.a (mon.0) 3556 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:49.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:49.520 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.786430+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.5 deep-scrub starts 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:47.787347+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.5 deep-scrub ok 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.075961+0000 osd.5 (osd.5) 314 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.079265+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.417315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.959827+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.e scrub starts 2024-02-10T08:48:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:48.962816+0000 osd.4 (osd.4) 64 : cluster [DBG] 6.e scrub ok 2024-02-10T08:48:49.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:49.183100+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-10T08:48:49.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:49.186667+0000 mon.b (mon.1) 835 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:49.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:49.187187+0000 mon.a (mon.0) 3556 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:48:50.468 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:49.563893+0000 mgr.y (mgr.14553) 630 : cluster [DBG] pgmap v1156: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.098690+0000 osd.3 (osd.3) 333 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.099942+0000 osd.3 (osd.3) 334 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.179786+0000 mon.a (mon.0) 3557 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.179907+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.199184+0000 mon.b (mon.1) 836 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.199995+0000 mon.a (mon.0) 3559 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:49.563893+0000 mgr.y (mgr.14553) 630 : cluster [DBG] pgmap v1156: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.098690+0000 osd.3 (osd.3) 333 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.099942+0000 osd.3 (osd.3) 334 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:48:50.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.179786+0000 mon.a (mon.0) 3557 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:50.472 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.179907+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-10T08:48:50.472 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.199184+0000 mon.b (mon.1) 836 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.472 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.199995+0000 mon.a (mon.0) 3559 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:50.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:49.563893+0000 mgr.y (mgr.14553) 630 : cluster [DBG] pgmap v1156: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.098690+0000 osd.3 (osd.3) 333 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.099942+0000 osd.3 (osd.3) 334 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.179786+0000 mon.a (mon.0) 3557 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.179907+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.199184+0000 mon.b (mon.1) 836 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.199995+0000 mon.a (mon.0) 3559 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:48:50.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:50.602 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:50.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.742+0000 7f71353cf700 1 -- 172.21.15.107:0/253538039 >> v1:172.21.15.107:6790/0 conn(0x7f71301007f0 legacy=0x7f7130102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 -- 172.21.15.107:0/253538039 shutdown_connections 2024-02-10T08:48:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 -- 172.21.15.107:0/253538039 >> 172.21.15.107:0/253538039 conn(0x7f71300f3640 msgr2=0x7f71300f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 -- 172.21.15.107:0/253538039 shutdown_connections 2024-02-10T08:48:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 -- 172.21.15.107:0/253538039 wait complete. 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 librados: starting msgr at 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 librados: starting objecter 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.743+0000 7f71353cf700 1 Processor -- start 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 -- start start 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 librados: setting wanted keys 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 librados: calling monclient init 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301173c0 con 0x7f7130104330 2024-02-10T08:48:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301185a0 con 0x7f71301007f0 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f71353cf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71301197a0 con 0x7f71300fda50 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f712effd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f71300fda50 0x7f71300f99b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42492/0 (socket says 172.21.15.107:42492) 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f712effd700 1 -- 172.21.15.107:0/1818583645 learned_addr learned my addr 172.21.15.107:0/1818583645 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f7130104330 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f71300fda50 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9101315 0 0) 0x7f71301173c0 con 0x7f7130104330 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7118003800 con 0x7f7130104330 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725970201 0 0) 0x7f71301197a0 con 0x7f71300fda50 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71301173c0 con 0x7f71300fda50 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter ms_handle_connect 0x7f71301007f0 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216713041 0 0) 0x7f71301185a0 con 0x7f71301007f0 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.744+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71301197a0 con 0x7f71301007f0 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530134130 0 0) 0x7f71301173c0 con 0x7f71300fda50 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71301185a0 con 0x7f71300fda50 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f71200019f0 con 0x7f71300fda50 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032967061 0 0) 0x7f7118003800 con 0x7f7130104330 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71301173c0 con 0x7f7130104330 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7128001b80 con 0x7f7130104330 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007390044 0 0) 0x7f71301197a0 con 0x7f71301007f0 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7118003800 con 0x7f71301007f0 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7124002060 con 0x7f71301007f0 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2266506263 0 0) 0x7f71301185a0 con 0x7f71300fda50 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.195:6789/0 conn(0x7f71301007f0 legacy=0x7f71300fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6789/0 conn(0x7f7130104330 legacy=0x7f7130115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713011a9a0 con 0x7f71300fda50 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7130119990 con 0x7f71300fda50 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f71353cf700 10 client.54335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7130119e00 con 0x7f71300fda50 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f71353cf700 1 librados: init done 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7120002380 con 0x7f71300fda50 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.745+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7120005ac0 con 0x7f71300fda50 2024-02-10T08:48:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.746+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f711c004f00 con 0x7f71300fda50 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.746+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7120001ee0 con 0x7f71300fda50 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(763..763 src has 420..763) v4 ==== 8597+0+0 (unknown 1163368776 0 0) 0x7f7120078ed0 con 0x7f71300fda50 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 osd_map(763..763 src has 420..763) v4 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 3 client.54335.objecter handle_osd_map got epochs [763,763] > 0 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 3 client.54335.objecter handle_osd_map decoding full epoch 763 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 20 client.54335.objecter dump_active .. 0 homeless 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v763) v1 ==== 54+0+16 (unknown 2229522718 0 2420000192) 0x7f712007b2e0 con 0x7f71300fda50 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f7118063c10 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 10 client.54335.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 1 2024-02-10T08:48:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 1 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 0 1 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 10 client.54335.objecter _send_command 1 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c004950 con 0x7f711c004530 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.747+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c004530 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.748+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c004950 con 0x7f711c004530 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.748+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.748+0000 7f7117fff700 10 client.54335.objecter _finish_command 1 = system:0 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.748+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:48:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.748+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 0 1 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 10 client.54335.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 2 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 2 2024-02-10T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c0051a0 osd=0 3 2024-02-10T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 0 2 2024-02-10T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 10 client.54335.objecter _send_command 2 2024-02-10T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.764+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c004d30 con 0x7f711c004530 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.765+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c004d30 con 0x7f711c004530 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.765+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.765+0000 7f7117fff700 10 client.54335.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.765+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 4 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.765+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 0 2 2024-02-10T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 10 client.54335.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 1 3 2024-02-10T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 10 client.54335.objecter _send_command 3 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c00c000 con 0x7f711c004150 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.766+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c004150 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.767+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c00c000 con 0x7f711c004150 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.767+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.767+0000 7f7117fff700 10 client.54335.objecter _finish_command 3 = system:0 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.767+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.767+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 1 3 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 10 client.54335.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 4 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 4 2024-02-10T08:48:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c003fe0 osd=1 3 2024-02-10T08:48:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 1 4 2024-02-10T08:48:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:48:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 10 client.54335.objecter _send_command 4 2024-02-10T08:48:50.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.782+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c0018b0 con 0x7f711c004150 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c0018b0 con 0x7f711c004150 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f7117fff700 10 client.54335.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 4 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 1 4 2024-02-10T08:48:50.787 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 10 client.54335.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 5 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 5 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 2 5 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 10 client.54335.objecter _send_command 5 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.784+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c003e70 con 0x7f711c00d2f0 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c00d2f0 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c003e70 con 0x7f711c00d2f0 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 10 client.54335.objecter _finish_command 5 = system:0 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:48:50.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.785+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 2 5 2024-02-10T08:48:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 10 client.54335.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 6 2024-02-10T08:48:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 6 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c003630 osd=2 3 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 2 6 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 10 client.54335.objecter _send_command 6 2024-02-10T08:48:50.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.800+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c00e710 con 0x7f711c00d2f0 2024-02-10T08:48:50.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c00e710 con 0x7f711c00d2f0 2024-02-10T08:48:50.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f7117fff700 10 client.54335.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.802 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 4 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 2 6 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 10 client.54335.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 7 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 7 2024-02-10T08:48:50.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 3 7 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 4 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 10 client.54335.objecter _send_command 7 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.801+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c015a40 con 0x7f711c011e80 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c011e80 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c015a40 con 0x7f711c011e80 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 10 client.54335.objecter _finish_command 7 = system:0 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 4 2024-02-10T08:48:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.802+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 3 7 2024-02-10T08:48:50.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 10 client.54335.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 8 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 8 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c0033c0 osd=3 3 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 3 8 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 4 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 10 client.54335.objecter _send_command 8 2024-02-10T08:48:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.817+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c00e180 con 0x7f711c011e80 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.818+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c00e180 con 0x7f711c011e80 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.818+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.818+0000 7f7117fff700 10 client.54335.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.818+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 4 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.818+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 3 8 2024-02-10T08:48:50.819 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 10 client.54335.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 9 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 9 2024-02-10T08:48:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 4 9 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 4 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 10 client.54335.objecter _send_command 9 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c00e4c0 con 0x7f711c00ce50 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.819+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c00ce50 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.820+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c00e4c0 con 0x7f711c00ce50 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.820+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.820+0000 7f7117fff700 10 client.54335.objecter _finish_command 9 = system:0 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.820+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 4 2024-02-10T08:48:50.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.820+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 4 9 2024-02-10T08:48:50.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 10 client.54335.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 10 2024-02-10T08:48:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 10 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c00e180 osd=4 3 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 4 10 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 4 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 10 client.54335.objecter _send_command 10 2024-02-10T08:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.835+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c00e4c0 con 0x7f711c00ce50 2024-02-10T08:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c00e4c0 con 0x7f711c00ce50 2024-02-10T08:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f7117fff700 10 client.54335.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 4 2024-02-10T08:48:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 4 10 2024-02-10T08:48:50.838 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 10 client.54335.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 11 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 11 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 5 11 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 4 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 10 client.54335.objecter _send_command 11 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.837+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c016d70 con 0x7f711c016800 2024-02-10T08:48:50.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c016800 2024-02-10T08:48:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c016d70 con 0x7f711c016800 2024-02-10T08:48:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:48:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 10 client.54335.objecter _finish_command 11 = system:0 2024-02-10T08:48:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 4 2024-02-10T08:48:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.838+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 5 11 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.674+0000 7fe6d1f14700 1 -- 172.21.15.107:0/64236925 >> v1:172.21.15.107:6790/0 conn(0x7fe6cc104330 legacy=0x7fe6cc1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.674+0000 7fe6d1f14700 1 -- 172.21.15.107:0/64236925 shutdown_connections 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.674+0000 7fe6d1f14700 1 -- 172.21.15.107:0/64236925 >> 172.21.15.107:0/64236925 conn(0x7fe6cc0f3640 msgr2=0x7fe6cc0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.675+0000 7fe6d1f14700 1 -- 172.21.15.107:0/64236925 shutdown_connections 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.675+0000 7fe6d1f14700 1 -- 172.21.15.107:0/64236925 wait complete. 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.675+0000 7fe6d1f14700 1 librados: starting msgr at 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.675+0000 7fe6d1f14700 1 librados: starting objecter 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.675+0000 7fe6d1f14700 1 Processor -- start 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 -- start start 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 librados: setting wanted keys 2024-02-10T08:48:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 librados: calling monclient init 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6cc117250 con 0x7fe6cc0fda50 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6cc118430 con 0x7fe6cc104330 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6d1f14700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6cc119630 con 0x7fe6cc1007f0 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6cb7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe6cc0fda50 0x7fe6cc0f9830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50654/0 (socket says 172.21.15.107:50654) 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6cb7fe700 1 -- 172.21.15.107:0/2276380318 learned_addr learned my addr 172.21.15.107:0/2276380318 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter ms_handle_connect 0x7fe6cc1007f0 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter ms_handle_connect 0x7fe6cc0fda50 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278308237 0 0) 0x7fe6cc119630 con 0x7fe6cc1007f0 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b4002220 con 0x7fe6cc1007f0 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789240022 0 0) 0x7fe6cc117250 con 0x7fe6cc0fda50 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6cc119630 con 0x7fe6cc0fda50 2024-02-10T08:48:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter ms_handle_connect 0x7fe6cc104330 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345512571 0 0) 0x7fe6b4002220 con 0x7fe6cc1007f0 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6cc117250 con 0x7fe6cc1007f0 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6c00016b0 con 0x7fe6cc1007f0 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116558661 0 0) 0x7fe6cc119630 con 0x7fe6cc0fda50 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6b4002220 con 0x7fe6cc0fda50 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.676+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6bc002880 con 0x7fe6cc0fda50 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642859281 0 0) 0x7fe6cc118430 con 0x7fe6cc104330 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6cc119630 con 0x7fe6cc104330 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3142210368 0 0) 0x7fe6cc117250 con 0x7fe6cc1007f0 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 >> v1:172.21.15.195:6789/0 conn(0x7fe6cc104330 legacy=0x7fe6cc1159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 >> v1:172.21.15.107:6789/0 conn(0x7fe6cc0fda50 legacy=0x7fe6cc0f9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc11a830 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc117440 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6d1f14700 10 client.54329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc1178b0 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6d1f14700 1 librados: init done 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe6c0001c00 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.677+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6c0003240 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.678+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8004f20 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.678+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe6c0002030 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(763..763 src has 420..763) v4 ==== 8597+0+0 (unknown 1163368776 0 0) 0x7fe6c0076590 con 0x7fe6cc1007f0 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 10 client.54329.objecter ms_dispatch 0x7fe6cc001550 osd_map(763..763 src has 420..763) v4 2024-02-10T08:48:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 3 client.54329.objecter handle_osd_map got epochs [763,763] > 0 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 3 client.54329.objecter handle_osd_map decoding full epoch 763 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 20 client.54329.objecter dump_active .. 0 homeless 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.679+0000 7fe6c8ff9700 10 client.54329.objecter ms_handle_connect 0x7fe6b4063da0 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.683+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe6c0049770 con 0x7fe6cc1007f0 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 10 client.54335.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 12 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 12 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c00e4c0 osd=5 3 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 5 12 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 4 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 10 client.54335.objecter _send_command 12 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.855+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c016d70 con 0x7f711c016800 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c016d70 con 0x7f711c016800 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f7117fff700 10 client.54335.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 4 2024-02-10T08:48:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 5 12 2024-02-10T08:48:50.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:48:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 10 client.54335.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 13 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 13 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 6 13 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 4 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 10 client.54335.objecter _send_command 13 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.857+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c026f60 con 0x7f711c023450 2024-02-10T08:48:50.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.858+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c023450 2024-02-10T08:48:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.859+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c026f60 con 0x7f711c023450 2024-02-10T08:48:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.859+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:48:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.859+0000 7f7117fff700 10 client.54335.objecter _finish_command 13 = system:0 2024-02-10T08:48:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.859+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 4 2024-02-10T08:48:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.859+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 6 13 2024-02-10T08:48:50.864 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:50.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.875+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fe6b80051d0 con 0x7fe6cc1007f0 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 10 client.54335.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 14 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 14 2024-02-10T08:48:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c016d70 osd=6 3 2024-02-10T08:48:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 6 14 2024-02-10T08:48:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 4 2024-02-10T08:48:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 10 client.54335.objecter _send_command 14 2024-02-10T08:48:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.882+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c026f60 con 0x7f711c023450 2024-02-10T08:48:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c026f60 con 0x7f711c023450 2024-02-10T08:48:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f7117fff700 10 client.54335.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 4 2024-02-10T08:48:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 6 14 2024-02-10T08:48:50.885 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:48:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 10 client.54335.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:48:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 15 2024-02-10T08:48:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 15 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 7 15 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 4 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 10 client.54335.objecter _send_command 15 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.884+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711c029380 con 0x7f711c027650 2024-02-10T08:48:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.885+0000 7f7117fff700 10 client.54335.objecter ms_handle_connect 0x7f711c027650 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.886+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f711c029380 con 0x7f711c027650 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.886+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.886+0000 7f7117fff700 10 client.54335.objecter _finish_command 15 = system:0 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.886+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 4 2024-02-10T08:48:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.886+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 7 15 2024-02-10T08:48:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:50.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 10 client.54335.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign -1 16 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 20 client.54335.objecter _get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 15 client.54335.objecter _session_command_op_remove -1 16 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 20 client.54335.objecter get_session s=0x7f711c026f60 osd=7 3 2024-02-10T08:48:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 15 client.54335.objecter _session_command_op_assign 7 16 2024-02-10T08:48:50.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 4 2024-02-10T08:48:50.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 10 client.54335.objecter _send_command 16 2024-02-10T08:48:50.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.901+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f711c029380 con 0x7f711c027650 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.902+0000 7f7117fff700 1 -- 172.21.15.107:0/1818583645 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f711c029380 con 0x7f711c027650 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.902+0000 7f7117fff700 10 client.54335.objecter ms_dispatch 0x7f7130001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.902+0000 7f7117fff700 10 client.54335.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.902+0000 7f7117fff700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 4 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.902+0000 7f7117fff700 15 client.54335.objecter _session_command_op_remove 7 16 2024-02-10T08:48:50.903 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:48:50.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 librados: watch_flush enter 2024-02-10T08:48:50.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 librados: watch_flush exit 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.0 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6801/3321596972 conn(0x7f711c004530 legacy=0x7f711c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0051a0 osd=0 1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6805/133516126 conn(0x7f711c004150 legacy=0x7f711c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003fe0 osd=1 1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.2 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6809/3575881728 conn(0x7f711c00d2f0 legacy=0x7f711c00d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c003630 osd=2 1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.3 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6813/2871825691 conn(0x7f711c011e80 legacy=0x7f711c014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c0033c0 osd=3 1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.4 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.195:6800/3310789251 conn(0x7f711c00ce50 legacy=0x7f711c015be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e180 osd=4 1 2024-02-10T08:48:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f712f7fe700 1 -- 172.21.15.107:0/1818583645 reap_dead start 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.5 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.195:6804/4178321486 conn(0x7f711c016800 legacy=0x7f711c021820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f712f7fe700 1 -- 172.21.15.107:0/1818583645 reap_dead start 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c00e4c0 osd=5 1 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.6 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.195:6808/1417479017 conn(0x7f711c023450 legacy=0x7f711c025840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c016d70 osd=6 1 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f712f7fe700 1 -- 172.21.15.107:0/1818583645 reap_dead start 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter close_session for osd.7 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.195:6812/1934176928 conn(0x7f711c027650 legacy=0x7f711c027ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter put_session s=0x7f711c026f60 osd=7 1 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 20 client.54335.objecter shutdown clearing up homeless session... 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 10 client.54335.objecter successfully canceled tick 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7118063c10 legacy=0x7f71180660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> v1:172.21.15.107:6790/0 conn(0x7f71300fda50 legacy=0x7f71300f99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 shutdown_connections 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 >> 172.21.15.107:0/1818583645 conn(0x7f71300f3640 msgr2=0x7f71301077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 shutdown_connections 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 -- 172.21.15.107:0/1818583645 wait complete. 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:50.903+0000 7f71353cf700 1 librados: shutdown 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:48:50.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:51.017 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.165 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:51.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:51.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.195+0000 7fe6c8ff9700 1 -- 172.21.15.107:0/2276380318 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v764) v1 ==== 113+0+0 (unknown 2949797639 0 0) 0x7fe6c0047b80 con 0x7fe6cc1007f0 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.197+0000 7fe6d1f14700 10 librados: watch_flush enter 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 10 librados: watch_flush exit 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 20 client.54329.objecter shutdown clearing up homeless session... 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 10 client.54329.objecter successfully canceled tick 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe6b4063da0 legacy=0x7fe6b4066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 >> v1:172.21.15.107:6790/0 conn(0x7fe6cc1007f0 legacy=0x7fe6cc0f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 shutdown_connections 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 >> 172.21.15.107:0/2276380318 conn(0x7fe6cc0f3640 msgr2=0x7fe6cc1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:51.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 shutdown_connections 2024-02-10T08:48:51.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 -- 172.21.15.107:0/2276380318 wait complete. 2024-02-10T08:48:51.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:51.198+0000 7fe6d1f14700 1 librados: shutdown 2024-02-10T08:48:51.200 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T08:48:51.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:51.299 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.318 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:51.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:51.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:48:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.470 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:48:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:51.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:48:51.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:51.598 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:51.599 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:51.599 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:49.702915+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:49.706254+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.117219+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.117973+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.747531+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.107:0/1818583645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.760106+0000 osd.7 (osd.7) 308 : cluster [DBG] 6.1c deep-scrub starts 2024-02-10T08:48:51.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:50.762665+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.1c deep-scrub ok 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.876225+0000 mon.c (mon.2) 483 : audit [INF] from='client.? v1:172.21.15.107:0/2276380318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:50.876644+0000 mon.a (mon.0) 3560 : audit [INF] from='client.54329 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:51.190272+0000 mon.a (mon.0) 3561 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:51.190363+0000 mon.a (mon.0) 3562 : audit [INF] from='client.54329 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.190419+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-10T08:48:51.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:51.194052+0000 mon.b (mon.1) 837 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.602 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:51.201037+0000 mon.a (mon.0) 3564 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.602 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:51.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:51.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:49.702915+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:49.706254+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.117219+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.117973+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.747531+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.107:0/1818583645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.760106+0000 osd.7 (osd.7) 308 : cluster [DBG] 6.1c deep-scrub starts 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:50.762665+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.1c deep-scrub ok 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.876225+0000 mon.c (mon.2) 483 : audit [INF] from='client.? v1:172.21.15.107:0/2276380318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:50.876644+0000 mon.a (mon.0) 3560 : audit [INF] from='client.54329 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:51.190272+0000 mon.a (mon.0) 3561 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:51.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:51.190363+0000 mon.a (mon.0) 3562 : audit [INF] from='client.54329 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.190419+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:51.194052+0000 mon.b (mon.1) 837 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:51.201037+0000 mon.a (mon.0) 3564 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:49.702915+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:49.706254+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.117219+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.117973+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.747531+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.107:0/1818583645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.760106+0000 osd.7 (osd.7) 308 : cluster [DBG] 6.1c deep-scrub starts 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:50.762665+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.1c deep-scrub ok 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.876225+0000 mon.c (mon.2) 483 : audit [INF] from='client.? v1:172.21.15.107:0/2276380318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:50.876644+0000 mon.a (mon.0) 3560 : audit [INF] from='client.54329 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:48:51.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:51.190272+0000 mon.a (mon.0) 3561 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:48:51.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:51.190363+0000 mon.a (mon.0) 3562 : audit [INF] from='client.54329 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:48:51.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.190419+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-10T08:48:51.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:51.194052+0000 mon.b (mon.1) 837 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:51.201037+0000 mon.a (mon.0) 3564 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:48:51.754 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:51.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:51.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:51.912 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:52.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:48:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:48:52.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T08:48:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:52.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.034548+0000 osd.5 (osd.5) 316 : cluster [DBG] 5.f deep-scrub starts 2024-02-10T08:48:52.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.038019+0000 osd.5 (osd.5) 317 : cluster [DBG] 5.f deep-scrub ok 2024-02-10T08:48:52.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.564414+0000 mgr.y (mgr.14553) 631 : cluster [DBG] pgmap v1159: 194 pgs: 2 unknown, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:52.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:51.808897+0000 mgr.y (mgr.14553) 632 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:52.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:52.190986+0000 mon.a (mon.0) 3565 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:48:52.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:52.191059+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-10T08:48:52.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:52.191564+0000 mon.b (mon.1) 838 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:52.191840+0000 mon.a (mon.0) 3567 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.034548+0000 osd.5 (osd.5) 316 : cluster [DBG] 5.f deep-scrub starts 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.038019+0000 osd.5 (osd.5) 317 : cluster [DBG] 5.f deep-scrub ok 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.564414+0000 mgr.y (mgr.14553) 631 : cluster [DBG] pgmap v1159: 194 pgs: 2 unknown, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:51.808897+0000 mgr.y (mgr.14553) 632 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:52.190986+0000 mon.a (mon.0) 3565 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:48:52.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:52.191059+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-10T08:48:52.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:52.191564+0000 mon.b (mon.1) 838 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:52.191840+0000 mon.a (mon.0) 3567 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.034548+0000 osd.5 (osd.5) 316 : cluster [DBG] 5.f deep-scrub starts 2024-02-10T08:48:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.038019+0000 osd.5 (osd.5) 317 : cluster [DBG] 5.f deep-scrub ok 2024-02-10T08:48:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.564414+0000 mgr.y (mgr.14553) 631 : cluster [DBG] pgmap v1159: 194 pgs: 2 unknown, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:51.808897+0000 mgr.y (mgr.14553) 632 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:48:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:52.190986+0000 mon.a (mon.0) 3565 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:48:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:52.191059+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-10T08:48:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:52.191564+0000 mon.b (mon.1) 838 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:52.191840+0000 mon.a (mon.0) 3567 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]: dispatch 2024-02-10T08:48:52.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:48:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:53.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.751285+0000 osd.1 (osd.1) 334 : cluster [DBG] 4.f scrub starts 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:51.754916+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.f scrub ok 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:52.093497+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:52.097157+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:52.468111+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f scrub starts 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:52.471827+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f scrub ok 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:53.057683+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:53.059125+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:53.187818+0000 mon.a (mon.0) 3568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:53.191787+0000 mon.a (mon.0) 3569 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]': finished 2024-02-10T08:48:53.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:53.191857+0000 mon.a (mon.0) 3570 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.751285+0000 osd.1 (osd.1) 334 : cluster [DBG] 4.f scrub starts 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:51.754916+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.f scrub ok 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:52.093497+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:52.097157+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:52.468111+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f scrub starts 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:52.471827+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f scrub ok 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:53.057683+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:53.059125+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:53.187818+0000 mon.a (mon.0) 3568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:53.191787+0000 mon.a (mon.0) 3569 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]': finished 2024-02-10T08:48:53.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:53.191857+0000 mon.a (mon.0) 3570 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.751285+0000 osd.1 (osd.1) 334 : cluster [DBG] 4.f scrub starts 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:51.754916+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.f scrub ok 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:52.093497+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.1c deep-scrub starts 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:52.097157+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.1c deep-scrub ok 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:52.468111+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f scrub starts 2024-02-10T08:48:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:52.471827+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f scrub ok 2024-02-10T08:48:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:53.057683+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:48:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:53.059125+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:48:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:53.187818+0000 mon.a (mon.0) 3568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:48:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:53.191787+0000 mon.a (mon.0) 3569 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-134", "mode": "readproxy"}]': finished 2024-02-10T08:48:53.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:53.191857+0000 mon.a (mon.0) 3570 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-10T08:48:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:48:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:54.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:48:54.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:48:54.680 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:54.701 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:48:54.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:53.564817+0000 mgr.y (mgr.14553) 633 : cluster [DBG] pgmap v1162: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:54.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.104462+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:48:54.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.105487+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:48:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:54.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:48:54.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:54.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:53.564817+0000 mgr.y (mgr.14553) 633 : cluster [DBG] pgmap v1162: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:54.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.104462+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:48:54.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.105487+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:48:54.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:53.564817+0000 mgr.y (mgr.14553) 633 : cluster [DBG] pgmap v1162: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T08:48:54.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.104462+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:48:54.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.105487+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:48:54.833 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:54.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:54.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:54.990 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:55.010 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:55.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:55.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- 172.21.15.107:0/2422818877 >> v1:172.21.15.107:6790/0 conn(0x7f76341007f0 legacy=0x7f7634102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- 172.21.15.107:0/2422818877 shutdown_connections 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- 172.21.15.107:0/2422818877 >> 172.21.15.107:0/2422818877 conn(0x7f76340f3640 msgr2=0x7f76340f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- 172.21.15.107:0/2422818877 shutdown_connections 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- 172.21.15.107:0/2422818877 wait complete. 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 librados: starting msgr at 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 librados: starting objecter 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 Processor -- start 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 -- start start 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 librados: setting wanted keys 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.107+0000 7f763b75e700 1 librados: calling monclient init 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f763b75e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76341173c0 con 0x7f76341007f0 2024-02-10T08:48:55.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f763b75e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76341185a0 con 0x7f76340fda50 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f763b75e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76341197a0 con 0x7f7634104330 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f7639cfb700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7634104330 0x7f7634115a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42518/0 (socket says 172.21.15.107:42518) 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f7639cfb700 1 -- 172.21.15.107:0/4271140124 learned_addr learned my addr 172.21.15.107:0/4271140124 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter ms_handle_connect 0x7f7634104330 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter ms_handle_connect 0x7f76341007f0 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter ms_handle_connect 0x7f76340fda50 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046440844 0 0) 0x7f76341197a0 con 0x7f7634104330 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7620003030 con 0x7f7634104330 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975043608 0 0) 0x7f76341173c0 con 0x7f76341007f0 2024-02-10T08:48:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76341197a0 con 0x7f76341007f0 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158537703 0 0) 0x7f76341185a0 con 0x7f76340fda50 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76341173c0 con 0x7f76340fda50 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3124571404 0 0) 0x7f76341197a0 con 0x7f76341007f0 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76341185a0 con 0x7f76341007f0 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345049687 0 0) 0x7f7620003030 con 0x7f7634104330 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76341197a0 con 0x7f7634104330 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f76240020a0 con 0x7f76341007f0 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f762c001c00 con 0x7f7634104330 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2662203813 0 0) 0x7f76341173c0 con 0x7f76340fda50 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7620003030 con 0x7f76340fda50 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7630001620 con 0x7f76340fda50 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3101485521 0 0) 0x7f76341197a0 con 0x7f7634104330 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 >> v1:172.21.15.195:6789/0 conn(0x7f76340fda50 legacy=0x7f76340f99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:55.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 >> v1:172.21.15.107:6789/0 conn(0x7f76341007f0 legacy=0x7f76340fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763411a9a0 con 0x7f7634104330 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.108+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76341175b0 con 0x7f7634104330 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f763b75e700 10 client.54341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7634117a20 con 0x7f7634104330 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f763b75e700 1 librados: init done 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f762c002240 con 0x7f7634104330 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f762c005d50 con 0x7f7634104330 2024-02-10T08:48:55.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.109+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7618004f20 con 0x7f7634104330 2024-02-10T08:48:55.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f762c001d60 con 0x7f7634104330 2024-02-10T08:48:55.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(766..766 src has 420..766) v4 ==== 8592+0+0 (unknown 700076855 0 0) 0x7f762c079180 con 0x7f7634104330 2024-02-10T08:48:55.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 10 client.54341.objecter ms_dispatch 0x7f7634001550 osd_map(766..766 src has 420..766) v4 2024-02-10T08:48:55.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 3 client.54341.objecter handle_osd_map got epochs [766,766] > 0 2024-02-10T08:48:55.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 3 client.54341.objecter handle_osd_map decoding full epoch 766 2024-02-10T08:48:55.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.110+0000 7f762a7fc700 20 client.54341.objecter dump_active .. 0 homeless 2024-02-10T08:48:55.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.111+0000 7f762a7fc700 10 client.54341.objecter ms_handle_connect 0x7f7620063c70 2024-02-10T08:48:55.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.115+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f762c04c380 con 0x7f7634104330 2024-02-10T08:48:55.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:48:55.132 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:55.148 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:55.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:48:55.169 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:48:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:55.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:48:55.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:48:55.304 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:55.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:55.329+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f76180051d0 con 0x7f7634104330 2024-02-10T08:48:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:48:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:55.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:48:55.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:55.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.002915+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:48:55.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.005825+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:48:55.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.135036+0000 osd.2 (osd.2) 285 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.135897+0000 osd.2 (osd.2) 286 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.415692+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.417163+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.703163+0000 osd.7 (osd.7) 310 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:54.704174+0000 osd.7 (osd.7) 311 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.060002+0000 osd.6 (osd.6) 318 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.061715+0000 osd.6 (osd.6) 319 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:55.171953+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:55.330499+0000 mon.c (mon.2) 484 : audit [INF] from='client.? v1:172.21.15.107:0/4271140124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:55.330936+0000 mon.a (mon.0) 3572 : audit [INF] from='client.54341 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.002915+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.005825+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.135036+0000 osd.2 (osd.2) 285 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.135897+0000 osd.2 (osd.2) 286 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.415692+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.417163+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.703163+0000 osd.7 (osd.7) 310 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:54.704174+0000 osd.7 (osd.7) 311 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.060002+0000 osd.6 (osd.6) 318 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.061715+0000 osd.6 (osd.6) 319 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:55.171953+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:55.330499+0000 mon.c (mon.2) 484 : audit [INF] from='client.? v1:172.21.15.107:0/4271140124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:55.330936+0000 mon.a (mon.0) 3572 : audit [INF] from='client.54341 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.480 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.002915+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.005825+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.135036+0000 osd.2 (osd.2) 285 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.135897+0000 osd.2 (osd.2) 286 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.415692+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:48:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.417163+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.703163+0000 osd.7 (osd.7) 310 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:54.704174+0000 osd.7 (osd.7) 311 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.060002+0000 osd.6 (osd.6) 318 : cluster [DBG] 234.6 scrub starts 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.061715+0000 osd.6 (osd.6) 319 : cluster [DBG] 234.6 scrub ok 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:55.171953+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:55.330499+0000 mon.c (mon.2) 484 : audit [INF] from='client.? v1:172.21.15.107:0/4271140124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:55.330936+0000 mon.a (mon.0) 3572 : audit [INF] from='client.54341 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:48:55.844 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:48:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:48:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:48:55] "GET /metrics HTTP/1.1" 200 191074 "" "Prometheus/2.33.4" 2024-02-10T08:48:56.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.176+0000 7f762a7fc700 1 -- 172.21.15.107:0/4271140124 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v767) v1 ==== 87+0+0 (unknown 1432993931 0 0) 0x7f762c04a790 con 0x7f7634104330 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 10 librados: watch_flush enter 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 10 librados: watch_flush exit 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 20 client.54341.objecter shutdown clearing up homeless session... 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 10 client.54341.objecter successfully canceled tick 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7620063c70 legacy=0x7f7620066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 >> v1:172.21.15.107:6790/0 conn(0x7f7634104330 legacy=0x7f7634115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 shutdown_connections 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 >> 172.21.15.107:0/4271140124 conn(0x7f76340f3640 msgr2=0x7f76341077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 shutdown_connections 2024-02-10T08:48:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 -- 172.21.15.107:0/4271140124 wait complete. 2024-02-10T08:48:56.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:56.178+0000 7f763b75e700 1 librados: shutdown 2024-02-10T08:48:56.180 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.012758+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.016733+0000 osd.5 (osd.5) 321 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.565173+0000 mgr.y (mgr.14553) 634 : cluster [DBG] pgmap v1163: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 950 B/s rd, 190 B/s wr, 1 op/s 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.710782+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:55.711728+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:48:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.169924+0000 mon.a (mon.0) 3573 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:56.175551+0000 mon.a (mon.0) 3574 : audit [INF] from='client.54341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.175619+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.175911+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.012758+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.016733+0000 osd.5 (osd.5) 321 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.565173+0000 mgr.y (mgr.14553) 634 : cluster [DBG] pgmap v1163: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 950 B/s rd, 190 B/s wr, 1 op/s 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.710782+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:55.711728+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.169924+0000 mon.a (mon.0) 3573 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:56.175551+0000 mon.a (mon.0) 3574 : audit [INF] from='client.54341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.175619+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-10T08:48:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.175911+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.012758+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.016733+0000 osd.5 (osd.5) 321 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.565173+0000 mgr.y (mgr.14553) 634 : cluster [DBG] pgmap v1163: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 950 B/s rd, 190 B/s wr, 1 op/s 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.710782+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:55.711728+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.169924+0000 mon.a (mon.0) 3573 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:48:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:56.175551+0000 mon.a (mon.0) 3574 : audit [INF] from='client.54341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:48:56.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.175619+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-10T08:48:56.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.175911+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:48:57.029 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:57.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:57.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:57.163 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:57.318 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:48:57.318 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:48:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.058311+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-02-10T08:48:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.061778+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-02-10T08:48:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.116635+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.a scrub starts 2024-02-10T08:48:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:56.119792+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.a scrub ok 2024-02-10T08:48:57.814 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.058311+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-02-10T08:48:57.814 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.061778+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.116635+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.a scrub starts 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:56.119792+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.a scrub ok 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.058311+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.061778+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.116635+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.a scrub starts 2024-02-10T08:48:57.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:56.119792+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.a scrub ok 2024-02-10T08:48:58.421 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:58.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:58.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:58.557 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:58.579 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:58.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:58.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:58.709 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:48:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:57.565945+0000 mgr.y (mgr.14553) 635 : cluster [DBG] pgmap v1165: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:48:58.711 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:58.731 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:58.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:58.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:48:58.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:48:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:57.565945+0000 mgr.y (mgr.14553) 635 : cluster [DBG] pgmap v1165: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:48:58.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:48:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:57.565945+0000 mgr.y (mgr.14553) 635 : cluster [DBG] pgmap v1165: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:48:58.868 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:58.922 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:48:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:58.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:48:59.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:48:59.054 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:59.812 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:48:59.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:48:59.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T08:48:59.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:48:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:48:59.966 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 -- 172.21.15.107:0/1153557408 >> v1:172.21.15.107:6790/0 conn(0x7fa1a0103260 legacy=0x7fa1a01056e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 -- 172.21.15.107:0/1153557408 shutdown_connections 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 -- 172.21.15.107:0/1153557408 >> 172.21.15.107:0/1153557408 conn(0x7fa1a00f2540 msgr2=0x7fa1a00f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 -- 172.21.15.107:0/1153557408 shutdown_connections 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 -- 172.21.15.107:0/1153557408 wait complete. 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 librados: starting msgr at 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 librados: starting objecter 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.514+0000 7fa1a595a700 1 Processor -- start 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 -- start start 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 librados: setting wanted keys 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 librados: calling monclient init 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa1a010e8a0 con 0x7fa1a0103260 2024-02-10T08:48:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa1a010ea70 con 0x7fa1a00ff720 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa1a595a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa1a011fc00 con 0x7fa1a00fc920 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa19effd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa1a00fc920 0x7fa1a00f84b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49226/0 (socket says 172.21.15.107:49226) 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa19effd700 1 -- 172.21.15.107:0/3286784858 learned_addr learned my addr 172.21.15.107:0/3286784858 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter ms_handle_connect 0x7fa1a00fc920 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter ms_handle_connect 0x7fa1a0103260 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750554600 0 0) 0x7fa1a011fc00 con 0x7fa1a00fc920 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188003030 con 0x7fa1a00fc920 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007555638 0 0) 0x7fa1a010e8a0 con 0x7fa1a0103260 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a011fc00 con 0x7fa1a0103260 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter ms_handle_connect 0x7fa1a00ff720 2024-02-10T08:48:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426209984 0 0) 0x7fa188003030 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a010e8a0 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335105677 0 0) 0x7fa1a010ea70 con 0x7fa1a00ff720 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188003030 con 0x7fa1a00ff720 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238049778 0 0) 0x7fa1a011fc00 con 0x7fa1a0103260 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a010ea70 con 0x7fa1a0103260 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa194001990 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.515+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa1980023d0 con 0x7fa1a0103260 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863366569 0 0) 0x7fa1a010e8a0 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 >> v1:172.21.15.195:6789/0 conn(0x7fa1a00ff720 legacy=0x7fa1a00f8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 >> v1:172.21.15.107:6789/0 conn(0x7fa1a0103260 legacy=0x7fa1a011c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0120de0 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a011fdf0 con 0x7fa1a00fc920 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa1a595a700 10 client.54347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:48:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a01202e0 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa1a595a700 1 librados: init done 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa194001cd0 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.516+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa194005bc0 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c004f20 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa194002930 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(767..767 src has 420..767) v4 ==== 8592+0+0 (unknown 1411037452 0 0) 0x7fa194078f60 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa187fff700 10 client.54347.objecter ms_dispatch 0x7fa1a0001550 osd_map(767..767 src has 420..767) v4 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa187fff700 3 client.54347.objecter handle_osd_map got epochs [767,767] > 0 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.517+0000 7fa187fff700 3 client.54347.objecter handle_osd_map decoding full epoch 767 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.518+0000 7fa187fff700 20 client.54347.objecter dump_active .. 0 homeless 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.518+0000 7fa187fff700 10 client.54347.objecter ms_handle_connect 0x7fa188063ba0 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.522+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa19404c160 con 0x7fa1a00fc920 2024-02-10T08:48:59.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:48:59.701+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fa18c0051d0 con 0x7fa1a00fc920 2024-02-10T08:48:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:48:59.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:48:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:00.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:00.103 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:00.123 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:00.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:00.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T08:49:00.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:00.290 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:00.316 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:00.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:00.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:00.449 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:00.546 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:00.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:00.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.628+0000 7fa187fff700 1 -- 172.21.15.107:0/3286784858 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v768) v1 ==== 113+0+0 (unknown 3939106757 0 0) 0x7fa19404a570 con 0x7fa1a00fc920 2024-02-10T08:49:00.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 10 librados: watch_flush enter 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 10 librados: watch_flush exit 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 20 client.54347.objecter shutdown clearing up homeless session... 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 10 client.54347.objecter successfully canceled tick 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa188063ba0 legacy=0x7fa188066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 >> v1:172.21.15.107:6790/0 conn(0x7fa1a00fc920 legacy=0x7fa1a00f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 shutdown_connections 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 >> 172.21.15.107:0/3286784858 conn(0x7fa1a00f2540 msgr2=0x7fa1a01066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 shutdown_connections 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 -- 172.21.15.107:0/3286784858 wait complete. 2024-02-10T08:49:00.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:00.630+0000 7fa1a595a700 1 librados: shutdown 2024-02-10T08:49:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T08:49:00.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:00.680 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:00.700 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:00.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:00.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:00.833 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:00.852 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:00.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:48:59.566327+0000 mgr.y (mgr.14553) 636 : cluster [DBG] pgmap v1166: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-02-10T08:49:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:59.702375+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.107:0/3286784858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:48:59.702622+0000 mon.a (mon.0) 3577 : audit [INF] from='client.54347 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:48:59.566327+0000 mgr.y (mgr.14553) 636 : cluster [DBG] pgmap v1166: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-02-10T08:49:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:59.702375+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.107:0/3286784858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:48:59.702622+0000 mon.a (mon.0) 3577 : audit [INF] from='client.54347 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:00.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:00.983 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:48:59.566327+0000 mgr.y (mgr.14553) 636 : cluster [DBG] pgmap v1166: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-02-10T08:49:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:59.702375+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.107:0/3286784858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:48:59.702622+0000 mon.a (mon.0) 3577 : audit [INF] from='client.54347 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:49:01.056 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:01.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:01.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:01.190 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:01.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:00.628283+0000 mon.a (mon.0) 3578 : audit [INF] from='client.54347 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:49:01.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:00.628343+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-10T08:49:01.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:01.183520+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-10T08:49:01.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:00.628283+0000 mon.a (mon.0) 3578 : audit [INF] from='client.54347 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:49:01.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:00.628343+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-10T08:49:01.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:01.183520+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-10T08:49:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:01.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:01.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:01.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:00.628283+0000 mon.a (mon.0) 3578 : audit [INF] from='client.54347 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:49:01.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:00.628343+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-10T08:49:01.973 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:01.183520+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-10T08:49:01.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:01.974 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:01.996 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:02.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:02.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:02.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T08:49:02.677 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:02.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:02.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:02.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:02.884 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.907+0000 7f024f674700 1 -- 172.21.15.107:0/3721998382 >> v1:172.21.15.107:6789/0 conn(0x7f02480fc910 legacy=0x7f02480fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.907+0000 7f024f674700 1 -- 172.21.15.107:0/3721998382 shutdown_connections 2024-02-10T08:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.907+0000 7f024f674700 1 -- 172.21.15.107:0/3721998382 >> 172.21.15.107:0/3721998382 conn(0x7f02480f2510 msgr2=0x7f02480f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.907+0000 7f024f674700 1 -- 172.21.15.107:0/3721998382 shutdown_connections 2024-02-10T08:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.907+0000 7f024f674700 1 -- 172.21.15.107:0/3721998382 wait complete. 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 librados: starting msgr at 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 librados: starting objecter 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 Processor -- start 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 -- start start 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 librados: setting wanted keys 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 librados: calling monclient init 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0248117400 con 0x7f02480fc910 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0248118600 con 0x7f02480ff710 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024f674700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0248119800 con 0x7f0248103250 2024-02-10T08:49:02.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024dc11700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0248103250 0x7f0248115ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49256/0 (socket says 172.21.15.107:49256) 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f024dc11700 1 -- 172.21.15.107:0/2805521019 learned_addr learned my addr 172.21.15.107:0/2805521019 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 10 client.?.objecter ms_handle_connect 0x7f02480fc910 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 10 client.?.objecter ms_handle_connect 0x7f0248103250 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445489886 0 0) 0x7f0248117400 con 0x7f02480fc910 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02340033f0 con 0x7f02480fc910 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293395257 0 0) 0x7f0248119800 con 0x7f0248103250 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0248117400 con 0x7f0248103250 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.908+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574634042 0 0) 0x7f02340033f0 con 0x7f02480fc910 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0248119800 con 0x7f02480fc910 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 10 client.?.objecter ms_handle_connect 0x7f02480ff710 2024-02-10T08:49:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0238002050 con 0x7f02480fc910 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048689831 0 0) 0x7f0248118600 con 0x7f02480ff710 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02340033f0 con 0x7f02480ff710 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898566917 0 0) 0x7f0248117400 con 0x7f0248103250 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0248118600 con 0x7f0248103250 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0240001dd0 con 0x7f0248103250 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2627476301 0 0) 0x7f0248119800 con 0x7f02480fc910 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6790/0 conn(0x7f0248103250 legacy=0x7f0248115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.195:6789/0 conn(0x7f02480ff710 legacy=0x7f02480613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024811aa00 con 0x7f02480fc910 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02481199f0 con 0x7f02480fc910 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f024f674700 10 client.28755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0248119e60 con 0x7f02480fc910 2024-02-10T08:49:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f024f674700 1 librados: init done 2024-02-10T08:49:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0238002320 con 0x7f02480fc910 2024-02-10T08:49:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.909+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0238003650 con 0x7f02480fc910 2024-02-10T08:49:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.910+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f022c004f00 con 0x7f02480fc910 2024-02-10T08:49:02.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0238001800 con 0x7f02480fc910 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(769..769 src has 420..769) v4 ==== 8592+0+0 (unknown 4264971097 0 0) 0x7f0238076aa0 con 0x7f02480fc910 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 osd_map(769..769 src has 420..769) v4 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 3 client.28755.objecter handle_osd_map got epochs [769,769] > 0 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 3 client.28755.objecter handle_osd_map decoding full epoch 769 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 20 client.28755.objecter dump_active .. 0 homeless 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v769) v1 ==== 54+0+16 (unknown 770346296 0 2420000192) 0x7f0238077060 con 0x7f02480fc910 2024-02-10T08:49:02.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f0234063ba0 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 10 client.28755.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 1 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 1 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 0 1 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 4 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 10 client.28755.objecter _send_command 1 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.911+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c004950 con 0x7f022c004530 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c004530 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c004950 con 0x7f022c004530 2024-02-10T08:49:02.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:49:02.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 10 client.28755.objecter _finish_command 1 = system:0 2024-02-10T08:49:02.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 4 2024-02-10T08:49:02.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.912+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 0 1 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 10 client.28755.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 2 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 2 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c0051a0 osd=0 3 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 0 2 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 4 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 10 client.28755.objecter _send_command 2 2024-02-10T08:49:02.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.927+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c004d30 con 0x7f022c004530 2024-02-10T08:49:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:01.566872+0000 mgr.y (mgr.14553) 637 : cluster [DBG] pgmap v1169: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:49:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:01.819632+0000 mgr.y (mgr.14553) 638 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:02.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c004d30 con 0x7f022c004530 2024-02-10T08:49:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f023e7fc700 10 client.28755.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 4 2024-02-10T08:49:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 0 2 2024-02-10T08:49:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:01.566872+0000 mgr.y (mgr.14553) 637 : cluster [DBG] pgmap v1169: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:49:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:01.819632+0000 mgr.y (mgr.14553) 638 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:02.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:49:02.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 10 client.28755.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:02.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 3 2024-02-10T08:49:02.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 1 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 4 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 10 client.28755.objecter _send_command 3 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.930+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c00bf60 con 0x7f022c003b40 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.931+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c003b40 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.932+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c00bf60 con 0x7f022c003b40 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.932+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.932+0000 7f023e7fc700 10 client.28755.objecter _finish_command 3 = system:0 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.932+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 4 2024-02-10T08:49:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.932+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 1 3 2024-02-10T08:49:02.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 10 client.28755.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 4 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 4 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c004270 osd=1 3 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 1 4 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 4 2024-02-10T08:49:02.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 10 client.28755.objecter _send_command 4 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.953+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c00d280 con 0x7f022c003b40 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.955+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c00d280 con 0x7f022c003b40 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.955+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.955+0000 7f023e7fc700 10 client.28755.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.955+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 4 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.955+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 1 4 2024-02-10T08:49:02.957 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 10 client.28755.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 5 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 5 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 2 5 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 4 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 10 client.28755.objecter _send_command 5 2024-02-10T08:49:02.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.957+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c003460 con 0x7f022c002070 2024-02-10T08:49:02.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.958+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c002070 2024-02-10T08:49:02.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.959+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c003460 con 0x7f022c002070 2024-02-10T08:49:02.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.959+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:49:02.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.959+0000 7f023e7fc700 10 client.28755.objecter _finish_command 5 = system:0 2024-02-10T08:49:02.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.959+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 4 2024-02-10T08:49:02.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.959+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 2 5 2024-02-10T08:49:02.970 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 10 client.28755.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 6 2024-02-10T08:49:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 6 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c003160 osd=2 3 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 2 6 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 4 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 10 client.28755.objecter _send_command 6 2024-02-10T08:49:02.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.976+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c001630 con 0x7f022c002070 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.977+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c001630 con 0x7f022c002070 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.977+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.977+0000 7f023e7fc700 10 client.28755.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.977+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 4 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.977+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 2 6 2024-02-10T08:49:02.978 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 10 client.28755.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 7 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 7 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 3 7 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 4 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 10 client.28755.objecter _send_command 7 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c00d080 con 0x7f022c016460 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.978+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c016460 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.979+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c00d080 con 0x7f022c016460 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.979+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.979+0000 7f023e7fc700 10 client.28755.objecter _finish_command 7 = system:0 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.979+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 4 2024-02-10T08:49:02.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.979+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 3 7 2024-02-10T08:49:02.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 10 client.28755.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:02.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 8 2024-02-10T08:49:02.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 8 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c00c8d0 osd=3 3 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 3 8 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 4 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 10 client.28755.objecter _send_command 8 2024-02-10T08:49:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.994+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c0177b0 con 0x7f022c016460 2024-02-10T08:49:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c0177b0 con 0x7f022c016460 2024-02-10T08:49:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 10 client.28755.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 4 2024-02-10T08:49:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 3 8 2024-02-10T08:49:02.997 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:49:02.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 10 client.28755.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 9 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 9 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 4 9 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 4 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 10 client.28755.objecter _send_command 9 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c01e630 con 0x7f022c01aab0 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.996+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c01aab0 2024-02-10T08:49:02.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.997+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c01e630 con 0x7f022c01aab0 2024-02-10T08:49:02.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.997+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:49:02.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.997+0000 7f023e7fc700 10 client.28755.objecter _finish_command 9 = system:0 2024-02-10T08:49:02.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.997+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 4 2024-02-10T08:49:02.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:02.997+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 4 9 2024-02-10T08:49:03.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 10 client.28755.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:03.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 10 2024-02-10T08:49:03.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:03.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:03.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 10 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c016fe0 osd=4 3 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 4 10 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 4 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 10 client.28755.objecter _send_command 10 2024-02-10T08:49:03.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.012+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c01e630 con 0x7f022c01aab0 2024-02-10T08:49:03.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c01e630 con 0x7f022c01aab0 2024-02-10T08:49:03.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:03.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f023e7fc700 10 client.28755.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 4 2024-02-10T08:49:03.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 4 10 2024-02-10T08:49:03.015 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 10 client.28755.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 11 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 11 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 5 11 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 4 2024-02-10T08:49:03.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 10 client.28755.objecter _send_command 11 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.014+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c01f890 con 0x7f022c020330 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c020330 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c01f890 con 0x7f022c020330 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 10 client.28755.objecter _finish_command 11 = system:0 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 4 2024-02-10T08:49:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.015+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 5 11 2024-02-10T08:49:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 10 client.28755.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 12 2024-02-10T08:49:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 12 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01e630 osd=5 3 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 5 12 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 4 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 10 client.28755.objecter _send_command 12 2024-02-10T08:49:03.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.030+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c01f850 con 0x7f022c020330 2024-02-10T08:49:03.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c01f850 con 0x7f022c020330 2024-02-10T08:49:03.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:03.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f023e7fc700 10 client.28755.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 4 2024-02-10T08:49:03.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 5 12 2024-02-10T08:49:03.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 10 client.28755.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 13 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 13 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 6 13 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 4 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 10 client.28755.objecter _send_command 13 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.032+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c01fc40 con 0x7f022c0234f0 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c0234f0 2024-02-10T08:49:03.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c01fc40 con 0x7f022c0234f0 2024-02-10T08:49:03.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:49:03.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 10 client.28755.objecter _finish_command 13 = system:0 2024-02-10T08:49:03.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 4 2024-02-10T08:49:03.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.033+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 6 13 2024-02-10T08:49:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:01.566872+0000 mgr.y (mgr.14553) 637 : cluster [DBG] pgmap v1169: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-10T08:49:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:01.819632+0000 mgr.y (mgr.14553) 638 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 10 client.28755.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 14 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 14 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01f850 osd=6 3 2024-02-10T08:49:03.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 6 14 2024-02-10T08:49:03.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 4 2024-02-10T08:49:03.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 10 client.28755.objecter _send_command 14 2024-02-10T08:49:03.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.048+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c01fc40 con 0x7f022c0234f0 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.049+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c01fc40 con 0x7f022c0234f0 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.049+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.049+0000 7f023e7fc700 10 client.28755.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.049+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 4 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.049+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 6 14 2024-02-10T08:49:03.050 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 10 client.28755.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 15 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 15 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 7 15 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 4 2024-02-10T08:49:03.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 10 client.28755.objecter _send_command 15 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c0280f0 con 0x7f022c028b90 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.050+0000 7f023e7fc700 10 client.28755.objecter ms_handle_connect 0x7f022c028b90 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.051+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f022c0280f0 con 0x7f022c028b90 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.051+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.051+0000 7f023e7fc700 10 client.28755.objecter _finish_command 15 = system:0 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.051+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 4 2024-02-10T08:49:03.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.051+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 7 15 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 10 client.28755.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign -1 16 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 20 client.28755.objecter _get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 15 client.28755.objecter _session_command_op_remove -1 16 2024-02-10T08:49:03.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 20 client.28755.objecter get_session s=0x7f022c01fc40 osd=7 3 2024-02-10T08:49:03.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 15 client.28755.objecter _session_command_op_assign 7 16 2024-02-10T08:49:03.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 4 2024-02-10T08:49:03.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 10 client.28755.objecter _send_command 16 2024-02-10T08:49:03.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.065+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f022c0280b0 con 0x7f022c028b90 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f023e7fc700 1 -- 172.21.15.107:0/2805521019 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f022c0280b0 con 0x7f022c028b90 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f023e7fc700 10 client.28755.objecter ms_dispatch 0x7f0248001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f023e7fc700 10 client.28755.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f023e7fc700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 4 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f023e7fc700 15 client.28755.objecter _session_command_op_remove 7 16 2024-02-10T08:49:03.067 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 librados: watch_flush enter 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 librados: watch_flush exit 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.0 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6801/3321596972 conn(0x7f022c004530 legacy=0x7f022c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c0051a0 osd=0 1 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.1 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6805/133516126 conn(0x7f022c003b40 legacy=0x7f022c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c004270 osd=1 1 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.2 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6809/3575881728 conn(0x7f022c002070 legacy=0x7f022c013b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c003160 osd=2 1 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.3 2024-02-10T08:49:03.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6813/2871825691 conn(0x7f022c016460 legacy=0x7f022c016880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c00c8d0 osd=3 1 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.4 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.195:6800/3310789251 conn(0x7f022c01aab0 legacy=0x7f022c01cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024dc11700 1 -- 172.21.15.107:0/2805521019 reap_dead start 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c016fe0 osd=4 1 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.5 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.195:6804/4178321486 conn(0x7f022c020330 legacy=0x7f022c0207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024dc11700 1 -- 172.21.15.107:0/2805521019 reap_dead start 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01e630 osd=5 1 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.6 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.195:6808/1417479017 conn(0x7f022c0234f0 legacy=0x7f022c0258e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01f850 osd=6 1 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter close_session for osd.7 2024-02-10T08:49:03.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.195:6812/1934176928 conn(0x7f022c028b90 legacy=0x7f022c029010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024dc11700 1 -- 172.21.15.107:0/2805521019 reap_dead start 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter put_session s=0x7f022c01fc40 osd=7 1 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 20 client.28755.objecter shutdown clearing up homeless session... 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 10 client.28755.objecter successfully canceled tick 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0234063ba0 legacy=0x7f0234066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> v1:172.21.15.107:6789/0 conn(0x7f02480fc910 legacy=0x7f0248060ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 shutdown_connections 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 >> 172.21.15.107:0/2805521019 conn(0x7f02480f2510 msgr2=0x7f02481066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 shutdown_connections 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 -- 172.21.15.107:0/2805521019 wait complete. 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:03.067+0000 7f024f674700 1 librados: shutdown 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.072 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:49:03.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:02.911271+0000 mon.a (mon.0) 3581 : audit [DBG] from='client.? v1:172.21.15.107:0/2805521019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:03.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:03.361403+0000 mon.b (mon.1) 839 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:03.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:03.361836+0000 mon.a (mon.0) 3582 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:02.911271+0000 mon.a (mon.0) 3581 : audit [DBG] from='client.? v1:172.21.15.107:0/2805521019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:03.361403+0000 mon.b (mon.1) 839 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:03.361836+0000 mon.a (mon.0) 3582 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:02.911271+0000 mon.a (mon.0) 3581 : audit [DBG] from='client.? v1:172.21.15.107:0/2805521019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:03.361403+0000 mon.b (mon.1) 839 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:04.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:03.361836+0000 mon.a (mon.0) 3582 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:04.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:03.567409+0000 mgr.y (mgr.14553) 639 : cluster [DBG] pgmap v1170: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:49:04.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:03.631591+0000 mon.a (mon.0) 3583 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:04.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:03.631635+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-10T08:49:04.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:03.633007+0000 mon.b (mon.1) 840 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:04.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:03.633898+0000 mon.a (mon.0) 3585 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:04.629515+0000 mon.a (mon.0) 3586 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:04.633327+0000 mon.a (mon.0) 3587 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:04.633408+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:03.567409+0000 mgr.y (mgr.14553) 639 : cluster [DBG] pgmap v1170: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:03.631591+0000 mon.a (mon.0) 3583 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:03.631635+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:03.633007+0000 mon.b (mon.1) 840 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:03.633898+0000 mon.a (mon.0) 3585 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:04.629515+0000 mon.a (mon.0) 3586 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:04.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:04.633327+0000 mon.a (mon.0) 3587 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:49:04.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:04.633408+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-10T08:49:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:03.567409+0000 mgr.y (mgr.14553) 639 : cluster [DBG] pgmap v1170: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:03.631591+0000 mon.a (mon.0) 3583 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:03.631635+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:03.633007+0000 mon.b (mon.1) 840 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:03.633898+0000 mon.a (mon.0) 3585 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:04.629515+0000 mon.a (mon.0) 3586 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:04.633327+0000 mon.a (mon.0) 3587 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]': finished 2024-02-10T08:49:05.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:04.633408+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-10T08:49:05.757 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:49:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:49:05.757 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:49:05.757 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:49:05.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:49:05.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:05] "GET /metrics HTTP/1.1" 200 191074 "" "Prometheus/2.33.4" 2024-02-10T08:49:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:04.660867+0000 mon.b (mon.1) 841 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:04.661200+0000 mon.a (mon.0) 3589 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:04.661662+0000 mon.b (mon.1) 842 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:04.661864+0000 mon.a (mon.0) 3590 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:04.660867+0000 mon.b (mon.1) 841 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:04.661200+0000 mon.a (mon.0) 3589 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:04.661662+0000 mon.b (mon.1) 842 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:04.661864+0000 mon.a (mon.0) 3590 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:04.660867+0000 mon.b (mon.1) 841 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:04.661200+0000 mon.a (mon.0) 3589 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:04.661662+0000 mon.b (mon.1) 842 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:04.661864+0000 mon.a (mon.0) 3590 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-134"}]: dispatch 2024-02-10T08:49:06.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T08:49:06.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:05.567978+0000 mgr.y (mgr.14553) 640 : cluster [DBG] pgmap v1173: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:05.652728+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:06.188021+0000 mon.a (mon.0) 3592 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:06.194960+0000 mon.b (mon.1) 843 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:06.198622+0000 mon.a (mon.0) 3593 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:05.567978+0000 mgr.y (mgr.14553) 640 : cluster [DBG] pgmap v1173: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:05.652728+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:06.188021+0000 mon.a (mon.0) 3592 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:06.194960+0000 mon.b (mon.1) 843 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:06.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:06.198622+0000 mon.a (mon.0) 3593 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:05.567978+0000 mgr.y (mgr.14553) 640 : cluster [DBG] pgmap v1173: 194 pgs: 194 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:49:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:05.652728+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-10T08:49:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:06.188021+0000 mon.a (mon.0) 3592 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-10T08:49:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:06.194960+0000 mon.b (mon.1) 843 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:06.198622+0000 mon.a (mon.0) 3593 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:07.552 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:07.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:07.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:07.691 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:07.712 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:07.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:07.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:07.847 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:07.867 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:07.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:07.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:07.995 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:07.999 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:08.012 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:08.018 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:08.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:08.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:06.983055+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:06.983310+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:07.187899+0000 mon.a (mon.0) 3594 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:08.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:07.187963+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-10T08:49:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:07.211846+0000 mon.b (mon.1) 844 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:07.212231+0000 mon.a (mon.0) 3596 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:07.568730+0000 mgr.y (mgr.14553) 641 : cluster [DBG] pgmap v1177: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 260 B/s wr, 1 op/s 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:06.983055+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:06.983310+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:07.187899+0000 mon.a (mon.0) 3594 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:07.187963+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:07.211846+0000 mon.b (mon.1) 844 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:07.212231+0000 mon.a (mon.0) 3596 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:07.568730+0000 mgr.y (mgr.14553) 641 : cluster [DBG] pgmap v1177: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 260 B/s wr, 1 op/s 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:06.983055+0000 osd.7 (osd.7) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:06.983310+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:07.187899+0000 mon.a (mon.0) 3594 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:07.187963+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:07.211846+0000 mon.b (mon.1) 844 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:07.212231+0000 mon.a (mon.0) 3596 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:08.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:07.568730+0000 mgr.y (mgr.14553) 641 : cluster [DBG] pgmap v1177: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 260 B/s wr, 1 op/s 2024-02-10T08:49:08.298 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:08.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:08.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:08.431 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:09.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:08.195777+0000 mon.a (mon.0) 3597 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:08.195830+0000 mon.a (mon.0) 3598 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:08.196270+0000 mon.b (mon.1) 845 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:08.196572+0000 mon.a (mon.0) 3599 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:08.912215+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.107:0/265779238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:08.912511+0000 mon.a (mon.0) 3600 : audit [INF] from='client.54359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:08.195777+0000 mon.a (mon.0) 3597 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:09.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:08.195830+0000 mon.a (mon.0) 3598 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-10T08:49:09.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:08.196270+0000 mon.b (mon.1) 845 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:08.196572+0000 mon.a (mon.0) 3599 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:08.912215+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.107:0/265779238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:08.912511+0000 mon.a (mon.0) 3600 : audit [INF] from='client.54359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:08.195777+0000 mon.a (mon.0) 3597 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:08.195830+0000 mon.a (mon.0) 3598 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:08.196270+0000 mon.b (mon.1) 845 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:08.196572+0000 mon.a (mon.0) 3599 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:08.912215+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.107:0/265779238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:08.912511+0000 mon.a (mon.0) 3600 : audit [INF] from='client.54359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.718+0000 7fa870a16700 1 -- 172.21.15.107:0/1691105896 >> v1:172.21.15.107:6790/0 conn(0x7fa86c0ff730 legacy=0x7fa86c101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.718+0000 7fa870a16700 1 -- 172.21.15.107:0/1691105896 shutdown_connections 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.718+0000 7fa870a16700 1 -- 172.21.15.107:0/1691105896 >> 172.21.15.107:0/1691105896 conn(0x7fa86c0f2510 msgr2=0x7fa86c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.718+0000 7fa870a16700 1 -- 172.21.15.107:0/1691105896 shutdown_connections 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.718+0000 7fa870a16700 1 -- 172.21.15.107:0/1691105896 wait complete. 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 librados: starting msgr at 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 librados: starting objecter 2024-02-10T08:49:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 Processor -- start 2024-02-10T08:49:09.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 -- start start 2024-02-10T08:49:09.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 librados: setting wanted keys 2024-02-10T08:49:09.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 librados: calling monclient init 2024-02-10T08:49:09.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa86c119570 con 0x7fa86c103270 2024-02-10T08:49:09.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa86c11a750 con 0x7fa86c0ff730 2024-02-10T08:49:09.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.719+0000 7fa870a16700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa86c11b950 con 0x7fa86c0fc930 2024-02-10T08:49:09.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa86a59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa86c0fc930 0x7fa86c060bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49288/0 (socket says 172.21.15.107:49288) 2024-02-10T08:49:09.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa86a59c700 1 -- 172.21.15.107:0/265779238 learned_addr learned my addr 172.21.15.107:0/265779238 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:09.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter ms_handle_connect 0x7fa86c103270 2024-02-10T08:49:09.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter ms_handle_connect 0x7fa86c0fc930 2024-02-10T08:49:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter ms_handle_connect 0x7fa86c0ff730 2024-02-10T08:49:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:09.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136132505 0 0) 0x7fa86c11b950 con 0x7fa86c0fc930 2024-02-10T08:49:09.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa854003110 con 0x7fa86c0fc930 2024-02-10T08:49:09.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780565552 0 0) 0x7fa86c119570 con 0x7fa86c103270 2024-02-10T08:49:09.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa86c11b950 con 0x7fa86c103270 2024-02-10T08:49:09.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058992647 0 0) 0x7fa86c11a750 con 0x7fa86c0ff730 2024-02-10T08:49:09.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa86c119570 con 0x7fa86c0ff730 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272491623 0 0) 0x7fa854003110 con 0x7fa86c0fc930 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa86c11a750 con 0x7fa86c0fc930 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa860001860 con 0x7fa86c0fc930 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247015488 0 0) 0x7fa86c11b950 con 0x7fa86c103270 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa854003110 con 0x7fa86c103270 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa864001b80 con 0x7fa86c103270 2024-02-10T08:49:09.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916659675 0 0) 0x7fa86c119570 con 0x7fa86c0ff730 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa86c11b950 con 0x7fa86c0ff730 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa85c0015d0 con 0x7fa86c0ff730 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1558669975 0 0) 0x7fa86c11a750 con 0x7fa86c0fc930 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 >> v1:172.21.15.195:6789/0 conn(0x7fa86c0ff730 legacy=0x7fa86c0612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 >> v1:172.21.15.107:6789/0 conn(0x7fa86c103270 legacy=0x7fa86c117cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c11cb50 con 0x7fa86c0fc930 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c11bb40 con 0x7fa86c0fc930 2024-02-10T08:49:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa870a16700 10 client.54359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c11c030 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa870a16700 1 librados: init done 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa860001fa0 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.720+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa8600059d0 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.721+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c004f20 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa860001b00 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(775..775 src has 420..775) v4 ==== 8592+0+0 (unknown 4128784690 0 0) 0x7fa860078df0 con 0x7fa86c0fc930 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 10 client.54359.objecter ms_dispatch 0x7fa86c001550 osd_map(775..775 src has 420..775) v4 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 3 client.54359.objecter handle_osd_map got epochs [775,775] > 0 2024-02-10T08:49:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 3 client.54359.objecter handle_osd_map decoding full epoch 775 2024-02-10T08:49:09.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 20 client.54359.objecter dump_active .. 0 homeless 2024-02-10T08:49:09.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.722+0000 7fa85b7fe700 10 client.54359.objecter ms_handle_connect 0x7fa854063e40 2024-02-10T08:49:09.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.726+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa86004bfd0 con 0x7fa86c0fc930 2024-02-10T08:49:09.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:08.911+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa84c0051d0 con 0x7fa86c0fc930 2024-02-10T08:49:09.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.197+0000 7fa85b7fe700 1 -- 172.21.15.107:0/265779238 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v776) v1 ==== 97+0+0 (unknown 3621367911 0 0) 0x7fa86004a3e0 con 0x7fa86c0fc930 2024-02-10T08:49:09.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 10 librados: watch_flush enter 2024-02-10T08:49:09.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 10 librados: watch_flush exit 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 20 client.54359.objecter shutdown clearing up homeless session... 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 10 client.54359.objecter successfully canceled tick 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa854063e40 legacy=0x7fa854066300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 >> v1:172.21.15.107:6790/0 conn(0x7fa86c0fc930 legacy=0x7fa86c060bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 shutdown_connections 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.200+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 >> 172.21.15.107:0/265779238 conn(0x7fa86c0f2510 msgr2=0x7fa86c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.203+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 shutdown_connections 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.203+0000 7fa870a16700 1 -- 172.21.15.107:0/265779238 wait complete. 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:09.203+0000 7fa870a16700 1 librados: shutdown 2024-02-10T08:49:09.648 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-02-10T08:49:10.282 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:09.191635+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:09.195939+0000 mon.a (mon.0) 3602 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:09.195999+0000 mon.a (mon.0) 3603 : audit [INF] from='client.54359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:09.196035+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:09.198941+0000 mon.b (mon.1) 846 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:09.204392+0000 mon.a (mon.0) 3605 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:09.569099+0000 mgr.y (mgr.14553) 642 : cluster [DBG] pgmap v1180: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.171808+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:09.191635+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:09.195939+0000 mon.a (mon.0) 3602 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:09.195999+0000 mon.a (mon.0) 3603 : audit [INF] from='client.54359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:09.196035+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:09.198941+0000 mon.b (mon.1) 846 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:09.204392+0000 mon.a (mon.0) 3605 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:09.569099+0000 mgr.y (mgr.14553) 642 : cluster [DBG] pgmap v1180: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-10T08:49:10.285 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.171808+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:10.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:10.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:10.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- 172.21.15.107:0/229252948 >> v1:172.21.15.107:6790/0 conn(0x7f355c103250 legacy=0x7f355c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:10.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- 172.21.15.107:0/229252948 shutdown_connections 2024-02-10T08:49:10.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- 172.21.15.107:0/229252948 >> 172.21.15.107:0/229252948 conn(0x7f355c0f2510 msgr2=0x7f355c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- 172.21.15.107:0/229252948 shutdown_connections 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- 172.21.15.107:0/229252948 wait complete. 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 librados: starting msgr at 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 librados: starting objecter 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 Processor -- start 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 -- start start 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 librados: setting wanted keys 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.379+0000 7f3564645700 1 librados: calling monclient init 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f355c117410 con 0x7f355c0ff710 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f355c1185f0 con 0x7f355c0fc910 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f355c1197f0 con 0x7f355c103250 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3561be0700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f355c0ff710 0x7f355c0f8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52342/0 (socket says 172.21.15.107:52342) 2024-02-10T08:49:10.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3562be2700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f355c103250 0x7f355c115ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50032/0 (socket says 172.21.15.107:50032) 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3561be0700 1 -- 172.21.15.107:0/678206178 learned_addr learned my addr 172.21.15.107:0/678206178 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter ms_handle_connect 0x7f355c0ff710 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter ms_handle_connect 0x7f355c103250 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter ms_handle_connect 0x7f355c0fc910 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151783127 0 0) 0x7f355c1197f0 con 0x7f355c103250 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3548003800 con 0x7f355c103250 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434620147 0 0) 0x7f355c117410 con 0x7f355c0ff710 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f355c1197f0 con 0x7f355c0ff710 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292065833 0 0) 0x7f355c1185f0 con 0x7f355c0fc910 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f355c117410 con 0x7f355c0fc910 2024-02-10T08:49:10.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235291364 0 0) 0x7f3548003800 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f355c1185f0 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f35540031e0 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 649954955 0 0) 0x7f355c1197f0 con 0x7f355c0ff710 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3548003800 con 0x7f355c0ff710 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f354c0018c0 con 0x7f355c0ff710 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219273080 0 0) 0x7f355c117410 con 0x7f355c0fc910 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f355c1197f0 con 0x7f355c0fc910 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3558001550 con 0x7f355c0fc910 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 248922195 0 0) 0x7f355c1185f0 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 >> v1:172.21.15.195:6789/0 conn(0x7f355c0fc910 legacy=0x7f355c0f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 >> v1:172.21.15.107:6789/0 conn(0x7f355c0ff710 legacy=0x7f355c0f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c11a9f0 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c1199e0 con 0x7f355c103250 2024-02-10T08:49:10.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 10 client.54365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c119e50 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3554002a30 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f3564645700 1 librados: init done 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.380+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3554003cc0 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.381+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3540004f20 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3554003ea0 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(777..777 src has 420..777) v4 ==== 8592+0+0 (unknown 486900114 0 0) 0x7f35540796a0 con 0x7f355c103250 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 10 client.54365.objecter ms_dispatch 0x7f355c001550 osd_map(777..777 src has 420..777) v4 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 3 client.54365.objecter handle_osd_map got epochs [777,777] > 0 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 3 client.54365.objecter handle_osd_map decoding full epoch 777 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 20 client.54365.objecter dump_active .. 0 homeless 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.382+0000 7f35537fe700 10 client.54365.objecter ms_handle_connect 0x7f3548063d10 2024-02-10T08:49:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.386+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f355404cab0 con 0x7f355c103250 2024-02-10T08:49:10.417 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:09.191635+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:09.195939+0000 mon.a (mon.0) 3602 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi107-112490-101", "overlaypool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:09.195999+0000 mon.a (mon.0) 3603 : audit [INF] from='client.54359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:09.196035+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:09.198941+0000 mon.b (mon.1) 846 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:09.204392+0000 mon.a (mon.0) 3605 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]: dispatch 2024-02-10T08:49:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:09.569099+0000 mgr.y (mgr.14553) 642 : cluster [DBG] pgmap v1180: 194 pgs: 6 creating+peering, 22 unknown, 166 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-10T08:49:10.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.171808+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:10.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:10.574+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f35400051d0 con 0x7f355c103250 2024-02-10T08:49:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:11.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:11.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:11.211 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:11.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.214+0000 7f35537fe700 1 -- 172.21.15.107:0/678206178 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v778) v1 ==== 113+0+0 (unknown 1858945999 0 0) 0x7f355404af20 con 0x7f355c103250 2024-02-10T08:49:11.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 10 librados: watch_flush enter 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 10 librados: watch_flush exit 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 20 client.54365.objecter shutdown clearing up homeless session... 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 10 client.54365.objecter successfully canceled tick 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3548063d10 legacy=0x7f35480661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 >> v1:172.21.15.107:6790/0 conn(0x7f355c103250 legacy=0x7f355c115ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 shutdown_connections 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 >> 172.21.15.107:0/678206178 conn(0x7f355c0f2510 msgr2=0x7f355c060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 shutdown_connections 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 -- 172.21.15.107:0/678206178 wait complete. 2024-02-10T08:49:11.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:11.216+0000 7f3564645700 1 librados: shutdown 2024-02-10T08:49:11.218 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T08:49:11.233 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:11.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:11.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:10.201738+0000 mon.a (mon.0) 3607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.213277+0000 mon.a (mon.0) 3608 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]': finished 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:10.213343+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.247989+0000 mon.b (mon.1) 847 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.248335+0000 mon.a (mon.0) 3610 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.575803+0000 mon.c (mon.2) 487 : audit [INF] from='client.? v1:172.21.15.107:0/678206178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:10.576066+0000 mon.a (mon.0) 3611 : audit [INF] from='client.54365 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:10.201738+0000 mon.a (mon.0) 3607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.213277+0000 mon.a (mon.0) 3608 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]': finished 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:10.213343+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.247989+0000 mon.b (mon.1) 847 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.248335+0000 mon.a (mon.0) 3610 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.575803+0000 mon.c (mon.2) 487 : audit [INF] from='client.? v1:172.21.15.107:0/678206178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:10.576066+0000 mon.a (mon.0) 3611 : audit [INF] from='client.54365 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:10.201738+0000 mon.a (mon.0) 3607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.213277+0000 mon.a (mon.0) 3608 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi107-112490-136", "mode": "writeback"}]': finished 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:10.213343+0000 mon.a (mon.0) 3609 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.247989+0000 mon.b (mon.1) 847 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.248335+0000 mon.a (mon.0) 3610 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.575803+0000 mon.c (mon.2) 487 : audit [INF] from='client.? v1:172.21.15.107:0/678206178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:10.576066+0000 mon.a (mon.0) 3611 : audit [INF] from='client.54365 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:49:11.962 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T08:49:11.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:49:12.114 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:12.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:12.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:12.266 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:12.392 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:12.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:12.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:12.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:11.212216+0000 mon.a (mon.0) 3612 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:49:12.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:11.212300+0000 mon.a (mon.0) 3613 : audit [INF] from='client.54365 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:49:12.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:11.212352+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-10T08:49:12.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:11.214867+0000 mon.b (mon.1) 848 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:11.267622+0000 mon.a (mon.0) 3615 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:11.569727+0000 mgr.y (mgr.14553) 643 : cluster [DBG] pgmap v1183: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:11.827475+0000 mgr.y (mgr.14553) 644 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:11.212216+0000 mon.a (mon.0) 3612 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:11.212300+0000 mon.a (mon.0) 3613 : audit [INF] from='client.54365 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:11.212352+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-10T08:49:12.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:11.214867+0000 mon.b (mon.1) 848 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:11.267622+0000 mon.a (mon.0) 3615 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:11.569727+0000 mgr.y (mgr.14553) 643 : cluster [DBG] pgmap v1183: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T08:49:12.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:11.827475+0000 mgr.y (mgr.14553) 644 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:12.524 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:11.212216+0000 mon.a (mon.0) 3612 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_count","val": "2"}]': finished 2024-02-10T08:49:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:11.212300+0000 mon.a (mon.0) 3613 : audit [INF] from='client.54365 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:49:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:11.212352+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-10T08:49:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:11.214867+0000 mon.b (mon.1) 848 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:11.267622+0000 mon.a (mon.0) 3615 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T08:49:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:11.569727+0000 mgr.y (mgr.14553) 643 : cluster [DBG] pgmap v1183: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T08:49:12.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:11.827475+0000 mgr.y (mgr.14553) 644 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:12.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:49:12.972 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:12.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:13.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:13.534 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:12.269596+0000 mon.a (mon.0) 3616 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:49:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:12.269659+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-10T08:49:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:12.271106+0000 mon.b (mon.1) 849 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:12.271385+0000 mon.a (mon.0) 3618 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:12.269596+0000 mon.a (mon.0) 3616 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:49:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:12.269659+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-10T08:49:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:12.271106+0000 mon.b (mon.1) 849 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:13.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:12.271385+0000 mon.a (mon.0) 3618 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:12.269596+0000 mon.a (mon.0) 3616 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_period","val": "600"}]': finished 2024-02-10T08:49:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:12.269659+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-10T08:49:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:12.271106+0000 mon.b (mon.1) 849 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:13.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:12.271385+0000 mon.a (mon.0) 3618 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T08:49:14.143 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:49:14.144 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:49:14.144 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-02-10T08:49:14.144 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-02-10T08:49:14.144 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:49:14.157 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:14.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:14.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:14.366 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:14.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:14.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:12.933566+0000 osd.6 (osd.6) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:12.933824+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:13.265879+0000 mon.a (mon.0) 3619 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:13.269577+0000 mon.a (mon.0) 3620 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:13.269648+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:13.271769+0000 mon.b (mon.1) 850 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:13.277901+0000 mon.a (mon.0) 3622 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:13.570223+0000 mgr.y (mgr.14553) 645 : cluster [DBG] pgmap v1186: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:13.571809+0000 osd.3 (osd.3) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:13.571994+0000 osd.3 (osd.3) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:12.933566+0000 osd.6 (osd.6) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:12.933824+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:13.265879+0000 mon.a (mon.0) 3619 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:14.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:13.269577+0000 mon.a (mon.0) 3620 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:13.269648+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:13.271769+0000 mon.b (mon.1) 850 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:13.277901+0000 mon.a (mon.0) 3622 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:13.570223+0000 mgr.y (mgr.14553) 645 : cluster [DBG] pgmap v1186: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:13.571809+0000 osd.3 (osd.3) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:13.571994+0000 osd.3 (osd.3) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:14.502 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:14.523 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:14.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:12.933566+0000 osd.6 (osd.6) 320 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:12.933824+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:13.265879+0000 mon.a (mon.0) 3619 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:13.269577+0000 mon.a (mon.0) 3620 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:13.269648+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:13.271769+0000 mon.b (mon.1) 850 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:13.277901+0000 mon.a (mon.0) 3622 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:13.570223+0000 mgr.y (mgr.14553) 645 : cluster [DBG] pgmap v1186: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-10T08:49:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:13.571809+0000 osd.3 (osd.3) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:14.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:13.571994+0000 osd.3 (osd.3) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:14.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:14.655 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:14.903 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:14.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:14.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T08:49:15.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:15.045 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:14.280819+0000 mon.a (mon.0) 3623 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:14.280888+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:14.282935+0000 mon.b (mon.1) 851 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:14.283395+0000 mon.a (mon.0) 3625 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:14.466027+0000 osd.2 (osd.2) 289 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:15.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:14.466231+0000 osd.2 (osd.2) 290 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:14.280819+0000 mon.a (mon.0) 3623 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:49:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:14.280888+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:14.282935+0000 mon.b (mon.1) 851 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:14.283395+0000 mon.a (mon.0) 3625 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:14.466027+0000 osd.2 (osd.2) 289 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:14.466231+0000 osd.2 (osd.2) 290 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:14.280819+0000 mon.a (mon.0) 3623 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:14.280888+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:14.282935+0000 mon.b (mon.1) 851 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:14.283395+0000 mon.a (mon.0) 3625 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:14.466027+0000 osd.2 (osd.2) 289 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:15.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:14.466231+0000 osd.2 (osd.2) 290 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:15.918 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:15] "GET /metrics HTTP/1.1" 200 191180 "" "Prometheus/2.33.4" 2024-02-10T08:49:16.240 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:16.486 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:16.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:16.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:16.623 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:15.281713+0000 mon.a (mon.0) 3626 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:49:16.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:15.281793+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-10T08:49:16.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:15.398697+0000 osd.1 (osd.1) 336 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:16.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:15.399063+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:16.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:15.570646+0000 mgr.y (mgr.14553) 646 : cluster [DBG] pgmap v1189: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:16.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:16.189685+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-10T08:49:16.626 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:16.647 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:15.281713+0000 mon.a (mon.0) 3626 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:15.281793+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:15.398697+0000 osd.1 (osd.1) 336 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:15.399063+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:15.570646+0000 mgr.y (mgr.14553) 646 : cluster [DBG] pgmap v1189: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:16.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:16.189685+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:15.281713+0000 mon.a (mon.0) 3626 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-136","var": "target_max_objects","val": "1"}]': finished 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:15.281793+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:15.398697+0000 osd.1 (osd.1) 336 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:15.399063+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:15.570646+0000 mgr.y (mgr.14553) 646 : cluster [DBG] pgmap v1189: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:16.189685+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-10T08:49:16.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:16.780 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:16.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:16.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.722+0000 7efbfd1e0700 1 -- 172.21.15.107:0/3259327306 >> v1:172.21.15.107:6790/0 conn(0x7efbf8104330 legacy=0x7efbf81067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.722+0000 7efbfd1e0700 1 -- 172.21.15.107:0/3259327306 shutdown_connections 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.722+0000 7efbfd1e0700 1 -- 172.21.15.107:0/3259327306 >> 172.21.15.107:0/3259327306 conn(0x7efbf80f3640 msgr2=0x7efbf80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.722+0000 7efbfd1e0700 1 -- 172.21.15.107:0/3259327306 shutdown_connections 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.722+0000 7efbfd1e0700 1 -- 172.21.15.107:0/3259327306 wait complete. 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 librados: starting msgr at 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 librados: starting objecter 2024-02-10T08:49:16.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 Processor -- start 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 -- start start 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 librados: setting wanted keys 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 librados: calling monclient init 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efbf806cc00 con 0x7efbf8104330 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efbf806cd90 con 0x7efbf80fda50 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.723+0000 7efbfd1e0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efbf806cf80 con 0x7efbf81007f0 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbf659c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7efbf81007f0 0x7efbf8114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50060/0 (socket says 172.21.15.107:50060) 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbf759e700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efbf8104330 0x7efbf81184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52360/0 (socket says 172.21.15.107:52360) 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbf659c700 1 -- 172.21.15.107:0/4128287537 learned_addr learned my addr 172.21.15.107:0/4128287537 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter ms_handle_connect 0x7efbf8104330 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter ms_handle_connect 0x7efbf81007f0 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter ms_handle_connect 0x7efbf80fda50 2024-02-10T08:49:16.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661413777 0 0) 0x7efbf806cc00 con 0x7efbf8104330 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe0003030 con 0x7efbf8104330 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662828577 0 0) 0x7efbf806cf80 con 0x7efbf81007f0 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf806cc00 con 0x7efbf81007f0 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395822528 0 0) 0x7efbf806cd90 con 0x7efbf80fda50 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf806cf80 con 0x7efbf80fda50 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304334277 0 0) 0x7efbe0003030 con 0x7efbf8104330 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf806cd90 con 0x7efbf8104330 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efbf0002320 con 0x7efbf8104330 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2679957838 0 0) 0x7efbf806cf80 con 0x7efbf80fda50 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbe0003030 con 0x7efbf80fda50 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83977910 0 0) 0x7efbf806cc00 con 0x7efbf81007f0 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf806cf80 con 0x7efbf81007f0 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efbe80015a0 con 0x7efbf80fda50 2024-02-10T08:49:16.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efbec001840 con 0x7efbf81007f0 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2048086733 0 0) 0x7efbf806cd90 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6790/0 conn(0x7efbf81007f0 legacy=0x7efbf8114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.195:6789/0 conn(0x7efbf80fda50 legacy=0x7efbf80fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf811cc00 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf8119c30 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbfd1e0700 10 client.28773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf811a0a0 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbfd1e0700 1 librados: init done 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7efbf0002960 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.724+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7efbf0003ac0 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.725+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efbe4004f00 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7efbf00024c0 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(783..783 src has 420..783) v4 ==== 8618+0+0 (unknown 4275375863 0 0) 0x7efbf0079520 con 0x7efbf8104330 2024-02-10T08:49:16.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 osd_map(783..783 src has 420..783) v4 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 3 client.28773.objecter handle_osd_map got epochs [783,783] > 0 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 3 client.28773.objecter handle_osd_map decoding full epoch 783 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 20 client.28773.objecter dump_active .. 0 homeless 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.726+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v783) v1 ==== 54+0+16 (unknown 1156222854 0 2420000192) 0x7efbf007b930 con 0x7efbf8104330 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe0063d10 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:49:16.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 1 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 1 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 0 1 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 4 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 10 client.28773.objecter _send_command 1 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe4004950 con 0x7efbe4004530 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe4004530 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.727+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe4004950 con 0x7efbe4004530 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.728+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.728+0000 7efbdffff700 10 client.28773.objecter _finish_command 1 = system:0 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.728+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 4 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.728+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 0 1 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 2 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 2 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe40051a0 osd=0 3 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 0 2 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 4 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 10 client.28773.objecter _send_command 2 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.743+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe4004d30 con 0x7efbe4004530 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe4004d30 con 0x7efbe4004530 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 10 client.28773.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 4 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 0 2 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 3 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 4 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 10 client.28773.objecter _send_command 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe400c000 con 0x7efbe4004150 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.745+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe4004150 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.746+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe400c000 con 0x7efbe4004150 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.746+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.746+0000 7efbdffff700 10 client.28773.objecter _finish_command 3 = system:0 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.746+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 4 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.746+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 1 3 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003fe0 osd=1 3 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 10 client.28773.objecter _send_command 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.761+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe40018b0 con 0x7efbe4004150 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe40018b0 con 0x7efbe4004150 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 10 client.28773.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 1 4 2024-02-10T08:49:16.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 5 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 5 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 2 5 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 4 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 10 client.28773.objecter _send_command 5 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe400de90 con 0x7efbe400d2f0 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.762+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe400d2f0 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.763+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe400de90 con 0x7efbe400d2f0 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.763+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.763+0000 7efbdffff700 10 client.28773.objecter _finish_command 5 = system:0 2024-02-10T08:49:16.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.763+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 4 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.763+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 2 5 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 6 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 6 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003630 osd=2 3 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 2 6 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 4 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 10 client.28773.objecter _send_command 6 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.777+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe40030a0 con 0x7efbe400d2f0 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe40030a0 con 0x7efbe400d2f0 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 10 client.28773.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 4 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 2 6 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 7 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 7 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 3 7 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 4 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 10 client.28773.objecter _send_command 7 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe4015a90 con 0x7efbe4011e80 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.779+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe4011e80 2024-02-10T08:49:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.780+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe4015a90 con 0x7efbe4011e80 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.780+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.780+0000 7efbdffff700 10 client.28773.objecter _finish_command 7 = system:0 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.780+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 4 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.780+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 3 7 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 8 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 8 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4003480 osd=3 3 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 3 8 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 4 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 10 client.28773.objecter _send_command 8 2024-02-10T08:49:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.794+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe400e270 con 0x7efbe4011e80 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe400e270 con 0x7efbe4011e80 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 10 client.28773.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 4 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 3 8 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 9 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 9 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 4 9 2024-02-10T08:49:16.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 4 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 10 client.28773.objecter _send_command 9 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe400cf20 con 0x7efbe400e3e0 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.796+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe400e3e0 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.797+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe400cf20 con 0x7efbe400e3e0 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.797+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.797+0000 7efbdffff700 10 client.28773.objecter _finish_command 9 = system:0 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:49:16.940 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.797+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 4 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.797+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 4 9 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 10 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 10 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe400e270 osd=4 3 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 4 10 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 4 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 10 client.28773.objecter _send_command 10 2024-02-10T08:49:16.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.812+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe400cf20 con 0x7efbe400e3e0 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe400cf20 con 0x7efbe400e3e0 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbdffff700 10 client.28773.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 4 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 4 10 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 11 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 11 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 5 11 2024-02-10T08:49:16.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 4 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 10 client.28773.objecter _send_command 11 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.814+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe4016b20 con 0x7efbe40166b0 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.815+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe40166b0 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.816+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe4016b20 con 0x7efbe40166b0 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.816+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.816+0000 7efbdffff700 10 client.28773.objecter _finish_command 11 = system:0 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.816+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 4 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.816+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 5 11 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 12 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 12 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe400cf20 osd=5 3 2024-02-10T08:49:16.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 5 12 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 4 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 10 client.28773.objecter _send_command 12 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.830+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe4016b20 con 0x7efbe40166b0 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.832+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe4016b20 con 0x7efbe40166b0 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.832+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.832+0000 7efbdffff700 10 client.28773.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.832+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 4 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.832+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 5 12 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 13 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 13 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 6 13 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 4 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 10 client.28773.objecter _send_command 13 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe4016f00 con 0x7efbe401f090 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.833+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe401f090 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.834+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe4016f00 con 0x7efbe401f090 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.834+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.834+0000 7efbdffff700 10 client.28773.objecter _finish_command 13 = system:0 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.834+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 4 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.834+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 6 13 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 14 2024-02-10T08:49:16.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 14 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe4016b20 osd=6 3 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 6 14 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 4 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 10 client.28773.objecter _send_command 14 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.848+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe4016f00 con 0x7efbe401f090 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe4016f00 con 0x7efbe401f090 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbdffff700 10 client.28773.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 4 2024-02-10T08:49:16.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 6 14 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 15 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 15 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 7 15 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 4 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 10 client.28773.objecter _send_command 15 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.850+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe4024d20 con 0x7efbe4024860 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 10 client.28773.objecter ms_handle_connect 0x7efbe4024860 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efbe4024d20 con 0x7efbe4024860 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 10 client.28773.objecter _finish_command 15 = system:0 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 4 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.851+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 7 15 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 10 client.28773.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign -1 16 2024-02-10T08:49:16.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 20 client.28773.objecter _get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_remove -1 16 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 20 client.28773.objecter get_session s=0x7efbe40241e0 osd=7 3 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 15 client.28773.objecter _session_command_op_assign 7 16 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 4 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 10 client.28773.objecter _send_command 16 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.866+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efbe4024d20 con 0x7efbe4024860 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbdffff700 1 -- 172.21.15.107:0/4128287537 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efbe4024d20 con 0x7efbe4024860 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbdffff700 10 client.28773.objecter ms_dispatch 0x7efbf8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbdffff700 10 client.28773.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbdffff700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 4 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbdffff700 15 client.28773.objecter _session_command_op_remove 7 16 2024-02-10T08:49:16.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 librados: watch_flush enter 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 librados: watch_flush exit 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.0 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6801/3321596972 conn(0x7efbe4004530 legacy=0x7efbe4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40051a0 osd=0 1 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.1 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6805/133516126 conn(0x7efbe4004150 legacy=0x7efbe400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003fe0 osd=1 1 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.2 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6809/3575881728 conn(0x7efbe400d2f0 legacy=0x7efbe400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003630 osd=2 1 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.3 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6813/2871825691 conn(0x7efbe4011e80 legacy=0x7efbe4014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4003480 osd=3 1 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.4 2024-02-10T08:49:16.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.195:6800/3310789251 conn(0x7efbe400e3e0 legacy=0x7efbe4015a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbf759e700 1 -- 172.21.15.107:0/4128287537 reap_dead start 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400e270 osd=4 1 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.5 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.195:6804/4178321486 conn(0x7efbe40166b0 legacy=0x7efbe401d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbf759e700 1 -- 172.21.15.107:0/4128287537 reap_dead start 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe400cf20 osd=5 1 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.6 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.195:6808/1417479017 conn(0x7efbe401f090 legacy=0x7efbe4021480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe4016b20 osd=6 1 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter close_session for osd.7 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.195:6812/1934176928 conn(0x7efbe4024860 legacy=0x7efbe4022ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter put_session s=0x7efbe40241e0 osd=7 1 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 20 client.28773.objecter shutdown clearing up homeless session... 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbfd1e0700 10 client.28773.objecter successfully canceled tick 2024-02-10T08:49:16.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.868+0000 7efbf759e700 1 -- 172.21.15.107:0/4128287537 reap_dead start 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6800/2634842009 conn(0x7efbe0063d10 legacy=0x7efbe00661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> v1:172.21.15.107:6789/0 conn(0x7efbf8104330 legacy=0x7efbf81184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 shutdown_connections 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 >> 172.21.15.107:0/4128287537 conn(0x7efbf80f3640 msgr2=0x7efbf81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 shutdown_connections 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 -- 172.21.15.107:0/4128287537 wait complete. 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:16.869+0000 7efbfd1e0700 1 librados: shutdown 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:49:16.953 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:17.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:17.268 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.289 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:17.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:17.421 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:16.284833+0000 mon.a (mon.0) 3629 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:49:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:16.727087+0000 mon.a (mon.0) 3630 : audit [DBG] from='client.? v1:172.21.15.107:0/4128287537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:16.284833+0000 mon.a (mon.0) 3629 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:49:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:16.727087+0000 mon.a (mon.0) 3630 : audit [DBG] from='client.? v1:172.21.15.107:0/4128287537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:17.440 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:16.284833+0000 mon.a (mon.0) 3629 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T08:49:17.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:16.727087+0000 mon.a (mon.0) 3630 : audit [DBG] from='client.? v1:172.21.15.107:0/4128287537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:17.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:17.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:17.780 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.800 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:17.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:17.992 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:17.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:18.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:18.126 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:18.148 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:18.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:18.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:18.279 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:18.377 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:18.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:18.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:18.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:17.571234+0000 mgr.y (mgr.14553) 647 : cluster [DBG] pgmap v1191: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:18.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:17.571234+0000 mgr.y (mgr.14553) 647 : cluster [DBG] pgmap v1191: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:18.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:17.571234+0000 mgr.y (mgr.14553) 647 : cluster [DBG] pgmap v1191: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:18.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:18.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:18.677 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:18.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:18.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:18.892 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:19.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.962+0000 7f53e598e700 1 -- 172.21.15.107:0/3982902578 >> v1:172.21.15.107:6790/0 conn(0x7f53e0104330 legacy=0x7f53e01067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.962+0000 7f53e598e700 1 -- 172.21.15.107:0/3982902578 shutdown_connections 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.962+0000 7f53e598e700 1 -- 172.21.15.107:0/3982902578 >> 172.21.15.107:0/3982902578 conn(0x7f53e00f3640 msgr2=0x7f53e00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.962+0000 7f53e598e700 1 -- 172.21.15.107:0/3982902578 shutdown_connections 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.962+0000 7f53e598e700 1 -- 172.21.15.107:0/3982902578 wait complete. 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 librados: starting msgr at 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 librados: starting objecter 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 Processor -- start 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 -- start start 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 librados: setting wanted keys 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 librados: calling monclient init 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53e00fa930 con 0x7f53e0104330 2024-02-10T08:49:19.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53e01187f0 con 0x7f53e01007f0 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53e598e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53e0119990 con 0x7f53e00fda50 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53deffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f53e00fda50 0x7f53e00f9d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50078/0 (socket says 172.21.15.107:50078) 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53deffd700 1 -- 172.21.15.107:0/927457836 learned_addr learned my addr 172.21.15.107:0/927457836 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53c7fff700 10 client.?.objecter ms_handle_connect 0x7f53e0104330 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53c7fff700 10 client.?.objecter ms_handle_connect 0x7f53e00fda50 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.963+0000 7f53c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 10 client.?.objecter ms_handle_connect 0x7f53e01007f0 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153384920 0 0) 0x7f53e0119990 con 0x7f53e00fda50 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c8003630 con 0x7f53e00fda50 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701849149 0 0) 0x7f53e00fa930 con 0x7f53e0104330 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e0119990 con 0x7f53e0104330 2024-02-10T08:49:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673346463 0 0) 0x7f53e01187f0 con 0x7f53e01007f0 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e00fa930 con 0x7f53e01007f0 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174127017 0 0) 0x7f53c8003630 con 0x7f53e00fda50 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53e01187f0 con 0x7f53e00fda50 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53d4001900 con 0x7f53e00fda50 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813589512 0 0) 0x7f53e0119990 con 0x7f53e0104330 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c8003630 con 0x7f53e0104330 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53d80021d0 con 0x7f53e0104330 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089519684 0 0) 0x7f53e00fa930 con 0x7f53e01007f0 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53e0119990 con 0x7f53e01007f0 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53d00018e0 con 0x7f53e01007f0 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3183005877 0 0) 0x7f53e01187f0 con 0x7f53e00fda50 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 >> v1:172.21.15.195:6789/0 conn(0x7f53e01007f0 legacy=0x7f53e0112990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 >> v1:172.21.15.107:6789/0 conn(0x7f53e0104330 legacy=0x7f53e01160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e011ab90 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0119b80 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53e598e700 10 client.54377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e0119ff0 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53e598e700 1 librados: init done 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f53d4001c40 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.964+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f53d4005b50 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.965+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc004f20 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f53d40021b0 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(783..783 src has 420..783) v4 ==== 8618+0+0 (unknown 4275375863 0 0) 0x7f53d4078ef0 con 0x7f53e00fda50 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 10 client.54377.objecter ms_dispatch 0x7f53e0001550 osd_map(783..783 src has 420..783) v4 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 3 client.54377.objecter handle_osd_map got epochs [783,783] > 0 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 3 client.54377.objecter handle_osd_map decoding full epoch 783 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 20 client.54377.objecter dump_active .. 0 homeless 2024-02-10T08:49:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.966+0000 7f53c7fff700 10 client.54377.objecter ms_handle_connect 0x7f53c8063c80 2024-02-10T08:49:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:18.970+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f53d404c0d0 con 0x7f53e00fda50 2024-02-10T08:49:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.149+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f53cc0051d0 con 0x7f53e00fda50 2024-02-10T08:49:19.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.310+0000 7f53c7fff700 1 -- 172.21.15.107:0/927457836 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v784) v1 ==== 91+0+0 (unknown 495453864 0 0) 0x7f53d404a4e0 con 0x7f53e00fda50 2024-02-10T08:49:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 10 librados: watch_flush enter 2024-02-10T08:49:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 10 librados: watch_flush exit 2024-02-10T08:49:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 20 client.54377.objecter shutdown clearing up homeless session... 2024-02-10T08:49:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 10 client.54377.objecter successfully canceled tick 2024-02-10T08:49:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 >> v1:172.21.15.107:6800/2634842009 conn(0x7f53c8063c80 legacy=0x7f53c8066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.312+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 >> v1:172.21.15.107:6790/0 conn(0x7f53e00fda50 legacy=0x7f53e00f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.313+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 shutdown_connections 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.313+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 >> 172.21.15.107:0/927457836 conn(0x7f53e00f3640 msgr2=0x7f53e01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.313+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 shutdown_connections 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.313+0000 7f53e598e700 1 -- 172.21.15.107:0/927457836 wait complete. 2024-02-10T08:49:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.313+0000 7f53e598e700 1 librados: shutdown 2024-02-10T08:49:19.314 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:49:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:19.150145+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.107:0/927457836' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:19.150408+0000 mon.a (mon.0) 3631 : audit [INF] from='client.54377 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:19.150145+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.107:0/927457836' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:19.150408+0000 mon.a (mon.0) 3631 : audit [INF] from='client.54377 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:49:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:19.150145+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.107:0/927457836' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:19.150408+0000 mon.a (mon.0) 3631 : audit [INF] from='client.54377 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.921+0000 7ff83c3f8700 1 -- 172.21.15.107:0/1386005458 >> v1:172.21.15.195:6789/0 conn(0x7ff834100830 legacy=0x7ff834102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.922+0000 7ff83c3f8700 1 -- 172.21.15.107:0/1386005458 shutdown_connections 2024-02-10T08:49:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.922+0000 7ff83c3f8700 1 -- 172.21.15.107:0/1386005458 >> 172.21.15.107:0/1386005458 conn(0x7ff83406fe70 msgr2=0x7ff8340f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.922+0000 7ff83c3f8700 1 -- 172.21.15.107:0/1386005458 shutdown_connections 2024-02-10T08:49:19.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.922+0000 7ff83c3f8700 1 -- 172.21.15.107:0/1386005458 wait complete. 2024-02-10T08:49:19.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 librados: starting msgr at 2024-02-10T08:49:19.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 librados: starting objecter 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 Processor -- start 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 -- start start 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 librados: setting wanted keys 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.923+0000 7ff83c3f8700 1 librados: calling monclient init 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff83c3f8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff834115200 con 0x7ff8340fd9e0 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff83c3f8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8341163e0 con 0x7ff834104340 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff83c3f8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8341175e0 con 0x7ff834100830 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff83a194700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8340fd9e0 0x7ff834102580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38092/0 (socket says 172.21.15.107:38092) 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff83a194700 1 -- 172.21.15.107:0/218921361 learned_addr learned my addr 172.21.15.107:0/218921361 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter ms_handle_connect 0x7ff8340fd9e0 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter ms_handle_connect 0x7ff834100830 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter ms_handle_connect 0x7ff834104340 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866406406 0 0) 0x7ff834115200 con 0x7ff8340fd9e0 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8200033f0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692298613 0 0) 0x7ff8341175e0 con 0x7ff834100830 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff834115200 con 0x7ff834100830 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643704706 0 0) 0x7ff8341163e0 con 0x7ff834104340 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.924+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8341175e0 con 0x7ff834104340 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750984045 0 0) 0x7ff8200033f0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8341163e0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff830002950 con 0x7ff8340fd9e0 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 124710690 0 0) 0x7ff834115200 con 0x7ff834100830 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8200033f0 con 0x7ff834100830 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff824001ec0 con 0x7ff834100830 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 759256807 0 0) 0x7ff8341175e0 con 0x7ff834104340 2024-02-10T08:49:19.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff834115200 con 0x7ff834104340 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff82c001c40 con 0x7ff834104340 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3139240905 0 0) 0x7ff8341163e0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 >> v1:172.21.15.107:6790/0 conn(0x7ff834100830 legacy=0x7ff834102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 >> v1:172.21.15.195:6789/0 conn(0x7ff834104340 legacy=0x7ff8341139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8341177d0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff834117a80 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff83c3f8700 10 client.28785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff834117ef0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff83c3f8700 1 librados: init done 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7ff830001810 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.925+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7ff830005ae0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.926+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818004f20 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.926+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7ff830018970 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(784..784 src has 420..784) v4 ==== 8618+0+0 (unknown 2152502808 0 0) 0x7ff830078eb0 con 0x7ff8340fd9e0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 10 client.28785.objecter ms_dispatch 0x7ff834001550 osd_map(784..784 src has 420..784) v4 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 3 client.28785.objecter handle_osd_map got epochs [784,784] > 0 2024-02-10T08:49:19.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 3 client.28785.objecter handle_osd_map decoding full epoch 784 2024-02-10T08:49:19.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 20 client.28785.objecter dump_active .. 0 homeless 2024-02-10T08:49:19.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.927+0000 7ff82b7fe700 10 client.28785.objecter ms_handle_connect 0x7ff820063c30 2024-02-10T08:49:19.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:19.931+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff83004c070 con 0x7ff8340fd9e0 2024-02-10T08:49:20.041 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:20.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:20.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.097+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff818000b90 con 0x7ff820063c30 2024-02-10T08:49:20.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.111+0000 7ff82b7fe700 1 -- 172.21.15.107:0/218921361 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+505162 (unknown 2965378022 0 805123511) 0x7ff818000b90 con 0x7ff820063c30 2024-02-10T08:49:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 10 librados: watch_flush enter 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 10 librados: watch_flush exit 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 20 client.28785.objecter shutdown clearing up homeless session... 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 10 client.28785.objecter successfully canceled tick 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 >> v1:172.21.15.107:6800/2634842009 conn(0x7ff820063c30 legacy=0x7ff8200660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 >> v1:172.21.15.107:6789/0 conn(0x7ff8340fd9e0 legacy=0x7ff834102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 shutdown_connections 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 >> 172.21.15.107:0/218921361 conn(0x7ff83406fe70 msgr2=0x7ff8340f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 shutdown_connections 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 -- 172.21.15.107:0/218921361 wait complete. 2024-02-10T08:49:20.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:20.115+0000 7ff83c3f8700 1 librados: shutdown 2024-02-10T08:49:20.125 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:49:20.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:20.311 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.041572+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.041662+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.296547+0000 mon.a (mon.0) 3632 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:19.306602+0000 mon.a (mon.0) 3633 : audit [INF] from='client.54377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.306660+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-10T08:49:20.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.571607+0000 mgr.y (mgr.14553) 648 : cluster [DBG] pgmap v1193: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:20.630 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:20.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.041572+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:20.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.041662+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:20.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.296547+0000 mon.a (mon.0) 3632 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:20.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:19.306602+0000 mon.a (mon.0) 3633 : audit [INF] from='client.54377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:20.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.306660+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.571607+0000 mgr.y (mgr.14553) 648 : cluster [DBG] pgmap v1193: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.041572+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.041662+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.296547+0000 mon.a (mon.0) 3632 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:19.306602+0000 mon.a (mon.0) 3633 : audit [INF] from='client.54377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.306660+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-10T08:49:20.633 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.571607+0000 mgr.y (mgr.14553) 648 : cluster [DBG] pgmap v1193: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:20.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:20.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:20.767 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:20.787 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:20.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:20.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:20.921 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:20.941 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:20.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:21.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:21.075 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:21.095 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:21.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:21.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:21.231 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:21.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.687938+0000 osd.7 (osd.7) 316 : cluster [DBG] 234.7 scrub starts 2024-02-10T08:49:21.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:19.690082+0000 osd.7 (osd.7) 317 : cluster [DBG] 234.7 scrub ok 2024-02-10T08:49:21.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.027756+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.e scrub starts 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.031221+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.e scrub ok 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.060385+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.e scrub starts 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.063273+0000 osd.2 (osd.2) 292 : cluster [DBG] 16.e scrub ok 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:20.098492+0000 mgr.y (mgr.14553) 649 : audit [DBG] from='client.28785 v1:172.21.15.107:0/218921361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.322202+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.324120+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.565559+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.566493+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.649683+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:20.649893+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.193889+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-10T08:49:21.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.687938+0000 osd.7 (osd.7) 316 : cluster [DBG] 234.7 scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:19.690082+0000 osd.7 (osd.7) 317 : cluster [DBG] 234.7 scrub ok 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.027756+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.e scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.031221+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.e scrub ok 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.060385+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.e scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.063273+0000 osd.2 (osd.2) 292 : cluster [DBG] 16.e scrub ok 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:20.098492+0000 mgr.y (mgr.14553) 649 : audit [DBG] from='client.28785 v1:172.21.15.107:0/218921361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.322202+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.324120+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.565559+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.566493+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.649683+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:21.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:20.649893+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:21.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.193889+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-10T08:49:21.516 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1193,"stamp":"2024-02-10T08:49:19.571402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006114,"num_objects":238,"num_object_clones":1,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":1338,"num_read_kb":1075,"num_write":786,"num_write_kb":4012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":13320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":4,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":943,"ondisk_log_size":943,"up":582,"acting":582,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":582,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2460960,"kb_used_data":56544,"kb_used_omap":59,"kb_used_meta":2404292,"kb_avail":747500256,"statfs":{"total":767960285184,"available":765440262144,"internally_reserved":0,"allocated":57901056,"data_stored":33713179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2461995627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":0,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.292151"},"pg_stats":[{"pgid":"252.1f","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327113+0000","last_change":"2024-02-10T08:49:07.199234+0000","last_active":"2024-02-10T08:49:13.327113+0000","last_peered":"2024-02-10T08:49:13.327113+0000","last_clean":"2024-02-10T08:49:13.327113+0000","last_became_active":"2024-02-10T08:49:07.198954+0000","last_became_peered":"2024-02-10T08:49:07.198954+0000","last_unstale":"2024-02-10T08:49:13.327113+0000","last_undegraded":"2024-02-10T08:49:13.327113+0000","last_fullsized":"2024-02-10T08:49:13.327113+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.1e","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280238+0000","last_change":"2024-02-10T08:49:07.204209+0000","last_active":"2024-02-10T08:49:14.280238+0000","last_peered":"2024-02-10T08:49:14.280238+0000","last_clean":"2024-02-10T08:49:14.280238+0000","last_became_active":"2024-02-10T08:49:07.203899+0000","last_became_peered":"2024-02-10T08:49:07.203899+0000","last_unstale":"2024-02-10T08:49:14.280238+0000","last_undegraded":"2024-02-10T08:49:14.280238+0000","last_fullsized":"2024-02-10T08:49:14.280238+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.1d","version":"0'0","reported_seq":18,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283216+0000","last_change":"2024-02-10T08:49:07.197522+0000","last_active":"2024-02-10T08:49:15.283216+0000","last_peered":"2024-02-10T08:49:15.283216+0000","last_clean":"2024-02-10T08:49:15.283216+0000","last_became_active":"2024-02-10T08:49:07.197200+0000","last_became_peered":"2024-02-10T08:49:07.197200+0000","last_unstale":"2024-02-10T08:49:15.283216+0000","last_undegraded":"2024-02-10T08:49:15.283216+0000","last_fullsized":"2024-02-10T08:49:15.283216+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"252.1c","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281836+0000","last_change":"2024-02-10T08:49:07.203877+0000","last_active":"2024-02-10T08:49:14.281836+0000","last_peered":"2024-02-10T08:49:14.281836+0000","last_clean":"2024-02-10T08:49:14.281836+0000","last_became_active":"2024-02-10T08:49:07.203561+0000","last_became_peered":"2024-02-10T08:49:07.203561+0000","last_unstale":"2024-02-10T08:49:14.281836+0000","last_undegraded":"2024-02-10T08:49:14.281836+0000","last_fullsized":"2024-02-10T08:49:14.281836+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.1b","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484873+0000","last_change":"2024-02-10T08:49:07.209393+0000","last_active":"2024-02-10T08:49:13.484873+0000","last_peered":"2024-02-10T08:49:13.484873+0000","last_clean":"2024-02-10T08:49:13.484873+0000","last_became_active":"2024-02-10T08:49:07.209079+0000","last_became_peered":"2024-02-10T08:49:07.209079+0000","last_unstale":"2024-02-10T08:49:13.484873+0000","last_undegraded":"2024-02-10T08:49:13.484873+0000","last_fullsized":"2024-02-10T08:49:13.484873+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"252.1a","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125669+0000","last_change":"2024-02-10T08:49:07.197926+0000","last_active":"2024-02-10T08:49:15.125669+0000","last_peered":"2024-02-10T08:49:15.125669+0000","last_clean":"2024-02-10T08:49:15.125669+0000","last_became_active":"2024-02-10T08:49:07.197229+0000","last_became_peered":"2024-02-10T08:49:07.197229+0000","last_unstale":"2024-02-10T08:49:15.125669+0000","last_undegraded":"2024-02-10T08:49:15.125669+0000","last_fullsized":"2024-02-10T08:49:15.125669+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"252.19","version":"0'0","reported_seq":16,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485618+0000","last_change":"2024-02-10T08:49:07.198296+0000","last_active":"2024-02-10T08:49:13.485618+0000","last_peered":"2024-02-10T08:49:13.485618+0000","last_clean":"2024-02-10T08:49:13.485618+0000","last_became_active":"2024-02-10T08:49:07.197954+0000","last_became_peered":"2024-02-10T08:49:07.197954+0000","last_unstale":"2024-02-10T08:49:13.485618+0000","last_undegraded":"2024-02-10T08:49:13.485618+0000","last_fullsized":"2024-02-10T08:49:13.485618+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.18","version":"777'2","reported_seq":18,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326318+0000","last_change":"2024-02-10T08:49:07.205883+0000","last_active":"2024-02-10T08:49:13.326318+0000","last_peered":"2024-02-10T08:49:13.326318+0000","last_clean":"2024-02-10T08:49:13.326318+0000","last_became_active":"2024-02-10T08:49:07.205362+0000","last_became_peered":"2024-02-10T08:49:07.205362+0000","last_unstale":"2024-02-10T08:49:13.326318+0000","last_undegraded":"2024-02-10T08:49:13.326318+0000","last_fullsized":"2024-02-10T08:49:13.326318+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.f","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327204+0000","last_change":"2024-02-10T08:49:07.207764+0000","last_active":"2024-02-10T08:49:13.327204+0000","last_peered":"2024-02-10T08:49:13.327204+0000","last_clean":"2024-02-10T08:49:13.327204+0000","last_became_active":"2024-02-10T08:49:07.207471+0000","last_became_peered":"2024-02-10T08:49:07.207471+0000","last_unstale":"2024-02-10T08:49:13.327204+0000","last_undegraded":"2024-02-10T08:49:13.327204+0000","last_fullsized":"2024-02-10T08:49:13.327204+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.e","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125850+0000","last_change":"2024-02-10T08:49:07.203880+0000","last_active":"2024-02-10T08:49:15.125850+0000","last_peered":"2024-02-10T08:49:15.125850+0000","last_clean":"2024-02-10T08:49:15.125850+0000","last_became_active":"2024-02-10T08:49:07.203456+0000","last_became_peered":"2024-02-10T08:49:07.203456+0000","last_unstale":"2024-02-10T08:49:15.125850+0000","last_undegraded":"2024-02-10T08:49:15.125850+0000","last_fullsized":"2024-02-10T08:49:15.125850+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.d","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326982+0000","last_change":"2024-02-10T08:49:07.198364+0000","last_active":"2024-02-10T08:49:13.326982+0000","last_peered":"2024-02-10T08:49:13.326982+0000","last_clean":"2024-02-10T08:49:13.326982+0000","last_became_active":"2024-02-10T08:49:07.198091+0000","last_became_peered":"2024-02-10T08:49:07.198091+0000","last_unstale":"2024-02-10T08:49:13.326982+0000","last_undegraded":"2024-02-10T08:49:13.326982+0000","last_fullsized":"2024-02-10T08:49:13.326982+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.c","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127152+0000","last_change":"2024-02-10T08:49:07.198644+0000","last_active":"2024-02-10T08:49:15.127152+0000","last_peered":"2024-02-10T08:49:15.127152+0000","last_clean":"2024-02-10T08:49:15.127152+0000","last_became_active":"2024-02-10T08:49:07.197965+0000","last_became_peered":"2024-02-10T08:49:07.197965+0000","last_unstale":"2024-02-10T08:49:15.127152+0000","last_undegraded":"2024-02-10T08:49:15.127152+0000","last_fullsized":"2024-02-10T08:49:15.127152+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.b","version":"0'0","reported_seq":18,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.280585+0000","last_change":"2024-02-10T08:49:07.197592+0000","last_active":"2024-02-10T08:49:15.280585+0000","last_peered":"2024-02-10T08:49:15.280585+0000","last_clean":"2024-02-10T08:49:15.280585+0000","last_became_active":"2024-02-10T08:49:07.197124+0000","last_became_peered":"2024-02-10T08:49:07.197124+0000","last_unstale":"2024-02-10T08:49:15.280585+0000","last_undegraded":"2024-02-10T08:49:15.280585+0000","last_fullsized":"2024-02-10T08:49:15.280585+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"252.a","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125491+0000","last_change":"2024-02-10T08:49:07.197129+0000","last_active":"2024-02-10T08:49:15.125491+0000","last_peered":"2024-02-10T08:49:15.125491+0000","last_clean":"2024-02-10T08:49:15.125491+0000","last_became_active":"2024-02-10T08:49:07.196833+0000","last_became_peered":"2024-02-10T08:49:07.196833+0000","last_unstale":"2024-02-10T08:49:15.125491+0000","last_undegraded":"2024-02-10T08:49:15.125491+0000","last_fullsized":"2024-02-10T08:49:15.125491+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.9","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326465+0000","last_change":"2024-02-10T08:49:07.198374+0000","last_active":"2024-02-10T08:49:13.326465+0000","last_peered":"2024-02-10T08:49:13.326465+0000","last_clean":"2024-02-10T08:49:13.326465+0000","last_became_active":"2024-02-10T08:49:07.198143+0000","last_became_peered":"2024-02-10T08:49:07.198143+0000","last_unstale":"2024-02-10T08:49:13.326465+0000","last_undegraded":"2024-02-10T08:49:13.326465+0000","last_fullsized":"2024-02-10T08:49:13.326465+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.8","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326293+0000","last_change":"2024-02-10T08:49:07.205696+0000","last_active":"2024-02-10T08:49:13.326293+0000","last_peered":"2024-02-10T08:49:13.326293+0000","last_clean":"2024-02-10T08:49:13.326293+0000","last_became_active":"2024-02-10T08:49:07.204967+0000","last_became_peered":"2024-02-10T08:49:07.204967+0000","last_unstale":"2024-02-10T08:49:13.326293+0000","last_undegraded":"2024-02-10T08:49:13.326293+0000","last_fullsized":"2024-02-10T08:49:13.326293+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.7","version":"777'2","reported_seq":20,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126333+0000","last_change":"2024-02-10T08:49:07.207844+0000","last_active":"2024-02-10T08:49:15.126333+0000","last_peered":"2024-02-10T08:49:15.126333+0000","last_clean":"2024-02-10T08:49:15.126333+0000","last_became_active":"2024-02-10T08:49:07.207224+0000","last_became_peered":"2024-02-10T08:49:07.207224+0000","last_unstale":"2024-02-10T08:49:15.126333+0000","last_undegraded":"2024-02-10T08:49:15.126333+0000","last_fullsized":"2024-02-10T08:49:15.126333+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"252.6","version":"0'0","reported_seq":15,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327344+0000","last_change":"2024-02-10T08:49:07.200548+0000","last_active":"2024-02-10T08:49:13.327344+0000","last_peered":"2024-02-10T08:49:13.327344+0000","last_clean":"2024-02-10T08:49:13.327344+0000","last_became_active":"2024-02-10T08:49:07.200380+0000","last_became_peered":"2024-02-10T08:49:07.200380+0000","last_unstale":"2024-02-10T08:49:13.327344+0000","last_undegraded":"2024-02-10T08:49:13.327344+0000","last_fullsized":"2024-02-10T08:49:13.327344+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.5","version":"777'1","reported_seq":18,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484292+0000","last_change":"2024-02-10T08:49:07.202761+0000","last_active":"2024-02-10T08:49:13.484292+0000","last_peered":"2024-02-10T08:49:13.484292+0000","last_clean":"2024-02-10T08:49:13.484292+0000","last_became_active":"2024-02-10T08:49:07.202348+0000","last_became_peered":"2024-02-10T08:49:07.202348+0000","last_unstale":"2024-02-10T08:49:13.484292+0000","last_undegraded":"2024-02-10T08:49:13.484292+0000","last_fullsized":"2024-02-10T08:49:13.484292+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.4","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282035+0000","last_change":"2024-02-10T08:49:07.203455+0000","last_active":"2024-02-10T08:49:14.282035+0000","last_peered":"2024-02-10T08:49:14.282035+0000","last_clean":"2024-02-10T08:49:14.282035+0000","last_became_active":"2024-02-10T08:49:07.203077+0000","last_became_peered":"2024-02-10T08:49:07.203077+0000","last_unstale":"2024-02-10T08:49:14.282035+0000","last_undegraded":"2024-02-10T08:49:14.282035+0000","last_fullsized":"2024-02-10T08:49:14.282035+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.3","version":"783'2","reported_seq":20,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-02-10T08:49:16.614204+0000","last_change":"2024-02-10T08:49:07.206935+0000","last_active":"2024-02-10T08:49:16.614204+0000","last_peered":"2024-02-10T08:49:16.614204+0000","last_clean":"2024-02-10T08:49:16.614204+0000","last_became_active":"2024-02-10T08:49:07.206372+0000","last_became_peered":"2024-02-10T08:49:07.206372+0000","last_unstale":"2024-02-10T08:49:16.614204+0000","last_undegraded":"2024-02-10T08:49:16.614204+0000","last_fullsized":"2024-02-10T08:49:16.614204+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"252.2","version":"0'0","reported_seq":16,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484892+0000","last_change":"2024-02-10T08:49:07.199351+0000","last_active":"2024-02-10T08:49:13.484892+0000","last_peered":"2024-02-10T08:49:13.484892+0000","last_clean":"2024-02-10T08:49:13.484892+0000","last_became_active":"2024-02-10T08:49:07.199031+0000","last_became_peered":"2024-02-10T08:49:07.199031+0000","last_unstale":"2024-02-10T08:49:13.484892+0000","last_undegraded":"2024-02-10T08:49:13.484892+0000","last_fullsized":"2024-02-10T08:49:13.484892+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.1","version":"0'0","reported_seq":16,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485851+0000","last_change":"2024-02-10T08:49:07.200156+0000","last_active":"2024-02-10T08:49:13.485851+0000","last_peered":"2024-02-10T08:49:13.485851+0000","last_clean":"2024-02-10T08:49:13.485851+0000","last_became_active":"2024-02-10T08:49:07.199844+0000","last_became_peered":"2024-02-10T08:49:07.199844+0000","last_unstale":"2024-02-10T08:49:13.485851+0000","last_undegraded":"2024-02-10T08:49:13.485851+0000","last_fullsized":"2024-02-10T08:49:13.485851+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.0","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279286+0000","last_change":"2024-02-10T08:49:07.203992+0000","last_active":"2024-02-10T08:49:14.279286+0000","last_peered":"2024-02-10T08:49:14.279286+0000","last_clean":"2024-02-10T08:49:14.279286+0000","last_became_active":"2024-02-10T08:49:07.203695+0000","last_became_peered":"2024-02-10T08:49:07.203695+0000","last_unstale":"2024-02-10T08:49:14.279286+0000","last_undegraded":"2024-02-10T08:49:14.279286+0000","last_fullsized":"2024-02-10T08:49:14.279286+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.11","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.124802+0000","last_change":"2024-02-10T08:49:07.204463+0000","last_active":"2024-02-10T08:49:15.124802+0000","last_peered":"2024-02-10T08:49:15.124802+0000","last_clean":"2024-02-10T08:49:15.124802+0000","last_became_active":"2024-02-10T08:49:07.204178+0000","last_became_peered":"2024-02-10T08:49:07.204178+0000","last_unstale":"2024-02-10T08:49:15.124802+0000","last_undegraded":"2024-02-10T08:49:15.124802+0000","last_fullsized":"2024-02-10T08:49:15.124802+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.7","version":"774'20","reported_seq":237,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485010+0000","last_change":"2024-02-10T08:47:46.185206+0000","last_active":"2024-02-10T08:49:13.485010+0000","last_peered":"2024-02-10T08:49:13.485010+0000","last_clean":"2024-02-10T08:49:13.485010+0000","last_became_active":"2024-02-10T08:45:46.132344+0000","last_became_peered":"2024-02-10T08:45:46.132344+0000","last_unstale":"2024-02-10T08:49:13.485010+0000","last_undegraded":"2024-02-10T08:49:13.485010+0000","last_fullsized":"2024-02-10T08:49:13.485010+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'16","last_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_deep_scrub":"669'16","last_deep_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_clean_scrub_stamp":"2024-02-10T08:47:22.551825+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.10","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281035+0000","last_change":"2024-02-10T08:49:07.206756+0000","last_active":"2024-02-10T08:49:14.281035+0000","last_peered":"2024-02-10T08:49:14.281035+0000","last_clean":"2024-02-10T08:49:14.281035+0000","last_became_active":"2024-02-10T08:49:07.206454+0000","last_became_peered":"2024-02-10T08:49:07.206454+0000","last_unstale":"2024-02-10T08:49:14.281035+0000","last_undegraded":"2024-02-10T08:49:14.281035+0000","last_fullsized":"2024-02-10T08:49:14.281035+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.6","version":"749'2","reported_seq":244,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279767+0000","last_change":"2024-02-10T08:48:55.061793+0000","last_active":"2024-02-10T08:49:14.279767+0000","last_peered":"2024-02-10T08:49:14.279767+0000","last_clean":"2024-02-10T08:49:14.279767+0000","last_became_active":"2024-02-10T08:47:49.222914+0000","last_became_peered":"2024-02-10T08:47:49.222914+0000","last_unstale":"2024-02-10T08:49:14.279767+0000","last_undegraded":"2024-02-10T08:49:14.279767+0000","last_fullsized":"2024-02-10T08:49:14.279767+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'2","last_scrub_stamp":"2024-02-10T08:48:55.061722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.061722+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.13","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126773+0000","last_change":"2024-02-10T08:49:07.201808+0000","last_active":"2024-02-10T08:49:15.126773+0000","last_peered":"2024-02-10T08:49:15.126773+0000","last_clean":"2024-02-10T08:49:15.126773+0000","last_became_active":"2024-02-10T08:49:07.201507+0000","last_became_peered":"2024-02-10T08:49:07.201507+0000","last_unstale":"2024-02-10T08:49:15.126773+0000","last_undegraded":"2024-02-10T08:49:15.126773+0000","last_fullsized":"2024-02-10T08:49:15.126773+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.5","version":"774'17","reported_seq":261,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281786+0000","last_change":"2024-02-10T08:48:34.935440+0000","last_active":"2024-02-10T08:49:14.281786+0000","last_peered":"2024-02-10T08:49:14.281786+0000","last_clean":"2024-02-10T08:49:14.281786+0000","last_became_active":"2024-02-10T08:46:29.211292+0000","last_became_peered":"2024-02-10T08:46:29.211292+0000","last_unstale":"2024-02-10T08:49:14.281786+0000","last_undegraded":"2024-02-10T08:49:14.281786+0000","last_fullsized":"2024-02-10T08:49:14.281786+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'13","last_scrub_stamp":"2024-02-10T08:48:34.935360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:34.935360+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.12","version":"0'0","reported_seq":18,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282114+0000","last_change":"2024-02-10T08:49:07.203965+0000","last_active":"2024-02-10T08:49:15.282114+0000","last_peered":"2024-02-10T08:49:15.282114+0000","last_clean":"2024-02-10T08:49:15.282114+0000","last_became_active":"2024-02-10T08:49:07.203646+0000","last_became_peered":"2024-02-10T08:49:07.203646+0000","last_unstale":"2024-02-10T08:49:15.282114+0000","last_undegraded":"2024-02-10T08:49:15.282114+0000","last_fullsized":"2024-02-10T08:49:15.282114+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"749'21","reported_seq":292,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127134+0000","last_change":"2024-02-10T08:48:05.926678+0000","last_active":"2024-02-10T08:49:15.127134+0000","last_peered":"2024-02-10T08:49:15.127134+0000","last_clean":"2024-02-10T08:49:15.127134+0000","last_became_active":"2024-02-10T08:47:49.231271+0000","last_became_peered":"2024-02-10T08:47:49.231271+0000","last_unstale":"2024-02-10T08:49:15.127134+0000","last_undegraded":"2024-02-10T08:49:15.127134+0000","last_fullsized":"2024-02-10T08:49:15.127134+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"693'17","last_scrub_stamp":"2024-02-10T08:48:05.926589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.926589+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.15","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280270+0000","last_change":"2024-02-10T08:49:07.203705+0000","last_active":"2024-02-10T08:49:14.280270+0000","last_peered":"2024-02-10T08:49:14.280270+0000","last_clean":"2024-02-10T08:49:14.280270+0000","last_became_active":"2024-02-10T08:49:07.203302+0000","last_became_peered":"2024-02-10T08:49:07.203302+0000","last_unstale":"2024-02-10T08:49:14.280270+0000","last_undegraded":"2024-02-10T08:49:14.280270+0000","last_fullsized":"2024-02-10T08:49:14.280270+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.3","version":"774'4","reported_seq":213,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484969+0000","last_change":"2024-02-10T08:48:03.075504+0000","last_active":"2024-02-10T08:49:13.484969+0000","last_peered":"2024-02-10T08:49:13.484969+0000","last_clean":"2024-02-10T08:49:13.484969+0000","last_became_active":"2024-02-10T08:45:46.129572+0000","last_became_peered":"2024-02-10T08:45:46.129572+0000","last_unstale":"2024-02-10T08:49:13.484969+0000","last_undegraded":"2024-02-10T08:49:13.484969+0000","last_fullsized":"2024-02-10T08:49:13.484969+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.075414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.075414+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000032,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"252.14","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279771+0000","last_change":"2024-02-10T08:49:07.197509+0000","last_active":"2024-02-10T08:49:14.279771+0000","last_peered":"2024-02-10T08:49:14.279771+0000","last_clean":"2024-02-10T08:49:14.279771+0000","last_became_active":"2024-02-10T08:49:07.197213+0000","last_became_peered":"2024-02-10T08:49:07.197213+0000","last_unstale":"2024-02-10T08:49:14.279771+0000","last_undegraded":"2024-02-10T08:49:14.279771+0000","last_fullsized":"2024-02-10T08:49:14.279771+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"749'1","reported_seq":225,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279377+0000","last_change":"2024-02-10T08:48:44.070853+0000","last_active":"2024-02-10T08:49:14.279377+0000","last_peered":"2024-02-10T08:49:14.279377+0000","last_clean":"2024-02-10T08:49:14.279377+0000","last_became_active":"2024-02-10T08:46:15.351881+0000","last_became_peered":"2024-02-10T08:46:15.351881+0000","last_unstale":"2024-02-10T08:49:14.279377+0000","last_undegraded":"2024-02-10T08:49:14.279377+0000","last_fullsized":"2024-02-10T08:49:14.279377+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'1","last_scrub_stamp":"2024-02-10T08:48:44.070730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.070730+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.17","version":"0'0","reported_seq":17,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125821+0000","last_change":"2024-02-10T08:49:07.207261+0000","last_active":"2024-02-10T08:49:15.125821+0000","last_peered":"2024-02-10T08:49:15.125821+0000","last_clean":"2024-02-10T08:49:15.125821+0000","last_became_active":"2024-02-10T08:49:07.206820+0000","last_became_peered":"2024-02-10T08:49:07.206820+0000","last_unstale":"2024-02-10T08:49:15.125821+0000","last_undegraded":"2024-02-10T08:49:15.125821+0000","last_fullsized":"2024-02-10T08:49:15.125821+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.1","version":"763'11","reported_seq":244,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.124733+0000","last_change":"2024-02-10T08:48:39.006369+0000","last_active":"2024-02-10T08:49:15.124733+0000","last_peered":"2024-02-10T08:49:15.124733+0000","last_clean":"2024-02-10T08:49:15.124733+0000","last_became_active":"2024-02-10T08:46:29.219928+0000","last_became_peered":"2024-02-10T08:46:29.219928+0000","last_unstale":"2024-02-10T08:49:15.124733+0000","last_undegraded":"2024-02-10T08:49:15.124733+0000","last_fullsized":"2024-02-10T08:49:15.124733+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'10","last_scrub_stamp":"2024-02-10T08:48:39.006289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:39.006289+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.16","version":"0'0","reported_seq":16,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486476+0000","last_change":"2024-02-10T08:49:07.203601+0000","last_active":"2024-02-10T08:49:13.486476+0000","last_peered":"2024-02-10T08:49:13.486476+0000","last_clean":"2024-02-10T08:49:13.486476+0000","last_became_active":"2024-02-10T08:49:07.203297+0000","last_became_peered":"2024-02-10T08:49:07.203297+0000","last_unstale":"2024-02-10T08:49:13.486476+0000","last_undegraded":"2024-02-10T08:49:13.486476+0000","last_fullsized":"2024-02-10T08:49:13.486476+0000","mapping_epoch":773,"log_start":"0'0","ondisk_log_start":"0'0","created":773,"last_epoch_clean":774,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:06.180785+0000","last_clean_scrub_stamp":"2024-02-10T08:49:06.180785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.0","version":"774'19","reported_seq":202,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125075+0000","last_change":"2024-02-10T08:48:28.968144+0000","last_active":"2024-02-10T08:49:15.125075+0000","last_peered":"2024-02-10T08:49:15.125075+0000","last_clean":"2024-02-10T08:49:15.125075+0000","last_became_active":"2024-02-10T08:46:29.210532+0000","last_became_peered":"2024-02-10T08:46:29.210532+0000","last_unstale":"2024-02-10T08:49:15.125075+0000","last_undegraded":"2024-02-10T08:49:15.125075+0000","last_fullsized":"2024-02-10T08:49:15.125075+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"685'16","last_scrub_stamp":"2024-02-10T08:48:28.968036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.968036+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"636'18","reported_seq":470,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.487890+0000","last_change":"2024-02-10T08:48:29.848382+0000","last_active":"2024-02-10T08:49:13.487890+0000","last_peered":"2024-02-10T08:49:13.487890+0000","last_clean":"2024-02-10T08:49:13.487890+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:49:13.487890+0000","last_undegraded":"2024-02-10T08:49:13.487890+0000","last_fullsized":"2024-02-10T08:49:13.487890+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:48:29.848301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.848301+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":848,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.483680+0000","last_change":"2024-02-10T08:47:58.959348+0000","last_active":"2024-02-10T08:49:13.483680+0000","last_peered":"2024-02-10T08:49:13.483680+0000","last_clean":"2024-02-10T08:49:13.483680+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:49:13.483680+0000","last_undegraded":"2024-02-10T08:49:13.483680+0000","last_fullsized":"2024-02-10T08:49:13.483680+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:47:58.959271+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.959271+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":783,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282643+0000","last_change":"2024-02-10T08:48:03.838139+0000","last_active":"2024-02-10T08:49:14.282643+0000","last_peered":"2024-02-10T08:49:14.282643+0000","last_clean":"2024-02-10T08:49:14.282643+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:49:14.282643+0000","last_undegraded":"2024-02-10T08:49:14.282643+0000","last_fullsized":"2024-02-10T08:49:14.282643+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.838064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.838064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":767,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485194+0000","last_change":"2024-02-10T08:48:01.033569+0000","last_active":"2024-02-10T08:49:13.485194+0000","last_peered":"2024-02-10T08:49:13.485194+0000","last_clean":"2024-02-10T08:49:13.485194+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:49:13.485194+0000","last_undegraded":"2024-02-10T08:49:13.485194+0000","last_fullsized":"2024-02-10T08:49:13.485194+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:01.033502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:01.033502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":724,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282185+0000","last_change":"2024-02-10T08:47:49.237658+0000","last_active":"2024-02-10T08:49:14.282185+0000","last_peered":"2024-02-10T08:49:14.282185+0000","last_clean":"2024-02-10T08:49:14.282185+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:49:14.282185+0000","last_undegraded":"2024-02-10T08:49:14.282185+0000","last_fullsized":"2024-02-10T08:49:14.282185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:47.472086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:47.472086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":230,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326648+0000","last_change":"2024-02-10T08:48:32.371647+0000","last_active":"2024-02-10T08:49:13.326648+0000","last_peered":"2024-02-10T08:49:13.326648+0000","last_clean":"2024-02-10T08:49:13.326648+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:49:13.326648+0000","last_undegraded":"2024-02-10T08:49:13.326648+0000","last_fullsized":"2024-02-10T08:49:13.326648+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.371572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.371572+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":89,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283229+0000","last_change":"2024-02-10T08:48:38.645376+0000","last_active":"2024-02-10T08:49:15.283229+0000","last_peered":"2024-02-10T08:49:15.283229+0000","last_clean":"2024-02-10T08:49:15.283229+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:49:15.283229+0000","last_undegraded":"2024-02-10T08:49:15.283229+0000","last_fullsized":"2024-02-10T08:49:15.283229+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.645301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.645301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":794,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282777+0000","last_change":"2024-02-10T08:48:56.119888+0000","last_active":"2024-02-10T08:49:14.282777+0000","last_peered":"2024-02-10T08:49:14.282777+0000","last_clean":"2024-02-10T08:49:14.282777+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:49:14.282777+0000","last_undegraded":"2024-02-10T08:49:14.282777+0000","last_fullsized":"2024-02-10T08:49:14.282777+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:56.119799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.119799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":166,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326689+0000","last_change":"2024-02-10T08:47:17.396762+0000","last_active":"2024-02-10T08:49:13.326689+0000","last_peered":"2024-02-10T08:49:13.326689+0000","last_clean":"2024-02-10T08:49:13.326689+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:49:13.326689+0000","last_undegraded":"2024-02-10T08:49:13.326689+0000","last_fullsized":"2024-02-10T08:49:13.326689+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.396696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.396696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":716,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486572+0000","last_change":"2024-02-10T08:48:55.016819+0000","last_active":"2024-02-10T08:49:13.486572+0000","last_peered":"2024-02-10T08:49:13.486572+0000","last_clean":"2024-02-10T08:49:13.486572+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:49:13.486572+0000","last_undegraded":"2024-02-10T08:49:13.486572+0000","last_fullsized":"2024-02-10T08:49:13.486572+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.016738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.016738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":942,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279990+0000","last_change":"2024-02-10T08:48:38.016707+0000","last_active":"2024-02-10T08:49:14.279990+0000","last_peered":"2024-02-10T08:49:14.279990+0000","last_clean":"2024-02-10T08:49:14.279990+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:49:14.279990+0000","last_undegraded":"2024-02-10T08:49:14.279990+0000","last_fullsized":"2024-02-10T08:49:14.279990+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:48:38.016615+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.016615+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":781,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282498+0000","last_change":"2024-02-10T08:48:30.851950+0000","last_active":"2024-02-10T08:49:14.282498+0000","last_peered":"2024-02-10T08:49:14.282498+0000","last_clean":"2024-02-10T08:49:14.282498+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:49:14.282498+0000","last_undegraded":"2024-02-10T08:49:14.282498+0000","last_fullsized":"2024-02-10T08:49:14.282498+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.851860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.851860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":815,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281494+0000","last_change":"2024-02-10T08:48:02.843328+0000","last_active":"2024-02-10T08:49:14.281494+0000","last_peered":"2024-02-10T08:49:14.281494+0000","last_clean":"2024-02-10T08:49:14.281494+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:49:14.281494+0000","last_undegraded":"2024-02-10T08:49:14.281494+0000","last_fullsized":"2024-02-10T08:49:14.281494+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.843266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.843266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":334,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485549+0000","last_change":"2024-02-10T08:48:40.860303+0000","last_active":"2024-02-10T08:49:13.485549+0000","last_peered":"2024-02-10T08:49:13.485549+0000","last_clean":"2024-02-10T08:49:13.485549+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:49:13.485549+0000","last_undegraded":"2024-02-10T08:49:13.485549+0000","last_fullsized":"2024-02-10T08:49:13.485549+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.860228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.860228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":344,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486947+0000","last_change":"2024-02-10T08:47:57.641284+0000","last_active":"2024-02-10T08:49:13.486947+0000","last_peered":"2024-02-10T08:49:13.486947+0000","last_clean":"2024-02-10T08:49:13.486947+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:49:13.486947+0000","last_undegraded":"2024-02-10T08:49:13.486947+0000","last_fullsized":"2024-02-10T08:49:13.486947+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.641182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.641182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":716,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127350+0000","last_change":"2024-02-10T08:47:49.965414+0000","last_active":"2024-02-10T08:49:15.127350+0000","last_peered":"2024-02-10T08:49:15.127350+0000","last_clean":"2024-02-10T08:49:15.127350+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:49:15.127350+0000","last_undegraded":"2024-02-10T08:49:15.127350+0000","last_fullsized":"2024-02-10T08:49:15.127350+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:47:26.935760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.935760+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":81,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126744+0000","last_change":"2024-02-10T08:47:49.226548+0000","last_active":"2024-02-10T08:49:15.126744+0000","last_peered":"2024-02-10T08:49:15.126744+0000","last_clean":"2024-02-10T08:49:15.126744+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:49:15.126744+0000","last_undegraded":"2024-02-10T08:49:15.126744+0000","last_fullsized":"2024-02-10T08:49:15.126744+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":89,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282906+0000","last_change":"2024-02-10T08:48:32.722180+0000","last_active":"2024-02-10T08:49:15.282906+0000","last_peered":"2024-02-10T08:49:15.282906+0000","last_clean":"2024-02-10T08:49:15.282906+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:49:15.282906+0000","last_undegraded":"2024-02-10T08:49:15.282906+0000","last_fullsized":"2024-02-10T08:49:15.282906+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.722121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.722121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":772,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.283204+0000","last_change":"2024-02-10T08:48:47.163067+0000","last_active":"2024-02-10T08:49:14.283204+0000","last_peered":"2024-02-10T08:49:14.283204+0000","last_clean":"2024-02-10T08:49:14.283204+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:49:14.283204+0000","last_undegraded":"2024-02-10T08:49:14.283204+0000","last_fullsized":"2024-02-10T08:49:14.283204+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.163011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.163011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":571,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.128035+0000","last_change":"2024-02-10T08:47:49.242441+0000","last_active":"2024-02-10T08:49:15.128035+0000","last_peered":"2024-02-10T08:49:15.128035+0000","last_clean":"2024-02-10T08:49:15.128035+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:49:15.128035+0000","last_undegraded":"2024-02-10T08:49:15.128035+0000","last_fullsized":"2024-02-10T08:49:15.128035+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:48.843209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.843209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":640,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125692+0000","last_change":"2024-02-10T08:48:56.061862+0000","last_active":"2024-02-10T08:49:15.125692+0000","last_peered":"2024-02-10T08:49:15.125692+0000","last_clean":"2024-02-10T08:49:15.125692+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:49:15.125692+0000","last_undegraded":"2024-02-10T08:49:15.125692+0000","last_fullsized":"2024-02-10T08:49:15.125692+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:48:56.061783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.061783+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":84,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282526+0000","last_change":"2024-02-10T08:47:49.236238+0000","last_active":"2024-02-10T08:49:15.282526+0000","last_peered":"2024-02-10T08:49:15.282526+0000","last_clean":"2024-02-10T08:49:15.282526+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:49:15.282526+0000","last_undegraded":"2024-02-10T08:49:15.282526+0000","last_fullsized":"2024-02-10T08:49:15.282526+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:49.582110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.582110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1033,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281875+0000","last_change":"2024-02-10T08:47:29.049751+0000","last_active":"2024-02-10T08:49:14.281875+0000","last_peered":"2024-02-10T08:49:14.281875+0000","last_clean":"2024-02-10T08:49:14.281875+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:49:14.281875+0000","last_undegraded":"2024-02-10T08:49:14.281875+0000","last_fullsized":"2024-02-10T08:49:14.281875+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:47:29.049685+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.049685+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":600,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127507+0000","last_change":"2024-02-10T08:48:48.962890+0000","last_active":"2024-02-10T08:49:15.127507+0000","last_peered":"2024-02-10T08:49:15.127507+0000","last_clean":"2024-02-10T08:49:15.127507+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:49:15.127507+0000","last_undegraded":"2024-02-10T08:49:15.127507+0000","last_fullsized":"2024-02-10T08:49:15.127507+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.962820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.962820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":174,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326792+0000","last_change":"2024-02-10T08:48:29.433315+0000","last_active":"2024-02-10T08:49:13.326792+0000","last_peered":"2024-02-10T08:49:13.326792+0000","last_clean":"2024-02-10T08:49:13.326792+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:49:13.326792+0000","last_undegraded":"2024-02-10T08:49:13.326792+0000","last_fullsized":"2024-02-10T08:49:13.326792+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:29.433223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.433223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":105,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283178+0000","last_change":"2024-02-10T08:48:51.755042+0000","last_active":"2024-02-10T08:49:15.283178+0000","last_peered":"2024-02-10T08:49:15.283178+0000","last_clean":"2024-02-10T08:49:15.283178+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:49:15.283178+0000","last_undegraded":"2024-02-10T08:49:15.283178+0000","last_fullsized":"2024-02-10T08:49:15.283178+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:48:51.754925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.754925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":88,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.281871+0000","last_change":"2024-02-10T08:47:50.656406+0000","last_active":"2024-02-10T08:49:15.281871+0000","last_peered":"2024-02-10T08:49:15.281871+0000","last_clean":"2024-02-10T08:49:15.281871+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:49:15.281871+0000","last_undegraded":"2024-02-10T08:49:15.281871+0000","last_fullsized":"2024-02-10T08:49:15.281871+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:47:13.515134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.515134+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":842,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-02-10T08:49:16.614014+0000","last_change":"2024-02-10T08:47:13.008447+0000","last_active":"2024-02-10T08:49:16.614014+0000","last_peered":"2024-02-10T08:49:16.614014+0000","last_clean":"2024-02-10T08:49:16.614014+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:49:16.614014+0000","last_undegraded":"2024-02-10T08:49:16.614014+0000","last_fullsized":"2024-02-10T08:49:16.614014+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:47:13.008374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.008374+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":780,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485877+0000","last_change":"2024-02-10T08:47:49.238554+0000","last_active":"2024-02-10T08:49:13.485877+0000","last_peered":"2024-02-10T08:49:13.485877+0000","last_clean":"2024-02-10T08:49:13.485877+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:49:13.485877+0000","last_undegraded":"2024-02-10T08:49:13.485877+0000","last_fullsized":"2024-02-10T08:49:13.485877+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:31.837641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:47:31.837641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":729,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485060+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:49:13.485060+0000","last_peered":"2024-02-10T08:49:13.485060+0000","last_clean":"2024-02-10T08:49:13.485060+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:49:13.485060+0000","last_undegraded":"2024-02-10T08:49:13.485060+0000","last_fullsized":"2024-02-10T08:49:13.485060+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.824249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.824249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":678,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127660+0000","last_change":"2024-02-10T08:47:50.476339+0000","last_active":"2024-02-10T08:49:15.127660+0000","last_peered":"2024-02-10T08:49:15.127660+0000","last_clean":"2024-02-10T08:49:15.127660+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:49:15.127660+0000","last_undegraded":"2024-02-10T08:49:15.127660+0000","last_fullsized":"2024-02-10T08:49:15.127660+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:46:45.859396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:45.859396+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":362,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485432+0000","last_change":"2024-02-10T08:48:31.859394+0000","last_active":"2024-02-10T08:49:13.485432+0000","last_peered":"2024-02-10T08:49:13.485432+0000","last_clean":"2024-02-10T08:49:13.485432+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:49:13.485432+0000","last_undegraded":"2024-02-10T08:49:13.485432+0000","last_fullsized":"2024-02-10T08:49:13.485432+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.859315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.859315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":935,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486316+0000","last_change":"2024-02-10T08:48:51.038110+0000","last_active":"2024-02-10T08:49:13.486316+0000","last_peered":"2024-02-10T08:49:13.486316+0000","last_clean":"2024-02-10T08:49:13.486316+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:49:13.486316+0000","last_undegraded":"2024-02-10T08:49:13.486316+0000","last_fullsized":"2024-02-10T08:49:13.486316+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.038025+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":749,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127300+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:49:15.127300+0000","last_peered":"2024-02-10T08:49:15.127300+0000","last_clean":"2024-02-10T08:49:15.127300+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:49:15.127300+0000","last_undegraded":"2024-02-10T08:49:15.127300+0000","last_fullsized":"2024-02-10T08:49:15.127300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":695,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125444+0000","last_change":"2024-02-10T08:46:09.490629+0000","last_active":"2024-02-10T08:49:15.125444+0000","last_peered":"2024-02-10T08:49:15.125444+0000","last_clean":"2024-02-10T08:49:15.125444+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:49:15.125444+0000","last_undegraded":"2024-02-10T08:49:15.125444+0000","last_fullsized":"2024-02-10T08:49:15.125444+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:09.490555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:09.490555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":996,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.124762+0000","last_change":"2024-02-10T08:47:49.260456+0000","last_active":"2024-02-10T08:49:15.124762+0000","last_peered":"2024-02-10T08:49:15.124762+0000","last_clean":"2024-02-10T08:49:15.124762+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:49:15.124762+0000","last_undegraded":"2024-02-10T08:49:15.124762+0000","last_fullsized":"2024-02-10T08:49:15.124762+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:39.499038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.499038+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":654,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127884+0000","last_change":"2024-02-10T08:47:27.942949+0000","last_active":"2024-02-10T08:49:15.127884+0000","last_peered":"2024-02-10T08:49:15.127884+0000","last_clean":"2024-02-10T08:49:15.127884+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:49:15.127884+0000","last_undegraded":"2024-02-10T08:49:15.127884+0000","last_fullsized":"2024-02-10T08:49:15.127884+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:47:27.942861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.942861+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":791,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280421+0000","last_change":"2024-02-10T08:48:54.105562+0000","last_active":"2024-02-10T08:49:14.280421+0000","last_peered":"2024-02-10T08:49:14.280421+0000","last_clean":"2024-02-10T08:49:14.280421+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:49:14.280421+0000","last_undegraded":"2024-02-10T08:49:14.280421+0000","last_fullsized":"2024-02-10T08:49:14.280421+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.105490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.105490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":336,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486410+0000","last_change":"2024-02-10T08:47:55.566362+0000","last_active":"2024-02-10T08:49:13.486410+0000","last_peered":"2024-02-10T08:49:13.486410+0000","last_clean":"2024-02-10T08:49:13.486410+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:49:13.486410+0000","last_undegraded":"2024-02-10T08:49:13.486410+0000","last_fullsized":"2024-02-10T08:49:13.486410+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.566281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.566281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":189,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327078+0000","last_change":"2024-02-10T08:48:05.524065+0000","last_active":"2024-02-10T08:49:13.327078+0000","last_peered":"2024-02-10T08:49:13.327078+0000","last_clean":"2024-02-10T08:49:13.327078+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:49:13.327078+0000","last_undegraded":"2024-02-10T08:49:13.327078+0000","last_fullsized":"2024-02-10T08:49:13.327078+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:05.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":619,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127743+0000","last_change":"2024-02-10T08:48:45.969499+0000","last_active":"2024-02-10T08:49:15.127743+0000","last_peered":"2024-02-10T08:49:15.127743+0000","last_clean":"2024-02-10T08:49:15.127743+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:49:15.127743+0000","last_undegraded":"2024-02-10T08:49:15.127743+0000","last_fullsized":"2024-02-10T08:49:15.127743+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.969416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.969416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":942,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125133+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:49:15.125133+0000","last_peered":"2024-02-10T08:49:15.125133+0000","last_clean":"2024-02-10T08:49:15.125133+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:49:15.125133+0000","last_undegraded":"2024-02-10T08:49:15.125133+0000","last_fullsized":"2024-02-10T08:49:15.125133+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":93,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283222+0000","last_change":"2024-02-10T08:48:36.653104+0000","last_active":"2024-02-10T08:49:15.283222+0000","last_peered":"2024-02-10T08:49:15.283222+0000","last_clean":"2024-02-10T08:49:15.283222+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:49:15.283222+0000","last_undegraded":"2024-02-10T08:49:15.283222+0000","last_fullsized":"2024-02-10T08:49:15.283222+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:48:36.652981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.652981+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":587,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125755+0000","last_change":"2024-02-10T08:48:42.984939+0000","last_active":"2024-02-10T08:49:15.125755+0000","last_peered":"2024-02-10T08:49:15.125755+0000","last_clean":"2024-02-10T08:49:15.125755+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:49:15.125755+0000","last_undegraded":"2024-02-10T08:49:15.125755+0000","last_fullsized":"2024-02-10T08:49:15.125755+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.984822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.984822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":189,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327000+0000","last_change":"2024-02-10T08:48:37.427622+0000","last_active":"2024-02-10T08:49:13.327000+0000","last_peered":"2024-02-10T08:49:13.327000+0000","last_clean":"2024-02-10T08:49:13.327000+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:49:13.327000+0000","last_undegraded":"2024-02-10T08:49:13.327000+0000","last_fullsized":"2024-02-10T08:49:13.327000+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:37.427518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:37.427518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":778,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280097+0000","last_change":"2024-02-10T08:48:43.081221+0000","last_active":"2024-02-10T08:49:14.280097+0000","last_peered":"2024-02-10T08:49:14.280097+0000","last_clean":"2024-02-10T08:49:14.280097+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:49:14.280097+0000","last_undegraded":"2024-02-10T08:49:14.280097+0000","last_fullsized":"2024-02-10T08:49:14.280097+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.081148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.081148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":184,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326791+0000","last_change":"2024-02-10T08:47:57.560707+0000","last_active":"2024-02-10T08:49:13.326791+0000","last_peered":"2024-02-10T08:49:13.326791+0000","last_clean":"2024-02-10T08:49:13.326791+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:49:13.326791+0000","last_undegraded":"2024-02-10T08:49:13.326791+0000","last_fullsized":"2024-02-10T08:49:13.326791+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:47:57.560587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.560587+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":359,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484851+0000","last_change":"2024-02-10T08:47:58.641964+0000","last_active":"2024-02-10T08:49:13.484851+0000","last_peered":"2024-02-10T08:49:13.484851+0000","last_clean":"2024-02-10T08:49:13.484851+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:49:13.484851+0000","last_undegraded":"2024-02-10T08:49:13.484851+0000","last_fullsized":"2024-02-10T08:49:13.484851+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:58.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.641895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":660,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281202+0000","last_change":"2024-02-10T08:47:49.238746+0000","last_active":"2024-02-10T08:49:14.281202+0000","last_peered":"2024-02-10T08:49:14.281202+0000","last_clean":"2024-02-10T08:49:14.281202+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:49:14.281202+0000","last_undegraded":"2024-02-10T08:49:14.281202+0000","last_fullsized":"2024-02-10T08:49:14.281202+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:39.455169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.455169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":747,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.283027+0000","last_change":"2024-02-10T08:48:39.036788+0000","last_active":"2024-02-10T08:49:14.283027+0000","last_peered":"2024-02-10T08:49:14.283027+0000","last_clean":"2024-02-10T08:49:14.283027+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:49:14.283027+0000","last_undegraded":"2024-02-10T08:49:14.283027+0000","last_fullsized":"2024-02-10T08:49:14.283027+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:39.036699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:39.036699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":806,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486404+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:49:13.486404+0000","last_peered":"2024-02-10T08:49:13.486404+0000","last_clean":"2024-02-10T08:49:13.486404+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:49:13.486404+0000","last_undegraded":"2024-02-10T08:49:13.486404+0000","last_fullsized":"2024-02-10T08:49:13.486404+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:13.825974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.825974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":226,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327211+0000","last_change":"2024-02-10T08:47:49.975699+0000","last_active":"2024-02-10T08:49:13.327211+0000","last_peered":"2024-02-10T08:49:13.327211+0000","last_clean":"2024-02-10T08:49:13.327211+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:49:13.327211+0000","last_undegraded":"2024-02-10T08:49:13.327211+0000","last_fullsized":"2024-02-10T08:49:13.327211+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.370523+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":361,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485420+0000","last_change":"2024-02-10T08:47:49.252790+0000","last_active":"2024-02-10T08:49:13.485420+0000","last_peered":"2024-02-10T08:49:13.485420+0000","last_clean":"2024-02-10T08:49:13.485420+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:49:13.485420+0000","last_undegraded":"2024-02-10T08:49:13.485420+0000","last_fullsized":"2024-02-10T08:49:13.485420+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.601673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.601673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":182,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326626+0000","last_change":"2024-02-10T08:48:36.452244+0000","last_active":"2024-02-10T08:49:13.326626+0000","last_peered":"2024-02-10T08:49:13.326626+0000","last_clean":"2024-02-10T08:49:13.326626+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:49:13.326626+0000","last_undegraded":"2024-02-10T08:49:13.326626+0000","last_fullsized":"2024-02-10T08:49:13.326626+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.452178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.452178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":94,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283007+0000","last_change":"2024-02-10T08:48:05.945279+0000","last_active":"2024-02-10T08:49:15.283007+0000","last_peered":"2024-02-10T08:49:15.283007+0000","last_clean":"2024-02-10T08:49:15.283007+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:49:15.283007+0000","last_undegraded":"2024-02-10T08:49:15.283007+0000","last_fullsized":"2024-02-10T08:49:15.283007+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.945162+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":720,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279768+0000","last_change":"2024-02-10T08:47:26.751598+0000","last_active":"2024-02-10T08:49:14.279768+0000","last_peered":"2024-02-10T08:49:14.279768+0000","last_clean":"2024-02-10T08:49:14.279768+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:49:14.279768+0000","last_undegraded":"2024-02-10T08:49:14.279768+0000","last_fullsized":"2024-02-10T08:49:14.279768+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.751532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.751532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":341,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485187+0000","last_change":"2024-02-10T08:48:54.704302+0000","last_active":"2024-02-10T08:49:13.485187+0000","last_peered":"2024-02-10T08:49:13.485187+0000","last_clean":"2024-02-10T08:49:13.485187+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:49:13.485187+0000","last_undegraded":"2024-02-10T08:49:13.485187+0000","last_fullsized":"2024-02-10T08:49:13.485187+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.704183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.704183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":857,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280074+0000","last_change":"2024-02-10T08:48:31.839331+0000","last_active":"2024-02-10T08:49:14.280074+0000","last_peered":"2024-02-10T08:49:14.280074+0000","last_clean":"2024-02-10T08:49:14.280074+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:49:14.280074+0000","last_undegraded":"2024-02-10T08:49:14.280074+0000","last_fullsized":"2024-02-10T08:49:14.280074+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:48:31.839259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.839259+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":345,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485644+0000","last_change":"2024-02-10T08:48:30.896370+0000","last_active":"2024-02-10T08:49:13.485644+0000","last_peered":"2024-02-10T08:49:13.485644+0000","last_clean":"2024-02-10T08:49:13.485644+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:49:13.485644+0000","last_undegraded":"2024-02-10T08:49:13.485644+0000","last_fullsized":"2024-02-10T08:49:13.485644+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.896306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.896306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":768,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485227+0000","last_change":"2024-02-10T08:47:16.775257+0000","last_active":"2024-02-10T08:49:13.485227+0000","last_peered":"2024-02-10T08:49:13.485227+0000","last_clean":"2024-02-10T08:49:13.485227+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:49:13.485227+0000","last_undegraded":"2024-02-10T08:49:13.485227+0000","last_fullsized":"2024-02-10T08:49:13.485227+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.775179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.775179+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":751,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125671+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:49:15.125671+0000","last_peered":"2024-02-10T08:49:15.125671+0000","last_clean":"2024-02-10T08:49:15.125671+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:49:15.125671+0000","last_undegraded":"2024-02-10T08:49:15.125671+0000","last_fullsized":"2024-02-10T08:49:15.125671+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":759,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.486494+0000","last_change":"2024-02-10T08:48:48.079333+0000","last_active":"2024-02-10T08:49:13.486494+0000","last_peered":"2024-02-10T08:49:13.486494+0000","last_clean":"2024-02-10T08:49:13.486494+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:49:13.486494+0000","last_undegraded":"2024-02-10T08:49:13.486494+0000","last_fullsized":"2024-02-10T08:49:13.486494+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.079269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.079269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":733,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.483796+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:49:13.483796+0000","last_peered":"2024-02-10T08:49:13.483796+0000","last_clean":"2024-02-10T08:49:13.483796+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:49:13.483796+0000","last_undegraded":"2024-02-10T08:49:13.483796+0000","last_fullsized":"2024-02-10T08:49:13.483796+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":795,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282797+0000","last_change":"2024-02-10T08:47:26.043612+0000","last_active":"2024-02-10T08:49:14.282797+0000","last_peered":"2024-02-10T08:49:14.282797+0000","last_clean":"2024-02-10T08:49:14.282797+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:49:14.282797+0000","last_undegraded":"2024-02-10T08:49:14.282797+0000","last_fullsized":"2024-02-10T08:49:14.282797+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.043548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.043548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":96,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283195+0000","last_change":"2024-02-10T08:47:53.993563+0000","last_active":"2024-02-10T08:49:15.283195+0000","last_peered":"2024-02-10T08:49:15.283195+0000","last_clean":"2024-02-10T08:49:15.283195+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:49:15.283195+0000","last_undegraded":"2024-02-10T08:49:15.283195+0000","last_fullsized":"2024-02-10T08:49:15.283195+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:47:30.847592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.847592+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":783,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281137+0000","last_change":"2024-02-10T08:47:18.043795+0000","last_active":"2024-02-10T08:49:14.281137+0000","last_peered":"2024-02-10T08:49:14.281137+0000","last_clean":"2024-02-10T08:49:14.281137+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:49:14.281137+0000","last_undegraded":"2024-02-10T08:49:14.281137+0000","last_fullsized":"2024-02-10T08:49:14.281137+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:18.043731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.043731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":175,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.383783+0000","last_change":"2024-02-10T08:47:49.933829+0000","last_active":"2024-02-10T08:49:15.383783+0000","last_peered":"2024-02-10T08:49:15.383783+0000","last_clean":"2024-02-10T08:49:15.383783+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:49:15.383783+0000","last_undegraded":"2024-02-10T08:49:15.383783+0000","last_fullsized":"2024-02-10T08:49:15.383783+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.043513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":184,"num_read_kb":179,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":333,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484737+0000","last_change":"2024-02-10T08:48:47.787415+0000","last_active":"2024-02-10T08:49:13.484737+0000","last_peered":"2024-02-10T08:49:13.484737+0000","last_clean":"2024-02-10T08:49:13.484737+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:49:13.484737+0000","last_undegraded":"2024-02-10T08:49:13.484737+0000","last_fullsized":"2024-02-10T08:49:13.484737+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.787351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":84,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.281115+0000","last_change":"2024-02-10T08:47:49.236998+0000","last_active":"2024-02-10T08:49:15.281115+0000","last_peered":"2024-02-10T08:49:15.281115+0000","last_clean":"2024-02-10T08:49:15.281115+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:49:15.281115+0000","last_undegraded":"2024-02-10T08:49:15.281115+0000","last_fullsized":"2024-02-10T08:49:15.281115+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.556602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":200,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327208+0000","last_change":"2024-02-10T08:47:59.541068+0000","last_active":"2024-02-10T08:49:13.327208+0000","last_peered":"2024-02-10T08:49:13.327208+0000","last_clean":"2024-02-10T08:49:13.327208+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:49:13.327208+0000","last_undegraded":"2024-02-10T08:49:13.327208+0000","last_fullsized":"2024-02-10T08:49:13.327208+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:59.541002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:47:59.541002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"711'43","reported_seq":128,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282942+0000","last_change":"2024-02-10T08:48:28.753355+0000","last_active":"2024-02-10T08:49:15.282942+0000","last_peered":"2024-02-10T08:49:15.282942+0000","last_clean":"2024-02-10T08:49:15.282942+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:49:15.282942+0000","last_undegraded":"2024-02-10T08:49:15.282942+0000","last_fullsized":"2024-02-10T08:49:15.282942+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'43","last_scrub_stamp":"2024-02-10T08:48:28.753288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.753288+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":773,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485280+0000","last_change":"2024-02-10T08:48:31.134699+0000","last_active":"2024-02-10T08:49:13.485280+0000","last_peered":"2024-02-10T08:49:13.485280+0000","last_clean":"2024-02-10T08:49:13.485280+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:49:13.485280+0000","last_undegraded":"2024-02-10T08:49:13.485280+0000","last_fullsized":"2024-02-10T08:49:13.485280+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.134575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.134575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":773,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126060+0000","last_change":"2024-02-10T08:46:33.460971+0000","last_active":"2024-02-10T08:49:15.126060+0000","last_peered":"2024-02-10T08:49:15.126060+0000","last_clean":"2024-02-10T08:49:15.126060+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:49:15.126060+0000","last_undegraded":"2024-02-10T08:49:15.126060+0000","last_fullsized":"2024-02-10T08:49:15.126060+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:33.460870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:33.460870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":747,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281174+0000","last_change":"2024-02-10T08:48:54.135964+0000","last_active":"2024-02-10T08:49:14.281174+0000","last_peered":"2024-02-10T08:49:14.281174+0000","last_clean":"2024-02-10T08:49:14.281174+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:49:14.281174+0000","last_undegraded":"2024-02-10T08:49:14.281174+0000","last_fullsized":"2024-02-10T08:49:14.281174+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.135900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.135900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":206,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326779+0000","last_change":"2024-02-10T08:47:18.403493+0000","last_active":"2024-02-10T08:49:13.326779+0000","last_peered":"2024-02-10T08:49:13.326779+0000","last_clean":"2024-02-10T08:49:13.326779+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:49:13.326779+0000","last_undegraded":"2024-02-10T08:49:13.326779+0000","last_fullsized":"2024-02-10T08:49:13.326779+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:47:18.403400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.403400+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":101,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282482+0000","last_change":"2024-02-10T08:48:42.684268+0000","last_active":"2024-02-10T08:49:15.282482+0000","last_peered":"2024-02-10T08:49:15.282482+0000","last_clean":"2024-02-10T08:49:15.282482+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:49:15.282482+0000","last_undegraded":"2024-02-10T08:49:15.282482+0000","last_fullsized":"2024-02-10T08:49:15.282482+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:48:42.684181+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.684181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283023+0000","last_change":"2024-02-10T08:47:49.256946+0000","last_active":"2024-02-10T08:49:15.283023+0000","last_peered":"2024-02-10T08:49:15.283023+0000","last_clean":"2024-02-10T08:49:15.283023+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:49:15.283023+0000","last_undegraded":"2024-02-10T08:49:15.283023+0000","last_fullsized":"2024-02-10T08:49:15.283023+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.363337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.363337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":171,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326616+0000","last_change":"2024-02-10T08:48:33.412887+0000","last_active":"2024-02-10T08:49:13.326616+0000","last_peered":"2024-02-10T08:49:13.326616+0000","last_clean":"2024-02-10T08:49:13.326616+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:49:13.326616+0000","last_undegraded":"2024-02-10T08:49:13.326616+0000","last_fullsized":"2024-02-10T08:49:13.326616+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.412775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.412775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":718,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125893+0000","last_change":"2024-02-10T08:48:50.100054+0000","last_active":"2024-02-10T08:49:15.125893+0000","last_peered":"2024-02-10T08:49:15.125893+0000","last_clean":"2024-02-10T08:49:15.125893+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:49:15.125893+0000","last_undegraded":"2024-02-10T08:49:15.125893+0000","last_fullsized":"2024-02-10T08:49:15.125893+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.099951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.099951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":818,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281383+0000","last_change":"2024-02-10T08:48:41.070217+0000","last_active":"2024-02-10T08:49:14.281383+0000","last_peered":"2024-02-10T08:49:14.281383+0000","last_clean":"2024-02-10T08:49:14.281383+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:49:14.281383+0000","last_undegraded":"2024-02-10T08:49:14.281383+0000","last_fullsized":"2024-02-10T08:49:14.281383+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.070126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.070126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":689,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127970+0000","last_change":"2024-02-10T08:48:31.973099+0000","last_active":"2024-02-10T08:49:15.127970+0000","last_peered":"2024-02-10T08:49:15.127970+0000","last_clean":"2024-02-10T08:49:15.127970+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:49:15.127970+0000","last_undegraded":"2024-02-10T08:49:15.127970+0000","last_fullsized":"2024-02-10T08:49:15.127970+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:48:31.973020+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.973020+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":654,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126424+0000","last_change":"2024-02-10T08:47:17.985317+0000","last_active":"2024-02-10T08:49:15.126424+0000","last_peered":"2024-02-10T08:49:15.126424+0000","last_clean":"2024-02-10T08:49:15.126424+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:49:15.126424+0000","last_undegraded":"2024-02-10T08:49:15.126424+0000","last_fullsized":"2024-02-10T08:49:15.126424+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:47:17.985235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.985235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":786,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126969+0000","last_change":"2024-02-10T08:48:38.003058+0000","last_active":"2024-02-10T08:49:15.126969+0000","last_peered":"2024-02-10T08:49:15.126969+0000","last_clean":"2024-02-10T08:49:15.126969+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:49:15.126969+0000","last_undegraded":"2024-02-10T08:49:15.126969+0000","last_fullsized":"2024-02-10T08:49:15.126969+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.002996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.002996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":102,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484483+0000","last_change":"2024-02-10T08:48:33.158780+0000","last_active":"2024-02-10T08:49:13.484483+0000","last_peered":"2024-02-10T08:49:13.484483+0000","last_clean":"2024-02-10T08:49:13.484483+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:49:13.484483+0000","last_undegraded":"2024-02-10T08:49:13.484483+0000","last_fullsized":"2024-02-10T08:49:13.484483+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.158671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.158671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":180,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327309+0000","last_change":"2024-02-10T08:48:54.417289+0000","last_active":"2024-02-10T08:49:13.327309+0000","last_peered":"2024-02-10T08:49:13.327309+0000","last_clean":"2024-02-10T08:49:13.327309+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:49:13.327309+0000","last_undegraded":"2024-02-10T08:49:13.327309+0000","last_fullsized":"2024-02-10T08:49:13.327309+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:48:54.417173+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.417173+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":805,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281309+0000","last_change":"2024-02-10T08:48:50.118042+0000","last_active":"2024-02-10T08:49:14.281309+0000","last_peered":"2024-02-10T08:49:14.281309+0000","last_clean":"2024-02-10T08:49:14.281309+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:49:14.281309+0000","last_undegraded":"2024-02-10T08:49:14.281309+0000","last_fullsized":"2024-02-10T08:49:14.281309+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.117976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":785,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282347+0000","last_change":"2024-02-10T08:46:38.449729+0000","last_active":"2024-02-10T08:49:14.282347+0000","last_peered":"2024-02-10T08:49:14.282347+0000","last_clean":"2024-02-10T08:49:14.282347+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:49:14.282347+0000","last_undegraded":"2024-02-10T08:49:14.282347+0000","last_fullsized":"2024-02-10T08:49:14.282347+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:38.449611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:38.449611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":582,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126202+0000","last_change":"2024-02-10T08:48:41.943374+0000","last_active":"2024-02-10T08:49:15.126202+0000","last_peered":"2024-02-10T08:49:15.126202+0000","last_clean":"2024-02-10T08:49:15.126202+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:49:15.126202+0000","last_undegraded":"2024-02-10T08:49:15.126202+0000","last_fullsized":"2024-02-10T08:49:15.126202+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.943287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.943287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":835,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126381+0000","last_change":"2024-02-10T08:47:49.978588+0000","last_active":"2024-02-10T08:49:15.126381+0000","last_peered":"2024-02-10T08:49:15.126381+0000","last_clean":"2024-02-10T08:49:15.126381+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:49:15.126381+0000","last_undegraded":"2024-02-10T08:49:15.126381+0000","last_fullsized":"2024-02-10T08:49:15.126381+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:46:49.515525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.515525+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":787,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279209+0000","last_change":"2024-02-10T08:48:36.966625+0000","last_active":"2024-02-10T08:49:14.279209+0000","last_peered":"2024-02-10T08:49:14.279209+0000","last_clean":"2024-02-10T08:49:14.279209+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:49:14.279209+0000","last_undegraded":"2024-02-10T08:49:14.279209+0000","last_fullsized":"2024-02-10T08:49:14.279209+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.966559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.966559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":344,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484559+0000","last_change":"2024-02-10T08:48:28.810015+0000","last_active":"2024-02-10T08:49:13.484559+0000","last_peered":"2024-02-10T08:49:13.484559+0000","last_clean":"2024-02-10T08:49:13.484559+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:49:13.484559+0000","last_undegraded":"2024-02-10T08:49:13.484559+0000","last_fullsized":"2024-02-10T08:49:13.484559+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:28.809930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.809930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":710,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485139+0000","last_change":"2024-02-10T08:48:47.081659+0000","last_active":"2024-02-10T08:49:13.485139+0000","last_peered":"2024-02-10T08:49:13.485139+0000","last_clean":"2024-02-10T08:49:13.485139+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:49:13.485139+0000","last_undegraded":"2024-02-10T08:49:13.485139+0000","last_fullsized":"2024-02-10T08:49:13.485139+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.081542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":863,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127327+0000","last_change":"2024-02-10T08:47:49.996933+0000","last_active":"2024-02-10T08:49:15.127327+0000","last_peered":"2024-02-10T08:49:15.127327+0000","last_clean":"2024-02-10T08:49:15.127327+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:49:15.127327+0000","last_undegraded":"2024-02-10T08:49:15.127327+0000","last_fullsized":"2024-02-10T08:49:15.127327+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:45:58.480952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:45:58.480952+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":629,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125579+0000","last_change":"2024-02-10T08:48:45.089054+0000","last_active":"2024-02-10T08:49:15.125579+0000","last_peered":"2024-02-10T08:49:15.125579+0000","last_clean":"2024-02-10T08:49:15.125579+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:49:15.125579+0000","last_undegraded":"2024-02-10T08:49:15.125579+0000","last_fullsized":"2024-02-10T08:49:15.125579+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.088965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.088965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":998,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485717+0000","last_change":"2024-02-10T08:47:49.363312+0000","last_active":"2024-02-10T08:49:13.485717+0000","last_peered":"2024-02-10T08:49:13.485717+0000","last_clean":"2024-02-10T08:49:13.485717+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:49:13.485717+0000","last_undegraded":"2024-02-10T08:49:13.485717+0000","last_fullsized":"2024-02-10T08:49:13.485717+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.886696+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":171,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327257+0000","last_change":"2024-02-10T08:48:38.471156+0000","last_active":"2024-02-10T08:49:13.327257+0000","last_peered":"2024-02-10T08:49:13.327257+0000","last_clean":"2024-02-10T08:49:13.327257+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:49:13.327257+0000","last_undegraded":"2024-02-10T08:49:13.327257+0000","last_fullsized":"2024-02-10T08:49:13.327257+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.471058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.471058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":171,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326831+0000","last_change":"2024-02-10T08:48:30.433484+0000","last_active":"2024-02-10T08:49:13.326831+0000","last_peered":"2024-02-10T08:49:13.326831+0000","last_clean":"2024-02-10T08:49:13.326831+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:49:13.326831+0000","last_undegraded":"2024-02-10T08:49:13.326831+0000","last_fullsized":"2024-02-10T08:49:13.326831+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.433421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.433421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":811,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485547+0000","last_change":"2024-02-10T08:47:24.808523+0000","last_active":"2024-02-10T08:49:13.485547+0000","last_peered":"2024-02-10T08:49:13.485547+0000","last_clean":"2024-02-10T08:49:13.485547+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:49:13.485547+0000","last_undegraded":"2024-02-10T08:49:13.485547+0000","last_fullsized":"2024-02-10T08:49:13.485547+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:24.808424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.808424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":527,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126621+0000","last_change":"2024-02-10T08:47:55.031372+0000","last_active":"2024-02-10T08:49:15.126621+0000","last_peered":"2024-02-10T08:49:15.126621+0000","last_clean":"2024-02-10T08:49:15.126621+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:49:15.126621+0000","last_undegraded":"2024-02-10T08:49:15.126621+0000","last_fullsized":"2024-02-10T08:49:15.126621+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.031301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.031301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":973,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127421+0000","last_change":"2024-02-10T08:46:08.457332+0000","last_active":"2024-02-10T08:49:15.127421+0000","last_peered":"2024-02-10T08:49:15.127421+0000","last_clean":"2024-02-10T08:49:15.127421+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:49:15.127421+0000","last_undegraded":"2024-02-10T08:49:15.127421+0000","last_fullsized":"2024-02-10T08:49:15.127421+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:46:08.457270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:08.457270+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":589,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126110+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:49:15.126110+0000","last_peered":"2024-02-10T08:49:15.126110+0000","last_clean":"2024-02-10T08:49:15.126110+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:49:15.126110+0000","last_undegraded":"2024-02-10T08:49:15.126110+0000","last_fullsized":"2024-02-10T08:49:15.126110+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":745,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280967+0000","last_change":"2024-02-10T08:47:49.268597+0000","last_active":"2024-02-10T08:49:14.280967+0000","last_peered":"2024-02-10T08:49:14.280967+0000","last_clean":"2024-02-10T08:49:14.280967+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:49:14.280967+0000","last_undegraded":"2024-02-10T08:49:14.280967+0000","last_fullsized":"2024-02-10T08:49:14.280967+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:47:19.053931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.053931+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":94,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282568+0000","last_change":"2024-02-10T08:47:49.932955+0000","last_active":"2024-02-10T08:49:15.282568+0000","last_peered":"2024-02-10T08:49:15.282568+0000","last_clean":"2024-02-10T08:49:15.282568+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:49:15.282568+0000","last_undegraded":"2024-02-10T08:49:15.282568+0000","last_fullsized":"2024-02-10T08:49:15.282568+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.671541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":764,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279761+0000","last_change":"2024-02-10T08:48:45.997431+0000","last_active":"2024-02-10T08:49:14.279761+0000","last_peered":"2024-02-10T08:49:14.279761+0000","last_clean":"2024-02-10T08:49:14.279761+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:49:14.279761+0000","last_undegraded":"2024-02-10T08:49:14.279761+0000","last_fullsized":"2024-02-10T08:49:14.279761+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.997364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.997364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":788,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126897+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:49:15.126897+0000","last_peered":"2024-02-10T08:49:15.126897+0000","last_clean":"2024-02-10T08:49:15.126897+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:49:15.126897+0000","last_undegraded":"2024-02-10T08:49:15.126897+0000","last_fullsized":"2024-02-10T08:49:15.126897+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":184,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326681+0000","last_change":"2024-02-10T08:47:49.241459+0000","last_active":"2024-02-10T08:49:13.326681+0000","last_peered":"2024-02-10T08:49:13.326681+0000","last_clean":"2024-02-10T08:49:13.326681+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:49:13.326681+0000","last_undegraded":"2024-02-10T08:49:13.326681+0000","last_fullsized":"2024-02-10T08:49:13.326681+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.257566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.257566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":722,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280214+0000","last_change":"2024-02-10T08:47:25.754309+0000","last_active":"2024-02-10T08:49:14.280214+0000","last_peered":"2024-02-10T08:49:14.280214+0000","last_clean":"2024-02-10T08:49:14.280214+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:49:14.280214+0000","last_undegraded":"2024-02-10T08:49:14.280214+0000","last_fullsized":"2024-02-10T08:49:14.280214+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.754229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.754229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":680,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126696+0000","last_change":"2024-02-10T08:47:49.793777+0000","last_active":"2024-02-10T08:49:15.126696+0000","last_peered":"2024-02-10T08:49:15.126696+0000","last_clean":"2024-02-10T08:49:15.126696+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:49:15.126696+0000","last_undegraded":"2024-02-10T08:49:15.126696+0000","last_fullsized":"2024-02-10T08:49:15.126696+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:19.946402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.946402+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":89,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283300+0000","last_change":"2024-02-10T08:47:56.921391+0000","last_active":"2024-02-10T08:49:15.283300+0000","last_peered":"2024-02-10T08:49:15.283300+0000","last_clean":"2024-02-10T08:49:15.283300+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:49:15.283300+0000","last_undegraded":"2024-02-10T08:49:15.283300+0000","last_fullsized":"2024-02-10T08:49:15.283300+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.921291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.921291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":89,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.282615+0000","last_change":"2024-02-10T08:47:58.934691+0000","last_active":"2024-02-10T08:49:15.282615+0000","last_peered":"2024-02-10T08:49:15.282615+0000","last_clean":"2024-02-10T08:49:15.282615+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:49:15.282615+0000","last_undegraded":"2024-02-10T08:49:15.282615+0000","last_fullsized":"2024-02-10T08:49:15.282615+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:58.934594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.934594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":768,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.124978+0000","last_change":"2024-02-10T08:48:40.049047+0000","last_active":"2024-02-10T08:49:15.124978+0000","last_peered":"2024-02-10T08:49:15.124978+0000","last_clean":"2024-02-10T08:49:15.124978+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:49:15.124978+0000","last_undegraded":"2024-02-10T08:49:15.124978+0000","last_fullsized":"2024-02-10T08:49:15.124978+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.048948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.048948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":719,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126243+0000","last_change":"2024-02-10T08:46:41.501224+0000","last_active":"2024-02-10T08:49:15.126243+0000","last_peered":"2024-02-10T08:49:15.126243+0000","last_clean":"2024-02-10T08:49:15.126243+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:49:15.126243+0000","last_undegraded":"2024-02-10T08:49:15.126243+0000","last_fullsized":"2024-02-10T08:49:15.126243+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.501157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.501157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":791,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.124659+0000","last_change":"2024-02-10T08:47:49.262275+0000","last_active":"2024-02-10T08:49:15.124659+0000","last_peered":"2024-02-10T08:49:15.124659+0000","last_clean":"2024-02-10T08:49:15.124659+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:49:15.124659+0000","last_undegraded":"2024-02-10T08:49:15.124659+0000","last_fullsized":"2024-02-10T08:49:15.124659+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:46:48.511488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.511488+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":759,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.484835+0000","last_change":"2024-02-10T08:48:36.153670+0000","last_active":"2024-02-10T08:49:13.484835+0000","last_peered":"2024-02-10T08:49:13.484835+0000","last_clean":"2024-02-10T08:49:13.484835+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:49:13.484835+0000","last_undegraded":"2024-02-10T08:49:13.484835+0000","last_fullsized":"2024-02-10T08:49:13.484835+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.153585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.153585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":785,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279207+0000","last_change":"2024-02-10T08:48:32.816285+0000","last_active":"2024-02-10T08:49:14.279207+0000","last_peered":"2024-02-10T08:49:14.279207+0000","last_clean":"2024-02-10T08:49:14.279207+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:49:14.279207+0000","last_undegraded":"2024-02-10T08:49:14.279207+0000","last_fullsized":"2024-02-10T08:49:14.279207+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.816211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.816211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":344,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485639+0000","last_change":"2024-02-10T08:47:56.600567+0000","last_active":"2024-02-10T08:49:13.485639+0000","last_peered":"2024-02-10T08:49:13.485639+0000","last_clean":"2024-02-10T08:49:13.485639+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:49:13.485639+0000","last_undegraded":"2024-02-10T08:49:13.485639+0000","last_fullsized":"2024-02-10T08:49:13.485639+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.600507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.600507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":731,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127228+0000","last_change":"2024-02-10T08:47:49.232286+0000","last_active":"2024-02-10T08:49:15.127228+0000","last_peered":"2024-02-10T08:49:15.127228+0000","last_clean":"2024-02-10T08:49:15.127228+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:49:15.127228+0000","last_undegraded":"2024-02-10T08:49:15.127228+0000","last_fullsized":"2024-02-10T08:49:15.127228+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.529230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.529230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":822,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125215+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:49:15.125215+0000","last_peered":"2024-02-10T08:49:15.125215+0000","last_clean":"2024-02-10T08:49:15.125215+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:49:15.125215+0000","last_undegraded":"2024-02-10T08:49:15.125215+0000","last_fullsized":"2024-02-10T08:49:15.125215+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":764,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280110+0000","last_change":"2024-02-10T08:48:53.059434+0000","last_active":"2024-02-10T08:49:14.280110+0000","last_peered":"2024-02-10T08:49:14.280110+0000","last_clean":"2024-02-10T08:49:14.280110+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:49:14.280110+0000","last_undegraded":"2024-02-10T08:49:14.280110+0000","last_fullsized":"2024-02-10T08:49:14.280110+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:53.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:53.059137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":344,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485724+0000","last_change":"2024-02-10T08:48:45.871861+0000","last_active":"2024-02-10T08:49:13.485724+0000","last_peered":"2024-02-10T08:49:13.485724+0000","last_clean":"2024-02-10T08:49:13.485724+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:49:13.485724+0000","last_undegraded":"2024-02-10T08:49:13.485724+0000","last_fullsized":"2024-02-10T08:49:13.485724+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.871773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":763,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127039+0000","last_change":"2024-02-10T08:46:15.348134+0000","last_active":"2024-02-10T08:49:15.127039+0000","last_peered":"2024-02-10T08:49:15.127039+0000","last_clean":"2024-02-10T08:49:15.127039+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:49:15.127039+0000","last_undegraded":"2024-02-10T08:49:15.127039+0000","last_fullsized":"2024-02-10T08:49:15.127039+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:05.554604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:05.554604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":166,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327117+0000","last_change":"2024-02-10T08:47:29.348776+0000","last_active":"2024-02-10T08:49:13.327117+0000","last_peered":"2024-02-10T08:49:13.327117+0000","last_clean":"2024-02-10T08:49:13.327117+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:49:13.327117+0000","last_undegraded":"2024-02-10T08:49:13.327117+0000","last_fullsized":"2024-02-10T08:49:13.327117+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.348714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":178,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327154+0000","last_change":"2024-02-10T08:47:58.529546+0000","last_active":"2024-02-10T08:49:13.327154+0000","last_peered":"2024-02-10T08:49:13.327154+0000","last_clean":"2024-02-10T08:49:13.327154+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:49:13.327154+0000","last_undegraded":"2024-02-10T08:49:13.327154+0000","last_fullsized":"2024-02-10T08:49:13.327154+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:47:58.529476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.529476+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":797,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.483904+0000","last_change":"2024-02-10T08:48:54.005915+0000","last_active":"2024-02-10T08:49:13.483904+0000","last_peered":"2024-02-10T08:49:13.483904+0000","last_clean":"2024-02-10T08:49:13.483904+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:49:13.483904+0000","last_undegraded":"2024-02-10T08:49:13.483904+0000","last_fullsized":"2024-02-10T08:49:13.483904+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.005830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.005830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":344,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485008+0000","last_change":"2024-02-10T08:48:50.762747+0000","last_active":"2024-02-10T08:49:13.485008+0000","last_peered":"2024-02-10T08:49:13.485008+0000","last_clean":"2024-02-10T08:49:13.485008+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:49:13.485008+0000","last_undegraded":"2024-02-10T08:49:13.485008+0000","last_fullsized":"2024-02-10T08:49:13.485008+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.762672+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":742,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282707+0000","last_change":"2024-02-10T08:47:49.229890+0000","last_active":"2024-02-10T08:49:14.282707+0000","last_peered":"2024-02-10T08:49:14.282707+0000","last_clean":"2024-02-10T08:49:14.282707+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:49:14.282707+0000","last_undegraded":"2024-02-10T08:49:14.282707+0000","last_fullsized":"2024-02-10T08:49:14.282707+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.149802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.149802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":835,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280311+0000","last_change":"2024-02-10T08:47:57.663357+0000","last_active":"2024-02-10T08:49:14.280311+0000","last_peered":"2024-02-10T08:49:14.280311+0000","last_clean":"2024-02-10T08:49:14.280311+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:49:14.280311+0000","last_undegraded":"2024-02-10T08:49:14.280311+0000","last_fullsized":"2024-02-10T08:49:14.280311+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:57.663290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.663290+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":774,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125078+0000","last_change":"2024-02-10T08:48:41.013610+0000","last_active":"2024-02-10T08:49:15.125078+0000","last_peered":"2024-02-10T08:49:15.125078+0000","last_clean":"2024-02-10T08:49:15.125078+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:49:15.125078+0000","last_undegraded":"2024-02-10T08:49:15.125078+0000","last_fullsized":"2024-02-10T08:49:15.125078+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.013543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.013543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":84,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.283272+0000","last_change":"2024-02-10T08:47:49.249250+0000","last_active":"2024-02-10T08:49:15.283272+0000","last_peered":"2024-02-10T08:49:15.283272+0000","last_clean":"2024-02-10T08:49:15.283272+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:49:15.283272+0000","last_undegraded":"2024-02-10T08:49:15.283272+0000","last_fullsized":"2024-02-10T08:49:15.283272+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:40.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:46:40.685831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":89,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.281179+0000","last_change":"2024-02-10T08:48:49.706321+0000","last_active":"2024-02-10T08:49:15.281179+0000","last_peered":"2024-02-10T08:49:15.281179+0000","last_clean":"2024-02-10T08:49:15.281179+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:49:15.281179+0000","last_undegraded":"2024-02-10T08:49:15.281179+0000","last_fullsized":"2024-02-10T08:49:15.281179+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:49.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:48:49.706258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":83,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.282113+0000","last_change":"2024-02-10T08:48:52.097227+0000","last_active":"2024-02-10T08:49:14.282113+0000","last_peered":"2024-02-10T08:49:14.282113+0000","last_clean":"2024-02-10T08:49:14.282113+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:49:14.282113+0000","last_undegraded":"2024-02-10T08:49:14.282113+0000","last_fullsized":"2024-02-10T08:49:14.282113+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.097161+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":787,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.280115+0000","last_change":"2024-02-10T08:48:41.102099+0000","last_active":"2024-02-10T08:49:14.280115+0000","last_peered":"2024-02-10T08:49:14.280115+0000","last_clean":"2024-02-10T08:49:14.280115+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:49:14.280115+0000","last_undegraded":"2024-02-10T08:49:14.280115+0000","last_fullsized":"2024-02-10T08:49:14.280115+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.102021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.102021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":595,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.127609+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:49:15.127609+0000","last_peered":"2024-02-10T08:49:15.127609+0000","last_clean":"2024-02-10T08:49:15.127609+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:49:15.127609+0000","last_undegraded":"2024-02-10T08:49:15.127609+0000","last_fullsized":"2024-02-10T08:49:15.127609+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":184,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326715+0000","last_change":"2024-02-10T08:47:49.241622+0000","last_active":"2024-02-10T08:49:13.326715+0000","last_peered":"2024-02-10T08:49:13.326715+0000","last_clean":"2024-02-10T08:49:13.326715+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:49:13.326715+0000","last_undegraded":"2024-02-10T08:49:13.326715+0000","last_fullsized":"2024-02-10T08:49:13.326715+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:27.336914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.336914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":188,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125914+0000","last_change":"2024-02-10T08:47:49.265858+0000","last_active":"2024-02-10T08:49:15.125914+0000","last_peered":"2024-02-10T08:49:15.125914+0000","last_clean":"2024-02-10T08:49:15.125914+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:49:15.125914+0000","last_undegraded":"2024-02-10T08:49:15.125914+0000","last_fullsized":"2024-02-10T08:49:15.125914+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:46:43.848897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.848897+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":774,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126892+0000","last_change":"2024-02-10T08:48:00.922194+0000","last_active":"2024-02-10T08:49:15.126892+0000","last_peered":"2024-02-10T08:49:15.126892+0000","last_clean":"2024-02-10T08:49:15.126892+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:49:15.126892+0000","last_undegraded":"2024-02-10T08:49:15.126892+0000","last_fullsized":"2024-02-10T08:49:15.126892+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:00.922104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:00.922104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":750,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126510+0000","last_change":"2024-02-10T08:48:43.032173+0000","last_active":"2024-02-10T08:49:15.126510+0000","last_peered":"2024-02-10T08:49:15.126510+0000","last_clean":"2024-02-10T08:49:15.126510+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:49:15.126510+0000","last_undegraded":"2024-02-10T08:49:15.126510+0000","last_fullsized":"2024-02-10T08:49:15.126510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.032094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.032094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":93,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.281802+0000","last_change":"2024-02-10T08:48:10.214294+0000","last_active":"2024-02-10T08:49:15.281802+0000","last_peered":"2024-02-10T08:49:15.281802+0000","last_clean":"2024-02-10T08:49:15.281802+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:49:15.281802+0000","last_undegraded":"2024-02-10T08:49:15.281802+0000","last_fullsized":"2024-02-10T08:49:15.281802+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:14.568443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:47:14.568443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":652,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126052+0000","last_change":"2024-02-10T08:48:32.990056+0000","last_active":"2024-02-10T08:49:15.126052+0000","last_peered":"2024-02-10T08:49:15.126052+0000","last_clean":"2024-02-10T08:49:15.126052+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:49:15.126052+0000","last_undegraded":"2024-02-10T08:49:15.126052+0000","last_fullsized":"2024-02-10T08:49:15.126052+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.989985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.989985+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":341,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485098+0000","last_change":"2024-02-10T08:48:55.711815+0000","last_active":"2024-02-10T08:49:13.485098+0000","last_peered":"2024-02-10T08:49:13.485098+0000","last_clean":"2024-02-10T08:49:13.485098+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:49:13.485098+0000","last_undegraded":"2024-02-10T08:49:13.485098+0000","last_fullsized":"2024-02-10T08:49:13.485098+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.711733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.711733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":184,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.326762+0000","last_change":"2024-02-10T08:47:49.241831+0000","last_active":"2024-02-10T08:49:13.326762+0000","last_peered":"2024-02-10T08:49:13.326762+0000","last_clean":"2024-02-10T08:49:13.326762+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:49:13.326762+0000","last_undegraded":"2024-02-10T08:49:13.326762+0000","last_fullsized":"2024-02-10T08:49:13.326762+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:24.365335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.365335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":736,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.281711+0000","last_change":"2024-02-10T08:46:37.473469+0000","last_active":"2024-02-10T08:49:14.281711+0000","last_peered":"2024-02-10T08:49:14.281711+0000","last_clean":"2024-02-10T08:49:14.281711+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:49:14.281711+0000","last_undegraded":"2024-02-10T08:49:14.281711+0000","last_fullsized":"2024-02-10T08:49:14.281711+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:37.473394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:37.473394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":655,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126863+0000","last_change":"2024-02-10T08:48:44.939276+0000","last_active":"2024-02-10T08:49:15.126863+0000","last_peered":"2024-02-10T08:49:15.126863+0000","last_clean":"2024-02-10T08:49:15.126863+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:49:15.126863+0000","last_undegraded":"2024-02-10T08:49:15.126863+0000","last_fullsized":"2024-02-10T08:49:15.126863+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:48:44.939188+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.939188+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":286,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125993+0000","last_change":"2024-02-10T08:47:57.938158+0000","last_active":"2024-02-10T08:49:15.125993+0000","last_peered":"2024-02-10T08:49:15.125993+0000","last_clean":"2024-02-10T08:49:15.125993+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:49:15.125993+0000","last_undegraded":"2024-02-10T08:49:15.125993+0000","last_fullsized":"2024-02-10T08:49:15.125993+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.938096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.938096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":762,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.483879+0000","last_change":"2024-02-10T08:47:56.907151+0000","last_active":"2024-02-10T08:49:13.483879+0000","last_peered":"2024-02-10T08:49:13.483879+0000","last_clean":"2024-02-10T08:49:13.483879+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:49:13.483879+0000","last_undegraded":"2024-02-10T08:49:13.483879+0000","last_fullsized":"2024-02-10T08:49:13.483879+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.907094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.907094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":170,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327301+0000","last_change":"2024-02-10T08:48:52.471904+0000","last_active":"2024-02-10T08:49:13.327301+0000","last_peered":"2024-02-10T08:49:13.327301+0000","last_clean":"2024-02-10T08:49:13.327301+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:49:13.327301+0000","last_undegraded":"2024-02-10T08:49:13.327301+0000","last_fullsized":"2024-02-10T08:49:13.327301+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:52.471831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.471831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":687,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125550+0000","last_change":"2024-02-10T08:47:50.341317+0000","last_active":"2024-02-10T08:49:15.125550+0000","last_peered":"2024-02-10T08:49:15.125550+0000","last_clean":"2024-02-10T08:49:15.125550+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:49:15.125550+0000","last_undegraded":"2024-02-10T08:49:15.125550+0000","last_fullsized":"2024-02-10T08:49:15.125550+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:21.938274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.938274+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":769,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126019+0000","last_change":"2024-02-10T08:46:43.502472+0000","last_active":"2024-02-10T08:49:15.126019+0000","last_peered":"2024-02-10T08:49:15.126019+0000","last_clean":"2024-02-10T08:49:15.126019+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:49:15.126019+0000","last_undegraded":"2024-02-10T08:49:15.126019+0000","last_fullsized":"2024-02-10T08:49:15.126019+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.502412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.502412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":610,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125890+0000","last_change":"2024-02-10T08:48:47.974449+0000","last_active":"2024-02-10T08:49:15.125890+0000","last_peered":"2024-02-10T08:49:15.125890+0000","last_clean":"2024-02-10T08:49:15.125890+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:49:15.125890+0000","last_undegraded":"2024-02-10T08:49:15.125890+0000","last_fullsized":"2024-02-10T08:49:15.125890+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.974390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.974390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":728,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.485487+0000","last_change":"2024-02-10T08:48:42.043077+0000","last_active":"2024-02-10T08:49:13.485487+0000","last_peered":"2024-02-10T08:49:13.485487+0000","last_clean":"2024-02-10T08:49:13.485487+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:49:13.485487+0000","last_undegraded":"2024-02-10T08:49:13.485487+0000","last_fullsized":"2024-02-10T08:49:13.485487+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.043006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.043006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":842,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.125259+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:49:15.125259+0000","last_peered":"2024-02-10T08:49:15.125259+0000","last_clean":"2024-02-10T08:49:15.125259+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:49:15.125259+0000","last_undegraded":"2024-02-10T08:49:15.125259+0000","last_fullsized":"2024-02-10T08:49:15.125259+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":182,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-02-10T08:49:13.327322+0000","last_change":"2024-02-10T08:48:02.600767+0000","last_active":"2024-02-10T08:49:13.327322+0000","last_peered":"2024-02-10T08:49:13.327322+0000","last_clean":"2024-02-10T08:49:13.327322+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:49:13.327322+0000","last_undegraded":"2024-02-10T08:49:13.327322+0000","last_fullsized":"2024-02-10T08:49:13.327322+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.600704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.600704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":765,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:15.126822+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:49:15.126822+0000","last_peered":"2024-02-10T08:49:15.126822+0000","last_clean":"2024-02-10T08:49:15.126822+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:49:15.126822+0000","last_undegraded":"2024-02-10T08:49:15.126822+0000","last_fullsized":"2024-02-10T08:49:15.126822+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":739,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-02-10T08:49:14.279968+0000","last_change":"2024-02-10T08:48:10.219049+0000","last_active":"2024-02-10T08:49:14.279968+0000","last_peered":"2024-02-10T08:49:14.279968+0000","last_clean":"2024-02-10T08:49:14.279968+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:49:14.279968+0000","last_undegraded":"2024-02-10T08:49:14.279968+0000","last_fullsized":"2024-02-10T08:49:14.279968+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.702592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":252,"num_pg":32,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":4,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":88,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":42,"num_object_clones":1,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":22,"num_read_kb":7,"num_write":62,"num_write_kb":3964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6500352,"data_stored":6500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":10732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":184,"num_read_kb":179,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299586,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311708,"kb_used_data":6292,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433444,"statfs":{"total":95995035648,"available":95675846656,"internally_reserved":0,"allocated":6443008,"data_stored":3428552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54900000000000004,"15min":0.60599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.37,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.76800000000000002,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.70699999999999996}]},{"osd":1,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.32300000000000001,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.73399999999999999,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67700000000000005,"15min":0.61799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.19900000000000001},"max":{"1min":0.80000000000000004,"5min":10.978,"15min":10.978},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.67100000000000004,"15min":0.60299999999999998},"min":{"1min":0.36899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.81599999999999995,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.625,"15min":0.67200000000000004},"min":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":7.5099999999999998,"15min":19.369},"last":0.75900000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.56999999999999995,"15min":0.59299999999999997},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":1.0269999999999999,"5min":7.8700000000000001,"15min":22.102},"last":0.67400000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65100000000000002,"5min":1.581,"15min":1.581},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.68700000000000006,"15min":0.71699999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67300000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.78300000000000003}]},{"osd":5,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.69199999999999995,"5min":4.46,"15min":19.326000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.58099999999999996,"15min":0.52200000000000002},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.57799999999999996},"min":{"1min":0.36299999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63400000000000001,"5min":4.718,"15min":22.687000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.503},"min":{"1min":0.34399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.621,"5min":5.484,"15min":5.484},"last":0.59999999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561224,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305520,"kb_used_data":8296,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439632,"statfs":{"total":95995035648,"available":95682183168,"internally_reserved":0,"allocated":8495104,"data_stored":5481905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.55600000000000005},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61499999999999999,"5min":3.9399999999999999,"15min":8.5250000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.751,"15min":0.63300000000000001},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.48799999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.73399999999999999,"15min":0.65300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.66300000000000003,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":0.69699999999999995,"5min":8.4399999999999995,"15min":18.402999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76800000000000002,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0509999999999999,"5min":10.772,"15min":10.772},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.84099999999999997,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.79500000000000004,"15min":0.70499999999999996},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.64300000000000002,"5min":9.2650000000000006,"15min":16.823},"last":0.496},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.439},"min":{"1min":0.26400000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.70999999999999996,"5min":3.915,"15min":3.915},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.371,"5min":0.254,"15min":0.254},"max":{"1min":0.70199999999999996,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.504,"15min":0.51200000000000001},"min":{"1min":0.40799999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.61499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307688,"kb_used_data":6368,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437464,"statfs":{"total":95995035648,"available":95679963136,"internally_reserved":0,"allocated":6520832,"data_stored":3451956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.5669999999999999,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.39500000000000002,"15min":0.46100000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.71799999999999997,"5min":1.4910000000000001,"15min":16.166},"last":0.25}]},{"osd":2,"last update":"Sat Feb 10 08:48:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.50600000000000001,"15min":0.64700000000000002},"min":{"1min":0.251,"5min":0.193,"15min":0.193},"max":{"1min":0.51500000000000001,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.46600000000000003,"15min":0.55500000000000005},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":0.54900000000000004,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.27300000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.47799999999999998,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":0.51100000000000001,"5min":4.8940000000000001,"15min":24.773},"last":0.311},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.50900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.80000000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82899999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83799999999999997,"5min":3.496,"15min":9.5600000000000005},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.622},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.82299999999999995,"5min":3.4710000000000001,"15min":9.532},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.874,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:48:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72499999999999998,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":12.147,"15min":12.147},"last":0.502},{"interface":"front","average":{"1min":0.62,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85499999999999998,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.68700000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738597,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303536,"kb_used_data":6312,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441616,"statfs":{"total":95995035648,"available":95684214784,"internally_reserved":0,"allocated":6463488,"data_stored":3452467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.46899999999999997,"15min":0.40300000000000002},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.156},"max":{"1min":0.59299999999999997,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.69699999999999995,"15min":0.48599999999999999},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.65000000000000002,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.53700000000000003,"15min":0.45700000000000002},"min":{"1min":0.252,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.496,"5min":11.987,"15min":11.987},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.501,"15min":0.48199999999999998},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.436}]},{"osd":3,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.502,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.379},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.51900000000000002,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:48:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.38,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.71799999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.931,"5min":4.609,"15min":4.609},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55900000000000005,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":1.7769999999999999,"15min":3.4079999999999999},"last":0.72599999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54100000000000004,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79900000000000004,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.752},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58199999999999996,"15min":0.621},"min":{"1min":0.39200000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Feb 10 08:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.317,"15min":0.317},"max":{"1min":0.76600000000000001,"5min":1.849,"15min":3.363},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55400000000000005,"15min":0.65600000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73799999999999999,"5min":0.92600000000000005,"15min":8.4299999999999997},"last":0.746}]}]},{"osd":2,"up_from":18,"seq":77309411547,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305476,"kb_used_data":8252,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439676,"statfs":{"total":95995035648,"available":95682228224,"internally_reserved":0,"allocated":8450048,"data_stored":5443288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.64600000000000002,"15min":0.499},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.64300000000000002,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.315,"5min":0.70399999999999996,"15min":0.56399999999999995},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":0.504,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.34599999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.63700000000000001,"15min":0.55700000000000005},"min":{"1min":0.253,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":9.6980000000000004,"15min":12.512},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.60399999999999998,"15min":0.53100000000000003},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.56599999999999995,"5min":12.875,"15min":12.875},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.61199999999999999,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50800000000000001,"5min":10.994999999999999,"15min":11.612},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":7.8280000000000003,"15min":11.710000000000001},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.42099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.504}]},{"osd":5,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.41099999999999998,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.79100000000000004,"5min":10.34,"15min":10.34},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.71199999999999997,"15min":0.66500000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3380000000000001,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.751,"15min":0.71499999999999997},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84299999999999997,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.80700000000000005,"15min":0.71099999999999997},"min":{"1min":0.41199999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.321,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.90500000000000003,"5min":10.539,"15min":10.539},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.78300000000000003,"15min":0.71099999999999997},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.357,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.63}]}]},{"osd":3,"up_from":25,"seq":107374182615,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303600,"kb_used_data":6376,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441552,"statfs":{"total":95995035648,"available":95684149248,"internally_reserved":0,"allocated":6529024,"data_stored":3491201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.20699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.497,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.25},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.251,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.58599999999999997,"5min":10.484999999999999,"15min":11.808},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.76900000000000002,"15min":0.67200000000000004},"min":{"1min":0.248,"5min":0.215,"15min":0.184},"max":{"1min":0.53800000000000003,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.312},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.26000000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.58799999999999997,"5min":17.343,"15min":17.343},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.754,"15min":0.68000000000000005},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":0.57299999999999995,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.443,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57199999999999995,"5min":5.0259999999999998,"15min":23.294},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:48:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.499},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.78400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55700000000000005,"15min":0.70199999999999996},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.73999999999999999,"15min":0.70299999999999996},"min":{"1min":0.39000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82999999999999996,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.54000000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58299999999999996,"15min":0.71499999999999997},"min":{"1min":0.42699999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67100000000000004,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.73399999999999999,"15min":0.749},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80400000000000005,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.746},"min":{"1min":0.45300000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79700000000000004,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":633,"seq":2718714298403,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309736,"kb_used_data":6368,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93435416,"statfs":{"total":95995035648,"available":95677865984,"internally_reserved":0,"allocated":6520832,"data_stored":3487951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.66400000000000003,"5min":0.754,"15min":0.754},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.497},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72799999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.505},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70899999999999996,"5min":0.746,"15min":0.746},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73899999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.41399999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68700000000000006,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.86099999999999999,"5min":0.87,"15min":0.87},"last":0.437}]},{"osd":6,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.42499999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70999999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.80400000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52100000000000002}]}]},{"osd":5,"up_from":37,"seq":158913790157,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313696,"kb_used_data":8280,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431456,"statfs":{"total":95995035648,"available":95673810944,"internally_reserved":0,"allocated":8478720,"data_stored":5475859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.69599999999999995,"15min":1.0489999999999999},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.216},"max":{"1min":0.73599999999999999,"5min":8.7400000000000002,"15min":107.551},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56999999999999995,"15min":0.97499999999999998},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.22900000000000001},"max":{"1min":0.64200000000000002,"5min":5.375,"15min":88.792000000000002},"last":0.41499999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.76800000000000002,"15min":1.006},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.216},"max":{"1min":0.77900000000000003,"5min":12.375999999999999,"15min":107.205},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.71499999999999997,"15min":0.88100000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.71299999999999997,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":1.0289999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.71599999999999997,"5min":10.794,"15min":54.174999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.753,"15min":1.0660000000000001},"min":{"1min":0.35699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76100000000000001,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.436}]},{"osd":3,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.745,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.80100000000000005,"15min":1.2869999999999999},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":0.754,"5min":8.7690000000000001,"15min":114.526},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.443,"15min":0.437},"min":{"1min":0.311,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64500000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.36599999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51700000000000002,"15min":0.72499999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":1.847,"5min":10.023999999999999,"15min":72.241},"last":0.379},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.58099999999999996},"min":{"1min":0.34599999999999997,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71199999999999997,"5min":10.013,"15min":22.469999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48099999999999998,"15min":0.72199999999999998},"min":{"1min":0.32900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1200000000000001,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.48599999999999999,"15min":0.754},"min":{"1min":0.313,"5min":0.253,"15min":0.253},"max":{"1min":1.8720000000000001,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2048000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2080768,"data_stored":2080768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:49:21.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.687938+0000 osd.7 (osd.7) 316 : cluster [DBG] 234.7 scrub starts 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:19.690082+0000 osd.7 (osd.7) 317 : cluster [DBG] 234.7 scrub ok 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.027756+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.e scrub starts 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.031221+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.e scrub ok 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.060385+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.e scrub starts 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.063273+0000 osd.2 (osd.2) 292 : cluster [DBG] 16.e scrub ok 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:20.098492+0000 mgr.y (mgr.14553) 649 : audit [DBG] from='client.28785 v1:172.21.15.107:0/218921361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.322202+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.324120+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.565559+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.566493+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:49:21.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.649683+0000 osd.5 (osd.5) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:49:21.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:20.649893+0000 osd.5 (osd.5) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:49:21.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.193889+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-10T08:49:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:22.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:22.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.359960+0000 osd.3 (osd.3) 339 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.361083+0000 osd.3 (osd.3) 340 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.572124+0000 mgr.y (mgr.14553) 650 : cluster [DBG] pgmap v1195: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.581760+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.583338+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:49:22.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:21.838497+0000 mgr.y (mgr.14553) 651 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.359960+0000 osd.3 (osd.3) 339 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.361083+0000 osd.3 (osd.3) 340 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.572124+0000 mgr.y (mgr.14553) 650 : cluster [DBG] pgmap v1195: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.581760+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.583338+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:49:22.616 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:21.838497+0000 mgr.y (mgr.14553) 651 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.359960+0000 osd.3 (osd.3) 339 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.361083+0000 osd.3 (osd.3) 340 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.572124+0000 mgr.y (mgr.14553) 650 : cluster [DBG] pgmap v1195: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.581760+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.583338+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:49:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:21.838497+0000 mgr.y (mgr.14553) 651 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.883501+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:49:23.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:21.886398+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.352568+0000 osd.3 (osd.3) 341 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.354101+0000 osd.3 (osd.3) 342 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.883501+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:21.886398+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.352568+0000 osd.3 (osd.3) 341 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:49:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.354101+0000 osd.3 (osd.3) 342 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:49:23.782 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:23.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.883501+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:49:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:21.886398+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:49:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.352568+0000 osd.3 (osd.3) 341 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:49:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.354101+0000 osd.3 (osd.3) 342 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:49:23.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:23.918 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:24.016 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:24.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:24.024 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:49:24.024 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:49:24.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:24.148 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:24.297 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:24.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:24.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.771160+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:49:24.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.774194+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.839453+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:22.842357+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.105054+0000 osd.2 (osd.2) 293 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.108057+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.177869+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.180809+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.339595+0000 osd.3 (osd.3) 343 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.340692+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:49:24.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.572681+0000 mgr.y (mgr.14553) 652 : cluster [DBG] pgmap v1196: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:24.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:24.433 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:24.455 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:24.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:24.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.771160+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.774194+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.839453+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:22.842357+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.105054+0000 osd.2 (osd.2) 293 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:49:24.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.108057+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.177869+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.180809+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.339595+0000 osd.3 (osd.3) 343 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.340692+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.572681+0000 mgr.y (mgr.14553) 652 : cluster [DBG] pgmap v1196: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:24.661 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:24.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 -- 172.21.15.107:0/2085552980 >> v1:172.21.15.107:6789/0 conn(0x7fa540103250 legacy=0x7fa5401056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 -- 172.21.15.107:0/2085552980 shutdown_connections 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 -- 172.21.15.107:0/2085552980 >> 172.21.15.107:0/2085552980 conn(0x7fa5400f2510 msgr2=0x7fa5400f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 -- 172.21.15.107:0/2085552980 shutdown_connections 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 -- 172.21.15.107:0/2085552980 wait complete. 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 librados: starting msgr at 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.751+0000 7fa544f00700 1 librados: starting objecter 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 Processor -- start 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 -- start start 2024-02-10T08:49:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 librados: setting wanted keys 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 librados: calling monclient init 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5400fbf80 con 0x7fa5400fc910 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5400fc150 con 0x7fa540103250 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa544f00700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5400fc340 con 0x7fa5400ff710 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa53dd9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa5400ff710 0x7fa540114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36912/0 (socket says 172.21.15.107:36912) 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa53dd9b700 1 -- 172.21.15.107:0/1925221337 learned_addr learned my addr 172.21.15.107:0/1925221337 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter ms_handle_connect 0x7fa5400fc910 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter ms_handle_connect 0x7fa5400ff710 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697982405 0 0) 0x7fa5400fbf80 con 0x7fa5400fc910 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa528003800 con 0x7fa5400fc910 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814804666 0 0) 0x7fa530001230 con 0x7fa5400ff710 2024-02-10T08:49:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5400fbf80 con 0x7fa5400ff710 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter ms_handle_connect 0x7fa540103250 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594323120 0 0) 0x7fa528003800 con 0x7fa5400fc910 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa530001230 con 0x7fa5400fc910 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa5340018c0 con 0x7fa5400fc910 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285943275 0 0) 0x7fa5400fc150 con 0x7fa540103250 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa528003800 con 0x7fa540103250 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 815618637 0 0) 0x7fa5400fbf80 con 0x7fa5400ff710 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5400fc150 con 0x7fa5400ff710 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.752+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa5300018d0 con 0x7fa5400ff710 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2277575447 0 0) 0x7fa530001230 con 0x7fa5400fc910 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6790/0 conn(0x7fa5400ff710 legacy=0x7fa540114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.195:6789/0 conn(0x7fa540103250 legacy=0x7fa5401183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54011cb90 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa540119b60 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa544f00700 10 client.28791.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa540119fd0 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa544f00700 1 librados: init done 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa534001f80 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.753+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa534003310 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa520004f00 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa5340026c0 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(785..785 src has 420..785) v4 ==== 8618+0+0 (unknown 1001657902 0 0) 0x7fa534076760 con 0x7fa5400fc910 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 osd_map(785..785 src has 420..785) v4 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa52f7fe700 3 client.28791.objecter handle_osd_map got epochs [785,785] > 0 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.754+0000 7fa52f7fe700 3 client.28791.objecter handle_osd_map decoding full epoch 785 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa52f7fe700 20 client.28791.objecter dump_active .. 0 homeless 2024-02-10T08:49:24.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v785) v1 ==== 54+0+16 (unknown 2201085864 0 2420000192) 0x7fa534078b70 con 0x7fa5400fc910 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 10 client.28791.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 1 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 1 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 0 1 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 4 2024-02-10T08:49:24.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 10 client.28791.objecter _send_command 1 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520004950 con 0x7fa520004530 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa528063cc0 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.755+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa520004530 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.756+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520004950 con 0x7fa520004530 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.756+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.756+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 1 = system:0 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.756+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 4 2024-02-10T08:49:24.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.756+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 0 1 2024-02-10T08:49:24.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 10 client.28791.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 2 2024-02-10T08:49:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 2 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa5200051a0 osd=0 3 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 0 2 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 4 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 10 client.28791.objecter _send_command 2 2024-02-10T08:49:24.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.773+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa520004d30 con 0x7fa520004530 2024-02-10T08:49:24.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa520004d30 con 0x7fa520004530 2024-02-10T08:49:24.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 4 2024-02-10T08:49:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 0 2 2024-02-10T08:49:24.777 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:49:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 10 client.28791.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 1 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 4 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 10 client.28791.objecter _send_command 3 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.775+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa52000c000 con 0x7fa520004150 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa520004150 2024-02-10T08:49:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa52000c000 con 0x7fa520004150 2024-02-10T08:49:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:49:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 3 = system:0 2024-02-10T08:49:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 4 2024-02-10T08:49:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.776+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 1 3 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.771160+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.774194+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.839453+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:22.842357+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.105054+0000 osd.2 (osd.2) 293 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.108057+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.177869+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:49:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.180809+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:49:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.339595+0000 osd.3 (osd.3) 343 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:49:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.340692+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:49:24.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.572681+0000 mgr.y (mgr.14553) 652 : cluster [DBG] pgmap v1196: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:24.795 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 10 client.28791.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 4 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 4 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520003fe0 osd=1 3 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 1 4 2024-02-10T08:49:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 4 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 10 client.28791.objecter _send_command 4 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.793+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa520001850 con 0x7fa520004150 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa520001850 con 0x7fa520004150 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 4 2024-02-10T08:49:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 1 4 2024-02-10T08:49:24.798 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:49:24.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 10 client.28791.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 5 2024-02-10T08:49:24.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 5 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 2 5 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 4 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 10 client.28791.objecter _send_command 5 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.795+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520003e70 con 0x7fa52000d2f0 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa52000d2f0 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520003e70 con 0x7fa52000d2f0 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 5 = system:0 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 4 2024-02-10T08:49:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.796+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 2 5 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 10 client.28791.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 6 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 6 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520003630 osd=2 3 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 2 6 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 4 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 10 client.28791.objecter _send_command 6 2024-02-10T08:49:24.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.811+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa52000e6e0 con 0x7fa52000d2f0 2024-02-10T08:49:24.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa52000e6e0 con 0x7fa52000d2f0 2024-02-10T08:49:24.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 4 2024-02-10T08:49:24.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 2 6 2024-02-10T08:49:24.815 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:49:24.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 10 client.28791.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 7 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 7 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 3 7 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 4 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 10 client.28791.objecter _send_command 7 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520015b00 con 0x7fa520011e80 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.813+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa520011e80 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.814+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520015b00 con 0x7fa520011e80 2024-02-10T08:49:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.814+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:49:24.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.814+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 7 = system:0 2024-02-10T08:49:24.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.814+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 4 2024-02-10T08:49:24.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.814+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 3 7 2024-02-10T08:49:24.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 10 client.28791.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 8 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 8 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa5200033e0 osd=3 3 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 3 8 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 4 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 10 client.28791.objecter _send_command 8 2024-02-10T08:49:24.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.828+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa52000e150 con 0x7fa520011e80 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa52000e150 con 0x7fa520011e80 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 4 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 3 8 2024-02-10T08:49:24.831 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:49:24.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 10 client.28791.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 9 2024-02-10T08:49:24.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 9 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 4 9 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 4 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 10 client.28791.objecter _send_command 9 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.830+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa52000e0b0 con 0x7fa52000ce50 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa52000ce50 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa52000e0b0 con 0x7fa52000ce50 2024-02-10T08:49:24.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:49:24.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 9 = system:0 2024-02-10T08:49:24.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 4 2024-02-10T08:49:24.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.831+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 4 9 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 10 client.28791.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 10 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 10 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa52000df10 osd=4 3 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 4 10 2024-02-10T08:49:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 4 2024-02-10T08:49:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 10 client.28791.objecter _send_command 10 2024-02-10T08:49:24.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.846+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa52000e0b0 con 0x7fa52000ce50 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa52000e0b0 con 0x7fa52000ce50 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 4 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 4 10 2024-02-10T08:49:24.849 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:49:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 10 client.28791.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 11 2024-02-10T08:49:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 11 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 5 11 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 4 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 10 client.28791.objecter _send_command 11 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520018030 con 0x7fa520017a90 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.848+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa520017a90 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.849+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520018030 con 0x7fa520017a90 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.849+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.849+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 11 = system:0 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.849+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 4 2024-02-10T08:49:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.849+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 5 11 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 10 client.28791.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 12 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 12 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa52000e0b0 osd=5 3 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 5 12 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 4 2024-02-10T08:49:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 10 client.28791.objecter _send_command 12 2024-02-10T08:49:24.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.863+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa520018030 con 0x7fa520017a90 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.865+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa520018030 con 0x7fa520017a90 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.865+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.865+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.865+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 4 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.865+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 5 12 2024-02-10T08:49:24.867 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 10 client.28791.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 13 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 13 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 6 13 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 4 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 10 client.28791.objecter _send_command 13 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520022c60 con 0x7fa52001f0e0 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.866+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa52001f0e0 2024-02-10T08:49:24.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.867+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520022c60 con 0x7fa52001f0e0 2024-02-10T08:49:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.867+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:49:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.867+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 13 = system:0 2024-02-10T08:49:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.867+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 4 2024-02-10T08:49:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.867+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 6 13 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 10 client.28791.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 14 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 14 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520018030 osd=6 3 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 6 14 2024-02-10T08:49:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 4 2024-02-10T08:49:24.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 10 client.28791.objecter _send_command 14 2024-02-10T08:49:24.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.881+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa520024410 con 0x7fa52001f0e0 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.883+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa520024410 con 0x7fa52001f0e0 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.883+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.883+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.883+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 4 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.883+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 6 14 2024-02-10T08:49:24.885 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 10 client.28791.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 15 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 15 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 7 15 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 4 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 10 client.28791.objecter _send_command 15 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520023d20 con 0x7fa520024960 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.884+0000 7fa52f7fe700 10 client.28791.objecter ms_handle_connect 0x7fa520024960 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.885+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa520023d20 con 0x7fa520024960 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.885+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.885+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 15 = system:0 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.885+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 4 2024-02-10T08:49:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.885+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 7 15 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 10 client.28791.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign -1 16 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 20 client.28791.objecter _get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 15 client.28791.objecter _session_command_op_remove -1 16 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 20 client.28791.objecter get_session s=0x7fa520024270 osd=7 3 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 15 client.28791.objecter _session_command_op_assign 7 16 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 4 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 10 client.28791.objecter _send_command 16 2024-02-10T08:49:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.903+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa520023ce0 con 0x7fa520024960 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.905+0000 7fa52f7fe700 1 -- 172.21.15.107:0/1925221337 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa520023ce0 con 0x7fa520024960 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.905+0000 7fa52f7fe700 10 client.28791.objecter ms_dispatch 0x7fa540001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.905+0000 7fa52f7fe700 10 client.28791.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.905+0000 7fa52f7fe700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 4 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.905+0000 7fa52f7fe700 15 client.28791.objecter _session_command_op_remove 7 16 2024-02-10T08:49:24.907 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 librados: watch_flush enter 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 librados: watch_flush exit 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.0 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6801/3321596972 conn(0x7fa520004530 legacy=0x7fa520002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200051a0 osd=0 1 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.1 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6805/133516126 conn(0x7fa520004150 legacy=0x7fa52000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003fe0 osd=1 1 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.2 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6809/3575881728 conn(0x7fa52000d2f0 legacy=0x7fa52000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520003630 osd=2 1 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.3 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6813/2871825691 conn(0x7fa520011e80 legacy=0x7fa520014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa5200033e0 osd=3 1 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.4 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.195:6800/3310789251 conn(0x7fa52000ce50 legacy=0x7fa520015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa53ed9d700 1 -- 172.21.15.107:0/1925221337 reap_dead start 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000df10 osd=4 1 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.5 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.195:6804/4178321486 conn(0x7fa520017a90 legacy=0x7fa52001d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa53ed9d700 1 -- 172.21.15.107:0/1925221337 reap_dead start 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa52000e0b0 osd=5 1 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.6 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.195:6808/1417479017 conn(0x7fa52001f0e0 legacy=0x7fa5200214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa53ed9d700 1 -- 172.21.15.107:0/1925221337 reap_dead start 2024-02-10T08:49:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520018030 osd=6 1 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter close_session for osd.7 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.195:6812/1934176928 conn(0x7fa520024960 legacy=0x7fa520024de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter put_session s=0x7fa520024270 osd=7 1 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 20 client.28791.objecter shutdown clearing up homeless session... 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 10 client.28791.objecter successfully canceled tick 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa528063cc0 legacy=0x7fa528066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> v1:172.21.15.107:6789/0 conn(0x7fa5400fc910 legacy=0x7fa5400fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 shutdown_connections 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 >> 172.21.15.107:0/1925221337 conn(0x7fa5400f2510 msgr2=0x7fa5400612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 shutdown_connections 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 -- 172.21.15.107:0/1925221337 wait complete. 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:24.906+0000 7fa544f00700 1 librados: shutdown 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:49:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:49:25.287 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:49:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:49:24+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-10T08:49:25.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T08:49:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.761432+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:49:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.762542+0000 osd.7 (osd.7) 321 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:49:25.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.852127+0000 osd.6 (osd.6) 322 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:23.855036+0000 osd.6 (osd.6) 323 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.113734+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.116598+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.383062+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.384177+0000 osd.3 (osd.3) 346 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:24.756199+0000 mon.a (mon.0) 3636 : audit [DBG] from='client.? v1:172.21.15.107:0/1925221337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:25.172979+0000 mon.a (mon.0) 3637 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:25.295766+0000 mon.b (mon.1) 852 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:25.296190+0000 mon.a (mon.0) 3638 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.761432+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.762542+0000 osd.7 (osd.7) 321 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.852127+0000 osd.6 (osd.6) 322 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:23.855036+0000 osd.6 (osd.6) 323 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.113734+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.116598+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.383062+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.384177+0000 osd.3 (osd.3) 346 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:24.756199+0000 mon.a (mon.0) 3636 : audit [DBG] from='client.? v1:172.21.15.107:0/1925221337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:25.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:25.172979+0000 mon.a (mon.0) 3637 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:25.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:25.295766+0000 mon.b (mon.1) 852 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:25.296190+0000 mon.a (mon.0) 3638 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.761432+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:49:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.762542+0000 osd.7 (osd.7) 321 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:49:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.852127+0000 osd.6 (osd.6) 322 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:23.855036+0000 osd.6 (osd.6) 323 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.113734+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.116598+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.383062+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.384177+0000 osd.3 (osd.3) 346 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:24.756199+0000 mon.a (mon.0) 3636 : audit [DBG] from='client.? v1:172.21.15.107:0/1925221337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:25.172979+0000 mon.a (mon.0) 3637 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:25.295766+0000 mon.b (mon.1) 852 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:25.296190+0000 mon.a (mon.0) 3638 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:25] "GET /metrics HTTP/1.1" 200 191209 "" "Prometheus/2.33.4" 2024-02-10T08:49:26.244 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:26.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:26.275 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:26.275 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:26.276 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:26.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:26.375 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.869735+0000 osd.1 (osd.1) 342 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:24.873413+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.006507+0000 osd.4 (osd.4) 69 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.009957+0000 osd.4 (osd.4) 70 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.573044+0000 mgr.y (mgr.14553) 653 : cluster [DBG] pgmap v1197: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.634416+0000 osd.0 (osd.0) 172 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:49:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.635391+0000 osd.0 (osd.0) 173 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.174686+0000 mon.a (mon.0) 3639 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.174762+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.177069+0000 mon.b (mon.1) 853 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.177381+0000 mon.a (mon.0) 3641 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.193164+0000 mon.a (mon.0) 3642 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.193228+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.224535+0000 mon.b (mon.1) 854 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.224906+0000 mon.a (mon.0) 3644 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.225366+0000 mon.b (mon.1) 855 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:26.225622+0000 mon.a (mon.0) 3645 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.869735+0000 osd.1 (osd.1) 342 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:24.873413+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.006507+0000 osd.4 (osd.4) 69 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.009957+0000 osd.4 (osd.4) 70 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.573044+0000 mgr.y (mgr.14553) 653 : cluster [DBG] pgmap v1197: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.634416+0000 osd.0 (osd.0) 172 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.635391+0000 osd.0 (osd.0) 173 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.174686+0000 mon.a (mon.0) 3639 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.174762+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.177069+0000 mon.b (mon.1) 853 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.177381+0000 mon.a (mon.0) 3641 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.193164+0000 mon.a (mon.0) 3642 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:26.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.193228+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-10T08:49:26.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.224535+0000 mon.b (mon.1) 854 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.224906+0000 mon.a (mon.0) 3644 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.225366+0000 mon.b (mon.1) 855 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:26.225622+0000 mon.a (mon.0) 3645 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.869735+0000 osd.1 (osd.1) 342 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:49:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:24.873413+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:49:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.006507+0000 osd.4 (osd.4) 69 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:49:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.009957+0000 osd.4 (osd.4) 70 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.573044+0000 mgr.y (mgr.14553) 653 : cluster [DBG] pgmap v1197: 194 pgs: 194 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.634416+0000 osd.0 (osd.0) 172 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.635391+0000 osd.0 (osd.0) 173 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.174686+0000 mon.a (mon.0) 3639 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.174762+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.177069+0000 mon.b (mon.1) 853 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.177381+0000 mon.a (mon.0) 3641 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.193164+0000 mon.a (mon.0) 3642 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]': finished 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.193228+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.224535+0000 mon.b (mon.1) 854 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.224906+0000 mon.a (mon.0) 3644 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.225366+0000 mon.b (mon.1) 855 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:26.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:26.225622+0000 mon.a (mon.0) 3645 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-136"}]: dispatch 2024-02-10T08:49:27.225 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:27.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:27.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:27.361 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:27.383 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:27.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:27.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:27.515 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:27.535 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:27.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:27.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:27.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.887544+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:25.890674+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.108372+0000 osd.2 (osd.2) 297 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.109190+0000 osd.2 (osd.2) 298 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.142001+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.144882+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:49:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:27.200988+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.887544+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:25.890674+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.108372+0000 osd.2 (osd.2) 297 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.109190+0000 osd.2 (osd.2) 298 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.142001+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.144882+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:49:27.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:27.200988+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-10T08:49:27.667 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.887544+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:25.890674+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.108372+0000 osd.2 (osd.2) 297 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.109190+0000 osd.2 (osd.2) 298 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.142001+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:49:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.144882+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:49:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:27.200988+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-10T08:49:28.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T08:49:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:28.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:28.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:28.581 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:28.601 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:28.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:28.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.779894+0000 osd.6 (osd.6) 324 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.780853+0000 osd.6 (osd.6) 325 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.792615+0000 osd.7 (osd.7) 322 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:26.795488+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:27.573704+0000 mgr.y (mgr.14553) 654 : cluster [DBG] pgmap v1201: 162 pgs: 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:28.202504+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-10T08:49:28.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:28.210090+0000 mon.b (mon.1) 856 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.732 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:28.210617+0000 mon.a (mon.0) 3648 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.733 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:28.752 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:28.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:28.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.779894+0000 osd.6 (osd.6) 324 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.780853+0000 osd.6 (osd.6) 325 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.792615+0000 osd.7 (osd.7) 322 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:26.795488+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:27.573704+0000 mgr.y (mgr.14553) 654 : cluster [DBG] pgmap v1201: 162 pgs: 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:28.202504+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:28.210090+0000 mon.b (mon.1) 856 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.872 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:28.210617+0000 mon.a (mon.0) 3648 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.779894+0000 osd.6 (osd.6) 324 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.780853+0000 osd.6 (osd.6) 325 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.792615+0000 osd.7 (osd.7) 322 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:26.795488+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:27.573704+0000 mgr.y (mgr.14553) 654 : cluster [DBG] pgmap v1201: 162 pgs: 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:28.202504+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:28.210090+0000 mon.b (mon.1) 856 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:28.210617+0000 mon.a (mon.0) 3648 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:28.906 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:28.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:29.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:29.038 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:29.057 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:29.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:29.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:29.189 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:29.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:29.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:29.575 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:29.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:29.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:29.722 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:28.105461+0000 osd.2 (osd.2) 299 : cluster [DBG] 16.a scrub starts 2024-02-10T08:49:29.722 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:28.108722+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.a scrub ok 2024-02-10T08:49:29.722 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:28.455637+0000 mon.a (mon.0) 3649 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:29.205879+0000 mon.a (mon.0) 3650 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.205936+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:29.385788+0000 mon.b (mon.1) 857 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:29.386163+0000 mon.a (mon.0) 3652 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:28.105461+0000 osd.2 (osd.2) 299 : cluster [DBG] 16.a scrub starts 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:28.108722+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.a scrub ok 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:28.455637+0000 mon.a (mon.0) 3649 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:29.205879+0000 mon.a (mon.0) 3650 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.205936+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-10T08:49:29.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:29.385788+0000 mon.b (mon.1) 857 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.724 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:29.386163+0000 mon.a (mon.0) 3652 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:28.105461+0000 osd.2 (osd.2) 299 : cluster [DBG] 16.a scrub starts 2024-02-10T08:49:29.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:28.108722+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.a scrub ok 2024-02-10T08:49:29.728 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:28.455637+0000 mon.a (mon.0) 3649 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T08:49:29.729 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:29.205879+0000 mon.a (mon.0) 3650 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:29.729 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.205936+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-10T08:49:29.729 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:29.385788+0000 mon.b (mon.1) 857 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.729 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:29.386163+0000 mon.a (mon.0) 3652 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T08:49:29.731 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:29.750 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:29.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:29.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:29.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:29.881 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:30.005 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:30.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:30.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:30.138 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.317+0000 7f426cb1f700 1 -- 172.21.15.107:0/2163850247 >> v1:172.21.15.107:6789/0 conn(0x7f4268104330 legacy=0x7f42681067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.317+0000 7f426cb1f700 1 -- 172.21.15.107:0/2163850247 shutdown_connections 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.317+0000 7f426cb1f700 1 -- 172.21.15.107:0/2163850247 >> 172.21.15.107:0/2163850247 conn(0x7f42680f3640 msgr2=0x7f42680f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.317+0000 7f426cb1f700 1 -- 172.21.15.107:0/2163850247 shutdown_connections 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.317+0000 7f426cb1f700 1 -- 172.21.15.107:0/2163850247 wait complete. 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 librados: starting msgr at 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 librados: starting objecter 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 Processor -- start 2024-02-10T08:49:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 -- start start 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 librados: setting wanted keys 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 librados: calling monclient init 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42680fcf70 con 0x7f4268104330 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42680fd100 con 0x7f42680fda50 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.318+0000 7f426cb1f700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42680fd290 con 0x7f42681007f0 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f4266d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4268104330 0x7f42681184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48898/0 (socket says 172.21.15.107:48898) 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f4265d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f42681007f0 0x7f4268114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60220/0 (socket says 172.21.15.107:60220) 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f4266d9d700 1 -- 172.21.15.107:0/1437801074 learned_addr learned my addr 172.21.15.107:0/1437801074 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter ms_handle_connect 0x7f4268104330 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:30.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter ms_handle_connect 0x7f42681007f0 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter ms_handle_connect 0x7f42680fda50 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190279490 0 0) 0x7f42680fcf70 con 0x7f4268104330 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c003030 con 0x7f4268104330 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689907893 0 0) 0x7f42680fd290 con 0x7f42681007f0 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42680fcf70 con 0x7f42681007f0 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275732183 0 0) 0x7f42680fd100 con 0x7f42680fda50 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42680fd290 con 0x7f42680fda50 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230001535 0 0) 0x7f424c003030 con 0x7f4268104330 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42680fd100 con 0x7f4268104330 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42600023f0 con 0x7f4268104330 2024-02-10T08:49:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3678732117 0 0) 0x7f42680fcf70 con 0x7f42681007f0 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f424c003030 con 0x7f42681007f0 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f42580018c0 con 0x7f42681007f0 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985416512 0 0) 0x7f42680fd290 con 0x7f42680fda50 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42680fcf70 con 0x7f42680fda50 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4254001000 con 0x7f42680fda50 2024-02-10T08:49:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1099158968 0 0) 0x7f42680fd100 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 >> v1:172.21.15.107:6790/0 conn(0x7f42681007f0 legacy=0x7f4268114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 >> v1:172.21.15.195:6789/0 conn(0x7f42680fda50 legacy=0x7f42680fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426811cc00 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4268119bd0 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f426cb1f700 10 client.28797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f426811a060 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f426cb1f700 1 librados: init done 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.319+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4260001870 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.320+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4260003c60 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.320+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250004f20 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4260003480 con 0x7f4268104330 2024-02-10T08:49:30.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(791..791 src has 420..791) v4 ==== 8592+0+0 (unknown 898867797 0 0) 0x7f4260079590 con 0x7f4268104330 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 10 client.28797.objecter ms_dispatch 0x7f4268001550 osd_map(791..791 src has 420..791) v4 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 3 client.28797.objecter handle_osd_map got epochs [791,791] > 0 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 3 client.28797.objecter handle_osd_map decoding full epoch 791 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 20 client.28797.objecter dump_active .. 0 homeless 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.321+0000 7f425f7fe700 10 client.28797.objecter ms_handle_connect 0x7f424c063d60 2024-02-10T08:49:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.326+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f426004c790 con 0x7f4268104330 2024-02-10T08:49:30.335 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:30.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:30.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:30.491 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:30.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:30.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.496+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4250000b90 con 0x7f424c063d60 2024-02-10T08:49:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.510+0000 7f425f7fe700 1 -- 172.21.15.107:0/1437801074 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+501258 (unknown 2965378022 0 2332520109) 0x7f4250000b90 con 0x7f424c063d60 2024-02-10T08:49:30.518 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:49:30.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 10 librados: watch_flush enter 2024-02-10T08:49:30.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 10 librados: watch_flush exit 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 20 client.28797.objecter shutdown clearing up homeless session... 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 10 client.28797.objecter successfully canceled tick 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 >> v1:172.21.15.107:6800/2634842009 conn(0x7f424c063d60 legacy=0x7f424c066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.514+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 >> v1:172.21.15.107:6789/0 conn(0x7f4268104330 legacy=0x7f42681184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.515+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 shutdown_connections 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.515+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 >> 172.21.15.107:0/1437801074 conn(0x7f42680f3640 msgr2=0x7f42681077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.515+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 shutdown_connections 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.515+0000 7f426cb1f700 1 -- 172.21.15.107:0/1437801074 wait complete. 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.515+0000 7f426cb1f700 1 librados: shutdown 2024-02-10T08:49:30.527 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:49:30.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:30.787 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.100075+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.101529+0000 osd.2 (osd.2) 302 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.382770+0000 osd.3 (osd.3) 347 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.384165+0000 osd.3 (osd.3) 348 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.574311+0000 mgr.y (mgr.14553) 655 : cluster [DBG] pgmap v1204: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 301 B/s rd, 0 op/s 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.653237+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:49:30.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.655077+0000 osd.0 (osd.0) 175 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:49:30.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.813508+0000 osd.6 (osd.6) 326 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:49:30.789 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.815318+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:49:30.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.888457+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:49:30.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:29.891349+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:49:30.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.207688+0000 mon.a (mon.0) 3653 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:30.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.207755+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-10T08:49:30.791 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.217149+0000 mon.b (mon.1) 858 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:30.791 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.217626+0000 mon.a (mon.0) 3655 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:30.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.100075+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:49:30.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.101529+0000 osd.2 (osd.2) 302 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:49:30.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.382770+0000 osd.3 (osd.3) 347 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:49:30.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.384165+0000 osd.3 (osd.3) 348 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:49:30.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.574311+0000 mgr.y (mgr.14553) 655 : cluster [DBG] pgmap v1204: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 301 B/s rd, 0 op/s 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.653237+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.655077+0000 osd.0 (osd.0) 175 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.813508+0000 osd.6 (osd.6) 326 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.815318+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.888457+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:29.891349+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.207688+0000 mon.a (mon.0) 3653 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.207755+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.217149+0000 mon.b (mon.1) 858 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.217626+0000 mon.a (mon.0) 3655 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.100075+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:49:30.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.101529+0000 osd.2 (osd.2) 302 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.382770+0000 osd.3 (osd.3) 347 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.384165+0000 osd.3 (osd.3) 348 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.574311+0000 mgr.y (mgr.14553) 655 : cluster [DBG] pgmap v1204: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 301 B/s rd, 0 op/s 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.653237+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.655077+0000 osd.0 (osd.0) 175 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.813508+0000 osd.6 (osd.6) 326 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.815318+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.888457+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:29.891349+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.207688+0000 mon.a (mon.0) 3653 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.207755+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.217149+0000 mon.b (mon.1) 858 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:30.855 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.217626+0000 mon.a (mon.0) 3655 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.072 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:31.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:31.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:31.211 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:31.312 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:31.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:31.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:31.446 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.190362+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.193313+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.498502+0000 mgr.y (mgr.14553) 656 : audit [DBG] from='client.28797 v1:172.21.15.107:0/1437801074' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.664583+0000 osd.0 (osd.0) 176 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.665643+0000 osd.0 (osd.0) 177 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.754056+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.107:0/552382960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:30.754296+0000 mon.a (mon.0) 3656 : audit [INF] from='client.54401 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.904328+0000 osd.7 (osd.7) 326 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.905782+0000 osd.7 (osd.7) 327 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.976939+0000 osd.4 (osd.4) 71 : cluster [DBG] 2.f scrub starts 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:30.980662+0000 osd.4 (osd.4) 72 : cluster [DBG] 2.f scrub ok 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.204978+0000 mon.a (mon.0) 3657 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:49:31.466 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.209268+0000 mon.a (mon.0) 3658 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]': finished 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.209343+0000 mon.a (mon.0) 3659 : audit [INF] from='client.54401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.209386+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.243782+0000 mon.b (mon.1) 859 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.244103+0000 mon.a (mon.0) 3661 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.244577+0000 mon.b (mon.1) 860 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.244794+0000 mon.a (mon.0) 3662 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:31.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:31.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:31.582 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:49:31.582 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-02-10T08:49:31.582 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-02-10T08:49:31.582 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:49:31.600 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- 172.21.15.107:0/617743404 >> v1:172.21.15.107:6789/0 conn(0x7fed300f9570 legacy=0x7fed30102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- 172.21.15.107:0/617743404 shutdown_connections 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- 172.21.15.107:0/617743404 >> 172.21.15.107:0/617743404 conn(0x7fed300f2510 msgr2=0x7fed300f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- 172.21.15.107:0/617743404 shutdown_connections 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- 172.21.15.107:0/617743404 wait complete. 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 librados: starting msgr at 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 librados: starting objecter 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 Processor -- start 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 -- start start 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 librados: setting wanted keys 2024-02-10T08:49:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.562+0000 7fed364bf700 1 librados: calling monclient init 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed364bf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed301173f0 con 0x7fed30104350 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed364bf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed301185d0 con 0x7fed300f89c0 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed364bf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed301197d0 con 0x7fed300f9570 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fed300f9570 0x7fed300ffe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60240/0 (socket says 172.21.15.107:60240) 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2f7fe700 1 -- 172.21.15.107:0/552382960 learned_addr learned my addr 172.21.15.107:0/552382960 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter ms_handle_connect 0x7fed300f9570 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter ms_handle_connect 0x7fed30104350 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter ms_handle_connect 0x7fed300f89c0 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196294734 0 0) 0x7fed301197d0 con 0x7fed300f9570 2024-02-10T08:49:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed180033f0 con 0x7fed300f9570 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471137220 0 0) 0x7fed301173f0 con 0x7fed30104350 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed301197d0 con 0x7fed30104350 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115640885 0 0) 0x7fed301185d0 con 0x7fed300f89c0 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed301173f0 con 0x7fed300f89c0 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.563+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847343487 0 0) 0x7fed180033f0 con 0x7fed300f9570 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed301185d0 con 0x7fed300f9570 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1844887550 0 0) 0x7fed301197d0 con 0x7fed30104350 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed180033f0 con 0x7fed30104350 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fed20002130 con 0x7fed300f9570 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fed28001bc0 con 0x7fed30104350 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 418062792 0 0) 0x7fed301173f0 con 0x7fed300f89c0 2024-02-10T08:49:31.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed301197d0 con 0x7fed300f89c0 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fed24001600 con 0x7fed300f89c0 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1025775680 0 0) 0x7fed301185d0 con 0x7fed300f9570 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 >> v1:172.21.15.195:6789/0 conn(0x7fed300f89c0 legacy=0x7fed300ff710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 >> v1:172.21.15.107:6789/0 conn(0x7fed30104350 legacy=0x7fed30115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3011a9d0 con 0x7fed300f9570 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed301175e0 con 0x7fed300f9570 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed364bf700 10 client.54401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30117a50 con 0x7fed300f9570 2024-02-10T08:49:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed364bf700 1 librados: init done 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fed200022d0 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.564+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fed200037a0 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c004f20 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fed20001880 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(791..791 src has 420..791) v4 ==== 8592+0+0 (unknown 898867797 0 0) 0x7fed20079370 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed2d7fa700 10 client.54401.objecter ms_dispatch 0x7fed30001550 osd_map(791..791 src has 420..791) v4 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed2d7fa700 3 client.54401.objecter handle_osd_map got epochs [791,791] > 0 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.565+0000 7fed2d7fa700 3 client.54401.objecter handle_osd_map decoding full epoch 791 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.566+0000 7fed2d7fa700 20 client.54401.objecter dump_active .. 0 homeless 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.566+0000 7fed2d7fa700 10 client.54401.objecter ms_handle_connect 0x7fed18063ce0 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.570+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fed2004c5d0 con 0x7fed300f9570 2024-02-10T08:49:31.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:30.752+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fed1c0051d0 con 0x7fed300f9570 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.209+0000 7fed2d7fa700 1 -- 172.21.15.107:0/552382960 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v792) v1 ==== 101+0+0 (unknown 3681602192 0 0) 0x7fed2004aa40 con 0x7fed300f9570 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 10 librados: watch_flush enter 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 10 librados: watch_flush exit 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 20 client.54401.objecter shutdown clearing up homeless session... 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 10 client.54401.objecter successfully canceled tick 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 >> v1:172.21.15.107:6800/2634842009 conn(0x7fed18063ce0 legacy=0x7fed180661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 >> v1:172.21.15.107:6790/0 conn(0x7fed300f9570 legacy=0x7fed300ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 shutdown_connections 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 >> 172.21.15.107:0/552382960 conn(0x7fed300f2510 msgr2=0x7fed301077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 shutdown_connections 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 -- 172.21.15.107:0/552382960 wait complete. 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:31.212+0000 7fed364bf700 1 librados: shutdown 2024-02-10T08:49:31.727 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:49:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.190362+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-10T08:49:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.193313+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-10T08:49:31.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.498502+0000 mgr.y (mgr.14553) 656 : audit [DBG] from='client.28797 v1:172.21.15.107:0/1437801074' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.664583+0000 osd.0 (osd.0) 176 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.665643+0000 osd.0 (osd.0) 177 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.754056+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.107:0/552382960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:30.754296+0000 mon.a (mon.0) 3656 : audit [INF] from='client.54401 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.904328+0000 osd.7 (osd.7) 326 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.905782+0000 osd.7 (osd.7) 327 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.976939+0000 osd.4 (osd.4) 71 : cluster [DBG] 2.f scrub starts 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:30.980662+0000 osd.4 (osd.4) 72 : cluster [DBG] 2.f scrub ok 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.204978+0000 mon.a (mon.0) 3657 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.209268+0000 mon.a (mon.0) 3658 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]': finished 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.209343+0000 mon.a (mon.0) 3659 : audit [INF] from='client.54401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.209386+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-10T08:49:31.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.243782+0000 mon.b (mon.1) 859 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.244103+0000 mon.a (mon.0) 3661 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.244577+0000 mon.b (mon.1) 860 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.790 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.244794+0000 mon.a (mon.0) 3662 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.190362+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-10T08:49:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.193313+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-10T08:49:31.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.498502+0000 mgr.y (mgr.14553) 656 : audit [DBG] from='client.28797 v1:172.21.15.107:0/1437801074' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.664583+0000 osd.0 (osd.0) 176 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.665643+0000 osd.0 (osd.0) 177 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.754056+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.107:0/552382960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:30.754296+0000 mon.a (mon.0) 3656 : audit [INF] from='client.54401 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.904328+0000 osd.7 (osd.7) 326 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.905782+0000 osd.7 (osd.7) 327 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.976939+0000 osd.4 (osd.4) 71 : cluster [DBG] 2.f scrub starts 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:30.980662+0000 osd.4 (osd.4) 72 : cluster [DBG] 2.f scrub ok 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.204978+0000 mon.a (mon.0) 3657 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.209268+0000 mon.a (mon.0) 3658 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]': finished 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.209343+0000 mon.a (mon.0) 3659 : audit [INF] from='client.54401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.209386+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.243782+0000 mon.b (mon.1) 859 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.244103+0000 mon.a (mon.0) 3661 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:31.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.244577+0000 mon.b (mon.1) 860 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:31.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.244794+0000 mon.a (mon.0) 3662 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-138"}]: dispatch 2024-02-10T08:49:32.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.016852+0000 osd.2 (osd.2) 303 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.019735+0000 osd.2 (osd.2) 304 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.377234+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.379089+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:31.574990+0000 mgr.y (mgr.14553) 657 : cluster [DBG] pgmap v1207: 194 pgs: 7 unknown, 187 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:31.847069+0000 mgr.y (mgr.14553) 658 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:32.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.211592+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.016852+0000 osd.2 (osd.2) 303 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.019735+0000 osd.2 (osd.2) 304 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.377234+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.379089+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:31.574990+0000 mgr.y (mgr.14553) 657 : cluster [DBG] pgmap v1207: 194 pgs: 7 unknown, 187 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:31.847069+0000 mgr.y (mgr.14553) 658 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:32.629 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.211592+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.016852+0000 osd.2 (osd.2) 303 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.019735+0000 osd.2 (osd.2) 304 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.377234+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.379089+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:31.574990+0000 mgr.y (mgr.14553) 657 : cluster [DBG] pgmap v1207: 194 pgs: 7 unknown, 187 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:31.847069+0000 mgr.y (mgr.14553) 658 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:32.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.211592+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-10T08:49:33.252 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:33.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:33.271 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1205,"stamp":"2024-02-10T08:49:30.205267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006090,"num_objects":235,"num_object_clones":1,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":1344,"num_read_kb":1081,"num_write":783,"num_write_kb":4012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":13320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":936,"ondisk_log_size":936,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":582,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2461400,"kb_used_data":56984,"kb_used_omap":59,"kb_used_meta":2404292,"kb_avail":747499816,"statfs":{"total":767960285184,"available":765439811584,"internally_reserved":0,"allocated":58351616,"data_stored":33972785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2461995627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":-2,"num_evict_kb":-2,"num_promote":-4,"num_flush_mode_high":-2,"num_flush_mode_low":0,"num_evict_mode_some":-2,"num_evict_mode_full":0,"num_objects_pinned":-2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013760"},"pg_stats":[{"pgid":"253.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"253.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"774'20","reported_seq":245,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.690202+0000","last_change":"2024-02-10T08:49:19.690171+0000","last_active":"2024-02-10T08:49:19.690202+0000","last_peered":"2024-02-10T08:49:19.690202+0000","last_clean":"2024-02-10T08:49:19.690202+0000","last_became_active":"2024-02-10T08:45:46.132344+0000","last_became_peered":"2024-02-10T08:45:46.132344+0000","last_unstale":"2024-02-10T08:49:19.690202+0000","last_undegraded":"2024-02-10T08:49:19.690202+0000","last_fullsized":"2024-02-10T08:49:19.690202+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'20","last_scrub_stamp":"2024-02-10T08:49:19.690086+0000","last_deep_scrub":"669'16","last_deep_scrub_stamp":"2024-02-10T08:47:22.551825+0000","last_clean_scrub_stamp":"2024-02-10T08:49:19.690086+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"253.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"749'2","reported_seq":247,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684772+0000","last_change":"2024-02-10T08:48:55.061793+0000","last_active":"2024-02-10T08:49:19.684772+0000","last_peered":"2024-02-10T08:49:19.684772+0000","last_clean":"2024-02-10T08:49:19.684772+0000","last_became_active":"2024-02-10T08:47:49.222914+0000","last_became_peered":"2024-02-10T08:47:49.222914+0000","last_unstale":"2024-02-10T08:49:19.684772+0000","last_undegraded":"2024-02-10T08:49:19.684772+0000","last_fullsized":"2024-02-10T08:49:19.684772+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'2","last_scrub_stamp":"2024-02-10T08:48:55.061722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.061722+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"253.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"774'17","reported_seq":264,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194787+0000","last_change":"2024-02-10T08:48:34.935440+0000","last_active":"2024-02-10T08:49:21.194787+0000","last_peered":"2024-02-10T08:49:21.194787+0000","last_clean":"2024-02-10T08:49:21.194787+0000","last_became_active":"2024-02-10T08:46:29.211292+0000","last_became_peered":"2024-02-10T08:46:29.211292+0000","last_unstale":"2024-02-10T08:49:21.194787+0000","last_undegraded":"2024-02-10T08:49:21.194787+0000","last_fullsized":"2024-02-10T08:49:21.194787+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'13","last_scrub_stamp":"2024-02-10T08:48:34.935360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:34.935360+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"253.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"749'21","reported_seq":295,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959496+0000","last_change":"2024-02-10T08:48:05.926678+0000","last_active":"2024-02-10T08:49:19.959496+0000","last_peered":"2024-02-10T08:49:19.959496+0000","last_clean":"2024-02-10T08:49:19.959496+0000","last_became_active":"2024-02-10T08:47:49.231271+0000","last_became_peered":"2024-02-10T08:47:49.231271+0000","last_unstale":"2024-02-10T08:49:19.959496+0000","last_undegraded":"2024-02-10T08:49:19.959496+0000","last_fullsized":"2024-02-10T08:49:19.959496+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"693'17","last_scrub_stamp":"2024-02-10T08:48:05.926589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.926589+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"253.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"774'4","reported_seq":217,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687031+0000","last_change":"2024-02-10T08:48:03.075504+0000","last_active":"2024-02-10T08:49:19.687031+0000","last_peered":"2024-02-10T08:49:19.687031+0000","last_clean":"2024-02-10T08:49:19.687031+0000","last_became_active":"2024-02-10T08:45:46.129572+0000","last_became_peered":"2024-02-10T08:45:46.129572+0000","last_unstale":"2024-02-10T08:49:19.687031+0000","last_undegraded":"2024-02-10T08:49:19.687031+0000","last_fullsized":"2024-02-10T08:49:19.687031+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.075414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.075414+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000032,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"253.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"749'1","reported_seq":228,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685086+0000","last_change":"2024-02-10T08:48:44.070853+0000","last_active":"2024-02-10T08:49:19.685086+0000","last_peered":"2024-02-10T08:49:19.685086+0000","last_clean":"2024-02-10T08:49:19.685086+0000","last_became_active":"2024-02-10T08:46:15.351881+0000","last_became_peered":"2024-02-10T08:46:15.351881+0000","last_unstale":"2024-02-10T08:49:19.685086+0000","last_undegraded":"2024-02-10T08:49:19.685086+0000","last_fullsized":"2024-02-10T08:49:19.685086+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'1","last_scrub_stamp":"2024-02-10T08:48:44.070730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.070730+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"253.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"763'11","reported_seq":247,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958633+0000","last_change":"2024-02-10T08:48:39.006369+0000","last_active":"2024-02-10T08:49:19.958633+0000","last_peered":"2024-02-10T08:49:19.958633+0000","last_clean":"2024-02-10T08:49:19.958633+0000","last_became_active":"2024-02-10T08:46:29.219928+0000","last_became_peered":"2024-02-10T08:46:29.219928+0000","last_unstale":"2024-02-10T08:49:19.958633+0000","last_undegraded":"2024-02-10T08:49:19.958633+0000","last_fullsized":"2024-02-10T08:49:19.958633+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"749'10","last_scrub_stamp":"2024-02-10T08:48:39.006289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:39.006289+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"253.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-10T08:49:28.197982+0000","last_change":"2024-02-10T08:49:28.197982+0000","last_active":"2024-02-10T08:49:28.197982+0000","last_peered":"2024-02-10T08:49:28.197982+0000","last_clean":"2024-02-10T08:49:28.197982+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-10T08:49:28.197982+0000","last_undegraded":"2024-02-10T08:49:28.197982+0000","last_fullsized":"2024-02-10T08:49:28.197982+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:28.197982+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.197982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"774'19","reported_seq":207,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213314+0000","last_change":"2024-02-10T08:48:28.968144+0000","last_active":"2024-02-10T08:49:26.213314+0000","last_peered":"2024-02-10T08:49:26.213314+0000","last_clean":"2024-02-10T08:49:26.213314+0000","last_became_active":"2024-02-10T08:46:29.210532+0000","last_became_peered":"2024-02-10T08:46:29.210532+0000","last_unstale":"2024-02-10T08:49:26.213314+0000","last_undegraded":"2024-02-10T08:49:26.213314+0000","last_fullsized":"2024-02-10T08:49:26.213314+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"685'16","last_scrub_stamp":"2024-02-10T08:48:28.968036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:45.107430+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.968036+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"636'18","reported_seq":475,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.674681+0000","last_change":"2024-02-10T08:48:29.848382+0000","last_active":"2024-02-10T08:49:21.674681+0000","last_peered":"2024-02-10T08:49:21.674681+0000","last_clean":"2024-02-10T08:49:21.674681+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:49:21.674681+0000","last_undegraded":"2024-02-10T08:49:21.674681+0000","last_fullsized":"2024-02-10T08:49:21.674681+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:48:29.848301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.848301+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":852,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684479+0000","last_change":"2024-02-10T08:47:58.959348+0000","last_active":"2024-02-10T08:49:19.684479+0000","last_peered":"2024-02-10T08:49:19.684479+0000","last_clean":"2024-02-10T08:49:19.684479+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:49:19.684479+0000","last_undegraded":"2024-02-10T08:49:19.684479+0000","last_fullsized":"2024-02-10T08:49:19.684479+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:47:58.959271+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.959271+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":786,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194947+0000","last_change":"2024-02-10T08:48:03.838139+0000","last_active":"2024-02-10T08:49:21.194947+0000","last_peered":"2024-02-10T08:49:21.194947+0000","last_clean":"2024-02-10T08:49:21.194947+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:49:21.194947+0000","last_undegraded":"2024-02-10T08:49:21.194947+0000","last_fullsized":"2024-02-10T08:49:21.194947+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.838064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.838064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":771,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.686905+0000","last_change":"2024-02-10T08:48:01.033569+0000","last_active":"2024-02-10T08:49:19.686905+0000","last_peered":"2024-02-10T08:49:19.686905+0000","last_clean":"2024-02-10T08:49:19.686905+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:49:19.686905+0000","last_undegraded":"2024-02-10T08:49:19.686905+0000","last_fullsized":"2024-02-10T08:49:19.686905+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:01.033502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:01.033502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":732,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:23.108146+0000","last_change":"2024-02-10T08:49:23.108122+0000","last_active":"2024-02-10T08:49:23.108146+0000","last_peered":"2024-02-10T08:49:23.108146+0000","last_clean":"2024-02-10T08:49:23.108146+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:49:23.108146+0000","last_undegraded":"2024-02-10T08:49:23.108146+0000","last_fullsized":"2024-02-10T08:49:23.108146+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.108061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":233,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544604+0000","last_change":"2024-02-10T08:48:32.371647+0000","last_active":"2024-02-10T08:49:19.544604+0000","last_peered":"2024-02-10T08:49:19.544604+0000","last_clean":"2024-02-10T08:49:19.544604+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:49:19.544604+0000","last_undegraded":"2024-02-10T08:49:19.544604+0000","last_fullsized":"2024-02-10T08:49:19.544604+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.371572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.371572+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174204+0000","last_change":"2024-02-10T08:48:38.645376+0000","last_active":"2024-02-10T08:49:26.174204+0000","last_peered":"2024-02-10T08:49:26.174204+0000","last_clean":"2024-02-10T08:49:26.174204+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:49:26.174204+0000","last_undegraded":"2024-02-10T08:49:26.174204+0000","last_fullsized":"2024-02-10T08:49:26.174204+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.645301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.645301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":797,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194517+0000","last_change":"2024-02-10T08:48:56.119888+0000","last_active":"2024-02-10T08:49:21.194517+0000","last_peered":"2024-02-10T08:49:21.194517+0000","last_clean":"2024-02-10T08:49:21.194517+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:49:21.194517+0000","last_undegraded":"2024-02-10T08:49:21.194517+0000","last_fullsized":"2024-02-10T08:49:21.194517+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:56.119799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.119799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":174,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:25.635459+0000","last_change":"2024-02-10T08:49:25.635459+0000","last_active":"2024-02-10T08:49:25.635459+0000","last_peered":"2024-02-10T08:49:25.635459+0000","last_clean":"2024-02-10T08:49:25.635459+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:49:25.635459+0000","last_undegraded":"2024-02-10T08:49:25.635459+0000","last_fullsized":"2024-02-10T08:49:25.635459+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.635394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.635394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":720,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.688016+0000","last_change":"2024-02-10T08:48:55.016819+0000","last_active":"2024-02-10T08:49:19.688016+0000","last_peered":"2024-02-10T08:49:19.688016+0000","last_clean":"2024-02-10T08:49:19.688016+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:49:19.688016+0000","last_undegraded":"2024-02-10T08:49:19.688016+0000","last_fullsized":"2024-02-10T08:49:19.688016+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.016738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.016738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":945,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685613+0000","last_change":"2024-02-10T08:48:38.016707+0000","last_active":"2024-02-10T08:49:19.685613+0000","last_peered":"2024-02-10T08:49:19.685613+0000","last_clean":"2024-02-10T08:49:19.685613+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:49:19.685613+0000","last_undegraded":"2024-02-10T08:49:19.685613+0000","last_fullsized":"2024-02-10T08:49:19.685613+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:48:38.016615+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.016615+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":784,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.195583+0000","last_change":"2024-02-10T08:48:30.851950+0000","last_active":"2024-02-10T08:49:21.195583+0000","last_peered":"2024-02-10T08:49:21.195583+0000","last_clean":"2024-02-10T08:49:21.195583+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:49:21.195583+0000","last_undegraded":"2024-02-10T08:49:21.195583+0000","last_fullsized":"2024-02-10T08:49:21.195583+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.851860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.851860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":818,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.195100+0000","last_change":"2024-02-10T08:48:02.843328+0000","last_active":"2024-02-10T08:49:21.195100+0000","last_peered":"2024-02-10T08:49:21.195100+0000","last_clean":"2024-02-10T08:49:21.195100+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:49:21.195100+0000","last_undegraded":"2024-02-10T08:49:21.195100+0000","last_fullsized":"2024-02-10T08:49:21.195100+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.843266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.843266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":337,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512084+0000","last_change":"2024-02-10T08:48:40.860303+0000","last_active":"2024-02-10T08:49:19.512084+0000","last_peered":"2024-02-10T08:49:19.512084+0000","last_clean":"2024-02-10T08:49:19.512084+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:49:19.512084+0000","last_undegraded":"2024-02-10T08:49:19.512084+0000","last_fullsized":"2024-02-10T08:49:19.512084+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.860228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.860228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":347,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.513060+0000","last_change":"2024-02-10T08:47:57.641284+0000","last_active":"2024-02-10T08:49:19.513060+0000","last_peered":"2024-02-10T08:49:19.513060+0000","last_clean":"2024-02-10T08:49:19.513060+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:49:19.513060+0000","last_undegraded":"2024-02-10T08:49:19.513060+0000","last_fullsized":"2024-02-10T08:49:19.513060+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.641182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.641182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":721,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214486+0000","last_change":"2024-02-10T08:47:49.965414+0000","last_active":"2024-02-10T08:49:26.214486+0000","last_peered":"2024-02-10T08:49:26.214486+0000","last_clean":"2024-02-10T08:49:26.214486+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:49:26.214486+0000","last_undegraded":"2024-02-10T08:49:26.214486+0000","last_fullsized":"2024-02-10T08:49:26.214486+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:47:26.935760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.935760+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":84,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959067+0000","last_change":"2024-02-10T08:47:49.226548+0000","last_active":"2024-02-10T08:49:19.959067+0000","last_peered":"2024-02-10T08:49:19.959067+0000","last_clean":"2024-02-10T08:49:19.959067+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:49:19.959067+0000","last_undegraded":"2024-02-10T08:49:19.959067+0000","last_fullsized":"2024-02-10T08:49:19.959067+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174192+0000","last_change":"2024-02-10T08:48:32.722180+0000","last_active":"2024-02-10T08:49:26.174192+0000","last_peered":"2024-02-10T08:49:26.174192+0000","last_clean":"2024-02-10T08:49:26.174192+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:49:26.174192+0000","last_undegraded":"2024-02-10T08:49:26.174192+0000","last_fullsized":"2024-02-10T08:49:26.174192+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.722121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.722121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":775,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.195166+0000","last_change":"2024-02-10T08:48:47.163067+0000","last_active":"2024-02-10T08:49:21.195166+0000","last_peered":"2024-02-10T08:49:21.195166+0000","last_clean":"2024-02-10T08:49:21.195166+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:49:21.195166+0000","last_undegraded":"2024-02-10T08:49:21.195166+0000","last_fullsized":"2024-02-10T08:49:21.195166+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.163011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.163011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":581,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213852+0000","last_change":"2024-02-10T08:49:25.010060+0000","last_active":"2024-02-10T08:49:26.213852+0000","last_peered":"2024-02-10T08:49:26.213852+0000","last_clean":"2024-02-10T08:49:26.213852+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:49:26.213852+0000","last_undegraded":"2024-02-10T08:49:26.213852+0000","last_fullsized":"2024-02-10T08:49:26.213852+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.009961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.009961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":645,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214005+0000","last_change":"2024-02-10T08:48:56.061862+0000","last_active":"2024-02-10T08:49:26.214005+0000","last_peered":"2024-02-10T08:49:26.214005+0000","last_clean":"2024-02-10T08:49:26.214005+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:49:26.214005+0000","last_undegraded":"2024-02-10T08:49:26.214005+0000","last_fullsized":"2024-02-10T08:49:26.214005+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:48:56.061783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.061783+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.173767+0000","last_change":"2024-02-10T08:49:22.842445+0000","last_active":"2024-02-10T08:49:26.173767+0000","last_peered":"2024-02-10T08:49:26.173767+0000","last_clean":"2024-02-10T08:49:26.173767+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:49:26.173767+0000","last_undegraded":"2024-02-10T08:49:26.173767+0000","last_fullsized":"2024-02-10T08:49:26.173767+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.842361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.842361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1038,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.674195+0000","last_change":"2024-02-10T08:47:29.049751+0000","last_active":"2024-02-10T08:49:21.674195+0000","last_peered":"2024-02-10T08:49:21.674195+0000","last_clean":"2024-02-10T08:49:21.674195+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:49:21.674195+0000","last_undegraded":"2024-02-10T08:49:21.674195+0000","last_fullsized":"2024-02-10T08:49:21.674195+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:47:29.049685+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.049685+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":605,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214051+0000","last_change":"2024-02-10T08:48:48.962890+0000","last_active":"2024-02-10T08:49:26.214051+0000","last_peered":"2024-02-10T08:49:26.214051+0000","last_clean":"2024-02-10T08:49:26.214051+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:49:26.214051+0000","last_undegraded":"2024-02-10T08:49:26.214051+0000","last_fullsized":"2024-02-10T08:49:26.214051+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.962820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.962820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":177,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544730+0000","last_change":"2024-02-10T08:48:29.433315+0000","last_active":"2024-02-10T08:49:19.544730+0000","last_peered":"2024-02-10T08:49:19.544730+0000","last_clean":"2024-02-10T08:49:19.544730+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:49:19.544730+0000","last_undegraded":"2024-02-10T08:49:19.544730+0000","last_fullsized":"2024-02-10T08:49:19.544730+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:29.433223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.433223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":109,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174971+0000","last_change":"2024-02-10T08:48:51.755042+0000","last_active":"2024-02-10T08:49:26.174971+0000","last_peered":"2024-02-10T08:49:26.174971+0000","last_clean":"2024-02-10T08:49:26.174971+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:49:26.174971+0000","last_undegraded":"2024-02-10T08:49:26.174971+0000","last_fullsized":"2024-02-10T08:49:26.174971+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:48:51.754925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.754925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":97,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.173754+0000","last_change":"2024-02-10T08:49:24.873487+0000","last_active":"2024-02-10T08:49:26.173754+0000","last_peered":"2024-02-10T08:49:26.173754+0000","last_clean":"2024-02-10T08:49:26.173754+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:49:26.173754+0000","last_undegraded":"2024-02-10T08:49:26.173754+0000","last_fullsized":"2024-02-10T08:49:26.173754+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:49:24.873416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.873416+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":849,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.675613+0000","last_change":"2024-02-10T08:47:13.008447+0000","last_active":"2024-02-10T08:49:26.675613+0000","last_peered":"2024-02-10T08:49:26.675613+0000","last_clean":"2024-02-10T08:49:26.675613+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:49:26.675613+0000","last_undegraded":"2024-02-10T08:49:26.675613+0000","last_fullsized":"2024-02-10T08:49:26.675613+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:47:13.008374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.008374+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":784,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687639+0000","last_change":"2024-02-10T08:47:49.238554+0000","last_active":"2024-02-10T08:49:19.687639+0000","last_peered":"2024-02-10T08:49:19.687639+0000","last_clean":"2024-02-10T08:49:19.687639+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:49:19.687639+0000","last_undegraded":"2024-02-10T08:49:19.687639+0000","last_fullsized":"2024-02-10T08:49:19.687639+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:31.837641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:47:31.837641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":733,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687034+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:49:19.687034+0000","last_peered":"2024-02-10T08:49:19.687034+0000","last_clean":"2024-02-10T08:49:19.687034+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:49:19.687034+0000","last_undegraded":"2024-02-10T08:49:19.687034+0000","last_fullsized":"2024-02-10T08:49:19.687034+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:17.824249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.824249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":688,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214242+0000","last_change":"2024-02-10T08:49:20.031309+0000","last_active":"2024-02-10T08:49:26.214242+0000","last_peered":"2024-02-10T08:49:26.214242+0000","last_clean":"2024-02-10T08:49:26.214242+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:49:26.214242+0000","last_undegraded":"2024-02-10T08:49:26.214242+0000","last_fullsized":"2024-02-10T08:49:26.214242+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:20.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.031226+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":365,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.511977+0000","last_change":"2024-02-10T08:48:31.859394+0000","last_active":"2024-02-10T08:49:19.511977+0000","last_peered":"2024-02-10T08:49:19.511977+0000","last_clean":"2024-02-10T08:49:19.511977+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:49:19.511977+0000","last_undegraded":"2024-02-10T08:49:19.511977+0000","last_fullsized":"2024-02-10T08:49:19.511977+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.859315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.859315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":941,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.674681+0000","last_change":"2024-02-10T08:48:51.038110+0000","last_active":"2024-02-10T08:49:21.674681+0000","last_peered":"2024-02-10T08:49:21.674681+0000","last_clean":"2024-02-10T08:49:21.674681+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:49:21.674681+0000","last_undegraded":"2024-02-10T08:49:21.674681+0000","last_fullsized":"2024-02-10T08:49:21.674681+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.038025+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":752,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959655+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:49:19.959655+0000","last_peered":"2024-02-10T08:49:19.959655+0000","last_clean":"2024-02-10T08:49:19.959655+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:49:19.959655+0000","last_undegraded":"2024-02-10T08:49:19.959655+0000","last_fullsized":"2024-02-10T08:49:19.959655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":703,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.361169+0000","last_change":"2024-02-10T08:49:21.361148+0000","last_active":"2024-02-10T08:49:21.361169+0000","last_peered":"2024-02-10T08:49:21.361169+0000","last_clean":"2024-02-10T08:49:21.361169+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:49:21.361169+0000","last_undegraded":"2024-02-10T08:49:21.361169+0000","last_fullsized":"2024-02-10T08:49:21.361169+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.361087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.361087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1001,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.659787+0000","last_change":"2024-02-10T08:47:49.260456+0000","last_active":"2024-02-10T08:49:21.659787+0000","last_peered":"2024-02-10T08:49:21.659787+0000","last_clean":"2024-02-10T08:49:21.659787+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:49:21.659787+0000","last_undegraded":"2024-02-10T08:49:21.659787+0000","last_fullsized":"2024-02-10T08:49:21.659787+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:46:39.499038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:46:39.499038+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":659,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.212912+0000","last_change":"2024-02-10T08:47:27.942949+0000","last_active":"2024-02-10T08:49:26.212912+0000","last_peered":"2024-02-10T08:49:26.212912+0000","last_clean":"2024-02-10T08:49:26.212912+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:49:26.212912+0000","last_undegraded":"2024-02-10T08:49:26.212912+0000","last_fullsized":"2024-02-10T08:49:26.212912+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:47:27.942861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.942861+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":794,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685558+0000","last_change":"2024-02-10T08:48:54.105562+0000","last_active":"2024-02-10T08:49:19.685558+0000","last_peered":"2024-02-10T08:49:19.685558+0000","last_clean":"2024-02-10T08:49:19.685558+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:49:19.685558+0000","last_undegraded":"2024-02-10T08:49:19.685558+0000","last_fullsized":"2024-02-10T08:49:19.685558+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.105490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.105490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":339,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512446+0000","last_change":"2024-02-10T08:47:55.566362+0000","last_active":"2024-02-10T08:49:19.512446+0000","last_peered":"2024-02-10T08:49:19.512446+0000","last_clean":"2024-02-10T08:49:19.512446+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:49:19.512446+0000","last_undegraded":"2024-02-10T08:49:19.512446+0000","last_fullsized":"2024-02-10T08:49:19.512446+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.566281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.566281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":192,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544994+0000","last_change":"2024-02-10T08:48:05.524065+0000","last_active":"2024-02-10T08:49:19.544994+0000","last_peered":"2024-02-10T08:49:19.544994+0000","last_clean":"2024-02-10T08:49:19.544994+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:49:19.544994+0000","last_undegraded":"2024-02-10T08:49:19.544994+0000","last_fullsized":"2024-02-10T08:49:19.544994+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:05.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":624,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213550+0000","last_change":"2024-02-10T08:48:45.969499+0000","last_active":"2024-02-10T08:49:26.213550+0000","last_peered":"2024-02-10T08:49:26.213550+0000","last_clean":"2024-02-10T08:49:26.213550+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:49:26.213550+0000","last_undegraded":"2024-02-10T08:49:26.213550+0000","last_fullsized":"2024-02-10T08:49:26.213550+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.969416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.969416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":945,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958630+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:49:19.958630+0000","last_peered":"2024-02-10T08:49:19.958630+0000","last_clean":"2024-02-10T08:49:19.958630+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:49:19.958630+0000","last_undegraded":"2024-02-10T08:49:19.958630+0000","last_fullsized":"2024-02-10T08:49:19.958630+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":97,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174602+0000","last_change":"2024-02-10T08:48:36.653104+0000","last_active":"2024-02-10T08:49:26.174602+0000","last_peered":"2024-02-10T08:49:26.174602+0000","last_clean":"2024-02-10T08:49:26.174602+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:49:26.174602+0000","last_undegraded":"2024-02-10T08:49:26.174602+0000","last_fullsized":"2024-02-10T08:49:26.174602+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:48:36.652981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.652981+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":592,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213557+0000","last_change":"2024-02-10T08:48:42.984939+0000","last_active":"2024-02-10T08:49:26.213557+0000","last_peered":"2024-02-10T08:49:26.213557+0000","last_clean":"2024-02-10T08:49:26.213557+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:49:26.213557+0000","last_undegraded":"2024-02-10T08:49:26.213557+0000","last_fullsized":"2024-02-10T08:49:26.213557+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.984822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.984822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":192,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544939+0000","last_change":"2024-02-10T08:48:37.427622+0000","last_active":"2024-02-10T08:49:19.544939+0000","last_peered":"2024-02-10T08:49:19.544939+0000","last_clean":"2024-02-10T08:49:19.544939+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:49:19.544939+0000","last_undegraded":"2024-02-10T08:49:19.544939+0000","last_fullsized":"2024-02-10T08:49:19.544939+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:37.427518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:37.427518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":781,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685680+0000","last_change":"2024-02-10T08:48:43.081221+0000","last_active":"2024-02-10T08:49:19.685680+0000","last_peered":"2024-02-10T08:49:19.685680+0000","last_clean":"2024-02-10T08:49:19.685680+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:49:19.685680+0000","last_undegraded":"2024-02-10T08:49:19.685680+0000","last_fullsized":"2024-02-10T08:49:19.685680+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.081148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.081148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":187,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544654+0000","last_change":"2024-02-10T08:47:57.560707+0000","last_active":"2024-02-10T08:49:19.544654+0000","last_peered":"2024-02-10T08:49:19.544654+0000","last_clean":"2024-02-10T08:49:19.544654+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:49:19.544654+0000","last_undegraded":"2024-02-10T08:49:19.544654+0000","last_fullsized":"2024-02-10T08:49:19.544654+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:47:57.560587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.560587+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":367,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:23.762672+0000","last_change":"2024-02-10T08:49:23.762672+0000","last_active":"2024-02-10T08:49:23.762672+0000","last_peered":"2024-02-10T08:49:23.762672+0000","last_clean":"2024-02-10T08:49:23.762672+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:49:23.762672+0000","last_undegraded":"2024-02-10T08:49:23.762672+0000","last_fullsized":"2024-02-10T08:49:23.762672+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.762550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.762550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":668,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.109287+0000","last_change":"2024-02-10T08:49:26.109262+0000","last_active":"2024-02-10T08:49:26.109287+0000","last_peered":"2024-02-10T08:49:26.109287+0000","last_clean":"2024-02-10T08:49:26.109287+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:49:26.109287+0000","last_undegraded":"2024-02-10T08:49:26.109287+0000","last_fullsized":"2024-02-10T08:49:26.109287+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.109194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.109194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":750,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194972+0000","last_change":"2024-02-10T08:48:39.036788+0000","last_active":"2024-02-10T08:49:21.194972+0000","last_peered":"2024-02-10T08:49:21.194972+0000","last_clean":"2024-02-10T08:49:21.194972+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:49:21.194972+0000","last_undegraded":"2024-02-10T08:49:21.194972+0000","last_fullsized":"2024-02-10T08:49:21.194972+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:39.036699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:39.036699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":810,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687931+0000","last_change":"2024-02-10T08:47:49.238189+0000","last_active":"2024-02-10T08:49:19.687931+0000","last_peered":"2024-02-10T08:49:19.687931+0000","last_clean":"2024-02-10T08:49:19.687931+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:49:19.687931+0000","last_undegraded":"2024-02-10T08:49:19.687931+0000","last_fullsized":"2024-02-10T08:49:19.687931+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:13.825974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.825974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":229,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545098+0000","last_change":"2024-02-10T08:47:49.975699+0000","last_active":"2024-02-10T08:49:19.545098+0000","last_peered":"2024-02-10T08:49:19.545098+0000","last_clean":"2024-02-10T08:49:19.545098+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:49:19.545098+0000","last_undegraded":"2024-02-10T08:49:19.545098+0000","last_fullsized":"2024-02-10T08:49:19.545098+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.370523+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":369,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:22.774270+0000","last_change":"2024-02-10T08:49:22.774270+0000","last_active":"2024-02-10T08:49:22.774270+0000","last_peered":"2024-02-10T08:49:22.774270+0000","last_clean":"2024-02-10T08:49:22.774270+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:49:22.774270+0000","last_undegraded":"2024-02-10T08:49:22.774270+0000","last_fullsized":"2024-02-10T08:49:22.774270+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.774198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.774198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":185,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544446+0000","last_change":"2024-02-10T08:48:36.452244+0000","last_active":"2024-02-10T08:49:19.544446+0000","last_peered":"2024-02-10T08:49:19.544446+0000","last_clean":"2024-02-10T08:49:19.544446+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:49:19.544446+0000","last_undegraded":"2024-02-10T08:49:19.544446+0000","last_fullsized":"2024-02-10T08:49:19.544446+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.452178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.452178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":98,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174422+0000","last_change":"2024-02-10T08:48:05.945279+0000","last_active":"2024-02-10T08:49:26.174422+0000","last_peered":"2024-02-10T08:49:26.174422+0000","last_clean":"2024-02-10T08:49:26.174422+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:49:26.174422+0000","last_undegraded":"2024-02-10T08:49:26.174422+0000","last_fullsized":"2024-02-10T08:49:26.174422+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.945162+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":728,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:23.855127+0000","last_change":"2024-02-10T08:49:23.855127+0000","last_active":"2024-02-10T08:49:23.855127+0000","last_peered":"2024-02-10T08:49:23.855127+0000","last_clean":"2024-02-10T08:49:23.855127+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:49:23.855127+0000","last_undegraded":"2024-02-10T08:49:23.855127+0000","last_fullsized":"2024-02-10T08:49:23.855127+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.855041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.855041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":344,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512866+0000","last_change":"2024-02-10T08:48:54.704302+0000","last_active":"2024-02-10T08:49:19.512866+0000","last_peered":"2024-02-10T08:49:19.512866+0000","last_clean":"2024-02-10T08:49:19.512866+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:49:19.512866+0000","last_undegraded":"2024-02-10T08:49:19.512866+0000","last_fullsized":"2024-02-10T08:49:19.512866+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.704183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.704183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":860,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.686589+0000","last_change":"2024-02-10T08:48:31.839331+0000","last_active":"2024-02-10T08:49:19.686589+0000","last_peered":"2024-02-10T08:49:19.686589+0000","last_clean":"2024-02-10T08:49:19.686589+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:49:19.686589+0000","last_undegraded":"2024-02-10T08:49:19.686589+0000","last_fullsized":"2024-02-10T08:49:19.686589+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:48:31.839259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.839259+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":348,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512143+0000","last_change":"2024-02-10T08:48:30.896370+0000","last_active":"2024-02-10T08:49:19.512143+0000","last_peered":"2024-02-10T08:49:19.512143+0000","last_clean":"2024-02-10T08:49:19.512143+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:49:19.512143+0000","last_undegraded":"2024-02-10T08:49:19.512143+0000","last_fullsized":"2024-02-10T08:49:19.512143+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.896306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.896306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":777,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:23.180885+0000","last_change":"2024-02-10T08:49:23.180885+0000","last_active":"2024-02-10T08:49:23.180885+0000","last_peered":"2024-02-10T08:49:23.180885+0000","last_clean":"2024-02-10T08:49:23.180885+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:49:23.180885+0000","last_undegraded":"2024-02-10T08:49:23.180885+0000","last_fullsized":"2024-02-10T08:49:23.180885+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.180813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.180813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":754,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959238+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:49:19.959238+0000","last_peered":"2024-02-10T08:49:19.959238+0000","last_clean":"2024-02-10T08:49:19.959238+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:49:19.959238+0000","last_undegraded":"2024-02-10T08:49:19.959238+0000","last_fullsized":"2024-02-10T08:49:19.959238+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":763,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.688015+0000","last_change":"2024-02-10T08:48:48.079333+0000","last_active":"2024-02-10T08:49:19.688015+0000","last_peered":"2024-02-10T08:49:19.688015+0000","last_clean":"2024-02-10T08:49:19.688015+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:49:19.688015+0000","last_undegraded":"2024-02-10T08:49:19.688015+0000","last_fullsized":"2024-02-10T08:49:19.688015+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.079269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.079269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":737,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684576+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:49:19.684576+0000","last_peered":"2024-02-10T08:49:19.684576+0000","last_clean":"2024-02-10T08:49:19.684576+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:49:19.684576+0000","last_undegraded":"2024-02-10T08:49:19.684576+0000","last_fullsized":"2024-02-10T08:49:19.684576+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":798,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.195064+0000","last_change":"2024-02-10T08:47:26.043612+0000","last_active":"2024-02-10T08:49:21.195064+0000","last_peered":"2024-02-10T08:49:21.195064+0000","last_clean":"2024-02-10T08:49:21.195064+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:49:21.195064+0000","last_undegraded":"2024-02-10T08:49:21.195064+0000","last_fullsized":"2024-02-10T08:49:21.195064+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.043548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.043548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":100,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.175041+0000","last_change":"2024-02-10T08:47:53.993563+0000","last_active":"2024-02-10T08:49:26.175041+0000","last_peered":"2024-02-10T08:49:26.175041+0000","last_clean":"2024-02-10T08:49:26.175041+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:49:26.175041+0000","last_undegraded":"2024-02-10T08:49:26.175041+0000","last_fullsized":"2024-02-10T08:49:26.175041+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:47:30.847592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.847592+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":786,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.193638+0000","last_change":"2024-02-10T08:47:18.043795+0000","last_active":"2024-02-10T08:49:21.193638+0000","last_peered":"2024-02-10T08:49:21.193638+0000","last_clean":"2024-02-10T08:49:21.193638+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:49:21.193638+0000","last_undegraded":"2024-02-10T08:49:21.193638+0000","last_fullsized":"2024-02-10T08:49:21.193638+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:18.043731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:18.043731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":189,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.173416+0000","last_change":"2024-02-10T08:47:49.933829+0000","last_active":"2024-02-10T08:49:26.173416+0000","last_peered":"2024-02-10T08:49:26.173416+0000","last_clean":"2024-02-10T08:49:26.173416+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:49:26.173416+0000","last_undegraded":"2024-02-10T08:49:26.173416+0000","last_fullsized":"2024-02-10T08:49:26.173416+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.043513+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":194,"num_read_kb":189,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":336,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512757+0000","last_change":"2024-02-10T08:48:47.787415+0000","last_active":"2024-02-10T08:49:19.512757+0000","last_peered":"2024-02-10T08:49:19.512757+0000","last_clean":"2024-02-10T08:49:19.512757+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:49:19.512757+0000","last_undegraded":"2024-02-10T08:49:19.512757+0000","last_fullsized":"2024-02-10T08:49:19.512757+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.787351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":88,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174496+0000","last_change":"2024-02-10T08:47:49.236998+0000","last_active":"2024-02-10T08:49:26.174496+0000","last_peered":"2024-02-10T08:49:26.174496+0000","last_clean":"2024-02-10T08:49:26.174496+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:49:26.174496+0000","last_undegraded":"2024-02-10T08:49:26.174496+0000","last_fullsized":"2024-02-10T08:49:26.174496+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.556602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":203,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545100+0000","last_change":"2024-02-10T08:47:59.541068+0000","last_active":"2024-02-10T08:49:19.545100+0000","last_peered":"2024-02-10T08:49:19.545100+0000","last_clean":"2024-02-10T08:49:19.545100+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:49:19.545100+0000","last_undegraded":"2024-02-10T08:49:19.545100+0000","last_fullsized":"2024-02-10T08:49:19.545100+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:59.541002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:47:59.541002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"711'43","reported_seq":134,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174324+0000","last_change":"2024-02-10T08:48:28.753355+0000","last_active":"2024-02-10T08:49:26.174324+0000","last_peered":"2024-02-10T08:49:26.174324+0000","last_clean":"2024-02-10T08:49:26.174324+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:49:26.174324+0000","last_undegraded":"2024-02-10T08:49:26.174324+0000","last_fullsized":"2024-02-10T08:49:26.174324+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'43","last_scrub_stamp":"2024-02-10T08:48:28.753288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.753288+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":777,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.686897+0000","last_change":"2024-02-10T08:48:31.134699+0000","last_active":"2024-02-10T08:49:19.686897+0000","last_peered":"2024-02-10T08:49:19.686897+0000","last_clean":"2024-02-10T08:49:19.686897+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:49:19.686897+0000","last_undegraded":"2024-02-10T08:49:19.686897+0000","last_fullsized":"2024-02-10T08:49:19.686897+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.134575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.134575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":781,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:24.384243+0000","last_change":"2024-02-10T08:49:24.384243+0000","last_active":"2024-02-10T08:49:24.384243+0000","last_peered":"2024-02-10T08:49:24.384243+0000","last_clean":"2024-02-10T08:49:24.384243+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:49:24.384243+0000","last_undegraded":"2024-02-10T08:49:24.384243+0000","last_fullsized":"2024-02-10T08:49:24.384243+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.384180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.384180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":750,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.192890+0000","last_change":"2024-02-10T08:48:54.135964+0000","last_active":"2024-02-10T08:49:21.192890+0000","last_peered":"2024-02-10T08:49:21.192890+0000","last_clean":"2024-02-10T08:49:21.192890+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:49:21.192890+0000","last_undegraded":"2024-02-10T08:49:21.192890+0000","last_fullsized":"2024-02-10T08:49:21.192890+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.135900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.135900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":216,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.674188+0000","last_change":"2024-02-10T08:49:21.583412+0000","last_active":"2024-02-10T08:49:21.674188+0000","last_peered":"2024-02-10T08:49:21.674188+0000","last_clean":"2024-02-10T08:49:21.674188+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:49:21.674188+0000","last_undegraded":"2024-02-10T08:49:21.674188+0000","last_fullsized":"2024-02-10T08:49:21.674188+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:49:21.583343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.583343+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":105,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.173656+0000","last_change":"2024-02-10T08:48:42.684268+0000","last_active":"2024-02-10T08:49:26.173656+0000","last_peered":"2024-02-10T08:49:26.173656+0000","last_clean":"2024-02-10T08:49:26.173656+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:49:26.173656+0000","last_undegraded":"2024-02-10T08:49:26.173656+0000","last_fullsized":"2024-02-10T08:49:26.173656+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:48:42.684181+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.684181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174511+0000","last_change":"2024-02-10T08:47:49.256946+0000","last_active":"2024-02-10T08:49:26.174511+0000","last_peered":"2024-02-10T08:49:26.174511+0000","last_clean":"2024-02-10T08:49:26.174511+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:49:26.174511+0000","last_undegraded":"2024-02-10T08:49:26.174511+0000","last_fullsized":"2024-02-10T08:49:26.174511+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:26.363337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.363337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":174,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544066+0000","last_change":"2024-02-10T08:48:33.412887+0000","last_active":"2024-02-10T08:49:19.544066+0000","last_peered":"2024-02-10T08:49:19.544066+0000","last_clean":"2024-02-10T08:49:19.544066+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:49:19.544066+0000","last_undegraded":"2024-02-10T08:49:19.544066+0000","last_fullsized":"2024-02-10T08:49:19.544066+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.412775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.412775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":721,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959410+0000","last_change":"2024-02-10T08:48:50.100054+0000","last_active":"2024-02-10T08:49:19.959410+0000","last_peered":"2024-02-10T08:49:19.959410+0000","last_clean":"2024-02-10T08:49:19.959410+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:49:19.959410+0000","last_undegraded":"2024-02-10T08:49:19.959410+0000","last_fullsized":"2024-02-10T08:49:19.959410+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.099951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.099951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":821,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.193216+0000","last_change":"2024-02-10T08:48:41.070217+0000","last_active":"2024-02-10T08:49:21.193216+0000","last_peered":"2024-02-10T08:49:21.193216+0000","last_clean":"2024-02-10T08:49:21.193216+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:49:21.193216+0000","last_undegraded":"2024-02-10T08:49:21.193216+0000","last_fullsized":"2024-02-10T08:49:21.193216+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.070126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.070126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":694,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213844+0000","last_change":"2024-02-10T08:48:31.973099+0000","last_active":"2024-02-10T08:49:26.213844+0000","last_peered":"2024-02-10T08:49:26.213844+0000","last_clean":"2024-02-10T08:49:26.213844+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:49:26.213844+0000","last_undegraded":"2024-02-10T08:49:26.213844+0000","last_fullsized":"2024-02-10T08:49:26.213844+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:48:31.973020+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.973020+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":659,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213029+0000","last_change":"2024-02-10T08:47:17.985317+0000","last_active":"2024-02-10T08:49:26.213029+0000","last_peered":"2024-02-10T08:49:26.213029+0000","last_clean":"2024-02-10T08:49:26.213029+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:49:26.213029+0000","last_undegraded":"2024-02-10T08:49:26.213029+0000","last_fullsized":"2024-02-10T08:49:26.213029+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:47:17.985235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.985235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":789,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959285+0000","last_change":"2024-02-10T08:48:38.003058+0000","last_active":"2024-02-10T08:49:19.959285+0000","last_peered":"2024-02-10T08:49:19.959285+0000","last_clean":"2024-02-10T08:49:19.959285+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:49:19.959285+0000","last_undegraded":"2024-02-10T08:49:19.959285+0000","last_fullsized":"2024-02-10T08:49:19.959285+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.002996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.002996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":106,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687189+0000","last_change":"2024-02-10T08:48:33.158780+0000","last_active":"2024-02-10T08:49:19.687189+0000","last_peered":"2024-02-10T08:49:19.687189+0000","last_clean":"2024-02-10T08:49:19.687189+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:49:19.687189+0000","last_undegraded":"2024-02-10T08:49:19.687189+0000","last_fullsized":"2024-02-10T08:49:19.687189+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.158671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.158671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":183,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545176+0000","last_change":"2024-02-10T08:48:54.417289+0000","last_active":"2024-02-10T08:49:19.545176+0000","last_peered":"2024-02-10T08:49:19.545176+0000","last_clean":"2024-02-10T08:49:19.545176+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:49:19.545176+0000","last_undegraded":"2024-02-10T08:49:19.545176+0000","last_fullsized":"2024-02-10T08:49:19.545176+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:48:54.417173+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.417173+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":808,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.193146+0000","last_change":"2024-02-10T08:48:50.118042+0000","last_active":"2024-02-10T08:49:21.193146+0000","last_peered":"2024-02-10T08:49:21.193146+0000","last_clean":"2024-02-10T08:49:21.193146+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:49:21.193146+0000","last_undegraded":"2024-02-10T08:49:21.193146+0000","last_fullsized":"2024-02-10T08:49:21.193146+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.117976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":793,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:24.116710+0000","last_change":"2024-02-10T08:49:24.116688+0000","last_active":"2024-02-10T08:49:24.116710+0000","last_peered":"2024-02-10T08:49:24.116710+0000","last_clean":"2024-02-10T08:49:24.116710+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:49:24.116710+0000","last_undegraded":"2024-02-10T08:49:24.116710+0000","last_fullsized":"2024-02-10T08:49:24.116710+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.116601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.116601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":587,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214128+0000","last_change":"2024-02-10T08:48:41.943374+0000","last_active":"2024-02-10T08:49:26.214128+0000","last_peered":"2024-02-10T08:49:26.214128+0000","last_clean":"2024-02-10T08:49:26.214128+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:49:26.214128+0000","last_undegraded":"2024-02-10T08:49:26.214128+0000","last_fullsized":"2024-02-10T08:49:26.214128+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.943287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.943287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":838,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959641+0000","last_change":"2024-02-10T08:47:49.978588+0000","last_active":"2024-02-10T08:49:19.959641+0000","last_peered":"2024-02-10T08:49:19.959641+0000","last_clean":"2024-02-10T08:49:19.959641+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:49:19.959641+0000","last_undegraded":"2024-02-10T08:49:19.959641+0000","last_fullsized":"2024-02-10T08:49:19.959641+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:46:49.515525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:49.515525+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":790,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685169+0000","last_change":"2024-02-10T08:48:36.966625+0000","last_active":"2024-02-10T08:49:19.685169+0000","last_peered":"2024-02-10T08:49:19.685169+0000","last_clean":"2024-02-10T08:49:19.685169+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:49:19.685169+0000","last_undegraded":"2024-02-10T08:49:19.685169+0000","last_fullsized":"2024-02-10T08:49:19.685169+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.966559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.966559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":347,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512649+0000","last_change":"2024-02-10T08:48:28.810015+0000","last_active":"2024-02-10T08:49:19.512649+0000","last_peered":"2024-02-10T08:49:19.512649+0000","last_clean":"2024-02-10T08:49:19.512649+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:49:19.512649+0000","last_undegraded":"2024-02-10T08:49:19.512649+0000","last_fullsized":"2024-02-10T08:49:19.512649+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:28.809930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.809930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":714,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687177+0000","last_change":"2024-02-10T08:48:47.081659+0000","last_active":"2024-02-10T08:49:19.687177+0000","last_peered":"2024-02-10T08:49:19.687177+0000","last_clean":"2024-02-10T08:49:19.687177+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:49:19.687177+0000","last_undegraded":"2024-02-10T08:49:19.687177+0000","last_fullsized":"2024-02-10T08:49:19.687177+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.081542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":871,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:20.324286+0000","last_change":"2024-02-10T08:49:20.324246+0000","last_active":"2024-02-10T08:49:20.324286+0000","last_peered":"2024-02-10T08:49:20.324286+0000","last_clean":"2024-02-10T08:49:20.324286+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:49:20.324286+0000","last_undegraded":"2024-02-10T08:49:20.324286+0000","last_fullsized":"2024-02-10T08:49:20.324286+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:20.324128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.324128+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":632,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959121+0000","last_change":"2024-02-10T08:48:45.089054+0000","last_active":"2024-02-10T08:49:19.959121+0000","last_peered":"2024-02-10T08:49:19.959121+0000","last_clean":"2024-02-10T08:49:19.959121+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:49:19.959121+0000","last_undegraded":"2024-02-10T08:49:19.959121+0000","last_fullsized":"2024-02-10T08:49:19.959121+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.088965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.088965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1004,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.674591+0000","last_change":"2024-02-10T08:47:49.363312+0000","last_active":"2024-02-10T08:49:21.674591+0000","last_peered":"2024-02-10T08:49:21.674591+0000","last_clean":"2024-02-10T08:49:21.674591+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:49:21.674591+0000","last_undegraded":"2024-02-10T08:49:21.674591+0000","last_fullsized":"2024-02-10T08:49:21.674591+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.886696+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545123+0000","last_change":"2024-02-10T08:48:38.471156+0000","last_active":"2024-02-10T08:49:19.545123+0000","last_peered":"2024-02-10T08:49:19.545123+0000","last_clean":"2024-02-10T08:49:19.545123+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:49:19.545123+0000","last_undegraded":"2024-02-10T08:49:19.545123+0000","last_fullsized":"2024-02-10T08:49:19.545123+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.471058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.471058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":174,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544086+0000","last_change":"2024-02-10T08:48:30.433484+0000","last_active":"2024-02-10T08:49:19.544086+0000","last_peered":"2024-02-10T08:49:19.544086+0000","last_clean":"2024-02-10T08:49:19.544086+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:49:19.544086+0000","last_undegraded":"2024-02-10T08:49:19.544086+0000","last_fullsized":"2024-02-10T08:49:19.544086+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.433421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.433421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"473'15","reported_seq":815,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687397+0000","last_change":"2024-02-10T08:47:24.808523+0000","last_active":"2024-02-10T08:49:19.687397+0000","last_peered":"2024-02-10T08:49:19.687397+0000","last_clean":"2024-02-10T08:49:19.687397+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:49:19.687397+0000","last_undegraded":"2024-02-10T08:49:19.687397+0000","last_fullsized":"2024-02-10T08:49:19.687397+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:24.808424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.808424+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":532,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.212960+0000","last_change":"2024-02-10T08:47:55.031372+0000","last_active":"2024-02-10T08:49:26.212960+0000","last_peered":"2024-02-10T08:49:26.212960+0000","last_clean":"2024-02-10T08:49:26.212960+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:49:26.212960+0000","last_undegraded":"2024-02-10T08:49:26.212960+0000","last_fullsized":"2024-02-10T08:49:26.212960+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.031301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.031301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":983,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:22.354177+0000","last_change":"2024-02-10T08:49:22.354177+0000","last_active":"2024-02-10T08:49:22.354177+0000","last_peered":"2024-02-10T08:49:22.354177+0000","last_clean":"2024-02-10T08:49:22.354177+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:49:22.354177+0000","last_undegraded":"2024-02-10T08:49:22.354177+0000","last_fullsized":"2024-02-10T08:49:22.354177+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:49:22.354105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.354105+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":594,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214439+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:49:26.214439+0000","last_peered":"2024-02-10T08:49:26.214439+0000","last_clean":"2024-02-10T08:49:26.214439+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:49:26.214439+0000","last_undegraded":"2024-02-10T08:49:26.214439+0000","last_fullsized":"2024-02-10T08:49:26.214439+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":748,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.193437+0000","last_change":"2024-02-10T08:47:49.268597+0000","last_active":"2024-02-10T08:49:21.193437+0000","last_peered":"2024-02-10T08:49:21.193437+0000","last_clean":"2024-02-10T08:49:21.193437+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:49:21.193437+0000","last_undegraded":"2024-02-10T08:49:21.193437+0000","last_fullsized":"2024-02-10T08:49:21.193437+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:47:19.053931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.053931+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":98,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174672+0000","last_change":"2024-02-10T08:47:49.932955+0000","last_active":"2024-02-10T08:49:26.174672+0000","last_peered":"2024-02-10T08:49:26.174672+0000","last_clean":"2024-02-10T08:49:26.174672+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:49:26.174672+0000","last_undegraded":"2024-02-10T08:49:26.174672+0000","last_fullsized":"2024-02-10T08:49:26.174672+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.671541+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":767,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685109+0000","last_change":"2024-02-10T08:48:45.997431+0000","last_active":"2024-02-10T08:49:19.685109+0000","last_peered":"2024-02-10T08:49:19.685109+0000","last_clean":"2024-02-10T08:49:19.685109+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:49:19.685109+0000","last_undegraded":"2024-02-10T08:49:19.685109+0000","last_fullsized":"2024-02-10T08:49:19.685109+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.997364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.997364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":791,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959241+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:49:19.959241+0000","last_peered":"2024-02-10T08:49:19.959241+0000","last_clean":"2024-02-10T08:49:19.959241+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:49:19.959241+0000","last_undegraded":"2024-02-10T08:49:19.959241+0000","last_fullsized":"2024-02-10T08:49:19.959241+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":192,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:20.566585+0000","last_change":"2024-02-10T08:49:20.566561+0000","last_active":"2024-02-10T08:49:20.566585+0000","last_peered":"2024-02-10T08:49:20.566585+0000","last_clean":"2024-02-10T08:49:20.566585+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:49:20.566585+0000","last_undegraded":"2024-02-10T08:49:20.566585+0000","last_fullsized":"2024-02-10T08:49:20.566585+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.566497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.566497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":725,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685403+0000","last_change":"2024-02-10T08:47:25.754309+0000","last_active":"2024-02-10T08:49:19.685403+0000","last_peered":"2024-02-10T08:49:19.685403+0000","last_clean":"2024-02-10T08:49:19.685403+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:49:19.685403+0000","last_undegraded":"2024-02-10T08:49:19.685403+0000","last_fullsized":"2024-02-10T08:49:19.685403+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:25.754229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:25.754229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":685,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213771+0000","last_change":"2024-02-10T08:47:49.793777+0000","last_active":"2024-02-10T08:49:26.213771+0000","last_peered":"2024-02-10T08:49:26.213771+0000","last_clean":"2024-02-10T08:49:26.213771+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:49:26.213771+0000","last_undegraded":"2024-02-10T08:49:26.213771+0000","last_fullsized":"2024-02-10T08:49:26.213771+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:19.946402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.946402+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174282+0000","last_change":"2024-02-10T08:47:56.921391+0000","last_active":"2024-02-10T08:49:26.174282+0000","last_peered":"2024-02-10T08:49:26.174282+0000","last_clean":"2024-02-10T08:49:26.174282+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:49:26.174282+0000","last_undegraded":"2024-02-10T08:49:26.174282+0000","last_fullsized":"2024-02-10T08:49:26.174282+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.921291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.921291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174743+0000","last_change":"2024-02-10T08:47:58.934691+0000","last_active":"2024-02-10T08:49:26.174743+0000","last_peered":"2024-02-10T08:49:26.174743+0000","last_clean":"2024-02-10T08:49:26.174743+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:49:26.174743+0000","last_undegraded":"2024-02-10T08:49:26.174743+0000","last_fullsized":"2024-02-10T08:49:26.174743+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:58.934594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.934594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":771,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958696+0000","last_change":"2024-02-10T08:48:40.049047+0000","last_active":"2024-02-10T08:49:19.958696+0000","last_peered":"2024-02-10T08:49:19.958696+0000","last_clean":"2024-02-10T08:49:19.958696+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:49:19.958696+0000","last_undegraded":"2024-02-10T08:49:19.958696+0000","last_fullsized":"2024-02-10T08:49:19.958696+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.048948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.048948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":722,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958397+0000","last_change":"2024-02-10T08:46:41.501224+0000","last_active":"2024-02-10T08:49:19.958397+0000","last_peered":"2024-02-10T08:49:19.958397+0000","last_clean":"2024-02-10T08:49:19.958397+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:49:19.958397+0000","last_undegraded":"2024-02-10T08:49:19.958397+0000","last_fullsized":"2024-02-10T08:49:19.958397+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:41.501157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:41.501157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":794,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958532+0000","last_change":"2024-02-10T08:47:49.262275+0000","last_active":"2024-02-10T08:49:19.958532+0000","last_peered":"2024-02-10T08:49:19.958532+0000","last_clean":"2024-02-10T08:49:19.958532+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:49:19.958532+0000","last_undegraded":"2024-02-10T08:49:19.958532+0000","last_fullsized":"2024-02-10T08:49:19.958532+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:46:48.511488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:48.511488+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":763,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687570+0000","last_change":"2024-02-10T08:48:36.153670+0000","last_active":"2024-02-10T08:49:19.687570+0000","last_peered":"2024-02-10T08:49:19.687570+0000","last_clean":"2024-02-10T08:49:19.687570+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:49:19.687570+0000","last_undegraded":"2024-02-10T08:49:19.687570+0000","last_fullsized":"2024-02-10T08:49:19.687570+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.153585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.153585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":788,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684910+0000","last_change":"2024-02-10T08:48:32.816285+0000","last_active":"2024-02-10T08:49:19.684910+0000","last_peered":"2024-02-10T08:49:19.684910+0000","last_clean":"2024-02-10T08:49:19.684910+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:49:19.684910+0000","last_undegraded":"2024-02-10T08:49:19.684910+0000","last_fullsized":"2024-02-10T08:49:19.684910+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.816211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.816211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":347,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512455+0000","last_change":"2024-02-10T08:47:56.600567+0000","last_active":"2024-02-10T08:49:19.512455+0000","last_peered":"2024-02-10T08:49:19.512455+0000","last_clean":"2024-02-10T08:49:19.512455+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:49:19.512455+0000","last_undegraded":"2024-02-10T08:49:19.512455+0000","last_fullsized":"2024-02-10T08:49:19.512455+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.600507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.600507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":734,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959574+0000","last_change":"2024-02-10T08:47:49.232286+0000","last_active":"2024-02-10T08:49:19.959574+0000","last_peered":"2024-02-10T08:49:19.959574+0000","last_clean":"2024-02-10T08:49:19.959574+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:49:19.959574+0000","last_undegraded":"2024-02-10T08:49:19.959574+0000","last_fullsized":"2024-02-10T08:49:19.959574+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:44.529230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:46:44.529230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":825,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958905+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:49:19.958905+0000","last_peered":"2024-02-10T08:49:19.958905+0000","last_clean":"2024-02-10T08:49:19.958905+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:49:19.958905+0000","last_undegraded":"2024-02-10T08:49:19.958905+0000","last_fullsized":"2024-02-10T08:49:19.958905+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":767,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685401+0000","last_change":"2024-02-10T08:48:53.059434+0000","last_active":"2024-02-10T08:49:19.685401+0000","last_peered":"2024-02-10T08:49:19.685401+0000","last_clean":"2024-02-10T08:49:19.685401+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:49:19.685401+0000","last_undegraded":"2024-02-10T08:49:19.685401+0000","last_fullsized":"2024-02-10T08:49:19.685401+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:53.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:53.059137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":347,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512018+0000","last_change":"2024-02-10T08:48:45.871861+0000","last_active":"2024-02-10T08:49:19.512018+0000","last_peered":"2024-02-10T08:49:19.512018+0000","last_clean":"2024-02-10T08:49:19.512018+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:49:19.512018+0000","last_undegraded":"2024-02-10T08:49:19.512018+0000","last_fullsized":"2024-02-10T08:49:19.512018+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.871773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":771,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:23.340760+0000","last_change":"2024-02-10T08:49:23.340760+0000","last_active":"2024-02-10T08:49:23.340760+0000","last_peered":"2024-02-10T08:49:23.340760+0000","last_clean":"2024-02-10T08:49:23.340760+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:49:23.340760+0000","last_undegraded":"2024-02-10T08:49:23.340760+0000","last_fullsized":"2024-02-10T08:49:23.340760+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.340696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.340696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":169,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545027+0000","last_change":"2024-02-10T08:47:29.348776+0000","last_active":"2024-02-10T08:49:19.545027+0000","last_peered":"2024-02-10T08:49:19.545027+0000","last_clean":"2024-02-10T08:49:19.545027+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:49:19.545027+0000","last_undegraded":"2024-02-10T08:49:19.545027+0000","last_fullsized":"2024-02-10T08:49:19.545027+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:47:29.348714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":181,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545055+0000","last_change":"2024-02-10T08:47:58.529546+0000","last_active":"2024-02-10T08:49:19.545055+0000","last_peered":"2024-02-10T08:49:19.545055+0000","last_clean":"2024-02-10T08:49:19.545055+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:49:19.545055+0000","last_undegraded":"2024-02-10T08:49:19.545055+0000","last_fullsized":"2024-02-10T08:49:19.545055+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:47:58.529476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.529476+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":801,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684592+0000","last_change":"2024-02-10T08:48:54.005915+0000","last_active":"2024-02-10T08:49:19.684592+0000","last_peered":"2024-02-10T08:49:19.684592+0000","last_clean":"2024-02-10T08:49:19.684592+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:49:19.684592+0000","last_undegraded":"2024-02-10T08:49:19.684592+0000","last_fullsized":"2024-02-10T08:49:19.684592+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.005830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.005830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":347,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512753+0000","last_change":"2024-02-10T08:48:50.762747+0000","last_active":"2024-02-10T08:49:19.512753+0000","last_peered":"2024-02-10T08:49:19.512753+0000","last_clean":"2024-02-10T08:49:19.512753+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:49:19.512753+0000","last_undegraded":"2024-02-10T08:49:19.512753+0000","last_fullsized":"2024-02-10T08:49:19.512753+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.762672+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":745,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194409+0000","last_change":"2024-02-10T08:47:49.229890+0000","last_active":"2024-02-10T08:49:21.194409+0000","last_peered":"2024-02-10T08:49:21.194409+0000","last_clean":"2024-02-10T08:49:21.194409+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:49:21.194409+0000","last_undegraded":"2024-02-10T08:49:21.194409+0000","last_fullsized":"2024-02-10T08:49:21.194409+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:15.149802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:15.149802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":838,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685517+0000","last_change":"2024-02-10T08:47:57.663357+0000","last_active":"2024-02-10T08:49:19.685517+0000","last_peered":"2024-02-10T08:49:19.685517+0000","last_clean":"2024-02-10T08:49:19.685517+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:49:19.685517+0000","last_undegraded":"2024-02-10T08:49:19.685517+0000","last_fullsized":"2024-02-10T08:49:19.685517+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:57.663290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.663290+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":777,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958769+0000","last_change":"2024-02-10T08:48:41.013610+0000","last_active":"2024-02-10T08:49:19.958769+0000","last_peered":"2024-02-10T08:49:19.958769+0000","last_clean":"2024-02-10T08:49:19.958769+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:49:19.958769+0000","last_undegraded":"2024-02-10T08:49:19.958769+0000","last_fullsized":"2024-02-10T08:49:19.958769+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.013543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.013543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174240+0000","last_change":"2024-02-10T08:49:21.886466+0000","last_active":"2024-02-10T08:49:26.174240+0000","last_peered":"2024-02-10T08:49:26.174240+0000","last_clean":"2024-02-10T08:49:26.174240+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:49:26.174240+0000","last_undegraded":"2024-02-10T08:49:26.174240+0000","last_fullsized":"2024-02-10T08:49:26.174240+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.886401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.886401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":93,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.174574+0000","last_change":"2024-02-10T08:48:49.706321+0000","last_active":"2024-02-10T08:49:26.174574+0000","last_peered":"2024-02-10T08:49:26.174574+0000","last_clean":"2024-02-10T08:49:26.174574+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:49:26.174574+0000","last_undegraded":"2024-02-10T08:49:26.174574+0000","last_fullsized":"2024-02-10T08:49:26.174574+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:49.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:48:49.706258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":86,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.195375+0000","last_change":"2024-02-10T08:48:52.097227+0000","last_active":"2024-02-10T08:49:21.195375+0000","last_peered":"2024-02-10T08:49:21.195375+0000","last_clean":"2024-02-10T08:49:21.195375+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:49:21.195375+0000","last_undegraded":"2024-02-10T08:49:21.195375+0000","last_fullsized":"2024-02-10T08:49:21.195375+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.097161+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":790,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685690+0000","last_change":"2024-02-10T08:48:41.102099+0000","last_active":"2024-02-10T08:49:19.685690+0000","last_peered":"2024-02-10T08:49:19.685690+0000","last_clean":"2024-02-10T08:49:19.685690+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:49:19.685690+0000","last_undegraded":"2024-02-10T08:49:19.685690+0000","last_fullsized":"2024-02-10T08:49:19.685690+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.102021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.102021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":600,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214176+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:49:26.214176+0000","last_peered":"2024-02-10T08:49:26.214176+0000","last_clean":"2024-02-10T08:49:26.214176+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:49:26.214176+0000","last_undegraded":"2024-02-10T08:49:26.214176+0000","last_fullsized":"2024-02-10T08:49:26.214176+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":187,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544646+0000","last_change":"2024-02-10T08:47:49.241622+0000","last_active":"2024-02-10T08:49:19.544646+0000","last_peered":"2024-02-10T08:49:19.544646+0000","last_clean":"2024-02-10T08:49:19.544646+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:49:19.544646+0000","last_undegraded":"2024-02-10T08:49:19.544646+0000","last_fullsized":"2024-02-10T08:49:19.544646+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:27.336914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:27.336914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":191,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958907+0000","last_change":"2024-02-10T08:47:49.265858+0000","last_active":"2024-02-10T08:49:19.958907+0000","last_peered":"2024-02-10T08:49:19.958907+0000","last_clean":"2024-02-10T08:49:19.958907+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:49:19.958907+0000","last_undegraded":"2024-02-10T08:49:19.958907+0000","last_fullsized":"2024-02-10T08:49:19.958907+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:46:43.848897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.848897+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":777,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959201+0000","last_change":"2024-02-10T08:48:00.922194+0000","last_active":"2024-02-10T08:49:19.959201+0000","last_peered":"2024-02-10T08:49:19.959201+0000","last_clean":"2024-02-10T08:49:19.959201+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:49:19.959201+0000","last_undegraded":"2024-02-10T08:49:19.959201+0000","last_fullsized":"2024-02-10T08:49:19.959201+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:00.922104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:00.922104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":753,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959734+0000","last_change":"2024-02-10T08:48:43.032173+0000","last_active":"2024-02-10T08:49:19.959734+0000","last_peered":"2024-02-10T08:49:19.959734+0000","last_clean":"2024-02-10T08:49:19.959734+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:49:19.959734+0000","last_undegraded":"2024-02-10T08:49:19.959734+0000","last_fullsized":"2024-02-10T08:49:19.959734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.032094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.032094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":102,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.173625+0000","last_change":"2024-02-10T08:49:25.890744+0000","last_active":"2024-02-10T08:49:26.173625+0000","last_peered":"2024-02-10T08:49:26.173625+0000","last_clean":"2024-02-10T08:49:26.173625+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:49:26.173625+0000","last_undegraded":"2024-02-10T08:49:26.173625+0000","last_fullsized":"2024-02-10T08:49:26.173625+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.890678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.890678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":657,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.212930+0000","last_change":"2024-02-10T08:48:32.990056+0000","last_active":"2024-02-10T08:49:26.212930+0000","last_peered":"2024-02-10T08:49:26.212930+0000","last_clean":"2024-02-10T08:49:26.212930+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:49:26.212930+0000","last_undegraded":"2024-02-10T08:49:26.212930+0000","last_fullsized":"2024-02-10T08:49:26.212930+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.989985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.989985+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":344,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.512732+0000","last_change":"2024-02-10T08:48:55.711815+0000","last_active":"2024-02-10T08:49:19.512732+0000","last_peered":"2024-02-10T08:49:19.512732+0000","last_clean":"2024-02-10T08:49:19.512732+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:49:19.512732+0000","last_undegraded":"2024-02-10T08:49:19.512732+0000","last_fullsized":"2024-02-10T08:49:19.512732+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.711733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.711733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":187,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.544674+0000","last_change":"2024-02-10T08:47:49.241831+0000","last_active":"2024-02-10T08:49:19.544674+0000","last_peered":"2024-02-10T08:49:19.544674+0000","last_clean":"2024-02-10T08:49:19.544674+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:49:19.544674+0000","last_undegraded":"2024-02-10T08:49:19.544674+0000","last_fullsized":"2024-02-10T08:49:19.544674+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:24.365335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:24.365335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":744,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-10T08:49:21.194927+0000","last_change":"2024-02-10T08:49:20.063342+0000","last_active":"2024-02-10T08:49:21.194927+0000","last_peered":"2024-02-10T08:49:21.194927+0000","last_clean":"2024-02-10T08:49:21.194927+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:49:21.194927+0000","last_undegraded":"2024-02-10T08:49:21.194927+0000","last_fullsized":"2024-02-10T08:49:21.194927+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.063276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.063276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":660,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.214287+0000","last_change":"2024-02-10T08:48:44.939276+0000","last_active":"2024-02-10T08:49:26.214287+0000","last_peered":"2024-02-10T08:49:26.214287+0000","last_clean":"2024-02-10T08:49:26.214287+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:49:26.214287+0000","last_undegraded":"2024-02-10T08:49:26.214287+0000","last_fullsized":"2024-02-10T08:49:26.214287+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:48:44.939188+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.939188+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":289,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958909+0000","last_change":"2024-02-10T08:47:57.938158+0000","last_active":"2024-02-10T08:49:19.958909+0000","last_peered":"2024-02-10T08:49:19.958909+0000","last_clean":"2024-02-10T08:49:19.958909+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:49:19.958909+0000","last_undegraded":"2024-02-10T08:49:19.958909+0000","last_fullsized":"2024-02-10T08:49:19.958909+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.938096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.938096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":766,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.684684+0000","last_change":"2024-02-10T08:47:56.907151+0000","last_active":"2024-02-10T08:49:19.684684+0000","last_peered":"2024-02-10T08:49:19.684684+0000","last_clean":"2024-02-10T08:49:19.684684+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:49:19.684684+0000","last_undegraded":"2024-02-10T08:49:19.684684+0000","last_fullsized":"2024-02-10T08:49:19.684684+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.907094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.907094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":173,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545190+0000","last_change":"2024-02-10T08:48:52.471904+0000","last_active":"2024-02-10T08:49:19.545190+0000","last_peered":"2024-02-10T08:49:19.545190+0000","last_clean":"2024-02-10T08:49:19.545190+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:49:19.545190+0000","last_undegraded":"2024-02-10T08:49:19.545190+0000","last_fullsized":"2024-02-10T08:49:19.545190+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:52.471831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.471831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":692,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213445+0000","last_change":"2024-02-10T08:47:50.341317+0000","last_active":"2024-02-10T08:49:26.213445+0000","last_peered":"2024-02-10T08:49:26.213445+0000","last_clean":"2024-02-10T08:49:26.213445+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:49:26.213445+0000","last_undegraded":"2024-02-10T08:49:26.213445+0000","last_fullsized":"2024-02-10T08:49:26.213445+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:47:21.938274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:21.938274+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":772,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959568+0000","last_change":"2024-02-10T08:46:43.502472+0000","last_active":"2024-02-10T08:49:19.959568+0000","last_peered":"2024-02-10T08:49:19.959568+0000","last_clean":"2024-02-10T08:49:19.959568+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:49:19.959568+0000","last_undegraded":"2024-02-10T08:49:19.959568+0000","last_fullsized":"2024-02-10T08:49:19.959568+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:46:43.502412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:46:43.502412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":615,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-10T08:49:26.213992+0000","last_change":"2024-02-10T08:48:47.974449+0000","last_active":"2024-02-10T08:49:26.213992+0000","last_peered":"2024-02-10T08:49:26.213992+0000","last_clean":"2024-02-10T08:49:26.213992+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:49:26.213992+0000","last_undegraded":"2024-02-10T08:49:26.213992+0000","last_fullsized":"2024-02-10T08:49:26.213992+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.974390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.974390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":732,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.687032+0000","last_change":"2024-02-10T08:48:42.043077+0000","last_active":"2024-02-10T08:49:19.687032+0000","last_peered":"2024-02-10T08:49:19.687032+0000","last_clean":"2024-02-10T08:49:19.687032+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:49:19.687032+0000","last_undegraded":"2024-02-10T08:49:19.687032+0000","last_fullsized":"2024-02-10T08:49:19.687032+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.043006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.043006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":845,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.958397+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:49:19.958397+0000","last_peered":"2024-02-10T08:49:19.958397+0000","last_clean":"2024-02-10T08:49:19.958397+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:49:19.958397+0000","last_undegraded":"2024-02-10T08:49:19.958397+0000","last_fullsized":"2024-02-10T08:49:19.958397+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":185,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.545207+0000","last_change":"2024-02-10T08:48:02.600767+0000","last_active":"2024-02-10T08:49:19.545207+0000","last_peered":"2024-02-10T08:49:19.545207+0000","last_clean":"2024-02-10T08:49:19.545207+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:49:19.545207+0000","last_undegraded":"2024-02-10T08:49:19.545207+0000","last_fullsized":"2024-02-10T08:49:19.545207+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.600704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.600704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":768,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.959116+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:49:19.959116+0000","last_peered":"2024-02-10T08:49:19.959116+0000","last_clean":"2024-02-10T08:49:19.959116+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:49:19.959116+0000","last_undegraded":"2024-02-10T08:49:19.959116+0000","last_fullsized":"2024-02-10T08:49:19.959116+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":742,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-10T08:49:19.685050+0000","last_change":"2024-02-10T08:48:10.219049+0000","last_active":"2024-02-10T08:49:19.685050+0000","last_peered":"2024-02-10T08:49:19.685050+0000","last_clean":"2024-02-10T08:49:19.685050+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:49:19.685050+0000","last_undegraded":"2024-02-10T08:49:19.685050+0000","last_fullsized":"2024-02-10T08:49:19.685050+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:48:03.702592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":253,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":42,"num_object_clones":1,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":22,"num_read_kb":7,"num_write":62,"num_write_kb":3964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6500352,"data_stored":6500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":10732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":194,"num_read_kb":189,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299588,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311768,"kb_used_data":6352,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433384,"statfs":{"total":95995035648,"available":95675785216,"internally_reserved":0,"allocated":6504448,"data_stored":3464337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54900000000000004,"15min":0.60599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.37,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.76800000000000002,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.32300000000000001,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.73399999999999999,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67700000000000005,"15min":0.61799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.19900000000000001},"max":{"1min":0.80000000000000004,"5min":10.978,"15min":10.978},"last":0.377}]},{"osd":2,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.67100000000000004,"15min":0.60299999999999998},"min":{"1min":0.36899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.81599999999999995,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.625,"15min":0.67200000000000004},"min":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":7.5099999999999998,"15min":19.369},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.56999999999999995,"15min":0.59299999999999997},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":1.0269999999999999,"5min":7.8700000000000001,"15min":22.102},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65100000000000002,"5min":1.581,"15min":1.581},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.68700000000000006,"15min":0.71699999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67300000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.627}]},{"osd":5,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.69199999999999995,"5min":4.46,"15min":19.326000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.58099999999999996,"15min":0.52200000000000002},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.57799999999999996},"min":{"1min":0.36299999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63400000000000001,"5min":4.718,"15min":22.687000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.503},"min":{"1min":0.34399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.621,"5min":5.484,"15min":5.484},"last":0.56699999999999995}]}]},{"osd":6,"up_from":44,"seq":188978561226,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305564,"kb_used_data":8340,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439588,"statfs":{"total":95995035648,"available":95682138112,"internally_reserved":0,"allocated":8540160,"data_stored":5508477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.55600000000000005},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61499999999999999,"5min":3.9399999999999999,"15min":8.5250000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.751,"15min":0.63300000000000001},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.747}]},{"osd":1,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.73399999999999999,"15min":0.65300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.66300000000000003,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.628},{"interface":"front","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":0.69699999999999995,"5min":8.4399999999999995,"15min":18.402999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76800000000000002,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0509999999999999,"5min":10.772,"15min":10.772},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.84099999999999997,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.79500000000000004,"15min":0.70499999999999996},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.64300000000000002,"5min":9.2650000000000006,"15min":16.823},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.66100000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.443,"15min":0.439},"min":{"1min":0.26400000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.70999999999999996,"5min":3.915,"15min":3.915},"last":0.50700000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.371,"5min":0.254,"15min":0.254},"max":{"1min":0.70199999999999996,"5min":7.5650000000000004,"15min":7.5650000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.504,"15min":0.51200000000000001},"min":{"1min":0.40799999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.72199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307732,"kb_used_data":6412,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437420,"statfs":{"total":95995035648,"available":95679918080,"internally_reserved":0,"allocated":6565888,"data_stored":3478528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.5669999999999999,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.39500000000000002,"15min":0.46100000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.71799999999999997,"5min":1.4910000000000001,"15min":16.166},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:48:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.50600000000000001,"15min":0.64700000000000002},"min":{"1min":0.251,"5min":0.193,"15min":0.193},"max":{"1min":0.51500000000000001,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.308},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.46600000000000003,"15min":0.55500000000000005},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.20100000000000001},"max":{"1min":0.54900000000000004,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.29499999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.47799999999999998,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":0.51100000000000001,"5min":4.8940000000000001,"15min":24.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.50900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.80000000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82899999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.374},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.441}]},{"osd":5,"last update":"Sat Feb 10 08:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83799999999999997,"5min":3.496,"15min":9.5600000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.622},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.82299999999999995,"5min":3.4710000000000001,"15min":9.532},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.874,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.496}]},{"osd":7,"last update":"Sat Feb 10 08:48:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.72499999999999998,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":12.147,"15min":12.147},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85499999999999998,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738599,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303600,"kb_used_data":6376,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441552,"statfs":{"total":95995035648,"available":95684149248,"internally_reserved":0,"allocated":6529024,"data_stored":3488260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45700000000000002,"15min":0.40500000000000003},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.156},"max":{"1min":1.4239999999999999,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.74199999999999999,"15min":0.5},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":3.8839999999999999,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.53700000000000003,"15min":0.45700000000000002},"min":{"1min":0.252,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.496,"5min":11.987,"15min":11.987},"last":0.313},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.501,"15min":0.48199999999999998},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53700000000000003,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.502,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.51900000000000002,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.42499999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:48:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.38,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.931,"5min":4.609,"15min":4.609},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55900000000000005,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":1.7769999999999999,"15min":3.4079999999999999},"last":0.80500000000000005}]},{"osd":6,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.33300000000000002},"max":{"1min":3.9609999999999999,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.61799999999999999},"min":{"1min":0.439,"5min":0.317,"15min":0.317},"max":{"1min":0.76600000000000001,"5min":1.849,"15min":3.363},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55400000000000005,"15min":0.65600000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73799999999999999,"5min":0.92600000000000005,"15min":8.4299999999999997},"last":0.70599999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411549,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305536,"kb_used_data":8312,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439616,"statfs":{"total":95995035648,"available":95682166784,"internally_reserved":0,"allocated":8511488,"data_stored":5479073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.64600000000000002,"15min":0.499},"min":{"1min":0.20699999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.64300000000000002,"5min":16.170000000000002,"15min":16.170000000000002},"last":1.2909999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.70399999999999996,"15min":0.56399999999999995},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":0.504,"5min":14.858000000000001,"15min":14.858000000000001},"last":0.379}]},{"osd":1,"last update":"Sat Feb 10 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.63700000000000001,"15min":0.55700000000000005},"min":{"1min":0.253,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":9.6980000000000004,"15min":12.512},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.60399999999999998,"15min":0.53100000000000003},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.56599999999999995,"5min":12.875,"15min":12.875},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.61199999999999999,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50800000000000001,"5min":10.994999999999999,"15min":11.612},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":7.8280000000000003,"15min":11.710000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.42099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.626}]},{"osd":5,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.41099999999999998,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.79100000000000004,"5min":10.34,"15min":10.34},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.71199999999999997,"15min":0.66500000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3380000000000001,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.753,"15min":0.71499999999999997},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.79800000000000004,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.57299999999999995}]},{"osd":7,"last update":"Sat Feb 10 08:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.90500000000000003,"5min":10.539,"15min":10.539},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.78300000000000003,"15min":0.71099999999999997},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.357,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.51300000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182617,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303648,"kb_used_data":6424,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441504,"statfs":{"total":95995035648,"available":95684100096,"internally_reserved":0,"allocated":6578176,"data_stored":3517781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.20699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.497,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.251,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.58599999999999997,"5min":10.484999999999999,"15min":11.808},"last":0.373}]},{"osd":1,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.76900000000000002,"15min":0.67200000000000004},"min":{"1min":0.248,"5min":0.215,"15min":0.184},"max":{"1min":0.53800000000000003,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.26000000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.58799999999999997,"5min":17.343,"15min":17.343},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.754,"15min":0.68000000000000005},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":0.57299999999999995,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.443,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57199999999999995,"5min":5.0259999999999998,"15min":23.294},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:48:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.78400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64100000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55700000000000005,"15min":0.70199999999999996},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.749},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.73999999999999999,"15min":0.70299999999999996},"min":{"1min":0.39000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82999999999999996,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58999999999999997,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79200000000000004,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.73199999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.746},"min":{"1min":0.45300000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79700000000000004,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.66500000000000004}]}]},{"osd":4,"up_from":633,"seq":2718714298405,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309796,"kb_used_data":6428,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93435356,"statfs":{"total":95995035648,"available":95677804544,"internally_reserved":0,"allocated":6582272,"data_stored":3524685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.66400000000000003,"5min":0.754,"15min":0.754},"last":0.52400000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72799999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70899999999999996,"5min":0.746,"15min":0.746},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73899999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.50900000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68700000000000006,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.86099999999999999,"5min":0.87,"15min":0.87},"last":0.53400000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.627}]},{"osd":7,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70999999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.80400000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63900000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790159,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313756,"kb_used_data":8340,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431396,"statfs":{"total":95995035648,"available":95673749504,"internally_reserved":0,"allocated":8540160,"data_stored":5511644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.69599999999999995,"15min":1.0489999999999999},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.216},"max":{"1min":0.73599999999999999,"5min":8.7400000000000002,"15min":107.551},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56999999999999995,"15min":0.97499999999999998},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.22900000000000001},"max":{"1min":0.64200000000000002,"5min":5.375,"15min":88.792000000000002},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.76800000000000002,"15min":1.006},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.216},"max":{"1min":0.77900000000000003,"5min":12.375999999999999,"15min":107.205},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.71499999999999997,"15min":0.88100000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.71299999999999997,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":1.0289999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.71599999999999997,"5min":10.794,"15min":54.174999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.753,"15min":1.0660000000000001},"min":{"1min":0.35699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76100000000000001,"5min":8.6890000000000001,"15min":86.183000000000007},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.745,"5min":13.691000000000001,"15min":44.395000000000003},"last":0.625},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.80100000000000005,"15min":1.2869999999999999},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":0.754,"5min":8.7690000000000001,"15min":114.526},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Feb 10 08:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.443,"15min":0.437},"min":{"1min":0.311,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64500000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.73299999999999998},"min":{"1min":0.28999999999999998,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":0.82299999999999995,"5min":10.023999999999999,"15min":72.241},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81000000000000005,"5min":10.013,"15min":22.469999999999999},"last":0.65900000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48099999999999998,"15min":0.72199999999999998},"min":{"1min":0.32900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1200000000000001,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.746},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.48599999999999999,"15min":0.754},"min":{"1min":0.313,"5min":0.253,"15min":0.253},"max":{"1min":1.8720000000000001,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2048000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2080768,"data_stored":2080768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":2076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:49:33.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph status --format=json 2024-02-10T08:49:33.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:33.384 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:33.482 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:33.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:33.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:33.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.463506+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:33.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.711966+0000 osd.6 (osd.6) 328 : cluster [DBG] 16.d scrub starts 2024-02-10T08:49:33.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.715169+0000 osd.6 (osd.6) 329 : cluster [DBG] 16.d scrub ok 2024-02-10T08:49:33.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.826475+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:32.829561+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:33.214203+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:33.222501+0000 mon.b (mon.1) 861 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:33.222954+0000 mon.a (mon.0) 3666 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.463506+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.711966+0000 osd.6 (osd.6) 328 : cluster [DBG] 16.d scrub starts 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.715169+0000 osd.6 (osd.6) 329 : cluster [DBG] 16.d scrub ok 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.826475+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:32.829561+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:33.214203+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-10T08:49:33.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:33.222501+0000 mon.b (mon.1) 861 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:33.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:33.222954+0000 mon.a (mon.0) 3666 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:33.618 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:33.639 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:33.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:33.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:33.770 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.463506+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.711966+0000 osd.6 (osd.6) 328 : cluster [DBG] 16.d scrub starts 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.715169+0000 osd.6 (osd.6) 329 : cluster [DBG] 16.d scrub ok 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.826475+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:32.829561+0000 osd.7 (osd.7) 329 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:49:33.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:33.214203+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-10T08:49:33.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:33.222501+0000 mon.b (mon.1) 861 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:33.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:33.222954+0000 mon.a (mon.0) 3666 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:34.042 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:34.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T08:49:34.319 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:34.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:34.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T08:49:34.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:34.455 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:34.476 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:34.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:34.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:33.302353+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:49:34.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:33.304168+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:49:34.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:33.575501+0000 mgr.y (mgr.14553) 659 : cluster [DBG] pgmap v1210: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:49:34.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:34.215893+0000 mon.a (mon.0) 3667 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:34.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:34.215986+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:34.252863+0000 mon.b (mon.1) 862 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:34.253198+0000 mon.a (mon.0) 3669 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:34.253616+0000 mon.b (mon.1) 863 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:34.253810+0000 mon.a (mon.0) 3670 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:33.302353+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:33.304168+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:33.575501+0000 mgr.y (mgr.14553) 659 : cluster [DBG] pgmap v1210: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:34.215893+0000 mon.a (mon.0) 3667 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:34.215986+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:34.252863+0000 mon.b (mon.1) 862 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:34.253198+0000 mon.a (mon.0) 3669 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.577 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:34.253616+0000 mon.b (mon.1) 863 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:34.253810+0000 mon.a (mon.0) 3670 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:34.609 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:34.631 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:34.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:34.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:33.302353+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:33.304168+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:33.575501+0000 mgr.y (mgr.14553) 659 : cluster [DBG] pgmap v1210: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:34.215893+0000 mon.a (mon.0) 3667 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:34.215986+0000 mon.a (mon.0) 3668 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:34.252863+0000 mon.b (mon.1) 862 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:34.253198+0000 mon.a (mon.0) 3669 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:34.253616+0000 mon.b (mon.1) 863 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:34.253810+0000 mon.a (mon.0) 3670 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-140"}]: dispatch 2024-02-10T08:49:34.804 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:34.812 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:34.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:34.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3 2024-02-10T08:49:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:35.237 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:35.483 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:34.308693+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:49:35.483 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:34.310459+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:49:35.483 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:34.639241+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:34.640384+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.012053+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 deep-scrub starts 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.015069+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 deep-scrub ok 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.216385+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:34.308693+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:34.310459+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:34.639241+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:34.640384+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.012053+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 deep-scrub starts 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.015069+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 deep-scrub ok 2024-02-10T08:49:35.484 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.216385+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:34.308693+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:34.310459+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:34.639241+0000 osd.0 (osd.0) 178 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:34.640384+0000 osd.0 (osd.0) 179 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.012053+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 deep-scrub starts 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.015069+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 deep-scrub ok 2024-02-10T08:49:35.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.216385+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-10T08:49:35.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:35] "GET /metrics HTTP/1.1" 200 191209 "" "Prometheus/2.33.4" 2024-02-10T08:49:35.994 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:49:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:35.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3' 2024-02-10T08:49:36.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:36.130 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.151 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:36.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:36.284 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.304 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:36.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:36.437 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:36.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:36.589 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.608 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.288075+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:49:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.294631+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:49:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.575981+0000 mgr.y (mgr.14553) 660 : cluster [DBG] pgmap v1213: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.985238+0000 osd.1 (osd.1) 348 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:49:36.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:35.988799+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:36.198045+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:36.199012+0000 mon.b (mon.1) 864 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:36.201081+0000 mon.a (mon.0) 3673 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.288075+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.294631+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.575981+0000 mgr.y (mgr.14553) 660 : cluster [DBG] pgmap v1213: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.985238+0000 osd.1 (osd.1) 348 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:35.988799+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:36.198045+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-10T08:49:36.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:36.199012+0000 mon.b (mon.1) 864 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:36.201081+0000 mon.a (mon.0) 3673 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.288075+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.294631+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.575981+0000 mgr.y (mgr.14553) 660 : cluster [DBG] pgmap v1213: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.985238+0000 osd.1 (osd.1) 348 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:35.988799+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:36.198045+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-10T08:49:36.737 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:36.199012+0000 mon.b (mon.1) 864 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.738 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:36.201081+0000 mon.a (mon.0) 3673 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:36.739 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.758 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:36.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:36.894 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:36.906 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:36.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:37.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:37.040 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:37.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:37.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7 2024-02-10T08:49:37.269 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:37.287 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:49:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:37.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7' 2024-02-10T08:49:37.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4 2024-02-10T08:49:37.417 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:37.436 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:49:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:37.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4' 2024-02-10T08:49:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.658+0000 7fefef80c700 1 -- 172.21.15.107:0/1002134036 >> v1:172.21.15.107:6789/0 conn(0x7fefe80ff710 legacy=0x7fefe8101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- 172.21.15.107:0/1002134036 shutdown_connections 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- 172.21.15.107:0/1002134036 >> 172.21.15.107:0/1002134036 conn(0x7fefe80f2510 msgr2=0x7fefe80f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- 172.21.15.107:0/1002134036 shutdown_connections 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- 172.21.15.107:0/1002134036 wait complete. 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 librados: starting msgr at 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 librados: starting objecter 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 Processor -- start 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- start start 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 librados: setting wanted keys 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 librados: calling monclient init 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefe8117450 con 0x7fefe8103250 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefe8118630 con 0x7fefe80ff710 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefef80c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fefe8119830 con 0x7fefe80fc910 2024-02-10T08:49:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.659+0000 7fefed5a8700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fefe80fc910 0x7fefe80f9c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60262/0 (socket says 172.21.15.107:60262) 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefed5a8700 1 -- 172.21.15.107:0/1284916364 learned_addr learned my addr 172.21.15.107:0/1284916364 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter ms_handle_connect 0x7fefe80fc910 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter ms_handle_connect 0x7fefe8103250 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197317562 0 0) 0x7fefe8119830 con 0x7fefe80fc910 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd0003030 con 0x7fefe80fc910 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter ms_handle_connect 0x7fefe80ff710 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065723345 0 0) 0x7fefe8117450 con 0x7fefe8103250 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8119830 con 0x7fefe8103250 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413730974 0 0) 0x7fefe8118630 con 0x7fefe80ff710 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8117450 con 0x7fefe80ff710 2024-02-10T08:49:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3534475032 0 0) 0x7fefd0003030 con 0x7fefe80fc910 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe8118630 con 0x7fefe80fc910 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefe0002990 con 0x7fefe80fc910 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481903508 0 0) 0x7fefe8119830 con 0x7fefe8103250 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefd0003030 con 0x7fefe8103250 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefd8001b00 con 0x7fefe8103250 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391836007 0 0) 0x7fefe8117450 con 0x7fefe80ff710 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe8119830 con 0x7fefe80ff710 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefdc001f90 con 0x7fefe80ff710 2024-02-10T08:49:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2507411286 0 0) 0x7fefe8118630 con 0x7fefe80fc910 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.195:6789/0 conn(0x7fefe80ff710 legacy=0x7fefe80fa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6789/0 conn(0x7fefe8103250 legacy=0x7fefe8115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe811aa30 con 0x7fefe80fc910 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe8117640 con 0x7fefe80fc910 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefef80c700 10 client.54407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe8117b30 con 0x7fefe80fc910 2024-02-10T08:49:37.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefef80c700 1 librados: init done 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fefe00017f0 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.660+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fefe0005b30 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.661+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fefd4004f00 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.661+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fefe0001350 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(797..797 src has 420..797) v4 ==== 8584+0+0 (unknown 566370847 0 0) 0x7fefe0078f90 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 osd_map(797..797 src has 420..797) v4 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 3 client.54407.objecter handle_osd_map got epochs [797,797] > 0 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 3 client.54407.objecter handle_osd_map decoding full epoch 797 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 20 client.54407.objecter dump_active .. 0 homeless 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v797) v1 ==== 54+0+16 (unknown 4293560388 0 2420000192) 0x7fefe007b380 con 0x7fefe80fc910 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd0063d60 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 10 client.54407.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 1 2024-02-10T08:49:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 1 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 0 1 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 4 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 10 client.54407.objecter _send_command 1 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.662+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4004950 con 0x7fefd4004530 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd4004530 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd4004950 con 0x7fefd4004530 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:49:37.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 10 client.54407.objecter _finish_command 1 = system:0 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 4 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.663+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 0 1 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.678+0000 7fefef80c700 10 client.54407.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 2 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 2 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd40051a0 osd=0 3 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 0 2 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 4 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 10 client.54407.objecter _send_command 2 2024-02-10T08:49:37.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.679+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd4004d30 con 0x7fefd4004530 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd4004d30 con 0x7fefd4004530 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefe67fc700 10 client.54407.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 4 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 0 2 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 10 client.54407.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 1 3 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 4 2024-02-10T08:49:37.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 10 client.54407.objecter _send_command 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.680+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd400c000 con 0x7fefd4004150 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd4004150 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd400c000 con 0x7fefd4004150 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 10 client.54407.objecter _finish_command 3 = system:0 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 4 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.681+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 1 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 10 client.54407.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 4 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 4 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003fe0 osd=1 3 2024-02-10T08:49:37.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 1 4 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 4 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 10 client.54407.objecter _send_command 4 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.696+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd4001850 con 0x7fefd4004150 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.697+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd4001850 con 0x7fefd4004150 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:49:37.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.697+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.697+0000 7fefe67fc700 10 client.54407.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.697+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 4 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.697+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 1 4 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 10 client.54407.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 5 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 5 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 2 5 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 4 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 10 client.54407.objecter _send_command 5 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4003e40 con 0x7fefd400c170 2024-02-10T08:49:37.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.698+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd400c170 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.699+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd4003e40 con 0x7fefd400c170 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.699+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.699+0000 7fefe67fc700 10 client.54407.objecter _finish_command 5 = system:0 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.699+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 4 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.699+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 2 5 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 10 client.54407.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 6 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 6 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003630 osd=2 3 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 2 6 2024-02-10T08:49:37.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 4 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 10 client.54407.objecter _send_command 6 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.713+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd400e6e0 con 0x7fefd400c170 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd400e6e0 con 0x7fefd400c170 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefe67fc700 10 client.54407.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 4 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 2 6 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 10 client.54407.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 7 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 7 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 3 7 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 4 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 10 client.54407.objecter _send_command 7 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.714+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4019e70 con 0x7fefd4016290 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd4016290 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd4019e70 con 0x7fefd4016290 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 10 client.54407.objecter _finish_command 7 = system:0 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 4 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.715+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 3 7 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 10 client.54407.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 8 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 8 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4003390 osd=3 3 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 3 8 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 4 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 10 client.54407.objecter _send_command 8 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.729+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd401b190 con 0x7fefd4016290 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd401b190 con 0x7fefd4016290 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 10 client.54407.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 4 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 3 8 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 10 client.54407.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 9 2024-02-10T08:49:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 9 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 4 9 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 4 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 10 client.54407.objecter _send_command 9 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd401e820 con 0x7fefd401c300 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.731+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd401c300 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.732+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd401e820 con 0x7fefd401c300 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.732+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.732+0000 7fefe67fc700 10 client.54407.objecter _finish_command 9 = system:0 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.732+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 4 2024-02-10T08:49:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.732+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 4 9 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 10 client.54407.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 10 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 10 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd401b190 osd=4 3 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 4 10 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 4 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 10 client.54407.objecter _send_command 10 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.747+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd401e820 con 0x7fefd401c300 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.748+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd401e820 con 0x7fefd401c300 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.748+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.748+0000 7fefe67fc700 10 client.54407.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.748+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 4 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.748+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 4 10 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 10 client.54407.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 11 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 11 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 5 11 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 4 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 10 client.54407.objecter _send_command 11 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4022b60 con 0x7fefd401eea0 2024-02-10T08:49:37.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.749+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd401eea0 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.750+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd4022b60 con 0x7fefd401eea0 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.750+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.750+0000 7fefe67fc700 10 client.54407.objecter _finish_command 11 = system:0 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.750+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 4 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.750+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 5 11 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 10 client.54407.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 12 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 12 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd401e820 osd=5 3 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 5 12 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 4 2024-02-10T08:49:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 10 client.54407.objecter _send_command 12 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.764+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd4024310 con 0x7fefd401eea0 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd4024310 con 0x7fefd401eea0 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 10 client.54407.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 4 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 5 12 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 10 client.54407.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 13 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 13 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 6 13 2024-02-10T08:49:37.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 4 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 10 client.54407.objecter _send_command 13 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4024f40 con 0x7fefd4024990 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd4024990 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd4024f40 con 0x7fefd4024990 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 10 client.54407.objecter _finish_command 13 = system:0 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 4 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.767+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 6 13 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 10 client.54407.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 14 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 14 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4024310 osd=6 3 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 6 14 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 4 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 10 client.54407.objecter _send_command 14 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.782+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd4024f40 con 0x7fefd4024990 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd4024f40 con 0x7fefd4024990 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefe67fc700 10 client.54407.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 4 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 6 14 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 10 client.54407.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 15 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 15 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 7 15 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 4 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 10 client.54407.objecter _send_command 15 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.783+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd402b530 con 0x7fefd4027a20 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.784+0000 7fefe67fc700 10 client.54407.objecter ms_handle_connect 0x7fefd4027a20 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.784+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fefd402b530 con 0x7fefd4027a20 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.785+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.785+0000 7fefe67fc700 10 client.54407.objecter _finish_command 15 = system:0 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.785+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 4 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.785+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 7 15 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 10 client.54407.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-10T08:49:37.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign -1 16 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 20 client.54407.objecter _get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 15 client.54407.objecter _session_command_op_remove -1 16 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 20 client.54407.objecter get_session s=0x7fefd4024f40 osd=7 3 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 15 client.54407.objecter _session_command_op_assign 7 16 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 4 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 10 client.54407.objecter _send_command 16 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.799+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fefd402b530 con 0x7fefd4027a20 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.800+0000 7fefe67fc700 1 -- 172.21.15.107:0/1284916364 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fefd402b530 con 0x7fefd4027a20 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.800+0000 7fefe67fc700 10 client.54407.objecter ms_dispatch 0x7fefe8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.800+0000 7fefe67fc700 10 client.54407.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.800+0000 7fefe67fc700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 4 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.800+0000 7fefe67fc700 15 client.54407.objecter _session_command_op_remove 7 16 2024-02-10T08:49:37.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 librados: watch_flush enter 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 librados: watch_flush exit 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.0 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6801/3321596972 conn(0x7fefd4004530 legacy=0x7fefd4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd40051a0 osd=0 1 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.1 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6805/133516126 conn(0x7fefd4004150 legacy=0x7fefd400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003fe0 osd=1 1 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.2 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6809/3575881728 conn(0x7fefd400c170 legacy=0x7fefd400c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003630 osd=2 1 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.3 2024-02-10T08:49:37.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6813/2871825691 conn(0x7fefd4016290 legacy=0x7fefd4018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4003390 osd=3 1 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.4 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.195:6800/3310789251 conn(0x7fefd401c300 legacy=0x7fefd400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401b190 osd=4 1 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.5 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.195:6804/4178321486 conn(0x7fefd401eea0 legacy=0x7fefd40212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefedda9700 1 -- 172.21.15.107:0/1284916364 reap_dead start 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefedda9700 1 -- 172.21.15.107:0/1284916364 reap_dead start 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd401e820 osd=5 1 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.6 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.195:6808/1417479017 conn(0x7fefd4024990 legacy=0x7fefd4022b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefedda9700 1 -- 172.21.15.107:0/1284916364 reap_dead start 2024-02-10T08:49:37.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024310 osd=6 1 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter close_session for osd.7 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.195:6812/1934176928 conn(0x7fefd4027a20 legacy=0x7fefd4029e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter put_session s=0x7fefd4024f40 osd=7 1 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 20 client.54407.objecter shutdown clearing up homeless session... 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 10 client.54407.objecter successfully canceled tick 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6800/2634842009 conn(0x7fefd0063d60 legacy=0x7fefd0066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> v1:172.21.15.107:6790/0 conn(0x7fefe80fc910 legacy=0x7fefe80f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 shutdown_connections 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 >> 172.21.15.107:0/1284916364 conn(0x7fefe80f2510 msgr2=0x7fefe80f5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 shutdown_connections 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 -- 172.21.15.107:0/1284916364 wait complete. 2024-02-10T08:49:37.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:36.801+0000 7fefef80c700 1 librados: shutdown 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.534 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:36.076235+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:36.077059+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:36.663779+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? v1:172.21.15.107:0/1284916364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:37.199430+0000 mon.a (mon.0) 3674 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.199602+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:37.240654+0000 mon.b (mon.1) 865 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:37.241026+0000 mon.a (mon.0) 3676 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:37.241492+0000 mon.b (mon.1) 866 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:37.241738+0000 mon.a (mon.0) 3677 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:37.567 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:37.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:36.076235+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:36.077059+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:36.663779+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? v1:172.21.15.107:0/1284916364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:37.199430+0000 mon.a (mon.0) 3674 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.199602+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:37.240654+0000 mon.b (mon.1) 865 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:37.241026+0000 mon.a (mon.0) 3676 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:37.241492+0000 mon.b (mon.1) 866 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:37.241738+0000 mon.a (mon.0) 3677 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:36.076235+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 scrub starts 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:36.077059+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 scrub ok 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:36.663779+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? v1:172.21.15.107:0/1284916364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:37.199430+0000 mon.a (mon.0) 3674 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.199602+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:37.240654+0000 mon.b (mon.1) 865 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:37.241026+0000 mon.a (mon.0) 3676 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:37.241492+0000 mon.b (mon.1) 866 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:37.241738+0000 mon.a (mon.0) 3677 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-142"}]: dispatch 2024-02-10T08:49:37.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:37.709 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-02-10T08:49:37.710 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T08:49:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.117384+0000 osd.4 (osd.4) 73 : cluster [DBG] 5.e scrub starts 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.120850+0000 osd.4 (osd.4) 74 : cluster [DBG] 5.e scrub ok 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.288331+0000 osd.5 (osd.5) 332 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.293064+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.295583+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.297242+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:37.576448+0000 mgr.y (mgr.14553) 661 : cluster [DBG] pgmap v1216: 194 pgs: 7 creating+peering, 19 unknown, 168 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T08:49:38.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.200364+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-10T08:49:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.117384+0000 osd.4 (osd.4) 73 : cluster [DBG] 5.e scrub starts 2024-02-10T08:49:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.120850+0000 osd.4 (osd.4) 74 : cluster [DBG] 5.e scrub ok 2024-02-10T08:49:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.288331+0000 osd.5 (osd.5) 332 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:49:38.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.293064+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.295583+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.297242+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:37.576448+0000 mgr.y (mgr.14553) 661 : cluster [DBG] pgmap v1216: 194 pgs: 7 creating+peering, 19 unknown, 168 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.200364+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.117384+0000 osd.4 (osd.4) 73 : cluster [DBG] 5.e scrub starts 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.120850+0000 osd.4 (osd.4) 74 : cluster [DBG] 5.e scrub ok 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.288331+0000 osd.5 (osd.5) 332 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.293064+0000 osd.5 (osd.5) 333 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.295583+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.297242+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:37.576448+0000 mgr.y (mgr.14553) 661 : cluster [DBG] pgmap v1216: 194 pgs: 7 creating+peering, 19 unknown, 168 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T08:49:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.200364+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.156685+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.160185+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.326340+0000 osd.5 (osd.5) 334 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.332198+0000 osd.5 (osd.5) 335 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.809125+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:38.810276+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:49:39.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.084127+0000 osd.1 (osd.1) 350 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:49:39.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.088226+0000 osd.1 (osd.1) 351 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:49:39.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.203513+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-10T08:49:39.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:39.210562+0000 mon.b (mon.1) 867 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:39.211996+0000 mon.a (mon.0) 3680 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.844 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.156685+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:49:39.844 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.160185+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:49:39.844 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.326340+0000 osd.5 (osd.5) 334 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.332198+0000 osd.5 (osd.5) 335 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.809125+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:38.810276+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.084127+0000 osd.1 (osd.1) 350 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.088226+0000 osd.1 (osd.1) 351 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.203513+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:39.210562+0000 mon.b (mon.1) 867 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:39.211996+0000 mon.a (mon.0) 3680 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.156685+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.160185+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:49:39.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.326340+0000 osd.5 (osd.5) 334 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.332198+0000 osd.5 (osd.5) 335 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.809125+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:38.810276+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.084127+0000 osd.1 (osd.1) 350 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.088226+0000 osd.1 (osd.1) 351 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.203513+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:39.210562+0000 mon.b (mon.1) 867 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:39.211996+0000 mon.a (mon.0) 3680 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T08:49:39.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T08:49:40.138 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:40.398 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:40.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:40.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T08:49:40.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:40.504 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:40.552 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.099849+0000 osd.2 (osd.2) 307 : cluster [DBG] 5.d scrub starts 2024-02-10T08:49:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.103505+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.d scrub ok 2024-02-10T08:49:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.308151+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:49:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.311157+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:39.576938+0000 mgr.y (mgr.14553) 662 : cluster [DBG] pgmap v1219: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:40.120146+0000 osd.1 (osd.1) 352 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:40.123714+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.172195+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.206204+0000 mon.a (mon.0) 3682 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:40.206285+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.216813+0000 mon.b (mon.1) 868 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.217753+0000 mon.a (mon.0) 3684 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.235928+0000 mon.b (mon.1) 869 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.236484+0000 mon.a (mon.0) 3685 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.236977+0000 mon.b (mon.1) 870 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:40.237205+0000 mon.a (mon.0) 3686 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.099849+0000 osd.2 (osd.2) 307 : cluster [DBG] 5.d scrub starts 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.103505+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.d scrub ok 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.308151+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.311157+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:39.576938+0000 mgr.y (mgr.14553) 662 : cluster [DBG] pgmap v1219: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:40.120146+0000 osd.1 (osd.1) 352 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:40.123714+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.172195+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.206204+0000 mon.a (mon.0) 3682 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:40.206285+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.216813+0000 mon.b (mon.1) 868 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.217753+0000 mon.a (mon.0) 3684 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.235928+0000 mon.b (mon.1) 869 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.236484+0000 mon.a (mon.0) 3685 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.236977+0000 mon.b (mon.1) 870 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:40.237205+0000 mon.a (mon.0) 3686 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.099849+0000 osd.2 (osd.2) 307 : cluster [DBG] 5.d scrub starts 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.103505+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.d scrub ok 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.308151+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.311157+0000 osd.5 (osd.5) 337 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:39.576938+0000 mgr.y (mgr.14553) 662 : cluster [DBG] pgmap v1219: 194 pgs: 32 unknown, 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:40.120146+0000 osd.1 (osd.1) 352 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:49:40.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:40.123714+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.172195+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.206204+0000 mon.a (mon.0) 3682 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-112490-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:40.206285+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.216813+0000 mon.b (mon.1) 868 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.217753+0000 mon.a (mon.0) 3684 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi107-112490-101","var": "dedup_tier","val": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.235928+0000 mon.b (mon.1) 869 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.236484+0000 mon.a (mon.0) 3685 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.236977+0000 mon.b (mon.1) 870 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:40.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:40.237205+0000 mon.a (mon.0) 3686 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi107-112490-101", "tierpool": "test-rados-api-smithi107-112490-144"}]: dispatch 2024-02-10T08:49:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:40.103011+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub starts 2024-02-10T08:49:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:40.106004+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.b scrub ok 2024-02-10T08:49:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:41.194134+0000 mon.a (mon.0) 3687 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:41.207810+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-10T08:49:41.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:40.103011+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub starts 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:40.106004+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.b scrub ok 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:41.194134+0000 mon.a (mon.0) 3687 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:41.207810+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:40.103011+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub starts 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:40.106004+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.b scrub ok 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:41.194134+0000 mon.a (mon.0) 3687 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:41.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:41.207810+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-10T08:49:41.868 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:42.104 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:42.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:42.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0 2024-02-10T08:49:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:42.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:42.436 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:49:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:42.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0' 2024-02-10T08:49:42.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:42.573 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:42.594 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:42.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:42.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:41.116192+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub starts 2024-02-10T08:49:42.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:41.119230+0000 osd.2 (osd.2) 312 : cluster [DBG] 2.e scrub ok 2024-02-10T08:49:42.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:41.577478+0000 mgr.y (mgr.14553) 663 : cluster [DBG] pgmap v1222: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:42.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:41.849090+0000 mgr.y (mgr.14553) 664 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:42.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.208376+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:42.210957+0000 mon.b (mon.1) 871 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:42.211311+0000 mon.a (mon.0) 3690 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:41.116192+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub starts 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:41.119230+0000 osd.2 (osd.2) 312 : cluster [DBG] 2.e scrub ok 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:41.577478+0000 mgr.y (mgr.14553) 663 : cluster [DBG] pgmap v1222: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:41.849090+0000 mgr.y (mgr.14553) 664 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.208376+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:42.210957+0000 mon.b (mon.1) 871 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:42.211311+0000 mon.a (mon.0) 3690 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:42.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 -- 172.21.15.107:0/997451770 >> v1:172.21.15.107:6790/0 conn(0x7fb4680fd9f0 legacy=0x7fb4680fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:42.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 -- 172.21.15.107:0/997451770 shutdown_connections 2024-02-10T08:49:42.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 -- 172.21.15.107:0/997451770 >> 172.21.15.107:0/997451770 conn(0x7fb46806c090 msgr2=0x7fb46806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:42.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 -- 172.21.15.107:0/997451770 shutdown_connections 2024-02-10T08:49:42.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 -- 172.21.15.107:0/997451770 wait complete. 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 librados: starting msgr at 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.724+0000 7fb46d1d1700 1 librados: starting objecter 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 Processor -- start 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 -- start start 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 librados: setting wanted keys 2024-02-10T08:49:42.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 librados: calling monclient init 2024-02-10T08:49:42.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4681174a0 con 0x7fb4680fd9f0 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb468118680 con 0x7fb468104350 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb46d1d1700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb468119880 con 0x7fb468100840 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb466d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb4680fd9f0 0x7fb4680f9db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51028/0 (socket says 172.21.15.107:51028) 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb466d9d700 1 -- 172.21.15.107:0/3818312292 learned_addr learned my addr 172.21.15.107:0/3818312292 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter ms_handle_connect 0x7fb4680fd9f0 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter ms_handle_connect 0x7fb468100840 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter ms_handle_connect 0x7fb468104350 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120881348 0 0) 0x7fb468119880 con 0x7fb468100840 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450003030 con 0x7fb468100840 2024-02-10T08:49:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706895977 0 0) 0x7fb4681174a0 con 0x7fb4680fd9f0 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.725+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb468119880 con 0x7fb4680fd9f0 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060876623 0 0) 0x7fb468118680 con 0x7fb468104350 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4681174a0 con 0x7fb468104350 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2453283169 0 0) 0x7fb450003030 con 0x7fb468100840 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb468118680 con 0x7fb468100840 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb45c001800 con 0x7fb468100840 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1792693803 0 0) 0x7fb468119880 con 0x7fb4680fd9f0 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb450003030 con 0x7fb4680fd9f0 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb4580010a0 con 0x7fb4680fd9f0 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693418530 0 0) 0x7fb4681174a0 con 0x7fb468104350 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb468119880 con 0x7fb468104350 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb460001c30 con 0x7fb468104350 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 283600561 0 0) 0x7fb468118680 con 0x7fb468100840 2024-02-10T08:49:42.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 >> v1:172.21.15.195:6789/0 conn(0x7fb468104350 legacy=0x7fb468115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 >> v1:172.21.15.107:6789/0 conn(0x7fb4680fd9f0 legacy=0x7fb4680f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46811aa80 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb468117690 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb46d1d1700 10 client.54413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb468117b00 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb46d1d1700 1 librados: init done 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb45c001e00 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.726+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb45c003230 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.727+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb454004f20 con 0x7fb468100840 2024-02-10T08:49:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.727+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb45c016140 con 0x7fb468100840 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(803..803 src has 420..803) v4 ==== 7789+0+0 (unknown 1988505857 0 0) 0x7fb45c076250 con 0x7fb468100840 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 10 client.54413.objecter ms_dispatch 0x7fb468001550 osd_map(803..803 src has 420..803) v4 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 3 client.54413.objecter handle_osd_map got epochs [803,803] > 0 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 3 client.54413.objecter handle_osd_map decoding full epoch 803 2024-02-10T08:49:42.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 20 client.54413.objecter dump_active .. 0 homeless 2024-02-10T08:49:42.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.728+0000 7fb44ffff700 10 client.54413.objecter ms_handle_connect 0x7fb450063be0 2024-02-10T08:49:42.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.732+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb45c049750 con 0x7fb468100840 2024-02-10T08:49:42.751 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:42.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:41.116192+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub starts 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:41.119230+0000 osd.2 (osd.2) 312 : cluster [DBG] 2.e scrub ok 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:41.577478+0000 mgr.y (mgr.14553) 663 : cluster [DBG] pgmap v1222: 162 pgs: 162 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:41.849090+0000 mgr.y (mgr.14553) 664 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.208376+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:42.210957+0000 mon.b (mon.1) 871 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:42.211311+0000 mon.a (mon.0) 3690 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:42.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6 2024-02-10T08:49:42.884 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:42.905 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:49:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:42.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6' 2024-02-10T08:49:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:42.908+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb4540051d0 con 0x7fb468100840 2024-02-10T08:49:42.929 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:43.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:43.037 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:43.160 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:43.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:43.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.210+0000 7fb44ffff700 1 -- 172.21.15.107:0/3818312292 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v804) v1 ==== 87+0+0 (unknown 2978766595 0 0) 0x7fb45c047b60 con 0x7fb468100840 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 10 librados: watch_flush enter 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 10 librados: watch_flush exit 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 20 client.54413.objecter shutdown clearing up homeless session... 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 10 client.54413.objecter successfully canceled tick 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb450063be0 legacy=0x7fb4500660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 >> v1:172.21.15.107:6790/0 conn(0x7fb468100840 legacy=0x7fb4680fa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 shutdown_connections 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 >> 172.21.15.107:0/3818312292 conn(0x7fb46806c090 msgr2=0x7fb4681067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 shutdown_connections 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 -- 172.21.15.107:0/3818312292 wait complete. 2024-02-10T08:49:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:43.213+0000 7fb46d1d1700 1 librados: shutdown 2024-02-10T08:49:43.216 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-02-10T08:49:43.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5 2024-02-10T08:49:43.294 INFO:teuthology.orchestra.run.smithi195.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:43.313 INFO:teuthology.orchestra.run.smithi195.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:49:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:43.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5' 2024-02-10T08:49:43.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2 2024-02-10T08:49:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.105513+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.108564+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.156183+0000 osd.4 (osd.4) 77 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.159692+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.397517+0000 osd.3 (osd.3) 357 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:49:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.399326+0000 osd.3 (osd.3) 358 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.705048+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:42.706425+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:42.910640+0000 mon.c (mon.2) 491 : audit [INF] from='client.? v1:172.21.15.107:0/3818312292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:42.910882+0000 mon.a (mon.0) 3691 : audit [INF] from='client.54413 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:43.206258+0000 mon.a (mon.0) 3692 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:43.210608+0000 mon.a (mon.0) 3693 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:43.210688+0000 mon.a (mon.0) 3694 : audit [INF] from='client.54413 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:43.210741+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:43.212664+0000 mon.b (mon.1) 872 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:43.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:43.214125+0000 mon.a (mon.0) 3696 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:43.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.105513+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-10T08:49:43.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.108564+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-10T08:49:43.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.156183+0000 osd.4 (osd.4) 77 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.159692+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.397517+0000 osd.3 (osd.3) 357 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.399326+0000 osd.3 (osd.3) 358 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.705048+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:42.706425+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:42.910640+0000 mon.c (mon.2) 491 : audit [INF] from='client.? v1:172.21.15.107:0/3818312292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:42.910882+0000 mon.a (mon.0) 3691 : audit [INF] from='client.54413 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:43.206258+0000 mon.a (mon.0) 3692 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:43.210608+0000 mon.a (mon.0) 3693 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:43.210688+0000 mon.a (mon.0) 3694 : audit [INF] from='client.54413 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:43.210741+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:43.212664+0000 mon.b (mon.1) 872 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:43.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:43.214125+0000 mon.a (mon.0) 3696 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.105513+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.108564+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.156183+0000 osd.4 (osd.4) 77 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.159692+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.397517+0000 osd.3 (osd.3) 357 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.399326+0000 osd.3 (osd.3) 358 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.705048+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:42.706425+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:42.910640+0000 mon.c (mon.2) 491 : audit [INF] from='client.? v1:172.21.15.107:0/3818312292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:42.910882+0000 mon.a (mon.0) 3691 : audit [INF] from='client.54413 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:43.206258+0000 mon.a (mon.0) 3692 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T08:49:43.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:43.210608+0000 mon.a (mon.0) 3693 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:43.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:43.210688+0000 mon.a (mon.0) 3694 : audit [INF] from='client.54413 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T08:49:43.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:43.210741+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-10T08:49:43.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:43.212664+0000 mon.b (mon.1) 872 : audit [INF] from='client.? v1:172.21.15.107:0/3092462557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:43.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:43.214125+0000 mon.a (mon.0) 3696 : audit [INF] from='client.48721 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]: dispatch 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8060 ms) 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12992 ms) 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7947 ms) 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-10T08:49:44.218 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14042 ms) 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7942 ms) 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8030 ms) 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: ok, hit_set contains 248:602f83fe:::foo:head 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8083 ms) 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,0 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: first is 1707554902 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,0 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,0 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,0 2024-02-10T08:49:44.219 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,1707554910,1707554911,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,1707554910,1707554911,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554902,1707554904,1707554905,1707554907,1707554908,1707554910,1707554911,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: got ls 1707554905,1707554907,1707554908,1707554910,1707554911,1707554913,1707554914,0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: first now 1707554905, trimmed 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20376 ms) 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: foo0 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-10T08:49:44.220 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13528 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17479 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21547 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5010 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3004 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2985 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3009 ms) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (236089 ms total) 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: 2024-02-10T08:49:44.221 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-10T08:49:44.222 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [==========] 72 tests from 4 test suites ran. (793983 ms total) 2024-02-10T08:49:44.222 INFO:tasks.workunit.client.0.smithi107.stdout: api_tier_pp: [ PASSED ] 72 tests. 2024-02-10T08:49:44.222 INFO:tasks.workunit.client.0.smithi107.stderr:+ exit 0 2024-02-10T08:49:44.222 INFO:tasks.workunit.client.0.smithi107.stderr:+ cleanup 2024-02-10T08:49:44.222 INFO:tasks.workunit.client.0.smithi107.stderr:+ pkill -P 112393 2024-02-10T08:49:44.261 INFO:tasks.workunit.client.0.smithi107.stderr:+ true 2024-02-10T08:49:44.261 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-10T08:49:44.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-10T08:49:44.332 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-02-10T08:49:44.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-10T08:49:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:49:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:44.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2' 2024-02-10T08:49:44.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1 2024-02-10T08:49:44.526 INFO:teuthology.orchestra.run.smithi107.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:43.361487+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:43.363192+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:43.577826+0000 mgr.y (mgr.14553) 665 : cluster [DBG] pgmap v1225: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:44.212784+0000 mon.a (mon.0) 3697 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:44.212861+0000 mon.a (mon.0) 3698 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-10T08:49:44.666 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:43.361487+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:49:44.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:43.363192+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:49:44.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:43.577826+0000 mgr.y (mgr.14553) 665 : cluster [DBG] pgmap v1225: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:44.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:44.212784+0000 mon.a (mon.0) 3697 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:44.667 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:44.212861+0000 mon.a (mon.0) 3698 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-10T08:49:44.668 DEBUG:teuthology.parallel:result is None 2024-02-10T08:49:44.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T08:49:44.736 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T08:49:44.736 DEBUG:teuthology.orchestra.run.smithi107:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-10T08:49:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:43.361487+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:49:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:43.363192+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:49:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:43.577826+0000 mgr.y (mgr.14553) 665 : cluster [DBG] pgmap v1225: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:44.212784+0000 mon.a (mon.0) 3697 : audit [INF] from='client.48721 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi107-112490-101"}]': finished 2024-02-10T08:49:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:44.212861+0000 mon.a (mon.0) 3698 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-10T08:49:44.820 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T08:49:44.820 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-10T08:49:44.830 INFO:tasks.thrashosds:joining thrashosds 2024-02-10T08:49:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.780+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/3064710033 >> v1:172.21.15.107:6790/0 conn(0x7f9f040fda50 legacy=0x7f9f040fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.780+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/3064710033 shutdown_connections 2024-02-10T08:49:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.780+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/3064710033 >> 172.21.15.107:0/3064710033 conn(0x7f9f040f3640 msgr2=0x7f9f040f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:45.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.780+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/3064710033 shutdown_connections 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/3064710033 wait complete. 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 librados: starting msgr at 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 librados: starting objecter 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 Processor -- start 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 -- start start 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 librados: setting wanted keys 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 librados: calling monclient init 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f040fcfc0 con 0x7f9f041007f0 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f040fd190 con 0x7f9f040fda50 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0c7a9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f040fd380 con 0x7f9f04104330 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0ad46700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9f04104330 0x7f9f041184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38366/0 (socket says 172.21.15.107:38366) 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f09d44700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9f041007f0 0x7f9f04114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51054/0 (socket says 172.21.15.107:51054) 2024-02-10T08:49:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.781+0000 7f9f0ad46700 1 -- 172.21.15.107:0/1877917204 learned_addr learned my addr 172.21.15.107:0/1877917204 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter ms_handle_connect 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter ms_handle_connect 0x7f9f041007f0 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter ms_handle_connect 0x7f9f040fda50 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862862787 0 0) 0x7f9f040fd380 con 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef0003030 con 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173311602 0 0) 0x7f9f040fcfc0 con 0x7f9f041007f0 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f040fd380 con 0x7f9f041007f0 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134231425 0 0) 0x7f9f040fd190 con 0x7f9f040fda50 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f040fcfc0 con 0x7f9f040fda50 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2694764811 0 0) 0x7f9ef0003030 con 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f040fd190 con 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9efc001d70 con 0x7f9f04104330 2024-02-10T08:49:45.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293080740 0 0) 0x7f9f040fd380 con 0x7f9f041007f0 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ef0003030 con 0x7f9f041007f0 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9f00001840 con 0x7f9f041007f0 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396012729 0 0) 0x7f9f040fcfc0 con 0x7f9f040fda50 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f040fd380 con 0x7f9f040fda50 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9ef4001430 con 0x7f9f040fda50 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 865507617 0 0) 0x7f9f040fd190 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 >> v1:172.21.15.195:6789/0 conn(0x7f9f040fda50 legacy=0x7f9f040fc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 >> v1:172.21.15.107:6789/0 conn(0x7f9f041007f0 legacy=0x7f9f04114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0411cc00 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f04119c30 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9f0c7a9700 10 client.54419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0411a0a0 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9f0c7a9700 1 librados: init done 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9efc002740 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.782+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9efc005e00 con 0x7f9f04104330 2024-02-10T08:49:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.783+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee8004f20 con 0x7f9f04104330 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.783+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9efc0028b0 con 0x7f9f04104330 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(805..805 src has 420..805) v4 ==== 7386+0+0 (unknown 2905194310 0 0) 0x7f9efc078e50 con 0x7f9f04104330 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 10 client.54419.objecter ms_dispatch 0x7f9f04001550 osd_map(805..805 src has 420..805) v4 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 3 client.54419.objecter handle_osd_map got epochs [805,805] > 0 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 3 client.54419.objecter handle_osd_map decoding full epoch 805 2024-02-10T08:49:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 20 client.54419.objecter dump_active .. 0 homeless 2024-02-10T08:49:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.784+0000 7f9efb7fe700 10 client.54419.objecter ms_handle_connect 0x7f9ef0063d10 2024-02-10T08:49:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.788+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9efc04c540 con 0x7f9f04104330 2024-02-10T08:49:45.878 INFO:teuthology.orchestra.run.smithi107.stderr:Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:49:45.881 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:45] "GET /metrics HTTP/1.1" 200 191307 "" "Prometheus/2.33.4" 2024-02-10T08:49:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-10T08:49:45.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 125: 'sudo docker kill -s 1 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1' 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.972+0000 7f46bb30a700 1 -- 172.21.15.107:0/3402159602 >> v1:172.21.15.107:6790/0 conn(0x7f46b4104350 legacy=0x7f46b41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.972+0000 7f46bb30a700 1 -- 172.21.15.107:0/3402159602 shutdown_connections 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.972+0000 7f46bb30a700 1 -- 172.21.15.107:0/3402159602 >> 172.21.15.107:0/3402159602 conn(0x7f46b40f2510 msgr2=0x7f46b40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- 172.21.15.107:0/3402159602 shutdown_connections 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- 172.21.15.107:0/3402159602 wait complete. 2024-02-10T08:49:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 librados: starting msgr at 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 librados: starting objecter 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 Processor -- start 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- start start 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 librados: setting wanted keys 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 librados: calling monclient init 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46b4117450 con 0x7f46b40f9570 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46b4118630 con 0x7f46b40f89c0 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46bb30a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46b4119830 con 0x7f46b4104350 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46b88a5700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f46b40f9570 0x7f46b40ffe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51078/0 (socket says 172.21.15.107:51078) 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.973+0000 7f46b88a5700 1 -- 172.21.15.107:0/4211902737 learned_addr learned my addr 172.21.15.107:0/4211902737 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter ms_handle_connect 0x7f46b40f9570 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter ms_handle_connect 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835631315 0 0) 0x7f46b4117450 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a00033f0 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817085713 0 0) 0x7f46b4118630 con 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b4117450 con 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189930717 0 0) 0x7f46a00033f0 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b4118630 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f46a4002910 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658074148 0 0) 0x7f46b4117450 con 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a00033f0 con 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f46b0001600 con 0x7f46b40f89c0 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3767025306 0 0) 0x7f46b4118630 con 0x7f46b40f9570 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6790/0 conn(0x7f46b4104350 legacy=0x7f46b4115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.195:6789/0 conn(0x7f46b40f89c0 legacy=0x7f46b40ff750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b411aa30 con 0x7f46b40f9570 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter ms_handle_connect 0x7f46b4104350 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b4118820 con 0x7f46b40f9570 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46bb30a700 10 client.28827.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b4118cd0 con 0x7f46b40f9570 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.974+0000 7f46bb30a700 1 librados: init done 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.975+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f46a4002ca0 con 0x7f46b40f9570 2024-02-10T08:49:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.975+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f46a4005bc0 con 0x7f46b40f9570 2024-02-10T08:49:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.975+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4698004f00 con 0x7f46b40f9570 2024-02-10T08:49:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f46a4018a50 con 0x7f46b40f9570 2024-02-10T08:49:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(805..805 src has 420..805) v4 ==== 7386+0+0 (unknown 2905194310 0 0) 0x7f46a40789b0 con 0x7f46b40f9570 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 osd_map(805..805 src has 420..805) v4 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 3 client.28827.objecter handle_osd_map got epochs [805,805] > 0 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 3 client.28827.objecter handle_osd_map decoding full epoch 805 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 20 client.28827.objecter dump_active .. 0 homeless 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v805) v1 ==== 54+0+16 (unknown 1577514061 0 2420000192) 0x7f46a4078dc0 con 0x7f46b40f9570 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f46a0063c00 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46bb30a700 10 client.28827.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.976+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 1 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 1 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 0 1 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 4 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 10 client.28827.objecter _send_command 1 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4698004950 con 0x7f4698004530 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698004530 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4698004950 con 0x7f4698004530 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 1 = system:0 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 4 2024-02-10T08:49:45.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.977+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 0 1 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 10 client.28827.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 2 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 2 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980051a0 osd=0 3 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 0 2 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 4 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 10 client.28827.objecter _send_command 2 2024-02-10T08:49:45.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.992+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4698004d30 con 0x7f4698004530 2024-02-10T08:49:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4698004d30 con 0x7f4698004530 2024-02-10T08:49:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 4 2024-02-10T08:49:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 0 2 2024-02-10T08:49:45.995 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-02-10T08:49:45.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 10 client.28827.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:45.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 3 2024-02-10T08:49:45.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 1 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 4 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 10 client.28827.objecter _send_command 3 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6805/133516126 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4698004190 con 0x7f46980088d0 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.994+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f46980088d0 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.995+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4698004190 con 0x7f46980088d0 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.995+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 3: 0 ) v1 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.995+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 3 = system:0 2024-02-10T08:49:45.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.995+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 4 2024-02-10T08:49:45.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:45.995+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 1 3 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 10 client.28827.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 4 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 4 2024-02-10T08:49:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f4698004020 osd=1 3 2024-02-10T08:49:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 1 4 2024-02-10T08:49:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 4 2024-02-10T08:49:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 10 client.28827.objecter _send_command 4 2024-02-10T08:49:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.012+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6805/133516126 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f46980017f0 con 0x7f46980088d0 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.013+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f46980017f0 con 0x7f46980088d0 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.013+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.013+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.013+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 4 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.013+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 1 4 2024-02-10T08:49:46.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 10 client.28827.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 5 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 5 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 2 5 2024-02-10T08:49:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 4 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 10 client.28827.objecter _send_command 5 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6809/3575881728 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4698010e50 con 0x7f4698003b70 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698003b70 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4698010e50 con 0x7f4698003b70 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 5: 0 ) v1 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 5 = system:0 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 4 2024-02-10T08:49:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.014+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 2 5 2024-02-10T08:49:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 10 client.28827.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 6 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 6 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f4698001c00 osd=2 3 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 2 6 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 4 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 10 client.28827.objecter _send_command 6 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.029+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6809/3575881728 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4698003100 con 0x7f4698003b70 2024-02-10T08:49:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.030+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9ee80051d0 con 0x7f9f04104330 2024-02-10T08:49:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f9efb7fe700 1 -- 172.21.15.107:0/1877917204 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2380 (unknown 3226119948 0 2667010590) 0x7f9efc04a9b0 con 0x7f9f04104330 2024-02-10T08:49:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4698003100 con 0x7f4698003b70 2024-02-10T08:49:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 4 2024-02-10T08:49:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 2 6 2024-02-10T08:49:46.032 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-02-10T08:49:46.032 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:49:46.032 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1200,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":805,"num_osds":8,"num_up_osds":8,"osd_up_since":1707554788,"num_in_osds":8,"osd_in_since":1707554868,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":154}],"num_pgs":154,"num_pools":7,"num_objects":193,"data_bytes":5801,"bytes_used":2535129088,"bytes_avail":765425156096,"bytes_total":767960285184},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"}},"servicemap":{"epoch":22,"modified":"2024-02-10T08:49:41.577325+0000","services":{"rgw":{"daemons":{"summary":"","24391":{"start_epoch":12,"start_stamp":"2024-02-10T08:33:25.211076+0000","gid":24391,"addr":"172.21.15.107:0/4187249279","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi107","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi107","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32537332","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"51e7526a-257b-41e2-b2a3-cd68464b4484","zone_name":"default","zonegroup_id":"5a6ddad7-fa42-479d-91fc-f2f04864c218","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 10 client.28827.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 7 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 7 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 3 7 2024-02-10T08:49:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 4 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 10 client.28827.objecter _send_command 7 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6813/2871825691 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f469800d320 con 0x7f4698010fc0 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.031+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698010fc0 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f469800d320 con 0x7f4698010fc0 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 7: 0 ) v1 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 7 = system:0 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 4 2024-02-10T08:49:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 3 7 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.032+0000 7f9f0c7a9700 10 librados: watch_flush enter 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 10 librados: watch_flush exit 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 20 client.54419.objecter shutdown clearing up homeless session... 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 10 client.54419.objecter successfully canceled tick 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9ef0063d10 legacy=0x7f9ef00661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 >> v1:172.21.15.107:6790/0 conn(0x7f9f04104330 legacy=0x7f9f041184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 shutdown_connections 2024-02-10T08:49:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 >> 172.21.15.107:0/1877917204 conn(0x7f9f040f3640 msgr2=0x7f9f041077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 shutdown_connections 2024-02-10T08:49:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 -- 172.21.15.107:0/1877917204 wait complete. 2024-02-10T08:49:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.033+0000 7f9f0c7a9700 1 librados: shutdown 2024-02-10T08:49:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 10 client.28827.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 8 2024-02-10T08:49:46.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 8 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469800cbb0 osd=3 3 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 3 8 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 4 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 10 client.28827.objecter _send_command 8 2024-02-10T08:49:46.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.047+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.107:6813/2871825691 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f46980122e0 con 0x7f4698010fc0 2024-02-10T08:49:46.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.048+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f46980122e0 con 0x7f4698010fc0 2024-02-10T08:49:46.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.048+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.048+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.048+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 4 2024-02-10T08:49:46.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.048+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 3 8 2024-02-10T08:49:46.131 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 10 client.28827.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 9 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 9 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 4 9 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 4 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 10 client.28827.objecter _send_command 9 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6800/3310789251 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f469801a3c0 con 0x7f4698016810 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.049+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698016810 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.050+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f469801a3c0 con 0x7f4698016810 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.050+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 9: 0 ) v1 2024-02-10T08:49:46.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.050+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 9 = system:0 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.050+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 4 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.050+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 4 9 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 10 client.28827.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 10 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 10 2024-02-10T08:49:46.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980122e0 osd=4 3 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 4 10 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 4 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 10 client.28827.objecter _send_command 10 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.066+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6800/3310789251 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f469801a3c0 con 0x7f4698016810 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f469801a3c0 con 0x7f4698016810 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 4 2024-02-10T08:49:46.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 4 10 2024-02-10T08:49:46.137 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-02-10T08:49:46.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 10 client.28827.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 11 2024-02-10T08:49:46.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 11 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 5 11 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 4 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 10 client.28827.objecter _send_command 11 2024-02-10T08:49:46.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6804/4178321486 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f469801b610 con 0x7f469801c0c0 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.068+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f469801c0c0 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.069+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f469801b610 con 0x7f469801c0c0 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.069+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 11: 0 ) v1 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.069+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 11 = system:0 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.069+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 4 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.069+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 5 11 2024-02-10T08:49:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 10 client.28827.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 12 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 12 2024-02-10T08:49:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469801a3c0 osd=5 3 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 5 12 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 4 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 10 client.28827.objecter _send_command 12 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.084+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6804/4178321486 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f469801b610 con 0x7f469801c0c0 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f469801b610 con 0x7f469801c0c0 2024-02-10T08:49:46.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 4 2024-02-10T08:49:46.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 5 12 2024-02-10T08:49:46.144 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 10 client.28827.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 13 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 13 2024-02-10T08:49:46.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 6 13 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 4 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 10 client.28827.objecter _send_command 13 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.086+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6808/1417479017 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46980271a0 con 0x7f4698023690 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698023690 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f46980271a0 con 0x7f4698023690 2024-02-10T08:49:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 13: 0 ) v1 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 13 = system:0 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 4 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.087+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 6 13 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 10 client.28827.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 14 2024-02-10T08:49:46.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 14 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f469801b610 osd=6 3 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 6 14 2024-02-10T08:49:46.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 4 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 10 client.28827.objecter _send_command 14 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.102+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6808/1417479017 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f46980271a0 con 0x7f4698023690 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f46980271a0 con 0x7f4698023690 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 4 2024-02-10T08:49:46.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 6 14 2024-02-10T08:49:46.151 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-02-10T08:49:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 10 client.28827.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:49:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 15 2024-02-10T08:49:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 15 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 7 15 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 4 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 10 client.28827.objecter _send_command 15 2024-02-10T08:49:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6812/1934176928 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f469802b510 con 0x7f4698027820 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.104+0000 7f46aa7fc700 10 client.28827.objecter ms_handle_connect 0x7f4698027820 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.105+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f469802b510 con 0x7f4698027820 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.105+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 15: 0 ) v1 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.105+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 15 = system:0 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.105+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 4 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.105+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 7 15 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 10 client.28827.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-10T08:49:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign -1 16 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 20 client.28827.objecter _get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_remove -1 16 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 20 client.28827.objecter get_session s=0x7f46980271a0 osd=7 3 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 15 client.28827.objecter _session_command_op_assign 7 16 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 4 2024-02-10T08:49:46.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 10 client.28827.objecter _send_command 16 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.121+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 --> v1:172.21.15.195:6812/1934176928 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f469802ccc0 con 0x7f4698027820 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.123+0000 7f46aa7fc700 1 -- 172.21.15.107:0/4211902737 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f469802ccc0 con 0x7f4698027820 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.123+0000 7f46aa7fc700 10 client.28827.objecter ms_dispatch 0x7f46b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.123+0000 7f46aa7fc700 10 client.28827.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.123+0000 7f46aa7fc700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 4 2024-02-10T08:49:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.123+0000 7f46aa7fc700 15 client.28827.objecter _session_command_op_remove 7 16 2024-02-10T08:49:46.157 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-02-10T08:49:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 librados: watch_flush enter 2024-02-10T08:49:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 librados: watch_flush exit 2024-02-10T08:49:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.0 2024-02-10T08:49:46.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6801/3321596972 conn(0x7f4698004530 legacy=0x7f4698002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980051a0 osd=0 1 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.1 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6805/133516126 conn(0x7f46980088d0 legacy=0x7f469800acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698004020 osd=1 1 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.2 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6809/3575881728 conn(0x7f4698003b70 legacy=0x7f469800f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f4698001c00 osd=2 1 2024-02-10T08:49:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.3 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6813/2871825691 conn(0x7f4698010fc0 legacy=0x7f4698011400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469800cbb0 osd=3 1 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.4 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.195:6800/3310789251 conn(0x7f4698016810 legacy=0x7f4698018c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46b98a7700 1 -- 172.21.15.107:0/4211902737 reap_dead start 2024-02-10T08:49:46.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980122e0 osd=4 1 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.5 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.195:6804/4178321486 conn(0x7f469801c0c0 legacy=0x7f469801c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46b98a7700 1 -- 172.21.15.107:0/4211902737 reap_dead start 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801a3c0 osd=5 1 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.6 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.195:6808/1417479017 conn(0x7f4698023690 legacy=0x7f4698025a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f469801b610 osd=6 1 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter close_session for osd.7 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.195:6812/1934176928 conn(0x7f4698027820 legacy=0x7f4698029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter put_session s=0x7f46980271a0 osd=7 1 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 20 client.28827.objecter shutdown clearing up homeless session... 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 10 client.28827.objecter successfully canceled tick 2024-02-10T08:49:46.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46b98a7700 1 -- 172.21.15.107:0/4211902737 reap_dead start 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6800/2634842009 conn(0x7f46a0063c00 legacy=0x7f46a00660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> v1:172.21.15.107:6789/0 conn(0x7f46b40f9570 legacy=0x7f46b40ffe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 shutdown_connections 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 >> 172.21.15.107:0/4211902737 conn(0x7f46b40f2510 msgr2=0x7f46b41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 shutdown_connections 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 -- 172.21.15.107:0/4211902737 wait complete. 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:46.124+0000 7f46bb30a700 1 librados: shutdown 2024-02-10T08:49:46.163 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-10T08:49:46.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T08:49:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:45.578211+0000 mgr.y (mgr.14553) 666 : cluster [DBG] pgmap v1227: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:45.977009+0000 mon.a (mon.0) 3699 : audit [DBG] from='client.? v1:172.21.15.107:0/4211902737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:46.031563+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? v1:172.21.15.107:0/1877917204' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:49:46.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:46.195823+0000 mon.a (mon.0) 3700 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:45.578211+0000 mgr.y (mgr.14553) 666 : cluster [DBG] pgmap v1227: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:45.977009+0000 mon.a (mon.0) 3699 : audit [DBG] from='client.? v1:172.21.15.107:0/4211902737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:46.031563+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? v1:172.21.15.107:0/1877917204' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:49:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:46.195823+0000 mon.a (mon.0) 3700 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:45.578211+0000 mgr.y (mgr.14553) 666 : cluster [DBG] pgmap v1227: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:45.977009+0000 mon.a (mon.0) 3699 : audit [DBG] from='client.? v1:172.21.15.107:0/4211902737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T08:49:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:46.031563+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? v1:172.21.15.107:0/1877917204' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T08:49:47.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:46.195823+0000 mon.a (mon.0) 3700 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T08:49:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:48.103 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-10T08:49:48.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:49:48.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:47.579193+0000 mgr.y (mgr.14553) 667 : cluster [DBG] pgmap v1228: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:48.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:47.646989+0000 mon.c (mon.2) 493 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:49:48.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:47.892667+0000 mon.a (mon.0) 3701 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:48.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:47.579193+0000 mgr.y (mgr.14553) 667 : cluster [DBG] pgmap v1228: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:48.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:47.646989+0000 mon.c (mon.2) 493 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:49:48.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:47.892667+0000 mon.a (mon.0) 3701 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:47.579193+0000 mgr.y (mgr.14553) 667 : cluster [DBG] pgmap v1228: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-10T08:49:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:47.646989+0000 mon.c (mon.2) 493 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:49:49.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:47.892667+0000 mon.a (mon.0) 3701 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:4742 197.c 197:30000000:test-rados-api-smithi107-112490-46:::head [pgnls start_epoch 347] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e347)", 2024-02-10T08:49:49.669 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 547.75994464300004, 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004231037, 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4742 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013951+0000", 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013947+0000", 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5713999999999999e-05 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013963+0000", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.173e-05 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013985+0000", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3070000000000003e-06 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.013993+0000", 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5900999999999999e-05 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.014049+0000", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1820000000000006e-06 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:41.014057+0000", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.003927923 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.017985+0000", 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5151000000000003e-05 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.018020+0000", 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016174900000000001 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:40:42.018182+0000", 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004231037 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.674 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.34349.0:1659 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94547991864448 gen 1] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:43:35.184846+0000", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 373.58904948999998, 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.90368862400000005, 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.34349", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1659 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184846+0000", 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184846+0000", 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999957 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184842+0000", 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4863e-05 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184857+0000", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7628000000000001e-05 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184874+0000", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9299999999999998e-06 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184884+0000", 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9979999999999998e-05 2024-02-10T08:49:49.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184944+0000", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0978000000000001e-05 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:35.184955+0000", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.89299231400000001 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:36.077947+0000", 2024-02-10T08:49:49.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000133125 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:36.078081+0000", 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010454088 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:43:36.088535+0000", 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.90368862400000005 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7300 222.d 222:b73f5add:test-rados-api-smithi107-112490-82::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e511)", 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:17.178934+0000", 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 331.59496204499999, 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043856328999999999, 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7300 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178934+0000", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178934+0000", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178930+0000", 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6042999999999999e-05 2024-02-10T08:49:49.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178946+0000", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9868e-05 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178966+0000", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1900000000000001e-06 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.178975+0000", 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9697999999999997e-05 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179025+0000", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7692e-05 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.179112+0000", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034087698999999999 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.213200+0000", 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093634000000000002 2024-02-10T08:49:49.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.214136+0000", 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025802279999999999 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.216716+0000", 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00598585 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:49:49.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.222702+0000", 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8365000000000002e-05 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.222751+0000", 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9156999999999998e-05 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:17.222790+0000", 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.043856328999999999 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7302 222.d 222:b73f5add:test-rados-api-smithi107-112490-82::foo:head [write 0~8] snapc 2=[2] ondisk+write+known_if_redirected e512)", 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:18.174885+0000", 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 330.59901011400001, 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.074097647000000003, 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:49:49.686 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7302 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174885+0000", 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174885+0000", 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999967 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174882+0000", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3788000000000001e-05 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174896+0000", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9018999999999999e-05 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174915+0000", 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5519999999999999e-06 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174922+0000", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7389999999999999e-05 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174970+0000", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2409999999999998e-06 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.174977+0000", 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049766077999999998 2024-02-10T08:49:49.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.224743+0000", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3789999999999995e-05 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.224837+0000", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018236500000000001 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.225019+0000", 2024-02-10T08:49:49.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077170659999999999 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.232736+0000", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010986888 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.243723+0000", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051906349999999999 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.248914+0000", 2024-02-10T08:49:49.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0375e-05 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.248944+0000", 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8983000000000003e-05 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:18.248983+0000", 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.074097647000000003 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.692 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7304 222.d 222:b73f5add:test-rados-api-smithi107-112490-82::foo:head [write 0~8] snapc 3=[3,2] ondisk+write+known_if_redirected e513)", 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:19.168030+0000", 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 329.605865967, 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082530107000000005, 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7304 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168030+0000", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168030+0000", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999952 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168025+0000", 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9440999999999999e-05 2024-02-10T08:49:49.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168044+0000", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4090000000000001e-05 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168068+0000", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6120000000000005e-06 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.168078+0000", 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005640824 2024-02-10T08:49:49.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.173719+0000", 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4590000000000001e-06 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.173726+0000", 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041446263999999997 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.215173+0000", 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9718999999999999e-05 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.215272+0000", 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019067300000000001 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.215463+0000", 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019250015999999998 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.234713+0000", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.114e-06 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.234720+0000", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015746506 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.250467+0000", 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2629e-05 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.250519+0000", 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0450000000000001e-05 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:19.250560+0000", 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082530107000000005 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.25339.0:7523 225.9 225:96e5096e:test-rados-api-smithi107-112490-87::602c57ffb51af99d6f3b54c0ee9587bb110fb990:head [writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:44:32.138866+0000", 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 316.63502995699997, 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024044928, 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.25339", 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:49:49.700 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7523 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138866+0000", 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138866+0000", 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138863+0000", 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2599999999999994e-06 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138872+0000", 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6050999999999999e-05 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138889+0000", 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5879999999999998e-06 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.138897+0000", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038261129999999999 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.142723+0000", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3080000000000004e-06 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.142730+0000", 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013747278999999999 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.156477+0000", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011975499999999999 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.156597+0000", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000143478 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.156740+0000", 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035557499999999999 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.160296+0000", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024865600000000001 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.162782+0000", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7710000000000002e-05 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.162830+0000", 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0456000000000004e-05 2024-02-10T08:49:49.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:44:32.162911+0000", 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024044928 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.1.13:93 234.1s0 234:80ca6862:test-rados-api-smithi107-112490-105::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e611)", 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:45:57.248708+0000", 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.52518709500001, 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.060539084999999999, 2024-02-10T08:49:49.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248708+0000", 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248708+0000", 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248706+0000", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4709999999999996e-06 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248716+0000", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7487999999999999e-05 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248733+0000", 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6219999999999998e-06 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248742+0000", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044520230000000003 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.253194+0000", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1999999999999994e-06 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.253202+0000", 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.055808199000000003 2024-02-10T08:49:49.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.309010+0000", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4228000000000007e-05 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.309105+0000", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014301200000000001 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.309248+0000", 2024-02-10T08:49:49.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.060539084999999999 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.1.13:94 234.1s0 234:80ca6862:test-rados-api-smithi107-112490-105::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e611)", 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:45:57.248755+0000", 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 231.525140956, 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.065341514000000003, 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248755+0000", 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248755+0000", 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248754+0000", 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3319999999999999e-06 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248759+0000", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5990000000000001e-06 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248765+0000", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0160000000000002e-06 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.248769+0000", 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004478236 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.253247+0000", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3349999999999997e-06 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.253252+0000", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056019215999999997 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.309272+0000", 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7272000000000002e-05 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.309339+0000", 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047571829999999999 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:45:57.314096+0000", 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.065341514000000003 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:942 241.c 241:31974020:test-rados-api-smithi107-112490-115::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e668)", 2024-02-10T08:49:49.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:03.283233+0000", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 165.49066305299999, 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048350199999999998, 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 942 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283233+0000", 2024-02-10T08:49:49.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283233+0000", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283231+0000", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9589999999999998e-06 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283237+0000", 2024-02-10T08:49:49.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2707999999999999e-05 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283250+0000", 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3550000000000001e-06 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283257+0000", 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5431e-05 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283292+0000", 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013637799999999999 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.283428+0000", 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011321879999999999 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.284561+0000", 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2254000000000002e-05 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.284643+0000", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038341999999999998 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.285026+0000", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0751999999999998e-05 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.285077+0000", 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002990562 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288068+0000", 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048350199999999998 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:943 241.c 241:31974020:test-rados-api-smithi107-112490-115::bar:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e668)", 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:03.288200+0000", 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 165.48569566500001, 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050659490000000001, 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.721 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 943 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288200+0000", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288200+0000", 2024-02-10T08:49:49.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.999999 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288199+0000", 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7389999999999999e-06 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288203+0000", 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2739999999999996e-06 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288213+0000", 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.066e-06 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288219+0000", 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8738000000000002e-05 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288248+0000", 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119515 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for blocked object", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.288367+0000", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00097452599999999997 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.289342+0000", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7737999999999993e-05 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.289419+0000", 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090085600000000001 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.290320+0000", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7623e-05 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.290378+0000", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028880479999999998 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:03.293266+0000", 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050659490000000001 2024-02-10T08:49:49.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(234.0s2 669/633 ECSubWrite(tid=20, reqid=osd.4.0:20, at_version=669'15, trim_to=0'0, roll_forward_to=665'13))", 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:04.266175+0000", 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 164.507720861, 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028400330000000001, 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266175+0000", 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266175+0000", 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266174+0000", 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8024000000000001e-05 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266202+0000", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6364000000000001e-05 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266219+0000", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9919999999999996e-06 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266228+0000", 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0326000000000002e-05 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266318+0000", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8439999999999998e-06 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.266326+0000", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026643280000000001 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.268990+0000", 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4542999999999999e-05 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:04.269015+0000", 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028400330000000001 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:989 241.c 241:30000000::::head [pgnls start_epoch 671] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e671)", 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:06.146078+0000", 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 162.62781771799999, 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021667460000000002, 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 989 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146078+0000", 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146078+0000", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146075+0000", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.397e-06 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146085+0000", 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2413e-05 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146097+0000", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7590000000000003e-06 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146103+0000", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0241999999999999e-05 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146143+0000", 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2024-02-10T08:49:49.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.146150+0000", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020132710000000001 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.148163+0000", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0679e-05 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.148184+0000", 2024-02-10T08:49:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0908e-05 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:06.148245+0000", 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021667460000000002 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:92 234.4s0 234:3fcfbd52:test-rados-api-smithi107-112490-121::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e693)", 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:33.158668+0000", 2024-02-10T08:49:49.736 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 135.61522708000001, 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024624320000000001, 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158668+0000", 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158668+0000", 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999924 2024-02-10T08:49:49.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158661+0000", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5364000000000001e-05 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158676+0000", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3756e-05 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158690+0000", 2024-02-10T08:49:49.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1720000000000001e-06 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158697+0000", 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4857000000000001e-05 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158742+0000", 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0458999999999999e-05 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158753+0000", 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002303336 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.161056+0000", 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4881000000000004e-05 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.161131+0000", 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024624320000000001 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(osd.5.37:93 234.4s0 234:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e693)", 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:47:33.158704+0000", 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 135.615191181, 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002502382, 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158704+0000", 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158704+0000", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158704+0000", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8999999999999997e-06 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158709+0000", 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4599999999999999e-06 2024-02-10T08:49:49.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158712+0000", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5069999999999999e-06 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158715+0000", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8601000000000002e-05 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158793+0000", 2024-02-10T08:49:49.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0040000000000001e-06 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.158797+0000", 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023462270000000002 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.161143+0000", 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3311000000000004e-05 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:47:33.161207+0000", 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002502382 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.48721.0:1928 247.1b e723/719 247:dae49067:test-rados-api-smithi107-112490-127::foo:head v 723'2, mlcod=0'0)", 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:48:03.172930+0000", 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.600965544, 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021374442, 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.172930+0000", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.172930+0000", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999986 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.172929+0000", 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019102114 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.192031+0000", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.867e-06 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.192040+0000", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.502e-06 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.192045+0000", 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7786e-05 2024-02-10T08:49:49.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.192093+0000", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6290000000000002e-05 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.192109+0000", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021650060000000001 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.194274+0000", 2024-02-10T08:49:49.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0159e-05 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:03.194304+0000", 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021374442 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(16.18 RELEASE e726)", 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:48:06.491348+0000", 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.28254730800001, 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031261650000000002, 2024-02-10T08:49:49.749 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491348+0000", 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491348+0000", 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491348+0000", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2449999999999998e-06 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491351+0000", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3300000000000005e-06 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491360+0000", 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6690000000000003e-06 2024-02-10T08:49:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491366+0000", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013147899999999999 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491497+0000", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.443e-06 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.491504+0000", 2024-02-10T08:49:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029353370000000001 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.494439+0000", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2595000000000001e-05 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.494452+0000", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2552e-05 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:06.494474+0000", 2024-02-10T08:49:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031261650000000002 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:2446 234.1s0 234:9b54dbdc:test-rados-api-smithi107-112490-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e763)", 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:48:50.187745+0000", 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.586150224000001, 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010145239, 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.754 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2446 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187745+0000", 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187745+0000", 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999971 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187743+0000", 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0342e-05 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187753+0000", 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3730999999999999e-05 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187767+0000", 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0309999999999998e-06 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187775+0000", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9777000000000001e-05 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187825+0000", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000148508 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.187973+0000", 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000495076 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.188468+0000", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082194950000000003 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.196688+0000", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011398319999999999 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.197827+0000", 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3100000000000002e-05 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:48:50.197891+0000", 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010145239 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:2880 252.c 252:30000000:test-rados-api-smithi107-112490-137:::head [pgnls start_epoch 777] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e777)", 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:49:10.236296+0000", 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.537599147000002, 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022775759999999999, 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.759 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2880 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236296+0000", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236296+0000", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999976 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236294+0000", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0212e-05 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236304+0000", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4782e-05 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236319+0000", 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6950000000000003e-06 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236326+0000", 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2205e-05 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236368+0000", 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8039999999999998e-06 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.236374+0000", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020879259999999999 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.238462+0000", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2775999999999999e-05 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.238494+0000", 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9690000000000004e-05 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:10.238574+0000", 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022775759999999999 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:3045 234.4s0 234:3a54c58e:test-rados-api-smithi107-112490-139::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e790)", 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:49:29.216704+0000", 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.557191306, 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051820779999999997, 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.764 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3045 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216704+0000", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216704+0000", 2024-02-10T08:49:49.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999981 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216702+0000", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.259e-06 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216711+0000", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.077e-05 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216721+0000", 2024-02-10T08:49:49.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8540000000000002e-06 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216727+0000", 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9404999999999999e-05 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216777+0000", 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141679 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.216918+0000", 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039665099999999998 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.217315+0000", 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047445099999999997 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.217789+0000", 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040393950000000003 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.221829+0000", 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7568999999999997e-05 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:29.221886+0000", 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051820779999999997 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.769 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.48721.0:3047 234.4s0 234:3a54c58e:test-rados-api-smithi107-112490-139::foo:head [set-redirect] snapc 0=[] ondisk+write+known_if_redirected e791)", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-02-10T08:49:30.212399+0000", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.561496611999999, 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035373570000000001, 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.48721", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3047 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212399+0000", 2024-02-10T08:49:49.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212399+0000", 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4294967295.9999962 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212395+0000", 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3648999999999999e-05 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-02-10T08:49:49.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212409+0000", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7306999999999999e-05 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212426+0000", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4930000000000004e-06 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212436+0000", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8520000000000003e-05 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212484+0000", 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3080000000000004e-06 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.212491+0000", 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021937850000000002 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.214684+0000", 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7571999999999997e-05 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.214742+0000", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000249666 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.214992+0000", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029205700000000003 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.215284+0000", 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061713999999999996 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.215901+0000", 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5471e-05 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-02-10T08:49:30.215936+0000", 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035373570000000001 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-02-10T08:49:49.775 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-02-10T08:49:49.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:49.159178+0000 mon.c (mon.2) 494 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:49:49.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:49.159782+0000 mon.c (mon.2) 495 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:49:49.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:49.164317+0000 mon.a (mon.0) 3702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:49.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:49.159178+0000 mon.c (mon.2) 494 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:49:49.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:49.159782+0000 mon.c (mon.2) 495 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:49:49.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:49.164317+0000 mon.a (mon.0) 3702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:49.159178+0000 mon.c (mon.2) 494 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:49:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:49.159782+0000 mon.c (mon.2) 495 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:49:50.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:49.164317+0000 mon.a (mon.0) 3702 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:49:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:49:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:49:50.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T08:49:50.865 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:49:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:50.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- 172.21.15.107:0/2688437412 >> v1:172.21.15.107:6790/0 conn(0x7fe9680f89c0 legacy=0x7fe9680f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:50.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- 172.21.15.107:0/2688437412 shutdown_connections 2024-02-10T08:49:50.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- 172.21.15.107:0/2688437412 >> 172.21.15.107:0/2688437412 conn(0x7fe9680f2510 msgr2=0x7fe9680f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:50.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- 172.21.15.107:0/2688437412 shutdown_connections 2024-02-10T08:49:50.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- 172.21.15.107:0/2688437412 wait complete. 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 librados: starting msgr at 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 librados: starting objecter 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 Processor -- start 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 -- start start 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 librados: setting wanted keys 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.880+0000 7fe96fd12700 1 librados: calling monclient init 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96fd12700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe968102b00 con 0x7fe968104350 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96fd12700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe968102cd0 con 0x7fe9680f89c0 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96fd12700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe968102ec0 con 0x7fe9680f9570 2024-02-10T08:49:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96d2ad700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe9680f9570 0x7fe968114da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46376/0 (socket says 172.21.15.107:46376) 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96d2ad700 1 -- 172.21.15.107:0/2972612683 learned_addr learned my addr 172.21.15.107:0/2972612683 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe96e2af700 1 --1- 172.21.15.107:0/2972612683 >> v1:172.21.15.107:6789/0 conn(0x7fe968104350 0x7fe9681184d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39838/0 (socket says 172.21.15.107:39838) 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter ms_handle_connect 0x7fe968104350 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter ms_handle_connect 0x7fe9680f89c0 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter ms_handle_connect 0x7fe9680f9570 2024-02-10T08:49:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868502723 0 0) 0x7fe968102b00 con 0x7fe968104350 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe954003800 con 0x7fe968104350 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230909115 0 0) 0x7fe968102cd0 con 0x7fe9680f89c0 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968102b00 con 0x7fe9680f89c0 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448853940 0 0) 0x7fe968102ec0 con 0x7fe9680f9570 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968102cd0 con 0x7fe9680f9570 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521820175 0 0) 0x7fe954003800 con 0x7fe968104350 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe968102ec0 con 0x7fe968104350 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.881+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe960001d70 con 0x7fe968104350 2024-02-10T08:49:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3883561228 0 0) 0x7fe968102b00 con 0x7fe9680f89c0 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe954003800 con 0x7fe9680f89c0 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe964002f20 con 0x7fe9680f89c0 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938416774 0 0) 0x7fe968102cd0 con 0x7fe9680f9570 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe968102b00 con 0x7fe9680f9570 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe958001840 con 0x7fe9680f9570 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 640981545 0 0) 0x7fe968102ec0 con 0x7fe968104350 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 >> v1:172.21.15.107:6790/0 conn(0x7fe9680f9570 legacy=0x7fe968114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 >> v1:172.21.15.195:6789/0 conn(0x7fe9680f89c0 legacy=0x7fe968102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96811cc20 con 0x7fe968104350 2024-02-10T08:49:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe968119c50 con 0x7fe968104350 2024-02-10T08:49:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe96fd12700 10 client.28833.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96811a0c0 con 0x7fe968104350 2024-02-10T08:49:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.882+0000 7fe96fd12700 1 librados: init done 2024-02-10T08:49:50.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.883+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe9600027b0 con 0x7fe968104350 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.883+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe960005da0 con 0x7fe968104350 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.883+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe960018d10 con 0x7fe968104350 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c004f20 con 0x7fe968104350 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(805..805 src has 754..805) v4 ==== 7386+0+0 (unknown 3422392587 0 0) 0x7fe960079c00 con 0x7fe968104350 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 10 client.28833.objecter ms_dispatch 0x7fe968001550 osd_map(805..805 src has 754..805) v4 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 3 client.28833.objecter handle_osd_map got epochs [805,805] > 0 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 3 client.28833.objecter handle_osd_map decoding full epoch 805 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 20 client.28833.objecter dump_active .. 0 homeless 2024-02-10T08:49:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.884+0000 7fe95effd700 10 client.28833.objecter ms_handle_connect 0x7fe954063cb0 2024-02-10T08:49:50.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:50.890+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe96004d2d0 con 0x7fe968104350 2024-02-10T08:49:50.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:49.579602+0000 mgr.y (mgr.14553) 668 : cluster [DBG] pgmap v1229: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-10T08:49:50.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:49.579602+0000 mgr.y (mgr.14553) 668 : cluster [DBG] pgmap v1229: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-10T08:49:51.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:49.579602+0000 mgr.y (mgr.14553) 668 : cluster [DBG] pgmap v1229: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-10T08:49:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.070+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fe94c0051d0 con 0x7fe968104350 2024-02-10T08:49:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.649+0000 7fe95effd700 1 -- 172.21.15.107:0/2972612683 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v806) v1 ==== 91+0+0 (unknown 3690968392 0 0) 0x7fe96004b6e0 con 0x7fe968104350 2024-02-10T08:49:51.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 10 librados: watch_flush enter 2024-02-10T08:49:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 10 librados: watch_flush exit 2024-02-10T08:49:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 20 client.28833.objecter shutdown clearing up homeless session... 2024-02-10T08:49:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 10 client.28833.objecter successfully canceled tick 2024-02-10T08:49:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe954063cb0 legacy=0x7fe954066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:51.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.651+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 >> v1:172.21.15.107:6789/0 conn(0x7fe968104350 legacy=0x7fe9681184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.652+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 shutdown_connections 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.652+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 >> 172.21.15.107:0/2972612683 conn(0x7fe9680f2510 msgr2=0x7fe968103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.652+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 shutdown_connections 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.653+0000 7fe96fd12700 1 -- 172.21.15.107:0/2972612683 wait complete. 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:51.653+0000 7fe96fd12700 1 librados: shutdown 2024-02-10T08:49:51.653 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-02-10T08:49:51.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:51.071603+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:51.071603+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:52.025 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:51.071603+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T08:49:52.025 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:49:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:49:52.559 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:49:52.559 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:49:52.559 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:49:52.560 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:51.580349+0000 mgr.y (mgr.14553) 669 : cluster [DBG] pgmap v1230: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:51.638706+0000 mon.a (mon.0) 3704 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:51.649723+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:51.649768+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:51.860048+0000 mgr.y (mgr.14553) 670 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:51.891884+0000 osd.7 (osd.7) 332 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:51.894799+0000 osd.7 (osd.7) 333 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:49:52.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:52.184707+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:49:52.895 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:52.188300+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:49:52.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:51.580349+0000 mgr.y (mgr.14553) 669 : cluster [DBG] pgmap v1230: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:52.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:51.638706+0000 mon.a (mon.0) 3704 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-10T08:49:52.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:51.649723+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:51.649768+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:51.860048+0000 mgr.y (mgr.14553) 670 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:51.891884+0000 osd.7 (osd.7) 332 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:51.894799+0000 osd.7 (osd.7) 333 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:52.184707+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:52.188300+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:51.580349+0000 mgr.y (mgr.14553) 669 : cluster [DBG] pgmap v1230: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:51.638706+0000 mon.a (mon.0) 3704 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:51.649723+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.107:0/2972612683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:51.649768+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-10T08:49:52.921 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:51.860048+0000 mgr.y (mgr.14553) 670 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:49:52.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:51.891884+0000 osd.7 (osd.7) 332 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:49:52.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:51.894799+0000 osd.7 (osd.7) 333 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:49:52.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:52.184707+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:49:52.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:52.188300+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:49:52.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T08:49:53.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.467+0000 7fb754b8b700 1 -- 172.21.15.107:0/4261626037 >> v1:172.21.15.107:6790/0 conn(0x7fb7501007f0 legacy=0x7fb750102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.467+0000 7fb754b8b700 1 -- 172.21.15.107:0/4261626037 shutdown_connections 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.467+0000 7fb754b8b700 1 -- 172.21.15.107:0/4261626037 >> 172.21.15.107:0/4261626037 conn(0x7fb7500f3640 msgr2=0x7fb7500f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.467+0000 7fb754b8b700 1 -- 172.21.15.107:0/4261626037 shutdown_connections 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.467+0000 7fb754b8b700 1 -- 172.21.15.107:0/4261626037 wait complete. 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 librados: starting msgr at 2024-02-10T08:49:53.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 librados: starting objecter 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 Processor -- start 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 -- start start 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 librados: setting wanted keys 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 librados: calling monclient init 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7501174a0 con 0x7fb750104330 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb750118680 con 0x7fb7500fda50 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb754b8b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb750119880 con 0x7fb7501007f0 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb74ed9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb750104330 0x7fb750115b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39854/0 (socket says 172.21.15.107:39854) 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb74ed9d700 1 -- 172.21.15.107:0/1588507206 learned_addr learned my addr 172.21.15.107:0/1588507206 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb74dd9b700 1 --1- 172.21.15.107:0/1588507206 >> v1:172.21.15.107:6790/0 conn(0x7fb7501007f0 0x7fb7500fa480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46390/0 (socket says 172.21.15.107:46390) 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter ms_handle_connect 0x7fb750104330 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:53.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter ms_handle_connect 0x7fb7500fda50 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter ms_handle_connect 0x7fb7501007f0 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.468+0000 7fb73f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022626338 0 0) 0x7fb750118680 con 0x7fb7500fda50 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7380033f0 con 0x7fb7500fda50 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46185523 0 0) 0x7fb7501174a0 con 0x7fb750104330 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750118680 con 0x7fb750104330 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395897888 0 0) 0x7fb750119880 con 0x7fb7501007f0 2024-02-10T08:49:53.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7501174a0 con 0x7fb7501007f0 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2049717343 0 0) 0x7fb7380033f0 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb750119880 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb740001870 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419991917 0 0) 0x7fb7501174a0 con 0x7fb7501007f0 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7380033f0 con 0x7fb7501007f0 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346114043 0 0) 0x7fb750118680 con 0x7fb750104330 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7501174a0 con 0x7fb750104330 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb744001ec0 con 0x7fb7501007f0 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb748002c50 con 0x7fb750104330 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1332785017 0 0) 0x7fb750119880 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 >> v1:172.21.15.107:6790/0 conn(0x7fb7501007f0 legacy=0x7fb7500fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 >> v1:172.21.15.107:6789/0 conn(0x7fb750104330 legacy=0x7fb750115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75011aa80 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb750118870 con 0x7fb7500fda50 2024-02-10T08:49:53.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb754b8b700 10 client.49153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:53.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb750118d60 con 0x7fb7500fda50 2024-02-10T08:49:53.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb754b8b700 1 librados: init done 2024-02-10T08:49:53.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb740001f30 con 0x7fb7500fda50 2024-02-10T08:49:53.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.469+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb740003270 con 0x7fb7500fda50 2024-02-10T08:49:53.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.470+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb730004f20 con 0x7fb7500fda50 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb740016200 con 0x7fb7500fda50 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(806..806 src has 754..806) v4 ==== 7386+0+0 (unknown 673814851 0 0) 0x7fb7400763a0 con 0x7fb7500fda50 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 10 client.49153.objecter ms_dispatch 0x7fb750001550 osd_map(806..806 src has 754..806) v4 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 3 client.49153.objecter handle_osd_map got epochs [806,806] > 0 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 3 client.49153.objecter handle_osd_map decoding full epoch 806 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 20 client.49153.objecter dump_active .. 0 homeless 2024-02-10T08:49:53.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.471+0000 7fb73f7fe700 10 client.49153.objecter ms_handle_connect 0x7fb738063d20 2024-02-10T08:49:53.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.476+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb740049b80 con 0x7fb7500fda50 2024-02-10T08:49:53.491 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:49:53.499 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:53.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.645+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb730000b90 con 0x7fb738063d20 2024-02-10T08:49:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.657+0000 7fb73f7fe700 1 -- 172.21.15.107:0/1588507206 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+410100 (unknown 2965378022 0 1065701330) 0x7fb730000b90 con 0x7fb738063d20 2024-02-10T08:49:53.663 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:49:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 10 librados: watch_flush enter 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 10 librados: watch_flush exit 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 20 client.49153.objecter shutdown clearing up homeless session... 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 10 client.49153.objecter successfully canceled tick 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb738063d20 legacy=0x7fb7380661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 >> v1:172.21.15.195:6789/0 conn(0x7fb7500fda50 legacy=0x7fb7500f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 shutdown_connections 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 >> 172.21.15.107:0/1588507206 conn(0x7fb7500f3640 msgr2=0x7fb7501077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 shutdown_connections 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 -- 172.21.15.107:0/1588507206 wait complete. 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:53.660+0000 7fb754b8b700 1 librados: shutdown 2024-02-10T08:49:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:49:54.557 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1232,"stamp":"2024-02-10T08:49:53.580546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1347,"num_read_kb":1099,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":13194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":462,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2469784,"kb_used_data":53080,"kb_used_omap":59,"kb_used_meta":2416580,"kb_avail":747491432,"statfs":{"total":767960285184,"available":765431226368,"internally_reserved":0,"allocated":54353920,"data_stored":28894763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2474578539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.370091"},"pg_stats":[{"pgid":"5.9","version":"636'18","reported_seq":499,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416599+0000","last_change":"2024-02-10T08:48:29.848382+0000","last_active":"2024-02-10T08:49:44.416599+0000","last_peered":"2024-02-10T08:49:44.416599+0000","last_clean":"2024-02-10T08:49:44.416599+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:49:44.416599+0000","last_undegraded":"2024-02-10T08:49:44.416599+0000","last_fullsized":"2024-02-10T08:49:44.416599+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:48:29.848301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.848301+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":871,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631166+0000","last_change":"2024-02-10T08:47:58.959348+0000","last_active":"2024-02-10T08:49:43.631166+0000","last_peered":"2024-02-10T08:49:43.631166+0000","last_clean":"2024-02-10T08:49:43.631166+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:49:43.631166+0000","last_undegraded":"2024-02-10T08:49:43.631166+0000","last_fullsized":"2024-02-10T08:49:43.631166+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:47:58.959271+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.959271+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":809,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210292+0000","last_change":"2024-02-10T08:49:41.119323+0000","last_active":"2024-02-10T08:49:43.210292+0000","last_peered":"2024-02-10T08:49:43.210292+0000","last_clean":"2024-02-10T08:49:43.210292+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:49:43.210292+0000","last_undegraded":"2024-02-10T08:49:43.210292+0000","last_fullsized":"2024-02-10T08:49:43.210292+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:41.119237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:49:41.119237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":790,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631040+0000","last_change":"2024-02-10T08:48:01.033569+0000","last_active":"2024-02-10T08:49:43.631040+0000","last_peered":"2024-02-10T08:49:43.631040+0000","last_clean":"2024-02-10T08:49:43.631040+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:49:43.631040+0000","last_undegraded":"2024-02-10T08:49:43.631040+0000","last_fullsized":"2024-02-10T08:49:43.631040+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:01.033502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:01.033502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":750,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210726+0000","last_change":"2024-02-10T08:49:23.108122+0000","last_active":"2024-02-10T08:49:43.210726+0000","last_peered":"2024-02-10T08:49:43.210726+0000","last_clean":"2024-02-10T08:49:43.210726+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:49:43.210726+0000","last_undegraded":"2024-02-10T08:49:43.210726+0000","last_fullsized":"2024-02-10T08:49:43.210726+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.108061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":252,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212126+0000","last_change":"2024-02-10T08:48:32.371647+0000","last_active":"2024-02-10T08:49:43.212126+0000","last_peered":"2024-02-10T08:49:43.212126+0000","last_clean":"2024-02-10T08:49:43.212126+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:49:43.212126+0000","last_undegraded":"2024-02-10T08:49:43.212126+0000","last_fullsized":"2024-02-10T08:49:43.212126+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.371572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.371572+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210687+0000","last_change":"2024-02-10T08:48:38.645376+0000","last_active":"2024-02-10T08:49:43.210687+0000","last_peered":"2024-02-10T08:49:43.210687+0000","last_clean":"2024-02-10T08:49:43.210687+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:49:43.210687+0000","last_undegraded":"2024-02-10T08:49:43.210687+0000","last_fullsized":"2024-02-10T08:49:43.210687+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.645301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.645301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":815,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210387+0000","last_change":"2024-02-10T08:48:56.119888+0000","last_active":"2024-02-10T08:49:43.210387+0000","last_peered":"2024-02-10T08:49:43.210387+0000","last_clean":"2024-02-10T08:49:43.210387+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:49:43.210387+0000","last_undegraded":"2024-02-10T08:49:43.210387+0000","last_fullsized":"2024-02-10T08:49:43.210387+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:56.119799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.119799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":193,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211797+0000","last_change":"2024-02-10T08:49:25.635459+0000","last_active":"2024-02-10T08:49:43.211797+0000","last_peered":"2024-02-10T08:49:43.211797+0000","last_clean":"2024-02-10T08:49:43.211797+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:49:43.211797+0000","last_undegraded":"2024-02-10T08:49:43.211797+0000","last_fullsized":"2024-02-10T08:49:43.211797+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.635394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.635394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":739,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.633087+0000","last_change":"2024-02-10T08:48:55.016819+0000","last_active":"2024-02-10T08:49:43.633087+0000","last_peered":"2024-02-10T08:49:43.633087+0000","last_clean":"2024-02-10T08:49:43.633087+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:49:43.633087+0000","last_undegraded":"2024-02-10T08:49:43.633087+0000","last_fullsized":"2024-02-10T08:49:43.633087+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.016738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.016738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":964,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631692+0000","last_change":"2024-02-10T08:48:38.016707+0000","last_active":"2024-02-10T08:49:43.631692+0000","last_peered":"2024-02-10T08:49:43.631692+0000","last_clean":"2024-02-10T08:49:43.631692+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:49:43.631692+0000","last_undegraded":"2024-02-10T08:49:43.631692+0000","last_fullsized":"2024-02-10T08:49:43.631692+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:48:38.016615+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.016615+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":807,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211046+0000","last_change":"2024-02-10T08:49:42.108649+0000","last_active":"2024-02-10T08:49:43.211046+0000","last_peered":"2024-02-10T08:49:43.211046+0000","last_clean":"2024-02-10T08:49:43.211046+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:49:43.211046+0000","last_undegraded":"2024-02-10T08:49:43.211046+0000","last_fullsized":"2024-02-10T08:49:43.211046+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.108568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.108568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":841,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209858+0000","last_change":"2024-02-10T08:49:40.106083+0000","last_active":"2024-02-10T08:49:43.209858+0000","last_peered":"2024-02-10T08:49:43.209858+0000","last_clean":"2024-02-10T08:49:43.209858+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:49:43.209858+0000","last_undegraded":"2024-02-10T08:49:43.209858+0000","last_fullsized":"2024-02-10T08:49:43.209858+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:40.106007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.106007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":356,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363027+0000","last_change":"2024-02-10T08:48:40.860303+0000","last_active":"2024-02-10T08:49:43.363027+0000","last_peered":"2024-02-10T08:49:43.363027+0000","last_clean":"2024-02-10T08:49:43.363027+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:49:43.363027+0000","last_undegraded":"2024-02-10T08:49:43.363027+0000","last_fullsized":"2024-02-10T08:49:43.363027+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.860228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.860228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":371,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363711+0000","last_change":"2024-02-10T08:49:30.905873+0000","last_active":"2024-02-10T08:49:43.363711+0000","last_peered":"2024-02-10T08:49:43.363711+0000","last_clean":"2024-02-10T08:49:43.363711+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:49:43.363711+0000","last_undegraded":"2024-02-10T08:49:43.363711+0000","last_fullsized":"2024-02-10T08:49:43.363711+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.905787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.905787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":739,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212133+0000","last_change":"2024-02-10T08:47:49.965414+0000","last_active":"2024-02-10T08:49:44.212133+0000","last_peered":"2024-02-10T08:49:44.212133+0000","last_clean":"2024-02-10T08:49:44.212133+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:49:44.212133+0000","last_undegraded":"2024-02-10T08:49:44.212133+0000","last_fullsized":"2024-02-10T08:49:44.212133+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:47:26.935760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.935760+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":104,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362241+0000","last_change":"2024-02-10T08:47:49.226548+0000","last_active":"2024-02-10T08:49:43.362241+0000","last_peered":"2024-02-10T08:49:43.362241+0000","last_clean":"2024-02-10T08:49:43.362241+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:49:43.362241+0000","last_undegraded":"2024-02-10T08:49:43.362241+0000","last_fullsized":"2024-02-10T08:49:43.362241+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211859+0000","last_change":"2024-02-10T08:48:32.722180+0000","last_active":"2024-02-10T08:49:43.211859+0000","last_peered":"2024-02-10T08:49:43.211859+0000","last_clean":"2024-02-10T08:49:43.211859+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:49:43.211859+0000","last_undegraded":"2024-02-10T08:49:43.211859+0000","last_fullsized":"2024-02-10T08:49:43.211859+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.722121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.722121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":793,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211128+0000","last_change":"2024-02-10T08:48:47.163067+0000","last_active":"2024-02-10T08:49:43.211128+0000","last_peered":"2024-02-10T08:49:43.211128+0000","last_clean":"2024-02-10T08:49:43.211128+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:49:43.211128+0000","last_undegraded":"2024-02-10T08:49:43.211128+0000","last_fullsized":"2024-02-10T08:49:43.211128+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.163011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.163011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":599,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211544+0000","last_change":"2024-02-10T08:49:25.010060+0000","last_active":"2024-02-10T08:49:44.211544+0000","last_peered":"2024-02-10T08:49:44.211544+0000","last_clean":"2024-02-10T08:49:44.211544+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:49:44.211544+0000","last_undegraded":"2024-02-10T08:49:44.211544+0000","last_fullsized":"2024-02-10T08:49:44.211544+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.009961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.009961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":663,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211857+0000","last_change":"2024-02-10T08:48:56.061862+0000","last_active":"2024-02-10T08:49:44.211857+0000","last_peered":"2024-02-10T08:49:44.211857+0000","last_clean":"2024-02-10T08:49:44.211857+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:49:44.211857+0000","last_undegraded":"2024-02-10T08:49:44.211857+0000","last_fullsized":"2024-02-10T08:49:44.211857+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:48:56.061783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.061783+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210292+0000","last_change":"2024-02-10T08:49:22.842445+0000","last_active":"2024-02-10T08:49:43.210292+0000","last_peered":"2024-02-10T08:49:43.210292+0000","last_clean":"2024-02-10T08:49:43.210292+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:49:43.210292+0000","last_undegraded":"2024-02-10T08:49:43.210292+0000","last_fullsized":"2024-02-10T08:49:43.210292+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.842361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.842361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1066,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213455+0000","last_change":"2024-02-10T08:49:39.103578+0000","last_active":"2024-02-10T08:49:44.213455+0000","last_peered":"2024-02-10T08:49:44.213455+0000","last_clean":"2024-02-10T08:49:44.213455+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:49:44.213455+0000","last_undegraded":"2024-02-10T08:49:44.213455+0000","last_fullsized":"2024-02-10T08:49:44.213455+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:49:39.103510+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.103510+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":623,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211878+0000","last_change":"2024-02-10T08:48:48.962890+0000","last_active":"2024-02-10T08:49:44.211878+0000","last_peered":"2024-02-10T08:49:44.211878+0000","last_clean":"2024-02-10T08:49:44.211878+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:49:44.211878+0000","last_undegraded":"2024-02-10T08:49:44.211878+0000","last_fullsized":"2024-02-10T08:49:44.211878+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.962820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.962820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":196,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212554+0000","last_change":"2024-02-10T08:48:29.433315+0000","last_active":"2024-02-10T08:49:43.212554+0000","last_peered":"2024-02-10T08:49:43.212554+0000","last_clean":"2024-02-10T08:49:43.212554+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:49:43.212554+0000","last_undegraded":"2024-02-10T08:49:43.212554+0000","last_fullsized":"2024-02-10T08:49:43.212554+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:29.433223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.433223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":127,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209827+0000","last_change":"2024-02-10T08:48:51.755042+0000","last_active":"2024-02-10T08:49:43.209827+0000","last_peered":"2024-02-10T08:49:43.209827+0000","last_clean":"2024-02-10T08:49:43.209827+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:49:43.209827+0000","last_undegraded":"2024-02-10T08:49:43.209827+0000","last_fullsized":"2024-02-10T08:49:43.209827+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:48:51.754925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.754925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":115,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211446+0000","last_change":"2024-02-10T08:49:24.873487+0000","last_active":"2024-02-10T08:49:43.211446+0000","last_peered":"2024-02-10T08:49:43.211446+0000","last_clean":"2024-02-10T08:49:43.211446+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:49:43.211446+0000","last_undegraded":"2024-02-10T08:49:43.211446+0000","last_fullsized":"2024-02-10T08:49:43.211446+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:49:24.873416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.873416+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":877,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.647345+0000","last_change":"2024-02-10T08:49:37.120919+0000","last_active":"2024-02-10T08:49:51.647345+0000","last_peered":"2024-02-10T08:49:51.647345+0000","last_clean":"2024-02-10T08:49:51.647345+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:49:51.647345+0000","last_undegraded":"2024-02-10T08:49:51.647345+0000","last_fullsized":"2024-02-10T08:49:51.647345+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:49:37.120855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.120855+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":808,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632773+0000","last_change":"2024-02-10T08:49:30.193392+0000","last_active":"2024-02-10T08:49:43.632773+0000","last_peered":"2024-02-10T08:49:43.632773+0000","last_clean":"2024-02-10T08:49:43.632773+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:49:43.632773+0000","last_undegraded":"2024-02-10T08:49:43.632773+0000","last_fullsized":"2024-02-10T08:49:43.632773+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.193317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.193317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":757,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631511+0000","last_change":"2024-02-10T08:49:38.332305+0000","last_active":"2024-02-10T08:49:43.631511+0000","last_peered":"2024-02-10T08:49:43.631511+0000","last_clean":"2024-02-10T08:49:43.631511+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:49:43.631511+0000","last_undegraded":"2024-02-10T08:49:43.631511+0000","last_fullsized":"2024-02-10T08:49:43.631511+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.332205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.332205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":706,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212185+0000","last_change":"2024-02-10T08:49:20.031309+0000","last_active":"2024-02-10T08:49:44.212185+0000","last_peered":"2024-02-10T08:49:44.212185+0000","last_clean":"2024-02-10T08:49:44.212185+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:49:44.212185+0000","last_undegraded":"2024-02-10T08:49:44.212185+0000","last_fullsized":"2024-02-10T08:49:44.212185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:20.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.031226+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":389,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362910+0000","last_change":"2024-02-10T08:49:32.829681+0000","last_active":"2024-02-10T08:49:43.362910+0000","last_peered":"2024-02-10T08:49:43.362910+0000","last_clean":"2024-02-10T08:49:43.362910+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:49:43.362910+0000","last_undegraded":"2024-02-10T08:49:43.362910+0000","last_fullsized":"2024-02-10T08:49:43.362910+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.829565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.829565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":965,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226959+0000","last_change":"2024-02-10T08:48:51.038110+0000","last_active":"2024-02-10T08:49:45.226959+0000","last_peered":"2024-02-10T08:49:45.226959+0000","last_clean":"2024-02-10T08:49:45.226959+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:49:45.226959+0000","last_undegraded":"2024-02-10T08:49:45.226959+0000","last_fullsized":"2024-02-10T08:49:45.226959+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.038025+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":772,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362725+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:49:43.362725+0000","last_peered":"2024-02-10T08:49:43.362725+0000","last_clean":"2024-02-10T08:49:43.362725+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:49:43.362725+0000","last_undegraded":"2024-02-10T08:49:43.362725+0000","last_fullsized":"2024-02-10T08:49:43.362725+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":723,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362114+0000","last_change":"2024-02-10T08:49:21.361148+0000","last_active":"2024-02-10T08:49:43.362114+0000","last_peered":"2024-02-10T08:49:43.362114+0000","last_clean":"2024-02-10T08:49:43.362114+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:49:43.362114+0000","last_undegraded":"2024-02-10T08:49:43.362114+0000","last_fullsized":"2024-02-10T08:49:43.362114+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.361087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.361087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1031,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588812+0000","last_change":"2024-02-10T08:49:34.310584+0000","last_active":"2024-02-10T08:49:45.588812+0000","last_peered":"2024-02-10T08:49:45.588812+0000","last_clean":"2024-02-10T08:49:45.588812+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:49:45.588812+0000","last_undegraded":"2024-02-10T08:49:45.588812+0000","last_fullsized":"2024-02-10T08:49:45.588812+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'14","last_scrub_stamp":"2024-02-10T08:49:34.310467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.310467+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211400+0000","last_change":"2024-02-10T08:49:42.159790+0000","last_active":"2024-02-10T08:49:44.211400+0000","last_peered":"2024-02-10T08:49:44.211400+0000","last_clean":"2024-02-10T08:49:44.211400+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:49:44.211400+0000","last_undegraded":"2024-02-10T08:49:44.211400+0000","last_fullsized":"2024-02-10T08:49:44.211400+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:49:42.159696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.159696+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":813,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632140+0000","last_change":"2024-02-10T08:48:54.105562+0000","last_active":"2024-02-10T08:49:43.632140+0000","last_peered":"2024-02-10T08:49:43.632140+0000","last_clean":"2024-02-10T08:49:43.632140+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:49:43.632140+0000","last_undegraded":"2024-02-10T08:49:43.632140+0000","last_fullsized":"2024-02-10T08:49:43.632140+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.105490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.105490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":363,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363337+0000","last_change":"2024-02-10T08:49:26.795580+0000","last_active":"2024-02-10T08:49:43.363337+0000","last_peered":"2024-02-10T08:49:43.363337+0000","last_clean":"2024-02-10T08:49:43.363337+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:49:43.363337+0000","last_undegraded":"2024-02-10T08:49:43.363337+0000","last_fullsized":"2024-02-10T08:49:43.363337+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.795492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.795492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":211,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211904+0000","last_change":"2024-02-10T08:48:05.524065+0000","last_active":"2024-02-10T08:49:43.211904+0000","last_peered":"2024-02-10T08:49:43.211904+0000","last_clean":"2024-02-10T08:49:43.211904+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:49:43.211904+0000","last_undegraded":"2024-02-10T08:49:43.211904+0000","last_fullsized":"2024-02-10T08:49:43.211904+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:05.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211244+0000","last_change":"2024-02-10T08:48:45.969499+0000","last_active":"2024-02-10T08:49:44.211244+0000","last_peered":"2024-02-10T08:49:44.211244+0000","last_clean":"2024-02-10T08:49:44.211244+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:49:44.211244+0000","last_undegraded":"2024-02-10T08:49:44.211244+0000","last_fullsized":"2024-02-10T08:49:44.211244+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.969416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.969416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":965,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361784+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:49:43.361784+0000","last_peered":"2024-02-10T08:49:43.361784+0000","last_clean":"2024-02-10T08:49:43.361784+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:49:43.361784+0000","last_undegraded":"2024-02-10T08:49:43.361784+0000","last_fullsized":"2024-02-10T08:49:43.361784+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":115,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211088+0000","last_change":"2024-02-10T08:48:36.653104+0000","last_active":"2024-02-10T08:49:43.211088+0000","last_peered":"2024-02-10T08:49:43.211088+0000","last_clean":"2024-02-10T08:49:43.211088+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:49:43.211088+0000","last_undegraded":"2024-02-10T08:49:43.211088+0000","last_fullsized":"2024-02-10T08:49:43.211088+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:48:36.652981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.652981+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":610,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211275+0000","last_change":"2024-02-10T08:48:42.984939+0000","last_active":"2024-02-10T08:49:44.211275+0000","last_peered":"2024-02-10T08:49:44.211275+0000","last_clean":"2024-02-10T08:49:44.211275+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:49:44.211275+0000","last_undegraded":"2024-02-10T08:49:44.211275+0000","last_fullsized":"2024-02-10T08:49:44.211275+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.984822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.984822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":211,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211010+0000","last_change":"2024-02-10T08:48:37.427622+0000","last_active":"2024-02-10T08:49:43.211010+0000","last_peered":"2024-02-10T08:49:43.211010+0000","last_clean":"2024-02-10T08:49:43.211010+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:49:43.211010+0000","last_undegraded":"2024-02-10T08:49:43.211010+0000","last_fullsized":"2024-02-10T08:49:43.211010+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:37.427518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:37.427518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631298+0000","last_change":"2024-02-10T08:48:43.081221+0000","last_active":"2024-02-10T08:49:43.631298+0000","last_peered":"2024-02-10T08:49:43.631298+0000","last_clean":"2024-02-10T08:49:43.631298+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:49:43.631298+0000","last_undegraded":"2024-02-10T08:49:43.631298+0000","last_fullsized":"2024-02-10T08:49:43.631298+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.081148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.081148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":206,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212199+0000","last_change":"2024-02-10T08:47:57.560707+0000","last_active":"2024-02-10T08:49:43.212199+0000","last_peered":"2024-02-10T08:49:43.212199+0000","last_clean":"2024-02-10T08:49:43.212199+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:49:43.212199+0000","last_undegraded":"2024-02-10T08:49:43.212199+0000","last_fullsized":"2024-02-10T08:49:43.212199+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:47:57.560587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.560587+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":386,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363474+0000","last_change":"2024-02-10T08:49:23.762672+0000","last_active":"2024-02-10T08:49:43.363474+0000","last_peered":"2024-02-10T08:49:43.363474+0000","last_clean":"2024-02-10T08:49:43.363474+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:49:43.363474+0000","last_undegraded":"2024-02-10T08:49:43.363474+0000","last_fullsized":"2024-02-10T08:49:43.363474+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.762550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.762550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":686,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209877+0000","last_change":"2024-02-10T08:49:26.109262+0000","last_active":"2024-02-10T08:49:43.209877+0000","last_peered":"2024-02-10T08:49:43.209877+0000","last_clean":"2024-02-10T08:49:43.209877+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:49:43.209877+0000","last_undegraded":"2024-02-10T08:49:43.209877+0000","last_fullsized":"2024-02-10T08:49:43.209877+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.109194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.109194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":768,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210880+0000","last_change":"2024-02-10T08:48:39.036788+0000","last_active":"2024-02-10T08:49:43.210880+0000","last_peered":"2024-02-10T08:49:43.210880+0000","last_clean":"2024-02-10T08:49:43.210880+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:49:43.210880+0000","last_undegraded":"2024-02-10T08:49:43.210880+0000","last_fullsized":"2024-02-10T08:49:43.210880+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:39.036699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:39.036699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":834,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632997+0000","last_change":"2024-02-10T08:49:26.144964+0000","last_active":"2024-02-10T08:49:43.632997+0000","last_peered":"2024-02-10T08:49:43.632997+0000","last_clean":"2024-02-10T08:49:43.632997+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:49:43.632997+0000","last_undegraded":"2024-02-10T08:49:43.632997+0000","last_fullsized":"2024-02-10T08:49:43.632997+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.144887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.144887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":253,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212425+0000","last_change":"2024-02-10T08:49:29.655187+0000","last_active":"2024-02-10T08:49:43.212425+0000","last_peered":"2024-02-10T08:49:43.212425+0000","last_clean":"2024-02-10T08:49:43.212425+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:49:43.212425+0000","last_undegraded":"2024-02-10T08:49:43.212425+0000","last_fullsized":"2024-02-10T08:49:43.212425+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:49:29.655086+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.655086+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":388,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362897+0000","last_change":"2024-02-10T08:49:22.774270+0000","last_active":"2024-02-10T08:49:43.362897+0000","last_peered":"2024-02-10T08:49:43.362897+0000","last_clean":"2024-02-10T08:49:43.362897+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:49:43.362897+0000","last_undegraded":"2024-02-10T08:49:43.362897+0000","last_fullsized":"2024-02-10T08:49:43.362897+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.774198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.774198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":204,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211723+0000","last_change":"2024-02-10T08:48:36.452244+0000","last_active":"2024-02-10T08:49:43.211723+0000","last_peered":"2024-02-10T08:49:43.211723+0000","last_clean":"2024-02-10T08:49:43.211723+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:49:43.211723+0000","last_undegraded":"2024-02-10T08:49:43.211723+0000","last_fullsized":"2024-02-10T08:49:43.211723+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.452178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.452178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":116,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211951+0000","last_change":"2024-02-10T08:48:05.945279+0000","last_active":"2024-02-10T08:49:43.211951+0000","last_peered":"2024-02-10T08:49:43.211951+0000","last_clean":"2024-02-10T08:49:43.211951+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:49:43.211951+0000","last_undegraded":"2024-02-10T08:49:43.211951+0000","last_fullsized":"2024-02-10T08:49:43.211951+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.945162+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":747,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631407+0000","last_change":"2024-02-10T08:49:23.855127+0000","last_active":"2024-02-10T08:49:43.631407+0000","last_peered":"2024-02-10T08:49:43.631407+0000","last_clean":"2024-02-10T08:49:43.631407+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:49:43.631407+0000","last_undegraded":"2024-02-10T08:49:43.631407+0000","last_fullsized":"2024-02-10T08:49:43.631407+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.855041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.855041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":363,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363481+0000","last_change":"2024-02-10T08:48:54.704302+0000","last_active":"2024-02-10T08:49:43.363481+0000","last_peered":"2024-02-10T08:49:43.363481+0000","last_clean":"2024-02-10T08:49:43.363481+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:49:43.363481+0000","last_undegraded":"2024-02-10T08:49:43.363481+0000","last_fullsized":"2024-02-10T08:49:43.363481+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.704183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.704183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":879,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631682+0000","last_change":"2024-02-10T08:48:31.839331+0000","last_active":"2024-02-10T08:49:43.631682+0000","last_peered":"2024-02-10T08:49:43.631682+0000","last_clean":"2024-02-10T08:49:43.631682+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:49:43.631682+0000","last_undegraded":"2024-02-10T08:49:43.631682+0000","last_fullsized":"2024-02-10T08:49:43.631682+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:48:31.839259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.839259+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":367,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363081+0000","last_change":"2024-02-10T08:48:30.896370+0000","last_active":"2024-02-10T08:49:43.363081+0000","last_peered":"2024-02-10T08:49:43.363081+0000","last_clean":"2024-02-10T08:49:43.363081+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:49:43.363081+0000","last_undegraded":"2024-02-10T08:49:43.363081+0000","last_fullsized":"2024-02-10T08:49:43.363081+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.896306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.896306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":796,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631690+0000","last_change":"2024-02-10T08:49:23.180885+0000","last_active":"2024-02-10T08:49:43.631690+0000","last_peered":"2024-02-10T08:49:43.631690+0000","last_clean":"2024-02-10T08:49:43.631690+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:49:43.631690+0000","last_undegraded":"2024-02-10T08:49:43.631690+0000","last_fullsized":"2024-02-10T08:49:43.631690+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.180813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.180813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":774,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362309+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:49:43.362309+0000","last_peered":"2024-02-10T08:49:43.362309+0000","last_clean":"2024-02-10T08:49:43.362309+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:49:43.362309+0000","last_undegraded":"2024-02-10T08:49:43.362309+0000","last_fullsized":"2024-02-10T08:49:43.362309+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":782,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.633043+0000","last_change":"2024-02-10T08:48:48.079333+0000","last_active":"2024-02-10T08:49:43.633043+0000","last_peered":"2024-02-10T08:49:43.633043+0000","last_clean":"2024-02-10T08:49:43.633043+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:49:43.633043+0000","last_undegraded":"2024-02-10T08:49:43.633043+0000","last_fullsized":"2024-02-10T08:49:43.633043+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.079269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.079269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":756,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631327+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:49:43.631327+0000","last_peered":"2024-02-10T08:49:43.631327+0000","last_clean":"2024-02-10T08:49:43.631327+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:49:43.631327+0000","last_undegraded":"2024-02-10T08:49:43.631327+0000","last_fullsized":"2024-02-10T08:49:43.631327+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":821,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210487+0000","last_change":"2024-02-10T08:49:31.019799+0000","last_active":"2024-02-10T08:49:43.210487+0000","last_peered":"2024-02-10T08:49:43.210487+0000","last_clean":"2024-02-10T08:49:43.210487+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:49:43.210487+0000","last_undegraded":"2024-02-10T08:49:43.210487+0000","last_fullsized":"2024-02-10T08:49:43.210487+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.019739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.019739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":118,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209858+0000","last_change":"2024-02-10T08:47:53.993563+0000","last_active":"2024-02-10T08:49:43.209858+0000","last_peered":"2024-02-10T08:49:43.209858+0000","last_clean":"2024-02-10T08:49:43.209858+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:49:43.209858+0000","last_undegraded":"2024-02-10T08:49:43.209858+0000","last_fullsized":"2024-02-10T08:49:43.209858+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:47:30.847592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:30.847592+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":809,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209795+0000","last_change":"2024-02-10T08:49:36.077156+0000","last_active":"2024-02-10T08:49:43.209795+0000","last_peered":"2024-02-10T08:49:43.209795+0000","last_clean":"2024-02-10T08:49:43.209795+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:49:43.209795+0000","last_undegraded":"2024-02-10T08:49:43.209795+0000","last_fullsized":"2024-02-10T08:49:43.209795+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:36.077066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:36.077066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":237,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:49.441923+0000","last_change":"2024-02-10T08:49:40.123788+0000","last_active":"2024-02-10T08:49:49.441923+0000","last_peered":"2024-02-10T08:49:49.441923+0000","last_clean":"2024-02-10T08:49:49.441923+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:49:49.441923+0000","last_undegraded":"2024-02-10T08:49:49.441923+0000","last_fullsized":"2024-02-10T08:49:49.441923+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:49:40.123719+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.123719+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":214,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":355,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363464+0000","last_change":"2024-02-10T08:48:47.787415+0000","last_active":"2024-02-10T08:49:43.363464+0000","last_peered":"2024-02-10T08:49:43.363464+0000","last_clean":"2024-02-10T08:49:43.363464+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:49:43.363464+0000","last_undegraded":"2024-02-10T08:49:43.363464+0000","last_fullsized":"2024-02-10T08:49:43.363464+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.787351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210291+0000","last_change":"2024-02-10T08:49:39.088323+0000","last_active":"2024-02-10T08:49:43.210291+0000","last_peered":"2024-02-10T08:49:43.210291+0000","last_clean":"2024-02-10T08:49:43.210291+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:49:43.210291+0000","last_undegraded":"2024-02-10T08:49:43.210291+0000","last_fullsized":"2024-02-10T08:49:43.210291+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.088230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.088230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":222,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212018+0000","last_change":"2024-02-10T08:47:59.541068+0000","last_active":"2024-02-10T08:49:43.212018+0000","last_peered":"2024-02-10T08:49:43.212018+0000","last_clean":"2024-02-10T08:49:43.212018+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:49:43.212018+0000","last_undegraded":"2024-02-10T08:49:43.212018+0000","last_fullsized":"2024-02-10T08:49:43.212018+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:59.541002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:47:59.541002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"711'43","reported_seq":157,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213810+0000","last_change":"2024-02-10T08:48:28.753355+0000","last_active":"2024-02-10T08:49:44.213810+0000","last_peered":"2024-02-10T08:49:44.213810+0000","last_clean":"2024-02-10T08:49:44.213810+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:49:44.213810+0000","last_undegraded":"2024-02-10T08:49:44.213810+0000","last_fullsized":"2024-02-10T08:49:44.213810+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'43","last_scrub_stamp":"2024-02-10T08:48:28.753288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.753288+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":796,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631669+0000","last_change":"2024-02-10T08:48:31.134699+0000","last_active":"2024-02-10T08:49:43.631669+0000","last_peered":"2024-02-10T08:49:43.631669+0000","last_clean":"2024-02-10T08:49:43.631669+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:49:43.631669+0000","last_undegraded":"2024-02-10T08:49:43.631669+0000","last_fullsized":"2024-02-10T08:49:43.631669+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.134575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.134575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":801,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363191+0000","last_change":"2024-02-10T08:49:24.384243+0000","last_active":"2024-02-10T08:49:43.363191+0000","last_peered":"2024-02-10T08:49:43.363191+0000","last_clean":"2024-02-10T08:49:43.363191+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:49:43.363191+0000","last_undegraded":"2024-02-10T08:49:43.363191+0000","last_fullsized":"2024-02-10T08:49:43.363191+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.384180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.384180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":768,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209297+0000","last_change":"2024-02-10T08:48:54.135964+0000","last_active":"2024-02-10T08:49:43.209297+0000","last_peered":"2024-02-10T08:49:43.209297+0000","last_clean":"2024-02-10T08:49:43.209297+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:49:43.209297+0000","last_undegraded":"2024-02-10T08:49:43.209297+0000","last_fullsized":"2024-02-10T08:49:43.209297+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.135900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.135900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":240,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589298+0000","last_change":"2024-02-10T08:49:21.583412+0000","last_active":"2024-02-10T08:49:45.589298+0000","last_peered":"2024-02-10T08:49:45.589298+0000","last_clean":"2024-02-10T08:49:45.589298+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:49:45.589298+0000","last_undegraded":"2024-02-10T08:49:45.589298+0000","last_fullsized":"2024-02-10T08:49:45.589298+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:49:21.583343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.583343+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":123,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210218+0000","last_change":"2024-02-10T08:48:42.684268+0000","last_active":"2024-02-10T08:49:43.210218+0000","last_peered":"2024-02-10T08:49:43.210218+0000","last_clean":"2024-02-10T08:49:43.210218+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:49:43.210218+0000","last_undegraded":"2024-02-10T08:49:43.210218+0000","last_fullsized":"2024-02-10T08:49:43.210218+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:48:42.684181+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.684181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211991+0000","last_change":"2024-02-10T08:49:35.015139+0000","last_active":"2024-02-10T08:49:43.211991+0000","last_peered":"2024-02-10T08:49:43.211991+0000","last_clean":"2024-02-10T08:49:43.211991+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:49:43.211991+0000","last_undegraded":"2024-02-10T08:49:43.211991+0000","last_fullsized":"2024-02-10T08:49:43.211991+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.015073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":193,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212123+0000","last_change":"2024-02-10T08:48:33.412887+0000","last_active":"2024-02-10T08:49:43.212123+0000","last_peered":"2024-02-10T08:49:43.212123+0000","last_clean":"2024-02-10T08:49:43.212123+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:49:43.212123+0000","last_undegraded":"2024-02-10T08:49:43.212123+0000","last_fullsized":"2024-02-10T08:49:43.212123+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.412775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.412775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362939+0000","last_change":"2024-02-10T08:48:50.100054+0000","last_active":"2024-02-10T08:49:43.362939+0000","last_peered":"2024-02-10T08:49:43.362939+0000","last_clean":"2024-02-10T08:49:43.362939+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:49:43.362939+0000","last_undegraded":"2024-02-10T08:49:43.362939+0000","last_fullsized":"2024-02-10T08:49:43.362939+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.099951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.099951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":839,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209513+0000","last_change":"2024-02-10T08:48:41.070217+0000","last_active":"2024-02-10T08:49:43.209513+0000","last_peered":"2024-02-10T08:49:43.209513+0000","last_clean":"2024-02-10T08:49:43.209513+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:49:43.209513+0000","last_undegraded":"2024-02-10T08:49:43.209513+0000","last_fullsized":"2024-02-10T08:49:43.209513+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.070126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.070126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":712,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211615+0000","last_change":"2024-02-10T08:48:31.973099+0000","last_active":"2024-02-10T08:49:44.211615+0000","last_peered":"2024-02-10T08:49:44.211615+0000","last_clean":"2024-02-10T08:49:44.211615+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:49:44.211615+0000","last_undegraded":"2024-02-10T08:49:44.211615+0000","last_fullsized":"2024-02-10T08:49:44.211615+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:48:31.973020+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.973020+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212121+0000","last_change":"2024-02-10T08:49:30.980749+0000","last_active":"2024-02-10T08:49:44.212121+0000","last_peered":"2024-02-10T08:49:44.212121+0000","last_clean":"2024-02-10T08:49:44.212121+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:49:44.212121+0000","last_undegraded":"2024-02-10T08:49:44.212121+0000","last_fullsized":"2024-02-10T08:49:44.212121+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:49:30.980666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.980666+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":809,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362522+0000","last_change":"2024-02-10T08:48:38.003058+0000","last_active":"2024-02-10T08:49:43.362522+0000","last_peered":"2024-02-10T08:49:43.362522+0000","last_clean":"2024-02-10T08:49:43.362522+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:49:43.362522+0000","last_undegraded":"2024-02-10T08:49:43.362522+0000","last_fullsized":"2024-02-10T08:49:43.362522+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.002996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.002996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":125,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632527+0000","last_change":"2024-02-10T08:48:33.158780+0000","last_active":"2024-02-10T08:49:43.632527+0000","last_peered":"2024-02-10T08:49:43.632527+0000","last_clean":"2024-02-10T08:49:43.632527+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:49:43.632527+0000","last_undegraded":"2024-02-10T08:49:43.632527+0000","last_fullsized":"2024-02-10T08:49:43.632527+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.158671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.158671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":202,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212559+0000","last_change":"2024-02-10T08:48:54.417289+0000","last_active":"2024-02-10T08:49:43.212559+0000","last_peered":"2024-02-10T08:49:43.212559+0000","last_clean":"2024-02-10T08:49:43.212559+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:49:43.212559+0000","last_undegraded":"2024-02-10T08:49:43.212559+0000","last_fullsized":"2024-02-10T08:49:43.212559+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:48:54.417173+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.417173+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":826,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209440+0000","last_change":"2024-02-10T08:48:50.118042+0000","last_active":"2024-02-10T08:49:43.209440+0000","last_peered":"2024-02-10T08:49:43.209440+0000","last_clean":"2024-02-10T08:49:43.209440+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:49:43.209440+0000","last_undegraded":"2024-02-10T08:49:43.209440+0000","last_fullsized":"2024-02-10T08:49:43.209440+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.117976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":811,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210899+0000","last_change":"2024-02-10T08:49:24.116688+0000","last_active":"2024-02-10T08:49:43.210899+0000","last_peered":"2024-02-10T08:49:43.210899+0000","last_clean":"2024-02-10T08:49:43.210899+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:49:43.210899+0000","last_undegraded":"2024-02-10T08:49:43.210899+0000","last_fullsized":"2024-02-10T08:49:43.210899+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.116601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.116601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":605,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211589+0000","last_change":"2024-02-10T08:48:41.943374+0000","last_active":"2024-02-10T08:49:44.211589+0000","last_peered":"2024-02-10T08:49:44.211589+0000","last_clean":"2024-02-10T08:49:44.211589+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:49:44.211589+0000","last_undegraded":"2024-02-10T08:49:44.211589+0000","last_fullsized":"2024-02-10T08:49:44.211589+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.943287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.943287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":863,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363494+0000","last_change":"2024-02-10T08:49:43.363252+0000","last_active":"2024-02-10T08:49:43.363494+0000","last_peered":"2024-02-10T08:49:43.363494+0000","last_clean":"2024-02-10T08:49:43.363494+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:49:43.363494+0000","last_undegraded":"2024-02-10T08:49:43.363494+0000","last_fullsized":"2024-02-10T08:49:43.363494+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:49:43.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:43.363195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":809,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631854+0000","last_change":"2024-02-10T08:48:36.966625+0000","last_active":"2024-02-10T08:49:43.631854+0000","last_peered":"2024-02-10T08:49:43.631854+0000","last_clean":"2024-02-10T08:49:43.631854+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:49:43.631854+0000","last_undegraded":"2024-02-10T08:49:43.631854+0000","last_fullsized":"2024-02-10T08:49:43.631854+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.966559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.966559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":371,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363377+0000","last_change":"2024-02-10T08:49:38.810361+0000","last_active":"2024-02-10T08:49:43.363377+0000","last_peered":"2024-02-10T08:49:43.363377+0000","last_clean":"2024-02-10T08:49:43.363377+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:49:43.363377+0000","last_undegraded":"2024-02-10T08:49:43.363377+0000","last_fullsized":"2024-02-10T08:49:43.363377+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.810280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.810280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":733,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631605+0000","last_change":"2024-02-10T08:48:47.081659+0000","last_active":"2024-02-10T08:49:43.631605+0000","last_peered":"2024-02-10T08:49:43.631605+0000","last_clean":"2024-02-10T08:49:43.631605+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:49:43.631605+0000","last_undegraded":"2024-02-10T08:49:43.631605+0000","last_fullsized":"2024-02-10T08:49:43.631605+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.081542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":891,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362736+0000","last_change":"2024-02-10T08:49:20.324246+0000","last_active":"2024-02-10T08:49:43.362736+0000","last_peered":"2024-02-10T08:49:43.362736+0000","last_clean":"2024-02-10T08:49:43.362736+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:49:43.362736+0000","last_undegraded":"2024-02-10T08:49:43.362736+0000","last_fullsized":"2024-02-10T08:49:43.362736+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:20.324128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.324128+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":652,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362192+0000","last_change":"2024-02-10T08:48:45.089054+0000","last_active":"2024-02-10T08:49:43.362192+0000","last_peered":"2024-02-10T08:49:43.362192+0000","last_clean":"2024-02-10T08:49:43.362192+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:49:43.362192+0000","last_undegraded":"2024-02-10T08:49:43.362192+0000","last_fullsized":"2024-02-10T08:49:43.362192+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.088965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.088965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1033,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226878+0000","last_change":"2024-02-10T08:49:35.294710+0000","last_active":"2024-02-10T08:49:45.226878+0000","last_peered":"2024-02-10T08:49:45.226878+0000","last_clean":"2024-02-10T08:49:45.226878+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:49:45.226878+0000","last_undegraded":"2024-02-10T08:49:45.226878+0000","last_fullsized":"2024-02-10T08:49:45.226878+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13","last_scrub_stamp":"2024-02-10T08:49:35.294638+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.294638+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":193,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212494+0000","last_change":"2024-02-10T08:48:38.471156+0000","last_active":"2024-02-10T08:49:43.212494+0000","last_peered":"2024-02-10T08:49:43.212494+0000","last_clean":"2024-02-10T08:49:43.212494+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:49:43.212494+0000","last_undegraded":"2024-02-10T08:49:43.212494+0000","last_fullsized":"2024-02-10T08:49:43.212494+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.471058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.471058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":193,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212628+0000","last_change":"2024-02-10T08:48:30.433484+0000","last_active":"2024-02-10T08:49:43.212628+0000","last_peered":"2024-02-10T08:49:43.212628+0000","last_clean":"2024-02-10T08:49:43.212628+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:49:43.212628+0000","last_undegraded":"2024-02-10T08:49:43.212628+0000","last_fullsized":"2024-02-10T08:49:43.212628+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.433421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.433421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"798'19","reported_seq":843,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632597+0000","last_change":"2024-02-10T08:49:37.293132+0000","last_active":"2024-02-10T08:49:43.632597+0000","last_peered":"2024-02-10T08:49:43.632597+0000","last_clean":"2024-02-10T08:49:43.632597+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:49:43.632597+0000","last_undegraded":"2024-02-10T08:49:43.632597+0000","last_fullsized":"2024-02-10T08:49:43.632597+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'19","last_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_deep_scrub":"798'19","last_deep_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.293068+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":550,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211472+0000","last_change":"2024-02-10T08:47:55.031372+0000","last_active":"2024-02-10T08:49:44.211472+0000","last_peered":"2024-02-10T08:49:44.211472+0000","last_clean":"2024-02-10T08:49:44.211472+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:49:44.211472+0000","last_undegraded":"2024-02-10T08:49:44.211472+0000","last_fullsized":"2024-02-10T08:49:44.211472+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.031301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.031301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":1008,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589514+0000","last_change":"2024-02-10T08:49:22.354177+0000","last_active":"2024-02-10T08:49:45.589514+0000","last_peered":"2024-02-10T08:49:45.589514+0000","last_clean":"2024-02-10T08:49:45.589514+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:49:45.589514+0000","last_undegraded":"2024-02-10T08:49:45.589514+0000","last_fullsized":"2024-02-10T08:49:45.589514+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:49:22.354105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.354105+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":612,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212098+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:49:44.212098+0000","last_peered":"2024-02-10T08:49:44.212098+0000","last_clean":"2024-02-10T08:49:44.212098+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:49:44.212098+0000","last_undegraded":"2024-02-10T08:49:44.212098+0000","last_fullsized":"2024-02-10T08:49:44.212098+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":771,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.209586+0000","last_change":"2024-02-10T08:49:29.101599+0000","last_active":"2024-02-10T08:49:43.209586+0000","last_peered":"2024-02-10T08:49:43.209586+0000","last_clean":"2024-02-10T08:49:43.209586+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:49:43.209586+0000","last_undegraded":"2024-02-10T08:49:43.209586+0000","last_fullsized":"2024-02-10T08:49:43.209586+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:49:29.101533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.101533+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":121,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210366+0000","last_change":"2024-02-10T08:49:35.988869+0000","last_active":"2024-02-10T08:49:43.210366+0000","last_peered":"2024-02-10T08:49:43.210366+0000","last_clean":"2024-02-10T08:49:43.210366+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:49:43.210366+0000","last_undegraded":"2024-02-10T08:49:43.210366+0000","last_fullsized":"2024-02-10T08:49:43.210366+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:35.988803+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.988803+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":786,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631655+0000","last_change":"2024-02-10T08:48:45.997431+0000","last_active":"2024-02-10T08:49:43.631655+0000","last_peered":"2024-02-10T08:49:43.631655+0000","last_clean":"2024-02-10T08:49:43.631655+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:49:43.631655+0000","last_undegraded":"2024-02-10T08:49:43.631655+0000","last_fullsized":"2024-02-10T08:49:43.631655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.997364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.997364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":811,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362481+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:49:43.362481+0000","last_peered":"2024-02-10T08:49:43.362481+0000","last_clean":"2024-02-10T08:49:43.362481+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:49:43.362481+0000","last_undegraded":"2024-02-10T08:49:43.362481+0000","last_fullsized":"2024-02-10T08:49:43.362481+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":211,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212320+0000","last_change":"2024-02-10T08:49:20.566561+0000","last_active":"2024-02-10T08:49:43.212320+0000","last_peered":"2024-02-10T08:49:43.212320+0000","last_clean":"2024-02-10T08:49:43.212320+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:49:43.212320+0000","last_undegraded":"2024-02-10T08:49:43.212320+0000","last_fullsized":"2024-02-10T08:49:43.212320+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.566497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.566497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":749,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631864+0000","last_change":"2024-02-10T08:49:26.780938+0000","last_active":"2024-02-10T08:49:43.631864+0000","last_peered":"2024-02-10T08:49:43.631864+0000","last_clean":"2024-02-10T08:49:43.631864+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:49:43.631864+0000","last_undegraded":"2024-02-10T08:49:43.631864+0000","last_fullsized":"2024-02-10T08:49:43.631864+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.780857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.780857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":703,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211532+0000","last_change":"2024-02-10T08:47:49.793777+0000","last_active":"2024-02-10T08:49:44.211532+0000","last_peered":"2024-02-10T08:49:44.211532+0000","last_clean":"2024-02-10T08:49:44.211532+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:49:44.211532+0000","last_undegraded":"2024-02-10T08:49:44.211532+0000","last_fullsized":"2024-02-10T08:49:44.211532+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:19.946402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:19.946402+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210834+0000","last_change":"2024-02-10T08:47:56.921391+0000","last_active":"2024-02-10T08:49:43.210834+0000","last_peered":"2024-02-10T08:49:43.210834+0000","last_clean":"2024-02-10T08:49:43.210834+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:49:43.210834+0000","last_undegraded":"2024-02-10T08:49:43.210834+0000","last_fullsized":"2024-02-10T08:49:43.210834+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.921291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.921291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210437+0000","last_change":"2024-02-10T08:47:58.934691+0000","last_active":"2024-02-10T08:49:43.210437+0000","last_peered":"2024-02-10T08:49:43.210437+0000","last_clean":"2024-02-10T08:49:43.210437+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:49:43.210437+0000","last_undegraded":"2024-02-10T08:49:43.210437+0000","last_fullsized":"2024-02-10T08:49:43.210437+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:58.934594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.934594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":791,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361961+0000","last_change":"2024-02-10T08:48:40.049047+0000","last_active":"2024-02-10T08:49:43.361961+0000","last_peered":"2024-02-10T08:49:43.361961+0000","last_clean":"2024-02-10T08:49:43.361961+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:49:43.361961+0000","last_undegraded":"2024-02-10T08:49:43.361961+0000","last_fullsized":"2024-02-10T08:49:43.361961+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.048948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.048948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":747,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363421+0000","last_change":"2024-02-10T08:49:31.379159+0000","last_active":"2024-02-10T08:49:43.363421+0000","last_peered":"2024-02-10T08:49:43.363421+0000","last_clean":"2024-02-10T08:49:43.363421+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:49:43.363421+0000","last_undegraded":"2024-02-10T08:49:43.363421+0000","last_fullsized":"2024-02-10T08:49:43.363421+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.379093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.379093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":819,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361993+0000","last_change":"2024-02-10T08:49:33.304282+0000","last_active":"2024-02-10T08:49:43.361993+0000","last_peered":"2024-02-10T08:49:43.361993+0000","last_clean":"2024-02-10T08:49:43.361993+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:49:43.361993+0000","last_undegraded":"2024-02-10T08:49:43.361993+0000","last_fullsized":"2024-02-10T08:49:43.361993+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:49:33.304177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:33.304177+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":782,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632800+0000","last_change":"2024-02-10T08:48:36.153670+0000","last_active":"2024-02-10T08:49:43.632800+0000","last_peered":"2024-02-10T08:49:43.632800+0000","last_clean":"2024-02-10T08:49:43.632800+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:49:43.632800+0000","last_undegraded":"2024-02-10T08:49:43.632800+0000","last_fullsized":"2024-02-10T08:49:43.632800+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.153585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.153585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":807,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631766+0000","last_change":"2024-02-10T08:48:32.816285+0000","last_active":"2024-02-10T08:49:43.631766+0000","last_peered":"2024-02-10T08:49:43.631766+0000","last_clean":"2024-02-10T08:49:43.631766+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:49:43.631766+0000","last_undegraded":"2024-02-10T08:49:43.631766+0000","last_fullsized":"2024-02-10T08:49:43.631766+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.816211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.816211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":371,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363134+0000","last_change":"2024-02-10T08:49:29.891433+0000","last_active":"2024-02-10T08:49:43.363134+0000","last_peered":"2024-02-10T08:49:43.363134+0000","last_clean":"2024-02-10T08:49:43.363134+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:49:43.363134+0000","last_undegraded":"2024-02-10T08:49:43.363134+0000","last_fullsized":"2024-02-10T08:49:43.363134+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.891353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.891353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":759,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362632+0000","last_change":"2024-02-10T08:49:42.399412+0000","last_active":"2024-02-10T08:49:43.362632+0000","last_peered":"2024-02-10T08:49:43.362632+0000","last_clean":"2024-02-10T08:49:43.362632+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:49:43.362632+0000","last_undegraded":"2024-02-10T08:49:43.362632+0000","last_fullsized":"2024-02-10T08:49:43.362632+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.399332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.399332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":845,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362361+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:49:43.362361+0000","last_peered":"2024-02-10T08:49:43.362361+0000","last_clean":"2024-02-10T08:49:43.362361+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:49:43.362361+0000","last_undegraded":"2024-02-10T08:49:43.362361+0000","last_fullsized":"2024-02-10T08:49:43.362361+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":786,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632064+0000","last_change":"2024-02-10T08:48:53.059434+0000","last_active":"2024-02-10T08:49:43.632064+0000","last_peered":"2024-02-10T08:49:43.632064+0000","last_clean":"2024-02-10T08:49:43.632064+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:49:43.632064+0000","last_undegraded":"2024-02-10T08:49:43.632064+0000","last_fullsized":"2024-02-10T08:49:43.632064+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:53.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:53.059137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":366,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362994+0000","last_change":"2024-02-10T08:48:45.871861+0000","last_active":"2024-02-10T08:49:43.362994+0000","last_peered":"2024-02-10T08:49:43.362994+0000","last_clean":"2024-02-10T08:49:43.362994+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:49:43.362994+0000","last_undegraded":"2024-02-10T08:49:43.362994+0000","last_fullsized":"2024-02-10T08:49:43.362994+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.871773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":791,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361780+0000","last_change":"2024-02-10T08:49:23.340760+0000","last_active":"2024-02-10T08:49:43.361780+0000","last_peered":"2024-02-10T08:49:43.361780+0000","last_clean":"2024-02-10T08:49:43.361780+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:49:43.361780+0000","last_undegraded":"2024-02-10T08:49:43.361780+0000","last_fullsized":"2024-02-10T08:49:43.361780+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.340696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.340696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":193,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211945+0000","last_change":"2024-02-10T08:49:42.706500+0000","last_active":"2024-02-10T08:49:43.211945+0000","last_peered":"2024-02-10T08:49:43.211945+0000","last_clean":"2024-02-10T08:49:43.211945+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:49:43.211945+0000","last_undegraded":"2024-02-10T08:49:43.211945+0000","last_fullsized":"2024-02-10T08:49:43.211945+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.706431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.706431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":200,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212355+0000","last_change":"2024-02-10T08:47:58.529546+0000","last_active":"2024-02-10T08:49:43.212355+0000","last_peered":"2024-02-10T08:49:43.212355+0000","last_clean":"2024-02-10T08:49:43.212355+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:49:43.212355+0000","last_undegraded":"2024-02-10T08:49:43.212355+0000","last_fullsized":"2024-02-10T08:49:43.212355+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:47:58.529476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.529476+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":820,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631251+0000","last_change":"2024-02-10T08:48:54.005915+0000","last_active":"2024-02-10T08:49:43.631251+0000","last_peered":"2024-02-10T08:49:43.631251+0000","last_clean":"2024-02-10T08:49:43.631251+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:49:43.631251+0000","last_undegraded":"2024-02-10T08:49:43.631251+0000","last_fullsized":"2024-02-10T08:49:43.631251+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.005830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.005830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":366,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363402+0000","last_change":"2024-02-10T08:48:50.762747+0000","last_active":"2024-02-10T08:49:43.363402+0000","last_peered":"2024-02-10T08:49:43.363402+0000","last_clean":"2024-02-10T08:49:43.363402+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:49:43.363402+0000","last_undegraded":"2024-02-10T08:49:43.363402+0000","last_fullsized":"2024-02-10T08:49:43.363402+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.762672+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":768,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210309+0000","last_change":"2024-02-10T08:49:28.108791+0000","last_active":"2024-02-10T08:49:43.210309+0000","last_peered":"2024-02-10T08:49:43.210309+0000","last_clean":"2024-02-10T08:49:43.210309+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:49:43.210309+0000","last_undegraded":"2024-02-10T08:49:43.210309+0000","last_fullsized":"2024-02-10T08:49:43.210309+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.108727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.108727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":862,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632220+0000","last_change":"2024-02-10T08:49:29.815400+0000","last_active":"2024-02-10T08:49:43.632220+0000","last_peered":"2024-02-10T08:49:43.632220+0000","last_clean":"2024-02-10T08:49:43.632220+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:49:43.632220+0000","last_undegraded":"2024-02-10T08:49:43.632220+0000","last_fullsized":"2024-02-10T08:49:43.632220+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:29.815324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.815324+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":797,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361884+0000","last_change":"2024-02-10T08:48:41.013610+0000","last_active":"2024-02-10T08:49:43.361884+0000","last_peered":"2024-02-10T08:49:43.361884+0000","last_clean":"2024-02-10T08:49:43.361884+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:49:43.361884+0000","last_undegraded":"2024-02-10T08:49:43.361884+0000","last_fullsized":"2024-02-10T08:49:43.361884+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.013543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.013543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210761+0000","last_change":"2024-02-10T08:49:21.886466+0000","last_active":"2024-02-10T08:49:43.210761+0000","last_peered":"2024-02-10T08:49:43.210761+0000","last_clean":"2024-02-10T08:49:43.210761+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:49:43.210761+0000","last_undegraded":"2024-02-10T08:49:43.210761+0000","last_fullsized":"2024-02-10T08:49:43.210761+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.886401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.886401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":111,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210360+0000","last_change":"2024-02-10T08:48:49.706321+0000","last_active":"2024-02-10T08:49:43.210360+0000","last_peered":"2024-02-10T08:49:43.210360+0000","last_clean":"2024-02-10T08:49:43.210360+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:49:43.210360+0000","last_undegraded":"2024-02-10T08:49:43.210360+0000","last_fullsized":"2024-02-10T08:49:43.210360+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:49.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:48:49.706258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":104,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210637+0000","last_change":"2024-02-10T08:48:52.097227+0000","last_active":"2024-02-10T08:49:43.210637+0000","last_peered":"2024-02-10T08:49:43.210637+0000","last_clean":"2024-02-10T08:49:43.210637+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:49:43.210637+0000","last_undegraded":"2024-02-10T08:49:43.210637+0000","last_fullsized":"2024-02-10T08:49:43.210637+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.097161+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":809,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631791+0000","last_change":"2024-02-10T08:48:41.102099+0000","last_active":"2024-02-10T08:49:43.631791+0000","last_peered":"2024-02-10T08:49:43.631791+0000","last_clean":"2024-02-10T08:49:43.631791+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:49:43.631791+0000","last_undegraded":"2024-02-10T08:49:43.631791+0000","last_fullsized":"2024-02-10T08:49:43.631791+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.102021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.102021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":618,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212108+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:49:44.212108+0000","last_peered":"2024-02-10T08:49:44.212108+0000","last_clean":"2024-02-10T08:49:44.212108+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:49:44.212108+0000","last_undegraded":"2024-02-10T08:49:44.212108+0000","last_fullsized":"2024-02-10T08:49:44.212108+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":211,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212397+0000","last_change":"2024-02-10T08:49:30.665713+0000","last_active":"2024-02-10T08:49:43.212397+0000","last_peered":"2024-02-10T08:49:43.212397+0000","last_clean":"2024-02-10T08:49:43.212397+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:49:43.212397+0000","last_undegraded":"2024-02-10T08:49:43.212397+0000","last_fullsized":"2024-02-10T08:49:43.212397+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":218,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362714+0000","last_change":"2024-02-10T08:49:37.297309+0000","last_active":"2024-02-10T08:49:43.362714+0000","last_peered":"2024-02-10T08:49:43.362714+0000","last_clean":"2024-02-10T08:49:43.362714+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:49:43.362714+0000","last_undegraded":"2024-02-10T08:49:43.362714+0000","last_fullsized":"2024-02-10T08:49:43.362714+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:49:37.297246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.297246+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":797,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362435+0000","last_change":"2024-02-10T08:48:00.922194+0000","last_active":"2024-02-10T08:49:43.362435+0000","last_peered":"2024-02-10T08:49:43.362435+0000","last_clean":"2024-02-10T08:49:43.362435+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:49:43.362435+0000","last_undegraded":"2024-02-10T08:49:43.362435+0000","last_fullsized":"2024-02-10T08:49:43.362435+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:00.922104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:00.922104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":773,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363549+0000","last_change":"2024-02-10T08:48:43.032173+0000","last_active":"2024-02-10T08:49:43.363549+0000","last_peered":"2024-02-10T08:49:43.363549+0000","last_clean":"2024-02-10T08:49:43.363549+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:49:43.363549+0000","last_undegraded":"2024-02-10T08:49:43.363549+0000","last_fullsized":"2024-02-10T08:49:43.363549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.032094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.032094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":120,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211389+0000","last_change":"2024-02-10T08:49:25.890744+0000","last_active":"2024-02-10T08:49:43.211389+0000","last_peered":"2024-02-10T08:49:43.211389+0000","last_clean":"2024-02-10T08:49:43.211389+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:49:43.211389+0000","last_undegraded":"2024-02-10T08:49:43.211389+0000","last_fullsized":"2024-02-10T08:49:43.211389+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.890678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.890678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":675,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211473+0000","last_change":"2024-02-10T08:48:32.990056+0000","last_active":"2024-02-10T08:49:44.211473+0000","last_peered":"2024-02-10T08:49:44.211473+0000","last_clean":"2024-02-10T08:49:44.211473+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:49:44.211473+0000","last_undegraded":"2024-02-10T08:49:44.211473+0000","last_fullsized":"2024-02-10T08:49:44.211473+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.989985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.989985+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":363,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.363399+0000","last_change":"2024-02-10T08:48:55.711815+0000","last_active":"2024-02-10T08:49:43.363399+0000","last_peered":"2024-02-10T08:49:43.363399+0000","last_clean":"2024-02-10T08:49:43.363399+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:49:43.363399+0000","last_undegraded":"2024-02-10T08:49:43.363399+0000","last_fullsized":"2024-02-10T08:49:43.363399+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.711733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.711733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":211,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.212470+0000","last_change":"2024-02-10T08:49:34.640456+0000","last_active":"2024-02-10T08:49:43.212470+0000","last_peered":"2024-02-10T08:49:43.212470+0000","last_clean":"2024-02-10T08:49:43.212470+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:49:43.212470+0000","last_undegraded":"2024-02-10T08:49:43.212470+0000","last_fullsized":"2024-02-10T08:49:43.212470+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:34.640388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.640388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":762,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.210215+0000","last_change":"2024-02-10T08:49:20.063342+0000","last_active":"2024-02-10T08:49:43.210215+0000","last_peered":"2024-02-10T08:49:43.210215+0000","last_clean":"2024-02-10T08:49:43.210215+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:49:43.210215+0000","last_undegraded":"2024-02-10T08:49:43.210215+0000","last_fullsized":"2024-02-10T08:49:43.210215+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.063276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.063276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":678,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211715+0000","last_change":"2024-02-10T08:48:44.939276+0000","last_active":"2024-02-10T08:49:44.211715+0000","last_peered":"2024-02-10T08:49:44.211715+0000","last_clean":"2024-02-10T08:49:44.211715+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:49:44.211715+0000","last_undegraded":"2024-02-10T08:49:44.211715+0000","last_fullsized":"2024-02-10T08:49:44.211715+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:48:44.939188+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.939188+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":309,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362719+0000","last_change":"2024-02-10T08:47:57.938158+0000","last_active":"2024-02-10T08:49:43.362719+0000","last_peered":"2024-02-10T08:49:43.362719+0000","last_clean":"2024-02-10T08:49:43.362719+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:49:43.362719+0000","last_undegraded":"2024-02-10T08:49:43.362719+0000","last_fullsized":"2024-02-10T08:49:43.362719+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.938096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.938096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":790,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631261+0000","last_change":"2024-02-10T08:49:39.311249+0000","last_active":"2024-02-10T08:49:43.631261+0000","last_peered":"2024-02-10T08:49:43.631261+0000","last_clean":"2024-02-10T08:49:43.631261+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:49:43.631261+0000","last_undegraded":"2024-02-10T08:49:43.631261+0000","last_fullsized":"2024-02-10T08:49:43.631261+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.311161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.311161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":192,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211268+0000","last_change":"2024-02-10T08:48:52.471904+0000","last_active":"2024-02-10T08:49:43.211268+0000","last_peered":"2024-02-10T08:49:43.211268+0000","last_clean":"2024-02-10T08:49:43.211268+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:49:43.211268+0000","last_undegraded":"2024-02-10T08:49:43.211268+0000","last_fullsized":"2024-02-10T08:49:43.211268+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:52.471831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.471831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":715,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211260+0000","last_change":"2024-02-10T08:49:38.160265+0000","last_active":"2024-02-10T08:49:44.211260+0000","last_peered":"2024-02-10T08:49:44.211260+0000","last_clean":"2024-02-10T08:49:44.211260+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:49:44.211260+0000","last_undegraded":"2024-02-10T08:49:44.211260+0000","last_fullsized":"2024-02-10T08:49:44.211260+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:49:38.160189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.160189+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":797,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361810+0000","last_change":"2024-02-10T08:49:29.384350+0000","last_active":"2024-02-10T08:49:43.361810+0000","last_peered":"2024-02-10T08:49:43.361810+0000","last_clean":"2024-02-10T08:49:43.361810+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:49:43.361810+0000","last_undegraded":"2024-02-10T08:49:43.361810+0000","last_fullsized":"2024-02-10T08:49:43.361810+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.384172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.384172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":633,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211395+0000","last_change":"2024-02-10T08:48:47.974449+0000","last_active":"2024-02-10T08:49:44.211395+0000","last_peered":"2024-02-10T08:49:44.211395+0000","last_clean":"2024-02-10T08:49:44.211395+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:49:44.211395+0000","last_undegraded":"2024-02-10T08:49:44.211395+0000","last_fullsized":"2024-02-10T08:49:44.211395+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.974390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.974390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":751,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.632017+0000","last_change":"2024-02-10T08:48:42.043077+0000","last_active":"2024-02-10T08:49:43.632017+0000","last_peered":"2024-02-10T08:49:43.632017+0000","last_clean":"2024-02-10T08:49:43.632017+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:49:43.632017+0000","last_undegraded":"2024-02-10T08:49:43.632017+0000","last_fullsized":"2024-02-10T08:49:43.632017+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.043006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.043006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":865,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.361780+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:49:43.361780+0000","last_peered":"2024-02-10T08:49:43.361780+0000","last_clean":"2024-02-10T08:49:43.361780+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:49:43.361780+0000","last_undegraded":"2024-02-10T08:49:43.361780+0000","last_fullsized":"2024-02-10T08:49:43.361780+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":204,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.211307+0000","last_change":"2024-02-10T08:48:02.600767+0000","last_active":"2024-02-10T08:49:43.211307+0000","last_peered":"2024-02-10T08:49:43.211307+0000","last_clean":"2024-02-10T08:49:43.211307+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:49:43.211307+0000","last_undegraded":"2024-02-10T08:49:43.211307+0000","last_fullsized":"2024-02-10T08:49:43.211307+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.600704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.600704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":788,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.362332+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:49:43.362332+0000","last_peered":"2024-02-10T08:49:43.362332+0000","last_clean":"2024-02-10T08:49:43.362332+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:49:43.362332+0000","last_undegraded":"2024-02-10T08:49:43.362332+0000","last_fullsized":"2024-02-10T08:49:43.362332+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":766,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-10T08:49:43.631545+0000","last_change":"2024-02-10T08:49:32.715296+0000","last_active":"2024-02-10T08:49:43.631545+0000","last_peered":"2024-02-10T08:49:43.631545+0000","last_clean":"2024-02-10T08:49:43.631545+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:49:43.631545+0000","last_undegraded":"2024-02-10T08:49:43.631545+0000","last_fullsized":"2024-02-10T08:49:43.631545+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.715176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.715176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":10732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":214,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299593,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316144,"kb_used_data":6632,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429008,"statfs":{"total":95995035648,"available":95671304192,"internally_reserved":0,"allocated":6791168,"data_stored":3611743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56100000000000005,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.751,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.59499999999999997,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70699999999999996,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.67400000000000004,"15min":0.60999999999999999},"min":{"1min":0.30399999999999999,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.80100000000000005,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.504},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.67700000000000005,"15min":0.61399999999999999},"min":{"1min":0.32100000000000001,"5min":0.28399999999999997,"15min":0.19900000000000001},"max":{"1min":0.64000000000000001,"5min":10.978,"15min":10.978},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67600000000000005,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.66000000000000003,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63600000000000001,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":7.5099999999999998,"15min":19.369},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57099999999999995,"15min":0.59299999999999997},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68899999999999995,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.496},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.72199999999999998,"15min":0.70099999999999996},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.67400000000000004,"5min":7.8700000000000001,"15min":22.102},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70399999999999996,"5min":1.581,"15min":1.581},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.66200000000000003,"15min":0.70899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.32300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.50900000000000001,"15min":0.54600000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65800000000000003,"5min":4.46,"15min":19.326000000000001},"last":0.378},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60799999999999998,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.23999999999999999,"15min":0.23899999999999999},"max":{"1min":0.76400000000000001,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.41199999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.74399999999999999,"5min":4.718,"15min":22.687000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.39700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70299999999999996,"5min":5.484,"15min":5.484},"last":0.39100000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561231,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307940,"kb_used_data":6620,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437212,"statfs":{"total":95995035648,"available":95679705088,"internally_reserved":0,"allocated":6778880,"data_stored":3611971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.55600000000000005},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61499999999999999,"5min":3.9399999999999999,"15min":8.5250000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.751,"15min":0.63300000000000001},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.73399999999999999,"15min":0.65300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.66300000000000003,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":0.69699999999999995,"5min":8.4399999999999995,"15min":18.402999999999999},"last":0.377}]},{"osd":2,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76800000000000002,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0509999999999999,"5min":10.772,"15min":10.772},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.84099999999999997,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.501}]},{"osd":3,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.79500000000000004,"15min":0.70499999999999996},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.64300000000000002,"5min":9.2650000000000006,"15min":16.823},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.443},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48899999999999999,"15min":0.47399999999999998},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.70999999999999996,"5min":3.915,"15min":3.915},"last":0.56100000000000005}]},{"osd":7,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":7.5650000000000004},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308000,"kb_used_data":6680,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437152,"statfs":{"total":95995035648,"available":95679643648,"internally_reserved":0,"allocated":6840320,"data_stored":3613646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.5669999999999999,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.372},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.39500000000000002,"15min":0.46100000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.71799999999999997,"5min":1.4910000000000001,"15min":16.166},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.47699999999999998,"15min":0.65100000000000002},"min":{"1min":0.246,"5min":0.193,"15min":0.193},"max":{"1min":1.637,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.317},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.44500000000000001,"15min":0.55600000000000005},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.93100000000000005,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.47799999999999998,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":0.51100000000000001,"5min":4.8940000000000001,"15min":24.773},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.50900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.80000000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81799999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.441,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73299999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83799999999999997,"5min":3.496,"15min":9.5600000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.63}]},{"osd":6,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.622},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.82299999999999995,"5min":3.4710000000000001,"15min":9.532},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.874,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.59499999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.73299999999999998,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":12.147,"15min":12.147},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78400000000000003,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307948,"kb_used_data":6628,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437204,"statfs":{"total":95995035648,"available":95679696896,"internally_reserved":0,"allocated":6787072,"data_stored":3611082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45700000000000002,"15min":0.40500000000000003},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.156},"max":{"1min":1.4239999999999999,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.37},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.74199999999999999,"15min":0.5},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":3.8839999999999999,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.54100000000000004,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.501,"5min":11.987,"15min":11.987},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.311}]},{"osd":3,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.502,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.51900000000000002,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.435,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75800000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.443},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.46300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79600000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.38200000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.931,"5min":4.609,"15min":4.609},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55900000000000005,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":1.7769999999999999,"15min":3.4079999999999999},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.33300000000000002},"max":{"1min":3.9609999999999999,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.48399999999999999,"5min":0.32500000000000001,"15min":0.317},"max":{"1min":0.94499999999999995,"5min":1.849,"15min":3.363},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.66000000000000003},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":8.4299999999999997},"last":0.48199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303840,"kb_used_data":6616,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441312,"statfs":{"total":95995035648,"available":95683903488,"internally_reserved":0,"allocated":6774784,"data_stored":3611231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.54800000000000004,"15min":0.502},"min":{"1min":0.214,"5min":0.183,"15min":0.183},"max":{"1min":1.2909999999999999,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.52800000000000002,"15min":0.56200000000000006},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":0.42299999999999999,"5min":10.567,"15min":14.858000000000001},"last":0.318}]},{"osd":1,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.51200000000000001,"15min":0.55700000000000005},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.63500000000000001,"5min":6.9720000000000004,"15min":12.512},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.56399999999999995,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.52200000000000002,"5min":12.875,"15min":12.875},"last":0.22900000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.61199999999999999,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50800000000000001,"5min":10.994999999999999,"15min":11.612},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":7.8280000000000003,"15min":11.710000000000001},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":1.298,"15min":1.298},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.41099999999999998,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.79100000000000004,"5min":10.34,"15min":10.34},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.71199999999999997,"15min":0.66500000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3380000000000001,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.753,"15min":0.71499999999999997},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.79800000000000004,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.71899999999999997,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.76600000000000001,"5min":10.539,"15min":10.539},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.77800000000000002,"15min":0.70899999999999996},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.32500000000000001},"max":{"1min":0.70199999999999996,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.57099999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182622,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303876,"kb_used_data":6652,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441276,"statfs":{"total":95995035648,"available":95683866624,"internally_reserved":0,"allocated":6811648,"data_stored":3611939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.20699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.497,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.251,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.58599999999999997,"5min":10.484999999999999,"15min":11.808},"last":0.252}]},{"osd":1,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.76900000000000002,"15min":0.67200000000000004},"min":{"1min":0.248,"5min":0.215,"15min":0.184},"max":{"1min":0.53800000000000003,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.26000000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.58799999999999997,"5min":17.343,"15min":17.343},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.754,"15min":0.68000000000000005},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":0.57299999999999995,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.443,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57199999999999995,"5min":5.0259999999999998,"15min":23.294},"last":0.23799999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:49:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84199999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55700000000000005,"15min":0.70199999999999996},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73399999999999999,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.62},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.73999999999999999,"15min":0.70299999999999996},"min":{"1min":0.39000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82999999999999996,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Feb 10 08:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58999999999999997,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79200000000000004,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.73199999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":8.6140000000000008,"15min":14.773999999999999},"last":0.629},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.746},"min":{"1min":0.45300000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79700000000000004,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.64100000000000001}]}]},{"osd":4,"up_from":633,"seq":2718714298410,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310004,"kb_used_data":6636,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93435148,"statfs":{"total":95995035648,"available":95677591552,"internally_reserved":0,"allocated":6795264,"data_stored":3612109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.66500000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36299999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.65200000000000002,"5min":0.754,"15min":0.754},"last":0.49099999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66100000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59999999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.315,"15min":0.315},"max":{"1min":0.63700000000000001,"5min":1.036,"15min":1.036},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":0.746,"15min":0.746},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.63600000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51700000000000002,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68600000000000005,"5min":0.87,"15min":0.87},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52800000000000002,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.67500000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.505},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69699999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.32200000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.624},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71699999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.55900000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790164,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312032,"kb_used_data":6616,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433120,"statfs":{"total":95995035648,"available":95675514880,"internally_reserved":0,"allocated":6774784,"data_stored":3611042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58099999999999996,"15min":1.05},"min":{"1min":0.29499999999999998,"5min":0.249,"15min":0.216},"max":{"1min":0.93500000000000005,"5min":7.923,"15min":107.551},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.53200000000000003,"15min":0.97499999999999998},"min":{"1min":0.36899999999999999,"5min":0.27800000000000002,"15min":0.255},"max":{"1min":0.85499999999999998,"5min":2.9590000000000001,"15min":88.792000000000002},"last":0.64900000000000002}]},{"osd":1,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":1.0069999999999999},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.216},"max":{"1min":0.82199999999999995,"5min":12.375999999999999,"15min":107.205},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.67700000000000005,"15min":0.88300000000000001},"min":{"1min":0.45100000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.90000000000000002,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.55400000000000005}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.66800000000000004,"15min":1.0349999999999999},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":1.4650000000000001,"5min":9.9600000000000009,"15min":54.174999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58299999999999996,"15min":1.073},"min":{"1min":0.40100000000000002,"5min":0.26900000000000002,"15min":0.26600000000000001},"max":{"1min":1.405,"5min":2.335,"15min":86.183000000000007},"last":0.57899999999999996}]},{"osd":3,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66700000000000004,"15min":0.86399999999999999},"min":{"1min":0.41799999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":1.0529999999999999,"5min":9.9900000000000002,"15min":44.395000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":1.29},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":0.92200000000000004,"5min":8.0120000000000005,"15min":114.526},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.73299999999999998},"min":{"1min":0.28999999999999998,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":0.82299999999999995,"5min":10.023999999999999,"15min":72.241},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81000000000000005,"5min":10.013,"15min":22.469999999999999},"last":0.433}]},{"osd":7,"last update":"Sat Feb 10 08:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.50800000000000001,"15min":0.72799999999999998},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.96699999999999997,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52200000000000002,"15min":0.75900000000000001},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.253},"max":{"1min":0.79200000000000004,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:49:54.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:53.265424+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:49:54.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:53.268894+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:49:54.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:53.580832+0000 mgr.y (mgr.14553) 671 : cluster [DBG] pgmap v1232: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:49:54.922 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:53.646606+0000 mgr.y (mgr.14553) 672 : audit [DBG] from='client.49153 v1:172.21.15.107:0/1588507206' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:53.265424+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:49:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:53.268894+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:49:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:53.580832+0000 mgr.y (mgr.14553) 671 : cluster [DBG] pgmap v1232: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:49:54.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:53.646606+0000 mgr.y (mgr.14553) 672 : audit [DBG] from='client.49153 v1:172.21.15.107:0/1588507206' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:53.265424+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:49:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:53.268894+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:49:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:53.580832+0000 mgr.y (mgr.14553) 671 : cluster [DBG] pgmap v1232: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:49:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:53.646606+0000 mgr.y (mgr.14553) 672 : audit [DBG] from='client.49153 v1:172.21.15.107:0/1588507206' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:49:55.144 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:49:55.144 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:49:55.144 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:49:55.144 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:49:55.144 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:49:55.755 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:54.608293+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:49:55.755 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:54.609740+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:49:55.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T08:49:55.928 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:54.608293+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:49:55.928 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:54.609740+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:49:55.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:54.608293+0000 osd.0 (osd.0) 182 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:49:55.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:54.609740+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:49:55.929 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:49:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:49:55] "GET /metrics HTTP/1.1" 200 186417 "" "Prometheus/2.33.4" 2024-02-10T08:49:56.021 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.190+0000 7fe23c285700 1 -- 172.21.15.107:0/2911492835 >> v1:172.21.15.107:6790/0 conn(0x7fe2341007f0 legacy=0x7fe234102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- 172.21.15.107:0/2911492835 shutdown_connections 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- 172.21.15.107:0/2911492835 >> 172.21.15.107:0/2911492835 conn(0x7fe2340f3640 msgr2=0x7fe2340f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- 172.21.15.107:0/2911492835 shutdown_connections 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- 172.21.15.107:0/2911492835 wait complete. 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 librados: starting msgr at 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 librados: starting objecter 2024-02-10T08:49:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 Processor -- start 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- start start 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 librados: setting wanted keys 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 librados: calling monclient init 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2341172a0 con 0x7fe234104330 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe234118480 con 0x7fe2340fda50 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe23c285700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe234119680 con 0x7fe2341007f0 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe239820700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe2341007f0 0x7fe2340fa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46404/0 (socket says 172.21.15.107:46404) 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.191+0000 7fe239820700 1 -- 172.21.15.107:0/1366410399 learned_addr learned my addr 172.21.15.107:0/1366410399 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter ms_handle_connect 0x7fe2341007f0 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter ms_handle_connect 0x7fe234104330 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342584544 0 0) 0x7fe234119680 con 0x7fe2341007f0 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe220003030 con 0x7fe2341007f0 2024-02-10T08:49:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter ms_handle_connect 0x7fe2340fda50 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 10 client.?.objecter resend_mon_ops 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360555894 0 0) 0x7fe2341172a0 con 0x7fe234104330 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe234119680 con 0x7fe234104330 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797320220 0 0) 0x7fe234118480 con 0x7fe2340fda50 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2341172a0 con 0x7fe2340fda50 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2148019457 0 0) 0x7fe220003030 con 0x7fe2341007f0 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe234118480 con 0x7fe2341007f0 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe224002cb0 con 0x7fe2341007f0 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1355966605 0 0) 0x7fe234119680 con 0x7fe234104330 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe220003030 con 0x7fe234104330 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe22c001970 con 0x7fe234104330 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334975061 0 0) 0x7fe2341172a0 con 0x7fe2340fda50 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe234119680 con 0x7fe2340fda50 2024-02-10T08:49:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe2300016a0 con 0x7fe2340fda50 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 640049442 0 0) 0x7fe234118480 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 >> v1:172.21.15.195:6789/0 conn(0x7fe2340fda50 legacy=0x7fe2340f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 >> v1:172.21.15.107:6789/0 conn(0x7fe234104330 legacy=0x7fe2341159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23411a880 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe234117490 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe23c285700 10 client.54443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe234117900 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.192+0000 7fe23c285700 1 librados: init done 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.193+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe224001ab0 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.193+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe224003860 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.193+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe218004f20 con 0x7fe2341007f0 2024-02-10T08:49:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe2240039c0 con 0x7fe2341007f0 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(806..806 src has 754..806) v4 ==== 7386+0+0 (unknown 673814851 0 0) 0x7fe224078dd0 con 0x7fe2341007f0 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 10 client.54443.objecter ms_dispatch 0x7fe234001550 osd_map(806..806 src has 754..806) v4 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 3 client.54443.objecter handle_osd_map got epochs [806,806] > 0 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 3 client.54443.objecter handle_osd_map decoding full epoch 806 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 20 client.54443.objecter dump_active .. 0 homeless 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.194+0000 7fe22b779700 10 client.54443.objecter ms_handle_connect 0x7fe220063c50 2024-02-10T08:49:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.198+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe22404c520 con 0x7fe2341007f0 2024-02-10T08:49:56.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:56.377+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe2180051d0 con 0x7fe2341007f0 2024-02-10T08:49:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:55.258786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-02-10T08:49:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:55.262448+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-02-10T08:49:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:55.581223+0000 mgr.y (mgr.14553) 673 : cluster [DBG] pgmap v1233: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:49:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:56.377958+0000 mon.c (mon.2) 496 : audit [INF] from='client.? v1:172.21.15.107:0/1366410399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:56.378276+0000 mon.a (mon.0) 3707 : audit [INF] from='client.54443 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:56.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:55.258786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-02-10T08:49:56.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:55.262448+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-02-10T08:49:56.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:55.581223+0000 mgr.y (mgr.14553) 673 : cluster [DBG] pgmap v1233: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:49:56.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:56.377958+0000 mon.c (mon.2) 496 : audit [INF] from='client.? v1:172.21.15.107:0/1366410399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:56.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:56.378276+0000 mon.a (mon.0) 3707 : audit [INF] from='client.54443 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:55.258786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d scrub starts 2024-02-10T08:49:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:55.262448+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub ok 2024-02-10T08:49:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:55.581223+0000 mgr.y (mgr.14553) 673 : cluster [DBG] pgmap v1233: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:49:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:56.377958+0000 mon.c (mon.2) 496 : audit [INF] from='client.? v1:172.21.15.107:0/1366410399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:57.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:56.378276+0000 mon.a (mon.0) 3707 : audit [INF] from='client.54443 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T08:49:57.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.212+0000 7fe22b779700 1 -- 172.21.15.107:0/1366410399 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v807) v1 ==== 101+0+0 (unknown 3782061003 0 0) 0x7fe22404a990 con 0x7fe2341007f0 2024-02-10T08:49:57.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 10 librados: watch_flush enter 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 10 librados: watch_flush exit 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 20 client.54443.objecter shutdown clearing up homeless session... 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 10 client.54443.objecter successfully canceled tick 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe220063c50 legacy=0x7fe220066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 >> v1:172.21.15.107:6790/0 conn(0x7fe2341007f0 legacy=0x7fe2340fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 shutdown_connections 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 >> 172.21.15.107:0/1366410399 conn(0x7fe2340f3640 msgr2=0x7fe2341077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 shutdown_connections 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 -- 172.21.15.107:0/1366410399 wait complete. 2024-02-10T08:49:57.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:49:57.215+0000 7fe23c285700 1 librados: shutdown 2024-02-10T08:49:57.217 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-02-10T08:49:57.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:49:57.587 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53393.0:1 195.1s0 195:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e330)", 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 573.55856619300005, 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441, 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.588 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53393", 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4089636740", 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027098+0000", 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027094+0000", 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1878999999999999e-05 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027106+0000", 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9205999999999999e-05 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027125+0000", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0699999999999999e-05 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.027136+0000", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017503530000000001 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028886+0000", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7790000000000003e-06 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:49:57.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:24.028891+0000", 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.008200303 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037091+0000", 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014759500000000001 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037239+0000", 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048123099999999999 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.037720+0000", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015191219999999999 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.039239+0000", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001215224 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040455+0000", 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0549e-05 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:25.040505+0000", 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.013407441 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1660 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94547991866752 gen 10] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 382.40073362700002, 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001, 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.594 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1660 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184930+0000", 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:49:57.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184927+0000", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.077e-05 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184938+0000", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3421000000000001e-05 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184951+0000", 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6079999999999998e-06 2024-02-10T08:49:57.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184959+0000", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1104000000000003e-05 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185020+0000", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0170000000000002e-06 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89132679699999995 2024-02-10T08:49:57.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076356+0000", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013946 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.076495+0000", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011721326000000001 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.088217+0000", 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90328652700000001 2024-02-10T08:49:57.598 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1661 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94547991863296 gen 4] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 382.40068534400001, 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999, 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1661 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.599 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184978+0000", 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-02-10T08:49:57.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184981+0000", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1167e-05 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184993+0000", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9099999999999996e-06 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184998+0000", 2024-02-10T08:49:57.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2206999999999997e-05 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185050+0000", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1770000000000004e-06 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185059+0000", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88333866299999997 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068398+0000", 2024-02-10T08:49:57.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172676 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.068570+0000", 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044926389999999997 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.073063+0000", 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88808445899999999 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:61 60.13 60:ca6ce4f9:test-rados-api-smithi107-112490-78::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.43694809200002, 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999, 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148716+0000", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148715+0000", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1810000000000003e-06 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148719+0000", 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9329999999999998e-06 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148725+0000", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1559999999999997e-06 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148729+0000", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4011000000000006e-05 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148813+0000", 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0549999999999996e-06 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148819+0000", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080372289999999999 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156857+0000", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8814999999999997e-05 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.156935+0000", 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017242200000000001 2024-02-10T08:49:57.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157108+0000", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016798900000000001 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.158788+0000", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072491250000000004 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166037+0000", 2024-02-10T08:49:57.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013801147 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179838+0000", 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0494000000000001e-05 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179889+0000", 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8433999999999998e-05 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179927+0000", 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031211223999999999 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:62 60.1f 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.43692367699998, 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006, 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148740+0000", 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4249999999999998e-06 2024-02-10T08:49:57.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148743+0000", 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2950000000000001e-06 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148748+0000", 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0949999999999998e-06 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148752+0000", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015047300000000001 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148903+0000", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.639e-06 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015726237000000001 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164634+0000", 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1116e-05 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164725+0000", 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016155299999999999 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164886+0000", 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016114943 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.181001+0000", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099756480000000002 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190977+0000", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033742560999999997 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224720+0000", 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1842999999999998e-05 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224771+0000", 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8983999999999998e-05 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224810+0000", 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076070277000000006 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.616 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:65 60.7 60:e0e5fb82:test-rados-api-smithi107-112490-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.43689809900002, 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999, 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148766+0000", 2024-02-10T08:49:57.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148765+0000", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.311e-06 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148773+0000", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5339999999999999e-06 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148778+0000", 2024-02-10T08:49:57.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0620000000000002e-06 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148782+0000", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014662 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148929+0000", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8380000000000001e-06 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148934+0000", 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015063796000000001 2024-02-10T08:49:57.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.163997+0000", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164103+0000", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017560199999999999 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.164279+0000", 2024-02-10T08:49:57.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016715187999999999 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180994+0000", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097823279999999999 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190776+0000", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8204999999999997e-05 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190825+0000", 2024-02-10T08:49:57.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5817999999999997e-05 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.190911+0000", 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042144793999999999 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:66 60.15 60:ab1b3caf:test-rados-api-smithi107-112490-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.436877552, 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002, 2024-02-10T08:49:57.622 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148786+0000", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2550000000000002e-06 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148789+0000", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1379999999999998e-06 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148794+0000", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8199999999999998e-06 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148798+0000", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109837 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148908+0000", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5809999999999996e-06 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148913+0000", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013380609999999999 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162294+0000", 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012214800000000001 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162416+0000", 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020241599999999999 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162619+0000", 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004775708 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.167394+0000", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057395455999999997 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224790+0000", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4917999999999999e-05 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224835+0000", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7835000000000002e-05 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.224923+0000", 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076136251000000002 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:69 60.13 60:ce87a4eb:test-rados-api-smithi107-112490-78::5a02e6184a18760c5f8050d0e5b120c3d7a79519:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e493)", 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 358.43685160000001, 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999, 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148812+0000", 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6509999999999999e-06 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148815+0000", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5979999999999996e-06 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148821+0000", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5329999999999996e-06 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148825+0000", 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7842999999999997e-05 2024-02-10T08:49:57.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148863+0000", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3610000000000003e-06 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.148868+0000", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084767599999999999 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157344+0000", 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010973599999999999 2024-02-10T08:49:57.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157454+0000", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020398000000000001 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.157658+0000", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004485096 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.162143+0000", 2024-02-10T08:49:57.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017667062000000001 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179810+0000", 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147684 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179958+0000", 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6878000000000002e-05 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.179995+0000", 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9448e-05 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180024+0000", 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031212100999999999 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(3.2 REQUEST e494)", 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 357.34089140999998, 2024-02-10T08:49:57.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998, 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244772+0000", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.517e-06 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244776+0000", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5489999999999999e-05 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244791+0000", 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9300000000000003e-06 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.244799+0000", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025046212000000002 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269845+0000", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455999999999999e-05 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269866+0000", 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3003999999999998e-05 2024-02-10T08:49:57.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.269899+0000", 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025126231999999998 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:46cb1ec1:reshard::reshard.0000000003:head)])", 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 169.390500143, 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003791034, 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.638 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195164+0000", 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195163+0000", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1680000000000003e-06 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195168+0000", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4689999999999997e-06 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195176+0000", 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.229e-06 2024-02-10T08:49:57.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.195183+0000", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036324640000000002 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.198815+0000", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000139739 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.198955+0000", 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003791034 2024-02-10T08:49:57.641 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:4798f7e9:::obj_delete_at_hint.0000000007:head)])", 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 169.30024492999999, 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040317240000000004, 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:49:57.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285419+0000", 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285418+0000", 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0409999999999998e-06 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285426+0000", 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2284e-05 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285438+0000", 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6699999999999997e-06 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.285445+0000", 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037752670000000001 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.289220+0000", 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023066100000000001 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.289451+0000", 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040317240000000004 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:4578209b:::obj_delete_at_hint.0000000014:head)])", 2024-02-10T08:49:57.645 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 169.209813311, 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038931109999999999, 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375851+0000", 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:49:57.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375850+0000", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.994e-06 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375855+0000", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1009999999999998e-06 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375864+0000", 2024-02-10T08:49:57.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5399999999999997e-06 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.375868+0000", 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037501779999999998 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.379618+0000", 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000125291 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.379744+0000", 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038931109999999999 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:479ba34f:::obj_delete_at_hint.0000000021:head)])", 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 169.119335855, 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038335589999999998, 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466328+0000", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466327+0000", 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7039999999999999e-06 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466332+0000", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5619999999999993e-06 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466342+0000", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2559999999999996e-06 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.466349+0000", 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003671058 2024-02-10T08:49:57.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.470020+0000", 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141342 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.470162+0000", 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038335589999999998 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.652 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.2 673/672 [PushReplyOp(4:47403372:::obj_delete_at_hint.0000000112:head)])", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 169.02897203200001, 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038613470000000002, 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556692+0000", 2024-02-10T08:49:57.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556691+0000", 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6399999999999999e-06 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556695+0000", 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7899999999999996e-06 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556701+0000", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5199999999999997e-06 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.556706+0000", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037127570000000001 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.560419+0000", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134046 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.560553+0000", 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038613470000000002 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1428 244.1c 244:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e693)", 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 144.42729756899999, 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046145960000000003, 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1428 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158366+0000", 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158363+0000", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4042e-05 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158377+0000", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3719e-05 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158391+0000", 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2459999999999997e-06 2024-02-10T08:49:57.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158398+0000", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4943e-05 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158443+0000", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000182914 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.158626+0000", 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034397 2024-02-10T08:49:57.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162066+0000", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9436000000000004e-05 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162135+0000", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010961 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162245+0000", 2024-02-10T08:49:57.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043155400000000001 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162676+0000", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203284 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162879+0000", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8809000000000001e-05 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162908+0000", 2024-02-10T08:49:57.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3383000000000002e-05 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162942+0000", 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9164e-05 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.162981+0000", 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046145960000000003 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.662 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1473 244.1c 244:3fcfbd52:test-rados-api-smithi107-112490-121::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e693)", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 144.397135464, 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029607119999999999, 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1473 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:49:57.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188528+0000", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188527+0000", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7519999999999998e-06 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188531+0000", 2024-02-10T08:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.498e-06 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188539+0000", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8809999999999998e-06 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188542+0000", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8841999999999999e-05 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188571+0000", 2024-02-10T08:49:57.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9135999999999997e-05 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.188640+0000", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026846190000000001 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191325+0000", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010083 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191426+0000", 2024-02-10T08:49:57.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3165999999999998e-05 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:33.191489+0000", 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029607119999999999 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.4.633:8 234.3s0 234:dae49067:test-rados-api-smithi107-112490-127::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e723)", 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.195083+0000", 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.39058109600001, 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016728881000000001, 2024-02-10T08:49:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195083+0000", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195083+0000", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195079+0000", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1593999999999999e-05 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195091+0000", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5458e-05 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195106+0000", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3890000000000002e-06 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195113+0000", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4351999999999998e-05 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195157+0000", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9165000000000003e-05 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.195256+0000", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011721809 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:49:57.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.206978+0000", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044697579999999999 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.211448+0000", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8176000000000006e-05 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.211516+0000", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029554499999999998 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.211812+0000", 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016728881000000001 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2771 251.18 251:18000000:%01:::head [pgnls start_epoch 771] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e771)", 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:04.640458+0000", 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.945205768999998, 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038627650000000002, 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.672 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2771 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640458+0000", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640458+0000", 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640454+0000", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8603000000000001e-05 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640472+0000", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3626999999999999e-05 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640496+0000", 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9520000000000006e-06 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640506+0000", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3534000000000004e-05 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640579+0000", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5100000000000001e-06 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.640587+0000", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035400309999999999 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.644127+0000", 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9964999999999999e-05 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.644187+0000", 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134012 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.644321+0000", 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038627650000000002 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2968 252.1b 252:d8000000:%01:::head [pgnls start_epoch 787] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e787)", 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:26.218606+0000", 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 31.367058117999999, 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029298000000000002, 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2968 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218606+0000", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218606+0000", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218603+0000", 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0316000000000001e-05 2024-02-10T08:49:57.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218613+0000", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3212e-05 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218626+0000", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3609999999999998e-05 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218660+0000", 2024-02-10T08:49:57.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1415000000000007e-05 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218731+0000", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5010000000000001e-06 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.218738+0000", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025797149999999998 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.221317+0000", 2024-02-10T08:49:57.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011921000000000001 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.221437+0000", 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8837000000000003e-05 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.221536+0000", 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029298000000000002 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(234.4s2 790/708 ECSubWrite(tid=797, reqid=client.48721.0:3045, at_version=790'22, trim_to=0'0, roll_forward_to=749'21))", 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:29.218402+0000", 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.367261535000001, 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031663540000000001, 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218402+0000", 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218402+0000", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218401+0000", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0216999999999999e-05 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218431+0000", 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0730999999999999e-05 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218442+0000", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2330000000000003e-06 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218448+0000", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1514000000000003e-05 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218490+0000", 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.221e-05 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.218502+0000", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030501479999999999 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.221552+0000", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6722999999999999e-05 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:29.221569+0000", 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031663540000000001 2024-02-10T08:49:57.685 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:49:57.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:49:57.686 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.134639+0000 osd.2 (osd.2) 315 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.138234+0000 osd.2 (osd.2) 316 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.300266+0000 osd.3 (osd.3) 361 : cluster [DBG] 2.b scrub starts 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.301362+0000 osd.3 (osd.3) 362 : cluster [DBG] 2.b scrub ok 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.902944+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.a scrub starts 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:56.906482+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.a scrub ok 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.156777+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.159812+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:49:57.212242+0000 mon.a (mon.0) 3708 : audit [INF] from='client.54443 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:57.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.212311+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.134639+0000 osd.2 (osd.2) 315 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.138234+0000 osd.2 (osd.2) 316 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.300266+0000 osd.3 (osd.3) 361 : cluster [DBG] 2.b scrub starts 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.301362+0000 osd.3 (osd.3) 362 : cluster [DBG] 2.b scrub ok 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.902944+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.a scrub starts 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:56.906482+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.a scrub ok 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.156777+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.159812+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:49:57.212242+0000 mon.a (mon.0) 3708 : audit [INF] from='client.54443 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:57.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.212311+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-10T08:49:57.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.134639+0000 osd.2 (osd.2) 315 : cluster [DBG] 16.17 scrub starts 2024-02-10T08:49:57.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.138234+0000 osd.2 (osd.2) 316 : cluster [DBG] 16.17 scrub ok 2024-02-10T08:49:57.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.300266+0000 osd.3 (osd.3) 361 : cluster [DBG] 2.b scrub starts 2024-02-10T08:49:57.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.301362+0000 osd.3 (osd.3) 362 : cluster [DBG] 2.b scrub ok 2024-02-10T08:49:57.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.902944+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.a scrub starts 2024-02-10T08:49:57.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:56.906482+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.a scrub ok 2024-02-10T08:49:57.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.156777+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:49:57.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.159812+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:49:57.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:49:57.212242+0000 mon.a (mon.0) 3708 : audit [INF] from='client.54443 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T08:49:57.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.212311+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-10T08:49:58.169 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:49:58.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T08:49:58.520 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.318082+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.322388+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.581716+0000 mgr.y (mgr.14553) 674 : cluster [DBG] pgmap v1235: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.924112+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:57.927309+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:49:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.131082+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.134380+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.318082+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.322388+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.581716+0000 mgr.y (mgr.14553) 674 : cluster [DBG] pgmap v1235: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.924112+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:57.927309+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.131082+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:49:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.134380+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.318082+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.322388+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.581716+0000 mgr.y (mgr.14553) 674 : cluster [DBG] pgmap v1235: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.924112+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:57.927309+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:49:58.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.131082+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:49:58.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.134380+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:49:59.856 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.183159+0000 osd.2 (osd.2) 317 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:49:59.856 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.186640+0000 osd.2 (osd.2) 318 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.955958+0000 osd.7 (osd.7) 334 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:58.959421+0000 osd.7 (osd.7) 335 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:59.156219+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:59.159800+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.183159+0000 osd.2 (osd.2) 317 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.186640+0000 osd.2 (osd.2) 318 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.955958+0000 osd.7 (osd.7) 334 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:58.959421+0000 osd.7 (osd.7) 335 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:49:59.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:59.156219+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:49:59.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:49:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:59.159800+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.183159+0000 osd.2 (osd.2) 317 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.186640+0000 osd.2 (osd.2) 318 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.955958+0000 osd.7 (osd.7) 334 : cluster [DBG] 5.9 scrub starts 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:58.959421+0000 osd.7 (osd.7) 335 : cluster [DBG] 5.9 scrub ok 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:59.156219+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 scrub starts 2024-02-10T08:49:59.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:49:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:59.159800+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.4 scrub ok 2024-02-10T08:50:00.135 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:00.135 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:00.135 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:50:00.135 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.119+0000 7f7f4cd50700 1 -- 172.21.15.107:0/2234759440 >> v1:172.21.15.107:6790/0 conn(0x7f7f481007f0 legacy=0x7f7f48102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 -- 172.21.15.107:0/2234759440 shutdown_connections 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 -- 172.21.15.107:0/2234759440 >> 172.21.15.107:0/2234759440 conn(0x7f7f480f3640 msgr2=0x7f7f480f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 -- 172.21.15.107:0/2234759440 shutdown_connections 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 -- 172.21.15.107:0/2234759440 wait complete. 2024-02-10T08:50:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 librados: starting msgr at 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.120+0000 7f7f4cd50700 1 librados: starting objecter 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 Processor -- start 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 -- start start 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 librados: setting wanted keys 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 librados: calling monclient init 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f481171d0 con 0x7f7f48104330 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f481183b0 con 0x7f7f480fda50 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f4cd50700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f481195b0 con 0x7f7f481007f0 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f45d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7f481007f0 0x7f7f480f9ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41998/0 (socket says 172.21.15.107:41998) 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f45d9b700 1 -- 172.21.15.107:0/3007238984 learned_addr learned my addr 172.21.15.107:0/3007238984 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f46d9d700 1 --1- 172.21.15.107:0/3007238984 >> v1:172.21.15.107:6789/0 conn(0x7f7f48104330 0x7f7f481159b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49276/0 (socket says 172.21.15.107:49276) 2024-02-10T08:50:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter ms_handle_connect 0x7f7f481007f0 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter ms_handle_connect 0x7f7f48104330 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter ms_handle_connect 0x7f7f480fda50 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081382331 0 0) 0x7f7f481195b0 con 0x7f7f481007f0 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f300038e0 con 0x7f7f481007f0 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138945929 0 0) 0x7f7f481171d0 con 0x7f7f48104330 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f481195b0 con 0x7f7f48104330 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597009317 0 0) 0x7f7f481183b0 con 0x7f7f480fda50 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f481171d0 con 0x7f7f480fda50 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.121+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2191062137 0 0) 0x7f7f300038e0 con 0x7f7f481007f0 2024-02-10T08:50:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f481183b0 con 0x7f7f481007f0 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f3c002de0 con 0x7f7f481007f0 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280611021 0 0) 0x7f7f481195b0 con 0x7f7f48104330 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f300038e0 con 0x7f7f48104330 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f40001c10 con 0x7f7f48104330 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719490769 0 0) 0x7f7f481171d0 con 0x7f7f480fda50 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f481195b0 con 0x7f7f480fda50 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f38001840 con 0x7f7f480fda50 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3297436010 0 0) 0x7f7f481183b0 con 0x7f7f481007f0 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 >> v1:172.21.15.195:6789/0 conn(0x7f7f480fda50 legacy=0x7f7f480f97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 >> v1:172.21.15.107:6789/0 conn(0x7f7f48104330 legacy=0x7f7f481159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:00.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4811a7b0 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f481173c0 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f4cd50700 10 client.54449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f481178b0 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f4cd50700 1 librados: init done 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7f3c002400 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.122+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f3c0038d0 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28004f20 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7f3c003180 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(807..807 src has 754..807) v4 ==== 7386+0+0 (unknown 1989828987 0 0) 0x7f7f3c078d20 con 0x7f7f481007f0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f377fe700 10 client.54449.objecter ms_dispatch 0x7f7f48001550 osd_map(807..807 src has 754..807) v4 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f377fe700 3 client.54449.objecter handle_osd_map got epochs [807,807] > 0 2024-02-10T08:50:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.123+0000 7f7f377fe700 3 client.54449.objecter handle_osd_map decoding full epoch 807 2024-02-10T08:50:00.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.124+0000 7f7f377fe700 20 client.54449.objecter dump_active .. 0 homeless 2024-02-10T08:50:00.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.124+0000 7f7f377fe700 10 client.54449.objecter ms_handle_connect 0x7f7f30063cc0 2024-02-10T08:50:00.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.128+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7f3c04c450 con 0x7f7f481007f0 2024-02-10T08:50:00.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.290+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f28000b90 con 0x7f7f30063cc0 2024-02-10T08:50:00.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.301+0000 7f7f377fe700 1 -- 172.21.15.107:0/3007238984 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+410082 (unknown 2965378022 0 1652068935) 0x7f7f28000b90 con 0x7f7f30063cc0 2024-02-10T08:50:00.308 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 10 librados: watch_flush enter 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 10 librados: watch_flush exit 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 20 client.54449.objecter shutdown clearing up homeless session... 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 10 client.54449.objecter successfully canceled tick 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7f30063cc0 legacy=0x7f7f30066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 >> v1:172.21.15.107:6790/0 conn(0x7f7f481007f0 legacy=0x7f7f480f9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 shutdown_connections 2024-02-10T08:50:00.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 >> 172.21.15.107:0/3007238984 conn(0x7f7f480f3640 msgr2=0x7f7f480f6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 shutdown_connections 2024-02-10T08:50:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 -- 172.21.15.107:0/3007238984 wait complete. 2024-02-10T08:50:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:00.304+0000 7f7f4cd50700 1 librados: shutdown 2024-02-10T08:50:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:50:00.486 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:00.766 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:59.518601+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:59.520570+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:49:59.582130+0000 mgr.y (mgr.14553) 675 : cluster [DBG] pgmap v1236: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.000117+0000 mon.a (mon.0) 3710 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.000141+0000 mon.a (mon.0) 3711 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.000153+0000 mon.a (mon.0) 3712 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.000165+0000 mon.a (mon.0) 3713 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.171870+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.175422+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.263028+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:50:00.767 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.264468+0000 osd.3 (osd.3) 364 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:50:00.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T08:50:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:59.518601+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:50:00.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:59.520570+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:49:59.582130+0000 mgr.y (mgr.14553) 675 : cluster [DBG] pgmap v1236: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.000117+0000 mon.a (mon.0) 3710 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.000141+0000 mon.a (mon.0) 3711 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.000153+0000 mon.a (mon.0) 3712 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.000165+0000 mon.a (mon.0) 3713 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.171870+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.175422+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.263028+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.264468+0000 osd.3 (osd.3) 364 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:50:00.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:59.518601+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:59.520570+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:49:59.582130+0000 mgr.y (mgr.14553) 675 : cluster [DBG] pgmap v1236: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.000117+0000 mon.a (mon.0) 3710 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.000141+0000 mon.a (mon.0) 3711 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.000153+0000 mon.a (mon.0) 3712 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.000165+0000 mon.a (mon.0) 3713 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.171870+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.175422+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.263028+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:50:00.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.264468+0000 osd.3 (osd.3) 364 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:50:00.951 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1236,"stamp":"2024-02-10T08:49:59.581934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1352,"num_read_kb":1104,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":13194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":462,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2469136,"kb_used_data":50200,"kb_used_omap":59,"kb_used_meta":2416580,"kb_avail":747492080,"statfs":{"total":767960285184,"available":765431889920,"internally_reserved":0,"allocated":51404800,"data_stored":26892027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2474578539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002113"},"pg_stats":[{"pgid":"5.9","version":"636'18","reported_seq":501,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679810+0000","last_change":"2024-02-10T08:48:29.848382+0000","last_active":"2024-02-10T08:49:51.679810+0000","last_peered":"2024-02-10T08:49:51.679810+0000","last_clean":"2024-02-10T08:49:51.679810+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:49:51.679810+0000","last_undegraded":"2024-02-10T08:49:51.679810+0000","last_fullsized":"2024-02-10T08:49:51.679810+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:48:29.848301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.848301+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":872,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226024+0000","last_change":"2024-02-10T08:47:58.959348+0000","last_active":"2024-02-10T08:49:45.226024+0000","last_peered":"2024-02-10T08:49:45.226024+0000","last_clean":"2024-02-10T08:49:45.226024+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:49:45.226024+0000","last_undegraded":"2024-02-10T08:49:45.226024+0000","last_fullsized":"2024-02-10T08:49:45.226024+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:47:58.959271+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.959271+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":810,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212514+0000","last_change":"2024-02-10T08:49:41.119323+0000","last_active":"2024-02-10T08:49:44.212514+0000","last_peered":"2024-02-10T08:49:44.212514+0000","last_clean":"2024-02-10T08:49:44.212514+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:49:44.212514+0000","last_undegraded":"2024-02-10T08:49:44.212514+0000","last_fullsized":"2024-02-10T08:49:44.212514+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:41.119237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:49:41.119237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":791,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226206+0000","last_change":"2024-02-10T08:48:01.033569+0000","last_active":"2024-02-10T08:49:45.226206+0000","last_peered":"2024-02-10T08:49:45.226206+0000","last_clean":"2024-02-10T08:49:45.226206+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:49:45.226206+0000","last_undegraded":"2024-02-10T08:49:45.226206+0000","last_fullsized":"2024-02-10T08:49:45.226206+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:01.033502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:01.033502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":751,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212366+0000","last_change":"2024-02-10T08:49:23.108122+0000","last_active":"2024-02-10T08:49:44.212366+0000","last_peered":"2024-02-10T08:49:44.212366+0000","last_clean":"2024-02-10T08:49:44.212366+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:49:44.212366+0000","last_undegraded":"2024-02-10T08:49:44.212366+0000","last_fullsized":"2024-02-10T08:49:44.212366+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.108061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":253,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589213+0000","last_change":"2024-02-10T08:48:32.371647+0000","last_active":"2024-02-10T08:49:45.589213+0000","last_peered":"2024-02-10T08:49:45.589213+0000","last_clean":"2024-02-10T08:49:45.589213+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:49:45.589213+0000","last_undegraded":"2024-02-10T08:49:45.589213+0000","last_fullsized":"2024-02-10T08:49:45.589213+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.371572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.371572+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212149+0000","last_change":"2024-02-10T08:48:38.645376+0000","last_active":"2024-02-10T08:49:44.212149+0000","last_peered":"2024-02-10T08:49:44.212149+0000","last_clean":"2024-02-10T08:49:44.212149+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:49:44.212149+0000","last_undegraded":"2024-02-10T08:49:44.212149+0000","last_fullsized":"2024-02-10T08:49:44.212149+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.645301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.645301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":816,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211931+0000","last_change":"2024-02-10T08:48:56.119888+0000","last_active":"2024-02-10T08:49:44.211931+0000","last_peered":"2024-02-10T08:49:44.211931+0000","last_clean":"2024-02-10T08:49:44.211931+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:49:44.211931+0000","last_undegraded":"2024-02-10T08:49:44.211931+0000","last_fullsized":"2024-02-10T08:49:44.211931+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:56.119799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.119799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":194,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588910+0000","last_change":"2024-02-10T08:49:25.635459+0000","last_active":"2024-02-10T08:49:45.588910+0000","last_peered":"2024-02-10T08:49:45.588910+0000","last_clean":"2024-02-10T08:49:45.588910+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:49:45.588910+0000","last_undegraded":"2024-02-10T08:49:45.588910+0000","last_fullsized":"2024-02-10T08:49:45.588910+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.635394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.635394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":740,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.227075+0000","last_change":"2024-02-10T08:48:55.016819+0000","last_active":"2024-02-10T08:49:45.227075+0000","last_peered":"2024-02-10T08:49:45.227075+0000","last_clean":"2024-02-10T08:49:45.227075+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:49:45.227075+0000","last_undegraded":"2024-02-10T08:49:45.227075+0000","last_fullsized":"2024-02-10T08:49:45.227075+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.016738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.016738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":965,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306177+0000","last_change":"2024-02-10T08:48:38.016707+0000","last_active":"2024-02-10T08:49:45.306177+0000","last_peered":"2024-02-10T08:49:45.306177+0000","last_clean":"2024-02-10T08:49:45.306177+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:49:45.306177+0000","last_undegraded":"2024-02-10T08:49:45.306177+0000","last_fullsized":"2024-02-10T08:49:45.306177+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:48:38.016615+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.016615+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":808,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212695+0000","last_change":"2024-02-10T08:49:42.108649+0000","last_active":"2024-02-10T08:49:44.212695+0000","last_peered":"2024-02-10T08:49:44.212695+0000","last_clean":"2024-02-10T08:49:44.212695+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:49:44.212695+0000","last_undegraded":"2024-02-10T08:49:44.212695+0000","last_fullsized":"2024-02-10T08:49:44.212695+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.108568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.108568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":842,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211677+0000","last_change":"2024-02-10T08:49:40.106083+0000","last_active":"2024-02-10T08:49:44.211677+0000","last_peered":"2024-02-10T08:49:44.211677+0000","last_clean":"2024-02-10T08:49:44.211677+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:49:44.211677+0000","last_undegraded":"2024-02-10T08:49:44.211677+0000","last_fullsized":"2024-02-10T08:49:44.211677+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:40.106007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.106007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":357,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416456+0000","last_change":"2024-02-10T08:48:40.860303+0000","last_active":"2024-02-10T08:49:44.416456+0000","last_peered":"2024-02-10T08:49:44.416456+0000","last_clean":"2024-02-10T08:49:44.416456+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:49:44.416456+0000","last_undegraded":"2024-02-10T08:49:44.416456+0000","last_fullsized":"2024-02-10T08:49:44.416456+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.860228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.860228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":372,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.417193+0000","last_change":"2024-02-10T08:49:30.905873+0000","last_active":"2024-02-10T08:49:44.417193+0000","last_peered":"2024-02-10T08:49:44.417193+0000","last_clean":"2024-02-10T08:49:44.417193+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:49:44.417193+0000","last_undegraded":"2024-02-10T08:49:44.417193+0000","last_fullsized":"2024-02-10T08:49:44.417193+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.905787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.905787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":744,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:55.262567+0000","last_change":"2024-02-10T08:49:55.262567+0000","last_active":"2024-02-10T08:49:55.262567+0000","last_peered":"2024-02-10T08:49:55.262567+0000","last_clean":"2024-02-10T08:49:55.262567+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:49:55.262567+0000","last_undegraded":"2024-02-10T08:49:55.262567+0000","last_fullsized":"2024-02-10T08:49:55.262567+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:49:55.262456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:55.262456+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":105,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590659+0000","last_change":"2024-02-10T08:47:49.226548+0000","last_active":"2024-02-10T08:49:45.590659+0000","last_peered":"2024-02-10T08:49:45.590659+0000","last_clean":"2024-02-10T08:49:45.590659+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:49:45.590659+0000","last_undegraded":"2024-02-10T08:49:45.590659+0000","last_fullsized":"2024-02-10T08:49:45.590659+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:11.590963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:47:11.590963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213728+0000","last_change":"2024-02-10T08:48:32.722180+0000","last_active":"2024-02-10T08:49:44.213728+0000","last_peered":"2024-02-10T08:49:44.213728+0000","last_clean":"2024-02-10T08:49:44.213728+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:49:44.213728+0000","last_undegraded":"2024-02-10T08:49:44.213728+0000","last_fullsized":"2024-02-10T08:49:44.213728+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.722121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.722121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":794,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213103+0000","last_change":"2024-02-10T08:48:47.163067+0000","last_active":"2024-02-10T08:49:44.213103+0000","last_peered":"2024-02-10T08:49:44.213103+0000","last_clean":"2024-02-10T08:49:44.213103+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:49:44.213103+0000","last_undegraded":"2024-02-10T08:49:44.213103+0000","last_fullsized":"2024-02-10T08:49:44.213103+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.163011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.163011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":599,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211544+0000","last_change":"2024-02-10T08:49:25.010060+0000","last_active":"2024-02-10T08:49:44.211544+0000","last_peered":"2024-02-10T08:49:44.211544+0000","last_clean":"2024-02-10T08:49:44.211544+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:49:44.211544+0000","last_undegraded":"2024-02-10T08:49:44.211544+0000","last_fullsized":"2024-02-10T08:49:44.211544+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.009961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.009961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":663,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211857+0000","last_change":"2024-02-10T08:48:56.061862+0000","last_active":"2024-02-10T08:49:44.211857+0000","last_peered":"2024-02-10T08:49:44.211857+0000","last_clean":"2024-02-10T08:49:44.211857+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:49:44.211857+0000","last_undegraded":"2024-02-10T08:49:44.211857+0000","last_fullsized":"2024-02-10T08:49:44.211857+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:48:56.061783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.061783+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213743+0000","last_change":"2024-02-10T08:49:22.842445+0000","last_active":"2024-02-10T08:49:44.213743+0000","last_peered":"2024-02-10T08:49:44.213743+0000","last_clean":"2024-02-10T08:49:44.213743+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:49:44.213743+0000","last_undegraded":"2024-02-10T08:49:44.213743+0000","last_fullsized":"2024-02-10T08:49:44.213743+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.842361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.842361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1068,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679416+0000","last_change":"2024-02-10T08:49:39.103578+0000","last_active":"2024-02-10T08:49:51.679416+0000","last_peered":"2024-02-10T08:49:51.679416+0000","last_clean":"2024-02-10T08:49:51.679416+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:49:51.679416+0000","last_undegraded":"2024-02-10T08:49:51.679416+0000","last_fullsized":"2024-02-10T08:49:51.679416+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:49:39.103510+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.103510+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":623,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211878+0000","last_change":"2024-02-10T08:48:48.962890+0000","last_active":"2024-02-10T08:49:44.211878+0000","last_peered":"2024-02-10T08:49:44.211878+0000","last_clean":"2024-02-10T08:49:44.211878+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:49:44.211878+0000","last_undegraded":"2024-02-10T08:49:44.211878+0000","last_fullsized":"2024-02-10T08:49:44.211878+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.962820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.962820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":197,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590466+0000","last_change":"2024-02-10T08:48:29.433315+0000","last_active":"2024-02-10T08:49:45.590466+0000","last_peered":"2024-02-10T08:49:45.590466+0000","last_clean":"2024-02-10T08:49:45.590466+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:49:45.590466+0000","last_undegraded":"2024-02-10T08:49:45.590466+0000","last_fullsized":"2024-02-10T08:49:45.590466+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:29.433223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.433223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":128,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212518+0000","last_change":"2024-02-10T08:48:51.755042+0000","last_active":"2024-02-10T08:49:44.212518+0000","last_peered":"2024-02-10T08:49:44.212518+0000","last_clean":"2024-02-10T08:49:44.212518+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:49:44.212518+0000","last_undegraded":"2024-02-10T08:49:44.212518+0000","last_fullsized":"2024-02-10T08:49:44.212518+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:48:51.754925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.754925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":116,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213347+0000","last_change":"2024-02-10T08:49:24.873487+0000","last_active":"2024-02-10T08:49:44.213347+0000","last_peered":"2024-02-10T08:49:44.213347+0000","last_clean":"2024-02-10T08:49:44.213347+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:49:44.213347+0000","last_undegraded":"2024-02-10T08:49:44.213347+0000","last_fullsized":"2024-02-10T08:49:44.213347+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:49:24.873416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.873416+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":878,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679743+0000","last_change":"2024-02-10T08:49:37.120919+0000","last_active":"2024-02-10T08:49:51.679743+0000","last_peered":"2024-02-10T08:49:51.679743+0000","last_clean":"2024-02-10T08:49:51.679743+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:49:51.679743+0000","last_undegraded":"2024-02-10T08:49:51.679743+0000","last_fullsized":"2024-02-10T08:49:51.679743+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:49:37.120855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.120855+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":809,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226901+0000","last_change":"2024-02-10T08:49:30.193392+0000","last_active":"2024-02-10T08:49:45.226901+0000","last_peered":"2024-02-10T08:49:45.226901+0000","last_clean":"2024-02-10T08:49:45.226901+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:49:45.226901+0000","last_undegraded":"2024-02-10T08:49:45.226901+0000","last_fullsized":"2024-02-10T08:49:45.226901+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.193317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.193317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":758,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226250+0000","last_change":"2024-02-10T08:49:38.332305+0000","last_active":"2024-02-10T08:49:45.226250+0000","last_peered":"2024-02-10T08:49:45.226250+0000","last_clean":"2024-02-10T08:49:45.226250+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:49:45.226250+0000","last_undegraded":"2024-02-10T08:49:45.226250+0000","last_fullsized":"2024-02-10T08:49:45.226250+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.332205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.332205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":706,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212185+0000","last_change":"2024-02-10T08:49:20.031309+0000","last_active":"2024-02-10T08:49:44.212185+0000","last_peered":"2024-02-10T08:49:44.212185+0000","last_clean":"2024-02-10T08:49:44.212185+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:49:44.212185+0000","last_undegraded":"2024-02-10T08:49:44.212185+0000","last_fullsized":"2024-02-10T08:49:44.212185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:20.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.031226+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":390,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416336+0000","last_change":"2024-02-10T08:49:32.829681+0000","last_active":"2024-02-10T08:49:44.416336+0000","last_peered":"2024-02-10T08:49:44.416336+0000","last_clean":"2024-02-10T08:49:44.416336+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:49:44.416336+0000","last_undegraded":"2024-02-10T08:49:44.416336+0000","last_fullsized":"2024-02-10T08:49:44.416336+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.829565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.829565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":967,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679716+0000","last_change":"2024-02-10T08:48:51.038110+0000","last_active":"2024-02-10T08:49:51.679716+0000","last_peered":"2024-02-10T08:49:51.679716+0000","last_clean":"2024-02-10T08:49:51.679716+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:49:51.679716+0000","last_undegraded":"2024-02-10T08:49:51.679716+0000","last_fullsized":"2024-02-10T08:49:51.679716+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.038025+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":773,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589284+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:49:45.589284+0000","last_peered":"2024-02-10T08:49:45.589284+0000","last_clean":"2024-02-10T08:49:45.589284+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:49:45.589284+0000","last_undegraded":"2024-02-10T08:49:45.589284+0000","last_fullsized":"2024-02-10T08:49:45.589284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":724,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588983+0000","last_change":"2024-02-10T08:49:21.361148+0000","last_active":"2024-02-10T08:49:45.588983+0000","last_peered":"2024-02-10T08:49:45.588983+0000","last_clean":"2024-02-10T08:49:45.588983+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:49:45.588983+0000","last_undegraded":"2024-02-10T08:49:45.588983+0000","last_fullsized":"2024-02-10T08:49:45.588983+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.361087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.361087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1033,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.650962+0000","last_change":"2024-02-10T08:49:34.310584+0000","last_active":"2024-02-10T08:49:51.650962+0000","last_peered":"2024-02-10T08:49:51.650962+0000","last_clean":"2024-02-10T08:49:51.650962+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:49:51.650962+0000","last_undegraded":"2024-02-10T08:49:51.650962+0000","last_fullsized":"2024-02-10T08:49:51.650962+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'14","last_scrub_stamp":"2024-02-10T08:49:34.310467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.310467+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211400+0000","last_change":"2024-02-10T08:49:42.159790+0000","last_active":"2024-02-10T08:49:44.211400+0000","last_peered":"2024-02-10T08:49:44.211400+0000","last_clean":"2024-02-10T08:49:44.211400+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:49:44.211400+0000","last_undegraded":"2024-02-10T08:49:44.211400+0000","last_fullsized":"2024-02-10T08:49:44.211400+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:49:42.159696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.159696+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":814,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306728+0000","last_change":"2024-02-10T08:48:54.105562+0000","last_active":"2024-02-10T08:49:45.306728+0000","last_peered":"2024-02-10T08:49:45.306728+0000","last_clean":"2024-02-10T08:49:45.306728+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:49:45.306728+0000","last_undegraded":"2024-02-10T08:49:45.306728+0000","last_fullsized":"2024-02-10T08:49:45.306728+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.105490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.105490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":364,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.417017+0000","last_change":"2024-02-10T08:49:26.795580+0000","last_active":"2024-02-10T08:49:44.417017+0000","last_peered":"2024-02-10T08:49:44.417017+0000","last_clean":"2024-02-10T08:49:44.417017+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:49:44.417017+0000","last_undegraded":"2024-02-10T08:49:44.417017+0000","last_fullsized":"2024-02-10T08:49:44.417017+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.795492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.795492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":212,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589569+0000","last_change":"2024-02-10T08:48:05.524065+0000","last_active":"2024-02-10T08:49:45.589569+0000","last_peered":"2024-02-10T08:49:45.589569+0000","last_clean":"2024-02-10T08:49:45.589569+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:49:45.589569+0000","last_undegraded":"2024-02-10T08:49:45.589569+0000","last_fullsized":"2024-02-10T08:49:45.589569+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:05.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211244+0000","last_change":"2024-02-10T08:48:45.969499+0000","last_active":"2024-02-10T08:49:44.211244+0000","last_peered":"2024-02-10T08:49:44.211244+0000","last_clean":"2024-02-10T08:49:44.211244+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:49:44.211244+0000","last_undegraded":"2024-02-10T08:49:44.211244+0000","last_fullsized":"2024-02-10T08:49:44.211244+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.969416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.969416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":966,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588724+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:49:45.588724+0000","last_peered":"2024-02-10T08:49:45.588724+0000","last_clean":"2024-02-10T08:49:45.588724+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:49:45.588724+0000","last_undegraded":"2024-02-10T08:49:45.588724+0000","last_fullsized":"2024-02-10T08:49:45.588724+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":116,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213005+0000","last_change":"2024-02-10T08:48:36.653104+0000","last_active":"2024-02-10T08:49:44.213005+0000","last_peered":"2024-02-10T08:49:44.213005+0000","last_clean":"2024-02-10T08:49:44.213005+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:49:44.213005+0000","last_undegraded":"2024-02-10T08:49:44.213005+0000","last_fullsized":"2024-02-10T08:49:44.213005+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:48:36.652981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.652981+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":610,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211275+0000","last_change":"2024-02-10T08:48:42.984939+0000","last_active":"2024-02-10T08:49:44.211275+0000","last_peered":"2024-02-10T08:49:44.211275+0000","last_clean":"2024-02-10T08:49:44.211275+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:49:44.211275+0000","last_undegraded":"2024-02-10T08:49:44.211275+0000","last_fullsized":"2024-02-10T08:49:44.211275+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.984822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.984822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":212,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588367+0000","last_change":"2024-02-10T08:48:37.427622+0000","last_active":"2024-02-10T08:49:45.588367+0000","last_peered":"2024-02-10T08:49:45.588367+0000","last_clean":"2024-02-10T08:49:45.588367+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:49:45.588367+0000","last_undegraded":"2024-02-10T08:49:45.588367+0000","last_fullsized":"2024-02-10T08:49:45.588367+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:37.427518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:37.427518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":801,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306237+0000","last_change":"2024-02-10T08:48:43.081221+0000","last_active":"2024-02-10T08:49:45.306237+0000","last_peered":"2024-02-10T08:49:45.306237+0000","last_clean":"2024-02-10T08:49:45.306237+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:49:45.306237+0000","last_undegraded":"2024-02-10T08:49:45.306237+0000","last_fullsized":"2024-02-10T08:49:45.306237+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.081148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.081148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":207,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589383+0000","last_change":"2024-02-10T08:47:57.560707+0000","last_active":"2024-02-10T08:49:45.589383+0000","last_peered":"2024-02-10T08:49:45.589383+0000","last_clean":"2024-02-10T08:49:45.589383+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:49:45.589383+0000","last_undegraded":"2024-02-10T08:49:45.589383+0000","last_fullsized":"2024-02-10T08:49:45.589383+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:47:57.560587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.560587+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":387,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416985+0000","last_change":"2024-02-10T08:49:23.762672+0000","last_active":"2024-02-10T08:49:44.416985+0000","last_peered":"2024-02-10T08:49:44.416985+0000","last_clean":"2024-02-10T08:49:44.416985+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:49:44.416985+0000","last_undegraded":"2024-02-10T08:49:44.416985+0000","last_fullsized":"2024-02-10T08:49:44.416985+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.762550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.762550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":687,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211830+0000","last_change":"2024-02-10T08:49:26.109262+0000","last_active":"2024-02-10T08:49:44.211830+0000","last_peered":"2024-02-10T08:49:44.211830+0000","last_clean":"2024-02-10T08:49:44.211830+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:49:44.211830+0000","last_undegraded":"2024-02-10T08:49:44.211830+0000","last_fullsized":"2024-02-10T08:49:44.211830+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.109194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.109194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":774,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.138309+0000","last_change":"2024-02-10T08:49:56.138309+0000","last_active":"2024-02-10T08:49:56.138309+0000","last_peered":"2024-02-10T08:49:56.138309+0000","last_clean":"2024-02-10T08:49:56.138309+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:49:56.138309+0000","last_undegraded":"2024-02-10T08:49:56.138309+0000","last_fullsized":"2024-02-10T08:49:56.138309+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:56.138238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:56.138238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":835,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.227041+0000","last_change":"2024-02-10T08:49:26.144964+0000","last_active":"2024-02-10T08:49:45.227041+0000","last_peered":"2024-02-10T08:49:45.227041+0000","last_clean":"2024-02-10T08:49:45.227041+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:49:45.227041+0000","last_undegraded":"2024-02-10T08:49:45.227041+0000","last_fullsized":"2024-02-10T08:49:45.227041+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.144887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.144887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":254,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588548+0000","last_change":"2024-02-10T08:49:29.655187+0000","last_active":"2024-02-10T08:49:45.588548+0000","last_peered":"2024-02-10T08:49:45.588548+0000","last_clean":"2024-02-10T08:49:45.588548+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:49:45.588548+0000","last_undegraded":"2024-02-10T08:49:45.588548+0000","last_fullsized":"2024-02-10T08:49:45.588548+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:49:29.655086+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.655086+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":389,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416279+0000","last_change":"2024-02-10T08:49:22.774270+0000","last_active":"2024-02-10T08:49:44.416279+0000","last_peered":"2024-02-10T08:49:44.416279+0000","last_clean":"2024-02-10T08:49:44.416279+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:49:44.416279+0000","last_undegraded":"2024-02-10T08:49:44.416279+0000","last_fullsized":"2024-02-10T08:49:44.416279+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.774198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.774198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":205,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588774+0000","last_change":"2024-02-10T08:48:36.452244+0000","last_active":"2024-02-10T08:49:45.588774+0000","last_peered":"2024-02-10T08:49:45.588774+0000","last_clean":"2024-02-10T08:49:45.588774+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:49:45.588774+0000","last_undegraded":"2024-02-10T08:49:45.588774+0000","last_fullsized":"2024-02-10T08:49:45.588774+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.452178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.452178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":117,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213854+0000","last_change":"2024-02-10T08:48:05.945279+0000","last_active":"2024-02-10T08:49:44.213854+0000","last_peered":"2024-02-10T08:49:44.213854+0000","last_clean":"2024-02-10T08:49:44.213854+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:49:44.213854+0000","last_undegraded":"2024-02-10T08:49:44.213854+0000","last_fullsized":"2024-02-10T08:49:44.213854+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.945162+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":748,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306037+0000","last_change":"2024-02-10T08:49:23.855127+0000","last_active":"2024-02-10T08:49:45.306037+0000","last_peered":"2024-02-10T08:49:45.306037+0000","last_clean":"2024-02-10T08:49:45.306037+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:49:45.306037+0000","last_undegraded":"2024-02-10T08:49:45.306037+0000","last_fullsized":"2024-02-10T08:49:45.306037+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.855041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.855041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":364,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416906+0000","last_change":"2024-02-10T08:48:54.704302+0000","last_active":"2024-02-10T08:49:44.416906+0000","last_peered":"2024-02-10T08:49:44.416906+0000","last_clean":"2024-02-10T08:49:44.416906+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:49:44.416906+0000","last_undegraded":"2024-02-10T08:49:44.416906+0000","last_fullsized":"2024-02-10T08:49:44.416906+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.704183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.704183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":880,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306694+0000","last_change":"2024-02-10T08:48:31.839331+0000","last_active":"2024-02-10T08:49:45.306694+0000","last_peered":"2024-02-10T08:49:45.306694+0000","last_clean":"2024-02-10T08:49:45.306694+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:49:45.306694+0000","last_undegraded":"2024-02-10T08:49:45.306694+0000","last_fullsized":"2024-02-10T08:49:45.306694+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:48:31.839259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.839259+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":373,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.894871+0000","last_change":"2024-02-10T08:49:51.894871+0000","last_active":"2024-02-10T08:49:51.894871+0000","last_peered":"2024-02-10T08:49:51.894871+0000","last_clean":"2024-02-10T08:49:51.894871+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:49:51.894871+0000","last_undegraded":"2024-02-10T08:49:51.894871+0000","last_fullsized":"2024-02-10T08:49:51.894871+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:51.894802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:49:51.894802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":797,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226387+0000","last_change":"2024-02-10T08:49:23.180885+0000","last_active":"2024-02-10T08:49:45.226387+0000","last_peered":"2024-02-10T08:49:45.226387+0000","last_clean":"2024-02-10T08:49:45.226387+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:49:45.226387+0000","last_undegraded":"2024-02-10T08:49:45.226387+0000","last_fullsized":"2024-02-10T08:49:45.226387+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.180813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.180813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":775,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589306+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:49:45.589306+0000","last_peered":"2024-02-10T08:49:45.589306+0000","last_clean":"2024-02-10T08:49:45.589306+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:49:45.589306+0000","last_undegraded":"2024-02-10T08:49:45.589306+0000","last_fullsized":"2024-02-10T08:49:45.589306+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":783,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.227028+0000","last_change":"2024-02-10T08:48:48.079333+0000","last_active":"2024-02-10T08:49:45.227028+0000","last_peered":"2024-02-10T08:49:45.227028+0000","last_clean":"2024-02-10T08:49:45.227028+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:49:45.227028+0000","last_undegraded":"2024-02-10T08:49:45.227028+0000","last_fullsized":"2024-02-10T08:49:45.227028+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.079269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.079269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":757,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226056+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:49:45.226056+0000","last_peered":"2024-02-10T08:49:45.226056+0000","last_clean":"2024-02-10T08:49:45.226056+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:49:45.226056+0000","last_undegraded":"2024-02-10T08:49:45.226056+0000","last_fullsized":"2024-02-10T08:49:45.226056+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":822,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212735+0000","last_change":"2024-02-10T08:49:31.019799+0000","last_active":"2024-02-10T08:49:44.212735+0000","last_peered":"2024-02-10T08:49:44.212735+0000","last_clean":"2024-02-10T08:49:44.212735+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:49:44.212735+0000","last_undegraded":"2024-02-10T08:49:44.212735+0000","last_fullsized":"2024-02-10T08:49:44.212735+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.019739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.019739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":124,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:52.188378+0000","last_change":"2024-02-10T08:49:52.188378+0000","last_active":"2024-02-10T08:49:52.188378+0000","last_peered":"2024-02-10T08:49:52.188378+0000","last_clean":"2024-02-10T08:49:52.188378+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:49:52.188378+0000","last_undegraded":"2024-02-10T08:49:52.188378+0000","last_fullsized":"2024-02-10T08:49:52.188378+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:49:52.188304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:52.188304+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":810,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211767+0000","last_change":"2024-02-10T08:49:36.077156+0000","last_active":"2024-02-10T08:49:44.211767+0000","last_peered":"2024-02-10T08:49:44.211767+0000","last_clean":"2024-02-10T08:49:44.211767+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:49:44.211767+0000","last_undegraded":"2024-02-10T08:49:44.211767+0000","last_fullsized":"2024-02-10T08:49:44.211767+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:36.077066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:36.077066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":243,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:54.449833+0000","last_change":"2024-02-10T08:49:40.123788+0000","last_active":"2024-02-10T08:49:54.449833+0000","last_peered":"2024-02-10T08:49:54.449833+0000","last_clean":"2024-02-10T08:49:54.449833+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:49:54.449833+0000","last_undegraded":"2024-02-10T08:49:54.449833+0000","last_fullsized":"2024-02-10T08:49:54.449833+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:49:40.123719+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.123719+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":224,"num_read_kb":219,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":356,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416925+0000","last_change":"2024-02-10T08:48:47.787415+0000","last_active":"2024-02-10T08:49:44.416925+0000","last_peered":"2024-02-10T08:49:44.416925+0000","last_clean":"2024-02-10T08:49:44.416925+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:49:44.416925+0000","last_undegraded":"2024-02-10T08:49:44.416925+0000","last_fullsized":"2024-02-10T08:49:44.416925+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.787351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211843+0000","last_change":"2024-02-10T08:49:39.088323+0000","last_active":"2024-02-10T08:49:44.211843+0000","last_peered":"2024-02-10T08:49:44.211843+0000","last_clean":"2024-02-10T08:49:44.211843+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:49:44.211843+0000","last_undegraded":"2024-02-10T08:49:44.211843+0000","last_fullsized":"2024-02-10T08:49:44.211843+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.088230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.088230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":223,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589726+0000","last_change":"2024-02-10T08:47:59.541068+0000","last_active":"2024-02-10T08:49:45.589726+0000","last_peered":"2024-02-10T08:49:45.589726+0000","last_clean":"2024-02-10T08:49:45.589726+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:49:45.589726+0000","last_undegraded":"2024-02-10T08:49:45.589726+0000","last_fullsized":"2024-02-10T08:49:45.589726+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:59.541002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:47:59.541002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"711'43","reported_seq":159,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653343+0000","last_change":"2024-02-10T08:48:28.753355+0000","last_active":"2024-02-10T08:49:51.653343+0000","last_peered":"2024-02-10T08:49:51.653343+0000","last_clean":"2024-02-10T08:49:51.653343+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:49:51.653343+0000","last_undegraded":"2024-02-10T08:49:51.653343+0000","last_fullsized":"2024-02-10T08:49:51.653343+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'43","last_scrub_stamp":"2024-02-10T08:48:28.753288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.753288+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":797,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226281+0000","last_change":"2024-02-10T08:48:31.134699+0000","last_active":"2024-02-10T08:49:45.226281+0000","last_peered":"2024-02-10T08:49:45.226281+0000","last_clean":"2024-02-10T08:49:45.226281+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:49:45.226281+0000","last_undegraded":"2024-02-10T08:49:45.226281+0000","last_fullsized":"2024-02-10T08:49:45.226281+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.134575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.134575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":802,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590024+0000","last_change":"2024-02-10T08:49:24.384243+0000","last_active":"2024-02-10T08:49:45.590024+0000","last_peered":"2024-02-10T08:49:45.590024+0000","last_clean":"2024-02-10T08:49:45.590024+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:49:45.590024+0000","last_undegraded":"2024-02-10T08:49:45.590024+0000","last_fullsized":"2024-02-10T08:49:45.590024+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.384180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.384180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":769,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212105+0000","last_change":"2024-02-10T08:48:54.135964+0000","last_active":"2024-02-10T08:49:44.212105+0000","last_peered":"2024-02-10T08:49:44.212105+0000","last_clean":"2024-02-10T08:49:44.212105+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:49:44.212105+0000","last_undegraded":"2024-02-10T08:49:44.212105+0000","last_fullsized":"2024-02-10T08:49:44.212105+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.135900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.135900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":242,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679415+0000","last_change":"2024-02-10T08:49:21.583412+0000","last_active":"2024-02-10T08:49:51.679415+0000","last_peered":"2024-02-10T08:49:51.679415+0000","last_clean":"2024-02-10T08:49:51.679415+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:49:51.679415+0000","last_undegraded":"2024-02-10T08:49:51.679415+0000","last_fullsized":"2024-02-10T08:49:51.679415+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:49:21.583343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.583343+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":124,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213900+0000","last_change":"2024-02-10T08:48:42.684268+0000","last_active":"2024-02-10T08:49:44.213900+0000","last_peered":"2024-02-10T08:49:44.213900+0000","last_clean":"2024-02-10T08:49:44.213900+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:49:44.213900+0000","last_undegraded":"2024-02-10T08:49:44.213900+0000","last_fullsized":"2024-02-10T08:49:44.213900+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:48:42.684181+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.684181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213890+0000","last_change":"2024-02-10T08:49:35.015139+0000","last_active":"2024-02-10T08:49:44.213890+0000","last_peered":"2024-02-10T08:49:44.213890+0000","last_clean":"2024-02-10T08:49:44.213890+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:49:44.213890+0000","last_undegraded":"2024-02-10T08:49:44.213890+0000","last_fullsized":"2024-02-10T08:49:44.213890+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.015073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":194,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590079+0000","last_change":"2024-02-10T08:48:33.412887+0000","last_active":"2024-02-10T08:49:45.590079+0000","last_peered":"2024-02-10T08:49:45.590079+0000","last_clean":"2024-02-10T08:49:45.590079+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:49:45.590079+0000","last_undegraded":"2024-02-10T08:49:45.590079+0000","last_fullsized":"2024-02-10T08:49:45.590079+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.412775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.412775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":742,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589830+0000","last_change":"2024-02-10T08:48:50.100054+0000","last_active":"2024-02-10T08:49:45.589830+0000","last_peered":"2024-02-10T08:49:45.589830+0000","last_clean":"2024-02-10T08:49:45.589830+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:49:45.589830+0000","last_undegraded":"2024-02-10T08:49:45.589830+0000","last_fullsized":"2024-02-10T08:49:45.589830+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.099951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.099951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":840,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212217+0000","last_change":"2024-02-10T08:48:41.070217+0000","last_active":"2024-02-10T08:49:44.212217+0000","last_peered":"2024-02-10T08:49:44.212217+0000","last_clean":"2024-02-10T08:49:44.212217+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:49:44.212217+0000","last_undegraded":"2024-02-10T08:49:44.212217+0000","last_fullsized":"2024-02-10T08:49:44.212217+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.070126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.070126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":712,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211615+0000","last_change":"2024-02-10T08:48:31.973099+0000","last_active":"2024-02-10T08:49:44.211615+0000","last_peered":"2024-02-10T08:49:44.211615+0000","last_clean":"2024-02-10T08:49:44.211615+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:49:44.211615+0000","last_undegraded":"2024-02-10T08:49:44.211615+0000","last_fullsized":"2024-02-10T08:49:44.211615+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:48:31.973020+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.973020+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212121+0000","last_change":"2024-02-10T08:49:30.980749+0000","last_active":"2024-02-10T08:49:44.212121+0000","last_peered":"2024-02-10T08:49:44.212121+0000","last_clean":"2024-02-10T08:49:44.212121+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:49:44.212121+0000","last_undegraded":"2024-02-10T08:49:44.212121+0000","last_fullsized":"2024-02-10T08:49:44.212121+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:49:30.980666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.980666+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":810,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590812+0000","last_change":"2024-02-10T08:48:38.003058+0000","last_active":"2024-02-10T08:49:45.590812+0000","last_peered":"2024-02-10T08:49:45.590812+0000","last_clean":"2024-02-10T08:49:45.590812+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:49:45.590812+0000","last_undegraded":"2024-02-10T08:49:45.590812+0000","last_fullsized":"2024-02-10T08:49:45.590812+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.002996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.002996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":126,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226495+0000","last_change":"2024-02-10T08:48:33.158780+0000","last_active":"2024-02-10T08:49:45.226495+0000","last_peered":"2024-02-10T08:49:45.226495+0000","last_clean":"2024-02-10T08:49:45.226495+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:49:45.226495+0000","last_undegraded":"2024-02-10T08:49:45.226495+0000","last_fullsized":"2024-02-10T08:49:45.226495+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.158671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.158671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":203,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588637+0000","last_change":"2024-02-10T08:48:54.417289+0000","last_active":"2024-02-10T08:49:45.588637+0000","last_peered":"2024-02-10T08:49:45.588637+0000","last_clean":"2024-02-10T08:49:45.588637+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:49:45.588637+0000","last_undegraded":"2024-02-10T08:49:45.588637+0000","last_fullsized":"2024-02-10T08:49:45.588637+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:48:54.417173+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.417173+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":827,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212177+0000","last_change":"2024-02-10T08:48:50.118042+0000","last_active":"2024-02-10T08:49:44.212177+0000","last_peered":"2024-02-10T08:49:44.212177+0000","last_clean":"2024-02-10T08:49:44.212177+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:49:44.212177+0000","last_undegraded":"2024-02-10T08:49:44.212177+0000","last_fullsized":"2024-02-10T08:49:44.212177+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.117976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":812,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212514+0000","last_change":"2024-02-10T08:49:24.116688+0000","last_active":"2024-02-10T08:49:44.212514+0000","last_peered":"2024-02-10T08:49:44.212514+0000","last_clean":"2024-02-10T08:49:44.212514+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:49:44.212514+0000","last_undegraded":"2024-02-10T08:49:44.212514+0000","last_fullsized":"2024-02-10T08:49:44.212514+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.116601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.116601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":605,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211589+0000","last_change":"2024-02-10T08:48:41.943374+0000","last_active":"2024-02-10T08:49:44.211589+0000","last_peered":"2024-02-10T08:49:44.211589+0000","last_clean":"2024-02-10T08:49:44.211589+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:49:44.211589+0000","last_undegraded":"2024-02-10T08:49:44.211589+0000","last_fullsized":"2024-02-10T08:49:44.211589+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.943287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.943287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":864,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590390+0000","last_change":"2024-02-10T08:49:43.363252+0000","last_active":"2024-02-10T08:49:45.590390+0000","last_peered":"2024-02-10T08:49:45.590390+0000","last_clean":"2024-02-10T08:49:45.590390+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:49:45.590390+0000","last_undegraded":"2024-02-10T08:49:45.590390+0000","last_fullsized":"2024-02-10T08:49:45.590390+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:49:43.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:43.363195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":810,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306033+0000","last_change":"2024-02-10T08:48:36.966625+0000","last_active":"2024-02-10T08:49:45.306033+0000","last_peered":"2024-02-10T08:49:45.306033+0000","last_clean":"2024-02-10T08:49:45.306033+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:49:45.306033+0000","last_undegraded":"2024-02-10T08:49:45.306033+0000","last_fullsized":"2024-02-10T08:49:45.306033+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.966559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.966559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":372,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416811+0000","last_change":"2024-02-10T08:49:38.810361+0000","last_active":"2024-02-10T08:49:44.416811+0000","last_peered":"2024-02-10T08:49:44.416811+0000","last_clean":"2024-02-10T08:49:44.416811+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:49:44.416811+0000","last_undegraded":"2024-02-10T08:49:44.416811+0000","last_fullsized":"2024-02-10T08:49:44.416811+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.810280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.810280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":734,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226297+0000","last_change":"2024-02-10T08:48:47.081659+0000","last_active":"2024-02-10T08:49:45.226297+0000","last_peered":"2024-02-10T08:49:45.226297+0000","last_clean":"2024-02-10T08:49:45.226297+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:49:45.226297+0000","last_undegraded":"2024-02-10T08:49:45.226297+0000","last_fullsized":"2024-02-10T08:49:45.226297+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.081542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":892,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589368+0000","last_change":"2024-02-10T08:49:20.324246+0000","last_active":"2024-02-10T08:49:45.589368+0000","last_peered":"2024-02-10T08:49:45.589368+0000","last_clean":"2024-02-10T08:49:45.589368+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:49:45.589368+0000","last_undegraded":"2024-02-10T08:49:45.589368+0000","last_fullsized":"2024-02-10T08:49:45.589368+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:20.324128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.324128+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":653,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589118+0000","last_change":"2024-02-10T08:48:45.089054+0000","last_active":"2024-02-10T08:49:45.589118+0000","last_peered":"2024-02-10T08:49:45.589118+0000","last_clean":"2024-02-10T08:49:45.589118+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:49:45.589118+0000","last_undegraded":"2024-02-10T08:49:45.589118+0000","last_fullsized":"2024-02-10T08:49:45.589118+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.088965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.088965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1035,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679730+0000","last_change":"2024-02-10T08:49:35.294710+0000","last_active":"2024-02-10T08:49:51.679730+0000","last_peered":"2024-02-10T08:49:51.679730+0000","last_clean":"2024-02-10T08:49:51.679730+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:49:51.679730+0000","last_undegraded":"2024-02-10T08:49:51.679730+0000","last_fullsized":"2024-02-10T08:49:51.679730+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13","last_scrub_stamp":"2024-02-10T08:49:35.294638+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.294638+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":194,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588591+0000","last_change":"2024-02-10T08:48:38.471156+0000","last_active":"2024-02-10T08:49:45.588591+0000","last_peered":"2024-02-10T08:49:45.588591+0000","last_clean":"2024-02-10T08:49:45.588591+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:49:45.588591+0000","last_undegraded":"2024-02-10T08:49:45.588591+0000","last_fullsized":"2024-02-10T08:49:45.588591+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.471058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.471058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":194,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590544+0000","last_change":"2024-02-10T08:48:30.433484+0000","last_active":"2024-02-10T08:49:45.590544+0000","last_peered":"2024-02-10T08:49:45.590544+0000","last_clean":"2024-02-10T08:49:45.590544+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:49:45.590544+0000","last_undegraded":"2024-02-10T08:49:45.590544+0000","last_fullsized":"2024-02-10T08:49:45.590544+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.433421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.433421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"798'19","reported_seq":844,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226689+0000","last_change":"2024-02-10T08:49:37.293132+0000","last_active":"2024-02-10T08:49:45.226689+0000","last_peered":"2024-02-10T08:49:45.226689+0000","last_clean":"2024-02-10T08:49:45.226689+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:49:45.226689+0000","last_undegraded":"2024-02-10T08:49:45.226689+0000","last_fullsized":"2024-02-10T08:49:45.226689+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'19","last_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_deep_scrub":"798'19","last_deep_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.293068+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":550,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211472+0000","last_change":"2024-02-10T08:47:55.031372+0000","last_active":"2024-02-10T08:49:44.211472+0000","last_peered":"2024-02-10T08:49:44.211472+0000","last_clean":"2024-02-10T08:49:44.211472+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:49:44.211472+0000","last_undegraded":"2024-02-10T08:49:44.211472+0000","last_fullsized":"2024-02-10T08:49:44.211472+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.031301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.031301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":1010,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652109+0000","last_change":"2024-02-10T08:49:22.354177+0000","last_active":"2024-02-10T08:49:51.652109+0000","last_peered":"2024-02-10T08:49:51.652109+0000","last_clean":"2024-02-10T08:49:51.652109+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:49:51.652109+0000","last_undegraded":"2024-02-10T08:49:51.652109+0000","last_fullsized":"2024-02-10T08:49:51.652109+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:49:22.354105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.354105+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":612,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212098+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:49:44.212098+0000","last_peered":"2024-02-10T08:49:44.212098+0000","last_clean":"2024-02-10T08:49:44.212098+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:49:44.212098+0000","last_undegraded":"2024-02-10T08:49:44.212098+0000","last_fullsized":"2024-02-10T08:49:44.212098+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":772,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211571+0000","last_change":"2024-02-10T08:49:29.101599+0000","last_active":"2024-02-10T08:49:44.211571+0000","last_peered":"2024-02-10T08:49:44.211571+0000","last_clean":"2024-02-10T08:49:44.211571+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:49:44.211571+0000","last_undegraded":"2024-02-10T08:49:44.211571+0000","last_fullsized":"2024-02-10T08:49:44.211571+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:49:29.101533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.101533+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":122,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213835+0000","last_change":"2024-02-10T08:49:35.988869+0000","last_active":"2024-02-10T08:49:44.213835+0000","last_peered":"2024-02-10T08:49:44.213835+0000","last_clean":"2024-02-10T08:49:44.213835+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:49:44.213835+0000","last_undegraded":"2024-02-10T08:49:44.213835+0000","last_fullsized":"2024-02-10T08:49:44.213835+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:35.988803+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.988803+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":787,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306484+0000","last_change":"2024-02-10T08:48:45.997431+0000","last_active":"2024-02-10T08:49:45.306484+0000","last_peered":"2024-02-10T08:49:45.306484+0000","last_clean":"2024-02-10T08:49:45.306484+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:49:45.306484+0000","last_undegraded":"2024-02-10T08:49:45.306484+0000","last_fullsized":"2024-02-10T08:49:45.306484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.997364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.997364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":812,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590793+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:49:45.590793+0000","last_peered":"2024-02-10T08:49:45.590793+0000","last_clean":"2024-02-10T08:49:45.590793+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:49:45.590793+0000","last_undegraded":"2024-02-10T08:49:45.590793+0000","last_fullsized":"2024-02-10T08:49:45.590793+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":212,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590227+0000","last_change":"2024-02-10T08:49:20.566561+0000","last_active":"2024-02-10T08:49:45.590227+0000","last_peered":"2024-02-10T08:49:45.590227+0000","last_clean":"2024-02-10T08:49:45.590227+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:49:45.590227+0000","last_undegraded":"2024-02-10T08:49:45.590227+0000","last_fullsized":"2024-02-10T08:49:45.590227+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.566497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.566497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":750,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306753+0000","last_change":"2024-02-10T08:49:26.780938+0000","last_active":"2024-02-10T08:49:45.306753+0000","last_peered":"2024-02-10T08:49:45.306753+0000","last_clean":"2024-02-10T08:49:45.306753+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:49:45.306753+0000","last_undegraded":"2024-02-10T08:49:45.306753+0000","last_fullsized":"2024-02-10T08:49:45.306753+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.780857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.780857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":708,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:53.268988+0000","last_change":"2024-02-10T08:49:53.268988+0000","last_active":"2024-02-10T08:49:53.268988+0000","last_peered":"2024-02-10T08:49:53.268988+0000","last_clean":"2024-02-10T08:49:53.268988+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:49:53.268988+0000","last_undegraded":"2024-02-10T08:49:53.268988+0000","last_fullsized":"2024-02-10T08:49:53.268988+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:53.268898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:53.268898+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212235+0000","last_change":"2024-02-10T08:47:56.921391+0000","last_active":"2024-02-10T08:49:44.212235+0000","last_peered":"2024-02-10T08:49:44.212235+0000","last_clean":"2024-02-10T08:49:44.212235+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:49:44.212235+0000","last_undegraded":"2024-02-10T08:49:44.212235+0000","last_fullsized":"2024-02-10T08:49:44.212235+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:56.921291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:41:56.647993+0000","last_clean_scrub_stamp":"2024-02-10T08:47:56.921291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213912+0000","last_change":"2024-02-10T08:47:58.934691+0000","last_active":"2024-02-10T08:49:44.213912+0000","last_peered":"2024-02-10T08:49:44.213912+0000","last_clean":"2024-02-10T08:49:44.213912+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:49:44.213912+0000","last_undegraded":"2024-02-10T08:49:44.213912+0000","last_fullsized":"2024-02-10T08:49:44.213912+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:58.934594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.934594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":792,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588755+0000","last_change":"2024-02-10T08:48:40.049047+0000","last_active":"2024-02-10T08:49:45.588755+0000","last_peered":"2024-02-10T08:49:45.588755+0000","last_clean":"2024-02-10T08:49:45.588755+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:49:45.588755+0000","last_undegraded":"2024-02-10T08:49:45.588755+0000","last_fullsized":"2024-02-10T08:49:45.588755+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.048948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.048948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":748,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590215+0000","last_change":"2024-02-10T08:49:31.379159+0000","last_active":"2024-02-10T08:49:45.590215+0000","last_peered":"2024-02-10T08:49:45.590215+0000","last_clean":"2024-02-10T08:49:45.590215+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:49:45.590215+0000","last_undegraded":"2024-02-10T08:49:45.590215+0000","last_fullsized":"2024-02-10T08:49:45.590215+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.379093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.379093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":820,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588580+0000","last_change":"2024-02-10T08:49:33.304282+0000","last_active":"2024-02-10T08:49:45.588580+0000","last_peered":"2024-02-10T08:49:45.588580+0000","last_clean":"2024-02-10T08:49:45.588580+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:49:45.588580+0000","last_undegraded":"2024-02-10T08:49:45.588580+0000","last_fullsized":"2024-02-10T08:49:45.588580+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:49:33.304177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:33.304177+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":783,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226827+0000","last_change":"2024-02-10T08:48:36.153670+0000","last_active":"2024-02-10T08:49:45.226827+0000","last_peered":"2024-02-10T08:49:45.226827+0000","last_clean":"2024-02-10T08:49:45.226827+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:49:45.226827+0000","last_undegraded":"2024-02-10T08:49:45.226827+0000","last_fullsized":"2024-02-10T08:49:45.226827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.153585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.153585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":808,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306037+0000","last_change":"2024-02-10T08:48:32.816285+0000","last_active":"2024-02-10T08:49:45.306037+0000","last_peered":"2024-02-10T08:49:45.306037+0000","last_clean":"2024-02-10T08:49:45.306037+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:49:45.306037+0000","last_undegraded":"2024-02-10T08:49:45.306037+0000","last_fullsized":"2024-02-10T08:49:45.306037+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.816211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.816211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":372,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416651+0000","last_change":"2024-02-10T08:49:29.891433+0000","last_active":"2024-02-10T08:49:44.416651+0000","last_peered":"2024-02-10T08:49:44.416651+0000","last_clean":"2024-02-10T08:49:44.416651+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:49:44.416651+0000","last_undegraded":"2024-02-10T08:49:44.416651+0000","last_fullsized":"2024-02-10T08:49:44.416651+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.891353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.891353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":760,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589121+0000","last_change":"2024-02-10T08:49:42.399412+0000","last_active":"2024-02-10T08:49:45.589121+0000","last_peered":"2024-02-10T08:49:45.589121+0000","last_clean":"2024-02-10T08:49:45.589121+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:49:45.589121+0000","last_undegraded":"2024-02-10T08:49:45.589121+0000","last_fullsized":"2024-02-10T08:49:45.589121+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.399332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.399332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":846,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589042+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:49:45.589042+0000","last_peered":"2024-02-10T08:49:45.589042+0000","last_clean":"2024-02-10T08:49:45.589042+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:49:45.589042+0000","last_undegraded":"2024-02-10T08:49:45.589042+0000","last_fullsized":"2024-02-10T08:49:45.589042+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":787,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306606+0000","last_change":"2024-02-10T08:48:53.059434+0000","last_active":"2024-02-10T08:49:45.306606+0000","last_peered":"2024-02-10T08:49:45.306606+0000","last_clean":"2024-02-10T08:49:45.306606+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:49:45.306606+0000","last_undegraded":"2024-02-10T08:49:45.306606+0000","last_fullsized":"2024-02-10T08:49:45.306606+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:53.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:53.059137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":367,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416440+0000","last_change":"2024-02-10T08:48:45.871861+0000","last_active":"2024-02-10T08:49:44.416440+0000","last_peered":"2024-02-10T08:49:44.416440+0000","last_clean":"2024-02-10T08:49:44.416440+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:49:44.416440+0000","last_undegraded":"2024-02-10T08:49:44.416440+0000","last_fullsized":"2024-02-10T08:49:44.416440+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.871773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":792,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590868+0000","last_change":"2024-02-10T08:49:23.340760+0000","last_active":"2024-02-10T08:49:45.590868+0000","last_peered":"2024-02-10T08:49:45.590868+0000","last_clean":"2024-02-10T08:49:45.590868+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:49:45.590868+0000","last_undegraded":"2024-02-10T08:49:45.590868+0000","last_fullsized":"2024-02-10T08:49:45.590868+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.340696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.340696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":194,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589610+0000","last_change":"2024-02-10T08:49:42.706500+0000","last_active":"2024-02-10T08:49:45.589610+0000","last_peered":"2024-02-10T08:49:45.589610+0000","last_clean":"2024-02-10T08:49:45.589610+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:49:45.589610+0000","last_undegraded":"2024-02-10T08:49:45.589610+0000","last_fullsized":"2024-02-10T08:49:45.589610+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.706431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.706431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":206,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:54.609853+0000","last_change":"2024-02-10T08:49:54.609853+0000","last_active":"2024-02-10T08:49:54.609853+0000","last_peered":"2024-02-10T08:49:54.609853+0000","last_clean":"2024-02-10T08:49:54.609853+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:49:54.609853+0000","last_undegraded":"2024-02-10T08:49:54.609853+0000","last_fullsized":"2024-02-10T08:49:54.609853+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:54.609749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:54.609749+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":821,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226017+0000","last_change":"2024-02-10T08:48:54.005915+0000","last_active":"2024-02-10T08:49:45.226017+0000","last_peered":"2024-02-10T08:49:45.226017+0000","last_clean":"2024-02-10T08:49:45.226017+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:49:45.226017+0000","last_undegraded":"2024-02-10T08:49:45.226017+0000","last_fullsized":"2024-02-10T08:49:45.226017+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.005830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.005830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":367,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416843+0000","last_change":"2024-02-10T08:48:50.762747+0000","last_active":"2024-02-10T08:49:44.416843+0000","last_peered":"2024-02-10T08:49:44.416843+0000","last_clean":"2024-02-10T08:49:44.416843+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:49:44.416843+0000","last_undegraded":"2024-02-10T08:49:44.416843+0000","last_fullsized":"2024-02-10T08:49:44.416843+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.762672+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":769,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211889+0000","last_change":"2024-02-10T08:49:28.108791+0000","last_active":"2024-02-10T08:49:44.211889+0000","last_peered":"2024-02-10T08:49:44.211889+0000","last_clean":"2024-02-10T08:49:44.211889+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:49:44.211889+0000","last_undegraded":"2024-02-10T08:49:44.211889+0000","last_fullsized":"2024-02-10T08:49:44.211889+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.108727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.108727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":863,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306766+0000","last_change":"2024-02-10T08:49:29.815400+0000","last_active":"2024-02-10T08:49:45.306766+0000","last_peered":"2024-02-10T08:49:45.306766+0000","last_clean":"2024-02-10T08:49:45.306766+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:49:45.306766+0000","last_undegraded":"2024-02-10T08:49:45.306766+0000","last_fullsized":"2024-02-10T08:49:45.306766+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:29.815324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.815324+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":798,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588881+0000","last_change":"2024-02-10T08:48:41.013610+0000","last_active":"2024-02-10T08:49:45.588881+0000","last_peered":"2024-02-10T08:49:45.588881+0000","last_clean":"2024-02-10T08:49:45.588881+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:49:45.588881+0000","last_undegraded":"2024-02-10T08:49:45.588881+0000","last_fullsized":"2024-02-10T08:49:45.588881+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.013543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.013543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212192+0000","last_change":"2024-02-10T08:49:21.886466+0000","last_active":"2024-02-10T08:49:44.212192+0000","last_peered":"2024-02-10T08:49:44.212192+0000","last_clean":"2024-02-10T08:49:44.212192+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:49:44.212192+0000","last_undegraded":"2024-02-10T08:49:44.212192+0000","last_fullsized":"2024-02-10T08:49:44.212192+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.886401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.886401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":112,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211884+0000","last_change":"2024-02-10T08:48:49.706321+0000","last_active":"2024-02-10T08:49:44.211884+0000","last_peered":"2024-02-10T08:49:44.211884+0000","last_clean":"2024-02-10T08:49:44.211884+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:49:44.211884+0000","last_undegraded":"2024-02-10T08:49:44.211884+0000","last_fullsized":"2024-02-10T08:49:44.211884+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:49.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:48:49.706258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":105,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211514+0000","last_change":"2024-02-10T08:48:52.097227+0000","last_active":"2024-02-10T08:49:44.211514+0000","last_peered":"2024-02-10T08:49:44.211514+0000","last_clean":"2024-02-10T08:49:44.211514+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:49:44.211514+0000","last_undegraded":"2024-02-10T08:49:44.211514+0000","last_fullsized":"2024-02-10T08:49:44.211514+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.097161+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":810,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306281+0000","last_change":"2024-02-10T08:48:41.102099+0000","last_active":"2024-02-10T08:49:45.306281+0000","last_peered":"2024-02-10T08:49:45.306281+0000","last_clean":"2024-02-10T08:49:45.306281+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:49:45.306281+0000","last_undegraded":"2024-02-10T08:49:45.306281+0000","last_fullsized":"2024-02-10T08:49:45.306281+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.102021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.102021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":618,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212108+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:49:44.212108+0000","last_peered":"2024-02-10T08:49:44.212108+0000","last_clean":"2024-02-10T08:49:44.212108+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:49:44.212108+0000","last_undegraded":"2024-02-10T08:49:44.212108+0000","last_fullsized":"2024-02-10T08:49:44.212108+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":212,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590307+0000","last_change":"2024-02-10T08:49:30.665713+0000","last_active":"2024-02-10T08:49:45.590307+0000","last_peered":"2024-02-10T08:49:45.590307+0000","last_clean":"2024-02-10T08:49:45.590307+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:49:45.590307+0000","last_undegraded":"2024-02-10T08:49:45.590307+0000","last_fullsized":"2024-02-10T08:49:45.590307+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":219,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589828+0000","last_change":"2024-02-10T08:49:37.297309+0000","last_active":"2024-02-10T08:49:45.589828+0000","last_peered":"2024-02-10T08:49:45.589828+0000","last_clean":"2024-02-10T08:49:45.589828+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:49:45.589828+0000","last_undegraded":"2024-02-10T08:49:45.589828+0000","last_fullsized":"2024-02-10T08:49:45.589828+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:49:37.297246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.297246+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":798,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590744+0000","last_change":"2024-02-10T08:48:00.922194+0000","last_active":"2024-02-10T08:49:45.590744+0000","last_peered":"2024-02-10T08:49:45.590744+0000","last_clean":"2024-02-10T08:49:45.590744+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:49:45.590744+0000","last_undegraded":"2024-02-10T08:49:45.590744+0000","last_fullsized":"2024-02-10T08:49:45.590744+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:00.922104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:00.922104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":774,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590523+0000","last_change":"2024-02-10T08:48:43.032173+0000","last_active":"2024-02-10T08:49:45.590523+0000","last_peered":"2024-02-10T08:49:45.590523+0000","last_clean":"2024-02-10T08:49:45.590523+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:49:45.590523+0000","last_undegraded":"2024-02-10T08:49:45.590523+0000","last_fullsized":"2024-02-10T08:49:45.590523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.032094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.032094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":121,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.213371+0000","last_change":"2024-02-10T08:49:25.890744+0000","last_active":"2024-02-10T08:49:44.213371+0000","last_peered":"2024-02-10T08:49:44.213371+0000","last_clean":"2024-02-10T08:49:44.213371+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:49:44.213371+0000","last_undegraded":"2024-02-10T08:49:44.213371+0000","last_fullsized":"2024-02-10T08:49:44.213371+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.890678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.890678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":675,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211473+0000","last_change":"2024-02-10T08:48:32.990056+0000","last_active":"2024-02-10T08:49:44.211473+0000","last_peered":"2024-02-10T08:49:44.211473+0000","last_clean":"2024-02-10T08:49:44.211473+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:49:44.211473+0000","last_undegraded":"2024-02-10T08:49:44.211473+0000","last_fullsized":"2024-02-10T08:49:44.211473+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.989985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.989985+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":364,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.416808+0000","last_change":"2024-02-10T08:48:55.711815+0000","last_active":"2024-02-10T08:49:44.416808+0000","last_peered":"2024-02-10T08:49:44.416808+0000","last_clean":"2024-02-10T08:49:44.416808+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:49:44.416808+0000","last_undegraded":"2024-02-10T08:49:44.416808+0000","last_fullsized":"2024-02-10T08:49:44.416808+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.711733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.711733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":212,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590393+0000","last_change":"2024-02-10T08:49:34.640456+0000","last_active":"2024-02-10T08:49:45.590393+0000","last_peered":"2024-02-10T08:49:45.590393+0000","last_clean":"2024-02-10T08:49:45.590393+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:49:45.590393+0000","last_undegraded":"2024-02-10T08:49:45.590393+0000","last_fullsized":"2024-02-10T08:49:45.590393+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:34.640388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.640388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":763,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212479+0000","last_change":"2024-02-10T08:49:20.063342+0000","last_active":"2024-02-10T08:49:44.212479+0000","last_peered":"2024-02-10T08:49:44.212479+0000","last_clean":"2024-02-10T08:49:44.212479+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:49:44.212479+0000","last_undegraded":"2024-02-10T08:49:44.212479+0000","last_fullsized":"2024-02-10T08:49:44.212479+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.063276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.063276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":678,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211715+0000","last_change":"2024-02-10T08:48:44.939276+0000","last_active":"2024-02-10T08:49:44.211715+0000","last_peered":"2024-02-10T08:49:44.211715+0000","last_clean":"2024-02-10T08:49:44.211715+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:49:44.211715+0000","last_undegraded":"2024-02-10T08:49:44.211715+0000","last_fullsized":"2024-02-10T08:49:44.211715+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:48:44.939188+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.939188+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":310,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589909+0000","last_change":"2024-02-10T08:47:57.938158+0000","last_active":"2024-02-10T08:49:45.589909+0000","last_peered":"2024-02-10T08:49:45.589909+0000","last_clean":"2024-02-10T08:49:45.589909+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:49:45.589909+0000","last_undegraded":"2024-02-10T08:49:45.589909+0000","last_fullsized":"2024-02-10T08:49:45.589909+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.938096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.938096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":791,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226124+0000","last_change":"2024-02-10T08:49:39.311249+0000","last_active":"2024-02-10T08:49:45.226124+0000","last_peered":"2024-02-10T08:49:45.226124+0000","last_clean":"2024-02-10T08:49:45.226124+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:49:45.226124+0000","last_undegraded":"2024-02-10T08:49:45.226124+0000","last_fullsized":"2024-02-10T08:49:45.226124+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.311161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.311161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":193,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.588963+0000","last_change":"2024-02-10T08:48:52.471904+0000","last_active":"2024-02-10T08:49:45.588963+0000","last_peered":"2024-02-10T08:49:45.588963+0000","last_clean":"2024-02-10T08:49:45.588963+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:49:45.588963+0000","last_undegraded":"2024-02-10T08:49:45.588963+0000","last_fullsized":"2024-02-10T08:49:45.588963+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:52.471831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.471831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":715,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211260+0000","last_change":"2024-02-10T08:49:38.160265+0000","last_active":"2024-02-10T08:49:44.211260+0000","last_peered":"2024-02-10T08:49:44.211260+0000","last_clean":"2024-02-10T08:49:44.211260+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:49:44.211260+0000","last_undegraded":"2024-02-10T08:49:44.211260+0000","last_fullsized":"2024-02-10T08:49:44.211260+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:49:38.160189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.160189+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":798,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589980+0000","last_change":"2024-02-10T08:49:29.384350+0000","last_active":"2024-02-10T08:49:45.589980+0000","last_peered":"2024-02-10T08:49:45.589980+0000","last_clean":"2024-02-10T08:49:45.589980+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:49:45.589980+0000","last_undegraded":"2024-02-10T08:49:45.589980+0000","last_fullsized":"2024-02-10T08:49:45.589980+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.384172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.384172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":633,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211395+0000","last_change":"2024-02-10T08:48:47.974449+0000","last_active":"2024-02-10T08:49:44.211395+0000","last_peered":"2024-02-10T08:49:44.211395+0000","last_clean":"2024-02-10T08:49:44.211395+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:49:44.211395+0000","last_undegraded":"2024-02-10T08:49:44.211395+0000","last_fullsized":"2024-02-10T08:49:44.211395+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.974390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.974390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":752,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.226346+0000","last_change":"2024-02-10T08:48:42.043077+0000","last_active":"2024-02-10T08:49:45.226346+0000","last_peered":"2024-02-10T08:49:45.226346+0000","last_clean":"2024-02-10T08:49:45.226346+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:49:45.226346+0000","last_undegraded":"2024-02-10T08:49:45.226346+0000","last_fullsized":"2024-02-10T08:49:45.226346+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.043006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.043006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":866,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589178+0000","last_change":"2024-02-10T08:47:13.690038+0000","last_active":"2024-02-10T08:49:45.589178+0000","last_peered":"2024-02-10T08:49:45.589178+0000","last_clean":"2024-02-10T08:49:45.589178+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:49:45.589178+0000","last_undegraded":"2024-02-10T08:49:45.589178+0000","last_fullsized":"2024-02-10T08:49:45.589178+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:47:13.689972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:13.689972+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":205,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.589089+0000","last_change":"2024-02-10T08:48:02.600767+0000","last_active":"2024-02-10T08:49:45.589089+0000","last_peered":"2024-02-10T08:49:45.589089+0000","last_clean":"2024-02-10T08:49:45.589089+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:49:45.589089+0000","last_undegraded":"2024-02-10T08:49:45.589089+0000","last_fullsized":"2024-02-10T08:49:45.589089+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.600704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.600704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":789,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.590669+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:49:45.590669+0000","last_peered":"2024-02-10T08:49:45.590669+0000","last_clean":"2024-02-10T08:49:45.590669+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:49:45.590669+0000","last_undegraded":"2024-02-10T08:49:45.590669+0000","last_fullsized":"2024-02-10T08:49:45.590669+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":767,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:45.306413+0000","last_change":"2024-02-10T08:49:32.715296+0000","last_active":"2024-02-10T08:49:45.306413+0000","last_peered":"2024-02-10T08:49:45.306413+0000","last_clean":"2024-02-10T08:49:45.306413+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:49:45.306413+0000","last_undegraded":"2024-02-10T08:49:45.306413+0000","last_fullsized":"2024-02-10T08:49:45.306413+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.715176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.715176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":10732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":224,"num_read_kb":219,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299594,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316156,"kb_used_data":6272,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428996,"statfs":{"total":95995035648,"available":95671291904,"internally_reserved":0,"allocated":6422528,"data_stored":3361401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56100000000000005,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.751,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.377},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.59499999999999997,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70699999999999996,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.67400000000000004,"15min":0.60999999999999999},"min":{"1min":0.30399999999999999,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.80100000000000005,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.67700000000000005,"15min":0.61399999999999999},"min":{"1min":0.32100000000000001,"5min":0.28399999999999997,"15min":0.19900000000000001},"max":{"1min":0.64000000000000001,"5min":10.978,"15min":10.978},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67600000000000005,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.66000000000000003,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63600000000000001,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":7.5099999999999998,"15min":19.369},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57099999999999995,"15min":0.59299999999999997},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68899999999999995,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.72199999999999998,"15min":0.70099999999999996},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.67400000000000004,"5min":7.8700000000000001,"15min":22.102},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70399999999999996,"5min":1.581,"15min":1.581},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.66200000000000003,"15min":0.70899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.50900000000000001,"15min":0.54600000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65800000000000003,"5min":4.46,"15min":19.326000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60799999999999998,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.23999999999999999,"15min":0.23899999999999999},"max":{"1min":0.76400000000000001,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.74399999999999999,"5min":4.718,"15min":22.687000000000001},"last":0.436},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.39700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70299999999999996,"5min":5.484,"15min":5.484},"last":0.39900000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561232,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":6260,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":6410240,"data_stored":3361629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.55600000000000005},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61499999999999999,"5min":3.9399999999999999,"15min":8.5250000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.751,"15min":0.63300000000000001},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.48099999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.73399999999999999,"15min":0.65300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.66300000000000003,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":0.69699999999999995,"5min":8.4399999999999995,"15min":18.402999999999999},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76800000000000002,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0509999999999999,"5min":10.772,"15min":10.772},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.84099999999999997,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.499}]},{"osd":3,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.79500000000000004,"15min":0.70499999999999996},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.64300000000000002,"5min":9.2650000000000006,"15min":16.823},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.443},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48899999999999999,"15min":0.47399999999999998},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.44700000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.70999999999999996,"5min":3.915,"15min":3.915},"last":0.46500000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":7.5650000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308012,"kb_used_data":6320,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437140,"statfs":{"total":95995035648,"available":95679631360,"internally_reserved":0,"allocated":6471680,"data_stored":3363304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.5669999999999999,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.221},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.39500000000000002,"15min":0.46100000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.71799999999999997,"5min":1.4910000000000001,"15min":16.166},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.47699999999999998,"15min":0.65100000000000002},"min":{"1min":0.246,"5min":0.193,"15min":0.193},"max":{"1min":1.637,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.44500000000000001,"15min":0.55600000000000005},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.93100000000000005,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.249}]},{"osd":3,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.47799999999999998,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":0.51100000000000001,"5min":4.8940000000000001,"15min":24.773},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.50900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.80000000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.307}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81799999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.317},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.441,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73299999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.44700000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76100000000000001,"5min":3.496,"15min":9.5600000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77700000000000002,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.622},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.82299999999999995,"5min":3.4710000000000001,"15min":9.532},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.874,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.73299999999999998,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":12.147,"15min":12.147},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78400000000000003,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.505}]}]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307588,"kb_used_data":6268,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437564,"statfs":{"total":95995035648,"available":95680065536,"internally_reserved":0,"allocated":6418432,"data_stored":3360740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45700000000000002,"15min":0.40500000000000003},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.156},"max":{"1min":1.4239999999999999,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.74199999999999999,"15min":0.5},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":3.8839999999999999,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.54100000000000004,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.501,"5min":11.987,"15min":11.987},"last":0.317},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.33700000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.502,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.51900000000000002,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.38300000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.435,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75800000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.435},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.46300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79600000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.42699999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.65600000000000003,"15min":0.59399999999999997},"min":{"1min":0.38700000000000001,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.3149999999999999,"5min":4.609,"15min":4.609},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.32300000000000001},"max":{"1min":3.2320000000000002,"5min":3.2320000000000002,"15min":3.4079999999999999},"last":0.47299999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.5},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.33300000000000002},"max":{"1min":3.9609999999999999,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.41899999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.48399999999999999,"5min":0.32500000000000001,"15min":0.317},"max":{"1min":0.94499999999999995,"5min":1.849,"15min":3.363},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.66000000000000003},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":8.4299999999999997},"last":0.45800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411555,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303852,"kb_used_data":6256,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441300,"statfs":{"total":95995035648,"available":95683891200,"internally_reserved":0,"allocated":6406144,"data_stored":3360889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.54800000000000004,"15min":0.502},"min":{"1min":0.214,"5min":0.183,"15min":0.183},"max":{"1min":1.2909999999999999,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.218},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.52800000000000002,"15min":0.56200000000000006},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":0.42299999999999999,"5min":10.567,"15min":14.858000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.51200000000000001,"15min":0.55700000000000005},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.63500000000000001,"5min":6.9720000000000004,"15min":12.512},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.56399999999999995,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.52200000000000002,"5min":12.875,"15min":12.875},"last":0.252}]},{"osd":3,"last update":"Sat Feb 10 08:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.61199999999999999,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50800000000000001,"5min":10.994999999999999,"15min":11.612},"last":0.309},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":7.8280000000000003,"15min":11.710000000000001},"last":0.28799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":1.298,"15min":1.298},"last":0.39100000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.41099999999999998,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.79100000000000004,"5min":10.34,"15min":10.34},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.71199999999999997,"15min":0.66500000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3380000000000001,"5min":9.6690000000000005,"15min":9.6690000000000005},"last":0.42399999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.753,"15min":0.71499999999999997},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.79800000000000004,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.71899999999999997,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.76600000000000001,"5min":10.539,"15min":10.539},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.77800000000000002,"15min":0.70899999999999996},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.32500000000000001},"max":{"1min":0.70199999999999996,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.48199999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182623,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303888,"kb_used_data":6292,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441264,"statfs":{"total":95995035648,"available":95683854336,"internally_reserved":0,"allocated":6443008,"data_stored":3361597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.20699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.497,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.251,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.58599999999999997,"5min":10.484999999999999,"15min":11.808},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.76900000000000002,"15min":0.67200000000000004},"min":{"1min":0.248,"5min":0.215,"15min":0.184},"max":{"1min":0.53800000000000003,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.26000000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.58799999999999997,"5min":17.343,"15min":17.343},"last":0.29899999999999999}]},{"osd":2,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.754,"15min":0.68000000000000005},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":0.57299999999999995,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.443,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57199999999999995,"5min":5.0259999999999998,"15min":23.294},"last":0.307}]},{"osd":4,"last update":"Sat Feb 10 08:49:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84199999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57199999999999995,"15min":0.70599999999999996},"min":{"1min":0.42099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.495},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.41299999999999998,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":0.82999999999999996,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.54900000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58999999999999997,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79200000000000004,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.73199999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.61399999999999999,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.81599999999999995,"5min":5.056,"15min":14.773999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59099999999999997,"15min":0.751},"min":{"1min":0.52700000000000002,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.56999999999999995}]}]},{"osd":4,"up_from":633,"seq":2718714298411,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309644,"kb_used_data":6276,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93435508,"statfs":{"total":95995035648,"available":95677960192,"internally_reserved":0,"allocated":6426624,"data_stored":3361767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.66500000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36299999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.65200000000000002,"5min":0.754,"15min":0.754},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66100000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59999999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.315,"15min":0.315},"max":{"1min":0.63700000000000001,"5min":1.036,"15min":1.036},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":0.746,"15min":0.746},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51700000000000002,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68600000000000005,"5min":0.87,"15min":0.87},"last":0.42499999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52800000000000002,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.67500000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69699999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44400000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.32200000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71699999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.5}]}]},{"osd":5,"up_from":37,"seq":158913790165,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312044,"kb_used_data":6256,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433108,"statfs":{"total":95995035648,"available":95675502592,"internally_reserved":0,"allocated":6406144,"data_stored":3360700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58099999999999996,"15min":1.05},"min":{"1min":0.29499999999999998,"5min":0.249,"15min":0.216},"max":{"1min":0.93500000000000005,"5min":7.923,"15min":107.551},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53200000000000003,"15min":0.97499999999999998},"min":{"1min":0.36899999999999999,"5min":0.27800000000000002,"15min":0.255},"max":{"1min":0.85499999999999998,"5min":2.9590000000000001,"15min":88.792000000000002},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":1.0069999999999999},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.216},"max":{"1min":0.82199999999999995,"5min":12.375999999999999,"15min":107.205},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.67700000000000005,"15min":0.88300000000000001},"min":{"1min":0.45100000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.90000000000000002,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.46200000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.66800000000000004,"15min":1.0349999999999999},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":1.4650000000000001,"5min":9.9600000000000009,"15min":54.174999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58299999999999996,"15min":1.073},"min":{"1min":0.40100000000000002,"5min":0.26900000000000002,"15min":0.26600000000000001},"max":{"1min":1.405,"5min":2.335,"15min":86.183000000000007},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66700000000000004,"15min":0.86399999999999999},"min":{"1min":0.41799999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":1.0529999999999999,"5min":9.9900000000000002,"15min":44.395000000000003},"last":0.436},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":1.29},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":0.92200000000000004,"5min":8.0120000000000005,"15min":114.526},"last":0.42999999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.314}]},{"osd":6,"last update":"Sat Feb 10 08:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.73299999999999998},"min":{"1min":0.28999999999999998,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":0.82299999999999995,"5min":10.023999999999999,"15min":72.241},"last":0.373},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81000000000000005,"5min":10.013,"15min":22.469999999999999},"last":0.38200000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.50800000000000001,"15min":0.72799999999999998},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.96699999999999997,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52200000000000002,"15min":0.75900000000000001},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.253},"max":{"1min":0.79200000000000004,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.41599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:50:00.958 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T08:50:00.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T08:50:00.958 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2024-02-10T08:50:00.958 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2024-02-10T08:50:00.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:50:01.054 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:01.229 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:01.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.224296+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:50:01.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.227183+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:50:01.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:00.291634+0000 mgr.y (mgr.14553) 676 : audit [DBG] from='client.54449 v1:172.21.15.107:0/3007238984' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:01.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.985891+0000 osd.6 (osd.6) 334 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:50:01.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:00.987852+0000 osd.6 (osd.6) 335 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:50:01.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.224296+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:50:01.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.227183+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:50:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:00.291634+0000 mgr.y (mgr.14553) 676 : audit [DBG] from='client.54449 v1:172.21.15.107:0/3007238984' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.985891+0000 osd.6 (osd.6) 334 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:50:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:00.987852+0000 osd.6 (osd.6) 335 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:50:02.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.224296+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:50:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.227183+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:50:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:00.291634+0000 mgr.y (mgr.14553) 676 : audit [DBG] from='client.54449 v1:172.21.15.107:0/3007238984' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.985891+0000 osd.6 (osd.6) 334 : cluster [DBG] 4.5 scrub starts 2024-02-10T08:50:02.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:00.987852+0000 osd.6 (osd.6) 335 : cluster [DBG] 4.5 scrub ok 2024-02-10T08:50:02.619 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:02.620 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:02.620 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:50:02.620 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:50:02.620 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:02.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:01.222604+0000 osd.5 (osd.5) 338 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:50:02.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:01.225680+0000 osd.5 (osd.5) 339 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:01.582923+0000 mgr.y (mgr.14553) 677 : cluster [DBG] pgmap v1237: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:01.870086+0000 mgr.y (mgr.14553) 678 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.269175+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.270722+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:01.222604+0000 osd.5 (osd.5) 338 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:01.225680+0000 osd.5 (osd.5) 339 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:01.582923+0000 mgr.y (mgr.14553) 677 : cluster [DBG] pgmap v1237: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:01.870086+0000 mgr.y (mgr.14553) 678 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.269175+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:50:02.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.270722+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:01.222604+0000 osd.5 (osd.5) 338 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:01.225680+0000 osd.5 (osd.5) 339 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:01.582923+0000 mgr.y (mgr.14553) 677 : cluster [DBG] pgmap v1237: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:01.870086+0000 mgr.y (mgr.14553) 678 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.269175+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:50:02.933 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.270722+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:50:03.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.178+0000 7f9f3b12f700 1 -- 172.21.15.107:0/928715171 >> v1:172.21.15.107:6790/0 conn(0x7f9f340fc910 legacy=0x7f9f340fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.178+0000 7f9f3b12f700 1 -- 172.21.15.107:0/928715171 shutdown_connections 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.178+0000 7f9f3b12f700 1 -- 172.21.15.107:0/928715171 >> 172.21.15.107:0/928715171 conn(0x7f9f340f2510 msgr2=0x7f9f340f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.178+0000 7f9f3b12f700 1 -- 172.21.15.107:0/928715171 shutdown_connections 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.178+0000 7f9f3b12f700 1 -- 172.21.15.107:0/928715171 wait complete. 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 librados: starting msgr at 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 librados: starting objecter 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 Processor -- start 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 -- start start 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 librados: setting wanted keys 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 librados: calling monclient init 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f34117350 con 0x7f9f340ff710 2024-02-10T08:50:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f34118530 con 0x7f9f34103250 2024-02-10T08:50:03.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f3b12f700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f34119730 con 0x7f9f340fc910 2024-02-10T08:50:03.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f38ecb700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9f340fc910 0x7f9f34060c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42010/0 (socket says 172.21.15.107:42010) 2024-02-10T08:50:03.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f38ecb700 1 -- 172.21.15.107:0/3124196061 learned_addr learned my addr 172.21.15.107:0/3124196061 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:03.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter ms_handle_connect 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter ms_handle_connect 0x7f9f340ff710 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter ms_handle_connect 0x7f9f34103250 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580352872 0 0) 0x7f9f34119730 con 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f240038e0 con 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.179+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645174289 0 0) 0x7f9f34118530 con 0x7f9f34103250 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f34119730 con 0x7f9f34103250 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845141758 0 0) 0x7f9f34117350 con 0x7f9f340ff710 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f34118530 con 0x7f9f340ff710 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2813189926 0 0) 0x7f9f240038e0 con 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f34117350 con 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9f28002e60 con 0x7f9f340fc910 2024-02-10T08:50:03.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668582124 0 0) 0x7f9f34118530 con 0x7f9f340ff710 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f240038e0 con 0x7f9f340ff710 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740082680 0 0) 0x7f9f34119730 con 0x7f9f34103250 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f34118530 con 0x7f9f34103250 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9f20001840 con 0x7f9f340ff710 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9f2c0019a0 con 0x7f9f34103250 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127578374 0 0) 0x7f9f34117350 con 0x7f9f340fc910 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 >> v1:172.21.15.195:6789/0 conn(0x7f9f34103250 legacy=0x7f9f34115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 >> v1:172.21.15.107:6789/0 conn(0x7f9f340ff710 legacy=0x7f9f34061330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3411a930 con 0x7f9f340fc910 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f34117540 con 0x7f9f340fc910 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9f280024f0 con 0x7f9f340fc910 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9f28003a20 con 0x7f9f340fc910 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f3b12f700 10 client.54455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:03.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f34117a30 con 0x7f9f340fc910 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.180+0000 7f9f3b12f700 1 librados: init done 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.181+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3405f7a0 con 0x7f9f340fc910 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.181+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9f28002660 con 0x7f9f340fc910 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(807..807 src has 754..807) v4 ==== 7386+0+0 (unknown 1989828987 0 0) 0x7f9f28078f70 con 0x7f9f340fc910 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 10 client.54455.objecter ms_dispatch 0x7f9f34001550 osd_map(807..807 src has 754..807) v4 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 3 client.54455.objecter handle_osd_map got epochs [807,807] > 0 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 3 client.54455.objecter handle_osd_map decoding full epoch 807 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 20 client.54455.objecter dump_active .. 0 homeless 2024-02-10T08:50:03.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.182+0000 7f9f31ffb700 10 client.54455.objecter ms_handle_connect 0x7f9f24063d30 2024-02-10T08:50:03.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.186+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9f2804c6c0 con 0x7f9f340fc910 2024-02-10T08:50:03.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T08:50:03.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.346+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f34108f80 con 0x7f9f24063d30 2024-02-10T08:50:03.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.357+0000 7f9f31ffb700 1 -- 172.21.15.107:0/3124196061 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+410100 (unknown 2965378022 0 1027714224) 0x7f9f34108f80 con 0x7f9f24063d30 2024-02-10T08:50:03.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 10 librados: watch_flush enter 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 10 librados: watch_flush exit 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 20 client.54455.objecter shutdown clearing up homeless session... 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 10 client.54455.objecter successfully canceled tick 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9f24063d30 legacy=0x7f9f240661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:03.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 >> v1:172.21.15.107:6790/0 conn(0x7f9f340fc910 legacy=0x7f9f34060c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 shutdown_connections 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 >> 172.21.15.107:0/3124196061 conn(0x7f9f340f2510 msgr2=0x7f9f341066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 shutdown_connections 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 -- 172.21.15.107:0/3124196061 wait complete. 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:03.360+0000 7f9f3b12f700 1 librados: shutdown 2024-02-10T08:50:03.367 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:50:03.515 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.158381+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:50:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.159164+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:50:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.202646+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:50:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.206097+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:50:03.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.209363+0000 osd.4 (osd.4) 87 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:02.212435+0000 osd.4 (osd.4) 88 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:03.100098+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.d scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:03.103109+0000 osd.1 (osd.1) 365 : cluster [DBG] 2.d scrub ok 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.158381+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.159164+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.202646+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.206097+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.209363+0000 osd.4 (osd.4) 87 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:02.212435+0000 osd.4 (osd.4) 88 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:03.100098+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.d scrub starts 2024-02-10T08:50:03.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:03.103109+0000 osd.1 (osd.1) 365 : cluster [DBG] 2.d scrub ok 2024-02-10T08:50:04.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.158381+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.10 deep-scrub starts 2024-02-10T08:50:04.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.159164+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.10 deep-scrub ok 2024-02-10T08:50:04.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.202646+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:50:04.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.206097+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:50:04.029 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.209363+0000 osd.4 (osd.4) 87 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:50:04.030 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:02.212435+0000 osd.4 (osd.4) 88 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:50:04.030 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:03.100098+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.d scrub starts 2024-02-10T08:50:04.030 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:03.103109+0000 osd.1 (osd.1) 365 : cluster [DBG] 2.d scrub ok 2024-02-10T08:50:04.067 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1237,"stamp":"2024-02-10T08:50:01.582408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1357,"num_read_kb":1109,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":13194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":462,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2465452,"kb_used_data":47596,"kb_used_omap":59,"kb_used_meta":2416580,"kb_avail":747495764,"statfs":{"total":767960285184,"available":765435662336,"internally_reserved":0,"allocated":48738304,"data_stored":25128615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2474578539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.934662"},"pg_stats":[{"pgid":"5.9","version":"636'18","reported_seq":508,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:58.959500+0000","last_change":"2024-02-10T08:49:58.959500+0000","last_active":"2024-02-10T08:49:58.959500+0000","last_peered":"2024-02-10T08:49:58.959500+0000","last_clean":"2024-02-10T08:49:58.959500+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:49:58.959500+0000","last_undegraded":"2024-02-10T08:49:58.959500+0000","last_fullsized":"2024-02-10T08:49:58.959500+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'18","last_scrub_stamp":"2024-02-10T08:49:58.959425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:58.959425+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":873,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835519+0000","last_change":"2024-02-10T08:47:58.959348+0000","last_active":"2024-02-10T08:49:51.835519+0000","last_peered":"2024-02-10T08:49:51.835519+0000","last_clean":"2024-02-10T08:49:51.835519+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:49:51.835519+0000","last_undegraded":"2024-02-10T08:49:51.835519+0000","last_fullsized":"2024-02-10T08:49:51.835519+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:47:58.959271+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:47:58.959271+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":811,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730603+0000","last_change":"2024-02-10T08:49:41.119323+0000","last_active":"2024-02-10T08:49:51.730603+0000","last_peered":"2024-02-10T08:49:51.730603+0000","last_clean":"2024-02-10T08:49:51.730603+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:49:51.730603+0000","last_undegraded":"2024-02-10T08:49:51.730603+0000","last_fullsized":"2024-02-10T08:49:51.730603+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:41.119237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:49:41.119237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":792,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835330+0000","last_change":"2024-02-10T08:48:01.033569+0000","last_active":"2024-02-10T08:49:51.835330+0000","last_peered":"2024-02-10T08:49:51.835330+0000","last_clean":"2024-02-10T08:49:51.835330+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:49:51.835330+0000","last_undegraded":"2024-02-10T08:49:51.835330+0000","last_fullsized":"2024-02-10T08:49:51.835330+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:01.033502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:01.033502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":752,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730307+0000","last_change":"2024-02-10T08:49:23.108122+0000","last_active":"2024-02-10T08:49:51.730307+0000","last_peered":"2024-02-10T08:49:51.730307+0000","last_clean":"2024-02-10T08:49:51.730307+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:49:51.730307+0000","last_undegraded":"2024-02-10T08:49:51.730307+0000","last_fullsized":"2024-02-10T08:49:51.730307+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.108061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.108061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":254,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835186+0000","last_change":"2024-02-10T08:48:32.371647+0000","last_active":"2024-02-10T08:49:51.835186+0000","last_peered":"2024-02-10T08:49:51.835186+0000","last_clean":"2024-02-10T08:49:51.835186+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:49:51.835186+0000","last_undegraded":"2024-02-10T08:49:51.835186+0000","last_fullsized":"2024-02-10T08:49:51.835186+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.371572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.371572+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652149+0000","last_change":"2024-02-10T08:48:38.645376+0000","last_active":"2024-02-10T08:49:51.652149+0000","last_peered":"2024-02-10T08:49:51.652149+0000","last_clean":"2024-02-10T08:49:51.652149+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:49:51.652149+0000","last_undegraded":"2024-02-10T08:49:51.652149+0000","last_fullsized":"2024-02-10T08:49:51.652149+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.645301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.645301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":817,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730216+0000","last_change":"2024-02-10T08:48:56.119888+0000","last_active":"2024-02-10T08:49:51.730216+0000","last_peered":"2024-02-10T08:49:51.730216+0000","last_clean":"2024-02-10T08:49:51.730216+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:49:51.730216+0000","last_undegraded":"2024-02-10T08:49:51.730216+0000","last_fullsized":"2024-02-10T08:49:51.730216+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:56.119799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.119799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":195,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835207+0000","last_change":"2024-02-10T08:49:25.635459+0000","last_active":"2024-02-10T08:49:51.835207+0000","last_peered":"2024-02-10T08:49:51.835207+0000","last_clean":"2024-02-10T08:49:51.835207+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:49:51.835207+0000","last_undegraded":"2024-02-10T08:49:51.835207+0000","last_fullsized":"2024-02-10T08:49:51.835207+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.635394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.635394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":741,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.838055+0000","last_change":"2024-02-10T08:48:55.016819+0000","last_active":"2024-02-10T08:49:51.838055+0000","last_peered":"2024-02-10T08:49:51.838055+0000","last_clean":"2024-02-10T08:49:51.838055+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:49:51.838055+0000","last_undegraded":"2024-02-10T08:49:51.838055+0000","last_fullsized":"2024-02-10T08:49:51.838055+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.016738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.016738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":971,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.906585+0000","last_change":"2024-02-10T08:49:56.906557+0000","last_active":"2024-02-10T08:49:56.906585+0000","last_peered":"2024-02-10T08:49:56.906585+0000","last_clean":"2024-02-10T08:49:56.906585+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:49:56.906585+0000","last_undegraded":"2024-02-10T08:49:56.906585+0000","last_fullsized":"2024-02-10T08:49:56.906585+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:49:56.906487+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:46:49.661956+0000","last_clean_scrub_stamp":"2024-02-10T08:49:56.906487+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":809,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730430+0000","last_change":"2024-02-10T08:49:42.108649+0000","last_active":"2024-02-10T08:49:51.730430+0000","last_peered":"2024-02-10T08:49:51.730430+0000","last_clean":"2024-02-10T08:49:51.730430+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:49:51.730430+0000","last_undegraded":"2024-02-10T08:49:51.730430+0000","last_fullsized":"2024-02-10T08:49:51.730430+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.108568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.108568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":843,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730252+0000","last_change":"2024-02-10T08:49:40.106083+0000","last_active":"2024-02-10T08:49:51.730252+0000","last_peered":"2024-02-10T08:49:51.730252+0000","last_clean":"2024-02-10T08:49:51.730252+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:49:51.730252+0000","last_undegraded":"2024-02-10T08:49:51.730252+0000","last_fullsized":"2024-02-10T08:49:51.730252+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:40.106007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.106007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":358,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836144+0000","last_change":"2024-02-10T08:48:40.860303+0000","last_active":"2024-02-10T08:49:51.836144+0000","last_peered":"2024-02-10T08:49:51.836144+0000","last_clean":"2024-02-10T08:49:51.836144+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:49:51.836144+0000","last_undegraded":"2024-02-10T08:49:51.836144+0000","last_fullsized":"2024-02-10T08:49:51.836144+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.860228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.860228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":373,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835810+0000","last_change":"2024-02-10T08:49:30.905873+0000","last_active":"2024-02-10T08:49:51.835810+0000","last_peered":"2024-02-10T08:49:51.835810+0000","last_clean":"2024-02-10T08:49:51.835810+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:49:51.835810+0000","last_undegraded":"2024-02-10T08:49:51.835810+0000","last_fullsized":"2024-02-10T08:49:51.835810+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.905787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.905787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":744,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:55.262567+0000","last_change":"2024-02-10T08:49:55.262567+0000","last_active":"2024-02-10T08:49:55.262567+0000","last_peered":"2024-02-10T08:49:55.262567+0000","last_clean":"2024-02-10T08:49:55.262567+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:49:55.262567+0000","last_undegraded":"2024-02-10T08:49:55.262567+0000","last_fullsized":"2024-02-10T08:49:55.262567+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:49:55.262456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:55.262456+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":111,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.301456+0000","last_change":"2024-02-10T08:49:56.301430+0000","last_active":"2024-02-10T08:49:56.301456+0000","last_peered":"2024-02-10T08:49:56.301456+0000","last_clean":"2024-02-10T08:49:56.301456+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:49:56.301456+0000","last_undegraded":"2024-02-10T08:49:56.301456+0000","last_fullsized":"2024-02-10T08:49:56.301456+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:56.301365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:49:56.301365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653254+0000","last_change":"2024-02-10T08:48:32.722180+0000","last_active":"2024-02-10T08:49:51.653254+0000","last_peered":"2024-02-10T08:49:51.653254+0000","last_clean":"2024-02-10T08:49:51.653254+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:49:51.653254+0000","last_undegraded":"2024-02-10T08:49:51.653254+0000","last_fullsized":"2024-02-10T08:49:51.653254+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.722121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.722121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":795,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730831+0000","last_change":"2024-02-10T08:48:47.163067+0000","last_active":"2024-02-10T08:49:51.730831+0000","last_peered":"2024-02-10T08:49:51.730831+0000","last_clean":"2024-02-10T08:49:51.730831+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:49:51.730831+0000","last_undegraded":"2024-02-10T08:49:51.730831+0000","last_fullsized":"2024-02-10T08:49:51.730831+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.163011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.163011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":599,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211544+0000","last_change":"2024-02-10T08:49:25.010060+0000","last_active":"2024-02-10T08:49:44.211544+0000","last_peered":"2024-02-10T08:49:44.211544+0000","last_clean":"2024-02-10T08:49:44.211544+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:49:44.211544+0000","last_undegraded":"2024-02-10T08:49:44.211544+0000","last_fullsized":"2024-02-10T08:49:44.211544+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.009961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:39:14.683242+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.009961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":663,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211857+0000","last_change":"2024-02-10T08:48:56.061862+0000","last_active":"2024-02-10T08:49:44.211857+0000","last_peered":"2024-02-10T08:49:44.211857+0000","last_clean":"2024-02-10T08:49:44.211857+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:49:44.211857+0000","last_undegraded":"2024-02-10T08:49:44.211857+0000","last_fullsized":"2024-02-10T08:49:44.211857+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:48:56.061783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:56.061783+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652100+0000","last_change":"2024-02-10T08:49:22.842445+0000","last_active":"2024-02-10T08:49:51.652100+0000","last_peered":"2024-02-10T08:49:51.652100+0000","last_clean":"2024-02-10T08:49:51.652100+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:49:51.652100+0000","last_undegraded":"2024-02-10T08:49:51.652100+0000","last_fullsized":"2024-02-10T08:49:51.652100+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.842361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.842361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1070,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680099+0000","last_change":"2024-02-10T08:49:39.103578+0000","last_active":"2024-02-10T08:49:56.680099+0000","last_peered":"2024-02-10T08:49:56.680099+0000","last_clean":"2024-02-10T08:49:56.680099+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:49:56.680099+0000","last_undegraded":"2024-02-10T08:49:56.680099+0000","last_fullsized":"2024-02-10T08:49:56.680099+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:49:39.103510+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:45:29.297292+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.103510+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":623,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211878+0000","last_change":"2024-02-10T08:48:48.962890+0000","last_active":"2024-02-10T08:49:44.211878+0000","last_peered":"2024-02-10T08:49:44.211878+0000","last_clean":"2024-02-10T08:49:44.211878+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:49:44.211878+0000","last_undegraded":"2024-02-10T08:49:44.211878+0000","last_fullsized":"2024-02-10T08:49:44.211878+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.962820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.962820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":198,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835328+0000","last_change":"2024-02-10T08:48:29.433315+0000","last_active":"2024-02-10T08:49:51.835328+0000","last_peered":"2024-02-10T08:49:51.835328+0000","last_clean":"2024-02-10T08:49:51.835328+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:49:51.835328+0000","last_undegraded":"2024-02-10T08:49:51.835328+0000","last_fullsized":"2024-02-10T08:49:51.835328+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:29.433223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:48:29.433223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"473'27","reported_seq":129,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652826+0000","last_change":"2024-02-10T08:48:51.755042+0000","last_active":"2024-02-10T08:49:51.652826+0000","last_peered":"2024-02-10T08:49:51.652826+0000","last_clean":"2024-02-10T08:49:51.652826+0000","last_became_active":"2024-02-10T08:47:49.243403+0000","last_became_peered":"2024-02-10T08:47:49.243403+0000","last_unstale":"2024-02-10T08:49:51.652826+0000","last_undegraded":"2024-02-10T08:49:51.652826+0000","last_fullsized":"2024-02-10T08:49:51.652826+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:48:51.754925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.754925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":117,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652916+0000","last_change":"2024-02-10T08:49:24.873487+0000","last_active":"2024-02-10T08:49:51.652916+0000","last_peered":"2024-02-10T08:49:51.652916+0000","last_clean":"2024-02-10T08:49:51.652916+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:49:51.652916+0000","last_undegraded":"2024-02-10T08:49:51.652916+0000","last_fullsized":"2024-02-10T08:49:51.652916+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:49:24.873416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.873416+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":878,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.679743+0000","last_change":"2024-02-10T08:49:37.120919+0000","last_active":"2024-02-10T08:49:51.679743+0000","last_peered":"2024-02-10T08:49:51.679743+0000","last_clean":"2024-02-10T08:49:51.679743+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:49:51.679743+0000","last_undegraded":"2024-02-10T08:49:51.679743+0000","last_fullsized":"2024-02-10T08:49:51.679743+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:49:37.120855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.120855+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":810,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837730+0000","last_change":"2024-02-10T08:49:30.193392+0000","last_active":"2024-02-10T08:49:51.837730+0000","last_peered":"2024-02-10T08:49:51.837730+0000","last_clean":"2024-02-10T08:49:51.837730+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:49:51.837730+0000","last_undegraded":"2024-02-10T08:49:51.837730+0000","last_fullsized":"2024-02-10T08:49:51.837730+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.193317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.193317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":759,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836675+0000","last_change":"2024-02-10T08:49:38.332305+0000","last_active":"2024-02-10T08:49:51.836675+0000","last_peered":"2024-02-10T08:49:51.836675+0000","last_clean":"2024-02-10T08:49:51.836675+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:49:51.836675+0000","last_undegraded":"2024-02-10T08:49:51.836675+0000","last_fullsized":"2024-02-10T08:49:51.836675+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.332205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.332205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":706,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212185+0000","last_change":"2024-02-10T08:49:20.031309+0000","last_active":"2024-02-10T08:49:44.212185+0000","last_peered":"2024-02-10T08:49:44.212185+0000","last_clean":"2024-02-10T08:49:44.212185+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:49:44.212185+0000","last_undegraded":"2024-02-10T08:49:44.212185+0000","last_fullsized":"2024-02-10T08:49:44.212185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:20.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.031226+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":391,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836055+0000","last_change":"2024-02-10T08:49:32.829681+0000","last_active":"2024-02-10T08:49:51.836055+0000","last_peered":"2024-02-10T08:49:51.836055+0000","last_clean":"2024-02-10T08:49:51.836055+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:49:51.836055+0000","last_undegraded":"2024-02-10T08:49:51.836055+0000","last_fullsized":"2024-02-10T08:49:51.836055+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.829565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.829565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":969,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680450+0000","last_change":"2024-02-10T08:48:51.038110+0000","last_active":"2024-02-10T08:49:56.680450+0000","last_peered":"2024-02-10T08:49:56.680450+0000","last_clean":"2024-02-10T08:49:56.680450+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:49:56.680450+0000","last_undegraded":"2024-02-10T08:49:56.680450+0000","last_fullsized":"2024-02-10T08:49:56.680450+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:48:51.038025+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":774,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651847+0000","last_change":"2024-02-10T08:47:16.815745+0000","last_active":"2024-02-10T08:49:51.651847+0000","last_peered":"2024-02-10T08:49:51.651847+0000","last_clean":"2024-02-10T08:49:51.651847+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:49:51.651847+0000","last_undegraded":"2024-02-10T08:49:51.651847+0000","last_fullsized":"2024-02-10T08:49:51.651847+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:16.815645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:47:16.815645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":725,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651443+0000","last_change":"2024-02-10T08:49:21.361148+0000","last_active":"2024-02-10T08:49:51.651443+0000","last_peered":"2024-02-10T08:49:51.651443+0000","last_clean":"2024-02-10T08:49:51.651443+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:49:51.651443+0000","last_undegraded":"2024-02-10T08:49:51.651443+0000","last_fullsized":"2024-02-10T08:49:51.651443+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.361087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.361087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1035,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680125+0000","last_change":"2024-02-10T08:49:34.310584+0000","last_active":"2024-02-10T08:49:56.680125+0000","last_peered":"2024-02-10T08:49:56.680125+0000","last_clean":"2024-02-10T08:49:56.680125+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:49:56.680125+0000","last_undegraded":"2024-02-10T08:49:56.680125+0000","last_fullsized":"2024-02-10T08:49:56.680125+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'14","last_scrub_stamp":"2024-02-10T08:49:34.310467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.310467+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211400+0000","last_change":"2024-02-10T08:49:42.159790+0000","last_active":"2024-02-10T08:49:44.211400+0000","last_peered":"2024-02-10T08:49:44.211400+0000","last_clean":"2024-02-10T08:49:44.211400+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:49:44.211400+0000","last_undegraded":"2024-02-10T08:49:44.211400+0000","last_fullsized":"2024-02-10T08:49:44.211400+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:49:42.159696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.159696+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":815,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836415+0000","last_change":"2024-02-10T08:48:54.105562+0000","last_active":"2024-02-10T08:49:51.836415+0000","last_peered":"2024-02-10T08:49:51.836415+0000","last_clean":"2024-02-10T08:49:51.836415+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:49:51.836415+0000","last_undegraded":"2024-02-10T08:49:51.836415+0000","last_fullsized":"2024-02-10T08:49:51.836415+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.105490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.105490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":365,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837933+0000","last_change":"2024-02-10T08:49:26.795580+0000","last_active":"2024-02-10T08:49:51.837933+0000","last_peered":"2024-02-10T08:49:51.837933+0000","last_clean":"2024-02-10T08:49:51.837933+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:49:51.837933+0000","last_undegraded":"2024-02-10T08:49:51.837933+0000","last_fullsized":"2024-02-10T08:49:51.837933+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.795492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.795492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":213,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835575+0000","last_change":"2024-02-10T08:48:05.524065+0000","last_active":"2024-02-10T08:49:51.835575+0000","last_peered":"2024-02-10T08:49:51.835575+0000","last_clean":"2024-02-10T08:49:51.835575+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:49:51.835575+0000","last_undegraded":"2024-02-10T08:49:51.835575+0000","last_fullsized":"2024-02-10T08:49:51.835575+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:05.523998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:05.523998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211244+0000","last_change":"2024-02-10T08:48:45.969499+0000","last_active":"2024-02-10T08:49:44.211244+0000","last_peered":"2024-02-10T08:49:44.211244+0000","last_clean":"2024-02-10T08:49:44.211244+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:49:44.211244+0000","last_undegraded":"2024-02-10T08:49:44.211244+0000","last_fullsized":"2024-02-10T08:49:44.211244+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.969416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.969416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":967,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.650936+0000","last_change":"2024-02-10T08:47:26.780396+0000","last_active":"2024-02-10T08:49:51.650936+0000","last_peered":"2024-02-10T08:49:51.650936+0000","last_clean":"2024-02-10T08:49:51.650936+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:49:51.650936+0000","last_undegraded":"2024-02-10T08:49:51.650936+0000","last_fullsized":"2024-02-10T08:49:51.650936+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:47:26.780326+0000","last_deep_scrub":"80'26","last_deep_scrub_stamp":"2024-02-10T08:35:09.492017+0000","last_clean_scrub_stamp":"2024-02-10T08:47:26.780326+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":122,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:50:00.175553+0000","last_change":"2024-02-10T08:50:00.175553+0000","last_active":"2024-02-10T08:50:00.175553+0000","last_peered":"2024-02-10T08:50:00.175553+0000","last_clean":"2024-02-10T08:50:00.175553+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:50:00.175553+0000","last_undegraded":"2024-02-10T08:50:00.175553+0000","last_fullsized":"2024-02-10T08:50:00.175553+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:50:00.175431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:50:00.175431+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":610,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211275+0000","last_change":"2024-02-10T08:48:42.984939+0000","last_active":"2024-02-10T08:49:44.211275+0000","last_peered":"2024-02-10T08:49:44.211275+0000","last_clean":"2024-02-10T08:49:44.211275+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:49:44.211275+0000","last_undegraded":"2024-02-10T08:49:44.211275+0000","last_fullsized":"2024-02-10T08:49:44.211275+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.984822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.984822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":213,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835220+0000","last_change":"2024-02-10T08:48:37.427622+0000","last_active":"2024-02-10T08:49:51.835220+0000","last_peered":"2024-02-10T08:49:51.835220+0000","last_clean":"2024-02-10T08:49:51.835220+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:49:51.835220+0000","last_undegraded":"2024-02-10T08:49:51.835220+0000","last_fullsized":"2024-02-10T08:49:51.835220+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:37.427518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:37.427518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":802,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837026+0000","last_change":"2024-02-10T08:48:43.081221+0000","last_active":"2024-02-10T08:49:51.837026+0000","last_peered":"2024-02-10T08:49:51.837026+0000","last_clean":"2024-02-10T08:49:51.837026+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:49:51.837026+0000","last_undegraded":"2024-02-10T08:49:51.837026+0000","last_fullsized":"2024-02-10T08:49:51.837026+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.081148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.081148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":213,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:59.520645+0000","last_change":"2024-02-10T08:49:59.520645+0000","last_active":"2024-02-10T08:49:59.520645+0000","last_peered":"2024-02-10T08:49:59.520645+0000","last_clean":"2024-02-10T08:49:59.520645+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:49:59.520645+0000","last_undegraded":"2024-02-10T08:49:59.520645+0000","last_fullsized":"2024-02-10T08:49:59.520645+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:49:59.520575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:59.520575+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":388,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837266+0000","last_change":"2024-02-10T08:49:23.762672+0000","last_active":"2024-02-10T08:49:51.837266+0000","last_peered":"2024-02-10T08:49:51.837266+0000","last_clean":"2024-02-10T08:49:51.837266+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:49:51.837266+0000","last_undegraded":"2024-02-10T08:49:51.837266+0000","last_fullsized":"2024-02-10T08:49:51.837266+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.762550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.762550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":688,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730179+0000","last_change":"2024-02-10T08:49:26.109262+0000","last_active":"2024-02-10T08:49:51.730179+0000","last_peered":"2024-02-10T08:49:51.730179+0000","last_clean":"2024-02-10T08:49:51.730179+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:49:51.730179+0000","last_undegraded":"2024-02-10T08:49:51.730179+0000","last_fullsized":"2024-02-10T08:49:51.730179+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.109194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.109194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":775,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.138335+0000","last_change":"2024-02-10T08:49:56.138309+0000","last_active":"2024-02-10T08:49:56.138335+0000","last_peered":"2024-02-10T08:49:56.138335+0000","last_clean":"2024-02-10T08:49:56.138335+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:49:56.138335+0000","last_undegraded":"2024-02-10T08:49:56.138335+0000","last_fullsized":"2024-02-10T08:49:56.138335+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:56.138238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:56.138238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":836,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837975+0000","last_change":"2024-02-10T08:49:26.144964+0000","last_active":"2024-02-10T08:49:51.837975+0000","last_peered":"2024-02-10T08:49:51.837975+0000","last_clean":"2024-02-10T08:49:51.837975+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:49:51.837975+0000","last_undegraded":"2024-02-10T08:49:51.837975+0000","last_fullsized":"2024-02-10T08:49:51.837975+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.144887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.561295+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.144887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":255,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835471+0000","last_change":"2024-02-10T08:49:29.655187+0000","last_active":"2024-02-10T08:49:51.835471+0000","last_peered":"2024-02-10T08:49:51.835471+0000","last_clean":"2024-02-10T08:49:51.835471+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:49:51.835471+0000","last_undegraded":"2024-02-10T08:49:51.835471+0000","last_fullsized":"2024-02-10T08:49:51.835471+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:49:29.655086+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.655086+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":390,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836001+0000","last_change":"2024-02-10T08:49:22.774270+0000","last_active":"2024-02-10T08:49:51.836001+0000","last_peered":"2024-02-10T08:49:51.836001+0000","last_clean":"2024-02-10T08:49:51.836001+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:49:51.836001+0000","last_undegraded":"2024-02-10T08:49:51.836001+0000","last_fullsized":"2024-02-10T08:49:51.836001+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:22.774198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:08.428189+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.774198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":206,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835230+0000","last_change":"2024-02-10T08:48:36.452244+0000","last_active":"2024-02-10T08:49:51.835230+0000","last_peered":"2024-02-10T08:49:51.835230+0000","last_clean":"2024-02-10T08:49:51.835230+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:49:51.835230+0000","last_undegraded":"2024-02-10T08:49:51.835230+0000","last_fullsized":"2024-02-10T08:49:51.835230+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.452178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.452178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":123,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:59.159883+0000","last_change":"2024-02-10T08:49:59.159883+0000","last_active":"2024-02-10T08:49:59.159883+0000","last_peered":"2024-02-10T08:49:59.159883+0000","last_clean":"2024-02-10T08:49:59.159883+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:49:59.159883+0000","last_undegraded":"2024-02-10T08:49:59.159883+0000","last_fullsized":"2024-02-10T08:49:59.159883+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:49:59.159804+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:48:05.945162+0000","last_clean_scrub_stamp":"2024-02-10T08:49:59.159804+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":749,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836726+0000","last_change":"2024-02-10T08:49:23.855127+0000","last_active":"2024-02-10T08:49:51.836726+0000","last_peered":"2024-02-10T08:49:51.836726+0000","last_clean":"2024-02-10T08:49:51.836726+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:49:51.836726+0000","last_undegraded":"2024-02-10T08:49:51.836726+0000","last_fullsized":"2024-02-10T08:49:51.836726+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.855041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.855041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":365,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836816+0000","last_change":"2024-02-10T08:48:54.704302+0000","last_active":"2024-02-10T08:49:51.836816+0000","last_peered":"2024-02-10T08:49:51.836816+0000","last_clean":"2024-02-10T08:49:51.836816+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:49:51.836816+0000","last_undegraded":"2024-02-10T08:49:51.836816+0000","last_fullsized":"2024-02-10T08:49:51.836816+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.704183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.704183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"473'26","reported_seq":886,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:50:00.987976+0000","last_change":"2024-02-10T08:50:00.987976+0000","last_active":"2024-02-10T08:50:00.987976+0000","last_peered":"2024-02-10T08:50:00.987976+0000","last_clean":"2024-02-10T08:50:00.987976+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:50:00.987976+0000","last_undegraded":"2024-02-10T08:50:00.987976+0000","last_fullsized":"2024-02-10T08:50:00.987976+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'26","last_scrub_stamp":"2024-02-10T08:50:00.987861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:00.987861+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":374,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.894893+0000","last_change":"2024-02-10T08:49:51.894871+0000","last_active":"2024-02-10T08:49:51.894893+0000","last_peered":"2024-02-10T08:49:51.894893+0000","last_clean":"2024-02-10T08:49:51.894893+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:49:51.894893+0000","last_undegraded":"2024-02-10T08:49:51.894893+0000","last_fullsized":"2024-02-10T08:49:51.894893+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:51.894802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:49:51.894802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":798,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836825+0000","last_change":"2024-02-10T08:49:23.180885+0000","last_active":"2024-02-10T08:49:51.836825+0000","last_peered":"2024-02-10T08:49:51.836825+0000","last_clean":"2024-02-10T08:49:51.836825+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:49:51.836825+0000","last_undegraded":"2024-02-10T08:49:51.836825+0000","last_fullsized":"2024-02-10T08:49:51.836825+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.180813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.180813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":776,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651659+0000","last_change":"2024-02-10T08:47:12.675300+0000","last_active":"2024-02-10T08:49:51.651659+0000","last_peered":"2024-02-10T08:49:51.651659+0000","last_clean":"2024-02-10T08:49:51.651659+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:49:51.651659+0000","last_undegraded":"2024-02-10T08:49:51.651659+0000","last_fullsized":"2024-02-10T08:49:51.651659+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:12.675236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:47:12.675236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":784,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.838016+0000","last_change":"2024-02-10T08:48:48.079333+0000","last_active":"2024-02-10T08:49:51.838016+0000","last_peered":"2024-02-10T08:49:51.838016+0000","last_clean":"2024-02-10T08:49:51.838016+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:49:51.838016+0000","last_undegraded":"2024-02-10T08:49:51.838016+0000","last_fullsized":"2024-02-10T08:49:51.838016+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:48.079269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:48.079269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":758,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835164+0000","last_change":"2024-02-10T08:47:54.969106+0000","last_active":"2024-02-10T08:49:51.835164+0000","last_peered":"2024-02-10T08:49:51.835164+0000","last_clean":"2024-02-10T08:49:51.835164+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:49:51.835164+0000","last_undegraded":"2024-02-10T08:49:51.835164+0000","last_fullsized":"2024-02-10T08:49:51.835164+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:54.969037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:47:54.969037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":823,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730728+0000","last_change":"2024-02-10T08:49:31.019799+0000","last_active":"2024-02-10T08:49:51.730728+0000","last_peered":"2024-02-10T08:49:51.730728+0000","last_clean":"2024-02-10T08:49:51.730728+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:49:51.730728+0000","last_undegraded":"2024-02-10T08:49:51.730728+0000","last_fullsized":"2024-02-10T08:49:51.730728+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.019739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.019739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":125,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:52.188418+0000","last_change":"2024-02-10T08:49:52.188378+0000","last_active":"2024-02-10T08:49:52.188418+0000","last_peered":"2024-02-10T08:49:52.188418+0000","last_clean":"2024-02-10T08:49:52.188418+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:49:52.188418+0000","last_undegraded":"2024-02-10T08:49:52.188418+0000","last_fullsized":"2024-02-10T08:49:52.188418+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:49:52.188304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:52.188304+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":811,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730103+0000","last_change":"2024-02-10T08:49:36.077156+0000","last_active":"2024-02-10T08:49:51.730103+0000","last_peered":"2024-02-10T08:49:51.730103+0000","last_clean":"2024-02-10T08:49:51.730103+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:49:51.730103+0000","last_undegraded":"2024-02-10T08:49:51.730103+0000","last_fullsized":"2024-02-10T08:49:51.730103+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:36.077066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:36.077066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":249,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:59.456841+0000","last_change":"2024-02-10T08:49:40.123788+0000","last_active":"2024-02-10T08:49:59.456841+0000","last_peered":"2024-02-10T08:49:59.456841+0000","last_clean":"2024-02-10T08:49:59.456841+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:49:59.456841+0000","last_undegraded":"2024-02-10T08:49:59.456841+0000","last_fullsized":"2024-02-10T08:49:59.456841+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:49:40.123719+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:49:40.123719+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":357,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837192+0000","last_change":"2024-02-10T08:48:47.787415+0000","last_active":"2024-02-10T08:49:51.837192+0000","last_peered":"2024-02-10T08:49:51.837192+0000","last_clean":"2024-02-10T08:49:51.837192+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:49:51.837192+0000","last_undegraded":"2024-02-10T08:49:51.837192+0000","last_fullsized":"2024-02-10T08:49:51.837192+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.787351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651599+0000","last_change":"2024-02-10T08:49:39.088323+0000","last_active":"2024-02-10T08:49:51.651599+0000","last_peered":"2024-02-10T08:49:51.651599+0000","last_clean":"2024-02-10T08:49:51.651599+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:49:51.651599+0000","last_undegraded":"2024-02-10T08:49:51.651599+0000","last_fullsized":"2024-02-10T08:49:51.651599+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.088230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.088230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":224,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835622+0000","last_change":"2024-02-10T08:47:59.541068+0000","last_active":"2024-02-10T08:49:51.835622+0000","last_peered":"2024-02-10T08:49:51.835622+0000","last_clean":"2024-02-10T08:49:51.835622+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:49:51.835622+0000","last_undegraded":"2024-02-10T08:49:51.835622+0000","last_fullsized":"2024-02-10T08:49:51.835622+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:59.541002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:41.054857+0000","last_clean_scrub_stamp":"2024-02-10T08:47:59.541002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"711'43","reported_seq":161,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680110+0000","last_change":"2024-02-10T08:48:28.753355+0000","last_active":"2024-02-10T08:49:56.680110+0000","last_peered":"2024-02-10T08:49:56.680110+0000","last_clean":"2024-02-10T08:49:56.680110+0000","last_became_active":"2024-02-10T08:47:49.228803+0000","last_became_peered":"2024-02-10T08:47:49.228803+0000","last_unstale":"2024-02-10T08:49:56.680110+0000","last_undegraded":"2024-02-10T08:49:56.680110+0000","last_fullsized":"2024-02-10T08:49:56.680110+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'43","last_scrub_stamp":"2024-02-10T08:48:28.753288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:28.753288+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":798,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835388+0000","last_change":"2024-02-10T08:48:31.134699+0000","last_active":"2024-02-10T08:49:51.835388+0000","last_peered":"2024-02-10T08:49:51.835388+0000","last_clean":"2024-02-10T08:49:51.835388+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:49:51.835388+0000","last_undegraded":"2024-02-10T08:49:51.835388+0000","last_fullsized":"2024-02-10T08:49:51.835388+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:31.134575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.134575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":803,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653186+0000","last_change":"2024-02-10T08:49:24.384243+0000","last_active":"2024-02-10T08:49:51.653186+0000","last_peered":"2024-02-10T08:49:51.653186+0000","last_clean":"2024-02-10T08:49:51.653186+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:49:51.653186+0000","last_undegraded":"2024-02-10T08:49:51.653186+0000","last_fullsized":"2024-02-10T08:49:51.653186+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.384180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.384180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":770,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.729912+0000","last_change":"2024-02-10T08:48:54.135964+0000","last_active":"2024-02-10T08:49:51.729912+0000","last_peered":"2024-02-10T08:49:51.729912+0000","last_clean":"2024-02-10T08:49:51.729912+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:49:51.729912+0000","last_undegraded":"2024-02-10T08:49:51.729912+0000","last_fullsized":"2024-02-10T08:49:51.729912+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.135900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.135900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":244,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680107+0000","last_change":"2024-02-10T08:49:21.583412+0000","last_active":"2024-02-10T08:49:56.680107+0000","last_peered":"2024-02-10T08:49:56.680107+0000","last_clean":"2024-02-10T08:49:56.680107+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:49:56.680107+0000","last_undegraded":"2024-02-10T08:49:56.680107+0000","last_fullsized":"2024-02-10T08:49:56.680107+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:49:21.583343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.583343+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":125,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652022+0000","last_change":"2024-02-10T08:48:42.684268+0000","last_active":"2024-02-10T08:49:51.652022+0000","last_peered":"2024-02-10T08:49:51.652022+0000","last_clean":"2024-02-10T08:49:51.652022+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:49:51.652022+0000","last_undegraded":"2024-02-10T08:49:51.652022+0000","last_fullsized":"2024-02-10T08:49:51.652022+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:48:42.684181+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.684181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653519+0000","last_change":"2024-02-10T08:49:35.015139+0000","last_active":"2024-02-10T08:49:51.653519+0000","last_peered":"2024-02-10T08:49:51.653519+0000","last_clean":"2024-02-10T08:49:51.653519+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:49:51.653519+0000","last_undegraded":"2024-02-10T08:49:51.653519+0000","last_fullsized":"2024-02-10T08:49:51.653519+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.015073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":195,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835160+0000","last_change":"2024-02-10T08:48:33.412887+0000","last_active":"2024-02-10T08:49:51.835160+0000","last_peered":"2024-02-10T08:49:51.835160+0000","last_clean":"2024-02-10T08:49:51.835160+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:49:51.835160+0000","last_undegraded":"2024-02-10T08:49:51.835160+0000","last_fullsized":"2024-02-10T08:49:51.835160+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.412775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.412775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":743,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651091+0000","last_change":"2024-02-10T08:48:50.100054+0000","last_active":"2024-02-10T08:49:51.651091+0000","last_peered":"2024-02-10T08:49:51.651091+0000","last_clean":"2024-02-10T08:49:51.651091+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:49:51.651091+0000","last_undegraded":"2024-02-10T08:49:51.651091+0000","last_fullsized":"2024-02-10T08:49:51.651091+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.099951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.099951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":841,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730030+0000","last_change":"2024-02-10T08:48:41.070217+0000","last_active":"2024-02-10T08:49:51.730030+0000","last_peered":"2024-02-10T08:49:51.730030+0000","last_clean":"2024-02-10T08:49:51.730030+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:49:51.730030+0000","last_undegraded":"2024-02-10T08:49:51.730030+0000","last_fullsized":"2024-02-10T08:49:51.730030+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.070126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.070126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":712,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211615+0000","last_change":"2024-02-10T08:48:31.973099+0000","last_active":"2024-02-10T08:49:44.211615+0000","last_peered":"2024-02-10T08:49:44.211615+0000","last_clean":"2024-02-10T08:49:44.211615+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:49:44.211615+0000","last_undegraded":"2024-02-10T08:49:44.211615+0000","last_fullsized":"2024-02-10T08:49:44.211615+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:48:31.973020+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:48:31.973020+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":682,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212121+0000","last_change":"2024-02-10T08:49:30.980749+0000","last_active":"2024-02-10T08:49:44.212121+0000","last_peered":"2024-02-10T08:49:44.212121+0000","last_clean":"2024-02-10T08:49:44.212121+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:49:44.212121+0000","last_undegraded":"2024-02-10T08:49:44.212121+0000","last_fullsized":"2024-02-10T08:49:44.212121+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:49:30.980666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.980666+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":811,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651421+0000","last_change":"2024-02-10T08:48:38.003058+0000","last_active":"2024-02-10T08:49:51.651421+0000","last_peered":"2024-02-10T08:49:51.651421+0000","last_clean":"2024-02-10T08:49:51.651421+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:49:51.651421+0000","last_undegraded":"2024-02-10T08:49:51.651421+0000","last_fullsized":"2024-02-10T08:49:51.651421+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.002996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.002996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":127,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837364+0000","last_change":"2024-02-10T08:48:33.158780+0000","last_active":"2024-02-10T08:49:51.837364+0000","last_peered":"2024-02-10T08:49:51.837364+0000","last_clean":"2024-02-10T08:49:51.837364+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:49:51.837364+0000","last_undegraded":"2024-02-10T08:49:51.837364+0000","last_fullsized":"2024-02-10T08:49:51.837364+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:33.158671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:33.158671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":204,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835569+0000","last_change":"2024-02-10T08:48:54.417289+0000","last_active":"2024-02-10T08:49:51.835569+0000","last_peered":"2024-02-10T08:49:51.835569+0000","last_clean":"2024-02-10T08:49:51.835569+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:49:51.835569+0000","last_undegraded":"2024-02-10T08:49:51.835569+0000","last_fullsized":"2024-02-10T08:49:51.835569+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:48:54.417173+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.417173+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":828,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730007+0000","last_change":"2024-02-10T08:48:50.118042+0000","last_active":"2024-02-10T08:49:51.730007+0000","last_peered":"2024-02-10T08:49:51.730007+0000","last_clean":"2024-02-10T08:49:51.730007+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:49:51.730007+0000","last_undegraded":"2024-02-10T08:49:51.730007+0000","last_fullsized":"2024-02-10T08:49:51.730007+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:50.117976+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.117976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":813,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730367+0000","last_change":"2024-02-10T08:49:24.116688+0000","last_active":"2024-02-10T08:49:51.730367+0000","last_peered":"2024-02-10T08:49:51.730367+0000","last_clean":"2024-02-10T08:49:51.730367+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:49:51.730367+0000","last_undegraded":"2024-02-10T08:49:51.730367+0000","last_fullsized":"2024-02-10T08:49:51.730367+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:24.116601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:24.116601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":605,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211589+0000","last_change":"2024-02-10T08:48:41.943374+0000","last_active":"2024-02-10T08:49:44.211589+0000","last_peered":"2024-02-10T08:49:44.211589+0000","last_clean":"2024-02-10T08:49:44.211589+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:49:44.211589+0000","last_undegraded":"2024-02-10T08:49:44.211589+0000","last_fullsized":"2024-02-10T08:49:44.211589+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.943287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.943287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":865,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653271+0000","last_change":"2024-02-10T08:49:43.363252+0000","last_active":"2024-02-10T08:49:51.653271+0000","last_peered":"2024-02-10T08:49:51.653271+0000","last_clean":"2024-02-10T08:49:51.653271+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:49:51.653271+0000","last_undegraded":"2024-02-10T08:49:51.653271+0000","last_fullsized":"2024-02-10T08:49:51.653271+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:49:43.363195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:43.363195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":816,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:57.927387+0000","last_change":"2024-02-10T08:49:57.927387+0000","last_active":"2024-02-10T08:49:57.927387+0000","last_peered":"2024-02-10T08:49:57.927387+0000","last_clean":"2024-02-10T08:49:57.927387+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:49:57.927387+0000","last_undegraded":"2024-02-10T08:49:57.927387+0000","last_fullsized":"2024-02-10T08:49:57.927387+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:57.927313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:57.927313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":373,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837055+0000","last_change":"2024-02-10T08:49:38.810361+0000","last_active":"2024-02-10T08:49:51.837055+0000","last_peered":"2024-02-10T08:49:51.837055+0000","last_clean":"2024-02-10T08:49:51.837055+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:49:51.837055+0000","last_undegraded":"2024-02-10T08:49:51.837055+0000","last_fullsized":"2024-02-10T08:49:51.837055+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:38.810280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.810280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":735,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836751+0000","last_change":"2024-02-10T08:48:47.081659+0000","last_active":"2024-02-10T08:49:51.836751+0000","last_peered":"2024-02-10T08:49:51.836751+0000","last_clean":"2024-02-10T08:49:51.836751+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:49:51.836751+0000","last_undegraded":"2024-02-10T08:49:51.836751+0000","last_fullsized":"2024-02-10T08:49:51.836751+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.081542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":893,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651924+0000","last_change":"2024-02-10T08:49:20.324246+0000","last_active":"2024-02-10T08:49:51.651924+0000","last_peered":"2024-02-10T08:49:51.651924+0000","last_clean":"2024-02-10T08:49:51.651924+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:49:51.651924+0000","last_undegraded":"2024-02-10T08:49:51.651924+0000","last_fullsized":"2024-02-10T08:49:51.651924+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:20.324128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.324128+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":654,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651558+0000","last_change":"2024-02-10T08:48:45.089054+0000","last_active":"2024-02-10T08:49:51.651558+0000","last_peered":"2024-02-10T08:49:51.651558+0000","last_clean":"2024-02-10T08:49:51.651558+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:49:51.651558+0000","last_undegraded":"2024-02-10T08:49:51.651558+0000","last_fullsized":"2024-02-10T08:49:51.651558+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.088965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.088965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1037,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680440+0000","last_change":"2024-02-10T08:49:35.294710+0000","last_active":"2024-02-10T08:49:56.680440+0000","last_peered":"2024-02-10T08:49:56.680440+0000","last_clean":"2024-02-10T08:49:56.680440+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:49:56.680440+0000","last_undegraded":"2024-02-10T08:49:56.680440+0000","last_fullsized":"2024-02-10T08:49:56.680440+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13","last_scrub_stamp":"2024-02-10T08:49:35.294638+0000","last_deep_scrub":"689'12","last_deep_scrub_stamp":"2024-02-10T08:47:29.886696+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.294638+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":195,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835527+0000","last_change":"2024-02-10T08:48:38.471156+0000","last_active":"2024-02-10T08:49:51.835527+0000","last_peered":"2024-02-10T08:49:51.835527+0000","last_clean":"2024-02-10T08:49:51.835527+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:49:51.835527+0000","last_undegraded":"2024-02-10T08:49:51.835527+0000","last_fullsized":"2024-02-10T08:49:51.835527+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:38.471058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:30.241405+0000","last_clean_scrub_stamp":"2024-02-10T08:48:38.471058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":195,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835424+0000","last_change":"2024-02-10T08:48:30.433484+0000","last_active":"2024-02-10T08:49:51.835424+0000","last_peered":"2024-02-10T08:49:51.835424+0000","last_clean":"2024-02-10T08:49:51.835424+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:49:51.835424+0000","last_undegraded":"2024-02-10T08:49:51.835424+0000","last_fullsized":"2024-02-10T08:49:51.835424+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:30.433421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:30.433421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"798'19","reported_seq":845,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837507+0000","last_change":"2024-02-10T08:49:37.293132+0000","last_active":"2024-02-10T08:49:51.837507+0000","last_peered":"2024-02-10T08:49:51.837507+0000","last_clean":"2024-02-10T08:49:51.837507+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:49:51.837507+0000","last_undegraded":"2024-02-10T08:49:51.837507+0000","last_fullsized":"2024-02-10T08:49:51.837507+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'19","last_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_deep_scrub":"798'19","last_deep_scrub_stamp":"2024-02-10T08:49:37.293068+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.293068+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":550,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211472+0000","last_change":"2024-02-10T08:47:55.031372+0000","last_active":"2024-02-10T08:49:44.211472+0000","last_peered":"2024-02-10T08:49:44.211472+0000","last_clean":"2024-02-10T08:49:44.211472+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:49:44.211472+0000","last_undegraded":"2024-02-10T08:49:44.211472+0000","last_fullsized":"2024-02-10T08:49:44.211472+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.031301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.031301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":1012,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:56.680153+0000","last_change":"2024-02-10T08:49:22.354177+0000","last_active":"2024-02-10T08:49:56.680153+0000","last_peered":"2024-02-10T08:49:56.680153+0000","last_clean":"2024-02-10T08:49:56.680153+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:49:56.680153+0000","last_undegraded":"2024-02-10T08:49:56.680153+0000","last_fullsized":"2024-02-10T08:49:56.680153+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:49:22.354105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:49:22.354105+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":612,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212098+0000","last_change":"2024-02-10T08:47:23.919561+0000","last_active":"2024-02-10T08:49:44.212098+0000","last_peered":"2024-02-10T08:49:44.212098+0000","last_clean":"2024-02-10T08:49:44.212098+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:49:44.212098+0000","last_undegraded":"2024-02-10T08:49:44.212098+0000","last_fullsized":"2024-02-10T08:49:44.212098+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:23.919470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:23.919470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":773,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.729981+0000","last_change":"2024-02-10T08:49:29.101599+0000","last_active":"2024-02-10T08:49:51.729981+0000","last_peered":"2024-02-10T08:49:51.729981+0000","last_clean":"2024-02-10T08:49:51.729981+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:49:51.729981+0000","last_undegraded":"2024-02-10T08:49:51.729981+0000","last_fullsized":"2024-02-10T08:49:51.729981+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:49:29.101533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.101533+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":123,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652184+0000","last_change":"2024-02-10T08:49:35.988869+0000","last_active":"2024-02-10T08:49:51.652184+0000","last_peered":"2024-02-10T08:49:51.652184+0000","last_clean":"2024-02-10T08:49:51.652184+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:49:51.652184+0000","last_undegraded":"2024-02-10T08:49:51.652184+0000","last_fullsized":"2024-02-10T08:49:51.652184+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:35.988803+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:49:35.988803+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":788,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835497+0000","last_change":"2024-02-10T08:48:45.997431+0000","last_active":"2024-02-10T08:49:51.835497+0000","last_peered":"2024-02-10T08:49:51.835497+0000","last_clean":"2024-02-10T08:49:51.835497+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:49:51.835497+0000","last_undegraded":"2024-02-10T08:49:51.835497+0000","last_fullsized":"2024-02-10T08:49:51.835497+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.997364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.997364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":813,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651351+0000","last_change":"2024-02-10T08:47:55.934857+0000","last_active":"2024-02-10T08:49:51.651351+0000","last_peered":"2024-02-10T08:49:51.651351+0000","last_clean":"2024-02-10T08:49:51.651351+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:49:51.651351+0000","last_undegraded":"2024-02-10T08:49:51.651351+0000","last_fullsized":"2024-02-10T08:49:51.651351+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:55.934788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:47:55.934788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":213,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835134+0000","last_change":"2024-02-10T08:49:20.566561+0000","last_active":"2024-02-10T08:49:51.835134+0000","last_peered":"2024-02-10T08:49:51.835134+0000","last_clean":"2024-02-10T08:49:51.835134+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:49:51.835134+0000","last_undegraded":"2024-02-10T08:49:51.835134+0000","last_fullsized":"2024-02-10T08:49:51.835134+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.566497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.566497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":751,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835965+0000","last_change":"2024-02-10T08:49:26.780938+0000","last_active":"2024-02-10T08:49:51.835965+0000","last_peered":"2024-02-10T08:49:51.835965+0000","last_clean":"2024-02-10T08:49:51.835965+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:49:51.835965+0000","last_undegraded":"2024-02-10T08:49:51.835965+0000","last_fullsized":"2024-02-10T08:49:51.835965+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:26.780857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:26.780857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":708,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:53.268988+0000","last_change":"2024-02-10T08:49:53.268988+0000","last_active":"2024-02-10T08:49:53.268988+0000","last_peered":"2024-02-10T08:49:53.268988+0000","last_clean":"2024-02-10T08:49:53.268988+0000","last_became_active":"2024-02-10T08:47:49.241725+0000","last_became_peered":"2024-02-10T08:47:49.241725+0000","last_unstale":"2024-02-10T08:49:53.268988+0000","last_undegraded":"2024-02-10T08:49:53.268988+0000","last_fullsized":"2024-02-10T08:49:53.268988+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:53.268898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:53.268898+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":118,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:57.159954+0000","last_change":"2024-02-10T08:49:57.159918+0000","last_active":"2024-02-10T08:49:57.159954+0000","last_peered":"2024-02-10T08:49:57.159954+0000","last_clean":"2024-02-10T08:49:57.159954+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:49:57.159954+0000","last_undegraded":"2024-02-10T08:49:57.159954+0000","last_fullsized":"2024-02-10T08:49:57.159954+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:57.159820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:57.159820+0000","last_clean_scrub_stamp":"2024-02-10T08:49:57.159820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":118,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:58.134458+0000","last_change":"2024-02-10T08:49:58.134458+0000","last_active":"2024-02-10T08:49:58.134458+0000","last_peered":"2024-02-10T08:49:58.134458+0000","last_clean":"2024-02-10T08:49:58.134458+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:49:58.134458+0000","last_undegraded":"2024-02-10T08:49:58.134458+0000","last_fullsized":"2024-02-10T08:49:58.134458+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:58.134385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:49:58.134385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":793,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652347+0000","last_change":"2024-02-10T08:48:40.049047+0000","last_active":"2024-02-10T08:49:51.652347+0000","last_peered":"2024-02-10T08:49:51.652347+0000","last_clean":"2024-02-10T08:49:51.652347+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:49:51.652347+0000","last_undegraded":"2024-02-10T08:49:51.652347+0000","last_fullsized":"2024-02-10T08:49:51.652347+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:40.048948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:40.048948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":749,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653127+0000","last_change":"2024-02-10T08:49:31.379159+0000","last_active":"2024-02-10T08:49:51.653127+0000","last_peered":"2024-02-10T08:49:51.653127+0000","last_clean":"2024-02-10T08:49:51.653127+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:49:51.653127+0000","last_undegraded":"2024-02-10T08:49:51.653127+0000","last_fullsized":"2024-02-10T08:49:51.653127+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:31.379093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:31.379093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"473'8","reported_seq":821,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.650843+0000","last_change":"2024-02-10T08:49:33.304282+0000","last_active":"2024-02-10T08:49:51.650843+0000","last_peered":"2024-02-10T08:49:51.650843+0000","last_clean":"2024-02-10T08:49:51.650843+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:49:51.650843+0000","last_undegraded":"2024-02-10T08:49:51.650843+0000","last_fullsized":"2024-02-10T08:49:51.650843+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8","last_scrub_stamp":"2024-02-10T08:49:33.304177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:33.304177+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":784,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837492+0000","last_change":"2024-02-10T08:48:36.153670+0000","last_active":"2024-02-10T08:49:51.837492+0000","last_peered":"2024-02-10T08:49:51.837492+0000","last_clean":"2024-02-10T08:49:51.837492+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:49:51.837492+0000","last_undegraded":"2024-02-10T08:49:51.837492+0000","last_fullsized":"2024-02-10T08:49:51.837492+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:36.153585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:36.153585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":809,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836126+0000","last_change":"2024-02-10T08:48:32.816285+0000","last_active":"2024-02-10T08:49:51.836126+0000","last_peered":"2024-02-10T08:49:51.836126+0000","last_clean":"2024-02-10T08:49:51.836126+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:49:51.836126+0000","last_undegraded":"2024-02-10T08:49:51.836126+0000","last_fullsized":"2024-02-10T08:49:51.836126+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:32.816211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.816211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":373,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836241+0000","last_change":"2024-02-10T08:49:29.891433+0000","last_active":"2024-02-10T08:49:51.836241+0000","last_peered":"2024-02-10T08:49:51.836241+0000","last_clean":"2024-02-10T08:49:51.836241+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:49:51.836241+0000","last_undegraded":"2024-02-10T08:49:51.836241+0000","last_fullsized":"2024-02-10T08:49:51.836241+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.891353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.891353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":761,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651707+0000","last_change":"2024-02-10T08:49:42.399412+0000","last_active":"2024-02-10T08:49:51.651707+0000","last_peered":"2024-02-10T08:49:51.651707+0000","last_clean":"2024-02-10T08:49:51.651707+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:49:51.651707+0000","last_undegraded":"2024-02-10T08:49:51.651707+0000","last_fullsized":"2024-02-10T08:49:51.651707+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.399332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.399332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":847,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652632+0000","last_change":"2024-02-10T08:47:17.785046+0000","last_active":"2024-02-10T08:49:51.652632+0000","last_peered":"2024-02-10T08:49:51.652632+0000","last_clean":"2024-02-10T08:49:51.652632+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:49:51.652632+0000","last_undegraded":"2024-02-10T08:49:51.652632+0000","last_fullsized":"2024-02-10T08:49:51.652632+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:47:17.784930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:47:17.784930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":788,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836337+0000","last_change":"2024-02-10T08:48:53.059434+0000","last_active":"2024-02-10T08:49:51.836337+0000","last_peered":"2024-02-10T08:49:51.836337+0000","last_clean":"2024-02-10T08:49:51.836337+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:49:51.836337+0000","last_undegraded":"2024-02-10T08:49:51.836337+0000","last_fullsized":"2024-02-10T08:49:51.836337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:53.059137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:53.059137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":368,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836100+0000","last_change":"2024-02-10T08:48:45.871861+0000","last_active":"2024-02-10T08:49:51.836100+0000","last_peered":"2024-02-10T08:49:51.836100+0000","last_clean":"2024-02-10T08:49:51.836100+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:49:51.836100+0000","last_undegraded":"2024-02-10T08:49:51.836100+0000","last_fullsized":"2024-02-10T08:49:51.836100+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:45.871773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:48:45.871773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":793,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651494+0000","last_change":"2024-02-10T08:49:23.340760+0000","last_active":"2024-02-10T08:49:51.651494+0000","last_peered":"2024-02-10T08:49:51.651494+0000","last_clean":"2024-02-10T08:49:51.651494+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:49:51.651494+0000","last_undegraded":"2024-02-10T08:49:51.651494+0000","last_fullsized":"2024-02-10T08:49:51.651494+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:23.340696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:23.340696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":195,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835572+0000","last_change":"2024-02-10T08:49:42.706500+0000","last_active":"2024-02-10T08:49:51.835572+0000","last_peered":"2024-02-10T08:49:51.835572+0000","last_clean":"2024-02-10T08:49:51.835572+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:49:51.835572+0000","last_undegraded":"2024-02-10T08:49:51.835572+0000","last_fullsized":"2024-02-10T08:49:51.835572+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:42.706431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:49:42.706431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":207,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:54.609892+0000","last_change":"2024-02-10T08:49:54.609853+0000","last_active":"2024-02-10T08:49:54.609892+0000","last_peered":"2024-02-10T08:49:54.609892+0000","last_clean":"2024-02-10T08:49:54.609892+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:49:54.609892+0000","last_undegraded":"2024-02-10T08:49:54.609892+0000","last_fullsized":"2024-02-10T08:49:54.609892+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:49:54.609749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:54.609749+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":822,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835596+0000","last_change":"2024-02-10T08:48:54.005915+0000","last_active":"2024-02-10T08:49:51.835596+0000","last_peered":"2024-02-10T08:49:51.835596+0000","last_clean":"2024-02-10T08:49:51.835596+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:49:51.835596+0000","last_undegraded":"2024-02-10T08:49:51.835596+0000","last_fullsized":"2024-02-10T08:49:51.835596+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:54.005830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:54.005830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":368,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836647+0000","last_change":"2024-02-10T08:48:50.762747+0000","last_active":"2024-02-10T08:49:51.836647+0000","last_peered":"2024-02-10T08:49:51.836647+0000","last_clean":"2024-02-10T08:49:51.836647+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:49:51.836647+0000","last_undegraded":"2024-02-10T08:49:51.836647+0000","last_fullsized":"2024-02-10T08:49:51.836647+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:48:50.762672+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":770,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730132+0000","last_change":"2024-02-10T08:49:28.108791+0000","last_active":"2024-02-10T08:49:51.730132+0000","last_peered":"2024-02-10T08:49:51.730132+0000","last_clean":"2024-02-10T08:49:51.730132+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:49:51.730132+0000","last_undegraded":"2024-02-10T08:49:51.730132+0000","last_fullsized":"2024-02-10T08:49:51.730132+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:28.108727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:28.108727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":864,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836501+0000","last_change":"2024-02-10T08:49:29.815400+0000","last_active":"2024-02-10T08:49:51.836501+0000","last_peered":"2024-02-10T08:49:51.836501+0000","last_clean":"2024-02-10T08:49:51.836501+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:49:51.836501+0000","last_undegraded":"2024-02-10T08:49:51.836501+0000","last_fullsized":"2024-02-10T08:49:51.836501+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:49:29.815324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.815324+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":799,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652509+0000","last_change":"2024-02-10T08:48:41.013610+0000","last_active":"2024-02-10T08:49:51.652509+0000","last_peered":"2024-02-10T08:49:51.652509+0000","last_clean":"2024-02-10T08:49:51.652509+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:49:51.652509+0000","last_undegraded":"2024-02-10T08:49:51.652509+0000","last_fullsized":"2024-02-10T08:49:51.652509+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.013543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:22.072982+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.013543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652227+0000","last_change":"2024-02-10T08:49:21.886466+0000","last_active":"2024-02-10T08:49:51.652227+0000","last_peered":"2024-02-10T08:49:51.652227+0000","last_clean":"2024-02-10T08:49:51.652227+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:49:51.652227+0000","last_undegraded":"2024-02-10T08:49:51.652227+0000","last_fullsized":"2024-02-10T08:49:51.652227+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:21.886401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:21.886401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":113,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651692+0000","last_change":"2024-02-10T08:48:49.706321+0000","last_active":"2024-02-10T08:49:51.651692+0000","last_peered":"2024-02-10T08:49:51.651692+0000","last_clean":"2024-02-10T08:49:51.651692+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:49:51.651692+0000","last_undegraded":"2024-02-10T08:49:51.651692+0000","last_fullsized":"2024-02-10T08:49:51.651692+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:49.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:48:49.706258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":111,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:49:58.186725+0000","last_change":"2024-02-10T08:49:58.186725+0000","last_active":"2024-02-10T08:49:58.186725+0000","last_peered":"2024-02-10T08:49:58.186725+0000","last_clean":"2024-02-10T08:49:58.186725+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:49:58.186725+0000","last_undegraded":"2024-02-10T08:49:58.186725+0000","last_fullsized":"2024-02-10T08:49:58.186725+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:49:58.186644+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:49:58.186644+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":811,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.837101+0000","last_change":"2024-02-10T08:48:41.102099+0000","last_active":"2024-02-10T08:49:51.837101+0000","last_peered":"2024-02-10T08:49:51.837101+0000","last_clean":"2024-02-10T08:49:51.837101+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:49:51.837101+0000","last_undegraded":"2024-02-10T08:49:51.837101+0000","last_fullsized":"2024-02-10T08:49:51.837101+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:41.102021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:41.102021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":618,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.212108+0000","last_change":"2024-02-10T08:47:28.929584+0000","last_active":"2024-02-10T08:49:44.212108+0000","last_peered":"2024-02-10T08:49:44.212108+0000","last_clean":"2024-02-10T08:49:44.212108+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:49:44.212108+0000","last_undegraded":"2024-02-10T08:49:44.212108+0000","last_fullsized":"2024-02-10T08:49:44.212108+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:28.929452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:28.929452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":213,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835245+0000","last_change":"2024-02-10T08:49:30.665713+0000","last_active":"2024-02-10T08:49:51.835245+0000","last_peered":"2024-02-10T08:49:51.835245+0000","last_clean":"2024-02-10T08:49:51.835245+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:49:51.835245+0000","last_undegraded":"2024-02-10T08:49:51.835245+0000","last_fullsized":"2024-02-10T08:49:51.835245+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:30.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:30.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":220,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651086+0000","last_change":"2024-02-10T08:49:37.297309+0000","last_active":"2024-02-10T08:49:51.651086+0000","last_peered":"2024-02-10T08:49:51.651086+0000","last_clean":"2024-02-10T08:49:51.651086+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:49:51.651086+0000","last_undegraded":"2024-02-10T08:49:51.651086+0000","last_fullsized":"2024-02-10T08:49:51.651086+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:49:37.297246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:37.297246+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":799,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651284+0000","last_change":"2024-02-10T08:48:00.922194+0000","last_active":"2024-02-10T08:49:51.651284+0000","last_peered":"2024-02-10T08:49:51.651284+0000","last_clean":"2024-02-10T08:49:51.651284+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:49:51.651284+0000","last_undegraded":"2024-02-10T08:49:51.651284+0000","last_fullsized":"2024-02-10T08:49:51.651284+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:00.922104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:48:00.922104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":775,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653416+0000","last_change":"2024-02-10T08:48:43.032173+0000","last_active":"2024-02-10T08:49:51.653416+0000","last_peered":"2024-02-10T08:49:51.653416+0000","last_clean":"2024-02-10T08:49:51.653416+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:49:51.653416+0000","last_undegraded":"2024-02-10T08:49:51.653416+0000","last_fullsized":"2024-02-10T08:49:51.653416+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:43.032094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:43.032094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":122,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.652893+0000","last_change":"2024-02-10T08:49:25.890744+0000","last_active":"2024-02-10T08:49:51.652893+0000","last_peered":"2024-02-10T08:49:51.652893+0000","last_clean":"2024-02-10T08:49:51.652893+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:49:51.652893+0000","last_undegraded":"2024-02-10T08:49:51.652893+0000","last_fullsized":"2024-02-10T08:49:51.652893+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:25.890678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.751695+0000","last_clean_scrub_stamp":"2024-02-10T08:49:25.890678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"473'15","reported_seq":675,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211473+0000","last_change":"2024-02-10T08:48:32.990056+0000","last_active":"2024-02-10T08:49:44.211473+0000","last_peered":"2024-02-10T08:49:44.211473+0000","last_clean":"2024-02-10T08:49:44.211473+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:49:44.211473+0000","last_undegraded":"2024-02-10T08:49:44.211473+0000","last_fullsized":"2024-02-10T08:49:44.211473+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:48:32.989985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:48:32.989985+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":365,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836735+0000","last_change":"2024-02-10T08:48:55.711815+0000","last_active":"2024-02-10T08:49:51.836735+0000","last_peered":"2024-02-10T08:49:51.836735+0000","last_clean":"2024-02-10T08:49:51.836735+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:49:51.836735+0000","last_undegraded":"2024-02-10T08:49:51.836735+0000","last_fullsized":"2024-02-10T08:49:51.836735+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:55.711733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:48:55.711733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":213,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835303+0000","last_change":"2024-02-10T08:49:34.640456+0000","last_active":"2024-02-10T08:49:51.835303+0000","last_peered":"2024-02-10T08:49:51.835303+0000","last_clean":"2024-02-10T08:49:51.835303+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:49:51.835303+0000","last_undegraded":"2024-02-10T08:49:51.835303+0000","last_fullsized":"2024-02-10T08:49:51.835303+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:34.640388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:34.640388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":764,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.730565+0000","last_change":"2024-02-10T08:49:20.063342+0000","last_active":"2024-02-10T08:49:51.730565+0000","last_peered":"2024-02-10T08:49:51.730565+0000","last_clean":"2024-02-10T08:49:51.730565+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:49:51.730565+0000","last_undegraded":"2024-02-10T08:49:51.730565+0000","last_fullsized":"2024-02-10T08:49:51.730565+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:20.063276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:49:20.063276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":678,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211715+0000","last_change":"2024-02-10T08:48:44.939276+0000","last_active":"2024-02-10T08:49:44.211715+0000","last_peered":"2024-02-10T08:49:44.211715+0000","last_clean":"2024-02-10T08:49:44.211715+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:49:44.211715+0000","last_undegraded":"2024-02-10T08:49:44.211715+0000","last_fullsized":"2024-02-10T08:49:44.211715+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:48:44.939188+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:48:44.939188+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":311,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651159+0000","last_change":"2024-02-10T08:47:57.938158+0000","last_active":"2024-02-10T08:49:51.651159+0000","last_peered":"2024-02-10T08:49:51.651159+0000","last_clean":"2024-02-10T08:49:51.651159+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:49:51.651159+0000","last_undegraded":"2024-02-10T08:49:51.651159+0000","last_fullsized":"2024-02-10T08:49:51.651159+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:57.938096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:47:57.938096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":792,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835248+0000","last_change":"2024-02-10T08:49:39.311249+0000","last_active":"2024-02-10T08:49:51.835248+0000","last_peered":"2024-02-10T08:49:51.835248+0000","last_clean":"2024-02-10T08:49:51.835248+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:49:51.835248+0000","last_undegraded":"2024-02-10T08:49:51.835248+0000","last_fullsized":"2024-02-10T08:49:51.835248+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:39.311161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:49:39.311161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":194,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835518+0000","last_change":"2024-02-10T08:48:52.471904+0000","last_active":"2024-02-10T08:49:51.835518+0000","last_peered":"2024-02-10T08:49:51.835518+0000","last_clean":"2024-02-10T08:49:51.835518+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:49:51.835518+0000","last_undegraded":"2024-02-10T08:49:51.835518+0000","last_fullsized":"2024-02-10T08:49:51.835518+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:52.471831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:52.471831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":715,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211260+0000","last_change":"2024-02-10T08:49:38.160265+0000","last_active":"2024-02-10T08:49:44.211260+0000","last_peered":"2024-02-10T08:49:44.211260+0000","last_clean":"2024-02-10T08:49:44.211260+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:49:44.211260+0000","last_undegraded":"2024-02-10T08:49:44.211260+0000","last_fullsized":"2024-02-10T08:49:44.211260+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:49:38.160189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:49:38.160189+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":799,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.653099+0000","last_change":"2024-02-10T08:49:29.384350+0000","last_active":"2024-02-10T08:49:51.653099+0000","last_peered":"2024-02-10T08:49:51.653099+0000","last_clean":"2024-02-10T08:49:51.653099+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:49:51.653099+0000","last_undegraded":"2024-02-10T08:49:51.653099+0000","last_fullsized":"2024-02-10T08:49:51.653099+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:29.384172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:49:29.384172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":633,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-10T08:49:44.211395+0000","last_change":"2024-02-10T08:48:47.974449+0000","last_active":"2024-02-10T08:49:44.211395+0000","last_peered":"2024-02-10T08:49:44.211395+0000","last_clean":"2024-02-10T08:49:44.211395+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:49:44.211395+0000","last_undegraded":"2024-02-10T08:49:44.211395+0000","last_fullsized":"2024-02-10T08:49:44.211395+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:47.974390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:48:47.974390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":753,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.836677+0000","last_change":"2024-02-10T08:48:42.043077+0000","last_active":"2024-02-10T08:49:51.836677+0000","last_peered":"2024-02-10T08:49:51.836677+0000","last_clean":"2024-02-10T08:49:51.836677+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:49:51.836677+0000","last_undegraded":"2024-02-10T08:49:51.836677+0000","last_fullsized":"2024-02-10T08:49:51.836677+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:42.043006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:48:42.043006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":872,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-02-10T08:50:00.264543+0000","last_change":"2024-02-10T08:50:00.264543+0000","last_active":"2024-02-10T08:50:00.264543+0000","last_peered":"2024-02-10T08:50:00.264543+0000","last_clean":"2024-02-10T08:50:00.264543+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:50:00.264543+0000","last_undegraded":"2024-02-10T08:50:00.264543+0000","last_fullsized":"2024-02-10T08:50:00.264543+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:50:00.264474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:00.264474+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":206,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835543+0000","last_change":"2024-02-10T08:48:02.600767+0000","last_active":"2024-02-10T08:49:51.835543+0000","last_peered":"2024-02-10T08:49:51.835543+0000","last_clean":"2024-02-10T08:49:51.835543+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:49:51.835543+0000","last_undegraded":"2024-02-10T08:49:51.835543+0000","last_fullsized":"2024-02-10T08:49:51.835543+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:48:02.600704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:48:02.600704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":790,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.651162+0000","last_change":"2024-02-10T08:47:10.668266+0000","last_active":"2024-02-10T08:49:51.651162+0000","last_peered":"2024-02-10T08:49:51.651162+0000","last_clean":"2024-02-10T08:49:51.651162+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:49:51.651162+0000","last_undegraded":"2024-02-10T08:49:51.651162+0000","last_fullsized":"2024-02-10T08:49:51.651162+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:47:10.668200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:47:10.668200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":768,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-02-10T08:49:51.835337+0000","last_change":"2024-02-10T08:49:32.715296+0000","last_active":"2024-02-10T08:49:51.835337+0000","last_peered":"2024-02-10T08:49:51.835337+0000","last_clean":"2024-02-10T08:49:51.835337+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:49:51.835337+0000","last_undegraded":"2024-02-10T08:49:51.835337+0000","last_fullsized":"2024-02-10T08:49:51.835337+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:49:32.715176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:49:32.715176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":10732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299595,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315412,"kb_used_data":5900,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429740,"statfs":{"total":95995035648,"available":95672053760,"internally_reserved":0,"allocated":6041600,"data_stored":3109485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56100000000000005,"15min":0.60499999999999998},"min":{"1min":0.33200000000000002,"5min":0.27800000000000002,"15min":0.246},"max":{"1min":0.751,"5min":5.3200000000000003,"15min":22.143000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.59499999999999997,"15min":0.58599999999999997},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70699999999999996,"5min":11.000999999999999,"15min":11.000999999999999},"last":0.66000000000000003}]},{"osd":1,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.67400000000000004,"15min":0.60999999999999999},"min":{"1min":0.30399999999999999,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":0.80100000000000005,"5min":9.2560000000000002,"15min":9.2560000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.67700000000000005,"15min":0.61399999999999999},"min":{"1min":0.32100000000000001,"5min":0.28399999999999997,"15min":0.19900000000000001},"max":{"1min":0.64000000000000001,"5min":10.978,"15min":10.978},"last":0.58999999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67600000000000005,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.66000000000000003,"5min":7.3949999999999996,"15min":7.3949999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63600000000000001,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":7.5099999999999998,"15min":19.369},"last":0.69299999999999995}]},{"osd":3,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57099999999999995,"15min":0.59299999999999997},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68899999999999995,"5min":9.3439999999999994,"15min":9.3439999999999994},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.72199999999999998,"15min":0.70099999999999996},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":0.67400000000000004,"5min":7.8700000000000001,"15min":22.102},"last":0.71599999999999997}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70399999999999996,"5min":1.581,"15min":1.581},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.66200000000000003,"15min":0.70899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.73099999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.50900000000000001,"15min":0.54600000000000004},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65800000000000003,"5min":4.46,"15min":19.326000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60799999999999998,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.23999999999999999,"15min":0.23899999999999999},"max":{"1min":0.76400000000000001,"5min":10.946999999999999,"15min":10.946999999999999},"last":0.627}]},{"osd":6,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.74399999999999999,"5min":4.718,"15min":22.687000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.39700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70299999999999996,"5min":5.484,"15min":5.484},"last":0.63800000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561233,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307208,"kb_used_data":5888,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437944,"statfs":{"total":95995035648,"available":95680454656,"internally_reserved":0,"allocated":6029312,"data_stored":3109713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52600000000000002,"15min":0.55600000000000005},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61499999999999999,"5min":3.9399999999999999,"15min":8.5250000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.751,"15min":0.63300000000000001},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":18.777999999999999,"15min":18.777999999999999},"last":0.372}]},{"osd":1,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.73399999999999999,"15min":0.65300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.66300000000000003,"5min":18.425999999999998,"15min":18.425999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":0.69699999999999995,"5min":8.4399999999999995,"15min":18.402999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76800000000000002,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0509999999999999,"5min":10.772,"15min":10.772},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.84099999999999997,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":18.829000000000001,"15min":18.829000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.79500000000000004,"15min":0.70499999999999996},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.26000000000000001},"max":{"1min":0.64300000000000002,"5min":9.2650000000000006,"15min":16.823},"last":0.311},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.443},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.68000000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.48899999999999999,"15min":0.47399999999999998},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.41699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":18.853999999999999,"15min":18.853999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26000000000000001},"max":{"1min":0.70999999999999996,"5min":3.915,"15min":3.915},"last":0.34499999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":7.5650000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.432}]}]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307268,"kb_used_data":5948,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437884,"statfs":{"total":95995035648,"available":95680393216,"internally_reserved":0,"allocated":6090752,"data_stored":3111388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.5669999999999999,"5min":4.8710000000000004,"15min":7.8780000000000001},"last":0.191},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.39500000000000002,"15min":0.46100000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.71799999999999997,"5min":1.4910000000000001,"15min":16.166},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.47699999999999998,"15min":0.65100000000000002},"min":{"1min":0.246,"5min":0.193,"15min":0.193},"max":{"1min":1.637,"5min":7.4820000000000002,"15min":34.985999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.44500000000000001,"15min":0.55600000000000005},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.93100000000000005,"5min":3.3380000000000001,"15min":21.614999999999998},"last":0.26700000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.44600000000000001,"15min":0.51400000000000001},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.20599999999999999},"max":{"1min":0.51600000000000001,"5min":4.8940000000000001,"15min":24.773},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.433,"15min":0.50800000000000001},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":2.222,"15min":16.120999999999999},"last":0.30599999999999999}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81799999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.499},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.441,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73299999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57799999999999996}]},{"osd":5,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.623,"15min":0.61199999999999999},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76100000000000001,"5min":3.496,"15min":9.5600000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77700000000000002,"5min":4.8310000000000004,"15min":7.8479999999999999},"last":0.623}]},{"osd":6,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.622},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.312},"max":{"1min":0.82299999999999995,"5min":3.4710000000000001,"15min":9.532},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67000000000000004,"15min":0.67500000000000004},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.874,"5min":4.8520000000000003,"15min":7.9000000000000004},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.73299999999999998,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":12.147,"15min":12.147},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78400000000000003,"5min":4.8019999999999996,"15min":7.9199999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738606,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307216,"kb_used_data":5896,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437936,"statfs":{"total":95995035648,"available":95680446464,"internally_reserved":0,"allocated":6037504,"data_stored":3108824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45700000000000002,"15min":0.40500000000000003},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.156},"max":{"1min":1.4239999999999999,"5min":11.960000000000001,"15min":11.960000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.74199999999999999,"15min":0.5},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":3.8839999999999999,"5min":20.757999999999999,"15min":20.757999999999999},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.54100000000000004,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.501,"5min":11.987,"15min":11.987},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.20599999999999999,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":0.307}]},{"osd":3,"last update":"Sat Feb 10 08:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42999999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.502,"5min":2.2919999999999998,"15min":8.5310000000000006},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.51900000000000002,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.435,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75800000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.46300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79600000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Feb 10 08:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.65600000000000003,"15min":0.59399999999999997},"min":{"1min":0.38700000000000001,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.3149999999999999,"5min":4.609,"15min":4.609},"last":0.498},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.32300000000000001},"max":{"1min":3.2320000000000002,"5min":3.2320000000000002,"15min":3.4079999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Sat Feb 10 08:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55700000000000005,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":1.8260000000000001,"15min":3.3879999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.33300000000000002},"max":{"1min":3.9609999999999999,"5min":4.5469999999999997,"15min":8.5730000000000004},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.48399999999999999,"5min":0.32500000000000001,"15min":0.317},"max":{"1min":0.94499999999999995,"5min":1.849,"15min":3.363},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.66000000000000003},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":8.4299999999999997},"last":0.51000000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411556,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303492,"kb_used_data":5884,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441660,"statfs":{"total":95995035648,"available":95684259840,"internally_reserved":0,"allocated":6025216,"data_stored":3108973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.54800000000000004,"15min":0.502},"min":{"1min":0.214,"5min":0.183,"15min":0.183},"max":{"1min":1.2909999999999999,"5min":16.170000000000002,"15min":16.170000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.52800000000000002,"15min":0.56200000000000006},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":0.42299999999999999,"5min":10.567,"15min":14.858000000000001},"last":0.30199999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.51200000000000001,"15min":0.55700000000000005},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.63500000000000001,"5min":6.9720000000000004,"15min":12.512},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.56399999999999995,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.52200000000000002,"5min":12.875,"15min":12.875},"last":0.316}]},{"osd":3,"last update":"Sat Feb 10 08:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.61199999999999999,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50800000000000001,"5min":10.994999999999999,"15min":11.612},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.61499999999999999,"15min":0.57399999999999995},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":7.8280000000000003,"15min":11.710000000000001},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":1.298,"15min":1.298},"last":0.51200000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.68700000000000006,"15min":0.70099999999999996},"min":{"1min":0.39000000000000001,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.76600000000000001,"5min":10.34,"15min":10.34},"last":0.495},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.62,"15min":0.66100000000000003},"min":{"1min":0.42399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.877,"5min":6.8970000000000002,"15min":9.6690000000000005},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Feb 10 08:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.753,"15min":0.71499999999999997},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":10.372999999999999,"15min":10.372999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.79800000000000004,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.71899999999999997,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.34200000000000003,"15min":0.33900000000000002},"max":{"1min":0.76600000000000001,"5min":10.539,"15min":10.539},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.77800000000000002,"15min":0.70899999999999996},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.32500000000000001},"max":{"1min":0.70199999999999996,"5min":9.6319999999999997,"15min":9.6319999999999997},"last":0.54600000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182624,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303528,"kb_used_data":5920,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441624,"statfs":{"total":95995035648,"available":95684222976,"internally_reserved":0,"allocated":6062080,"data_stored":3109681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.20699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.497,"5min":23.443000000000001,"15min":23.443000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.251,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.58599999999999997,"5min":10.484999999999999,"15min":11.808},"last":0.23499999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.76900000000000002,"15min":0.67200000000000004},"min":{"1min":0.248,"5min":0.215,"15min":0.184},"max":{"1min":0.53800000000000003,"5min":20.902000000000001,"15min":20.902000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.26000000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.58799999999999997,"5min":17.343,"15min":17.343},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.754,"15min":0.68000000000000005},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22},"max":{"1min":0.57299999999999995,"5min":17.201000000000001,"15min":17.201000000000001},"last":0.314},{"interface":"front","average":{"1min":0.371,"5min":0.443,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57199999999999995,"5min":5.0259999999999998,"15min":23.294},"last":0.29499999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:49:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84199999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.35699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57199999999999995,"15min":0.70599999999999996},"min":{"1min":0.42099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":3.1389999999999998,"15min":12.579000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.73999999999999999,"15min":0.70599999999999996},"min":{"1min":0.41299999999999998,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":0.82999999999999996,"5min":17.292000000000002,"15min":17.292000000000002},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58999999999999997,"15min":0.71699999999999997},"min":{"1min":0.42499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79200000000000004,"5min":2.8370000000000002,"15min":16.091000000000001},"last":0.373},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.73199999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":17.260999999999999,"15min":17.260999999999999},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.61399999999999999,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.81599999999999995,"5min":5.056,"15min":14.773999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59099999999999997,"15min":0.751},"min":{"1min":0.52700000000000002,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":1.6779999999999999,"15min":12.606999999999999},"last":0.42699999999999999}]}]},{"osd":4,"up_from":633,"seq":2718714298411,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309644,"kb_used_data":6276,"kb_used_omap":59,"kb_used_meta":303300,"kb_avail":93435508,"statfs":{"total":95995035648,"available":95677960192,"internally_reserved":0,"allocated":6426624,"data_stored":3361767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":310579819},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.66500000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36299999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.65200000000000002,"5min":0.754,"15min":0.754},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66100000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59999999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.315,"15min":0.315},"max":{"1min":0.63700000000000001,"5min":1.036,"15min":1.036},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":0.746,"15min":0.746},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.61799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51700000000000002,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68600000000000005,"5min":0.87,"15min":0.87},"last":0.42499999999999999}]},{"osd":6,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52800000000000002,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.67500000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69699999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44400000000000001}]},{"osd":7,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.32200000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71799999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71699999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.5}]}]},{"osd":5,"up_from":37,"seq":158913790166,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311684,"kb_used_data":5884,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433468,"statfs":{"total":95995035648,"available":95675871232,"internally_reserved":0,"allocated":6025216,"data_stored":3108784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58099999999999996,"15min":1.05},"min":{"1min":0.29499999999999998,"5min":0.249,"15min":0.216},"max":{"1min":0.93500000000000005,"5min":7.923,"15min":107.551},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53200000000000003,"15min":0.97499999999999998},"min":{"1min":0.36899999999999999,"5min":0.27800000000000002,"15min":0.255},"max":{"1min":0.85499999999999998,"5min":2.9590000000000001,"15min":88.792000000000002},"last":0.497}]},{"osd":1,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":1.0069999999999999},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.216},"max":{"1min":0.82199999999999995,"5min":12.375999999999999,"15min":107.205},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.67700000000000005,"15min":0.88300000000000001},"min":{"1min":0.45100000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.90000000000000002,"5min":12.845000000000001,"15min":72.272000000000006},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.66800000000000004,"15min":1.0349999999999999},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":1.4650000000000001,"5min":9.9600000000000009,"15min":54.174999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58299999999999996,"15min":1.073},"min":{"1min":0.40100000000000002,"5min":0.26900000000000002,"15min":0.26600000000000001},"max":{"1min":1.405,"5min":2.335,"15min":86.183000000000007},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:49:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66700000000000004,"15min":0.86399999999999999},"min":{"1min":0.41799999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":1.0529999999999999,"5min":9.9900000000000002,"15min":44.395000000000003},"last":0.435},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63300000000000001,"15min":1.29},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.29099999999999998},"max":{"1min":0.92200000000000004,"5min":8.0120000000000005,"15min":114.526},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.376},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.316,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.35299999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53400000000000003,"15min":0.73299999999999998},"min":{"1min":0.28999999999999998,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":0.82299999999999995,"5min":10.023999999999999,"15min":72.241},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81000000000000005,"5min":10.013,"15min":22.469999999999999},"last":0.41799999999999998}]},{"osd":7,"last update":"Sat Feb 10 08:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.50800000000000001,"15min":0.72799999999999998},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.96699999999999997,"5min":2.3039999999999998,"15min":66.844999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52200000000000002,"15min":0.75900000000000001},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.253},"max":{"1min":0.79200000000000004,"5min":2.2320000000000002,"15min":72.257000000000005},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:50:04.073 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-10T08:50:04.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:03.347152+0000 mgr.y (mgr.14553) 679 : audit [DBG] from='client.54455 v1:172.21.15.107:0/3124196061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:04.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:03.583361+0000 mgr.y (mgr.14553) 680 : cluster [DBG] pgmap v1238: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:04.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.262272+0000 osd.3 (osd.3) 367 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:50:04.712 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.264236+0000 osd.3 (osd.3) 368 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:50:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:03.347152+0000 mgr.y (mgr.14553) 679 : audit [DBG] from='client.54455 v1:172.21.15.107:0/3124196061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:03.583361+0000 mgr.y (mgr.14553) 680 : cluster [DBG] pgmap v1238: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:04.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.262272+0000 osd.3 (osd.3) 367 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:50:04.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.264236+0000 osd.3 (osd.3) 368 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:50:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:03.347152+0000 mgr.y (mgr.14553) 679 : audit [DBG] from='client.54455 v1:172.21.15.107:0/3124196061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:50:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:03.583361+0000 mgr.y (mgr.14553) 680 : cluster [DBG] pgmap v1238: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.262272+0000 osd.3 (osd.3) 367 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:50:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.264236+0000 osd.3 (osd.3) 368 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(osd.7.52:62 60.1f e493/135 60:f893ecdf:test-rados-api-smithi107-112490-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head v 493'43, mlcod=439'41)", 2024-02-10T08:50:05.251 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 366.08230646300001, 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014157032999999999, 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166607+0000", 2024-02-10T08:50:05.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166606+0000", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1691999999999999e-05 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166618+0000", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5758000000000001e-05 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166633+0000", 2024-02-10T08:50:05.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1700000000000002e-06 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166640+0000", 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9800000000000002e-05 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166719+0000", 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2801000000000001e-05 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.166742+0000", 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013993217000000001 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180735+0000", 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8994999999999999e-05 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.180764+0000", 2024-02-10T08:50:05.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014157032999999999 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7522 60.d 60:b4f23613:test-rados-api-smithi107-112490-87::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e526)", 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:32.094401+0000", 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 333.15451285900002, 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028993570999999999, 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:05.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7522 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094401+0000", 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094401+0000", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094399+0000", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7929999999999992e-06 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094408+0000", 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.241e-05 2024-02-10T08:50:05.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094421+0000", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2790000000000004e-06 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094427+0000", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5862000000000001e-05 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094483+0000", 2024-02-10T08:50:05.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6359999999999999e-06 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.094489+0000", 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002673244 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.097162+0000", 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7472999999999994e-05 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.097249+0000", 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104899 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:05.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.097354+0000", 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037919899999999999 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.097733+0000", 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02556804 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.123301+0000", 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0261e-05 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.123332+0000", 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2854999999999994e-05 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:32.123395+0000", 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028993570999999999 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7534 60.d 60:b4f23613:test-rados-api-smithi107-112490-87::foo:head [write 6~1] snapc 25=[25] ondisk+write+known_if_redirected e527)", 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:33.116788+0000", 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 332.13212574800002, 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054290126000000001, 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7534 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.264 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116788+0000", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116788+0000", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116784+0000", 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4735999999999999e-05 2024-02-10T08:50:05.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116799+0000", 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9831999999999999e-05 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116819+0000", 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7010000000000008e-06 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116829+0000", 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0078999999999999e-05 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116889+0000", 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8209999999999997e-06 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.116898+0000", 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013999849999999999 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.130898+0000", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100892 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.130999+0000", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000194904 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.131194+0000", 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095546639999999992 2024-02-10T08:50:05.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.140749+0000", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030198071999999999 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.170947+0000", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5037000000000003e-05 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.170992+0000", 2024-02-10T08:50:05.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6342999999999996e-05 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.171078+0000", 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054290126000000001 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7537 60.d 60:b4f23613:test-rados-api-smithi107-112490-87::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e527)", 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:33.185251+0000", 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 332.063662443, 2024-02-10T08:50:05.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015018626, 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7537 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185251+0000", 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185251+0000", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185250+0000", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4099999999999999e-06 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185256+0000", 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6919999999999997e-06 2024-02-10T08:50:05.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185265+0000", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5879999999999997e-06 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185271+0000", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4854000000000002e-05 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185306+0000", 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3255000000000003e-05 2024-02-10T08:50:05.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.185379+0000", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013976056000000001 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.199355+0000", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00042412899999999998 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.199779+0000", 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036901700000000002 2024-02-10T08:50:05.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.200148+0000", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2030000000000002e-05 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.200190+0000", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9853999999999998e-05 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:33.200270+0000", 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015018626 2024-02-10T08:50:05.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7543 60.d 60:b4f23613:test-rados-api-smithi107-112490-87::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e528)", 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:34.128983+0000", 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 331.11993075499998, 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054705941000000001, 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:05.276 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7543 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.128983+0000", 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.128983+0000", 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.128982+0000", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7490000000000004e-06 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.128987+0000", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0162e-05 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.128998+0000", 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7049999999999998e-06 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.129002+0000", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5142999999999999e-05 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.129037+0000", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6607000000000006e-05 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.129124+0000", 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022259245 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.151383+0000", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00042863599999999998 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.151812+0000", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031746591999999997 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.183559+0000", 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3454000000000001e-05 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.183602+0000", 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7089999999999995e-05 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:34.183689+0000", 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054705941000000001 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53936.0:3 231.6 231:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e571)", 2024-02-10T08:50:05.281 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:45:17.114798+0000", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 288.13411563, 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076166364, 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53936", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/130121399", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114798+0000", 2024-02-10T08:50:05.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114798+0000", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114795+0000", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0603e-05 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114806+0000", 2024-02-10T08:50:05.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2210000000000003e-06 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114815+0000", 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8560000000000007e-06 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114823+0000", 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4192e-05 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114867+0000", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2970000000000003e-06 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.114873+0000", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003128703 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.118001+0000", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020664899999999999 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:50:05.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.118208+0000", 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019207762 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.137416+0000", 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101357 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.137517+0000", 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016185700000000001 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.137679+0000", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034745099999999998 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.138026+0000", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052804242000000001 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.190831+0000", 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6826999999999997e-05 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.190877+0000", 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7250999999999995e-05 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.190965+0000", 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076166364 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.288 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53936.0:4 231.6 231:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e571)", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:45:17.191931+0000", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 288.05698331600001, 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026042777, 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53936", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/130121399", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 4 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191931+0000", 2024-02-10T08:50:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191931+0000", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191929+0000", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5830000000000003e-06 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191935+0000", 2024-02-10T08:50:05.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6500000000000008e-06 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191945+0000", 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5420000000000002e-06 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191952+0000", 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7067000000000003e-05 2024-02-10T08:50:05.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191989+0000", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3305000000000003e-05 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.192072+0000", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025704200999999999 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.217776+0000", 2024-02-10T08:50:05.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2065999999999997e-05 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.217858+0000", 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011510000000000001 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.217973+0000", 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026042777 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53936.0:5 231.6 231:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e571)", 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:45:17.191960+0000", 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 288.05695393600001, 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027033393999999999, 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53936", 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/130121399", 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 5 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191960+0000", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191960+0000", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191959+0000", 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.544e-06 2024-02-10T08:50:05.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191964+0000", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3619999999999997e-06 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191969+0000", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4689999999999999e-06 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.191974+0000", 2024-02-10T08:50:05.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024679299999999998 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.192220+0000", 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7589999999999997e-05 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.192268+0000", 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025732023 2024-02-10T08:50:05.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218000+0000", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9465e-05 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218069+0000", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128133 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218198+0000", 2024-02-10T08:50:05.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031397400000000001 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218511+0000", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035880999999999999 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218870+0000", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3136999999999997e-05 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218913+0000", 2024-02-10T08:50:05.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9898e-05 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:45:17.218993+0000", 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027033393999999999 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:644 239.c 239:36322a8f:test-rados-api-smithi107-112490-111::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e651)", 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:46.160645+0000", 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 199.088268715, 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023348151000000001, 2024-02-10T08:50:05.300 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 644 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160645+0000", 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160645+0000", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160644+0000", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1370000000000004e-06 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160648+0000", 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9949999999999999e-06 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160655+0000", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4809999999999998e-06 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160659+0000", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9539999999999998e-05 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160688+0000", 2024-02-10T08:50:05.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2203e-05 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.160760+0000", 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023232843999999999 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.183993+0000", 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023348151000000001 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.12 673/672 [PushReplyOp(4:4a2e2f65:::obj_delete_at_hint.0000000071:head)])", 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.373017+0000", 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 176.87589706099999, 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089918057999999995, 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373017+0000", 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373017+0000", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373016+0000", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3909999999999997e-06 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373021+0000", 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1207e-05 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373032+0000", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6309999999999997e-06 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.373039+0000", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089655093000000005 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.462694+0000", 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000240712 2024-02-10T08:50:05.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.462935+0000", 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089918057999999995 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.12 673/672 [PushReplyOp(4:4b3bdbb4:::obj_delete_at_hint.0000000074:head)])", 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.635243+0000", 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 176.613670668, 2024-02-10T08:50:05.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089826998000000005, 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635243+0000", 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635243+0000", 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635243+0000", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.456e-06 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635246+0000", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2960000000000001e-06 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635254+0000", 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4719999999999995e-06 2024-02-10T08:50:05.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.635262+0000", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089699321999999998 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.724961+0000", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010884599999999999 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.725070+0000", 2024-02-10T08:50:05.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089826998000000005 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1235 243.0 243:0630f6e9:test-rados-api-smithi107-112490-119::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e685)", 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:20.148949+0000", 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 165.099964677, 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040868120000000004, 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1235 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.148949+0000", 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.148949+0000", 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999948 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.148944+0000", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0732000000000001e-05 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.148965+0000", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3938e-05 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.148989+0000", 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2062000000000001e-05 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.149001+0000", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0816000000000002e-05 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.149062+0000", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5830000000000003e-06 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.149068+0000", 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001192296 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.150261+0000", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027920400000000002 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.150540+0000", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016329370000000001 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152173+0000", 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4935000000000002e-05 2024-02-10T08:50:05.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152228+0000", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6602999999999996e-05 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152324+0000", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038921700000000002 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152713+0000", 2024-02-10T08:50:05.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000233007 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152946+0000", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6027e-05 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.152972+0000", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9856e-05 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.153002+0000", 2024-02-10T08:50:05.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3710000000000001e-05 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:20.153036+0000", 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040868120000000004 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(234.3s2 723/599 ECSubWrite(tid=1631, reqid=osd.4.633:8, at_version=723'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.207022+0000", 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 122.04189208699999, 2024-02-10T08:50:05.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042336580000000004, 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207022+0000", 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207022+0000", 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207021+0000", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088227800000000001 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207903+0000", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1889e-05 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207925+0000", 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2639999999999997e-06 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207934+0000", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0330000000000001e-05 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207985+0000", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3915e-05 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.207999+0000", 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032154340000000001 2024-02-10T08:50:05.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.211214+0000", 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1492000000000002e-05 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.211255+0000", 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042336580000000004 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.323 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(6.15 REQUEST e744)", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:28.805019+0000", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 96.443894822999994, 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023763500000000002, 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805019+0000", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805019+0000", 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805019+0000", 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.988e-06 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805022+0000", 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.275e-05 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805034+0000", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2230000000000001e-06 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805042+0000", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1130000000000006e-05 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805103+0000", 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1349999999999992e-06 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805111+0000", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022304030000000002 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.807341+0000", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8006e-05 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.807359+0000", 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6035000000000003e-05 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.807395+0000", 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023763500000000002 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2280 250.0 250:00000000:test-rados-api-smithi107-112490-133:::head [pgnls start_epoch 758] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e758)", 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:45.235526+0000", 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.013387570000006, 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022775299999999998, 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2280 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235526+0000", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235526+0000", 2024-02-10T08:50:05.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235522+0000", 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8542e-05 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235540+0000", 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9430000000000001e-05 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235570+0000", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4600000000000001e-05 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235584+0000", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012454799999999999 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235709+0000", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0069999999999997e-06 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.235714+0000", 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015477150000000001 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.237262+0000", 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7805999999999997e-05 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.237300+0000", 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050429499999999998 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.237804+0000", 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022775299999999998 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2345 250.0 250:00000000:%01:::head [pgnls start_epoch 760] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e760)", 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:47.177582+0000", 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 78.071331891, 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001727719, 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2345 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177582+0000", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177582+0000", 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:05.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177578+0000", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3431999999999999e-05 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177592+0000", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6662000000000001e-05 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177608+0000", 2024-02-10T08:50:05.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2749999999999998e-06 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177616+0000", 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9506999999999997e-05 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177685+0000", 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8800000000000008e-06 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.177693+0000", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011735070000000001 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.178867+0000", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8312000000000002e-05 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.178905+0000", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040476299999999999 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:47.179310+0000", 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001727719 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2785 251.11 251:88000000:%01:::head [pgnls start_epoch 771] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e771)", 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:04.652547+0000", 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 60.596366545999999, 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018083039999999999, 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2785 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652547+0000", 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652547+0000", 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652544+0000", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4769e-05 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652558+0000", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1423000000000001e-05 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652580+0000", 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2679999999999994e-06 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652588+0000", 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9397e-05 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652658+0000", 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9100000000000002e-06 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.652663+0000", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001508449 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.654172+0000", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6711999999999997e-05 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.654209+0000", 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014704099999999999 2024-02-10T08:50:05.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.654356+0000", 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018083039999999999 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(234.0s1 774/633 ECSubWrite(tid=50, reqid=client.48721.0:2866, at_version=774'19, trim_to=0'0, roll_forward_to=749'18))", 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:07.193821+0000", 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 58.055093286000002, 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024298010000000001, 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.345 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193821+0000", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193821+0000", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193820+0000", 2024-02-10T08:50:05.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2994000000000003e-05 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193853+0000", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3710000000000006e-06 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193862+0000", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4719999999999995e-06 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.193871+0000", 2024-02-10T08:50:05.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018609799999999999 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.194057+0000", 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1449e-05 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.194068+0000", 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002169336 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:05.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.196238+0000", 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2595000000000001e-05 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.196250+0000", 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024298010000000001 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2941 252.0 252:00000000:%01:::head [pgnls start_epoch 787] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e787)", 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:26.199175+0000", 2024-02-10T08:50:05.349 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 39.049739356000003, 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026497869999999998, 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2941 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199175+0000", 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199175+0000", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199170+0000", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6922000000000002e-05 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199187+0000", 2024-02-10T08:50:05.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0704999999999999e-05 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199208+0000", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2750000000000001e-06 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199217+0000", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3485000000000005e-05 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199291+0000", 2024-02-10T08:50:05.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9279999999999998e-06 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.199297+0000", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001986294 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.201283+0000", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5283e-05 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.201328+0000", 2024-02-10T08:50:05.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00049601199999999999 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:26.201824+0000", 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026497869999999998 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:3080 253.1f 253:f8000000:%01:::head [pgnls start_epoch 792] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e792)", 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:31.240062+0000", 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.008851726000003, 2024-02-10T08:50:05.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026878119999999999, 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3080 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240062+0000", 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240062+0000", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240059+0000", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2565000000000001e-05 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240072+0000", 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8346e-05 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240090+0000", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5980000000000002e-06 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240097+0000", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8331000000000001e-05 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240155+0000", 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1800000000000001e-06 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.240161+0000", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024366959999999999 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.242598+0000", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3142999999999997e-05 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.242641+0000", 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108964 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:05.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.242750+0000", 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026878119999999999 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:05.359 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.280671+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.280843+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.468212+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.469065+0000 osd.0 (osd.0) 187 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:04.677718+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:05.175379+0000 osd.1 (osd.1) 366 : cluster [DBG] 16.b scrub starts 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:05.178774+0000 osd.1 (osd.1) 367 : cluster [DBG] 16.b scrub ok 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:05.254096+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.a scrub starts 2024-02-10T08:50:05.870 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:05.257094+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.a scrub ok 2024-02-10T08:50:05.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T08:50:05.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:05] "GET /metrics HTTP/1.1" 200 186417 "" "Prometheus/2.33.4" 2024-02-10T08:50:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.280671+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:05.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.280843+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.468212+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.469065+0000 osd.0 (osd.0) 187 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:04.677718+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:05.175379+0000 osd.1 (osd.1) 366 : cluster [DBG] 16.b scrub starts 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:05.178774+0000 osd.1 (osd.1) 367 : cluster [DBG] 16.b scrub ok 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:05.254096+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.a scrub starts 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:05.257094+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.a scrub ok 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.280671+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.280843+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.468212+0000 osd.0 (osd.0) 186 : cluster [DBG] 5.5 scrub starts 2024-02-10T08:50:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.469065+0000 osd.0 (osd.0) 187 : cluster [DBG] 5.5 scrub ok 2024-02-10T08:50:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:04.677718+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-10T08:50:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:05.175379+0000 osd.1 (osd.1) 366 : cluster [DBG] 16.b scrub starts 2024-02-10T08:50:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:05.178774+0000 osd.1 (osd.1) 367 : cluster [DBG] 16.b scrub ok 2024-02-10T08:50:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:05.254096+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.a scrub starts 2024-02-10T08:50:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:05.257094+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.a scrub ok 2024-02-10T08:50:06.136 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:07.004 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:05.583874+0000 mgr.y (mgr.14553) 681 : cluster [DBG] pgmap v1240: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-10T08:50:07.005 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.199026+0000 osd.1 (osd.1) 368 : cluster [DBG] 4.f scrub starts 2024-02-10T08:50:07.005 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.202545+0000 osd.1 (osd.1) 369 : cluster [DBG] 4.f scrub ok 2024-02-10T08:50:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:05.583874+0000 mgr.y (mgr.14553) 681 : cluster [DBG] pgmap v1240: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-10T08:50:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.199026+0000 osd.1 (osd.1) 368 : cluster [DBG] 4.f scrub starts 2024-02-10T08:50:07.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.202545+0000 osd.1 (osd.1) 369 : cluster [DBG] 4.f scrub ok 2024-02-10T08:50:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:05.583874+0000 mgr.y (mgr.14553) 681 : cluster [DBG] pgmap v1240: 154 pgs: 154 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-10T08:50:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.199026+0000 osd.1 (osd.1) 368 : cluster [DBG] 4.f scrub starts 2024-02-10T08:50:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.202545+0000 osd.1 (osd.1) 369 : cluster [DBG] 4.f scrub ok 2024-02-10T08:50:07.769 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:07.769 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:07.770 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:50:07.770 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.516195+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.517289+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.925606+0000 osd.7 (osd.7) 336 : cluster [DBG] 5.4 deep-scrub starts 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:06.926570+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.4 deep-scrub ok 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.266409+0000 osd.5 (osd.5) 344 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:50:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.269656+0000 osd.5 (osd.5) 345 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:50:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.516195+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:50:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.517289+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:50:08.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.925606+0000 osd.7 (osd.7) 336 : cluster [DBG] 5.4 deep-scrub starts 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:06.926570+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.4 deep-scrub ok 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.266409+0000 osd.5 (osd.5) 344 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.269656+0000 osd.5 (osd.5) 345 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.516195+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.517289+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.925606+0000 osd.7 (osd.7) 336 : cluster [DBG] 5.4 deep-scrub starts 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:06.926570+0000 osd.7 (osd.7) 337 : cluster [DBG] 5.4 deep-scrub ok 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.266409+0000 osd.5 (osd.5) 344 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:50:08.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.269656+0000 osd.5 (osd.5) 345 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:50:08.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T08:50:08.692 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.160323+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.161241+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.211602+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.215060+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.555040+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.558316+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:07.584722+0000 mgr.y (mgr.14553) 682 : cluster [DBG] pgmap v1241: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.258521+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:50:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.261676+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:50:09.074 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2024-02-10T08:50:09.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd pool set default.rgw.control pgp_num 22 2024-02-10T08:50:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.160323+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:50:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.161241+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:50:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.211602+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:50:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.215060+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.555040+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.558316+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:07.584722+0000 mgr.y (mgr.14553) 682 : cluster [DBG] pgmap v1241: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.258521+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.261676+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.160323+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.161241+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.211602+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1a scrub starts 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.215060+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1a scrub ok 2024-02-10T08:50:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.555040+0000 osd.0 (osd.0) 190 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:50:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.558316+0000 osd.0 (osd.0) 191 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:50:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:07.584722+0000 mgr.y (mgr.14553) 682 : cluster [DBG] pgmap v1241: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.258521+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:50:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.261676+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:50:09.357 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.942361+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.945422+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.968886+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:08.971760+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:09.274651+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:50:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:09.277551+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:50:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.942361+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:50:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.945422+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:50:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.968886+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:50:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:08.971760+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:09.274651+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:09.277551+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.942361+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.945422+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.968886+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:08.971760+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:09.274651+0000 osd.5 (osd.5) 348 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:50:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:09.277551+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:50:10.316 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:10.316 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:10.316 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:50:10.316 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:50:10.316 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:10.336 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:10.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:09.585090+0000 mgr.y (mgr.14553) 683 : cluster [DBG] pgmap v1242: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.260648+0000 osd.1 (osd.1) 370 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.264104+0000 osd.1 (osd.1) 371 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.295693+0000 osd.5 (osd.5) 350 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.298917+0000 osd.5 (osd.5) 351 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:09.585090+0000 mgr.y (mgr.14553) 683 : cluster [DBG] pgmap v1242: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:10.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.260648+0000 osd.1 (osd.1) 370 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:50:10.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.264104+0000 osd.1 (osd.1) 371 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:50:10.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.295693+0000 osd.5 (osd.5) 350 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:50:10.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.298917+0000 osd.5 (osd.5) 351 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:50:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:09.585090+0000 mgr.y (mgr.14553) 683 : cluster [DBG] pgmap v1242: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.260648+0000 osd.1 (osd.1) 370 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:50:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.264104+0000 osd.1 (osd.1) 371 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:50:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.295693+0000 osd.5 (osd.5) 350 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:50:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.298917+0000 osd.5 (osd.5) 351 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:50:11.142 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.540+0000 7fba346e7700 1 -- 172.21.15.107:0/4143798697 >> v1:172.21.15.107:6790/0 conn(0x7fba2c103250 legacy=0x7fba2c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.540+0000 7fba346e7700 1 -- 172.21.15.107:0/4143798697 shutdown_connections 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.540+0000 7fba346e7700 1 -- 172.21.15.107:0/4143798697 >> 172.21.15.107:0/4143798697 conn(0x7fba2c0f2510 msgr2=0x7fba2c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- 172.21.15.107:0/4143798697 shutdown_connections 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- 172.21.15.107:0/4143798697 wait complete. 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 librados: starting msgr at 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 librados: starting objecter 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 Processor -- start 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- start start 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 librados: setting wanted keys 2024-02-10T08:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 librados: calling monclient init 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c1193f0 con 0x7fba2c0fc910 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c11a5d0 con 0x7fba2c0ff710 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba346e7700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c11b7d0 con 0x7fba2c103250 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba32483700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba2c0fc910 0x7fba2c0fb510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37958/0 (socket says 172.21.15.107:37958) 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.541+0000 7fba32c84700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fba2c103250 0x7fba2c117cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37182/0 (socket says 172.21.15.107:37182) 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba32483700 1 -- 172.21.15.107:0/529108461 learned_addr learned my addr 172.21.15.107:0/529108461 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter ms_handle_connect 0x7fba2c0fc910 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter ms_handle_connect 0x7fba2c103250 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter ms_handle_connect 0x7fba2c0ff710 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852758767 0 0) 0x7fba2c1193f0 con 0x7fba2c0fc910 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba18003030 con 0x7fba2c0fc910 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849908073 0 0) 0x7fba2c11b7d0 con 0x7fba2c103250 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c1193f0 con 0x7fba2c103250 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919423041 0 0) 0x7fba2c11a5d0 con 0x7fba2c0ff710 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c11b7d0 con 0x7fba2c0ff710 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 924971292 0 0) 0x7fba18003030 con 0x7fba2c0fc910 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c11a5d0 con 0x7fba2c0fc910 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fba280019f0 con 0x7fba2c0fc910 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179903821 0 0) 0x7fba2c1193f0 con 0x7fba2c103250 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba18003030 con 0x7fba2c103250 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fba240017b0 con 0x7fba2c103250 2024-02-10T08:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707677370 0 0) 0x7fba2c11b7d0 con 0x7fba2c0ff710 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c1193f0 con 0x7fba2c0ff710 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fba1c0017c0 con 0x7fba2c0ff710 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778993070 0 0) 0x7fba2c11a5d0 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 >> v1:172.21.15.107:6790/0 conn(0x7fba2c103250 legacy=0x7fba2c117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 >> v1:172.21.15.195:6789/0 conn(0x7fba2c0ff710 legacy=0x7fba2c0fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c11c9d0 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c11b9c0 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba346e7700 10 client.28863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c11beb0 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba346e7700 1 librados: init done 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fba280024e0 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.542+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fba28005b50 con 0x7fba2c0fc910 2024-02-10T08:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.543+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba10004f20 con 0x7fba2c0fc910 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fba28002040 con 0x7fba2c0fc910 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(808..808 src has 754..808) v4 ==== 7386+0+0 (unknown 338350848 0 0) 0x7fba28078a90 con 0x7fba2c0fc910 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 10 client.28863.objecter ms_dispatch 0x7fba2c001550 osd_map(808..808 src has 754..808) v4 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 3 client.28863.objecter handle_osd_map got epochs [808,808] > 0 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 3 client.28863.objecter handle_osd_map decoding full epoch 808 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 20 client.28863.objecter dump_active .. 0 homeless 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.544+0000 7fba237fe700 10 client.28863.objecter ms_handle_connect 0x7fba18063d10 2024-02-10T08:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.548+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fba2804c120 con 0x7fba2c0fc910 2024-02-10T08:50:11.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:11.716+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"} v 0) v1 -- 0x7fba100051d0 con 0x7fba2c0fc910 2024-02-10T08:50:12.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.111448+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.a scrub starts 2024-02-10T08:50:12.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:10.114354+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.a scrub ok 2024-02-10T08:50:12.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:11.304463+0000 osd.1 (osd.1) 372 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:50:12.008 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:11.308521+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:50:12.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:12.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.111448+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.a scrub starts 2024-02-10T08:50:12.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:10.114354+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.a scrub ok 2024-02-10T08:50:12.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:11.304463+0000 osd.1 (osd.1) 372 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:50:12.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:11.308521+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:50:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.111448+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.a scrub starts 2024-02-10T08:50:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:10.114354+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.a scrub ok 2024-02-10T08:50:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:11.304463+0000 osd.1 (osd.1) 372 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:50:12.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:11.308521+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:50:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.715+0000 7fba237fe700 1 -- 172.21.15.107:0/529108461 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]=0 set pool 5 pgp_num to 22 v809) v1 ==== 146+0+0 (unknown 3472499976 0 0) 0x7fba2804a530 con 0x7fba2c0fc910 2024-02-10T08:50:12.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 10 librados: watch_flush enter 2024-02-10T08:50:12.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 10 librados: watch_flush exit 2024-02-10T08:50:12.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 20 client.28863.objecter shutdown clearing up homeless session... 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 10 client.28863.objecter successfully canceled tick 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 >> v1:172.21.15.107:6800/2634842009 conn(0x7fba18063d10 legacy=0x7fba180661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 >> v1:172.21.15.107:6789/0 conn(0x7fba2c0fc910 legacy=0x7fba2c0fb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 shutdown_connections 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 >> 172.21.15.107:0/529108461 conn(0x7fba2c0f2510 msgr2=0x7fba2c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 shutdown_connections 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 -- 172.21.15.107:0/529108461 wait complete. 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:12.718+0000 7fba346e7700 1 librados: shutdown 2024-02-10T08:50:12.719 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 5 pgp_num to 22 2024-02-10T08:50:12.748 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:12.748 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:50:12.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.53375.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 595.74386890200003, 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999, 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.53375", 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/4067602482", 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.749 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002094+0000", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002090+0000", 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2568000000000001e-05 2024-02-10T08:50:12.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002103+0000", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6772000000000001e-05 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002120+0000", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4579999999999999e-06 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002129+0000", 2024-02-10T08:50:12.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2737e-05 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002182+0000", 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.963e-06 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:17.002186+0000", 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.005764858 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.007951+0000", 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014669699999999999 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008097+0000", 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045664099999999998 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008554+0000", 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038511399999999999 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.008939+0000", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023091299999999999 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009170+0000", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3881999999999996e-05 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:18.009234+0000", 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071402279999999 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.754 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.27867.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 571.73968787199999, 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001, 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.27867", 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/10211293", 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1 2024-02-10T08:50:12.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006275+0000", 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006272+0000", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2092e-05 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006284+0000", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1758e-05 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006296+0000", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5979999999999997e-06 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006304+0000", 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015616800000000001 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006460+0000", 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4299999999999999e-06 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:41.006465+0000", 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0093833510000001 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.015848+0000", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017003600000000001 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016018+0000", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00054150900000000002 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.016560+0000", 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013759289999999999 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.017936+0000", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045079799999999998 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018386+0000", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0971999999999997e-05 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:40:42.018457+0000", 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0121826350000001 2024-02-10T08:50:12.760 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.34349.0:1662 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94547991871360 gen 9] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 397.56095989800002, 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997, 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.34349", 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/987143937", 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1662 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.761 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185003+0000", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.184999+0000", 2024-02-10T08:50:12.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8840000000000006e-06 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185009+0000", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3593e-05 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185023+0000", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8629999999999996e-06 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185029+0000", 2024-02-10T08:50:12.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0421999999999999e-05 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185079+0000", 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8589999999999999e-06 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:35.185086+0000", 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88982775599999997 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.074914+0000", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016209899999999999 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.075076+0000", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005269442 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:36.080345+0000", 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89534263999999997 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.765 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7071 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e493)", 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 373.59919720900001, 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994, 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7071 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146765+0000", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146764+0000", 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.878e-06 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146771+0000", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0343000000000001e-05 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146781+0000", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4099999999999999e-06 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146786+0000", 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5929000000000001e-05 2024-02-10T08:50:12.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146822+0000", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1459999999999995e-05 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.146914+0000", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080156775999999999 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:43:59.227071+0000", 2024-02-10T08:50:12.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080305250999999994 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7073 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e494)", 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 372.57216318399998, 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997, 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:12.770 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7073 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173799+0000", 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173794+0000", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2813e-05 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173817+0000", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0312e-05 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173848+0000", 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1270999999999999e-05 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173859+0000", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4023999999999996e-05 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173923+0000", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.131e-06 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.173930+0000", 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079391889999999993 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.181869+0000", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014406799999999999 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182013+0000", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018625099999999999 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.182199+0000", 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017831909999999999 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.183983+0000", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037849075000000003 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.221832+0000", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000174136 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222006+0000", 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6679000000000001e-05 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.222103+0000", 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048303147999999997 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7074 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 372.51120722399997, 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998, 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.776 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7074 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:50:12.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234755+0000", 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999943 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234750+0000", 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0859000000000001e-05 2024-02-10T08:50:12.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234771+0000", 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8118000000000001e-05 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234799+0000", 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.234811+0000", 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038360999999999999 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235195+0000", 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839000000000001 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.235323+0000", 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038646028999999998 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:00.273969+0000", 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039213657999999998 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.781 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7076 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [write 0~2] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 371.66317536999998, 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999, 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.782 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7076 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082787+0000", 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082784+0000", 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1815e-05 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082796+0000", 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4978e-05 2024-02-10T08:50:12.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082811+0000", 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8789999999999992e-06 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082820+0000", 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6817000000000002e-05 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082877+0000", 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8079999999999994e-06 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.082886+0000", 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013785067 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096671+0000", 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109211 2024-02-10T08:50:12.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096780+0000", 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000188168 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.096968+0000", 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01095494 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.107923+0000", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175246 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108098+0000", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011893200000000001 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108217+0000", 2024-02-10T08:50:12.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2880999999999995e-05 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108300+0000", 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025512956999999999 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.7.52:92 60.12 60:49b62e28:test-rados-api-smithi107-112490-78::bfb5db8bf32239a5b4437a6edddde56f73a76483:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e495)", 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:50:12.789 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 371.63759270600002, 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002, 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108370+0000", 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108368+0000", 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3590000000000008e-06 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108376+0000", 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9339999999999995e-06 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108385+0000", 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1519999999999997e-06 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108391+0000", 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0077999999999997e-05 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108442+0000", 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0480000000000001e-05 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108532+0000", 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018637699999999999 2024-02-10T08:50:12.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2", 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.108718+0000", 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036315000000000002 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.109082+0000", 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049288513999999999 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158370+0000", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4228999999999998e-05 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158434+0000", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2422000000000004e-05 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158527+0000", 2024-02-10T08:50:12.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050156851000000002 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.25339.0:7078 220.1e 220:79c5a5b6:test-rados-api-smithi107-112490-78::foo-chunk:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e495)", 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 371.58776377100003, 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001, 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.796 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.25339", 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/949878047", 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 7078 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158199+0000", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158195+0000", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9330999999999998e-05 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158214+0000", 2024-02-10T08:50:12.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4609000000000001e-05 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158239+0000", 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9750000000000002e-06 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158249+0000", 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013279000000000001 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158381+0000", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4767999999999998e-05 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.158476+0000", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022729646999999999 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181206+0000", 2024-02-10T08:50:12.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000387876 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181594+0000", 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203425 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181797+0000", 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6589999999999999e-05 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181844+0000", 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8802999999999997e-05 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:44:01.181933+0000", 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023733737000000001 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:469 237.10 237.10 (undecoded) ondisk+read+ignore_overlay+known_if_redirected e633)", 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 224.489751748, 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.944453247, 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 469 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.803 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256211+0000", 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256208+0000", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0508e-05 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256219+0000", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3166999999999999e-05 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256232+0000", 2024-02-10T08:50:12.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9609999999999999e-06 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256239+0000", 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8158000000000001e-05 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256297+0000", 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8319999999999995e-06 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:28.256306+0000", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94434135299999999 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.200647+0000", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6847999999999999e-05 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.200664+0000", 2024-02-10T08:50:12.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.944453247 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.0.8:1 234.7s0 234:ffb4a4cf:test-rados-api-smithi107-112490-115::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e668)", 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 189.48296595299999, 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030410150000000002, 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.808 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262997+0000", 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.262993+0000", 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1848000000000001e-05 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263005+0000", 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2612000000000001e-05 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263018+0000", 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0649999999999999e-06 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263026+0000", 2024-02-10T08:50:12.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8707999999999993e-05 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263095+0000", 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6760000000000004e-06 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263100+0000", 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027235660000000002 2024-02-10T08:50:12.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.265824+0000", 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011065000000000001 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.265934+0000", 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010323399999999999 2024-02-10T08:50:12.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266038+0000", 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030410150000000002 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.0.8:2 234.7s0 234:ffb4a4cf:test-rados-api-smithi107-112490-115::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e668)", 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:50:12.814 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 189.48292643900001, 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003812757, 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263036+0000", 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.293e-06 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263041+0000", 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9949999999999996e-06 2024-02-10T08:50:12.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263046+0000", 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9299999999999999e-06 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263049+0000", 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107218 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263156+0000", 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2320000000000001e-06 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.263161+0000", 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002893077 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266054+0000", 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5282999999999999e-05 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266110+0000", 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073932200000000005 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:03.266849+0000", 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003812757 2024-02-10T08:50:12.819 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:969 241.5 241:a0000000:%01:::head [pgnls start_epoch 669] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e669)", 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 188.459761738, 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028089130000000001, 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.820 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 969 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286201+0000", 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286198+0000", 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.944e-06 2024-02-10T08:50:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286208+0000", 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2311e-05 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286220+0000", 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4770000000000002e-06 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286226+0000", 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0899000000000003e-05 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286267+0000", 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9780000000000002e-06 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.286273+0000", 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002599334 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.288872+0000", 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8223000000000003e-05 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.288910+0000", 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9705999999999997e-05 2024-02-10T08:50:12.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:04.289010+0000", 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028089130000000001 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(6.16 REQUEST e675)", 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:50:12.826 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 182.48950674299999, 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026645219999999999, 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256456+0000", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256455+0000", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2129999999999999e-06 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256460+0000", 2024-02-10T08:50:12.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9454e-05 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256480+0000", 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4569999999999999e-05 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256494+0000", 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105244 2024-02-10T08:50:12.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256600+0000", 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6909999999999999e-05 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.256617+0000", 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00246321 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259080+0000", 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.615e-05 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259096+0000", 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4400999999999999e-05 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:10.259120+0000", 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026645219999999999 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1373 243.1a 243:58000000::::head [pgnls start_epoch 687] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e687)", 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 165.50613511700001, 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021910319999999999, 2024-02-10T08:50:12.832 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1373 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239827+0000", 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239823+0000", 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5954e-05 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239839+0000", 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5633e-05 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239865+0000", 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0718e-05 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239876+0000", 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8201999999999995e-05 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239944+0000", 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1449999999999996e-06 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.239950+0000", 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001918131 2024-02-10T08:50:12.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.241868+0000", 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7027000000000003e-05 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.241905+0000", 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011327099999999999 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:27.242019+0000", 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021910319999999999 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1750 245.10 245:08000000::::head [pgnls start_epoch 708] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 144.52511039199999, 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0319844600000001, 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.838 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1750 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220852+0000", 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220849+0000", 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1374e-05 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220861+0000", 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7833999999999999e-05 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220879+0000", 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5169999999999999e-06 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220887+0000", 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4659e-05 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220942+0000", 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1242e-05 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:48.220953+0000", 2024-02-10T08:50:12.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.031753017 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252706+0000", 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5024000000000001e-05 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252751+0000", 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5668999999999996e-05 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:49.252837+0000", 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0319844600000001 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2305 250.13 250:c8000000:test-rados-api-smithi107-112490-133:::head [pgnls start_epoch 758] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e758)", 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:45.254889+0000", 2024-02-10T08:50:12.844 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 87.491073708000002, 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029832499999999998, 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2305 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254889+0000", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254889+0000", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254884+0000", 2024-02-10T08:50:12.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7640000000000001e-05 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254902+0000", 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8742999999999999e-05 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254930+0000", 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1356e-05 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.254952+0000", 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016040100000000001 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.255112+0000", 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1126e-05 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.255123+0000", 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025719039999999999 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.257695+0000", 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2719000000000002e-05 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.257738+0000", 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134141 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.257872+0000", 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029832499999999998 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2774 251.c 251:30000000:%01:::head [pgnls start_epoch 771] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e771)", 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:04.645331+0000", 2024-02-10T08:50:12.851 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 68.100631854, 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037532120000000001, 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2774 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645331+0000", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645331+0000", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645326+0000", 2024-02-10T08:50:12.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6787999999999999e-05 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645343+0000", 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4547000000000001e-05 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645367+0000", 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1056999999999999e-05 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645378+0000", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2557000000000004e-05 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645441+0000", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5679999999999996e-06 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.645448+0000", 2024-02-10T08:50:12.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035142369999999999 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.648962+0000", 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6554000000000003e-05 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.648998+0000", 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5624999999999994e-05 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:04.649084+0000", 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037532120000000001 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2871 252.5 252:a5753f94:test-rados-api-smithi107-112490-137::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e777)", 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:10.224067+0000", 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.521895551999997, 2024-02-10T08:50:12.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045846410000000004, 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2871 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224067+0000", 2024-02-10T08:50:12.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224067+0000", 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224063+0000", 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0892e-05 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224074+0000", 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6824999999999999e-05 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224091+0000", 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3470000000000004e-06 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224098+0000", 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6637000000000004e-05 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224184+0000", 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9890000000000003e-06 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.224191+0000", 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021473909999999998 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.226339+0000", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022067500000000001 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.226559+0000", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001559766 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.228119+0000", 2024-02-10T08:50:12.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6044000000000007e-05 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.228185+0000", 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030712799999999999 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.228492+0000", 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9533000000000002e-05 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.228542+0000", 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109962 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.228652+0000", 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045846410000000004 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:3055 253.c 253:30000000:%01:::head [pgnls start_epoch 792] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e792)", 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:31.224230+0000", 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.5217326, 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079410260000000003, 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3055 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:12.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224230+0000", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224230+0000", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224226+0000", 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7428e-05 2024-02-10T08:50:12.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224243+0000", 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8053000000000001e-05 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224261+0000", 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.969e-06 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224270+0000", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8471999999999999e-05 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224319+0000", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5879999999999997e-06 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.224324+0000", 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077399210000000003 2024-02-10T08:50:12.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.232064+0000", 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5772e-05 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.232090+0000", 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1044000000000005e-05 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:12.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:31.232171+0000", 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079410260000000003 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:12.870 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:11.585792+0000 mgr.y (mgr.14553) 684 : cluster [DBG] pgmap v1243: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:11.716974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:11.880692+0000 mgr.y (mgr.14553) 685 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.115447+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.115678+0000 osd.7 (osd.7) 341 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.263746+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub starts 2024-02-10T08:50:12.871 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.266631+0000 osd.1 (osd.1) 375 : cluster [DBG] 5.c scrub ok 2024-02-10T08:50:13.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:11.585792+0000 mgr.y (mgr.14553) 684 : cluster [DBG] pgmap v1243: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:13.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:11.716974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-10T08:50:13.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:11.880692+0000 mgr.y (mgr.14553) 685 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.115447+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.115678+0000 osd.7 (osd.7) 341 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.263746+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub starts 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.266631+0000 osd.1 (osd.1) 375 : cluster [DBG] 5.c scrub ok 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:11.585792+0000 mgr.y (mgr.14553) 684 : cluster [DBG] pgmap v1243: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:11.716974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:11.880692+0000 mgr.y (mgr.14553) 685 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.115447+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.115678+0000 osd.7 (osd.7) 341 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.263746+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub starts 2024-02-10T08:50:13.165 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.266631+0000 osd.1 (osd.1) 375 : cluster [DBG] 5.c scrub ok 2024-02-10T08:50:13.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T08:50:13.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-10T08:50:13.629 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:13.912 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:13.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:12.715343+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-10T08:50:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.715422+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-10T08:50:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.936574+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:50:13.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:12.940303+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:50:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:12.715343+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-10T08:50:13.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.715422+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-10T08:50:13.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.936574+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:50:13.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:12.940303+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:50:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:12.715343+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? v1:172.21.15.107:0/529108461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-10T08:50:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.715422+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-10T08:50:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.936574+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:50:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:12.940303+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:50:14.785 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.057268+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:50:14.785 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.060170+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:50:14.785 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.321256+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.c scrub starts 2024-02-10T08:50:14.785 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.322167+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.c scrub ok 2024-02-10T08:50:14.785 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.543542+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.544615+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.586216+0000 mgr.y (mgr.14553) 686 : cluster [DBG] pgmap v1245: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.806813+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.807712+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.963796+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:13.966687+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:14.325742+0000 osd.5 (osd.5) 352 : cluster [DBG] 16.c scrub starts 2024-02-10T08:50:14.786 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:14.328642+0000 osd.5 (osd.5) 353 : cluster [DBG] 16.c scrub ok 2024-02-10T08:50:15.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.057268+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.060170+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.321256+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.c scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.322167+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.c scrub ok 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.543542+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.544615+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.586216+0000 mgr.y (mgr.14553) 686 : cluster [DBG] pgmap v1245: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.806813+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.807712+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.963796+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:13.966687+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:14.325742+0000 osd.5 (osd.5) 352 : cluster [DBG] 16.c scrub starts 2024-02-10T08:50:15.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:14.328642+0000 osd.5 (osd.5) 353 : cluster [DBG] 16.c scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.057268+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.060170+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.321256+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.c scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.322167+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.c scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.543542+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.544615+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.586216+0000 mgr.y (mgr.14553) 686 : cluster [DBG] pgmap v1245: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.806813+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.807712+0000 osd.7 (osd.7) 343 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.963796+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.11 scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:13.966687+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.11 scrub ok 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:14.325742+0000 osd.5 (osd.5) 352 : cluster [DBG] 16.c scrub starts 2024-02-10T08:50:15.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:14.328642+0000 osd.5 (osd.5) 353 : cluster [DBG] 16.c scrub ok 2024-02-10T08:50:15.309 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:15.310 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:15.310 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-02-10T08:50:15.310 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:15.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.813+0000 7faf69dbf700 1 -- 172.21.15.107:0/3517853695 >> v1:172.21.15.107:6790/0 conn(0x7faf64103250 legacy=0x7faf641056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.813+0000 7faf69dbf700 1 -- 172.21.15.107:0/3517853695 shutdown_connections 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 -- 172.21.15.107:0/3517853695 >> 172.21.15.107:0/3517853695 conn(0x7faf640f2510 msgr2=0x7faf640f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 -- 172.21.15.107:0/3517853695 shutdown_connections 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 -- 172.21.15.107:0/3517853695 wait complete. 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 librados: starting msgr at 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 librados: starting objecter 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 Processor -- start 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 -- start start 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 librados: setting wanted keys 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 librados: calling monclient init 2024-02-10T08:50:15.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.814+0000 7faf69dbf700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf640fc1a0 con 0x7faf640ff710 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf64118720 con 0x7faf64103250 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faf64119900 con 0x7faf640fc910 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf637fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faf640fc910 0x7faf640fb5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37206/0 (socket says 172.21.15.107:37206) 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf637fe700 1 -- 172.21.15.107:0/3341007807 learned_addr learned my addr 172.21.15.107:0/3341007807 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter ms_handle_connect 0x7faf640ff710 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter ms_handle_connect 0x7faf640fc910 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter ms_handle_connect 0x7faf64103250 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560575813 0 0) 0x7faf64119900 con 0x7faf640fc910 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c0033f0 con 0x7faf640fc910 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659251714 0 0) 0x7faf640fc1a0 con 0x7faf640ff710 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf64119900 con 0x7faf640ff710 2024-02-10T08:50:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055348490 0 0) 0x7faf64118720 con 0x7faf64103250 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf640fc1a0 con 0x7faf64103250 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694529243 0 0) 0x7faf4c0033f0 con 0x7faf640fc910 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf64118720 con 0x7faf640fc910 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf54001980 con 0x7faf640fc910 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2779347754 0 0) 0x7faf64119900 con 0x7faf640ff710 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf4c0033f0 con 0x7faf640ff710 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413988780 0 0) 0x7faf640fc1a0 con 0x7faf64103250 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf64119900 con 0x7faf64103250 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf58001840 con 0x7faf640ff710 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c002240 con 0x7faf64103250 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 632815008 0 0) 0x7faf64118720 con 0x7faf640fc910 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.195:6789/0 conn(0x7faf64103250 legacy=0x7faf64115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6789/0 conn(0x7faf640ff710 legacy=0x7faf641128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6411ab00 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf64119af0 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 10 client.54467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf64119fe0 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf69dbf700 1 librados: init done 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7faf54002d00 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.815+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf54005bc0 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.816+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf50004f40 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7faf54018b10 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(809..809 src has 754..809) v4 ==== 7386+0+0 (unknown 1251552056 0 0) 0x7faf54078b10 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 10 client.54467.objecter ms_dispatch 0x7faf64001550 osd_map(809..809 src has 754..809) v4 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 3 client.54467.objecter handle_osd_map got epochs [809,809] > 0 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 3 client.54467.objecter handle_osd_map decoding full epoch 809 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 20 client.54467.objecter dump_active .. 0 homeless 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1111 (unknown 409899127 0 2452703222) 0x7faf54078fa0 con 0x7faf640fc910 2024-02-10T08:50:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.817+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf4c063d70 2024-02-10T08:50:15.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf50004cb0 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892351662 0 0) 0x7faf50004cb0 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf64118720 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2651163528 0 0) 0x7faf64118720 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf50004cb0 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c001da0 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2553195556 0 0) 0x7faf50004cb0 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf50002260 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7faf5c001c00 con 0x7faf500027b0 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6789/0 conn(0x7faf500027b0 legacy=0x7faf50005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.818+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6789/0 conn(0x7faf500027b0 legacy=0x7faf50005920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf5000adc0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363651175 0 0) 0x7faf5000adc0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50004cb0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2663686606 0 0) 0x7faf50004cb0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf5000adc0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c0027d0 con 0x7faf500070d0 2024-02-10T08:50:15.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3198907254 0 0) 0x7faf5000adc0 con 0x7faf500070d0 2024-02-10T08:50:15.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.825+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7faf5000c020 con 0x7faf500070d0 2024-02-10T08:50:15.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.0 v1:172.21.15.107:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7faf5c002b20 con 0x7faf500070d0 2024-02-10T08:50:15.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6789/0 conn(0x7faf500070d0 legacy=0x7faf50009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6789/0 conn(0x7faf500070d0 legacy=0x7faf50009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.827 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a: {} 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf50001ee0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.827+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818478970 0 0) 0x7faf50001ee0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5000adc0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3819769657 0 0) 0x7faf5000adc0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf50001ee0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c003900 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 337788830 0 0) 0x7faf50001ee0 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf50012980 con 0x7faf50001ab0 2024-02-10T08:50:15.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7faf5c005ae0 con 0x7faf50001ab0 2024-02-10T08:50:15.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.195:6789/0 conn(0x7faf50001ab0 legacy=0x7faf5000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.195:6789/0 conn(0x7faf50001ab0 legacy=0x7faf5000b720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.828+0000 7faf63fff700 1 -- 172.21.15.107:0/3341007807 reap_dead start 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf50013f40 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108594006 0 0) 0x7faf50013f40 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50001ee0 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 632996902 0 0) 0x7faf50001ee0 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf50013f40 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c005450 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3112430654 0 0) 0x7faf50013f40 con 0x7faf50012b50 2024-02-10T08:50:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.835+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.195:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7faf500141b0 con 0x7faf50012b50 2024-02-10T08:50:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.1 v1:172.21.15.195:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7faf5c0057c0 con 0x7faf50012b50 2024-02-10T08:50:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.195:6789/0 conn(0x7faf50012b50 legacy=0x7faf50011220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.195:6789/0 conn(0x7faf50012b50 legacy=0x7faf50011220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.838 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b: {} 2024-02-10T08:50:15.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf500126b0 con 0x7faf50011b80 2024-02-10T08:50:15.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078697631 0 0) 0x7faf500126b0 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.837+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50013f40 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 308957347 0 0) 0x7faf50013f40 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf500126b0 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c0063d0 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3438720511 0 0) 0x7faf500126b0 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf5001ab10 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7faf5c0062b0 con 0x7faf50011b80 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6790/0 conn(0x7faf50011b80 legacy=0x7faf50011fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.838+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6790/0 conn(0x7faf50011b80 legacy=0x7faf50011fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.844+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7faf5001b280 con 0x7faf5001ae30 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.844+0000 7faf60ff9700 10 client.54467.objecter ms_handle_connect 0x7faf5001ae30 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.844+0000 7faf60ff9700 10 client.54467.objecter resend_mon_ops 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512299994 0 0) 0x7faf5001b280 con 0x7faf5001ae30 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf500126b0 con 0x7faf5001ae30 2024-02-10T08:50:15.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3880562636 0 0) 0x7faf500126b0 con 0x7faf5001ae30 2024-02-10T08:50:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf5001b280 con 0x7faf5001ae30 2024-02-10T08:50:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7faf5c006610 con 0x7faf5001ae30 2024-02-10T08:50:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1427301737 0 0) 0x7faf5001b280 con 0x7faf5001ae30 2024-02-10T08:50:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.845+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 --> v1:172.21.15.107:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7faf5001c520 con 0x7faf5001ae30 2024-02-10T08:50:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 <== mon.2 v1:172.21.15.107:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7faf5c0069b0 con 0x7faf5001ae30 2024-02-10T08:50:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6790/0 conn(0x7faf5001ae30 legacy=0x7faf500193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf60ff9700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6790/0 conn(0x7faf5001ae30 legacy=0x7faf500193b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-10T08:50:15.847 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c: {} 2024-02-10T08:50:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 10 librados: watch_flush enter 2024-02-10T08:50:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 10 librados: watch_flush exit 2024-02-10T08:50:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 20 client.54467.objecter shutdown clearing up homeless session... 2024-02-10T08:50:15.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 10 client.54467.objecter successfully canceled tick 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6800/2634842009 conn(0x7faf4c063d70 legacy=0x7faf4c066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 >> v1:172.21.15.107:6790/0 conn(0x7faf640fc910 legacy=0x7faf640fb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf63fff700 1 -- 172.21.15.107:0/3341007807 reap_dead start 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 shutdown_connections 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.846+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 >> 172.21.15.107:0/3341007807 conn(0x7faf640f2510 msgr2=0x7faf640f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.847+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 shutdown_connections 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.847+0000 7faf69dbf700 1 -- 172.21.15.107:0/3341007807 wait complete. 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:15.847+0000 7faf69dbf700 1 librados: shutdown 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-10T08:50:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-10T08:50:15.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:14.056037+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:50:15.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:14.059684+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:50:15.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:14.056037+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:50:15.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:14.059684+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:50:15.866 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:15] "GET /metrics HTTP/1.1" 200 186422 "" "Prometheus/2.33.4" 2024-02-10T08:50:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:14.056037+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:50:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:14.059684+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:50:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:16.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T08:50:16.891 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:15.327734+0000 osd.3 (osd.3) 371 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:50:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:15.328584+0000 osd.3 (osd.3) 372 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:50:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:15.586588+0000 mgr.y (mgr.14553) 687 : cluster [DBG] pgmap v1246: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:16.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:15.817920+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? v1:172.21.15.107:0/3341007807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:50:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:15.327734+0000 osd.3 (osd.3) 371 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:50:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:15.328584+0000 osd.3 (osd.3) 372 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:50:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:15.586588+0000 mgr.y (mgr.14553) 687 : cluster [DBG] pgmap v1246: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:16.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:15.817920+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? v1:172.21.15.107:0/3341007807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:50:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:15.327734+0000 osd.3 (osd.3) 371 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:50:17.089 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:15.328584+0000 osd.3 (osd.3) 372 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:50:17.089 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:15.586588+0000 mgr.y (mgr.14553) 687 : cluster [DBG] pgmap v1246: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:17.089 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:15.817920+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? v1:172.21.15.107:0/3341007807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:50:17.748 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:17.748 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-02-10T08:50:17.748 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-02-10T08:50:17.748 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-02-10T08:50:17.748 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:17.772 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:16.541195+0000 osd.0 (osd.0) 194 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:50:17.772 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:16.542066+0000 osd.0 (osd.0) 195 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:50:17.772 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:17.271955+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub starts 2024-02-10T08:50:17.772 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:17.275423+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.f scrub ok 2024-02-10T08:50:18.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:16.541195+0000 osd.0 (osd.0) 194 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:50:18.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:16.542066+0000 osd.0 (osd.0) 195 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:17.271955+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub starts 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:17.275423+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.f scrub ok 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:16.541195+0000 osd.0 (osd.0) 194 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:16.542066+0000 osd.0 (osd.0) 195 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:17.271955+0000 osd.5 (osd.5) 354 : cluster [DBG] 5.f scrub starts 2024-02-10T08:50:18.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:17.275423+0000 osd.5 (osd.5) 355 : cluster [DBG] 5.f scrub ok 2024-02-10T08:50:18.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T08:50:18.639 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.710+0000 7f0944328700 1 -- 172.21.15.107:0/1126687187 >> v1:172.21.15.107:6789/0 conn(0x7f093c0ff710 legacy=0x7f093c101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 -- 172.21.15.107:0/1126687187 shutdown_connections 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 -- 172.21.15.107:0/1126687187 >> 172.21.15.107:0/1126687187 conn(0x7f093c0f2510 msgr2=0x7f093c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 -- 172.21.15.107:0/1126687187 shutdown_connections 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 -- 172.21.15.107:0/1126687187 wait complete. 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 librados: starting msgr at 2024-02-10T08:50:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 librados: starting objecter 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 Processor -- start 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 -- start start 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 librados: setting wanted keys 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.711+0000 7f0944328700 1 librados: calling monclient init 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f0944328700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f093c116210 con 0x7f093c0fc910 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f0944328700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f093c1173f0 con 0x7f093c103250 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f0944328700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f093c1185f0 con 0x7f093c0ff710 2024-02-10T08:50:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09418c3700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f093c0ff710 0x7f093c0f8f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37250/0 (socket says 172.21.15.107:37250) 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09418c3700 1 -- 172.21.15.107:0/934026568 learned_addr learned my addr 172.21.15.107:0/934026568 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter ms_handle_connect 0x7f093c0ff710 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter ms_handle_connect 0x7f093c0fc910 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter ms_handle_connect 0x7f093c103250 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321095887 0 0) 0x7f093c1185f0 con 0x7f093c0ff710 2024-02-10T08:50:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09280033f0 con 0x7f093c0ff710 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042619291 0 0) 0x7f093c116210 con 0x7f093c0fc910 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c1185f0 con 0x7f093c0fc910 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251439703 0 0) 0x7f093c1173f0 con 0x7f093c103250 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c116210 con 0x7f093c103250 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116740543 0 0) 0x7f09280033f0 con 0x7f093c0ff710 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f093c1173f0 con 0x7f093c0ff710 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f092c002b90 con 0x7f093c0ff710 2024-02-10T08:50:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3905201362 0 0) 0x7f093c116210 con 0x7f093c103250 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09280033f0 con 0x7f093c103250 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0934001d10 con 0x7f093c103250 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930868681 0 0) 0x7f093c1185f0 con 0x7f093c0fc910 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f093c116210 con 0x7f093c0fc910 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f09380017c0 con 0x7f093c0fc910 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1557126961 0 0) 0x7f093c1173f0 con 0x7f093c0ff710 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 >> v1:172.21.15.195:6789/0 conn(0x7f093c103250 legacy=0x7f093c1148d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 >> v1:172.21.15.107:6789/0 conn(0x7f093c0fc910 legacy=0x7f093c0f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.712+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c1197f0 con 0x7f093c0ff710 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c116400 con 0x7f093c0ff710 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f0944328700 10 client.54479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c1168f0 con 0x7f093c0ff710 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f0944328700 1 librados: init done 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f092c001a60 con 0x7f093c0ff710 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f092c0037a0 con 0x7f093c0ff710 2024-02-10T08:50:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.713+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0920004f20 con 0x7f093c0ff710 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f092c002200 con 0x7f093c0ff710 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(809..809 src has 754..809) v4 ==== 7386+0+0 (unknown 1251552056 0 0) 0x7f092c078c40 con 0x7f093c0ff710 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 10 client.54479.objecter ms_dispatch 0x7f093c001550 osd_map(809..809 src has 754..809) v4 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 3 client.54479.objecter handle_osd_map got epochs [809,809] > 0 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 3 client.54479.objecter handle_osd_map decoding full epoch 809 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.714+0000 7f09337fe700 20 client.54479.objecter dump_active .. 0 homeless 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.715+0000 7f09337fe700 10 client.54479.objecter ms_handle_connect 0x7f0928063d60 2024-02-10T08:50:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.718+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f092c04c3d0 con 0x7f093c0ff710 2024-02-10T08:50:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:18.899+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f09200051d0 con 0x7f093c0ff710 2024-02-10T08:50:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:17.113230+0000 osd.2 (osd.2) 325 : cluster [DBG] 6.f scrub starts 2024-02-10T08:50:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:17.116355+0000 osd.2 (osd.2) 326 : cluster [DBG] 6.f scrub ok 2024-02-10T08:50:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:17.587448+0000 mgr.y (mgr.14553) 688 : cluster [DBG] pgmap v1247: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.048875+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:50:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.049940+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:50:19.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:17.113230+0000 osd.2 (osd.2) 325 : cluster [DBG] 6.f scrub starts 2024-02-10T08:50:19.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:17.116355+0000 osd.2 (osd.2) 326 : cluster [DBG] 6.f scrub ok 2024-02-10T08:50:19.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:17.587448+0000 mgr.y (mgr.14553) 688 : cluster [DBG] pgmap v1247: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:19.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.048875+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:50:19.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.049940+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:50:19.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:17.113230+0000 osd.2 (osd.2) 325 : cluster [DBG] 6.f scrub starts 2024-02-10T08:50:19.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:17.116355+0000 osd.2 (osd.2) 326 : cluster [DBG] 6.f scrub ok 2024-02-10T08:50:19.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:17.587448+0000 mgr.y (mgr.14553) 688 : cluster [DBG] pgmap v1247: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:50:19.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.048875+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:50:19.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.049940+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:50:19.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.738+0000 7f09337fe700 1 -- 172.21.15.107:0/934026568 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v811) v1 ==== 113+0+0 (unknown 1030266413 0 0) 0x7f092c04a840 con 0x7f093c0ff710 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 10 librados: watch_flush enter 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 10 librados: watch_flush exit 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 20 client.54479.objecter shutdown clearing up homeless session... 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 10 client.54479.objecter successfully canceled tick 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0928063d60 legacy=0x7f0928066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:19.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.741+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 >> v1:172.21.15.107:6790/0 conn(0x7f093c0ff710 legacy=0x7f093c0f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.742+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 shutdown_connections 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.742+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 >> 172.21.15.107:0/934026568 conn(0x7f093c0f2510 msgr2=0x7f093c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.742+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 shutdown_connections 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.742+0000 7f0944328700 1 -- 172.21.15.107:0/934026568 wait complete. 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:19.742+0000 7f0944328700 1 librados: shutdown 2024-02-10T08:50:19.743 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.534739+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.535547+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.544452+0000 osd.3 (osd.3) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.544623+0000 osd.3 (osd.3) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:18.731875+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-10T08:50:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:18.900723+0000 mon.c (mon.2) 498 : audit [INF] from='client.? v1:172.21.15.107:0/934026568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:19.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:18.900979+0000 mon.a (mon.0) 3719 : audit [INF] from='client.54479 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.534739+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.535547+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.544452+0000 osd.3 (osd.3) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.544623+0000 osd.3 (osd.3) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:18.731875+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:18.900723+0000 mon.c (mon.2) 498 : audit [INF] from='client.? v1:172.21.15.107:0/934026568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:20.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:18.900979+0000 mon.a (mon.0) 3719 : audit [INF] from='client.54479 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.534739+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.535547+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.544452+0000 osd.3 (osd.3) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.544623+0000 osd.3 (osd.3) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:18.731875+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:18.900723+0000 mon.c (mon.2) 498 : audit [INF] from='client.? v1:172.21.15.107:0/934026568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:20.170 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:18.900979+0000 mon.a (mon.0) 3719 : audit [INF] from='client.54479 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:469 237.10 237:08000000:test-rados-api-smithi107-112490-107:::head [pgnls start_epoch 633] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_overlay+known_if_redirected e634)", 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:29.217101+0000", 2024-02-10T08:50:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 231.06373575000001, 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97975531599999999, 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 469 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217101+0000", 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217101+0000", 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217099+0000", 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2387e-05 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217111+0000", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0763e-05 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217122+0000", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6590000000000002e-06 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.217132+0000", 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034567439999999999 2024-02-10T08:50:20.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.220589+0000", 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7890000000000006e-06 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.220598+0000", 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.975641745 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:30.196240+0000", 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0877000000000002e-05 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:30.196281+0000", 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057574000000000004 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:30.196857+0000", 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97975531599999999 2024-02-10T08:50:20.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(234.0s0 634/633 [PushReplyOp(234:072f2071:test-rados-api-smithi107-112490-107::foo:head)])", 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:29.225181+0000", 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 231.055655831, 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040020120000000001, 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225181+0000", 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225181+0000", 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225181+0000", 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225181+0000", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2289999999999999e-05 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225194+0000", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2800000000000003e-06 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.225199+0000", 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038009459999999999 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.229000+0000", 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018349600000000001 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.229183+0000", 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040020120000000001 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.292 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(234.0s0 634/633 [PushReplyOp(234:072f2071:test-rados-api-smithi107-112490-107::foo:2)])", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:29.310323+0000", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 230.970514654, 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041754299999999999, 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310323+0000", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310323+0000", 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310323+0000", 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310323+0000", 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0566e-05 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310333+0000", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5750000000000002e-06 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.310338+0000", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040100470000000001 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.314348+0000", 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150242 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.314498+0000", 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041754299999999999 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(234.0s0 634/633 [PushReplyOp(234:16c7d2d7:test-rados-api-smithi107-112490-107::bar:head)])", 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:29.396041+0000", 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 230.88479593700001, 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042070739999999999, 2024-02-10T08:50:20.296 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396041+0000", 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396041+0000", 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396041+0000", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396041+0000", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0101e-05 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396051+0000", 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2429999999999999e-06 2024-02-10T08:50:20.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.396056+0000", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004065468 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.400121+0000", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127262 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.400248+0000", 2024-02-10T08:50:20.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042070739999999999 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(234.0s0 634/633 [PushReplyOp(234:16c7d2d7:test-rados-api-smithi107-112490-107::bar:2)])", 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:29.481966+0000", 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 230.79887125799999, 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041826779999999996, 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481966+0000", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481966+0000", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481966+0000", 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481966+0000", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2889999999999998e-06 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481974+0000", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9720000000000001e-06 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.481977+0000", 2024-02-10T08:50:20.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004074823 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.486052+0000", 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6594000000000004e-05 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:29.486149+0000", 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041826779999999996 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:645 239.16 239:6c126df5:test-rados-api-smithi107-112490-111::bar:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e651)", 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:46:46.184179+0000", 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 214.09665809800001, 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031797091, 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 645 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.304 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184179+0000", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184179+0000", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184177+0000", 2024-02-10T08:50:20.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9439999999999999e-06 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184184+0000", 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1738e-05 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184196+0000", 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0010000000000001e-06 2024-02-10T08:50:20.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184204+0000", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4576000000000002e-05 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184239+0000", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5514999999999999e-05 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184304+0000", 2024-02-10T08:50:20.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9293000000000001e-05 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,2", 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184383+0000", 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027095899999999998 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:20.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.184654+0000", 2024-02-10T08:50:20.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031241549 2024-02-10T08:50:20.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.215896+0000", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9454e-05 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.215925+0000", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7555999999999998e-05 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.215953+0000", 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3288000000000001e-05 2024-02-10T08:50:20.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:46:46.215976+0000", 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031797091 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(4.1b 673/672 [PushReplyOp(4:db9d130d:gc::gc.24:head)])", 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:47:08.186942+0000", 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 192.093895449, 2024-02-10T08:50:20.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049856370000000002, 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186942+0000", 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186942+0000", 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186941+0000", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5340000000000001e-06 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186946+0000", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3110000000000002e-06 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186952+0000", 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5090000000000001e-06 2024-02-10T08:50:20.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.186957+0000", 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047244979999999997 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.191682+0000", 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024549400000000002 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:47:08.191927+0000", 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049856370000000002 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1928 247.1b 247:dae49067:test-rados-api-smithi107-112490-127::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected e723)", 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.162497+0000", 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 137.11834006999999, 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032104557999999998, 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.317 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1928 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.162497+0000", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.162497+0000", 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-02-10T08:50:20.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.162494+0000", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045661230000000001 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.167060+0000", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8858000000000001e-05 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.167079+0000", 2024-02-10T08:50:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5550000000000003e-06 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.167088+0000", 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3318e-05 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.167141+0000", 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7770000000000002e-06 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:20.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.167146+0000", 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015495769999999999 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.168695+0000", 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000219688 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.168915+0000", 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035897839999999999 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172505+0000", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8836000000000003e-05 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172574+0000", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013079000000000001 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,3", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172704+0000", 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0051516050000000001 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.177856+0000", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011044534999999999 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.188901+0000", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055947840000000002 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194495+0000", 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6934000000000001e-05 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194552+0000", 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9404000000000003e-05 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194602+0000", 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032104557999999998 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.324 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(osd.4.0:23 247.1b e722/719 ondisk, result = 0)", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.172803+0000", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 137.108033901, 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034096320000000001, 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172803+0000", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172803+0000", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172802+0000", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0900000000000001e-06 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172808+0000", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1077000000000001e-05 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172819+0000", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1499999999999999e-06 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172827+0000", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033643259999999999 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176191+0000", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3559999999999999e-06 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176201+0000", 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2404e-05 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176213+0000", 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034096320000000001 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(osd.4.0:23 247.1b e723/719 ondisk, result = 0)", 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.172842+0000", 2024-02-10T08:50:20.328 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 137.107995203, 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003440593, 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172842+0000", 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172842+0000", 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172841+0000", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.732e-06 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172845+0000", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3540000000000002e-06 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172852+0000", 2024-02-10T08:50:20.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7920000000000001e-05 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.172870+0000", 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033651499999999999 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176235+0000", 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5199999999999997e-06 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176241+0000", 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1758999999999998e-05 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.176283+0000", 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003440593 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:1929 247.1b 247:dae49067:test-rados-api-smithi107-112490-127::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e723)", 2024-02-10T08:50:20.332 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:03.194790+0000", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 137.08604731599999, 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017492506000000001, 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 1929 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194790+0000", 2024-02-10T08:50:20.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194790+0000", 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194788+0000", 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.455e-06 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194794+0000", 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0543000000000001e-05 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194805+0000", 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2970000000000003e-06 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194810+0000", 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6421000000000002e-05 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194847+0000", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000113194 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.194960+0000", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017121925 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.212082+0000", 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112445 2024-02-10T08:50:20.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.212194+0000", 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8120000000000003e-05 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:03.212282+0000", 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017492506000000001 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2132 249.6 249:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:21.167789+0000", 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 119.11304864500001, 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003546614, 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2132 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167789+0000", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167789+0000", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167785+0000", 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2223000000000001e-05 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167807+0000", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8668000000000002e-05 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167826+0000", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8859999999999993e-06 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167835+0000", 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6314000000000003e-05 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167891+0000", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9070000000000001e-06 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.167898+0000", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002260111 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.170158+0000", 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119533 2024-02-10T08:50:20.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.170277+0000", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000145282 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.170423+0000", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045782899999999998 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.170881+0000", 2024-02-10T08:50:20.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036786300000000001 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.171248+0000", 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1628000000000003e-05 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.171280+0000", 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1090999999999998e-05 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.171311+0000", 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4128e-05 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:21.171335+0000", 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003546614 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(6.15 REQUEST e744)", 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:28.805016+0000", 2024-02-10T08:50:20.344 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 111.475821, 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003357493, 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805016+0000", 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805016+0000", 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:20.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805016+0000", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9790000000000004e-06 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805020+0000", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4459e-05 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805034+0000", 2024-02-10T08:50:20.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805042+0000", 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5266000000000002e-05 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805098+0000", 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5839999999999995e-06 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:20.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.805106+0000", 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003200179 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.808306+0000", 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9332e-05 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.808336+0000", 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8114000000000001e-05 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:28.808374+0000", 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003357493 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2152 249.6 249:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e744)", 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:31.193886+0000", 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 109.08695174099999, 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063466119999999997, 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2152 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193886+0000", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193886+0000", 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999957 2024-02-10T08:50:20.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193881+0000", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7458999999999999e-05 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193908+0000", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4958999999999999e-05 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193933+0000", 2024-02-10T08:50:20.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7110000000000001e-06 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.193942+0000", 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7084999999999996e-05 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.194039+0000", 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4630999999999997e-05 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.194124+0000", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119894 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.194244+0000", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00084338600000000001 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.195087+0000", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5367999999999998e-05 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.195113+0000", 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050589069999999996 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.200171+0000", 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0457999999999999e-05 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.200202+0000", 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0230999999999999e-05 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:31.200232+0000", 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063466119999999997 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2287 250.1c 250:38000000:test-rados-api-smithi107-112490-133:::head [pgnls start_epoch 758] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e758)", 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:45.242331+0000", 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 95.038506002000005, 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046972730000000001, 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2287 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242331+0000", 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242331+0000", 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242328+0000", 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2683e-05 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242340+0000", 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6497000000000001e-05 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242357+0000", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2439999999999993e-06 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242365+0000", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4087999999999999e-05 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242439+0000", 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7349999999999999e-06 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.242446+0000", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044105969999999996 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.246856+0000", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5853999999999999e-05 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.246892+0000", 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136234 2024-02-10T08:50:20.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:45.247029+0000", 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046972730000000001 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(234.1s2 763/633 ECSubWrite(tid=794, reqid=client.48721.0:2446, at_version=763'11, trim_to=0'0, roll_forward_to=749'10))", 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:48:50.191159+0000", 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 90.089678282999998, 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050749979999999998, 2024-02-10T08:50:20.360 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191159+0000", 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191159+0000", 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191158+0000", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0103999999999998e-05 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191188+0000", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.653e-06 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191197+0000", 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1030000000000002e-06 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.191205+0000", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00094390499999999998 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.192149+0000", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1968e-05 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.192161+0000", 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00405332 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.196214+0000", 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9593999999999999e-05 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:48:50.196234+0000", 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050749979999999998 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:2866 234.0s0 234:1a0e16ef:test-rados-api-smithi107-112490-137::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e774)", 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:07.189602+0000", 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 73.091235148999999, 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070896520000000001, 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 2866 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189602+0000", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189602+0000", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189599+0000", 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3998999999999999e-05 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189613+0000", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8734e-05 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189641+0000", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0468999999999999e-05 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.189652+0000", 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002228299 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.191880+0000", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022334199999999999 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.192104+0000", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00065398799999999999 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.192758+0000", 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034312520000000001 2024-02-10T08:50:20.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.196189+0000", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040888200000000001 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.196598+0000", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4066000000000004e-05 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.196692+0000", 2024-02-10T08:50:20.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070896520000000001 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(234.5s1 774/633 ECSubWrite(tid=709, reqid=client.48721.0:2867, at_version=774'17, trim_to=0'0, roll_forward_to=749'16))", 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:07.201076+0000", 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 73.079761293999994, 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057543359999999997, 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201076+0000", 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201076+0000", 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201075+0000", 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.021e-05 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201105+0000", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.719e-05 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201122+0000", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1689999999999992e-06 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201130+0000", 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2692000000000001e-05 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201203+0000", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0297e-05 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.201213+0000", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055836039999999998 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.206797+0000", 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3609000000000003e-05 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:07.206830+0000", 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057543359999999997 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.0.8:5 234.0s0 234:1a0e16ef:test-rados-api-smithi107-112490-137::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e777)", 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:10.217064+0000", 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 70.063773702000006, 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027429669999999998, 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.374 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217064+0000", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217064+0000", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217063+0000", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9432999999999998e-05 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217082+0000", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1889999999999998e-06 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217087+0000", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7169999999999999e-06 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217092+0000", 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104947 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217197+0000", 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5260000000000004e-06 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.217202+0000", 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014851090000000001 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.218687+0000", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7710000000000001e-05 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.218744+0000", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010621649999999999 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:10.219807+0000", 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027429669999999998 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.378 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.48721.0:3150 254.1 254:872209dc:test-rados-api-smithi107-112490-141::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e795)", 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-02-10T08:49:34.218154+0000", 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.062683382000003, 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015703894999999999, 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.48721", 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "v1:172.21.15.107:0/3092462557", 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 3150 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.379 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218154+0000", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218154+0000", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999967 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218150+0000", 2024-02-10T08:50:20.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3506000000000001e-05 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218164+0000", 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9970999999999999e-05 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218184+0000", 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8919999999999995e-06 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-02-10T08:50:20.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.218194+0000", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020842740000000001 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.220278+0000", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9550000000000004e-06 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.220288+0000", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026811750000000001 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-02-10T08:50:20.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.222969+0000", 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013800499999999999 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.223107+0000", 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017454600000000001 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.223282+0000", 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00087825700000000002 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.224160+0000", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010511330000000001 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.225211+0000", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085450419999999992 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.233756+0000", 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5915999999999998e-05 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.233812+0000", 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5732000000000002e-05 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-02-10T08:49:34.233858+0000", 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015703894999999999 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-02-10T08:50:20.385 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-02-10T08:50:20.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-10T08:50:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:20.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.506870+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:20.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.507262+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:20.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.574251+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.575504+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.587937+0000 mgr.y (mgr.14553) 689 : cluster [DBG] pgmap v1249: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:19.737751+0000 mon.a (mon.0) 3720 : audit [INF] from='client.54479 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.737841+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.909532+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:50:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:19.911481+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:50:20.752 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.506870+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.507262+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.574251+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.575504+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.587937+0000 mgr.y (mgr.14553) 689 : cluster [DBG] pgmap v1249: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:19.737751+0000 mon.a (mon.0) 3720 : audit [INF] from='client.54479 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.737841+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.909532+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:50:20.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:19.911481+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:50:20.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.276224+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:20.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.276555+0000 osd.1 (osd.1) 377 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:20.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.339451+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:50:20.880 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.342721+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:50:21.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.276224+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.276555+0000 osd.1 (osd.1) 377 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.339451+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.342721+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.506870+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.507262+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.574251+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.575504+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.587937+0000 mgr.y (mgr.14553) 689 : cluster [DBG] pgmap v1249: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:21.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:19.737751+0000 mon.a (mon.0) 3720 : audit [INF] from='client.54479 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.737841+0000 mon.a (mon.0) 3721 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.909532+0000 osd.7 (osd.7) 344 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:19.911481+0000 osd.7 (osd.7) 345 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.276224+0000 osd.1 (osd.1) 376 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.276555+0000 osd.1 (osd.1) 377 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.339451+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:50:21.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.342721+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:50:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.745085+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-10T08:50:21.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.883386+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:50:21.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:20.886478+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:50:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.745085+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-10T08:50:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.883386+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:50:21.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:20.886478+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:50:22.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.745085+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-10T08:50:22.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.883386+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:50:22.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:20.886478+0000 osd.7 (osd.7) 347 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 -- 172.21.15.107:0/3995229538 >> v1:172.21.15.107:6789/0 conn(0x7fb970103250 legacy=0x7fb9701056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 -- 172.21.15.107:0/3995229538 shutdown_connections 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 -- 172.21.15.107:0/3995229538 >> 172.21.15.107:0/3995229538 conn(0x7fb9700f2510 msgr2=0x7fb9700f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 -- 172.21.15.107:0/3995229538 shutdown_connections 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 -- 172.21.15.107:0/3995229538 wait complete. 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 librados: starting msgr at 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 librados: starting objecter 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.623+0000 7fb975041700 1 Processor -- start 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 -- start start 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 librados: setting wanted keys 2024-02-10T08:50:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 librados: calling monclient init 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb970116340 con 0x7fb9700fc910 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb970117520 con 0x7fb970103250 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb975041700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb970118720 con 0x7fb9700ff710 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb96e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb9700ff710 0x7fb9700f9340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34814/0 (socket says 172.21.15.107:34814) 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb96e59c700 1 -- 172.21.15.107:0/459386678 learned_addr learned my addr 172.21.15.107:0/459386678 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb96ed9d700 1 --1- 172.21.15.107:0/459386678 >> v1:172.21.15.107:6789/0 conn(0x7fb9700fc910 0x7fb9700f8c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40798/0 (socket says 172.21.15.107:40798) 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter ms_handle_connect 0x7fb9700ff710 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter ms_handle_connect 0x7fb9700fc910 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372313346 0 0) 0x7fb970118720 con 0x7fb9700ff710 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958003030 con 0x7fb9700ff710 2024-02-10T08:50:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter ms_handle_connect 0x7fb970103250 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131443062 0 0) 0x7fb970116340 con 0x7fb9700fc910 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970118720 con 0x7fb9700fc910 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755031600 0 0) 0x7fb970117520 con 0x7fb970103250 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970116340 con 0x7fb970103250 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 208078356 0 0) 0x7fb958003030 con 0x7fb9700ff710 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb970117520 con 0x7fb9700ff710 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb9600018e0 con 0x7fb9700ff710 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139860894 0 0) 0x7fb970118720 con 0x7fb9700fc910 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb958003030 con 0x7fb9700fc910 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.624+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb964001960 con 0x7fb9700fc910 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4227705029 0 0) 0x7fb970116340 con 0x7fb970103250 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb970118720 con 0x7fb970103250 2024-02-10T08:50:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb968002250 con 0x7fb970103250 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988097259 0 0) 0x7fb970117520 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 >> v1:172.21.15.195:6789/0 conn(0x7fb970103250 legacy=0x7fb970114a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 >> v1:172.21.15.107:6789/0 conn(0x7fb9700fc910 legacy=0x7fb9700f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970119920 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb970116530 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb975041700 10 client.54485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9701169a0 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb975041700 1 librados: init done 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fb960002310 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.625+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fb960005a00 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.626+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c004f20 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.626+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fb960001e70 con 0x7fb9700ff710 2024-02-10T08:50:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(812..812 src has 754..812) v4 ==== 7386+0+0 (unknown 1795769617 0 0) 0x7fb960078930 con 0x7fb9700ff710 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 10 client.54485.objecter ms_dispatch 0x7fb970001550 osd_map(812..812 src has 754..812) v4 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 3 client.54485.objecter handle_osd_map got epochs [812,812] > 0 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 3 client.54485.objecter handle_osd_map decoding full epoch 812 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 20 client.54485.objecter dump_active .. 0 homeless 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.627+0000 7fb957fff700 10 client.54485.objecter ms_handle_connect 0x7fb958063c20 2024-02-10T08:50:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.631+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb96004c000 con 0x7fb9700ff710 2024-02-10T08:50:22.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:22.807+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fb95c0059e0 con 0x7fb9700ff710 2024-02-10T08:50:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:21.588877+0000 mgr.y (mgr.14553) 690 : cluster [DBG] pgmap v1252: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:50:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:21.891718+0000 mgr.y (mgr.14553) 691 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.407722+0000 osd.5 (osd.5) 358 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:50:23.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.411913+0000 osd.5 (osd.5) 359 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:50:23.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:21.588877+0000 mgr.y (mgr.14553) 690 : cluster [DBG] pgmap v1252: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:50:23.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:21.891718+0000 mgr.y (mgr.14553) 691 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:23.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.407722+0000 osd.5 (osd.5) 358 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:50:23.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.411913+0000 osd.5 (osd.5) 359 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:50:23.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:21.588877+0000 mgr.y (mgr.14553) 690 : cluster [DBG] pgmap v1252: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:50:23.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:21.891718+0000 mgr.y (mgr.14553) 691 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:23.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.407722+0000 osd.5 (osd.5) 358 : cluster [DBG] 16.1f scrub starts 2024-02-10T08:50:23.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.411913+0000 osd.5 (osd.5) 359 : cluster [DBG] 16.1f scrub ok 2024-02-10T08:50:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.758+0000 7fb957fff700 1 -- 172.21.15.107:0/459386678 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v813) v1 ==== 133+0+0 (unknown 980905142 0 0) 0x7fb96004a410 con 0x7fb9700ff710 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.761+0000 7fb975041700 10 librados: watch_flush enter 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.761+0000 7fb975041700 10 librados: watch_flush exit 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.761+0000 7fb975041700 20 client.54485.objecter shutdown clearing up homeless session... 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.762+0000 7fb975041700 10 client.54485.objecter successfully canceled tick 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.762+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 >> v1:172.21.15.107:6800/2634842009 conn(0x7fb958063c20 legacy=0x7fb9580660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.762+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 >> v1:172.21.15.107:6790/0 conn(0x7fb9700ff710 legacy=0x7fb9700f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.763+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 shutdown_connections 2024-02-10T08:50:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.763+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 >> 172.21.15.107:0/459386678 conn(0x7fb9700f2510 msgr2=0x7fb9701066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.763+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 shutdown_connections 2024-02-10T08:50:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.763+0000 7fb975041700 1 -- 172.21.15.107:0/459386678 wait complete. 2024-02-10T08:50:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:23.763+0000 7fb975041700 1 librados: shutdown 2024-02-10T08:50:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-10T08:50:24.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:22.808219+0000 mon.c (mon.2) 499 : audit [INF] from='client.? v1:172.21.15.107:0/459386678' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:22.808540+0000 mon.a (mon.0) 3723 : audit [INF] from='client.54485 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.865862+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:24.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.866100+0000 osd.6 (osd.6) 345 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:24.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.966322+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:50:24.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:22.967210+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:22.808219+0000 mon.c (mon.2) 499 : audit [INF] from='client.? v1:172.21.15.107:0/459386678' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:22.808540+0000 mon.a (mon.0) 3723 : audit [INF] from='client.54485 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.865862+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.866100+0000 osd.6 (osd.6) 345 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.966322+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:22.967210+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:22.808219+0000 mon.c (mon.2) 499 : audit [INF] from='client.? v1:172.21.15.107:0/459386678' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.083 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:22.808540+0000 mon.a (mon.0) 3723 : audit [INF] from='client.54485 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T08:50:24.084 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.865862+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:24.084 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.866100+0000 osd.6 (osd.6) 345 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:24.084 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.966322+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:50:24.084 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:22.967210+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:50:24.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T08:50:24.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.017404+0000 osd.2 (osd.2) 329 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:50:24.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.018233+0000 osd.2 (osd.2) 330 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:50:24.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.065675+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.e scrub starts 2024-02-10T08:50:24.764 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.068539+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.e scrub ok 2024-02-10T08:50:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.589407+0000 mgr.y (mgr.14553) 692 : cluster [DBG] pgmap v1253: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:23.755064+0000 mon.a (mon.0) 3724 : audit [INF] from='client.54485 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.755148+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-10T08:50:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.864291+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:50:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.865324+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:50:24.804 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:25.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.017404+0000 osd.2 (osd.2) 329 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:50:25.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.018233+0000 osd.2 (osd.2) 330 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:50:25.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.065675+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.e scrub starts 2024-02-10T08:50:25.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.068539+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.e scrub ok 2024-02-10T08:50:25.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.589407+0000 mgr.y (mgr.14553) 692 : cluster [DBG] pgmap v1253: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:23.755064+0000 mon.a (mon.0) 3724 : audit [INF] from='client.54485 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.755148+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.864291+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.865324+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.995679+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:23.997056+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:24.466755+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.18 deep-scrub starts 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:24.469775+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.18 deep-scrub ok 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.995679+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:23.997056+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:50:25.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:24.466755+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.18 deep-scrub starts 2024-02-10T08:50:25.023 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:24.469775+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.18 deep-scrub ok 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.017404+0000 osd.2 (osd.2) 329 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.018233+0000 osd.2 (osd.2) 330 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.065675+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.e scrub starts 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.068539+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.e scrub ok 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.589407+0000 mgr.y (mgr.14553) 692 : cluster [DBG] pgmap v1253: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:23.755064+0000 mon.a (mon.0) 3724 : audit [INF] from='client.54485 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.755148+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.864291+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:50:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.865324+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:50:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.995679+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:50:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:23.997056+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:50:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:24.466755+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.18 deep-scrub starts 2024-02-10T08:50:25.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:24.469775+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.18 deep-scrub ok 2024-02-10T08:50:25.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:25] "GET /metrics HTTP/1.1" 200 186408 "" "Prometheus/2.33.4" 2024-02-10T08:50:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:25.213406+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.a scrub starts 2024-02-10T08:50:25.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:25.216525+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.a scrub ok 2024-02-10T08:50:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:25.213406+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.a scrub starts 2024-02-10T08:50:25.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:25.216525+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.a scrub ok 2024-02-10T08:50:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:25.213406+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.a scrub starts 2024-02-10T08:50:26.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:25.216525+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.a scrub ok 2024-02-10T08:50:26.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 -- 172.21.15.107:0/3159987838 >> v1:172.21.15.107:6790/0 conn(0x7f0d700ff720 legacy=0x7f0d70101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 -- 172.21.15.107:0/3159987838 shutdown_connections 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 -- 172.21.15.107:0/3159987838 >> 172.21.15.107:0/3159987838 conn(0x7f0d7006c090 msgr2=0x7f0d7006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 -- 172.21.15.107:0/3159987838 shutdown_connections 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 -- 172.21.15.107:0/3159987838 wait complete. 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 librados: starting msgr at 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.626+0000 7f0d75602700 1 librados: starting objecter 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 Processor -- start 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 -- start start 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 librados: setting wanted keys 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 librados: calling monclient init 2024-02-10T08:50:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d70105670 con 0x7f0d700fc960 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d700fbf10 con 0x7f0d700ff720 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d75602700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d700fc0a0 con 0x7f0d70103320 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d6effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0d700fc960 0x7f0d700fb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40822/0 (socket says 172.21.15.107:40822) 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d6effd700 1 -- 172.21.15.107:0/279809893 learned_addr learned my addr 172.21.15.107:0/279809893 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter ms_handle_connect 0x7f0d700fc960 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter ms_handle_connect 0x7f0d70103320 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter ms_handle_connect 0x7f0d700ff720 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.627+0000 7f0d57fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506767040 0 0) 0x7f0d700fc0a0 con 0x7f0d70103320 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d580038e0 con 0x7f0d70103320 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269201572 0 0) 0x7f0d70105670 con 0x7f0d700fc960 2024-02-10T08:50:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d700fc0a0 con 0x7f0d700fc960 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208494876 0 0) 0x7f0d700fbf10 con 0x7f0d700ff720 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70105670 con 0x7f0d700ff720 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211885015 0 0) 0x7f0d580038e0 con 0x7f0d70103320 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d700fbf10 con 0x7f0d70103320 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0d68001b80 con 0x7f0d70103320 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036444753 0 0) 0x7f0d700fc0a0 con 0x7f0d700fc960 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d580038e0 con 0x7f0d700fc960 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0d640019f0 con 0x7f0d700fc960 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221859346 0 0) 0x7f0d70105670 con 0x7f0d700ff720 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d700fc0a0 con 0x7f0d700ff720 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0d60002d90 con 0x7f0d700ff720 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295469830 0 0) 0x7f0d700fbf10 con 0x7f0d70103320 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 >> v1:172.21.15.195:6789/0 conn(0x7f0d700ff720 legacy=0x7f0d701195a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 >> v1:172.21.15.107:6789/0 conn(0x7f0d700fc960 legacy=0x7f0d700fb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700fc230 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7011e3f0 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d75602700 10 client.54491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7011e740 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d75602700 1 librados: init done 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0d68002180 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.628+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0d68003620 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.629+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c004f20 con 0x7f0d70103320 2024-02-10T08:50:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.629+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0d68001ce0 con 0x7f0d70103320 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(813..813 src has 754..813) v4 ==== 7386+0+0 (unknown 901430569 0 0) 0x7f0d680763e0 con 0x7f0d70103320 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 10 client.54491.objecter ms_dispatch 0x7f0d70001550 osd_map(813..813 src has 754..813) v4 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 3 client.54491.objecter handle_osd_map got epochs [813,813] > 0 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 3 client.54491.objecter handle_osd_map decoding full epoch 813 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 20 client.54491.objecter dump_active .. 0 homeless 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.630+0000 7f0d57fff700 10 client.54491.objecter ms_handle_connect 0x7f0d58063c20 2024-02-10T08:50:26.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.634+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0d68049ad0 con 0x7f0d70103320 2024-02-10T08:50:26.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:26.812+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f0d5c0051d0 con 0x7f0d70103320 2024-02-10T08:50:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:25.312369+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:50:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:25.314510+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:50:27.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:25.589956+0000 mgr.y (mgr.14553) 693 : cluster [DBG] pgmap v1255: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:25.312369+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:50:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:25.314510+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:50:27.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:25.589956+0000 mgr.y (mgr.14553) 693 : cluster [DBG] pgmap v1255: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:25.312369+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.0 scrub starts 2024-02-10T08:50:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:25.314510+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.0 scrub ok 2024-02-10T08:50:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:25.589956+0000 mgr.y (mgr.14553) 693 : cluster [DBG] pgmap v1255: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:27.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.779+0000 7f0d57fff700 1 -- 172.21.15.107:0/279809893 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v814) v1 ==== 113+0+0 (unknown 2830115581 0 0) 0x7f0d68047f40 con 0x7f0d70103320 2024-02-10T08:50:27.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 10 librados: watch_flush enter 2024-02-10T08:50:27.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 10 librados: watch_flush exit 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 20 client.54491.objecter shutdown clearing up homeless session... 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 10 client.54491.objecter successfully canceled tick 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0d58063c20 legacy=0x7f0d580660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 >> v1:172.21.15.107:6790/0 conn(0x7f0d70103320 legacy=0x7f0d7011ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 shutdown_connections 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 >> 172.21.15.107:0/279809893 conn(0x7f0d7006c090 msgr2=0x7f0d7006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 shutdown_connections 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 -- 172.21.15.107:0/279809893 wait complete. 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:27.783+0000 7f0d75602700 1 librados: shutdown 2024-02-10T08:50:27.785 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T08:50:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:26.814593+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.107:0/279809893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:26.814826+0000 mon.a (mon.0) 3726 : audit [INF] from='client.54491 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:26.814593+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.107:0/279809893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:26.814826+0000 mon.a (mon.0) 3726 : audit [INF] from='client.54491 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:26.814593+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.107:0/279809893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:26.814826+0000 mon.a (mon.0) 3726 : audit [INF] from='client.54491 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:28.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-10T08:50:28.867 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:27.026595+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:50:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:27.029493+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:50:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:27.590784+0000 mgr.y (mgr.14553) 694 : cluster [DBG] pgmap v1256: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:27.780915+0000 mon.a (mon.0) 3727 : audit [INF] from='client.54491 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:28.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:27.780972+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-10T08:50:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:27.026595+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:50:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:27.029493+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:50:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:27.590784+0000 mgr.y (mgr.14553) 694 : cluster [DBG] pgmap v1256: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:27.780915+0000 mon.a (mon.0) 3727 : audit [INF] from='client.54491 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:27.780972+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-10T08:50:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:27.026595+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:50:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:27.029493+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:50:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:27.590784+0000 mgr.y (mgr.14553) 694 : cluster [DBG] pgmap v1256: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:27.780915+0000 mon.a (mon.0) 3727 : audit [INF] from='client.54491 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:29.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:27.780972+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-10T08:50:30.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:28.350592+0000 osd.3 (osd.3) 377 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:50:30.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:28.351600+0000 osd.3 (osd.3) 378 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:50:30.136 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:28.350592+0000 osd.3 (osd.3) 377 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:50:30.137 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:28.351600+0000 osd.3 (osd.3) 378 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:50:30.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:28.350592+0000 osd.3 (osd.3) 377 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:50:30.194 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:28.351600+0000 osd.3 (osd.3) 378 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:50:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.705+0000 7fd13dc5e700 1 -- 172.21.15.107:0/331435166 >> v1:172.21.15.107:6790/0 conn(0x7fd138100810 legacy=0x7fd138102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.705+0000 7fd13dc5e700 1 -- 172.21.15.107:0/331435166 shutdown_connections 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.705+0000 7fd13dc5e700 1 -- 172.21.15.107:0/331435166 >> 172.21.15.107:0/331435166 conn(0x7fd1380f3630 msgr2=0x7fd1380f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.705+0000 7fd13dc5e700 1 -- 172.21.15.107:0/331435166 shutdown_connections 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.705+0000 7fd13dc5e700 1 -- 172.21.15.107:0/331435166 wait complete. 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 librados: starting msgr at 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 librados: starting objecter 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 Processor -- start 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 -- start start 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 librados: setting wanted keys 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 librados: calling monclient init 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd138117430 con 0x7fd1380fda30 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd138118630 con 0x7fd138100810 2024-02-10T08:50:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd13dc5e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd138119830 con 0x7fd138104320 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd137fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd138104320 0x7fd138115af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43446/0 (socket says 172.21.15.107:43446) 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd137fff700 1 -- 172.21.15.107:0/1270765770 learned_addr learned my addr 172.21.15.107:0/1270765770 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd1377fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1380fda30 0x7fd138100db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37582/0 (socket says 172.21.15.107:37582) 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd134ff9700 10 client.?.objecter ms_handle_connect 0x7fd138104320 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd134ff9700 10 client.?.objecter ms_handle_connect 0x7fd1380fda30 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.706+0000 7fd134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 10 client.?.objecter ms_handle_connect 0x7fd138100810 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217201730 0 0) 0x7fd138119830 con 0x7fd138104320 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124003030 con 0x7fd138104320 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679084270 0 0) 0x7fd138117430 con 0x7fd1380fda30 2024-02-10T08:50:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd138119830 con 0x7fd1380fda30 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828095750 0 0) 0x7fd138118630 con 0x7fd138100810 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd138117430 con 0x7fd138100810 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237366883 0 0) 0x7fd124003030 con 0x7fd138104320 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd138118630 con 0x7fd138104320 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd130001d70 con 0x7fd138104320 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1686317686 0 0) 0x7fd138119830 con 0x7fd1380fda30 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd124003030 con 0x7fd1380fda30 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd1200017c0 con 0x7fd1380fda30 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 403356974 0 0) 0x7fd138117430 con 0x7fd138100810 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd138119830 con 0x7fd138100810 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd128001fa0 con 0x7fd138100810 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103820232 0 0) 0x7fd138118630 con 0x7fd138104320 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 >> v1:172.21.15.195:6789/0 conn(0x7fd138100810 legacy=0x7fd1381014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 >> v1:172.21.15.107:6789/0 conn(0x7fd1380fda30 legacy=0x7fd138100db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13811aa30 con 0x7fd138104320 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd138117620 con 0x7fd138104320 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd13dc5e700 10 client.54497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd138117b10 con 0x7fd138104320 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd13dc5e700 1 librados: init done 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fd130002740 con 0x7fd138104320 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.707+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fd130005e70 con 0x7fd138104320 2024-02-10T08:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.708+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd118004f20 con 0x7fd138104320 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.708+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fd1300028b0 con 0x7fd138104320 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(814..814 src has 754..814) v4 ==== 7386+0+0 (unknown 3597579617 0 0) 0x7fd130078e00 con 0x7fd138104320 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 10 client.54497.objecter ms_dispatch 0x7fd138001550 osd_map(814..814 src has 754..814) v4 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 3 client.54497.objecter handle_osd_map got epochs [814,814] > 0 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 3 client.54497.objecter handle_osd_map decoding full epoch 814 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 20 client.54497.objecter dump_active .. 0 homeless 2024-02-10T08:50:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.709+0000 7fd134ff9700 10 client.54497.objecter ms_handle_connect 0x7fd124063d10 2024-02-10T08:50:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.713+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd13004c490 con 0x7fd138104320 2024-02-10T08:50:30.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:30.894+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fd1180059e0 con 0x7fd138104320 2024-02-10T08:50:31.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:29.010826+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:29.014526+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:29.348559+0000 osd.3 (osd.3) 379 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:29.349500+0000 osd.3 (osd.3) 380 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:29.591225+0000 mgr.y (mgr.14553) 695 : cluster [DBG] pgmap v1258: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:29.010826+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:29.014526+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:29.348559+0000 osd.3 (osd.3) 379 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:50:31.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:29.349500+0000 osd.3 (osd.3) 380 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:50:31.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:29.591225+0000 mgr.y (mgr.14553) 695 : cluster [DBG] pgmap v1258: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:29.010826+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:50:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:29.014526+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:50:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:29.348559+0000 osd.3 (osd.3) 379 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:50:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:29.349500+0000 osd.3 (osd.3) 380 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:50:31.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:29.591225+0000 mgr.y (mgr.14553) 695 : cluster [DBG] pgmap v1258: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:31.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.805+0000 7fd134ff9700 1 -- 172.21.15.107:0/1270765770 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v816) v1 ==== 133+0+0 (unknown 3205798300 0 0) 0x7fd13004a8a0 con 0x7fd138104320 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.807+0000 7fd13dc5e700 10 librados: watch_flush enter 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.807+0000 7fd13dc5e700 10 librados: watch_flush exit 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.807+0000 7fd13dc5e700 20 client.54497.objecter shutdown clearing up homeless session... 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.807+0000 7fd13dc5e700 10 client.54497.objecter successfully canceled tick 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.808+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 >> v1:172.21.15.107:6800/2634842009 conn(0x7fd124063d10 legacy=0x7fd1240661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.808+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 >> v1:172.21.15.107:6790/0 conn(0x7fd138104320 legacy=0x7fd138115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.808+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 shutdown_connections 2024-02-10T08:50:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.808+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 >> 172.21.15.107:0/1270765770 conn(0x7fd1380f3630 msgr2=0x7fd138107b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.811+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 shutdown_connections 2024-02-10T08:50:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.811+0000 7fd13dc5e700 1 -- 172.21.15.107:0/1270765770 wait complete. 2024-02-10T08:50:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:31.811+0000 7fd13dc5e700 1 librados: shutdown 2024-02-10T08:50:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-10T08:50:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.039337+0000 osd.4 (osd.4) 103 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:50:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.042563+0000 osd.4 (osd.4) 104 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.347923+0000 osd.3 (osd.3) 381 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.348971+0000 osd.3 (osd.3) 382 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.757009+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.757201+0000 osd.5 (osd.5) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:30.797711+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:30.896821+0000 mon.c (mon.2) 501 : audit [INF] from='client.? v1:172.21.15.107:0/1270765770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:30.897102+0000 mon.a (mon.0) 3730 : audit [INF] from='client.54497 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.039337+0000 osd.4 (osd.4) 103 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.042563+0000 osd.4 (osd.4) 104 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:50:32.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.347923+0000 osd.3 (osd.3) 381 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.348971+0000 osd.3 (osd.3) 382 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.757009+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.757201+0000 osd.5 (osd.5) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:30.797711+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:30.896821+0000 mon.c (mon.2) 501 : audit [INF] from='client.? v1:172.21.15.107:0/1270765770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:31 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:30.897102+0000 mon.a (mon.0) 3730 : audit [INF] from='client.54497 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.039337+0000 osd.4 (osd.4) 103 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:50:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.042563+0000 osd.4 (osd.4) 104 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:50:32.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.347923+0000 osd.3 (osd.3) 381 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.348971+0000 osd.3 (osd.3) 382 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.757009+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.757201+0000 osd.5 (osd.5) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:30.797711+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:30.896821+0000 mon.c (mon.2) 501 : audit [INF] from='client.? v1:172.21.15.107:0/1270765770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:30.897102+0000 mon.a (mon.0) 3730 : audit [INF] from='client.54497 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T08:50:32.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T08:50:32.889 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:31.065133+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:50:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:31.068175+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:31.592135+0000 mgr.y (mgr.14553) 696 : cluster [DBG] pgmap v1260: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:31.806315+0000 mon.a (mon.0) 3731 : audit [INF] from='client.54497 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:31.806374+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:31.902649+0000 mgr.y (mgr.14553) 697 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:32.453075+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:32.456723+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:31.065133+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:31.068175+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:31.592135+0000 mgr.y (mgr.14553) 696 : cluster [DBG] pgmap v1260: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:31.806315+0000 mon.a (mon.0) 3731 : audit [INF] from='client.54497 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:31.806374+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-10T08:50:32.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:31.902649+0000 mgr.y (mgr.14553) 697 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:32.453075+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:50:32.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:32.456723+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:31.065133+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.1a scrub starts 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:31.068175+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1a scrub ok 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:31.592135+0000 mgr.y (mgr.14553) 696 : cluster [DBG] pgmap v1260: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:31.806315+0000 mon.a (mon.0) 3731 : audit [INF] from='client.54497 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:31.806374+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:31.902649+0000 mgr.y (mgr.14553) 697 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:32.453075+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 deep-scrub starts 2024-02-10T08:50:33.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:32.456723+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 deep-scrub ok 2024-02-10T08:50:34.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:32.082101+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-02-10T08:50:34.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:32.085605+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-02-10T08:50:34.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:33.288966+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:50:34.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:33.292626+0000 osd.1 (osd.1) 381 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:50:34.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:33.605704+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:50:34.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:33.606610+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:50:34.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:32.082101+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-02-10T08:50:34.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:32.085605+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-02-10T08:50:34.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:33.288966+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:50:34.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:33.292626+0000 osd.1 (osd.1) 381 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:50:34.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:33.605704+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:50:34.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:33.606610+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:32.082101+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.c scrub starts 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:32.085605+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.c scrub ok 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:33.288966+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:33.292626+0000 osd.1 (osd.1) 381 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:33.605704+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:50:34.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:33.606610+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:50:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.701+0000 7f688a1ff700 1 -- 172.21.15.107:0/200473432 >> v1:172.21.15.107:6790/0 conn(0x7f6884104390 legacy=0x7f6884106850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.701+0000 7f688a1ff700 1 -- 172.21.15.107:0/200473432 shutdown_connections 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.701+0000 7f688a1ff700 1 -- 172.21.15.107:0/200473432 >> 172.21.15.107:0/200473432 conn(0x7f688406fe70 msgr2=0x7f68840f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.701+0000 7f688a1ff700 1 -- 172.21.15.107:0/200473432 shutdown_connections 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.701+0000 7f688a1ff700 1 -- 172.21.15.107:0/200473432 wait complete. 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 librados: starting msgr at 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 librados: starting objecter 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 Processor -- start 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 -- start start 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 librados: setting wanted keys 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 librados: calling monclient init 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6884117480 con 0x7f68840fda90 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6884118660 con 0x7f68841007c0 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f688a1ff700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6884119860 con 0x7f6884104390 2024-02-10T08:50:34.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f68837fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f68840fda90 0x7f68840f9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37608/0 (socket says 172.21.15.107:37608) 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6883fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6884104390 0x7f6884115b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43462/0 (socket says 172.21.15.107:43462) 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f68837fe700 1 -- 172.21.15.107:0/739964763 learned_addr learned my addr 172.21.15.107:0/739964763 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 10 client.?.objecter ms_handle_connect 0x7f6884104390 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 10 client.?.objecter ms_handle_connect 0x7f68840fda90 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587987752 0 0) 0x7f6884119860 con 0x7f6884104390 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6870003030 con 0x7f6884104390 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.702+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119468855 0 0) 0x7f6884117480 con 0x7f68840fda90 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884119860 con 0x7f68840fda90 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 10 client.?.objecter ms_handle_connect 0x7f68841007c0 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228562868 0 0) 0x7f6884118660 con 0x7f68841007c0 2024-02-10T08:50:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884117480 con 0x7f68841007c0 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619149424 0 0) 0x7f6870003030 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6884118660 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f687c003190 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230458604 0 0) 0x7f6884119860 con 0x7f68840fda90 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6870003030 con 0x7f68840fda90 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f686c001940 con 0x7f68840fda90 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1719268410 0 0) 0x7f6884118660 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 >> v1:172.21.15.195:6789/0 conn(0x7f68841007c0 legacy=0x7f68840fa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 >> v1:172.21.15.107:6789/0 conn(0x7f68840fda90 legacy=0x7f68840f9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688411aa60 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884117670 con 0x7f6884104390 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f688a1ff700 10 client.54503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6884117ae0 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f688a1ff700 1 librados: init done 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f687c002720 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.703+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f687c003ab0 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.704+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864004f20 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.704+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f687c002250 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(816..816 src has 765..816) v4 ==== 7386+0+0 (unknown 188230262 0 0) 0x7f687c079060 con 0x7f6884104390 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 10 client.54503.objecter ms_dispatch 0x7f6884001550 osd_map(816..816 src has 765..816) v4 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 3 client.54503.objecter handle_osd_map got epochs [816,816] > 0 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 3 client.54503.objecter handle_osd_map decoding full epoch 816 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 20 client.54503.objecter dump_active .. 0 homeless 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.705+0000 7f6880ff9700 10 client.54503.objecter ms_handle_connect 0x7f6870063c60 2024-02-10T08:50:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.709+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f687c04c750 con 0x7f6884104390 2024-02-10T08:50:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:34.887+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f68640051d0 con 0x7f6884104390 2024-02-10T08:50:35.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:33.592638+0000 mgr.y (mgr.14553) 698 : cluster [DBG] pgmap v1262: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:33.592638+0000 mgr.y (mgr.14553) 698 : cluster [DBG] pgmap v1262: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:35.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:33.592638+0000 mgr.y (mgr.14553) 698 : cluster [DBG] pgmap v1262: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:35.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.829+0000 7f6880ff9700 1 -- 172.21.15.107:0/739964763 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v817) v1 ==== 113+0+0 (unknown 295174335 0 0) 0x7f687c04abc0 con 0x7f6884104390 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 10 librados: watch_flush enter 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 10 librados: watch_flush exit 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 20 client.54503.objecter shutdown clearing up homeless session... 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 10 client.54503.objecter successfully canceled tick 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6870063c60 legacy=0x7f6870066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 >> v1:172.21.15.107:6790/0 conn(0x7f6884104390 legacy=0x7f6884115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 shutdown_connections 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 >> 172.21.15.107:0/739964763 conn(0x7f688406fe70 msgr2=0x7f68840f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 shutdown_connections 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 -- 172.21.15.107:0/739964763 wait complete. 2024-02-10T08:50:35.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:35.832+0000 7f688a1ff700 1 librados: shutdown 2024-02-10T08:50:35.835 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T08:50:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:34.891062+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.107:0/739964763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:34.891751+0000 mon.a (mon.0) 3733 : audit [INF] from='client.54503 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.552332+0000 osd.0 (osd.0) 202 : cluster [DBG] 4.b scrub starts 2024-02-10T08:50:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.554160+0000 osd.0 (osd.0) 203 : cluster [DBG] 4.b scrub ok 2024-02-10T08:50:35.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:35] "GET /metrics HTTP/1.1" 200 186408 "" "Prometheus/2.33.4" 2024-02-10T08:50:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:34.891062+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.107:0/739964763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:34.891751+0000 mon.a (mon.0) 3733 : audit [INF] from='client.54503 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.552332+0000 osd.0 (osd.0) 202 : cluster [DBG] 4.b scrub starts 2024-02-10T08:50:35.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.554160+0000 osd.0 (osd.0) 203 : cluster [DBG] 4.b scrub ok 2024-02-10T08:50:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:34.891062+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.107:0/739964763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:34.891751+0000 mon.a (mon.0) 3733 : audit [INF] from='client.54503 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.552332+0000 osd.0 (osd.0) 202 : cluster [DBG] 4.b scrub starts 2024-02-10T08:50:36.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.554160+0000 osd.0 (osd.0) 203 : cluster [DBG] 4.b scrub ok 2024-02-10T08:50:36.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-10T08:50:36.902 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.593207+0000 mgr.y (mgr.14553) 699 : cluster [DBG] pgmap v1263: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-02-10T08:50:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:35.828923+0000 mon.a (mon.0) 3734 : audit [INF] from='client.54503 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.828998+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-10T08:50:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.968171+0000 osd.4 (osd.4) 109 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:50:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:35.972235+0000 osd.4 (osd.4) 110 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:50:36.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.593207+0000 mgr.y (mgr.14553) 699 : cluster [DBG] pgmap v1263: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-02-10T08:50:36.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:35.828923+0000 mon.a (mon.0) 3734 : audit [INF] from='client.54503 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:36.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.828998+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-10T08:50:36.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.968171+0000 osd.4 (osd.4) 109 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:50:36.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:35.972235+0000 osd.4 (osd.4) 110 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:50:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.593207+0000 mgr.y (mgr.14553) 699 : cluster [DBG] pgmap v1263: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-02-10T08:50:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:35.828923+0000 mon.a (mon.0) 3734 : audit [INF] from='client.54503 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.828998+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-10T08:50:37.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.968171+0000 osd.4 (osd.4) 109 : cluster [DBG] 16.19 scrub starts 2024-02-10T08:50:37.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:35.972235+0000 osd.4 (osd.4) 110 : cluster [DBG] 16.19 scrub ok 2024-02-10T08:50:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:36.283371+0000 osd.3 (osd.3) 383 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:50:38.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:36.284326+0000 osd.3 (osd.3) 384 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:50:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:36.283371+0000 osd.3 (osd.3) 383 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:50:38.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:36.284326+0000 osd.3 (osd.3) 384 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:50:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:36.283371+0000 osd.3 (osd.3) 383 : cluster [DBG] 2.19 scrub starts 2024-02-10T08:50:38.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:36.284326+0000 osd.3 (osd.3) 384 : cluster [DBG] 2.19 scrub ok 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.711+0000 7fcda0505700 1 -- 172.21.15.107:0/2336119114 >> v1:172.21.15.107:6790/0 conn(0x7fcd980fc910 legacy=0x7fcd980fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.711+0000 7fcda0505700 1 -- 172.21.15.107:0/2336119114 shutdown_connections 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.711+0000 7fcda0505700 1 -- 172.21.15.107:0/2336119114 >> 172.21.15.107:0/2336119114 conn(0x7fcd980f2510 msgr2=0x7fcd980f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.711+0000 7fcda0505700 1 -- 172.21.15.107:0/2336119114 shutdown_connections 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 -- 172.21.15.107:0/2336119114 wait complete. 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 librados: starting msgr at 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 librados: starting objecter 2024-02-10T08:50:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 Processor -- start 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 -- start start 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 librados: setting wanted keys 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 librados: calling monclient init 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd980fbfb0 con 0x7fcd98103250 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd980fc180 con 0x7fcd980ff710 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcda0505700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd9811bb20 con 0x7fcd980fc910 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcd9e2a1700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcd980fc910 0x7fcd980fb7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43484/0 (socket says 172.21.15.107:43484) 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcd9e2a1700 1 -- 172.21.15.107:0/566704489 learned_addr learned my addr 172.21.15.107:0/566704489 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.712+0000 7fcd8f7fe700 10 client.?.objecter ms_handle_connect 0x7fcd980fc910 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 10 client.?.objecter ms_handle_connect 0x7fcd98103250 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 10 client.?.objecter ms_handle_connect 0x7fcd980ff710 2024-02-10T08:50:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926961486 0 0) 0x7fcd980fbfb0 con 0x7fcd98103250 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd840038e0 con 0x7fcd98103250 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886117500 0 0) 0x7fcd9811bb20 con 0x7fcd980fc910 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd980fbfb0 con 0x7fcd980fc910 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601222931 0 0) 0x7fcd980fc180 con 0x7fcd980ff710 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9811bb20 con 0x7fcd980ff710 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297301605 0 0) 0x7fcd980fbfb0 con 0x7fcd980fc910 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd980fc180 con 0x7fcd980fc910 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd880015d0 con 0x7fcd980fc910 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904559043 0 0) 0x7fcd840038e0 con 0x7fcd98103250 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd980fbfb0 con 0x7fcd98103250 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd90001bd0 con 0x7fcd98103250 2024-02-10T08:50:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680158108 0 0) 0x7fcd9811bb20 con 0x7fcd980ff710 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd840038e0 con 0x7fcd980ff710 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd940018c0 con 0x7fcd980ff710 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 16098076 0 0) 0x7fcd980fc180 con 0x7fcd980fc910 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 >> v1:172.21.15.195:6789/0 conn(0x7fcd980ff710 legacy=0x7fcd98114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 >> v1:172.21.15.107:6789/0 conn(0x7fcd98103250 legacy=0x7fcd981183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9811ccc0 con 0x7fcd980fc910 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9811bd10 con 0x7fcd980fc910 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcda0505700 10 client.54509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9811c180 con 0x7fcd980fc910 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcda0505700 1 librados: init done 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fcd88002400 con 0x7fcd980fc910 2024-02-10T08:50:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.713+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fcd88003a10 con 0x7fcd980fc910 2024-02-10T08:50:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.714+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c004f20 con 0x7fcd980fc910 2024-02-10T08:50:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fcd88003220 con 0x7fcd980fc910 2024-02-10T08:50:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(817..817 src has 765..817) v4 ==== 7386+0+0 (unknown 1435226702 0 0) 0x7fcd88078d10 con 0x7fcd980fc910 2024-02-10T08:50:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 10 client.54509.objecter ms_dispatch 0x7fcd98001550 osd_map(817..817 src has 765..817) v4 2024-02-10T08:50:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 3 client.54509.objecter handle_osd_map got epochs [817,817] > 0 2024-02-10T08:50:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 3 client.54509.objecter handle_osd_map decoding full epoch 817 2024-02-10T08:50:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 20 client.54509.objecter dump_active .. 0 homeless 2024-02-10T08:50:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.715+0000 7fcd8f7fe700 10 client.54509.objecter ms_handle_connect 0x7fcd84063e00 2024-02-10T08:50:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.719+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcd8804c4c0 con 0x7fcd980fc910 2024-02-10T08:50:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:38.895+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fcd7c0059e0 con 0x7fcd980fc910 2024-02-10T08:50:39.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:37.594115+0000 mgr.y (mgr.14553) 700 : cluster [DBG] pgmap v1265: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:50:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:37.594115+0000 mgr.y (mgr.14553) 700 : cluster [DBG] pgmap v1265: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:50:39.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:37.594115+0000 mgr.y (mgr.14553) 700 : cluster [DBG] pgmap v1265: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T08:50:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.838+0000 7fcd8f7fe700 1 -- 172.21.15.107:0/566704489 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v818) v1 ==== 133+0+0 (unknown 956777302 0 0) 0x7fcd8804a930 con 0x7fcd980fc910 2024-02-10T08:50:39.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.842+0000 7fcda0505700 10 librados: watch_flush enter 2024-02-10T08:50:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.844+0000 7fcda0505700 10 librados: watch_flush exit 2024-02-10T08:50:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 20 client.54509.objecter shutdown clearing up homeless session... 2024-02-10T08:50:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 10 client.54509.objecter successfully canceled tick 2024-02-10T08:50:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 >> v1:172.21.15.107:6800/2634842009 conn(0x7fcd84063e00 legacy=0x7fcd840662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:39.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 >> v1:172.21.15.107:6790/0 conn(0x7fcd980fc910 legacy=0x7fcd980fb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 shutdown_connections 2024-02-10T08:50:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 >> 172.21.15.107:0/566704489 conn(0x7fcd980f2510 msgr2=0x7fcd980612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 shutdown_connections 2024-02-10T08:50:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 -- 172.21.15.107:0/566704489 wait complete. 2024-02-10T08:50:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:39.845+0000 7fcda0505700 1 librados: shutdown 2024-02-10T08:50:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-10T08:50:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:38.897763+0000 mon.c (mon.2) 503 : audit [INF] from='client.? v1:172.21.15.107:0/566704489' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:38.898033+0000 mon.a (mon.0) 3736 : audit [INF] from='client.54509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.655000+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:50:40.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.656439+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:50:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:38.897763+0000 mon.c (mon.2) 503 : audit [INF] from='client.? v1:172.21.15.107:0/566704489' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:38.898033+0000 mon.a (mon.0) 3736 : audit [INF] from='client.54509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.655000+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:50:40.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.656439+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:50:40.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:38.897763+0000 mon.c (mon.2) 503 : audit [INF] from='client.? v1:172.21.15.107:0/566704489' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:38.898033+0000 mon.a (mon.0) 3736 : audit [INF] from='client.54509 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T08:50:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.655000+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:50:40.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.656439+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:50:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:40.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T08:50:41.064 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.072131+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.e scrub starts 2024-02-10T08:50:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.075645+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub ok 2024-02-10T08:50:41.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.594608+0000 mgr.y (mgr.14553) 701 : cluster [DBG] pgmap v1266: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:39.839552+0000 mon.a (mon.0) 3737 : audit [INF] from='client.54509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:39.839636+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:40.200693+0000 osd.3 (osd.3) 385 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:40.201600+0000 osd.3 (osd.3) 386 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.072131+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.e scrub starts 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.075645+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub ok 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.594608+0000 mgr.y (mgr.14553) 701 : cluster [DBG] pgmap v1266: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:39.839552+0000 mon.a (mon.0) 3737 : audit [INF] from='client.54509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:39.839636+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:40.200693+0000 osd.3 (osd.3) 385 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:50:41.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:40.201600+0000 osd.3 (osd.3) 386 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.072131+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.e scrub starts 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.075645+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.e scrub ok 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.594608+0000 mgr.y (mgr.14553) 701 : cluster [DBG] pgmap v1266: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:39.839552+0000 mon.a (mon.0) 3737 : audit [INF] from='client.54509 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:39.839636+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-10T08:50:41.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:40.200693+0000 osd.3 (osd.3) 385 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:50:41.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:40.201600+0000 osd.3 (osd.3) 386 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:50:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:40.820554+0000 osd.2 (osd.2) 331 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:50:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:40.823526+0000 osd.2 (osd.2) 332 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.136786+0000 mon.c (mon.2) 504 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.136964+0000 mon.c (mon.2) 505 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.137120+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.137140+0000 mon.c (mon.2) 506 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.137456+0000 mon.a (mon.0) 3740 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.137626+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.182297+0000 osd.6 (osd.6) 350 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:50:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.185202+0000 osd.6 (osd.6) 351 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.459742+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.462636+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:40.820554+0000 osd.2 (osd.2) 331 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:40.823526+0000 osd.2 (osd.2) 332 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.136786+0000 mon.c (mon.2) 504 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.136964+0000 mon.c (mon.2) 505 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.137120+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.137140+0000 mon.c (mon.2) 506 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.137456+0000 mon.a (mon.0) 3740 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.137626+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.182297+0000 osd.6 (osd.6) 350 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:50:42.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.185202+0000 osd.6 (osd.6) 351 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:50:42.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.459742+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:50:42.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.462636+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:50:42.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:40.820554+0000 osd.2 (osd.2) 331 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:50:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:40.823526+0000 osd.2 (osd.2) 332 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:50:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.136786+0000 mon.c (mon.2) 504 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.136964+0000 mon.c (mon.2) 505 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.137120+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.137140+0000 mon.c (mon.2) 506 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.137456+0000 mon.a (mon.0) 3740 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.137626+0000 mon.a (mon.0) 3741 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.182297+0000 osd.6 (osd.6) 350 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.185202+0000 osd.6 (osd.6) 351 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.459742+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:50:42.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.462636+0000 osd.5 (osd.5) 367 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:50:43.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.595409+0000 mgr.y (mgr.14553) 702 : cluster [DBG] pgmap v1268: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:43.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.792096+0000 osd.2 (osd.2) 333 : cluster [DBG] 6.1 deep-scrub starts 2024-02-10T08:50:43.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.793018+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.1 deep-scrub ok 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.848338+0000 mon.a (mon.0) 3742 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.848449+0000 mon.a (mon.0) 3743 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.848522+0000 mon.a (mon.0) 3744 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.848586+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:41.904589+0000 mgr.y (mgr.14553) 703 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.964869+0000 osd.4 (osd.4) 113 : cluster [DBG] 2.f scrub starts 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:41.968311+0000 osd.4 (osd.4) 114 : cluster [DBG] 2.f scrub ok 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:42.445092+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.d scrub starts 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:42.448241+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.d scrub ok 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:42.581521+0000 osd.0 (osd.0) 206 : cluster [DBG] 16.f scrub starts 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:42.585359+0000 osd.0 (osd.0) 207 : cluster [DBG] 16.f scrub ok 2024-02-10T08:50:43.037 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.595409+0000 mgr.y (mgr.14553) 702 : cluster [DBG] pgmap v1268: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.792096+0000 osd.2 (osd.2) 333 : cluster [DBG] 6.1 deep-scrub starts 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.793018+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.1 deep-scrub ok 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.848338+0000 mon.a (mon.0) 3742 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.848449+0000 mon.a (mon.0) 3743 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.848522+0000 mon.a (mon.0) 3744 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.848586+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:41.904589+0000 mgr.y (mgr.14553) 703 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.964869+0000 osd.4 (osd.4) 113 : cluster [DBG] 2.f scrub starts 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:41.968311+0000 osd.4 (osd.4) 114 : cluster [DBG] 2.f scrub ok 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:42.445092+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.d scrub starts 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:42.448241+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.d scrub ok 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:42.581521+0000 osd.0 (osd.0) 206 : cluster [DBG] 16.f scrub starts 2024-02-10T08:50:43.038 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:42.585359+0000 osd.0 (osd.0) 207 : cluster [DBG] 16.f scrub ok 2024-02-10T08:50:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- 172.21.15.107:0/3269819824 >> v1:172.21.15.107:6790/0 conn(0x7fafd0103250 legacy=0x7fafd01056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- 172.21.15.107:0/3269819824 shutdown_connections 2024-02-10T08:50:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- 172.21.15.107:0/3269819824 >> 172.21.15.107:0/3269819824 conn(0x7fafd00f2510 msgr2=0x7fafd00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- 172.21.15.107:0/3269819824 shutdown_connections 2024-02-10T08:50:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- 172.21.15.107:0/3269819824 wait complete. 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 librados: starting msgr at 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 librados: starting objecter 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 Processor -- start 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 -- start start 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 librados: setting wanted keys 2024-02-10T08:50:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.270+0000 7fafd4ac0700 1 librados: calling monclient init 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafd4ac0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafd0061800 con 0x7fafd0103250 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafd4ac0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafd00619d0 con 0x7fafd00ff710 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafd4ac0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafd0119770 con 0x7fafd00fc910 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafce59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fafd00fc910 0x7fafd0060d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35672/0 (socket says 172.21.15.107:35672) 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafce59c700 1 -- 172.21.15.107:0/2789896969 learned_addr learned my addr 172.21.15.107:0/2789896969 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter ms_handle_connect 0x7fafd00fc910 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter ms_handle_connect 0x7fafd0103250 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter ms_handle_connect 0x7fafd00ff710 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417087703 0 0) 0x7fafd0119770 con 0x7fafd00fc910 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc0033f0 con 0x7fafd00fc910 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435987170 0 0) 0x7fafd00619d0 con 0x7fafd00ff710 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd0119770 con 0x7fafd00ff710 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006808507 0 0) 0x7fafd0061800 con 0x7fafd0103250 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd00619d0 con 0x7fafd0103250 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232664111 0 0) 0x7fafbc0033f0 con 0x7fafd00fc910 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd0061800 con 0x7fafd00fc910 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafc0001d50 con 0x7fafd00fc910 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3833568465 0 0) 0x7fafd0119770 con 0x7fafd00ff710 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafbc0033f0 con 0x7fafd00ff710 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1638518723 0 0) 0x7fafd00619d0 con 0x7fafd0103250 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd0119770 con 0x7fafd0103250 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafb8001840 con 0x7fafd00ff710 2024-02-10T08:50:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafc8001ec0 con 0x7fafd0103250 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.271+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1998439145 0 0) 0x7fafd0061800 con 0x7fafd00fc910 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 >> v1:172.21.15.195:6789/0 conn(0x7fafd00ff710 legacy=0x7fafd01128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 >> v1:172.21.15.107:6789/0 conn(0x7fafd0103250 legacy=0x7fafd0115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd011a990 con 0x7fafd00fc910 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd0119960 con 0x7fafd00fc910 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafd4ac0700 10 client.54515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0119e50 con 0x7fafd00fc910 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafd4ac0700 1 librados: init done 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fafc0002760 con 0x7fafd00fc910 2024-02-10T08:50:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.272+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fafc0005e90 con 0x7fafd00fc910 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb0004f20 con 0x7fafd00fc910 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fafc00022c0 con 0x7fafd00fc910 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(820..820 src has 765..820) v4 ==== 7473+0+0 (unknown 495136936 0 0) 0x7fafc0078e30 con 0x7fafd00fc910 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafc77fe700 10 client.54515.objecter ms_dispatch 0x7fafd0001550 osd_map(820..820 src has 765..820) v4 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafc77fe700 3 client.54515.objecter handle_osd_map got epochs [820,820] > 0 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.273+0000 7fafc77fe700 3 client.54515.objecter handle_osd_map decoding full epoch 820 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.274+0000 7fafc77fe700 20 client.54515.objecter dump_active .. 0 homeless 2024-02-10T08:50:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.274+0000 7fafc77fe700 10 client.54515.objecter ms_handle_connect 0x7fafbc0681e0 2024-02-10T08:50:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.278+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fafc004c470 con 0x7fafd00fc910 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.595409+0000 mgr.y (mgr.14553) 702 : cluster [DBG] pgmap v1268: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.792096+0000 osd.2 (osd.2) 333 : cluster [DBG] 6.1 deep-scrub starts 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.793018+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.1 deep-scrub ok 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.848338+0000 mon.a (mon.0) 3742 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.848449+0000 mon.a (mon.0) 3743 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.848522+0000 mon.a (mon.0) 3744 : audit [INF] from='mgr.14553 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.848586+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:41.904589+0000 mgr.y (mgr.14553) 703 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.964869+0000 osd.4 (osd.4) 113 : cluster [DBG] 2.f scrub starts 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:41.968311+0000 osd.4 (osd.4) 114 : cluster [DBG] 2.f scrub ok 2024-02-10T08:50:43.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:42.445092+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.d scrub starts 2024-02-10T08:50:43.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:42.448241+0000 osd.5 (osd.5) 369 : cluster [DBG] 6.d scrub ok 2024-02-10T08:50:43.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:42.581521+0000 osd.0 (osd.0) 206 : cluster [DBG] 16.f scrub starts 2024-02-10T08:50:43.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:42.585359+0000 osd.0 (osd.0) 207 : cluster [DBG] 16.f scrub ok 2024-02-10T08:50:43.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.456+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fafb00051d0 con 0x7fafd00fc910 2024-02-10T08:50:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.861+0000 7fafc77fe700 1 -- 172.21.15.107:0/2789896969 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v821) v1 ==== 113+0+0 (unknown 3028888770 0 0) 0x7fafc004a880 con 0x7fafd00fc910 2024-02-10T08:50:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.863+0000 7fafd4ac0700 10 librados: watch_flush enter 2024-02-10T08:50:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.863+0000 7fafd4ac0700 10 librados: watch_flush exit 2024-02-10T08:50:43.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.863+0000 7fafd4ac0700 20 client.54515.objecter shutdown clearing up homeless session... 2024-02-10T08:50:43.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.863+0000 7fafd4ac0700 10 client.54515.objecter successfully canceled tick 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 >> v1:172.21.15.107:6800/2634842009 conn(0x7fafbc0681e0 legacy=0x7fafbc06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 >> v1:172.21.15.107:6790/0 conn(0x7fafd00fc910 legacy=0x7fafd0060d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 shutdown_connections 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 >> 172.21.15.107:0/2789896969 conn(0x7fafd00f2510 msgr2=0x7fafd01066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 shutdown_connections 2024-02-10T08:50:43.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 -- 172.21.15.107:0/2789896969 wait complete. 2024-02-10T08:50:43.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:43.864+0000 7fafd4ac0700 1 librados: shutdown 2024-02-10T08:50:43.867 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T08:50:44.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:42.859896+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:43.398714+0000 osd.1 (osd.1) 382 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:43.402313+0000 osd.1 (osd.1) 383 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:43.458218+0000 mon.c (mon.2) 507 : audit [INF] from='client.? v1:172.21.15.107:0/2789896969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:43.458476+0000 mon.a (mon.0) 3747 : audit [INF] from='client.54515 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:42.859896+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:43.398714+0000 osd.1 (osd.1) 382 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:43.402313+0000 osd.1 (osd.1) 383 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:43.458218+0000 mon.c (mon.2) 507 : audit [INF] from='client.? v1:172.21.15.107:0/2789896969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:43.458476+0000 mon.a (mon.0) 3747 : audit [INF] from='client.54515 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:42.859896+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-10T08:50:44.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:43.398714+0000 osd.1 (osd.1) 382 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:50:44.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:43.402313+0000 osd.1 (osd.1) 383 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:50:44.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:43.458218+0000 mon.c (mon.2) 507 : audit [INF] from='client.? v1:172.21.15.107:0/2789896969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:43.458476+0000 mon.a (mon.0) 3747 : audit [INF] from='client.54515 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:44.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-10T08:50:44.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:43.595834+0000 mgr.y (mgr.14553) 704 : cluster [DBG] pgmap v1271: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:44.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:43.861972+0000 mon.a (mon.0) 3748 : audit [INF] from='client.54515 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:44.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:43.862038+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-10T08:50:44.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:43.595834+0000 mgr.y (mgr.14553) 704 : cluster [DBG] pgmap v1271: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:44.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:43.861972+0000 mon.a (mon.0) 3748 : audit [INF] from='client.54515 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:44.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:43.862038+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-10T08:50:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:43.595834+0000 mgr.y (mgr.14553) 704 : cluster [DBG] pgmap v1271: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:50:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:43.861972+0000 mon.a (mon.0) 3748 : audit [INF] from='client.54515 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:43.862038+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-10T08:50:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:45] "GET /metrics HTTP/1.1" 200 186416 "" "Prometheus/2.33.4" 2024-02-10T08:50:45.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:44.995178+0000 osd.7 (osd.7) 350 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:44.998436+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.021528+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.021569+0000 osd.0 (osd.0) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.253831+0000 osd.3 (osd.3) 387 : cluster [DBG] 6.b scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.255073+0000 osd.3 (osd.3) 388 : cluster [DBG] 6.b scrub ok 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.544120+0000 osd.0 (osd.0) 210 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.545572+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:44.995178+0000 osd.7 (osd.7) 350 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:44.998436+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.021528+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:45.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.021569+0000 osd.0 (osd.0) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.253831+0000 osd.3 (osd.3) 387 : cluster [DBG] 6.b scrub starts 2024-02-10T08:50:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.255073+0000 osd.3 (osd.3) 388 : cluster [DBG] 6.b scrub ok 2024-02-10T08:50:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.544120+0000 osd.0 (osd.0) 210 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:50:45.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.545572+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:44.995178+0000 osd.7 (osd.7) 350 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:44.998436+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.021528+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.021569+0000 osd.0 (osd.0) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.253831+0000 osd.3 (osd.3) 387 : cluster [DBG] 6.b scrub starts 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.255073+0000 osd.3 (osd.3) 388 : cluster [DBG] 6.b scrub ok 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.544120+0000 osd.0 (osd.0) 210 : cluster [DBG] 4.16 scrub starts 2024-02-10T08:50:46.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.545572+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub ok 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.631+0000 7f7f58d55700 1 -- 172.21.15.107:0/3878179273 >> v1:172.21.15.107:6790/0 conn(0x7f7f540f8a20 legacy=0x7f7f540f8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.631+0000 7f7f58d55700 1 -- 172.21.15.107:0/3878179273 shutdown_connections 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.631+0000 7f7f58d55700 1 -- 172.21.15.107:0/3878179273 >> 172.21.15.107:0/3878179273 conn(0x7f7f540f2510 msgr2=0x7f7f540f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.631+0000 7f7f58d55700 1 -- 172.21.15.107:0/3878179273 shutdown_connections 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.631+0000 7f7f58d55700 1 -- 172.21.15.107:0/3878179273 wait complete. 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 librados: starting msgr at 2024-02-10T08:50:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 librados: starting objecter 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 Processor -- start 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 -- start start 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 librados: setting wanted keys 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 librados: calling monclient init 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f54117250 con 0x7f7f540f8a20 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f54118430 con 0x7f7f54104350 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f58d55700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f54119610 con 0x7f7f540f95d0 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f51d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7f540f95d0 0x7f7f540ffae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35694/0 (socket says 172.21.15.107:35694) 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.632+0000 7f7f51d9b700 1 -- 172.21.15.107:0/3899889301 learned_addr learned my addr 172.21.15.107:0/3899889301 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter ms_handle_connect 0x7f7f540f8a20 2024-02-10T08:50:46.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter ms_handle_connect 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter ms_handle_connect 0x7f7f54104350 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174860283 0 0) 0x7f7f54117250 con 0x7f7f540f8a20 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f400038e0 con 0x7f7f540f8a20 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682412284 0 0) 0x7f7f54119610 con 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54117250 con 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361200233 0 0) 0x7f7f54118430 con 0x7f7f54104350 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54119610 con 0x7f7f54104350 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 906851376 0 0) 0x7f7f54117250 con 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f54118430 con 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f440019f0 con 0x7f7f540f95d0 2024-02-10T08:50:46.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012221042 0 0) 0x7f7f400038e0 con 0x7f7f540f8a20 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f54117250 con 0x7f7f540f8a20 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f3c002e40 con 0x7f7f540f8a20 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129539791 0 0) 0x7f7f54119610 con 0x7f7f54104350 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f400038e0 con 0x7f7f54104350 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f4c001c40 con 0x7f7f54104350 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563629661 0 0) 0x7f7f54118430 con 0x7f7f540f95d0 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 >> v1:172.21.15.195:6789/0 conn(0x7f7f54104350 legacy=0x7f7f541159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 >> v1:172.21.15.107:6789/0 conn(0x7f7f540f8a20 legacy=0x7f7f540ff3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5411a810 con 0x7f7f540f95d0 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f54119800 con 0x7f7f540f95d0 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f58d55700 10 client.54521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f54119c70 con 0x7f7f540f95d0 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.633+0000 7f7f58d55700 1 librados: init done 2024-02-10T08:50:46.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.634+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f7f440023f0 con 0x7f7f540f95d0 2024-02-10T08:50:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.634+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f7f44005a00 con 0x7f7f540f95d0 2024-02-10T08:50:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.634+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f34004f20 con 0x7f7f540f95d0 2024-02-10T08:50:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f7f44002560 con 0x7f7f540f95d0 2024-02-10T08:50:46.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(822..822 src has 765..822) v4 ==== 7473+0+0 (unknown 3657556204 0 0) 0x7f7f44078860 con 0x7f7f540f95d0 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 10 client.54521.objecter ms_dispatch 0x7f7f54001550 osd_map(822..822 src has 765..822) v4 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 3 client.54521.objecter handle_osd_map got epochs [822,822] > 0 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 3 client.54521.objecter handle_osd_map decoding full epoch 822 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 20 client.54521.objecter dump_active .. 0 homeless 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.635+0000 7f7f4b7fe700 10 client.54521.objecter ms_handle_connect 0x7f7f40063c30 2024-02-10T08:50:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.640+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7f4404bf00 con 0x7f7f540f95d0 2024-02-10T08:50:46.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:46.815+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f7f340059e0 con 0x7f7f540f95d0 2024-02-10T08:50:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.596150+0000 mgr.y (mgr.14553) 705 : cluster [DBG] pgmap v1273: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-02-10T08:50:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.782432+0000 osd.2 (osd.2) 335 : cluster [DBG] 16.a scrub starts 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.785716+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.a scrub ok 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.877234+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.994940+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:45.996248+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:46.142096+0000 osd.6 (osd.6) 352 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:46.144089+0000 osd.6 (osd.6) 353 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:46.817657+0000 mon.c (mon.2) 508 : audit [INF] from='client.? v1:172.21.15.107:0/3899889301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:46.817963+0000 mon.a (mon.0) 3751 : audit [INF] from='client.54521 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:46.830127+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:46.833588+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:50:47.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.596150+0000 mgr.y (mgr.14553) 705 : cluster [DBG] pgmap v1273: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.782432+0000 osd.2 (osd.2) 335 : cluster [DBG] 16.a scrub starts 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.785716+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.a scrub ok 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.877234+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.994940+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:45.996248+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:46.142096+0000 osd.6 (osd.6) 352 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:46.144089+0000 osd.6 (osd.6) 353 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:46.817657+0000 mon.c (mon.2) 508 : audit [INF] from='client.? v1:172.21.15.107:0/3899889301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:46.817963+0000 mon.a (mon.0) 3751 : audit [INF] from='client.54521 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:46.830127+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:50:47.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:46.833588+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:50:47.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.229+0000 7f7f4b7fe700 1 -- 172.21.15.107:0/3899889301 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v823) v1 ==== 133+0+0 (unknown 1113160712 0 0) 0x7f7f4404a310 con 0x7f7f540f95d0 2024-02-10T08:50:47.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 10 librados: watch_flush enter 2024-02-10T08:50:47.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 10 librados: watch_flush exit 2024-02-10T08:50:47.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 20 client.54521.objecter shutdown clearing up homeless session... 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 10 client.54521.objecter successfully canceled tick 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 >> v1:172.21.15.107:6800/2634842009 conn(0x7f7f40063c30 legacy=0x7f7f400660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 >> v1:172.21.15.107:6790/0 conn(0x7f7f540f95d0 legacy=0x7f7f540ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 shutdown_connections 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 >> 172.21.15.107:0/3899889301 conn(0x7f7f540f2510 msgr2=0x7f7f541067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 shutdown_connections 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 -- 172.21.15.107:0/3899889301 wait complete. 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:47.231+0000 7f7f58d55700 1 librados: shutdown 2024-02-10T08:50:47.233 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.596150+0000 mgr.y (mgr.14553) 705 : cluster [DBG] pgmap v1273: 154 pgs: 1 peering, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.782432+0000 osd.2 (osd.2) 335 : cluster [DBG] 16.a scrub starts 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.785716+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.a scrub ok 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.877234+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.994940+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:45.996248+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:46.142096+0000 osd.6 (osd.6) 352 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:50:47.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:46.144089+0000 osd.6 (osd.6) 353 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:50:47.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:46.817657+0000 mon.c (mon.2) 508 : audit [INF] from='client.? v1:172.21.15.107:0/3899889301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:46.817963+0000 mon.a (mon.0) 3751 : audit [INF] from='client.54521 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T08:50:47.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:46.830127+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.d deep-scrub starts 2024-02-10T08:50:47.288 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:46.833588+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.d deep-scrub ok 2024-02-10T08:50:47.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T08:50:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:47.229474+0000 mon.a (mon.0) 3752 : audit [INF] from='client.54521 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:47.229530+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-10T08:50:48.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:47.596837+0000 mgr.y (mgr.14553) 706 : cluster [DBG] pgmap v1276: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 121 B/s, 4 objects/s recovering 2024-02-10T08:50:48.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:47.229474+0000 mon.a (mon.0) 3752 : audit [INF] from='client.54521 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:48.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:47.229530+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-10T08:50:48.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:47.596837+0000 mgr.y (mgr.14553) 706 : cluster [DBG] pgmap v1276: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 121 B/s, 4 objects/s recovering 2024-02-10T08:50:48.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:47.229474+0000 mon.a (mon.0) 3752 : audit [INF] from='client.54521 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T08:50:48.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:47.229530+0000 mon.a (mon.0) 3753 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-10T08:50:48.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:47.596837+0000 mgr.y (mgr.14553) 706 : cluster [DBG] pgmap v1276: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 121 B/s, 4 objects/s recovering 2024-02-10T08:50:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:48.525245+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:50:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:48.526757+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:50:49.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:48.815767+0000 osd.2 (osd.2) 339 : cluster [DBG] 5.6 deep-scrub starts 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:48.818859+0000 osd.2 (osd.2) 340 : cluster [DBG] 5.6 deep-scrub ok 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:49.168709+0000 mon.c (mon.2) 509 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:48.525245+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:48.526757+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:48.815767+0000 osd.2 (osd.2) 339 : cluster [DBG] 5.6 deep-scrub starts 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:48.818859+0000 osd.2 (osd.2) 340 : cluster [DBG] 5.6 deep-scrub ok 2024-02-10T08:50:49.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:49.168709+0000 mon.c (mon.2) 509 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:50:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:48.525245+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.3 scrub starts 2024-02-10T08:50:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:48.526757+0000 osd.0 (osd.0) 213 : cluster [DBG] 5.3 scrub ok 2024-02-10T08:50:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:48.815767+0000 osd.2 (osd.2) 339 : cluster [DBG] 5.6 deep-scrub starts 2024-02-10T08:50:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:48.818859+0000 osd.2 (osd.2) 340 : cluster [DBG] 5.6 deep-scrub ok 2024-02-10T08:50:49.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:49.168709+0000 mon.c (mon.2) 509 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.159+0000 7f78bfa9d700 1 -- 172.21.15.107:0/4011030028 >> v1:172.21.15.107:6789/0 conn(0x7f78b81007f0 legacy=0x7f78b8102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.159+0000 7f78bfa9d700 1 -- 172.21.15.107:0/4011030028 shutdown_connections 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.159+0000 7f78bfa9d700 1 -- 172.21.15.107:0/4011030028 >> 172.21.15.107:0/4011030028 conn(0x7f78b80f3640 msgr2=0x7f78b80f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.159+0000 7f78bfa9d700 1 -- 172.21.15.107:0/4011030028 shutdown_connections 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.159+0000 7f78bfa9d700 1 -- 172.21.15.107:0/4011030028 wait complete. 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 librados: starting msgr at 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 librados: starting objecter 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 Processor -- start 2024-02-10T08:50:50.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 -- start start 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 librados: setting wanted keys 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 librados: calling monclient init 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78b8117400 con 0x7f78b81007f0 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78b81185e0 con 0x7f78b8104330 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bfa9d700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78b81197e0 con 0x7f78b80fda50 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bd839700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f78b80fda50 0x7f78b80f9bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48430/0 (socket says 172.21.15.107:48430) 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bd839700 1 -- 172.21.15.107:0/3331193797 learned_addr learned my addr 172.21.15.107:0/3331193797 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78bd038700 1 --1- 172.21.15.107:0/3331193797 >> v1:172.21.15.107:6789/0 conn(0x7f78b81007f0 0x7f78b80fa2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53106/0 (socket says 172.21.15.107:53106) 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78aef78700 10 client.?.objecter ms_handle_connect 0x7f78b80fda50 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78aef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78aef78700 10 client.?.objecter ms_handle_connect 0x7f78b81007f0 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.160+0000 7f78aef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:50.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448039108 0 0) 0x7f78b8117400 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a4003030 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833397760 0 0) 0x7f78b81197e0 con 0x7f78b80fda50 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b8117400 con 0x7f78b80fda50 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 10 client.?.objecter ms_handle_connect 0x7f78b8104330 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553722721 0 0) 0x7f78b81185e0 con 0x7f78b8104330 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b81197e0 con 0x7f78b8104330 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142962401 0 0) 0x7f78a4003030 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78b81185e0 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78b4000fe0 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1043602392 0 0) 0x7f78b8117400 con 0x7f78b80fda50 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78a4003030 con 0x7f78b80fda50 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78a80018c0 con 0x7f78b80fda50 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3495625655 0 0) 0x7f78b81185e0 con 0x7f78b81007f0 2024-02-10T08:50:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 >> v1:172.21.15.107:6790/0 conn(0x7f78b80fda50 legacy=0x7f78b80f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 >> v1:172.21.15.195:6789/0 conn(0x7f78b8104330 legacy=0x7f78b8115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b811a9e0 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b81175f0 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78bfa9d700 10 client.28929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b8117a60 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78bfa9d700 1 librados: init done 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f78b4002290 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.161+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f78b4003720 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.162+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c004f20 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.162+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f78b4001840 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(823..823 src has 765..823) v4 ==== 7473+0+0 (unknown 3116402894 0 0) 0x7f78b4078cd0 con 0x7f78b81007f0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 10 client.28929.objecter ms_dispatch 0x7f78b8001550 osd_map(823..823 src has 765..823) v4 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 3 client.28929.objecter handle_osd_map got epochs [823,823] > 0 2024-02-10T08:50:50.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 3 client.28929.objecter handle_osd_map decoding full epoch 823 2024-02-10T08:50:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 20 client.28929.objecter dump_active .. 0 homeless 2024-02-10T08:50:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.163+0000 7f78aef78700 10 client.28929.objecter ms_handle_connect 0x7f78a40639e0 2024-02-10T08:50:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.167+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f78b404c3d0 con 0x7f78b81007f0 2024-02-10T08:50:50.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:50.347+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f789c0051d0 con 0x7f78b81007f0 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:49.572777+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:49.575734+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:49.597221+0000 mgr.y (mgr.14553) 707 : cluster [DBG] pgmap v1277: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:49.987434+0000 mon.c (mon.2) 510 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:50:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:49.988053+0000 mon.c (mon.2) 511 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:50:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:49.992517+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:50:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:49.572777+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:50:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:49.575734+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:49.597221+0000 mgr.y (mgr.14553) 707 : cluster [DBG] pgmap v1277: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:49.987434+0000 mon.c (mon.2) 510 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:49.988053+0000 mon.c (mon.2) 511 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:49.992517+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:49.572777+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:49.575734+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:49.597221+0000 mgr.y (mgr.14553) 707 : cluster [DBG] pgmap v1277: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:49.987434+0000 mon.c (mon.2) 510 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:49.988053+0000 mon.c (mon.2) 511 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:50:50.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:49.992517+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:50:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.245+0000 7f78aef78700 1 -- 172.21.15.107:0/3331193797 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v824) v1 ==== 113+0+0 (unknown 2509839621 0 0) 0x7f78b40020f0 con 0x7f78b81007f0 2024-02-10T08:50:51.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.248+0000 7f78bfa9d700 10 librados: watch_flush enter 2024-02-10T08:50:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 10 librados: watch_flush exit 2024-02-10T08:50:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 20 client.28929.objecter shutdown clearing up homeless session... 2024-02-10T08:50:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 10 client.28929.objecter successfully canceled tick 2024-02-10T08:50:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 >> v1:172.21.15.107:6800/2634842009 conn(0x7f78a40639e0 legacy=0x7f78a4065ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 >> v1:172.21.15.107:6789/0 conn(0x7f78b81007f0 legacy=0x7f78b80fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 shutdown_connections 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 >> 172.21.15.107:0/3331193797 conn(0x7f78b80f3640 msgr2=0x7f78b81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 shutdown_connections 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 -- 172.21.15.107:0/3331193797 wait complete. 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:51.249+0000 7f78bfa9d700 1 librados: shutdown 2024-02-10T08:50:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T08:50:51.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:50.348644+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:51.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.543024+0000 osd.0 (osd.0) 214 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:50:51.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.543883+0000 osd.0 (osd.0) 215 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:50:51.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.738745+0000 osd.2 (osd.2) 341 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:50:51.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.741815+0000 osd.2 (osd.2) 342 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:50:51.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:50.348644+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:51.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.543024+0000 osd.0 (osd.0) 214 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:50:51.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.543883+0000 osd.0 (osd.0) 215 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:50:51.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.738745+0000 osd.2 (osd.2) 341 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:50:51.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.741815+0000 osd.2 (osd.2) 342 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:50:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:50.348644+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.543024+0000 osd.0 (osd.0) 214 : cluster [DBG] 6.10 scrub starts 2024-02-10T08:50:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.543883+0000 osd.0 (osd.0) 215 : cluster [DBG] 6.10 scrub ok 2024-02-10T08:50:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.738745+0000 osd.2 (osd.2) 341 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:50:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.741815+0000 osd.2 (osd.2) 342 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:50:52.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-10T08:50:52.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:50:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:50:52.395 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.988542+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:50:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:50.991794+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:50:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.140953+0000 osd.6 (osd.6) 354 : cluster [DBG] 16.d scrub starts 2024-02-10T08:50:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.143591+0000 osd.6 (osd.6) 355 : cluster [DBG] 16.d scrub ok 2024-02-10T08:50:52.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:51.246062+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.246142+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.262981+0000 osd.3 (osd.3) 389 : cluster [DBG] 6.1f deep-scrub starts 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.264740+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1f deep-scrub ok 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.272090+0000 osd.1 (osd.1) 384 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.275201+0000 osd.1 (osd.1) 385 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.577415+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.580967+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.597702+0000 mgr.y (mgr.14553) 708 : cluster [DBG] pgmap v1279: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.765757+0000 osd.2 (osd.2) 343 : cluster [DBG] 16.e scrub starts 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.768641+0000 osd.2 (osd.2) 344 : cluster [DBG] 16.e scrub ok 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.897866+0000 osd.4 (osd.4) 115 : cluster [DBG] 5.e scrub starts 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:51.901317+0000 osd.4 (osd.4) 116 : cluster [DBG] 5.e scrub ok 2024-02-10T08:50:52.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:51.915697+0000 mgr.y (mgr.14553) 709 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.988542+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:50.991794+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.140953+0000 osd.6 (osd.6) 354 : cluster [DBG] 16.d scrub starts 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.143591+0000 osd.6 (osd.6) 355 : cluster [DBG] 16.d scrub ok 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:51.246062+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.246142+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.262981+0000 osd.3 (osd.3) 389 : cluster [DBG] 6.1f deep-scrub starts 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.264740+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1f deep-scrub ok 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.272090+0000 osd.1 (osd.1) 384 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.275201+0000 osd.1 (osd.1) 385 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.577415+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:50:52.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.580967+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.597702+0000 mgr.y (mgr.14553) 708 : cluster [DBG] pgmap v1279: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.765757+0000 osd.2 (osd.2) 343 : cluster [DBG] 16.e scrub starts 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.768641+0000 osd.2 (osd.2) 344 : cluster [DBG] 16.e scrub ok 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.897866+0000 osd.4 (osd.4) 115 : cluster [DBG] 5.e scrub starts 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:51.901317+0000 osd.4 (osd.4) 116 : cluster [DBG] 5.e scrub ok 2024-02-10T08:50:52.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:51.915697+0000 mgr.y (mgr.14553) 709 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.988542+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 deep-scrub starts 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:50.991794+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 deep-scrub ok 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.140953+0000 osd.6 (osd.6) 354 : cluster [DBG] 16.d scrub starts 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.143591+0000 osd.6 (osd.6) 355 : cluster [DBG] 16.d scrub ok 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:51.246062+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? v1:172.21.15.107:0/3331193797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.246142+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.262981+0000 osd.3 (osd.3) 389 : cluster [DBG] 6.1f deep-scrub starts 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.264740+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1f deep-scrub ok 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.272090+0000 osd.1 (osd.1) 384 : cluster [DBG] 16.9 deep-scrub starts 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.275201+0000 osd.1 (osd.1) 385 : cluster [DBG] 16.9 deep-scrub ok 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.577415+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.15 scrub starts 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.580967+0000 osd.5 (osd.5) 373 : cluster [DBG] 5.15 scrub ok 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.597702+0000 mgr.y (mgr.14553) 708 : cluster [DBG] pgmap v1279: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.765757+0000 osd.2 (osd.2) 343 : cluster [DBG] 16.e scrub starts 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.768641+0000 osd.2 (osd.2) 344 : cluster [DBG] 16.e scrub ok 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.897866+0000 osd.4 (osd.4) 115 : cluster [DBG] 5.e scrub starts 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:51.901317+0000 osd.4 (osd.4) 116 : cluster [DBG] 5.e scrub ok 2024-02-10T08:50:52.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:51.915697+0000 mgr.y (mgr.14553) 709 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:50:53.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.186597+0000 osd.6 (osd.6) 356 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:50:53.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.187536+0000 osd.6 (osd.6) 357 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.315280+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.318769+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.617425+0000 osd.5 (osd.5) 374 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:52.621151+0000 osd.5 (osd.5) 375 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.186597+0000 osd.6 (osd.6) 356 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.187536+0000 osd.6 (osd.6) 357 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.315280+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.318769+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.617425+0000 osd.5 (osd.5) 374 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:50:53.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:52.621151+0000 osd.5 (osd.5) 375 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.186597+0000 osd.6 (osd.6) 356 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.187536+0000 osd.6 (osd.6) 357 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.315280+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.318769+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.617425+0000 osd.5 (osd.5) 374 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:50:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:52.621151+0000 osd.5 (osd.5) 375 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/3541181847 >> v1:172.21.15.107:6790/0 conn(0x7f0b940fda50 legacy=0x7f0b940fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/3541181847 shutdown_connections 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/3541181847 >> 172.21.15.107:0/3541181847 conn(0x7f0b940f3640 msgr2=0x7f0b940f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/3541181847 shutdown_connections 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/3541181847 wait complete. 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 librados: starting msgr at 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.074+0000 7f0b9a7b3700 1 librados: starting objecter 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 Processor -- start 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 -- start start 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 librados: setting wanted keys 2024-02-10T08:50:54.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 librados: calling monclient init 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b940fd000 con 0x7f0b940fda50 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b940fd1d0 con 0x7f0b94104330 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b9a7b3700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0b940fd3e0 con 0x7f0b941007f0 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b937fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0b941007f0 0x7f0b94114d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48460/0 (socket says 172.21.15.107:48460) 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b937fe700 1 -- 172.21.15.107:0/877881189 learned_addr learned my addr 172.21.15.107:0/877881189 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b93fff700 1 --1- 172.21.15.107:0/877881189 >> v1:172.21.15.107:6789/0 conn(0x7f0b940fda50 0x7f0b940fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53128/0 (socket says 172.21.15.107:53128) 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter ms_handle_connect 0x7f0b941007f0 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter ms_handle_connect 0x7f0b940fda50 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619310332 0 0) 0x7f0b940fd3e0 con 0x7f0b941007f0 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c0033f0 con 0x7f0b941007f0 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619913344 0 0) 0x7f0b940fd000 con 0x7f0b940fda50 2024-02-10T08:50:54.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b940fd3e0 con 0x7f0b940fda50 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter ms_handle_connect 0x7f0b94104330 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.075+0000 7f0b917fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016950956 0 0) 0x7f0b940fd1d0 con 0x7f0b94104330 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b940fd000 con 0x7f0b94104330 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673151269 0 0) 0x7f0b940fd3e0 con 0x7f0b940fda50 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b940fd1d0 con 0x7f0b940fda50 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031009726 0 0) 0x7f0b7c0033f0 con 0x7f0b941007f0 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b940fd3e0 con 0x7f0b941007f0 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b84002050 con 0x7f0b940fda50 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b880019d0 con 0x7f0b941007f0 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462002667 0 0) 0x7f0b940fd000 con 0x7f0b94104330 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b7c0033f0 con 0x7f0b94104330 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 638809606 0 0) 0x7f0b940fd1d0 con 0x7f0b940fda50 2024-02-10T08:50:54.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 >> v1:172.21.15.107:6790/0 conn(0x7f0b941007f0 legacy=0x7f0b94114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 >> v1:172.21.15.195:6789/0 conn(0x7f0b94104330 legacy=0x7f0b941184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9411cc00 con 0x7f0b940fda50 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b94119c30 con 0x7f0b940fda50 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b9a7b3700 10 client.28935.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9411a0a0 con 0x7f0b940fda50 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b9a7b3700 1 librados: init done 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f0b84002fd0 con 0x7f0b940fda50 2024-02-10T08:50:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.076+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f0b840035e0 con 0x7f0b940fda50 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.077+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80004f20 con 0x7f0b940fda50 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.077+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f0b84001880 con 0x7f0b940fda50 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(824..824 src has 765..824) v4 ==== 7473+0+0 (unknown 2270510290 0 0) 0x7f0b84078c00 con 0x7f0b940fda50 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 10 client.28935.objecter ms_dispatch 0x7f0b94001550 osd_map(824..824 src has 765..824) v4 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 3 client.28935.objecter handle_osd_map got epochs [824,824] > 0 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 3 client.28935.objecter handle_osd_map decoding full epoch 824 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 20 client.28935.objecter dump_active .. 0 homeless 2024-02-10T08:50:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.078+0000 7f0b917fa700 10 client.28935.objecter ms_handle_connect 0x7f0b7c063d30 2024-02-10T08:50:54.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.082+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0b8404c2e0 con 0x7f0b940fda50 2024-02-10T08:50:54.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.260+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f0b800059e0 con 0x7f0b940fda50 2024-02-10T08:50:54.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.318+0000 7f0b917fa700 1 -- 172.21.15.107:0/877881189 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v825) v1 ==== 133+0+0 (unknown 184756426 0 0) 0x7f0b8404a750 con 0x7f0b940fda50 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 10 librados: watch_flush enter 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 10 librados: watch_flush exit 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 20 client.28935.objecter shutdown clearing up homeless session... 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 10 client.28935.objecter successfully canceled tick 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0b7c063d30 legacy=0x7f0b7c0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 >> v1:172.21.15.107:6789/0 conn(0x7f0b940fda50 legacy=0x7f0b940fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 shutdown_connections 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 >> 172.21.15.107:0/877881189 conn(0x7f0b940f3640 msgr2=0x7f0b941077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 shutdown_connections 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 -- 172.21.15.107:0/877881189 wait complete. 2024-02-10T08:50:54.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:54.320+0000 7f0b9a7b3700 1 librados: shutdown 2024-02-10T08:50:54.322 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-10T08:50:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:53.551227+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:50:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:53.552513+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:50:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:53.598275+0000 mgr.y (mgr.14553) 710 : cluster [DBG] pgmap v1280: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 74 B/s, 2 objects/s recovering 2024-02-10T08:50:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:53.772737+0000 osd.2 (osd.2) 345 : cluster [DBG] 5.b scrub starts 2024-02-10T08:50:54.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:53.775909+0000 osd.2 (osd.2) 346 : cluster [DBG] 5.b scrub ok 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:54.261000+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:53.551227+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:53.552513+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:53.598275+0000 mgr.y (mgr.14553) 710 : cluster [DBG] pgmap v1280: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 74 B/s, 2 objects/s recovering 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:53.772737+0000 osd.2 (osd.2) 345 : cluster [DBG] 5.b scrub starts 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:53.775909+0000 osd.2 (osd.2) 346 : cluster [DBG] 5.b scrub ok 2024-02-10T08:50:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:54.261000+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:53.551227+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:53.552513+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:53.598275+0000 mgr.y (mgr.14553) 710 : cluster [DBG] pgmap v1280: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 74 B/s, 2 objects/s recovering 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:53.772737+0000 osd.2 (osd.2) 345 : cluster [DBG] 5.b scrub starts 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:53.775909+0000 osd.2 (osd.2) 346 : cluster [DBG] 5.b scrub ok 2024-02-10T08:50:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:54.261000+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T08:50:55.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T08:50:55.490 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:54.317724+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.317781+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.354886+0000 osd.1 (osd.1) 388 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.357890+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.555020+0000 osd.0 (osd.0) 218 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.556193+0000 osd.0 (osd.0) 219 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.873434+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.877102+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:55.204649+0000 osd.3 (osd.3) 391 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:50:55.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:55.206119+0000 osd.3 (osd.3) 392 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:54.317724+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.317781+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.354886+0000 osd.1 (osd.1) 388 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.357890+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.555020+0000 osd.0 (osd.0) 218 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.556193+0000 osd.0 (osd.0) 219 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.873434+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.877102+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:55.204649+0000 osd.3 (osd.3) 391 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:50:55.635 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:55.206119+0000 osd.3 (osd.3) 392 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:54.317724+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? v1:172.21.15.107:0/877881189' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.317781+0000 mon.a (mon.0) 3760 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.354886+0000 osd.1 (osd.1) 388 : cluster [DBG] 6.1d scrub starts 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.357890+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.1d scrub ok 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.555020+0000 osd.0 (osd.0) 218 : cluster [DBG] 16.8 scrub starts 2024-02-10T08:50:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.556193+0000 osd.0 (osd.0) 219 : cluster [DBG] 16.8 scrub ok 2024-02-10T08:50:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.873434+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:50:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.877102+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:50:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:55.204649+0000 osd.3 (osd.3) 391 : cluster [DBG] 16.16 scrub starts 2024-02-10T08:50:55.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:55.206119+0000 osd.3 (osd.3) 392 : cluster [DBG] 16.16 scrub ok 2024-02-10T08:50:55.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:50:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:50:55] "GET /metrics HTTP/1.1" 200 186415 "" "Prometheus/2.33.4" 2024-02-10T08:50:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.996596+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:50:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:54.999499+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:50:56.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:55.598821+0000 mgr.y (mgr.14553) 711 : cluster [DBG] pgmap v1282: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:56.253286+0000 osd.3 (osd.3) 393 : cluster [DBG] 2.12 deep-scrub starts 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:56.254210+0000 osd.3 (osd.3) 394 : cluster [DBG] 2.12 deep-scrub ok 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.996596+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:54.999499+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:55.598821+0000 mgr.y (mgr.14553) 711 : cluster [DBG] pgmap v1282: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:56.253286+0000 osd.3 (osd.3) 393 : cluster [DBG] 2.12 deep-scrub starts 2024-02-10T08:50:56.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:56.254210+0000 osd.3 (osd.3) 394 : cluster [DBG] 2.12 deep-scrub ok 2024-02-10T08:50:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.996596+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-02-10T08:50:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:54.999499+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-02-10T08:50:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:55.598821+0000 mgr.y (mgr.14553) 711 : cluster [DBG] pgmap v1282: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:50:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:56.253286+0000 osd.3 (osd.3) 393 : cluster [DBG] 2.12 deep-scrub starts 2024-02-10T08:50:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:56.254210+0000 osd.3 (osd.3) 394 : cluster [DBG] 2.12 deep-scrub ok 2024-02-10T08:50:57.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3432343758 >> v1:172.21.15.107:6789/0 conn(0x7f8cc0106590 legacy=0x7f8cc0108a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:57.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3432343758 shutdown_connections 2024-02-10T08:50:57.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3432343758 >> 172.21.15.107:0/3432343758 conn(0x7f8cc00f3640 msgr2=0x7f8cc00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3432343758 shutdown_connections 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3432343758 wait complete. 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 librados: starting msgr at 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.362+0000 7f8cc4bb2700 1 librados: starting objecter 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 Processor -- start 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 -- start start 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 librados: setting wanted keys 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 librados: calling monclient init 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8cc01195f0 con 0x7f8cc006b1b0 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8cc011a790 con 0x7f8cc0106590 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cc4bb2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8cc011b990 con 0x7f8cc006bca0 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cbe59c700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8cc006b1b0 0x7f8cc01028a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53158/0 (socket says 172.21.15.107:53158) 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8cbe59c700 1 -- 172.21.15.107:0/3418037924 learned_addr learned my addr 172.21.15.107:0/3418037924 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:50:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cc006bca0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cc006b1b0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781019162 0 0) 0x7f8cc011b990 con 0x7f8cc006bca0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca80038e0 con 0x7f8cc006bca0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cc0106590 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034510467 0 0) 0x7f8cc01195f0 con 0x7f8cc006b1b0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.363+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc011b990 con 0x7f8cc006b1b0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053344740 0 0) 0x7f8cc011a790 con 0x7f8cc0106590 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc01195f0 con 0x7f8cc0106590 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944904398 0 0) 0x7f8ca80038e0 con 0x7f8cc006bca0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc011a790 con 0x7f8cc006bca0 2024-02-10T08:50:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8cb0001880 con 0x7f8cc006bca0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894600840 0 0) 0x7f8cc011b990 con 0x7f8cc006b1b0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca80038e0 con 0x7f8cc006b1b0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8cb40019f0 con 0x7f8cc006b1b0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764188779 0 0) 0x7f8cc01195f0 con 0x7f8cc0106590 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc011b990 con 0x7f8cc0106590 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3630749337 0 0) 0x7f8cc011a790 con 0x7f8cc006bca0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 >> v1:172.21.15.195:6789/0 conn(0x7f8cc0106590 legacy=0x7f8cc0117c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 >> v1:172.21.15.107:6789/0 conn(0x7f8cc006b1b0 legacy=0x7f8cc01028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc011cb90 con 0x7f8cc006bca0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc01197e0 con 0x7f8cc006bca0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8cc4bb2700 10 client.54539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc0119c50 con 0x7f8cc006bca0 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8cc4bb2700 1 librados: init done 2024-02-10T08:50:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f8cb0001f90 con 0x7f8cc006bca0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.364+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f8cb0003380 con 0x7f8cc006bca0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.365+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca0004f20 con 0x7f8cc006bca0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.365+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f8cb0016210 con 0x7f8cc006bca0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(825..825 src has 775..825) v4 ==== 7473+0+0 (unknown 435623105 0 0) 0x7f8cb00762c0 con 0x7f8cc006bca0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 10 client.54539.objecter ms_dispatch 0x7f8cc0001550 osd_map(825..825 src has 775..825) v4 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 3 client.54539.objecter handle_osd_map got epochs [825,825] > 0 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 3 client.54539.objecter handle_osd_map decoding full epoch 825 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 20 client.54539.objecter dump_active .. 0 homeless 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.366+0000 7f8caf7fe700 10 client.54539.objecter ms_handle_connect 0x7f8ca8063c40 2024-02-10T08:50:57.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.370+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8cb0049900 con 0x7f8cc006bca0 2024-02-10T08:50:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:56.602265+0000 osd.0 (osd.0) 220 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:50:57.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:56.603594+0000 osd.0 (osd.0) 221 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:50:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:56.602265+0000 osd.0 (osd.0) 220 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:50:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:57 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:56.603594+0000 osd.0 (osd.0) 221 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:50:57.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:57.546+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f8ca00051d0 con 0x7f8cc006bca0 2024-02-10T08:50:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:56.602265+0000 osd.0 (osd.0) 220 : cluster [DBG] 16.5 scrub starts 2024-02-10T08:50:57.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:56.603594+0000 osd.0 (osd.0) 221 : cluster [DBG] 16.5 scrub ok 2024-02-10T08:50:58.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.334+0000 7f8caf7fe700 1 -- 172.21.15.107:0/3418037924 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v826) v1 ==== 113+0+0 (unknown 1582534404 0 0) 0x7f8cb0047d10 con 0x7f8cc006bca0 2024-02-10T08:50:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 10 librados: watch_flush enter 2024-02-10T08:50:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 10 librados: watch_flush exit 2024-02-10T08:50:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 20 client.54539.objecter shutdown clearing up homeless session... 2024-02-10T08:50:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 10 client.54539.objecter successfully canceled tick 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 >> v1:172.21.15.107:6800/2634842009 conn(0x7f8ca8063c40 legacy=0x7f8ca8066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 >> v1:172.21.15.107:6790/0 conn(0x7f8cc006bca0 legacy=0x7f8cc0102fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 shutdown_connections 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 >> 172.21.15.107:0/3418037924 conn(0x7f8cc00f3640 msgr2=0x7f8cc01089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 shutdown_connections 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.339+0000 7f8cc4bb2700 1 -- 172.21.15.107:0/3418037924 wait complete. 2024-02-10T08:50:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:50:58.340+0000 7f8cc4bb2700 1 librados: shutdown 2024-02-10T08:50:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T08:50:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:57.284290+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:50:58.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:57.285358+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:57.547043+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.107:0/3418037924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:57.547320+0000 mon.a (mon.0) 3761 : audit [INF] from='client.54539 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:57.599606+0000 mgr.y (mgr.14553) 712 : cluster [DBG] pgmap v1283: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:57.284290+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:57.285358+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:57.547043+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.107:0/3418037924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:57.547320+0000 mon.a (mon.0) 3761 : audit [INF] from='client.54539 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:57.599606+0000 mgr.y (mgr.14553) 712 : cluster [DBG] pgmap v1283: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:57.284290+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:50:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:57.285358+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:50:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:57.547043+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.107:0/3418037924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:57.547320+0000 mon.a (mon.0) 3761 : audit [INF] from='client.54539 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:50:58.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:57.599606+0000 mgr.y (mgr.14553) 712 : cluster [DBG] pgmap v1283: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:50:59.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-10T08:50:59.371 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:50:59.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:57.989339+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:50:59.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:57.992316+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:50:59.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:50:58.334412+0000 mon.a (mon.0) 3762 : audit [INF] from='client.54539 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:50:59.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:58.334469+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-10T08:50:59.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:59.256658+0000 osd.3 (osd.3) 397 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:59.260157+0000 osd.3 (osd.3) 398 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:57.989339+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:57.992316+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:50:58.334412+0000 mon.a (mon.0) 3762 : audit [INF] from='client.54539 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:58.334469+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:59.256658+0000 osd.3 (osd.3) 397 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:50:59.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:50:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:59.260157+0000 osd.3 (osd.3) 398 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:57.989339+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub starts 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:57.992316+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.8 scrub ok 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:50:58.334412+0000 mon.a (mon.0) 3762 : audit [INF] from='client.54539 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:58.334469+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:59.256658+0000 osd.3 (osd.3) 397 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:50:59.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:50:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:59.260157+0000 osd.3 (osd.3) 398 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:51:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:59.599966+0000 mgr.y (mgr.14553) 713 : cluster [DBG] pgmap v1285: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:59.776653+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.1 deep-scrub starts 2024-02-10T08:51:00.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:50:59.777442+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.1 deep-scrub ok 2024-02-10T08:51:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:59.599966+0000 mgr.y (mgr.14553) 713 : cluster [DBG] pgmap v1285: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:59.776653+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.1 deep-scrub starts 2024-02-10T08:51:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:50:59.777442+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.1 deep-scrub ok 2024-02-10T08:51:00.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:59.599966+0000 mgr.y (mgr.14553) 713 : cluster [DBG] pgmap v1285: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:00.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:59.776653+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.1 deep-scrub starts 2024-02-10T08:51:00.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:50:59.777442+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.1 deep-scrub ok 2024-02-10T08:51:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.282+0000 7f11f63e8700 1 -- 172.21.15.107:0/636277395 >> v1:172.21.15.107:6790/0 conn(0x7f11f01007f0 legacy=0x7f11f0102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.282+0000 7f11f63e8700 1 -- 172.21.15.107:0/636277395 shutdown_connections 2024-02-10T08:51:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.282+0000 7f11f63e8700 1 -- 172.21.15.107:0/636277395 >> 172.21.15.107:0/636277395 conn(0x7f11f00f3640 msgr2=0x7f11f00f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:01.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.282+0000 7f11f63e8700 1 -- 172.21.15.107:0/636277395 shutdown_connections 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 -- 172.21.15.107:0/636277395 wait complete. 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 librados: starting msgr at 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 librados: starting objecter 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 Processor -- start 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 -- start start 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 librados: setting wanted keys 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 librados: calling monclient init 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11f01173a0 con 0x7f11f00fda50 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11f0118580 con 0x7f11f01007f0 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.283+0000 7f11f63e8700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11f0119780 con 0x7f11f0104330 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11effff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f11f00fda50 0x7f11f00f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53184/0 (socket says 172.21.15.107:53184) 2024-02-10T08:51:01.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11effff700 1 -- 172.21.15.107:0/2789121924 learned_addr learned my addr 172.21.15.107:0/2789121924 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter ms_handle_connect 0x7f11f00fda50 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter ms_handle_connect 0x7f11f0104330 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter ms_handle_connect 0x7f11f01007f0 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873506029 0 0) 0x7f11f01173a0 con 0x7f11f00fda50 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8003030 con 0x7f11f00fda50 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66790579 0 0) 0x7f11f0119780 con 0x7f11f0104330 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f01173a0 con 0x7f11f0104330 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011336147 0 0) 0x7f11f0118580 con 0x7f11f01007f0 2024-02-10T08:51:01.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f0119780 con 0x7f11f01007f0 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959006755 0 0) 0x7f11d8003030 con 0x7f11f00fda50 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f0118580 con 0x7f11f00fda50 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11e0001940 con 0x7f11f00fda50 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624134113 0 0) 0x7f11f0119780 con 0x7f11f01007f0 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d8003030 con 0x7f11f01007f0 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11e4002c20 con 0x7f11f01007f0 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757651407 0 0) 0x7f11f01173a0 con 0x7f11f0104330 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f0119780 con 0x7f11f0104330 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11e80019a0 con 0x7f11f0104330 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2386665935 0 0) 0x7f11f0118580 con 0x7f11f00fda50 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 >> v1:172.21.15.107:6790/0 conn(0x7f11f0104330 legacy=0x7f11f0115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 >> v1:172.21.15.195:6789/0 conn(0x7f11f01007f0 legacy=0x7f11f00fa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f011a980 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f0119970 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11f63e8700 10 client.28947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f0119de0 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11f63e8700 1 librados: init done 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f11e0001cd0 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.284+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f11e0005ca0 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.285+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc004f20 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f11e0002270 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(826..826 src has 775..826) v4 ==== 7473+0+0 (unknown 3182734503 0 0) 0x7f11e0078ae0 con 0x7f11f00fda50 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 10 client.28947.objecter ms_dispatch 0x7f11f0001550 osd_map(826..826 src has 775..826) v4 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 3 client.28947.objecter handle_osd_map got epochs [826,826] > 0 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 3 client.28947.objecter handle_osd_map decoding full epoch 826 2024-02-10T08:51:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 20 client.28947.objecter dump_active .. 0 homeless 2024-02-10T08:51:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.286+0000 7f11ed7fa700 10 client.28947.objecter ms_handle_connect 0x7f11d8063cc0 2024-02-10T08:51:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.290+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f11e004c180 con 0x7f11f00fda50 2024-02-10T08:51:01.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:01.468+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f11dc0059e0 con 0x7f11f00fda50 2024-02-10T08:51:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:00.928574+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:51:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:00.932098+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:51:01.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:00.928574+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:51:01.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:00.932098+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:51:01.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:00.928574+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1 scrub starts 2024-02-10T08:51:01.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:00.932098+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1 scrub ok 2024-02-10T08:51:02.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:02.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.349+0000 7f11ed7fa700 1 -- 172.21.15.107:0/2789121924 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v827) v1 ==== 133+0+0 (unknown 2366697984 0 0) 0x7f11e004a5f0 con 0x7f11f00fda50 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 10 librados: watch_flush enter 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 10 librados: watch_flush exit 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 20 client.28947.objecter shutdown clearing up homeless session... 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 10 client.28947.objecter successfully canceled tick 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 >> v1:172.21.15.107:6800/2634842009 conn(0x7f11d8063cc0 legacy=0x7f11d8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.353+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 >> v1:172.21.15.107:6789/0 conn(0x7f11f00fda50 legacy=0x7f11f00f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.354+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 shutdown_connections 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.354+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 >> 172.21.15.107:0/2789121924 conn(0x7f11f00f3640 msgr2=0x7f11f01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:02.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.354+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 shutdown_connections 2024-02-10T08:51:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.354+0000 7f11f63e8700 1 -- 172.21.15.107:0/2789121924 wait complete. 2024-02-10T08:51:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:02.354+0000 7f11f63e8700 1 librados: shutdown 2024-02-10T08:51:02.355 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-10T08:51:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:01.420450+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:51:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:01.423582+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:51:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:01.469274+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:51:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:01.600769+0000 mgr.y (mgr.14553) 714 : cluster [DBG] pgmap v1286: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:01.926526+0000 mgr.y (mgr.14553) 715 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:02.240232+0000 osd.3 (osd.3) 399 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:02.242050+0000 osd.3 (osd.3) 400 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:01.420450+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:01.423582+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:01.469274+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:01.600769+0000 mgr.y (mgr.14553) 714 : cluster [DBG] pgmap v1286: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:01.926526+0000 mgr.y (mgr.14553) 715 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:02.240232+0000 osd.3 (osd.3) 399 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:51:02.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:02.242050+0000 osd.3 (osd.3) 400 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:51:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:01.420450+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.4 scrub starts 2024-02-10T08:51:02.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:01.423582+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.4 scrub ok 2024-02-10T08:51:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:01.469274+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T08:51:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:01.600769+0000 mgr.y (mgr.14553) 714 : cluster [DBG] pgmap v1286: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:01.926526+0000 mgr.y (mgr.14553) 715 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:02.240232+0000 osd.3 (osd.3) 399 : cluster [DBG] 4.14 scrub starts 2024-02-10T08:51:02.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:02.242050+0000 osd.3 (osd.3) 400 : cluster [DBG] 4.14 scrub ok 2024-02-10T08:51:03.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T08:51:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:01.854388+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:01.855410+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:02.349023+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:02.349127+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:03.209222+0000 osd.3 (osd.3) 401 : cluster [DBG] 4.10 deep-scrub starts 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:03.211825+0000 osd.3 (osd.3) 402 : cluster [DBG] 4.10 deep-scrub ok 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:01.854388+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:51:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:01.855410+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:51:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:02.349023+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:51:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:02.349127+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-10T08:51:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:03.209222+0000 osd.3 (osd.3) 401 : cluster [DBG] 4.10 deep-scrub starts 2024-02-10T08:51:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:03.211825+0000 osd.3 (osd.3) 402 : cluster [DBG] 4.10 deep-scrub ok 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:01.854388+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:01.855410+0000 osd.7 (osd.7) 361 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:02.349023+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.107:0/2789121924' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:02.349127+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:03.209222+0000 osd.3 (osd.3) 401 : cluster [DBG] 4.10 deep-scrub starts 2024-02-10T08:51:03.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:03.211825+0000 osd.3 (osd.3) 402 : cluster [DBG] 4.10 deep-scrub ok 2024-02-10T08:51:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:03.534375+0000 osd.0 (osd.0) 222 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:51:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:03.535639+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:51:04.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:03.601088+0000 mgr.y (mgr.14553) 716 : cluster [DBG] pgmap v1288: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:03.534375+0000 osd.0 (osd.0) 222 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:51:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:03.535639+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:51:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:03.601088+0000 mgr.y (mgr.14553) 716 : cluster [DBG] pgmap v1288: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:03.534375+0000 osd.0 (osd.0) 222 : cluster [DBG] 6.9 scrub starts 2024-02-10T08:51:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:03.535639+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.9 scrub ok 2024-02-10T08:51:04.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:03.601088+0000 mgr.y (mgr.14553) 716 : cluster [DBG] pgmap v1288: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.247+0000 7f17628e4700 1 -- 172.21.15.107:0/2655392315 >> v1:172.21.15.107:6789/0 conn(0x7f175c0ff710 legacy=0x7f175c101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.247+0000 7f17628e4700 1 -- 172.21.15.107:0/2655392315 shutdown_connections 2024-02-10T08:51:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.247+0000 7f17628e4700 1 -- 172.21.15.107:0/2655392315 >> 172.21.15.107:0/2655392315 conn(0x7f175c0f2510 msgr2=0x7f175c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.247+0000 7f17628e4700 1 -- 172.21.15.107:0/2655392315 shutdown_connections 2024-02-10T08:51:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.247+0000 7f17628e4700 1 -- 172.21.15.107:0/2655392315 wait complete. 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 librados: starting msgr at 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 librados: starting objecter 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 Processor -- start 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 -- start start 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 librados: setting wanted keys 2024-02-10T08:51:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 librados: calling monclient init 2024-02-10T08:51:05.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f175c117380 con 0x7f175c0ff710 2024-02-10T08:51:05.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f175c118560 con 0x7f175c103250 2024-02-10T08:51:05.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17628e4700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f175c119760 con 0x7f175c0fc910 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f175bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f175c0fc910 0x7f175c0f9b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55166/0 (socket says 172.21.15.107:55166) 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f175bfff700 1 -- 172.21.15.107:0/1527829853 learned_addr learned my addr 172.21.15.107:0/1527829853 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17597fa700 10 client.?.objecter ms_handle_connect 0x7f175c0ff710 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17597fa700 10 client.?.objecter ms_handle_connect 0x7f175c0fc910 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.248+0000 7f17597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 10 client.?.objecter ms_handle_connect 0x7f175c103250 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288087804 0 0) 0x7f175c119760 con 0x7f175c0fc910 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17440038e0 con 0x7f175c0fc910 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230292834 0 0) 0x7f175c117380 con 0x7f175c0ff710 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f175c119760 con 0x7f175c0ff710 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982148326 0 0) 0x7f175c118560 con 0x7f175c103250 2024-02-10T08:51:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f175c117380 con 0x7f175c103250 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655228787 0 0) 0x7f17440038e0 con 0x7f175c0fc910 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f175c118560 con 0x7f175c0fc910 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f17500019f0 con 0x7f175c0fc910 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941996 0 0) 0x7f175c119760 con 0x7f175c0ff710 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17440038e0 con 0x7f175c0ff710 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f174c000fe0 con 0x7f175c0ff710 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346122436 0 0) 0x7f175c117380 con 0x7f175c103250 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f175c119760 con 0x7f175c103250 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1754001b80 con 0x7f175c103250 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1820704313 0 0) 0x7f175c118560 con 0x7f175c0fc910 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 >> v1:172.21.15.195:6789/0 conn(0x7f175c103250 legacy=0x7f175c115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 >> v1:172.21.15.107:6789/0 conn(0x7f175c0ff710 legacy=0x7f175c0fa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c11a960 con 0x7f175c0fc910 2024-02-10T08:51:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175c119950 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17628e4700 10 client.54551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c119e40 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.249+0000 7f17628e4700 1 librados: init done 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.250+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1750002360 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.250+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f17500059a0 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.250+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1748004f20 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.250+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1750005b00 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(827..827 src has 775..827) v4 ==== 7473+0+0 (unknown 3732259973 0 0) 0x7f1750078a30 con 0x7f175c0fc910 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 10 client.54551.objecter ms_dispatch 0x7f175c001550 osd_map(827..827 src has 775..827) v4 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 3 client.54551.objecter handle_osd_map got epochs [827,827] > 0 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 3 client.54551.objecter handle_osd_map decoding full epoch 827 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 20 client.54551.objecter dump_active .. 0 homeless 2024-02-10T08:51:05.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.251+0000 7f17597fa700 10 client.54551.objecter ms_handle_connect 0x7f1744063c70 2024-02-10T08:51:05.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.255+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f175004c220 con 0x7f175c0fc910 2024-02-10T08:51:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.430+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f17480051d0 con 0x7f175c0fc910 2024-02-10T08:51:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.486+0000 7f17597fa700 1 -- 172.21.15.107:0/1527829853 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v828) v1 ==== 113+0+0 (unknown 2597421603 0 0) 0x7f1750051c50 con 0x7f175c0fc910 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 10 librados: watch_flush enter 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 10 librados: watch_flush exit 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 20 client.54551.objecter shutdown clearing up homeless session... 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 10 client.54551.objecter successfully canceled tick 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1744063c70 legacy=0x7f1744066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.488+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 >> v1:172.21.15.107:6790/0 conn(0x7f175c0fc910 legacy=0x7f175c0f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.489+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 shutdown_connections 2024-02-10T08:51:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.489+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 >> 172.21.15.107:0/1527829853 conn(0x7f175c0f2510 msgr2=0x7f175c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.491+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 shutdown_connections 2024-02-10T08:51:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.491+0000 7f17628e4700 1 -- 172.21.15.107:0/1527829853 wait complete. 2024-02-10T08:51:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:05.491+0000 7f17628e4700 1 librados: shutdown 2024-02-10T08:51:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T08:51:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:05.430991+0000 mon.c (mon.2) 513 : audit [INF] from='client.? v1:172.21.15.107:0/1527829853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:05.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:05.431284+0000 mon.a (mon.0) 3767 : audit [INF] from='client.54551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:05.834 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:05.430991+0000 mon.c (mon.2) 513 : audit [INF] from='client.? v1:172.21.15.107:0/1527829853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:05.834 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:05.431284+0000 mon.a (mon.0) 3767 : audit [INF] from='client.54551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:05.835 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:05] "GET /metrics HTTP/1.1" 200 186415 "" "Prometheus/2.33.4" 2024-02-10T08:51:05.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:05.430991+0000 mon.c (mon.2) 513 : audit [INF] from='client.? v1:172.21.15.107:0/1527829853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:05.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:05.431284+0000 mon.a (mon.0) 3767 : audit [INF] from='client.54551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:06.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-10T08:51:06.564 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:04.695292+0000 osd.5 (osd.5) 376 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:51:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:04.698757+0000 osd.5 (osd.5) 377 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:05.485799+0000 mon.a (mon.0) 3768 : audit [INF] from='client.54551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:05.485863+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:05.601438+0000 mgr.y (mgr.14553) 717 : cluster [DBG] pgmap v1290: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:05.740972+0000 osd.7 (osd.7) 362 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:05.742388+0000 osd.7 (osd.7) 363 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:06.202024+0000 osd.3 (osd.3) 403 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:06.203289+0000 osd.3 (osd.3) 404 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:51:06.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:04.695292+0000 osd.5 (osd.5) 376 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:04.698757+0000 osd.5 (osd.5) 377 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:05.485799+0000 mon.a (mon.0) 3768 : audit [INF] from='client.54551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:05.485863+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:05.601438+0000 mgr.y (mgr.14553) 717 : cluster [DBG] pgmap v1290: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:05.740972+0000 osd.7 (osd.7) 362 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:05.742388+0000 osd.7 (osd.7) 363 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:06.202024+0000 osd.3 (osd.3) 403 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:51:06.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:06.203289+0000 osd.3 (osd.3) 404 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:51:06.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:04.695292+0000 osd.5 (osd.5) 376 : cluster [DBG] 4.15 deep-scrub starts 2024-02-10T08:51:06.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:04.698757+0000 osd.5 (osd.5) 377 : cluster [DBG] 4.15 deep-scrub ok 2024-02-10T08:51:06.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:05.485799+0000 mon.a (mon.0) 3768 : audit [INF] from='client.54551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:06.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:05.485863+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-10T08:51:06.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:05.601438+0000 mgr.y (mgr.14553) 717 : cluster [DBG] pgmap v1290: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-02-10T08:51:06.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:05.740972+0000 osd.7 (osd.7) 362 : cluster [DBG] 16.1e scrub starts 2024-02-10T08:51:06.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:05.742388+0000 osd.7 (osd.7) 363 : cluster [DBG] 16.1e scrub ok 2024-02-10T08:51:06.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:06.202024+0000 osd.3 (osd.3) 403 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:51:06.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:06.203289+0000 osd.3 (osd.3) 404 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:51:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:06.467472+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:06.467803+0000 osd.4 (osd.4) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.229103+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:51:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.230823+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-10T08:51:07.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.231785+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:51:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:06.467472+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:06.467803+0000 osd.4 (osd.4) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.229103+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:51:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.230823+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-10T08:51:07.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.231785+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:51:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:06.467472+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:06.467803+0000 osd.4 (osd.4) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.229103+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.14 scrub starts 2024-02-10T08:51:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.230823+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-10T08:51:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.231785+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.14 scrub ok 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/1366329222 >> v1:172.21.15.107:6790/0 conn(0x7f3bc40fc910 legacy=0x7f3bc40fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/1366329222 shutdown_connections 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/1366329222 >> 172.21.15.107:0/1366329222 conn(0x7f3bc40f2510 msgr2=0x7f3bc40f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/1366329222 shutdown_connections 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/1366329222 wait complete. 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 librados: starting msgr at 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 librados: starting objecter 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 Processor -- start 2024-02-10T08:51:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 -- start start 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 librados: setting wanted keys 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.440+0000 7f3bcc3ba700 1 librados: calling monclient init 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bcc3ba700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bc4115330 con 0x7f3bc4103250 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bcc3ba700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bc4116510 con 0x7f3bc40ff710 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bcc3ba700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bc4117710 con 0x7f3bc40fc910 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bca156700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3bc40fc910 0x7f3bc40f8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55188/0 (socket says 172.21.15.107:55188) 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bca957700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3bc4103250 0x7f3bc41139f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53210/0 (socket says 172.21.15.107:53210) 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bca156700 1 -- 172.21.15.107:0/458168187 learned_addr learned my addr 172.21.15.107:0/458168187 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter ms_handle_connect 0x7f3bc4103250 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter ms_handle_connect 0x7f3bc40fc910 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter ms_handle_connect 0x7f3bc40ff710 2024-02-10T08:51:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768657358 0 0) 0x7f3bc4117710 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb00033f0 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390307680 0 0) 0x7f3bc4115330 con 0x7f3bc4103250 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4117710 con 0x7f3bc4103250 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337464538 0 0) 0x7f3bc4116510 con 0x7f3bc40ff710 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4115330 con 0x7f3bc40ff710 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121896845 0 0) 0x7f3bb00033f0 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bc4116510 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3bc0002e10 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1458460331 0 0) 0x7f3bc4117710 con 0x7f3bc4103250 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bb00033f0 con 0x7f3bc4103250 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.441+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3bbc001be0 con 0x7f3bc4103250 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2226399747 0 0) 0x7f3bc4116510 con 0x7f3bc40fc910 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 >> v1:172.21.15.195:6789/0 conn(0x7f3bc40ff710 legacy=0x7f3bc40f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 >> v1:172.21.15.107:6789/0 conn(0x7f3bc4103250 legacy=0x7f3bc41139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4118910 con 0x7f3bc40fc910 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc4117900 con 0x7f3bc40fc910 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bcc3ba700 10 client.54557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc4117d70 con 0x7f3bc40fc910 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bcc3ba700 1 librados: init done 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3bc0001b60 con 0x7f3bc40fc910 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.442+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3bc00038f0 con 0x7f3bc40fc910 2024-02-10T08:51:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba8004f20 con 0x7f3bc40fc910 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3bc0003a50 con 0x7f3bc40fc910 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(829..829 src has 775..829) v4 ==== 7473+0+0 (unknown 2468266680 0 0) 0x7f3bc0078cc0 con 0x7f3bc40fc910 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bbb7fe700 10 client.54557.objecter ms_dispatch 0x7f3bc4001550 osd_map(829..829 src has 775..829) v4 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bbb7fe700 3 client.54557.objecter handle_osd_map got epochs [829,829] > 0 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.443+0000 7f3bbb7fe700 3 client.54557.objecter handle_osd_map decoding full epoch 829 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.444+0000 7f3bbb7fe700 20 client.54557.objecter dump_active .. 0 homeless 2024-02-10T08:51:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.444+0000 7f3bbb7fe700 10 client.54557.objecter ms_handle_connect 0x7f3bb0068030 2024-02-10T08:51:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.448+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3bc004c300 con 0x7f3bc40fc910 2024-02-10T08:51:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:08.628+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f3ba80059e0 con 0x7f3bc40fc910 2024-02-10T08:51:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.595208+0000 osd.0 (osd.0) 224 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:51:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.597340+0000 osd.0 (osd.0) 225 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:51:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.601925+0000 mgr.y (mgr.14553) 718 : cluster [DBG] pgmap v1292: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.736155+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 deep-scrub starts 2024-02-10T08:51:08.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:07.741130+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.9 deep-scrub ok 2024-02-10T08:51:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.595208+0000 osd.0 (osd.0) 224 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:51:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.597340+0000 osd.0 (osd.0) 225 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:51:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.601925+0000 mgr.y (mgr.14553) 718 : cluster [DBG] pgmap v1292: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.736155+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 deep-scrub starts 2024-02-10T08:51:08.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:07.741130+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.9 deep-scrub ok 2024-02-10T08:51:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.595208+0000 osd.0 (osd.0) 224 : cluster [DBG] 4.6 scrub starts 2024-02-10T08:51:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.597340+0000 osd.0 (osd.0) 225 : cluster [DBG] 4.6 scrub ok 2024-02-10T08:51:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.601925+0000 mgr.y (mgr.14553) 718 : cluster [DBG] pgmap v1292: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.736155+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.9 deep-scrub starts 2024-02-10T08:51:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:07.741130+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.9 deep-scrub ok 2024-02-10T08:51:09.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.509+0000 7f3bbb7fe700 1 -- 172.21.15.107:0/458168187 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v830) v1 ==== 133+0+0 (unknown 4132359518 0 0) 0x7f3bc004a770 con 0x7f3bc40fc910 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 10 librados: watch_flush enter 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 10 librados: watch_flush exit 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 20 client.54557.objecter shutdown clearing up homeless session... 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 10 client.54557.objecter successfully canceled tick 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3bb0068030 legacy=0x7f3bb006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.512+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 >> v1:172.21.15.107:6790/0 conn(0x7f3bc40fc910 legacy=0x7f3bc40f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.513+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 shutdown_connections 2024-02-10T08:51:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.513+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 >> 172.21.15.107:0/458168187 conn(0x7f3bc40f2510 msgr2=0x7f3bc4060260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:09.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.513+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 shutdown_connections 2024-02-10T08:51:09.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.513+0000 7f3bcc3ba700 1 -- 172.21.15.107:0/458168187 wait complete. 2024-02-10T08:51:09.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:09.513+0000 7f3bcc3ba700 1 librados: shutdown 2024-02-10T08:51:09.514 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-10T08:51:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.496977+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:51:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.500794+0000 osd.1 (osd.1) 393 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:51:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.566868+0000 osd.0 (osd.0) 226 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:51:09.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.569477+0000 osd.0 (osd.0) 227 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:08.629096+0000 mon.c (mon.2) 514 : audit [INF] from='client.? v1:172.21.15.107:0/458168187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:08.629384+0000 mon.a (mon.0) 3771 : audit [INF] from='client.54557 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.776822+0000 osd.2 (osd.2) 349 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:08.778383+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.216568+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:51:09.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.218341+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:51:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.496977+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:51:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.500794+0000 osd.1 (osd.1) 393 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:51:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.566868+0000 osd.0 (osd.0) 226 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.569477+0000 osd.0 (osd.0) 227 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:08.629096+0000 mon.c (mon.2) 514 : audit [INF] from='client.? v1:172.21.15.107:0/458168187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:08.629384+0000 mon.a (mon.0) 3771 : audit [INF] from='client.54557 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.776822+0000 osd.2 (osd.2) 349 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:08.778383+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.216568+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.218341+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.496977+0000 osd.1 (osd.1) 392 : cluster [DBG] 6.4 deep-scrub starts 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.500794+0000 osd.1 (osd.1) 393 : cluster [DBG] 6.4 deep-scrub ok 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.566868+0000 osd.0 (osd.0) 226 : cluster [DBG] 4.3 scrub starts 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.569477+0000 osd.0 (osd.0) 227 : cluster [DBG] 4.3 scrub ok 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:08.629096+0000 mon.c (mon.2) 514 : audit [INF] from='client.? v1:172.21.15.107:0/458168187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:08.629384+0000 mon.a (mon.0) 3771 : audit [INF] from='client.54557 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T08:51:09.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.776822+0000 osd.2 (osd.2) 349 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:51:09.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:08.778383+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:51:09.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.216568+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.0 scrub starts 2024-02-10T08:51:09.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.218341+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.0 scrub ok 2024-02-10T08:51:10.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T08:51:10.516 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:09.509441+0000 mon.a (mon.0) 3772 : audit [INF] from='client.54557 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.509528+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-10T08:51:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.602354+0000 mgr.y (mgr.14553) 719 : cluster [DBG] pgmap v1294: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.967527+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:09.970393+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:09.509441+0000 mon.a (mon.0) 3772 : audit [INF] from='client.54557 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.509528+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.602354+0000 mgr.y (mgr.14553) 719 : cluster [DBG] pgmap v1294: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.967527+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:51:10.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:09.970393+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:51:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:09.509441+0000 mon.a (mon.0) 3772 : audit [INF] from='client.54557 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T08:51:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.509528+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-10T08:51:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.602354+0000 mgr.y (mgr.14553) 719 : cluster [DBG] pgmap v1294: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.967527+0000 osd.4 (osd.4) 123 : cluster [DBG] 2.13 scrub starts 2024-02-10T08:51:10.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:09.970393+0000 osd.4 (osd.4) 124 : cluster [DBG] 2.13 scrub ok 2024-02-10T08:51:12.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:12.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:11.603075+0000 mgr.y (mgr.14553) 720 : cluster [DBG] pgmap v1295: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T08:51:12.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:11.891835+0000 osd.4 (osd.4) 125 : cluster [DBG] 4.d scrub starts 2024-02-10T08:51:12.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:11.895694+0000 osd.4 (osd.4) 126 : cluster [DBG] 4.d scrub ok 2024-02-10T08:51:12.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:11.933102+0000 mgr.y (mgr.14553) 721 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:12.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:11.603075+0000 mgr.y (mgr.14553) 720 : cluster [DBG] pgmap v1295: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T08:51:12.522 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:11.891835+0000 osd.4 (osd.4) 125 : cluster [DBG] 4.d scrub starts 2024-02-10T08:51:12.522 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:11.895694+0000 osd.4 (osd.4) 126 : cluster [DBG] 4.d scrub ok 2024-02-10T08:51:12.522 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:11.933102+0000 mgr.y (mgr.14553) 721 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:11.603075+0000 mgr.y (mgr.14553) 720 : cluster [DBG] pgmap v1295: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T08:51:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:11.891835+0000 osd.4 (osd.4) 125 : cluster [DBG] 4.d scrub starts 2024-02-10T08:51:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:11.895694+0000 osd.4 (osd.4) 126 : cluster [DBG] 4.d scrub ok 2024-02-10T08:51:12.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:11.933102+0000 mgr.y (mgr.14553) 721 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 -- 172.21.15.107:0/3479698600 >> v1:172.21.15.107:6790/0 conn(0x7f96940fc910 legacy=0x7f96940fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 -- 172.21.15.107:0/3479698600 shutdown_connections 2024-02-10T08:51:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 -- 172.21.15.107:0/3479698600 >> 172.21.15.107:0/3479698600 conn(0x7f96940f2510 msgr2=0x7f96940f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 -- 172.21.15.107:0/3479698600 shutdown_connections 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 -- 172.21.15.107:0/3479698600 wait complete. 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 librados: starting msgr at 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.773+0000 7f969c1d9700 1 librados: starting objecter 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 Processor -- start 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 -- start start 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 librados: setting wanted keys 2024-02-10T08:51:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 librados: calling monclient init 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96941171d0 con 0x7f96940fc910 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f9699774700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f96940ff710 0x7f9694061350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47630/0 (socket says 172.21.15.107:47630) 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96941183b0 con 0x7f9694103250 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f9699774700 1 -- 172.21.15.107:0/2756070648 learned_addr learned my addr 172.21.15.107:0/2756070648 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96941195b0 con 0x7f96940ff710 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f9699f75700 1 --1- 172.21.15.107:0/2756070648 >> v1:172.21.15.107:6789/0 conn(0x7f96940fc910 0x7f9694060c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57384/0 (socket says 172.21.15.107:57384) 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter ms_handle_connect 0x7f96940ff710 2024-02-10T08:51:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter ms_handle_connect 0x7f96940fc910 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800078326 0 0) 0x7f96941171d0 con 0x7f96940fc910 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680003030 con 0x7f96940fc910 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter ms_handle_connect 0x7f9694103250 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115657394 0 0) 0x7f96941195b0 con 0x7f96940ff710 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.774+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96941171d0 con 0x7f96940ff710 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117993040 0 0) 0x7f96941183b0 con 0x7f9694103250 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96941195b0 con 0x7f9694103250 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515162911 0 0) 0x7f9680003030 con 0x7f96940fc910 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96941183b0 con 0x7f96940fc910 2024-02-10T08:51:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f96900020a0 con 0x7f96940fc910 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276549025 0 0) 0x7f96941171d0 con 0x7f96940ff710 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9680003030 con 0x7f96940ff710 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9684001940 con 0x7f96940ff710 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647049579 0 0) 0x7f96941195b0 con 0x7f9694103250 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96941171d0 con 0x7f9694103250 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f968c001d70 con 0x7f9694103250 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2283958897 0 0) 0x7f96941183b0 con 0x7f96940fc910 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 >> v1:172.21.15.107:6790/0 conn(0x7f96940ff710 legacy=0x7f9694061350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 >> v1:172.21.15.195:6789/0 conn(0x7f9694103250 legacy=0x7f96941158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969411a7b0 con 0x7f96940fc910 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96941173c0 con 0x7f96940fc910 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f969c1d9700 10 client.28965.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9694117830 con 0x7f96940fc910 2024-02-10T08:51:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f969c1d9700 1 librados: init done 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f9690001130 con 0x7f96940fc910 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.775+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f9690003690 con 0x7f96940fc910 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.776+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9678004f20 con 0x7f96940fc910 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.776+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f9690001770 con 0x7f96940fc910 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(830..830 src has 775..830) v4 ==== 7473+0+0 (unknown 928873182 0 0) 0x7f9690078c80 con 0x7f96940fc910 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 10 client.28965.objecter ms_dispatch 0x7f9694001550 osd_map(830..830 src has 775..830) v4 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 3 client.28965.objecter handle_osd_map got epochs [830,830] > 0 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 3 client.28965.objecter handle_osd_map decoding full epoch 830 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 20 client.28965.objecter dump_active .. 0 homeless 2024-02-10T08:51:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.777+0000 7f968affd700 10 client.28965.objecter ms_handle_connect 0x7f9680063c20 2024-02-10T08:51:12.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.781+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f969004c380 con 0x7f96940fc910 2024-02-10T08:51:12.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:12.954+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f96780051d0 con 0x7f96940fc910 2024-02-10T08:51:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.238+0000 7f968affd700 1 -- 172.21.15.107:0/2756070648 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v831) v1 ==== 113+0+0 (unknown 1641185423 0 0) 0x7f969004a7f0 con 0x7f96940fc910 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 10 librados: watch_flush enter 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 10 librados: watch_flush exit 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 20 client.28965.objecter shutdown clearing up homeless session... 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 10 client.28965.objecter successfully canceled tick 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 >> v1:172.21.15.107:6800/2634842009 conn(0x7f9680063c20 legacy=0x7f96800660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 >> v1:172.21.15.107:6789/0 conn(0x7f96940fc910 legacy=0x7f9694060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 shutdown_connections 2024-02-10T08:51:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 >> 172.21.15.107:0/2756070648 conn(0x7f96940f2510 msgr2=0x7f96941066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 shutdown_connections 2024-02-10T08:51:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 -- 172.21.15.107:0/2756070648 wait complete. 2024-02-10T08:51:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:13.241+0000 7f969c1d9700 1 librados: shutdown 2024-02-10T08:51:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T08:51:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:12.241541+0000 osd.3 (osd.3) 409 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:51:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:12.242867+0000 osd.3 (osd.3) 410 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:51:13.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:12.956095+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:13.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:12.241541+0000 osd.3 (osd.3) 409 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:51:13.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:12.242867+0000 osd.3 (osd.3) 410 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:51:13.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:12.956095+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:13.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:12.241541+0000 osd.3 (osd.3) 409 : cluster [DBG] 16.1a scrub starts 2024-02-10T08:51:13.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:12.242867+0000 osd.3 (osd.3) 410 : cluster [DBG] 16.1a scrub ok 2024-02-10T08:51:13.541 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:12.956095+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:13.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-10T08:51:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:13.239373+0000 mon.a (mon.0) 3775 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:13.239411+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:13.603425+0000 mgr.y (mgr.14553) 722 : cluster [DBG] pgmap v1297: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:13.239373+0000 mon.a (mon.0) 3775 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:13.239411+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-10T08:51:14.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:13.603425+0000 mgr.y (mgr.14553) 722 : cluster [DBG] pgmap v1297: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-02-10T08:51:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:13.239373+0000 mon.a (mon.0) 3775 : audit [INF] from='client.? v1:172.21.15.107:0/2756070648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:13.239411+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-10T08:51:14.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:13.603425+0000 mgr.y (mgr.14553) 722 : cluster [DBG] pgmap v1297: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-02-10T08:51:15.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.166290+0000 osd.3 (osd.3) 411 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:51:15.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.167790+0000 osd.3 (osd.3) 412 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:51:15.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.166290+0000 osd.3 (osd.3) 411 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:51:15.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.167790+0000 osd.3 (osd.3) 412 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:51:15.761 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:15] "GET /metrics HTTP/1.1" 200 186418 "" "Prometheus/2.33.4" 2024-02-10T08:51:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.166290+0000 osd.3 (osd.3) 411 : cluster [DBG] 16.4 scrub starts 2024-02-10T08:51:15.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.167790+0000 osd.3 (osd.3) 412 : cluster [DBG] 16.4 scrub ok 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 -- 172.21.15.107:0/2495885495 >> v1:172.21.15.107:6789/0 conn(0x7f47780fda50 legacy=0x7f47780fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 -- 172.21.15.107:0/2495885495 shutdown_connections 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 -- 172.21.15.107:0/2495885495 >> 172.21.15.107:0/2495885495 conn(0x7f47780f3640 msgr2=0x7f47780f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 -- 172.21.15.107:0/2495885495 shutdown_connections 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 -- 172.21.15.107:0/2495885495 wait complete. 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 librados: starting msgr at 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.969+0000 7f477fd1a700 1 librados: starting objecter 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 Processor -- start 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 -- start start 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 librados: setting wanted keys 2024-02-10T08:51:15.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 librados: calling monclient init 2024-02-10T08:51:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4778119550 con 0x7f47780fda50 2024-02-10T08:51:15.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f477811a730 con 0x7f47781007f0 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477fd1a700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f477811b930 con 0x7f4778104330 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477dab6700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47780fda50 0x7f47781024b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57396/0 (socket says 172.21.15.107:57396) 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477e2b7700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4778104330 0x7f4778117da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47642/0 (socket says 172.21.15.107:47642) 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f477dab6700 1 -- 172.21.15.107:0/595508309 learned_addr learned my addr 172.21.15.107:0/595508309 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter ms_handle_connect 0x7f47780fda50 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter ms_handle_connect 0x7f4778104330 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter ms_handle_connect 0x7f47781007f0 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516354345 0 0) 0x7f4778119550 con 0x7f47780fda50 2024-02-10T08:51:15.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47640038e0 con 0x7f47780fda50 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697240807 0 0) 0x7f477811b930 con 0x7f4778104330 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4778119550 con 0x7f4778104330 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934976571 0 0) 0x7f477811a730 con 0x7f47781007f0 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477811b930 con 0x7f47781007f0 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213551080 0 0) 0x7f47640038e0 con 0x7f47780fda50 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f477811a730 con 0x7f47780fda50 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4768002e10 con 0x7f47780fda50 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214318717 0 0) 0x7f4778119550 con 0x7f4778104330 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47640038e0 con 0x7f4778104330 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.970+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4770001e00 con 0x7f4778104330 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290248282 0 0) 0x7f477811b930 con 0x7f47781007f0 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4778119550 con 0x7f47781007f0 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f47740018c0 con 0x7f47781007f0 2024-02-10T08:51:15.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 851226453 0 0) 0x7f477811a730 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 >> v1:172.21.15.107:6790/0 conn(0x7f4778104330 legacy=0x7f4778117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 >> v1:172.21.15.195:6789/0 conn(0x7f47781007f0 legacy=0x7f4778102bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477811cb30 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477811bb20 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f477fd1a700 10 client.28971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f477811c010 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f477fd1a700 1 librados: init done 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f47680023a0 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.971+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4768003810 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475c004f20 con 0x7f47780fda50 2024-02-10T08:51:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4768001f00 con 0x7f47780fda50 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(831..831 src has 775..831) v4 ==== 7473+0+0 (unknown 1419514620 0 0) 0x7f4768078da0 con 0x7f47780fda50 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f476effd700 10 client.28971.objecter ms_dispatch 0x7f4778001550 osd_map(831..831 src has 775..831) v4 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f476effd700 3 client.28971.objecter handle_osd_map got epochs [831,831] > 0 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.972+0000 7f476effd700 3 client.28971.objecter handle_osd_map decoding full epoch 831 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.973+0000 7f476effd700 20 client.28971.objecter dump_active .. 0 homeless 2024-02-10T08:51:15.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.973+0000 7f476effd700 10 client.28971.objecter ms_handle_connect 0x7f4764063cd0 2024-02-10T08:51:15.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:15.977+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f476804c4e0 con 0x7f47780fda50 2024-02-10T08:51:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.153+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f475c0059e0 con 0x7f47780fda50 2024-02-10T08:51:16.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.339+0000 7f476effd700 1 -- 172.21.15.107:0/595508309 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v832) v1 ==== 133+0+0 (unknown 2935153050 0 0) 0x7f476804a950 con 0x7f47780fda50 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 10 librados: watch_flush enter 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 10 librados: watch_flush exit 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 20 client.28971.objecter shutdown clearing up homeless session... 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 10 client.28971.objecter successfully canceled tick 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4764063cd0 legacy=0x7f4764066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.341+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 >> v1:172.21.15.107:6789/0 conn(0x7f47780fda50 legacy=0x7f47781024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.342+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 shutdown_connections 2024-02-10T08:51:16.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.342+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 >> 172.21.15.107:0/595508309 conn(0x7f47780f3640 msgr2=0x7f47781077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:16.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.344+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 shutdown_connections 2024-02-10T08:51:16.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.344+0000 7f477fd1a700 1 -- 172.21.15.107:0/595508309 wait complete. 2024-02-10T08:51:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:16.344+0000 7f477fd1a700 1 librados: shutdown 2024-02-10T08:51:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:14.503902+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:14.507558+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:14.836703+0000 osd.7 (osd.7) 366 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:14.839676+0000 osd.7 (osd.7) 367 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.603822+0000 mgr.y (mgr.14553) 723 : cluster [DBG] pgmap v1298: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:16.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:16.154668+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:14.503902+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:14.507558+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:14.836703+0000 osd.7 (osd.7) 366 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:14.839676+0000 osd.7 (osd.7) 367 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.603822+0000 mgr.y (mgr.14553) 723 : cluster [DBG] pgmap v1298: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:16.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:16.154668+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:14.503902+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.7 scrub starts 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:14.507558+0000 osd.1 (osd.1) 395 : cluster [DBG] 4.7 scrub ok 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:14.836703+0000 osd.7 (osd.7) 366 : cluster [DBG] 1.0 scrub starts 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:14.839676+0000 osd.7 (osd.7) 367 : cluster [DBG] 1.0 scrub ok 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.603822+0000 mgr.y (mgr.14553) 723 : cluster [DBG] pgmap v1298: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:16.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:16.154668+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T08:51:17.159 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-10T08:51:17.159 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-10T08:51:17.159 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-10T08:51:17.159 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-10T08:51:17.159 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T08:51:17.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.499823+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.503390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.729352+0000 osd.5 (osd.5) 378 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:15.732513+0000 osd.5 (osd.5) 379 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:16.029561+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:51:17.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:16.033356+0000 osd.6 (osd.6) 359 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:16.339239+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:16.339296+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:16.787182+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:16.790185+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.499823+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.503390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.729352+0000 osd.5 (osd.5) 378 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:15.732513+0000 osd.5 (osd.5) 379 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:16.029561+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:16.033356+0000 osd.6 (osd.6) 359 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:51:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:16.339239+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:17.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:16.339296+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-10T08:51:17.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:16.787182+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-02-10T08:51:17.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:16.790185+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-02-10T08:51:17.433 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.499823+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.503390+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.729352+0000 osd.5 (osd.5) 378 : cluster [DBG] 16.11 scrub starts 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:15.732513+0000 osd.5 (osd.5) 379 : cluster [DBG] 16.11 scrub ok 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:16.029561+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.a deep-scrub starts 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:16.033356+0000 osd.6 (osd.6) 359 : cluster [DBG] 4.a deep-scrub ok 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:16.339239+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? v1:172.21.15.107:0/595508309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:16.339296+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-10T08:51:17.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:16.787182+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-02-10T08:51:17.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:16.790185+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.604284+0000 mgr.y (mgr.14553) 724 : cluster [DBG] pgmap v1300: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.661130+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.a deep-scrub starts 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.664021+0000 osd.5 (osd.5) 381 : cluster [DBG] 6.a deep-scrub ok 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.752851+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.e scrub starts 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.756063+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.e scrub ok 2024-02-10T08:51:18.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.604284+0000 mgr.y (mgr.14553) 724 : cluster [DBG] pgmap v1300: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:18.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.661130+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.a deep-scrub starts 2024-02-10T08:51:18.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.664021+0000 osd.5 (osd.5) 381 : cluster [DBG] 6.a deep-scrub ok 2024-02-10T08:51:18.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.752851+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.e scrub starts 2024-02-10T08:51:18.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.756063+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.e scrub ok 2024-02-10T08:51:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.604284+0000 mgr.y (mgr.14553) 724 : cluster [DBG] pgmap v1300: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.661130+0000 osd.5 (osd.5) 380 : cluster [DBG] 6.a deep-scrub starts 2024-02-10T08:51:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.664021+0000 osd.5 (osd.5) 381 : cluster [DBG] 6.a deep-scrub ok 2024-02-10T08:51:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.752851+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.e scrub starts 2024-02-10T08:51:18.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.756063+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.e scrub ok 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.131+0000 7f4eb6b75700 1 -- 172.21.15.107:0/3922994321 >> v1:172.21.15.107:6790/0 conn(0x7f4eb0103250 legacy=0x7f4eb01056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.131+0000 7f4eb6b75700 1 -- 172.21.15.107:0/3922994321 shutdown_connections 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.131+0000 7f4eb6b75700 1 -- 172.21.15.107:0/3922994321 >> 172.21.15.107:0/3922994321 conn(0x7f4eb00f2510 msgr2=0x7f4eb00f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.131+0000 7f4eb6b75700 1 -- 172.21.15.107:0/3922994321 shutdown_connections 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.131+0000 7f4eb6b75700 1 -- 172.21.15.107:0/3922994321 wait complete. 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 librados: starting msgr at 2024-02-10T08:51:19.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 librados: starting objecter 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 Processor -- start 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 -- start start 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 librados: setting wanted keys 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 librados: calling monclient init 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4eb00fbf20 con 0x7f4eb0103250 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4eb00fc0f0 con 0x7f4eb00fc910 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb6b75700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4eb00fc2e0 con 0x7f4eb00ff710 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eaffff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4eb00ff710 0x7f4eb0114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47652/0 (socket says 172.21.15.107:47652) 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eaffff700 1 -- 172.21.15.107:0/2984125547 learned_addr learned my addr 172.21.15.107:0/2984125547 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:19.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eb5112700 1 --1- 172.21.15.107:0/2984125547 >> v1:172.21.15.107:6789/0 conn(0x7f4eb0103250 0x7f4eb01183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57428/0 (socket says 172.21.15.107:57428) 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter ms_handle_connect 0x7f4eb00ff710 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter ms_handle_connect 0x7f4eb0103250 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800814296 0 0) 0x7f4eb00fc2e0 con 0x7f4eb00ff710 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea00038e0 con 0x7f4eb00ff710 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575550528 0 0) 0x7f4eb00fbf20 con 0x7f4eb0103250 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb00fc2e0 con 0x7f4eb0103250 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter ms_handle_connect 0x7f4eb00fc910 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.132+0000 7f4eadffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251162705 0 0) 0x7f4eb00fc0f0 con 0x7f4eb00fc910 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb00fbf20 con 0x7f4eb00fc910 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 631689179 0 0) 0x7f4ea00038e0 con 0x7f4eb00ff710 2024-02-10T08:51:19.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4eb00fc0f0 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4e9c0019e0 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606033021 0 0) 0x7f4eb00fc2e0 con 0x7f4eb0103250 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ea00038e0 con 0x7f4eb0103250 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4ea8002f80 con 0x7f4eb0103250 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934099969 0 0) 0x7f4eb00fbf20 con 0x7f4eb00fc910 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4eb00fc2e0 con 0x7f4eb00fc910 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4ea4001880 con 0x7f4eb00fc910 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708514658 0 0) 0x7f4eb00fc0f0 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 >> v1:172.21.15.195:6789/0 conn(0x7f4eb00fc910 legacy=0x7f4eb00fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 >> v1:172.21.15.107:6789/0 conn(0x7f4eb0103250 legacy=0x7f4eb01183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb011cb30 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb0119b60 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4e9c002450 con 0x7f4eb00ff710 2024-02-10T08:51:19.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4e9c005b10 con 0x7f4eb00ff710 2024-02-10T08:51:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eb6b75700 10 client.54575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb0119fd0 con 0x7f4eb00ff710 2024-02-10T08:51:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.133+0000 7f4eb6b75700 1 librados: init done 2024-02-10T08:51:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.134+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb005f7a0 con 0x7f4eb00ff710 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.134+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4e9c001fb0 con 0x7f4eb00ff710 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(832..832 src has 775..832) v4 ==== 7473+0+0 (unknown 2339875570 0 0) 0x7f4e9c078ae0 con 0x7f4eb00ff710 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 10 client.54575.objecter ms_dispatch 0x7f4eb0001550 osd_map(832..832 src has 775..832) v4 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 3 client.54575.objecter handle_osd_map got epochs [832,832] > 0 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 3 client.54575.objecter handle_osd_map decoding full epoch 832 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 20 client.54575.objecter dump_active .. 0 homeless 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.135+0000 7f4eadffb700 10 client.54575.objecter ms_handle_connect 0x7f4ea0063c80 2024-02-10T08:51:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.139+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4e9c04c180 con 0x7f4eb00ff710 2024-02-10T08:51:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.302+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4eb0108f50 con 0x7f4eb00ff710 2024-02-10T08:51:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.303+0000 7f4eadffb700 1 -- 172.21.15.107:0/2984125547 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v832) v1 ==== 74+0+22087 (unknown 1400943034 0 1167207953) 0x7f4e9c04a590 con 0x7f4eb00ff710 2024-02-10T08:51:19.304 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:51:19.304 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":832,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","created":"2024-02-10T08:27:44.316161+0000","modified":"2024-02-10T08:51:16.334505+0000","last_up_change":"2024-02-10T08:46:28.181193+0000","last_in_change":"2024-02-10T08:47:48.190549+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T08:31:09.850435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"552","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T08:33:16.728206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-10T08:33:16.960794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T08:33:18.379465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T08:33:20.354391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.16","ready_epoch":135,"last_epoch_started":134,"last_epoch_clean":134,"source_version":"0'0","target_version":"0'0"},"last_change":"809","last_force_op_resend":"0","last_force_op_resend_prenautilus":"135","last_force_op_resend_preluminous":"133","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T08:33:22.423222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-10T08:36:32.714443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"e30a7307-1dfd-4dfa-b4c4-89b83eb24167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":3321596972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":3321596972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":3321596972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":3321596972}]},"public_addr":"172.21.15.107:6801/3321596972","cluster_addr":"172.21.15.107:6802/3321596972","heartbeat_back_addr":"172.21.15.107:6804/3321596972","heartbeat_front_addr":"172.21.15.107:6803/3321596972","state":["exists","up"]},{"osd":1,"uuid":"c5fe7280-6de7-4c89-a9f2-22eebe4c1c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":133516126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":133516126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":133516126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":133516126}]},"public_addr":"172.21.15.107:6805/133516126","cluster_addr":"172.21.15.107:6806/133516126","heartbeat_back_addr":"172.21.15.107:6808/133516126","heartbeat_front_addr":"172.21.15.107:6807/133516126","state":["exists","up"]},{"osd":2,"uuid":"3ae7131b-69d5-4cbd-b1ca-29556fac7fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":819,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":3575881728}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":3575881728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":3575881728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":3575881728}]},"public_addr":"172.21.15.107:6809/3575881728","cluster_addr":"172.21.15.107:6810/3575881728","heartbeat_back_addr":"172.21.15.107:6812/3575881728","heartbeat_front_addr":"172.21.15.107:6811/3575881728","state":["exists","up"]},{"osd":3,"uuid":"1ffb2633-18bb-4048-90eb-7130e0cc7d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6813","nonce":2871825691}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6814","nonce":2871825691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6816","nonce":2871825691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6815","nonce":2871825691}]},"public_addr":"172.21.15.107:6813/2871825691","cluster_addr":"172.21.15.107:6814/2871825691","heartbeat_back_addr":"172.21.15.107:6816/2871825691","heartbeat_front_addr":"172.21.15.107:6815/2871825691","state":["exists","up"]},{"osd":4,"uuid":"1b7529a6-19fc-4435-be7f-28876750838a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":479,"up_from":633,"up_thru":819,"down_at":480,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6800","nonce":3310789251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6801","nonce":3310789251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6803","nonce":3310789251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6802","nonce":3310789251}]},"public_addr":"172.21.15.195:6800/3310789251","cluster_addr":"172.21.15.195:6801/3310789251","heartbeat_back_addr":"172.21.15.195:6803/3310789251","heartbeat_front_addr":"172.21.15.195:6802/3310789251","state":["exists","up"]},{"osd":5,"uuid":"bdc1b91c-94ce-4287-8019-535b326da3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":819,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6804","nonce":4178321486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6805","nonce":4178321486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6807","nonce":4178321486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6806","nonce":4178321486}]},"public_addr":"172.21.15.195:6804/4178321486","cluster_addr":"172.21.15.195:6805/4178321486","heartbeat_back_addr":"172.21.15.195:6807/4178321486","heartbeat_front_addr":"172.21.15.195:6806/4178321486","state":["exists","up"]},{"osd":6,"uuid":"217fdf91-33a7-40e2-a948-51b05ed63031","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6808","nonce":1417479017}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6809","nonce":1417479017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6811","nonce":1417479017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6810","nonce":1417479017}]},"public_addr":"172.21.15.195:6808/1417479017","cluster_addr":"172.21.15.195:6809/1417479017","heartbeat_back_addr":"172.21.15.195:6811/1417479017","heartbeat_front_addr":"172.21.15.195:6810/1417479017","state":["exists","up"]},{"osd":7,"uuid":"f0cd42da-4080-4e6a-843b-d73e10ce0948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6812","nonce":1934176928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6813","nonce":1934176928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6815","nonce":1934176928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6814","nonce":1934176928}]},"public_addr":"172.21.15.195:6812/1934176928","cluster_addr":"172.21.15.195:6813/1934176928","heartbeat_back_addr":"172.21.15.195:6815/1934176928","heartbeat_front_addr":"172.21.15.195:6814/1934176928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:45.021570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:20.276558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:19.507264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:18.544625+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:51:06.467805+0000","dead_epoch":479},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:30.757203+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:22.866103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:12.115680+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.b","osds":[3,1,6]},{"pgid":"2.12","osds":[3,7,1]},{"pgid":"2.13","osds":[4,2,7]},{"pgid":"2.1d","osds":[3,4,2]},{"pgid":"4.1","osds":[4,0,7]},{"pgid":"6.1","osds":[2,1,7]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"16.1d","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1675475667":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/682167559":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/3360068681":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/2916003802":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1214732392":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/810931708":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1610921635":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/942960899":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/2081774553":"2024-02-11T08:28:29.992672+0000","172.21.15.107:6800/4244374259":"2024-02-11T08:28:07.029499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":27,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":57,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":77,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":83,"snaps":[{"begin":2,"length":1}]},{"pool":93,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":106,"snaps":[{"begin":2,"length":1}]},{"pool":111,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":124,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":132,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":176,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T08:51:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 10 librados: watch_flush enter 2024-02-10T08:51:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 10 librados: watch_flush exit 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 20 client.54575.objecter shutdown clearing up homeless session... 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 10 client.54575.objecter successfully canceled tick 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 >> v1:172.21.15.107:6800/2634842009 conn(0x7f4ea0063c80 legacy=0x7f4ea0066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 >> v1:172.21.15.107:6790/0 conn(0x7f4eb00ff710 legacy=0x7f4eb0114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 shutdown_connections 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 >> 172.21.15.107:0/2984125547 conn(0x7f4eb00f2510 msgr2=0x7f4eb00612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 shutdown_connections 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 -- 172.21.15.107:0/2984125547 wait complete. 2024-02-10T08:51:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:19.305+0000 7f4eb6b75700 1 librados: shutdown 2024-02-10T08:51:19.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.582277+0000 osd.1 (osd.1) 398 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:51:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:17.586018+0000 osd.1 (osd.1) 399 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:51:19.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:19.303366+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? v1:172.21.15.107:0/2984125547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.582277+0000 osd.1 (osd.1) 398 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:51:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:17.586018+0000 osd.1 (osd.1) 399 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:51:19.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:19.303366+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? v1:172.21.15.107:0/2984125547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.582277+0000 osd.1 (osd.1) 398 : cluster [DBG] 3.0 scrub starts 2024-02-10T08:51:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:17.586018+0000 osd.1 (osd.1) 399 : cluster [DBG] 3.0 scrub ok 2024-02-10T08:51:19.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:19.303366+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? v1:172.21.15.107:0/2984125547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:20.129 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T08:51:20.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:18.537009+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:18.540720+0000 osd.1 (osd.1) 401 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:18.537009+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:18.540720+0000 osd.1 (osd.1) 401 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:19.604629+0000 mgr.y (mgr.14553) 725 : cluster [DBG] pgmap v1301: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:19.813537+0000 osd.2 (osd.2) 355 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:51:20.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:19.817006+0000 osd.2 (osd.2) 356 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:51:20.400 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:18.537009+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.2 scrub starts 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:18.540720+0000 osd.1 (osd.1) 401 : cluster [DBG] 4.2 scrub ok 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:19.604629+0000 mgr.y (mgr.14553) 725 : cluster [DBG] pgmap v1301: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:19.813537+0000 osd.2 (osd.2) 355 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:51:20.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:19.817006+0000 osd.2 (osd.2) 356 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:51:20.625 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:19.604629+0000 mgr.y (mgr.14553) 725 : cluster [DBG] pgmap v1301: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:51:20.625 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:19.813537+0000 osd.2 (osd.2) 355 : cluster [DBG] 4.1c scrub starts 2024-02-10T08:51:20.625 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:19.817006+0000 osd.2 (osd.2) 356 : cluster [DBG] 4.1c scrub ok 2024-02-10T08:51:21.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:19.945566+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:51:21.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:19.948483+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:51:21.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:19.945566+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:51:21.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:19.948483+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:51:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:19.945566+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.1a scrub starts 2024-02-10T08:51:21.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:19.948483+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.1a scrub ok 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.115+0000 7f38e4462700 1 -- 172.21.15.107:0/2619444682 >> v1:172.21.15.107:6790/0 conn(0x7f38dc0f95d0 legacy=0x7f38dc102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.115+0000 7f38e4462700 1 -- 172.21.15.107:0/2619444682 shutdown_connections 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.115+0000 7f38e4462700 1 -- 172.21.15.107:0/2619444682 >> 172.21.15.107:0/2619444682 conn(0x7f38dc0f2510 msgr2=0x7f38dc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.115+0000 7f38e4462700 1 -- 172.21.15.107:0/2619444682 shutdown_connections 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.115+0000 7f38e4462700 1 -- 172.21.15.107:0/2619444682 wait complete. 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 librados: starting msgr at 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 librados: starting objecter 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 Processor -- start 2024-02-10T08:51:22.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 -- start start 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 librados: setting wanted keys 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.116+0000 7f38e4462700 1 librados: calling monclient init 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38dc1173e0 con 0x7f38dc0f8a20 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38dc1185c0 con 0x7f38dc0f95d0 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38dc1197c0 con 0x7f38dc104350 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e21fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f38dc0f8a20 0x7f38dc0ff570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46928/0 (socket says 172.21.15.107:46928) 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e21fe700 1 -- 172.21.15.107:0/3538753242 learned_addr learned my addr 172.21.15.107:0/3538753242 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e29ff700 1 --1- 172.21.15.107:0/3538753242 >> v1:172.21.15.107:6790/0 conn(0x7f38dc104350 0x7f38dc115aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46902/0 (socket says 172.21.15.107:46902) 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter ms_handle_connect 0x7f38dc104350 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter ms_handle_connect 0x7f38dc0f8a20 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:22.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter ms_handle_connect 0x7f38dc0f95d0 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099234071 0 0) 0x7f38dc1197c0 con 0x7f38dc104350 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c80038e0 con 0x7f38dc104350 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008364741 0 0) 0x7f38dc1173e0 con 0x7f38dc0f8a20 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc1197c0 con 0x7f38dc0f8a20 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393315057 0 0) 0x7f38dc1185c0 con 0x7f38dc0f95d0 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc1173e0 con 0x7f38dc0f95d0 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3863065857 0 0) 0x7f38c80038e0 con 0x7f38dc104350 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38dc1185c0 con 0x7f38dc104350 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f38d4001e00 con 0x7f38dc104350 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624341937 0 0) 0x7f38dc1197c0 con 0x7f38dc0f8a20 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c80038e0 con 0x7f38dc0f8a20 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f38d8001880 con 0x7f38dc0f8a20 2024-02-10T08:51:22.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3060720045 0 0) 0x7f38dc1173e0 con 0x7f38dc0f95d0 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38dc1197c0 con 0x7f38dc0f95d0 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f38d0002d60 con 0x7f38dc0f95d0 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3959388117 0 0) 0x7f38dc1185c0 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 >> v1:172.21.15.195:6789/0 conn(0x7f38dc0f95d0 legacy=0x7f38dc0ffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 >> v1:172.21.15.107:6789/0 conn(0x7f38dc0f8a20 legacy=0x7f38dc0ff570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc11a9c0 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc1175d0 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 10 client.54581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38dc117a40 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.117+0000 7f38e4462700 1 librados: init done 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.118+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f38d4002770 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.118+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f38d4005d40 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.118+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c0004f20 con 0x7f38dc104350 2024-02-10T08:51:22.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.119+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f38d40028e0 con 0x7f38dc104350 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.119+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(832..832 src has 775..832) v4 ==== 7473+0+0 (unknown 2339875570 0 0) 0x7f38d4078d00 con 0x7f38dc104350 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.119+0000 7f38cf7fe700 10 client.54581.objecter ms_dispatch 0x7f38dc001550 osd_map(832..832 src has 775..832) v4 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.119+0000 7f38cf7fe700 3 client.54581.objecter handle_osd_map got epochs [832,832] > 0 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.119+0000 7f38cf7fe700 3 client.54581.objecter handle_osd_map decoding full epoch 832 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.120+0000 7f38cf7fe700 20 client.54581.objecter dump_active .. 0 homeless 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.120+0000 7f38cf7fe700 10 client.54581.objecter ms_handle_connect 0x7f38c8063cb0 2024-02-10T08:51:22.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.124+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f38d404c340 con 0x7f38dc104350 2024-02-10T08:51:22.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.289+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38c00051d0 con 0x7f38dc104350 2024-02-10T08:51:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.290+0000 7f38cf7fe700 1 -- 172.21.15.107:0/3538753242 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v832) v1 ==== 74+0+22087 (unknown 1400943034 0 1167207953) 0x7f38d404a750 con 0x7f38dc104350 2024-02-10T08:51:22.291 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:51:22.291 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":832,"fsid":"275c2754-c7ee-11ee-95b8-87774f69a715","created":"2024-02-10T08:27:44.316161+0000","modified":"2024-02-10T08:51:16.334505+0000","last_up_change":"2024-02-10T08:46:28.181193+0000","last_in_change":"2024-02-10T08:47:48.190549+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-10T08:31:09.850435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"552","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T08:33:16.728206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-10T08:33:16.960794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T08:33:18.379465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T08:33:20.354391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.16","ready_epoch":135,"last_epoch_started":134,"last_epoch_clean":134,"source_version":"0'0","target_version":"0'0"},"last_change":"809","last_force_op_resend":"0","last_force_op_resend_prenautilus":"135","last_force_op_resend_preluminous":"133","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T08:33:22.423222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-10T08:36:32.714443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"e30a7307-1dfd-4dfa-b4c4-89b83eb24167","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":3321596972}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":3321596972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":3321596972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":3321596972}]},"public_addr":"172.21.15.107:6801/3321596972","cluster_addr":"172.21.15.107:6802/3321596972","heartbeat_back_addr":"172.21.15.107:6804/3321596972","heartbeat_front_addr":"172.21.15.107:6803/3321596972","state":["exists","up"]},{"osd":1,"uuid":"c5fe7280-6de7-4c89-a9f2-22eebe4c1c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":133516126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":133516126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":133516126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":133516126}]},"public_addr":"172.21.15.107:6805/133516126","cluster_addr":"172.21.15.107:6806/133516126","heartbeat_back_addr":"172.21.15.107:6808/133516126","heartbeat_front_addr":"172.21.15.107:6807/133516126","state":["exists","up"]},{"osd":2,"uuid":"3ae7131b-69d5-4cbd-b1ca-29556fac7fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":819,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":3575881728}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":3575881728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":3575881728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":3575881728}]},"public_addr":"172.21.15.107:6809/3575881728","cluster_addr":"172.21.15.107:6810/3575881728","heartbeat_back_addr":"172.21.15.107:6812/3575881728","heartbeat_front_addr":"172.21.15.107:6811/3575881728","state":["exists","up"]},{"osd":3,"uuid":"1ffb2633-18bb-4048-90eb-7130e0cc7d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6813","nonce":2871825691}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6814","nonce":2871825691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6816","nonce":2871825691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6815","nonce":2871825691}]},"public_addr":"172.21.15.107:6813/2871825691","cluster_addr":"172.21.15.107:6814/2871825691","heartbeat_back_addr":"172.21.15.107:6816/2871825691","heartbeat_front_addr":"172.21.15.107:6815/2871825691","state":["exists","up"]},{"osd":4,"uuid":"1b7529a6-19fc-4435-be7f-28876750838a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":479,"up_from":633,"up_thru":819,"down_at":480,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6800","nonce":3310789251}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6801","nonce":3310789251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6803","nonce":3310789251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6802","nonce":3310789251}]},"public_addr":"172.21.15.195:6800/3310789251","cluster_addr":"172.21.15.195:6801/3310789251","heartbeat_back_addr":"172.21.15.195:6803/3310789251","heartbeat_front_addr":"172.21.15.195:6802/3310789251","state":["exists","up"]},{"osd":5,"uuid":"bdc1b91c-94ce-4287-8019-535b326da3e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":819,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6804","nonce":4178321486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6805","nonce":4178321486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6807","nonce":4178321486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6806","nonce":4178321486}]},"public_addr":"172.21.15.195:6804/4178321486","cluster_addr":"172.21.15.195:6805/4178321486","heartbeat_back_addr":"172.21.15.195:6807/4178321486","heartbeat_front_addr":"172.21.15.195:6806/4178321486","state":["exists","up"]},{"osd":6,"uuid":"217fdf91-33a7-40e2-a948-51b05ed63031","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6808","nonce":1417479017}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6809","nonce":1417479017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6811","nonce":1417479017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6810","nonce":1417479017}]},"public_addr":"172.21.15.195:6808/1417479017","cluster_addr":"172.21.15.195:6809/1417479017","heartbeat_back_addr":"172.21.15.195:6811/1417479017","heartbeat_front_addr":"172.21.15.195:6810/1417479017","state":["exists","up"]},{"osd":7,"uuid":"f0cd42da-4080-4e6a-843b-d73e10ce0948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6812","nonce":1934176928}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6813","nonce":1934176928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6815","nonce":1934176928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.195:6814","nonce":1934176928}]},"public_addr":"172.21.15.195:6812/1934176928","cluster_addr":"172.21.15.195:6813/1934176928","heartbeat_back_addr":"172.21.15.195:6815/1934176928","heartbeat_front_addr":"172.21.15.195:6814/1934176928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:45.021570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:20.276558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:19.507264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:18.544625+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:51:06.467805+0000","dead_epoch":479},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:30.757203+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:22.866103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T08:50:12.115680+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.b","osds":[3,1,6]},{"pgid":"2.12","osds":[3,7,1]},{"pgid":"2.13","osds":[4,2,7]},{"pgid":"2.1d","osds":[3,4,2]},{"pgid":"4.1","osds":[4,0,7]},{"pgid":"6.1","osds":[2,1,7]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"16.1d","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1675475667":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/682167559":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/3360068681":"2024-02-11T08:33:39.974787+0000","172.21.15.107:0/2916003802":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1214732392":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/810931708":"2024-02-11T08:28:29.992672+0000","172.21.15.107:0/1610921635":"2024-02-11T08:28:07.029499+0000","172.21.15.107:0/942960899":"2024-02-11T08:33:39.974787+0000","172.21.15.107:6800/2081774553":"2024-02-11T08:28:29.992672+0000","172.21.15.107:6800/4244374259":"2024-02-11T08:28:07.029499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":27,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":57,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":77,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":83,"snaps":[{"begin":2,"length":1}]},{"pool":93,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":106,"snaps":[{"begin":2,"length":1}]},{"pool":111,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":124,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":132,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":176,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 10 librados: watch_flush enter 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 10 librados: watch_flush exit 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 20 client.54581.objecter shutdown clearing up homeless session... 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 10 client.54581.objecter successfully canceled tick 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 >> v1:172.21.15.107:6800/2634842009 conn(0x7f38c8063cb0 legacy=0x7f38c8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 >> v1:172.21.15.107:6790/0 conn(0x7f38dc104350 legacy=0x7f38dc115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 shutdown_connections 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 >> 172.21.15.107:0/3538753242 conn(0x7f38dc0f2510 msgr2=0x7f38dc1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 shutdown_connections 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 -- 172.21.15.107:0/3538753242 wait complete. 2024-02-10T08:51:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:22.292+0000 7f38e4462700 1 librados: shutdown 2024-02-10T08:51:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:20.701834+0000 osd.5 (osd.5) 382 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:51:22.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:20.705697+0000 osd.5 (osd.5) 383 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:20.993397+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 deep-scrub starts 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:20.998796+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 deep-scrub ok 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.605312+0000 mgr.y (mgr.14553) 726 : cluster [DBG] pgmap v1302: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.677150+0000 osd.5 (osd.5) 384 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.680617+0000 osd.5 (osd.5) 385 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:21.943929+0000 mgr.y (mgr.14553) 727 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.997094+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1a deep-scrub starts 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.206646+0000 osd.3 (osd.3) 413 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.207623+0000 osd.3 (osd.3) 414 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:22.290755+0000 mon.c (mon.2) 516 : audit [DBG] from='client.? v1:172.21.15.107:0/3538753242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:22.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:20.701834+0000 osd.5 (osd.5) 382 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:20.705697+0000 osd.5 (osd.5) 383 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:20.993397+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 deep-scrub starts 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:20.998796+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 deep-scrub ok 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.605312+0000 mgr.y (mgr.14553) 726 : cluster [DBG] pgmap v1302: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.677150+0000 osd.5 (osd.5) 384 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.680617+0000 osd.5 (osd.5) 385 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:21.943929+0000 mgr.y (mgr.14553) 727 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.997094+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1a deep-scrub starts 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.206646+0000 osd.3 (osd.3) 413 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.207623+0000 osd.3 (osd.3) 414 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:51:22.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:22.290755+0000 mon.c (mon.2) 516 : audit [DBG] from='client.? v1:172.21.15.107:0/3538753242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:20.701834+0000 osd.5 (osd.5) 382 : cluster [DBG] 4.4 scrub starts 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:20.705697+0000 osd.5 (osd.5) 383 : cluster [DBG] 4.4 scrub ok 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:20.993397+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 deep-scrub starts 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:20.998796+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 deep-scrub ok 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.605312+0000 mgr.y (mgr.14553) 726 : cluster [DBG] pgmap v1302: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.677150+0000 osd.5 (osd.5) 384 : cluster [DBG] 16.3 scrub starts 2024-02-10T08:51:22.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.680617+0000 osd.5 (osd.5) 385 : cluster [DBG] 16.3 scrub ok 2024-02-10T08:51:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:21.943929+0000 mgr.y (mgr.14553) 727 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.997094+0000 osd.4 (osd.4) 127 : cluster [DBG] 4.1a deep-scrub starts 2024-02-10T08:51:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.206646+0000 osd.3 (osd.3) 413 : cluster [DBG] 6.6 scrub starts 2024-02-10T08:51:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.207623+0000 osd.3 (osd.3) 414 : cluster [DBG] 6.6 scrub ok 2024-02-10T08:51:22.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:22.290755+0000 mon.c (mon.2) 516 : audit [DBG] from='client.? v1:172.21.15.107:0/3538753242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T08:51:23.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T08:51:23.347 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:23.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.491588+0000 osd.1 (osd.1) 402 : cluster [DBG] 16.b scrub starts 2024-02-10T08:51:23.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:21.495228+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub ok 2024-02-10T08:51:23.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.002692+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1a deep-scrub ok 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.673148+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.676829+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.491588+0000 osd.1 (osd.1) 402 : cluster [DBG] 16.b scrub starts 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:21.495228+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub ok 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.002692+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1a deep-scrub ok 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.673148+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:51:23.618 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.676829+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:51:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.491588+0000 osd.1 (osd.1) 402 : cluster [DBG] 16.b scrub starts 2024-02-10T08:51:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:21.495228+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub ok 2024-02-10T08:51:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.002692+0000 osd.4 (osd.4) 128 : cluster [DBG] 4.1a deep-scrub ok 2024-02-10T08:51:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.673148+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-02-10T08:51:23.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:23 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.676829+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-02-10T08:51:24.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.536192+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:51:24.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:22.539348+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:51:24.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:23.550795+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:23.551082+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:23.605764+0000 mgr.y (mgr.14553) 728 : cluster [DBG] pgmap v1303: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.536192+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:22.539348+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:23.550795+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:23.551082+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:24.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:24 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:23.605764+0000 mgr.y (mgr.14553) 728 : cluster [DBG] pgmap v1303: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.536192+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.15 deep-scrub starts 2024-02-10T08:51:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:22.539348+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.15 deep-scrub ok 2024-02-10T08:51:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:23.550795+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:23.551082+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:24.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:23.605764+0000 mgr.y (mgr.14553) 728 : cluster [DBG] pgmap v1303: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.244+0000 7f2106541700 1 -- 172.21.15.107:0/698797190 >> v1:172.21.15.107:6790/0 conn(0x7f21000ff720 legacy=0x7f2100101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.244+0000 7f2106541700 1 -- 172.21.15.107:0/698797190 shutdown_connections 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.244+0000 7f2106541700 1 -- 172.21.15.107:0/698797190 >> 172.21.15.107:0/698797190 conn(0x7f21000f2540 msgr2=0x7f21000f4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.244+0000 7f2106541700 1 -- 172.21.15.107:0/698797190 shutdown_connections 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.244+0000 7f2106541700 1 -- 172.21.15.107:0/698797190 wait complete. 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 librados: starting msgr at 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 librados: starting objecter 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 Processor -- start 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 -- start start 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 librados: setting wanted keys 2024-02-10T08:51:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 librados: calling monclient init 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2100119500 con 0x7f2100103260 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f210011a700 con 0x7f21000fc920 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f2106541700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f210011b900 con 0x7f21000ff720 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.245+0000 7f20ff7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f21000ff720 0x7f21000612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46916/0 (socket says 172.21.15.107:46916) 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20ff7fe700 1 -- 172.21.15.107:0/2931963554 learned_addr learned my addr 172.21.15.107:0/2931963554 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter ms_handle_connect 0x7f2100103260 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter ms_handle_connect 0x7f21000ff720 2024-02-10T08:51:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361072101 0 0) 0x7f210011b900 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8003030 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter ms_handle_connect 0x7f21000fc920 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536780113 0 0) 0x7f2100119500 con 0x7f2100103260 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210011b900 con 0x7f2100103260 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 629710788 0 0) 0x7f20e8003030 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2100119500 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f20f4003030 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668289231 0 0) 0x7f210011a700 con 0x7f21000fc920 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8003030 con 0x7f21000fc920 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3580542026 0 0) 0x7f2100119500 con 0x7f21000ff720 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 >> v1:172.21.15.195:6789/0 conn(0x7f21000fc920 legacy=0x7f2100060bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 >> v1:172.21.15.107:6789/0 conn(0x7f2100103260 legacy=0x7f2100117ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210011cb00 con 0x7f21000ff720 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210011baf0 con 0x7f21000ff720 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.246+0000 7f2106541700 10 client.54587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f210011bfe0 con 0x7f21000ff720 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 1 librados: init done 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f20f4000ed0 con 0x7f21000ff720 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f20f40038e0 con 0x7f21000ff720 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 10 client.54587.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 15 client.54587.objecter _session_command_op_assign -1 1 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 20 client.54587.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 10 client.54587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.247+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20ec0028e0 con 0x7f21000ff720 2024-02-10T08:51:25.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.248+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f20f4003a40 con 0x7f21000ff720 2024-02-10T08:51:25.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f20f4078db0 con 0x7f21000ff720 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 10 client.54587.objecter ms_dispatch 0x7f2100001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 3 client.54587.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 3 client.54587.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 10 client.54587.objecter checking command 1 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter _get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 15 client.54587.objecter _session_command_op_remove -1 1 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter _get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 15 client.54587.objecter _session_command_op_assign 0 1 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 4 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 10 client.54587.objecter _send_command 1 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6801/3321596972 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20e80738b0 con 0x7f20e806fc40 2024-02-10T08:51:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 20 client.54587.objecter dump_active .. 0 homeless 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f20f4079140 con 0x7f21000ff720 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 10 client.54587.objecter ms_handle_connect 0x7f20e8063bc0 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.249+0000 7f20fd7fa700 10 client.54587.objecter ms_handle_connect 0x7f20e806fc40 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.250+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== osd.0 v1:172.21.15.107:6801/3321596972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f20e80738b0 con 0x7f20e806fc40 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.250+0000 7f20fd7fa700 10 client.54587.objecter ms_dispatch 0x7f2100001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.250+0000 7f20fd7fa700 10 client.54587.objecter _finish_command 1 = system:0 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.250+0000 7f20fd7fa700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 4 2024-02-10T08:51:25.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.250+0000 7f20fd7fa700 15 client.54587.objecter _session_command_op_remove 0 1 2024-02-10T08:51:25.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 10 client.54587.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 15 client.54587.objecter _session_command_op_assign -1 2 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 20 client.54587.objecter _get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 20 client.54587.objecter _get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 15 client.54587.objecter _session_command_op_remove -1 2 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 20 client.54587.objecter get_session s=0x7f20e806f590 osd=0 3 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 15 client.54587.objecter _session_command_op_assign 0 2 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 4 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 10 client.54587.objecter _send_command 2 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 --> v1:172.21.15.107:6801/3321596972 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f20ec003c90 con 0x7f20e806fc40 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f20fd7fa700 1 -- 172.21.15.107:0/2931963554 <== osd.0 v1:172.21.15.107:6801/3321596972 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3652764523) 0x7f20ec003c90 con 0x7f20e806fc40 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f20fd7fa700 10 client.54587.objecter ms_dispatch 0x7f2100001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:25.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f20fd7fa700 10 client.54587.objecter _finish_command 2 = system:0 2024-02-10T08:51:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f20fd7fa700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 4 2024-02-10T08:51:25.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.265+0000 7f20fd7fa700 15 client.54587.objecter _session_command_op_remove 0 2 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 10 librados: watch_flush enter 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 10 librados: watch_flush exit 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 10 client.54587.objecter close_session for osd.0 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 >> v1:172.21.15.107:6801/3321596972 conn(0x7f20e806fc40 legacy=0x7f20e80720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 20 client.54587.objecter put_session s=0x7f20e806f590 osd=0 1 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 20 client.54587.objecter shutdown clearing up homeless session... 2024-02-10T08:51:25.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 10 client.54587.objecter successfully canceled tick 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 >> v1:172.21.15.107:6800/2634842009 conn(0x7f20e8063bc0 legacy=0x7f20e8066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 >> v1:172.21.15.107:6790/0 conn(0x7f21000ff720 legacy=0x7f21000612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2104ade700 1 -- 172.21.15.107:0/2931963554 reap_dead start 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 shutdown_connections 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 >> 172.21.15.107:0/2931963554 conn(0x7f21000f2540 msgr2=0x7f21001066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 shutdown_connections 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 -- 172.21.15.107:0/2931963554 wait complete. 2024-02-10T08:51:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:25.266+0000 7f2106541700 1 librados: shutdown 2024-02-10T08:51:25.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:24.368569+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-10T08:51:25.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:24.368569+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-10T08:51:25.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:24.368569+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-10T08:51:25.920 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:25] "GET /metrics HTTP/1.1" 200 186425 "" "Prometheus/2.33.4" 2024-02-10T08:51:25.939 INFO:teuthology.orchestra.run.smithi107.stdout:34359738623 2024-02-10T08:51:25.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T08:51:26.207 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:24.528423+0000 osd.1 (osd.1) 406 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:51:26.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:24.531321+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:51:26.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:25.606226+0000 mgr.y (mgr.14553) 729 : cluster [DBG] pgmap v1305: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 B/s rd, 0 op/s 2024-02-10T08:51:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:24.528423+0000 osd.1 (osd.1) 406 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:51:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:24.531321+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:51:26.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:26 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:25.606226+0000 mgr.y (mgr.14553) 729 : cluster [DBG] pgmap v1305: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 B/s rd, 0 op/s 2024-02-10T08:51:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:24.528423+0000 osd.1 (osd.1) 406 : cluster [DBG] 5.12 scrub starts 2024-02-10T08:51:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:24.531321+0000 osd.1 (osd.1) 407 : cluster [DBG] 5.12 scrub ok 2024-02-10T08:51:26.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:25.606226+0000 mgr.y (mgr.14553) 729 : cluster [DBG] pgmap v1305: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 B/s rd, 0 op/s 2024-02-10T08:51:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:27.218326+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:51:27.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:27.219833+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:51:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:27.218326+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:51:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:27.219833+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:51:27.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:27.218326+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.17 scrub starts 2024-02-10T08:51:27.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:27 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:27.219833+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.17 scrub ok 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 -- 172.21.15.107:0/2824969450 >> v1:172.21.15.107:6790/0 conn(0x7f206c0ff710 legacy=0x7f206c101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 -- 172.21.15.107:0/2824969450 shutdown_connections 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 -- 172.21.15.107:0/2824969450 >> 172.21.15.107:0/2824969450 conn(0x7f206c0f2510 msgr2=0x7f206c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 -- 172.21.15.107:0/2824969450 shutdown_connections 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 -- 172.21.15.107:0/2824969450 wait complete. 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 librados: starting msgr at 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 librados: starting objecter 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.982+0000 7f2071da9700 1 Processor -- start 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 -- start start 2024-02-10T08:51:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 librados: setting wanted keys 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 librados: calling monclient init 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f206c117360 con 0x7f206c0fc910 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f206c118540 con 0x7f206c0ff710 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f2071da9700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f206c119740 con 0x7f206c103250 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f206b7fe700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f206c0fc910 0x7f206c060d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46956/0 (socket says 172.21.15.107:46956) 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f206bfff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f206c103250 0x7f206c115a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46944/0 (socket says 172.21.15.107:46944) 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f206b7fe700 1 -- 172.21.15.107:0/2259095664 learned_addr learned my addr 172.21.15.107:0/2259095664 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter ms_handle_connect 0x7f206c103250 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter ms_handle_connect 0x7f206c0fc910 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858421046 0 0) 0x7f206c119740 con 0x7f206c103250 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2058003030 con 0x7f206c103250 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter ms_handle_connect 0x7f206c0ff710 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429913123 0 0) 0x7f206c117360 con 0x7f206c0fc910 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c119740 con 0x7f206c0fc910 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801435537 0 0) 0x7f206c118540 con 0x7f206c0ff710 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c117360 con 0x7f206c0ff710 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 46385405 0 0) 0x7f2058003030 con 0x7f206c103250 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f206c118540 con 0x7f206c103250 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2064001bc0 con 0x7f206c103250 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917333404 0 0) 0x7f206c119740 con 0x7f206c0fc910 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2058003030 con 0x7f206c0fc910 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f205c0019f0 con 0x7f206c0fc910 2024-02-10T08:51:27.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265324860 0 0) 0x7f206c117360 con 0x7f206c0ff710 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f206c119740 con 0x7f206c0ff710 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2597658815 0 0) 0x7f206c118540 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.983+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 >> v1:172.21.15.195:6789/0 conn(0x7f206c0ff710 legacy=0x7f206c061440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 >> v1:172.21.15.107:6789/0 conn(0x7f206c0fc910 legacy=0x7f206c060d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c11a940 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c117550 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 10 client.54593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c1179c0 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 1 librados: init done 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2064002760 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2064005df0 con 0x7f206c103250 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 10 client.54593.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 15 client.54593.objecter _session_command_op_assign -1 1 2024-02-10T08:51:27.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 20 client.54593.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 10 client.54593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.984+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f204c0028b0 con 0x7f206c103250 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f20640022c0 con 0x7f206c103250 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f2064078d50 con 0x7f206c103250 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 10 client.54593.objecter ms_dispatch 0x7f206c001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 3 client.54593.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 3 client.54593.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 10 client.54593.objecter checking command 1 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter _get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 15 client.54593.objecter _session_command_op_remove -1 1 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter _get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:27.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 15 client.54593.objecter _session_command_op_assign 1 1 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 4 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 10 client.54593.objecter _send_command 1 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6805/133516126 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20580738f0 con 0x7f205806fbe0 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 20 client.54593.objecter dump_active .. 0 homeless 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f2064079110 con 0x7f206c103250 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.985+0000 7f20697fa700 10 client.54593.objecter ms_handle_connect 0x7f2058063c40 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 10 client.54593.objecter ms_handle_connect 0x7f205806fbe0 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== osd.1 v1:172.21.15.107:6805/133516126 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f20580738f0 con 0x7f205806fbe0 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 10 client.54593.objecter ms_dispatch 0x7f206c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 10 client.54593.objecter _finish_command 1 = system:0 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 4 2024-02-10T08:51:27.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:27.986+0000 7f20697fa700 15 client.54593.objecter _session_command_op_remove 1 1 2024-02-10T08:51:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 client.54593.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 15 client.54593.objecter _session_command_op_assign -1 2 2024-02-10T08:51:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter _get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter _get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 15 client.54593.objecter _session_command_op_remove -1 2 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter get_session s=0x7f205806f530 osd=1 3 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 15 client.54593.objecter _session_command_op_assign 1 2 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 4 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 client.54593.objecter _send_command 2 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 --> v1:172.21.15.107:6805/133516126 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f204c001140 con 0x7f205806fbe0 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f20697fa700 1 -- 172.21.15.107:0/2259095664 <== osd.1 v1:172.21.15.107:6805/133516126 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1957187935) 0x7f204c001140 con 0x7f205806fbe0 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f20697fa700 10 client.54593.objecter ms_dispatch 0x7f206c001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f20697fa700 10 client.54593.objecter _finish_command 2 = system:0 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f20697fa700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 4 2024-02-10T08:51:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f20697fa700 15 client.54593.objecter _session_command_op_remove 1 2 2024-02-10T08:51:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 librados: watch_flush enter 2024-02-10T08:51:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 librados: watch_flush exit 2024-02-10T08:51:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 client.54593.objecter close_session for osd.1 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 >> v1:172.21.15.107:6805/133516126 conn(0x7f205806fbe0 legacy=0x7f2058072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter put_session s=0x7f205806f530 osd=1 1 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 20 client.54593.objecter shutdown clearing up homeless session... 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 10 client.54593.objecter successfully canceled tick 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2058063c40 legacy=0x7f2058066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 >> v1:172.21.15.107:6790/0 conn(0x7f206c103250 legacy=0x7f206c115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f206bfff700 1 -- 172.21.15.107:0/2259095664 reap_dead start 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 shutdown_connections 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 >> 172.21.15.107:0/2259095664 conn(0x7f206c0f2510 msgr2=0x7f206c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.001+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 shutdown_connections 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.002+0000 7f2071da9700 1 -- 172.21.15.107:0/2259095664 wait complete. 2024-02-10T08:51:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:28.002+0000 7f2071da9700 1 librados: shutdown 2024-02-10T08:51:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:27.607044+0000 mgr.y (mgr.14553) 730 : cluster [DBG] pgmap v1306: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:28.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:27.807951+0000 osd.2 (osd.2) 357 : cluster [DBG] 4.f deep-scrub starts 2024-02-10T08:51:28.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:27.814015+0000 osd.2 (osd.2) 358 : cluster [DBG] 4.f deep-scrub ok 2024-02-10T08:51:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:27.607044+0000 mgr.y (mgr.14553) 730 : cluster [DBG] pgmap v1306: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:27.807951+0000 osd.2 (osd.2) 357 : cluster [DBG] 4.f deep-scrub starts 2024-02-10T08:51:28.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:27.814015+0000 osd.2 (osd.2) 358 : cluster [DBG] 4.f deep-scrub ok 2024-02-10T08:51:28.711 INFO:teuthology.orchestra.run.smithi107.stdout:55834575098 2024-02-10T08:51:28.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T08:51:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:27.607044+0000 mgr.y (mgr.14553) 730 : cluster [DBG] pgmap v1306: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:27.807951+0000 osd.2 (osd.2) 357 : cluster [DBG] 4.f deep-scrub starts 2024-02-10T08:51:28.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:28 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:27.814015+0000 osd.2 (osd.2) 358 : cluster [DBG] 4.f deep-scrub ok 2024-02-10T08:51:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:28.918101+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:28.921010+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.210956+0000 osd.3 (osd.3) 417 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.212758+0000 osd.3 (osd.3) 418 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:28.918101+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:51:29.517 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:28.921010+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:51:29.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.210956+0000 osd.3 (osd.3) 417 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:51:29.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:29 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.212758+0000 osd.3 (osd.3) 418 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:51:29.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:28.918101+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.8 scrub starts 2024-02-10T08:51:29.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:28.921010+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.8 scrub ok 2024-02-10T08:51:29.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.210956+0000 osd.3 (osd.3) 417 : cluster [DBG] 4.1d scrub starts 2024-02-10T08:51:29.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:29 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.212758+0000 osd.3 (osd.3) 418 : cluster [DBG] 4.1d scrub ok 2024-02-10T08:51:30.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:28.706385+0000 osd.5 (osd.5) 388 : cluster [DBG] 16.c scrub starts 2024-02-10T08:51:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:28.709357+0000 osd.5 (osd.5) 389 : cluster [DBG] 16.c scrub ok 2024-02-10T08:51:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.007297+0000 osd.6 (osd.6) 364 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:51:30.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.010280+0000 osd.6 (osd.6) 365 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:51:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.607456+0000 mgr.y (mgr.14553) 731 : cluster [DBG] pgmap v1307: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.899642+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:51:30.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:30 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:29.900771+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:51:30.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:28.706385+0000 osd.5 (osd.5) 388 : cluster [DBG] 16.c scrub starts 2024-02-10T08:51:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:28.709357+0000 osd.5 (osd.5) 389 : cluster [DBG] 16.c scrub ok 2024-02-10T08:51:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.007297+0000 osd.6 (osd.6) 364 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:51:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.010280+0000 osd.6 (osd.6) 365 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:51:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.607456+0000 mgr.y (mgr.14553) 731 : cluster [DBG] pgmap v1307: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.899642+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:51:30.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:29.900771+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:51:30.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:28.706385+0000 osd.5 (osd.5) 388 : cluster [DBG] 16.c scrub starts 2024-02-10T08:51:30.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:28.709357+0000 osd.5 (osd.5) 389 : cluster [DBG] 16.c scrub ok 2024-02-10T08:51:30.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.007297+0000 osd.6 (osd.6) 364 : cluster [DBG] 2.11 scrub starts 2024-02-10T08:51:30.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.010280+0000 osd.6 (osd.6) 365 : cluster [DBG] 2.11 scrub ok 2024-02-10T08:51:30.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.607456+0000 mgr.y (mgr.14553) 731 : cluster [DBG] pgmap v1307: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:30.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.899642+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.4 scrub starts 2024-02-10T08:51:30.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:29.900771+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.4 scrub ok 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 -- 172.21.15.107:0/4060765051 >> v1:172.21.15.107:6789/0 conn(0x7f6b640fda50 legacy=0x7f6b640fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 -- 172.21.15.107:0/4060765051 shutdown_connections 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 -- 172.21.15.107:0/4060765051 >> 172.21.15.107:0/4060765051 conn(0x7f6b640f3640 msgr2=0x7f6b640f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 -- 172.21.15.107:0/4060765051 shutdown_connections 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 -- 172.21.15.107:0/4060765051 wait complete. 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 librados: starting msgr at 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.940+0000 7f6b6887e700 1 librados: starting objecter 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 Processor -- start 2024-02-10T08:51:30.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 -- start start 2024-02-10T08:51:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 librados: setting wanted keys 2024-02-10T08:51:30.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 librados: calling monclient init 2024-02-10T08:51:30.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b64117570 con 0x7f6b641007f0 2024-02-10T08:51:30.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b64118750 con 0x7f6b64104330 2024-02-10T08:51:30.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6887e700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b64119950 con 0x7f6b640fda50 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6259c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6b640fda50 0x7f6b640f9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38108/0 (socket says 172.21.15.107:38108) 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b6259c700 1 -- 172.21.15.107:0/3070790548 learned_addr learned my addr 172.21.15.107:0/3070790548 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b61d9b700 1 --1- 172.21.15.107:0/3070790548 >> v1:172.21.15.107:6789/0 conn(0x7f6b641007f0 0x7f6b640fa4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35812/0 (socket says 172.21.15.107:35812) 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter ms_handle_connect 0x7f6b641007f0 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter ms_handle_connect 0x7f6b640fda50 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161638758 0 0) 0x7f6b64117570 con 0x7f6b641007f0 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b4c003030 con 0x7f6b641007f0 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317964522 0 0) 0x7f6b64119950 con 0x7f6b640fda50 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b64117570 con 0x7f6b640fda50 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter ms_handle_connect 0x7f6b64104330 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.941+0000 7f6b537fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512049288 0 0) 0x7f6b4c003030 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b64119950 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318190587 0 0) 0x7f6b64117570 con 0x7f6b640fda50 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b4c003030 con 0x7f6b640fda50 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6b580018e0 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6b54001f90 con 0x7f6b640fda50 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638662335 0 0) 0x7f6b64118750 con 0x7f6b64104330 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b64117570 con 0x7f6b64104330 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4207048222 0 0) 0x7f6b64119950 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 >> v1:172.21.15.107:6790/0 conn(0x7f6b640fda50 legacy=0x7f6b640f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 >> v1:172.21.15.195:6789/0 conn(0x7f6b64104330 legacy=0x7f6b64115c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6411ab50 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b64119b40 con 0x7f6b641007f0 2024-02-10T08:51:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b6887e700 10 client.29001.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b64119fb0 con 0x7f6b641007f0 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b6887e700 1 librados: init done 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f6b58001ff0 con 0x7f6b641007f0 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.942+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6b58005a90 con 0x7f6b641007f0 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b6887e700 10 client.29001.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b6887e700 15 client.29001.objecter _session_command_op_assign -1 1 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b6887e700 20 client.29001.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b6887e700 10 client.29001.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b440028b0 con 0x7f6b641007f0 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.943+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f6b58005bf0 con 0x7f6b641007f0 2024-02-10T08:51:30.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f6b58078970 con 0x7f6b641007f0 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 10 client.29001.objecter ms_dispatch 0x7f6b64001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 3 client.29001.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 3 client.29001.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 10 client.29001.objecter checking command 1 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter _get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 15 client.29001.objecter _session_command_op_remove -1 1 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter _get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 15 client.29001.objecter _session_command_op_assign 2 1 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 4 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 10 client.29001.objecter _send_command 1 2024-02-10T08:51:30.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6809/3575881728 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b4c073960 con 0x7f6b4c06fc50 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 20 client.29001.objecter dump_active .. 0 homeless 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f6b58078da0 con 0x7f6b641007f0 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 10 client.29001.objecter ms_handle_connect 0x7f6b4c063c50 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.944+0000 7f6b537fe700 10 client.29001.objecter ms_handle_connect 0x7f6b4c06fc50 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.945+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== osd.2 v1:172.21.15.107:6809/3575881728 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6b4c073960 con 0x7f6b4c06fc50 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.945+0000 7f6b537fe700 10 client.29001.objecter ms_dispatch 0x7f6b64001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.945+0000 7f6b537fe700 10 client.29001.objecter _finish_command 1 = system:0 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.945+0000 7f6b537fe700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 4 2024-02-10T08:51:30.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.945+0000 7f6b537fe700 15 client.29001.objecter _session_command_op_remove 2 1 2024-02-10T08:51:30.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 10 client.29001.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 15 client.29001.objecter _session_command_op_assign -1 2 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 20 client.29001.objecter _get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 20 client.29001.objecter _get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 15 client.29001.objecter _session_command_op_remove -1 2 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 20 client.29001.objecter get_session s=0x7f6b4c06f5a0 osd=2 3 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 15 client.29001.objecter _session_command_op_assign 2 2 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 4 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 10 client.29001.objecter _send_command 2 2024-02-10T08:51:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.958+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 --> v1:172.21.15.107:6809/3575881728 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b44001140 con 0x7f6b4c06fc50 2024-02-10T08:51:30.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b537fe700 1 -- 172.21.15.107:0/3070790548 <== osd.2 v1:172.21.15.107:6809/3575881728 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3015930934) 0x7f6b44001140 con 0x7f6b4c06fc50 2024-02-10T08:51:30.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b537fe700 10 client.29001.objecter ms_dispatch 0x7f6b64001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:30.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b537fe700 10 client.29001.objecter _finish_command 2 = system:0 2024-02-10T08:51:30.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b537fe700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 4 2024-02-10T08:51:30.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b537fe700 15 client.29001.objecter _session_command_op_remove 2 2 2024-02-10T08:51:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 10 librados: watch_flush enter 2024-02-10T08:51:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 10 librados: watch_flush exit 2024-02-10T08:51:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 10 client.29001.objecter close_session for osd.2 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 >> v1:172.21.15.107:6809/3575881728 conn(0x7f6b4c06fc50 legacy=0x7f6b4c0720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 20 client.29001.objecter put_session s=0x7f6b4c06f5a0 osd=2 1 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 20 client.29001.objecter shutdown clearing up homeless session... 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 10 client.29001.objecter successfully canceled tick 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6b4c063c50 legacy=0x7f6b4c066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 >> v1:172.21.15.107:6789/0 conn(0x7f6b641007f0 legacy=0x7f6b640fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b62d9d700 1 -- 172.21.15.107:0/3070790548 reap_dead start 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 shutdown_connections 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 >> 172.21.15.107:0/3070790548 conn(0x7f6b640f3640 msgr2=0x7f6b641077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 shutdown_connections 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 -- 172.21.15.107:0/3070790548 wait complete. 2024-02-10T08:51:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:30.959+0000 7f6b6887e700 1 librados: shutdown 2024-02-10T08:51:31.712 INFO:teuthology.orchestra.run.smithi107.stdout:77309411574 2024-02-10T08:51:31.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T08:51:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:32.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:31 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:32.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:31.608128+0000 mgr.y (mgr.14553) 732 : cluster [DBG] pgmap v1308: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:32.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:31.945796+0000 mgr.y (mgr.14553) 733 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:32.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:31.608128+0000 mgr.y (mgr.14553) 732 : cluster [DBG] pgmap v1308: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:32.496 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:32 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:31.945796+0000 mgr.y (mgr.14553) 733 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:31.608128+0000 mgr.y (mgr.14553) 732 : cluster [DBG] pgmap v1308: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:32.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:32 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:31.945796+0000 mgr.y (mgr.14553) 733 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:33.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:31.771122+0000 osd.5 (osd.5) 390 : cluster [DBG] 16.1f deep-scrub starts 2024-02-10T08:51:33.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:31.775228+0000 osd.5 (osd.5) 391 : cluster [DBG] 16.1f deep-scrub ok 2024-02-10T08:51:33.348 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:31.771122+0000 osd.5 (osd.5) 390 : cluster [DBG] 16.1f deep-scrub starts 2024-02-10T08:51:33.348 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:31.775228+0000 osd.5 (osd.5) 391 : cluster [DBG] 16.1f deep-scrub ok 2024-02-10T08:51:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:31.771122+0000 osd.5 (osd.5) 390 : cluster [DBG] 16.1f deep-scrub starts 2024-02-10T08:51:33.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:33 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:31.775228+0000 osd.5 (osd.5) 391 : cluster [DBG] 16.1f deep-scrub ok 2024-02-10T08:51:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- 172.21.15.107:0/1187510748 >> v1:172.21.15.107:6789/0 conn(0x7f30280fc910 legacy=0x7f30280fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- 172.21.15.107:0/1187510748 shutdown_connections 2024-02-10T08:51:33.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- 172.21.15.107:0/1187510748 >> 172.21.15.107:0/1187510748 conn(0x7f30280f2510 msgr2=0x7f30280f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- 172.21.15.107:0/1187510748 shutdown_connections 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- 172.21.15.107:0/1187510748 wait complete. 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 librados: starting msgr at 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 librados: starting objecter 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 Processor -- start 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 -- start start 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 librados: setting wanted keys 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.887+0000 7f302f0dd700 1 librados: calling monclient init 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302f0dd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3028119480 con 0x7f30280ff710 2024-02-10T08:51:33.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302f0dd700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f302811a660 con 0x7f3028103250 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302f0dd700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f302811b840 con 0x7f30280fc910 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302ce79700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f30280fc910 0x7f3028060b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38126/0 (socket says 172.21.15.107:38126) 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302ce79700 1 -- 172.21.15.107:0/1167552889 learned_addr learned my addr 172.21.15.107:0/1167552889 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter ms_handle_connect 0x7f30280ff710 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter ms_handle_connect 0x7f30280fc910 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195860469 0 0) 0x7f302811b840 con 0x7f30280fc910 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30180038e0 con 0x7f30280fc910 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter ms_handle_connect 0x7f3028103250 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109145094 0 0) 0x7f3028119480 con 0x7f30280ff710 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302811b840 con 0x7f30280ff710 2024-02-10T08:51:33.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559508214 0 0) 0x7f302811a660 con 0x7f3028103250 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028119480 con 0x7f3028103250 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481750391 0 0) 0x7f30180038e0 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f302811a660 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f301c002050 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1766156904 0 0) 0x7f302811b840 con 0x7f30280ff710 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30180038e0 con 0x7f30280ff710 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f30140019f0 con 0x7f30280ff710 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710758438 0 0) 0x7f302811a660 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 >> v1:172.21.15.195:6789/0 conn(0x7f3028103250 legacy=0x7f3028117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 >> v1:172.21.15.107:6789/0 conn(0x7f30280ff710 legacy=0x7f3028061230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302811ca40 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.888+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302811ba30 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f301c001b40 con 0x7f30280fc910 2024-02-10T08:51:33.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f301c0039a0 con 0x7f30280fc910 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 10 client.54605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f302811bea0 con 0x7f30280fc910 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 1 librados: init done 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 10 client.54605.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 15 client.54605.objecter _session_command_op_assign -1 1 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 20 client.54605.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 10 client.54605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.889+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f302805f7a0 con 0x7f30280fc910 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f301c003ba0 con 0x7f30280fc910 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f301c078e80 con 0x7f30280fc910 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 10 client.54605.objecter ms_dispatch 0x7f3028001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 3 client.54605.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 3 client.54605.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:33.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 10 client.54605.objecter checking command 1 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter _get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 15 client.54605.objecter _session_command_op_remove -1 1 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter _get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 15 client.54605.objecter _session_command_op_assign 3 1 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 4 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 10 client.54605.objecter _send_command 1 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6813/2871825691 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3018073540 con 0x7f301806f830 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 20 client.54605.objecter dump_active .. 0 homeless 2024-02-10T08:51:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f301c079220 con 0x7f30280fc910 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 10 client.54605.objecter ms_handle_connect 0x7f3018063a30 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.890+0000 7f3025ffb700 10 client.54605.objecter ms_handle_connect 0x7f301806f830 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.891+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== osd.3 v1:172.21.15.107:6813/2871825691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3018073540 con 0x7f301806f830 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.891+0000 7f3025ffb700 10 client.54605.objecter ms_dispatch 0x7f3028001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.891+0000 7f3025ffb700 10 client.54605.objecter _finish_command 1 = system:0 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.891+0000 7f3025ffb700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 4 2024-02-10T08:51:33.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.891+0000 7f3025ffb700 15 client.54605.objecter _session_command_op_remove 3 1 2024-02-10T08:51:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 10 client.54605.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 15 client.54605.objecter _session_command_op_assign -1 2 2024-02-10T08:51:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 20 client.54605.objecter _get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 20 client.54605.objecter _get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 15 client.54605.objecter _session_command_op_remove -1 2 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 20 client.54605.objecter get_session s=0x7f301806f180 osd=3 3 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 15 client.54605.objecter _session_command_op_assign 3 2 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 4 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 10 client.54605.objecter _send_command 2 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 --> v1:172.21.15.107:6813/2871825691 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3028061b30 con 0x7f301806f830 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f3025ffb700 1 -- 172.21.15.107:0/1167552889 <== osd.3 v1:172.21.15.107:6813/2871825691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 4071372866) 0x7f3028061b30 con 0x7f301806f830 2024-02-10T08:51:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f3025ffb700 10 client.54605.objecter ms_dispatch 0x7f3028001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:33.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f3025ffb700 10 client.54605.objecter _finish_command 2 = system:0 2024-02-10T08:51:33.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f3025ffb700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 4 2024-02-10T08:51:33.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.905+0000 7f3025ffb700 15 client.54605.objecter _session_command_op_remove 3 2 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 10 librados: watch_flush enter 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 10 librados: watch_flush exit 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 10 client.54605.objecter close_session for osd.3 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 >> v1:172.21.15.107:6813/2871825691 conn(0x7f301806f830 legacy=0x7f3018071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 20 client.54605.objecter put_session s=0x7f301806f180 osd=3 1 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 20 client.54605.objecter shutdown clearing up homeless session... 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 10 client.54605.objecter successfully canceled tick 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3018063a30 legacy=0x7f3018065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 >> v1:172.21.15.107:6790/0 conn(0x7f30280fc910 legacy=0x7f3028060b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302d67a700 1 -- 172.21.15.107:0/1167552889 reap_dead start 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 shutdown_connections 2024-02-10T08:51:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 >> 172.21.15.107:0/1167552889 conn(0x7f30280f2510 msgr2=0x7f30281066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 shutdown_connections 2024-02-10T08:51:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 -- 172.21.15.107:0/1167552889 wait complete. 2024-02-10T08:51:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:33.906+0000 7f302f0dd700 1 librados: shutdown 2024-02-10T08:51:34.252 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:33.185019+0000 osd.3 (osd.3) 419 : cluster [DBG] 2.b scrub starts 2024-02-10T08:51:34.252 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:33.185988+0000 osd.3 (osd.3) 420 : cluster [DBG] 2.b scrub ok 2024-02-10T08:51:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:33.185019+0000 osd.3 (osd.3) 419 : cluster [DBG] 2.b scrub starts 2024-02-10T08:51:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:33.185988+0000 osd.3 (osd.3) 420 : cluster [DBG] 2.b scrub ok 2024-02-10T08:51:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:33.608732+0000 mgr.y (mgr.14553) 734 : cluster [DBG] pgmap v1309: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:33.810947+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:51:34.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:34 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:33.814000+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:33.185019+0000 osd.3 (osd.3) 419 : cluster [DBG] 2.b scrub starts 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:33.185988+0000 osd.3 (osd.3) 420 : cluster [DBG] 2.b scrub ok 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:33.608732+0000 mgr.y (mgr.14553) 734 : cluster [DBG] pgmap v1309: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:33.810947+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:33.814000+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:33.608732+0000 mgr.y (mgr.14553) 734 : cluster [DBG] pgmap v1309: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:34.647 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:33.810947+0000 osd.4 (osd.4) 129 : cluster [DBG] 6.17 scrub starts 2024-02-10T08:51:34.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:34 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:33.814000+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.17 scrub ok 2024-02-10T08:51:34.667 INFO:teuthology.orchestra.run.smithi107.stdout:107374182643 2024-02-10T08:51:34.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T08:51:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.022267+0000 osd.6 (osd.6) 366 : cluster [DBG] 5.11 deep-scrub starts 2024-02-10T08:51:35.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.025220+0000 osd.6 (osd.6) 367 : cluster [DBG] 5.11 deep-scrub ok 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.332764+0000 osd.0 (osd.0) 228 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.334499+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.022267+0000 osd.6 (osd.6) 366 : cluster [DBG] 5.11 deep-scrub starts 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.025220+0000 osd.6 (osd.6) 367 : cluster [DBG] 5.11 deep-scrub ok 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.332764+0000 osd.0 (osd.0) 228 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:51:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.334499+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:51:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.022267+0000 osd.6 (osd.6) 366 : cluster [DBG] 5.11 deep-scrub starts 2024-02-10T08:51:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.025220+0000 osd.6 (osd.6) 367 : cluster [DBG] 5.11 deep-scrub ok 2024-02-10T08:51:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.332764+0000 osd.0 (osd.0) 228 : cluster [DBG] 4.1e scrub starts 2024-02-10T08:51:35.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:35 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.334499+0000 osd.0 (osd.0) 229 : cluster [DBG] 4.1e scrub ok 2024-02-10T08:51:35.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:35] "GET /metrics HTTP/1.1" 200 186425 "" "Prometheus/2.33.4" 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.467394+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.d scrub starts 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:34.470493+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub ok 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.175647+0000 osd.3 (osd.3) 421 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.177430+0000 osd.3 (osd.3) 422 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.320130+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.323208+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.609446+0000 mgr.y (mgr.14553) 735 : cluster [DBG] pgmap v1310: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-02-10T08:51:36.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.767415+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:51:36.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:36 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.770980+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:51:36.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.467394+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.d scrub starts 2024-02-10T08:51:36.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:34.470493+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub ok 2024-02-10T08:51:36.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.175647+0000 osd.3 (osd.3) 421 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.177430+0000 osd.3 (osd.3) 422 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.320130+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.323208+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.609446+0000 mgr.y (mgr.14553) 735 : cluster [DBG] pgmap v1310: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.767415+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.770980+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.467394+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.d scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:34.470493+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.175647+0000 osd.3 (osd.3) 421 : cluster [DBG] 4.19 scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.177430+0000 osd.3 (osd.3) 422 : cluster [DBG] 4.19 scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.320130+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.1f scrub starts 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.323208+0000 osd.0 (osd.0) 231 : cluster [DBG] 2.1f scrub ok 2024-02-10T08:51:36.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.609446+0000 mgr.y (mgr.14553) 735 : cluster [DBG] pgmap v1310: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-02-10T08:51:36.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.767415+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.13 scrub starts 2024-02-10T08:51:36.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:36 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.770980+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.13 scrub ok 2024-02-10T08:51:36.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- 172.21.15.107:0/2185783891 >> v1:172.21.15.107:6789/0 conn(0x7f147c0fc910 legacy=0x7f147c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- 172.21.15.107:0/2185783891 shutdown_connections 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- 172.21.15.107:0/2185783891 >> 172.21.15.107:0/2185783891 conn(0x7f147c0f2510 msgr2=0x7f147c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- 172.21.15.107:0/2185783891 shutdown_connections 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- 172.21.15.107:0/2185783891 wait complete. 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 librados: starting msgr at 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 librados: starting objecter 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 Processor -- start 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 -- start start 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 librados: setting wanted keys 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.856+0000 7f1484253700 1 librados: calling monclient init 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1484253700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f147c0fa880 con 0x7f147c0ff710 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1484253700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f147c118790 con 0x7f147c103250 2024-02-10T08:51:36.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1484253700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f147c119970 con 0x7f147c0fc910 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1481fef700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f147c0fc910 0x7f147c0f9c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38140/0 (socket says 172.21.15.107:38140) 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1481fef700 1 -- 172.21.15.107:0/4023715540 learned_addr learned my addr 172.21.15.107:0/4023715540 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter ms_handle_connect 0x7f147c0fc910 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter ms_handle_connect 0x7f147c0ff710 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter ms_handle_connect 0x7f147c103250 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006916718 0 0) 0x7f147c119970 con 0x7f147c0fc910 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1468003030 con 0x7f147c0fc910 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41071865 0 0) 0x7f147c0fa880 con 0x7f147c0ff710 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c119970 con 0x7f147c0ff710 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859693148 0 0) 0x7f147c118790 con 0x7f147c103250 2024-02-10T08:51:36.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c0fa880 con 0x7f147c103250 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459402824 0 0) 0x7f1468003030 con 0x7f147c0fc910 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147c118790 con 0x7f147c0fc910 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1478002130 con 0x7f147c0fc910 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3386259803 0 0) 0x7f147c119970 con 0x7f147c0ff710 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1468003030 con 0x7f147c0ff710 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f146c001800 con 0x7f147c0ff710 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262139147 0 0) 0x7f147c0fa880 con 0x7f147c103250 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147c119970 con 0x7f147c103250 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1474001cb0 con 0x7f147c103250 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2908486051 0 0) 0x7f147c118790 con 0x7f147c0fc910 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 >> v1:172.21.15.195:6789/0 conn(0x7f147c103250 legacy=0x7f147c116060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 >> v1:172.21.15.107:6789/0 conn(0x7f147c0ff710 legacy=0x7f147c112930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.857+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c11ab90 con 0x7f147c0fc910 2024-02-10T08:51:36.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c1177e0 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 10 client.54611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c117c50 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 1 librados: init done 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1478002e20 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1478003a70 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 10 client.54611.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 15 client.54611.objecter _session_command_op_assign -1 1 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 20 client.54611.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 10 client.54611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.858+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14600028b0 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1478003c70 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f1478078d00 con 0x7f147c0fc910 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 10 client.54611.objecter ms_dispatch 0x7f147c001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 3 client.54611.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 3 client.54611.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 10 client.54611.objecter checking command 1 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter _get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 15 client.54611.objecter _session_command_op_remove -1 1 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter _get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 15 client.54611.objecter _session_command_op_assign 4 1 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 4 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 10 client.54611.objecter _send_command 1 2024-02-10T08:51:36.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.195:6800/3310789251 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1468073800 con 0x7f146806faf0 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 20 client.54611.objecter dump_active .. 0 homeless 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f1478079110 con 0x7f147c0fc910 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.859+0000 7f1472ffd700 10 client.54611.objecter ms_handle_connect 0x7f1468063cc0 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 10 client.54611.objecter ms_handle_connect 0x7f146806faf0 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== osd.4 v1:172.21.15.195:6800/3310789251 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1468073800 con 0x7f146806faf0 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 10 client.54611.objecter ms_dispatch 0x7f147c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 10 client.54611.objecter _finish_command 1 = system:0 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 4 2024-02-10T08:51:36.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.860+0000 7f1472ffd700 15 client.54611.objecter _session_command_op_remove 4 1 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 10 client.54611.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 15 client.54611.objecter _session_command_op_assign -1 2 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 20 client.54611.objecter _get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 20 client.54611.objecter _get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 15 client.54611.objecter _session_command_op_remove -1 2 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 20 client.54611.objecter get_session s=0x7f1478002130 osd=4 3 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 15 client.54611.objecter _session_command_op_assign 4 2 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 4 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 10 client.54611.objecter _send_command 2 2024-02-10T08:51:36.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.874+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 --> v1:172.21.15.195:6800/3310789251 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1460001140 con 0x7f146806faf0 2024-02-10T08:51:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1472ffd700 1 -- 172.21.15.107:0/4023715540 <== osd.4 v1:172.21.15.195:6800/3310789251 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+13 (unknown 0 0 1742254341) 0x7f1460001140 con 0x7f146806faf0 2024-02-10T08:51:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1472ffd700 10 client.54611.objecter ms_dispatch 0x7f147c001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1472ffd700 10 client.54611.objecter _finish_command 2 = system:0 2024-02-10T08:51:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1472ffd700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 4 2024-02-10T08:51:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1472ffd700 15 client.54611.objecter _session_command_op_remove 4 2 2024-02-10T08:51:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 10 librados: watch_flush enter 2024-02-10T08:51:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 10 librados: watch_flush exit 2024-02-10T08:51:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 10 client.54611.objecter close_session for osd.4 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 >> v1:172.21.15.195:6800/3310789251 conn(0x7f146806faf0 legacy=0x7f1468071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 20 client.54611.objecter put_session s=0x7f1478002130 osd=4 1 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 20 client.54611.objecter shutdown clearing up homeless session... 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 10 client.54611.objecter successfully canceled tick 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1468063cc0 legacy=0x7f1468066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 >> v1:172.21.15.107:6790/0 conn(0x7f147c0fc910 legacy=0x7f147c0f9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f14827f0700 1 -- 172.21.15.107:0/4023715540 reap_dead start 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 shutdown_connections 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 >> 172.21.15.107:0/4023715540 conn(0x7f147c0f2510 msgr2=0x7f147c0f54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.875+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 shutdown_connections 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.876+0000 7f1484253700 1 -- 172.21.15.107:0/4023715540 wait complete. 2024-02-10T08:51:36.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:36.876+0000 7f1484253700 1 librados: shutdown 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.491483+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.c scrub starts 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.494507+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.c scrub ok 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.700901+0000 osd.5 (osd.5) 392 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:35.704330+0000 osd.5 (osd.5) 393 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:36.155686+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:36.157637+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:36.666493+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:51:37.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:37 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:36.669635+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.491483+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.c scrub starts 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.494507+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.c scrub ok 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.700901+0000 osd.5 (osd.5) 392 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:35.704330+0000 osd.5 (osd.5) 393 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:36.155686+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:36.157637+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:36.666493+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:51:37.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:36.669635+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.491483+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.c scrub starts 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.494507+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.c scrub ok 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.700901+0000 osd.5 (osd.5) 392 : cluster [DBG] 16.1d scrub starts 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:35.704330+0000 osd.5 (osd.5) 393 : cluster [DBG] 16.1d scrub ok 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:36.155686+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.1b scrub starts 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:36.157637+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.1b scrub ok 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:36.666493+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.2 scrub starts 2024-02-10T08:51:37.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:37 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:36.669635+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.2 scrub ok 2024-02-10T08:51:37.683 INFO:teuthology.orchestra.run.smithi107.stdout:2718714298431 2024-02-10T08:51:37.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T08:51:37.952 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:37.041847+0000 osd.6 (osd.6) 368 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:51:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:37.042824+0000 osd.6 (osd.6) 369 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:51:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:37.610116+0000 mgr.y (mgr.14553) 736 : cluster [DBG] pgmap v1311: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:37.768002+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.e deep-scrub starts 2024-02-10T08:51:38.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:38 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:37.770945+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.e deep-scrub ok 2024-02-10T08:51:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:37.041847+0000 osd.6 (osd.6) 368 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:51:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:37.042824+0000 osd.6 (osd.6) 369 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:51:38.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:37.610116+0000 mgr.y (mgr.14553) 736 : cluster [DBG] pgmap v1311: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:37.768002+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.e deep-scrub starts 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:37.770945+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.e deep-scrub ok 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:37.041847+0000 osd.6 (osd.6) 368 : cluster [DBG] 2.14 scrub starts 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:37.042824+0000 osd.6 (osd.6) 369 : cluster [DBG] 2.14 scrub ok 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:37.610116+0000 mgr.y (mgr.14553) 736 : cluster [DBG] pgmap v1311: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:37.768002+0000 osd.4 (osd.4) 133 : cluster [DBG] 6.e deep-scrub starts 2024-02-10T08:51:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:38 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:37.770945+0000 osd.4 (osd.4) 134 : cluster [DBG] 6.e deep-scrub ok 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.285899+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.286909+0000 osd.0 (osd.0) 233 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.810478+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.814087+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.866924+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:51:39.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:38.867810+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.285899+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.286909+0000 osd.0 (osd.0) 233 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.810478+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.814087+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.866924+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:38.867810+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:51:39.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.285899+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 deep-scrub starts 2024-02-10T08:51:39.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.286909+0000 osd.0 (osd.0) 233 : cluster [DBG] 5.5 deep-scrub ok 2024-02-10T08:51:39.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.810478+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.9 scrub starts 2024-02-10T08:51:39.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.814087+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.9 scrub ok 2024-02-10T08:51:39.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.866924+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.8 scrub starts 2024-02-10T08:51:39.646 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:39 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:38.867810+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.8 scrub ok 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.914+0000 7f9302dfa700 1 -- 172.21.15.107:0/2350990952 >> v1:172.21.15.107:6790/0 conn(0x7f92fc0fc910 legacy=0x7f92fc0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.914+0000 7f9302dfa700 1 -- 172.21.15.107:0/2350990952 shutdown_connections 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.914+0000 7f9302dfa700 1 -- 172.21.15.107:0/2350990952 >> 172.21.15.107:0/2350990952 conn(0x7f92fc0f2510 msgr2=0x7f92fc0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.914+0000 7f9302dfa700 1 -- 172.21.15.107:0/2350990952 shutdown_connections 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.914+0000 7f9302dfa700 1 -- 172.21.15.107:0/2350990952 wait complete. 2024-02-10T08:51:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 librados: starting msgr at 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 librados: starting objecter 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 Processor -- start 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 -- start start 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 librados: setting wanted keys 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 librados: calling monclient init 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92fc1173b0 con 0x7f92fc0fc910 2024-02-10T08:51:39.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92fc118590 con 0x7f92fc0ff710 2024-02-10T08:51:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9302dfa700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92fc119790 con 0x7f92fc103250 2024-02-10T08:51:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9301397700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f92fc103250 0x7f92fc115a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34974/0 (socket says 172.21.15.107:34974) 2024-02-10T08:51:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9300b96700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f92fc0fc910 0x7f92fc0f9b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33564/0 (socket says 172.21.15.107:33564) 2024-02-10T08:51:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f9301397700 1 -- 172.21.15.107:0/1018617308 learned_addr learned my addr 172.21.15.107:0/1018617308 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter ms_handle_connect 0x7f92fc103250 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter ms_handle_connect 0x7f92fc0fc910 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter ms_handle_connect 0x7f92fc0ff710 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:39.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871275872 0 0) 0x7f92fc1173b0 con 0x7f92fc0fc910 2024-02-10T08:51:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ec003030 con 0x7f92fc0fc910 2024-02-10T08:51:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414484223 0 0) 0x7f92fc119790 con 0x7f92fc103250 2024-02-10T08:51:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92fc1173b0 con 0x7f92fc103250 2024-02-10T08:51:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35753050 0 0) 0x7f92fc118590 con 0x7f92fc0ff710 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.915+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92fc119790 con 0x7f92fc0ff710 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160694791 0 0) 0x7f92fc1173b0 con 0x7f92fc103250 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92fc118590 con 0x7f92fc103250 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791067589 0 0) 0x7f92ec003030 con 0x7f92fc0fc910 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92fc1173b0 con 0x7f92fc0fc910 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92f4001bc0 con 0x7f92fc103250 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92f00020a0 con 0x7f92fc0fc910 2024-02-10T08:51:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35568305 0 0) 0x7f92fc119790 con 0x7f92fc0ff710 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92ec003030 con 0x7f92fc0ff710 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92e8001620 con 0x7f92fc0ff710 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2322951158 0 0) 0x7f92fc118590 con 0x7f92fc103250 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 >> v1:172.21.15.195:6789/0 conn(0x7f92fc0ff710 legacy=0x7f92fc0fa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 >> v1:172.21.15.107:6789/0 conn(0x7f92fc0fc910 legacy=0x7f92fc0f9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc11a990 con 0x7f92fc103250 2024-02-10T08:51:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc1175a0 con 0x7f92fc103250 2024-02-10T08:51:39.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f92f4001df0 con 0x7f92fc103250 2024-02-10T08:51:39.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f92f4005e00 con 0x7f92fc103250 2024-02-10T08:51:39.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f9302dfa700 10 client.54617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc117a10 con 0x7f92fc103250 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.916+0000 7f9302dfa700 1 librados: init done 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f9302dfa700 10 client.54617.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f9302dfa700 15 client.54617.objecter _session_command_op_assign -1 1 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f9302dfa700 20 client.54617.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f9302dfa700 10 client.54617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92fc05f7a0 con 0x7f92fc103250 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.917+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f92f4002300 con 0x7f92fc103250 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f92f4078e70 con 0x7f92fc103250 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 10 client.54617.objecter ms_dispatch 0x7f92fc001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 3 client.54617.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 3 client.54617.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 10 client.54617.objecter checking command 1 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter _get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 15 client.54617.objecter _session_command_op_remove -1 1 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter _get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 15 client.54617.objecter _session_command_op_assign 5 1 2024-02-10T08:51:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 4 2024-02-10T08:51:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 10 client.54617.objecter _send_command 1 2024-02-10T08:51:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.195:6804/4178321486 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92ec0739a0 con 0x7f92ec06fc90 2024-02-10T08:51:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 20 client.54617.objecter dump_active .. 0 homeless 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f92f40792a0 con 0x7f92fc103250 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.918+0000 7f92f9ffb700 10 client.54617.objecter ms_handle_connect 0x7f92ec063c70 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 10 client.54617.objecter ms_handle_connect 0x7f92ec06fc90 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== osd.5 v1:172.21.15.195:6804/4178321486 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f92ec0739a0 con 0x7f92ec06fc90 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 10 client.54617.objecter ms_dispatch 0x7f92fc001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:39.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 10 client.54617.objecter _finish_command 1 = system:0 2024-02-10T08:51:39.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 4 2024-02-10T08:51:39.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.919+0000 7f92f9ffb700 15 client.54617.objecter _session_command_op_remove 5 1 2024-02-10T08:51:39.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 10 client.54617.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:39.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 15 client.54617.objecter _session_command_op_assign -1 2 2024-02-10T08:51:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 20 client.54617.objecter _get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 20 client.54617.objecter _get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 15 client.54617.objecter _session_command_op_remove -1 2 2024-02-10T08:51:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 20 client.54617.objecter get_session s=0x7f92ec06f5e0 osd=5 3 2024-02-10T08:51:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 15 client.54617.objecter _session_command_op_assign 5 2 2024-02-10T08:51:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 4 2024-02-10T08:51:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 10 client.54617.objecter _send_command 2 2024-02-10T08:51:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 --> v1:172.21.15.195:6804/4178321486 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f92fc0475e0 con 0x7f92ec06fc90 2024-02-10T08:51:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f92f9ffb700 1 -- 172.21.15.107:0/1018617308 <== osd.5 v1:172.21.15.195:6804/4178321486 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2308010022) 0x7f92fc0475e0 con 0x7f92ec06fc90 2024-02-10T08:51:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f92f9ffb700 10 client.54617.objecter ms_dispatch 0x7f92fc001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f92f9ffb700 10 client.54617.objecter _finish_command 2 = system:0 2024-02-10T08:51:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f92f9ffb700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 4 2024-02-10T08:51:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.934+0000 7f92f9ffb700 15 client.54617.objecter _session_command_op_remove 5 2 2024-02-10T08:51:39.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 10 librados: watch_flush enter 2024-02-10T08:51:39.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 10 librados: watch_flush exit 2024-02-10T08:51:39.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 10 client.54617.objecter close_session for osd.5 2024-02-10T08:51:39.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 >> v1:172.21.15.195:6804/4178321486 conn(0x7f92ec06fc90 legacy=0x7f92ec072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 20 client.54617.objecter put_session s=0x7f92ec06f5e0 osd=5 1 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 20 client.54617.objecter shutdown clearing up homeless session... 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 10 client.54617.objecter successfully canceled tick 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 >> v1:172.21.15.107:6800/2634842009 conn(0x7f92ec063c70 legacy=0x7f92ec066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 >> v1:172.21.15.107:6790/0 conn(0x7f92fc103250 legacy=0x7f92fc115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9301397700 1 -- 172.21.15.107:0/1018617308 reap_dead start 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 shutdown_connections 2024-02-10T08:51:39.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 >> 172.21.15.107:0/1018617308 conn(0x7f92fc0f2510 msgr2=0x7f92fc0612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:39.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 shutdown_connections 2024-02-10T08:51:39.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 -- 172.21.15.107:0/1018617308 wait complete. 2024-02-10T08:51:39.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:39.935+0000 7f9302dfa700 1 librados: shutdown 2024-02-10T08:51:40.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:39.011227+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:51:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:39.012189+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:51:40.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:40 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:39.610506+0000 mgr.y (mgr.14553) 737 : cluster [DBG] pgmap v1312: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:40.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:39.011227+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:51:40.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:39.012189+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:51:40.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:39.610506+0000 mgr.y (mgr.14553) 737 : cluster [DBG] pgmap v1312: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:39.011227+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.7 scrub starts 2024-02-10T08:51:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:39.012189+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.7 scrub ok 2024-02-10T08:51:40.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:39.610506+0000 mgr.y (mgr.14553) 737 : cluster [DBG] pgmap v1312: 154 pgs: 1 active+clean+scrubbing, 153 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:40.756 INFO:teuthology.orchestra.run.smithi107.stdout:158913790186 2024-02-10T08:51:40.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T08:51:41.024 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:41.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:39.708455+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:39.711347+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.060914+0000 osd.6 (osd.6) 372 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.063808+0000 osd.6 (osd.6) 373 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.232491+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.233441+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:39.708455+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:39.711347+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.060914+0000 osd.6 (osd.6) 372 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.063808+0000 osd.6 (osd.6) 373 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:51:41.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.232491+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:51:41.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.233441+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:39.708455+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.16 scrub starts 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:39.711347+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.16 scrub ok 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.060914+0000 osd.6 (osd.6) 372 : cluster [DBG] 5.2 scrub starts 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.063808+0000 osd.6 (osd.6) 373 : cluster [DBG] 5.2 scrub ok 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.232491+0000 osd.0 (osd.0) 234 : cluster [DBG] 6.16 deep-scrub starts 2024-02-10T08:51:41.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.233441+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.16 deep-scrub ok 2024-02-10T08:51:42.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:41 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:42.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.934854+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:51:42.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:40.935706+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:41.179056+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:41.179817+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:41.282024+0000 osd.0 (osd.0) 236 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:41.282891+0000 osd.0 (osd.0) 237 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:41.611265+0000 mgr.y (mgr.14553) 738 : cluster [DBG] pgmap v1313: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:41.956749+0000 mgr.y (mgr.14553) 739 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.934854+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:40.935706+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:41.179056+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:51:42.636 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:41.179817+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:51:42.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:41.282024+0000 osd.0 (osd.0) 236 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:51:42.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:41.282891+0000 osd.0 (osd.0) 237 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:51:42.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:41.611265+0000 mgr.y (mgr.14553) 738 : cluster [DBG] pgmap v1313: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:51:42.637 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:42 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:41.956749+0000 mgr.y (mgr.14553) 739 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.934854+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.10 scrub starts 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:40.935706+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.10 scrub ok 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:41.179056+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.13 scrub starts 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:41.179817+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.13 scrub ok 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:41.282024+0000 osd.0 (osd.0) 236 : cluster [DBG] 16.15 scrub starts 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:41.282891+0000 osd.0 (osd.0) 237 : cluster [DBG] 16.15 scrub ok 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:41.611265+0000 mgr.y (mgr.14553) 738 : cluster [DBG] pgmap v1313: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T08:51:42.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:42 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:41.956749+0000 mgr.y (mgr.14553) 739 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.898+0000 7f626283b700 1 -- 172.21.15.107:0/2636724176 >> v1:172.21.15.107:6790/0 conn(0x7f625c103250 legacy=0x7f625c1056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.898+0000 7f626283b700 1 -- 172.21.15.107:0/2636724176 shutdown_connections 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.898+0000 7f626283b700 1 -- 172.21.15.107:0/2636724176 >> 172.21.15.107:0/2636724176 conn(0x7f625c0f2510 msgr2=0x7f625c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.898+0000 7f626283b700 1 -- 172.21.15.107:0/2636724176 shutdown_connections 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.898+0000 7f626283b700 1 -- 172.21.15.107:0/2636724176 wait complete. 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 librados: starting msgr at 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 librados: starting objecter 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 Processor -- start 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 -- start start 2024-02-10T08:51:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 librados: setting wanted keys 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 librados: calling monclient init 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f625c117200 con 0x7f625c0fc910 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f625c1183e0 con 0x7f625c0ff710 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f626283b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f625c1195c0 con 0x7f625c103250 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f6260dd8700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f625c103250 0x7f625c1158e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35004/0 (socket says 172.21.15.107:35004) 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f6260dd8700 1 -- 172.21.15.107:0/3423228225 learned_addr learned my addr 172.21.15.107:0/3423228225 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.899+0000 7f625bfff700 1 --1- 172.21.15.107:0/3423228225 >> v1:172.21.15.107:6789/0 conn(0x7f625c0fc910 0x7f625c060c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33586/0 (socket says 172.21.15.107:33586) 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter ms_handle_connect 0x7f625c103250 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter ms_handle_connect 0x7f625c0fc910 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter ms_handle_connect 0x7f625c0ff710 2024-02-10T08:51:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075369389 0 0) 0x7f625c1195c0 con 0x7f625c103250 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6244003030 con 0x7f625c103250 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619118700 0 0) 0x7f625c117200 con 0x7f625c0fc910 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c1195c0 con 0x7f625c0fc910 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550517704 0 0) 0x7f625c1183e0 con 0x7f625c0ff710 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c117200 con 0x7f625c0ff710 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877697953 0 0) 0x7f625c1195c0 con 0x7f625c0fc910 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f625c1183e0 con 0x7f625c0fc910 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f624c001a30 con 0x7f625c0fc910 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409887987 0 0) 0x7f6244003030 con 0x7f625c103250 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f625c1195c0 con 0x7f625c103250 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f62540017b0 con 0x7f625c103250 2024-02-10T08:51:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570320620 0 0) 0x7f625c117200 con 0x7f625c0ff710 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6244003030 con 0x7f625c0ff710 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f6250001840 con 0x7f625c0ff710 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480315249 0 0) 0x7f625c1183e0 con 0x7f625c0fc910 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 >> v1:172.21.15.107:6790/0 conn(0x7f625c103250 legacy=0x7f625c1158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 >> v1:172.21.15.195:6789/0 conn(0x7f625c0ff710 legacy=0x7f625c061370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c11a7c0 con 0x7f625c0fc910 2024-02-10T08:51:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c1173f0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f626283b700 10 client.29025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c1178e0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f626283b700 1 librados: init done 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f624c0023a0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.900+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f624c005950 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.901+0000 7f626283b700 10 client.29025.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.901+0000 7f626283b700 15 client.29025.objecter _session_command_op_assign -1 1 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.901+0000 7f626283b700 20 client.29025.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.901+0000 7f626283b700 10 client.29025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.901+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.107:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62480028b0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f624c0188c0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(833..833 src has 775..833) v4 ==== 7473+0+0 (unknown 3904291536 0 0) 0x7f624c0788f0 con 0x7f625c0fc910 2024-02-10T08:51:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 10 client.29025.objecter ms_dispatch 0x7f625c001550 osd_map(833..833 src has 775..833) v4 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 3 client.29025.objecter handle_osd_map got epochs [833,833] > 0 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 3 client.29025.objecter handle_osd_map decoding full epoch 833 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 10 client.29025.objecter checking command 1 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter _get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 15 client.29025.objecter _session_command_op_remove -1 1 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter _get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 15 client.29025.objecter _session_command_op_assign 6 1 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 4 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 10 client.29025.objecter _send_command 1 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.195:6808/1417479017 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6244073a20 con 0x7f624406fd10 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 20 client.29025.objecter dump_active .. 0 homeless 2024-02-10T08:51:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 4195621587 0 0) 0x7f624c078d20 con 0x7f625c0fc910 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.902+0000 7f62597fa700 10 client.29025.objecter ms_handle_connect 0x7f6244063d80 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.903+0000 7f62597fa700 10 client.29025.objecter ms_handle_connect 0x7f624406fd10 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.904+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== osd.6 v1:172.21.15.195:6808/1417479017 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6244073a20 con 0x7f624406fd10 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.904+0000 7f62597fa700 10 client.29025.objecter ms_dispatch 0x7f625c001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.904+0000 7f62597fa700 10 client.29025.objecter _finish_command 1 = system:0 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.904+0000 7f62597fa700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 4 2024-02-10T08:51:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.904+0000 7f62597fa700 15 client.29025.objecter _session_command_op_remove 6 1 2024-02-10T08:51:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 client.29025.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 15 client.29025.objecter _session_command_op_assign -1 2 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter _get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter _get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 15 client.29025.objecter _session_command_op_remove -1 2 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter get_session s=0x7f624406f660 osd=6 3 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 15 client.29025.objecter _session_command_op_assign 6 2 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 4 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 client.29025.objecter _send_command 2 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 --> v1:172.21.15.195:6808/1417479017 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6248001140 con 0x7f624406fd10 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f62597fa700 1 -- 172.21.15.107:0/3423228225 <== osd.6 v1:172.21.15.195:6808/1417479017 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 25011676) 0x7f6248001140 con 0x7f624406fd10 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f62597fa700 10 client.29025.objecter ms_dispatch 0x7f625c001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f62597fa700 10 client.29025.objecter _finish_command 2 = system:0 2024-02-10T08:51:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f62597fa700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 4 2024-02-10T08:51:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f62597fa700 15 client.29025.objecter _session_command_op_remove 6 2 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 librados: watch_flush enter 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 librados: watch_flush exit 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 client.29025.objecter close_session for osd.6 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 >> v1:172.21.15.195:6808/1417479017 conn(0x7f624406fd10 legacy=0x7f62440721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter put_session s=0x7f624406f660 osd=6 1 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 20 client.29025.objecter shutdown clearing up homeless session... 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.918+0000 7f626283b700 10 client.29025.objecter successfully canceled tick 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 >> v1:172.21.15.107:6800/2634842009 conn(0x7f6244063d80 legacy=0x7f6244066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 >> v1:172.21.15.107:6789/0 conn(0x7f625c0fc910 legacy=0x7f625c060c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f6260dd8700 1 -- 172.21.15.107:0/3423228225 reap_dead start 2024-02-10T08:51:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 shutdown_connections 2024-02-10T08:51:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 >> 172.21.15.107:0/3423228225 conn(0x7f625c0f2510 msgr2=0x7f625c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 shutdown_connections 2024-02-10T08:51:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 -- 172.21.15.107:0/3423228225 wait complete. 2024-02-10T08:51:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:42.919+0000 7f626283b700 1 librados: shutdown 2024-02-10T08:51:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:42.643222+0000 osd.5 (osd.5) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:43.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:42.643585+0000 osd.5 (osd.5) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:42.643222+0000 osd.5 (osd.5) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:42.643585+0000 osd.5 (osd.5) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:42.643222+0000 osd.5 (osd.5) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:43.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:43 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:42.643585+0000 osd.5 (osd.5) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:43.883 INFO:teuthology.orchestra.run.smithi107.stdout:188978561254 2024-02-10T08:51:43.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T08:51:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:44.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.324544+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-10T08:51:44.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.611727+0000 mgr.y (mgr.14553) 740 : cluster [DBG] pgmap v1315: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:44.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.249828+0000 osd.3 (osd.3) 427 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:51:44.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.251744+0000 osd.3 (osd.3) 428 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:51:44.672 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.324544+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-10T08:51:44.672 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.611727+0000 mgr.y (mgr.14553) 740 : cluster [DBG] pgmap v1315: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:44.672 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.249828+0000 osd.3 (osd.3) 427 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:51:44.672 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:44 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.251744+0000 osd.3 (osd.3) 428 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:51:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.324544+0000 mon.a (mon.0) 3781 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-10T08:51:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.611727+0000 mgr.y (mgr.14553) 740 : cluster [DBG] pgmap v1315: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.249828+0000 osd.3 (osd.3) 427 : cluster [DBG] 4.11 scrub starts 2024-02-10T08:51:44.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:44 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.251744+0000 osd.3 (osd.3) 428 : cluster [DBG] 4.11 scrub ok 2024-02-10T08:51:45.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.526910+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:51:45.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.530539+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.834594+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:43.835897+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.917688+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.920618+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.526910+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.530539+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.834594+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:43.835897+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.917688+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:51:45.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.920618+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.526910+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub starts 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.530539+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.9 scrub ok 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.834594+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.5 scrub starts 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:43.835897+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.5 scrub ok 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.917688+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-02-10T08:51:45.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.920618+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-02-10T08:51:45.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:45 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:45] "GET /metrics HTTP/1.1" 200 186422 "" "Prometheus/2.33.4" 2024-02-10T08:51:46.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- 172.21.15.107:0/563045084 >> v1:172.21.15.107:6789/0 conn(0x7f86d4104330 legacy=0x7f86d41067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- 172.21.15.107:0/563045084 shutdown_connections 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- 172.21.15.107:0/563045084 >> 172.21.15.107:0/563045084 conn(0x7f86d40f3640 msgr2=0x7f86d40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- 172.21.15.107:0/563045084 shutdown_connections 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- 172.21.15.107:0/563045084 wait complete. 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 librados: starting msgr at 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 librados: starting objecter 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 Processor -- start 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 -- start start 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 librados: setting wanted keys 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.005+0000 7f86dbbdc700 1 librados: calling monclient init 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86dbbdc700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d4117520 con 0x7f86d40fda50 2024-02-10T08:51:46.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86dbbdc700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d4118700 con 0x7f86d41007f0 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86dbbdc700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d4119900 con 0x7f86d4104330 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86da179700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f86d4104330 0x7f86d4115be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35026/0 (socket says 172.21.15.107:35026) 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86d9978700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86d40fda50 0x7f86d40f9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33608/0 (socket says 172.21.15.107:33608) 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86da179700 1 -- 172.21.15.107:0/1705410985 learned_addr learned my addr 172.21.15.107:0/1705410985 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter ms_handle_connect 0x7f86d4104330 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter ms_handle_connect 0x7f86d40fda50 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter ms_handle_connect 0x7f86d41007f0 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173932860 0 0) 0x7f86d4119900 con 0x7f86d4104330 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003630 con 0x7f86d4104330 2024-02-10T08:51:46.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024925456 0 0) 0x7f86d4117520 con 0x7f86d40fda50 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4119900 con 0x7f86d40fda50 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483111224 0 0) 0x7f86d4118700 con 0x7f86d41007f0 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4117520 con 0x7f86d41007f0 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029969160 0 0) 0x7f86c0003630 con 0x7f86d4104330 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d4118700 con 0x7f86d4104330 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409091684 0 0) 0x7f86d4119900 con 0x7f86d40fda50 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c0003630 con 0x7f86d40fda50 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86cc0030e0 con 0x7f86d4104330 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86c4001840 con 0x7f86d40fda50 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343510047 0 0) 0x7f86d4117520 con 0x7f86d41007f0 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d4119900 con 0x7f86d41007f0 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86d0001a10 con 0x7f86d41007f0 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2581633073 0 0) 0x7f86d4118700 con 0x7f86d4104330 2024-02-10T08:51:46.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 >> v1:172.21.15.195:6789/0 conn(0x7f86d41007f0 legacy=0x7f86d40fa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 >> v1:172.21.15.107:6789/0 conn(0x7f86d40fda50 legacy=0x7f86d40f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d411ab00 con 0x7f86d4104330 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.006+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d4117710 con 0x7f86d4104330 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 10 client.54629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d4117c00 con 0x7f86d4104330 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 1 librados: init done 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f86cc0032c0 con 0x7f86d4104330 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f86cc003bf0 con 0x7f86d4104330 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 10 client.54629.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 15 client.54629.objecter _session_command_op_assign -1 1 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 20 client.54629.objecter _get_session osd=-1 returning homeless 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 10 client.54629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:46.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.007+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.107:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86b80028b0 con 0x7f86d4104330 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f86cc003df0 con 0x7f86d4104330 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(834..834 src has 775..834) v4 ==== 7473+0+0 (unknown 1291156150 0 0) 0x7f86cc079170 con 0x7f86d4104330 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 10 client.54629.objecter ms_dispatch 0x7f86d4001550 osd_map(834..834 src has 775..834) v4 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 3 client.54629.objecter handle_osd_map got epochs [834,834] > 0 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 3 client.54629.objecter handle_osd_map decoding full epoch 834 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 10 client.54629.objecter checking command 1 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter _get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 15 client.54629.objecter _session_command_op_remove -1 1 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter _get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 15 client.54629.objecter _session_command_op_assign 7 1 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 4 2024-02-10T08:51:46.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 10 client.54629.objecter _send_command 1 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.195:6812/1934176928 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c00738c0 con 0x7f86c006fbb0 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 20 client.54629.objecter dump_active .. 0 homeless 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_get_version_reply(handle=1 version=834) v2 ==== 24+0+0 (unknown 3924589088 0 0) 0x7f86cc0794f0 con 0x7f86d4104330 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.008+0000 7f86caffd700 10 client.54629.objecter ms_handle_connect 0x7f86c0063d20 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.009+0000 7f86caffd700 10 client.54629.objecter ms_handle_connect 0x7f86c006fbb0 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.009+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== osd.7 v1:172.21.15.195:6812/1934176928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c00738c0 con 0x7f86c006fbb0 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.010+0000 7f86caffd700 10 client.54629.objecter ms_dispatch 0x7f86d4001550 command_reply(tid 1: 0 ) v1 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.010+0000 7f86caffd700 10 client.54629.objecter _finish_command 1 = system:0 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.010+0000 7f86caffd700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 4 2024-02-10T08:51:46.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.010+0000 7f86caffd700 15 client.54629.objecter _session_command_op_remove 7 1 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 client.54629.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 15 client.54629.objecter _session_command_op_assign -1 2 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter _get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter _get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 15 client.54629.objecter _session_command_op_remove -1 2 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter get_session s=0x7f86c006f500 osd=7 3 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 15 client.54629.objecter _session_command_op_assign 7 2 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 4 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 client.54629.objecter _send_command 2 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 --> v1:172.21.15.195:6812/1934176928 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f86b8001140 con 0x7f86c006fbb0 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86caffd700 1 -- 172.21.15.107:0/1705410985 <== osd.7 v1:172.21.15.195:6812/1934176928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2476650775) 0x7f86b8001140 con 0x7f86c006fbb0 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86caffd700 10 client.54629.objecter ms_dispatch 0x7f86d4001550 command_reply(tid 2: 0 ) v1 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86caffd700 10 client.54629.objecter _finish_command 2 = system:0 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86caffd700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 4 2024-02-10T08:51:46.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86caffd700 15 client.54629.objecter _session_command_op_remove 7 2 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 librados: watch_flush enter 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 librados: watch_flush exit 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 client.54629.objecter close_session for osd.7 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 >> v1:172.21.15.195:6812/1934176928 conn(0x7f86c006fbb0 legacy=0x7f86c0072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter put_session s=0x7f86c006f500 osd=7 1 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 20 client.54629.objecter shutdown clearing up homeless session... 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.024+0000 7f86dbbdc700 10 client.54629.objecter successfully canceled tick 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 >> v1:172.21.15.107:6800/2634842009 conn(0x7f86c0063d20 legacy=0x7f86c00661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 >> v1:172.21.15.107:6790/0 conn(0x7f86d4104330 legacy=0x7f86d4115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86da179700 1 -- 172.21.15.107:0/1705410985 reap_dead start 2024-02-10T08:51:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 shutdown_connections 2024-02-10T08:51:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 >> 172.21.15.107:0/1705410985 conn(0x7f86d40f3640 msgr2=0x7f86d41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 shutdown_connections 2024-02-10T08:51:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 -- 172.21.15.107:0/1705410985 wait complete. 2024-02-10T08:51:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:46.025+0000 7f86dbbdc700 1 librados: shutdown 2024-02-10T08:51:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.702605+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.f scrub starts 2024-02-10T08:51:46.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:44.706119+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.f scrub ok 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:45.612099+0000 mgr.y (mgr.14553) 741 : cluster [DBG] pgmap v1316: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:45.869720+0000 osd.4 (osd.4) 139 : cluster [DBG] 4.c scrub starts 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:45.873191+0000 osd.4 (osd.4) 140 : cluster [DBG] 4.c scrub ok 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.702605+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.f scrub starts 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:44.706119+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.f scrub ok 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:45.612099+0000 mgr.y (mgr.14553) 741 : cluster [DBG] pgmap v1316: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:45.869720+0000 osd.4 (osd.4) 139 : cluster [DBG] 4.c scrub starts 2024-02-10T08:51:46.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:46 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:45.873191+0000 osd.4 (osd.4) 140 : cluster [DBG] 4.c scrub ok 2024-02-10T08:51:46.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.702605+0000 osd.5 (osd.5) 400 : cluster [DBG] 5.f scrub starts 2024-02-10T08:51:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:44.706119+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.f scrub ok 2024-02-10T08:51:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:45.612099+0000 mgr.y (mgr.14553) 741 : cluster [DBG] pgmap v1316: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:45.869720+0000 osd.4 (osd.4) 139 : cluster [DBG] 4.c scrub starts 2024-02-10T08:51:46.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:46 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:45.873191+0000 osd.4 (osd.4) 140 : cluster [DBG] 4.c scrub ok 2024-02-10T08:51:46.829 INFO:teuthology.orchestra.run.smithi107.stdout:223338299616 2024-02-10T08:51:46.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T08:51:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:45.967296+0000 osd.2 (osd.2) 363 : cluster [DBG] 16.17 deep-scrub starts 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:45.971084+0000 osd.2 (osd.2) 364 : cluster [DBG] 16.17 deep-scrub ok 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.280887+0000 osd.3 (osd.3) 429 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.281833+0000 osd.3 (osd.3) 430 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:45.967296+0000 osd.2 (osd.2) 363 : cluster [DBG] 16.17 deep-scrub starts 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:45.971084+0000 osd.2 (osd.2) 364 : cluster [DBG] 16.17 deep-scrub ok 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.280887+0000 osd.3 (osd.3) 429 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:51:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:47 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.281833+0000 osd.3 (osd.3) 430 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:51:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:45.967296+0000 osd.2 (osd.2) 363 : cluster [DBG] 16.17 deep-scrub starts 2024-02-10T08:51:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:45.971084+0000 osd.2 (osd.2) 364 : cluster [DBG] 16.17 deep-scrub ok 2024-02-10T08:51:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.280887+0000 osd.3 (osd.3) 429 : cluster [DBG] 2.1b scrub starts 2024-02-10T08:51:47.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:47 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.281833+0000 osd.3 (osd.3) 430 : cluster [DBG] 2.1b scrub ok 2024-02-10T08:51:48.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:46.488667+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.a deep-scrub starts 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:46.491777+0000 osd.1 (osd.1) 415 : cluster [DBG] 2.a deep-scrub ok 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:46.695623+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:46.698898+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.007045+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.a scrub starts 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.010149+0000 osd.2 (osd.2) 366 : cluster [DBG] 5.a scrub ok 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.362093+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.363082+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.612854+0000 mgr.y (mgr.14553) 742 : cluster [DBG] pgmap v1317: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.939489+0000 osd.4 (osd.4) 141 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.942473+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:51:48.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.028227+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.028424+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.277949+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.c scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.279082+0000 osd.3 (osd.3) 434 : cluster [DBG] 6.c scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:46.488667+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.a deep-scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:46.491777+0000 osd.1 (osd.1) 415 : cluster [DBG] 2.a deep-scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:46.695623+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:46.698898+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.007045+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.a scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.010149+0000 osd.2 (osd.2) 366 : cluster [DBG] 5.a scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.362093+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.363082+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.612854+0000 mgr.y (mgr.14553) 742 : cluster [DBG] pgmap v1317: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:48.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.939489+0000 osd.4 (osd.4) 141 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:51:48.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.942473+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:51:48.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.028227+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:48.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.028424+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:48.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.277949+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.c scrub starts 2024-02-10T08:51:48.434 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:48 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.279082+0000 osd.3 (osd.3) 434 : cluster [DBG] 6.c scrub ok 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:46.488667+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.a deep-scrub starts 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:46.491777+0000 osd.1 (osd.1) 415 : cluster [DBG] 2.a deep-scrub ok 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:46.695623+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.7 scrub starts 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:46.698898+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.7 scrub ok 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.007045+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.a scrub starts 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.010149+0000 osd.2 (osd.2) 366 : cluster [DBG] 5.a scrub ok 2024-02-10T08:51:48.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.362093+0000 osd.0 (osd.0) 238 : cluster [DBG] 6.0 scrub starts 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.363082+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 scrub ok 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.612854+0000 mgr.y (mgr.14553) 742 : cluster [DBG] pgmap v1317: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.939489+0000 osd.4 (osd.4) 141 : cluster [DBG] 5.1 scrub starts 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.942473+0000 osd.4 (osd.4) 142 : cluster [DBG] 5.1 scrub ok 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.028227+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.028424+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.277949+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.c scrub starts 2024-02-10T08:51:48.788 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:48 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.279082+0000 osd.3 (osd.3) 434 : cluster [DBG] 6.c scrub ok 2024-02-10T08:51:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 -- 172.21.15.107:0/1805297585 >> v1:172.21.15.107:6789/0 conn(0x7f1d6c0fd9f0 legacy=0x7f1d6c0fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 -- 172.21.15.107:0/1805297585 shutdown_connections 2024-02-10T08:51:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 -- 172.21.15.107:0/1805297585 >> 172.21.15.107:0/1805297585 conn(0x7f1d6c06c090 msgr2=0x7f1d6c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 -- 172.21.15.107:0/1805297585 shutdown_connections 2024-02-10T08:51:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 -- 172.21.15.107:0/1805297585 wait complete. 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 librados: starting msgr at 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.017+0000 7f1d7148b700 1 librados: starting objecter 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 Processor -- start 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 -- start start 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 librados: setting wanted keys 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 librados: calling monclient init 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c1175b0 con 0x7f1d6c104350 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c1187b0 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d7148b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d6c1199b0 con 0x7f1d6c100840 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d6a7fc700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1d6c100840 0x7f1d6c0fa590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35034/0 (socket says 172.21.15.107:35034) 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d6a7fc700 1 -- 172.21.15.107:0/3851223897 learned_addr learned my addr 172.21.15.107:0/3851223897 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter ms_handle_connect 0x7f1d6c100840 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter ms_handle_connect 0x7f1d6c104350 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter ms_handle_connect 0x7f1d6c0fd9f0 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326480748 0 0) 0x7f1d6c1199b0 con 0x7f1d6c100840 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d58003030 con 0x7f1d6c100840 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672050642 0 0) 0x7f1d6c1175b0 con 0x7f1d6c104350 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c1199b0 con 0x7f1d6c104350 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909530705 0 0) 0x7f1d6c1187b0 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.018+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c1175b0 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2132538723 0 0) 0x7f1d58003030 con 0x7f1d6c100840 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6c1187b0 con 0x7f1d6c100840 2024-02-10T08:51:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d54001960 con 0x7f1d6c100840 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014576699 0 0) 0x7f1d6c1199b0 con 0x7f1d6c104350 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d58003030 con 0x7f1d6c104350 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d64001b80 con 0x7f1d6c104350 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3288309417 0 0) 0x7f1d6c1175b0 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6c1199b0 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d5c001f90 con 0x7f1d6c0fd9f0 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3072131075 0 0) 0x7f1d6c1187b0 con 0x7f1d6c100840 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 >> v1:172.21.15.195:6789/0 conn(0x7f1d6c0fd9f0 legacy=0x7f1d6c0f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 >> v1:172.21.15.107:6789/0 conn(0x7f1d6c104350 legacy=0x7f1d6c115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c11abb0 con 0x7f1d6c100840 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c1177a0 con 0x7f1d6c100840 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d7148b700 10 client.54635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c117c10 con 0x7f1d6c100840 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d7148b700 1 librados: init done 2024-02-10T08:51:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1d54002070 con 0x7f1d6c100840 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.019+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1d54005b30 con 0x7f1d6c100840 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.020+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c004f20 con 0x7f1d6c100840 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.020+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1d54001bd0 con 0x7f1d6c100840 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(835..835 src has 775..835) v4 ==== 7473+0+0 (unknown 791941780 0 0) 0x7f1d540789d0 con 0x7f1d6c100840 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 10 client.54635.objecter ms_dispatch 0x7f1d6c001550 osd_map(835..835 src has 775..835) v4 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 3 client.54635.objecter handle_osd_map got epochs [835,835] > 0 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 3 client.54635.objecter handle_osd_map decoding full epoch 835 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 20 client.54635.objecter dump_active .. 0 homeless 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.021+0000 7f1d63fff700 10 client.54635.objecter ms_handle_connect 0x7f1d58063c20 2024-02-10T08:51:49.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.025+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1d5404c010 con 0x7f1d6c100840 2024-02-10T08:51:49.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.187+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1d4c0051d0 con 0x7f1d6c100840 2024-02-10T08:51:49.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.187+0000 7f1d63fff700 1 -- 172.21.15.107:0/3851223897 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 981975939) 0x7f1d5404a420 con 0x7f1d6c100840 2024-02-10T08:51:49.188 INFO:teuthology.orchestra.run.smithi107.stdout:34359738628 2024-02-10T08:51:49.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 10 librados: watch_flush enter 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 10 librados: watch_flush exit 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 20 client.54635.objecter shutdown clearing up homeless session... 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 10 client.54635.objecter successfully canceled tick 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1d58063c20 legacy=0x7f1d580660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 >> v1:172.21.15.107:6790/0 conn(0x7f1d6c100840 legacy=0x7f1d6c0fa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 shutdown_connections 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 >> 172.21.15.107:0/3851223897 conn(0x7f1d6c06c090 msgr2=0x7f1d6c06c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 shutdown_connections 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 -- 172.21.15.107:0/3851223897 wait complete. 2024-02-10T08:51:49.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:49.189+0000 7f1d7148b700 1 librados: shutdown 2024-02-10T08:51:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.850271+0000 osd.6 (osd.6) 374 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:47.850496+0000 osd.6 (osd.6) 375 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.358426+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.665288+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.668611+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:49.188052+0000 mon.c (mon.2) 517 : audit [DBG] from='client.? v1:172.21.15.107:0/3851223897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.281793+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.282609+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.850271+0000 osd.6 (osd.6) 374 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:47.850496+0000 osd.6 (osd.6) 375 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.358426+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-10T08:51:49.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.665288+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:51:49.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.668611+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:51:49.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:49.188052+0000 mon.c (mon.2) 517 : audit [DBG] from='client.? v1:172.21.15.107:0/3851223897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:51:49.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.281793+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:51:49.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:49 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.282609+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.850271+0000 osd.6 (osd.6) 374 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:47.850496+0000 osd.6 (osd.6) 375 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.358426+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.665288+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.7 scrub starts 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.668611+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.7 scrub ok 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:49.188052+0000 mon.c (mon.2) 517 : audit [DBG] from='client.? v1:172.21.15.107:0/3851223897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.281793+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-02-10T08:51:49.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:49 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.282609+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-02-10T08:51:49.987 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738623 got 34359738628 for osd.0 2024-02-10T08:51:49.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T08:51:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.915039+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:48.917163+0000 osd.7 (osd.7) 377 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.316716+0000 osd.2 (osd.2) 367 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.316878+0000 osd.2 (osd.2) 368 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.358309+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-10T08:51:50.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.613200+0000 mgr.y (mgr.14553) 743 : cluster [DBG] pgmap v1320: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.950475+0000 osd.7 (osd.7) 378 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.951874+0000 osd.7 (osd.7) 379 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:49.997401+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:50.319918+0000 mon.c (mon.2) 519 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:50.320506+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:51:50.538 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:50 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:50.326516+0000 mon.a (mon.0) 3784 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.915039+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:48.917163+0000 osd.7 (osd.7) 377 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.316716+0000 osd.2 (osd.2) 367 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.316878+0000 osd.2 (osd.2) 368 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.358309+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.613200+0000 mgr.y (mgr.14553) 743 : cluster [DBG] pgmap v1320: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.950475+0000 osd.7 (osd.7) 378 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.951874+0000 osd.7 (osd.7) 379 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:49.997401+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:50.319918+0000 mon.c (mon.2) 519 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:51:50.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:50.320506+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:50.326516+0000 mon.a (mon.0) 3784 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.915039+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.1c scrub starts 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:48.917163+0000 osd.7 (osd.7) 377 : cluster [DBG] 6.1c scrub ok 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.316716+0000 osd.2 (osd.2) 367 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.316878+0000 osd.2 (osd.2) 368 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.358309+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.613200+0000 mgr.y (mgr.14553) 743 : cluster [DBG] pgmap v1320: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.950475+0000 osd.7 (osd.7) 378 : cluster [DBG] 2.5 scrub starts 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.951874+0000 osd.7 (osd.7) 379 : cluster [DBG] 2.5 scrub ok 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:49.997401+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:50.319918+0000 mon.c (mon.2) 519 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:51:50.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:50.320506+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:51:50.686 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:50.326516+0000 mon.a (mon.0) 3784 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:51:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.992660+0000 osd.2 (osd.2) 369 : cluster [DBG] 6.f scrub starts 2024-02-10T08:51:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:49.995978+0000 osd.2 (osd.2) 370 : cluster [DBG] 6.f scrub ok 2024-02-10T08:51:51.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:50.003696+0000 osd.1 (osd.1) 416 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:50.003926+0000 osd.1 (osd.1) 417 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:50.377320+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:51.281146+0000 osd.3 (osd.3) 437 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.992660+0000 osd.2 (osd.2) 369 : cluster [DBG] 6.f scrub starts 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:49.995978+0000 osd.2 (osd.2) 370 : cluster [DBG] 6.f scrub ok 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:50.003696+0000 osd.1 (osd.1) 416 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:50.003926+0000 osd.1 (osd.1) 417 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:50.377320+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-10T08:51:51.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:51 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:51.281146+0000 osd.3 (osd.3) 437 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.992660+0000 osd.2 (osd.2) 369 : cluster [DBG] 6.f scrub starts 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:49.995978+0000 osd.2 (osd.2) 370 : cluster [DBG] 6.f scrub ok 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:50.003696+0000 osd.1 (osd.1) 416 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:50.003926+0000 osd.1 (osd.1) 417 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:50.377320+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-10T08:51:51.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:51.281146+0000 osd.3 (osd.3) 437 : cluster [DBG] 4.0 deep-scrub starts 2024-02-10T08:51:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.163+0000 7fa3110b0700 1 -- 172.21.15.107:0/3156527276 >> v1:172.21.15.107:6789/0 conn(0x7fa30c104330 legacy=0x7fa30c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.163+0000 7fa3110b0700 1 -- 172.21.15.107:0/3156527276 shutdown_connections 2024-02-10T08:51:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.163+0000 7fa3110b0700 1 -- 172.21.15.107:0/3156527276 >> 172.21.15.107:0/3156527276 conn(0x7fa30c0f3640 msgr2=0x7fa30c0f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:52.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.163+0000 7fa3110b0700 1 -- 172.21.15.107:0/3156527276 shutdown_connections 2024-02-10T08:51:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.163+0000 7fa3110b0700 1 -- 172.21.15.107:0/3156527276 wait complete. 2024-02-10T08:51:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 librados: starting msgr at 2024-02-10T08:51:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 librados: starting objecter 2024-02-10T08:51:52.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 Processor -- start 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 -- start start 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 librados: setting wanted keys 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.164+0000 7fa3110b0700 1 librados: calling monclient init 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa30c117390 con 0x7fa30c0fda50 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa30c118570 con 0x7fa30c104330 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa30c119770 con 0x7fa30c1007f0 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa30a59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa30c1007f0 0x7fa30c0fa090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42246/0 (socket says 172.21.15.107:42246) 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa30ad9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa30c0fda50 0x7fa30c0f9980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55016/0 (socket says 172.21.15.107:55016) 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa30a59c700 1 -- 172.21.15.107:0/809535384 learned_addr learned my addr 172.21.15.107:0/809535384 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter ms_handle_connect 0x7fa30c1007f0 2024-02-10T08:51:52.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter ms_handle_connect 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter ms_handle_connect 0x7fa30c104330 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734897759 0 0) 0x7fa30c117390 con 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f80038e0 con 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164793196 0 0) 0x7fa30c119770 con 0x7fa30c1007f0 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30c117390 con 0x7fa30c1007f0 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902207491 0 0) 0x7fa30c118570 con 0x7fa30c104330 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30c119770 con 0x7fa30c104330 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721531982 0 0) 0x7fa2f80038e0 con 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa30c118570 con 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2f40018d0 con 0x7fa30c0fda50 2024-02-10T08:51:52.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147514112 0 0) 0x7fa30c117390 con 0x7fa30c1007f0 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f80038e0 con 0x7fa30c1007f0 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2fc001840 con 0x7fa30c1007f0 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3845062678 0 0) 0x7fa30c119770 con 0x7fa30c104330 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa30c117390 con 0x7fa30c104330 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3787567844 0 0) 0x7fa30c118570 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 >> v1:172.21.15.107:6790/0 conn(0x7fa30c1007f0 legacy=0x7fa30c0fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 >> v1:172.21.15.195:6789/0 conn(0x7fa30c104330 legacy=0x7fa30c115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c11a970 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c117580 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 10 client.29043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30c117a70 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.165+0000 7fa3110b0700 1 librados: init done 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.166+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fa2f40022c0 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.166+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fa2f40059c0 con 0x7fa30c0fda50 2024-02-10T08:51:52.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.166+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec004f20 con 0x7fa30c0fda50 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fa2f4001e20 con 0x7fa30c0fda50 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7fa2f40788c0 con 0x7fa30c0fda50 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 10 client.29043.objecter ms_dispatch 0x7fa30c001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 3 client.29043.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 3 client.29043.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.167+0000 7fa303fff700 20 client.29043.objecter dump_active .. 0 homeless 2024-02-10T08:51:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.168+0000 7fa303fff700 10 client.29043.objecter ms_handle_connect 0x7fa2f8063cd0 2024-02-10T08:51:52.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.171+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa2f404bf60 con 0x7fa30c0fda50 2024-02-10T08:51:52.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:51:51 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:51:52.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.333+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa2ec0051d0 con 0x7fa30c0fda50 2024-02-10T08:51:52.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.334+0000 7fa303fff700 1 -- 172.21.15.107:0/809535384 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 2661585867) 0x7fa2f404a370 con 0x7fa30c0fda50 2024-02-10T08:51:52.335 INFO:teuthology.orchestra.run.smithi107.stdout:55834575102 2024-02-10T08:51:52.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 10 librados: watch_flush enter 2024-02-10T08:51:52.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 10 librados: watch_flush exit 2024-02-10T08:51:52.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 20 client.29043.objecter shutdown clearing up homeless session... 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 10 client.29043.objecter successfully canceled tick 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 >> v1:172.21.15.107:6800/2634842009 conn(0x7fa2f8063cd0 legacy=0x7fa2f8066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 >> v1:172.21.15.107:6789/0 conn(0x7fa30c0fda50 legacy=0x7fa30c0f9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 shutdown_connections 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.335+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 >> 172.21.15.107:0/809535384 conn(0x7fa30c0f3640 msgr2=0x7fa30c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.336+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 shutdown_connections 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.336+0000 7fa3110b0700 1 -- 172.21.15.107:0/809535384 wait complete. 2024-02-10T08:51:52.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:52.336+0000 7fa3110b0700 1 librados: shutdown 2024-02-10T08:51:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:51.296684+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:51:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:51.614020+0000 mgr.y (mgr.14553) 744 : cluster [DBG] pgmap v1322: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:51:52.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:51.959894+0000 mgr.y (mgr.14553) 745 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:52.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:52.334766+0000 mon.a (mon.0) 3786 : audit [DBG] from='client.? v1:172.21.15.107:0/809535384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:51:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:51.296684+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:51:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:51.614020+0000 mgr.y (mgr.14553) 744 : cluster [DBG] pgmap v1322: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:51:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:51.959894+0000 mgr.y (mgr.14553) 745 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:52 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:52.334766+0000 mon.a (mon.0) 3786 : audit [DBG] from='client.? v1:172.21.15.107:0/809535384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:51:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:51.296684+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.0 deep-scrub ok 2024-02-10T08:51:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:51.614020+0000 mgr.y (mgr.14553) 744 : cluster [DBG] pgmap v1322: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T08:51:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:51.959894+0000 mgr.y (mgr.14553) 745 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:51:52.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:52 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:52.334766+0000 mon.a (mon.0) 3786 : audit [DBG] from='client.? v1:172.21.15.107:0/809535384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T08:51:53.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575098 got 55834575102 for osd.1 2024-02-10T08:51:53.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T08:51:53.313 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:53.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:51.974446+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:51:53.658 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:51.975465+0000 osd.2 (osd.2) 372 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:52.643424+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:52.646419+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:51.974446+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:51.975465+0000 osd.2 (osd.2) 372 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:52.643424+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:51:53.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:52.646419+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:51:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:51.974446+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.10 scrub starts 2024-02-10T08:51:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:51.975465+0000 osd.2 (osd.2) 372 : cluster [DBG] 16.10 scrub ok 2024-02-10T08:51:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:52.643424+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.18 scrub starts 2024-02-10T08:51:53.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:52.646419+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.18 scrub ok 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:53.322562+0000 osd.0 (osd.0) 240 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:53.323695+0000 osd.0 (osd.0) 241 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:53.614466+0000 mgr.y (mgr.14553) 746 : cluster [DBG] pgmap v1323: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:53.834012+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:53.835035+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:53.322562+0000 osd.0 (osd.0) 240 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:51:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:53.323695+0000 osd.0 (osd.0) 241 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:51:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:53.614466+0000 mgr.y (mgr.14553) 746 : cluster [DBG] pgmap v1323: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:53.834012+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:51:54.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:54 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:53.835035+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:51:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:53.322562+0000 osd.0 (osd.0) 240 : cluster [DBG] 16.0 scrub starts 2024-02-10T08:51:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:53.323695+0000 osd.0 (osd.0) 241 : cluster [DBG] 16.0 scrub ok 2024-02-10T08:51:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:53.614466+0000 mgr.y (mgr.14553) 746 : cluster [DBG] pgmap v1323: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:51:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:53.834012+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1c scrub starts 2024-02-10T08:51:54.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:53.835035+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1c scrub ok 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- 172.21.15.107:0/2902975334 >> v1:172.21.15.107:6789/0 conn(0x7f4120104330 legacy=0x7f41201067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- 172.21.15.107:0/2902975334 shutdown_connections 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- 172.21.15.107:0/2902975334 >> 172.21.15.107:0/2902975334 conn(0x7f41200f3640 msgr2=0x7f41200f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- 172.21.15.107:0/2902975334 shutdown_connections 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- 172.21.15.107:0/2902975334 wait complete. 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 librados: starting msgr at 2024-02-10T08:51:55.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 librados: starting objecter 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 Processor -- start 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 -- start start 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 librados: setting wanted keys 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.215+0000 7f4125d31700 1 librados: calling monclient init 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f4125d31700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4120119620 con 0x7f4120104330 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f4125d31700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f412011a800 con 0x7f41201007f0 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f4125d31700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f412011ba00 con 0x7f41200fda50 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411f7fe700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f41200fda50 0x7f41201025a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42268/0 (socket says 172.21.15.107:42268) 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411f7fe700 1 -- 172.21.15.107:0/2705068440 learned_addr learned my addr 172.21.15.107:0/2705068440 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter ms_handle_connect 0x7f41200fda50 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter ms_handle_connect 0x7f4120104330 2024-02-10T08:51:55.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter ms_handle_connect 0x7f41201007f0 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066146041 0 0) 0x7f4120119620 con 0x7f4120104330 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c003800 con 0x7f4120104330 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159688529 0 0) 0x7f412011ba00 con 0x7f41200fda50 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4120119620 con 0x7f41200fda50 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719474116 0 0) 0x7f412011a800 con 0x7f41201007f0 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f412011ba00 con 0x7f41201007f0 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583497100 0 0) 0x7f410c003800 con 0x7f4120104330 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f412011a800 con 0x7f4120104330 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4110002ea0 con 0x7f4120104330 2024-02-10T08:51:55.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3044792745 0 0) 0x7f4120119620 con 0x7f41200fda50 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c003800 con 0x7f41200fda50 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325111493 0 0) 0x7f412011ba00 con 0x7f41201007f0 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4120119620 con 0x7f41201007f0 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4118001d70 con 0x7f41200fda50 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.216+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4108002870 con 0x7f41201007f0 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3133722058 0 0) 0x7f412011a800 con 0x7f4120104330 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 >> v1:172.21.15.107:6790/0 conn(0x7f41200fda50 legacy=0x7f41201025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 >> v1:172.21.15.195:6789/0 conn(0x7f41201007f0 legacy=0x7f4120102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412011cc00 con 0x7f4120104330 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4120119810 con 0x7f4120104330 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f4125d31700 10 client.29049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4120119cc0 con 0x7f4120104330 2024-02-10T08:51:55.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f4125d31700 1 librados: init done 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f4110002270 con 0x7f4120104330 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.217+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f4110003600 con 0x7f4120104330 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.218+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100004f20 con 0x7f4120104330 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.218+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f4110001820 con 0x7f4120104330 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7f4110076430 con 0x7f4120104330 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 10 client.29049.objecter ms_dispatch 0x7f4120001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 3 client.29049.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 3 client.29049.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 20 client.29049.objecter dump_active .. 0 homeless 2024-02-10T08:51:55.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.219+0000 7f411cff9700 10 client.29049.objecter ms_handle_connect 0x7f410c063ce0 2024-02-10T08:51:55.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.223+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4110049ad0 con 0x7f4120104330 2024-02-10T08:51:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.392+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f41000051d0 con 0x7f4120104330 2024-02-10T08:51:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.392+0000 7f411cff9700 1 -- 172.21.15.107:0/2705068440 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3174006534) 0x7f4110047f40 con 0x7f4120104330 2024-02-10T08:51:55.393 INFO:teuthology.orchestra.run.smithi107.stdout:77309411579 2024-02-10T08:51:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.394+0000 7f4125d31700 10 librados: watch_flush enter 2024-02-10T08:51:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.394+0000 7f4125d31700 10 librados: watch_flush exit 2024-02-10T08:51:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.394+0000 7f4125d31700 20 client.29049.objecter shutdown clearing up homeless session... 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.394+0000 7f4125d31700 10 client.29049.objecter successfully canceled tick 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.394+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 >> v1:172.21.15.107:6800/2634842009 conn(0x7f410c063ce0 legacy=0x7f410c0661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 >> v1:172.21.15.107:6789/0 conn(0x7f4120104330 legacy=0x7f4120117da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 shutdown_connections 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 >> 172.21.15.107:0/2705068440 conn(0x7f41200f3640 msgr2=0x7f41200f64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 shutdown_connections 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 -- 172.21.15.107:0/2705068440 wait complete. 2024-02-10T08:51:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:55.395+0000 7f4125d31700 1 librados: shutdown 2024-02-10T08:51:55.680 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:51:55] "GET /metrics HTTP/1.1" 200 186421 "" "Prometheus/2.33.4" 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.280142+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.b scrub starts 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.282030+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub ok 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.863746+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.866686+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.955673+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.956816+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:55.393481+0000 mon.a (mon.0) 3787 : audit [DBG] from='client.? v1:172.21.15.107:0/2705068440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.280142+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.b scrub starts 2024-02-10T08:51:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.282030+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub ok 2024-02-10T08:51:55.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.863746+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:51:55.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.866686+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:51:55.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.955673+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:51:55.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.956816+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:51:55.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:55 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:55.393481+0000 mon.a (mon.0) 3787 : audit [DBG] from='client.? v1:172.21.15.107:0/2705068440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.280142+0000 osd.0 (osd.0) 242 : cluster [DBG] 4.b scrub starts 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.282030+0000 osd.0 (osd.0) 243 : cluster [DBG] 4.b scrub ok 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.863746+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.866686+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.955673+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.17 scrub starts 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.956816+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.17 scrub ok 2024-02-10T08:51:55.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:55.393481+0000 mon.a (mon.0) 3787 : audit [DBG] from='client.? v1:172.21.15.107:0/2705068440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T08:51:56.212 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411574 got 77309411579 for osd.2 2024-02-10T08:51:56.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T08:51:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.992839+0000 osd.2 (osd.2) 373 : cluster [DBG] 16.a scrub starts 2024-02-10T08:51:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:54.996105+0000 osd.2 (osd.2) 374 : cluster [DBG] 16.a scrub ok 2024-02-10T08:51:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:55.614886+0000 mgr.y (mgr.14553) 747 : cluster [DBG] pgmap v1324: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-02-10T08:51:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.992839+0000 osd.2 (osd.2) 373 : cluster [DBG] 16.a scrub starts 2024-02-10T08:51:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:54.996105+0000 osd.2 (osd.2) 374 : cluster [DBG] 16.a scrub ok 2024-02-10T08:51:56.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:56 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:55.614886+0000 mgr.y (mgr.14553) 747 : cluster [DBG] pgmap v1324: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-02-10T08:51:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.992839+0000 osd.2 (osd.2) 373 : cluster [DBG] 16.a scrub starts 2024-02-10T08:51:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:54.996105+0000 osd.2 (osd.2) 374 : cluster [DBG] 16.a scrub ok 2024-02-10T08:51:56.787 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:55.614886+0000 mgr.y (mgr.14553) 747 : cluster [DBG] pgmap v1324: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-02-10T08:51:58.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2566777341 >> v1:172.21.15.107:6790/0 conn(0x7f2eac0ff710 legacy=0x7f2eac101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2566777341 shutdown_connections 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2566777341 >> 172.21.15.107:0/2566777341 conn(0x7f2eac0f2510 msgr2=0x7f2eac0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2566777341 shutdown_connections 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2566777341 wait complete. 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 librados: starting msgr at 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 librados: starting objecter 2024-02-10T08:51:58.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.333+0000 7f2eb1c39700 1 Processor -- start 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 -- start start 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 librados: setting wanted keys 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 librados: calling monclient init 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2eac117380 con 0x7f2eac103250 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2eac118560 con 0x7f2eac0fc910 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eb1c39700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2eac119760 con 0x7f2eac0ff710 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eabfff700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2eac103250 0x7f2eac115a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55056/0 (socket says 172.21.15.107:55056) 2024-02-10T08:51:58.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eaaffd700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2eac0ff710 0x7f2eac0fa220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42288/0 (socket says 172.21.15.107:42288) 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2eabfff700 1 -- 172.21.15.107:0/2658595559 learned_addr learned my addr 172.21.15.107:0/2658595559 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter ms_handle_connect 0x7f2eac103250 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter ms_handle_connect 0x7f2eac0ff710 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter ms_handle_connect 0x7f2eac0fc910 2024-02-10T08:51:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 10 client.?.objecter resend_mon_ops 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270490897 0 0) 0x7f2eac117380 con 0x7f2eac103250 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e98003030 con 0x7f2eac103250 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043907290 0 0) 0x7f2eac119760 con 0x7f2eac0ff710 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eac117380 con 0x7f2eac0ff710 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056721118 0 0) 0x7f2eac118560 con 0x7f2eac0fc910 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eac119760 con 0x7f2eac0fc910 2024-02-10T08:51:58.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196861955 0 0) 0x7f2e98003030 con 0x7f2eac103250 2024-02-10T08:51:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eac118560 con 0x7f2eac103250 2024-02-10T08:51:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.334+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2ea4001ca0 con 0x7f2eac103250 2024-02-10T08:51:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573152281 0 0) 0x7f2eac117380 con 0x7f2eac0ff710 2024-02-10T08:51:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e98003030 con 0x7f2eac0ff710 2024-02-10T08:51:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2e9c002e10 con 0x7f2eac0ff710 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406550418 0 0) 0x7f2eac119760 con 0x7f2eac0fc910 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eac117380 con 0x7f2eac0fc910 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2e940018c0 con 0x7f2eac0fc910 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 473142338 0 0) 0x7f2eac118560 con 0x7f2eac103250 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 >> v1:172.21.15.107:6790/0 conn(0x7f2eac0ff710 legacy=0x7f2eac0fa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:58.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 >> v1:172.21.15.195:6789/0 conn(0x7f2eac0fc910 legacy=0x7f2eac0f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac11a960 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eac119950 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2eb1c39700 10 client.29055.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac119e40 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2eb1c39700 1 librados: init done 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f2ea4002800 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.335+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f2ea4005ef0 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.336+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c004f20 con 0x7f2eac103250 2024-02-10T08:51:58.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.336+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f2ea4002360 con 0x7f2eac103250 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7f2ea4078e10 con 0x7f2eac103250 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 10 client.29055.objecter ms_dispatch 0x7f2eac001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 3 client.29055.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 3 client.29055.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 20 client.29055.objecter dump_active .. 0 homeless 2024-02-10T08:51:58.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.337+0000 7f2ea8ff9700 10 client.29055.objecter ms_handle_connect 0x7f2e98063cc0 2024-02-10T08:51:58.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.341+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2ea404c4b0 con 0x7f2eac103250 2024-02-10T08:51:58.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.506+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2e8c0051d0 con 0x7f2eac103250 2024-02-10T08:51:58.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.506+0000 7f2ea8ff9700 1 -- 172.21.15.107:0/2658595559 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 1654882922) 0x7f2ea4002970 con 0x7f2eac103250 2024-02-10T08:51:58.507 INFO:teuthology.orchestra.run.smithi107.stdout:107374182648 2024-02-10T08:51:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.507+0000 7f2eb1c39700 10 librados: watch_flush enter 2024-02-10T08:51:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.507+0000 7f2eb1c39700 10 librados: watch_flush exit 2024-02-10T08:51:58.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.507+0000 7f2eb1c39700 20 client.29055.objecter shutdown clearing up homeless session... 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.507+0000 7f2eb1c39700 10 client.29055.objecter successfully canceled tick 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 >> v1:172.21.15.107:6800/2634842009 conn(0x7f2e98063cc0 legacy=0x7f2e98066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 >> v1:172.21.15.107:6789/0 conn(0x7f2eac103250 legacy=0x7f2eac115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 shutdown_connections 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 >> 172.21.15.107:0/2658595559 conn(0x7f2eac0f2510 msgr2=0x7f2eac0612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 shutdown_connections 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 -- 172.21.15.107:0/2658595559 wait complete. 2024-02-10T08:51:58.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:51:58.508+0000 7f2eb1c39700 1 librados: shutdown 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:57.615549+0000 mgr.y (mgr.14553) 748 : cluster [DBG] pgmap v1325: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:58.221718+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:58.222633+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:51:58.506954+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? v1:172.21.15.107:0/2658595559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:57.615549+0000 mgr.y (mgr.14553) 748 : cluster [DBG] pgmap v1325: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:58.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:58.221718+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:51:58.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:58.222633+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:51:58.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:58 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:51:58.506954+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? v1:172.21.15.107:0/2658595559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:51:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:57.615549+0000 mgr.y (mgr.14553) 748 : cluster [DBG] pgmap v1325: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:51:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:58.221718+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.11 scrub starts 2024-02-10T08:51:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:58.222633+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.11 scrub ok 2024-02-10T08:51:59.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:51:58.506954+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? v1:172.21.15.107:0/2658595559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T08:51:59.156 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182643 got 107374182648 for osd.3 2024-02-10T08:51:59.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T08:51:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:51:59.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:58.912210+0000 osd.6 (osd.6) 378 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:58.913183+0000 osd.6 (osd.6) 379 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.263485+0000 osd.3 (osd.3) 441 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.264423+0000 osd.3 (osd.3) 442 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:58.912210+0000 osd.6 (osd.6) 378 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:58.913183+0000 osd.6 (osd.6) 379 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.263485+0000 osd.3 (osd.3) 441 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:51:59.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:51:59 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.264423+0000 osd.3 (osd.3) 442 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:52:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:58.912210+0000 osd.6 (osd.6) 378 : cluster [DBG] 16.6 scrub starts 2024-02-10T08:52:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:58.913183+0000 osd.6 (osd.6) 379 : cluster [DBG] 16.6 scrub ok 2024-02-10T08:52:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.263485+0000 osd.3 (osd.3) 441 : cluster [DBG] 2.1d scrub starts 2024-02-10T08:52:00.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:51:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.264423+0000 osd.3 (osd.3) 442 : cluster [DBG] 2.1d scrub ok 2024-02-10T08:52:00.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:52:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:52:00.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.513772+0000 osd.5 (osd.5) 408 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:52:00.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.516957+0000 osd.5 (osd.5) 409 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:52:00.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.615951+0000 mgr.y (mgr.14553) 749 : cluster [DBG] pgmap v1326: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:52:00.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.513772+0000 osd.5 (osd.5) 408 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:52:00.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.516957+0000 osd.5 (osd.5) 409 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:52:00.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.615951+0000 mgr.y (mgr.14553) 749 : cluster [DBG] pgmap v1326: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:52:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.513772+0000 osd.5 (osd.5) 408 : cluster [DBG] 6.19 scrub starts 2024-02-10T08:52:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.516957+0000 osd.5 (osd.5) 409 : cluster [DBG] 6.19 scrub ok 2024-02-10T08:52:01.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.615951+0000 mgr.y (mgr.14553) 749 : cluster [DBG] pgmap v1326: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T08:52:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.128+0000 7fca5c7b2700 1 -- 172.21.15.107:0/980709145 >> v1:172.21.15.107:6789/0 conn(0x7fca540ff710 legacy=0x7fca54101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.128+0000 7fca5c7b2700 1 -- 172.21.15.107:0/980709145 shutdown_connections 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.128+0000 7fca5c7b2700 1 -- 172.21.15.107:0/980709145 >> 172.21.15.107:0/980709145 conn(0x7fca540f2510 msgr2=0x7fca540f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.128+0000 7fca5c7b2700 1 -- 172.21.15.107:0/980709145 shutdown_connections 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 -- 172.21.15.107:0/980709145 wait complete. 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 librados: starting msgr at 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 librados: starting objecter 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 Processor -- start 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 -- start start 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 librados: setting wanted keys 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 librados: calling monclient init 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca54061870 con 0x7fca540ff710 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca54118720 con 0x7fca54103250 2024-02-10T08:52:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5c7b2700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca541198c0 con 0x7fca540fc910 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5a54e700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fca540fc910 0x7fca54060e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45530/0 (socket says 172.21.15.107:45530) 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.129+0000 7fca5a54e700 1 -- 172.21.15.107:0/4243706688 learned_addr learned my addr 172.21.15.107:0/4243706688 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter ms_handle_connect 0x7fca540fc910 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter ms_handle_connect 0x7fca540ff710 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter ms_handle_connect 0x7fca54103250 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010826908 0 0) 0x7fca541198c0 con 0x7fca540fc910 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca400033f0 con 0x7fca540fc910 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10509467 0 0) 0x7fca54061870 con 0x7fca540ff710 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca541198c0 con 0x7fca540ff710 2024-02-10T08:52:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137929176 0 0) 0x7fca54118720 con 0x7fca54103250 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54061870 con 0x7fca54103250 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987537911 0 0) 0x7fca400033f0 con 0x7fca540fc910 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca54118720 con 0x7fca540fc910 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fca500019d0 con 0x7fca540fc910 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2986222211 0 0) 0x7fca541198c0 con 0x7fca540ff710 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca400033f0 con 0x7fca540ff710 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fca44002060 con 0x7fca540ff710 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 905478302 0 0) 0x7fca54061870 con 0x7fca54103250 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca541198c0 con 0x7fca54103250 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fca4c001bc0 con 0x7fca54103250 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482289172 0 0) 0x7fca54118720 con 0x7fca540fc910 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 >> v1:172.21.15.195:6789/0 conn(0x7fca54103250 legacy=0x7fca54115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 >> v1:172.21.15.107:6789/0 conn(0x7fca540ff710 legacy=0x7fca541128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5411aa60 con 0x7fca540fc910 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca54117770 con 0x7fca540fc910 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca5c7b2700 10 client.54659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca54117be0 con 0x7fca540fc910 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca5c7b2700 1 librados: init done 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fca500023c0 con 0x7fca540fc910 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.130+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fca50005a00 con 0x7fca540fc910 2024-02-10T08:52:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.131+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca38004f20 con 0x7fca540fc910 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.131+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fca50001f20 con 0x7fca540fc910 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7fca50078a90 con 0x7fca540fc910 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 10 client.54659.objecter ms_dispatch 0x7fca54001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 3 client.54659.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 3 client.54659.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 20 client.54659.objecter dump_active .. 0 homeless 2024-02-10T08:52:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.132+0000 7fca4b7fe700 10 client.54659.objecter ms_handle_connect 0x7fca40063cc0 2024-02-10T08:52:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.136+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fca5004c110 con 0x7fca540fc910 2024-02-10T08:52:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.298+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fca380051d0 con 0x7fca540fc910 2024-02-10T08:52:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.298+0000 7fca4b7fe700 1 -- 172.21.15.107:0/4243706688 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+14 (unknown 1823589948 0 605629904) 0x7fca5004a580 con 0x7fca540fc910 2024-02-10T08:52:01.299 INFO:teuthology.orchestra.run.smithi107.stdout:2718714298436 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 10 librados: watch_flush enter 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 10 librados: watch_flush exit 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 20 client.54659.objecter shutdown clearing up homeless session... 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 10 client.54659.objecter successfully canceled tick 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 >> v1:172.21.15.107:6800/2634842009 conn(0x7fca40063cc0 legacy=0x7fca40066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 >> v1:172.21.15.107:6790/0 conn(0x7fca540fc910 legacy=0x7fca54060e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 shutdown_connections 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 >> 172.21.15.107:0/4243706688 conn(0x7fca540f2510 msgr2=0x7fca541066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 shutdown_connections 2024-02-10T08:52:01.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 -- 172.21.15.107:0/4243706688 wait complete. 2024-02-10T08:52:01.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:01.300+0000 7fca5c7b2700 1 librados: shutdown 2024-02-10T08:52:01.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.917781+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.b deep-scrub starts 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:51:59.920993+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.b deep-scrub ok 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:00.886838+0000 osd.7 (osd.7) 384 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:00.889806+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:01.299030+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? v1:172.21.15.107:0/4243706688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.917781+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.b deep-scrub starts 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:51:59.920993+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.b deep-scrub ok 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:00.886838+0000 osd.7 (osd.7) 384 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:00.889806+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:52:01.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:01 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:01.299030+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? v1:172.21.15.107:0/4243706688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:52:01.962 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.917781+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.b deep-scrub starts 2024-02-10T08:52:01.962 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:51:59.920993+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.b deep-scrub ok 2024-02-10T08:52:01.962 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:00.886838+0000 osd.7 (osd.7) 384 : cluster [DBG] 2.0 scrub starts 2024-02-10T08:52:01.962 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:00.889806+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.0 scrub ok 2024-02-10T08:52:01.962 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:01.299030+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? v1:172.21.15.107:0/4243706688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T08:52:02.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2718714298431 got 2718714298436 for osd.4 2024-02-10T08:52:02.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T08:52:02.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:52:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:52:02.366 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:02.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:01.330425+0000 osd.0 (osd.0) 244 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:01.331439+0000 osd.0 (osd.0) 245 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:01.546655+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:01.550267+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:01.616563+0000 mgr.y (mgr.14553) 750 : cluster [DBG] pgmap v1327: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:01.961780+0000 mgr.y (mgr.14553) 751 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:01.330425+0000 osd.0 (osd.0) 244 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:01.331439+0000 osd.0 (osd.0) 245 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:01.546655+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:01.550267+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:01.616563+0000 mgr.y (mgr.14553) 750 : cluster [DBG] pgmap v1327: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:52:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:02 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:01.961780+0000 mgr.y (mgr.14553) 751 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:01.330425+0000 osd.0 (osd.0) 244 : cluster [DBG] 16.14 scrub starts 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:01.331439+0000 osd.0 (osd.0) 245 : cluster [DBG] 16.14 scrub ok 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:01.546655+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.15 deep-scrub starts 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:01.550267+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.15 deep-scrub ok 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:01.616563+0000 mgr.y (mgr.14553) 750 : cluster [DBG] pgmap v1327: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T08:52:03.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:01.961780+0000 mgr.y (mgr.14553) 751 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:04.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 -- 172.21.15.107:0/1771249196 >> v1:172.21.15.107:6789/0 conn(0x7f5594104350 legacy=0x7f55941067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 -- 172.21.15.107:0/1771249196 shutdown_connections 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 -- 172.21.15.107:0/1771249196 >> 172.21.15.107:0/1771249196 conn(0x7f55940f2510 msgr2=0x7f55940f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 -- 172.21.15.107:0/1771249196 shutdown_connections 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 -- 172.21.15.107:0/1771249196 wait complete. 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 librados: starting msgr at 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.308+0000 7f55988ea700 1 librados: starting objecter 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 Processor -- start 2024-02-10T08:52:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 -- start start 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 librados: setting wanted keys 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 librados: calling monclient init 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5594117470 con 0x7f5594104350 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5594118650 con 0x7f55940f9570 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55988ea700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5594119850 con 0x7f55940f89c0 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f559259c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f55940f89c0 0x7f55940ff760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45550/0 (socket says 172.21.15.107:45550) 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f559259c700 1 -- 172.21.15.107:0/2827644980 learned_addr learned my addr 172.21.15.107:0/2827644980 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter ms_handle_connect 0x7f55940f89c0 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter ms_handle_connect 0x7f5594104350 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:04.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997610517 0 0) 0x7f5594117470 con 0x7f5594104350 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c0038e0 con 0x7f5594104350 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739199856 0 0) 0x7f5594119850 con 0x7f55940f89c0 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5594117470 con 0x7f55940f89c0 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter ms_handle_connect 0x7f55940f9570 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.309+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 946054859 0 0) 0x7f557c0038e0 con 0x7f5594104350 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5594119850 con 0x7f5594104350 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183181995 0 0) 0x7f5594118650 con 0x7f55940f9570 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c0038e0 con 0x7f55940f9570 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f558c001360 con 0x7f5594104350 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1767305389 0 0) 0x7f5594117470 con 0x7f55940f89c0 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5594118650 con 0x7f55940f89c0 2024-02-10T08:52:04.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f55840018d0 con 0x7f55940f89c0 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3706711990 0 0) 0x7f5594119850 con 0x7f5594104350 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 >> v1:172.21.15.107:6790/0 conn(0x7f55940f89c0 legacy=0x7f55940ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 >> v1:172.21.15.195:6789/0 conn(0x7f55940f9570 legacy=0x7f55940ffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559411aa50 con 0x7f5594104350 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5594117660 con 0x7f5594104350 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55988ea700 10 client.29067.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5594117ad0 con 0x7f5594104350 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55988ea700 1 librados: init done 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f558c0032b0 con 0x7f5594104350 2024-02-10T08:52:04.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.310+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f558c003950 con 0x7f5594104350 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.311+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5574004f20 con 0x7f5594104350 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.311+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f558c001ba0 con 0x7f5594104350 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7f558c076730 con 0x7f5594104350 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 10 client.29067.objecter ms_dispatch 0x7f5594001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 3 client.29067.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 3 client.29067.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:52:04.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 20 client.29067.objecter dump_active .. 0 homeless 2024-02-10T08:52:04.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.312+0000 7f55837fe700 10 client.29067.objecter ms_handle_connect 0x7f557c063ce0 2024-02-10T08:52:04.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.316+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f558c049d70 con 0x7f5594104350 2024-02-10T08:52:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.480+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f55740051d0 con 0x7f5594104350 2024-02-10T08:52:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.480+0000 7f55837fe700 1 -- 172.21.15.107:0/2827644980 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 2992343877) 0x7f558c0481e0 con 0x7f5594104350 2024-02-10T08:52:04.481 INFO:teuthology.orchestra.run.smithi107.stdout:158913790191 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 10 librados: watch_flush enter 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 10 librados: watch_flush exit 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 20 client.29067.objecter shutdown clearing up homeless session... 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 10 client.29067.objecter successfully canceled tick 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 >> v1:172.21.15.107:6800/2634842009 conn(0x7f557c063ce0 legacy=0x7f557c0661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 >> v1:172.21.15.107:6789/0 conn(0x7f5594104350 legacy=0x7f5594115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 shutdown_connections 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 >> 172.21.15.107:0/2827644980 conn(0x7f55940f2510 msgr2=0x7f55941077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 shutdown_connections 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 -- 172.21.15.107:0/2827644980 wait complete. 2024-02-10T08:52:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:04.482+0000 7f55988ea700 1 librados: shutdown 2024-02-10T08:52:04.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:02.897506+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.d scrub starts 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:02.900915+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.d scrub ok 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:03.541827+0000 osd.5 (osd.5) 412 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:03.545686+0000 osd.5 (osd.5) 413 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:03.617109+0000 mgr.y (mgr.14553) 752 : cluster [DBG] pgmap v1328: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:03.833539+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:03.836448+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:04.481421+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? v1:172.21.15.107:0/2827644980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:02.897506+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.d scrub starts 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:02.900915+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.d scrub ok 2024-02-10T08:52:04.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:03.541827+0000 osd.5 (osd.5) 412 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:52:04.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:03.545686+0000 osd.5 (osd.5) 413 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:52:04.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:03.617109+0000 mgr.y (mgr.14553) 752 : cluster [DBG] pgmap v1328: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:04.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:03.833539+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:52:04.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:03.836448+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:52:04.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:04 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:04.481421+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? v1:172.21.15.107:0/2827644980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:02.897506+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.d scrub starts 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:02.900915+0000 osd.2 (osd.2) 378 : cluster [DBG] 5.d scrub ok 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:03.541827+0000 osd.5 (osd.5) 412 : cluster [DBG] 16.1b scrub starts 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:03.545686+0000 osd.5 (osd.5) 413 : cluster [DBG] 16.1b scrub ok 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:03.617109+0000 mgr.y (mgr.14553) 752 : cluster [DBG] pgmap v1328: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:03.833539+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.2 scrub starts 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:03.836448+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2024-02-10T08:52:05.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:04.481421+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? v1:172.21.15.107:0/2827644980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T08:52:05.264 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790186 got 158913790191 for osd.5 2024-02-10T08:52:05.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T08:52:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:05.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:52:05] "GET /metrics HTTP/1.1" 200 186421 "" "Prometheus/2.33.4" 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.392832+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.393873+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.816797+0000 osd.4 (osd.4) 145 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.819959+0000 osd.4 (osd.4) 146 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.994343+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:04.997563+0000 osd.7 (osd.7) 387 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.392832+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.393873+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:52:05.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.816797+0000 osd.4 (osd.4) 145 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:52:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.819959+0000 osd.4 (osd.4) 146 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:52:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.994343+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:52:05.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:05 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:04.997563+0000 osd.7 (osd.7) 387 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.392832+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 deep-scrub starts 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.393873+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 deep-scrub ok 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.816797+0000 osd.4 (osd.4) 145 : cluster [DBG] 16.2 scrub starts 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.819959+0000 osd.4 (osd.4) 146 : cluster [DBG] 16.2 scrub ok 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.994343+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.3 scrub starts 2024-02-10T08:52:06.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:04.997563+0000 osd.7 (osd.7) 387 : cluster [DBG] 6.3 scrub ok 2024-02-10T08:52:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.588336+0000 osd.5 (osd.5) 414 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:52:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.591300+0000 osd.5 (osd.5) 415 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:52:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.617477+0000 mgr.y (mgr.14553) 753 : cluster [DBG] pgmap v1329: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.832257+0000 osd.4 (osd.4) 147 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:52:07.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.835827+0000 osd.4 (osd.4) 148 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.588336+0000 osd.5 (osd.5) 414 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.591300+0000 osd.5 (osd.5) 415 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.617477+0000 mgr.y (mgr.14553) 753 : cluster [DBG] pgmap v1329: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.832257+0000 osd.4 (osd.4) 147 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.835827+0000 osd.4 (osd.4) 148 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.588336+0000 osd.5 (osd.5) 414 : cluster [DBG] 2.3 scrub starts 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.591300+0000 osd.5 (osd.5) 415 : cluster [DBG] 2.3 scrub ok 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.617477+0000 mgr.y (mgr.14553) 753 : cluster [DBG] pgmap v1329: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.832257+0000 osd.4 (osd.4) 147 : cluster [DBG] 16.19 deep-scrub starts 2024-02-10T08:52:07.125 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:06 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.835827+0000 osd.4 (osd.4) 148 : cluster [DBG] 16.19 deep-scrub ok 2024-02-10T08:52:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.348+0000 7f198cb7c700 1 -- 172.21.15.107:0/259147776 >> v1:172.21.15.107:6790/0 conn(0x7f19880ff710 legacy=0x7f1988101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.348+0000 7f198cb7c700 1 -- 172.21.15.107:0/259147776 shutdown_connections 2024-02-10T08:52:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.348+0000 7f198cb7c700 1 -- 172.21.15.107:0/259147776 >> 172.21.15.107:0/259147776 conn(0x7f19880f2510 msgr2=0x7f19880f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.348+0000 7f198cb7c700 1 -- 172.21.15.107:0/259147776 shutdown_connections 2024-02-10T08:52:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.348+0000 7f198cb7c700 1 -- 172.21.15.107:0/259147776 wait complete. 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 librados: starting msgr at 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 librados: starting objecter 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 Processor -- start 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 -- start start 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 librados: setting wanted keys 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.349+0000 7f198cb7c700 1 librados: calling monclient init 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f198cb7c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19880fbff0 con 0x7f1988103250 2024-02-10T08:52:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f198cb7c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19880fc1c0 con 0x7f19880fc910 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f198cb7c700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f198811bb20 con 0x7f19880ff710 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f1985d9b700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f19880ff710 0x7f1988114cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45564/0 (socket says 172.21.15.107:45564) 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f1986d9d700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1988103250 0x7f19881183e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44610/0 (socket says 172.21.15.107:44610) 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f1985d9b700 1 -- 172.21.15.107:0/2667789578 learned_addr learned my addr 172.21.15.107:0/2667789578 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter ms_handle_connect 0x7f19880ff710 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter ms_handle_connect 0x7f1988103250 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901496940 0 0) 0x7f198811bb20 con 0x7f19880ff710 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19700038e0 con 0x7f19880ff710 2024-02-10T08:52:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter ms_handle_connect 0x7f19880fc910 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365405560 0 0) 0x7f19880fbff0 con 0x7f1988103250 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f198811bb20 con 0x7f1988103250 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123483700 0 0) 0x7f19880fc1c0 con 0x7f19880fc910 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19880fbff0 con 0x7f19880fc910 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530542371 0 0) 0x7f19700038e0 con 0x7f19880ff710 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19880fc1c0 con 0x7f19880ff710 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f19780015d0 con 0x7f19880ff710 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242394606 0 0) 0x7f198811bb20 con 0x7f1988103250 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19700038e0 con 0x7f1988103250 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1980001df0 con 0x7f1988103250 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427498236 0 0) 0x7f19880fbff0 con 0x7f19880fc910 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f198811bb20 con 0x7f19880fc910 2024-02-10T08:52:07.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963486837 0 0) 0x7f19880fc1c0 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 >> v1:172.21.15.195:6789/0 conn(0x7f19880fc910 legacy=0x7f19880fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 >> v1:172.21.15.107:6789/0 conn(0x7f1988103250 legacy=0x7f19881183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198811ccc0 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.350+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198811bd10 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f198cb7c700 10 client.54671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f198811c180 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f198cb7c700 1 librados: init done 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1978001af0 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1978003770 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.351+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1968004f20 con 0x7f19880ff710 2024-02-10T08:52:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.352+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1978003970 con 0x7f19880ff710 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.352+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(837..837 src has 787..837) v4 ==== 7473+0+0 (unknown 3199310970 0 0) 0x7f1978078da0 con 0x7f19880ff710 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.352+0000 7f19777fe700 10 client.54671.objecter ms_dispatch 0x7f1988001550 osd_map(837..837 src has 787..837) v4 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.352+0000 7f19777fe700 3 client.54671.objecter handle_osd_map got epochs [837,837] > 0 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.352+0000 7f19777fe700 3 client.54671.objecter handle_osd_map decoding full epoch 837 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.353+0000 7f19777fe700 20 client.54671.objecter dump_active .. 0 homeless 2024-02-10T08:52:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.353+0000 7f19777fe700 10 client.54671.objecter ms_handle_connect 0x7f1970063ca0 2024-02-10T08:52:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.357+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f197804c440 con 0x7f19880ff710 2024-02-10T08:52:07.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.522+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f19680051d0 con 0x7f19880ff710 2024-02-10T08:52:07.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.523+0000 7f19777fe700 1 -- 172.21.15.107:0/2667789578 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 653121922) 0x7f197804a8b0 con 0x7f19880ff710 2024-02-10T08:52:07.523 INFO:teuthology.orchestra.run.smithi107.stdout:188978561258 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 10 librados: watch_flush enter 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 10 librados: watch_flush exit 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 20 client.54671.objecter shutdown clearing up homeless session... 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 10 client.54671.objecter successfully canceled tick 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1970063ca0 legacy=0x7f1970066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 >> v1:172.21.15.107:6790/0 conn(0x7f19880ff710 legacy=0x7f1988114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 shutdown_connections 2024-02-10T08:52:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 >> 172.21.15.107:0/2667789578 conn(0x7f19880f2510 msgr2=0x7f1988061280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 shutdown_connections 2024-02-10T08:52:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.524+0000 7f198cb7c700 1 -- 172.21.15.107:0/2667789578 wait complete. 2024-02-10T08:52:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:07.525+0000 7f198cb7c700 1 librados: shutdown 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.896328+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:05.897121+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.480451+0000 osd.1 (osd.1) 418 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.483871+0000 osd.1 (osd.1) 419 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.630298+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub starts 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.633414+0000 osd.5 (osd.5) 417 : cluster [DBG] 6.d scrub ok 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.806353+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:52:07.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.809836+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.888948+0000 osd.6 (osd.6) 380 : cluster [DBG] 16.d scrub starts 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.891353+0000 osd.6 (osd.6) 381 : cluster [DBG] 16.d scrub ok 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:07.523609+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? v1:172.21.15.107:0/2667789578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.896328+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:05.897121+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.480451+0000 osd.1 (osd.1) 418 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.483871+0000 osd.1 (osd.1) 419 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.630298+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub starts 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.633414+0000 osd.5 (osd.5) 417 : cluster [DBG] 6.d scrub ok 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.806353+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:52:07.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.809836+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:52:07.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.888948+0000 osd.6 (osd.6) 380 : cluster [DBG] 16.d scrub starts 2024-02-10T08:52:07.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.891353+0000 osd.6 (osd.6) 381 : cluster [DBG] 16.d scrub ok 2024-02-10T08:52:07.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:07 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:07.523609+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? v1:172.21.15.107:0/2667789578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.896328+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.1 scrub starts 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:05.897121+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.1 scrub ok 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.480451+0000 osd.1 (osd.1) 418 : cluster [DBG] 16.9 scrub starts 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.483871+0000 osd.1 (osd.1) 419 : cluster [DBG] 16.9 scrub ok 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.630298+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub starts 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.633414+0000 osd.5 (osd.5) 417 : cluster [DBG] 6.d scrub ok 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.806353+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1b scrub starts 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.809836+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1b scrub ok 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.888948+0000 osd.6 (osd.6) 380 : cluster [DBG] 16.d scrub starts 2024-02-10T08:52:08.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.891353+0000 osd.6 (osd.6) 381 : cluster [DBG] 16.d scrub ok 2024-02-10T08:52:08.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:07.523609+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? v1:172.21.15.107:0/2667789578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T08:52:08.386 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561254 got 188978561258 for osd.6 2024-02-10T08:52:08.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T08:52:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:08.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.939228+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:52:08.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:06.942288+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:52:08.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:07.331385+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:52:08.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:07.332898+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:07.618350+0000 mgr.y (mgr.14553) 754 : cluster [DBG] pgmap v1330: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:07.854228+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:07.856087+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.939228+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:06.942288+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:07.331385+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:07.332898+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:07.618350+0000 mgr.y (mgr.14553) 754 : cluster [DBG] pgmap v1330: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:07.854228+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:52:08.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:08 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:07.856087+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:52:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.939228+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-02-10T08:52:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:06.942288+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-02-10T08:52:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:07.331385+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1f scrub starts 2024-02-10T08:52:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:07.332898+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.1f scrub ok 2024-02-10T08:52:09.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:07.618350+0000 mgr.y (mgr.14553) 754 : cluster [DBG] pgmap v1330: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:07.854228+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.1f scrub starts 2024-02-10T08:52:09.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:07.856087+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.1f scrub ok 2024-02-10T08:52:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:08.304544+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.b scrub starts 2024-02-10T08:52:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:08.306034+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.b scrub ok 2024-02-10T08:52:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:08.304544+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.b scrub starts 2024-02-10T08:52:09.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:09 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:08.306034+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.b scrub ok 2024-02-10T08:52:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:08.304544+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.b scrub starts 2024-02-10T08:52:10.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:08.306034+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.b scrub ok 2024-02-10T08:52:10.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:52:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.544+0000 7fe614b29700 1 -- 172.21.15.107:0/2063478969 >> v1:172.21.15.107:6790/0 conn(0x7fe6100fc910 legacy=0x7fe6100fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.544+0000 7fe614b29700 1 -- 172.21.15.107:0/2063478969 shutdown_connections 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.544+0000 7fe614b29700 1 -- 172.21.15.107:0/2063478969 >> 172.21.15.107:0/2063478969 conn(0x7fe6100f2510 msgr2=0x7fe6100f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.544+0000 7fe614b29700 1 -- 172.21.15.107:0/2063478969 shutdown_connections 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.544+0000 7fe614b29700 1 -- 172.21.15.107:0/2063478969 wait complete. 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 librados: starting msgr at 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 librados: starting objecter 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 Processor -- start 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 -- start start 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 librados: setting wanted keys 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 librados: calling monclient init 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6100fbf80 con 0x7fe610103250 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6100fc150 con 0x7fe6100ff710 2024-02-10T08:52:10.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe614b29700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6100fc340 con 0x7fe6100fc910 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe60e59c700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe6100fc910 0x7fe6100fb740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34684/0 (socket says 172.21.15.107:34684) 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe60e59c700 1 -- 172.21.15.107:0/4085075339 learned_addr learned my addr 172.21.15.107:0/4085075339 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe610103250 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe6100fc910 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe6100ff710 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.545+0000 7fe5ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716701300 0 0) 0x7fe6100fbf80 con 0x7fe610103250 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f8003030 con 0x7fe610103250 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605119257 0 0) 0x7fe61010a610 con 0x7fe6100fc910 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6100fbf80 con 0x7fe6100fc910 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287597887 0 0) 0x7fe6100fc150 con 0x7fe6100ff710 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61010a610 con 0x7fe6100ff710 2024-02-10T08:52:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058914906 0 0) 0x7fe5f8003030 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6100fc150 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe608001b80 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 427300995 0 0) 0x7fe6100fbf80 con 0x7fe6100fc910 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5f8003030 con 0x7fe6100fc910 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe604001f50 con 0x7fe6100fc910 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143139800 0 0) 0x7fe61010a610 con 0x7fe6100ff710 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6100fbf80 con 0x7fe6100ff710 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe6000019f0 con 0x7fe6100ff710 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3001662415 0 0) 0x7fe6100fc150 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 >> v1:172.21.15.107:6790/0 conn(0x7fe6100fc910 legacy=0x7fe6100fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 >> v1:172.21.15.195:6789/0 conn(0x7fe6100ff710 legacy=0x7fe610114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61011cb90 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe610119b60 con 0x7fe610103250 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe614b29700 10 client.29079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:10.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe610119fd0 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe614b29700 1 librados: init done 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7fe608002290 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.546+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7fe608003620 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.547+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0004f20 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.547+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7fe608001df0 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(838..838 src has 787..838) v4 ==== 7473+0+0 (unknown 452076572 0 0) 0x7fe6080766b0 con 0x7fe610103250 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 10 client.29079.objecter ms_dispatch 0x7fe610001550 osd_map(838..838 src has 787..838) v4 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 3 client.29079.objecter handle_osd_map got epochs [838,838] > 0 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 3 client.29079.objecter handle_osd_map decoding full epoch 838 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 20 client.29079.objecter dump_active .. 0 homeless 2024-02-10T08:52:10.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.548+0000 7fe5ff7fe700 10 client.29079.objecter ms_handle_connect 0x7fe5f8063c80 2024-02-10T08:52:10.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.552+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe608049cf0 con 0x7fe610103250 2024-02-10T08:52:10.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.718+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe5f00051d0 con 0x7fe610103250 2024-02-10T08:52:10.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.718+0000 7fe5ff7fe700 1 -- 172.21.15.107:0/4085075339 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 1067850811) 0x7fe608048100 con 0x7fe610103250 2024-02-10T08:52:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:223338299621 2024-02-10T08:52:10.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 10 librados: watch_flush enter 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 10 librados: watch_flush exit 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 20 client.29079.objecter shutdown clearing up homeless session... 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 10 client.29079.objecter successfully canceled tick 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 >> v1:172.21.15.107:6800/2634842009 conn(0x7fe5f8063c80 legacy=0x7fe5f8066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 >> v1:172.21.15.107:6789/0 conn(0x7fe610103250 legacy=0x7fe6101183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 shutdown_connections 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 >> 172.21.15.107:0/4085075339 conn(0x7fe6100f2510 msgr2=0x7fe6101066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 shutdown_connections 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 -- 172.21.15.107:0/4085075339 wait complete. 2024-02-10T08:52:10.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:10.720+0000 7fe614b29700 1 librados: shutdown 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.491314+0000 osd.0 (osd.0) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.491361+0000 osd.0 (osd.0) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.618889+0000 mgr.y (mgr.14553) 755 : cluster [DBG] pgmap v1331: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.702179+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.900077+0000 osd.6 (osd.6) 384 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.903002+0000 osd.6 (osd.6) 385 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.932176+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.8 deep-scrub starts 2024-02-10T08:52:11.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.935651+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.8 deep-scrub ok 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.491314+0000 osd.0 (osd.0) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.491361+0000 osd.0 (osd.0) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.618889+0000 mgr.y (mgr.14553) 755 : cluster [DBG] pgmap v1331: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.702179+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.900077+0000 osd.6 (osd.6) 384 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.903002+0000 osd.6 (osd.6) 385 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.932176+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.8 deep-scrub starts 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.935651+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.8 deep-scrub ok 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.491314+0000 osd.0 (osd.0) 246 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.491361+0000 osd.0 (osd.0) 247 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.618889+0000 mgr.y (mgr.14553) 755 : cluster [DBG] pgmap v1331: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T08:52:11.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.702179+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-10T08:52:11.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.900077+0000 osd.6 (osd.6) 384 : cluster [DBG] 16.12 scrub starts 2024-02-10T08:52:11.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.903002+0000 osd.6 (osd.6) 385 : cluster [DBG] 16.12 scrub ok 2024-02-10T08:52:11.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.932176+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.8 deep-scrub starts 2024-02-10T08:52:11.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.935651+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.8 deep-scrub ok 2024-02-10T08:52:11.537 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299616 got 223338299621 for osd.7 2024-02-10T08:52:11.537 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T08:52:11.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:52:11.808 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.925787+0000 osd.2 (osd.2) 383 : cluster [DBG] 16.e scrub starts 2024-02-10T08:52:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:09.928809+0000 osd.2 (osd.2) 384 : cluster [DBG] 16.e scrub ok 2024-02-10T08:52:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.331784+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:52:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.332918+0000 osd.3 (osd.3) 450 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.527436+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.527719+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.709687+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:10.718951+0000 mon.a (mon.0) 3792 : audit [DBG] from='client.? v1:172.21.15.107:0/4085075339' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.925787+0000 osd.2 (osd.2) 383 : cluster [DBG] 16.e scrub starts 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:09.928809+0000 osd.2 (osd.2) 384 : cluster [DBG] 16.e scrub ok 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.331784+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.332918+0000 osd.3 (osd.3) 450 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.527436+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.527719+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.709687+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-10T08:52:11.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:11 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:10.718951+0000 mon.a (mon.0) 3792 : audit [DBG] from='client.? v1:172.21.15.107:0/4085075339' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:52:11.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.925787+0000 osd.2 (osd.2) 383 : cluster [DBG] 16.e scrub starts 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:09.928809+0000 osd.2 (osd.2) 384 : cluster [DBG] 16.e scrub ok 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.331784+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.13 scrub starts 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.332918+0000 osd.3 (osd.3) 450 : cluster [DBG] 6.13 scrub ok 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.527436+0000 osd.4 (osd.4) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.527719+0000 osd.4 (osd.4) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.709687+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-10T08:52:11.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:10.718951+0000 mon.a (mon.0) 3792 : audit [DBG] from='client.? v1:172.21.15.107:0/4085075339' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T08:52:12.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:52:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:52:13.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.881594+0000 osd.2 (osd.2) 385 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:10.884655+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.423886+0000 osd.0 (osd.0) 248 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.424934+0000 osd.0 (osd.0) 249 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.512253+0000 osd.1 (osd.1) 420 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.515967+0000 osd.1 (osd.1) 421 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.619600+0000 mgr.y (mgr.14553) 756 : cluster [DBG] pgmap v1334: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.787264+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.790173+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.940914+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:11.941964+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:52:13.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:11.963436+0000 mgr.y (mgr.14553) 757 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:13.149 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.881594+0000 osd.2 (osd.2) 385 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:10.884655+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.423886+0000 osd.0 (osd.0) 248 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.424934+0000 osd.0 (osd.0) 249 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.512253+0000 osd.1 (osd.1) 420 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.515967+0000 osd.1 (osd.1) 421 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.619600+0000 mgr.y (mgr.14553) 756 : cluster [DBG] pgmap v1334: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.787264+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.790173+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.940914+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:11.941964+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:52:13.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:11.963436+0000 mgr.y (mgr.14553) 757 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.881594+0000 osd.2 (osd.2) 385 : cluster [DBG] 16.1c scrub starts 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:10.884655+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.1c scrub ok 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.423886+0000 osd.0 (osd.0) 248 : cluster [DBG] 3.1 scrub starts 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.424934+0000 osd.0 (osd.0) 249 : cluster [DBG] 3.1 scrub ok 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.512253+0000 osd.1 (osd.1) 420 : cluster [DBG] 4.12 scrub starts 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.515967+0000 osd.1 (osd.1) 421 : cluster [DBG] 4.12 scrub ok 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.619600+0000 mgr.y (mgr.14553) 756 : cluster [DBG] pgmap v1334: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.787264+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.790173+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.940914+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.15 scrub starts 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:11.941964+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.15 scrub ok 2024-02-10T08:52:13.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:12 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:11.963436+0000 mgr.y (mgr.14553) 757 : audit [DBG] from='client.34451 v1:172.21.15.195:0/3505385625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T08:52:13.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:12.329560+0000 osd.3 (osd.3) 451 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:52:13.719 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:12.330459+0000 osd.3 (osd.3) 452 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:52:13.719 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:12.833820+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:52:13.719 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:12.837008+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:52:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:12.329560+0000 osd.3 (osd.3) 451 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:52:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:12.330459+0000 osd.3 (osd.3) 452 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:52:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:12.833820+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:52:14.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:12.837008+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:52:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:12.329560+0000 osd.3 (osd.3) 451 : cluster [DBG] 2.1e scrub starts 2024-02-10T08:52:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:12.330459+0000 osd.3 (osd.3) 452 : cluster [DBG] 2.1e scrub ok 2024-02-10T08:52:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:12.833820+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-02-10T08:52:14.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:12.837008+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 -- 172.21.15.107:0/2951841465 >> v1:172.21.15.107:6790/0 conn(0x7f39901007f0 legacy=0x7f3990102be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 -- 172.21.15.107:0/2951841465 shutdown_connections 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 -- 172.21.15.107:0/2951841465 >> 172.21.15.107:0/2951841465 conn(0x7f39900f3640 msgr2=0x7f39900f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 -- 172.21.15.107:0/2951841465 shutdown_connections 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 -- 172.21.15.107:0/2951841465 wait complete. 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 librados: starting msgr at 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 librados: starting objecter 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.704+0000 7f3995261700 1 Processor -- start 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 -- start start 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 librados: setting wanted keys 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 librados: calling monclient init 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f39900fd000 con 0x7f39900fda50 2024-02-10T08:52:14.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f39900fd190 con 0x7f39901007f0 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3995261700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f39900fd360 con 0x7f3990104330 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f398effd700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f39900fda50 0x7f39900fc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55176/0 (socket says 172.21.15.107:55176) 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f398effd700 1 -- 172.21.15.107:0/1923939337 learned_addr learned my addr 172.21.15.107:0/1923939337 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f398f7fe700 1 --1- 172.21.15.107:0/1923939337 >> v1:172.21.15.107:6790/0 conn(0x7f3990104330 0x7f39901184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34704/0 (socket says 172.21.15.107:34704) 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter ms_handle_connect 0x7f39900fda50 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter ms_handle_connect 0x7f3990104330 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter ms_handle_connect 0x7f39901007f0 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550322455 0 0) 0x7f39900fd000 con 0x7f39900fda50 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3978003030 con 0x7f39900fda50 2024-02-10T08:52:14.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461984666 0 0) 0x7f39900fd360 con 0x7f3990104330 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900fd000 con 0x7f3990104330 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191722629 0 0) 0x7f39900fd190 con 0x7f39901007f0 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900fd360 con 0x7f39901007f0 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 937819508 0 0) 0x7f3978003030 con 0x7f39900fda50 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39900fd190 con 0x7f39900fda50 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132869741 0 0) 0x7f39900fd000 con 0x7f3990104330 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3978003030 con 0x7f3990104330 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f39840019f0 con 0x7f39900fda50 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3988001b00 con 0x7f3990104330 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625806037 0 0) 0x7f39900fd360 con 0x7f39901007f0 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39900fd000 con 0x7f39901007f0 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3980001070 con 0x7f39901007f0 2024-02-10T08:52:14.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046506645 0 0) 0x7f39900fd190 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.705+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 >> v1:172.21.15.107:6790/0 conn(0x7f3990104330 legacy=0x7f39901184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 >> v1:172.21.15.195:6789/0 conn(0x7f39901007f0 legacy=0x7f3990114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399011cc00 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3990119c30 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3995261700 10 client.29085.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f399011a120 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3995261700 1 librados: init done 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f3984001be0 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.706+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f3984005a90 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c004f20 con 0x7f39900fda50 2024-02-10T08:52:14.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f3984004fc0 con 0x7f39900fda50 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(839..839 src has 787..839) v4 ==== 7473+0+0 (unknown 2033404990 0 0) 0x7f3984078af0 con 0x7f39900fda50 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3977fff700 10 client.29085.objecter ms_dispatch 0x7f3990001550 osd_map(839..839 src has 787..839) v4 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3977fff700 3 client.29085.objecter handle_osd_map got epochs [839,839] > 0 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.707+0000 7f3977fff700 3 client.29085.objecter handle_osd_map decoding full epoch 839 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.708+0000 7f3977fff700 20 client.29085.objecter dump_active .. 0 homeless 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.708+0000 7f3977fff700 10 client.29085.objecter ms_handle_connect 0x7f3978063c30 2024-02-10T08:52:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.714+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f398404c190 con 0x7f39900fda50 2024-02-10T08:52:14.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.878+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f397c000b90 con 0x7f3978063c30 2024-02-10T08:52:14.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:13.889+0000 7f3977fff700 1 -- 172.21.15.107:0/1923939337 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+410330 (unknown 2965378022 0 176233021) 0x7f397c000b90 con 0x7f3978063c30 2024-02-10T08:52:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 10 librados: watch_flush enter 2024-02-10T08:52:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 10 librados: watch_flush exit 2024-02-10T08:52:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 20 client.29085.objecter shutdown clearing up homeless session... 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 10 client.29085.objecter successfully canceled tick 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 >> v1:172.21.15.107:6800/2634842009 conn(0x7f3978063c30 legacy=0x7f39780660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 >> v1:172.21.15.107:6789/0 conn(0x7f39900fda50 legacy=0x7f39900fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 shutdown_connections 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 >> 172.21.15.107:0/1923939337 conn(0x7f39900f3640 msgr2=0x7f39901077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 shutdown_connections 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 -- 172.21.15.107:0/1923939337 wait complete. 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:14.183+0000 7f3995261700 1 librados: shutdown 2024-02-10T08:52:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:52:14.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:13.305815+0000 osd.3 (osd.3) 453 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:52:14.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:13.307921+0000 osd.3 (osd.3) 454 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:52:14.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:13.620076+0000 mgr.y (mgr.14553) 758 : cluster [DBG] pgmap v1335: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:13.860270+0000 osd.2 (osd.2) 389 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:13.861845+0000 osd.2 (osd.2) 390 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:13.879044+0000 mgr.y (mgr.14553) 759 : audit [DBG] from='client.29085 v1:172.21.15.107:0/1923939337' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:13.305815+0000 osd.3 (osd.3) 453 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:13.307921+0000 osd.3 (osd.3) 454 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:13.620076+0000 mgr.y (mgr.14553) 758 : cluster [DBG] pgmap v1335: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:13.860270+0000 osd.2 (osd.2) 389 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:13.861845+0000 osd.2 (osd.2) 390 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:52:14.925 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:14 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:13.879044+0000 mgr.y (mgr.14553) 759 : audit [DBG] from='client.29085 v1:172.21.15.107:0/1923939337' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:14.945 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1335,"stamp":"2024-02-10T08:52:13.619760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1487,"num_read_kb":1239,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":13772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":462,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2436092,"kb_used_data":14140,"kb_used_omap":59,"kb_used_meta":2420676,"kb_avail":747525124,"statfs":{"total":767960285184,"available":765465726976,"internally_reserved":0,"allocated":14479360,"data_stored":4437191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2478772843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002469"},"pg_stats":[{"pgid":"5.9","version":"829'19","reported_seq":572,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934013+0000","last_change":"2024-02-10T08:51:07.741210+0000","last_active":"2024-02-10T08:52:09.934013+0000","last_peered":"2024-02-10T08:52:09.934013+0000","last_clean":"2024-02-10T08:52:09.934013+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:52:09.934013+0000","last_undegraded":"2024-02-10T08:52:09.934013+0000","last_fullsized":"2024-02-10T08:52:09.934013+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'19","last_scrub_stamp":"2024-02-10T08:51:07.741135+0000","last_deep_scrub":"829'19","last_deep_scrub_stamp":"2024-02-10T08:51:07.741135+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.741135+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":914,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332554+0000","last_change":"2024-02-10T08:51:22.676912+0000","last_active":"2024-02-10T08:52:10.332554+0000","last_peered":"2024-02-10T08:52:10.332554+0000","last_clean":"2024-02-10T08:52:10.332554+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:52:10.332554+0000","last_undegraded":"2024-02-10T08:52:10.332554+0000","last_fullsized":"2024-02-10T08:52:10.332554+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:51:22.676835+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.676835+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":848,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703834+0000","last_change":"2024-02-10T08:51:17.756145+0000","last_active":"2024-02-10T08:52:09.703834+0000","last_peered":"2024-02-10T08:52:09.703834+0000","last_clean":"2024-02-10T08:52:09.703834+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:52:09.703834+0000","last_undegraded":"2024-02-10T08:52:09.703834+0000","last_fullsized":"2024-02-10T08:52:09.703834+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:17.756068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.756068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":833,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332946+0000","last_change":"2024-02-10T08:51:17.664135+0000","last_active":"2024-02-10T08:52:10.332946+0000","last_peered":"2024-02-10T08:52:10.332946+0000","last_clean":"2024-02-10T08:52:10.332946+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:52:10.332946+0000","last_undegraded":"2024-02-10T08:52:10.332946+0000","last_fullsized":"2024-02-10T08:52:10.332946+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:17.664025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:17.664025+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.664025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":794,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.884774+0000","last_change":"2024-02-10T08:52:10.884774+0000","last_active":"2024-02-10T08:52:10.884774+0000","last_peered":"2024-02-10T08:52:10.884774+0000","last_clean":"2024-02-10T08:52:10.884774+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:52:10.884774+0000","last_undegraded":"2024-02-10T08:52:10.884774+0000","last_fullsized":"2024-02-10T08:52:10.884774+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:10.884679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:10.884679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":296,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925313+0000","last_change":"2024-02-10T08:51:54.282145+0000","last_active":"2024-02-10T08:52:09.925313+0000","last_peered":"2024-02-10T08:52:09.925313+0000","last_clean":"2024-02-10T08:52:09.925313+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:52:09.925313+0000","last_undegraded":"2024-02-10T08:52:09.925313+0000","last_fullsized":"2024-02-10T08:52:09.925313+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:51:54.282038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.282038+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701359+0000","last_change":"2024-02-10T08:51:34.470562+0000","last_active":"2024-02-10T08:52:09.701359+0000","last_peered":"2024-02-10T08:52:09.701359+0000","last_clean":"2024-02-10T08:52:09.701359+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:52:09.701359+0000","last_undegraded":"2024-02-10T08:52:09.701359+0000","last_fullsized":"2024-02-10T08:52:09.701359+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:34.470498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.470498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":859,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703411+0000","last_change":"2024-02-10T08:51:47.010221+0000","last_active":"2024-02-10T08:52:09.703411+0000","last_peered":"2024-02-10T08:52:09.703411+0000","last_clean":"2024-02-10T08:52:09.703411+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:52:09.703411+0000","last_undegraded":"2024-02-10T08:52:09.703411+0000","last_fullsized":"2024-02-10T08:52:09.703411+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.010152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.010152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":232,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925236+0000","last_change":"2024-02-10T08:51:03.535754+0000","last_active":"2024-02-10T08:52:09.925236+0000","last_peered":"2024-02-10T08:52:09.925236+0000","last_clean":"2024-02-10T08:52:09.925236+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:52:09.925236+0000","last_undegraded":"2024-02-10T08:52:09.925236+0000","last_fullsized":"2024-02-10T08:52:09.925236+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:03.535646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:03.535646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":782,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333733+0000","last_change":"2024-02-10T08:51:31.775340+0000","last_active":"2024-02-10T08:52:10.333733+0000","last_peered":"2024-02-10T08:52:10.333733+0000","last_clean":"2024-02-10T08:52:10.333733+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:52:10.333733+0000","last_undegraded":"2024-02-10T08:52:10.333733+0000","last_fullsized":"2024-02-10T08:52:10.333733+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:31.775249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:31.775249+0000","last_clean_scrub_stamp":"2024-02-10T08:51:31.775249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":1008,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335384+0000","last_change":"2024-02-10T08:51:16.033467+0000","last_active":"2024-02-10T08:52:10.335384+0000","last_peered":"2024-02-10T08:52:10.335384+0000","last_clean":"2024-02-10T08:52:10.335384+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:52:10.335384+0000","last_undegraded":"2024-02-10T08:52:10.335384+0000","last_fullsized":"2024-02-10T08:52:10.335384+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:51:16.033361+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:51:16.033361+0000","last_clean_scrub_stamp":"2024-02-10T08:51:16.033361+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":846,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703856+0000","last_change":"2024-02-10T08:51:16.790256+0000","last_active":"2024-02-10T08:52:09.703856+0000","last_peered":"2024-02-10T08:52:09.703856+0000","last_clean":"2024-02-10T08:52:09.703856+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:52:09.703856+0000","last_undegraded":"2024-02-10T08:52:09.703856+0000","last_fullsized":"2024-02-10T08:52:09.703856+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:16.790188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:51:16.790188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":885,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703622+0000","last_change":"2024-02-10T08:51:59.921088+0000","last_active":"2024-02-10T08:52:09.703622+0000","last_peered":"2024-02-10T08:52:09.703622+0000","last_clean":"2024-02-10T08:52:09.703622+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:52:09.703622+0000","last_undegraded":"2024-02-10T08:52:09.703622+0000","last_fullsized":"2024-02-10T08:52:09.703622+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.921000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:59.921000+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.921000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":400,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933622+0000","last_change":"2024-02-10T08:51:28.921089+0000","last_active":"2024-02-10T08:52:09.933622+0000","last_peered":"2024-02-10T08:52:09.933622+0000","last_clean":"2024-02-10T08:52:09.933622+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:52:09.933622+0000","last_undegraded":"2024-02-10T08:52:09.933622+0000","last_fullsized":"2024-02-10T08:52:09.933622+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:28.921014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:51:28.921014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934383+0000","last_change":"2024-02-10T08:51:05.742473+0000","last_active":"2024-02-10T08:52:09.934383+0000","last_peered":"2024-02-10T08:52:09.934383+0000","last_clean":"2024-02-10T08:52:09.934383+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:52:09.934383+0000","last_undegraded":"2024-02-10T08:52:09.934383+0000","last_fullsized":"2024-02-10T08:52:09.934383+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:05.742394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:05.742394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":782,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701816+0000","last_change":"2024-02-10T08:51:11.895780+0000","last_active":"2024-02-10T08:52:09.701816+0000","last_peered":"2024-02-10T08:52:09.701816+0000","last_clean":"2024-02-10T08:52:09.701816+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:52:09.701816+0000","last_undegraded":"2024-02-10T08:52:09.701816+0000","last_fullsized":"2024-02-10T08:52:09.701816+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:51:11.895697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:11.895697+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333050+0000","last_change":"2024-02-10T08:51:33.186070+0000","last_active":"2024-02-10T08:52:10.333050+0000","last_peered":"2024-02-10T08:52:10.333050+0000","last_clean":"2024-02-10T08:52:10.333050+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:52:10.333050+0000","last_undegraded":"2024-02-10T08:52:10.333050+0000","last_fullsized":"2024-02-10T08:52:10.333050+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:33.185993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:51:33.185993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701596+0000","last_change":"2024-02-10T08:51:35.494575+0000","last_active":"2024-02-10T08:52:09.701596+0000","last_peered":"2024-02-10T08:52:09.701596+0000","last_clean":"2024-02-10T08:52:09.701596+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:52:09.701596+0000","last_undegraded":"2024-02-10T08:52:09.701596+0000","last_fullsized":"2024-02-10T08:52:09.701596+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.494511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.494511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":837,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.704070+0000","last_change":"2024-02-10T08:51:49.996088+0000","last_active":"2024-02-10T08:52:09.704070+0000","last_peered":"2024-02-10T08:52:09.704070+0000","last_clean":"2024-02-10T08:52:09.704070+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:52:09.704070+0000","last_undegraded":"2024-02-10T08:52:09.704070+0000","last_fullsized":"2024-02-10T08:52:09.704070+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.995985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.995985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":642,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701306+0000","last_change":"2024-02-10T08:52:05.835916+0000","last_active":"2024-02-10T08:52:09.701306+0000","last_peered":"2024-02-10T08:52:09.701306+0000","last_clean":"2024-02-10T08:52:09.701306+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:52:09.701306+0000","last_undegraded":"2024-02-10T08:52:09.701306+0000","last_fullsized":"2024-02-10T08:52:09.701306+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.835835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:05.835835+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.835835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":706,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701512+0000","last_change":"2024-02-10T08:51:45.873280+0000","last_active":"2024-02-10T08:52:09.701512+0000","last_peered":"2024-02-10T08:52:09.701512+0000","last_clean":"2024-02-10T08:52:09.701512+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:52:09.701512+0000","last_undegraded":"2024-02-10T08:52:09.701512+0000","last_fullsized":"2024-02-10T08:52:09.701512+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:45.873199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:45.873199+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701866+0000","last_change":"2024-02-10T08:51:46.491852+0000","last_active":"2024-02-10T08:52:09.701866+0000","last_peered":"2024-02-10T08:52:09.701866+0000","last_clean":"2024-02-10T08:52:09.701866+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:52:09.701866+0000","last_undegraded":"2024-02-10T08:52:09.701866+0000","last_fullsized":"2024-02-10T08:52:09.701866+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:46.491782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:46.491782+0000","last_clean_scrub_stamp":"2024-02-10T08:51:46.491782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1138,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703741+0000","last_change":"2024-02-10T08:52:02.900985+0000","last_active":"2024-02-10T08:52:09.703741+0000","last_peered":"2024-02-10T08:52:09.703741+0000","last_clean":"2024-02-10T08:52:09.703741+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:52:09.703741+0000","last_undegraded":"2024-02-10T08:52:09.703741+0000","last_fullsized":"2024-02-10T08:52:09.703741+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:52:02.900920+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:50:46.833593+0000","last_clean_scrub_stamp":"2024-02-10T08:52:02.900920+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":666,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701664+0000","last_change":"2024-02-10T08:51:37.771083+0000","last_active":"2024-02-10T08:52:09.701664+0000","last_peered":"2024-02-10T08:52:09.701664+0000","last_clean":"2024-02-10T08:52:09.701664+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:52:09.701664+0000","last_undegraded":"2024-02-10T08:52:09.701664+0000","last_fullsized":"2024-02-10T08:52:09.701664+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:37.770953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:37.770953+0000","last_clean_scrub_stamp":"2024-02-10T08:51:37.770953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":235,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925597+0000","last_change":"2024-02-10T08:50:39.656525+0000","last_active":"2024-02-10T08:52:09.925597+0000","last_peered":"2024-02-10T08:52:09.925597+0000","last_clean":"2024-02-10T08:52:09.925597+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:52:09.925597+0000","last_undegraded":"2024-02-10T08:52:09.925597+0000","last_fullsized":"2024-02-10T08:52:09.925597+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:39.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:50:39.656444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"833'34","reported_seq":57,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703895+0000","last_change":"2024-02-10T08:51:27.814091+0000","last_active":"2024-02-10T08:52:09.703895+0000","last_peered":"2024-02-10T08:52:09.703895+0000","last_clean":"2024-02-10T08:52:09.703895+0000","last_became_active":"2024-02-10T08:50:42.864182+0000","last_became_peered":"2024-02-10T08:50:42.864182+0000","last_unstale":"2024-02-10T08:52:09.703895+0000","last_undegraded":"2024-02-10T08:52:09.703895+0000","last_fullsized":"2024-02-10T08:52:09.703895+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'34","last_scrub_stamp":"2024-02-10T08:51:27.814022+0000","last_deep_scrub":"833'34","last_deep_scrub_stamp":"2024-02-10T08:51:27.814022+0000","last_clean_scrub_stamp":"2024-02-10T08:51:27.814022+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":159,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701189+0000","last_change":"2024-02-10T08:51:43.530660+0000","last_active":"2024-02-10T08:52:09.701189+0000","last_peered":"2024-02-10T08:52:09.701189+0000","last_clean":"2024-02-10T08:52:09.701189+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:52:09.701189+0000","last_undegraded":"2024-02-10T08:52:09.701189+0000","last_fullsized":"2024-02-10T08:52:09.701189+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:51:43.530546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:43.530546+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":943,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.834258+0000","last_change":"2024-02-10T08:50:51.901411+0000","last_active":"2024-02-10T08:52:10.834258+0000","last_peered":"2024-02-10T08:52:10.834258+0000","last_clean":"2024-02-10T08:52:10.834258+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:52:10.834258+0000","last_undegraded":"2024-02-10T08:52:10.834258+0000","last_fullsized":"2024-02-10T08:52:10.834258+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:50:51.901321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:50:51.901321+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":851,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333442+0000","last_change":"2024-02-10T08:52:06.633522+0000","last_active":"2024-02-10T08:52:10.333442+0000","last_peered":"2024-02-10T08:52:10.333442+0000","last_clean":"2024-02-10T08:52:10.333442+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:52:10.333442+0000","last_undegraded":"2024-02-10T08:52:10.333442+0000","last_fullsized":"2024-02-10T08:52:10.333442+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.633419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.633419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":800,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333064+0000","last_change":"2024-02-10T08:52:03.545779+0000","last_active":"2024-02-10T08:52:10.333064+0000","last_peered":"2024-02-10T08:52:10.333064+0000","last_clean":"2024-02-10T08:52:10.333064+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:52:10.333064+0000","last_undegraded":"2024-02-10T08:52:10.333064+0000","last_fullsized":"2024-02-10T08:52:10.333064+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:03.545690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:03.545690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":744,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701863+0000","last_change":"2024-02-10T08:50:39.075746+0000","last_active":"2024-02-10T08:52:09.701863+0000","last_peered":"2024-02-10T08:52:09.701863+0000","last_clean":"2024-02-10T08:52:09.701863+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:52:09.701863+0000","last_undegraded":"2024-02-10T08:52:09.701863+0000","last_fullsized":"2024-02-10T08:52:09.701863+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:39.075652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:39.075652+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":433,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.935780+0000","last_change":"2024-02-10T08:52:09.935732+0000","last_active":"2024-02-10T08:52:09.935780+0000","last_peered":"2024-02-10T08:52:09.935780+0000","last_clean":"2024-02-10T08:52:09.935780+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:52:09.935780+0000","last_undegraded":"2024-02-10T08:52:09.935780+0000","last_fullsized":"2024-02-10T08:52:09.935780+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.935655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:09.935655+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.935655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":1036,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333616+0000","last_change":"2024-02-10T08:51:44.706204+0000","last_active":"2024-02-10T08:52:10.333616+0000","last_peered":"2024-02-10T08:52:10.333616+0000","last_clean":"2024-02-10T08:52:10.333616+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:52:10.333616+0000","last_undegraded":"2024-02-10T08:52:10.333616+0000","last_fullsized":"2024-02-10T08:52:10.333616+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:51:44.706124+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.706124+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":818,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333369+0000","last_change":"2024-02-10T08:51:48.279152+0000","last_active":"2024-02-10T08:52:10.333369+0000","last_peered":"2024-02-10T08:52:10.333369+0000","last_clean":"2024-02-10T08:52:10.333369+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:52:10.333369+0000","last_undegraded":"2024-02-10T08:52:10.333369+0000","last_fullsized":"2024-02-10T08:52:10.333369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:48.279085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.279085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":762,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332994+0000","last_change":"2024-02-10T08:51:12.243000+0000","last_active":"2024-02-10T08:52:10.332994+0000","last_peered":"2024-02-10T08:52:10.332994+0000","last_clean":"2024-02-10T08:52:10.332994+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:52:10.332994+0000","last_undegraded":"2024-02-10T08:52:10.332994+0000","last_fullsized":"2024-02-10T08:52:10.332994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:12.242874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:12.242874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1098,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332819+0000","last_change":"2024-02-10T08:51:09.218426+0000","last_active":"2024-02-10T08:52:10.332819+0000","last_peered":"2024-02-10T08:52:10.332819+0000","last_clean":"2024-02-10T08:52:10.332819+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:52:10.332819+0000","last_undegraded":"2024-02-10T08:52:10.332819+0000","last_fullsized":"2024-02-10T08:52:10.332819+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'14","last_scrub_stamp":"2024-02-10T08:51:09.218347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:09.218347+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":720,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701095+0000","last_change":"2024-02-10T08:51:00.932182+0000","last_active":"2024-02-10T08:52:09.701095+0000","last_peered":"2024-02-10T08:52:09.701095+0000","last_clean":"2024-02-10T08:52:09.701095+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:52:09.701095+0000","last_undegraded":"2024-02-10T08:52:09.701095+0000","last_fullsized":"2024-02-10T08:52:09.701095+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:51:00.932102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:00.932102+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":857,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335897+0000","last_change":"2024-02-10T08:51:39.012269+0000","last_active":"2024-02-10T08:52:10.335897+0000","last_peered":"2024-02-10T08:52:10.335897+0000","last_clean":"2024-02-10T08:52:10.335897+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:52:10.335897+0000","last_undegraded":"2024-02-10T08:52:10.335897+0000","last_fullsized":"2024-02-10T08:52:10.335897+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:39.012192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:51:39.012192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933910+0000","last_change":"2024-02-10T08:52:04.997651+0000","last_active":"2024-02-10T08:52:09.933910+0000","last_peered":"2024-02-10T08:52:09.933910+0000","last_clean":"2024-02-10T08:52:09.933910+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:52:09.933910+0000","last_undegraded":"2024-02-10T08:52:09.933910+0000","last_fullsized":"2024-02-10T08:52:09.933910+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.997567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.997567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":255,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925405+0000","last_change":"2024-02-10T08:51:41.282955+0000","last_active":"2024-02-10T08:52:09.925405+0000","last_peered":"2024-02-10T08:52:09.925405+0000","last_clean":"2024-02-10T08:52:09.925405+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:52:09.925405+0000","last_undegraded":"2024-02-10T08:52:09.925405+0000","last_fullsized":"2024-02-10T08:52:09.925405+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:41.282896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:41.282896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":685,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.700982+0000","last_change":"2024-02-10T08:51:47.942559+0000","last_active":"2024-02-10T08:52:09.700982+0000","last_peered":"2024-02-10T08:52:09.700982+0000","last_clean":"2024-02-10T08:52:09.700982+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:52:09.700982+0000","last_undegraded":"2024-02-10T08:52:09.700982+0000","last_fullsized":"2024-02-10T08:52:09.700982+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.942478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.942478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":1009,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332804+0000","last_change":"2024-02-10T08:51:51.296797+0000","last_active":"2024-02-10T08:52:10.332804+0000","last_peered":"2024-02-10T08:52:10.332804+0000","last_clean":"2024-02-10T08:52:10.332804+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:52:10.332804+0000","last_undegraded":"2024-02-10T08:52:10.332804+0000","last_fullsized":"2024-02-10T08:52:10.332804+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:51:51.296693+0000","last_deep_scrub":"473'38","last_deep_scrub_stamp":"2024-02-10T08:51:51.296693+0000","last_clean_scrub_stamp":"2024-02-10T08:51:51.296693+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":159,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.700915+0000","last_change":"2024-02-10T08:51:15.503458+0000","last_active":"2024-02-10T08:52:09.700915+0000","last_peered":"2024-02-10T08:52:09.700915+0000","last_clean":"2024-02-10T08:52:09.700915+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:52:09.700915+0000","last_undegraded":"2024-02-10T08:52:09.700915+0000","last_fullsized":"2024-02-10T08:52:09.700915+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:51:15.503393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.503393+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":653,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701104+0000","last_change":"2024-02-10T08:52:03.836543+0000","last_active":"2024-02-10T08:52:09.701104+0000","last_peered":"2024-02-10T08:52:09.701104+0000","last_clean":"2024-02-10T08:52:09.701104+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:52:09.701104+0000","last_undegraded":"2024-02-10T08:52:09.701104+0000","last_fullsized":"2024-02-10T08:52:09.701104+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:03.836454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:03.836454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":255,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925414+0000","last_change":"2024-02-10T08:52:01.331511+0000","last_active":"2024-02-10T08:52:09.925414+0000","last_peered":"2024-02-10T08:52:09.925414+0000","last_clean":"2024-02-10T08:52:09.925414+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:52:09.925414+0000","last_undegraded":"2024-02-10T08:52:09.925414+0000","last_fullsized":"2024-02-10T08:52:09.925414+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:01.331444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:01.331444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":844,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335414+0000","last_change":"2024-02-10T08:51:40.063896+0000","last_active":"2024-02-10T08:52:10.335414+0000","last_peered":"2024-02-10T08:52:10.335414+0000","last_clean":"2024-02-10T08:52:10.335414+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:52:10.335414+0000","last_undegraded":"2024-02-10T08:52:10.335414+0000","last_fullsized":"2024-02-10T08:52:10.335414+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.063814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.063814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925539+0000","last_change":"2024-02-10T08:51:08.569596+0000","last_active":"2024-02-10T08:52:09.925539+0000","last_peered":"2024-02-10T08:52:09.925539+0000","last_clean":"2024-02-10T08:52:09.925539+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:52:09.925539+0000","last_undegraded":"2024-02-10T08:52:09.925539+0000","last_fullsized":"2024-02-10T08:52:09.925539+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:51:08.569484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.569484+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":430,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934263+0000","last_change":"2024-02-10T08:51:49.952004+0000","last_active":"2024-02-10T08:52:09.934263+0000","last_peered":"2024-02-10T08:52:09.934263+0000","last_clean":"2024-02-10T08:52:09.934263+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:52:09.934263+0000","last_undegraded":"2024-02-10T08:52:09.934263+0000","last_fullsized":"2024-02-10T08:52:09.934263+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.951887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.951887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":730,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703506+0000","last_change":"2024-02-10T08:52:05.897186+0000","last_active":"2024-02-10T08:52:09.703506+0000","last_peered":"2024-02-10T08:52:09.703506+0000","last_clean":"2024-02-10T08:52:09.703506+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:52:09.703506+0000","last_undegraded":"2024-02-10T08:52:09.703506+0000","last_fullsized":"2024-02-10T08:52:09.703506+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.897125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:41.793022+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.897125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703970+0000","last_change":"2024-02-10T08:51:45.971164+0000","last_active":"2024-02-10T08:52:09.703970+0000","last_peered":"2024-02-10T08:52:09.703970+0000","last_clean":"2024-02-10T08:52:09.703970+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:52:09.703970+0000","last_undegraded":"2024-02-10T08:52:09.703970+0000","last_fullsized":"2024-02-10T08:52:09.703970+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:45.971088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:45.971088+0000","last_clean_scrub_stamp":"2024-02-10T08:51:45.971088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":877,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333592+0000","last_change":"2024-02-10T08:51:48.668695+0000","last_active":"2024-02-10T08:52:10.333592+0000","last_peered":"2024-02-10T08:52:10.333592+0000","last_clean":"2024-02-10T08:52:10.333592+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:52:10.333592+0000","last_undegraded":"2024-02-10T08:52:10.333592+0000","last_fullsized":"2024-02-10T08:52:10.333592+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:48.668616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:32.456728+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.668616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":292,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925692+0000","last_change":"2024-02-10T08:51:07.597408+0000","last_active":"2024-02-10T08:52:09.925692+0000","last_peered":"2024-02-10T08:52:09.925692+0000","last_clean":"2024-02-10T08:52:09.925692+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:52:09.925692+0000","last_undegraded":"2024-02-10T08:52:09.925692+0000","last_fullsized":"2024-02-10T08:52:09.925692+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:51:07.597344+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.597344+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":432,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933374+0000","last_change":"2024-02-10T08:52:00.889889+0000","last_active":"2024-02-10T08:52:09.933374+0000","last_peered":"2024-02-10T08:52:09.933374+0000","last_clean":"2024-02-10T08:52:09.933374+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:52:09.933374+0000","last_undegraded":"2024-02-10T08:52:09.933374+0000","last_fullsized":"2024-02-10T08:52:09.933374+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:00.889812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:50.991799+0000","last_clean_scrub_stamp":"2024-02-10T08:52:00.889812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":243,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925149+0000","last_change":"2024-02-10T08:50:33.606684+0000","last_active":"2024-02-10T08:52:09.925149+0000","last_peered":"2024-02-10T08:52:09.925149+0000","last_clean":"2024-02-10T08:52:09.925149+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:52:09.925149+0000","last_undegraded":"2024-02-10T08:52:09.925149+0000","last_fullsized":"2024-02-10T08:52:09.925149+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:33.606614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:50:33.606614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":160,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701707+0000","last_change":"2024-02-10T08:51:08.500925+0000","last_active":"2024-02-10T08:52:09.701707+0000","last_peered":"2024-02-10T08:52:09.701707+0000","last_clean":"2024-02-10T08:52:09.701707+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:52:09.701707+0000","last_undegraded":"2024-02-10T08:52:09.701707+0000","last_fullsized":"2024-02-10T08:52:09.701707+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:51:08.500803+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:51:08.500803+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.500803+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":791,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335231+0000","last_change":"2024-02-10T08:52:09.903082+0000","last_active":"2024-02-10T08:52:10.335231+0000","last_peered":"2024-02-10T08:52:10.335231+0000","last_clean":"2024-02-10T08:52:10.335231+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:52:10.335231+0000","last_undegraded":"2024-02-10T08:52:10.335231+0000","last_fullsized":"2024-02-10T08:52:10.335231+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.903006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.903006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934123+0000","last_change":"2024-02-10T08:51:29.900861+0000","last_active":"2024-02-10T08:52:09.934123+0000","last_peered":"2024-02-10T08:52:09.934123+0000","last_clean":"2024-02-10T08:52:09.934123+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:52:09.934123+0000","last_undegraded":"2024-02-10T08:52:09.934123+0000","last_fullsized":"2024-02-10T08:52:09.934123+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:29.900777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:06.926578+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.900777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"832'33","reported_seq":930,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335733+0000","last_change":"2024-02-10T08:51:20.998887+0000","last_active":"2024-02-10T08:52:10.335733+0000","last_peered":"2024-02-10T08:52:10.335733+0000","last_clean":"2024-02-10T08:52:10.335733+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:52:10.335733+0000","last_undegraded":"2024-02-10T08:52:10.335733+0000","last_fullsized":"2024-02-10T08:52:10.335733+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'33","last_scrub_stamp":"2024-02-10T08:51:20.998801+0000","last_deep_scrub":"832'33","last_deep_scrub_stamp":"2024-02-10T08:51:20.998801+0000","last_clean_scrub_stamp":"2024-02-10T08:51:20.998801+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":411,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933654+0000","last_change":"2024-02-10T08:51:14.839753+0000","last_active":"2024-02-10T08:52:09.933654+0000","last_peered":"2024-02-10T08:52:09.933654+0000","last_clean":"2024-02-10T08:52:09.933654+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:52:09.933654+0000","last_undegraded":"2024-02-10T08:52:09.933654+0000","last_fullsized":"2024-02-10T08:52:09.933654+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:14.839681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:51:14.839681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":839,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332871+0000","last_change":"2024-02-10T08:52:05.591433+0000","last_active":"2024-02-10T08:52:10.332871+0000","last_peered":"2024-02-10T08:52:10.332871+0000","last_clean":"2024-02-10T08:52:10.332871+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:52:10.332871+0000","last_undegraded":"2024-02-10T08:52:10.332871+0000","last_fullsized":"2024-02-10T08:52:10.332871+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.591323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":818,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333184+0000","last_change":"2024-02-10T08:51:49.282695+0000","last_active":"2024-02-10T08:52:10.333184+0000","last_peered":"2024-02-10T08:52:10.333184+0000","last_clean":"2024-02-10T08:52:10.333184+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:52:10.333184+0000","last_undegraded":"2024-02-10T08:52:10.333184+0000","last_fullsized":"2024-02-10T08:52:10.333184+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.282616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.282616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":825,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333659+0000","last_change":"2024-02-10T08:51:46.698978+0000","last_active":"2024-02-10T08:52:10.333659+0000","last_peered":"2024-02-10T08:52:10.333659+0000","last_clean":"2024-02-10T08:52:10.333659+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:52:10.333659+0000","last_undegraded":"2024-02-10T08:52:10.333659+0000","last_fullsized":"2024-02-10T08:52:10.333659+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:46.698902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:46.698902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":799,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332592+0000","last_change":"2024-02-10T08:51:15.732610+0000","last_active":"2024-02-10T08:52:10.332592+0000","last_peered":"2024-02-10T08:52:10.332592+0000","last_clean":"2024-02-10T08:52:10.332592+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:52:10.332592+0000","last_undegraded":"2024-02-10T08:52:10.332592+0000","last_fullsized":"2024-02-10T08:52:10.332592+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:15.732518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.732518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":860,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703925+0000","last_change":"2024-02-10T08:50:48.818974+0000","last_active":"2024-02-10T08:52:09.703925+0000","last_peered":"2024-02-10T08:52:09.703925+0000","last_clean":"2024-02-10T08:52:09.703925+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:52:09.703925+0000","last_undegraded":"2024-02-10T08:52:09.703925+0000","last_fullsized":"2024-02-10T08:52:09.703925+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:48.818866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:48.818866+0000","last_clean_scrub_stamp":"2024-02-10T08:50:48.818866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":162,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701752+0000","last_change":"2024-02-10T08:51:14.507631+0000","last_active":"2024-02-10T08:52:09.701752+0000","last_peered":"2024-02-10T08:52:09.701752+0000","last_clean":"2024-02-10T08:52:09.701752+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:52:09.701752+0000","last_undegraded":"2024-02-10T08:52:09.701752+0000","last_fullsized":"2024-02-10T08:52:09.701752+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:51:14.507563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:14.507563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":848,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703463+0000","last_change":"2024-02-10T08:50:59.777513+0000","last_active":"2024-02-10T08:52:09.703463+0000","last_peered":"2024-02-10T08:52:09.703463+0000","last_clean":"2024-02-10T08:52:09.703463+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:52:09.703463+0000","last_undegraded":"2024-02-10T08:52:09.703463+0000","last_fullsized":"2024-02-10T08:52:09.703463+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:59.777445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:59.777445+0000","last_clean_scrub_stamp":"2024-02-10T08:50:59.777445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":416,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.690202+0000","last_change":"2024-02-10T08:51:17.586120+0000","last_active":"2024-02-10T08:52:10.690202+0000","last_peered":"2024-02-10T08:52:10.690202+0000","last_clean":"2024-02-10T08:52:10.690202+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:52:10.690202+0000","last_undegraded":"2024-02-10T08:52:10.690202+0000","last_fullsized":"2024-02-10T08:52:10.690202+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:51:17.586026+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.586026+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":399,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934237+0000","last_change":"2024-02-10T08:51:43.835976+0000","last_active":"2024-02-10T08:52:09.934237+0000","last_peered":"2024-02-10T08:52:09.934237+0000","last_clean":"2024-02-10T08:52:09.934237+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:52:09.934237+0000","last_undegraded":"2024-02-10T08:52:09.934237+0000","last_fullsized":"2024-02-10T08:52:09.934237+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:43.835902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:51:43.835902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701059+0000","last_change":"2024-02-10T08:50:43.402411+0000","last_active":"2024-02-10T08:52:09.701059+0000","last_peered":"2024-02-10T08:52:09.701059+0000","last_clean":"2024-02-10T08:52:09.701059+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:52:09.701059+0000","last_undegraded":"2024-02-10T08:52:09.701059+0000","last_fullsized":"2024-02-10T08:52:09.701059+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:43.402320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:50:43.402320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":266,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925589+0000","last_change":"2024-02-10T08:51:38.286982+0000","last_active":"2024-02-10T08:52:09.925589+0000","last_peered":"2024-02-10T08:52:09.925589+0000","last_clean":"2024-02-10T08:52:09.925589+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:52:09.925589+0000","last_undegraded":"2024-02-10T08:52:09.925589+0000","last_fullsized":"2024-02-10T08:52:09.925589+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:38.286913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:38.286913+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.286913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"822'44","reported_seq":72,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332792+0000","last_change":"2024-02-10T08:51:20.705787+0000","last_active":"2024-02-10T08:52:10.332792+0000","last_peered":"2024-02-10T08:52:10.332792+0000","last_clean":"2024-02-10T08:52:10.332792+0000","last_became_active":"2024-02-10T08:50:42.866275+0000","last_became_peered":"2024-02-10T08:50:42.866275+0000","last_unstale":"2024-02-10T08:52:10.332792+0000","last_undegraded":"2024-02-10T08:52:10.332792+0000","last_fullsized":"2024-02-10T08:52:10.332792+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'44","last_scrub_stamp":"2024-02-10T08:51:20.705702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:20.705702+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":839,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333027+0000","last_change":"2024-02-10T08:51:36.669755+0000","last_active":"2024-02-10T08:52:10.333027+0000","last_peered":"2024-02-10T08:52:10.333027+0000","last_clean":"2024-02-10T08:52:10.333027+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:52:10.333027+0000","last_undegraded":"2024-02-10T08:52:10.333027+0000","last_fullsized":"2024-02-10T08:52:10.333027+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:36.669648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:51:36.669648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":840,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333682+0000","last_change":"2024-02-10T08:51:22.207745+0000","last_active":"2024-02-10T08:52:10.333682+0000","last_peered":"2024-02-10T08:52:10.333682+0000","last_clean":"2024-02-10T08:52:10.333682+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:52:10.333682+0000","last_undegraded":"2024-02-10T08:52:10.333682+0000","last_fullsized":"2024-02-10T08:52:10.333682+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:22.207631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.207631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703270+0000","last_change":"2024-02-10T08:51:51.975568+0000","last_active":"2024-02-10T08:52:09.703270+0000","last_peered":"2024-02-10T08:52:09.703270+0000","last_clean":"2024-02-10T08:52:09.703270+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:52:09.703270+0000","last_undegraded":"2024-02-10T08:52:09.703270+0000","last_fullsized":"2024-02-10T08:52:09.703270+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:51.975472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:51.975472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":307,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925370+0000","last_change":"2024-02-10T08:50:48.526828+0000","last_active":"2024-02-10T08:52:09.925370+0000","last_peered":"2024-02-10T08:52:09.925370+0000","last_clean":"2024-02-10T08:52:09.925370+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:52:09.925370+0000","last_undegraded":"2024-02-10T08:52:09.925370+0000","last_fullsized":"2024-02-10T08:52:09.925370+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:50:48.526761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:50:48.526761+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":167,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701598+0000","last_change":"2024-02-10T08:51:18.540799+0000","last_active":"2024-02-10T08:52:09.701598+0000","last_peered":"2024-02-10T08:52:09.701598+0000","last_clean":"2024-02-10T08:52:09.701598+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:52:09.701598+0000","last_undegraded":"2024-02-10T08:52:09.701598+0000","last_fullsized":"2024-02-10T08:52:09.701598+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:51:18.540724+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:51:18.540724+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701718+0000","last_change":"2024-02-10T08:51:01.423644+0000","last_active":"2024-02-10T08:52:09.701718+0000","last_peered":"2024-02-10T08:52:09.701718+0000","last_clean":"2024-02-10T08:52:09.701718+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:52:09.701718+0000","last_undegraded":"2024-02-10T08:52:09.701718+0000","last_fullsized":"2024-02-10T08:52:09.701718+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:01.423585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_clean_scrub_stamp":"2024-02-10T08:51:01.423585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925213+0000","last_change":"2024-02-10T08:51:47.363190+0000","last_active":"2024-02-10T08:52:09.925213+0000","last_peered":"2024-02-10T08:52:09.925213+0000","last_clean":"2024-02-10T08:52:09.925213+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:52:09.925213+0000","last_undegraded":"2024-02-10T08:52:09.925213+0000","last_fullsized":"2024-02-10T08:52:09.925213+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.363092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.363092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":780,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333435+0000","last_change":"2024-02-10T08:50:55.206205+0000","last_active":"2024-02-10T08:52:10.333435+0000","last_peered":"2024-02-10T08:52:10.333435+0000","last_clean":"2024-02-10T08:52:10.333435+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:52:10.333435+0000","last_undegraded":"2024-02-10T08:52:10.333435+0000","last_fullsized":"2024-02-10T08:52:10.333435+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:55.206123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:55.206123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":883,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703377+0000","last_change":"2024-02-10T08:51:38.867904+0000","last_active":"2024-02-10T08:52:09.703377+0000","last_peered":"2024-02-10T08:52:09.703377+0000","last_clean":"2024-02-10T08:52:09.703377+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:52:09.703377+0000","last_undegraded":"2024-02-10T08:52:09.703377+0000","last_fullsized":"2024-02-10T08:52:09.703377+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:38.867817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.867817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":755,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701228+0000","last_change":"2024-02-10T08:51:38.814227+0000","last_active":"2024-02-10T08:52:09.701228+0000","last_peered":"2024-02-10T08:52:09.701228+0000","last_clean":"2024-02-10T08:52:09.701228+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:52:09.701228+0000","last_undegraded":"2024-02-10T08:52:09.701228+0000","last_fullsized":"2024-02-10T08:52:09.701228+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:51:38.814098+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.814098+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":720,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701980+0000","last_change":"2024-02-10T08:50:41.968410+0000","last_active":"2024-02-10T08:52:09.701980+0000","last_peered":"2024-02-10T08:52:09.701980+0000","last_clean":"2024-02-10T08:52:09.701980+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:52:09.701980+0000","last_undegraded":"2024-02-10T08:52:09.701980+0000","last_fullsized":"2024-02-10T08:52:09.701980+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:50:41.968315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:50:41.968315+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333401+0000","last_change":"2024-02-10T08:52:08.306127+0000","last_active":"2024-02-10T08:52:10.333401+0000","last_peered":"2024-02-10T08:52:10.333401+0000","last_clean":"2024-02-10T08:52:10.333401+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:52:10.333401+0000","last_undegraded":"2024-02-10T08:52:10.333401+0000","last_fullsized":"2024-02-10T08:52:10.333401+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:08.306037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:08.306037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":168,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332926+0000","last_change":"2024-02-10T08:51:35.704431+0000","last_active":"2024-02-10T08:52:10.332926+0000","last_peered":"2024-02-10T08:52:10.332926+0000","last_clean":"2024-02-10T08:52:10.332926+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:52:10.332926+0000","last_undegraded":"2024-02-10T08:52:10.332926+0000","last_fullsized":"2024-02-10T08:52:10.332926+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.704334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.704334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":243,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925788+0000","last_change":"2024-02-10T08:50:45.545641+0000","last_active":"2024-02-10T08:52:09.925788+0000","last_peered":"2024-02-10T08:52:09.925788+0000","last_clean":"2024-02-10T08:52:09.925788+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:52:09.925788+0000","last_undegraded":"2024-02-10T08:52:09.925788+0000","last_fullsized":"2024-02-10T08:52:09.925788+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:45.545577+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:50:45.545577+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":870,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703360+0000","last_change":"2024-02-10T08:51:40.935809+0000","last_active":"2024-02-10T08:52:09.703360+0000","last_peered":"2024-02-10T08:52:09.703360+0000","last_clean":"2024-02-10T08:52:09.703360+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:52:09.703360+0000","last_undegraded":"2024-02-10T08:52:09.703360+0000","last_fullsized":"2024-02-10T08:52:09.703360+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.935714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:02.159169+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.935714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":855,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703771+0000","last_change":"2024-02-10T08:52:06.942398+0000","last_active":"2024-02-10T08:52:09.703771+0000","last_peered":"2024-02-10T08:52:09.703771+0000","last_clean":"2024-02-10T08:52:09.703771+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:52:09.703771+0000","last_undegraded":"2024-02-10T08:52:09.703771+0000","last_fullsized":"2024-02-10T08:52:09.703771+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.942295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.942295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":648,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701398+0000","last_change":"2024-02-10T08:52:04.820047+0000","last_active":"2024-02-10T08:52:09.701398+0000","last_peered":"2024-02-10T08:52:09.701398+0000","last_clean":"2024-02-10T08:52:09.701398+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:52:09.701398+0000","last_undegraded":"2024-02-10T08:52:09.701398+0000","last_fullsized":"2024-02-10T08:52:09.701398+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.819963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.819963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":902,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333690+0000","last_change":"2024-02-10T08:51:27.219904+0000","last_active":"2024-02-10T08:52:10.333690+0000","last_peered":"2024-02-10T08:52:10.333690+0000","last_clean":"2024-02-10T08:52:10.333690+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:52:10.333690+0000","last_undegraded":"2024-02-10T08:52:10.333690+0000","last_fullsized":"2024-02-10T08:52:10.333690+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:51:27.219837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:27.219837+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335234+0000","last_change":"2024-02-10T08:51:29.010376+0000","last_active":"2024-02-10T08:52:10.335234+0000","last_peered":"2024-02-10T08:52:10.335234+0000","last_clean":"2024-02-10T08:52:10.335234+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:52:10.335234+0000","last_undegraded":"2024-02-10T08:52:10.335234+0000","last_fullsized":"2024-02-10T08:52:10.335234+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:29.010284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.010284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934077+0000","last_change":"2024-02-10T08:51:01.855495+0000","last_active":"2024-02-10T08:52:09.934077+0000","last_peered":"2024-02-10T08:52:09.934077+0000","last_clean":"2024-02-10T08:52:09.934077+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:52:09.934077+0000","last_undegraded":"2024-02-10T08:52:09.934077+0000","last_fullsized":"2024-02-10T08:52:09.934077+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:01.855415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:01.855415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":776,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332785+0000","last_change":"2024-02-10T08:51:21.680721+0000","last_active":"2024-02-10T08:52:10.332785+0000","last_peered":"2024-02-10T08:52:10.332785+0000","last_clean":"2024-02-10T08:52:10.332785+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:52:10.332785+0000","last_undegraded":"2024-02-10T08:52:10.332785+0000","last_fullsized":"2024-02-10T08:52:10.332785+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:21.680621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:51:21.680621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":930,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333398+0000","last_change":"2024-02-10T08:51:02.242143+0000","last_active":"2024-02-10T08:52:10.333398+0000","last_peered":"2024-02-10T08:52:10.333398+0000","last_clean":"2024-02-10T08:52:10.333398+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:52:10.333398+0000","last_undegraded":"2024-02-10T08:52:10.333398+0000","last_fullsized":"2024-02-10T08:52:10.333398+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:02.242057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:02.242057+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":691,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333078+0000","last_change":"2024-02-10T08:50:56.254276+0000","last_active":"2024-02-10T08:52:10.333078+0000","last_peered":"2024-02-10T08:52:10.333078+0000","last_clean":"2024-02-10T08:52:10.333078+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:52:10.333078+0000","last_undegraded":"2024-02-10T08:52:10.333078+0000","last_fullsized":"2024-02-10T08:52:10.333078+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:56.254214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:56.254214+0000","last_clean_scrub_stamp":"2024-02-10T08:50:56.254214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1104,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333312+0000","last_change":"2024-02-10T08:52:01.550377+0000","last_active":"2024-02-10T08:52:10.333312+0000","last_peered":"2024-02-10T08:52:10.333312+0000","last_clean":"2024-02-10T08:52:10.333312+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:52:10.333312+0000","last_undegraded":"2024-02-10T08:52:10.333312+0000","last_fullsized":"2024-02-10T08:52:10.333312+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13","last_scrub_stamp":"2024-02-10T08:52:01.550272+0000","last_deep_scrub":"711'13","last_deep_scrub_stamp":"2024-02-10T08:52:01.550272+0000","last_clean_scrub_stamp":"2024-02-10T08:52:01.550272+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925883+0000","last_change":"2024-02-10T08:51:40.233510+0000","last_active":"2024-02-10T08:52:09.925883+0000","last_peered":"2024-02-10T08:52:09.925883+0000","last_clean":"2024-02-10T08:52:09.925883+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:52:09.925883+0000","last_undegraded":"2024-02-10T08:52:09.925883+0000","last_fullsized":"2024-02-10T08:52:09.925883+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.233445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:40.233445+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.233445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925690+0000","last_change":"2024-02-10T08:51:53.323765+0000","last_active":"2024-02-10T08:52:09.925690+0000","last_peered":"2024-02-10T08:52:09.925690+0000","last_clean":"2024-02-10T08:52:09.925690+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:52:09.925690+0000","last_undegraded":"2024-02-10T08:52:09.925690+0000","last_fullsized":"2024-02-10T08:52:09.925690+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:53.323700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:53.323700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"798'19","reported_seq":881,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333270+0000","last_change":"2024-02-10T08:51:04.698846+0000","last_active":"2024-02-10T08:52:10.333270+0000","last_peered":"2024-02-10T08:52:10.333270+0000","last_clean":"2024-02-10T08:52:10.333270+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:52:10.333270+0000","last_undegraded":"2024-02-10T08:52:10.333270+0000","last_fullsized":"2024-02-10T08:52:10.333270+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'19","last_scrub_stamp":"2024-02-10T08:51:04.698764+0000","last_deep_scrub":"798'19","last_deep_scrub_stamp":"2024-02-10T08:51:04.698764+0000","last_clean_scrub_stamp":"2024-02-10T08:51:04.698764+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":593,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701195+0000","last_change":"2024-02-10T08:51:09.970486+0000","last_active":"2024-02-10T08:52:09.701195+0000","last_peered":"2024-02-10T08:52:09.701195+0000","last_clean":"2024-02-10T08:52:09.701195+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:52:09.701195+0000","last_undegraded":"2024-02-10T08:52:09.701195+0000","last_fullsized":"2024-02-10T08:52:09.701195+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:09.970399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:51:09.970399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":1075,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333509+0000","last_change":"2024-02-10T08:51:07.231908+0000","last_active":"2024-02-10T08:52:10.333509+0000","last_peered":"2024-02-10T08:52:10.333509+0000","last_clean":"2024-02-10T08:52:10.333509+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:52:10.333509+0000","last_undegraded":"2024-02-10T08:52:10.333509+0000","last_fullsized":"2024-02-10T08:52:10.333509+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:51:07.231794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.231794+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":655,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701948+0000","last_change":"2024-02-10T08:51:33.814101+0000","last_active":"2024-02-10T08:52:09.701948+0000","last_peered":"2024-02-10T08:52:09.701948+0000","last_clean":"2024-02-10T08:52:09.701948+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:52:09.701948+0000","last_undegraded":"2024-02-10T08:52:09.701948+0000","last_fullsized":"2024-02-10T08:52:09.701948+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:33.814004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:33.814004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":810,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703352+0000","last_change":"2024-02-10T08:51:08.778455+0000","last_active":"2024-02-10T08:52:09.703352+0000","last_peered":"2024-02-10T08:52:09.703352+0000","last_clean":"2024-02-10T08:52:09.703352+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:52:09.703352+0000","last_undegraded":"2024-02-10T08:52:09.703352+0000","last_fullsized":"2024-02-10T08:52:09.703352+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:51:08.778387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.778387+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":160,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701701+0000","last_change":"2024-02-10T08:50:52.318856+0000","last_active":"2024-02-10T08:52:09.701701+0000","last_peered":"2024-02-10T08:52:09.701701+0000","last_clean":"2024-02-10T08:52:09.701701+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:52:09.701701+0000","last_undegraded":"2024-02-10T08:52:09.701701+0000","last_fullsized":"2024-02-10T08:52:09.701701+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:52.318773+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:50:52.318773+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":830,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335581+0000","last_change":"2024-02-10T08:51:37.042907+0000","last_active":"2024-02-10T08:52:10.335581+0000","last_peered":"2024-02-10T08:52:10.335581+0000","last_clean":"2024-02-10T08:52:10.335581+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:52:10.335581+0000","last_undegraded":"2024-02-10T08:52:10.335581+0000","last_fullsized":"2024-02-10T08:52:10.335581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:37.042831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:37.042831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":855,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333323+0000","last_change":"2024-02-10T08:51:41.179889+0000","last_active":"2024-02-10T08:52:10.333323+0000","last_peered":"2024-02-10T08:52:10.333323+0000","last_clean":"2024-02-10T08:52:10.333323+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:52:10.333323+0000","last_undegraded":"2024-02-10T08:52:10.333323+0000","last_fullsized":"2024-02-10T08:52:10.333323+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:41.179821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:41.179821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925498+0000","last_change":"2024-02-10T08:50:50.543955+0000","last_active":"2024-02-10T08:52:09.925498+0000","last_peered":"2024-02-10T08:52:09.925498+0000","last_clean":"2024-02-10T08:52:09.925498+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:52:09.925498+0000","last_undegraded":"2024-02-10T08:52:09.925498+0000","last_fullsized":"2024-02-10T08:52:09.925498+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:50.543887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:50.543887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":793,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335944+0000","last_change":"2024-02-10T08:51:58.913272+0000","last_active":"2024-02-10T08:52:10.335944+0000","last_peered":"2024-02-10T08:52:10.335944+0000","last_clean":"2024-02-10T08:52:10.335944+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:52:10.335944+0000","last_undegraded":"2024-02-10T08:52:10.335944+0000","last_fullsized":"2024-02-10T08:52:10.335944+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:58.913187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:58.913187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":767,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701325+0000","last_change":"2024-02-10T08:51:35.771062+0000","last_active":"2024-02-10T08:52:09.701325+0000","last_peered":"2024-02-10T08:52:09.701325+0000","last_clean":"2024-02-10T08:52:09.701325+0000","last_became_active":"2024-02-10T08:50:42.866200+0000","last_became_peered":"2024-02-10T08:50:42.866200+0000","last_unstale":"2024-02-10T08:52:09.701325+0000","last_undegraded":"2024-02-10T08:52:09.701325+0000","last_fullsized":"2024-02-10T08:52:09.701325+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":820,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:51:35.770985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.770985+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701501+0000","last_change":"2024-02-10T08:51:22.539464+0000","last_active":"2024-02-10T08:52:09.701501+0000","last_peered":"2024-02-10T08:52:09.701501+0000","last_clean":"2024-02-10T08:52:09.701501+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:52:09.701501+0000","last_undegraded":"2024-02-10T08:52:09.701501+0000","last_fullsized":"2024-02-10T08:52:09.701501+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:22.539355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:22.539355+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.539355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701781+0000","last_change":"2024-02-10T08:51:24.531448+0000","last_active":"2024-02-10T08:52:09.701781+0000","last_peered":"2024-02-10T08:52:09.701781+0000","last_clean":"2024-02-10T08:52:09.701781+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:52:09.701781+0000","last_undegraded":"2024-02-10T08:52:09.701781+0000","last_fullsized":"2024-02-10T08:52:09.701781+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:24.531342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:51:24.531342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":835,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332814+0000","last_change":"2024-02-10T08:51:58.222709+0000","last_active":"2024-02-10T08:52:10.332814+0000","last_peered":"2024-02-10T08:52:10.332814+0000","last_clean":"2024-02-10T08:52:10.332814+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:52:10.332814+0000","last_undegraded":"2024-02-10T08:52:10.332814+0000","last_fullsized":"2024-02-10T08:52:10.332814+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:58.222637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:58.222637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":786,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333515+0000","last_change":"2024-02-10T08:50:59.260262+0000","last_active":"2024-02-10T08:52:10.333515+0000","last_peered":"2024-02-10T08:52:10.333515+0000","last_clean":"2024-02-10T08:52:10.333515+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:52:10.333515+0000","last_undegraded":"2024-02-10T08:52:10.333515+0000","last_fullsized":"2024-02-10T08:52:10.333515+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:59.260164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:59.260164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"827'10","reported_seq":860,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332702+0000","last_change":"2024-02-10T08:51:03.211894+0000","last_active":"2024-02-10T08:52:10.332702+0000","last_peered":"2024-02-10T08:52:10.332702+0000","last_clean":"2024-02-10T08:52:10.332702+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:52:10.332702+0000","last_undegraded":"2024-02-10T08:52:10.332702+0000","last_fullsized":"2024-02-10T08:52:10.332702+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"827'10","last_scrub_stamp":"2024-02-10T08:51:03.211831+0000","last_deep_scrub":"827'10","last_deep_scrub_stamp":"2024-02-10T08:51:03.211831+0000","last_clean_scrub_stamp":"2024-02-10T08:51:03.211831+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":825,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333426+0000","last_change":"2024-02-10T08:51:39.711442+0000","last_active":"2024-02-10T08:52:10.333426+0000","last_peered":"2024-02-10T08:52:10.333426+0000","last_clean":"2024-02-10T08:52:10.333426+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:52:10.333426+0000","last_undegraded":"2024-02-10T08:52:10.333426+0000","last_fullsized":"2024-02-10T08:52:10.333426+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:39.711351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:39.711351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":851,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335202+0000","last_change":"2024-02-10T08:51:34.025342+0000","last_active":"2024-02-10T08:52:10.335202+0000","last_peered":"2024-02-10T08:52:10.335202+0000","last_clean":"2024-02-10T08:52:10.335202+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:52:10.335202+0000","last_undegraded":"2024-02-10T08:52:10.335202+0000","last_fullsized":"2024-02-10T08:52:10.335202+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:34.025230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:34.025230+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.025230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933964+0000","last_change":"2024-02-10T08:50:54.999584+0000","last_active":"2024-02-10T08:52:09.933964+0000","last_peered":"2024-02-10T08:52:09.933964+0000","last_clean":"2024-02-10T08:52:09.933964+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:52:09.933964+0000","last_undegraded":"2024-02-10T08:52:09.933964+0000","last_fullsized":"2024-02-10T08:52:09.933964+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.999503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.999503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":798,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333287+0000","last_change":"2024-02-10T08:51:15.167860+0000","last_active":"2024-02-10T08:52:10.333287+0000","last_peered":"2024-02-10T08:52:10.333287+0000","last_clean":"2024-02-10T08:52:10.333287+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:52:10.333287+0000","last_undegraded":"2024-02-10T08:52:10.333287+0000","last_fullsized":"2024-02-10T08:52:10.333287+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:15.167795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.167795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":889,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333056+0000","last_change":"2024-02-10T08:51:44.251816+0000","last_active":"2024-02-10T08:52:10.333056+0000","last_peered":"2024-02-10T08:52:10.333056+0000","last_clean":"2024-02-10T08:52:10.333056+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:52:10.333056+0000","last_undegraded":"2024-02-10T08:52:10.333056+0000","last_fullsized":"2024-02-10T08:52:10.333056+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:51:44.251748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.251748+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":830,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335881+0000","last_change":"2024-02-10T08:51:54.956900+0000","last_active":"2024-02-10T08:52:10.335881+0000","last_peered":"2024-02-10T08:52:10.335881+0000","last_clean":"2024-02-10T08:52:10.335881+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:52:10.335881+0000","last_undegraded":"2024-02-10T08:52:10.335881+0000","last_fullsized":"2024-02-10T08:52:10.335881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.956821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.956821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933505+0000","last_change":"2024-02-10T08:51:54.866778+0000","last_active":"2024-02-10T08:52:09.933505+0000","last_peered":"2024-02-10T08:52:09.933505+0000","last_clean":"2024-02-10T08:52:09.933505+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:52:09.933505+0000","last_undegraded":"2024-02-10T08:52:09.933505+0000","last_fullsized":"2024-02-10T08:52:09.933505+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.866690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.866690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":835,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333489+0000","last_change":"2024-02-10T08:52:10.333022+0000","last_active":"2024-02-10T08:52:10.333489+0000","last_peered":"2024-02-10T08:52:10.333489+0000","last_clean":"2024-02-10T08:52:10.333489+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:52:10.333489+0000","last_undegraded":"2024-02-10T08:52:10.333489+0000","last_fullsized":"2024-02-10T08:52:10.333489+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:10.332923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:10.332923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":232,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925491+0000","last_change":"2024-02-10T08:50:56.603701+0000","last_active":"2024-02-10T08:52:09.925491+0000","last_peered":"2024-02-10T08:52:09.925491+0000","last_clean":"2024-02-10T08:52:09.925491+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:52:09.925491+0000","last_undegraded":"2024-02-10T08:52:09.925491+0000","last_fullsized":"2024-02-10T08:52:09.925491+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:56.603602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:50:56.603602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":244,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925589+0000","last_change":"2024-02-10T08:51:34.334569+0000","last_active":"2024-02-10T08:52:09.925589+0000","last_peered":"2024-02-10T08:52:09.925589+0000","last_clean":"2024-02-10T08:52:09.925589+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:52:09.925589+0000","last_undegraded":"2024-02-10T08:52:09.925589+0000","last_fullsized":"2024-02-10T08:52:09.925589+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:34.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.334504+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":863,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332569+0000","last_change":"2024-02-10T08:51:52.646508+0000","last_active":"2024-02-10T08:52:10.332569+0000","last_peered":"2024-02-10T08:52:10.332569+0000","last_clean":"2024-02-10T08:52:10.332569+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:52:10.332569+0000","last_undegraded":"2024-02-10T08:52:10.332569+0000","last_fullsized":"2024-02-10T08:52:10.332569+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:52.646424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:24.469783+0000","last_clean_scrub_stamp":"2024-02-10T08:51:52.646424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933990+0000","last_change":"2024-02-10T08:51:48.917250+0000","last_active":"2024-02-10T08:52:09.933990+0000","last_peered":"2024-02-10T08:52:09.933990+0000","last_clean":"2024-02-10T08:52:09.933990+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:52:09.933990+0000","last_undegraded":"2024-02-10T08:52:09.933990+0000","last_fullsized":"2024-02-10T08:52:09.933990+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:51:48.917167+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.917167+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703416+0000","last_change":"2024-02-10T08:51:54.996240+0000","last_active":"2024-02-10T08:52:09.703416+0000","last_peered":"2024-02-10T08:52:09.703416+0000","last_clean":"2024-02-10T08:52:09.703416+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:52:09.703416+0000","last_undegraded":"2024-02-10T08:52:09.703416+0000","last_fullsized":"2024-02-10T08:52:09.703416+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.996111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.996111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":906,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335990+0000","last_change":"2024-02-10T08:52:07.856178+0000","last_active":"2024-02-10T08:52:10.335990+0000","last_peered":"2024-02-10T08:52:10.335990+0000","last_clean":"2024-02-10T08:52:10.335990+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:52:10.335990+0000","last_undegraded":"2024-02-10T08:52:10.335990+0000","last_fullsized":"2024-02-10T08:52:10.335990+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:52:07.856094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:52:07.856094+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":841,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332904+0000","last_change":"2024-02-10T08:52:04.393940+0000","last_active":"2024-02-10T08:52:10.332904+0000","last_peered":"2024-02-10T08:52:10.332904+0000","last_clean":"2024-02-10T08:52:10.332904+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:52:10.332904+0000","last_undegraded":"2024-02-10T08:52:10.332904+0000","last_fullsized":"2024-02-10T08:52:10.332904+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.393877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:04.393877+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.393877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701432+0000","last_change":"2024-02-10T08:50:54.357965+0000","last_active":"2024-02-10T08:52:09.701432+0000","last_peered":"2024-02-10T08:52:09.701432+0000","last_clean":"2024-02-10T08:52:09.701432+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:52:09.701432+0000","last_undegraded":"2024-02-10T08:52:09.701432+0000","last_fullsized":"2024-02-10T08:52:09.701432+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.357895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701143+0000","last_change":"2024-02-10T08:51:21.495313+0000","last_active":"2024-02-10T08:52:09.701143+0000","last_peered":"2024-02-10T08:52:09.701143+0000","last_clean":"2024-02-10T08:52:09.701143+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:52:09.701143+0000","last_undegraded":"2024-02-10T08:52:09.701143+0000","last_fullsized":"2024-02-10T08:52:09.701143+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:21.495232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:51:21.495232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703589+0000","last_change":"2024-02-10T08:51:19.817098+0000","last_active":"2024-02-10T08:52:09.703589+0000","last_peered":"2024-02-10T08:52:09.703589+0000","last_clean":"2024-02-10T08:52:09.703589+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:52:09.703589+0000","last_undegraded":"2024-02-10T08:52:09.703589+0000","last_fullsized":"2024-02-10T08:52:09.703589+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:51:19.817010+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:51:19.817010+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335511+0000","last_change":"2024-02-10T08:51:19.948572+0000","last_active":"2024-02-10T08:52:10.335511+0000","last_peered":"2024-02-10T08:52:10.335511+0000","last_clean":"2024-02-10T08:52:10.335511+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:52:10.335511+0000","last_undegraded":"2024-02-10T08:52:10.335511+0000","last_fullsized":"2024-02-10T08:52:10.335511+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:19.948488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:19.948488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":661,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701790+0000","last_change":"2024-02-10T08:51:44.920711+0000","last_active":"2024-02-10T08:52:09.701790+0000","last_peered":"2024-02-10T08:52:09.701790+0000","last_clean":"2024-02-10T08:52:09.701790+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:52:09.701790+0000","last_undegraded":"2024-02-10T08:52:09.701790+0000","last_fullsized":"2024-02-10T08:52:09.701790+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:44.920624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.920624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925515+0000","last_change":"2024-02-10T08:50:54.556265+0000","last_active":"2024-02-10T08:52:09.925515+0000","last_peered":"2024-02-10T08:52:09.925515+0000","last_clean":"2024-02-10T08:52:09.925515+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:52:09.925515+0000","last_undegraded":"2024-02-10T08:52:09.925515+0000","last_fullsized":"2024-02-10T08:52:09.925515+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.556197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.556197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":257,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333352+0000","last_change":"2024-02-10T08:51:29.212829+0000","last_active":"2024-02-10T08:52:10.333352+0000","last_peered":"2024-02-10T08:52:10.333352+0000","last_clean":"2024-02-10T08:52:10.333352+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:52:10.333352+0000","last_undegraded":"2024-02-10T08:52:10.333352+0000","last_fullsized":"2024-02-10T08:52:10.333352+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:51:29.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.212763+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":841,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333247+0000","last_change":"2024-02-10T08:51:47.281899+0000","last_active":"2024-02-10T08:52:10.333247+0000","last_peered":"2024-02-10T08:52:10.333247+0000","last_clean":"2024-02-10T08:52:10.333247+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:52:10.333247+0000","last_undegraded":"2024-02-10T08:52:10.333247+0000","last_fullsized":"2024-02-10T08:52:10.333247+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.281837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.281837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":817,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333734+0000","last_change":"2024-02-10T08:52:07.332969+0000","last_active":"2024-02-10T08:52:10.333734+0000","last_peered":"2024-02-10T08:52:10.333734+0000","last_clean":"2024-02-10T08:52:10.333734+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:52:10.333734+0000","last_undegraded":"2024-02-10T08:52:10.333734+0000","last_fullsized":"2024-02-10T08:52:10.333734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:07.332902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:51.264745+0000","last_clean_scrub_stamp":"2024-02-10T08:52:07.332902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":166,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701098+0000","last_change":"2024-02-10T08:52:06.483942+0000","last_active":"2024-02-10T08:52:09.701098+0000","last_peered":"2024-02-10T08:52:09.701098+0000","last_clean":"2024-02-10T08:52:09.701098+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:52:09.701098+0000","last_undegraded":"2024-02-10T08:52:09.701098+0000","last_fullsized":"2024-02-10T08:52:09.701098+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.483874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:51.275205+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.483874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"832'19","reported_seq":722,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701337+0000","last_change":"2024-02-10T08:51:22.002805+0000","last_active":"2024-02-10T08:52:09.701337+0000","last_peered":"2024-02-10T08:52:09.701337+0000","last_clean":"2024-02-10T08:52:09.701337+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:52:09.701337+0000","last_undegraded":"2024-02-10T08:52:09.701337+0000","last_fullsized":"2024-02-10T08:52:09.701337+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'19","last_scrub_stamp":"2024-02-10T08:51:22.002701+0000","last_deep_scrub":"832'19","last_deep_scrub_stamp":"2024-02-10T08:51:22.002701+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.002701+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934031+0000","last_change":"2024-02-10T08:51:53.835128+0000","last_active":"2024-02-10T08:52:09.934031+0000","last_peered":"2024-02-10T08:52:09.934031+0000","last_clean":"2024-02-10T08:52:09.934031+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:52:09.934031+0000","last_undegraded":"2024-02-10T08:52:09.934031+0000","last_fullsized":"2024-02-10T08:52:09.934031+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:53.835040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:51:53.835040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925618+0000","last_change":"2024-02-10T08:50:53.552617+0000","last_active":"2024-02-10T08:52:09.925618+0000","last_peered":"2024-02-10T08:52:09.925618+0000","last_clean":"2024-02-10T08:52:09.925618+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:52:09.925618+0000","last_undegraded":"2024-02-10T08:52:09.925618+0000","last_fullsized":"2024-02-10T08:52:09.925618+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:53.552521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:53.552521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":806,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.928908+0000","last_change":"2024-02-10T08:52:09.928881+0000","last_active":"2024-02-10T08:52:09.928908+0000","last_peered":"2024-02-10T08:52:09.928908+0000","last_clean":"2024-02-10T08:52:09.928908+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:52:09.928908+0000","last_undegraded":"2024-02-10T08:52:09.928908+0000","last_fullsized":"2024-02-10T08:52:09.928908+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.928813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.928813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":721,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701648+0000","last_change":"2024-02-10T08:52:06.809920+0000","last_active":"2024-02-10T08:52:09.701648+0000","last_peered":"2024-02-10T08:52:09.701648+0000","last_clean":"2024-02-10T08:52:09.701648+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:52:09.701648+0000","last_undegraded":"2024-02-10T08:52:09.701648+0000","last_fullsized":"2024-02-10T08:52:09.701648+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:52:06.809842+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.809842+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":353,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333378+0000","last_change":"2024-02-10T08:51:59.264492+0000","last_active":"2024-02-10T08:52:10.333378+0000","last_peered":"2024-02-10T08:52:10.333378+0000","last_clean":"2024-02-10T08:52:10.333378+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:52:10.333378+0000","last_undegraded":"2024-02-10T08:52:10.333378+0000","last_fullsized":"2024-02-10T08:52:10.333378+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.264427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.264427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":833,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332646+0000","last_change":"2024-02-10T08:51:59.517087+0000","last_active":"2024-02-10T08:52:10.332646+0000","last_peered":"2024-02-10T08:52:10.332646+0000","last_clean":"2024-02-10T08:52:10.332646+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:52:10.332646+0000","last_undegraded":"2024-02-10T08:52:10.332646+0000","last_fullsized":"2024-02-10T08:52:10.332646+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.516966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.516966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":231,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925826+0000","last_change":"2024-02-10T08:50:42.585429+0000","last_active":"2024-02-10T08:52:09.925826+0000","last_peered":"2024-02-10T08:52:09.925826+0000","last_clean":"2024-02-10T08:52:09.925826+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:52:09.925826+0000","last_undegraded":"2024-02-10T08:52:09.925826+0000","last_fullsized":"2024-02-10T08:52:09.925826+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:42.585363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:42.585363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":753,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701015+0000","last_change":"2024-02-10T08:50:54.877195+0000","last_active":"2024-02-10T08:52:09.701015+0000","last_peered":"2024-02-10T08:52:09.701015+0000","last_clean":"2024-02-10T08:52:09.701015+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:52:09.701015+0000","last_undegraded":"2024-02-10T08:52:09.701015+0000","last_fullsized":"2024-02-10T08:52:09.701015+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:50:54.877106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.877106+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":836,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333629+0000","last_change":"2024-02-10T08:51:06.203366+0000","last_active":"2024-02-10T08:52:10.333629+0000","last_peered":"2024-02-10T08:52:10.333629+0000","last_clean":"2024-02-10T08:52:10.333629+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:52:10.333629+0000","last_undegraded":"2024-02-10T08:52:10.333629+0000","last_fullsized":"2024-02-10T08:52:10.333629+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:06.203294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:06.203294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":676,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:11.790259+0000","last_change":"2024-02-10T08:52:11.790259+0000","last_active":"2024-02-10T08:52:11.790259+0000","last_peered":"2024-02-10T08:52:11.790259+0000","last_clean":"2024-02-10T08:52:11.790259+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:52:11.790259+0000","last_undegraded":"2024-02-10T08:52:11.790259+0000","last_fullsized":"2024-02-10T08:52:11.790259+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:11.790177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:11.790177+0000","last_clean_scrub_stamp":"2024-02-10T08:52:11.790177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":794,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332806+0000","last_change":"2024-02-10T08:51:28.709454+0000","last_active":"2024-02-10T08:52:10.332806+0000","last_peered":"2024-02-10T08:52:10.332806+0000","last_clean":"2024-02-10T08:52:10.332806+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:52:10.332806+0000","last_undegraded":"2024-02-10T08:52:10.332806+0000","last_fullsized":"2024-02-10T08:52:10.332806+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:28.709361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:28.709361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":909,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333068+0000","last_change":"2024-02-10T08:51:35.177551+0000","last_active":"2024-02-10T08:52:10.333068+0000","last_peered":"2024-02-10T08:52:10.333068+0000","last_clean":"2024-02-10T08:52:10.333068+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:52:10.333068+0000","last_undegraded":"2024-02-10T08:52:10.333068+0000","last_fullsized":"2024-02-10T08:52:10.333068+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:51:35.177438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.177438+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":248,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925874+0000","last_change":"2024-02-10T08:51:35.323278+0000","last_active":"2024-02-10T08:52:09.925874+0000","last_peered":"2024-02-10T08:52:09.925874+0000","last_clean":"2024-02-10T08:52:09.925874+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:52:09.925874+0000","last_undegraded":"2024-02-10T08:52:09.925874+0000","last_fullsized":"2024-02-10T08:52:09.925874+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.323215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.323215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":832,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333139+0000","last_change":"2024-02-10T08:51:36.157770+0000","last_active":"2024-02-10T08:52:10.333139+0000","last_peered":"2024-02-10T08:52:10.333139+0000","last_clean":"2024-02-10T08:52:10.333139+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:52:10.333139+0000","last_undegraded":"2024-02-10T08:52:10.333139+0000","last_fullsized":"2024-02-10T08:52:10.333139+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:36.157645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:36.157645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":810,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.334909+0000","last_change":"2024-02-10T08:52:06.891432+0000","last_active":"2024-02-10T08:52:10.334909+0000","last_peered":"2024-02-10T08:52:10.334909+0000","last_clean":"2024-02-10T08:52:10.334909+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:52:10.334909+0000","last_undegraded":"2024-02-10T08:52:10.334909+0000","last_fullsized":"2024-02-10T08:52:10.334909+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.891358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.891358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":11310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299622,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311452,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433700,"statfs":{"total":95995035648,"available":95676108800,"internally_reserved":0,"allocated":1789952,"data_stored":545607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.60399999999999998},"min":{"1min":0.314,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64600000000000002,"5min":1.3049999999999999,"15min":22.143000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.61599999999999999,"5min":1.0740000000000001,"15min":11.000999999999999},"last":0.625}]},{"osd":1,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.60899999999999999},"min":{"1min":0.36299999999999999,"5min":0.28100000000000003,"15min":0.27200000000000002},"max":{"1min":0.65400000000000003,"5min":1.333,"15min":9.2560000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.60899999999999999},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.22500000000000001},"max":{"1min":0.621,"5min":1.1240000000000001,"15min":10.978},"last":0.69299999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49399999999999999,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.253},"max":{"1min":0.623,"5min":0.81599999999999995,"15min":7.3949999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.66400000000000003},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.251},"max":{"1min":0.76100000000000001,"5min":1.3200000000000001,"15min":19.369},"last":0.68200000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.505,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.78300000000000003,"15min":9.3439999999999994},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55000000000000004,"15min":0.69799999999999995},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.314},"max":{"1min":0.72399999999999998,"5min":1.345,"15min":22.102},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64900000000000002,"5min":1.581,"15min":1.581},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58199999999999996,"15min":0.68200000000000005},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79600000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.86399999999999999,"5min":4.46,"15min":19.326000000000001},"last":0.501},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.51900000000000002},"min":{"1min":0.38200000000000001,"5min":0.27100000000000002,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":6.8650000000000002,"15min":10.946999999999999},"last":0.66700000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.38800000000000001,"5min":0.26800000000000002,"15min":0.23000000000000001},"max":{"1min":0.80800000000000005,"5min":4.718,"15min":22.687000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.437,"5min":0.27800000000000002,"15min":0.25900000000000001},"max":{"1min":0.71299999999999997,"5min":1.8899999999999999,"15min":5.484},"last":0.53500000000000003}]}]},{"osd":6,"up_from":44,"seq":188978561260,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441904,"statfs":{"total":95995035648,"available":95684509696,"internally_reserved":0,"allocated":1777664,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47499999999999998,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.76500000000000001,"15min":8.5250000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.30599999999999999,"15min":0.254},"max":{"1min":0.75700000000000001,"5min":2.649,"15min":18.777999999999999},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.65500000000000003},"min":{"1min":0.38300000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":18.425999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":2.4769999999999999,"15min":18.402999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.63},"min":{"1min":0.40500000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71899999999999997,"5min":2.4319999999999999,"15min":10.772},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.69499999999999995,"5min":2.3570000000000002,"15min":18.829000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"min":{"1min":0.34499999999999997,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":9.2650000000000006,"15min":16.823},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53300000000000003,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.502}]},{"osd":4,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.36699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.69899999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.49399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68999999999999995,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.38500000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.27600000000000002,"15min":0.224},"max":{"1min":0.75,"5min":0.75,"15min":18.853999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.33400000000000002,"5min":0.29499999999999998,"15min":0.26000000000000001},"max":{"1min":0.73999999999999999,"5min":0.76500000000000001,"15min":3.915},"last":0.40000000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":7.5650000000000004},"last":0.434},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.0229999999999999},"last":0.45300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303272,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441880,"statfs":{"total":95995035648,"available":95684485120,"internally_reserved":0,"allocated":1802240,"data_stored":539217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.45100000000000001},"min":{"1min":0.23499999999999999,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.48599999999999999,"5min":1.5669999999999999,"15min":7.8780000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.25800000000000001,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.51600000000000001,"5min":1.2669999999999999,"15min":16.166},"last":0.442}]},{"osd":2,"last update":"Sat Feb 10 08:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.65500000000000003},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.193},"max":{"1min":0.55200000000000005,"5min":1.637,"15min":34.985999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.55700000000000005},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.47399999999999998,"5min":0.93100000000000005,"15min":21.614999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35499999999999998,"15min":0.51600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.47099999999999997,"5min":0.73199999999999998,"15min":24.773},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.505},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45800000000000002,"5min":0.80000000000000004,"15min":16.120999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81299999999999994,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58799999999999997,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63}]},{"osd":5,"last update":"Sat Feb 10 08:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.34599999999999997,"15min":0.26800000000000002},"max":{"1min":0.77600000000000002,"5min":0.86899999999999999,"15min":9.5600000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.94699999999999995,"15min":7.8479999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57499999999999996,"15min":0.61899999999999999},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":0.76600000000000001,"5min":0.91500000000000004,"15min":9.532},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60499999999999998,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.33200000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":7.9000000000000004},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.67000000000000004},"min":{"1min":0.42599999999999999,"5min":0.39400000000000002,"15min":0.34799999999999998},"max":{"1min":0.75,"5min":0.88300000000000001,"15min":12.147},"last":0.623},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.70599999999999996},"min":{"1min":0.50700000000000001,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":7.9199999999999999},"last":0.67800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738633,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441908,"statfs":{"total":95995035648,"available":95684513792,"internally_reserved":0,"allocated":1773568,"data_stored":537231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.34999999999999998,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.156},"max":{"1min":0.58099999999999996,"5min":1.4239999999999999,"15min":11.960000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.379,"15min":0.496},"min":{"1min":0.223,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":3.8839999999999999,"15min":20.757999999999999},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34899999999999998,"15min":0.44900000000000001},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.46700000000000003,"5min":0.79400000000000004,"15min":11.987},"last":0.245},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34999999999999998,"15min":0.48099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":0.47899999999999998,"5min":0.64000000000000001,"15min":11.303000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":8.5310000000000006},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.442,"15min":0.47699999999999998},"min":{"1min":0.28100000000000003,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.86399999999999999,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82199999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63300000000000001,"15min":0.59299999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.67700000000000005,"5min":3.931,"15min":4.609},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.43099999999999999,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":0.72099999999999997,"5min":3.2320000000000002,"15min":3.4079999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.318},"max":{"1min":0.874,"5min":1.1910000000000001,"15min":3.3879999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.373,"15min":0.33300000000000002},"max":{"1min":0.85199999999999998,"5min":3.9609999999999999,"15min":8.5730000000000004},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.629},"min":{"1min":0.47999999999999998,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":0.79400000000000004,"5min":0.94499999999999995,"15min":3.363},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61199999999999999,"15min":0.66400000000000003},"min":{"1min":0.52800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77500000000000002,"5min":0.93899999999999995,"15min":8.4299999999999997},"last":0.72899999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411583,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299156,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445996,"statfs":{"total":95995035648,"available":95688699904,"internally_reserved":0,"allocated":1781760,"data_stored":545315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.495},"min":{"1min":0.214,"5min":0.20699999999999999,"15min":0.183},"max":{"1min":0.53300000000000003,"5min":1.2909999999999999,"15min":16.170000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.55800000000000005},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":0.49099999999999999,"5min":1.458,"15min":14.858000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34899999999999998,"15min":0.55200000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.499,"5min":0.63500000000000001,"15min":12.512},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.35199999999999998,"15min":0.52400000000000002},"min":{"1min":0.27900000000000003,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":12.875},"last":0.27600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35799999999999998,"15min":0.53900000000000003},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45600000000000002,"5min":1.3069999999999999,"15min":11.612},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41999999999999998,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.22600000000000001},"max":{"1min":0.49299999999999999,"5min":3.484,"15min":11.710000000000001},"last":0.316}]},{"osd":4,"last update":"Sat Feb 10 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53300000000000003,"15min":0.51500000000000001},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.45500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.996,"5min":1.298,"15min":1.298},"last":0.37}]},{"osd":5,"last update":"Sat Feb 10 08:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56899999999999995,"15min":0.70299999999999996},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.28799999999999998},"max":{"1min":0.82499999999999996,"5min":0.871,"15min":10.34},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57199999999999995,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.31900000000000001},"max":{"1min":0.83699999999999997,"5min":1.3380000000000001,"15min":9.6690000000000005},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.58699999999999997,"15min":0.71899999999999997},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":10.372999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.59199999999999997,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.372,"15min":0.35699999999999998},"max":{"1min":0.95899999999999996,"5min":1.321,"15min":9.7420000000000009},"last":0.42899999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.70199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.33900000000000002},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":10.539},"last":0.497},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.59699999999999998,"15min":0.70999999999999996},"min":{"1min":0.47699999999999998,"5min":0.41499999999999998,"15min":0.32500000000000001},"max":{"1min":0.91100000000000003,"5min":1.357,"15min":9.6319999999999997},"last":0.44400000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182651,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299160,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445992,"statfs":{"total":95995035648,"available":95688695808,"internally_reserved":0,"allocated":1982464,"data_stored":640464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.57399999999999995},"min":{"1min":0.216,"5min":0.19900000000000001,"15min":0.184},"max":{"1min":0.45800000000000002,"5min":1.8879999999999999,"15min":23.443000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.40100000000000002,"15min":0.59499999999999997},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":0.58899999999999997,"5min":6.6369999999999996,"15min":11.808},"last":0.377}]},{"osd":1,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.41999999999999998,"15min":0.67400000000000004},"min":{"1min":0.25,"5min":0.215,"15min":0.184},"max":{"1min":0.56599999999999995,"5min":6.5810000000000004,"15min":20.902000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.373,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":2.9950000000000001,"15min":17.343},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.49199999999999999,"15min":0.67800000000000005},"min":{"1min":0.28299999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51600000000000001,"5min":12.664999999999999,"15min":17.201000000000001},"last":0.312},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36499999999999999,"15min":0.63200000000000001},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57599999999999996,"5min":1.206,"15min":23.294},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.36699999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.40600000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.69899999999999995},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":12.579000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.67100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.28799999999999998},"max":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":17.292000000000002},"last":0.54100000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.71699999999999997},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":1.244,"15min":16.091000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.74299999999999999},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.32100000000000001},"max":{"1min":0.67700000000000005,"5min":0.80400000000000005,"15min":17.260999999999999},"last":0.46300000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.70399999999999996},"min":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.35799999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":14.773999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.747},"min":{"1min":0.49199999999999999,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.82699999999999996,"15min":12.606999999999999},"last":0.42099999999999999}]}]},{"osd":4,"up_from":633,"seq":2718714298439,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309216,"kb_used_data":1752,"kb_used_omap":59,"kb_used_meta":307396,"kb_avail":93435936,"statfs":{"total":95995035648,"available":95678398464,"internally_reserved":0,"allocated":1794048,"data_stored":545973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":314774123},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.67600000000000005,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.54000000000000004},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72599999999999998,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70799999999999996,"5min":1.036,"15min":1.036},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69899999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.33800000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65300000000000002,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.436},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80900000000000005,"5min":0.87,"15min":0.87},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50600000000000001,"15min":0.53200000000000003},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.745,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78800000000000003,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.48699999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.34899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57399999999999995,"15min":0.59999999999999998},"min":{"1min":0.40500000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46899999999999997}]}]},{"osd":5,"up_from":37,"seq":158913790193,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307344,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437808,"statfs":{"total":95995035648,"available":95680315392,"internally_reserved":0,"allocated":1777664,"data_stored":537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51500000000000001,"15min":1.054},"min":{"1min":0.31900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.73799999999999999,"5min":1.151,"15min":107.551},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49299999999999999,"15min":0.97499999999999998},"min":{"1min":0.372,"5min":0.29399999999999998,"15min":0.255},"max":{"1min":0.71899999999999997,"5min":2.2589999999999999,"15min":88.792000000000002},"last":0.67500000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53600000000000003,"15min":1.0109999999999999},"min":{"1min":0.36299999999999999,"5min":0.28699999999999998,"15min":0.216},"max":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":107.205},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.88100000000000001},"min":{"1min":0.43099999999999999,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.74399999999999999,"5min":0.90000000000000002,"15min":72.272000000000006},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":1.034},"min":{"1min":0.38600000000000001,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.66600000000000004,"5min":1.4650000000000001,"15min":54.174999999999997},"last":0.626},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":1.073},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.26600000000000001},"max":{"1min":0.752,"5min":1.6559999999999999,"15min":86.183000000000007},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.86299999999999999},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.70899999999999996,"5min":1.0529999999999999,"15min":44.395000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56000000000000005,"15min":1.288},"min":{"1min":0.40200000000000002,"5min":0.35699999999999998,"15min":0.29099999999999998},"max":{"1min":0.79000000000000004,"5min":1.7010000000000001,"15min":114.526},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.48999999999999999},"min":{"1min":0.41399999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66100000000000003,"5min":1.002,"15min":1.002},"last":0.79600000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47999999999999998,"15min":0.73099999999999998},"min":{"1min":0.34499999999999997,"5min":0.26800000000000002,"15min":0.223},"max":{"1min":0.70599999999999996,"5min":1.847,"15min":72.241},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.502,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.23499999999999999},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":22.469999999999999},"last":0.66400000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.72799999999999998},"min":{"1min":0.40799999999999997,"5min":0.28100000000000003,"15min":0.24099999999999999},"max":{"1min":0.73499999999999999,"5min":1.1200000000000001,"15min":66.844999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55300000000000005,"15min":0.755},"min":{"1min":0.40400000000000003,"5min":0.313,"15min":0.253},"max":{"1min":0.71799999999999997,"5min":1.8720000000000001,"15min":72.257000000000005},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:52:14.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:13.305815+0000 osd.3 (osd.3) 453 : cluster [DBG] 16.7 scrub starts 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:13.307921+0000 osd.3 (osd.3) 454 : cluster [DBG] 16.7 scrub ok 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:13.620076+0000 mgr.y (mgr.14553) 758 : cluster [DBG] pgmap v1335: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:13.860270+0000 osd.2 (osd.2) 389 : cluster [DBG] 16.1 scrub starts 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:13.861845+0000 osd.2 (osd.2) 390 : cluster [DBG] 16.1 scrub ok 2024-02-10T08:52:15.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:13.879044+0000 mgr.y (mgr.14553) 759 : audit [DBG] from='client.29085 v1:172.21.15.107:0/1923939337' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:15.260 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:52:15] "GET /metrics HTTP/1.1" 200 186420 "" "Prometheus/2.33.4" 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:14.503469+0000 osd.1 (osd.1) 422 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:14.507574+0000 osd.1 (osd.1) 423 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:14.875150+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:14.878827+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:15.634067+0000 osd.5 (osd.5) 418 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:52:15.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:15.637539+0000 osd.5 (osd.5) 419 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:14.503469+0000 osd.1 (osd.1) 422 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:14.507574+0000 osd.1 (osd.1) 423 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:14.875150+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:14.878827+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:15.634067+0000 osd.5 (osd.5) 418 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:52:15.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:15 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:15.637539+0000 osd.5 (osd.5) 419 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:14.503469+0000 osd.1 (osd.1) 422 : cluster [DBG] 16.13 scrub starts 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:14.507574+0000 osd.1 (osd.1) 423 : cluster [DBG] 16.13 scrub ok 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:14.875150+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.18 scrub starts 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:14.878827+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.18 scrub ok 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:15.634067+0000 osd.5 (osd.5) 418 : cluster [DBG] 4.15 scrub starts 2024-02-10T08:52:16.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:15.637539+0000 osd.5 (osd.5) 419 : cluster [DBG] 4.15 scrub ok 2024-02-10T08:52:16.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:15.620556+0000 mgr.y (mgr.14553) 760 : cluster [DBG] pgmap v1336: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:52:16.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:16 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:15.620556+0000 mgr.y (mgr.14553) 760 : cluster [DBG] pgmap v1336: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:52:17.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:15.620556+0000 mgr.y (mgr.14553) 760 : cluster [DBG] pgmap v1336: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.118+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3105595849 >> v1:172.21.15.107:6790/0 conn(0x7f1f6c0f9550 legacy=0x7f1f6c102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3105595849 shutdown_connections 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3105595849 >> 172.21.15.107:0/3105595849 conn(0x7f1f6c0f2510 msgr2=0x7f1f6c0f4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3105595849 shutdown_connections 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3105595849 wait complete. 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 librados: starting msgr at 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 librados: starting objecter 2024-02-10T08:52:17.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 Processor -- start 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 -- start start 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 librados: setting wanted keys 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.119+0000 7f1f73b6b700 1 librados: calling monclient init 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f73b6b700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f6c1173a0 con 0x7f1f6c104330 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f73b6b700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f6c118580 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f73b6b700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f6c119780 con 0x7f1f6c0f9550 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f71106700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1f6c0f9550 0x7f1f6c0ffcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34718/0 (socket says 172.21.15.107:34718) 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f71106700 1 -- 172.21.15.107:0/3773392756 learned_addr learned my addr 172.21.15.107:0/3773392756 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter ms_handle_connect 0x7f1f6c0f9550 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter ms_handle_connect 0x7f1f6c0f8a60 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133201476 0 0) 0x7f1f6c119780 con 0x7f1f6c0f9550 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f580038e0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter ms_handle_connect 0x7f1f6c104330 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635540237 0 0) 0x7f1f6c118580 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c119780 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104293753 0 0) 0x7f1f6c1173a0 con 0x7f1f6c104330 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c118580 con 0x7f1f6c104330 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886778797 0 0) 0x7f1f580038e0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f6c1173a0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1f5c0015d0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446450611 0 0) 0x7f1f6c119780 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f580038e0 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.120+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1f680018c0 con 0x7f1f6c0f8a60 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333185172 0 0) 0x7f1f6c118580 con 0x7f1f6c104330 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f6c119780 con 0x7f1f6c104330 2024-02-10T08:52:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1f64001a20 con 0x7f1f6c104330 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 900334543 0 0) 0x7f1f6c1173a0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 >> v1:172.21.15.195:6789/0 conn(0x7f1f6c0f8a60 legacy=0x7f1f6c0ff5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 >> v1:172.21.15.107:6789/0 conn(0x7f1f6c104330 legacy=0x7f1f6c115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c11a980 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c118770 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f73b6b700 10 client.54689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c118be0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f73b6b700 1 librados: init done 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f1f5c001a90 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.121+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f1f5c0038a0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f50004f20 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f1f5c003a00 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(839..839 src has 787..839) v4 ==== 7473+0+0 (unknown 2033404990 0 0) 0x7f1f5c078c70 con 0x7f1f6c0f9550 2024-02-10T08:52:17.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f62ffd700 10 client.54689.objecter ms_dispatch 0x7f1f6c001550 osd_map(839..839 src has 787..839) v4 2024-02-10T08:52:17.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f62ffd700 3 client.54689.objecter handle_osd_map got epochs [839,839] > 0 2024-02-10T08:52:17.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.122+0000 7f1f62ffd700 3 client.54689.objecter handle_osd_map decoding full epoch 839 2024-02-10T08:52:17.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.123+0000 7f1f62ffd700 20 client.54689.objecter dump_active .. 0 homeless 2024-02-10T08:52:17.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.123+0000 7f1f62ffd700 10 client.54689.objecter ms_handle_connect 0x7f1f58068180 2024-02-10T08:52:17.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.127+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1f5c04c3b0 con 0x7f1f6c0f9550 2024-02-10T08:52:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.284+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 --> v1:172.21.15.107:6800/2634842009 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f50000b90 con 0x7f1f58068180 2024-02-10T08:52:17.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.296+0000 7f1f62ffd700 1 -- 172.21.15.107:0/3773392756 <== mgr.14553 v1:172.21.15.107:6800/2634842009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+410328 (unknown 2965378022 0 1319299528) 0x7f1f50000b90 con 0x7f1f58068180 2024-02-10T08:52:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 10 librados: watch_flush enter 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 10 librados: watch_flush exit 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 20 client.54689.objecter shutdown clearing up homeless session... 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 10 client.54689.objecter successfully canceled tick 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 >> v1:172.21.15.107:6800/2634842009 conn(0x7f1f58068180 legacy=0x7f1f5806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 >> v1:172.21.15.107:6790/0 conn(0x7f1f6c0f9550 legacy=0x7f1f6c0ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 shutdown_connections 2024-02-10T08:52:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 >> 172.21.15.107:0/3773392756 conn(0x7f1f6c0f2510 msgr2=0x7f1f6c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 shutdown_connections 2024-02-10T08:52:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 -- 172.21.15.107:0/3773392756 wait complete. 2024-02-10T08:52:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:17.299+0000 7f1f73b6b700 1 librados: shutdown 2024-02-10T08:52:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-02-10T08:52:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:17.268721+0000 osd.3 (osd.3) 455 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:52:17.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:17.269835+0000 osd.3 (osd.3) 456 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:52:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:17.268721+0000 osd.3 (osd.3) 455 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:52:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:17 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:17.269835+0000 osd.3 (osd.3) 456 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:52:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:17.268721+0000 osd.3 (osd.3) 455 : cluster [DBG] 2.12 scrub starts 2024-02-10T08:52:18.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:17.269835+0000 osd.3 (osd.3) 456 : cluster [DBG] 2.12 scrub ok 2024-02-10T08:52:18.093 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1336,"stamp":"2024-02-10T08:52:15.620247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1487,"num_read_kb":1239,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":13772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":462,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2436092,"kb_used_data":14140,"kb_used_omap":59,"kb_used_meta":2420676,"kb_avail":747525124,"statfs":{"total":767960285184,"available":765465726976,"internally_reserved":0,"allocated":14479360,"data_stored":4437191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":2478772843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002606"},"pg_stats":[{"pgid":"5.9","version":"829'19","reported_seq":572,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934013+0000","last_change":"2024-02-10T08:51:07.741210+0000","last_active":"2024-02-10T08:52:09.934013+0000","last_peered":"2024-02-10T08:52:09.934013+0000","last_clean":"2024-02-10T08:52:09.934013+0000","last_became_active":"2024-02-10T08:46:29.217648+0000","last_became_peered":"2024-02-10T08:46:29.217648+0000","last_unstale":"2024-02-10T08:52:09.934013+0000","last_undegraded":"2024-02-10T08:52:09.934013+0000","last_fullsized":"2024-02-10T08:52:09.934013+0000","mapping_epoch":633,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'19","last_scrub_stamp":"2024-02-10T08:51:07.741135+0000","last_deep_scrub":"829'19","last_deep_scrub_stamp":"2024-02-10T08:51:07.741135+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.741135+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'34","reported_seq":914,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332554+0000","last_change":"2024-02-10T08:51:22.676912+0000","last_active":"2024-02-10T08:52:10.332554+0000","last_peered":"2024-02-10T08:52:10.332554+0000","last_clean":"2024-02-10T08:52:10.332554+0000","last_became_active":"2024-02-10T08:43:15.114185+0000","last_became_peered":"2024-02-10T08:43:15.114185+0000","last_unstale":"2024-02-10T08:52:10.332554+0000","last_undegraded":"2024-02-10T08:52:10.332554+0000","last_fullsized":"2024-02-10T08:52:10.332554+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:51:22.676835+0000","last_deep_scrub":"80'22","last_deep_scrub_stamp":"2024-02-10T08:35:05.510644+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.676835+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":848,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703834+0000","last_change":"2024-02-10T08:51:17.756145+0000","last_active":"2024-02-10T08:52:09.703834+0000","last_peered":"2024-02-10T08:52:09.703834+0000","last_clean":"2024-02-10T08:52:09.703834+0000","last_became_active":"2024-02-10T08:43:15.119014+0000","last_became_peered":"2024-02-10T08:43:15.119014+0000","last_unstale":"2024-02-10T08:52:09.703834+0000","last_undegraded":"2024-02-10T08:52:09.703834+0000","last_fullsized":"2024-02-10T08:52:09.703834+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:17.756068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:37:19.833287+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.756068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":833,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332946+0000","last_change":"2024-02-10T08:51:17.664135+0000","last_active":"2024-02-10T08:52:10.332946+0000","last_peered":"2024-02-10T08:52:10.332946+0000","last_clean":"2024-02-10T08:52:10.332946+0000","last_became_active":"2024-02-10T08:46:15.332234+0000","last_became_peered":"2024-02-10T08:46:15.332234+0000","last_unstale":"2024-02-10T08:52:10.332946+0000","last_undegraded":"2024-02-10T08:52:10.332946+0000","last_fullsized":"2024-02-10T08:52:10.332946+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:17.664025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:17.664025+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.664025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":794,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.884774+0000","last_change":"2024-02-10T08:52:10.884774+0000","last_active":"2024-02-10T08:52:10.884774+0000","last_peered":"2024-02-10T08:52:10.884774+0000","last_clean":"2024-02-10T08:52:10.884774+0000","last_became_active":"2024-02-10T08:47:49.237291+0000","last_became_peered":"2024-02-10T08:47:49.237291+0000","last_unstale":"2024-02-10T08:52:10.884774+0000","last_undegraded":"2024-02-10T08:52:10.884774+0000","last_fullsized":"2024-02-10T08:52:10.884774+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:10.884679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:10.884679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'15","reported_seq":296,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925313+0000","last_change":"2024-02-10T08:51:54.282145+0000","last_active":"2024-02-10T08:52:09.925313+0000","last_peered":"2024-02-10T08:52:09.925313+0000","last_clean":"2024-02-10T08:52:09.925313+0000","last_became_active":"2024-02-10T08:47:49.239592+0000","last_became_peered":"2024-02-10T08:47:49.239592+0000","last_unstale":"2024-02-10T08:52:09.925313+0000","last_undegraded":"2024-02-10T08:52:09.925313+0000","last_fullsized":"2024-02-10T08:52:09.925313+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:51:54.282038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.282038+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701359+0000","last_change":"2024-02-10T08:51:34.470562+0000","last_active":"2024-02-10T08:52:09.701359+0000","last_peered":"2024-02-10T08:52:09.701359+0000","last_clean":"2024-02-10T08:52:09.701359+0000","last_became_active":"2024-02-10T08:47:49.241738+0000","last_became_peered":"2024-02-10T08:47:49.241738+0000","last_unstale":"2024-02-10T08:52:09.701359+0000","last_undegraded":"2024-02-10T08:52:09.701359+0000","last_fullsized":"2024-02-10T08:52:09.701359+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:34.470498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.470498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":859,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703411+0000","last_change":"2024-02-10T08:51:47.010221+0000","last_active":"2024-02-10T08:52:09.703411+0000","last_peered":"2024-02-10T08:52:09.703411+0000","last_clean":"2024-02-10T08:52:09.703411+0000","last_became_active":"2024-02-10T08:46:29.209171+0000","last_became_peered":"2024-02-10T08:46:29.209171+0000","last_unstale":"2024-02-10T08:52:09.703411+0000","last_undegraded":"2024-02-10T08:52:09.703411+0000","last_fullsized":"2024-02-10T08:52:09.703411+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.010152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.010152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":232,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925236+0000","last_change":"2024-02-10T08:51:03.535754+0000","last_active":"2024-02-10T08:52:09.925236+0000","last_peered":"2024-02-10T08:52:09.925236+0000","last_clean":"2024-02-10T08:52:09.925236+0000","last_became_active":"2024-02-10T08:46:15.372037+0000","last_became_peered":"2024-02-10T08:46:15.372037+0000","last_unstale":"2024-02-10T08:52:09.925236+0000","last_undegraded":"2024-02-10T08:52:09.925236+0000","last_fullsized":"2024-02-10T08:52:09.925236+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:03.535646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:03.535646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":782,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333733+0000","last_change":"2024-02-10T08:51:31.775340+0000","last_active":"2024-02-10T08:52:10.333733+0000","last_peered":"2024-02-10T08:52:10.333733+0000","last_clean":"2024-02-10T08:52:10.333733+0000","last_became_active":"2024-02-10T08:43:15.126169+0000","last_became_peered":"2024-02-10T08:43:15.126169+0000","last_unstale":"2024-02-10T08:52:10.333733+0000","last_undegraded":"2024-02-10T08:52:10.333733+0000","last_fullsized":"2024-02-10T08:52:10.333733+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:31.775249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:31.775249+0000","last_clean_scrub_stamp":"2024-02-10T08:51:31.775249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"654'44","reported_seq":1008,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335384+0000","last_change":"2024-02-10T08:51:16.033467+0000","last_active":"2024-02-10T08:52:10.335384+0000","last_peered":"2024-02-10T08:52:10.335384+0000","last_clean":"2024-02-10T08:52:10.335384+0000","last_became_active":"2024-02-10T08:47:49.242002+0000","last_became_peered":"2024-02-10T08:47:49.242002+0000","last_unstale":"2024-02-10T08:52:10.335384+0000","last_undegraded":"2024-02-10T08:52:10.335384+0000","last_fullsized":"2024-02-10T08:52:10.335384+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'44","last_scrub_stamp":"2024-02-10T08:51:16.033361+0000","last_deep_scrub":"654'44","last_deep_scrub_stamp":"2024-02-10T08:51:16.033361+0000","last_clean_scrub_stamp":"2024-02-10T08:51:16.033361+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":846,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703856+0000","last_change":"2024-02-10T08:51:16.790256+0000","last_active":"2024-02-10T08:52:09.703856+0000","last_peered":"2024-02-10T08:52:09.703856+0000","last_clean":"2024-02-10T08:52:09.703856+0000","last_became_active":"2024-02-10T08:46:15.373933+0000","last_became_peered":"2024-02-10T08:46:15.373933+0000","last_unstale":"2024-02-10T08:52:09.703856+0000","last_undegraded":"2024-02-10T08:52:09.703856+0000","last_fullsized":"2024-02-10T08:52:09.703856+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:16.790188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.313935+0000","last_clean_scrub_stamp":"2024-02-10T08:51:16.790188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":885,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703622+0000","last_change":"2024-02-10T08:51:59.921088+0000","last_active":"2024-02-10T08:52:09.703622+0000","last_peered":"2024-02-10T08:52:09.703622+0000","last_clean":"2024-02-10T08:52:09.703622+0000","last_became_active":"2024-02-10T08:46:15.367325+0000","last_became_peered":"2024-02-10T08:46:15.367325+0000","last_unstale":"2024-02-10T08:52:09.703622+0000","last_undegraded":"2024-02-10T08:52:09.703622+0000","last_fullsized":"2024-02-10T08:52:09.703622+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.921000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:59.921000+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.921000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":400,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933622+0000","last_change":"2024-02-10T08:51:28.921089+0000","last_active":"2024-02-10T08:52:09.933622+0000","last_peered":"2024-02-10T08:52:09.933622+0000","last_clean":"2024-02-10T08:52:09.933622+0000","last_became_active":"2024-02-10T08:43:15.126623+0000","last_became_peered":"2024-02-10T08:43:15.126623+0000","last_unstale":"2024-02-10T08:52:09.933622+0000","last_undegraded":"2024-02-10T08:52:09.933622+0000","last_fullsized":"2024-02-10T08:52:09.933622+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:28.921014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:46.484656+0000","last_clean_scrub_stamp":"2024-02-10T08:51:28.921014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934383+0000","last_change":"2024-02-10T08:51:05.742473+0000","last_active":"2024-02-10T08:52:09.934383+0000","last_peered":"2024-02-10T08:52:09.934383+0000","last_clean":"2024-02-10T08:52:09.934383+0000","last_became_active":"2024-02-10T08:46:29.215756+0000","last_became_peered":"2024-02-10T08:46:29.215756+0000","last_unstale":"2024-02-10T08:52:09.934383+0000","last_undegraded":"2024-02-10T08:52:09.934383+0000","last_fullsized":"2024-02-10T08:52:09.934383+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:05.742394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:05.742394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"473'31","reported_seq":782,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701816+0000","last_change":"2024-02-10T08:51:11.895780+0000","last_active":"2024-02-10T08:52:09.701816+0000","last_peered":"2024-02-10T08:52:09.701816+0000","last_clean":"2024-02-10T08:52:09.701816+0000","last_became_active":"2024-02-10T08:47:49.241747+0000","last_became_peered":"2024-02-10T08:47:49.241747+0000","last_unstale":"2024-02-10T08:52:09.701816+0000","last_undegraded":"2024-02-10T08:52:09.701816+0000","last_fullsized":"2024-02-10T08:52:09.701816+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'31","last_scrub_stamp":"2024-02-10T08:51:11.895697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:11.895697+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333050+0000","last_change":"2024-02-10T08:51:33.186070+0000","last_active":"2024-02-10T08:52:10.333050+0000","last_peered":"2024-02-10T08:52:10.333050+0000","last_clean":"2024-02-10T08:52:10.333050+0000","last_became_active":"2024-02-10T08:47:49.224836+0000","last_became_peered":"2024-02-10T08:47:49.224836+0000","last_unstale":"2024-02-10T08:52:10.333050+0000","last_undegraded":"2024-02-10T08:52:10.333050+0000","last_fullsized":"2024-02-10T08:52:10.333050+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:33.185993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:33.297424+0000","last_clean_scrub_stamp":"2024-02-10T08:51:33.185993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701596+0000","last_change":"2024-02-10T08:51:35.494575+0000","last_active":"2024-02-10T08:52:09.701596+0000","last_peered":"2024-02-10T08:52:09.701596+0000","last_clean":"2024-02-10T08:52:09.701596+0000","last_became_active":"2024-02-10T08:47:49.256294+0000","last_became_peered":"2024-02-10T08:47:49.256294+0000","last_unstale":"2024-02-10T08:52:09.701596+0000","last_undegraded":"2024-02-10T08:52:09.701596+0000","last_fullsized":"2024-02-10T08:52:09.701596+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.494511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.494511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":837,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.704070+0000","last_change":"2024-02-10T08:51:49.996088+0000","last_active":"2024-02-10T08:52:09.704070+0000","last_peered":"2024-02-10T08:52:09.704070+0000","last_clean":"2024-02-10T08:52:09.704070+0000","last_became_active":"2024-02-10T08:46:29.209045+0000","last_became_peered":"2024-02-10T08:46:29.209045+0000","last_unstale":"2024-02-10T08:52:09.704070+0000","last_undegraded":"2024-02-10T08:52:09.704070+0000","last_fullsized":"2024-02-10T08:52:09.704070+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.995985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.691633+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.995985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":642,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701306+0000","last_change":"2024-02-10T08:52:05.835916+0000","last_active":"2024-02-10T08:52:09.701306+0000","last_peered":"2024-02-10T08:52:09.701306+0000","last_clean":"2024-02-10T08:52:09.701306+0000","last_became_active":"2024-02-10T08:47:49.241234+0000","last_became_peered":"2024-02-10T08:47:49.241234+0000","last_unstale":"2024-02-10T08:52:09.701306+0000","last_undegraded":"2024-02-10T08:52:09.701306+0000","last_fullsized":"2024-02-10T08:52:09.701306+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.835835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:05.835835+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.835835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"473'20","reported_seq":706,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701512+0000","last_change":"2024-02-10T08:51:45.873280+0000","last_active":"2024-02-10T08:52:09.701512+0000","last_peered":"2024-02-10T08:52:09.701512+0000","last_clean":"2024-02-10T08:52:09.701512+0000","last_became_active":"2024-02-10T08:46:29.218647+0000","last_became_peered":"2024-02-10T08:46:29.218647+0000","last_unstale":"2024-02-10T08:52:09.701512+0000","last_undegraded":"2024-02-10T08:52:09.701512+0000","last_fullsized":"2024-02-10T08:52:09.701512+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:45.873199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:45.873199+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701866+0000","last_change":"2024-02-10T08:51:46.491852+0000","last_active":"2024-02-10T08:52:09.701866+0000","last_peered":"2024-02-10T08:52:09.701866+0000","last_clean":"2024-02-10T08:52:09.701866+0000","last_became_active":"2024-02-10T08:47:49.235871+0000","last_became_peered":"2024-02-10T08:47:49.235871+0000","last_unstale":"2024-02-10T08:52:09.701866+0000","last_undegraded":"2024-02-10T08:52:09.701866+0000","last_fullsized":"2024-02-10T08:52:09.701866+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:46.491782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:46.491782+0000","last_clean_scrub_stamp":"2024-02-10T08:51:46.491782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"583'17","reported_seq":1138,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703741+0000","last_change":"2024-02-10T08:52:02.900985+0000","last_active":"2024-02-10T08:52:09.703741+0000","last_peered":"2024-02-10T08:52:09.703741+0000","last_clean":"2024-02-10T08:52:09.703741+0000","last_became_active":"2024-02-10T08:43:21.061080+0000","last_became_peered":"2024-02-10T08:43:21.061080+0000","last_unstale":"2024-02-10T08:52:09.703741+0000","last_undegraded":"2024-02-10T08:52:09.703741+0000","last_fullsized":"2024-02-10T08:52:09.703741+0000","mapping_epoch":468,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"583'17","last_scrub_stamp":"2024-02-10T08:52:02.900920+0000","last_deep_scrub":"583'17","last_deep_scrub_stamp":"2024-02-10T08:50:46.833593+0000","last_clean_scrub_stamp":"2024-02-10T08:52:02.900920+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":666,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701664+0000","last_change":"2024-02-10T08:51:37.771083+0000","last_active":"2024-02-10T08:52:09.701664+0000","last_peered":"2024-02-10T08:52:09.701664+0000","last_clean":"2024-02-10T08:52:09.701664+0000","last_became_active":"2024-02-10T08:47:49.242386+0000","last_became_peered":"2024-02-10T08:47:49.242386+0000","last_unstale":"2024-02-10T08:52:09.701664+0000","last_undegraded":"2024-02-10T08:52:09.701664+0000","last_fullsized":"2024-02-10T08:52:09.701664+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:37.770953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:37.770953+0000","last_clean_scrub_stamp":"2024-02-10T08:51:37.770953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":235,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925597+0000","last_change":"2024-02-10T08:50:39.656525+0000","last_active":"2024-02-10T08:52:09.925597+0000","last_peered":"2024-02-10T08:52:09.925597+0000","last_clean":"2024-02-10T08:52:09.925597+0000","last_became_active":"2024-02-10T08:46:15.365648+0000","last_became_peered":"2024-02-10T08:46:15.365648+0000","last_unstale":"2024-02-10T08:52:09.925597+0000","last_undegraded":"2024-02-10T08:52:09.925597+0000","last_fullsized":"2024-02-10T08:52:09.925597+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:39.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:42.008755+0000","last_clean_scrub_stamp":"2024-02-10T08:50:39.656444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"833'34","reported_seq":57,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703895+0000","last_change":"2024-02-10T08:51:27.814091+0000","last_active":"2024-02-10T08:52:09.703895+0000","last_peered":"2024-02-10T08:52:09.703895+0000","last_clean":"2024-02-10T08:52:09.703895+0000","last_became_active":"2024-02-10T08:50:42.864182+0000","last_became_peered":"2024-02-10T08:50:42.864182+0000","last_unstale":"2024-02-10T08:52:09.703895+0000","last_undegraded":"2024-02-10T08:52:09.703895+0000","last_fullsized":"2024-02-10T08:52:09.703895+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'34","last_scrub_stamp":"2024-02-10T08:51:27.814022+0000","last_deep_scrub":"833'34","last_deep_scrub_stamp":"2024-02-10T08:51:27.814022+0000","last_clean_scrub_stamp":"2024-02-10T08:51:27.814022+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"56'1","reported_seq":159,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701189+0000","last_change":"2024-02-10T08:51:43.530660+0000","last_active":"2024-02-10T08:52:09.701189+0000","last_peered":"2024-02-10T08:52:09.701189+0000","last_clean":"2024-02-10T08:52:09.701189+0000","last_became_active":"2024-02-10T08:47:49.233892+0000","last_became_peered":"2024-02-10T08:47:49.233892+0000","last_unstale":"2024-02-10T08:52:09.701189+0000","last_undegraded":"2024-02-10T08:52:09.701189+0000","last_fullsized":"2024-02-10T08:52:09.701189+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:51:43.530546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:43.530546+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"636'22","reported_seq":943,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.834258+0000","last_change":"2024-02-10T08:50:51.901411+0000","last_active":"2024-02-10T08:52:10.834258+0000","last_peered":"2024-02-10T08:52:10.834258+0000","last_clean":"2024-02-10T08:52:10.834258+0000","last_became_active":"2024-02-10T08:46:30.199529+0000","last_became_peered":"2024-02-10T08:46:30.199529+0000","last_unstale":"2024-02-10T08:52:10.834258+0000","last_undegraded":"2024-02-10T08:52:10.834258+0000","last_fullsized":"2024-02-10T08:52:10.834258+0000","mapping_epoch":634,"log_start":"63'8","ondisk_log_start":"63'8","created":59,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'22","last_scrub_stamp":"2024-02-10T08:50:51.901321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:50:51.901321+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":851,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333442+0000","last_change":"2024-02-10T08:52:06.633522+0000","last_active":"2024-02-10T08:52:10.333442+0000","last_peered":"2024-02-10T08:52:10.333442+0000","last_clean":"2024-02-10T08:52:10.333442+0000","last_became_active":"2024-02-10T08:47:49.237861+0000","last_became_peered":"2024-02-10T08:47:49.237861+0000","last_unstale":"2024-02-10T08:52:10.333442+0000","last_undegraded":"2024-02-10T08:52:10.333442+0000","last_fullsized":"2024-02-10T08:52:10.333442+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.633419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:45:23.003138+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.633419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":800,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333064+0000","last_change":"2024-02-10T08:52:03.545779+0000","last_active":"2024-02-10T08:52:10.333064+0000","last_peered":"2024-02-10T08:52:10.333064+0000","last_clean":"2024-02-10T08:52:10.333064+0000","last_became_active":"2024-02-10T08:47:49.237856+0000","last_became_peered":"2024-02-10T08:47:49.237856+0000","last_unstale":"2024-02-10T08:52:10.333064+0000","last_undegraded":"2024-02-10T08:52:10.333064+0000","last_fullsized":"2024-02-10T08:52:10.333064+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:03.545690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:03.545690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"473'19","reported_seq":744,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701863+0000","last_change":"2024-02-10T08:50:39.075746+0000","last_active":"2024-02-10T08:52:09.701863+0000","last_peered":"2024-02-10T08:52:09.701863+0000","last_clean":"2024-02-10T08:52:09.701863+0000","last_became_active":"2024-02-10T08:47:49.242616+0000","last_became_peered":"2024-02-10T08:47:49.242616+0000","last_unstale":"2024-02-10T08:52:09.701863+0000","last_undegraded":"2024-02-10T08:52:09.701863+0000","last_fullsized":"2024-02-10T08:52:09.701863+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:39.075652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:39.075652+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":433,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.935780+0000","last_change":"2024-02-10T08:52:09.935732+0000","last_active":"2024-02-10T08:52:09.935780+0000","last_peered":"2024-02-10T08:52:09.935780+0000","last_clean":"2024-02-10T08:52:09.935780+0000","last_became_active":"2024-02-10T08:47:49.252422+0000","last_became_peered":"2024-02-10T08:47:49.252422+0000","last_unstale":"2024-02-10T08:52:09.935780+0000","last_undegraded":"2024-02-10T08:52:09.935780+0000","last_fullsized":"2024-02-10T08:52:09.935780+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.935655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:09.935655+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.935655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"636'12","reported_seq":1036,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333616+0000","last_change":"2024-02-10T08:51:44.706204+0000","last_active":"2024-02-10T08:52:10.333616+0000","last_peered":"2024-02-10T08:52:10.333616+0000","last_clean":"2024-02-10T08:52:10.333616+0000","last_became_active":"2024-02-10T08:46:29.210237+0000","last_became_peered":"2024-02-10T08:46:29.210237+0000","last_unstale":"2024-02-10T08:52:10.333616+0000","last_undegraded":"2024-02-10T08:52:10.333616+0000","last_fullsized":"2024-02-10T08:52:10.333616+0000","mapping_epoch":633,"log_start":"74'9","ondisk_log_start":"74'9","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'12","last_scrub_stamp":"2024-02-10T08:51:44.706124+0000","last_deep_scrub":"636'12","last_deep_scrub_stamp":"2024-02-10T08:48:51.038025+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.706124+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":818,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333369+0000","last_change":"2024-02-10T08:51:48.279152+0000","last_active":"2024-02-10T08:52:10.333369+0000","last_peered":"2024-02-10T08:52:10.333369+0000","last_clean":"2024-02-10T08:52:10.333369+0000","last_became_active":"2024-02-10T08:33:24.126224+0000","last_became_peered":"2024-02-10T08:33:24.126224+0000","last_unstale":"2024-02-10T08:52:10.333369+0000","last_undegraded":"2024-02-10T08:52:10.333369+0000","last_fullsized":"2024-02-10T08:52:10.333369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:48.279085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:40.893261+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.279085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":762,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332994+0000","last_change":"2024-02-10T08:51:12.243000+0000","last_active":"2024-02-10T08:52:10.332994+0000","last_peered":"2024-02-10T08:52:10.332994+0000","last_clean":"2024-02-10T08:52:10.332994+0000","last_became_active":"2024-02-10T08:36:37.133318+0000","last_became_peered":"2024-02-10T08:36:37.133318+0000","last_unstale":"2024-02-10T08:52:10.332994+0000","last_undegraded":"2024-02-10T08:52:10.332994+0000","last_fullsized":"2024-02-10T08:52:10.332994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:12.242874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:12.242874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"711'14","reported_seq":1098,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332819+0000","last_change":"2024-02-10T08:51:09.218426+0000","last_active":"2024-02-10T08:52:10.332819+0000","last_peered":"2024-02-10T08:52:10.332819+0000","last_clean":"2024-02-10T08:52:10.332819+0000","last_became_active":"2024-02-10T08:47:49.225459+0000","last_became_peered":"2024-02-10T08:47:49.225459+0000","last_unstale":"2024-02-10T08:52:10.332819+0000","last_undegraded":"2024-02-10T08:52:10.332819+0000","last_fullsized":"2024-02-10T08:52:10.332819+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'14","last_scrub_stamp":"2024-02-10T08:51:09.218347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:09.218347+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"473'28","reported_seq":720,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701095+0000","last_change":"2024-02-10T08:51:00.932182+0000","last_active":"2024-02-10T08:52:09.701095+0000","last_peered":"2024-02-10T08:52:09.701095+0000","last_clean":"2024-02-10T08:52:09.701095+0000","last_became_active":"2024-02-10T08:46:30.198673+0000","last_became_peered":"2024-02-10T08:46:30.198673+0000","last_unstale":"2024-02-10T08:52:09.701095+0000","last_undegraded":"2024-02-10T08:52:09.701095+0000","last_fullsized":"2024-02-10T08:52:09.701095+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-10T08:51:00.932102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:00.932102+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":857,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335897+0000","last_change":"2024-02-10T08:51:39.012269+0000","last_active":"2024-02-10T08:52:10.335897+0000","last_peered":"2024-02-10T08:52:10.335897+0000","last_clean":"2024-02-10T08:52:10.335897+0000","last_became_active":"2024-02-10T08:43:15.137215+0000","last_became_peered":"2024-02-10T08:43:15.137215+0000","last_unstale":"2024-02-10T08:52:10.335897+0000","last_undegraded":"2024-02-10T08:52:10.335897+0000","last_fullsized":"2024-02-10T08:52:10.335897+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:39.012192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:21.617907+0000","last_clean_scrub_stamp":"2024-02-10T08:51:39.012192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933910+0000","last_change":"2024-02-10T08:52:04.997651+0000","last_active":"2024-02-10T08:52:09.933910+0000","last_peered":"2024-02-10T08:52:09.933910+0000","last_clean":"2024-02-10T08:52:09.933910+0000","last_became_active":"2024-02-10T08:43:15.133871+0000","last_became_peered":"2024-02-10T08:43:15.133871+0000","last_unstale":"2024-02-10T08:52:09.933910+0000","last_undegraded":"2024-02-10T08:52:09.933910+0000","last_fullsized":"2024-02-10T08:52:09.933910+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.997567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.997567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":255,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925405+0000","last_change":"2024-02-10T08:51:41.282955+0000","last_active":"2024-02-10T08:52:09.925405+0000","last_peered":"2024-02-10T08:52:09.925405+0000","last_clean":"2024-02-10T08:52:09.925405+0000","last_became_active":"2024-02-10T08:47:49.241538+0000","last_became_peered":"2024-02-10T08:47:49.241538+0000","last_unstale":"2024-02-10T08:52:09.925405+0000","last_undegraded":"2024-02-10T08:52:09.925405+0000","last_fullsized":"2024-02-10T08:52:09.925405+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:41.282896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:41.282896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":685,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.700982+0000","last_change":"2024-02-10T08:51:47.942559+0000","last_active":"2024-02-10T08:52:09.700982+0000","last_peered":"2024-02-10T08:52:09.700982+0000","last_clean":"2024-02-10T08:52:09.700982+0000","last_became_active":"2024-02-10T08:46:29.214403+0000","last_became_peered":"2024-02-10T08:46:29.214403+0000","last_unstale":"2024-02-10T08:52:09.700982+0000","last_undegraded":"2024-02-10T08:52:09.700982+0000","last_fullsized":"2024-02-10T08:52:09.700982+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.942478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:34.622869+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.942478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"473'38","reported_seq":1009,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332804+0000","last_change":"2024-02-10T08:51:51.296797+0000","last_active":"2024-02-10T08:52:10.332804+0000","last_peered":"2024-02-10T08:52:10.332804+0000","last_clean":"2024-02-10T08:52:10.332804+0000","last_became_active":"2024-02-10T08:46:15.352008+0000","last_became_peered":"2024-02-10T08:46:15.352008+0000","last_unstale":"2024-02-10T08:52:10.332804+0000","last_undegraded":"2024-02-10T08:52:10.332804+0000","last_fullsized":"2024-02-10T08:52:10.332804+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'38","last_scrub_stamp":"2024-02-10T08:51:51.296693+0000","last_deep_scrub":"473'38","last_deep_scrub_stamp":"2024-02-10T08:51:51.296693+0000","last_clean_scrub_stamp":"2024-02-10T08:51:51.296693+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"56'1","reported_seq":159,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.700915+0000","last_change":"2024-02-10T08:51:15.503458+0000","last_active":"2024-02-10T08:52:09.700915+0000","last_peered":"2024-02-10T08:52:09.700915+0000","last_clean":"2024-02-10T08:52:09.700915+0000","last_became_active":"2024-02-10T08:47:49.244603+0000","last_became_peered":"2024-02-10T08:47:49.244603+0000","last_unstale":"2024-02-10T08:52:09.700915+0000","last_undegraded":"2024-02-10T08:52:09.700915+0000","last_fullsized":"2024-02-10T08:52:09.700915+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'1","last_scrub_stamp":"2024-02-10T08:51:15.503393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.503393+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":653,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701104+0000","last_change":"2024-02-10T08:52:03.836543+0000","last_active":"2024-02-10T08:52:09.701104+0000","last_peered":"2024-02-10T08:52:09.701104+0000","last_clean":"2024-02-10T08:52:09.701104+0000","last_became_active":"2024-02-10T08:46:29.210259+0000","last_became_peered":"2024-02-10T08:46:29.210259+0000","last_unstale":"2024-02-10T08:52:09.701104+0000","last_undegraded":"2024-02-10T08:52:09.701104+0000","last_fullsized":"2024-02-10T08:52:09.701104+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:03.836454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:03.836454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":255,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925414+0000","last_change":"2024-02-10T08:52:01.331511+0000","last_active":"2024-02-10T08:52:09.925414+0000","last_peered":"2024-02-10T08:52:09.925414+0000","last_clean":"2024-02-10T08:52:09.925414+0000","last_became_active":"2024-02-10T08:47:49.240596+0000","last_became_peered":"2024-02-10T08:47:49.240596+0000","last_unstale":"2024-02-10T08:52:09.925414+0000","last_undegraded":"2024-02-10T08:52:09.925414+0000","last_fullsized":"2024-02-10T08:52:09.925414+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:01.331444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:01.331444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":844,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335414+0000","last_change":"2024-02-10T08:51:40.063896+0000","last_active":"2024-02-10T08:52:10.335414+0000","last_peered":"2024-02-10T08:52:10.335414+0000","last_clean":"2024-02-10T08:52:10.335414+0000","last_became_active":"2024-02-10T08:46:15.350158+0000","last_became_peered":"2024-02-10T08:46:15.350158+0000","last_unstale":"2024-02-10T08:52:10.335414+0000","last_undegraded":"2024-02-10T08:52:10.335414+0000","last_fullsized":"2024-02-10T08:52:10.335414+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.063814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.063814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"473'29","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925539+0000","last_change":"2024-02-10T08:51:08.569596+0000","last_active":"2024-02-10T08:52:09.925539+0000","last_peered":"2024-02-10T08:52:09.925539+0000","last_clean":"2024-02-10T08:52:09.925539+0000","last_became_active":"2024-02-10T08:46:15.362855+0000","last_became_peered":"2024-02-10T08:46:15.362855+0000","last_unstale":"2024-02-10T08:52:09.925539+0000","last_undegraded":"2024-02-10T08:52:09.925539+0000","last_fullsized":"2024-02-10T08:52:09.925539+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'29","last_scrub_stamp":"2024-02-10T08:51:08.569484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.569484+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":430,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934263+0000","last_change":"2024-02-10T08:51:49.952004+0000","last_active":"2024-02-10T08:52:09.934263+0000","last_peered":"2024-02-10T08:52:09.934263+0000","last_clean":"2024-02-10T08:52:09.934263+0000","last_became_active":"2024-02-10T08:46:30.199952+0000","last_became_peered":"2024-02-10T08:46:30.199952+0000","last_unstale":"2024-02-10T08:52:09.934263+0000","last_undegraded":"2024-02-10T08:52:09.934263+0000","last_fullsized":"2024-02-10T08:52:09.934263+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.951887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.951887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":730,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703506+0000","last_change":"2024-02-10T08:52:05.897186+0000","last_active":"2024-02-10T08:52:09.703506+0000","last_peered":"2024-02-10T08:52:09.703506+0000","last_clean":"2024-02-10T08:52:09.703506+0000","last_became_active":"2024-02-10T08:47:49.238301+0000","last_became_peered":"2024-02-10T08:47:49.238301+0000","last_unstale":"2024-02-10T08:52:09.703506+0000","last_undegraded":"2024-02-10T08:52:09.703506+0000","last_fullsized":"2024-02-10T08:52:09.703506+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.897125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:41.793022+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.897125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703970+0000","last_change":"2024-02-10T08:51:45.971164+0000","last_active":"2024-02-10T08:52:09.703970+0000","last_peered":"2024-02-10T08:52:09.703970+0000","last_clean":"2024-02-10T08:52:09.703970+0000","last_became_active":"2024-02-10T08:47:49.229599+0000","last_became_peered":"2024-02-10T08:47:49.229599+0000","last_unstale":"2024-02-10T08:52:09.703970+0000","last_undegraded":"2024-02-10T08:52:09.703970+0000","last_fullsized":"2024-02-10T08:52:09.703970+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:45.971088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:45.971088+0000","last_clean_scrub_stamp":"2024-02-10T08:51:45.971088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":877,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333592+0000","last_change":"2024-02-10T08:51:48.668695+0000","last_active":"2024-02-10T08:52:10.333592+0000","last_peered":"2024-02-10T08:52:10.333592+0000","last_clean":"2024-02-10T08:52:10.333592+0000","last_became_active":"2024-02-10T08:47:49.237859+0000","last_became_peered":"2024-02-10T08:47:49.237859+0000","last_unstale":"2024-02-10T08:52:10.333592+0000","last_undegraded":"2024-02-10T08:52:10.333592+0000","last_fullsized":"2024-02-10T08:52:10.333592+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:48.668616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:32.456728+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.668616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"620'30","reported_seq":292,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925692+0000","last_change":"2024-02-10T08:51:07.597408+0000","last_active":"2024-02-10T08:52:09.925692+0000","last_peered":"2024-02-10T08:52:09.925692+0000","last_clean":"2024-02-10T08:52:09.925692+0000","last_became_active":"2024-02-10T08:47:49.239890+0000","last_became_peered":"2024-02-10T08:47:49.239890+0000","last_unstale":"2024-02-10T08:52:09.925692+0000","last_undegraded":"2024-02-10T08:52:09.925692+0000","last_fullsized":"2024-02-10T08:52:09.925692+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"620'30","last_scrub_stamp":"2024-02-10T08:51:07.597344+0000","last_deep_scrub":"620'30","last_deep_scrub_stamp":"2024-02-10T08:47:30.370523+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.597344+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":432,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933374+0000","last_change":"2024-02-10T08:52:00.889889+0000","last_active":"2024-02-10T08:52:09.933374+0000","last_peered":"2024-02-10T08:52:09.933374+0000","last_clean":"2024-02-10T08:52:09.933374+0000","last_became_active":"2024-02-10T08:47:49.252105+0000","last_became_peered":"2024-02-10T08:47:49.252105+0000","last_unstale":"2024-02-10T08:52:09.933374+0000","last_undegraded":"2024-02-10T08:52:09.933374+0000","last_fullsized":"2024-02-10T08:52:09.933374+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:00.889812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:50.991799+0000","last_clean_scrub_stamp":"2024-02-10T08:52:00.889812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":243,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925149+0000","last_change":"2024-02-10T08:50:33.606684+0000","last_active":"2024-02-10T08:52:09.925149+0000","last_peered":"2024-02-10T08:52:09.925149+0000","last_clean":"2024-02-10T08:52:09.925149+0000","last_became_active":"2024-02-10T08:46:30.199656+0000","last_became_peered":"2024-02-10T08:46:30.199656+0000","last_unstale":"2024-02-10T08:52:09.925149+0000","last_undegraded":"2024-02-10T08:52:09.925149+0000","last_fullsized":"2024-02-10T08:52:09.925149+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:33.606614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:50:33.606614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"725'2","reported_seq":160,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701707+0000","last_change":"2024-02-10T08:51:08.500925+0000","last_active":"2024-02-10T08:52:09.701707+0000","last_peered":"2024-02-10T08:52:09.701707+0000","last_clean":"2024-02-10T08:52:09.701707+0000","last_became_active":"2024-02-10T08:47:49.249076+0000","last_became_peered":"2024-02-10T08:47:49.249076+0000","last_unstale":"2024-02-10T08:52:09.701707+0000","last_undegraded":"2024-02-10T08:52:09.701707+0000","last_fullsized":"2024-02-10T08:52:09.701707+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'2","last_scrub_stamp":"2024-02-10T08:51:08.500803+0000","last_deep_scrub":"725'2","last_deep_scrub_stamp":"2024-02-10T08:51:08.500803+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.500803+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":791,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335231+0000","last_change":"2024-02-10T08:52:09.903082+0000","last_active":"2024-02-10T08:52:10.335231+0000","last_peered":"2024-02-10T08:52:10.335231+0000","last_clean":"2024-02-10T08:52:10.335231+0000","last_became_active":"2024-02-10T08:46:15.351493+0000","last_became_peered":"2024-02-10T08:46:15.351493+0000","last_unstale":"2024-02-10T08:52:10.335231+0000","last_undegraded":"2024-02-10T08:52:10.335231+0000","last_fullsized":"2024-02-10T08:52:10.335231+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.903006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.903006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934123+0000","last_change":"2024-02-10T08:51:29.900861+0000","last_active":"2024-02-10T08:52:09.934123+0000","last_peered":"2024-02-10T08:52:09.934123+0000","last_clean":"2024-02-10T08:52:09.934123+0000","last_became_active":"2024-02-10T08:43:15.118899+0000","last_became_peered":"2024-02-10T08:43:15.118899+0000","last_unstale":"2024-02-10T08:52:09.934123+0000","last_undegraded":"2024-02-10T08:52:09.934123+0000","last_fullsized":"2024-02-10T08:52:09.934123+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:29.900777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:06.926578+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.900777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"832'33","reported_seq":930,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335733+0000","last_change":"2024-02-10T08:51:20.998887+0000","last_active":"2024-02-10T08:52:10.335733+0000","last_peered":"2024-02-10T08:52:10.335733+0000","last_clean":"2024-02-10T08:52:10.335733+0000","last_became_active":"2024-02-10T08:46:29.215128+0000","last_became_peered":"2024-02-10T08:46:29.215128+0000","last_unstale":"2024-02-10T08:52:10.335733+0000","last_undegraded":"2024-02-10T08:52:10.335733+0000","last_fullsized":"2024-02-10T08:52:10.335733+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'33","last_scrub_stamp":"2024-02-10T08:51:20.998801+0000","last_deep_scrub":"832'33","last_deep_scrub_stamp":"2024-02-10T08:51:20.998801+0000","last_clean_scrub_stamp":"2024-02-10T08:51:20.998801+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":411,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933654+0000","last_change":"2024-02-10T08:51:14.839753+0000","last_active":"2024-02-10T08:52:09.933654+0000","last_peered":"2024-02-10T08:52:09.933654+0000","last_clean":"2024-02-10T08:52:09.933654+0000","last_became_active":"2024-02-10T08:46:15.362650+0000","last_became_peered":"2024-02-10T08:46:15.362650+0000","last_unstale":"2024-02-10T08:52:09.933654+0000","last_undegraded":"2024-02-10T08:52:09.933654+0000","last_fullsized":"2024-02-10T08:52:09.933654+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:14.839681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:31:10.597600+0000","last_clean_scrub_stamp":"2024-02-10T08:51:14.839681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":839,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332871+0000","last_change":"2024-02-10T08:52:05.591433+0000","last_active":"2024-02-10T08:52:10.332871+0000","last_peered":"2024-02-10T08:52:10.332871+0000","last_clean":"2024-02-10T08:52:10.332871+0000","last_became_active":"2024-02-10T08:43:15.126827+0000","last_became_peered":"2024-02-10T08:43:15.126827+0000","last_unstale":"2024-02-10T08:52:10.332871+0000","last_undegraded":"2024-02-10T08:52:10.332871+0000","last_fullsized":"2024-02-10T08:52:10.332871+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:05.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:52:05.591323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":818,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333184+0000","last_change":"2024-02-10T08:51:49.282695+0000","last_active":"2024-02-10T08:52:10.333184+0000","last_peered":"2024-02-10T08:52:10.333184+0000","last_clean":"2024-02-10T08:52:10.333184+0000","last_became_active":"2024-02-10T08:33:18.108660+0000","last_became_peered":"2024-02-10T08:33:18.108660+0000","last_unstale":"2024-02-10T08:52:10.333184+0000","last_undegraded":"2024-02-10T08:52:10.333184+0000","last_fullsized":"2024-02-10T08:52:10.333184+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:49.282616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:44:00.274550+0000","last_clean_scrub_stamp":"2024-02-10T08:51:49.282616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":825,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333659+0000","last_change":"2024-02-10T08:51:46.698978+0000","last_active":"2024-02-10T08:52:10.333659+0000","last_peered":"2024-02-10T08:52:10.333659+0000","last_clean":"2024-02-10T08:52:10.333659+0000","last_became_active":"2024-02-10T08:46:29.210688+0000","last_became_peered":"2024-02-10T08:46:29.210688+0000","last_unstale":"2024-02-10T08:52:10.333659+0000","last_undegraded":"2024-02-10T08:52:10.333659+0000","last_fullsized":"2024-02-10T08:52:10.333659+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:46.698902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:46.698902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":799,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332592+0000","last_change":"2024-02-10T08:51:15.732610+0000","last_active":"2024-02-10T08:52:10.332592+0000","last_peered":"2024-02-10T08:52:10.332592+0000","last_clean":"2024-02-10T08:52:10.332592+0000","last_became_active":"2024-02-10T08:47:49.237067+0000","last_became_peered":"2024-02-10T08:47:49.237067+0000","last_unstale":"2024-02-10T08:52:10.332592+0000","last_undegraded":"2024-02-10T08:52:10.332592+0000","last_fullsized":"2024-02-10T08:52:10.332592+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:15.732518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.732518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":860,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703925+0000","last_change":"2024-02-10T08:50:48.818974+0000","last_active":"2024-02-10T08:52:09.703925+0000","last_peered":"2024-02-10T08:52:09.703925+0000","last_clean":"2024-02-10T08:52:09.703925+0000","last_became_active":"2024-02-10T08:43:15.119595+0000","last_became_peered":"2024-02-10T08:43:15.119595+0000","last_unstale":"2024-02-10T08:52:09.703925+0000","last_undegraded":"2024-02-10T08:52:09.703925+0000","last_fullsized":"2024-02-10T08:52:09.703925+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:48.818866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:48.818866+0000","last_clean_scrub_stamp":"2024-02-10T08:50:48.818866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"473'17","reported_seq":162,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701752+0000","last_change":"2024-02-10T08:51:14.507631+0000","last_active":"2024-02-10T08:52:09.701752+0000","last_peered":"2024-02-10T08:52:09.701752+0000","last_clean":"2024-02-10T08:52:09.701752+0000","last_became_active":"2024-02-10T08:47:49.250301+0000","last_became_peered":"2024-02-10T08:47:49.250301+0000","last_unstale":"2024-02-10T08:52:09.701752+0000","last_undegraded":"2024-02-10T08:52:09.701752+0000","last_fullsized":"2024-02-10T08:52:09.701752+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'17","last_scrub_stamp":"2024-02-10T08:51:14.507563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:14.507563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":848,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703463+0000","last_change":"2024-02-10T08:50:59.777513+0000","last_active":"2024-02-10T08:52:09.703463+0000","last_peered":"2024-02-10T08:52:09.703463+0000","last_clean":"2024-02-10T08:52:09.703463+0000","last_became_active":"2024-02-10T08:46:15.362460+0000","last_became_peered":"2024-02-10T08:46:15.362460+0000","last_unstale":"2024-02-10T08:52:09.703463+0000","last_undegraded":"2024-02-10T08:52:09.703463+0000","last_fullsized":"2024-02-10T08:52:09.703463+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:59.777445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:59.777445+0000","last_clean_scrub_stamp":"2024-02-10T08:50:59.777445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"80'6","reported_seq":416,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.690202+0000","last_change":"2024-02-10T08:51:17.586120+0000","last_active":"2024-02-10T08:52:10.690202+0000","last_peered":"2024-02-10T08:52:10.690202+0000","last_clean":"2024-02-10T08:52:10.690202+0000","last_became_active":"2024-02-10T08:47:49.229501+0000","last_became_peered":"2024-02-10T08:47:49.229501+0000","last_unstale":"2024-02-10T08:52:10.690202+0000","last_undegraded":"2024-02-10T08:52:10.690202+0000","last_fullsized":"2024-02-10T08:52:10.690202+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'6","last_scrub_stamp":"2024-02-10T08:51:17.586026+0000","last_deep_scrub":"80'6","last_deep_scrub_stamp":"2024-02-10T08:47:24.043513+0000","last_clean_scrub_stamp":"2024-02-10T08:51:17.586026+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":399,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934237+0000","last_change":"2024-02-10T08:51:43.835976+0000","last_active":"2024-02-10T08:52:09.934237+0000","last_peered":"2024-02-10T08:52:09.934237+0000","last_clean":"2024-02-10T08:52:09.934237+0000","last_became_active":"2024-02-10T08:43:15.134094+0000","last_became_peered":"2024-02-10T08:43:15.134094+0000","last_unstale":"2024-02-10T08:52:09.934237+0000","last_undegraded":"2024-02-10T08:52:09.934237+0000","last_fullsized":"2024-02-10T08:52:09.934237+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:43.835902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.787351+0000","last_clean_scrub_stamp":"2024-02-10T08:51:43.835902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701059+0000","last_change":"2024-02-10T08:50:43.402411+0000","last_active":"2024-02-10T08:52:09.701059+0000","last_peered":"2024-02-10T08:52:09.701059+0000","last_clean":"2024-02-10T08:52:09.701059+0000","last_became_active":"2024-02-10T08:47:49.236685+0000","last_became_peered":"2024-02-10T08:47:49.236685+0000","last_unstale":"2024-02-10T08:52:09.701059+0000","last_undegraded":"2024-02-10T08:52:09.701059+0000","last_fullsized":"2024-02-10T08:52:09.701059+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:43.402320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.556602+0000","last_clean_scrub_stamp":"2024-02-10T08:50:43.402320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":266,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925589+0000","last_change":"2024-02-10T08:51:38.286982+0000","last_active":"2024-02-10T08:52:09.925589+0000","last_peered":"2024-02-10T08:52:09.925589+0000","last_clean":"2024-02-10T08:52:09.925589+0000","last_became_active":"2024-02-10T08:47:49.241958+0000","last_became_peered":"2024-02-10T08:47:49.241958+0000","last_unstale":"2024-02-10T08:52:09.925589+0000","last_undegraded":"2024-02-10T08:52:09.925589+0000","last_fullsized":"2024-02-10T08:52:09.925589+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:38.286913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:38.286913+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.286913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"822'44","reported_seq":72,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332792+0000","last_change":"2024-02-10T08:51:20.705787+0000","last_active":"2024-02-10T08:52:10.332792+0000","last_peered":"2024-02-10T08:52:10.332792+0000","last_clean":"2024-02-10T08:52:10.332792+0000","last_became_active":"2024-02-10T08:50:42.866275+0000","last_became_peered":"2024-02-10T08:50:42.866275+0000","last_unstale":"2024-02-10T08:52:10.332792+0000","last_undegraded":"2024-02-10T08:52:10.332792+0000","last_fullsized":"2024-02-10T08:52:10.332792+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'44","last_scrub_stamp":"2024-02-10T08:51:20.705702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:20.705702+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":839,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333027+0000","last_change":"2024-02-10T08:51:36.669755+0000","last_active":"2024-02-10T08:52:10.333027+0000","last_peered":"2024-02-10T08:52:10.333027+0000","last_clean":"2024-02-10T08:52:10.333027+0000","last_became_active":"2024-02-10T08:47:49.238071+0000","last_became_peered":"2024-02-10T08:47:49.238071+0000","last_unstale":"2024-02-10T08:52:10.333027+0000","last_undegraded":"2024-02-10T08:52:10.333027+0000","last_fullsized":"2024-02-10T08:52:10.333027+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:36.669648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:42.473021+0000","last_clean_scrub_stamp":"2024-02-10T08:51:36.669648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":840,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333682+0000","last_change":"2024-02-10T08:51:22.207745+0000","last_active":"2024-02-10T08:52:10.333682+0000","last_peered":"2024-02-10T08:52:10.333682+0000","last_clean":"2024-02-10T08:52:10.333682+0000","last_became_active":"2024-02-10T08:46:29.214590+0000","last_became_peered":"2024-02-10T08:46:29.214590+0000","last_unstale":"2024-02-10T08:52:10.333682+0000","last_undegraded":"2024-02-10T08:52:10.333682+0000","last_fullsized":"2024-02-10T08:52:10.333682+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:22.207631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.207631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703270+0000","last_change":"2024-02-10T08:51:51.975568+0000","last_active":"2024-02-10T08:52:09.703270+0000","last_peered":"2024-02-10T08:52:09.703270+0000","last_clean":"2024-02-10T08:52:09.703270+0000","last_became_active":"2024-02-10T08:46:15.362675+0000","last_became_peered":"2024-02-10T08:46:15.362675+0000","last_unstale":"2024-02-10T08:52:09.703270+0000","last_undegraded":"2024-02-10T08:52:09.703270+0000","last_fullsized":"2024-02-10T08:52:09.703270+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:51.975472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:51.975472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"624'10","reported_seq":307,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925370+0000","last_change":"2024-02-10T08:50:48.526828+0000","last_active":"2024-02-10T08:52:09.925370+0000","last_peered":"2024-02-10T08:52:09.925370+0000","last_clean":"2024-02-10T08:52:09.925370+0000","last_became_active":"2024-02-10T08:46:15.356837+0000","last_became_peered":"2024-02-10T08:46:15.356837+0000","last_unstale":"2024-02-10T08:52:09.925370+0000","last_undegraded":"2024-02-10T08:52:09.925370+0000","last_fullsized":"2024-02-10T08:52:09.925370+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'10","last_scrub_stamp":"2024-02-10T08:50:48.526761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:50:48.526761+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"473'25","reported_seq":167,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701598+0000","last_change":"2024-02-10T08:51:18.540799+0000","last_active":"2024-02-10T08:52:09.701598+0000","last_peered":"2024-02-10T08:52:09.701598+0000","last_clean":"2024-02-10T08:52:09.701598+0000","last_became_active":"2024-02-10T08:47:49.235447+0000","last_became_peered":"2024-02-10T08:47:49.235447+0000","last_unstale":"2024-02-10T08:52:09.701598+0000","last_undegraded":"2024-02-10T08:52:09.701598+0000","last_fullsized":"2024-02-10T08:52:09.701598+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'25","last_scrub_stamp":"2024-02-10T08:51:18.540724+0000","last_deep_scrub":"84'15","last_deep_scrub_stamp":"2024-02-10T08:35:34.249424+0000","last_clean_scrub_stamp":"2024-02-10T08:51:18.540724+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701718+0000","last_change":"2024-02-10T08:51:01.423644+0000","last_active":"2024-02-10T08:52:09.701718+0000","last_peered":"2024-02-10T08:52:09.701718+0000","last_clean":"2024-02-10T08:52:09.701718+0000","last_became_active":"2024-02-10T08:47:49.256463+0000","last_became_peered":"2024-02-10T08:47:49.256463+0000","last_unstale":"2024-02-10T08:52:09.701718+0000","last_undegraded":"2024-02-10T08:52:09.701718+0000","last_fullsized":"2024-02-10T08:52:09.701718+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:01.423585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:49:35.015073+0000","last_clean_scrub_stamp":"2024-02-10T08:51:01.423585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925213+0000","last_change":"2024-02-10T08:51:47.363190+0000","last_active":"2024-02-10T08:52:09.925213+0000","last_peered":"2024-02-10T08:52:09.925213+0000","last_clean":"2024-02-10T08:52:09.925213+0000","last_became_active":"2024-02-10T08:46:15.372250+0000","last_became_peered":"2024-02-10T08:46:15.372250+0000","last_unstale":"2024-02-10T08:52:09.925213+0000","last_undegraded":"2024-02-10T08:52:09.925213+0000","last_fullsized":"2024-02-10T08:52:09.925213+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.363092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.363092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":780,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333435+0000","last_change":"2024-02-10T08:50:55.206205+0000","last_active":"2024-02-10T08:52:10.333435+0000","last_peered":"2024-02-10T08:52:10.333435+0000","last_clean":"2024-02-10T08:52:10.333435+0000","last_became_active":"2024-02-10T08:47:49.225090+0000","last_became_peered":"2024-02-10T08:47:49.225090+0000","last_unstale":"2024-02-10T08:52:10.333435+0000","last_undegraded":"2024-02-10T08:52:10.333435+0000","last_fullsized":"2024-02-10T08:52:10.333435+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:55.206123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:55.206123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":883,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703377+0000","last_change":"2024-02-10T08:51:38.867904+0000","last_active":"2024-02-10T08:52:09.703377+0000","last_peered":"2024-02-10T08:52:09.703377+0000","last_clean":"2024-02-10T08:52:09.703377+0000","last_became_active":"2024-02-10T08:47:49.236554+0000","last_became_peered":"2024-02-10T08:47:49.236554+0000","last_unstale":"2024-02-10T08:52:09.703377+0000","last_undegraded":"2024-02-10T08:52:09.703377+0000","last_fullsized":"2024-02-10T08:52:09.703377+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:38.867817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.867817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"473'23","reported_seq":755,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701228+0000","last_change":"2024-02-10T08:51:38.814227+0000","last_active":"2024-02-10T08:52:09.701228+0000","last_peered":"2024-02-10T08:52:09.701228+0000","last_clean":"2024-02-10T08:52:09.701228+0000","last_became_active":"2024-02-10T08:47:49.242469+0000","last_became_peered":"2024-02-10T08:47:49.242469+0000","last_unstale":"2024-02-10T08:52:09.701228+0000","last_undegraded":"2024-02-10T08:52:09.701228+0000","last_fullsized":"2024-02-10T08:52:09.701228+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'23","last_scrub_stamp":"2024-02-10T08:51:38.814098+0000","last_deep_scrub":"78'17","last_deep_scrub_stamp":"2024-02-10T08:34:54.039584+0000","last_clean_scrub_stamp":"2024-02-10T08:51:38.814098+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"56'2","reported_seq":720,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701980+0000","last_change":"2024-02-10T08:50:41.968410+0000","last_active":"2024-02-10T08:52:09.701980+0000","last_peered":"2024-02-10T08:52:09.701980+0000","last_clean":"2024-02-10T08:52:09.701980+0000","last_became_active":"2024-02-10T08:46:30.201079+0000","last_became_peered":"2024-02-10T08:46:30.201079+0000","last_unstale":"2024-02-10T08:52:09.701980+0000","last_undegraded":"2024-02-10T08:52:09.701980+0000","last_fullsized":"2024-02-10T08:52:09.701980+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2024-02-10T08:50:41.968315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:50:41.968315+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333401+0000","last_change":"2024-02-10T08:52:08.306127+0000","last_active":"2024-02-10T08:52:10.333401+0000","last_peered":"2024-02-10T08:52:10.333401+0000","last_clean":"2024-02-10T08:52:10.333401+0000","last_became_active":"2024-02-10T08:47:49.225823+0000","last_became_peered":"2024-02-10T08:47:49.225823+0000","last_unstale":"2024-02-10T08:52:10.333401+0000","last_undegraded":"2024-02-10T08:52:10.333401+0000","last_fullsized":"2024-02-10T08:52:10.333401+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:08.306037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:08.306037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":168,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332926+0000","last_change":"2024-02-10T08:51:35.704431+0000","last_active":"2024-02-10T08:52:10.332926+0000","last_peered":"2024-02-10T08:52:10.332926+0000","last_clean":"2024-02-10T08:52:10.332926+0000","last_became_active":"2024-02-10T08:47:49.238251+0000","last_became_peered":"2024-02-10T08:47:49.238251+0000","last_unstale":"2024-02-10T08:52:10.332926+0000","last_undegraded":"2024-02-10T08:52:10.332926+0000","last_fullsized":"2024-02-10T08:52:10.332926+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.704334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.704334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"473'19","reported_seq":243,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925788+0000","last_change":"2024-02-10T08:50:45.545641+0000","last_active":"2024-02-10T08:52:09.925788+0000","last_peered":"2024-02-10T08:52:09.925788+0000","last_clean":"2024-02-10T08:52:09.925788+0000","last_became_active":"2024-02-10T08:46:15.367321+0000","last_became_peered":"2024-02-10T08:46:15.367321+0000","last_unstale":"2024-02-10T08:52:09.925788+0000","last_undegraded":"2024-02-10T08:52:09.925788+0000","last_fullsized":"2024-02-10T08:52:09.925788+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:45.545577+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:44:38.409486+0000","last_clean_scrub_stamp":"2024-02-10T08:50:45.545577+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":870,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703360+0000","last_change":"2024-02-10T08:51:40.935809+0000","last_active":"2024-02-10T08:52:09.703360+0000","last_peered":"2024-02-10T08:52:09.703360+0000","last_clean":"2024-02-10T08:52:09.703360+0000","last_became_active":"2024-02-10T08:47:49.236205+0000","last_became_peered":"2024-02-10T08:47:49.236205+0000","last_unstale":"2024-02-10T08:52:09.703360+0000","last_undegraded":"2024-02-10T08:52:09.703360+0000","last_fullsized":"2024-02-10T08:52:09.703360+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.935714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:02.159169+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.935714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":855,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703771+0000","last_change":"2024-02-10T08:52:06.942398+0000","last_active":"2024-02-10T08:52:09.703771+0000","last_peered":"2024-02-10T08:52:09.703771+0000","last_clean":"2024-02-10T08:52:09.703771+0000","last_became_active":"2024-02-10T08:46:29.209122+0000","last_became_peered":"2024-02-10T08:46:29.209122+0000","last_unstale":"2024-02-10T08:52:09.703771+0000","last_undegraded":"2024-02-10T08:52:09.703771+0000","last_fullsized":"2024-02-10T08:52:09.703771+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.942295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.942295+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":648,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701398+0000","last_change":"2024-02-10T08:52:04.820047+0000","last_active":"2024-02-10T08:52:09.701398+0000","last_peered":"2024-02-10T08:52:09.701398+0000","last_clean":"2024-02-10T08:52:09.701398+0000","last_became_active":"2024-02-10T08:47:49.241351+0000","last_became_peered":"2024-02-10T08:47:49.241351+0000","last_unstale":"2024-02-10T08:52:09.701398+0000","last_undegraded":"2024-02-10T08:52:09.701398+0000","last_fullsized":"2024-02-10T08:52:09.701398+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.819963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.819963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"473'12","reported_seq":902,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333690+0000","last_change":"2024-02-10T08:51:27.219904+0000","last_active":"2024-02-10T08:52:10.333690+0000","last_peered":"2024-02-10T08:52:10.333690+0000","last_clean":"2024-02-10T08:52:10.333690+0000","last_became_active":"2024-02-10T08:47:49.225942+0000","last_became_peered":"2024-02-10T08:47:49.225942+0000","last_unstale":"2024-02-10T08:52:10.333690+0000","last_undegraded":"2024-02-10T08:52:10.333690+0000","last_fullsized":"2024-02-10T08:52:10.333690+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'12","last_scrub_stamp":"2024-02-10T08:51:27.219837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:27.219837+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335234+0000","last_change":"2024-02-10T08:51:29.010376+0000","last_active":"2024-02-10T08:52:10.335234+0000","last_peered":"2024-02-10T08:52:10.335234+0000","last_clean":"2024-02-10T08:52:10.335234+0000","last_became_active":"2024-02-10T08:47:49.241991+0000","last_became_peered":"2024-02-10T08:47:49.241991+0000","last_unstale":"2024-02-10T08:52:10.335234+0000","last_undegraded":"2024-02-10T08:52:10.335234+0000","last_fullsized":"2024-02-10T08:52:10.335234+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:29.010284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.010284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934077+0000","last_change":"2024-02-10T08:51:01.855495+0000","last_active":"2024-02-10T08:52:09.934077+0000","last_peered":"2024-02-10T08:52:09.934077+0000","last_clean":"2024-02-10T08:52:09.934077+0000","last_became_active":"2024-02-10T08:46:29.218354+0000","last_became_peered":"2024-02-10T08:46:29.218354+0000","last_unstale":"2024-02-10T08:52:09.934077+0000","last_undegraded":"2024-02-10T08:52:09.934077+0000","last_fullsized":"2024-02-10T08:52:09.934077+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:01.855415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:01.855415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":776,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332785+0000","last_change":"2024-02-10T08:51:21.680721+0000","last_active":"2024-02-10T08:52:10.332785+0000","last_peered":"2024-02-10T08:52:10.332785+0000","last_clean":"2024-02-10T08:52:10.332785+0000","last_became_active":"2024-02-10T08:46:29.214145+0000","last_became_peered":"2024-02-10T08:46:29.214145+0000","last_unstale":"2024-02-10T08:52:10.332785+0000","last_undegraded":"2024-02-10T08:52:10.332785+0000","last_fullsized":"2024-02-10T08:52:10.332785+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:21.680621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:47.081542+0000","last_clean_scrub_stamp":"2024-02-10T08:51:21.680621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"473'20","reported_seq":930,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333398+0000","last_change":"2024-02-10T08:51:02.242143+0000","last_active":"2024-02-10T08:52:10.333398+0000","last_peered":"2024-02-10T08:52:10.333398+0000","last_clean":"2024-02-10T08:52:10.333398+0000","last_became_active":"2024-02-10T08:47:49.231930+0000","last_became_peered":"2024-02-10T08:47:49.231930+0000","last_unstale":"2024-02-10T08:52:10.333398+0000","last_undegraded":"2024-02-10T08:52:10.333398+0000","last_fullsized":"2024-02-10T08:52:10.333398+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:02.242057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:02.242057+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":691,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333078+0000","last_change":"2024-02-10T08:50:56.254276+0000","last_active":"2024-02-10T08:52:10.333078+0000","last_peered":"2024-02-10T08:52:10.333078+0000","last_clean":"2024-02-10T08:52:10.333078+0000","last_became_active":"2024-02-10T08:47:49.225406+0000","last_became_peered":"2024-02-10T08:47:49.225406+0000","last_unstale":"2024-02-10T08:52:10.333078+0000","last_undegraded":"2024-02-10T08:52:10.333078+0000","last_fullsized":"2024-02-10T08:52:10.333078+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:56.254214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:56.254214+0000","last_clean_scrub_stamp":"2024-02-10T08:50:56.254214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"711'13","reported_seq":1104,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333312+0000","last_change":"2024-02-10T08:52:01.550377+0000","last_active":"2024-02-10T08:52:10.333312+0000","last_peered":"2024-02-10T08:52:10.333312+0000","last_clean":"2024-02-10T08:52:10.333312+0000","last_became_active":"2024-02-10T08:47:49.238344+0000","last_became_peered":"2024-02-10T08:47:49.238344+0000","last_unstale":"2024-02-10T08:52:10.333312+0000","last_undegraded":"2024-02-10T08:52:10.333312+0000","last_fullsized":"2024-02-10T08:52:10.333312+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13","last_scrub_stamp":"2024-02-10T08:52:01.550272+0000","last_deep_scrub":"711'13","last_deep_scrub_stamp":"2024-02-10T08:52:01.550272+0000","last_clean_scrub_stamp":"2024-02-10T08:52:01.550272+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925883+0000","last_change":"2024-02-10T08:51:40.233510+0000","last_active":"2024-02-10T08:52:09.925883+0000","last_peered":"2024-02-10T08:52:09.925883+0000","last_clean":"2024-02-10T08:52:09.925883+0000","last_became_active":"2024-02-10T08:46:15.366871+0000","last_became_peered":"2024-02-10T08:46:15.366871+0000","last_unstale":"2024-02-10T08:52:09.925883+0000","last_undegraded":"2024-02-10T08:52:09.925883+0000","last_fullsized":"2024-02-10T08:52:09.925883+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:40.233445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:40.233445+0000","last_clean_scrub_stamp":"2024-02-10T08:51:40.233445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":237,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925690+0000","last_change":"2024-02-10T08:51:53.323765+0000","last_active":"2024-02-10T08:52:09.925690+0000","last_peered":"2024-02-10T08:52:09.925690+0000","last_clean":"2024-02-10T08:52:09.925690+0000","last_became_active":"2024-02-10T08:46:15.373987+0000","last_became_peered":"2024-02-10T08:46:15.373987+0000","last_unstale":"2024-02-10T08:52:09.925690+0000","last_undegraded":"2024-02-10T08:52:09.925690+0000","last_fullsized":"2024-02-10T08:52:09.925690+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:53.323700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:53.323700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"798'19","reported_seq":881,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333270+0000","last_change":"2024-02-10T08:51:04.698846+0000","last_active":"2024-02-10T08:52:10.333270+0000","last_peered":"2024-02-10T08:52:10.333270+0000","last_clean":"2024-02-10T08:52:10.333270+0000","last_became_active":"2024-02-10T08:43:15.126933+0000","last_became_peered":"2024-02-10T08:43:15.126933+0000","last_unstale":"2024-02-10T08:52:10.333270+0000","last_undegraded":"2024-02-10T08:52:10.333270+0000","last_fullsized":"2024-02-10T08:52:10.333270+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"798'19","last_scrub_stamp":"2024-02-10T08:51:04.698764+0000","last_deep_scrub":"798'19","last_deep_scrub_stamp":"2024-02-10T08:51:04.698764+0000","last_clean_scrub_stamp":"2024-02-10T08:51:04.698764+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":593,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701195+0000","last_change":"2024-02-10T08:51:09.970486+0000","last_active":"2024-02-10T08:52:09.701195+0000","last_peered":"2024-02-10T08:52:09.701195+0000","last_clean":"2024-02-10T08:52:09.701195+0000","last_became_active":"2024-02-10T08:46:30.196362+0000","last_became_peered":"2024-02-10T08:46:30.196362+0000","last_unstale":"2024-02-10T08:52:09.701195+0000","last_undegraded":"2024-02-10T08:52:09.701195+0000","last_fullsized":"2024-02-10T08:52:09.701195+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:09.970399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:35:45.760869+0000","last_clean_scrub_stamp":"2024-02-10T08:51:09.970399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"465'10","reported_seq":1075,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333509+0000","last_change":"2024-02-10T08:51:07.231908+0000","last_active":"2024-02-10T08:52:10.333509+0000","last_peered":"2024-02-10T08:52:10.333509+0000","last_clean":"2024-02-10T08:52:10.333509+0000","last_became_active":"2024-02-10T08:43:15.136954+0000","last_became_peered":"2024-02-10T08:43:15.136954+0000","last_unstale":"2024-02-10T08:52:10.333509+0000","last_undegraded":"2024-02-10T08:52:10.333509+0000","last_fullsized":"2024-02-10T08:52:10.333509+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'10","last_scrub_stamp":"2024-02-10T08:51:07.231794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:07.231794+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":655,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701948+0000","last_change":"2024-02-10T08:51:33.814101+0000","last_active":"2024-02-10T08:52:09.701948+0000","last_peered":"2024-02-10T08:52:09.701948+0000","last_clean":"2024-02-10T08:52:09.701948+0000","last_became_active":"2024-02-10T08:46:29.210537+0000","last_became_peered":"2024-02-10T08:46:29.210537+0000","last_unstale":"2024-02-10T08:52:09.701948+0000","last_undegraded":"2024-02-10T08:52:09.701948+0000","last_fullsized":"2024-02-10T08:52:09.701948+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:33.814004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:33.814004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"103'7","reported_seq":810,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703352+0000","last_change":"2024-02-10T08:51:08.778455+0000","last_active":"2024-02-10T08:52:09.703352+0000","last_peered":"2024-02-10T08:52:09.703352+0000","last_clean":"2024-02-10T08:52:09.703352+0000","last_became_active":"2024-02-10T08:47:49.237831+0000","last_became_peered":"2024-02-10T08:47:49.237831+0000","last_unstale":"2024-02-10T08:52:09.703352+0000","last_undegraded":"2024-02-10T08:52:09.703352+0000","last_fullsized":"2024-02-10T08:52:09.703352+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'7","last_scrub_stamp":"2024-02-10T08:51:08.778387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:08.778387+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"473'19","reported_seq":160,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701701+0000","last_change":"2024-02-10T08:50:52.318856+0000","last_active":"2024-02-10T08:52:09.701701+0000","last_peered":"2024-02-10T08:52:09.701701+0000","last_clean":"2024-02-10T08:52:09.701701+0000","last_became_active":"2024-02-10T08:47:49.226295+0000","last_became_peered":"2024-02-10T08:47:49.226295+0000","last_unstale":"2024-02-10T08:52:09.701701+0000","last_undegraded":"2024-02-10T08:52:09.701701+0000","last_fullsized":"2024-02-10T08:52:09.701701+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:50:52.318773+0000","last_deep_scrub":"473'19","last_deep_scrub_stamp":"2024-02-10T08:47:23.671541+0000","last_clean_scrub_stamp":"2024-02-10T08:50:52.318773+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":830,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335581+0000","last_change":"2024-02-10T08:51:37.042907+0000","last_active":"2024-02-10T08:52:10.335581+0000","last_peered":"2024-02-10T08:52:10.335581+0000","last_clean":"2024-02-10T08:52:10.335581+0000","last_became_active":"2024-02-10T08:33:18.108571+0000","last_became_peered":"2024-02-10T08:33:18.108571+0000","last_unstale":"2024-02-10T08:52:10.335581+0000","last_undegraded":"2024-02-10T08:52:10.335581+0000","last_fullsized":"2024-02-10T08:52:10.335581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:37.042831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:37.042831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":855,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333323+0000","last_change":"2024-02-10T08:51:41.179889+0000","last_active":"2024-02-10T08:52:10.333323+0000","last_peered":"2024-02-10T08:52:10.333323+0000","last_clean":"2024-02-10T08:52:10.333323+0000","last_became_active":"2024-02-10T08:47:49.228717+0000","last_became_peered":"2024-02-10T08:47:49.228717+0000","last_unstale":"2024-02-10T08:52:10.333323+0000","last_undegraded":"2024-02-10T08:52:10.333323+0000","last_fullsized":"2024-02-10T08:52:10.333323+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:41.179821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:21.099708+0000","last_clean_scrub_stamp":"2024-02-10T08:51:41.179821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925498+0000","last_change":"2024-02-10T08:50:50.543955+0000","last_active":"2024-02-10T08:52:09.925498+0000","last_peered":"2024-02-10T08:52:09.925498+0000","last_clean":"2024-02-10T08:52:09.925498+0000","last_became_active":"2024-02-10T08:47:49.239985+0000","last_became_peered":"2024-02-10T08:47:49.239985+0000","last_unstale":"2024-02-10T08:52:09.925498+0000","last_undegraded":"2024-02-10T08:52:09.925498+0000","last_fullsized":"2024-02-10T08:52:09.925498+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:50.543887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:50.543887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":793,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335944+0000","last_change":"2024-02-10T08:51:58.913272+0000","last_active":"2024-02-10T08:52:10.335944+0000","last_peered":"2024-02-10T08:52:10.335944+0000","last_clean":"2024-02-10T08:52:10.335944+0000","last_became_active":"2024-02-10T08:46:15.351095+0000","last_became_peered":"2024-02-10T08:46:15.351095+0000","last_unstale":"2024-02-10T08:52:10.335944+0000","last_undegraded":"2024-02-10T08:52:10.335944+0000","last_fullsized":"2024-02-10T08:52:10.335944+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:58.913187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:58.913187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"473'19","reported_seq":767,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701325+0000","last_change":"2024-02-10T08:51:35.771062+0000","last_active":"2024-02-10T08:52:09.701325+0000","last_peered":"2024-02-10T08:52:09.701325+0000","last_clean":"2024-02-10T08:52:09.701325+0000","last_became_active":"2024-02-10T08:50:42.866200+0000","last_became_peered":"2024-02-10T08:50:42.866200+0000","last_unstale":"2024-02-10T08:52:09.701325+0000","last_undegraded":"2024-02-10T08:52:09.701325+0000","last_fullsized":"2024-02-10T08:52:09.701325+0000","mapping_epoch":819,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":820,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:51:35.770985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.770985+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701501+0000","last_change":"2024-02-10T08:51:22.539464+0000","last_active":"2024-02-10T08:52:09.701501+0000","last_peered":"2024-02-10T08:52:09.701501+0000","last_clean":"2024-02-10T08:52:09.701501+0000","last_became_active":"2024-02-10T08:47:49.251347+0000","last_became_peered":"2024-02-10T08:47:49.251347+0000","last_unstale":"2024-02-10T08:52:09.701501+0000","last_undegraded":"2024-02-10T08:52:09.701501+0000","last_fullsized":"2024-02-10T08:52:09.701501+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:22.539355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:22.539355+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.539355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701781+0000","last_change":"2024-02-10T08:51:24.531448+0000","last_active":"2024-02-10T08:52:09.701781+0000","last_peered":"2024-02-10T08:52:09.701781+0000","last_clean":"2024-02-10T08:52:09.701781+0000","last_became_active":"2024-02-10T08:47:49.222424+0000","last_became_peered":"2024-02-10T08:47:49.222424+0000","last_unstale":"2024-02-10T08:52:09.701781+0000","last_undegraded":"2024-02-10T08:52:09.701781+0000","last_fullsized":"2024-02-10T08:52:09.701781+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:24.531342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:40:36.354073+0000","last_clean_scrub_stamp":"2024-02-10T08:51:24.531342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":835,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332814+0000","last_change":"2024-02-10T08:51:58.222709+0000","last_active":"2024-02-10T08:52:10.332814+0000","last_peered":"2024-02-10T08:52:10.332814+0000","last_clean":"2024-02-10T08:52:10.332814+0000","last_became_active":"2024-02-10T08:46:15.349542+0000","last_became_peered":"2024-02-10T08:46:15.349542+0000","last_unstale":"2024-02-10T08:52:10.332814+0000","last_undegraded":"2024-02-10T08:52:10.332814+0000","last_fullsized":"2024-02-10T08:52:10.332814+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:58.222637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:58.222637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":786,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333515+0000","last_change":"2024-02-10T08:50:59.260262+0000","last_active":"2024-02-10T08:52:10.333515+0000","last_peered":"2024-02-10T08:52:10.333515+0000","last_clean":"2024-02-10T08:52:10.333515+0000","last_became_active":"2024-02-10T08:46:15.366169+0000","last_became_peered":"2024-02-10T08:46:15.366169+0000","last_unstale":"2024-02-10T08:52:10.333515+0000","last_undegraded":"2024-02-10T08:52:10.333515+0000","last_fullsized":"2024-02-10T08:52:10.333515+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:59.260164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:59.260164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"827'10","reported_seq":860,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332702+0000","last_change":"2024-02-10T08:51:03.211894+0000","last_active":"2024-02-10T08:52:10.332702+0000","last_peered":"2024-02-10T08:52:10.332702+0000","last_clean":"2024-02-10T08:52:10.332702+0000","last_became_active":"2024-02-10T08:47:49.225679+0000","last_became_peered":"2024-02-10T08:47:49.225679+0000","last_unstale":"2024-02-10T08:52:10.332702+0000","last_undegraded":"2024-02-10T08:52:10.332702+0000","last_fullsized":"2024-02-10T08:52:10.332702+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"827'10","last_scrub_stamp":"2024-02-10T08:51:03.211831+0000","last_deep_scrub":"827'10","last_deep_scrub_stamp":"2024-02-10T08:51:03.211831+0000","last_clean_scrub_stamp":"2024-02-10T08:51:03.211831+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":825,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333426+0000","last_change":"2024-02-10T08:51:39.711442+0000","last_active":"2024-02-10T08:52:10.333426+0000","last_peered":"2024-02-10T08:52:10.333426+0000","last_clean":"2024-02-10T08:52:10.333426+0000","last_became_active":"2024-02-10T08:33:18.108303+0000","last_became_peered":"2024-02-10T08:33:18.108303+0000","last_unstale":"2024-02-10T08:52:10.333426+0000","last_undegraded":"2024-02-10T08:52:10.333426+0000","last_fullsized":"2024-02-10T08:52:10.333426+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:39.711351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:39.711351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":851,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335202+0000","last_change":"2024-02-10T08:51:34.025342+0000","last_active":"2024-02-10T08:52:10.335202+0000","last_peered":"2024-02-10T08:52:10.335202+0000","last_clean":"2024-02-10T08:52:10.335202+0000","last_became_active":"2024-02-10T08:46:29.221577+0000","last_became_peered":"2024-02-10T08:46:29.221577+0000","last_unstale":"2024-02-10T08:52:10.335202+0000","last_undegraded":"2024-02-10T08:52:10.335202+0000","last_fullsized":"2024-02-10T08:52:10.335202+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:34.025230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:51:34.025230+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.025230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933964+0000","last_change":"2024-02-10T08:50:54.999584+0000","last_active":"2024-02-10T08:52:09.933964+0000","last_peered":"2024-02-10T08:52:09.933964+0000","last_clean":"2024-02-10T08:52:09.933964+0000","last_became_active":"2024-02-10T08:46:29.217709+0000","last_became_peered":"2024-02-10T08:46:29.217709+0000","last_unstale":"2024-02-10T08:52:09.933964+0000","last_undegraded":"2024-02-10T08:52:09.933964+0000","last_fullsized":"2024-02-10T08:52:09.933964+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.999503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.999503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":798,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333287+0000","last_change":"2024-02-10T08:51:15.167860+0000","last_active":"2024-02-10T08:52:10.333287+0000","last_peered":"2024-02-10T08:52:10.333287+0000","last_clean":"2024-02-10T08:52:10.333287+0000","last_became_active":"2024-02-10T08:47:49.231588+0000","last_became_peered":"2024-02-10T08:47:49.231588+0000","last_unstale":"2024-02-10T08:52:10.333287+0000","last_undegraded":"2024-02-10T08:52:10.333287+0000","last_fullsized":"2024-02-10T08:52:10.333287+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:15.167795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:15.167795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"473'19","reported_seq":889,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333056+0000","last_change":"2024-02-10T08:51:44.251816+0000","last_active":"2024-02-10T08:52:10.333056+0000","last_peered":"2024-02-10T08:52:10.333056+0000","last_clean":"2024-02-10T08:52:10.333056+0000","last_became_active":"2024-02-10T08:43:15.136975+0000","last_became_peered":"2024-02-10T08:43:15.136975+0000","last_unstale":"2024-02-10T08:52:10.333056+0000","last_undegraded":"2024-02-10T08:52:10.333056+0000","last_fullsized":"2024-02-10T08:52:10.333056+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:51:44.251748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.251748+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":830,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335881+0000","last_change":"2024-02-10T08:51:54.956900+0000","last_active":"2024-02-10T08:52:10.335881+0000","last_peered":"2024-02-10T08:52:10.335881+0000","last_clean":"2024-02-10T08:52:10.335881+0000","last_became_active":"2024-02-10T08:33:18.107493+0000","last_became_peered":"2024-02-10T08:33:18.107493+0000","last_unstale":"2024-02-10T08:52:10.335881+0000","last_undegraded":"2024-02-10T08:52:10.335881+0000","last_fullsized":"2024-02-10T08:52:10.335881+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.956821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.956821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933505+0000","last_change":"2024-02-10T08:51:54.866778+0000","last_active":"2024-02-10T08:52:09.933505+0000","last_peered":"2024-02-10T08:52:09.933505+0000","last_clean":"2024-02-10T08:52:09.933505+0000","last_became_active":"2024-02-10T08:46:29.216693+0000","last_became_peered":"2024-02-10T08:46:29.216693+0000","last_unstale":"2024-02-10T08:52:09.933505+0000","last_undegraded":"2024-02-10T08:52:09.933505+0000","last_fullsized":"2024-02-10T08:52:09.933505+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.866690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:39.567819+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.866690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":835,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333489+0000","last_change":"2024-02-10T08:52:10.333022+0000","last_active":"2024-02-10T08:52:10.333489+0000","last_peered":"2024-02-10T08:52:10.333489+0000","last_clean":"2024-02-10T08:52:10.333489+0000","last_became_active":"2024-02-10T08:46:15.347671+0000","last_became_peered":"2024-02-10T08:46:15.347671+0000","last_unstale":"2024-02-10T08:52:10.333489+0000","last_undegraded":"2024-02-10T08:52:10.333489+0000","last_fullsized":"2024-02-10T08:52:10.333489+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:10.332923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:52:10.332923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":232,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925491+0000","last_change":"2024-02-10T08:50:56.603701+0000","last_active":"2024-02-10T08:52:09.925491+0000","last_peered":"2024-02-10T08:52:09.925491+0000","last_clean":"2024-02-10T08:52:09.925491+0000","last_became_active":"2024-02-10T08:46:15.361021+0000","last_became_peered":"2024-02-10T08:46:15.361021+0000","last_unstale":"2024-02-10T08:52:09.925491+0000","last_undegraded":"2024-02-10T08:52:09.925491+0000","last_fullsized":"2024-02-10T08:52:09.925491+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:56.603602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:47:29.348714+0000","last_clean_scrub_stamp":"2024-02-10T08:50:56.603602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"473'20","reported_seq":244,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925589+0000","last_change":"2024-02-10T08:51:34.334569+0000","last_active":"2024-02-10T08:52:09.925589+0000","last_peered":"2024-02-10T08:52:09.925589+0000","last_clean":"2024-02-10T08:52:09.925589+0000","last_became_active":"2024-02-10T08:46:15.366425+0000","last_became_peered":"2024-02-10T08:46:15.366425+0000","last_unstale":"2024-02-10T08:52:09.925589+0000","last_undegraded":"2024-02-10T08:52:09.925589+0000","last_fullsized":"2024-02-10T08:52:09.925589+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'20","last_scrub_stamp":"2024-02-10T08:51:34.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:34.334504+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":863,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332569+0000","last_change":"2024-02-10T08:51:52.646508+0000","last_active":"2024-02-10T08:52:10.332569+0000","last_peered":"2024-02-10T08:52:10.332569+0000","last_clean":"2024-02-10T08:52:10.332569+0000","last_became_active":"2024-02-10T08:46:29.214755+0000","last_became_peered":"2024-02-10T08:46:29.214755+0000","last_unstale":"2024-02-10T08:52:10.332569+0000","last_undegraded":"2024-02-10T08:52:10.332569+0000","last_fullsized":"2024-02-10T08:52:10.332569+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:52.646424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:24.469783+0000","last_clean_scrub_stamp":"2024-02-10T08:51:52.646424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"763'2","reported_seq":410,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.933990+0000","last_change":"2024-02-10T08:51:48.917250+0000","last_active":"2024-02-10T08:52:09.933990+0000","last_peered":"2024-02-10T08:52:09.933990+0000","last_clean":"2024-02-10T08:52:09.933990+0000","last_became_active":"2024-02-10T08:43:15.118159+0000","last_became_peered":"2024-02-10T08:43:15.118159+0000","last_unstale":"2024-02-10T08:52:09.933990+0000","last_undegraded":"2024-02-10T08:52:09.933990+0000","last_fullsized":"2024-02-10T08:52:09.933990+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'2","last_scrub_stamp":"2024-02-10T08:51:48.917167+0000","last_deep_scrub":"763'2","last_deep_scrub_stamp":"2024-02-10T08:48:50.762672+0000","last_clean_scrub_stamp":"2024-02-10T08:51:48.917167+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":812,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703416+0000","last_change":"2024-02-10T08:51:54.996240+0000","last_active":"2024-02-10T08:52:09.703416+0000","last_peered":"2024-02-10T08:52:09.703416+0000","last_clean":"2024-02-10T08:52:09.703416+0000","last_became_active":"2024-02-10T08:47:49.229494+0000","last_became_peered":"2024-02-10T08:47:49.229494+0000","last_unstale":"2024-02-10T08:52:09.703416+0000","last_undegraded":"2024-02-10T08:52:09.703416+0000","last_fullsized":"2024-02-10T08:52:09.703416+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:54.996111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:54.996111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"473'19","reported_seq":906,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335990+0000","last_change":"2024-02-10T08:52:07.856178+0000","last_active":"2024-02-10T08:52:10.335990+0000","last_peered":"2024-02-10T08:52:10.335990+0000","last_clean":"2024-02-10T08:52:10.335990+0000","last_became_active":"2024-02-10T08:47:49.222711+0000","last_became_peered":"2024-02-10T08:47:49.222711+0000","last_unstale":"2024-02-10T08:52:10.335990+0000","last_undegraded":"2024-02-10T08:52:10.335990+0000","last_fullsized":"2024-02-10T08:52:10.335990+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'19","last_scrub_stamp":"2024-02-10T08:52:07.856094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:52:07.856094+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":841,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332904+0000","last_change":"2024-02-10T08:52:04.393940+0000","last_active":"2024-02-10T08:52:10.332904+0000","last_peered":"2024-02-10T08:52:10.332904+0000","last_clean":"2024-02-10T08:52:10.332904+0000","last_became_active":"2024-02-10T08:46:15.349951+0000","last_became_peered":"2024-02-10T08:46:15.349951+0000","last_unstale":"2024-02-10T08:52:10.332904+0000","last_undegraded":"2024-02-10T08:52:10.332904+0000","last_fullsized":"2024-02-10T08:52:10.332904+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:04.393877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:04.393877+0000","last_clean_scrub_stamp":"2024-02-10T08:52:04.393877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":150,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701432+0000","last_change":"2024-02-10T08:50:54.357965+0000","last_active":"2024-02-10T08:52:09.701432+0000","last_peered":"2024-02-10T08:52:09.701432+0000","last_clean":"2024-02-10T08:52:09.701432+0000","last_became_active":"2024-02-10T08:47:49.248914+0000","last_became_peered":"2024-02-10T08:47:49.248914+0000","last_unstale":"2024-02-10T08:52:09.701432+0000","last_undegraded":"2024-02-10T08:52:09.701432+0000","last_fullsized":"2024-02-10T08:52:09.701432+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.357895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":155,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701143+0000","last_change":"2024-02-10T08:51:21.495313+0000","last_active":"2024-02-10T08:52:09.701143+0000","last_peered":"2024-02-10T08:52:09.701143+0000","last_clean":"2024-02-10T08:52:09.701143+0000","last_became_active":"2024-02-10T08:47:49.234953+0000","last_became_peered":"2024-02-10T08:47:49.234953+0000","last_unstale":"2024-02-10T08:52:09.701143+0000","last_undegraded":"2024-02-10T08:52:09.701143+0000","last_fullsized":"2024-02-10T08:52:09.701143+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:21.495232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:28.906474+0000","last_clean_scrub_stamp":"2024-02-10T08:51:21.495232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"473'34","reported_seq":148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.703589+0000","last_change":"2024-02-10T08:51:19.817098+0000","last_active":"2024-02-10T08:52:09.703589+0000","last_peered":"2024-02-10T08:52:09.703589+0000","last_clean":"2024-02-10T08:52:09.703589+0000","last_became_active":"2024-02-10T08:48:10.219414+0000","last_became_peered":"2024-02-10T08:48:10.219414+0000","last_unstale":"2024-02-10T08:52:09.703589+0000","last_undegraded":"2024-02-10T08:52:09.703589+0000","last_fullsized":"2024-02-10T08:52:09.703589+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'34","last_scrub_stamp":"2024-02-10T08:51:19.817010+0000","last_deep_scrub":"473'34","last_deep_scrub_stamp":"2024-02-10T08:48:52.097161+0000","last_clean_scrub_stamp":"2024-02-10T08:51:19.817010+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":853,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.335511+0000","last_change":"2024-02-10T08:51:19.948572+0000","last_active":"2024-02-10T08:52:10.335511+0000","last_peered":"2024-02-10T08:52:10.335511+0000","last_clean":"2024-02-10T08:52:10.335511+0000","last_became_active":"2024-02-10T08:46:29.214895+0000","last_became_peered":"2024-02-10T08:46:29.214895+0000","last_unstale":"2024-02-10T08:52:10.335511+0000","last_undegraded":"2024-02-10T08:52:10.335511+0000","last_fullsized":"2024-02-10T08:52:10.335511+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:19.948488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:19.948488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":661,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701790+0000","last_change":"2024-02-10T08:51:44.920711+0000","last_active":"2024-02-10T08:52:09.701790+0000","last_peered":"2024-02-10T08:52:09.701790+0000","last_clean":"2024-02-10T08:52:09.701790+0000","last_became_active":"2024-02-10T08:46:29.218432+0000","last_became_peered":"2024-02-10T08:46:29.218432+0000","last_unstale":"2024-02-10T08:52:09.701790+0000","last_undegraded":"2024-02-10T08:52:09.701790+0000","last_fullsized":"2024-02-10T08:52:09.701790+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:44.920624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:44.920624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925515+0000","last_change":"2024-02-10T08:50:54.556265+0000","last_active":"2024-02-10T08:52:09.925515+0000","last_peered":"2024-02-10T08:52:09.925515+0000","last_clean":"2024-02-10T08:52:09.925515+0000","last_became_active":"2024-02-10T08:47:49.240426+0000","last_became_peered":"2024-02-10T08:47:49.240426+0000","last_unstale":"2024-02-10T08:52:09.925515+0000","last_undegraded":"2024-02-10T08:52:09.925515+0000","last_fullsized":"2024-02-10T08:52:09.925515+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:54.556197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.556197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"473'18","reported_seq":257,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333352+0000","last_change":"2024-02-10T08:51:29.212829+0000","last_active":"2024-02-10T08:52:10.333352+0000","last_peered":"2024-02-10T08:52:10.333352+0000","last_clean":"2024-02-10T08:52:10.333352+0000","last_became_active":"2024-02-10T08:47:49.224837+0000","last_became_peered":"2024-02-10T08:47:49.224837+0000","last_unstale":"2024-02-10T08:52:10.333352+0000","last_undegraded":"2024-02-10T08:52:10.333352+0000","last_fullsized":"2024-02-10T08:52:10.333352+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'18","last_scrub_stamp":"2024-02-10T08:51:29.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:29.212763+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":841,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333247+0000","last_change":"2024-02-10T08:51:47.281899+0000","last_active":"2024-02-10T08:52:10.333247+0000","last_peered":"2024-02-10T08:52:10.333247+0000","last_clean":"2024-02-10T08:52:10.333247+0000","last_became_active":"2024-02-10T08:43:15.125717+0000","last_became_peered":"2024-02-10T08:43:15.125717+0000","last_unstale":"2024-02-10T08:52:10.333247+0000","last_undegraded":"2024-02-10T08:52:10.333247+0000","last_fullsized":"2024-02-10T08:52:10.333247+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:47.281837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:47.281837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":817,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333734+0000","last_change":"2024-02-10T08:52:07.332969+0000","last_active":"2024-02-10T08:52:10.333734+0000","last_peered":"2024-02-10T08:52:10.333734+0000","last_clean":"2024-02-10T08:52:10.333734+0000","last_became_active":"2024-02-10T08:33:24.126929+0000","last_became_peered":"2024-02-10T08:33:24.126929+0000","last_unstale":"2024-02-10T08:52:10.333734+0000","last_undegraded":"2024-02-10T08:52:10.333734+0000","last_fullsized":"2024-02-10T08:52:10.333734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:07.332902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:51.264745+0000","last_clean_scrub_stamp":"2024-02-10T08:52:07.332902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":166,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701098+0000","last_change":"2024-02-10T08:52:06.483942+0000","last_active":"2024-02-10T08:52:09.701098+0000","last_peered":"2024-02-10T08:52:09.701098+0000","last_clean":"2024-02-10T08:52:09.701098+0000","last_became_active":"2024-02-10T08:48:10.213884+0000","last_became_peered":"2024-02-10T08:48:10.213884+0000","last_unstale":"2024-02-10T08:52:09.701098+0000","last_undegraded":"2024-02-10T08:52:09.701098+0000","last_fullsized":"2024-02-10T08:52:09.701098+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.483874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:50:51.275205+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.483874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"832'19","reported_seq":722,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701337+0000","last_change":"2024-02-10T08:51:22.002805+0000","last_active":"2024-02-10T08:52:09.701337+0000","last_peered":"2024-02-10T08:52:09.701337+0000","last_clean":"2024-02-10T08:52:09.701337+0000","last_became_active":"2024-02-10T08:46:30.197007+0000","last_became_peered":"2024-02-10T08:46:30.197007+0000","last_unstale":"2024-02-10T08:52:09.701337+0000","last_undegraded":"2024-02-10T08:52:09.701337+0000","last_fullsized":"2024-02-10T08:52:09.701337+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'19","last_scrub_stamp":"2024-02-10T08:51:22.002701+0000","last_deep_scrub":"832'19","last_deep_scrub_stamp":"2024-02-10T08:51:22.002701+0000","last_clean_scrub_stamp":"2024-02-10T08:51:22.002701+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":407,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.934031+0000","last_change":"2024-02-10T08:51:53.835128+0000","last_active":"2024-02-10T08:52:09.934031+0000","last_peered":"2024-02-10T08:52:09.934031+0000","last_clean":"2024-02-10T08:52:09.934031+0000","last_became_active":"2024-02-10T08:43:15.122347+0000","last_became_peered":"2024-02-10T08:43:15.122347+0000","last_unstale":"2024-02-10T08:52:09.934031+0000","last_undegraded":"2024-02-10T08:52:09.934031+0000","last_fullsized":"2024-02-10T08:52:09.934031+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:53.835040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:43:59.379146+0000","last_clean_scrub_stamp":"2024-02-10T08:51:53.835040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":250,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925618+0000","last_change":"2024-02-10T08:50:53.552617+0000","last_active":"2024-02-10T08:52:09.925618+0000","last_peered":"2024-02-10T08:52:09.925618+0000","last_clean":"2024-02-10T08:52:09.925618+0000","last_became_active":"2024-02-10T08:47:49.240864+0000","last_became_peered":"2024-02-10T08:47:49.240864+0000","last_unstale":"2024-02-10T08:52:09.925618+0000","last_undegraded":"2024-02-10T08:52:09.925618+0000","last_fullsized":"2024-02-10T08:52:09.925618+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:53.552521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:50:53.552521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":806,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.928908+0000","last_change":"2024-02-10T08:52:09.928881+0000","last_active":"2024-02-10T08:52:09.928908+0000","last_peered":"2024-02-10T08:52:09.928908+0000","last_clean":"2024-02-10T08:52:09.928908+0000","last_became_active":"2024-02-10T08:46:30.200804+0000","last_became_peered":"2024-02-10T08:46:30.200804+0000","last_unstale":"2024-02-10T08:52:09.928908+0000","last_undegraded":"2024-02-10T08:52:09.928908+0000","last_fullsized":"2024-02-10T08:52:09.928908+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:09.928813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:52:09.928813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"473'9","reported_seq":721,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701648+0000","last_change":"2024-02-10T08:52:06.809920+0000","last_active":"2024-02-10T08:52:09.701648+0000","last_peered":"2024-02-10T08:52:09.701648+0000","last_clean":"2024-02-10T08:52:09.701648+0000","last_became_active":"2024-02-10T08:47:49.244203+0000","last_became_peered":"2024-02-10T08:47:49.244203+0000","last_unstale":"2024-02-10T08:52:09.701648+0000","last_undegraded":"2024-02-10T08:52:09.701648+0000","last_fullsized":"2024-02-10T08:52:09.701648+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'9","last_scrub_stamp":"2024-02-10T08:52:06.809842+0000","last_deep_scrub":"257'7","last_deep_scrub_stamp":"2024-02-10T08:42:03.646375+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.809842+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":353,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333378+0000","last_change":"2024-02-10T08:51:59.264492+0000","last_active":"2024-02-10T08:52:10.333378+0000","last_peered":"2024-02-10T08:52:10.333378+0000","last_clean":"2024-02-10T08:52:10.333378+0000","last_became_active":"2024-02-10T08:46:30.198332+0000","last_became_peered":"2024-02-10T08:46:30.198332+0000","last_unstale":"2024-02-10T08:52:10.333378+0000","last_undegraded":"2024-02-10T08:52:10.333378+0000","last_fullsized":"2024-02-10T08:52:10.333378+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.264427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:38:30.389076+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.264427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":833,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332646+0000","last_change":"2024-02-10T08:51:59.517087+0000","last_active":"2024-02-10T08:52:10.332646+0000","last_peered":"2024-02-10T08:52:10.332646+0000","last_clean":"2024-02-10T08:52:10.332646+0000","last_became_active":"2024-02-10T08:47:49.237095+0000","last_became_peered":"2024-02-10T08:47:49.237095+0000","last_unstale":"2024-02-10T08:52:10.332646+0000","last_undegraded":"2024-02-10T08:52:10.332646+0000","last_fullsized":"2024-02-10T08:52:10.332646+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:59.516966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:59.516966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":231,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925826+0000","last_change":"2024-02-10T08:50:42.585429+0000","last_active":"2024-02-10T08:52:09.925826+0000","last_peered":"2024-02-10T08:52:09.925826+0000","last_clean":"2024-02-10T08:52:09.925826+0000","last_became_active":"2024-02-10T08:46:15.361459+0000","last_became_peered":"2024-02-10T08:46:15.361459+0000","last_unstale":"2024-02-10T08:52:09.925826+0000","last_undegraded":"2024-02-10T08:52:09.925826+0000","last_fullsized":"2024-02-10T08:52:09.925826+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:50:42.585363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:50:42.585363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"473'15","reported_seq":753,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.701015+0000","last_change":"2024-02-10T08:50:54.877195+0000","last_active":"2024-02-10T08:52:09.701015+0000","last_peered":"2024-02-10T08:52:09.701015+0000","last_clean":"2024-02-10T08:52:09.701015+0000","last_became_active":"2024-02-10T08:47:49.242072+0000","last_became_peered":"2024-02-10T08:47:49.242072+0000","last_unstale":"2024-02-10T08:52:09.701015+0000","last_undegraded":"2024-02-10T08:52:09.701015+0000","last_fullsized":"2024-02-10T08:52:09.701015+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'15","last_scrub_stamp":"2024-02-10T08:50:54.877106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:50:54.877106+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":836,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333629+0000","last_change":"2024-02-10T08:51:06.203366+0000","last_active":"2024-02-10T08:52:10.333629+0000","last_peered":"2024-02-10T08:52:10.333629+0000","last_clean":"2024-02-10T08:52:10.333629+0000","last_became_active":"2024-02-10T08:46:15.347761+0000","last_became_peered":"2024-02-10T08:46:15.347761+0000","last_unstale":"2024-02-10T08:52:10.333629+0000","last_undegraded":"2024-02-10T08:52:10.333629+0000","last_fullsized":"2024-02-10T08:52:10.333629+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:06.203294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:17.081938+0000","last_clean_scrub_stamp":"2024-02-10T08:51:06.203294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":676,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-02-10T08:52:11.790259+0000","last_change":"2024-02-10T08:52:11.790259+0000","last_active":"2024-02-10T08:52:11.790259+0000","last_peered":"2024-02-10T08:52:11.790259+0000","last_clean":"2024-02-10T08:52:11.790259+0000","last_became_active":"2024-02-10T08:47:49.241245+0000","last_became_peered":"2024-02-10T08:47:49.241245+0000","last_unstale":"2024-02-10T08:52:11.790259+0000","last_undegraded":"2024-02-10T08:52:11.790259+0000","last_fullsized":"2024-02-10T08:52:11.790259+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:11.790177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:52:11.790177+0000","last_clean_scrub_stamp":"2024-02-10T08:52:11.790177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":794,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.332806+0000","last_change":"2024-02-10T08:51:28.709454+0000","last_active":"2024-02-10T08:52:10.332806+0000","last_peered":"2024-02-10T08:52:10.332806+0000","last_clean":"2024-02-10T08:52:10.332806+0000","last_became_active":"2024-02-10T08:46:30.199644+0000","last_became_peered":"2024-02-10T08:46:30.199644+0000","last_unstale":"2024-02-10T08:52:10.332806+0000","last_undegraded":"2024-02-10T08:52:10.332806+0000","last_fullsized":"2024-02-10T08:52:10.332806+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:28.709361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:36:35.870864+0000","last_clean_scrub_stamp":"2024-02-10T08:51:28.709361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"473'27","reported_seq":909,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333068+0000","last_change":"2024-02-10T08:51:35.177551+0000","last_active":"2024-02-10T08:52:10.333068+0000","last_peered":"2024-02-10T08:52:10.333068+0000","last_clean":"2024-02-10T08:52:10.333068+0000","last_became_active":"2024-02-10T08:46:15.349100+0000","last_became_peered":"2024-02-10T08:46:15.349100+0000","last_unstale":"2024-02-10T08:52:10.333068+0000","last_undegraded":"2024-02-10T08:52:10.333068+0000","last_fullsized":"2024-02-10T08:52:10.333068+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'27","last_scrub_stamp":"2024-02-10T08:51:35.177438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:19.093397+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.177438+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":248,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:09.925874+0000","last_change":"2024-02-10T08:51:35.323278+0000","last_active":"2024-02-10T08:52:09.925874+0000","last_peered":"2024-02-10T08:52:09.925874+0000","last_clean":"2024-02-10T08:52:09.925874+0000","last_became_active":"2024-02-10T08:46:30.199613+0000","last_became_peered":"2024-02-10T08:46:30.199613+0000","last_unstale":"2024-02-10T08:52:09.925874+0000","last_undegraded":"2024-02-10T08:52:09.925874+0000","last_fullsized":"2024-02-10T08:52:09.925874+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:35.323215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:46:44.049633+0000","last_clean_scrub_stamp":"2024-02-10T08:51:35.323215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":832,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.333139+0000","last_change":"2024-02-10T08:51:36.157770+0000","last_active":"2024-02-10T08:52:10.333139+0000","last_peered":"2024-02-10T08:52:10.333139+0000","last_clean":"2024-02-10T08:52:10.333139+0000","last_became_active":"2024-02-10T08:43:15.136781+0000","last_became_peered":"2024-02-10T08:43:15.136781+0000","last_unstale":"2024-02-10T08:52:10.333139+0000","last_undegraded":"2024-02-10T08:52:10.333139+0000","last_fullsized":"2024-02-10T08:52:10.333139+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:51:36.157645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:33:23.102654+0000","last_clean_scrub_stamp":"2024-02-10T08:51:36.157645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":810,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-02-10T08:52:10.334909+0000","last_change":"2024-02-10T08:52:06.891432+0000","last_active":"2024-02-10T08:52:10.334909+0000","last_peered":"2024-02-10T08:52:10.334909+0000","last_clean":"2024-02-10T08:52:10.334909+0000","last_became_active":"2024-02-10T08:48:10.218684+0000","last_became_peered":"2024-02-10T08:48:10.218684+0000","last_unstale":"2024-02-10T08:52:10.334909+0000","last_undegraded":"2024-02-10T08:52:10.334909+0000","last_fullsized":"2024-02-10T08:52:10.334909+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T08:52:06.891358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T08:48:03.702592+0000","last_clean_scrub_stamp":"2024-02-10T08:52:06.891358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":11310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299622,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311452,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433700,"statfs":{"total":95995035648,"available":95676108800,"internally_reserved":0,"allocated":1789952,"data_stored":545607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.60399999999999998},"min":{"1min":0.314,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64600000000000002,"5min":1.3049999999999999,"15min":22.143000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.61599999999999999,"5min":1.0740000000000001,"15min":11.000999999999999},"last":0.625}]},{"osd":1,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.60899999999999999},"min":{"1min":0.36299999999999999,"5min":0.28100000000000003,"15min":0.27200000000000002},"max":{"1min":0.65400000000000003,"5min":1.333,"15min":9.2560000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.49199999999999999,"15min":0.60899999999999999},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.22500000000000001},"max":{"1min":0.621,"5min":1.1240000000000001,"15min":10.978},"last":0.69299999999999995}]},{"osd":2,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49399999999999999,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.253},"max":{"1min":0.623,"5min":0.81599999999999995,"15min":7.3949999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.66400000000000003},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.251},"max":{"1min":0.76100000000000001,"5min":1.3200000000000001,"15min":19.369},"last":0.68200000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.505,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.78300000000000003,"15min":9.3439999999999994},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55000000000000004,"15min":0.69799999999999995},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.314},"max":{"1min":0.72399999999999998,"5min":1.345,"15min":22.102},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64900000000000002,"5min":1.581,"15min":1.581},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58199999999999996,"15min":0.68200000000000005},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79600000000000004,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.86399999999999999,"5min":4.46,"15min":19.326000000000001},"last":0.501},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.51900000000000002},"min":{"1min":0.38200000000000001,"5min":0.27100000000000002,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":6.8650000000000002,"15min":10.946999999999999},"last":0.66700000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.38800000000000001,"5min":0.26800000000000002,"15min":0.23000000000000001},"max":{"1min":0.80800000000000005,"5min":4.718,"15min":22.687000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.437,"5min":0.27800000000000002,"15min":0.25900000000000001},"max":{"1min":0.71299999999999997,"5min":1.8899999999999999,"15min":5.484},"last":0.53500000000000003}]}]},{"osd":6,"up_from":44,"seq":188978561260,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441904,"statfs":{"total":95995035648,"available":95684509696,"internally_reserved":0,"allocated":1777664,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47499999999999998,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.69299999999999995,"5min":0.76500000000000001,"15min":8.5250000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.30599999999999999,"15min":0.254},"max":{"1min":0.75700000000000001,"5min":2.649,"15min":18.777999999999999},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.65500000000000003},"min":{"1min":0.38300000000000001,"5min":0.26400000000000001,"15min":0.23999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":18.425999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":2.4769999999999999,"15min":18.402999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.63},"min":{"1min":0.40500000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71899999999999997,"5min":2.4319999999999999,"15min":10.772},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":0.69499999999999995,"5min":2.3570000000000002,"15min":18.829000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"min":{"1min":0.34499999999999997,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":9.2650000000000006,"15min":16.823},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53300000000000003,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":2.5840000000000001,"15min":3.7389999999999999},"last":0.502}]},{"osd":4,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.36699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.69899999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.49399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68999999999999995,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.38500000000000001}]},{"osd":5,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.27600000000000002,"15min":0.224},"max":{"1min":0.75,"5min":0.75,"15min":18.853999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.33400000000000002,"5min":0.29499999999999998,"15min":0.26000000000000001},"max":{"1min":0.73999999999999999,"5min":0.76500000000000001,"15min":3.915},"last":0.40000000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":7.5650000000000004},"last":0.434},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.0229999999999999},"last":0.45300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303272,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441880,"statfs":{"total":95995035648,"available":95684485120,"internally_reserved":0,"allocated":1802240,"data_stored":539217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.45100000000000001},"min":{"1min":0.23499999999999999,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.48599999999999999,"5min":1.5669999999999999,"15min":7.8780000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.25800000000000001,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":0.51600000000000001,"5min":1.2669999999999999,"15min":16.166},"last":0.442}]},{"osd":2,"last update":"Sat Feb 10 08:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.65500000000000003},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.193},"max":{"1min":0.55200000000000005,"5min":1.637,"15min":34.985999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.55700000000000005},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.47399999999999998,"5min":0.93100000000000005,"15min":21.614999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35499999999999998,"15min":0.51600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.47099999999999997,"5min":0.73199999999999998,"15min":24.773},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.505},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45800000000000002,"5min":0.80000000000000004,"15min":16.120999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81299999999999994,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58799999999999997,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63}]},{"osd":5,"last update":"Sat Feb 10 08:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.34599999999999997,"15min":0.26800000000000002},"max":{"1min":0.77600000000000002,"5min":0.86899999999999999,"15min":9.5600000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.94699999999999995,"15min":7.8479999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57499999999999996,"15min":0.61899999999999999},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":0.76600000000000001,"5min":0.91500000000000004,"15min":9.532},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60499999999999998,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.33200000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":7.9000000000000004},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.67000000000000004},"min":{"1min":0.42599999999999999,"5min":0.39400000000000002,"15min":0.34799999999999998},"max":{"1min":0.75,"5min":0.88300000000000001,"15min":12.147},"last":0.623},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.70599999999999996},"min":{"1min":0.50700000000000001,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":7.9199999999999999},"last":0.67800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738633,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441908,"statfs":{"total":95995035648,"available":95684513792,"internally_reserved":0,"allocated":1773568,"data_stored":537231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 08:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.34999999999999998,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.156},"max":{"1min":0.58099999999999996,"5min":1.4239999999999999,"15min":11.960000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.379,"15min":0.496},"min":{"1min":0.223,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":3.8839999999999999,"15min":20.757999999999999},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34899999999999998,"15min":0.44900000000000001},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.20899999999999999},"max":{"1min":0.46700000000000003,"5min":0.79400000000000004,"15min":11.987},"last":0.245},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34999999999999998,"15min":0.48099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.19400000000000001},"max":{"1min":0.47899999999999998,"5min":0.64000000000000001,"15min":11.303000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.23599999999999999,"15min":0.22800000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":8.5310000000000006},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.442,"15min":0.47699999999999998},"min":{"1min":0.28100000000000003,"5min":0.27200000000000002,"15min":0.22600000000000001},"max":{"1min":0.86399999999999999,"5min":4.0780000000000003,"15min":5.9779999999999998},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82199999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Feb 10 08:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63300000000000001,"15min":0.59299999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.67700000000000005,"5min":3.931,"15min":4.609},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.43099999999999999,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":0.72099999999999997,"5min":3.2320000000000002,"15min":3.4079999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Feb 10 08:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.318},"max":{"1min":0.874,"5min":1.1910000000000001,"15min":3.3879999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.373,"15min":0.33300000000000002},"max":{"1min":0.85199999999999998,"5min":3.9609999999999999,"15min":8.5730000000000004},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Feb 10 08:51:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.629},"min":{"1min":0.47999999999999998,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":0.79400000000000004,"5min":0.94499999999999995,"15min":3.363},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61199999999999999,"15min":0.66400000000000003},"min":{"1min":0.52800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77500000000000002,"5min":0.93899999999999995,"15min":8.4299999999999997},"last":0.72899999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411583,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299156,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445996,"statfs":{"total":95995035648,"available":95688699904,"internally_reserved":0,"allocated":1781760,"data_stored":545315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.495},"min":{"1min":0.214,"5min":0.20699999999999999,"15min":0.183},"max":{"1min":0.53300000000000003,"5min":1.2909999999999999,"15min":16.170000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.55800000000000005},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.187},"max":{"1min":0.49099999999999999,"5min":1.458,"15min":14.858000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Feb 10 08:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34899999999999998,"15min":0.55200000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.499,"5min":0.63500000000000001,"15min":12.512},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.35199999999999998,"15min":0.52400000000000002},"min":{"1min":0.27900000000000003,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":12.875},"last":0.27600000000000002}]},{"osd":3,"last update":"Sat Feb 10 08:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35799999999999998,"15min":0.53900000000000003},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45600000000000002,"5min":1.3069999999999999,"15min":11.612},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41999999999999998,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.22600000000000001},"max":{"1min":0.49299999999999999,"5min":3.484,"15min":11.710000000000001},"last":0.316}]},{"osd":4,"last update":"Sat Feb 10 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53300000000000003,"15min":0.51500000000000001},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.45500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.996,"5min":1.298,"15min":1.298},"last":0.37}]},{"osd":5,"last update":"Sat Feb 10 08:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56899999999999995,"15min":0.70299999999999996},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.28799999999999998},"max":{"1min":0.82499999999999996,"5min":0.871,"15min":10.34},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57199999999999995,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.31900000000000001},"max":{"1min":0.83699999999999997,"5min":1.3380000000000001,"15min":9.6690000000000005},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Feb 10 08:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.58699999999999997,"15min":0.71899999999999997},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":10.372999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.59199999999999997,"15min":0.71399999999999997},"min":{"1min":0.49099999999999999,"5min":0.372,"15min":0.35699999999999998},"max":{"1min":0.95899999999999996,"5min":1.321,"15min":9.7420000000000009},"last":0.42899999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.70199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.33900000000000002},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":10.539},"last":0.497},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.59699999999999998,"15min":0.70999999999999996},"min":{"1min":0.47699999999999998,"5min":0.41499999999999998,"15min":0.32500000000000001},"max":{"1min":0.91100000000000003,"5min":1.357,"15min":9.6319999999999997},"last":0.44400000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182651,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299160,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445992,"statfs":{"total":95995035648,"available":95688695808,"internally_reserved":0,"allocated":1982464,"data_stored":640464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.57399999999999995},"min":{"1min":0.216,"5min":0.19900000000000001,"15min":0.184},"max":{"1min":0.45800000000000002,"5min":1.8879999999999999,"15min":23.443000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.40100000000000002,"15min":0.59499999999999997},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":0.58899999999999997,"5min":6.6369999999999996,"15min":11.808},"last":0.377}]},{"osd":1,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.41999999999999998,"15min":0.67400000000000004},"min":{"1min":0.25,"5min":0.215,"15min":0.184},"max":{"1min":0.56599999999999995,"5min":6.5810000000000004,"15min":20.902000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.373,"15min":0.60299999999999998},"min":{"1min":0.23100000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":2.9950000000000001,"15min":17.343},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Feb 10 08:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.49199999999999999,"15min":0.67800000000000005},"min":{"1min":0.28299999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51600000000000001,"5min":12.664999999999999,"15min":17.201000000000001},"last":0.312},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36499999999999999,"15min":0.63200000000000001},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57599999999999996,"5min":1.206,"15min":23.294},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.36699999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.40600000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Feb 10 08:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.69899999999999995},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":12.579000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.67100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.28799999999999998},"max":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":17.292000000000002},"last":0.54100000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.71699999999999997},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":1.244,"15min":16.091000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.74299999999999999},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.32100000000000001},"max":{"1min":0.67700000000000005,"5min":0.80400000000000005,"15min":17.260999999999999},"last":0.46300000000000002}]},{"osd":7,"last update":"Sat Feb 10 08:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.70399999999999996},"min":{"1min":0.42199999999999999,"5min":0.38900000000000001,"15min":0.35799999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":14.773999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.747},"min":{"1min":0.49199999999999999,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.82699999999999996,"15min":12.606999999999999},"last":0.42099999999999999}]}]},{"osd":4,"up_from":633,"seq":2718714298439,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309216,"kb_used_data":1752,"kb_used_omap":59,"kb_used_meta":307396,"kb_avail":93435936,"statfs":{"total":95995035648,"available":95678398464,"internally_reserved":0,"allocated":1794048,"data_stored":545973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60821,"internal_metadata":314774123},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.67600000000000005,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.54000000000000004},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72599999999999998,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70799999999999996,"5min":1.036,"15min":1.036},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69899999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.33800000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65300000000000002,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.436},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80900000000000005,"5min":0.87,"15min":0.87},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50600000000000001,"15min":0.53200000000000003},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.745,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78800000000000003,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.48699999999999999}]},{"osd":7,"last update":"Sat Feb 10 08:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.34899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57399999999999995,"15min":0.59999999999999998},"min":{"1min":0.40500000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46899999999999997}]}]},{"osd":5,"up_from":37,"seq":158913790193,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307344,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437808,"statfs":{"total":95995035648,"available":95680315392,"internally_reserved":0,"allocated":1777664,"data_stored":537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51500000000000001,"15min":1.054},"min":{"1min":0.31900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.73799999999999999,"5min":1.151,"15min":107.551},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49299999999999999,"15min":0.97499999999999998},"min":{"1min":0.372,"5min":0.29399999999999998,"15min":0.255},"max":{"1min":0.71899999999999997,"5min":2.2589999999999999,"15min":88.792000000000002},"last":0.67500000000000004}]},{"osd":1,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53600000000000003,"15min":1.0109999999999999},"min":{"1min":0.36299999999999999,"5min":0.28699999999999998,"15min":0.216},"max":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":107.205},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.88100000000000001},"min":{"1min":0.43099999999999999,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.74399999999999999,"5min":0.90000000000000002,"15min":72.272000000000006},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":1.034},"min":{"1min":0.38600000000000001,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.66600000000000004,"5min":1.4650000000000001,"15min":54.174999999999997},"last":0.626},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":1.073},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.26600000000000001},"max":{"1min":0.752,"5min":1.6559999999999999,"15min":86.183000000000007},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Feb 10 08:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52500000000000002,"15min":0.86299999999999999},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.70899999999999996,"5min":1.0529999999999999,"15min":44.395000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56000000000000005,"15min":1.288},"min":{"1min":0.40200000000000002,"5min":0.35699999999999998,"15min":0.29099999999999998},"max":{"1min":0.79000000000000004,"5min":1.7010000000000001,"15min":114.526},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Feb 10 08:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.48999999999999999},"min":{"1min":0.41399999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66100000000000003,"5min":1.002,"15min":1.002},"last":0.79600000000000004}]},{"osd":6,"last update":"Sat Feb 10 08:51:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47999999999999998,"15min":0.73099999999999998},"min":{"1min":0.34499999999999997,"5min":0.26800000000000002,"15min":0.223},"max":{"1min":0.70599999999999996,"5min":1.847,"15min":72.241},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.502,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.23499999999999999},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":22.469999999999999},"last":0.66400000000000003}]},{"osd":7,"last update":"Sat Feb 10 08:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.72799999999999998},"min":{"1min":0.40799999999999997,"5min":0.28100000000000003,"15min":0.24099999999999999},"max":{"1min":0.73499999999999999,"5min":1.1200000000000001,"15min":66.844999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55300000000000005,"15min":0.755},"min":{"1min":0.40400000000000003,"5min":0.313,"15min":0.253},"max":{"1min":0.71799999999999997,"5min":1.8720000000000001,"15min":72.257000000000005},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3101,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T08:52:18.100 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T08:52:18.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-10T08:52:18.368 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T08:52:18.950 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:17.285788+0000 mgr.y (mgr.14553) 761 : audit [DBG] from='client.54689 v1:172.21.15.107:0/3773392756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:18.950 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:17.621325+0000 mgr.y (mgr.14553) 762 : cluster [DBG] pgmap v1337: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:18.950 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:17.285788+0000 mgr.y (mgr.14553) 761 : audit [DBG] from='client.54689 v1:172.21.15.107:0/3773392756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:18.950 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:18 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:17.621325+0000 mgr.y (mgr.14553) 762 : cluster [DBG] pgmap v1337: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:17.285788+0000 mgr.y (mgr.14553) 761 : audit [DBG] from='client.54689 v1:172.21.15.107:0/3773392756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:52:19.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:17.621325+0000 mgr.y (mgr.14553) 762 : cluster [DBG] pgmap v1337: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T08:52:20.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:18.962008+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub starts 2024-02-10T08:52:20.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:18.965502+0000 osd.4 (osd.4) 158 : cluster [DBG] 2.f scrub ok 2024-02-10T08:52:20.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.242997+0000 osd.3 (osd.3) 457 : cluster [DBG] 16.16 deep-scrub starts 2024-02-10T08:52:20.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.244377+0000 osd.3 (osd.3) 458 : cluster [DBG] 16.16 deep-scrub ok 2024-02-10T08:52:20.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:18.962008+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub starts 2024-02-10T08:52:20.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:18.965502+0000 osd.4 (osd.4) 158 : cluster [DBG] 2.f scrub ok 2024-02-10T08:52:20.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.242997+0000 osd.3 (osd.3) 457 : cluster [DBG] 16.16 deep-scrub starts 2024-02-10T08:52:20.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:19 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.244377+0000 osd.3 (osd.3) 458 : cluster [DBG] 16.16 deep-scrub ok 2024-02-10T08:52:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:18.962008+0000 osd.4 (osd.4) 157 : cluster [DBG] 2.f scrub starts 2024-02-10T08:52:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:18.965502+0000 osd.4 (osd.4) 158 : cluster [DBG] 2.f scrub ok 2024-02-10T08:52:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.242997+0000 osd.3 (osd.3) 457 : cluster [DBG] 16.16 deep-scrub starts 2024-02-10T08:52:20.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.244377+0000 osd.3 (osd.3) 458 : cluster [DBG] 16.16 deep-scrub ok 2024-02-10T08:52:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.256+0000 7f03ba547700 1 -- 172.21.15.107:0/3503537620 >> v1:172.21.15.107:6789/0 conn(0x7f03b40fda50 legacy=0x7f03b40fde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.256+0000 7f03ba547700 1 -- 172.21.15.107:0/3503537620 shutdown_connections 2024-02-10T08:52:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.256+0000 7f03ba547700 1 -- 172.21.15.107:0/3503537620 >> 172.21.15.107:0/3503537620 conn(0x7f03b40f3640 msgr2=0x7f03b40f5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.256+0000 7f03ba547700 1 -- 172.21.15.107:0/3503537620 shutdown_connections 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.256+0000 7f03ba547700 1 -- 172.21.15.107:0/3503537620 wait complete. 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 librados: starting msgr at 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 librados: starting objecter 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 Processor -- start 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 -- start start 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 librados: setting wanted keys 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 librados: calling monclient init 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f03b41173c0 con 0x7f03b41007f0 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f03b41185a0 con 0x7f03b4104330 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03ba547700 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f03b41197a0 con 0x7f03b40fda50 2024-02-10T08:52:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b3fff700 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f03b40fda50 0x7f03b40f99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40526/0 (socket says 172.21.15.107:40526) 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b3fff700 1 -- 172.21.15.107:0/4104854515 learned_addr learned my addr 172.21.15.107:0/4104854515 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter ms_handle_connect 0x7f03b40fda50 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter ms_handle_connect 0x7f03b41007f0 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342190179 0 0) 0x7f03b41197a0 con 0x7f03b40fda50 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394003030 con 0x7f03b40fda50 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter ms_handle_connect 0x7f03b4104330 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 10 client.?.objecter resend_mon_ops 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.257+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147024219 0 0) 0x7f03b41173c0 con 0x7f03b41007f0 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b41197a0 con 0x7f03b41007f0 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802555031 0 0) 0x7f03b41185a0 con 0x7f03b4104330 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b41173c0 con 0x7f03b4104330 2024-02-10T08:52:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378943064 0 0) 0x7f0394003030 con 0x7f03b40fda50 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03b41185a0 con 0x7f03b40fda50 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f03a4002070 con 0x7f03b40fda50 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3973964498 0 0) 0x7f03b41197a0 con 0x7f03b41007f0 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0394003030 con 0x7f03b41007f0 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f039c001970 con 0x7f03b41007f0 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 144471718 0 0) 0x7f03b41173c0 con 0x7f03b4104330 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03b41197a0 con 0x7f03b4104330 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f03ac001b80 con 0x7f03b4104330 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2647638455 0 0) 0x7f03b41185a0 con 0x7f03b40fda50 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 >> v1:172.21.15.195:6789/0 conn(0x7f03b4104330 legacy=0x7f03b4115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 >> v1:172.21.15.107:6789/0 conn(0x7f03b41007f0 legacy=0x7f03b40fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b411a9a0 con 0x7f03b40fda50 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b41175b0 con 0x7f03b40fda50 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03ba547700 10 client.54695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4117a20 con 0x7f03b40fda50 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03ba547700 1 librados: init done 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 3797172637 0 0) 0x7f03a4002ba0 con 0x7f03b40fda50 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.258+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 633151545 0 0) 0x7f03a4003820 con 0x7f03b40fda50 2024-02-10T08:52:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.259+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0398004f20 con 0x7f03b40fda50 2024-02-10T08:52:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.259+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3695920119 0 0) 0x7f03a4002210 con 0x7f03b40fda50 2024-02-10T08:52:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(839..839 src has 787..839) v4 ==== 7473+0+0 (unknown 2033404990 0 0) 0x7f03a4078c90 con 0x7f03b40fda50 2024-02-10T08:52:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 10 client.54695.objecter ms_dispatch 0x7f03b4001550 osd_map(839..839 src has 787..839) v4 2024-02-10T08:52:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 3 client.54695.objecter handle_osd_map got epochs [839,839] > 0 2024-02-10T08:52:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 3 client.54695.objecter handle_osd_map decoding full epoch 839 2024-02-10T08:52:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 20 client.54695.objecter dump_active .. 0 homeless 2024-02-10T08:52:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.260+0000 7f03b1ffb700 10 client.54695.objecter ms_handle_connect 0x7f0394063c20 2024-02-10T08:52:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.264+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f03a404c3d0 con 0x7f03b40fda50 2024-02-10T08:52:20.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.426+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7f03980051d0 con 0x7f03b40fda50 2024-02-10T08:52:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.431+0000 7f03b1ffb700 1 -- 172.21.15.107:0/4104854515 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v30) v1 ==== 118+0+0 (unknown 1095616568 0 0) 0x7f03a404a840 con 0x7f03b40fda50 2024-02-10T08:52:20.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 10 librados: watch_flush enter 2024-02-10T08:52:20.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 10 librados: watch_flush exit 2024-02-10T08:52:20.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 20 client.54695.objecter shutdown clearing up homeless session... 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 10 client.54695.objecter successfully canceled tick 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 >> v1:172.21.15.107:6800/2634842009 conn(0x7f0394063c20 legacy=0x7f03940660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 >> v1:172.21.15.107:6790/0 conn(0x7f03b40fda50 legacy=0x7f03b40f99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 shutdown_connections 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 >> 172.21.15.107:0/4104854515 conn(0x7f03b40f3640 msgr2=0x7f03b41077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:52:20.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 shutdown_connections 2024-02-10T08:52:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 -- 172.21.15.107:0/4104854515 wait complete. 2024-02-10T08:52:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-02-10T08:52:20.433+0000 7f03ba547700 1 librados: shutdown 2024-02-10T08:52:20.537 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x[71034]: ::ffff:172.21.15.195 - - [10/Feb/2024:08:52:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-10T08:52:20.740 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE Bus STOPPING 2024-02-10T08:52:20.740 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:52:20.741 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE Bus STOPPED 2024-02-10T08:52:20.741 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE Bus STARTING 2024-02-10T08:52:20.741 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE Serving on http://:::9283 2024-02-10T08:52:20.741 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:20] ENGINE Bus STARTED 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.373487+0000 osd.0 (osd.0) 250 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.374696+0000 osd.0 (osd.0) 251 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.621740+0000 mgr.y (mgr.14553) 763 : cluster [DBG] pgmap v1338: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.962069+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.e scrub starts 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: cluster 2024-02-10T08:52:19.965506+0000 osd.4 (osd.4) 160 : cluster [DBG] 5.e scrub ok 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.427730+0000 mon.c (mon.2) 523 : audit [INF] from='client.? v1:172.21.15.107:0/4104854515' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.428004+0000 mon.a (mon.0) 3793 : audit [INF] from='client.54695 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.431462+0000 mon.a (mon.0) 3794 : audit [INF] from='client.54695 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.456115+0000 mon.c (mon.2) 524 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.457472+0000 mon.c (mon.2) 525 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.458080+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: audit 2024-02-10T08:52:20.464101+0000 mon.a (mon.0) 3795 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.373487+0000 osd.0 (osd.0) 250 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.374696+0000 osd.0 (osd.0) 251 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.621740+0000 mgr.y (mgr.14553) 763 : cluster [DBG] pgmap v1338: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.962069+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.e scrub starts 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: cluster 2024-02-10T08:52:19.965506+0000 osd.4 (osd.4) 160 : cluster [DBG] 5.e scrub ok 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.427730+0000 mon.c (mon.2) 523 : audit [INF] from='client.? v1:172.21.15.107:0/4104854515' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.428004+0000 mon.a (mon.0) 3793 : audit [INF] from='client.54695 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.431462+0000 mon.a (mon.0) 3794 : audit [INF] from='client.54695 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T08:52:21.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.456115+0000 mon.c (mon.2) 524 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:52:21.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.457472+0000 mon.c (mon.2) 525 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:52:21.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.458080+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:52:21.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: audit 2024-02-10T08:52:20.464101+0000 mon.a (mon.0) 3795 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.373487+0000 osd.0 (osd.0) 250 : cluster [DBG] 16.18 scrub starts 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.374696+0000 osd.0 (osd.0) 251 : cluster [DBG] 16.18 scrub ok 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.621740+0000 mgr.y (mgr.14553) 763 : cluster [DBG] pgmap v1338: 154 pgs: 154 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.962069+0000 osd.4 (osd.4) 159 : cluster [DBG] 5.e scrub starts 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: cluster 2024-02-10T08:52:19.965506+0000 osd.4 (osd.4) 160 : cluster [DBG] 5.e scrub ok 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.427730+0000 mon.c (mon.2) 523 : audit [INF] from='client.? v1:172.21.15.107:0/4104854515' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.428004+0000 mon.a (mon.0) 3793 : audit [INF] from='client.54695 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.431462+0000 mon.a (mon.0) 3794 : audit [INF] from='client.54695 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.456115+0000 mon.c (mon.2) 524 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:52:21.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.457472+0000 mon.c (mon.2) 525 : audit [DBG] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:52:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.458080+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.14553 v1:172.21.15.107:0/2765981786' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T08:52:21.038 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: audit 2024-02-10T08:52:20.464101+0000 mon.a (mon.0) 3795 : audit [INF] from='mgr.14553 ' entity='mgr.y' 2024-02-10T08:52:21.047 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T08:52:21.057 INFO:tasks.cephadm:Teardown begin 2024-02-10T08:52:21.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:52:21.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:52:21.125 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T08:52:21.125 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T08:52:21.145 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T08:52:21.186 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T08:52:21.186 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T08:52:21.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a 2024-02-10T08:52:21.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:21 smithi107 systemd[1]: Stopping Ceph mon.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:21.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:21 smithi107 bash[289369]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon.a" found: no such container 2024-02-10T08:52:21.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:52:21.499+0000 7fe8ac9b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T08:52:21.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:21 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a[68223]: debug 2024-02-10T08:52:21.499+0000 7fe8ac9b7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T08:52:22.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Feb 10 08:52:22 smithi107 bash[289408]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-a 2024-02-10T08:52:22.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:52:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug there is no tcmu-runner data available 2024-02-10T08:52:22.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T08:52:22.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:22.563 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T08:52:22.564 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T08:52:22.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.c 2024-02-10T08:52:22.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:22 smithi107 systemd[1]: Stopping Ceph mon.c for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:22.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:22 smithi107 bash[289527]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon.c" found: no such container 2024-02-10T08:52:23.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:52:22.877+0000 7fc83bb40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T08:52:23.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c[74747]: debug 2024-02-10T08:52:22.877+0000 7fc83bb40700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T08:52:23.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE Bus STOPPING 2024-02-10T08:52:23.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T08:52:23.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE Bus STOPPED 2024-02-10T08:52:23.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE Bus STARTING 2024-02-10T08:52:23.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE Serving on http://:::9283 2024-02-10T08:52:23.181 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:22 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: [10/Feb/2024:08:52:22] ENGINE Bus STARTED 2024-02-10T08:52:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Feb 10 08:52:23 smithi107 bash[289566]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-c 2024-02-10T08:52:23.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T08:52:23.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:23.876 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T08:52:23.876 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T08:52:23.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.b 2024-02-10T08:52:24.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:23 smithi195 systemd[1]: Stopping Ceph mon.b for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:24.273 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:23 smithi195 bash[152690]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon.b" found: no such container 2024-02-10T08:52:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:52:24.272+0000 7fb681d80700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T08:52:24.537 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b[69563]: debug 2024-02-10T08:52:24.272+0000 7fb681d80700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T08:52:25.037 INFO:journalctl@ceph.mon.b.smithi195.stdout:Feb 10 08:52:24 smithi195 bash[152729]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mon-b 2024-02-10T08:52:25.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T08:52:25.186 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:25.186 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T08:52:25.186 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T08:52:25.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y 2024-02-10T08:52:25.488 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:25 smithi107 systemd[1]: Stopping Ceph mgr.y for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:25.488 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:25 smithi107 bash[289696]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr.y" found: no such container 2024-02-10T08:52:25.852 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:25 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y[68503]: debug 2024-02-10T08:52:25.486+0000 7f1427a57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:26.180 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Feb 10 08:52:25 smithi107 bash[289734]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-y 2024-02-10T08:52:26.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T08:52:26.423 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:26.423 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T08:52:26.424 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T08:52:26.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.x 2024-02-10T08:52:26.787 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:26 smithi195 systemd[1]: Stopping Ceph mgr.x for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:26.787 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:26 smithi195 bash[152848]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr.x" found: no such container 2024-02-10T08:52:27.281 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Feb 10 08:52:27 smithi195 bash[152889]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-mgr-x 2024-02-10T08:52:27.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T08:52:27.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:27.489 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T08:52:27.489 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T08:52:27.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.0 2024-02-10T08:52:27.787 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:27 smithi107 systemd[1]: Stopping Ceph osd.0 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:27.787 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:27 smithi107 bash[289854]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.0" found: no such container 2024-02-10T08:52:28.180 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0[78238]: debug 2024-02-10T08:52:27.785+0000 7faac2cad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:28.180 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0[78238]: debug 2024-02-10T08:52:27.785+0000 7faac2cad700 -1 osd.0 839 *** Got signal Terminated *** 2024-02-10T08:52:28.180 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:27 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0[78238]: debug 2024-02-10T08:52:27.785+0000 7faac2cad700 -1 osd.0 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:52:30.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:30.143Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:30.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:30.144Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:33.430 INFO:journalctl@ceph.osd.0.smithi107.stdout:Feb 10 08:52:33 smithi107 bash[289895]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-0 2024-02-10T08:52:33.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:33.577Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:33.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:33.577Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:35.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T08:52:35.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:35.313 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T08:52:35.313 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T08:52:35.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.1 2024-02-10T08:52:35.585 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:35 smithi107 systemd[1]: Stopping Ceph osd.1 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:35.585 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:35 smithi107 bash[290137]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.1" found: no such container 2024-02-10T08:52:35.930 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:52:35.583+0000 7f40be7e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:35.930 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:52:35.583+0000 7f40be7e8700 -1 osd.1 839 *** Got signal Terminated *** 2024-02-10T08:52:35.930 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:35 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1[82618]: debug 2024-02-10T08:52:35.583+0000 7f40be7e8700 -1 osd.1 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:52:40.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:52:40.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:40.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:40.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:40.431 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:40.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:41.180 INFO:journalctl@ceph.osd.1.smithi107.stdout:Feb 10 08:52:41 smithi107 bash[290178]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-1 2024-02-10T08:52:42.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T08:52:42.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:42.970 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T08:52:42.970 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T08:52:42.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.2 2024-02-10T08:52:43.180 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:43 smithi107 systemd[1]: Stopping Ceph osd.2 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:43.181 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:43 smithi107 bash[290418]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.2" found: no such container 2024-02-10T08:52:43.577 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2[87274]: debug 2024-02-10T08:52:43.304+0000 7fbb90530700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:43.577 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2[87274]: debug 2024-02-10T08:52:43.304+0000 7fbb90530700 -1 osd.2 839 *** Got signal Terminated *** 2024-02-10T08:52:43.577 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2[87274]: debug 2024-02-10T08:52:43.304+0000 7fbb90530700 -1 osd.2 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:52:43.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:52:43.577Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:43.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:43.577Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:43.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:43 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:43.578Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:48.930 INFO:journalctl@ceph.osd.2.smithi107.stdout:Feb 10 08:52:48 smithi107 bash[290457]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-2 2024-02-10T08:52:50.248 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:52:50.145Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:50.248 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:50.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:50.248 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:50.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:50.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T08:52:50.607 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:50.607 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T08:52:50.607 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T08:52:50.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.3 2024-02-10T08:52:50.924 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:50 smithi107 systemd[1]: Stopping Ceph osd.3 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:50.924 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:50 smithi107 bash[290694]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.3" found: no such container 2024-02-10T08:52:51.180 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3[91952]: debug 2024-02-10T08:52:50.923+0000 7f0ddb345700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:51.180 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3[91952]: debug 2024-02-10T08:52:50.923+0000 7f0ddb345700 -1 osd.3 839 *** Got signal Terminated *** 2024-02-10T08:52:51.180 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:50 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3[91952]: debug 2024-02-10T08:52:50.923+0000 7f0ddb345700 -1 osd.3 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:52:53.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:53.514+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:53.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:52:53.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:53.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:53.578Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:52:53.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:52:53 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:52:53.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:52:54.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:53 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:53.855+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:54.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:54.528+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:55.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:54.827+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:55.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:52:54 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:52:54.973+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:52:55.781 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:55.524+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:56.037 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:52:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:52:56.001+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:52:56.037 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:55 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:55.780+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:56.680 INFO:journalctl@ceph.osd.3.smithi107.stdout:Feb 10 08:52:56 smithi107 bash[290734]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-3 2024-02-10T08:52:57.014 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:56.743+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:52:57.014 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:56.542+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:57.015 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:56 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:56.785+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:57.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:52:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:52:57.013+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:52:58.016 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:57.774+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:52:58.017 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:57.562+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:58.017 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:57 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:57.802+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:58.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T08:52:58.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:52:58.107 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T08:52:58.108 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T08:52:58.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4 2024-02-10T08:52:58.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:52:58.016+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:52:58.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 systemd[1]: Stopping Ceph osd.4 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:52:58.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 bash[153021]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.4" found: no such container 2024-02-10T08:52:58.771 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:58.538+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:52:58.771 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:58.418+0000 7f568de81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:52:58.772 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:58.418+0000 7f568de81700 -1 osd.4 839 *** Got signal Terminated *** 2024-02-10T08:52:58.772 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:58.418+0000 7f568de81700 -1 osd.4 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:52:58.772 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:58.770+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:52:59.024 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:58.830+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:52:59.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:52:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:52:59.023+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:52:59.860 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:52:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:52:59.784+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:52:59.860 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:52:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:52:59.564+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:00.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:00.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:00.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:52:59 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:52:59.859+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:00.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:00.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:00.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:00.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:00.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:00 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:00.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:00.860 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:53:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:53:00.753+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:53:00.944 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:00.593+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:01.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:01.066+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:01.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:00 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:00.860+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:02.037 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:53:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:53:01.717+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:53:02.038 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:01.611+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:02.038 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:01 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:01.828+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:02.537 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:02.114+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:03.037 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:53:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4[137920]: debug 2024-02-10T08:53:02.684+0000 7f5686df6700 -1 osd.4 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.107797+0000 front 2024-02-10T08:52:31.107783+0000 (oldest deadline 2024-02-10T08:52:55.807733+0000) 2024-02-10T08:53:03.037 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:02.785+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:03.037 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:02.568+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:03.038 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:02 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:02.568+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:03.537 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:03.163+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:03.537 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:03.163+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:03.894 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:03.783+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:03.894 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:03.557+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:03.894 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:03 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:03.557+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:03.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:03.579Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:03.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:03.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:03.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:03 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:03.580Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:04.287 INFO:journalctl@ceph.osd.4.smithi195.stdout:Feb 10 08:53:03 smithi195 bash[153059]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-4 2024-02-10T08:53:04.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:04.117+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:04.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:04.117+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:04.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:04.576+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:04.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:04.576+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:05.115 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:04 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:04.828+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:05.493 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:05.114+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:05.493 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:05.114+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:05.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T08:53:05.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:05.542 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T08:53:05.542 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T08:53:05.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.5 2024-02-10T08:53:05.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:05.578+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:05.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:05.578+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:05.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 systemd[1]: Stopping Ceph osd.5 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:05.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 bash[153295]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.5" found: no such container 2024-02-10T08:53:06.093 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:05.891+0000 7fa4b3654700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:53:06.094 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:05.891+0000 7fa4b3654700 -1 osd.5 839 *** Got signal Terminated *** 2024-02-10T08:53:06.094 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:05.891+0000 7fa4b3654700 -1 osd.5 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:53:06.094 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:05.843+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:06.094 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:05 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:05.843+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:06.537 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:06.093+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:06.537 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:06.093+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:06.884 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:06.619+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:06.884 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:06.619+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:07.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:07.100+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:07.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:07.100+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:07.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:06.883+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:07.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:06 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:06.883+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:07.929 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:07.624+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:07.929 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:07.624+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:08.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:08.105+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:08.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:08.105+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:08.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:07.928+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:08.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:07 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:07.928+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:08.950 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:08.621+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:08.950 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:08.621+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:09.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:09.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:09.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:09.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:09.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:08.950+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:09.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:08 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:08.950+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:09.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:09.579+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:09.999 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:09.579+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:10.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.047+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:10.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.047+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:10.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.047+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:10.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:09.998+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:10.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:09.998+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:10.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:09 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:09.998+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:10.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:10.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:10.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:10.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:10.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:10 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:10.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:10.999 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:10.567+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:31.234494+0000 front 2024-02-10T08:52:31.234327+0000 (oldest deadline 2024-02-10T08:52:52.934371+0000) 2024-02-10T08:53:11.000 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:10 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5[78146]: debug 2024-02-10T08:53:10.567+0000 7fa4ac5c9700 -1 osd.5 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:40.535979+0000 front 2024-02-10T08:52:40.535988+0000 (oldest deadline 2024-02-10T08:53:02.235848+0000) 2024-02-10T08:53:11.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:11.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:11.288 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:10.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:11.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:11.033+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:11.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:11.033+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:11.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:11 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:11.033+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:11.787 INFO:journalctl@ceph.osd.5.smithi195.stdout:Feb 10 08:53:11 smithi195 bash[153335]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-5 2024-02-10T08:53:12.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:12.062+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:12.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:12.062+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:12.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:12.062+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:12.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:12.021+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:12.288 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:12.021+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:12.288 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:12 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:12.021+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:13.063 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:13.021+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:13.063 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:13.021+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:13.063 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:13.021+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:13.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T08:53:13.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:13.277 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T08:53:13.277 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T08:53:13.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.6 2024-02-10T08:53:13.343 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.062+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:13.343 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.062+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:13.343 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.062+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:13.595 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 systemd[1]: Stopping Ceph osd.6 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:13.595 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 bash[153568]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.6" found: no such container 2024-02-10T08:53:13.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:13.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:13.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:13.580Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:13.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:13 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:13.580Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:14.003 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.594+0000 7fe9a9311700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:53:14.003 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.594+0000 7fe9a9311700 -1 osd.6 839 *** Got signal Terminated *** 2024-02-10T08:53:14.003 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:13 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:13.594+0000 7fe9a9311700 -1 osd.6 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:53:14.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:14.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:14.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:14.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:14.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:14.068+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:14.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:14.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:14.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:14.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:14.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:14 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:14.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:15.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:15.046+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:15.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:15.046+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:15.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:15.046+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:15.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:15.028+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:15.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:15.028+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:15.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:15 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:15.028+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:16.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.009+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:16.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.009+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:16.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.009+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:16.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:16.068+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:16.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:16.068+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:16.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:16.068+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:17.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:17.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:17.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:17.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:16 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:16.998+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:55.036058+0000 front 2024-02-10T08:52:55.036138+0000 (oldest deadline 2024-02-10T08:53:16.136041+0000) 2024-02-10T08:53:17.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:17.053+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:17.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:17.053+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:17.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:17 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:17.053+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:18.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:18.083+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:18.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:18.083+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:18.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:18.083+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:18.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:18.029+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:30.231381+0000 front 2024-02-10T08:52:30.231484+0000 (oldest deadline 2024-02-10T08:52:54.931292+0000) 2024-02-10T08:53:18.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:18.029+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:37.232029+0000 front 2024-02-10T08:52:37.232105+0000 (oldest deadline 2024-02-10T08:53:02.532185+0000) 2024-02-10T08:53:18.288 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:18.029+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:48.134653+0000 front 2024-02-10T08:52:48.134621+0000 (oldest deadline 2024-02-10T08:53:09.834617+0000) 2024-02-10T08:53:18.288 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:18 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6[82758]: debug 2024-02-10T08:53:18.029+0000 7fe9a428a700 -1 osd.6 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:55.036058+0000 front 2024-02-10T08:52:55.036138+0000 (oldest deadline 2024-02-10T08:53:16.136041+0000) 2024-02-10T08:53:19.287 INFO:journalctl@ceph.osd.6.smithi195.stdout:Feb 10 08:53:19 smithi195 bash[153608]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-6 2024-02-10T08:53:19.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:19.077+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:19.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:19.077+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:19.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:19.077+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:19.288 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:19 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:19.077+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:20.403 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:20.057+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:20.403 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:20.057+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:20.403 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:20.057+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:20.403 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:20.057+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:20.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:20.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:20.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:20.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:20.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:20 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:20.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:20.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T08:53:20.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:20.743 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T08:53:20.744 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T08:53:20.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.7 2024-02-10T08:53:21.022 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 systemd[1]: Stopping Ceph osd.7 for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:21.022 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:20 smithi195 bash[153847]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd.7" found: no such container 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.020+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.020+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.020+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.020+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.067+0000 7faf3cdd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.067+0000 7faf3cdd4700 -1 osd.7 839 *** Got signal Terminated *** 2024-02-10T08:53:21.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:21 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:21.067+0000 7faf3cdd4700 -1 osd.7 839 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T08:53:22.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.012+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:22.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.012+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:22.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.012+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:22.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.012+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:23.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.989+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:23.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.989+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:23.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.989+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:23.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:22 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:22.989+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:23.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:23.580Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:23.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:23.581Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:23.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:23 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:23.581Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:24.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:24.025+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:24.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:24.025+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:24.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:24.025+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:24.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:24 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:24.025+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:25.537 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:25.044+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:25.537 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:25.044+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:25.537 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:25.044+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:25.537 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:25 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:25.044+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:26.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:26.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6803 osd.0 since back 2024-02-10T08:52:27.837737+0000 front 2024-02-10T08:52:27.837839+0000 (oldest deadline 2024-02-10T08:52:53.137674+0000) 2024-02-10T08:53:26.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:26.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6807 osd.1 since back 2024-02-10T08:52:39.539147+0000 front 2024-02-10T08:52:39.539138+0000 (oldest deadline 2024-02-10T08:53:05.438983+0000) 2024-02-10T08:53:26.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:26.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6811 osd.2 since back 2024-02-10T08:52:45.439467+0000 front 2024-02-10T08:52:45.439452+0000 (oldest deadline 2024-02-10T08:53:09.539445+0000) 2024-02-10T08:53:26.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:26.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.107:6815 osd.3 since back 2024-02-10T08:52:54.240534+0000 front 2024-02-10T08:52:54.240604+0000 (oldest deadline 2024-02-10T08:53:18.340659+0000) 2024-02-10T08:53:26.287 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7[87376]: debug 2024-02-10T08:53:26.002+0000 7faf37d4d700 -1 osd.7 839 heartbeat_check: no reply from 172.21.15.195:6802 osd.4 since back 2024-02-10T08:53:03.041410+0000 front 2024-02-10T08:53:03.041441+0000 (oldest deadline 2024-02-10T08:53:25.941467+0000) 2024-02-10T08:53:26.787 INFO:journalctl@ceph.osd.7.smithi195.stdout:Feb 10 08:53:26 smithi195 bash[153887]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-osd-7 2024-02-10T08:53:28.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T08:53:28.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:28.112 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T08:53:28.112 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-10T08:53:28.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@rgw.foo.a 2024-02-10T08:53:28.403 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:28 smithi107 systemd[1]: Stopping Ceph rgw.foo.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:28.404 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:28 smithi107 bash[290967]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw.foo.a" found: no such container 2024-02-10T08:53:28.680 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw-foo-a[94732]: debug 2024-02-10T08:53:28.402+0000 7fb5c7ccb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-10T08:53:28.680 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:28 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw-foo-a[94732]: debug 2024-02-10T08:53:28.402+0000 7fb5ded95880 -1 shutting down 2024-02-10T08:53:30.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:30.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:30.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:30.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:30.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:30 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:30.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:33.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:33.581Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:33.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:33.582Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:33.930 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:33 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:33.582Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:38.680 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:38 smithi107 bash[291006]: time="2024-02-10T08:53:38Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-10T08:53:39.180 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Feb 10 08:53:38 smithi107 bash[291006]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-rgw-foo-a 2024-02-10T08:53:39.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@rgw.foo.a.service' 2024-02-10T08:53:39.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:39.535 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-10T08:53:39.535 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T08:53:39.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-275c2754-c7ee-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T08:53:39.828 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 systemd[1]: Stopping Ceph prometheus.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:39.828 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 bash[154134]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus.a" found: no such container 2024-02-10T08:53:40.106 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T08:53:40.106 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-10T08:53:40.106 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-10T08:53:40.106 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.827Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.828Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.828Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-10T08:53:40.107 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:39 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a[93306]: ts=2024-02-10T08:53:39.828Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-10T08:53:40.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=error ts=2024-02-10T08:53:40.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:40.430 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:40.150Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:40.431 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:40 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:40.150Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:40.532 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Feb 10 08:53:40 smithi195 bash[154175]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T08:53:40.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-275c2754-c7ee-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T08:53:40.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T08:53:40.574 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T08:53:40.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T08:53:41.932 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:41.629Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi107.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.107:8443: connect: connection refused" 2024-02-10T08:53:41.932 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=warn ts=2024-02-10T08:53:41.630Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi195.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.195:8443: connect: connection refused" 2024-02-10T08:53:41.933 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:41 smithi107 systemd[1]: Stopping Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:41.933 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:41 smithi107 bash[291437]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager.a" found: no such container 2024-02-10T08:53:42.322 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:41 smithi107 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a[101316]: level=info ts=2024-02-10T08:53:41.945Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-10T08:53:42.680 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:42 smithi107 bash[291475]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T08:53:43.180 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:42 smithi107 systemd[1]: Stopping Ceph node-exporter.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:43.180 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:42 smithi107 bash[291579]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter.a" found: no such container 2024-02-10T08:53:43.181 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:42 smithi107 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-10T08:53:43.181 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Feb 10 08:53:42 smithi107 systemd[1]: Stopped Ceph alertmanager.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:53:43.680 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:43 smithi107 bash[291620]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T08:53:44.030 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:43 smithi107 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T08:53:44.030 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:43 smithi107 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T08:53:44.030 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Feb 10 08:53:43 smithi107 systemd[1]: Stopped Ceph node-exporter.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:53:44.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T08:53:45.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:45 smithi195 systemd[1]: Stopping Ceph iscsi.iscsi.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:45.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:45 smithi195 bash[154531]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi.iscsi.a" found: no such container 2024-02-10T08:53:45.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:45 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a[91202]: debug Shutdown received 2024-02-10T08:53:55.776 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:55 smithi195 bash[154571]: time="2024-02-10T08:53:55Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-10T08:53:56.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:55 smithi195 bash[154571]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a 2024-02-10T08:53:56.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:56 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T08:53:56.881 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:56 smithi195 systemd[1]: Stopping Ceph node-exporter.b for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:56.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:56 smithi195 bash[154645]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-iscsi-iscsi-a-tcmu 2024-02-10T08:53:56.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:56 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-10T08:53:56.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi195.stdout:Feb 10 08:53:56 smithi195 systemd[1]: Stopped Ceph iscsi.iscsi.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:53:57.287 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:56 smithi195 bash[154722]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter.b" found: no such container 2024-02-10T08:53:57.659 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:57 smithi195 bash[154763]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T08:53:57.936 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:57 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T08:53:57.937 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:57 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T08:53:57.937 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Feb 10 08:53:57 smithi195 systemd[1]: Stopped Ceph node-exporter.b for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:53:58.267 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 systemd[1]: Stopping Ceph grafana.a for 275c2754-c7ee-11ee-95b8-87774f69a715... 2024-02-10T08:53:58.267 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 bash[154897]: Error: no container with name or ID "ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana.a" found: no such container 2024-02-10T08:53:58.537 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a[93672]: t=2024-02-10T08:53:58+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-10T08:53:58.995 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 bash[154936]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715-grafana-a 2024-02-10T08:53:59.248 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 systemd[1]: ceph-275c2754-c7ee-11ee-95b8-87774f69a715@grafana.a.service: Succeeded. 2024-02-10T08:53:59.248 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Feb 10 08:53:58 smithi195 systemd[1]: Stopped Ceph grafana.a for 275c2754-c7ee-11ee-95b8-87774f69a715. 2024-02-10T08:53:59.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:53:59.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:53:59.421 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T08:53:59.423 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/ubuntu@smithi107.front.sepia.ceph.com/crash 2024-02-10T08:53:59.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash -- . 2024-02-10T08:53:59.478 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T08:53:59.478 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-02-10T08:53:59.480 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/ubuntu@smithi195.front.sepia.ceph.com/crash 2024-02-10T08:53:59.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash -- . 2024-02-10T08:53:59.516 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T08:53:59.516 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-02-10T08:53:59.517 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T08:53:59.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T08:53:59.550 INFO:teuthology.orchestra.run.smithi107.stdout:2024-02-10T08:29:43.313177+0000 mon.a (mon.0) 164 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T08:53:59.552 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T08:53:59.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[SEC\]' /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T08:53:59.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]' /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T08:53:59.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[WRN\]' /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-10T08:53:59.766 INFO:teuthology.orchestra.run.smithi107.stdout:2024-02-10T08:29:43.313177+0000 mon.a (mon.0) 164 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T08:53:59.768 INFO:tasks.cephadm:Compressing logs... 2024-02-10T08:53:59.768 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T08:53:59.811 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T08:53:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T08:53:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log: 89.1% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T08:53:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T08:53:59.838 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:: No such file or directory 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T08:53:59.839 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T08:53:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:gzip /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log: -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T08:53:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T08:53:59.840 INFO:teuthology.orchestra.run.smithi195.stderr: 81.6% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T08:53:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/objectstore_tool.7.log 2024-02-10T08:53:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.3.log: /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-10T08:53:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log: 91.3% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T08:53:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T08:53:59.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.x.log: /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T08:53:59.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T08:53:59.850 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log 2024-02-10T08:53:59.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log: 91.8% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T08:53:59.850 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T08:53:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.4.log: 95.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/objectstore_tool.7.log.gz 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr: 91.9% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.5.log: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T08:53:59.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/tcmu-runner.log: 67.6% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T08:53:59.858 INFO:teuthology.orchestra.run.smithi107.stderr: 95.0% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T08:53:59.859 INFO:teuthology.orchestra.run.smithi195.stderr: 85.8% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T08:53:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log 2024-02-10T08:53:59.875 INFO:teuthology.orchestra.run.smithi107.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T08:53:59.883 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T08:53:59.892 INFO:teuthology.orchestra.run.smithi107.stderr: 92.0% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T08:54:00.046 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.1.log: 93.4% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-10T08:54:00.262 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T08:54:01.156 INFO:teuthology.orchestra.run.smithi195.stderr: 92.1% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T08:54:01.516 INFO:teuthology.orchestra.run.smithi107.stderr: 91.6% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T08:54:02.284 INFO:teuthology.orchestra.run.smithi107.stderr: 94.1% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T08:54:02.298 INFO:teuthology.orchestra.run.smithi107.stderr: 91.4% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T08:54:05.406 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T08:54:05.498 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T08:54:05.650 INFO:teuthology.orchestra.run.smithi195.stderr: 94.1% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T08:54:05.763 INFO:teuthology.orchestra.run.smithi195.stderr: 94.1% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T08:54:05.764 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-02-10T08:54:05.765 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m5.936s 2024-02-10T08:54:05.765 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m23.437s 2024-02-10T08:54:05.765 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.870s 2024-02-10T08:54:05.828 INFO:teuthology.orchestra.run.smithi107.stderr: 94.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T08:54:06.230 INFO:teuthology.orchestra.run.smithi107.stderr: 94.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T08:54:06.764 INFO:teuthology.orchestra.run.smithi107.stderr: 94.2% -- replaced with /var/log/ceph/275c2754-c7ee-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T08:54:06.766 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-02-10T08:54:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m6.941s 2024-02-10T08:54:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m25.571s 2024-02-10T08:54:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.941s 2024-02-10T08:54:06.767 INFO:tasks.cephadm:Archiving logs... 2024-02-10T08:54:06.767 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/ubuntu@smithi107.front.sepia.ceph.com/log 2024-02-10T08:54:06.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T08:54:10.555 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/ubuntu@smithi195.front.sepia.ceph.com/log 2024-02-10T08:54:10.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T08:54:14.066 INFO:tasks.cephadm:Removing cluster... 2024-02-10T08:54:14.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 --force 2024-02-10T08:54:14.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 275c2754-c7ee-11ee-95b8-87774f69a715 --force 2024-02-10T08:54:15.042 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T08:54:15.043 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T08:54:15.062 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T08:54:15.080 INFO:tasks.cephadm:Teardown complete 2024-02-10T08:54:15.080 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T08:54:15.090 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T08:54:15.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T08:54:15.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T08:54:15.199 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T08:54:15.209 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T08:54:15.210 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:54:15.212 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:54:15.227 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2024-02-10T08:54:15.228 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2024-02-10T08:54:15.232 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:54:15.232 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-02-10T08:54:15.233 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +104us[ +104us] +/- 83ms 2024-02-10T08:54:15.233 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 12 +187us[ +191us] +/- 58ms 2024-02-10T08:54:15.233 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv03.front.sepia.ceph.com 3 6 377 11 -270us[ -266us] +/- 76ms 2024-02-10T08:54:15.233 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +102us[ +102us] +/- 83ms 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 +189us[ +211us] +/- 57ms 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -274us[ -274us] +/- 76ms 2024-02-10T08:54:15.234 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:54:15.234 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T08:54:15.245 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T08:54:15.245 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T08:54:15.288 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T08:54:15.380 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T08:54:15.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:54:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:54:15.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:54:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:54:15.529 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T08:54:15.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T08:54:15.588 INFO:teuthology.task.internal:Duration was 2255.035005 seconds 2024-02-10T08:54:15.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T08:54:15.639 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T08:54:15.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T08:54:15.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T08:54:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:54:15.688 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:54:16.046 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T08:54:16.046 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-02-10T08:54:16.047 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T08:54:16.069 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-02-10T08:54:16.069 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T08:54:16.092 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T08:54:16.093 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T08:54:16.111 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T08:54:16.226 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T08:54:16.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T08:54:16.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T08:54:16.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T08:54:16.533 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T08:54:16.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T08:54:16.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T08:54:16.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T08:54:16.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:54:16.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:54:16.626 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-02-10T08:54:16.653 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-02-10T08:54:16.689 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:54:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:54:16.706 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:54:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:54:16.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T08:54:16.758 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T08:54:16.758 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/smithi107 2024-02-10T08:54:16.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T08:54:16.823 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278/remote/smithi195 2024-02-10T08:54:16.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T08:54:16.877 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T08:54:16.877 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T08:54:16.879 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T08:54:16.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T08:54:16.936 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T08:54:16.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T08:54:16.964 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T08:54:16.965 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T08:54:16.967 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T08:54:16.982 INFO:teuthology.orchestra.run.smithi107.stdout: 525742 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 08:54 /home/ubuntu/cephtest 2024-02-10T08:54:16.984 INFO:teuthology.orchestra.run.smithi195.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 08:54 /home/ubuntu/cephtest 2024-02-10T08:54:16.986 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T08:54:17.013 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T08:54:17.114 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T08:54:17.142 DEBUG:teuthology.nuke:shortname: smithi107 2024-02-10T08:54:17.142 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T08:54:17.167 DEBUG:teuthology.nuke:shortname: smithi195 2024-02-10T08:54:17.167 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T08:54:17.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:39:28.462834', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMtn8ILoP4HsAh7Ccwf7LEa4Ga+FAKgUtwKtiedwvBSIwiFxvoItbSI56QhzmKcEEWeDL/2t0sU/IrUVX4ZR+w='} 2024-02-10T08:54:17.191 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:39:28.464164', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5zvCgoyqaesxlZVllbXdfOTzJx7alf2x1fJNVN5QFM8hnn7LeBY3y4/Bg7B6N3csAJOMjOk9k39nbTE3iCqgo='} 2024-02-10T08:54:17.221 INFO:teuthology.orchestra.console.smithi107:Power off 2024-02-10T08:54:17.222 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T08:54:17.248 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-02-10T08:54:17.249 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:17.264 INFO:teuthology.orchestra.console.smithi195:Power off 2024-02-10T08:54:17.265 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T08:54:17.274 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-02-10T08:54:17.287 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-02-10T08:54:17.288 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:17.312 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-10T08:54:21.275 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:21.313 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:21.387 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-02-10T08:54:21.425 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-10T08:54:25.392 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:25.426 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:25.503 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-02-10T08:54:25.538 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-02-10T08:54:25.639 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-02-10T08:54:29.508 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:54:29.617 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-02-10T08:54:29.718 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-02-10T08:54:29.822 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 2255.0350046157837 failure_reason: '"2024-02-10T08:29:43.313177+0000 mon.a (mon.0) 164 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-10T08:54:29.822 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T08:54:29.910 INFO:teuthology.run:FAIL